Node.js monitoring and alerts with practical examples: practical implementation guide

Node.js apps are fast to build, but they only feel fast when latency, errors, and dependencies are under control. This guide shows exactly what to monitor, how to instrument useful metrics and logs, how to write alert rules that reduce noise, how to build a focused dashboard, and how to run a simple incident workflow. You can pilot everything locally first, then scale out safely.

Workflow Overview

A pragmatic flow to get from signals to action:

1. Identify signals that reflect user experience and service health.
2. Instrument metrics and structured logs in your Node.js service.
3. Collect and store data in your monitoring stack.
4. Visualize with a focused dashboard for fast triage.
5. Define actionable alerts with clear thresholds and durations.
6. Respond with a lightweight incident workflow and verify recovery.
7. Review and refine thresholds, panels, and runbooks after each event.

What To Monitor In Node.js

Track a small set of high-value signals first.

Golden signals

• Latency: p50, p95, p99 for key routes and overall service.
• Throughput: requests per second.
• Errors: 5xx rate and error types.
• Saturation: CPU, memory (RSS and heap), event loop lag.

Node.js internals

• Event loop lag: sustained lag indicates saturation or GC pressure.
• Garbage collection: major GC time and frequency.
• Memory: heap used vs max, RSS growth (leak suspicion if monotonically rising under steady load).
• Process restarts: unexpected exits or crash loops.

Dependencies

• MongoDB: operation latency, error count, connection pool in-use vs max.
• Redis: command latency, timeouts, connection errors, queue/backlog depth.
• External APIs: call latency, non-2xx rate, circuit breaker opens.

Health and readiness

• Healthcheck status and time to respond.
• Queue depth and worker success/failure rate.

Instrumentation Examples

Below is a minimal Express app with Prometheus-style metrics using prom-client, event loop lag measurement, and structured JSON logs with a request ID.

// app.js
'use strict';

const express = require('express');
const { performance, monitorEventLoopDelay } = require('perf_hooks');
const client = require('prom-client');
const crypto = require('crypto');

const app = express();
const register = new client.Registry();

// Label all metrics with a service name for easy filtering
const serviceName = process.env.SERVICE_NAME || 'api';
register.setDefaultLabels({ service: serviceName });

// Default Node.js process metrics (CPU, memory, GC, etc.)
client.collectDefaultMetrics({ register });

// Event loop lag gauge (seconds)
const ell = monitorEventLoopDelay({ resolution: 20 });
ell.enable();
const eventLoopLag = new client.Gauge({
  name: 'event_loop_lag_seconds',
  help: 'Event loop lag in seconds (mean over scrape interval)',
});
register.registerMetric(eventLoopLag);
setInterval(() => {
  eventLoopLag.set(ell.mean / 1é); // mean is in nanoseconds
  ell.reset();
}, 1000);

// HTTP request duration histogram
const httpDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5]
});
register.registerMetric(httpDuration);

// Total error counter
const httpErrors = new client.Counter({
  name: 'http_request_errors_total',
  help: 'Total number of failed HTTP requests',
  labelNames: ['route', 'status_code', 'cause']
});
register.registerMetric(httpErrors);

// Simple request ID middleware for log correlation
app.use((req, res, next) => {
  req.id = req.headers['x-request-id'] || crypto.randomUUID();
  res.setHeader('x-request-id', req.id);
  next();
});

// Timing + logging middleware
app.use(async (req, res, next) => {
  const start = performance.now();
  res.on('finish', () => {
    const ms = performance.now() - start;
    const route = req.route ? req.route.path : req.path;
    httpDuration.labels(req.method, route, String(res.statusCode)).observe(ms / 1000);
    const log = {
      ts: new Date().toISOString(),
      level: res.statusCode >= 500 ? 'error' : 'info',
      msg: 'request',
      req_id: req.id,
      method: req.method,
      route,
      status: res.statusCode,
      latency_ms: Math.round(ms),
      service: serviceName
    };
    // Emit to stdout; forward by your log collector
    process.stdout.write(JSON.stringify(log) + '\n');
  });
  next();
});

// Sample routes
app.get('/api/todos', (req, res) => {
  // Simulate variable work
  setTimeout(() => res.json([{ id: 1, title: 'learn monitoring' }]), Math.random() * 80);
});

app.get('/api/error', (req, res) => {
  const err = new Error('forced');
  httpErrors.labels('/api/error', '500', 'forced').inc();
  res.status(500).json({ error: err.message });
});

// Metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', register.contentType);
  res.end(await register.metrics());
});

const port = process.env.PORT || 3000;
app.listen(port, () => {
  // Simple startup log
  const log = { ts: new Date().toISOString(), level: 'info', msg: 'listening', port, service: serviceName };
  process.stdout.write(JSON.stringify(log) + '\n');
});

Sample structured logs (stdout) for quick visual checks:

{"ts":"2025-01-01T12:00:00.000Z","level":"info","msg":"request","req_id":"c1b2","method":"GET","route":"/api/todos","status":200,"latency_ms":42,"service":"api"}
{"ts":"2025-01-01T12:00:01.000Z","level":"error","msg":"request","req_id":"c1b3","method":"GET","route":"/api/error","status":500,"latency_ms":3,"service":"api"}

Alert Rules And Log Signals

Start with alerts that reflect user pain and sustained saturation. Below are Prometheus-style alert examples that match the metric names above.

Latency SLO breach (p95 > 300 ms for 10 minutes):

groups:
  - name: node-api-latency
    rules:
      - alert: NodeServiceHighLatencyP95
        expr: |
          histogram_quantile(
            0.95,
            sum(rate(http_request_duration_seconds_bucket{service="api"}[5m])) by (le)
          ) > 0.3
        for: 10m
        labels:
          severity: page
          service: api
        annotations:
          summary: "High latency p95 on api"
          description: "p95 latency > 300ms for 10m"

Error rate over 2% for 10 minutes (uses histogram _count as total requests):

groups:
  - name: node-api-errors
    rules:
      - alert: NodeServiceHighErrorRate
        expr: |
          sum(rate(http_request_errors_total{service="api"}[5m]))
          /
          sum(rate(http_request_duration_seconds_count{service="api"}[5m]))
          > 0.02
        for: 10m
        labels:
          severity: page
          service: api
        annotations:
          summary: "High error rate on api"
          description: ">2% errors for 10m"

Event loop lag sustained above 200 ms for 5 minutes:

groups:
  - name: node-api-ell
    rules:
      - alert: NodeServiceEventLoopLagHigh
        expr: event_loop_lag_seconds{service="api"} > 0.2
        for: 5m
        labels:
          severity: ticket
          service: api
        annotations:
          summary: "Event loop lag high"
          description: ">200ms lag for 5m suggests saturation or GC pressure"

Optional dependency alerts (adapt names to your gauges):

• MongoDB pool saturation: db_pool_in_use / db_pool_max > 0.8 for 10m.
• Redis backlog: worker_queue_backlog > 1000 for 10m.

Log signals to watch

• 5xx spikes by route:
• Filter: level=error AND status>=500, group by route.
• Dependency errors:
• Mongo network errors: message contains "MongoNetworkError" or code "ETIMEDOUT".
• Redis timeouts: message contains "RedisTimeoutError" or "ECONNRESET".
• Slow responders:
• latency_ms>500 with status<500 indicates slowness without outright failure.

Keep alerts actionable

• Tie alerts to user-visible symptoms (latency and error rate) before internal metrics.
• Use a time window (for:) to avoid paging on transients.
• Route noisy internals (like brief GC pauses) to a ticket or notification instead of a page.

Dashboards That Drive Action

Organize around decisions, not data dumps.

Top panel row (golden signals)

• Latency p50/p95/p99 (overall and top 3 routes):
• Query: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="api"}[5m])) by (le, route))
• Throughput (rps):
• Query: sum(rate(http_request_duration_seconds_count{service="api"}[1m]))
• Error rate (%):
• Query: 100 * sum(rate(http_request_errors_total{service="api"}[5m])) / sum(rate(http_request_duration_seconds_count{service="api"}[5m]))

Resource health

• Event loop lag (seconds): event_loop_lag_seconds{service="api"}
• Memory RSS and heap used: process_resident_memory_bytes, nodejs_heap_size_used_bytes
• CPU user/system: process_cpu_user_seconds_total, process_cpu_system_seconds_total (as rates)

Drilldowns

• Slowest routes: sort p95 by route.
• Dependency latency and errors: panels per MongoDB/Redis metric.
• Recent deploy marker: include a build/version label on metrics (e.g., default labels) and annotate changes.

Link to logs

• Add dashboard links that pass route and x-request-id to your log viewer to jump from a slow panel to exact requests.

Incident Response Workflows

Aim for fast triage and safe mitigation.

1. Acknowledge and size impact

• Check golden signals: is this user-visible (latency or error rate)?
• Identify blast radius: single route, entire service, or dependency.

1. Form a hypothesis in 2-3 minutes

• High latency with normal CPU: likely dependency slowness.
• High event loop lag and rising RSS: GC pressure or synchronous work on the main thread.
• Error rate spike after a change: regression or config drift.

1. Mitigate

• Roll back or disable the risky change.
• Shed load carefully (rate limits) on specific routes if needed.
• Restart misbehaving replicas only if you understand the impact.

1. Verify recovery

• Latency and error rate return to normal bands.
• Event loop lag stabilizes.
• Logs show reduced 5xx and dependency errors.

1. Capture notes and improve

• Record the trigger, fix, and follow-ups (dashboards to add, thresholds to tune, code to optimize).

Runbook starters

• Memory/RSS rising steadily: take a heap snapshot in a safe environment, check for large buffers or caches without TTL.
• Dependency slow: add timeouts and retries with jitter; consider a circuit breaker and bulkhead isolation for hot paths.
• Event loop lag: move CPU-heavy or blocking work to a worker thread or separate process; avoid sync filesystem and crypto calls on the main thread.

Local Pilot Plan

Start small, measure, and inspect locally before broad rollout.

Scope

• One service, two routes (/api/todos and /api/error).
• Metrics: request duration histogram, error counter, event loop lag gauge, default process metrics.
• Alerts: p95 latency, error rate.
• Dashboard: golden signals and event loop lag.

Steps

1. Add the middleware and metrics from the sample app.
2. Expose /metrics and confirm it returns metrics text.
3. Generate load and errors locally:

• Hit /api/todos repeatedly to build latency histograms.
• Hit /api/error to increment error counters.

1. Validate alerts by temporarily lowering thresholds to trigger within a few minutes, then restore sane levels.
2. Build a minimal dashboard with 4 panels: p95 latency, rps, error rate, event loop lag.
3. Add a request ID to client requests (x-request-id) and confirm logs and responses include it for correlation.

Exit criteria

• You can see p50/p95 move under load and stabilize after load stops.
• Error rate alert triggers only during forced failures and quiets when they stop.
• Event loop lag remains near zero under normal load and rises under stress tests.

Scale-out

• Add route labels to your top 5 endpoints.
• Add dependency metrics next (MongoDB pool usage, Redis latency/backlog).
• Roll to one production service at a time and verify panels and alerts.

Conclusion

Effective Node.js monitoring starts with a few high-signal metrics: latency, throughput, error rate, and saturation. Instrument them with clear labels, expose a /metrics endpoint, and emit structured logs with request IDs. Keep the first alerts simple and user-centric, and build a dashboard that makes triage fast. Pilot locally, verify behavior, then expand to more endpoints and dependencies. Tune thresholds after real incidents and keep runbooks short and actionable.