Express API monitoring and alerts with practical examples: practical implementation guide

Intro

Monitoring is how you turn an Express API from a black box into a service you can trust. In this guide you will:

• Add production-grade metrics and logs to an Express API
• Define alert rules that catch real issues without noisy pages
• Build dashboards that answer on-call questions fast
• Practice an incident response flow that shortens MTTR

This is hands-on. You will get code, rules, and a small local pilot you can run before touching production.

Workflow Overview

A clear, staged approach reduces rework and confusion across roles and helps teams move from ideas to working implementations.

1. Instrument

• Add metrics and logs in the API process.
• Expose a /metrics endpoint and health probes.

1. Collect

• Scrape or push metrics to your time-series store.
• Ship structured logs to a searchable store.

1. Visualize

• Build dashboards for golden signals and top slow routes.

1. Alert

• Define high-signal rules tied to user impact and SLOs.

1. Respond

• Triage, mitigate, and communicate using a short runbook.

1. Improve

• Tune thresholds, add missing labels, and fix noisy alerts.

Key Metrics and Signals

Express sits at the HTTP edge. Focus first on the golden signals: latency, traffic, errors, and saturation.

Metrics to collect

• Requests total: counter labeled by method, route, and status_code.
• Errors total: counter, filter status_code >= 500.
• Request duration: histogram in milliseconds, labeled by method, route, status_code.
• Active requests: gauge to catch saturation during spikes.
• Process metrics: event loop lag, heap used, CPU, open handles.
• Dependency latency/errors: outbound HTTP, DB calls (e.g., MongoDB), cache.

Express instrumentation example

Add lightweight metrics using prom-client. Labels should be low-cardinality. Prefer route templates (e.g., /users/:id) over raw paths.

// metrics.js
const client = require("prom-client");

// Default Node.js process metrics
client.collectDefaultMetrics({ prefix: "node_" });

const httpRequestDurationMs = new client.Histogram({
  name: "http_request_duration_ms",
  help: "Duration of HTTP requests in ms",
  labelNames: ["method", "route", "status_code"],
  buckets: [10, 50, 100, 300, 500, 1000, 2000, 5000]
});

const httpRequestsTotal = new client.Counter({
  name: "http_requests_total",
  help: "Total HTTP requests",
  labelNames: ["method", "route", "status_code"]
});

const httpRequestsErrorsTotal = new client.Counter({
  name: "http_requests_errors_total",
  help: "Total HTTP 5xx responses",
  labelNames: ["method", "route"]
});

const activeRequests = new client.Gauge({
  name: "http_active_requests",
  help: "In-flight HTTP requests",
  labelNames: ["route"]
});

function metricsMiddleware(req, res, next) {
  const route = req.route && req.route.path ? req.route.path : req.path || "unknown";
  activeRequests.labels(route).inc();
  const end = httpRequestDurationMs.startTimer();

  res.on("finish", () => {
    const status = res.statusCode;
    const labels = { method: req.method, route, status_code: String(status) };
    httpRequestsTotal.labels(labels.method, labels.route, labels.status_code).inc();
    if (status >= 500) {
      httpRequestsErrorsTotal.labels(labels.method, labels.route).inc();
    }
    end(labels);
    activeRequests.labels(route).dec();
  });

  next();
}

async function metricsEndpoint(req, res) {
  res.set("Content-Type", client.register.contentType);
  res.end(await client.register.metrics());
}

module.exports = {
  metricsMiddleware,
  metricsEndpoint,
};

Integrate with Express:

// server.js
const express = require("express");
const { metricsMiddleware, metricsEndpoint } = require("./metrics");
const pino = require("pino");
const pinoHttp = require("pino-http");

const app = express();
const logger = pino({ level: process.env.LOG_LEVEL || "info" });
app.use(pinoHttp({ logger, genReqId: () => crypto.randomUUID() }));
app.use(express.json());
app.use(metricsMiddleware);

app.get("/healthz", (req, res) => res.status(200).send("ok"));
app.get("/readyz", (req, res) => res.status(200).send("ready"));

// Example routes
app.get("/users/:id", async (req, res) => {
  // Simulate work
  await new Promise(r => setTimeout(r, Math.random() * 200));
  res.json({ id: req.params.id });
});

app.get("/error", (req, res) => {
  res.status(500).json({ error: "boom" });
});

app.get("/metrics", metricsEndpoint);

const port = process.env.PORT || 3000;
app.listen(port, () => logger.info({ port }, "api-listening"));

Tip: ensure route labels use templates (e.g., /users/:id). Many routers expose req.route.path after matching. Fallbacks to raw paths can cause high cardinality.

Alert Rules That Matter

Alert rules should be tied to user-visible impact and sustained conditions. Avoid single-sample spikes. Use a short-for page window (5-15m) and temper with for clauses.

Below are example rules in a Prometheus-like syntax. Adapt names to your metric schema and stack.

Error rate

Page when 5xx ratio exceeds 2% for 5 minutes.

- alert: HighErrorRate
  expr: (
    sum(rate(http_requests_errors_total[5m]))
    /
    sum(rate(http_requests_total[5m]))
  ) > 0.02
  for: 5m
  labels:
    severity: page
  annotations:
    summary: "Elevated 5xx error rate (>2% for 5m)"
    description: "Investigate recent deploys, dependencies, and logs."

High latency (p95)

Page when p95 latency exceeds 300 ms for 10 minutes.

- alert: HighLatencyP95
  expr: |
    histogram_quantile(0.95,
      sum(rate(http_request_duration_ms_bucket[5m])) by (le)
    ) > 0.3
  for: 10m
  labels:
    severity: page
  annotations:
    summary: "p95 latency > 300ms for 10m"
    description: "Check saturated workers, DB, and hot routes."

Note: Duration in seconds in many systems. 0.3 s == 300 ms.

Saturation

Page when in-flight requests are consistently high.

- alert: SaturatedWorkers
  expr: avg_over_time(http_active_requests[5m]) > 100
  for: 10m
  labels:
    severity: page
  annotations:
    summary: "High in-flight HTTP requests"
    description: "Scale out or shed load if needed."

Tune the threshold to your concurrency capacity.

No traffic or sudden drop

Warn when total request rate drops near zero during expected traffic hours.

- alert: NoTraffic
  expr: sum(rate(http_requests_total[5m])) < 1
  for: 10m
  labels:
    severity: warn
  annotations:
    summary: "Request rate near zero"
    description: "Possible upstream issue, DNS, or LB route."

Logs and Traces

When an alert fires, you need fast answers. Structure logs for search and correlation.

Recommended log fields

• ts: ISO timestamp
• level: info, warn, error
• msg: short message
• request_id: a UUID generated per request
• method, route, path
• status, latency_ms
• user_id or account_id (if available and safe)
• remote_ip, user_agent (watch PII policies)
• service, version, commit_sha

Pino example with correlation

const pino = require("pino");
const pinoHttp = require("pino-http");
const { randomUUID } = require("crypto");

const logger = pino({ level: process.env.LOG_LEVEL || "info" });

app.use(pinoHttp({
  logger,
  genReqId: (req) => req.headers["x-request-id"] || randomUUID(),
  customLogLevel: function (res, err) {
    if (res.statusCode >= 500 || err) return "error";
    if (res.statusCode >= 400) return "warn";
    return "info";
  },
  customSuccessMessage: function (req, res) {
    return "request-complete";
  },
  customErrorMessage: function (req, res, err) {
    return "request-error";
  },
  serializers: {
    req(req) {
      return {
        id: req.id,
        method: req.method,
        route: req.route && req.route.path ? req.route.path : req.url,
      };
    },
    res(res) {
      return { statusCode: res.statusCode };
    },
  },
}));

Useful log signals

• Spikes in 5xx and 499/ClientClosedRequest equivalents
• Timeouts (ETIMEDOUT), resets (ECONNRESET), pool exhaustion
• Repeated errors tied to one route, tenant, or version
• Long tails: a few requests taking seconds while median is low

If you adopt distributed tracing later, start by propagating request_id as trace id headers through your stack.

Dashboards That Work

Dashboards should answer the top 5 on-call questions in 1 minute.

Recommended panels:

• Traffic: requests per second, by route and method
• Latency: p50, p95, p99 trends; top N slow routes now
• Errors: error rate by status_code and route
• Saturation: in-flight requests, event loop lag, CPU, heap used, GC pauses
• Dependencies: DB latency and errors, outbound HTTP
• Version breakdown: metrics split by version/commit for quick rollback calls

Layout tips:

• Keep a single-page triage view with linked driller dashboards per service.
• Annotate with deploy markers.
• Use consistent colors and units; milliseconds for latency is readable at a glance.

Incident Response Workflow

A lightweight, repeatable flow turns alerts into fast recovery.

1. Acknowledge and assess

• Confirm the alert and check the triage dashboard.
• Identify user impact (error rate, latency, drop in traffic).

1. Stabilize

• Roll back recent changes if metrics point to a regression.
• Scale out, add a circuit breaker, or rate limit noisy tenants.

1. Diagnose

• Filter logs by request_id, route, and version.
• Compare hot routes and dependency panels.

1. Mitigate

• Ship a config fix, feature flag, or hot patch.
• Add temporary timeouts or retries for flaky dependencies.

1. Verify

• Ensure alerts clear and golden signals return to baseline.

1. Learn

• Capture what detected the issue, what helped, and what was missing.
• Tune alerts and dashboards. Add labels or new metrics if needed.

Keep a short runbook per alert with owner, checks, and remedies.

Local Pilot Plan

Start small, measure, and inspect locally before rollout. A narrow, measurable pilot reduces risk and clarifies next steps.

1) Run the API locally

Install dependencies:

npm i express prom-client pino pino-http
node server.js

Hit endpoints and check metrics:

curl -s localhost:3000/healthz
curl -s localhost:3000/users/123
curl -s localhost:3000/error
curl -s localhost:3000/metrics | head

2) Optional: scrape metrics and alert locally

Use a minimal docker-compose to run a metrics store, a dashboard, and an alerting component.

version: "3.9"
services:
  api:
    build: .
    ports: ["3000:3000"]
  prometheus:
    image: prom/prometheus: latest
    ports: ["9090:9090"]
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml: ro
  alertmanager:
    image: prom/alertmanager: latest
    ports: ["9093:9093"]
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml: ro
  grafana:
    image: grafana/grafana: latest
    ports: ["3001:3000"]

Example prometheus.yml:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: "express-api"
    metrics_path: /metrics
    static_configs:
      - targets: ["api:3000"]
rule_files:
  - rules.yml

Example rules.yml with two alerts:

groups:
- name: express
  rules:
  - alert: HighErrorRate
    expr: (sum(rate(http_requests_errors_total[5m])) / sum(rate(http_requests_total[5m]))) > 0.02
    for: 5m
    labels: { severity: page }
    annotations:
      summary: "Elevated 5xx error rate"
  - alert: HighLatencyP95
    expr: histogram_quantile(0.95, sum(rate(http_request_duration_ms_bucket[5m])) by (le)) > 0.3
    for: 10m
    labels: { severity: page }
    annotations:
      summary: "p95 latency > 300ms"

3) Generate test traffic

# steady good traffic
for i in $(seq 1 200); do curl -s localhost:3000/users/$i > /dev/null; done

# induce latency and errors
curl -s localhost:3000/error > /dev/null

Watch the dashboard and validate that alerts trigger only when conditions persist.

4) Define pass-fail and document

• Pass if metrics populate, dashboards render, and alerts fire and clear as expected.
• Fail if labels explode in cardinality, alerts are noisy, or metrics are missing.
• Capture thresholds, label sets, and any gaps to fix before staging rollout.

Conclusion

You now have a practical baseline for Express API monitoring:

• Golden-signal metrics and low-cardinality labels
• Actionable alert rules that reflect user impact
• Logs that correlate requests and speed diagnosis
• Dashboards that answer on-call questions fast
• A simple incident response flow and a safe local pilot

Next steps:

• Tune thresholds to your real traffic and capacity
• Add dependency metrics for databases and outbound calls
• Expand logs with request_id propagation across services
• Pilot on a single service in staging, then roll out gradually

A small, measurable start and a clear workflow help teams move from ideas to a reliable, observable Express API.