## Intro

Monitoring turns an Express API from a black box into a service you can trust. In this guide you will:

- Add production-grade metrics and logs to an Express API

- Define alert rules that catch real issues without noisy pages

- Build dashboards that answer on-call questions fast

- Practice an incident response flow that reduces mean time to resolution

This is hands-on. You will get code, rules, and a small local pilot you can run before touching production.

## Workflow Overview

A clear, staged approach reduces rework and helps teams move from ideas to working implementations.

- Instrument

- Add metrics and logs in the API process

- Expose a /metrics endpoint and health probes

- Collect

- Scrape or push metrics to your time-series store

- Ship structured logs to a searchable store

- Visualize

- Build dashboards for golden signals and top slow routes

- Alert

- Define high-signal rules tied to user impact and SLOs

- Respond

- Triage, mitigate, and communicate using a short runbook

- Improve

- Tune thresholds, add missing labels, and fix noisy alerts

## Key Metrics and Signals

Express sits at the HTTP edge. Focus first on the golden signals: latency, traffic, errors, and saturation.

### Metrics to collect

- Requests total: counter labeled by method, route, and status_code

- Errors total: counter, filter status_code >= 500

- Request duration: histogram in milliseconds, labeled by method, route, status_code

- Active requests: gauge to catch saturation during spikes

- Process metrics: event loop lag, heap used, CPU, open handles

- Dependency latency/errors: outbound HTTP, DB calls (e.g., MongoDB), cache

### Express instrumentation example

Add lightweight metrics using prom-client . Labels should be low-cardinality. Prefer route templates (e.g., /users/:id ) over raw paths.

// metrics.js
const client = require("prom-client");

// Default Node.js process metrics
client.collectDefaultMetrics({ prefix: "node_" });

const httpRequestDurationMs = new client.Histogram({
 name: "http_request_duration_ms",
 help: "Duration of HTTP requests in ms",
 labelNames: ["method", "route", "status_code"],
 buckets: [10, 50, 100, 300, 500, 1000, 2000, 5000]
});

const httpRequestsTotal = new client.Counter({
 name: "http_requests_total",
 help: "Total HTTP requests",
 labelNames: ["method", "route", "status_code"]
});

const httpRequestsErrorsTotal = new client.Counter({
 name: "http_requests_errors_total",
 help: "Total HTTP 5xx responses",
 labelNames: ["method", "route" ]
});

const activeRequests = new client.Gauge({
 name: "http_active_requests",
 help: "In-flight HTTP requests",
 labelNames: ["route" ]
});

function metricsMiddleware(req, res, next) {
 const route = req.route && req.route.path ? req.route.path : req.path || "unknown\n activeRequests.labels(route).inc();
 const end = httpRequestDurationMs.startTimer();

 res.on("finish", () => {
 const status = res.statusCode;
 const labels = { method: req.method, route, status_code: String(status) };
 httpRequestsTotal.labels(labels.method, labels.route, labels.status_code).inc();
 if (status >= 500) {
 httpRequestsErrorsTotal.labels(labels.method, labels.route).inc();
 }
 end(labels);
 activeRequests.labels(route).dec();
 });

 next();
}

async function metricsEndpoint(req, res) {
 res.set("Content-Type", client.register.contentType);
 res.end(await client.register.metrics());
}

module.exports = {
 metricsMiddleware,
 metricsEndpoint
}; 
 Integrate with Express:

// server.js
const express = require("express");
const { metricsMiddleware, metricsEndpoint } = require("metrics");
const pino = require("pino");
const pinoHttp = require("pino-http");

const app = express();
const logger = pino({ level: process.env.LOG_LEVEL || "info" });
app.use(pinoHttp({ logger, genReqId: () => crypto.randomUUID() }));
app.use(express.json());
app.use(metricsMiddleware);

app.get("healthz", (req, res) => res.status(200).send("ok\
app.get("readyz", (req, res) => res.status(200).send("ready\);

// Example routes
app.get("users/:id", async (req, res) => {
 // Simulate work
 await new Promise(r => setTimeout(r, Math.random() * 200));
 res.json({ id: req.params.id });
});

app.get("error", (req, res) => {
 res.status(500).json({ error: "boom" });
});

app.get("metrics", metricsEndpoint);

const port = process.env.PORT || 3000;
app.listen(port, () => logger.info({ port }, "api-listening\); 
 Tip: Ensure route labels use templates (e.g., /users/:id ). Most routers expose req.route.path after matching. If your router doesn't, add a small mapping layer — falling back to raw paths causes high cardinality that breaks Prometheus.

## Alert Rules That Matter

Alert rules should be tied to user-visible impact and sustained conditions. Avoid single-sample spikes. Use a short-for page window (5–15 min) and temper with for clauses.

Below are example rules in Prometheus syntax. Adapt metric names to your schema and stack.

### Error rate

Page when 5xx ratio exceeds 2% for 5 minutes.

- alert: HighErrorRate
 expr: (
 sum(rate(http_requests_errors_total[5m]))
 /
 sum(rate(http_requests_total[5m]))
 ) > 0.02
 for: 5m
 labels:
 severity: page
 annotations:
 summary: "Elevated 5xx error rate (>2% for 5m)"\n description: "Investigate recent deploys, dependencies, and logs." 

### High latency (p95)

 Page when p95 latency exceeds 300 ms for 10 minutes.

- alert: HighLatencyP95
 expr: |
 histogram_quantile(0.95,
 sum(rate(http_request_duration_ms_bucket[5m])) by (le)
 ) > 0.3
 for: 10m
 labels:
 severity: page
 annotations:
 summary: "p95 latency > 300ms for 10m"\n description: "Check saturated workers, DB, and hot routes." 

 Note: Prometheus stores durations in seconds. 0.3 == 300 ms.

</div>

### Saturation

Page when in-flight requests are consistently high relative to your concurrency limit.

- alert: SaturatedWorkers
 expr: avg_over_time(http_active_requests[5m]) > 100
 for: 10m
 labels:
 severity: page
 annotations:
 summary: "High in-flight HTTP requests"\n description: "Scale out or shed load if needed." 
 Tune the threshold to your actual concurrency capacity (e.g., worker count × max connections per worker).

### No traffic or sudden drop

Warn when total request rate drops near zero during expected traffic hours.

- alert: NoTraffic
 expr: sum(rate(http_requests_total[5m])) < 1
 for: 10m
 labels:
 severity: warn
 annotations:
 summary: "Request rate near zero"\n description: "Possible upstream issue, DNS, or LB route." 

## Logs and Traces

 When an alert fires, you need fast answers. Structure logs for search and correlation.

### Recommended log fields

- ts : ISO timestamp

- level : info, warn, error

- msg : short message

- request_id : a UUID generated per request

- method , route , path

- status , latency_ms

- user_id or account_id (if available and safe)

- remote_ip , user_agent (watch PII policies)

- service , version , commit_sha

### Pino example with correlation

const pino = require("pino");
const pinoHttp = require("pino-http");
const { randomUUID } = require("crypto");

const logger = pino({ level: process.env.LOG_LEVEL || "info" });

app.use(pinoHttp({
 logger,
 genReqId: (req) => req.headers["x-request-id\) || randomUUID(),
 customLogLevel: function (res, err) {
 if (res.statusCode >= 500 || err) return "error
 if (res.statusCode >= 400) return "warn
 return "info
 },
 customSuccessMessage: function (req, res) {
 return "request-complete
 },
 customErrorMessage: function (req, res, err) {
 return "request-error
 },
 serializers: {
 req(req) {
 return {
 id: req.id,
 method: req.method,
 route: req.route && req.route.path ? req.route.path : req.url,
 };
 },
 res(res) {
 return { statusCode: res.statusCode };
 },
 },
})); 

### Useful log signals

- Spikes in 5xx and 499/ClientClosedRequest equivalents

- Timeouts ( ETIMEDOUT ), resets ( ECONNRESET ), pool exhaustion

- Repeated errors tied to one route, tenant, or version

- Long tails: a few requests taking seconds while median is low

 If you adopt distributed tracing later, start by propagating request_id as trace-id headers through your stack.

## Dashboards That Work

Dashboards should answer the top 5 on-call questions in 1 minute.

Recommended panels:

- Traffic: requests per second, by route and method

- Latency: p50, p95, p99 trends; top N slow routes now

- Errors: error rate by status_code and route

- Saturation: in-flight requests, event loop lag, CPU, heap used, GC pauses

- Dependencies: DB latency and errors, outbound HTTP

- Version breakdown: metrics split by version/commit for quick rollback calls

Layout tips:

- Keep a single-page triage view with linked drill-down dashboards per service

- Annotate with deploy markers

- Use consistent colors and units; milliseconds for latency is readable at a glance

## Incident Response Workflow

A lightweight, repeatable flow turns alerts into fast recovery.

- Acknowledge and assess

- Confirm the alert and check the triage dashboard

- Identify user impact (error rate, latency, drop in traffic)

- Stabilize

- Roll back recent changes if metrics point to a regression

- Scale out, add a circuit breaker, or rate-limit noisy tenants

- Diagnose

- Filter logs by request_id , route, and version

- Compare hot routes and dependency panels

- Mitigate

- Ship a config fix, feature flag, or hot patch

- Add temporary timeouts or retries for flaky dependencies

- Verify

- Ensure alerts clear and golden signals return to baseline

- Learn

- Capture what detected the issue, what helped, and what was missing

- Tune alerts and dashboards. Add labels or new metrics if needed

Keep a short runbook per alert with owner, checks, and remedies.

## Local Pilot Plan

Start small, measure, and inspect locally before rollout. A narrow, measurable pilot reduces risk and clarifies next steps.

### 1) Run the API locally

Install dependencies:

npm i express prom-client pino pino-http
node server.js 
 Hit endpoints and check metrics:

curl -s localhost:3000/healthz
curl -s localhost:3000/users/123
curl -s localhost:3000/error
curl -s localhost:3000/metrics | head 

### 2) Optional: scrape metrics and alert locally

 Use a minimal docker-compose to run a metrics store, a dashboard, and an alerting component.

version: "3.9"
services:
 api:
 build: .
 ports: ["3000:3000\
 prometheus:
 image: prom/prometheus:latest
 ports: ["9090:9090\
 volumes:
 - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
 alertmanager:
 image: prom/alertmanager:latest
 ports: ["9093:9093\
 volumes:
 - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
 grafana:
 image: grafana/grafana:latest
 ports: ["3001:3000\) 
 Example prometheus.yml :

global:
 scrape_interval: 15s
scrape_configs:
 - job_name: "express-api"
 metrics_path: /metrics
 static_configs:
 - targets: ["api:3000\)
rule_files:
 - rules.yml 
 Example rules.yml with two alerts:

groups:
 - name: express
 rules:
 - alert: HighErrorRate
 expr: (sum(rate(http_requests_errors_total[5m])) / sum(rate(http_requests_total[5m]))) > 0.02
 for: 5m
 labels: { severity: page }
 annotations:
 summary: "Elevated 5xx error rate"
 - alert: HighLatencyP95
 expr: histogram_quantile(0.95, sum(rate(http_request_duration_ms_bucket[5m])) by (le)) > 0.3
 for: 10m
 labels: { severity: page }
 annotations:
 summary: "p95 latency > 300ms" 

### 3) Generate test traffic

 # steady good traffic
for i in $(seq 1 200); do curl -s localhost:3000/users/$i > /dev/null; done

# induce latency and errors
curl -s localhost:3000/error > /dev/null 
 Watch the dashboard and validate that alerts trigger only when conditions persist.

### 4) Define pass-fail and document

- Pass if metrics populate, dashboards render, and alerts fire and clear as expected

- Fail if labels explode in cardinality, alerts are noisy, or metrics are missing

- Capture thresholds, label sets, and any gaps to fix before staging rollout

## Conclusion

You now have a practical baseline for Express API monitoring:

- Golden-signal metrics and low-cardinality labels

- Actionable alert rules that reflect user impact

- Logs that correlate requests and speed diagnosis

- Dashboards that answer on-call questions fast

- A simple incident response flow and a safe local pilot

Next steps:

- Tune thresholds to your real traffic and capacity

- Add dependency metrics for databases and outbound calls

- Expand logs with request_id propagation across services

- Pilot on a single service in staging, then roll out gradually

A small, measurable start and a clear workflow help teams move from ideas to a reliable, observable Express API.