Express API production operations checklist with practical examples: practical implementation guide

Running an Express API in production is more than starting a server. You need secure defaults, reliable deploys, deep visibility, tested backups, and a plan for upgrades and incidents. This practical checklist gives you copy-paste examples and concrete exit criteria so your team can operate with confidence.

Workflow Overview

Use an explicit, staged flow so each step has clear owners and exit criteria:

1. Plan

• Define SLOs: availability (for example, 99.9%), p95 latency, and error budgets.
• Choose runtime: Node LTS, Express version, and process manager.
• Decide observability stack: logs, metrics, traces, and alert targets.

1. Harden

• Lock environment config and secrets.
• Add security middleware, timeouts, and graceful shutdown.

1. Observe

• Emit JSON logs with request IDs.
• Expose /healthz, /readyz, and /metrics.

1. Verify

• Load test to SLOs.
• Chaos basics: terminate a process; verify no request loss.
• Run backup and restore drill.

1. Release

• Zero-downtime rollout (drain, swap, or rolling).
• Monitor for 30 minutes; watch for regressions.

1. Operate

• On-call runbooks for high latency, error spikes, disk pressure, and dependency failures.

A concise, reviewed checklist reduces rework and increases consistency across releases.

Configuration and hardening

Apply these defaults before your first production request.

• Node and Express
• Use Node LTS and pin with engines in package.json.
• Set NODE_ENV=production.
• Avoid synchronous CPU-heavy work in request paths.

• Security middleware and limits
• helmet() for common headers.
• compression() only when upstream does not already compress.
• express.json/urlencoded with strict size limits.
• express-rate-limit or gateway-level rate limiting.
• Disable x-powered-by and validate CORS with an allowlist.

• Trust proxy
• If behind a reverse proxy or load balancer, set app.set('trust proxy', true) or a subnet so req.ip and protocol are correct.

• Timeouts and headers
• Keep-alive and header timeouts prevent hung sockets and slowloris.
• Always set a server-side request timeout.

• Errors and 404s
• Central error handler with consistent JSON shape and no stack traces to clients.

• Graceful shutdown
• Handle SIGTERM and SIGINT to drain, then close the server. Refuse traffic by flipping readiness before shutdown.

Practical starter server (adapt as needed):

// server.js
const express = require('express');
const helmet = require('helmet');
const compression = require('compression');
const rateLimit = require('express-rate-limit');
const pino = require('pino');
const pinoHttp = require('pino-http');
const crypto = require('crypto');

const app = express();
app.set('env', process.env.NODE_ENV || 'production');
app.disable('x-powered-by');
app.set('trust proxy', true); // if behind a proxy

app.use(helmet());
app.use(compression());
app.use(express.json({ limit: '1mb', strict: true }));
app.use(express.urlencoded({ extended: false, limit: '1mb' }));

// Request ID
app.use((req, res, next) => {
  const rid = req.headers['x-request-id'] || crypto.randomUUID();
  res.setHeader('x-request-id', rid);
  req.id = rid;
  next();
});

const logger = pino({ level: process.env.LOG_LEVEL || 'info' });
app.use(pinoHttp({ logger, genReqId: req => req.id }));

// Rate limit (tune per endpoint)
app.use(rateLimit({ windowMs: 60_000, max: 300, standardHeaders: true, legacyHeaders: false }));

// Health and readiness
let isReady = false;
app.get('/healthz', (req, res) => res.status(200).json({ ok: true }));
app.get('/readyz', (req, res) => isReady ? res.json({ ready: true }) : res.status(503).json({ ready: false }));

// Example route
app.get('/v1/ping', (req, res) => res.json({ pong: true }));

// 404 and error handler
app.use((req, res) => res.status(404).json({ error: 'Not Found' }));
app.use((err, req, res, next) => { // eslint-disable-line no-unused-vars
  req.log.error({ err }, 'Unhandled error');
  res.status(500).json({ error: 'Internal Server Error' });
});

const server = app.listen(process.env.PORT || 3000, () => {
  logger.info({ port: server.address().port }, 'server started');
  isReady = true;
});

// Timeouts and graceful shutdown
server.keepAliveTimeout = 61_000; // > LB keepalive
server.headersTimeout = 65_000;   // > keepAliveTimeout

const shutdown = () => {
  logger.info('shutdown start');
  isReady = false;
  server.close(err => {
    if (err) logger.error({ err }, 'server close error');
    logger.info('shutdown complete');
    process.exit(err ? 1 : 0);
  });
  setTimeout(() => {
    logger.error('forced shutdown');
    process.exit(1);
  }, 30_000).unref();
};

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);
process.on('unhandledRejection', err => { logger.error({ err }, 'unhandledRejection'); shutdown(); });
process.on('uncaughtException', err => { logger.error({ err }, 'uncaughtException'); shutdown(); });

Observability and monitoring

Make every request observable and alert on user-facing symptoms.

• Logging
• JSON logs, one line per event, with fields: ts, level, msg, request_id, method, route, status, latency_ms, user_id (if available).
• Do not log secrets or PII.

• Metrics
• Expose /metrics in Prometheus format.
• Track request count, error count, and latency histograms by method/route/status.

• Tracing
• Propagate x-request-id (and trace headers if using a tracer). Sample at a low percentage for high-QPS services.

• Alerts (align to SLOs)
• p95 latency above threshold for 5m.
• 5xx rate above threshold for 5m.
• Readiness failing across more than one instance.
• Disk or memory pressure near limits.

Example metrics using prom-client:

const client = require('prom-client');
client.collectDefaultMetrics();
const httpLatency = new client.Histogram({
  name: 'http_request_duration_ms',
  help: 'HTTP request latency in ms',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [5, 15, 50, 100, 300, 500, 1000, 2000]
});

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    const route = req.route && req.route.path ? req.route.path : req.path;
    httpLatency.labels(req.method, route, String(res.statusCode)).observe(Date.now() - start);
  });
  next();
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.end(await client.register.metrics());
});

Runtime operations

Operate for zero-downtime and predictable behavior.

• Process manager
• Use a supervisor (for example, a service manager or PM2) to restart on crash and manage instances.
• Prefer one process per CPU with graceful shutdown. Avoid in-process clustering if your orchestrator handles replicas.

• Reverse proxy
• Terminate TLS, set proxy headers (X-Forwarded-*), and keep keep-alive enabled.

Example NGINX snippet:

upstream api {
  server 127.0.0.1:3000;
  keepalive 64;
}
server {
  listen 443 ssl http2;
  server_name api.example.com;
  location / {
    proxy_pass http://api;
    proxy_http_version 1.1;
    proxy_set_header Connection '';
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Request-Id $request_id;
  }
}

• Zero-downtime release
• Step 1: mark instance not ready (readyz -> 503),
• Step 2: drain connections via load balancer,
• Step 3: stop process after server.close completes,
• Step 4: start new instance, wait until readyz=200, then add to pool.

• Resource limits
• Set ulimit -n (file descriptors) high enough for concurrent sockets.
• Tune Node heap via --max-old-space-size if needed; monitor GC.

• Scaling
• Keep the API stateless; use external stores for sessions or caching.
• For sticky sessions (if unavoidable), ensure LB stickiness and rate limit per client.

Data management

Your API is only as reliable as your last verified restore.

• MongoDB connections
• Use the SRV URI with TLS, tune maxPoolSize to match concurrency.
• Timeouts: serverSelectionTimeoutMS and socketTimeoutMS.

• Backups
• Schedule dumps and keep them off-host with retention.
• Test restore regularly into a staging database.

Examples:

# Backup
mongodump --uri "$MONGODB_URI" \
  --gzip --archive=/backups/$(date +%F).gz

# Restore (into a test DB)
mongorestore --uri "$MONGODB_RESTORE_URI" \
  --gzip --archive=/backups/2024-01-15.gz --drop

• Migrations
• Versioned, idempotent migration scripts.
• Run migrations before enabling new code paths.
• Add TTL indexes for expiring data automatically.

Maintenance and upgrades

Keep velocity high without breaking production.

• Dependencies
• Lock with a package-lock and update on a schedule.
• Review security advisories and update high-risk packages quickly.

• Node and Express
• Track Node LTS and plan minor upgrades quarterly.
• Use canary releases for major bumps and monitor p95/5xx.

• Operational hygiene
• Rotate secrets and tokens.
• Prune logs; ensure disk is monitored and rotated.
• Run incident postmortems and add checklist items to prevent repeats.

Local Pilot Plan

Start small, measure, and iterate. Keep the first pilot narrow and easy to inspect locally before deployment.

Pilot scope:

• One endpoint (/v1/ping),
• /healthz and /readyz,
• JSON logging with request IDs,
• Prometheus metrics with latency histogram,
• Graceful shutdown and timeouts.

Steps:

1. Scaffold a minimal Express server using the example in this guide.
2. Add request ID and pino logging. Verify logs show request_id and latency.
3. Add prom-client metrics and expose /metrics. Confirm counters and histograms change under load.
4. Implement readiness gating (flip isReady on startup and shutdown).
5. Load test with a small tool (for example, autocannon) to your latency SLO:

autocannon -c 50 -d 30 http://localhost:3000/v1/ping

1. Send SIGTERM during load; verify no 5xx spikes and no dropped requests.
2. If using MongoDB, connect with a small query, set TTL index in a test collection, and run one backup and restore cycle.

Exit criteria:

• p95 latency within target under expected load,
• No errors during controlled shutdown,
• Metrics and logs are complete and useful for debugging.

Common production pitfalls

Avoid these frequent sources of incidents:

• No graceful shutdown: requests drop on deploy.
• Missing timeouts: connections hang and workers exhaust.
• trust proxy not set: client IPs wrong; rate limiting ineffective.
• Unlimited payloads: large bodies crash the process.
• Event loop blocking: synchronous crypto, zlib, or JSON on big payloads.
• Memory leaks: retaining req/res or large caches without bounds.
• Log sprawl: unbounded logs fill disk; rotate and cap size.
• Unbounded retries: thundering herds; add jittered backoff and circuit breakers.
• Mongo pool too large: timeouts and contention under load.
• Secrets in logs: scrub sensitive fields.
• No 404 handler: framework defaults leak internals.
• No env validation: app boots with bad settings and fails under traffic.

Conclusion

A production-ready Express API is deliberate: secure defaults, strong observability, safe deploys, tested backups, and a repeatable workflow. Start with the local pilot, graduate to a zero-downtime rollout, and keep iterating.

Preflight checklist:

• [ ] NODE_ENV=production and Node LTS pinned
• [ ] helmet, compression (as needed), size limits, rate limits
• [ ] trust proxy configured correctly
• [ ] Keep-alive and header timeouts set; request timeout enforced
• [ ] /healthz, /readyz, /metrics exposed and monitored
• [ ] JSON logs with request IDs; PII scrubbed
• [ ] Graceful shutdown on SIGTERM/SIGINT
• [ ] Backups scheduled and restore tested
• [ ] Migrations versioned and idempotent
• [ ] Alerts on p95 latency, error rate, readiness, and resource limits
• [ ] Runbook for high-latency and error spikes

Adopt the workflow, keep the pilot narrow, and expand with confidence as you prove reliability at each step.