Production operations for REST APIs succeed when they are boring: predictable rollouts, observable behavior, quick diagnosis, and fast recovery. This guide gives you a practical, step-by-step checklist you can run in real environments. It includes concrete configuration patterns, validation commands, expected results, failure-mode playbooks, and a repeatable daily-to-monthly operations routine. All values, thresholds, and command outputs in this article are constructed examples. Tune them to your workloads and risk tolerance. ## Version and Environment Inventory You cannot operate what you cannot name. Before any change, collect the exact versions, topology, and critical dependencies for the API. Store this inventory in a versioned location accessible to operations and developers.
ItemExample Value (Constructed)How to Verify
RuntimeNode.js v18.19.1node -v
API Buildv1.7.3 commit 3f2a9c1grep VERSION .env && git rev-parse --short HEAD
OSUbuntu 22.04 LTSlsb_release -a or cat /etc/os-release
TLSLet's Encrypt, expires 2026-03-01openssl s_client -connect api.example.com:443 -servername api.example.com < /dev/null 2>/dev/null | openssl x509 -noout -dates
DB EngineMongoDB 6.0 primary in region Amongod --version && mongo --eval 'db.hello()'
Dependenciesexpress 4.18, mongoose 7.xnpm ls --prod --depth=0
Third-PartyStripe, OpenAI APIReview integration keys and endpoints
Health Endpoints/healthz, /readyz, /versioncurl -fsS https://api.example.com/healthz
Time SyncNTP activetimedatectl status
Minimum inventory prerequisites: - A written topology: inbound (load balancer, WAF), API hosts, database cluster, caches, outbound egress. - A documented release identifier attached to each instance (environment variable, file, or header). - Credentials rotation policy and current rotation dates. ## Safe Configuration Path Adopt an incremental, low-risk sequence. Each step is independently verifiable and reversible. ### 1. Add Versioned Health and Readiness Endpoints Goal: unambiguous, fast checks that never hit expensive dependencies unless necessary. Constructed example (Express): // server-health.js const express = require('express'); const router = express.Router(); // Liveness: process is up and event loop responsive router.get('/healthz', (req, res) => { res.set('Cache-Control', 'no-store'); res.json({ status: 'ok', uptime_s: process.uptime() }); }); // Readiness: dependencies needed for serving traffic router.get('/readyz', async (req, res) => { try { // Example: ping DB with a lightweight command await req.app.locals.db.command({ ping: 1 }); res.json({ ready: true }); } catch (err) { res.status(503).json({ ready: false, reason: 'db_unavailable' }); } }); // Version: immutable release info router.get('/version', (req, res) => { res.json({ version: process.env.VERSION || 'dev', commit: process.env.GIT_COMMIT || 'unknown' }); }); module.exports = router; Expected results: - /healthz returns 200 within 10 ms on host. - /readyz returns 200 only when database and required dependencies are reachable. - /version returns immutable identifiers for traceability. Verification: curl -fsS https://api.example.com/healthz curl -fsS https://api.example.com/readyz curl -fsS https://api.example.com/version ### 2. Set Safe HTTP Server Timeouts Prevent hung connections from exhausting resources. Constructed example (Node.js HTTP): // server-timeouts.js const http = require('http'); const app = require('./app'); const server = http.createServer(app); server.headersTimeout = 65000; // time to receive full headers server.requestTimeout = 30000; // overall per-request timeout server.keepAliveTimeout = 60000; // keep-alive connection reuse time server.maxRequestsPerSocket = 100; // mitigate slowloris-style risks server.listen(process.env.PORT || 8080); Expected: requests exceeding 30 seconds are terminated with 408 or 504 (depending on proxy) and logged. ### 3. Enforce Rate Limits and Size Limits - Apply an overall request rate limit per client IP or token. - Enforce maximum payload size for JSON bodies (for example 1 MB). Constructed example (Express): const express = require('express'); const rateLimit = require('express-rate-limit'); const app = express(); app.use(express.json({ limit: '1mb' })); const limiter = rateLimit({ windowMs: 60 * 1000, max: 300, standardHeaders: true, legacyHeaders: false }); app.use(limiter); Verification: # Send more than max requests per minute from same IP and expect 429 for i in $(seq 1 400); do curl -s -o /dev/null -w "%{http_code}\n" https://api.example.com/healthz; done | sort | uniq -c ### 4. Add Secure Headers and Strict TLS - Force TLS 1.2+. - Add HSTS, X-Content-Type-Options, Referrer-Policy, and a minimal Content-Security-Policy for docs or HTML responses. Constructed example (Express headers): const helmet = require('helmet'); app.use(helmet({ hsts: { maxAge: 31536000, includeSubDomains: true }, contentSecurityPolicy: false // for pure JSON APIs; enable for HTML })); TLS verification: openssl s_client -connect api.example.com:443 -servername api.example.com < /dev/null 2>/dev/null | openssl x509 -noout -text | grep -E "TLS|Signature Algorithm|DNS:" ### 5. Structured, Context-Rich Logging - Use JSON logs with request ID, user ID or API key hash, route, status, latency, and error codes. - Emit one log line per request and separate error logs with stack traces. Constructed example (Express + pino): const pino = require('pino'); const pinoHttp = require('pino-http'); const logger = pino({ level: process.env.LOG_LEVEL || 'info' }); app.use(pinoHttp({ logger, genReqId: req => req.headers['x-request-id'] || crypto.randomUUID() })); Expected logs (constructed): {"ts":"2026-08-01T12:00:00Z","level":"info","req_id":"1d2e","route":"/v1/payments","status":201,"latency_ms":83} ### 6. Connection Pools, Timeouts, and Retries - Set database pool sizes to match CPU concurrency and avoid exhaustion. - Use short timeouts and bounded retries with jitter for external calls (for example Stripe or OpenAI API) to avoid cascades. Constructed example (MongoDB): const { MongoClient } = require('mongodb'); const client = new MongoClient(process.env.MONGO_URI, { maxPoolSize: 20, minPoolSize: 5, serverSelectionTimeoutMS: 3000, socketTimeoutMS: 5000 }); Constructed example (fetch with retries and idempotency): async function postWithIdempotency(url, body, key) { const headers = { 'Content-Type': 'application/json', 'Idempotency-Key': key }; for (let i = 0; i < 3; i++) { try { const res = await fetch(url, { method: 'POST', headers, body: JSON.stringify(body), timeout: 5000 }); if (res.ok) return res.json(); if (res.status >= 500) continue; // retry on server errors throw new Error(`non-retryable status ${res.status}`); } catch (e) { await new Promise(r => setTimeout(r, 100 * Math.pow(2, i))); } } throw new Error('exhausted retries'); } ### 7. Backups and Restore Tests - Automate logical backups (and snapshots if available). - Regularly restore to a disposable database and validate referential integrity. Constructed example (MongoDB logical backup and restore): # Backup mongodump --uri "$MONGO_URI" --archive=/var/backups/api-$(date +%F).gz --gzip # Restore to temp DB for validation mongorestore --nsInclude mydb.* --archive=/var/backups/api-2026-08-01.gz --gzip --nsFrom 'mydb.*' --nsTo 'mydb_restore.*' Expected results: - Backup completes with non-empty archive and logs checksum. - Restore succeeds; basic counts match production within tolerance. ### 8. Secrets Hygiene - Load secrets from environment or OS secrets store; never from source control. - Rotate keys on a fixed cadence; validate that rotated keys are active and old keys are revoked. Verification: # Process sees expected secrets cat /proc/$(pgrep -f "node .*app.js" | head -1)/environ | tr '\0' '\n' | grep -E 'API_KEY|DB_PASSWORD' ## Verification and Diagnostics Verification makes configuration real. Run these checks after each change and on a schedule. ### 1. Smoke Tests and Golden Paths - Health endpoints return 200. - A representative GET and POST per major route complete with p50 latency within target. Constructed example smoke test: set -euo pipefail HOST=https://api.example.com curl -fsS $HOST/healthz curl -fsS $HOST/readyz # Golden GET time curl -fsS "$HOST/v1/customers?limit=5" # Golden POST with idempotency IDEMP=$(uuidgen) HTTP=$(curl -s -o /dev/null -w "%{http_code}" -H "Idempotency-Key: $IDEMP" -H "Content-Type: application/json" -d '{"amount":1000,"currency":"usd"}' "$HOST/v1/payments") [ "$HTTP" = "201" ] ### 2. Observability Checks - Logs show one line per request with request ID. - Metrics report traffic rate, success ratio, and latency breakdowns. - Trace sampling (if used) captures slow endpoints. Constructed example core SLI targets:
MetricTarget (Constructed)Alert When
Availability (2xx, 3xx)>= 99.9% rolling 30 days< 99.5% over 1 hour
p95 latency (GET)<= 200 ms> 400 ms for 15 min
p95 latency (POST)<= 500 ms> 800 ms for 15 min
Error rate (5xx)< 0.5%> 2% for 10 min
DB connection utilization< 80%> 90% for 5 min
### 3. Configuration Validation - Request and headers timeouts take effect (verify with a deliberately slow upstream). - Rate limiting responds with 429 when exceeded. - Payload limits return 413 for oversize bodies. Constructed example tests: # Oversized payload python - <<'PY' import requests print(requests.post('https://api.example.com/v1/echo', data='x'*(2*1024*1024)).status_code) PY # Slow upstream test: expect timeout per server.requestTimeout ### 4. Backup-and-Restore Acceptance - Restore latest backup to a temp DB. - Run an integrity query set: counts by collection, sample document roundtrips, index presence. Constructed example: mongo --quiet <<'JS' use mydb_restore printjson(db.runCommand({ dbStats: 1 })) printjson(db.customers.countDocuments()) printjson(db.orders.countDocuments()) JS Expected: counts are within an expected delta; indexes exist; no errors. ## Failure Modes and Recovery Prepare for the failures you are most likely to see first.
SymptomLikely CauseFirst FixesVerify Recovery
Spike of 5xx on POSTDependency outage (payments, ML API)Enable circuit breaker or queue; reduce timeout to 3 s with 2 retries; serve 202 Accepted with queued job IDError rate back under 0.5%; queue depth stabilizes
Many 429 responsesAggressive rate limitRaise per-key bucket for trusted clients; add burst capacity; communicate limitsp95 latency steady; 2xx ratio increases without saturation
Slow p95 latencyDB contention or missing indexAdd index, bump DB pool to match CPU, review N+1 queriesp95 returns to target; DB lock time drops
Memory growth over hoursLeak or unbounded cachesCap cache size, audit request bodies, restart with heap dumpHeap usage plateaus; GC pauses normal
TLS errors in clientsExpired or weak ciphersRenew cert; enforce TLS 1.2+; fix SNI mismatchSuccessful handshakes; no new TLS alerts
Thread/socket exhaustionSlowloris or timeouts too longReduce headersTimeout; set maxRequestsPerSocket; enable request size limitsNew connections succeed; 5xx due to timeout fall
### Recovery Steps and Rollback - Roll forward fix, then validate with smoke tests and SLIs. - Roll back release if fixes fail to stabilize within a defined window (for example 15 minutes) and error budget is at risk. Constructed example rollback checklist: - Identify last known good build ID (for example v1.7.2, commit a1b2c3). - Stop the API process safely (drain connections if supported). - Replace the application binary or bundle with the last known good one. - Reapply configuration diffs as needed (config kept separate from code). - Start the service and run smoke tests. - Announce rollback completion and open a follow-up task for root cause analysis. Validation after recovery: - /readyz returns 200. - SLIs return to target ranges for at least 15 minutes. - Error logs show only normal baseline rates. ## Operations Checklist Run these routines to keep the API healthy. ### Pre-Deploy (Per Release) - [ ] Confirm inventory is up to date (runtime, build ID, dependencies, DB engine). - [ ] Check /version , /healthz , /readyz in the target environment. - [ ] Review config diffs: timeouts, rate limits, secrets, feature flags. - [ ] Backup verification: latest backup completed and restore tested in the last week. - [ ] Dry-run the golden-path smoke tests against staging. ### During Deploy - [ ] Deploy during a low-traffic window when possible. - [ ] Watch live metrics: availability, error rate, p95 latency, DB pool utilization. - [ ] Run smoke tests and confirm expected results. - [ ] Hold for 10-15 minutes and confirm stability. ### Post-Deploy (Same Day) - [ ] Update inventory with the release ID and timestamp. - [ ] Review logs for new error classes or warnings. - [ ] Communicate changes to stakeholders (noting any new limits or headers). ### Daily - [ ] Check SLI dashboard for availability, error rate, and latency. - [ ] Skim error logs and top 5 slow endpoints. - [ ] Verify that the last backup completed and space is healthy. - [ ] Rotate and compress logs if near limits. ### Weekly - [ ] Restore a backup to a temp DB and run integrity checks. - [ ] Exercise dependency timeouts and circuit breakers with a short test. - [ ] Review rate limit counters and adjust thresholds for fairness and safety. - [ ] Refresh expiring credentials due in the next 14 days. ### Monthly - [ ] Run a controlled failover drill for one dependency. - [ ] Review SLI targets and adjust alerts to reduce noise while protecting user experience. - [ ] Reassess pool sizes and concurrency based on traffic trends. - [ ] Audit dependencies for security updates and deprecations. ## Practical Examples Across Common Stacks - Express API on Node.js: Add helmet for headers, pino for JSON logs, express-rate-limit for fairness, and health endpoints as shown. - MongoDB-backed APIs: Start with maxPoolSize near 10 per CPU core and refine using connection utilization and latency. Keep serverSelectionTimeoutMS short (1-3 s) to avoid request stalls. - Payment calls (constructed example like Stripe): Use an Idempotency-Key header on POSTs that create charges. On upstream timeouts, retry at most twice with exponential backoff and jitter. Fallback to 202 Accepted with a status endpoint for finalization when possible. - ML or content calls (constructed example like OpenAI API): Bound prompt or payload sizes server-side, set 5 s connect timeout and 30 s read timeout, and use a small circuit breaker window to avoid cascading failures. ## Verification Script Bundle (Constructed Example) Bundle these checks into one script you can run after each change. #!/usr/bin/env bash set -euo pipefail HOST=${HOST:-https://api.example.com} say() { printf "[%s] %s\n" "$(date -Is)" "$*"; } say "Health checks" curl -fsS $HOST/healthz >/dev/null curl -fsS $HOST/readyz >/dev/null say "Version" curl -fsS $HOST/version | jq -r '.version,.commit' say "Golden GET" time curl -fsS "$HOST/v1/customers?limit=1" >/dev/null say "Golden POST idempotent" IDEMP=$(uuidgen) code=$(curl -s -o /dev/null -w "%{http_code}" -H "Idempotency-Key: $IDEMP" -H "Content-Type: application/json" -d '{"amount":500,"currency":"usd"}' "$HOST/v1/payments") [ "$code" = "201" ] || { echo "Expected 201, got $code"; exit 1; } say "Rate limit" rc=$(for i in $(seq 1 400); do curl -s -o /dev/null -w "%{http_code}\n" $HOST/healthz; done | sort | uniq -c) echo "$rc" ## Conclusion Start small, measure, and grow. Implement the health endpoints, timeouts, structured logging, rate and size limits, and connection tuning. Prove them with the smoke tests and SLI checks. Run the daily, weekly, and monthly routines so reliability becomes predictable. Then expand the checklist to your specific routes, data stores, and third-party dependencies. A narrow, verifiable pilot is the best first step: pick one endpoint, run the safe configuration path, verify with the provided scripts, and document the observed improvements. Repeat for the next most critical path until your entire REST API reaches the same operational standard. When every change follows a rehearsed sequence and every incident has a documented playbook, operations stop being a source of surprise and start being a source of confidence.