REST API Production Operations Checklist with Practical Examples

Production operations for REST APIs succeed when they are boring: predictable rollouts, observable behavior, quick diagnosis, and fast recovery. This guide gives you a practical, step-by-step checklist you can run in real environments. It includes concrete configuration patterns, validation commands, expected results, failure-mode playbooks, and a repeatable daily-to-monthly operations routine.

All values, thresholds, and command outputs in this article are constructed examples. Tune them to your workloads and risk tolerance.

Version and Environment Inventory

You cannot operate what you cannot name. Before any change, collect the exact versions, topology, and critical dependencies for the API. Store this inventory in a versioned location accessible to operations and developers.

Minimum inventory prerequisites:

• A written topology: inbound (load balancer, WAF), API hosts, database cluster, caches, outbound egress.
• A documented release identifier attached to each instance (environment variable, file, or header).
• Credentials rotation policy and current rotation dates.

Safe Configuration Path

Adopt an incremental, low-risk sequence. Each step is independently verifiable and reversible.

1. Add Versioned Health and Readiness Endpoints

Goal: unambiguous, fast checks that never hit expensive dependencies unless necessary.

Constructed example (Express):

// server-health.js
const express = require('express');
const router = express.Router();

// Liveness: process is up and event loop responsive
router.get('/healthz', (req, res) => {
  res.set('Cache-Control', 'no-store');
  res.json({ status: 'ok', uptime_s: process.uptime() });
});

// Readiness: dependencies needed for serving traffic
router.get('/readyz', async (req, res) => {
  try {
    // Example: ping DB with a lightweight command
    await req.app.locals.db.command({ ping: 1 });
    res.json({ ready: true });
  } catch (err) {
    res.status(503).json({ ready: false, reason: 'db_unavailable' });
  }
});

// Version: immutable release info
router.get('/version', (req, res) => {
  res.json({ version: process.env.VERSION || 'dev', commit: process.env.GIT_COMMIT || 'unknown' });
});

module.exports = router;

Expected results:

• /healthz returns 200 within 10 ms on host.
• /readyz returns 200 only when database and required dependencies are reachable.
• /version returns immutable identifiers for traceability.

Verification:

curl -fsS https://api.example.com/healthz
curl -fsS https://api.example.com/readyz
curl -fsS https://api.example.com/version

2. Set Safe HTTP Server Timeouts

Prevent hung connections from exhausting resources.

Constructed example (Node.js HTTP):

// server-timeouts.js
const http = require('http');
const app = require('./app');
const server = http.createServer(app);

server.headersTimeout = 65000;      // time to receive full headers
server.requestTimeout = 30000;      // overall per-request timeout
server.keepAliveTimeout = 60000;    // keep-alive connection reuse time
server.maxRequestsPerSocket = 100;  // mitigate slowloris-style risks

server.listen(process.env.PORT || 8080);

Expected: requests exceeding 30 seconds are terminated with 408 or 504 (depending on proxy) and logged.

3. Enforce Rate Limits and Size Limits

• Apply an overall request rate limit per client IP or token.
• Enforce maximum payload size for JSON bodies (for example 1 MB).

Constructed example (Express):

const express = require('express');
const rateLimit = require('express-rate-limit');
const app = express();

app.use(express.json({ limit: '1mb' }));

const limiter = rateLimit({
  windowMs: 60 * 1000,
  max: 300,
  standardHeaders: true,
  legacyHeaders: false
});
app.use(limiter);

Verification:

# Send more than max requests per minute from same IP and expect 429
for i in $(seq 1 400); do curl -s -o /dev/null -w "%{http_code}\n" https://api.example.com/healthz; done | sort | uniq -c

4. Add Secure Headers and Strict TLS

• Force TLS 1.2+.
• Add HSTS, X-Content-Type-Options, Referrer-Policy, and a minimal Content-Security-Policy for docs or HTML responses.

Constructed example (Express headers):

const helmet = require('helmet');
app.use(helmet({
  hsts: { maxAge: 31536000, includeSubDomains: true },
  contentSecurityPolicy: false // for pure JSON APIs; enable for HTML
}));

TLS verification:

openssl s_client -connect api.example.com:443 -servername api.example.com < /dev/null 2>/dev/null | openssl x509 -noout -text | grep -E "TLS|Signature Algorithm|DNS:"

5. Structured, Context-Rich Logging

• Use JSON logs with request ID, user ID or API key hash, route, status, latency, and error codes.
• Emit one log line per request and separate error logs with stack traces.

Constructed example (Express + pino):

const pino = require('pino');
const pinoHttp = require('pino-http');
const logger = pino({ level: process.env.LOG_LEVEL || 'info' });

app.use(pinoHttp({ logger, genReqId: req => req.headers['x-request-id'] || crypto.randomUUID() }));

Expected logs (constructed):

{"ts":"2026-08-01T12:00:00Z","level":"info","req_id":"1d2e","route":"/v1/payments","status":201,"latency_ms":83}

6. Connection Pools, Timeouts, and Retries

• Set database pool sizes to match CPU concurrency and avoid exhaustion.
• Use short timeouts and bounded retries with jitter for external calls (for example Stripe or OpenAI API) to avoid cascades.

Constructed example (MongoDB):

const { MongoClient } = require('mongodb');
const client = new MongoClient(process.env.MONGO_URI, {
  maxPoolSize: 20,
  minPoolSize: 5,
  serverSelectionTimeoutMS: 3000,
  socketTimeoutMS: 5000
});

Constructed example (fetch with retries and idempotency):

async function postWithIdempotency(url, body, key) {
  const headers = { 'Content-Type': 'application/json', 'Idempotency-Key': key };
  for (let i = 0; i < 3; i++) {
    try {
      const res = await fetch(url, { method: 'POST', headers, body: JSON.stringify(body), timeout: 5000 });
      if (res.ok) return res.json();
      if (res.status >= 500) continue; // retry on server errors
      throw new Error(`non-retryable status ${res.status}`);
    } catch (e) {
      await new Promise(r => setTimeout(r, 100 * Math.pow(2, i)));
    }
  }
  throw new Error('exhausted retries');
}

7. Backups and Restore Tests

• Automate logical backups (and snapshots if available).
• Regularly restore to a disposable database and validate referential integrity.

Constructed example (MongoDB logical backup and restore):

# Backup
mongodump --uri "$MONGO_URI" --archive=/var/backups/api-$(date +%F).gz --gzip

# Restore to temp DB for validation
mongorestore --nsInclude mydb.* --archive=/var/backups/api-2026-08-01.gz --gzip --nsFrom 'mydb.*' --nsTo 'mydb_restore.*'

Expected results:

• Backup completes with non-empty archive and logs checksum.
• Restore succeeds; basic counts match production within tolerance.

8. Secrets Hygiene

• Load secrets from environment or OS secrets store; never from source control.
• Rotate keys on a fixed cadence; validate that rotated keys are active and old keys are revoked.

Verification:

# Process sees expected secrets
cat /proc/$(pgrep -f "node .*app.js" | head -1)/environ | tr '\0' '\n' | grep -E 'API_KEY|DB_PASSWORD'

Verification and Diagnostics

Verification makes configuration real. Run these checks after each change and on a schedule.

1. Smoke Tests and Golden Paths

• Health endpoints return 200.
• A representative GET and POST per major route complete with p50 latency within target.

Constructed example smoke test:

set -euo pipefail
HOST=https://api.example.com

curl -fsS $HOST/healthz
curl -fsS $HOST/readyz

# Golden GET
time curl -fsS "$HOST/v1/customers?limit=5"

# Golden POST with idempotency
IDEMP=$(uuidgen)
HTTP=$(curl -s -o /dev/null -w "%{http_code}" -H "Idempotency-Key: $IDEMP" -H "Content-Type: application/json" -d '{"amount":1000,"currency":"usd"}' "$HOST/v1/payments")
[ "$HTTP" = "201" ]

2. Observability Checks

• Logs show one line per request with request ID.
• Metrics report traffic rate, success ratio, and latency breakdowns.
• Trace sampling (if used) captures slow endpoints.

Constructed example core SLI targets:

3. Configuration Validation

• Request and headers timeouts take effect (verify with a deliberately slow upstream).
• Rate limiting responds with 429 when exceeded.
• Payload limits return 413 for oversize bodies.

Constructed example tests:

# Oversized payload
python - <<'PY'
import requests
print(requests.post('https://api.example.com/v1/echo', data='x'*(2*1024*1024)).status_code)
PY

# Slow upstream test: expect timeout per server.requestTimeout

4. Backup-and-Restore Acceptance

• Restore latest backup to a temp DB.
• Run an integrity query set: counts by collection, sample document roundtrips, index presence.

Constructed example:

mongo --quiet <<'JS'
use mydb_restore
printjson(db.runCommand({ dbStats: 1 }))
printjson(db.customers.countDocuments())
printjson(db.orders.countDocuments())
JS

Expected: counts are within an expected delta; indexes exist; no errors.

Failure Modes and Recovery

Prepare for the failures you are most likely to see first.

Recovery Steps and Rollback

• Roll forward fix, then validate with smoke tests and SLIs.
• Roll back release if fixes fail to stabilize within a defined window (for example 15 minutes) and error budget is at risk.

Constructed example rollback checklist:

1. Identify last known good build ID (for example v1.7.2, commit a1b2c3).
2. Stop the API process safely (drain connections if supported).
3. Replace the application binary or bundle with the last known good one.
4. Reapply configuration diffs as needed (config kept separate from code).
5. Start the service and run smoke tests.
6. Announce rollback completion and open a follow-up task for root cause analysis.

Validation after recovery:

• /readyz returns 200.
• SLIs return to target ranges for at least 15 minutes.
• Error logs show only normal baseline rates.

Operations Checklist

Run these routines to keep the API healthy.

Pre-Deploy (Per Release)

• [ ] Confirm inventory is up to date (runtime, build ID, dependencies, DB engine).
• [ ] Check /version, /healthz, /readyz in the target environment.
• [ ] Review config diffs: timeouts, rate limits, secrets, feature flags.
• [ ] Backup verification: latest backup completed and restore tested in the last week.
• [ ] Dry-run the golden-path smoke tests against staging.

During Deploy

• [ ] Deploy during a low-traffic window when possible.
• [ ] Watch live metrics: availability, error rate, p95 latency, DB pool utilization.
• [ ] Run smoke tests and confirm expected results.
• [ ] Hold for 10-15 minutes and confirm stability.

Post-Deploy (Same Day)

• [ ] Update inventory with the release ID and timestamp.
• [ ] Review logs for new error classes or warnings.
• [ ] Communicate changes to stakeholders (noting any new limits or headers).

Daily

• [ ] Check SLI dashboard for availability, error rate, and latency.
• [ ] Skim error logs and top 5 slow endpoints.
• [ ] Verify that the last backup completed and space is healthy.
• [ ] Rotate and compress logs if near limits.

Weekly

• [ ] Restore a backup to a temp DB and run integrity checks.
• [ ] Exercise dependency timeouts and circuit breakers with a short test.
• [ ] Review rate limit counters and adjust thresholds for fairness and safety.
• [ ] Refresh expiring credentials due in the next 14 days.

Monthly

• [ ] Run a controlled failover drill for one dependency.
• [ ] Review SLI targets and adjust alerts to reduce noise while protecting user experience.
• [ ] Reassess pool sizes and concurrency based on traffic trends.
• [ ] Audit dependencies for security updates and deprecations.

Practical Examples Across Common Stacks

• Express API on Node.js: Add helmet for headers, pino for JSON logs, express-rate-limit for fairness, and health endpoints as shown.
• MongoDB-backed APIs: Start with maxPoolSize near 10 per CPU core and refine using connection utilization and latency. Keep serverSelectionTimeoutMS short (1-3 s) to avoid request stalls.
• Payment calls (constructed example like Stripe): Use an Idempotency-Key header on POSTs that create charges. On upstream timeouts, retry at most twice with exponential backoff and jitter. Fallback to 202 Accepted with a status endpoint for finalization when possible.
• ML or content calls (constructed example like OpenAI API): Bound prompt or payload sizes server-side, set 5 s connect timeout and 30 s read timeout, and use a small circuit breaker window to avoid cascading failures.

Verification Script Bundle (Constructed Example)

Bundle these checks into one script you can run after each change.

#!/usr/bin/env bash
set -euo pipefail
HOST=${HOST:-https://api.example.com}

say() { printf "[%s] %s\n" "$(date -Is)" "$*"; }

say "Health checks"
curl -fsS $HOST/healthz >/dev/null
curl -fsS $HOST/readyz >/dev/null

say "Version"
curl -fsS $HOST/version | jq -r '.version,.commit'

say "Golden GET"
time curl -fsS "$HOST/v1/customers?limit=1" >/dev/null

say "Golden POST idempotent"
IDEMP=$(uuidgen)
code=$(curl -s -o /dev/null -w "%{http_code}" -H "Idempotency-Key: $IDEMP" -H "Content-Type: application/json" -d '{"amount":500,"currency":"usd"}' "$HOST/v1/payments")
[ "$code" = "201" ] || { echo "Expected 201, got $code"; exit 1; }

say "Rate limit"
rc=$(for i in $(seq 1 400); do curl -s -o /dev/null -w "%{http_code}\n" $HOST/healthz; done | sort | uniq -c)
echo "$rc"

Conclusion

Start small, measure, and grow. Implement the health endpoints, timeouts, structured logging, rate and size limits, and connection tuning. Prove them with the smoke tests and SLI checks. Run the daily, weekly, and monthly routines so reliability becomes predictable. Then expand the checklist to your specific routes, data stores, and third-party dependencies.

A narrow, verifiable pilot is the best first step: pick one endpoint, run the safe configuration path, verify with the provided scripts, and document the observed improvements. Repeat for the next most critical path until your entire REST API reaches the same operational standard. When every change follows a rehearsed sequence and every incident has a documented playbook, operations stop being a source of surprise and start being a source of confidence.