## Intro Node.js powers a large share of modern web services, but production reliability depends less on framework choice and more on disciplined operations. This article provides an actionable Node.js production operations checklist with practical examples. It is written for developers, DevOps engineers, and technical startup teams who need to move from an observed problem to a verified fix without guesswork. Every item in this checklist follows the same pattern: identify the installed version and topology, run a read-only observation, define the expected result and failure signal, make the smallest justified change, verify the outcome, and keep a tested recovery path. Throughout, we use explicit placeholders and never show real credentials or sensitive production identifiers. The goal is operational safety. Observe before changing, limit the blast radius, verify results, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before touching anything, know exactly what you are running and where. A version inventory prevents the classic mistake of applying documentation or patches meant for a different Node.js release. It also gives you a baseline to compare against after a change or incident. ### Read-Only Observation Commands Run these commands from a production shell or via your configuration management tool. They are read-only and safe. # Node.js version node --version # Expected output: v20.11.0 (or your supported version) # Full platform details node -p "process.platform + ' ' + process.arch" # Expected output: linux x64 # Running process details (adapt to your process manager) ps aux | grep node # Expected output: one or more node processes with their start commands and uptimes # List globally installed npm packages (often reveals untracked tools) npm ls -g --depth=0 # Expected output: a tree of global packages with versions ### Documenting the Topology Record the following in a shared operations runbook: - Node.js version and build flags (if any custom build is used) - Operating system and architecture - Process manager (systemd, PM2, Kubernetes, Docker, etc.) - Number of instances and their distribution across hosts - Reverse proxy or load balancer configuration (nginx, HAProxy, cloud LB) - Upstream dependencies (databases, caches, message queues) and their connection settings - Environment variables that affect runtime behavior (but never store secrets in plain text) A concrete example: for a typical Express API behind nginx on Ubuntu 22.04, the topology might look like: Hosts: 3 x t3.medium EC2 instances OS: Ubuntu 22.04 LTS Node.js: v20.11.0 Process manager: PM2 in cluster mode (2 instances per host) Load balancer: AWS Application Load Balancer -> nginx -> PM2 Database: MongoDB 6.0 (primary + two replicas) Cache: Redis 7.2 Message queue: RabbitMQ 3.12 ### Smallest Justified Change If you find an unsupported version (e.g. Node.js 12 that is end-of-life), do not upgrade in place during peak traffic. Instead: - Pin the current working version in your deployment manifest. - Test the upgrade in a staging environment that mirrors production. - Schedule a maintenance window. - Perform the upgrade one instance at a time behind the load balancer. - Monitor error rates, latency, and memory usage after each instance upgrade. Recovery path: if the new version causes unexpected behavior, roll back the instance to the previous image or package version and document the failure. ## Safe Configuration Path Configuration changes are a leading cause of production incidents. The safe path follows a strict sequence: observe current configuration, make a backup, apply the change to one instance, verify, then roll out gradually. ### Observing Current Configuration For Node.js apps, configuration often lives in environment variables, .env files, JSON/YAML config files, or command-line arguments. Never assume; inspect. # Print environment variables (redact secrets before sharing) printenv | sort # Expected output: a sorted list of variables; look for NODE_ENV, PORT, LOG_LEVEL, etc. # If using PM2, show the environment for a specific process pm2 env 0 # Expected output: environment for process id 0 # If using Docker, inspect the container's environment docker inspect --format '{{range .Config.Env}}{{println .}}{{end}}' # Expected output: list of env vars inside the container ### Backing Up Configuration Before editing, create a timestamped backup. For a file-based config: cp /etc/myapp/config.json /etc/myapp/config.json.backup.$(date +%Y%m%d%H%M%S) # Verify the backup exists ls -la /etc/myapp/ If configuration is stored in a secret manager or external system, export a copy to a secure location appropriate for your organization. ### Applying a Change with Verification Example: change the log level from info to debug on one instance to troubleshoot an issue. - Change the environment variable in your process manager or .env file. - Restart only that instance. - Verify the new value is active: # If using systemd systemctl show myapp --property=Environment # Expected output: Environment=LOG_LEVEL=debug ... # If using PM2 pm2 env 0 | grep LOG_LEVEL # Expected output: LOG_LEVEL=debug - Verify the app behaves correctly: check logs, run a health endpoint, and watch error rates. Recovery path: if the change causes problems, revert to the backup configuration and restart the instance. Document which instance was changed and when. ## Verification and Diagnostics After any change, verify the system behaves as expected. Diagnostics help you confirm health and isolate issues. ### Health Checks and Metrics Every production Node.js service should expose a health endpoint. A simple Express example: app.get('/health', (req, res) => { res.status(200).json({ status: 'ok', uptime: process.uptime() }); }); Verify it responds: curl -s http://localhost:3000/health # Expected output: {"status":"ok","uptime":123.45} If the health check fails, capture diagnostic data: # Check CPU and memory usage of the node process ps -p -o %cpu,%mem,cmd # Expected output: CPU and memory percentages for the process # Take a heap snapshot if memory is high (requires inspector) node --heapsnapshot-signal=SIGUSR2 --inspect app.js # Then send SIGUSR2 to the process to write a snapshot kill -USR2 ### Log Analysis Search logs for errors or anomalies: # If logs are written to a file tail -n 100 /var/log/myapp/app.log | grep -i error # Expected output: recent error lines, if any # If using journald journalctl -u myapp --since "1 hour ago" | grep -i error # Expected output: errors from the last hour Compare the error rate before and after the change. Use a monitoring tool like Prometheus metrics endpoint, Datadog, or New Relic to track error counts, latency percentiles, and saturation. ### Common Diagnostic Commands - node --trace-warnings app.js to get stack traces for warnings - node --abort-on-uncaught-exception app.js to generate core dumps on fatal errors (useful for post-mortem debugging) - NODE_DEBUG=module,http node app.js to see internal debugging output for specified modules - strace -p to see system calls (use with caution in production due to overhead) ## Failure Modes and Recovery Even with careful operations, failures happen. Plan for them. ### Common Failure Modes - Unhandled promise rejections crash the process. In Node.js 15+, unhandled rejections throw and exit by default. Mitigate by adding a global handler that logs and optionally exits gracefully: process.on('unhandledRejection', (reason, promise) => { console.error('Unhandled Rejection at:', promise, 'reason:', reason); // Optionally: process.exit(1); }); - Memory leaks cause the process to grow until the OOM killer terminates it. Use --max-old-space-size to set a limit and monitor RSS. If a leak is suspected, take heap snapshots over time and compare. - Event loop blocking makes the service unresponsive. Diagnose with tools like clinic doctor or built-in --prof to profile CPU usage. - Exhausted file descriptors can happen with too many open sockets or files. Check limits: ulimit -n # Expected output: current file descriptor limit # Increase in /etc/security/limits.conf if needed - Database connection pool exhaustion due to misconfigured pool size or leaking connections. Monitor pool usage metrics from your database driver (e.g. pg.Pool in pg has events for acquire and release). ### Recovery Procedures For each failure mode, define a recovery runbook. Example for a crashed process: - Detect : process is down (health check fails, monitoring alert fires). - Diagnose : check exit code and logs. systemctl status myapp or pm2 logs --err . - Restart : if safe, restart via your process manager. Ensure automatic restart is configured (e.g. restart: always in Docker, Restart=always in systemd). - Verify : ensure health endpoint responds and traffic is flowing. - Root cause : after service is restored, analyze logs and metrics to find why it crashed. For a memory leak, immediate recovery is to restart the process. Long-term fix: find and patch the leak, then deploy. Always document the recovery steps in the runbook and test them during game days or planned drills. ## Operations Checklist Here is a consolidated checklist for routine operations and incident response. Each item names a single accountable owner and a review cadence.
#Checklist ItemOwnerFrequencyVerification Command / Signal
1Verify Node.js version is supportedDevOps Engineer (Anna Singh)Monthlynode --version matches supported range
2Check process uptime and restartsOn-call SRE (Mark Lee)Dailypm2 ls or systemctl status myapp shows stable uptime
3Review error rate and latency metricsBackend Lead (Priya Shah)WeeklyDashboard thresholds not exceeded; no new alerts
4Rotate secrets and API keysSecurity Engineer (David Kim)QuarterlyConfirm old keys invalid, new keys working
5Test backup and restore procedureDatabase Admin (Chris Chen)Bi-monthlyRestore a backup to staging and verify data integrity
6Update dependencies with security patchesDevOps Engineer (Anna Singh)Weeklynpm audit returns zero high/critical vulnerabilities
7Review configuration changes historyTech Lead (Miguel Lopez)MonthlyGit log shows approved changes only
8Run load test in stagingQA Engineer (Sarah Jones)Before major releasesResponse time < 200ms at 1000 RPS, error rate < 0.1%
9Validate monitoring alerts and notification channelsOn-call SRE (Mark Lee)MonthlyTest alert fires and message received
10Check disk space and log rotationSysadmin (Tom Wilson)WeeklyDisk usage < 80%, logs rotated correctly
### Using the Checklist - Owner : one person is accountable for each item, not a whole team. If an item cannot be completed, that owner escalates. - Frequency : some items are daily, others monthly or quarterly. Adjust based on your risk profile. - Verification : every item has a concrete command or signal to verify completion. No vague "check health" without a measurable outcome. ## Common Pitfalls and How to Avoid Them Even experienced teams fall into these traps. Here are the most frequent and how to avoid or recover. ### 1. Running with an Unsupported Node.js Version Why it happens : teams postpone upgrades because the app works, and nobody tracks end-of-life dates. Avoidance : subscribe to Node.js release announcements, set calendar reminders, and plan upgrades well before EOL. Recovery : schedule a maintenance window, test upgrade in staging, and roll out gradually. Do not jump multiple major versions at once; upgrade stepwise. ### 2. Logging Secrets in Error Messages or Logs Why it happens : developers add debug logs that include request headers or environment variables, and they slip into production logs. Avoidance : use a logging library that supports redaction (e.g. pino with redact option). Review logs in staging for sensitive data. Recovery : immediately rotate exposed secrets, purge logs if possible, and update logging code to redact. ### 3. Not Setting Memory Limits in Containers Why it happens : Node.js sees the host's total memory, not the container limit, and may allocate too much heap, causing OOM kills. Avoidance : set --max-old-space-size based on container memory limit, or use NODE_OPTIONS=--max-old-space-size=... . Monitor memory usage. Recovery : restart with adjusted limits. Tune GC settings if needed. ### 4. Ignoring Deprecation Warnings Why it happens : warnings are easy to ignore, but deprecated APIs may be removed in the next major version, breaking your app. Avoidance : run with --throw-deprecation in CI to fail tests on deprecation warnings. Fix them before upgrading. Recovery : if you encounter a removal, check the Node.js migration guide for the specific version and update code accordingly. ### 5. No Graceful Shutdown Handling Why it happens : many apps just exit on SIGTERM, dropping in-flight requests. Avoidance : implement graceful shutdown: catch SIGTERM/SIGINT, stop accepting new connections, finish pending requests, then exit. const server = app.listen(3000); process.on('SIGTERM', () => { console.log('SIGTERM received, shutting down gracefully'); server.close(() => { console.log('HTTP server closed'); // close database connections, etc. process.exit(0); }); // Force shutdown after 30s setTimeout(() => { console.error('Could not close connections in time, forcefully shutting down'); process.exit(1); }, 30000); }); Recovery : during deployment, ensure your orchestrator sends SIGTERM and waits for exit before SIGKILL. If requests are still dropped, adjust the timeout. ### 6. Overlooking File Descriptor Limits Why it happens : default limits may be too low for high-concurrency services, causing EMFILE errors. Avoidance : set ulimit -n 65535 in your process manager or container spec. Monitor open file descriptors. Recovery : increase the limit and restart the process. Investigate if file descriptors are leaked (e.g. connections not closed). ## Conclusion A Node.js production operations checklist is only useful when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure. As a next step, choose one low-risk verification from this article, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as Express API, MongoDB, and Redis. Then expand to the full checklist, assign owners, and set review cadences. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With these practices, you can operate Node.js in production with confidence.