Nginx Production Operations Checklist: Version-Scoped, Verifiable, Reversible

This checklist moves operators from an observed problem to a verified result across bare-metal (systemd), Docker, and Kubernetes NGINX Ingress Controller topologies. Every step captures state first, limits blast radius, uses explicit placeholders, verifies the outcome, and documents a tested rollback. Supported Nginx versions: 1.20.x–1.26.x (stable and mainline as of 2024). NGINX Plus-only features are called out where they differ from OSS.

Prerequisites & Assumptions

Before any operation, confirm the following baseline. Missing prerequisites are the most common cause of failed reloads and silent misconfigurations.

Nginx version range and nginx -V interpretation Run nginx -V (capital V) to print the configure arguments. The output reveals the OpenSSL version, PCRE, zlib, and dynamic module paths. Example annotated output for Nginx 1.24.0 on Ubuntu 22.04:

nginx version: nginx/1.24.0
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx \
  --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf \
  --error-log-path=/var/log/nginx/error.log \
  --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid \
  --lock-path=/var/lock/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp \
  --http-proxy-temp-path=/var/cache/nginx/proxy_temp \
  --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
  --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
  --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
  --with-http_ssl_module --with-http_v2_module --with-http_realip_module \
  --with-http_stub_status_module --with-http_gzip_static_module \
  --with-threads --with-file-aio --with-http_v3_module \
  --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-... -fstack-protector-strong \
  -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' \
  --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' \
  --add-dynamic-module=../ngx_http_geoip2_module

Key fields to record: nginx version, built with OpenSSL, configure arguments (especially --with-http_v3_module for QUIC/HTTP3, --with-http_stub_status_module for metrics, dynamic module paths).

Deployment topology matrix

• Topology — Manager — Config Source — Reload Mechanism — Version Mapping
• Bare metal — systemd (nginx.service) — /etc/nginx/ files — systemctl reload nginx / nginx -s reload — Package version (e.g., nginx=1.24.0-1~jammy)
• Docker (official) — Container runtime — Bind-mounted /etc/nginx/ or baked image — docker kill -s HUP <container> or docker exec <container> nginx -s reload — Image tag nginx:1.24-alpine embeds that Nginx version
• Kubernetes Ingress Controller — Deployment + ConfigMap — ConfigMap nginx-configuration + TLS Secrets — kubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx — Controller v1.9.x embeds Nginx 1.25.x; check kubectl describe pod for --nginx-version

Required tools nginx (binary in PATH), systemctl / docker / kubectl, openssl (>= 1.1.1 for TLS 1.3), curl (with --resolve support), jq, ss or netstat, audit2allow (SELinux), certbot (for Let's Encrypt scenario).

Permissions and capabilities

• Sudo for config write (/etc/nginx/) and reload (systemctl reload nginx).
• Read access to /var/log/nginx/ (or container log driver).
• Non-root port 80/443: grant CAP_NET_BIND_SERVICE to the Nginx binary (setcap 'cap_net_bind_service=+ep' /usr/sbin/nginx) or run master as root with workers dropping privileges (user nginx; in nginx.conf).
• Docker: container must run with --cap-add=NET_BIND_SERVICE if not root.
• Kubernetes: Ingress Controller pod typically runs as root (hostNetwork) or with NET_BIND_SERVICE capability.

Security baseline (apply before any change)

• TLS 1.2+ only: ssl_protocols TLSv1.2 TLSv1.3;
• Strong ciphers: ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
• HSTS: add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; (only on HTTPS server blocks).
• server_tokens off; in http block.
• Rate limiting zones defined in http: limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

Architecture context Nginx uses a master/worker process model. The master reads configuration, manages worker processes, and handles signals. Workers share memory zones (SSL session cache, rate limit counters, upstream health state). Cache loader/manager processes populate disk cache on startup. Graceful reload (nginx -s reload / SIGHUP) starts new workers with new config while old workers drain existing connections. Binary upgrade (kill -USR2 master) replaces the executable without dropping connections—requires master_process on; and pid file writable.

Version & Environment Inventory

Capture a complete, timestamped snapshot before any change. Run all commands as read-only; they must not mutate state.

Commands and expected output

1. Version and module inventory

   nginx -v
   nginx -V 2>&1 | head -20

Expected: version line (e.g., nginx version: nginx/1.24.0) and configure arguments as shown in Prerequisites. Failure signal: non-zero exit, nginx: command not found, or missing expected modules (e.g., --with-http_v3_module absent when HTTP/3 required).

1. Full configuration dump with syntax test

   nginx -T 2>&1 | tee /tmp/nginx-inventory-$(date +%F_%H%M%S).txt

Expected: concatenated config from nginx.conf and all include files, preceded by nginx: the configuration file /etc/nginx/nginx.conf syntax is ok and nginx: configuration file /etc/nginx/nginx.conf test is successful. Failure signal: nginx: [emerg] ... syntax error, missing included file, or permission denied reading a config file. Capture the exact line number and file.

1. Process and resource limits

   systemctl status nginx --no-pager
   # or Docker
   docker inspect --format '{{.State.Pid}} {{.HostConfig.CapAdd}}' <container>
   # or Kubernetes
   kubectl describe pod -n ingress-nginx -l app.kubernetes.io/component=controller | head -40
   ps auxf | grep nginx
   ulimit -n
   cat /proc/$(cat /var/run/nginx.pid)/limits 2>/dev/null | grep 'Max open files'

Expected: master PID, worker count matching worker_processes (or auto), open file limit >= worker_connections worker_processes 2. Failure signal: worker count mismatch, ulimit -n < 1024, SELinux/AppArmor denials in dmesg or audit.log.

Extract from nginx -T output or nginx -V configure arguments: error_log, access_log, pid file. Verify directory permissions: ls -ld /var/log/nginx /var/run/nginx.pid.

1. Log and pid paths

Annotated nginx -T truncated example (shows include hierarchy):

# /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
    multi_accept on;
}

http {
    include /etc/nginx/mime.types;
    default_type application/octet-stream;
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                    '$status $body_bytes_sent "$http_referer" '
                    '"$http_user_agent" "$http_x_forwarded_for"';
    access_log /var/log/nginx/access.log main;

    # Security baseline
    server_tokens off;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256...';

    # Rate limit zone
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

    # Upstream block
    upstream app_backend {
        least_conn;
        server 192.0.2.10:8080 max_fails=3 fail_timeout=30s;
        server 192.0.2.11:8080 max_fails=3 fail_timeout=30s;
        keepalive 32;
    }

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

# /etc/nginx/sites-enabled/app.example.com.conf
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name app.example.com;

    ssl_certificate /etc/nginx/ssl/app.example.com/fullchain.pem;
    ssl_certificate_key /etc/nginx/ssl/app.example.com/privkey.pem;
    ssl_trusted_certificate /etc/nginx/ssl/app.example.com/chain.pem;
    ssl_stapling on;
    ssl_stapling_verify on;

    location /healthz {
        proxy_pass http://app_backend;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }

    location /api/ {
        limit_req zone=api burst=20 nodelay;
        proxy_pass http://app_backend;
        proxy_read_timeout 60s;
        proxy_set_header Host $host;
    }
}

Safe Configuration Path

Follow this exact workflow for every configuration change. Never edit in place without a timestamped backup.

Workflow

1. Snapshot current config

   BACKUP_DIR="/etc/nginx.bak.$(date +%F_%H%M%S)"
   cp -a /etc/nginx "$BACKUP_DIR"
   echo "Backup saved to $BACKUP_DIR"

Example: increase proxy_read_timeout for /api/ only.

1. Edit a single scoped block

   # Edit the specific file
   vim /etc/nginx/sites-enabled/app.example.com.conf

Change:

    location /api/ {
        limit_req zone=api burst=20 nodelay;
        proxy_pass http://app_backend;
   +    proxy_read_timeout 60s;
        proxy_set_header Host $host;
    }

1. Syntax test

   nginx -t

Expected success:

   nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
   nginx: configuration file /etc/nginx/nginx.conf test is successful

Failure example (missing semicolon):

   nginx: [emerg] invalid parameter "proxy_read_timeout 60s" in /etc/nginx/sites-enabled/app.example.com.conf:27
   nginx: configuration file /etc/nginx/nginx.conf test failed

Do not proceed until nginx -t passes.

1. Diff against backup

   nginx -T 2>&1 | grep -v '^nginx:' | diff -u "$BACKUP_DIR/nginx.conf" -

Verify only the intended lines changed.

1. Graceful reload

   # Bare metal
   systemctl reload nginx
   # or
   nginx -s reload

   # Docker
   docker exec <container> nginx -s reload

   # Kubernetes (ConfigMap change)
   kubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx

1. Post-reload verification

   # Confirm new config active
   nginx -T 2>&1 | grep -A2 'proxy_read_timeout 60s'

   # Health check
   curl -v --resolve app.example.com:443:192.0.2.100 https://app.example.com/healthz

Expected: 200 OK, Server: nginx header, no 5xx in access log for the request.

Version-specific gotchas

• Dynamic modules: load_module /usr/lib/nginx/modules/ngx_http_geoip2_module.so; must appear before events block. Path changes across package versions.
• ssl_certificate/ssl_certificate_key must be readable by the worker user (nginx or www-data). Test: sudo -u nginx cat /etc/nginx/ssl/app.example.com/privkey.pem >/dev/null.
• include directive ordering matters: later files override earlier ones for same directive in same context.
• Kubernetes Ingress Controller: ConfigMap changes require controller pod restart; nginx -s reload inside the pod does not persist.

Blast radius control

• Change one server or location block at a time.
• Avoid global http block changes (e.g., ssl_ciphers) without canary: deploy to one node, verify, then roll out.
• Use nginx -T diff to confirm scope.

Rollback procedure

# Restore backup and reload
mv /etc/nginx /etc/nginx.failed.$(date +%F_%H%M%S)
mv "$BACKUP_DIR" /etc/nginx
nginx -t && systemctl reload nginx
# Verify old config active
nginx -T 2>&1 | grep -c 'proxy_read_timeout 60s'  # should be 0

Verification & Diagnostics

Use these checks to confirm healthy operation and diagnose anomalies. Each produces an observable signal.

Health checks

# Synthetic request with SNI and forced IP (bypasses DNS)
curl -v --resolve app.example.com:443:192.0.2.100 https://app.example.com/healthz

Expected: HTTP/2 200, Server: nginx, response body OK. Failure: non-2xx, curl: (56) Recv failure: Connection reset by peer, TLS handshake failure.

# Process health
systemctl is-active nginx  # returns "active"
ps auxf | grep nginx       # master + N workers, same master PID as pid file

Metrics collection Enable stub_status in a protected location:

location /nginx_status {
    stub_status on;
    allow 192.0.2.0/24;  # monitoring subnet
    deny all;
}

Scrape:

curl -s http://192.0.2.100/nginx_status

Output:

Active connections: 43
server accepts handled requests
 12345 12345 67890
Reading: 6 Writing: 1 Waiting: 36

Prometheus exporter (sidecar or DaemonSet): nginx-prometheus-exporter scrapes /nginx_status and exposes /metrics.

Access log latency analysis

# 95th percentile request time and upstream time (last 1000 lines)
awk '{print $NF, $(NF-1)}' /var/log/nginx/access.log | tail -1000 | \
  sort -n | awk '{a[NR]=$1} END {print "p95 request_time:", a[int(NR*0.95)]}'

Log format must include $request_time and $upstream_response_time.

TLS validation

openssl s_client -connect app.example.com:443 -servername app.example.com -verify_return_error < /dev/null

Expected: Verify return code: 0 (ok), Protocol: TLSv1.3, Cipher: TLS_AES_256_GCM_SHA384, cert notAfter > 30 days, OCSP stapling response present. Failure: verify error:num=10:certificate has expired, verify error:num=20:unable to get local issuer certificate, no OCSP response.

Worker health

# Worker count vs configured
grep 'worker_processes' /etc/nginx/nginx.conf
ps -eo pid,ppid,cmd | grep nginx | grep -v grep | wc -l

Expected: worker count == worker_processes (or CPU cores if auto). Failure: fewer workers (OOM kill, segfault), master PID changed unexpectedly (unplanned reload/restart).

Log correlation

# Recent errors/warnings
grep -E 'error|warn|alert|emerg' /var/log/nginx/error.log | tail -20

# 5xx responses in access log
awk '$9 ~ /^5[0-9]{2}$/' /var/log/nginx/access.log | tail -20

Failure signals: connect() failed (111: Connection refused) while connecting to upstream, upstream timed out (110: Connection timed out), SSL_do_handshake() failed, worker process exited on signal 11 (SIGSEGV).

Failure Modes & Recovery

Each failure mode includes detection, root cause isolation, and a tested recovery path.

1. Config syntax error on reload

• Detection: nginx -t fails; systemctl reload nginx returns error; new workers do not start.
• Recovery: Do not reload. Fix syntax in edited file. Run nginx -t until clean. Then reload.
• Rollback: If already reloaded and old workers gone, restore backup (see Safe Configuration Path rollback).

2. TLS cert/key mismatch or expired

• Detection: openssl s_client shows verify error:num=10 or openssl x509 -noout -modulus -in cert.pem | openssl md5 != same for key.
• Recovery: Replace cert/key pair atomically (write new files, then mv into place). Ensure permissions: chmod 640 /etc/nginx/ssl/app.example.com/privkey.pem; chown root:nginx /etc/nginx/ssl/app.example.com/privkey.pem. Run nginx -t then reload.
• Rollback: Keep previous cert/key in /etc/nginx/ssl/app.example.com/prev/; swap back on failure.

3. Upstream unavailable (502 Bad Gateway)

• Detection: error.log shows connect() failed (111: Connection refused) while connecting to upstream, access log shows 502.
• Root cause isolation:
• Check proxy_pass DNS: resolver 127.0.0.11 valid=10s; (Docker) or explicit resolver 8.8.8.8; + set $upstream http://app_backend; proxy_pass $upstream; for dynamic resolution.
• Verify upstream health: curl -v http://192.0.2.10:8080/healthz from Nginx host.
• Firewall/Security Groups: ss -tlnp | grep :8080 on upstream; iptables -L -n / cloud SG rules.
• Recovery: Fix upstream or DNS; Nginx re-resolves on next request (with resolver).
• OSS note: health_check in match block is NGINX Plus only. OSS uses proxy_next_upstream error timeout http_502 http_503 http_504; + external active checks (e.g., nginx-upstream-check-module or sidecar).

4. Worker OOM / too many open files

• Detection: error.log shows accept() failed (24: Too many open files), worker process exited on signal 9 (SIGKILL) (OOM killer). dmesg | grep -i kill.
• Recovery:
• Raise limits: ulimit -n 65535 in systemd override (LimitNOFILE=65535), worker_rlimit_nofile 65535; in main context, worker_connections 4096; in events.
• Math: worker_rlimit_nofile >= worker_connections * 2 (listening + upstream sockets).
• Restart required (not reload) after limit changes: systemctl restart nginx.
• Rollback: Revert limits, restart.

5. Binary upgrade failure

• Procedure (zero-downtime binary replacement):

  # 1. Start new master with new binary (USR2)
  kill -USR2 $(cat /var/run/nginx.pid)
  # 2. Verify new master PID in /var/run/nginx.pid.newbin
  sleep 2
  # 3. Gracefully shut down old workers (WINCH)
  kill -WINCH $(cat /var/run/nginx.pid.oldbin)
  # 4. Wait for old workers to drain (monitor connections)
  # 5. If new master healthy, quit old master (QUIT)
  kill -QUIT $(cat /var/run/nginx.pid.oldbin)

• Rollback (if new master unhealthy):

  kill -HUP $(cat /var/run/nginx.pid.oldbin)   # old master reaps workers, resumes
  kill -QUIT $(cat /var/run/nginx.pid)         # quit new master

6. SELinux/AppArmor denial

• Detection: audit.log shows avc: denied { read } for pid=... comm="nginx" path="/etc/nginx/ssl/...".
• Recovery:

  grep nginx /var/log/audit/audit.log | audit2allow -M nginx_local
  semodule -i nginx_local.pp

• Verify: nginx -t && systemctl reload nginx.

Consolidated Operations Checklist

• Phase — Action — Command — Expected Signal — Failure Signal — Rollback/Recovery — Blast Radius — Frequency
• Inventory — Capture version & modules — nginx -V 2>&1 \| tee /tmp/inv-\$(date +%F).txt — OpenSSL 1.1.1+, required modules listed — Missing --with-http_v3_module, non-zero exit — N/A (read-only) — None — Weekly / pre-change
• Inventory — Dump full config — nginx -T 2>&1 \| tee /tmp/dump-\$(date +%F).txt — Syntax ok, all includes resolved — [emerg] syntax error, missing file — N/A — None — Weekly / pre-change
• Inventory — Check process & limits — ps auxf \| grep nginx; cat /proc/\$(cat /var/run/nginx.pid)/limits — Worker count matches, Max open files >= 65535 — Worker count mismatch, low ulimit — Investigate systemd LimitNOFILE — None — Daily
• Config — Timestamped backup — cp -a /etc/nginx /etc/nginx.bak.\$(date +%F_%H%M%S) — Directory created — Permission denied — N/A — Config dir — Pre-change
• Config — Edit single block — vim /etc/nginx/sites-enabled/app.example.com.conf — File saved — Syntax error on save — Discard changes — One location — Per change
• Config — Syntax test — nginx -t — syntax is ok, test is successful — [emerg] with line/file — Fix syntax, retest — None — Per change
• Config — Diff vs backup — nginx -T \| grep -v '^nginx:' \| diff -u /etc/nginx.bak.../nginx.conf - — Only intended lines changed — Unexpected diffs — Restore backup, re-edit — Config scope — Per change
• Config — Graceful reload — systemctl reload nginx — reloaded, new workers spawned — Job failed, old workers persist — Restore backup, reload — All workers — Per change
• Verify — Health check — curl -v --resolve app.example.com:443:192.0.2.100 https://app.example.com/healthz — 200 OK, Server: nginx — Non-2xx, TLS error, timeout — Rollback config — Single endpoint — Post-change / daily
• Verify — TLS validation — openssl s_client -connect app.example.com:443 -servername app.example.com -verify_return_error — Verify return code: 0, TLSv1.3, cert >30d — Expired cert, chain incomplete, no OCSP — Rollback cert/key — TLS termination — Weekly / post-renewal
• Verify — Metrics scrape — curl -s http://192.0.2.100/nginx_status — Active connections >0, counters increment — Connection refused, empty — Check stub_status location, allow list — Metrics endpoint — Daily
• Verify — Log scan — grep -E 'error|warn' /var/log/nginx/error.log \| tail -20 — No new [alert]/[emerg] — Upstream connect refused, SSL handshake fail — Investigate upstream/TLS — Log analysis — Hourly / alert-driven
• Recovery — Config rollback — mv /etc/nginx /etc/nginx.failed.\$(date +%F); mv /etc/nginx.bak... /etc/nginx; nginx -t && systemctl reload nginx — Old config active, healthz 200 — Reload fails — Binary upgrade rollback — Entire config — On failure
• Recovery — Cert rollback — mv /etc/nginx/ssl/app.example.com /etc/nginx/ssl/app.example.com.bad; mv /etc/nginx/ssl/app.example.com.prev /etc/nginx/ssl/app.example.com; systemctl reload nginx — Valid cert served — Cert still invalid — Re-run certbot — TLS termination — On renewal failure
• Recovery — Binary upgrade rollback — kill -HUP \$(cat /var/run/nginx.pid.oldbin); kill -QUIT \$(cat /var/run/nginx.pid) — Old master resumes, new master exits — Old master gone — Full service restart — Process tree — On upgrade failure

Realistic Technical Scenario: Zero-Downtime TLS Certificate Rotation

Context: Bare metal, systemd, Nginx 1.24.0, serving app.example.com with Let's Encrypt certificates managed by certbot. Certificate expires in 7 days. Goal: rotate without dropping connections.

Step-by-step execution

1. Pre-change snapshot

   BACKUP_DIR="/etc/nginx.bak.$(date +%F_%H%M%S)"
   cp -a /etc/nginx "$BACKUP_DIR"
   cp -a /etc/letsencrypt/live/app.example.com /etc/letsencrypt/live/app.example.com.prev.$(date +%F)

1. Renew certificate (certbot)

   certbot renew --cert-name app.example.com --dry-run  # test first
   certbot renew --cert-name app.example.com            # actual renewal

Expected: Congratulations, all renewals succeeded. New cert in /etc/letsencrypt/live/app.example.com/.

1. Deploy new cert/key with correct permissions

   CERT_DIR="/etc/nginx/ssl/app.example.com"
   mkdir -p "$CERT_DIR"
   cp /etc/letsencrypt/live/app.example.com/fullchain.pem "$CERT_DIR/fullchain.pem.new"
   cp /etc/letsencrypt/live/app.example.com/privkey.pem "$CERT_DIR/privkey.pem.new"
   chmod 640 "$CERT_DIR/privkey.pem.new"
   chown root:nginx "$CERT_DIR/privkey.pem.new"
   mv "$CERT_DIR/fullchain.pem.new" "$CERT_DIR/fullchain.pem"
   mv "$CERT_DIR/privkey.pem.new" "$CERT_DIR/privkey.pem"

1. Verify cert/key match and validity

   openssl x509 -noout -modulus -in "$CERT_DIR/fullchain.pem" | openssl md5
   openssl rsa -noout -modulus -in "$CERT_DIR/privkey.pem" | openssl md5
   # Outputs must match
   openssl x509 -noout -dates -in "$CERT_DIR/fullchain.pem"
   # notAfter > 30 days

1. Syntax test and diff

   nginx -t
   nginx -T 2>&1 | grep -v '^nginx:' | diff -u "$BACKUP_DIR/nginx.conf" -

Expected: only SSL file timestamps/size differ; no directive changes.

1. Graceful reload

   systemctl reload nginx

1. Post-reload verification

   # TLS handshake and cert details
   openssl s_client -connect app.example.com:443 -servername app.example.com -verify_return_error < /dev/null 2>&1 | \
     grep -E 'Protocol|Cipher|Verify return|notAfter|subject=|issuer='

   # Synthetic health check with forced IP
   curl -v --resolve app.example.com:443:192.0.2.100 https://app.example.com/healthz 2>&1 | \
     grep -E 'HTTP/|Server:|< OK'

   # Log check
   sleep 2
   grep -E 'error|warn' /var/log/nginx/error.log | tail -10

Expected: Protocol: TLSv1.3, Verify return code: 0, notAfter matches new cert, HTTP/2 200, no new errors.

1. Rollback plan (if verification fails)

   # Restore previous cert/key
   mv /etc/nginx/ssl/app.example.com /etc/nginx/ssl/app.example.com.failed.$(date +%F_%H%M%S)
   mv /etc/letsencrypt/live/app.example.com.prev.$(date +%F) /etc/letsencrypt/live/app.example.com
   cp /etc/letsencrypt/live/app.example.com/fullchain.pem /etc/nginx/ssl/app.example.com/
   cp /etc/letsencrypt/live/app.example.com/privkey.pem /etc/nginx/ssl/app.example.com/
   chmod 640 /etc/nginx/ssl/app.example.com/privkey.pem
   chown root:nginx /etc/nginx/ssl/app.example.com/privkey.pem
   systemctl reload nginx
   # Verify old cert active
   openssl s_client -connect app.example.com:443 -servername app.example.com < /dev/null 2>&1 | grep 'notAfter'

Trade-offs and risks

• Certbot renewal may fail due to rate limits or DNS challenges; --dry-run mitigates.
• Atomic mv of cert files prevents Nginx reading partial files during reload.
• Worker processes hold open file descriptors to old cert; graceful reload ensures new workers pick up new files while old workers finish TLS handshakes with old cert.
• If OCSP stapling enabled, ssl_stapling_file may need update; nginx -s reload refreshes staple from ssl_trusted_certificate.

Conclusion

This checklist converts Nginx operations from reactive debugging into a repeatable, version-scoped discipline. Every phase—inventory, safe configuration, verification, failure recovery—ties a specific command to an observable signal and a tested rollback. The TLS rotation scenario demonstrates end-to-end execution: snapshot, change, verify, recover. Operators should adopt one low-risk verification (e.g., daily nginx -t + healthz curl), record the baseline, and expand coverage weekly. A reliable workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.