This checklist moves operators from an observed problem to a verified result across bare-metal (systemd), Docker, and Kubernetes NGINX Ingress Controller topologies. Every step captures state first, limits blast radius, uses explicit placeholders, verifies the outcome, and documents a tested rollback. Supported Nginx versions: 1.20.x–1.26.x (stable and mainline as of 2024). NGINX Plus-only features are called out where they differ from OSS.

## Prerequisites & Assumptions

Before any operation, confirm the following baseline. Missing prerequisites are the most common cause of failed reloads and silent misconfigurations.

Nginx version range and nginx -V interpretation Run nginx -V (capital V) to print the configure arguments. The output reveals the OpenSSL version, PCRE, zlib, and dynamic module paths. Example annotated output for Nginx 1.24.0 on Ubuntu 22.04:

nginx version: nginx/1.24.0
built by gcc 11.4.0 (Ubuntu 11.4.0-1ubuntu1~22.04)
built with OpenSSL 3.0.2 15 Mar 2022
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx \
 --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf \
 --error-log-path=/var/log/nginx/error.log \
 --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid \
 --lock-path=/var/lock/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp \
 --http-proxy-temp-path=/var/cache/nginx/proxy_temp \
 --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
 --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
 --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
 --with-http_ssl_module --with-http_v2_module --with-http_realip_module \
 --with-http_stub_status_module --with-http_gzip_static_module \
 --with-threads --with-file-aio --with-http_v3_module \
 --with-cc-opt='-g -O2 -ffile-prefix-map=/build/nginx-... -fstack-protector-strong \
 -Wformat -Werror=format-security -fPIC -Wdate-time -D_FORTIFY_SOURCE=2' \
 --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -fPIC' \
 --add-dynamic-module=../ngx_http_geoip2_module 
 Key fields to record: nginx version , built with OpenSSL , configure arguments (especially --with-http_v3_module for QUIC/HTTP3, --with-http_stub_status_module for metrics, dynamic module paths).

Deployment topology matrix

- Topology — Manager — Config Source — Reload Mechanism — Version Mapping

- Bare metal — systemd ( nginx.service ) — /etc/nginx/ files — systemctl reload nginx / nginx -s reload — Package version (e.g., nginx=1.24.0-1~jammy )

- Docker (official) — Container runtime — Bind-mounted /etc/nginx/ or baked image — docker kill -s HUP <container> or docker exec <container> nginx -s reload — Image tag nginx:1.24-alpine embeds that Nginx version

- Kubernetes Ingress Controller — Deployment + ConfigMap — ConfigMap nginx-configuration + TLS Secrets — kubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx — Controller v1.9.x embeds Nginx 1.25.x; check kubectl describe pod for --nginx-version

Required tools nginx (binary in PATH), systemctl / docker / kubectl , openssl (>= 1.1.1 for TLS 1.3), curl (with --resolve support), jq , ss or netstat , audit2allow (SELinux), certbot (for Let's Encrypt scenario).

Permissions and capabilities

- Sudo for config write ( /etc/nginx/ ) and reload ( systemctl reload nginx ).

- Read access to /var/log/nginx/ (or container log driver).

- Non-root port 80/443: grant CAP_NET_BIND_SERVICE to the Nginx binary ( setcap 'cap_net_bind_service=+ep' /usr/sbin/nginx ) or run master as root with workers dropping privileges ( user nginx; in nginx.conf ).

- Docker: container must run with --cap-add=NET_BIND_SERVICE if not root.

- Kubernetes: Ingress Controller pod typically runs as root (hostNetwork) or with NET_BIND_SERVICE capability.

Security baseline (apply before any change)

- TLS 1.2+ only: ssl_protocols TLSv1.2 TLSv1.3;

- Strong ciphers: ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';

- HSTS: add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; (only on HTTPS server blocks).

- server_tokens off; in http block.

- Rate limiting zones defined in http : limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

Architecture context Nginx uses a master/worker process model. The master reads configuration, manages worker processes, and handles signals. Workers share memory zones (SSL session cache, rate limit counters, upstream health state). Cache loader/manager processes populate disk cache on startup. Graceful reload ( nginx -s reload / SIGHUP ) starts new workers with new config while old workers drain existing connections. Binary upgrade ( kill -USR2 master) replaces the executable without dropping connections—requires master_process on; and pid file writable.

## Version & Environment Inventory

Capture a complete, timestamped snapshot before any change. Run all commands as read-only; they must not mutate state.

Commands and expected output

- Version and module inventory

nginx -v
 nginx -V 2>&1 | head -20 
 Expected: version line (e.g., nginx version: nginx/1.24.0 ) and configure arguments as shown in Prerequisites. Failure signal: non-zero exit, nginx: command not found , or missing expected modules (e.g., --with-http_v3_module absent when HTTP/3 required).

- Full configuration dump with syntax test

nginx -T 2>&1 | tee /tmp/nginx-inventory-$(date +%F_%H%M%S).txt 
 Expected: concatenated config from nginx.conf and all include files, preceded by nginx: the configuration file /etc/nginx/nginx.conf syntax is ok and nginx: configuration file /etc/nginx/nginx.conf test is successful . Failure signal: nginx: [emerg] ... syntax error, missing included file, or permission denied reading a config file. Capture the exact line number and file.

- Process and resource limits

systemctl status nginx --no-pager
 # or Docker
 docker inspect --format '{{.State.Pid}} {{.HostConfig.CapAdd}}' <container>
 # or Kubernetes
 kubectl describe pod -n ingress-nginx -l app.kubernetes.io/component=controller | head -40
 ps auxf | grep nginx
 ulimit -n
 cat /proc/$(cat /var/run/nginx.pid)/limits 2>/dev/null | grep 'Max open files' 
 Expected: master PID, worker count matching worker_processes (or auto ), open file limit >= worker_connections worker_processes 2 . Failure signal: worker count mismatch, ulimit -n < 1024, SELinux/AppArmor denials in dmesg or audit.log .

Extract from nginx -T output or nginx -V configure arguments: error_log , access_log , pid file. Verify directory permissions: ls -ld /var/log/nginx /var/run/nginx.pid .

- Log and pid paths

Annotated nginx -T truncated example (shows include hierarchy):

# /etc/nginx/nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;

events {
 worker_connections 1024;
 multi_accept on;
}

http {
 include /etc/nginx/mime.types;
 default_type application/octet-stream;
 log_format main '$remote_addr - $remote_user [$time_local] "$request" '
 '$status $body_bytes_sent "$http_referer" '
 '"$http_user_agent" "$http_x_forwarded_for"';
 access_log /var/log/nginx/access.log main;

 # Security baseline
 server_tokens off;
 ssl_protocols TLSv1.2 TLSv1.3;
 ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256...';

 # Rate limit zone
 limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

 # Upstream block
 upstream app_backend {
 least_conn;
 server 192.0.2.10:8080 max_fails=3 fail_timeout=30s;
 server 192.0.2.11:8080 max_fails=3 fail_timeout=30s;
 keepalive 32;
 }

 include /etc/nginx/conf.d/*.conf;
 include /etc/nginx/sites-enabled/*;
}

# /etc/nginx/sites-enabled/app.example.com.conf
server {
 listen 443 ssl http2;
 listen [::]:443 ssl http2;
 server_name app.example.com;

 ssl_certificate /etc/nginx/ssl/app.example.com/fullchain.pem;
 ssl_certificate_key /etc/nginx/ssl/app.example.com/privkey.pem;
 ssl_trusted_certificate /etc/nginx/ssl/app.example.com/chain.pem;
 ssl_stapling on;
 ssl_stapling_verify on;

 location /healthz {
 proxy_pass http://app_backend;
 proxy_set_header Host $host;
 proxy_set_header X-Real-IP $remote_addr;
 }

 location /api/ {
 limit_req zone=api burst=20 nodelay;
 proxy_pass http://app_backend;
 proxy_read_timeout 60s;
 proxy_set_header Host $host;
 }
} 

## Safe Configuration Path

 Follow this exact workflow for every configuration change. Never edit in place without a timestamped backup.

Workflow

- Snapshot current config

BACKUP_DIR="/etc/nginx.bak.$(date +%F_%H%M%S)"
 cp -a /etc/nginx "$BACKUP_DIR"
 echo "Backup saved to $BACKUP_DIR" 
 Example: increase proxy_read_timeout for /api/ only.

- Edit a single scoped block

# Edit the specific file
 vim /etc/nginx/sites-enabled/app.example.com.conf 
 Change:

location /api/ {
 limit_req zone=api burst=20 nodelay;
 proxy_pass http://app_backend;
 + proxy_read_timeout 60s;
 proxy_set_header Host $host;
 } 

- Syntax test

 nginx -t 
 Expected success:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
 nginx: configuration file /etc/nginx/nginx.conf test is successful 
 Failure example (missing semicolon):

nginx: [emerg] invalid parameter "proxy_read_timeout 60s" in /etc/nginx/sites-enabled/app.example.com.conf:27
 nginx: configuration file /etc/nginx/nginx.conf test failed 
 Do not proceed until nginx -t passes.

- Diff against backup

nginx -T 2>&1 | grep -v '^nginx:' | diff -u "$BACKUP_DIR/nginx.conf" - 
 Verify only the intended lines changed.

- Graceful reload

# Bare metal
 systemctl reload nginx
 # or
 nginx -s reload

 # Docker
 docker exec <container> nginx -s reload

 # Kubernetes (ConfigMap change)
 kubectl rollout restart deployment/ingress-nginx-controller -n ingress-nginx 

- Post-reload verification

 # Confirm new config active
 nginx -T 2>&1 | grep -A2 'proxy_read_timeout 60s'

 # Health check
 curl -v --resolve app.example.com:443:192.0.2.100 https://app.example.com/healthz 
 Expected: 200 OK , Server: nginx header, no 5xx in access log for the request.

Version-specific gotchas

- Dynamic modules: load_module /usr/lib/nginx/modules/ngx_http_geoip2_module.so; must appear before events block. Path changes across package versions.

- ssl_certificate / ssl_certificate_key must be readable by the worker user ( nginx or www-data ). Test: sudo -u nginx cat /etc/nginx/ssl/app.example.com/privkey.pem >/dev/null .

- include directive ordering matters: later files override earlier ones for same directive in same context.

- Kubernetes Ingress Controller: ConfigMap changes require controller pod restart; nginx -s reload inside the pod does not persist.

Blast radius control

- Change one server or location block at a time.

- Avoid global http block changes (e.g., ssl_ciphers ) without canary: deploy to one node, verify, then roll out.

- Use nginx -T diff to confirm scope.

Rollback procedure

# Restore backup and reload
mv /etc/nginx /etc/nginx.failed.$(date +%F_%H%M%S)
mv "$BACKUP_DIR" /etc/nginx
nginx -t && systemctl reload nginx
# Verify old config active
nginx -T 2>&1 | grep -c 'proxy_read_timeout 60s' # should be 0 

## Verification & Diagnostics

 Use these checks to confirm healthy operation and diagnose anomalies. Each produces an observable signal.

Health checks

# Synthetic request with SNI and forced IP (bypasses DNS)
curl -v --resolve app.example.com:443:192.0.2.100 https://app.example.com/healthz 
 Expected: HTTP/2 200 , Server: nginx , response body OK . Failure: non-2xx, curl: (56) Recv failure: Connection reset by peer , TLS handshake failure.

# Process health
systemctl is-active nginx # returns "active"
ps auxf | grep nginx # master + N workers, same master PID as pid file 
 Metrics collection Enable stub_status in a protected location:

location /nginx_status {
 stub_status on;
 allow 192.0.2.0/24; # monitoring subnet
 deny all;
} 
 Scrape:

curl -s http://192.0.2.100/nginx_status 
 Output:

Active connections: 43
server accepts handled requests
 12345 12345 67890
Reading: 6 Writing: 1 Waiting: 36 
 Prometheus exporter (sidecar or DaemonSet): nginx-prometheus-exporter scrapes /nginx_status and exposes /metrics .

Access log latency analysis

# 95th percentile request time and upstream time (last 1000 lines)
awk '{print $NF, $(NF-1)}' /var/log/nginx/access.log | tail -1000 | \
 sort -n | awk '{a[NR]=$1} END {print "p95 request_time:", a[int(NR*0.95)]}' 
 Log format must include $request_time and $upstream_response_time .

TLS validation

openssl s_client -connect app.example.com:443 -servername app.example.com -verify_return_error < /dev/null 
 Expected: Verify return code: 0 (ok) , Protocol: TLSv1.3 , Cipher: TLS_AES_256_GCM_SHA384 , cert notAfter > 30 days, OCSP stapling response present. Failure: verify error:num=10:certificate has expired , verify error:num=20:unable to get local issuer certificate , no OCSP response.

Worker health

# Worker count vs configured
grep 'worker_processes' /etc/nginx/nginx.conf
ps -eo pid,ppid,cmd | grep nginx | grep -v grep | wc -l 
 Expected: worker count == worker_processes (or CPU cores if auto ). Failure: fewer workers (OOM kill, segfault), master PID changed unexpectedly (unplanned reload/restart).

Log correlation

# Recent errors/warnings
grep -E 'error|warn|alert|emerg' /var/log/nginx/error.log | tail -20

# 5xx responses in access log
awk '$9 ~ /^5[0-9]{2}$/' /var/log/nginx/access.log | tail -20 
 Failure signals: connect() failed (111: Connection refused) while connecting to upstream , upstream timed out (110: Connection timed out) , SSL_do_handshake() failed , worker process exited on signal 11 (SIGSEGV) .

## Failure Modes & Recovery

Each failure mode includes detection, root cause isolation, and a tested recovery path.

1. Config syntax error on reload

- Detection: nginx -t fails; systemctl reload nginx returns error; new workers do not start.

- Recovery: Do not reload. Fix syntax in edited file. Run nginx -t until clean. Then reload.

- Rollback: If already reloaded and old workers gone, restore backup (see Safe Configuration Path rollback).

2. TLS cert/key mismatch or expired

- Detection: openssl s_client shows verify error:num=10 or openssl x509 -noout -modulus -in cert.pem | openssl md5 != same for key.

- Recovery: Replace cert/key pair atomically (write new files, then mv into place). Ensure permissions: chmod 640 /etc/nginx/ssl/app.example.com/privkey.pem; chown root:nginx /etc/nginx/ssl/app.example.com/privkey.pem . Run nginx -t then reload.

- Rollback: Keep previous cert/key in /etc/nginx/ssl/app.example.com/prev/ ; swap back on failure.

3. Upstream unavailable (502 Bad Gateway)

- Detection: error.log shows connect() failed (111: Connection refused) while connecting to upstream , access log shows 502.

- Root cause isolation:

- Check proxy_pass DNS: resolver 127.0.0.11 valid=10s; (Docker) or explicit resolver 8.8.8.8; + set $upstream http://app_backend; proxy_pass $upstream; for dynamic resolution.

- Verify upstream health: curl -v http://192.0.2.10:8080/healthz from Nginx host.

- Firewall/Security Groups: ss -tlnp | grep :8080 on upstream; iptables -L -n / cloud SG rules.

- Recovery: Fix upstream or DNS; Nginx re-resolves on next request (with resolver ).

- OSS note: health_check in match block is NGINX Plus only. OSS uses proxy_next_upstream error timeout http_502 http_503 http_504; + external active checks (e.g., nginx-upstream-check-module or sidecar).

4. Worker OOM / too many open files

- Detection: error.log shows accept() failed (24: Too many open files) , worker process exited on signal 9 (SIGKILL) (OOM killer). dmesg | grep -i kill .

- Recovery:

- Raise limits: ulimit -n 65535 in systemd override ( LimitNOFILE=65535 ), worker_rlimit_nofile 65535; in main context, worker_connections 4096; in events .

- Math: worker_rlimit_nofile >= worker_connections * 2 (listening + upstream sockets).

- Restart required (not reload) after limit changes: systemctl restart nginx .

- Rollback: Revert limits, restart.

5. Binary upgrade failure

- Procedure (zero-downtime binary replacement):

# 1. Start new master with new binary (USR2)
 kill -USR2 $(cat /var/run/nginx.pid)
 # 2. Verify new master PID in /var/run/nginx.pid.newbin
 sleep 2
 # 3. Gracefully shut down old workers (WINCH)
 kill -WINCH $(cat /var/run/nginx.pid.oldbin)
 # 4. Wait for old workers to drain (monitor connections)
 # 5. If new master healthy, quit old master (QUIT)
 kill -QUIT $(cat /var/run/nginx.pid.oldbin) 

- Rollback (if new master unhealthy):

 kill -HUP $(cat /var/run/nginx.pid.oldbin) # old master reaps workers, resumes
 kill -QUIT $(cat /var/run/nginx.pid) # quit new master 
 6. SELinux/AppArmor denial

- Detection: audit.log shows avc: denied { read } for pid=... comm="nginx" path="/etc/nginx/ssl/..." .

- Recovery:

grep nginx /var/log/audit/audit.log | audit2allow -M nginx_local
 semodule -i nginx_local.pp 

- Verify: nginx -t && systemctl reload nginx .

## Consolidated Operations Checklist

- Phase — Action — Command — Expected Signal — Failure Signal — Rollback/Recovery — Blast Radius — Frequency

- Inventory — Capture version & modules — nginx -V 2>&1 \&#124; tee /tmp/inv-\$(date +%F).txt — OpenSSL 1.1.1+, required modules listed — Missing --with-http_v3_module , non-zero exit — N/A (read-only) — None — Weekly / pre-change

- Inventory — Dump full config — nginx -T 2>&1 \&#124; tee /tmp/dump-\$(date +%F).txt — Syntax ok, all includes resolved — [emerg] syntax error, missing file — N/A — None — Weekly / pre-change

- Inventory — Check process & limits — ps auxf \&#124; grep nginx; cat /proc/\$(cat /var/run/nginx.pid)/limits — Worker count matches, Max open files >= 65535 — Worker count mismatch, low ulimit — Investigate systemd LimitNOFILE — None — Daily

- Config — Timestamped backup — cp -a /etc/nginx /etc/nginx.bak.\$(date +%F_%H%M%S) — Directory created — Permission denied — N/A — Config dir — Pre-change

- Config — Edit single block — vim /etc/nginx/sites-enabled/app.example.com.conf — File saved — Syntax error on save — Discard changes — One location — Per change

- Config — Syntax test — nginx -t — syntax is ok , test is successful — [emerg] with line/file — Fix syntax, retest — None — Per change

- Config — Diff vs backup — nginx -T \&#124; grep -v '^nginx:' \&#124; diff -u /etc/nginx.bak.../nginx.conf - — Only intended lines changed — Unexpected diffs — Restore backup, re-edit — Config scope — Per change

- Config — Graceful reload — systemctl reload nginx — reloaded , new workers spawned — Job failed , old workers persist — Restore backup, reload — All workers — Per change

- Verify — Health check — curl -v --resolve app.example.com:443:192.0.2.100 https://app.example.com/healthz — 200 OK , Server: nginx — Non-2xx, TLS error, timeout — Rollback config — Single endpoint — Post-change / daily

- Verify — TLS validation — openssl s_client -connect app.example.com:443 -servername app.example.com -verify_return_error — Verify return code: 0 , TLSv1.3, cert >30d — Expired cert, chain incomplete, no OCSP — Rollback cert/key — TLS termination — Weekly / post-renewal

- Verify — Metrics scrape — curl -s http://192.0.2.100/nginx_status — Active connections >0, counters increment — Connection refused, empty — Check stub_status location, allow list — Metrics endpoint — Daily

- Verify — Log scan — grep -E 'error&#124;warn' /var/log/nginx/error.log \&#124; tail -20 — No new [alert] / [emerg] — Upstream connect refused, SSL handshake fail — Investigate upstream/TLS — Log analysis — Hourly / alert-driven

- Recovery — Config rollback — mv /etc/nginx /etc/nginx.failed.\$(date +%F); mv /etc/nginx.bak... /etc/nginx; nginx -t && systemctl reload nginx — Old config active, healthz 200 — Reload fails — Binary upgrade rollback — Entire config — On failure

- Recovery — Cert rollback — mv /etc/nginx/ssl/app.example.com /etc/nginx/ssl/app.example.com.bad; mv /etc/nginx/ssl/app.example.com.prev /etc/nginx/ssl/app.example.com; systemctl reload nginx — Valid cert served — Cert still invalid — Re-run certbot — TLS termination — On renewal failure

- Recovery — Binary upgrade rollback — kill -HUP \$(cat /var/run/nginx.pid.oldbin); kill -QUIT \$(cat /var/run/nginx.pid) — Old master resumes, new master exits — Old master gone — Full service restart — Process tree — On upgrade failure

## Realistic Technical Scenario: Zero-Downtime TLS Certificate Rotation

 Context : Bare metal, systemd, Nginx 1.24.0, serving app.example.com with Let's Encrypt certificates managed by certbot . Certificate expires in 7 days. Goal: rotate without dropping connections.

Step-by-step execution

- Pre-change snapshot

BACKUP_DIR="/etc/nginx.bak.$(date +%F_%H%M%S)"
 cp -a /etc/nginx "$BACKUP_DIR"
 cp -a /etc/letsencrypt/live/app.example.com /etc/letsencrypt/live/app.example.com.prev.$(date +%F) 

- Renew certificate (certbot)

 certbot renew --cert-name app.example.com --dry-run # test first
 certbot renew --cert-name app.example.com # actual renewal 
 Expected: Congratulations, all renewals succeeded. New cert in /etc/letsencrypt/live/app.example.com/ .

- Deploy new cert/key with correct permissions

CERT_DIR="/etc/nginx/ssl/app.example.com"
 mkdir -p "$CERT_DIR"
 cp /etc/letsencrypt/live/app.example.com/fullchain.pem "$CERT_DIR/fullchain.pem.new"
 cp /etc/letsencrypt/live/app.example.com/privkey.pem "$CERT_DIR/privkey.pem.new"
 chmod 640 "$CERT_DIR/privkey.pem.new"
 chown root:nginx "$CERT_DIR/privkey.pem.new"
 mv "$CERT_DIR/fullchain.pem.new" "$CERT_DIR/fullchain.pem"
 mv "$CERT_DIR/privkey.pem.new" "$CERT_DIR/privkey.pem" 

- Verify cert/key match and validity

 openssl x509 -noout -modulus -in "$CERT_DIR/fullchain.pem" | openssl md5
 openssl rsa -noout -modulus -in "$CERT_DIR/privkey.pem" | openssl md5
 # Outputs must match
 openssl x509 -noout -dates -in "$CERT_DIR/fullchain.pem"
 # notAfter > 30 days 

- Syntax test and diff

 nginx -t
 nginx -T 2>&1 | grep -v '^nginx:' | diff -u "$BACKUP_DIR/nginx.conf" - 
 Expected: only SSL file timestamps/size differ; no directive changes.

- Graceful reload

systemctl reload nginx 

- Post-reload verification

 # TLS handshake and cert details
 openssl s_client -connect app.example.com:443 -servername app.example.com -verify_return_error < /dev/null 2>&1 | \
 grep -E 'Protocol|Cipher|Verify return|notAfter|subject=|issuer='

 # Synthetic health check with forced IP
 curl -v --resolve app.example.com:443:192.0.2.100 https://app.example.com/healthz 2>&1 | \
 grep -E 'HTTP/|Server:|< OK'

 # Log check
 sleep 2
 grep -E 'error|warn' /var/log/nginx/error.log | tail -10 
 Expected: Protocol: TLSv1.3 , Verify return code: 0 , notAfter matches new cert, HTTP/2 200 , no new errors.

- Rollback plan (if verification fails)

# Restore previous cert/key
 mv /etc/nginx/ssl/app.example.com /etc/nginx/ssl/app.example.com.failed.$(date +%F_%H%M%S)
 mv /etc/letsencrypt/live/app.example.com.prev.$(date +%F) /etc/letsencrypt/live/app.example.com
 cp /etc/letsencrypt/live/app.example.com/fullchain.pem /etc/nginx/ssl/app.example.com/
 cp /etc/letsencrypt/live/app.example.com/privkey.pem /etc/nginx/ssl/app.example.com/
 chmod 640 /etc/nginx/ssl/app.example.com/privkey.pem
 chown root:nginx /etc/nginx/ssl/app.example.com/privkey.pem
 systemctl reload nginx
 # Verify old cert active
 openssl s_client -connect app.example.com:443 -servername app.example.com < /dev/null 2>&1 | grep 'notAfter' 
 Trade-offs and risks

- Certbot renewal may fail due to rate limits or DNS challenges; --dry-run mitigates.

- Atomic mv of cert files prevents Nginx reading partial files during reload.

- Worker processes hold open file descriptors to old cert; graceful reload ensures new workers pick up new files while old workers finish TLS handshakes with old cert.

- If OCSP stapling enabled, ssl_stapling_file may need update; nginx -s reload refreshes staple from ssl_trusted_certificate .

## Conclusion

This checklist converts Nginx operations from reactive debugging into a repeatable, version-scoped discipline. Every phase—inventory, safe configuration, verification, failure recovery—ties a specific command to an observable signal and a tested rollback. The TLS rotation scenario demonstrates end-to-end execution: snapshot, change, verify, recover. Operators should adopt one low-risk verification (e.g., daily nginx -t + healthz curl), record the baseline, and expand coverage weekly. A reliable workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.