Nginx Upgrade and Migration Playbook: Low-Risk Strategies for Linux

A practical, low-risk playbook for upgrading or migrating Nginx on Linux (Debian/Ubuntu and RHEL-family) that prioritizes inventory, a narrow pilot, observable verification, and a tested rollback. The guide covers in-place minor/major upgrades and zero-downtime host migration with concrete commands, validation checks, and failure-mode recovery steps.

Prerequisites, Versions & Assumptions

Before starting, confirm the following environment details. Replace all placeholders (e.g., <CURRENT_VERSION>, <OLD_HOST_IP>) with values from your infrastructure.

Operating System & Init System

• Debian 11/12, Ubuntu 20.04/22.04/24.04 (systemd)
• RHEL 8/9, AlmaLinux 8/9, Rocky Linux 8/9 (systemd)
• Root or sudo access with systemctl privileges

Nginx Variant & Repository Source

• OS default repository (stable, older versions)
• nginx.org official repository: stable or mainline branch
• Identify current source: apt-cache policy nginx (Debian/Ubuntu) or dnf repoinfo nginx (RHEL-family)

Required Modules (verify with nginx -V 2>&1)

• Core: ngx_http_ssl_module, ngx_http_v2_module, ngx_http_realip_module
• Third-party (common): ngx_brotli (filter + static), ngx_http_geoip2_module, ngx_http_js_module (njs)
• Dynamic modules must be reinstalled or rebuilt for the target version

Security Posture

• SELinux (Enforcing/Permissive/Disabled) on RHEL-family; AppArmor on Debian/Ubuntu
• httpd_can_network_connect boolean for proxy upstream connections (SELinux)
• Certificate/key permissions: 640 owned by root:nginx (or www-data)

Network & DNS

• Reachability between old and new hosts for rsync (SSH, port 22)
• DNS TTL (Time To Live) set to 60 seconds or lower at least one TTL period before migration
• Layer 4/7 Load Balancer (LB) access for weight-based cutover (preferred over DNS for true zero-downtime)

Assumptions

• Configuration resides in /etc/nginx with includes in /etc/nginx/conf.d and/or /etc/nginx/sites-enabled
• TLS assets in /etc/letsencrypt (Certbot) or /etc/nginx/ssl
• Web roots under /var/www
• Systemd unit name is nginx.service

Architecture & Reload Mechanics

Understanding Nginx process behavior prevents surprise connection drops during upgrades.

Master/Worker Model

• Master process: Reads configuration, manages workers, handles signals (SIGHUP reload, SIGQUIT graceful shutdown, SIGTERM fast shutdown).
• Worker processes: Handle connections. Count typically equals CPU cores (worker_processes auto;).
• Shared memory zones: limit_req_zone, limit_conn_zone, proxy_cache_path keys zone — survive reload but not binary replacement.

Graceful Reload (SIGHUP / systemctl reload nginx / nginx -s reload)

1. Master parses new config; if valid, starts new workers with new config.
2. Old workers receive shutdown signal, stop accepting new connections, drain in-flight requests.
3. worker_shutdown_timeout (default varies; set explicitly, e.g., 30s) bounds drain time.
4. Old master exits after all old workers terminate.

Binary Upgrade (Live Binary Replacement)

• Supported via nginx -s reload after package installs new binary.
• Risk: Incompatible dynamic modules or changed directive syntax cause new workers to fail start, leaving old workers running (safe) or master failing (downtime).
• Critical: nginx -t validates syntax only. It does not verify:
• Certificate/key readability at runtime (permissions, SELinux context)
• Upstream connectivity
• Dynamic module .so load success
• Runtime directive deprecation warnings (e.g., ssl on;)

Graceful Shutdown (nginx -s quit / SIGQUIT)

• Workers stop accepting immediately, finish current requests, then exit.
• Use for controlled drain before package downgrade or host decommission.

Systemd Drop-In for Tuned Shutdown Create /etc/systemd/system/nginx.service.d/override.conf:

[Service]
# Ensure reload uses SIGHUP (default) and stop uses SIGQUIT with adequate timeout
KillSignal=SIGQUIT
TimeoutStopSec=60
# Explicit reload command (default is usually correct)
ExecReload=/usr/sbin/nginx -s reload

Apply: systemctl daemon-reload.

Repository Source Management (OS repo vs. nginx.org)

Pinning the nginx.org repository gives control over version selection (stable vs. mainline) and access to newer modules.

Debian/Ubuntu: nginx.org Mainline Repository

# Install prerequisites
sudo apt-get update && sudo apt-get install -y curl gnupg2 ca-certificates lsb-release

# Import nginx.org signing key
curl -fsSL https://nginx.org/keys/nginx_signing.key | sudo gpg --dearmor -o /usr/share/keyrings/nginx-archive-keyring.gpg

# Add mainline repository (replace 'mainline' with 'stable' for stable branch)
echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/mainline/ubuntu $(lsb_release -cs) nginx" | sudo tee /etc/apt/sources.list.d/nginx-mainline.list

# Pin priority to prefer nginx.org over OS repo (optional but recommended)
cat <<'EOF' | sudo tee /etc/apt/preferences.d/nginx-pin
Package: nginx*
Pin: origin nginx.org
Pin-Priority: 900
EOF

sudo apt-get update
apt-cache policy nginx  # Verify candidate version shows <TARGET_VERSION> from nginx.org

RHEL-family (RHEL 9, AlmaLinux 9, Rocky 9): nginx.org Mainline Repository

# Add repository
sudo dnf config-manager --add-repo=https://nginx.org/packages/mainline/rhel/9/x86_64/
# For stable branch: https://nginx.org/packages/rhel/9/x86_64/

# Import GPG key
sudo rpm --import https://nginx.org/keys/nginx_signing.key

# Verify
dnf repoinfo nginx-mainline
dnf --showduplicates list nginx  # Lists available versions; identify <TARGET_VERSION>

Dynamic Module Handling (nginx.org packages)

nginx.org packages split dynamic modules into separate RPMs/DEBs (e.g., nginx-module-brotli, nginx-module-njs). Install matching version:

# Debian/Ubuntu
sudo apt-get install -y nginx-module-brotli=<TARGET_VERSION>-1~$(lsb_release -cs)

# RHEL-family
sudo dnf install -y nginx-module-brotli-<TARGET_VERSION>-1.el9.ngx

Load in nginx.conf main context (before http block):

load_module modules/ngx_http_brotli_filter_module.so;
load_module modules/ngx_http_brotli_static_module.so;

Security Hardening Checklist

Apply these before and after any upgrade/migration.

TLS Configuration (validate in nginx -T output)

• Protocols: ssl_protocols TLSv1.2 TLSv1.3; (disable TLSv1, TLSv1.1)
• Ciphers: ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
• Prefer server ciphers: ssl_prefer_server_ciphers off; (modern clients negotiate best)
• HSTS: add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; (only if all subdomains HTTPS)
• OCSP Stapling: ssl_stapling on; ssl_stapling_verify on; resolver 1.1.1.1 8.8.8.8 valid=300s; resolver_timeout 5s;
• Session reuse: ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m;

Certificate/Key Permissions

# Certbot/Let's Encrypt paths (adjust for your CA)
sudo chown root:nginx /etc/letsencrypt/live/<YOUR_DOMAIN>/privkey.pem
sudo chmod 640 /etc/letsencrypt/live/<YOUR_DOMAIN>/privkey.pem
sudo chown root:nginx /etc/letsencrypt/live/<YOUR_DOMAIN>/fullchain.pem
sudo chmod 644 /etc/letsencrypt/live/<YOUR_DOMAIN>/fullchain.pem

Note: nginx -t does not validate that the master process can read these files at runtime (master runs as root, workers as nginx/www-data). Test with sudo -u nginx cat /etc/letsencrypt/live/<YOUR_DOMAIN>/privkey.pem.

SELinux (RHEL-family)

# Allow proxy upstream connections (required for proxy_pass)
sudo setsebool -P httpd_can_network_connect 1

# Allow non-standard pilot port (e.g., 8081)
sudo semanage port -a -t http_port_t -p tcp 8081

# Restore contexts after config/rsync transfer
sudo restorecon -Rv /etc/nginx
sudo restorecon -Rv /etc/letsencrypt
sudo restorecon -Rv /var/www

AppArmor (Debian/Ubuntu)

• After binary upgrade, reload profile: sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.nginx
• Verify status: sudo aa-status | grep nginx

Deprecated Directive Migration

• Replace ssl on; with listen 443 ssl; (or listen 443 ssl http2;)
• Replace spdy with http2
• Remove ssl_certificate/ssl_certificate_key from http block if duplicated in server blocks

Performance Baselines & Tuning Knobs

Capture baselines before any change. Compare post-change against these.

Baseline Metrics (capture over 10–15 minutes at typical load)

• RPS (Requests Per Second)
• p50 / p95 / p99 latency (ms)
• 5xx rate (% of total responses)
• Active connections (ss -s or nginx_status module)
• Worker CPU/memory (pidstat -p $(pgrep -d, nginx))

Key Tuning Directives (set in http or main context)

# Worker tuning
worker_processes auto;
worker_rlimit_nofile 65535;        # Matches 'ulimit -n' for worker process
worker_shutdown_timeout 30s;       # Graceful drain window

# Event model (Linux defaults to epoll; explicit is fine)
events {
    worker_connections 4096;
    multi_accept on;               # Accept multiple connections per notify
    use epoll;
}

# Buffer sizes (adjust for workload)
http {
    client_body_buffer_size 16k;
    client_max_body_size 10m;
    proxy_buffers 8 16k;
    proxy_buffer_size 16k;
    proxy_busy_buffers_size 32k;

    # Keepalive
    keepalive_timeout 65;
    keepalive_requests 1000;
    upstream backend {
        server 127.0.0.1:8080;
        keepalive 32;              # Connections cached per worker
    }

    # Compression
    gzip on;
    gzip_vary on;
    gzip_min_length 1024;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
    brotli on;                     # Requires ngx_brotli module
    brotli_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
}

Validation: After changes, run synthetic load (see Validation section) and confirm p95 latency within 10% of baseline, 5xx rate < 0.1%.

Common Failures (Expanded)

• Failure Mode — Symptom — Root Cause — Immediate Mitigation
• Missing dynamic module — nginx: [emerg] module "ngx_http_brotli_filter_module.so" not found — Package upgraded but nginx-module-brotli not installed for new version — Install matching module package; verify load_module path in nginx.conf
• Deprecated ssl on; — nginx: [emerg] the "ssl" directive is deprecated — Config uses legacy directive removed in Nginx 1.15+ — Replace with listen 443 ssl; in all server blocks
• Include path change — nginx: [emerg] open() "/etc/nginx/conf.d/.conf" failed — Package upgrade altered default nginx.conf include paths — Diff nginx -T pre/post; restore custom includes; check include /etc/nginx/sites-enabled/;
• systemd unit ExecReload change — systemctl reload nginx fails or hangs — Upstream package changed reload command (rare) — Check systemctl cat nginx; create drop-in with correct ExecReload=/usr/sbin/nginx -s reload
• Log rotation breakage — Logs stop writing; disk fills — logrotate config references old paths or missing postrotate script — Verify /etc/logrotate.d/nginx; test logrotate --debug /etc/logrotate.d/nginx
• IPv6 bind failure — nginx: [emerg] bind() to [::]:80 failed (99: Cannot assign requested address) — IPv6 disabled on host/kernel but config has listen [::]:80 — Remove IPv6 listen or enable IPv6 (sysctl net.ipv6.conf.all.disable_ipv6=0)
• Third-party module ABI mismatch — Worker segfault on start (check dmesg / coredumpctl) — Module compiled against different Nginx version/ABI — Rebuild module from source against target Nginx headers (nginx -V shows --with-cc-opt)
• Certificate/key permission denied — Workers log SSL_CTX_use_PrivateKey_file failed after reload — Key file 600 or wrong group; master (root) reads, workers (nginx) cannot — Set 640 root:nginx; restorecon on RHEL; test sudo -u nginx cat /path/key.pem

Troubleshooting Deep-Dive

Use these commands when verification fails or behavior is unexpected.

Service State & Logs

# Full systemd status with recent logs
systemctl status nginx -l

# Last 100 journal lines, no pager
journalctl -u nginx -n 100 --no-pager

# Follow error log in real time
tail -f /var/log/nginx/error.log

Process & Network Inspection

# Verify master/worker PIDs and open sockets
ss -ltnp | grep nginx

# Trace network/syscall activity on master PID (replace PID)
strace -f -e trace=network,signal -p $(cat /run/nginx.pid)

# Check file descriptors for a worker
ls -l /proc/$(pgrep -f 'nginx: worker' | head -1)/fd/

Configuration Diagnostics

# Full effective config with line numbers (requires root for included SSL files)
sudo nginx -T | grep -n '<directive>'  # Locate directive origin

# Show all server_name values
grep -R "server_name" /etc/nginx/sites-enabled /etc/nginx/conf.d 2>/dev/null

# Diff pre-change vs post-change effective config
diff -u /var/backups/nginx-effective-pre.conf /var/tmp/nginx-effective-post.conf

Module & Build Verification

# Parse build flags for required modules
nginx -V 2>&1 | tr ' ' '\n' | grep -E 'ngx_http_(ssl|v2|brotli|js|geoip2)_module'

# Check dynamic module load order (main context only)
grep '^load_module' /etc/nginx/nginx.conf

TLS Verification

# TLS 1.3 handshake with SNI (Server Name Indication)
openssl s_client -connect <HOST>:443 -servername <YOUR_DOMAIN> -tls1_3 -brief < /dev/null

# Certificate chain validation
openssl s_client -connect <HOST>:443 -servername <YOUR_DOMAIN> -showcerts < /dev/null

Validation & Acceptance Criteria

Run pre-change and post-change validation. Automate where possible.

1. Configuration Diff

# Pre-change (save during inventory)
sudo nginx -T > /var/backups/nginx-effective-pre.conf

# Post-change
sudo nginx -T > /var/tmp/nginx-effective-post.conf
diff -u /var/backups/nginx-effective-pre.conf /var/tmp/nginx-effective-post.conf

Accept: Only intentional changes (version-specific directives, new modules). No accidental drops.

2. Synthetic Transaction Script (Healthz + Static + Proxied)

Save as validate.sh, run for 60–300 seconds:

#!/usr/bin/env bash
set -euo pipefail
URLS=(
  "http://<HOST>/healthz"
  "http://<HOST>/static/logo.png"
  "http://<HOST>/api/proxied-endpoint"
)
DURATION=60
END=$((SECONDS + DURATION))
declare -A CODES
declare -A TIMES

while [ $SECONDS -lt $END ]; do
  for URL in "${URLS[@]}"; do
    OUT=$(curl -s -o /dev/null -w 'http_code=%{http_code} time_total=%{time_total}\n' --max-time 5 "$URL" 2>/dev/null || echo "http_code=000 time_total=0")
    CODE=$(echo "$OUT" | cut -d' ' -f1 | cut -d= -f2)
    TIME=$(echo "$OUT" | cut -d' ' -f2 | cut -d= -f2)
    CODES["$CODE"]=$((${CODES["$CODE"]:-0} + 1))
    TIMES["$URL"]+="$TIME "
  done
  sleep 1
done

echo "=== HTTP Code Distribution ==="
for C in "${!CODES[@]}"; do echo "$C: ${CODES[$C]}"; done | sort

echo "=== Latency Summary (seconds) ==="
for U in "${!TIMES[@]}"; do
  # Requires awk for percentile calc; simplified avg here
  AVG=$(echo "${TIMES[$U]}" | awk '{sum+=$1} END {if (NR) print sum/NR; else print 0}')
  echo "$U avg: $AVG"
done

Accept Criteria:

• 100% 2xx/3xx on healthz/static; 2xx on proxied (per app contract)
• Zero 000 (connection failures)
• p95 latency ≤ 1.1 × baseline p95

3. Concurrency Smoke Test (wrk or ab)

# wrk: 50 connections, 10 threads, 30s
wrk -t10 -c50 -d30s http://<HOST>/healthz

# ab: 1000 requests, 50 concurrent
ab -n 1000 -c 50 http://<HOST>/healthz

Accept: Error rate 0%, p95 within baseline.

4. Log Parsing for 5xx / Latency Spikes

# Last 5 minutes of access log: 5xx count and rate
sudo awk '$9 ~ /^5/ {c5++} {ctotal++} END {print "5xx:", c5, "total:", ctotal, "rate:", (ctotal?c5/ctotal*100:0)"%"}' /var/log/nginx/access.log

# p95 latency from $request_time (if logged in custom format)
sudo awk '{print $NF}' /var/log/nginx/access.log | sort -n | awk '{a[NR]=$1} END {print "p95:", a[int(NR*0.95)]}'

5. Observability Snapshot (Prometheus/Grafana or goaccess)

# Quick goaccess HTML report (run on log directory)
sudo goaccess /var/log/nginx/access.log --log-format=COMBINED -o /tmp/nginx-report.html --date-format=%d/%b/%Y --time-format=%H:%M:%S

Compare pre/post dashboards for RPS, latency, error rate, active connections.

Rollback Procedures (Per Strategy)

Strategy 1: In-Place Minor/Major Upgrade (Package Downgrade + Config Restore)

Trigger: Service fails to start, unknown directive, module load failure, elevated 5xx > threshold. Time to Restore: 2–5 minutes.

# 1. Stop current service (graceful drain)
sudo nginx -s quit
# Wait for workers to exit (max worker_shutdown_timeout)
sleep 35

# 2. Downgrade package
# Debian/Ubuntu (replace with actual previous version from apt-cache policy)
sudo apt-get install -y nginx=<PREVIOUS_VERSION> nginx-module-brotli=<PREVIOUS_VERSION>

# RHEL-family
sudo dnf downgrade -y nginx-<PREVIOUS_VERSION> nginx-module-brotli-<PREVIOUS_VERSION>

# 3. Restore configuration from backup
sudo tar -C / -xzf /var/backups/nginx-etc-<BACKUP_DATE>.tgz

# 4. Validate and start
sudo nginx -t && sudo systemctl start nginx

# 5. Verify
curl -sI http://127.0.0.1/healthz

Strategy 2: In-Place Major Upgrade with Canary Port (Config Revert + Package Downgrade)

Trigger: Canary vhost on port 8081 shows errors; main traffic unaffected. Time to Restore: 1–2 minutes (disable canary only) or 5 minutes (full downgrade).

# Fast: Disable canary vhost only
sudo rm /etc/nginx/sites-enabled/example-8081.conf
sudo nginx -t && sudo systemctl reload nginx

# Full rollback: Same as Strategy 1

Strategy 3: New-Host Migration (DNS/LB Revert + Old Host Standby)

Trigger: Post-cutover 5xx > 0.5%, TLS failures, latency spike > 2× baseline. Time to Restore: Minutes (LB weight flip) to TTL expiry (DNS).

# Preferred: Load Balancer weight flip (instant)
# Set web-new weight=0, web-old weight=100 in LB console/API

# Fallback: DNS revert (subject to client TTL caching)
# Update A/AAAA record back to <OLD_HOST_IP>
# Monitor old host logs for traffic return

# Keep old host running until new host validated offline
# On new host: stop nginx, investigate
sudo systemctl stop nginx

Graceful Drain vs. Fast Stop

• Scenario — Command — Behavior
• Planned maintenance / rollback — nginx -s quit / systemctl stop nginx (with KillSignal=SIGQUIT) — Workers finish in-flight requests (up to worker_shutdown_timeout), then exit. Zero connection drops if timeout sufficient.
• Emergency / hung workers — nginx -s stop / systemctl kill -s SIGTERM nginx — Immediate termination. In-flight requests dropped. Use only when graceful drain stalls.

Realistic Technical Scenario (End-to-End Walkthrough)

Objective: Upgrade Nginx from 1.22 (Ubuntu 22.04 OS repo) to 1.26 (nginx.org mainline) with HTTP/2, TLS 1.3, and third-party ngx_brotli. Migrate to new host with 60s TTL DNS cutover. Validate with 50 RPS synthetic load for 10 minutes. Rollback trigger: >0.5% 5xx.

Environment

• Old Host: <OLD_HOST> (Ubuntu 22.04, Nginx 1.22 from OS repo, ngx_brotli compiled from source)
• New Host: <NEW_HOST> (Ubuntu 22.04, fresh install)
• Domain: <YOUR_DOMAIN> (DNS TTL=60s, managed externally)
• Load: ~200 RPS production, mixed static + proxied API

Phase 1: Inventory & Baseline (Old Host)

# Capture versions, modules, config
nginx -V 2>&1 | tee /var/backups/nginx-build-info.txt
sudo nginx -T > /var/backups/nginx-effective-pre.conf
sudo ss -ltnp | grep nginx > /var/backups/listeners-pre.txt
apt-cache policy nginx > /var/backups/pkg-policy-pre.txt

# Baseline performance (run 10 min)
./validate.sh  # Custom script from Validation section, targeting old host
# Record: p95=45ms, 5xx=0.02%, RPS=200

Phase 2: New Host Preparation

# On <NEW_HOST>: Install nginx.org mainline + brotli module
curl -fsSL https://nginx.org/keys/nginx_signing.key | sudo gpg --dearmor -o /usr/share/keyrings/nginx-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/mainline/ubuntu jammy nginx" | sudo tee /etc/apt/sources.list.d/nginx-mainline.list
cat <<'EOF' | sudo tee /etc/apt/preferences.d/nginx-pin
Package: nginx*
Pin: origin nginx.org
Pin-Priority: 900
EOF
sudo apt-get update
sudo apt-get install -y nginx nginx-module-brotli

# Verify modules
nginx -V 2>&1 | grep -E 'brotli|ssl|v2'

# Transfer config & TLS (run from <OLD_HOST>)
sudo rsync -a --delete /etc/nginx/ <NEW_HOST>:/etc/nginx/
sudo rsync -a --delete /etc/letsencrypt/ <NEW_HOST>:/etc/letsencrypt/
sudo rsync -a --delete /var/www/ <NEW_HOST>:/var/www/

# Fix permissions on new host
sudo chown root:nginx /etc/letsencrypt/live/<YOUR_DOMAIN>/privkey.pem
sudo chmod 640 /etc/letsencrypt/live/<YOUR_DOMAIN>/privkey.pem
sudo restorecon -Rv /etc/nginx /etc/letsencrypt /var/www 2>/dev/null || true  # No-op on Ubuntu/AppArmor

Phase 3: Pilot Validation on New Host (No Production Traffic)

# On <NEW_HOST>: Enable pilot port 8081 for single vhost
sudo cp /etc/nginx/sites-available/app.conf /etc/nginx/sites-available/app-pilot.conf
sudo sed -i 's/listen 80;/listen 8081;/' /etc/nginx/sites-available/app-pilot.conf
sudo sed -i 's/listen 443 ssl;/listen 8443 ssl;/' /etc/nginx/sites-available/app-pilot.conf
sudo ln -s /etc/nginx/sites-available/app-pilot.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

# Verify pilot from old host (or admin workstation)
curl -sI --resolve <YOUR_DOMAIN>:8081:<NEW_HOST_IP> http://<YOUR_DOMAIN>:8081/healthz
curl -sIk --resolve <YOUR_DOMAIN>:8443:<NEW_HOST_IP> https://<YOUR_DOMAIN>:8443/healthz
# Check error.log: clean
sudo tail -n 20 /var/log/nginx/error.log

Phase 4: Full Validation on New Host (Production Ports, No DNS)

# Enable main config on 80/443
sudo rm /etc/nginx/sites-enabled/app-pilot.conf
sudo nginx -t && sudo systemctl reload nginx

# End-to-end with --resolve (bypasses DNS)
curl -sI --resolve <YOUR_DOMAIN>:80:<NEW_HOST_IP> http://<YOUR_DOMAIN>/
curl -sIk --resolve <YOUR_DOMAIN>:443:<NEW_HOST_IP> https://<YOUR_DOMAIN>/

# TLS 1.3 verification
openssl s_client -connect <NEW_HOST_IP>:443 -servername <YOUR_DOMAIN> -tls1_3 -brief < /dev/null

# Synthetic load: 50 RPS for 10 minutes (600s)
./validate.sh  # Modify DURATION=600, target <NEW_HOST_IP> with --resolve
# Acceptance: 5xx=0%, p95 ≤ 50ms (within 10% of 45ms baseline)

Phase 5: Cutover

# Update DNS A record to <NEW_HOST_IP> (TTL=60s already set)
# Monitor TTL expiry: watch -n 5 'dig +short <YOUR_DOMAIN>'

# Or LB cutover: set <NEW_HOST> weight=100, <OLD_HOST> weight=0

Phase 6: Post-Cutover Monitoring (15 minutes)

# On <NEW_HOST>
sudo tail -f /var/log/nginx/error.log &
# In another terminal, watch 5xx rate
watch -n 10 "awk '\$9 ~ /^5/ {c5++} {ctotal++} END {print \"5xx rate:\", ctotal?c5/ctotal*100:0 \"%\"}' /var/log/nginx/access.log"

# Synthetic validation continues
./validate.sh  # DURATION=900

Phase 7: Rollback Decision Point

• If 5xx > 0.5% sustained for 2 minutes OR p95 > 90ms OR TLS errors in logs:
• Execute LB weight flip back to <OLD_HOST> (instant)
• Or DNS revert to <OLD_HOST_IP> (wait for TTL)
• Investigate <NEW_HOST> offline
• Else: Decommission <OLD_HOST> after 24h observation.

Phase 8: Documentation

Record in runbook:

• Versions: 1.22 (OS) → 1.26.0-1~jammy (nginx.org mainline)
• Module: ngx_brotli 1.0.0 (nginx.org package)
• Timings: Pilot 15m, Validation 10m, Cutover 2m, Monitoring 15m
• Metrics: Pre p95=45ms/5xx=0.02% → Post p95=48ms/5xx=0.00%
• Commands used (exact, with versions)
• Issues: None / [list any]

Quick Reference (One-Liners)

Inventory

nginx -V 2>&1; sudo nginx -t; sudo nginx -T > /var/tmp/nginx-effective.conf; sudo ss -ltnp; apt-cache policy nginx || dnf repoinfo nginx

Backup

sudo tar -C /etc -czf /var/backups/nginx-etc-$(date +%F).tgz nginx; sudo tar -C /var -czf /var/backups/nginx-web-$(date +%F).tgz www nginx ssl letsencrypt 2>/dev/null

Pilot (Port 8081)

sudo cp /etc/nginx/sites-available/app.conf /etc/nginx/sites-available/app-8081.conf; sudo sed -i 's/listen 80;/listen 8081;/' /etc/nginx/sites-available/app-8081.conf; sudo ln -sf /etc/nginx/sites-available/app-8081.conf /etc/nginx/sites-enabled/; sudo nginx -t && sudo systemctl reload nginx

Verify

curl -sI --resolve <DOMAIN>:80:<IP> http://<DOMAIN>/; curl -sIk --resolve <DOMAIN>:443:<IP> https://<DOMAIN>/; openssl s_client -connect <IP>:443 -servername <DOMAIN> -tls1_3 -brief < /dev/null

Rollback (In-Place)

sudo nginx -s quit; sleep 35; sudo apt-get install -y nginx=<PREV_VER>; sudo tar -C / -xzf /var/backups/nginx-etc-<DATE>.tgz; sudo nginx -t && sudo systemctl start nginx

Rollback (Migration)

# LB: set old host weight=100, new host weight=0
# DNS: revert A record to <OLD_HOST_IP>

Conclusion

You can upgrade or migrate Nginx with confidence by keeping the first pilot small, validating changes with concrete checks, and preparing fast recovery steps. Start with a tight inventory that captures version, modules, repository source, TLS assets, and performance baselines. Choose the lowest-risk path that meets your objectives: in-place minor upgrade for patch-level changes, canary port for major version shifts with module dependencies, or new-host migration when uptime is non-negotiable. Prove each change in isolation using synthetic transactions, log analysis, and TLS verification before any traffic cutover. With backups in hand, a verified plan to downgrade packages or flip load balancer weights, and a short, objective checklist tied to measurable thresholds (5xx rate, p95 latency, error log cleanliness), you reduce risk, shorten maintenance windows, and make future upgrades routine rather than stressful.