A practical, low-risk playbook for upgrading or migrating Nginx on Linux (Debian/Ubuntu and RHEL-family) that prioritizes inventory, a narrow pilot, observable verification, and a tested rollback. The guide covers in-place minor/major upgrades and zero-downtime host migration with concrete commands, validation checks, and failure-mode recovery steps.

## Prerequisites, Versions & Assumptions

Before starting, confirm the following environment details. Replace all placeholders (e.g., <CURRENT_VERSION> ,  ) with values from your infrastructure.

Operating System & Init System

- Debian 11/12, Ubuntu 20.04/22.04/24.04 (systemd)

- RHEL 8/9, AlmaLinux 8/9, Rocky Linux 8/9 (systemd)

- Root or sudo access with systemctl privileges

Nginx Variant & Repository Source

- OS default repository (stable, older versions)

- nginx.org official repository: stable or mainline branch

- Identify current source: apt-cache policy nginx (Debian/Ubuntu) or dnf repoinfo nginx (RHEL-family)

Required Modules (verify with nginx -V 2>&1 )

- Core: ngx_http_ssl_module , ngx_http_v2_module , ngx_http_realip_module

- Third-party (common): ngx_brotli (filter + static), ngx_http_geoip2_module , ngx_http_js_module (njs)

- Dynamic modules must be reinstalled or rebuilt for the target version

Security Posture

- SELinux (Enforcing/Permissive/Disabled) on RHEL-family; AppArmor on Debian/Ubuntu

- httpd_can_network_connect boolean for proxy upstream connections (SELinux)

- Certificate/key permissions: 640 owned by root:nginx (or www-data )

Network & DNS

- Reachability between old and new hosts for rsync (SSH, port 22)

- DNS TTL (Time To Live) set to 60 seconds or lower at least one TTL period before migration

- Layer 4/7 Load Balancer (LB) access for weight-based cutover (preferred over DNS for true zero-downtime)

Assumptions

- Configuration resides in /etc/nginx with includes in /etc/nginx/conf.d and/or /etc/nginx/sites-enabled

- TLS assets in /etc/letsencrypt (Certbot) or /etc/nginx/ssl

- Web roots under /var/www

- Systemd unit name is nginx.service

## Architecture & Reload Mechanics

Understanding Nginx process behavior prevents surprise connection drops during upgrades.

Master/Worker Model

- Master process : Reads configuration, manages workers, handles signals (SIGHUP reload, SIGQUIT graceful shutdown, SIGTERM fast shutdown).

- Worker processes : Handle connections. Count typically equals CPU cores ( worker_processes auto; ).

- Shared memory zones : limit_req_zone , limit_conn_zone , proxy_cache_path keys zone — survive reload but not binary replacement.

Graceful Reload (SIGHUP / systemctl reload nginx / nginx -s reload )

- Master parses new config; if valid, starts new workers with new config.

- Old workers receive shutdown signal, stop accepting new connections, drain in-flight requests.

- worker_shutdown_timeout (default varies; set explicitly, e.g., 30s ) bounds drain time.

- Old master exits after all old workers terminate.

Binary Upgrade (Live Binary Replacement)

- Supported via nginx -s reload after package installs new binary.

- Risk: Incompatible dynamic modules or changed directive syntax cause new workers to fail start, leaving old workers running (safe) or master failing (downtime).

- Critical : nginx -t validates syntax only. It does not verify:

- Certificate/key readability at runtime (permissions, SELinux context)

- Upstream connectivity

- Dynamic module .so load success

- Runtime directive deprecation warnings (e.g., ssl on; )

Graceful Shutdown ( nginx -s quit / SIGQUIT)

- Workers stop accepting immediately, finish current requests, then exit.

- Use for controlled drain before package downgrade or host decommission.

Systemd Drop-In for Tuned Shutdown Create /etc/systemd/system/nginx.service.d/override.conf :

[Service]
# Ensure reload uses SIGHUP (default) and stop uses SIGQUIT with adequate timeout
KillSignal=SIGQUIT
TimeoutStopSec=60
# Explicit reload command (default is usually correct)
ExecReload=/usr/sbin/nginx -s reload 
 Apply: systemctl daemon-reload .

## Repository Source Management (OS repo vs. nginx.org)

Pinning the nginx.org repository gives control over version selection (stable vs. mainline) and access to newer modules.

### Debian/Ubuntu: nginx.org Mainline Repository

# Install prerequisites
sudo apt-get update && sudo apt-get install -y curl gnupg2 ca-certificates lsb-release

# Import nginx.org signing key
curl -fsSL https://nginx.org/keys/nginx_signing.key | sudo gpg --dearmor -o /usr/share/keyrings/nginx-archive-keyring.gpg

# Add mainline repository (replace 'mainline' with 'stable' for stable branch)
echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/mainline/ubuntu $(lsb_release -cs) nginx" | sudo tee /etc/apt/sources.list.d/nginx-mainline.list

# Pin priority to prefer nginx.org over OS repo (optional but recommended)
cat <<'EOF' | sudo tee /etc/apt/preferences.d/nginx-pin
Package: nginx*
Pin: origin nginx.org
Pin-Priority: 900
EOF

sudo apt-get update
apt-cache policy nginx # Verify candidate version shows <TARGET_VERSION> from nginx.org 

### RHEL-family (RHEL 9, AlmaLinux 9, Rocky 9): nginx.org Mainline Repository

 # Add repository
sudo dnf config-manager --add-repo=https://nginx.org/packages/mainline/rhel/9/x86_64/
# For stable branch: https://nginx.org/packages/rhel/9/x86_64/

# Import GPG key
sudo rpm --import https://nginx.org/keys/nginx_signing.key

# Verify
dnf repoinfo nginx-mainline
dnf --showduplicates list nginx # Lists available versions; identify <TARGET_VERSION> 

### Dynamic Module Handling (nginx.org packages)

 nginx.org packages split dynamic modules into separate RPMs/DEBs (e.g., nginx-module-brotli , nginx-module-njs ). Install matching version:

# Debian/Ubuntu
sudo apt-get install -y nginx-module-brotli=<TARGET_VERSION>-1~$(lsb_release -cs)

# RHEL-family
sudo dnf install -y nginx-module-brotli-<TARGET_VERSION>-1.el9.ngx 
 Load in nginx.conf main context (before http block):

load_module modules/ngx_http_brotli_filter_module.so;
load_module modules/ngx_http_brotli_static_module.so; 

## Security Hardening Checklist

 Apply these before and after any upgrade/migration.

TLS Configuration (validate in nginx -T output)

- Protocols: ssl_protocols TLSv1.2 TLSv1.3; (disable TLSv1, TLSv1.1)

- Ciphers: ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';

- Prefer server ciphers: ssl_prefer_server_ciphers off; (modern clients negotiate best)

- HSTS: add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always; (only if all subdomains HTTPS)

- OCSP Stapling: ssl_stapling on; ssl_stapling_verify on; resolver 1.1.1.1 8.8.8.8 valid=300s; resolver_timeout 5s;

- Session reuse: ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m;

Certificate/Key Permissions

# Certbot/Let's Encrypt paths (adjust for your CA)
sudo chown root:nginx /etc/letsencrypt/live/<YOUR_DOMAIN>/privkey.pem
sudo chmod 640 /etc/letsencrypt/live/<YOUR_DOMAIN>/privkey.pem
sudo chown root:nginx /etc/letsencrypt/live/<YOUR_DOMAIN>/fullchain.pem
sudo chmod 644 /etc/letsencrypt/live/<YOUR_DOMAIN>/fullchain.pem 
 Note : nginx -t does not validate that the master process can read these files at runtime (master runs as root, workers as nginx / www-data ). Test with sudo -u nginx cat /etc/letsencrypt/live/<YOUR_DOMAIN>/privkey.pem .

SELinux (RHEL-family)

# Allow proxy upstream connections (required for proxy_pass)
sudo setsebool -P httpd_can_network_connect 1

# Allow non-standard pilot port (e.g., 8081)
sudo semanage port -a -t http_port_t -p tcp 8081

# Restore contexts after config/rsync transfer
sudo restorecon -Rv /etc/nginx
sudo restorecon -Rv /etc/letsencrypt
sudo restorecon -Rv /var/www 
 AppArmor (Debian/Ubuntu)

- After binary upgrade, reload profile: sudo apparmor_parser -r /etc/apparmor.d/usr.sbin.nginx

- Verify status: sudo aa-status | grep nginx

Deprecated Directive Migration

- Replace ssl on; with listen 443 ssl; (or listen 443 ssl http2; )

- Replace spdy with http2

- Remove ssl_certificate / ssl_certificate_key from http block if duplicated in server blocks

## Performance Baselines & Tuning Knobs

Capture baselines before any change. Compare post-change against these.

Baseline Metrics (capture over 10–15 minutes at typical load)

- RPS (Requests Per Second)

- p50 / p95 / p99 latency (ms)

- 5xx rate (% of total responses)

- Active connections ( ss -s or nginx_status module)

- Worker CPU/memory ( pidstat -p $(pgrep -d, nginx) )

Key Tuning Directives (set in http or main context)

# Worker tuning
worker_processes auto;
worker_rlimit_nofile 65535; # Matches 'ulimit -n' for worker process
worker_shutdown_timeout 30s; # Graceful drain window

# Event model (Linux defaults to epoll; explicit is fine)
events {
 worker_connections 4096;
 multi_accept on; # Accept multiple connections per notify
 use epoll;
}

# Buffer sizes (adjust for workload)
http {
 client_body_buffer_size 16k;
 client_max_body_size 10m;
 proxy_buffers 8 16k;
 proxy_buffer_size 16k;
 proxy_busy_buffers_size 32k;

 # Keepalive
 keepalive_timeout 65;
 keepalive_requests 1000;
 upstream backend {
 server 127.0.0.1:8080;
 keepalive 32; # Connections cached per worker
 }

 # Compression
 gzip on;
 gzip_vary on;
 gzip_min_length 1024;
 gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
 brotli on; # Requires ngx_brotli module
 brotli_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
} 
 Validation : After changes, run synthetic load (see Validation section) and confirm p95 latency within 10% of baseline, 5xx rate < 0.1%.

## Common Failures (Expanded)

- Failure Mode — Symptom — Root Cause — Immediate Mitigation

- Missing dynamic module — nginx: [emerg] module "ngx_http_brotli_filter_module.so" not found — Package upgraded but nginx-module-brotli not installed for new version — Install matching module package; verify load_module path in nginx.conf

- Deprecated ssl on; — nginx: [emerg] the "ssl" directive is deprecated — Config uses legacy directive removed in Nginx 1.15+ — Replace with listen 443 ssl; in all server blocks

- Include path change — nginx: [emerg] open() "/etc/nginx/conf.d/ .conf" failed — Package upgrade altered default nginx.conf include paths — Diff nginx -T pre/post; restore custom includes; check include /etc/nginx/sites-enabled/ ;

- systemd unit ExecReload change — systemctl reload nginx fails or hangs — Upstream package changed reload command (rare) — Check systemctl cat nginx ; create drop-in with correct ExecReload=/usr/sbin/nginx -s reload

- Log rotation breakage — Logs stop writing; disk fills — logrotate config references old paths or missing postrotate script — Verify /etc/logrotate.d/nginx ; test logrotate --debug /etc/logrotate.d/nginx

- IPv6 bind failure — nginx: [emerg] bind() to [::]:80 failed (99: Cannot assign requested address) — IPv6 disabled on host/kernel but config has listen [::]:80 — Remove IPv6 listen or enable IPv6 ( sysctl net.ipv6.conf.all.disable_ipv6=0 )

- Third-party module ABI mismatch — Worker segfault on start (check dmesg / coredumpctl ) — Module compiled against different Nginx version/ABI — Rebuild module from source against target Nginx headers ( nginx -V shows --with-cc-opt )

- Certificate/key permission denied — Workers log SSL_CTX_use_PrivateKey_file failed after reload — Key file 600 or wrong group; master (root) reads, workers ( nginx ) cannot — Set 640 root:nginx ; restorecon on RHEL; test sudo -u nginx cat /path/key.pem

## Troubleshooting Deep-Dive

Use these commands when verification fails or behavior is unexpected.

Service State & Logs

# Full systemd status with recent logs
systemctl status nginx -l

# Last 100 journal lines, no pager
journalctl -u nginx -n 100 --no-pager

# Follow error log in real time
tail -f /var/log/nginx/error.log 
 Process & Network Inspection

# Verify master/worker PIDs and open sockets
ss -ltnp | grep nginx

# Trace network/syscall activity on master PID (replace PID)
strace -f -e trace=network,signal -p $(cat /run/nginx.pid)

# Check file descriptors for a worker
ls -l /proc/$(pgrep -f 'nginx: worker' | head -1)/fd/ 
 Configuration Diagnostics

# Full effective config with line numbers (requires root for included SSL files)
sudo nginx -T | grep -n '<directive>' # Locate directive origin

# Show all server_name values
grep -R "server_name" /etc/nginx/sites-enabled /etc/nginx/conf.d 2>/dev/null

# Diff pre-change vs post-change effective config
diff -u /var/backups/nginx-effective-pre.conf /var/tmp/nginx-effective-post.conf 
 Module & Build Verification

# Parse build flags for required modules
nginx -V 2>&1 | tr ' ' '\n' | grep -E 'ngx_http_(ssl|v2|brotli|js|geoip2)_module'

# Check dynamic module load order (main context only)
grep '^load_module' /etc/nginx/nginx.conf 
 TLS Verification

# TLS 1.3 handshake with SNI (Server Name Indication)
openssl s_client -connect <HOST>:443 -servername <YOUR_DOMAIN> -tls1_3 -brief < /dev/null

# Certificate chain validation
openssl s_client -connect <HOST>:443 -servername <YOUR_DOMAIN> -showcerts < /dev/null 

## Validation & Acceptance Criteria

 Run pre-change and post-change validation. Automate where possible.

### 1. Configuration Diff

# Pre-change (save during inventory)
sudo nginx -T > /var/backups/nginx-effective-pre.conf

# Post-change
sudo nginx -T > /var/tmp/nginx-effective-post.conf
diff -u /var/backups/nginx-effective-pre.conf /var/tmp/nginx-effective-post.conf 
 Accept : Only intentional changes (version-specific directives, new modules). No accidental drops.

### 2. Synthetic Transaction Script (Healthz + Static + Proxied)

Save as validate.sh , run for 60–300 seconds:

#!/usr/bin/env bash
set -euo pipefail
URLS=(
 "http://<HOST>/healthz"
 "http://<HOST>/static/logo.png"
 "http://<HOST>/api/proxied-endpoint"
)
DURATION=60
END=$((SECONDS + DURATION))
declare -A CODES
declare -A TIMES

while [ $SECONDS -lt $END ]; do
 for URL in "${URLS[@]}"; do
 OUT=$(curl -s -o /dev/null -w 'http_code=%{http_code} time_total=%{time_total}\n' --max-time 5 "$URL" 2>/dev/null || echo "http_code=000 time_total=0")
 CODE=$(echo "$OUT" | cut -d' ' -f1 | cut -d= -f2)
 TIME=$(echo "$OUT" | cut -d' ' -f2 | cut -d= -f2)
 CODES["$CODE"]=$((${CODES["$CODE"]:-0} + 1))
 TIMES["$URL"]+="$TIME "
 done
 sleep 1
done

echo "=== HTTP Code Distribution ==="
for C in "${!CODES[@]}"; do echo "$C: ${CODES[$C]}"; done | sort

echo "=== Latency Summary (seconds) ==="
for U in "${!TIMES[@]}"; do
 # Requires awk for percentile calc; simplified avg here
 AVG=$(echo "${TIMES[$U]}" | awk '{sum+=$1} END {if (NR) print sum/NR; else print 0}')
 echo "$U avg: $AVG"
done 
 Accept Criteria :

- 100% 2xx / 3xx on healthz/static; 2xx on proxied (per app contract)

- Zero 000 (connection failures)

- p95 latency ≤ 1.1 × baseline p95

### 3. Concurrency Smoke Test ( wrk or ab )

# wrk: 50 connections, 10 threads, 30s
wrk -t10 -c50 -d30s http://<HOST>/healthz

# ab: 1000 requests, 50 concurrent
ab -n 1000 -c 50 http://<HOST>/healthz 
 Accept : Error rate 0%, p95 within baseline.

### 4. Log Parsing for 5xx / Latency Spikes

# Last 5 minutes of access log: 5xx count and rate
sudo awk '$9 ~ /^5/ {c5++} {ctotal++} END {print "5xx:", c5, "total:", ctotal, "rate:", (ctotal?c5/ctotal*100:0)"%"}' /var/log/nginx/access.log

# p95 latency from $request_time (if logged in custom format)
sudo awk '{print $NF}' /var/log/nginx/access.log | sort -n | awk '{a[NR]=$1} END {print "p95:", a[int(NR*0.95)]}' 

### 5. Observability Snapshot (Prometheus/Grafana or goaccess )

 # Quick goaccess HTML report (run on log directory)
sudo goaccess /var/log/nginx/access.log --log-format=COMBINED -o /tmp/nginx-report.html --date-format=%d/%b/%Y --time-format=%H:%M:%S 
 Compare pre/post dashboards for RPS, latency, error rate, active connections.

## Rollback Procedures (Per Strategy)

### Strategy 1: In-Place Minor/Major Upgrade (Package Downgrade + Config Restore)

Trigger : Service fails to start, unknown directive, module load failure, elevated 5xx > threshold. Time to Restore : 2–5 minutes.

# 1. Stop current service (graceful drain)
sudo nginx -s quit
# Wait for workers to exit (max worker_shutdown_timeout)
sleep 35

# 2. Downgrade package
# Debian/Ubuntu (replace with actual previous version from apt-cache policy)
sudo apt-get install -y nginx=<PREVIOUS_VERSION> nginx-module-brotli=<PREVIOUS_VERSION>

# RHEL-family
sudo dnf downgrade -y nginx-<PREVIOUS_VERSION> nginx-module-brotli-<PREVIOUS_VERSION>

# 3. Restore configuration from backup
sudo tar -C / -xzf /var/backups/nginx-etc-<BACKUP_DATE>.tgz

# 4. Validate and start
sudo nginx -t && sudo systemctl start nginx

# 5. Verify
curl -sI http://127.0.0.1/healthz 

### Strategy 2: In-Place Major Upgrade with Canary Port (Config Revert + Package Downgrade)

 Trigger : Canary vhost on port 8081 shows errors; main traffic unaffected. Time to Restore : 1–2 minutes (disable canary only) or 5 minutes (full downgrade).

# Fast: Disable canary vhost only
sudo rm /etc/nginx/sites-enabled/example-8081.conf
sudo nginx -t && sudo systemctl reload nginx

# Full rollback: Same as Strategy 1 

### Strategy 3: New-Host Migration (DNS/LB Revert + Old Host Standby)

 Trigger : Post-cutover 5xx > 0.5%, TLS failures, latency spike > 2× baseline. Time to Restore : Minutes (LB weight flip) to TTL expiry (DNS).

# Preferred: Load Balancer weight flip (instant)
# Set web-new weight=0, web-old weight=100 in LB console/API

# Fallback: DNS revert (subject to client TTL caching)
# Update A/AAAA record back to 
# Monitor old host logs for traffic return

# Keep old host running until new host validated offline
# On new host: stop nginx, investigate
sudo systemctl stop nginx 

### Graceful Drain vs. Fast Stop

- Scenario — Command — Behavior

- Planned maintenance / rollback — nginx -s quit / systemctl stop nginx (with KillSignal=SIGQUIT ) — Workers finish in-flight requests (up to worker_shutdown_timeout ), then exit. Zero connection drops if timeout sufficient.

- Emergency / hung workers — nginx -s stop / systemctl kill -s SIGTERM nginx — Immediate termination. In-flight requests dropped. Use only when graceful drain stalls.

## Realistic Technical Scenario (End-to-End Walkthrough)

 Objective : Upgrade Nginx from 1.22 (Ubuntu 22.04 OS repo) to 1.26 (nginx.org mainline) with HTTP/2, TLS 1.3, and third-party ngx_brotli . Migrate to new host with 60s TTL DNS cutover. Validate with 50 RPS synthetic load for 10 minutes. Rollback trigger: >0.5% 5xx.

Environment

- Old Host :  (Ubuntu 22.04, Nginx 1.22 from OS repo, ngx_brotli compiled from source)

- New Host : <NEW_HOST> (Ubuntu 22.04, fresh install)

- Domain : <YOUR_DOMAIN> (DNS TTL=60s, managed externally)

- Load : ~200 RPS production, mixed static + proxied API

### Phase 1: Inventory & Baseline (Old Host)

# Capture versions, modules, config
nginx -V 2>&1 | tee /var/backups/nginx-build-info.txt
sudo nginx -T > /var/backups/nginx-effective-pre.conf
sudo ss -ltnp | grep nginx > /var/backups/listeners-pre.txt
apt-cache policy nginx > /var/backups/pkg-policy-pre.txt

# Baseline performance (run 10 min)
./validate.sh # Custom script from Validation section, targeting old host
# Record: p95=45ms, 5xx=0.02%, RPS=200 

### Phase 2: New Host Preparation

 # On <NEW_HOST>: Install nginx.org mainline + brotli module
curl -fsSL https://nginx.org/keys/nginx_signing.key | sudo gpg --dearmor -o /usr/share/keyrings/nginx-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/mainline/ubuntu jammy nginx" | sudo tee /etc/apt/sources.list.d/nginx-mainline.list
cat <<'EOF' | sudo tee /etc/apt/preferences.d/nginx-pin
Package: nginx*
Pin: origin nginx.org
Pin-Priority: 900
EOF
sudo apt-get update
sudo apt-get install -y nginx nginx-module-brotli

# Verify modules
nginx -V 2>&1 | grep -E 'brotli|ssl|v2'

# Transfer config & TLS (run from )
sudo rsync -a --delete /etc/nginx/ <NEW_HOST>:/etc/nginx/
sudo rsync -a --delete /etc/letsencrypt/ <NEW_HOST>:/etc/letsencrypt/
sudo rsync -a --delete /var/www/ <NEW_HOST>:/var/www/

# Fix permissions on new host
sudo chown root:nginx /etc/letsencrypt/live/<YOUR_DOMAIN>/privkey.pem
sudo chmod 640 /etc/letsencrypt/live/<YOUR_DOMAIN>/privkey.pem
sudo restorecon -Rv /etc/nginx /etc/letsencrypt /var/www 2>/dev/null || true # No-op on Ubuntu/AppArmor 

### Phase 3: Pilot Validation on New Host (No Production Traffic)

 # On <NEW_HOST>: Enable pilot port 8081 for single vhost
sudo cp /etc/nginx/sites-available/app.conf /etc/nginx/sites-available/app-pilot.conf
sudo sed -i 's/listen 80;/listen 8081;/' /etc/nginx/sites-available/app-pilot.conf
sudo sed -i 's/listen 443 ssl;/listen 8443 ssl;/' /etc/nginx/sites-available/app-pilot.conf
sudo ln -s /etc/nginx/sites-available/app-pilot.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

# Verify pilot from old host (or admin workstation)
curl -sI --resolve <YOUR_DOMAIN>:8081:<NEW_HOST_IP> http://<YOUR_DOMAIN>:8081/healthz
curl -sIk --resolve <YOUR_DOMAIN>:8443:<NEW_HOST_IP> https://<YOUR_DOMAIN>:8443/healthz
# Check error.log: clean
sudo tail -n 20 /var/log/nginx/error.log 

### Phase 4: Full Validation on New Host (Production Ports, No DNS)

 # Enable main config on 80/443
sudo rm /etc/nginx/sites-enabled/app-pilot.conf
sudo nginx -t && sudo systemctl reload nginx

# End-to-end with --resolve (bypasses DNS)
curl -sI --resolve <YOUR_DOMAIN>:80:<NEW_HOST_IP> http://<YOUR_DOMAIN>/
curl -sIk --resolve <YOUR_DOMAIN>:443:<NEW_HOST_IP> https://<YOUR_DOMAIN>/

# TLS 1.3 verification
openssl s_client -connect <NEW_HOST_IP>:443 -servername <YOUR_DOMAIN> -tls1_3 -brief < /dev/null

# Synthetic load: 50 RPS for 10 minutes (600s)
./validate.sh # Modify DURATION=600, target <NEW_HOST_IP> with --resolve
# Acceptance: 5xx=0%, p95 ≤ 50ms (within 10% of 45ms baseline) 

### Phase 5: Cutover

 # Update DNS A record to <NEW_HOST_IP> (TTL=60s already set)
# Monitor TTL expiry: watch -n 5 'dig +short <YOUR_DOMAIN>'

# Or LB cutover: set <NEW_HOST> weight=100,  weight=0 

### Phase 6: Post-Cutover Monitoring (15 minutes)

 # On <NEW_HOST>
sudo tail -f /var/log/nginx/error.log &
# In another terminal, watch 5xx rate
watch -n 10 "awk '\$9 ~ /^5/ {c5++} {ctotal++} END {print \"5xx rate:\", ctotal?c5/ctotal*100:0 \"%\"}' /var/log/nginx/access.log"

# Synthetic validation continues
./validate.sh # DURATION=900 

### Phase 7: Rollback Decision Point

- If 5xx > 0.5% sustained for 2 minutes OR p95 > 90ms OR TLS errors in logs:

- Execute LB weight flip back to (instant)

- Or DNS revert to (wait for TTL)

- Investigate offline

- Else : Decommission after 24h observation.

### Phase 8: Documentation

 Record in runbook:

- Versions: 1.22 (OS) → 1.26.0-1~jammy (nginx.org mainline)

- Module: ngx_brotli 1.0.0 (nginx.org package)

- Timings: Pilot 15m, Validation 10m, Cutover 2m, Monitoring 15m

- Metrics: Pre p95=45ms/5xx=0.02% → Post p95=48ms/5xx=0.00%

- Commands used (exact, with versions)

- Issues: None / [list any]

## Quick Reference (One-Liners)

Inventory

nginx -V 2>&1; sudo nginx -t; sudo nginx -T > /var/tmp/nginx-effective.conf; sudo ss -ltnp; apt-cache policy nginx || dnf repoinfo nginx 
 Backup

sudo tar -C /etc -czf /var/backups/nginx-etc-$(date +%F).tgz nginx; sudo tar -C /var -czf /var/backups/nginx-web-$(date +%F).tgz www nginx ssl letsencrypt 2>/dev/null 
 Pilot (Port 8081)

sudo cp /etc/nginx/sites-available/app.conf /etc/nginx/sites-available/app-8081.conf; sudo sed -i 's/listen 80;/listen 8081;/' /etc/nginx/sites-available/app-8081.conf; sudo ln -sf /etc/nginx/sites-available/app-8081.conf /etc/nginx/sites-enabled/; sudo nginx -t && sudo systemctl reload nginx 
 Verify

curl -sI --resolve <DOMAIN>:80:<IP> http://<DOMAIN>/; curl -sIk --resolve <DOMAIN>:443:<IP> https://<DOMAIN>/; openssl s_client -connect <IP>:443 -servername <DOMAIN> -tls1_3 -brief < /dev/null 
 Rollback (In-Place)

sudo nginx -s quit; sleep 35; sudo apt-get install -y nginx=<PREV_VER>; sudo tar -C / -xzf /var/backups/nginx-etc-<DATE>.tgz; sudo nginx -t && sudo systemctl start nginx 
 Rollback (Migration)

# LB: set old host weight=100, new host weight=0
# DNS: revert A record to  

## Conclusion

 You can upgrade or migrate Nginx with confidence by keeping the first pilot small, validating changes with concrete checks, and preparing fast recovery steps. Start with a tight inventory that captures version, modules, repository source, TLS assets, and performance baselines. Choose the lowest-risk path that meets your objectives: in-place minor upgrade for patch-level changes, canary port for major version shifts with module dependencies, or new-host migration when uptime is non-negotiable. Prove each change in isolation using synthetic transactions, log analysis, and TLS verification before any traffic cutover. With backups in hand, a verified plan to downgrade packages or flip load balancer weights, and a short, objective checklist tied to measurable thresholds (5xx rate, p95 latency, error log cleanliness), you reduce risk, shorten maintenance windows, and make future upgrades routine rather than stressful.