Nginx Capacity Planning with Practical Examples

Intro

Capacity planning for Nginx is about translating a performance goal into safe, observable configuration and infrastructure changes. This guide shows how to inventory your environment, size key knobs (workers, connections, file descriptors, timeouts), verify with load tests, and recover from common failure modes—using small, reversible steps and concrete examples.

Who this is for: developers, SREs, and DevOps teams running Nginx as a reverse proxy, TLS terminator, or static content server on Linux, containers, or Kubernetes. Placeholders like <HOST>, <ENV>, and <PATH> are intentional; never paste secrets, tokens, or private identifiers.

Version and Environment Inventory

Start by observing, not changing. Record what is running, where, and how it is built.

• Supported scope: open source Nginx 1.14+ (most commands also apply to newer LTS and mainline versions).
• Topologies covered: edge TLS termination, reverse proxy to upstream app servers, static asset serving, and container/Kubernetes deployments when relevant to limits and recovery.

1. Identify the exact build and modules

# Capture Nginx version, compiler flags, and linked modules
TZ=UTC date -Is
nginx -V 2>&1 | tee /tmp/nginx-version-<ENV>.txt

# Validate configuration (read-only)
nginx -t

1. Map the deployment

• Host OS/distro and kernel: uname -a, /etc/os-release
• Process manager: systemctl status nginx or ps -eo pid,cmd | grep nginx
• Listener ports: ss -tanp | grep -E ':(80|443)\s'
• TLS offload: Verify cert/key paths exist and ownership is correct (do not copy the contents into notes).
• Upstream topology: how many upstream servers per service, connection reuse (keepalive) status, and average response sizes.

1. Baseline resource limits (read-only)

# Process-level file descriptor limit
sudo -u <nginx_user> sh -c 'ulimit -n; echo $PPID'  # or check via systemd LimitNOFILE

# System-wide FD ceiling and sockets summary
cat /proc/sys/fs/file-max
ss -s

# Per-worker threads and RSS (sanity)
ps -o pid,psr,pcpu,pmem,rss,cmd -C nginx

1. Define the goal and failure signal before any change

• Example goal: peak 2,000 RPS at p95 < 150 ms on TLS, reverse proxy to 4 upstreams.
• Primary signals: 5xx rate, 499 rate, Nginx accept errors, queue/backlog overflow, CPU > 85% sustained, memory pressure, saturation in stub_status.

Safe Configuration Path

Plan the smallest justified change. Keep a rollback ready. Test in staging first.

Key sizing concepts

• worker_processes: usually equals CPU cores. Use auto unless you have a reason not to.
• worker_connections: upper bound for concurrent connections per worker (includes client and upstream connections). For reverse proxying, a single request typically consumes one client + one upstream connection.
• Effective proxied concurrency ≈ (worker_processes × worker_connections) / 2 (rule of thumb; buffers, keepalive, and idle sockets affect this).
• File descriptors (FDs) must cover all concurrent sockets. Budget 2× concurrent proxied connections plus 10–20% headroom for logs, files, and control sockets.

Minimal-risk example change

Prerequisites

• Nginx 1.14+ on Linux, managed by systemd.
• Goal: support ~16k concurrent proxied connections on a 4 vCPU host with TLS.
• Blast radius: a single Nginx instance on <HOST>. Recovery: immediate config rollback and reload.

Proposed patch (excerpts)

# /etc/nginx/nginx.conf (snippets)
user  nginx;
worker_processes auto;            # 4 on a 4 vCPU host
worker_rlimit_nofile 200000;      # allow high FD usage inside workers

events {
    worker_connections 8192;      # 4 workers * 8192 ~= 32768 total connections
    multi_accept on;
    # consider reuseport only if tested per kernel/stack
    # use   listen 443 reuseport;  in server blocks if adopted
}

http {
    # Keepalive trade-off: better CPU efficiency but consumes FDs
    keepalive_timeout 15s;
    keepalive_requests 1000;

    # Reverse proxy tuning (example)
    upstream app_backend {
        server 10.0.0.11:8080;
        server 10.0.0.12:8080;
        keepalive 256;            # upstream connection reuse pool
    }

    server {
        listen 443 ssl http2;     # HTTP/2 improves multiplexing, adds CPU cost
        server_name <HOSTNAME>;
        ssl_certificate     /etc/ssl/certs/<cert>.pem;
        ssl_certificate_key /etc/ssl/private/<key>.pem;

        location /api/ {
            proxy_pass http://app_backend;
            proxy_http_version 1.1;
            proxy_set_header Connection "";  # allow upstream keepalive
            proxy_read_timeout 30s;          # ensure matches app SLO
            proxy_send_timeout 30s;
            proxy_connect_timeout 5s;
        }
    }
}

OS limits to match

# System-wide FD ceiling and per-service limit (systemd example)
# /etc/systemd/system/nginx.service.d/limits.conf
[Service]
LimitNOFILE=200000

# Reload systemd and restart gracefully during a maintenance window
sudo systemctl daemon-reload
sudo systemctl restart nginx

# Kernel queues and backlog
sudo sysctl -w net.core.somaxconn=8192
sudo sysctl -w net.ipv4.ip_local_port_range="20000 65000"  # for high outbound fan-out

Verification

nginx -t
sudo nginx -s reload
ss -tan 'sport = :443' | wc -l
journalctl -u nginx -n 100 --no-pager | grep -i -E 'error|crit|alert'

Rollback

• Keep a copy of the prior nginx.conf (e.g., /etc/nginx/nginx.conf.bak-<ts>).
• To rollback: restore the file, nginx -t, then sudo nginx -s reload.

Verification and Diagnostics

Observe first, then pressure-test.

Read-only health checks

# Config dump for audit
nginx -T > /tmp/nginx-conf-<ENV>.txt

# Lightweight status (stub_status)
# Add in a protected block (only on internal/admin network)
# location /nginx_status { stub_status; allow 127.0.0.1; deny all; }

curl -s http://127.0.0.1/nginx_status
# Expect Active connections, accepts/handled/requests, Reading/Writing/Waiting

Load test procedure (example)

Prerequisites: staging environment that mirrors prod limits, a client host near the server, and a realistic dataset.

• Tool suggestion: wrk, vegeta, hey, or k6. Example with wrk:

# 2k RPS for 5 minutes with 200 connections, HTTP/1.1 keepalive
wrk -t8 -c200 -d300s --rate 2000 https://<HOST>/api/test

• Success signal: target RPS with stable latency and error rate < 0.5%.
• Collect during test: top, mpstat -P ALL 5, iostat -xz 5, ss -s, curl /nginx_status.

What to look for

• 502/504 rising: upstream saturation or timeouts.
• 499 (client closed request): client impatience, slow upstream, or load balancer health checks timing out.
• Accept queue overflows: netstat -s or dmesg may show drops; increase somaxconn and validate listen backlog if needed.
• TLS CPU saturation: openssl speed baseline vs server CPU; consider enabling session resumption and ECDHE curves with hardware support.

Simple sizing math example

• 4 workers × 8192 worker_connections = 32,768 total connections.
• Reverse proxy typically uses 2 sockets per in-flight request; effective proxied concurrency ceiling ≈ 16,384.
• Ensure FDs > 2 × 16,384 × 1.2 headroom ≈ 39,322; using 200,000 is sufficient for growth.
• With p95 upstream time 100 ms, a single worker can often sustain thousands of RPS; confirm with tests.

Failure Modes and Recovery

Common failure signals, likely causes, and safe responses.

1. 502 Bad Gateway

• Causes: upstream process crashed, DNS failure, proxy_connect_timeout too low, exhausted ephemeral ports when proxying to many hosts.
• Actions: confirm upstream health (curl http://<upstream>:<port>/health), raise proxy_connect_timeout moderately (e.g., 1s -> 3s), ensure ip_local_port_range wide enough, and scale upstreams.
• Recovery: revert timeout if latency SLO degrades after change.

1. 504 Gateway Timeout

• Causes: proxy_read_timeout too small for slow endpoints, upstream saturation, or network issues.
• Actions: profile slow endpoints, right-size proxy_read_timeout for specific locations (do not globally raise without need), add caching or queueing upstream.

1. 499 Client Closed Request

• Causes: client impatience, load balancer idle timeout shorter than Nginx/backend, or slow TLS handshake under CPU pressure.
• Actions: align timeouts (LB > Nginx > upstream), reduce payload size, enable keepalive, ensure TLS params are efficient.

1. Accept queue/backlog saturation

• Symptoms: increased SYN backlog drops, rising latency under burst.
• Actions: raise net.core.somaxconn, validate listen ... backlog=<n> if explicitly set, consider reuseport after benchmarking.

1. Memory pressure / worker exits

• Causes: insufficient RAM for buffers, large in-flight responses, or logs.
• Actions: reduce proxy_buffering for large streaming endpoints, set client_max_body_size thoughtfully, tune proxy_buffers and proxy_busy_buffers_size, ensure swap policy is appropriate.

Containers and Kubernetes notes

• Docker: set --ulimit nofile=200000:200000 or via compose, ensure host sysctl matches.
• Kubernetes: specify containerSecurityContext allowPrivilegeEscalation: false and readOnlyRootFilesystem: true as applicable; tune FDs via hostConfig or init scripts; Nginx Ingress Controller has its own knobs—only adjust those that map to your goal (e.g., max-worker-connections).

Operations Checklist

Use this as a repeatable runbook for a single host or pod.

• Define the SLO and peak target (RPS, tail latency, error budget).
• Inventory
• nginx -V, nginx -t, nginx -T (to a timestamped file)
• OS/kernel, CPU cores, memory, process manager
• Current FDs: ulimit -n, cat /proc/sys/fs/file-max
• Observe baseline under load
• Enable and secure stub_status
• Record CPU, memory, socket stats, 4xx/5xx rates, and 499s
• Plan minimal change
• worker_processes auto, worker_connections sized to goal
• Set worker_rlimit_nofile and systemd LimitNOFILE
• Right-size keepalive and proxy timeouts for your app
• OS: somaxconn, ip_local_port_range
• Verify safely
• nginx -t; sudo nginx -s reload
• Load test with realistic RPS and payloads
• Compare metrics to expected signals
• Decide next step
• If CPU-bound: scale out (more pods/instances), tune TLS, enable caching
• If FD/connection-bound: raise limits cautiously or offload idle connections (shorter keepalive)
• If upstream-bound: add replicas, reduce per-connection concurrency, cache, or shard
• Document
• Final config, limits, and recovery path stored in version control

Conclusion

Nginx capacity planning works best when it is observable, incremental, and reversible. Start with an inventory, set a clear performance target, change one knob at a time with a known blast radius, and verify each step with load tests and status signals. Align Nginx limits with OS and container limits, size for both client and upstream sockets, and keep a ready rollback. This disciplined loop—observe, adjust, verify, recover—builds operational safety without surprises.