Tuning an NGINX Ingress Controller for production traffic is not about finding a single magic setting. It is a disciplined process of measuring baseline behavior, applying targeted changes to connection handling, buffering, TLS, and traffic routing, then verifying the impact with real load. This guide walks through that process with concrete configuration examples, a safe rollout strategy, and rollback procedures you can run today. ## Architecture & Traffic Flow A request typically follows this path: Client → Cloud Load Balancer (TLS termination or passthrough) → NodePort or LoadBalancer Service → Ingress Controller Pod (NGINX master and worker processes) → ClusterIP Service → Backend Pods. The externalTrafficPolicy on the controller Service changes both hop count and source IP visibility. With Local , the load balancer sends traffic only to nodes that run a controller pod, preserving the client IP in X-Forwarded-For and eliminating a kube-proxy hop. With Cluster , traffic may route to any node, adding a kube-proxy hop and masking the client IP behind the node IP. Inside the pod, NGINX runs a master process and a configurable number of worker processes. Each worker handles up to worker_connections concurrent connections. Upstream keepalive pools ( upstream-keepalive ) are per-worker, so the total idle upstream connections equal worker-processes × upstream-keepalive . Understanding this multiplication is critical when sizing connection limits. ## Inventory & Baseline Before changing anything, capture your current environment. Run these commands to record versions, topology, and existing annotations: kubectl version --short kubectl -n ingress-nginx get deploy -o wide kubectl -n ingress-nginx describe deploy ingress-nginx-controller kubectl get nodes -o wide kubectl -n ingress-nginx get pods -o wide kubectl -n ingress-nginx get svc kubectl -n ingress-nginx describe svc ingress-nginx-controller kubectl get ingress --all-namespaces -o yaml | grep -E "nginx.ingress.kubernetes.io" Target Kubernetes 1.29 or newer (1.27 reached EOL June 2024) and NGINX Ingress Controller 1.11.x (Helm chart 4.11+). The version skew policy requires the controller minor version to be less than or equal to the Kubernetes minor version. ### Step 0: Echo Test App Deploy an isolated workload for safe, repeatable measurement. This echo server returns a static response and exposes a predictable latency profile. apiVersion: apps/v1 kind: Deployment metadata: name: echo namespace: default spec: replicas: 2 selector: matchLabels: app: echo template: metadata: labels: app: echo spec: containers: - name: echo image: hashicorp/http-echo:0.2.3 args: ["-text=ok"] ports: - containerPort: 5678 resources: requests: cpu: 50m memory: 64Mi limits: cpu: 200m memory: 128Mi --- apiVersion: v1 kind: Service metadata: name: echo namespace: default spec: selector: app: echo ports: - name: http port: 80 targetPort: 5678 --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: echo namespace: default annotations: nginx.ingress.kubernetes.io/proxy-read-timeout: "30" nginx.ingress.kubernetes.io/proxy-send-timeout: "30" nginx.ingress.kubernetes.io/keepalive: "64" nginx.ingress.kubernetes.io/limit-rps: "100" nginx.ingress.kubernetes.io/proxy-buffer-size: "16k" nginx.ingress.kubernetes.io/proxy-buffers: "8 16k" spec: ingressClassName: nginx rules: - host: echo.example.test http: paths: - path: / pathType: Prefix backend: service: name: echo port: number: 80 Apply and smoke-test: kubectl apply -f echo.yaml kubectl get endpoints echo -n default curl -H "Host: echo.example.test" http://LB_ADDRESS/ Record a 60-second baseline at steady load (for example, 100 RPS) capturing p50, p95, and p99 latency. This baseline is your reference for every subsequent change. ## Controller Configuration (ConfigMap) Update the nginx-configuration ConfigMap in the ingress-nginx namespace. Keys marked "Restart" require a pod restart; others reload live via the controller's reload mechanism. ### Recommended ConfigMap Values
KeyTypeDefaultRecommended RangeReload/RestartVersion
worker-processesstring"1""auto" or 2-4Restart1.5.0+
worker-connectionsstring"16384"16384-65536Restart1.5.0+
keep-alivestring"75"60-300Reload1.0+
keep-alive-requestsstring"1000"1000-10000Reload1.0+
upstream-keepalivestring"0" (disabled)32-256Reload1.0+
upstream-keepalive-requestsstring"1000"100-10000Reload1.3+
upstream-keepalive-timeoutstring"60s"30-300sReload1.3+
proxy-buffer-sizestring"4k/8k"8k-16kReload1.0+
proxy-buffers-numberstring"8"8-16Reload1.0+
proxy-buffersstring"8 4k/8k""8 16k"Reload1.0+
proxy-busy-buffers-sizestring"8k/16k"16k-32kReload1.0+
use-http2string"true""true"Reload1.0+
http2-max-concurrent-streamsstring"128"128-256Reload1.3+
http2-max-field-sizestring"4k"4k-16kReload1.3+
http2-max-header-sizestring"16k"16k-32kReload1.3+
proxy-request-bufferingstring"on""on"/"off" (per-Ingress for large uploads)Reload1.0+
proxy-read-timeoutstring"60"30-300Reload1.0+
proxy-send-timeoutstring"60"30-300Reload1.0+
proxy-next-upstreamstring"error timeout""error timeout http_500 http_502 http_503 http_504"Reload1.0+
proxy-intercept-errorsstring"off""on" for custom error pagesReload1.0+
ssl-session-cachestring"shared:SSL:10m""shared:SSL:10m-50m"Reload1.0+
ssl-session-timeoutstring"10m"10m-60mReload1.0+
ssl-session-ticketsstring"on""on"/"off" (FIPS/PCI may require off)Reload1.0+
ssl-prefer-server-ciphersstring"off""off" (TLS 1.3)Reload1.0+
ssl-ciphersstringMozilla IntermediateModern suite for TLS 1.2 compatReload1.0+
ssl-ecdh-curvestring"X25519:P-256""X25519:P-256"Reload1.0+
ssl-dh-paramstring""path to custom DH param fileReload1.0+
enable-ssl-chain-completionstring"false""true"Reload1.3+
ssl-staplingstring"off""on"Reload1.3+
ssl-stapling-verifystring"off""on"Reload1.3+
log-format-escape-jsonstring"false""true"Reload1.3+
error-log-levelstring"warn""warn"/"notice"Reload1.0+
limit-req-zonestring"""zone=global:10m rate=1000r/s"Reload1.0+
limit-conn-zonestring"""zone=conn:10m"Reload1.0+
hstsstring"false""true"Reload1.3+
hsts-max-agestring"2592000"31536000 (1 year)Reload1.3+
proxy-hide-headersstring"""Server,X-Powered-By"Reload1.0+
enable-underscores-in-headersstring"false""false"Reload1.0+
### Example ConfigMap Patch apiVersion: v1 kind: ConfigMap metadata: name: nginx-configuration namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx data: worker-processes: "auto" worker-connections: "32768" keep-alive: "100" keep-alive-requests: "10000" upstream-keepalive: "128" upstream-keepalive-requests: "5000" upstream-keepalive-timeout: "60s" proxy-buffer-size: "16k" proxy-buffers-number: "8" proxy-buffers: "8 16k" proxy-busy-buffers-size: "32k" use-http2: "true" http2-max-concurrent-streams: "256" http2-max-field-size: "16k" http2-max-header-size: "32k" proxy-request-buffering: "on" proxy-read-timeout: "60" proxy-send-timeout: "60" proxy-next-upstream: "error timeout http_500 http_502 http_503 http_504" proxy-intercept-errors: "off" ssl-session-cache: "shared:SSL:20m" ssl-session-timeout: "20m" ssl-session-tickets: "on" ssl-prefer-server-ciphers: "off" ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384" ssl-ecdh-curve: "X25519:P-256" enable-ssl-chain-completion: "true" ssl-stapling: "on" ssl-stapling-verify: "on" log-format-escape-json: "true" error-log-level: "warn" limit-req-zone: "zone=global:10m rate=2000r/s" limit-conn-zone: "zone=conn:10m" hsts: "true" hsts-max-age: "31536000" proxy-hide-headers: "Server,X-Powered-By" enable-underscores-in-headers: "false" Apply and verify the reload: kubectl -n ingress-nginx apply -f nginx-configmap.yaml kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | grep -i reload Security note: TLS 1.3 is preferred; the cipher list above covers TLS 1.2 fallback. Disable ssl-session-tickets if FIPS or PCI-DSS compliance requires it. Enable OCSP stapling ( ssl-stapling: "on" ) and HSTS for production. ## TLS Optimization TLS handshake CPU dominates at high connection rates. Session reuse via ssl-session-cache and ssl-session-tickets avoids full handshakes on subsequent connections. For TLS 1.2 clients, explicit ssl-ciphers and ssl-ecdh-curve ensure modern curves (X25519) and AEAD ciphers. Use ECDSA certificates where possible (smaller, faster verification). Cert-manager with a dns01 solver automates wildcard certificate rotation and keeps secrets current. ## Service Traffic Policy externalTrafficPolicy: Local preserves client IP and removes one kube-proxy hop. It requires a controller pod on every node that receives load balancer traffic. Use Pod anti-affinity and topologySpreadConstraints to guarantee distribution across nodes. externalTrafficPolicy: Cluster balances evenly but masks the client IP behind the node IP. Patch the Service: apiVersion: v1 kind: Service metadata: name: ingress-nginx-controller namespace: ingress-nginx spec: externalTrafficPolicy: Local Apply and verify: kubectl -n ingress-nginx apply -f svc-patch.yaml curl -H "Host: echo.example.test" http://LB_ADDRESS/ -v 2>&1 | grep -i "x-forwarded-for" kubectl -n ingress-nginx get endpoints ingress-nginx-controller -o wide Warning: If endpoints are missing on some nodes, traffic will not reach those nodes. Fix scheduling or revert to Cluster . ## Per-Route Annotations Annotations let you tune behavior for specific Ingress resources without affecting the entire controller. Apply these to high-traffic Ingress objects first.
AnnotationScopeDefaultExampleUse CaseVersion
nginx.ingress.kubernetes.io/limit-rpsIngressunlimited"1000"Per-IP rate limit1.0+
nginx.ingress.kubernetes.io/limit-burstIngress0"2000"Burst allowance1.0+
nginx.ingress.kubernetes.io/limit-connectionsIngressunlimited"100"Per-IP concurrent connections1.3+
nginx.ingress.kubernetes.io/keepaliveIngress0 (disabled)"100"Upstream keepalive pool1.0+
nginx.ingress.kubernetes.io/proxy-body-sizeIngress"1m""20m"Max upload size1.0+
nginx.ingress.kubernetes.io/proxy-buffer-sizeIngressglobal"16k"Large header handling1.0+
nginx.ingress.kubernetes.io/proxy-buffersIngressglobal"8 16k"Response buffering1.0+
nginx.ingress.kubernetes.io/proxy-read-timeoutIngress"60""60"Slow backend tolerance1.0+
nginx.ingress.kubernetes.io/proxy-send-timeoutIngress"60""60"Slow client tolerance1.0+
nginx.ingress.kubernetes.io/backend-protocolIngress"HTTP""GRPC"gRPC upstream1.0+
nginx.ingress.kubernetes.io/grpc-pass-throughIngress"false""true"TLS passthrough for gRPC1.3+
nginx.ingress.kubernetes.io/proxy-request-bufferingIngress"on""off"Large uploads streaming1.0+
nginx.ingress.kubernetes.io/canaryIngress"false""true"Canary rollout1.0+
nginx.ingress.kubernetes.io/canary-weightIngress"0""10"Canary traffic %1.0+
nginx.ingress.kubernetes.io/mirror-targetIngress"""mirror-svc"Traffic shadowing1.3+
nginx.ingress.kubernetes.io/rewrite-targetIngress"""/"Path rewriting1.0+
nginx.ingress.kubernetes.io/enable-corsIngress"false""true"CORS headers1.0+
nginx.ingress.kubernetes.io/auth-urlIngress"""https://auth.example.com"External auth1.0+
nginx.ingress.kubernetes.io/limit-req-zoneIngressglobal"zone=api:10m rate=500r/s"Custom rate limit zone1.3+
Example for a high-traffic API gateway: apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: api-gateway namespace: production annotations: nginx.ingress.kubernetes.io/limit-rps: "1000" nginx.ingress.kubernetes.io/limit-burst: "2000" nginx.ingress.kubernetes.io/limit-connections: "200" nginx.ingress.kubernetes.io/keepalive: "100" nginx.ingress.kubernetes.io/proxy-body-size: "20m" nginx.ingress.kubernetes.io/proxy-buffer-size: "16k" nginx.ingress.kubernetes.io/proxy-buffers: "8 16k" nginx.ingress.kubernetes.io/proxy-read-timeout: "60" nginx.ingress.kubernetes.io/proxy-send-timeout: "60" nginx.ingress.kubernetes.io/backend-protocol: "HTTP" spec: ingressClassName: nginx rules: - host: api.example.com http: paths: - path: / pathType: Prefix backend: service: name: api-gateway port: number: 80 ## Safer Rollouts (PDB, HPA, PreStop) ### PodDisruptionBudget A PDB with maxUnavailable: 1 allows one pod down during voluntary disruptions (node upgrades, deployments) while maintaining HA. apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: ingress-nginx-pdb namespace: ingress-nginx spec: maxUnavailable: 1 selector: matchLabels: app.kubernetes.io/name: ingress-nginx ### Graceful Termination The default image lacks a /wait-shutdown binary. Use a preStop hook that quits NGINX gracefully and sleeps to allow in-flight requests to drain. apiVersion: apps/v1 kind: Deployment metadata: name: ingress-nginx-controller namespace: ingress-nginx spec: template: spec: terminationGracePeriodSeconds: 60 containers: - name: controller lifecycle: preStop: exec: command: ["/bin/sh","-c","nginx -s quit && sleep 30"] ### HorizontalPodAutoscaler with Custom Metrics Scale on both CPU and request rate (requires Prometheus Adapter). apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: ingress-nginx-hpa namespace: ingress-nginx spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: ingress-nginx-controller minReplicas: 3 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 60 - type: Pods pods: metric: name: nginx_ingress_controller_requests target: type: AverageValue averageValue: "1000" behavior: scaleDown: stabilizationWindowSeconds: 300 policies: - type: Percent value: 10 periodSeconds: 60 scaleUp: stabilizationWindowSeconds: 0 policies: - type: Percent value: 100 periodSeconds: 15 - type: Pods value: 4 periodSeconds: 15 selectPolicy: Max ## Verification & Load Testing ### curl Timing Breakdown curl -H "Host: echo.example.test" -w "namelookup:%{time_namelookup} connect:%{time_connect} appconnect:%{time_appconnect} starttransfer:%{time_starttransfer} total:%{time_total}\n" -o /dev/null -s http://LB_ADDRESS/ Expected: appconnect near zero for HTTP; for HTTPS, appconnect drops on subsequent requests with session reuse. starttransfer and total should be stable across runs. ### In-Cluster Load Generation Use a pinned hey image or k6 for repeatable tests. kubectl run -it loadgen --image=docker.io/rakyll/hey@sha256:2a5b3c4d5e6f7g8h9i0j --restart=Never --rm -- -z 60s -c 100 -q 100 -H "Host: echo.example.test" http://ingress-nginx-controller.ingress-nginx.svc.cluster.local/ k6 script for CI: import http from 'k6/http'; import { check, sleep } from 'k6'; export const options = { stages: [ { duration: '30s', target: 50 }, { duration: '2m', target: 200 }, { duration: '30s', target: 500 }, { duration: '30s', target: 0 } ], thresholds: { http_req_duration: ['p(95)<500', 'p(99)<1000'], http_req_failed: ['rate<0.01'] } }; export default function () { const res = http.get('http://ingress-nginx-controller.ingress-nginx.svc.cluster.local/', { headers: { Host: 'echo.example.test' } }); check(res, { 'status is 200': (r) => r.status === 200 }); sleep(0.1); } ### Controller Metrics and Logs Metrics endpoint in v1.11+: /metrics on port 10254 (Prometheus format). Configure a ServiceMonitor for Prometheus. kubectl -n ingress-nginx exec -it deploy/ingress-nginx-controller -- curl -s localhost:10254/metrics | grep -E "nginx_ingress_controller_requests|nginx_ingress_controller_request_duration_seconds|nginx_ingress_controller_upstream_latency|nginx_ingress_controller_connections_active|nginx_ingress_controller_ssl_handshakes_total" Key metrics: - nginx_ingress_controller_requests_total (counter by code) - nginx_ingress_controller_request_duration_seconds_bucket (histogram) - nginx_ingress_controller_upstream_latency_seconds_bucket (histogram) - nginx_ingress_controller_connections_active (gauge) - nginx_ingress_controller_connections_waiting (gauge) - nginx_ingress_controller_ssl_handshakes_total (counter) Grafana dashboard: import the community dashboard from kubernetes-mixin/ingress-nginx (JSON available in their repository). JSON log format for structured parsing: kubectl -n ingress-nginx logs deploy/ingress-nginx-controller --tail=100 | jq . ### Endpoint Distribution Check kubectl -n ingress-nginx get endpoints ingress-nginx-controller -o wide Confirm ready addresses on all nodes receiving LB traffic when using Local . ## Failure Modes & Rollback ### Playbook 1: 502/504 Spike - Check upstream readiness: kubectl get pods -n -l app= - Check proxy-read-timeout on Ingress; increase if backend legitimately slow. - Check NGINX error logs: kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | grep "upstream timed out" - Fix: raise timeout, scale backend, or tune backend performance. - Rollback: revert timeout annotation. ### Playbook 2: High Latency p99 - Check nginx_ingress_controller_connections_active vs worker-connections × worker-processes . - Check TLS handshake rate: nginx_ingress_controller_ssl_handshakes_total increasing rapidly. - Check buffer flushing: large responses with small proxy-buffers . - Check DNS resolution: resolver in ConfigMap if using upstream domain names. - Fix: increase worker-connections , enable TLS session cache, tune buffers, add resolver. - Rollback: revert ConfigMap. ### Playbook 3: Client IP Missing - Verify externalTrafficPolicy: Local on Service. - Verify use-forwarded-headers: "true" and compute-full-forwarded-for: "true" in ConfigMap. - If L4 LB with PROXY protocol: enable use-proxy-protocol: "true" in ConfigMap and annotate Service with service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" (AWS NLB) or equivalent. - Fix: adjust Service policy and ConfigMap. - Rollback: revert to Cluster policy. ### Playbook 4: Certificate Errors - Check cert-manager Certificate status: kubectl get certificate -n - Verify secret contains tls.crt , tls.key , ca.crt (for chain completion). - Check SNI matching: kubectl -n ingress-nginx exec deploy/ingress-nginx-controller -- nginx -T | grep server_name - Fix: renew cert, enable enable-ssl-chain-completion: "true" , verify secret format. - Rollback: revert to previous valid secret. ### Quick Rollback Commands # ConfigMap revert kubectl -n ingress-nginx apply -f nginx-configmap-prev.yaml # Deployment rollback kubectl -n ingress-nginx rollout undo deploy/ingress-nginx-controller # Annotation removal kubectl annotate ingress myapp nginx.ingress.kubernetes.io/limit-rps- --overwrite # Service policy revert kubectl -n ingress-nginx patch svc ingress-nginx-controller -p '{"spec":{"externalTrafficPolicy":"Cluster"}}' # Scale up for capacity kubectl -n ingress-nginx scale deploy ingress-nginx-controller --replicas=10 ## Operations Checklist - Define a narrow pilot with one Ingress and the echo test route. - Inventory versions: Kubernetes 1.29+, Controller 1.11+, Helm chart 4.11+, Cert-manager 1.14+. - Record baseline: RPS and p50/p95/p99 for 60s at steady load. - Right-size controller replicas (min 3) and resources (requests 500m/512Mi, limits 2/1Gi); confirm HPA and PDB. - Apply controller ConfigMap changes: worker-processes , worker-connections , keep-alive , upstream-keepalive , buffers, timeouts, TLS, logging, rate limiting. - Verify curl timing and controller metrics after reload. - Tune TLS: enable session cache, tickets (if allowed), OCSP stapling, HSTS; verify appconnect improvements on HTTPS. - Choose Service externalTrafficPolicy based on client IP requirements; confirm endpoint distribution. - Add per-Ingress annotations for buffering, body size, keepalive, rate limits, canary as needed. - Re-test and compare against baseline; document deltas. - If regressions appear: roll back ConfigMap or deployment; scale up temporarily. - Schedule quarterly review: retest assumptions, versions, and tunables. ## Conclusion Ingress performance is not one magic flag; it is a small set of focused, measurable changes that compound. By inventorying your environment, establishing a baseline, and applying safe, reversible tuning to connections, buffers, TLS, and traffic policy, you can improve latency and throughput without risking outages. Start with a narrow pilot route, verify each change with simple load and curl timing, and keep rollback steps ready. With the checklist in hand, you can repeat the process for each high-traffic Ingress and maintain performance as your traffic and services evolve.