NGINX Ingress Controller Performance Tuning: A Production Guide

Tuning an NGINX Ingress Controller for production traffic is not about finding a single magic setting. It is a disciplined process of measuring baseline behavior, applying targeted changes to connection handling, buffering, TLS, and traffic routing, then verifying the impact with real load. This guide walks through that process with concrete configuration examples, a safe rollout strategy, and rollback procedures you can run today.

Architecture & Traffic Flow

A request typically follows this path: Client → Cloud Load Balancer (TLS termination or passthrough) → NodePort or LoadBalancer Service → Ingress Controller Pod (NGINX master and worker processes) → ClusterIP Service → Backend Pods.

The externalTrafficPolicy on the controller Service changes both hop count and source IP visibility. With Local, the load balancer sends traffic only to nodes that run a controller pod, preserving the client IP in X-Forwarded-For and eliminating a kube-proxy hop. With Cluster, traffic may route to any node, adding a kube-proxy hop and masking the client IP behind the node IP.

Inside the pod, NGINX runs a master process and a configurable number of worker processes. Each worker handles up to worker_connections concurrent connections. Upstream keepalive pools (upstream-keepalive) are per-worker, so the total idle upstream connections equal worker-processes × upstream-keepalive. Understanding this multiplication is critical when sizing connection limits.

Inventory & Baseline

Before changing anything, capture your current environment. Run these commands to record versions, topology, and existing annotations:

kubectl version --short
kubectl -n ingress-nginx get deploy -o wide
kubectl -n ingress-nginx describe deploy ingress-nginx-controller
kubectl get nodes -o wide
kubectl -n ingress-nginx get pods -o wide
kubectl -n ingress-nginx get svc
kubectl -n ingress-nginx describe svc ingress-nginx-controller
kubectl get ingress --all-namespaces -o yaml | grep -E "nginx.ingress.kubernetes.io"

Target Kubernetes 1.29 or newer (1.27 reached EOL June 2024) and NGINX Ingress Controller 1.11.x (Helm chart 4.11+). The version skew policy requires the controller minor version to be less than or equal to the Kubernetes minor version.

Step 0: Echo Test App

Deploy an isolated workload for safe, repeatable measurement. This echo server returns a static response and exposes a predictable latency profile.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
      - name: echo
        image: hashicorp/http-echo:0.2.3
        args: ["-text=ok"]
        ports:
        - containerPort: 5678
        resources:
          requests:
            cpu: 50m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi
---
apiVersion: v1
kind: Service
metadata:
  name: echo
  namespace: default
spec:
  selector:
    app: echo
  ports:
  - name: http
    port: 80
    targetPort: 5678
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/proxy-read-timeout: "30"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "30"
    nginx.ingress.kubernetes.io/keepalive: "64"
    nginx.ingress.kubernetes.io/limit-rps: "100"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
    nginx.ingress.kubernetes.io/proxy-buffers: "8 16k"
spec:
  ingressClassName: nginx
  rules:
  - host: echo.example.test
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: echo
            port:
              number: 80

Apply and smoke-test:

kubectl apply -f echo.yaml
kubectl get endpoints echo -n default
curl -H "Host: echo.example.test" http://LB_ADDRESS/

Record a 60-second baseline at steady load (for example, 100 RPS) capturing p50, p95, and p99 latency. This baseline is your reference for every subsequent change.

Controller Configuration (ConfigMap)

Update the nginx-configuration ConfigMap in the ingress-nginx namespace. Keys marked "Restart" require a pod restart; others reload live via the controller's reload mechanism.

Recommended ConfigMap Values

Example ConfigMap Patch

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  worker-processes: "auto"
  worker-connections: "32768"
  keep-alive: "100"
  keep-alive-requests: "10000"
  upstream-keepalive: "128"
  upstream-keepalive-requests: "5000"
  upstream-keepalive-timeout: "60s"
  proxy-buffer-size: "16k"
  proxy-buffers-number: "8"
  proxy-buffers: "8 16k"
  proxy-busy-buffers-size: "32k"
  use-http2: "true"
  http2-max-concurrent-streams: "256"
  http2-max-field-size: "16k"
  http2-max-header-size: "32k"
  proxy-request-buffering: "on"
  proxy-read-timeout: "60"
  proxy-send-timeout: "60"
  proxy-next-upstream: "error timeout http_500 http_502 http_503 http_504"
  proxy-intercept-errors: "off"
  ssl-session-cache: "shared:SSL:20m"
  ssl-session-timeout: "20m"
  ssl-session-tickets: "on"
  ssl-prefer-server-ciphers: "off"
  ssl-ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"
  ssl-ecdh-curve: "X25519:P-256"
  enable-ssl-chain-completion: "true"
  ssl-stapling: "on"
  ssl-stapling-verify: "on"
  log-format-escape-json: "true"
  error-log-level: "warn"
  limit-req-zone: "zone=global:10m rate=2000r/s"
  limit-conn-zone: "zone=conn:10m"
  hsts: "true"
  hsts-max-age: "31536000"
  proxy-hide-headers: "Server,X-Powered-By"
  enable-underscores-in-headers: "false"

Apply and verify the reload:

kubectl -n ingress-nginx apply -f nginx-configmap.yaml
kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | grep -i reload

Security note: TLS 1.3 is preferred; the cipher list above covers TLS 1.2 fallback. Disable ssl-session-tickets if FIPS or PCI-DSS compliance requires it. Enable OCSP stapling (ssl-stapling: "on") and HSTS for production.

TLS Optimization

TLS handshake CPU dominates at high connection rates. Session reuse via ssl-session-cache and ssl-session-tickets avoids full handshakes on subsequent connections. For TLS 1.2 clients, explicit ssl-ciphers and ssl-ecdh-curve ensure modern curves (X25519) and AEAD ciphers. Use ECDSA certificates where possible (smaller, faster verification). Cert-manager with a dns01 solver automates wildcard certificate rotation and keeps secrets current.

Service Traffic Policy

externalTrafficPolicy: Local preserves client IP and removes one kube-proxy hop. It requires a controller pod on every node that receives load balancer traffic. Use Pod anti-affinity and topologySpreadConstraints to guarantee distribution across nodes. externalTrafficPolicy: Cluster balances evenly but masks the client IP behind the node IP.

Patch the Service:

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  externalTrafficPolicy: Local

Apply and verify:

kubectl -n ingress-nginx apply -f svc-patch.yaml
curl -H "Host: echo.example.test" http://LB_ADDRESS/ -v 2>&1 | grep -i "x-forwarded-for"
kubectl -n ingress-nginx get endpoints ingress-nginx-controller -o wide

Warning: If endpoints are missing on some nodes, traffic will not reach those nodes. Fix scheduling or revert to Cluster.

Per-Route Annotations

Annotations let you tune behavior for specific Ingress resources without affecting the entire controller. Apply these to high-traffic Ingress objects first.

Example for a high-traffic API gateway:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api-gateway
  namespace: production
  annotations:
    nginx.ingress.kubernetes.io/limit-rps: "1000"
    nginx.ingress.kubernetes.io/limit-burst: "2000"
    nginx.ingress.kubernetes.io/limit-connections: "200"
    nginx.ingress.kubernetes.io/keepalive: "100"
    nginx.ingress.kubernetes.io/proxy-body-size: "20m"
    nginx.ingress.kubernetes.io/proxy-buffer-size: "16k"
    nginx.ingress.kubernetes.io/proxy-buffers: "8 16k"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "60"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "60"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: api-gateway
            port:
              number: 80

Safer Rollouts (PDB, HPA, PreStop)

PodDisruptionBudget

A PDB with maxUnavailable: 1 allows one pod down during voluntary disruptions (node upgrades, deployments) while maintaining HA.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ingress-nginx-pdb
  namespace: ingress-nginx
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx

Graceful Termination

The default image lacks a /wait-shutdown binary. Use a preStop hook that quits NGINX gracefully and sleeps to allow in-flight requests to drain.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: controller
        lifecycle:
          preStop:
            exec:
              command: ["/bin/sh","-c","nginx -s quit && sleep 30"]

HorizontalPodAutoscaler with Custom Metrics

Scale on both CPU and request rate (requires Prometheus Adapter).

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ingress-nginx-hpa
  namespace: ingress-nginx
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ingress-nginx-controller
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Pods
    pods:
      metric:
        name: nginx_ingress_controller_requests
      target:
        type: AverageValue
        averageValue: "1000"
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Percent
        value: 100
        periodSeconds: 15
      - type: Pods
        value: 4
        periodSeconds: 15
      selectPolicy: Max

Verification & Load Testing

curl Timing Breakdown

curl -H "Host: echo.example.test" -w "namelookup:%{time_namelookup} connect:%{time_connect} appconnect:%{time_appconnect} starttransfer:%{time_starttransfer} total:%{time_total}\n" -o /dev/null -s http://LB_ADDRESS/

Expected: appconnect near zero for HTTP; for HTTPS, appconnect drops on subsequent requests with session reuse. starttransfer and total should be stable across runs.

In-Cluster Load Generation

Use a pinned hey image or k6 for repeatable tests.

kubectl run -it loadgen --image=docker.io/rakyll/hey@sha256:2a5b3c4d5e6f7g8h9i0j --restart=Never --rm -- -z 60s -c 100 -q 100 -H "Host: echo.example.test" http://ingress-nginx-controller.ingress-nginx.svc.cluster.local/

k6 script for CI:

import http from 'k6/http';
import { check, sleep } from 'k6';
export const options = {
  stages: [
    { duration: '30s', target: 50 },
    { duration: '2m', target: 200 },
    { duration: '30s', target: 500 },
    { duration: '30s', target: 0 }
  ],
  thresholds: { http_req_duration: ['p(95)<500', 'p(99)<1000'], http_req_failed: ['rate<0.01'] }
};
export default function () {
  const res = http.get('http://ingress-nginx-controller.ingress-nginx.svc.cluster.local/', { headers: { Host: 'echo.example.test' } });
  check(res, { 'status is 200': (r) => r.status === 200 });
  sleep(0.1);
}

Controller Metrics and Logs

Metrics endpoint in v1.11+: /metrics on port 10254 (Prometheus format). Configure a ServiceMonitor for Prometheus.

kubectl -n ingress-nginx exec -it deploy/ingress-nginx-controller -- curl -s localhost:10254/metrics | grep -E "nginx_ingress_controller_requests|nginx_ingress_controller_request_duration_seconds|nginx_ingress_controller_upstream_latency|nginx_ingress_controller_connections_active|nginx_ingress_controller_ssl_handshakes_total"

Key metrics:

• nginx_ingress_controller_requests_total (counter by code)
• nginx_ingress_controller_request_duration_seconds_bucket (histogram)
• nginx_ingress_controller_upstream_latency_seconds_bucket (histogram)
• nginx_ingress_controller_connections_active (gauge)
• nginx_ingress_controller_connections_waiting (gauge)
• nginx_ingress_controller_ssl_handshakes_total (counter)

Grafana dashboard: import the community dashboard from kubernetes-mixin/ingress-nginx (JSON available in their repository).

JSON log format for structured parsing:

kubectl -n ingress-nginx logs deploy/ingress-nginx-controller --tail=100 | jq .

Endpoint Distribution Check

kubectl -n ingress-nginx get endpoints ingress-nginx-controller -o wide

Confirm ready addresses on all nodes receiving LB traffic when using Local.

Failure Modes & Rollback

Playbook 1: 502/504 Spike

• Check upstream readiness: kubectl get pods -n <backend-ns> -l app=<backend>
• Check proxy-read-timeout on Ingress; increase if backend legitimately slow.
• Check NGINX error logs: kubectl -n ingress-nginx logs deploy/ingress-nginx-controller | grep "upstream timed out"
• Fix: raise timeout, scale backend, or tune backend performance.
• Rollback: revert timeout annotation.

Playbook 2: High Latency p99

• Check nginx_ingress_controller_connections_active vs worker-connections × worker-processes.
• Check TLS handshake rate: nginx_ingress_controller_ssl_handshakes_total increasing rapidly.
• Check buffer flushing: large responses with small proxy-buffers.
• Check DNS resolution: resolver in ConfigMap if using upstream domain names.
• Fix: increase worker-connections, enable TLS session cache, tune buffers, add resolver.
• Rollback: revert ConfigMap.

Playbook 3: Client IP Missing

• Verify externalTrafficPolicy: Local on Service.
• Verify use-forwarded-headers: "true" and compute-full-forwarded-for: "true" in ConfigMap.
• If L4 LB with PROXY protocol: enable use-proxy-protocol: "true" in ConfigMap and annotate Service with service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*" (AWS NLB) or equivalent.
• Fix: adjust Service policy and ConfigMap.
• Rollback: revert to Cluster policy.

Playbook 4: Certificate Errors

• Check cert-manager Certificate status: kubectl get certificate -n <ns>
• Verify secret contains tls.crt, tls.key, ca.crt (for chain completion).
• Check SNI matching: kubectl -n ingress-nginx exec deploy/ingress-nginx-controller -- nginx -T | grep server_name
• Fix: renew cert, enable enable-ssl-chain-completion: "true", verify secret format.
• Rollback: revert to previous valid secret.

Quick Rollback Commands

# ConfigMap revert
kubectl -n ingress-nginx apply -f nginx-configmap-prev.yaml

# Deployment rollback
kubectl -n ingress-nginx rollout undo deploy/ingress-nginx-controller

# Annotation removal
kubectl annotate ingress myapp nginx.ingress.kubernetes.io/limit-rps- --overwrite

# Service policy revert
kubectl -n ingress-nginx patch svc ingress-nginx-controller -p '{"spec":{"externalTrafficPolicy":"Cluster"}}'

# Scale up for capacity
kubectl -n ingress-nginx scale deploy ingress-nginx-controller --replicas=10

Operations Checklist

• Define a narrow pilot with one Ingress and the echo test route.
• Inventory versions: Kubernetes 1.29+, Controller 1.11+, Helm chart 4.11+, Cert-manager 1.14+.
• Record baseline: RPS and p50/p95/p99 for 60s at steady load.
• Right-size controller replicas (min 3) and resources (requests 500m/512Mi, limits 2/1Gi); confirm HPA and PDB.
• Apply controller ConfigMap changes: worker-processes, worker-connections, keep-alive, upstream-keepalive, buffers, timeouts, TLS, logging, rate limiting.
• Verify curl timing and controller metrics after reload.
• Tune TLS: enable session cache, tickets (if allowed), OCSP stapling, HSTS; verify appconnect improvements on HTTPS.
• Choose Service externalTrafficPolicy based on client IP requirements; confirm endpoint distribution.
• Add per-Ingress annotations for buffering, body size, keepalive, rate limits, canary as needed.
• Re-test and compare against baseline; document deltas.
• If regressions appear: roll back ConfigMap or deployment; scale up temporarily.
• Schedule quarterly review: retest assumptions, versions, and tunables.

Conclusion

Ingress performance is not one magic flag; it is a small set of focused, measurable changes that compound. By inventorying your environment, establishing a baseline, and applying safe, reversible tuning to connections, buffers, TLS, and traffic policy, you can improve latency and throughput without risking outages. Start with a narrow pilot route, verify each change with simple load and curl timing, and keep rollback steps ready. With the checklist in hand, you can repeat the process for each high-traffic Ingress and maintain performance as your traffic and services evolve.