Kubernetes Ingress Capacity Planning with Practical Examples

Intro

Your Kubernetes Ingress is the front door to your platform. If it is under-sized or misconfigured, the symptoms are painful: intermittent 5xx errors, timeouts, and sudden outages during traffic spikes. This guide gives you a repeatable, practical approach to Ingress capacity planning. You will inventory your environment, estimate load and headroom, apply safe defaults, validate with load testing, and prepare recovery steps for common failures.

By the end, you will have a template you can run in staging and then production to keep your Ingress healthy as traffic evolves.

1. Environment and Version Inventory

Before tuning, establish a baseline.

Run the following to capture versions and current footprint:

kubectl version --short
kubectl get ingressclass
kubectl get pods -A | grep -E "ingress|gateway"
kubectl top nodes
kubectl top pods -A | grep ingress
kubectl get ingress --all-namespaces | wc -l
kubectl get services --all-namespaces | wc -l
kubectl get endpoints --all-namespaces | wc -l

Record the results in a simple table you can maintain in version control:

Why endpoints matter: most controllers build an in-memory routing table and upstream pool. More Ingress rules, hosts, and endpoints typically increase memory and reload time.

Tip: keep this inventory updated whenever you add services or upgrade the controller.

2. Right-size your baseline

Start with a sizing hypothesis based on traffic and latency goals, then configure conservative requests and limits you can tune with data.

2.1 Estimate concurrency and RPS per pod

Use a quick back-of-the-envelope based on Little's Law:

• concurrency = RPS x average latency (in seconds)
• target RPS per pod = desired cluster RPS / planned replicas

Example:

• Target steady-state: 2,000 RPS
• Average end-to-end latency: 100 ms (0.1 s)
• Concurrency estimate: 2,000 x 0.1 = 200 concurrent requests
• Plan 30% headroom: 260 concurrent
• Start with 4 Ingress pods: ~65 concurrent per pod

If your average response size is large or TLS termination is enabled, expect higher CPU per request and size up accordingly.

2.2 Set resource requests and limits

Begin with requests slightly above observed 80th percentile usage and limits high enough to absorb short spikes without sustained throttling.

# Deployment snippet for ingress-nginx
resources:
  requests:
    cpu: 300m
    memory: 512Mi
  limits:
    cpu: "2"
    memory: 2Gi

After rollout, monitor with:

kubectl top pod -n ingress-nginx

Tune requests toward your p80 usage so the HPA can react accurately to load.

2.3 Ingress-NGINX performance knobs (safe starting points)

Place these in the controller ConfigMap and tune from there:

# ingress-nginx ConfigMap
worker-processes: "auto"
worker-connections: "16384"
keep-alive: "75"
keep-alive-requests: "1000"
proxy-read-timeout: "60s"
proxy-send-timeout: "60s"
proxy-body-size: "16m"
upstream-keepalive-connections: "64"

Notes:

• worker-processes auto lets NGINX match CPU cores.
• worker-connections sets the max concurrent connections per worker. Combined with file descriptor limits, this caps total concurrency.
• keep-alive reduces TLS and TCP handshake overhead for repeat callers.

Apply and watch the controller logs for a clean reload.

2.4 Horizontal Pod Autoscaling (CPU and memory)

Prefer autoscaling/v2 to use multiple metrics:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: ingress-nginx-controller
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 75

Add a PodDisruptionBudget to maintain availability during maintenance:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: ingress-nginx-pdb
  namespace: ingress-nginx
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx

Increase resilience by spreading replicas:

# Add to Deployment spec
spec:
  template:
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
        labelSelector:
          matchLabels:
            app.kubernetes.io/name: ingress-nginx
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app.kubernetes.io/name: ingress-nginx
            topologyKey: kubernetes.io/hostname

3. Validate with load and observe

Prove your configuration under realistic conditions before trusting it in production.

3.1 Run a targeted load test

Pick a representative path that exercises typical upstream behavior (cache hit rate, auth, payload size). Use hey for a quick test:

# 10,000 requests at 100 concurrent connections
hey -n 10000 -c 100 https://ingress.example.com/

To align concurrency with your estimate, use the calculation from section 2.1. For spike tests, ramp concurrency up over 1 to 3 minutes and observe scaling.

Tools you can use:

• hey: simple HTTP benchmarking
• vegeta: attack and rate-based tests
• k6: scripted scenarios and threshold assertions

3.2 Watch system behavior while testing

Run these in parallel:

watch -n 2 kubectl top pods -n ingress-nginx
watch -n 2 kubectl get hpa -n ingress-nginx
kubectl logs -n ingress-nginx deploy/ingress-nginx-controller -f

Key signals to track:

• Error rates: 5xx from Ingress, 499 client canceled, and upstream 5xx
• Latency: p50, p95, p99; sustained p99 > 500 ms usually triggers scale-out
• CPU throttling and OOM events on controller pods
• HPA decisions and stabilization behavior
• Connection metrics: active, accepted, handled (if you scrape NGINX metrics)

Set alerts for outliers, not just averages. For example, fire an alert if error rate > 1% for 5 minutes or p99 latency > 500 ms for 5 minutes.

4. Common failure modes and quick recovery

Use these patterns to detect and fix issues fast.

1. Resource starvation (CPU throttling or OOMKilled)

• Symptoms: high request latency, 5xx bursts, CrashLoopBackOff, logs show throttling.
• Verify: kubectl describe pod ... | grep -i -E "throttling|oom".
• Fix: raise limits, align requests with p80 usage, scale replicas. Example:
• Increase CPU limit to 3 cores and memory to 3 GiB temporarily.
• If OOM, reduce routes per controller (shard by ingressClass or namespace) or raise memory.

1. Misconfigured HPA (thrashing or sluggish scaling)

• Symptoms: frequent up/down within minutes, or slow to add pods during spikes.
• Verify: kubectl describe hpa for events like FailedGetResourceMetric.
• Fix: add stabilization and rate limits:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60

1. TLS and certificate problems

• Symptoms: handshake failures, sudden 495/525-style errors, spike in CPU.
• Verify: kubectl get secrets -n ingress-nginx for TLS secrets; openssl s_client -connect host:443 to inspect cert.
• Fix: confirm cert-manager renewals; enable session reuse and caches via ConfigMap; roll back to previous working secret if needed.

1. Connection exhaustion (node or NGINX)

• Symptoms: timeouts under spikes, SYN backlog drops, node conntrack full, or NGINX worker_connections exceeded.
• Verify: controller logs; node metrics for conntrack; cloud load balancer connection limits.
• Fix: increase NGINX worker-connections, scale replicas, or split traffic across multiple IngressClasses. If node-level limits are hit, spread pods across more nodes.

1. Config or version upgrade regressions

• Symptoms: controller fails readiness, 404s for valid routes, unexpected redirects.
• Verify: kubectl rollout status deploy/ingress-nginx-controller -n ingress-nginx and logs for config errors.
• Fix: rollback quickly:

kubectl rollout undo deployment/ingress-nginx-controller -n ingress-nginx

For ConfigMap regressions, revert your IaC commit and re-apply.

Always keep manifests in version control and export a snapshot of Ingress, Services, and ConfigMaps before major changes.

5. Operations checklist and alert thresholds

Run this lightweight loop to keep capacity healthy and predictable.

• Weekly: review kubectl top pods -n ingress-nginx and compare to HPA targets.
• Monthly: validate HPA bounds (minReplicas, maxReplicas) vs. traffic trends.
• After every config or version change: run a short load test and confirm error and latency SLOs.
• Certificates: monitor expiry and auto-renewal status.
• Chaos and failover: drain a node once per quarter to verify PDB and spreading rules.
• Documentation: update your capacity worksheet with current requests/limits, replicas, and latest test results.

Suggested alert thresholds and next actions:

Putting it all together: a quick path

1. Inventory your version, controller, nodes, and current usage.
2. Estimate concurrency and per-pod RPS; add 30% headroom.
3. Apply conservative requests/limits and NGINX ConfigMap defaults.
4. Enable HPA with CPU and memory, plus stabilization behavior.
5. Add a PDB and spread replicas across nodes and zones.
6. Run a representative load test and watch error, latency, and resource signals.
7. Tune iteratively and document the new baseline.

Conclusion

Ingress capacity planning is not a one-time exercise. Treat it as a small operational loop: measure, hypothesize, change, and verify. Start with an explicit inventory, translate traffic goals into concurrency and RPS, apply safe defaults, and validate under load. Add guardrails with HPA, PDBs, and replica spreading, and keep a clear rollback path for upgrades and config changes. With this playbook, you can scale confidently, avoid surprise bottlenecks, and keep your cluster’s front door fast and reliable.