Helm capacity planning with practical examples: practical implementation guide

Intro

Helm capacity planning is the discipline of predicting and controlling how much CPU, memory, and replicas your Kubernetes workloads need, then encoding those decisions in your charts. Done well, it cuts cost, prevents outages, and makes scaling predictable. This guide gives you a practitioner-focused workflow, concrete values.yaml and HPA snippets, and a safe pilot you can run in a sandbox before rolling wider.

Workflow Overview

Use this repeatable sequence for each service:

1. Define targets: expected peak RPS, concurrency, latency SLO, and error budget.
2. Baseline locally: run the service in Docker or a test namespace and measure CPU%, RSS memory, and P95 latency under representative load.
3. Estimate per-request cost: derive CPU ms/req and steady-state memory per worker.
4. Choose pod shape: set requests and limits to fit node sizing and QoS goals.
5. Size replicas: compute replicas for peak with headroom, then set HPA targets.
6. Instrument: expose app and Nginx metrics; scrape with Prometheus.
7. Validate: run a controlled load and check throttling, OOM, and latency.
8. Roll out gradually: increase traffic while watching key signals.

Key Concepts and Signals

Core definitions:

• CPU request and limit: reserved millicores and the cap. Throttle risk rises near the limit.
• Memory request and limit: reserved MiB and the cap. OOMKill occurs when usage exceeds the limit.
• QoS classes: Guaranteed (requests=limits), Burstable (requests<limits), BestEffort (no requests). Guaranteed reduces eviction risk; Burstable allows CPU burst.

Primary signals to watch:

• CPU utilization per pod (target 50-70% at peak for headroom)
• RSS memory and GC behavior (target <80% of limit at peak)
• P95 latency and error rate
• Queue depth or backlog for workers
• Nginx active connections and 5xx rates

Back-of-the-envelope formulas:

Pick safety_factor between 1.3 and 2.0 depending on burstiness.

• rps_per_pod ~= concurrency_per_pod / p95_service_time_sec
• replicas_needed = ceil((peak_rps / rps_per_pod) * safety_factor)
• cpu_request_per_pod ~= p95_cpu_ms_per_req * rps_per_pod / 1000 (to cores)

Practical Examples

Example A: Nginx fronting a static site or upstream API Assumptions: p95 10 ms per request on cache hit, target peak 2000 rps, Nginx uses about 60-100 Mi steady memory per worker.

• rps_per_pod: with worker_processes=2, concurrency_per_pod ~= 2 * 1024 using epoll. Concurrency is often limited by CPU; assume safe rps_per_pod=1500.
• replicas_needed = ceil((2000 / 1500) * 1.4) = 2
• cpu_request_per_pod: assume 0.2 cpu at target; memory_request 128Mi.

values.yaml

replicaCount: 2
resources:
  requests:
    cpu: 200m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi
nginx:
  workerProcesses: 2
  workerConnections: 2048
hpa:
  enabled: true
  minReplicas: 2
  maxReplicas: 10
  targetCPUUtilizationPercentage: 60

HPA template (autoscaling/v2)

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: {{ include "chart.fullname" . }}
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: {{ include "chart.fullname" . }}
  minReplicas: {{ .Values.hpa.minReplicas }}
  maxReplicas: {{ .Values.hpa.maxReplicas }}
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: {{ .Values.hpa.targetCPUUtilizationPercentage }}

Example B: CPU-bound API (Go or Node.js) Assumptions: p95 50 ms/req at 1 vCPU saturation; target peak 600 rps.

To reduce replicas, increase per-pod CPU: try 2 vCPU request.

• rps_per_pod at 60% CPU target: 0.6 cpu * (1 / 0.050) = 12 rps if single-threaded. Go can use multiple threads; assume 4 workers -> ~48 rps.
• replicas_needed = ceil((600 / 48) * 1.5) = 19
• New rps_per_pod ~= 2 * 0.6 / 0.050 = 24 rps per worker; with 4 workers ~96 rps.
• replicas_needed = ceil((600 / 96) * 1.4) = 9

values.yaml

replicaCount: 9
resources:
  requests:
    cpu: "2"
    memory: 512Mi
  limits:
    cpu: "3"
    memory: 1Gi
env:
  - name: GOMAXPROCS
    value: "4"
  - name: NODE_OPTIONS
    value: "--max-old-space-size=512"
hpa:
  enabled: true
  minReplicas: 6
  maxReplicas: 20
  targetCPUUtilizationPercentage: 65

Example C: Background worker pulling from a queue Assumptions: each job takes 200 ms CPU and 300 ms wall time; target 500 jobs/s; worker concurrency 10.

• jobs_per_pod = 10 / 0.300 ~= 33 jobs/s (wall time bound)
• cpu_request_per_pod = (0.200 s * 33) = 6.6 cpu seconds/s -> 6.6 cores to hold 100% CPU; at 70% target, request ~4.6 cores. That is too large; reduce concurrency to 4 per pod or use more pods.
• With concurrency 4: jobs_per_pod ~= 13; cpu_request ~= (0.200 * 13) = 2.6 cores; request 2, limit 3.
• replicas_needed = ceil((500 / 13) * 1.3) = 50

values.yaml

replicaCount: 50
resources:
  requests:
    cpu: "2"
    memory: 512Mi
  limits:
    cpu: "3"
    memory: 768Mi
env:
  - name: WORKER_CONCURRENCY
    value: "4"
hpa:
  enabled: true
  minReplicas: 20
  maxReplicas: 120
  targetCPUUtilizationPercentage: 60

Prometheus queries to validate capacity

• RPS: sum(rate(http_requests_total[5m])) by (pod)
• Latency: histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, pod))
• CPU: sum(rate(container_cpu_usage_seconds_total{image!=""}[5m])) by (pod)
• Memory: container_memory_working_set_bytes{image!=""}

GitLab CI/CD example job for Helm rollout

deploy:
  stage: deploy
  image: alpine/helm:3.14.0
  script:
    - helm upgrade --install api charts/api \
        --namespace prod \
        --set resources.requests.cpu=2 \
        --set resources.requests.memory=512Mi \
        --set hpa.targetCPUUtilizationPercentage=65
  rules:
    - if: "$CI_COMMIT_BRANCH == 'main'"

Local Pilot Plan

Keep the first pilot narrow, measurable, and easy to inspect locally:

1. Select one service with stable traffic and clear SLOs.
2. Add instrumentation: request rate, P95 latency, CPU, RSS.
3. Implement values.yaml with explicit requests/limits and an HPA.
4. In a sandbox namespace, run a short load while watching Prometheus.
5. Check for throttling, OOM, and latency regressions.
6. Freeze the chart version and document the chosen pod shape and scaling policy.
7. Increase traffic gradually and recheck the same dashboards.

Operational Safety Margins

Practical defaults

• CPU headroom: target 60-70% utilization at peak; HPA target 60-65%.
• Memory headroom: size so peak stays under 70-80% of limit.
• N+1: keep at least one spare replica above the minimum to absorb failures.
• Readiness gates: set readinessProbe to protect latency during warmup.
• QoS: for latency-critical services, consider requests close to limits; for bursty web traffic, allow a 1.5x-2x CPU limit above request.

Common risks

• OOMKill from sudden traffic or leaks: watch working set, avoid memory limits too close to steady state.
• CPU throttling: sustained near-limit CPU induces latency spikes; lower HPA target or raise limits.
• Pod density fragmentation: oversized pods reduce scheduling flexibility; prefer more moderate pod sizes.
• Node overcommit: align requests with node capacity and cluster autoscaler.
• I/O and ephemeral storage: set requests/limits if the workload writes to disk.

Controls to add

• PodDisruptionBudget to keep minimum replicas during maintenance.
• ResourceQuota and LimitRange per namespace to prevent runaway pods.
• Alerts: high CPU for 10m, memory >80% limit, latency SLO burn, 5xx rate.

Conclusion

Capacity planning with Helm is about measuring, modeling, and encoding resource decisions as code. Start with clear targets, baseline locally, choose a sensible pod shape with headroom, set HPA targets, and validate under load. Keep a safety margin, watch the right signals, and grow iteratively. The examples and snippets here should let you ship a small, safe pilot, learn from it, and scale confidently.