Kubernetes capacity planning with practical examples: practical implementation guide

Intro

Capacity planning in Kubernetes is about translating real demand into the right mix of requests, limits, replicas, and nodes, then proving the result with measurements. This guide gives you a practical, example-driven workflow you can apply to a single service today and extend across a cluster.

You will learn how to:

Define SLOs and baseline load.
Size CPU and memory requests/limits from measurements.
Pick scaling signals for HPA and use VPA safely.
Plan node types and pod density with safety margins.
Include storage and network in your math.
Build Prometheus dashboards tied to risk.

Related topics used in examples: Docker images, Helm values, GitLab CI/CD burst patterns, Nginx ingress, Ceph-backed storage, and Prometheus metrics.

Workflow Overview

Use this repeatable sequence for each workload:

Set SLOs and forecast demand.
Measure a baseline on one replica.
Convert measurements to requests/limits.
Choose replicas and HPA signals; enable VPA in recommend-only mode.
Plan nodes and pod density; assert headroom.
Include storage and network budgets.
Add dashboards and alerts linked to SLOs and saturation.
Run a pilot, review results, then widen scope.

Each step yields artifacts you can keep in Git: values.yaml, Deployment, HPA, and dashboard configs.

Define SLOs and baselines

Pick concrete targets and inputs you will size against:

Availability SLO: 99.9% monthly for the service.
Latency SLO: p95 HTTP latency <= 200 ms at peak.
Traffic: forecast peak QPS, payload sizes, and background jobs.
Batch windows: GitLab CI/CD spikes at top of the hour; nightly data jobs.

Measure a single replica under realistic data:

CPU per request (ms CPU per request at 1 vCPU baseline).
Memory working set (MiB) at steady state and during spikes (GC, compactions).
I/O: requests per second and bytes for storage and network.

Example quick commands:

kubectl top pods -n app
kubectl describe node <nodeName> | egrep -i "capacity|allocatable|pods"

If you front the service with Nginx Ingress, enable access logs and export QPS and latency histograms to Prometheus.

Requests and limits sizing

Translate measurements to Kubernetes resource specs. The goal is to avoid CPU throttling and OOMKills while preserving bin packing efficiency.

Assume a stateless API behind Nginx with these baseline measurements on one replica:

Peak QPS: 200
CPU per request: 2 ms on a 1 vCPU baseline
Steady memory working set: 350 MiB, p95 spike: 500 MiB

CPU math:

CPU needed = QPS CPU_per_request = 200 0.002 = 0.4 vCPU
Add 20% headroom for jitter and libraries: 0.48 vCPU
Round requests.cpu to 600m; set limits.cpu to 1200m to allow burst without starve

Memory math:

Request = p95 working set 1.2 = 500 MiB 1.2 = 600 MiB
Limit = Request * 1.5 = 900 MiB (gives room before OOM while surfacing leaks)

Helm values example (values.yaml):

resources:
  requests:
    cpu: "600m"
    memory: "600Mi"
  limits:
    cpu: "1200m"
    memory: "900Mi"

Deployment excerpt:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 2
  template:
    spec:
      containers:
      - name: api
        image: registry.local/api:1.2.3
        resources:
          requests:
            cpu: "600m"
            memory: "600Mi"
          limits:
            cpu: "1200m"
            memory: "900Mi"

Rules of thumb you can defend:

Start requests near p95 observed usage; keep limits 1.5x to 2x requests for CPU-bound services.
For memory, set limits cautiously. Too tight limits cause OOMKills. Too loose hides leaks. Track restarts.
Recompute after each release that changes performance characteristics.

Replicas and autoscaling

Choose initial replicas for availability, then add autoscaling.

Replicas for availability:

Minimum 2 replicas per zone for HA.
If single-zone: 3 replicas across nodes to tolerate one node drain.

HPA using CPU utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleUp:
      policies:
      - type: Percent
        value: 100
        periodSeconds: 60
      stabilizationWindowSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300

Custom metrics (recommended when latency or backlog matters more than CPU):

Use request rate per replica, queue depth, or p90 latency as signals via Prometheus Adapter.

Example custom metric HPA metric stanza:

  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "150"

VPA in recommend-only mode to avoid fighting HPA:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"

Read recommendations and adjust requests in Git, then redeploy.

Time-windowed scaling for batch spikes:

Use a CronJob that patches Deployment replicas before and after a known window.
Alternatively, feed a backlog size metric to HPA so it reacts automatically.

Nodes and bin packing

Compute allocatable, cap pod density, and avoid noisy neighbors.

Example node type:

Instance: 4 vCPU, 16 GiB RAM
kube-reserved + system: 0.5 vCPU, 1.5 GiB
Allocatable: 3.5 vCPU, 14.5 GiB

With the api pod sized at 600m CPU and 600Mi request:

Max CPU-bound pods per node by CPU = floor(3.5 / 0.6) = 5
By memory = floor(14.5 GiB / 0.6 GiB) = 24
CPU is the bottleneck, so plan 5 api pods per node max.

Best practices:

Set a per-node pod cap below theoretical max (for example 30) to leave space for DaemonSets and surge during rollouts.
Use PodTopologySpread to distribute replicas across nodes and zones.
For noisy workloads (builds, GitLab CI runners), isolate with taints/tolerations or separate node pools.

Storage and network

Do not ignore I/O; many incidents come from saturating disks or links while CPU looks fine.

Ceph capacity:

Replication factor 3 means effective capacity is roughly raw * 0.33 before metadata and overhead. Keep pools under ~70% to avoid latency cliffs.
Plan for recovery traffic during OSD failure; keep network headroom.

Throughput and IOPS:

Measure p95 read/write IOPS and MiB/s during steady state and maintenance windows (backups, compaction, reindexing).
For log-heavy services, consider local NVMe with a bounded retention and ship to object storage asynchronously.

Ingress and egress:

For Nginx Ingress, track requests per second, response sizes, and upstream latency histograms. Size CPU for TLS termination and gzip.
Budget cross-zone and cross-cluster egress; put chatty services in the same zone when possible.

Observability

Tie dashboards and alerts to SLOs and saturation, not just host metrics.

Prometheus queries you can start with:

CPU per pod:

rate(container_cpu_usage_seconds_total{container!="",pod=~"api-.*"}[5m])

Memory working set per pod:

container_memory_working_set_bytes{container!="",pod=~"api-.*"}

Nginx request rate and latency (assuming exported metrics):

sum(rate(nginx_ingress_controller_requests[1m])) by (ingress)
histogram_quantile(0.95, sum(rate(nginx_ingress_controller_request_duration_seconds_bucket[5m])) by (le, ingress))

Queue depth or backlog custom metric from the app for HPA decisions.

Alerts:

Burn rate against latency SLO.
Node allocatable CPU > 80% for 10m.
Pod OOMKill count > 0 in 5m.
Ceph pool fullness > 70% and OSD nearfull.

Safety margins and cost

Aim for resilience without waste.

Keep cluster-wide CPU and memory utilization under 70% during steady state to absorb spikes and disruptions.
For critical services, maintain N+1 node capacity in each failure domain.
Track cost per request: sum node, storage, and egress cost divided by successful requests. Use this to tune requests and limits over time.
Revisit size after major code changes or dependency upgrades.

Local Pilot Plan

Start small, measure, and expand.

Pilot scope:

One stateless API behind Nginx Ingress.
Add requests/limits, an HPA on CPU at 70% target, and VPA in recommend-only mode.
Dashboards for CPU, memory, QPS, p95 latency, and error rate.

Steps:

Add resource requests/limits and deploy to a staging namespace.
Run a controlled load test to the target peak QPS.
Record CPU/request and memory working set; recompute requests if needed.
Enable HPA and verify scaling behavior and stabilization.
Validate headroom: keep nodes under 70% CPU during peak.
Document results and roll to production in one region or cluster.

Acceptance criteria:

p95 latency <= SLO at target peak.
No OOMKills or throttling warnings.
HPA scales within 2 minutes of a step increase.
Node and pod eviction rate is near zero during the test.

Rollback plan:

Keep previous Deployment revision and HPA disabled manifest ready.
If alerts fire, scale replicas back to the last known good value and raise requests.

Conclusion

A practical capacity plan in Kubernetes starts with SLOs, turns measurements into requests and limits, chooses scaling signals that reflect real load, and keeps enough headroom to ride out failures and bursts. Run a small pilot on one service, verify against clear acceptance criteria, and expand iteratively. As your workloads and traffic evolve, revisit assumptions, keep dashboards honest, and adjust requests, limits, and node pools with evidence.