Kubernetes Deployment Capacity Planning: A Practical Guide with Examples

Intro

Kubernetes Deployment capacity planning means estimating and configuring the CPU, memory, and replica counts your workloads need to run reliably without wasting infrastructure. Every Deployment you create consumes cluster resources. Set requests and limits too low, and pods may be evicted or throttled under load. Set them too high, and you overprovision, inflating costs and starving other workloads. A structured approach separates estimation, configuration, and verification, reducing guesswork and rework. This guide walks through a practical method with concrete commands and examples.

You will learn how to inventory your cluster, choose safe starting values, observe actual usage, configure autoscaling, diagnose failures, and maintain a repeatable checklist. The approach works for stateless web services, APIs, and similar workloads. By the end, you will be able to size Deployments with confidence and avoid common resource-related outages.

Version and Environment Inventory

Before setting any capacity parameters, confirm your Kubernetes version and available API resources. Capacity-related fields like requests, limits, and autoscaling behavior have evolved across versions. Run these commands on a cluster with kubectl access:

kubectl version --short
kubectl api-resources | grep -E 'deployments|horizontalpodautoscalers|resourcequotas'

Expected output will show your client and server versions, then a list of relevant API groups. For example, on Kubernetes 1.28:

Client Version: v1.28.2
Server Version: v1.28.4
deployments                         apps/v1
horizontalpodautoscalers            autoscaling/v2
resourcequotas                      v1

If you use a managed Kubernetes service, the version may appear in the cloud console. Record the exact minor version because HPA v2 fields differ from v1. Also verify that the metrics API is available if you plan to autoscale on CPU or memory:

kubectl get --raw /apis/metrics.k8s.io/v1beta1 | head -n 5

You should see a resource list, not a 404 error. If the metrics API is missing, install the metrics-server or your provider's equivalent. For capacity planning, you also need to know node capacity and existing resource pressure:

kubectl describe nodes | grep -A 5 'Allocated resources'

The output shows CPU and memory requests and limits as percentages. Use this to gauge headroom. If nodes are already over 80% requested, avoid adding large Deployments without cluster expansion. Document these versions and numbers before proceeding; they affect how you interpret metrics and configure autoscaling.

Safe Configuration Path

Begin capacity planning with a single Deployment and conservative estimates. Do not try to size all services at once. A narrow pilot lets you observe real usage and adjust without risking production. We will create a simple web application Deployment with explicit requests and limits, then scale it manually and via HPA. The accountable owner for this initial sizing is the service's engineering lead, who should revisit these values after the first week of production traffic and monthly thereafter.

Step 1: Start with a baseline manifest

Assume you have a stateless web app that typically uses 100 millicores (0.1 CPU) and 128 MiB of memory per instance under light load. Peak may triple that. Set initial requests to the typical use and limits to the expected peak. Create web-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: web
        image: nginx:1.25
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "300m"
            memory: "256Mi"

Apply it:

kubectl apply -f web-deployment.yaml

Expected result:

deployment.apps/web-app created

Check the pods:

kubectl get pods -l app=web

You should see two pods running after a few seconds. The requests guarantee each pod gets at least the specified resources. The limits cap how much it can burst. If the pod exceeds its memory limit, it is OOM-killed. If it exceeds CPU limit, it is throttled. This is why limits should not be set arbitrarily high: throttling may degrade latency before you notice.

Step 2: Observe actual usage before scaling

Let the Deployment run under realistic traffic for a few hours or days. Monitor resource consumption per pod:

kubectl top pods -l app=web

Example output:

NAME                          CPU(cores)   MEMORY(bytes)
web-app-6b4d8c9f5-abcde       45m          102Mi
web-app-6b4d8c9f5-fghij       52m          118Mi

These are actual usage samples. If typical usage is much lower than requests, you can lower requests to improve bin packing. If it approaches limits, increase limits or replica count. For a quick 5-minute average, use kubectl top pods --containers for per-container detail. Record these values; they inform the next step.

Step 3: Configure Horizontal Pod Autoscaler (HPA)

Autoscaling based on CPU or memory is the primary reactive capacity mechanism. Create an HPA targeting 60% average CPU utilization across the pods:

kubectl autoscale deployment web-app --cpu=60% --min=2 --max=8

Expected output:

horizontalpodautoscaler.autoscaling/web-app autoscaled

Verify the HPA:

kubectl get hpa web-app

Example output:

NAME      REFERENCE            TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
web-app   Deployment/web-app   10%/60%   2         8         2          5m

The TARGETS column shows current utilization vs target. Wait a few minutes and repeat if no load. To simulate load, you can temporarily run a load generator pod. However, do not rely on HPA alone; it reacts after the fact. Set a reasonable minimum replica count for base load to avoid cold starts.

Step 4: Adjust requests and limits iteratively

Once you have metrics, tune the manifest. Suppose the average pod uses 80m CPU and 150 MiB memory at peak. Your original limits were too low; raise them to 500m CPU and 512 MiB memory, but keep requests at 100m and 128 MiB to avoid over-reserving. Update the Deployment:

kubectl patch deployment web-app -p '{"spec":{"template":{"spec":{"containers":[{"name":"web","resources":{"limits":{"cpu":"500m","memory":"512Mi"}}}]}}}}'

This triggers a rolling update. Watch the rollout:

kubectl rollout status deployment/web-app

Expected output:

deployment "web-app" successfully rolled out

Re-check usage and HPA. This iterative loop is central to capacity planning. The engineering lead should review these settings weekly for the first month, then monthly, or after any significant traffic change.

Step 5: Consider Node Affinity and Pod Disruption Budgets (optional)

Capacity planning also includes availability. If your Deployment must not have all replicas on one node, use pod anti-affinity. For example, to spread pods across nodes, add this to the pod spec:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - web
        topologyKey: kubernetes.io/hostname

Additionally, define a PodDisruptionBudget (PDB) to limit voluntary disruptions during maintenance. Example PDB:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: web

Apply with kubectl apply -f pdb.yaml. This ensures at least one pod remains available during node drains. The platform team owns PDB policies and reviews them quarterly.

Verification and Diagnostics

After configuration, verify that capacity settings are effective and that the system behaves as expected under load. Use multiple observability signals.

Check resource requests and limits

kubectl get deployment web-app -o json | jq '.spec.template.spec.containers[0].resources'

Expected output:

{
  "limits": {
    "cpu": "500m",
    "memory": "512Mi"
  },
  "requests": {
    "cpu": "100m",
    "memory": "128Mi"
  }
}

Check HPA status and events

kubectl describe hpa web-app

Look for scaling events. Example snippet:

Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  10m   horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target

If you see repeated rapid scaling, check for flapping caused by low thresholds or noisy metrics. Adjust the target or add stabilization windows as described later.

Load test to confirm scaling

Generate load using a temporary pod with a simple HTTP load tool. For example, run hey from a debug container:

kubectl run load-generator --image=williamyeh/hey --rm -it -- -n 10000 -c 10 http://web-app-service/

Monitor the HPA and pod count during the test:

watch kubectl get hpa, pods

After the test, check that the replica count decreased back to the minimum once the load stopped. This confirms the HPA scales down correctly. The service owner should run such a test before any major release or traffic event.

Diagnose capacity issues

If pods are pending, check node resources:

kubectl describe pod <pod-name> | grep -A 3 Events

A common event is 0/2 nodes are available: 2 Insufficient cpu. This indicates that the cluster lacks capacity for the requested CPU. You must either reduce requests, add nodes, or scale down other workloads. If pods are OOM-killed, inspect the termination reason:

kubectl get pod <pod-name> -o json | jq '.status.containerStatuses[0].state.terminated.reason'

Output OOMKilled means the memory limit was too low. Increase the limit or fix the application leak. Use kubectl describe node to see memory pressure. These diagnostic steps should be documented in your runbook for on-call engineers.

Failure Modes and Recovery

Capacity planning can go wrong in several ways. Knowing the common failures and how to recover is essential. Each failure mode below lists a named owner who should be alerted and the typical recovery action.

OOMKilled and pod evictions

Failure: Pods exceed memory limits and are killed, causing restarts and potential downtime.

Detection: kubectl get pods shows RESTARTS increasing, and kubectl describe pod shows OOMKilled.

Recovery: Increase memory limits temporarily, but investigate the cause. If the application needs more memory at peak, adjust limits and possibly requests. For example, to set memory limit to 1Gi:

kubectl set resources deployment web-app --limits=memory=1Gi

Then verify with kubectl rollout status. If the app has a memory leak, fixing the code is the long-term solution. The application developer owns memory usage and should be alerted on the first OOMKilled event.

CPU throttling causing high latency

Failure: CPU limits are too low; pods are throttled but not killed, leading to slow responses.

Detection: Check container metrics for container_cpu_cfs_throttled_seconds_total, or correlate high latency with CPU limit hits in your monitoring dashboard.

Recovery: Increase CPU limits. For example:

kubectl set resources deployment web-app --limits=cpu=1

Monitor latency after the change. The SRE on call should perform this recovery and then schedule a capacity review with the service owner.

Node resource exhaustion and pending pods

Failure: Nodes run out of allocatable CPU or memory, so new pods cannot be scheduled.

Detection: kubectl get pods shows Pending state; kubectl describe node shows high resource pressure.

Recovery: Reduce resource requests for non-critical Deployments, add nodes, or use cluster autoscaler if available. To reduce requests temporarily:

kubectl set resources deployment web-app --requests=cpu=50m,memory=64Mi

This can starve pods if misused, so only do it when you understand the workload. The cluster administrator owns node capacity and should be notified when pressure exceeds 80%.

HPA misconfiguration causing flapping

Failure: HPA target utilization is set too low or metrics are noisy, causing rapid scale-up and scale-down, destabilizing the service.

Detection: HPA events show frequent SuccessfulRescale with alternating sizes.

Recovery: Adjust the target percentage or add a stabilization window. For example, to set scale-down stabilization to 5 minutes:

kubectl patch hpa web-app --type='json' -p='[{"op": "replace", "path": "/spec/behavior/scaleDown/stabilizationWindowSeconds", "value": 300}]'

Verify with kubectl get hpa web-app -o yaml. The service owner should review HPA settings monthly and after any traffic pattern change.

Rollback of a bad capacity change

If a change causes problems, roll back the Deployment to the previous revision:

kubectl rollout undo deployment/web-app

This reverts to the last known good configuration. Check the rollout status. To roll back to a specific revision, first list revisions:

kubectl rollout history deployment/web-app

Then use --to-revision=<n>. The release manager owns rollback decisions and should document the incident.

Common Pitfalls and How to Avoid Them

Beyond failure modes, several practices frequently trip up teams. Recognizing these pitfalls early saves time and resources.

Setting requests and limits too close together

Why it happens: Teams copy production values to development without adjusting for lower traffic, or they set both to the same high number to avoid throttling.

Consequence: Over-reservation wastes cluster capacity and leads to poor bin packing, increasing costs.

How to avoid: Base requests on actual average usage measured with kubectl top, and limits on peak usage plus a safety margin of 20 to 30 percent. Revisit monthly.

Ignoring node capacity and topology

Why it happens: Developers focus on pod-level limits and forget that nodes have finite resources.

Consequence: Pods may be unschedulable even though the cluster appears to have free resources if requests are skewed.

How to avoid: Use kubectl describe nodes regularly to check Allocated resources. Consider node affinity and anti-affinity to spread load. Alert when node requests exceed 70%.

Over-relying on HPA without minimum replicas

Why it happens: Teams assume HPA will handle all scaling, including from zero, but HPA only scales based on current load and may be slow to react to sudden spikes.

Consequence: Cold starts and dropped traffic during traffic bursts.

How to avoid: Set a minimum replica count that can handle baseline traffic plus some buffer. Use burstable CPU requests to allow spikes without throttling. The service owner should define this minimum.

Not monitoring actual usage after rollout

Why it happens: After deploying with initial guesses, teams move on to the next task.

Consequence: Resource settings become stale, leading to over- or under-provisioning.

How to avoid: Establish a review cadence (monthly) and use automated dashboards to track resource utilization trends. The engineering manager owns this process.

Using resource quotas as a blunt instrument

Why it happens: Administrators set tight quotas to control costs, but they restrict legitimate workloads.

Consequence: Deployments fail to schedule, frustrating developers.

How to avoid: Coordinate quotas with actual capacity plans. Review quota usage monthly and adjust based on projected growth. The platform team owns quota policy.

Operations Checklist

Use this checklist to review capacity for any Kubernetes Deployment before and after changes. The service owner is accountable for each item; the engineering manager reviews the completed checklist monthly.

Run through this checklist during initial deployment, after traffic changes, and periodically (e.g., monthly).

Conclusion

Kubernetes Deployment capacity planning is an ongoing process, not a one-time task. Begin with a baseline manifest that sets conservative requests and limits. Use kubectl top and HPA events to observe actual behavior. Adjust iteratively, and always verify with commands and load tests. When failures occur, know how to detect and recover quickly. Use the operations checklist to review capacity regularly. By following these practical steps, you reduce the risk of resource-related outages and keep your cluster efficient. Start with one small Deployment, measure, and then expand your process to other workloads. Capacity planning is a key part of running Kubernetes reliably.