Kubernetes Service Capacity Planning: A Practical Guide with Examples

Introduction

Capacity planning for Kubernetes Services is the practice of determining the right amount of compute, memory, and networking resources your service needs to handle current and future traffic while remaining cost-efficient and resilient. Done well, it prevents outages caused by resource exhaustion, avoids wasting money on overprovisioned clusters, and gives your team confidence when scaling up or rolling out new features.

This guide focuses on practical, hands-on techniques for developers, DevOps consultants, and technical startup teams running production Kubernetes workloads. We will cover how to analyze your service's resource usage, configure requests and limits, scale horizontally and vertically, diagnose performance bottlenecks, and prepare recovery procedures for common failure modes. Rather than treating capacity planning as a one-time exercise, we emphasize an iterative loop: observe, measure, adjust, and verify.

By the end of this article, you will understand how to apply Kubernetes-native tools such as the Metrics Server, Horizontal Pod Autoscaler (HPA), Vertical Pod Autoscaler (VPA), and kubectl commands to make informed capacity decisions for Services. Every recommendation includes concrete commands, configuration snippets, and expected outputs so you can test them in your own environment.

1. Version and Environment Inventory

Before touching any live configuration, you must know exactly what you are working with. Start by recording the Kubernetes server and client versions, because HPA, VPA, and resource management behaviors change between releases. Run the following commands and save the output:

kubectl version --short
# Example output:
# Client Version: v1.25.3
# Server Version: v1.25.3

Next, verify that the Metrics Server is installed and returning data. The Metrics Server aggregates CPU and memory usage from the kubelet and is required for HPA and kubectl top. Check its availability with:

kubectl get deployment metrics-server -n kube-system
# Expected output if installed:
# NAME             READY   UP-TO-DATE   AVAILABLE   AGE
# metrics-server   1/1     1            1           10d

If the deployment is missing, install it according to your cluster provider's documentation. Without Metrics Server, kubectl top commands will fail with an error like "Metrics API not available".

Gather contextual information about the Services you intend to plan for. List all Services in your namespace and note their types:

kubectl get svc -n your-namespace
# Example output:
# NAME           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
# frontend       ClusterIP   10.96.0.10      <none>        80/TCP           5d
# api            LoadBalancer 10.96.0.11    35.190.10.20   443:31000/TCP    5d
# internal-db    ClusterIP   None            <none>        5432/TCP         5d

Document which Services serve production traffic, their average response times, and any known scaling constraints. This inventory is the baseline for all further analysis.

Finally, review your cluster's node capacity and current allocation using kubectl describe nodes. Look for the "Allocated resources" section to see how much CPU and memory are already requested by pods:

kubectl describe nodes | grep -A5 "Allocated resources"
# Example condensed output:
# Allocated resources:
#   (Total limits may be over 100 percent, i.e., overcommitted.)
#   CPU Requests  CPU Limits  Memory Requests  Memory Limits
#   1200m (30%)   3000m (75%) 2Gi (25%)        4Gi (50%)

This overview tells you how much headroom exists cluster-wide before you even start scaling a single Service.

2. Measuring Current Resource Usage

Capacity planning starts with accurate measurements of your Service's current resource consumption. Use kubectl top to observe CPU and memory usage of pods backing your Service:

kubectl top pods -l app=frontend --sort-by=cpu
# Example output:
# NAME                        CPU(cores)   MEMORY(bytes)
# frontend-7c5f9b8c6d-2xq4l   150m         256Mi
# frontend-7c5f9b8c6d-7z8pt   120m         248Mi
# frontend-7c5f9b8c6d-k9m2n   180m         270Mi

Take these measurements during peak traffic hours to see the maximum load. Let the collection run for at least a full business cycle (e.g., 24-48 hours) to capture daily peaks and valleys. For longer-term trends, integrate Prometheus and Grafana, which provide historical data and queries like rate(container_cpu_usage_seconds_total[5m]) or sum by (pod) (container_memory_working_set_bytes).

Analyze the data to determine the baseline and peak resource needs. For example, if the frontend pods above use 150-180m CPU at peak and 50m at idle, you might set a CPU request of 100m (to cover average load) and a limit of 500m (to allow bursts) initially, then refine after observing behavior.

Also monitor network metrics for your Service. Kubernetes Services do not have built-in bandwidth limits, but the underlying node network capacity matters. If your Service is a LoadBalancer, check cloud provider metrics for total bytes in/out and connection rates. Use kubectl get endpoints <service-name> to see how traffic is distributed across pods.

kubectl get endpoints frontend
# Example output:
# NAME       ENDPOINTS                                         AGE
# frontend   10.244.1.5:8080,10.244.2.3:8080,10.244.3.7:8080   5d

If all traffic hits a single pod, you may have a session affinity misconfiguration or a client-side load balancing issue. Ensure your Service's sessionAffinity is set appropriately (default is None) unless you specifically need sticky sessions.

3. Setting Resource Requests and Limits

Resource requests and limits are the foundation of Service capacity planning. Requests guarantee a minimum amount of CPU and memory for your containers and are used by the scheduler to place pods on nodes. Limits cap the maximum resource usage. Setting these correctly prevents both starvation and noisy neighbor problems.

For each Deployment that backs your Service, define resources in the pod template. Here is an example YAML snippet for the frontend Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: yourregistry/frontend:1.2.3
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

Choose requests based on measured average usage plus a small buffer (e.g., 10-20%). Choose limits based on expected peak usage, but avoid setting them too low or your pod will be OOMKilled (for memory) or throttled (for CPU). For memory, a common practice is to set request equal to limit to avoid unexpected evictions due to overcommitment. However, for bursty workloads, you can set limit higher than request.

A good starting formula:

• CPU request = p95 CPU usage over 7 days, rounded up to nearest 10m.
• Memory request = p95 memory usage + 20% headroom.
• CPU limit = either 2x request or p99 usage, whichever is higher, but not exceeding node allocatable CPU per pod.
• Memory limit = request * 1.5 or 2x, but be cautious: if the limit is too high, a memory leak can consume the entire node.

Apply the updated Deployment:

kubectl apply -f frontend-deployment.yaml
kubectl rollout status deployment/frontend
# Expected output: deployment "frontend" successfully rolled out

After the rollout, verify the pods are running with the new resource settings:

kubectl get pods -l app=frontend -o jsonpath='{.items[*].spec.containers[0].resources}'
# Example output: {"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"100m","memory":"128Mi"}}

Monitor the pods for a few hours to ensure no OOMKills or excessive throttling occurs. If memory usage consistently remains near the limit, increase the limit; if CPU throttling is high (check container_cpu_cfs_throttled_seconds_total metric), either raise the CPU limit or optimize the code.

4. Scaling Strategies: Horizontal and Vertical

Horizontal Pod Autoscaler (HPA)

HPA automatically adjusts the number of pod replicas based on observed CPU utilization or custom metrics. It is ideal for Services with variable traffic patterns. To create an HPA for the frontend Deployment targeting 60% average CPU utilization:

kubectl autoscale deployment frontend --cpu-percent=60 --min=3 --max=10

Or define an HPA manifest:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: frontend-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: frontend
  minReplicas: 3
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60

Apply it and check the HPA status:

kubectl apply -f frontend-hpa.yaml
kubectl get hpa frontend-hpa
# Example output:
# NAME           REFERENCE             TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
# frontend-hpa   Deployment/frontend   45%/60%         3         10        3          5m

As load increases, the HPA will scale up replicas. Verify events:

kubectl describe hpa frontend-hpa
# Look for events like: "New size: 4; reason: cpu resource utilization (percentage of request) above target"

Note that HPA reacts to current metrics, not predictions. It can take a few minutes to scale, so plan for burst capacity by setting minReplicas appropriately or using custom metrics for faster reaction.

Vertical Pod Autoscaler (VPA)

VPA adjusts CPU and memory requests and limits for containers based on usage. It is useful for stable workloads where horizontal scaling is not possible (e.g., stateful databases) or when you want to rightsize resources. Unlike HPA, VPA typically requires pod restarts to apply new resources, so use it with care.

Install VPA from the Kubernetes autoscaler repository if not present. Then create a VPA resource:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: frontend-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       Deployment
    name:       frontend
  updatePolicy:
    updateMode: "Auto"

VPA in Auto mode will evict pods to apply new resource recommendations. This can cause brief downtime, so for critical Services, use "Off" mode to get recommendations only:

kubectl get vpa frontend-vpa
# Example output (recommendations shown later via describe):
# NAME           MODE   CPU     MEM       PROVIDED   AGE
# frontend-vpa   Off    150m    200Mi     True       10m

Review recommendations:

kubectl describe vpa frontend-vpa
# Recommendation section may show:
#   Container: frontend
#     Lower Bound:   CPU 100m, Memory 128Mi
#     Target:        CPU 150m, Memory 200Mi
#     Upper Bound:   CPU 300m, Memory 400Mi

Apply the target recommendations manually to your Deployment.

Manual Scaling and Pre-warming

Sometimes you need to scale manually before a known event (e.g., product launch). Use:

kubectl scale deployment frontend --replicas=8
kubectl rollout status deployment/frontend

Ensure the Service's load balancer and endpoints update accordingly.

5. Service Configuration for Capacity

Beyond pod resources, the Service object itself has settings that affect capacity. Key fields:

• Type: ClusterIP, NodePort, LoadBalancer, or ExternalName. LoadBalancer provides external access but may have cloud provider rate limits.
• ExternalTrafficPolicy: Cluster (default) or Local. With Local, traffic is routed only to nodes with running pods, preserving client source IP but potentially causing imbalance.
• SessionAffinity: None or ClientIP. ClientIP keeps requests from the same client to the same pod, which can help with caches but may cause uneven load distribution.
• ipFamilyPolicy: SingleStack or DualStack for IPv4/IPv6.

For high-traffic Services, use externalTrafficPolicy: Local and ensure even pod distribution across nodes to avoid overloading a single node's network. However, Local policy may not work well if your load balancer health checks do not reflect pod readiness properly.

Example Service with LoadBalancer and Local policy:

apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  type: LoadBalancer
  externalTrafficPolicy: Local
  selector:
    app: frontend
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

Check the Service endpoints and node allocation after scaling:

kubectl get svc frontend -o wide
kubectl get endpoints frontend
kubectl get pods -l app=frontend -o wide | awk '{print $1, $7}'

This shows which nodes the pods are on, helping you identify potential network bottlenecks.

6. Verification and Diagnostics

After any capacity change, verify that traffic flows correctly and resources are within expected ranges.

Run a load test against your Service. For an internal ClusterIP Service, use a temporary pod with a load generation tool like hey or ab:

kubectl run load-generator --image=busybox --restart=Never -- sh -c "while true; do wget -q -O- http://frontend; done"
# Better: use a tool like hey or vegeta from a pod with proper image.

Use kubectl port-forward to access the Service locally and send requests:

kubectl port-forward svc/frontend 8080:80
# In another terminal:
hey -z 30s -c 50 http://localhost:8080
# Example output:
#   Total: 30.0167 secs
#   Slowest: 0.2345 secs
#   Fastest: 0.0009 secs
#   Average: 0.0121 secs
#   Requests/sec: 4123.4567

During the load test, monitor pod metrics:

kubectl top pods -l app=frontend --sort-by=cpu
# Watch for pods approaching limits
kubectl describe pods -l app=frontend | grep -E "State|Reason|OOMKilled|Throttling"

Also check HPA events:

kubectl describe hpa frontend-hpa
# Look for scaling events

If you see OOMKilled events, increase memory limits. If CPU throttling occurs (indicated by container_cpu_cfs_throttled_seconds_total metric), raise CPU limits or reduce CPU requests to give more headroom.

Diagnostics should also include network-level checks. Use kubectl exec into a pod and test connectivity to Service DNS:

kubectl exec -it frontend-7c5f9b8c6d-2xq4l -- nslookup frontend
# Expected: resolves to ClusterIP
kubectl exec -it frontend-7c5f9b8c6d-2xq4l -- wget -q -O- http://frontend
# Expected: returns your app's response

7. Common Pitfalls and How to Avoid Them

Capacity planning mistakes can lead to outages, performance degradation, or inflated cloud bills. Here are the most frequent pitfalls and how to sidestep them.

1. Setting Requests Too High or Too Low

• Why it happens: Teams guess values without measuring actual usage, or copy from other projects.
• Consequence: High requests waste cluster capacity and prevent pods from being scheduled; low requests cause overcommitment and can lead to node pressure and evictions.
• How to avoid: Always base requests on measured metrics. Use VPA recommendations as a starting point, then adjust after observing real traffic.

2. Ignoring Memory vs. CPU Differences

• Why it happens: Treating CPU and memory the same way. CPU is compressible (throttled), memory is not (OOMKilled).
• Consequence: Setting a low memory limit causes pod restarts; setting a high CPU limit with low request can starve other pods.
• How to avoid: Set memory limits close to requests to avoid eviction, and set CPU limits higher for burst tolerance.

3. Not Considering Pod Startup Time

• Why it happens: HPA scales up, but new pods take time to initialize (e.g., loading cache, connecting to databases).
• Consequence: During a traffic spike, the Service may be overwhelmed before new pods are ready, causing timeouts.
• How to avoid: Use readiness probes to delay traffic to not-ready pods, and pre-warm pods before expected spikes. Consider using container startup probes for slow-initializing apps.

4. Overlooking Network Bandwidth

• Why it happens: Monitoring CPU/memory only; network capacity is often ignored.
• Consequence: A single node can become a network bottleneck, saturating its NIC and causing high latency for all pods on it.
• How to avoid: Distribute pods across nodes using pod anti-affinity, monitor node network usage (e.g., via node exporter), and consider using a Service mesh or ingress controller with better load balancing.

5. Not Testing Failure Scenarios

• Why it happens: Teams assume everything works until it doesn't; no chaos engineering or failure drills.
• Consequence: When a pod fails or a node goes down, the Service may degrade unpredictably, and recovery is slow.
• How to avoid: Regularly simulate pod failures (kubectl delete pod) and node drains. Ensure your HPA minReplicas covers a single node failure. Test the Service's behavior with reduced replicas.

6. Relying Solely on Automatic Scaling

• Why it happens: Trusting HPA to handle all scenarios without manual oversight.
• Consequence: HPA may not scale fast enough for sudden spikes, or may scale down too aggressively, causing flapping.
• How to avoid: Set appropriate minReplicas based on baseline load. Use custom metrics for faster scaling. Monitor HPA events and set alerts for scaling activity.

8. Failure Modes and Recovery Procedures

Even with careful planning, failures occur. Define recovery procedures for common scenarios.

Pod OOMKilled

Symptom: Pod restarts with reason OOMKilled, application crashes. Diagnosis: kubectl describe pod <pod-name> shows last state terminated with reason OOMKilled. Memory usage exceeded limit. Recovery: Increase memory limit temporarily to stabilize, then investigate memory leak. Update Deployment.

kubectl set resources deployment frontend --limits=memory=1Gi --requests=memory=512Mi
kubectl rollout restart deployment/frontend

After recovery, analyze memory usage patterns and adjust limits permanently.

CPU Throttling

Symptom: High latency, slow responses, but pods are running. CPU usage hits limit; throttling metrics spike. Diagnosis: Check kubectl top pods and Prometheus for container_cpu_cfs_throttled_seconds_total. Recovery: Raise CPU limit or optimize code.

kubectl set resources deployment frontend --limits=cpu=1 --requests=cpu=200m

Node Not Ready

Symptom: Pods on a node become unknown, Service endpoints missing. Diagnosis: kubectl get nodes shows NotReady. kubectl describe node <node> shows issues. Recovery: Drain the node if possible, or let the cluster reschedule pods. Ensure enough capacity.

kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
# Pods will be rescheduled to other nodes; update HPA if necessary.

Service Endpoints Missing

Symptom: Service has no endpoints, traffic fails. Diagnosis: kubectl get endpoints <service> shows empty list. Check selector labels and pod readiness. Recovery: Fix selector or readiness probe. Example: if selector is app: frontend but pods have app: frontend-app, update Service selector.

kubectl patch svc frontend -p '{"spec":{"selector":{"app":"frontend-app"}}}'

HPA Not Scaling

Symptom: HPA target metrics show unknown or no scaling despite high load. Diagnosis: kubectl describe hpa shows events. Common cause: metrics server not available, or missing resource requests on pods (HPA requires requests for CPU utilization). Recovery: Ensure Metrics Server is running, and pods have resource requests. Fix and reapply HPA.

9. Operations Checklist for Capacity Planning

Use this checklist before deploying a new Service or modifying capacity. Each item should have a single accountable owner and a review cadence.

Assign owners clearly and set calendar reminders for these reviews. Documentation of decisions and changes is critical; update runbooks accordingly.

10. Conclusion

Kubernetes Service capacity planning is a continuous process that combines measurement, configuration, and proactive scaling. By following the systematic approach in this guide, you can avoid common pitfalls, ensure your Services handle traffic gracefully, and recover quickly from failures. Start with a thorough environment inventory, set realistic resource requests and limits, leverage HPA and VPA appropriately, and regularly review performance data. Capacity planning is not a one-time task but an ongoing discipline that pays dividends in reliability and cost efficiency. As your traffic patterns evolve, revisit your assumptions and adjust. With the right practices and tools, you can keep your Kubernetes Services running smoothly at any scale.

Further Reading

• Kubernetes Documentation: Managing Resources for Containers
• Kubernetes Documentation: Horizontal Pod Autoscaler
• Kubernetes Documentation: Vertical Pod Autoscaler
• Prometheus and Grafana for Kubernetes monitoring
• Cloud provider-specific load balancer limits and quotas