Kubectl Capacity Planning with Practical Examples

Intro

Capacity planning in Kubernetes often starts with a problem: a pod is evicted, a node runs out of memory, or a deployment cannot schedule. Teams then rush to increase limits or scale replicas without knowing the actual resource usage. The result is wasted cloud spend or repeated incidents.

This article provides a practical, command-driven workflow for kubectl capacity planning. It covers how to inspect current usage, set resource requests and limits, choose scaling strategies, verify changes, and recover when things go wrong. Every section includes concrete commands with expected output and failure signals.

The target audience is developers, DevOps engineers, and SREs who manage Kubernetes workloads and want to move from reactive guesswork to proactive, evidence-based capacity decisions.

Version and Environment Inventory

Before changing anything, know your environment. Check kubectl version and cluster connectivity.

kubectl version --short

Example output:

Client Version: v1.27.2
Kustomize Version: v5.0.1
Server Version: v1.26.5

If client and server major.minor differ by more than one version, be cautious with new API fields. Next, confirm which context and namespace you are targeting.

kubectl config current-context
kubectl get namespaces

This prevents applying changes to the wrong cluster or namespace. For capacity planning, relevant components are Deployments, StatefulSets, DaemonSets, and Pods. Identify those with uncertain resource settings.

kubectl get pods -o wide --all-namespaces

The output shows node placement, status, and restarts. Pods that are Pending or have many restarts likely have capacity issues.

Practical check: run kubectl describe pod <name> -n <namespace> and look for events like FailedScheduling with messages such as 0/3 nodes are available: insufficient memory. That tells you the current request exceeds available node capacity. Record this before making any change.

Safe Configuration Path

Resource requests and limits are the foundation of capacity planning. Requests define the guaranteed amount of CPU and memory a pod can use, while limits cap maximum usage. Without them, the scheduler cannot make informed decisions, and a noisy neighbor can consume all node resources.

Set requests based on observed usage, not guesses. The most reliable way is to use the Vertical Pod Autoscaler (VPA) in recommendation mode or inspect metrics. First, check current usage:

kubectl top pods -n <namespace>

Example output:

NAME                       CPU(cores)   MEMORY(bytes)
nginx-6c8b449f8f-g9f8q     5m           61Mi
redis-68b8d9d5c9-5j4kp     8m           5Mi

If kubectl top fails with Error from server (NotFound): the server could not find the requested resource, metrics-server is not installed. Install it or use cloud provider monitoring.

For a typical web application using around 50m CPU and 120Mi memory at normal load, set requests with a small buffer to cover startup spikes: 100m CPU and 128Mi memory. Limits can be set higher, say 500m CPU and 512Mi memory, to allow bursts while preventing runaway consumption.

Example Deployment snippet:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: app
        image: nginx:1.25
        resources:
          requests:
            cpu: "100m"
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

Apply it with:

kubectl apply -f deployment.yaml -n <namespace>

After modifying resources, perform a rolling restart if necessary. Some changes require pod recreation.

kubectl rollout restart deployment/web-app -n <namespace>

Check rollout status:

kubectl rollout status deployment/web-app -n <namespace>

Expected output:

deployment "web-app" successfully rolled out

If the rollout hangs, pods may be unable to schedule with the new requests. Check events and adjust.

Verification and Diagnostics

After applying resource changes or scaling, verify that pods are running and healthy. Use the standard commands:

kubectl get pods -o wide -n <namespace>

Look for status Running and READY columns showing 1/1. For deeper inspection:

kubectl describe pod <pod-name> -n <namespace>

Key sections in describe output:

• Conditions: Ready should be True
• Containers.State: should show Running without restart count increasing
• Events: warnings about resource issues, e.g., OOMKilled if limit was too low

To see actual resource usage after a change:

kubectl top pods -n <namespace>

Compare usage with requests and limits. If usage is consistently near limits, consider raising limits or optimizing the application. If usage is far below requests, you are over-provisioning and wasting capacity.

For application-level verification, use kubectl port-forward to access a pod locally:

kubectl port-forward pod/<pod-name> 8080:80 -n <namespace>

Then open http://localhost:8080 in a browser or use curl.

If you changed the number of replicas, verify load distribution:

kubectl get endpoints <service-name> -n <namespace>

Expected output includes IP addresses of all ready pods. If some pods are missing from endpoints, they are not ready and will not receive traffic.

Failure Modes and Recovery

Capacity planning failures typically fall into these categories:

OOMKilled (Memory Limit Too Low)

When a container uses more memory than its limit, the kernel kills the process with an OOMKilled exit code. Check pod status:

kubectl get pods -n <namespace>

Output shows RESTARTS count increasing and STATUS as CrashLoopBackOff. Describe the pod:

kubectl describe pod <pod-name> -n <namespace>

Look for:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Recovery: Increase memory limit to at least 20-30% above observed peak usage. Also check if the application has a memory leak; merely raising limits may postpone the issue.

CPU Throttling

If CPU usage reaches the limit, the kernel throttles the container, causing increased latency but not termination. Check container metrics:

kubectl top pods -n <namespace>

If CPU usage is at or near limit, inspect throttling with kubectl describe under Conditions or use metrics-server with additional tooling. Recovery: Increase CPU limit or, if the workload is bursty, consider setting limit higher than request to allow bursts.

Eviction Due to Node Pressure

If a node runs out of memory, the kubelet evicts pods based on their request/limit and priority. Evicted pods show status Failed with reason Evicted. Recovery may involve rescheduling or adding capacity. Check node status:

kubectl describe node <node-name>

Look for MemoryPressure condition set to True. Then consider reducing memory requests, moving pods to other nodes, or scaling the cluster.

Scheduling Failure

New pods may not schedule if requests exceed available node resources. Events show FailedScheduling with insufficient cpu or insufficient memory. Recovery: reduce requests if overestimated, or add nodes / enable cluster autoscaling.

Operations Checklist

Use this checklist to perform capacity planning changes safely. For each item, we name an accountable owner and review cadence.

Common Pitfalls and How to Avoid Them

Using Requests Equal to Limits

Many teams set requests = limits for simplicity. This guarantees quality of service but prevents CPU bursting and may waste allocated memory since memory requests are counted against node capacity even if unused. For memory, overcommit is allowed, but too much overcommit risks evictions. For CPU, allow a higher limit than request to enable bursts when needed, but be aware of throttling.

Setting Limits Without Historical Data

Guessing resource needs often leads to OOMKills or wasted capacity. Use monitoring data over at least two weeks, covering peak and off-peak usage, before setting values. Tools like Prometheus and Grafana can help visualize trends.

Not Considering Startup Overhead

Many applications use more resources during startup (e.g., loading caches). Set requests based on steady-state usage but ensure limits can accommodate the startup burst, or use a startupProbe to delay readiness until after the burst.

Ignoring the Impact of Replicas on Node Capacity

Each replica consumes its requests from node resources. Increasing replicas without checking node capacity can cause scheduling failures. Use kubectl describe nodes to see allocated resources and compare with capacity.

Relying Solely on Horizontal Scaling

Horizontal Pod Autoscaler (HPA) is useful but only works if pods have requests set properly. HPA also responds slowly compared to traffic spikes. Combine HPA with reasonable resource buffers and, if needed, Vertical Pod Autoscaler in recommendation mode.

Conclusion

Effective kubectl capacity planning is not about guessing limits; it is about observing actual usage, setting evidence-based requests and limits, verifying changes, and planning for recovery. The workflow is:

1. Inventory the environment and current state.
2. Baseline usage with kubectl top.
3. Adjust requests and limits in small, documented increments.
4. Verify rollout with kubectl rollout status and kubectl get pods.
5. Monitor for OOM kills, throttling, and evictions.
6. Revisit regularly as traffic patterns change.

By following these practices, you can avoid common pitfalls, reduce cloud costs, and maintain application reliability. Start small: choose one deployment, apply the checklist, and iterate. Your future self, and your cluster, will thank you.