## Intro Capacity planning in Kubernetes often starts with a problem: a pod is evicted, a node runs out of memory, or a deployment cannot schedule. Teams then rush to increase limits or scale replicas without knowing the actual resource usage. The result is wasted cloud spend or repeated incidents. This article provides a practical, command-driven workflow for kubectl capacity planning. It covers how to inspect current usage, set resource requests and limits, choose scaling strategies, verify changes, and recover when things go wrong. Every section includes concrete commands with expected output and failure signals. The target audience is developers, DevOps engineers, and SREs who manage Kubernetes workloads and want to move from reactive guesswork to proactive, evidence-based capacity decisions. ## Version and Environment Inventory Before changing anything, know your environment. Check kubectl version and cluster connectivity. ```bash kubectl version --short ``` Example output: ``` Client Version: v1.27.2 Kustomize Version: v5.0.1 Server Version: v1.26.5 ``` If client and server major.minor differ by more than one version, be cautious with new API fields. Next, confirm which context and namespace you are targeting. ```bash kubectl config current-context kubectl get namespaces ``` This prevents applying changes to the wrong cluster or namespace. For capacity planning, relevant components are Deployments, StatefulSets, DaemonSets, and Pods. Identify those with uncertain resource settings. ```bash kubectl get pods -o wide --all-namespaces ``` The output shows node placement, status, and restarts. Pods that are Pending or have many restarts likely have capacity issues. Practical check: run `kubectl describe pod -n ` and look for events like `FailedScheduling` with messages such as `0/3 nodes are available: insufficient memory`. That tells you the current request exceeds available node capacity. Record this before making any change. ## Safe Configuration Path Resource requests and limits are the foundation of capacity planning. Requests define the guaranteed amount of CPU and memory a pod can use, while limits cap maximum usage. Without them, the scheduler cannot make informed decisions, and a noisy neighbor can consume all node resources. Set requests based on observed usage, not guesses. The most reliable way is to use the Vertical Pod Autoscaler (VPA) in recommendation mode or inspect metrics. First, check current usage: ```bash kubectl top pods -n ``` Example output: ``` NAME CPU(cores) MEMORY(bytes) nginx-6c8b449f8f-g9f8q 5m 61Mi redis-68b8d9d5c9-5j4kp 8m 5Mi ``` If `kubectl top` fails with `Error from server (NotFound): the server could not find the requested resource`, metrics-server is not installed. Install it or use cloud provider monitoring. For a typical web application using around 50m CPU and 120Mi memory at normal load, set requests with a small buffer to cover startup spikes: 100m CPU and 128Mi memory. Limits can be set higher, say 500m CPU and 512Mi memory, to allow bursts while preventing runaway consumption. Example Deployment snippet: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: web-app spec: replicas: 3 selector: matchLabels: app: web-app template: metadata: labels: app: web-app spec: containers: - name: app image: nginx:1.25 resources: requests: cpu: "100m" memory: "128Mi" limits: cpu: "500m" memory: "512Mi" ``` Apply it with: ```bash kubectl apply -f deployment.yaml -n ``` After modifying resources, perform a rolling restart if necessary. Some changes require pod recreation. ```bash kubectl rollout restart deployment/web-app -n ``` Check rollout status: ```bash kubectl rollout status deployment/web-app -n ``` Expected output: ``` deployment "web-app" successfully rolled out ``` If the rollout hangs, pods may be unable to schedule with the new requests. Check events and adjust. ## Verification and Diagnostics After applying resource changes or scaling, verify that pods are running and healthy. Use the standard commands: ```bash kubectl get pods -o wide -n ``` Look for status `Running` and `READY` columns showing `1/1`. For deeper inspection: ```bash kubectl describe pod -n ``` Key sections in describe output: - `Conditions`: `Ready` should be `True` - `Containers.State`: should show `Running` without restart count increasing - `Events`: warnings about resource issues, e.g., `OOMKilled` if limit was too low To see actual resource usage after a change: ```bash kubectl top pods -n ``` Compare usage with requests and limits. If usage is consistently near limits, consider raising limits or optimizing the application. If usage is far below requests, you are over-provisioning and wasting capacity. For application-level verification, use `kubectl port-forward` to access a pod locally: ```bash kubectl port-forward pod/ 8080:80 -n ``` Then open `http://localhost:8080` in a browser or use curl. If you changed the number of replicas, verify load distribution: ```bash kubectl get endpoints -n ``` Expected output includes IP addresses of all ready pods. If some pods are missing from endpoints, they are not ready and will not receive traffic. ## Failure Modes and Recovery Capacity planning failures typically fall into these categories: ### OOMKilled (Memory Limit Too Low) When a container uses more memory than its limit, the kernel kills the process with an OOMKilled exit code. Check pod status: ```bash kubectl get pods -n ``` Output shows `RESTARTS` count increasing and `STATUS` as `CrashLoopBackOff`. Describe the pod: ```bash kubectl describe pod -n ``` Look for: ``` Last State: Terminated Reason: OOMKilled Exit Code: 137 ``` Recovery: Increase memory limit to at least 20-30% above observed peak usage. Also check if the application has a memory leak; merely raising limits may postpone the issue. ### CPU Throttling If CPU usage reaches the limit, the kernel throttles the container, causing increased latency but not termination. Check container metrics: ```bash kubectl top pods -n ``` If CPU usage is at or near limit, inspect throttling with `kubectl describe` under `Conditions` or use metrics-server with additional tooling. Recovery: Increase CPU limit or, if the workload is bursty, consider setting limit higher than request to allow bursts. ### Eviction Due to Node Pressure If a node runs out of memory, the kubelet evicts pods based on their request/limit and priority. Evicted pods show status `Failed` with reason `Evicted`. Recovery may involve rescheduling or adding capacity. Check node status: ```bash kubectl describe node ``` Look for `MemoryPressure` condition set to `True`. Then consider reducing memory requests, moving pods to other nodes, or scaling the cluster. ### Scheduling Failure New pods may not schedule if requests exceed available node resources. Events show `FailedScheduling` with `insufficient cpu` or `insufficient memory`. Recovery: reduce requests if overestimated, or add nodes / enable cluster autoscaling. ## Operations Checklist Use this checklist to perform capacity planning changes safely. For each item, we name an accountable owner and review cadence. | # | Check | Command / Action | Expected Result | Owner | Review Frequency | |---|-------|------------------|-----------------|-------|------------------| | 1 | Confirm context and namespace | `kubectl config current-context` and `kubectl get ns` | Correct cluster and namespace | Platform Engineer | Every change | | 2 | Baseline current usage | `kubectl top pods -n ` | CPU/memory usage per pod | DevOps Lead | Weekly | | 3 | Identify pods without requests/limits | `kubectl get pods -o json \| jq '.items[] | select(.spec.containers[].resources.requests == null) \| .metadata.name'` | List of non-compliant pods | SRE | Monthly | | 4 | Set requests/limits in manifests | Edit YAML and apply | Pods run with defined resources | Application Owner | Per release | | 5 | Verify rollout | `kubectl rollout status deployment/ -n ` | `successfully rolled out` | Release Manager | Each deploy | | 6 | Check actual vs requested | `kubectl top pods` vs configured requests | Usage reasonable | DevOps Lead | Weekly | | 7 | Test failure recovery | Intentional scale down and up | Pods reschedule successfully | Platform Engineer | Quarterly | ## Common Pitfalls and How to Avoid Them ### Using Requests Equal to Limits Many teams set `requests = limits` for simplicity. This guarantees quality of service but prevents CPU bursting and may waste allocated memory since memory requests are counted against node capacity even if unused. For memory, overcommit is allowed, but too much overcommit risks evictions. For CPU, allow a higher limit than request to enable bursts when needed, but be aware of throttling. ### Setting Limits Without Historical Data Guessing resource needs often leads to OOMKills or wasted capacity. Use monitoring data over at least two weeks, covering peak and off-peak usage, before setting values. Tools like Prometheus and Grafana can help visualize trends. ### Not Considering Startup Overhead Many applications use more resources during startup (e.g., loading caches). Set requests based on steady-state usage but ensure limits can accommodate the startup burst, or use a `startupProbe` to delay readiness until after the burst. ### Ignoring the Impact of Replicas on Node Capacity Each replica consumes its requests from node resources. Increasing replicas without checking node capacity can cause scheduling failures. Use `kubectl describe nodes` to see allocated resources and compare with capacity. ### Relying Solely on Horizontal Scaling Horizontal Pod Autoscaler (HPA) is useful but only works if pods have requests set properly. HPA also responds slowly compared to traffic spikes. Combine HPA with reasonable resource buffers and, if needed, Vertical Pod Autoscaler in recommendation mode. ## Conclusion Effective kubectl capacity planning is not about guessing limits; it is about observing actual usage, setting evidence-based requests and limits, verifying changes, and planning for recovery. The workflow is: 1. Inventory the environment and current state. 2. Baseline usage with `kubectl top`. 3. Adjust requests and limits in small, documented increments. 4. Verify rollout with `kubectl rollout status` and `kubectl get pods`. 5. Monitor for OOM kills, throttling, and evictions. 6. Revisit regularly as traffic patterns change. By following these practices, you can avoid common pitfalls, reduce cloud costs, and maintain application reliability. Start small: choose one deployment, apply the checklist, and iterate. Your future self, and your cluster, will thank you.