## Intro Capacity planning in Kubernetes is the practice of predicting how much CPU, memory, and other resources your pods will need, so the cluster can schedule them without waste or outages. Advanced pod configuration is where this gets real: you do not just set a memory limit and hope for the best. You observe actual usage, tune requests and limits, handle init containers, account for security context overhead, and plan for failure. This article is for developers, DevOps engineers, and technical startup teams who are already familiar with basic Kubernetes concepts like pods and deployments, and now need to make informed decisions about sizing workloads. We will cover version and environment inventory, a safe configuration path, verification and diagnostics, failure modes and recovery, and a concise operations checklist. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before you touch any pod configuration for capacity planning, know what you are running. This means capturing the Kubernetes version, the container runtime, cluster topology, and existing resource usage. ### Check Kubernetes version and API availability Run: ```bash kubectl version --short kubectl get nodes -o wide kubectl get pods -A -o wide ``` If you are using a managed Kubernetes service such as EKS, GKE, or AKS, the version is linked to the control plane; you may need to use their CLI or console to check upgrade channels. For example, on EKS: ```bash eksctl get cluster --name my-cluster ``` Make sure the API resources you need are available. For resource requests and limits, they have existed since early Kubernetes versions (v1.0), but for fields like `startupProbe` (v1.16+) or `ephemeral-storage` requests (v1.8+ as alpha, stable in v1.25), you need at least that version. Check quickly with: ```bash kubectl api-versions | grep -E 'apps/v1|v1$' ``` ### Capture current pod resource usage Do not guess what your pods need. Use the Kubernetes metrics server. If it is not installed, install it (in most managed clusters it is a one-click add-on). Once metrics-server is running, get live usage: ```bash kubectl top pods -n your-namespace kubectl top nodes ``` Example output: ``` NAME CPU(cores) MEMORY(bytes) web-app-6d4f7c9b8-abcde 150m 320Mi worker-9f7c8d6b5-xyz12 20m 180Mi ``` These numbers are instantaneous and are not sufficient for capacity planning. You need history. Install Prometheus and Grafana, or use a managed offering like Datadog, New Relic, or GKE Cloud Monitoring. With Prometheus, you can query typical usage: ```promql avg_over_time(container_memory_working_set_bytes{namespace="your-namespace", container!="POD"}[7d]) ``` This gives the average working set memory over the last 7 days, which is a more realistic baseline than the current spike. ### Inspect existing pod specifications Look at what is currently configured: ```bash kubectl get pod -n -o yaml ``` Look for existing `resources` fields. If they are missing, that is a red flag: without requests, the scheduler cannot make good decisions; without limits, a runaway pod can starve neighbors. Record all relevant configuration in a version-controlled inventory, even if it is a simple spreadsheet or a Git repository with the current manifests. ### Document prerequisites and dependencies List any cluster-level configurations that affect capacity planning: - Node capacity and taints/tolerations: `kubectl describe nodes` - Namespace resource quotas and limit ranges: `kubectl get resourcequota,limitrange -n ` - Pod Security Admission (PSA) levels (if using v1.23+): `kubectl get ns -o yaml | grep pod-security` - Network policies that might affect metrics scraping - Any custom admission webhooks that mutate pod specs For example, a LimitRange can impose defaults or max/min values: ```yaml apiVersion: v1 kind: LimitRange metadata: name: default-limits spec: limits: - default: cpu: "1" memory: "1Gi" defaultRequest: cpu: "0.5" memory: "512Mi" max: cpu: "2" memory: "2Gi" min: cpu: "50m" memory: "100Mi" type: Container ``` If such a LimitRange exists, pods without explicit requests/limits get the defaults, and pods exceeding max are rejected. Know these before changing your pod spec. ## Safe Configuration Path Now that you have the environment inventory, you can design a safe capacity plan. The principle is: start small, observe, adjust, and always have a rollback path. ### Start with resource requests and limits Resource requests are what the scheduler uses to place pods. Limits are the hard ceiling. Setting them correctly prevents both overcommitment (too many pods on one node causing CPU throttling or OOM kills) and underutilization (wasting money). Use the observed usage from the Version and Environment Inventory. A common starting point: - **CPU request**: set to the 75th percentile of observed usage over a week. For many web services, this might be 100m to 500m. - **Memory request**: set to the average working set memory plus a small buffer (e.g., 10-20%). Use `container_memory_working_set_bytes`, not `container_memory_usage_bytes`, because working set excludes page cache. - **CPU limit**: set higher than request, perhaps 2-3x the request, but be aware of CPU throttling. If your application can burst, allow some headroom; if it is CPU-intensive and latency-sensitive, you might set limit equal to request to avoid throttling. - **Memory limit**: set above the expected peak usage, but not so high that a leak goes unnoticed. A rule of thumb is limit = request * 1.5 or 2, plus headroom for spikes. Example manifest for a typical web app: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: web-app spec: replicas: 3 selector: matchLabels: app: web-app template: metadata: labels: app: web-app spec: containers: - name: app image: myregistry/web-app:1.4.2 ports: - containerPort: 8080 resources: requests: cpu: "200m" memory: "256Mi" limits: cpu: "500m" memory: "512Mi" readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 5 periodSeconds: 10 livenessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 15 periodSeconds: 20 ``` Note the probes: they are essential for capacity planning because a pod that is not ready should not receive traffic, and a pod that is hung but not dead can cause false capacity assumptions. ### Consider Init Containers and Sidecars Init containers run before app containers and can consume resources aggressively during startup. They can also have their own requests and limits. If your init container does database migrations or large data downloads, it may need more memory than the main app initially. Set them explicitly: ```yaml initContainers: - name: init-db image: busybox:1.36 command: ['sh', '-c', 'echo "Initializing DB"; sleep 10'] resources: requests: cpu: "50m" memory: "64Mi" limits: cpu: "100m" memory: "128Mi" ``` Sidecar containers (e.g., service mesh proxies like Envoy, logging agents) are part of the pod and must be included in the total resource calculation. For a pod with an Istio sidecar, typical sidecar requests are: ```yaml resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 256Mi ``` Add these to the main app requests when calculating node capacity. ### Use PriorityClass and Pod Disruption Budgets Capacity planning is not just about resources; it is about availability. Define PriorityClass for critical pods so that during node pressure, lower-priority pods are evicted first: ```yaml apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: high-priority value: 1000 globalDefault: false description: "For critical production services." ``` Then in your pod spec: ```yaml priorityClassName: high-priority ``` Pod Disruption Budgets (PDBs) ensure that voluntary disruptions (e.g., node drains) do not take down too many replicas: ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: web-app-pdb spec: minAvailable: 2 selector: matchLabels: app: web-app ``` With 3 replicas, this allows one pod to be down at a time, so during a node drain, the cluster respects your availability needs. ### Security Context and Its Effect on Capacity Security context settings can impact capacity. For example, running a container as a non-root user with a read-only root filesystem does not directly consume CPU or memory, but if you also have `allowPrivilegeEscalation: false` and drop capabilities, certain applications might behave differently (more CPU due to no acceleration). More importantly, if you use hostPath volumes or device plugins (like GPU), capacity planning must account for those special resources. Example security context that might affect GPU allocation: ```yaml securityContext: runAsNonRoot: true runAsUser: 1000 fsGroup: 2000 seccompProfile: type: RuntimeDefault resources: limits: nvidia.com/gpu: 1 ``` If you request a GPU, the scheduler needs to find a node with that resource. This is advanced pod configuration because you are not just planning CPU and memory. ### Apply Changes Incrementally Do not change all pods at once. Use canary deployments or simply modify one replica at a time if possible. For a deployment, you can use the rolling update strategy with `maxUnavailable: 0` and `maxSurge: 1` to add a new pod before terminating old ones, minimizing impact. Before applying to production, test in a staging environment with similar resource constraints. Use `kubectl apply --dry-run=client` to validate syntax, and server-side dry run to check admission: ```bash kubectl apply -f new-deployment.yaml --dry-run=server ``` Then apply and monitor: ```bash kubectl apply -f new-deployment.yaml kubectl rollout status deployment/web-app -n your-namespace ``` If something goes wrong, rollback: ```bash kubectl rollout undo deployment/web-app -n your-namespace ``` Keep the previous manifest in version control. ## Verification and Diagnostics After applying new resource settings, verify that the pods are scheduled, running, and performing as expected. Use a systematic approach. ### Check scheduling and pod status ```bash kubectl get pods -n your-namespace -o wide ``` Look for `Pending` pods, which indicate insufficient resources or other scheduling constraints. Describe the pod for details: ```bash kubectl describe pod -n your-namespace ``` In the events, you might see messages like: ``` 0/3 nodes are available: 3 Insufficient memory. ``` That tells you your request is too high for the current nodes. ### Confirm actual resource usage vs requests After the pod runs for some time, compare actual usage to requests: ```bash kubectl top pod -n your-namespace ``` If actual usage is consistently below requests, you may be over-provisioning and wasting capacity. If it is close to limits, you risk throttling or OOM. ### Use monitoring dashboards Set up Grafana dashboards to visualize resource usage over time. Key metrics: - CPU usage vs request vs limit - Memory usage (working set) vs request vs limit - CPU throttling (`container_cpu_cfs_throttled_seconds_total`) - OOM kill events (`kube_pod_container_status_last_terminated_reason="OOMKilled"`) - Node capacity utilization Example Prometheus alert for memory close to limit: ```yaml - alert: PodMemoryNearLimit expr: (sum(container_memory_working_set_bytes{container!="POD"}) by (pod, namespace) / sum(kube_pod_container_resource_limits{resource="memory"}) by (pod, namespace)) > 0.85 for: 10m labels: severity: warning annotations: summary: "Pod {{ $labels.pod }} memory usage is above 85% of limit" ``` ### Validate Liveness and Readiness A pod that is running but not ready is useless for capacity. Check readiness: ```bash kubectl get pods -n your-namespace -o jsonpath='{range .items[*]}{.metadata.name}{" ready="}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' ``` If a pod is not ready, debug the readiness probe. For HTTP probes, use `kubectl port-forward` to test the endpoint manually: ```bash kubectl port-forward pod/ 8080:8080 -n your-namespace curl localhost:8080/healthz ``` ### Test Failure Scenarios Proactively test what happens when a pod is killed or a node dies. Use `kubectl delete pod ` to simulate a crash and watch the deployment recreate it. Ensure the new pod gets scheduled and becomes ready. Simulate node pressure by cordoning a node and draining it: ```bash kubectl cordon kubectl drain --ignore-daemonsets --delete-emptydir-data ``` Observe that pods are rescheduled according to their priorities and PDBs. ## Failure Modes and Recovery Even with careful planning, failures happen. Know the common failure modes related to capacity and how to recover. ### Out-of-Memory (OOM) Kills Symptom: pod restart with reason `OOMKilled`. Check: ```bash kubectl get pod -n your-namespace -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' ``` This often means the memory limit is too low for the workload's peak usage. Recovery: 1. Increase the memory limit temporarily to stop the bleeding (if the app can handle it). 2. Analyze memory usage patterns to find the cause (e.g., memory leak, large batch process). 3. Fix the application if it is a leak, or adjust the request/limit to match the true peak, leaving headroom. 4. Consider using `startupProbe` if the app needs more time and memory during initialization. Example patch to increase memory limit: ```bash kubectl patch deployment web-app -n your-namespace -p '{"spec":{"template":{"spec":{"containers":[{"name":"app","resources":{"limits":{"memory":"1Gi"}}}]}}}}' ``` ### CPU Throttling Symptom: high CPU throttling metrics, application latency spikes. Check: ```bash kubectl top pod -n your-namespace ``` If CPU usage is at the limit, the container is being throttled. Recovery: - Increase CPU limit, or set it equal to request to avoid throttling if your workload is latency-sensitive. - Optimize the application to use less CPU. - Add more replicas to distribute load. ### Pending Pods Due to Insufficient Resources If pods stay `Pending`, describe them to see why. Common reasons: insufficient CPU/memory, no matching node for node selectors/affinity/taints, or resource quotas exceeded. Recovery: - Scale up nodes (increase node count or node size). - Lower the pod's resource requests. - Remove or adjust nodeSelector/affinity to allow scheduling on other nodes. - Check namespace resource quotas and either raise the quota or delete unused resources. Example quota check: ```bash kubectl get resourcequota -n your-namespace kubectl describe resourcequota -n your-namespace ``` ### Evictions Under Node Pressure When a node runs out of memory or disk, kubelet evicts pods. Pods with no requests or low priority are evicted first. Recovery: - Identify evicted pods: `kubectl get pods -n your-namespace --field-selector=status.phase=Failed` and look for reason `Evicted`. - Delete evicted pods to clean up: `kubectl delete pod -n your-namespace` - Reduce node memory pressure by adding nodes or reducing workloads. - Ensure critical pods have higher priority and adequate requests. ### Node Failure If a node goes down unexpectedly, pods on it become `Terminating` and are rescheduled if the node does not recover. However, if you have stateful workloads with local storage, data may be lost. Recovery depends on your storage: if using persistent volumes with remote storage (e.g., EBS, GCE PD), the pod can be rescheduled and reattach the volume. If using `emptyDir`, data is lost. Test recovery by simulating node failure in a test environment: shut down a node or use `kubectl delete node ` (in a test cluster) and watch how pods are redistributed. ### Misconfigured Requests/Limits Leading to Overcommit Setting requests too low can lead to node overcommitment. When a node is overcommitted and all pods try to use their requests simultaneously, the node may run out of actual resources, causing evictions or OOM kills. Recovery: - Increase requests to more accurately reflect real usage. - Use LimitRange to set minimum requests in the namespace. - Enable the Kubernetes scheduler's `NodeResourcesFit` plugin with appropriate scoring to reduce overcommit. ## Operations Checklist Use this checklist before and after making capacity configuration changes. Replace the example values with your own. | Step | Command or Action | Expected Result | Owner | |------|-------------------|-----------------|-------| | 1. Capture baseline | `kubectl top pods -n prod` and note peak usage over 7 days | A table of current CPU/memory usage | Priya Shah, SRE | | 2. Check current pod specs | `kubectl get pods -n prod -o yaml` and review `resources` | Documented current requests/limits | Priya Shah | | 3. Verify metrics server | `kubectl get deployment metrics-server -n kube-system` | Metrics server is running | Marcus Chen | | 4. Set new resource requests/limits in manifest | Edit deployment YAML with new values, e.g., request CPU 200m, memory 256Mi; limit CPU 500m, memory 512Mi | Manifest updated and version-controlled | Marcus Chen | | 5. Dry-run apply | `kubectl apply -f new-deployment.yaml --dry-run=server` | No errors, validation passed | Priya Shah | | 6. Apply change | `kubectl apply -f new-deployment.yaml` | Deployment updated, rollout starts | Priya Shah | | 7. Monitor rollout | `kubectl rollout status deployment/web-app -n prod` | "deployment successfully rolled out" | Marcus Chen | | 8. Check pod resource usage after 30 min | `kubectl top pods -n prod --sort-by=memory` | New pods usage within limits | Priya Shah | | 9. Review throttling/OOM metrics | Prometheus: `rate(container_cpu_cfs_throttled_seconds_total[5m])` and OOM kill events | No significant throttling or OOM kills | Marcus Chen | | 10. Test resilience | `kubectl delete pod web-app- -n prod` and observe recreation | New pod is scheduled and becomes ready | Priya Shah | | 11. Document recovery | Update runbook with rollback commands | Runbook has `kubectl rollout undo` steps | Marcus Chen | ## Conclusion Advanced pod configuration for capacity planning in Kubernetes is not a one-time task but an ongoing practice. By following the steps outlined in this article - from environment inventory and safe configuration to verification, failure handling, and operations checklists - you can avoid the common pitfalls that lead to application outages and wasted resources. Remember: always observe before changing, make small incremental adjustments, use concrete metrics to guide your decisions, and have a rollback plan. The commands and examples provided here are a starting point; adapt them to your specific environment and workload characteristics. As a next step, choose one low-risk verification from the checklist, run it in a non-production namespace, and record the results. Then apply the same discipline to your production workloads, one at a time. With careful capacity planning, your Kubernetes clusters will be more stable, cost-efficient, and ready for growth.