Kubernetes Advanced Pod Configuration: Capacity Planning with Practical Examples

Intro

Capacity planning in Kubernetes is the practice of predicting how much CPU, memory, and other resources your pods will need, so the cluster can schedule them without waste or outages. Advanced pod configuration is where this gets real: you do not just set a memory limit and hope for the best. You observe actual usage, tune requests and limits, handle init containers, account for security context overhead, and plan for failure.

This article is for developers, DevOps engineers, and technical startup teams who are already familiar with basic Kubernetes concepts like pods and deployments, and now need to make informed decisions about sizing workloads. We will cover version and environment inventory, a safe configuration path, verification and diagnostics, failure modes and recovery, and a concise operations checklist.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before you touch any pod configuration for capacity planning, know what you are running. This means capturing the Kubernetes version, the container runtime, cluster topology, and existing resource usage.

Check Kubernetes version and API availability

Run:

kubectl version --short
kubectl get nodes -o wide
kubectl get pods -A -o wide

If you are using a managed Kubernetes service such as EKS, GKE, or AKS, the version is linked to the control plane; you may need to use their CLI or console to check upgrade channels. For example, on EKS:

eksctl get cluster --name my-cluster

Make sure the API resources you need are available. For resource requests and limits, they have existed since early Kubernetes versions (v1.0), but for fields like startupProbe (v1.16+) or ephemeral-storage requests (v1.8+ as alpha, stable in v1.25), you need at least that version. Check quickly with:

kubectl api-versions | grep -E 'apps/v1|v1$'

Capture current pod resource usage

Do not guess what your pods need. Use the Kubernetes metrics server. If it is not installed, install it (in most managed clusters it is a one-click add-on). Once metrics-server is running, get live usage:

kubectl top pods -n your-namespace
kubectl top nodes

Example output:

NAME                        CPU(cores)   MEMORY(bytes)
web-app-6d4f7c9b8-abcde     150m         320Mi
worker-9f7c8d6b5-xyz12      20m          180Mi

These numbers are instantaneous and are not sufficient for capacity planning. You need history. Install Prometheus and Grafana, or use a managed offering like Datadog, New Relic, or GKE Cloud Monitoring. With Prometheus, you can query typical usage:

avg_over_time(container_memory_working_set_bytes{namespace="your-namespace", container!="POD"}[7d])

This gives the average working set memory over the last 7 days, which is a more realistic baseline than the current spike.

Inspect existing pod specifications

Look at what is currently configured:

kubectl get pod <pod-name> -n <namespace> -o yaml

Look for existing resources fields. If they are missing, that is a red flag: without requests, the scheduler cannot make good decisions; without limits, a runaway pod can starve neighbors. Record all relevant configuration in a version-controlled inventory, even if it is a simple spreadsheet or a Git repository with the current manifests.

Document prerequisites and dependencies

List any cluster-level configurations that affect capacity planning:

• Node capacity and taints/tolerations: kubectl describe nodes
• Namespace resource quotas and limit ranges: kubectl get resourcequota,limitrange -n <namespace>
• Pod Security Admission (PSA) levels (if using v1.23+): kubectl get ns <namespace> -o yaml | grep pod-security
• Network policies that might affect metrics scraping
• Any custom admission webhooks that mutate pod specs

For example, a LimitRange can impose defaults or max/min values:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
spec:
  limits:
  - default:
      cpu: "1"
      memory: "1Gi"
    defaultRequest:
      cpu: "0.5"
      memory: "512Mi"
    max:
      cpu: "2"
      memory: "2Gi"
    min:
      cpu: "50m"
      memory: "100Mi"
    type: Container

If such a LimitRange exists, pods without explicit requests/limits get the defaults, and pods exceeding max are rejected. Know these before changing your pod spec.

Safe Configuration Path

Now that you have the environment inventory, you can design a safe capacity plan. The principle is: start small, observe, adjust, and always have a rollback path.

Start with resource requests and limits

Resource requests are what the scheduler uses to place pods. Limits are the hard ceiling. Setting them correctly prevents both overcommitment (too many pods on one node causing CPU throttling or OOM kills) and underutilization (wasting money).

Use the observed usage from the Version and Environment Inventory. A common starting point:

• CPU request: set to the 75th percentile of observed usage over a week. For many web services, this might be 100m to 500m.
• Memory request: set to the average working set memory plus a small buffer (e.g., 10-20%). Use container_memory_working_set_bytes, not container_memory_usage_bytes, because working set excludes page cache.
• CPU limit: set higher than request, perhaps 2-3x the request, but be aware of CPU throttling. If your application can burst, allow some headroom; if it is CPU-intensive and latency-sensitive, you might set limit equal to request to avoid throttling.
• Memory limit: set above the expected peak usage, but not so high that a leak goes unnoticed. A rule of thumb is limit = request * 1.5 or 2, plus headroom for spikes.

Example manifest for a typical web app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: app
        image: myregistry/web-app:1.4.2
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: "200m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"
        readinessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20

Note the probes: they are essential for capacity planning because a pod that is not ready should not receive traffic, and a pod that is hung but not dead can cause false capacity assumptions.

Consider Init Containers and Sidecars

Init containers run before app containers and can consume resources aggressively during startup. They can also have their own requests and limits. If your init container does database migrations or large data downloads, it may need more memory than the main app initially. Set them explicitly:

initContainers:
- name: init-db
  image: busybox:1.36
  command: ['sh', '-c', 'echo "Initializing DB"; sleep 10']
  resources:
    requests:
      cpu: "50m"
      memory: "64Mi"
    limits:
      cpu: "100m"
      memory: "128Mi"

Sidecar containers (e.g., service mesh proxies like Envoy, logging agents) are part of the pod and must be included in the total resource calculation. For a pod with an Istio sidecar, typical sidecar requests are:

resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    cpu: 500m
    memory: 256Mi

Add these to the main app requests when calculating node capacity.

Use PriorityClass and Pod Disruption Budgets

Capacity planning is not just about resources; it is about availability. Define PriorityClass for critical pods so that during node pressure, lower-priority pods are evicted first:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000
globalDefault: false
description: "For critical production services."

Then in your pod spec:

priorityClassName: high-priority

Pod Disruption Budgets (PDBs) ensure that voluntary disruptions (e.g., node drains) do not take down too many replicas:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: web-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: web-app

With 3 replicas, this allows one pod to be down at a time, so during a node drain, the cluster respects your availability needs.

Security Context and Its Effect on Capacity

Security context settings can impact capacity. For example, running a container as a non-root user with a read-only root filesystem does not directly consume CPU or memory, but if you also have allowPrivilegeEscalation: false and drop capabilities, certain applications might behave differently (more CPU due to no acceleration). More importantly, if you use hostPath volumes or device plugins (like GPU), capacity planning must account for those special resources.

Example security context that might affect GPU allocation:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 2000
  seccompProfile:
    type: RuntimeDefault
resources:
  limits:
    nvidia.com/gpu: 1

If you request a GPU, the scheduler needs to find a node with that resource. This is advanced pod configuration because you are not just planning CPU and memory.

Apply Changes Incrementally

Do not change all pods at once. Use canary deployments or simply modify one replica at a time if possible. For a deployment, you can use the rolling update strategy with maxUnavailable: 0 and maxSurge: 1 to add a new pod before terminating old ones, minimizing impact.

Before applying to production, test in a staging environment with similar resource constraints. Use kubectl apply --dry-run=client to validate syntax, and server-side dry run to check admission:

kubectl apply -f new-deployment.yaml --dry-run=server

Then apply and monitor:

kubectl apply -f new-deployment.yaml
kubectl rollout status deployment/web-app -n your-namespace

If something goes wrong, rollback:

kubectl rollout undo deployment/web-app -n your-namespace

Keep the previous manifest in version control.

Verification and Diagnostics

After applying new resource settings, verify that the pods are scheduled, running, and performing as expected. Use a systematic approach.

Check scheduling and pod status

kubectl get pods -n your-namespace -o wide

Look for Pending pods, which indicate insufficient resources or other scheduling constraints. Describe the pod for details:

kubectl describe pod <pod-name> -n your-namespace

In the events, you might see messages like:

0/3 nodes are available: 3 Insufficient memory.

That tells you your request is too high for the current nodes.

Confirm actual resource usage vs requests

After the pod runs for some time, compare actual usage to requests:

kubectl top pod <pod-name> -n your-namespace

If actual usage is consistently below requests, you may be over-provisioning and wasting capacity. If it is close to limits, you risk throttling or OOM.

Use monitoring dashboards

Set up Grafana dashboards to visualize resource usage over time. Key metrics:

• CPU usage vs request vs limit
• Memory usage (working set) vs request vs limit
• CPU throttling (container_cpu_cfs_throttled_seconds_total)
• OOM kill events (kube_pod_container_status_last_terminated_reason="OOMKilled")
• Node capacity utilization

Example Prometheus alert for memory close to limit:

- alert: PodMemoryNearLimit
  expr: (sum(container_memory_working_set_bytes{container!="POD"}) by (pod, namespace) / sum(kube_pod_container_resource_limits{resource="memory"}) by (pod, namespace)) > 0.85
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Pod {{ $labels.pod }} memory usage is above 85% of limit"

Validate Liveness and Readiness

A pod that is running but not ready is useless for capacity. Check readiness:

kubectl get pods -n your-namespace -o jsonpath='{range .items[*]}{.metadata.name}{" ready="}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}'

If a pod is not ready, debug the readiness probe. For HTTP probes, use kubectl port-forward to test the endpoint manually:

kubectl port-forward pod/<pod-name> 8080:8080 -n your-namespace
curl localhost:8080/healthz

Test Failure Scenarios

Proactively test what happens when a pod is killed or a node dies. Use kubectl delete pod <pod-name> to simulate a crash and watch the deployment recreate it. Ensure the new pod gets scheduled and becomes ready.

Simulate node pressure by cordoning a node and draining it:

kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

Observe that pods are rescheduled according to their priorities and PDBs.

Failure Modes and Recovery

Even with careful planning, failures happen. Know the common failure modes related to capacity and how to recover.

Out-of-Memory (OOM) Kills

Symptom: pod restart with reason OOMKilled. Check:

kubectl get pod <pod-name> -n your-namespace -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

This often means the memory limit is too low for the workload's peak usage. Recovery:

1. Increase the memory limit temporarily to stop the bleeding (if the app can handle it).
2. Analyze memory usage patterns to find the cause (e.g., memory leak, large batch process).
3. Fix the application if it is a leak, or adjust the request/limit to match the true peak, leaving headroom.
4. Consider using startupProbe if the app needs more time and memory during initialization.

Example patch to increase memory limit:

kubectl patch deployment web-app -n your-namespace -p '{"spec":{"template":{"spec":{"containers":[{"name":"app","resources":{"limits":{"memory":"1Gi"}}}]}}}}'

CPU Throttling

Symptom: high CPU throttling metrics, application latency spikes. Check:

kubectl top pod <pod-name> -n your-namespace

If CPU usage is at the limit, the container is being throttled. Recovery:

• Increase CPU limit, or set it equal to request to avoid throttling if your workload is latency-sensitive.
• Optimize the application to use less CPU.
• Add more replicas to distribute load.

Pending Pods Due to Insufficient Resources

If pods stay Pending, describe them to see why. Common reasons: insufficient CPU/memory, no matching node for node selectors/affinity/taints, or resource quotas exceeded.

Recovery:

• Scale up nodes (increase node count or node size).
• Lower the pod's resource requests.
• Remove or adjust nodeSelector/affinity to allow scheduling on other nodes.
• Check namespace resource quotas and either raise the quota or delete unused resources.

Example quota check:

kubectl get resourcequota -n your-namespace
kubectl describe resourcequota <quota-name> -n your-namespace

Evictions Under Node Pressure

When a node runs out of memory or disk, kubelet evicts pods. Pods with no requests or low priority are evicted first. Recovery:

• Identify evicted pods: kubectl get pods -n your-namespace --field-selector=status.phase=Failed and look for reason Evicted.
• Delete evicted pods to clean up: kubectl delete pod <pod-name> -n your-namespace
• Reduce node memory pressure by adding nodes or reducing workloads.
• Ensure critical pods have higher priority and adequate requests.

Node Failure

If a node goes down unexpectedly, pods on it become Terminating and are rescheduled if the node does not recover. However, if you have stateful workloads with local storage, data may be lost. Recovery depends on your storage: if using persistent volumes with remote storage (e.g., EBS, GCE PD), the pod can be rescheduled and reattach the volume. If using emptyDir, data is lost.

Test recovery by simulating node failure in a test environment: shut down a node or use kubectl delete node <node-name> (in a test cluster) and watch how pods are redistributed.

Misconfigured Requests/Limits Leading to Overcommit

Setting requests too low can lead to node overcommitment. When a node is overcommitted and all pods try to use their requests simultaneously, the node may run out of actual resources, causing evictions or OOM kills. Recovery:

• Increase requests to more accurately reflect real usage.
• Use LimitRange to set minimum requests in the namespace.
• Enable the Kubernetes scheduler's NodeResourcesFit plugin with appropriate scoring to reduce overcommit.

Operations Checklist

Use this checklist before and after making capacity configuration changes. Replace the example values with your own.

Conclusion

Advanced pod configuration for capacity planning in Kubernetes is not a one-time task but an ongoing practice. By following the steps outlined in this article - from environment inventory and safe configuration to verification, failure handling, and operations checklists - you can avoid the common pitfalls that lead to application outages and wasted resources.

Remember: always observe before changing, make small incremental adjustments, use concrete metrics to guide your decisions, and have a rollback plan. The commands and examples provided here are a starting point; adapt them to your specific environment and workload characteristics.

As a next step, choose one low-risk verification from the checklist, run it in a non-production namespace, and record the results. Then apply the same discipline to your production workloads, one at a time. With careful capacity planning, your Kubernetes clusters will be more stable, cost-efficient, and ready for growth.