Kubernetes Debug Running Pod Performance: A Practical Operations Guide

Intro

When a Kubernetes pod is already running and users report slowness, timeouts, or intermittent failures, you need a systematic approach that moves from observed symptom to verified fix without causing new incidents. Debugging a running pod is different from debugging a deployment that fails to start: the pod is serving traffic, so every action must minimize disruption and preserve evidence.

This practical guide walks through the process of tuning performance for a live pod. It covers environment inventory, safe configuration changes, targeted diagnostics, common failure modes, and an operations checklist. Along the way, you will use real commands such as kubectl top, kubectl exec, and kubectl logs, interpret their output, and make changes with controlled blast radius. The focus is on Linux containers running in Kubernetes 1.28+ with kubectl 1.28 or later, but the principles apply to any recent version.

The goal is operational safety: observe before changing, limit the scope of any intervention, verify results with metrics, and document rollback paths. You will learn how to diagnose CPU throttling, memory pressure, disk I/O bottlenecks, and network issues without restarting a pod unnecessarily.

Version and Environment Inventory

Before touching a running pod, establish the exact versions and cluster state. This prevents false assumptions and ensures any tuning advice matches your environment.

Run the following read-only commands:

kubectl version --short
kubectl get nodes -o wide
kubectl describe node <your-node-name>

Expected output includes Kubernetes server version (e.g., Server Version: v1.28.7), node kernel version, container runtime (e.g., containerd://1.7.11), and operating system. Record these in your incident notes. If the node shows high %CPU or %MEM, treat the node as a candidate bottleneck before blaming the pod.

Next, inventory the pod itself:

kubectl get pod <pod-name> -n <namespace> -o yaml

Look for the pod's resource requests and limits under spec.containers[].resources. Example snippet:

resources:
  requests:
    memory: "128Mi"
    cpu: "250m"
  limits:
    memory: "256Mi"
    cpu: "500m"

If requests are missing, the pod may be starved or overcommitted. If limits equal requests, the QoS class is Guaranteed, which gives higher priority; otherwise it is Burstable or BestEffort. This matters during node pressure.

Prerequisites for debugging:

• You have kubectl access with get, describe, logs, exec, and top permissions.
• You can read metrics-server output. If kubectl top reports error: Metrics API not available, install or enable metrics-server before relying on CPU/memory metrics.
• You have a way to generate test traffic, such as hey, wrk, or curl in a loop.

Keep changes local: reproduce the issue in a staging namespace with a copied pod spec before altering production. Use kubectl port-forward to test service endpoints without exposing them externally.

Safe Configuration Path

Changing a running pod's configuration is risky. Kubernetes does not allow modifying most pod fields directly; you must either update the Deployment or terminate and recreate the pod. For performance tuning, you often adjust resource limits, environment variables that affect runtime behavior, or annotations. Always follow a safe sequence:

1. Observe current behavior for at least five minutes with load.
2. Define one hypothesis (e.g., "CPU limit is too low causing throttling").
3. Make the smallest change to address that hypothesis.
4. Compare before and after metrics.
5. Roll back if no improvement or regression occurs.

Example: Increase CPU limit due to throttling

Check current throttling:

kubectl exec <pod-name> -- cat /sys/fs/cgroup/cpu/cpu.stat

Look for nr_throttled and throttled_time. If throttled_time increases steadily under load, the pod is hitting its CPU limit.

Proposed change in the Deployment manifest:

resources:
  limits:
    cpu: "1000m"   # increased from 500m
  requests:
    cpu: "500m"

Apply with:

kubectl apply -f updated-deployment.yaml
kubectl rollout status deployment/<deployment-name> -n <namespace>

After rollout, generate the same load and compare nr_throttled. If throttling drops to zero or near zero, the change worked. If not, consider other bottlenecks.

Example: Adjust garbage collection environment variable

Many runtimes (Java, Node.js) have environment variables that affect memory usage. Suppose a Node.js application occasionally exceeds memory limits and gets OOMKilled. You can set NODE_OPTIONS=--max-old-space-size=512 to limit heap. First, test locally,

then update the Deployment:

env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=512"

Apply and verify the pod restarts with the new environment using kubectl describe pod and check the environment section.

Always keep a copy of the previous manifest for rollback:

kubectl rollout undo deployment/<deployment-name>

Verification and Diagnostics

Diagnostics should be layered: start with cluster-level metrics, then pod-level, then process-level inside the container.

Cluster and pod resource usage

Use kubectl top to get current usage:

kubectl top node
kubectl top pod -n <namespace>

Example output:

NAME          CPU(cores)   MEMORY(bytes)
my-app-pod    720m         289Mi

If the pod consistently uses more than 80% of its limit, consider raising the limit. If it uses less than 20% of its request, you may overcommit and cause node pressure elsewhere.

Application logs and events

Look for error patterns:

kubectl logs <pod-name> --tail=100 -n <namespace>
kubectl logs <pod-name> --previous  # if restarted
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>

Events often reveal OOMKilled, failed probes, or scheduling issues.

Inside the container

Exec into the pod to inspect processes and system metrics:

kubectl exec -it <pod-name> -n <namespace> -- /bin/sh

Inside, run:

ps aux
cat /proc/1/status | grep -E 'VmRSS|Threads'

Check for thread count growth (potential leaks) and RSS memory (actual usage). If your image lacks these tools, use kubectl debug to create a temporary debug container:

kubectl debug <pod-name> -n <namespace> --image=nicolaka/netshoot --target=<pod-name>

Then, from the debug container, you can run curl or tcpdump against the pod's network namespace.

Network performance

Test internal service latency:

kubectl exec <pod-name> -- curl -o /dev/null -s -w '%{time_total}\n' http://<service-name>.<namespace>.svc.cluster.local:80

High latency could indicate service mesh overhead, DNS issues, or CNI problems. Check DNS resolution:

kubectl exec <pod-name> -- nslookup <service-name>

Profiling application code

For deeper analysis, use language-specific profilers. For example, in a Java application, attach jcmd (if present in the container) to get heap histogram. Or, for Node.js, enable the inspector port and connect from your machine via kubectl port-forward.

Always correlate metrics with the time of user-reported issues. Use a monitoring system like Prometheus to graph CPU, memory, and request duration over time.

Failure Modes and Recovery

Performance problems often lead to pod restarts or evictions. Understand common failure modes and how to recover quickly.

OOMKilled (Exit code 137)

If a pod restarts with reason OOMKilled:

kubectl describe pod <pod-name> | grep -A 5 "Last State"

Output:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

This means the container exceeded its memory limit. Recovery options:

• Increase memory limit in Deployment.
• Reduce application memory usage (e.g., lower cache size).
• Fix memory leak.

Temporary mitigation: set a lower memory.limit_in_bytes within the cgroup and let the kernel kill the process gracefully; but better to fix the root cause.

CPU throttling

If CPU throttling is severe, response times increase and the application may fail readiness probes. Check:

kubectl describe pod <pod-name> | grep -A 3 "Conditions"

If readiness probe fails, the pod becomes unready and traffic is cut. Recovery: increase CPU limit, or set CPU request equal to limit to gain Guaranteed QoS and avoid throttling in some environments.

Disk pressure

If the node's disk is full, pods may be evicted. Check node conditions:

kubectl describe node <node-name> | grep -A 5 "Conditions"

Look for DiskPressure with status True. Recovery: clean up unused images (crictl rmi --prune on node), remove unneeded logs, or add disk capacity.

Pod eviction due to node pressure

If pod is evicted, it gets terminated and rescheduled. Check reason:

kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A 10 "status"

Look for message: The node was low on resource: memory.. Recovery: reduce pod memory request or add more nodes.

CrashLoopBackOff

If pod repeatedly crashes after start:

kubectl get pods -n <namespace>
# status shows CrashLoopBackOff
kubectl logs <pod-name> --previous

Often caused by application misconfiguration or resource limits too low. Check exit code: 137 OOM, 139 segfault, 143 SIGTERM from liveness probe failure.

For each failure, have a documented recovery runbook. In an emergency, you can roll back to a previous version:

kubectl rollout undo deployment/<deployment-name>

Operations Checklist for Pod Performance Tuning

Use this checklist before and after any tuning on a running pod. Each item should have a single accountable owner (e.g., on-call engineer) and be revisited whenever a performance incident occurs or at least monthly as part of capacity review.

This checklist ensures accountability and consistent execution. Store it in your team's runbook or automation tool.

Common Pitfalls and How to Avoid Them

Many Kubernetes performance issues arise from misunderstandings. Here are frequent mistakes and how to avoid or recover from them.

• Setting limits too low without understanding application behavior: Developers often copy example limits (e.g., 128Mi) and then wonder why the app gets OOMKilled. Avoid by profiling memory usage under load and set limits to at least 2x average usage. Recovery: increase limits and set up horizontal pod autoscaling.

• Ignoring CPU throttling: Throttling can silently degrade performance. Many teams only look at memory. Use kubectl top and cpu.stat to detect throttling. Recovery: raise CPU limit or use cpushares tuning.

• Not distinguishing between requests and limits: Requests are used for scheduling, limits for enforcement. If you set a high limit but low request, the node may become overcommitted and the pod evicted. Recovery: set requests close to limits for critical workloads.

• Using latest tags in production: This leads to unpredictable performance because the container changes without a manifest change. Use immutable tags and record the image digest. Recovery: pin images to digest and redeploy.

• Disabling liveness probes because they fail during load: Some teams remove probes to stop restarts, masking the real issue. Instead, adjust probe thresholds (failureThreshold: 3, periodSeconds: 10, timeoutSeconds: 5) to be more tolerant, and fix the underlying performance problem.

• Not reserving resources for system daemons: If you set pod requests too high relative to node capacity, kubelet may not have enough. Always leave room for kubelet, container runtime, and OS. Recovery: set systemReserved and kubeReserved on nodes.

• Debugging a live pod by attaching a debugger without considering impact: A debugger can pause the application and worsen performance. Use non-intrusive tools like pprof sampling or use a sidecar debug container with shareProcessNamespace: true to inspect without stopping the main process.

• Relying solely on kubectl logs for performance: Logs are useful but not for every metric. Use monitoring, tracing, and profiling together.

Conclusion

Tuning a running Kubernetes pod requires a careful, measured approach. You must first understand the environment and versions, then use read-only diagnostics to form hypotheses, make one small change at a time, and verify with metrics. Always have a rollback path.

Key takeaways:

• Establish a baseline before changes.
• Check CPU throttling, memory limits, and node pressure first; they are the most common bottlenecks.
• Use kubectl top, kubectl describe, kubectl exec, and kubectl debug to gather evidence.
• Apply changes via Deployment updates with version control and rollback ability.
• Document every step and assign a single owner for each checklist item.

As a next step, choose one low-risk verification, such as checking CPU throttling on a critical pod. Record the current state, run the command, compare with expected values, and then consider whether resource tuning is needed. Revisit this process monthly to keep performance healthy.

A reliable workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.