## Intro

When a Kubernetes pod is already running and users report slowness, timeouts, or intermittent failures, you need a systematic approach that moves from observed symptom to verified fix without causing new incidents. Debugging a running pod is different from debugging a deployment that fails to start: the pod is serving traffic, so every action must minimize disruption and preserve evidence.

This practical guide walks through the process of tuning performance for a live pod. It covers environment inventory, safe configuration changes, targeted diagnostics, common failure modes, and an operations checklist. Along the way, you will use real commands such as `kubectl top`, `kubectl exec`, and `kubectl logs`, interpret their output, and make changes with controlled blast radius. The focus is on Linux containers running in Kubernetes 1.28+ with `kubectl` 1.28 or later, but the principles apply to any recent version.

The goal is operational safety: observe before changing, limit the scope of any intervention, verify results with metrics, and document rollback paths. You will learn how to diagnose CPU throttling, memory pressure, disk I/O bottlenecks, and network issues without restarting a pod unnecessarily.

## Version and Environment Inventory

Before touching a running pod, establish the exact versions and cluster state. This prevents false assumptions and ensures any tuning advice matches your environment.

Run the following read-only commands:

```bash
kubectl version --short
kubectl get nodes -o wide
kubectl describe node <your-node-name>
```

Expected output includes Kubernetes server version (e.g., `Server Version: v1.28.7`), node kernel version, container runtime (e.g., `containerd://1.7.11`), and operating system. Record these in your incident notes. If the node shows high `%CPU` or `%MEM`, treat the node as a candidate bottleneck before blaming the pod.

Next, inventory the pod itself:

```bash
kubectl get pod <pod-name> -n <namespace> -o yaml
```

Look for the pod's resource requests and limits under `spec.containers[].resources`. Example snippet:

```yaml
resources:
  requests:
    memory: "128Mi"
    cpu: "250m"
  limits:
    memory: "256Mi"
    cpu: "500m"
```

If requests are missing, the pod may be starved or overcommitted. If limits equal requests, the QoS class is Guaranteed, which gives higher priority; otherwise it is Burstable or BestEffort. This matters during node pressure.

Prerequisites for debugging:

- You have `kubectl` access with `get`, `describe`, `logs`, `exec`, and `top` permissions.
- You can read metrics-server output. If `kubectl top` reports `error: Metrics API not available`, install or enable metrics-server before relying on CPU/memory metrics.
- You have a way to generate test traffic, such as `hey`, `wrk`, or `curl` in a loop.

Keep changes local: reproduce the issue in a staging namespace with a copied pod spec before altering production. Use `kubectl port-forward` to test service endpoints without exposing them externally.

## Safe Configuration Path

Changing a running pod's configuration is risky. Kubernetes does not allow modifying most pod fields directly; you must either update the Deployment or terminate and recreate the pod. For performance tuning, you often adjust resource limits, environment variables that affect runtime behavior, or annotations. Always follow a safe sequence:

1. Observe current behavior for at least five minutes with load.
2. Define one hypothesis (e.g., "CPU limit is too low causing throttling").
3. Make the smallest change to address that hypothesis.
4. Compare before and after metrics.
5. Roll back if no improvement or regression occurs.

### Example: Increase CPU limit due to throttling

Check current throttling:

```bash
kubectl exec <pod-name> -- cat /sys/fs/cgroup/cpu/cpu.stat
```

Look for `nr_throttled` and `throttled_time`. If `throttled_time` increases steadily under load, the pod is hitting its CPU limit.

Proposed change in the Deployment manifest:

```yaml
resources:
  limits:
    cpu: "1000m"   # increased from 500m
  requests:
    cpu: "500m"
```

Apply with:

```bash
kubectl apply -f updated-deployment.yaml
kubectl rollout status deployment/<deployment-name> -n <namespace>
```

After rollout, generate the same load and compare `nr_throttled`. If throttling drops to zero or near zero, the change worked. If not, consider other bottlenecks.

### Example: Adjust garbage collection environment variable

Many runtimes (Java, Node.js) have environment variables that affect memory usage. Suppose a Node.js application occasionally exceeds memory limits and gets OOMKilled. You can set `NODE_OPTIONS=--max-old-space-size=512` to limit heap. First, test locally,

then update the Deployment:

```yaml
env:
  - name: NODE_OPTIONS
    value: "--max-old-space-size=512"
```

Apply and verify the pod restarts with the new environment using `kubectl describe pod` and check the environment section.

Always keep a copy of the previous manifest for rollback:

```bash
kubectl rollout undo deployment/<deployment-name>
```

## Verification and Diagnostics

Diagnostics should be layered: start with cluster-level metrics, then pod-level, then process-level inside the container.

### Cluster and pod resource usage

Use `kubectl top` to get current usage:

```bash
kubectl top node
kubectl top pod -n <namespace>
```

Example output:

```
NAME          CPU(cores)   MEMORY(bytes)
my-app-pod    720m         289Mi
```

If the pod consistently uses more than 80% of its limit, consider raising the limit. If it uses less than 20% of its request, you may overcommit and cause node pressure elsewhere.

### Application logs and events

Look for error patterns:

```bash
kubectl logs <pod-name> --tail=100 -n <namespace>
kubectl logs <pod-name> --previous  # if restarted
kubectl get events -n <namespace> --field-selector involvedObject.name=<pod-name>
```

Events often reveal OOMKilled, failed probes, or scheduling issues.

### Inside the container

Exec into the pod to inspect processes and system metrics:

```bash
kubectl exec -it <pod-name> -n <namespace> -- /bin/sh
```

Inside, run:

```sh
ps aux
cat /proc/1/status | grep -E 'VmRSS|Threads'
```

Check for thread count growth (potential leaks) and RSS memory (actual usage). If your image lacks these tools, use `kubectl debug` to create a temporary debug container:

```bash
kubectl debug <pod-name> -n <namespace> --image=nicolaka/netshoot --target=<pod-name>
```

Then, from the debug container, you can run `curl` or `tcpdump` against the pod's network namespace.

### Network performance

Test internal service latency:

```bash
kubectl exec <pod-name> -- curl -o /dev/null -s -w '%{time_total}\n' http://<service-name>.<namespace>.svc.cluster.local:80
```

High latency could indicate service mesh overhead, DNS issues, or CNI problems. Check DNS resolution:

```bash
kubectl exec <pod-name> -- nslookup <service-name>
```

### Profiling application code

For deeper analysis, use language-specific profilers. For example, in a Java application, attach `jcmd` (if present in the container) to get heap histogram. Or, for Node.js, enable the inspector port and connect from your machine via `kubectl port-forward`.

Always correlate metrics with the time of user-reported issues. Use a monitoring system like Prometheus to graph CPU, memory, and request duration over time.

## Failure Modes and Recovery

Performance problems often lead to pod restarts or evictions. Understand common failure modes and how to recover quickly.

### OOMKilled (Exit code 137)

If a pod restarts with reason `OOMKilled`:

```bash
kubectl describe pod <pod-name> | grep -A 5 "Last State"
```

Output:

```
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
```

This means the container exceeded its memory limit. Recovery options:

- Increase memory limit in Deployment.
- Reduce application memory usage (e.g., lower cache size).
- Fix memory leak.

Temporary mitigation: set a lower `memory.limit_in_bytes` within the cgroup and let the kernel kill the process gracefully; but better to fix the root cause.

### CPU throttling

If CPU throttling is severe, response times increase and the application may fail readiness probes. Check:

```bash
kubectl describe pod <pod-name> | grep -A 3 "Conditions"
```

If readiness probe fails, the pod becomes unready and traffic is cut. Recovery: increase CPU limit, or set CPU request equal to limit to gain Guaranteed QoS and avoid throttling in some environments.

### Disk pressure

If the node's disk is full, pods may be evicted. Check node conditions:

```bash
kubectl describe node <node-name> | grep -A 5 "Conditions"
```

Look for `DiskPressure` with status `True`. Recovery: clean up unused images (`crictl rmi --prune` on node), remove unneeded logs, or add disk capacity.

### Pod eviction due to node pressure

If pod is evicted, it gets terminated and rescheduled. Check reason:

```bash
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A 10 "status"
```

Look for `message: The node was low on resource: memory.`. Recovery: reduce pod memory request or add more nodes.

### CrashLoopBackOff

If pod repeatedly crashes after start:

```bash
kubectl get pods -n <namespace>
# status shows CrashLoopBackOff
kubectl logs <pod-name> --previous
```

Often caused by application misconfiguration or resource limits too low. Check exit code: 137 OOM, 139 segfault, 143 SIGTERM from liveness probe failure.

For each failure, have a documented recovery runbook. In an emergency, you can roll back to a previous version:

```bash
kubectl rollout undo deployment/<deployment-name>
```

## Operations Checklist for Pod Performance Tuning

Use this checklist before and after any tuning on a running pod. Each item should have a single accountable owner (e.g., on-call engineer) and be revisited whenever a performance incident occurs or at least monthly as part of capacity review.

| Checklist Item | Action | Owner | Frequency | Expected Result |
|---|---|---|---|---|
| Version inventory | Record Kubernetes, node OS, runtime versions | Platform engineer Priya Shah | Quarterly or on major upgrade | Documented versions match supported matrix |
| Baseline metrics | Capture pod CPU, memory, network, latency over 1 hour under normal load | DevOps lead Marcus Chen | Before any tuning or after code deploy | Baseline stored in monitoring dashboard |
| Resource request/limit review | Compare actual usage vs requests and limits | Application owner Sarah Ali | Monthly or before capacity planning | Gaps less than 20% for CPU, 10% for memory |
| Throttling check | Check `cpu.stat` inside pod for `nr_throttled` | On-call engineer Alex Kim | During incident or after limit change | Throttled time remains zero or acceptable threshold |
| Log analysis | Grep logs for errors, slow queries, timeouts | Application developer Diego Martinez | Weekly during sprint | No new error spikes |
| Probe tuning | Verify liveness/readiness probe thresholds are realistic | DevOps consultant Olga Petrova | On deployment of health checks | Probes pass with typical load, fail under extreme but recover |
| Rollback plan | Ensure previous manifest version is stored in Git and `kubectl rollout undo` works | Release manager Tom Wilson | Before every production change | Rollback tested in staging; undoes in <5 minutes |
| Post-change verification | After tuning, run load test and compare metrics | QA engineer Emily Zhang | After each tuning | Performance meets defined SLOs |

This checklist ensures accountability and consistent execution. Store it in your team's runbook or automation tool.

## Common Pitfalls and How to Avoid Them

Many Kubernetes performance issues arise from misunderstandings. Here are frequent mistakes and how to avoid or recover from them.

- **Setting limits too low without understanding application behavior**: Developers often copy example limits (e.g., 128Mi) and then wonder why the app gets OOMKilled. Avoid by profiling memory usage under load and set limits to at least 2x average usage. Recovery: increase limits and set up horizontal pod autoscaling.

- **Ignoring CPU throttling**: Throttling can silently degrade performance. Many teams only look at memory. Use `kubectl top` and `cpu.stat` to detect throttling. Recovery: raise CPU limit or use `cpushares` tuning.

- **Not distinguishing between requests and limits**: Requests are used for scheduling, limits for enforcement. If you set a high limit but low request, the node may become overcommitted and the pod evicted. Recovery: set requests close to limits for critical workloads.

- **Using `latest` tags in production**: This leads to unpredictable performance because the container changes without a manifest change. Use immutable tags and record the image digest. Recovery: pin images to digest and redeploy.

- **Disabling liveness probes because they fail during load**: Some teams remove probes to stop restarts, masking the real issue. Instead, adjust probe thresholds (`failureThreshold: 3`, `periodSeconds: 10`, `timeoutSeconds: 5`) to be more tolerant, and fix the underlying performance problem.

- **Not reserving resources for system daemons**: If you set pod requests too high relative to node capacity, kubelet may not have enough. Always leave room for kubelet, container runtime, and OS. Recovery: set `systemReserved` and `kubeReserved` on nodes.

- **Debugging a live pod by attaching a debugger without considering impact**: A debugger can pause the application and worsen performance. Use non-intrusive tools like `pprof` sampling or use a sidecar debug container with `shareProcessNamespace: true` to inspect without stopping the main process.

- **Relying solely on `kubectl logs` for performance**: Logs are useful but not for every metric. Use monitoring, tracing, and profiling together.

## Conclusion

Tuning a running Kubernetes pod requires a careful, measured approach. You must first understand the environment and versions, then use read-only diagnostics to form hypotheses, make one small change at a time, and verify with metrics. Always have a rollback path.

Key takeaways:

- Establish a baseline before changes.
- Check CPU throttling, memory limits, and node pressure first; they are the most common bottlenecks.
- Use `kubectl top`, `kubectl describe`, `kubectl exec`, and `kubectl debug` to gather evidence.
- Apply changes via Deployment updates with version control and rollback ability.
- Document every step and assign a single owner for each checklist item.

As a next step, choose one low-risk verification, such as checking CPU throttling on a critical pod. Record the current state, run the command, compare with expected values, and then consider whether resource tuning is needed. Revisit this process monthly to keep performance healthy.

A reliable workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.