Kubernetes: Debugging a Running Pod – Advanced Concepts and Practical Examples

Intro

Debugging a running pod in Kubernetes requires more than checking logs. When a pod is misbehaving but not crashing, you need to inspect its internal state without disrupting the workload. This article covers advanced techniques for debugging live pods, including ephemeral containers, process inspection, network analysis, and safe recovery workflows. It targets developers, DevOps engineers, and SREs who need to diagnose issues in production-like environments.

The core principle is operational safety: observe before changing, limit the blast radius, use placeholders for secrets, verify results, and document recovery steps. We will explore commands, expected outputs, and failure signals to help you move from symptom to solution.

Prerequisites and Environment Inventory

Before debugging, confirm your cluster and tools are ready.

• Kubernetes version 1.23 or later for kubectl debug with ephemeral containers.
• kubectl client version within one minor version of the server.
• Cluster access with permission to create pods and exec into containers.
• Enable the EphemeralContainers feature gate if on older versions (GA in 1.25).

Verify your environment with:

kubectl version --short
# Example output:
# Client Version: v1.27.2
# Server Version: v1.27.3

Check that ephemeral containers are allowed by listing API resources:

kubectl api-resources | grep -i ephemeral
# Example output:
# ephemeralcontainers   pods   true   EphemeralContainer

If the resource is missing, you may need to update your cluster or use older debugging methods like modifying the deployment spec.

Understanding Pod States and Debugging Entry Points

A pod's STATUS field from kubectl get pods gives the first clue. Common statuses include Running, Pending, CrashLoopBackOff, Error, and Completed. Each state suggests different debugging approaches.

• Running: Pod is scheduled and at least one container is running. Internal issues may still exist.
• Pending: Pod cannot be scheduled, often due to resource constraints or node selectors.
• CrashLoopBackOff: A container starts and then exits with an error repeatedly. Logs from the previous instance are crucial.
• Error: A container terminated with an error.
• Completed: Container exited successfully; for debugging, you may need to keep it running.

To see detailed events and conditions:

kubectl describe pod <pod-name> -n <namespace>

Review the Events section for scheduling failures, image pull errors, or probe failures.

Advanced Debugging with Ephemeral Containers

Ephemeral containers are the modern way to debug a running pod without restarting it. They share the pod's network, process namespace, and optionally other namespaces, allowing you to inspect from inside.

When to Use Ephemeral Containers

• The pod has no shell in its existing containers (distroless images).
• You need to attach debugging tools without modifying the original container.
• You want to observe process-level activity without stopping the container.

Creating an Ephemeral Container with kubectl debug

The basic command:

kubectl debug -it <pod-name> --image=busybox --target=<container-name> -n <namespace>

Example:

kubectl debug -it my-app-pod --image=busybox --target=my-app -n production

You will be dropped into a shell inside the ephemeral container. Now you can inspect files, run network commands, or use debugging tools.

Sharing Process Namespace

By default, the ephemeral container does not share the process namespace, so you won't see the target container's processes. To share it, you must create the ephemeral container using a JSON patch or a YAML file because kubectl debug does not have a flag for this.

Example patch file debug-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: my-app-pod
  namespace: production
spec:
  ephemeralContainers:
  - name: debugger
    image: busybox
    command:
    - sleep
    - "3600"
    targetContainerName: my-app
    securityContext:
      runAsUser: 0
    stdin: true
    tty: true

Apply with:

kubectl replace --raw /api/v1/namespaces/production/pods/my-app-pod/ephemeralcontainers -f debug-pod.yaml

Then attach:

kubectl attach -it -n production my-app-pod -c debugger

Inside, run ps aux to see the target container's processes.

Copying Files

Ephemeral containers can be used to copy files to and from the pod using kubectl cp. This is useful for extracting core dumps or adding debugging tools.

From pod to local:

kubectl cp production/my-app-pod:/app/data -c debugger ./local-data

From local to pod:

kubectl cp ./my-tool production/my-app-pod:/tmp/my-tool -c debugger

Security Considerations

Ephemeral containers run with the same privileges as the user specified in the security context. Avoid running as root unless necessary, and clean up after debugging. The ephemeral container remains until the pod is deleted; you can remove it by patching the pod.

Inspecting Processes and Runtime State

When you cannot use ephemeral containers, you can still gather information with kubectl exec if the target container has a shell.

Executing Commands in the Target Container

kubectl exec -it <pod-name> -n <namespace> -- /bin/sh

If the container runs as a non-root user, you may need --user or --privileged flags, but use them with caution.

Checking Process List

kubectl exec <pod-name> -n <namespace> -- ps aux
# Example output:
# USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
# root 1 0.1 0.3 123456 7890 ? Ssl 10:00 0:01 python app.py
# root 250 0.0 0.1 65432 1234 ? Ss 10:05 0:00 /bin/sh

This shows whether the expected process is running and its resource usage.

Filesystem Inspection

kubectl exec <pod-name> -n <namespace> -- df -h
kubectl exec <pod-name> -n <namespace> -- du -sh /app

Check for disk space issues or unexpected files.

Runtime Information via /proc

You can access /proc inside the container for runtime details:

kubectl exec <pod-name> -n <namespace> -- cat /proc/1/status

This gives memory, CPU, and other details about PID 1.

Network Debugging Inside the Pod

Network issues are common. Use tools from inside the pod to test connectivity and DNS.

Testing DNS Resolution

kubectl exec <pod-name> -n <namespace> -- nslookup kubernetes.default
# Expected output:
# Server: 10.96.0.10
# Address: 10.96.0.10:53
# Name: kubernetes.default.svc.cluster.local
# Address: 10.96.0.1

If DNS fails, check the resolv.conf and the cluster DNS service.

Checking Connectivity with curl

kubectl exec <pod-name> -n <namespace> -- curl -v http://service-name.namespace.svc.cluster.local

If curl is not available, use wget or nc.

Inspecting Socket and Routing

kubectl exec <pod-name> -n <namespace> -- netstat -tulpn
kubectl exec <pod-name> -n <namespace> -- ip route

These commands help identify listening ports and routing problems.

Using a Debug Container for Network Tools

If the target container lacks network tools, create an ephemeral container with network tools image, like nicolaka/netshoot:

kubectl debug -it <pod-name> --image=nicolaka/netshoot --target=<container-name> -n <namespace>

Then run tcpdump, nmap, etc.

Debugging with Core Dumps and Memory Analysis

For crashes or memory issues, you may need to capture a core dump.

Enabling Core Dumps

Set ulimit -c unlimited in the container and ensure /proc/sys/kernel/core_pattern is writable or redirect to a mounted volume.

Example pod spec:

apiVersion: v1
kind: Pod
metadata:
  name: core-dump-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    volumeMounts:
    - name: core-dumps
      mountPath: /cores
    securityContext:
      capabilities:
        add: ["SYS_PTRACE"]
  volumes:
  - name: core-dumps
    emptyDir: {}

Then, inside the container, set ulimit -c unlimited && cd /cores && ./app.

After crash, retrieve the core file:

kubectl cp core-dump-pod:/cores/core ./core

Analyze with gdb:

gdb ./app ./core

Failure Modes and Recovery Strategies

Common Failure Modes

1. OOMKilled (Out of Memory)

• Symptom: Pod restarts with reason OOMKilled in kubectl describe pod.
• Verification: Check kubectl get pod -o yaml for lastState.terminated.reason: OOMKilled.
• Recovery: Increase memory limits or optimize the application.

1. CrashLoopBackOff

• Symptom: Pod status shows CrashLoopBackOff.
• Verification: kubectl logs <pod> --previous shows error from last crash.
• Recovery: Fix the application error.

1. ImagePullBackOff

• Symptom: Pod cannot pull image, status ImagePullBackOff.
• Verification: kubectl describe pod shows registry authentication or image not found.
• Recovery: Correct image name or add imagePullSecrets.

1. Crash due to Misconfiguration

• Symptom: Application exits immediately.
• Debug: Use kubectl exec to run the command manually if possible.

Recovery Workflows

Always have a rollback plan. Use kubectl rollout undo for deployments:

kubectl rollout undo deployment/my-app -n production

Check rollout status:

kubectl rollout status deployment/my-app -n production

If a pod is stuck, you can force delete it:

kubectl delete pod <pod-name> -n <namespace> --grace-period=0 --force

But be cautious: this may cause data loss.

Operations Checklist

Follow this checklist for systematic debugging:

1. Identify the pod and namespace: kubectl get pods -n production
2. Check pod status and events: kubectl describe pod my-app-pod -n production
3. View logs: kubectl logs my-app-pod -c my-app -n production --tail=100
4. If crash-looping, view previous logs: kubectl logs my-app-pod -c my-app -n production --previous
5. Check resource usage: kubectl top pod my-app-pod -n production
6. Attach ephemeral container if needed: kubectl debug -it my-app-pod --image=busybox --target=my-app -n production
7. Test network: From inside, run curl http://service or nslookup.
8. Inspect processes: ps aux, df -h, du -sh /app.
9. Make one change at a time and observe.
10. Document findings and recovery steps.

Conclusion

Debugging a running pod in Kubernetes requires a systematic approach that balances observation and intervention. Advanced techniques like ephemeral containers provide powerful ways to inspect live workloads without disruption. By following the principles of safe debugging, you can isolate issues quickly and recover with minimal impact. Always verify your environment, understand the pod state, and use the right tools for diagnosis. With practice, you will handle even the most complex pod problems with confidence.

As a next step, choose one technique from this article, such as ephemeral containers or process inspection, and apply it to a non-critical pod in your cluster. Record the commands and outputs, and build your own debugging runbook.