## Intro Debugging a running pod in Kubernetes requires more than checking logs. When a pod is misbehaving but not crashing, you need to inspect its internal state without disrupting the workload. This article covers advanced techniques for debugging live pods, including ephemeral containers, process inspection, network analysis, and safe recovery workflows. It targets developers, DevOps engineers, and SREs who need to diagnose issues in production-like environments. The core principle is operational safety: observe before changing, limit the blast radius, use placeholders for secrets, verify results, and document recovery steps. We will explore commands, expected outputs, and failure signals to help you move from symptom to solution. ## Prerequisites and Environment Inventory Before debugging, confirm your cluster and tools are ready. - Kubernetes version 1.23 or later for kubectl debug with ephemeral containers. - kubectl client version within one minor version of the server. - Cluster access with permission to create pods and exec into containers. - Enable the EphemeralContainers feature gate if on older versions (GA in 1.25). Verify your environment with: kubectl version --short # Example output: # Client Version: v1.27.2 # Server Version: v1.27.3 Check that ephemeral containers are allowed by listing API resources: kubectl api-resources | grep -i ephemeral # Example output: # ephemeralcontainers pods true EphemeralContainer If the resource is missing, you may need to update your cluster or use older debugging methods like modifying the deployment spec. ## Understanding Pod States and Debugging Entry Points A pod's STATUS field from kubectl get pods gives the first clue. Common statuses include Running , Pending , CrashLoopBackOff , Error , and Completed . Each state suggests different debugging approaches. - Running : Pod is scheduled and at least one container is running. Internal issues may still exist. - Pending : Pod cannot be scheduled, often due to resource constraints or node selectors. - CrashLoopBackOff : A container starts and then exits with an error repeatedly. Logs from the previous instance are crucial. - Error : A container terminated with an error. - Completed : Container exited successfully; for debugging, you may need to keep it running. To see detailed events and conditions: kubectl describe pod -n Review the Events section for scheduling failures, image pull errors, or probe failures. ## Advanced Debugging with Ephemeral Containers Ephemeral containers are the modern way to debug a running pod without restarting it. They share the pod's network, process namespace, and optionally other namespaces, allowing you to inspect from inside. ### When to Use Ephemeral Containers - The pod has no shell in its existing containers (distroless images). - You need to attach debugging tools without modifying the original container. - You want to observe process-level activity without stopping the container. ### Creating an Ephemeral Container with kubectl debug The basic command: kubectl debug -it --image=busybox --target= -n Example: kubectl debug -it my-app-pod --image=busybox --target=my-app -n production You will be dropped into a shell inside the ephemeral container. Now you can inspect files, run network commands, or use debugging tools. ### Sharing Process Namespace By default, the ephemeral container does not share the process namespace, so you won't see the target container's processes. To share it, you must create the ephemeral container using a JSON patch or a YAML file because kubectl debug does not have a flag for this. Example patch file debug-pod.yaml : apiVersion: v1 kind: Pod metadata: name: my-app-pod namespace: production spec: ephemeralContainers: - name: debugger image: busybox command: - sleep - "3600" targetContainerName: my-app securityContext: runAsUser: 0 stdin: true tty: true Apply with: kubectl replace --raw /api/v1/namespaces/production/pods/my-app-pod/ephemeralcontainers -f debug-pod.yaml Then attach: kubectl attach -it -n production my-app-pod -c debugger Inside, run ps aux to see the target container's processes. ### Copying Files Ephemeral containers can be used to copy files to and from the pod using kubectl cp . This is useful for extracting core dumps or adding debugging tools. From pod to local: kubectl cp production/my-app-pod:/app/data -c debugger ./local-data From local to pod: kubectl cp ./my-tool production/my-app-pod:/tmp/my-tool -c debugger ### Security Considerations Ephemeral containers run with the same privileges as the user specified in the security context. Avoid running as root unless necessary, and clean up after debugging. The ephemeral container remains until the pod is deleted; you can remove it by patching the pod. ## Inspecting Processes and Runtime State When you cannot use ephemeral containers, you can still gather information with kubectl exec if the target container has a shell. ### Executing Commands in the Target Container kubectl exec -it -n -- /bin/sh If the container runs as a non-root user, you may need --user or --privileged flags, but use them with caution. ### Checking Process List kubectl exec -n -- ps aux # Example output: # USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND # root 1 0.1 0.3 123456 7890 ? Ssl 10:00 0:01 python app.py # root 250 0.0 0.1 65432 1234 ? Ss 10:05 0:00 /bin/sh This shows whether the expected process is running and its resource usage. ### Filesystem Inspection kubectl exec -n -- df -h kubectl exec -n -- du -sh /app Check for disk space issues or unexpected files. ### Runtime Information via /proc You can access /proc inside the container for runtime details: kubectl exec -n -- cat /proc/1/status This gives memory, CPU, and other details about PID 1. ## Network Debugging Inside the Pod Network issues are common. Use tools from inside the pod to test connectivity and DNS. ### Testing DNS Resolution kubectl exec -n -- nslookup kubernetes.default # Expected output: # Server: 10.96.0.10 # Address: 10.96.0.10:53 # Name: kubernetes.default.svc.cluster.local # Address: 10.96.0.1 If DNS fails, check the resolv.conf and the cluster DNS service. ### Checking Connectivity with curl kubectl exec -n -- curl -v http://service-name.namespace.svc.cluster.local If curl is not available, use wget or nc . ### Inspecting Socket and Routing kubectl exec -n -- netstat -tulpn kubectl exec -n -- ip route These commands help identify listening ports and routing problems. ### Using a Debug Container for Network Tools If the target container lacks network tools, create an ephemeral container with network tools image, like nicolaka/netshoot : kubectl debug -it --image=nicolaka/netshoot --target= -n Then run tcpdump , nmap , etc.

Quick check 2 of 2 When can you use kubectl debug to create a copy of a Pod? - When the pod is running normally - When you want to change the pod's name - When you can't run kubectl exec because the container lacks a shell or crashes on startup - When you need to delete the pod

The passage says: 'Sometimes Pod configuration options make it difficult to troubleshoot in certain situations. For example, you can't run kubectl exec to troubleshoot your container if your container image does not include a shell or if your application crashes on startup. In these situations you can use kubectl debug to create a copy of the Pod with configuration values changed to aid debugging.'

## Debugging with Core Dumps and Memory Analysis For crashes or memory issues, you may need to capture a core dump. ### Enabling Core Dumps Set ulimit -c unlimited in the container and ensure /proc/sys/kernel/core_pattern is writable or redirect to a mounted volume. Example pod spec: apiVersion: v1 kind: Pod metadata: name: core-dump-pod spec: containers: - name: app image: myapp:latest volumeMounts: - name: core-dumps mountPath: /cores securityContext: capabilities: add: ["SYS_PTRACE"] volumes: - name: core-dumps emptyDir: {} Then, inside the container, set ulimit -c unlimited && cd /cores && ./app . After crash, retrieve the core file: kubectl cp core-dump-pod:/cores/core ./core Analyze with gdb: gdb ./app ./core ## Failure Modes and Recovery Strategies ### Common Failure Modes - OOMKilled (Out of Memory) - Symptom: Pod restarts with reason OOMKilled in kubectl describe pod . - Verification: Check kubectl get pod -o yaml for lastState.terminated.reason: OOMKilled . - Recovery: Increase memory limits or optimize the application. - CrashLoopBackOff - Symptom: Pod status shows CrashLoopBackOff . - Verification: kubectl logs --previous shows error from last crash. - Recovery: Fix the application error. - ImagePullBackOff - Symptom: Pod cannot pull image, status ImagePullBackOff . - Verification: kubectl describe pod shows registry authentication or image not found. - Recovery: Correct image name or add imagePullSecrets. - Crash due to Misconfiguration - Symptom: Application exits immediately. - Debug: Use kubectl exec to run the command manually if possible. ### Recovery Workflows Always have a rollback plan. Use kubectl rollout undo for deployments: kubectl rollout undo deployment/my-app -n production Check rollout status: kubectl rollout status deployment/my-app -n production If a pod is stuck, you can force delete it: kubectl delete pod -n --grace-period=0 --force But be cautious: this may cause data loss. ## Operations Checklist Follow this checklist for systematic debugging: - Identify the pod and namespace : kubectl get pods -n production - Check pod status and events : kubectl describe pod my-app-pod -n production - View logs : kubectl logs my-app-pod -c my-app -n production --tail=100 - If crash-looping, view previous logs : kubectl logs my-app-pod -c my-app -n production --previous - Check resource usage : kubectl top pod my-app-pod -n production - Attach ephemeral container if needed : kubectl debug -it my-app-pod --image=busybox --target=my-app -n production - Test network : From inside, run curl http://service or nslookup . - Inspect processes : ps aux , df -h , du -sh /app . - Make one change at a time and observe. - Document findings and recovery steps . ## Conclusion Debugging a running pod in Kubernetes requires a systematic approach that balances observation and intervention. Advanced techniques like ephemeral containers provide powerful ways to inspect live workloads without disruption. By following the principles of safe debugging, you can isolate issues quickly and recover with minimal impact. Always verify your environment, understand the pod state, and use the right tools for diagnosis. With practice, you will handle even the most complex pod problems with confidence. As a next step, choose one technique from this article, such as ephemeral containers or process inspection, and apply it to a non-critical pod in your cluster. Record the commands and outputs, and build your own debugging runbook.