## Intro Kubernetes pods are the smallest deployable units, but their lifecycle is full of traps: images that never pull, probes that fail, nodes that cannot schedule, and containers that crash on startup. When a pod is stuck, knowing which state it is in and why is the fastest path to recovery. This article is a field guide for developers, DevOps engineers, and SREs who need to move from a reported symptom to a verified fix without guessing. It walks through common failure states, the exact commands to inspect them, the logs and events that reveal root cause, and the smallest safe changes that restore service. We will cover: - How to inventory your cluster and pod state before making changes - The pod lifecycle from Pending to Running to Terminated - Five common failure modes: Pending, ImagePullBackOff, CrashLoopBackOff, Probe failures, and OOMKilled - How to use kubectl describe , kubectl logs , events, and metrics - Recovery procedures with examples and expected outputs - An operations checklist to avoid making things worse All examples assume Kubernetes 1.24 or later and a working kubectl context. Replace resource names as needed. ## Version and Environment Inventory Before changing anything, capture the current state. The goal is to observe without mutating. Think of it as a read-only diagnostic pass. ### 1. Confirm cluster version and API availability kubectl version --short # Client Version: v1.27.3 # Server Version: v1.27.1 If the client and server differ by more than one minor version, upgrade the client to match before debugging; some kubectl flags may not exist on older servers. ### 2. Get a broad view of pods kubectl get pods -n default -o wide Example output: NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-6d4cf56db6-8z7wv 0/1 ImagePullBackOff 1 5m minikube api-85f9b8c7f-4k9s2 1/1 Running 0 2d 10.244.0.5 worker-01 Look at the STATUS , RESTARTS , and NODE columns. A high restart count suggests a crash loop; a missing IP often means the pod is not scheduled. ### 3. Check cluster-level events kubectl get events -n default --sort-by=.lastTimestamp Events are short-lived (default 1 hour). If the pod has been failing for a long time, you may need to rely on describe or logs for older history. ### 4. Inspect node conditions kubectl get nodes -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,MEMORY_PRESSURE:.status.conditions[?(@.type=="MemoryPressure")].status,DISK_PRESSURE:.status.conditions[?(@.type=="DiskPressure")].status' If a node is NotReady, pods on it will remain Pending or be evicted. ### 5. Verify the pod's controller kubectl get deploy,rs,sts,ds -n default If a pod is managed by a Deployment, the ReplicaSet name in kubectl describe pod shows which controller created it. This matters because you may need to roll back a Deployment rather than directly edit a pod. Checklist before intervention: - Cluster version confirmed - Pod list captured - Events reviewed - Node health checked - Controller identified Only after this inventory should you move to diagnosis and change.

Quick check 1 of 2 Which of the following is NOT a cause of a CrashLoopBackOff container restart loop, according to the reference? - An application exits immediately due to a misconfiguration - A liveness probe kills the container - A missing dependency such as a database or config file - The container image tag does not exist

Image tag non-existence causes ImagePullBackOff, not CrashLoopBackOff. CrashLoopBackOff is caused by application exits, missing dependencies, liveness probe failures, or OOM kills.

## Safe Configuration Path Treat every change as a potential production incident. Use a local or staging cluster first, then promote. ### 1. Reproduce locally Use a tool like kind or minikube to create a throwaway cluster: kind create cluster --name debug Apply the same manifest (after removing or overriding secrets): kubectl apply -f pod.yaml If the failure reproduces, you can safely experiment. If not, the issue may be environmental (node capacity, network policy, registry access). ### 2. Use dry-run and diff Always preview changes: kubectl apply -f pod.yaml --dry-run=client For modifications to existing objects: kubectl diff -f pod.yaml kubectl diff shows what would change without applying it. ### 3. Change one thing at a time If you update both an image tag and a resource limit simultaneously, you will not know which fixed it. Apply a minimal patch: kubectl set image deployment/nginx nginx=nginx:1.25.1 Then wait and observe before the next change. ### 4. Protect secrets Never put secrets in manifests you commit. Use placeholders: env: - name: DB_PASSWORD value: REPLACE_ME Then inject the real value via a Secret: kubectl create secret generic db-secret --from-literal=password='actual-password' Reference it in the pod: valueFrom: secretKeyRef: name: db-secret key: password ### 5. Verify after every change Use kubectl rollout status for Deployments: kubectl rollout status deployment/nginx Expected when healthy: Waiting for deployment "nginx" rollout to finish: 0 of 1 updated replicas are available... deployment "nginx" successfully rolled out If it times out, roll back: kubectl rollout undo deployment/nginx ### 6. Test traffic locally before exposing Use port-forward instead of immediately creating a LoadBalancer: kubectl port-forward pod/nginx-6d4cf56db6-8z7wv 8080:80 Then curl: curl localhost:8080 If that works, create a Service with type ClusterIP first, then NodePort or Ingress as needed. ## Verification and Diagnostics This section is the heart of troubleshooting: how to pinpoint why a pod is stuck. ### The pod lifecycle in Kubernetes A pod goes through these phases: - Pending : Scheduled but containers not yet created (image pulling, scheduling delay). - Running : All containers started, but may still be failing readiness checks. - Succeeded/Terminated : All containers completed successfully (for Jobs) or terminated. - Failed : At least one container exited with non-zero status. - Unknown : State cannot be determined (usually node communication issue). Within Running, container statuses include: - Waiting (with a reason like CrashLoopBackOff or ImagePullBackOff ) - Running - Terminated (with exit code) ### Diagnostic commands ### kubectl describe pod This is the first detailed command to run. kubectl describe pod nginx-6d4cf56db6-8z7wv Key sections: - Events at the bottom: Often contain the direct error message. - Conditions : PodScheduled , Initialized , ContainersReady , Ready . - Container statuses : State , Reason , Exit Code . Example event: Warning Failed 4m (x5 over 5m) kubelet Failed to pull image "nginx:latest": rpc error: code = Unknown desc = Error response from daemon: manifest for nginx:latest not found: manifest unknown: manifest unknown ### kubectl logs For a running container: kubectl logs pod-name -c container-name For a crashed container, use --previous : kubectl logs pod-name --previous This shows stdout/stderr before the last termination. ### kubectl get events Already shown, but combine with a watch: kubectl get events -n default --watch & kubectl apply -f pod.yaml Observe events as they stream. ### kubectl top If metrics-server is installed: kubectl top pod kubectl top node This reveals CPU/memory usage causing evictions or OOM. ### Interpreting common exit codes

Exit Code	Meaning	Typical Cause
0	Success	Normal shutdown
1	General error	Application crash
2	Misuse of shell builtins	Wrong command syntax
126	Command cannot execute	Permission problem
127	Command not found	Missing binary or PATH
128+n	Signal n	e.g., 137 = SIGKILL (OOM or liveness probe killing)

### Example: debugging a CrashLoopBackOff - Get pod details: kubectl get pod crashpod -o yaml - Check last state and exit code: lastState: terminated: exitCode: 1 reason: Error - View previous logs: kubectl logs crashpod --previous - If logs show a missing config file, fix the mount or environment variable. Quick check 2 of 2 What does it mean when a Pod is in the CrashLoopBackOff state? - The container is running and healthy - The container has failed to start and is being restarted with exponential backoff - The image cannot be pulled from the registry - The Pod is unschedulable due to insufficient resources

CrashLoopBackOff indicates that the backoff delay mechanism is in effect for a container that is failing and restarting repeatedly.

## Failure Modes and Recovery Here are the five most common pod failure states, each with diagnosis and recovery steps. ### 1. Pod stuck in Pending Symptoms : kubectl get pods shows Pending , no node assigned (NODE column empty). Causes : - Insufficient resources on any node (CPU, memory, GPU) - Node taints not tolerated - Node selectors/affinity rules unmatched - PersistentVolumeClaim bound to unavailable storage Diagnosis : kubectl describe pod pending-pod Look at Events for messages like: 0/3 nodes are available: 3 Insufficient memory. Or check node taints: kubectl get nodes -o json | jq '.items[].spec.taints' Recovery : - If resource insufficiency, reduce requests or add nodes. - If taints, add toleration to pod spec: tolerations: - key: "node-role.kubernetes.io/control-plane" operator: "Exists" effect: "NoSchedule" - If node selector mismatch, adjust selector or label a node. ### 2. ImagePullBackOff Symptoms : Pod status ImagePullBackOff or ErrImagePull . Causes : - Image tag does not exist - Registry authentication failure - Network policy blocking registry - Rate limits from Docker Hub Diagnosis : kubectl describe pod imagepull-pod | grep -A5 Events Typical event: Failed to pull image "myrepo/myapp:v1.2": rpc error: code = Unknown desc = Error response from daemon: pull access denied for myrepo/myapp, repository does not exist or may require 'docker login' Check if image exists locally: docker pull myrepo/myapp:v1.2 Recovery : - Correct the image tag. - If private registry, create a Secret and set imagePullSecrets : kubectl create secret docker-registry regcred \ --docker-server=myregistry.com \ --docker-username=user \ --docker-password=pass Then in pod spec: imagePullSecrets: - name: regcred - If rate limited, use a pull-through cache or different registry. ### 3. CrashLoopBackOff Symptoms : Pod restarts repeatedly, status CrashLoopBackOff , high RESTARTS count. Causes : - Application exits immediately due to misconfiguration - Missing dependency (database, config file) - Liveness probe killing the container - Out-of-memory (OOM) kill Diagnosis : kubectl logs crashpod --previous Common errors: - Error: Cannot find module '/app/index.js' (Node.js) - panic: runtime error: invalid memory address or nil pointer dereference (Go) - Exception in thread "main" java.lang.IllegalStateException: No database connection (Java) Check if OOM: kubectl describe pod crashpod | grep -i oom If OOM, you may see: Last State: Terminated Reason: OOMKilled Exit Code: 137 Recovery : - Fix the underlying application error. - If OOM, increase memory limit or reduce application memory usage. - If liveness probe is too aggressive, adjust initialDelaySeconds , periodSeconds , or the probe command. ### 4. Probe failures (readiness/liveness) Symptoms : Pod is Running but not Ready; traffic is not sent (readiness) or container is restarted (liveness). Diagnosis : kubectl get pods # NAME READY STATUS RESTARTS AGE # web-6d4cf56db6-8z7wv 0/1 Running 1 10m Readiness failing: RESTARTS remains low. Liveness failing: RESTARTS increases. Check probe configuration: kubectl get pod web-6d4cf56db6-8z7wv -o yaml | less Find readinessProbe and livenessProbe sections. Common probe failures: - HTTP endpoint returns 4xx/5xx - TCP port not open - Command exits non-zero Recovery : - Test the endpoint manually inside the pod: kubectl exec -it pod-name -- curl localhost:8080/healthz - Adjust probe path, port, or thresholds. - Increase timeoutSeconds if application is slow to start. - Consider a startupProbe for slow-starting apps. ### 5. Pod evicted or terminated unexpectedly Symptoms : Pod disappears from node, or enters Failed with Evicted status. Causes : - Node resource pressure (memory, disk) - Node cordoned/drained - Preemption by higher-priority pods Diagnosis : kubectl describe pod evicted-pod | grep -A10 Events Look for: The node was low on resource: memory. Or check node conditions: kubectl describe node worker-01 Recovery : - Free resources on node (delete unused pods, resize requests) - Add more nodes - Set appropriate priority classes to avoid preemption of critical pods ## Operations Checklist Use this checklist after any failure to ensure you have a complete picture and safe recovery. ### Immediate response - [ ] Capture pod list with timestamps: kubectl get pods -o wide > pods-$(date +%s).txt - [ ] Capture describe output: kubectl describe pod > describe-.txt - [ ] Capture logs (current and previous): kubectl logs --previous > logs--prev.txt - [ ] Capture events: kubectl get events > events.txt - [ ] Note the exact time and any recent changes (deployments, config updates) ### Diagnosis - [ ] Determine pod phase and reason from describe - [ ] Check container last state and exit code - [ ] Review logs for application errors - [ ] Check resource usage ( kubectl top if available) - [ ] Verify node health and taints - [ ] If probe failure, test endpoint manually ### Recovery - [ ] Decide on minimal change (image, config, resource, probe) - [ ] Apply change in staging first if possible - [ ] Use kubectl apply or kubectl set for the change - [ ] Monitor rollout with kubectl rollout status - [ ] Verify pod readiness and traffic (port-forward test) - [ ] Roll back if not healthy: kubectl rollout undo - [ ] Document root cause and fix ### Prevention - [ ] Add liveness and readiness probes to all long-running pods - [ ] Set appropriate resource requests and limits - [ ] Use explicit image tags instead of latest - [ ] Implement PodDisruptionBudgets for critical services - [ ] Set up monitoring alerts on pod restarts and failures - [ ] Regularly review node capacity and auto-scaling policies ## Conclusion Kubernetes pod lifecycle troubleshooting is systematic: observe first, then diagnose, then apply a minimal fix, and always verify. Rushing to kubectl delete pod may provide temporary relief but hides the root cause. This article covered the essential techniques: - Capturing environment and pod state before changes - Understanding pod phases and statuses - Using kubectl describe , logs , and events to find root causes - Five common failure modes and their recovery procedures - An operations checklist to ensure thorough incident response As a next step, choose one failing pod in your cluster and walk through the diagnostic process outlined here. Record the observed state, identify the cause, apply the smallest fix, and verify the result. Over time, these habits will turn pod debugging from a fire drill into a calm, repeatable procedure. Remember: the goal is not just to restart a pod, but to understand why it failed and to prevent recurrence.