Kubernetes Pod Lifecycle Troubleshooting with Practical Examples

Intro

Kubernetes pods are the smallest deployable units, but their lifecycle is full of traps: images that never pull, probes that fail, nodes that cannot schedule, and containers that crash on startup. When a pod is stuck, knowing which state it is in and why is the fastest path to recovery.

This article is a field guide for developers, DevOps engineers, and SREs who need to move from a reported symptom to a verified fix without guessing. It walks through common failure states, the exact commands to inspect them, the logs and events that reveal root cause, and the smallest safe changes that restore service.

We will cover:

• How to inventory your cluster and pod state before making changes
• The pod lifecycle from Pending to Running to Terminated
• Five common failure modes: Pending, ImagePullBackOff, CrashLoopBackOff, Probe failures, and OOMKilled
• How to use kubectl describe, kubectl logs, events, and metrics
• Recovery procedures with examples and expected outputs
• An operations checklist to avoid making things worse

All examples assume Kubernetes 1.24 or later and a working kubectl context. Replace resource names as needed.

Version and Environment Inventory

Before changing anything, capture the current state. The goal is to observe without mutating. Think of it as a read-only diagnostic pass.

1. Confirm cluster version and API availability

kubectl version --short
# Client Version: v1.27.3
# Server Version: v1.27.1

If the client and server differ by more than one minor version, upgrade the client to match before debugging; some kubectl flags may not exist on older servers.

2. Get a broad view of pods

kubectl get pods -n default -o wide

Example output:

NAME                     READY   STATUS             RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
nginx-6d4cf56db6-8z7wv   0/1     ImagePullBackOff    1          5m    <none>       minikube       <none>           <none>
api-85f9b8c7f-4k9s2      1/1     Running             0          2d    10.244.0.5   worker-01      <none>           <none>

Look at the STATUS, RESTARTS, and NODE columns. A high restart count suggests a crash loop; a missing IP often means the pod is not scheduled.

3. Check cluster-level events

kubectl get events -n default --sort-by=.lastTimestamp

Events are short-lived (default 1 hour). If the pod has been failing for a long time, you may need to rely on describe or logs for older history.

4. Inspect node conditions

kubectl get nodes -o custom-columns='NAME:.metadata.name,READY:.status.conditions[?(@.type=="Ready")].status,MEMORY_PRESSURE:.status.conditions[?(@.type=="MemoryPressure")].status,DISK_PRESSURE:.status.conditions[?(@.type=="DiskPressure")].status'

If a node is NotReady, pods on it will remain Pending or be evicted.

5. Verify the pod's controller

kubectl get deploy,rs,sts,ds -n default

If a pod is managed by a Deployment, the ReplicaSet name in kubectl describe pod shows which controller created it. This matters because you may need to roll back a Deployment rather than directly edit a pod.

Checklist before intervention:

• Cluster version confirmed
• Pod list captured
• Events reviewed
• Node health checked
• Controller identified

Only after this inventory should you move to diagnosis and change.

Safe Configuration Path

Treat every change as a potential production incident. Use a local or staging cluster first, then promote.

1. Reproduce locally

Use a tool like kind or minikube to create a throwaway cluster:

kind create cluster --name debug

Apply the same manifest (after removing or overriding secrets):

kubectl apply -f pod.yaml

If the failure reproduces, you can safely experiment. If not, the issue may be environmental (node capacity, network policy, registry access).

2. Use dry-run and diff

Always preview changes:

kubectl apply -f pod.yaml --dry-run=client

For modifications to existing objects:

kubectl diff -f pod.yaml

kubectl diff shows what would change without applying it.

3. Change one thing at a time

If you update both an image tag and a resource limit simultaneously, you will not know which fixed it. Apply a minimal patch:

kubectl set image deployment/nginx nginx=nginx:1.25.1

Then wait and observe before the next change.

4. Protect secrets

Never put secrets in manifests you commit. Use placeholders:

env:
  - name: DB_PASSWORD
    value: REPLACE_ME

Then inject the real value via a Secret:

kubectl create secret generic db-secret --from-literal=password='actual-password'

Reference it in the pod:

valueFrom:
  secretKeyRef:
    name: db-secret
    key: password

5. Verify after every change

Use kubectl rollout status for Deployments:

kubectl rollout status deployment/nginx

Expected when healthy:

Waiting for deployment "nginx" rollout to finish: 0 of 1 updated replicas are available...
deployment "nginx" successfully rolled out

If it times out, roll back:

kubectl rollout undo deployment/nginx

6. Test traffic locally before exposing

Use port-forward instead of immediately creating a LoadBalancer:

kubectl port-forward pod/nginx-6d4cf56db6-8z7wv 8080:80

Then curl:

curl localhost:8080

If that works, create a Service with type ClusterIP first, then NodePort or Ingress as needed.

Verification and Diagnostics

This section is the heart of troubleshooting: how to pinpoint why a pod is stuck.

The pod lifecycle in Kubernetes

A pod goes through these phases:

1. Pending: Scheduled but containers not yet created (image pulling, scheduling delay).
2. Running: All containers started, but may still be failing readiness checks.
3. Succeeded/Terminated: All containers completed successfully (for Jobs) or terminated.
4. Failed: At least one container exited with non-zero status.
5. Unknown: State cannot be determined (usually node communication issue).

Within Running, container statuses include:

• Waiting (with a reason like CrashLoopBackOff or ImagePullBackOff)
• Running
• Terminated (with exit code)

Diagnostic commands

kubectl describe pod

This is the first detailed command to run.

kubectl describe pod nginx-6d4cf56db6-8z7wv

Key sections:

• Events at the bottom: Often contain the direct error message.
• Conditions: PodScheduled, Initialized, ContainersReady, Ready.
• Container statuses: State, Reason, Exit Code.

Example event:

Warning  Failed     4m (x5 over 5m)   kubelet            Failed to pull image "nginx:latest": rpc error: code = Unknown desc = Error response from daemon: manifest for nginx:latest not found: manifest unknown: manifest unknown

kubectl logs

For a running container:

kubectl logs pod-name -c container-name

For a crashed container, use --previous:

kubectl logs pod-name --previous

This shows stdout/stderr before the last termination.

kubectl get events

Already shown, but combine with a watch:

kubectl get events -n default --watch &
kubectl apply -f pod.yaml

Observe events as they stream.

kubectl top

If metrics-server is installed:

kubectl top pod
kubectl top node

This reveals CPU/memory usage causing evictions or OOM.

Interpreting common exit codes

Example: debugging a CrashLoopBackOff

1. Get pod details:

   kubectl get pod crashpod -o yaml

1. Check last state and exit code:

   lastState:
     terminated:
       exitCode: 1
       reason: Error

1. View previous logs:

   kubectl logs crashpod --previous

1. If logs show a missing config file, fix the mount or environment variable.

Failure Modes and Recovery

Here are the five most common pod failure states, each with diagnosis and recovery steps.

1. Pod stuck in Pending

Symptoms: kubectl get pods shows Pending, no node assigned (NODE column empty).

Causes:

• Insufficient resources on any node (CPU, memory, GPU)
• Node taints not tolerated
• Node selectors/affinity rules unmatched
• PersistentVolumeClaim bound to unavailable storage

Diagnosis:

kubectl describe pod pending-pod

Look at Events for messages like:

0/3 nodes are available: 3 Insufficient memory.

Or check node taints:

kubectl get nodes -o json | jq '.items[].spec.taints'

Recovery:

• If resource insufficiency, reduce requests or add nodes.
• If taints, add toleration to pod spec:

  tolerations:
  - key: "node-role.kubernetes.io/control-plane"
    operator: "Exists"
    effect: "NoSchedule"

• If node selector mismatch, adjust selector or label a node.

2. ImagePullBackOff

Symptoms: Pod status ImagePullBackOff or ErrImagePull.

Causes:

• Image tag does not exist
• Registry authentication failure
• Network policy blocking registry
• Rate limits from Docker Hub

Diagnosis:

kubectl describe pod imagepull-pod | grep -A5 Events

Typical event:

Failed to pull image "myrepo/myapp:v1.2": rpc error: code = Unknown desc = Error response from daemon: pull access denied for myrepo/myapp, repository does not exist or may require 'docker login'

Check if image exists locally:

docker pull myrepo/myapp:v1.2

Recovery:

• Correct the image tag.
• If private registry, create a Secret and set imagePullSecrets:

  kubectl create secret docker-registry regcred \
    --docker-server=myregistry.com \
    --docker-username=user \
    --docker-password=pass

Then in pod spec:

  imagePullSecrets:
  - name: regcred

• If rate limited, use a pull-through cache or different registry.

3. CrashLoopBackOff

Symptoms: Pod restarts repeatedly, status CrashLoopBackOff, high RESTARTS count.

Causes:

• Application exits immediately due to misconfiguration
• Missing dependency (database, config file)
• Liveness probe killing the container
• Out-of-memory (OOM) kill

Diagnosis:

kubectl logs crashpod --previous

Common errors:

• Error: Cannot find module '/app/index.js' (Node.js)
• panic: runtime error: invalid memory address or nil pointer dereference (Go)
• Exception in thread "main" java.lang.IllegalStateException: No database connection (Java)

Check if OOM:

kubectl describe pod crashpod | grep -i oom

If OOM, you may see:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Recovery:

• Fix the underlying application error.
• If OOM, increase memory limit or reduce application memory usage.
• If liveness probe is too aggressive, adjust initialDelaySeconds, periodSeconds, or the probe command.

4. Probe failures (readiness/liveness)

Symptoms: Pod is Running but not Ready; traffic is not sent (readiness) or container is restarted (liveness).

Diagnosis:

kubectl get pods
# NAME                     READY   STATUS    RESTARTS   AGE
# web-6d4cf56db6-8z7wv     0/1     Running   1          10m

Readiness failing: RESTARTS remains low. Liveness failing: RESTARTS increases.

Check probe configuration:

kubectl get pod web-6d4cf56db6-8z7wv -o yaml | less

Find readinessProbe and livenessProbe sections.

Common probe failures:

• HTTP endpoint returns 4xx/5xx
• TCP port not open
• Command exits non-zero

Recovery:

• Test the endpoint manually inside the pod:

  kubectl exec -it pod-name -- curl localhost:8080/healthz

• Adjust probe path, port, or thresholds.
• Increase timeoutSeconds if application is slow to start.
• Consider a startupProbe for slow-starting apps.

5. Pod evicted or terminated unexpectedly

Symptoms: Pod disappears from node, or enters Failed with Evicted status.

Causes:

• Node resource pressure (memory, disk)
• Node cordoned/drained
• Preemption by higher-priority pods

Diagnosis:

kubectl describe pod evicted-pod | grep -A10 Events

Look for:

The node was low on resource: memory.

Or check node conditions:

kubectl describe node worker-01

Recovery:

• Free resources on node (delete unused pods, resize requests)
• Add more nodes
• Set appropriate priority classes to avoid preemption of critical pods

Operations Checklist

Use this checklist after any failure to ensure you have a complete picture and safe recovery.

Immediate response

• [ ] Capture pod list with timestamps: kubectl get pods -o wide > pods-$(date +%s).txt
• [ ] Capture describe output: kubectl describe pod <name> > describe-<name>.txt
• [ ] Capture logs (current and previous): kubectl logs <name> --previous > logs-<name>-prev.txt
• [ ] Capture events: kubectl get events > events.txt
• [ ] Note the exact time and any recent changes (deployments, config updates)

Diagnosis

• [ ] Determine pod phase and reason from describe
• [ ] Check container last state and exit code
• [ ] Review logs for application errors
• [ ] Check resource usage (kubectl top if available)
• [ ] Verify node health and taints
• [ ] If probe failure, test endpoint manually

Recovery

• [ ] Decide on minimal change (image, config, resource, probe)
• [ ] Apply change in staging first if possible
• [ ] Use kubectl apply or kubectl set for the change
• [ ] Monitor rollout with kubectl rollout status
• [ ] Verify pod readiness and traffic (port-forward test)
• [ ] Roll back if not healthy: kubectl rollout undo
• [ ] Document root cause and fix

Prevention

• [ ] Add liveness and readiness probes to all long-running pods
• [ ] Set appropriate resource requests and limits
• [ ] Use explicit image tags instead of latest
• [ ] Implement PodDisruptionBudgets for critical services
• [ ] Set up monitoring alerts on pod restarts and failures
• [ ] Regularly review node capacity and auto-scaling policies

Conclusion

Kubernetes pod lifecycle troubleshooting is systematic: observe first, then diagnose, then apply a minimal fix, and always verify. Rushing to kubectl delete pod may provide temporary relief but hides the root cause.

This article covered the essential techniques:

• Capturing environment and pod state before changes
• Understanding pod phases and statuses
• Using kubectl describe, logs, and events to find root causes
• Five common failure modes and their recovery procedures
• An operations checklist to ensure thorough incident response

As a next step, choose one failing pod in your cluster and walk through the diagnostic process outlined here. Record the observed state, identify the cause, apply the smallest fix, and verify the result. Over time, these habits will turn pod debugging from a fire drill into a calm, repeatable procedure.

Remember: the goal is not just to restart a pod, but to understand why it failed and to prevent recurrence.