Kubernetes Debug Running Pod: Common Errors and Fixes with Practical Examples

Intro

Debugging a running pod in Kubernetes often starts with an error message like ImagePullBackOff, CrashLoopBackOff, or OOMKilled. These common errors can stop a deployment in its tracks, but with a systematic approach you can diagnose the root cause and apply a targeted fix quickly.

This article is a practical guide for developers, DevOps engineers, and technical startup teams who need to troubleshoot Kubernetes pods in development, staging, or production clusters. We cover the most frequent pod errors, explain what they mean, and provide concrete kubectl commands and configuration snippets to identify and resolve them.

You will learn how to:

• Inspect pod status and events with kubectl get and kubectl describe
• Retrieve logs from current and previous container instances
• Fix image-related errors, crash loops, resource limits, and probe failures
• Verify a fix by checking rollout status and pod health
• Use an operations checklist to avoid common pitfalls

All examples use Kubernetes 1.25+ and assume a working kubectl configured to access your cluster. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery steps.

Version and Environment Inventory

Before debugging, establish a clear picture of your Kubernetes environment. Run the following commands to capture the current state and relevant versions:

kubectl version --short
kubectl cluster-info
kubectl get nodes
kubectl get pods -o wide -n <namespace>

For example, if you are troubleshooting a pod named web-app-7d5c6f8b9-xyz in the default namespace, the output might look like:

NAME                            READY   STATUS             RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
web-app-7d5c6f8b9-xyz           0/1     CrashLoopBackOff    5         10m   10.244.1.5   worker-01    <none>           <none>

This already tells you the pod is restarting repeatedly (RESTARTS = 5) and is in a CrashLoopBackOff state. Next, collect detailed information about the pod:

kubectl describe pod web-app-7d5c6f8b9-xyz -n default

The Events section at the bottom of the describe output is often the fastest way to spot the problem. Here is a typical crash loop event sequence:

Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  2m                   default-scheduler  Successfully assigned default/web-app-7d5c6f8b9-xyz to worker-01
  Normal   Pulled     105s                 kubelet            Successfully pulled image "myapp:1.0" in 3.2s
  Normal   Created    105s                 kubelet            Created container app
  Normal   Started    105s                 kubelet            Started container app
  Warning  BackOff    104s                 kubelet            Back-off restarting failed container

To see logs from the current and previous container instances, use:

kubectl logs web-app-7d5c6f8b9-xyz -n default
kubectl logs web-app-7d5c6f8b9-xyz -n default --previous

Always capture the output of these commands before making any changes. This read-only observation forms your baseline and helps you verify whether a fix actually worked.

Safe Configuration Path

When a pod fails to run, resist the urge to delete and recreate it immediately. Instead, follow a safe configuration path: understand the pod's configuration, spot the misconfiguration, and apply the smallest change that addresses the root cause.

A common source of errors is a bad pod specification. Retrieve the pod definition in YAML format:

kubectl get pod web-app-7d5c6f8b9-xyz -n default -o yaml

Review key sections:

• spec.containers[].image – is the image name and tag correct?
• spec.containers[].resources – are requests and limits reasonable?
• spec.containers[].readinessProbe / livenessProbe – are probes correctly configured?
• spec.containers[].env – are environment variables set properly?

For example, a pod might fail because the livenessProbe has a much shorter initialDelaySeconds than the application startup time. This causes the kubelet to kill the container before it is ready.

Example of a problematic liveness probe:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

If the app needs 30 seconds to start, the probe will fail for the first 25 seconds and trigger a restart. A safer configuration would be:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 5

This gives the app enough time to start and tolerates transient failures.

To edit a live pod's configuration, you typically cannot change most fields directly. Instead, edit the deployment that manages the pod:

kubectl edit deployment web-app -n default

Make your change, save, and watch the rollout:

kubectl rollout status deployment/web-app -n default

If the rollout fails, you can roll back to the previous revision:

kubectl rollout undo deployment/web-app -n default

For testing configuration changes locally before applying to the cluster, use kubectl apply --dry-run=client or kubectl diff:

kubectl apply -f web-app.yaml --dry-run=client
kubectl diff -f web-app.yaml

This safe path minimizes risk and gives you a clear rollback plan.

Verification and Diagnostics

After you apply a fix, you must verify that the pod actually recovers. Use the following commands to confirm health:

1. Check pod status:

kubectl get pods -n default -l app=web-app

Look for Running status and a low restart count.

1. Check rollout status for the deployment:

kubectl rollout status deployment/web-app -n default

A successful rollout prints: deployment "web-app" successfully rolled out.

1. Inspect pod events for any new warnings:

kubectl describe pod <new-pod-name> -n default | tail -20

1. Verify the application is actually serving traffic. Use port-forward to test locally:

kubectl port-forward pod/<new-pod-name> 8080:8080 -n default

Then open http://localhost:8080 in a browser or use curl:

curl -I http://localhost:8080/health

You should see an HTTP 200 response if the app is healthy.

1. For deeper diagnostics, execute commands inside the container using kubectl exec:

kubectl exec -it <new-pod-name> -n default -- /bin/sh

Inside the shell, you can check processes, network connectivity, or file mounts.

Remember to capture expected output in every step. For example, a healthy pod check should show:

NAME                        READY   STATUS    RESTARTS   AGE
web-app-6c9f7b8f-abcde      1/1     Running   0          2m

If the pod remains in CrashLoopBackOff, do not assume the fix worked. Continue to the failure modes section.

Failure Modes and Recovery

Kubernetes pods fail in many ways. Here are the most common error types, their likely causes, and specific fixes.

1. ImagePullBackOff or ErrImagePull

Symptom: Pod status shows ImagePullBackOff or ErrImagePull. Describe the pod and you will see events like:

Warning  Failed     2m (x4 over 3m)   kubelet  Failed to pull image "myapp:latest": rpc error: code = NotFound desc = failed to pull and unpack image

Causes:

• Incorrect image name or tag
• Image does not exist in the registry
• Authentication failure to a private registry
• Network issues between the node and the registry

Diagnosis:

kubectl describe pod <pod-name> | grep -A 5 Events

Check the exact image reference. Verify it exists in the registry. For private registries, ensure the imagePullSecrets are configured.

Fix:

• Correct the image name/tag in your deployment YAML.
• If using a private registry, create and reference a secret:

apiVersion: v1
kind: Secret
metadata:
  name: regcred
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: <base64-encoded-docker-config>

Then in your pod spec:

spec:
  imagePullSecrets:
    - name: regcred
  containers:
    - name: app
      image: myregistry.com/myapp:1.0

• For network issues, check node connectivity to the registry.

2. CrashLoopBackOff

Symptom: Pod restarts repeatedly. The container starts and then exits with a non-zero code.

Causes:

• Application error inside the container (e.g., misconfiguration, missing dependency)
• Liveness probe failing and killing the container
• Out of memory (OOMKilled)
• Incorrect command or arguments

Diagnosis:

Check previous container logs to see the error at exit:

kubectl logs <pod-name> --previous

For an OOM kill, the pod status will show:

State:          Terminated
  Reason:       OOMKilled
  Exit Code:    137

Fix:

• Fix the application error based on logs.
• For OOM, increase the memory limit or reduce the application's memory usage. Example resource block:

resources:
  requests:
    memory: "128Mi"
    cpu: "250m"
  limits:
    memory: "256Mi"
    cpu: "500m"

• Adjust liveness probe parameters as shown in Safe Configuration Path.
• Ensure the command and args are correct in the pod spec.

3. Pending Pods

Symptom: Pod stays in Pending state.

Causes:

• Insufficient CPU or memory on nodes
• No nodes match node selectors or affinity rules
• PersistentVolumeClaim cannot be bound

Diagnosis:

kubectl describe pod <pod-name> | grep -A 10 Events

Look for messages like:

Warning  FailedScheduling  10s (x2 over 20s)  default-scheduler  0/3 nodes are available: 1 Insufficient memory, 2 node(s) didn't match node selector.

Fix:

• Reduce resource requests, or add more nodes to the cluster.
• Adjust node selectors or affinities to match available nodes.
• Ensure storage classes and PVCs are correctly defined.

4. Readiness or Liveness Probe Failures

Symptom: Pod is Running but not Ready, or is killed by liveness probe.

Causes:

• Probe endpoint returns non-200 status or times out
• Probe path or port incorrect
• initialDelaySeconds too short

Diagnosis:

Check pod events and probe configuration:

kubectl describe pod <pod-name> | grep -E 'Readiness|Liveness'

Fix:

• Correct the probe endpoint, path, and port.
• Increase initialDelaySeconds to accommodate application startup.
• For HTTP probes, ensure the application listens on the specified port.

5. CreateContainerConfigError

Symptom: Pod cannot start because the container configuration is invalid.

Causes:

• Missing ConfigMap or Secret referenced in the pod spec
• Invalid volume mounts
• Invalid environment variable definitions

Diagnosis:

kubectl describe pod <pod-name> | grep -A 10 Events

Look for messages like:

Warning  Failed     5m    kubelet  Error: configmap "app-config" not found

Fix:

• Create the missing ConfigMap or Secret, or correct the reference name.
• Fix volume mounts or environment variable syntax.

Recovery Best Practices

After applying a fix, always roll out the deployment and verify:

kubectl apply -f fixed-deployment.yaml
kubectl rollout status deployment/<name>

If the new pod still fails, roll back:

kubectl rollout undo deployment/<name>

Document the error, the fix, and the verification result in your team's runbook for future reference.

Operations Checklist

Use this checklist to systematically debug a pod issue. Replace the example values with your own resource names and parameters.

1. Identify the failing pod

• Command: kubectl get pods -n default -l app=web-app
• Expected output: list of pods; note the one with non-Running status.

1. Gather detailed pod information

• Command: kubectl describe pod web-app-7d5c6f8b9-xyz -n default
• Look for: events, container states, last state (e.g., Terminated with OOMKilled), readiness/liveness probe details.

1. Retrieve application logs

• Current logs: kubectl logs web-app-7d5c6f8b9-xyz -n default
• Previous logs: kubectl logs web-app-7d5c6f8b9-xyz -n default --previous
• Expected output: application error stack trace or exit message.

1. Check resource constraints

• Command: kubectl top pod web-app-7d5c6f8b9-xyz -n default
• Compare memory/CPU usage against limits in kubectl get pod -o yaml.

1. Verify configuration references

• Ensure all ConfigMaps, Secrets, and PersistentVolumeClaims referenced exist and are correctly named.

1. Apply the smallest fix

• Example: change image tag from latest to a specific version, increase memory limit from 256Mi to 512Mi, or fix the probe path.
• Update the deployment YAML and apply via kubectl apply -f deployment.yaml.

1. Monitor the rollout

• Command: kubectl rollout status deployment/web-app -n default
• Expected output: deployment "web-app" successfully rolled out.

1. Verify pod health

• Command: kubectl get pods -n default -l app=web-app
• Expected output: pod shows Running and Ready 1/1.

1. Test application functionality

• Use kubectl port-forward to access the app locally and confirm the expected response (e.g., HTTP 200 from /health).

1. Document the incident

• Record error, root cause, fix, and verification steps in your runbook or ticketing system.

This checklist ensures you do not skip critical diagnostic steps and reduces the chance of introducing new issues.

Conclusion

Debugging a running pod in Kubernetes is a methodical process. By starting with kubectl get and kubectl describe, examining logs, and understanding the common error patterns, you can quickly narrow down the root cause and apply a targeted fix.

Remember to always observe the current state first, make minimal changes, and verify each fix. Use the commands and examples in this article as a reference, but adapt them to your specific environment and application.

As a next step, choose one low-risk verification for a pod error you have encountered or anticipate. Practice the diagnostic flow: capture pod events, inspect logs, identify the failure reason, and apply a documented fix. Then, ensure your team has a runbook entry for that error so that future incidents are resolved faster.

A reliable troubleshooting workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With these practices, you can turn Kubernetes pod debugging from a firefight into a structured process.