## Intro Kubernetes events are one of the first places operators look when something goes wrong, but they are also one of the most misunderstood. An event is not an error by itself; it is a timestamped record of what happened to an object in the cluster, such as a pod being scheduled, a container crashing, or a volume failing to mount. Common errors like `CrashLoopBackOff`, `ImagePullBackOff`, `FailedScheduling`, or `FailedMount` appear as events, and each one points to a different underlying problem. This article covers the most frequent Kubernetes event errors, how to read them, how to fix them with practical commands, and how to prevent them from recurring. We focus on real-world operations for developers, DevOps engineers, and startup teams running Kubernetes in production or staging. The goal is to give you a clear path from observing an event to verifying the fix, without dangerous guesswork. Every recommendation includes concrete `kubectl` commands, expected output, and recovery steps. Before changing anything, capture the current state and understand the blast radius. Then make one scoped change and verify the result. This operational safety pattern runs through the entire article. Keep in mind that Kubernetes versions and distributions (vanilla, EKS, GKE, AKS, OpenShift) may differ in event wording or defaults. The commands shown here work on Kubernetes 1.24 and later unless noted. Always check your cluster version with `kubectl version` before applying fixes. ## Version and Environment Inventory Before diagnosing event errors, know exactly what you are working with. Run these commands to establish a baseline: ```bash kubectl version --short kubectl cluster-info kubectl get nodes -o wide ``` Example output (truncated) for a healthy cluster: ``` Client Version: v1.27.3 K8s Master: https://192.168.49.2:8443 NAME STATUS ROLES AGE VERSION INTERNAL-IP minikube Ready control-plane 10d v1.26.1 192.168.49.2 worker1 Ready 10d v1.26.1 192.168.49.3 ``` This tells you the API server address, node versions, and overall health. If a node is `NotReady`, events on pods scheduled on that node will cascade. Address node issues first. Next, identify the namespace and resources you care about. Use `kubectl get all -n ` to see pods, services, deployments, and more. For a specific pod, get detailed information: ```bash kubectl get pods -n myapp -o wide NAME READY STATUS RESTARTS AGE web-5db6d9c8f4-abcde 0/1 CrashLoopBackOff 5 10m web-5db6d9c8f4-fghij 1/1 Running 0 5m ``` Here one pod is crash looping. To see events for that pod: ```bash kubectl describe pod web-5db6d9c8f4-abcde -n myapp ``` Look under the `Events` section at the bottom. It will show timestamps, reason, and message. Example: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 10m default-scheduler Successfully assigned myapp/web-5db6d9c8f4-abcde to worker1 Normal Pulled 9m kubelet Container image "nginx:latest" already present on machine Normal Created 9m kubelet Created container web Normal Started 9m kubelet Started container web Warning BackOff 8m kubelet Back-off restarting failed container ``` This shows the container started but then failed, triggering a backoff. To see why it failed, fetch the logs: ```bash kubectl logs web-5db6d9c8f4-abcde -n myapp --previous ``` The `--previous` flag is crucial for CrashLoopBackOff because it gets logs from the previous container instance, which contains the error that caused the restart. Without it, you may only see the current (likely failing) start-up logs. If the pod is pending due to scheduling issues, look at events for `FailedScheduling` and node conditions. Always record these outputs in your incident notes. Environment inventory is not a one-time step; repeat it after any change to compare states. ## Safe Configuration Path The safest way to fix event errors is to first reproduce them in a controlled environment. Avoid making changes directly to production manifests without testing. Use a local cluster like minikube, kind, or k3s for quick experiments. For example, to test a manifest before applying to production: ```bash kubectl apply --dry-run=client -f deployment.yaml # or for server-side validation without persisting: kubectl apply --dry-run=server -f deployment.yaml ``` If the manifest passes, you can apply it to a staging namespace. Always keep manifests versioned in Git and use a deployment strategy like rolling update to limit blast radius. When a pod is continuously restarting, inspect the deployment's rollout status: ```bash kubectl rollout status deployment/web -n myapp ``` If it is stuck, you can pause the rollout: ```bash kubectl rollout pause deployment/web -n myapp ``` Then make changes and resume or rollback: ```bash kubectl rollout undo deployment/web -n myapp ``` For configuration errors like missing ConfigMap or Secret, the pod may fail to start. Use `kubectl describe` to see the event, then check if the resources exist: ```bash kubectl get configmap,secret -n myapp ``` If missing, create them from the correct files or values. Avoid embedding secrets in manifests; use Kubernetes Secrets and reference them. After fixing the configuration, verify the pod is healthy: ```bash kubectl get pods -n myapp -w # Ctrl+C to stop watching ``` Watch for the status to become `Running` and ready `1/1` without restarts. A safe configuration path always includes a rollback plan. ## Verification and Diagnostics Once you have applied a fix, you must verify it works as expected. Do not assume; check with commands and logs. For a pod that is now running, test connectivity from inside the cluster or via port-forward: ```bash kubectl port-forward pod/web-5db6d9c8f4-abcde 8080:80 -n myapp ``` Then in another terminal: ```bash curl -I http://localhost:8080 ``` Expected output should include `HTTP/1.1 200 OK` if the container serves HTTP. If not, examine the logs again. To diagnose container crashes, get the exit code: ```bash kubectl get pods -n myapp -o jsonpath='{.items[0].status.containerStatuses[0].lastState.terminated.exitCode}' ``` Common exit codes: - `0`: normal exit (but if it exits immediately, maybe the container has no foreground process) - `1`: application error - `2`: misuse of shell builtins - `126`: command invoked cannot execute - `127`: command not found - `137`: SIGKILL (often OOM kill or pod termination) - `139`: SIGSEGV (segmentation fault) - `143`: SIGTERM (graceful termination) For example, exit code 137 indicates the container was killed, likely due to memory limits. Check the pod's resource usage and limits: ```bash kubectl top pod -n myapp kubectl describe pod web-5db6d9c8f4-abcde -n myapp | grep -A5 Limits ``` If the container exceeded memory, increase the limit or reduce memory usage. Event errors often require checking cluster-level events: ```bash kubectl get events -n myapp --sort-by=.lastTimestamp ``` You can filter events by field selector, for example, to see only warnings: ```bash kubectl get events -n myapp --field-selector type=Warning ``` This helps you focus on problems. If you use monitoring tools like Prometheus or Loki, correlate event timestamps with metrics and logs for a complete picture. ## Failure Modes and Recovery Let's dive into specific common event errors and how to recover from them. ### CrashLoopBackOff **What it looks like:** Pod status `CrashLoopBackOff`, events show `BackOff` warning. **Why it happens:** Container starts and then exits with a non-zero code, repeatedly. **Common causes and fixes:** - **Application error:** Check logs with `kubectl logs --previous`. Fix the code or configuration. For example, if a Python app fails because a module is missing, add the dependency to the image or requirements. - **Missing file or config:** The container expects a config file but it is not mounted. Ensure ConfigMap or Secret is mounted correctly. - **Liveness probe misconfiguration:** If the liveness probe is too strict, the container may be killed even though it is healthy. Adjust `initialDelaySeconds`, `periodSeconds`, `timeoutSeconds`, and `failureThreshold`. - **Command or arguments wrong:** The pod's `command` or `args` may be incorrect. Check the image's entrypoint. If the image expects to run as non-root, set `securityContext.runAsUser`. **Recovery:** Fix the root cause, then restart the pod by deleting it (the deployment will recreate it) or rolling out a new version. ### ImagePullBackOff **What it looks like:** Pod status `ImagePullBackOff` or `ErrImagePull`, events show `Failed to pull image`. **Why it happens:** The container image cannot be pulled from the registry. **Common causes and fixes:** - **Image name or tag incorrect:** Double-check spelling, registry, and tag. Use `docker pull ` locally to test. - **Private registry credentials missing:** Create a Secret with registry credentials and reference it in `imagePullSecrets`. - **Registry unreachable:** Check network connectivity from nodes to registry. - **Rate limiting:** Docker Hub rate limits anonymous pulls. Use authenticated pulls or a mirror. **Recovery:** Correct the image reference or credentials, then delete the pod to force a new pull, or update the deployment. ### FailedScheduling **What it looks like:** Pod stuck in `Pending`, events show `FailedScheduling` with reasons like `0/3 nodes are available: 3 Insufficient cpu.` **Why it happens:** No node has enough resources, or there are taints/tolerations, or node selectors/affinity rules cannot be satisfied. **Common causes and fixes:** - **Resource requests too high:** Reduce `resources.requests` or scale up nodes. - **Taints and tolerations:** Nodes may have taints; pods need matching tolerations. - **Node selector mismatch:** Pod's `nodeSelector` may not match any node labels. - **Pod anti-affinity:** Rules may prevent scheduling. **Recovery:** Adjust the pod spec or cluster capacity. Use `kubectl describe pod` to see the exact reason. ### FailedMount **What it looks like:** Container cannot start, events show `FailedMount` or `MountVolume.SetUp failed`. **Why it happens:** Volume cannot be mounted, often due to missing PersistentVolume (PV) or PersistentVolumeClaim (PVC), or incorrect secret/configmap name. **Common causes and fixes:** - **PVC not bound:** Check `kubectl get pvc -n `. If pending, there may be no matching PV or StorageClass. - **Secret/ConfigMap missing:** Create the required resource or fix the reference. - **NFS or cloud volume issues:** Check storage backend connectivity and permissions. **Recovery:** Fix the underlying storage or reference, then recreate the pod. ## Advanced Diagnostics with Events For deeper troubleshooting, use JSON output and jq to filter events: ```bash kubectl get events -n myapp -o json | jq '.items[] | select(.reason=="FailedScheduling") | .message' ``` You can also watch events in real time with `kubectl get events -w` in a separate terminal while you apply changes. This gives immediate feedback. If events are not providing enough detail, check the kubelet logs on the node where the pod is scheduled: ```bash journalctl -u kubelet -f # or if kubelet runs as a container: docker logs kubelet ``` These logs often contain more detailed error messages, especially for mount issues or container runtime problems. For events that are old and purged (the default retention is 1 hour), you may need to rely on logs from monitoring systems or increase event TTL in kube-apiserver if needed for forensics. ## Common Pitfalls and How to Avoid Them Many Kubernetes event errors share avoidable pitfalls. Here are the most frequent ones we see in the field. ### Ignoring the `--previous` flag When a container is crash looping, `kubectl logs ` shows logs from the current (often just starting) container. The actual error is in the previous container's logs. **Always** use `--previous` for CrashLoopBackOff. Without it, you miss the root cause and waste time. **How to avoid:** Make `kubectl logs --previous` your first command for any pod that restarts. ### Assuming events are errors Not all events are bad. `Normal` events like `Scheduled`, `Pulled`, `Created`, and `Started` are expected. Only `Warning` events indicate problems. Filtering by type helps you focus. **How to avoid:** Use `kubectl get events --field-selector type=Warning` to see only warnings. ### Changing too many things at once In panic, teams often modify multiple configurations simultaneously. If the pod starts working, they do not know which change fixed it; if it fails, they have a mess. This violates the principle of one change at a time. **How to avoid:** Make one scoped change, verify, then proceed. Keep a rollback plan. ### Not checking resource limits Many CrashLoopBackOff issues are due to OOMKilled because memory limits are too low. The pod shows exit code 137. Increasing limits or optimizing memory usage is often the fix. **How to avoid:** Set reasonable resource requests and limits, monitor usage with `kubectl top`, and adjust before limits are hit. ### Forgetting imagePullSecrets for private registries When moving from public to private images, pods fail with `ImagePullBackOff` because credentials are missing. This is a classic configuration oversight. **How to avoid:** Always include `imagePullSecrets` in your pod spec when using private registries. Test pulling the image manually on a node to confirm credentials. ### Not using `kubectl describe` enough `kubectl describe pod` shows the event history that `kubectl get events` might miss or that has been purged. It also shows container states, probes, and mount information all in one place. Many operators jump straight to logs and miss the event clues. **How to avoid:** Make `kubectl describe pod ` part of your initial triage, right after `kubectl get pods`. ## Owners and Review Cadence In a team setting, troubleshooting Kubernetes events should have clear ownership. We recommend assigning a rotation of on-call engineers who are responsible for initial triage. For example, Priya Shah, Engineering Lead, owns the incident response process and reviews event error trends weekly. The on-call engineer investigates each event error, documents the root cause, and updates the runbook. A post-incident review is held for any event error that causes more than 10 minutes of downtime or repeated occurrences. This ensures that fixes are not just band-aids but lead to long-term improvements. ## Operations Checklist Use this checklist when you encounter a Kubernetes event error. It is a compact version of the workflows described above, with concrete commands. | Step | Action | Command / Example | Owner | Frequency | |------|--------|-------------------|-------|-----------| | 1 | Identify the failing resource | `kubectl get pods -n myapp -o wide` | On-call engineer | Every incident | | 2 | Get detailed events and state | `kubectl describe pod -n myapp` | On-call engineer | Every incident | | 3 | Check previous logs for crash | `kubectl logs -n myapp --previous` | On-call engineer | Every incident | | 4 | Examine cluster warnings | `kubectl get events -n myapp --field-selector type=Warning` | On-call engineer | Every incident | | 5 | Diagnose the root cause | Based on event reason and logs | On-call engineer | Every incident | | 6 | Apply minimal fix in staging first | `kubectl apply --dry-run=server -f fix.yaml` | Developer | Before production change | | 7 | Verify fix in staging | `kubectl port-forward` and `curl` | Developer | After each fix | | 8 | Roll out to production with rollback plan | `kubectl apply -f fix.yaml` (deployment) | Engineering Lead | After staging verification | | 9 | Monitor for recurrence | `kubectl get pods -w` and alerts | On-call engineer | 24 hours post-fix | | 10 | Update runbook and share learnings | Documentation in wiki | On-call engineer | Weekly review | ## Conclusion Kubernetes event errors are manageable if you approach them systematically. Start with environment inventory, use `kubectl describe` and `kubectl logs --previous` to read the event story, apply one scoped fix at a time, and verify thoroughly. Common errors like `CrashLoopBackOff`, `ImagePullBackOff`, `FailedScheduling`, and `FailedMount` have well-known causes and fixes described above. Avoid the pitfalls of ignoring previous logs, mixing changes, and neglecting resource limits. Assign clear ownership and review incidents regularly to improve processes. As a next step, choose one low-risk verification from this article, record the current state of your cluster, run the documented commands, compare the result with the expected signal, and note any discrepancies. Build a habit of checking events before and after every change. A reliable operational workflow makes failures visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With practice, you will turn Kubernetes events from a source of confusion into a powerful diagnostic tool.