## Intro

Kubernetes failures can be noisy and confusing under pressure. This article gives you a practical, safe-first workflow with concrete examples, the exact commands to run, and small, targeted fixes you can verify quickly. You will learn how to:

- Triage impact and find the failing component fast

- Read events, pod and node logs, and status fields

- Classify failures and run focused root-cause checks

- Apply safe remediations and verify outcomes

- Rehearse everything locally before touching production

Tooling you will see in examples: Docker images and registries, Helm releases, GitLab CI/CD, Nginx Ingress, Ceph-backed StorageClasses, and Prometheus metrics.

## Workflow Overview

Use this end-to-end path. Each step favors read-only inspection before any change.

### 1. Scope and Locate the Issue

Confirm context and namespace:

kubectl config current-context
kubectl get ns
kubectl get pods -A -o wide 
 Identify the failing workload and component (pod, deployment, statefulset, daemonset, job, cronjob, service, ingress).

### 2. Collect Fast Signals (Read-Only)

High-level health:

kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get deploy,sts,ds -A 
 Events tell the story in order:

kubectl get events -A --sort-by=.metadata.creationTimestamp 
 Describe adds decision context from the scheduler and controllers:

kubectl -n <ns> describe pod <pod> 
 Logs for the current and previous container attempt:

kubectl -n <ns> logs <pod> -c <container> --tail=200
kubectl -n <ns> logs <pod> -c <container> --previous --tail=200 
 Service and endpoint wiring:

kubectl -n <ns> get svc,ep
kubectl -n <ns> describe svc <name> 
 Storage signals:

kubectl -n <ns> get pvc
kubectl -n <ns> describe pvc <pvc>
kubectl get storageclass 
 Node pressure and kubelet:

kubectl describe node <node>
# If you have node SSH access
sudo journalctl -u kubelet -n 200 --no-pager 

### 3. Classify the Failure Quickly

 Common pod statuses and what they hint at:

- Pending : scheduler cannot place the pod (resources, nodeSelector, taints, tolerations, affinity, PVC Pending)

- ContainerCreating : image pull latency, volume mount issues, CNI init, secrets/config mount

- ImagePullBackOff : bad image reference or credentials

- CrashLoopBackOff : app fails repeatedly at start

- OOMKilled : memory limit too low or leaks

- RunContainerError : entrypoint or permission problem

- 0/1 ready (or similar) : readiness probe failing

- NodeNotReady : node health, network, kubelet, or cloud issues

### 4. Root-Cause Checks and Safe Remediations

Below are targeted playbooks with minimal-risk commands first.

### A. ImagePullBackOff

Checks:

kubectl -n <ns> describe pod <pod> | sed -n '/Events/,$p'
kubectl -n <ns> get sa <sa> -o yaml
kubectl -n <ns> get secret 
 Fixes:

# Correct the image name or tag
kubectl -n <ns> set image deploy/<name> <container>=<repo>/<image>:<tag>
kubectl -n <ns> rollout status deploy/<name>

# Add or fix the registry secret
kubectl -n <ns> create secret docker-registry regcred \
 --docker-server=<registry> --docker-username=<user> \
 --docker-password=<pass> --docker-email=<email>
# Reference it in the pod spec under imagePullSecrets 

### B. CrashLoopBackOff

 Checks:

kubectl -n <ns> logs <pod> -c <container> --previous --tail=200
kubectl -n <ns> get pod <pod> -o yaml 
 Typical causes and fixes:

- Bad configuration or missing env/secret key: fix the ConfigMap/Secret key and rollout restart.

kubectl -n <ns> rollout restart deploy/<name>
 kubectl -n <ns> rollout status deploy/<name> 

- App needs startup time: increase readinessProbe.initialDelaySeconds or timeoutSeconds .

- Bad entrypoint: correct command and args to the expected binary or script.

### C. OOMKilled

 Checks:

kubectl -n <ns> describe pod <pod> | grep -i -E 'oom|memory'
kubectl top pod -n <ns> <pod> 
 Fixes:

- Raise memory limit or reduce usage. Ensure requests reflect realistic needs to avoid eviction/scheduling churn.

- For memory spikes, add heap caps (e.g., -Xmx for JVM), worker limits, or enable app-level memory limits.

### D. Pending Pod (Unschedulable)

Checks:

kubectl -n <ns> describe pod <pod> | sed -n '/Events/,$p'
kubectl get nodes 
 Fixes:

- Lower inflated requests or right-size them based on real usage from kubectl top .

- Remove overly strict nodeSelector or adjust labels.

- Add a compatible node pool if the workload legitimately needs more resources.

### E. Readiness or Liveness Probe Failing

Checks:

kubectl -n <ns> describe pod <pod>
kubectl -n <ns> logs <pod> -c <container> --tail=200 
 Fixes:

- Correct path, port, or command. Add initialDelaySeconds for cold starts.

- Ensure your app binds on 0.0.0.0 , not 127.0.0.1 .

- For HTTP probes, verify the path returns 2xx within timeoutSeconds .

### F. Service Has No Endpoints (Traffic Black Hole)

Checks:

kubectl -n <ns> get svc <name> -o yaml
kubectl -n <ns> get pods -l <selector> -o wide --show-labels
kubectl -n <ns> get ep <name> -o yaml 
 Fixes:

- Align labels and selectors. If using Helm, ensure values render consistent labels on both Service and Deployment.

### G. PVC Pending or Mount Failures

Checks:

kubectl -n <ns> describe pvc <pvc>
kubectl get storageclass 
 Fixes:

- Set the correct storageClassName . Match access mode to workload (ReadWriteOnce for most single-writer cases). Request a realistic size.

- If using Ceph, ensure the chosen StorageClass exists and the cluster can provision volumes (check ceph status and provisioner logs).

### H. DNS or Cluster DNS Add-on Issues

Checks:

kubectl -n <ns> run dnsutils --rm -it --image=busybox:1.36 -- nslookup kubernetes.default
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs deploy/coredns --tail=200 
 Fixes:

- Correct CoreDNS ConfigMap if misconfigured (e.g., stub domains, forward zones).

- Loosen NetworkPolicies to allow UDP/TCP 53 to CoreDNS from your namespaces.

### I. Node NotReady or Pressure

Checks:

kubectl describe node <node> 
 Common issues include DiskPressure , MemoryPressure , PIDPressure , or network agent problems.

Fixes (safe order):

# Cordon the node to stop new scheduling, then drain if you must move pods
kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# After remediation, uncordon
kubectl uncordon <node> 

### 5. Verify, Then Roll Back If Needed

- Check rollout and health:

 kubectl -n <ns> rollout status deploy/<name>
kubectl -n <ns> get pods,svc,ep
kubectl -n <ns> get events --sort-by=.metadata.creationTimestamp | tail -n 20 

- If the change did not help, undo quickly:

 kubectl -n <ns> rollout undo deploy/<name> 

- With Helm-managed apps, prefer a dry run before actual changes:

 helm -n <ns> upgrade <rel> <chart> --dry-run --values values.yaml 

## Practical Mini Playbooks

### CrashLoopBackOff from Missing ConfigMap Key

 Symptom:

kubectl -n app describe pod api-xyz | sed -n '/Events/,$p'
kubectl -n app logs api-xyz -c api --previous --tail=200
# shows: KeyError: 'MISSING_URL' 
 Fix: Add the missing key to the ConfigMap or update envFrom to the correct name, then restart the rollout.

kubectl -n app apply -f configmap.yaml
kubectl -n app rollout restart deploy/api
kubectl -n app rollout status deploy/api 

### Service Has No Endpoints Due to Label Mismatch

 Symptom:

kubectl -n web get svc nginx
kubectl -n web get ep nginx -o yaml # empty subsets
kubectl -n web get deploy nginx -o yaml | grep labels -A5 
 Fix: Align Service selector with Deployment pod template labels, then apply.

kubectl -n web apply -f svc.yaml -f deploy.yaml
kubectl -n web get ep nginx -o yaml # endpoints now populated 

### PVC Pending with Wrong StorageClass

 Symptom:

kubectl -n data describe pvc db-data | sed -n '/Events/,$p'
# ProvisioningFailed: storageclass "ceph-block" not found 
 Fix: Set storageClassName to an existing class (for example, ceph-rbd exposed by your cluster), then reapply.

kubectl -n data apply -f pvc.yaml
kubectl -n data get pvc db-data 

### ImagePullBackOff Due to Missing Credentials

 Symptom:

kubectl -n ci describe pod runner-abc | sed -n '/Events/,$p'
# error: unauthorized: authentication required 
 Fix: Create a docker-registry secret and reference it via imagePullSecrets .

kubectl -n ci create secret docker-registry regcred \
 --docker-server=registry.example.com \
 --docker-username=ci-user --docker-password='$TOKEN' --docker-email=ci@example.com
kubectl -n ci patch sa default -p '{"imagePullSecrets":[{"name":"regcred"}]}'
kubectl -n ci rollout restart deploy/runner 

## Local Pilot Plan

 Practice in a sandbox cluster so you can observe each signal end-to-end.

### Scope

- Goal: detect, diagnose, fix, and verify 4 failures in under 30 minutes total.

- Environment: a local cluster (for example, kind or minikube) and kubectl.

### Setup

- Deploy a simple app (Nginx Deployment + ClusterIP Service + optional Ingress) and a small API Deployment with ConfigMap and Secret.

- Install metrics-server if you want kubectl top .

### Exercises (Induce, Then Fix)

- Label mismatch

- Break: change Service selector to a non-matching label.

- Detect: endpoints empty. Fix: align labels and selectors.

- Readiness probe fail

- Break: set readiness path to /bad .

- Detect: probe errors in describe. Fix: correct path and tune initialDelaySeconds .

- ImagePullBackOff

- Break: change image tag to a non-existent tag.

- Detect: events show pull error. Fix: correct tag or add imagePullSecret .

- PVC Pending

- Break: request a non-existent StorageClass.

- Detect: PVC events show ProvisioningFailed . Fix: set an existing class.

### Measurement

For each exercise, record: time to first signal, root cause, fix applied, and verification command output.

Success criteria: green rollout status and healthy endpoints/pods after each fix.

### Safety

- Use only non-destructive commands until you understand the failure.

- Prefer rollout restarts and spec fixes over pod deletes. If deleting, ensure a controller owns the pod so it will be recreated.

## Conclusion

You now have a practical, staged troubleshooting workflow and repeatable playbooks for the most common Kubernetes errors. Next steps:

- Turn this workflow into a team runbook with copy-paste commands

- Rehearse quarterly in a sandbox so muscle memory stays fresh

- Add quick preflight checks to your GitLab CI/CD or Helm pipelines (for example, kubectl apply --dry-run=server , helm --dry-run ) to catch label, probe, and storage misconfigurations early

- Instrument with Prometheus and alerts for readiness, restart spikes, and PVC provisioning failures

When incidents happen, start with read-only inspection, classify fast, apply the smallest safe fix, and verify. Consistency beats improvisation under pressure.