Kubernetes Troubleshooting: Practical Workflow with Real Examples

Intro

Kubernetes failures can be noisy and confusing under pressure. This article gives you a practical, safe-first workflow with concrete examples, the exact commands to run, and small, targeted fixes you can verify quickly. You will learn how to:

• Triage impact and find the failing component fast
• Read events, pod and node logs, and status fields
• Classify failures and run focused root-cause checks
• Apply safe remediations and verify outcomes
• Rehearse everything locally before touching production

Tooling you will see in examples: Docker images and registries, Helm releases, GitLab CI/CD, Nginx Ingress, Ceph-backed StorageClasses, and Prometheus metrics.

Workflow Overview

Use this end-to-end path. Each step favors read-only inspection before any change.

1. Scope and Locate the Issue

Confirm context and namespace:

kubectl config current-context
kubectl get ns
kubectl get pods -A -o wide

Identify the failing workload and component (pod, deployment, statefulset, daemonset, job, cronjob, service, ingress).

2. Collect Fast Signals (Read-Only)

High-level health:

kubectl get nodes -o wide
kubectl get pods -A --field-selector=status.phase!=Running
kubectl get deploy,sts,ds -A

Events tell the story in order:

kubectl get events -A --sort-by=.metadata.creationTimestamp

Describe adds decision context from the scheduler and controllers:

kubectl -n <ns> describe pod <pod>

Logs for the current and previous container attempt:

kubectl -n <ns> logs <pod> -c <container> --tail=200
kubectl -n <ns> logs <pod> -c <container> --previous --tail=200

Service and endpoint wiring:

kubectl -n <ns> get svc,ep
kubectl -n <ns> describe svc <name>

Storage signals:

kubectl -n <ns> get pvc
kubectl -n <ns> describe pvc <pvc>
kubectl get storageclass

Node pressure and kubelet:

kubectl describe node <node>
# If you have node SSH access
sudo journalctl -u kubelet -n 200 --no-pager

3. Classify the Failure Quickly

Common pod statuses and what they hint at:

• Pending: scheduler cannot place the pod (resources, nodeSelector, taints, tolerations, affinity, PVC Pending)
• ContainerCreating: image pull latency, volume mount issues, CNI init, secrets/config mount
• ImagePullBackOff: bad image reference or credentials
• CrashLoopBackOff: app fails repeatedly at start
• OOMKilled: memory limit too low or leaks
• RunContainerError: entrypoint or permission problem
• 0/1 ready (or similar): readiness probe failing
• NodeNotReady: node health, network, kubelet, or cloud issues

4. Root-Cause Checks and Safe Remediations

Below are targeted playbooks with minimal-risk commands first.

A. ImagePullBackOff

Checks:

kubectl -n <ns> describe pod <pod> | sed -n '/Events/,$p'
kubectl -n <ns> get sa <sa> -o yaml
kubectl -n <ns> get secret

Fixes:

# Correct the image name or tag
kubectl -n <ns> set image deploy/<name> <container>=<repo>/<image>:<tag>
kubectl -n <ns> rollout status deploy/<name>

# Add or fix the registry secret
kubectl -n <ns> create secret docker-registry regcred \
  --docker-server=<registry> --docker-username=<user> \
  --docker-password=<pass> --docker-email=<email>
# Reference it in the pod spec under imagePullSecrets

B. CrashLoopBackOff

Checks:

kubectl -n <ns> logs <pod> -c <container> --previous --tail=200
kubectl -n <ns> get pod <pod> -o yaml

Typical causes and fixes:

• Bad configuration or missing env/secret key: fix the ConfigMap/Secret key and rollout restart.

  kubectl -n <ns> rollout restart deploy/<name>
  kubectl -n <ns> rollout status deploy/<name>

• App needs startup time: increase readinessProbe.initialDelaySeconds or timeoutSeconds.
• Bad entrypoint: correct command and args to the expected binary or script.

C. OOMKilled

Checks:

kubectl -n <ns> describe pod <pod> | grep -i -E 'oom|memory'
kubectl top pod -n <ns> <pod>

Fixes:

• Raise memory limit or reduce usage. Ensure requests reflect realistic needs to avoid eviction/scheduling churn.
• For memory spikes, add heap caps (e.g., -Xmx for JVM), worker limits, or enable app-level memory limits.

D. Pending Pod (Unschedulable)

Checks:

kubectl -n <ns> describe pod <pod> | sed -n '/Events/,$p'
kubectl get nodes

Fixes:

• Lower inflated requests or right-size them based on real usage from kubectl top.
• Remove overly strict nodeSelector or adjust labels.
• Add a compatible node pool if the workload legitimately needs more resources.

E. Readiness or Liveness Probe Failing

Checks:

kubectl -n <ns> describe pod <pod>
kubectl -n <ns> logs <pod> -c <container> --tail=200

Fixes:

• Correct path, port, or command. Add initialDelaySeconds for cold starts.
• Ensure your app binds on 0.0.0.0, not 127.0.0.1.
• For HTTP probes, verify the path returns 2xx within timeoutSeconds.

F. Service Has No Endpoints (Traffic Black Hole)

Checks:

kubectl -n <ns> get svc <name> -o yaml
kubectl -n <ns> get pods -l <selector> -o wide --show-labels
kubectl -n <ns> get ep <name> -o yaml

Fixes:

• Align labels and selectors. If using Helm, ensure values render consistent labels on both Service and Deployment.

G. PVC Pending or Mount Failures

Checks:

kubectl -n <ns> describe pvc <pvc>
kubectl get storageclass

Fixes:

• Set the correct storageClassName. Match access mode to workload (ReadWriteOnce for most single-writer cases). Request a realistic size.
• If using Ceph, ensure the chosen StorageClass exists and the cluster can provision volumes (check ceph status and provisioner logs).

H. DNS or Cluster DNS Add-on Issues

Checks:

kubectl -n <ns> run dnsutils --rm -it --image=busybox:1.36 -- nslookup kubernetes.default
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs deploy/coredns --tail=200

Fixes:

• Correct CoreDNS ConfigMap if misconfigured (e.g., stub domains, forward zones).
• Loosen NetworkPolicies to allow UDP/TCP 53 to CoreDNS from your namespaces.

I. Node NotReady or Pressure

Checks:

kubectl describe node <node>

Common issues include DiskPressure, MemoryPressure, PIDPressure, or network agent problems.

Fixes (safe order):

# Cordon the node to stop new scheduling, then drain if you must move pods
kubectl cordon <node>
kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

# After remediation, uncordon
kubectl uncordon <node>

5. Verify, Then Roll Back If Needed

• Check rollout and health:

kubectl -n <ns> rollout status deploy/<name>
kubectl -n <ns> get pods,svc,ep
kubectl -n <ns> get events --sort-by=.metadata.creationTimestamp | tail -n 20

• If the change did not help, undo quickly:

kubectl -n <ns> rollout undo deploy/<name>

• With Helm-managed apps, prefer a dry run before actual changes:

helm -n <ns> upgrade <rel> <chart> --dry-run --values values.yaml

Practical Mini Playbooks

CrashLoopBackOff from Missing ConfigMap Key

Symptom:

kubectl -n app describe pod api-xyz | sed -n '/Events/,$p'
kubectl -n app logs api-xyz -c api --previous --tail=200
# shows: KeyError: 'MISSING_URL'

Fix: Add the missing key to the ConfigMap or update envFrom to the correct name, then restart the rollout.

kubectl -n app apply -f configmap.yaml
kubectl -n app rollout restart deploy/api
kubectl -n app rollout status deploy/api

Service Has No Endpoints Due to Label Mismatch

Symptom:

kubectl -n web get svc nginx
kubectl -n web get ep nginx -o yaml  # empty subsets
kubectl -n web get deploy nginx -o yaml | grep labels -A5

Fix: Align Service selector with Deployment pod template labels, then apply.

kubectl -n web apply -f svc.yaml -f deploy.yaml
kubectl -n web get ep nginx -o yaml  # endpoints now populated

PVC Pending with Wrong StorageClass

Symptom:

kubectl -n data describe pvc db-data | sed -n '/Events/,$p'
# ProvisioningFailed: storageclass "ceph-block" not found

Fix: Set storageClassName to an existing class (for example, ceph-rbd exposed by your cluster), then reapply.

kubectl -n data apply -f pvc.yaml
kubectl -n data get pvc db-data

ImagePullBackOff Due to Missing Credentials

Symptom:

kubectl -n ci describe pod runner-abc | sed -n '/Events/,$p'
# error: unauthorized: authentication required

Fix: Create a docker-registry secret and reference it via imagePullSecrets.

kubectl -n ci create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=ci-user --docker-password='$TOKEN' [email protected]
kubectl -n ci patch sa default -p '{"imagePullSecrets":[{"name":"regcred"}]}'
kubectl -n ci rollout restart deploy/runner

Local Pilot Plan

Practice in a sandbox cluster so you can observe each signal end-to-end.

Scope

• Goal: detect, diagnose, fix, and verify 4 failures in under 30 minutes total.
• Environment: a local cluster (for example, kind or minikube) and kubectl.

Setup

• Deploy a simple app (Nginx Deployment + ClusterIP Service + optional Ingress) and a small API Deployment with ConfigMap and Secret.
• Install metrics-server if you want kubectl top.

Exercises (Induce, Then Fix)

1. Label mismatch

• Break: change Service selector to a non-matching label.
• Detect: endpoints empty. Fix: align labels and selectors.

1. Readiness probe fail

• Break: set readiness path to /bad.
• Detect: probe errors in describe. Fix: correct path and tune initialDelaySeconds.

1. ImagePullBackOff

• Break: change image tag to a non-existent tag.
• Detect: events show pull error. Fix: correct tag or add imagePullSecret.

1. PVC Pending

• Break: request a non-existent StorageClass.
• Detect: PVC events show ProvisioningFailed. Fix: set an existing class.

Measurement

For each exercise, record: time to first signal, root cause, fix applied, and verification command output.

Success criteria: green rollout status and healthy endpoints/pods after each fix.

Safety

• Use only non-destructive commands until you understand the failure.
• Prefer rollout restarts and spec fixes over pod deletes. If deleting, ensure a controller owns the pod so it will be recreated.

Conclusion

You now have a practical, staged troubleshooting workflow and repeatable playbooks for the most common Kubernetes errors. Next steps:

• Turn this workflow into a team runbook with copy-paste commands
• Rehearse quarterly in a sandbox so muscle memory stays fresh
• Add quick preflight checks to your GitLab CI/CD or Helm pipelines (for example, kubectl apply --dry-run=server, helm --dry-run) to catch label, probe, and storage misconfigurations early
• Instrument with Prometheus and alerts for readiness, restart spikes, and PVC provisioning failures

When incidents happen, start with read-only inspection, classify fast, apply the smallest safe fix, and verify. Consistency beats improvisation under pressure.