Kubernetes troubleshooting with practical examples: practical implementation guide

Intro

Kubernetes failures can be noisy and confusing under pressure. This article gives you a practical, safe-first workflow with concrete examples, the exact commands to run, and small, targeted fixes you can verify quickly. You will learn how to:

• Triage impact and find the failing component fast
• Read events, pod and node logs, and status fields
• Classify failures and run focused root-cause checks
• Apply safe remediations and verify outcomes
• Rehearse everything locally before touching production

Related tooling you may touch in examples: Docker images and registries, Helm releases, GitLab CI/CD, Nginx services, Ceph-backed StorageClasses, and Prometheus metrics.

Workflow Overview

Use this end-to-end path. Each step favors read-only inspection before any change.

1. Scope and locate the issue

• Confirm context and namespace:

  kubectl config current-context
  kubectl get ns
  kubectl get pods -A -o wide

• Identify the failing workload and component (pod, deployment, statefulset, daemonset, job, cronjob, service, ingress).

1. Collect fast signals (read-only)

• High-level health:

  kubectl get nodes -o wide
  kubectl get pods -A --field-selector=status.phase!=Running
  kubectl get deploy, sts, ds -A

• Events tell the story in order:

  kubectl get events -A --sort-by=.metadata.creationTimestamp

• Describe adds decision context from the scheduler and controllers:

  kubectl -n <ns> describe pod <pod>

• Logs for the current and previous container attempt:

  kubectl -n <ns> logs <pod> -c <container> --tail=200
  kubectl -n <ns> logs <pod> -c <container> --previous --tail=200

• Service and endpoint wiring:

  kubectl -n <ns> get svc, ep
  kubectl -n <ns> describe svc <name>

• Storage signals:

  kubectl -n <ns> get pvc
  kubectl -n <ns> describe pvc <pvc>
  kubectl get storageclass

• Node pressure and kubelet:

  kubectl describe node <node>
  # If you have node SSH access
  sudo journalctl -u kubelet -n 200 --no-pager

1. Classify the failure quickly

Common pod statuses and what they hint at:

• Pending: scheduler cannot place the pod (resources, nodeSelector, taints, tolerations, affinity, PVC Pending)
• ContainerCreating: image pull latency, volume mount issues, CNI init, secrets/config mount
• ImagePullBackOff: bad image reference or credentials
• CrashLoopBackOff: app fails repeatedly at start
• OOMKilled: memory limit too low or leaks
• RunContainerError: entrypoint or permission problem
• 0/1 ready (or similar): readiness probe failing
• NodeNotReady: node health, network, kubelet, or cloud issues

1. Root-cause checks and safe remediations

Below are targeted playbooks with minimal-risk commands first.

A. ImagePullBackOff

Checks:

• Events and describe will show the registry error. Verify the image string.

  kubectl -n <ns> describe pod <pod> | sed -n '/Events/,$p'

• Confirm image exists and the tag is correct. If private, verify imagePullSecrets on the service account or pod spec.

  kubectl -n <ns> get sa <sa> -o yaml
  kubectl -n <ns> get secret

Fixes:

• Correct the image name or tag in Deployment/StatefulSet, then rollout:

  kubectl -n <ns> set image deploy/<name> <container>=<repo>/<image>:<tag>
  kubectl -n <ns> rollout status deploy/<name>

• Add or fix the registry secret:

  kubectl -n <ns> create secret docker-registry regcred \
    --docker-server=<registry> --docker-username=<user> \
    --docker-password=<pass> --docker-email=<email>
  # Reference it in the pod spec under imagePullSecrets

B. CrashLoopBackOff

Checks:

• Get previous logs to catch the crash reason:

  kubectl -n <ns> logs <pod> -c <container> --previous --tail=200

• Inspect probes, command/args, env, volumes, workingDir, and securityContext:

  kubectl -n <ns> get pod <pod> -o yaml

Typical causes and fixes:

• Bad configuration or missing env/secret key: fix the ConfigMap/Secret key and rollout restart.

  kubectl -n <ns> rollout restart deploy/<name>
  kubectl -n <ns> rollout status deploy/<name>

• App needs startup time: increase readiness/liveness initialDelaySeconds or timeouts.
• Bad entrypoint: correct command/args to the expected binary or script.

C. OOMKilled

Checks:

• Confirm termination reason and memory usage:

  kubectl -n <ns> describe pod <pod> | grep -i -E 'oom|memory'
  kubectl top pod -n <ns> <pod>

Fixes:

• Raise memory limit or reduce usage. Ensure requests reflect realistic needs to avoid eviction/scheduling churn.
• For memory spikes, add heap or worker caps, or enable app-level limits.

D. Pending pod (unschedulable)

Checks:

• Scheduling events tell you why it cannot place the pod:

  kubectl -n <ns> describe pod <pod> | sed -n '/Events/,$p'
  kubectl get nodes

• Look for resource requests exceeding cluster capacity, missing tolerations, or node selectors that match no nodes.

Fixes:

• Lower inflated requests or right-size them based on real usage.
• Remove overly strict node selectors or adjust labels.
• Add a compatible node pool if the workload legitimately needs more resources.

E. Readiness or liveness probe failing

Checks:

• See probe failure details in describe and logs.

  kubectl -n <ns> describe pod <pod>
  kubectl -n <ns> logs <pod> -c <container> --tail=200

• Exec probe: ensure command is present and executable. HTTP probe: confirm path and port. TCP probe: confirm the port is listening.

Fixes:

• Correct path, port, or command. Add initialDelaySeconds for cold starts. Ensure your app binds on 0.0.0.0, not 127.0.0.1.

F. Service has no endpoints (traffic black hole)

Checks:

• Compare Service selector with pod labels:

  kubectl -n <ns> get svc <name> -o yaml
  kubectl -n <ns> get pods -l <selector> -o wide --show-labels
  kubectl -n <ns> get ep <name> -o yaml

Fixes:

• Align labels and selectors. If using Helm, ensure values render consistent labels on both Service and Deployment.

G. PVC Pending or Mount failures

Checks:

• PVC events show provisioner errors:

  kubectl -n <ns> describe pvc <pvc>
  kubectl get storageclass

• Verify accessModes and storageClassName match the provisioner (for example, RBD vs filesystem). Confirm quota and capacity.

Fixes:

• Set the correct storageClassName. Match access mode to workload (ReadWriteOnce for most single-writer cases). Request a realistic size.
• If using Ceph, ensure the chosen StorageClass exists and the cluster can provision volumes.

H. DNS or cluster DNS add-on issues

Checks:

• Test resolution from a throwaway pod:

  kubectl -n <ns> run dnsutils --rm -it --image=busybox:1.36 -- nslookup kubernetes.default

• Inspect CoreDNS pods and logs:

  kubectl -n kube-system get pods -l k8s-app=kube-dns
  kubectl -n kube-system logs deploy/coredns --tail=200

• Ensure NetworkPolicies are not blocking UDP/TCP 53 to CoreDNS.

Fixes:

• Correct CoreDNS configmap if misconfigured. Loosen policies to allow DNS from your namespaces.

I. Node NotReady or pressure

Checks:

• Node conditions and taints:

  kubectl describe node <node>

• Common issues include DiskPressure, MemoryPressure, PIDPressure, or network agent problems.

Fixes (safe order):

• Cordon the node to stop new scheduling, then drain if you must move pods:

  kubectl cordon <node>
  kubectl drain <node> --ignore-daemonsets --delete-emptydir-data

• After remediation, uncordon:

  kubectl uncordon <node>

1. Verify, then roll back if needed

• Check rollout and health:

  kubectl -n <ns> rollout status deploy/<name>
  kubectl -n <ns> get pods, svc, ep
  kubectl -n <ns> get events --sort-by=.metadata.creationTimestamp | tail -n 20

• If the change did not help, undo quickly:

  kubectl -n <ns> rollout undo deploy/<name>

• With Helm-managed apps, prefer a dry run before actual changes:

  helm -n <ns> upgrade <rel> <chart> --dry-run --values values.yaml

Practical mini playbooks

1. CrashLoopBackOff from missing ConfigMap key

• Symptom:

  kubectl -n app describe pod api-xyz | sed -n '/Events/,$p'
  kubectl -n app logs api-xyz -c api --previous --tail=200
  # shows: KeyError: MISSING_URL

• Fix: add the missing key to the ConfigMap or update envFrom to the correct name, then restart the rollout.

  kubectl -n app apply -f configmap.yaml
  kubectl -n app rollout restart deploy/api
  kubectl -n app rollout status deploy/api

1. Service has no endpoints due to label mismatch

• Symptom:

  kubectl -n web get svc nginx
  kubectl -n web get ep nginx -o yaml  # empty subsets
  kubectl -n web get deploy nginx -o yaml | grep labels -n

• Fix: align Service selector with Deployment pod template labels, then apply.

  kubectl -n web apply -f svc.yaml -f deploy.yaml
  kubectl -n web get ep nginx -o yaml  # endpoints now populated

1. PVC Pending with wrong StorageClass

• Symptom:

  kubectl -n data describe pvc db-data | sed -n '/Events/,$p'
  # ProvisioningFailed: storageclass "ceph-block" not found

• Fix: set storageClassName to an existing class, for example "ceph-rbd" exposed by your cluster, then reapply.

  kubectl -n data apply -f pvc.yaml
  kubectl -n data get pvc db-data

1. ImagePullBackOff due to missing credentials

• Symptom:

  kubectl -n ci describe pod runner-abc | sed -n '/Events/,$p'
  # error: unauthorized: authentication required

• Fix: create a docker-registry secret and reference it via imagePullSecrets.

  kubectl -n ci create secret docker-registry regcred \
    --docker-server=registry.example.com \
    --docker-username=ci-user --docker-password='$TOKEN' [email protected]
  kubectl -n ci patch sa default -p '{"imagePullSecrets":[{"name":"regcred"}]}'
  kubectl -n ci rollout restart deploy/runner

Local Pilot Plan

Practice in a sandbox cluster so you can observe each signal end-to-end.

Scope

• Goal: detect, diagnose, fix, and verify 4 failures in under 30 minutes total.
• Environment: a local cluster (for example, kind or minikube) and kubectl.

Setup

• Deploy a simple app (Nginx Deployment + ClusterIP Service + optional Ingress) and a small API Deployment with ConfigMap and Secret.
• Install metrics-server if you want kubectl top.

Exercises (induce, then fix)

1. Label mismatch

• Break: change Service selector to a non-matching label.
• Detect: endpoints empty. Fix: align labels and selectors.

1. Readiness probe fail

• Break: set readiness path to /bad.
• Detect: probe errors in describe. Fix: correct path and tune initialDelaySeconds.

1. ImagePullBackOff

• Break: change image tag to a non-existent tag.
• Detect: events show pull error. Fix: correct tag or add imagePullSecret.

1. PVC Pending

• Break: request a non-existent StorageClass.
• Detect: PVC events show ProvisioningFailed. Fix: set an existing class.

Measurement

• For each exercise, record: time to first signal, root cause, fix applied, and verification command output.
• Success criteria: green rollout status and healthy endpoints/pods after each fix.

Safety

• Use only non-destructive commands until you understand the failure.
• Prefer rollout restarts and spec fixes over pod deletes. If deleting, ensure a controller owns the pod so it will be recreated.

Conclusion

You now have a practical, staged troubleshooting workflow and repeatable playbooks for the most common Kubernetes errors. Next steps:

• Turn this workflow into a team runbook with copy-paste commands
• Rehearse quarterly in a sandbox so muscle memory stays fresh
• Add quick preflight checks to your GitLab CI/CD or Helm pipelines (for example, kubectl apply --dry-run=server, helm --dry-run) to catch label, probe, and storage misconfigurations early
• Instrument with Prometheus and alerts for readiness, restart spikes, and PVC provisioning failures

When incidents happen, start with read-only inspection, classify fast, apply the smallest safe fix, and verify. Consistency beats improvisation under pressure.