Kubernetes Pod Configuration Mistakes: A Practical Troubleshooting Guide

Intro

Misconfigured Pods are among the most frequent causes of production incidents in Kubernetes. The symptoms are familiar: Pods stuck in Pending, looping through CrashLoopBackOff, failing readiness probes, or consuming more resources than expected. Yet many teams still troubleshoot by deleting and recreating resources, hoping the next rollout will behave differently.

This guide replaces guesswork with a systematic approach to Kubernetes Pod configuration. It is written for developers, DevOps consultants, and technical startup teams who operate clusters day to day. It focuses on four practical areas: Pod configuration mistakes, Pod validation, Pod rollback, and Pod troubleshooting. For each, you will find concrete kubectl commands, expected outputs, failure signals, and recovery decisions that match the scenario.

The operating principle throughout is safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths before an incident forces a rushed decision.

Version and Environment Inventory

Before touching a Pod, you need to know what you are working with. Version and Environment Inventory means identifying the relevant component (Pod, Deployment, ReplicaSet), the supported version range, prerequisites, a read-only observation, the smallest justified change, and the command or signal that verifies the outcome.

Start by capturing the current state without modifying anything. Record the Kubernetes server version, the namespace where the Pod runs, and the exact image tags in use. This establishes a baseline and prevents confusion when comparing before and after states.

kubectl version --short
kubectl get pods -n payments -o wide
kubectl get deployment -n payments -o yaml | grep -A 2 image:

Example output from a healthy baseline:

Client Version: v1.28.2
Server Version: v1.28.4
NAME                          READY   STATUS    RESTARTS   AGE   IP            NODE
checkout-7d9f8c5b6d-abcde     1/1     Running   0          2m    10.244.1.5    node-1
checkout-7d9f8c5b6d-fghij     1/1     Running   0          2m    10.244.2.7    node-2

To see events, scheduling details, and recent warnings, describe the Pod:

kubectl describe pod checkout-7d9f8c5b6d-abcde -n payments

Pay attention to the Events section at the bottom. A Pod that is stuck in Pending because no node has enough resources will show events like FailedScheduling with messages such as 0/3 nodes are available: 3 Insufficient memory.

Separate observation from intervention. Do not delete or edit the Pod yet. Capture current state and timestamps, protect credentials and private material, then plan one scoped change at a time.

Related resources such as Deployment, ReplicaSet, and Node should be included only when they affect prerequisites, compatibility, security, observability, or recovery. For example, if a Pod is always scheduled to the same node, check node taints and labels, because that can reveal a missing toleration in the Pod spec.

Keep the local test small. Apply one manifest, inspect the generated resources, and verify traffic with kubectl port-forward or a local Service type before moving to a cloud load balancer or ingress controller.

Common Pod Configuration Mistakes

The most frequent configuration errors fall into a few repeating categories. Each mistake below includes a real command to reproduce, a description of the failure, and a fix that has been verified in practice.

Missing or Incorrect Container Port Declaration

A container may listen on port 8080 inside, but if the Pod spec does not declare containerPort, Services cannot route traffic correctly. Worse, if the wrong port is declared, traffic goes nowhere.

apiVersion: v1
kind: Pod
metadata:
  name: webapp
spec:
  containers:
  - name: webapp
    image: myapp:1.2.3
    ports:
    - containerPort: 8080   # the app actually listens on 8080, so this is correct

To verify inside the container, run:

kubectl exec -it webapp -- sh -c 'netstat -tulpn | grep LISTEN'

Expected output if the app listens on 8080:

tcp        0      0 :::8080                 :::*                    LISTEN      -

If the output shows a different port or no listener, fix the containerPort declaration in the manifest.

Resource Limits Missing or Too Tight

Without resource requests and limits, a Pod can be evicted when a node runs out of memory, or it can starve other workloads. Conversely, limits that are too low cause OOMKilled or CPU throttling.

Set realistic values based on observed usage:

resources:
  requests:
    memory: "128Mi"
    cpu: "250m"
  limits:
    memory: "256Mi"
    cpu: "500m"

To check current usage, use kubectl top:

kubectl top pod -n payments

Example output:

NAME                          CPU(cores)   MEMORY(bytes)
checkout-7d9f8c5b6d-abcde     120m         150Mi
checkout-7d9f8c5b6d-fghij     98m          142Mi

If memory usage is consistently near 256Mi, increase the limit to 512Mi and update the request accordingly. If a Pod was OOMKilled, check the termination reason:

kubectl describe pod checkout-7d9f8c5b6d-abcde -n payments | grep -A 5 'Last State'

Look for Reason: OOMKilled. The fix is to raise the memory limit or reduce the workload footprint.

Probes Misconfigured

Liveness and readiness probes are essential, but wrong configurations cause restarts or traffic loss. A common mistake is using a TCP probe on an HTTP endpoint, or setting initialDelaySeconds too low for applications with a slow startup.

Example of a problematic HTTP probe:

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 5   # too short for an app that takes 15 seconds to start
  periodSeconds: 10

With this probe, the container may never become ready before the liveness probe starts failing, causing CrashLoopBackOff. Increase initialDelaySeconds to 30 and periodSeconds to 20.

To observe probe failures, describe the Pod and look for Unhealthy events:

kubectl describe pod webapp -n payments | grep -A 10 Events

If you see Liveness probe failed: HTTP probe failed with statuscode: 500, the application itself is failing; if it is connection refused, the probe is hitting the wrong port.

Hardcoded Environment Variables or Secrets in Plain Text

Putting database passwords or API keys directly in the Pod spec is a serious mistake. Use Kubernetes Secrets and reference them via valueFrom.

Wrong approach:

env:
- name: DB_PASSWORD
  value: "Sup3rSecret!"   # never do this

Correct approach:

env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: app-secrets
      key: db-password

Create the Secret with a placeholder first, then inject the real value via an external secrets manager or a sealed secret.

kubectl create secret generic app-secrets --from-literal=db-password='CHANGE_ME'

Verify the Pod can read the Secret without printing it:

kubectl exec webapp -- printenv DB_PASSWORD

Do not echo the value in logs or commands that end up in shell history. Use a masked value if you must confirm existence.

Incorrect Image Pull Policy or Missing Registry Authentication

If your image is private, a Pod may fail with ImagePullBackOff because the cluster cannot authenticate. The image pull policy also matters: Always pulls every time, which is safe for mutable tags but slow; IfNotPresent is faster for immutable tags.

imagePullPolicy: IfNotPresent   # use with immutable tags
imagePullSecrets:
- name: regcred

Create the pull secret:

kubectl create secret docker-registry regcred \
  --docker-server=registry.example.com \
  --docker-username=deploy \
  --docker-password=xxxx \
  [email protected]

If a Pod is in ImagePullBackOff, check the events:

kubectl describe pod webapp | grep -A 5 Events

Look for messages like failed to authorize: failed to fetch anonymous token or pull access denied. Fix the secret or image tag and then delete the Pod to force a new pull.

Safe Configuration Path

The Safe Configuration Path is a sequence that reduces risk at every step. It consists of five phases: observe, plan, change, verify, and document. For Kubernetes Pod configuration, this means naming the relevant component, supported version, prerequisites, a read-only observation, the smallest justified change, and the verification command.

Phase 1: Observe

Do not change anything yet. Collect the current state:

kubectl get pods -n payments -o wide
kubectl describe pod <pod-name> -n payments
kubectl logs <pod-name> --previous -n payments   # if the container restarted
kubectl rollout status deployment/checkout -n payments

Example output when a rollout is stuck:

Waiting for deployment "checkout" rollout to finish: 1 old replicas are pending termination...

Phase 2: Plan

Identify the exact field to change. If a Pod is in CrashLoopBackOff, inspect the previous logs first:

kubectl logs checkout-7d9f8c5b6d-abcde --previous -n payments

Suppose the log shows:

panic: runtime error: invalid memory address or nil pointer dereference

That points to an application bug, not a configuration issue. But if the log shows Error: could not connect to database at postgres:5432, the configuration is likely wrong: the database hostname or credentials may be incorrect.

Choose the smallest change that addresses the observed failure. If it is a configuration error, edit the Deployment (not the Pod directly, since Pods are ephemeral):

kubectl edit deployment checkout -n payments

Phase 3: Change

Apply the change via a merge patch to avoid overwriting unrelated fields:

kubectl patch deployment checkout -n payments -p '{"spec":{"template":{"spec":{"containers":[{"name":"checkout","env":[{"name":"DB_HOST","value":"postgres.internal"}]}]}}}}'

Then watch the rollout:

kubectl rollout status deployment/checkout -n payments

Expected output on success:

deployment "checkout" successfully rolled out

Phase 4: Verify

Confirm the new Pod is running and ready, and that traffic flows. Use port-forward to test locally:

kubectl port-forward deployment/checkout 8080:8080 -n payments
curl http://localhost:8080/healthz

Expected:

{"status":"ok"}

Phase 5: Document

Record what changed, why, and how to roll back. For example, note in your runbook: "Increased liveness probe initialDelaySeconds from 5 to 30 for checkout deployment due to slow startup. Rollback command: kubectl rollout undo deployment/checkout -n payments"

Verification and Diagnostics

Verification is not a single kubectl get after a rollout. It is a sequence of checks that confirm the Pod is healthy from the inside out: process, resource usage, network reachability, and application-level response.

Check Pod Status and Readiness

Start with the basics:

kubectl get pods -n payments -o wide

All Pods should be Running and READY 1/1. If a Pod is Running but 0/1, readiness is failing. Describe to see why:

kubectl describe pod checkout-7d9f8c5b6d-xyz -n payments

Look for events like Readiness probe failed: Get "http://10.244.1.6:8080/ready": dial tcp 10.244.1.6:8080: connect: connection refused.

Inspect Logs for Errors

kubectl logs checkout-7d9f8c5b6d-xyz -n payments --tail=50

If the container has restarted, include the previous instance:

kubectl logs checkout-7d9f8c5b6d-xyz -n payments --previous

Filter for errors with grep:

kubectl logs checkout-7d9f8c5b6d-xyz -n payments --previous | grep -i error

Test Network Connectivity from Inside the Pod

kubectl exec -it checkout-7d9f8c5b6d-xyz -n payments -- sh
# inside the Pod
ping -c 3 postgres.internal   # if ping is installed
nc -zv postgres.internal 5432

If the Pod can resolve the DNS name but cannot connect to the port, there may be a NetworkPolicy blocking traffic.

Validate Resource Usage Against Limits

kubectl top pod checkout-7d9f8c5b6d-xyz -n payments --containers

Example output:

POD                          NAME        CPU(cores)   MEMORY(bytes)
checkout-7d9f8c5b6d-xyz      checkout    75m          198Mi

If memory is near the limit (say 256Mi limit, using 198Mi), consider raising the limit before the next traffic spike.

Check the Deployment's Rollout History

kubectl rollout history deployment/checkout -n payments

Example:

deployment.apps/checkout
REVISION  CHANGE-CAUSE
1         <none>
2         kubectl set image deployment/checkout checkout=myapp:1.2.4

This history is critical for rollback decisions.

Failure Modes and Recovery

Every Pod configuration can fail in predictable ways. Knowing the typical failure modes and their recovery commands shortens incident time.

CrashLoopBackOff

Symptom: Pod restarts repeatedly, STATUS shows CrashLoopBackOff, and RESTARTS count climbs.

Diagnosis:

kubectl describe pod webapp -n payments | grep -A 10 'Last State'
kubectl logs webapp --previous -n payments

Common causes: application crash at startup, missing configuration file, wrong command or arguments, OOMKilled.

Recovery: fix the root cause (e.g., add the missing ConfigMap, increase memory limit), then delete the Pod to restart cleanly:

kubectl delete pod webapp -n payments

If the Pod is managed by a Deployment, it will be recreated automatically.

ImagePullBackOff

Symptom: Pod cannot pull the image, status ImagePullBackOff or ErrImagePull.

Diagnosis:

kubectl describe pod webapp -n payments | grep -A 10 Events

Look for messages like failed to pull image "myapp:latest": pull access denied or repository does not exist.

Recovery: correct the image name or tag, ensure the pull secret exists and is referenced by the Pod spec. Then delete the Pod.

Pending Due to Insufficient Resources

Symptom: Pod stays in Pending, READY is 0/1, and events show FailedScheduling.

Diagnosis:

kubectl describe pod bigdata -n analytics | grep -A 5 Events

Example message: 0/3 nodes are available: 3 Insufficient cpu.

Recovery: reduce the resource requests, scale down other workloads, or add nodes to the cluster. If the request is justified, increase cluster capacity.

Readiness Probe Failing

Symptom: Pod is Running but not Ready; Service does not route traffic to it.

Diagnosis:

kubectl get pods -n payments
kubectl describe pod webapp -n payments | grep -A 10 Events

Look for Readiness probe failed: HTTP probe failed with statuscode: 503.

Recovery: fix the probe path, port, or delay. For an application that takes 30 seconds to become ready, set initialDelaySeconds: 30 and periodSeconds: 10.

OOMKilled

Symptom: Pod terminates with reason OOMKilled, exit code 137.

Diagnosis:

kubectl describe pod webapp -n payments | grep -A 5 'Last State'

Output shows Reason: OOMKilled.

Recovery: increase the memory limit or reduce memory usage. Use a vertical scaling tool or adjust the application memory settings. Then roll out a new revision.

Rollback Recovery

If a recent change caused the failure, roll back to a previous revision:

kubectl rollout undo deployment/checkout -n payments
kubectl rollout status deployment/checkout -n payments

To roll back to a specific revision:

kubectl rollout undo deployment/checkout --to-revision=1 -n payments

Always verify after rollback:

kubectl get pods -n payments -o wide
kubectl logs deployment/checkout --tail=20 -n payments

Operations Checklist

Use this checklist to standardize Pod configuration troubleshooting and changes. Each item includes a concrete example and the expected result.

1. Record cluster and namespace context.

• Command: kubectl config current-context && kubectl get ns payments
• Expected: context name printed and namespace exists.

1. Capture Pod baseline.

• Command: kubectl get pods -n payments -o wide --show-labels
• Expected: all Pods listed with labels and IPs.

1. Describe the failing Pod and save events to a file.

• Command: kubectl describe pod webapp-abc123 -n payments > /tmp/webapp-events.txt
• Expected: file contains recent events like probe failures or scheduling errors.

1. Check previous container logs if restarted.

• Command: kubectl logs webapp-abc123 --previous -n payments | tail -50
• Expected: startup errors or crash stack traces visible.

1. Validate the Pod spec against the desired configuration.

• Compare the running spec with your source manifest: kubectl get pod webapp-abc123 -n payments -o yaml | diff - desired-pod.yaml
• Expected: only expected differences (e.g., default fields).

1. Apply the smallest change via patch, not full replace.

• Example: kubectl patch deployment webapp -n payments -p '{"spec":{"template":{"spec":{"containers":[{"name":"webapp","resources":{"limits":{"memory":"512Mi"}}}]}}}}'
• Expected: patch succeeds with deployment.apps/webapp patched.

1. Watch rollout status.

• Command: kubectl rollout status deployment/webapp -n payments
• Expected: deployment "webapp" successfully rolled out within timeout.

1. Verify Pod readiness and application health.

• Commands: kubectl get pods -n payments, then kubectl port-forward deployment/webapp 8080:8080 -n payments and curl localhost:8080/healthz.
• Expected: Pods 1/1 Running, health endpoint returns 200 OK.

1. Confirm resource usage is within limits.

• Command: kubectl top pod -n payments
• Expected: memory and CPU below limits, with headroom.

1. Document the change and rollback command.

• Example note: "Increased webapp memory limit from 256Mi to 512Mi to prevent OOMKilled. Rollback: kubectl rollout undo deployment/webapp -n payments."
• Expected: entry in team runbook or incident log.

Conclusion

Kubernetes Pod configuration mistakes are inevitable, but they do not have to cause extended downtime. The practical examples in this guide showed how to observe Pod state before changing anything, diagnose common failure modes with specific kubectl commands, apply the smallest safe change, and verify the result before declaring victory.

Every recommendation here is version-scoped, observable, and reversible where Kubernetes permits. Copying a command without checking prerequisites and expected output is not an operations procedure; it is a gamble.

As a next step, choose one low-risk verification from the Operations Checklist, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as Deployment, ReplicaSet, and Node in your own cluster. Then pick the most frequent Pod configuration mistake in your environment and build a runbook entry for it using the Safe Configuration Path phases.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Your future on-call self will thank you.