Kubernetes Deployment Errors: Causes, Diagnosis, and Safe Fixes

Intro

Kubernetes Deployment errors can halt releases, exhaust on-call engineers, and undermine confidence in the platform. The good news is that most failures follow a small set of patterns: misconfigured container images, missing resources, broken probes, and rollouts that move too fast. The bad news is that without a systematic approach, you can waste hours guessing at YAML indentation or chasing logs that do not point to the real problem.

This article gives you a repeatable diagnosis workflow, not a collection of copy-paste commands. You will learn how to inventory the environment safely, inspect a Deployment without making changes, interpret the most common error messages, and recover with minimal blast radius. Every recommendation includes a concrete command, the expected output, and a fallback plan. The target audience is developers who own their services, DevOps consultants who must fix clusters they did not build, and startup teams where one person wears many hats.

Operational safety is the underlying principle: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the outcome, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before touching a broken Deployment, you need to know what you are dealing with. Kubernetes behavior changes between versions; a fix that works on 1.28 may not apply to 1.30. Start with read-only observations.

Example inventory commands:

kubectl version --short
# Client Version: v1.30.2
# Server Version: v1.30.0

kubectl get deployment -n production
# NAME        READY   UP-TO-DATE   AVAILABLE   AGE
# web-app     0/3     3            0           15m

kubectl get pods -n production -l app=web-app -o wide
# NAME                      READY   STATUS             RESTARTS   AGE   IP       NODE
# web-app-6d4b7c9f5-abcde   0/1     ImagePullBackOff   0          5m    <none>   node-pool-1
# web-app-6d4b7c9f5-fghij   0/1     ImagePullBackOff   0          5m    <none>   node-pool-2
# web-app-6d4b7c9f5-klmno   0/1     ImagePullBackOff   0          5m    <none>   node-pool-3

Notice the ImagePullBackOff status. This tells you the Pod could not pull the container image, but not why. To find out, describe the Pod:

kubectl describe pod -n production web-app-6d4b7c9f5-abcde
# Events:
#   Type     Reason     Age   From               Message
#   ----     ------     ----  ----               -------
#   Normal   Scheduled  5m    default-scheduler  Successfully assigned production/web-app-6d4b7c9f5-abcde to node-pool-1
#   Warning  Failed     5m    kubelet            Failed to pull image "myregistry.example.com/web-app:v1.25": rpc error: code = NotFound desc = failed to pull and unpack image: not found
#   Warning  Failed     4m    kubelet            Failed to pull image "myregistry.example.com/web-app:v1.25": rpc error: code = NotFound desc = failed to pull and unpack image: not found
#   Normal   Pulling    3m    kubelet            Pulling image "myregistry.example.com/web-app:v1.25"

The event says not found. This usually means the image tag does not exist in the registry or the image name is misspelled. Before changing anything, check the Deployment spec:

kubectl get deployment -n production web-app -o yaml | grep -A5 image
#       image: myregistry.example.com/web-app:v1.25
#       imagePullPolicy: Always

Compare with the tags actually present in your registry. If the correct tag is v1.25.0, you have found your root cause. The fix is a one-line change, but do not apply it yet. Document the current state, then make the change.

Practical parallel: For a Deployment with many Pods, use kubectl get pods -n production -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].state}{"\n"}{end}' to get a compact view of each Pod's current state.

Safe Configuration Path

Once you have identified a likely root cause, you need a safe way to apply and verify the fix. The golden rule: make the smallest possible change and verify each step. Do not bundle multiple fixes into one kubectl apply because if something else breaks, you will not know which change caused it.

Using kubectl edit vs kubectl apply

kubectl edit opens a live object in your editor. It is fast but dangerous because you can accidentally change multiple fields. A safer approach is to export the existing YAML, make a controlled edit, and diff before applying.

kubectl get deployment -n production web-app -o yaml > web-app-current.yaml
# Edit web-app-current.yaml to change image tag
kubectl diff -f web-app-current.yaml
# -  image: myregistry.example.com/web-app:v1.25
# +  image: myregistry.example.com/web-app:v1.25.0

The kubectl diff output shows exactly what will change. If it looks correct, apply:

kubectl apply -f web-app-current.yaml
# deployment.apps/web-app configured

Then monitor the rollout:

kubectl rollout status deployment/web-app -n production
# Waiting for deployment "web-app" rollout to finish: 0 of 3 updated replicas are available...
# deployment "web-app" successfully rolled out

If the rollout does not succeed, do not ignore it. Investigate the new Pods. kubectl rollout status will hang or time out; you can use --timeout=1m to avoid indefinite blocking.

Progressive Rollout with maxSurge and maxUnavailable

By default, Kubernetes may replace all Pods at once depending on your strategy. To limit the blast radius, configure a RollingUpdate strategy with maxSurge: 1 and maxUnavailable: 0. This ensures at most one extra Pod is created and no old Pods are terminated until new ones are ready.

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxSurge: 1
    maxUnavailable: 0

This is especially useful when rolling back a bad change. You can pause the rollout if you suspect a problem:

kubectl rollout pause deployment/web-app -n production
# deployment.apps/web-app paused

While paused, inspect the new Pods, run smoke tests, and then resume:

kubectl rollout resume deployment/web-app -n production
# deployment.apps/web-app resumed

Protecting Secrets and Config

Never hardcode secrets in the Deployment spec. Use environment variables from a Secret or ConfigMap. For example:

env:
  - name: DATABASE_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-secret
        key: password

Before applying any change, ensure you are not exposing secrets in logs or manifests. Use kubectl get secret db-secret -o yaml to verify the secret exists but do not print its values in documentation.

Verification and Diagnostics

After applying a fix, many engineers assume the problem is solved if the Pod is running. That is insufficient. You must verify that the application is healthy and serving traffic correctly.

Readiness and Liveness Probes

A Pod can be Running but not ready to receive traffic. Kubernetes uses readiness probes to determine when to add a Pod to a Service's endpoints. If your Deployment has no readiness probe, the Pod is considered ready as soon as its container starts, which may be too early.

Example HTTP readiness probe:

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3

Check the probe status with kubectl describe pod:

kubectl describe pod web-app-6d4b7c9f5-new123 -n production
# Events:
#   Type     Reason     Age   From               Message
#   ----     ------     ----  ----               -------
#   Warning  Unhealthy  5s    kubelet            Readiness probe failed: HTTP probe failed with statuscode: 503

A 503 means the application is not ready, even though the container is running. Investigate the application logs.

Checking Pod Logs

Use kubectl logs to inspect the current container output:

kubectl logs web-app-6d4b7c9f5-new123 -n production --tail=50

If the container is crash-looping, use --previous to get logs from the last terminated container:

kubectl logs web-app-6d4b7c9f5-new123 -n production --previous

Look for stack traces, connection errors, or configuration issues. For example, a common error is Exception in thread "main" java.net.UnknownHostException: db-service. That indicates the application cannot resolve the database Service name, usually because the Service does not exist or the application uses the wrong namespace.

End-to-End Verification with Service and Port-Forward

Do not only check Pods; verify that a Service can route to them. List the endpoints for the Service:

kubectl get endpoints web-app -n production
# NAME        ENDPOINTS                                         AGE
# web-app     10.244.1.5:8080,10.244.2.6:8080,10.244.3.7:8080   20m

If ENDPOINTS is empty, the Service selector does not match the Pod labels. Check the Service selector and the Pod labels:

kubectl get svc web-app -n production -o jsonpath='{.spec.selector}'
# {"app":"web-app"}

kubectl get pods -n production -l app=web-app --show-labels
# NAME                      READY   STATUS    RESTARTS   AGE   LABELS
# web-app-6d4b7c9f5-new123   1/1     Running   0          2m    app=web-app,pod-template-hash=6d4b7c9f5

They match, but the endpoints might be empty if the Pods are not ready. Ensure readiness probes pass.

For a quick manual test, use port-forward to access a Pod directly:

kubectl port-forward pod/web-app-6d4b7c9f5-new123 8080:8080 -n production
# Forwarding from 127.0.0.1:8080 -> 8080

Then curl the health endpoint:

curl http://localhost:8080/healthz
# OK

Failure Modes and Recovery

Deployments fail in predictable ways. Knowing these patterns lets you recover faster.

ImagePullBackOff

Cause: The image does not exist, the registry is unreachable, credentials are wrong, or the imagePullPolicy is incorrect. The Pod shows ImagePullBackOff or ErrImagePull.

Diagnosis:

kubectl describe pod <pod-name> -n <namespace>

Look for events like Failed to pull image ... not found or unauthorized: authentication required.

Fix:

• Verify the image name and tag.
• Check registry connectivity from the node.
• Ensure the imagePullSecret is present and correctly referenced.
• For public images, test with docker pull or crictl pull on the node.

Recovery: Correct the image specification and apply. Kubernetes will retry automatically after the backoff period.

CrashLoopBackOff

Cause: The container starts and then exits with a non-zero code. Reasons include application misconfiguration, missing dependencies, or failing probes.

Diagnosis:

kubectl describe pod <pod-name> -n <namespace>
# Last State:     Terminated
#   Reason:       Error
#   Exit Code:    1

Check logs:

kubectl logs <pod-name> -n <namespace> --previous

Fix:

• Examine the application startup. Does it require a database that is not ready?
• Check environment variables and mounted ConfigMaps.
• Adjust liveness probe timings if the app starts slowly.

Recovery: After fixing, delete the Pod to force a new one (do not delete the Deployment):

kubectl delete pod <pod-name> -n <namespace>

Pending Pods (Unschedulable)

Cause: Insufficient resources (CPU, memory, GPU), node selectors or affinity rules that cannot be satisfied, or taints and tolerations mismatch.

Diagnosis:

kubectl describe pod <pod-name> -n <namespace>
# Events:
#   Warning  FailedScheduling  2m   default-scheduler  0/3 nodes are available: 3 Insufficient memory.

Fix:

• Reduce resource requests or increase cluster capacity.
• Adjust node selectors or affinity.
• Add necessary tolerations.

Recovery: Update the Deployment spec and apply. New Pods should schedule.

Rollout Stuck (ProgressDeadlineExceeded)

Cause: The Deployment cannot reach the desired state within the progress deadline (default 10 minutes). Often due to failed readiness probes or insufficient resources.

Diagnosis:

kubectl rollout status deployment/<name> -n <namespace>
# error: deployment "web-app" exceeded its progress deadline

Inspect the new ReplicaSet:

kubectl describe rs <new-replica-set> -n <namespace>

Fix:

• Fix the underlying issue with the new Pods.
• If you need to pause the rollout, use kubectl rollout pause.

Recovery: If the new version is bad, rollback to the previous revision:

kubectl rollout undo deployment/web-app -n production
# deployment.apps/web-app rolled back

Or rollback to a specific revision:

kubectl rollout undo deployment/web-app -n production --to-revision=2

Operations Checklist

Use this checklist before and after any Deployment change. Every item names an owner and the frequency of review.

• [Owner: DevOps Engineer; Review: After every change] Capture current Deployment state: kubectl get deployment <name> -n <namespace> -o yaml > backup-$(date +%Y%m%d).yaml. This provides a point-in-time backup.
• [Owner: Application Developer; Review: Before every release] Verify the container image exists and is pullable. Test with docker pull or a registry API call.
• [Owner: DevOps Engineer; Review: Weekly] Check that the Deployment has valid readiness and liveness probes. Use kubectl get deployment <name> -n <namespace> -o jsonpath='{.spec.template.spec.containers[*].readinessProbe}' to confirm.
• [Owner: Platform Team; Review: Monthly] Review resource requests and limits against actual usage. Use kubectl top pods -n <namespace> and compare with resources.requests in the spec.
• [Owner: DevOps Engineer; Review: Every rollout] Run kubectl rollout status deployment/<name> --timeout=5m after applying. If it fails, investigate immediately rather than waiting.
• [Owner: Security Officer; Review: Quarterly] Audit image registries and pull secrets. Ensure no secrets are stored in plaintext in Deployment manifests.
• [Owner: DevOps Engineer; Review: Quarterly] Test rollback procedure: kubectl rollout undo deployment/<name> --to-revision=<previous> in a staging environment.
• [Owner: Application Developer; Review: On every code change] Check application logs for startup errors using kubectl logs <pod> --previous after a crash.

Common Pitfalls and How to Avoid Them

Even experienced engineers make these mistakes. Recognizing them saves time and prevents outages.

1. Not Using Namespaces Correctly

Pitfall: You edit or rollback a Deployment in the wrong namespace. This is common in clusters with multiple environments.

Avoidance: Always specify -n <namespace> in every command. Set your current context with kubectl config set-context --current --namespace=<namespace> if you work mainly in one namespace.

Recovery: If you changed a Deployment in the wrong namespace, undo the change immediately. If you have a backup, restore it. Otherwise, check the rollout history and rollback to the previous revision.

2. Ignoring Pod Disruption Budgets

Pitfall: You scale down or delete Pods during maintenance without considering a Pod Disruption Budget (PDB). This can cause unexpected downtime or violation of SLOs.

Avoidance: Before any voluntary disruption, check PDBs: kubectl get pdb -n <namespace>. Respect the minAvailable or maxUnavailable settings.

Recovery: If you violated a PDB, scale the Deployment back up to meet the budget, and ensure you do not schedule further disruptions until recovered.

3. Overwriting Manual Patches with kubectl apply

Pitfall: You manually change a field using kubectl edit or kubectl patch, and later your CI/CD pipeline applies an old manifest that overwrites your change.

Avoidance: Store the canonical manifest in version control and make changes there. Use kubectl apply --server-side or kubectl replace cautiously, understanding the implications. Ensure CI/CD uses the correct manifest.

Recovery: If a change was overwritten, reapply the patch. Better, fix the source manifest and redeploy.

4. Misconfiguring Rolling Update Parameters

Pitfall: Setting maxUnavailable: 3 on a 3-replica Deployment causes all Pods to be terminated before new ones start, leading to downtime.

Avoidance: Use maxUnavailable: 0 and maxSurge: 1 for zero-downtime rolling updates. This ensures old Pods remain until new Pods are ready.

Recovery: If you already caused downtime, the rollout will eventually complete if new Pods are healthy; if not, rollback to previous version.

5. Forgetting to Update Health Probes with Application Changes

Pitfall: You change the application's health endpoint path or port but forget to update the readiness probe. Pods never become ready.

Avoidance: Treat probe configurations as part of the application definition. Test probes in staging before production. Use environment variables or consistent paths.

Recovery: Update the probe path/port in the Deployment spec and apply. The new Pods will pass probes.

6. Overlooking Image Tag Mutability

Pitfall: Using the latest tag or a mutable tag like v1 can lead to unexpected image changes. When a new image is pushed with the same tag, a rolling update may pull different versions for different Pods.

Avoidance: Use immutable tags like v1.25.0 or a commit SHA. Enable tag immutability in your registry if available.

Recovery: If inconsistent versions are running, set the image to a specific immutable tag and trigger a rolling update.

Conclusion

Kubernetes Deployment errors are inevitable, but they do not have to be stressful. A systematic approach—inventory, diagnose, change minimally, verify, and recover—turns a chaotic incident into a routine operation. The commands and patterns in this article give you the tools to handle the most common failure modes safely.

Start by applying one new practice: perhaps adding a readiness probe where it is missing, or using kubectl diff before applying changes. Document your current state, run the documented check, and compare the result with the expected output. Review dependencies such as ReplicaSets, Pods, and Horizontal Pod Autoscalers when they affect your Deployment's behavior.

A reliable workflow makes failures visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Your cluster will be more stable, and your team will sleep better.