## Intro Kubernetes Deployment errors can halt releases, exhaust on-call engineers, and undermine confidence in the platform. The good news is that most failures follow a small set of patterns: misconfigured container images, missing resources, broken probes, and rollouts that move too fast. The bad news is that without a systematic approach, you can waste hours guessing at YAML indentation or chasing logs that do not point to the real problem. This article gives you a repeatable diagnosis workflow, not a collection of copy-paste commands. You will learn how to inventory the environment safely, inspect a Deployment without making changes, interpret the most common error messages, and recover with minimal blast radius. Every recommendation includes a concrete command, the expected output, and a fallback plan. The target audience is developers who own their services, DevOps consultants who must fix clusters they did not build, and startup teams where one person wears many hats. Operational safety is the underlying principle: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the outcome, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before touching a broken Deployment, you need to know what you are dealing with. Kubernetes behavior changes between versions; a fix that works on 1.28 may not apply to 1.30. Start with read-only observations. **Example inventory commands:** ```bash kubectl version --short # Client Version: v1.30.2 # Server Version: v1.30.0 ``` ```bash kubectl get deployment -n production # NAME READY UP-TO-DATE AVAILABLE AGE # web-app 0/3 3 0 15m ``` ```bash kubectl get pods -n production -l app=web-app -o wide # NAME READY STATUS RESTARTS AGE IP NODE # web-app-6d4b7c9f5-abcde 0/1 ImagePullBackOff 0 5m node-pool-1 # web-app-6d4b7c9f5-fghij 0/1 ImagePullBackOff 0 5m node-pool-2 # web-app-6d4b7c9f5-klmno 0/1 ImagePullBackOff 0 5m node-pool-3 ``` Notice the `ImagePullBackOff` status. This tells you the Pod could not pull the container image, but not why. To find out, describe the Pod: ```bash kubectl describe pod -n production web-app-6d4b7c9f5-abcde # Events: # Type Reason Age From Message # ---- ------ ---- ---- ------- # Normal Scheduled 5m default-scheduler Successfully assigned production/web-app-6d4b7c9f5-abcde to node-pool-1 # Warning Failed 5m kubelet Failed to pull image "myregistry.example.com/web-app:v1.25": rpc error: code = NotFound desc = failed to pull and unpack image: not found # Warning Failed 4m kubelet Failed to pull image "myregistry.example.com/web-app:v1.25": rpc error: code = NotFound desc = failed to pull and unpack image: not found # Normal Pulling 3m kubelet Pulling image "myregistry.example.com/web-app:v1.25" ``` The event says `not found`. This usually means the image tag does not exist in the registry or the image name is misspelled. Before changing anything, check the Deployment spec: ```bash kubectl get deployment -n production web-app -o yaml | grep -A5 image # image: myregistry.example.com/web-app:v1.25 # imagePullPolicy: Always ``` Compare with the tags actually present in your registry. If the correct tag is `v1.25.0`, you have found your root cause. The fix is a one-line change, but do not apply it yet. Document the current state, then make the change. **Practical parallel:** For a Deployment with many Pods, use `kubectl get pods -n production -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].state}{"\n"}{end}'` to get a compact view of each Pod's current state. ## Safe Configuration Path Once you have identified a likely root cause, you need a safe way to apply and verify the fix. The golden rule: make the smallest possible change and verify each step. Do not bundle multiple fixes into one `kubectl apply` because if something else breaks, you will not know which change caused it. ### Using `kubectl edit` vs `kubectl apply` `kubectl edit` opens a live object in your editor. It is fast but dangerous because you can accidentally change multiple fields. A safer approach is to export the existing YAML, make a controlled edit, and diff before applying. ```bash kubectl get deployment -n production web-app -o yaml > web-app-current.yaml # Edit web-app-current.yaml to change image tag kubectl diff -f web-app-current.yaml # - image: myregistry.example.com/web-app:v1.25 # + image: myregistry.example.com/web-app:v1.25.0 ``` The `kubectl diff` output shows exactly what will change. If it looks correct, apply: ```bash kubectl apply -f web-app-current.yaml # deployment.apps/web-app configured ``` Then monitor the rollout: ```bash kubectl rollout status deployment/web-app -n production # Waiting for deployment "web-app" rollout to finish: 0 of 3 updated replicas are available... # deployment "web-app" successfully rolled out ``` If the rollout does not succeed, do not ignore it. Investigate the new Pods. `kubectl rollout status` will hang or time out; you can use `--timeout=1m` to avoid indefinite blocking. ### Progressive Rollout with `maxSurge` and `maxUnavailable` By default, Kubernetes may replace all Pods at once depending on your strategy. To limit the blast radius, configure a `RollingUpdate` strategy with `maxSurge: 1` and `maxUnavailable: 0`. This ensures at most one extra Pod is created and no old Pods are terminated until new ones are ready. ```yaml strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 ``` This is especially useful when rolling back a bad change. You can pause the rollout if you suspect a problem: ```bash kubectl rollout pause deployment/web-app -n production # deployment.apps/web-app paused ``` While paused, inspect the new Pods, run smoke tests, and then resume: ```bash kubectl rollout resume deployment/web-app -n production # deployment.apps/web-app resumed ``` ### Protecting Secrets and Config Never hardcode secrets in the Deployment spec. Use environment variables from a Secret or ConfigMap. For example: ```yaml env: - name: DATABASE_PASSWORD valueFrom: secretKeyRef: name: db-secret key: password ``` Before applying any change, ensure you are not exposing secrets in logs or manifests. Use `kubectl get secret db-secret -o yaml` to verify the secret exists but do not print its values in documentation. ## Verification and Diagnostics After applying a fix, many engineers assume the problem is solved if the Pod is running. That is insufficient. You must verify that the application is healthy and serving traffic correctly. ### Readiness and Liveness Probes A Pod can be `Running` but not ready to receive traffic. Kubernetes uses readiness probes to determine when to add a Pod to a Service's endpoints. If your Deployment has no readiness probe, the Pod is considered ready as soon as its container starts, which may be too early. **Example HTTP readiness probe:** ```yaml readinessProbe: httpGet: path: /healthz port: 8080 initialDelaySeconds: 10 periodSeconds: 5 timeoutSeconds: 2 failureThreshold: 3 ``` Check the probe status with `kubectl describe pod`: ```bash kubectl describe pod web-app-6d4b7c9f5-new123 -n production # Events: # Type Reason Age From Message # ---- ------ ---- ---- ------- # Warning Unhealthy 5s kubelet Readiness probe failed: HTTP probe failed with statuscode: 503 ``` A 503 means the application is not ready, even though the container is running. Investigate the application logs. ### Checking Pod Logs Use `kubectl logs` to inspect the current container output: ```bash kubectl logs web-app-6d4b7c9f5-new123 -n production --tail=50 ``` If the container is crash-looping, use `--previous` to get logs from the last terminated container: ```bash kubectl logs web-app-6d4b7c9f5-new123 -n production --previous ``` Look for stack traces, connection errors, or configuration issues. For example, a common error is `Exception in thread "main" java.net.UnknownHostException: db-service`. That indicates the application cannot resolve the database Service name, usually because the Service does not exist or the application uses the wrong namespace. ### End-to-End Verification with Service and Port-Forward Do not only check Pods; verify that a Service can route to them. List the endpoints for the Service: ```bash kubectl get endpoints web-app -n production # NAME ENDPOINTS AGE # web-app 10.244.1.5:8080,10.244.2.6:8080,10.244.3.7:8080 20m ``` If ENDPOINTS is empty, the Service selector does not match the Pod labels. Check the Service selector and the Pod labels: ```bash kubectl get svc web-app -n production -o jsonpath='{.spec.selector}' # {"app":"web-app"} kubectl get pods -n production -l app=web-app --show-labels # NAME READY STATUS RESTARTS AGE LABELS # web-app-6d4b7c9f5-new123 1/1 Running 0 2m app=web-app,pod-template-hash=6d4b7c9f5 ``` They match, but the endpoints might be empty if the Pods are not ready. Ensure readiness probes pass. For a quick manual test, use port-forward to access a Pod directly: ```bash kubectl port-forward pod/web-app-6d4b7c9f5-new123 8080:8080 -n production # Forwarding from 127.0.0.1:8080 -> 8080 ``` Then curl the health endpoint: ```bash curl http://localhost:8080/healthz # OK ``` ## Failure Modes and Recovery Deployments fail in predictable ways. Knowing these patterns lets you recover faster. ### ImagePullBackOff **Cause:** The image does not exist, the registry is unreachable, credentials are wrong, or the imagePullPolicy is incorrect. The Pod shows `ImagePullBackOff` or `ErrImagePull`. **Diagnosis:** ```bash kubectl describe pod -n ``` Look for events like `Failed to pull image ... not found` or `unauthorized: authentication required`. **Fix:** - Verify the image name and tag. - Check registry connectivity from the node. - Ensure the imagePullSecret is present and correctly referenced. - For public images, test with `docker pull` or `crictl pull` on the node. **Recovery:** Correct the image specification and apply. Kubernetes will retry automatically after the backoff period. ### CrashLoopBackOff **Cause:** The container starts and then exits with a non-zero code. Reasons include application misconfiguration, missing dependencies, or failing probes. **Diagnosis:** ```bash kubectl describe pod -n # Last State: Terminated # Reason: Error # Exit Code: 1 ``` Check logs: ```bash kubectl logs -n --previous ``` **Fix:** - Examine the application startup. Does it require a database that is not ready? - Check environment variables and mounted ConfigMaps. - Adjust liveness probe timings if the app starts slowly. **Recovery:** After fixing, delete the Pod to force a new one (do not delete the Deployment): ```bash kubectl delete pod -n ``` ### Pending Pods (Unschedulable) **Cause:** Insufficient resources (CPU, memory, GPU), node selectors or affinity rules that cannot be satisfied, or taints and tolerations mismatch. **Diagnosis:** ```bash kubectl describe pod -n # Events: # Warning FailedScheduling 2m default-scheduler 0/3 nodes are available: 3 Insufficient memory. ``` **Fix:** - Reduce resource requests or increase cluster capacity. - Adjust node selectors or affinity. - Add necessary tolerations. **Recovery:** Update the Deployment spec and apply. New Pods should schedule. ### Rollout Stuck (ProgressDeadlineExceeded) **Cause:** The Deployment cannot reach the desired state within the progress deadline (default 10 minutes). Often due to failed readiness probes or insufficient resources. **Diagnosis:** ```bash kubectl rollout status deployment/ -n # error: deployment "web-app" exceeded its progress deadline ``` Inspect the new ReplicaSet: ```bash kubectl describe rs -n ``` **Fix:** - Fix the underlying issue with the new Pods. - If you need to pause the rollout, use `kubectl rollout pause`. **Recovery:** If the new version is bad, rollback to the previous revision: ```bash kubectl rollout undo deployment/web-app -n production # deployment.apps/web-app rolled back ``` Or rollback to a specific revision: ```bash kubectl rollout undo deployment/web-app -n production --to-revision=2 ``` ## Operations Checklist Use this checklist before and after any Deployment change. Every item names an owner and the frequency of review. - **[Owner: DevOps Engineer; Review: After every change]** Capture current Deployment state: `kubectl get deployment -n -o yaml > backup-$(date +%Y%m%d).yaml`. This provides a point-in-time backup. - **[Owner: Application Developer; Review: Before every release]** Verify the container image exists and is pullable. Test with `docker pull` or a registry API call. - **[Owner: DevOps Engineer; Review: Weekly]** Check that the Deployment has valid readiness and liveness probes. Use `kubectl get deployment -n -o jsonpath='{.spec.template.spec.containers[*].readinessProbe}'` to confirm. - **[Owner: Platform Team; Review: Monthly]** Review resource requests and limits against actual usage. Use `kubectl top pods -n ` and compare with `resources.requests` in the spec. - **[Owner: DevOps Engineer; Review: Every rollout]** Run `kubectl rollout status deployment/ --timeout=5m` after applying. If it fails, investigate immediately rather than waiting. - **[Owner: Security Officer; Review: Quarterly]** Audit image registries and pull secrets. Ensure no secrets are stored in plaintext in Deployment manifests. - **[Owner: DevOps Engineer; Review: Quarterly]** Test rollback procedure: `kubectl rollout undo deployment/ --to-revision=` in a staging environment. - **[Owner: Application Developer; Review: On every code change]** Check application logs for startup errors using `kubectl logs --previous` after a crash. ## Common Pitfalls and How to Avoid Them Even experienced engineers make these mistakes. Recognizing them saves time and prevents outages. ### 1. Not Using Namespaces Correctly **Pitfall:** You edit or rollback a Deployment in the wrong namespace. This is common in clusters with multiple environments. **Avoidance:** Always specify `-n ` in every command. Set your current context with `kubectl config set-context --current --namespace=` if you work mainly in one namespace. **Recovery:** If you changed a Deployment in the wrong namespace, undo the change immediately. If you have a backup, restore it. Otherwise, check the rollout history and rollback to the previous revision. ### 2. Ignoring Pod Disruption Budgets **Pitfall:** You scale down or delete Pods during maintenance without considering a Pod Disruption Budget (PDB). This can cause unexpected downtime or violation of SLOs. **Avoidance:** Before any voluntary disruption, check PDBs: `kubectl get pdb -n `. Respect the `minAvailable` or `maxUnavailable` settings. **Recovery:** If you violated a PDB, scale the Deployment back up to meet the budget, and ensure you do not schedule further disruptions until recovered. ### 3. Overwriting Manual Patches with `kubectl apply` **Pitfall:** You manually change a field using `kubectl edit` or `kubectl patch`, and later your CI/CD pipeline applies an old manifest that overwrites your change. **Avoidance:** Store the canonical manifest in version control and make changes there. Use `kubectl apply --server-side` or `kubectl replace` cautiously, understanding the implications. Ensure CI/CD uses the correct manifest. **Recovery:** If a change was overwritten, reapply the patch. Better, fix the source manifest and redeploy. ### 4. Misconfiguring Rolling Update Parameters **Pitfall:** Setting `maxUnavailable: 3` on a 3-replica Deployment causes all Pods to be terminated before new ones start, leading to downtime. **Avoidance:** Use `maxUnavailable: 0` and `maxSurge: 1` for zero-downtime rolling updates. This ensures old Pods remain until new Pods are ready. **Recovery:** If you already caused downtime, the rollout will eventually complete if new Pods are healthy; if not, rollback to previous version. ### 5. Forgetting to Update Health Probes with Application Changes **Pitfall:** You change the application's health endpoint path or port but forget to update the readiness probe. Pods never become ready. **Avoidance:** Treat probe configurations as part of the application definition. Test probes in staging before production. Use environment variables or consistent paths. **Recovery:** Update the probe path/port in the Deployment spec and apply. The new Pods will pass probes. ### 6. Overlooking Image Tag Mutability **Pitfall:** Using the `latest` tag or a mutable tag like `v1` can lead to unexpected image changes. When a new image is pushed with the same tag, a rolling update may pull different versions for different Pods. **Avoidance:** Use immutable tags like `v1.25.0` or a commit SHA. Enable tag immutability in your registry if available. **Recovery:** If inconsistent versions are running, set the image to a specific immutable tag and trigger a rolling update. ## Conclusion Kubernetes Deployment errors are inevitable, but they do not have to be stressful. A systematic approach—inventory, diagnose, change minimally, verify, and recover—turns a chaotic incident into a routine operation. The commands and patterns in this article give you the tools to handle the most common failure modes safely. Start by applying one new practice: perhaps adding a readiness probe where it is missing, or using `kubectl diff` before applying changes. Document your current state, run the documented check, and compare the result with the expected output. Review dependencies such as ReplicaSets, Pods, and Horizontal Pod Autoscalers when they affect your Deployment's behavior. A reliable workflow makes failures visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Your cluster will be more stable, and your team will sleep better.