## Intro Kubernetes is the de facto container orchestrator, but image-related issues remain one of the most common sources of deployment failures. A misconfigured image name, missing registry credentials, or an incompatible architecture can cause pods to crash loop or fail to start entirely. For developers and DevOps engineers, quickly diagnosing and fixing these issues is critical to maintaining service reliability and delivery speed. This guide provides a practical approach to troubleshooting Kubernetes image problems. It covers environment inventory, safe configuration practices, verification commands, common failure modes, and recovery steps. Each section includes concrete commands and expected outputs. By the end, you will have a repeatable methodology to resolve image-related issues in your clusters. ## Version and Environment Inventory Before troubleshooting, establish the Kubernetes and container runtime versions, cluster topology, and the affected workloads. This scoping prevents wasted effort and helps identify version-specific quirks. ### Collect Cluster and Runtime Versions Run the following commands from a machine with kubectl configured: ```bash kubectl version --short kubectl get nodes -o wide ``` Expected output shows client and server versions, and node details including container runtime (e.g., containerd, CRI-O) and OS image. For example: ``` Client Version: v1.27.3 Server Version: v1.27.3 NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME node-1 Ready control-plane 10d v1.27.3 10.0.0.1 Ubuntu 22.04.2 LTS 5.15.0-76-generic containerd://1.7.1 node-2 Ready 10d v1.27.3 10.0.0.2 Ubuntu 22.04.2 LTS 5.15.0-76-generic containerd://1.7.1 ``` For node-level detail, describe a node: ```bash kubectl describe node ``` Look for the "Container Runtime Version" and "Kubelet Version" fields. Example snippet: ``` System Info: Machine ID: 9b2f3c4d5e6f7a8b9c0d1e2f3a4b5c6d System UUID: 12345678-1234-1234-1234-123456789abc Boot ID: a1b2c3d4-e5f6-7890-abcd-ef1234567890 Kernel Version: 5.15.0-76-generic OS Image: Ubuntu 22.04.2 LTS Operating System: linux Architecture: amd64 Container Runtime Version: containerd://1.7.1 Kubelet Version: v1.27.3 Kube-Proxy Version: v1.27.3 ``` ### Identify Affected Workloads and Images List pods in all namespaces with image information: ```bash kubectl get pods -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATUS:.status.phase,IMAGES:.spec.containers[*].image ``` This helps pinpoint which pods are failing and the images they reference. Example output: ``` NAMESPACE NAME STATUS IMAGES default web-6d4b8c9f5-abcde ImagePullBackOff nginx:1.21.6 default api-7f8b9c6d5-xyzab CrashLoopBackOff myregistry.com/myapp:2.3.1 kube-system coredns-5d78c9d8f-12345 Running registry.k8s.io/coredns/coredns:v1.10.1 ``` ### Prerequisites Check Ensure you have: - kubectl installed and configured with appropriate permissions. - Access to container registry (e.g., Docker Hub, private registry) to pull images for testing. - Ability to inspect logs and events. If you suspect registry connectivity issues, test from within the cluster (e.g., using a debug pod) or from a node if possible. ## Safe Configuration Path When fixing image issues, it is crucial to make changes incrementally and safely. Avoid modifying production workloads directly without testing. Use canary deployments, rolling updates, or recreate pods in a controlled manner. ### Correcting Image References Common image mistakes include typos, wrong tags, or missing registry prefixes. To fix, edit the deployment: ```bash kubectl edit deployment -n ``` Or apply a patch: ```bash kubectl patch deployment -n --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"nginx:1.21.6"}]' ``` After patching, the deployment rollout triggers automatically. Monitor with: ```bash kubectl rollout status deployment/ -n ``` Expected: `deployment "" successfully rolled out` ### Managing Registry Credentials If pods fail with `ImagePullBackOff` or `ErrImagePull` due to authentication, ensure a Secret of type `docker-registry` exists and is referenced. Create a secret from existing Docker config or credentials: ```bash kubectl create secret docker-registry regcred \ --docker-server=myregistry.example.com \ --docker-username=myuser \ --docker-password=mypassword \ --docker-email=myuser@example.com \ -n default ``` Then add `imagePullSecrets` to the pod spec or service account. Patch service account: ```bash kubectl patch serviceaccount default -n default -p '{"imagePullSecrets": [{"name": "regcred"}]}' ``` Verify secret exists: ```bash kubectl get secret regcred -n default -o yaml ``` Expected output shows base64-encoded `.dockerconfigjson`. ### Setting Image Pull Policy The `imagePullPolicy` field controls when kubelet pulls images. Defaults: `Always` for `:latest` tag, `IfNotPresent` for specific tags. For troubleshooting, you may set to `Always` to force pull latest: ```yaml spec: containers: - name: app image: myapp:1.0 imagePullPolicy: Always ``` ## Verification and Diagnostics After applying fixes, verify that images are pulled correctly and pods are healthy. ### Check Pod Status and Events ```bash kubectl get pods -n kubectl describe pod -n ``` In the events section, look for `Pulled`, `Created`, `Started` messages. Failure events like `Failed to pull image` or `Back-off pulling image` indicate ongoing issues. Example event block: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 2m default-scheduler Successfully assigned default/web-6d4b8c9f5-abcde to node-2 Normal Pulling 2m kubelet Pulling image "nginx:1.21.6" Warning Failed 2m kubelet Failed to pull image "nginx:1.21.6": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:1.21.6": failed to resolve reference "docker.io/library/nginx:1.21.6": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed Warning Failed 2m kubelet Error: ErrImagePull Normal BackOff 2m (x3 over 2m) kubelet Back-off pulling image "nginx:1.21.6" Warning Failed 2m (x3 over 2m) kubelet Error: ImagePullBackOff ``` ### Inspect Logs ```bash kubectl logs -n ``` For previous crashed container: ```bash kubectl logs -n --previous ``` If logs show application errors unrelated to image pull, the image itself may be incompatible with the container runtime (e.g., wrong architecture). Example log from a crash loop: ``` standard_init_linux.go:228: exec user process caused: exec format error ``` This often indicates the image was built for a different CPU architecture (e.g., arm64 image on amd64 node). ### Verify Image Digest Pods can reference images by digest for immutability. To get the digest from a running pod: ```bash kubectl get pod -n -o jsonpath='{.status.containerStatuses[0].imageID}' ``` Expected output like `docker.io/library/nginx@sha256:abcdef1234567890...`. ### Test Image Pull Manually If possible, pull the image on the node or from a debug container: ```bash kubectl run -it --rm debug --image=busybox --restart=Never -- sh ``` Inside debug container, try `nslookup ` and `wget` to check connectivity. (Note: busybox may not have `wget`; use `nc` or `curl` in appropriate image.) Alternatively, check kubelet logs on the node for pull errors: ```bash journalctl -u kubelet -f | grep -i image ``` ## Failure Modes and Recovery ### Common Failure Modes - **ImagePullBackOff**: Kubernetes repeatedly attempts to pull but fails. Causes: - Incorrect image name or tag - Registry authentication failure - Network connectivity issues - Rate limiting on registry - **ErrImagePull**: Immediate pull error, often due to missing credentials or invalid image reference. - **CrashLoopBackOff**: Image pulled successfully but container exits quickly. This may be due to application error, missing dependencies, or wrong architecture. - **InvalidImageName**: The image field is malformed (e.g., contains uppercase letters or invalid characters). ### Recovery Steps For each failure, follow these steps: #### ImagePullBackOff / ErrImagePull - Check pod events for detailed error: ```bash kubectl describe pod -n | grep -A 10 Events ``` - Verify image name and tag against registry. Use `docker pull` or `crictl pull` from a node if possible. - Check secret existence and correctness: ```bash kubectl get secret -n -o yaml ``` - Validate network connectivity from the node to the registry. - Fix the issue and trigger a new rollout: ```bash kubectl rollout restart deployment/ -n ``` #### CrashLoopBackOff - Check logs: ```bash kubectl logs -n --previous ``` - If the error indicates missing file or architecture mismatch, rebuild the image with correct multi-arch support or use an appropriate tag. - Update the deployment with the corrected image. #### InvalidImageName - Edit the deployment and correct the image field using proper format: `[registry/]image[:tag][@digest]`. - Ensure lowercase and no spaces. ### Rollback Strategies If a change causes new failures, rollback to a known good version: - Use `kubectl rollout undo deployment/ -n ` to revert to previous revision. - Check rollout history: ```bash kubectl rollout history deployment/ -n ``` - If the cluster is in a bad state, consider scaling down the broken workload and scaling up a stable one. ## Common Pitfalls and How to Avoid Them Many image issues stem from avoidable configuration mistakes or process gaps. Here are the most common pitfalls and how to prevent them. ### 1. Using the `latest` Tag **Problem**: The `latest` tag is mutable and can be overwritten, causing unpredictable deployments. Combined with default `imagePullPolicy: Always`, kubelet may pull a new version without notice. **Avoidance**: Always use explicit version tags or digests in production. For example, use `nginx:1.21.6` instead of `nginx:latest`. If you must use `latest`, set `imagePullPolicy: IfNotPresent` to avoid unexpected pulls, but this is not recommended for production. **Recovery**: If a `latest` pull broke your deployment, rollback to the previous revision with `kubectl rollout undo deployment/` and then pin a specific tag. ### 2. Mismatched Architectures **Problem**: Containers built for one CPU architecture (e.g., arm64) fail on nodes of another (e.g., amd64) with `exec format error` and `CrashLoopBackOff`. **Avoidance**: Build multi-arch images using Docker Buildx or similar tools, and use manifests that include all required architectures. When deploying, specify the correct image variant or rely on the manifest list auto-selection. **Recovery**: Verify node architecture with `kubectl get nodes -o wide` and inspect image manifest for supported architectures. Rebuild or use a compatible image tag. ### 3. Missing or Incorrect Image Pull Secrets **Problem**: Private registries require credentials, but the pod or service account lacks `imagePullSecrets`, causing `ImagePullBackOff` with authentication errors. **Avoidance**: Store registry credentials as a `docker-registry` secret and reference it either in the pod spec or, better, in the default service account for the namespace. Use `kubectl create secret docker-registry` as shown earlier. **Recovery**: Create the secret, patch the service account, and restart the affected pods. Verify with `kubectl describe pod` that the pull succeeds. ### 4. Registry Rate Limits **Problem**: Public registries like Docker Hub enforce pull rate limits. Exceeding them causes `ImagePullBackOff` with `toomanyrequests` errors. **Avoidance**: Use a pull-through cache (e.g., Harbor, Artifactory) or authenticated Docker Hub accounts with higher limits. Monitor registry pull metrics. **Recovery**: Wait for the rate limit window to reset or switch to a cache/private registry. Implement exponential backoff in CI/CD to reduce burst pulls. ### 5. Misconfigured Network Policies or Firewall **Problem**: Firewall or network policies block access to the registry, leading to timeouts and pull failures. **Avoidance**: Ensure egress traffic to registry hosts is allowed. Test connectivity from nodes using `telnet 443` or `curl -v https:///v2/`. **Recovery**: Update firewall rules or network policies, then restart affected pods. ### 6. Forgetting to Update Image References in All Places **Problem**: A fix is applied to one deployment but not to others referencing the same image, leading to inconsistent behavior. **Avoidance**: Use a single source of truth for image references, such as a Helm values file or Kustomize overlay, and update it globally. Automate image tag updates with CI/CD. **Recovery**: Search for all workloads using the problematic image with `kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].image | contains("myapp:old")) | .metadata.namespace + "/" + .metadata.name'` and update them. ## Operations Checklist Use the following checklist to systematically verify image health and prevent recurring issues. Assign a single accountable owner for each item and a review cadence. - [ ] Confirm cluster and node versions are compatible with required images. Owner: Platform Engineer (e.g., John Doe). Review: monthly or after cluster upgrades. - [ ] Ensure all image references are spelled correctly and use valid tags/digests. Owner: Application Developer (e.g., Jane Smith). Review: per deployment. - [ ] Verify `imagePullSecrets` are present and correctly configured for private registries. Owner: DevOps Engineer (e.g., Alex Johnson). Review: quarterly or when registry credentials rotate. - [ ] Set appropriate `imagePullPolicy` for the deployment strategy. Owner: Application Developer. Review: per deployment. - [ ] After deployment, monitor pod status and events for successful pulls. Owner: Site Reliability Engineer (e.g., Emily Brown). Review: continuous monitoring with alerts. - [ ] Regularly review registry rate limits and use pull-through caches if necessary. Owner: Platform Engineer. Review: monthly. - [ ] Use immutable tags (digests) for production workloads to avoid unexpected changes. Owner: DevOps Engineer. Review: per release process. - [ ] Implement CI checks that validate image existence and architecture before deployment. Owner: CI/CD Pipeline Maintainer (e.g., Michael Lee). Review: quarterly. The table below summarizes common failure events and typical causes: | Failure Event | Typical Cause | Key Diagnostic Command | |---|---|---| | ImagePullBackOff | Authentication, wrong image, network | `kubectl describe pod` | | ErrImagePull | Missing credentials, invalid image | `kubectl get events` | | CrashLoopBackOff | Application error, architecture mismatch | `kubectl logs --previous` | | InvalidImageName | Malformed image field | `kubectl describe pod` | ## Conclusion Troubleshooting Kubernetes image issues requires a systematic approach: start by inventorying the environment, then make safe configuration changes, verify through diagnostics, and have recovery plans ready. By following the commands and checklists in this guide, teams can reduce downtime and improve deployment reliability. The key is to be methodical: collect the right information, apply fixes carefully, and verify the results. Remember that image issues are often a symptom of broader process gaps, such as lack of image governance or CI/CD validation. Addressing those root causes will prevent future failures.