Kubernetes Images Troubleshooting: Commands, Logs, and Recovery in Practice

Intro

Kubernetes is the de facto container orchestrator, but image-related issues remain one of the most common sources of deployment failures. A misconfigured image name, missing registry credentials, or an incompatible architecture can cause pods to crash loop or fail to start entirely. For developers and DevOps engineers, quickly diagnosing and fixing these issues is critical to maintaining service reliability and delivery speed.

This guide provides a practical approach to troubleshooting Kubernetes image problems. It covers environment inventory, safe configuration practices, verification commands, common failure modes, and recovery steps. Each section includes concrete commands and expected outputs. By the end, you will have a repeatable methodology to resolve image-related issues in your clusters.

Version and Environment Inventory

Before troubleshooting, establish the Kubernetes and container runtime versions, cluster topology, and the affected workloads. This scoping prevents wasted effort and helps identify version-specific quirks.

Collect Cluster and Runtime Versions

Run the following commands from a machine with kubectl configured:

kubectl version --short
kubectl get nodes -o wide

Expected output shows client and server versions, and node details including container runtime (e.g., containerd, CRI-O) and OS image. For example:

Client Version: v1.27.3
Server Version: v1.27.3
NAME        STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node-1      Ready    control-plane   10d   v1.27.3   10.0.0.1      <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.7.1
node-2      Ready    <none>          10d   v1.27.3   10.0.0.2      <none>        Ubuntu 22.04.2 LTS   5.15.0-76-generic   containerd://1.7.1

For node-level detail, describe a node:

kubectl describe node <node-name>

Look for the "Container Runtime Version" and "Kubelet Version" fields. Example snippet:

System Info:
  Machine ID:                 9b2f3c4d5e6f7a8b9c0d1e2f3a4b5c6d
  System UUID:                12345678-1234-1234-1234-123456789abc
  Boot ID:                    a1b2c3d4-e5f6-7890-abcd-ef1234567890
  Kernel Version:             5.15.0-76-generic
  OS Image:                   Ubuntu 22.04.2 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.7.1
  Kubelet Version:            v1.27.3
  Kube-Proxy Version:         v1.27.3

Identify Affected Workloads and Images

List pods in all namespaces with image information:

kubectl get pods -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,STATUS:.status.phase,IMAGES:.spec.containers[*].image

This helps pinpoint which pods are failing and the images they reference. Example output:

NAMESPACE   NAME                     STATUS             IMAGES
default     web-6d4b8c9f5-abcde      ImagePullBackOff   nginx:1.21.6
default     api-7f8b9c6d5-xyzab     CrashLoopBackOff   myregistry.com/myapp:2.3.1
kube-system coredns-5d78c9d8f-12345 Running            registry.k8s.io/coredns/coredns:v1.10.1

Prerequisites Check

Ensure you have:

• kubectl installed and configured with appropriate permissions.
• Access to container registry (e.g., Docker Hub, private registry) to pull images for testing.
• Ability to inspect logs and events.

If you suspect registry connectivity issues, test from within the cluster (e.g., using a debug pod) or from a node if possible.

Safe Configuration Path

When fixing image issues, it is crucial to make changes incrementally and safely. Avoid modifying production workloads directly without testing. Use canary deployments, rolling updates, or recreate pods in a controlled manner.

Correcting Image References

Common image mistakes include typos, wrong tags, or missing registry prefixes. To fix, edit the deployment:

kubectl edit deployment <deployment-name> -n <namespace>

Or apply a patch:

kubectl patch deployment <deployment-name> -n <namespace> --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/containers/0/image", "value":"nginx:1.21.6"}]'

After patching, the deployment rollout triggers automatically. Monitor with:

kubectl rollout status deployment/<deployment-name> -n <namespace>

Expected: deployment "<deployment-name>" successfully rolled out

Managing Registry Credentials

If pods fail with ImagePullBackOff or ErrImagePull due to authentication, ensure a Secret of type docker-registry exists and is referenced.

Create a secret from existing Docker config or credentials:

kubectl create secret docker-registry regcred \
  --docker-server=myregistry.example.com \
  --docker-username=myuser \
  --docker-password=mypassword \
  [email protected] \
  -n default

Then add imagePullSecrets to the pod spec or service account. Patch service account:

kubectl patch serviceaccount default -n default -p '{"imagePullSecrets": [{"name": "regcred"}]}'

Verify secret exists:

kubectl get secret regcred -n default -o yaml

Expected output shows base64-encoded .dockerconfigjson.

Setting Image Pull Policy

The imagePullPolicy field controls when kubelet pulls images. Defaults: Always for :latest tag, IfNotPresent for specific tags. For troubleshooting, you may set to Always to force pull latest:

spec:
  containers:
  - name: app
    image: myapp:1.0
    imagePullPolicy: Always

Verification and Diagnostics

After applying fixes, verify that images are pulled correctly and pods are healthy.

Check Pod Status and Events

kubectl get pods -n <namespace>
kubectl describe pod <pod-name> -n <namespace>

In the events section, look for Pulled, Created, Started messages. Failure events like Failed to pull image or Back-off pulling image indicate ongoing issues. Example event block:

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  2m                 default-scheduler  Successfully assigned default/web-6d4b8c9f5-abcde to node-2
  Normal   Pulling    2m                 kubelet            Pulling image "nginx:1.21.6"
  Warning  Failed     2m                 kubelet            Failed to pull image "nginx:1.21.6": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginx:1.21.6": failed to resolve reference "docker.io/library/nginx:1.21.6": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     2m                 kubelet            Error: ErrImagePull
  Normal   BackOff    2m (x3 over 2m)    kubelet            Back-off pulling image "nginx:1.21.6"
  Warning  Failed     2m (x3 over 2m)    kubelet            Error: ImagePullBackOff

Inspect Logs

kubectl logs <pod-name> -n <namespace>

For previous crashed container:

kubectl logs <pod-name> -n <namespace> --previous

If logs show application errors unrelated to image pull, the image itself may be incompatible with the container runtime (e.g., wrong architecture). Example log from a crash loop:

standard_init_linux.go:228: exec user process caused: exec format error

This often indicates the image was built for a different CPU architecture (e.g., arm64 image on amd64 node).

Verify Image Digest

Pods can reference images by digest for immutability. To get the digest from a running pod:

kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.status.containerStatuses[0].imageID}'

Expected output like docker.io/library/nginx@sha256:abcdef1234567890....

Test Image Pull Manually

If possible, pull the image on the node or from a debug container:

kubectl run -it --rm debug --image=busybox --restart=Never -- sh

Inside debug container, try nslookup <registry-host> and wget to check connectivity. (Note: busybox may not have wget; use nc or curl in appropriate image.)

Alternatively, check kubelet logs on the node for pull errors:

journalctl -u kubelet -f | grep -i image

Failure Modes and Recovery

Common Failure Modes

• ImagePullBackOff: Kubernetes repeatedly attempts to pull but fails. Causes:
• Incorrect image name or tag
• Registry authentication failure
• Network connectivity issues
• Rate limiting on registry
• ErrImagePull: Immediate pull error, often due to missing credentials or invalid image reference.
• CrashLoopBackOff: Image pulled successfully but container exits quickly. This may be due to application error, missing dependencies, or wrong architecture.
• InvalidImageName: The image field is malformed (e.g., contains uppercase letters or invalid characters).

Recovery Steps

For each failure, follow these steps:

ImagePullBackOff / ErrImagePull

• Check pod events for detailed error:

kubectl describe pod <pod-name> -n <namespace> | grep -A 10 Events

• Verify image name and tag against registry. Use docker pull or crictl pull from a node if possible.

• Check secret existence and correctness:

kubectl get secret <secret-name> -n <namespace> -o yaml

• Validate network connectivity from the node to the registry.

• Fix the issue and trigger a new rollout:

kubectl rollout restart deployment/<deployment-name> -n <namespace>

CrashLoopBackOff

• Check logs:

kubectl logs <pod-name> -n <namespace> --previous

• If the error indicates missing file or architecture mismatch, rebuild the image with correct multi-arch support or use an appropriate tag.
• Update the deployment with the corrected image.

InvalidImageName

• Edit the deployment and correct the image field using proper format: [registry/]image[:tag][@digest].
• Ensure lowercase and no spaces.

Rollback Strategies

If a change causes new failures, rollback to a known good version:

• Use kubectl rollout undo deployment/<deployment-name> -n <namespace> to revert to previous revision.
• Check rollout history:

kubectl rollout history deployment/<deployment-name> -n <namespace>

• If the cluster is in a bad state, consider scaling down the broken workload and scaling up a stable one.

Common Pitfalls and How to Avoid Them

Many image issues stem from avoidable configuration mistakes or process gaps. Here are the most common pitfalls and how to prevent them.

1. Using the latest Tag

Problem: The latest tag is mutable and can be overwritten, causing unpredictable deployments. Combined with default imagePullPolicy: Always, kubelet may pull a new version without notice.

Avoidance: Always use explicit version tags or digests in production. For example, use nginx:1.21.6 instead of nginx:latest. If you must use latest, set imagePullPolicy: IfNotPresent to avoid unexpected pulls, but this is not recommended for production.

Recovery: If a latest pull broke your deployment, rollback to the previous revision with kubectl rollout undo deployment/<deployment-name> and then pin a specific tag.

2. Mismatched Architectures

Problem: Containers built for one CPU architecture (e.g., arm64) fail on nodes of another (e.g., amd64) with exec format error and CrashLoopBackOff.

Avoidance: Build multi-arch images using Docker Buildx or similar tools, and use manifests that include all required architectures. When deploying, specify the correct image variant or rely on the manifest list auto-selection.

Recovery: Verify node architecture with kubectl get nodes -o wide and inspect image manifest for supported architectures. Rebuild or use a compatible image tag.

3. Missing or Incorrect Image Pull Secrets

Problem: Private registries require credentials, but the pod or service account lacks imagePullSecrets, causing ImagePullBackOff with authentication errors.

Avoidance: Store registry credentials as a docker-registry secret and reference it either in the pod spec or, better, in the default service account for the namespace. Use kubectl create secret docker-registry as shown earlier.

Recovery: Create the secret, patch the service account, and restart the affected pods. Verify with kubectl describe pod that the pull succeeds.

4. Registry Rate Limits

Problem: Public registries like Docker Hub enforce pull rate limits. Exceeding them causes ImagePullBackOff with toomanyrequests errors.

Avoidance: Use a pull-through cache (e.g., Harbor, Artifactory) or authenticated Docker Hub accounts with higher limits. Monitor registry pull metrics.

Recovery: Wait for the rate limit window to reset or switch to a cache/private registry. Implement exponential backoff in CI/CD to reduce burst pulls.

5. Misconfigured Network Policies or Firewall

Problem: Firewall or network policies block access to the registry, leading to timeouts and pull failures.

Avoidance: Ensure egress traffic to registry hosts is allowed. Test connectivity from nodes using telnet <registry-host> 443 or curl -v https://<registry-host>/v2/.

Recovery: Update firewall rules or network policies, then restart affected pods.

6. Forgetting to Update Image References in All Places

Problem: A fix is applied to one deployment but not to others referencing the same image, leading to inconsistent behavior.

Avoidance: Use a single source of truth for image references, such as a Helm values file or Kustomize overlay, and update it globally. Automate image tag updates with CI/CD.

Recovery: Search for all workloads using the problematic image with kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].image | contains("myapp:old")) | .metadata.namespace + "/" + .metadata.name' and update them.

Operations Checklist

Use the following checklist to systematically verify image health and prevent recurring issues. Assign a single accountable owner for each item and a review cadence.

• [ ] Confirm cluster and node versions are compatible with required images. Owner: Platform Engineer (e.g., John Doe). Review: monthly or after cluster upgrades.
• [ ] Ensure all image references are spelled correctly and use valid tags/digests. Owner: Application Developer (e.g., Jane Smith). Review: per deployment.
• [ ] Verify imagePullSecrets are present and correctly configured for private registries. Owner: DevOps Engineer (e.g., Alex Johnson). Review: quarterly or when registry credentials rotate.
• [ ] Set appropriate imagePullPolicy for the deployment strategy. Owner: Application Developer. Review: per deployment.
• [ ] After deployment, monitor pod status and events for successful pulls. Owner: Site Reliability Engineer (e.g., Emily Brown). Review: continuous monitoring with alerts.
• [ ] Regularly review registry rate limits and use pull-through caches if necessary. Owner: Platform Engineer. Review: monthly.
• [ ] Use immutable tags (digests) for production workloads to avoid unexpected changes. Owner: DevOps Engineer. Review: per release process.
• [ ] Implement CI checks that validate image existence and architecture before deployment. Owner: CI/CD Pipeline Maintainer (e.g., Michael Lee). Review: quarterly.

The table below summarizes common failure events and typical causes:

Conclusion

Troubleshooting Kubernetes image issues requires a systematic approach: start by inventorying the environment, then make safe configuration changes, verify through diagnostics, and have recovery plans ready. By following the commands and checklists in this guide, teams can reduce downtime and improve deployment reliability. The key is to be methodical: collect the right information, apply fixes carefully, and verify the results. Remember that image issues are often a symptom of broader process gaps, such as lack of image governance or CI/CD validation. Addressing those root causes will prevent future failures.