## Intro Kubernetes images are at the heart of every workload, and knowing how to inspect, update, and troubleshoot them is essential for developers, DevOps consultants, and technical startup teams. This article moves from observed problem to verified result by connecting basic commands, concrete examples, a cheat sheet, and operational practices. We focus on practical safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the outcome, and document how to recover if the expected state is not reached. Every command below is version-scoped (Kubernetes 1.24+ unless noted) and assumes kubectl is installed, configured with a current context, and that you have read access to the relevant namespace. The goal of operational safety is not just to run commands but to understand what they change and how to revert. We will cover version and environment inventory, the safe configuration path for image updates, verification and diagnostics, failure modes and recovery, and an operations checklist you can reuse across clusters. ## Version and Environment Inventory Before touching images, establish the cluster and component versions. This inventory is read-only and forms the baseline for any later change. Key components to inventory: - Kubernetes control plane and node versions - Container runtime (containerd, CRI-O, or Docker via cri-dockerd) - CNI plugin version - Image registry endpoints and authentication - Existing image pull policies on workloads Start with version discovery commands: kubectl version --short # Expected output example: # Client Version: v1.28.2 # Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 # Server Version: v1.28.2 If kubectl version --short is deprecated in newer releases, use: kubectl version Which prints full JSON, including client and server versions. Capture this baseline in your runbook. Check node runtime version with: kubectl get nodes -o wide # NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME # node1 Ready control-plane 10d v1.28.2 192.168.1.10 Ubuntu 22.04.3 LTS 5.15.0-91-generic containerd://1.7.8 # node2 Ready 10d v1.28.2 192.168.1.11 Ubuntu 22.04.3 LTS 5.15.0-91-generic containerd://1.7.8 The CONTAINER-RUNTIME column lists the runtime and its version, which matters for image compatibility (e.g., some features require containerd 1.6+). Example: if you need to use image streaming (e.g., for large images), ensure the runtime supports it. containerd 1.7 with CRI 1.2 and KEP 3668 enables it; older versions do not. Inventory existing image usage across all namespaces with: kubectl get pods --all-namespaces -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}' Sample output: default nginx-deployment-6b474476c4-8g2kf nginx:1.25.3 default redis-cache-0 redis:7.2-alpine kube-system coredns-5d78c9869d-8h4wp registry.k8s.io/coredns/coredns:v1.11.1 This read-only inventory shows all running images. Compare against your approved image list to spot drift. Prerequisites for safe image operations: - kubectl version compatible with server (within one minor version) - Registry credentials configured as a secret if pulling from private registries - Sufficient node disk space for image pulls (check with df -h on nodes or use monitoring) - Network egress to registry endpoints A scoped read-only observation for image health: kubectl get events --all-namespaces --field-selector reason=FailedToPullImage If events are empty, images are pulling normally in the last hour (default event TTL). Otherwise, note the namespace, pod, and image causing failure. Document the current state with timestamp and cluster context: date -u && kubectl config current-context && kubectl get pods -A -o wide > inventory-$(date +%Y%m%d-%H%M).txt This text file is your rollback reference for later changes.
Quick check 1 of 2 Which command is used to update a container image in a pod using a JSON patch? - kubectl set image pod valid-pod kubernetes-serve-hostname=new image - kubectl patch pod valid-pod -p '{"spec":{"containers":[{"name":"kubernetes-serve-hostname","image":"new image"}]}}' - kubectl edit pod valid-pod - kubectl apply -f pod.yaml
The provided reference passage explicitly shows the command to update a container's image using kubectl patch with a merge patch where spec.containers[*].name is required as a merge key.
## Safe Configuration Path Changing an image should follow the smallest justified change: one image at a time in one workload, with an observable verification. ### Image Update in a Deployment Assume you have a Deployment web running nginx:1.25.2 and want to update to nginx:1.25.3 . First, confirm current image: kubectl get deployment web -o jsonpath='{.spec.template.spec.containers[0].image}' # nginx:1.25.2 Observe the current rollout status: kubectl rollout status deployment/web --timeout=60s # deployment "web" successfully rolled out Now update the image using the imperative command (smallest change): kubectl set image deployment/web nginx=nginx:1.25.3 # deployment.apps/web image updated Watch the rollout and verify: kubectl rollout status deployment/web --timeout=120s # Waiting for deployment "web" rollout to finish: 1 out of 2 new replicas have been updated... # deployment "web" successfully rolled out Confirm the new image is live: kubectl get pods -l app=web -o jsonpath='{.items[*].spec.containers[0].image}{"\n"}' # nginx:1.25.3 nginx:1.25.3 Rollback if needed (recovery path): kubectl rollout undo deployment/web # deployment.apps/web rolled back kubectl rollout status deployment/web # deployment "web" successfully rolled out Check image after rollback: kubectl get deployment web -o jsonpath='{.spec.template.spec.containers[0].image}' # nginx:1.25.2 ### Image Pull Policy By default, Kubernetes uses IfNotPresent for images with an explicit tag (like nginx:1.25.3 ) and Always for images with latest tag. This can cause unexpected caching. Example: set pull policy to Always to ensure latest image in a development namespace. Edit deployment: kubectl patch deployment web -p '{"spec":{"template":{"spec":{"containers":[{"name":"nginx","imagePullPolicy":"Always"}]}}}}' # deployment.apps/web patched Verify with: kubectl get deployment web -o jsonpath='{.spec.template.spec.containers[0].imagePullPolicy}' # Always Note: Always adds registry load and may slow pod starts. For production, use explicit tags and IfNotPresent (default) or a digest. ### Using Image Digests for Immutability Pinning to a digest prevents accidental changes if the tag is overwritten. Find digest: docker buildx imagetools inspect nginx:1.25.3 --format '{{.Manifest.Digest}}' # sha256:... (actual hash) Or use skopeo inspect docker://nginx:1.25.3 | jq -r '.Digest' . Update deployment to use digest: kubectl set image deployment/web nginx=nginx@sha256:abcdef1234567890... Verify: kubectl get deployment web -o jsonpath='{.spec.template.spec.containers[0].image}' # nginx@sha256:abcdef1234567890... Rollback works the same via kubectl rollout undo . ### Private Registry Credentials If your image is in a private registry, create a Secret and add it to the pod spec. Create secret: kubectl create secret docker-registry my-registry-secret \ --docker-server=registry.example.com \ --docker-username=myuser \ --docker-password='s3cr3t' \ --docker-email=user@example.com # secret/my-registry-secret created Patch the service account or deployment to use it (service account is recommended for cluster-wide default): kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"my-registry-secret"}]}' Or on the pod spec directly: kubectl patch deployment web -p '{"spec":{"template":{"spec":{"imagePullSecrets":[{"name":"my-registry-secret"}]}}}}' Never store plaintext passwords in deployment YAML; use Secrets and avoid committing them to git. ### Keeping Local Test Small for Safe Configuration Before moving to external load balancer or ingress, test locally with kubectl port-forward : kubectl port-forward deployment/web 8080:80 # Forwarding from 127.0.0.1:8080 -> 80 # Forwarding from [::1]:8080 -> 80 Then test with curl http://localhost:8080 . This checks the pod directly without exposing to the network. ## Verification and Diagnostics After an image change, verify the app is healthy and the correct image is running. ### Pod Status and Image Details List pods with images: kubectl get pods -o wide # NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES # web-5f6cf84df-8n2k5 1/1 Running 0 2m 10.244.2.5 node2 # web-5f6cf84df-v9xqt 1/1 Running 0 2m 10.244.1.7 node1 Get detailed image information: kubectl get pods -o jsonpath='{.items[*].spec.containers[0].image}' # nginx:1.25.3 nginx:1.25.3 Check current deployment image: kubectl get deployment web -o jsonpath='{.spec.template.spec.containers[0].image}' # nginx:1.25.3 ### Logs and Events for Diagnostics If pods are crash-looping, get logs: kubectl logs deployment/web --tail=50 # Sample: 2024/01/15 error: image not found For previous container instance: kubectl logs --previous Describe pod for events: kubectl describe pod # Events: # Type Reason Age From Message # ---- ------ ---- ---- ------- # Normal Scheduled 3m default-scheduler Successfully assigned default/web-5f6cf84df-8n2k5 to node2 # Normal Pulled 2m59s kubelet Container image "nginx:1.25.3" already present on machine # Normal Created 2m59s kubelet Created container nginx # Normal Started 2m58s kubelet Started container nginx If image pull fails, you'll see Failed with ErrImagePull or ImagePullBackOff . ### Check Image Pull Errors Across Cluster kubectl get events --all-namespaces --field-selector reason=FailedToPullImage,reason=ErrImagePull ### Verifying Runtime Image Inside a running pod, confirm the actual process image (requires sh): kubectl exec -it -- cat /etc/os-release # NAME="Alpine Linux" # ID=alpine # VERSION_ID=3.19.0 Or check the container's image metadata from node: crictl images | grep nginx if you have node access. ### Rollout History Check rollout history for image changes: kubectl rollout history deployment/web # REVISION CHANGE-CAUSE # 1 # 2 If you used --record (deprecated in 1.27) or annotate with change-cause, you'll see details. To view a revision's image: kubectl rollout history deployment/web --revision=2 ### Health Check Verification If your pod has readiness probes, verify they pass: kubectl get pods -l app=web -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}{"\n"}' # True True And endpoints: kubectl get endpoints web # NAME ENDPOINTS AGE # web 10.244.1.7:80,10.244.2.5:80 5m ## Failure Modes and Recovery Image-related failures typically fall into pull failures, wrong image, missing secrets, or runtime incompatibility. This section covers common failure modes and explicit recovery steps. ### ImagePullBackOff Cause: image does not exist, tag incorrect, registry unreachable, or private registry credentials missing. Observe: kubectl get pods # NAME READY STATUS RESTARTS AGE # web-7fd69d9c6-5k4j2 0/1 ImagePullBackOff 0 30s Get details: kubectl describe pod web-7fd69d9c6-5k4j2 | tail -20 # Events: # Type Reason Age From Message # ---- ------ ---- ---- ------- # Normal Scheduled 35s default-scheduler Successfully assigned default/web-7fd69d9c6-5k4j2 to node1 # Normal Pulling 34s kubelet Pulling image "nginx:1.25.9" # Warning Failed 33s kubelet Failed to pull image "nginx:1.25.9": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:1.25.9": failed to resolve reference "docker.io/library/nginx:1.25.9": not found # Warning Failed 33s kubelet Error: ErrImagePull # Normal BackOff 32s kubelet Back-off pulling image "nginx:1.25.9" Recovery: - Correct the image tag in the deployment: kubectl set image deployment/web nginx=nginx:1.25.3 - Verify rollout: kubectl rollout status deployment/web - If private registry, verify secret exists and is correct: kubectl get secret my-registry-secret -o yaml Ensure .dockerconfigjson contains correct auth. ### CrashLoopBackOff due to Wrong Image Architecture You may pull an ARM64 image on AMD64 node. Pod starts but crashes with exec format error. Observe: kubectl logs # exec /entrypoint.sh: exec format error Recovery: - Verify node architecture: kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.architecture}' # amd64 amd64 - Use multi-arch image or correct architecture tag. For example, use nginx:1.25.3 which supports amd64 and arm64, or specify --platform=linux/amd64 if building. - Update image and verify. ### Image Pull Secret Missing If pod can't pull from private registry, status shows ImagePullBackOff with unauthorized: authentication required . Observe: kubectl describe pod | grep -A5 "Failed to pull image" # Warning Failed 10s (x3 over 30s) kubelet Failed to pull image "registry.example.com/myapp:1.0": rpc error: code = Unknown desc = failed to pull and unpack image "registry.example.com/myapp:1.0": failed to resolve reference "registry.example.com/myapp:1.0": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed Recovery: - Create or update the secret as shown earlier. - Add imagePullSecrets to the pod or service account. - Delete the pod to force re-pull: kubectl delete pod New pod should start. ### Image Update Caused Application Failure If new image is running but app is failing (e.g., wrong entrypoint, missing config), you can rollback. Quick rollback: kubectl rollout undo deployment/web Or to a specific revision: kubectl rollout undo deployment/web --to-revision=1 Then verify: kubectl rollout status deployment/web kubectl get pods -l app=web -o jsonpath='{.items[*].spec.containers[0].image}' ### Disk Pressure from Large Images If nodes run out of disk due to many images or large layers, pods may be evicted. Observe node conditions: kubectl describe node | grep -A5 "Conditions:" # Conditions: # Type Status LastHeartbeatTime LastTransitionTime Reason Message # ---- ------ ----------------- ------------------ ------ ------- # MemoryPressure False Tue, 16 Jan 2024 10:00:00 +0000 Tue, 16 Jan 2024 09:00:00 +0000 KubeletHasSufficientMemory kubelet has sufficient memory available # DiskPressure True Tue, 16 Jan 2024 10:05:00 +0000 Tue, 16 Jan 2024 10:04:30 +0000 KubeletHasDiskPressure kubelet has disk pressure Recovery: - Remove unused images on node: crictl rmi --prune (if you have node access) - Set image garbage collection threshold: kubelet flag --image-gc-high-threshold=85 (default 85%) - Use smaller images or image streaming. ### Failed to Pull due to Network or Registry Downtime If registry is unavailable, pods will show ImagePullBackOff with connection refused or timeout . Check connectivity from node: kubectl debug node/ -it --image=busybox -- sh # inside debug pod: wget -O- https://registry-1.docker.io/v2/ ; echo $? Recovery: - Fix registry endpoint/DNS/firewall. - Use a local mirror or --registry-mirror in container runtime. - Update image source if using a different registry.
Quick check 2 of 2 What is the purpose of using an image scanning tool before deploying containers? - To improve container performance - To scan an image for vulnerabilities before deployment - To automatically update images - To create container images
The reference passage lists 'Using an image scanning tool to scan an image before deploying containers' as a security measure, implying its purpose is to identify vulnerabilities.
## Operations Checklist Use this checklist before, during, and after any image-related operation to maintain safety and recoverability. ### 1. Pre-Change Checklist - [ ] Record current cluster version: kubectl version - [ ] Record current image for the workload: kubectl get deployment -o jsonpath='{.spec.template.spec.containers[0].image}' - [ ] Note rollout status: kubectl rollout status deployment/ - [ ] Verify registry connectivity: curl -I https://registry-1.docker.io/v2/ (or your registry) - [ ] Check disk space on nodes: kubectl describe nodes | grep -A5 DiskPressure - [ ] Backup deployment YAML: kubectl get deployment -o yaml > backup--$(date +%F).yaml ### 2. Change Execution - [ ] Apply one image change with kubectl set image or patch - [ ] Watch rollout progress: kubectl rollout status deployment/ --timeout=90s - [ ] If new pods not coming up, check events: kubectl get events --field-selector reason=FailedToPullImage,reason=ErrImagePull ### 3. Post-Change Verification - [ ] Confirm new image running: kubectl get pods -l app=