## Intro Kubernetes init containers often fail for reasons that are not obvious from pod status alone. An operator sees `Init:0/1` or `Init:Error` and needs a reliable path from symptom to resolution. This guide provides practical, command-driven troubleshooting steps for Kubernetes init containers, with concrete examples, expected output, and recovery verification. It is intended for developers, DevOps engineers, and platform teams who run Kubernetes in production or staging. We cover version and environment inventory, safe configuration, verification and diagnostics, failure modes and recovery, and an operations checklist. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery before an incident forces a decision. ## Version and Environment Inventory Before troubleshooting an init container failure, gather precise information about your Kubernetes cluster and the affected pod. This reduces false assumptions and narrows the problem space. ### Check cluster version and API deprecations ```bash kubectl version --short ``` Expected output (example): ``` Client Version: v1.28.2 Server Version: v1.28.2 ``` If the init container uses a beta API that was removed in the server version (for example, `extensions/v1beta1` removed in 1.16), the pod will fail to create. Confirm with: ```bash kubectl explain pod.spec.initContainers | head -20 ``` If the command returns `error: the server doesn't have a resource type "pod"` or similar, check cluster connectivity and credentials. ### Identify pod, namespace, and init container names ```bash kubectl get pods -n -o wide ``` Example output: ``` NAME READY STATUS RESTARTS AGE IP NODE web-app-6d4b8c-abc 0/1 Init:0/2 0 5m 10.244.2.5 node-1 ``` The `STATUS` column shows the init container state: `Init:0/2` means 0 of 2 init containers completed. To list init container names in a pod: ```bash kubectl get pod web-app-6d4b8c-abc -n -o jsonpath='{.spec.initContainers[*].name}' ``` Expected output: ``` init-db init-migrate ``` ### Inspect pod events for first signal ```bash kubectl describe pod web-app-6d4b8c-abc -n ``` Look for events at the bottom. Example: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 5m default-scheduler Successfully assigned default/web-app-6d4b8c-abc to node-1 Warning Failed 4m kubelet Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "bootstrap.sh": executable file not found in $PATH: unknown ``` This event immediately points to a missing executable in the init container image. ## Safe Configuration Path When modifying any configuration, always capture the current state and timestamp, then apply the smallest change that can be reversed. Never put secrets directly in manifests; use Kubernetes Secrets or external secret stores. ### Capture current manifest and checksum ```bash kubectl get pod web-app-6d4b8c-abc -n -o yaml > pod-before.yaml shasum -a 256 pod-before.yaml ``` Example output: ``` e1c7f8f2b6a4d... pod-before.yaml ``` ### Common misconfiguration: wrong command or arguments Many init containers fail because the `command` or `args` override the image's default entrypoint incorrectly. For example, this manifest intends to wait for a database but uses a shell built-in incorrectly: ```yaml initContainers: - name: init-mysql image: busybox:1.36 command: ["until nc -z mysql 3306; do sleep 2; done"] ``` The command is a single string, but `command` expects an array. It should be: ```yaml initContainers: - name: init-mysql image: busybox:1.36 command: ['sh', '-c', 'until nc -z mysql 3306; do echo waiting for mysql; sleep 2; done'] ``` After correcting, apply the change: ```bash kubectl apply -f deployment.yaml kubectl rollout status deployment/web-app -n --timeout=120s ``` Expected output when successful: ``` deployment "web-app" successfully rolled out ``` ### Environment variables and secrets Use secret references instead of literal values: ```yaml env: - name: DB_PASSWORD valueFrom: secretKeyRef: name: db-secret key: password ``` Verify the secret exists before the pod is created: ```bash kubectl get secret db-secret -n ``` If missing, init container may fail with `CreateContainerConfigError`. See Failure Modes. ### Version compatibility notes - Kubernetes 1.20+ supports ephemeral containers for debugging, but init containers are standard. - If using `restartPolicy: Always` in a pod with init containers, failed init containers restart according to policy. ## Verification and Diagnostics This section shows step-by-step diagnostic commands to pinpoint the exact failure. ### Get init container logs ```bash kubectl logs web-app-6d4b8c-abc -n -c init-mysql --tail=50 ``` If the init container crashed and restarted, use `--previous`: ```bash kubectl logs web-app-6d4b8c-abc -n -c init-mysql --previous ``` Example log for a connection refusal: ``` /bin/sh: nc: not found ``` This indicates missing `nc` in busybox. Use `busybox:1.36` with `nc` via `busybox-extras` or use a different image like `alpine/socat`. ### Check exit codes and termination reason ```bash kubectl get pod web-app-6d4b8c-abc -n -o jsonpath='{.status.initContainerStatuses[?(@.name=="init-mysql")].state}' ``` Example output: ```json {"terminated":{"exitCode":127,"reason":"ContainerCannotRun","message":"exec: \"nc\": executable file not found in $PATH","startedAt":"...","finishedAt":"...","containerID":"..."}} ``` Common exit codes: - `126`: permission denied or command not executable - `127`: command not found - `137`: SIGKILL (OOM or liveness failure) - `1`: general application error ### Inspect resource limits and node conditions If init container is stuck in `Init:0/1` for a long time, check if it is OOMKilled: ```bash kubectl describe pod web-app-6d4b8c-abc -n | grep -A5 'State' ``` Example: ``` State: Waiting Reason: CrashLoopBackOff Last State: Terminated Reason: OOMKilled Exit Code: 137 ``` Increase memory limit or optimize the init container. ### Debug with ephemeral containers (K8s 1.23+) If the init container cannot start at all, you can run an ephemeral debug container to inspect the environment: ```bash kubectl debug -it web-app-6d4b8c-abc -n --image=busybox:1.36 --target=init-mysql ``` Then test connectivity manually. ## Failure Modes and Recovery This section covers specific failure modes, their causes, and recovery steps. ### Failure: `Init:Error` or `Init:CrashLoopBackOff` Cause: the init container process exits with non-zero code repeatedly. Recovery: 1. Get logs and exit code as above. 2. Fix the command, script bug, or missing dependency. 3. Apply fix and force a new rollout if needed: ```bash kubectl rollout restart deployment/web-app -n ``` 4. Monitor: ```bash kubectl get pods -n -w ``` ### Failure: `Init:CreateContainerConfigError` Cause: referenced ConfigMap or Secret does not exist, or permissions issue. Example event: ``` Warning Failed 4s kubelet Error: secret "db-secret" not found ``` Recovery: ```bash kubectl create secret generic db-secret --from-literal=password='S3cureP@ss' -n # or apply the secret manifest kubectl apply -f db-secret.yaml ``` Then delete the pod to restart init containers: ```bash kubectl delete pod web-app-6d4b8c-abc -n ``` ### Failure: `Init:ImagePullBackOff` Cause: wrong image name, tag, authentication, or network policy. Check events: ```bash kubectl describe pod web-app-6d4b8c-abc -n | grep -A5 'Events' ``` Example: ``` Warning Failed 10s kubelet Failed to pull image "myregistry/init:v2": rpc error: code = NotFound desc = failed to pull and unpack image "myregistry/init:v2": failed to resolve reference "myregistry/init:v2": not found ``` Recovery: correct image name/tag, create imagePullSecret: ```bash kubectl create secret docker-registry regcred --docker-server=myregistry --docker-username=user --docker-password=pass --docker-email=user@example.com ``` Add `imagePullSecrets` to pod spec. ### Failure: init container completes but app container fails This is not an init container failure per se, but verify init container succeeded: ```bash kubectl get pod web-app-6d4b8c-abc -n -o jsonpath='{.status.initContainerStatuses[*].state.terminated.exitCode}' ``` If output is `0`, init containers are successful; troubleshoot app container separately. ## Operations Checklist Use this checklist before and after any change to init containers. ### Before change - [ ] Confirm cluster version and API compatibility. - [ ] Capture current pod manifest with timestamp: `kubectl get pod -o yaml > pod-$(date +%Y%m%d%H%M%S).yaml` - [ ] Identify init container names: `kubectl get pod -o jsonpath='{.spec.initContainers[*].name}'` - [ ] Note current status: `kubectl get pods -n ` - [ ] Check if any secrets/configmaps referenced exist. ### After change - [ ] Apply change: `kubectl apply -f updated-manifest.yaml` - [ ] Watch rollout: `kubectl rollout status deployment/ --timeout=120s` - [ ] Verify new pod reached `Running` with all init containers completed: ```bash kubectl wait --for=condition=Initialized pod/web-app-6d4b8c-abc -n --timeout=60s ``` Expected: `pod/web-app-6d4b8c-abc condition met` - [ ] Test application endpoint or functionality. - [ ] Document the failure mode, cause, and fix in runbook. ### Example runbook entry ``` Date: 2025-03-15 Issue: Init container init-mysql exited 127, nc not found. Fix: Changed image from busybox:1.35 to alpine/socat:1.7.4.3 and adjusted command. Verification: pod reached Running in 45s, log shows "waiting for mysql" then success. ``` ## Conclusion Troubleshooting Kubernetes init containers requires a methodical approach: gather version and environment details, inspect events and logs, check exit codes, and apply minimal fixes with verification. The examples in this guide provide a template for common failures such as missing binaries, bad commands, missing secrets, and image pull errors. Always observe before changing, safeguard credentials, and define recovery verification before an incident. Choose one low-risk verification from this guide, record the current state, run the documented check, compare the result, and review dependencies like pod lifecycle and configuration. A reliable workflow makes failure visible and limits changes to the intended resource.