Kubernetes Init Containers Troubleshooting: Practical Examples and Recovery Guide

Intro

Kubernetes init containers often fail for reasons that are not obvious from pod status alone. An operator sees Init:0/1 or Init:Error and needs a reliable path from symptom to resolution. This guide provides practical, command-driven troubleshooting steps for Kubernetes init containers, with concrete examples, expected output, and recovery verification. It is intended for developers, DevOps engineers, and platform teams who run Kubernetes in production or staging. We cover version and environment inventory, safe configuration, verification and diagnostics, failure modes and recovery, and an operations checklist. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery before an incident forces a decision.

Version and Environment Inventory

Before troubleshooting an init container failure, gather precise information about your Kubernetes cluster and the affected pod. This reduces false assumptions and narrows the problem space.

Check cluster version and API deprecations

kubectl version --short

Expected output (example):

Client Version: v1.28.2
Server Version: v1.28.2

If the init container uses a beta API that was removed in the server version (for example, extensions/v1beta1 removed in 1.16), the pod will fail to create. Confirm with:

kubectl explain pod.spec.initContainers | head -20

If the command returns error: the server doesn't have a resource type "pod" or similar, check cluster connectivity and credentials.

Identify pod, namespace, and init container names

kubectl get pods -n <namespace> -o wide

Example output:

NAME                READY   STATUS     RESTARTS   AGE   IP           NODE
web-app-6d4b8c-abc  0/1     Init:0/2   0          5m    10.244.2.5   node-1

The STATUS column shows the init container state: Init:0/2 means 0 of 2 init containers completed. To list init container names in a pod:

kubectl get pod web-app-6d4b8c-abc -n <namespace> -o jsonpath='{.spec.initContainers[*].name}'

Expected output:

init-db init-migrate

Inspect pod events for first signal

kubectl describe pod web-app-6d4b8c-abc -n <namespace>

Look for events at the bottom. Example:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  5m    default-scheduler  Successfully assigned default/web-app-6d4b8c-abc to node-1
  Warning  Failed     4m    kubelet            Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "bootstrap.sh": executable file not found in $PATH: unknown

This event immediately points to a missing executable in the init container image.

Safe Configuration Path

When modifying any configuration, always capture the current state and timestamp, then apply the smallest change that can be reversed. Never put secrets directly in manifests; use Kubernetes Secrets or external secret stores.

Capture current manifest and checksum

kubectl get pod web-app-6d4b8c-abc -n <namespace> -o yaml > pod-before.yaml
shasum -a 256 pod-before.yaml

Example output:

e1c7f8f2b6a4d...  pod-before.yaml

Common misconfiguration: wrong command or arguments

Many init containers fail because the command or args override the image's default entrypoint incorrectly. For example, this manifest intends to wait for a database but uses a shell built-in incorrectly:

initContainers:
  - name: init-mysql
    image: busybox:1.36
    command: ["until nc -z mysql 3306; do sleep 2; done"]

The command is a single string, but command expects an array. It should be:

initContainers:
  - name: init-mysql
    image: busybox:1.36
    command: ['sh', '-c', 'until nc -z mysql 3306; do echo waiting for mysql; sleep 2; done']

After correcting, apply the change:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/web-app -n <namespace> --timeout=120s

Expected output when successful:

deployment "web-app" successfully rolled out

Environment variables and secrets

Use secret references instead of literal values:

env:
  - name: DB_PASSWORD
    valueFrom:
      secretKeyRef:
        name: db-secret
        key: password

Verify the secret exists before the pod is created:

kubectl get secret db-secret -n <namespace>

If missing, init container may fail with CreateContainerConfigError. See Failure Modes.

Version compatibility notes

• Kubernetes 1.20+ supports ephemeral containers for debugging, but init containers are standard.
• If using restartPolicy: Always in a pod with init containers, failed init containers restart according to policy.

Verification and Diagnostics

This section shows step-by-step diagnostic commands to pinpoint the exact failure.

Get init container logs

kubectl logs web-app-6d4b8c-abc -n <namespace> -c init-mysql --tail=50

If the init container crashed and restarted, use --previous:

kubectl logs web-app-6d4b8c-abc -n <namespace> -c init-mysql --previous

Example log for a connection refusal:

/bin/sh: nc: not found

This indicates missing nc in busybox. Use busybox:1.36 with nc via busybox-extras or use a different image like alpine/socat.

Check exit codes and termination reason

kubectl get pod web-app-6d4b8c-abc -n <namespace> -o jsonpath='{.status.initContainerStatuses[?(@.name=="init-mysql")].state}'

Example output:

{"terminated":{"exitCode":127,"reason":"ContainerCannotRun","message":"exec: \"nc\": executable file not found in $PATH","startedAt":"...","finishedAt":"...","containerID":"..."}}

Common exit codes:

• 126: permission denied or command not executable
• 127: command not found
• 137: SIGKILL (OOM or liveness failure)
• 1: general application error

Inspect resource limits and node conditions

If init container is stuck in Init:0/1 for a long time, check if it is OOMKilled:

kubectl describe pod web-app-6d4b8c-abc -n <namespace> | grep -A5 'State'

Example:

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Increase memory limit or optimize the init container.

Debug with ephemeral containers (K8s 1.23+)

If the init container cannot start at all, you can run an ephemeral debug container to inspect the environment:

kubectl debug -it web-app-6d4b8c-abc -n <namespace> --image=busybox:1.36 --target=init-mysql

Then test connectivity manually.

Failure Modes and Recovery

This section covers specific failure modes, their causes, and recovery steps.

Failure: Init:Error or Init:CrashLoopBackOff

Cause: the init container process exits with non-zero code repeatedly. Recovery:

1. Get logs and exit code as above.
2. Fix the command, script bug, or missing dependency.
3. Apply fix and force a new rollout if needed:

kubectl rollout restart deployment/web-app -n <namespace>

1. Monitor:

kubectl get pods -n <namespace> -w

Failure: Init:CreateContainerConfigError

Cause: referenced ConfigMap or Secret does not exist, or permissions issue. Example event:

Warning  Failed     4s    kubelet            Error: secret "db-secret" not found

Recovery:

kubectl create secret generic db-secret --from-literal=password='S3cureP@ss' -n <namespace>
# or apply the secret manifest
kubectl apply -f db-secret.yaml

Then delete the pod to restart init containers:

kubectl delete pod web-app-6d4b8c-abc -n <namespace>

Failure: Init:ImagePullBackOff

Cause: wrong image name, tag, authentication, or network policy. Check events:

kubectl describe pod web-app-6d4b8c-abc -n <namespace> | grep -A5 'Events'

Example:

Warning  Failed     10s   kubelet            Failed to pull image "myregistry/init:v2": rpc error: code = NotFound desc = failed to pull and unpack image "myregistry/init:v2": failed to resolve reference "myregistry/init:v2": not found

Recovery: correct image name/tag, create imagePullSecret:

kubectl create secret docker-registry regcred --docker-server=myregistry --docker-username=user --docker-password=pass [email protected]

Add imagePullSecrets to pod spec.

Failure: init container completes but app container fails

This is not an init container failure per se, but verify init container succeeded:

kubectl get pod web-app-6d4b8c-abc -n <namespace> -o jsonpath='{.status.initContainerStatuses[*].state.terminated.exitCode}'

If output is 0, init containers are successful; troubleshoot app container separately.

Operations Checklist

Use this checklist before and after any change to init containers.

Before change

• [ ] Confirm cluster version and API compatibility.
• [ ] Capture current pod manifest with timestamp: kubectl get pod <name> -o yaml > pod-$(date +%Y%m%d%H%M%S).yaml
• [ ] Identify init container names: kubectl get pod <name> -o jsonpath='{.spec.initContainers[*].name}'
• [ ] Note current status: kubectl get pods -n <namespace>
• [ ] Check if any secrets/configmaps referenced exist.

After change

• [ ] Apply change: kubectl apply -f updated-manifest.yaml
• [ ] Watch rollout: kubectl rollout status deployment/<deployment> --timeout=120s
• [ ] Verify new pod reached Running with all init containers completed:

kubectl wait --for=condition=Initialized pod/web-app-6d4b8c-abc -n <namespace> --timeout=60s

Expected: pod/web-app-6d4b8c-abc condition met

• [ ] Test application endpoint or functionality.
• [ ] Document the failure mode, cause, and fix in runbook.

Example runbook entry

Date: 2025-03-15
Issue: Init container init-mysql exited 127, nc not found.
Fix: Changed image from busybox:1.35 to alpine/socat:1.7.4.3 and adjusted command.
Verification: pod reached Running in 45s, log shows "waiting for mysql" then success.

Conclusion

Troubleshooting Kubernetes init containers requires a methodical approach: gather version and environment details, inspect events and logs, check exit codes, and apply minimal fixes with verification. The examples in this guide provide a template for common failures such as missing binaries, bad commands, missing secrets, and image pull errors. Always observe before changing, safeguard credentials, and define recovery verification before an incident. Choose one low-risk verification from this guide, record the current state, run the documented check, compare the result, and review dependencies like pod lifecycle and configuration. A reliable workflow makes failure visible and limits changes to the intended resource.