Kubernetes Init Containers: Configuration Mistakes and How to Avoid Them

Intro

Kubernetes init containers are a powerful mechanism for preparing an application pod before the main containers start. They run to completion before the app containers launch, making them ideal for tasks like waiting for dependencies, setting up file permissions, or performing database migrations. However, init container configuration is easy to get wrong, and mistakes can lead to pods stuck in Init:Error, Init:CrashLoopBackOff, or silent behavior changes that only surface later.

This article walks through common configuration mistakes with practical examples, showing how to diagnose, fix, and prevent them. You will learn to validate init container behavior, roll back bad changes safely, and troubleshoot failures using concrete kubectl commands and manifest snippets. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery steps before an incident forces your hand.

We'll assume you have a working Kubernetes cluster (version 1.20 or later for stable init container features, but examples work on most modern versions), kubectl configured, and basic familiarity with pods and deployments. All examples use a demo namespace called init-demo to keep things isolated.

Version and Environment Inventory

Before changing any init container configuration, you need to know exactly what you're running. Gather the Kubernetes server version, the API resources involved, and the current state of your pods. This section covers the read-only observations that inform every later step.

Check cluster version and API support

Init containers have been stable since Kubernetes 1.6, but certain fields like restartPolicy: Always for sidecar-style init containers (alpha in 1.28, behind a feature gate) may not be available in your distribution. Confirm your server version:

kubectl version --short

Expected output shows both client and server versions, for example:

Client Version: v1.28.2
Server Version: v1.27.4

If your server version is older than 1.20, be cautious with newer init container features. Document the version in your runbook; it matters when troubleshooting.

Inventory the current pod state

Suppose you have a deployment with an init container that copies a config file into a shared volume. Get the current pods with wide output:

kubectl get pods -n init-demo -o wide

Example output:

NAME                           READY   STATUS     RESTARTS   AGE   IP            NODE
webapp-7f4c8d6b9c-5g2kx        0/1     Init:0/1   0          12s   10.244.1.8    worker-1
webapp-7f4c8d6b9c-8mzpl        0/1     Init:0/1   0          12s   10.244.1.9    worker-1

The STATUS column shows Init:0/1, meaning the pod has one init container and it hasn't completed yet. This is normal during startup, but if it stays like that, something is wrong.

Capture events and describe the pod

Use kubectl describe to see init container statuses and events:

kubectl describe pod webapp-7f4c8d6b9c-5g2kx -n init-demo

Focus on the Init Containers section and the Events at the bottom. A healthy init container that has finished will show:

Init Containers:
  init-config:
    Container ID:  containerd://abcd1234...
    Image:         busybox:1.36
    State:          Terminated
      Reason:       Completed
      Exit Code:    0

If the init container is failing, you'll see State: Waiting with Reason: CrashLoopBackOff or Error, and events like:

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Normal   Scheduled  30s   default-scheduler  Successfully assigned init-demo/webapp-7f4c8d6b9c-5g2kx to worker-1
  Warning  Failed     29s   kubelet            Error: Init container init-config failed: exit code 1

Always collect this information before making changes. It tells you if the problem is in the init container itself or in the pod's scheduling or environment.

Keep the test small

For any configuration change, start with a minimal manifest in a dedicated namespace, not your production deployment. For example, use a simple pod like this:

apiVersion: v1
kind: Pod
metadata:
  name: init-test
  namespace: init-demo
spec:
  initContainers:
  - name: init-echo
    image: busybox:1.36
    command: ['sh', '-c', 'echo "init done" && sleep 2']
  containers:
  - name: main
    image: nginx:1.25

Apply it and watch the status transition:

kubectl apply -f init-test.yaml
kubectl get pod init-test -n init-demo --watch

The watch output should show Init:0/1, then PodInitializing, then Running. If it goes to Error or CrashLoopBackOff, you have a quick signal to investigate.

Safe Configuration Path

Now that you understand the current environment, follow a safe path for changing init container configuration. The principle: observe, change one thing, verify, and be ready to roll back.

Identify the exact mistake

Common init container configuration mistakes include:

Wrong command or arguments: The init container exits non-zero because the command is incorrect.
Missing or incorrect volume mounts: The init container writes to a path that isn't shared with the main container, so the data is lost.
Image pull failures: Wrong image tag or missing registry credentials.
Resource limits too low: The init container gets OOMKilled before completing.
No restart policy awareness: Init containers always restart until they succeed (unless you use restartPolicy: Never on the pod, which is rare). A failing init container with a bad command will cause endless restarts.

Let's illustrate with a concrete mistake: an init container that waits for a database to be ready using a command that never succeeds because it's using the wrong service name.

initContainers:
- name: wait-for-db
  image: busybox:1.36
  command: ['sh', '-c', 'until nc -z database 5432; do echo waiting; sleep 2; done']

If the service is actually named postgres-svc, the init container will loop forever (or until it hits the pod's activeDeadlineSeconds, if set). The pod will stay Init:0/1 indefinitely.

Apply the smallest justified change

Rather than editing multiple fields at once, fix one thing. For the example above, change the hostname to the correct service name:

initContainers:
- name: wait-for-db
  image: busybox:1.36
  command: ['sh', '-c', 'until nc -z postgres-svc 5432; do echo waiting; sleep 2; done']

Apply with kubectl apply and watch the pod:

kubectl apply -f deployment.yaml
kubectl get pods -n init-demo --watch

Expected progression: Init:0/1 (waiting for DB), then once the DB is reachable, PodInitializing, then Running for the main container.

Use placeholders instead of secrets

When init containers need sensitive data (like database passwords), never hardcode them. Use Kubernetes Secrets and environment variables or volume mounts. For example, to pass a password to an init container that initializes a schema:

apiVersion: v1
kind: Secret
metadata:
  name: db-init-secret
  namespace: init-demo
type: Opaque
stringData:
  DB_PASS: "s3cr3t"   # In real life, use a vault or external secrets operator
---
apiVersion: v1
kind: Pod
metadata:
  name: db-migrate
  namespace: init-demo
spec:
  initContainers:
  - name: run-migration
    image: postgres:16
    env:
    - name: PGPASSWORD
      valueFrom:
        secretKeyRef:
          name: db-init-secret
          key: DB_PASS
    command: ['sh', '-c', 'psql -h postgres-svc -U app -d appdb -f /migrations/001.sql']
  containers:
  - name: main
    image: nginx:1.25

Apply and verify the secret is used, not printed in logs. Avoid using kubectl logs on init containers in production if they might expose secret values; use kubectl logs <pod> -c <init-container> with caution.

Rollback strategy

If the change breaks things, you need a quick way to revert. With deployments, you can use rollout history and undo:

kubectl rollout history deployment webapp -n init-demo
kubectl rollout undo deployment webapp -n init-demo

For raw pods, you'll need to reapply a known-good manifest. Keep a backup of the previous YAML either in version control or as a file:

kubectl get pod webapp -n init-demo -o yaml > webapp-backup.yaml   # before the change
# if change fails:
kubectl delete pod webapp -n init-demo
kubectl apply -f webapp-backup.yaml

Document this rollback procedure in your runbook before making changes, not after.

Verification and Diagnostics

After applying a configuration change, verify that the init container behaves as intended and that the main container starts correctly. This section gives exact commands and expected outputs for different scenarios.

Check init container logs

Init container logs are often the first place to look. Use -c to specify the init container name:

kubectl logs webapp-7f4c8d6b9c-5g2kx -c init-config -n init-demo

For a successful init container that copies files, you might see no output if the command is silent. To verify, use a command that prints a clear completion message, like:

command: ['sh', '-c', 'cp /config/* /data/ && echo "config copied successfully"']

Then kubectl logs should show:

config copied successfully

If the init container is crash-looping, use --previous to see logs from the last failed attempt:

kubectl logs webapp-7f4c8d6b9c-5g2kx -c init-config -n init-demo --previous

Example failure log:

cp: can't create '/data/config.txt': No such file or directory

This tells you the /data directory doesn't exist in the init container's filesystem, likely because the volume mount is wrong.

Describe the pod for init container status

kubectl describe shows the state of each init container. For the failure above, you'd see:

Init Containers:
  init-config:
    State:          Terminated
      Reason:       Error
      Exit Code:    1
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
    Ready:          False
    Restart Count:  4

The Restart Count increasing indicates a crash loop. Check events for more context.

Validate the main container readiness

Once the init container completes, ensure the main container becomes ready. Use kubectl get pods to see the READY column change from 0/1 to 1/1. For deployments, check rollout status:

kubectl rollout status deployment webapp -n init-demo

Expected output:

deployment "webapp" successfully rolled out

If it hangs, the main container might be failing its readiness probe after the init container succeeded. In that case, the init container is not the problem, but its side effects could be. For example, if the init container sets file permissions incorrectly, the main app might fail to read a file.

Test connectivity or data

After the pod is running, verify that the init container's work had the intended effect. For example, if the init container populated a shared volume with a config file, exec into the main container and check:

kubectl exec -it webapp-7f4c8d6b9c-5g2kx -n init-demo -- cat /etc/app/config.yaml

Expected output shows the config content you expect. If it's missing or wrong, review the volume mounts between init and main containers. They must use the same name and mountPath.

Failure Modes and Recovery

Even with careful planning, init containers can fail. This section covers common failure scenarios, how to diagnose them quickly, and how to recover with minimal downtime.

Failure mode 1: Init container exit code non-zero

Symptom: Pod stuck in Init:Error or Init:CrashLoopBackOff.

Diagnosis:

kubectl describe pod <pod-name> -n init-demo
kubectl logs <pod-name> -c <init-container-name> --previous

Example: An init container that runs a database migration script and the script fails due to a duplicate column. Logs show:

psql:/migrations/002.sql:3: ERROR: column "email" of relation "users" already exists

Recovery: Fix the migration script (e.g., make it idempotent) and update the configmap or image that contains the script. If the init container image is baked, build a new image and update the deployment. Then delete the stuck pod to restart cleanly:

kubectl delete pod webapp-7f4c8d6b9c-5g2kx -n init-demo

The deployment controller will create a new pod that runs the fixed init container.

Failure mode 2: Init container times out

Symptom: Pod stays in Init:0/1 for a long time, then may be killed if activeDeadlineSeconds is set on the pod.

Diagnosis: Check the init container command. If it's waiting for an external service, ensure the service is reachable from within the cluster. Use a temporary pod to test connectivity:

kubectl run test-conn --rm -it --image=busybox:1.36 -n init-demo -- sh -c 'nc -zv postgres-svc 5432'

Expected output if reachable:

postgres-svc (10.96.0.10:5432) open

If not reachable, check the service, endpoints, and network policies.

Recovery: Correct the service name or network policy, apply the change, and delete the stuck pod.

Failure mode 3: Image pull failure

Symptom: Pod in Init:ErrImagePull or Init:ImagePullBackOff.

Diagnosis: kubectl describe pod shows events like:

Warning  Failed     30s   kubelet            Failed to pull image "myrepo/init-helper:v1.0": rpc error: code = NotFound desc = failed to pull and unpack image

Check the image name and tag. Ensure it exists in the registry and that the node has pull credentials if needed (imagePullSecrets in the pod spec).

Recovery: Fix the image reference in the pod or deployment spec. If it's a private registry, add or update the imagePullSecrets:

spec:
  imagePullSecrets:
  - name: regcred
  initContainers:
  - name: init-helper
    image: myrepo/init-helper:v1.1

Apply and the pod should restart with the corrected image.

Failure mode 4: Resource exhaustion

Symptom: Init container repeatedly restarts; describe shows OOMKilled in last state.

Diagnosis: kubectl describe pod shows:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

Check resource requests/limits for the init container. If not set, the container may consume more than the node can provide.

Recovery: Set appropriate limits in the init container spec. For example:

initContainers:
- name: data-loader
  image: myrepo/loader:1.0
  resources:
    requests:
      memory: "256Mi"
      cpu: "250m"
    limits:
      memory: "512Mi"
      cpu: "500m"

Apply the change and the init container should complete without OOM.

Rollback strategies for init container changes

If a new init container configuration causes pod failures, the fastest recovery is often to revert to the previous deployment revision. For a Deployment:

kubectl rollout undo deployment webapp -n init-demo

For a StatefulSet or DaemonSet, you may need to roll back the manifest manually. Always keep a known-good manifest version in git and apply it:

kubectl apply -f deployment-webapp-known-good.yaml

If the init container change was made directly to a running pod (not recommended), delete the pod to let the controller recreate it with the old spec.

Operations Checklist

Use this checklist before and after any init container configuration change to minimize risk and speed up recovery.

Pre-change checklist

[ ] Confirm cluster version and API compatibility for init container features you plan to use (kubectl version --short).
[ ] Record current pod status and init container state (kubectl get pods -n <namespace> -o wide, kubectl describe pod <pod-name>).
[ ] Capture current deployment revision and history (kubectl rollout history deployment <name>).
[ ] Backup the current manifest (kubectl get deploy <name> -o yaml > deploy-backup.yaml).
[ ] Test the new init container logic in a scratch pod in a separate namespace (kubectl run init-test --image=... --restart=Never --command -- <test command>).
[ ] Ensure secrets are not hardcoded; use Kubernetes Secrets or a secrets manager.
[ ] Set resource requests and limits for the init container.
[ ] Document the expected outcome and rollback command.

Post-change checklist

[ ] Watch pod progression (kubectl get pods -n <namespace> --watch), confirm transition from Init:0/1 to Running.
[ ] Check init container logs for success message or errors (kubectl logs <pod> -c <init-container>).
[ ] Verify main container readiness (kubectl get pods shows 1/1 ready; kubectl rollout status deployment <name> succeeds).
[ ] Validate the side effect of the init container (e.g., exec into main container and check file/data).
[ ] If failure occurs, run kubectl rollout undo deployment <name> or reapply known-good manifest.
[ ] Update documentation with any new findings or adjustments.

Example runbook entry

Here's a concrete example for a database migration init container in a deployment named api-server:

Runbook: api-server init container migration

Before rolling out a new migration version:

   kubectl get deploy api-server -n prod -o yaml > api-server-backup.yaml
   kubectl rollout history deployment api-server -n prod

Apply the new deployment with updated init container image tag:

   kubectl apply -f api-server-deploy.yaml

Watch pods:

   kubectl get pods -n prod -l app=api-server --watch

Expected: pods go through Init:0/1 to Running.

If pods are stuck in Init:Error, check logs:

   kubectl logs <new-pod> -c db-migrate -n prod --previous

If migration fails, rollback:

   kubectl rollout undo deployment api-server -n prod

After successful rollout, confirm migration by checking a database table version or an application health endpoint.

Conclusion

Init containers are a critical part of many Kubernetes workloads, but their configuration pitfalls can lead to silent failures or prolonged outages. By following a structured approach—version inventory, safe configuration, thorough verification, and explicit recovery plans—you can avoid common mistakes and respond quickly when things go wrong.

Remember the core principles: observe before changing, limit blast radius, use placeholders instead of secrets, verify results with concrete commands, and document recovery steps before you need them. Apply these practices to your init container configurations and you'll build more reliable and maintainable Kubernetes applications.

Next time you modify an init container, start with the Operations Checklist and a backup manifest. The few minutes of preparation will save hours of debugging when an init container misbehaves in production.