Docker Init Containers: Monitoring and Alerting in Practice

Intro

Docker init containers are special containers in a pod that run before the main application containers start. They perform setup tasks such as waiting for dependencies, preparing configuration files, or running database migrations. Because init containers run to completion before the main containers begin, their success or failure directly impacts the availability of your application. Without proper monitoring and alerting, a failing init container can silently prevent your application from starting, leading to extended downtime and difficult debugging.

This guide provides a practical approach to monitoring Docker init containers. We cover what to monitor, useful metrics, alert rules, log signals, dashboards, and incident response workflows. By the end, you will be able to implement a basic monitoring setup for init containers using standard tools, verify its operation, and handle common failure scenarios. The examples use Docker Engine with the built-in healthcheck and log mechanisms, plus Prometheus for metrics and alerting, but the concepts apply to any container orchestration platform.

Version and Environment Inventory

Before configuring monitoring and alerts, document your environment precisely. This inventory ensures that the monitoring configuration matches the runtime and simplifies troubleshooting.

Runtime Versions

Record the versions of the container runtime and monitoring tools. For example:

• Docker Engine: 20.10.12 or later (verify with docker version)
• Docker Compose: 1.29.2 or later (if using Compose for local orchestration)
• Prometheus: 2.32.1 or later
• Alertmanager: 0.23.0 or later
• Grafana: 8.3.4 or later (for dashboards)

Topology

Describe how your init containers are deployed. Common scenarios include:

• A single init container in a Docker Compose service that runs database migrations.
• Multiple init containers that run sequentially in a Kubernetes pod.
• Init containers in a Docker Swarm service that wait for a secret to be available.

For this guide, we assume a simple setup: a web application service with an init container that waits for a PostgreSQL database to accept connections. The application main container starts only after the init container exits successfully.

Prerequisites

You need:

• A running Docker host with access to the Docker CLI.
• A Prometheus server and Alertmanager (can be run as containers).
• Basic familiarity with container logs and Prometheus metrics.
• A target application that uses an init container.

Safe Configuration Path

Monitoring init containers requires two aspects: collecting metrics and logs from the init containers, and defining alert rules based on their lifecycle. We will implement a scoped configuration that monitors a single service's init container, then expand later.

Step 1: Enable Docker Container Logging for Init Containers

Init containers may have a short lifespan, so their logs can be missed. Configure Docker's logging driver to ensure logs are retained. For local testing, the default json-file driver is fine, but you can add options for rotation.

Create a Docker Compose file docker-compose.yml with the following content:

version: "3.8"
services:
  db:
    image: postgres:13
    environment:
      POSTGRES_PASSWORD: secret
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5
  web:
    build: .
    init: true
    depends_on:
      db:
        condition: service_healthy
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
    ports:
      - "8080:80"

The init: true option for the web service tells Docker to use an init process (tini) as PID 1, but it doesn't add an init container. To add a true init container, we'd define a separate service that runs to completion before the web service starts. In Compose, we can simulate an init container using a service with a command that exits and then depends_on with condition: service_completed_successfully (if using Compose v2). Here is an example:

version: "3.8"
services:
  migrate:
    image: migrate/migrate
    command: ["-path", "/migrations", "-database", "postgres://postgres: secret@db:5432/mydb?sslmode=disable", "up"]
    volumes:
      - ./migrations:/migrations
    depends_on:
      db:
        condition: service_healthy
  web:
    build: .
    depends_on:
      migrate:
        condition: service_completed_successfully
    ports:
      - "8080:80"

Here, migrate is the init container. It runs the database migrations and then exits. The web service starts only if the migration succeeds.

Step 2: Collect Docker Daemon Metrics

The Docker daemon exposes metrics at http://localhost:9323/metrics if experimental features are enabled. For init containers, the most useful metrics are container start/stop events and exit codes, which are not directly exposed as Prometheus metrics by the daemon. However, you can install cAdvisor (or use the Docker Prometheus exporter) to collect container-level metrics including CPU, memory, and network. For init containers, we care about whether they succeeded, how long they took, and resource usage.

Run cAdvisor to collect metrics for all containers:

docker run \
  --volume=/:/rootfs: ro \
  --volume=/var/run:/var/run: ro \
  --volume=/sys:/sys: ro \
  --volume=/var/lib/docker/:/var/lib/docker: ro \
  --volume=/dev/disk/:/dev/disk: ro \
  --publish=8081:8080 \
  --detach=true \
  --name=cadvisor \
  gcr.io/cadvisor/cadvisor: v0.39.3

cAdvisor exposes metrics at http://localhost:8081/metrics. We'll use these metrics to build alerts.

Step 3: Configure Prometheus to Scrape Metrics

Create a prometheus.yml configuration file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

Run Prometheus with this config:

docker run -d --name prometheus -p 9090:9090 \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  prom/prometheus: v2.32.1

Step 4: Define Alert Rules for Init Container Failures

Since init containers exit quickly, we cannot rely on a "container down" alert. Instead, we should alert on the exit code of the init container. cAdvisor exposes a metric container_last_seen but not exit codes. We can use a custom script that checks the exit code via the Docker API and exposes it as a Prometheus metric using the textfile collector from Node Exporter, or we can use the Docker event stream to trigger alerts.

A simpler approach is to use the docker events command to watch for die events on init containers and log a message if the exit code is non-zero. This can be integrated with alerting via a custom exporter.

Here, we'll create a small shell script that runs periodically and writes a metric to a file for Node Exporter's textfile collector.

First, run Node Exporter with textfile collector enabled:

docker run -d --name node-exporter -p 9100:9100 \
  -v /path/to/textfile_collector:/var/lib/node_exporter/textfile_collector \
  prom/node-exporter: v1.3.1 --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

Create a script check_init_container.sh:

#!/bin/bash
# Check the exit code of the most recent 'migrate' container
CONTAINER_NAME="migrate"
EXIT_CODE=$(docker inspect --format='{{.State.ExitCode}}' $(docker ps -a -q --filter "name=${CONTAINER_NAME}" --latest) 2>/dev/null || echo 1)
STATUS=$(docker inspect --format='{{.State.Status}}' $(docker ps -a -q --filter "name=${CONTAINER_NAME}" --latest) 2>/dev/null || echo "missing")
if [ "$STATUS" == "exited" ]; then
  if [ "$EXIT_CODE" -eq 0 ]; then
    METRIC="docker_init_container_exit_code{container=\"${CONTAINER_NAME}\",status=\"success\"} 0"
  else
    METRIC="docker_init_container_exit_code{container=\"${CONTAINER_NAME}\",status=\"failure\"} 1"
  fi
else
  METRIC="docker_init_container_exit_code{container=\"${CONTAINER_NAME}\",status=\"unknown\"} -1"
fi
echo "$METRIC" > /path/to/textfile_collector/docker_init_container.prom

Set up a cron job to run this script every minute. Then configure Prometheus to scrape Node Exporter:

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node-exporter:9100']

Now define an alert rule that fires when the exit code metric is 1 (failure) or -1 (unknown). Create alert.rules.yml:

groups:
  - name: docker_init_containers
    rules:
      - alert: InitContainerFailed
        expr: docker_init_container_exit_code{status="failure"} == 1
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Init container {{ $labels.container }} failed with exit code 1"
          description: "The init container {{ $labels.container }} on instance {{ $labels.instance }} has exited with a non-zero code."
      - alert: InitContainerUnknown
        expr: docker_init_container_exit_code{status="unknown"} == -1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Init container {{ $labels.container }} status unknown"
          description: "The init container {{ $labels.container }} has not reported a successful completion."

Update Prometheus config to include the rules file and set up Alertmanager.

Step 5: Configure Alertmanager

Create alertmanager.yml:

global:
  resolve_timeout: 5m
route:
  receiver: 'default'
receivers:
  - name: 'default'
    webhook_configs:
      - url: 'http://example.com/alerts'  # replace with your webhook

Run Alertmanager:

docker run -d --name alertmanager -p 9093:9093 \
  -v $(pwd)/alertmanager.yml:/etc/alertmanager/alertmanager.yml \
  prom/alertmanager: v0.23.0

Add Alertmanager as the alerting endpoint in Prometheus config.

Verification and Diagnostics

After configuring monitoring and alerts, verify that they work correctly.

Check 1: Verify Metrics Collection

Access Prometheus at http://localhost:9090 and query docker_init_container_exit_code. You should see the metric with value 0 if the init container succeeded. If you see no data, check that the Node Exporter textfile collector is reading the file and that the script ran successfully.

Run the script manually and check the output:

bash check_init_container.sh
cat /path/to/textfile_collector/docker_init_container.prom

Expected output:

docker_init_container_exit_code{container="migrate",status="success"} 0

Check 2: Simulate a Failure

To test the alert, force an init container failure. Modify the migration command to an invalid SQL statement or point to a non-existent database. Then run docker-compose up and observe the init container exit code.

Run the check script again. The metric should show status="failure" and value 1. In Prometheus, the alert InitContainerFailed should become pending after 1 minute, then firing. You can see alerts in the Prometheus UI under Alerts, or in Alertmanager.

Check 3: Verify Logs

Check the logs of the init container to confirm the reason for failure:

docker logs <migrate-container-id>

Look for error messages indicating the migration failure. These logs are essential for diagnostics and can be included in alert notifications by using a custom exporter that also captures recent log lines.

Failure Modes and Recovery

Common Failure Modes

1. Init container never exits (hangs): The init container may wait indefinitely for a dependency that never becomes available. This prevents the main container from starting. Monitor the init container's start time and set an alert if it runs longer than expected (e.g., > 5 minutes).
2. Init container exits with non-zero code: This indicates a setup task failed. The main container will not start. Immediate alerting is critical.
3. Init container is not scheduled: In orchestrators like Kubernetes, if the node lacks resources, the init container may remain Pending. Docker Compose does not have this issue, but in Docker Swarm, placement constraints could cause a similar problem.
4. Monitoring metrics missing: If the textfile collector script fails, the metric disappears. Configure an alert on up for the Node Exporter to detect this.

Recovery Steps

1. For a hanging init container, kill it manually and investigate the dependency:

   docker kill <container-id>
   docker logs <container-id>

Fix the underlying issue (e.g., incorrect database host, network problem) and restart.

1. For a non-zero exit, check logs, correct the configuration (e.g., migration script error), and redeploy.
2. If the init container is not scheduled, inspect available resources and adjust constraints.
3. If monitoring fails, manually run the check script and ensure the cron job is active.

Rollback

If a new init container version causes failures, roll back to the previous working image. For example, in Compose, change the image tag to the previous version and run docker-compose up -d. The main application should then start normally after the init container completes.

Operations Checklist

Use this checklist for ongoing operations and reviews:

Conclusion

Monitoring Docker init containers is essential for ensuring application availability. By tracking exit codes, logs, and resource usage, and configuring alerts based on those signals, you can detect failures early and reduce downtime. Start with a narrow pilot, as suggested in the evidence: implement monitoring for one service's init container, verify the metrics and alerts, then expand to other services.

We covered setting up logging, collecting metrics with cAdvisor and a custom script exposed via Node Exporter, defining Prometheus alert rules, and configuring Alertmanager. We also provided verification steps and a recovery guide for common failures. Use the operations checklist to maintain a robust monitoring system. With these practices, you can turn init container monitoring from an afterthought into a reliable part of your operational workflow.