Docker monitoring and alerts with practical examples: practical implementation guide

Intro

Good Docker monitoring answers three fast questions during incidents: what changed, what is breaking, and where to fix it. For most teams that means watching container CPU and memory, restarts and OOM kills, image drift, and network or disk pressure. This guide gives you a practical, narrow pilot you can run on a single host to prove value quickly, then extend confidently. It includes metrics to collect, concrete alert rules, log signals, dashboards, verification steps, common failure modes, and rollback instructions.

The approach is deliberately simple: enable the Docker Engine metrics endpoint, add cAdvisor for per-container stats, scrape with Prometheus, route alerts via Alertmanager, and apply log rotation. All steps are safe to test locally before any multi-host deployment.

Version and Environment Inventory

Before changing anything, inventory what you have so you can measure improvement and roll back safely.

• Record versions and topology
• Run: docker version and keep both client and server versions.
• Run: docker info and note Storage Driver, Logging Driver, Cgroup Driver, and number of running containers.
• OS: uname -a and /etc/os-release.
• Hostname and IPs: hostname -f and ip -br a.
• Networks: docker network ls and docker network inspect <network> for production networks.
• Baseline configuration
• Back up: sudo cp -a /etc/docker/daemon.json /etc/docker/daemon.json.bak.$(date +%s) 2>/dev/null || true
• Validate JSON format: sudo sh -c 'test -f /etc/docker/daemon.json && jq . /etc/docker/daemon.json >/dev/null' || echo ok
• Note current log options: docker info | grep -i 'Logging Driver' -A2
• Access and safety
• Ensure you have sudo rights to restart Docker: sudo systemctl status docker.
• Maintenance window: restarting the Docker daemon is brief but can interrupt control-plane operations (start/stop, pulls). Running containers normally continue; schedule during a low-traffic window.

Safe Configuration Path

The aim is a scoped pilot on one host: bind metrics to localhost, confirm collection, then add a few focused alerts. Keep credentials and network exposure minimal.

Pick a narrow pilot scope

Use the following pilot scope as a default and expand later.

Configure Docker Engine metrics and log rotation

1. Create or edit /etc/docker/daemon.json with conservative defaults. Example (constructed example):

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "experimental": true,
  "metrics-addr": "127.0.0.1:9323"
}

Notes:

• experimental must be true to expose the metrics endpoint.
• Binding to 127.0.0.1 avoids network exposure; scrape locally.
• If you already use a non-default log driver, preserve it and just add metrics settings.

1. Validate JSON and restart Docker safely:

• sudo jq . /etc/docker/daemon.json >/dev/null
• sudo systemctl restart docker
• Verify: curl -s http://127.0.0.1:9323/metrics | head -n 5 should print Prometheus-format lines.

Add cAdvisor for per-container metrics

Run cAdvisor with read-only host mounts. The following keeps it local and avoids exposing its UI externally.

sudo docker run -d \
  --name=cadvisor \
  --restart unless-stopped \
  -p 127.0.0.1:8080:8080 \
  -v /:/rootfs: ro \
  -v /var/run:/var/run: ro \
  -v /sys:/sys: ro \
  -v /var/lib/docker/:/var/lib/docker: ro \
  gcr.io/cadvisor/cadvisor: latest

Verify: curl -s http://127.0.0.1:8080/metrics | head -n 5

Run a local Prometheus and Alertmanager pilot

Use local configuration files so everything is auditable and easy to revert.

1. Create ./prometheus/ with:

• prometheus.yml (constructed example):

global:
  scrape_interval: 15s
  evaluation_interval: 15s
rule_files:
  - /etc/prometheus/alerts.yml
scrape_configs:
  - job_name: 'docker-engine'
    static_configs:
      - targets: ['127.0.0.1:9323']
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['127.0.0.1:8080']

• alerts.yml with focused alerts (constructed examples):

# CPU near a full core for 5m
- alert: ContainerHighCPU
  expr: rate(container_cpu_usage_seconds_total{container!=""}[5m]) > 0.9
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.container }} high CPU"
    description: "Container CPU > 90% of one core for 5m"

# Memory > 90% of limit (when a limit is set)
- alert: ContainerMemoryNearLimit
  expr: (container_memory_working_set_bytes{container!=""} / container_spec_memory_limit_bytes{container!=""} > 0.9) and on(container) (container_spec_memory_limit_bytes{container!=""} > 0)
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.container }} memory near limit"
    description: "Working set > 90% of limit for 5m"

# Restart loop detection via start time changes
- alert: ContainerCrashLoop
  expr: changes(container_start_time_seconds{container!=""}[5m]) > 2
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.container }} crash looping"
    description: "Container restarted multiple times in 5m"

# OOM kill signal (if exposed by cAdvisor)
- alert: ContainerOOMKilled
  expr: increase(container_oom_events_total{container!=""}[5m]) > 0
  for: 0m
  labels:
    severity: critical
  annotations:
    summary: "{{ $labels.container }} OOM killed"
    description: "One or more OOM events in the last 5m"

1. Start Prometheus, binding to localhost:

sudo docker run -d \
  --name=prometheus \
  --restart unless-stopped \
  -p 127.0.0.1:9090:9090 \
  -v $(pwd)/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml: ro \
  -v $(pwd)/prometheus/alerts.yml:/etc/prometheus/alerts.yml: ro \
  prom/prometheus: latest

1. Start a minimal Alertmanager and route alerts locally. Create ./alertmanager/alertmanager.yml (constructed example):

route:
  receiver: 'stdout'
receivers:
  - name: 'stdout'
    webhook_configs:
      - url: 'http://127.0.0.1:5001/'
        send_resolved: true

Start Alertmanager:

sudo docker run -d \
  --name=alertmanager \
  --restart unless-stopped \
  -p 127.0.0.1:9093:9093 \
  -v $(pwd)/alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml: ro \
  prom/alertmanager: latest

For a quick local receiver, you can run a simple HTTP listener in another terminal to observe alert payloads (constructed example):

python3 -m http.server 5001

Alternatively, configure your team chat or email webhook after you verify alerts locally.

Core metrics and signals to watch

Use the following as your initial dashboard and alert palette.

| Engine health | Confirms Docker control-plane status | curl -s 127.0.0.1:9323/metrics | head |

Useful Docker log signals

• Container exits and restarts: docker ps --format '{{.Names}} {{.Status}}' and docker inspect -f '{{.Name}} RestartCount={{.RestartCount}}' <container>
• OOMKilled flag: docker inspect -f '{{.Name}} OOMKilled={{.State.OOMKilled}}' <container>
• Engine-level events: docker events --since 10m --filter type=container --filter event=oom --filter event=die
• Application errors: docker logs --since 10m <container>; look for stack traces and rate of errors per minute.

Verification and Diagnostics

Prove each piece works before moving on. The following steps keep verification targeted and reversible.

1) Check metrics endpoints

• Engine metrics: curl -s 127.0.0.1:9323/metrics | grep -m1 ^# should show a HELP or TYPE line.
• cAdvisor metrics: curl -s 127.0.0.1:8080/metrics | grep -m1 ^# should show Prometheus headers.
• Prometheus targets: open http://127.0.0.1:9090/targets in a browser or via port-forwarding; both jobs should be UP.

2) Fire a CPU alert (constructed example)

Create a CPU-hogging test container:

docker run -d --name cpuhog --rm alpine:3 sh -c 'while :; do :; done'

• Query in Prometheus: rate(container_cpu_usage_seconds_total{container="cpuhog"}[5m]) should trend near 1.0 on a single core.
• Within 5-10 minutes, the ContainerHighCPU alert should trigger.
• Stop the test: docker rm -f cpuhog

3) Trigger a crash loop alert (constructed example)

Start a container that exits immediately and restarts:

docker run -d --name crashloop --restart always alpine:3 sh -c 'echo boom; sleep 1; exit 1'

• Verify restarts: docker ps --format '{{.Names}} {{.Status}}' | grep crashloop
• Query: changes(container_start_time_seconds{container="crashloop"}[5m])
• The ContainerCrashLoop alert should fire. Clean up: docker rm -f crashloop

4) Optional: Simulate memory pressure (constructed example)

If you have a test image that can allocate memory (for example, a small utility image in your org), run it with a low memory limit and try to exceed it. Confirm whether container_oom_events_total increases and alerts fire. If you cannot safely simulate OOM, validate by inspecting an application container after a known OOM incident:

docker inspect -f '{{.Name}} OOMKilled={{.State.OOMKilled}}' <container>

5) Validate alert routing

• Open http://127.0.0.1:9093/#/alerts to see active alerts.
• Confirm your local webhook receiver shows POSTs when alerts fire.
• If integrating with chat/email, temporarily lower thresholds and confirm a single notification arrives.

Failure Modes and Recovery

Common issues and quick fixes

• Metrics endpoint not reachable
• Symptom: curl 127.0.0.1:9323/metrics fails.
• Cause: experimental not enabled, port in use, or syntax error in daemon.json.
• Fix: validate JSON with jq ., set experimental: true, confirm metrics-addr, restart Docker.

• cAdvisor permission or mount problems
• Symptom: cAdvisor target DOWN or missing container metrics.
• Cause: missing mounts or read errors on /var/lib/docker or /sys.
• Fix: ensure mounts are present and read-only, restart cAdvisor.

• Chatty or duplicate alerts
• Symptom: flood of notifications.
• Cause: short for: durations, noisy queries, missing group_by in receivers.
• Fix: raise thresholds, add for: windows, de-duplicate at Alertmanager, and use labels to group.

• Containers do not respect memory alerts
• Symptom: Memory alert fires but no limit is set; alert feels unactionable.
• Cause: Containers without memory limits report unlimited spec; ratio-based alerts are not meaningful.
• Fix: Use absolute working set thresholds per host or enforce limits with docker update --memory where appropriate.

• Docker restart side effects
• Symptom: Brief control-plane disruptions.
• Cause: daemon restart.
• Fix: schedule during a window; verify after restart that docker ps and docker pull work.

Rollback and recovery procedures

• Revert Docker daemon configuration
• sudo cp -a /etc/docker/daemon.json.bak.* /etc/docker/daemon.json (choose the latest good backup)
• sudo systemctl restart docker
• Verify: docker ps and curl 127.0.0.1:9323/metrics (expect failure if metrics removed; this is normal after rollback)

• Disable monitoring components quickly
• Stop cAdvisor: docker rm -f cadvisor
• Stop Prometheus: docker rm -f prometheus
• Stop Alertmanager: docker rm -f alertmanager

• Silence noisy alerts while investigating
• In Alertmanager, create a temporary silence matching severity=warning or the specific alert name.
• Increase for: duration in alerts.yml from 5m to 15m for non-critical alerts, reload Prometheus by restarting the container.

• Recover disk space from runaway logs
• Confirm log driver and rotation: docker info | grep -i 'Logging Driver' -A2
• If using json-file, ensure max-size and max-file are set; update daemon.json and restart Docker.

Operations Checklist

Use this checklist weekly and during incidents.

• Baseline and capacity
• Review top 5 containers by CPU and memory in Prometheus for the last 7 days.
• Confirm no container runs without an intended memory limit if you rely on ratio-based memory alerts.
• Check disk space on /var/lib/docker: df -h /var/lib/docker.

• Signals and alerts
• Ensure Prometheus targets are UP.
• Review alert history and confirm no persistent noise; tune thresholds or for: windows if needed.
• Confirm at least one test alert reaches your chosen channel.

• Logs and events
• Scan for frequent container restarts: docker ps --format '{{.Names}} {{.Status}}' | grep -E 'Up|Restarting'.
• Sample engine events: docker events --since 24h --filter type=container --filter event=oom --filter event=die | head -n 20.

• Incident response workflow (follow in order)
• Identify the loudest failing container by alert context.
• Inspect state quickly: docker inspect -f '{{.Name}} State={{.State.Status}} OOM={{.State.OOMKilled}} Restarts={{.RestartCount}}' <container>.
• Get logs around failure time: docker logs --since 10m <container>.
• If CPU runaway, consider a temporary cap: docker update --cpus 1.0 <container> (constructed example; verify impact first).
• If crash looping, disable --restart or scale to zero while triaging: docker update --restart=no <container> or recreate without restart.
• Document root cause and prevention (limit, code fix, startup probe changes) before re-enabling auto-restart.

Practical dashboards (queries you can paste)

• CPU: topk(10, rate(container_cpu_usage_seconds_total{container!=""}[5m]))
• Memory working set: topk(10, container_memory_working_set_bytes{container!=""})
• Restarts: sum by (container) (changes(container_start_time_seconds{container!=""}[1h]))
• OOM events: increase(container_oom_events_total{container!=""}[1h])

What to expand after the pilot is quiet

• Add node-level collectors (filesystem, network) and alerts for disk pressure and inode exhaustion.
• Introduce dashboards for image and tag drift to catch unexpected upgrades.
• Secure metrics with TLS or isolate scrapers on a monitoring network when moving beyond localhost.
• Integrate Alertmanager with chat, ticketing, and on-call systems, with routing by service or team.

Conclusion

You now have a working, verifiable baseline for Docker monitoring and alerts: engine and container metrics collected locally, actionable alert rules for CPU, memory, restarts, and OOM kills, and a checklist that keeps operations repeatable. Because the pilot is narrow and locally inspectable, it is easy to tune thresholds, validate routing, and roll back if needed. Once your alerts are quiet, specific, and trusted, expand coverage to node resources, team-specific dashboards, and production-grade alert routing.