Docker Compose Monitoring and Alerts: A Practical Implementation Guide

Docker Compose simplifies multi-container application deployment, but its built-in observability is limited to basic container state and logs. Production workloads require structured metrics, actionable alerts, and dashboards that surface issues before users notice them. This guide walks through a complete monitoring stack for Docker Compose environments using Prometheus, Grafana, Alertmanager, and cAdvisor, with concrete configuration examples, alerting rules, and operational procedures you can adapt to your stack.

Architecture and Component Selection

A practical monitoring stack for Docker Compose consists of four core services:

• Prometheus scrapes metrics from exporters and application endpoints, stores time-series data, and evaluates alerting rules.
• Alertmanager deduplicates, groups, and routes alerts to notification channels such as Slack, PagerDuty, or email.
• Grafana visualizes metrics through dashboards and provides ad-hoc query capabilities.
• cAdvisor exposes container-level resource usage (CPU, memory, disk, network) for each container on the host.

Optional but recommended additions include Node Exporter for host-level metrics (disk, network, system load) and application-specific exporters (e.g., nginx-prometheus-exporter, redis_exporter, postgres_exporter) for service-level golden signals.

All components run as containers defined in a single docker-compose.monitoring.yml file, deployed alongside your application stack or on a dedicated observability host. This keeps the monitoring plane independent of application deployments while sharing the same Docker network for service discovery.

Example Compose File Structure

version: "3.8"

services:
  prometheus:
    image: prom/prometheus:v2.53.0
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./prometheus/rules:/etc/prometheus/rules:ro
      - prometheus_data:/prometheus
    ports:
      - "9090:9090"
    networks:
      - monitoring
    restart: unless-stopped

  alertmanager:
    image: prom/alertmanager:v0.27.0
    command:
      - "--config.file=/etc/alertmanager/alertmanager.yml"
      - "--storage.path=/alertmanager"
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - alertmanager_data:/alertmanager
    ports:
      - "9093:9093"
    networks:
      - monitoring
    restart: unless-stopped

  grafana:
    image: grafana/grafana:10.4.0
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password
      - GF_INSTALL_PLUGINS=grafana-piechart-panel
    volumes:
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
      - grafana_data:/var/lib/grafana
    secrets:
      - grafana_admin_password
    ports:
      - "3000:3000"
    networks:
      - monitoring
    restart: unless-stopped

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    command:
      - "--docker_only=true"
      - "--housekeeping_interval=10s"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - "8080:8080"
    networks:
      - monitoring
    restart: unless-stopped

  node-exporter:
    image: prom/node-exporter:v1.7.0
    command:
      - "--path.rootfs=/host"
    pid: host
    volumes:
      - /:/host:ro
      - /sys:/sys:ro
      - /proc:/proc:ro
    ports:
      - "9100:9100"
    networks:
      - monitoring
    restart: unless-stopped

networks:
  monitoring:
    driver: bridge

volumes:
  prometheus_data:
  alertmanager_data:
  grafana_data:

secrets:
  grafana_admin_password:
    file: ./secrets/grafana_admin_password.txt

Key operational notes:

• Pin image tags to specific versions (not latest) to avoid surprise upgrades.
• Store secrets (Grafana admin password, Alertmanager webhook URLs) in Docker secrets or a .env file excluded from version control.
• Mount configuration as read-only (:ro) where possible to prevent accidental modification.
• Use named volumes for persistent data (Prometheus TSDB, Alertmanager state, Grafana dashboards) so container recreation preserves history.

Prometheus Configuration and Service Discovery

Prometheus discovers scrape targets through static configuration or Docker service discovery. For Docker Compose, the dockersd mechanism automatically detects containers with specific labels, eliminating manual target management.

Prometheus Configuration (prometheus/prometheus.yml)

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    environment: "production"
    compose_project: "myapp"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

rule_files:
  - "rules/*.yml"

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "cadvisor"
    static_configs:
      - targets: ["cadvisor:8080"]

  - job_name: "node-exporter"
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: "docker-compose-services"
    dockerswarm_sd_configs:
      - host: unix:///var/run/docker.sock
        role: services
    relabel_configs:
      - source_labels: [__meta_docker_service_label_prometheus_job]
        action: keep
        regex: .+
      - source_labels: [__meta_docker_service_label_prometheus_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: ${1}:${2}
      - source_labels: [__meta_docker_service_label_prometheus_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
        replacement: ${1}

Enabling Application Metrics via Labels

Add Prometheus scrape labels to your application services in the main docker-compose.yml:

services:
  api:
    image: myorg/api:v1.12.0
    ports:
      - "8000:8000"
    labels:
      - "prometheus.job=api"
      - "prometheus.port=8000"
      - "prometheus.path=/metrics"
    networks:
      - app-network
      - monitoring
    deploy:
      replicas: 3

Prometheus will automatically discover the api service, scrape /metrics on port 8000, and tag metrics with job="api". This pattern scales across dozens of services without touching prometheus.yml.

Alerting Rules That Reduce Noise

Effective alerts fire on symptoms (user-facing impact) rather than causes (internal state). Define rules in prometheus/rules/ grouped by domain.

Infrastructure Alerts (prometheus/rules/infrastructure.yml)

groups:
  - name: infrastructure
    interval: 30s
    rules:
      - alert: ContainerDown
        expr: |
          absent(container_last_seen{job="cadvisor"}) or
          (time() - container_last_seen{job="cadvisor"}) > 60
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Container {{ $labels.name }} down for > 1m"
          description: "Container {{ $labels.name }} (image: {{ $labels.image }}) has not reported metrics for over 60 seconds."
          runbook_url: "https://wiki.example.com/runbooks/container-down"

      - alert: ContainerHighCPU
        expr: |
          (rate(container_cpu_usage_seconds_total{job="cadvisor",container!=""}[5m]) /
          container_spec_cpu_quota{job="cadvisor",container!=""} / 100000) > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} CPU > 85% for 5m"
          description: "Container {{ $labels.name }} sustained CPU usage above 85% of quota. Check for runaway processes or undersized limits."

      - alert: ContainerHighMemory
        expr: |
          (container_memory_usage_bytes{job="cadvisor",container!=""} /
          container_spec_memory_limit_bytes{job="cadvisor",container!=""}) > 0.85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} memory > 85% for 5m"
          description: "Container {{ $labels.name }} memory usage exceeds 85% of limit. Risk of OOM kill."

      - alert: HostDiskSpaceCritical
        expr: |
          (node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"} /
          node_filesystem_size_bytes{mountpoint="/",fstype!="tmpfs"}) < 0.1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Host disk space < 10% on {{ $labels.instance }}"
          description: "Root filesystem on {{ $labels.instance }} has less than 10% free. Clean up logs, images, or expand volume."

Application Golden Signals (prometheus/rules/application.yml)

groups:
  - name: application
    interval: 30s
    rules:
      - alert: APIHighErrorRate
        expr: |
          sum(rate(http_requests_total{job="api",code=~"5.."}[2m])) by (job)
          /
          sum(rate(http_requests_total{job="api"}[2m])) by (job)
          > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "API 5xx error rate > 5% for 2m"
          description: "Service {{ $labels.job }} returning 5xx errors on > 5% of requests. Check upstream dependencies and logs."

      - alert: APIHighLatency
        expr: |
          histogram_quantile(0.95,
            sum(rate(http_request_duration_seconds_bucket{job="api"}[5m])) by (le, job)
          ) > 1.0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "API p95 latency > 1s for 5m"
          description: "Service {{ $labels.job }} p95 request latency exceeds 1 second. Investigate database, cache, or external calls."

      - alert: APITrafficDrop
        expr: |
          sum(rate(http_requests_total{job="api"}[5m])) by (job) < 0.1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "API traffic dropped near zero for 10m"
          description: "Service {{ $labels.job }} receiving < 0.1 req/s. Possible routing issue, deployment failure, or upstream outage."

Alertmanager Routing (alertmanager/alertmanager.yml)

global:
  resolve_timeout: 5m
  slack_api_url: "https://hooks.slack.com/services/XXX/YYY/ZZZ"

route:
  group_by: ["alertname", "job", "severity"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: "slack-critical"
  routes:
    - match:
        severity: "critical"
      receiver: "slack-critical"
      continue: true
    - match:
        severity: "warning"
      receiver: "slack-warning"
      group_interval: 15m
      repeat_interval: 12h

receivers:
  - name: "slack-critical"
    slack_configs:
      - channel: "#alerts-critical"
        title: "[{{ .Status }}] {{ .GroupLabels.alertname }}"
        text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"
        send_resolved: true
        icon_emoji: ":fire:"
  - name: "slack-warning"
    slack_configs:
      - channel: "#alerts-warning"
        title: "[{{ .Status }}] {{ .GroupLabels.alertname }}"
        text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"
        send_resolved: true
        icon_emoji: ":warning:"

inhibit_rules:
  - source_match:
      severity: "critical"
    target_match:
      severity: "warning"
    equal: ["job", "instance"]

This configuration routes critical alerts to a high-visibility channel with short repeat intervals, while warnings go to a lower-noise channel with longer grouping. Inhibition prevents warning storms when a critical alert already covers the same service.

Grafana Dashboards and Provisioning

Provision dashboards as code so they version-control alongside infrastructure. Grafana reads dashboard JSON from a mounted directory at startup.

Provisioning Config (grafana/provisioning/dashboards/dashboard-provider.yml)

apiVersion: 1

providers:
  - name: "Docker Compose Monitoring"
    orgId: 1
    folder: "Infrastructure"
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards

Key Dashboard Panels

Create or import dashboards covering:

1. Container Overview: Table of all containers with status, CPU %, memory %, restart count, and uptime. Use container_last_seen, container_cpu_usage_seconds_total, container_memory_usage_bytes, and container_start_time_seconds from cAdvisor.
2. Resource Saturation: Heatmap of CPU and memory utilization across services over time. Spot trends before limits are hit.
3. Request Rate, Errors, Duration (RED): For each instrumented service, show requests/sec, error rate (5xx / total), and p50/p95/p99 latency from application /metrics endpoints.
4. Host Resources: Node Exporter panels for disk I/O, network throughput, load average, and filesystem usage.
5. Alert Status: Table of currently firing alerts from Alertmanager API (/api/v2/alerts) with silence buttons.

Store dashboard JSON files in grafana/dashboards/ and commit them. On Grafana restart, dashboards appear automatically without manual import.

Operational Procedures

Deploying the Stack

# Create secrets directory and generate Grafana password
mkdir -p secrets
openssl rand -base64 32 > secrets/grafana_admin_password.txt
chmod 600 secrets/grafana_admin_password.txt

# Deploy monitoring stack
docker compose -f docker-compose.monitoring.yml up -d

# Verify all services healthy
docker compose -f docker-compose.monitoring.yml ps

Verifying Metrics Collection

# Check Prometheus targets
curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, instance: .labels.instance, health: .health}'

# Query a sample metric
curl -s "http://localhost:9090/api/v1/query?query=container_cpu_usage_seconds_total" | jq '.data.result[0]'

# Check Alertmanager status
curl -s http://localhost:9093/api/v2/status | jq '.'

Adding a New Service to Monitoring

• Add Prometheus labels to the service in your application docker-compose.yml.
• Ensure the service joins the monitoring network.
• Redeploy the application stack: docker compose up -d.
• Verify the target appears in Prometheus UI (Status → Targets) within 30 seconds.
• Create or update alerting rules if the service exposes new SLIs.
• Add panels to the relevant Grafana dashboard.

Rotating Secrets

# Generate new Grafana password
openssl rand -base64 32 > secrets/grafana_admin_password.txt.new
# Update secret (requires stack restart for Grafana to pick up)
mv secrets/grafana_admin_password.txt.new secrets/grafana_admin_password.txt
docker compose -f docker-compose.monitoring.yml up -d --force-recreate grafana

Backup and Disaster Recovery

• Prometheus data: Snapshot via API (POST /api/v1/admin/tsdb/snapshot) or stop container and tar prometheus_data volume.
• Grafana dashboards: Already in Git via provisioning. Export dashboard JSON from UI as backup.
• Alertmanager config: In Git. Silence state stored in alertmanager_data volume; recreate from config on restore.

Test restore quarterly: spin up a fresh host, mount volume backups, start stack, verify dashboards show history and alerts evaluate correctly.

Failure Modes and Mitigations

Conclusion

Monitoring Docker Compose workloads requires more than docker stats and log tailing. A stack built on Prometheus, Alertmanager, Grafana, and cAdvisor provides service-level visibility, actionable alerts, and historical analysis without agent installation or host modification. The compose-based deployment keeps operations simple: version-controlled configuration, secret management via Docker secrets, and independent scaling of the observability plane. Start with infrastructure alerts (container health, resource saturation), add application golden signals as services expose /metrics endpoints, and iterate dashboards based on incident retrospectives. Treat monitoring code like application code—review, test, version, and deploy through the same pipeline.