Docker Compose Production Monitoring and Alerts: A Step-by-Step Guide

Intro

When you run production workloads with Docker Compose, monitoring and alerting are critical to keep services healthy, catch issues early, and respond quickly. Unlike managed container platforms, a Compose-based deployment often puts the burden of observability on your team. This guide gives you a practical, step-by-step approach to set up monitoring and alerts for a Docker Compose production stack using widely available open-source tools.

You will learn what to monitor, which metrics matter, how to configure alert rules, how to examine log signals, how to build a basic dashboard, and how to handle incidents. Every step includes concrete commands, configuration snippets, and expected outputs so you can implement and verify each piece.

A structured approach to monitoring and alerts reduces rework by separating research, brief, generation, evaluation, approval, and publishing. This article follows that principle: we start with an inventory, then configure, verify, and plan recovery. A clear monitoring pilot should be narrow, measurable, and easy to inspect locally before deployment; here we begin with a minimal but complete setup.

Version and Environment Inventory

Before adding monitoring, document your environment precisely. This prevents version mismatches and helps when troubleshooting.

Prerequisites

• A Linux server (or VM) running Docker Engine 20.10+ and Docker Compose v2.0+.
• Access to the server via SSH with sudo privileges.
• A running Docker Compose application. For this guide, we assume a simple web application with a database, but the steps generalize.
• Basic familiarity with the command line and YAML.

Gather Versions

Run the following commands to confirm your setup:

docker --version
# Docker version 24.0.7, build afdd53b

docker compose version
# Docker Compose version v2.23.0

Define Your Stack Topology

List your existing Compose services and their configurations. For example:

# docker-compose.yml
version: '3.8'
services:
  web:
    image: nginx:1.25
    ports:
      - "80:80"
    depends_on:
      - app
  app:
    build: ./app
    environment:
      DB_HOST: db
    depends_on:
      - db
  db:
    image: postgres:16
    environment:
      POSTGRES_PASSWORD: secret
    volumes:
      - dbdata:/var/lib/postgresql/data
volumes:
  dbdata:

This example stack has three services: web (nginx reverse proxy), app (custom application), and db (PostgreSQL). Your stack may differ, but identify the components that need monitoring.

Inventory Required Exposed Ports and Dependencies

Document which ports are exposed and which services depend on each other. This helps you know what to monitor and avoid port conflicts when adding monitoring exporters.

Safe Configuration Path

We will set up monitoring without altering your existing services, using a separate monitoring stack. This keeps the production services untouched and reduces risk. We'll use Prometheus for metrics collection, Alertmanager for alert routing, Node Exporter and cAdvisor for host and container metrics, and Grafana for dashboards.

Add Monitoring Services to Your Compose File

Create a separate monitoring-compose.yml to avoid modifying your production file. This file defines the monitoring services and attaches them to the same network as your application.

First, create an external Docker network so both stacks can communicate:

docker network create monitoring

Then, create monitoring-compose.yml:

version: '3.8'
services:
  prometheus:
    image: prom/prometheus:v2.48.0
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
    ports:
      - "9090:9090"
    networks:
      - monitoring

  alertmanager:
    image: prom/alertmanager:v0.26.0
    volumes:
      - ./alertmanager/alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"
    networks:
      - monitoring

  node-exporter:
    image: prom/node-exporter:v1.7.0
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    command:
      - '--path.procfs=/host/proc'
      - '--path.sysfs=/host/sys'
      - '--path.rootfs=/rootfs'
    ports:
      - "9100:9100"
    networks:
      - monitoring

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.47.0
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    ports:
      - "8080:8080"
    networks:
      - monitoring

  grafana:
    image: grafana/grafana:10.2.1
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - grafana_data:/var/lib/grafana
    networks:
      - monitoring

volumes:
  prometheus_data:
  grafana_data:

networks:
  monitoring:
    external: true

This configuration starts Prometheus on port 9090, Alertmanager on 9093, Node Exporter on 9100, cAdvisor on 8080, and Grafana on 3000. They all join the external monitoring network.

Configure Prometheus

Prometheus needs a configuration file to scrape metrics from the exporters and your application services.

Create prometheus/prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

rule_files:
  - "alerts.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

  # Add your application services if they expose /metrics
  # - job_name: 'web'
  #   static_configs:
  #     - targets: ['web:80']

If your application exposes Prometheus metrics, add a scrape config for it. Otherwise, we will rely on cAdvisor for container-level metrics.

Define Alert Rules

Create prometheus/alerts.yml with rules for common failure conditions:

groups:
  - name: docker_compose_alerts
    rules:
      - alert: ContainerDown
        expr: absent(container_last_seen{name=~".+"})
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Container down (instance {{ $labels.instance }})"
          description: "A container has been down for more than 1 minute."

      - alert: HighCPUUsage
        expr: sum(rate(container_cpu_usage_seconds_total{name=~".+"}[5m])) by (name) > 0.8
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage (container {{ $labels.name }})"
          description: "Container CPU usage is above 80% for 5 minutes."

      - alert: HighMemoryUsage
        expr: container_memory_usage_bytes{name=~".+"} / container_spec_memory_limit_bytes{name=~".+"} > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage (container {{ $labels.name }})"
          description: "Container memory usage is above 90% of limit for 5 minutes."

These rules alert if a container disappears, CPU usage exceeds 80%, or memory usage exceeds 90% of its limit.

Configure Alertmanager

Create alertmanager/alertmanager.yml:

route:
  receiver: 'default'
receivers:
  - name: 'default'
    email_configs:
      - to: '[email protected]'
        from: '[email protected]'
        smarthost: 'smtp.example.com:587'
        auth_username: '[email protected]'
        auth_password: 'password'

Replace the email settings with your own. For other notification channels (Slack, PagerDuty, etc.), refer to Alertmanager documentation.

Start the Monitoring Stack

Start the monitoring services:

docker compose -f monitoring-compose.yml up -d

Verify all containers are running:

docker compose -f monitoring-compose.yml ps

Expected output shows all services as "Up".

Verification and Diagnostics

Now we verify that each monitoring component works correctly.

Check Prometheus Targets

Open Prometheus web UI at http://<server-ip>:9090/targets. All targets should show "UP" (green). Alternatively, use the API:

curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health: .health}'

Example output:

{"job":"prometheus","health":"up"}
{"job":"node","health":"up"}
{"job":"cadvisor","health":"up"}

If a target is down, check the error message in the UI and verify network connectivity and service names.

Check Alert Rules

In Prometheus UI, go to "Alerts". You should see the three alerts we defined. They may be in "inactive" state initially. You can test by stopping a container temporarily:

docker stop <container_name>

After 1 minute, the ContainerDown alert should fire (state "pending" then "firing"). Restart the container after testing.

Verify cAdvisor Metrics

cAdvisor exposes container metrics. Query a sample metric:

curl -s 'http://localhost:8080/metrics' | grep 'container_cpu_usage_seconds_total'

You should see metrics with container names.

Verify Node Exporter Metrics

Similarly, check node metrics:

curl -s 'http://localhost:9100/metrics' | grep 'node_cpu_seconds_total'

Test Alertmanager

Trigger a test alert by using the Alertmanager API or by temporarily stopping a service. Check Alertmanager UI at http://<server-ip>:9093 for alerts. If email is configured, you should receive a notification (check your email).

Set Up Grafana Dashboard

Access Grafana at http://<server-ip>:3000 (default login admin/admin).

• Add Prometheus as a data source:
• Go to Configuration > Data Sources > Add data source > Prometheus.
• Set URL to http://prometheus:9090 (since Grafana is on the same Docker network).
• Click "Save & Test".

• Import a Docker monitoring dashboard:
• Go to Dashboards > Import.
• Enter dashboard ID 193 for a popular Docker monitoring dashboard, or use ID 14282 for a more comprehensive one.
• Select the Prometheus data source and import.

• Verify that panels show data (CPU, memory, network, etc.). If some panels are empty, ensure your Prometheus queries match the metrics available (cAdvisor version differences may require adjustments).

Inspect Logs for Signals

Container logs are essential for debugging. Use Docker's logging drivers and a log aggregator if needed. For quick checks:

docker logs <container_name> --tail 50

For production, consider using the json-file logging driver and shipping logs to a central system like Loki or ELK. Docker Compose can configure logging per service:

services:
  web:
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"

Then, use a log processing tool to detect patterns and alert on them.

Failure Modes and Recovery

Monitoring itself can fail or produce false alarms. Prepare for common issues.

Prometheus Data Loss

If Prometheus data is lost (e.g., volume deleted), you lose historical metrics but monitoring can restart. To mitigate, regularly back up the Prometheus data volume or use remote storage. For simplicity, ensure the prometheus_data volume is included in your backup strategy.

Alert Storms

Too many alerts can overwhelm. Use alert grouping and inhibition rules in Alertmanager. For example, group alerts by service and environment to reduce noise. Also, set appropriate for durations to avoid flapping.

False Positives

If an alert fires but the service is healthy, adjust thresholds. For example, memory usage may spike during normal operation; increase the threshold or the duration. Use Grafana to observe trends and refine.

Monitoring Stack Downtime

If Prometheus or Alertmanager goes down, you may miss alerts. Consider running them in a highly available setup or have an external uptime monitoring service (like UptimeRobot) to ping your monitoring endpoints.

Rollback Plan

If the monitoring configuration causes issues, you can stop the monitoring stack without affecting your application:

docker compose -f monitoring-compose.yml down

This removes the monitoring containers but leaves volumes intact. To remove volumes as well (data loss), add -v flag.

Recovery Checks

After any change or recovery, run through the verification steps again to ensure all targets are up and alerts are functioning.

Common Pitfalls and How to Avoid Them

Setting up monitoring for Docker Compose has several recurring mistakes. Here are the most frequent ones and how to prevent or recover from them.

Hardcoding Container Names in Alert Rules

Why it happens: Developers often copy alert expressions that reference specific container names, such as container_memory_usage_bytes{name="web"}. When containers are recreated or scaled, the names change, and the alerts silently break.

How to avoid: Use label matchers that are stable, such as image, com.docker.compose.service, or com.docker.compose.project. For example, instead of name="web", use com.docker.compose.service="web". If you must use names, use a regex like name=~"web.*" to catch suffixes.

Recovery: If alerts stop firing, review the container_last_seen metric in Prometheus to see the current labels. Update your alert expressions accordingly.

Using the Same Network as Production Without Isolation

Why it happens: Teams might add monitoring services directly to the production Compose file to keep things simple. This couples the monitoring stack to production and can cause port conflicts or accidental config changes.

How to avoid: Keep monitoring in a separate Compose file and use an external network. This way, you can restart or update monitoring without touching production. As shown earlier, create the network with docker network create monitoring and attach both stacks to it.

Recovery: If you have already mixed them, extract the monitoring services into their own file, create the network, and redeploy. Test in a staging environment first.

Ignoring Storage Growth

Why it happens: Prometheus stores time-series data, which can grow indefinitely. Without retention limits, the disk fills up, causing Prometheus to crash or stop ingesting data.

How to avoid: Set a retention period or size limit in the Prometheus command. For example, add --storage.tsdb.retention.time=15d to keep 15 days of data. Monitor disk usage with a separate alert.

Recovery: If disk is full, stop Prometheus, remove old data blocks manually (or delete the volume), and restart with proper retention settings. Ensure you have a backup if historical data is needed.

Not Testing Alerts End-to-End

Why it happens: Teams configure alerts and assume they work, but notifications might be misrouted or blocked by email filters.

How to avoid: Regularly trigger a test alert (e.g., stop a test container) and verify that the notification reaches the intended channel. Include this in your operations checklist.

Recovery: If an alert didn't fire, check the Alertmanager logs and the notification provider's delivery status. Adjust the routing or credentials.

Over-Alerting and Alert Fatigue

Why it happens: Alert rules with low thresholds or short for durations generate too many alerts, causing teams to ignore them.

How to avoid: Start with conservative thresholds and longer durations. Use Grafana to observe normal behavior for a week, then adjust. Group alerts and use severity levels to prioritize.

Recovery: If you already have alert fatigue, review recent alerts in Alertmanager, identify the noisiest ones, and tune them. Delete or pause alerts that are not actionable.

Operations Checklist

Use this checklist for ongoing operations. Assign a single owner for each area and set a review cadence.

Perform these checks regularly and after any changes. The owner is responsible for the check, and the review frequency indicates how often the result should be revisited.

Conclusion

You now have a complete monitoring and alerting setup for your Docker Compose production deployment. You started with an environment inventory, added a separate monitoring stack with Prometheus, Alertmanager, Node Exporter, cAdvisor, and Grafana, defined alert rules, verified each component, and prepared recovery procedures.

Next steps:

• Customize alert rules for your application-specific metrics.
• Add more dashboards for your services.
• Set up a log aggregation system and integrate with Alertmanager.
• Test your incident response process with a fire drill.
• Regularly review and update thresholds based on observed patterns.

A clear monitoring pilot should be narrow, measurable, and easy to inspect locally before deployment. Start with the basics, get them working reliably, then expand. Monitoring is an ongoing discipline, not a one-time setup.