Docker Compose Watch Monitoring and Alerts: A Practical Implementation Guide

Intro

Docker Compose Watch gives developers a fast feedback loop by rebuilding and restarting services when source files change. But in any non-trivial project, you also need to know when things go wrong: a service crash, a failed build, a misconfigured mount, or a resource limit. Monitoring and alerting turn Compose Watch from a convenience into a reliable part of your operations workflow.

This guide walks through a practical implementation for Docker Compose Watch monitoring, alerts, dashboards, and incident response. It assumes you are comfortable with Docker and Compose basics, and it focuses on the operational layer: what to watch, how to detect problems, and how to respond safely. Every section includes exact commands, configuration snippets, expected outputs, and recovery steps.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths. Do not treat this as a copy-paste checklist; adapt the examples to your environment and version constraints.

Version and Environment Inventory

Before you monitor anything, know what you are running. A monitoring rule that assumes the wrong version or topology will either miss failures or flood you with false alarms.

Start by capturing your current Docker and Compose versions. Run:

docker version --format '{{.Server.Version}}'
docker compose version

Expected output example:

24.0.7
Docker Compose version v2.21.0

Here 24.0.7 is the Docker engine version and v2.21.0 is the Compose plugin version. Compose Watch requires Docker Compose v2.17.0 or later, and the watch attribute is only fully supported with BuildKit and the Compose v2 file format (typically compose.yaml). If you see an older version, upgrade before enabling watch.

Next, list the services in your Compose project and their state:

docker compose ps

Example output:

NAME                IMAGE               COMMAND                  SERVICE             CREATED             STATUS              PORTS
project-web-1       project-web:latest  "gunicorn app:app"       web                 2 minutes ago       Up 2 minutes        0.0.0.0:8000->8000/tcp
project-db-1        postgres:15         "docker-entrypoint.s…"  db                  2 minutes ago       Up 2 minutes        5432/tcp

Note the service names (web, db), container names, and health status. A missing Up under STATUS is the first failure signal.

For read-only inspection of a suspicious container, use:

docker inspect project-web-1 --format '{{json .State.Health}}'

Expected output (if a healthcheck is defined):

{"Status":"healthy","FailingStreak":0,"Log":[{"Start":"2024-01-01T00:00:00Z","End":"2024-01-01T00:00:05Z","ExitCode":0,"Output":"ok"}]}

If no healthcheck is defined, the field is null. You can add one later; we cover that in the Verification section.

Lastly, confirm your volume mounts are correct before any change. Data loss is a common Compose failure. Use:

docker inspect project-db-1 --format '{{json .Mounts}}'

Example output:

[{"Type":"volume","Name":"project_pgdata","Source":"/var/lib/docker/volumes/project_pgdata/_data","Destination":"/var/lib/postgresql/data","Driver":"local","Mode":"z","RW":true}]

This shows a named volume bound to /var/lib/postgresql/data. If you expected a bind mount to ./data, this is a mismatch. Correcting it before a restart prevents hours of debugging later.

Safe Configuration Path

The watch section in your Compose file controls how changes are synced. A misconfigured watch rule can cause infinite rebuild loops, container churn, or missed syncs. Here is a safe starting configuration:

services:
  web:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - .:/app
    watch:
      - action: sync
        path: ./app
        target: /app
        ignore:
          - node_modules/
          - .venv/
      - action: rebuild
        path: requirements.txt
    environment:
      FLASK_ENV: development

Key points:

• Use action: sync for directories where only file content changes, not dependencies.
• Use action: rebuild for manifest files like requirements.txt or Dockerfile changes.
• Always include an ignore list for large generated directories (node_modules, .venv, __pycache__) to reduce sync load.

To verify your watch configuration is valid, run:

docker compose config --quiet

If the configuration is invalid, this command exits with a non-zero status and prints errors to stderr. Example of a syntax error:

yaml: line 10: did not find expected key

Fix the YAML indentation before proceeding.

Now test the sync path manually. Start the stack:

docker compose up --build -d

Watch the logs for the watcher startup message:

docker compose logs -f web

Example log line:

web-1  | [watch] watching /app for changes

Trigger a sync by editing a file in ./app, for example:

echo "print('hello')" >> app/main.py

You should see the watcher log the change and sync the file without restarting the container. If instead the container restarts or rebuilds, you have the wrong action type.

Verification and Diagnostics

After configuring watch, verify that your application behaves correctly under sync and rebuild actions. The first step is to define a healthcheck inside your service. For a Python web app, add to compose.yaml:

services:
  web:
    # ...
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s

This healthcheck runs curl -f against a /health endpoint every 10 seconds, considers the container unhealthy after 3 consecutive failures, and waits 30 seconds before starting checks during startup.

Apply the change and recreate the service:

docker compose up -d web

Check health status:

docker inspect --format='{{.State.Health.Status}}' project-web-1

Expected output:

healthy

If it says unhealthy, inspect the healthcheck logs:

docker inspect --format='{{json .State.Health.Log}}' project-web-1 | jq

This shows each healthcheck attempt with exit codes and output. A common failure is the endpoint returning 404 because the route is not defined.

For deeper diagnostics, attach to a running container without stopping it:

docker compose exec web sh

Inside the container, check that your files synced correctly:

ls -la /app

Compare the timestamps with your host files. If a file is missing or stale, the watch sync may be ignoring the correct path. Recheck your ignore patterns and target path.

Finally, test the container's filesystem persistence. Stop the container, recreate it, and see if data survives:

docker compose stop web
docker compose up -d web
docker compose exec web ls /app/data

If the directory is empty, your data was written to the container's ephemeral filesystem. Move that path to a volume or bind mount as described in the inventory section.

Monitoring and Metrics

Docker Compose Watch itself does not expose metrics, but you can monitor the containers and the watch events by combining Docker's built-in metrics with log-based alerts. This section shows a minimal setup using cAdvisor, Prometheus, and Alertmanager, all run via Compose.

Add a monitoring stack to your compose.yaml:

services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.47.2
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    depends_on:
      - web

  prometheus:
    image: prom/prometheus:v2.47.2
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      - cadvisor

  alertmanager:
    image: prom/alertmanager:v0.26.0
    ports:
      - "9093:9093"
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml

Create prometheus.yml with a scrape config for cAdvisor:

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

Add a container restart alert to alertrules.yml (loaded by Prometheus):

groups:
- name: container-alerts
  rules:
  - alert: ContainerKilled
    expr: time() - container_last_seen{name=~"project-web-.*"} > 60
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Container {{ $labels.name }} has been down for more than 1 minute"

This alert fires when a container matching project-web-.* has not been seen for 60 seconds, allowing a 1-minute grace period. In alertmanager.yml, route alerts to email or Slack.

Test the alert by stopping the web container:

docker compose stop web

Within about 2 minutes (60s metric plus 1m for clause), Prometheus will show the alert in http://localhost:9090/alerts, and Alertmanager will send a notification.

Dashboard and Visualization

A dashboard helps you spot anomalies quickly. Use Grafana to visualize cAdvisor metrics.

Add Grafana to your Compose stack:

grafana:
  image: grafana/grafana:10.1.5
  ports:
    - "3000:3000"
  environment:
    GF_SECURITY_ADMIN_PASSWORD: admin
  volumes:
    - grafana-storage:/var/lib/grafana
  depends_on:
    - prometheus

Start the stack and access Grafana at http://localhost:3000 (login with admin / admin, change it immediately). Add Prometheus as a data source with URL http://prometheus:9090.

Create a new dashboard and add the following panels using Prometheus queries:

1. CPU usage per container:

   rate(container_cpu_usage_seconds_total{name=~"project-.*"}[5m])

1. Memory usage:

   container_memory_usage_bytes{name=~"project-.*"}

1. Restart count (critical for watch-triggered rebuilds):

   increase(container_restart_count{name=~"project-.*"}[1h])

1. Filesystem usage (watch sync can fill disk):

   container_fs_usage_bytes{name=~"project-.*"}

Set alert thresholds on the dashboard. For example, a restart count greater than 5 in the last hour suggests a watch rebuild loop; investigate immediately.

Failure Modes and Recovery

Here are the common failure modes with Docker Compose Watch and how to recover from each.

1. Watch is not triggering syncs or rebuilds

Symptom: You edit a file, but the container contents do not change; no log entries from the watcher.

Cause: Usually a misconfigured watch path or a missing target. The path is relative to the Compose file, not the host root. Also, the watcher may be ignoring the file because of an ignore pattern.

Diagnosis: Run docker compose config --services to verify services and check the watch section is present. Inspect the container filesystem versus host:

docker compose exec web sh -c 'stat /app/file.py'
stat host/path/file.py

If timestamps differ and the file content is stale, the sync is not happening.

Recovery: Fix the path and target to be exact matches. Remove overly broad ignores. Restart the watch process by running docker compose watch in the foreground and observing logs.

2. Infinite rebuild loop

Symptom: The container constantly rebuilds and restarts, consuming CPU and causing downtime.

Cause: A file that changes during the build process (e.g., a generated artifact) is being watched, so the watcher triggers another build.

Diagnosis: Check the watch logs for frequent rebuild actions:

docker compose logs -f web | grep -i rebuild

If you see repeated rebuilds, the trigger file is changing.

Recovery: Move generated files out of the watched directory or add them to the ignore list. For manifests, use action: sync where possible if a rebuild is unnecessary. Alternatively, use a .dockerignore to prevent the file from being copied into the build context.

3. Container exits after sync due to missing dependency

Symptom: After a sync, the container exits with a non-zero code.

Cause: The synced file references a new library that is not installed in the image because the sync occurred before the rebuild.

Diagnosis: Check container exit code and logs:

docker inspect project-web-1 --format='{{.State.ExitCode}}'
docker logs project-web-1 --tail 50

If the error is a ModuleNotFoundError, the dependency is missing.

Recovery: Change the watch action to rebuild for files that alter dependencies, or manually rebuild with docker compose up --build web and then restart the watch.

4. Data loss after container recreation

Symptom: The application loses state after a watch-triggered rebuild.

Cause: Data is stored in the container's writable layer, which is destroyed on rebuild. The service wrote to a path not backed by a volume or bind mount.

Diagnosis: Compare data before and after rebuild. Inspect mounts:

docker inspect project-db-1 --format='{{json .Mounts}}'

If the data directory is not listed, it is ephemeral.

Recovery: Add a volume to the service definition and migrate the data. Stop the container, copy data out if possible, add the volume, and restart. Going forward, ensure all stateful paths are volumes.

Operations Checklist

Use this checklist for each watch-enabled service before and during development. Assign one owner per item to avoid ambiguity.

Ownership note: The lead developer owns the overall watch configuration and is accountable for the watch action definitions. The DevOps engineer owns monitoring and alerting infrastructure. Review the checklist in a weekly standup, and update owners if team composition changes.

Change control: Any change to the watch section or monitoring rules must go through a pull request with a review by at least one other engineer. After merging, run the verification steps above and record the results in the deployment log.

Common Pitfalls

Avoid these mistakes that frequently trip up teams using Compose Watch.

1. Watching too many files - Including a large directory like node_modules in the sync path can overwhelm the watcher and cause delays. Always define ignore patterns. Example: ignore */.log, /__pycache__, /node_modules.

1. Confusing sync and rebuild - Using rebuild for every change defeats the purpose of watch; using sync for dependency changes leaves the container in a broken state. Use sync for source files, rebuild for manifests and Dockerfile changes.

1. Not setting resource limits - Watch-driven rebuilds can spike CPU and memory. Define deploy.resources.limits for services in development to prevent host destabilization:

   services:
     web:
       deploy:
         resources:
           limits:
             cpus: '1.0'
             memory: 512M

1. Ignoring watch logs - The watcher emits useful information about what it is doing. Run docker compose logs -f in a separate terminal during development; do not silence them.

1. Assuming watch works in production - Compose Watch is designed for development. In production, use immutable images and orchestration. If you must use it for small-scale deployments, disable watch with docker compose up (not watch) and monitor closely.

1. Hardcoding secrets in watch-triggered code - A sync may copy files with secrets into the container. Use environment variables and secret mounts; never commit secrets to the repository.

Conclusion

Docker Compose Watch can dramatically speed up development, but it is not a fire-and-forget tool. Effective monitoring and alerting require you to understand your environment, configure watch actions correctly, verify the container state after each change, and prepare for failures.

Start with the inventory and safe configuration sections, then add healthchecks and metrics. The operational checklist and common pitfalls provide a framework for keeping your Compose Watch setup healthy over time. Remember to review alert thresholds and ownership assignments regularly, as applications and teams evolve.

As a next step, implement one monitoring aspect from this guide: define a healthcheck for your most critical service, set up a restart alert, or create a dashboard panel for restart counts. Document the current state, run the check, and compare with the expected signal. Then expand to the rest of your stack. Reliable watch monitoring makes development faster and safer.