## Intro Docker Compose Watch gives developers a fast feedback loop by rebuilding and restarting services when source files change. But in any non-trivial project, you also need to know when things go wrong: a service crash, a failed build, a misconfigured mount, or a resource limit. Monitoring and alerting turn Compose Watch from a convenience into a reliable part of your operations workflow. This guide walks through a practical implementation for Docker Compose Watch monitoring, alerts, dashboards, and incident response. It assumes you are comfortable with Docker and Compose basics, and it focuses on the operational layer: what to watch, how to detect problems, and how to respond safely. Every section includes exact commands, configuration snippets, expected outputs, and recovery steps. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths. Do not treat this as a copy-paste checklist; adapt the examples to your environment and version constraints. ## Version and Environment Inventory Before you monitor anything, know what you are running. A monitoring rule that assumes the wrong version or topology will either miss failures or flood you with false alarms. Start by capturing your current Docker and Compose versions. Run: ```bash docker version --format '{{.Server.Version}}' docker compose version ``` Expected output example: ```text 24.0.7 Docker Compose version v2.21.0 ``` Here `24.0.7` is the Docker engine version and `v2.21.0` is the Compose plugin version. Compose Watch requires Docker Compose v2.17.0 or later, and the `watch` attribute is only fully supported with BuildKit and the Compose v2 file format (typically `compose.yaml`). If you see an older version, upgrade before enabling watch. Next, list the services in your Compose project and their state: ```bash docker compose ps ``` Example output: ```text NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS project-web-1 project-web:latest "gunicorn app:app" web 2 minutes ago Up 2 minutes 0.0.0.0:8000->8000/tcp project-db-1 postgres:15 "docker-entrypoint.s…" db 2 minutes ago Up 2 minutes 5432/tcp ``` Note the service names (`web`, `db`), container names, and health status. A missing `Up` under STATUS is the first failure signal. For read-only inspection of a suspicious container, use: ```bash docker inspect project-web-1 --format '{{json .State.Health}}' ``` Expected output (if a healthcheck is defined): ```json {"Status":"healthy","FailingStreak":0,"Log":[{"Start":"2024-01-01T00:00:00Z","End":"2024-01-01T00:00:05Z","ExitCode":0,"Output":"ok"}]} ``` If no healthcheck is defined, the field is `null`. You can add one later; we cover that in the Verification section. Lastly, confirm your volume mounts are correct before any change. Data loss is a common Compose failure. Use: ```bash docker inspect project-db-1 --format '{{json .Mounts}}' ``` Example output: ```json [{"Type":"volume","Name":"project_pgdata","Source":"/var/lib/docker/volumes/project_pgdata/_data","Destination":"/var/lib/postgresql/data","Driver":"local","Mode":"z","RW":true}] ``` This shows a named volume bound to `/var/lib/postgresql/data`. If you expected a bind mount to `./data`, this is a mismatch. Correcting it before a restart prevents hours of debugging later. ## Safe Configuration Path The `watch` section in your Compose file controls how changes are synced. A misconfigured watch rule can cause infinite rebuild loops, container churn, or missed syncs. Here is a safe starting configuration: ```yaml services: web: build: . ports: - "8000:8000" volumes: - .:/app watch: - action: sync path: ./app target: /app ignore: - node_modules/ - .venv/ - action: rebuild path: requirements.txt environment: FLASK_ENV: development ``` Key points: - Use `action: sync` for directories where only file content changes, not dependencies. - Use `action: rebuild` for manifest files like `requirements.txt` or `Dockerfile` changes. - Always include an `ignore` list for large generated directories (`node_modules`, `.venv`, `__pycache__`) to reduce sync load. To verify your watch configuration is valid, run: ```bash docker compose config --quiet ``` If the configuration is invalid, this command exits with a non-zero status and prints errors to stderr. Example of a syntax error: ```text yaml: line 10: did not find expected key ``` Fix the YAML indentation before proceeding. Now test the sync path manually. Start the stack: ```bash docker compose up --build -d ``` Watch the logs for the watcher startup message: ```bash docker compose logs -f web ``` Example log line: ```text web-1 | [watch] watching /app for changes ``` Trigger a sync by editing a file in `./app`, for example: ```bash echo "print('hello')" >> app/main.py ``` You should see the watcher log the change and sync the file without restarting the container. If instead the container restarts or rebuilds, you have the wrong `action` type. ## Verification and Diagnostics After configuring watch, verify that your application behaves correctly under sync and rebuild actions. The first step is to define a healthcheck inside your service. For a Python web app, add to `compose.yaml`: ```yaml services: web: # ... healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8000/health"] interval: 10s timeout: 5s retries: 3 start_period: 30s ``` This healthcheck runs `curl -f` against a `/health` endpoint every 10 seconds, considers the container unhealthy after 3 consecutive failures, and waits 30 seconds before starting checks during startup. Apply the change and recreate the service: ```bash docker compose up -d web ``` Check health status: ```bash docker inspect --format='{{.State.Health.Status}}' project-web-1 ``` Expected output: ```text healthy ``` If it says `unhealthy`, inspect the healthcheck logs: ```bash docker inspect --format='{{json .State.Health.Log}}' project-web-1 | jq ``` This shows each healthcheck attempt with exit codes and output. A common failure is the endpoint returning 404 because the route is not defined. For deeper diagnostics, attach to a running container without stopping it: ```bash docker compose exec web sh ``` Inside the container, check that your files synced correctly: ```bash ls -la /app ``` Compare the timestamps with your host files. If a file is missing or stale, the watch sync may be ignoring the correct path. Recheck your `ignore` patterns and `target` path. Finally, test the container's filesystem persistence. Stop the container, recreate it, and see if data survives: ```bash docker compose stop web docker compose up -d web docker compose exec web ls /app/data ``` If the directory is empty, your data was written to the container's ephemeral filesystem. Move that path to a volume or bind mount as described in the inventory section. ## Monitoring and Metrics Docker Compose Watch itself does not expose metrics, but you can monitor the containers and the watch events by combining Docker's built-in metrics with log-based alerts. This section shows a minimal setup using cAdvisor, Prometheus, and Alertmanager, all run via Compose. Add a monitoring stack to your `compose.yaml`: ```yaml services: cadvisor: image: gcr.io/cadvisor/cadvisor:v0.47.2 ports: - "8080:8080" volumes: - /:/rootfs:ro - /var/run:/var/run:ro - /sys:/sys:ro - /var/lib/docker/:/var/lib/docker:ro depends_on: - web prometheus: image: prom/prometheus:v2.47.2 ports: - "9090:9090" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml depends_on: - cadvisor alertmanager: image: prom/alertmanager:v0.26.0 ports: - "9093:9093" volumes: - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml ``` Create `prometheus.yml` with a scrape config for cAdvisor: ```yaml global: scrape_interval: 15s scrape_configs: - job_name: 'cadvisor' static_configs: - targets: ['cadvisor:8080'] - job_name: 'node-exporter' static_configs: - targets: ['node-exporter:9100'] ``` Add a container restart alert to `alertrules.yml` (loaded by Prometheus): ```yaml groups: - name: container-alerts rules: - alert: ContainerKilled expr: time() - container_last_seen{name=~"project-web-.*"} > 60 for: 1m labels: severity: critical annotations: summary: "Container {{ $labels.name }} has been down for more than 1 minute" ``` This alert fires when a container matching `project-web-.*` has not been seen for 60 seconds, allowing a 1-minute grace period. In `alertmanager.yml`, route alerts to email or Slack. Test the alert by stopping the web container: ```bash docker compose stop web ``` Within about 2 minutes (60s metric plus 1m `for` clause), Prometheus will show the alert in `http://localhost:9090/alerts`, and Alertmanager will send a notification. ## Dashboard and Visualization A dashboard helps you spot anomalies quickly. Use Grafana to visualize cAdvisor metrics. Add Grafana to your Compose stack: ```yaml grafana: image: grafana/grafana:10.1.5 ports: - "3000:3000" environment: GF_SECURITY_ADMIN_PASSWORD: admin volumes: - grafana-storage:/var/lib/grafana depends_on: - prometheus ``` Start the stack and access Grafana at `http://localhost:3000` (login with `admin` / `admin`, change it immediately). Add Prometheus as a data source with URL `http://prometheus:9090`. Create a new dashboard and add the following panels using Prometheus queries: 1. **CPU usage per container**: ```promql rate(container_cpu_usage_seconds_total{name=~"project-.*"}[5m]) ``` 2. **Memory usage**: ```promql container_memory_usage_bytes{name=~"project-.*"} ``` 3. **Restart count** (critical for watch-triggered rebuilds): ```promql increase(container_restart_count{name=~"project-.*"}[1h]) ``` 4. **Filesystem usage** (watch sync can fill disk): ```promql container_fs_usage_bytes{name=~"project-.*"} ``` Set alert thresholds on the dashboard. For example, a restart count greater than 5 in the last hour suggests a watch rebuild loop; investigate immediately. ## Failure Modes and Recovery Here are the common failure modes with Docker Compose Watch and how to recover from each. ### 1. Watch is not triggering syncs or rebuilds **Symptom**: You edit a file, but the container contents do not change; no log entries from the watcher. **Cause**: Usually a misconfigured `watch` path or a missing `target`. The path is relative to the Compose file, not the host root. Also, the watcher may be ignoring the file because of an `ignore` pattern. **Diagnosis**: Run `docker compose config --services` to verify services and check the `watch` section is present. Inspect the container filesystem versus host: ```bash docker compose exec web sh -c 'stat /app/file.py' stat host/path/file.py ``` If timestamps differ and the file content is stale, the sync is not happening. **Recovery**: Fix the `path` and `target` to be exact matches. Remove overly broad ignores. Restart the watch process by running `docker compose watch` in the foreground and observing logs. ### 2. Infinite rebuild loop **Symptom**: The container constantly rebuilds and restarts, consuming CPU and causing downtime. **Cause**: A file that changes during the build process (e.g., a generated artifact) is being watched, so the watcher triggers another build. **Diagnosis**: Check the watch logs for frequent rebuild actions: ```bash docker compose logs -f web | grep -i rebuild ``` If you see repeated rebuilds, the trigger file is changing. **Recovery**: Move generated files out of the watched directory or add them to the `ignore` list. For manifests, use `action: sync` where possible if a rebuild is unnecessary. Alternatively, use a `.dockerignore` to prevent the file from being copied into the build context. ### 3. Container exits after sync due to missing dependency **Symptom**: After a sync, the container exits with a non-zero code. **Cause**: The synced file references a new library that is not installed in the image because the sync occurred before the rebuild. **Diagnosis**: Check container exit code and logs: ```bash docker inspect project-web-1 --format='{{.State.ExitCode}}' docker logs project-web-1 --tail 50 ``` If the error is a `ModuleNotFoundError`, the dependency is missing. **Recovery**: Change the watch action to `rebuild` for files that alter dependencies, or manually rebuild with `docker compose up --build web` and then restart the watch. ### 4. Data loss after container recreation **Symptom**: The application loses state after a watch-triggered rebuild. **Cause**: Data is stored in the container's writable layer, which is destroyed on rebuild. The service wrote to a path not backed by a volume or bind mount. **Diagnosis**: Compare data before and after rebuild. Inspect mounts: ```bash docker inspect project-db-1 --format='{{json .Mounts}}' ``` If the data directory is not listed, it is ephemeral. **Recovery**: Add a volume to the service definition and migrate the data. Stop the container, copy data out if possible, add the volume, and restart. Going forward, ensure all stateful paths are volumes. ## Operations Checklist Use this checklist for each watch-enabled service before and during development. Assign one owner per item to avoid ambiguity. | # | Checklist Item | Owner | How to Verify | Review Frequency | |---|---|---|---|---| | 1 | Confirm Docker and Compose versions meet minimums | Lead Developer (e.g., Priya Shah) | Run `docker compose version`; compare to required | Every release | | 2 | Review `watch` configuration for correct paths and ignores | DevOps Engineer (e.g., Marcus Chen) | `docker compose config --quiet`; manual test sync | Weekly | | 3 | Ensure healthchecks are defined for all stateful services | Backend Lead (e.g., Ana Lopez) | `docker inspect --format='{{.State.Health.Status}}'` | Every PR | | 4 | Verify data persistence with a restart test | QA Engineer (e.g., Tom Nguyen) | Stop, recreate, check data exists | Before each deploy | | 5 | Monitor restart counts for rebuild loops | SRE (e.g., Jordan Patel) | Prometheus query `increase(container_restart_count[1h])` | Daily | | 6 | Test alert routing end-to-end | On-call Engineer (rotating) | Trigger a test alert, confirm notification | Monthly | | 7 | Review sync and rebuild performance (CPU, disk I/O) | DevOps Engineer | Grafana dashboard; check cAdvisor metrics | Weekly | **Ownership note**: The lead developer owns the overall watch configuration and is accountable for the watch action definitions. The DevOps engineer owns monitoring and alerting infrastructure. Review the checklist in a weekly standup, and update owners if team composition changes. **Change control**: Any change to the `watch` section or monitoring rules must go through a pull request with a review by at least one other engineer. After merging, run the verification steps above and record the results in the deployment log. ## Common Pitfalls Avoid these mistakes that frequently trip up teams using Compose Watch. 1. **Watching too many files** - Including a large directory like `node_modules` in the sync path can overwhelm the watcher and cause delays. Always define ignore patterns. Example: ignore `**/*.log`, `**/__pycache__`, `**/node_modules`. 2. **Confusing sync and rebuild** - Using `rebuild` for every change defeats the purpose of watch; using `sync` for dependency changes leaves the container in a broken state. Use `sync` for source files, `rebuild` for manifests and Dockerfile changes. 3. **Not setting resource limits** - Watch-driven rebuilds can spike CPU and memory. Define `deploy.resources.limits` for services in development to prevent host destabilization: ```yaml services: web: deploy: resources: limits: cpus: '1.0' memory: 512M ``` 4. **Ignoring watch logs** - The watcher emits useful information about what it is doing. Run `docker compose logs -f` in a separate terminal during development; do not silence them. 5. **Assuming watch works in production** - Compose Watch is designed for development. In production, use immutable images and orchestration. If you must use it for small-scale deployments, disable watch with `docker compose up` (not `watch`) and monitor closely. 6. **Hardcoding secrets in watch-triggered code** - A sync may copy files with secrets into the container. Use environment variables and secret mounts; never commit secrets to the repository. ## Conclusion Docker Compose Watch can dramatically speed up development, but it is not a fire-and-forget tool. Effective monitoring and alerting require you to understand your environment, configure watch actions correctly, verify the container state after each change, and prepare for failures. Start with the inventory and safe configuration sections, then add healthchecks and metrics. The operational checklist and common pitfalls provide a framework for keeping your Compose Watch setup healthy over time. Remember to review alert thresholds and ownership assignments regularly, as applications and teams evolve. As a next step, implement one monitoring aspect from this guide: define a healthcheck for your most critical service, set up a restart alert, or create a dashboard panel for restart counts. Document the current state, run the check, and compare with the expected signal. Then expand to the rest of your stack. Reliable watch monitoring makes development faster and safer.