## Intro Docker containers are isolated but not automatically constrained. Without explicit resource limits, a single container can consume every available CPU cycle or byte of memory on the host, starving neighboring services, triggering the kernel OOM killer, and degrading the entire node. Monitoring and alerting turn this silent risk into a visible, manageable problem. This guide walks through a practical, end-to-end approach: inventorying your current Docker environment, enabling metrics collection, configuring Prometheus and cAdvisor, defining alert rules, validating them under real load, and responding to incidents. You will learn which metrics actually matter, how to avoid common false positives, and how to make monitoring a routine operational practice rather than an afterthought. By the end, you will have a repeatable workflow. We will use Docker's native capabilities plus a Prometheus-based stack as concrete examples, but the principles transfer to any monitoring system. The goal is not just to see numbers on a dashboard; it is to know when a container is about to blow past its limits and to act before users notice. ## Version and Environment Inventory Before you can set meaningful thresholds, you need to know what you are running and what limits already exist. Start with a complete inventory of your Docker hosts. ### Check Docker and OS versions Run the following on each Docker host: ```bash docker version cat /etc/os-release uname -r ``` Example output (trimmed for readability): ```text Client: Docker Engine - Community Version: 24.0.7 API version: 1.43 Go version: go1.20.10 Git commit: afdd53b Built: Thu Oct 26 09:11:35 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.7 API version: 1.43 (minimum version 1.12) Go version: go1.20.10 Git commit: afdd53b Built: Thu Oct 26 09:09:17 2023 OS/Arch: linux/amd64 Experimental: false ``` This confirms Docker Engine 24.0.7 on a Linux host (Ubuntu 22.04 in this example). The API version is 1.43. Note the OS release and kernel version; resource accounting features like cgroup v2 behave differently across kernel versions, and you need to know your baseline. ### List containers and their resource limits Use `docker ps` to list running containers: ```bash docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.Status}}" ``` Then inspect a specific container (replace `` with its name or ID) to see its runtime resource settings: ```bash docker inspect --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}} {{.HostConfig.CpuShares}}' ``` Example output: ```text 1073741824 500000000 1024 ``` Interpretation: - Memory limit: 1073741824 bytes = 1 GiB - NanoCpus: 500000000 = 0.5 CPU (nano CPUs are 1e-9 of a CPU) - CpuShares: 1024 (default weight, only relevant under CPU contention) If no limits are set, you will typically see `0` for memory and `0` for NanoCpus. A zero means the container can use all host resources, which is often the root cause of noisy neighbor incidents. ### Summarize your environment Create a table of containers and their limits to assess risk. For example: | Container Name | Image | Memory Limit | CPU Limit | Status | |---|---|---|---|---| | web | nginx:1.25 | 512 MiB | 0.5 CPU | Up 2 hours | | db | postgres:16 | 2 GiB | 1.0 CPU | Up 5 days | | cache | redis:7 | 256 MiB | 0.25 CPU | Up 1 day | This table immediately tells you which containers are most constrained. In this example, `cache` has the tightest memory limit and deserves closer monitoring. The `web` container has no history yet; you should watch it for a few days before trusting its limits. **Owner and cadence:** Assign an infrastructure engineer (for example, Jordan Lee) to maintain this inventory and update it weekly or whenever containers are added or removed. The update should be part of your regular change review process. ## Safe Configuration Path Do not roll out monitoring and limits to everything at once. Start with a single non-critical container so you can validate your setup without risking production. ### Step 1: Enable the Docker metrics endpoint Docker Engine can expose metrics in Prometheus format. On Linux, configure the daemon to listen on a local TCP socket. Create or edit `/etc/docker/daemon.json`: ```json { "metrics-addr": "127.0.0.1:9323", "experimental": true } ``` Then restart Docker: ```bash sudo systemctl restart docker ``` Verify the endpoint: ```bash curl http://127.0.0.1:9323/metrics | head -n 20 ``` Expected output includes lines like: ```text # HELP engine_daemon_container_states_containers The count of containers in various states # TYPE engine_daemon_container_states_containers gauge engine_daemon_container_states_containers{state="running"} 3 engine_daemon_container_states_containers{state="paused"} 0 engine_daemon_container_states_containers{state="stopped"} 0 ``` Note: The `experimental` key is required on some Docker versions to enable the metrics endpoint. Check your specific Docker version's documentation if the endpoint does not appear. ### Step 2: Set resource limits on a pilot container If your containers lack limits, add them to a test container first. For example, run an nginx container with memory and CPU limits: ```bash docker run -d --name web-pilot --memory=512m --cpus=0.5 -p 8080:80 nginx:1.25 ``` Verify the limits are applied: ```bash docker inspect web-pilot --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}}' ``` Output: ```text 536870912 500000000 ``` This corresponds to 512 MiB and 0.5 CPU. ### Step 3: Deploy cAdvisor for detailed container metrics While Docker's native metrics include some container information, per-container resource usage like working set memory or CPU throttling is best exported by cAdvisor. Run cAdvisor as a container: ```bash docker run -d \ --name=cadvisor \ --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:ro \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --publish=8080:8080 \ gcr.io/cadvisor/cadvisor:v0.47.0 ``` Access cAdvisor metrics at `http://:8080/metrics`. You will see detailed metrics like `container_memory_working_set_bytes`, `container_cpu_usage_seconds_total`, and `container_spec_memory_limit_bytes`. ### Step 4: Configure Prometheus to scrape metrics Add scrape configurations to your `prometheus.yml` for both the Docker daemon and cAdvisor: ```yaml scrape_configs: - job_name: 'docker' static_configs: - targets: [':9323'] - job_name: 'cadvisor' static_configs: - targets: [':8080'] ``` Reload Prometheus to apply the configuration. ### Step 5: Define alert rules A high-value alert is for container memory usage near its limit. In your Prometheus rules file: ```yaml groups: - name: docker_container_alerts rules: - alert: ContainerMemoryUsageHigh expr: (container_memory_working_set_bytes / container_spec_memory_limit_bytes) > 0.9 for: 5m labels: severity: warning annotations: summary: "Container {{ $labels.name }} memory usage above 90% of limit" description: "Container {{ $labels.name }} has used {{ $value | humanizePercentage }} of its memory limit for 5 minutes." ``` This alert fires when a container uses more than 90% of its memory limit for 5 consecutive minutes. The `for` clause prevents flapping from short spikes. **Owner and cadence:** The alert rules are owned by the platform monitoring team lead (e.g., Alex Morgan). Review alert thresholds monthly and after any major application change to ensure they still match expected usage patterns. ## Verification and Diagnostics After configuring monitoring, verify that metrics are collected, alerts actually fire, and your dashboards show meaningful data. ### Check Prometheus target health Query Prometheus for target status: ```bash curl -s http://:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .job, health: .health, lastError: .lastError}' ``` Expected output shows `"health":"up"` for each target. If a target is down, check the exporter container or daemon status. ### Inspect key metrics Use the Prometheus query browser or API to run these queries: - Container memory usage vs limit (ratio): ```promql container_memory_working_set_bytes{name="web-pilot"} / container_spec_memory_limit_bytes{name="web-pilot"} ``` - Container CPU throttling rate (seconds throttled per second, over 5 minutes): ```promql rate(container_cpu_cfs_throttled_seconds_total{name="web-pilot"}[5m]) ``` A non-zero throttling rate means the container is being restricted below its CPU limit, often due to bursty workloads. ### Generate test load To verify alerts, create load inside the pilot container. For memory stress, run a command that allocates memory. The `stress` tool is ideal but may not be installed. You can use a simple shell loop with `head` from `/dev/zero` as a fallback: ```bash docker exec web-pilot sh -c "tail /dev/zero -c 400M | head -c 400M > /dev/null & sleep 60; kill %1" ``` This allocates about 400 MiB and holds it for 60 seconds. Since the container limit is 512 MiB, this should push usage above 90% if the container was already using some memory. Observe the metric ratio increase in Prometheus, and after 5 minutes (if you set `for: 5m`), the alert should fire. ### Check alert delivery Confirm you received a notification through your Alertmanager configuration. If not, check Alertmanager logs and routing rules. A common mistake is forgetting to route alerts to a receiver; verify your `alertmanager.yml` includes a route and receiver for the `warning` severity. ## Failure Modes and Recovery Monitoring systems themselves can fail. Here are common failure modes and how to recover. ### Alert flapping If an alert repeatedly fires and resolves, the threshold is likely too tight or the `for` duration too short. Increase `for` to 10 minutes or adjust the threshold. For example, if a container routinely spikes to 91% for 4 minutes, but is otherwise fine, a 90% threshold with `for: 5m` will flap. Either raise the threshold to 95% or extend `for` to 15 minutes. ### Missing metrics due to exporter crash If cAdvisor or the Docker metrics endpoint goes down, Prometheus will show the target as `down`. Set up an alert for that: ```yaml - alert: ExporterDown expr: up == 0 for: 1m labels: severity: critical annotations: summary: "Exporter {{ $labels.job }} is down" ``` To recover, restart the container or service and verify the target comes back up. For cAdvisor, `docker restart cadvisor` usually works. If it repeatedly crashes, check its logs for root cause. ### Alert rule misconfiguration If alerts do not fire when expected, check rule evaluation in the Prometheus UI under Status > Rules. Look for errors. Validate YAML syntax and reload. A common syntax error is incorrect indentation in the rules file. ### False positives due to limit misread cAdvisor may report `container_spec_memory_limit_bytes` as 0 for containers without an explicit limit, causing division by zero in your query. Modify the alert expression to filter out zero limits: ```promql (container_memory_working_set_bytes / container_spec_memory_limit_bytes) > 0.9 and container_spec_memory_limit_bytes > 0 ``` This prevents false alarms for unlimited containers. Better yet, set limits on all containers so this edge case disappears. ### Rollback monitoring changes If a configuration change causes issues, revert to the previous configuration. For Docker `daemon.json` changes, restore the original file and restart Docker. For Prometheus config, revert the file and reload. Keep version-controlled backups of these configurations. ## Common Pitfalls and How to Avoid Them Over years of operating container fleets, certain mistakes appear repeatedly. Here is how to sidestep them. **Pitfall 1: Setting memory limits without monitoring working set vs RSS.** Memory limits apply to the container's total memory usage, but the kernel's OOM killer uses a broader metric than `container_memory_working_set_bytes`. cAdvisor's working set excludes inactive file cache, so your alert may fire after the container is already in trouble. To avoid this, also monitor `container_memory_usage_bytes` and set alerts at 85% of the limit for working set and 95% for total usage. **Pitfall 2: Ignoring CPU throttling.** A container may stay below its CPU limit on average but still be throttled during bursts. This causes latency spikes that users notice. Always alert on `container_cpu_cfs_throttled_seconds_total` with a non-zero rate over 5 minutes. If throttling is common, increase the CPU limit or adjust the application's behavior. **Pitfall 3: Trusting default alert templates without testing.** Prometheus alert expressions often look correct but fail silently because labels do not match or metrics are missing. Always test alerts with known load. Use `promtool test rules` to validate rule files before deploying. **Pitfall 4: Setting limits without benchmarking first.** Picking arbitrary limits like 512 MiB for a database can cause OOM kills during peak traffic. Benchmark your application under realistic load and set limits at least 20% above observed peak usage. Revisit limits after every significant code change. **Pitfall 5: Forgetting to monitor the monitoring stack itself.** If Prometheus or cAdvisor goes down, you lose visibility. Place the monitoring stack on a separate host or at least in a different failure domain, and set up external health checks. ## Operations Checklist Use this checklist for regular review and incident response. ### Daily checks - [ ] Verify Prometheus targets are up: `up == 1` in Prometheus UI or via API. - [ ] Check for any active alerts in Alertmanager. Acknowledge them if they are being worked on. - [ ] Review the top 5 containers by memory and CPU usage: query `topk(5, container_memory_working_set_bytes)` and `topk(5, rate(container_cpu_usage_seconds_total[5m]))`. - [ ] Owner: On-call SRE (rotating). ### Weekly review - [ ] Analyze resource usage trends for all containers. Identify any container that has exceeded 80% of its limit for more than 1 hour in the past week. - [ ] Test alerting by simulating a resource spike in a dedicated test container. Owner: Platform engineer (e.g., Jordan Lee). - [ ] Check for updates to cAdvisor and Prometheus exporters. Plan upgrades during maintenance windows. - [ ] Owner: Monitoring lead (Alex Morgan). Frequency: Every Monday. ### Incident response steps When a resource alert fires, follow this sequence: 1. Acknowledge the alert in Alertmanager and note the time. 2. Identify the affected container(s). Query Prometheus for the specific metric: `container_memory_working_set_bytes{name="db"}`. 3. Check current resource usage on the host: `docker stats --no-stream` to see memory, CPU, and PIDs. 4. Inspect container logs for errors: `docker logs --tail 100`. 5. If the container is unresponsive due to OOM, restart it: `docker restart `. 6. If the issue persists, temporarily increase resource limits: `docker update --memory 1g --cpus 1 `. 7. After resolving, investigate root cause and adjust limits or application code. Document the incident. ### Key metrics table | Metric | Description | Alert Suggestion | |---|---|---| | `container_memory_working_set_bytes` | Memory currently in use by container, excluding inactive file cache | >90% of limit for 5 min | | `container_memory_usage_bytes` | Total memory usage including cache | >95% of limit for 5 min | | `container_cpu_usage_seconds_total` | Total CPU time consumed (counter) | Rate > 0.9 x limit over 15 min | | `container_spec_memory_limit_bytes` | Memory limit configured for container | Use as denominator in ratio alerts | | `container_cpu_cfs_throttled_seconds_total` | Total CPU throttling time (counter) | Any non-zero rate over 5 min | | `up` | Exporter health (1=up, 0=down) | == 0 for 1 min | ## Conclusion Monitoring Docker resource limits transforms a hidden risk into a managed operational practice. In this guide, we covered a full workflow: inventorying your environment, enabling metrics with Docker and cAdvisor, configuring Prometheus alerts, validating with test load, handling common failure modes, and following a daily and weekly checklist. Start small: pick one non-critical container, apply limits, and monitor it for a week. Watch how it behaves under real traffic. Then expand to all containers, integrate alerts into your incident management tool, and make weekly reviews a habit. The effort pays off in fewer OOM surprises, fewer noisy neighbor incidents, and a more stable platform for everyone.