Docker Resource Limits: Monitoring, Alerting, and Practical Operations

Intro

Docker containers are isolated but not automatically constrained. Without explicit resource limits, a single container can consume every available CPU cycle or byte of memory on the host, starving neighboring services, triggering the kernel OOM killer, and degrading the entire node. Monitoring and alerting turn this silent risk into a visible, manageable problem.

This guide walks through a practical, end-to-end approach: inventorying your current Docker environment, enabling metrics collection, configuring Prometheus and cAdvisor, defining alert rules, validating them under real load, and responding to incidents. You will learn which metrics actually matter, how to avoid common false positives, and how to make monitoring a routine operational practice rather than an afterthought.

By the end, you will have a repeatable workflow. We will use Docker's native capabilities plus a Prometheus-based stack as concrete examples, but the principles transfer to any monitoring system. The goal is not just to see numbers on a dashboard; it is to know when a container is about to blow past its limits and to act before users notice.

Version and Environment Inventory

Before you can set meaningful thresholds, you need to know what you are running and what limits already exist. Start with a complete inventory of your Docker hosts.

Check Docker and OS versions

Run the following on each Docker host:

docker version
cat /etc/os-release
uname -r

Example output (trimmed for readability):

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:11:35 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:        go1.20.10
  Git commit:        afdd53b
  Built:             Thu Oct 26 09:09:17 2023
  OS/Arch:           linux/amd64
  Experimental:      false

This confirms Docker Engine 24.0.7 on a Linux host (Ubuntu 22.04 in this example). The API version is 1.43. Note the OS release and kernel version; resource accounting features like cgroup v2 behave differently across kernel versions, and you need to know your baseline.

List containers and their resource limits

Use docker ps to list running containers:

docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Image}}\t{{.Status}}"

Then inspect a specific container (replace <container> with its name or ID) to see its runtime resource settings:

docker inspect <container> --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}} {{.HostConfig.CpuShares}}'

Example output:

1073741824 500000000 1024

Interpretation:

• Memory limit: 1073741824 bytes = 1 GiB
• NanoCpus: 500000000 = 0.5 CPU (nano CPUs are 1e-9 of a CPU)
• CpuShares: 1024 (default weight, only relevant under CPU contention)

If no limits are set, you will typically see 0 for memory and 0 for NanoCpus. A zero means the container can use all host resources, which is often the root cause of noisy neighbor incidents.

Summarize your environment

Create a table of containers and their limits to assess risk. For example:

This table immediately tells you which containers are most constrained. In this example, cache has the tightest memory limit and deserves closer monitoring. The web container has no history yet; you should watch it for a few days before trusting its limits.

Owner and cadence: Assign an infrastructure engineer (for example, Jordan Lee) to maintain this inventory and update it weekly or whenever containers are added or removed. The update should be part of your regular change review process.

Safe Configuration Path

Do not roll out monitoring and limits to everything at once. Start with a single non-critical container so you can validate your setup without risking production.

Step 1: Enable the Docker metrics endpoint

Docker Engine can expose metrics in Prometheus format. On Linux, configure the daemon to listen on a local TCP socket. Create or edit /etc/docker/daemon.json:

{
  "metrics-addr": "127.0.0.1:9323",
  "experimental": true
}

Then restart Docker:

sudo systemctl restart docker

Verify the endpoint:

curl http://127.0.0.1:9323/metrics | head -n 20

Expected output includes lines like:

# HELP engine_daemon_container_states_containers The count of containers in various states
# TYPE engine_daemon_container_states_containers gauge
engine_daemon_container_states_containers{state="running"} 3
engine_daemon_container_states_containers{state="paused"} 0
engine_daemon_container_states_containers{state="stopped"} 0

Note: The experimental key is required on some Docker versions to enable the metrics endpoint. Check your specific Docker version's documentation if the endpoint does not appear.

Step 2: Set resource limits on a pilot container

If your containers lack limits, add them to a test container first. For example, run an nginx container with memory and CPU limits:

docker run -d --name web-pilot --memory=512m --cpus=0.5 -p 8080:80 nginx:1.25

Verify the limits are applied:

docker inspect web-pilot --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}}'

Output:

536870912 500000000

This corresponds to 512 MiB and 0.5 CPU.

Step 3: Deploy cAdvisor for detailed container metrics

While Docker's native metrics include some container information, per-container resource usage like working set memory or CPU throttling is best exported by cAdvisor. Run cAdvisor as a container:

docker run -d \
  --name=cadvisor \
  --volume=/:/rootfs:ro \
  --volume=/var/run:/var/run:ro \
  --volume=/sys:/sys:ro \
  --volume=/var/lib/docker/:/var/lib/docker:ro \
  --publish=8080:8080 \
  gcr.io/cadvisor/cadvisor:v0.47.0

Access cAdvisor metrics at http://<host>:8080/metrics. You will see detailed metrics like container_memory_working_set_bytes, container_cpu_usage_seconds_total, and container_spec_memory_limit_bytes.

Step 4: Configure Prometheus to scrape metrics

Add scrape configurations to your prometheus.yml for both the Docker daemon and cAdvisor:

scrape_configs:
  - job_name: 'docker'
    static_configs:
      - targets: ['<host-ip>:9323']
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['<host-ip>:8080']

Reload Prometheus to apply the configuration.

Step 5: Define alert rules

A high-value alert is for container memory usage near its limit. In your Prometheus rules file:

groups:
  - name: docker_container_alerts
    rules:
      - alert: ContainerMemoryUsageHigh
        expr: (container_memory_working_set_bytes / container_spec_memory_limit_bytes) > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} memory usage above 90% of limit"
          description: "Container {{ $labels.name }} has used {{ $value | humanizePercentage }} of its memory limit for 5 minutes."

This alert fires when a container uses more than 90% of its memory limit for 5 consecutive minutes. The for clause prevents flapping from short spikes.

Owner and cadence: The alert rules are owned by the platform monitoring team lead (e.g., Alex Morgan). Review alert thresholds monthly and after any major application change to ensure they still match expected usage patterns.

Verification and Diagnostics

After configuring monitoring, verify that metrics are collected, alerts actually fire, and your dashboards show meaningful data.

Check Prometheus target health

Query Prometheus for target status:

curl -s http://<prometheus>:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .job, health: .health, lastError: .lastError}'

Expected output shows "health":"up" for each target. If a target is down, check the exporter container or daemon status.

Inspect key metrics

Use the Prometheus query browser or API to run these queries:

• Container memory usage vs limit (ratio):

container_memory_working_set_bytes{name="web-pilot"} / container_spec_memory_limit_bytes{name="web-pilot"}

• Container CPU throttling rate (seconds throttled per second, over 5 minutes):

rate(container_cpu_cfs_throttled_seconds_total{name="web-pilot"}[5m])

A non-zero throttling rate means the container is being restricted below its CPU limit, often due to bursty workloads.

Generate test load

To verify alerts, create load inside the pilot container. For memory stress, run a command that allocates memory. The stress tool is ideal but may not be installed. You can use a simple shell loop with head from /dev/zero as a fallback:

docker exec web-pilot sh -c "tail /dev/zero -c 400M | head -c 400M > /dev/null & sleep 60; kill %1"

This allocates about 400 MiB and holds it for 60 seconds. Since the container limit is 512 MiB, this should push usage above 90% if the container was already using some memory. Observe the metric ratio increase in Prometheus, and after 5 minutes (if you set for: 5m), the alert should fire.

Check alert delivery

Confirm you received a notification through your Alertmanager configuration. If not, check Alertmanager logs and routing rules. A common mistake is forgetting to route alerts to a receiver; verify your alertmanager.yml includes a route and receiver for the warning severity.

Failure Modes and Recovery

Monitoring systems themselves can fail. Here are common failure modes and how to recover.

Alert flapping

If an alert repeatedly fires and resolves, the threshold is likely too tight or the for duration too short. Increase for to 10 minutes or adjust the threshold. For example, if a container routinely spikes to 91% for 4 minutes, but is otherwise fine, a 90% threshold with for: 5m will flap. Either raise the threshold to 95% or extend for to 15 minutes.

Missing metrics due to exporter crash

If cAdvisor or the Docker metrics endpoint goes down, Prometheus will show the target as down. Set up an alert for that:

- alert: ExporterDown
  expr: up == 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Exporter {{ $labels.job }} is down"

To recover, restart the container or service and verify the target comes back up. For cAdvisor, docker restart cadvisor usually works. If it repeatedly crashes, check its logs for root cause.

Alert rule misconfiguration

If alerts do not fire when expected, check rule evaluation in the Prometheus UI under Status > Rules. Look for errors. Validate YAML syntax and reload. A common syntax error is incorrect indentation in the rules file.

False positives due to limit misread

cAdvisor may report container_spec_memory_limit_bytes as 0 for containers without an explicit limit, causing division by zero in your query. Modify the alert expression to filter out zero limits:

(container_memory_working_set_bytes / container_spec_memory_limit_bytes) > 0.9
and
container_spec_memory_limit_bytes > 0

This prevents false alarms for unlimited containers. Better yet, set limits on all containers so this edge case disappears.

Rollback monitoring changes

If a configuration change causes issues, revert to the previous configuration. For Docker daemon.json changes, restore the original file and restart Docker. For Prometheus config, revert the file and reload. Keep version-controlled backups of these configurations.

Common Pitfalls and How to Avoid Them

Over years of operating container fleets, certain mistakes appear repeatedly. Here is how to sidestep them.

Pitfall 1: Setting memory limits without monitoring working set vs RSS.

Memory limits apply to the container's total memory usage, but the kernel's OOM killer uses a broader metric than container_memory_working_set_bytes. cAdvisor's working set excludes inactive file cache, so your alert may fire after the container is already in trouble. To avoid this, also monitor container_memory_usage_bytes and set alerts at 85% of the limit for working set and 95% for total usage.

Pitfall 2: Ignoring CPU throttling.

A container may stay below its CPU limit on average but still be throttled during bursts. This causes latency spikes that users notice. Always alert on container_cpu_cfs_throttled_seconds_total with a non-zero rate over 5 minutes. If throttling is common, increase the CPU limit or adjust the application's behavior.

Pitfall 3: Trusting default alert templates without testing.

Prometheus alert expressions often look correct but fail silently because labels do not match or metrics are missing. Always test alerts with known load. Use promtool test rules to validate rule files before deploying.

Pitfall 4: Setting limits without benchmarking first.

Picking arbitrary limits like 512 MiB for a database can cause OOM kills during peak traffic. Benchmark your application under realistic load and set limits at least 20% above observed peak usage. Revisit limits after every significant code change.

Pitfall 5: Forgetting to monitor the monitoring stack itself.

If Prometheus or cAdvisor goes down, you lose visibility. Place the monitoring stack on a separate host or at least in a different failure domain, and set up external health checks.

Operations Checklist

Use this checklist for regular review and incident response.

Daily checks

• [ ] Verify Prometheus targets are up: up == 1 in Prometheus UI or via API.
• [ ] Check for any active alerts in Alertmanager. Acknowledge them if they are being worked on.
• [ ] Review the top 5 containers by memory and CPU usage: query topk(5, container_memory_working_set_bytes) and topk(5, rate(container_cpu_usage_seconds_total[5m])).
• [ ] Owner: On-call SRE (rotating).

Weekly review

• [ ] Analyze resource usage trends for all containers. Identify any container that has exceeded 80% of its limit for more than 1 hour in the past week.
• [ ] Test alerting by simulating a resource spike in a dedicated test container. Owner: Platform engineer (e.g., Jordan Lee).
• [ ] Check for updates to cAdvisor and Prometheus exporters. Plan upgrades during maintenance windows.
• [ ] Owner: Monitoring lead (Alex Morgan). Frequency: Every Monday.

Incident response steps

When a resource alert fires, follow this sequence:

1. Acknowledge the alert in Alertmanager and note the time.
2. Identify the affected container(s). Query Prometheus for the specific metric: container_memory_working_set_bytes{name="db"}.
3. Check current resource usage on the host: docker stats --no-stream to see memory, CPU, and PIDs.
4. Inspect container logs for errors: docker logs <container> --tail 100.
5. If the container is unresponsive due to OOM, restart it: docker restart <container>.
6. If the issue persists, temporarily increase resource limits: docker update --memory 1g --cpus 1 <container>.
7. After resolving, investigate root cause and adjust limits or application code. Document the incident.

Key metrics table

Conclusion

Monitoring Docker resource limits transforms a hidden risk into a managed operational practice. In this guide, we covered a full workflow: inventorying your environment, enabling metrics with Docker and cAdvisor, configuring Prometheus alerts, validating with test load, handling common failure modes, and following a daily and weekly checklist.

Start small: pick one non-critical container, apply limits, and monitor it for a week. Watch how it behaves under real traffic. Then expand to all containers, integrate alerts into your incident management tool, and make weekly reviews a habit. The effort pays off in fewer OOM surprises, fewer noisy neighbor incidents, and a more stable platform for everyone.