## Intro In production, containers often compete for finite CPU, memory, and I/O. Without explicit limits, one container can degrade an entire host, causing latency spikes or outages. Docker resource limits let operators enforce boundaries, protect critical services, and maintain predictable performance. This article provides a practical checklist for planning, configuring, verifying, and maintaining Docker resource limits in production. You will find concrete commands, configuration examples, and expected outputs to guide you. ## Version and Environment Inventory Before setting resource limits, document your environment. Confirm the Docker daemon version, storage driver, and orchestration layer (if any). Resource limit support varies by Docker version and runtime. For example, CPU limits require Docker 1.13+, while cgroup v2 support arrived in Docker 20.10. Run the following command to verify Docker version and cgroup version: ```bash docker info --format '{{.ServerVersion}} | cgroup driver: {{.CgroupDriver}} | cgroup version: {{.CgroupVersion}}' ``` Expected output (example): ``` 20.10.12 | cgroup driver: cgroupfs | cgroup version: 1 ``` Also check the host kernel and operating system: `uname -r`. Ensure you have permission to modify cgroup settings (root or sudo). Record these details for your runbook. If using orchestrators like Kubernetes, know that Docker resource limits may be managed by the scheduler. Here we focus on Docker Engine directly, but many principles apply. Next, identify the workloads. Categorize containers as critical, normal, or best-effort. Critical services need guaranteed resources, while best-effort can be constrained tightly. Inventory current usage with `docker stats --no-stream` to establish baselines. Example output from `docker stats --no-stream`: ``` CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS f1e2d3c4b5a6 web 5.00% 150MiB / 512MiB 29.30% 1.2MB / 800kB 4.1MB / 0B 12 b7c8d9e0f1a2 db 12.00% 800MiB / 1GiB 78.13% 2.5MB / 1.8MB 10MB / 20MB 25 ``` Use this data to set initial limits and identify which containers need attention. ## Safe Configuration Path Apply limits gradually. Start with non-critical containers in a staging environment, then expand. Define limits at container runtime using the `docker run` command or in Compose files with the `deploy.resources` section (for Swarm) or `resources` (for newer Compose versions). ### Step 1: Set CPU Limits CPU limits are relative weights when using the default scheduler. For hard limits, use `--cpus` (e.g., `--cpus=1.5` allows 1.5 cores). If you need proportional weight, `--cpu-shares` exists, but current Docker documentation focuses on `--cpus` and does not cover `--cpu-shares`. **Hard CPU limit example:** ```bash docker run -d --name web --cpus=1.5 --memory=512m nginx ``` This container can use at most 1.5 CPU cores. To confirm, run `docker inspect web --format '{{.HostConfig.NanoCpus}}'` and you will see `1500000000` (1.5 * 1e9 nanocpus). **CPU limit example for multiple containers:** ```bash docker run -d --name app1 --cpus=0.5 myimage docker run -d --name app2 --cpus=1.0 myimage ``` App1 is limited to 0.5 CPU and app2 to 1.0 CPU. When CPU contention occurs, app2 can use up to twice as much CPU time as app1. If you need relative weight instead, `--cpu-shares` still exists, but it is not covered in current Docker documentation. **CPU quota and period (advanced):** To set a CPU quota directly, use `--cpu-quota=50000 --cpu-period=100000` for 50% of one core. ```bash docker run -d --name batch --cpu-quota=50000 --cpu-period=100000 mybatchjob ``` This is equivalent to 0.5 CPU. ### Step 2: Set Memory Limits Always set memory limits to prevent OOM incidents. Use `--memory` (for RAM) and `--memory-swap` (for total memory+swap). If `--memory-swap` is omitted, it defaults to twice the memory limit. **Memory limit with swap disabled:** ```bash docker run -d --name api --memory=1g --memory-swap=1g myapp ``` This prevents the container from using swap, i.e., total memory is capped at 1 GB. If the container tries to exceed this, it will be OOM killed. **Memory limit with swap allowed:** ```bash docker run -d --name cache --memory=512m --memory-swap=1g redis ``` This allows 512 MB RAM plus 512 MB swap. Swap usage may degrade performance; use with caution. **Memory reservation (soft limit):** ```bash docker run -d --name web --memory=1g --memory-reservation=750m nginx ``` Under memory pressure, the kernel tries to reclaim memory down to the reservation, but will not OOM kill above that reservation unless the hard limit is exceeded. ### Step 3: Set Blkio Limits (Optional) For disk I/O, use `--device-read-bps`, `--device-write-bps`, `--device-read-iops`, `--device-write-iops`. **Limit write throughput on a specific device:** ```bash docker run -d --name db --device-write-bps /dev/sda:10mb postgres ``` This limits writes to 10 MB/s on /dev/sda. Useful for preventing a noisy neighbor from saturating disk I/O. **Limit read IOPS:** ```bash docker run -d --name logger --device-read-iops /dev/sda:100 myloggingapp ``` This limits reads to 100 IOPS. ### Step 4: Use Compose for Repeatability **For Compose v3 (Swarm), use the `deploy` section:** ```yaml version: "3.8" services: web: image: nginx deploy: resources: limits: cpus: '0.5' memory: 256M reservations: cpus: '0.25' memory: 128M ``` Note: `reservations` are honored only in Swarm mode. **For Compose v2 (non-Swarm), use top-level service keys:** ```yaml version: "2.4" services: web: image: nginx mem_limit: 256m memswap_limit: 512m cpus: 0.5 cpu_shares: 512 ``` For Compose v1 with the `docker-compose` standalone tool, you can use `mem_limit` and `cpus` as well. **Example full `docker-compose.yml` for a production-like stack:** ```yaml version: "3.8" services: frontend: image: nginx:1.21 deploy: resources: limits: cpus: '0.5' memory: 256M restart: always backend: image: myapp:2.0 deploy: resources: limits: cpus: '1.0' memory: 1G reservations: cpus: '0.5' memory: 512M environment: - DB_HOST=db restart: always db: image: postgres:13 deploy: resources: limits: memory: 2G cpus: '2.0' volumes: - pgdata:/var/lib/postgresql/data restart: always volumes: pgdata: ``` When deploying to Swarm, use `docker stack deploy -c docker-compose.yml mystack`. For non-Swarm, use `docker compose up -d` with a version that supports the `deploy` key (ignored) or use `mem_limit`/`cpus` directly. ### Step 5: Update Existing Containers Existing containers require recreation to apply new limits. Use `docker update` for some settings (CPU shares, memory limits) if the container is running, but not for all (e.g., `--cpus`). For consistent results, recreate the container with the desired flags. **Update memory limit on a running container:** ```bash docker update -m 512m mycontainer ``` **Update CPU shares:** ```bash docker update --cpu-shares 512 mycontainer ``` **Note:** `docker update` cannot change `--cpus`, `--memory-swap`, or blkio settings after creation. For those, you must stop, remove, and recreate the container. Always use version-controlled configuration for recreation. ## Verification and Diagnostics After applying limits, verify they are active and observe behavior under load. ### Inspect Container Limits Use `docker inspect` to confirm settings: ```bash docker inspect --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}}' container_name ``` Example output: ``` 524288000 1500000000 ``` This indicates 512 MB memory (in bytes) and 1.5 CPUs (in nanocpus, 1.5 * 1e9). **Inspect all resource-related settings:** ```bash docker inspect --format 'Memory: {{.HostConfig.Memory}}, MemorySwap: {{.HostConfig.MemorySwap}}, CpuShares: {{.HostConfig.CpuShares}}, NanoCpus: {{.HostConfig.NanoCpus}}' web ``` Example output: ``` Memory: 536870912, MemorySwap: 1073741824, CpuShares: 1024, NanoCpus: 500000000 ``` ### Monitor Runtime Usage Use `docker stats` to see current consumption and limits: ```bash docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}" ``` Expected output (hypothetical): ``` NAME CPU % MEM USAGE / LIMIT MEM % web 15.00% 120MiB / 512MiB 23.44% api 0.50% 300MiB / 1GiB 29.30% db 45.00% 1.2GiB / 2GiB 60.00% ``` Note the "MEM USAGE / LIMIT" column shows the applied limit. CPU % is relative to the number of cores available on the host. With a limit of 1.5 CPUs, the max CPU % shown would be 150%. ### Simulate Load to Verify Enforcement For memory, run a stress command inside the container: ```bash docker exec -it web stress --vm 1 --vm-bytes 600M ``` If the memory limit is 512 MB, the container should be OOM killed. Observe that the container is killed or throttled when exceeding the limit. Check exit code and logs: `docker inspect web --format '{{.State.OOMKilled}}'` should output `true` after an OOM kill. For CPU, run a CPU-intensive task and verify usage does not exceed the limit using `docker stats`. With `--cpus=1.5`, CPU % should max at 150% on a multicore host. **Example CPU stress test:** ```bash docker exec -it web bash -c "yes > /dev/null &" ``` Then observe `docker stats web`: ``` CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS f1e2d3c4b5a6 web 150.00% 150MiB / 512MiB 29.30% 1.2MB / 800kB 4.1MB / 0B 3 ``` CPU % will be throttled at exactly 150%. ### Check cgroup Parameters Inspect the host cgroup files (requires root): ```bash cat /sys/fs/cgroup/memory/docker//memory.limit_in_bytes ``` Should show `536870912` for 512 MB. On cgroup v2 systems, the path may differ: `/sys/fs/cgroup/system.slice/docker-.scope/memory.max`. ## Failure Modes and Recovery Understanding failure modes helps in recovery planning. Common issues include: - **OOM Kills**: Container exceeds memory limit, kernel kills a process. The container may restart (if restart policy allows) or remain stopped. Mitigation: Set memory limits with headroom, monitor memory usage, and scale horizontally. - **CPU Starvation**: Container is throttled excessively, causing slow response. Mitigation: Adjust `--cpus` or remove CPU limits for critical services. - **I/O Bottleneck**: Blkio limits too low slow down database writes. Mitigation: Increase limits or separate data volumes. - **Incorrect Limits Due to Misconfiguration**: Typos in flags or Compose files can lead to no limits or unintended restrictions. Mitigation: Use `docker inspect` to verify after creation. ### OOM Kill Example Suppose a web container has `--memory=512m` and `--memory-swap=512m`. A sudden spike causes the application to allocate more than 512 MB. The kernel OOM killer terminates the process inside the container. Docker records this in the container state: ```bash docker inspect web --format '{{.State.OOMKilled}} {{.State.ExitCode}}' ``` Output: ``` true 137 ``` Exit code 137 indicates the process was killed by SIGKILL (OOM). The container may restart if `--restart=always` is set. To diagnose, check `docker logs web` and host dmesg for OOM events. ### Rollback Steps To rollback changes: - Remove or adjust limits by recreating the container without the limit flags. - If the container was killed, restart it with previous configuration. - In Compose, revert the resource section and run `docker compose up -d` again. **Example: Remove memory limit from a running container** You can use `docker update -m 0 container_name` but note that not all settings can be updated. Safer: recreate from image with original flags. Here's a recipe: ```bash # Stop and remove the container docker stop web && docker rm web # Recreate without memory limit docker run -d --name web --cpus=1.5 nginx ``` ### Recovery Checks After rollback, verify: - Container is running: `docker ps` shows the container. - Resource usage is normal: `docker stats` shows expected values. - Application health endpoints respond (e.g., `curl http://localhost/health` returns 200). - Host-level metrics (CPU, memory, disk I/O) are within normal range. Additionally, monitor host-level metrics to ensure no other container is affected. ## Operations Checklist Use this checklist for routine operations and reviews: - [ ] Confirm Docker version and cgroup support (`docker info`). - [ ] Inventory all production containers and their resource usage baselines. - [ ] Define resource tiers (guaranteed, burst, best-effort) and corresponding limits. - [ ] Apply limits using documented `docker run` flags or Compose files. - [ ] Verify limits with `docker inspect` and `docker stats`. - [ ] Test limits under load to ensure no unexpected terminations. - [ ] Set up monitoring and alerting for resource usage (e.g., memory >80% of limit). - [ ] Document recovery procedures for OOM kills and throttling. - [ ] Schedule periodic reviews to adjust limits based on changing workloads. - [ ] Keep container images and Docker Engine updated to benefit from resource management improvements. - [ ] Train team members on resource limit configurations and troubleshooting. ### Resource Tier Definition Table | Tier | Example Services | CPU Limit (--cpus) | Memory Limit | CPU Shares | Notes | |------------|------------------------|--------------------|--------------|------------|-------| | Critical | PostgreSQL, Redis | 2.0 | 2 GB | 2048 | Guaranteed resources; no compromise | | Normal | Web frontend, API | 0.5 | 512 MB | 1024 | Adequate for typical load | | Best-effort| Batch jobs, analytics | 0.25 | 256 MB | 512 | May be throttled; can be killed under pressure | ### Monitoring and Alerting Setup Use a monitoring tool like Prometheus, Datadog, or cAdvisor to collect container metrics. Set alerts based on thresholds: - **Memory**: Alert when container memory usage > 80% of limit for more than 5 minutes. - **CPU**: Alert when container CPU throttling count increases (cgroup `cpu.stat` throttled time) or CPU usage is near limit for extended period. - **OOM**: Alert on any OOM kill event (container exit code 137 or cgroup `memory.events` oom_kill count). - **Disk I/O**: Alert when blkio wait time is high or throughput approaches limit. Example Prometheus alert rule for memory: ```yaml groups: - name: docker rules: - alert: ContainerMemoryHigh expr: (container_memory_usage_bytes / container_spec_memory_limit_bytes) > 0.8 for: 5m labels: severity: warning annotations: summary: "Container {{ $labels.name }} memory usage above 80%" ``` ### Maintenance Tasks - Regularly review `docker stats` to identify containers approaching limits. - Update Compose files in version control, not just production. - After deployments, confirm new containers have limits applied automatically. - Test restore of configuration from version control. - Keep an audit log of resource limit changes. ### Backups Resource configurations should be stored in infrastructure-as-code (Compose files or scripts) and versioned. Back up these files along with other configurations. **Example Git workflow:** ```bash # Commit Compose file changes git add docker-compose.yml git commit -m "Update resource limits for db service: memory 2g, cpus 2.0" git push origin main ``` Store backups in a remote repository and consider automated backups of the entire config directory. ## Conclusion Docker resource limits are essential for production stability. By following this checklist, you can systematically apply, verify, and maintain limits to prevent resource contention and outages. Start with a pilot on a small set of containers, measure the impact, and gradually expand. Document your configurations and keep them under version control. With these practices, your containerized applications will run predictably and efficiently. Remember: resource limits are not set-and-forget. Continuously monitor, review, and adjust based on real workload patterns. A solid resource management strategy is a key part of a reliable container platform.