Intro
Docker containers share the host kernel and, by default, can consume as much CPU and memory as the host allows. Without explicit resource limits, a single container can starve other containers or even the host operating system, leading to performance degradation or outages. Configuring resource limits is essential for predictable performance and stability, but many teams make mistakes that either fail to enforce limits or over-restrict containers, causing crashes or poor behavior.
This guide walks through common Docker resource limit configuration mistakes, shows how to set limits correctly with practical examples, and provides validation, rollback, and troubleshooting workflows. By following these steps, you can avoid resource contention issues and ensure your containers run reliably in production.
Version and Environment Inventory
Before applying resource limits, document your Docker version, host configuration, and container topology. This inventory helps reproduce issues and understand the impact of changes.
Check Docker Version and Info
Run the following commands to verify Docker version and server details:
docker version
Expected output includes client and server versions. For example:
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:08:17 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.7
Confirm your Docker version supports the resource limit flags (--cpus, --memory). These flags are available since Docker 1.13 and later. If you are on an older version, upgrade first.
docker info
Check the Cgroup Driver and Cgroup Version in the output. Resource limits rely on cgroups. Example snippet:
Cgroup Driver: systemd
Cgroup Version: 2
Host Resource Inventory
List the host's CPU and memory to understand available capacity:
lscpu | grep -E '^CPU\(s\):|^Model name'
free -h
Example output:
CPU(s): 8
Model name: Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
total used free shared buff/cache available
Mem: 31Gi 2.1Gi 27Gi 123Mi 1.6Gi 28Gi
Swap: 2.0Gi 0B 2.0Gi
Container Topology
List running containers and their current resource usage:
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
docker stats --no-stream
docker stats shows CPU and memory usage per container. This baseline helps decide appropriate limits.
Prerequisites
- Docker Engine with cgroups v1 or v2 enabled.
- Root or sudo access to the Docker host for applying limits.
- A test environment to validate changes before production.
Safe Configuration Path
Setting resource limits incorrectly can hurt performance or cause containers to be killed. Follow a scoped approach: start with a single container in a test environment, measure its behavior, then gradually adjust.
Common Mistake 1: Not Setting Any Limits
Without limits, a container can consume all CPU and memory. To set limits for a new container:
docker run -d --name web --cpus="1.5" --memory="512m" nginx
This command restricts the container to 1.5 CPU cores and 512 MB of memory.
Common Mistake 2: Misunderstanding CPU Limits
Using --cpus sets a quota on CPU time. For example, --cpus="1.5" limits the container to 150% of one CPU core. If you need to pin to specific cores, use --cpuset-cpus:
docker run -d --name cpu-pinned --cpuset-cpus="0-1" nginx
However, using --cpuset-cpus without --cpus still allows the container to use 100% of the pinned cores. Combine both for strict control:
docker run -d --name cpu-limited --cpus="1.0" --cpuset-cpus="0-1" nginx
This limits to one CPU's worth of time across cores 0 and 1.
Common Mistake 3: Setting Memory Limit Too Low
If a container exceeds its memory limit, the kernel OOM (out-of-memory) killer terminates the process. Set memory limits based on actual usage with headroom. Use --memory-reservation as a soft limit that Docker uses for scheduling but does not hard-kill:
docker run -d --name app --memory="1g" --memory-reservation="750m" myapp
This sets a hard limit of 1 GB and a soft reservation of 750 MB.
Common Mistake 4: Forgetting Swap Limits
By default, a container can use as much swap as the host has available. To control swap, set --memory-swap in conjunction with --memory. For example, to allow no swap:
docker run -d --name no-swap --memory="512m" --memory-swap="512m" myapp
Here, --memory-swap equals --memory, so no swap is allowed. To allow 1 GB total (memory + swap), set --memory-swap="1g":
docker run -d --name with-swap --memory="512m" --memory-swap="1g" myapp
Common Mistake 5: Setting Limits Only at Container Start
Limits can also be set in Docker Compose or via update. For existing containers, you can update limits with:
docker update --cpus="2.0" --memory="1g" web
Note: You cannot update --memory-swap on a running container; you must recreate it.
Docker Compose Example
In docker-compose.yml, specify limits under deploy.resources.limits (for Swarm) or mem_limit and cpus (for Compose v2/v3). Use the following for Compose v3 with Swarm-compatible syntax:
version: '3.8'
services:
web:
image: nginx
deploy:
resources:
limits:
cpus: '1.5'
memory: 512M
reservations:
cpus: '0.5'
memory: 256M
Deploy with docker stack deploy -c docker-compose.yml mystack if using Swarm, or use docker compose up (Compose v2) which also respects these limits.
Step-by-Step Safe Implementation
- Measure baseline: Run the container without limits and monitor
docker statsfor typical and peak usage. - Apply conservative limits: Start with limits 20-30% above observed peak.
- Load test: Simulate realistic traffic and monitor for throttling or OOM kills.
- Adjust gradually: Increase or decrease limits based on observed behavior.
- Document and version control: Store configuration in Docker Compose or Kubernetes manifests.
Verification and Diagnostics
After setting limits, verify they are enforced and diagnose any issues.
Inspect Container Configuration
Use docker inspect to see the applied limits:
docker inspect web --format '{{.HostConfig.NanoCpus}} {{.HostConfig.Memory}}'
For example, output:
1500000000 536870912
NanoCpus is in billionths of a CPU (1.5 x 1e9 = 1500000000), and Memory is in bytes (512 MB = 536870912).
Check cgroup Limits Directly
Access the container's cgroup files on the host. For cgroup v2, the path is /sys/fs/cgroup/system.slice/docker-<container-id>.scope/. Find the container ID with docker ps --no-trunc. Then check:
cat /sys/fs/cgroup/system.slice/docker-<container-id>.scope/cpu.max
Example output for a limit of 1.5 CPUs:
150000 100000
This means the container can use 150000 microseconds of CPU time per 100000 microseconds period.
For memory:
cat /sys/fs/cgroup/system.slice/docker-<container-id>.scope/memory.max
Output:
536870912
Monitor Runtime Statistics
Use docker stats to observe live resource usage:
docker stats web
Output includes CPU % and MEM USAGE / LIMIT. Check that MEM USAGE stays below the limit.
Test CPU Throttling
To see if CPU is being throttled, check the nr_throttled counter in the cgroup:
cat /sys/fs/cgroup/system.slice/docker-<container-id>.scope/cpu.stat | grep nr_throttled
If the value is increasing, the container is hitting its CPU limit and being throttled. Consider raising the limit.
Test Memory OOM
Deliberately exceed the memory limit to see OOM behavior. Run a container with a small memory limit and a command that allocates memory:
docker run --rm --memory="50m" alpine sh -c 'tail /dev/zero'
This command reads from /dev/zero indefinitely, consuming memory. After a few seconds, the kernel OOM killer will terminate the process. Check the container exit code:
docker run --rm --memory="50m" alpine sh -c 'tail /dev/zero'; echo $?
Expected exit code is 137 (128 + 9, SIGKILL), indicating OOM kill.
Failure Modes and Recovery
Resource limits can cause failures if misconfigured. Understand common failure modes and how to recover.
Failure Mode 1: Container OOM Killed
Symptom: Container stops unexpectedly, exit code 137, logs show OOM kill.
Diagnosis: Check docker inspect for State.OOMKilled:
docker inspect web --format '{{.State.OOMKilled}}'
If true, the container exceeded its memory limit.
Recovery: Increase memory limit or reduce memory usage. Update the limit:
docker update --memory="1g" web
If the container crashed, restart it with corrected settings.
Failure Mode 2: CPU Starvation
Symptom: Application is slow, high latency, but CPU usage on host is not maxed. nr_throttled increases.
Diagnosis: Check cgroup CPU throttling as shown earlier.
Recovery: Increase CPU limit:
docker update --cpus="2.0" web
Failure Mode 3: Incorrect Swap Configuration
Symptom: Container uses excessive swap, causing disk I/O and slowdown.
Diagnosis: Check swap usage in docker stats (if available) or in cgroup memory.swap.current.
Recovery: Recreate the container with appropriate --memory-swap:
docker stop web && docker rm web
docker run -d --name web --memory="512m" --memory-swap="512m" nginx
Failure Mode 4: Limit Not Applied
Symptom: Container uses more resources than expected, docker inspect shows no limits.
Cause: Limits specified in a compose file but not applied due to using docker-compose up with unsupported fields, or forgetting to deploy the stack.
Diagnosis: Inspect the running container:
docker inspect web --format '{{.HostConfig.NanoCpus}} {{.HostConfig.Memory}}'
If values are 0 0, no limits are set.
Recovery: Apply limits explicitly via docker update or recreate with correct flags.
Rollback and Recovery Best Practices
- Keep configuration in version control.
- Use
docker updatefor quick limit changes without recreating the container. - For swap changes, recreate the container with a script or compose file.
- Test rollback by restoring previous limits and verifying with
docker stats.
Common Pitfalls and How to Avoid Them
Besides the mistakes already described, here are additional pitfalls that often trip up teams:
Pitfall 1: Ignoring CPU Throttling Metrics
Many teams set CPU limits but never check whether the container is being throttled. Throttling can cause latency spikes even if CPU usage appears below the limit. Why it happens: The container hits the CPU quota within a scheduling period and is forced to wait. How to avoid: Regularly monitor nr_throttled in cgroup cpu.stat. If it increases under normal load, raise the CPU limit or adjust the period (advanced).
Pitfall 2: Not Setting Memory Reservation
A memory hard limit alone can lead to aggressive swapping or OOM kills under memory pressure. Why it happens: The kernel may reclaim pages from the container, causing performance degradation before the hard limit is reached. How to avoid: Set a --memory-reservation below the hard limit to signal a softer boundary. This helps the kernel make better decisions.
Pitfall 3: Overcommitting Host Resources
If the sum of container limits exceeds host capacity, the host may become unstable. Why it happens: Teams assume limits are reservations, but they are only caps; containers can still compete for resources. How to avoid: Use docker stats and host monitoring to ensure total limits do not exceed 80-90% of host capacity. Leave headroom for the host OS and system daemons.
Pitfall 4: Forgetting to Update Compose Files
After manually adjusting limits with docker update, teams forget to update the source-of-truth compose file. Why it happens: Quick fixes in production lead to configuration drift. How to avoid: Always update the compose file or manifest first, then apply changes via a deployment pipeline. Use version control to track changes.
Operations Checklist
Use this checklist for ongoing management of Docker resource limits. Assign a single owner (e.g., DevOps Engineer) for each recurring task and review frequency.
| Task | Command or Action | Owner | Frequency |
|---|---|---|---|
| Review container resource usage | docker stats --no-stream | SRE on call | Weekly |
| Check for OOM kills | docker inspect --format '{{.State.OOMKilled}}' <container> | Application owner | After incidents |
| Verify limits in configuration | docker inspect --format '{{.HostConfig.NanoCpus}} {{.HostConfig.Memory}}' <container> | DevOps lead | Monthly |
| Audit throttling | Check cpu.stat nr_throttled per container | Performance engineer | Monthly |
| Update limits based on trends | Adjust --cpus and --memory as needed | Service owner | Quarterly |
| Test OOM behavior in staging | Run memory stress test in non-production | QA lead | Before major releases |
| Document limits in CMDB | Maintain a record of resource allocations | Configuration manager | Continuously |
Additional Considerations
- Set default limits for daemon: You can configure some default resource limits in
/etc/docker/daemon.jsonusingdefault-ulimits, but CPU and memory defaults are not directly supported. Use orchestration policies or templates to enforce defaults. - Use orchestrator features: If using Kubernetes, set resource requests and limits in pod specs. For Docker Swarm, use service constraints and resource reservations.
- Monitor with tools: Integrate
docker statsoutput into monitoring systems like Prometheus via cAdvisor or node exporter for alerting on resource usage and throttling.
Conclusion
Configuring Docker resource limits is a critical practice for maintaining performance and stability. By understanding common mistakes such as neglecting limits, misunderstanding CPU semantics, setting memory too low, and ignoring swap, you can avoid production issues. The safe configuration path involves measuring baseline usage, applying conservative limits, load testing, and gradually adjusting. Verification through docker inspect, cgroup files, and docker stats ensures limits are effective. When failures occur, quick recovery is possible with docker update or container recreation. Finally, an operations checklist with clear ownership and review frequencies helps keep resource management consistent over time.
Start with one container, apply the examples from this guide, and measure the impact. Then expand to your entire fleet with confidence.