Docker Resource Limits: Production Operations Checklist with Practical Examples

Intro

In production, containers often compete for finite CPU, memory, and I/O. Without explicit limits, one container can degrade an entire host, causing latency spikes or outages. Docker resource limits let operators enforce boundaries, protect critical services, and maintain predictable performance.

This article provides a practical checklist for planning, configuring, verifying, and maintaining Docker resource limits in production. You will find concrete commands, configuration examples, and expected outputs to guide you.

Version and Environment Inventory

Before setting resource limits, document your environment. Confirm the Docker daemon version, storage driver, and orchestration layer (if any). Resource limit support varies by Docker version and runtime. For example, CPU limits require Docker 1.13+, while cgroup v2 support arrived in Docker 20.10.

Run the following command to verify Docker version and cgroup version:

docker info --format '{{.ServerVersion}} | cgroup driver: {{.CgroupDriver}} | cgroup version: {{.CgroupVersion}}'

Expected output (example):

20.10.12 | cgroup driver: cgroupfs | cgroup version: 1

Also check the host kernel and operating system: uname -r. Ensure you have permission to modify cgroup settings (root or sudo). Record these details for your runbook.

If using orchestrators like Kubernetes, know that Docker resource limits may be managed by the scheduler. Here we focus on Docker Engine directly, but many principles apply.

Next, identify the workloads. Categorize containers as critical, normal, or best-effort. Critical services need guaranteed resources, while best-effort can be constrained tightly. Inventory current usage with docker stats --no-stream to establish baselines.

Example output from docker stats --no-stream:

CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
f1e2d3c4b5a6   web       5.00%     150MiB / 512MiB       29.30%    1.2MB / 800kB    4.1MB / 0B       12
b7c8d9e0f1a2   db        12.00%    800MiB / 1GiB         78.13%    2.5MB / 1.8MB    10MB / 20MB      25

Use this data to set initial limits and identify which containers need attention.

Safe Configuration Path

Apply limits gradually. Start with non-critical containers in a staging environment, then expand. Define limits at container runtime using the docker run command or in Compose files with the deploy.resources section (for Swarm) or resources (for newer Compose versions).

Step 1: Set CPU Limits

CPU limits are relative weights when using the default scheduler. For hard limits, use --cpus (e.g., --cpus=1.5 allows 1.5 cores). If you need proportional weight, --cpu-shares exists, but current Docker documentation focuses on --cpus and does not cover --cpu-shares.

Hard CPU limit example:

docker run -d --name web --cpus=1.5 --memory=512m nginx

This container can use at most 1.5 CPU cores. To confirm, run docker inspect web --format '{{.HostConfig.NanoCpus}}' and you will see 1500000000 (1.5 * 1e9 nanocpus).

CPU limit example for multiple containers:

docker run -d --name app1 --cpus=0.5 myimage
docker run -d --name app2 --cpus=1.0 myimage

App1 is limited to 0.5 CPU and app2 to 1.0 CPU. When CPU contention occurs, app2 can use up to twice as much CPU time as app1. If you need relative weight instead, --cpu-shares still exists, but it is not covered in current Docker documentation.

CPU quota and period (advanced):

To set a CPU quota directly, use --cpu-quota=50000 --cpu-period=100000 for 50% of one core.

docker run -d --name batch --cpu-quota=50000 --cpu-period=100000 mybatchjob

This is equivalent to 0.5 CPU.

Step 2: Set Memory Limits

Always set memory limits to prevent OOM incidents. Use --memory (for RAM) and --memory-swap (for total memory+swap). If --memory-swap is omitted, it defaults to twice the memory limit.

Memory limit with swap disabled:

docker run -d --name api --memory=1g --memory-swap=1g myapp

This prevents the container from using swap, i.e., total memory is capped at 1 GB. If the container tries to exceed this, it will be OOM killed.

Memory limit with swap allowed:

docker run -d --name cache --memory=512m --memory-swap=1g redis

This allows 512 MB RAM plus 512 MB swap. Swap usage may degrade performance; use with caution.

Memory reservation (soft limit):

docker run -d --name web --memory=1g --memory-reservation=750m nginx

Under memory pressure, the kernel tries to reclaim memory down to the reservation, but will not OOM kill above that reservation unless the hard limit is exceeded.

Step 3: Set Blkio Limits (Optional)

For disk I/O, use --device-read-bps, --device-write-bps, --device-read-iops, --device-write-iops.

Limit write throughput on a specific device:

docker run -d --name db --device-write-bps /dev/sda:10mb postgres

This limits writes to 10 MB/s on /dev/sda. Useful for preventing a noisy neighbor from saturating disk I/O.

Limit read IOPS:

docker run -d --name logger --device-read-iops /dev/sda:100 myloggingapp

This limits reads to 100 IOPS.

Step 4: Use Compose for Repeatability

For Compose v3 (Swarm), use the deploy section:

version: "3.8"
services:
  web:
    image: nginx
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 256M
        reservations:
          cpus: '0.25'
          memory: 128M

Note: reservations are honored only in Swarm mode.

For Compose v2 (non-Swarm), use top-level service keys:

version: "2.4"
services:
  web:
    image: nginx
    mem_limit: 256m
    memswap_limit: 512m
    cpus: 0.5
    cpu_shares: 512

For Compose v1 with the docker-compose standalone tool, you can use mem_limit and cpus as well.

Example full docker-compose.yml for a production-like stack:

version: "3.8"
services:
  frontend:
    image: nginx:1.21
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 256M
    restart: always
  backend:
    image: myapp:2.0
    deploy:
      resources:
        limits:
          cpus: '1.0'
          memory: 1G
        reservations:
          cpus: '0.5'
          memory: 512M
    environment:
      - DB_HOST=db
    restart: always
  db:
    image: postgres:13
    deploy:
      resources:
        limits:
          memory: 2G
          cpus: '2.0'
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always
volumes:
  pgdata:

When deploying to Swarm, use docker stack deploy -c docker-compose.yml mystack. For non-Swarm, use docker compose up -d with a version that supports the deploy key (ignored) or use mem_limit/cpus directly.

Step 5: Update Existing Containers

Existing containers require recreation to apply new limits. Use docker update for some settings (CPU shares, memory limits) if the container is running, but not for all (e.g., --cpus). For consistent results, recreate the container with the desired flags.

Update memory limit on a running container:

docker update -m 512m mycontainer

Update CPU shares:

docker update --cpu-shares 512 mycontainer

Note: docker update cannot change --cpus, --memory-swap, or blkio settings after creation. For those, you must stop, remove, and recreate the container. Always use version-controlled configuration for recreation.

Verification and Diagnostics

After applying limits, verify they are active and observe behavior under load.

Inspect Container Limits

Use docker inspect to confirm settings:

docker inspect --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}}' container_name

Example output:

524288000 1500000000

This indicates 512 MB memory (in bytes) and 1.5 CPUs (in nanocpus, 1.5 * 1e9).

Inspect all resource-related settings:

docker inspect --format 'Memory: {{.HostConfig.Memory}}, MemorySwap: {{.HostConfig.MemorySwap}}, CpuShares: {{.HostConfig.CpuShares}}, NanoCpus: {{.HostConfig.NanoCpus}}' web

Example output:

Memory: 536870912, MemorySwap: 1073741824, CpuShares: 1024, NanoCpus: 500000000

Monitor Runtime Usage

Use docker stats to see current consumption and limits:

docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.MemPerc}}"

Expected output (hypothetical):

NAME      CPU %     MEM USAGE / LIMIT     MEM %
web       15.00%    120MiB / 512MiB       23.44%
api       0.50%     300MiB / 1GiB         29.30%
db        45.00%    1.2GiB / 2GiB         60.00%

Note the "MEM USAGE / LIMIT" column shows the applied limit. CPU % is relative to the number of cores available on the host. With a limit of 1.5 CPUs, the max CPU % shown would be 150%.

Simulate Load to Verify Enforcement

For memory, run a stress command inside the container:

docker exec -it web stress --vm 1 --vm-bytes 600M

If the memory limit is 512 MB, the container should be OOM killed. Observe that the container is killed or throttled when exceeding the limit. Check exit code and logs: docker inspect web --format '{{.State.OOMKilled}}' should output true after an OOM kill.

For CPU, run a CPU-intensive task and verify usage does not exceed the limit using docker stats. With --cpus=1.5, CPU % should max at 150% on a multicore host.

Example CPU stress test:

docker exec -it web bash -c "yes > /dev/null &"

Then observe docker stats web:

CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
f1e2d3c4b5a6   web       150.00%   150MiB / 512MiB       29.30%    1.2MB / 800kB    4.1MB / 0B       3

CPU % will be throttled at exactly 150%.

Check cgroup Parameters

Inspect the host cgroup files (requires root):

cat /sys/fs/cgroup/memory/docker/<container_id>/memory.limit_in_bytes

Should show 536870912 for 512 MB.

On cgroup v2 systems, the path may differ: /sys/fs/cgroup/system.slice/docker-<container_id>.scope/memory.max.

Failure Modes and Recovery

Understanding failure modes helps in recovery planning. Common issues include:

• OOM Kills: Container exceeds memory limit, kernel kills a process. The container may restart (if restart policy allows) or remain stopped. Mitigation: Set memory limits with headroom, monitor memory usage, and scale horizontally.
• CPU Starvation: Container is throttled excessively, causing slow response. Mitigation: Adjust --cpus or remove CPU limits for critical services.
• I/O Bottleneck: Blkio limits too low slow down database writes. Mitigation: Increase limits or separate data volumes.
• Incorrect Limits Due to Misconfiguration: Typos in flags or Compose files can lead to no limits or unintended restrictions. Mitigation: Use docker inspect to verify after creation.

OOM Kill Example

Suppose a web container has --memory=512m and --memory-swap=512m. A sudden spike causes the application to allocate more than 512 MB. The kernel OOM killer terminates the process inside the container. Docker records this in the container state:

docker inspect web --format '{{.State.OOMKilled}} {{.State.ExitCode}}'

Output:

true 137

Exit code 137 indicates the process was killed by SIGKILL (OOM). The container may restart if --restart=always is set. To diagnose, check docker logs web and host dmesg for OOM events.

Rollback Steps

To rollback changes:

• Remove or adjust limits by recreating the container without the limit flags.
• If the container was killed, restart it with previous configuration.
• In Compose, revert the resource section and run docker compose up -d again.

Example: Remove memory limit from a running container

You can use docker update -m 0 container_name but note that not all settings can be updated. Safer: recreate from image with original flags. Here's a recipe:

# Stop and remove the container
docker stop web && docker rm web
# Recreate without memory limit
docker run -d --name web --cpus=1.5 nginx

Recovery Checks

After rollback, verify:

• Container is running: docker ps shows the container.
• Resource usage is normal: docker stats shows expected values.
• Application health endpoints respond (e.g., curl http://localhost/health returns 200).
• Host-level metrics (CPU, memory, disk I/O) are within normal range.

Additionally, monitor host-level metrics to ensure no other container is affected.

Operations Checklist

Use this checklist for routine operations and reviews:

• [ ] Confirm Docker version and cgroup support (docker info).
• [ ] Inventory all production containers and their resource usage baselines.
• [ ] Define resource tiers (guaranteed, burst, best-effort) and corresponding limits.
• [ ] Apply limits using documented docker run flags or Compose files.
• [ ] Verify limits with docker inspect and docker stats.
• [ ] Test limits under load to ensure no unexpected terminations.
• [ ] Set up monitoring and alerting for resource usage (e.g., memory >80% of limit).
• [ ] Document recovery procedures for OOM kills and throttling.
• [ ] Schedule periodic reviews to adjust limits based on changing workloads.
• [ ] Keep container images and Docker Engine updated to benefit from resource management improvements.
• [ ] Train team members on resource limit configurations and troubleshooting.

Resource Tier Definition Table

Monitoring and Alerting Setup

Use a monitoring tool like Prometheus, Datadog, or cAdvisor to collect container metrics. Set alerts based on thresholds:

• Memory: Alert when container memory usage > 80% of limit for more than 5 minutes.
• CPU: Alert when container CPU throttling count increases (cgroup cpu.stat throttled time) or CPU usage is near limit for extended period.
• OOM: Alert on any OOM kill event (container exit code 137 or cgroup memory.events oom_kill count).
• Disk I/O: Alert when blkio wait time is high or throughput approaches limit.

Example Prometheus alert rule for memory:

groups:
- name: docker
  rules:
  - alert: ContainerMemoryHigh
    expr: (container_memory_usage_bytes / container_spec_memory_limit_bytes) > 0.8
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Container {{ $labels.name }} memory usage above 80%"

Maintenance Tasks

• Regularly review docker stats to identify containers approaching limits.
• Update Compose files in version control, not just production.
• After deployments, confirm new containers have limits applied automatically.
• Test restore of configuration from version control.
• Keep an audit log of resource limit changes.

Backups

Resource configurations should be stored in infrastructure-as-code (Compose files or scripts) and versioned. Back up these files along with other configurations.

Example Git workflow:

# Commit Compose file changes
git add docker-compose.yml
git commit -m "Update resource limits for db service: memory 2g, cpus 2.0"
git push origin main

Store backups in a remote repository and consider automated backups of the entire config directory.

Conclusion

Docker resource limits are essential for production stability. By following this checklist, you can systematically apply, verify, and maintain limits to prevent resource contention and outages. Start with a pilot on a small set of containers, measure the impact, and gradually expand. Document your configurations and keep them under version control. With these practices, your containerized applications will run predictably and efficiently.

Remember: resource limits are not set-and-forget. Continuously monitor, review, and adjust based on real workload patterns. A solid resource management strategy is a key part of a reliable container platform.