Docker Resource Limits Architecture Explained with Practical Examples

Intro

Docker resource limits are a critical operational control for running containers reliably and cost-effectively. When you set a memory limit of 512 MB, Docker enforces it through the Linux kernel's cgroups, but how that enforcement behaves depends on the resource type, the container runtime, and the host configuration. This article explains the architecture behind Docker resource limits and gives you practical commands to observe, configure, and verify them.

We will cover:

• How Docker interacts with cgroups v1 and v2 to enforce limits.
• CPU limits, including shares, quotas, and CPU sets.
• Memory limits, including OOM behavior and swap.
• Disk I/O and PID limits.
• How to inspect current limits and configure them via Docker CLI and Compose.
• Failure modes, recovery steps, and an operations checklist.

Throughout, we use concrete commands and expected outputs so you can follow along on your own host.

Prerequisites and Environment Inventory

Before changing any resource limit, know your environment.

Docker Version and Cgroup Driver

Resource limit behavior depends on the Docker version and the cgroup driver configured on the host. Run:

docker version --format '{{.Server.Version}}'

Expected output example: 24.0.5

Check the cgroup driver:

docker info --format '{{.CgroupDriver}} {{.CgroupVersion}}'

Expected output example: systemd 2

If you see cgroupfs and cgroup version 1, some newer features like memory.high may not be available. Most modern distributions default to cgroup v2 with the systemd driver.

Host Kernel and Distribution

Confirm the kernel supports cgroup v2:

grep cgroup /proc/filesystems

Expected output includes nodev cgroup2.

Identify Running Containers and Their Current Limits

List running containers:

docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

Then inspect limits for a specific container, e.g., web:

docker inspect web --format '{{json .HostConfig.Resources}}'

Expected output example:

{"CpusetCpus":"","CpuShares":1024,"Memory":0,"NanoCpus":0,"MemorySwap":0,"PidsLimit":0}

Here Memory: 0 means no memory limit, and NanoCpus: 0 means no CPU quota.

Verify cgroup Files for a Container

Find the cgroup path for the container:

docker inspect web --format '{{.HostConfig.CgroupParent}} {{.Id}}'

Then, using the container ID, look at its cgroup directory (example for cgroup v2):

cat /sys/fs/cgroup/system.slice/docker-<container-id>.scope/memory.max

If no limit is set, this file contains max.

Safe Configuration Path

Step 1: Capture Baseline Metrics

Before changing limits, measure current usage. Use docker stats:

docker stats --no-stream web

Expected output example:

CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
c2f2...        web       0.15%     120MiB / 1.945GiB     6.03%     1.2kB / 0B       0B / 0B          7

Note the MEM USAGE / LIMIT column. If LIMIT shows the host's total memory, the container is unlimited.

Step 2: Choose the Right Limit Type

• CPU: Use CPU shares for relative weighting when multiple containers compete. Use a quota (NanoCpus) for an absolute limit. Use cpuset to pin to specific cores.
• Memory: Set a hard limit (--memory) and optionally a swap limit (--memory-swap). Understand OOM behavior.
• Disk I/O: Use blkio limits on cgroup v1; on cgroup v2, use --device-read-bps and --device-write-bps for block devices.
• PIDs: Limit the number of processes to prevent fork bombs with --pids-limit.

Step 3: Apply Limits via Docker Run

Example: run a container with 512 MB memory, 0.5 CPU quota, and 100 PIDs:

docker run -d --name web \
  --memory 512m \
  --memory-swap 1g \
  --cpus 0.5 \
  --pids-limit 100 \
  nginx:alpine

Verify:

docker inspect web --format '{{json .HostConfig.Resources}}'

Expected output includes "Memory":536870912, "NanoCpus":500000000, "PidsLimit":100.

Step 4: Apply Limits via Docker Compose

In docker-compose.yml:

services:
  web:
    image: nginx:alpine
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 512M
        reservations:
          memory: 128M
    pids_limit: 100

Note: The deploy key is only honored by Docker Swarm. For docker compose (non-Swarm), use the resources key at the service level (Compose v2.24+ supports limits and reservations under resources). Alternatively, use mem_limit and cpus for older compatibility:

services:
  web:
    image: nginx:alpine
    mem_limit: 512m
    cpus: 0.5
    pids_limit: 100

Apply with docker compose up -d.

Verification and Diagnostics

After setting limits, verify enforcement from inside and outside the container.

Inside the Container: Check Visible Memory

The container sees the host's memory, not the limit, unless you set a memory reservation. This confuses many. Run:

docker exec web cat /proc/meminfo | head -3

Expected output shows host total memory, not 512 MB. To see the cgroup limit, read:

docker exec web cat /sys/fs/cgroup/memory.max

Expected output: 536870912 (512 MB in bytes).

Stress Test Memory Limit

Use a tool like stress inside the container:

docker exec web sh -c "apk add --no-cache stress-ng && stress-ng --vm 1 --vm-bytes 600M --timeout 30s"

After a few seconds, check if the process is OOM-killed:

docker inspect web --format '{{.State.OOMKilled}} {{.State.ExitCode}}'

Expected output: true 137 (137 = 128 + 9, SIGKILL).

Inspect cgroup Events

On cgroup v2, monitor memory.events:

docker exec web cat /sys/fs/cgroup/memory.events

Look for oom and oom_kill counters increasing.

CPU Throttling Verification

To test CPU quota, run a CPU-intensive task and observe throttling:

docker run --rm --cpus 0.5 alpine sh -c "time crunch 100000000000"

Check docker stats CPU% should stay near 50%.

For cgroup v2, read cpu.stat:

docker exec web cat /sys/fs/cgroup/cpu.stat | grep throttled

Expected output includes nr_throttled and throttled_time.

Failure Modes and Recovery

OOM Kills Inside Container

If the application inside the container is killed, the container may exit or restart. Check:

docker inspect web --format '{{.State.OOMKilled}}'

If true, increase memory limit or reduce application footprint. Note: if --memory-swap is not set, swap is disabled and the limit equals --memory. To allow swap, set --memory-swap > --memory.

Host-level OOM

If the host runs out of memory, the kernel may kill other processes. Use dmesg to see OOM kills:

dmesg | grep -i oom

Recovery: add host memory, reduce container limits, or adjust overcommit settings.

CPU Starvation

If a container is throttled excessively, application performance degrades. Check nr_throttled in cpu.stat; if high, consider increasing --cpus or moving to cpuset.

Disk I/O Limits Not Applied

On some storage drivers, blkio limits may not work. Verify with docker info | grep Storage Driver. For overlay2, blkio limits may be ignored; use device-specific limits or switch to cgroup v2.

PIDs Limit Exceeded

If the container hits the PIDs limit, fork fails: "Resource temporarily unavailable". Increase --pids-limit if legitimate, else fix the application.

Common Pitfalls and How to Avoid Them

1. Assuming the container sees its memory limit: Many apps read /proc/meminfo and think they have the host's memory, causing OOM. Solution: set environment variables or use cgroup-aware libraries.
2. Setting CPU shares instead of quota: CPU shares are relative and do not cap usage when the CPU is idle. Use --cpus for hard limits.
3. Not setting --memory-swap: If you set --memory without --memory-swap, swap is disabled, and the container cannot use swap. This may surprise applications that expect swap.
4. Ignoring PID limits: A fork bomb can exhaust the host's PID space. Always set --pids-limit for untrusted containers.
5. Not using cgroup v2: cgroup v2 provides better isolation and features like memory.high for soft limits. Migrate if possible.
6. Overcommitting host resources: Summing container limits can exceed host capacity. Monitor overall usage and set appropriate reservations.

Operations Checklist

Use this checklist before and after changing Docker resource limits.

Conclusion

Docker resource limits are essential for stable multi-tenant environments. By understanding the underlying cgroup mechanisms and using the practical commands provided here, you can set appropriate CPU, memory, I/O, and PID limits, diagnose enforcement issues, and recover from failures. Always start with observation, make incremental changes, and verify with concrete metrics.

Next step: select one container in your environment, follow the Safe Configuration Path to set a memory limit, verify with a stress test, and document the results. Then adopt the Operations Checklist to maintain limits over time.