## Intro

Docker resource limits are a critical operational control for running containers reliably and cost-effectively. When you set a memory limit of 512 MB, Docker enforces it through the Linux kernel's cgroups, but how that enforcement behaves depends on the resource type, the container runtime, and the host configuration. This article explains the architecture behind Docker resource limits and gives you practical commands to observe, configure, and verify them.

We will cover:

- How Docker interacts with cgroups v1 and v2 to enforce limits.
- CPU limits, including shares, quotas, and CPU sets.
- Memory limits, including OOM behavior and swap.
- Disk I/O and PID limits.
- How to inspect current limits and configure them via Docker CLI and Compose.
- Failure modes, recovery steps, and an operations checklist.

Throughout, we use concrete commands and expected outputs so you can follow along on your own host.

## Prerequisites and Environment Inventory

Before changing any resource limit, know your environment.

### Docker Version and Cgroup Driver

Resource limit behavior depends on the Docker version and the cgroup driver configured on the host. Run:

```bash
docker version --format '{{.Server.Version}}'
```

Expected output example: `24.0.5`

Check the cgroup driver:

```bash
docker info --format '{{.CgroupDriver}} {{.CgroupVersion}}'
```

Expected output example: `systemd 2`

If you see `cgroupfs` and cgroup version 1, some newer features like memory.high may not be available. Most modern distributions default to cgroup v2 with the systemd driver.

### Host Kernel and Distribution

Confirm the kernel supports cgroup v2:

```bash
grep cgroup /proc/filesystems
```

Expected output includes `nodev cgroup2`.

### Identify Running Containers and Their Current Limits

List running containers:

```bash
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
```

Then inspect limits for a specific container, e.g., `web`:

```bash
docker inspect web --format '{{json .HostConfig.Resources}}'
```

Expected output example:

```json
{"CpusetCpus":"","CpuShares":1024,"Memory":0,"NanoCpus":0,"MemorySwap":0,"PidsLimit":0}
```

Here `Memory: 0` means no memory limit, and `NanoCpus: 0` means no CPU quota.

### Verify cgroup Files for a Container

Find the cgroup path for the container:

```bash
docker inspect web --format '{{.HostConfig.CgroupParent}} {{.Id}}'
```

Then, using the container ID, look at its cgroup directory (example for cgroup v2):

```bash
cat /sys/fs/cgroup/system.slice/docker-<container-id>.scope/memory.max
```

If no limit is set, this file contains `max`.

## Safe Configuration Path

### Step 1: Capture Baseline Metrics

Before changing limits, measure current usage. Use `docker stats`:

```bash
docker stats --no-stream web
```

Expected output example:

```
CONTAINER ID   NAME      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O          BLOCK I/O        PIDS
c2f2...        web       0.15%     120MiB / 1.945GiB     6.03%     1.2kB / 0B       0B / 0B          7
```

Note the `MEM USAGE / LIMIT` column. If LIMIT shows the host's total memory, the container is unlimited.

### Step 2: Choose the Right Limit Type

- **CPU**: Use CPU shares for relative weighting when multiple containers compete. Use a quota (NanoCpus) for an absolute limit. Use cpuset to pin to specific cores.
- **Memory**: Set a hard limit (`--memory`) and optionally a swap limit (`--memory-swap`). Understand OOM behavior.
- **Disk I/O**: Use blkio limits on cgroup v1; on cgroup v2, use `--device-read-bps` and `--device-write-bps` for block devices.
- **PIDs**: Limit the number of processes to prevent fork bombs with `--pids-limit`.

### Step 3: Apply Limits via Docker Run

Example: run a container with 512 MB memory, 0.5 CPU quota, and 100 PIDs:

```bash
docker run -d --name web \
  --memory 512m \
  --memory-swap 1g \
  --cpus 0.5 \
  --pids-limit 100 \
  nginx:alpine
```

Verify:

```bash
docker inspect web --format '{{json .HostConfig.Resources}}'
```

Expected output includes `"Memory":536870912`, `"NanoCpus":500000000`, `"PidsLimit":100`.

### Step 4: Apply Limits via Docker Compose

In `docker-compose.yml`:

```yaml
services:
  web:
    image: nginx:alpine
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 512M
        reservations:
          memory: 128M
    pids_limit: 100
```

Note: The `deploy` key is only honored by Docker Swarm. For `docker compose` (non-Swarm), use the `resources` key at the service level (Compose v2.24+ supports `limits` and `reservations` under `resources`). Alternatively, use `mem_limit` and `cpus` for older compatibility:

```yaml
services:
  web:
    image: nginx:alpine
    mem_limit: 512m
    cpus: 0.5
    pids_limit: 100
```

Apply with `docker compose up -d`.

## Verification and Diagnostics

After setting limits, verify enforcement from inside and outside the container.

### Inside the Container: Check Visible Memory

The container sees the host's memory, not the limit, unless you set a memory reservation. This confuses many. Run:

```bash
docker exec web cat /proc/meminfo | head -3
```

Expected output shows host total memory, not 512 MB. To see the cgroup limit, read:

```bash
docker exec web cat /sys/fs/cgroup/memory.max
```

Expected output: `536870912` (512 MB in bytes).

### Stress Test Memory Limit

Use a tool like `stress` inside the container:

```bash
docker exec web sh -c "apk add --no-cache stress-ng && stress-ng --vm 1 --vm-bytes 600M --timeout 30s"
```

After a few seconds, check if the process is OOM-killed:

```bash
docker inspect web --format '{{.State.OOMKilled}} {{.State.ExitCode}}'
```

Expected output: `true 137` (137 = 128 + 9, SIGKILL).

### Inspect cgroup Events

On cgroup v2, monitor memory.events:

```bash
docker exec web cat /sys/fs/cgroup/memory.events
```

Look for `oom` and `oom_kill` counters increasing.

### CPU Throttling Verification

To test CPU quota, run a CPU-intensive task and observe throttling:

```bash
docker run --rm --cpus 0.5 alpine sh -c "time crunch 100000000000"
```

Check `docker stats` CPU% should stay near 50%.

For cgroup v2, read `cpu.stat`:

```bash
docker exec web cat /sys/fs/cgroup/cpu.stat | grep throttled
```

Expected output includes `nr_throttled` and `throttled_time`.

## Failure Modes and Recovery

### OOM Kills Inside Container

If the application inside the container is killed, the container may exit or restart. Check:

```bash
docker inspect web --format '{{.State.OOMKilled}}'
```

If true, increase memory limit or reduce application footprint. Note: if `--memory-swap` is not set, swap is disabled and the limit equals `--memory`. To allow swap, set `--memory-swap` > `--memory`.

### Host-level OOM

If the host runs out of memory, the kernel may kill other processes. Use `dmesg` to see OOM kills:

```bash
dmesg | grep -i oom
```

Recovery: add host memory, reduce container limits, or adjust overcommit settings.

### CPU Starvation

If a container is throttled excessively, application performance degrades. Check `nr_throttled` in `cpu.stat`; if high, consider increasing `--cpus` or moving to cpuset.

### Disk I/O Limits Not Applied

On some storage drivers, blkio limits may not work. Verify with `docker info | grep Storage Driver`. For overlay2, blkio limits may be ignored; use device-specific limits or switch to cgroup v2.

### PIDs Limit Exceeded

If the container hits the PIDs limit, fork fails: "Resource temporarily unavailable". Increase `--pids-limit` if legitimate, else fix the application.

## Common Pitfalls and How to Avoid Them

1. **Assuming the container sees its memory limit**: Many apps read `/proc/meminfo` and think they have the host's memory, causing OOM. Solution: set environment variables or use cgroup-aware libraries.
2. **Setting CPU shares instead of quota**: CPU shares are relative and do not cap usage when the CPU is idle. Use `--cpus` for hard limits.
3. **Not setting `--memory-swap`**: If you set `--memory` without `--memory-swap`, swap is disabled, and the container cannot use swap. This may surprise applications that expect swap.
4. **Ignoring PID limits**: A fork bomb can exhaust the host's PID space. Always set `--pids-limit` for untrusted containers.
5. **Not using cgroup v2**: cgroup v2 provides better isolation and features like memory.high for soft limits. Migrate if possible.
6. **Overcommitting host resources**: Summing container limits can exceed host capacity. Monitor overall usage and set appropriate reservations.

## Operations Checklist

Use this checklist before and after changing Docker resource limits.

| Step | Action | Owner | Frequency |
|------|--------|-------|-----------|
| 1 | Baseline current metrics with `docker stats` | DevOps Engineer (e.g., Alex Johnson) | Before every limit change |
| 2 | Review application profile and determine required limits | Application Owner (e.g., Priya Shah) | Pre-deployment |
| 3 | Apply limits in staging environment first | DevOps Engineer | Every change |
| 4 | Run load tests to verify performance under limits | QA Lead | Every release |
| 5 | Monitor OOM kills and throttling metrics | SRE Team | Continuously (alerts) |
| 6 | Document limits in version control | DevOps Engineer | Every change |
| 7 | Review limits quarterly and adjust based on trends | Engineering Manager | Quarterly |

## Conclusion

Docker resource limits are essential for stable multi-tenant environments. By understanding the underlying cgroup mechanisms and using the practical commands provided here, you can set appropriate CPU, memory, I/O, and PID limits, diagnose enforcement issues, and recover from failures. Always start with observation, make incremental changes, and verify with concrete metrics.

Next step: select one container in your environment, follow the Safe Configuration Path to set a memory limit, verify with a stress test, and document the results. Then adopt the Operations Checklist to maintain limits over time.