## Intro Docker containers share the host's kernel and resources. Without explicit limits, a single container can exhaust memory or CPU, causing instability for other services or the host itself. Resource limits are a critical part of running containers safely in development and production. However, setting these limits often leads to confusing errors: containers killed with exit code 137, processes unable to allocate memory, or CPU throttling causing sluggish performance. Misconfigured limits can also cause containers to fail to start or behave unpredictably. This guide covers the most common Docker resource limit errors, how to diagnose them, and practical fixes. We will focus on memory and CPU limits, with notes on PID limits. You will learn how to inspect current settings, apply safe configurations, verify them, and recover from failures. ## Version and Environment Inventory Before changing resource limits, confirm your environment. ### Check Docker Version Resource limit features depend on the Docker version and the underlying init system. Use: ```bash docker version ``` Expected output includes client and server versions. For example: ``` Client: Docker Engine - Community Version: 24.0.7 API version: 1.43 Go version: go1.20.10 Git commit: 311b9ff Built: Thu Oct 26 09:08:15 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.7 API version: 1.43 (minimum version 1.12) Go version: go1.20.10 Git commit: 311b9ff Built: Thu Oct 26 09:08:15 2023 OS/Arch: linux/amd64 Experimental: false ``` ### Check Kernel Support Memory limits rely on cgroups. Verify cgroup v2 or v1 is active: ```bash docker info | grep -i cgroup ``` On modern systems, you might see: ``` Cgroup Driver: systemd Cgroup Version: 2 ``` If cgroup v2 is not supported, memory limits may not be enforced correctly. ### Identify Running Containers List containers and their current resource usage: ```bash docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Status}}" ``` This gives a quick overview. ### Prerequisites You will need: - Docker Engine 20.10 or newer (for reliable cgroup v2 support) - Appropriate permissions to run Docker commands (typically root or docker group) - A test container image (e.g., `alpine` or `ubuntu`) ## Safe Configuration Path Resource limits can be set at container runtime, in Docker Compose files, or by updating existing containers. We will cover each with examples. ### Runtime Flags The `docker run` command accepts `--memory` (or `-m`) for memory limits and `--cpus` for CPU limits. Memory limit example: ```bash docker run -d --name limited-nginx --memory 256m nginx ``` This limits the container to 256 MB of RAM. If the container exceeds this, the kernel may kill the process inside the container, and the container will exit with code 137. CPU limit example: ```bash docker run -d --name limited-cpu --cpus 1.5 nginx ``` This allows the container to use up to 1.5 CPU cores. If the container tries to use more, it will be throttled. Combined limits: ```bash docker run -d --name limited-app --memory 512m --memory-swap 1g --cpus 2 myapp ``` The `--memory-swap` sets total memory+swap. If you set `--memory` and `--memory-swap` to the same value, swap is disabled. PID limit: ```bash docker run -d --name limited-pids --pids-limit 100 nginx ``` This limits the number of processes inside the container. ### Docker Compose Configuration In a `docker-compose.yml`, use the `deploy.resources` section for Docker Swarm or the top-level `resources` (in Compose v2.x with `docker compose` and non-swarm mode) or the legacy `mem_limit`, `cpus` fields. Modern Compose (v2.x) example: ```yaml services: web: image: nginx deploy: resources: limits: cpus: '0.5' memory: 512M reservations: cpus: '0.25' memory: 256M ``` Note: The `deploy` section is used by Docker Swarm and by `docker compose` when using the `--compatibility` flag or in recent versions with `docker compose up`. For local development, you can use the simpler legacy options: ```yaml services: web: image: nginx mem_limit: 512m cpus: 0.5 ``` ### Updating Existing Containers You cannot change resource limits on a running container directly. You must stop the container, recreate it with new limits, or use `docker update` for some settings. Using `docker update`: `docker update` can change memory, CPU, and other limits on a running container without restarting. For example: ```bash docker update --memory 512m --cpus 1 mycontainer docker update --memory-swap -1 mycontainer ``` The `-1` for memory-swap resets it to unlimited swap (but memory limit remains). However, some changes may require a restart if the container's init process needs to see the new limits. ## Verification and Diagnostics After setting limits, verify they are applied and test their behavior. ### Inspect Container Limits Use `docker inspect` to see the effective limits: ```bash docker inspect limited-nginx --format '{{.HostConfig.Memory}}' ``` Output would be the number of bytes, e.g., `268435456` for 256 MB. For CPU: ```bash docker inspect limited-cpu --format '{{.HostConfig.NanoCpus}}' ``` Output would be `1500000000` for 1.5 CPUs (since 1 CPU = 1e9 NanoCpus). ### Check Runtime Statistics Use `docker stats` to observe real-time usage: ```bash docker stats limited-nginx --no-stream ``` Output: ``` CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS abc123def456 limited-nginx 0.00% 2.5MiB / 256MiB 0.98% 0B / 0B 0B / 0B 5 ``` The `MEM USAGE / LIMIT` column shows the current usage against the limit. ### Testing Memory Limits To see how a container behaves when it exceeds memory limits, run a container with a memory limit and then execute a memory-consuming command inside it. ```bash docker run -d --name mem-test --memory 128m alpine sleep 300 docker exec mem-test sh -c "tail /dev/zero" ``` The `tail /dev/zero` command will quickly consume memory. After a few seconds, check the container status: ```bash docker ps -a | grep mem-test ``` You should see the container has exited with code 137. Check the logs: ```bash docker logs mem-test ``` It may show nothing because the process was killed by the OOM killer. On the host, you can check `dmesg` for OOM messages. ### Testing CPU Limits To observe CPU throttling, run a CPU-intensive task with a low CPU limit: ```bash docker run -d --name cpu-test --cpus 0.5 alpine sh -c "while true; do :; done" ``` Then monitor CPU usage: ```bash docker stats cpu-test --no-stream ``` You should see CPU % around 50% (0.5 CPU). If you check the container's cgroup CPU statistics, you will see throttling counts. ## Failure Modes and Recovery Understanding common errors and how to recover is essential. ### Container Exits with Code 137 Exit code 137 indicates the container was killed, usually due to OOM (out of memory) or sometimes due to a manual kill. If your container is being killed unexpectedly, check memory limits and usage. Diagnosis: - Check exit code: `docker inspect --format '{{.State.ExitCode}}'` - Check logs for OOM messages: `docker logs ` may not show much; check host `dmesg` or `journalctl` for OOM events. - Inspect memory limit: `docker inspect --format '{{.HostConfig.Memory}}'` - Check peak memory usage before death if available. Fix: - Increase the memory limit if the application legitimately needs more memory: `docker update --memory 512m ` or recreate with a higher limit. - Optimize the application's memory usage (e.g., fix memory leaks). - Add swap space if appropriate: allow swap by setting `--memory-swap` larger than `--memory`. Rollback: If the increased limit causes host issues, revert to the previous limit and scale down the application or add more host memory. ### Container Fails to Start Due to Invalid Limits Sometimes you may specify limits that are invalid, such as `--memory 10g` on a host with less memory. Docker will still start the container, but the limit may not be achievable. More commonly, you might get an error like: ``` docker: Error response from daemon: invalid memory limit: 0. ``` Or if you specify a CPU limit that is too low, e.g., `--cpus 0.001`, Docker may accept it but the container might be starved. Diagnosis: - Check the exact error message from `docker run`. - Ensure the limit is a positive number and within host capabilities. Fix: - Correct the limit value. - On cgroup v2, avoid setting swap limits without memory limits, as it may cause errors. ### CPU Throttling and Performance Issues If your container is not performing as expected, it may be hitting CPU limits. Symptoms include slow response times and high CPU throttling metrics. Diagnosis: - Check container CPU stats: `docker stats ` shows CPU %; compare with limit. - Inspect cgroup CPU throttling: on cgroup v2, check `cat /sys/fs/cgroup/system.slice/docker-.scope/cpu.stat` and look at `nr_throttled` and `throttled_time`. Fix: - Increase CPU limit if the workload needs more. - Optimize the application to use fewer CPU cycles. - Adjust scheduling or parallelization. ### PID Limit Exceeded If a container tries to fork more processes than allowed, it will get errors like `fork: retry: Resource temporarily unavailable`. Diagnosis: - Check container logs for fork errors. - Inspect PID limit: `docker inspect --format '{{.HostConfig.PidsLimit}}'` (default is 0, meaning unlimited, but you may have set a limit). - Check current PID count: `docker exec ps -e | wc -l` Fix: - Increase `--pids-limit` if the application legitimately needs more processes. - Reduce number of processes by optimizing the application (e.g., use worker threads instead of processes). ## Common Pitfalls and How to Avoid Them Resource limits are powerful but easy to get wrong. Here are frequent mistakes and how to avoid or recover from them. ### Setting Limits Too Low for the Workload Many teams start with arbitrary low limits without profiling actual usage, leading to frequent OOM kills or CPU starvation. Why it happens: Lack of understanding of the application's resource requirements. How to avoid: Baseline your application in a staging environment using tools like `docker stats` or application performance monitoring. Set limits with at least 20-30% headroom above peak usage. Example: If your web app peaks at 700 MB, set `--memory 1g` instead of `--memory 512m`. Recovery: Increase limits incrementally and monitor for stability. ### Ignoring Swap Configuration By default, Docker may not allow swap unless you explicitly set it. If your container occasionally needs more memory than the limit but you want to avoid OOM kills, enabling swap can help, but it may slow down the container. Why it happens: Misunderstanding of `--memory-swap` semantics. How to avoid: Understand that `--memory-swap` is the total of memory+swap. If you set `--memory 512m --memory-swap 1g`, the container has 512 MB RAM and 512 MB swap. If you set both to the same value, swap is disabled. Recovery: Adjust `--memory-swap` carefully and test performance. ### Relying Solely on Runtime Flags When using orchestration or Compose, setting limits only via `docker run` flags can lead to inconsistency when containers are recreated. Why it happens: Teams may not have a standard way to define limits. How to avoid: Define limits in your Docker Compose file or orchestration manifests (Kubernetes, Swarm). This ensures limits are version-controlled and applied consistently. Recovery: Refactor to use Compose or orchestration configurations. ### Not Monitoring After Deployment Setting limits is not a one-time task. Applications change, and limits need adjustment. Why it happens: Lack of ongoing monitoring and alerting. How to avoid: Set up monitoring with tools like Prometheus, Datadog, or cAdvisor. Alert on metrics like container memory usage approaching the limit or CPU throttling. Recovery: Review and adjust limits based on usage trends, at least quarterly or after significant application changes. ### Misunderstanding CPU Limits Docker's `--cpus` is a hard limit on CPU time, not a reservation. If you set `--cpus 1` for a bursty workload that occasionally needs 2 CPUs, it will be throttled even if the host has idle CPUs. Why it happens: Confusion between CPU limits and CPU reservations. Docker does not have a separate CPU reservation flag, but you can use CPU shares (`--cpu-shares`) to influence scheduling when there is contention. How to avoid: Understand your workload's CPU pattern. For bursty workloads, consider setting a higher CPU limit or using `--cpu-shares` to give it more weight. Recovery: Adjust `--cpus` or `--cpu-shares` based on performance observations. ## Operations Checklist Use this checklist when working with Docker resource limits. | Step | Action | Command / Notes | Owner | Frequency | |------|--------|-----------------|-------|-----------| | 1 | Verify Docker version and cgroup support | `docker version`, `docker info \| grep -i cgroup` | DevOps Engineer | Monthly or after infrastructure changes | | 2 | Identify containers and current resource usage | `docker ps`, `docker stats` | DevOps Engineer | Weekly | | 3 | Decide on limits based on workload needs | Base on application profiling and testing | Application Owner (e.g., Priya Shah, Engineering Lead) | Per release or quarterly | | 4 | Apply limits using docker run flags or Compose | Use `--memory`, `--cpus`, `--pids-limit` | DevOps Engineer | Per deployment | | 5 | Verify limits are applied | `docker inspect` to check HostConfig | DevOps Engineer | Immediately after deployment | | 6 | Monitor resource usage and throttling | `docker stats`, check cgroup stats | Monitoring Team | Continuous | | 7 | Test failure scenarios (OOM, CPU throttle) | Run stress tests in staging | QA Lead | Before each major release | | 8 | Document limits and reasoning | Update runbooks | DevOps Engineer | As changes occur | | 9 | Set up alerting for resource usage | Use monitoring tools | Monitoring Team | Initial setup, then maintain | | 10 | Review and adjust limits periodically | Compare usage trends | Application Owner and DevOps Lead | Quarterly | ## Conclusion Docker resource limits are essential for stable and predictable container operations. Common errors such as exit code 137, CPU throttling, and PID exhaustion can be diagnosed and fixed with the right commands and configuration. Start with a narrow pilot to test limits before broad rollout, and always verify with `docker inspect` and `docker stats`. Document your limits and review them as application needs change. By following the safe configuration path and having a rollback plan, you can avoid most resource-related issues and keep your containerized applications running smoothly. Additional resources: - Docker documentation: [Runtime options with Memory, CPUs, and GPUs](https://docs.docker.com/config/containers/resource_constraints/) - Docker Compose resource constraints: [Compose file reference](https://docs.docker.com/compose/compose-file/)