Docker Resource Limits: Common Errors and Practical Fixes

Intro

Docker containers share the host's kernel and resources. Without explicit limits, a single container can exhaust memory or CPU, causing instability for other services or the host itself. Resource limits are a critical part of running containers safely in development and production.

However, setting these limits often leads to confusing errors: containers killed with exit code 137, processes unable to allocate memory, or CPU throttling causing sluggish performance. Misconfigured limits can also cause containers to fail to start or behave unpredictably.

This guide covers the most common Docker resource limit errors, how to diagnose them, and practical fixes. We will focus on memory and CPU limits, with notes on PID limits. You will learn how to inspect current settings, apply safe configurations, verify them, and recover from failures.

Version and Environment Inventory

Before changing resource limits, confirm your environment.

Check Docker Version

Resource limit features depend on the Docker version and the underlying init system. Use:

docker version

Expected output includes client and server versions. For example:

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        311b9ff
 Built:             Thu Oct 26 09:08:15 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:15 2023
  OS/Arch:          linux/amd64
  Experimental:     false

Check Kernel Support

Memory limits rely on cgroups. Verify cgroup v2 or v1 is active:

docker info | grep -i cgroup

On modern systems, you might see:

Cgroup Driver: systemd
Cgroup Version: 2

If cgroup v2 is not supported, memory limits may not be enforced correctly.

Identify Running Containers

List containers and their current resource usage:

docker ps --format "table {{.ID}}\t{{.Names}}\t{{.Status}}"

This gives a quick overview.

Prerequisites

You will need:

• Docker Engine 20.10 or newer (for reliable cgroup v2 support)
• Appropriate permissions to run Docker commands (typically root or docker group)
• A test container image (e.g., alpine or ubuntu)

Safe Configuration Path

Resource limits can be set at container runtime, in Docker Compose files, or by updating existing containers. We will cover each with examples.

Runtime Flags

The docker run command accepts --memory (or -m) for memory limits and --cpus for CPU limits.

Memory limit example:

docker run -d --name limited-nginx --memory 256m nginx

This limits the container to 256 MB of RAM. If the container exceeds this, the kernel may kill the process inside the container, and the container will exit with code 137.

CPU limit example:

docker run -d --name limited-cpu --cpus 1.5 nginx

This allows the container to use up to 1.5 CPU cores. If the container tries to use more, it will be throttled.

Combined limits:

docker run -d --name limited-app --memory 512m --memory-swap 1g --cpus 2 myapp

The --memory-swap sets total memory+swap. If you set --memory and --memory-swap to the same value, swap is disabled.

PID limit:

docker run -d --name limited-pids --pids-limit 100 nginx

This limits the number of processes inside the container.

Docker Compose Configuration

In a docker-compose.yml, use the deploy.resources section for Docker Swarm or the top-level resources (in Compose v2.x with docker compose and non-swarm mode) or the legacy mem_limit, cpus fields.

Modern Compose (v2.x) example:

services:
  web:
    image: nginx
    deploy:
      resources:
        limits:
          cpus: '0.5'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M

Note: The deploy section is used by Docker Swarm and by docker compose when using the --compatibility flag or in recent versions with docker compose up. For local development, you can use the simpler legacy options:

services:
  web:
    image: nginx
    mem_limit: 512m
    cpus: 0.5

Updating Existing Containers

You cannot change resource limits on a running container directly. You must stop the container, recreate it with new limits, or use docker update for some settings.

Using docker update:

docker update can change memory, CPU, and other limits on a running container without restarting. For example:

docker update --memory 512m --cpus 1 mycontainer
docker update --memory-swap -1 mycontainer

The -1 for memory-swap resets it to unlimited swap (but memory limit remains).

However, some changes may require a restart if the container's init process needs to see the new limits.

Verification and Diagnostics

After setting limits, verify they are applied and test their behavior.

Inspect Container Limits

Use docker inspect to see the effective limits:

docker inspect limited-nginx --format '{{.HostConfig.Memory}}'

Output would be the number of bytes, e.g., 268435456 for 256 MB.

For CPU:

docker inspect limited-cpu --format '{{.HostConfig.NanoCpus}}'

Output would be 1500000000 for 1.5 CPUs (since 1 CPU = 1e9 NanoCpus).

Check Runtime Statistics

Use docker stats to observe real-time usage:

docker stats limited-nginx --no-stream

Output:

CONTAINER ID   NAME            CPU %     MEM USAGE / LIMIT   MEM %     NET I/O     BLOCK I/O   PIDS
abc123def456   limited-nginx   0.00%     2.5MiB / 256MiB     0.98%     0B / 0B     0B / 0B     5

The MEM USAGE / LIMIT column shows the current usage against the limit.

Testing Memory Limits

To see how a container behaves when it exceeds memory limits, run a container with a memory limit and then execute a memory-consuming command inside it.

docker run -d --name mem-test --memory 128m alpine sleep 300
docker exec mem-test sh -c "tail /dev/zero"

The tail /dev/zero command will quickly consume memory. After a few seconds, check the container status:

docker ps -a | grep mem-test

You should see the container has exited with code 137.

Check the logs:

docker logs mem-test

It may show nothing because the process was killed by the OOM killer. On the host, you can check dmesg for OOM messages.

Testing CPU Limits

To observe CPU throttling, run a CPU-intensive task with a low CPU limit:

docker run -d --name cpu-test --cpus 0.5 alpine sh -c "while true; do :; done"

Then monitor CPU usage:

docker stats cpu-test --no-stream

You should see CPU % around 50% (0.5 CPU). If you check the container's cgroup CPU statistics, you will see throttling counts.

Failure Modes and Recovery

Understanding common errors and how to recover is essential.

Container Exits with Code 137

Exit code 137 indicates the container was killed, usually due to OOM (out of memory) or sometimes due to a manual kill. If your container is being killed unexpectedly, check memory limits and usage.

Diagnosis:

• Check exit code: docker inspect <container> --format '{{.State.ExitCode}}'
• Check logs for OOM messages: docker logs <container> may not show much; check host dmesg or journalctl for OOM events.
• Inspect memory limit: docker inspect <container> --format '{{.HostConfig.Memory}}'
• Check peak memory usage before death if available.

Fix:

• Increase the memory limit if the application legitimately needs more memory: docker update --memory 512m <container> or recreate with a higher limit.
• Optimize the application's memory usage (e.g., fix memory leaks).
• Add swap space if appropriate: allow swap by setting --memory-swap larger than --memory.

Rollback:

If the increased limit causes host issues, revert to the previous limit and scale down the application or add more host memory.

Container Fails to Start Due to Invalid Limits

Sometimes you may specify limits that are invalid, such as --memory 10g on a host with less memory. Docker will still start the container, but the limit may not be achievable. More commonly, you might get an error like:

docker: Error response from daemon: invalid memory limit: 0.

Or if you specify a CPU limit that is too low, e.g., --cpus 0.001, Docker may accept it but the container might be starved.

Diagnosis:

• Check the exact error message from docker run.
• Ensure the limit is a positive number and within host capabilities.

Fix:

• Correct the limit value.
• On cgroup v2, avoid setting swap limits without memory limits, as it may cause errors.

CPU Throttling and Performance Issues

If your container is not performing as expected, it may be hitting CPU limits. Symptoms include slow response times and high CPU throttling metrics.

Diagnosis:

• Check container CPU stats: docker stats <container> shows CPU %; compare with limit.
• Inspect cgroup CPU throttling: on cgroup v2, check cat /sys/fs/cgroup/system.slice/docker-<container-id>.scope/cpu.stat and look at nr_throttled and throttled_time.

Fix:

• Increase CPU limit if the workload needs more.
• Optimize the application to use fewer CPU cycles.
• Adjust scheduling or parallelization.

PID Limit Exceeded

If a container tries to fork more processes than allowed, it will get errors like fork: retry: Resource temporarily unavailable.

Diagnosis:

• Check container logs for fork errors.
• Inspect PID limit: docker inspect <container> --format '{{.HostConfig.PidsLimit}}' (default is 0, meaning unlimited, but you may have set a limit).
• Check current PID count: docker exec <container> ps -e | wc -l

Fix:

• Increase --pids-limit if the application legitimately needs more processes.
• Reduce number of processes by optimizing the application (e.g., use worker threads instead of processes).

Common Pitfalls and How to Avoid Them

Resource limits are powerful but easy to get wrong. Here are frequent mistakes and how to avoid or recover from them.

Setting Limits Too Low for the Workload

Many teams start with arbitrary low limits without profiling actual usage, leading to frequent OOM kills or CPU starvation.

Why it happens: Lack of understanding of the application's resource requirements.

How to avoid: Baseline your application in a staging environment using tools like docker stats or application performance monitoring. Set limits with at least 20-30% headroom above peak usage.

Example: If your web app peaks at 700 MB, set --memory 1g instead of --memory 512m.

Recovery: Increase limits incrementally and monitor for stability.

Ignoring Swap Configuration

By default, Docker may not allow swap unless you explicitly set it. If your container occasionally needs more memory than the limit but you want to avoid OOM kills, enabling swap can help, but it may slow down the container.

Why it happens: Misunderstanding of --memory-swap semantics.

How to avoid: Understand that --memory-swap is the total of memory+swap. If you set --memory 512m --memory-swap 1g, the container has 512 MB RAM and 512 MB swap. If you set both to the same value, swap is disabled.

Recovery: Adjust --memory-swap carefully and test performance.

Relying Solely on Runtime Flags

When using orchestration or Compose, setting limits only via docker run flags can lead to inconsistency when containers are recreated.

Why it happens: Teams may not have a standard way to define limits.

How to avoid: Define limits in your Docker Compose file or orchestration manifests (Kubernetes, Swarm). This ensures limits are version-controlled and applied consistently.

Recovery: Refactor to use Compose or orchestration configurations.

Not Monitoring After Deployment

Setting limits is not a one-time task. Applications change, and limits need adjustment.

Why it happens: Lack of ongoing monitoring and alerting.

How to avoid: Set up monitoring with tools like Prometheus, Datadog, or cAdvisor. Alert on metrics like container memory usage approaching the limit or CPU throttling.

Recovery: Review and adjust limits based on usage trends, at least quarterly or after significant application changes.

Misunderstanding CPU Limits

Docker's --cpus is a hard limit on CPU time, not a reservation. If you set --cpus 1 for a bursty workload that occasionally needs 2 CPUs, it will be throttled even if the host has idle CPUs.

Why it happens: Confusion between CPU limits and CPU reservations. Docker does not have a separate CPU reservation flag, but you can use CPU shares (--cpu-shares) to influence scheduling when there is contention.

How to avoid: Understand your workload's CPU pattern. For bursty workloads, consider setting a higher CPU limit or using --cpu-shares to give it more weight.

Recovery: Adjust --cpus or --cpu-shares based on performance observations.

Operations Checklist

Use this checklist when working with Docker resource limits.

Conclusion

Docker resource limits are essential for stable and predictable container operations. Common errors such as exit code 137, CPU throttling, and PID exhaustion can be diagnosed and fixed with the right commands and configuration. Start with a narrow pilot to test limits before broad rollout, and always verify with docker inspect and docker stats. Document your limits and review them as application needs change.

By following the safe configuration path and having a rollback plan, you can avoid most resource-related issues and keep your containerized applications running smoothly.

Additional resources:

• Docker documentation: Runtime options with Memory, CPUs, and GPUs
• Docker Compose resource constraints: Compose file reference