Learn how to estimate Docker Swarm cluster capacity, avoid resource exhaustion, and scale safely. This guide includes step-by-step sizing, monitoring, and operational checklists, with practical commands and real-world examples.

## Introduction

Docker Swarm clusters often fail at the worst possible moment: a traffic spike hits, a service leaks memory, and suddenly nodes are starved for CPU or RAM. Containers get evicted, the scheduler can't place new tasks, and users see timeouts. Capacity planning is the discipline of estimating how much CPU, memory, disk, and network your services will need, then monitoring actual usage to adjust before problems occur. Done right, it prevents both overprovisioning waste and underprovisioning outages.

This guide walks through a hands-on approach to sizing your Swarm cluster. You will inventory your current environment, set safe resource reservations and limits, verify configurations, diagnose failures, and follow a repeatable operations checklist. Every step includes concrete commands and expected outputs so you can apply it directly to your own cluster.

## Inventory Your Environment

Before you can plan capacity, you need a clear picture of what you are running. Record the Docker Engine and Swarm versions, node hardware specs, and the services you expect to run. Use the following commands to gather details.

First, check the Docker version and Swarm status:

```bash
docker version --format '{{.Server.Version}}'
docker info --format '{{.Swarm.LocalNodeState}} {{.Swarm.Nodes}}'
```

Expected output example:

```
20.10.12
active 5
```

This indicates Docker Engine 20.10.12 and an active Swarm with five nodes.

List all nodes with their roles and availability:

```bash
docker node ls
```

Output columns: `ID`, `HOSTNAME`, `STATUS`, `AVAILABILITY`, `MANAGER STATUS`. Confirm all nodes are `Ready` and `Active`.

Inspect each node's resources:

```bash
docker node inspect --format '{{.Description.Hostname}} CPUs={{.Description.Resources.NanoCPUs}} Memory={{.Description.Resources.MemoryBytes}}' <node-id>
```

`NanoCPUs` are billionths of a CPU (1e9 = 1 CPU). `MemoryBytes` is in bytes. Example:

```
worker-01 CPUs=4000000000 Memory=16777216000
```

That means 4 CPUs and 16 GB of RAM.

Note the operating system and kernel versions, as they affect resource limits:

```bash
uname -a
```

List current services and their resource reservations/limits:

```bash
docker service ls
docker service inspect --format '{{.Spec.Name}} Reservations: CPU={{.Spec.TaskTemplate.Resources.Reservations.NanoCPUs}} Mem={{.Spec.TaskTemplate.Resources.Reservations.MemoryBytes}} Limits: CPU={{.Spec.TaskTemplate.Resources.Limits.NanoCPUs}} Mem={{.Spec.TaskTemplate.Resources.Limits.MemoryBytes}}' <service-name>
```

If resources are not set, they show as `0`. This inventory is your baseline for planning.

## Configure Safe Resource Limits

After inventory, configure resource reservations and limits for each service. Reservations guarantee minimum resources; limits cap maximum usage. Set them in your service definition (Compose file or `docker service create` command). Example for a web service:

```yaml
# docker-compose.yml
version: '3.8'
services:
  web:
    image: nginx:latest
    deploy:
      replicas: 3
      resources:
        reservations:
          cpus: '0.25'
          memory: 128M
        limits:
          cpus: '0.5'
          memory: 256M
```

Deploy with:

```bash
docker stack deploy -c docker-compose.yml mystack
```

Verify with `docker service inspect --pretty mystack_web` to see the resources section.

Set placement constraints to put services on appropriate nodes. For example, only on nodes labeled with SSD:

```yaml
placement:
  constraints:
    - node.labels.storage == ssd
```

Add labels to nodes with:

```bash
docker node update --label-add storage=ssd node1
```

Use update configurations to control rolling updates and avoid capacity spikes:

```yaml
update_config:
  parallelism: 1
  delay: 10s
  failure_action: rollback
  max_failure_ratio: 0.2
```

This updates one task at a time with a 10-second delay, rolling back if more than 20% fail.

For capacity planning, monitor actual usage and adjust limits gradually. Start with conservative limits based on expected peak load, then tune using monitoring data. For example, if your web service typically uses 150 MB of memory under load, set the reservation to 128 MB and the limit to 256 MB to leave headroom.

## Verify Resource Configurations

After setting limits, verify they work as intended. Use the following commands and expected outcomes.

Check service tasks and their resource usage:

```bash
docker stats --no-stream
```

Output shows per-container CPU percentage, memory usage, and limit. Compare with your settings.

Inspect node resource availability:

```bash
docker node inspect --format '{{.Description.Hostname}} CPUs={{.Description.Resources.NanoCPUs}} Memory={{.Description.Resources.MemoryBytes}}' <node-id>
```

Simulate node removal to test capacity:

```bash
docker node update --availability drain <node-id>
```

Draining ensures tasks are rescheduled elsewhere, verifying capacity.

Test rollback by introducing a memory limit too low and observing failure:

```bash
docker service update --limit-memory 32M mystack_web
```

If the container exceeds 32 MB, Docker kills it, and the service may roll back if configured.

Check events for OOM:

```bash
docker events --filter type=container --filter event=oom
```

This shows out-of-memory events, indicating capacity issues.

Use `docker service ps` to see task attempts and errors:

```bash
docker service ps mystack_web
```

Look for `REJECTED` or `FAILED` tasks due to insufficient resources. Expected results: healthy services show running tasks, no OOM events, and resource utilization below limits.

## Diagnose Capacity Failures

Understanding common failure modes helps you recover quickly.

### Node Resource Exhaustion

A node runs out of memory or CPU. Symptoms include tasks evicted, OOM kills, or the scheduler cannot place tasks. Recovery steps:

1. Drain the node to move workloads:
   ```bash
   docker node update --availability drain nodeX
   ```
2. Investigate with `docker stats` to find the culprit container.
3. Adjust limits or add capacity. For example, if a container consistently uses 90% of its memory limit, increase the limit or reduce replicas.

### Service Fails to Scale

Scaling up results in "no suitable node" errors. Check constraints and available resources:

```bash
docker service scale mystack_web=10
docker service ps mystack_web
```

Pending tasks indicate insufficient resources or unmet constraints. Adjust constraints or add nodes.

### Unbounded Memory Growth

A container leaks memory and hits its limit, causing repeated restarts. Cap memory with:

```bash
docker service update --limit-memory 512M mystack_web
```

But ensure monitoring alerts before hitting the limit. Rollback to previous config if needed:

```bash
docker service rollback mystack_web
```

### Inadequate Reservations

Services with no reservations may be scheduled on overcrowded nodes. Set reservations to guarantee minimum capacity for each service.

### Network Saturation

Overlay network bandwidth is exceeded. Monitor with `docker network inspect` and use network traffic tools. Consider service mesh or separate networks.

Always keep a capacity buffer: at least 20-30% headroom on each node for failover and spikes.

## Common Pitfalls and How to Avoid Them

Capacity planning mistakes are common. Here are pitfalls and how to avoid them:

### Pitfall: No Resource Limits

**Why it happens:** Teams omit limits to avoid complexity or assume containers will behave.

**How to avoid:** Set limits and reservations for every service in production. Use a template in your Compose files and review regularly. For example:

```yaml
deploy:
  resources:
    limits:
      cpus: '0.5'
      memory: 256M
```

### Pitfall: Overprovisioning

**Why it happens:** Fear of outages leads to generous limits that waste cluster capacity and increase costs.

**How to avoid:** Monitor actual usage with `docker stats` and set limits based on the 95th percentile plus a 20% buffer. Adjust quarterly.

### Pitfall: Ignoring Node Headroom

**Why it happens:** Teams focus on individual services but forget to leave headroom on nodes for system processes and failover.

**How to avoid:** Ensure each node's total allocated resources do not exceed 70-80% of capacity. Use `docker node inspect` to sum reservations.

### Pitfall: Not Testing Failover

**Why it happens:** Assumptions about rescheduling are never validated.

**How to avoid:** Regularly drain a node in a staging environment and verify all services reschedule without errors. Document the process.

## Operations Checklist

Use this checklist weekly or before any major deployment. Assign an owner to each item and revisit monthly.

| Task | Command / Action | Expected Result | Owner | Review Frequency |
|------|------------------|-----------------|-------|------------------|
| Check node status | `docker node ls` | All nodes Ready and Active | DevOps lead | Weekly |
| Check resource usage | `docker stats --no-stream` | CPU/mem below limits, no spikes | SRE on-call | Daily |
| Review service resource configs | `docker service inspect --pretty` on each service | Reservations/limits appropriate | Service owner | Monthly |
| Monitor events for OOM | `docker events --since 24h --filter event=oom` | No OOM events | SRE on-call | Daily |
| Verify capacity headroom | `docker node inspect` sum resources vs usage | At least 25% headroom per node | DevOps lead | Weekly |
| Test scaling | `docker service scale <service>=<current+1>` then scale back | Scaling succeeds, no pending tasks | QA engineer | Before release |
| Check logs for resource errors | `docker service logs <service>` | No resource-related errors | Service owner | Weekly |
| Review failed tasks | `docker service ps <service>` | No recent failures due to resources | SRE on-call | Weekly |
| Backup cluster state | `docker swarm unlock-key` (if autolock) and save configs | Key and configs stored safely | DevOps lead | Monthly |
| Document changes | Update capacity planning doc | Current baseline and projections recorded | Engineering manager | Monthly |

Additionally:

- Set up alerts: use monitoring tools (e.g., Prometheus, cAdvisor) to alert when node CPU > 80% or memory > 85%.
- Perform load testing: before scaling, simulate peak load to validate capacity.
- Regularly review placement constraints and node labels for alignment with capacity planning.

## Conclusion

Capacity planning for Docker Swarm is an ongoing process of measuring, adjusting, and verifying. By inventorying your environment, setting explicit resource reservations and limits, monitoring usage, and preparing for failures, you maintain a stable cluster. The next steps are to establish a baseline using this guide's commands, set up monitoring alerts, and schedule regular capacity reviews. Start small with a pilot service, measure actual usage, and scale confidently.