## Intro

Docker Secrets capacity planning with practical examples means moving from an observed problem to a verified result. You should start by identifying the installed Docker version, the deployment topology (Swarm, Compose, or standalone), the operating system and kernel, and the exact component you are inspecting. Do not guess: run a read-only command first, record the current state, then decide whether a change is justified.

This article targets developers, DevOps consultants, and technical startup teams who need to plan secret capacity before it becomes a bottleneck. It connects Docker Secrets scaling, resources, limits, and sizing to concrete commands, expected output, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of real secrets in examples, verify the result, and document how to recover if the expected state is not reached.

Docker Secrets is a feature of Docker Swarm mode and, with the Compose `secrets` top-level element, can be used with `docker compose` even on a single node. Secrets are stored encrypted in the Raft log and made available to services as files under `/run/secrets/`. Capacity planning involves understanding how many secrets a service can mount, how large a secret can be, how many services can use a secret, how encryption and rotation affect performance, and how to monitor for saturation.

## Version and Environment Inventory

Before sizing anything, confirm the Docker version, the running mode, and the storage driver. This is critical because secret size limits and behavior differ between Docker versions and between Swarm and Compose. Run the following read-only checks:

```bash
docker version --format '{{.Server.Version}}'
docker info --format '{{.Swarm.LocalNodeState}} {{.Driver}}'
```

Expected output should show the server version (for example, `24.0.7`) and either `active` for an active Swarm node or `inactive` if you are not in Swarm mode. If the node is not in Swarm mode, `docker secret` commands will fail with `Error response from daemon: This node is not a swarm manager`.

If you use Docker Compose, check the Compose plugin version:

```bash
docker compose version
```

Typical output: `Docker Compose version v2.23.0`. Compose supports secrets only with the Compose Specification and a compatible runtime; older `docker-compose` (v1) does not support secrets.

Confirm where existing secrets are stored and how they are mounted. List secrets with:

```bash
docker secret ls
```

Output columns: `ID`, `NAME`, `CREATED`, `UPDATED`. For example:

```
ID                          NAME                CREATED             UPDATED
vqj0f2a7x4n9k3p1m8t5r6w    db_password         2 days ago          2 days ago
```

Inspect a specific secret's metadata (not its value):

```bash
docker secret inspect db_password
```

This returns JSON with `ID`, `Version`, `CreatedAt`, `UpdatedAt`, and `Spec.Name`. The actual secret value is never shown.

For a Compose project, list secrets defined in the compose file and their files:

```bash
docker compose config --secrets
```

This reveals how many secrets are wired to services and whether any are unused.

When data is involved, confirm where files are stored before changing containers. Docker Secrets are not stored in a named volume; they are encrypted in the Raft log and mounted as a tmpfs at `/run/secrets/<secret_name>` inside the container. A bind mount or volume can be used for application data, but secrets should always use the Docker Secrets mechanism, never environment variables or files baked into the image.

For example, a service defined as:

```yaml
services:
  db:
    image: postgres:16
    secrets:
      - db_password
secrets:
  db_password:
    file: ./secrets/db_password.txt
```

will have the secret available at `/run/secrets/db_password` inside the `db` container. Verify with:

```bash
docker compose exec db ls -l /run/secrets/
```

Expected output:

```
-r--r--r-- 1 root root 24 Jan 15 10:00 db_password
```

The file is read-only and owned by root. If the application runs as a non-root user, you may need to adjust ownership or use a group with read access, as discussed later.

As part of environment inventory, perform a restart test: create a service, add a secret, stop the container, recreate it, and confirm the application still sees the secret. For Swarm:

```bash
docker service create --name secret-test --secret db_password alpine sleep 3600
docker service ps secret-test
docker service rm secret-test
docker service create --name secret-test --secret db_password alpine sleep 3600
```

Then exec into the new container and verify `/run/secrets/db_password` exists. If the secret is missing, the service definition or secret reference is wrong.

## Understanding Docker Secrets Limits and Sizing

Docker Secrets has documented and undocumented limits that directly affect capacity planning. The two hard limits are:

- Maximum secret size: **500 KB** per secret. This is enforced by the API; any attempt to create a larger secret returns `Error response from daemon: rpc error: code = InvalidArgument desc = invalid secret: exceeds maximum size of 500KB`.
- Maximum number of secrets per service: no official hard limit, but the practical limit is determined by the total size of mounted tmpfs and the container's memory limits. Every mounted secret consumes memory in the container's tmpfs. For a service with 10 secrets each 100 KB, the tmpfs uses at least 1 MB of memory, plus overhead.

Swarm stores secrets in the Raft log, which is replicated to all manager nodes. A 500 KB secret is stored in Raft, and every manager keeps a copy. If you have 5 managers and 10 such secrets, that is 25 MB of Raft storage just for secrets. Raft log size can affect performance if it grows too large; Docker recommends keeping the Raft log under a few hundred megabytes.

To check the current Raft log size on a manager:

```bash
du -sh /var/lib/docker/swarm/raft/
```

If the directory grows beyond expectations, inspect the number and size of secrets:

```bash
docker secret ls -q | xargs -I {} sh -c 'docker secret inspect {} | jq -r ".[0].Spec.Name, .[0].ID"'
```

But secret values are not stored on disk in plaintext, so their individual sizes are not directly exposed. You can approximate by reading the source files before creating the secret. Use placeholders:

```bash
head -c 500K /dev/zero | base64 > secret_placeholder.txt
```

Then create the secret from that placeholder to test the API limit.

For services, the tmpfs mount is sized to the secret's actual size plus a small overhead. Docker does not allow specifying a custom tmpfs size for secrets. If a secret is 400 KB, the tmpfs size is 400 KB plus a few bytes. If a service mounts multiple secrets, each gets its own tmpfs mount, and the total memory usage is the sum of the secret sizes plus per-mount overhead (typically 4 KB per mount).

Worked example: A service mounts 8 secrets, each 200 KB. Total secret data = 1.6 MB. Overhead per mount ~ 4 KB, so total ~ 1.632 MB. If the container has a memory limit of 64 MB, that is less than 3% of memory, negligible. But if you have 50 services on the same node each mounting those 8 secrets, the per-container memory still stays low, but the Raft storage on managers could grow if the secrets are large and numerous.

Monitoring secret usage requires checking the container memory cgroup for the tmpfs mounts. Use `docker stats` or `docker inspect` for memory usage. There is no built-in metric for secret tmpfs specifically, but you can see the mounts:

```bash
docker inspect <container_id> | jq '.[0].Mounts[] | select(.Type=="tmpfs")'
```

Output shows `Source`, `Destination`, `Type`, and `Options`. The `Source` is empty for tmpfs; the `Options` contain `size` if explicitly set, but for secrets, Docker sets it automatically. If you see `size=524288` for a 512 KB secret, that confirms the mount size.

## Safe Configuration Path

When changing secret configurations, use a staged approach: define the secret, assign it to a test service, verify, then roll out to production. Follow these steps:

1. Create a secret from a file or stdin. Use a placeholder for testing:

```bash
echo "test_secret_value" | docker secret create test_secret -
```

This avoids storing the real value in shell history. For a real secret, use a file with restricted permissions.

2. Attach the secret to a service in a test environment:

```bash
docker service create --name test-app --secret test_secret alpine cat /run/secrets/test_secret
```

Expected output: the service starts and prints `test_secret_value`, then exits. Check service logs:

```bash
docker service logs test-app
```

3. Verify the secret is mounted and readable, but not writable:

```bash
docker exec $(docker ps -q --filter name=test-app) ls -l /run/secrets/test_secret
```

Output should show `-r--r--r--`. If the application needs write access, that is a design problem; secrets are meant to be read-only.

4. Roll back if needed: if the secret causes issues, remove the secret from the service without deleting the secret object itself:

```bash
docker service update --secret-rm test_secret test-app
```

Then observe. To remove the secret entirely, first ensure no service uses it:

```bash
docker secret rm test_secret
```

If a service still uses it, you get `Error response from daemon: rpc error: code = InvalidArgument desc = secret 'test_secret' is in use by the following service: test-app`.

For Compose, configuration changes are made in the YAML file. Use `docker compose config` to validate before applying:

```bash
docker compose config
```

This prints the resolved configuration and catches syntax errors or missing files. Then apply:

```bash
docker compose up -d
```

If the secret file changes, you must recreate the service to pick up the new secret value:

```bash
docker compose up -d --force-recreate
```

Docker does not automatically update running containers when a secret file changes; the secret content is copied into the Raft at creation time. For Swarm, rotating a secret requires creating a new secret under a new name and updating the service to use it, because secret versions are immutable.

## Verification and Diagnostics

Verifying secret delivery and performance requires checking three levels: the service health, the secret mount, and the Raft health on managers.

First, confirm the service is running and has the expected secret references:

```bash
docker service inspect <service_name> --format '{{json .Spec.TaskTemplate.ContainerSpec.Secrets}}' | jq
```

Output is an array of objects with `SecretID`, `SecretName`, and `File.Name`. For example:

```json
[
  {
    "File": {
      "Name": "db_password",
      "UID": "0",
      "GID": "0",
      "Mode": 292
    },
    "SecretID": "vqj0f2a7x4n9k3p1m8t5r6w",
    "SecretName": "db_password"
  }
]
```

The `Mode` 292 is octal 0444 (read-only for all). To change the UID/GID or mode, you can specify them in the compose file or using `docker service update` with `--secret-add` and `--secret-rm`.

Next, verify the secret is correctly mounted inside the container:

```bash
docker exec <container_id> cat /run/secrets/db_password
```

This should output the secret value. If the file is empty or the path is wrong, check the service definition and the secret name case sensitivity (secret names are case-sensitive).

For diagnostics when a service fails to start due to missing secret, check the service tasks:

```bash
docker service ps <service_name> --no-trunc
```

Look for `Rejected` or `Failed` states. A common error is `secret not found: <name>`, which means the secret was deleted or never created in the swarm.

Monitor Raft health with:

```bash
docker info | grep -A5 'Raft'
```

Look for `Raft: Snapshot: ...` and `Nodes: ...`. If the Raft log is too large (hundreds of MB), the manager may become slow. Use `docker node ls` to ensure all managers are reachable.

To test secret size limits without consuming space, create a placeholder secret close to 500 KB:

```bash
head -c 499K /dev/urandom | base64 > /tmp/big_secret.txt
docker secret create big_secret /tmp/big_secret.txt
```

If it succeeds, the secret size is within limit. If you attempt 501 KB, it will fail with the error mentioned earlier.

## Failure Modes and Recovery

Several failure modes affect Docker Secrets capacity and availability.

**Secret too large**: Creating a secret over 500 KB fails. The creation attempt returns an error immediately, and no secret is created. Recovery: reduce the secret size by splitting it into multiple secrets or by compressing the data if possible. Do not try to bypass the limit by mounting a file from a volume; that defeats the purpose of secrets.

**Secret in use cannot be removed**: `docker secret rm` fails if any service references the secret. Recovery: update all services to remove the secret reference, then delete the secret. For Swarm:

```bash
docker service update --secret-rm <secret_name> <service_name>
```

For Compose, remove the secret from the service's `secrets:` list and run `docker compose up -d`.

**Secret rotation breakage**: When you rotate a secret by creating a new one and updating the service, the service may restart and fail if the application expects a specific filename. Recovery: use the `target` parameter in secrets to control the mounted filename. For example, in Compose:

```yaml
secrets:
  db_password_v2:
    file: ./new_password.txt
services:
  db:
    secrets:
      - source: db_password_v2
        target: db_password
```

This keeps the mount at `/run/secrets/db_password` even though the secret name changed. Test this in a non-production environment first.

**Secrets lost due to manager quorum loss**: If a majority of manager nodes are lost, the swarm loses quorum and cannot process new secret creation or updates. Recovery: restore enough managers to regain quorum, or force a new cluster if all managers are lost, but secrets are stored in Raft and may be unrecoverable without backups. Always back up the Raft directory (`/var/lib/docker/swarm/`) on a manager, but be aware it contains encrypted secret data; the encryption key is stored there as well.

**Container cannot read secret due to permissions**: The default mode is 0444 (read-only for all), but if you set `mode: 0400`, only root can read it. If the container runs as non-root and you did not set UID/GID, the application gets permission denied. Recovery: set the UID/GID to the application user in the secret definition.

```yaml
secrets:
  app_secret:
    file: ./secret.txt
    uid: "1000"
    gid: "1000"
    mode: 0400
```

Then recreate the service.

**Memory pressure from many secrets**: If a service mounts dozens of large secrets, the tmpfs mounts consume memory. If the container hits its memory limit, it may be OOM-killed. Recovery: reduce the number of secrets or their sizes, or increase the container memory limit. Monitor with `docker stats` and set alerts.

## Common Pitfalls and How to Avoid Them

1. **Using environment variables for secrets**: Many teams start with `environment: DB_PASSWORD=...` and then later try to move to Docker Secrets. The secret is visible in `docker inspect` and in process lists. Avoid by using Docker Secrets from the start. For local development, use a `.env` file that is gitignored, but in production use secrets.

2. **Baking secrets into images**: Hardcoding credentials in a Dockerfile copies them into the image layers. Even if you delete the line later, the secret remains in the image history. Use multi-stage builds and avoid copying secret files into the final image. If you must use a secret at build time, use BuildKit secrets with `--mount=type=secret`.

3. **Not planning for rotation**: Secret rotation is not automatic; you must create a new secret and update the service. Plan a rotation schedule (e.g., every 90 days) and automate the update process. Use the `target` option to keep filename stable. Document the rotation owner (e.g., "Priya Shah, Engineering Lead") and revisit it monthly.

4. **Ignoring Raft storage growth**: Large secrets fill the Raft log, causing slower consensus and possible availability issues. Monitor `/var/lib/docker/swarm/raft` size weekly. The operations owner (e.g., "DevOps rotation") should alert at 200 MB and take action at 500 MB.

5. **Mixing Compose v1 and v2**: The old `docker-compose` (v1) does not support secrets, causing `unsupported Compose file version` errors. Always use the Compose plugin `docker compose` (v2). Run `docker compose version` to confirm.

6. **Creating secrets from uncontrolled sources**: If you create a secret from a file that contains a trailing newline, the secret value includes that newline, which can break applications. Use `printf` instead of `echo` to avoid extra newline, or trim the file. Always test with a known placeholder.

7. **Overlooking secret name collisions**: In a swarm, secret names are unique. Creating a secret with an existing name fails with `Error response from daemon: rpc error: code = AlreadyExists`. Use versioning in names (e.g., `db_password_v1`, `db_password_v2`) to avoid conflicts during rotation.

## Operations Checklist for Secret Capacity

Use this checklist before deploying or scaling secrets. Each item includes a concrete command or action, the expected result, and the owner.

| Item | Action | Expected Result | Owner | Frequency |
|------|--------|-----------------|-------|-----------|
| Docker version check | `docker version --format '{{.Server.Version}}'` | Version >= 24.0 | Release engineer | Monthly |
| Swarm mode check | `docker info --format '{{.Swarm.LocalNodeState}}'` | `active` or intentionally inactive | DevOps lead | Weekly |
| Secret list audit | `docker secret ls` | No unused secrets older than 90 days | Security officer | Quarterly |
| Secret size audit | For each secret, check source file size | All < 500 KB, ideally < 100 KB | Developer owner | Monthly |
| Service secret mounts | `docker service inspect <svc> --format '{{json .Spec.TaskTemplate.ContainerSpec.Secrets}}'` | Only necessary secrets mounted | Service owner | Monthly |
| Raft log size | `du -sh /var/lib/docker/swarm/raft` | < 200 MB | Infrastructure owner | Weekly |
| Secret rotation / test | Create test secret and attach to staging service | Service starts and can read secret | Release manager | Each rotation |
| Recovery drill | Simulate missing secret and restore in staging | Recovery time < 15 min | DevOps engineer | Quarterly |

Each item should have a single accountable owner (not a group), as listed. Revisit the table monthly in a production readiness meeting.

## Conclusion

Docker Secrets capacity planning is not a one-time calculation; it requires ongoing monitoring, version awareness, and clear ownership. The practical limits—500 KB per secret, Raft storage growth, tmpfs memory, and service mount counts—are only useful when you observe them in your environment.

A reliable workflow makes failure visible: run the read-only commands, record the current state, make one scoped change, verify with a concrete signal, and define recovery before an incident forces the decision. Protect sensitive values by using placeholders in examples and real secrets only in controlled files.

Document your secret inventory, rotation schedule, and size thresholds. Assign owners for auditing and recovery. Test secret delivery in a staging environment with a restart. These habits prevent surprises and keep your capacity planning grounded in actual data rather than guesswork.