Docker Secrets Capacity Planning with Practical Examples

Intro

Docker Secrets capacity planning with practical examples means moving from an observed problem to a verified result. You should start by identifying the installed Docker version, the deployment topology (Swarm, Compose, or standalone), the operating system and kernel, and the exact component you are inspecting. Do not guess: run a read-only command first, record the current state, then decide whether a change is justified.

This article targets developers, DevOps consultants, and technical startup teams who need to plan secret capacity before it becomes a bottleneck. It connects Docker Secrets scaling, resources, limits, and sizing to concrete commands, expected output, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of real secrets in examples, verify the result, and document how to recover if the expected state is not reached.

Docker Secrets is a feature of Docker Swarm mode and, with the Compose secrets top-level element, can be used with docker compose even on a single node. Secrets are stored encrypted in the Raft log and made available to services as files under /run/secrets/. Capacity planning involves understanding how many secrets a service can mount, how large a secret can be, how many services can use a secret, how encryption and rotation affect performance, and how to monitor for saturation.

Version and Environment Inventory

Before sizing anything, confirm the Docker version, the running mode, and the storage driver. This is critical because secret size limits and behavior differ between Docker versions and between Swarm and Compose. Run the following read-only checks:

docker version --format '{{.Server.Version}}'
docker info --format '{{.Swarm.LocalNodeState}} {{.Driver}}'

Expected output should show the server version (for example, 24.0.7) and either active for an active Swarm node or inactive if you are not in Swarm mode. If the node is not in Swarm mode, docker secret commands will fail with Error response from daemon: This node is not a swarm manager.

If you use Docker Compose, check the Compose plugin version:

docker compose version

Typical output: Docker Compose version v2.23.0. Compose supports secrets only with the Compose Specification and a compatible runtime; older docker-compose (v1) does not support secrets.

Confirm where existing secrets are stored and how they are mounted. List secrets with:

docker secret ls

Output columns: ID, NAME, CREATED, UPDATED. For example:

ID                          NAME                CREATED             UPDATED
vqj0f2a7x4n9k3p1m8t5r6w    db_password         2 days ago          2 days ago

Inspect a specific secret's metadata (not its value):

docker secret inspect db_password

This returns JSON with ID, Version, CreatedAt, UpdatedAt, and Spec.Name. The actual secret value is never shown.

For a Compose project, list secrets defined in the compose file and their files:

docker compose config --secrets

This reveals how many secrets are wired to services and whether any are unused.

When data is involved, confirm where files are stored before changing containers. Docker Secrets are not stored in a named volume; they are encrypted in the Raft log and mounted as a tmpfs at /run/secrets/<secret_name> inside the container. A bind mount or volume can be used for application data, but secrets should always use the Docker Secrets mechanism, never environment variables or files baked into the image.

For example, a service defined as:

services:
  db:
    image: postgres:16
    secrets:
      - db_password
secrets:
  db_password:
    file: ./secrets/db_password.txt

will have the secret available at /run/secrets/db_password inside the db container. Verify with:

docker compose exec db ls -l /run/secrets/

Expected output:

-r--r--r-- 1 root root 24 Jan 15 10:00 db_password

The file is read-only and owned by root. If the application runs as a non-root user, you may need to adjust ownership or use a group with read access, as discussed later.

As part of environment inventory, perform a restart test: create a service, add a secret, stop the container, recreate it, and confirm the application still sees the secret. For Swarm:

docker service create --name secret-test --secret db_password alpine sleep 3600
docker service ps secret-test
docker service rm secret-test
docker service create --name secret-test --secret db_password alpine sleep 3600

Then exec into the new container and verify /run/secrets/db_password exists. If the secret is missing, the service definition or secret reference is wrong.

Understanding Docker Secrets Limits and Sizing

Docker Secrets has documented and undocumented limits that directly affect capacity planning. The two hard limits are:

• Maximum secret size: 500 KB per secret. This is enforced by the API; any attempt to create a larger secret returns Error response from daemon: rpc error: code = InvalidArgument desc = invalid secret: exceeds maximum size of 500KB.
• Maximum number of secrets per service: no official hard limit, but the practical limit is determined by the total size of mounted tmpfs and the container's memory limits. Every mounted secret consumes memory in the container's tmpfs. For a service with 10 secrets each 100 KB, the tmpfs uses at least 1 MB of memory, plus overhead.

Swarm stores secrets in the Raft log, which is replicated to all manager nodes. A 500 KB secret is stored in Raft, and every manager keeps a copy. If you have 5 managers and 10 such secrets, that is 25 MB of Raft storage just for secrets. Raft log size can affect performance if it grows too large; Docker recommends keeping the Raft log under a few hundred megabytes.

To check the current Raft log size on a manager:

du -sh /var/lib/docker/swarm/raft/

If the directory grows beyond expectations, inspect the number and size of secrets:

docker secret ls -q | xargs -I {} sh -c 'docker secret inspect {} | jq -r ".[0].Spec.Name, .[0].ID"'

But secret values are not stored on disk in plaintext, so their individual sizes are not directly exposed. You can approximate by reading the source files before creating the secret. Use placeholders:

head -c 500K /dev/zero | base64 > secret_placeholder.txt

Then create the secret from that placeholder to test the API limit.

For services, the tmpfs mount is sized to the secret's actual size plus a small overhead. Docker does not allow specifying a custom tmpfs size for secrets. If a secret is 400 KB, the tmpfs size is 400 KB plus a few bytes. If a service mounts multiple secrets, each gets its own tmpfs mount, and the total memory usage is the sum of the secret sizes plus per-mount overhead (typically 4 KB per mount).

Worked example: A service mounts 8 secrets, each 200 KB. Total secret data = 1.6 MB. Overhead per mount ~ 4 KB, so total ~ 1.632 MB. If the container has a memory limit of 64 MB, that is less than 3% of memory, negligible. But if you have 50 services on the same node each mounting those 8 secrets, the per-container memory still stays low, but the Raft storage on managers could grow if the secrets are large and numerous.

Monitoring secret usage requires checking the container memory cgroup for the tmpfs mounts. Use docker stats or docker inspect for memory usage. There is no built-in metric for secret tmpfs specifically, but you can see the mounts:

docker inspect <container_id> | jq '.[0].Mounts[] | select(.Type=="tmpfs")'

Output shows Source, Destination, Type, and Options. The Source is empty for tmpfs; the Options contain size if explicitly set, but for secrets, Docker sets it automatically. If you see size=524288 for a 512 KB secret, that confirms the mount size.

Safe Configuration Path

When changing secret configurations, use a staged approach: define the secret, assign it to a test service, verify, then roll out to production. Follow these steps:

1. Create a secret from a file or stdin. Use a placeholder for testing:

echo "test_secret_value" | docker secret create test_secret -

This avoids storing the real value in shell history. For a real secret, use a file with restricted permissions.

1. Attach the secret to a service in a test environment:

docker service create --name test-app --secret test_secret alpine cat /run/secrets/test_secret

Expected output: the service starts and prints test_secret_value, then exits. Check service logs:

docker service logs test-app

1. Verify the secret is mounted and readable, but not writable:

docker exec $(docker ps -q --filter name=test-app) ls -l /run/secrets/test_secret

Output should show -r--r--r--. If the application needs write access, that is a design problem; secrets are meant to be read-only.

1. Roll back if needed: if the secret causes issues, remove the secret from the service without deleting the secret object itself:

docker service update --secret-rm test_secret test-app

Then observe. To remove the secret entirely, first ensure no service uses it:

docker secret rm test_secret

If a service still uses it, you get Error response from daemon: rpc error: code = InvalidArgument desc = secret 'test_secret' is in use by the following service: test-app.

For Compose, configuration changes are made in the YAML file. Use docker compose config to validate before applying:

docker compose config

This prints the resolved configuration and catches syntax errors or missing files. Then apply:

docker compose up -d

If the secret file changes, you must recreate the service to pick up the new secret value:

docker compose up -d --force-recreate

Docker does not automatically update running containers when a secret file changes; the secret content is copied into the Raft at creation time. For Swarm, rotating a secret requires creating a new secret under a new name and updating the service to use it, because secret versions are immutable.

Verification and Diagnostics

Verifying secret delivery and performance requires checking three levels: the service health, the secret mount, and the Raft health on managers.

First, confirm the service is running and has the expected secret references:

docker service inspect <service_name> --format '{{json .Spec.TaskTemplate.ContainerSpec.Secrets}}' | jq

Output is an array of objects with SecretID, SecretName, and File.Name. For example:

[
  {
    "File": {
      "Name": "db_password",
      "UID": "0",
      "GID": "0",
      "Mode": 292
    },
    "SecretID": "vqj0f2a7x4n9k3p1m8t5r6w",
    "SecretName": "db_password"
  }
]

The Mode 292 is octal 0444 (read-only for all). To change the UID/GID or mode, you can specify them in the compose file or using docker service update with --secret-add and --secret-rm.

Next, verify the secret is correctly mounted inside the container:

docker exec <container_id> cat /run/secrets/db_password

This should output the secret value. If the file is empty or the path is wrong, check the service definition and the secret name case sensitivity (secret names are case-sensitive).

For diagnostics when a service fails to start due to missing secret, check the service tasks:

docker service ps <service_name> --no-trunc

Look for Rejected or Failed states. A common error is secret not found: <name>, which means the secret was deleted or never created in the swarm.

Monitor Raft health with:

docker info | grep -A5 'Raft'

Look for Raft: Snapshot: ... and Nodes: .... If the Raft log is too large (hundreds of MB), the manager may become slow. Use docker node ls to ensure all managers are reachable.

To test secret size limits without consuming space, create a placeholder secret close to 500 KB:

head -c 499K /dev/urandom | base64 > /tmp/big_secret.txt
docker secret create big_secret /tmp/big_secret.txt

If it succeeds, the secret size is within limit. If you attempt 501 KB, it will fail with the error mentioned earlier.

Failure Modes and Recovery

Several failure modes affect Docker Secrets capacity and availability.

Secret too large: Creating a secret over 500 KB fails. The creation attempt returns an error immediately, and no secret is created. Recovery: reduce the secret size by splitting it into multiple secrets or by compressing the data if possible. Do not try to bypass the limit by mounting a file from a volume; that defeats the purpose of secrets.

Secret in use cannot be removed: docker secret rm fails if any service references the secret. Recovery: update all services to remove the secret reference, then delete the secret. For Swarm:

docker service update --secret-rm <secret_name> <service_name>

For Compose, remove the secret from the service's secrets: list and run docker compose up -d.

Secret rotation breakage: When you rotate a secret by creating a new one and updating the service, the service may restart and fail if the application expects a specific filename. Recovery: use the target parameter in secrets to control the mounted filename. For example, in Compose:

secrets:
  db_password_v2:
    file: ./new_password.txt
services:
  db:
    secrets:
      - source: db_password_v2
        target: db_password

This keeps the mount at /run/secrets/db_password even though the secret name changed. Test this in a non-production environment first.

Secrets lost due to manager quorum loss: If a majority of manager nodes are lost, the swarm loses quorum and cannot process new secret creation or updates. Recovery: restore enough managers to regain quorum, or force a new cluster if all managers are lost, but secrets are stored in Raft and may be unrecoverable without backups. Always back up the Raft directory (/var/lib/docker/swarm/) on a manager, but be aware it contains encrypted secret data; the encryption key is stored there as well.

Container cannot read secret due to permissions: The default mode is 0444 (read-only for all), but if you set mode: 0400, only root can read it. If the container runs as non-root and you did not set UID/GID, the application gets permission denied. Recovery: set the UID/GID to the application user in the secret definition.

secrets:
  app_secret:
    file: ./secret.txt
    uid: "1000"
    gid: "1000"
    mode: 0400

Then recreate the service.

Memory pressure from many secrets: If a service mounts dozens of large secrets, the tmpfs mounts consume memory. If the container hits its memory limit, it may be OOM-killed. Recovery: reduce the number of secrets or their sizes, or increase the container memory limit. Monitor with docker stats and set alerts.

Common Pitfalls and How to Avoid Them

1. Using environment variables for secrets: Many teams start with environment: DB_PASSWORD=... and then later try to move to Docker Secrets. The secret is visible in docker inspect and in process lists. Avoid by using Docker Secrets from the start. For local development, use a .env file that is gitignored, but in production use secrets.

1. Baking secrets into images: Hardcoding credentials in a Dockerfile copies them into the image layers. Even if you delete the line later, the secret remains in the image history. Use multi-stage builds and avoid copying secret files into the final image. If you must use a secret at build time, use BuildKit secrets with --mount=type=secret.

1. Not planning for rotation: Secret rotation is not automatic; you must create a new secret and update the service. Plan a rotation schedule (e.g., every 90 days) and automate the update process. Use the target option to keep filename stable. Document the rotation owner (e.g., "Priya Shah, Engineering Lead") and revisit it monthly.

1. Ignoring Raft storage growth: Large secrets fill the Raft log, causing slower consensus and possible availability issues. Monitor /var/lib/docker/swarm/raft size weekly. The operations owner (e.g., "DevOps rotation") should alert at 200 MB and take action at 500 MB.

1. Mixing Compose v1 and v2: The old docker-compose (v1) does not support secrets, causing unsupported Compose file version errors. Always use the Compose plugin docker compose (v2). Run docker compose version to confirm.

1. Creating secrets from uncontrolled sources: If you create a secret from a file that contains a trailing newline, the secret value includes that newline, which can break applications. Use printf instead of echo to avoid extra newline, or trim the file. Always test with a known placeholder.

1. Overlooking secret name collisions: In a swarm, secret names are unique. Creating a secret with an existing name fails with Error response from daemon: rpc error: code = AlreadyExists. Use versioning in names (e.g., db_password_v1, db_password_v2) to avoid conflicts during rotation.

Operations Checklist for Secret Capacity

Use this checklist before deploying or scaling secrets. Each item includes a concrete command or action, the expected result, and the owner.

Each item should have a single accountable owner (not a group), as listed. Revisit the table monthly in a production readiness meeting.

Conclusion

Docker Secrets capacity planning is not a one-time calculation; it requires ongoing monitoring, version awareness, and clear ownership. The practical limits—500 KB per secret, Raft storage growth, tmpfs memory, and service mount counts—are only useful when you observe them in your environment.

A reliable workflow makes failure visible: run the read-only commands, record the current state, make one scoped change, verify with a concrete signal, and define recovery before an incident forces the decision. Protect sensitive values by using placeholders in examples and real secrets only in controlled files.

Document your secret inventory, rotation schedule, and size thresholds. Assign owners for auditing and recovery. Test secret delivery in a staging environment with a restart. These habits prevent surprises and keep your capacity planning grounded in actual data rather than guesswork.