Intro
Managing sensitive data like passwords, API keys, and TLS certificates inside containers is a common operational challenge. Storing secrets in images, environment variables, or application code creates security risks and complicates rotation. Docker Secrets provides a built-in mechanism for securely transmitting and managing secrets in Docker Swarm.
This guide focuses on the commands and workflows you need to work with Docker Secrets effectively. You will learn how to create, inspect, use, and rotate secrets, along with practical examples and troubleshooting techniques. The goal is operational safety: understand the current state before making changes, limit the blast radius, protect actual secret values, and always verify the result.
Prerequisites and Environment Inventory
Before using Docker Secrets, you must be running Docker in Swarm mode. Secrets are a Swarm feature and are not available for standalone containers. Check your Docker version and Swarm status with:
docker version
Look for the server version and ensure it is 1.13 or higher (the version that introduced secrets). To verify Swarm mode is active:
docker info --format '{{.Swarm.LocalNodeState}}'
The expected output is active. If it is inactive, initialize a swarm with:
docker swarm init
On a manager node, you can manage secrets. Worker nodes can access secrets only if the service is scheduled there.
A quick environment inventory helps identify what is running and where secrets might be in use. Use these commands:
# List running containers with names, status, and ports
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"
# List services in the swarm
docker service ls
# Inspect a service to see its secret references
docker service inspect <service-name> --pretty
For example, you might see a service named web that references a secret called db_password.
When dealing with data volumes, it is important to know where files are stored. A named volume like app_data:/var/lib/app is managed by Docker and persists across container rebuilds. A bind mount like ./data:/var/lib/app maps a host directory and is useful for development but can cause permission and portability issues. Test persistence by stopping, recreating, and checking the data.
Creating and Managing Secrets
Creating Secrets
There are several ways to create a secret. The most common is from a file or from standard input.
From a file:
The file should contain the secret value without trailing newlines. For example, create a file db_password.txt with the content s3cr3tP@ss. Then run:
docker secret create db_password db_password.txt
This command outputs the secret ID, like a1b2c3d4e5f6. The secret is now stored encrypted in the Swarm's Raft log and available to services.
From standard input:
You can pipe a value directly, but be careful with shell history. For example:
echo "s3cr3tP@ss" | docker secret create db_password -
Using - as the source tells Docker to read from stdin. Note that using echo may expose the secret in process lists. Safer alternatives use a password manager or a file.
Using a label:
You can add labels to secrets for easier management:
docker secret create --label env=prod db_password db_password.txt
Labels help organize secrets across environments.
Listing Secrets
To see all secrets in the swarm:
docker secret ls
Example output:
ID NAME CREATED UPDATED
abc123def456 db_password 5 minutes ago 5 minutes ago
xyz789uvw012 api_key 2 hours ago 2 hours ago
This shows the secret IDs, names, and timestamps. Note that secret values are never displayed.
Inspecting Secrets
To view metadata about a secret:
docker secret inspect db_password
This returns JSON, including the ID, name, labels, and creation date, but never the secret value. Example:
[
{
"ID": "abc123def456",
"Version": {
"Index": 1
},
"CreatedAt": "2025-03-15T10:00:00Z",
"UpdatedAt": "2025-03-15T10:00:00Z",
"Spec": {
"Name": "db_password",
"Labels": {}
}
}
]
Removing Secrets
To remove a secret that is no longer needed:
docker secret rm db_password
Note: You cannot remove a secret that is currently in use by a running service. First, update or remove the service, then remove the secret.
Using Secrets in Services
Secrets are mounted as files inside containers at /run/secrets/<secret-name>. To use a secret, you need to grant a service access to it.
Creating a Service with a Secret
This example creates a service with a secret:
docker service create \
--name web \
--secret db_password \
--publish 80:80 \
nginx:latest
The secret will be mounted at /run/secrets/db_password inside the container. You can verify by execing into the running container:
docker exec -it $(docker ps -q -f name=web) cat /run/secrets/db_password
This prints the secret value. Be cautious with this command in production.
You can specify a custom target path:
docker service create \
--name web \
--secret source=db_password,target=/etc/secrets/db_password \
--publish 80:80 \
nginx:latest
The secret file will be at /etc/secrets/db_password.
You can also set file permissions for the secret:
docker service create \
--name web \
--secret source=db_password,target=/run/secrets/db_password,mode=0400 \
--publish 80:80 \
nginx:latest
This restricts read access to the file owner only.
Updating a Service to Add or Remove Secrets
To add a secret to an existing service:
docker service update --secret-add db_password web
To remove a secret:
docker service update --secret-rm db_password web
Note: This will trigger a rolling update of the service, recreating the containers.
Using Secrets in Docker Compose
In Compose files (version 3.1+), you can define secrets in the secrets top-level key. Example:
version: '3.8'
services:
web:
image: nginx:latest
secrets:
- db_password
environment:
- DB_PASSWORD_FILE=/run/secrets/db_password
deploy:
replicas: 2
ports:
- "80:80"
secrets:
db_password:
external: true
Here, the secret db_password must already exist in the swarm. The service mounts it at /run/secrets/db_password, and the application can read the value from that file.
Alternatively, you can define the secret content in the Compose file using file::
secrets:
db_password:
file: ./db_password.txt
However, keeping secret values out of the Compose file and source control is safer.
Rotating Secrets
Secret rotation is critical for security. Docker does not automatically rotate secrets, so you must create a new secret and update services to use it. Here is a safe rotation procedure:
- Create a new secret with a new value:
docker secret create db_password_v2 new_secret_value.txt
- Update the service to use both old and new secrets temporarily, allowing applications to accept both. For example, if your application can read multiple secret files, add the new secret:
docker service update --secret-add db_password_v2 web
- Update the application to use the new secret and then remove the old one from the service:
docker service update --secret-rm db_password web
- Remove the old secret from the swarm after confirming no service uses it:
docker secret rm db_password
This process minimizes downtime and avoids locking out services.
Troubleshooting and Diagnostics
Secret Not Accessible in Container
If the secret file is missing or unreadable, check the following:
- Ensure the service has the secret attached. Use
docker service inspect <service> --prettyand look for theSecretssection. - Check the target path. By default, it is
/run/secrets/<name>. If you specified a custom target, ensure the application reads from that path. - Check file permissions. If you set a restrictive mode, ensure the user running the application has read access.
- If the service was created before the secret was added, update the service with
--secret-add.
Removing a Secret in Use
If you try docker secret rm db_password and get an error like Error response from daemon: rpc error: code = InvalidArgument desc = secret 'db_password' is in use by the following service: web, you must first remove the secret from all services. Use docker service update --secret-rm db_password <service> for each service, then retry removal.
Secret Values and Shell Exposure
Avoid exposing secret values in shell history or process lists. Instead of echo "secret" | docker secret create, use a file generated from a password manager or a here-document without command line exposure. For example:
docker secret create db_password - <<EOF
s3cr3tP@ss
EOF
This still may expose in some shells; using a file is safer.
Debugging with Docker Logs and Exec
To see recent logs from a service container:
docker service logs web --tail 100
To enter a running container for inspection:
docker exec -it <container-id> sh
Then check the secret file:
cat /run/secrets/db_password
Use these commands to verify that the secret is present and contains the expected value.
Common Pitfalls and Mistakes
Storing Secrets in Images or Environment Variables
Mistake: Embedding secrets in Dockerfiles or using environment variables like ENV DB_PASSWORD=secret.
Why it happens: It seems convenient and works in local development.
How to avoid: Use Docker Secrets for any sensitive data in Swarm. For local development without Swarm, consider using a secrets manager or .env files that are excluded from version control. Environment variables are visible in docker inspect and can be leaked in logs or process listings.
Not Rotating Secrets Regularly
Mistake: Using the same secret indefinitely.
Why it happens: Rotation is manual and can cause downtime if not done carefully.
How to avoid: Implement a rotation schedule (e.g., every 90 days) and use the multi-secret approach described earlier to avoid downtime. Automate the process with scripts or CI/CD pipelines.
Creating Secrets with Trailing Newlines
Mistake: When creating a secret from a file, the file may contain a trailing newline. This newline becomes part of the secret value and can cause authentication failures.
Why it happens: Text editors often add a newline at the end of a file.
How to avoid: Use printf without a trailing newline to create the file, or strip newlines before creating the secret:
printf 's3cr3tP@ss' > db_password.txt
docker secret create db_password db_password.txt
Forgetting to Remove Unused Secrets
Mistake: Leaving old secrets in the swarm after rotation or service removal.
Why it happens: It is easy to forget or assume they are harmless.
How to avoid: Regularly audit secrets with docker secret ls and remove those not in use. Use labels to track ownership and expiration.
Mismanaging Secret Permissions
Mistake: Setting secret file permissions too open (e.g., 0644) or too restrictive (e.g., 0000) causing application errors.
Why it happens: Lack of understanding of container user contexts.
How to avoid: Use mode=0400 for most secrets and ensure the application runs as a user that can read the file. Test in a non-production environment.
Best Practices and Security Considerations
- Least privilege: Grant secrets only to services that need them. Avoid using
--secretbroadly; attach per service. - Encryption in transit and at rest: Secrets are encrypted during transmission to containers and stored encrypted in the Raft log. However, they are stored unencrypted on container filesystems; minimize access.
- Regular rotation: Rotate secrets periodically and after any suspected compromise.
- Avoid logging secrets: Ensure applications do not log secret values. Use monitoring to detect accidental exposure.
- Backup secrets securely: If you need to backup secrets, use encrypted storage and strict access controls.
- Use labels for management: Label secrets with environment, owner, and expiration date to facilitate audits.
Conclusion
Docker Secrets is a powerful feature for managing sensitive data in Swarm clusters. By following the commands and examples in this guide, you can securely create, use, and rotate secrets while minimizing operational risks. Remember to verify your environment, use safe practices, and regularly audit your secrets.
Next steps: pick one low-risk service, attach a secret, and verify its accessibility inside the container. Then plan a rotation procedure and document it for your team. With careful management, Docker Secrets can be a cornerstone of your container security posture.