## Intro

Managing sensitive data like passwords, API keys, and TLS certificates inside containers is a common operational challenge. Storing secrets in images, environment variables, or application code creates security risks and complicates rotation. Docker Secrets provides a built-in mechanism for securely transmitting and managing secrets in Docker Swarm.

This guide focuses on the commands and workflows you need to work with Docker Secrets effectively. You will learn how to create, inspect, use, and rotate secrets, along with practical examples and troubleshooting techniques. The goal is operational safety: understand the current state before making changes, limit the blast radius, protect actual secret values, and always verify the result.

## Prerequisites and Environment Inventory

Before using Docker Secrets, you must be running Docker in Swarm mode. Secrets are a Swarm feature and are not available for standalone containers. Check your Docker version and Swarm status with:

```bash
docker version
```

Look for the server version and ensure it is 1.13 or higher (the version that introduced secrets). To verify Swarm mode is active:

```bash
docker info --format '{{.Swarm.LocalNodeState}}'
```

The expected output is `active`. If it is `inactive`, initialize a swarm with:

```bash
docker swarm init
```

On a manager node, you can manage secrets. Worker nodes can access secrets only if the service is scheduled there.

A quick environment inventory helps identify what is running and where secrets might be in use. Use these commands:

```bash
# List running containers with names, status, and ports
docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

# List services in the swarm
docker service ls

# Inspect a service to see its secret references
docker service inspect <service-name> --pretty
```

For example, you might see a service named `web` that references a secret called `db_password`.

When dealing with data volumes, it is important to know where files are stored. A named volume like `app_data:/var/lib/app` is managed by Docker and persists across container rebuilds. A bind mount like `./data:/var/lib/app` maps a host directory and is useful for development but can cause permission and portability issues. Test persistence by stopping, recreating, and checking the data.

## Creating and Managing Secrets

### Creating Secrets

There are several ways to create a secret. The most common is from a file or from standard input.

**From a file:**

The file should contain the secret value without trailing newlines. For example, create a file `db_password.txt` with the content `s3cr3tP@ss`. Then run:

```bash
docker secret create db_password db_password.txt
```

This command outputs the secret ID, like `a1b2c3d4e5f6`. The secret is now stored encrypted in the Swarm's Raft log and available to services.

**From standard input:**

You can pipe a value directly, but be careful with shell history. For example:

```bash
echo "s3cr3tP@ss" | docker secret create db_password -
```

Using `-` as the source tells Docker to read from stdin. Note that using `echo` may expose the secret in process lists. Safer alternatives use a password manager or a file.

**Using a label:**

You can add labels to secrets for easier management:

```bash
docker secret create --label env=prod db_password db_password.txt
```

Labels help organize secrets across environments.

### Listing Secrets

To see all secrets in the swarm:

```bash
docker secret ls
```

Example output:

```
ID                          NAME          CREATED             UPDATED
abc123def456                db_password   5 minutes ago       5 minutes ago
xyz789uvw012                api_key       2 hours ago         2 hours ago
```

This shows the secret IDs, names, and timestamps. Note that secret values are never displayed.

### Inspecting Secrets

To view metadata about a secret:

```bash
docker secret inspect db_password
```

This returns JSON, including the ID, name, labels, and creation date, but never the secret value. Example:

```json
[
  {
    "ID": "abc123def456",
    "Version": {
      "Index": 1
    },
    "CreatedAt": "2025-03-15T10:00:00Z",
    "UpdatedAt": "2025-03-15T10:00:00Z",
    "Spec": {
      "Name": "db_password",
      "Labels": {}
    }
  }
]
```

### Removing Secrets

To remove a secret that is no longer needed:

```bash
docker secret rm db_password
```

Note: You cannot remove a secret that is currently in use by a running service. First, update or remove the service, then remove the secret.

## Using Secrets in Services

Secrets are mounted as files inside containers at `/run/secrets/<secret-name>`. To use a secret, you need to grant a service access to it.

### Creating a Service with a Secret

This example creates a service with a secret:

```bash
docker service create \
  --name web \
  --secret db_password \
  --publish 80:80 \
  nginx:latest
```

The secret will be mounted at `/run/secrets/db_password` inside the container. You can verify by execing into the running container:

```bash
docker exec -it $(docker ps -q -f name=web) cat /run/secrets/db_password
```

This prints the secret value. Be cautious with this command in production.

You can specify a custom target path:

```bash
docker service create \
  --name web \
  --secret source=db_password,target=/etc/secrets/db_password \
  --publish 80:80 \
  nginx:latest
```

The secret file will be at `/etc/secrets/db_password`.

You can also set file permissions for the secret:

```bash
docker service create \
  --name web \
  --secret source=db_password,target=/run/secrets/db_password,mode=0400 \
  --publish 80:80 \
  nginx:latest
```

This restricts read access to the file owner only.

### Updating a Service to Add or Remove Secrets

To add a secret to an existing service:

```bash
docker service update --secret-add db_password web
```

To remove a secret:

```bash
docker service update --secret-rm db_password web
```

Note: This will trigger a rolling update of the service, recreating the containers.

### Using Secrets in Docker Compose

In Compose files (version 3.1+), you can define secrets in the `secrets` top-level key. Example:

```yaml
version: '3.8'

services:
  web:
    image: nginx:latest
    secrets:
      - db_password
    environment:
      - DB_PASSWORD_FILE=/run/secrets/db_password
    deploy:
      replicas: 2
    ports:
      - "80:80"

secrets:
  db_password:
    external: true
```

Here, the secret `db_password` must already exist in the swarm. The service mounts it at `/run/secrets/db_password`, and the application can read the value from that file.

Alternatively, you can define the secret content in the Compose file using `file:`:

```yaml
secrets:
  db_password:
    file: ./db_password.txt
```

However, keeping secret values out of the Compose file and source control is safer.

## Rotating Secrets

Secret rotation is critical for security. Docker does not automatically rotate secrets, so you must create a new secret and update services to use it. Here is a safe rotation procedure:

1. Create a new secret with a new value:

```bash
docker secret create db_password_v2 new_secret_value.txt
```

2. Update the service to use both old and new secrets temporarily, allowing applications to accept both. For example, if your application can read multiple secret files, add the new secret:

```bash
docker service update --secret-add db_password_v2 web
```

3. Update the application to use the new secret and then remove the old one from the service:

```bash
docker service update --secret-rm db_password web
```

4. Remove the old secret from the swarm after confirming no service uses it:

```bash
docker secret rm db_password
```

This process minimizes downtime and avoids locking out services.

## Troubleshooting and Diagnostics

### Secret Not Accessible in Container

If the secret file is missing or unreadable, check the following:

- Ensure the service has the secret attached. Use `docker service inspect <service> --pretty` and look for the `Secrets` section.
- Check the target path. By default, it is `/run/secrets/<name>`. If you specified a custom target, ensure the application reads from that path.
- Check file permissions. If you set a restrictive mode, ensure the user running the application has read access.
- If the service was created before the secret was added, update the service with `--secret-add`.

### Removing a Secret in Use

If you try `docker secret rm db_password` and get an error like `Error response from daemon: rpc error: code = InvalidArgument desc = secret 'db_password' is in use by the following service: web`, you must first remove the secret from all services. Use `docker service update --secret-rm db_password <service>` for each service, then retry removal.

### Secret Values and Shell Exposure

Avoid exposing secret values in shell history or process lists. Instead of `echo "secret" | docker secret create`, use a file generated from a password manager or a here-document without command line exposure. For example:

```bash
docker secret create db_password - <<EOF
s3cr3tP@ss
EOF
```

This still may expose in some shells; using a file is safer.

### Debugging with Docker Logs and Exec

To see recent logs from a service container:

```bash
docker service logs web --tail 100
```

To enter a running container for inspection:

```bash
docker exec -it <container-id> sh
```

Then check the secret file:

```bash
cat /run/secrets/db_password
```

Use these commands to verify that the secret is present and contains the expected value.

## Common Pitfalls and Mistakes

### Storing Secrets in Images or Environment Variables

**Mistake:** Embedding secrets in Dockerfiles or using environment variables like `ENV DB_PASSWORD=secret`.

**Why it happens:** It seems convenient and works in local development.

**How to avoid:** Use Docker Secrets for any sensitive data in Swarm. For local development without Swarm, consider using a secrets manager or `.env` files that are excluded from version control. Environment variables are visible in `docker inspect` and can be leaked in logs or process listings.

### Not Rotating Secrets Regularly

**Mistake:** Using the same secret indefinitely.

**Why it happens:** Rotation is manual and can cause downtime if not done carefully.

**How to avoid:** Implement a rotation schedule (e.g., every 90 days) and use the multi-secret approach described earlier to avoid downtime. Automate the process with scripts or CI/CD pipelines.

### Creating Secrets with Trailing Newlines

**Mistake:** When creating a secret from a file, the file may contain a trailing newline. This newline becomes part of the secret value and can cause authentication failures.

**Why it happens:** Text editors often add a newline at the end of a file.

**How to avoid:** Use `printf` without a trailing newline to create the file, or strip newlines before creating the secret:

```bash
printf 's3cr3tP@ss' > db_password.txt
docker secret create db_password db_password.txt
```

### Forgetting to Remove Unused Secrets

**Mistake:** Leaving old secrets in the swarm after rotation or service removal.

**Why it happens:** It is easy to forget or assume they are harmless.

**How to avoid:** Regularly audit secrets with `docker secret ls` and remove those not in use. Use labels to track ownership and expiration.

### Mismanaging Secret Permissions

**Mistake:** Setting secret file permissions too open (e.g., `0644`) or too restrictive (e.g., `0000`) causing application errors.

**Why it happens:** Lack of understanding of container user contexts.

**How to avoid:** Use `mode=0400` for most secrets and ensure the application runs as a user that can read the file. Test in a non-production environment.

## Best Practices and Security Considerations

- **Least privilege:** Grant secrets only to services that need them. Avoid using `--secret` broadly; attach per service.
- **Encryption in transit and at rest:** Secrets are encrypted during transmission to containers and stored encrypted in the Raft log. However, they are stored unencrypted on container filesystems; minimize access.
- **Regular rotation:** Rotate secrets periodically and after any suspected compromise.
- **Avoid logging secrets:** Ensure applications do not log secret values. Use monitoring to detect accidental exposure.
- **Backup secrets securely:** If you need to backup secrets, use encrypted storage and strict access controls.
- **Use labels for management:** Label secrets with environment, owner, and expiration date to facilitate audits.

## Conclusion

Docker Secrets is a powerful feature for managing sensitive data in Swarm clusters. By following the commands and examples in this guide, you can securely create, use, and rotate secrets while minimizing operational risks. Remember to verify your environment, use safe practices, and regularly audit your secrets.

Next steps: pick one low-risk service, attach a secret, and verify its accessibility inside the container. Then plan a rotation procedure and document it for your team. With careful management, Docker Secrets can be a cornerstone of your container security posture.