## Intro Docker logging drivers control how container logs are captured, stored, and forwarded. They are a critical part of container security because logs often contain sensitive data such as application errors, user identifiers, internal URLs, and stack traces. Misconfigured logging drivers can expose that data through overly permissive file permissions, unauthenticated remote endpoints, or unencrypted transport. This guide focuses on hardening Docker logging drivers for developers, DevOps consultants, and technical startup teams. It connects core security concepts - hardening, access control, secrets handling, and permissions - to specific Docker commands, configuration snippets, expected outputs, failure signals, and recovery decisions. Every step is designed to be verified safely in a test environment before being applied to production. The goal is operational safety: observe the current state before making changes, limit the blast radius of each change, avoid putting real secrets in configuration files or shell history, verify the result with concrete output, and document a recovery path if the expected state is not reached. ## Version and Environment Inventory Before changing any logging configuration, document the Docker environment. This inventory prevents applying commands to the wrong host, using unsupported options, or missing prerequisites such as a specific Docker version or plugin. Run the following read-only commands to capture the current state: ```bash # Docker version and API version docker version --format '{{.Server.Version}}' # Example output: 24.0.7 # Docker info: logging driver, storage driver, root dir docker info --format 'Logging Driver: {{.LoggingDriver}} | Storage Driver: {{.Driver}} | Docker Root: {{.DockerRootDir}}' # Example output: Logging Driver: json-file | Storage Driver: overlay2 | Docker Root: /var/lib/docker # List running containers with names, status, and ports docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" # Example output: # NAMES STATUS PORTS # web-server Up 2 hours 0.0.0.0:8080->80/tcp # postgres-db Up 2 hours 5432/tcp ``` For a specific container, inspect its current logging configuration: ```bash docker inspect --format '{{ json .HostConfig.LogConfig }}' # Example output: {"Type":"json-file","Config":{"max-size":"10m","max-file":"3"}} ``` If the container is part of a Compose project, use: ```bash docker compose ps docker compose logs -f --tail 50 docker compose exec sh ``` Never run these commands against a production host without first testing on a staging or local environment. Record the output and timestamp before making any changes. If you need to inspect logs that may contain secrets, avoid dumping them directly to the terminal. Redirect to a file with restrictive permissions: ```bash docker logs > /tmp/app-logs.txt 2>&1 chmod 600 /tmp/app-logs.txt ``` Also confirm where persistent data is stored. A named volume such as `app_data:/var/lib/app` is managed by Docker and survives container rebuilds. A bind mount such as `./data:/var/lib/app` maps a host directory and is useful for development, but can introduce permission, portability, and backup issues. Before modifying logging, ensure you know whether logs themselves are stored on a volume, a bind mount, or the container filesystem, because that affects retention and backup strategy. A quick restart test verifies that application data and log configuration survive a container recreate: ```bash docker stop docker rm docker run -d --name [same options] docker logs --tail 20 ``` If logs are lost after this test, the container was likely writing to its writable layer instead of a persistent location, which is a separate but related hardening issue. ## Understanding Docker Logging Drivers and Their Security Implications Docker supports several built-in logging drivers: `json-file`, `journald`, `syslog`, `gelf`, `fluentd`, `awslogs`, `splunk`, and others. The default driver is `json-file`, which writes logs as JSON objects to the host filesystem. Security considerations vary by driver: - **json-file**: logs are stored as files under `/var/lib/docker/containers//`. File permissions are controlled by the Docker daemon's umask setting, which is often `022`, resulting in files readable by all local users. This is a common permission hardening target. - **syslog**: logs are sent to a local or remote syslog daemon. If using UDP, logs are unencrypted and can be spoofed or sniffed on the network. If using TCP, authentication is often absent unless TLS is configured. - **gelf (Graylog Extended Log Format)**: sends logs as UDP or TCP to a Graylog server. UDP is unreliable and insecure without network isolation. TCP can be secured with TLS. - **fluentd**: sends logs to a Fluentd aggregator. Plain TCP or HTTP without TLS exposes log data in transit. - **splunk**: sends logs to Splunk via HTTP Event Collector (HEC). It requires a token, but the channel should be encrypted with HTTPS. - **awslogs**: sends logs to Amazon CloudWatch Logs. It requires IAM credentials with appropriate permissions; hardcoding credentials in container environment variables is a common mistake. Hardening a logging driver means selecting the right driver for your environment, locking down file permissions or network transport, avoiding credential exposure, and ensuring that log rotation and retention do not create security gaps. ## Safe Configuration Path The safest way to change the logging driver or its options is to modify the Docker daemon configuration file (`/etc/docker/daemon.json`) or the per-container configuration. Before editing, back up the current file and validate its syntax. **Step 1: Observe current daemon configuration** ```bash cat /etc/docker/daemon.json # Example output: # { # "log-driver": "json-file", # "log-opts": { # "max-size": "10m", # "max-file": "3" # } # } ``` If the file does not exist, Docker uses default settings. **Step 2: Back up and edit the configuration** ```bash sudo cp /etc/docker/daemon.json /etc/docker/daemon.json.bak.$(date +%Y%m%d) sudo nano /etc/docker/daemon.json ``` For example, to set `json-file` with rotation limits and a restrictive default umask, you could configure: ```json { "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "5", "mode": "non-blocking", "max-buffer-size": "4m" } } ``` Note: `mode` and `max-buffer-size` are available for `json-file` in Docker 20.10 and later. Setting `mode` to `non-blocking` prevents the container from being blocked if the logging driver cannot keep up, but it may drop logs under heavy load. This is a trade-off between availability and log completeness. To change file permissions, you cannot set umask directly in `daemon.json`. Instead, set the system-wide umask for the Docker daemon service. For systemd systems, create an override file: ```bash sudo mkdir -p /etc/systemd/system/docker.service.d sudo tee /etc/systemd/system/docker.service.d/override.conf < /dev/null # Find the log file for the exited container (use the container ID) container_id=$(docker ps -lq) ls -l /var/lib/docker/containers/$container_id/*-json.log # Expected permissions: -rw-r----- 1 root root ... ``` If permissions are still `644`, the umask override did not apply. Check `systemctl show docker -p UMask` and restart again. ## Hardening the json-file Driver The `json-file` driver is the most commonly used and often overlooked for security. Here are specific hardening steps: **Enable log rotation** Unbounded log files can fill the disk and cause denial of service. Set `max-size` and `max-file` for every container, either globally in `daemon.json` or per container in `docker run`: ```bash docker run -d \ --log-driver json-file \ --log-opt max-size=10m \ --log-opt max-file=3 \ --name web \ nginx:alpine ``` In Compose: ```yaml services: web: image: nginx:alpine logging: driver: json-file options: max-size: "10m" max-file: "3" ``` Verify rotation by filling the log and checking that old files are deleted. You can force rotation by sending signals if the application logs heavily, or simply wait until the size cap is reached. **Set restrictive permissions** As described above, use a umask of `0027` or `0077` for the Docker daemon to prevent other local users from reading logs. This is especially important on shared hosts. After applying, verify with: ```bash find /var/lib/docker/containers -name "*-json.log" -ls ``` **Avoid storing sensitive data in container logs** Even with restrictive permissions, logs may be shipped to central systems. Never log full credit card numbers, passwords, or tokens. Use log redaction in application code or a logging sidecar that can scrub data. For example, if using a logging proxy like Fluentd, you can apply a filter to mask sensitive fields: ```xml @type record_transformer enable_ruby true message ${record["message"].gsub(/\b\d{4}-\d{4}-\d{4}-\d{4}\b/, '[REDACTED]')} ``` This is outside Docker itself but is part of the overall logging security posture. ## Securing Remote Logging Drivers If you use a remote logging driver such as `syslog`, `gelf`, `fluentd`, `splunk`, or `awslogs`, consider the following: **Use TLS where possible** For `syslog`, use TCP with TLS: ```bash docker run -d \ --log-driver syslog \ --log-opt syslog-address=tcp+tls://logs.example.com:6514 \ --log-opt syslog-tls-ca-cert=/etc/docker/ca.pem \ --log-opt syslog-tls-cert=/etc/docker/cert.pem \ --log-opt syslog-tls-key=/etc/docker/key.pem \ --name app \ myapp:latest ``` For `gelf` with UDP, use a dedicated isolated network; for TCP, use TLS if the server supports it. For `fluentd`, use TLS: ```bash docker run -d \ --log-driver fluentd \ --log-opt fluentd-address=localhost:24224 \ --log-opt fluentd-async=true \ --log-opt fluentd-sub-second-precision=true \ --name app \ myapp:latest ``` Fluentd does not natively support TLS in the Docker driver, so you would need to run a TLS proxy in front of Fluentd or use a sidecar. **Protect credentials** For `splunk`, the HEC token is passed via `--log-opt splunk-token=...`. Never put the token in a shell script or Compose file that is committed to version control. Use Docker secrets or environment variables from a secure store: ```bash echo "splunk-token-value" | docker secret create splunk_token - docker service create \ --name app \ --secret splunk_token \ --log-driver splunk \ --log-opt splunk-token-source=secret \ --log-opt splunk-token-secret=splunk_token \ myapp:latest ``` For `awslogs`, store AWS credentials in IAM roles for EC2 or ECS instead of environment variables. If you must use static keys, use Docker Compose secrets: ```yaml services: app: image: myapp logging: driver: awslogs options: awslogs-region: us-east-1 awslogs-group: my-log-group awslogs-stream: my-stream secrets: - aws_creds secrets: aws_creds: file: ./aws_credentials ``` Inside the container, the credentials are available as a file, not in the environment. **Network isolation** Always run remote logging traffic over a private network or VPN. For local development, use a dedicated Docker network and restrict egress with firewall rules. ## Access Control and Permissions for Log Files On the host, log files under `/var/lib/docker/containers` should be accessible only to the root user and the Docker group. The default `docker` group may have read access, but consider whether all users in that group need it. You can tighten group membership: ```bash grep docker /etc/group # Example output: docker:x:999:alice,bob # Remove a user from the docker group sudo gpasswd -d bob docker ``` If a non-root process needs to read logs, use `sudo` with a specific command rather than granting broad group membership. For containers that write logs to a mounted directory (via bind mount), ensure the container runs as a non-root user and that the mounted directory has appropriate ownership and permissions on the host: ```bash docker run -d \ --user 1000:1000 \ -v /var/log/myapp:/var/log/app \ --name app \ myapp:latest ``` On the host, set `/var/log/myapp` to be owned by UID 1000 and mode `700` or `750` as appropriate. ## Verification and Diagnostics After making changes, verify thoroughly. Use the following checks: **1. Confirm the active logging driver for a container** ```bash docker inspect --format '{{.HostConfig.LogConfig.Type}}' # Expected output: json-file (or your configured driver) ``` **2. Confirm driver options** ```bash docker inspect --format '{{json .HostConfig.LogConfig.Config}}' # Example output: {"max-file":"3","max-size":"10m"} ``` **3. Test log generation and rotation** For `json-file`, you can force rotation by sending a `USR1` signal to the container if the application handles it, or simply generate enough logs. For example, with a busybox container: ```bash docker run -d --name logtest --log-opt max-size=1k --log-opt max-file=2 busybox sh -c 'while true; do echo "Log line $(date)"; sleep 0.1; done' ``` Wait until the log file exceeds 1KB, then check that only two log files remain: ```bash ls -l /var/lib/docker/containers/$(docker inspect logtest --format '{{.Id}}')/*-json.log # Expected: two files, e.g., container-json.log and container-json.log.1 ``` **4. Check log file permissions** ```bash stat -c '%a %U %G %n' /var/lib/docker/containers/$(docker inspect logtest --format '{{.Id}}')/*-json.log # Expected output: 640 root root ... ``` **5. Test remote logging driver connectivity** If using `syslog` or `gelf`, send a test log and verify it appears at the destination: ```bash docker run --rm --log-driver syslog --log-opt syslog-address=udp://syslog-server:514 alpine echo "Test log $(date)" ``` Then check the syslog server's logs for that message. **6. Check for dropped logs** If using `non-blocking` mode for `json-file`, monitor the daemon for buffer overruns: ```bash journalctl -u docker | grep -i "dropping logs" ``` If you see messages about dropping logs, increase `max-buffer-size` or switch back to blocking mode to ensure completeness. ## Failure Modes and Recovery Things can go wrong when changing logging configuration. Here are common failure modes and how to recover: **Malformed daemon.json prevents Docker from starting** Symptoms: `systemctl restart docker` fails, `docker ps` returns "Cannot connect to the Docker daemon". Recovery: ```bash sudo cp /etc/docker/daemon.json.bak.YYYYMMDD /etc/docker/daemon.json sudo systemctl restart docker ``` If you do not have a backup, remove the file (after saving a copy elsewhere) and restart Docker. Then reconstruct the configuration step by step, validating JSON after each change. **Container fails to start with specified logging driver** Symptoms: `docker run` outputs an error like "failed to initialize logging driver: ..." or the container exits immediately. Diagnosis: ```bash docker logs # Example error: "Failed to connect to syslog server" ``` Recovery: - Check the logging driver options for typos. - Verify that the remote logging endpoint is reachable and credentials are correct. - Temporarily switch the container to `json-file` to confirm the application itself works. **Logs are not being shipped to the remote server** Symptoms: No logs appear at the destination. Diagnosis: - Check Docker daemon logs: `journalctl -u docker -f` - Test network connectivity from the Docker host: `nc -vz ` - Check firewall rules. - For TLS, validate certificates and hostname. **Log rotation not working** Symptoms: Log file grows beyond `max-size` and is not rotated. Diagnosis: - Confirm that `max-size` is set correctly in the container's log config. - Note that `json-file` rotation only happens when Docker writes a new log entry; if the container is idle, the file may exceed max-size until new logs are written. - Check that the `max-file` option is not set to 0. **Permission changes cause application unable to write logs** If you changed umask and now a container running as non-root cannot write to its log file (this is unlikely because Docker creates the log file as root, but possible with bind-mounted logs), adjust the bind mount permissions or run the container with appropriate user and group IDs. ## Operations Checklist Use this checklist before and after making logging driver changes: | Step | Action | Command / Check | Expected Result | |------|--------|-----------------|-----------------| | 1 | Backup current Docker daemon config | `cp /etc/docker/daemon.json /etc/docker/daemon.json.bak` | Backup file exists | | 2 | Record current logging driver | `docker info --format '{{.LoggingDriver}}'` | Output matches inventory | | 3 | Record container log configs | `docker inspect --format '{{json .HostConfig.LogConfig}}'` | Config documented | | 4 | Apply change to one container first | `docker run --log-driver ... --log-opt ...` | Container starts | | 5 | Verify new driver active | `docker inspect --format '{{.HostConfig.LogConfig.Type}}'` | Expected driver name | | 6 | Verify options | `docker inspect --format '{{json .HostConfig.LogConfig.Config}}'` | Expected options | | 7 | Generate test logs | `docker exec sh -c 'echo test'` | Log entry appears | | 8 | Check log file permissions | `stat -c '%a' /var/lib/docker/containers//*-json.log` | 640 or stricter | | 9 | Test rotation | Write logs until max-size, check file count | Only max-file count remains | | 10 | Test remote delivery (if applicable) | Send test log, check receiver | Log appears at destination | | 11 | Check Docker daemon logs for errors | `journalctl -u docker -n 100 --no-pager` | No unexpected errors | | 12 | Document the change and rollback plan | Update runbook | Runbook updated | For each step, if the actual result differs from expected, stop and investigate before proceeding. Rollback to the backup configuration if necessary. ## Conclusion Hardening Docker logging drivers is an essential part of container security. By carefully selecting the appropriate driver, enforcing log rotation, restricting file permissions, protecting credentials, and using encrypted transport for remote logging, you can significantly reduce the risk of data exposure and denial-of-service attacks. This guide has provided a systematic approach: inventory the environment, understand the security implications of each driver, make controlled configuration changes, verify results with concrete commands and expected outputs, and prepare for failure with documented recovery steps. As a next step, choose one low-risk hardening measure from this article, such as enabling log rotation with restrictive permissions. Implement it in a staging environment, run the verification checks, and document the outcome. Once confident, apply the same change to production during a maintenance window, following the operations checklist. Remember that security is an ongoing process. Regularly review logging configurations, rotate credentials, and stay informed about new Docker features and best practices. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces a decision.