## Intro Running Docker in production requires more than building and shipping images. Security is an ongoing operational responsibility that spans configuration, monitoring, maintenance, backups, and upgrades. Without a repeatable process, teams risk exposure to container escapes, privilege escalation, secret leaks, and supply-chain attacks. This article provides a practical operations checklist for Docker security in production. Each section includes concrete commands, configuration snippets, expected outputs, and verification steps. The goal is to give developers, DevOps consultants, and technical startup teams a reference they can adapt to their own environments. We cover: - Inventorying Docker versions and environment prerequisites - Applying a safe configuration path with scoped hardening - Verifying the security posture with observable checks - Handling failure modes and recovery procedures - A repeatable operations checklist for ongoing security By the end, you will have a clear set of actions to harden Docker hosts, enforce runtime policies, detect anomalies, and recover from incidents. ## Version and Environment Inventory Before making any security changes, document the current Docker environment. This includes Docker Engine and CLI versions, container runtime, host operating system, storage and network drivers, and current security-related settings. ### Check Docker Versions Run the following on each Docker host: ```bash docker version ``` Example output (truncated): ``` Client: Docker Engine - Community Version: 24.0.5 API version: 1.43 Go version: go1.20.4 Git commit: ced0996 Built: Fri Jul 21 19:01:22 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.5 API version: 1.43 (minimum version 1.12) Go version: go1.20.4 Git commit: a61e2b4 Built: Fri Jul 21 19:01:22 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.21 GitCommit: 3dce8eb055cbb6878b1e53c1b34f8f1b3e3f0c1a runc: Version: 1.1.7 GitCommit: v1.1.7-0-g860f061 docker-init: Version: 0.19.0 GitCommit: de40ad0 ``` Record the Engine version and the components (containerd, runc). Security patches often target specific versions, so keeping this inventory up to date is critical. ### Inspect Current Security Configuration Use `docker info` to see general security settings: ```bash docker info --format '{{json .SecurityOptions}}' ``` Example output: ```json ["name=seccomp,profile=builtin","name=cgroupns"] ``` If you see `name=userns` in the output, user namespace remapping is enabled. If not, it may be a candidate for hardening (discussed later). Also check the default seccomp profile and AppArmor/SELinux status: ```bash docker info | grep -E 'Security Options|Kernel Version|Operating System' ``` ### Identify Privileged Containers List currently running containers and check for privileged mode: ```bash docker ps --quiet | xargs docker inspect --format '{{.Name}}: Privileged={{.HostConfig.Privileged}}' ``` Example output: ``` /mysql: Privileged=false /web: Privileged=true /cache: Privileged=false ``` The `/web` container is running privileged, which is risky. You will address this in the safe configuration path. ### Prerequisites Ensure you have: - SSH access to Docker hosts with sudo (or equivalent) - A test or staging environment to validate changes before production - Backup of Docker daemon configuration (`/etc/docker/daemon.json`) - Backup of any systemd unit files or init scripts for Docker - Ability to restart the Docker daemon (requires scheduling maintenance window) - Log aggregation and monitoring tools (e.g., ELK, Prometheus, Datadog) Document the inventory in a spreadsheet or configuration management database. This baseline will be used for verification later. ## Safe Configuration Path Apply security hardening in a controlled, incremental manner. Start with the lowest-risk, highest-impact changes: daemon configuration, container runtime settings, and network policies. Always test on a non-production host first. ### 1. Harden the Docker Daemon The Docker daemon runs with root privileges and controls all containers. A misconfigured daemon can expose the host. Edit `/etc/docker/daemon.json` (create if missing) and add or modify the following: ```json { "icc": false, "log-level": "info", "userland-proxy": false, "no-new-privileges": true, "live-restore": true, "iptables": true, "ip6tables": true, "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" }, "default-ulimits": { "nofile": { "Name": "nofile", "Hard": 64000, "Soft": 64000 } } } ``` Key settings: - `"icc": false`: Disables inter-container communication by default. Containers must be explicitly linked or on the same user-defined network to communicate. - `"no-new-privileges": true`: Prevents containers from gaining additional privileges via setuid/setgid binaries or other mechanisms. - `"live-restore": true`: Keeps containers running when the Docker daemon is stopped, reducing downtime during upgrades. After editing, restart the Docker daemon: ```bash sudo systemctl restart docker ``` Verify the settings: ```bash docker info --format '{{json .SecurityOptions}}' ``` Expected output should now include `no-new-privileges`. Actually, `no-new-privileges` is a per-container flag; the daemon setting is not shown there. Instead, check `docker info` for the daemon-level settings: ```bash sudo docker info | grep -i 'Default Runtime\|Cgroup Driver\|Security Options' ``` But to verify `icc: false`, run: ```bash docker network inspect bridge | grep -i 'com.docker.network.bridge.enable_icc' ``` Expected output: ``` "com.docker.network.bridge.enable_icc": "false" ``` ### 2. Enable User Namespace Remapping User namespaces map the container's root user to a non-root user on the host, limiting the impact of container escapes. This is a powerful security control but can break some workloads (e.g., those requiring specific file permissions). Test thoroughly. To enable, add to `/etc/docker/daemon.json`: ```json { "userns-remap": "default" } ``` Then restart Docker: ```bash sudo systemctl restart docker ``` Check that the remapped user exists: ```bash id dockremap ``` Example output: ``` uid=140000(dockremap) gid=140000(dockremap) groups=140000(dockremap) ``` Note: If you have existing containers or volumes, they may have permission issues after enabling. Plan to recreate containers and adjust volume ownership. Refer to Docker's documentation for migration steps. ### 3. Restrict Container Capabilities By default, Docker containers get a subset of Linux capabilities. You can further reduce privileges by dropping all capabilities and adding back only what is needed. Use the `--cap-drop` and `--cap-add` flags. Example: Run an Nginx container with minimal capabilities: ```bash docker run -d --name web \ --cap-drop ALL \ --cap-add NET_BIND_SERVICE \ --security-opt no-new-privileges \ nginx:alpine ``` Here we drop all capabilities, then add only `NET_BIND_SERVICE` (to bind to port 80). We also set `no-new-privileges` to prevent privilege escalation. Verify the capabilities of a running container: ```bash docker inspect --format '{{.HostConfig.CapAdd}} {{.HostConfig.CapDrop}}' web ``` Expected output: ``` [NET_BIND_SERVICE] [ALL] ``` ### 4. Apply Resource Limits Unlimited resource usage can lead to denial of service. Set memory and CPU limits on containers: ```bash docker run -d --name api \ --memory 512m \ --cpus 1.5 \ --pids-limit 100 \ myapp:1.0 ``` These flags ensure the container uses at most 512 MB RAM, 1.5 CPU cores, and 100 processes/threads. `--pids-limit` prevents fork bombs. Check limits: ```bash docker inspect --format '{{.HostConfig.Memory}} {{.HostConfig.NanoCpus}} {{.HostConfig.PidsLimit}}' api ``` Expected output: ``` 536870912 1500000000 100 ``` ### 5. Use Read-Only Root Filesystem Many applications can run with a read-only root filesystem, preventing attackers from writing malicious files. Use `--read-only` and mount writable volumes for specific paths (e.g., `/tmp`, `/var/run`). ```bash docker run -d --name web \ --read-only \ --tmpfs /tmp \ --tmpfs /var/run \ nginx:alpine ``` Verify: ```bash docker inspect --format '{{.HostConfig.ReadonlyRootfs}}' web ``` Expected output: ``` true ``` ### 6. Secure the Docker Socket The Docker socket (`/var/run/docker.sock`) grants full control over the daemon. Never expose it to containers unless absolutely necessary. If a container needs to manage Docker (e.g., Portainer), use a proxy like `docker-socket-proxy` with restricted permissions. Check if any container mounts the socket: ```bash docker ps --quiet | xargs docker inspect --format '{{.Name}}: {{range .Mounts}}{{if eq .Source "/var/run/docker.sock"}}YES{{end}}{{end}}' ``` If any container shows `YES`, evaluate whether it truly needs the socket and restrict with a proxy. ### 7. Network Segmentation Use user-defined bridge networks to isolate containers. Avoid using the default bridge network in production. Create a network: ```bash docker network create --driver bridge frontend ``` Run containers attached to this network: ```bash docker run -d --name web --network frontend nginx:alpine ``` Container communication across different networks is blocked unless explicitly allowed. ### 8. Configure Logging and Monitoring Ensure logs are captured and rotated. The daemon configuration above sets JSON file logging with rotation. For production, consider shipping logs to a central system using a logging driver like `fluentd` or `splunk`. Example `daemon.json` addition: ```json { "log-driver": "fluentd", "log-opts": { "fluentd-address": "localhost:24224", "tag": "docker.{{.Name}}" } } ``` Then restart Docker. Test by running a container and checking the log destination. ## Verification and Diagnostics After applying changes, verify that the security controls are effective. Use automated tools and manual checks. ### 1. Run Docker Bench for Security Docker Bench for Security is a script that checks for common best practices. Run it on the host: ```bash git clone https://github.com/docker/docker-bench-security.git cd docker-bench-security sudo ./docker-bench-security.sh ``` The output is a long report with `[WARN]`, `[INFO]`, and `[PASS]` items. Example excerpt: ``` [INFO] 1 - Host Configuration [WARN] 1.1 - Ensure a separate partition for containers has been created [PASS] 1.2 - Ensure the container host has been Hardened [PASS] 1.3 - Ensure Docker is up to date [INFO] 2 - Docker daemon configuration [WARN] 2.1 - Ensure network traffic is restricted between containers on the default bridge [PASS] 2.2 - Ensure the logging level is set to 'info' ... ``` Address warnings as appropriate for your environment. Note that some warnings may be acceptable due to operational constraints; document exceptions. ### 2. Scan Images for Vulnerabilities Use image scanning tools like Trivy, Clair, or Docker Scout. Example with Trivy: ```bash trivy image nginx:alpine ``` Example output (truncated): ``` nginx:alpine (alpine 3.18.3) Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0) ``` If vulnerabilities are found, update the base image or apply patches. Integrate scanning into your CI/CD pipeline to catch issues early. ### 3. Verify Seccomp and AppArmor Profiles Check that containers are using the default seccomp profile (or a custom one). ```bash docker inspect --format '{{.HostConfig.SecurityOpt}}' web ``` Expected output includes `seccomp=...` and possibly `apparmor=...` if AppArmor is enabled. If you have a custom seccomp profile, test it by attempting a forbidden syscall inside the container. For example, with a profile that blocks `mkdir`, running `mkdir /tmp/test` should fail with `Operation not permitted`. ### 4. Test User Namespace Remapping After enabling `userns-remap`, run a container and check the ownership of files created in a volume from the host perspective. ```bash docker run -d --name test -v /tmp/testvol:/data alpine sleep 300 docker exec test touch /data/testfile ls -l /tmp/testvol ``` Expected output shows `testfile` owned by `dockremap` (e.g., `140000`) rather than root. ### 5. Confirm No Privileged Containers Re-run the privileged container check: ```bash docker ps --quiet | xargs docker inspect --format '{{.Name}}: Privileged={{.HostConfig.Privileged}}' ``` Output should show `Privileged=false` for all containers. If any are `true`, remediate immediately. ### 6. Check Network Isolation Test inter-container communication. With `icc: false`, containers on the default bridge cannot communicate. Create two test containers on the default bridge: ```bash docker run -d --name test1 alpine sleep 300 docker run -d --name test2 alpine sleep 300 ``` Get IP of `test1`: ```bash docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' test1 ``` Example output: `172.17.0.2` From `test2`, try to ping `test1`: ```bash docker exec test2 ping -c 2 172.17.0.2 ``` Expected result: ping fails (100% packet loss) because `icc` is disabled. If they need to communicate, put them on a user-defined network. ### 7. Monitor Runtime Anomalies Use tools like Falco or Sysdig to detect suspicious behavior. Install Falco on the host according to its documentation. Then test by running a container that tries to write to `/etc` inside the container: ```bash docker run --rm -it alpine sh -c "echo bad > /etc/test" ``` Falco should generate an alert. Example Falco log line: ``` 12:34:56.789012345: Warning Container with writable /etc directory detected (user=root command=sh container_id=abc123) ``` Configure alerts to go to your incident response channel. ## Failure Modes and Recovery Security changes can break applications. Plan for failures and know how to roll back quickly. ### Common Failure Modes - **Application requires privileges**: Some applications legitimately need certain capabilities or even privileged mode (e.g., system daemons, network tools). If you drop capabilities and the app fails, add back only the necessary capabilities. - **User namespace remapping breaks file permissions**: Existing volumes may become inaccessible because of UID/GID mapping. Solutions: recreate volumes with correct ownership, or disable `userns-remap`. - **Read-only root filesystem causes write errors**: Apps may try to write to arbitrary locations. Use `--tmpfs` or volume mounts for required writable paths. - **Network isolation blocks legitimate traffic**: Internal services may need communication on the default bridge. Use user-defined networks with explicit connectivity. - **Resource limits cause OOM kills or throttling**: Too-low limits can crash containers. Adjust based on observed usage. ### Rollback Procedures For any daemon configuration change, keep a backup of `/etc/docker/daemon.json`. To rollback: ```bash sudo cp /etc/docker/daemon.json.backup /etc/docker/daemon.json sudo systemctl restart docker ``` For container-level changes, simply recreate the container with previous flags. Document the exact steps to revert each change in your runbook. ### Recovery Checks After rollback, verify that the environment is functioning: - Run `docker ps` to ensure containers are up. - Check application logs for errors. - Test connectivity. - Re-run the relevant verification commands from the previous section. ## Operations Checklist Use this checklist for regular security reviews and as part of your operational routine. ### Daily Operations - [ ] Monitor Docker host resource usage (CPU, memory, disk) for anomalies. Owner: On-call DevOps engineer. Check via Prometheus alerts. - [ ] Check Docker daemon logs (`journalctl -u docker`) for errors or security events. - [ ] Review container logs for suspicious activity. - [ ] Verify that monitoring/alerting (e.g., Falco) is functioning. ### Weekly Operations - [ ] Run Docker Bench for Security and review any new warnings. Owner: Security lead. - [ ] Scan images for new vulnerabilities (if not integrated into CI/CD). - [ ] Review user access to Docker hosts and socket permissions. Owner: Infrastructure lead. - [ ] Check for orphaned containers, images, volumes, and networks; clean up: ```bash docker system prune -f --volumes ``` Use caution with `--volumes` as it deletes unused volumes. ### Monthly Operations - [ ] Update Docker Engine and CLI to the latest stable version. Owner: Infrastructure lead, scheduled for first Tuesday of each month. - [ ] Review and update seccomp/AppArmor profiles if needed. - [ ] Rotate secrets and credentials used by containers (e.g., registry credentials, API keys). - [ ] Test backup and restore procedures. - [ ] Review resource limits and adjust based on performance trends. - [ ] Evaluate new security features in Docker releases and plan adoption. ### Incident Response In case of a security incident: - Isolate the affected host or container. - Capture forensic data: container logs, image metadata, host logs. - Identify the vulnerability (e.g., exposed port, vulnerable dependency). - Patch and redeploy. - Document the incident and update the checklist to prevent recurrence. ## Common Pitfalls and How to Avoid Them 1. **Ignoring the Docker socket exposure**: Many developers mount `/var/run/docker.sock` into containers for convenience, giving those containers root-equivalent access to the host. Use a proxy like `docker-socket-proxy` with read-only or restricted API access, and only for containers that truly need it. 2. **Running containers as root**: By default, Docker containers run as root. If an attacker escapes the container, they get root on the host. Use the `USER` directive in Dockerfiles to run as a non-root user, and avoid `--privileged` unless absolutely necessary. 3. **Not setting resource limits**: Without memory or CPU limits, a single container can exhaust host resources, causing denial of service. Always set `--memory` and `--cpus` in production, and use orchestrators like Kubernetes to enforce limits. 4. **Using outdated base images**: Base images often contain known vulnerabilities. Regularly update images and scan them with tools like Trivy in CI/CD. Pin image versions to avoid unexpected changes. 5. **Disabling security features for simplicity**: Turning off seccomp, AppArmor, or SELinux to get an app working leaves the host exposed. Instead, create custom profiles that grant only the required syscalls or permissions. 6. **Storing secrets in images**: Hardcoding passwords or API keys in Dockerfiles or environment variables is a common mistake. Use Docker secrets, Kubernetes secrets, or external secret managers like Vault, and never bake secrets into image layers. 7. **Neglecting network segmentation**: Relying on the default bridge network allows all containers to communicate. Use user-defined networks with explicit connectivity and consider network policies (e.g., Calico, Cilium) in Kubernetes. ## Conclusion Docker security is not a one-time setup; it requires continuous operational effort. This checklist provides a foundation for hardening your Docker environment, verifying security controls, and recovering from failures. Start with the inventory, apply safe configuration changes incrementally, and establish regular review cycles. Adapt the checklist to fit your specific workloads and risk profile. By following these practices, you can significantly reduce the attack surface of your containerized applications and maintain a robust security posture in production.