## Intro Debugging Docker containers in production presents unique challenges: containers are ephemeral, often immutable, and tightly integrated with orchestration and networking layers. A methodical approach prevents costly downtime and accelerates root cause analysis. This article provides a practical production checklist for debugging Docker containers, covering environment inventory, safe configuration, diagnostics, failure recovery, and a repeatable operations checklist. By following these steps, developers and DevOps teams can reduce mean time to recovery (MTTR) and maintain system stability. A clear operations checklist separates investigation from guesswork, reducing rework and ensuring consistent debugging outcomes. Start with a narrow, measurable pilot issue that is easy to inspect locally before scaling your debugging efforts. ## Version and Environment Inventory Before diving into debugging, establish a precise snapshot of the environment. This inventory helps reproduce issues and provides a baseline for changes. ### Docker Version and Host OS Run `docker version` and `docker info` on the affected host. Note the client and server versions, storage driver, and kernel version. For example: ```bash $ docker version --format '{{.Server.Version}}' 20.10.12 ``` Expected output: the Docker Engine version. If the version is outdated, upgrade may resolve known bugs. Record the host OS with `cat /etc/os-release` or `uname -a`. ### Container Topology and Orchestration Identify whether containers are managed by Docker Compose, Swarm, or Kubernetes. For standalone containers, use `docker ps -a` to list all containers and their status. For orchestrated environments, retrieve cluster state with the appropriate tools (e.g., `kubectl get pods` for Kubernetes). Capture the container's image ID and tags: ```bash $ docker inspect -f '{{.Config.Image}} {{.Image}}' nginx:1.21 sha256:2bcabc23b45489fb0885d69a06ba1d648aeda973fae7bb981bafbb884165e514 ``` This shows the image name and digest, which is crucial for rolling back to a known good image. ### Network and Resource Prerequisites Document network modes, port mappings, and resource limits. Use `docker inspect` to extract detailed configuration: ```bash $ docker inspect -f '{{json .NetworkSettings.Networks}}' | python -m json.tool ``` Check CPU and memory limits with `docker stats --no-stream ` to see if the container is hitting its cap. Also, verify logs are being captured by the logging driver (e.g., `json-file`, `journald`). ## Safe Configuration Path Debugging often requires configuration changes. To avoid causing further issues, implement changes in a scoped, reversible manner. ### Use Environment-Specific Overrides Never modify a running container's configuration directly. Instead, use environment variables or configuration files that can be version-controlled and rolled back. For example, to change a container's log level, restart it with an updated environment variable: ```bash $ docker run -d --name myapp -e LOG_LEVEL=debug myapp:latest ``` This command starts a new container with the debug log level. The old container can be stopped and kept for comparison. ### Enable Debug Logging Safely If the application supports debug logging, enable it temporarily via environment or config file. For Docker daemon itself, enable debug mode by editing `/etc/docker/daemon.json` and restarting the daemon: ```json { "debug": true } ``` After restart, verify debug logs appear in the daemon log (e.g., `journalctl -u docker.service -f`). Be aware that debug logging increases verbosity and may affect performance; disable after troubleshooting. ### Avoid Privileged Mode and Host Modifications When needing to inspect network traffic or system calls, use specialized tools that do not require running the container with `--privileged`. For example, `nsenter` allows running commands in the container's namespaces without changing its privileges. To get a shell inside a container for debugging, prefer `docker exec` over `docker attach` to avoid interfering with the container's main process. ## Verification and Diagnostics This section provides concrete checks and expected outcomes to aid debugging. ### Check Container Health If a container has a healthcheck defined, query its status: ```bash $ docker inspect --format='{{.State.Health.Status}}' healthy ``` Expected output: `healthy` or `unhealthy`. If unhealthy, inspect the healthcheck log: ```bash $ docker inspect --format='{{json .State.Health}}' | python -m json.tool ``` This shows the last healthcheck output and exit code. ### Examine Logs Use `docker logs` to view container output. For a container with a lot of logs, use `--since` and `--until` to filter by time, and `--tail` to limit lines. Example: ```bash $ docker logs --since=2023-10-01T10:00:00 --until=2023-10-01T11:00:00 --tail=100 ``` Expected output: recent log lines. Look for error stack traces, timeouts, or connection failures. ### Diagnose Network Connectivity If the container cannot reach external services, test connectivity from inside the container: ```bash $ docker exec -it bash root@container:/# curl -v https://example.com ``` Expected output: HTTP response headers. If `curl` fails, check DNS resolution (`nslookup example.com`) and routing (`ip route`). Also inspect `iptables` rules on the host (requires root on host). ### Inspect Process and Resource Usage Use `docker top ` to see processes running inside the container. Compare with host processes using `ps aux | grep `. To check for memory leaks, monitor RSS over time with `docker stats`. A steadily increasing memory usage may indicate a leak. ### Analyze File System Changes To see what files have changed since the container started, compare the container's file system with the image using `docker diff `. This shows files added (A), changed (C), or deleted (D). Example: ```bash $ docker diff C /var/log/app.log A /tmp/newfile.txt D /etc/oldconfig.conf ``` This is useful for identifying unintended modifications or confirming that config changes were applied. ## Failure Modes and Recovery Even with careful debugging, failures can occur. Prepare for common failure modes and know how to recover. ### Container Crash Loop If a container repeatedly starts and exits, inspect its exit code and logs: ```bash $ docker inspect -f '{{.State.ExitCode}} {{.State.Error}}' ``` Common exit codes: 1 (application error), 137 (SIGKILL, often OOM), 143 (SIGTERM, graceful termination). To temporarily prevent restart, stop the container: `docker stop `. Then run the container in foreground with `--rm` to see output directly: ```bash $ docker run --rm -it --name debug bash ``` This gives an interactive shell to test startup manually. ### Out of Memory (OOM) If the host runs out of memory, the kernel may kill a container process. Check `dmesg | grep -i oom` on the host. To prevent OOM, increase container memory limit: `docker update --memory 512m `. Also analyze application memory usage patterns. ### Network Communication Failure If the container cannot connect to other containers or external services, verify network configuration. For user-defined bridge networks, ensure containers are attached to the correct network: `docker network inspect `. Check firewall rules: `iptables -L -n -v | grep DOCKER`. Rollback by reverting recent network changes or restarting the Docker daemon: `systemctl restart docker`. ### Configuration Rollback Always keep previous configurations in version control. If a configuration change causes issues, revert to the last known good version and restart the container. For image-related issues, rollback to a previous image tag: ```bash $ docker run -d --name myapp_rollback myapp:1.0.2 ``` Then update the service or load balancer to point to the new container. ### Data Loss or Corruption If a volume is corrupted, restore from backups. Verify backup integrity regularly. For critical data, use a database migration or restore procedure. Test recovery drills in a staging environment. ## Common Pitfalls and How to Avoid Them Even experienced teams make mistakes when debugging containers in production. Here are the most common pitfalls, why they happen, and how to avoid or recover from them. ### Pitfall 1: Modifying a Running Container Directly **Why it happens:** It feels quick to `docker exec` into a container and change a config file or install a package to test a fix. **Risk:** Changes are lost when the container restarts, and you now have a configuration that is not reproducible. **How to avoid:** Always make changes via image rebuild, environment variables, or mounted config files. If you must make an emergency change, document it and immediately create a proper fix in your build pipeline. ### Pitfall 2: Ignoring Resource Limits **Why it happens:** Developers often focus on application errors and overlook container resource limits. **Risk:** A container hitting its memory limit can be killed by the OOM killer, causing unexplained restarts. CPU throttling can make the application appear hung. **How to avoid:** Set appropriate resource limits in your deployment manifest. Monitor `docker stats` regularly. When debugging, always check if the container is near its limit. ### Pitfall 3: Relying on `latest` Tag **Why it happens:** Using `latest` is convenient during development. **Risk:** In production, `latest` can point to a different image than the one tested, leading to unexpected behavior and making rollbacks difficult. **How to avoid:** Always use explicit version tags or image digests in production. For example, use `myapp:1.2.3` or `myapp@sha256:...`. ### Pitfall 4: Overlooking Logging Configuration **Why it happens:** Teams assume logs are being captured correctly. **Risk:** If the logging driver is misconfigured or disk space runs out, you lose visibility exactly when you need it most. **How to avoid:** Verify the logging driver with `docker info`. Set up log rotation to prevent disk exhaustion. Use a centralized log aggregation system. ### Pitfall 5: Running Containers as Root **Why it happens:** Many base images default to root for simplicity. **Risk:** A compromised container gives an attacker root access to the host. **How to avoid:** Specify a non-root user in your Dockerfile (`USER` directive) and ensure your application does not need root. Use Docker's `--user` flag if necessary. ## Operations Checklist Use the following checklist for systematic debugging and ongoing operations. This checklist helps teams turn debugging challenges into repeatable, measurable tasks. Assign a single accountable owner for each step and set a review cadence (e.g., weekly for monitoring, monthly for checklist review). | Step | Action | Owner (Example) | Verification | Frequency | |------|--------|-----------------|--------------|-----------| | 1 | Record environment details (Docker version, host OS, orchestration) | Priya Shah, DevOps Engineer | `docker version` and `docker info` output captured | Every incident | | 2 | Identify the failing container and its purpose | On-call engineer | `docker ps -a` shows status | Every incident | | 3 | Review container logs for errors | On-call engineer | `docker logs --tail=100` shows no unexpected errors | Every incident | | 4 | Check container health status | On-call engineer | `docker inspect --format='{{.State.Health.Status}}'` returns healthy | Every incident | | 5 | Test network connectivity from inside the container | Network specialist (if needed) | `docker exec curl -v ` succeeds | When network issues suspected | | 6 | Inspect resource usage (CPU, memory, disk I/O) | Priya Shah, DevOps Engineer | `docker stats --no-stream` shows utilization below 80% | Weekly | | 7 | Compare container filesystem to image | On-call engineer | `docker diff` shows only expected changes | When changes suspected | | 8 | Reproduce the issue with minimal configuration | Developer lead (e.g., Alex Chen) | Container runs with `--rm` and minimal env without error | During root cause analysis | | 9 | Apply fix in a scoped manner (new container, env override) | Developer lead | `docker run` with updated config succeeds | During fix | | 10 | Monitor after fix and document root cause | On-call engineer | `docker stats` and logs show no errors for 24 hours | Post-fix and review | ### Regular Operations Review Perform regular reviews of container health, log rotation, and resource alerts. Set up monitoring with Prometheus and Grafana, or use the Docker built-in `docker stats` with a collection script. Define alert thresholds for CPU, memory, and restarts. The operations checklist itself should be reviewed monthly by the DevOps team lead to incorporate new learnings. ## Conclusion Docker container debugging in production demands a systematic approach. This checklist provides a practical framework: start with an environment inventory, make scoped configuration changes, use concrete diagnostic commands, anticipate failure modes, follow a repeatable operations checklist, and avoid common pitfalls. By adopting these practices, teams can reduce downtime and improve reliability. Remember to keep a narrow, measurable focus for each debugging session, always verify fixes with observable checks, and assign clear ownership for each step. Regularly review and update your debugging procedures to cover new scenarios and container technologies. --- Note: All commands and outputs are illustrative. Adapt them to your specific environment and container runtime.