## Intro Debugging a containerized application often feels like solving a mystery. Processes are isolated from the host, file systems are ephemeral, and networking adds layers of indirection. When your web app stops responding or crashes, the path to resolution is rarely obvious. To debug effectively, you need to understand the architecture behind Docker container debugging: the components involved, how data flows between them, and where failures commonly occur. This guide provides that architectural view, along with practical examples, commands, and configuration snippets you can use immediately. Docker container debugging is not a single tool but a combination of Docker's built-in features and external utilities. The core components include the Docker CLI, the Docker daemon, container runtime (runc or containerd), logging drivers, exec and inspect commands, and monitoring tools. By understanding how these pieces interact, you can diagnose issues faster and avoid common pitfalls. This guide covers version and environment inventory, safe configuration, verification and diagnostics, failure modes, and an operations checklist. Throughout this article, we will use a simple web application container as a running example. It is a Node.js app that occasionally crashes due to unhandled exceptions, and we will debug it step by step. All commands are tested on Linux (Ubuntu 22.04) with Docker Engine 24.0, but they should work with minor modifications on other systems. ## Version and Environment Inventory Before debugging any container, establish a clear baseline. Know the Docker version, the container's image and runtime, and the host's operating system. This information is often critical when researching known issues or determining compatibility. ### Prerequisites - Docker Engine 20.10 or later (we use 24.0). - Basic familiarity with the command line. - A container to debug (we use webapp:1.0). - Access to container logs and the ability to execute commands inside containers. ### Checking Docker Version and Info Run the following commands on the host: ```bash docker version --format '{{.Server.Version}}' ``` Expected output (yours may differ): ``` 24.0.5 ``` ```bash docker info --format '{{.OperatingSystem}} {{.Architecture}} {{.CgroupDriver}}' ``` Expected output: ``` Ubuntu 22.04.3 LTS x86_64 systemd ``` ### Inspecting the Container Identify the container you want to debug: ```bash docker ps -a --filter "name=webapp" --format "table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.Names}}" ``` Example output: ``` CONTAINER ID IMAGE STATUS NAMES 3f4a5b6c7d8e webapp:1.0 Up 10 minutes (unhealthy) webapp ``` Note the "(unhealthy)" status; this indicates the container's health check is failing. We will investigate later. ### Topology In a typical Docker debugging setup, you have: - **Host**: Runs the Docker daemon and provides resources. - **Docker daemon (dockerd)**: Manages containers, images, networks, and volumes. - **Container runtime (containerd / runc)**: Actually creates and runs containers. - **Container**: The application process and its dependencies. - **Logging driver**: Captures container stdout/stderr and forwards to a destination (json-file by default). - **Debugging tools**: docker exec, docker logs, docker inspect, system tools like strace, tcpdump, etc. Data flow: Application writes to stdout/stderr -> container runtime -> logging driver -> log file or remote endpoint. Control flow: User issues docker exec -> Docker CLI -> daemon -> runtime -> execute command in container namespace. ## Safe Configuration Path Before making any changes, implement a safe debugging configuration. This means enabling appropriate logging, setting up health checks, and ensuring you can execute commands in the container without disrupting production if possible. We will do this in a scoped, reversible way. ### Configuring Logging Drivers Docker's default logging driver is json-file, which stores logs on the host. For debugging, it is often helpful to configure a different driver or adjust options. We will retain json-file but set a max size to avoid disk fill. Create a container with logging options: ```bash docker run -d --name webapp \ --log-driver json-file \ --log-opt max-size=10m \ --log-opt max-file=3 \ -p 8080:8080 \ webapp:1.0 ``` This limits the log files to 10 MB each and keeps 3 rotated files. If you need to change logging for an existing container, you must recreate it. To see the current logging config: ```bash docker inspect --format '{{.HostConfig.LogConfig.Type}} {{.HostConfig.LogConfig.Config}}' webapp ``` Output: ``` json-file map[max-file:3 max-size:10m] ``` ### Setting Up Health Checks A health check tells Docker whether the application is functioning. For our Node.js app, we can add an HTTP health endpoint. Modify the Dockerfile to include HEALTHCHECK: ```dockerfile FROM node:18-alpine WORKDIR /app COPY package*.json ./ RUN npm install COPY . . EXPOSE 8080 HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ CMD curl -f http://localhost:8080/health || exit 1 CMD ["node", "server.js"] ``` Rebuild the image: ```bash docker build -t webapp:1.1 . ``` Run the new container: ```bash docker run -d --name webapp-healthy \ --log-driver json-file --log-opt max-size=10m --log-opt max-file=3 \ -p 8080:8080 \ webapp:1.1 ``` Check health status after a minute: ```bash docker ps --format '{{.HealthStatus}}' --filter name=webapp-healthy ``` Expected: healthy. ### Enabling Exec Access For debugging, you need to run commands inside the container. Ensure the container is not running with --read-only or a restricted security profile that prevents exec. We will use docker exec for interactive debugging. ## Verification and Diagnostics Now that the container is running with proper logging and health checks, we can diagnose issues. Let us simulate a failure in our Node.js app: it crashes when a certain endpoint is hit due to an unhandled promise rejection. ### Observing Logs First, check the logs: ```bash docker logs webapp-healthy ``` You might see something like: ``` Listening on port 8080 ... (node:1) UnhandledPromiseRejectionWarning: Error: Something went wrong at /app/server.js:15:11 ... ``` This tells us where the error is. We can also follow logs live: ```bash docker logs -f --tail 50 webapp-healthy ``` ### Executing Commands in the Container We can inspect the running process and environment: ```bash docker exec webapp-healthy ps aux ``` Expected output (simplified): ``` USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND root 1 0.0 0.5 723456 45678 ? Ssl 12:34 0:00 node server.js root 12 0.0 0.0 15568 2024 ? Rs 12:35 0:00 ps aux ``` If the container is unhealthy, we might want to inspect the health check logs: ```bash docker inspect --format '{{json .State.Health}}' webapp-healthy ``` Output includes log entries from health check runs. ### Using docker inspect for Configuration docker inspect reveals a wealth of information: ```bash docker inspect webapp-healthy ``` Key fields: NetworkSettings.IPAddress, Mounts, Config.Env, HostConfig.Binds, etc. For debugging, check the network settings to ensure port mapping: ```bash docker inspect --format '{{range $p, $conf := .NetworkSettings.Ports}}{{$p}} -> {{(index $conf 0).HostPort}}{{"\n"}}{{end}}' webapp-healthy ``` Output: ``` 8080/tcp -> 8080 ``` ### Copying Files for Analysis Sometimes you need to extract files from a container, such as core dumps or logs written to disk. Use docker cp: ```bash docker cp webapp-healthy:/app/server.js ./server.js ``` ### Debugging with strace If the process is behaving strangely, strace can trace system calls. First, install strace inside the container (if not present): ```bash docker exec -it webapp-healthy sh -c "apk add --no-cache strace" ``` Then attach to the process: ```bash docker exec -it webapp-healthy strace -p 1 ``` This might be heavy; use it with caution on production containers. ## Failure Modes and Recovery Understanding common failure modes helps you prepare for and recover from issues. We will cover typical problems: container exiting immediately, unresponsive container, disk full due to logs, and network misconfiguration. ### Container Exits Immediately If a container starts and exits, check its exit code and logs. ```bash docker ps -a --filter "name=webapp" --format "{{.Status}}" ``` Example: ``` Exited (1) 10 seconds ago ``` Then inspect logs: ```bash docker logs webapp ``` Often it is a missing dependency or configuration error. Fix the Dockerfile or entrypoint and restart. ### Unresponsive Container If a container hangs and does not respond to commands, it may be out of memory or have a deadlocked process. Check resource usage: ```bash docker stats --no-stream webapp-healthy ``` Example output: ``` CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 3f4a5b6c7d8e webapp-healthy 0.00% 45.2MiB / 3.8GiB 1.16% 2.1kB / 0B 0B / 0B 2 ``` If memory is high, consider increasing limits or profiling the app. You can also use docker exec to run diagnostic commands, but if the container is truly hung, you may need to restart it. ### Disk Full Due to Logs If the host disk fills up, container logging may be the culprit. Check disk usage of Docker logs: ```bash sudo du -sh /var/lib/docker/containers/*/*-json.log ``` If a file is huge, rotate or remove it. Better, configure log rotation from the start. To clear logs without restarting the container, use: ```bash truncate -s 0 /var/lib/docker/containers//-json.log ``` Get container ID with docker ps -q. ### Network Misconfiguration If the app cannot connect to other services, inspect the network: ```bash docker network inspect bridge ``` Check if containers are on the same network. For custom networks, create one: ```bash docker network create mynet ``` Then run containers with --network mynet. Use docker exec to test connectivity: ```bash docker exec webapp-healthy curl http://database:5432 ``` ### Recovery Steps - For immediate recovery, restart the container: docker restart webapp-healthy. - If the issue persists, recreate the container with updated configuration. - For code bugs, fix the application and rebuild the image. ### Rollback Keep previous image versions to rollback: ```bash docker tag webapp:1.1 webapp:stable docker run -d --name webapp-stable ... webapp:stable ``` If the new version fails, switch back to the stable image. ## Common Pitfalls and How to Avoid Them Even experienced Docker users fall into predictable traps when debugging containers. Here are the most common pitfalls, why they happen, and how to avoid or recover from them. ### Pitfall 1: Neglecting Log Rotation **Why it happens**: Default logging configuration stores all container output in a single JSON file without size limits. Over time, this file can fill the host disk, causing system-wide failures. **How to avoid**: Always set log rotation options when running containers, either via the --log-opt flags or in the Docker daemon configuration file (/etc/docker/daemon.json). For example, add this to daemon.json: ```json { "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } } ``` Then restart the Docker daemon. **Recovery**: If disk is already full, identify large log files with `sudo du -sh /var/lib/docker/containers/*/*-json.log` and truncate them as shown in the previous section. Then implement rotation to prevent recurrence. ### Pitfall 2: Ignoring Health Check Status **Why it happens**: Many developers do not configure health checks for their containers. Without them, Docker cannot report whether the application is actually working, only whether the process is running. A container may be "Up" but unresponsive. **How to avoid**: Always define a health check in your Dockerfile or when running the container. Use a lightweight command that verifies the application's core functionality, such as an HTTP endpoint or a simple script. For example: ```dockerfile HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \ CMD curl -f http://localhost:8080/health || exit 1 ``` **Recovery**: If a container is unhealthy, inspect its health check logs with `docker inspect --format '{{json .State.Health}}' ` to understand what is failing. ### Pitfall 3: Modifying a Running Container Instead of Recreating **Why it happens**: It is tempting to use docker exec to modify a running container (e.g., install missing packages) rather than updating the Dockerfile and recreating the container. However, these changes are lost when the container is restarted or replaced, leading to inconsistent environments. **How to avoid**: Treat containers as immutable. Any required change should be reflected in the Dockerfile or entrypoint, and the container should be recreated. Use docker exec only for diagnostics, not for persistent changes. **Recovery**: If you have already modified a running container, document the changes and immediately rebuild the image with those changes incorporated. ### Pitfall 4: Overlooking Network Isolation **Why it happens**: Containers on different networks cannot communicate by default. Developers often assume that all containers can reach each other, leading to connection refused errors. **How to avoid**: Explicitly define networks and attach containers that need to communicate. Use custom networks for better isolation and DNS resolution. For example: ```bash docker network create backend docker run -d --network backend --name db postgres docker run -d --network backend --name webapp myapp ``` **Recovery**: If a container cannot reach another service, check `docker network inspect ` to see if both containers are on the same network. If not, recreate one with the correct --network flag. ### Pitfall 5: Running Containers as Root **Why it happens**: Many base images run processes as root by default. If the application is compromised, the attacker gains root privileges inside the container, which can potentially lead to host compromise. **How to avoid**: Always create a non-root user in your Dockerfile and run the application as that user. For example: ```dockerfile RUN addgroup -S appgroup && adduser -S appuser -G appgroup USER appuser ``` **Recovery**: If you have containers running as root, plan to rebuild them with a non-root user. This is especially important for production environments. ### Pitfall 6: Using Latest Tag in Production **Why it happens**: Using the "latest" tag makes it easy to pull the newest image, but it is not deterministic. The image content can change without warning, leading to unexpected behavior and making rollback difficult. **How to avoid**: Use explicit version tags for all images, and consider using image digests for absolute immutability. For example, use `myapp:1.4.2` instead of `myapp:latest`. **Recovery**: If you have been using latest, start tagging images with specific versions moving forward. You can also use `docker image inspect myapp:latest` to find the digest and tag it with a version for current deployments. ## Operations Checklist Use this checklist for routine debugging and maintenance. Each item has an owner (a single accountable role, not a group) and a review frequency to ensure accountability. | Step | Action | Command/Snippet | Owner | Review Frequency | |------|--------|------------------|-------|------------------| | 1 | Verify Docker version and environment | `docker version` and `docker info` | DevOps Engineer (e.g., Priya Shah) | Monthly or after host updates | | 2 | List running containers with status | `docker ps -a` | On-call engineer | Daily or per incident | | 3 | Check container logs | `docker logs --tail 100` | Application owner (e.g., Alex Chen) | On every deployment or incident | | 4 | Inspect container configuration | `docker inspect ` | DevOps Engineer | When configuration changes | | 5 | Check health status | `docker inspect --format '{{.State.Health.Status}}' ` | On-call engineer | Continuous monitoring (with alerts) | | 6 | View resource usage | `docker stats --no-stream` | SRE (e.g., Maria Garcia) | Weekly or when performance issues arise | | 7 | Execute diagnostic commands inside container | `docker exec -it sh` | Developer on-call | Only during debugging | | 8 | Verify logging configuration | `docker inspect --format '{{.HostConfig.LogConfig.Type}}' ` | DevOps Engineer | After container creation or monthly | | 9 | Test network connectivity from container | `docker exec ping ` | Network engineer (if available) or on-call | When network issues reported | | 10 | Backup or copy important files from container | `docker cp : ` | Data owner | Before container deletion or migration | | 11 | Clean up unused containers and images | `docker system prune` (with caution) | DevOps Engineer | Weekly (scheduled) | | 12 | Document findings and update runbooks | - | Incident responder | After every significant incident | ## Conclusion Debugging Docker containers requires a systematic approach grounded in understanding the architecture. By inventorying your environment, configuring logging and health checks safely, using diagnostic commands effectively, and preparing for common failures, you can resolve issues quickly and prevent future ones. The commands and configurations in this guide provide a solid foundation. Start by applying these techniques to your current containers, and build a debugging checklist tailored to your applications. Remember to keep images versioned for easy rollback, and always monitor logs and health status. With practice, container debugging becomes a manageable routine rather than a mystery.