Docker Container Debugging Architecture Explained with Practical Examples

Intro

Debugging a containerized application often feels like solving a mystery. Processes are isolated from the host, file systems are ephemeral, and networking adds layers of indirection. When your web app stops responding or crashes, the path to resolution is rarely obvious. To debug effectively, you need to understand the architecture behind Docker container debugging: the components involved, how data flows between them, and where failures commonly occur. This guide provides that architectural view, along with practical examples, commands, and configuration snippets you can use immediately.

Docker container debugging is not a single tool but a combination of Docker's built-in features and external utilities. The core components include the Docker CLI, the Docker daemon, container runtime (runc or containerd), logging drivers, exec and inspect commands, and monitoring tools. By understanding how these pieces interact, you can diagnose issues faster and avoid common pitfalls. This guide covers version and environment inventory, safe configuration, verification and diagnostics, failure modes, and an operations checklist.

Throughout this article, we will use a simple web application container as a running example. It is a Node.js app that occasionally crashes due to unhandled exceptions, and we will debug it step by step. All commands are tested on Linux (Ubuntu 22.04) with Docker Engine 24.0, but they should work with minor modifications on other systems.

Version and Environment Inventory

Before debugging any container, establish a clear baseline. Know the Docker version, the container's image and runtime, and the host's operating system. This information is often critical when researching known issues or determining compatibility.

Prerequisites

• Docker Engine 20.10 or later (we use 24.0).
• Basic familiarity with the command line.
• A container to debug (we use webapp:1.0).
• Access to container logs and the ability to execute commands inside containers.

Checking Docker Version and Info

Run the following commands on the host:

docker version --format '{{.Server.Version}}'

Expected output (yours may differ):

24.0.5

docker info --format '{{.OperatingSystem}} {{.Architecture}} {{.CgroupDriver}}'

Expected output:

Ubuntu 22.04.3 LTS x86_64 systemd

Inspecting the Container

Identify the container you want to debug:

docker ps -a --filter "name=webapp" --format "table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.Names}}"

Example output:

CONTAINER ID   IMAGE        STATUS                     NAMES
3f4a5b6c7d8e   webapp:1.0   Up 10 minutes (unhealthy)  webapp

Note the "(unhealthy)" status; this indicates the container's health check is failing. We will investigate later.

Topology

In a typical Docker debugging setup, you have:

• Host: Runs the Docker daemon and provides resources.
• Docker daemon (dockerd): Manages containers, images, networks, and volumes.
• Container runtime (containerd / runc): Actually creates and runs containers.
• Container: The application process and its dependencies.
• Logging driver: Captures container stdout/stderr and forwards to a destination (json-file by default).
• Debugging tools: docker exec, docker logs, docker inspect, system tools like strace, tcpdump, etc.

Data flow: Application writes to stdout/stderr -> container runtime -> logging driver -> log file or remote endpoint. Control flow: User issues docker exec -> Docker CLI -> daemon -> runtime -> execute command in container namespace.

Safe Configuration Path

Before making any changes, implement a safe debugging configuration. This means enabling appropriate logging, setting up health checks, and ensuring you can execute commands in the container without disrupting production if possible. We will do this in a scoped, reversible way.

Configuring Logging Drivers

Docker's default logging driver is json-file, which stores logs on the host. For debugging, it is often helpful to configure a different driver or adjust options. We will retain json-file but set a max size to avoid disk fill.

Create a container with logging options:

docker run -d --name webapp \
  --log-driver json-file \
  --log-opt max-size=10m \
  --log-opt max-file=3 \
  -p 8080:8080 \
  webapp:1.0

This limits the log files to 10 MB each and keeps 3 rotated files. If you need to change logging for an existing container, you must recreate it. To see the current logging config:

docker inspect --format '{{.HostConfig.LogConfig.Type}} {{.HostConfig.LogConfig.Config}}' webapp

Output:

json-file map[max-file:3 max-size:10m]

Setting Up Health Checks

A health check tells Docker whether the application is functioning. For our Node.js app, we can add an HTTP health endpoint. Modify the Dockerfile to include HEALTHCHECK:

FROM node:18-alpine
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 8080
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:8080/health || exit 1
CMD ["node", "server.js"]

Rebuild the image:

docker build -t webapp:1.1 .

Run the new container:

docker run -d --name webapp-healthy \
  --log-driver json-file --log-opt max-size=10m --log-opt max-file=3 \
  -p 8080:8080 \
  webapp:1.1

Check health status after a minute:

docker ps --format '{{.HealthStatus}}' --filter name=webapp-healthy

Expected: healthy.

Enabling Exec Access

For debugging, you need to run commands inside the container. Ensure the container is not running with --read-only or a restricted security profile that prevents exec. We will use docker exec for interactive debugging.

Verification and Diagnostics

Now that the container is running with proper logging and health checks, we can diagnose issues. Let us simulate a failure in our Node.js app: it crashes when a certain endpoint is hit due to an unhandled promise rejection.

Observing Logs

First, check the logs:

docker logs webapp-healthy

You might see something like:

Listening on port 8080
...
(node:1) UnhandledPromiseRejectionWarning: Error: Something went wrong
    at /app/server.js:15:11
...

This tells us where the error is. We can also follow logs live:

docker logs -f --tail 50 webapp-healthy

Executing Commands in the Container

We can inspect the running process and environment:

docker exec webapp-healthy ps aux

Expected output (simplified):

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.5 723456 45678 ?        Ssl  12:34   0:00 node server.js
root        12  0.0  0.0  15568  2024 ?        Rs   12:35   0:00 ps aux

If the container is unhealthy, we might want to inspect the health check logs:

docker inspect --format '{{json .State.Health}}' webapp-healthy

Output includes log entries from health check runs.

Using docker inspect for Configuration

docker inspect reveals a wealth of information:

docker inspect webapp-healthy

Key fields: NetworkSettings.IPAddress, Mounts, Config.Env, HostConfig.Binds, etc. For debugging, check the network settings to ensure port mapping:

docker inspect --format '{{range $p, $conf := .NetworkSettings.Ports}}{{$p}} -> {{(index $conf 0).HostPort}}{{"\n"}}{{end}}' webapp-healthy

Output:

8080/tcp -> 8080

Copying Files for Analysis

Sometimes you need to extract files from a container, such as core dumps or logs written to disk. Use docker cp:

docker cp webapp-healthy:/app/server.js ./server.js

Debugging with strace

If the process is behaving strangely, strace can trace system calls. First, install strace inside the container (if not present):

docker exec -it webapp-healthy sh -c "apk add --no-cache strace"

Then attach to the process:

docker exec -it webapp-healthy strace -p 1

This might be heavy; use it with caution on production containers.

Failure Modes and Recovery

Understanding common failure modes helps you prepare for and recover from issues. We will cover typical problems: container exiting immediately, unresponsive container, disk full due to logs, and network misconfiguration.

Container Exits Immediately

If a container starts and exits, check its exit code and logs.

docker ps -a --filter "name=webapp" --format "{{.Status}}"

Example:

Exited (1) 10 seconds ago

Then inspect logs:

docker logs webapp

Often it is a missing dependency or configuration error. Fix the Dockerfile or entrypoint and restart.

Unresponsive Container

If a container hangs and does not respond to commands, it may be out of memory or have a deadlocked process. Check resource usage:

docker stats --no-stream webapp-healthy

Example output:

CONTAINER ID   NAME              CPU %     MEM USAGE / LIMIT     MEM %     NET I/O       BLOCK I/O   PIDS
3f4a5b6c7d8e   webapp-healthy    0.00%     45.2MiB / 3.8GiB      1.16%     2.1kB / 0B   0B / 0B     2

If memory is high, consider increasing limits or profiling the app. You can also use docker exec to run diagnostic commands, but if the container is truly hung, you may need to restart it.

Disk Full Due to Logs

If the host disk fills up, container logging may be the culprit. Check disk usage of Docker logs:

sudo du -sh /var/lib/docker/containers/*/*-json.log

If a file is huge, rotate or remove it. Better, configure log rotation from the start. To clear logs without restarting the container, use:

truncate -s 0 /var/lib/docker/containers/<container-id>/<container-id>-json.log

Get container ID with docker ps -q.

Network Misconfiguration

If the app cannot connect to other services, inspect the network:

docker network inspect bridge

Check if containers are on the same network. For custom networks, create one:

docker network create mynet

Then run containers with --network mynet. Use docker exec to test connectivity:

docker exec webapp-healthy curl http://database:5432

Recovery Steps

• For immediate recovery, restart the container: docker restart webapp-healthy.
• If the issue persists, recreate the container with updated configuration.
• For code bugs, fix the application and rebuild the image.

Rollback

Keep previous image versions to rollback:

docker tag webapp:1.1 webapp:stable
docker run -d --name webapp-stable ... webapp:stable

If the new version fails, switch back to the stable image.

Common Pitfalls and How to Avoid Them

Even experienced Docker users fall into predictable traps when debugging containers. Here are the most common pitfalls, why they happen, and how to avoid or recover from them.

Pitfall 1: Neglecting Log Rotation

Why it happens: Default logging configuration stores all container output in a single JSON file without size limits. Over time, this file can fill the host disk, causing system-wide failures.

How to avoid: Always set log rotation options when running containers, either via the --log-opt flags or in the Docker daemon configuration file (/etc/docker/daemon.json). For example, add this to daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

Then restart the Docker daemon.

Recovery: If disk is already full, identify large log files with sudo du -sh /var/lib/docker/containers//-json.log and truncate them as shown in the previous section. Then implement rotation to prevent recurrence.

Pitfall 2: Ignoring Health Check Status

Why it happens: Many developers do not configure health checks for their containers. Without them, Docker cannot report whether the application is actually working, only whether the process is running. A container may be "Up" but unresponsive.

How to avoid: Always define a health check in your Dockerfile or when running the container. Use a lightweight command that verifies the application's core functionality, such as an HTTP endpoint or a simple script. For example:

HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://localhost:8080/health || exit 1

Recovery: If a container is unhealthy, inspect its health check logs with docker inspect --format '{{json .State.Health}}' <container> to understand what is failing.

Pitfall 3: Modifying a Running Container Instead of Recreating

Why it happens: It is tempting to use docker exec to modify a running container (e.g., install missing packages) rather than updating the Dockerfile and recreating the container. However, these changes are lost when the container is restarted or replaced, leading to inconsistent environments.

How to avoid: Treat containers as immutable. Any required change should be reflected in the Dockerfile or entrypoint, and the container should be recreated. Use docker exec only for diagnostics, not for persistent changes.

Recovery: If you have already modified a running container, document the changes and immediately rebuild the image with those changes incorporated.

Pitfall 4: Overlooking Network Isolation

Why it happens: Containers on different networks cannot communicate by default. Developers often assume that all containers can reach each other, leading to connection refused errors.

How to avoid: Explicitly define networks and attach containers that need to communicate. Use custom networks for better isolation and DNS resolution. For example:

docker network create backend
docker run -d --network backend --name db postgres
docker run -d --network backend --name webapp myapp

Recovery: If a container cannot reach another service, check docker network inspect <network> to see if both containers are on the same network. If not, recreate one with the correct --network flag.

Pitfall 5: Running Containers as Root

Why it happens: Many base images run processes as root by default. If the application is compromised, the attacker gains root privileges inside the container, which can potentially lead to host compromise.

How to avoid: Always create a non-root user in your Dockerfile and run the application as that user. For example:

RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

Recovery: If you have containers running as root, plan to rebuild them with a non-root user. This is especially important for production environments.

Pitfall 6: Using Latest Tag in Production

Why it happens: Using the "latest" tag makes it easy to pull the newest image, but it is not deterministic. The image content can change without warning, leading to unexpected behavior and making rollback difficult.

How to avoid: Use explicit version tags for all images, and consider using image digests for absolute immutability. For example, use myapp:1.4.2 instead of myapp:latest.

Recovery: If you have been using latest, start tagging images with specific versions moving forward. You can also use docker image inspect myapp:latest to find the digest and tag it with a version for current deployments.

Operations Checklist

Use this checklist for routine debugging and maintenance. Each item has an owner (a single accountable role, not a group) and a review frequency to ensure accountability.

Conclusion

Debugging Docker containers requires a systematic approach grounded in understanding the architecture. By inventorying your environment, configuring logging and health checks safely, using diagnostic commands effectively, and preparing for common failures, you can resolve issues quickly and prevent future ones. The commands and configurations in this guide provide a solid foundation. Start by applying these techniques to your current containers, and build a debugging checklist tailored to your applications. Remember to keep images versioned for easy rollback, and always monitor logs and health status. With practice, container debugging becomes a manageable routine rather than a mystery.