## Intro

Docker is a core building block of modern software delivery, yet its internal architecture is often treated as a black box. This guide explains how Docker fits together end to end: the client-daemon model, images and layers, runtimes, networking, storage, security, and diagnostics. Along the way, you will run practical commands that mirror what you do in day-to-day engineering, so you can troubleshoot confidently and design safer, more reliable systems.

## The Big Picture: Components and Responsibilities

Docker follows a client-server model with a runtime stack under the hood:

- Docker client: The CLI (docker) that you use locally or remotely. It parses commands and talks to the Docker daemon via the Docker API.

- Docker daemon (dockerd): The server that builds images, manages containers, networks, and volumes, and exposes the REST API over a Unix socket or TCP.

- containerd and runc: containerd orchestrates container lifecycle and images for dockerd. runc is the low-level OCI runtime that actually creates containers using Linux primitives.

- Linux kernel features: Namespaces (PID, NET, MNT, UTS, IPC), cgroups, capabilities, seccomp, and AppArmor/SELinux power isolation and resource limits.

- Images and registries: Images are read-only, layered filesystems. Registries (Docker Hub, private registries) store and distribute them.

- Containers, networks, and volumes: Containers are runnable instances of images with a thin writable layer. Networks connect containers; volumes (or bind mounts) persist and share data.

Think of the Docker client as the remote control, the daemon as the control plane, containerd/runc as the execution engine, and the kernel as the isolation and resource manager.

## How Images and Layers Work

Images are built from a Dockerfile, where each instruction typically creates a layer. Layers are content-addressed and cached, which speeds up rebuilds and lets multiple images share base layers on disk.

Example: a tiny image that runs curl.

# Dockerfile
FROM alpine:3.20
RUN apk add --no-cache curl
CMD ["curl", "-sS", "https://example.com"] 
 Build and run:

docker build -t mycurl .
docker run --rm mycurl 
 Tip: Order Dockerfile instructions so that rarely changing steps (e.g., apt/apk adds) come before frequently changing code copies. This maximizes cache hits and faster rebuilds.

To see layer sharing and disk use:

docker image inspect mycurl --format '{{json .RootFS.Layers}}' | jq '.'
docker system df 

## Data Flow: Build, Pull/Push, and Run

- Build: The client tars your build context and sends it to the daemon. The daemon executes the Dockerfile, producing a layered image.

- Pull/Push: The daemon authenticates to a registry, negotiates manifests, downloads or uploads layers by digest, and caches them locally.

- Run: The daemon asks containerd to create a container using the image’s root filesystem and configuration. runc sets up namespaces, cgroups, and mounts, then execs the process. The container runs as a regular Linux process, isolated from the host.

 Trace activity in real time:

# Show daemon events as you build and run
docker events --since 10m &
docker build -t demo .
docker run --rm demo
kill %1 

## Control Flow: How the Client Talks to the Daemon

 The client sends HTTP requests to the Docker API over a Unix socket by default. You can target remote daemons via SSH or TLS-secured TCP. This separation enables remote management, RBAC via proxies, and clear security boundaries.

Examples:

# Local (Unix socket)
docker ps

# Remote via SSH (no daemon TCP port needed)
export DOCKER_HOST=ssh://ubuntu@prod01
docker ps

# Explicit API check
docker version 
 If you must expose TCP, use TLS with client certificates and firewall rules. Prefer SSH transport for simplicity and security.

## Networking in Practice

Docker provides several network drivers:

- bridge: The default local network with NAT. User-defined bridges add embedded DNS and better isolation.

- host: Shares the host’s network stack (no isolation). Useful for performance-sensitive cases.

- none: No networking.

- macvlan/ipvlan: Attach containers directly to the physical network with unique MAC/IP addresses.

Create an isolated app network and test service discovery:

docker network create appnet
docker run -d --name web --network appnet -p 8080:80 nginx:alpine
# Use a tiny curl image to reach the web container by name
docker run --rm --network appnet curlimages/curl:8.8.0 curl -sS http://web 
 Inspect and diagnose:

docker network inspect appnet 
 If you see IP conflicts (especially on laptops or VPNs), configure custom address pools:

{
 "default-address-pools": [
 { "base": "10.90.0.0/16", "size": 24 }
 ]
} 

## Storage: overlay2, Volumes, and Bind Mounts

 Most Linux hosts use overlay2, a union filesystem that stacks image layers plus a small writable container layer. For persistent data, use volumes or bind mounts rather than relying on the container’s writable layer.

- Volumes: Managed by Docker, stored under /var/lib/docker/volumes. Ideal for databases and state.

- Bind mounts: Map an existing host path into the container. Ideal for development or when you must control the host path.

Example with a named volume:

docker volume create pgdata
docker run -d --name db \
 -e POSTGRES_PASSWORD=secret \
 -v pgdata:/var/lib/postgresql/data \
 postgres:16 
 Check disk usage and prune stale data carefully:

docker system df
docker image prune -f
docker container prune -f
docker volume prune # double-check before running in prod 

## Safe Configuration: Hardening the Daemon and Containers

 Follow a conservative baseline for production:

- Do not expose the daemon without TLS. Prefer SSH transport.

- Enable user namespace remapping to reduce host-UID privilege exposure.

- Use live-restore so running containers survive daemon restarts.

- Limit container privileges: drop capabilities, use seccomp/AppArmor, set read-only filesystems, and run as a non-root user where possible.

- Configure log rotation to avoid unbounded json-file growth.

Minimal daemon.json:

{
 "userns-remap": "default",
 "live-restore": true,
 "log-driver": "json-file",
 "log-opts": { "max-size": "10m", "max-file": "5" }
} 
 Apply and verify:

sudo systemctl restart docker
docker info --format '{{.SecurityOptions}}' 
 Run a web server safely on port 80 without full root capabilities:

docker run -d --name web-safe \
 --read-only \
 --tmpfs /tmp \
 --cap-drop ALL --cap-add NET_BIND_SERVICE \
 --user 1000:1000 \
 -p 8080:80 \
 nginx:alpine 
 To pin exact content and avoid tag drift, use image digests:

docker pull nginx:alpine
docker inspect --format='{{index .RepoDigests 0}}' nginx:alpine
# Example: nginx@sha256:...
docker run nginx@sha256:... 
 Rootless mode is another strong option where supported; it runs dockerd and containers without root on the host, further narrowing risk.

## Verification and Diagnostics

Start with an environment inventory:

docker version
docker info | less 
 End-to-end check:

docker run --rm hello-world 
 When something misbehaves, use this short toolkit:

- Logs and details: docker logs -f <id> , docker inspect <id> , docker top <id>

- System activity: docker events --since 10m , docker stats

- Networks: docker network ls , docker network inspect <net>

- Disk pressure: docker system df , df -h , du -sh /var/lib/docker

- Host services: journalctl -u docker -u containerd --since 1h , systemctl status docker

## Common Failure Modes and Recovery

- Daemon will not start: Check journalctl -u docker . Verify daemon.json syntax and storage driver compatibility. Try reverting recent config changes.

- Containers fail to start after host reboot: Enable live-restore . Inspect errors with docker inspect --format '{{.State.Error}}' <id> .

- Low disk space on overlay2: Pulls and starts may fail. Assess with docker system df and host df -h . Prune unused artifacts and expand the filesystem if needed.

- Network conflicts or DNS issues: Inspect with docker network inspect . Consider custom default-address-pools . Ensure VPN/host routes do not overlap container CIDRs.

- Log file bloat: Set json-file rotation (daemon-level) or run-time log options per container.

Rollback strategies:

- Pin and revert by digest or a previous tag:

docker run --rm nginx@sha256:<previous> 

- For services (e.g., in Docker Swarm), use built-in rollback:

 docker service update --rollback my_service 

- Keep volume backups for stateful services; test restore regularly.

## Practical Walkthrough: From Zero to Verified

- Record versions and drivers:

 docker version
docker info | egrep 'Storage Driver|Cgroup|Security Options' 

- Build and run a sample image (cache-friendly ordering):

 FROM alpine:3.20
RUN apk add --no-cache curl
COPY . /app
WORKDIR /app
CMD ["sh", "-c", "curl -sS https://example.com"] 
 docker build -t sample .
docker run --rm sample 

- Create an app network and validate connectivity:

 docker network create appnet
docker run -d --name web --network appnet nginx:alpine
docker run --rm --network appnet curlimages/curl:8.8.0 curl -sS http://web 

- Persist data with a named volume:

 docker volume create appdata
docker run -d --name kv \
 -v appdata:/data \
 redis:7-alpine 

- Apply safe daemon settings and confirm:

 sudo tee /etc/docker/daemon.json >/dev/null <<'JSON'
{
 "userns-remap": "default",
 "live-restore": true,
 "log-driver": "json-file",
 "log-opts": { "max-size": "10m", "max-file": "5" }
}
JSON
sudo systemctl restart docker
docker info --format '{{.SecurityOptions}}' 

## Operations Checklist

- Monitor daemon health: docker info, system logs, and alert on failures.

- Keep images current: pull updates, rebuild with patches, and test in staging.

- Control disk usage: docker system df; prune unused images, containers, and volumes on a schedule (with safeguards).

- Verify backups: regularly restore volumes in a non-prod environment.

- Harden defaults: userns-remap, live-restore, log rotation, and minimized privileges for containers.

- Document changes: track daemon.json edits, image updates, and network CIDRs.

 Quick disk check:

docker system df 

## Conclusion

 Understanding Docker’s architecture pays off every time you debug a flaky deploy, chase a networking glitch, or harden a production host. Keep the mental model clear: the client controls the daemon; the daemon orchestrates containerd and runc; the kernel enforces isolation; images and layers are content-addressed and cached; and networks and volumes round out the runtime environment. With the examples and checklists above, you can build, run, secure, diagnose, and recover Docker workloads with confidence.