## Intro Slow Docker builds silently drain developer productivity and CI pipeline throughput. The build cache can dramatically reduce build times, but only if you understand its architecture, when it is invalidated, and how to share it across builds. This guide explains Docker Build Cache fundamentals with practical examples you can run locally: inspect cache hits and misses, structure Dockerfiles for optimal layer caching, use BuildKit cache mounts for package dependencies, and share cache across CI runs. By the end, you will be able to diagnose cache invalidation, recover from common mistakes, and make builds reliably faster. We will focus on the modern BuildKit builder, which Docker Desktop enables by default, and also cover legacy behavior where relevant. You will learn which Dockerfile instructions affect the cache, how to check whether a layer was cached, and how to avoid cache misses that cause redundant downloads or compilations. ## Version and Environment Inventory Before changing anything, observe what you are running. Check the Docker version and confirm BuildKit is enabled. Run `docker version` and look for `Server: Docker Engine` and `Client: Docker Engine`. To verify BuildKit, run `docker buildx version`. If the output shows `github.com/docker/buildx`, you are using BuildKit. Alternatively, use `docker system info --format '{{ .Builder }}'`; on older versions this may show empty for the legacy builder. You can set `DOCKER_BUILDKIT=1` in your environment to explicitly enable BuildKit on supported versions, or configure it in the Docker daemon (`/etc/docker/daemon.json`) with `"features": {"buildkit": true}`. Note the storage driver, which affects layer handling: `docker info --format '{{ .Driver }}'`. Common drivers are overlay2 on modern Linux, but on some systems you might see aufs or devicemapper. This is rarely a cache issue by itself, but useful when you need to inspect layer sizes or prune cache data. Before each build, capture the current state: `docker system df` shows reclaimed space usage by images, containers, volumes, and build cache. For a quick cache inventory, use `docker buildx du` (BuildKit) to show how much disk the builder cache occupies. Example output: ```text Total: 1.234GB Reclaimable: 234.5MB ``` This tells you how much cache you have and how much can be reclaimed without breaking current builds. Do not prune cache before you understand what it contains; use `docker buildx du --verbose` to list cache entries by type and size, including regular layers and cache mounts. For data safety, avoid bind mounts for cache directories in production builds. Bind mounts expose host-specific paths and can leak state across builds. Prefer named volumes or BuildKit cache mounts, which are managed by Docker and isolated. A simple restart test for a service that writes local files: stop and recreate the container, then verify that data persists if it is in a volume. If data disappears, it was probably written to the container writable layer. ## Safe Configuration Path Build cache behavior is mostly controlled by Dockerfile structure and build arguments. The safest changes are read-only observations first, then one-scope modifications with a verification build. ### Dockerfile Instruction Cache Rules Each Dockerfile instruction creates a layer, and the layer cache is keyed by the instruction type, arguments, and parent layer. The most important rules: - `FROM base:tag` is cached by the image ID; changing the tag may or may not change the ID. - `RUN` commands are cached by the exact command string. Any difference, including a comment or whitespace, invalidates the cache. - `COPY` and `ADD` are cached based on file content checksums, not timestamps. File metadata changes like permissions or ownership also affect the checksum. - `ENV`, `LABEL`, `EXPOSE`, and `CMD` usually do not create large layers but still affect subsequent layers if changed. To verify that a layer was reused, build with `--progress=plain` (BuildKit) instead of the default fancy progress. For each step, the output shows `CACHED` if the layer was reused. Example: ```text #2 [1/4] FROM python:3.11-slim #2 CACHED #3 [2/4] RUN apt-get update && apt-get install -y curl #3 CACHED #4 [3/4] COPY requirements.txt . #4 sha256:1234... #5 [4/4] RUN pip install -r requirements.txt #5 0.532 done ``` In this example, the `FROM` and `RUN apt-get` steps are cached, but the `COPY` step wasn't because `requirements.txt` changed, so the subsequent `pip install` also ran fresh. If you see `CACHED` on every step, the entire image was reused from cache. Legacy builder (non-BuildKit) shows cache status differently: `---> Using cache` for a cached layer and `---> Running in ...` for a rebuilt layer. ### Practical Optimization: Order Matters Place instructions that change less frequently before those that change frequently. For a Python application, a typical optimal order is: ```dockerfile FROM python:3.11-slim WORKDIR /app # Install system dependencies first (changes rarely) RUN apt-get update && apt-get install -y --no-install-recommends gcc libpq-dev && rm -rf /var/lib/apt/lists/* # Copy dependency manifest and install Python packages (changes moderately) COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt # Copy application code (changes frequently) COPY . . CMD ["python", "app.py"] ``` Each step caches independently. If you change only application code, the first three layers are reused, and only the `COPY . .` and `CMD` (if changed) are rebuilt. If you instead copy everything at once and then install dependencies, any source code change invalidates the dependency installation layer. ### Using Build Arguments Without Invalidating Cache Build arguments (`ARG`) can be used in a way that does not break cache for later layers. An `ARG` used only in a layer does not affect cache if the value is unchanged. However, if you use `ARG` in a `RUN` command, the layer cache key includes the argument value. To avoid invalidating the entire cache when passing versionized parameters, define the `ARG` just before the `RUN` that uses it, not at the top. Example: ```dockerfile FROM node:18-alpine ... # ARG declared right before use ARG NODE_ENV=production RUN NODE_ENV=$NODE_ENV npm ci ``` This allows the layer before `ARG` to be reused even if `NODE_ENV` changes. If `ARG` were at the top, any difference in `NODE_ENV` would invalidate every subsequent layer. ### Multi-Stage Builds for Cache Isolation Multi-stage builds create separate cache chains per stage. The final stage only includes what you copy from earlier stages, often with `COPY --from=builder`. This is powerful because you can change the build stage without affecting the cache of the final runtime stage. Example: ```dockerfile # Compile stage FROM golang:1.21 AS builder WORKDIR /src COPY go.mod go.sum ./ RUN go mod download COPY . . RUN CGO_ENABLED=0 go build -o /app/main . # Runtime stage FROM alpine:3.19 COPY --from=builder /app/main /usr/local/bin/app CMD ["app"] ``` The `builder` stage can be rebuilt often without changing the runtime stage cache. You can also use `COPY --from=builder` with a cache mount to reuse the build cache between builds (see below). ## Verification and Diagnostics After you modify a Dockerfile, verify that the cache behaves as expected. Build twice and compare timings and cache status. ### Measure Build Time and Cache Hits Use the `time` command around your build: ```bash time docker build -t myapp . ``` First build might take minutes; second build with no changes should complete in milliseconds because all layers are cached. If the second build still takes a long time, something invalidated the cache. Use `docker build --progress=plain` to see per-step status. For a more quantitative check, use `docker buildx build --progress=plain --output=type=image` and look for `cache` lines in the output. BuildKit supports a `--metadata-file` option that outputs build metadata including cache information. Example: ```bash docker buildx build --metadata-file build-metadata.json -t myapp . cat build-metadata.json ``` The JSON contains a `containerimage.buildinfo` with layer digests and sizes, not directly cache hit/miss, but combined with a second build you can compare layer digests to see if layers were reused. For CI debugging, you can run `docker buildx du --verbose` to see cache entries by layer digest and size. This helps identify which layers are consuming the most cache space and whether cache mounts are being used effectively. ### Simulating Cache Invalidation Tests Create deliberate changes to see how cache invalidates: 1. Change a line in the source code but not `requirements.txt` or `Dockerfile`. 2. Rebuild and observe: only the `COPY . .` step should rebuild; dependency installation should be cached. 3. Add a comment to the `Dockerfile` at the top: the first `RUN` after that comment will rebuild because the parent layer changes. 4. Change an environment variable: layers after that `ENV` will rebuild. By performing these tests on a sample project, you gain intuition for which changes cause cache misses. ### Using Docker Build Cloud or Remote Builders If you use remote builders (e.g., Docker Build Cloud, GitHub Actions with Buildx), you can share cache across machines. BuildKit supports cache export and import using registry or local directories. For example, to export cache to a Docker Hub repository: ```bash docker buildx build --cache-to type=registry,ref=myuser/myapp:cache --cache-from type=registry,ref=myuser/myapp:cache -t myapp:latest . ``` On a fresh CI runner, import the cache with `--cache-from` to reuse previously built layers. This is a game-changer for CI pipelines where each job starts from scratch. ### Diagnosing Cache Misses with BuildGraph BuildKit has a debug feature to dump the build graph. Set `BUILDKIT_PROGRESS=plain` and `BUILDKIT_DEBUG=1` to see detailed logs. For Linux, you can use `docker buildx debug` to attach to the builder and run commands. The build graph shows dependencies between steps and cache keys. You can install `buildgraph` viewer or use `docker buildx imagetools` to inspect layers. ## Failure Modes and Recovery ### Common Failure: Cache Bloat and Disk Space Exhaustion Symptoms: `docker build` fails with `no space left on device`, or the system disk fills up. Build cache can grow unbounded, especially with frequent builds using cache mounts (which store dependency caches outside the image). Recovery: Prune build cache with `docker buildx prune`. This removes unused cache, but be careful: if you remove cache that is still needed for current builds, they will rebuild from scratch. Use `docker buildx prune --keep-storage 10GB` to keep a certain amount. For high-traffic CI, set up a periodic prune with `docker system prune -f --filter "until=24h"`. Prevention: Use `--cache-to` and `--cache-from` to store cache persistently and avoid rebuilding from scratch. Monitor `docker buildx du` as part of your health checks. ### Common Failure: Cache Invalidation Due to Copious Context If your build context includes many files or large files, the `COPY . .` step may take a long time and invalidate cache frequently. Use a `.dockerignore` file to exclude `.git`, `node_modules`, logs, and other irrelevant files. Example `.dockerignore`: ```text .git node_modules *.log dist .env ``` The `.dockerignore` not only speeds up the context transfer but also prevents cache misses from unnecessary file changes. ### Common Failure: Secrets Leaked into Cache Never `COPY` or `RUN` with secrets that end up in a cached layer. For example, `RUN curl -H "Authorization: Bearer $TOKEN"` embeds the token in the layer, and the layer may be shared in cache images. To avoid this, use BuildKit secrets: `RUN --mount=type=secret,id=mysecret curl ...` with the secret provided at build time via `--secret id=mysecret,src=./secret.txt`. The secret is not stored in the image. If you accidentally leaked a secret, you must rebuild the image without it and consider all previous layers compromised. ### Common Failure: Cache Mount Misuse and Orphaned Cache Cache mounts (`--mount=type=cache,target=/root/.cache`) are great for package managers, but if not used correctly, they can cause builds to use stale dependencies or leave orphaned cache entries. For example, if you use a cache mount for `/go/pkg/mod` in a Go build, and the `go.sum` changes, the module downloads might still be partially reused. To avoid stale cache, always include the dependency lock file in the layer that populates the cache, or use a versioned cache mount ID like `id=gomod-${GO_VERSION}`. Clean up unused cache mounts with `docker buildx prune --filter type=exec.cachemount`. ### Common Failure: Legacy Builder Cache and BuildKit Confusion If you sometimes build with the legacy builder and sometimes with BuildKit, cache behavior differs. Legacy builder caches only on the local daemon and has no cache mounts or export. This can lead to inconsistent builds on different machines. Standardize on BuildKit for all builds. In CI, set `DOCKER_BUILDKIT=1`. For Docker Compose, use `COMPOSE_DOCKER_CLI_BUILD=1` and `DOCKER_BUILDKIT=1` when building. ## Operations Checklist Here is a monthly operational checklist for managing Docker build cache. Update the responsible owner and review date as needed for your environment. | Check | Command or Action | Expected Result | Owner | Frequency | |---|---|---|---|---| | Cache disk usage | `docker buildx du` | Total cache size within budget (e.g., <5GB per builder) | DevOps Lead (Alex Rivera) | Weekly | | Build time baseline | `time docker build ...` on a fixed tag | No more than 20% regression vs. baseline | Build Engineer (Priya Shah) | Monthly | | Cache export validity | Inspect `--cache-to` registry image (e.g., `docker pull myuser/myapp:cache`) | Pull succeeds and contains required layers | DevOps Lead | Weekly | | Secret leakage check | Scan layers with `docker history --no-trunc` and grep for tokens | No secrets found in any layer | Security Officer (Jordan Lee) | Quarterly | | `.dockerignore` coverage | Compare `docker build --progress=plain` context size with and without ignore | Context size reduced by at least 50% | Developer (Sam Chen) | Monthly | | Cache mount efficiency | Run build twice; record `CACHED` steps and total time; ensure cache mount reduces time | Second build with no changes fully cached; with dependency change, package reinstall uses cache mount | Build Engineer | Monthly | | Multi-stage cache isolation | Modify a file only used in builder stage, rebuild, verify runtime stage not rebuilt | Runtime stage layer IDs unchanged | Developer | As needed | | CI cache reuse | Run build on clean CI runner with `--cache-from`, compare to local build | Build time close to local cached build | DevOps Lead | Weekly | | Prune old cache | `docker buildx prune --keep-storage 10GB` or scheduled cron | Disk reclaimed without breaking active builds | DevOps Lead | Monthly | | Documentation review | Update this checklist and cache-related runbooks | All changes reflected in docs | Team Lead (Morgan Diaz) | Quarterly | For each check, the owner is accountable for verifying the result and raising issues. Review this checklist at the monthly ops meeting to adjust budgets, frequencies, and owners as the project evolves. ## Pitfalls to Avoid 1. **Placing `COPY . .` early in the Dockerfile**: This invalidates all subsequent layers on every code change. Always place `COPY . .` as late as possible. 2. **Using `ADD` for remote files**: `ADD https://...` always downloads and busts cache unpredictably. Prefer `RUN curl` or `wget` and verify checksums, or use multi-stage with `COPY`. 3. **Ignoring `.dockerignore`**: Without it, the build context may include sensitive files and unnecessary data, causing slow transfers and frequent cache misses. 4. **Using `ARG` at the top for version-dependent builds**: This can invalidate the entire cache when the argument changes. Declare `ARG` just before use. 5. **Not using multi-stage builds**: Single-stage images are bloated and have poor cache granularity. Multi-stage separates build-time dependencies from runtime, allowing smaller images and better cache reuse. 6. **Running package installs with cache mounts without version control**: If the lock file doesn't change, packages will be cached, but if the lock file changes, the cache mount might still hold old dependencies. Always bind the cache mount to the lock file state, e.g., `--mount=type=cache,target=/root/.npm,id=npm-${{ hashFiles('package-lock.json') }}` in CI. 7. **Using legacy builder out of habit**: This forfeits the performance and features of BuildKit. Enable BuildKit globally via environment or daemon config. 8. **Storing secrets in image layers**: Even if removed later, they remain in history. Use BuildKit secrets or runtime secret mounts. 9. **Pruning aggressively without analysis**: Removing cache used by active builds can lead to painful rebuilds. Always check `docker buildx du` and keep a safe retention. 10. **Ignoring cache export in CI**: Without cache import/export, each CI job starts from zero, multiplying build times and resource usage. Each pitfall can be avoided with the practices described in earlier sections. When a mistake occurs, refer to the Failure Modes section for recovery steps. ## Conclusion Docker build cache architecture is not magic; it follows deterministic rules that you can observe, test, and optimize. By mastering layer caching, multi-stage builds, cache mounts, and cache sharing, you can cut build times by an order of magnitude and reduce CI costs. The key is to treat the cache as a valuable asset: monitor it, structure your Dockerfiles to maximize reuse, and avoid common pitfalls that silently invalidate it. Start with one practical improvement from this guide—for example, reorganize a Dockerfile or add a `.dockerignore`—measure the before and after build times using the commands provided, and iterate until your builds are reliably fast. Remember that a reliable build system makes failures visible, protects sensitive data, and gives you confidence that your Docker images are built efficiently and reproducibly.