## Intro Docker layer caching can dramatically speed up image builds and reduce CI costs, but many teams only understand the basics: "if the Dockerfile line doesn't change, the cache is reused." Reality is more complex. BuildKit, multi-stage builds, cache mounts, and even the order of COPY instructions can silently invalidate your cache or cause you to miss out on significant optimizations. This article goes beyond the basics to explain advanced layer caching concepts with practical examples. You will learn how Docker's cache works under the hood, how to inspect it, how to design Dockerfiles for maximum cache reuse, and how to recover when the cache lets you down. Every concept is paired with a concrete command, expected output, and troubleshooting guidance so you can apply these techniques immediately. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before debugging cache behavior, establish a clear picture of your Docker environment. The cache implementation differs between the legacy builder and BuildKit, and behavior varies across Docker versions. ### Check Docker Version and Builder Run the following commands to identify your Docker version and which builder is active: ```bash docker version --format '{{.Server.Version}}' docker buildx version docker system info --format '{{.Driver}}' ``` Example expected output: ``` 24.0.7 v0.12.1 overlay2 ``` If BuildKit is enabled (default in Docker 23+), you'll see **buildkit** as the builder driver. To verify directly: ```bash docker buildx ls ``` Look for a builder with `*` indicating it is current. If you see only the default builder with `DOCKER_BUILDKIT=0` in its configuration, you are using the legacy builder and should consider enabling BuildKit for better cache control. ### Prerequisites - Docker Engine 20.10 or later (BuildKit enabled by default) - Basic familiarity with Dockerfiles and `docker build` - A test project with a Dockerfile for experimentation ### Observe Current Cache State To see how much disk space the build cache is using: ```bash docker system df ``` Sample output (truncated): ``` TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 12 3 2.45GB 1.2GB (49%) Containers 5 2 150MB 100MB (66%) Local Volumes 8 4 1.1GB 500MB (45%) Build Cache 42 18 3.8GB 2.9GB (76%) ``` This shows build cache usage and how much could be reclaimed with `docker builder prune`. ### Data Location and Restart Test When cache is stored in a volume, confirm where files persist. For a named volume: ```bash docker volume inspect app_data ``` Output includes the mountpoint on the host. If using a bind mount (e.g., `./data:/var/lib/app`), ensure the host path exists and is writable. A restart test verifies persistence: ```bash docker run -d --name test-app -v app_data:/var/lib/app myimage # write some data inside the container docker exec test-app sh -c "echo hello > /var/lib/app/test.txt" docker stop test-app docker rm test-app docker run -d --name test-app -v app_data:/var/lib/app myimage docker exec test-app cat /var/lib/app/test.txt ``` If the file does not appear, the volume was not mounted correctly, or the application wrote to a different path. ## Safe Configuration Path Configuring cache behavior should be deliberate and reversible. The following steps guide you through enabling BuildKit, using cache mounts, and configuring external cache storage. ### Enable BuildKit For Docker 23+, BuildKit is enabled by default. For older versions, set the environment variable: ```bash export DOCKER_BUILDKIT=1 docker build -t myapp:latest . ``` Verify BuildKit is active by checking for cache mount support: ```bash docker build --progress=plain --no-cache -t test . 2>&1 | grep -i buildkit ``` If you don't see errors, BuildKit is likely active. You can also check: ```bash docker buildx debug build --invoke /bin/true . ``` ### Use Cache Mounts for Dependencies One of the most impactful advanced techniques is using cache mounts in `RUN` instructions to persist package manager caches between builds. Example for a Node.js project: ```dockerfile # syntax=docker/dockerfile:1 FROM node:20-alpine WORKDIR /app COPY package.json yarn.lock ./ RUN --mount=type=cache,target=/root/.yarn \ yarn install --frozen-lockfile COPY . . RUN yarn build ``` Here, the Yarn cache directory `/root/.yarn` is stored in a cache mount that persists across builds even if the `RUN` layer is invalidated. The `COPY package.json yarn.lock` line ensures dependency installation only reruns when those files change. To verify the cache mount is working, build twice with a change only to source code (not package files). The second build should skip the `yarn install` step and show `CACHED` for that layer: ``` #7 [3/5] COPY package.json yarn.lock ./ #7 CACHED #8 [4/5] RUN --mount=type=cache,target=/root/.yarn yarn install --frozen-lockfile #8 CACHED ``` For Python with pip: ```dockerfile RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt ``` For Go modules: ```dockerfile RUN --mount=type=cache,target=/go/pkg/mod go mod download ``` ### Configure External Cache (Registry or Local) BuildKit supports exporting cache to a registry or local directory to share across CI runs. For a registry cache: ```bash docker buildx build --cache-to=type=registry,ref=myregistry/cache:myapp --cache-from=type=registry,ref=myregistry/cache:myapp -t myapp:latest . ``` For a local cache directory: ```bash docker buildx build --cache-to=type=local,dest=./build-cache --cache-from=type=local,src=./build-cache -t myapp:latest . ``` This is especially useful in CI where you cannot rely on the local Docker cache persisting between jobs. ### Safety: Use Placeholders, Not Secrets In build arguments or environment variables, never pass real secrets directly. Use BuildKit secrets: ```dockerfile # syntax=docker/dockerfile:1 FROM alpine RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret ``` Pass the secret at build time: ```bash docker build --secret id=mysecret,src=./secret.txt -t myapp . ``` This ensures the secret is not baked into the image layer or cache. ## Verification and Diagnostics After configuring caching, you need to verify it works and diagnose when it does not. This section provides concrete commands for inspecting cache usage, analyzing layer rebuilds, and troubleshooting common problems. ### Inspect Build Output for Cache Hits Always build with `--progress=plain` to see a clear layer-by-layer log: ```bash docker build --progress=plain -t myapp:latest . ``` Sample output: ``` #1 [internal] load build definition from Dockerfile #1 transferring dockerfile: 150B done #1 DONE 0.0s #2 [internal] load .dockerignore #2 transferring context: 2B done #2 DONE 0.0s #3 [internal] load metadata for docker.io/library/node:20-alpine #3 DONE 1.2s #4 [1/5] FROM docker.io/library/node:20-alpine #4 CACHED #5 [2/5] WORKDIR /app #5 CACHED #6 [3/5] COPY package.json yarn.lock ./ #6 CACHED #7 [4/5] RUN --mount=type=cache,target=/root/.yarn yarn install --frozen-lockfile #7 CACHED #8 [5/5] COPY . . #8 CACHED #9 exporting to image #9 exporting layers 0.2s done #9 writing image sha256:... done ``` Every line with `CACHED` means the layer was reused. If you see `DONE` with a timestamp, that layer was rebuilt. ### Analyze Cache with BuildKit Debug For deeper analysis, use `buildx debug` to inspect the build graph and cache entries: ```bash docker buildx debug build --invoke /bin/true . ``` This prints the entire build plan, including steps and their cache keys. You can also use `--print=outline`: ```bash docker buildx build --print=outline -t myapp:latest . ``` Expected output shows the Dockerfile steps with their corresponding stages and dependencies. ### Check Cache Size and Prune To see current cache usage: ```bash docker builder du ``` Sample output: ``` ID RECLAIMABLE SIZE LAST ACCESSED m3p5n7v9w2x1y4z6a8b0c true 1.5GB 2 hours ago ... Total: 3.8GB ``` To prune only unused cache: ```bash docker builder prune ``` To prune everything including active cache (use with caution): ```bash docker builder prune -a ``` ### Troubleshooting Common Cache Misses **Symptom: A layer is rebuilt even though its instruction did not change.** - Cause: A previous layer changed, changing the context for this layer. In Docker, each layer's cache key includes the content of all previous layers. - Fix: Reorder instructions to put frequently changing instructions later. For example, move `COPY . .` after dependency installation. **Symptom: Cache is not used at all.** - Check if BuildKit is enabled: `docker buildx ls` - Check if `--no-cache` was inadvertently passed. - Check if the Dockerfile has a `# syntax` directive that is incompatible with your BuildKit version. **Symptom: Cache mounts do not seem to persist across builds.** - Ensure you are using BuildKit. Legacy builder ignores `--mount` flags. - Ensure the `RUN` instruction with `--mount` is not invalidated by earlier layer changes. The cache mount persists regardless, but the `RUN` will re-execute, and the mount will be reused only if the previous layer cache is valid. - Verify the target path matches the tool's cache directory. For example, Yarn cache may be at `/usr/local/share/.cache/yarn` depending on the base image. ## Failure Modes and Recovery Even with careful design, cache failures happen. This section covers common failure modes, how to detect them, and how to recover without breaking your build. ### Failure: Cache Poisoning with Stale Dependencies If you use a floating tag like `node:20` or rely on a package manager that resolves versions loosely, you may get a cached layer with outdated dependencies. Even if the Dockerfile hasn't changed, the underlying package versions may have changed upstream, but the cache will not know. **Detection:** - Review the build log for the timestamp of the `RUN` layer. If it is older than expected, you may be using an old cache. - In CI, monitor for security advisories in dependencies and force a cache bust periodically. **Recovery:** - Pin base images by digest: `FROM node:20-alpine@sha256:...` - Use dependency lock files and ensure they are copied before `RUN` install commands. - Periodically use `--no-cache` or `docker builder prune -af` to force a full rebuild, but only in a controlled manner. ### Failure: Cache Bloat and Disk Pressure Over time, build cache can consume large amounts of disk space, causing failed builds due to no space left on device. **Detection:** - Run `docker system df` and check the `Build Cache` row. - Monitor disk usage on the Docker data root (`/var/lib/docker`). **Recovery:** - Use `docker builder prune` with filters to remove old cache: ```bash docker builder prune --filter "until=24h" ``` - Set a retention policy in the Docker daemon configuration: ```json { "builder": { "gc": { "enabled": true, "defaultKeepStorage": "10GB" } } } ``` - Use external cache with registry and manage retention there. ### Failure: Cache Invalidation Due to File Permissions or Metadata Changing file permissions or ownership in the build context can invalidate cache for `COPY` layers, even if file contents are identical. **Detection:** - Compare checksums of the context sent to the builder. Use `docker build --progress=plain` and look for the `transferring context` size. If it differs unexpectedly, something changed. **Recovery:** - Use `.dockerignore` to exclude irrelevant files that might change permissions. - Normalize permissions before build using a script or in CI. - Be consistent with how you construct the build context. ### Failure: Cache Mount Not Working in Multi-Stage Builds Cache mounts are scoped to a single stage unless shared. If you need to share a cache mount between stages, you must explicitly define it in each stage. **Example:** ```dockerfile # syntax=docker/dockerfile:1 FROM node:20-alpine AS deps WORKDIR /app COPY package.json yarn.lock ./ RUN --mount=type=cache,target=/root/.yarn yarn install --frozen-lockfile FROM node:20-alpine AS builder WORKDIR /app COPY --from=deps /app/node_modules ./node_modules COPY . . RUN yarn build ``` Here, the cache mount is only used in the `deps` stage. If you need the cache mount in the `builder` stage as well, you would define another `--mount` there. ## Operations Checklist Use this checklist to ensure your layer caching strategy is robust and measurable. Each item includes a verification command and expected signal. | Item | Verification | Expected Signal | Frequency | |------|--------------|-----------------|-----------| | BuildKit enabled | `docker buildx ls` | Active builder with BuildKit driver | Once per environment | | Dockerfile uses cache mounts | `grep -n "RUN --mount=type=cache" Dockerfile` | At least one match for dependency installation | Every Dockerfile review | | Base images pinned | `grep -n "^FROM" Dockerfile` | Images include digest or specific tag | Every Dockerfile review | | `.dockerignore` exists | `test -f .dockerignore && echo OK` | `OK` | Every project | | Cache size under control | `docker system df` | Build cache < 5GB (adjust to your limit) | Weekly | | Cache pruning scheduled | `crontab -l` or CI job | Runs `docker builder prune --filter until=168h` | Weekly | | External cache configured (CI) | `docker buildx build --cache-from=type=registry,ref=...` succeeds | Build log shows `CACHED` for most layers | Every CI run | | Secret handling safe | `docker build --secret id=mysecret,src=./secret.txt` and inspect image: `docker history myapp` | No secret string in history | Every build with secrets | Additionally, perform a restart test for any volume-backed cache: ```bash docker run -d --name cache-test -v cache-vol:/cache myimage sh -c "echo test > /cache/test && sleep 1000" docker stop cache-test docker start cache-test docker exec cache-test cat /cache/test ``` Expected output: `test`. If not, the volume mount is misconfigured. ## Conclusion Docker layer caching is a powerful feature, but its advanced concepts require understanding and deliberate design. By implementing cache mounts, using BuildKit features like external cache and secrets, and regularly verifying cache behavior, you can achieve faster builds and more reliable pipelines. Start with one low-risk improvement: add a cache mount to your dependency installation step. Run the build twice and confirm the second build shows `CACHED` for that step. Then, explore external caching for CI. Always observe before changing, and keep recovery plans ready for cache failures. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.