## Intro Docker's build cache is one of the most powerful features for speeding up image builds, but it is also one of the most misunderstood in production environments. When the cache works, builds are fast and predictable. When it does not, developers lose time debugging inconsistent builds and operators may deploy images that are larger or older than expected. This article provides a production operations checklist for Docker Build Cache, with practical examples that move you from an observed problem to a verified result. We will cover the essential elements: identifying the installed Docker and BuildKit version, understanding your deployment topology, checking prerequisites, and inspecting the exact component that is misbehaving. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. This guide is written for developers, DevOps consultants, and technical startup teams who manage Docker in production. It connects Docker Build Cache operations, checklist items, best practices, and maintenance to real commands, expected outputs, failure signals, and recovery decisions. ## Version and Environment Inventory Before you touch the cache, you need to know what you are working with. Version and environment inventory answers: which Docker version, which BuildKit version, what storage driver, and what build context are you using? These details drastically affect cache behavior. ### Check Docker and BuildKit versions ```bash docker version ``` Expected output includes separate sections for Client and Server. Look for the `OS/Arch`, `Version`, and `Git commit` lines. For BuildKit, check if it is enabled: ```bash docker buildx version ``` If BuildKit is enabled, you will see `buildx` in the output. BuildKit is the default builder on Docker Desktop and on many Linux distributions with Docker 23.0+. To confirm BuildKit is being used for builds: ```bash docker buildx ls ``` If you see a builder named `default` with `BuildKit` in the driver column, you are using BuildKit. If you see `docker` as the driver, you are using the legacy builder. BuildKit has different cache semantics and is required for advanced cache features like `--mount=type=cache`. ### Understand the storage driver ```bash docker info --format '{{.Driver}}' ``` Common drivers are `overlay2`, `btrfs`, `zfs`, and `vfs`. The storage driver affects how layers are stored and how cache invalidation works. For most Linux systems, `overlay2` is recommended. If you are on `vfs`, builds will be significantly slower and cache performance will be poor. ### Check disk space and cache size ```bash docker system df ``` This shows disk usage by images, containers, local volumes, and build cache. The `Build Cache` column appears only with BuildKit. If it is large, that is not necessarily bad, but if disk pressure is causing evictions, you may see cache misses. Monitor disk space regularly; a full disk is a common cause of unexpected cache invalidation. ### Inspect the build context The build context is everything you send to the Docker daemon for a build. A large context slows builds and can include files that invalidate cache unnecessarily. Use `.dockerignore` to exclude irrelevant files: ```dockerignore # .dockerignore .git *.log node_modules tmp ``` Without a proper `.dockerignore`, a change in a log file or `.git` directory will invalidate layers that copy the entire context. This is a frequent source of cache misses. ### Separate observation from intervention Always capture current state and timestamps first. Protect credentials and private material. Only then make a scoped change when you understand its blast radius and recovery path. For example, if you suspect the cache is not being used, do not clear it immediately. Instead, run `docker buildx du` to see cache usage, and inspect which layers are being rebuilt with `docker build --progress=plain`. ## Safe Configuration Path This section covers how to safely configure and test Docker Build Cache changes. The principle is: make the smallest justified change and verify its effect before applying to production. ### Enable BuildKit explicitly To ensure BuildKit is used for all builds, you can set the environment variable: ```bash export DOCKER_BUILDKIT=1 ``` Or configure the Docker daemon (`/etc/docker/daemon.json`) with: ```json { "features": { "buildkit": true } } ``` Then restart Docker: ```bash sudo systemctl restart docker ``` Verify with `docker buildx ls` that the default builder is now BuildKit. ### Configure cache mounts in Dockerfile BuildKit allows you to mount a cache directory that persists between builds, even if the layer that uses it is invalidated. For example, for a Node.js project: ```dockerfile # syntax=docker/dockerfile:1.4 FROM node:20-alpine AS build WORKDIR /app COPY package*.json ./ RUN --mount=type=cache,target=/root/.npm \ npm install COPY . . RUN npm run build ``` The `--mount=type=cache` on the `npm install` command keeps the npm cache across builds. Without it, any change to `package.json` would force a full re-install. With it, only the changed packages are downloaded. ### Test in a staging environment Never change the cache configuration directly in production. Use a staging Docker host or a local environment that mimics production. For example, create a test Compose file: ```yaml # docker-compose.test.yml version: '3.8' services: app: build: context: . dockerfile: Dockerfile image: myapp:test ``` Run a build with verbose output to see cache hits: ```bash docker compose -f docker-compose.test.yml build --progress=plain ``` Look for `CACHED` in the output for layers that should be reused. If a layer is rebuilt unexpectedly, inspect the Dockerfile and context. ### Data persistence verification When dealing with volumes and bind mounts, a small production-like test should include a restart test. Create a container with a named volume: ```bash docker run -d --name app -v app_data:/var/lib/app myapp:latest ``` Write a test file, stop and recreate the container, and confirm the file persists: ```bash docker exec app sh -c 'echo test > /var/lib/app/test.txt' docker stop app docker rm app docker run -d --name app -v app_data:/var/lib/app myapp:latest docker exec app cat /var/lib/app/test.txt ``` If the file is not there, the service was writing to the container filesystem instead of the volume. This is a common cache-related mistake when the build process assumes data will persist. ## Verification and Diagnostics This section focuses on how to verify that the build cache is working correctly and how to diagnose when it is not. ### Observe cache hits with plain progress The default build output hides detailed cache information. Use `--progress=plain` to see every layer and its cache status: ```bash docker build --progress=plain -t myapp:latest . ``` Output example: ```text #1 [internal] load build definition from Dockerfile #1 transferring dockerfile: 32B done #1 DONE 0.0s #2 [internal] load .dockerignore #2 transferring context: 2B done #2 DONE 0.0s #3 [internal] load metadata for docker.io/library/node:20-alpine #3 DONE 0.6s #4 [1/4] FROM docker.io/library/node:20-alpine #4 CACHED #5 [2/4] WORKDIR /app #5 CACHED #6 [3/4] COPY package*.json ./ #6 CACHED #7 [4/4] COPY . . #7 DONE 0.1s ``` The `CACHED` lines tell you the layer was reused. If a layer shows `DONE` instead of `CACHED`, it was rebuilt. Check what changed in the context or Dockerfile instructions. ### Use `docker buildx du` for cache usage ```bash docker buildx du ``` This shows cache usage per builder. If you see a high value, the cache is large. If it is empty or small, the cache may have been cleared recently. ### Inspect cache metadata BuildKit stores cache metadata in its internal database. You can view cache entries with: ```bash docker buildx du --verbose ``` This outputs detailed cache records including the command that created the layer and the cache key. This is invaluable when debugging why a layer is invalidated. ### Diagnose common cache misses 1. **Dockerfile instruction change**: Any change in the instruction text alters the cache key for that layer and all subsequent layers. 2. **Context file change**: A change in any file in the context (not excluded by `.dockerignore`) that is copied into the image will invalidate the `COPY` layer and all subsequent layers. 3. **Build arguments change**: Using `ARG` and `--build-arg` with changing values invalidates layers that use that argument. 4. **Base image change**: If the base image is updated (even with the same tag), all layers based on it may be invalidated. Pinning the base image digest prevents unexpected invalidation. 5. **BuildKit cache export/import issues**: If you are using remote caches (e.g., registry cache), network or authentication issues can cause cache misses. Always run `docker build --progress=plain` to see which layer is rebuilt and then trace back to its dependency. ## Failure Modes and Recovery Even with careful setup, cache failures happen. This section outlines common failure modes and recovery steps. ### Failure: Cache is never hit Symptom: Every build rebuilds all layers. Possible causes: - BuildKit not enabled (using legacy builder with no cache support for some features). - `.dockerignore` missing, causing context changes to invalidate `COPY` layers. - Using `ADD` with a URL or archive that changes. - Running builds as different users or with different build args. Recovery: - Enable BuildKit. - Add a robust `.dockerignore`. - Use `COPY` instead of `ADD` when possible. - Standardize build environment: same user, same build command, same base image digest. ### Failure: Cache is stale Symptom: Builds are fast but the resulting image does not contain the latest code or dependencies. Possible causes: - Cache is being reused incorrectly due to a bug or misconfiguration. - Remote cache is out of sync. - Using mutable tags for base images or dependencies. Recovery: - Force a rebuild with `--no-cache` to get a fresh image. - Pin base images by digest. - Pin dependency versions in your package manifest. - For remote caches, ensure they are updated properly. ### Failure: Disk full due to cache Symptom: Docker daemon errors about no space left on device, builds fail unpredictably. Recovery: - Clean cache with `docker builder prune` (with `--filter` to keep recent cache). - Set up a cron job to prune old cache regularly. - Use cache mounts and remote caches to reduce local storage needs. - Move Docker data directory to a larger disk if necessary. ### Failure: Security concerns with cache Symptom: Sensitive data (e.g., secrets) is found in cache layers. Recovery: - Use BuildKit secrets (`--secret`) to avoid baking secrets into layers. - Avoid copying `.env` files or keys into the image. - Periodically scan images for secrets. - If a leak occurs, rotate the secrets immediately and purge the cache. ## Operations Checklist Use this checklist as a runbook for managing Docker Build Cache in production. Each item includes a concrete command or check. ### 1. Verify environment - [ ] Docker version: `docker version` (note client and server versions). - [ ] BuildKit enabled: `docker buildx ls` shows BuildKit driver. - [ ] Storage driver: `docker info --format '{{.Driver}}'` is `overlay2` or similar high-performance driver. - [ ] Disk space: `docker system df` shows sufficient free space on the build host. ### 2. Inspect build context and Dockerfile - [ ] `.dockerignore` is present and excludes unnecessary files. - [ ] Dockerfile uses multi-stage builds to minimize final image size. - [ ] Cache mounts are used for package managers and build dependencies. - [ ] Base images are pinned by digest for reproducibility. - [ ] Changing `ARG` values are not used in early layers unless necessary. ### 3. Test cache behavior - [ ] Run `docker build --progress=plain` and review cache status for each layer. - [ ] Identify any layer that is rebuilt unexpectedly and determine the cause. - [ ] Run two consecutive builds with no changes; the second should have all layers `CACHED`. - [ ] Test cache invalidation: change a file and confirm only the affected layer and subsequent layers rebuild. ### 4. Monitor and maintain - [ ] Set up monitoring for disk space on Docker host. - [ ] Schedule regular `docker builder prune` with appropriate filters (keep last 7 days cache). - [ ] Consider using a remote cache (e.g., `--cache-from` with registry) for CI/CD pipelines. - [ ] Document any custom cache configuration in the project repository. ### 5. Recovery procedures - [ ] In case of suspected cache corruption, purge with `docker builder prune -a` and rebuild from scratch. - [ ] If builds are slow due to cache misses, check for context changes and `.dockerignore`. - [ ] If security incident involving cache, rotate secrets and purge affected cache. ### 6. Ownership and review - **Owner**: DevOps Engineer (or Platform Engineer) – name a single person responsible for the Docker build infrastructure. - **Review frequency**: Monthly review of cache performance and configuration. The owner should check cache hit rates, disk usage, and any recent incidents. For high-churn projects, weekly review may be necessary. ## Common Pitfalls and How to Avoid Them 1. **Not using `.dockerignore`** – The context becomes huge, and any minor file change invalidates entire layers. **Why it happens**: Developers often start without a `.dockerignore` and forget to add it. **Avoid**: Add a `.dockerignore` from the start with common excludes. **Recover**: Add the `.dockerignore` and rebuild; cache performance improves. 2. **Using `latest` tag for base images** – The base image may change, invalidating all layers. **Why**: Convenience; developers don't want to update digests. **Avoid**: Pin digests in `FROM` lines and use Dependabot to update. **Recover**: If a `latest` base image changes and breaks your build, identify the new digest and pin it. 3. **Not enabling BuildKit** – Missing out on advanced cache features like cache mounts and inline caching. **Why**: Not knowing about BuildKit or using an old Docker version. **Avoid**: Upgrade Docker and enable BuildKit by default. **Recover**: Enable BuildKit and adapt Dockerfile to use cache mounts. 4. **Using `ADD` instead of `COPY`** – `ADD` can fetch remote URLs or extract tarballs, causing surprising cache invalidation. **Why**: Habit or misunderstanding. **Avoid**: Use `COPY` for local files; use `RUN curl` for remote downloads separately. **Recover**: Replace `ADD` with `COPY` to stabilize cache. 5. **Building with no cache in CI** – CI pipelines often run `docker build` without cache, wasting time and resources. **Why**: Lack of cache persistence between CI runs. **Avoid**: Use remote cache (e.g., `--cache-from` with registry) or a persistent buildkit daemon (e.g., `docker buildx` with remote driver). **Recover**: Implement a cache strategy in CI to speed up builds. 6. **Ignoring disk management** – Build cache grows over time and fills the disk, causing failures. **Why**: No monitoring or maintenance. **Avoid**: Set up alerts for disk usage and schedule `docker builder prune`. **Recover**: Prune cache and add a cron job. 7. **Baking secrets into images** – Secrets like API keys end up in cache layers, becoming a security risk. **Why**: Copying `.env` files or passing secrets as build args. **Avoid**: Use `--secret` with BuildKit or runtime secrets. **Recover**: Rotate exposed secrets, purge cache, and fix the Dockerfile. ## Conclusion Docker Build Cache in production is not a set-and-forget feature. It requires version awareness, careful configuration, monitoring, and recovery planning. The checklist and examples in this article provide a practical framework to ensure your builds are fast, consistent, and secure. Start with one low-risk verification: run `docker build --progress=plain` on a representative project and examine which layers are cached. Record the current state, compare with expected, and then decide if changes are needed. Remember that every change should be small, observable, and reversible where possible. A reliable operational workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By following this checklist, you can master Docker Build Cache in your production environment.