## Intro A local Docker Build Cache lab lets you reproduce build behavior safely before changing production pipelines. The goal is to move from an observed problem to a verified result using a controlled environment where you can inspect caching, test changes, and learn recovery steps without affecting shared infrastructure. This guide walks through a complete workflow: checking the installed Docker and BuildKit versions, understanding your topology, verifying prerequisites, observing current cache state, making small scoped changes, and confirming the outcome. It is written for developers, DevOps consultants, and technical startup teams who need practical, hands-on examples. Throughout, we keep operational safety in mind: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before touching any build configuration, establish exactly what you are working with. Run the following read-only commands and record the output with timestamps. # Docker and BuildKit versions docker version docker buildx version # Docker daemon information docker info --format '{{.ServerVersion}}' docker info --format '{{json .Driver}}' # Current builder and buildkit support docker buildx ls Expected output for docker version includes a client and server section. Note the API version and OS/Arch. For BuildKit, the default builder should be listed with a driver like docker or docker-container . Example output (Docker 24.0.7): Client: Docker Engine - Community Version: 24.0.7 API version: 1.43 Go version: go1.20.10 Git commit: afdd53b Built: Thu Oct 26 09:08:14 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.7 API version: 1.43 (minimum version 1.12) Go version: go1.20.10 Git commit: 311b9ff Built: Thu Oct 26 09:07:50 2023 OS/Arch: linux/amd64 Experimental: false Key point: Docker 23.0+ enables BuildKit by default for the docker build command. If you are on an older version, you may need to set DOCKER_BUILDKIT=1 or use docker buildx build explicitly. ### Check Build Cache Storage Cache data can live in different places depending on the builder. For the default docker driver using BuildKit, cache is stored within the Docker data directory, typically /var/lib/docker . For the docker-container driver, a dedicated volume stores cache and build state. # List current builder and inspect its configuration docker buildx inspect --bootstrap # For docker-container driver, check the underlying container docker ps --filter "name=buildx_buildkit" Example output from docker buildx inspect --bootstrap : Name: default Driver: docker Nodes: Name: default Endpoint: default Status: running Buildkit: v0.12.4 Platforms: linux/amd64, linux/arm64, linux/386 If you need to isolate cache for lab testing, create a dedicated builder: # Create a new builder using docker-container driver docker buildx create --name lab-builder --driver docker-container --use # Verify the builder is active docker buildx ls Now docker buildx inspect --bootstrap lab-builder will show the cache volume name (usually buildx_buildkit_lab-builder0_cache ) and other details. ### Data Storage for Build Artifacts When testing applications that produce or consume files, decide where the data should live before starting containers. A named volume is managed by Docker and is easier to reuse across container rebuilds. A bind mount maps a host directory directly and is useful for local development, but it can introduce permission and portability issues. Example: run a build that writes to a named volume # Create a named volume docker volume create build-artifacts # Run a container with the volume mounted docker run --rm -v build-artifacts:/app/data alpine sh -c 'echo hello > /app/data/greeting.txt' # Verify data persists after container exit docker run --rm -v build-artifacts:/app/data alpine cat /app/data/greeting.txt # Expected: hello Example: use a bind mount for development mkdir -p ./local-data docker run --rm -v "$(pwd)/local-data:/app/data" alpine sh -c 'echo hello > /app/data/greeting.txt' cat ./local-data/greeting.txt # Expected: hello Restart test for data persistence: - Run a service container with a volume. - Write a marker file inside the container (or have the app do it). - Stop and remove the container. - Recreate the container with the same volume. - Check that the marker file still exists. If the file is missing, the service was likely writing to the container's writable layer instead of the volume. ## Safe Configuration Path The safe path is: observe current state, back up existing configuration if any, make one scoped change, verify, and have a rollback plan. For Docker Build Cache, the most common configuration changes involve: - Choosing the BuildKit driver and cache backend - Setting cache mount options in Dockerfiles - Configuring remote cache (registry, S3, etc.) - Tuning garbage collection and retention Start with a simple experiment: enable BuildKit and compare cache reuse. ### Enable BuildKit for Docker Build If you are on Docker <23.0 or want to be explicit, set the environment variable: # Enable BuildKit for the current shell export DOCKER_BUILDKIT=1 # Build an image with buildkit docker build -t myapp:test . With Docker 23.0+, BuildKit is the default builder and no environment variable is needed. You can also use docker buildx build for more control: docker buildx build -t myapp:test . ### Example Dockerfile with Cache Mounts Cache mounts allow build steps to reuse package manager caches across builds, speeding up repeat builds. The following Dockerfile uses a cache mount for apt packages. # syntax=docker/dockerfile:1.4 FROM ubuntu:22.04 RUN --mount=type=cache,target=/var/cache/apt \ apt-get update && apt-get install -y curl Build the image twice and observe the second build is faster and reuses the cache layer. # First build docker build -t cache-test . # Second build (cache should be reused) docker build -t cache-test . Look for lines like => CACHED in the build output. For the apt-get step, you should see something similar to: => [2/2] RUN --mount=type=cache,target=/var/cache/apt apt-get update && apt-get install -y curl 0.5s On subsequent builds, the output may say => CACHED for that step. ### Configure Garbage Collection for BuildKit BuildKit has a garbage collector that removes unused cache data. You can control its behavior in the builder configuration. For a docker-container driver builder, you can pass configuration when creating the builder. # Create a builder with custom GC policy docker buildx create --name lab-gc --driver docker-container \ --driver-opt env.BUILDKIT_STEP_EXECUTOR_DISABLE_CACHE=0 \ --config ./buildkitd.toml First, create a buildkitd.toml file: [worker.oci] gc = true gckeepstorage = 5000 # keep cache up to 5GB [[worker.oci.gcpolicy]] keepBytes = 512000000 # 512MB keepDuration = 172800 # 48 hours filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout" ] [[worker.oci.gcpolicy]] all = true keepBytes = 1024000000 # 1GB Then inspect the builder to confirm the config was applied: docker buildx inspect --bootstrap lab-gc Caution: Changing GC settings can affect disk usage and cache retention. In a lab, start with conservative values and observe over a few builds before applying to production. ## Verification and Diagnostics Once you have a build running or completed, verify cache behavior with concrete commands. The goal is to answer: Did the cache actually get used? Which layers were cached? Where does the builder store cache data? ### Inspect Build Output Run a build with progress output set to plain to see detailed layer information: docker buildx build --progress=plain -t myapp:test . Look for lines like #34 CACHED or #34 DONE 1.2s . CACHED indicates layer reuse. To force a no-cache build for comparison: docker buildx build --no-cache --progress=plain -t myapp:test . Compare build times with the time command: time docker buildx build -t myapp:test . Example outputs: - Cached build: real 0m2.345s - No-cache build: real 1m12.890s ### Use docker buildx du to See Cache Size Buildx provides a command to show disk usage of the cache: docker buildx du Sample output: ID RECLAIMABLE SIZE LAST ACCESSED 8ldp1z5l91 true 812.5MB 3 minutes ago 3q8p0c2m4n false 1.2GB 1 hour ago You can reclaim space with docker buildx prune (see recovery section). ### Inspect a Specific Docker Layer To see which files a build layer added, use docker history : docker history myapp:test Example output: IMAGE CREATED CREATED BY SIZE abc123456789 2 minutes ago RUN /bin/sh -c apt-get update && apt-get in... 45MB Or use docker image inspect to get layer IDs and then docker save to extract if needed. For more advanced debugging, run a container from the image and inspect the filesystem. ### Check Container Logs for BuildKit Daemon If you use a docker-container builder, you can view its logs: docker logs buildx_buildkit_lab-builder0 This may output warnings or errors about cache operations. ### Test Data Persistence with Containers As mentioned earlier, if your build produces data that must persist, test with a simple scenario. Create a container that writes a file, exit, then run another container with the same volume to read the file. # Write a file docker run --rm -v testvol:/data alpine sh -c 'echo important > /data/file.txt' # Read it back docker run --rm -v testvol:/data alpine cat /data/file.txt # Expected: important If the read returns an error, the volume was not attached or the path differs. ## Failure Modes and Recovery Despite careful setup, things go wrong. Here are common failure modes for Docker Build Cache and how to recover. ### Failure: Cache Not Being Reused Symptom: Every build takes as long as a no-cache build. You see no CACHED lines in build output. Likely causes: - Dockerfile instructions are often invalidating cache (e.g., COPY . . after a frequently changing file, or RUN apt-get update with no version pinning). - Build context is huge and includes unnecessary files, causing layer changes. - BuildKit is not enabled or using a different builder. - Cache was pruned or expired due to GC. Diagnosis: - Run docker buildx ls to confirm active builder. - Check docker buildx du to see if cache exists. - Inspect build output with --progress=plain to see which step breaks the cache chain. Recovery: - Reorder Dockerfile to place frequently changing steps later. - Use a .dockerignore to exclude unnecessary files from context. - Pin base image tags and dependencies (e.g., FROM ubuntu:22.04 instead of ubuntu:latest ). - For package managers, use cache mounts ( --mount=type=cache ) as shown earlier. ### Failure: Disk Space Exhaustion Symptom: Build fails with no space left on device or Docker daemon reports disk pressure. Diagnosis: - Run df -h on host. - Run docker system df to see disk usage by Docker objects. Example output: TYPE TOTAL ACTIVE SIZE RECLAIMABLE Images 12 2 5.32GB 2.11GB (39%) Containers 4 1 78MB 12MB (15%) Local Volumes 8 2 1.8GB 300MB (16%) Build Cache 25 0 9.2GB 9.2GB (100%) Recovery: - docker buildx prune to remove build cache. Use filters to remove only old or unused cache: docker buildx prune --filter until=24h . - Remove unused images and volumes: docker system prune -a (careful: removes all unused data). - If using a dedicated builder, consider reducing GC keep storage or increasing host disk size. ### Failure: Corrupted or Inconsistent Cache Symptom: Build sometimes uses cached layer but produces wrong output, or build fails with cache-related errors. Cause: BuildKit bug, interrupted build leaving partial cache, or manual edits to cache files. Recovery: - Clear the entire build cache: docker buildx prune --force . - Recreate the builder if using docker-container driver: docker buildx rm lab-builder && docker buildx create --name lab-builder --driver docker-container . - Rebuild from scratch without cache: docker buildx build --no-cache -t myapp:test . . ### Failure: Secrets Exposed in Build Cache Symptom: Secrets passed via build args or copied into image layers end up in cache and can be extracted. Prevention: - Never use build args for secrets. Use BuildKit secret mounts: RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret . - Use multi-stage builds to avoid copying sensitive files into final layers. Recovery: - Purge build cache and rebuild. Rotate any exposed secrets. ## Operations Checklist Run through this checklist in your lab before and after any build cache experiment. Each item includes the owner and review frequency.
CheckActionExpected SignalOwnerReview Frequency
Version compatibilityRun docker version and docker buildx versionDocker 23.0+ and BuildKit v0.12+Priya Shah, Engineering LeadMonthly
Builder activedocker buildx lsdefault builder with docker driver and running statusPriya ShahWeekly
Cache storagedocker buildx duTotal reclaimable size < 80% of diskAlex Chen, DevOpsWeekly
Dockerfile cache efficiencyBuild twice and compare times; check for CACHED linesSecond build at least 50% fasterDeveloper on dutyPer project
Data persistenceRestart test with volumeFile remains after container recreateQA EngineerPer release
Secrets handlingReview Dockerfile for build args or copied secretsNo secret in build args; secret mounts usedSecurity OfficerQuarterly
Recovery readinessDocument and test docker buildx prune commandCache cleared and builds still succeedAlex Chen, DevOpsMonthly
Perform these checks in a dedicated lab environment before applying changes to CI/CD. Record results and compare trends over time. ## Common Pitfalls and How to Avoid Them - Not using BuildKit - Why it happens: Older Docker versions or habit. - Avoid: Upgrade to Docker 23.0+ or explicitly set DOCKER_BUILDKIT=1 . Use docker buildx commands. - Frequently changing base image tag - Why: Using latest or floating tags invalidates cache frequently. - Avoid: Pin exact versions, e.g., FROM node:18.17.1-alpine . - Copying entire project directory early - Why: COPY . . before installing dependencies causes cache of dependency install to be invalidated on any source change. - Avoid: First copy only dependency manifest files (package.json, requirements.txt) and install dependencies, then copy the rest. - Example Dockerfile: FROM node:18-alpine WORKDIR /app COPY package*.json ./ RUN npm install COPY . . - Forgetting .dockerignore - Why: Build context includes node_modules, .git, logs, etc., making context huge and causing layer changes. - Avoid: Create a .dockerignore with at minimum node_modules , .git , *.log . - Not using cache mounts for package managers - Why: Each build re-downloads packages. - Avoid: Use --mount=type=cache for apt, pip, npm, etc. - Misunderstanding docker buildx prune - Why: Running prune without filters removes all cache, slowing next builds. - Avoid: Use --filter until=24h or --filter label=... to remove only old cache. - Storing secrets in build layers - Why: Trying to pass secrets via build args or copying .env files. - Avoid: Use BuildKit secret mounts or runtime environment variables (in orchestration, not image). - Assuming cache is shared across builders - Why: Each builder has its own cache storage; using different builder names for different projects prevents reuse. - Avoid: Use a consistent builder for related projects or configure remote cache. ## Conclusion A local Docker Build Cache lab empowers you to test caching behavior, diagnose failures, and practice recovery without impacting production. The key is to work methodically: inventory your environment, make one change at a time, verify with concrete commands, and document both success and failure signals. Start with a low-risk verification: run docker buildx ls to see your builder, build a simple image twice, and observe cache reuse with --progress=plain . Record the build times and cache size before and after. Remember that cache management is an operational responsibility. Regular checks for disk usage, secret handling, and cache efficiency keep builds fast and secure. When in doubt, never assume the cache works—prove it with commands and keep a rollback plan ready. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.