## Intro

Layer caching can turn a 12-minute Docker build into a 90-second rebuild, but only when the cache has enough room, the right storage driver, and a BuildKit configuration that survives CI runners being recycled. Capacity planning for Docker layer caching means answering a few concrete questions before the first cache miss forces an emergency cleanup:

- How much disk will the cache realistically consume for this project's image history?
- Which cache backend should be used: a local directory, a Docker-managed volume, or a remote registry cache?
- What signal tells you the cache is working, and which failure modes silently produce full rebuilds?
- What is the smallest safe change when the cache outgrows its current allocation?

This guide is written for developers, DevOps consultants, and technical startup teams who operate their own build infrastructure. It connects cache sizing, scaling, resource limits, and monitoring to specific commands, expected output, and recovery decisions. The underlying principle is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Throughout the article, examples assume Docker Engine 24.0 or later with BuildKit enabled by default. On Docker Desktop, BuildKit is on by default. On Linux, you can confirm with `docker buildx version`; if you see `buildx` output rather than an error, BuildKit is available. The commands are copy-pasteable, but replace paths, image names, and thresholds with values from your own environment.

## Version and Environment Inventory

Before you tune anything, know exactly what is running and where the cache lives. Inventory the Docker version, buildx plugins, disk usage, and builder configuration. This step is read-only and should produce timestamps and baseline numbers that you can compare against after any change.

Run the following commands and record the output:

```bash
# Docker and buildx versions
docker version --format '{{.Client.Version}} {{.Server.Version}}'
docker buildx version

# Disk usage summary for Docker
docker system df

# Builder details, including cache location and driver
docker buildx ls
```

A typical `docker system df` output looks like this:

```
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          12        3         4.213GB   2.111GB (50%)
Containers      8         2         1.02MB    0B (0%)
Local Volumes   5         1         1.89GB    1.34GB (71%)
Build Cache     25        0         15.75GB   15.75GB (100%)
```

The `Build Cache` line is the one to watch for layer caching capacity planning. If that number is near your disk limit or if it is growing without bound, you have a sizing problem. Also note the `docker buildx ls` output: the default builder may have a `docker-container` driver with a dedicated cache volume, or it may fall back to the docker driver that stores cache on the host's `var/lib/docker`. A common surprise is that a CI runner created a new builder with an empty cache on every pipeline run, so the layer cache never has a chance to help.

### Where the cache actually lives

For a standard docker driver, layer cache is stored under `/var/lib/docker/overlay2` (or the equivalent path for your storage driver). For a `docker-container` builder, the cache may reside inside the builder container's own filesystem, often in a volume such as `buildx_buildkit_builder0_state`. To see the exact mount, inspect the builder container:

```bash
docker buildx inspect --bootstrap
```

Output includes the builder's name, driver, and node platform. If you see `Driver: docker-container` and `Platforms: linux/amd64, linux/arm64`, the builder is remote and its cache is not directly on the host filesystem unless a cache mount is configured.

For a simple sanity check on the host, look at the overlay directories' disk usage:

```bash
sudo du -sh /var/lib/docker/overlay2/* 2>/dev/null | sort -h | tail -5
```

This shows which layers consume the most space. Extremely large layers often come from accidental inclusion of build artifacts, vendored dependencies, or caches from earlier steps that were not cleaned up.

### Production-like local test for cache persistence

A small but critical test is whether the cache survives a builder restart. Create a trivial Dockerfile and build twice with a timestamp to verify the second build is fast:

```bash
# Dockerfile
FROM alpine:3.18
RUN echo "build $(date +%s)" > /tmp/build-time

# First build
docker build -t cache-test:first .

# Second build immediately after
docker build -t cache-test:second .
```

In the second build, BuildKit should show `[2/2] RUN echo "build ...` as `CACHED`. If you see the `RUN` step execute again, the cache is not persisting between builds. The fix depends on the driver: for docker-container builders, ensure the builder is not recreated every time (use `docker buildx create --name mybuilder --use` once, and reuse that builder). For CI runners, ensure the builder's cache volume is not wiped between jobs.

## Sizing the Cache with Practical Numbers

Capacity planning starts with measuring how much cache one typical build produces and how often you build. For a mid-sized Node.js service, the layer cache after ten builds might look like this:

| Build component | Typical size | Rebuilt every time? | Cache utility |
|---|---|---|---|
| Base image (node:20-alpine) | 180 MB | No, reused | High |
| npm dependencies layer | 350 MB | Only when package.json changes | Very high |
| Source code copy | 50 MB | Yes, unless tooling uses mount cache | Low as layer, high as mount |
| Test artifacts | 200 MB | Yes, but should not be cached | None, creates churn |
| Build output (dist) | 120 MB | Yes, but can be cached with max mode | Medium |

If a team runs 20 builds per day on a single builder, a naive cache that keeps everything would grow roughly 20 x (350 MB + 50 MB + 200 MB + 120 MB) = 14.4 GB per day. Most of that is useless because the source code and test artifacts change every build while the dependency layer changes only occasionally. A well-configured cache prunes aggressively or uses a remote registry cache to store only the meaningful layers.

A realistic cache budget for that team might be 5 to 10 GB on disk, with a retention policy based on age rather than size. Use `docker buildx prune` with a filter to keep only cache entries accessed within the last 7 days:

```bash
docker buildx prune --filter "until=168h" --force
```

This is a safe starting point because it removes only cache data, not images or containers. Monitor the reclaimed space with `docker system df` before and after. If builds suddenly slow down because common dependencies are pruned, increase the retention window to 14 days and adjust the storage allocation accordingly.

### Compute the cache budget from image history

For a precise plan, inspect the actual layer sizes for a representative image:

```bash
docker history --no-trunc your-image:latest
```

This lists each layer with its size and the command that created it. Identify the layers that change least frequently. Those are the layers worth caching. For a Python application, the dependency installation layer (`RUN pip install -r requirements.txt`) is the prime candidate. If that layer is 800 MB, you want enough cache space to keep at least one copy per Python version in active use. Reserve 2 copies per active branch or PR pipeline, plus a buffer for base image updates. The formula is:

```
Cache GB = (sum of high-value layer sizes in GB) x (number of concurrent branches) x (safety factor of 1.5)
```

As a concrete example, a Python monorepo with a 800 MB dependencies layer, 500 MB base image, and 5 active feature branches would need about (0.8 + 0.5) x 5 x 1.5 = 9.75 GB of cache space just for those layers. Adding in ephemeral build cache from unmount steps might push it to 15 GB. Compare that against the available disk on the builder host.

## Safe Configuration Path

Changing cache settings is an intervention. Follow a scoped process: record current state, backup any cache that might be irreplaceable (for remote caches, note the registry URL and credentials), make one change, and verify the outcome before moving on. For a local builder, start with builder configuration rather than Docker daemon global settings.

### Use a dedicated builder with explicit cache settings

Create a named builder with a docker-container driver and an explicit cache directory or cache-to/cache-from configuration:

```bash
docker buildx create \
  --name mybuilder \
  --driver docker-container \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10485760 \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=1048576 \
  --use

docker buildx inspect --bootstrap
```

This sets log limits that prevent unbounded growth in the builder container. The cache itself is stored inside the builder's volume. You can also mount a dedicated host directory for cache if you want easier backup:

```bash
docker buildx create \
  --name mybuilder \
  --driver docker-container \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10485760 \
  --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=1048576 \
  --config /etc/buildkit/buildkitd.toml \
  --use
```

With a custom `buildkitd.toml` you can set cache policies, garbage collection thresholds, and worker counts. An example minimal `buildkitd.toml`:

```toml
[worker.oci]
  enabled = true
  gc = true
  gckeepstorage = 10737418240  # 10 GiB
  [[worker.oci.gcpolicy]]
    keepBytes = 5368709120     # 5 GiB
    keepDuration = 604800      # 7 days
    filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout" ]
  [[worker.oci.gcpolicy]]
    all = true
    keepBytes = 1073741824     # 1 GiB
```

This configuration runs garbage collection automatically when cache exceeds 10 GiB, retaining at least 5 GiB of the specifically identified cache types for 7 days. Adjust `gckeepstorage` and `keepBytes` to match your computed budget. After creating the builder, run a test build and check `docker system df` to see if the cache is being bounded.

### Remote registry cache for ephemeral CI environments

In CI pipelines where runners are destroyed after each job, the local build cache disappears. The solution is to push and pull cache images to a container registry. Use `cache-to` and `cache-from` options in `docker buildx build`:

```yaml
# GitHub Actions example step
- name: Build and push with cache
  uses: docker/build-push-action@v5
  with:
    context: .
    push: true
    tags: yourrepo/app:latest
    cache-from: type=registry,ref=yourrepo/app:buildcache
    cache-to: type=registry,ref=yourrepo/app:buildcache,mode=max
```

The `mode=max` exports cache for all layers, including intermediate stages, while the default `min` mode exports cache only for the final image layers. Max mode uses more storage in the registry but dramatically improves cache hits for multi-stage builds. Registry cache capacity planning is different: you are paying for object storage per GB-month. Monitor the size of the cache image with `docker manifest inspect` or by checking the registry's UI. A typical cache image might be 1 to 2 GB; if it grows beyond 5 GB, prune or set a retention rule in the registry.

### Docker daemon GC and storage driver

On a standalone Docker host, the `docker system events` and daemon logs reveal when the overlay2 storage is filling up. Enable daemon garbage collection by setting `--storage-opt overlay2.size=` is not a valid option; instead, rely on periodic `docker system prune` or BuildKit's own GC. For the classic image builder, the layer cache is managed by the daemon and can be pruned with:

```bash
docker image prune --filter "until=168h" --force
```

For BuildKit builders, always use `docker buildx prune` rather than `docker image prune`. They operate on different cache stores.

## Verification and Diagnostics

After configuring cache settings, verify that cache hits actually occur in your builds. The most direct signal is the BuildKit output. A cached step shows `CACHED` next to the command:

```
 => [stage-1 3/5] COPY --from=builder /app/dist ./dist      0.0s
 => [stage-1 4/5] RUN npm install --production              0.0s
 => [stage-1 5/5] CMD ["node", "server.js"]                0.0s
 => exporting to image                                       0.4s
 => => exporting layers                                      0.0s
 => => writing image sha256:...                              0.0s
```

If you see `RUN npm install` executing rather than `CACHED`, the cache was missed for that step. To diagnose why, run the build with verbose BuildKit logging:

```bash
docker buildx build --progress=plain --no-cache-filter=install .
```

The `--no-cache-filter` option forces a rebuild of a specific stage or step, which is useful for testing. To see cache metadata for an image, use:

```bash
docker buildx imagetools inspect yourrepo/app:buildcache
```

This shows the manifest list and can indicate whether the cache image was pushed correctly. For local cache, you can inspect the cache records using `docker buildx du`:

```bash
docker buildx du
```

Example output:

```
ID                                              RECLAIMABLE  SIZE       LAST ACCESSED
3nq7k9nm0v3w74l0a1e2z7g5e                      true         180MB      2 hours ago
...
Total:                                          15.75GB
```

Use `docker buildx du --verbose` to see which cache entries belong to which build. This is helpful when you need to determine whether a specific dependency layer is cached.

### Monitoring cache hit rate

For ongoing verification, instrument builds with BuildKit's tracing support. Export build traces to a collector such as Jaeger or OpenTelemetry. Add `--build-arg BUILDKIT_INLINE_CACHE=1` to include cache metadata in the final image, which allows subsequent builds to use the image itself as a cache source. For CI metrics, parse the build log for the string `CACHED` and compute `cached_steps / total_steps`. A healthy cache hit rate for dependency installation steps should be above 80% for repeated builds with unchanged dependency files.

Set up a simple cron job or CI check that builds a sentinel Dockerfile and reports the cache status:

```bash
#!/bin/bash
# cache-health-check.sh
IMAGE=cache-sentinel
OUTPUT=$(docker build -t $IMAGE . 2>&1)
if echo "$OUTPUT" | grep -q "CACHED"; then
  echo "Cache hit detected"
else
  echo "Cache miss or failure; output:"
  echo "$OUTPUT"
  exit 1
fi
```

Run this every hour or after each builder restart. If it fails, investigate before trusting the cache in production builds.

## Failure Modes and Recovery

Layer caching fails silently more often than you think. The most common failure mode is the cache being empty or incomplete because the builder was recreated, the cache directory was wiped, or the CI runner had a fresh checkout. Symptoms include sudden build time increases and log output that never shows `CACHED` for previously cached steps. Diagnosis: run `docker buildx ls` to see if the builder is still present and `docker buildx du` to check cache size. If the cache size is 0 or very small, the cache was lost. Recovery is to rebuild the cache by running a full build with `cache-to` enabled, and then update the CI configuration to persist the builder between runs or use a remote cache.

Another failure mode is cache poisoning or incorrect layer reuse when a dependency version changes without changing the Dockerfile. For example, if you copy `package.json` and `package-lock.json` separately but order them incorrectly, the cache may reuse an old `npm install` layer even though `package-lock.json` changed. Symptoms: builds succeed but with outdated dependencies. Fix the Dockerfile to copy the lockfile before running install, and use `--no-cache-filter=install` to force a clean install after the change. Prevent by ensuring the cache key for the install step includes the content hash of both files.

The opposite problem is cache thrash: the cache grows unbounded and fills the disk, causing build failures with `no space left on device`. Symptoms include build processes killed, `docker buildx build` errors about cannot allocate memory or write layer, and daemon logs showing disk full. Recovery steps:

1. Free space immediately: `docker system prune -af --volumes` (careful: this removes all unused images, containers, volumes, and cache).
2. Restrict to cache only: `docker buildx prune --all --force`.
3. Increase the builder's GC aggressiveness by editing `buildkitd.toml` as shown earlier, lowering `gckeepstorage` and `keepBytes`.
4. Add disk usage alerts on the host, e.g., `df -h /var/lib/docker` threshold at 80%.

For remote registry cache, failure mode is registry capacity limits or egress costs. If the cache image grows too large, pushes may be rejected. Monitor the registry storage and set a retention policy. Use `mode=min` instead of `mode=max` to reduce cache image size if storage costs are high, but accept lower cache hit rates.

Finally, secrets leaking into cache layers is a security failure. If you accidentally built an image with a secret in a layer that was cached and pushed as a cache image, you must invalidate and purge that cache entry. Use `docker buildx prune --filter "ref==yourrepo/app:buildcache"` after deleting the registry image, and rotate the exposed secret.

## Common Pitfalls and How to Avoid Them

Pitfall 1: Ignoring the builder context. Many teams enable BuildKit but continue using the legacy docker driver for local builds, which does not support all cache features. Check `docker buildx ls` and ensure the builder marked with `*` uses `docker-container` or `remote` driver for advanced cache-to/cache-from. If it says `default * docker`, you are not using BuildKit's full capabilities.

Pitfall 2: Forgetting to order Dockerfile steps from least to most frequently changing. If you `COPY . .` early in the Dockerfile, every code change invalidates all subsequent layers, including dependency installs. Place dependency manifest copies and installs before copying the full source tree.

Pitfall 3: Not using cache mounts for package manager caches. `RUN --mount=type=cache,target=/root/.npm npm install` lets BuildKit reuse the npm cache across builds even when dependencies change. Without it, the package manager downloads everything on every dependency change.

Pitfall 4: Over-pruning with broad `docker system prune`. That command removes all unused images and containers as well as cache. If you prune images that are used as cache sources via inline cache, you lose the cache. Use `docker buildx prune` with filters instead.

Pitfall 5: Assuming cache size equals disk usage on the host. With a docker-container builder, the cache lives inside a volume that may not be counted by `df /var/lib/docker`. Always use `docker buildx du` and `docker buildx ls` to see the actual cache size and location.

Recovery from any of these pitfalls: revert the last configuration change, run a clean build with explicit cache-to, and verify with the cache-health-check script.

## Operations Checklist

Capacity planning is an ongoing practice. Review cache performance monthly, or more frequently if build times fluctuate. Assign an accountable owner for cache tuning, typically the CI/CD platform engineer or a senior developer in a startup. The owner should:

- Weekly: check `docker buildx du` and `docker system df` on each builder; ensure cache size stays within 70% of allocated disk.
- Monthly: review cache hit rate from CI logs; if below 80% for dependency layers, investigate causes and adjust cache retention or Dockerfile structure.
- Quarterly: reassess the cache budget using the formula in the sizing section; update retention policies and builder configurations.
- On every image or builder change: run the cache-health-check script and compare build times to baseline.

Use this concrete checklist before rolling out cache changes to a production-like environment:

```markdown
- [ ] Current builder name and driver recorded: `docker buildx ls`
- [ ] Cache size before change: `docker buildx du` total
- [ ] Backup or note of remote cache reference if any: e.g., yourrepo/app:buildcache
- [ ] Change made: (e.g., added cache-to max mode, adjusted gckeepstorage)
- [ ] Test build completed: `docker build .` shows expected CACHED steps
- [ ] Cache size after change: `docker buildx du` total
- [ ] Build time recorded: time before and after
- [ ] Rollback command identified: e.g., restore previous buildkitd.toml, recreate builder
```

When a cache decision involves spending money (more disk or registry storage), the owner should document the monthly cost and the expected reduction in CI build minutes. Review that trade-off with the team lead every quarter.

## Conclusion

Docker layer caching capacity planning turns an invisible performance feature into a managed resource. The steps are concrete: inventory your builder and cache location, compute a cache budget from real layer sizes, configure a dedicated builder with explicit GC limits or a remote cache for ephemeral CI, verify cache hits with logged commands, and prepare recovery procedures for the common failure modes of cache loss, thrash, and poisoning.

Start with a single low-risk change: create a named builder with a `buildkitd.toml` that sets `gckeepstorage` to your computed budget, then run two builds of your application and compare times. Record the cache size before and after, and set up a weekly `docker buildx du` check. Extend to registry cache only after local cache is stable and observed.

A reliable caching setup makes its own performance visible. When a build slows down, you should be able to point to the cache hit rate, the cache size, and the retention policy, and know exactly which knob to turn.