Docker BuildKit Troubleshooting with Practical Examples

Intro

Docker BuildKit is the modern builder backend for Docker images, offering faster builds, better caching, and advanced features like secrets and parallel stage execution. But when a build fails with a cryptic error, the power of BuildKit can make troubleshooting feel more complex than the legacy builder. This guide provides practical, command-driven steps to diagnose and fix common BuildKit problems, from missing build cache to configuration errors.

We will focus on operational safety: observe the current state before making changes, limit the blast radius of any modification, use placeholders for secrets, verify each fix, and document recovery steps. The article is aimed at developers, DevOps engineers, and technical startup teams who use Docker in production or CI pipelines.

You will learn how to inspect BuildKit version and environment, check logs, interpret cache behavior, adjust configuration safely, and recover from frequent failure modes. Each section includes concrete commands, expected outputs, and recovery decisions.

Version and Environment Inventory

Before diving into specific errors, establish a baseline. Knowing the exact BuildKit version and how Docker is configured will help you avoid chasing issues that are already fixed in a later release or caused by environment differences.

Check Docker and BuildKit Version

Run docker version to see the client and server versions. BuildKit is built into the Docker daemon, but its version may differ depending on the Docker release.

docker version --format '{{.Server.Version}}'

Example output:

24.0.5

To check the BuildKit version specifically, use docker buildx version if you have the buildx plugin (often included).

docker buildx version

Example output:

github.com/docker/buildx v0.11.2 9872040

Verify BuildKit is Enabled

BuildKit is the default builder for Docker 23.0 and later. You can confirm by running a simple build and looking for the BuildKit output format. Or check the daemon configuration:

docker info --format '{{.Builder}}'

If the output is buildkit, it is active. If it says legacy, you need to enable BuildKit.

Understand the Build Environment

BuildKit can run in different contexts: the default Docker daemon, a Docker container (e.g., docker-container driver), or a Kubernetes pod. List your builders:

docker buildx ls

Example:

NAME/NODE       DRIVER/ENDPOINT             STATUS  PLATFORMS
default *       docker
  default       default                     running linux/amd64, linux/arm64

Knowing the driver matters because some features like multi-platform builds only work with the docker-container driver.

Inspect Current Build Resources

Check for existing build containers, networks, or volumes that might conflict.

docker ps -a --filter "name=buildx_buildkit"

If you see a stopped BuildKit container, it may need to be restarted or cleaned up.

Practical Production-like Test

Before debugging a complex pipeline, reproduce the issue in a minimal local project. Create a simple Dockerfile:

FROM alpine:3.18
RUN echo "hello" > /tmp/hello.txt

Build it with BuildKit:

docker build -t test-buildkit .

If this succeeds, but your real build fails, the problem is likely in your Dockerfile instructions or build context, not BuildKit itself.

To confirm that files are persisted correctly, run a container from the image and check the filesystem:

docker run --rm test-buildkit cat /tmp/hello.txt

Expected output:

hello

Then test restart behavior by running the container with a named volume and writing data:

docker run -d --name data-test -v app_data:/data alpine sh -c "echo persistent > /data/file.txt && sleep 300"
docker exec data-test cat /data/file.txt

Output:

persistent

Now stop and remove the container, then run a new one with the same volume:

docker rm -f data-test
docker run --rm -v app_data:/data alpine cat /data/file.txt

If the output is still persistent, your volume setup is correct. If not, data was written to the container layer instead of the volume.

Safe Configuration Path

BuildKit configuration can come from environment variables, daemon settings, or buildx builder options. Changing configuration incorrectly can break all builds, so follow a systematic approach: observe, back up, change one item, test, and rollback if needed.

Key Configuration Locations

1. Environment variables: DOCKER_BUILDKIT=1 to force BuildKit (for older Docker clients).
2. Daemon configuration file: /etc/docker/daemon.json (Linux) or Docker Desktop settings (Mac/Windows).
3. Buildx builder configuration: stored in ~/.docker/buildx/ or in the builder container.

Safe Change Procedure

Before making changes, capture the current configuration:

docker buildx inspect --bootstrap

Example output (truncated):

Name:   default
Driver: docker
Nodes:
Name:      default
Endpoint:  default
Status:    running
Buildkit:  v0.12.4

Save the output to a file for rollback reference.

Also back up the daemon.json if it exists:

sudo cp /etc/docker/daemon.json /etc/docker/daemon.json.backup

When you change a setting, restart Docker and verify the change took effect. For example, to enable verbose BuildKit logging, add the following to daemon.json:

{
  "debug": true,
  "features": {
    "buildkit": true
  }
}

Then restart Docker:

sudo systemctl restart docker

Check that the daemon restarted without errors:

sudo journalctl -u docker -n 50 --no-pager

Look for lines like level=info msg="Daemon has completed initialization".

Dealing with Secrets and Sensitive Data

Never hardcode secrets in Dockerfiles or build arguments. Use BuildKit secrets:

# Dockerfile
FROM alpine
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret

Pass the secret at build time:

docker build --secret id=mysecret,src=./secret.txt -t myapp .

If you accidentally expose a secret, revoke it immediately and rotate credentials. Use docker history to see if secrets were baked into an image:

docker history --no-trunc myapp

If you see the secret value, rebuild and push a new image; do not use the compromised one.

Verification and Diagnostics

When a build fails, the first step is to gather logs and understand the error. BuildKit provides detailed output, but sometimes you need to increase verbosity or inspect internal state.

Enable Detailed Build Logs

By default, docker build shows progress and errors. To see more detail, use --progress=plain:

docker build --progress=plain -t myapp .

This prints each step's output without the fancy progress bars, which is useful for reading errors.

To get even more detail, set the BuildKit log level via environment variable before running the build:

export BUILDKIT_PROGRESS=plain
docker build -t myapp .

Or run the daemon with debug logging (as described in the previous section).

Examine BuildKit Daemon Logs

If the build daemon itself crashes or behaves unexpectedly, inspect its logs. For the default Docker daemon:

sudo journalctl -u docker -f

For a buildx container driver, find the container name and view logs:

docker ps -a --filter "name=buildx_buildkit"
# note the container id, then:
docker logs <container-id> --tail 100

Diagnose Cache Misses

A common frustration is builds not using cache. To see what caused a cache miss, run with --progress=plain and look for lines like CACHED or RUN with no cache. For example:

#7 [stage-0 2/4] RUN apt-get update && apt-get install -y curl
#7 0.523 + apt-get update
#7 1.234 ...
#7 DONE 12.3s

If a step is not CACHED, something changed in the preceding layers or the command itself. Use docker buildx du to check cache usage:

docker buildx du

Example:

Total: 123.4MB

If the cache is too small, you may need to increase the cache size in the builder configuration.

Interactive Debugging

You can run a container with the same filesystem as a failed build step. Use docker buildx debug if available, or manually replicate the Dockerfile instructions in a base image.

docker run -it --rm alpine:3.18 sh

Inside the container, execute the failing command to see the error interactively.

Health Check for BuildKit

Verify BuildKit is functioning by checking the BuildKit API endpoint (if exposed). For a container driver, find the port:

docker port buildx_buildkit_default

Example:

1234/tcp -> 0.0.0.0:1234

Then use docker buildx inspect --bootstrap to ensure the builder is running.

Failure Modes and Recovery

This section covers the most frequent BuildKit failures, their symptoms, and step-by-step recovery.

Failure 1: "BuildKit is enabled but the build client is not using it"

Symptom: You see output that looks like legacy builder (e.g., Sending build context to Docker daemon) even though Docker version is 23+.

Cause: The DOCKER_BUILDKIT environment variable may be set to 0, or the client is old.

Recovery:

1. Check environment: echo $DOCKER_BUILDKIT (should be empty or 1).
2. Unset it if it is 0: unset DOCKER_BUILDKIT.
3. Or force BuildKit: DOCKER_BUILDKIT=1 docker build -t myapp ..
4. Upgrade Docker if the version is below 23.0.

Failure 2: "failed to solve: rpc error: code = Unknown desc = failed to compute cache key"

Symptom: Build fails with cache key error, often when copying files.

Cause: The build context is too large, or there is a file that changes frequently (like a log file) that invalidates cache, or there are permission issues.

Recovery:

1. Check the build context size: docker build --progress=plain -t myapp . 2>&1 | grep "transferred"
2. Use .dockerignore to exclude unnecessary files. Create a .dockerignore file with patterns like:

.git
*.log
node_modules

1. If the error persists, try clearing the build cache: docker builder prune -f.
2. Rebuild.

Failure 3: "BuildKit not supported by daemon"

Symptom: Error message: ERROR: BuildKit is enabled but the buildx component is missing or broken.

Cause: The buildx plugin is not installed or outdated.

Recovery:

1. Install or update buildx: follow Docker documentation for your OS.
2. Alternatively, use the built-in BuildKit without buildx by setting DOCKER_BUILDKIT=1 and using docker build (not docker buildx build).

Failure 4: "no space left on device" during build

Symptom: Build fails with no space left on device, even though disk has free space.

Cause: BuildKit uses a separate storage area (often in /var/lib/docker/buildkit or the builder container) which may be full due to cache growth.

Recovery:

1. Check disk usage: df -h /var/lib/docker/buildkit (or docker system df).
2. Prune BuildKit cache: docker builder prune -a -f (careful: removes all build cache).
3. If using a container driver, increase the volume size or prune the container: docker rm -f buildx_buildkit_default && docker buildx create --use.
4. Rebuild.

Failure 5: "failed to load cache key: pull access denied"

Symptom: Build fails when trying to pull a base image or cache from a registry, with access denied.

Cause: Authentication issue or missing permissions in the registry.

Recovery:

1. Log in to the registry: docker login <registry>.
2. Verify you can pull the image manually: docker pull <image>.
3. If using a buildx container driver, ensure the builder container has access to credentials. Use --push flag with docker buildx build and proper --secret for registry auth if needed.
4. Check your registry permissions.

Operations Checklist

Use this checklist before and after any BuildKit troubleshooting session to ensure you cover essential steps and minimize risk.

Owner and Review Frequency

Each checklist item has a single accountable owner to avoid ambiguity. For example, the DevOps Lead owns configuration integrity and reviews it monthly or before any change. The Developer on-call handles build failures and reviews daily. The Security Officer audits secret handling quarterly. These reviews should be documented in your incident management system.

Common Pitfalls and How to Avoid Them

Even experienced Docker users encounter BuildKit pitfalls. Below are the most frequent ones and preventive measures.

Pitfall 1: Ignoring .dockerignore

Why it happens: Developers often forget to create or update .dockerignore, causing huge build contexts and cache invalidation.

How to avoid: Always include a .dockerignore in the project root. Start with patterns for VCS, logs, and dependencies:

.git
.gitignore
*.log
node_modules
target/

Recovery: If a build is slow or cache misses constantly, check the build context size with docker build --progress=plain . | grep "transferred". Add missing patterns to .dockerignore and rebuild.

Pitfall 2: Misusing Build Arguments for Secrets

Why it happens: Passing secrets via ARG or ENV is convenient but leaves them in image layers.

How to avoid: Use BuildKit secrets (RUN --mount=type=secret) for sensitive data during build. Never embed secrets in the final image.

Recovery: If you suspect secrets were baked in, run docker history --no-trunc <image> and search for the secret. If found, revoke the secret and rebuild without it, then push a clean image.

Pitfall 3: Over-pruning Cache Aggressively

Why it happens: In an attempt to fix issues, users run docker builder prune -a -f without considering the impact.

How to avoid: Use targeted pruning: docker builder prune --filter "until=24h" to remove only old cache. Or use docker buildx prune --keep-storage=10GB to limit pruning.

Recovery: If you lose cache unnecessarily, rebuild will take longer. To prevent future over-pruning, document cache management policies and use filters.

Pitfall 4: Running BuildKit in Unsupported Environments

Why it happens: Trying to use BuildKit features (like multi-platform builds) with the default docker driver, which does not support them.

How to avoid: Check the driver capabilities with docker buildx ls. If you need advanced features, create a new builder with the docker-container driver:

docker buildx create --name mybuilder --driver docker-container --use
docker buildx inspect --bootstrap

Recovery: Switch to the appropriate builder using docker buildx use mybuilder.

Pitfall 5: Not Monitoring BuildKit Resource Usage

Why it happens: BuildKit containers can consume significant CPU and memory, especially during parallel builds, leading to host resource exhaustion.

How to avoid: Set resource limits for the builder container when creating a custom builder:

docker buildx create --driver docker-container --driver-opt env.BUILDKIT_STEP_LOG_MAX_SIZE=10m --driver-opt env.BUILDKIT_STEP_LOG_MAX_SPEED=1m mybuilder

Also monitor with docker stats buildx_buildkit_mybuilder.

Recovery: If the builder is unresponsive, restart it: docker restart buildx_buildkit_mybuilder.

Conclusion

Docker BuildKit troubleshooting requires a methodical approach: start with version and environment checks, inspect logs, adjust configuration safely, and recover from known failure modes. The practical examples in this guide should help you resolve most build issues quickly.

Remember to always observe the current state, make one change at a time, verify the result, and document the recovery process. BuildKit's advanced features can significantly improve build performance, but they also introduce new complexities. By following the checklists and avoiding common pitfalls, you can keep your builds reliable and your team productive.

For a next step, choose a low-risk verification from the Operations Checklist, such as testing a minimal build or reviewing cache usage. Record the current state, run the command, compare the result with the expected output, and then investigate any discrepancies. With practice, you will develop an intuitive sense for BuildKit behavior and be able to troubleshoot any build failure with confidence.