## Intro

Docker BuildKit is the modern build engine for Docker images, offering faster builds, advanced caching, and more flexibility. However, its power also introduces new security considerations. A misconfigured BuildKit can expose build secrets, allow unauthorized access, or inadvertently leak data. This guide provides a practical, step-by-step approach to hardening BuildKit in your environment, ensuring your image builds are secure without sacrificing performance.

A clear hardening process reduces rework and confusion. By starting with a narrow, measurable pilot, you can inspect and adjust security settings locally before rolling them out broadly. This article walks you through version checks, access control, secret management, permission tightening, network exposure limitation, and verification. Each step includes concrete commands and expected outputs so you can follow along and confirm your configuration.

## Version and Environment Inventory

Before making any changes, you need a clear picture of your current Docker and BuildKit setup. This inventory establishes the baseline and helps identify which security features are available. BuildKit is included with Docker Engine 18.09 and later, and is the default builder since Docker 23.0. Older versions may not support all hardening options.

First, check your Docker and BuildKit versions:

```
docker version
```

Expected output (example from a recent version):

```
Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        ced0996
 Built:             Tue Jul 25 02:52:41 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       a61e2b4
  Built:            Tue Jul 25 02:52:41 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6878b9c4c58f6ac5a6d5b69a2c8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```

To determine if BuildKit is enabled, run:

```
docker buildx version
```

If BuildKit is available, you will see output similar to:

```
github.com/docker/buildx v0.11.2 9872040
```

If `docker buildx` is not found, you may need to install the buildx plugin or enable BuildKit by setting the environment variable `DOCKER_BUILDKIT=1` for each build command or in your shell profile.

You can also inspect the current builder configuration:

```
docker buildx ls
```

Example output:

```
NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS
default * docker
  default default running 20.10.24 linux/amd64, linux/arm64, linux/arm/v7, linux/arm/v6
```

Note the BuildKit version shown. Some security features may require a minimum BuildKit version. For example, `--secret` and `--ssh` options have been available for a long time, but newer features like `--mount=type=cache` with `sharing=private` require BuildKit 0.10 or later.

Another important check is whether you are using the default Docker daemon configuration or a custom one. The daemon configuration file is typically at `/etc/docker/daemon.json`. You can view its contents (requires root or sudo):

```
sudo cat /etc/docker/daemon.json
```

If the file does not exist, Docker uses default settings. Important daemon settings for BuildKit security include `"buildkit": true` (optional, as BuildKit is default), `"features": { "buildkit": true }` (deprecated), or `"builder": { "gc": { "enabled": true, "defaultKeepStorage": "10GB" } }` (garbage collection).

Also note the current user and group, as they affect access permissions to the Docker socket, which is the main entry point for build operations.

Finally, check whether you have any remote builders configured, as those may have separate security considerations:

```
docker buildx inspect default
```

This shows details like driver, platforms, and buildkit version for the builder.

Now that you have a full inventory, you can move on to implementing security hardening in a controlled manner.

## Safe Configuration Path

Hardening BuildKit involves several areas: access control, secrets management, build permissions, network exposure, and build container isolation. Because changes can affect build functionality, it is important to apply them incrementally and test each step. A narrow, measurable pilot is the best first step, as it allows you to verify each change locally before broader rollout.

### 1. Restrict Access to the Docker Daemon and BuildKit

BuildKit operations are triggered through the Docker daemon. The primary security control is who can access the Docker socket (`/var/run/docker.sock`). By default, this socket is owned by root and the `docker` group. Ensure only trusted users are in the `docker` group. Adding a user to the `docker` group grants them full control over the daemon, effectively root-equivalent on the host, because they can mount any directory or run privileged containers. Therefore, restrict group membership.

To list users in the `docker` group:

```
grep docker /etc/group
```

If you need to add a user, use:

```
sudo usermod -aG docker username
```

But be aware of the security implications. For stronger isolation, consider using rootless Docker or remote BuildKit with authentication, but those are beyond this guide.

Additionally, you can configure the Docker daemon to require TLS for remote connections. If you must expose the Docker API over the network, enable TLS authentication. The daemon configuration would include:

```json
{
  "tls": true,
  "tlscert": "/etc/docker/certs/server-cert.pem",
  "tlskey": "/etc/docker/certs/server-key.pem",
  "tlsverify": true,
  "tlscacert": "/etc/docker/certs/ca.pem"
}
```

Then restart Docker. Without TLS, anyone who can reach the port can control Docker. Even with TLS, ensure certificates are properly secured.

### 2. Use BuildKit Secrets for Sensitive Data

Never embed secrets like passwords, API keys, or tokens directly in a Dockerfile `RUN` command. BuildKit provides `--secret` to mount secret files or environment variables during build steps without persisting them in the image layers. This is a major security improvement over older methods like `COPY`ing secrets and then deleting them, which leaves them in image history.

Example Dockerfile using a secret:

```dockerfile
# syntax=docker/dockerfile:1
FROM alpine:latest
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
```

Build with:

```
echo "supersecret" > secret.txt
docker build --secret id=mysecret,src=secret.txt -t myapp .
```

The secret is available only during that `RUN` instruction and is not stored in the final image or intermediate layers (unless you intentionally copy it into a layer). You can also use `--secret id=mysecret,env=MY_SECRET` to pass an environment variable.

If you need SSH agent forwarding for private repositories, use `--ssh`:

```dockerfile
RUN --mount=type=ssh git clone git@github.com:example/repo.git
```

Build with:

```
docker build --ssh default -t myapp .
```

This avoids copying SSH private keys into the image.

### 3. Harden Build Container Permissions

BuildKit runs each `RUN` instruction in a container with certain privileges. By default, build containers run as root. You can change the user inside the build steps using the `USER` directive in the Dockerfile. For steps that need root (e.g., installing packages), you can switch back to a non-root user for the remaining steps:

```dockerfile
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y curl
USER appuser
RUN curl ...
```

This reduces the risk if a build step is compromised. Additionally, you can use the `--security=sandbox` flag in `RUN` to run in a more restricted seccomp/AppArmor profile (requires BuildKit 0.10+). For example:

```dockerfile
RUN --security=sandbox apt-get update
```

By default, `RUN` uses `sandbox`, but you can also use `insecure` to disable the sandbox (not recommended unless needed). Keep the default.

### 4. Control Network Access During Builds

BuildKit allows you to control network access per build step. By default, build steps have network access. If your build does not need network for certain steps (e.g., just copying files), you can run with `--network=none`:

```dockerfile
RUN --network=none go build -o app .
```

This prevents any network connections from that step, reducing the attack surface. For steps that need network, you can restrict to a specific custom network using `--network=bridge` or a user-defined network. For example, in a multi-stage build, you might isolate the download stage.

At the builder level, you can disable network access for all builds if you are using a remote builder, but that is less practical. Instead, use per-step control.

### 5. Configure BuildKit Garbage Collection and Cache

BuildKit caches layers to speed up builds, but this cache can contain sensitive data if secrets or private code were copied into a layer. Configure garbage collection (GC) to periodically prune cache. The Docker daemon has a default GC policy, but you can adjust it in `/etc/docker/daemon.json`:

```json
{
  "builder": {
    "gc": {
      "enabled": true,
      "defaultKeepStorage": "20GB",
      "policy": [
        { "keepStorage": "10GB", "filter": ["unused-for=24h"] },
        { "keepStorage": "5GB", "filter": ["unused-for=168h"] }
      ]
    }
  }
}
```

This example keeps up to 10GB of cache used in the last 24 hours and 5GB of cache used in the last week, and removes anything unused for longer. Adjust to your needs. After changing, restart Docker.

### 6. Enable Content Trust (Optional)

Content trust ensures that the base images you pull are signed by a trusted publisher. Enable it with:

```
export DOCKER_CONTENT_TRUST=1
```

Then when you pull an image, Docker verifies the signature. For builds, you may want to pin base images by digest in your Dockerfile:

```dockerfile
FROM alpine@sha256:123456...
```

This prevents accidental use of a different version.

### Example: Building a Simple App Securely

Let's walk through a concrete example. Assume you have a Node.js application that needs an npm package from a private registry. With BuildKit, you can:

- Use a multi-stage build.
- Mount a secret for the npm token.
- Use `--network=none` where appropriate.
- Run as non-root in the final stage.

Dockerfile:

```dockerfile
# syntax=docker/dockerfile:1
FROM node:18 AS builder
WORKDIR /app
COPY package*.json ./
RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) npm install
COPY . .
RUN --network=none npm run build

FROM node:18-alpine
WORKDIR /app
USER node
COPY --from=builder /app/dist ./dist
EXPOSE 3000
CMD ["node", "dist/index.js"]
```

Build with:

```
echo "your_npm_token" > npm_token.txt
docker build --secret id=npm_token,src=npm_token.txt -t myapp:latest .
```

This ensures the token is not stored in the image. The `--network=none` on the build step prevents any unintended network access. The final stage runs as `node` user.

## Verification and Diagnostics

After applying hardening measures, you must verify they are effective. Here are checks for each area.

### 1. Confirm Secrets Are Not in Image Layers

After building with `--secret`, run a container from the image and ensure the secret is not present. For the example above:

```
docker run -it --rm myapp:latest sh -c 'ls -la /run/secrets/ 2>/dev/null; echo $NPM_TOKEN'
```

Expected: no `/run/secrets` directory and empty `$NPM_TOKEN`. Also check image history to ensure no layer contains the secret string:

```
docker history myapp:latest
```

Look for any layer that might have included the secret. For a more thorough check, use `docker save` and inspect the tarball, or use tools like `dive` to analyze layers. Example:

```
dive myapp:latest
```

In `dive`, you can browse each layer's file changes and search for the secret.

### 2. Check BuildKit Cache for Secrets

If you suspect secrets may have been cached in previous builds, run:

```
docker builder prune --all --force
```

This removes all build cache. To check current cache usage:

```
docker system df
```

This shows space used by build cache under `Build Cache`.

### 3. Verify Network Restrictions

To ensure `--network=none` worked, you can temporarily add a step that fails if network is available. For example, in your Dockerfile, add a test `RUN` (not for production) that tries to ping an external IP and fails if successful. Better: after building with `--network=none`, inspect the build logs. BuildKit logs network usage? Not directly. Instead, you can use a side-effect test: create a `RUN` step with `--network=none` that attempts to download a file and should fail if the step actually runs. But since `--network=none` is a build-time flag, the step will fail if it tries network. So, if your build succeeded with `--network=none` on a step that previously needed network, then the restriction worked. You can also run `docker build --progress=plain` to see full output, including any network errors if they occurred.

### 4. Check Container User Permissions

Run a container from the final image and check the user:

```
docker run -it --rm myapp:latest id
```

Expected output for above: `uid=1000(node) gid=1000(node) groups=1000(node)`. If it shows `root`, then the `USER` directive did not apply to the final stage.

### 5. Verify Docker Daemon TLS (If Configured)

If you enabled TLS on the Docker daemon, test remote access without certificates to ensure it is denied:

```
docker -H tcp://your-server:2376 info
```

Should return a certificate error. With proper client certificates, it should succeed.

### 6. Check BuildKit Version and Features

Run:

```
docker buildx ls
```

Ensure the buildkit version is as expected. For specific feature support, refer to the BuildKit release notes.

## Common Pitfalls and How to Avoid Them

Even with careful configuration, common mistakes can undermine BuildKit security. Here are some pitfalls and how to avoid or recover from them.

### Pitfall 1: Accidentally Baking Secrets into Image Layers

**Why it happens:** Developers use `COPY` or `ADD` to include a `.env` file or private key, then delete it in a later layer, assuming it is gone. However, the secret remains in the intermediate layer and can be extracted with `docker history` or by pulling the image.

**How to avoid:** Always use BuildKit secrets (`--secret` or `--mount=type=secret`) for any sensitive data needed during build. If a secret must be a file in the final image, consider using runtime secrets via an orchestrator or environment variables instead.

**Recovery:** If a secret is discovered in an image, rotate the credential immediately, rebuild the image without the secret, and prune the build cache with `docker builder prune --all --force` to remove any cached layers containing it.

### Pitfall 2: Overly Broad Access to the Docker Socket

**Why it happens:** To simplify development, teams add many users to the `docker` group, granting them root-equivalent access. This expands the attack surface if any of those users' accounts are compromised.

**How to avoid:** Only add users who absolutely need to build images, and consider using rootless Docker or a remote BuildKit with authentication for stronger isolation. Regularly audit group membership with `grep docker /etc/group`.

**Recovery:** Remove unnecessary users from the `docker` group with `sudo gpasswd -d username docker`, and implement a review process for new group additions.

### Pitfall 3: Missing Network Isolation for Sensitive Build Steps

**Why it happens:** Developers assume builds are isolated and do not realize that build containers have network access by default. A compromised base image or dependency could exfiltrate data.

**How to avoid:** Apply `--network=none` to any step that does not require network access, such as compilation or file copying steps. For steps that need network, limit to a private network if possible.

**Recovery:** Review Dockerfiles and add `--network=none` where appropriate. If a step fails because it needs network, assess whether the network requirement can be removed or whether a more restricted network can be used.

### Pitfall 4: Running Build Containers as Root

**Why it happens:** Default Dockerfile `RUN` instructions execute as root, and developers often overlook switching to a non-root user. If a build step is compromised, the attacker gains root privileges within the build environment.

**How to avoid:** Use the `USER` directive to switch to a non-root user for all steps that do not require root, especially in the final stage. For steps that must run as root, consider using `--security=sandbox` to restrict capabilities.

**Recovery:** Refactor the Dockerfile to create a dedicated user and switch to it after prerequisite root steps. Test the built image to ensure the application runs with the non-root user.

### Pitfall 5: Ignoring Build Cache Security

**Why it happens:** Build cache is seen as ephemeral and not a security risk. However, cached layers may contain secrets or vulnerable dependencies that persist longer than expected.

**How to avoid:** Configure BuildKit garbage collection with appropriate retention policies, and periodically prune cache manually with `docker builder prune`. Also, avoid using cache mounts for sensitive data unless necessary, and use `sharing=private` for cache mounts that may contain secrets.

**Recovery:** If cache is suspected to contain secrets, prune it immediately and rebuild. Adjust GC policies to prevent long-term retention of unused cache.

## Failure Modes and Recovery

Despite careful hardening, issues can arise. Here are common failure modes and how to recover.

### 1. Build Fails Due to `--network=none`

If a step requires network but you set `--network=none`, the build will fail with a network error, such as:

```
error: failed to solve: process "/bin/sh -c apt-get update" did not complete successfully: exit code: 100
```

Recovery: Remove `--network=none` from that step, or adjust the step to use a network if absolutely necessary. Then rebuild.

### 2. Secret Mount Not Working

If the secret is not found, the build fails with something like:

```
error: failed to solve: secret mysecret not found
```

Check that you passed `--secret` correctly and the source file exists. Also verify the mount syntax in the Dockerfile. The mount must be on the same line as the `RUN` command.

### 3. Secret Leakage Due to Caching

If a secret was used in a build step that also copied files or produced output, it might be cached. For example, if you use `RUN --mount=type=secret ... && curl -H ... > output.txt`, the output file may contain the secret. To recover, prune the cache and rebuild without the secret in the output:

```
docker builder prune --all --force
```

Then fix the Dockerfile to avoid writing secrets to files, or ensure those files are not included in the final image.

### 4. TLS Misconfiguration Causes Docker Daemon Failure

If the TLS settings are wrong, the Docker daemon may fail to start. Check logs:

```
journalctl -u docker.service
```

Look for certificate errors. Recovery: correct the paths or certificate contents in `/etc/docker/daemon.json`, or revert to no TLS if necessary, then restart Docker.

### 5. User Permission Issues in Container

If you set `USER` to a non-root user but later steps need root, the build fails with permission denied. Recovery: Temporarily switch back to root with `USER root` before the step that needs it, then switch back.

### 6. BuildKit Garbage Collection Too Aggressive

If you set GC policy too low, you may lose cache frequently, slowing builds. Monitor cache hit rate with `docker build --progress=plain` to see cache usage. Adjust `keepStorage` values upward if needed.

In all cases, maintain backups of your Dockerfile and build scripts, and use version control so you can roll back changes easily.

## Operations Checklist

Use this checklist for routine security reviews and before deploying changes. Assign a single accountable owner for each major item and set a review frequency.

| Checklist Item | Owner (example) | Review Frequency |
|----------------|-----------------|------------------|
| Docker and BuildKit versions are up to date | DevOps Engineer | Monthly |
| Only authorized users in docker group | System Administrator | Quarterly |
| Dockerfiles use secrets, not COPY for sensitive data | Security Champion | Every build |
| Base images pinned by digest | DevOps Engineer | Every build |
| Non-root user in final stage | Application Developer | Every build |
| Network isolation applied where possible | Application Developer | Every build |
| Build cache pruning policy is effective | DevOps Engineer | Monthly |
| TLS certificates for remote Docker daemon are valid | System Administrator | Quarterly |

### Pre-Build Checklist

- [ ] Ensure Docker and BuildKit versions are up to date.
- [ ] Confirm only authorized users are in the `docker` group.
- [ ] Set `DOCKER_BUILDKIT=1` (or rely on default) to ensure BuildKit is used.
- [ ] Review Dockerfile for secrets; replace `COPY secret` with `--secret`.
- [ ] Pin base images by digest where possible.
- [ ] Identify steps that need network; plan to use `--network=none` for others.
- [ ] Plan non-root user for runtime stage.

### Post-Build Verification Checklist

- [ ] Run `docker history` to check for unusual layers.
- [ ] Run `docker run` to verify user is non-root.
- [ ] Use `dive` or similar to inspect for secrets.
- [ ] Confirm build cache size is acceptable.

### Regular Operations

- [ ] Periodically prune build cache (`docker builder prune`).
- [ ] Monitor Docker daemon logs for unauthorized access attempts.
- [ ] Review and update secret management processes.
- [ ] Keep BuildKit and Docker updated with security patches.

### Incident Response

- [ ] If a secret is suspected in an image, immediately rebuild without the secret and prune cache.
- [ ] Rotate any exposed credentials.
- [ ] Investigate how the secret was leaked and update processes.

## Conclusion

Hardening Docker BuildKit is essential for secure image builds. By following the steps in this guide, you can control access to the build environment, securely manage secrets, restrict build permissions and network access, and verify your configurations. Start with a small, measurable pilot: take one Dockerfile, apply these hardening techniques, and test thoroughly. Once confident, expand to other projects.

Remember that security is an ongoing process. Regularly review your build configurations, stay updated with Docker and BuildKit releases, and incorporate new security features as they become available. A clear hardening process reduces rework and helps your team produce reliable, secure containers.