## Intro

Docker BuildKit is the modern build engine for Docker, offering faster builds, better caching, and advanced features like parallel execution and secrets handling. However, the build cache and state that BuildKit maintains can become a critical asset in your development workflow. Losing it can mean slower builds, increased network usage, and lost reproducibility. This guide provides a practical approach to backing up and restoring Docker BuildKit's state, including the build cache and configuration, to ensure your build environment can be recovered quickly and safely. We cover inventory, safe configuration, verification, failure modes, and a checklist for ongoing operations.

## Version and Environment Inventory

Before implementing any backup strategy, it is essential to understand the exact version and configuration of Docker and BuildKit in your environment. This inventory helps you locate the relevant files and determine the appropriate backup method.

First, check the Docker version and confirm BuildKit is enabled:

```bash
docker version
```

Example output snippet:

```
Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 ...
Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  ...
```

To verify BuildKit is active, check the builder info:

```bash
docker buildx version
```

Output example:

```
github.com/docker/buildx v0.11.2 docker/docker-ce-desktop: latest
```

If you are using the default builder with BuildKit, you can inspect it:

```bash
docker buildx inspect
```

The output shows the builder name, driver, and node platforms. For example:

```
Name:   default
Driver: docker
Nodes:
Name:      default
Endpoint:  docker-desktop
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6
```

BuildKit stores its state in several locations depending on the driver and operating system. For the Docker driver (the default), the build cache and metadata are stored within the Docker daemon's data directory. On Linux, this is typically `/var/lib/docker`. On macOS and Windows, it is within the Docker Desktop virtual machine or Hyper-V disk, which is less directly accessible from the host.

For a more controllable backup, consider using the `docker-container` driver, which runs BuildKit in a dedicated container. This driver stores its state in a Docker volume that you can easily back up. To create a builder with the `docker-container` driver:

```bash
docker buildx create --name mybuilder --driver docker-container --use
```

Then, you can inspect its volume:

```bash
docker volume ls | grep mybuilder
```

You will see a volume like `mybuilder0_buildkit_state`. This volume contains the BuildKit cache and metadata. Backing up this volume is a reliable method.

Additionally, note the Docker configuration file, usually `/etc/docker/daemon.json` on Linux, which may contain BuildKit-specific settings like `"features": {"buildkit": true}`. This file should also be part of your backup.

Table 1 summarizes the common locations to inventory:

| Component | Location (Linux) | Backup Method |
|---|---|---|
| Docker daemon data (includes BuildKit cache) | `/var/lib/docker` | Stop Docker, copy directory |
| BuildKit state volume (docker-container driver) | Docker volume `mybuilder0_buildkit_state` | `docker run --rm -v mybuilder0_buildkit_state:/data -v /backup:/backup alpine tar czf /backup/buildkit-state.tar.gz -C /data .` |
| Docker daemon configuration | `/etc/docker/daemon.json` | Copy file |
| BuildKit configuration (if custom) | `/etc/buildkit/buildkitd.toml` | Copy file |

## Safe Configuration Path

After inventory, set up backups in a way that minimizes risk and avoids disrupting ongoing builds. The safest approach is to use a dedicated backup script that freezes or snapshots the state without stopping the BuildKit service if possible, or schedules downtime if necessary.

For the Docker daemon data directory, the recommended method is to stop the Docker daemon before copying the directory to ensure consistency. However, if you cannot afford downtime, you can use filesystem snapshots if your storage supports them (e.g., LVM, ZFS, Btrfs). Alternatively, use `tar` with appropriate flags to copy while the daemon is running, but be aware that this may result in an inconsistent backup if a build is actively writing. The risk is low for cache-only data, but it's safer to pause builds during the backup window.

For the `docker-container` driver, you can back up the state volume without stopping the builder by using a temporary container to tar the volume contents. However, to ensure a consistent snapshot, it's best to stop the builder container first:

```bash
docker buildx stop mybuilder
```

Then run the backup:

```bash
docker run --rm \
  -v mybuilder0_buildkit_state:/data \
  -v /backup:/backup \
  alpine tar czf /backup/buildkit-state-$(date +%Y%m%d).tar.gz -C /data .
```

After backup, restart the builder:

```bash
docker buildx start mybuilder
```

To avoid manual steps, create a backup script and schedule it with cron. For example, a script `/usr/local/bin/backup-buildkit.sh`:

```bash
#!/bin/bash
# Backup BuildKit state for mybuilder
set -e
BACKUP_DIR="/backup/buildkit"
VOLUME="mybuilder0_buildkit_state"
TIMESTAMP=$(date +%Y%m%d%H%M%S)
mkdir -p "$BACKUP_DIR"
echo "Stopping builder..."
docker buildx stop mybuilder
echo "Backing up volume..."
docker run --rm -v "$VOLUME:/data" -v "$BACKUP_DIR:/backup" alpine tar czf "/backup/buildkit-state-$TIMESTAMP.tar.gz" -C /data .
echo "Starting builder..."
docker buildx start mybuilder
echo "Backup completed: $BACKUP_DIR/buildkit-state-$TIMESTAMP.tar.gz"
```

Make it executable and add to cron (e.g., daily at 2 AM):

```bash
chmod +x /usr/local/bin/backup-buildkit.sh
echo "0 2 * * * /usr/local/bin/backup-buildkit.sh >> /var/log/buildkit-backup.log 2>&1" | crontab -
```

For the Docker daemon configuration file, simply copy it to the backup directory:

```bash
cp /etc/docker/daemon.json /backup/daemon.json
```

Ensure your backup directory has sufficient space and is itself backed up to offsite or remote storage according to your disaster recovery plan.

## Verification and Diagnostics

After setting up backups, you must verify that the backup is complete and restorable. Do not assume the backup works; test it regularly.

First, check the backup file's existence and size:

```bash
ls -lh /backup/buildkit-state-*.tar.gz
```

Example output:

```
-rw-r--r-- 1 root root 245M Jul 26 02:00 /backup/buildkit-state-20240726020000.tar.gz
```

Verify the tar archive integrity:

```bash
tar tzf /backup/buildkit-state-20240726020000.tar.gz | head
```

You should see the contents of the BuildKit state, such as directories like `runc-native`, `containerd`, `tmp`, etc. For example:

```
./
./runc-native/
./runc-native/metadata.db
./containerd/
./containerd/daemon/
...
```

To test restore without affecting the current environment, create a temporary builder with a new volume and restore the backup into it:

```bash
# Create a temporary builder
docker buildx create --name restoretest --driver docker-container
# Stop it and restore backup into its volume
VOLUME_NAME=$(docker volume ls --format '{{.Name}}' | grep restoretest)
docker buildx stop restoretest
docker run --rm -v "$VOLUME_NAME:/data" -v /backup:/backup alpine sh -c "cd /data && tar xzf /backup/buildkit-state-20240726020000.tar.gz"
docker buildx start restoretest
# Now use the restored builder to see if cache is present
docker buildx use restoretest
docker buildx du
```

The `docker buildx du` command shows disk usage of the cache. If the cache was restored, you should see cache entries with sizes. For example:

```
ID                                      RECLAIMABLE SIZE    LAST ACCESSED
8f2e1c3d4b5a67890abcdef1234567890       true        123MB   2 days ago
...
```

If the cache entries appear, the restore is successful. You can also run a sample build that should hit the cache and observe faster build times.

Additionally, verify the Docker daemon configuration restore by copying back the daemon.json and restarting Docker:

```bash
sudo systemctl restart docker
sudo docker info | grep -A2 'BuildKit'
```

Expected output includes `BuildKit: true` or similar.

After verification, remove the temporary builder:

```bash
docker buildx rm restoretest
```

## Failure Modes and Recovery

Backups can fail, restores may not work as expected, and BuildKit itself may encounter issues. Understanding common failure modes helps you recover quickly.

**Backup failure due to insufficient disk space**

If the backup directory runs out of space, the `tar` command will fail with an error like `No space left on device`. Monitor disk usage and set alerts, for example using `df -h /backup` and configuring a monitoring system to alert when usage exceeds 80%. Recovery: free space and rerun the backup.

**Inconsistent backup due to running builds**

If you back up a live volume without stopping the builder, the `tar` may capture a mid-write state, leading to a corrupt cache on restore. To minimize this, always stop the builder before backup. If you cannot stop it, use filesystem snapshots. Recovery: if a restore fails due to corruption, you may need to start with a fresh BuildKit state and rebuild caches.

**Restore to a different version**

BuildKit state format may change between versions. Restoring a backup from a much older version to a newer version might not work. Always test restore in a staging environment. Recovery: if incompatible, you may need to keep the same version or rebuild the cache.

**Volume name mismatch**

If you restore to a builder with a different volume name, you must ensure the target volume is empty and mounted correctly. For example, if the backup was from volume `mybuilder0_buildkit_state` and you restore to a new builder `newbuilder`, the volume name will be `newbuilder0_buildkit_state`. Use the correct volume name in the restore command.

**Corrupted tar archive**

Always verify the archive with `tar tzf` before relying on it. Keep multiple backup generations to increase the chance of a good copy. For example, keep daily backups for 7 days, weekly for 4 weeks, and monthly for 3 months.

**Builder not starting after restore**

If the restored builder fails to start, check the BuildKit logs:

```bash
docker logs buildx_buildkit_mybuilder0
```

Look for errors like database corruption. If the cache is corrupted, you can delete the volume and start fresh, then rebuild caches:

```bash
docker buildx rm mybuilder
# Recreate from backup if needed
```

**Rollback planning**

Always have a rollback plan in case a restore introduces problems. Keep the previous backup version. If a restore fails, you can re-create the builder from the previous known-good backup. Document the exact commands and test them.

Table 2 summarizes common failure modes and recovery actions:

| Failure Mode | Symptom | Recovery Action |
|---|---|---|
| Backup disk full | `tar` error: No space left on device | Free space, rerun backup |
| Inconsistent backup | Restore fails with unexpected files | Restore from earlier backup |
| Version mismatch | Builder does not start after restore | Use same BuildKit version or rebuild cache |
| Corrupted archive | `tar tzf` reports errors | Restore from another backup generation |
| Restore to wrong volume | Data not visible in builder | Ensure correct volume name and empty target |

## Common Pitfalls and How to Avoid Them

Beyond explicit failures, there are several common mistakes that can undermine your BuildKit backup strategy.

**Pitfall 1: Not testing restores regularly**

Many teams set up backups but never test restore. A backup is only as good as its ability to restore. Without testing, you may discover at the worst time that your backups are unusable.

*Avoidance*: Schedule a quarterly restore test as part of your operations checklist. Use the temporary builder method described earlier to validate the backup without affecting production.

**Pitfall 2: Backing up the wrong volume or directory**

If you are using the `docker-container` driver, the BuildKit state is in a Docker volume, not in the host filesystem directly. Backing up `/var/lib/docker/volumes` without understanding the volume layout may capture incomplete data.

*Avoidance*: Use `docker volume ls` to identify the exact volume name and back it up using the `tar` method described. Validate the archive contents to ensure they match expected BuildKit files.

**Pitfall 3: Not stopping the builder during backup**

Backing up a running builder can lead to an inconsistent state, especially if a build is in progress. While it may work for cache-only data, it is risky.

*Avoidance*: Always stop the builder with `docker buildx stop` before backup, or use filesystem snapshots if downtime is not acceptable.

**Pitfall 4: Ignoring BuildKit configuration files**

The builder setup includes more than just cache data. If you have custom BuildKit configurations (e.g., `buildkitd.toml`) or Docker daemon settings, losing them can break your build environment even if cache is restored.

*Avoidance*: Include configuration files in your backup script, and store them in version control for easy recovery.

**Pitfall 5: Not retaining multiple backup generations**

A single backup can be corrupted or incomplete. Relying on one backup increases risk.

*Avoidance*: Implement a retention policy: keep 7 daily, 4 weekly, and 3 monthly backups. Automate rotation using a tool like `logrotate` or a custom script.

**Pitfall 6: Forgetting about Docker Desktop specifics**

On macOS and Windows, the Docker daemon runs inside a VM, and the BuildKit state is within the VM disk image. Backing up host directories may not capture BuildKit state effectively.

*Avoidance*: Use the `docker-container` driver even on Docker Desktop to have a dedicated volume that can be backed up consistently. Alternatively, use Docker Desktop's built-in backup features or export the VM.

## Operations Checklist

To ensure ongoing reliability, incorporate the following checks and procedures into your regular operations. Assign a single owner for each item and set a review frequency.

| Checklist Item | Owner | Frequency |
|---|---|---|
| Inventory Docker and BuildKit versions; update documentation | DevOps Engineer (e.g., Priya Shah) | Monthly |
| Confirm backup script runs successfully; review logs for errors | System Administrator (e.g., Mark Chen) | Daily |
| Test a restore to a non-production builder and verify cache presence | QA Lead (e.g., Laura Rodriguez) | Quarterly |
| Rotate backups: keep daily for 7 days, weekly for 4 weeks, monthly for 3 months | Backup Operator (e.g., Sam Patel) | Automated, verify weekly |
| Monitor backup storage capacity and alert if usage exceeds 80% | Monitoring Team (e.g., DevOps Engineer) | Continuous, review monthly |
| Document environment-specific commands in runbook | Technical Writer (e.g., Emily Nguyen) | On change, review quarterly |
| Conduct a disaster recovery drill | IT Manager (e.g., David Kim) | Twice a year |
| Review builder configurations and adjust caching parameters | Build Engineer (e.g., Alex Johnson) | Monthly |

Each owner should document the outcome of their checks and updates in a shared log. For example, the monthly version inventory should record the Docker and BuildKit versions and any changes to the backup procedure.

## Conclusion

Backing up and restoring Docker BuildKit state is a straightforward but critical practice for maintaining development efficiency. By following the steps in this guide, you can protect your build cache, reduce rebuild times, and recover quickly from failures. Start with a simple backup of the BuildKit volume using the `docker-container` driver, verify the restore in a test environment, and then integrate the process into your regular operations with scheduled scripts and monitoring. Regular testing and documentation will ensure your backups are always ready when you need them.

For further reading, consult the official Docker Buildx documentation and BuildKit repository for the latest best practices and version-specific details.