Docker BuildKit Backup and Restore: A Practical Guide with Worked Examples

Intro

Docker BuildKit is the modern build engine for Docker, offering faster builds, better caching, and advanced features like parallel execution and secrets handling. However, the build cache and state that BuildKit maintains can become a critical asset in your development workflow. Losing it can mean slower builds, increased network usage, and lost reproducibility. This guide provides a practical approach to backing up and restoring Docker BuildKit's state, including the build cache and configuration, to ensure your build environment can be recovered quickly and safely. We cover inventory, safe configuration, verification, failure modes, and a checklist for ongoing operations.

Version and Environment Inventory

Before implementing any backup strategy, it is essential to understand the exact version and configuration of Docker and BuildKit in your environment. This inventory helps you locate the relevant files and determine the appropriate backup method.

First, check the Docker version and confirm BuildKit is enabled:

docker version

Example output snippet:

Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 ...
Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  ...

To verify BuildKit is active, check the builder info:

docker buildx version

Output example:

github.com/docker/buildx v0.11.2 docker/docker-ce-desktop: latest

If you are using the default builder with BuildKit, you can inspect it:

docker buildx inspect

The output shows the builder name, driver, and node platforms. For example:

Name:   default
Driver: docker
Nodes:
Name:      default
Endpoint:  docker-desktop
Status:    running
Platforms: linux/amd64, linux/arm64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

BuildKit stores its state in several locations depending on the driver and operating system. For the Docker driver (the default), the build cache and metadata are stored within the Docker daemon's data directory. On Linux, this is typically /var/lib/docker. On macOS and Windows, it is within the Docker Desktop virtual machine or Hyper-V disk, which is less directly accessible from the host.

For a more controllable backup, consider using the docker-container driver, which runs BuildKit in a dedicated container. This driver stores its state in a Docker volume that you can easily back up. To create a builder with the docker-container driver:

docker buildx create --name mybuilder --driver docker-container --use

Then, you can inspect its volume:

docker volume ls | grep mybuilder

You will see a volume like mybuilder0_buildkit_state. This volume contains the BuildKit cache and metadata. Backing up this volume is a reliable method.

Additionally, note the Docker configuration file, usually /etc/docker/daemon.json on Linux, which may contain BuildKit-specific settings like "features": {"buildkit": true}. This file should also be part of your backup.

Table 1 summarizes the common locations to inventory:

Safe Configuration Path

After inventory, set up backups in a way that minimizes risk and avoids disrupting ongoing builds. The safest approach is to use a dedicated backup script that freezes or snapshots the state without stopping the BuildKit service if possible, or schedules downtime if necessary.

For the Docker daemon data directory, the recommended method is to stop the Docker daemon before copying the directory to ensure consistency. However, if you cannot afford downtime, you can use filesystem snapshots if your storage supports them (e.g., LVM, ZFS, Btrfs). Alternatively, use tar with appropriate flags to copy while the daemon is running, but be aware that this may result in an inconsistent backup if a build is actively writing. The risk is low for cache-only data, but it's safer to pause builds during the backup window.

For the docker-container driver, you can back up the state volume without stopping the builder by using a temporary container to tar the volume contents. However, to ensure a consistent snapshot, it's best to stop the builder container first:

docker buildx stop mybuilder

Then run the backup:

docker run --rm \
  -v mybuilder0_buildkit_state:/data \
  -v /backup:/backup \
  alpine tar czf /backup/buildkit-state-$(date +%Y%m%d).tar.gz -C /data .

After backup, restart the builder:

docker buildx start mybuilder

To avoid manual steps, create a backup script and schedule it with cron. For example, a script /usr/local/bin/backup-buildkit.sh:

#!/bin/bash
# Backup BuildKit state for mybuilder
set -e
BACKUP_DIR="/backup/buildkit"
VOLUME="mybuilder0_buildkit_state"
TIMESTAMP=$(date +%Y%m%d%H%M%S)
mkdir -p "$BACKUP_DIR"
echo "Stopping builder..."
docker buildx stop mybuilder
echo "Backing up volume..."
docker run --rm -v "$VOLUME:/data" -v "$BACKUP_DIR:/backup" alpine tar czf "/backup/buildkit-state-$TIMESTAMP.tar.gz" -C /data .
echo "Starting builder..."
docker buildx start mybuilder
echo "Backup completed: $BACKUP_DIR/buildkit-state-$TIMESTAMP.tar.gz"

Make it executable and add to cron (e.g., daily at 2 AM):

chmod +x /usr/local/bin/backup-buildkit.sh
echo "0 2 * * * /usr/local/bin/backup-buildkit.sh >> /var/log/buildkit-backup.log 2>&1" | crontab -

For the Docker daemon configuration file, simply copy it to the backup directory:

cp /etc/docker/daemon.json /backup/daemon.json

Ensure your backup directory has sufficient space and is itself backed up to offsite or remote storage according to your disaster recovery plan.

Verification and Diagnostics

After setting up backups, you must verify that the backup is complete and restorable. Do not assume the backup works; test it regularly.

First, check the backup file's existence and size:

ls -lh /backup/buildkit-state-*.tar.gz

Example output:

-rw-r--r-- 1 root root 245M Jul 26 02:00 /backup/buildkit-state-20240726020000.tar.gz

Verify the tar archive integrity:

tar tzf /backup/buildkit-state-20240726020000.tar.gz | head

You should see the contents of the BuildKit state, such as directories like runc-native, containerd, tmp, etc. For example:

./
./runc-native/
./runc-native/metadata.db
./containerd/
./containerd/daemon/
...

To test restore without affecting the current environment, create a temporary builder with a new volume and restore the backup into it:

# Create a temporary builder
docker buildx create --name restoretest --driver docker-container
# Stop it and restore backup into its volume
VOLUME_NAME=$(docker volume ls --format '{{.Name}}' | grep restoretest)
docker buildx stop restoretest
docker run --rm -v "$VOLUME_NAME:/data" -v /backup:/backup alpine sh -c "cd /data && tar xzf /backup/buildkit-state-20240726020000.tar.gz"
docker buildx start restoretest
# Now use the restored builder to see if cache is present
docker buildx use restoretest
docker buildx du

The docker buildx du command shows disk usage of the cache. If the cache was restored, you should see cache entries with sizes. For example:

ID                                      RECLAIMABLE SIZE    LAST ACCESSED
8f2e1c3d4b5a67890abcdef1234567890       true        123MB   2 days ago
...

If the cache entries appear, the restore is successful. You can also run a sample build that should hit the cache and observe faster build times.

Additionally, verify the Docker daemon configuration restore by copying back the daemon.json and restarting Docker:

sudo systemctl restart docker
sudo docker info | grep -A2 'BuildKit'

Expected output includes BuildKit: true or similar.

After verification, remove the temporary builder:

docker buildx rm restoretest

Failure Modes and Recovery

Backups can fail, restores may not work as expected, and BuildKit itself may encounter issues. Understanding common failure modes helps you recover quickly.

Backup failure due to insufficient disk space

If the backup directory runs out of space, the tar command will fail with an error like No space left on device. Monitor disk usage and set alerts, for example using df -h /backup and configuring a monitoring system to alert when usage exceeds 80%. Recovery: free space and rerun the backup.

Inconsistent backup due to running builds

If you back up a live volume without stopping the builder, the tar may capture a mid-write state, leading to a corrupt cache on restore. To minimize this, always stop the builder before backup. If you cannot stop it, use filesystem snapshots. Recovery: if a restore fails due to corruption, you may need to start with a fresh BuildKit state and rebuild caches.

Restore to a different version

BuildKit state format may change between versions. Restoring a backup from a much older version to a newer version might not work. Always test restore in a staging environment. Recovery: if incompatible, you may need to keep the same version or rebuild the cache.

Volume name mismatch

If you restore to a builder with a different volume name, you must ensure the target volume is empty and mounted correctly. For example, if the backup was from volume mybuilder0_buildkit_state and you restore to a new builder newbuilder, the volume name will be newbuilder0_buildkit_state. Use the correct volume name in the restore command.

Corrupted tar archive

Always verify the archive with tar tzf before relying on it. Keep multiple backup generations to increase the chance of a good copy. For example, keep daily backups for 7 days, weekly for 4 weeks, and monthly for 3 months.

Builder not starting after restore

If the restored builder fails to start, check the BuildKit logs:

docker logs buildx_buildkit_mybuilder0

Look for errors like database corruption. If the cache is corrupted, you can delete the volume and start fresh, then rebuild caches:

docker buildx rm mybuilder
# Recreate from backup if needed

Rollback planning

Always have a rollback plan in case a restore introduces problems. Keep the previous backup version. If a restore fails, you can re-create the builder from the previous known-good backup. Document the exact commands and test them.

Table 2 summarizes common failure modes and recovery actions:

Common Pitfalls and How to Avoid Them

Beyond explicit failures, there are several common mistakes that can undermine your BuildKit backup strategy.

Pitfall 1: Not testing restores regularly

Many teams set up backups but never test restore. A backup is only as good as its ability to restore. Without testing, you may discover at the worst time that your backups are unusable.

Avoidance: Schedule a quarterly restore test as part of your operations checklist. Use the temporary builder method described earlier to validate the backup without affecting production.

Pitfall 2: Backing up the wrong volume or directory

If you are using the docker-container driver, the BuildKit state is in a Docker volume, not in the host filesystem directly. Backing up /var/lib/docker/volumes without understanding the volume layout may capture incomplete data.

Avoidance: Use docker volume ls to identify the exact volume name and back it up using the tar method described. Validate the archive contents to ensure they match expected BuildKit files.

Pitfall 3: Not stopping the builder during backup

Backing up a running builder can lead to an inconsistent state, especially if a build is in progress. While it may work for cache-only data, it is risky.

Avoidance: Always stop the builder with docker buildx stop before backup, or use filesystem snapshots if downtime is not acceptable.

Pitfall 4: Ignoring BuildKit configuration files

The builder setup includes more than just cache data. If you have custom BuildKit configurations (e.g., buildkitd.toml) or Docker daemon settings, losing them can break your build environment even if cache is restored.

Avoidance: Include configuration files in your backup script, and store them in version control for easy recovery.

Pitfall 5: Not retaining multiple backup generations

A single backup can be corrupted or incomplete. Relying on one backup increases risk.

Avoidance: Implement a retention policy: keep 7 daily, 4 weekly, and 3 monthly backups. Automate rotation using a tool like logrotate or a custom script.

Pitfall 6: Forgetting about Docker Desktop specifics

On macOS and Windows, the Docker daemon runs inside a VM, and the BuildKit state is within the VM disk image. Backing up host directories may not capture BuildKit state effectively.

Avoidance: Use the docker-container driver even on Docker Desktop to have a dedicated volume that can be backed up consistently. Alternatively, use Docker Desktop's built-in backup features or export the VM.

Operations Checklist

To ensure ongoing reliability, incorporate the following checks and procedures into your regular operations. Assign a single owner for each item and set a review frequency.

Each owner should document the outcome of their checks and updates in a shared log. For example, the monthly version inventory should record the Docker and BuildKit versions and any changes to the backup procedure.

Conclusion

Backing up and restoring Docker BuildKit state is a straightforward but critical practice for maintaining development efficiency. By following the steps in this guide, you can protect your build cache, reduce rebuild times, and recover quickly from failures. Start with a simple backup of the BuildKit volume using the docker-container driver, verify the restore in a test environment, and then integrate the process into your regular operations with scheduled scripts and monitoring. Regular testing and documentation will ensure your backups are always ready when you need them.

For further reading, consult the official Docker Buildx documentation and BuildKit repository for the latest best practices and version-specific details.