GitLab Runner Backup and Restore: A Practical Guide with Examples

Intro

GitLab Runner is the workhorse of GitLab CI/CD pipelines. It executes the jobs defined in your .gitlab-ci.yml files on the infrastructure you specify, whether that is a shell on a virtual machine, a Docker container, or a Kubernetes pod. When a runner fails, pipelines stall, deployments freeze, and developer productivity takes a hit. To recover quickly and avoid losing critical build artifacts, caches, and configurations, you need a solid backup and restore plan. This guide walks through practical backup strategies for the three most common GitLab Runner deployment types: binary installation on Linux, Docker container deployment, and Kubernetes deployment. You will learn how to back up configuration files, Docker volumes, and Kubernetes resources; how to verify those backups; how to diagnose and recover from failures; and how to maintain a repeatable backup schedule. The examples use realistic commands and a sample environment so you can adapt them directly to your own infrastructure.

Version and Environment Inventory

Before you back anything up, you must know exactly what you have. The backup process differs significantly depending on how the runner is installed and how it executes jobs. The three primary deployment modes are:

• Binary installation on Linux/macOS: The runner is installed directly on the host operating system, often as a system service. This is common for shell and Docker executors on standalone machines.
• Docker container deployment: The runner itself runs inside a Docker container. The container may use the Docker executor (spawning sibling containers) or another executor.
• Kubernetes deployment: The runner is installed in a Kubernetes cluster, typically via a Helm chart, and uses the Kubernetes executor to run each job in a dedicated pod.

Prerequisites

You need the following in place before implementing a backup routine:

• Backup storage: A secure location with enough space for your archives. Use a separate disk, an S3 bucket, a network share, or a dedicated backup server. Do not store backups on the same physical disk as the runner; a disk failure would then destroy both the runner and its backup.
• Access credentials: Permissions to read runner configuration files, to access Docker volumes, and to execute kubectl and helm commands if you have Kubernetes runners.
• Tools: tar, rsync, docker, kubectl, and helm, depending on your environment.

Inventory Checklist

Create a table that lists every runner, its hostname, executor, configuration file path, and any data volumes or persistent storage it uses. The table below shows a sample inventory for three runners. Replace the entries with your own values and keep the table updated as runners are added, removed, or changed. Assign a Single Accountable Owner (SAO) for the inventory, usually the CI/CD infrastructure engineer. That person is responsible for updating the inventory within one business day after any change and reviewing it monthly with the team.

For each runner, note whether the configuration file is the only thing you need to back up, or if there are also Docker volumes, cache directories, or persistent volumes. For example, runner-1 has a Docker volume named runner-cache that stores build artifacts and layer caches; if that volume is lost, builds will have to re-download dependencies and recompile, but the runner itself can be restored from config.toml alone.

Safe Configuration Path

The main configuration file for GitLab Runner is config.toml. On Linux binary installations, it is typically located at /etc/gitlab-runner/config.toml. This file contains the runner registration token, executor settings, concurrency limits, and other options. Backing it up is essential but not sufficient for runners that use container executors with cached images or volumes. The sections below show how to back up the configuration file for each deployment type, along with any associated data.

Backing Up the Configuration File (Binary Installation)

For a Linux binary installation, run the following command as a user with sudo privileges. It creates a compressed archive of the configuration file and the certs directory (if it exists):

sudo tar -czf gitlab-runner-config-$(date +%Y%m%d).tar.gz \
  /etc/gitlab-runner/config.toml \
  /etc/gitlab-runner/certs/ 2>/dev/null || true

Expected output: a file named gitlab-runner-config-YYYYMMDD.tar.gz in the current directory. For example, if today is January 15, 2025, the file is gitlab-runner-config-20250115.tar.gz. The archive contains the config.toml and any files in the certs directory. If the certs directory does not exist, the command still succeeds because of the 2>/dev/null || true; the archive will contain only config.toml.

After creating the archive, generate a checksum so you can verify integrity later:

sha256sum gitlab-runner-config-20250115.tar.gz > gitlab-runner-config-20250115.tar.gz.sha256

Store the checksum file alongside the archive and verify it before restoring.

Backing Up the Configuration File (Docker Container)

If the runner runs in a Docker container, extract config.toml from the running container:

docker cp <container_name>:/etc/gitlab-runner/config.toml ./config.toml

Replace <container_name> with the actual name or ID of the container, e.g., gitlab-runner. Expected output: a local file config.toml in the current directory with no error message. You should then archive this file along with any mounted volumes or other relevant data. If the container was started with a volume mount for the config, such as -v /srv/gitlab-runner/config:/etc/gitlab-runner, you can back up the host directory directly instead of using docker cp.

Backing Up Docker Volumes

Runners using the Docker executor often store cache and build artifacts in named Docker volumes. To back up a volume, first identify the volumes in use:

docker volume ls

Example output:

DRIVER    VOLUME NAME
local     runner-cache
local     runner-artifacts

Back up each volume by running a temporary Alpine container that mounts the volume and a host directory, then creates a tar archive:

docker run --rm -v runner-cache:/data -v $(pwd):/backup alpine \
  tar czf /backup/runner-cache.tar.gz -C /data .

Expected output: a file runner-cache.tar.gz in the current directory. This archive contains all files from the volume. Repeat for each volume, replacing runner-cache with the appropriate name. For large volumes, you may want to exclude certain directories (e.g., .cache) to reduce backup size; you can do so by adding tar options like --exclude='.cache' before the -C flag. After the backup, create a checksum for each archive and store it securely.

Backing Up Kubernetes Manifests and Data

For GitLab Runner deployed on Kubernetes, the primary configuration is in the Helm values file and any custom resources you created. The runner's runtime data may be stored in PersistentVolumeClaims (PVCs) for cache and artifacts. Back up the Helm release values with:

helm get values <release-name> -n <namespace> > values-backup.yaml

For example, if the release is named gitlab-runner in the namespace gitlab, run:

helm get values gitlab-runner -n gitlab > values-backup.yaml

Expected output: a YAML file values-backup.yaml containing all custom values used to install or upgrade the release. This file preserves your non-default settings.

To back up the runtime Kubernetes resources (ConfigMaps, Secrets, Deployments, and Pods), run:

kubectl get configmap, secret, deployment, pod -n <namespace> -o yaml > k8s-resources-backup.yaml

Example:

kubectl get configmap, secret, deployment, pod -n gitlab -o yaml > k8s-resources-backup.yaml

Expected output: a YAML file containing the definitions of all those resources in the namespace. Note that Secret resources contain base64-encoded data; when you restore them, they will be recreated exactly as they were. For PVC data, you need to back up the underlying storage volume. The method depends on your storage provider. If you use a cloud provider, take snapshots of the PVCs. If you use a local provisioner, you may need to copy data from the pod using kubectl cp or a similar tool. We cover PVC backup in more detail in the disaster recovery section.

Verification and Diagnostics

Backing up is only half the job. You must verify that backups are valid and restorable. Regular testing prevents the classic scenario of finding out during an emergency that your backup is empty, corrupted, or incomplete.

Verify Backup Files

Start by listing the contents of the archive to ensure it is not empty and contains the expected files. For a config archive:

tar -tzf gitlab-runner-config-20250115.tar.gz

Expected output: a list of files, including etc/gitlab-runner/config.toml. If the certs directory was included, those files appear as well. If the archive is empty or listing fails, the backup did not complete correctly; re-run the backup command.

For Docker volume backups:

tar -tzf runner-cache.tar.gz | head

Expected output: the first ten file paths from the volume. If the volume was empty, there will be no output; in that case, verify that the volume indeed had no data or that the mount was correct.

Always verify the checksum of the archive against the stored checksum:

sha256sum -c gitlab-runner-config-20250115.tar.gz.sha256

Expected output: gitlab-runner-config-20250115.tar.gz: OK. If the output says FAILED, the archive is corrupted or has been modified; create a fresh backup.

Test Restore in a Sandbox

Before relying on a backup, test the restore process in an isolated environment. For a binary runner config, extract the archive to a temporary directory and compare the restored file with the original:

mkdir /tmp/restore-test
tar -xzf gitlab-runner-config-20250115.tar.gz -C /tmp/restore-test
diff /etc/gitlab-runner/config.toml /tmp/restore-test/etc/gitlab-runner/config.toml

Expected output: no differences (empty diff output). If diff prints differences, investigate whether the backup was taken after the original changed, or if the extraction path is wrong.

For Docker volume backups, create a new volume, restore the archive into it, and then run a container to verify the files are present:

docker volume create restore-test-volume
docker run --rm -v restore-test-volume:/data -v $(pwd):/backup alpine \
  tar xzf /backup/runner-cache.tar.gz -C /data

Then run a temporary container using the volume and list files:

docker run --rm -v restore-test-volume:/data alpine ls -la /data

Expected output: a listing of files and directories that should match the original volume contents. If the listing is empty or incomplete, the backup did not capture the data correctly.

For Kubernetes backups, test by restoring to a separate namespace. Use the backup YAML files to recreate resources, but be careful with Secrets and ConfigMaps; they may need to be recreated with the same names before restoring Deployments. A full test of PVC restore depends on your storage provider's snapshot capability.

Failure Modes and Recovery

Backup and restore processes can fail for many reasons. Understanding common failure modes helps you plan mitigation and recover faster. The table below summarizes typical failures, symptoms, and mitigations. For each failure mode, the Recovery Owner is the person or role responsible for restoring service. In most organizations, this is the CI/CD infrastructure engineer or the on-call SRE. The recovery owner should document the incident, including the cause, recovery steps taken, and time to restore, within 24 hours of resolution.

Rollback Strategy

If a restore introduces problems, you need a way to roll back to the previous state. For configuration files, always keep the current file as a backup before overwriting:

sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak

Then restore the new file. If the runner fails to start or behaves incorrectly, revert:

sudo cp /etc/gitlab-runner/config.toml.bak /etc/gitlab-runner/config.toml
sudo gitlab-runner restart

Expected output: the runner restarts successfully (no error messages). Ensure the rollback file is from a known-good state; you may want to maintain timestamped copies of config.toml for a few days.

For Docker volumes, you can similarly keep a copy of the existing volume before restoring from backup. For example, create a backup of the current volume, then restore the previous backup if needed.

Disaster Recovery Steps

In a full disaster, follow these steps to bring a runner back from backups:

1. Restore the configuration file: Copy the backed-up config.toml to /etc/gitlab-runner/config.toml (or the appropriate path). Ensure ownership and permissions are correct:

   sudo cp /path/to/backup/config.toml /etc/gitlab-runner/config.toml
   sudo chown gitlab-runner:gitlab-runner /etc/gitlab-runner/config.toml
   sudo chmod 600 /etc/gitlab-runner/config.toml

1. Restore Docker volumes: Recreate the volume if needed and populate it from the backup archive:

   docker volume create runner-cache
   docker run --rm -v runner-cache:/data -v $(pwd):/backup alpine \
     tar xzf /backup/runner-cache.tar.gz -C /data

1. Reinstall the runner binary if needed: If the host was rebuilt, install GitLab Runner using the official repository or the install script:

   curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
   sudo apt-get install gitlab-runner

1. Verify runner registration: Run gitlab-runner verify to check that the runner is registered with the GitLab instance:

   sudo gitlab-runner verify

Expected output: messages indicating each configured runner is alive or needs attention. If the runner was registered with a token that is no longer valid, re-register using gitlab-runner register with the appropriate token from GitLab.

1. Test with a simple pipeline: Trigger a minimal CI job to confirm end-to-end functionality. Create a test project or use an existing one, and run a pipeline with a simple job, e.g., a job that runs echo. If the runner picks up the job and completes it successfully, your restore was successful.

Common Pitfalls and How to Avoid Them

Even with good intentions, teams often make mistakes in backup and restore processes. Here are frequent pitfalls and how to avoid them.

1. Not testing restores regularly. Backups can be corrupted or incomplete without anyone noticing. A restore test is the only way to guarantee recoverability.

• Why it happens: Backups run automatically, and no one checks them until an emergency.
• How to avoid: Schedule a monthly test restore (or more frequently for critical runners). Automate the test with a script and have the CI/CD engineer review the results.

1. Storing backups on the same disk as the runner. A disk failure destroys both the runner and its backups.

• Why it happens: Convenience; local backups are easy to set up.
• How to avoid: Use a separate physical disk, a network share, or cloud storage. At minimum, copy backups off the host regularly.

1. Forgetting to back up secrets and tokens. In Kubernetes, Secrets are often backed up as part of the resource YAML, but in binary or Docker setups, tokens may reside only in config.toml. Losing the config.toml means losing registration tokens.

• Why it happens: Tokens are not visible; they are hidden in configuration.
• How to avoid: Ensure config.toml is backed up, and also store the runner registration token securely in a password manager or secret store. For Kubernetes, ensure Secrets are included in your backup (or use an external secret management system).

1. Assuming Docker volumes are backed up when only the configuration is saved. The config.toml does not contain volume data. If you rely on caches stored in volumes, you need separate volume backups.

• Why it happens: Misunderstanding what config.toml contains.
• How to avoid: Document which runners have volumes and include them in the backup plan. Run automated volume backups at the same frequency as config backups.

1. Using a single monolithic backup script that fails silently. If the script fails partway, you might end up with incomplete backups but no error reported.

• Why it happens: Lack of error handling; the script runs and exits 0 even if some commands fail.
• How to avoid: Add set -e and explicit checks in your backup script. For each command, check the exit status and log errors. Send notifications on failure.

1. Not securing backup storage. Backup archives may contain sensitive information (tokens, SSH keys, build artifacts). If backups are stored unencrypted on a shared server, they can be compromised.

• Why it happens: Overlooking security of backups.
• How to avoid: Encrypt backups before sending to storage (e.g., using gpg or openssl). Restrict access to backup files and use access controls.

Operations Checklist

Use the following checklists to maintain a consistent backup routine. Assign a Backup Owner (typically the CI/CD engineer) who is responsible for executing these checklists and escalating any failures. The owner should review the checklist outcomes with the team during the weekly infrastructure meeting.

Daily Backup Checklist

Run these checks every day, preferably automated and logged:

• [ ] Backup script executed successfully (check log for errors).
• [ ] Backup files exist and have expected size (e.g., config archive > 1 KB, volume archive > 10 MB).
• [ ] Backup storage has sufficient free space (e.g., at least 20% free).
• [ ] Checksum files generated and match stored checksums.

Weekly Verification Checklist

Perform these tasks every week to ensure backups are usable:

• [ ] Test restore of configuration files to a sandbox and compare with live config.
• [ ] Validate Docker volume backups by restoring to a test volume and verifying file integrity.
• [ ] Check Kubernetes resource backup YAML for completeness (all expected resources present).
• [ ] Review backup logs for any anomalies and update documentation if needed.

Quarterly Disaster Recovery Drill

Conduct a full DR drill every quarter to test the complete restore process:

• [ ] Simulate a complete runner failure (e.g., shut down a test runner, delete its config and volumes).
• [ ] Restore from the latest backups using the documented steps.
• [ ] Verify runner registration with gitlab-runner verify.
• [ ] Trigger a test pipeline and confirm successful job execution.
• [ ] Record recovery time (target: under 30 minutes) and identify any issues or improvements.

Conclusion

A robust backup and restore strategy for GitLab Runner is not just about copying files; it is about ensuring that you can recover quickly and completely when things go wrong. This guide walked through the fundamentals: inventorying your runners, backing up configuration files, Docker volumes, and Kubernetes resources, verifying backups, and planning for failures. By implementing the practices and checklists outlined here, you can minimize downtime and data loss. Start small: pick one runner, create a backup, test the restore, and then expand to cover all runners. Measure your recovery time in drills and continuously improve. With a reliable backup and restore process, your CI/CD infrastructure remains resilient, and your development team can keep shipping without interruption.