GitLab Runner Upgrade and Migration with Practical Examples

Intro

Upgrading or migrating GitLab Runner is rarely a single command. It is a sequence of observations, scoped changes, and verifications that protect your CI/CD pipeline from avoidable failures. Whether you run runners on Docker, Kubernetes, or a plain Linux host, the same operational discipline applies: know your current version, understand your deployment topology, prepare a rollback path, and verify every step against an expected signal.

This guide is written for developers, DevOps consultants, and technical startup teams who need a repeatable method for moving GitLab Runner from one version to another or from one execution environment to another. It connects version upgrades, configuration changes, migration steps, rollback procedures, and post-change validation to concrete commands, expected output, and recovery decisions.

The examples use placeholders like RUNNER_TOKEN and gitlab.example.com. Replace them with your own values, but never put real credentials, tokens, or private keys into documentation or scripts that might be shared.

Version and Environment Inventory

Before you change anything, record what you have. The goal of this phase is to create a known starting point. If something breaks later, you can compare against this snapshot and decide whether to roll back or fix forward.

Identify the Installed Version

Run the version command on the runner host or inside the runner container. For a Docker-based runner, use:

docker exec -it gitlab-runner gitlab-runner --version

Expected output looks like:

Version:      16.10.0
Git revision: 1a2b3c4d
Git branch:   16-10-stable
GO version:   go1.21.5
Built:        2024-04-18T12:00:00+0000
OS/Arch:      linux/amd64

For a Kubernetes deployment, locate the runner pod and check the version:

kubectl get pods -n gitlab-runners
kubectl exec -n gitlab-runners <runner-pod-name> -- gitlab-runner --version

Note the version number somewhere durable. This is your baseline for the upgrade path and your rollback target if needed.

Document the Deployment Topology

GitLab Runner can run as a shell executor on a VM, as a Docker executor on a host, or as a Kubernetes executor inside a cluster. Each has different upgrade mechanics.

• Shell executor: The runner binary is installed directly on the host (often via package manager). Upgrade usually means apt upgrade gitlab-runner or yum update gitlab-runner.
• Docker executor: The runner itself runs in a container, and jobs run in separate containers. Upgrading the runner container is often as simple as pulling a new image and restarting.
• Kubernetes executor: The runner is deployed as pods in a cluster. Upgrade is managed through Helm chart updates or manifest changes.

Draw the topology before proceeding. It helps you understand the blast radius of each change.

Capture Current Configuration

Back up the runner configuration file before changing anything. For a Linux host, the configuration typically lives at /etc/gitlab-runner/config.toml. For Docker, you may mount it from a host volume. For Kubernetes, it is often stored in a ConfigMap or Secret.

# Linux host
sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak-$(date +%Y%m%d)

# Docker (if config is in a named volume)
docker run --rm -v gitlab-runner-config:/config alpine cat /config/config.toml > config.toml.bak

# Kubernetes
kubectl get configmap gitlab-runner -n gitlab-runners -o yaml > runner-configmap.yaml

Store these backups securely. They are your fastest rollback mechanism for configuration-only changes.

Prerequisites Check

Upgrading GitLab Runner may require a minimum GitLab instance version. Check the GitLab Runner compatibility chart to confirm your GitLab server supports the runner version you plan to install. If you run a self-managed GitLab, you can check the instance version:

# On the GitLab server (or via admin web UI)
sudo gitlab-rake gitlab:env:info

Look for GitLab version and compare it to the runner requirements. If your GitLab instance is older than the runner's minimum supported version, upgrade GitLab first or choose an older runner release.

Safe Configuration Path

Configuration changes carry hidden risks. A small typo in config.toml can prevent the runner from starting, which stops all CI jobs assigned to it. Follow a safe path: back up, change one thing at a time, verify, and be ready to restore.

Before Changing Configuration

Make a backup of config.toml as shown earlier. Also record the current runner status:

sudo gitlab-runner status

Expected output on a systemd-based host:

● gitlab-runner.service - GitLab Runner
   Loaded: loaded (/etc/systemd/system/gitlab-runner.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2024-04-23 10:00:00 UTC; 1h ago

Note whether the service is active and how long it has been running. A short uptime may indicate a recent crash or restart.

Making a Scoped Change

Suppose you need to change the Docker image used for jobs from ruby:3.2 to ruby:3.3. Open config.toml and locate the relevant [[runners]] section:

[[runners]]
  name = "shared-docker-runner"
  url = "https://gitlab.example.com/"
  token = "RUNNER_TOKEN"
  executor = "docker"
  [runners.docker]
    image = "ruby:3.2"

Change image to ruby:3.3, save, and restart the runner:

sudo gitlab-runner restart

Then verify the change took effect without breaking the runner:

sudo gitlab-runner verify

Expected output:

Verifying runner... is alive

If the verification fails, restore the backup immediately:

sudo cp /etc/gitlab-runner/config.toml.bak-YYYYMMDD /etc/gitlab-runner/config.toml
sudo gitlab-runner restart

Using Environment Variables Safely

Never hard-code secrets in config.toml if you can avoid it. GitLab Runner supports environment variables for many settings. For Docker, you can pass the registration token as an environment variable:

docker run -d --name gitlab-runner \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -e CI_SERVER_URL=https://gitlab.example.com/ \
  -e REGISTRATION_TOKEN=YOUR_TOKEN \
  gitlab/gitlab-runner:latest

For Kubernetes, use a Secret and reference it in the Helm values or deployment manifest.

Verification and Diagnostics

After any upgrade or migration, prove that the runner works. Verification goes beyond gitlab-runner verify; it should include a real job execution and careful inspection of logs.

Runner Registration Check

Confirm the runner is still registered with your GitLab instance. In the GitLab UI, go to Settings > CI/CD > Runners for the relevant project or group. The runner should appear with a green circle (online) or a gray circle (offline but registered). You can also use the API:

curl --header "PRIVATE-TOKEN: <your_access_token>" \
  "https://gitlab.example.com/api/v4/runners/all?scope=active"

Look for the runner's ID and "active": true. If the runner is missing or inactive, investigate the registration token and the runner's configuration.

Test Pipeline

Create a trivial .gitlab-ci.yml to run on the runner. For a Docker executor, a simple job like this works:

test-runner:
  script:
    - echo "Runner is alive"
    - docker --version

If the job succeeds, you have a basic level of confidence. For shell executor, the docker command may not exist, so adjust the script accordingly.

For a more thorough test, run a job that exercises the specific changes you made. If you upgraded the Docker image, use a command that checks the image version:

test-ruby-version:
  image: ruby:3.3
  script:
    - ruby -v

Expected output includes ruby 3.3.x.

Inspect Logs

Runner logs are essential for diagnosing problems. For systemd-based Linux:

sudo journalctl -u gitlab-runner -n 50 --no-pager

Look for errors like ERROR: Failed to start, Registering runner... failed, or executor-specific messages. For Docker:

docker logs gitlab-runner --tail 50

For Kubernetes:

kubectl logs -n gitlab-runners <runner-pod-name> --tail=50

If a job fails, check the job log in GitLab UI for the specific error. Common issues include missing Docker socket permissions, incompatible executor settings, or network timeouts.

Failure Modes and Recovery

Upgrades and migrations do not always go smoothly. Prepare for the most likely failure modes and know how to recover.

Runner Fails to Start After Upgrade

Symptom: gitlab-runner status shows failed or activating, and journalctl reports errors about missing libraries or incompatible configuration.

Cause: The new runner version may require a newer system library, or your config.toml contains deprecated settings.

Recovery: Restore the previous runner version if you used a package manager and kept the old package. On Debian/Ubuntu:

sudo apt install gitlab-runner=<old_version>

If you used Docker, pull the previous image tag:

docker pull gitlab/gitlab-runner:<old_version>
docker stop gitlab-runner
docker rm gitlab-runner
docker run -d --name gitlab-runner \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:<old_version>

Then restore your backed-up config.toml if needed.

Runner Not Picking Up Jobs

Symptom: Runner shows as online in GitLab UI, but jobs stay pending.

Cause: Runner tags do not match job tags, or the runner is not enabled for the project, or the runner is paused.

Recovery: Check the job tags in .gitlab-ci.yml and the runner's tag list. Verify the runner is enabled for the project: in GitLab UI, go to Settings > CI/CD > Runners, find the runner, and ensure the project is listed under Enable for this project. Use gitlab-runner list to see configured runners and their settings:

sudo gitlab-runner list

Look for Executor and Tags lines. If tags are missing, add them to config.toml and restart.

Docker Executor Permission Errors

Symptom: Jobs fail with permission denied when trying to access /var/run/docker.sock.

Cause: The runner user does not have permission to the Docker socket, often due to missing group membership.

Recovery: For a shell runner, add the gitlab-runner user to the docker group:

sudo usermod -aG docker gitlab-runner
sudo gitlab-runner restart

For Docker executor, ensure the runner container is started with -v /var/run/docker.sock:/var/run/docker.sock and that the socket permissions allow the container user.

Configuration Rollback

Always keep a known-good config.toml backup. If a change causes the runner to misbehave, restore the backup and restart:

sudo cp /etc/gitlab-runner/config.toml.good /etc/gitlab-runner/config.toml
sudo gitlab-runner restart

Then verify with gitlab-runner verify and a test pipeline.

Operations Checklist

Use this checklist for any upgrade or migration. Each item includes an owner (a specific person or role) and a review frequency.

Common Pitfalls and How to Avoid Them

Skipping the Version Compatibility Check

Pitfall: You upgrade the runner to a version that requires a newer GitLab server than you run. The runner may fail to register or behave unpredictably.

Why it happens: Excitement about new runner features or applying security patches without reading the release notes.

How to avoid: Before any upgrade, open the GitLab Runner releases page and confirm the minimum GitLab version. If your GitLab instance is older, upgrade GitLab first, or choose an older runner that supports your GitLab version.

Overwriting the Configuration File Without Backup

Pitfall: You edit config.toml directly, make a syntax error, and the runner fails to start. Without a backup, you must reconstruct the configuration from memory or from GitLab UI settings.

Why it happens: Admins often rely on version control for code but forget that runner configuration is not always versioned.

How to avoid: Make a timestamped backup before every change. Better yet, commit config.toml (with secrets redacted) to a private Git repository and use a config management tool.

Changing Multiple Variables at Once

Pitfall: You update the runner version, change the Docker image, and modify tags in a single maintenance window. A job fails, and you cannot tell which change caused it.

Why it happens: Trying to be efficient by batching changes.

How to avoid: Change one thing at a time. If you must batch, ensure you can apply and roll back each independently. In a Kubernetes deployment, use Helm rollback or kubectl to revert specific changes.

Ignoring Runner Tags and Project Enablement

Pitfall: After migration, the runner is online but never picks up jobs. You spend hours checking logs when the issue is simply that the job tags do not match or the runner is not enabled for the project.

Why it happens: Tags and enablement are easy to overlook when focusing on the technical execution.

How to avoid: After any migration, review the runner's tags and project enablement in the GitLab UI. Update config.toml if tags are missing, and explicitly enable the runner for the target projects.

Not Testing with a Non-Critical Pipeline First

Pitfall: You upgrade the runner and immediately route production CI jobs to it. A failure halts the team's delivery pipeline.

Why it happens: Pressure to get the upgrade done quickly or lack of a staging environment.

How to avoid: Run a test pipeline on a spare runner or a low-traffic project first. Gradually shift production traffic after the test succeeds. Many teams use runner tags like staging and production to control job routing.

Conclusion

The value of a structured approach to GitLab Runner upgrade and migration is not in the commands themselves but in the discipline of observing before changing, limiting the blast radius, verifying every step, and knowing how to recover. Copying a command without checking prerequisites and expected output is not an operations procedure; it is a guess.

As a next step, choose one low-risk verification from this guide, such as checking your current runner version and comparing it to the supported range. Record the current state, run the documented check, and review any dependencies like Docker or Kubernetes before making a larger change. Then apply the safe configuration path to a single scoped change and confirm the result with a test pipeline.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With that foundation, you can upgrade and migrate GitLab Runner with confidence.