GitLab Runner Production Operations Checklist with Practical Examples

Intro

Operating GitLab Runner in production means more than installing a binary and hoping jobs complete. It requires a repeatable, observable workflow that moves from a detected problem to a verified resolution. Without a checklist, operators often skip read-only inspection, change configuration blindly, and struggle to recover when the expected state is not reached.

This article provides a practical operations checklist for GitLab Runner, aimed at developers, DevOps consultants, and technical startup teams who manage self-hosted CI/CD infrastructure. It connects core operational areas - version and environment inventory, safe configuration paths, verification and diagnostics, failure modes and recovery, and a consolidated operations checklist - to concrete commands, expected outputs, failure signals, and recovery decisions.

The guiding principle is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify after every change, and document how to roll back if the expected state is not reached. Every recommendation is version-scoped to GitLab Runner 15.x and 16.x, the widely deployed stable releases, unless otherwise noted.

Version and Environment Inventory

Before touching any setting, know exactly what you are running and where. A version and environment inventory establishes the current state so that any subsequent change is informed and reversible.

Start by identifying the installed GitLab Runner version and the deployment topology. Run the read-only version command on the runner host:

gitlab-runner --version

Expected output example:

Version:      16.5.0
Git revision: 8f1ad6c1
Git branch:   16-5-stable
GO version:   go1.20.7
Built:        2023-09-22T06:12:13+00:00
OS/Arch:      linux/amd64

Compare this version against the GitLab instance version you are using. GitLab Runner follows a compatibility policy where the runner should ideally be within one minor version of the GitLab server, or at least not older than the oldest supported runner version. Check compatibility in the GitLab documentation for your specific GitLab release.

Next, determine how the runner is installed and managed. Use your package manager or systemd to see the unit status:

systemctl status gitlab-runner

Look for the Loaded: line to confirm the unit file location, and Active: to see whether the service is running. If the runner is deployed as a Docker container, inspect the container:

docker ps --filter "name=gitlab-runner"

Capture the runner's registration details without exposing secrets. The list command shows configured runners and their executors:

sudo gitlab-runner list

Output example:

Listing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml
my-project-runner                                  Executor=docker Token=glrt-xxxxxx URL=https://gitlab.example.com

Never print the full token in logs or articles. Treat any output containing Token= as sensitive and redact it before pasting into tickets or documentation.

Record the output of the following command to understand the host environment if the runner uses the shell executor:

uname -a && lscpu | head -n 5

This captures the kernel, architecture, and CPU model, which can matter for build performance and compatibility.

Create a small inventory table like the one below and keep it updated:

Complete this inventory before any change. It will serve as a baseline for rollback and for diagnosing issues later.

Safe Configuration Path

Changing GitLab Runner configuration can break all builds if done carelessly. The safe path involves validating syntax, backing up the current configuration, making a minimal change, and reloading the service with verification.

The central configuration file is /etc/gitlab-runner/config.toml. Before editing, create a timestamped backup:

sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak.$(date +%Y%m%d%H%M%S)

Then validate the current configuration by running the runner in debug mode without starting a job:

sudo gitlab-runner verify

This command loads the config and checks each runner against the GitLab instance. It does not execute a job, making it safe for validation. Expected output for a valid configuration:

Verifying runner... is alive

If you need to make a change, such as increasing the Docker pull policy or adjusting the concurrency, first open the config file:

sudo editor /etc/gitlab-runner/config.toml

Make one scoped change at a time. For example, to increase global concurrency from 1 to 2, locate the top-level concurrent field and change it:

concurrent = 2

Do not combine unrelated modifications in one edit; if something goes wrong, you need to know exactly which change caused it.

After editing, run verify again. If it reports an error, fix it before reloading the service. Do not restart the runner until the configuration passes validation.

To apply the change, reload the systemd service:

sudo systemctl reload gitlab-runner

Some configurations may require a full restart instead of reload. Check the service documentation. After reload, confirm the service is active and jobs are being accepted:

systemctl status gitlab-runner
sudo gitlab-runner list

If the runner is containerized, the process differs. For a Docker-deployed runner, edit the mounted config file on the host, then restart the container:

docker restart gitlab-runner

Always keep the backup file for at least a week. If a change causes failures, roll back by restoring the backup and restarting the service:

sudo cp /etc/gitlab-runner/config.toml.bak.20231025120000 /etc/gitlab-runner/config.toml
sudo systemctl restart gitlab-runner

Verification and Diagnostics

After any change, verify that the runner behaves correctly under load. Diagnostics help isolate problems when jobs fail or hang.

Start with a single test job that exercises the runner end-to-end. Create a minimal .gitlab-ci.yml in a scratch project:

test_runner:
  script:
    - echo "Runner is working"
    - gitlab-runner --version

Trigger the pipeline and watch the job log. A successful job will show Job succeeded in the pipeline view. If the job is not picked up, check the runner's connection to GitLab by running:

sudo gitlab-runner verify

If the verification fails with an authentication error, the registration token may have changed or been revoked. Re-register the runner with a new token as a last resort.

For more detailed diagnostics, enable debug logging temporarily. Edit config.toml and set the log level:

log_level = "debug"

Reload the runner, run a test job, then inspect the logs:

sudo journalctl -u gitlab-runner -f

Look for lines containing Checking for jobs... and Job succeeded or specific error messages. Turn off debug logging after troubleshooting to avoid log bloat.

If jobs execute but fail, examine the job logs in GitLab UI. Common failure signals include:

• ERROR: Preparation failed: Error response from daemon: ... indicating a Docker image pull issue.
• ERROR: Job failed: exit code 1 meaning the script itself returned an error.
• ERROR: Failed to remove network for build suggesting a Docker networking problem.

Check the health of the Docker daemon if the executor is Docker:

docker info

Expected output includes Server Version: and a list of storage drivers. If Docker is unhealthy, restart it:

sudo systemctl restart docker

Then restart the runner service to re-establish connections.

Failure Modes and Recovery

Understanding common failure modes and their recovery paths is critical for minimal downtime. The table below summarizes frequent issues.

Always have a rollback plan for configuration or version changes. If an upgrade causes failures, downgrade to the previous version:

sudo apt-get install gitlab-runner=16.4.1

Then verify the version:

gitlab-runner --version

For severe production incidents, consider switching to a standby runner registered to the same project if available.

Operations Checklist

Consolidate the above into a regular checklist that any operator can follow. Perform these tasks at the stated frequency.

Assign a single accountable owner per item to avoid diffusion of responsibility. For example, the CI/CD Engineer owns the version check and monthly cleanup, while the Security Team reviews runner access quarterly. Revisit this checklist every quarter to adjust frequencies based on incident history and infrastructure changes.

Common Pitfalls

Several mistakes recur in GitLab Runner production operations. Here are the most common ones and how to avoid or recover from them.

Editing config.toml without backup Operators often edit the configuration directly and then cannot revert when something breaks. Always create a timestamped backup before editing, and keep backups for at least a week.

Ignoring version compatibility Using a runner version far older or newer than the GitLab instance can cause unexpected behavior. Check the compatibility matrix before upgrading either component. Upgrade the runner one minor version at a time if possible.

Leaving debug logging enabled Debug logging generates large log files and may expose sensitive information in plain text. Enable it only during active troubleshooting and disable it immediately after.

Running too many concurrent jobs on one host Setting concurrent too high for available resources leads to timeouts and OOM kills. Monitor CPU and memory usage. Increase concurrency gradually and test with representative workloads.

Not cleaning up Docker images and volumes Disk space fills up, causing build failures. Schedule a regular docker system prune with appropriate filters and monitor disk usage with df -h.

Treating the registration token as permanent Tokens can be revoked or rotated. If authentication fails, re-register the runner instead of trying to guess the token. Keep the registration token in a secure vault, not in plain text on disk.

Conclusion

A GitLab Runner production operations checklist is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying commands without checking prerequisites and expected output is not an operations procedure.

Start with a low-risk verification: run gitlab-runner --version and gitlab-runner verify on your runner, record the current state, and compare the results with expected signals. Then move on to backing up your configuration and scheduling regular checks.

By following this checklist, you make failures visible, protect sensitive values, limit changes to the intended resource, and define recovery verification before an incident forces a decision. Reliable operations come from disciplined observation and incremental, verified changes.