## Intro

Operating GitLab Runner in production means more than installing a binary and hoping jobs complete. It requires a repeatable, observable workflow that moves from a detected problem to a verified resolution. Without a checklist, operators often skip read-only inspection, change configuration blindly, and struggle to recover when the expected state is not reached.

This article provides a practical operations checklist for GitLab Runner, aimed at developers, DevOps consultants, and technical startup teams who manage self-hosted CI/CD infrastructure. It connects core operational areas - version and environment inventory, safe configuration paths, verification and diagnostics, failure modes and recovery, and a consolidated operations checklist - to concrete commands, expected outputs, failure signals, and recovery decisions.

The guiding principle is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify after every change, and document how to roll back if the expected state is not reached. Every recommendation is version-scoped to GitLab Runner 15.x and 16.x, the widely deployed stable releases, unless otherwise noted.

## Version and Environment Inventory

Before touching any setting, know exactly what you are running and where. A version and environment inventory establishes the current state so that any subsequent change is informed and reversible.

Start by identifying the installed GitLab Runner version and the deployment topology. Run the read-only version command on the runner host:

gitlab-runner --version 
 Expected output example:

Version: 16.5.0
Git revision: 8f1ad6c1
Git branch: 16-5-stable
GO version: go1.20.7
Built: 2023-09-22T06:12:13+00:00
OS/Arch: linux/amd64 
 Compare this version against the GitLab instance version you are using. GitLab Runner follows a compatibility policy where the runner should ideally be within one minor version of the GitLab server, or at least not older than the oldest supported runner version. Check compatibility in the GitLab documentation for your specific GitLab release.

Next, determine how the runner is installed and managed. Use your package manager or systemd to see the unit status:

systemctl status gitlab-runner 
 Look for the Loaded: line to confirm the unit file location, and Active: to see whether the service is running. If the runner is deployed as a Docker container, inspect the container:

docker ps --filter "name=gitlab-runner" 
 Capture the runner's registration details without exposing secrets. The list command shows configured runners and their executors:

sudo gitlab-runner list 
 Output example:

Listing configured runners ConfigFile=/etc/gitlab-runner/config.toml
my-project-runner Executor=docker Token=glrt-xxxxxx URL=https://gitlab.example.com 
 Never print the full token in logs or articles. Treat any output containing Token= as sensitive and redact it before pasting into tickets or documentation.

Record the output of the following command to understand the host environment if the runner uses the shell executor:

uname -a && lscpu | head -n 5 
 This captures the kernel, architecture, and CPU model, which can matter for build performance and compatibility.

Create a small inventory table like the one below and keep it updated:

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Attribute</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Example Value</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Command to Verify</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Runner version</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">16.5.0</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">gitlab-runner --version</code></td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Executor type</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">docker</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">gitlab-runner list</code> (look at Executor column)</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Config file path</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">/etc/gitlab-runner/config.toml</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">systemctl status gitlab-runner</code> (Loaded line)</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Service status</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">active (running)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">systemctl status gitlab-runner</code></td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Host OS and kernel</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Ubuntu 22.04, 5.15.0-84-generic</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">uname -a</code> and <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">cat /etc/os-release</code></td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Docker version (if used)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">24.0.6</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">docker --version</code></td></tr></tbody>
</table>
</div>
Complete this inventory before any change. It will serve as a baseline for rollback and for diagnosing issues later.

## Safe Configuration Path

Changing GitLab Runner configuration can break all builds if done carelessly. The safe path involves validating syntax, backing up the current configuration, making a minimal change, and reloading the service with verification.

The central configuration file is /etc/gitlab-runner/config.toml . Before editing, create a timestamped backup:

sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak.$(date +%Y%m%d%H%M%S) 
 Then validate the current configuration by running the runner in debug mode without starting a job:

sudo gitlab-runner verify 
 This command loads the config and checks each runner against the GitLab instance. It does not execute a job, making it safe for validation. Expected output for a valid configuration:

Verifying runner... is alive 
 If you need to make a change, such as increasing the Docker pull policy or adjusting the concurrency, first open the config file:

sudo editor /etc/gitlab-runner/config.toml 
 Make one scoped change at a time. For example, to increase global concurrency from 1 to 2, locate the top-level concurrent field and change it:

concurrent = 2 
 Do not combine unrelated modifications in one edit; if something goes wrong, you need to know exactly which change caused it.

After editing, run verify again. If it reports an error, fix it before reloading the service. Do not restart the runner until the configuration passes validation.

To apply the change, reload the systemd service:

sudo systemctl reload gitlab-runner 
 Some configurations may require a full restart instead of reload. Check the service documentation. After reload, confirm the service is active and jobs are being accepted:

systemctl status gitlab-runner
sudo gitlab-runner list 
 If the runner is containerized, the process differs. For a Docker-deployed runner, edit the mounted config file on the host, then restart the container:

docker restart gitlab-runner 
 Always keep the backup file for at least a week. If a change causes failures, roll back by restoring the backup and restarting the service:

sudo cp /etc/gitlab-runner/config.toml.bak.20231025120000 /etc/gitlab-runner/config.toml
sudo systemctl restart gitlab-runner 

## Verification and Diagnostics

 After any change, verify that the runner behaves correctly under load. Diagnostics help isolate problems when jobs fail or hang.

Start with a single test job that exercises the runner end-to-end. Create a minimal .gitlab-ci.yml in a scratch project:

test_runner:
 script:
 - echo "Runner is working"
 - gitlab-runner --version 
 Trigger the pipeline and watch the job log. A successful job will show Job succeeded in the pipeline view. If the job is not picked up, check the runner's connection to GitLab by running:

sudo gitlab-runner verify 
 If the verification fails with an authentication error, the registration token may have changed or been revoked. Re-register the runner with a new token as a last resort.

For more detailed diagnostics, enable debug logging temporarily. Edit config.toml and set the log level:

log_level = "debug" 
 Reload the runner, run a test job, then inspect the logs:

sudo journalctl -u gitlab-runner -f 
 Look for lines containing Checking for jobs... and Job succeeded or specific error messages. Turn off debug logging after troubleshooting to avoid log bloat.

If jobs execute but fail, examine the job logs in GitLab UI. Common failure signals include:

- ERROR: Preparation failed: Error response from daemon: ... indicating a Docker image pull issue.

- ERROR: Job failed: exit code 1 meaning the script itself returned an error.

- ERROR: Failed to remove network for build suggesting a Docker networking problem.

Check the health of the Docker daemon if the executor is Docker:

docker info 
 Expected output includes Server Version: and a list of storage drivers. If Docker is unhealthy, restart it:

sudo systemctl restart docker 
 Then restart the runner service to re-establish connections.

## Failure Modes and Recovery

Understanding common failure modes and their recovery paths is critical for minimal downtime. The table below summarizes frequent issues.

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Failure Mode</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Symptom</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Likely Cause</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Recovery Step</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Verification</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Runner not picking up jobs</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Pipeline remains pending, no job log</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Runner offline or not connected</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Check <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">systemctl status gitlab-runner</code>, run <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">gitlab-runner verify</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Runner shows active and verify succeeds</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Authentication failure</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">ERROR: Failed to authorize</code> in verify</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Invalid or revoked token</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Re-register runner with new token</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">gitlab-runner verify</code> succeeds, jobs run</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Docker image pull failure</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">ERROR: Preparation failed: image pull failed</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Network issue or incorrect image name</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Test pull manually: <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">docker pull &lt;image&gt;</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Pull succeeds, rerun job</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Disk space full</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Jobs fail with &quot;No space left on device&quot;</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Build artifacts not cleaned</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Run <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">docker system prune -a --volumes</code> (be careful!)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">df -h</code> shows freed space, jobs succeed</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Runner process crash</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Service failed, logs show panic</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Bug or resource limit</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Restart service, upgrade runner version</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Service stable for 24 hours</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Slow build times</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Jobs take longer than expected</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Insufficient concurrency or resources</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Increase concurrency in config.toml, scale runner horizontally</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Job duration decreases</td></tr></tbody>
</table>
</div>
Always have a rollback plan for configuration or version changes. If an upgrade causes failures, downgrade to the previous version:

sudo apt-get install gitlab-runner=16.4.1 
 Then verify the version:

gitlab-runner --version 
 For severe production incidents, consider switching to a standby runner registered to the same project if available.

## Operations Checklist

Consolidate the above into a regular checklist that any operator can follow. Perform these tasks at the stated frequency.

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Task</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Frequency</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Owner</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Command / Action</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Expected Result</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Verify runner version and compatibility</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Monthly</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">CI/CD Engineer</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">gitlab-runner --version</code> and compare with GitLab version</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Version within supported range</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Check service status</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Daily (automated)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Monitoring system</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">systemctl status gitlab-runner</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Active (running)</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Review runner logs for errors</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Weekly</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">CI/CD Engineer</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">journalctl -u gitlab-runner --since &quot;7 days ago&quot;</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">No critical errors</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Validate configuration</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Before any change</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">CI/CD Engineer</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">gitlab-runner verify</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">All runners alive</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Backup config file</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Before any change</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">CI/CD Engineer</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">cp config.toml config.toml.bak.&lt;timestamp&gt;</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Backup file exists</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Test a minimal job</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">After any change</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">CI/CD Engineer</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Create/run test pipeline</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Job succeeds</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Clean up Docker resources</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Monthly</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">System Administrator</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">docker system prune --filter &quot;until=720h&quot;</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Freed disk space</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Review registered runners and tokens</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Quarterly</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Security Team</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">gitlab-runner list</code> and review access</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Only necessary runners present</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Update runner version</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Quarterly or per policy</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">CI/CD Engineer</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Follow upgrade documentation</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">New version running</td></tr></tbody>
</table>
</div>
Assign a single accountable owner per item to avoid diffusion of responsibility. For example, the CI/CD Engineer owns the version check and monthly cleanup, while the Security Team reviews runner access quarterly. Revisit this checklist every quarter to adjust frequencies based on incident history and infrastructure changes.

## Common Pitfalls

Several mistakes recur in GitLab Runner production operations. Here are the most common ones and how to avoid or recover from them.

Editing config.toml without backup Operators often edit the configuration directly and then cannot revert when something breaks. Always create a timestamped backup before editing, and keep backups for at least a week.

Ignoring version compatibility Using a runner version far older or newer than the GitLab instance can cause unexpected behavior. Check the compatibility matrix before upgrading either component. Upgrade the runner one minor version at a time if possible.

Leaving debug logging enabled Debug logging generates large log files and may expose sensitive information in plain text. Enable it only during active troubleshooting and disable it immediately after.

Running too many concurrent jobs on one host Setting concurrent too high for available resources leads to timeouts and OOM kills. Monitor CPU and memory usage. Increase concurrency gradually and test with representative workloads.

Not cleaning up Docker images and volumes Disk space fills up, causing build failures. Schedule a regular docker system prune with appropriate filters and monitor disk usage with df -h .

Treating the registration token as permanent Tokens can be revoked or rotated. If authentication fails, re-register the runner instead of trying to guess the token. Keep the registration token in a secure vault, not in plain text on disk.

## Conclusion

A GitLab Runner production operations checklist is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying commands without checking prerequisites and expected output is not an operations procedure.

Start with a low-risk verification: run gitlab-runner --version and gitlab-runner verify on your runner, record the current state, and compare the results with expected signals. Then move on to backing up your configuration and scheduling regular checks.

By following this checklist, you make failures visible, protect sensitive values, limit changes to the intended resource, and define recovery verification before an incident forces a decision. Reliable operations come from disciplined observation and incremental, verified changes.