GitLab Runner Configuration Mistakes: Diagnose, Fix, and Prevent Them

Intro

Misconfigured GitLab Runners are a leading cause of flaky pipelines, mysterious job failures, and wasted engineering hours. The problem is rarely obvious from GitLab's UI: a job that fails with exit code 137, a runner that never picks up tagged jobs, or an executor that silently uses the wrong Docker image. Each symptom points to a specific configuration mistake, and each mistake has a diagnostic command, a fix, and a way to prevent recurrence.

This article is for developers, DevOps engineers, and technical leads who manage self-hosted GitLab Runners. It is organized around observable symptoms rather than abstract settings. For each mistake, you will find the exact command or configuration to inspect, the expected output when things are correct, the failure signal that indicates a problem, and a recovery path that minimizes downtime.

The examples assume GitLab Runner 15.x or later on Linux, using the gitlab-runner binary and /etc/gitlab-runner/config.toml. Before making any change, run sudo gitlab-runner --version and sudo gitlab-runner list to capture your current version and registered runners. Always back up config.toml with sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak-$(date +%Y%m%d) before editing.

Version and Environment Inventory

A reliable troubleshooting session starts with a precise inventory of the runner version, operating system, executor, and registration state. Skipping this step leads to applying fixes for the wrong version or missing a mismatch between the runner and your GitLab instance.

Run these read-only commands on the runner host:

sudo gitlab-runner --version
sudo gitlab-runner list
sudo systemctl status gitlab-runner --no-pager

Expected output for gitlab-runner list includes a table with columns for Executor, Token, and URL. If you see ERROR: Failed to load config file or the runner list is empty while jobs are pending, the runner is not reading the expected config.toml.

The version command should return something like Version: 15.11.0 and Git revision: 5e9f5f4e. If you are using GitLab.com, your runner version should be within the supported version policy: the minor version should not be more than two releases behind the current GitLab minor version. For example, if your GitLab instance is 16.0, a runner version 15.9 is still supported, but 15.0 is too old and may miss security patches or features.

Capture the executor for each runner. A common mistake is assuming the executor is docker when it is actually shell. Run sudo gitlab-runner list and look at the Executor column. If it says shell, jobs run directly on the host and may have side effects like leftover processes or files. If it says docker, jobs run in containers defined by the image keyword in .gitlab-ci.yml or the runner's [[runners]] section.

Document the runner's architecture: uname -m should return x86_64, aarch64, etc. If you use Docker executor with multi-arch images, ensure the runner host and base images match. For example, a runner on an ARM64 host cannot run amd64 Docker images without emulation.

Finally, record the GitLab instance URL and registration token scope. A runner can be shared, group, or project specific. Run sudo gitlab-runner list and note the URL and Token columns (the token is partially masked). This classification affects which projects can use the runner and where you need to look for configuration conflicts.

Common Configuration Mistakes and Their Symptoms

Mistake 1: Wrong Executor Type

Symptom: Pipeline jobs fail with ERROR: Preparation failed: Cannot connect to the Docker daemon even though Docker is installed, or jobs behave inconsistently between environments.

Why it happens: The runner's executor in config.toml is set to docker, but the Docker daemon is not accessible, or the reverse: it is set to shell when the team expected container isolation.

Diagnosis:

sudo grep -A2 'executor' /etc/gitlab-runner/config.toml
sudo docker info

If executor = "docker" and docker info returns Cannot connect to the Docker daemon, the runner cannot start Docker containers. If executor = "shell" and you see host-level processes from previous jobs, you have no container isolation.

Fix: Edit config.toml with sudo nano /etc/gitlab-runner/config.toml and set the correct executor. For Docker:

[[runners]]
  name = "my-docker-runner"
  url = "https://gitlab.example.com/"
  token = "YOUR_TOKEN"
  executor = "docker"
  [runners.docker]
    image = "alpine:latest"
    privileged = false

For shell:

[[runners]]
  name = "my-shell-runner"
  url = "https://gitlab.example.com/"
  token = "YOUR_TOKEN"
  executor = "shell"

After editing, restart the runner: sudo gitlab-runner restart. Verify with sudo gitlab-runner verify and a test pipeline.

Mistake 2: Missing or Incorrect Tags

Symptom: Jobs wait indefinitely with This job is stuck because the project has no runners online that can run this job or are assigned to the wrong runner.

Why it happens: The job's tags in .gitlab-ci.yml do not match any runner's tags in config.toml, or the runner has no tags while the job requires a specific tag.

Diagnosis:

sudo grep -A5 'tags' /etc/gitlab-runner/config.toml
grep -A3 'tags:' .gitlab-ci.yml

Example config.toml runner tags:

[[runners]]
  name = "docker-builder"
  tags = ["docker", "linux", "amd64"]

Example job tags:

build:
  stage: build
  tags: ["docker", "linux"]
  script: echo "Building..."

The job runs only if all its tags are present on the runner. If the runner has tags = ["docker"] but the job also requires linux, the runner is not a match.

Fix: Align tags. If a job needs multiple capabilities, either add all tags to one runner or split into separate runners. Use the --tag-list option at registration: sudo gitlab-runner register --tag-list "docker,linux,amd64". Or edit config.toml and restart.

Prevention: Document tag conventions and use CI linting (.gitlab-ci.yml editor in GitLab UI) to catch missing tags before merge.

Mistake 3: Insufficient Disk Space for Docker Images

Symptom: Jobs fail with no space left on device or the runner logs show Error pulling image with disk-related errors. Builds slow down due to repeated image pulls.

Why it happens: The Docker executor caches images on the runner host, and over time the cache fills the disk. The runner does not automatically prune unused images unless configured.

Diagnosis:

df -h /var/lib/docker
sudo docker system df

If /var/lib/docker usage is above 80% or docker system df shows many unused images and containers, disk pressure is likely causing failures.

Fix: Clean up unused Docker resources immediately:

sudo docker system prune -af --volumes

Then set up periodic pruning. One approach is a cron job on the runner host:

# Run every Sunday at 3 AM
0 3 * * 0 docker system prune -af --volumes

Alternatively, configure the runner's pre_build_script or use GitLab's image cleanup policy if using Kubernetes executor. For Docker executor, consider using DOCKER_IMAGE_CLEANUP in the runner's config:

[runners.docker]
  image_cleanup = true

This setting removes images that are no longer used by any job after each build, but it can slow down subsequent jobs if they rely on cached layers. Carefully weigh the trade-off.

Mistake 4: Wrong Docker Image or Registry Authentication

Symptom: Jobs fail at the start with ERROR: Job failed: image pull failed: access denied or the wrong image tag is used, leading to unpredictable builds.

Why it happens: The image name in .gitlab-ci.yml or runner config does not match the intended registry, or the runner lacks credentials to pull from a private registry.

Diagnosis:

Inspect the job log for the exact pull command. The runner logs the image name and pull attempt. Also check the runner's Docker config for registry credentials:

sudo grep -A10 'runners.docker' /etc/gitlab-runner/config.toml
sudo cat /root/.docker/config.json   # if runner runs as root

If your project uses a private registry, the runner must authenticate. The config.toml should include a DOCKER_AUTH_CONFIG environment variable or use GitLab's CI_REGISTRY variables.

Fix: For private registries, configure the runner with the appropriate DOCKER_AUTH_CONFIG. Example in config.toml:

[[runners]]
  environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"registry.example.com\":{\"auth\":\"BASE64_ENCODED_CREDENTIALS\"}}}"]

For GitLab's own container registry, the built-in CI_REGISTRY_USER and CI_REGISTRY_PASSWORD usually suffice if the job uses docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY before pulling.

Always verify the image tag. If you specify image: ruby:latest and the latest tag changes unexpectedly, builds become non-reproducible. Pin versions: image: ruby:3.1.2.

Mistake 5: Ignoring Runner Concurrency Limits

Symptom: The runner host becomes overloaded, CPU and memory usage spike, and jobs time out. Or some jobs are never picked up because the runner is busy with long-running jobs.

Why it happens: The runner's concurrent setting (global) and limit for a specific runner are too high for the host's resources, or they are not set at all, allowing unlimited concurrent jobs.

Diagnosis:

Check the global concurrent value and runner-specific limits:

sudo grep -B2 -A3 'concurrent\|limit' /etc/gitlab-runner/config.toml

Also monitor host load during a pipeline spike: uptime, htop.

Fix: Set reasonable limits. For example, on a machine with 4 CPUs and 8 GB RAM, limit to 2 concurrent jobs for a Docker runner:

concurrent = 2
[[runners]]
  name = "docker-runner"
  limit = 2

Restart the runner and verify: sudo gitlab-runner restart && sudo gitlab-runner verify. Then observe CPU/memory during peak usage.

Prevention: Right-size runners based on workload. Use GitLab's runner metrics (if enabled) to track job queuing times and adjust concurrent and limit accordingly.

Safe Configuration Path

Changing a runner's configuration on a production system requires a methodical approach. A small mistake can take down all CI for a team. Follow these steps to minimize risk.

Step 1: Capture the Current State

Before editing, record the current config.toml and runner status:

sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak-$(date +%Y%m%d%H%M%S)
sudo gitlab-runner list > ~/runner-list-before.txt
sudo systemctl status gitlab-runner --no-pager > ~/runner-status-before.txt

Also note the GitLab version and runner version:

sudo gitlab-runner --version

Step 2: Make a Single Change in a Test Environment

If possible, apply the change first on a staging runner or a test instance. This is particularly important for executor changes or authentication settings. Use a non-production project to trigger a test pipeline.

Example of a safe edit using sed to replace an old image:

sudo sed -i 's/image = "ruby:2.7"/image = "ruby:3.1"/' /etc/gitlab-runner/config.toml

Then validate the syntax:

sudo gitlab-runner verify

Expected output if syntax is valid: Verifying runner... is alive for each runner. If you see ERROR: Invalid config file, do not proceed; restore the backup:

sudo cp /etc/gitlab-runner/config.toml.bak-$(ls -t /etc/gitlab-runner/config.toml.bak-* | head -1 | cut -d- -f2-) /etc/gitlab-runner/config.toml

Step 3: Apply in Production with a Rollback Plan

If the change passes verification on staging, apply it to the production runner:

1. Stop the runner service: sudo gitlab-runner stop (or sudo systemctl stop gitlab-runner).
2. Edit the config file with your change.
3. Start the runner: sudo gitlab-runner start.
4. Check status: sudo systemctl status gitlab-runner --no-pager. Look for active (running).
5. Trigger a test pipeline in a low-risk project.

If the pipeline fails or the runner does not come back, roll back immediately:

sudo gitlab-runner stop
sudo cp /etc/gitlab-runner/config.toml.bak-YYYYMMDDHHMMSS /etc/gitlab-runner/config.toml
sudo gitlab-runner start
sudo gitlab-runner verify

Replace YYYYMMDDHHMMSS with the backup file timestamp you created in Step 1.

Step 4: Verify and Document

After a successful change, record what was changed, why, and how to verify. Update your runbook or wiki. Confirm that the expected behavior is observed: jobs run, tags match, concurrency limits are respected, etc.

Verification and Diagnostics

Effective verification goes beyond checking that the runner is alive. You must confirm the runner behaves correctly under real job conditions. Here are key diagnostics.

Check Runner Registration and Health

sudo gitlab-runner verify

This command checks connectivity to the GitLab instance and validates the runner token. Expected output:

Runtime platform                                    arch=amd64 os=linux pid=12345 revision=5e9f5f4e version=15.11.0
Running in system-mode.

Verifying runner... is alive                        runner=AbC123

If a runner is not alive, check network connectivity: curl -v https://gitlab.example.com/api/v4/runners. Ensure the runner's token is still valid; if not, re-register the runner.

Test Job Execution with a Minimal Pipeline

Create a test project and add a .gitlab-ci.yml:

test-runner:
  script:
    - echo "Runner works"
    - uname -a

If this job succeeds, the runner can execute basic commands. To test Docker executor, add image: alpine and a Docker-specific command:

test-docker:
  image: alpine
  script:
    - docker info   # will fail if Docker is not available inside the container unless using Docker-in-Docker

The docker info command will only work if the runner is configured with Docker-in-Docker or privileged mode. A more reliable Docker test is:

test-docker:
  image: docker:20.10
  services:
    - docker:20.10-dind
  script:
    - docker info

If the job fails with Cannot connect to the Docker daemon, the Docker-in-Docker service is not running or the runner's privileged setting is false. For CI pipelines that need to build Docker images, set privileged = true in config.toml under [runners.docker] or use a CI variable to disable TLS.

Inspect Runner Logs for Errors

Logs often contain the root cause of job failures. On systemd systems:

sudo journalctl -u gitlab-runner -n 50 --no-pager

Look for lines containing ERROR, panic, or failed. For example, a permission error when accessing the Docker socket:

ERROR: Failed to create container: permission denied while trying to connect to the Docker daemon socket

This indicates the gitlab-runner user does not have access to /var/run/docker.sock. Fix by adding the user to the docker group:

sudo usermod -aG docker gitlab-runner
sudo systemctl restart gitlab-runner

Failure Modes and Recovery

Failure: Runner Offline or Not Picking Up Jobs

Possible causes: network issues, invalid token, runner service stopped, or runner run_untagged set to false while jobs have no tags.

Recovery steps:

1. Check service status: sudo systemctl status gitlab-runner.
2. Check runner registration: sudo gitlab-runner list.
3. Ensure the runner is assigned to the correct project/group and that the job's tags match.
4. If using a proxy, set HTTP_PROXY and HTTPS_PROXY in the runner's environment.
5. If the runner is online but not picking jobs, verify run_untagged in config.toml. For jobs without tags, the runner must have run_untagged = true:

[[runners]]
  run_untagged = true

Failure: Job Fails with Exit Code 137

This usually means the job was killed due to out-of-memory (OOM). The runner's container exceeded memory limits, or the host ran out of memory.

Diagnosis: Check the host's memory usage: free -h. Inspect the container's memory limit: sudo docker inspect <container-id> | grep Memory. If you did not set a memory limit, consider setting one in the runner's Docker config:

[runners.docker]
  memory = "2g"
  memory_swap = "2g"

Also review the job's resource usage in GitLab's UI (if monitoring is enabled).

Recovery: Increase the memory limit if the job legitimately needs more, or optimize the job to use less memory. Restart the failed job after making changes.

Failure: Cache or Artifacts Not Working

Symptoms: Jobs can't find cached dependencies, or artifacts are missing after the pipeline.

Diagnosis: Check the runner's cache configuration. For distributed cache, the runner's config.toml should have [runners.cache] settings pointing to an S3 bucket or GCS. If using local cache, it is stored under the runner's build directory, typically /home/gitlab-runner/cache.

Common mistakes:

• The cache key is not consistent across jobs (cache:key missing or changing).
• The runner does not have access to the cache storage.
• Artifacts path in .gitlab-ci.yml is incorrect.

Fix: Define a clear cache key in .gitlab-ci.yml:

cache:
  key: "$CI_COMMIT_REF_SLUG"
  paths:
    - .m2/repository

For artifacts, ensure the artifacts:paths are relative to the build directory and the job exits successfully.

Operations Checklist

Use this checklist before and after any runner configuration change. Each item has an owner and a review frequency.

Assign single accountable owners, not groups. For example, "DevOps Engineer" should be a named individual like Priya Shah. Review frequencies are minimums; in fast-changing environments, review more often.

Conclusion

GitLab Runner misconfigurations can be costly, but they are almost always preventable with systematic diagnosis and safe change practices. The key is to observe before altering, make one change at a time, verify with concrete commands, and have a rollback plan. Document the current version and environment, align executor and tags with job requirements, manage disk and memory limits, and regularly verify runner health.

For a low-risk next step, choose one verification from the Operations Checklist, run it on your runner, and compare the output to the expected result. If you find a mismatch, apply the corresponding fix from the relevant section, then trigger a test pipeline to confirm. Over time, integrate these checks into your routine maintenance and monitoring to keep your CI/CD pipelines reliable.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.