GitLab Runner troubleshooting with practical examples: practical implementation guide

When a GitLab Runner stops picking up jobs, fails a checkout, or times out pushing artifacts, the pressure is real: development slows, and deployment windows slip. This guide gives you a reliable, low-risk way to diagnose and fix common GitLab Runner issues using practical command examples, scoped configuration changes, and reversible steps. You will inventory your environment, choose a safe configuration path, verify behavior with expected results, and resolve frequent failure modes. The goal is to move from confusion to repeatable, measured troubleshooting you can trust.

Version and Environment Inventory

Before changing anything, capture a simple inventory. This keeps your work measurable and makes rollback safer.

Prerequisites

• Shell access to the runner host(s)
• Permissions to view system logs and edit runner configuration files
• Access to GitLab project and group settings (to view runners, tokens, and job logs)

Minimum version info to capture

• GitLab Runner version and installation method
• GitLab server version (self-managed or SaaS)
• Executor type per runner: shell, Docker, Kubernetes, or others
• Operating system and init system (systemd, upstart, Windows services)
• Network constraints: proxies, firewalls, private registries, DNS

Commands and locations

Linux/macOS:

• Show runner version:

  gitlab-runner --version

Windows (PowerShell):

  gitlab-runner --version

• List configured runners on the host:

  sudo gitlab-runner list

Expected: runner names with short token hashes and executor types.

• Find configuration file:
• Linux default: /etc/gitlab-runner/config.toml
• Windows default: C:\GitLab-Runner\config.toml

• Service status (Linux systemd):

  systemctl status gitlab-runner

• Logs (Linux with systemd):

  journalctl -u gitlab-runner -n 200 --no-pager

Constructed example: Inventory snapshot

• GitLab Runner 16.7, package install on Ubuntu 22.04
• Two runners: shell and Docker executors
• GitLab.com SaaS
• Outbound via corporate proxy
• systemd service active for 5 days

Why this matters

Changes are easier to reason about when you can compare behavior before and after. Keep this inventory with timestamps so you can roll back precisely if necessary.

Safe Configuration Path

Troubleshooting should start with the smallest, safest change and expand only if required. Follow this path and back up before edits.

Safety steps

1. Backup runner configuration:

Linux:

   sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak.$(date +%Y%m%d%H%M%S)
   sudo chmod 600 /etc/gitlab-runner/config.toml.bak.*

Windows (PowerShell):

   Copy-Item C:\GitLab-Runner\config.toml C:\GitLab-Runner\config.toml.bak.$(Get-Date -Format yyyyMMddHHmmss)

1. Avoid global changes. Edit only the specific runner section in config.toml.
2. Change one thing at a time. After each change, restart the service and verify.
3. Keep a simple change log: timestamp, what changed, why, expected effect.

Scoped configuration examples

In /etc/gitlab-runner/config.toml (constructed example), set a conservative limit first:

• Adjust concurrency safely:

  concurrent = 2
  check_interval = 0

Expected effect: at most 2 concurrent jobs across all runners on this host.

• Start runner in debug mode interactively to avoid persistent verbosity:

  sudo gitlab-runner run --debug

Press Ctrl+C when done. For persistent debug logging, prefer system journal filters over changing runner log level globally.

• Set environment variables in the runner section only, not system-wide:

Network proxy configuration (constructed example):

  runners
    name = "shell-runner"
    executor = "shell"
    environment = [
      "HTTPS_PROXY=http://proxy.corp.example:8080",
      "NO_PROXY=localhost,127.0.0.1,.svc,.local"
    ]

Expected effect: only this runner uses the proxy; localhost and cluster-local addresses bypass it.

Restart safely

• Linux systemd:

  sudo systemctl restart gitlab-runner
  sudo systemctl is-active gitlab-runner

• Windows:

  Restart-Service gitlab-runner
  Get-Service gitlab-runner

Verification and Diagnostics

Verification means producing an observable, expected result. Start narrow and local, then expand.

Basic health checks

• Runner can reach GitLab:

  gitlab-runner verify --delete

Expected: lists registered runners and reports whether they are reachable. Use cautiously; the --delete flag removes unreachable runner references from local cache, not from GitLab.

• Service starts cleanly:

  systemctl status gitlab-runner
  journalctl -u gitlab-runner -n 50 --no-pager

Expected: no crash loops, no permission denied on config or cache paths.

• In the GitLab UI, confirm the runner appears as online. If not, re-register safely:

Token validity:

  sudo gitlab-runner register

Use a non-admin registration token scoped to the group or project. Expected: runner receives a unique token and appears online.

• File system sanity (Linux):

  df -h
  getent passwd gitlab-runner || id gitlab-runner
  sudo -u gitlab-runner touch /tmp/runner-perm-check

Expected: sufficient free space; the gitlab-runner user exists and can create files in temp.

• DNS and network reachability:

  getent hosts gitlab.com || nslookup gitlab.com
  curl -sI https://gitlab.com/.well-known/security.txt | head -n 1

Expected: DNS resolves; HTTP 200/301/302 status line appears.

Executor-specific spot checks

• Shell executor: Ensure required tools are on PATH for the service user.

  sudo -u gitlab-runner which git bash sh

Expected: paths resolve. If not, add to PATH in the runner environment.

• Docker executor (constructed example): Confirm the host can contact the registry.

  docker info
  docker login REGISTRY_URL
  docker pull alpine:3.18

Expected: no TLS or auth errors. If using a self-signed cert, place it in the engine trust store as documented for your OS.

• Kubernetes executor (constructed example):

  kubectl get nodes
  kubectl auth can-i create pods --as system:serviceaccount:runner-namespace:runner-sa

Expected: the service account used by the runner can create pods; nodes are Ready.

Log patterns to look for

• authentication: invalid token, 403, runner not authorized
• network: context deadline exceeded, i/o timeout, TLS handshake timeout
• executor: not found, permission denied, device or resource busy
• artifacts/cache: 403 during upload, invalid credentials, bucket not found

Constructed example: Job stuck at pending

• Observed: job is pending for 10+ minutes. Runner is offline in the UI.
• Checks: service status shows crash loop; logs show permission denied on /var/lib/gitlab-runner.
• Fix: chown the directory to the runner user and restart.

  sudo chown -R gitlab-runner:gitlab-runner /var/lib/gitlab-runner
  sudo systemctl restart gitlab-runner

Expected: runner appears online; pending jobs start within concurrency limits.

Failure Modes and Recovery

Use the table to quickly map symptom to first checks and actions. Then follow the detailed workflows below.

Constructed examples are labeled below.

1. Runner offline or unauthorized

Checks:

• UI: runner status is offline
• Host: service running but logs show unauthorized or invalid token

Recovery steps:

• Re-register with a scoped token:

  sudo gitlab-runner register

Choose the correct GitLab URL and enter the registration token from the project or group. Assign tags that match your jobs.

• Confirm the runner name and tags in config.toml match expectations.
• Restart the service and verify online status.

Rollback: If the new registration created a duplicate, disable the old runner in the UI and remove its section from config.toml after confirming no jobs use it.

2. Jobs stuck in pending due to tag or concurrency mismatch (constructed example)

Checks:

• Job requires tags: linux, shell
• Runner tags: only linux
• concurrent = 0 in config

Recovery steps:

• Add missing tag or remove unnecessary job tag requirement.
• Set a safe concurrency:

  concurrent = 2

• Restart and watch the queue.
• Expected: jobs begin within a minute; queue depth decreases.

Rollback: If high load causes resource contention, reduce concurrent to 1 temporarily.

3. Checkout or Git authentication failures

Symptoms:

• fatal: could not read Username for https://... No such device or address
• Host key verification failed

Checks:

• Verify SSH keys or HTTPS credentials for the runner user.
• For SSH: populate known_hosts for the runner user:

  sudo -u gitlab-runner ssh-keyscan -t rsa gitlab.com >> ~gitlab-runner/.ssh/known_hosts
  sudo chmod 600 ~gitlab-runner/.ssh/known_hosts

Recovery steps:

• If using HTTPS with a token, ensure the variable is masked and available to the job.
• Validate the URL and credentials by cloning manually as the runner user.

Rollback: Revert recent credential variable changes if the last known good state worked.

4. Artifacts and cache upload/download errors (constructed example)

Symptoms:

• 403 during upload to artifacts endpoint or S3-compatible store
• Cache always re-computes

Checks:

• Validate project or group settings for artifacts and cache.
• Inspect cache key used by the job; ensure it is stable and not overwritten each run unnecessarily.

Recovery steps:

• Set explicit and stable cache keys in the job definition to avoid cache misses:

  cache:
    key: "deps-v1"
    paths:
      - vendor/

• For external cache stores, export credentials via runner environment rather than hardcoding in scripts.

Rollback: If changing the cache backend, keep the previous backend configured until the new one has successful hits for multiple jobs.

5. Network and proxy problems

Symptoms:

• i/o timeout to GitLab or registries
• TLS handshake timeout behind proxy

Checks:

• curl to GitLab endpoints with and without proxy
• NO_PROXY includes internal hosts and 127.0.0.1

Recovery steps:

• Set HTTPS_PROXY and NO_PROXY only for affected runner(s) in config.toml.
• For registries with self-signed certs, trust the CA on the host.

Rollback: Remove proxy environment entries if they break internal connectivity.

6. Docker executor image pull failures (constructed example)

Symptoms: error response from daemon: pull access denied

Checks:

• docker login status and credentials scope
• Proxy interfering with docker pull

Recovery steps:

• docker login REGISTRY_URL with a read-only robot account.
• If private CA, install it in the Docker engine trust store and restart the engine.

Rollback: Revert to previously working base image tag to reduce moving parts.

7. Kubernetes executor pod lifecycle issues (constructed example)

Symptoms: pods Pending or ImagePullBackOff

Checks:

• kubectl get events -n runner-namespace
• RBAC: kubectl auth can-i create pods ...
• ImagePullSecrets present on the runner ServiceAccount

Recovery steps:

• Grant minimal RBAC for pods, secrets (read if needed), and events.
• Attach the correct imagePullSecret to the ServiceAccount.

Rollback: Scale runner replicas to 0, revert recent RBAC changes, and scale back up after validation.

8. Permissions and filesystem ownership

Symptoms: permission denied creating cache or artifacts directory

Checks:

• id gitlab-runner, ls -l target directories
• SELinux enforcing blocks (Linux):

  getenforce
  ausearch -m avc -ts recent

Recovery steps:

• chown directories to gitlab-runner user.
• For SELinux, set proper contexts (constructed example):

  sudo chcon -Rt svirt_sandbox_file_t /var/lib/gitlab-runner

Prefer permanent policy adjustments via semanage fcontext if available.

Rollback: Revert ownership or context changes if they affect other services; restore from the config backup as needed.

Operations Checklist

Use this concise checklist for ongoing operations and quick incident response.

Daily or pre-deploy quick checks

• Confirm runner online status in GitLab and note queue depth.
• systemctl status gitlab-runner shows running; no recent crash loops.
• Disk space above a safe threshold (constructed example: > 20%).

When a job stalls or fails unexpectedly

• Capture context: job ID, runner name, executor, recent changes.
• Verify tags and concurrent settings.
• Inspect last 200 log lines: journalctl -u gitlab-runner -n 200 --no-pager.
• Re-run a single job with minimal variables to isolate.

Change management

• Back up config.toml with timestamp.
• Edit only one variable at a time; record expected outcome.
• Restart runner and verify within 5 minutes.
• If outcome diverges, roll back immediately using the backup.

Network and credentials hygiene

• Validate DNS resolution and proxy env for runner users.
• Rotate credentials on a schedule; test new creds with a single runner first.

Capacity and performance

• Track average job wait time and success rate.
• Increase concurrent cautiously (step by 1) and observe CPU, memory, and I/O.

Executor-specific

• Shell: ensure required build tools and shells are installed for the runner user.
• Docker: verify engine health, registry login, and CA trust.
• Kubernetes: confirm node capacity, RBAC, and image pull secrets.

Rollback and recovery

• Keep at least one known-good runner untouched during major changes.
• Maintain a playbook for re-registering a runner with correct tags and executors.
• If a change causes widespread failures, restore config.toml from backup and restart.

Conclusion

Troubleshooting GitLab Runner becomes predictable when you start with a clear inventory, make one small change at a time, and verify each step against expected results. Use the safe configuration path, lean on targeted logs and health checks, and prefer reversible changes. The symptom-to-action table and the operations checklist give you fast paths to common fixes and safe rollbacks. By treating each change as an experiment with a rollback plan, you reduce the blast radius of mistakes and build confidence that your CI/CD pipeline will stay reliable under pressure.

Constructed next steps

• Pick one runner and run the Verification and Diagnostics section end to end. Timebox to 20 minutes and record findings.
• Apply one improvement from the Failure Modes and Recovery section that addresses your top symptom.
• Monitor for one day: queue depth, job success rate, and mean job start latency. If stable or improved, propagate to additional runners gradually.

Focused, measurable steps make troubleshooting faster and safer. Start narrow, verify locally, then expand with confidence.