When a GitLab Runner stops picking up jobs, fails a checkout, or times out pushing artifacts, the pressure is real: development slows, and deployment windows slip. This guide gives you a reliable, low-risk way to diagnose and fix common GitLab Runner issues using practical command examples, scoped configuration changes, and reversible steps. You will inventory your environment, choose a safe configuration path, verify behavior with expected results, and resolve frequent failure modes. The goal is to move from confusion to repeatable, measured troubleshooting you can trust. ## Version and Environment Inventory Before changing anything, capture a simple inventory. This keeps your work measurable and makes rollback safer. ### Prerequisites - Shell access to the runner host(s) - Permissions to view system logs and edit runner configuration files - Access to GitLab project and group settings (to view runners, tokens, and job logs) ### Minimum version info to capture - GitLab Runner version and installation method - GitLab server version (self-managed or SaaS) - Executor type per runner: shell, Docker, Kubernetes, or others - Operating system and init system (systemd, upstart, Windows services) - Network constraints: proxies, firewalls, private registries, DNS ### Commands and locations Linux/macOS: - Show runner version: gitlab-runner --version Windows (PowerShell): gitlab-runner --version - List configured runners on the host: sudo gitlab-runner list Expected: runner names with short token hashes and executor types. - Find configuration file: - Linux default: /etc/gitlab-runner/config.toml - Windows default: C:\GitLab-Runner\config.toml - Service status (Linux systemd): systemctl status gitlab-runner - Logs (Linux with systemd): journalctl -u gitlab-runner -n 200 --no-pager ### Constructed example: Inventory snapshot - GitLab Runner 16.7, package install on Ubuntu 22.04 - Two runners: shell and Docker executors - GitLab.com SaaS - Outbound via corporate proxy - systemd service active for 5 days ### Why this matters Changes are easier to reason about when you can compare behavior before and after. Keep this inventory with timestamps so you can roll back precisely if necessary. ## Safe Configuration Path Troubleshooting should start with the smallest, safest change and expand only if required. Follow this path and back up before edits. ### Safety steps - Backup runner configuration: Linux: sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak.$(date +%Y%m%d%H%M%S) sudo chmod 600 /etc/gitlab-runner/config.toml.bak.* Windows (PowerShell): Copy-Item C:\GitLab-Runner\config.toml C:\GitLab-Runner\config.toml.bak.$(Get-Date -Format yyyyMMddHHmmss) - Avoid global changes. Edit only the specific runner section in config.toml. - Change one thing at a time. After each change, restart the service and verify. - Keep a simple change log: timestamp, what changed, why, expected effect. ### Scoped configuration examples In /etc/gitlab-runner/config.toml (constructed example), set a conservative limit first: - Adjust concurrency safely: concurrent = 2 check_interval = 0 Expected effect: at most 2 concurrent jobs across all runners on this host. - Start runner in debug mode interactively to avoid persistent verbosity: sudo gitlab-runner run --debug Press Ctrl+C when done. For persistent debug logging, prefer system journal filters over changing runner log level globally. - Set environment variables in the runner section only, not system-wide: Network proxy configuration (constructed example): runners name = "shell-runner" executor = "shell" environment = [ "HTTPS_PROXY=http://proxy.corp.example:8080", "NO_PROXY=localhost,127.0.0.1,.svc,.local" ] Expected effect: only this runner uses the proxy; localhost and cluster-local addresses bypass it. ### Restart safely - Linux systemd: sudo systemctl restart gitlab-runner sudo systemctl is-active gitlab-runner - Windows: Restart-Service gitlab-runner Get-Service gitlab-runner ## Verification and Diagnostics Verification means producing an observable, expected result. Start narrow and local, then expand. ### Basic health checks - Runner can reach GitLab: gitlab-runner verify --delete Expected: lists registered runners and reports whether they are reachable. Use cautiously; the --delete flag removes unreachable runner references from local cache, not from GitLab. - Service starts cleanly: systemctl status gitlab-runner journalctl -u gitlab-runner -n 50 --no-pager Expected: no crash loops, no permission denied on config or cache paths. - In the GitLab UI, confirm the runner appears as online. If not, re-register safely: Token validity: sudo gitlab-runner register Use a non-admin registration token scoped to the group or project. Expected: runner receives a unique token and appears online. - File system sanity (Linux): df -h getent passwd gitlab-runner || id gitlab-runner sudo -u gitlab-runner touch /tmp/runner-perm-check Expected: sufficient free space; the gitlab-runner user exists and can create files in temp. - DNS and network reachability: getent hosts gitlab.com || nslookup gitlab.com curl -sI https://gitlab.com/.well-known/security.txt | head -n 1 Expected: DNS resolves; HTTP 200/301/302 status line appears. ### Executor-specific spot checks - Shell executor: Ensure required tools are on PATH for the service user. sudo -u gitlab-runner which git bash sh Expected: paths resolve. If not, add to PATH in the runner environment. - Docker executor (constructed example): Confirm the host can contact the registry. docker info docker login REGISTRY_URL docker pull alpine:3.18 Expected: no TLS or auth errors. If using a self-signed cert, place it in the engine trust store as documented for your OS. - Kubernetes executor (constructed example): kubectl get nodes kubectl auth can-i create pods --as system:serviceaccount:runner-namespace:runner-sa Expected: the service account used by the runner can create pods; nodes are Ready. ### Log patterns to look for - authentication: invalid token, 403, runner not authorized - network: context deadline exceeded, i/o timeout, TLS handshake timeout - executor: not found, permission denied, device or resource busy - artifacts/cache: 403 during upload, invalid credentials, bucket not found ### Constructed example: Job stuck at pending - Observed: job is pending for 10+ minutes. Runner is offline in the UI. - Checks: service status shows crash loop; logs show permission denied on /var/lib/gitlab-runner . - Fix: chown the directory to the runner user and restart. sudo chown -R gitlab-runner:gitlab-runner /var/lib/gitlab-runner sudo systemctl restart gitlab-runner Expected: runner appears online; pending jobs start within concurrency limits. ## Failure Modes and Recovery Use the table to quickly map symptom to first checks and actions. Then follow the detailed workflows below.
SymptomProbable causeFirst checks
Runner shows offlineInvalid token or service downsystemctl status; gitlab-runner verify
Jobs stuck pendingConcurrency zero or tags mismatchconcurrent value; job tags vs runner tags
Checkout failsSSH key or Git auth misconfiggit config; deploy keys; credentials helper
Artifacts upload 403Wrong credentials or endpointproject settings; URL; env vars for creds
Cache missesKey mismatch or backend unreachablecache key; backend auth; network route
Docker image pull failsRegistry auth or TLS trustdocker login; certs; proxy bypass
Kubernetes pod failsServiceAccount RBAC or quotaevents; can-i; namespace quota
Permission deniedFile ownership or SELinuxid; ls -Z; chown; setenforce 0 (temporary)
Constructed examples are labeled below. ### 1. Runner offline or unauthorized Checks: - UI: runner status is offline - Host: service running but logs show unauthorized or invalid token Recovery steps: - Re-register with a scoped token: sudo gitlab-runner register Choose the correct GitLab URL and enter the registration token from the project or group. Assign tags that match your jobs. - Confirm the runner name and tags in config.toml match expectations. - Restart the service and verify online status. Rollback: If the new registration created a duplicate, disable the old runner in the UI and remove its section from config.toml after confirming no jobs use it. ### 2. Jobs stuck in pending due to tag or concurrency mismatch (constructed example) Checks: - Job requires tags: linux , shell - Runner tags: only linux - concurrent = 0 in config Recovery steps: - Add missing tag or remove unnecessary job tag requirement. - Set a safe concurrency: concurrent = 2 - Restart and watch the queue. - Expected: jobs begin within a minute; queue depth decreases. Rollback: If high load causes resource contention, reduce concurrent to 1 temporarily. ### 3. Checkout or Git authentication failures Symptoms: - fatal: could not read Username for https://... No such device or address - Host key verification failed Checks: - Verify SSH keys or HTTPS credentials for the runner user. - For SSH: populate known_hosts for the runner user: sudo -u gitlab-runner ssh-keyscan -t rsa gitlab.com >> ~gitlab-runner/.ssh/known_hosts sudo chmod 600 ~gitlab-runner/.ssh/known_hosts Recovery steps: - If using HTTPS with a token, ensure the variable is masked and available to the job. - Validate the URL and credentials by cloning manually as the runner user. Rollback: Revert recent credential variable changes if the last known good state worked. ### 4. Artifacts and cache upload/download errors (constructed example) Symptoms: - 403 during upload to artifacts endpoint or S3-compatible store - Cache always re-computes Checks: - Validate project or group settings for artifacts and cache. - Inspect cache key used by the job; ensure it is stable and not overwritten each run unnecessarily. Recovery steps: - Set explicit and stable cache keys in the job definition to avoid cache misses: cache: key: "deps-v1" paths: - vendor/ - For external cache stores, export credentials via runner environment rather than hardcoding in scripts. Rollback: If changing the cache backend, keep the previous backend configured until the new one has successful hits for multiple jobs. ### 5. Network and proxy problems Symptoms: - i/o timeout to GitLab or registries - TLS handshake timeout behind proxy Checks: - curl to GitLab endpoints with and without proxy - NO_PROXY includes internal hosts and 127.0.0.1 Recovery steps: - Set HTTPS_PROXY and NO_PROXY only for affected runner(s) in config.toml . - For registries with self-signed certs, trust the CA on the host. Rollback: Remove proxy environment entries if they break internal connectivity. ### 6. Docker executor image pull failures (constructed example) Symptoms: error response from daemon: pull access denied Checks: - docker login status and credentials scope - Proxy interfering with docker pull Recovery steps: - docker login REGISTRY_URL with a read-only robot account. - If private CA, install it in the Docker engine trust store and restart the engine. Rollback: Revert to previously working base image tag to reduce moving parts. ### 7. Kubernetes executor pod lifecycle issues (constructed example) Symptoms: pods Pending or ImagePullBackOff Checks: - kubectl get events -n runner-namespace - RBAC: kubectl auth can-i create pods ... - ImagePullSecrets present on the runner ServiceAccount Recovery steps: - Grant minimal RBAC for pods, secrets (read if needed), and events. - Attach the correct imagePullSecret to the ServiceAccount. Rollback: Scale runner replicas to 0, revert recent RBAC changes, and scale back up after validation. ### 8. Permissions and filesystem ownership Symptoms: permission denied creating cache or artifacts directory Checks: - id gitlab-runner , ls -l target directories - SELinux enforcing blocks (Linux): getenforce ausearch -m avc -ts recent Recovery steps: - chown directories to gitlab-runner user. - For SELinux, set proper contexts (constructed example): sudo chcon -Rt svirt_sandbox_file_t /var/lib/gitlab-runner Prefer permanent policy adjustments via semanage fcontext if available. Rollback: Revert ownership or context changes if they affect other services; restore from the config backup as needed. ## Operations Checklist Use this concise checklist for ongoing operations and quick incident response. ### Daily or pre-deploy quick checks - Confirm runner online status in GitLab and note queue depth. - systemctl status gitlab-runner shows running; no recent crash loops. - Disk space above a safe threshold (constructed example: > 20%). ### When a job stalls or fails unexpectedly - Capture context: job ID, runner name, executor, recent changes. - Verify tags and concurrent settings. - Inspect last 200 log lines: journalctl -u gitlab-runner -n 200 --no-pager . - Re-run a single job with minimal variables to isolate. ### Change management - Back up config.toml with timestamp. - Edit only one variable at a time; record expected outcome. - Restart runner and verify within 5 minutes. - If outcome diverges, roll back immediately using the backup. ### Network and credentials hygiene - Validate DNS resolution and proxy env for runner users. - Rotate credentials on a schedule; test new creds with a single runner first. ### Capacity and performance - Track average job wait time and success rate. - Increase concurrent cautiously (step by 1) and observe CPU, memory, and I/O. ### Executor-specific - Shell: ensure required build tools and shells are installed for the runner user. - Docker: verify engine health, registry login, and CA trust. - Kubernetes: confirm node capacity, RBAC, and image pull secrets. ### Rollback and recovery - Keep at least one known-good runner untouched during major changes. - Maintain a playbook for re-registering a runner with correct tags and executors. - If a change causes widespread failures, restore config.toml from backup and restart. ## Conclusion Troubleshooting GitLab Runner becomes predictable when you start with a clear inventory, make one small change at a time, and verify each step against expected results. Use the safe configuration path, lean on targeted logs and health checks, and prefer reversible changes. The symptom-to-action table and the operations checklist give you fast paths to common fixes and safe rollbacks. By treating each change as an experiment with a rollback plan, you reduce the blast radius of mistakes and build confidence that your CI/CD pipeline will stay reliable under pressure. ### Constructed next steps - Pick one runner and run the Verification and Diagnostics section end to end. Timebox to 20 minutes and record findings. - Apply one improvement from the Failure Modes and Recovery section that addresses your top symptom. - Monitor for one day: queue depth, job success rate, and mean job start latency. If stable or improved, propagate to additional runners gradually. Focused, measurable steps make troubleshooting faster and safer. Start narrow, verify locally, then expand with confidence.