GitLab Runner: Common Errors, Root Causes, and Reliable Fixes (With Practical Examples)

Intro

GitLab Runner is the engine behind your CI/CD jobs. When it fails, pipelines stall and logs can be cryptic. This guide shows you a proven way to diagnose and fix the most common runner errors safely and quickly. You will learn how to inventory your environment, apply scoped configuration changes, validate fixes with concrete commands, and recover without breaking working pipelines.

1) Inventory your runner and environment

Start by writing down exactly what you are running. Versions and executors determine both error messages and fixes.

• Runner version and platform:

gitlab-runner --version

• Registered runners and executors:

gitlab-runner list

• If using Docker executor:

docker --version
docker info | sed -n '1,20p'

• If runner itself runs in Docker:

docker ps --filter name=gitlab-runner --format '{{.Names}}  {{.Image}}'

• If using Kubernetes executor:

kubectl version --short
kubectl get nodes

Also note:

• Operating system and architecture (Linux/Windows/macOS; amd64/arm64).
• Network mode (host/bridge), proxies, custom DNS, and whether TLS inspection or a corporate CA is in play.
• Whether jobs require privileged mode (Docker-in-Docker, BuildKit) or specific volumes.

2) Make safe, scoped changes

Use a deliberate workflow to avoid introducing new failures.

1. Back up config first:

• Linux package: /etc/gitlab-runner/config.toml
• Dockerized runner: docker cp gitlab-runner:/etc/gitlab-runner/config.toml ./config.toml.bak

1. Edit only the affected runner scope (avoid global changes unless needed). Example: extend a timeout for one runner:

[[runners]]
  name = "my-runner"
  url = "https://gitlab.example.com/"
  token = "..."
  executor = "docker"
  timeout = 3600

1. Validate the runner and reload:

gitlab-runner verify
# Linux packages
sudo systemctl restart gitlab-runner
# Dockerized runner
docker restart gitlab-runner

1. Use a minimal test job to confirm behavior before re-running production pipelines:

# .gitlab-ci.yml
verify-runner:
  image: alpine:3
  script:
    - echo "Runner OK"
    - apk add --no-cache curl >/dev/null 2>&1 || true

3) Common errors, root causes, and practical fixes

3.1 exec: "docker": executable file not found in $PATH

• When: Docker executor but Docker binary is absent or inaccessible.
• Check and fix:
• Ensure Docker is installed and running on the host:
• Ubuntu/Debian: sudo apt-get update && sudo apt-get install -y docker.io
• RHEL/CentOS: sudo yum install -y docker and sudo systemctl enable --now docker
• Confirm the binary is resolvable:

which docker && docker info

• If the runner runs inside a container, use the Docker socket or Docker-in-Docker:
• Socket mount: -v /var/run/docker.sock:/var/run/docker.sock
• DinD service (requires privileged):

[runners.docker]
  privileged = true
  services = ["docker:24-dind"]

3.2 ERROR: Job failed: exit code 1

• When: Generic failure; the script within the job failed.
• Diagnose:
• Look for the last non-empty command in the job log.
• Reproduce locally with the same image and working directory:

docker run --rm -v "$PWD":/builds/project -w /builds/project my-image:latest sh -lc "<the failing command>"

• Fix tips:
• Ensure required tools exist in the image or install them in the job.
• Make the failing command verbose (e.g., set -euxo pipefail, npm --verbose, mvn -X).

3.3 dial tcp: lookup gitlab.example.com: no such host

• When: DNS resolution failure inside runner host or job container.
• Diagnose:

nslookup gitlab.example.com || dig gitlab.example.com +short

• Fix options:
• Set valid DNS resolvers on the host (/etc/resolv.conf) or in Docker daemon (/etc/docker/daemon.json):

{
  "dns": ["1.1.1.1", "8.8.8.8"]
}

• For stubborn cases, test host networking in runner docker config:

[runners.docker]
  network_mode = "host"

• If corporate DNS or proxy is required, add environment variables and no_proxy to the runner:

[runners]
  environment = ["http_proxy=http://proxy:3128", "https_proxy=http://proxy:3128", "no_proxy=localhost,127.0.0.1,.example.com"]

3.4 SSL certificate problem: unable to get local issuer certificate

• When: Corporate MITM or private CA; GitLab or registry cert not trusted.
• Fix:
• Install the CA on the runner host (Linux): place corp-ca.crt in /usr/local/share/ca-certificates/ then sudo update-ca-certificates.
• For Docker jobs, bake the CA into the image or mount it and set env vars:

variables:
  SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt

• For GitLab Runner, use tls-ca-file in config.toml if needed:

[[runners]]
  tls-ca-file = "/etc/gitlab-runner/corp-ca.crt"

3.5 permission denied (checkout, cache, artifacts, or workspace)

• When: Mismatched UIDs/GIDs, restrictive mounts, or non-root user in the container.
• Diagnose:

id
ls -ld /builds /cache

• Fix options:
• Run container with a matching user or root:

[runners.docker]
  user = "root"

• Align permissions on the host:

sudo chown -R gitlab-runner:gitlab-runner /var/lib/gitlab-runner
sudo chmod -R u+rwX /var/lib/gitlab-runner

• If using Kubernetes, set securityContext or use an image that supports non-root.

3.6 Cache is not found (cache not restored)

• When: Cache key mismatch, wrong paths, or external cache misconfig.
• Fix:
• Use a stable key with fallbacks and clear paths:

cache:
  key:
    files:
      - package-lock.json
    prefix: "node"
  fallback_keys:
    - "node"
  paths:
    - node_modules/

• For S3/MinIO cache, validate credentials and bucket:

gitlab-runner verify --delete
gitlab-runner cache-archiver --help

• Keep paths small and specific; avoid caching build outputs that change constantly.

3.7 Failed to remove network: network ... not found / orphaned Docker resources

• When: Interrupted cleanup or daemon restarts.
• Fix:

sudo systemctl restart docker || true
docker network prune -f
docker container prune -f
docker image prune -f

• Consider a periodic cleanup job on the runner host.

3.8 This job is stuck; no runners online or no matching tags

• When: Runner offline, paused, not assigned to the project, or tags do not match.
• Diagnose and fix:
• In GitLab UI, confirm the runner is online and not paused.
• Check tags on the job and runner. They must intersect:

lint:
  tags: [docker, amd64]

• Ensure runner is shared or assigned to the project and not locked to another project.
• Increase concurrency if all builds are queued:

concurrent = 4

3.9 pull access denied / image not found / rate limited

• When: Private registry without auth or Docker Hub limits.
• Fix:
• Provide registry credentials via DOCKER_AUTH_CONFIG:

variables:
  DOCKER_AUTH_CONFIG: >
    {"auths": {"registry.example.com": {"auth": "<base64(user:pass)>"}}}

• Or log in during before_script:

before_script:
  - echo "$CI_REGISTRY_PASSWORD" | docker login -u "$CI_REGISTRY_USER" --password-stdin "$CI_REGISTRY"

• Pin images to digests to avoid surprise changes:

image: alpine@sha256:1b231...

3.10 Docker-in-Docker build fails without privileged or with nested cgroups

• When: Using docker:dind but the runner lacks privileged or proper mounts.
• Fix:

[runners.docker]
  privileged = true
  volumes = ["/certs/client", "/cache"]
  services = ["docker:24-dind"]

• If builds need BuildKit, enable it explicitly:

variables:
  DOCKER_BUILDKIT: "1"

3.11 Git clone fails: Host key verification failed or auth errors

• When: SSH host key unknown or tokens/credentials missing.
• Fix options:
• Prefer HTTPS with CI_JOB_TOKEN (default in GitLab) and ensure corporate CA is installed if using TLS inspection.
• For SSH, add known_hosts within the job:

before_script:
  - mkdir -p ~/.ssh && chmod 700 ~/.ssh
  - ssh-keyscan -t rsa gitlab.example.com >> ~/.ssh/known_hosts

4) Verification and diagnostics workflow

Use the following steps after each change:

• Verify runner liveness:

gitlab-runner verify

• Run a minimal pipeline (the verify-runner job above) and confirm success.
• Enable debug logs temporarily:

gitlab-runner --debug run

• For Docker executor, compare host vs container behavior with the same image to isolate environment differences.

5) Recovery playbook

Have a rollback path for each change.

• Malformed config.toml (runner fails to start):

sudo cp /etc/gitlab-runner/config.toml.bak /etc/gitlab-runner/config.toml
sudo systemctl restart gitlab-runner

• Resource exhaustion (no space left on device / cannot allocate memory):

df -h
free -h
# Cleanup
gitlab-runner cleanup || true
docker system prune -af --volumes

• Network mode changes break connectivity: test and revert quickly.

nc -vz gitlab.example.com 443 || curl -vkI https://gitlab.example.com
# If failing, revert network_mode or proxy settings

• Runner container unhealthy: recreate from a known-good image and re-use the same config.toml.

6) Quick reference table

7) Operational checklist

Run this weekly or when troubleshooting:

• Runner is alive: gitlab-runner verify
• No stuck jobs in GitLab UI; tags match
• Disk usage under 80%: df -h; prune Docker: docker system prune -af
• Backup updated: copy config.toml and document changes
• Review runner logs for new warnings

Conclusion

Troubleshooting GitLab Runner becomes predictable with a clear process: inventory your environment, change one thing at a time, verify with a minimal job, and keep a rollback ready. Use the common error fixes in this guide as starting points, then document what works in your environment. Over time, your runners will be faster to repair, easier to operate, and far less likely to block critical pipelines.