GitLab Runner Advanced Concepts Explained with Practical Examples

Intro

This guide explains advanced GitLab Runner concepts with concrete, low-risk procedures you can apply in production. It is written for engineers who operate runners across Docker and Kubernetes, and who need to move from an observed problem to a verified result.

The operating principles used throughout are:

• Observe before changing: capture version, topology, and current state first.
• Make the smallest justified change with a clear blast radius and rollback.
• Use placeholders instead of secrets and avoid copying production identifiers.
• Verify outcomes with explicit commands and signals; document recovery steps.

What follows focuses on the runner component itself: executors, isolation, capacity and concurrency, caching and artifacts, network and security options, observability, failure modes, and safe recovery.

Version and Environment Inventory

Start every investigation by inventorying version and topology. Separate observation (read-only) from intervention (changes).

Prerequisites:

• Shell access to the runner host or control plane
• The runner service installed and running
• Basic familiarity with your chosen executor (shell, Docker, or Kubernetes)

Read-only inventory examples (no changes):

1. Identify runner version and platform

$ gitlab-runner --version
$ uname -a

Signal: a semantic version (for example, 16.11.x or later) and OS/arch. If multiple hosts are used, capture each host and timestamp.

1. List registered runners on a host

$ sudo gitlab-runner list

Signal: runner names, IDs, scopes (shared/group/project), and tags. Note which are protected and whether they are locked to projects.

1. Capture active configuration (read-only)

$ sudo cat /etc/gitlab-runner/config.toml

Signal: executor type per runner ([runners.docker], [runners.kubernetes], [runners.machine] if still present), concurrent value, cache settings, volumes, helper image settings, and any metrics listener.

1. Service and logs

$ systemctl status gitlab-runner
$ journalctl -u gitlab-runner -n 200 --no-pager

For Docker executor hosts:

$ docker info --format '{{.ServerVersion}} {{.OperatingSystem}}'
$ docker ps --filter label=com.gitlab.gitlab-runner.job.id --format 'table {{.ID}}\t{{.Image}}\t{{.Status}}'

For Kubernetes executor:

$ kubectl -n <RUNNER_NAMESPACE> get pods -l app=gitlab-runner
$ kubectl -n <JOB_NAMESPACE> get pods -l gitlab-job=true

Before any change, define the expected end state (for example, lower job queue time, smaller image pull failures, or reproducible cache hits) and the failure signal (for example, specific error lines or HTTP codes).

Safe Configuration Path

Make only one scoped change at a time, and verify it. Below are common advanced adjustments with blast radius and rollback.

Example A: Control runner concurrency on a single host

• Component: global concurrency
• Supported: all recent versions
• Blast radius: all jobs scheduled on this host
• Prerequisite: current job queue metrics or observed CPU/RAM pressure

Observation (current value):

$ grep -E '^concurrent' /etc/gitlab-runner/config.toml || echo 'concurrent not set (defaults to 1)'

Intervention (set to 4):

$ sudo sed -i 's/^concurrent = .*/concurrent = 4/' /etc/gitlab-runner/config.toml || echo 'concurrent = 4' | sudo tee -a /etc/gitlab-runner/config.toml
$ sudo systemctl reload gitlab-runner || sudo systemctl restart gitlab-runner

Verification:

• Run two pipelines and confirm up to 4 jobs run parallel on this host.
• Watch CPU and memory; if saturation occurs, rollback to the previous value.

Rollback:

• Restore the prior file backup or set the old integer, then reload.

Example B: Harden Docker executor isolation and reproducibility

• Component: [runners.docker]
• Blast radius: jobs on this specific runner
• Goal: predictable images and fewer network pulls

Change:

[runners.docker]
  image = "alpine:3.19"
  pull_policy = ["if-not-present"]
  privileged = false
  shm_size = 268435456
  volumes = ["/cache"]

Prerequisites and cautions:

• privileged = false improves isolation; jobs needing Docker-in-Docker may fail.
• shm_size helps Chrome/Playwright and other browsers avoid OOM in /dev/shm.

Verification:

• Re-run a representative job. Signals: cached image reuse in logs, no DinD failures if not required.

Rollback:

• Restore previous docker section, reload runner.

Example C: Enable S3-compatible cache for faster builds

• Component: [runners.cache]
• Blast radius: cache usage for jobs on this runner
• Prerequisite: cache bucket exists; use read/write credentials with minimal scope

Config snippet with placeholders only:

[cache]
  Type = "s3"
  Shared = true
  [cache.s3]
    ServerAddress = "s3.example.com"
    BucketName = "<CACHE_BUCKET>"
    BucketLocation = "<REGION>"
    Insecure = false
  [cache.s3.access]
    AccessKey = "<ACCESS_KEY>"
    SecretKey = "<SECRET_KEY>"

Verification:

• Add a cache section to your .gitlab-ci.yml and run a job:

cache:
  key: "$CI_COMMIT_REF_SLUG"
  paths:
    - node_modules/

• Signals: job logs show Creating cache and Successfully uploaded cache; subsequent jobs show Restoring cache.

Rollback:

• Comment out the [cache] block and reload the runner.

Example D: Enable metrics to observe capacity

• Component: runner metrics endpoint
• Blast radius: none; read-only endpoint

[metrics]
  listen_address = ":9252"

Verification:

$ curl -s localhost:9252/metrics | grep gitlab_runner

Signals: metrics such as gitlab_runner_jobs_running, gitlab_runner_jobs_queued.

Verification and Diagnostics

Use targeted checks to validate connectivity, registration, and executor health.

1. Validate runner registration and reachability

$ gitlab-runner verify

Signal: registered runners printed with OK; failures show HTTP or auth errors. If you see Unauthorized, suspect a revoked token or incorrect URL.

1. Dry-run jobs locally to isolate executor issues

For Docker executor, run a job locally using your .gitlab-ci.yml:

$ gitlab-runner exec docker <job_name> \
  --docker-image alpine:3.19 \
  --env CI_DEBUG_TRACE=true

Signal: the job script runs locally; if it fails here, the problem is within the job definition or image, not GitLab networking.

1. Inspect live job containers (Docker)

$ docker ps --filter label=com.gitlab.gitlab-runner.job.id -a
$ docker logs <CONTAINER_ID>

Signals: image pull errors, permission issues, or OOM kills.

1. Inspect live job pods (Kubernetes)

$ kubectl -n <JOB_NAMESPACE> get pods -l gitlab-job=true
$ kubectl -n <JOB_NAMESPACE> describe pod/<POD_NAME>
$ kubectl -n <JOB_NAMESPACE> logs <POD_NAME> --all-containers

Signals: Admission or RBAC errors, ImagePullBackOff, or node scheduling failures.

1. Confirm networking and proxy behavior

Add a temporary diagnostic step to a non-production job:

script:
  - apk add --no-cache curl
  - curl -I --max-time 5 https://gitlab.com

Signal: 200/301/302 indicates basic outbound reachability; timeouts suggest firewall or proxy issues.

Failure Modes and Recovery

Below are frequent failure patterns, their signals, and safe recovery actions.

1. Invalid or rotated runner token

• Signals: verify shows Unauthorized; jobs stuck pending with Runner not found.
• Recovery: on the runner host, remove the stale entry and re-register with a new token. Keep the same tags to avoid pipeline disruption.

$ sudo gitlab-runner unregister --name <RUNNER_NAME>
$ sudo gitlab-runner register \
  --url https://<GITLAB_URL>/ \
  --token <RUNNER_TOKEN> \
  --executor docker \
  --docker-image alpine:3.19 \
  --tag-list "linux,x86_64" \
  --non-interactive
$ sudo systemctl restart gitlab-runner

1. Saturated host or mis-set concurrency

• Signals: long queue times; host CPU at 100%; OOM kills in logs.
• Recovery: reduce concurrent in config.toml, add resource limits (Docker: cpus, memory), or scale out additional runners.

[runners.docker]
  cpus = "2"
  memory = "4g"

1. Image pull failures and rate limits

• Signals: Too Many Requests or pull timeouts.
• Recovery: pin images to an internal registry mirror, set pull_policy = ["if-not-present"], or pre-pull critical images via a daemonset (Kubernetes) or cron (Docker host). Verify by re-running a job and confirming no external pulls.

1. Cache does not restore

• Signals: job log shows No URL found for cache or cache miss on every run.
• Recovery: ensure identical cache key, correct bucket/region, and that Shared = true when multiple runners use the same cache. Test by running two consecutive jobs on different runners with the same cache key.

1. Kubernetes RBAC or namespace drift

• Signals: Forbidden errors on pod creation; ImagePullBackOff due to missing secrets.
• Recovery: verify service account, role bindings, and imagePullSecrets in the runner chart config. Rollback by reapplying the last known-good ConfigMap/Secret and restarting the runner deployment.

1. Disk pressure and leftover containers

• Signals: no space left on device; many exited containers; large /var/lib/docker.
• Recovery: implement periodic cleanup:

$ docker system prune --volumes --filter "until=24h" -f

• Then add lifecycle tasks off-hours. Verify via docker system df.

Operations Checklist

Use this repeatable checklist for routine runner operations. Adjust to your environment.

• Inventory
• [ ] Record gitlab-runner --version and OS/arch per host
• [ ] List runners and tags; confirm scopes and protected status
• [ ] Capture config.toml with timestamp and secure backup

• Capacity and isolation
• [ ] Set a justified concurrent value; monitor CPU/RAM saturation
• [ ] For Docker, pin image, set pull_policy, shm_size, and volumes
• [ ] For Kubernetes, confirm namespaces, RBAC, node selectors, and tolerations

• Caching and artifacts
• [ ] Configure S3-compatible cache with least-privilege keys
• [ ] Verify one cache create and one restore on different runners
• [ ] Align artifacts: when, what, and retention

• Network and security
• [ ] Test outbound connectivity and proxy rules from jobs
• [ ] Avoid privileged unless required; document why when enabled
• [ ] Store tokens in secure variables; never embed secrets in files

• Observability
• [ ] Enable the metrics endpoint; scrape and alert on queued/running jobs
• [ ] Centralize runner logs; retain recent job logs for diagnosis

• Change management
• [ ] Make one change at a time with blast radius noted
• [ ] Verify signals immediately; roll back quickly if unexpected
• [ ] Keep a canary runner for testing upgrades before fleet rollout

Conclusion

Advanced GitLab Runner operations are predictable when you separate observation from change, make small scoped updates, and verify each step against clear signals. Start with a clean inventory of version, executor, and configuration. Apply targeted improvements to concurrency, isolation, and caching. Use diagnostics to confirm registration, container or pod health, and network reachability. Plan for common failures with prepared rollbacks and routine cleanup.

Choose one low-risk improvement now, such as enabling the metrics endpoint or setting a conservative concurrent value. Capture the current state, apply the change, verify the outcome in real pipelines, and document both the success criteria and the recovery path. That workflow makes failures visible, protects sensitive values, and keeps builds fast and reliable.