GitLab Runner Performance Tuning with Practical Examples

Intro

Fast, predictable CI requires more than bigger machines. It needs a clear method: observe the current state, make the smallest justified change, verify the result, and document how to roll back. This article walks through practical GitLab Runner performance tuning for developers and operators, with concrete examples you can adapt to Docker, Kubernetes, or shell executors.

Goals:

• Reduce queue time and job duration without sacrificing reliability.
• Pinpoint bottlenecks (CPU, memory, I/O, image pulls, cache thrash).
• Apply safe, reversible configuration changes with explicit verification.

Assumptions: examples target Linux hosts running GitLab Runner 13+ (systemd-managed service). Replace placeholders like YOUR_GITLAB_URL, RUNNER_NAME, and S3_BUCKET with your values.

Version and Environment Inventory

Before changing anything, capture a factual snapshot of what you have. Keep observation read-only and timestamped.

1. Identify versions and topology

• Runner binary and OS:

  date -Is
  gitlab-runner --version
  uname -a

• Executor type and runners registered on this host:

  gitlab-runner list

• Service state and recent logs (Linux/systemd):

  systemctl status gitlab-runner --no-pager
  journalctl -u gitlab-runner -n 200 --no-pager

• For Kubernetes executor, also note cluster version and node shape:

  kubectl version --short
  kubectl get nodes -o wide

1. Record hardware and resource pressure

• CPU and memory:

  lscpu | egrep 'CPU\(s\)|Model name'
  free -h

• Disk capacity and I/O headroom:

  df -hT
  iostat -xz 1 3  # if sysstat is installed

• Docker cache pressure (Docker executor):

  docker system df

1. Capture runner configuration (read-only)

• Path typically /etc/gitlab-runner/config.toml:

  sudo sed -n '1,200p' /etc/gitlab-runner/config.toml | sed 's/\(token = \).*/\1"REDACTED"/'

Note the global concurrent value, per-runner limit, executor, pull policy, cache settings, and any Kubernetes resource limits.

1. Baseline CI symptoms

• From GitLab UI, open a few recent pipelines and record for each job:
• Queue time (created_at to started_at)
• Total duration
• Time in "Preparing environment" (often image pulls) vs script time
• Export rough medians for the busiest project. Write down what "good" looks like (e.g., queue < 30s, job < 6m).

Only after you know what exists should you plan one change with a known blast radius and recovery path.

Safe Configuration Path

Every tuning step below follows one pattern: observe -> change one scoped item -> verify -> record -> roll back if needed. Adjust values to your hardware and workload.

1) Right-size concurrency

Scope: global runner concurrency on a single host.

• Observe:
• In config.toml, find:

    concurrent = 1

• At peak hours, check CPU steal/wait and load average while N jobs run.
• Smallest justified change:
• Start with concurrent equal to the number of physical cores for CPU-bound tasks, up to 1.5x for I/O-heavy tasks. Cap per-runner limit to prevent one runner from starving others.
• Edit and verify:

  sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak.$(date +%s)
  sudo editor /etc/gitlab-runner/config.toml
  # Example changes:
  # concurrent = 8
  # [[runners]]
  #   name = "RUNNER_NAME"
  #   limit = 6

  sudo systemctl restart gitlab-runner
  gitlab-runner verify

• Expected result: queue time decreases without CPU pegged >90% for prolonged periods and without memory OOM events.
• Failure signal: rising job failures with exit codes related to OOM/timeouts or host load spikes. Recovery: restore the backup config and restart the service.

2) Cut Docker image pull latency

Scope: Docker executor pulling images repeatedly increases "Preparing environment" time.

• Observe: in job logs, look for repeated Pulling docker image and long download times. Check local cache:

  docker images --format '{{.Repository}}:{{.Tag}}\t{{.Size}}' | sort

• Smallest justified change: set pull_policy to if-not-present and standardize a slim base image.
• Edit and verify:

  # /etc/gitlab-runner/config.toml
  [[runners]]
    executor = "docker"
    [runners.docker]
      image = "alpine:3.19"
      pull_policy = "if-not-present"
      shm_size = 134217728  # 128MB for Chrome/Playwright-like tests; adjust as needed

Then restart runner and run one pipeline that uses the configured image.

• Expected result: subsequent jobs reuse local images; preparing time drops.
• Failure signal: outdated images not refreshed. If you rely on frequent updates, use pull_policy = "always" only for those jobs or tags. Recovery: revert policy or pin image digests.

Tip: if your registry bandwidth is a bottleneck, use a local registry mirror or pre-pull heavy images via cron during off-peak hours.

3) Optimize Git clone and repository handling

Scope: network and disk overhead from fetching source code.

• Observe: check clone duration at job start. Large histories or submodules slow down runs.
• Smallest justified change: use shallow clones and selective strategies.
• Add to the project or group CI variables or .gitlab-ci.yml:

  variables:
    GIT_DEPTH: "20"        # try 1 for most projects; increase if your build needs more history
    GIT_SUBMODULE_STRATEGY: recursive
    GIT_STRATEGY: fetch     # avoids wiping workspace when cache is warm

• Expected result: smaller network transfer and faster start times.
• Failure signal: builds needing full history (e.g., versioning from tags) break. Recovery: bump GIT_DEPTH or set job-specific overrides.

4) Make caches work for you (and not against you)

Scope: reduce redundant downloads/builds; avoid cache storms.

• Observe: compare job time with and without a warm cache; inspect cache size growth. If using the default local cache, watch disk usage under /var/lib/gitlab-runner.
• Smallest justified change: move to shared S3-compatible cache for multiple runners; key caches precisely.
• Runner cache config (example):

  # /etc/gitlab-runner/config.toml
  [runners.cache]
    Type = "s3"
    Path = "cache/"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "https://s3.YOUR_REGION.example.com"
      AccessKey = "REDACTED"
      SecretKey = "REDACTED"
      BucketName = "S3_BUCKET"
      Insecure = false

• Job-level cache example (Node.js):

  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
    policy: pull-push

• Expected result: consistent cache hits across runners; total pipeline time falls.
• Failure signal: cache poisoning (stale or cross-branch conflicts) or runaway storage costs. Recovery: include branch in key for risky paths, limit paths, or temporarily set policy: pull on default branch.

5) Kubernetes executor resource sanity

Scope: avoid CPU throttling and memory eviction in K8s pods.

• Observe:

  kubectl get pods -n YOUR_NS
  kubectl describe pod YOUR_POD -n YOUR_NS | egrep 'Reason|Limits|Requests'

• Smallest justified change: set requests/limits aligned to typical job usage; prefer requests near median and limits near p95.
• Runner config (excerpt):

  [[runners]]
    executor = "kubernetes"
    [runners.kubernetes]
      image = "alpine:3.19"
      poll_timeout = 600
      cpu_request = "500m"
      memory_request = "512Mi"
      cpu_limit = "2"
      memory_limit = "2Gi"
      pull_policy = "IfNotPresent"

• Expected result: fewer Pending pods (insufficient resources), stable job duration.
• Failure signal: Pending/Failed pods with node pressure or OOMKilled. Recovery: lower limits/requests, add nodes, or spread runners across node groups.

Verification and Diagnostics

Use the same checks each time you tune so your improvements are measurable and repeatable.

1. Connectivity and health

gitlab-runner verify           # all runners in config; prints "is alive" for each
systemctl is-active gitlab-runner && echo OK

1. Queue time and duration trends

• In GitLab UI, compare medians for queue time and total duration before vs after your change across at least 20 recent jobs.
• For automated checks, export job data via the GitLab API and compute deltas per stage.

1. Runner logs and debug level

• Temporarily increase verbosity:

  # /etc/gitlab-runner/config.toml
  log_level = "debug"

Then restart and tail logs during a single pipeline. Revert to info when finished.

1. Metrics endpoint (if supported by your version)

• Enable the built-in Prometheus metrics endpoint on the runner host and verify locally:

  # /etc/gitlab-runner/config.toml
  listen_address = ":9252"

  sudo systemctl restart gitlab-runner
  curl -s http://127.0.0.1:9252/metrics | head

• Track active builds, queue depth, and executor errors in your monitoring stack.

1. Docker/Kubernetes environment checks

• Docker cache and pull performance:

  /usr/bin/time -f '%E %C' docker pull alpine:3.19
  docker system df

• Kubernetes scheduling pressure:

  kubectl top nodes
  kubectl describe node YOUR_NODE | egrep 'Allocatable|Pods:|CPU|memory'

Failure Modes and Recovery

Have a rollback plan before you change anything. Here are common pitfalls and how to resolve them safely.

1. Over-concurrency (host thrash)

• Symptoms: CPU >95% for minutes, rising job timeouts, OOM, or swap storms.
• Immediate recovery: reduce concurrent and per-runner limit, then restart the runner.
• Long-term: scale out with additional runners or bigger instances; separate heavy and light jobs with tags.

1. Image pull bottlenecks

• Symptoms: long "Preparing environment" with frequent pulls, registry 429/5xx.
• Immediate recovery: switch to pull_policy = "if-not-present"; temporarily pin images by digest; pre-pull via cron or a DaemonSet (K8s).
• Long-term: add a local registry mirror, slim your images, consolidate base layers.

1. Cache storms and stale caches

• Symptoms: sudden cache miss rates, high egress from cache backend, inconsistent build outputs.
• Immediate recovery: narrow cache keys (include branch or lockfile), mark risky caches as policy: pull until stabilized.
• Long-term: separate caches by toolchain, impose size limits, and document cache invalidation conventions.

1. Disk I/O saturation or no space left

• Symptoms: slow jobs, Docker layer extraction stalls, ENOSPC errors.
• Immediate recovery: cleanup safe artifacts and images during off-peak:

  docker system prune -af --volumes

Move runner cache to a larger/faster disk and set [runners.cache].Path accordingly.

• Long-term: provision more IOPS, use faster storage for /var/lib/docker and cache directories, and right-size caches.

1. Stuck or offline runners

• Symptoms: jobs remain pending; runner shows offline in GitLab.
• Immediate recovery:

  gitlab-runner verify
  systemctl restart gitlab-runner

If needed, re-register:

  gitlab-runner register \
    --url https://YOUR_GITLAB_URL \
    --registration-token REG_TOKEN \
    --name RUNNER_NAME \
    --executor docker \
    --tag-list default,docker \
    --non-interactive

• Long-term: automate health checks and alerts for runner liveness.

Operations Checklist

Run this lightweight checklist regularly to keep CI fast and safe.

Daily

• Review queue time and duration medians for top projects.
• Confirm runner health: gitlab-runner verify and systemctl is-active.
• Scan disk usage for Docker and cache directories.

Weekly

• Reassess concurrent and per-runner limit against actual CPU/memory headroom.
• Examine image pull logs; pre-pull or mirror heavy bases.
• Validate cache hit rates and adjust keys or paths if they drift.

Monthly or after major changes

• Re-baseline performance metrics and update your "good" thresholds.
• Audit runner config for drift; diff against a known-good template.
• For Kubernetes, verify requests/limits align to current job profiles; scale nodes if needed.

Conclusion

Effective GitLab Runner tuning is disciplined and incremental. Start with an inventory, make one scoped change, and verify with clear signals: lower queue time, shorter and more stable job durations, and healthy resource utilization. Keep sensitive values out of logs and configs, prefer reversible edits, and document recovery steps before you need them.

As your workload evolves, revisit concurrency, image strategy, caching, and executor resources. The combination of concrete measurements, minimal safe changes, and reliable verification will keep your pipelines fast without trading off reliability.