GitLab CI/CD performance directly impacts engineering velocity. Long queue times delay feedback, overloaded runners cause flaky failures, and misconfigured caches waste bandwidth and storage. This guide provides a staged, measurable approach to finding and fixing bottlenecks without risking your delivery pipeline. You will inventory your environment, establish baselines, apply targeted optimizations, validate each change, and build a repeatable operating routine. ## Inventory Your Environment First Before making changes, capture what you have. This clarifies which tuning levers exist and which documentation applies. Record the following: - GitLab edition and version (SaaS vs. self-managed) - Runner versions and executors (shell, Docker, Kubernetes) - Cache and artifact backends (local disk, S3, GCS, NFS) - Network locality (where runners sit relative to GitLab, registries, and package mirrors) - Repository traits (size, submodules, LFS usage) - Access you actually hold (runner config, project settings, API token) Quick commands to gather data: # From a job log echo "$CI_SERVER_VERSION $CI_RUNNER_EXECUTOR $CI_PROJECT_PATH" # From a runner host gitlab-runner --version gitlab-runner list # From GitLab API (replace placeholders) curl -s --header "PRIVATE-TOKEN: " https:///api/v4/version Use this table to standardize your inventory:

Field	Example Value	Notes
GitLab version	16.11 SaaS	Affects available features and defaults
Runner version(s)	16.11	Keep aligned with server where possible
Runner executors	shell, docker	Drives caching strategy and isolation
Cache backend	S3, bucket gitlab-runner-cache	Enables cross-runner cache sharing
Artifact policy	expire_in: 2 days	Keep artifacts small and short-lived
Repo traits	2.1 GB, 5 submodules, LFS on	Heavy clone if not tuned
Network locality	Runners in same region as GitLab	Reduces transfer latency
Access	Runner config, project maintainer, read_api token	Required to apply tuning changes

## Safe Configuration Path Apply changes in small steps with clear rollback. Follow this sequence: (1) baseline metrics, (2) quick wins, (3) throughput sizing, (4) latency and cache tuning, (5) safe parallelization, (6) re-measure and expand only after validation. ### 1. Establish a Measurable Baseline Pick one or two representative pipelines (for example, main branch build and test) and capture three to five recent runs. Record pipeline duration, critical job durations, job queue time, and artifact upload/download time. Add simple timing around heavy steps in your scripts: set -euo pipefail stamp() { date +"%Y-%m-%d %H:%M:%S"; } step() { echo "[STEP] $(stamp) $*"; } step "Install deps" # your install commands step "Run tests" # your test commands Use the API to collect baseline data: # Last 10 successful pipelines and durations curl -s --header "PRIVATE-TOKEN: " \ "https:///api/v4/projects//pipelines?status=success&per_page=10" | jq '[.[] | {id, status, duration}]' # Jobs for a specific pipeline curl -s --header "PRIVATE-TOKEN: " \ "https:///api/v4/projects//pipelines//jobs" | jq '[.[] | {name, status, duration, queued_duration}]' ### 2. Quick Wins in .gitlab-ci.yml Reduce Git clone overhead with shallow fetch: variables: GIT_STRATEGY: fetch GIT_DEPTH: "1" GIT_SUBMODULE_STRATEGY: none Right-size artifacts. Do not archive large, reproducible outputs. Keep expirations short: artifacts: when: on_success expire_in: 2 days paths: - build/reports/ Add caches for dependency directories you know are safe to reuse. Key caches by lockfiles so they invalidate only when dependencies change. Node.js example: cache: key: files: - package-lock.json paths: - node_modules/ policy: pull-push Python example: cache: key: files: - requirements.txt paths: - .venv/ - $PIP_CACHE_DIR policy: pull-push Maven example: cache: key: maven-$CI_COMMIT_REF_SLUG paths: - .m2/repository policy: pull-push ### 3. Throughput Sizing on Runners Set global concurrency so the host stays busy but not overloaded. For CPU-bound jobs, start with concurrent near the vCPU count, then adjust for I/O and memory pressure. In /etc/gitlab-runner/config.toml , review these knobs: concurrent = 6 check_interval = 0 runners name = "build-runner-01" url = "https://gitlab.example.com/" token = "REDACTED" executor = "docker" request_concurrency = 2 [runners.cache] Type = "s3" Path = "runner" Shared = true [runners.cache.s3] ServerAddress = "s3.amazonaws.com" BucketName = "gitlab-runner-cache" BucketLocation = "us-east-1" Insecure = false Restart the runner to apply changes: systemctl restart gitlab-runner Sizing guidelines (starting points; tune to your workload): - CPU-bound builds: concurrent ≈ vCPU count, request_concurrency = 2 to 4 - I/O-heavy builds: concurrent < vCPU (for example, 0.5 × vCPU) - Memory-heavy tests: cap concurrent so the largest job has at least 1.5× its peak memory available ### 4. Latency and Cache Tuning - Co-locate runners near GitLab and your container/package registries to reduce transfer time. - Prefer remote cache (S3/GCS) for fleets so caches survive runner restarts and hit across machines. - Keep cache keys stable within a branch but different across branches to avoid poisoning. - Avoid archiving huge caches; cache only what you cannot re-download quickly (dependency directories, not build outputs). ### 5. Safe Parallelization Split long-running test jobs using GitLab parallel . Start small and ensure isolation. Fixed-count example: unit_tests: stage: test script: run_tests.sh "$CI_NODE_INDEX" "$CI_NODE_TOTAL" parallel: 4 Matrix example: build: stage: build parallel: matrix: - PLATFORM: [linux, darwin] ARCH: [amd64, arm64] script: - ./build.sh "$PLATFORM" "$ARCH" Implementation notes: - Ensure tests do not share mutable global state (databases, ports, temp directories). Parameterize per node using $CI_NODE_INDEX . - Gate parallelism behind a feature flag variable so you can roll back quickly: workflow: rules: - if: '$ENABLE_PARALLEL == "1"' ### 6. Re-measure and Decide Next Steps Compare durations, queue time, and artifact transfer time against your baseline. Keep only changes that produce consistent wins across multiple runs. ## Verification and Diagnostics Verification proves you improved the right thing. Use these checks. ### Pipeline and Job Timing - Compare three to five runs before and three to five runs after a change to smooth random variance. - Inspect job details for Queued time vs. Running time . Queued signals capacity issues; Running signals job-level inefficiency. ### Runner Health - On runner hosts, watch CPU, memory, disk I/O, and network during peak hours. If CPU is idle while jobs wait in queue, you likely need more registered runners or increased concurrent . - Validate runner connectivity: gitlab-runner verify ### Symptom-to-Metric Guide Symptom Metric to Check Likely Root Cause First Action Long job queued time queued_duration Not enough runner capacity Increase concurrent or add runners Long clone times Job logs, repo size Deep history, submodules Use GIT_DEPTH , disable submodules Slow dependency steps Step-level timers Missing or unstable caches Add cache keyed by lockfiles Slow artifacts Upload/download time Oversized artifacts Reduce paths, set expire_in Flaky tests after split Failure rate per node Shared resources Isolate with CI_NODE_INDEX ## Failure Modes and Recovery Common issues and how to recover safely. ### 1. Cache Poisoning or Cache Misses Symptoms: Build uses wrong dependency versions, or caches never hit. Likely cause: Overly broad or unstable cache keys. Recovery: - Narrow keys to lockfile or manifest files (for Node: key.files: package-lock.json ). - Clear caches. For local caches, remove /var/lib/gitlab-runner/cache/* on the runner. For S3, delete the affected prefix. ### 2. Runner Overload and Timeouts Symptoms: Rising queued_duration , sporadic job failures under load. Causes: concurrent too high for hardware, network saturation. Recovery: - Reduce concurrent in config.toml and restart gitlab-runner . - Add more runners or move heavy jobs to larger machines. ### 3. Flaky Tests After Parallelization Symptoms: Intermittent failures, port collisions, shared temp directories. Recovery: - Parameterize per-node resources using $CI_NODE_INDEX . Isolate temp directories and databases per node. - Roll back: set ENABLE_PARALLEL=0 in project variables, or revert the .gitlab-ci.yml change. ### 4. Artifact Bloat Slows Pipelines Symptoms: Long upload/download times. Causes: Archiving large, reproducible binaries or entire workspaces. Recovery: - Limit artifacts to needed reports and small deliverables. Set expire_in aggressively. - Use caches for frequently reused content and artifacts for minimal deliverables. ### 5. Shallow Clones Break Versioning Symptoms: Build scripts cannot find tags or history. Cause: GIT_DEPTH=1 when scripts require tags. Recovery: - Increase GIT_DEPTH to include needed history, or fetch tags explicitly: script: - git fetch --tags --depth=50 ### 6. Remote Cache Permission Errors Symptoms: 403 or 404 during cache push/pull. Recovery: - Verify credentials and bucket policies for the runner instance profile or access keys. - Temporarily disable cache ( policy: pull ) to unblock builds, then fix permissions. ## Operations Checklist Use this routine to keep performance under control. ### Daily - Scan queued_duration on the busiest jobs. - Triage the slowest job from the last 24 hours. ### Weekly - Review pipeline duration trends and runner resource graphs. - Prune caches older than your average branch life. ### Per Change - Baseline three runs, apply one change, re-measure three runs. - Document deltas and keep or revert the change. Change Log Template:

Change	Baseline Metric	After Metric	Delta	Keep or Revert	Owner
GIT_DEPTH=1 on repo X	Clone 180s	Clone 35s	-81%	Keep	A. Dev
Cache node_modules	Install 240s	Install 55s	-77%	Keep	B. Ops
Parallel tests ×4	Test 28m	Test 9m	-68%	Keep	C. QA
Reduce artifact paths	Upload 120s	Upload 20s	-83%	Keep	D. Dev

## Practical Examples ### 1. Reduce Queue Time by Adding Capacity - Baseline: queued_duration ~ 300s during peak hours. - Change: Increase concurrent from 4 to 8 on two runners (total logical capacity from 8 to 16), keeping CPU utilization under 75%. - Expected result: queued_duration falls below 60s. - Verify: Compare average queued_duration over 10 pipelines before and after. - Rollback: If CPU crosses 90% and failures rise, revert concurrent and plan to add a third runner instead. ### 2. Shrink Clone Times with Shallow Fetch - Baseline: Clone 150s on a 2 GB repo with submodules. - Change: GIT_STRATEGY: fetch , GIT_DEPTH: 1 , GIT_SUBMODULE_STRATEGY: none . - Expected result: Clone under 40s. - Verify: Job log start-to-first-command timing. - Rollback: If build requires tags or submodules, raise GIT_DEPTH or re-enable submodules selectively. ### 3. Speed Up Dependencies with Lockfile-Keyed Cache - Baseline: Node install 200s cold, 200s every build. - Change: Cache node_modules keyed by package-lock.json . - Expected result: Cached install ~40-60s, invalidates only on dependency change. - Verify: Observe cache restore message and install duration. - Rollback: If cache conflicts occur across branches, include $CI_COMMIT_REF_SLUG in the key. ### 4. Split Long Tests Safely - Baseline: Single test job 30m. - Change: parallel: 4 and pass $CI_NODE_INDEX to shard tests. - Expected result: 8-12m total test time depending on overhead. - Verify: Ensure each shard duration is within 10-20% of others; adjust sharding logic if not. - Rollback: Set ENABLE_PARALLEL=0 to disable while investigating flakiness. ## What Good Looks Like - Queue time: Under 60s for routine pipelines during business hours. - Clone time: Proportional to GIT_DEPTH and repo size; under 60s for most repos with shallow fetch. - Dependency steps: Under 1 minute with warm caches for typical language ecosystems. - Artifacts: Upload/download under 30s when limited to reports and small deliverables. - Stability: Failure rate does not increase after tuning; if it does, roll back the last change. ## Conclusion You now have a safe, staged method to tune GitLab CI/CD performance: inventory what exists, baseline what matters, apply small reversible changes, verify results, and operationalize the routine. Start with the smallest pilot that is easy to measure and inspect, keep what consistently improves throughput or latency, and roll it out deliberately. Revisit your inventory and sizing each quarter or after significant workload shifts (new languages, larger repos, or new test suites). With disciplined measurement and careful rollbacks, pipeline speed becomes a managed, predictable part of your engineering system.