## Intro GitLab Runner is the workhorse of GitLab CI/CD, executing the jobs defined in your pipelines. As your team and project count grow, a single runner quickly becomes a bottleneck: queues lengthen, pipelines time out, and developers wait. Capacity planning is the practice of determining how many runners, of what size, and in what configuration you need to meet your throughput and latency targets without wasting resources. This article provides a practical guide to GitLab Runner capacity planning with concrete examples. You will learn how to inventory your current environment, estimate resource needs, choose safe configuration options, verify performance, handle failures, and maintain an operational checklist. By the end, you will have a repeatable method to size your runner fleet and avoid common scaling pitfalls. ## Version and Environment Inventory Before planning capacity, you must know what you have. This section covers how to inventory your GitLab Runner version, topology, and prerequisites. ### GitLab Runner Version Start by checking the runner version on each host. Use the following command on the runner machine: gitlab-runner --version Expected output: Version: 15.9.0 Note the version; features like ff_use_direct_download or docker+machine autoscaling may vary. Different versions have different capabilities, so knowing the exact version helps when planning upgrades or troubleshooting. ### Topology Overview Document your runner topology thoroughly. This includes: - Runner types : Shared, group, or project-specific runners. - Executors : Shell, Docker, Kubernetes, etc. - Host types : Physical, virtual machines, or cloud instances. - Network location : On-premises or cloud. An inventory table helps you understand your current capacity. Here is an example for a typical mid-sized organization:

Runner Name	Host	Executor	Concurrency Limit	Current Jobs/Min
shared-docker-1	vm-build-01	docker	4	12
shared-docker-2	vm-build-02	docker	4	10
project-alpha	k8s-cluster	kubernetes	10	30

In this example, the shared Docker runners handle general workloads, while the Kubernetes runner is dedicated to a specific project with higher throughput needs. ### Prerequisites for Capacity Planning You will need the following before you can effectively plan capacity: - Access to runner configuration files ( config.toml ) and the GitLab admin area. - Monitoring for runner hosts : CPU, memory, disk I/O metrics. - Historical job data from GitLab : pipeline duration, job queue times. - Understanding of your CI job resource characteristics : Are they CPU-intensive, I/O-heavy, or memory-hungry? Without this data, any capacity planning is guesswork. For example, if you do not know that your test suite is I/O-heavy, you might over-provision CPU and still have slow jobs. ### Identifying Bottlenecks Use GitLab's UI or API to check current runner utilization. For example, list all runners and their status: curl --header "PRIVATE-TOKEN: " "https://gitlab.example.com/api/v4/runners/all" The JSON response includes fields like online , status , and maximum_timeout . Look for runners that are always busy or jobs that wait long in the pending state. A runner that is constantly at its concurrency limit and has a backlog of pending jobs is a clear bottleneck. ## Estimating Resource Needs Once you have an inventory, you can estimate how much capacity you need. This involves understanding your workload and translating that into runner specifications. ### Characterizing Your Job Types Different jobs have different resource profiles. Start by categorizing your typical pipeline jobs: - Build jobs : Often CPU-intensive, such as compiling code or building Docker images. - Test jobs : May be CPU or memory intensive depending on the test framework and environment. - Deployment jobs : Usually light on resources but may require specific tools or network access. For each category, measure the average duration and resource consumption. You can do this by running a representative job and monitoring it with tools like htop or Prometheus. For example, a typical build job might take 5 minutes and use 2 CPU cores and 4 GB of memory at peak. ### Calculating Throughput Requirements Determine how many jobs you need to run per minute during peak times. This is your target throughput. For example, if your team runs 200 pipelines per day, with an average of 3 jobs per pipeline, that is 600 jobs per day. Assuming a 8-hour workday with peak at 2x average, peak throughput is about 3.75 jobs per minute. To handle spikes, plan for at least 5 jobs per minute. Then, calculate how many concurrent jobs you need. If a job takes 5 minutes on average, to sustain 5 jobs per minute, you need to run 25 jobs concurrently (5 jobs/min x 5 min = 25 concurrent jobs). This is a simplified calculation; in reality, job durations vary, so add a buffer. ### Sizing Runners With the concurrency number, you can size your runner fleet. A single runner has a concurrent limit, which is the number of jobs it can run simultaneously. For Docker executor, this is limited by host resources: CPU cores, memory, and disk I/O. A common rule of thumb: concurrent = number of CPU cores - 1 (leave one core for the host OS and Docker daemon). For memory, allocate at least 512 MB per job plus overhead, so a host with 16 GB RAM could run about 12 jobs (16 GB / 1.5 GB per job ≈ 10, leave some for host). For example, if you need 25 concurrent jobs, you could have three 8-core VMs, each with concurrent = 7 , giving 21 concurrent jobs, plus a Kubernetes runner for overflow. Or, you could use larger machines with more cores. ### Considering Peak vs. Average Capacity planning should focus on peak usage, not average. If your team has a deploy day or monthly release, you will see spikes. Plan for the 95th percentile of load, not the mean. Monitor your GitLab instance over a few weeks to identify patterns. For instance, if load doubles every Monday morning due to weekend commits, size for that spike. ## Safe Configuration Path Capacity planning involves making configuration changes that can impact your entire CI system. Follow a safe, incremental approach. ### Start with a Baseline Before scaling, establish a performance baseline. Run a representative job multiple times and record durations. For example, a typical build job may take 5 minutes. Measure queue time: if jobs wait 10 minutes, your total pipeline time is 15 minutes, indicating a capacity problem. This baseline helps you evaluate the impact of changes. ### Use the Right Executor Choose an executor that matches your workload and infrastructure: - Shell executor : Simple, but jobs run on the host, risking resource contention and security issues. Best for lightweight tasks or when you need direct host access. - Docker executor : Isolates jobs in containers, providing consistent environments. Adds some overhead but is the most common choice for general CI. - Kubernetes executor : Dynamic scaling, ideal for cloud-native setups, but complex to tune. Best for large, fluctuating workloads. For most teams, Docker executor on dedicated VMs is a safe default. It balances isolation, performance, and ease of management. ### Concurrency Settings The concurrent setting in config.toml controls how many jobs a runner can run simultaneously. Set it based on available CPU and memory. A rule of thumb: concurrent = number of CPU cores - 1 for CPU-bound jobs, but adjust based on memory. For example, on a 4-core VM with 8 GB RAM, you might set concurrent = 3 to leave some headroom. Example config.toml snippet: concurrent = 3 check_interval = 0 [[runners]] name = "docker-runner" url = "https://gitlab.example.com" token = "TOKEN" executor = "docker" [runners.docker] image = "alpine:latest" privileged = false volumes = ["/cache"] Adjust concurrent incrementally. Monitor resource utilization after each change. For example, increase from 3 to 4, wait a day, and check CPU and memory saturation. ### Resource Limits Define resource limits per job to prevent a single job from consuming all host resources. For Docker executor, you can set memory and cpus in the runner configuration or in job definitions. For Kubernetes executor, use resource requests and limits. Example job-level limits for Kubernetes: job: variables: KUBERNETES_CPU_REQUEST: "500m" KUBERNETES_MEMORY_REQUEST: "512Mi" KUBERNETES_CPU_LIMIT: "1" KUBERNETES_MEMORY_LIMIT: "1Gi" This prevents resource starvation and improves stability. Set limits based on your measured job requirements, with some headroom. ### Autoscaling For variable workloads, consider autoscaling. GitLab Runner offers several options: - docker+machine (deprecated): Uses Docker Machine to create and destroy VMs on demand. - docker-autoscaler executor : The newer, recommended approach for cloud providers. - Kubernetes executor with cluster autoscaler : Scales pods and nodes automatically. Autoscaling helps manage cost but requires careful tuning to avoid underprovisioning during spikes. For example, set a minimum number of idle runners to handle sudden bursts, and configure scale-down delays to prevent thrashing. ### Scoped Pilot Before rolling out changes globally, pilot on a single runner or project. Use a separate runner registered with a new token and limit it to a test project. Monitor the pilot for at least a week before expanding. For example, create a new runner with concurrent = 2 and assign it only to a low-traffic project. Observe how it performs relative to your baseline. ## Verification and Diagnostics After implementing capacity changes, verify they work and diagnose any issues. ### Check Runner Status Use the GitLab UI or API to confirm runners are online and accepting jobs. Example command: curl --header "PRIVATE-TOKEN: " "https://gitlab.example.com/api/v4/runners/all?scope=active" Expected: a list of runners with online: true and status: "online" . If a runner is offline, check its logs and host status. ### Monitor Queue Times Track how long jobs wait in pending state. You can query the GitLab API for pipeline durations, or use Prometheus metrics if enabled. A healthy system should have average queue time under 1 minute. For example, if you see average queue times of 5 minutes, you need more capacity. ### Resource Utilization On runner hosts, monitor CPU and memory. Use tools like htop , vmstat , or Prometheus node exporter. Example vmstat output: procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 2048576 123456 345678 0 0 10 20 100 200 20 5 70 5 0 If us (user CPU) is consistently above 80%, you need more runners or higher concurrency. ### Test Job Execution Create a simple test job that outputs resource usage inside the container. For Docker executor, you can run: test: script: - nproc - free -h - cat /proc/cpuinfo | grep "cpu cores" Expected output shows the allocated resources, e.g., nproc returns 1 if you set cpus=1 . This verifies that resource limits are applied correctly. ### Logs and Events Check runner logs for errors. On systemd systems: sudo journalctl -u gitlab-runner -f Look for warnings about job timeouts, disk space, or network issues. For example, messages like ERROR: Job failed (system failure): preparing environment: context canceled can indicate resource exhaustion. ## Failure Modes and Recovery Despite planning, capacity issues can arise. Be prepared to handle them. ### Common Failure Modes - Resource exhaustion : Host runs out of CPU or memory, causing jobs to fail or slow down. This often happens when concurrency is set too high for the host resources. - Disk full : Runner host disk fills with job artifacts or Docker images, preventing new jobs. Docker images can accumulate quickly if not cleaned regularly. - Network saturation : Too many concurrent jobs overwhelm network bandwidth, causing pipeline failures, especially for jobs that pull large dependencies. - Queue buildup : Sudden spike in jobs leads to long waiting times and timeouts. This can cascade if jobs have strict timeouts. - Autoscaling lag : Autoscaler cannot provision resources fast enough, leading to job failures, particularly in cloud environments with slow instance startup times. ### Rollback Procedures If a configuration change causes instability, revert to the previous known-good configuration. Keep backups of config.toml before changes. Example: cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak After rollback, restart the runner: sudo gitlab-runner restart For example, if you increased concurrent from 4 to 8 and started seeing memory issues, revert to 4 and restart. ### Recovery Checks After recovering from a failure, verify: - Runners are online and jobs are executing. - Queue times have returned to acceptable levels. - Resource utilization is within safe margins. - No data loss or corrupted artifacts. ### Capacity Cushion Always maintain a capacity cushion (e.g., 20% spare capacity) to absorb sudden load. This prevents cascading failures during traffic spikes. For example, if your peak demand is 25 concurrent jobs, provision for 30 to handle unexpected bursts. ## Common Pitfalls and How to Avoid Them Capacity planning is full of subtle traps. Here are the most common mistakes and how to steer clear. ### Overestimating or Underestimating Concurrency Pitfall : Setting concurrent too high leads to resource contention and failed jobs; setting it too low wastes resources and causes long queues. Why it happens : Planners often use a simple rule of thumb without considering actual job resource usage or host capacity. How to avoid : Measure actual resource consumption per job type. Use monitoring to determine the maximum sustainable concurrency on a given host. Start conservative and increase incrementally. Recovery : If you have overprovisioned, reduce concurrent and restart. If underprovisioned, add more runners or increase concurrency with adequate host resources. ### Ignoring Peak Load Variability Pitfall : Sizing for average load leads to poor performance during spikes. Why it happens : Teams look at daily or weekly averages and miss periodic peaks. How to avoid : Analyze historical job data to identify peak periods (deploy days, end of sprint, etc.). Plan for the 95th percentile of load. Recovery : Implement autoscaling to handle peaks, or manually add temporary runners before known high-load events. ### Not Setting Resource Limits Pitfall : A single job consumes excessive resources, starving other jobs on the same runner. Why it happens : Default configurations often have no limits, and developers may not be aware of their job's resource needs. How to avoid : Set resource limits in runner configuration or job definitions. Educate teams on estimating resource requirements. Recovery : Add limits to new jobs immediately. For existing jobs, gradually apply limits based on measured usage to avoid breaking them. ### Neglecting Disk Cleanup Pitfall : Runner disks fill up with old Docker images and job artifacts, causing failures. Why it happens : Docker images accumulate with every pipeline, and cleanup scripts are often missing. How to avoid : Schedule regular cleanup of unused Docker images and old artifacts. For Docker executor, use docker system prune -f weekly or set up a cron job. Recovery : If disk is full, manually remove images and artifacts, then implement a cleanup policy. ### Failing to Monitor Proactively Pitfall : Issues are detected only after users complain or pipelines fail. Why it happens : Teams lack proper monitoring or alerting, or they ignore early warning signs. How to avoid : Set up monitoring dashboards for runner hosts and GitLab metrics. Configure alerts for high queue times, CPU usage, and disk space. Recovery : Implement monitoring retroactively. Start with basic checks like queue time and runner status, then expand. ## Operations Checklist Use this checklist for ongoing capacity management. Each item should have a designated owner and a review frequency to ensure accountability. ### Daily Checks - Monitor runner status and queue lengths : Check the GitLab UI for any offline runners or jobs stuck in pending. Owner: CI/CD Engineer. Frequency: Daily. - Check disk space on runner hosts : Ensure usage is below threshold. Owner: System Administrator. Frequency: Daily. - Review CI job failure rates : Look for spikes in failures. Owner: DevOps Lead. Frequency: Daily. ### Weekly Checks - Review average job duration and queue time trends : Identify any upward trends. Owner: CI/CD Engineer. Frequency: Weekly. - Check resource utilization (CPU, memory) over the week : Compare with baselines. Owner: System Administrator. Frequency: Weekly. - Identify any jobs consuming excessive resources : Flag for optimization. Owner: DevOps Lead. Frequency: Weekly. ### Monthly Checks - Re-evaluate capacity needs based on project growth : Look at pipeline count trends. Owner: Engineering Manager. Frequency: Monthly. - Test autoscaling configuration if applicable : Ensure scale-up and scale-down work correctly. Owner: CI/CD Engineer. Frequency: Monthly. - Update runner version if needed : Check for security patches and feature improvements. Owner: DevOps Lead. Frequency: Monthly. ### Proactive Adjustments - Add runners when average queue time exceeds 2 minutes. - Increase concurrency when CPU utilization consistently exceeds 70%. - Add disk space when usage exceeds 80%. - Review safety margins during peak periods, especially before known high-load events like releases.

Metric	Safe Threshold	Action	Owner
Average queue time	< 2 minutes	Add runners if exceeded	CI/CD Engineer
CPU utilization	< 70%	Increase concurrency or add runners	System Administrator
Disk usage	< 80%	Clean up artifacts, expand disk	System Administrator
Job failure rate	< 5%	Investigate causes, adjust resources	DevOps Lead

## Conclusion GitLab Runner capacity planning is essential for maintaining fast, reliable CI/CD pipelines. By inventorying your environment, configuring runners safely, monitoring performance, and preparing for failures, you can ensure your team's productivity remains high. Start with a baseline, make incremental changes, and always keep a capacity cushion. Implement the operations checklist with clear owners and review frequencies to stay ahead of growth. With these practices, you can scale your GitLab Runner fleet effectively and avoid costly downtime. Regularly revisit your capacity plan, at least monthly, to account for team growth, new projects, and changing workloads. Remember, capacity planning is not a one-time task but an ongoing process of measurement and adjustment.