GitLab CI/CD Capacity Planning with Practical Examples

Intro

GitLab CI/CD pipelines are the backbone of modern software delivery, but they can become a bottleneck if their capacity is not planned carefully. Capacity planning is not a one-time task; it is an ongoing process of estimating resource needs, monitoring usage, and adjusting configurations as your team and projects grow. Without proper planning, pipelines slow down, jobs queue excessively, runner resources are wasted, and developers waste time waiting for feedback.

This guide provides a practical approach to GitLab CI/CD capacity planning. It covers how to inventory your current environment, estimate resource requirements, configure runners safely, verify your setup, anticipate failure modes, and establish an operational checklist. Throughout, we include concrete commands, configuration snippets, and examples with expected outputs. By the end, you will have a repeatable framework to keep your CI/CD infrastructure running smoothly as demand changes.

Version and Environment Inventory

Before making any changes, you must understand your current GitLab setup and runner infrastructure. This includes the GitLab version, runner types and configuration, project mix, and existing resource utilization. A clear inventory helps you identify bottlenecks and plan for growth.

GitLab Server Version and Topology

First, record the GitLab version and the server's role. GitLab instances can be self-managed or hosted on GitLab.com. For self-managed, SSH to the server and run:

gitlab-rake gitlab:env:info

Expected output excerpt:

System information
System:         Ubuntu 20.04
Current User:   git
Using RVM:      no
Ruby Version:   2.7.6p219
Gem Version:    3.1.6
Bundler Version:2.3.5
Rake Version:   13.0.6
Redis Version:  6.2.7
Git Version:    2.33.1
Go Version:     unknown

GitLab information
Version:        15.11.2
Revision:       abc123
Directory:      /opt/gitlab
DB Adapter:     PostgreSQL
DB Version:     13.8
URL:            https://gitlab.example.com
HTTP Clone URL: https://gitlab.example.com/some-group/some-project.git
SSH Clone URL:  [email protected]:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: google_oauth2

Take note of the GitLab version, database, and external integrations. GitLab.com users can skip this step but should note that GitLab.com has shared runners with defined quotas (e.g., free tier has 400 CI minutes per month for private projects, while paid tiers get more). Self-managed instances require you to manage your own runners and their underlying infrastructure.

Runner Inventory

Runners execute the actual CI/CD jobs. List the registered runners and their configuration:

gitlab-runner list

Example output:

Listing configured runners    ConfigFile=/etc/gitlab-runner/config.toml
my-runner-1                   Executor=docker Token=abc123 URL=https://gitlab.example.com
my-runner-2                   Executor=shell  Token=def456 URL=https://gitlab.example.com

For each runner, check its /etc/gitlab-runner/config.toml file to understand its executor type, concurrency, and other settings. A typical Docker executor runner configuration might look like this:

concurrent = 4
check_interval = 0

[[runners]]
  name = "my-runner-1"
  url = "https://gitlab.example.com"
  token = "abc123"
  executor = "docker"
  [runners.docker]
    image = "alpine:latest"
    privileged = false
    volumes = ["/cache"]
    shm_size = 0

The concurrent setting limits how many jobs this runner can run simultaneously. Knowing your current executor types is crucial because different executors have different performance characteristics and capacity limits. For example, Docker executors can be resource-intensive due to container overhead, while shell executors run jobs directly on the host.

Project and Pipeline Utilization

Identify how many projects are using CI/CD and their pipeline frequency. In GitLab, you can analyze pipeline activity via the UI (e.g., the project's CI/CD Analytics page) or via the API. For a self-managed instance, use the Rails console:

gitlab-rails console

Then query:

Project.where('ci_pipelines_count > 0').count

This gives the number of projects with at least one pipeline. To find the total pipeline count per day (approximate):

Ci::Pipeline.where('created_at > ?', 1.week.ago).group("DATE(created_at)").count

Expected output may look like:

{"2023-05-01"=>45, "2023-05-02"=>52, "2023-05-03"=>61}

Also check job wait times. In the GitLab UI, navigate to a project's CI/CD analytics, or use the API to list jobs and their queued_duration field. High queued durations indicate a lack of runner capacity.

Capacity Planning Fundamentals

Capacity planning is about matching supply (runner resources) with demand (pipeline jobs) while maintaining acceptable performance. This section explains how to estimate resource requirements based on your workload.

Understanding Workload Patterns

Your CI/CD workload is driven by the number of jobs and their resource consumption. Start by categorizing jobs:

• Build jobs: Compile code, run tests, build containers. They can be CPU and memory intensive.
• Test jobs: Run automated tests, often parallelizable. They may require specific services (databases, caches).
• Deploy jobs: Push artifacts or deploy to environments. Usually short but may need network access.
• Utility jobs: Lint, scan, or clean up. Typically lightweight.

Analyze your existing pipelines to understand the distribution of job types and their average duration. For example, you might find that 70% of jobs are tests, 20% are builds, and 10% are deploys. This breakdown helps you decide whether to use specialized runners or adjust concurrency.

Estimating Resource Needs

To estimate CPU and memory requirements, you can measure actual job usage. If you use Docker executors, use docker stats during a job run. For example, a typical Node.js build job might use:

• CPU: 1.5 cores average
• Memory: 2 GB peak

If you have a runner host with 4 CPU cores and 8 GB RAM, you can safely run 2 such jobs concurrently (4 cores / 1.5 cores per job ≈ 2.6, but leave overhead). Adjust the concurrent setting accordingly.

For GitLab.com shared runners, resource availability is managed by GitLab, but you can control usage via CI/CD minutes and job timeouts.

Calculating Runner Capacity

A simple formula for runner capacity:

Max concurrent jobs = floor(available CPU cores / average CPU per job)

and also ensure memory is not overcommitted:

Max concurrent jobs = floor(available memory / average memory per job)

Take the minimum of the two. For example, a host with 8 cores and 16 GB RAM, and jobs averaging 2 cores and 3 GB RAM, can run max 4 jobs concurrently (8/2=4, 16/3=5.3, min=4).

Always reserve some resources for the OS and runner overhead. A common practice is to leave at least 1 core and 1-2 GB RAM free.

Safe Configuration Path

Capacity planning involves making configuration changes that can impact all pipeline users. To do this safely, adopt a scoped, incremental approach. Start with a narrow pilot group, measure the impact, and then roll out gradually.

Understanding Runner Scaling Options

There are two main ways to scale GitLab CI/CD capacity:

• Horizontal scaling: Add more runners or increase concurrency on existing runners. This is the most common approach.
• Vertical scaling: Increase the resources (CPU, memory) of the machine hosting the runner. This helps when individual jobs are resource-heavy.

For self-managed runners, you can deploy them on dedicated machines or use autoscaling (e.g., with Docker Machine or the GitLab Runner Autoscaler on cloud platforms). For GitLab.com, you can purchase additional CI minutes or use your own runners.

Example: Adjusting Concurrency in config.toml

Suppose you have a Docker executor runner on a host with 4 CPUs and 8 GB RAM. The current concurrent is 1, but you observe job queuing. You can increase concurrency to 2 or 3, but you must ensure the host has enough resources to avoid overcommitment.

Edit /etc/gitlab-runner/config.toml and set:

concurrent = 3

Then restart the runner:

sudo gitlab-runner restart

Check the runner status:

sudo gitlab-runner status

Expected output:

gitlab-runner: Service is running

Monitor the host's CPU and memory during a typical pipeline to ensure jobs don't fail due to resource limits. For Docker executors, consider setting resource limits per job via [runners.docker] options such as cpu_limit and memory_limit to prevent a single job from consuming all resources.

Example snippet:

[runners.docker]
  cpu_limit = "1"
  memory_limit = "2g"

This limits each job container to 1 CPU and 2 GB memory, preventing one job from starving others.

Adding a New Runner

If a single host cannot handle the load, add another runner. This can be another machine or a containerized runner on the same host. To register a new runner:

1. In GitLab, go to your project or group's Settings > CI/CD > Runners and get the registration token.
2. On the target machine, run:

sudo gitlab-runner register \
  --url https://gitlab.example.com \
  --registration-token PROJECT_REGISTRATION_TOKEN \
  --executor docker \
  --description "additional-runner" \
  --docker-image alpine:latest

Follow the prompts to set tags and other options. After registration, the runner appears in the list:

gitlab-runner list

Expected output includes the new runner.

Using Tags to Control Job Distribution

Tags allow you to route jobs to specific runners. For example, you may have a dedicated runner for heavy builds with more resources. In your .gitlab-ci.yml, you can specify:

job:
  tags:
    - heavy-build

Ensure the runner is configured with the same tag:

[[runners]]
  name = "heavy-build-runner"
  tags = ["heavy-build"]

This prevents light jobs from occupying the heavy runner and ensures heavy jobs get the resources they need.

Verification and Diagnostics

After making configuration changes, you must verify that the system behaves as expected and that capacity meets demand. This section provides concrete commands and checks.

Monitor Job Queue and Execution

Use the GitLab API to check the number of pending jobs for a specific runner. First, get the runner's ID from the UI or list runners:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners"

Find your runner ID, then query its jobs:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/<runner_id>/jobs?status=running"

This shows currently running jobs. To see pending jobs for all runners, use:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/jobs?scope=pending"

If pending jobs exist, it means runners are not picking them up fast enough. Check runner availability and concurrency.

Inspect Runner Logs

Runner logs are essential for diagnosing issues. Use:

sudo journalctl -u gitlab-runner -f

This tails the runner service logs. Look for errors such as job timeouts, image pull failures, or disk space issues. For Docker executor, also check container logs:

docker logs <container-id>

Measure Resource Utilization

On the runner host, use standard Linux tools:

top

or

vmstat 1

Observe CPU load, memory usage, and I/O. For Docker executors, docker stats gives per-container metrics:

docker stats

Expected output:

CONTAINER ID   NAME              CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O   PIDS
abc123         runner-abc-job1   95.00%    1.2GiB / 3.8GiB       31.58%    1.2MB / 2.3MB     0B / 0B     25
def456         runner-def-job2   12.00%    800MiB / 3.8GiB       20.53%    500kB / 1.1MB     0B / 0B     18

High CPU or memory usage may indicate that you need to reduce concurrency or increase resources.

Check Pipeline Performance

Use GitLab's built-in CI/CD analytics to view pipeline duration, success rates, and job wait times. For a project, navigate to Analytics > CI/CD Analytics. You can also use the API to get pipeline data:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/:id/pipelines?per_page=20"

Look for pipelines with long durations or high queued durations. A queued duration over a few minutes suggests insufficient runner capacity.

Common Pitfalls and How to Avoid Them

Even with careful planning, teams often fall into traps that degrade CI/CD performance. Here are the most common pitfalls, why they happen, and how to avoid or recover from them.

Pitfall 1: Overcommitting Runner Resources

Why it happens: Teams increase concurrent beyond what the host can support, assuming more parallelism is always better. This leads to resource contention, job timeouts, and cascading failures.

How to avoid: Always calculate capacity based on actual job resource usage (as shown earlier). Test with a gradual increase, monitoring CPU and memory. Set resource limits per job container to prevent one job from hogging resources.

Recovery: If you have overcommitted, reduce concurrent and restart the runner. Clean up any zombie processes or containers. Monitor for improvement.

Pitfall 2: Ignoring Job Queue Metrics

Why it happens: Teams focus on pipeline success but overlook job wait times. Over time, queue depth grows silently, causing frustration.

How to avoid: Set up alerts for pending job counts. You can use GitLab's built-in monitoring or external tools like Prometheus. Review queue metrics weekly. Assign an owner, e.g., the DevOps engineer, to review queue metrics every Monday morning.

Recovery: If you notice high queue depth, immediately check runner availability and capacity. Add runners or increase concurrency as needed.

Pitfall 3: Using the Wrong Executor Type

Why it happens: Some teams default to Docker executors without considering whether shell or Kubernetes executors would be more efficient for their workload.

How to avoid: Evaluate your job types. For simple scripts, shell executor may be faster due to no container overhead. For complex environments, Docker provides isolation. Kubernetes executor offers dynamic scaling but adds complexity.

Recovery: If jobs are slow due to executor overhead, consider changing executor type for a subset of runners and tagging jobs accordingly.

Pitfall 4: Neglecting Disk Space

Why it happens: Docker builds and caches accumulate over time, filling up runner disks. This causes jobs to fail with "no space left on device".

How to avoid: Schedule regular cleanup via cron or use GitLab's cleanup policies for container registry. Set up disk usage alerts.

Recovery: Manually prune Docker resources:

docker system prune -a --volumes

Pitfall 5: Hardcoding Runner Tags Without Strategy

Why it happens: Teams add tags ad hoc, leading to job routing confusion and underutilized runners.

How to avoid: Define a tagging strategy upfront. For example, use tags like small, medium, heavy based on resource requirements. Document the mapping.

Recovery: Audit existing tags and consolidate. Update .gitlab-ci.yml files to align with the new strategy.

Failure Modes and Recovery

Even with careful planning, things can go wrong. This section describes common failure modes in GitLab CI/CD capacity and how to recover from them.

Runner Overload and Job Failures

Symptom: Jobs fail with errors like "no space left on device" or "connection refused" when starting containers. The runner host may be out of disk space or memory.

Diagnosis: Check disk space with df -h and memory with free -h. If disk is full, clean up Docker images and volumes:

docker system prune -a --volumes

Recovery: Reduce concurrent temporarily, add disk space, or move runner cache to a larger volume. Ensure you monitor afterward.

Unresponsive Runner Service

Symptom: Jobs remain pending, and the runner does not pick them up. gitlab-runner status shows the service is not running.

Diagnosis: Check runner logs for crash reasons. Restart the service:

sudo gitlab-runner restart

Recovery: If the runner fails to start, verify the configuration file syntax with gitlab-runner verify. Look for misconfigurations like invalid TOML or wrong executor settings. Roll back to a known-good configuration if needed.

Auto-scaling Failures (Cloud Runners)

If using autoscaling (e.g., with AWS Spot Instances), instances may be terminated unexpectedly, causing job failures.

Symptom: Jobs fail with "runner system failure" or "job aborted".

Diagnosis: Check the runner's autoscaling configuration and cloud provider logs. Ensure you have a fallback capacity with on-demand instances.

Recovery: Adjust the autoscaler settings to maintain a minimum number of instances. Test with a simple job to confirm stability.

Network Issues Between GitLab and Runners

Symptom: Runners lose connection to GitLab, leading to stuck jobs.

Diagnosis: From the runner host, test connectivity:

curl -I https://gitlab.example.com

If it fails, check firewall rules and network routing.

Recovery: Ensure outbound HTTPS is allowed. If behind a proxy, configure the runner accordingly.

Operations Checklist

Use this checklist as a repeatable guide for ongoing capacity management. Assign a single accountable owner for each item and set a review frequency.

Each owner should document their review findings and track action items. For example, the DevOps Engineer might record that runner disk usage reached 80% on 2023-06-15 and schedule cleanup for the next week.

Conclusion

Capacity planning for GitLab CI/CD is an essential practice for maintaining fast, reliable software delivery. By taking a systematic approach, you can avoid common pitfalls such as overloading runners, wasting resources, and delaying developer feedback.

In this guide, you learned how to inventory your current environment, safely configure runners, verify capacity with concrete diagnostics, recover from failures, and maintain an operational checklist. The key takeaway is to start small: choose a narrow pilot, measure impact, and expand gradually. Continuously monitor your pipelines and adjust as your team grows.

For next steps, consider setting up automated monitoring for runner health and job queues, exploring autoscaling options if you are on a cloud platform, and regularly reviewing your CI/CD analytics. With these practices, you can keep your GitLab CI/CD infrastructure running smoothly and focus on delivering value to your users.