GitLab Runner Monitoring and Alerts: A Practical Guide for Reliable CI/CD

Intro

GitLab Runner is the workhorse of GitLab CI/CD. It executes jobs on your infrastructure, whether that is a bare-metal server, a virtual machine, a Docker container, or a Kubernetes cluster. When runners fail silently, misbehave under load, or drift out of sync with the GitLab instance, pipelines break and teams lose trust in their automation.

Monitoring and alerting for GitLab Runner closes that gap. It turns raw runner activity into observable signals: which runners are busy, how long jobs wait, whether the runner can reach GitLab, and whether the right executor is actually doing the work. With the right metrics, dashboards, and alerts, an operator can move from "a job failed" to "this runner on this host exhausted its disk at 14:32 and recovered after cleanup" in minutes.

This guide is written for developers, DevOps consultants, and technical startup teams who operate one or more GitLab Runners. It focuses on practical monitoring and alerting: the commands to run, the metrics to collect, the dashboards to build, and the alerts to configure. Every example is concrete and version-scoped, with expected output and a recovery path. No secrets or production identifiers appear in any example.

Throughout, the goal is operational safety: observe before changing, limit the blast radius, verify the result, and document recovery before an incident forces the decision.

Version and Environment Inventory

Before you can monitor a runner effectively, you must know exactly what you are running. A runner's behavior depends on its version, its executor type, and its deployment topology. A Docker executor on Kubernetes behaves differently from a shell executor on a standalone VM, and both differ from the GitLab instance's built-in runners.

Start with read-only discovery. On the runner host, run:

gitlab-runner --version

Example output on a typical installation:

Version:      16.5.0
Git revision: 8b1f2e3d
Git branch:   16-5-stable
GO version:   go1.21.3
Built:        2023-10-20T14:00:00Z
OS/Arch:      linux/amd64

This tells you the runner version, which matters for features like FF_USE_LEGACY_VOLUMES_MOUNTING or the Kubernetes executor's helper_image. Check the GitLab Runner documentation for the version's supported configuration keys before making changes.

Next, list all configured runners on this host:

gitlab-runner list

Expected output includes each runner's name, executor, and token prefix. For example:

Listing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml
my-docker-runner                                    Executor=docker Token=glrt-xxxx URL=https://gitlab.example.com
my-shell-runner                                     Executor=shell Token=glrt-yyyy URL=https://gitlab.example.com

If a runner does not appear here, it may be registered on a different host or managed by the GitLab instance (shared runners). To see all runners from the GitLab side, an administrator can use the API:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/all?per_page=100"

This returns JSON with each runner's id, description, active, online, status, and ip_address. The online field is critical: a runner can be registered but offline. Capture this output before any change to establish a baseline.

Example response snippet:

[
  {
    "id": 42,
    "description": "docker-runner-prod-01",
    "active": true,
    "online": true,
    "status": "online",
    "ip_address": "10.0.1.15"
  }
]

Document the deployment topology: how many runners, which executors, where they run, and which tags they expose. Store this inventory in version control or a wiki so it is available during an incident. Revisit it monthly or whenever a runner is added, removed, or upgraded.

Prerequisites for monitoring: ensure the runner process is running on the host, the Prometheus metrics endpoint is enabled (see next section), and you have SSH or equivalent access to the runner host. Blast radius of any read-only command is zero; no recovery path needed.

Safe Configuration Path

To monitor GitLab Runner with Prometheus, enable the built-in metrics listener. This is a small configuration change in /etc/gitlab-runner/config.toml.

First, check the current configuration:

sudo cat /etc/gitlab-runner/config.toml

Look for a [session_server] or listen_address entry. By default, the metrics listener is disabled. To enable it, edit the file and add or uncomment the listen_address line under the global section:

concurrent = 4
check_interval = 0

[session_server]
  session_timeout = 1800

listen_address = "0.0.0.0:9252"

The listen_address sets the address and port for the Prometheus metrics endpoint. Using 0.0.0.0 exposes the endpoint on all interfaces; restrict to a specific IP if the host is multi-homed. Port 9252 is conventional but can be any free port.

Blast radius: this change exposes metrics over the network. If the host is publicly reachable, restrict access with a firewall or bind to localhost (127.0.0.1:9252) and use a reverse proxy with authentication. Always verify the configuration before restarting the runner to avoid a syntax error that takes the runner offline.

After editing, validate the configuration syntax:

gitlab-runner verify

Expected output:

Verifying runner... is alive

If the syntax is invalid, the command reports an error and the runner will not start. In that case, revert the change before proceeding.

Then restart the runner to apply changes:

sudo systemctl restart gitlab-runner

Verify the metrics endpoint is responding:

curl -s http://localhost:9252/metrics | head

Expected output includes # HELP and # TYPE lines for gitlab_runner_* metrics. For example:

# HELP gitlab_runner_jobs_total Total number of jobs handled by the runner
# TYPE gitlab_runner_jobs_total counter
gitlab_runner_jobs_total{runner="my-docker-runner",state="success"} 123

If you see connection refused, the runner did not bind the port. Check /var/log/gitlab-runner/ for errors and re-check the listen_address syntax.

Recovery: if the runner fails to start after the change, restore the previous config.toml from backup or comment out the listen_address line, then restart. Because this change is additive and scoped, recovery is straightforward.

Verification and Diagnostics

With metrics available, you can diagnose runner health and performance. Start with the most important metrics: job counts, job duration, and failure rates. Use curl to query the endpoint and grep to filter relevant metrics.

For example, to see all counter metrics for failed jobs:

curl -s http://localhost:9252/metrics | grep 'gitlab_runner_jobs_total{.*state="failed"'

Expected output:

gitlab_runner_jobs_total{runner="my-docker-runner",state="failed"} 7

This tells you 7 jobs have failed on this runner since it started. Compare with the total jobs and success count to compute a failure rate. Use the runner's id or name label to isolate a specific runner in a multi-runner host.

Another key metric is job duration. GitLab Runner exposes gitlab_runner_job_duration_seconds as a histogram. To see the total count of jobs and cumulative duration, query:

curl -s http://localhost:9252/metrics | grep 'gitlab_runner_job_duration_seconds'

Sample output:

gitlab_runner_job_duration_seconds_bucket{runner="my-docker-runner",le="60"} 10
gitlab_runner_job_duration_seconds_bucket{runner="my-docker-runner",le="300"} 15
gitlab_runner_job_duration_seconds_bucket{runner="my-docker-runner",le="+Inf"} 23
gitlab_runner_job_duration_seconds_sum{runner="my-docker-runner"} 4500
gitlab_runner_job_duration_seconds_count{runner="my-docker-runner"} 23

To compute the average job duration, divide _sum by _count. In this example, 4500 / 23 = approximately 195 seconds. If your jobs are usually 2 minutes and the average climbs to 10 minutes, you may have resource contention or network latency.

For real-time status, use the gitlab-runner status command on the host:

sudo gitlab-runner status

Example output:

gitlab-runner: Service is running

If the service is not running, check systemd logs:

sudo journalctl -u gitlab-runner -n 50 --no-pager

Look for repeated ERROR entries, connection timeouts to the GitLab instance, or executor failures. Also verify the runner can reach GitLab:

curl -I https://gitlab.example.com

Expected: HTTP/2 200 or similar. If you get a connection error, the runner will not pick up jobs. Network issues are often DNS, firewall, or proxy misconfiguration.

Diagnostics should always include a timestamped log of commands run and outputs seen. Use script or terminal logging to preserve evidence for post-incident review.

Failure Modes and Recovery

Runners fail in predictable ways. The following are common failure modes, how to detect them, and how to recover.

Runner offline

Symptoms: Jobs assigned to the runner stay pending. The GitLab UI shows the runner as offline. The runner host may still be up.

Detection: On the GitLab instance, check runner status via API:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/all?per_page=100" | jq '.[] | select(.description=="my-docker-runner") | {online, status}'

If online is false, the runner is not connected. On the runner host, check the service:

sudo systemctl status gitlab-runner

Recovery: Restart the service:

sudo systemctl restart gitlab-runner

Then verify it is online again via the API. If the runner repeatedly goes offline, check network connectivity and the runner's registration token validity. A runner may lose its connection if the GitLab instance's URL changes or the token is revoked.

Disk full on runner host

Symptoms: Jobs fail with no space left on device, especially for Docker executor where images and caches accumulate.

Detection: On the runner host:

df -h /var/lib/docker

Look for Use% at 100%. Also check GitLab Runner's own build directory:

du -sh /home/gitlab-runner/builds

Recovery: Clean up unused Docker resources:

docker system prune -af --volumes

Warning: This removes all stopped containers, unused networks, dangling images, and unused volumes. For GitLab Runner, it also removes the build cache. Use with caution in production. A safer first step is to prune only dangling images:

docker image prune -f

After cleanup, monitor disk usage and consider adding a cron job or logrotate to prevent recurrence. Set an alert on disk usage above 80%.

Runner picks up jobs but they wait in queue

Symptoms: Jobs remain in pending state even though the runner is online and idle. The runner's concurrent limit may be reached, or the runner is tagged incorrectly.

Detection: In GitLab UI, check the runner's tags. If the job has tags that don't match this runner, it won't be picked. Also check the runner's concurrent setting:

grep 'concurrent' /etc/gitlab-runner/config.toml

If concurrent = 1 and multiple jobs are waiting, the runner processes one at a time.

Recovery: Increase concurrent to a higher number (e.g., 4) if the host has capacity. Edit config.toml, verify, and restart. To match tags, either add the required tag to the runner or adjust the job's tags in .gitlab-ci.yml. Ensure the runner is not assigned to a specific project if it should be shared.

Executor misconfiguration

Symptoms: Jobs fail immediately with executor-specific errors, e.g., "executor not found" or "missing docker image".

Detection: Run a test job or check runner registration:

gitlab-runner list

Confirm the executor is correct. For Docker executor, test Docker on the host:

docker run hello-world

Recovery: Re-register the runner with the correct executor and settings, or edit config.toml manually. For Docker executor, ensure the Docker daemon is running and the runner has access to the Docker socket. After changes, verify with a trivial pipeline job.

Operations Checklist

Use this checklist as a recurring health check for your GitLab Runners. Assign a single owner per item (not a team) and set a review cadence: daily for critical items, weekly for capacity, monthly for inventory.

Run these checks manually or automate them with a cron job that sends results to a dashboard. The owner is responsible for acting on failures and updating the checklist if the environment changes.

Common Pitfalls and How to Avoid Them

Several recurring mistakes undermine GitLab Runner monitoring. Understanding why they happen helps you avoid them.

Ignoring the metrics endpoint entirely

Why it happens: Teams assume the GitLab UI shows enough information. The UI shows job status and runner online/offline, but not performance metrics like job duration, queue time, or failure rate trends.

How to avoid: Enable the Prometheus metrics endpoint as described earlier, even if you don't have a full monitoring stack. The raw metrics are valuable for ad-hoc debugging.

Recovery: If you're already in a failure, enable metrics, restart the runner, and immediately scrape the endpoint to see current counters and histograms.

Alerting on the wrong metrics

Why it happens: Teams set alerts on metrics like gitlab_runner_jobs_total (a counter that always increases) without applying a rate, or alert on a single runner instead of aggregate. This creates noisy or meaningless alerts.

How to avoid: Use PromQL expressions with rate() for counters, and think about what indicates a real problem: sustained failure rate above threshold, job duration above SLO, or offline runner. Test alerts in staging before production.

Recovery: If an alert is too noisy, tune the threshold or silence it temporarily while fixing the root cause. Document the alert's purpose and owner.

Not monitoring the runner host itself

Why it happens: Teams monitor the runner process but forget the host's CPU, memory, disk, and network. A runner can be online but starved of resources, causing slow jobs or timeouts.

How to avoid: Install a node exporter (e.g., Prometheus node_exporter) on each runner host and include host metrics in your dashboards and alerts.

Recovery: If the host is already constrained, reduce concurrency, move some runners to other hosts, or scale up resources. Monitor the effect on job duration.

Running outdated runner versions

Why it happens: Upgrading runners is often neglected because it seems low-risk. However, older runners may lack bug fixes or features that affect performance and security.

How to avoid: Subscribe to GitLab release announcements and schedule upgrades quarterly. Keep a test environment to validate new versions before production.

Recovery: If a runner is outdated and causing issues, upgrade it following the official upgrade guide. Always back up config.toml first.

Overlooking executor-specific issues

Why it happens: Monitoring generic runner metrics misses problems unique to Docker, Kubernetes, or shell executors, such as Docker daemon failures, Kubernetes pod scheduling, or shell environment drift.

How to avoid: Include executor-specific checks in your monitoring. For Docker, monitor docker info and image pull times. For Kubernetes, monitor pod lifecycle and resource usage.

Recovery: When a specific executor fails, check its dependencies and logs before restarting the runner. For Docker, restart the Docker daemon if necessary. For Kubernetes, check node capacity and pod events.

Conclusion

GitLab Runner monitoring and alerts turn a black box into a transparent system. By versioning your inventory, enabling metrics safely, diagnosing with concrete commands, and preparing for common failure modes, you reduce pipeline downtime and increase team confidence in CI/CD.

This guide has provided a repeatable process: discover the environment, enable observability with a minimal change, verify with expected outputs, and document recovery paths. The operations checklist and common pitfalls sections give you a starting point for your own procedures.

As a next step, choose one low-risk verification from this article, such as enabling the metrics endpoint on a single test runner. Record the current state, apply the change, verify the endpoint responds, and observe the metrics. Then expand to all runners and build dashboards and alerts that fit your team's SLOs.

A reliable monitoring workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With these practices, your GitLab Runners will be a dependable foundation for continuous delivery.