Practical GitLab CI/CD Monitoring and Alerts

Intro

When a pipeline fails at 3 a.m., the difference between a 10-minute recovery and a three-hour firefight often comes down to what you can observe before you change anything. GitLab CI/CD provides native monitoring, metrics, and alerting capabilities, but many teams only scratch the surface. They see a red pipeline, restart the job, and hope for the best. That is not monitoring; that is guessing.

This article lays out a practical, command-first approach to GitLab CI/CD monitoring and alerts. It is written for developers, platform engineers, and technical startup teams who need to move from an observed problem to a verified result. You will find concrete commands, expected outputs, and failure signals for key areas: version and environment inventory, safe configuration, verification and diagnostics, failure modes and recovery, and an operations checklist. Each section includes a real-world example with placeholders you can adapt to your own environment.

The goal is operational safety: observe before changing, limit blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before you can monitor anything, you must know exactly what you are running and how it is deployed. This section covers how to inventory your GitLab installation, runners, and environment in a read-only, reversible way.

Identify the installed version and topology

Start by determining the GitLab version and deployment type (self-managed or GitLab.com). Use the GitLab API or the Rails console for self-managed instances.

Read-only command (API):

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/version"

Expected output:

{
  "version": "17.5.2",
  "revision": "c7a5f3b"
}

Failure signal: If you get a 401, your token lacks read_api scope. If you get a connection refused, the GitLab service may not be running on that host.

For self-managed instances, you can also check the version locally:

sudo gitlab-rake gitlab:env:info

Expected output snippet:

GitLab information
Version:        17.5.2
Directory:      /opt/gitlab
DB Adapter:     PostgreSQL

Inventory runners and their status

Runners execute CI jobs. You need to know which runners are available and whether they are online.

Command:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/all?per_page=50"

Expected output (truncated):

[
  {
    "id": 42,
    "description": "docker-shared-runner-1",
    "active": true,
    "online": true,
    "status": "online"
  },
  {
    "id": 43,
    "description": "kubernetes-runner-prod",
    "active": true,
    "online": false,
    "status": "offline"
  }
]

Failure signal: A runner with online: false means jobs assigned to it will be stuck in pending. Investigate the runner machine or Kubernetes pod.

Blast radius and recovery for inventory

Inventory commands are read-only and safe. The blast radius is zero. However, never use tokens with write privileges for inventory. Use a dedicated read-only token, or a project access token with read_api scope for API calls.

Recovery: If you accidentally run a write command, revert any changes immediately and rotate tokens if exposed.

Safe Configuration Path

Configuration changes affect monitoring visibility. This section shows how to enable Prometheus metrics, configure alerting, and adjust settings without breaking your pipeline.

Enable Prometheus metrics on GitLab

GitLab self-managed can expose Prometheus metrics if configured. By default, the bundled Prometheus is enabled, but you may need to enable the exporter for GitLab itself.

Check current setting (read-only):

sudo gitlab-rails runner "puts Gitlab::CurrentSettings.current_application_settings.prometheus_metrics_enabled"

Expected output:

true

If it returns false, you can enable it via the Admin Area UI under Settings > Metrics and profiling, or via the Rails console (with caution). The smallest justified change is to toggle the setting in the UI; a Rails console change is riskier.

Before/after verification:

curl http://localhost:9090/metrics | head -n 20

Expected output contains:

# HELP gitlab_ci_pipeline_size_avg Average pipeline size
# TYPE gitlab_ci_pipeline_size_avg gauge

Configure external alerting

GitLab can send alerts to webhooks or integrations. A common pattern is to configure a webhook to send pipeline failure events to a chat tool or incident management system.

Example: project webhook for pipeline events.

1. Go to your project's Settings > Webhooks.
2. Add the URL and select "Pipeline events".
3. Set a secret token.

Verify with a test event:

# In the webhook settings, click "Test" and select "Pipeline events"

Expected signal: Your receiver gets a JSON payload with object_kind: pipeline and object_attributes.status equal to failed or success.

Blast radius and recovery

A webhook configuration change only affects that project. To revert, delete the webhook. If you accidentally trigger a test event that pages someone, acknowledge the false alarm in your incident channel.

Verification and Diagnostics

Once monitoring is in place, you need to verify that it works and diagnose problems when it does not. This section provides commands to check pipeline status, inspect job logs, and query metrics.

Check pipeline status

Use the GitLab API to list recent pipelines and their statuses.

Command:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/123/pipelines?per_page=5"

Expected output:

[
  {
    "id": 98765,
    "status": "failed",
    "ref": "main",
    "sha": "a1b2c3d4...",
    "created_at": "2025-03-15T10:20:30Z"
  },
  {
    "id": 98764,
    "status": "success",
    "ref": "main",
    "sha": "e5f6a7b8..."
  }
]

Failure signal: A pipeline with status failed requires immediate diagnosis. Check the failed job's log.

Inspect job logs

Command:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/123/jobs/456/trace"

Expected output: Raw job log text. Look for error patterns such as ERROR, fatal, or exit code non-zero.

Diagnostic command for runner-side errors:

sudo gitlab-runner status

Expected output:

gitlab-runner: Service is running

If the runner is stopped, start it and check its logs at /var/log/gitlab-runner/.

Query Prometheus metrics

If you have Prometheus scraping GitLab, use PromQL to diagnose CI-specific metrics.

Example: pipeline duration trend.

avg_over_time(gitlab_ci_pipeline_duration_seconds[1h])

Expected output: A graph showing average pipeline duration over the last hour. A spike may indicate infrastructure issues.

Example: runner availability.

gitlab_runner_autoscaling_machine_states{state="available"}

If this metric drops to zero, your autoscaling group may have no capacity.

Failure Modes and Recovery

Pipelines fail for many reasons. This section covers common failure modes, how to detect them, and recovery steps.

Failure mode: Runner offline

Observation: Jobs stay pending for an extended time.

Command:

curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/123/jobs?scope=pending"

Expected output: A list of jobs with status pending. No jobs being executed.

Diagnosis: Check runner status using gitlab-runner status or the Kubernetes pod logs.

Recovery: Restart the runner service or scale up the runner pod. For Kubernetes runners, check kubectl get pods -n gitlab-runner and restart the runner pod if necessary.

Failure mode: Pipeline fails due to environment variable missing

Observation: Job fails with command not found or a missing variable error.

Command:

# In the job log, search for "missing" or "undefined"
curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/123/jobs/456/trace" | grep -i "missing\|undefined"

Expected output: A line like undefined variable: DEPLOY_TOKEN.

Recovery: Add the variable in project CI/CD settings (Settings > CI/CD > Variables) with the appropriate scope and protection.

Verification: Re-run the pipeline and confirm the job passes.

Failure mode: Disk space full on runner

Observation: Jobs fail with No space left on device in the log.

Diagnosis: SSH into the runner and run df -h.

Recovery: Clean up Docker cache (docker system prune -f) or increase disk size. For Kubernetes runners, check PVC capacity.

Verification: Re-run a lightweight job to confirm the runner can execute.

Common Pitfalls and How to Avoid Them

Pitfall 1: Using shared secrets in monitoring scripts

Why it happens: Teams hardcode tokens or passwords in shell scripts for convenience.

How to avoid: Use GitLab CI/CD variables masked and protected. For external scripts, use environment variables or secret managers.

Recovery: If a secret leaks, rotate it immediately and audit access logs.

Pitfall 2: Ignoring runner capacity metrics

Why it happens: Teams focus only on pipeline success/failure and miss slow queue times.

How to avoid: Set up alerts on gitlab_runner_autoscaling_machine_states and queue depth metrics. Use Prometheus alert rules: gitlab_ci_pipeline_queue_duration_seconds > 300.

Recovery: If alerts fire, scale runners or optimize job scheduling.

Pitfall 3: Over-alerting on every pipeline failure

Why it happens: Teams configure webhooks to alert on all pipeline events, leading to alert fatigue.

How to avoid: Alert only on failures on the default branch or protected branches, and use severity levels. In webhook settings, filter by branch or use a custom script to filter payloads.

Recovery: Fine-tune alert rules and mute non-critical notifications.

Pitfall 4: Not version-pinning monitoring tools

Why it happens: Teams install agents or exporters without verifying compatibility with their GitLab version.

How to avoid: Check the compatibility matrix in the GitLab documentation before installing any monitoring integration.

Recovery: If an incompatible version is installed, roll back to a supported version and verify metrics appear.

Operations Checklist

Here is a practical checklist for ongoing GitLab CI/CD monitoring. Assign an owner for each item and review frequency.

How to use this checklist:

• Each owner is a single accountable person, not a group. If the owner changes, update the table.
• Review frequencies are minimums; adjust based on your team's needs.
• Automate as many checks as possible using CI schedules or cron jobs.

Incident Walkthrough: Diagnosing a Flaky Pipeline

Let's apply these practices to a realistic scenario.

Scenario: Your team notices that the deploy-to-staging pipeline fails intermittently. Failures occur about once every ten runs, always at the same job, with the error connection refused when pushing to a Docker registry.

Step 1: Observe (read-only)

curl --header "PRIVATE-TOKEN: <token>" "https://gitlab.example.com/api/v4/projects/123/pipelines?per_page=20&status=failed"

Output: Several failed pipelines, all in job docker-push. Retrieve the job trace:

curl --header "PRIVATE-TOKEN: <token>" "https://gitlab.example.com/api/v4/projects/123/jobs/789/trace"

Log snippet:

Error: error building image: error building stage: failed to execute command: waiting for connection: dial tcp 10.0.1.23:2376: connect: connection refused

Step 2: Diagnose

The error points to the Docker daemon on the runner. Check runner status:

sudo gitlab-runner status

Output: gitlab-runner: Service is running

Check disk space:

df -h

Output shows: /var/lib/docker 92% used. High disk usage can cause Docker daemon instability.

Step 3: Smallest justified change

Clean up Docker cache:

docker system prune -af --volumes

Expected output:

Deleted Containers:
...
Total reclaimed space: 15.6GB

Step 4: Verify

Re-run the failed pipeline manually:

# In GitLab UI, click "Retry" on the failed pipeline, or use API
curl --request POST --header "PRIVATE-TOKEN: <token>" "https://gitlab.example.com/api/v4/projects/123/pipelines/98765/retry"

Expected outcome: Pipeline succeeds.

Step 5: Prevent recurrence

Set up an alert on disk usage for runners. Use Prometheus node exporter or a simple cron job:

# Cron job on runner host runs hourly
0 * * * * /usr/bin/df -h /var/lib/docker | awk 'NR==2 {print $5}' | mail -s "Runner disk usage" [email protected]

This alert ensures you know before disk space becomes critical.

Conclusion

GitLab CI/CD monitoring and alerts are only useful when commands are version-scoped, observable, and reversible. Copying a command without checking prerequisites and expected output is not an operations procedure; it is a gamble. The practices in this article give you a framework for inventory, safe configuration, verification, failure recovery, and ongoing operations.

Start with one low-risk verification: inventory your GitLab version and runner status. Record the current state, run the documented read-only check, and compare with the expected signal. Then gradually add Prometheus metrics, alerting, and the operations checklist. Review dependencies such as GitLab Runner, Docker, and Kubernetes only where they affect your monitoring and recovery.

A reliable monitoring workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Assign a single owner for each checklist item and revisit it regularly. This is the difference between reacting to outages and preventing them.