GitLab CI/CD Production Operations Checklist with Practical Examples

Intro

Running GitLab CI/CD in production is more than installing the software. It requires a disciplined approach to configuration, monitoring, maintenance, backups, and upgrades. This checklist provides practical, actionable steps to keep your GitLab CI/CD environment reliable, secure, and efficient. We focus on self-managed GitLab instances, whether deployed on bare metal, virtual machines, or cloud instances, and avoid container-specific advice unless explicitly noted. By following this guide, you will establish a baseline for version inventory, implement safe configuration changes, verify system health, plan for failure recovery, and maintain a repeatable operations routine.

Version and Environment Inventory

Before making any changes, document your current setup. This inventory is the foundation for troubleshooting and planning.

Determine component versions

Run the following commands on your GitLab server (assuming an Omnibus installation):

sudo gitlab-rake gitlab:env:info

Expected output includes version lines like:

GitLab version: 16.7.0
GitLab Shell version: 14.33.0
PostgreSQL version: 14.8

Also check the GitLab Runner version on each runner:

gitlab-runner --version

Expected output:

Version:      16.7.0
Git revision: 1234abcd
Git branch:   16-7-stable
GO version:   go1.21.4
Built:        2023-12-18T14:00:00Z
OS/Arch:      linux/amd64

Map your topology

Record how GitLab components are distributed. For a typical single-server Omnibus install, all services run on one host. For larger setups, you may have separate PostgreSQL, Redis, Gitaly, and multiple Runner hosts. Note hostnames, IP addresses, resource allocations, and network connectivity.

Check prerequisites

Ensure the operating system meets GitLab's requirements and that required packages and ports are accessible. Common ports: 80/443 for HTTP/HTTPS, 22 for SSH, and 5050 for container registry (if enabled). Verify with:

sudo ss -tulpn | grep -E ':(80|443|22|5050)'

Expected output shows listening sockets for those ports.

Document your backup and restore procedures

Know where backups are stored, their frequency, and how to restore. A simple check:

sudo ls -lh /var/opt/gitlab/backups/

Expected output shows timestamped backup files, e.g., 1699999999_2023_11_15_16.7.0_gitlab_backup.tar.

Keep this inventory in a version-controlled document and update it on every change. Assign a single owner, such as the DevOps lead, to maintain it and review it quarterly.

Safe Configuration Path

Avoid making broad, untested changes directly in production. Instead, adopt a scoped, incremental approach.

Use configuration management and version control

Store /etc/gitlab/gitlab.rb in a Git repository. Before each change, create a branch and validate the configuration syntax:

sudo gitlab-ctl reconfigure --dry-run

Expected output includes Would reconfigure GitLab... without actually applying changes.

Test on a staging environment

Maintain a staging instance that mirrors production as closely as possible. Apply changes there first, run your test suite, and observe for any issues.

Implement feature flags and gradual rollout

For new CI/CD features, use GitLab's feature flags or environment variables to enable them for a subset of projects. For example, to enable a new runner feature for one project, you can define a CI variable only for that project.

Example: Changing CI/CD settings

Suppose you want to increase the artifact expiration time. Instead of changing globally, set it per project in .gitlab-ci.yml:

default:
  artifacts:
    expire_in: 1 week

Validate the YAML syntax:

python3 -c "import yaml, sys; yaml.safe_load(open('.gitlab-ci.yml'))"

No output means syntax is valid.

Document rollback procedure

For every change, note the previous configuration and how to revert. Use git revert or restore the previous gitlab.rb and reconfigure.

By following a safe configuration path, you reduce the risk of service disruption and make troubleshooting easier. The configuration owner should approve all changes and review them monthly.

Verification and Diagnostics

After applying changes, verify that the system is functioning correctly. Use built-in commands and monitoring tools.

Check service health

Run:

sudo gitlab-ctl status

Expected output shows each service as run: service_name: (pid) 123s; run: log: (pid) 456s.

Run self-checks

GitLab provides several checks:

sudo gitlab-rake gitlab:check

This runs various sanity checks, such as database connectivity and repository integrity. Look for lines like Checking GitLab ... Finished with no errors.

Monitor key metrics

Use Prometheus metrics exposed by GitLab (if enabled) or external monitoring. For example, to check background jobs queue:

sudo gitlab-rails runner 'puts Sidekiq::Queue.new.size'

Expected output: 0 (or a small number). High numbers indicate problems.

Check CI/CD pipeline execution

Create a test project with a simple .gitlab-ci.yml and run a pipeline. Verify that the job completes and expected artifacts are produced.

Verify backup integrity

Periodically test restore in a sandbox environment. A backup that cannot be restored is useless. Assign the operations engineer to perform a restore test monthly and document the results.

Keep logs accessible and review them regularly:

sudo gitlab-ctl tail

Use this verification section each time you make a change or on a schedule to catch issues early.

Failure Modes and Recovery

Production systems will fail. Plan for common failure modes and know how to recover.

Disk space exhaustion

Symptom: GitLab becomes unresponsive, logs show No space left on device.

Recovery: Identify large files:

sudo du -sh /var/opt/gitlab/* | sort -h

Clean up old artifacts, logs, or container images. If needed, expand the disk. Set up monitoring alerts for disk usage above 80% to avoid complete exhaustion.

Database corruption

Symptom: GitLab errors, PostgreSQL logs show corruption.

Recovery: Restore from the most recent good backup. Before restoring, stop services:

sudo gitlab-ctl stop
sudo gitlab-backup restore BACKUP=1699999999
sudo gitlab-ctl start

Then verify with sudo gitlab-rake gitlab:check.

Configuration mistake

Symptom: Services fail after reconfigure.

Recovery: Revert gitlab.rb to the previous known-good version and run sudo gitlab-ctl reconfigure. If that fails, restore from backup.

Runner failures

Symptom: Jobs stuck pending, runner offline.

Recovery: Check runner status:

sudo gitlab-runner status

Restart if needed:

sudo gitlab-runner restart

Review runner logs for errors.

Network or certificate issues

Symptom: Web interface inaccessible or SSL errors.

Recovery: Check firewall rules, DNS, and certificate validity. Renew or correct certificates and reconfigure.

High resource usage

Symptom: Slow response, high CPU/memory.

Recovery: Identify processes:

top

Tune GitLab settings (e.g., reduce worker processes) or scale up resources.

Always have a tested backup and restore procedure. Document each failure and recovery in an incident log to improve future responses. The incident commander (rotating role) should review the log weekly.

Common Pitfalls and How to Avoid Them

Operating GitLab CI/CD comes with recurring pitfalls. Here are the most common ones, why they happen, and how to prevent or recover from them.

Pitfall 1: Not testing backups

Why it happens: Teams assume backups are fine because they are created successfully, but restoration may fail due to missing configuration, permissions, or incompatible versions.

How to avoid: Schedule monthly restore tests in an isolated environment. Use the exact backup file and follow the documented procedure. Log the test result and time taken.

Recovery: If a restore fails, troubleshoot immediately. Check backup logs, storage integrity, and compatibility. Do not wait for a real disaster.

Pitfall 2: Making configuration changes without dry-run

Why it happens: In a rush to fix an issue, engineers edit gitlab.rb directly and run reconfigure, causing unexpected service restarts or misconfigurations.

How to avoid: Always use --dry-run first and test in staging. Keep gitlab.rb in version control and review diffs.

Recovery: If a change breaks production, revert to the previous version from Git and reconfigure. If that fails, restore from backup.

Pitfall 3: Ignoring runner maintenance

Why it happens: Runners are often set up once and forgotten until pipeline failures occur.

How to avoid: Regularly update runner versions, monitor runner status, and prune unused runners. Use a configuration management tool to enforce consistency.

Recovery: When runners fail, check logs, restart, and if necessary, re-register the runner with a new token.

Pitfall 4: Running out of disk space

Why it happens: Artifacts, logs, and container images accumulate without cleanup policies.

How to avoid: Set artifact expiration in .gitlab-ci.yml, configure log rotation, and schedule regular cleanup jobs. Monitor disk usage trends.

Recovery: Immediately delete old artifacts and logs, expand disk if possible, and implement retention policies.

Pitfall 5: Skipping upgrade preparation

Why it happens: Teams upgrade without reading release notes or backing up, leading to unexpected breaking changes or data loss.

How to avoid: Read upgrade notes for each minor version, backup before upgrading, and use a staging environment to test.

Recovery: If upgrade fails, restore from backup and analyze the cause before retrying.

Operations Checklist

Use this checklist for routine operations and periodic reviews. Assign each item to an owner (e.g., DevOps engineer) and set a review frequency.

Daily checks (Owner: On-call DevOps engineer)

• Review sudo gitlab-ctl status for all services running.
• Check system resource usage (CPU, memory, disk).
• Monitor Sidekiq queue size: sudo gitlab-rails runner 'puts Sidekiq::Queue.new.size' should be low.
• Check for failed pipeline jobs in critical projects.

Weekly checks (Owner: DevOps lead)

• Review logs for errors (sudo gitlab-ctl tail or centralized log search).
• Verify backup completion and storage space.
• Check for available GitLab updates and security patches.

Monthly checks (Owner: Operations engineer)

• Test restore procedure in a sandbox environment.
• Review user accounts and permissions; remove stale accounts.
• Check runner fleet for offline runners and update runner versions.
• Review configuration changes in version control and document any drift.

Quarterly checks (Owner: DevOps manager)

• Perform a full security audit (TLS certificates, firewall rules).
• Review CI/CD pipeline efficiency: artifact sizes, job durations, and runner utilization.
• Evaluate capacity planning: disk growth, CPU/memory trends.
• Update the environment inventory document.

Before and after upgrades (Owner: Release manager)

• Read upgrade notes and plan for downtime if necessary.
• Backup before upgrading.
• Upgrade one minor version at a time (for Omnibus: sudo apt-get update && sudo apt-get install gitlab-ce=<version>).
• After upgrade, run sudo gitlab-ctl reconfigure and sudo gitlab-rake db:migrate if required.
• Verify system with sudo gitlab-rake gitlab:check.

Keep this checklist in a shared location and review it with your team to ensure no step is missed.

Conclusion

Operating GitLab CI/CD in production requires a structured approach. This checklist has covered version inventory, safe configuration practices, verification and diagnostics, failure modes and recovery, common pitfalls, and a routine operations checklist. By implementing these steps, you can reduce downtime, improve reliability, and respond effectively to incidents. Start with a narrow pilot to test your procedures, then expand. Remember to keep your documentation up to date and continuously review and improve your operations processes.