## Intro GitLab CI/CD upgrades and migrations often feel risky because a single misstep can break pipelines across many projects. This guide turns that risk into a controlled process. You will learn how to inventory your current installation, choose the right upgrade path, test changes safely, and recover quickly if something goes wrong. The focus is on practical commands, expected outputs, and decision points that work in real environments. This article is written for DevOps engineers, platform teams, and technical leads who manage self-hosted GitLab instances or plan a migration between deployment models. It covers upgrades within the same installation type, migrations such as moving from source to Omnibus or from on-premises to cloud, and rollback procedures when an upgrade fails. The examples use placeholders like gitlab.example.com and RUNNER_TOKEN so you can adapt them to your environment without exposing real credentials. The core principle is operational safety: observe before changing, limit the blast radius, use read-only checks before and after each step, and always have a tested recovery path. Never run a command you cannot reverse or verify. ## Version and Environment Inventory Before any upgrade or migration, you must know exactly what you are running, where it is deployed, and what depends on it. This section shows how to collect that information using GitLab's own CLI and API. ### Capture the Current Version and Topology Log in to your GitLab server and run the following read-only command to get the installed version: sudo gitlab-rake gitlab:env:info Expected output includes key lines like: GitLab version: 15.11.0 GitLab revision: 8f2f1a2b3c4 GitLab database adapter: PostgreSQL GitLab database version: 13.8 GitLab Shell version: 14.19.0 If you use Omnibus, you can also run sudo gitlab-ctl status to see the state of all services. For source installs, use cd /home/git/gitlab && sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production . Next, determine your deployment type. The most common are: - Omnibus package (all-in-one) - Source installation from Git repository - Helm chart on Kubernetes - Docker container Each has a documented upgrade path. For example, upgrading Omnibus requires apt or yum commands, while Kubernetes upgrades require helm upgrade with a new chart version. Check the background migration status with: sudo gitlab-rake gitlab:background_migrations:status If migrations are still pending, do not proceed with the upgrade until they finish. ### Identify Dependencies and External Components GitLab CI/CD relies on GitLab Runner, which may be installed on separate machines or containers. List registered runners using the admin area or API: curl --header "PRIVATE-TOKEN: " "https://gitlab.example.com/api/v4/runners/all" The response is JSON. Note the version field for each runner. Runner versions must be compatible with the GitLab server version. GitLab maintains a compatibility chart: a runner one major version behind is usually fine, but a runner three major versions behind may fail to communicate. Also check external integrations like object storage (S3, GCS) for CI artifacts, registry for container images, and PostgreSQL/Redis versions. Use the built-in health checks: sudo gitlab-rake gitlab:check This command verifies database connectivity, Redis availability, and filesystem permissions. Any error here must be resolved before an upgrade. ### Establish a Baseline with Read-Only Checks Before changing anything, record the current working state. Create a simple test project and pipeline to confirm CI/CD is functional. For example, create a .gitlab-ci.yml with a trivial job: stages: - test ping: stage: test script: - echo "The CI system is alive" - gitlab-runner --version Run the pipeline and save its log and status. This baseline will be compared after the upgrade to ensure no regression. Also capture the list of active projects, users, and CI variables (without their values) using the API or database. This helps with rollback if needed. ## Safe Configuration Path A safe upgrade or migration is not just about running the right commands; it is about sequencing changes so each step is reversible and verifiable. This section outlines a recommended path and the specific commands for each step. ### Pre-Upgrade Checklist - Backup everything. For Omnibus, run: sudo gitlab-backup create STRATEGY=copy This creates a tar file in /var/opt/gitlab/backups containing database, repositories, uploads, and configuration. Verify the backup: sudo ls -lh /var/opt/gitlab/backups/ The output should show a recent .tar file with a non-zero size. Also back up /etc/gitlab/gitlab-secrets.json and /etc/gitlab/gitlab.rb separately. - Review the upgrade path. GitLab requires stepwise upgrades for major versions. You cannot jump from 14.0 to 16.5 directly. Use the official upgrade tool or the documented path. For example, if you are on 14.0, the path might be 14.0 -> 14.3 -> 14.6 -> 14.9 -> 15.0 -> 15.4 -> 15.8 -> 16.0 -> 16.3 -> 16.5. Each step must be completed and validated before the next. - Check for deprecations. Review the deprecation list for your current version. Use sudo gitlab-rake gitlab:check and look for warnings. Address any deprecations that could break after upgrade. ### Perform the Upgrade (Omnibus Example) Assuming you are upgrading from 15.11.0 to 16.0.0 on Ubuntu, the commands are: sudo apt-get update sudo apt-get install gitlab-ce=16.0.0-ce.0 For other versions or distributions, replace accordingly. After installation, GitLab runs migrations automatically on first start. Monitor the process: sudo gitlab-ctl tail Wait for the output to show "gitlab Reconfigured!" or similar. Then check the status: sudo gitlab-ctl status All services should be run: ... (pid ...) with no down states. Run the post-install checks: sudo gitlab-rake gitlab:check Look for any new errors. If all clear, proceed to the next version in the upgrade path. ### Handling Migrations (Omnibus to Kubernetes, etc.) Migrations between deployment types are more complex. For example, moving from Omnibus to Helm chart involves: - Export data from Omnibus using backup and import into a PostgreSQL instance managed by the chart. - Transfer repositories and uploads to object storage. - Update DNS and load balancer. - Reconfigure runners to point to the new instance. A detailed migration script is beyond this scope, but the principle is the same: each data movement step must be verified with checksums and test restores before cutover. ## Verification and Diagnostics After each upgrade or migration, you must verify that the system works as expected. This section provides specific checks and diagnostic commands. ### Core Health Checks Run the following commands in order: sudo gitlab-ctl status sudo gitlab-rake gitlab:check curl -I https://gitlab.example.com The first should show all services running. The second should end with "Checking GitLab ... Finished" without errors. The third should return HTTP 200 or 302 (redirect to sign-in). Check background migrations again: sudo gitlab-rake gitlab:background_migrations:status Should show no pending migrations. If any are stuck, see the failure section below. ### Functional CI/CD Verification Create a new project or use a test project. Push a simple .gitlab-ci.yml : stages: - verify upgrade-check: stage: verify script: - echo "Post-upgrade pipeline works" - gitlab-runner --version - git --version Run the pipeline and ensure it succeeds. Then check runner registration: sudo gitlab-runner list Output should show all runners and their executors. If any runner is missing or shows an error, troubleshoot before proceeding. Use the API to verify version consistency: curl --header "PRIVATE-TOKEN: " "https://gitlab.example.com/api/v4/version" Response: {"version": "16.0.0", "revision": "abcdefg"} Compare this to the installed version. ### Logs and Metrics Check logs for errors: sudo gitlab-ctl tail gitlab-rails sudo gitlab-ctl tail postgresql Look for exceptions, failed migrations, or connection errors. Also monitor system metrics like CPU, memory, and disk usage during the first pipeline runs after upgrade. Use htop , df -h , and free -m . If you use Prometheus, check the built-in metrics endpoint: curl https://gitlab.example.com/-/metrics This should return a large text payload of metrics. Any failure indicates a problem with the monitoring integration. ## Failure Modes and Recovery Even with careful planning, upgrades can fail. This section describes common failure modes, how to diagnose them, and recovery steps. ### Common Upgrade Failures and Fixes - Background migrations stuck. Sometimes migrations do not finish automatically, often due to long-running data or locks. Diagnose with: sudo gitlab-rake gitlab:background_migrations:status If stuck, check the logs: sudo gitlab-ctl tail sidekiq Look for deadlocks or timeouts. Often, restarting Sidekiq helps: sudo gitlab-ctl restart sidekiq If still stuck, consult GitLab support or run the migration manually with caution. - Database migration failure. If gitlab:check reports a failed migration, retrieve the migration ID from the error and check its status in the schema_migrations table. You can attempt to re-run it, but first ensure no partial changes. Rolling back a migration is not officially supported, so restoring from backup may be safer. - Runner incompatibility. After upgrade, runners may fail to pick up jobs. Check runner logs: sudo gitlab-runner --debug run Look for "401 Unauthorized" or "version mismatch". Update the runner to a compatible version: sudo apt-get update && sudo apt-get install gitlab-runner=16.0.0 Replace with an appropriate version from the compatibility chart. - Configuration file errors. Omnibus errors often stem from a bad gitlab.rb . Run: sudo gitlab-ctl reconfigure If reconfigure fails, examine /var/log/gitlab/reconfigure/current and fix the configuration parameter. ### Rollback Procedure If an upgrade causes severe issues and you need to roll back, follow these steps: - Stop GitLab services: sudo gitlab-ctl stop - Restore the backup from before the upgrade: sudo gitlab-backup restore BACKUP= For example, if your backup file is 1699999999_2023_11_14_15.11.0_gitlab_backup.tar , use BACKUP=1699999999_2023_11_14_15.11.0 . - Restore secrets and configuration: sudo cp /path/to/backup/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json sudo cp /path/to/backup/gitlab.rb /etc/gitlab/gitlab.rb - Reconfigure and restart: sudo gitlab-ctl reconfigure sudo gitlab-ctl restart - Verify with health checks and the baseline pipeline. Note: A rollback is only possible if you have a complete backup taken immediately before the upgrade. Always test your backup restore process in a staging environment to ensure it works. ## Operations Checklist This checklist consolidates the key steps into a repeatable runbook. Assign an owner to each item and review the checklist quarterly.
StepActionCommand / CheckExpected ResultOwnerReview Frequency
1Inventory current version and topologysudo gitlab-rake gitlab:env:infoVersion matches planned starting pointPriya Shah, Engineering LeadQuarterly
2Verify background migrations completesudo gitlab-rake gitlab:background_migrations:statusNo pending migrationsMarcus Chen, DevOps EngineerBefore each upgrade
3Create full backup with copy strategysudo gitlab-backup create STRATEGY=copyBackup file exists in /var/opt/gitlab/backupsPriya ShahBefore each upgrade
4Check runner compatibilityAPI call to list runnersAll runners within supported version rangeMarcus ChenBefore each upgrade
5Perform stepwise upgrade following pathsudo apt-get install gitlab-ce=<version>Services restart without errorsMarcus ChenPer upgrade step
6Run post-upgrade health checkssudo gitlab-rake gitlab:checkNo errors reportedMarcus ChenAfter each step
7Run functional pipeline testPush test project and run pipelinePipeline succeedsPriya ShahAfter each step
8Verify rollback procedure in stagingSimulate restore in non-productionRestore completes and system worksPriya ShahQuarterly
9Update documentation and runbookEdit internal wikiRunbook reflects current version and stepsMarcus ChenAfter each upgrade
## Common Pitfalls and How to Avoid Them Many upgrade failures are caused by avoidable mistakes. Here are the most common pitfalls, why they happen, and how to recover. ### Pitfall 1: Skipping Backup or Not Testing Restore Why it happens: Teams assume the upgrade will succeed and do not want to spend time on backups. Consequence: If the upgrade corrupts data, there is no way to recover. How to avoid: Always run a full backup with gitlab-backup create and also copy secrets. Test the restore process in a staging environment at least once. Document the restore steps. Recovery: If you have no backup, you may need to rely on database snapshots or filesystem snapshots if available. Otherwise, data loss is probable. ### Pitfall 2: Jumping Multiple Major Versions Why it happens: Developers want to get to the latest version quickly or assume backward compatibility. Consequence: The upgrade may fail with migration errors or leave the database in an inconsistent state. How to avoid: Follow the official upgrade path. Upgrade one major version at a time, running checks between each. Recovery: If you skipped versions, you might be able to restore from backup and follow the correct path. Otherwise, consult support. ### Pitfall 3: Ignoring Runner Compatibility Why it happens: Teams focus on the server and forget about runners, which may be on separate machines. Consequence: After the server upgrade, runners fail to register jobs or execute them with errors. How to avoid: Check runner versions before upgrading and upgrade runners as needed. Maintain a compatibility matrix. Recovery: Update runners to a compatible version. If the runner cannot be updated immediately, pin the server to a version that supports the old runner until you can update both. ### Pitfall 4: Not Monitoring Background Migrations Why it happens: Background migrations run asynchronously, and teams may assume they completed. Consequence: Some features may not work correctly because the database schema or data is not fully migrated. How to avoid: Always check migration status after upgrade and do not proceed until it is clear. Recovery: If migrations are stuck, identify the job and resolve issues such as locks or timeouts. Restart Sidekiq if needed. In worst case, restore to a previous version and re-run with more resources. ### Pitfall 5: Changing Configuration Without Version Control Why it happens: Engineers directly edit /etc/gitlab/gitlab.rb or other config files without tracking changes. Consequence: A bad configuration change can break the instance, and it is hard to undo. How to avoid: Keep configuration files in version control and use a configuration management tool. Always test changes in staging. Recovery: Restore the previous configuration from version control or backup and reconfigure. ## Conclusion GitLab CI/CD upgrades and migrations do not have to be scary. By following a structured process—inventory, backup, stepwise upgrade, verification, and documented rollback—you can minimize downtime and avoid data loss. This guide has provided concrete commands and checklists that you can adapt to your environment. The next step is to apply this to your own system. Start with a low-risk verification: check your current version, ensure all background migrations are complete, and create a test backup. Then plan your upgrade path with the official tool. Remember that every environment is unique, so test in staging first and always have a rollback plan. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With these practices, you can keep your GitLab CI/CD platform secure, performant, and up to date.