## Intro GitLab CI/CD is a powerful tool for automating software delivery, but its configuration can be a double-edged sword. A single misplaced keyword, an incorrect variable, or a misunderstood rule can break your pipeline, waste developer time, and delay releases. Many teams learn these lessons the hard way, often after a production incident or a failed deployment. This guide aims to short-circuit that pain by cataloging the most common GitLab CI/CD configuration mistakes, providing practical examples, and explaining how to validate changes, roll back safely, and troubleshoot effectively. Whether you are a developer new to GitLab CI/CD, a DevOps consultant helping multiple clients, or a technical startup team building your first pipelines, this article will give you actionable steps to avoid pitfalls and recover quickly when something goes wrong. We will focus on real-world scenarios, not theory. By the end, you will have a checklist to review your own configurations and a set of commands to diagnose issues in any GitLab environment. ## Version and Environment Inventory Before diving into mistakes, it is essential to know exactly what you are working with. GitLab CI/CD behavior can vary between versions, runners, and executors. A configuration that works on GitLab.com might fail on a self-managed instance due to missing features or different defaults. Therefore, the first step in troubleshooting any pipeline issue is to establish a clear inventory of your environment. ### Check Your GitLab Version The GitLab version determines which CI/CD features are available. For example, the rules keyword was introduced in GitLab 12.3, and workflow: rules in 12.5. If you are on an older version, these constructs will cause a syntax error. To check your GitLab version, navigate to the help page (e.g., https://gitlab.example.com/help ) or run the following command on the server if you have SSH access: sudo gitlab-rake gitlab:env:info This command outputs a summary including the GitLab version, for example: GitLab version: 15.11.0 Make a note of this version because certain features and syntaxes are version-dependent. For instance, parallel: matrix was introduced in GitLab 13.3, and using it on an earlier version will cause an "unknown keyword" error. ### Runner Versions and Executors Runners execute the jobs defined in your .gitlab-ci.yml . They come in different types (shared, group, project) and can use various executors (shell, Docker, Kubernetes, etc.). The executor affects how commands run and what environment is available. To list runners registered for a project, use the GitLab API or the UI. With a personal access token, you can run: curl --header "PRIVATE-TOKEN: " "https://gitlab.example.com/api/v4/projects//runners" The response is a JSON array with runner details including id , description , active , runner_type , and ip_address . Note the runner tags because they control job routing. A common mistake is assigning a tag to a job that no runner has. For example, if your job has tags: [docker] but all runners are tagged shell , the job will be stuck in pending state indefinitely. To check the runner version, you can run: gitlab-runner --version It should match or be compatible with your GitLab version. GitLab maintains a runner version compatibility chart; using a runner that is too old can cause job failures because of missing features. ### Understand Your Topology Understand where your runners are hosted. Are they on the same network as your GitLab server? Do they have access to external services like package registries or deployment targets? Network issues often manifest as pipeline failures with timeouts or "connection refused" errors. Document the expected network path for each job. For example, a job that deploys to AWS needs outbound internet access and appropriate IAM credentials; if the runner is in a restricted subnet, the job will fail. ### Prerequisites Before modifying any configuration, ensure you have the following: - Access to the project repository and the ability to push changes to a branch. - Permission to view pipeline logs and runner status. - A local clone of the repository for quick edits and validation. - Knowledge of the deployment environment (servers, cloud accounts, etc.). With this inventory, you can proceed to examine your configuration with confidence. ## Safe Configuration Path When you identify a mistake or want to introduce a new CI/CD feature, resist the urge to make sweeping changes directly on the main branch. Instead, follow a safe configuration path that minimizes risk and allows for quick rollback. ### Start Small and Scoped The first useful pilot should be narrow, measurable, and easy to inspect locally before deployment. For GitLab CI/CD, this means making one change at a time in a feature branch and testing it in isolation. For example, if you need to add a deployment job, first add it with a simple echo command to ensure the job runs in the correct environment. Only after that works should you add the actual deployment logic. ### Use Branch-Based Development Create a branch for your CI/CD changes and open a merge request. This allows you to review the configuration diff and run the pipeline in the context of that branch. GitLab will automatically run the pipeline for the branch if CI is enabled. You can see the pipeline status directly in the merge request, which helps catch errors before merging. As an added safety measure, you can configure the merge request to require a successful pipeline before merging: # In your project settings, under Merge requests -> Pipelines must succeed This prevents accidental merges of broken configurations. ### Validate Syntax Before Pushing A common mistake is pushing a .gitlab-ci.yml with invalid YAML or unknown keywords. GitLab will fail the pipeline immediately, but you can catch these errors locally before even pushing. GitLab provides a CI Lint tool in the UI under CI/CD > Pipelines > CI Lint. However, for local validation, you can use the gitlab-ci-lint command if you have the GitLab CLI ( glab ) installed: glab ci lint If you are not using glab , you can use a generic YAML parser like yamllint to at least check syntax, though it will not understand GitLab-specific keywords. Better yet, install the GitLab Workflow extension for VS Code or use the online lint tool. Additionally, you can add a pre-commit hook to run glab ci lint automatically before each commit. ### Leverage rules and workflow Carefully Many configuration mistakes stem from misunderstanding the rules keyword and the interplay between only / except (deprecated) and rules . A safe path is to define a global workflow that controls when pipelines are created, and then use rules on individual jobs for finer control. For example: workflow: rules: - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - if: '$CI_COMMIT_BRANCH == "main"' - if: '$CI_COMMIT_TAG' This ensures pipelines are only created for merge requests, main branch commits, and tags. Without workflow , you might get duplicate pipelines or pipelines for unwanted events. Note that only and except are deprecated and may be removed in future GitLab versions; migrate to rules as soon as possible. ### Incremental Changes with Comments Always add comments explaining why a change was made. This helps future maintainers understand the context and can prevent them from reverting a fix that addressed a subtle issue. Additionally, consider using the extends keyword to reuse common job definitions, reducing duplication and the chance of inconsistent configurations. For example: .deploy_template: &deploy_template stage: deploy script: - echo "Deploying..." only: - main deploy_production: extends: .deploy_template environment: production ### Example of a Safe Change Suppose you need to add a test stage that runs unit tests. Instead of adding it directly to main , create a branch add-test-stage and modify .gitlab-ci.yml as follows: stages: - build - test - deploy build_job: stage: build script: echo "Building..." test_job: stage: test script: echo "Running tests..." Push the branch and observe the pipeline. If successful, open a merge request. This isolated change ensures that if something goes wrong, you can simply delete the branch and start over without affecting the main pipeline. ## Verification and Diagnostics After making changes, you need to verify that the configuration works as intended. This involves checking pipeline status, examining job logs, and possibly running commands on the runner to diagnose issues. ### Pipeline Status and Job Logs The first place to look is the pipeline page in GitLab. A pipeline can have statuses: passed , failed , pending , running , canceled , or skipped . Click on a failed job to see its log. Common errors include shell command failures, missing dependencies, or permission issues. The log will show the exact command that failed and the exit code. For example, a failed npm install will show output like: npm ERR! code E404 npm ERR! 404 Not Found - GET https://registry.npmjs.org/undefined npm ERR! A complete log of this run can be found in: This indicates a dependency issue. ### Using gitlab-runner for Local Debugging If a job fails due to environment issues on the runner, you can use the gitlab-runner command to run a job locally for debugging. For example, if you have a runner with the shell executor, you can simulate the job environment by running: gitlab-runner exec shell test_job This will execute the test_job script locally using the same environment as the runner, which can reveal missing packages or wrong paths. Note that gitlab-runner exec works only for shell, docker, and a few other executors, and it may not exactly replicate the CI environment, but it is very useful for quick checks. ### Verifying Configuration Syntax As mentioned earlier, use glab ci lint to validate your .gitlab-ci.yml . The output will indicate if the syntax is valid and may show a list of jobs with their stages and scripts. For example: Valid! - build_job (stage: build) - test_job (stage: test) If there is an error, it will point to the line and provide a hint. You can also use the API for linting without CLI: curl --header "Content-Type: application/json" --header "PRIVATE-TOKEN: " "https://gitlab.example.com/api/v4/ci/lint" --data '{"content": "stages: [build, test]\nbuild_job:\n stage: build\n script: echo Building"}' The API returns a JSON response with status and errors . ### Checking Runner Availability A job stuck in pending state often means no runner is available to pick it up. Use the GitLab UI to see which runners are active and their tags. You can also use the API: curl --header "PRIVATE-TOKEN: " "https://gitlab.example.com/api/v4/projects//runners?scope=active" If no runners appear, you need to register one or adjust the job tags. Also, check if the runner is paused or if there are runner limits (e.g., maximum jobs per runner). ### Diagnosing with Variables Sometimes a job behaves differently because of a variable set at the project or group level. To inspect the variables available to a job, add a temporary job that prints them (but be careful not to expose secrets). For example: debug_vars: stage: test script: - env | sort Run this job and check the log for unexpected values. Remember to remove this debug job before merging to avoid leaking sensitive information. You can also use echo $VARIABLE_NAME for specific variables, but avoid printing secrets like passwords or tokens. ### Expected Results for Common Checks - glab ci lint returns Valid! with no errors. - gitlab-runner exec completes with exit code 0 and expected output. - The pipeline status is passed and all jobs show green checkmarks. - No jobs remain in pending for more than a few minutes. ## Failure Modes and Recovery Despite careful configuration, things can go wrong. It is crucial to understand common failure modes and have a plan to recover quickly. ### Common Configuration Mistakes and Their Symptoms The following are the most frequent mistakes teams make, with details on why they happen and how to recover: ### 1. YAML Syntax Errors Symptom: The pipeline fails immediately with a message like jobs config should contain at least one visible job or YAML syntax error . Why it happens: Often, indentation is wrong, or a special character is not quoted. YAML is whitespace-sensitive, so a missing space can break the entire file. Recovery: Use glab ci lint to identify the line with the error. Fix the indentation or quoting, then commit again. ### 2. Invalid Keywords Symptom: Error message like unknown keyword: workflow or jobs:build:rules:rule if invalid expression . Why it happens: Using a keyword not supported by your GitLab version. For example, using workflow: rules in GitLab 11.x. Recovery: Check your GitLab version (as described earlier). If the keyword is too new, upgrade GitLab or avoid that keyword. If you must support older versions, use conditional includes based on the version, but that adds complexity. ### 3. Incorrect rules Logic Symptom: A job that should run on the main branch does not run, and the pipeline may skip the job unexpectedly. The job is not listed in the pipeline, or it shows skipped . Why it happens: A rule condition is too restrictive or the order of rules is incorrect. Remember that rules are evaluated in order until the first match, so if a generic rule comes first and matches, later specific rules are ignored. Recovery: Review the rules conditions and test with different branch names. Use the CI Lint tool to simulate pipeline creation for a specific branch or tag. ### 4. Missing Runner Tags Symptom: A job with tags: [docker] stays in pending forever. No runner picks it up. Why it happens: No runner has the specified tag, or all runners with that tag are inactive or busy. Recovery: Either add a runner with that tag (by editing the runner's configuration or registering a new one) or remove the tag if not needed. Alternatively, change the job to use a runner that is available. ### 5. Environment Variables Not Set Symptom: A job fails because a required variable is missing. The log shows an error like undefined variable or access denied . Why it happens: The variable is not defined in the project, group, or instance settings, or it is defined but not protected and the job runs on a protected branch. Recovery: Set the variable in GitLab CI/CD settings (Settings > CI/CD > Variables) or in the job definition using variables . Ensure the variable's protection settings match the branch. ### 6. Incorrect Working Directory Symptom: Commands run in a different directory than expected, causing file not found errors. Why it happens: By default, jobs run in the project directory, but custom scripts may change the directory, or the job may expect a different location. Recovery: Use cd explicitly in the script or set working_directory in the job (if using a custom executor). For Docker executors, the default is /builds// , but it can be overridden. ### 7. Dependency Installation Failures Symptom: apt-get install or bundle install fails due to network or missing keys. Why it happens: The runner may not have internet access, or the package repository is unreachable. Sometimes missing GPG keys or outdated caches cause issues. Recovery: Fix the source or use a cached image. Use cache in GitLab CI to cache dependencies between builds. For example: cache: paths: - vendor/ This speeds up builds and reduces network reliance. ### Rollback Strategies When a configuration change causes a pipeline to break, the fastest recovery is to revert the change. Since your changes should be in a merge request, you can simply revert the merge request or push a revert commit. If the change was merged directly, use git revert to undo it. git revert git push origin main This creates a new commit that undoes the previous change. Alternatively, if you have a known-good version of .gitlab-ci.yml , you can restore it via git checkout : git checkout -- .gitlab-ci.yml git commit -m "Revert .gitlab-ci.yml to known good state" git push origin main ### Recovery Checks After rolling back, verify that the pipeline returns to a passing state. Check the pipeline for the main branch and ensure all jobs run as expected. Also, monitor the next few pipeline runs to confirm stability. ### Preventing Future Failures - Use the CI Lint tool regularly, ideally as part of your pre-commit or CI process. - Write tests for your CI configuration if possible (e.g., using gitlab-ci-lint in a pre-commit hook or a dedicated CI job that lints the file). - Keep configurations modular with extends and include to reduce complexity and isolate changes. - Document any non-obvious settings in comments or a separate README. ## Common Pitfalls to Avoid Beyond the specific mistakes above, here are common behavioral and process pitfalls that can undermine your CI/CD setup: ### 1. Overcomplicating the Pipeline Many teams try to do everything in one pipeline, leading to a massive .gitlab-ci.yml that is hard to maintain. Why it happens: Lack of modularity and desire for a single source of truth. How to avoid: Break pipelines into stages and use include to import reusable YAML snippets. Consider using child pipelines or multi-project pipelines for complex workflows. ### 2. Ignoring Pipeline Performance Slow pipelines frustrate developers and reduce productivity. Why it happens: Not caching dependencies, using inefficient Docker images, or running unnecessary jobs. How to avoid: Use cache and artifacts to reuse build outputs. Profile pipeline duration and optimize the slowest jobs. Use needs to run jobs in parallel when possible. ### 3. Hand-Rolling Security Storing secrets in the repository or in plaintext variables is a serious risk. Why it happens: Convenience and lack of awareness of GitLab's secret management features. How to avoid: Use GitLab CI/CD variables with the "Masked" and "Protected" options. For more advanced secrets, use HashiCorp Vault integration or GitLab's external secrets. ### 4. Not Having a Test Environment for CI/CD Changes Applying changes directly to production pipelines without testing is risky. Why it happens: Lack of a staging or development environment for CI/CD. How to avoid: Use a dedicated test project or a separate branch with separate runner tags to test changes before merging. ### 5. Forgetting to Clean Up Old artifacts, caches, and runner registrations can accumulate and cause issues. Why it happens: No regular maintenance plan. How to avoid: Schedule periodic cleanup of artifacts and caches (GitLab has expiration settings). Remove inactive runners and unused variables. ## Operations Checklist To maintain a healthy GitLab CI/CD setup, follow this repeatable checklist for operations and reviews. Assign an owner and review frequency for each item to ensure accountability.
TaskOwnerFrequencyCommand/ToolExpected Result
Validate .gitlab-ci.ymlCI/CD Lead (e.g., Priya Shah, DevOps Engineer)Before every pushglab ci lintValid! message
Review pipeline statusDeveloper on merge requestAfter each pushGitLab UIAll jobs passed
Check runner availabilitySystem AdministratorWeeklyAPI or UIRunners online with correct tags
Audit CI/CD variablesSecurity Officer (e.g., Marcus Lee, DevSecOps)MonthlyGitLab settingsOnly necessary variables present, values correct and masked
Test disaster recoveryCI/CD LeadQuarterlySimulate failure on a branchRollback succeeds, pipeline recovers within 15 minutes
Update runner versionsSystem AdministratorAs neededgitlab-runner --versionLatest stable version
Review deprecated keywordsCI/CD LeadWith each GitLab upgradeCheck release notesNo deprecated only/except if avoidable
### Daily Operations - For every merge request, ensure the pipeline passes before merging. - Monitor pipeline duration and optimize slow jobs. - Keep job logs for auditing; you can store them in GitLab for a limited time by default. ### Troubleshooting Workflow When a pipeline fails, follow this sequence: - Look at the failed job log and note the error message. - Check if it is a syntax error, environment issue, or script failure. - If syntax error, run glab ci lint to identify the line. - If environment issue, try gitlab-runner exec locally to reproduce. - If script failure, examine the command and its dependencies. - Fix and push a new commit to a branch, then validate. ### Monthly Review Meeting Schedule a monthly 30-minute meeting with the CI/CD Lead, System Administrator, and a developer representative to go over pipeline health, runner utilization, and upcoming changes. Document decisions in the project wiki and assign action items. ## Conclusion GitLab CI/CD configuration mistakes are common but avoidable. By establishing a clear environment inventory, following a safe configuration path, verifying changes rigorously, and knowing how to recover from failures, you can maintain robust pipelines. The key is to treat CI/CD configuration as code: version it, review it, and test it. Use the checklist provided to periodically audit your setup and stay ahead of potential issues. With these practices, your team can ship faster with confidence. As a final tip, remember that the goal of CI/CD is to automate repetitive tasks and reduce human error, not to introduce new sources of failure. Keep your configurations simple, documented, and well-tested. Happy building!