When a job runs on GitLab, a separate process called GitLab Runner picks it up, prepares an execution environment, runs your scripts, and reports back. Understanding how that runner is built and how it moves data gives you control over stability, speed, and troubleshooting.

This guide explains GitLab Runner architecture from an operator's point of view. You will learn the core components, how control and data move during a job, and where failures typically happen. We walk through a safe path to start with the shell executor, then add practical Docker and Kubernetes examples. Finally, we cover verification steps, failure modes, recovery, and a repeatable operations checklist.

## Version and Environment Inventory

Before you change anything, capture a minimal inventory so you can reproduce and debug:

- GitLab edition and version (self-managed or SaaS)

- GitLab Runner version and installation method

- OS and init system for each runner host (for example, Linux with systemd)

- Network egress to GitLab and to any container registries or artifact stores

- Planned executor types (shell, Docker, Kubernetes)

- Runner tokens, scope, and tags

- Concurrency limits and expected workload volume

Commands that help you capture the above on a Linux runner host:

gitlab-runner --version

- Show runner version:

sudo systemctl status gitlab-runner

- Confirm service status (systemd example):

uname -a lsb_release -a (if available) or cat /etc/os-release

- Check OS basics:

curl -I https://gitlab.example.com/ (replace with your host)

- Verify outbound HTTPS to GitLab:

Topology to write down (text is fine):

- Where runners live (single VM, fleet of VMs, Kubernetes cluster)

- Whether runners are shared or project/group-specific

- Executor isolation expectations (host shell vs containers vs pods)

- Data storage paths for builds, cache, and artifacts

## Architecture Overview: Components and Roles

At a high level, GitLab Runner continuously polls the GitLab coordinator API, claims a job that matches its tags and permissions, prepares an execution environment via an executor, runs the job scripts, uploads artifacts and cache, and reports status.

Key components and their roles:

- Coordinator : the GitLab API endpoints that advertise jobs and accept results

- Runner manager : the daemon that polls for work and orchestrates executors

- Executor : the adapter that actually runs the job (shell, Docker, Kubernetes, etc.)

- Workspace : the builds directory where job files are checked out and executed

- Cache and artifacts : persisted outputs uploaded after the job completes

- Tags and concurrency : scheduling knobs that govern who runs what and how many at once

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Component</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Role</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Example knobs</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Coordinator (GitLab)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Queues jobs and records results</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">URL, registration and runner tokens</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Runner manager</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Polls for jobs and orchestrates executors</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">concurrent</code>, <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">check_interval</code></td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Executor</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Runs scripts in an environment</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">executor = shell|docker|kubernetes</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Workspace</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Stores repo checkout and job files</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">builds_dir, permissions</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Cache</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Speeds up repeated downloads</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Cache backend, key, paths</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Artifacts</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Stores job outputs for later use</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Expiration, when, paths</td></tr></tbody>
</table>
</div>

## Data Flow and Control Flow

Control flow for a single job:

- A job enters the project queue in GitLab with required tags and rules.

- The runner manager polls the coordinator and sees a matching job.

- The runner claims the job and downloads job metadata.

- The executor prepares the environment:

- Shell: switches to the service user and prepares the workspace.

- Docker: pulls the image if needed, mounts volumes, sets env.

- Kubernetes: creates a pod with the specified image and mounts.

- The runner fetches the repository according to strategy (fresh, fetch, or none) and checks out the commit.

- The runner executes before_script, script, and after_script steps.

- The runner uploads artifacts and cache if configured.

- The runner sends job logs and final status back to GitLab.

Data flow highlights:

- Input : job metadata, environment variables, repository data, and container images (if used)

- Runtime : stdout/stderr streamed to job logs, temporary files in the workspace

- Output : artifacts archive and cache content uploaded to GitLab or configured storage

Scheduling knobs:

- Tags : a job runs only on a runner that advertises all required tags

- Protected scope : restricts runners to protected refs if enabled

- Concurrency : the concurrent value caps how many jobs the runner can run in parallel

Minimal config.toml knobs that affect control flow and throughput:

concurrent = 1
check_interval = 0

[session_server]
 session_timeout = 1800 

## Safe Configuration Path: Start With Shell Executor

 Start with a simple, auditable pilot on a single VM using the shell executor. This keeps isolation complexity low while you validate tags, scheduling, logs, and artifact handling.

Prerequisites:

- Linux VM with outbound HTTPS to your GitLab instance

- A dedicated system user, for example gitlab-runner

- A project or group runner token and a unique tag, for example local-shell

Steps (Debian/Ubuntu example):

- Create a dedicated user and install the runner.

sudo useradd --create-home --shell /bin/bash gitlab-runner || true
curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
sudo apt-get install -y gitlab-runner 

- Install and enable the service under the dedicated user.

 sudo gitlab-runner install --user=gitlab-runner --service=systemd
sudo systemctl enable --now gitlab-runner 

- Register a shell executor with a unique tag.

 sudo -u gitlab-runner gitlab-runner register \
 --url https://gitlab.example.com/ \
 --registration-token YOUR_PROJECT_OR_GROUP_TOKEN \
 --description "local-shell" \
 --tag-list "local-shell" \
 --executor shell \
 --locked=false \
 --run-untagged=false 

- Set conservative concurrency and confirm directories.

 sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak
sudo sed -i 's/^concurrent = .*/concurrent = 1/' /etc/gitlab-runner/config.toml
sudo systemctl restart gitlab-runner 
 Expected results:

- systemctl status gitlab-runner shows active (running)

- gitlab-runner list shows the new runner with executor: shell

- A job tagged local-shell is picked up quickly and runs on this host

A minimal verification job (store in your project) to prove end-to-end behavior:

stages: [verify]

runner_verify:
 stage: verify
 tags: [local-shell]
 script:
 - echo "hello from runner on $(hostname)"
 - echo "workspace: $PWD"
 - uname -a
 artifacts:
 when: always
 expire_in: 1 day
 paths:
 - verify.log
 after_script:
 - echo "verification complete" | tee verify.log 

## Practical Example: Docker Executor

 Move to the Docker executor when you need isolated, reproducible toolchains. The runner launches each job in a container based on the job's image or a default image.

Registration (runner host must have Docker installed and permission for the runner user to access the Docker socket):

sudo -u gitlab-runner gitlab-runner register \
 --url https://gitlab.example.com/ \
 --registration-token YOUR_TOKEN \
 --description "docker-shared" \
 --tag-list "docker" \
 --executor docker 
 Edit /etc/gitlab-runner/config.toml to set sane defaults:

runners
 name = "docker-shared"
 url = "https://gitlab.example.com/"
 token = "REDACTED"
 executor = "docker"
 [runners.docker]
 image = "alpine:3.19"
 privileged = false
 disable_cache = false
 pull_policy = ["if-not-present"]
 volumes = ["/cache"] 
 An example job that uses this runner:

stages: [build]

container_build:
 stage: build
 tags: [docker]
 image: alpine:3.19
 script:
 - apk add --no-cache bash
 - echo "build step in container: $(uname -a)"
 artifacts:
 paths: ["build/output.txt"] 
 Executor tradeoffs at a glance:

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Executor</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Isolation</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Overhead</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Typical use</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">shell</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Low (host)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Minimal</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Fast pilot, trusted hosts</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">docker</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Medium (container)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Moderate</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Reproducible toolchains</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">kubernetes</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">High (pod)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Higher</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Elastic scale across nodes</td></tr></tbody>
</table>
</div>

## Practical Example: Kubernetes Executor

Use the Kubernetes executor when you want to scale horizontally across a cluster. Each job runs in a pod created on demand.

Prerequisites:

- A reachable Kubernetes cluster

- Service account with permissions to create pods in a chosen namespace

- Network egress from the cluster to GitLab and any registries

Register and configure (simplified):

sudo -u gitlab-runner gitlab-runner register \
 --url https://gitlab.example.com/ \
 --registration-token YOUR_TOKEN \
 --description "k8s-exec" \
 --tag-list "k8s" \
 --executor kubernetes 
 Then edit /etc/gitlab-runner/config.toml:

runners
 name = "k8s-exec"
 url = "https://gitlab.example.com/"
 token = "REDACTED"
 executor = "kubernetes"
 [runners.kubernetes]
 namespace = "ci"
 image = "alpine:3.19"
 service_account = "gitlab-runner"
 poll_timeout = 600 
 A simple job for this executor:

stages: [test]

cluster_smoke:
 stage: test
 tags: [k8s]
 image: alpine:3.19
 script:
 - echo "running in k8s pod"
 - cat /etc/os-release || true 
 Expected behavior:

- The runner creates a pod per job in the ci namespace

- Logs stream to the job details page

- Artifacts upload to GitLab when the pod exits

## Verification and Diagnostics

Confirm core health after any change:

- Runner service state

- systemctl status gitlab-runner

- journalctl -u gitlab-runner -n 200 --no-pager

- Runner registration

- gitlab-runner list

- Network reachability

- curl -I https://gitlab.example.com/ from the runner

- Job scheduling

- Trigger the verification job with the matching tag

- Observe that the job is picked up by the expected executor

- Artifacts and cache

- Confirm the artifact is attached to the job

- If using cache, confirm cache hits on subsequent runs

For deeper inspection (constructed examples):

sudo -u gitlab-runner ls -lah /home/gitlab-runner/builds

- Shell executor workspace: check permissions and free space

docker info and docker images (on the runner host)

- Docker executor image pulls and volume mounts

kubectl -n ci get pods and kubectl -n ci describe pod <name>

- Kubernetes executor pod events

Expected verification outcomes:

- Jobs tagged for local-shell, docker, and k8s route only to matching runners

- The job logs show environment details as scripted

- No stuck jobs in pending for longer than the polling interval

## Failure Modes and Recovery

Common breakages and practical remedies:

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Symptom</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Likely cause</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Fix</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Job stays Pending</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">No runner with matching tags</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Add correct tag or add a matching runner</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Runner not responding</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Network egress blocked or URL mismatch</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Verify URL, certificates, and outbound HTTPS</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Permission denied in workspace</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Wrong user or dir perms</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Ensure runner user owns builds dir</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Docker image pull fails</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Registry auth or network</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Log in to registry or open egress</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Kubernetes pod CrashLoopBackOff</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Missing image or init script error</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Use a known-good image and re-run</td></tr></tbody>
</table>
</div>
Recovery steps you can execute safely:

- Pause or unregister a misbehaving runner

- Pause in the GitLab UI for quick isolation

- Or run: gitlab-runner unregister --name "local-shell"

- Restore known-good configuration

- Keep a backup of /etc/gitlab-runner/config.toml

- sudo cp /etc/gitlab-runner/config.toml.bak /etc/gitlab-runner/config.toml

- sudo systemctl restart gitlab-runner

- Retry a minimal verification job

- Confirm scheduling and logs before enabling more concurrency

Rollback guidance:

- Limit the blast radius by keeping concurrent = 1 during changes

- Use unique tags per executor so you can redirect jobs quickly

- Revert to the last working executor (for example, shell) while you fix Docker or Kubernetes configuration

## Operations Checklist

Use this checklist after installation, after any change, and periodically:

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Frequency</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Check</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Command or evidence</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Daily</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Service is up</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">systemctl status gitlab-runner</code> is active</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Daily</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Jobs not stuck Pending</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Recent jobs start within expected time</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Daily</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Disk space OK</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">df -h</code> on builds and cache paths</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Weekly</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Runner version drift</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Compare <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">gitlab-runner --version</code> to target</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Weekly</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Tag alignment</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Jobs have tags that match available runners</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Weekly</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Artifact and cache health</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Artifacts attached, cache hits observed</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Monthly</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Concurrency vs capacity</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Adjust <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">concurrent</code> based on load</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Monthly</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Security review</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Review runner user, volumes, and permissions</td></tr></tbody>
</table>
</div>
Practical tips:

- Keep one small verification job that you can trigger on demand for each executor

- Back up config.toml before any change

- Document the mapping of projects to runner tags so onboarding is predictable

## Conclusion

You have seen how GitLab Runner components fit together, how control and data flow during a job, and where to look when things go wrong. Starting with a narrow shell-executor pilot lets you validate scheduling, logging, and artifacts with minimal risk. From there, you can add Docker for reproducible toolchains and Kubernetes for elastic scale. Use the verification steps after each change, watch for common failure modes, and keep the checklist handy to sustain predictable operations as your usage grows.