GitLab Runner Networking Troubleshooting: A Practical Field Guide

Intro

When a GitLab Runner suddenly stops picking up jobs, the cause is often not the runner itself but the network path between the runner, GitLab, and external services such as container registries or package mirrors. This guide provides a practical, command-driven approach to diagnosing and fixing GitLab Runner networking issues. It is written for developers, DevOps consultants, and technical startup teams who need to move from an observed problem to a verified result without guessing.

The focus is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. We will cover DNS resolution, port reachability, connectivity to GitLab, proxy configuration, Docker and Kubernetes networking, and provide a checklist and common pitfalls.

Every example uses explicit placeholders like <gitlab.example.com> or <runner-ip>; replace them with your actual values. Never place real credentials, tokens, private keys, or production identifiers in scripts or documentation.

Version and Environment Inventory

Before touching anything, establish what you are working with. Know the GitLab Runner version, the host operating system, the executor (shell, Docker, Kubernetes, etc.), and where the runner sits in the network topology. A misdiagnosis often starts with assuming a version or topology that is not present.

Step 1: Identify the installed version

Run this on the machine where the runner is installed:

gitlab-runner --version

Expected output example:

Version:      16.4.0
Git revision: 1a2b3c4d
Git branch:   16-4-stable
GO version:   go1.20.5
Built:        2023-09-22T14:30:00Z
OS/Arch:      linux/amd64

If the command is not found, the runner is not installed or not in PATH. Check common locations: /usr/local/bin/gitlab-runner, /usr/bin/gitlab-runner, or the installation directory you used.

Step 2: Determine the deployment topology

Ask: is this runner installed directly on a VM or bare-metal host, or is it running inside a container? Is it a shared runner managed by GitLab, a group runner, or a project-specific runner? The network path is different in each case.

For a Docker-based runner, run:

docker ps --filter "name=gitlab-runner" --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"

This shows the runner container name, image, and status. If the container is restarting repeatedly, that is a signal to inspect logs: docker logs <container-name>.

For a Kubernetes-based runner, check the pod status:

kubectl get pods -n gitlab-runner -o wide

Look for pods in CrashLoopBackOff, Pending, or with restarts. The -o wide option shows the node IP and pod IP, useful for cross-node connectivity diagnosis.

Step 3: Capture current observable state with read-only commands

Use commands that do not change configuration:

• ip addr show or ifconfig to list network interfaces and IPs.
• ip route show to view routing table.
• cat /etc/resolv.conf to see DNS servers.
• ss -tulpn | grep gitlab-runner to see listening ports (if any).
• systemctl status gitlab-runner (systemd) or service gitlab-runner status (SysV) to see service state.

Record the output and timestamp. If you need to share logs with support, redact sensitive information.

Step 4: Define expected result and failure signal

Before making changes, write down what a healthy runner looks like in your environment. For example:

• Expected: gitlab-runner verify returns Verifying runner... is alive.
• Failure signal: ERROR: Failed to register this runner. Perhaps you are having network problems.

This step forces you to think about the target state, not just the immediate error.

Safe Configuration Path

Configuration changes should be small, scoped, and reversible. Before editing config.toml, know its current values and have a backup.

Read the current configuration

The main config file is usually /etc/gitlab-runner/config.toml. Display it with:

sudo cat /etc/gitlab-runner/config.toml

Note the [[runners]] sections, especially url, token, executor, and any pre_clone_script, environment, or extra_hosts entries. For Docker executor, network_mode and dns settings matter.

Make a backup

Before any change:

sudo cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak.$(date +%Y%m%d%H%M%S)

This gives you a fast recovery path: restore the backup and restart the runner.

Example change: adding a custom DNS server for Docker executor

If the runner's jobs fail to resolve internal hostnames, you might add a DNS server in the runner's Docker configuration. In config.toml, find the [[runners]] section and add or modify dns:

[[runners]]
  name = "my-docker-runner"
  url = "https://gitlab.example.com/"
  token = "REDACTED"
  executor = "docker"
  [runners.docker]
    image = "alpine:latest"
    dns = ["10.0.0.2", "8.8.8.8"]

After editing, validate the TOML syntax:

gitlab-runner verify

Or use a TOML parser if available. Then restart the runner:

sudo systemctl restart gitlab-runner

Verify that the runner reconnects:

sudo gitlab-runner status

Expected: gitlab-runner: Service is running.

If the runner fails to start, restore the backup and check logs: journalctl -u gitlab-runner -n 50 --no-pager.

Example change: adjusting network mode for Docker executor

Sometimes jobs cannot reach services on the host's network because the default bridge network isolates them. Changing network_mode to host can solve it, but be aware of security implications. In config.toml:

[runners.docker]
  network_mode = "host"

Blast radius: using host network means containers share the host's network stack, which may expose host ports or conflict with other services. Only use after verifying that bridge networking is not required for isolation. Document the decision and the review date.

Verification and rollback

After any configuration change, run a test job or at least a gitlab-runner verify. If the change does not produce the expected result, roll back:

sudo cp /etc/gitlab-runner/config.toml.bak.<timestamp> /etc/gitlab-runner/config.toml
sudo systemctl restart gitlab-runner

Never edit config.toml by hand on a live production runner without a backup and a maintenance window.

Verification and Diagnostics

Network issues often show up as: runner cannot register, runner cannot connect to GitLab, jobs fail to clone repository, or jobs cannot reach external services. Use these targeted checks.

1. Test connectivity to GitLab

The runner must reach the GitLab instance's HTTP(S) endpoint. From the runner host (or inside the runner container), run:

curl -I https://gitlab.example.com

Expected: HTTP/1.1 200 OK or 302 redirect to login. If you get Could not resolve host, DNS is failing. If connection times out, a firewall or routing issue is likely. If you get an SSL certificate error, the CA is not trusted.

For a more detailed TLS check:

echo | openssl s_client -connect gitlab.example.com:443 -servername gitlab.example.com 2>/dev/null | openssl x509 -noout -dates -subject

This shows certificate validity dates and subject, helping detect expired certificates or hostname mismatch.

2. Test DNS resolution

From the runner host, resolve GitLab's hostname:

nslookup gitlab.example.com

Or with dig:

dig +short gitlab.example.com

If no answer, check /etc/resolv.conf and /etc/hosts. For Docker executor, the DNS used inside jobs may differ; test inside a job or a temporary container:

docker run --rm alpine nslookup gitlab.example.com

If the host resolves but the container does not, the Docker daemon's DNS configuration is wrong. Fix by adjusting dns in config.toml or Docker daemon settings.

3. Test port reachability

GitLab Runner typically needs outbound HTTPS (443) and possibly SSH (22) if using SSH URL for repository. Use nc (netcat) or telnet:

nc -zv gitlab.example.com 443

Expected: Connection to gitlab.example.com 443 port [tcp/https] succeeded!

If not, test with telnet gitlab.example.com 443. A firewall may be blocking. Use traceroute to see where packets stop:

traceroute -n -T -p 443 gitlab.example.com

4. Check runner registration and token

The runner token is used to authenticate with GitLab. If the token is invalid or the runner is not registered, it will not connect. Run:

sudo gitlab-runner verify

Expected: Verifying runner... is alive. If you see ERROR: Verifying runner... failed, check the token in config.toml and compare with GitLab UI under Settings > CI/CD > Runners. Do not share the token.

5. Inspect logs

Runner logs are essential. For systemd:

journalctl -u gitlab-runner -f

For Docker runner:

docker logs -f gitlab-runner

Look for lines like:

ERROR: Failed to register this runner. Perhaps you are having network problems
ERROR: Registering runner... failed runner=xxxx status=couldn't execute POST

These indicate connectivity or authentication issues. For job-level network errors, check job logs; they may show names resolution failures, TLS errors, or timeouts.

6. Test from within a job

Create a simple test job in .gitlab-ci.yml to run network diagnostics inside the CI environment:

network-test:
  script:
    - nslookup gitlab.example.com
    - curl -I https://gitlab.example.com
    - nc -zv registry.example.com 443

Run the job and examine output. This quickly reveals whether the issue is runner-level or job-level.

Failure Modes and Recovery

Here are common network failure modes with symptoms, causes, and recovery steps.

A. DNS resolution failure

Symptom: curl: (6) Could not resolve host: gitlab.example.com; nslookup returns server can't find gitlab.example.com: NXDOMAIN.

Possible causes:

• Wrong DNS servers in /etc/resolv.conf or Docker daemon.
• DNS server unreachable due to firewall.
• Hostname not in DNS or /etc/hosts missing entry.

Recovery:

• Verify DNS server IPs and reachability: ping <dns-ip> (if ICMP allowed) or nc -zv <dns-ip> 53.
• Add correct nameserver to /etc/resolv.conf or Docker config.
• For temporary fix, add to /etc/hosts:

  192.0.2.10 gitlab.example.com

But this is not scalable; prefer proper DNS.

• Document the DNS configuration owner and review quarterly.

B. Connection timeout to GitLab

Symptom: curl: (7) Failed to connect to gitlab.example.com port 443: Connection timed out.

Possible causes:

• Firewall/security group blocking outbound 443.
• No route to host (missing route, wrong gateway).
• GitLab service down or listening on different port.

Recovery:

• Check security group rules (cloud) or iptables/firewalld on host.
• Verify routing table: ip route show; ensure default route exists.
• Test with telnet gitlab.example.com 443 from a machine that can connect to isolate.
• If GitLab is behind a proxy, ensure runner has proxy settings.

C. TLS certificate errors

Symptom: curl: (60) SSL certificate problem: unable to get local issuer certificate.

Possible causes:

• Self-signed or internal CA not trusted.
• Certificate expired or hostname mismatch.

Recovery:

• For self-signed, add CA to system trust store or configure runner to use ca_file in config.toml:

  [[runners]]
    url = "https://gitlab.example.com/"
    tls-ca-file = "/etc/gitlab-runner/certs/ca.crt"

Then restart runner.

• Check certificate dates: openssl s_client -connect gitlab.example.com:443 | openssl x509 -noout -dates.
• Do not disable TLS verification (tls-skip-verify) unless absolutely necessary and with a plan to fix the root cause.

D. Proxy misconfiguration

Symptom: Runner connects to GitLab, but jobs fail to reach external sites; runner logs show proxy errors.

Possible causes:

• Environment variables HTTP_PROXY, HTTPS_PROXY, NO_PROXY not set or incorrectly set.
• Proxy requires authentication not provided.

Recovery:

• Set proxy in runner's config.toml under [[runners]]:

  environment = ["HTTP_PROXY=http://proxy.example.com:3128", "HTTPS_PROXY=http://proxy.example.com:3128", "NO_PROXY=localhost,127.0.0.1,.example.com"]

Or set in Docker executor via env or pre_build_script.

• Ensure NO_PROXY includes GitLab hostname if GitLab is internal.
• Test with curl -x http://proxy.example.com:3128 https://gitlab.example.com.

E. Docker network issues

Symptom: Jobs cannot reach host services or other containers; DNS inside containers fails.

Possible causes:

• Default bridge network isolation.
• Docker daemon DNS settings.
• Port mapping not set for service containers.

Recovery:

• Use network_mode = "host" (with careful consideration) or define a custom network in Docker executor config:

  [runners.docker]
    network_mode = "bridge"
    dns = ["10.0.0.2"]

• For services, use links or define a network in docker-compose if used.
• Test from a temporary container: docker run --rm --network=host alpine nslookup gitlab.example.com.

F. Firewall on runner host

Symptom: Runner cannot make outbound connections, but other hosts can.

Possible causes: local firewall rules block egress.

Recovery:

• For iptables: sudo iptables -L -n -v to see rules; add allowed outbound rules.
• For firewalld: sudo firewall-cmd --list-all; ensure http and https services are allowed.
• For cloud security groups, review egress rules.
• Document changes in change management.

Each recovery step should be followed by verification: rerun the failed command or test job to confirm the issue is resolved.

Operations Checklist

Use this checklist before, during, and after troubleshooting. Assign a single accountable owner for each item and set a review frequency.

Use this table as a template; adapt owners and frequencies to your team. The key is to have a named person accountable for each area and a regular review to catch drift.

Common Pitfalls and How to Avoid Them

Many network issues persist because of avoidable mistakes. Here are the most frequent ones, why they happen, and how to prevent or recover from them.

Pitfall 1: Assuming the problem is the runner when it is the network

Why it happens: Teams see a failed job and immediately restart the runner or reinstall, without checking basic connectivity. This wastes time and may mask the real issue temporarily.

How to avoid: Always start with network diagnostics: DNS, curl, port tests, and logs. Only after ruling out network should you touch runner configuration.

Recovery: If you have already restarted and the issue persists, go through the Verification and Diagnostics section systematically; do not skip steps.

Pitfall 2: Editing config.toml without a backup

Why it happens: In a hurry, operators edit the file directly and then cannot revert, or they introduce syntax errors that break the runner completely.

How to avoid: Always cp the file to a timestamped backup before editing. Use a TOML linter or gitlab-runner verify after changes. Keep backups for at least 30 days.

Recovery: If the runner fails to start after an edit, restore the backup and restart. Examine the error in logs to understand what went wrong.

Pitfall 3: Ignoring proxy settings in CI jobs

Why it happens: The runner host may have proxy environment variables set, but Docker executor jobs often do not inherit them. Developers expect jobs to behave like their local shell.

How to avoid: Set proxy variables explicitly in runner configuration or job scripts. Use NO_PROXY to exclude internal addresses. Test from a job, not just the host.

Recovery: Check job logs for failed downloads or DNS errors. Add proxy environment to the runner's config.toml or use a before_script to set them.

Pitfall 4: Not isolating DNS issues between host and container

Why it happens: nslookup works on the host, so operators conclude DNS is fine, but containers use Docker daemon's DNS (often 8.8.8.8 or internal Docker DNS) and cannot resolve internal names.

How to avoid: Test DNS from inside a container that mimics the runner's environment. Use docker run --rm alpine nslookup ... or run a test job.

Recovery: Configure dns in [runners.docker] to use internal DNS servers, or set Docker daemon DNS globally.

Pitfall 5: Too many changes at once

Why it happens: Under pressure, operators change firewall rules, DNS, proxy, and runner config all at once; if the issue resolves, they don't know which change fixed it, and if it breaks, they don't know which to roll back.

How to avoid: Make one change at a time, verify after each, and document. Use the checklist to track.

Recovery: If you are in a multi-change mess, revert to the last known good backup of each modified component one by one, testing after each revert.

Pitfall 6: Hardcoding IP addresses instead of using DNS

Why it happens: To "quickly fix" DNS issues, someone adds an IP to /etc/hosts or in code. Later, the IP changes and everything breaks mysteriously.

How to avoid: Fix DNS properly; use hostnames in configuration. If you must use /etc/hosts temporarily, add a comment with the date and reason, and schedule a permanent fix.

Recovery: Search for hardcoded IPs in config.toml, CI scripts, and /etc/hosts; replace with hostnames and ensure DNS works.

Conclusion

Effective GitLab Runner networking troubleshooting is a disciplined process, not a collection of random commands. It requires knowing your environment, observing current state with read-only commands, making small and reversible changes, verifying each step, and documenting recovery paths.

The skills and checklists in this guide apply to self-managed GitLab, cloud runners, and hybrid setups. By following the version and environment inventory, safe configuration practices, targeted diagnostics, and a clear recovery plan, you reduce downtime and build confidence in your CI/CD infrastructure.

As a next step, choose one low-risk verification from the Operations Checklist and run it today. Record the current state, execute the check, compare the output to the expected result, and note any gaps. Then assign an owner and set a review date. Over time, this practice will make network issues rare and quick to resolve.