GitLab CI/CD networking troubleshooting with practical examples: practical implementation guide

When GitLab CI/CD jobs fail to reach package repositories, artifact stores, container registries, or internal APIs, the root cause is often networking. DNS, ports, routing, proxies, TLS, and firewalls can behave differently inside a runner than on a developer workstation. This guide gives you practical, safe steps to diagnose and fix those differences with reproducible examples. You will create a small, read-only diagnostic job, compare outputs to expected results, and apply targeted changes without risking unrelated parts of your pipeline.

The approach is simple: capture a clear inventory, run a narrow pilot check, observe the network path, and fix only what your evidence supports. By keeping the pilot small and measurable, you can iterate quickly and reduce rework.

Version and Environment Inventory

Before touching configuration, capture answers to these questions. Consistent inventory prevents chasing the wrong layer.

What to record

How to collect it (Linux examples)

• Runner version and executor (run on the runner host):

gitlab-runner --version
cat /etc/gitlab-runner/config.toml

• DNS configuration visible to jobs (inside the job):

cat /etc/resolv.conf || true
getent hosts gitlab.com || true

• Proxy-related environment (inside the job):

( env | grep -iE '^(http|https|no)_proxy=' || true ) | sort

• Network paths (on runner host or inside job if permitted):

ip addr show
ip route show
ess -tupan | head -n 50

Windows runner snippets

• DNS resolution:

Resolve-DnsName example.com

• Port reachability:

Test-NetConnection example.com -Port 443

Document these findings alongside job IDs so you can compare good vs failing runs.

Safe Configuration Path

The safest way to troubleshoot is to introduce a small, read-only diagnostic job that probes only the endpoints you care about. Avoid global changes until the job proves what is broken.

1) Add a diagnostic job

The following job uses standard Linux tools. It does not modify system state and exits non-zero on clear failures.

# .gitlab-ci.yml (constructed example)
stages:
  - verify

net-verify:
  stage: verify
  image: alpine:3.19 # for docker executor; remove 'image' for shell executor
  variables:
    # Set proxies here only if your environment requires them
    # http_proxy: "http://proxy.local:3128"
    # https_proxy: "http://proxy.local:3128"
    # NO_PROXY: ".svc,.cluster.local,10.0.0.0/8,127.0.0.1, localhost"
  before_script:
    - apk add --no-cache bind-tools curl openssl iproute2 busybox-extras
  script:
    - set -euxo pipefail
    - echo "== DNS =="
    - cat /etc/resolv.conf || true
    - dig +short example.com || true
    - echo "== TCP 443 reachability =="
    - nc -vz -w5 example.com 443 || true
    - echo "== TLS handshake and SNI =="
    - echo | openssl s_client -connect example.com:443 -servername example.com -brief 2>/dev/null || true
    - echo "== HTTP HEAD with timing =="
    - curl -fsSILv https://example.com -o /dev/null -w 'code=%{http_code} connect=%{time_connect} ttfb=%{time_starttransfer}\n'
    - echo "== Route and addresses =="
    - ip addr show || true
    - ip route show || true

Notes:

• Replace example.com with your real target (artifact store, registry, API).
• If you use a shell executor on a locked-down host, install equivalent tools via the OS package manager and run them in the job script.
• Keep secrets out of command lines. Use masked CI variables where authentication is required for HTTP(S) checks.

2) Check DNS first

A large fraction of CI connectivity failures are name resolution problems. Validate:

• The job resolves the same names your workstation resolves.
• The answer types (A vs AAAA) make sense for your network.
• The search domain does not rewrite names unexpectedly.

Diagnostics:

getent hosts internal.api.local || true
dig +search internal.api A AAAA +timeout=2 +tries=1 || true

If IPv6 answers are returned but your network does not carry IPv6 end-to-end, prefer IPv4 at call sites during investigation:

curl -4 -fsSIL https://internal.api.local

3) Validate ports and TLS

• Ports: Confirm the target port is open from the job.

nc -vz -w3 registry.internal 443 || true

• TLS: Confirm SNI, certificate chain, and protocol are acceptable. Look for certificate verify errors or unexpected issuers (for example, a corporate intercept proxy).

echo | openssl s_client -connect registry.internal:443 -servername registry.internal -showcerts 2>/dev/null | awk '/Server certificate/ {p=1} p{print}'

If a corporate CA signs certificates, install it in the runner environment rather than disabling verification. For example, on Debian/Ubuntu-based runners:

# Write corporate-ca.crt via a protected CI variable or baked image, then:
sudo cp corporate-ca.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

Avoid disabling TLS verification globally. If you must temporarily bypass verification to confirm that only trust is the issue, scope it to a single command and immediately revert once the CA is installed.

4) Inspect routing and NAT

• Confirm a default route exists and is sane:

ip route show

• Trace the path and spot blackholes:

traceroute -n registry.internal || true

• If your organization uses NAT, ask NetSec which egress IP your runner uses so they can check firewall logs. Keep this process outside the CI job; do not rely on calling external what-is-my-ip endpoints from CI.

5) Handle proxies deliberately

If your jobs must traverse a proxy, set environment variables at the narrowest scope that works:

variables:
  http_proxy: "http://proxy.local:3128"
  https_proxy: "http://proxy.local:3128"
  NO_PROXY: ".local,.svc,.cluster.local,10.0.0.0/8,127.0.0.1, localhost"

Tips:

• NO_PROXY supports domain suffixes like .example.com. List internal hostnames and loopback addresses to avoid proxying local services.
• If only some jobs need the proxy, define variables at the job level, not globally.
• Some tools read lowercase and uppercase variants; set both if in doubt.

6) Executor-specific notes

• Shell executor: The job shares the host network. Host firewalls (ufw, nftables, Windows Firewall) directly affect connectivity.
• Docker executor: The job runs behind a user-defined bridge. Name resolution, MTU, and outbound NAT are controlled by the Docker bridge and the host. If you must add extra trust roots or hosts mappings, prefer image baking or runner configuration over ad-hoc flags.
• Kubernetes executor: NetworkPolicies may block egress. Confirm namespace egress rules, DNS policies, and whether the cluster uses a proxy or egress gateway.

Verification and Diagnostics

After each change, rerun the diagnostic job and compare its outputs to the expectations below. Keep diffs small and focused on the endpoint you are trying to reach.

Expected results by layer

Sample healthy outputs (constructed examples)

• DNS:

$ dig +short registry.internal
10.50.12.34

• TCP:

$ nc -vz -w3 registry.internal 443
registry.internal (10.50.12.34:443) open

• TLS:

$ echo | openssl s_client -connect registry.internal:443 -servername registry.internal -brief
Protocol  : TLSv1.3
Ciphersuite: TLS_AES_256_GCM_SHA384
Server certificate
 subject=CN = registry.internal
 issuer=CN = Corp-Root-CA
 Verification: OK

• HTTP timing:

$ curl -fsSILv https://registry.internal -o /dev/null -w 'code=%{http_code} connect=%{time_connect} ttfb=%{time_starttransfer}\n'
< HTTP/1.1 200 OK
code=200 connect=0.020 ttfb=0.085

If outputs do not match expectations, return to the Safe Configuration Path and adjust only the relevant layer.

Failure Modes and Recovery

This section lists common symptoms, causes, and safe fixes you can roll back.

Common symptoms and focused fixes

Rollback and recovery

• .gitlab-ci.yml:
• Keep all diagnostic changes in a dedicated verify job.
• To roll back, remove only that job or variables you introduced.
• Runner trust store:
• Before adding a corporate CA, back up trust directories (for example, /etc/ssl/certs and /usr/local/share/ca-certificates on Debian/Ubuntu).
• To roll back, remove the added CA file and run update-ca-certificates.
• Proxy changes:
• Remove job-level proxy variables to revert to prior behavior.
• Docker or Kubernetes executor settings:
• Back up runner configuration before edits.
• Revert to the previous config file and restart the runner if changes do not help.

Always rerun the diagnostic job after rollback to confirm you returned to the known baseline.

Operations Checklist

Use this checklist to ensure consistent, low-risk troubleshooting.

1. Inventory

• Record GitLab and Runner versions, executor, OS, DNS resolvers, and proxy variables.
• Capture the expected egress IP from NetSec if applicable.

1. Prepare diagnostics

• Add a verify job that runs dig, nc, openssl s_client, curl, ip route.
• Limit scope to the target endpoint(s). No secrets on command lines.

1. Run and observe

• Save job logs. Compare DNS answers, TCP status, TLS chain, HTTP codes, and timing.
• Note whether results differ across runners or environments.

1. Apply focused fixes

• DNS: correct resolvers or DNS policy for the runner.
• Ports: request firewall egress rules; test with nc.
• TLS: install corporate CA in the runner trust store.
• Proxies: set http_proxy/https_proxy/NO_PROXY at the job level.
• Routing: fix default route or host firewall, then traceroute again.

1. Verify and document

• Rerun verify job; confirm outputs meet expectations.
• Record what changed and keep the verify job available for future checks.

1. Roll back if needed

• Revert only the changes you introduced.
• Confirm baseline by rerunning the verify job.

Conclusion

Networking inside GitLab CI/CD differs from developer laptops in subtle but important ways. By capturing a clear environment inventory, introducing a narrow diagnostic job, and validating each layer in order (DNS, ports, TLS, routing, proxies, firewall), you can isolate root causes quickly and fix them with minimal risk. Keep the diagnostic job handy in your repository to verify new environments, runners, or endpoints. When issues recur, follow the checklist to reproduce results, apply targeted changes, and confirm outcomes.