When GitLab CI/CD jobs fail to reach package repositories, artifact stores, container registries, or internal APIs, the root cause is often networking. DNS, ports, routing, proxies, TLS, and firewalls can behave differently inside a runner than on a developer workstation. This guide gives you practical, safe steps to diagnose and fix those differences with reproducible examples. You will create a small, read-only diagnostic job, compare outputs to expected results, and apply targeted changes without risking unrelated parts of your pipeline. The approach is simple: capture a clear inventory, run a narrow pilot check, observe the network path, and fix only what your evidence supports. By keeping the pilot small and measurable, you can iterate quickly and reduce rework. ## Version and Environment Inventory Before touching configuration, capture answers to these questions. Consistent inventory prevents chasing the wrong layer. ## What to record

Item	Example value
GitLab edition and version	GitLab.com SaaS or self-managed 17.x
GitLab Runner version	17.x (shell, Docker, or Kubernetes executor)
Runner OS and architecture	Ubuntu 22.04 x86_64; Windows Server 2022
Executor type	shell, docker, kubernetes
Network zone	office LAN, DC VLAN 120, k8s namespace ci
DNS resolvers used by jobs	10.0.0.10, 10.0.0.11
Proxy settings in jobs	http_proxy, https_proxy, NO_PROXY
Expected egress IP(s)	NAT IP 203.0.113.45 (constructed example)
Firewall owners and change window	NetSec on-call, daily 16:00-17:00

## How to collect it (Linux examples) - Runner version and executor (run on the runner host): gitlab-runner --version cat /etc/gitlab-runner/config.toml - DNS configuration visible to jobs (inside the job): cat /etc/resolv.conf || true getent hosts gitlab.com || true - Proxy-related environment (inside the job): ( env | grep -iE '^(http|https|no)_proxy=' || true ) | sort - Network paths (on runner host or inside job if permitted): ip addr show ip route show ss -tupan | head -n 50 ## Windows runner snippets - DNS resolution: Resolve-DnsName example.com - Port reachability: Test-NetConnection example.com -Port 443 Document these findings alongside job IDs so you can compare good vs failing runs. ## Safe Configuration Path The safest way to troubleshoot is to introduce a small, read-only diagnostic job that probes only the endpoints you care about. Avoid global changes until the job proves what is broken. ## 1) Add a diagnostic job The following job uses standard Linux tools. It does not modify system state and exits non-zero on clear failures. # .gitlab-ci.yml (constructed example) stages: - verify net-verify: stage: verify image: alpine:3.20 # for docker executor; remove 'image' for shell executor variables: # Set proxies here only if your environment requires them # http_proxy: "http://proxy.internal:3128" # https_proxy: "http://proxy.internal:3128" # NO_PROXY: "registry.internal,.svc,.cluster.local,10.0.0.0/8,127.0.0.1,localhost" before_script: - apk add --no-cache bind-tools curl openssl iproute2 busybox-extras script: - set -euxo pipefail - echo "== DNS ==" - cat /etc/resolv.conf || true - dig +short registry.internal || true - echo "== TCP 443 reachability ==" - nc -vz -w5 registry.internal 443 || true - echo "== TLS handshake and SNI ==" - echo | openssl s_client -connect registry.internal:443 -servername registry.internal -brief 2>/dev/null || true - echo "== HTTP HEAD with timing ==" - curl -fsSILv https://registry.internal -o /dev/null -w 'code=%{http_code} connect=%{time_connect} ttfb=%{time_starttransfer}\n' - echo "== Route and addresses ==" - ip addr show || true - ip route show || true Notes: - Replace registry.internal with your real target (artifact store, registry, API). - If you use a shell executor on a locked-down host, install equivalent tools via the OS package manager and run them in the job script. - Keep secrets out of command lines. Use masked CI variables where authentication is required for HTTP(S) checks. ## 2) Check DNS first A large fraction of CI connectivity failures are name resolution problems. Validate: - The job resolves the same names your workstation resolves. - The answer types (A vs AAAA) make sense for your network. - The search domain does not rewrite names unexpectedly. Diagnostics: getent hosts internal.api.local || true dig +search internal.api A AAAA +timeout=2 +tries=1 || true If IPv6 answers are returned but your network does not carry IPv6 end-to-end, prefer IPv4 at call sites during investigation: curl -4 -fsSIL https://internal.api.local ## 3) Validate ports and TLS - Ports: Confirm the target port is open from the job. nc -vz -w3 registry.internal 443 || true - TLS: Confirm SNI, certificate chain, and protocol are acceptable. Look for certificate verify errors or unexpected issuers (for example, a corporate intercept proxy). echo | openssl s_client -connect registry.internal:443 -servername registry.internal -showcerts 2>/dev/null | awk '/Server certificate/ {p=1} p{print}' If a corporate CA signs certificates, install it in the runner environment rather than disabling verification. For example, on Debian/Ubuntu-based runners: # Write corporate-ca.crt via a protected CI variable or baked image, then: sudo cp corporate-ca.crt /usr/local/share/ca-certificates/ sudo update-ca-certificates Avoid disabling TLS verification globally. If you must temporarily bypass verification to confirm that only trust is the issue, scope it to a single command and immediately revert once the CA is installed. ## 4) Inspect routing and NAT - Confirm a default route exists and is sane: ip route show - Trace the path and spot blackholes: traceroute -n registry.internal || true - If your organization uses NAT, ask NetSec which egress IP your runner uses so they can check firewall logs. Keep this process outside the CI job; do not rely on calling external what-is-my-ip endpoints from CI. ## 5) Handle proxies deliberately If your jobs must traverse a proxy, set environment variables at the narrowest scope that works: variables: http_proxy: "http://proxy.internal:3128" https_proxy: "http://proxy.internal:3128" NO_PROXY: "registry.internal,.svc,.cluster.local,10.0.0.0/8,127.0.0.1,localhost" Tips: - NO_PROXY supports domain suffixes like .example.com . List internal hostnames and loopback addresses to avoid proxying local services. - If only some jobs need the proxy, define variables at the job level, not globally. - Some tools read lowercase and uppercase variants; set both if in doubt. ## 6) Executor-specific notes - Shell executor : The job shares the host network. Host firewalls (ufw, nftables, Windows Firewall) directly affect connectivity. - Docker executor : The job runs behind a user-defined bridge. Name resolution, MTU, and outbound NAT are controlled by the Docker bridge and the host. If you must add extra trust roots or hosts mappings, prefer image baking or runner configuration over ad-hoc flags. - Kubernetes executor : NetworkPolicies may block egress. Confirm namespace egress rules, DNS policies, and whether the cluster uses a proxy or egress gateway. ## Verification and Diagnostics After each change, rerun the diagnostic job and compare its outputs to the expectations below. Keep diffs small and focused on the endpoint you are trying to reach. ## Expected results by layer

Layer	Healthy example
DNS	`dig` returns A/AAAA for target; `getent hosts` shows expected IPs
TCP	`nc -vz host 443` reports succeeded; no timeout
TLS	`openssl s_client` shows a valid chain to a trusted root; SNI matches
HTTP	`curl -I` returns 200/302/401 per endpoint policy; timing fields are low
Route	`ip route` shows a default via correct gateway; `traceroute` progresses

## Sample healthy outputs (constructed examples) - DNS: $ dig +short registry.internal 10.50.12.34 - TCP: $ nc -vz -w3 registry.internal 443 registry.internal (10.50.12.34:443) open - TLS: $ echo | openssl s_client -connect registry.internal:443 -servername registry.internal -brief Protocol : TLSv1.3 Ciphersuite: TLS_AES_256_GCM_SHA384 Server certificate subject=CN = registry.internal issuer=CN = Corp-Root-CA Verification: OK - HTTP timing: $ curl -fsSILv https://registry.internal -o /dev/null -w 'code=%{http_code} connect=%{time_connect} ttfb=%{time_starttransfer}\n' < HTTP/1.1 200 OK code=200 connect=0.020 ttfb=0.085 If outputs do not match expectations, return to the Safe Configuration Path and adjust only the relevant layer. ## Failure Modes and Recovery This section lists common symptoms, causes, and safe fixes you can roll back. ## Common symptoms and focused fixes

Symptom	Likely cause	Safe first fix
`getent hosts` times out	Wrong DNS server or blocked UDP/53	Point runner to correct resolvers; verify with `dig`
`nc` to 443 times out	Firewall blocking egress or proxy required	Add proxy vars at job scope or request firewall rule
TLS verify failed	Missing corporate CA or TLS intercept	Install CA in runner trust store; avoid disabling verification
`curl` resolves IPv6 only	Split-horizon DNS, IPv6 not routed	Prefer IPv4 (`-4`) or correct DNS view for CI
`traceroute` stops at hop 1	Local firewall or default route wrong	Fix host firewall; confirm default route and gateway
Slow HTTP TTFB	Proxy throttling or packet loss	Bypass proxy for internal hosts via `NO_PROXY`; retry off-peak

## Rollback and recovery - .gitlab-ci.yml : - Keep all diagnostic changes in a dedicated verify job. - To roll back, remove only that job or variables you introduced. - Runner trust store : - Before adding a corporate CA, back up trust directories (for example, /etc/ssl/certs and /usr/local/share/ca-certificates on Debian/Ubuntu). - To roll back, remove the added CA file and run update-ca-certificates . - Proxy changes : - Remove job-level proxy variables to revert to prior behavior. - Docker or Kubernetes executor settings : - Back up runner configuration before edits. - Revert to the previous config file and restart the runner if changes do not help. Always rerun the diagnostic job after rollback to confirm you returned to the known baseline. ## Operations Checklist Use this checklist to ensure consistent, low-risk troubleshooting. - Inventory - Record GitLab and Runner versions, executor, OS, DNS resolvers, and proxy variables. - Capture the expected egress IP from NetSec if applicable. - Prepare diagnostics - Add a verify job that runs dig , nc , openssl s_client , curl , ip route . - Limit scope to the target endpoint(s). No secrets on command lines. - Run and observe - Save job logs. Compare DNS answers, TCP status, TLS chain, HTTP codes, and timing. - Note whether results differ across runners or environments. - Apply focused fixes - DNS: correct resolvers or DNS policy for the runner. - Ports: request firewall egress rules; test with nc . - TLS: install corporate CA in the runner trust store. - Proxies: set http_proxy / https_proxy / NO_PROXY at the job level. - Routing: fix default route or host firewall, then traceroute again. - Verify and document - Rerun verify job; confirm outputs meet expectations. - Record what changed and keep the verify job available for future checks. - Roll back if needed - Revert only the changes you introduced. - Confirm baseline by rerunning the verify job. ## Conclusion Networking inside GitLab CI/CD differs from developer laptops in subtle but important ways. By capturing a clear environment inventory, introducing a narrow diagnostic job, and validating each layer in order (DNS, ports, TLS, routing, proxies, firewall), you can isolate root causes quickly and fix them with minimal risk. Keep the diagnostic job handy in your repository to verify new environments, runners, or endpoints. When issues recur, follow the checklist to reproduce results, apply targeted changes, and confirm outcomes.