## Intro Docker BuildKit is the modern backend for building container images. It brings parallel builds, better caching, and advanced features like secrets and SSH forwarding, but it also introduces new networking behavior that can confuse even experienced developers. When a build that worked with the legacy builder suddenly fails with DNS errors, timeout messages, or inability to reach external services, the root cause often lies in BuildKit's network configuration. This guide provides a systematic approach to diagnosing and fixing BuildKit networking issues. You will learn how to gather the right information about your environment, apply safe configuration changes, test connectivity effectively, and recover from common failure modes. Every command includes expected output so you can compare results directly. By the end of this article, you will be able to resolve issues such as: - DNS resolution failures inside build containers - Cannot connect to external package repositories - Host services not reachable from build steps - Slow network performance during builds We will avoid guesswork by following a structured troubleshooting path. The techniques shown here work on Linux, macOS, and Windows hosts running Docker 20.10 or later with BuildKit enabled by default. ## Version and Environment Inventory Before changing any configuration, you need a clear picture of your system. Start by collecting version information for Docker and BuildKit. This helps identify known bugs and ensures you are using a supported setup. Open a terminal and run: ```bash docker version ``` Expected output includes client and server versions, like: ``` Client: Docker Engine - Community Version: 24.0.5 API version: 1.43 Go version: go1.20.6 Git commit: ced0996 Built: Fri Jul 21 18:20:40 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 24.0.5 API version: 1.43 (minimum version 1.12) Go version: go1.20.6 Git commit: a61e2b4 Built: Fri Jul 21 18:20:40 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.6.21 GitCommit: 3dce8eb055cbb6874e1c4c0f4e5c5b1d5a0a4e1 runc: Version: 1.1.7 GitCommit: v1.1.7-0-g860f061 docker-init: Version: 0.19.0 GitCommit: de40ad0 ``` Note the Docker Engine version and whether BuildKit is the default builder. Starting with Docker 23.0, BuildKit is the default for all builds. To check the current builder explicitly: ```bash docker buildx ls ``` Typical output: ``` NAME/NODE DRIVER/ENDPOINT STATUS BUILDKIT PLATFORMS default * docker default default running 23.0.3 linux/amd64, linux/arm64, linux/arm/v7, linux/arm/v6 ``` If you see "buildkit" under BUILDKIT, you are on BuildKit. If not, you can enable it by setting the environment variable `DOCKER_BUILDKIT=1` or configuring the daemon. Next, examine the Docker daemon networking configuration. The daemon manages the default bridge network, DNS settings, and MTU. Run: ```bash docker network ls ``` Output lists existing networks: ``` NETWORK ID NAME DRIVER SCOPE 9f904e7c5d7a bridge bridge local 2a9bc1e4b9dc host host local c24e0d1f3a5f none null local ``` BuildKit build containers typically attach to a dedicated network. You can inspect the default bridge to see its subnet and gateway: ```bash docker network inspect bridge ``` Relevant fields include: ```json [ { "Name": "bridge", "Id": "...", "Created": "...", "Scope": "local", "Driver": "bridge", "EnableIPv6": false, "IPAM": { "Driver": "default", "Options": null, "Config": [ { "Subnet": "172.17.0.0/16", "Gateway": "172.17.0.1" } ] }, "Internal": false, "Attachable": false, "Ingress": false, "ConfigFrom": { "Network": "" }, "ConfigOnly": false, "Containers": {}, "Options": { "com.docker.network.bridge.default_bridge": "true", "com.docker.network.bridge.enable_icc": "true", "com.docker.network.bridge.enable_ip_masquerade": "true", "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0", "com.docker.network.bridge.name": "docker0", "com.docker.network.driver.mtu": "1500" }, "Labels": {} } ] ``` Note the MTU value. In some environments, such as VPN or overlay networks, the host MTU may be lower than 1500. BuildKit inherits the MTU from the Docker daemon by default, and a mismatch can cause network stalls or timeouts. Also check the host's DNS configuration, because BuildKit forwards DNS queries to the host's resolver. On Linux, inspect `/etc/resolv.conf`: ```bash cat /etc/resolv.conf ``` Example: ``` nameserver 127.0.0.53 options edns0 trust-ad search example.com ``` On macOS, use `scutil --dns`. On Windows, run `ipconfig /all` and look for DNS Servers. Finally, confirm you can reach the internet from the host. A quick test: ```bash ping -c 1 8.8.8.8 ``` If this fails, the problem may be host-level, not BuildKit-specific. Similarly, test DNS resolution: ```bash nslookup google.com ``` Gather all this information into a simple inventory table for reference: | Component | Version / Value | Notes | |---|---|---| | Docker Engine | 24.0.5 | Check for latest patch | | BuildKit version | 23.0.3 (from buildx ls) | Default builder | | Default network subnet | 172.17.0.0/16 | Gateway 172.17.0.1 | | MTU on docker0 | 1500 | Compare with host interface | | Host DNS server | 127.0.0.53 | Local systemd-resolved | | Host internet access | OK | ping 8.8.8.8 works | This inventory allows you to spot mismatches quickly and provides a baseline for later comparisons. ## Safe Configuration Path Configuration changes can break existing builds. Always make small, reversible changes and test after each one. Prefer BuildKit-specific options over global Docker daemon changes whenever possible. This section outlines safe ways to adjust BuildKit networking behavior. ### Use Build-Time DNS Settings BuildKit lets you override DNS servers and search domains for build containers without affecting the host. You can set these in the Dockerfile using the `--dns` and `--dns-search` build arguments or via the `docker build` command line. For a single build, use: ```bash docker build --dns 8.8.8.8 --dns 1.1.1.1 -t myimage . ``` Alternatively, you can define them in the Dockerfile using the `ARG` and `RUN` instructions with explicit `--network` settings, but the command line is simpler. If you need to set DNS permanently for all builds, configure the Docker daemon. On Linux, edit `/etc/docker/daemon.json` and add: ```json { "dns": ["8.8.8.8", "1.1.1.1"], "dns-search": ["example.com"] } ``` Then restart Docker: ```bash sudo systemctl restart docker ``` This applies to all containers, including build containers. However, it is a global change, so test with the command-line override first. ### Configure Network Mode for Builds By default, BuildKit runs each build step in its own network namespace with NAT to the host. This isolation can cause issues when the build needs to access services on the host or on a local network. You can change the network mode per build using the `--network` flag. For example, to allow the build to use the host's network stack directly, which can bypass many routing issues: ```bash docker build --network=host -t myimage . ``` This is useful for accessing host services or when the build environment has custom routes. However, it reduces isolation and may not be allowed in some CI environments. Another option is to use a custom Docker network with specific settings. First create a network: ```bash docker network create --subnet=192.168.100.0/24 buildnet ``` Then build using that network: ```bash docker buildx build --network=buildnet -t myimage . ``` This gives you control over the subnet and gateway, which can help if the default bridge conflicts with your local network. ### Adjust MTU Settings If you suspect MTU mismatch, especially over VPN, you can set a lower MTU for the Docker daemon. In `daemon.json`: ```json { "mtu": 1400 } ``` Restart Docker and test builds. Alternatively, you can set the MTU for a specific network: ```bash docker network create -o "com.docker.network.driver.mtu=1400" buildnet-low-mtu ``` Then use `--network=buildnet-low-mtu` in your build. This is less invasive than changing the daemon default. ### Use Proxy Settings If your environment requires an HTTP/HTTPS proxy, ensure the build can use it. You can pass proxy variables as build args: ```bash docker build --build-arg HTTP_PROXY=http://proxy.example.com:3128 \ --build-arg HTTPS_PROXY=http://proxy.example.com:3128 \ --build-arg NO_PROXY=localhost,127.0.0.1,.example.com \ -t myimage . ``` But these args must be used in the Dockerfile via `ARG` and `ENV` to affect subsequent `RUN` commands: ```dockerfile ARG HTTP_PROXY ARG HTTPS_PROXY ARG NO_PROXY ENV HTTP_PROXY=$HTTP_PROXY \ HTTPS_PROXY=$HTTPS_PROXY \ NO_PROXY=$NO_PROXY RUN apt-get update && apt-get install -y curl ``` Alternatively, configure the Docker client to use a proxy by setting environment variables for the `docker` command itself, but that does not always propagate into build containers. The safest method is explicit build args. ### Configuration Summary All the above changes are reversible. Command-line flags affect only a single build. `daemon.json` changes require a daemon restart but can be reverted by restoring the file and restarting. Custom networks can be removed with `docker network rm`. Keep a record of changes so you can roll back if needed. ## Verification and Diagnostics After applying a configuration change, verify connectivity from within a build context. The easiest way is to create a temporary container using the same network settings as your build and run network tests. You can also add diagnostic steps directly to a Dockerfile during troubleshooting, but remember to remove them later. ### Test DNS Resolution in Build Run a one-off build with a simple test. Create a temporary Dockerfile named `Dockerfile.test` with: ```dockerfile FROM alpine:3.18 RUN nslookup google.com ``` Then build it: ```bash docker build -f Dockerfile.test --progress=plain . ``` Expected output from the `nslookup` command shows a successful resolution: ``` #6 [2/2] RUN nslookup google.com #6 sha256:... #6 0.123 Server: 10.0.2.3 #6 0.123 Address: 10.0.2.3:53 #6 0.123 #6 0.124 Non-authoritative answer: #6 0.124 Name: google.com #6 0.124 Address: 142.250.72.14 #6 DONE 0.2s ``` If DNS fails, you will see an error like `server can't find google.com: NXDOMAIN` or a timeout. In that case, check the DNS server address shown. It may be the BuildKit internal DNS (e.g., 10.0.2.3) which forwards to the host. If the host DNS is misconfigured, the internal DNS will also fail. To test with explicit DNS, use the `--dns` flag: ```bash docker build --dns 8.8.8.8 -f Dockerfile.test --progress=plain . ``` If this works, the issue is with the default DNS configuration. ### Test External Connectivity Use a simple HTTP request from within the build. Add this to your test Dockerfile: ```dockerfile FROM alpine:3.18 RUN wget -O- https://example.com | head -n 5 ``` Build and check for a response. A successful run shows the beginning of the example.com page. A failure will show `connection refused` or `timeout`. If you need to test TCP connectivity to a specific port, use `nc` (netcat). For example, to test connection to a database at host `db.example.com` port 5432: ```dockerfile FROM alpine:3.18 RUN apk add --no-cache netcat-openbsd && \ nc -zv db.example.com 5432 ``` Expected output: ``` db.example.com (192.168.1.100:5432) open ``` If it fails, you will see a timeout or `Connection refused`. ### Inspect BuildKit Network Namespace For deeper diagnostics, you can inspect the network namespace of a running build container. First, start a build that sleeps, using a multi-stage image: ```dockerfile FROM alpine:3.18 RUN sleep 300 ``` In another terminal, while the build is running, find the container: ```bash docker ps ``` Look for a container with the image `alpine:3.18` and command `sleep 300`. Note its ID. Then inspect its network settings: ```bash docker inspect | grep -A10 NetworkSettings ``` This shows the IP address, gateway, and DNS. You can also exec into the container (if it remains running) and run network commands: ```bash docker exec -it sh ``` Inside, run `ip addr`, `ip route`, `cat /etc/resolv.conf`, etc. This gives you a direct view of the build environment. ### Use BuildKit Debug Tools BuildKit has a built-in debug server that can be enabled for verbose logs. Set `BUILDKIT_PROGRESS=plain` to see more output, and use `--progress=plain` to disable the fancy UI. For even more detail, set the environment variable `BUILDKIT_DEBUG=1` before running `docker build`: ```bash BUILDKIT_DEBUG=1 docker build --progress=plain -t myimage . ``` This outputs internal BuildKit logs, including network-related messages, to the console. Search for lines containing `network`, `dns`, or `dial` to spot issues. ### Diagnostic Table Summarize common tests and expected outcomes: | Test | Command | Expected Result | Failure Indicates | |---|---|---|---| | DNS lookup | `nslookup google.com` inside build | IP address returned | DNS misconfiguration | | HTTP connectivity | `wget -O- https://example.com` | HTTP 200 and content | Firewall or routing issue | | TCP port check | `nc -zv host port` | "open" | Service not reachable or port blocked | | BuildKit debug log | `BUILDKIT_DEBUG=1 docker build ...` | No network errors | N/A (log reveals issues) | ## Failure Modes and Recovery Understanding common failure modes helps you recover quickly. Here are typical BuildKit networking problems, how to identify them, and steps to fix them. ### DNS Resolution Failure **Symptoms**: `RUN apt-get update` or `RUN npm install` fails with `Could not resolve host` or `Temporary failure in name resolution`. **Cause**: The build container's DNS server is unreachable or misconfigured. This can happen if the host's DNS is broken, the Docker daemon's DNS settings are wrong, or there is a firewall blocking port 53. **Diagnosis**: Run the DNS test from the verification section. Check `/etc/resolv.conf` inside the container with `docker exec` or by adding `RUN cat /etc/resolv.conf` to a temporary Dockerfile. The nameserver should be a valid IP reachable from the container. **Recovery**: Override DNS with `--dns 8.8.8.8` (or your corporate DNS). If that works, make the change permanent in `daemon.json` or in your build configuration. Also check that the host's `/etc/resolv.conf` is correct. If using systemd-resolved, ensure it is running and not misconfigured. ### Timeout Connecting to External Services **Symptoms**: Build hangs for a long time and eventually fails with `connection timed out` or `no route to host`. **Cause**: Network isolation, MTU mismatch, or firewall blocking outbound connections. VPNs often cause MTU issues. **Diagnosis**: Test connectivity to the target service from the host first. If the host works but the build does not, the issue is likely in the Docker network. Check MTU: compare `docker network inspect bridge` MTU with the host interface MTU (`ip link show`). A mismatch can cause packet loss. **Recovery**: Set MTU to match the host's effective MTU. Use `--network=host` as a temporary workaround to see if that resolves it. If yes, you know it is a container network issue. Adjust the MTU in `daemon.json` or create a custom network with the correct MTU. ### Cannot Access Host Services **Symptoms**: Build attempts to connect to a service running on the host (e.g., a local database or API) but fails. **Cause**: By default, build containers are on a separate bridge network and cannot reach the host via `localhost` or `127.0.0.1`. They must use the host's IP on the bridge network (often `172.17.0.1`). **Diagnosis**: From within a build container, try `ping 172.17.0.1`. If that works, use that IP in your build. Alternatively, use the special DNS name `host.docker.internal` on Docker Desktop (macOS/Windows). **Recovery**: Use `--network=host` for the build (if acceptable). Or, in your Dockerfile, use `--add-host=host.docker.internal:host-gateway` to map the hostname to the host's gateway IP. For Linux, you may need to add `extra_hosts` in the daemon or use `--add-host` flag: ```bash docker build --add-host=host.docker.internal:host-gateway -t myimage . ``` Then in the Dockerfile, refer to `host.docker.internal`. ### Firewall or Security Group Blocking **Symptoms**: Specific ports are blocked, but others work. For example, HTTPS (443) works but HTTP (80) fails. **Cause**: Host firewall or cloud security group rules are restricting outbound traffic from Docker's bridge network. **Diagnosis**: From the host, test the port using `nc` or `telnet`. Then from inside a container, test the same. If the container fails but the host succeeds, the firewall is likely blocking the Docker bridge interface (`docker0`). **Recovery**: Adjust firewall rules to allow traffic from the Docker subnet (e.g., `172.17.0.0/16`). On Linux with iptables: ```bash sudo iptables -I OUTPUT -s 172.17.0.0/16 -j ACCEPT ``` But be cautious with firewall changes; always follow your security policy. Alternatively, use `--network=host` to bypass Docker's bridge (though that may have its own restrictions). ### Rollback Strategies If a configuration change makes things worse, revert it immediately. Keep a backup of `daemon.json` and any custom scripts. For command-line changes, simply remove the flags. For `daemon.json`, restore the previous file and restart Docker. For custom networks, delete them with `docker network rm`. Document each change so you know what to undo. ## Common Pitfalls and How to Avoid Them Here are frequent mistakes developers make when dealing with BuildKit networking, along with prevention and recovery tips. ### Assuming BuildKit behaves like the legacy builder **Why it happens**: Developers are used to the old builder's networking, which was less isolated and often shared the host network. **How to avoid**: Understand that BuildKit creates a separate network namespace for each build step by default. Test connectivity early in a new build pipeline. **Recovery**: If a build fails due to isolation, try `--network=host` as a quick test, then adjust your Dockerfile or network settings accordingly. ### Overlooking MTU mismatch **Why it happens**: MTU is often not considered until symptoms like slow transfers or timeouts appear, especially over VPNs. **How to avoid**: Always check MTU when builds hang during network operations. Compare host MTU with Docker's MTU. **Recovery**: Set the correct MTU in `daemon.json` or use a custom network with the appropriate MTU. ### Hardcoding DNS servers without testing **Why it happens**: Teams hardcode public DNS like 8.8.8.8 in daemon.json, which may bypass corporate DNS and cause internal name resolution failures. **How to avoid**: Test with `--dns` on a single build first. Use corporate DNS servers if available. **Recovery**: Remove hardcoded DNS from daemon.json and configure appropriate DNS in the build environment. ### Ignoring proxy requirements **Why it happens**: In corporate environments, builds fail because proxy variables are not passed correctly. **How to avoid**: Always pass proxy settings as build args and use them in Dockerfile via ARG/ENV. Test with a simple `RUN curl` command. **Recovery**: Add missing proxy build args and rebuild. ### Not documenting network changes **Why it happens**: Teams make quick fixes without recording them, leading to confusion later. **How to avoid**: Keep a runbook of network configuration changes and their rationale. **Recovery**: Review git history and daemon.json backups to understand past changes. ## Operations Checklist Use the following checklist to troubleshoot systematically and prevent future issues. Run through these items in order, from basic checks to advanced diagnostics. ### Pre-flight Checks - [ ] Confirm Docker and BuildKit versions with `docker version` and `docker buildx ls`. - [ ] Verify host internet connectivity: `ping 8.8.8.8` and `nslookup google.com`. - [ ] Check default bridge network settings: `docker network inspect bridge`. - [ ] Note MTU and DNS settings. ### During Troubleshooting - [ ] Reproduce the issue with a minimal Dockerfile (e.g., single `RUN` that fails). - [ ] Test DNS resolution inside a build using `nslookup`. - [ ] Test external connectivity with `wget` or `curl`. - [ ] Test specific TCP ports with `nc`. - [ ] Try overriding DNS using `--dns 8.8.8.8`. - [ ] Try `--network=host` to isolate container network issues. - [ ] Inspect running build container with `docker exec` and `ip addr`. - [ ] Enable BuildKit debug logs: `BUILDKIT_DEBUG=1 docker build --progress=plain ...`. ### After Fix - [ ] Confirm the build completes successfully. - [ ] Document the change and the reason. - [ ] If a global change was made (e.g., `daemon.json`), ensure it does not affect other builds. - [ ] Consider adding regression tests (e.g., a healthcheck build in CI). ### Routine Maintenance - [ ] Periodically update Docker and BuildKit to get network fixes. - [ ] Monitor DNS infrastructure health. - [ ] Review firewall rules for Docker subnets. - [ ] Keep a runbook with common symptoms and fixes. ## Conclusion Docker BuildKit networking issues can be complex, but a structured approach simplifies troubleshooting. Begin by gathering version and environment details, then make small, reversible configuration changes. Use targeted diagnostic tests to verify connectivity and identify root causes. Knowing common failure modes such as DNS misconfiguration, MTU mismatch, and host access problems allows for quick recovery. This guide provided practical commands and expected outputs to guide you through each step. By following the operations checklist, you can resolve issues efficiently and prevent recurrences. Remember to document your findings and adjust your build processes accordingly. With these tools, you can confidently handle BuildKit networking challenges and keep your builds running smoothly.