Docker BuildKit Networking Troubleshooting with Practical Examples

Intro

Docker BuildKit is the modern backend for building container images. It brings parallel builds, better caching, and advanced features like secrets and SSH forwarding, but it also introduces new networking behavior that can confuse even experienced developers. When a build that worked with the legacy builder suddenly fails with DNS errors, timeout messages, or inability to reach external services, the root cause often lies in BuildKit's network configuration.

This guide provides a systematic approach to diagnosing and fixing BuildKit networking issues. You will learn how to gather the right information about your environment, apply safe configuration changes, test connectivity effectively, and recover from common failure modes. Every command includes expected output so you can compare results directly.

By the end of this article, you will be able to resolve issues such as:

• DNS resolution failures inside build containers
• Cannot connect to external package repositories
• Host services not reachable from build steps
• Slow network performance during builds

We will avoid guesswork by following a structured troubleshooting path. The techniques shown here work on Linux, macOS, and Windows hosts running Docker 20.10 or later with BuildKit enabled by default.

Version and Environment Inventory

Before changing any configuration, you need a clear picture of your system. Start by collecting version information for Docker and BuildKit. This helps identify known bugs and ensures you are using a supported setup.

Open a terminal and run:

docker version

Expected output includes client and server versions, like:

Client: Docker Engine - Community
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.20.6
 Git commit:        ced0996
 Built:             Fri Jul 21 18:20:40 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.6
  Git commit:       a61e2b4
  Built:            Fri Jul 21 18:20:40 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6874e1c4c0f4e5c5b1d5a0a4e1
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Note the Docker Engine version and whether BuildKit is the default builder. Starting with Docker 23.0, BuildKit is the default for all builds. To check the current builder explicitly:

docker buildx ls

Typical output:

NAME/NODE       DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
default *       docker
  default       default         running 23.0.3               linux/amd64, linux/arm64, linux/arm/v7, linux/arm/v6

If you see "buildkit" under BUILDKIT, you are on BuildKit. If not, you can enable it by setting the environment variable DOCKER_BUILDKIT=1 or configuring the daemon.

Next, examine the Docker daemon networking configuration. The daemon manages the default bridge network, DNS settings, and MTU. Run:

docker network ls

Output lists existing networks:

NETWORK ID     NAME      DRIVER    SCOPE
9f904e7c5d7a   bridge    bridge    local
2a9bc1e4b9dc   host      host      local
c24e0d1f3a5f   none      null      local

BuildKit build containers typically attach to a dedicated network. You can inspect the default bridge to see its subnet and gateway:

docker network inspect bridge

Relevant fields include:

[
    {
        "Name": "bridge",
        "Id": "...",
        "Created": "...",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

Note the MTU value. In some environments, such as VPN or overlay networks, the host MTU may be lower than 1500. BuildKit inherits the MTU from the Docker daemon by default, and a mismatch can cause network stalls or timeouts.

Also check the host's DNS configuration, because BuildKit forwards DNS queries to the host's resolver. On Linux, inspect /etc/resolv.conf:

cat /etc/resolv.conf

Example:

nameserver 127.0.0.53
options edns0 trust-ad
search example.com

On macOS, use scutil --dns. On Windows, run ipconfig /all and look for DNS Servers.

Finally, confirm you can reach the internet from the host. A quick test:

ping -c 1 8.8.8.8

If this fails, the problem may be host-level, not BuildKit-specific. Similarly, test DNS resolution:

nslookup google.com

Gather all this information into a simple inventory table for reference:

This inventory allows you to spot mismatches quickly and provides a baseline for later comparisons.

Safe Configuration Path

Configuration changes can break existing builds. Always make small, reversible changes and test after each one. Prefer BuildKit-specific options over global Docker daemon changes whenever possible. This section outlines safe ways to adjust BuildKit networking behavior.

Use Build-Time DNS Settings

BuildKit lets you override DNS servers and search domains for build containers without affecting the host. You can set these in the Dockerfile using the --dns and --dns-search build arguments or via the docker build command line.

For a single build, use:

docker build --dns 8.8.8.8 --dns 1.1.1.1 -t myimage .

Alternatively, you can define them in the Dockerfile using the ARG and RUN instructions with explicit --network settings, but the command line is simpler. If you need to set DNS permanently for all builds, configure the Docker daemon. On Linux, edit /etc/docker/daemon.json and add:

{
  "dns": ["8.8.8.8", "1.1.1.1"],
  "dns-search": ["example.com"]
}

Then restart Docker:

sudo systemctl restart docker

This applies to all containers, including build containers. However, it is a global change, so test with the command-line override first.

Configure Network Mode for Builds

By default, BuildKit runs each build step in its own network namespace with NAT to the host. This isolation can cause issues when the build needs to access services on the host or on a local network. You can change the network mode per build using the --network flag.

For example, to allow the build to use the host's network stack directly, which can bypass many routing issues:

docker build --network=host -t myimage .

This is useful for accessing host services or when the build environment has custom routes. However, it reduces isolation and may not be allowed in some CI environments. Another option is to use a custom Docker network with specific settings. First create a network:

docker network create --subnet=192.168.100.0/24 buildnet

Then build using that network:

docker buildx build --network=buildnet -t myimage .

This gives you control over the subnet and gateway, which can help if the default bridge conflicts with your local network.

Adjust MTU Settings

If you suspect MTU mismatch, especially over VPN, you can set a lower MTU for the Docker daemon. In daemon.json:

{
  "mtu": 1400
}

Restart Docker and test builds. Alternatively, you can set the MTU for a specific network:

docker network create -o "com.docker.network.driver.mtu=1400" buildnet-low-mtu

Then use --network=buildnet-low-mtu in your build. This is less invasive than changing the daemon default.

Use Proxy Settings

If your environment requires an HTTP/HTTPS proxy, ensure the build can use it. You can pass proxy variables as build args:

docker build --build-arg HTTP_PROXY=http://proxy.example.com:3128 \
             --build-arg HTTPS_PROXY=http://proxy.example.com:3128 \
             --build-arg NO_PROXY=localhost,127.0.0.1,.example.com \
             -t myimage .

But these args must be used in the Dockerfile via ARG and ENV to affect subsequent RUN commands:

ARG HTTP_PROXY
ARG HTTPS_PROXY
ARG NO_PROXY
ENV HTTP_PROXY=$HTTP_PROXY \
    HTTPS_PROXY=$HTTPS_PROXY \
    NO_PROXY=$NO_PROXY

RUN apt-get update && apt-get install -y curl

Alternatively, configure the Docker client to use a proxy by setting environment variables for the docker command itself, but that does not always propagate into build containers. The safest method is explicit build args.

Configuration Summary

All the above changes are reversible. Command-line flags affect only a single build. daemon.json changes require a daemon restart but can be reverted by restoring the file and restarting. Custom networks can be removed with docker network rm. Keep a record of changes so you can roll back if needed.

Verification and Diagnostics

After applying a configuration change, verify connectivity from within a build context. The easiest way is to create a temporary container using the same network settings as your build and run network tests. You can also add diagnostic steps directly to a Dockerfile during troubleshooting, but remember to remove them later.

Test DNS Resolution in Build

Run a one-off build with a simple test. Create a temporary Dockerfile named Dockerfile.test with:

FROM alpine:3.18
RUN nslookup google.com

Then build it:

docker build -f Dockerfile.test --progress=plain .

Expected output from the nslookup command shows a successful resolution:

#6 [2/2] RUN nslookup google.com
#6 sha256:...
#6 0.123 Server:  10.0.2.3
#6 0.123 Address: 10.0.2.3:53
#6 0.123
#6 0.124 Non-authoritative answer:
#6 0.124 Name:    google.com
#6 0.124 Address: 142.250.72.14
#6 DONE 0.2s

If DNS fails, you will see an error like server can't find google.com: NXDOMAIN or a timeout. In that case, check the DNS server address shown. It may be the BuildKit internal DNS (e.g., 10.0.2.3) which forwards to the host. If the host DNS is misconfigured, the internal DNS will also fail.

To test with explicit DNS, use the --dns flag:

docker build --dns 8.8.8.8 -f Dockerfile.test --progress=plain .

If this works, the issue is with the default DNS configuration.

Test External Connectivity

Use a simple HTTP request from within the build. Add this to your test Dockerfile:

FROM alpine:3.18
RUN wget -O- https://example.com | head -n 5

Build and check for a response. A successful run shows the beginning of the example.com page. A failure will show connection refused or timeout.

If you need to test TCP connectivity to a specific port, use nc (netcat). For example, to test connection to a database at host db.example.com port 5432:

FROM alpine:3.18
RUN apk add --no-cache netcat-openbsd && \
    nc -zv db.example.com 5432

Expected output:

db.example.com (192.168.1.100:5432) open

If it fails, you will see a timeout or Connection refused.

Inspect BuildKit Network Namespace

For deeper diagnostics, you can inspect the network namespace of a running build container. First, start a build that sleeps, using a multi-stage image:

FROM alpine:3.18
RUN sleep 300

In another terminal, while the build is running, find the container:

docker ps

Look for a container with the image alpine:3.18 and command sleep 300. Note its ID. Then inspect its network settings:

docker inspect <container-id> | grep -A10 NetworkSettings

This shows the IP address, gateway, and DNS. You can also exec into the container (if it remains running) and run network commands:

docker exec -it <container-id> sh

Inside, run ip addr, ip route, cat /etc/resolv.conf, etc. This gives you a direct view of the build environment.

Use BuildKit Debug Tools

BuildKit has a built-in debug server that can be enabled for verbose logs. Set BUILDKIT_PROGRESS=plain to see more output, and use --progress=plain to disable the fancy UI. For even more detail, set the environment variable BUILDKIT_DEBUG=1 before running docker build:

BUILDKIT_DEBUG=1 docker build --progress=plain -t myimage .

This outputs internal BuildKit logs, including network-related messages, to the console. Search for lines containing network, dns, or dial to spot issues.

Diagnostic Table

Summarize common tests and expected outcomes:

Failure Modes and Recovery

Understanding common failure modes helps you recover quickly. Here are typical BuildKit networking problems, how to identify them, and steps to fix them.

DNS Resolution Failure

Symptoms: RUN apt-get update or RUN npm install fails with Could not resolve host or Temporary failure in name resolution.

Cause: The build container's DNS server is unreachable or misconfigured. This can happen if the host's DNS is broken, the Docker daemon's DNS settings are wrong, or there is a firewall blocking port 53.

Diagnosis: Run the DNS test from the verification section. Check /etc/resolv.conf inside the container with docker exec or by adding RUN cat /etc/resolv.conf to a temporary Dockerfile. The nameserver should be a valid IP reachable from the container.

Recovery: Override DNS with --dns 8.8.8.8 (or your corporate DNS). If that works, make the change permanent in daemon.json or in your build configuration. Also check that the host's /etc/resolv.conf is correct. If using systemd-resolved, ensure it is running and not misconfigured.

Timeout Connecting to External Services

Symptoms: Build hangs for a long time and eventually fails with connection timed out or no route to host.

Cause: Network isolation, MTU mismatch, or firewall blocking outbound connections. VPNs often cause MTU issues.

Diagnosis: Test connectivity to the target service from the host first. If the host works but the build does not, the issue is likely in the Docker network. Check MTU: compare docker network inspect bridge MTU with the host interface MTU (ip link show). A mismatch can cause packet loss.

Recovery: Set MTU to match the host's effective MTU. Use --network=host as a temporary workaround to see if that resolves it. If yes, you know it is a container network issue. Adjust the MTU in daemon.json or create a custom network with the correct MTU.

Cannot Access Host Services

Symptoms: Build attempts to connect to a service running on the host (e.g., a local database or API) but fails.

Cause: By default, build containers are on a separate bridge network and cannot reach the host via localhost or 127.0.0.1. They must use the host's IP on the bridge network (often 172.17.0.1).

Diagnosis: From within a build container, try ping 172.17.0.1. If that works, use that IP in your build. Alternatively, use the special DNS name host.docker.internal on Docker Desktop (macOS/Windows).

Recovery: Use --network=host for the build (if acceptable). Or, in your Dockerfile, use --add-host=host.docker.internal:host-gateway to map the hostname to the host's gateway IP. For Linux, you may need to add extra_hosts in the daemon or use --add-host flag:

docker build --add-host=host.docker.internal:host-gateway -t myimage .

Then in the Dockerfile, refer to host.docker.internal.

Firewall or Security Group Blocking

Symptoms: Specific ports are blocked, but others work. For example, HTTPS (443) works but HTTP (80) fails.

Cause: Host firewall or cloud security group rules are restricting outbound traffic from Docker's bridge network.

Diagnosis: From the host, test the port using nc or telnet. Then from inside a container, test the same. If the container fails but the host succeeds, the firewall is likely blocking the Docker bridge interface (docker0).

Recovery: Adjust firewall rules to allow traffic from the Docker subnet (e.g., 172.17.0.0/16). On Linux with iptables:

sudo iptables -I OUTPUT -s 172.17.0.0/16 -j ACCEPT

But be cautious with firewall changes; always follow your security policy. Alternatively, use --network=host to bypass Docker's bridge (though that may have its own restrictions).

Rollback Strategies

If a configuration change makes things worse, revert it immediately. Keep a backup of daemon.json and any custom scripts. For command-line changes, simply remove the flags. For daemon.json, restore the previous file and restart Docker. For custom networks, delete them with docker network rm. Document each change so you know what to undo.

Common Pitfalls and How to Avoid Them

Here are frequent mistakes developers make when dealing with BuildKit networking, along with prevention and recovery tips.

Assuming BuildKit behaves like the legacy builder

Why it happens: Developers are used to the old builder's networking, which was less isolated and often shared the host network.

How to avoid: Understand that BuildKit creates a separate network namespace for each build step by default. Test connectivity early in a new build pipeline.

Recovery: If a build fails due to isolation, try --network=host as a quick test, then adjust your Dockerfile or network settings accordingly.

Overlooking MTU mismatch

Why it happens: MTU is often not considered until symptoms like slow transfers or timeouts appear, especially over VPNs.

How to avoid: Always check MTU when builds hang during network operations. Compare host MTU with Docker's MTU.

Recovery: Set the correct MTU in daemon.json or use a custom network with the appropriate MTU.

Hardcoding DNS servers without testing

Why it happens: Teams hardcode public DNS like 8.8.8.8 in daemon.json, which may bypass corporate DNS and cause internal name resolution failures.

How to avoid: Test with --dns on a single build first. Use corporate DNS servers if available.

Recovery: Remove hardcoded DNS from daemon.json and configure appropriate DNS in the build environment.

Ignoring proxy requirements

Why it happens: In corporate environments, builds fail because proxy variables are not passed correctly.

How to avoid: Always pass proxy settings as build args and use them in Dockerfile via ARG/ENV. Test with a simple RUN curl command.

Recovery: Add missing proxy build args and rebuild.

Not documenting network changes

Why it happens: Teams make quick fixes without recording them, leading to confusion later.

How to avoid: Keep a runbook of network configuration changes and their rationale.

Recovery: Review git history and daemon.json backups to understand past changes.

Operations Checklist

Use the following checklist to troubleshoot systematically and prevent future issues. Run through these items in order, from basic checks to advanced diagnostics.

Pre-flight Checks

• [ ] Confirm Docker and BuildKit versions with docker version and docker buildx ls.
• [ ] Verify host internet connectivity: ping 8.8.8.8 and nslookup google.com.
• [ ] Check default bridge network settings: docker network inspect bridge.
• [ ] Note MTU and DNS settings.

During Troubleshooting

• [ ] Reproduce the issue with a minimal Dockerfile (e.g., single RUN that fails).
• [ ] Test DNS resolution inside a build using nslookup.
• [ ] Test external connectivity with wget or curl.
• [ ] Test specific TCP ports with nc.
• [ ] Try overriding DNS using --dns 8.8.8.8.
• [ ] Try --network=host to isolate container network issues.
• [ ] Inspect running build container with docker exec and ip addr.
• [ ] Enable BuildKit debug logs: BUILDKIT_DEBUG=1 docker build --progress=plain ....

After Fix

• [ ] Confirm the build completes successfully.
• [ ] Document the change and the reason.
• [ ] If a global change was made (e.g., daemon.json), ensure it does not affect other builds.
• [ ] Consider adding regression tests (e.g., a healthcheck build in CI).

Routine Maintenance

• [ ] Periodically update Docker and BuildKit to get network fixes.
• [ ] Monitor DNS infrastructure health.
• [ ] Review firewall rules for Docker subnets.
• [ ] Keep a runbook with common symptoms and fixes.

Conclusion

Docker BuildKit networking issues can be complex, but a structured approach simplifies troubleshooting. Begin by gathering version and environment details, then make small, reversible configuration changes. Use targeted diagnostic tests to verify connectivity and identify root causes. Knowing common failure modes such as DNS misconfiguration, MTU mismatch, and host access problems allows for quick recovery.

This guide provided practical commands and expected outputs to guide you through each step. By following the operations checklist, you can resolve issues efficiently and prevent recurrences. Remember to document your findings and adjust your build processes accordingly. With these tools, you can confidently handle BuildKit networking challenges and keep your builds running smoothly.