## Intro Docker Compose simplifies multi-container application deployment by defining services, networks, and volumes in a single YAML file. However, when moving from development to production, networking issues often surface: containers cannot reach each other, DNS resolution fails, ports are not exposed correctly, or external traffic is blocked by firewalls. These problems can cause service outages and slow down incident response. This guide provides practical, step-by-step methods to diagnose and resolve Docker Compose networking issues in production environments. You will learn how to inspect your Docker environment, verify DNS and connectivity, check firewall rules, and apply safe configuration changes with rollback options. Each section includes concrete commands and expected outputs so you can follow along and adapt them to your setup. By the end of this guide, you will have a repeatable troubleshooting workflow that reduces downtime and builds confidence in your Docker Compose deployments. ## Version and Environment Inventory Before troubleshooting, gather precise information about your Docker and Compose versions, network topology, and running containers. This baseline helps identify version-specific bugs and ensures you apply compatible fixes. ### Check Docker and Docker Compose Versions Run the following commands to see installed versions: ```bash docker --version docker compose version ``` Example output: ``` Docker version 24.0.7, build afdd53b Docker Compose version v2.21.0 ``` Note: Docker Compose v2 is a plugin. If you use the older standalone `docker-compose`, the command is `docker-compose --version`. Knowing the exact version is crucial because networking behavior (especially DNS) changed between major versions. ### List Running Containers and Networks Inspect the current state of your deployment: ```bash docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}" docker network ls ``` Example: ``` NAMES IMAGE STATUS PORTS web nginx:alpine Up 2 hours 0.0.0.0:80->80/tcp app myapp:latest Up 2 hours 8000/tcp db postgres:15 Up 2 hours 5432/tcp NETWORK ID NAME DRIVER SCOPE abc123def456 bridge bridge local ghi789jkl012 myapp_default bridge local ``` The `myapp_default` network is typically created by Docker Compose for your project. Note which containers are attached to which networks using `docker network inspect`. ### Check Docker Compose File and Environment Review your `docker-compose.yml` or `docker-compose.yaml` for network definitions and service dependencies. Also check for environment variable substitutions that might affect ports or network settings. Example snippet: ```yaml services: app: build: . ports: - "${APP_PORT:-8000}:8000" environment: DB_HOST: db depends_on: - db db: image: postgres:15 volumes: - db_data:/var/lib/postgresql/data networks: - backend networks: backend: driver: bridge ``` Ensure the Compose file version is compatible with your Docker Engine. Docker Compose v2 ignores the `version` key, so it's optional. ### Prerequisites for Troubleshooting - Shell access to the Docker host with sufficient permissions (typically sudo or being in the docker group). - Basic understanding of Linux networking commands (`ip`, `ss`, `ping`, `curl`, `dig`). - A recent backup of your Docker Compose configuration and persistent volumes (for rollback if needed). - Network access to the host and, if applicable, cloud provider firewall settings. ## Safe Configuration Path When applying network changes in production, follow a scoped approach: make minimal, reversible changes and test after each step. Avoid broad modifications that affect multiple services simultaneously. ### Understand Docker Compose Networking Basics Docker Compose creates a default network for each project (named `_default`). Containers can reach each other by service name (which is also the container name) due to embedded DNS. For explicit control, define custom networks. Example of a custom network with a subnet: ```yaml networks: backend: driver: bridge ipam: config: - subnet: 172.28.0.0/16 ``` Avoid using the default network in production if you need IP range control or network isolation. ### Verify DNS Resolution Inside Containers Docker's embedded DNS server at `127.0.0.11` resolves container names. Test from inside a container: ```bash docker exec -it app nslookup db ``` Expected output: ``` Server: 127.0.0.11 Address: 127.0.0.11#53 Name: db Address: 172.28.0.3 ``` If DNS fails, check that both containers are on the same custom network. Containers on different networks cannot resolve each other by name unless you use a network alias or connect them via an external network. ### Check Port Publishing and Binding Port publishing maps a host port to a container port. Use `docker port` to verify: ```bash docker port ``` Example: ``` 80/tcp -> 0.0.0.0:80 80/tcp -> :::80 ``` This shows that host port 80 is forwarded to container port 80. If you need to change the host port, edit the Compose file `ports` section and run `docker compose up -d` to recreate the container. Important: Changing ports may disrupt existing connections. Always test the new port before removing the old one. You can temporarily expose both ports: ```yaml ports: - "8080:80" - "80:80" ``` Then migrate traffic and remove the old port. ### Use Environment Variables for Configuration Hardcoding IP addresses and ports makes troubleshooting harder. Use environment variables with sensible defaults. In Compose file: ```yaml services: app: environment: DB_HOST: ${DB_HOST:-db} DB_PORT: ${DB_PORT:-5432} ``` In `.env` file: ``` DB_HOST=db DB_PORT=5432 ``` This allows quick changes without editing the Compose file and is safer for production. ### Avoid Host Networking Mode Unless absolutely necessary, avoid `network_mode: host` because it bypasses Docker's network isolation and can cause port conflicts. Prefer bridge networks for better control and security. ## Verification and Diagnostics After any configuration change, verify connectivity and performance. This section provides commands and expected outputs to confirm that networking is functioning correctly. ### Test Inter-Container Communication From within a container, use `ping` (if available) or `curl` to reach another service. Many minimal images lack `ping`; use `wget` or a simple TCP check. Example: ```bash docker compose exec app sh -c "wget -qO- http://db:5432/ || echo 'Connection failed'" ``` For a TCP port check without extra tools, use `nc` (netcat) if present, or install it temporarily (not recommended in production). Instead, use `docker run` with a temporary container on the same network: ```bash docker run --rm -it --network myapp_default alpine sh -c "apk add --no-cache busybox-extras && telnet db 5432" ``` Expected output if connection succeeds: ``` Connected to db ``` If it times out, check the service's listening interface. For example, if a database is configured to listen only on localhost inside the container, it won't accept connections from other containers. The fix is to configure the service to listen on `0.0.0.0`. ### Diagnose DNS Issues If service names do not resolve, inspect the DNS configuration inside the container: ```bash docker exec -it app cat /etc/resolv.conf ``` Expected: ``` nameserver 127.0.0.11 options ndots:0 ``` If it shows a different nameserver (like `8.8.8.8`), Docker's embedded DNS is not being used, possibly because you set `dns` in the Compose file or the container uses a custom network with DNS overridden. Check network connectivity to the Docker DNS: ```bash docker exec -it app sh -c 'echo -e "GET / HTTP/1.0\r\n\r\n" | nc 127.0.0.11 53' ``` This is not a standard DNS query, but it verifies the port is reachable. For a real test, use `dig` if available: ```bash docker exec -it app dig db @127.0.0.11 ``` ### Verify External Connectivity To test outbound internet access from a container: ```bash docker exec -it app wget -qO- http://example.com ``` If this fails, check the host's firewall and IP forwarding. On Linux, ensure IP forwarding is enabled: ```bash sysctl net.ipv4.ip_forward ``` Should output `net.ipv4.ip_forward = 1`. If not, enable it temporarily: ```bash sudo sysctl -w net.ipv4.ip_forward=1 ``` To make permanent, add `net.ipv4.ip_forward=1` to `/etc/sysctl.conf` or a file in `/etc/sysctl.d/`. ### Check Firewall Rules Firewalls (host-based like iptables, or cloud security groups) often block Docker traffic. Docker manipulates iptables automatically, but manual rules can interfere. List iptables rules for the DOCKER chain: ```bash sudo iptables -L DOCKER -n -v ``` Example snippet: ``` Chain DOCKER (2 references) pkts bytes target prot opt in out source destination 0 0 ACCEPT tcp -- !br-abc123def456 br-abc123def456 0.0.0.0/0 172.28.0.2 tcp dpt:5432 ``` This shows that traffic to the database container on port 5432 is allowed. If no rule exists, Docker may be unable to route. For cloud environments, check security groups or firewall settings to ensure the required ports (e.g., 80, 443) are open from the internet or specific IP ranges. ### Examine Docker Network Configuration Inspect the network to see connected containers and IP addresses: ```bash docker network inspect myapp_default ``` Output includes a list of containers with their IPv4 addresses. Verify that the IPs are within the expected subnet and that each container has a unique IP. ### Monitor Logs for Network Errors Container logs often reveal connection issues: ```bash docker logs --tail 50 app ``` Look for errors like "connection refused", "name resolution failed", or "timeout". Adjust the tail number based on the log volume. ## Failure Modes and Recovery Even with careful planning, networking issues can occur. This section outlines common failure modes, their impact, and how to recover safely. ### Common Failure Modes - **DNS Resolution Failure** - **Symptom**: Application logs show `getaddrinfo ENOTFOUND db` or similar. - **Cause**: Containers not on same network, custom DNS settings, or service name mismatch. - **Recovery**: Ensure both containers are attached to the same network (or an external network). Check networks in Compose file. If using custom DNS, remove or adjust the `dns` option. - **Port Binding Conflict** - **Symptom**: `docker compose up` fails with `Bind for 0.0.0.0:80 failed: port is already allocated`. - **Cause**: Another process on the host is using port 80, or a previous container still holds it. - **Recovery**: Identify the process with `sudo lsof -i :80` or `sudo ss -tulpn | grep :80`. Stop the conflicting process or change the host port in Compose. - **Container Cannot Reach External Network** - **Symptom**: Outbound requests from container time out. - **Cause**: IP forwarding disabled, iptables misconfiguration, or Docker daemon not started correctly. - **Recovery**: Enable IP forwarding as shown earlier. Restart Docker daemon (`sudo systemctl restart docker`) after ensuring no critical containers are running (use `docker compose down` first if needed). - **Inter-Container Communication Works by IP but Not by Name** - **Symptom**: `ping 172.28.0.3` works but `ping db` fails. - **Cause**: Custom `/etc/hosts` entries in container overriding Docker DNS, or `network_mode` set incorrectly. - **Recovery**: Remove any manual `/etc/hosts` modifications and ensure the container is on the docker network. Use `docker exec` to check `/etc/hosts`; it should contain the container's own IP and hostname, but not entries for other services (Docker DNS handles those). - **Firewall Blocking Specific Ports** - **Symptom**: Host can reach container port internally, but external clients cannot. - **Cause**: Security group or host firewall blocking the published port. - **Recovery**: Update cloud security group to allow ingress on the port, or add iptables rules. For example, to allow port 8080: ```bash sudo iptables -A INPUT -p tcp --dport 8080 -j ACCEPT ``` Note: This rule is not persistent; use your distribution's firewall management tool (e.g., `ufw allow 8080/tcp`). ### Rollback Strategies Always have a rollback plan before making network changes. - Keep previous Compose file: Before editing, copy `docker-compose.yml` to `docker-compose.yml.bak`. If changes break networking, restore and `docker compose up -d`. - Use version control: Store Compose files in Git. Rollback via `git checkout` and redeploy. - For port changes: Keep the old port mapped temporarily until the new port is confirmed working. Example: ```yaml ports: - "80:80" # old - "8080:80" # new ``` Once verified, remove the old mapping and update load balancer accordingly. - For network reconfiguration: If you need to change network subnet, first create a new network and gradually move services. Docker may require container recreation, so plan for brief downtime. ### Verification After Recovery After applying a fix, always verify: - `docker compose ps` shows all services up and healthy. - DNS resolution works: `docker exec app nslookup db`. - Port connectivity: `curl -I http://localhost:80` from host. - Application logs show no new errors. ## Common Pitfalls and How to Avoid Them Even experienced operators hit recurring snags with Docker Compose networking. Here are common pitfalls and how to steer clear of them. ### Forgetting to Attach Services to the Same Custom Network By default, Compose creates a network per project, but if you define multiple networks and forget to list a service under `networks`, it may be left on the default network. Then other services cannot resolve it by name. Always explicitly define and attach services to the intended networks. ### Overriding Docker's DNS Unintentionally Setting `dns` or `dns_search` in the Compose file for a service replaces Docker's embedded DNS, breaking internal name resolution. Only override DNS if you need external DNS servers; then ensure internal names are still resolvable via `extra_hosts` or network aliases. ### Using `network_mode: host` for Convenience Host mode bypasses Docker networking, which can cause port conflicts and security issues. It also makes the container dependent on the host's network stack. Use bridge networks unless there is a specific reason, such as needing low-latency access to host services. ### Not Checking IP Forwarding Without IP forwarding enabled, containers cannot reach the external network even if the firewall is correct. Many minimal Linux distributions disable it by default. Always verify `sysctl net.ipv4.ip_forward` after kernel updates or system reboots. ### Ignoring Cloud Security Groups Docker's iptables rules control traffic on the host, but cloud providers add another layer. A port might be published correctly but still blocked by the security group. Always check both layers when external access fails. ### Hardcoding IP Addresses Using fixed IP addresses in application configuration or Compose files leads to breakage when containers are recreated with new IPs. Rely on DNS service names and use environment variables for any external dependencies. ### Not Testing Rollback Procedures A rollback plan is only as good as its last test. Regularly simulate a failed deployment and practice restoring from backup. This ensures your team can act quickly under pressure. ## Operations Checklist Use this checklist before and after any network change in Docker Compose production. Each item has an accountable owner and a review frequency. | Step | Action | Command / Check | Owner | Review Frequency | |------|--------|-----------------|-------|------------------| | 1 | Backup Compose file | `cp docker-compose.yml docker-compose.yml.bak` | Priya Shah, Engineering Lead | Before every change | | 2 | Record current state | `docker ps`, `docker network ls` | Priya Shah | Before and after change | | 3 | Test DNS resolution | `docker exec nslookup ` | Alex Rivera, DevOps Engineer | After every change | | 4 | Verify port bindings | `docker port ` | Alex Rivera | After every change | | 5 | Check firewall rules | `sudo iptables -L DOCKER -n -v` | Maria Gomez, Security Specialist | Monthly and after firewall changes | | 6 | Enable IP forwarding | `sysctl net.ipv4.ip_forward` | Alex Rivera | After every reboot | | 7 | Apply configuration change | Edit file, `docker compose up -d` | Priya Shah | As needed | | 8 | Verify after change | Repeat steps 2-6, check logs | Alex Rivera | Immediately after change | | 9 | Update rollback plan | Document the change and how to revert | Priya Shah | After every change | | 10 | Monitor for regressions | `docker logs --tail 100 ` | Priya Shah | Weekly | ## Conclusion Troubleshooting Docker Compose networking in production requires a systematic approach: understand your environment, apply safe changes, verify with concrete commands, and be ready to roll back. By following the steps in this guide, you can quickly identify DNS misconfigurations, port conflicts, firewall blocks, and routing problems. Remember to keep your Compose files version-controlled, use environment variables for flexibility, and test changes in a staging environment whenever possible. The first pilot of any network change should be narrow and measurable, allowing you to inspect results locally before scaling. Use the operations checklist as a routine practice before and after deployments. With these tools, you can minimize downtime and maintain reliable Docker Compose production services. Next steps: Start by inventorying your current Docker environment. Run the version and network commands to establish a baseline. Then, test your inter-container DNS and connectivity. If issues arise, apply the diagnostics in order, and always have a rollback plan ready.