Intro
Linux networking troubleshooting works best as a repeatable, evidence-driven process rather than a collection of memorized commands. When a service stops responding, a deployment cannot reach its database, or DNS returns unexpected answers, an operator needs to move from symptom to root cause without making the problem worse.
This guide is written for developers, DevOps consultants, and startup engineers who operate Linux systems directly. It covers the four areas that cause most incidents: DNS resolution, local port state, end-to-end connectivity, and the diagnostic workflow that ties them together. Every section shows a read-only command first, explains the output you should expect, identifies common failure signals, and only then introduces the smallest safe change with a verification step and a rollback path.
Operational safety is the recurring theme. Observe before you modify, limit the blast radius of any change, keep secrets and production identifiers out of your command history and documentation, and always define what success looks like before you press Enter.
Version and Environment Inventory
Before troubleshooting, you need a reliable inventory of the system you are about to inspect. Guessing the distribution, kernel version, or network manager can lead you to run commands that are not installed, produce misleading output, or modify the wrong configuration file.
Start with three read-only commands that work on virtually every Linux distribution.
cat /etc/os-release
uname -r
ip -brief address/etc/os-release shows the distribution and release version, for example PRETTY_NAME="Ubuntu 22.04.3 LTS" or VERSION_ID="9" on Debian 9. uname -r prints the running kernel version like 5.15.0-91-generic. ip -brief address lists all interfaces and their IPv4 and IPv6 addresses in a compact table.
Sample output from the last command might look like this:
lo UNKNOWN 127.0.0.1/8 ::1/128
eth0 UP 10.0.0.42/24 2001:db8::42/64From this you learn the interface name (eth0, not the legacy eth1 or a predictable name like enp3s0), the link state (UP), and the configured addresses. If eth0 were DOWN instead of UP, connectivity issues would be expected and the next step would be to inspect ip link show eth0 and the network service logs, not to start testing remote ports.
Next, identify which DNS resolver the system is using and which services are listening on which ports. Both observations are read-only.
cat /etc/resolv.conf
ss -lntup/etc/resolv.conf may contain nameserver 127.0.0.53, indicating systemd-resolved as the local stub resolver, or nameserver 10.0.0.2, meaning direct use of a local DNS server. ss -lntup shows listening TCP and UDP sockets with the owning process. Look for :22 (sshd), :80 or :443 (a web server), and :53 if a local resolver like dnsmasq is running.
Example output:
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=812,fd=3))
LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=1200,fd=6))
LISTEN 0 128 [::]:22 [::]:* users:(("sshd",pid=812,fd=4))If ss is not available, install the iproute2 package (usually already present) or use the older netstat -lntup, but verify that the package version is appropriate for your distribution. On Alpine Linux, for example, ss is part of iproute2, which may need to be added with apk add iproute2.
Record this inventory before making any change. It gives you a known-good baseline and makes any drift obvious later.
Safe Configuration Path
When a change is required, follow a path that minimizes risk: back up the file, modify one setting at a time, restart only the affected service, verify, and know how to roll back.
Consider a common scenario: an application reports that it cannot connect to a PostgreSQL database at db.internal.example.com. The application is running on the same host, and you suspect a DNS misconfiguration. Before editing anything, verify the current resolution and connectivity.
First, test DNS resolution with a read-only query:
getent hosts db.internal.example.comExpected output includes an IP address:
10.0.1.15 db.internal.example.comIf the command returns nothing or an error like Name or service not known, DNS is not resolving the name. Check /etc/hosts and /etc/resolv.conf, and test upstream resolution with dig or nslookup if available.
Suppose you discover that /etc/resolv.conf points to a nameserver that no longer exists. You decide to switch to a known-good resolver such as Cloudflare's 1.1.1.1. This is a single-scope change with an easy rollback.
- Back up the current file:
cp /etc/resolv.conf /etc/resolv.conf.bak.$(date +%Y%m%d)- Check whether
/etc/resolv.confis a symlink (common on systems usingsystemd-resolvedor NetworkManager). If it is, edit the target file or use the appropriate tool (resolvectlfor systemd-resolved,nmclifor NetworkManager) to avoid your changes being overwritten.
- Make a minimal edit. If editing the file directly, replace the
nameserverline with:
nameserver 1.1.1.1
nameserver 8.8.8.8- Verify resolution immediately:
getent hosts db.internal.example.com- If the name still does not resolve, restore the backup:
cp /etc/resolv.conf.bak.20240115 /etc/resolv.confThis five-step pattern – observe, backup, minimal change, verify, rollback – applies to most configuration edits. Never edit multiple network settings at once, because if the service fails you will not know which change caused the problem.
For services that are managed by systemd, use systemctl edit rather than directly editing unit files where possible. For example, to add an environment variable to a service without touching the packaged unit file:
systemctl edit myapp.serviceThen add:
[Service]
Environment="MYAPP_DNS=1.1.1.1"Reload and restart only that service:
systemctl daemon-reload
systemctl restart myapp.serviceVerify with systemctl status myapp.service and by checking the application logs for a successful startup message. If the service fails, systemctl revert myapp.service removes the override and returns to the packaged defaults.
Verification and Diagnostics
Verification is not the same as checking that a command executed without error. A command can exit successfully and still hide a misconfiguration. You must compare the observed output against an expected, version-appropriate result.
DNS Verification
Use dig for detailed, scriptable DNS queries. dig is part of the bind-utils package on Red Hat derivatives and dnsutils on Debian/Ubuntu.
dig +short db.internal.example.comExpected output is a bare IP address:
10.0.1.15If you get no output or a timeout, run the full query to see the response code:
dig db.internal.example.comLook for the status and ANSWER sections. A SERVFAIL status means the authoritative server could not answer; NXDOMAIN means the name definitely does not exist; NOERROR with zero answers means the name exists but has no A record.
When debugging a public domain, you can query a specific resolver to isolate problems:
dig @1.1.1.1 example.com AIf this works but the system default resolver fails, the problem is with the local DNS configuration, not the domain itself.
Port Verification
To check whether a local service is actually listening on the expected port, ss is the standard tool.
ss -lntp | grep ':5432'Example output for PostgreSQL:
LISTEN 0 4096 127.0.0.1:5432 0.0.0.0:* users:(("postgres",pid=1500,fd=7))If the line is missing, the service is not listening at all. If it says 127.0.0.1:5432, it is bound only to localhost, so remote clients will be refused. If it says 0.0.0.0:5432, it is accepting connections on all interfaces.
To test from the client perspective, use nc (netcat) or telnet:
nc -zv 10.0.1.15 5432The -z flag scans without sending data, -v makes it verbose. A successful connection prints something like:
Connection to 10.0.1.15 5432 port [tcp/postgresql] succeeded!A failure prints nothing or Connection refused. Remember that Connection refused means the packet reached the host but nothing was listening, while a timeout often indicates a firewall dropping packets silently.
Connectivity Verification
ping is useful for basic reachability, but ICMP can be blocked while TCP services still work. Do not rely on ping alone. Use traceroute or tracepath to see the route, and mtr for a continuous combined view.
traceroute -n 10.0.1.15The -n flag prevents reverse DNS lookups, which can stall the output. Each hop line shows the round-trip time to that router. A line of asterisks indicates a router that does not respond to ICMP, but subsequent hops may still appear.
For a quick test of a specific TCP port along the way, use nc with a timeout:
nc -zvw 5 10.0.1.15 443The -w 5 sets a five-second timeout. If the connection succeeds, the port is open and reachable.
Failure Modes and Recovery
Recognizing common failure patterns speeds up diagnosis and prevents misguided fixes.
DNS Failure: Name or service not known
Why it happens: The resolver cannot map the hostname to an IP address. Causes include a missing /etc/hosts entry, an unreachable nameserver, or the name genuinely not existing.
How to diagnose:
getent hosts db.internal.example.com
cat /etc/resolv.conf
ping -c 1 1.1.1.1If /etc/resolv.conf points to an unreachable IP, ping to that IP will fail while ping 1.1.1.1 succeeds. Fix by editing /etc/resolv.conf as described earlier, or by reconfiguring the network manager.
Recovery verification: getent hosts returns an IP, and dig shows a NOERROR status.
Port Failure: Connection refused
Why it happens: The target port is not open on the destination host. Either no process is listening, the service is bound to a different interface (e.g., localhost only), or a firewall is actively rejecting the connection.
How to diagnose:
ss -lntp | grep ':5432' # on the server
nc -zv 10.0.1.15 5432 # from the clientIf ss shows the service listening on 127.0.0.1:5432, change the service configuration to listen on 0.0.0.0 or the specific server IP. For PostgreSQL, edit postgresql.conf and set listen_addresses = '*', then restart PostgreSQL. If ss shows no listener at all, the service is not running; check systemctl status postgresql and start it.
Recovery verification: nc -zv from a remote host succeeds.
Connectivity Failure: Timeout
Why it happens: Packets are being dropped, usually by a firewall, or the route is wrong.
How to diagnose:
traceroute -n 10.0.1.15
iptables -L -n -v # or nft list ruleset
ip route showIf traceroute stops at a certain hop and never reaches the destination, inspect the firewall rules on that hop. For the local host, check iptables -L -n -v for DROP or REJECT rules affecting the destination port. A common mistake is a ufw default deny policy:
ufw status verboseIf ufw is active and the port is not allowed, add a rule:
ufw allow 5432/tcpRecovery verification: The port test from the client now returns succeeded.
Interface Failure: ip link shows DOWN
Why it happens: The network cable is unplugged, the virtual interface is administratively down, or the driver failed to initialize.
How to diagnose:
ip link show eth0
dmesg | tail -20If the state is DOWN, try bringing it up administratively:
ip link set eth0 upThen check again. If it remains DOWN, inspect dmesg for driver errors such as link is not ready or failed to allocate. For virtual machines, verify that the hypervisor virtual network adapter is connected.
Recovery verification: ip -brief address shows UP and an IP address.
Operations Checklist
A checklist turns incident response into a systematic process. Use the following steps in order, and document the output of each command in your incident notes.
1. Baseline Inventory (Owner: On-call engineer, reviewed at incident start)
cat /etc/os-release
uname -r
ip -brief address
cat /etc/resolv.conf
ss -lntupRecord the output. This tells you what you are working with and prevents later confusion.
2. Reproduce the Failure (Owner: On-call engineer, reviewed as soon as reported)
Run the exact command the application or user reports as failing. Note the exact error message. For example:
nc -zv db.internal.example.com 5432If it fails with Connection refused, you have a port problem. If it hangs and times out, you have a reachability problem.
3. Isolate the Layer (Owner: Network engineer if escalated, reviewed after reproduction)
- DNS:
dig +short db.internal.example.com– does it return an IP? - Port:
ss -lntp | grep 5432on the server – is the service listening? - Route:
ip route get 10.0.1.15– which interface and gateway are used? - Firewall:
iptables -L -n -v | grep 5432– any DROP rules?
4. Make One Scoped Change (Owner: On-call engineer, reviewed before execution)
If DNS is the issue, edit /etc/resolv.conf after backup. If the port is closed, adjust the service configuration. Document the exact change in the incident log.
5. Verify and Rollback (Owner: On-call engineer, reviewed immediately after change)
Run the same diagnostic from step 3. If the result is still failing, revert the change using the backup or systemctl revert. If the result is successful, monitor for at least five minutes to ensure stability.
6. Post-Incident Review (Owner: Engineering lead, reviewed within 48 hours)
Ask: What was the root cause? What command or check would have caught this before users noticed? Add that check to your monitoring or runbook.
This checklist should be revisited after every incident or at least quarterly to keep it current with your infrastructure changes.
Common Pitfalls and How to Avoid Them
Even experienced operators make these mistakes. Recognizing them saves time and prevents outages.
1. Editing /etc/resolv.conf and losing changes after reboot
Why it happens: On many systems, /etc/resolv.conf is managed by NetworkManager, systemd-resolved, or DHCP. A manual edit is overwritten.
How to avoid: Check if the file is a symlink with ls -l /etc/resolv.conf. If it is, use the appropriate management tool. For NetworkManager, use nmcli con mod "Wired connection 1" ipv4.dns "1.1.1.1 8.8.8.8" followed by nmcli con up "Wired connection 1". For systemd-resolved, edit /etc/systemd/resolved.conf and set DNS=1.1.1.1 8.8.8.8.
2. Testing connectivity with ping only
Why it happens: ping is the first tool that comes to mind, but ICMP is often blocked while TCP ports are open.
How to avoid: Always test the actual service port with nc or telnet. If ICMP is blocked, ping will fail even though the service is fine.
3. Checking the wrong interface
Why it happens: A host may have multiple interfaces, and the service might be bound to a different one than you think.
How to avoid: Use ss -lntp to see the exact bind address. Compare it with ip -brief address to know which interface that address belongs to. If the bind address is 127.0.0.1, remote clients will always be refused.
4. Assuming a firewall is not present
Why it happens: Many cloud images and minimal installs have iptables or nftables rules by default, and ufw or firewalld may be active without the operator's knowledge.
How to avoid: Always check firewall status: ufw status or firewall-cmd --state or iptables -L -n. If policies exist, review them before blaming DNS or the application.
5. Logging into production and running multiple changes at once
Why it happens: Pressure to fix quickly leads to changing DNS, firewall, and service configuration simultaneously.
How to avoid: Change one thing at a time. After each change, verify. If the problem remains, roll back that change before trying the next. This isolates the cause and makes recovery predictable.
Conclusion
Linux network troubleshooting is a skill that rewards discipline. The commands themselves are not difficult; the challenge is to know what output to expect, to interpret unexpected output correctly, and to make changes that can be undone.
Start with a complete inventory: OS version, kernel, interfaces, DNS resolver, and listening ports. Reproduce the failure with the exact command that is failing. Isolate the problem to one layer - DNS, port, route, or firewall. Make a single scoped change with a backup or rollback plan. Verify with a command whose expected output you defined before making the change. Then document the incident and update your runbook.
Selecting one low-risk verification to automate is a good first step after reading this guide. For example, create a cron job that runs nc -zv db.internal.example.com 5432 > /dev/null 2>&1 || echo "DB port check failed at $(date)" >> /var/log/db-port-check.log every five minutes. This makes the failure visible before an application reports it.
A reliable workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces a hurried decision. With these practices, you can resolve network issues with confidence and keep your systems available.