Kubernetes networking troubleshooting with practical examples: practical implementation guide

Kubernetes networking is powerful but layered: name resolution, service routing, pod overlays, node iptables/IPVS, cluster DNS, NetworkPolicy, and cloud or on-prem firewalls all influence traffic. When something breaks, guessing is slow and risky. This guide gives you a safe, reproducible method to isolate failures, validate expected results, and recover quickly with minimal disruption. It includes concrete commands, controlled test workloads, and rollback options.

What you will get:

• A short environment inventory you can capture once and reuse.
• A safe test namespace and echo workload to probe common paths.
• A diagnostic sequence from DNS to pod, service, node, and egress.
• Expected outputs, failure modes, and targeted recovery steps.
• A practical operations checklist for ongoing use.

Scope: This article focuses on troubleshooting. It does not prescribe a specific CNI (for example, Calico, Cilium, Flannel) but shows how to interrogate whichever you run.

Version and Environment Inventory

Before changing anything, record the current state. This prevents misdiagnosis and enables quick rollback.

Prerequisites:

• kubectl access with view permissions to all namespaces and kube-system.
• Optional node SSH access for low-risk, read-only checks (iptables listing, routes). If you do not have node access, skip those steps.

Capture versions and modes:

kubectl version -o yaml | sed -n '1,80p'   # Truncated display for readability
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl -n kube-system get cm kube-proxy -o yaml | grep -i mode -n || true
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system get svc -l k8s-app=kube-dns -o wide || \
  kubectl -n kube-system get svc -l k8s-app=coredns -o wide

If you have node access, capture routing and firewall state safely (list-only):

# On a node (read-only)
sudo ip route
sudo iptables -S | head -n 50
sudo iptables -L -n | head -n 50
# If IPVS is used
sudo ipvsadm -Ln 2>/dev/null | head -n 50 || true

Record your CNI plugin and status:

kubectl -n kube-system get pods -o wide | egrep -i 'calico|cilium|flannel|weave|antrea' || true
kubectl -n kube-system logs deploy/coredns --tail=100 || \
  kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100

Tip: Note MTU settings (node interfaces, CNI config). MTU mismatch often causes intermittent or large-payload failures.

Safe Configuration Path

Use a temporary namespace and minimal workloads you can quickly remove.

Principles:

• Contain all tests in one namespace (for example, net-test).
• Prefer non-disruptive, read-only commands first.
• When you add policies or routes for testing, document and set a clear removal path.

Create the test namespace and two tiny echo apps: one HTTP, one UDP DNS client.

kubectl create namespace net-test

# Simple HTTP echo server and Service (constructed example)
cat <<'YAML' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: echo-a
  namespace: net-test
  labels:
    app: echo-a
spec:
  containers:
  - name: http-echo
    image: hashicorp/http-echo:0.2.3
    args: ["-text=hello-from-echo-a"]
    ports:
    - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: echo-a
  namespace: net-test
spec:
  selector:
    app: echo-a
  ports:
  - name: http
    port: 80
    targetPort: 5678
YAML

# A busybox test pod for curl, nslookup, ping
kubectl -n net-test run net-debug --image=busybox:1.36 --restart=Never -- sleep 36000
kubectl -n net-test wait --for=condition=ready pod/echo-a --timeout=90s || true
kubectl -n net-test get pods -o wide

Expected results:

• echo-a Pod is Running and Ready; Service has a ClusterIP.
• net-debug Pod is Running.

Rollback: To remove all test artifacts, run kubectl delete ns net-test.

Verification and Diagnostics

Work from simplest, most informative checks to those requiring more access. Stop when you find the first failing hop.

1) DNS basics inside the cluster

From the net-debug pod, test service discovery and cluster DNS.

# Get into the pod
kubectl -n net-test exec -it net-debug -- sh

# Inside the pod
nslookup kubernetes.default.svc.cluster.local
nslookup echo-a.net-test.svc.cluster.local

# If nslookup is missing, install via busybox tools or use getent hosts if available

Expected results:

• Both names resolve to ClusterIPs. Failures indicate DNS or search path problems.

If name resolution fails:

• Check CoreDNS/Kube-DNS pod status and logs.
• Ensure the net-debug pod has /etc/resolv.conf pointing to the cluster DNS service IP.

Commands:

# Back outside the pod
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide || \
  kubectl -n kube-system get pods -l k8s-app=coredns -o wide
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 || \
  kubectl -n kube-system logs -l k8s-app=coredns --tail=200
kubectl -n net-test exec net-debug -- cat /etc/resolv.conf

Typical signals:

• SERVFAIL or timeout: DNS service unreachable or CoreDNS plugins misconfigured.
• NXDOMAIN for service names: Kubernetes DNS plugin issues or wrong search domains.

2) Pod-to-Service (ClusterIP) path

From net-debug, curl the echo-a Service by DNS and IP.

# Inside net-debug
wget -qO- http://echo-a
wget -qO- http://<ECHO_A_CLUSTER_IP>

Expected results:

• Response text: hello-from-echo-a.

If DNS works but IP fails, kube-proxy rules or CNI routing may be broken.

Check kube-proxy mode and rules:

# Outside
kubectl -n kube-system get cm kube-proxy -o yaml | grep -i mode -n || true
# On a node (read-only): expect KUBE- service chains for iptables mode
sudo iptables-save | egrep 'KUBE-SVC|KUBE-SEP' | head -n 20 || true
sudo ipvsadm -Ln | head -n 20 || true

3) Pod-to-Pod connectivity (same and cross node)

Find echo-a Pod IP and ping/curl it directly.

kubectl -n net-test get pod -o wide -l app=echo-a
# Inside net-debug
PING_IP=$(kubectl -n net-test get pod -l app=echo-a -o jsonpath='{.items[0].status.podIP}')
ping -c2 $PING_IP || true
# curl via Pod IP and port
wget -qO- http://$PING_IP:5678

Expected results:

• Pings may be blocked by ICMP policy; success is informative but failure is not conclusive.
• HTTP should succeed. If not, suspect CNI, NetworkPolicy, or routes.

4) NetworkPolicy checks

List policies in the namespace and test allowances.

kubectl -n net-test get networkpolicy

If you need to validate policy effects, apply a constructed default-deny and then an allow rule, then roll back.

# Default deny all ingress (constructed example)
cat <<'YAML' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: net-test
spec:
  podSelector: {}
  policyTypes:
  - Ingress
YAML

# Allow net-debug to echo-a
cat <<'YAML' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-debug-to-echo
  namespace: net-test
spec:
  podSelector:
    matchLabels:
      app: echo-a
  ingress:
  - from:
    - podSelector:
        matchLabels:
          run: net-debug
YAML

Expected results:

• With default-deny applied, curl from net-debug to echo-a should fail with timeout.
• With allow rule applied, curl should succeed again.

Rollback:

kubectl -n net-test delete networkpolicy default-deny allow-debug-to-echo --ignore-not-found

5) NodePort and external access

If the workload is intended to be externally reachable via NodePort, expose it and test from outside the cluster.

kubectl -n net-test expose pod echo-a --type=NodePort --name=echo-a-nodeport --port=80 --target-port=5678
kubectl -n net-test get svc echo-a-nodeport -o wide

From a workstation or another node:

# Replace NODE_IP and NODEPORT accordingly
curl -sS http://NODE_IP:NODEPORT

If this fails while ClusterIP works, check node firewall or cloud security groups for the NodePort range (default 30000-32767/TCP).

6) Egress to the internet

From net-debug, test external DNS and HTTP.

# Inside net-debug
nslookup example.com
wget -qO- https://ifconfig.me || wget -qO- http://ifconfig.me || true

If DNS to the internet fails but cluster DNS works, your cluster DNS forwarders or upstream resolvers may be blocked. If HTTP egress fails, check NAT gateway, egress gateways, or cloud route tables.

7) MTU and fragmentation

Jumbo or overlay MTU mismatch causes silent drops for larger packets.

# Inside net-debug, pick a reachable external IP (constructed example 1.1.1.1)
ping -s 1472 -M do -c 2 1.1.1.1 || true
ping -s 1200 -M do -c 2 1.1.1.1 || true

Expected results:

• A larger payload may fail while a smaller one passes. If so, align MTU between node interfaces and CNI overlay.

8) CoreDNS configuration sanity

Check CoreDNS ConfigMap for unusual rewrites or upstreams.

kubectl -n kube-system get configmap coredns -o yaml

Look for:

• Loop or heavy rewrite plugins.
• Upstream resolvers unreachable from cluster nodes.
• Cache plugin configuration that is overly aggressive.

9) Observability with events and logs

Quick wins often come from events and targeted logs.

kubectl get events -A --sort-by=.lastTimestamp | tail -n 30
kubectl -n kube-system logs -l k8s-app=kube-proxy --tail=200
# For your CNI, example selectors (may differ):
kubectl -n kube-system logs -l k8s-app=calico-node --tail=200 || true
kubectl -n kube-system logs -l k8s-app=cilium --tail=200 || true

Common network paths and where to look

Failure Modes and Recovery

The table below maps common symptoms to concrete recovery steps and rollbacks. Examples are constructed; adapt values to your environment.

| ClusterIP curl fails, Pod IP works | kube-proxy tables broken | iptables-save | grep KUBE-SVC or ipvsadm -Ln | Restart kube-proxy DaemonSet; reconcile config | Roll back kube-proxy config change | | Cross-node pod reachability fails | CNI not ready or MTU mismatch | kubectl -n kube-system logs <cni-daemon>; MTU test | Restart CNI DaemonSet; align MTU | Restore previous CNI config | | NodePort unreachable externally | Node firewall / SG blocks | curl NODE_IP:NODEPORT from outside; check SG | Open NodePort range; allow node security group | Revert firewall/SG change | | Egress DNS works, HTTP fails | NAT/egress route issue | wget -qO- ifconfig.me fails | Fix NAT gateway / SNAT; correct routes | Revert route/NAT edits | | Intermittent large-payload drops | MTU fragmentation blocked | ping -M do -s 1472 fails; smaller works | Lower overlay MTU; enable path MTU discovery | Revert MTU change if ineffective |

Rollback and safety notes

• Always apply changes to the net-test namespace or non-production first.
• Backup ConfigMaps and iptables/IPVS state before edits:

kubectl -n kube-system get cm coredns -o yaml > coredns.bak.yaml
sudo iptables-save > iptables-$(date +%s).bak

• For NetworkPolicy experiments, delete policies to restore previous behavior:

kubectl -n net-test delete networkpolicy --all

• For CNI changes, coordinate a rolling node drain and uncordon plan; be prepared to revert to the prior DaemonSet image or config.

Ports and components to know

Use this as a quick reference when firewalling between nodes, control plane, and pods (constructed, adjust to your setup).

Verification wrap-up

After each fix, re-run the minimal test set to confirm recovery and avoid regressions:

1. DNS inside pods:

kubectl -n net-test exec net-debug -- nslookup echo-a.net-test.svc.cluster.local

1. ClusterIP routing:

kubectl -n net-test exec net-debug -- wget -qO- http://echo-a

1. Pod IP direct:

ECHO_IP=$(kubectl -n net-test get pod -l app=echo-a -o jsonpath='{.items[0].status.podIP}')
kubectl -n net-test exec net-debug -- wget -qO- http://$ECHO_IP:5678

1. Optional external:

curl -sS http://NODE_IP:NODEPORT || true

Expected: all previously failing steps now pass.

Operations Checklist

Use this concise list during incidents and routine checks.

• Capture inventory
• kubectl version, cluster-info, nodes -o wide
• kube-proxy mode; CNI pods and logs
• Note DNS Service IP and CoreDNS status
• Prepare safe test scope
• Create net-test namespace
• Deploy echo-a and net-debug pods
• Confirm readiness
• Run diagnostics in order
• In-pod DNS: nslookup kubernetes and echo-a
• Service HTTP via DNS and ClusterIP
• Pod IP direct access
• NetworkPolicy presence and effects
• NodePort from external (if relevant)
• Egress DNS and HTTP
• MTU checks if symptoms fit
• Identify failure domain
• DNS vs proxy/routing vs policy vs firewall vs MTU
• Apply targeted fix
• Edit ConfigMap, policy, firewall, or CNI/kube-proxy as indicated
• Verify and rollback if needed
• Re-run core tests
• If no improvement, revert the last change cleanly
• Clean up
• Delete net-test namespace when finished
• Archive logs and commands for post-incident review

Conclusion

Kubernetes networking problems become tractable when you approach them methodically. Start by inventorying versions, modes, and CNI status. Use a safe, temporary namespace and a minimal test workload to probe DNS, Service routing, pod paths, NodePort, and egress. Compare outputs to expected results to pinpoint the failing layer. When you make changes, keep them scoped and reversible, verify fixes immediately, and roll back if results do not improve. With the commands, examples, and checklists in this guide, you can diagnose and recover from most cluster networking issues confidently and repeatably.