Kubernetes Networking Troubleshooting: A Step-by-Step Guide with Commands

Introduction

Kubernetes networking spans name resolution, service routing, pod overlays, node iptables or IPVS, cluster DNS, NetworkPolicy, and cloud or on-prem firewalls. When traffic breaks, guessing wastes time and risks disruption. This guide gives you a reproducible method to isolate failures, validate expected results, and recover quickly. It includes concrete commands, controlled test workloads, and rollback steps.

What you will get:

A one-time environment inventory you can reuse
A safe test namespace with an echo workload to probe common paths
A diagnostic sequence from DNS through pod, service, node, and egress
Expected outputs, failure modes, and targeted recovery steps
A concise operations checklist for ongoing use

Scope: This article focuses on troubleshooting. It does not prescribe a specific CNI (Calico, Cilium, Flannel, etc.) but shows how to interrogate whichever you run.

Version and Environment Inventory

Before changing anything, record the current state. This prevents misdiagnosis and enables quick rollback.

Prerequisites:

kubectl access with view permissions to all namespaces and kube-system
Optional node SSH access for read-only checks (iptables listing, routes). Skip if unavailable.

Capture versions and modes:

kubectl version -o yaml | sed -n '1,80p'
kubectl cluster-info
kubectl get nodes -o wide
kubectl get pods -A -o wide
kubectl -n kube-system get cm kube-proxy -o yaml | grep -i mode -n || true
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system get svc -l k8s-app=kube-dns -o wide || \
 kubectl -n kube-system get svc -l k8s-app=coredns -o wide

If you have node access, capture routing and firewall state (list-only):

# On a node (read-only)
sudo ip route
sudo iptables -S | head -n 50
sudo iptables -L -n | head -n 50
# If IPVS is used
sudo ipvsadm -Ln 2>/dev/null | head -n 50 || true

Record your CNI plugin and status:

kubectl -n kube-system get pods -o wide | egrep -i 'calico|cilium|flannel|weave|antrea' || true
kubectl -n kube-system logs deploy/coredns --tail=100 || \
 kubectl -n kube-system logs -l k8s-app=kube-dns --tail=100

Tip: Note MTU settings on node interfaces and in CNI config. MTU mismatch often causes intermittent or large-payload failures.

Safe Configuration Path

Use a temporary namespace and minimal workloads you can quickly remove.

Principles:

Contain all tests in one namespace (for example, net-test)
Prefer non-disruptive, read-only commands first
When you add policies or routes for testing, document and set a clear removal path

Create the test namespace and two tiny echo apps:

kubectl create namespace net-test

# HTTP echo server and Service
cat <<'YAML' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: echo-a
  namespace: net-test
  labels:
    app: echo-a
spec:
  containers:
  - name: http-echo
    image: hashicorp/http-echo:0.2.3
    args: ["-text=hello-from-echo-a"]
    ports:
    - containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
  name: echo-a
  namespace: net-test
spec:
  selector:
    app: echo-a
  ports:
  - name: http
    port: 80
    targetPort: 5678
YAML

# Busybox test pod for curl, nslookup, ping
kubectl -n net-test run net-debug --image=busybox:1.36 --restart=Never -- sleep 36000
kubectl -n net-test wait --for=condition=ready pod/echo-a --timeout=90s || true
kubectl -n net-test get pods -o wide

Expected results:

echo-a Pod is Running and Ready; Service has a ClusterIP
net-debug Pod is Running

Rollback: Remove all test artifacts with kubectl delete ns net-test.

Verification and Diagnostics

Work from simplest checks to those requiring more access. Stop when you find the first failing hop.

1) DNS basics inside the cluster

From the net-debug pod, test service discovery and cluster DNS.

# Get into the pod
kubectl -n net-test exec -it net-debug -- sh

# Inside the pod
nslookup kubernetes.default.svc.cluster.local
nslookup echo-a.net-test.svc.cluster.local

Expected: Both names resolve to ClusterIPs. Failures indicate DNS or search path problems.

If name resolution fails:

Check CoreDNS/Kube-DNS pod status and logs
Ensure the net-debug pod has /etc/resolv.conf pointing to the cluster DNS service IP

Commands:

# Back outside the pod
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide || \
 kubectl -n kube-system get pods -l k8s-app=coredns -o wide
kubectl -n kube-system logs -l k8s-app=k8s-dns --tail=200 || \
 kubectl -n kube-system logs -l k8s-app=coredns --tail=200
kubectl -n net-test exec net-debug -- cat /etc/resolv.conf

Typical signals:

SERVFAIL or timeout: DNS service unreachable or CoreDNS plugins misconfigured
NXDOMAIN for service names: Kubernetes DNS plugin issues or wrong search domains

2) Pod-to-Service (ClusterIP) path

From net-debug, curl the echo-a Service by DNS and IP.

# Inside net-debug
wget -qO- http://echo-a
wget -qO- http://<ECHO_A_CLUSTER_IP>

Expected: Response text hello-from-echo-a.

If DNS works but IP fails: kube-proxy rules or CNI routing may be broken.

Check kube-proxy mode and rules:

# Outside
kubectl -n kube-system get cm kube-proxy -o yaml | grep -i mode -n || true
# On a node (read-only): expect KUBE- service chains for iptables mode
sudo iptables-save | egrep 'KUBE-SVC|KUBE-SEP' | head -n 20 || true
sudo ipvsadm -Ln | head -n 20 || true

3) Pod-to-Pod connectivity (same and cross node)

Find echo-a Pod IP and ping/curl it directly.

kubectl -n net-test get pod -o wide -l app=echo-a
# Inside net-debug
PING_IP=$(kubectl -n net-test get pod -l app=echo-a -o jsonpath='{.items[0].status.podIP}')
ping -c2 $PING_IP || true
wget -qO- http://$PING_IP:5678

Expected:

Pings may be blocked by ICMP policy; success is informative but failure is not conclusive
HTTP should succeed. If not, suspect CNI, NetworkPolicy, or routes

4) NetworkPolicy checks

List policies in the namespace and test allowances.

kubectl -n net-test get networkpolicy

To validate policy effects, apply a default-deny then an allow rule, then roll back:

# Default deny all ingress
cat <<'YAML' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: net-test
spec:
  podSelector: {}
  policyTypes:
  - Ingress
YAML

# Allow net-debug to echo-a
cat <<'YAML' | kubectl apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-debug-to-echo
  namespace: net-test
spec:
  podSelector:
    matchLabels:
      app: echo-a
  ingress:
  - from:
    - podSelector:
        matchLabels:
          run: net-debug
YAML

Expected:

With default-deny applied, curl from net-debug to echo-a should fail with timeout
With allow rule applied, curl should succeed again

Rollback:

kubectl -n net-test delete networkpolicy default-deny allow-debug-to-echo --ignore-not-found

5) NodePort and external access

If the workload is intended to be externally reachable via NodePort, expose it and test from outside the cluster.

kubectl -n net-test expose pod echo-a --type=NodePort --name=echo-a-nodeport --port=80 --target-port=5678
kubectl -n net-test get svc echo-a-nodeport -o wide

From a workstation or another node:

# Replace NODE_IP and NODEPORT accordingly
curl -sS http://NODE_IP:NODEPORT

If this fails while ClusterIP works: check node firewall or cloud security groups for the NodePort range (default 30000-32767/TCP).

6) Egress to the internet

From net-debug, test external DNS and HTTP.

# Inside net-debug
nslookup example.com
wget -qO- https://ifconfig.me || wget -qO- http://ifconfig.me || true

If DNS to the internet fails but cluster DNS works: your cluster DNS forwarders or upstream resolvers may be blocked. If HTTP egress fails, check NAT gateway, egress gateways, or cloud route tables.

7) MTU and fragmentation

Jumbo or overlay MTU mismatch causes silent drops for larger packets.

# Inside net-debug, pick a reachable external IP (example 1.1.1.1)
ping -s 1472 -M do -c 2 1.1.1.1 || true
ping -s 1200 -M do -c 2 1.1.1.1 || true

Expected: A larger payload may fail while a smaller one passes. If so, align MTU between node interfaces and CNI overlay. For VXLAN overlays, a common fix is setting the CNI MTU to 1450 when the node interface is 1500.

8) CoreDNS configuration sanity

Check CoreDNS ConfigMap for unusual rewrites or upstreams.

kubectl -n kube-system get configmap coredns -o yaml

Look for:

Loop or heavy rewrite plugins
Upstream resolvers unreachable from cluster nodes
Cache plugin configuration that is overly aggressive

9) Observability with events and logs

Quick wins often come from events and targeted logs.

kubectl get events -A --sort-by=.lastTimestamp | tail -n 30
kubectl -n kube-system logs -l k8s-app=kube-proxy --tail=200
# For your CNI, example selectors (may differ):
kubectl -n kube-system logs -l k8s-app=calico-node --tail=200 || true
kubectl -n kube-system logs -l k8s-app=cilium --tail=200 || true

Common Network Paths and Where to Look

Use this reference to map symptoms to components:

Pod -> ClusterIP Service — Test from pod; primary components: kube-proxy, CNI routing; typical fix area: kube-proxy rules, CNI readiness
Pod -> Pod (same node) — Test from pod; primary components: CNI veth/bridge; typical fix area: pod sandbox, CNI, policy
Pod -> Pod (cross node) — Test from pod; primary components: CNI overlay, node routes; typical fix area: node routing, MTU, CNI daemon
Pod -> Internet (egress) — Test from pod; primary components: NAT/egress GW, routes; typical fix area: NAT gateways, firewall, egress policy
External -> NodePort — Test from workstation; primary components: node firewall, kube-proxy; typical fix area: node firewall, security groups

Failure Modes and Recovery

The list below maps common symptoms to concrete recovery steps and rollbacks. Adapt values to your environment.

Symptom: Service DNS times out

Likely cause: CoreDNS down or unreachable
Verification: kubectl -n kube-system get pods -l k8s-app=coredns
Recovery: Restart or scale CoreDNS; ensure DNS Service IP reachable from pods
Rollback: Revert any CoreDNS ConfigMap edits

Symptom: Service name NXDOMAIN

Likely cause: Wrong search domains or plugin
Verification: kubectl -n net-test exec net-debug -- cat /etc/resolv.conf
Recovery: Fix search suffix; correct CoreDNS plugins
Rollback: Restore previous ConfigMap

Symptom: ClusterIP curl fails, Pod IP works

Likely cause: kube-proxy tables broken
Verification: iptables-save | grep KUBE-SVC or ipvsadm -Ln
Recovery: Restart kube-proxy DaemonSet; reconcile config
Rollback: Roll back kube-proxy config change

Symptom: Cross-node pod reachability fails

Likely cause: CNI not ready or MTU mismatch
Verification: kubectl -n kube-system logs <cni-daemon>; MTU test
Recovery: Restart CNI DaemonSet; align MTU
Rollback: Restore previous CNI config

Symptom: NodePort unreachable externally

Likely cause: Node firewall / security group blocks
Verification: curl NODE_IP:NODEPORT from outside; check security group
Recovery: Open NodePort range; allow node security group
Rollback: Revert firewall/security group change

Symptom: Egress DNS works, HTTP fails

Likely cause: NAT/egress route issue
Verification: wget -qO- ifconfig.me fails
Recovery: Fix NAT gateway / SNAT; correct routes
Rollback: Revert route/NAT edits

Symptom: Intermittent large-payload drops

Likely cause: MTU fragmentation blocked
Verification: ping -M do -s 1472 fails; smaller works
Recovery: Lower overlay MTU; enable path MTU discovery
Rollback: Revert MTU change if ineffective

Rollback and safety notes

Always apply changes to the net-test namespace or non-production first
Backup ConfigMaps and iptables/IPVS state before edits:

  kubectl -n kube-system get cm coredns -o yaml > coredns.bak.yaml
  sudo iptables-save > iptables-$(date +%s).bak

For NetworkPolicy experiments, delete policies to restore previous behavior:

  kubectl -n net-test delete networkpolicy --all

For CNI changes, coordinate a rolling node drain and uncordon plan; be prepared to revert to the prior DaemonSet image or config

Ports and Components to Know

Quick reference when firewalling between nodes, control plane, and pods (adjust to your setup):

kube-apiserver — TCP/6443 — Control plane API
kubelet — TCP/10250 — Node kubelet API
CoreDNS — UDP/53, TCP/53 — Cluster DNS
NodePort range — TCP/30000-32767 — External access to Services
etcd (if self-managed) — TCP/2379-2380 — Control plane storage

Verification Wrap-Up

After each fix, re-run the minimal test set to confirm recovery and avoid regressions:

DNS inside pods:

   kubectl -n net-test exec net-debug -- nslookup echo-a.net-test.svc.cluster.local

ClusterIP routing:

   kubectl -n net-test exec net-debug -- wget -qO- http://echo-a

Pod IP direct:

   ECHO_IP=$(kubectl -n net-test get pod -l app=echo-a -o jsonpath='{.items[0].status.podIP}')
   kubectl -n net-test exec net-debug -- wget -qO- http://$ECHO_IP:5678

Optional external:

   curl -sS http://NODE_IP:NODEPORT || true

Expected: All previously failing steps now pass.

Operations Checklist

Use this concise list during incidents and routine checks.

Capture inventory
kubectl version, cluster-info, nodes -o wide
kube-proxy mode; CNI pods and logs
Note DNS Service IP and CoreDNS status
Prepare safe test scope
Create net-test namespace
Deploy echo-a and net-debug pods
Confirm readiness
Run diagnostics in order
In-pod DNS: nslookup kubernetes and echo-a
Service HTTP via DNS and ClusterIP
Pod IP direct access
NetworkPolicy presence and effects
NodePort from external (if relevant)
Egress DNS and HTTP
MTU checks if symptoms fit
Identify failure domain
DNS vs proxy/routing vs policy vs firewall vs MTU
Apply targeted fix
Edit ConfigMap, policy, firewall, or CNI/kube-proxy as indicated
Verify and rollback if needed
Re-run core tests
If no improvement, revert the last change cleanly
Clean up
Delete net-test namespace when finished
Archive logs and commands for post-incident review

Conclusion

Kubernetes networking problems become tractable when you approach them methodically. Start by inventorying versions, modes, and CNI status. Use a safe, temporary namespace and a minimal test workload to probe DNS, Service routing, pod paths, NodePort, and egress. Compare outputs to expected results to pinpoint the failing layer. When you make changes, keep them scoped and reversible, verify fixes immediately, and roll back if results do not improve. With the commands, examples, and checklists in this guide, you can diagnose and recover from most cluster networking issues confidently and repeatably.