Kubernetes Architecture Networking Troubleshooting: Practical Examples and Commands

Intro

Kubernetes networking is often the most challenging aspect of cluster operations. When pods cannot reach each other, services fail to load-balance, or DNS lookups time out, developers and operators need a systematic approach to diagnose and resolve issues quickly. This guide provides practical, command-driven troubleshooting techniques for Kubernetes networking, grounded in real-world scenarios. We will cover DNS resolution, service and pod connectivity, kube-proxy and CNI behavior, and safe diagnostic methods. By following the examples, you can reduce downtime and build confidence in your Kubernetes networking stack.

Networking issues can manifest in subtle ways: a service that intermittently returns 503 errors, pods stuck in ContainerCreating state, or DNS queries that sporadically time out. A structured methodology prevents flailing and reduces mean time to resolution. This guide emphasizes hands-on commands, expected outputs, and concrete recovery steps.

Before diving into specific symptoms, establish a baseline understanding of your cluster's networking components. This includes the Container Network Interface (CNI) plugin, kube-proxy mode, DNS add-on, and the underlying node network topology. Each component has its own failure modes and diagnostic surfaces.

Version and Environment Inventory

Before troubleshooting, establish a clear inventory of your Kubernetes environment. This includes cluster version, CNI plugin, kube-proxy mode, and node operating system. Knowing these details prevents misdiagnosis and helps when consulting documentation or community forums.

Start by checking the Kubernetes server version:

kubectl version --short

Expected output example:

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2

Identify the CNI plugin in use. Common CNIs include Calico, Flannel, Cilium, and Weave. You can often find this by listing pods in the kube-system namespace:

kubectl get pods -n kube-system | grep -E 'calico|flannel|cilium|weave'

Example output:

calico-node-abcde                         1/1     Running   0          5d
calico-kube-controllers-12345             1/1     Running   0          5d

If no CNI pods are visible, check for CNI configuration files on the node:

ls /etc/cni/net.d/

Common files include 10-calico.conflist, 10-flannel.conflist, or 05-cilium.conf.

Check the kube-proxy mode:

kubectl get pods -n kube-system -l k8s-app=kube-proxy -o yaml | grep -A1 'mode:'

Typical output includes mode: iptables or mode: ipvs. You can also check the kube-proxy ConfigMap:

kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode

If using kubeadm, the kube-proxy ConfigMap often contains:

mode: "iptables"

or

mode: "ipvs"

Gather node information:

kubectl get nodes -o wide

This shows node IPs, OS, and container runtime. Example:

NAME     STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node01   Ready    control-plane   10d   v1.28.2   172.31.0.10    <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.2
node02   Ready    <none>          10d   v1.28.2   172.31.0.11    <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.2

Check the pod network CIDR to ensure it does not overlap with node or service CIDRs:

kubectl cluster-info dump | grep -m 1 cluster-cidr

or inspect the kube-controller-manager manifest on a control plane node:

sudo grep cluster-cidr /etc/kubernetes/manifests/kube-controller-manager.yaml

Example output:

- --cluster-cidr=10.244.0.0/16

Prerequisites for safe troubleshooting:

• Access to cluster with kubectl configured.
• Permission to view and execute commands in relevant namespaces.
• Understanding of cluster topology (e.g., single vs multi-node, cloud provider).
• A change control process for any modifications.
• Ability to SSH into nodes if necessary for node-level diagnostics.
• Network tools like ping, curl, dig, tcpdump, and iptables available or installable.

A scoped pilot, as recommended for initial deployment, is also useful for troubleshooting: test changes on one service or namespace before rolling out cluster-wide.

Safe Configuration Path

When making changes to fix networking issues, always start with narrow, reversible modifications. For example, if you suspect a kube-proxy configuration problem, back up the current ConfigMap before editing:

kubectl get configmap kube-proxy -n kube-system -o yaml > kube-proxy-config-backup.yaml

Then edit with kubectl edit configmap kube-proxy -n kube-system and restart kube-proxy pods:

kubectl rollout restart daemonset kube-proxy -n kube-system

For DNS changes, modify CoreDNS configuration only after backing up:

kubectl get configmap coredns -n kube-system -o yaml > coredns-backup.yaml

Use kubectl apply -f with a modified file rather than imperative edits to track changes. After applying, restart CoreDNS:

kubectl rollout restart deployment coredns -n kube-system

When adjusting network policies, start by allowing all traffic in a test namespace to confirm the policy engine works:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-all
  namespace: test-ns
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - {}
  egress:
    - {}

Apply this policy with:

kubectl apply -f allow-all.yaml -n test-ns

Then test connectivity from a pod in that namespace. If it works, gradually tighten the policy. If it fails, the issue may be in the CNI's policy engine.

Always use kubectl diff -f file.yaml before applying to see what will change:

kubectl diff -f networkpolicy.yaml

Example output showing proposed changes:

+  ingress:
+  - from:
+    - podSelector:
+        matchLabels:
+          role: frontend

For CNI changes, consult vendor documentation. Many CNIs allow per-node or per-pool configuration. Avoid cluster-wide changes without canary testing. For example, Calico allows setting per-node configurations using node-specific overrides. Create a node-specific FelixConfiguration:

apiVersion: crd.projectcalico.org/v1
kind: FelixConfiguration
metadata:
  name: node.node02
spec:
  ipipEnabled: false

Apply to only that node and monitor before rolling out to others.

Verification and Diagnostics

Systematic verification is key. Start with DNS because many connectivity issues stem from name resolution failures.

DNS Resolution Checks

Check DNS resolution from within a pod:

kubectl run -it --rm debug --image=busybox --restart=Never -- nslookup kubernetes.default

Expected output:

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      kubernetes.default
Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local

If DNS fails, check CoreDNS pods:

kubectl get pods -n kube-system -l k8s-app=kube-dns

Expected output:

NAME                      READY   STATUS    RESTARTS   AGE
coredns-787d4945fb-2v8hx   1/1     Running   0          5d
coredns-787d4945fb-6pwqn   1/1     Running   0          5d

Check CoreDNS logs:

kubectl logs -n kube-system -l k8s-app=kube-dns

Look for errors like SERVFAIL or NXDOMAIN for internal names. If pods can't reach the CoreDNS service IP, test connectivity to the service ClusterIP:

kubectl get svc kube-dns -n kube-system

Example:

NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   10d

From a pod, run:

kubectl exec -it <pod-name> -- nc -zv 10.96.0.10 53

If connection is refused, kube-proxy or CoreDNS might be down.

Pod-to-Pod Connectivity

Test pod-to-pod connectivity: get pod IPs and run ping or curl.

kubectl get pods -o wide

Example:

NAME                     READY   STATUS    IP           NODE
web-5dcb957ccc-abcde     1/1     Running   10.244.1.5   node02
api-6f49d5c5b5-xyzab     1/1     Running   10.244.2.3   node01

From one pod, ping another pod's IP:

kubectl exec web-5dcb957ccc-abcde -- ping -c 4 10.244.2.3

Expected output if successful:

PING 10.244.2.3 (10.244.2.3): 56 data bytes
64 bytes from 10.244.2.3: seq=0 ttl=62 time=0.823 ms
64 bytes from 10.244.2.3: seq=1 ttl=62 time=0.745 ms
64 bytes from 10.244.2.3: seq=2 ttl=62 time=0.711 ms
64 bytes from 10.244.2.3: seq=3 ttl=62 time=0.698 ms

--- 10.244.2.3 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss

If ping fails, test cross-node communication using a non-ICMP protocol, as some clusters disable ICMP. Use nc or curl:

kubectl exec web-5dcb957ccc-abcde -- nc -zv 10.244.2.3 80

Expected: Connection to 10.244.2.3 80 port [tcp/http] succeeded!

Service Connectivity

Test service connectivity:

kubectl get svc

Example:

NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
my-service   ClusterIP   10.96.123.45    <none>        80/TCP     2h

From a pod, curl the service:

kubectl exec <pod-name> -- curl http://my-service

Or if the service name resolves:

kubectl exec <pod-name> -- curl http://my-service.default.svc.cluster.local

Expected: HTTP 200 and response body.

Check service endpoints:

kubectl get endpoints my-service

Example:

NAME         ENDPOINTS           AGE
my-service   10.244.1.5:8080     2h

If endpoints are empty, the service selector does not match any pods. Check the selector:

kubectl describe svc my-service

Compare the selector with pod labels:

kubectl get pods -l app=my-app --show-labels

kube-proxy Rules Verification

Check kube-proxy rules (if iptables mode). On a node, run:

sudo iptables-save | grep my-service

Expected output includes DNAT rules:

-A KUBE-SERVICES -d 10.96.123.45/32 -p tcp -m comment --comment "default/my-service: cluster IP" -m tcp --dport 80 -j KUBE-SVC-ABCDEF123456

Then inspect the KUBE-SVC chain:

sudo iptables-save | grep KUBE-SVC-ABCDEF123456

Expected:

-A KUBE-SVC-ABCDEF123456 -m comment --comment "default/my-service:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-XYZ
-A KUBE-SVC-ABCDEF123456 -m comment --comment "default/my-service:" -j KUBE-SEP-UVW

Each KUBE-SEP chain corresponds to an endpoint.

If kube-proxy is in IPVS mode, use ipvsadm:

sudo ipvsadm -L -n

Expected output shows virtual services and real servers:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.123.45:80 rr
  -> 10.244.1.5:8080              Masq    1      0          0

Use kubectl describe to examine events and endpoints for services and pods:

kubectl describe pod <pod-name>

Look for events related to network setup errors.

Failure Modes and Recovery

Common networking failure modes include DNS misconfiguration, kube-proxy not running, CNI plugin errors, and network policy blocking traffic.

DNS Failures

If CoreDNS pods are crashing, check logs:

kubectl logs -n kube-system -l k8s-app=kube-dns --previous

Common causes:

• Misconfigured Corefile (e.g., wrong upstream server, forward plugin errors).
• Resource limits (CPU/memory) causing OOM kills.
• Node DNS issues affecting CoreDNS.

Recovery steps:

1. Restore CoreDNS ConfigMap from backup:

kubectl apply -f coredns-backup.yaml

1. Check and adjust resource limits:

kubectl get deployment coredns -n kube-system -o yaml

Look for resources under container spec. Increase if needed.

1. Restart CoreDNS:

kubectl rollout restart deployment coredns -n kube-system

Example: If you see SERVFAIL for external domains, check the forward directive in Corefile. A misconfigured upstream may point to a non-existent resolver. Correct it and restart.

kube-proxy Failure

If kube-proxy pods are not running, services may not get iptables rules. Check:

kubectl get pods -n kube-system -l k8s-app=kube-proxy

If pods are Pending or CrashLoopBackOff, view logs:

kubectl logs -n kube-system -l k8s-app=kube-proxy

Common errors include:

• Failed to list *v1.Service: Unauthorized due to RBAC misconfiguration.
• Failed to start IPVS manager if IPVS modules are missing.

Recovery:

• If RBAC, verify ClusterRole and ClusterRoleBinding for kube-proxy.
• If IPVS, install required kernel modules on nodes and set mode to ipvs.
• If config issue, restore kube-proxy ConfigMap from backup and restart.

Delete pods to force recreation:

kubectl delete pod -n kube-system -l k8s-app=kube-proxy

Kubernetes will recreate them via DaemonSet.

CNI Errors

Pods stuck in ContainerCreating with CNI errors. Check pod description:

kubectl describe pod <pod-name>

Look for events like:

Warning  FailedCreatePodSandBox  2m   kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "...": plugin type="calico" failed (add): unable to allocate IP address: no available IPs in pool

This indicates IP exhaustion. Check IP pools for Calico:

calicoctl get ippool -o wide

If exhausted, expand the pool or clean up unused IPs.

Check node's CNI logs, often in /var/log/calico or via kubectl logs for CNI pod:

kubectl logs -n kube-system calico-node-abcde

Ensure CNI configuration file is correct and binaries exist:

ls /opt/cni/bin
cat /etc/cni/net.d/10-calico.conflist

If binaries missing, reinstall CNI or copy from a working node.

Network Policy Overly Restrictive

If a new policy breaks existing traffic, delete the policy to restore connectivity:

kubectl delete networkpolicy <policy-name> -n <namespace>

Example:

kubectl delete networkpolicy default-deny -n production

Then verify traffic flows. If policy was intended, modify it to allow necessary traffic.

Rollback Strategies

• Keep backups of all configuration (ConfigMaps, YAML files).
• Use version control for manifests.
• Test changes in a separate namespace or cluster.
• Have a canary deployment for critical updates.
• Use kubectl rollout undo for Deployments or DaemonSets:

kubectl rollout undo daemonset kube-proxy -n kube-system

• For iptables changes, revert by restoring previous rules and restarting kube-proxy.

Example: If an iptables rule change broke node networking, revert by running iptables-restore with a backup saved before the change, then restart kube-proxy.

Operations Checklist

Use this checklist for regular network health checks and troubleshooting:

1. Verify kube-proxy pods are running: kubectl get pods -n kube-system -l k8s-app=kube-proxy
2. Verify CoreDNS pods: kubectl get pods -n kube-system -l k8s-app=kube-dns
3. Test DNS resolution periodically from a test pod: kubectl run -it --rm debug --image=busybox -- nslookup kubernetes.default
4. Check node network interfaces and routes: ip route and ip addr on nodes.
5. Monitor CNI plugin health and logs: kubectl logs -n kube-system -l k8s-app=calico-node --tail=50
6. Review network policies for unintended restrictions: kubectl get networkpolicies --all-namespaces
7. Validate service endpoints match pods: kubectl get endpoints --all-namespaces
8. Perform connectivity tests between pods on different nodes: Use ping or nc as shown earlier.
9. Keep configuration backups and document changes.
10. Use kubectl diff before applying network changes.

Recommended frequency and tools:

For each item, document the expected result and set alerts for deviations using monitoring tools like Prometheus.

Conclusion

Troubleshooting Kubernetes networking requires a methodical approach: gather environment details, make safe configuration changes, verify with concrete commands, understand failure modes, and follow an operational checklist. The examples in this guide provide a foundation for diagnosing and resolving common issues. Start with a pilot scope, keep backups, and always verify before and after changes. With these practices, you can maintain a reliable and performant Kubernetes network.

Remember that networking is a layered system. Issues can arise at the DNS layer, the service abstraction layer, the kube-proxy implementation, or the CNI dataplane. A systematic elimination approach, starting with DNS and then moving to pod and service connectivity, often reveals the root cause quickly. Always correlate logs and events from multiple components, and when in doubt, isolate the problem by testing from different namespaces or nodes. By building a robust diagnostic toolkit and following the safe change practices outlined here, you can minimize downtime and keep your applications reachable.