## Intro Kube-proxy is the workhorse that implements Kubernetes Services on every node. When it is misconfigured or unhealthy, pods may still run, but traffic stops flowing to the right place. This article provides a production-focused checklist for operating kube-proxy safely: from version and topology inventory, through configuration, verification, diagnostics, and recovery. Each section includes concrete commands, expected output, and failure signals, so you can move from an observed symptom to a verified fix. This guide is written for platform engineers, DevOps consultants, and technical founders who need to maintain kube-proxy in production without guesswork. It emphasizes operational safety: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document recovery paths. You will learn how to: - Identify the installed kube-proxy version and mode (iptables, IPVS, etc.) - Safely inspect and change configuration on live nodes - Verify that kube-proxy is programming data plane rules correctly - Diagnose common failure modes such as missing endpoints or stale rules - Use a runbook-style checklist to reduce mean time to recovery All examples assume a Kubernetes cluster running v1.28 or later, with kubectl access and permission to view node-level resources. Replace placeholders like and with your actual values. ## Version and Environment Inventory Before changing anything, you need a clear picture of what is running. Kube-proxy can be deployed as a DaemonSet (common) or as a static pod in some distributions. Its behavior differs significantly by version and mode. ### 1. Identify kube-proxy deployment topology First, check whether kube-proxy is managed by a DaemonSet: kubectl get daemonset -n kube-system | grep kube-proxy Expected output on a default cluster: kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 5m2s If you see no output, check for a static pod: ls /etc/kubernetes/manifests/ Look for a file named kube-proxy.yaml . On managed services (EKS, GKE, AKS), kube-proxy is often hidden from the user; consult your provider's documentation. ### 2. Determine kube-proxy version and image Run: kubectl get daemonset kube-proxy -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}' Expected output example: registry.k8s.io/kube-proxy:v1.28.2 Verify that the version is compatible with your control plane. Kubernetes allows a skew of two minor versions for kube-proxy. If your API server is v1.28, kube-proxy v1.26 is the oldest supported. Using an older version may cause flag changes or missing features. ### 3. Confirm kube-proxy running mode Kube-proxy uses one of three main modes: iptables , ipvs , or userspace (rare). The mode is usually set via the --proxy-mode flag. On a node, examine the kube-proxy logs: kubectl logs -n kube-system daemonset/kube-proxy | head -20 Look for a line like: I0101 12:00:00.000000 1 server.go:567] "Using iptables Proxier" If you see "Using ipvs Proxier" , mode is IPVS. If userspace , upgrade immediately because it is deprecated and slower. To inspect without logs, check the metrics endpoint if enabled: kubectl exec -n kube-system daemonset/kube-proxy -- curl -s localhost:10249/metrics | grep proxy_mode Expected output showing mode as a label: kubeproxy_sync_proxy_rules_duration_seconds_count{mode="iptables"} 42 ### 4. Read-only observation: check ConfigMap Kube-proxy configuration is often stored in a ConfigMap named kube-proxy in kube-system : kubectl get configmap kube-proxy -n kube-system -o yaml Look for fields like mode , clusterCIDR , iptables.syncPeriod , etc. Use kubectl describe for a summary: kubectl describe configmap kube-proxy -n kube-system Example output snippet: Name: kube-proxy Namespace: kube-system Labels: app=kube-proxy Annotations: kubeadm.kubernetes.io/component-config.hash: sha256:... Data ==== config.conf: ---- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration clientConnection: kubeconfig: /var/lib/kube-proxy/kubeconfig.conf clusterCIDR: 10.244.0.0/16 mode: iptables Capture the hash annotation because changes to the ConfigMap must be reflected there. ### 5. Blast radius and prerequisites Before modifying anything, understand that kube-proxy runs one pod per node. Changing its ConfigMap or DaemonSet specification triggers a rolling restart across all nodes, potentially disrupting service traffic for seconds. For a single node change, you can cordon and drain the node first, but for config changes, a wider impact is expected. In production, schedule changes during a low-traffic window. Prerequisites for safe inventory: cluster access with read-only RBAC is sufficient. You do not need write permissions to collect this information. ## Safe Configuration Path Configuration changes should follow a strict process: inspect, back up, modify one item, verify, and be ready to roll back. ### 1. Inspect current ConfigMap and DaemonSet Before editing, save a backup of both resources: kubectl get configmap kube-proxy -n kube-system -o yaml > kube-proxy-cm-backup-$(date +%Y%m%d).yaml kubectl get daemonset kube-proxy -n kube-system -o yaml > kube-proxy-ds-backup-$(date +%Y%m%d).yaml These files are your rollback artifacts. ### 2. Identify the change needed Common safe changes: - Adjust iptables.syncPeriod from 30s to 60s to reduce CPU on nodes with many Services (requires restart). - Enable metricsBindAddress if currently disabled (allows Prometheus scraping). - Change mode from iptables to ipvs for better performance at scale (requires node kernel modules and careful testing). - Update clusterCIDR if the pod network changed. Never change multiple settings at once in production. Choose one. ### 3. Modify ConfigMap via kubectl edit or patch Example: increase sync period. Current ConfigMap may contain: apiVersion: v1 data: config.conf: |- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration iptables: syncPeriod: 30s mode: iptables kind: ConfigMap metadata: name: kube-proxy namespace: kube-system Use kubectl patch for a targeted change: kubectl patch configmap kube-proxy -n kube-system --type merge -p '{"data":{"config.conf":"apiVersion: kubeproxy.config.k8s.io/v1alpha1\nkind: KubeProxyConfiguration\niptables:\n syncPeriod: 60s\nmode: iptables\n"}}' Alternatively, use kubectl edit configmap kube-proxy -n kube-system and modify the value. ### 4. Trigger rollout and observe If the ConfigMap is mounted into the kube-proxy pod as a file, kube-proxy may not automatically reload. You must restart the pods: kubectl rollout restart daemonset kube-proxy -n kube-system Monitor the rollout: kubectl rollout status daemonset kube-proxy -n kube-system Expected output: Waiting for daemon set "kube-proxy" rollout to finish: 0 out of 3 new pods have been updated... Successfully rolled out! If you see error: daemonset "kube-proxy" has timed out , investigate pod logs. ### 5. Verify the change took effect On a node, inspect the kube-proxy logs after restart: kubectl logs -n kube-system daemonset/kube-proxy --tail=30 Look for: I0101 12:30:00.000000 1 config.go:315] "Using iptables Proxier" I0101 12:30:00.000000 1 server.go:661] "Successfully set iptables sync period without needing a full resync" syncPeriod="60s" For metric-based verification: kubectl exec -n kube-system daemonset/kube-proxy -- curl -s localhost:10249/metrics | grep iptables_sync_period_seconds Expected: # HELP kubeproxy_sync_proxy_rules_duration_seconds... If the sync period did not change, double-check the ConfigMap YAML indentation and that the pod restarted with the updated volume. ### 6. Rollback path If the change causes issues, restore the backup ConfigMap and restart again: kubectl apply -f kube-proxy-cm-backup-YYYYMMDD.yaml kubectl rollout restart daemonset kube-proxy -n kube-system Because you backed up both ConfigMap and DaemonSet, you can also restore the DaemonSet if the pod template was altered. ## Verification and Diagnostics Healthy kube-proxy means the data plane (iptables rules, IPVS tables) matches the desired Services and Endpoints. This section provides a repeatable verification routine. ### 1. Check kube-proxy pod health and readiness All pods should be Running and Ready: kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide Expected output: NAME READY STATUS RESTARTS AGE IP NODE kube-proxy-abcde 1/1 Running 0 10m 10.0.0.10 node-1 kube-proxy-fghij 1/1 Running 0 10m 10.0.0.11 node-2 kube-proxy-klmno 1/1 Running 0 10m 10.0.0.12 node-3 If a pod is not ready, check its logs: kubectl logs -n kube-system kube-proxy-abcde --tail=50 Look for fatal errors like: E0101 12:00:00.000000 1 server.go:321] "Failed to run proxier" err="can't set ipset" ### 2. Verify iptables rules for a Service Choose a test Service (e.g., default/my-service ). On a node where kube-proxy runs, list the relevant iptables chain: sudo iptables-save | grep -A 10 'KUBE-SVC-' Expected output includes a chain for your Service with a Virtual IP (ClusterIP) and rules jumping to KUBE-SEP-* chains for each endpoint. Example for Service with ClusterIP 10.96.0.10 and two endpoints: -A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "default/my-service:http cluster IP" -m tcp --dport 80 -j KUBE-SVC-XXXXXXXXXXXXXXXX -A KUBE-SVC-XXXXXXXXXXXXXXXX -m comment --comment "default/my-service:http" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-YYYYYYYYYYYYYYYY -A KUBE-SVC-XXXXXXXXXXXXXXXX -m comment --comment "default/my-service:http" -j KUBE-SEP-ZZZZZZZZZZZZZZZZ If no KUBE-SERVICES rule exists for that ClusterIP, kube-proxy is not syncing correctly. Check endpoint slices: kubectl get endpointslice -l kubernetes.io/service-name=my-service -n default If EndpointSlices are empty, the Service has no ready pods, which is not a kube-proxy fault. ### 3. Check IPVS rules if in IPVS mode On the node, run: sudo ipvsadm -Ln Expected output includes a virtual service entry: TCP 10.96.0.10:80 rr -> 10.244.1.5:8080 Masq 1 0 0 -> 10.244.2.6:8080 Masq 1 0 0 If the virtual service is missing or has no real servers, kube-proxy failed to program IPVS. ### 4. Verify metrics endpoint If metricsBindAddress is set (e.g., 0.0.0.0:10249 ), query it from inside a pod or via node proxy: kubectl exec -n kube-system daemonset/kube-proxy -- curl -s http://localhost:10249/healthz Expected output: ok For metrics count of sync errors: kubectl exec -n kube-system daemonset/kube-proxy -- curl -s http://localhost:10249/metrics | grep sync_proxy_rules_last_timestamp_seconds Look for a recent timestamp; if the timestamp is old, sync is stuck. ### 5. Validate connectivity end-to-end From a pod in the cluster, test a Service DNS name and port: kubectl run test-pod --rm -it --image=busybox --restart=Never -- sh -c 'wget -qO- http://my-service.default.svc.cluster.local' Expected output if the Service returns a web page: HTML content. If connection hangs, check kube-proxy and endpoint readiness. ## Failure Modes and Recovery Kube-proxy failures manifest as connection timeouts, refused connections, or random load-balancing anomalies. Here are the common failure modes with recovery steps. ### 1. Kube-proxy pod crash-looping Symptoms : kubectl get pods -n kube-system shows CrashLoopBackOff for kube-proxy pods. Diagnosis : Check logs: kubectl logs -n kube-system kube-proxy-abcde --previous Common causes: - Invalid ConfigMap syntax leading to parse error. - Missing kubeconfig file due to secret removal. - Insufficient memory or CPU on node. - Incompatible flags if running unsupported version. Recovery : Revert ConfigMap to last known good backup, then restart: kubectl apply -f kube-proxy-cm-backup.yaml kubectl rollout restart daemonset kube-proxy -n kube-system If the issue is node resources, cordon the node and investigate memory pressure: kubectl cordon node-1 kubectl drain node-1 --ignore-daemonsets ### 2. Service ClusterIP unreachable but pods healthy Symptoms : Application logs show connection timeouts to Service IP, but curl to individual pod IPs works. Diagnosis : Check if kube-proxy is running on the node where the client pod runs: kubectl get pod -n kube-system -l k8s-app=kube-proxy -o wide If no kube-proxy on that node (e.g., after a node replacement and DaemonSet scheduling issue), that explains the failure. Recovery : Ensure the DaemonSet has correct tolerations and node selector, then uncordon or add the node: kubectl describe daemonset kube-proxy -n kube-system | grep -A5 Tolerations If missing a toleration for a tainted node, add it: kubectl patch daemonset kube-proxy -n kube-system -p '{"spec":{"template":{"spec":{"tolerations":[{"key":"node-role.kubernetes.io/master","operator":"Exists","effect":"NoSchedule"}]}}}}' ### 3. Stale iptables rules causing misrouting Symptoms : Traffic to a Service is sent to a pod that no longer exists, or to the wrong port, after scaling down. Diagnosis : Inspect iptables for the Service's chain: sudo iptables-save | grep KUBE-SVC-XXXXXXXXXXXXXXXX If you see KUBE-SEP-* rules pointing to terminated pod IPs, the sync did not occur. Recovery : Force a full sync by restarting kube-proxy: kubectl delete pod -n kube-system -l k8s-app=kube-proxy This recreates all iptables rules from scratch. Verify after restart. ### 4. IPVS mode not working due to missing kernel modules Symptoms : Kube-proxy pod logs show errors like "can't create ipvs" or "failed to load kernel module". Diagnosis : On the node, check kernel modules: sudo lsmod | grep ip_vs If output is empty, required modules are missing. Recovery : Load the modules (temporary): sudo modprobe ip_vs sudo modprobe ip_vs_rr sudo modprobe ip_vs_wrr sudo modprobe ip_vs_sh For persistence, consult your node OS documentation. Alternatively, switch back to iptables mode temporarily by editing the ConfigMap and restarting. ### 5. Metrics endpoint not accessible Symptoms : Prometheus cannot scrape kube-proxy metrics. Diagnosis : Check if metricsBindAddress is set: kubectl get configmap kube-proxy -n kube-system -o yaml | grep metricsBindAddress If missing, metrics are disabled by default in newer versions. Recovery : Add metricsBindAddress: 0.0.0.0:10249 under metricsBindAddress in the ConfigMap, then restart. Note: exposing metrics on all interfaces may be a security risk; restrict via network policy if possible. ## Operations Checklist Use this table as a pre-flight and periodic audit for kube-proxy in production. Replace placeholder names with your cluster values.

#	Check	Command	Expected Result	Failure Signal	Recovery Action
1	Kube-proxy version matches cluster	`kubectl get ds kube-proxy -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}'`	Image tag within supported skew (e.g., v1.28.x for API v1.28)	Version too old or too new	Upgrade or downgrade kube-proxy according to version skew policy
2	All kube-proxy pods running	`kubectl get pods -n kube-system -l k8s-app=kube-proxy`	All pods `1/1 Running`, Ready	Pods in CrashLoopBackOff or Pending	Inspect logs, restore ConfigMap, check node resources
3	Proxy mode is expected	Check pod logs or metrics for `"Using iptables Proxier"` or `"Using ipvs Proxier"`	Mode matches intended config	Mode mismatch (e.g., iptables when IPVS expected)	Update ConfigMap `mode` field and restart
4	ConfigMap hash matches pod template	`kubectl get configmap kube-proxy -n kube-system -o yaml \| grep kubeadm.kubernetes.io/component-config.hash`	Hash exists and matches DaemonSet template annotation	Hash missing or different after manual edit	Reapply ConfigMap via kubeadm or update annotation
5	Data plane rules exist for Services	On node: `sudo iptables-save \| grep KUBE-SERVICES` (or `ipvsadm -Ln` for IPVS)	Rules for ClusterIPs present	No KUBE-SERVICES chain or missing Service entries	Restart kube-proxy to force full sync
6	Metrics endpoint healthy	`curl -s http://localhost:10249/healthz` from node or pod	`ok`	Connection refused or timeout	Check metricsBindAddress, restart if needed
7	No continuous sync errors	Query metrics `kubeproxy_sync_proxy_rules_last_timestamp_seconds`	Timestamp updated within sync period	Timestamp stale (>2 sync periods)	Check for API server connectivity, node resource pressure
8	Service connectivity from pod	`kubectl run test --rm -it --image=busybox -- wget -qO- http://<service>.<ns>.svc`	Returns expected response	Timeout or 503	Verify endpoints, check kube-proxy, test individual pod IP
9	Node scheduling coverage	`kubectl get nodes -o name \| wc -l` equals `kubectl get pods -n kube-system -l k8s-app=kube-proxy --field-selector=status.phase=Running \| wc -l` (minus control-plane if DS tolerates)	Count match	Missing pods on some nodes	Check tolerations, taints, node selector, or resource issues
10	Backup exists before change	Look for `kube-proxy-cm-backup-.yaml` and `kube-proxy-ds-backup-``.yaml` in your repo or local	Backup files present and recent	No backup	Create backup immediately before any modification

Run this checklist monthly and after any network plugin upgrade, Kubernetes upgrade, or significant scaling event. ## Conclusion Operating kube-proxy in production requires disciplined observation, minimal intervention, and reliable verification. This checklist gives you a repeatable path from symptom to resolution: know your version and topology, inspect configuration safely, verify data plane programming, diagnose common failures, and recover with backups. The key takeaways: - Always record the current state before changing anything. - Make one configuration change at a time and verify its effect. - Use read-only commands (kubectl get, logs, iptables-save, ipvsadm) to observe. - Keep backups of ConfigMap and DaemonSet as rollback points. - Understand the blast radius: kube-proxy restarts affect all Services on a node. As a next step, choose one low-risk verification from the Operations Checklist (e.g., check ConfigMap hash or sync timestamp) and run it on your cluster. Document the output, compare to the expected result, and address any discrepancies. Over time, build a runbook specific to your environment, incorporating these patterns and your own recovery steps. A reliable kube-proxy operations workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.