Kube-Proxy Production Operations Checklist: Practical Examples for Reliable Clusters

Intro

Kube-proxy is the workhorse that implements Kubernetes Services on every node. When it is misconfigured or unhealthy, pods may still run, but traffic stops flowing to the right place. This article provides a production-focused checklist for operating kube-proxy safely: from version and topology inventory, through configuration, verification, diagnostics, and recovery. Each section includes concrete commands, expected output, and failure signals, so you can move from an observed symptom to a verified fix.

This guide is written for platform engineers, DevOps consultants, and technical founders who need to maintain kube-proxy in production without guesswork. It emphasizes operational safety: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document recovery paths.

You will learn how to:

• Identify the installed kube-proxy version and mode (iptables, IPVS, etc.)
• Safely inspect and change configuration on live nodes
• Verify that kube-proxy is programming data plane rules correctly
• Diagnose common failure modes such as missing endpoints or stale rules
• Use a runbook-style checklist to reduce mean time to recovery

All examples assume a Kubernetes cluster running v1.28 or later, with kubectl access and permission to view node-level resources. Replace placeholders like <node-name> and <service-name> with your actual values.

Version and Environment Inventory

Before changing anything, you need a clear picture of what is running. Kube-proxy can be deployed as a DaemonSet (common) or as a static pod in some distributions. Its behavior differs significantly by version and mode.

1. Identify kube-proxy deployment topology

First, check whether kube-proxy is managed by a DaemonSet:

kubectl get daemonset -n kube-system | grep kube-proxy

Expected output on a default cluster:

kube-proxy           1         1         1       1            1           kubernetes.io/os=linux   5m2s

If you see no output, check for a static pod:

ls /etc/kubernetes/manifests/

Look for a file named kube-proxy.yaml. On managed services (EKS, GKE, AKS), kube-proxy is often hidden from the user; consult your provider's documentation.

2. Determine kube-proxy version and image

Run:

kubectl get daemonset kube-proxy -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}'

Expected output example:

registry.k8s.io/kube-proxy:v1.28.2

Verify that the version is compatible with your control plane. Kubernetes allows a skew of two minor versions for kube-proxy. If your API server is v1.28, kube-proxy v1.26 is the oldest supported. Using an older version may cause flag changes or missing features.

3. Confirm kube-proxy running mode

Kube-proxy uses one of three main modes: iptables, ipvs, or userspace (rare). The mode is usually set via the --proxy-mode flag. On a node, examine the kube-proxy logs:

kubectl logs -n kube-system daemonset/kube-proxy | head -20

Look for a line like:

I0101 12:00:00.000000       1 server.go:567] "Using iptables Proxier"

If you see "Using ipvs Proxier", mode is IPVS. If userspace, upgrade immediately because it is deprecated and slower.

To inspect without logs, check the metrics endpoint if enabled:

kubectl exec -n kube-system daemonset/kube-proxy -- curl -s localhost:10249/metrics | grep proxy_mode

Expected output showing mode as a label:

kubeproxy_sync_proxy_rules_duration_seconds_count{mode="iptables"} 42

4. Read-only observation: check ConfigMap

Kube-proxy configuration is often stored in a ConfigMap named kube-proxy in kube-system:

kubectl get configmap kube-proxy -n kube-system -o yaml

Look for fields like mode, clusterCIDR, iptables.syncPeriod, etc. Use kubectl describe for a summary:

kubectl describe configmap kube-proxy -n kube-system

Example output snippet:

Name:         kube-proxy
Namespace:    kube-system
Labels:       app=kube-proxy
Annotations:  kubeadm.kubernetes.io/component-config.hash: sha256:...

Data
====
config.conf:
----
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clientConnection:
  kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
clusterCIDR: 10.244.0.0/16
mode: iptables

Capture the hash annotation because changes to the ConfigMap must be reflected there.

5. Blast radius and prerequisites

Before modifying anything, understand that kube-proxy runs one pod per node. Changing its ConfigMap or DaemonSet specification triggers a rolling restart across all nodes, potentially disrupting service traffic for seconds. For a single node change, you can cordon and drain the node first, but for config changes, a wider impact is expected. In production, schedule changes during a low-traffic window.

Prerequisites for safe inventory: cluster access with read-only RBAC is sufficient. You do not need write permissions to collect this information.

Safe Configuration Path

Configuration changes should follow a strict process: inspect, back up, modify one item, verify, and be ready to roll back.

1. Inspect current ConfigMap and DaemonSet

Before editing, save a backup of both resources:

kubectl get configmap kube-proxy -n kube-system -o yaml > kube-proxy-cm-backup-$(date +%Y%m%d).yaml
kubectl get daemonset kube-proxy -n kube-system -o yaml > kube-proxy-ds-backup-$(date +%Y%m%d).yaml

These files are your rollback artifacts.

2. Identify the change needed

Common safe changes:

• Adjust iptables.syncPeriod from 30s to 60s to reduce CPU on nodes with many Services (requires restart).
• Enable metricsBindAddress if currently disabled (allows Prometheus scraping).
• Change mode from iptables to ipvs for better performance at scale (requires node kernel modules and careful testing).
• Update clusterCIDR if the pod network changed.

Never change multiple settings at once in production. Choose one.

3. Modify ConfigMap via kubectl edit or patch

Example: increase sync period. Current ConfigMap may contain:

apiVersion: v1
data:
  config.conf: |-
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    kind: KubeProxyConfiguration
    iptables:
      syncPeriod: 30s
    mode: iptables
kind: ConfigMap
metadata:
  name: kube-proxy
  namespace: kube-system

Use kubectl patch for a targeted change:

kubectl patch configmap kube-proxy -n kube-system --type merge -p '{"data":{"config.conf":"apiVersion: kubeproxy.config.k8s.io/v1alpha1\nkind: KubeProxyConfiguration\niptables:\n  syncPeriod: 60s\nmode: iptables\n"}}'

Alternatively, use kubectl edit configmap kube-proxy -n kube-system and modify the value.

4. Trigger rollout and observe

If the ConfigMap is mounted into the kube-proxy pod as a file, kube-proxy may not automatically reload. You must restart the pods:

kubectl rollout restart daemonset kube-proxy -n kube-system

Monitor the rollout:

kubectl rollout status daemonset kube-proxy -n kube-system

Expected output:

Waiting for daemon set "kube-proxy" rollout to finish: 0 out of 3 new pods have been updated...
Successfully rolled out!

If you see error: daemonset "kube-proxy" has timed out, investigate pod logs.

5. Verify the change took effect

On a node, inspect the kube-proxy logs after restart:

kubectl logs -n kube-system daemonset/kube-proxy --tail=30

Look for:

I0101 12:30:00.000000       1 config.go:315] "Using iptables Proxier"
I0101 12:30:00.000000       1 server.go:661] "Successfully set iptables sync period without needing a full resync" syncPeriod="60s"

For metric-based verification:

kubectl exec -n kube-system daemonset/kube-proxy -- curl -s localhost:10249/metrics | grep iptables_sync_period_seconds

Expected:

# HELP kubeproxy_sync_proxy_rules_duration_seconds...

If the sync period did not change, double-check the ConfigMap YAML indentation and that the pod restarted with the updated volume.

6. Rollback path

If the change causes issues, restore the backup ConfigMap and restart again:

kubectl apply -f kube-proxy-cm-backup-YYYYMMDD.yaml
kubectl rollout restart daemonset kube-proxy -n kube-system

Because you backed up both ConfigMap and DaemonSet, you can also restore the DaemonSet if the pod template was altered.

Verification and Diagnostics

Healthy kube-proxy means the data plane (iptables rules, IPVS tables) matches the desired Services and Endpoints. This section provides a repeatable verification routine.

1. Check kube-proxy pod health and readiness

All pods should be Running and Ready:

kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide

Expected output:

NAME               READY   STATUS    RESTARTS   AGE   IP           NODE
kube-proxy-abcde   1/1     Running   0          10m   10.0.0.10    node-1
kube-proxy-fghij   1/1     Running   0          10m   10.0.0.11    node-2
kube-proxy-klmno   1/1     Running   0          10m   10.0.0.12    node-3

If a pod is not ready, check its logs:

kubectl logs -n kube-system kube-proxy-abcde --tail=50

Look for fatal errors like:

E0101 12:00:00.000000       1 server.go:321] "Failed to run proxier" err="can't set ipset"

2. Verify iptables rules for a Service

Choose a test Service (e.g., default/my-service). On a node where kube-proxy runs, list the relevant iptables chain:

sudo iptables-save | grep -A 10 'KUBE-SVC-'

Expected output includes a chain for your Service with a Virtual IP (ClusterIP) and rules jumping to KUBE-SEP-* chains for each endpoint.

Example for Service with ClusterIP 10.96.0.10 and two endpoints:

-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment "default/my-service:http cluster IP" -m tcp --dport 80 -j KUBE-SVC-XXXXXXXXXXXXXXXX
-A KUBE-SVC-XXXXXXXXXXXXXXXX -m comment --comment "default/my-service:http" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-YYYYYYYYYYYYYYYY
-A KUBE-SVC-XXXXXXXXXXXXXXXX -m comment --comment "default/my-service:http" -j KUBE-SEP-ZZZZZZZZZZZZZZZZ

If no KUBE-SERVICES rule exists for that ClusterIP, kube-proxy is not syncing correctly. Check endpoint slices:

kubectl get endpointslice -l kubernetes.io/service-name=my-service -n default

If EndpointSlices are empty, the Service has no ready pods, which is not a kube-proxy fault.

3. Check IPVS rules if in IPVS mode

On the node, run:

sudo ipvsadm -Ln

Expected output includes a virtual service entry:

TCP  10.96.0.10:80 rr
  -> 10.244.1.5:8080              Masq    1      0          0
  -> 10.244.2.6:8080              Masq    1      0          0

If the virtual service is missing or has no real servers, kube-proxy failed to program IPVS.

4. Verify metrics endpoint

If metricsBindAddress is set (e.g., 0.0.0.0:10249), query it from inside a pod or via node proxy:

kubectl exec -n kube-system daemonset/kube-proxy -- curl -s http://localhost:10249/healthz

Expected output: ok

For metrics count of sync errors:

kubectl exec -n kube-system daemonset/kube-proxy -- curl -s http://localhost:10249/metrics | grep sync_proxy_rules_last_timestamp_seconds

Look for a recent timestamp; if the timestamp is old, sync is stuck.

5. Validate connectivity end-to-end

From a pod in the cluster, test a Service DNS name and port:

kubectl run test-pod --rm -it --image=busybox --restart=Never -- sh -c 'wget -qO- http://my-service.default.svc.cluster.local'

Expected output if the Service returns a web page: HTML content. If connection hangs, check kube-proxy and endpoint readiness.

Failure Modes and Recovery

Kube-proxy failures manifest as connection timeouts, refused connections, or random load-balancing anomalies. Here are the common failure modes with recovery steps.

1. Kube-proxy pod crash-looping

Symptoms: kubectl get pods -n kube-system shows CrashLoopBackOff for kube-proxy pods.

Diagnosis: Check logs:

kubectl logs -n kube-system kube-proxy-abcde --previous

Common causes:

• Invalid ConfigMap syntax leading to parse error.
• Missing kubeconfig file due to secret removal.
• Insufficient memory or CPU on node.
• Incompatible flags if running unsupported version.

Recovery: Revert ConfigMap to last known good backup, then restart:

kubectl apply -f kube-proxy-cm-backup.yaml
kubectl rollout restart daemonset kube-proxy -n kube-system

If the issue is node resources, cordon the node and investigate memory pressure:

kubectl cordon node-1
kubectl drain node-1 --ignore-daemonsets

2. Service ClusterIP unreachable but pods healthy

Symptoms: Application logs show connection timeouts to Service IP, but curl to individual pod IPs works.

Diagnosis: Check if kube-proxy is running on the node where the client pod runs:

kubectl get pod -n kube-system -l k8s-app=kube-proxy -o wide

If no kube-proxy on that node (e.g., after a node replacement and DaemonSet scheduling issue), that explains the failure.

Recovery: Ensure the DaemonSet has correct tolerations and node selector, then uncordon or add the node:

kubectl describe daemonset kube-proxy -n kube-system | grep -A5 Tolerations

If missing a toleration for a tainted node, add it:

kubectl patch daemonset kube-proxy -n kube-system -p '{"spec":{"template":{"spec":{"tolerations":[{"key":"node-role.kubernetes.io/master","operator":"Exists","effect":"NoSchedule"}]}}}}'

3. Stale iptables rules causing misrouting

Symptoms: Traffic to a Service is sent to a pod that no longer exists, or to the wrong port, after scaling down.

Diagnosis: Inspect iptables for the Service's chain:

sudo iptables-save | grep KUBE-SVC-XXXXXXXXXXXXXXXX

If you see KUBE-SEP-* rules pointing to terminated pod IPs, the sync did not occur.

Recovery: Force a full sync by restarting kube-proxy:

kubectl delete pod -n kube-system -l k8s-app=kube-proxy

This recreates all iptables rules from scratch. Verify after restart.

4. IPVS mode not working due to missing kernel modules

Symptoms: Kube-proxy pod logs show errors like "can't create ipvs" or "failed to load kernel module".

Diagnosis: On the node, check kernel modules:

sudo lsmod | grep ip_vs

If output is empty, required modules are missing.

Recovery: Load the modules (temporary):

sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh

For persistence, consult your node OS documentation. Alternatively, switch back to iptables mode temporarily by editing the ConfigMap and restarting.

5. Metrics endpoint not accessible

Symptoms: Prometheus cannot scrape kube-proxy metrics.

Diagnosis: Check if metricsBindAddress is set:

kubectl get configmap kube-proxy -n kube-system -o yaml | grep metricsBindAddress

If missing, metrics are disabled by default in newer versions.

Recovery: Add metricsBindAddress: 0.0.0.0:10249 under metricsBindAddress in the ConfigMap, then restart. Note: exposing metrics on all interfaces may be a security risk; restrict via network policy if possible.

Operations Checklist

Use this table as a pre-flight and periodic audit for kube-proxy in production. Replace placeholder names with your cluster values.

Run this checklist monthly and after any network plugin upgrade, Kubernetes upgrade, or significant scaling event.

Conclusion

Operating kube-proxy in production requires disciplined observation, minimal intervention, and reliable verification. This checklist gives you a repeatable path from symptom to resolution: know your version and topology, inspect configuration safely, verify data plane programming, diagnose common failures, and recover with backups.

The key takeaways:

• Always record the current state before changing anything.
• Make one configuration change at a time and verify its effect.
• Use read-only commands (kubectl get, logs, iptables-save, ipvsadm) to observe.
• Keep backups of ConfigMap and DaemonSet as rollback points.
• Understand the blast radius: kube-proxy restarts affect all Services on a node.

As a next step, choose one low-risk verification from the Operations Checklist (e.g., check ConfigMap hash or sync timestamp) and run it on your cluster. Document the output, compare to the expected result, and address any discrepancies. Over time, build a runbook specific to your environment, incorporating these patterns and your own recovery steps.

A reliable kube-proxy operations workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.