## Intro Kubernetes networking relies on kube-proxy to implement Service abstractions, translating virtual IPs and ports into routable endpoints. It runs on every node, watching the control plane and programming the data plane—typically iptables, IPVS, or userspace. When kube-proxy misbehaves, the whole cluster feels it: services become unreachable, connection setup stalls, and latency spikes. Performance tuning is not a one-time tweak; it is an iterative process of observing, hypothesizing, changing, and verifying. This guide targets cluster operators, DevOps engineers, and site reliability engineers who manage production Kubernetes. It connects practical tuning actions—adjusting sync intervals, switching proxy modes, right-sizing resources—to measurable outcomes. You will learn to gather the right metrics, apply configuration changes safely, and avoid common pitfalls that turn a small change into an outage. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before touching a single flag, know your kube-proxy version and how it is deployed. Versions differ in default modes, available metrics, and known issues. Run: ```bash kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide ``` Expected output shows one pod per node. Note the image tag, e.g., `v1.28.2`. Then inspect one pod: ```bash kubectl describe pod -n kube-system kube-proxy- | grep -A2 "Image:" ``` Prerequisites: - Cluster access with `kubectl` and permissions to read pods, configmaps, and daemonsets in `kube-system`. - Metrics endpoint enabled (usually on port 10249). Check configuration: ```bash kubectl get configmap kube-proxy -n kube-system -o yaml ``` Look for `metricsBindAddress: 0.0.0.0:10249` and `mode: iptables` or `ipvs`. If metrics are disabled, enable them in the configmap and restart kube-proxy. Blast radius: reading is safe. Changes to configmap affect all nodes. Always back up: ```bash kubectl get configmap kube-proxy -n kube-system -o yaml > kube-proxy-config-backup-$(date +%Y%m%d).yaml ``` Verification: after any change, confirm pods are running and config is applied: ```bash kubectl rollout status daemonset kube-proxy -n kube-system ``` Wait for success. If `kubectl rollout status` fails, check pod logs: ```bash kubectl logs -n kube-system kube-proxy- --tail=50 ``` ## Safe Configuration Path Kube-proxy is configured via a ConfigMap in `kube-system`. Editing it directly is risky; instead, follow a controlled process. ### Baseline Metrics Before changing anything, capture performance baselines. Use `curl` against the metrics endpoint on one node. If you are on the node: ```bash curl -s http://127.0.0.1:10249/metrics | grep -E 'kubeproxy_sync_proxy_rules_duration_seconds|kubeproxy_network_programming_duration_seconds' ``` If not on node, use `kubectl port-forward`: ```bash kubectl port-forward -n kube-system pod/kube-proxy- 10249:10249 ``` Then in another terminal: `curl -s http://127.0.0.1:10249/metrics`. Key metrics: - `kubeproxy_sync_proxy_rules_duration_seconds`: histogram of time to sync rules. High percentiles (p99 > 1s) indicate iptables churn. - `kubeproxy_network_programming_duration_seconds`: time to program network rules. High values mean slow updates. Record these values. If your Prometheus scrapes kube-proxy, use queries like: ```promql histogram_quantile(0.99, sum(rate(kubeproxy_sync_proxy_rules_duration_seconds_bucket[5m])) by (le)) ``` ### Minimal Change Example Suppose p99 sync duration is 2.3s. A likely culprit is the default `iptables` sync period of 30 seconds being too short? Actually, the sync period (`iptables.syncPeriod`) defaults to 30s. If your cluster has many services and endpoints, iptables can become slow, and frequent syncs worsen the load. Increasing syncPeriod can reduce CPU usage but delays updates. A safer first step is to switch to IPVS mode if supported. To change mode: 1. Edit ConfigMap: ```bash kubectl edit configmap kube-proxy -n kube-system ``` 2. Change `mode: "ipvs"` (or set if absent). Save and exit. 3. Restart kube-proxy pods: ```bash kubectl rollout restart daemonset kube-proxy -n kube-system ``` 4. Verify mode: ```bash kubectl logs -n kube-system kube-proxy- | grep "Using ipvs Proxier" ``` Recovery: if services break, revert the ConfigMap change and restart again. Prerequisites for IPVS: node kernel modules (`ip_vs`, `ip_vs_rr`, etc.) must be loaded. Check: ```bash lsmod | grep ip_vs ``` If missing, kube-proxy falls back to iptables (or fails). Enable modules via node configuration before switching. ## Verification and Diagnostics Tuning is only as good as your verification. After changes, compare metrics and test actual service connectivity. ### Service Connectivity Test From a pod inside the cluster, test a service: ```bash kubectl run test-pod --image=busybox --restart=Never --rm -it -- wget -qO- --timeout=5 http://..svc.cluster.local:8080 ``` Replace with your service details. Success shows `200 OK` or expected output. If timeout, check endpoints: ```bash kubectl get endpoints -n ``` Ensure at least one IP. If none, kube-proxy cannot route. ### Latency Measurement Use `curl` with timing from a client pod: ```bash kubectl exec -it -- curl -w "time_namelookup: %{time_namelookup}\ntime_connect: %{time_connect}\ntime_starttransfer: %{time_starttransfer}\ntime_total: %{time_total}\n" -o /dev/null -s http://..svc.cluster.local:8080 ``` Record `time_connect` before and after tuning. Example output: ``` time_namelookup: 0.004 time_connect: 0.012 time_starttransfer: 0.030 time_total: 0.035 ``` If `time_connect` is high (>100ms), investigate kube-proxy rule programming or node network latency. ### Diagnostic Commands - Check kube-proxy logs: ```bash kubectl logs -n kube-system kube-proxy- --tail=100 ``` - Dump current iptables rules (on node): ```bash sudo iptables-save | grep -c "KUBE-SERVICES" ``` Large rule count (thousands) can slow sync. With IPVS, check virtual server table: ```bash sudo ipvsadm -L -n | head ``` ## Failure Modes and Recovery Kube-proxy can fail in subtle ways. Recognize these common failure modes and know how to recover. ### 1. ConfigMap Misconfiguration **Symptom:** After editing ConfigMap, kube-proxy pods crash loop or services unreachable. **Why:** Invalid YAML or unsupported options. **Recovery:** Restore from backup: ```bash kubectl apply -f kube-proxy-config-backup-.yaml kubectl rollout restart daemonset kube-proxy -n kube-system ``` Then check pod status and logs. ### 2. IPVS Kernel Modules Missing **Symptom:** kube-proxy logs show `Failed to load kernel modules` or `IPVS required kernel modules not found`. Pods may still run with iptables fallback, but performance suffers. **Why:** Node image lacks IPVS modules. **Recovery:** Install kernel modules on node (e.g., via package manager) or revert to iptables mode. Document node provisioning requirements. ### 3. Conntrack Table Exhaustion **Symptom:** `nf_conntrack: table full, dropping packet` in kernel logs. Connection timeouts under load. **Why:** High connection churn, especially with iptables mode. Default `nf_conntrack_max` too low. **Recovery:** Increase conntrack table size (with caution): ```bash sudo sysctl -w net.netfilter.nf_conntrack_max=1048576 ``` Make permanent in `/etc/sysctl.conf`. Consider reducing connection tracking load by using IPVS (which has less conntrack overhead) or tuning pod lifecycle. ### 4. High Sync Duration After Node Scale **Symptom:** p99 sync duration spikes above 5s when many pods/endpoints change. **Why:** iptables is O(n) in number of rules. With thousands of endpoints, rule updates become slow. **Recovery:** Switch to IPVS, which uses hash tables (O(1) lookups) and scales better. Or adjust `iptables.syncPeriod` to 60s to reduce sync frequency, but this delays updates. Always test changes in a non-production cluster first. ## Common Pitfalls and How to Avoid Them 1. **Changing settings without baselines.** Without pre-change metrics, you cannot know if tuning helped or hurt. Always capture baseline metrics for at least 24 hours. 2. **Ignoring node resource limits.** kube-proxy uses CPU and memory, especially IPVS mode with many services. Ensure the DaemonSet has sensible resource requests/limits. Check current usage: ```bash kubectl top pod -n kube-system -l k8s-app=kube-proxy ``` If near limits, adjust in DaemonSet spec. 3. **Overlooking kernel parameters.** For iptables, `net.netfilter.nf_conntrack_max` and `net.ipv4.ip_forward` are critical. For IPVS, `net.ipv4.vs.conntrack` may matter. Align node sysctl settings with kube-proxy requirements. 4. **Restarting kube-proxy during high traffic.** Restart causes temporary disruption as rules are reprogrammed. Schedule changes during maintenance windows or use rolling updates with `kubectl rollout restart` and monitor. 5. **Not testing recovery.** Before making a change, have a rollback plan: backup config, know the exact command to revert, and verify rollback works. 6. **Misunderstanding IPVS scheduler.** Default round-robin may not be optimal for all workloads. Consider `lc` (least connections) or `sh` (source hashing) if needed. Set in ConfigMap under `ipvs.scheduler`. Document why. ## Operations Checklist Use this checklist for every kube-proxy change. Each item is assigned to a specific owner (e.g., SRE lead) and has a review frequency. | Step | Action | Owner | Frequency / Review | |---|---|---|---| | 1 | Capture baseline metrics (sync duration, CPU, memory) | On-call SRE | Before every change | | 2 | Backup current ConfigMap | Platform engineer | Before every change | | 3 | Assess blast radius: how many nodes? critical services? | Team lead | Before every change | | 4 | Test change in staging cluster | DevOps engineer | Quarterly or per change | | 5 | Apply change to production via GitOps or approved PR | Release manager | Per change | | 6 | Verify metrics and service connectivity | On-call SRE | Within 15 minutes after change | | 7 | Document outcome in runbook | Owner (assign) | Immediately after change | | 8 | Review performance trends weekly | SRE lead | Weekly | | 9 | Revisit tuning parameters if workload changes | Engineering manager | Monthly or on major release | Example: Priya Shah, Engineering Lead, reviews sync duration histograms every Monday and decides if further tuning is needed. ## Conclusion Kube-proxy performance tuning is an ongoing discipline. Start with environment inventory and baseline metrics, make minimal changes with clear rollback plans, verify outcomes with concrete commands, and learn from failure modes. By following safe configuration practices and avoiding common pitfalls, you can keep Kubernetes networking fast and reliable. Assign clear ownership and revisit tuning decisions regularly as your cluster evolves. The payoff is lower latency, fewer dropped connections, and a more predictable platform for your applications.