Kubernetes Virtual IPs: Advanced Concepts and Practical Examples

Intro

Kubernetes virtual IPs are at the heart of service networking. Every time you create a Service, Kubernetes assigns a stable virtual IP (the cluster IP) that does not belong to any single pod. Instead, the control plane programs every node to rewrite traffic destined for that virtual IP to one of the ready pods behind the service. This indirection gives you stable endpoints, load balancing, and discovery even as pods are created, destroyed, or scaled.

This article is a deep dive for developers, DevOps consultants, and technical startup teams who already understand basic Kubernetes objects and want to operate virtual IPs with confidence. You will learn the internals of kube-proxy in its three main modes - iptables, IPVS, and eBPF - and how each mode installs the data path that turns a virtual IP into pod traffic. We will reinforce every concept with concrete commands, expected output, and failure signals. When things go wrong, you will know what to inspect first and how to recover.

We will start by establishing a clear version and environment inventory. From there, we will explore the safe configuration path, detailed verification and diagnostics, common failure modes and recovery procedures, and finally an operations checklist you can reuse in your own clusters. Throughout, the focus is on operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before you change a single line of configuration, you need to know exactly what you are running. For virtual IP behavior, the relevant components are the Kubernetes API server version, the kube-proxy version, and the mode kube-proxy is using. The mode determines how virtual IPs are programmed on each node. A cluster running kube-proxy in iptables mode will show very different debugging commands than one using IPVS or the eBPF-based replacement.

Confirm the Kubernetes and kube-proxy versions

Run these read-only commands on a control plane node or from a machine with kubectl access:

kubectl version --short

Expected output (example):

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.4

Next, identify the kube-proxy version on a node:

kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide

NAME               READY   STATUS    RESTARTS   AGE   IP           NODE
kube-proxy-8fz6s   1/1     Running   0          10d   10.0.0.4     node-1
kube-proxy-m2v8x   1/1     Running   0          10d   10.0.0.5     node-2

Check the container image version:

kubectl get pods -n kube-system -l k8s-app=kube-proxy -o jsonpath='{.items[0].spec.containers[0].image}'

registry.k8s.io/kube-proxy:v1.28.4

Determine the kube-proxy mode

kube-proxy mode is often set via a ConfigMap. Inspect it:

kubectl describe configmap kube-proxy -n kube-system

Look for a line like mode: ipvs or mode: iptables. If the ConfigMap does not show a mode, inspect the kube-proxy daemonset arguments:

kubectl describe daemonset kube-proxy -n kube-system

In the pod template, locate the --proxy-mode flag. If absent, the default mode is iptables on most Linux distributions.

On a node, you can also read the current mode from kube-proxy logs:

kubectl logs -n kube-system kube-proxy-8fz6s | head -20

Expected log excerpt:

I1205 10:00:01.123456       1 server_others.go:72] "Using iptables Proxier"

or

I1205 10:00:01.223456       1 server_others.go:72] "Using ipvs Proxier"

Check prerequisites for each mode

Each kube-proxy mode has prerequisites:

• iptables: Requires the iptables user-space tool and kernel modules ip_tables, iptable_nat, iptable_filter. On most modern distributions these are present by default.
• IPVS: Requires kernel modules ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, and nf_conntrack. To load them on a systemd node, run:

modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack

Verify they are loaded:

lsmod | grep ip_vs

ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  0
ip_vs                 172032  6 ip_vs_sh,ip_vs_wrr,ip_vs_rr
nf_conntrack          155648  7 ip_vs,nf_nat,nf_conntrack_netlink

• eBPF mode (e.g., Cilium, Calico eBPF): Requires a kernel with BPF and BPF JIT enabled, and often specific capabilities. For Cilium, you can check its status:

kubectl -n kube-system exec ds/cilium -- cilium status

Establish a read-only baseline

Capture a snapshot of the current virtual IP programming. For iptables mode, list the NAT rules for services:

sudo iptables-save -t nat | grep -i 'KUBE-SERVICES' | head -20

For IPVS mode, list the virtual services:

sudo ipvsadm -L -n

Example output for a service with ClusterIP 10.96.0.10:

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.0.10:443 rr
  -> 10.244.1.5:8443              Masq    1      0          0
  -> 10.244.2.3:8443              Masq    1      0          0

Store these snapshots with timestamps and node names so you can compare before and after any change.

Safe Configuration Path

When you need to change virtual IP behavior, the safest approach is to introduce the change in a test namespace or a single service first, validate the data path, and then roll it out. Never modify a production service's kube-proxy settings on all nodes simultaneously without a canary.

Example: switching a service from iptables to IPVS handling

Suppose you want to test IPVS for a specific service, even though the cluster kube-proxy is in iptables mode. You cannot switch just one service; the mode is cluster-wide per node. However, you can test a new kube-proxy mode on a single node by cordoning that node and running kube-proxy with the new mode as a separate process.

First, cordon the node and drain it of workloads except daemonsets:

kubectl cordon node-2
kubectl drain node-2 --ignore-daemonsets --delete-emptydir-data

On node-2, stop the existing kube-proxy (assuming it runs as a static pod or systemd service). For a static pod, remove the manifest temporarily. Then run kube-proxy manually with IPVS mode:

sudo kube-proxy \
  --kubeconfig=/etc/kubernetes/kube-proxy.conf \
  --proxy-mode=ipvs \
  --v=2

Wait for the IPVS virtual server table to populate:

sudo ipvsadm -L -n

You should see entries for all services after a few seconds.

Validate with a test deployment

Create a simple test namespace and deployment:

apiVersion: v1
kind: Namespace
metadata:
  name: vip-test
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: vip-test
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: vip-test
spec:
  selector:
    app: nginx
  ports:
  - port: 80
    targetPort: 80
  type: ClusterIP

Apply and verify the service has a cluster IP:

kubectl apply -f test.yaml
kubectl get svc -n vip-test nginx

NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
nginx   ClusterIP   10.104.12.33    <none>        80/TCP    10s

From node-2, curl the virtual IP:

curl -s http://10.104.12.33

You should see the nginx welcome page. At the same time, watch the IPVS connections:

sudo ipvsadm -L -n --stats | grep 10.104.12.33

If traffic works, record the configuration and continue testing more services. If not, revert by restarting the original kube-proxy and uncordoning the node.

Safe configuration checklist

• Always make a backup of the kube-proxy ConfigMap or node configuration before changes: kubectl get configmap kube-proxy -n kube-system -o yaml > kube-proxy-backup.yaml
• Use a canary node or namespace.
• Verify with read-only commands before and after.
• Have a rollback plan: restore the backup and restart kube-proxy.
• For cloud load balancers or ingress, first test with a local port-forward to isolate application issues from networking issues: kubectl -n vip-test port-forward svc/nginx 8080:80.

Verification and Diagnostics

Once you have a stable configuration, you need ongoing verification and the ability to diagnose failures quickly. This section covers the commands and signals for each kube-proxy mode.

Verifying virtual IP reachability

The simplest check is from a pod on the same node and a pod on a different node. Run a temporary pod with curl:

kubectl run tmp --rm -it --image=curlimages/curl -- /bin/sh

Inside the pod, test the service by name and by IP:

curl http://nginx.vip-test.svc.cluster.local
curl http://10.104.12.33

If both work, the virtual IP is programmed correctly on at least one node. To verify all nodes, run a pod on each node using a daemonset or a pod with nodeSelector.

Inspecting iptables rules

For iptables mode, list the chain that handles service traffic:

sudo iptables -t nat -L KUBE-SERVICES -n -v | head -30

Look for a rule matching your service's cluster IP and port. Example:

Chain KUBE-SERVICES (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SVC-2IRACUALRELARSND  tcp  --  *      *       0.0.0.0/0            10.104.12.33         /* vip-test/nginx cluster IP */ tcp dpt:80

Follow the target chain to see the endpoints:

sudo iptables -t nat -L KUBE-SVC-2IRACUALRELARSND -n -v

Chain KUBE-SVC-2IRACUALRELARSND (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 KUBE-SEP-AJ5Q5TXBVHZIL7ZI  all  --  *      *       0.0.0.0/0            0.0.0.0/0            statistic mode random probability 0.33333333349
    0     0 KUBE-SEP-BL5Q5TXBVHZIL8ZI  all  --  *      *       0.0.0.0/0            0.0.0.0/0            statistic mode random probability 0.50000000000
    0     0 KUBE-SEP-CJ5Q5TXBVHZIL9ZI  all  --  *      *       0.0.0.0/0            0.0.0.0/0

Each KUBE-SEP-xxx chain points to a pod IP. Verify that the endpoint IPs match the ready pods:

kubectl get endpoints -n vip-test nginx

NAME    ENDPOINTS                                      AGE
nginx   10.244.1.5:80,10.244.2.3:80,10.244.3.8:80     10m

If the iptables rules do not match the endpoints, kube-proxy has not updated the node. Check kube-proxy logs for errors.

Inspecting IPVS rules

For IPVS mode, use ipvsadm:

sudo ipvsadm -L -n | grep -A5 10.104.12.33

Expected output:

TCP  10.104.12.33:80 rr
  -> 10.244.1.5:80                 Masq    1      0          0
  -> 10.244.2.3:80                 Masq    1      0          0
  -> 10.244.3.8:80                 Masq    1      0          0

The scheduler (rr for round-robin) is chosen by the kube-proxy configuration. Check the real server status:

sudo ipvsadm -L -n --stats | grep -A5 10.104.12.33

This shows active connections and errors per backend. If a backend shows many InActConn or zero Weight, it may be marked as unavailable.

eBPF mode verification

For clusters using Cilium or similar eBPF-based service handling, use the cilium CLI:

kubectl -n kube-system exec ds/cilium -- cilium service list

Find your service:

ID   Frontend           Service Type   Backend
2    10.104.12.33:80    ClusterIP      1 => 10.244.1.5:80
                                       2 => 10.244.2.3:80
                                       3 => 10.244.3.8:80

To trace a packet, use cilium monitor:

kubectl -n kube-system exec ds/cilium -- cilium monitor --type drop

Look for drops related to your service IP.

Diagnostic command summary

For any mode, also inspect kube-proxy logs:

kubectl logs -n kube-system kube-proxy-8fz6s --tail=50

Increase verbosity temporarily if needed by editing the daemonset args to include --v=4, then restart.

Failure Modes and Recovery

Virtual IP failures can manifest as connection timeouts, intermittent 504 errors, or traffic going to the wrong pod. We will cover the most common failure modes and step-by-step recovery.

Failure: no endpoints behind the service

If a service selector does not match any pods, the virtual IP is still assigned, but kube-proxy does not create any forwarding rules. Clients connecting to the virtual IP will hang or receive connection refused (depending on the mode).

Diagnosis:

kubectl get endpoints -n vip-test nginx

If the ENDPOINTS column is empty or shows <none>, the selector is wrong or the pods are not ready.

Check pod labels and readiness:

kubectl get pods -n vip-test -l app=nginx -o wide
kubectl describe pod -n vip-test <pod-name>

Recovery:

• Correct the service selector to match the pod labels.
• Ensure pods pass readiness probes. If no readiness probe is defined, the pod is ready as soon as it starts, but it may still not accept connections; add a proper readiness probe.
• If using headless service (clusterIP: None), virtual IP behavior is different; you should use DNS-based discovery.

Failure: kube-proxy is not running or is crashed

If kube-proxy pods are not running on a node, that node will not have any virtual IP programming. Pods on that node cannot reach services via cluster IP.

Diagnosis:

kubectl get pods -n kube-system -l k8s-app=kube-proxy -o wide

Look for CrashLoopBackOff or Error states.

Check logs of a crashed pod:

kubectl logs -n kube-system kube-proxy-8fz6s --previous

Common causes:

• Missing kernel modules for the mode (e.g., IPVS modules not loaded).
• Invalid kubeconfig or missing RBAC permissions for the kube-proxy service account.
• Node resource pressure.

Recovery:

• Load required kernel modules as shown earlier.
• Verify the kube-proxy service account and cluster role binding: kubectl get clusterrolebinding kube-proxy.
• If all else fails, restart the pod by deleting it: kubectl delete pod -n kube-system kube-proxy-8fz6s (the daemonset will recreate it).

Failure: virtual IP reachable but slow or unbalanced

This can happen in IPVS mode if the scheduler is not appropriate for the workload, or if one backend pod is overloaded. It can also happen in iptables mode when there are too many rules and packet traversal is slow.

Diagnosis:

• Check per-backend statistics with ipvsadm -L -n --stats.
• Look for high InActConn on a single backend.
• Check CPU usage of kube-proxy: kubectl top pod -n kube-system kube-proxy-8fz6s.
• In iptables mode, count the number of rules: iptables -t nat -L KUBE-SERVICES -n | wc -l; large numbers (>1000) can cause performance issues.

Recovery:

• If using IPVS, switch scheduler to lc (least-connection) or wlc (weighted least-connection) for better dynamic balancing. Edit the kube-proxy ConfigMap scheduler field and restart kube-proxy.
• For iptables, consider migrating to IPVS or eBPF mode if the cluster is large.
• Scale backend pods to distribute load.

Failure: external traffic to NodePort or LoadBalancer does not reach the virtual IP

This often occurs when the NodePort range is blocked by firewall rules or the cloud provider's security group does not open the node ports.

Diagnosis:

kubectl get svc -n vip-test nginx -o wide

Note the NodePort, e.g., 30345. Then from a remote machine, test connectivity to any node IP on that port:

curl http://<node-public-ip>:30345

If timeout, check firewall rules on the node and cloud security groups. Also verify kube-proxy is listening on the node port:

sudo netstat -tlnp | grep 30345

Recovery:

• Open the NodePort range in your firewall or security group.
• If using a cloud load balancer, ensure the load balancer target group includes all nodes and health checks pass.

Recovery verification

After any recovery action, always re-verify the virtual IP from a client pod:

kubectl run curl-test --rm -it --image=curlimages/curl -- sh

curl -v http://nginx.vip-test.svc.cluster.local

Expected output should show HTTP/1.1 200 OK.

Operations Checklist

Use this checklist before and after any change to virtual IP configuration in a production cluster.

Pre-change:

• [ ] Record cluster and kube-proxy versions: kubectl version --short
• [ ] Determine kube-proxy mode: check ConfigMap or logs
• [ ] Capture current virtual IP programming: iptables, ipvsadm, or cilium output
• [ ] Identify blast radius: which nodes and services will be affected?
• [ ] Prepare rollback plan: backup ConfigMap and node configuration
• [ ] Schedule a maintenance window if needed

During change:

• [ ] Apply change to a canary node or namespace first
• [ ] Monitor kube-proxy logs for errors
• [ ] Verify virtual IP programming on the canary: iptables -t nat -L KUBE-SERVICES or ipvsadm -L -n
• [ ] Test service connectivity from a pod on the canary node

Post-change:

• [ ] Verify endpoints match pods: kubectl get endpoints
• [ ] Run a connectivity test from a pod on each node
• [ ] Check for CPU/memory spikes in kube-proxy pods
• [ ] Document the change and any new metrics or baselines
• [ ] If failure occurs, execute rollback plan and verify restoration

Ongoing verification:

• [ ] Monitor kube-proxy pod status in kube-system
• [ ] Periodically review ipvsadm -L -n --stats for connection distribution
• [ ] Set alerts for service endpoint count drops (use Prometheus metrics kube_endpoint_address_available)
• [ ] Keep kube-proxy version in sync with control plane

Conclusion

Kubernetes virtual IPs are a powerful abstraction, but they depend on correctly functioning kube-proxy instances on every node. By understanding the underlying mechanisms - whether iptables, IPVS, or eBPF - you can diagnose and fix issues that are not obvious from application logs alone.

In this article, we moved from environment inventory to safe configuration, verification, and recovery. We provided concrete commands and expected outputs for each mode. The key takeaway is to always observe current state before making changes, limit changes to a canary, and have a rollback plan.

As a next step, pick one low-risk verification in your own cluster: record the kube-proxy mode, capture the virtual IP programming for one service, and test connectivity from a pod on a different node. Compare the results with the expected behavior described here. If something differs, investigate using the diagnostic commands. This hands-on practice will deepen your understanding and prepare you for real incidents.