Kubernetes Virtual IPs Architecture Explained with Practical Examples

Intro

Kubernetes Virtual IPs (VIPs) are the stable network identity that lets clients reach a set of interchangeable Pods. Instead of tracking individual Pod IPs, you create a Service, get a single virtual IP, and Kubernetes makes sure traffic lands on a healthy backend. The trick is that the virtual IP is just that: virtual. It exists in the cluster networking rules, not on any single interface. Understanding how those rules are built, updated, and sometimes broken is the difference between a Service that quietly works and a Service that silently drops traffic.

This article is for developers, DevOps consultants, and technical startup teams who need to move from “the Service is down” to a verified root cause. We will walk through the components, data flow, design decisions, and operational commands behind Kubernetes VIPs, with concrete examples and expected output. The focus is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and know how to recover if the expected state is not reached.

Version and Environment Inventory

Before you touch anything, capture your environment. For Kubernetes VIPs, the relevant components are the Service, Endpoints/EndpointSlices, and the kube-proxy implementation on each node. The supported version range matters because kube-proxy modes and default behaviors have shifted over releases. For example, iptables mode is stable and widely used, IPVS mode is available in most distributions since Kubernetes 1.11, and EndpointSlices became the default for Services with large backends around 1.21.

Start with read-only observations. Identify the Kubernetes version and the proxy mode on a control-plane node:

kubectl version --short
kubectl get nodes -o wide
kubectl get configmap kube-proxy -n kube-system -o yaml | grep -A 2 'mode:'

Expected output will look something like this (version numbers vary):

Client Version: v1.28.0
Server Version: v1.28.2

For the proxy mode, if the config map has mode: ipvs, that is your mode. If it is absent or says mode: "", kube-proxy defaults to iptables. This matters because troubleshooting commands differ: for iptables you use iptables-save, for IPVS you use ipvsadm -Ln.

Next, list the Services and their VIPs in the namespace you care about:

kubectl get services -n production

Example output:

NAME        TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
orders      ClusterIP   10.96.120.45    <none>        8080/TCP   12d
payments    ClusterIP   10.96.0.12      <none>        443/TCP    30d
web         LoadBalancer 10.99.23.1     35.224.12.8    80:30021/TCP 5d

That ClusterIP column is your virtual IP. It is allocated from the cluster's service CIDR and is only meaningful with the cluster networking rules. Capture these values and the service definitions before making changes:

kubectl get service orders -n production -o yaml > orders-service-backup.yaml
kubectl get endpointslices -n production -l kubernetes.io/service-name=orders -o yaml > orders-endpointslices-backup.yaml

Prerequisites for safe work: you need cluster-admin or at least permission to get and describe Services, EndpointSlices, and kube-proxy config; and you need shell access to a node to inspect iptables/IPVS rules. Always separate observation from intervention. Capture current state and timestamps first, protect credentials, then change one scoped item only when you understand its blast radius and recovery path.

How Kubernetes Virtual IPs Actually Work

The virtual IP is realized by kube-proxy on each node. kube-proxy watches the Kubernetes API for Service and EndpointSlice changes and then programs the node's kernel networking rules so that traffic sent to the ClusterIP gets rewritten to a Pod IP. There are three main modes:

• iptables: The default in many clusters. kube-proxy writes chains in the NAT table. For each Service, there is a chain KUBE-SERVICES that matches destination ClusterIP:port and jumps to a KUBE-SVC-... chain. That chain uses statistic matching to distribute to a KUBE-SEP-... chain per Endpoint, where DNAT rewrites the destination to the Pod IP and port.
• IPVS: Uses the Linux IP Virtual Server for load balancing. It creates a virtual server for the ClusterIP and real servers for each Pod. IPVS supports more scheduling algorithms (round robin, least connections, etc.) and scales to more Services and Endpoints with less overhead.
• Userspace: The oldest mode, where kube-proxy itself acts as a proxy and traffic goes through user space. This is slow and rarely used, but it still exists in some legacy clusters.

Let's follow a concrete request. Suppose you have a Deployment of 3 Pods labeled app: orders-api, and a Service orders with ClusterIP 10.96.120.45 and port 8080. A client inside the cluster connects to 10.96.120.45:8080. On the node where the client Pod runs, the iptables rules created by kube-proxy will look like this (simplified from iptables-save -t nat):

-A KUBE-SERVICES -d 10.96.120.45/32 -p tcp -m comment --comment "production/orders:http cluster IP" -m tcp --dport 8080 -j KUBE-SVC-XYZABC
-A KUBE-SVC-XYZABC -m comment --comment "production/orders:http" -m statistic --mode random --probability 0.33333333349 -j KUBE-SEP-AAA111
-A KUBE-SVC-XYZABC -m comment --comment "production/orders:http" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-BBB222
-A KUBE-SVC-XYZABC -j KUBE-SEP-CCC333
-A KUBE-SEP-AAA111 -s 10.244.1.5/32 -m comment --comment "production/orders:http" -j KUBE-MARK-MASQ
-A KUBE-SEP-AAA111 -p tcp -m comment --comment "production/orders:http" -m tcp -j DNAT --to-destination 10.244.1.5:8080
... and similar for BBB222 and CCC333

Here you can see the random distribution: the first endpoint gets 1/3 of connections, the second gets 1/2 of the remaining, and the third gets the rest. The actual probabilities are computed to approximate equal distribution. If kube-proxy is in IPVS mode, you would instead see:

ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.96.120.45:8080 rr
  -> 10.244.1.5:8080              Masq    1      0          0
  -> 10.244.2.8:8080              Masq    1      0          0
  -> 10.244.3.9:8080              Masq    1      0          0

The rr scheduler means round robin, and each real server has weight 1. This is a cleaner model for debugging connection distribution.

The key design trade-off: the virtual IP does not exist as an interface. You cannot ping it and expect a reply. It is a pattern match in the kernel's network stack. That is why a Service can have zero Pods without any error, but connections to the ClusterIP will hang or be refused depending on the proxy mode and the publishNotReadyAddresses setting.

Safe Configuration Path

When you need to change a Service, VIP, or related networking config, follow a safe path that starts with observation and ends with verification. Let's walk through a realistic task: adding a new Service for a payment API and selecting the right backend Pods.

First, verify that the Pods are running and have the labels you plan to use:

kubectl get pods -n production -l app=payments-api --show-labels

Expected:

NAME                            READY   STATUS    RESTARTS   AGE   LABELS
payments-api-6c9f8f7b4d-abcde   1/1     Running   0          3d    app=payments-api,version=v2
payments-api-6c9f8f7b4d-fghij   1/1     Running   0          3d    app=payments-api,version=v2

Then create the Service with a YAML manifest, but do not apply it blindly. Dry-run and validate:

kubectl apply -f payments-service.yaml --dry-run=server

If there are errors in the YAML or the selector, you see them here. The manifest might look like this:

apiVersion: v1
kind: Service
metadata:
  name: payments
  namespace: production
  labels:
    app: payments-api
spec:
  selector:
    app: payments-api
    version: v2
  ports:
  - name: https
    protocol: TCP
    port: 443
    targetPort: 8443
  type: ClusterIP

Notice the selector uses two labels. This is safer than a broad selector because it limits the blast radius to only version: v2 Pods. If you later deploy version: v3, it will not receive traffic until you update the Service selector. Keep the local test small: apply one manifest, inspect the generated resources, and test with kubectl port-forward before moving to a cloud load balancer or ingress controller. For example:

kubectl apply -f payments-service.yaml
kubectl get service payments -n production -o wide

Expected ClusterIP is assigned (e.g., 10.96.0.12) and ENDPOINTS shows the Pod IPs. Then test connectivity with a temporary pod:

kubectl run -it --rm debug --image=busybox -- sh
# Inside the pod:
wget -qO- --timeout=2 https://10.96.0.12:443 --no-check-certificate

If you get a response or a specific application error (not a timeout), the VIP path works. A timeout before any response suggests the routing rules are missing or the Pods are not listening on targetPort.

Always separate observation from intervention. Before applying a change, record the current Service and EndpointSlices as shown earlier. Know how to roll back: kubectl delete -f payments-service.yaml removes the Service and its VIP, but the Pods remain. If you modified an existing Service, restore from your backup YAML.

Verification and Diagnostics

When a Service's VIP is not working, the problem usually sits in one of four layers: the Service definition (selector mismatch), the EndpointSlice (no ready Pods), the kube-proxy rules (stale or missing), or the network policy (blocking traffic). Let's go through a systematic diagnostic path.

Step 1: Confirm the Service and its virtual IP.

kubectl describe service orders -n production

Look for the Endpoints field. If it is empty (<none>), no Pods match the selector, or the matched Pods are not ready. For example:

Endpoints:          <none>

This is a common failure: the selector in the Service spec does not match any running Pod labels. Verify the selector with:

kubectl get pods -n production --selector=app=orders-api

If no Pods are listed, the labels are wrong. Fix the selector to match actual Pod labels, or update the Deployment to use the expected labels. Be careful: changing labels on a running Deployment can cause a rolling update. Instead, change the Service selector if you control it.

Step 2: Check the EndpointSlice for the Service.

EndpointSlices are the successor to Endpoints and are used by kube-proxy to program rules. List them:

kubectl get endpointslices -n production -l kubernetes.io/service-name=orders

Example with one slice and three endpoints:

NAME           ADDRESSTYPE   PORTS   ENDPOINTS   AGE
orders-abcde   IPv4          8080    3           12m

If the EndpointSlice count is zero or the endpoint addresses are wrong, kube-proxy has nothing to program. Describe the slice to see the endpoint readiness:

kubectl describe endpointslices orders-abcde -n production

Look for conditions: ready: true on each endpoint. If an endpoint is not ready, the Pod may be failing readiness probes.

Step 3: Inspect the node's network rules.

On a worker node (use kubectl get nodes -o wide to find an IP, then SSH), check the proxy rules. For iptables:

sudo iptables-save -t nat | grep 10.96.120.45

You should see the KUBE-SERVICES chain match and a DNAT to a Pod IP. If nothing appears, kube-proxy has not programmed the VIP on this node. This can happen if kube-proxy is not running or is stuck. Check kube-proxy Pods:

kubectl get pods -n kube-system -l k8s-app=kube-proxy
kubectl logs -n kube-system kube-proxy-<node-name> --tail=50

Look for errors about writing iptables rules or connectivity to the API server. If the logs are clean but rules are missing, restart the kube-proxy Pod on the affected node:

kubectl delete pod -n kube-system kube-proxy-<node-name>

It will be recreated by its DaemonSet and will repopulate the rules.

For IPVS mode, use sudo ipvsadm -Ln and verify the virtual server exists with the correct real servers. Also check that the kernel modules are loaded:

lsmod | grep ip_vs

If IPVS is not functional, kube-proxy may fall back to iptables depending on configuration. Check kube-proxy logs for messages like Failed to load kernel module ip_vs.

Step 4: Test from a client Pod and capture traffic.

Run a debug Pod and attempt a connection:

kubectl run -it --rm debug --image=nicolaka/netshoot -- bash
# Inside:
curl -v http://10.96.120.45:8080/health

A successful response with HTTP 200 confirms the full path. If the connection hangs, use tcpdump on the network namespace or conntrack -L to see where packets stall.

Failure Modes and Recovery

Understanding common failure modes helps you respond quickly and without panic. Here are the most frequent issues with Kubernetes VIPs, why they happen, and how to recover.

1. Empty Endpoints (Selector Mismatch or Pods Not Ready)

Symptoms: Service exists, ClusterIP is assigned, but connections time out or are refused. kubectl describe service shows Endpoints: <none>.

Why it happens: The Service's selector does not match any running Pod labels, or all matching Pods are failing readiness probes. This is often caused by a typo in the label key or value, or by a Deployment that was scaled to zero.

How to avoid: Use kubectl get pods --show-labels to verify the exact labels, and test the selector with kubectl get pods -l <selector> before creating the Service. In the Service manifest, keep the selector simple and aligned with the Pod template.

How to recover: Update the Service selector to match the actual Pods, or fix the readiness probe so Pods become Ready. Apply the change and verify endpoints appear within a few seconds.

2. Stale kube-proxy Rules

Symptoms: Endpoints are present in the API, but traffic from a client to the ClusterIP still fails or goes to an old Pod IP. On the node, iptables-save shows rules for an IP that no longer exists.

Why it happens: kube-proxy failed to update the node's rules due to an API connection issue, a crash, or resource pressure. This can happen during a network partition or when kube-proxy is OOMKilled.

How to avoid: Monitor kube-proxy logs and metrics for rule sync delays. Ensure the node has sufficient memory and that the API server is reachable from all nodes.

How to recover: Delete the kube-proxy Pod on the affected node to force a restart. It will resync the rules from the API. If the problem persists, check the kube-proxy config and logs for the specific error.

3. IPVS Mode Issues with Port Mapping

Symptoms: In IPVS mode, connections to the ClusterIP work from some Pods but not others, or certain ports are unreachable. ipvsadm -Ln shows incorrect ports or the virtual server is missing.

Why it happens: IPVS uses different kernel modules and has nuanced handling of targetPort and named ports. Also, IPVS may not handle all protocols correctly (e.g., SCTP).

How to avoid: Test IPVS mode in a staging cluster before enabling it in production. Ensure all kernel modules are loaded and that kube-proxy is configured with strictARP: false if using a CNI that does not support ARP responses.

How to recover: Switch back to iptables mode by editing the kube-proxy config map and restarting kube-proxy DaemonSet, or fix the IPVS configuration and ensure the real servers are correct.

4. Network Policies Blocking Traffic

Symptoms: The Service VIP and endpoints are correct, kube-proxy rules exist, but connections from one Pod fail while others succeed. The client Pod just times out.

Why it happens: A NetworkPolicy may be blocking egress from the client Pod or ingress to the backend Pods. Network policies are enforced by the CNI plugin and can interrupt the DNAT flow.

How to avoid: Audit NetworkPolicies before deploying new Services. Use a policy tool or manually inspect all policies in the namespace for conflicting rules.

How to recover: Identify the blocking policy with kubectl get networkpolicies -n <namespace>. Modify the policy to allow the required traffic (e.g., from client Pod labels to backend Pod labels on the target port), then test again.

5. Externally Visible VIPs (LoadBalancer) Not Working

Symptoms: A Service of type LoadBalancer has an external IP but it is not reachable from outside the cluster. The ClusterIP works internally, but external traffic fails.

Why it happens: Cloud provider load balancer misconfiguration, security group rules, or firewall settings. The load balancer may not be forwarding to the node ports correctly, or health checks are failing and marking nodes unhealthy.

How to avoid: Test with a ClusterIP and kubectl port-forward before creating a LoadBalancer. When using a cloud provider, check the provider's load balancer logs and health check status in the cloud console.

How to recover: Recreate the Service if the load balancer is stuck. Verify that node ports are open on all nodes and that the cloud firewall allows traffic from the load balancer to the nodes on the node port range (default 30000-32767).

Operations Checklist

Use this checklist when deploying or modifying a Kubernetes Service to minimize risk and speed up recovery. Each item includes a concrete command or verification step, an accountable owner (not a group), and a review frequency.

After completing the checklist, document the results and any exceptions in your runbook. The accountable owner should sign off on production changes and ensure the review frequency is met. This prevents silent misconfigurations from becoming outages.

Common Pitfalls and How to Avoid Them

Beyond the failure modes above, several subtle mistakes repeatedly trip up teams working with Kubernetes VIPs. Recognize these early signs.

Assuming the ClusterIP is pingable

Newcomers often run ping 10.96.120.45 and get no response, then assume the Service is down. The ClusterIP is not an interface; it is a NAT rule. The correct test is to attempt a TCP connection to the Service port from a Pod or node. Use curl, wget, or nc inside the cluster.

Using --service-cluster-ip-range inconsistently

If your cluster was initialized with a particular service CIDR, all Services must have ClusterIPs in that range. Changing it later is destructive and requires recreating the cluster. Always confirm the CIDR with kubectl cluster-info dump | grep service-cluster-ip-range before creating many Services. If you see errors like ClusterIP out of range, you have a mismatch.

Not handling headless Services differently

A headless Service (with clusterIP: None) has no virtual IP. Instead, DNS returns the individual Pod IPs. Many operators treat it like a regular Service and wonder why the VIP does not work. If your Service spec says clusterIP: None, you must use Pod IPs directly or through a custom DNS resolver. Avoid headless Services unless you specifically need Pod-level discovery.

Ignoring session affinity when state is involved

If your application stores session state on the Pod, you need sessionAffinity: ClientIP on the Service. Otherwise, kube-proxy may route a client to a different Pod on each connection, breaking state. Add the field and test with repeated requests from the same IP. Be aware that this limits load balancing and may cause uneven distribution.

Forgetting to delete a LoadBalancer Service

Removing a LoadBalancer Service can leave orphaned cloud resources (e.g., an ELB) that continue to incur costs. Always check your cloud provider's console after deleting such Services, or use kubectl get svc -A | grep LoadBalancer to audit. Some providers have finalizers that slow deletion; ensure they complete.

Overcomplicating selectors

A selector with many labels is brittle. If one label changes during a Deployment update, the Service loses all backends. Use the minimum labels necessary to identify the correct Pods, and avoid labels that are likely to change (such as version if you do rolling updates with both versions concurrently).

Conclusion

Kubernetes Virtual IPs are the glue between stable network names and ephemeral Pod IPs. Understanding how kube-proxy programs the node's kernel rules, how EndpointSlices feed those rules, and how to diagnose failures is essential for anyone running production Services. The architecture is deliberately simple, but misconfigurations in selectors, proxy mode, or network policies can silently break connectivity.

A reliable operational workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Start with one low-risk verification from the checklist: record the current state of a Service, run a diagnostic command, compare the result with the expected output, and review dependencies like EndpointSlices and kube-proxy logs. Then you can confidently manage virtual IPs across your cluster without guesswork.