Kubernetes Pod Affinity and Anti-Affinity: Common Errors and Practical Fixes

Intro

Kubernetes pod affinity and anti-affinity rules let you control which nodes your pods land on based on the labels of other pods running on those nodes. This is powerful for optimizing latency, improving resilience, and enforcing co-location or separation. However, misconfigurations often lead to pods stuck in Pending state, unexpected scheduling, or cluster inefficiencies. This guide walks through the most common errors, how to diagnose them, and safe fixes with practical examples.

Version and Environment Inventory

Before troubleshooting affinity issues, establish your environment baseline. Check your Kubernetes version, because affinity syntax and behavior have evolved. Use kubectl version --short to see client and server versions. For example:

kubectl version --short

Expected output (example):

Client Version: v1.24.3
Server Version: v1.24.3

Note: In newer versions, --short is deprecated; use kubectl version without flags.

Next, understand your cluster topology. For affinity to work, nodes must have the relevant labels. For example, for a topology key like topology.kubernetes.io/zone, each node should have a zone label. List nodes with their labels:

kubectl get nodes --show-labels

Look for labels such as kubernetes.io/hostname, topology.kubernetes.io/zone, topology.kubernetes.io/region, or custom labels like disktype=ssd.

Prerequisites:

Cluster with at least two nodes (to demonstrate anti-affinity).
kubectl configured with access to the cluster.
Understanding of labels and selectors.

Safe Configuration Path

Start with a minimal example and test in a non-production namespace if possible. Here we create a scenario: a web app and a cache that should be co-located (affinity), and web replicas that should be spread across nodes (anti-affinity).

Example: Pod Affinity

We want a web pod to schedule on a node that already runs a pod with label app=cache. Define the pod as follows:

apiVersion: v1
kind: Pod
metadata:
  name: web-affinity
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - cache
        topologyKey: kubernetes.io/hostname
  containers:
  - name: web
    image: nginx

Apply it:

kubectl apply -f web-affinity.yaml

If no cache pod exists, the web pod will remain Pending. This is a common error: unsatisfiable affinity. We'll cover diagnosis later.

Example: Pod Anti-Affinity

To spread web pods across nodes, use anti-affinity with topologyKey: kubernetes.io/hostname:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deploy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: web
            topologyKey: kubernetes.io/hostname
      containers:
      - name: web
        image: nginx

This ensures no two web pods from this deployment run on the same node. If you have fewer nodes than replicas, pods will be Pending.

Verification and Diagnostics

Checking Pod Status

To see if pods are unschedulable due to affinity, use:

kubectl get pods

Look for pods with status Pending. Then describe the pod:

kubectl describe pod <pod-name>

In the Events section, look for messages like:

Warning  FailedScheduling  2m (x25 over 5m)  default-scheduler  0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules.

This indicates the scheduler could not find a node satisfying the rules.

Checking Node Labels

Ensure all topology keys used in your affinity rules have corresponding labels on nodes. For example, if you use topologyKey: topology.kubernetes.io/zone, check nodes:

kubectl get nodes -L topology.kubernetes.io/zone

If a node lacks the label, it will not be considered for pods using that topologyKey.

Simulating Scheduling

There is no built-in dry-run for scheduling, but you can use tools like the Kubernetes scheduler in debug mode or third-party validators. A simpler check: temporarily scale down replicas to see if one pod can schedule.

Failure Modes and Recovery

Common Errors and Fixes

Unsatisfiable Pod Affinity: Pod requires a label that no existing pod has. Fix: Ensure the target pod exists and has the correct label. Example: Before applying web-affinity.yaml, create a cache pod:

  apiVersion: v1
  kind: Pod
  metadata:
    name: cache-pod
    labels:
      app: cache
  spec:
    containers:
    - name: redis
      image: redis

Then apply the web pod. It should schedule on the same node.

Topology Key Missing on Nodes: If nodes don't have the topology key label, pods using that key will fail to schedule. Fix: Add the label to nodes:

  kubectl label nodes <node-name> topology.kubernetes.io/zone=us-east-1a

Too Few Nodes for Anti-Affinity: If you require anti-affinity with requiredDuringScheduling and have more replicas than nodes, some pods will be Pending. Fix: Either increase node count, use preferredDuringScheduling for soft anti-affinity, or reduce replicas.

Label Selector Mismatch: Ensure the label selector in affinity matches the labels on the pods you want to attract/repel. Use kubectl get pods --show-labels to verify.

Invalid Topology Key: If the topology key is not a valid label key, the pod will be rejected by the API server. Example: using topologyKey: "failure-domain.beta.kubernetes.io/zone" which is deprecated in newer versions. Use topology.kubernetes.io/zone instead.

Rollback and Recovery

If an affinity rule causes outage, remove or modify the affinity section in the pod spec. For a Deployment, update the pod template:

kubectl edit deployment web-deploy

Remove the affinity block or change it. Then delete the pending pods to force recreation:

kubectl delete pods -l app=web

For a single pod, delete it and reapply without affinity.

Common Pitfalls and How to Avoid Them

1. Overly Restrictive Affinity Rules

Why it happens: Using requiredDuringScheduling with complex or multiple affinity terms can make it nearly impossible for the scheduler to find a suitable node. For example, requiring both a specific zone and a specific hostname label may be unsatisfiable in a small cluster.

How to avoid: Prefer preferredDuringScheduling for non-critical constraints. If you must use required, test in a staging environment first. Also, ensure that the topology keys you use have enough nodes per domain.

Recovery: If pods are stuck, modify the deployment to use preferredDuringScheduling or relax the selector. Delete the stuck pods to reschedule.

2. Ignoring Node Labels

Why it happens: Assuming that nodes have standard labels like topology.kubernetes.io/zone when they may not, especially in custom or on-prem clusters.

How to avoid: Always check node labels before writing affinity rules. Use kubectl get nodes --show-labels and ensure the required labels are present. If not, add them manually or via node labels in your infrastructure.

Recovery: Add missing labels to nodes using kubectl label nodes.

3. Forgetting to Update Labels When Scaling

Why it happens: When new nodes are added to the cluster, they may not have the same custom labels as existing nodes, leading to inconsistent scheduling behavior.

How to avoid: Use infrastructure as code to label nodes automatically, or at least have a runbook for labeling new nodes.

Recovery: Label the new nodes as needed and delete pending pods to reschedule.

4. Confusing Pod Affinity with Node Affinity

Why it happens: Pod affinity is based on labels of other pods, whereas node affinity is based on labels of nodes. Misunderstanding this can lead to incorrect selectors and topology keys.

How to avoid: Clearly understand the difference: pod affinity uses podAffinity or podAntiAffinity in the pod spec and selects pods, while node affinity uses nodeAffinity and selects nodes. Always verify the selector matches the intended resource type.

Recovery: Review the YAML for the correct affinity type and fix the selector accordingly.

5. Using Deprecated Topology Keys

Why it happens: Older documentation and examples may still use deprecated keys like failure-domain.beta.kubernetes.io/zone or beta.kubernetes.io/arch.

How to avoid: Stick to the stable topology.kubernetes.io/* labels. The kubernetes.io/hostname label is still valid.

Recovery: Replace deprecated keys with the stable ones and reapply.

6. Not Considering Scheduling Capacity

Why it happens: Anti-affinity required rules limit pod density. If you have more replicas than available nodes, some pods will never schedule.

How to avoid: Use preferredDuringScheduling for anti-affinity when possible, or ensure you have enough nodes to satisfy the replica count. Consider using pod topology spread constraints for more flexible spreading.

Recovery: Scale down replicas, add nodes, or switch to preferred anti-affinity.

Operations Checklist

Confirm Kubernetes version and that labels exist on nodes.
Validate that the label selectors in affinity rules match existing pod labels.
Ensure the topology key is correct and present on all relevant nodes.
Prefer preferredDuringScheduling for soft constraints to avoid hard blocking.
Monitor pod events for scheduling failures.
Test affinity rules in a staging namespace before production.
Keep node labels updated, especially when adding new nodes.
Use kubectl describe to quickly identify unschedulable pods and reasons.
Document your affinity policies for team reference.

Conclusion

Pod affinity and anti-affinity are valuable for controlling pod placement, but they require careful configuration. By understanding common errors like unsatisfiable rules, missing labels, and topology key issues, you can quickly diagnose and fix scheduling problems. Always start with a simple test, verify labels, and consider using soft constraints for better resilience. Regularly review your scheduling policies to align with cluster capacity and application needs.