## Intro

Kubernetes pod affinity and anti-affinity rules let you control which nodes your pods land on based on the labels of other pods running on those nodes. This is powerful for optimizing latency, improving resilience, and enforcing co-location or separation. However, misconfigurations often lead to pods stuck in Pending state, unexpected scheduling, or cluster inefficiencies. This guide walks through the most common errors, how to diagnose them, and safe fixes with practical examples.

## Version and Environment Inventory

Before troubleshooting affinity issues, establish your environment baseline. Check your Kubernetes version, because affinity syntax and behavior have evolved. Use `kubectl version --short` to see client and server versions. For example:

```bash
kubectl version --short
```

Expected output (example):

```
Client Version: v1.24.3
Server Version: v1.24.3
```

Note: In newer versions, `--short` is deprecated; use `kubectl version` without flags.

Next, understand your cluster topology. For affinity to work, nodes must have the relevant labels. For example, for a topology key like `topology.kubernetes.io/zone`, each node should have a zone label. List nodes with their labels:

```bash
kubectl get nodes --show-labels
```

Look for labels such as `kubernetes.io/hostname`, `topology.kubernetes.io/zone`, `topology.kubernetes.io/region`, or custom labels like `disktype=ssd`.

**Prerequisites:**

- Cluster with at least two nodes (to demonstrate anti-affinity).
- `kubectl` configured with access to the cluster.
- Understanding of labels and selectors.

## Safe Configuration Path

Start with a minimal example and test in a non-production namespace if possible. Here we create a scenario: a web app and a cache that should be co-located (affinity), and web replicas that should be spread across nodes (anti-affinity).

### Example: Pod Affinity

We want a web pod to schedule on a node that already runs a pod with label `app=cache`. Define the pod as follows:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: web-affinity
spec:
  affinity:
    podAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
      - labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - cache
        topologyKey: kubernetes.io/hostname
  containers:
  - name: web
    image: nginx
```

Apply it:

```bash
kubectl apply -f web-affinity.yaml
```

If no cache pod exists, the web pod will remain Pending. This is a common error: unsatisfiable affinity. We'll cover diagnosis later.

### Example: Pod Anti-Affinity

To spread web pods across nodes, use anti-affinity with `topologyKey: kubernetes.io/hostname`:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-deploy
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: web
            topologyKey: kubernetes.io/hostname
      containers:
      - name: web
        image: nginx
```

This ensures no two web pods from this deployment run on the same node. If you have fewer nodes than replicas, pods will be Pending.

## Verification and Diagnostics

### Checking Pod Status

To see if pods are unschedulable due to affinity, use:

```bash
kubectl get pods
```

Look for pods with status `Pending`. Then describe the pod:

```bash
kubectl describe pod <pod-name>
```

In the Events section, look for messages like:

```
Warning  FailedScheduling  2m (x25 over 5m)  default-scheduler  0/3 nodes are available: 1 node(s) didn't match pod affinity/anti-affinity, 2 node(s) didn't match pod anti-affinity rules.
```

This indicates the scheduler could not find a node satisfying the rules.

### Checking Node Labels

Ensure all topology keys used in your affinity rules have corresponding labels on nodes. For example, if you use `topologyKey: topology.kubernetes.io/zone`, check nodes:

```bash
kubectl get nodes -L topology.kubernetes.io/zone
```

If a node lacks the label, it will not be considered for pods using that topologyKey.

### Simulating Scheduling

There is no built-in dry-run for scheduling, but you can use tools like the Kubernetes scheduler in debug mode or third-party validators. A simpler check: temporarily scale down replicas to see if one pod can schedule.

## Failure Modes and Recovery

### Common Errors and Fixes

- **Unsatisfiable Pod Affinity**: Pod requires a label that no existing pod has. Fix: Ensure the target pod exists and has the correct label. Example: Before applying `web-affinity.yaml`, create a cache pod:

  ```yaml
  apiVersion: v1
  kind: Pod
  metadata:
    name: cache-pod
    labels:
      app: cache
  spec:
    containers:
    - name: redis
      image: redis
  ```

  Then apply the web pod. It should schedule on the same node.

- **Topology Key Missing on Nodes**: If nodes don't have the topology key label, pods using that key will fail to schedule. Fix: Add the label to nodes:

  ```bash
  kubectl label nodes <node-name> topology.kubernetes.io/zone=us-east-1a
  ```

- **Too Few Nodes for Anti-Affinity**: If you require anti-affinity with `requiredDuringScheduling` and have more replicas than nodes, some pods will be Pending. Fix: Either increase node count, use `preferredDuringScheduling` for soft anti-affinity, or reduce replicas.

- **Label Selector Mismatch**: Ensure the label selector in affinity matches the labels on the pods you want to attract/repel. Use `kubectl get pods --show-labels` to verify.

- **Invalid Topology Key**: If the topology key is not a valid label key, the pod will be rejected by the API server. Example: using `topologyKey: "failure-domain.beta.kubernetes.io/zone"` which is deprecated in newer versions. Use `topology.kubernetes.io/zone` instead.

### Rollback and Recovery

If an affinity rule causes outage, remove or modify the affinity section in the pod spec. For a Deployment, update the pod template:

```bash
kubectl edit deployment web-deploy
```

Remove the affinity block or change it. Then delete the pending pods to force recreation:

```bash
kubectl delete pods -l app=web
```

For a single pod, delete it and reapply without affinity.

## Common Pitfalls and How to Avoid Them

### 1. Overly Restrictive Affinity Rules

**Why it happens**: Using `requiredDuringScheduling` with complex or multiple affinity terms can make it nearly impossible for the scheduler to find a suitable node. For example, requiring both a specific zone and a specific hostname label may be unsatisfiable in a small cluster.

**How to avoid**: Prefer `preferredDuringScheduling` for non-critical constraints. If you must use `required`, test in a staging environment first. Also, ensure that the topology keys you use have enough nodes per domain.

**Recovery**: If pods are stuck, modify the deployment to use `preferredDuringScheduling` or relax the selector. Delete the stuck pods to reschedule.

### 2. Ignoring Node Labels

**Why it happens**: Assuming that nodes have standard labels like `topology.kubernetes.io/zone` when they may not, especially in custom or on-prem clusters.

**How to avoid**: Always check node labels before writing affinity rules. Use `kubectl get nodes --show-labels` and ensure the required labels are present. If not, add them manually or via node labels in your infrastructure.

**Recovery**: Add missing labels to nodes using `kubectl label nodes`.

### 3. Forgetting to Update Labels When Scaling

**Why it happens**: When new nodes are added to the cluster, they may not have the same custom labels as existing nodes, leading to inconsistent scheduling behavior.

**How to avoid**: Use infrastructure as code to label nodes automatically, or at least have a runbook for labeling new nodes.

**Recovery**: Label the new nodes as needed and delete pending pods to reschedule.

### 4. Confusing Pod Affinity with Node Affinity

**Why it happens**: Pod affinity is based on labels of other pods, whereas node affinity is based on labels of nodes. Misunderstanding this can lead to incorrect selectors and topology keys.

**How to avoid**: Clearly understand the difference: pod affinity uses `podAffinity` or `podAntiAffinity` in the pod spec and selects pods, while node affinity uses `nodeAffinity` and selects nodes. Always verify the selector matches the intended resource type.

**Recovery**: Review the YAML for the correct affinity type and fix the selector accordingly.

### 5. Using Deprecated Topology Keys

**Why it happens**: Older documentation and examples may still use deprecated keys like `failure-domain.beta.kubernetes.io/zone` or `beta.kubernetes.io/arch`.

**How to avoid**: Stick to the stable `topology.kubernetes.io/*` labels. The `kubernetes.io/hostname` label is still valid.

**Recovery**: Replace deprecated keys with the stable ones and reapply.

### 6. Not Considering Scheduling Capacity

**Why it happens**: Anti-affinity required rules limit pod density. If you have more replicas than available nodes, some pods will never schedule.

**How to avoid**: Use `preferredDuringScheduling` for anti-affinity when possible, or ensure you have enough nodes to satisfy the replica count. Consider using pod topology spread constraints for more flexible spreading.

**Recovery**: Scale down replicas, add nodes, or switch to preferred anti-affinity.

## Operations Checklist

- Confirm Kubernetes version and that labels exist on nodes.
- Validate that the label selectors in affinity rules match existing pod labels.
- Ensure the topology key is correct and present on all relevant nodes.
- Prefer `preferredDuringScheduling` for soft constraints to avoid hard blocking.
- Monitor pod events for scheduling failures.
- Test affinity rules in a staging namespace before production.
- Keep node labels updated, especially when adding new nodes.
- Use `kubectl describe` to quickly identify unschedulable pods and reasons.
- Document your affinity policies for team reference.

## Conclusion

Pod affinity and anti-affinity are valuable for controlling pod placement, but they require careful configuration. By understanding common errors like unsatisfiable rules, missing labels, and topology key issues, you can quickly diagnose and fix scheduling problems. Always start with a simple test, verify labels, and consider using soft constraints for better resilience. Regularly review your scheduling policies to align with cluster capacity and application needs.