Kubernetes Pod Affinity and Anti-Affinity: A Production Operations Checklist with Practical Examples

Intro

Kubernetes Pod Affinity and Anti-Affinity are scheduling features that control pod placement relative to other pods based on labels. While they are powerful for optimizing performance, ensuring high availability, and enforcing data locality or separation, misconfiguration can lead to unschedulable pods, uneven distribution, and operational headaches.

This article provides a practical production operations checklist for managing pod affinity and anti-affinity. It is written for developers, DevOps engineers, and technical startup teams responsible for deploying and maintaining Kubernetes workloads. We cover environment prerequisites, safe configuration paths, verification techniques, failure modes, and a repeatable operations checklist. The goal is to help you avoid common pitfalls and keep your clusters healthy.

Affinity and anti-affinity rules are not set-and-forget. They require planning, validation, and ongoing review. We include concrete commands, YAML snippets, and scenarios to illustrate each step. Whether you are new to these features or looking to refine your operational practices, this guide will help you manage scheduling constraints with confidence.

Version and Environment Inventory

Before modifying affinity rules, establish a clear picture of your Kubernetes environment. This inventory helps you understand the available scheduling domain and avoid misconfigurations due to version differences or missing labels.

1.1 Kubernetes Version

Affinity and anti-affinity have been stable since Kubernetes 1.6, but newer versions have added improvements such as namespace selectors and more flexible matching. Check your cluster version with:

kubectl version --short

Expected output:

Client Version: v1.27.3
Server Version: v1.27.3

Ensure you run a supported version. As of this writing, Kubernetes 1.24 and later are commonly used in production. If you are on an older version, review the Kubernetes changelog for scheduling changes.

1.2 Topology Key Configuration

The topology key is the node label that defines the scheduling domain for your affinity rule. Common topology keys include:

• kubernetes.io/hostname (node-level)
• topology.kubernetes.io/zone (availability zone)
• topology.kubernetes.io/region (region)
• Custom labels such as rack, datacenter, or failure-domain.beta.kubernetes.io/zone (deprecated).

List node labels to see what is available:

kubectl get nodes --show-labels

Example output:

NAME     STATUS   ROLES    AGE   VERSION   LABELS
node-1   Ready    <none>   10d   v1.27.3   kubernetes.io/hostname=node-1,topology.kubernetes.io/zone=us-east-1a
node-2   Ready    <none>   10d   v1.27.3   kubernetes.io/hostname=node-2,topology.kubernetes.io/zone=us-east-1b

Ensure labels intended as topology keys are present and consistent across nodes. Inconsistent labels can lead to pods being unschedulable.

1.3 Cluster Topology and Capacity

Understand cluster size and node distribution. Affinity rules constrain placement, so you need enough nodes in each topology domain to satisfy requirements. For example, if you require pod anti-affinity with topologyKey: kubernetes.io/hostname for three replicas, you need at least three nodes.

Check node count and capacity:

kubectl get nodes
kubectl describe nodes | grep -A 5 "Allocatable"

Review current pod distribution to anticipate conflicts:

kubectl get pods -o wide --all-namespaces

1.4 Prerequisites for Production Use

• Access: You need cluster-admin or appropriate RBAC permissions to modify workloads and view events.
• Observability: Have monitoring in place (e.g., Prometheus, Grafana) to track pod scheduling and node resource usage.
• Backup: Ensure you can roll back deployments easily using version control and deployment tools.
• Testing: Use a staging cluster or namespace to test affinity changes before promoting to production.

Safe Configuration Path

Introducing affinity rules should be a controlled, incremental process. Follow these steps to minimize risk.

2.1 Start with a Narrow, Measurable Pilot

Choose a single non-critical workload with clear placement requirements. For example, a cache service that should run on the same node as its dependent application, or replicas that should spread across zones.

Define success metrics: number of pods scheduled, distribution across topology domains, and performance impact.

2.2 Define Clear Intent

Use descriptive labels on pods to make affinity rules readable and maintainable. For example, label application pods with app: myapp and tier: cache or tier: frontend.

Example label assignment in a Deployment spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis-cache
spec:
  replicas: 2
  selector:
    matchLabels:
      app: redis-cache
  template:
    metadata:
      labels:
        app: redis-cache
        tier: cache
    spec:
      containers:
      - name: redis
        image: redis:6.2

Apply the manifest and verify labels:

kubectl apply -f redis-cache.yaml
kubectl get pods -l app=redis-cache --show-labels

2.3 Choose Affinity Type

There are three types:

• requiredDuringSchedulingIgnoredDuringExecution: Hard requirement. If not satisfied, pod remains unscheduled.
• preferredDuringSchedulingIgnoredDuringExecution: Soft preference. Scheduler tries to satisfy but will place pod anyway.
• requiredDuringSchedulingRequiredDuringExecution: Planned but not yet available. Avoid using.

Start with soft preferences (preferred...) when possible, especially for less critical constraints. Reserve hard requirements for essential placement rules and ensure capacity exists to satisfy them.

Example of a pod anti-affinity soft rule to spread pods across nodes:

spec:
  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        podAffinityTerm:
          labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values:
              - myapp
          topologyKey: kubernetes.io/hostname

2.4 Use Namespace Selectors if Needed

By default, podAffinityTerm only matches pods in the same namespace. If you need cross-namespace affinity, add a namespaceSelector to the term. This feature is stable in v1.24 and requires the podAffinityNamespaceSelector feature gate on older clusters.

Example with namespace selector:

podAffinityTerm:
  labelSelector:
    matchLabels:
      app: myapp
  topologyKey: kubernetes.io/hostname
  namespaceSelector:
    matchLabels:
      team: data

2.5 Validate Before Applying

Use kubectl apply --dry-run=client or --dry-run=server to check syntax and admission. Also, use kubectl diff to see changes:

kubectl apply -f deployment.yaml --dry-run=client
kubectl diff -f deployment.yaml

2.6 Gradual Rollout

Apply affinity changes to a canary deployment or a subset of replicas. Monitor pod scheduling and application behavior before scaling out. Use deployment strategies like RollingUpdate to minimize impact.

Verification and Diagnostics

After applying affinity rules, verify that pods are scheduled as intended and diagnose any issues.

3.1 Check Pod Placement

Use kubectl get pods -o wide to see which nodes the pods landed on:

kubectl get pods -l app=myapp -o wide

Expected output:

NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
myapp-6d4b7c9f5-abcde   1/1     Running   0          5m    10.244.1.5   node-1   <none>           <none>
myapp-6d4b7c9f5-fghij   1/1     Running   0          5m    10.244.2.7   node-2   <none>           <none>

If anti-affinity is working, replicas should be on different nodes. If they are on the same node, check your topology key and labels.

3.2 Inspect Scheduler Events

If a pod is pending, describe it to see why:

kubectl describe pod <pod-name>

Look for events similar to:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules.

This indicates your rule is too restrictive.

3.3 Use kubectl get events for Cluster-wide View

kubectl get events --sort-by=.lastTimestamp

Filter by reason:

kubectl get events | grep FailedScheduling

3.4 Validate Labels on Nodes and Pods

Mislabeling is a common cause. Verify node labels:

kubectl get nodes --show-labels

Verify pod labels:

kubectl get pods -l <selector> --show-labels

3.5 Test Scheduler with Simulated Pods

Create a dummy pod with the same affinity rules to see if scheduling works without affecting real workload. Use a simple pod spec and check its status.

3.6 Check Topology Spread

Use a simple script to count pods per node:

kubectl get pods -l app=myapp -o wide | awk '{print $7}' | sort | uniq -c

This will show distribution.

Failure Modes and Recovery

Affinity rules can cause pods to remain unscheduled, affect cluster autoscaling, or cause overcommitment. Be prepared to diagnose and recover.

4.1 Unschedulable Pods

Symptom: Pods stuck in Pending state.

Cause: Hard affinity requirements cannot be met due to insufficient nodes or conflicting labels.

Recovery:

• Check events as described.
• If necessary, relax the rule by switching from required to preferred.
• Add more nodes with appropriate labels.
• Correct pod labels so they match the selector.

Example of relaxing:

Change this:

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - labelSelector:
        matchExpressions:
        - key: app
          operator: In
          values:
          - myapp
      topologyKey: kubernetes.io/hostname

to this:

affinity:
  podAntiAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      podAffinityTerm:
        labelSelector:
          matchExpressions:
          - key: app
            operator: In
            values:
            - myapp
        topologyKey: kubernetes.io/hostname

Then update the deployment:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/myapp

4.2 Overly Broad Topology Key

Symptom: Pods are spread too thinly, causing underutilization or excessive network latency.

Cause: Using topologyKey: topology.kubernetes.io/zone when kubernetes.io/hostname would be better, or vice versa.

Recovery: Adjust topology key to match the intended failure domain. Monitor application performance after change.

4.3 Affinity with Node Selector Conflicts

Combining node selectors with pod affinity can create impossible constraints. For example, a node selector requires disktype: ssd but affinity requires co-location with pods on nodes labeled disktype: hdd.

Recovery: Review all constraints together. Simplify by removing conflicting rules or combining them into a single node affinity.

4.4 Cluster Autoscaler Interference

Hard anti-affinity may prevent the cluster autoscaler from scaling down nodes if pods cannot be moved. Conversely, soft rules may allow overpacking, causing resource pressure.

Recovery: Use appropriate pod disruption budgets and consider node affinity in conjunction with pod anti-affinity to guide autoscaler. Test autoscaling scenarios in staging.

4.5 Rollback Strategy

Always maintain versioned manifests. If a new affinity configuration causes issues, roll back to the previous version:

kubectl rollout undo deployment/myapp

Or apply previous manifest:

kubectl apply -f deployment-previous.yaml

After rollback, verify pod distribution returns to expected state.

Operations Checklist

Use this repeatable checklist whenever you plan to modify affinity rules or during periodic reviews.

5.1 Pre-Deployment Checklist

• [ ] Confirm Kubernetes version supports required features (namespace selectors, etc.).
• [ ] Verify node labels for topology keys are consistent and correct.
• [ ] Ensure enough nodes in each topology domain to satisfy hard requirements.
• [ ] Identify affected workloads and their current placement.
• [ ] Define desired placement in terms of labels and topology keys.
• [ ] Choose affinity type (required vs preferred) based on criticality.
• [ ] Write or update manifests with clear comments.
• [ ] Validate manifests with kubectl apply --dry-run=client and kubectl diff.
• [ ] Prepare rollback plan and have previous manifests accessible.
• [ ] Schedule change during maintenance window if risk is high.

5.2 Deployment Checklist

• [ ] Apply changes to a canary or single replica first.
• [ ] Monitor pod status and events immediately after apply.
• [ ] Check pod distribution across expected topology domains.
• [ ] Verify application health and performance (response times, error rates).
• [ ] If canary is successful, gradually scale to full desired state.
• [ ] Update documentation and runbooks with new affinity rules.

5.3 Post-Deployment Verification

• [ ] Run kubectl get pods -o wide -l <selector> and confirm node placement.
• [ ] Run kubectl describe pod for any warnings.
• [ ] Check cluster events for scheduling failures.
• [ ] Confirm no nodes are under excessive resource pressure due to placement changes.
• [ ] Validate that autoscaling (if used) behaves as expected.

5.4 Ongoing Maintenance

• [ ] Periodically review affinity rules against current cluster topology and workload changes.
• [ ] Update node labels when adding or removing nodes to maintain consistency.
• [ ] Monitor for unschedulable pods using alerts on FailedScheduling events.
• [ ] Keep manifests in version control and enforce review process.
• [ ] Test disaster recovery scenarios: if a zone fails, can pods reschedule with current rules?

Common Pitfalls and How to Avoid Them

Beyond failure modes, several subtle pitfalls can trip up even experienced operators.

6.1 Ignoring Topology Key Semantics

Pitfall: Using topologyKey: kubernetes.io/hostname for anti-affinity when you actually need zone-level spreading. This can lead to all replicas landing in the same zone if multiple nodes are in that zone.

Why it happens: A misunderstanding of what the topology key represents. The key defines the scope of the rule.

How to avoid: Document the intended failure domain for each workload. For high availability across zones, use topology.kubernetes.io/zone as the key. For node-level separation, use kubernetes.io/hostname. Review the actual node labels before setting the key.

6.2 Mixing Pod Affinity and Anti-Affinity Incorrectly

Pitfall: Using both pod affinity and anti-affinity on the same pod without considering their interaction. For example, requiring affinity to a cache pod and anti-affinity from other replicas of the same app may be impossible if the cache pod runs on a node already occupied by a replica.

Why it happens: Operators may add rules incrementally without a holistic view.

How to avoid: Draw a diagram of desired placement and test with a small number of pods. Use kubectl describe pod to see scheduler decisions.

6.3 Forgetting to Update Labels

Pitfall: Changing application labels (e.g., renaming tier: backend to tier: api) without updating existing affinity rules that reference the old label.

Why it happens: Labels are often changed during refactoring or new deployments.

How to avoid: Use a label management policy. When changing labels, search for affinity rules referencing them using grep -r "old-label" manifests/. Update rules and pods simultaneously.

6.4 Relying Solely on Hard Requirements

Pitfall: Setting all affinity rules as requiredDuringSchedulingIgnoredDuringExecution without capacity planning. This can lead to pods being unschedulable if node count drops or labels drift.

Why it happens: A desire for strict control, but without considering dynamic cluster changes.

How to avoid: Prefer soft rules for non-critical constraints. If hard rules are necessary, ensure cluster autoscaling can add nodes with matching labels and that there is a monitoring alert for unschedulable pods.

6.5 Neglecting to Test with Realistic Workloads

Pitfall: Testing affinity rules with dummy pods that have minimal resource requests, only to find that real pods with larger requests cannot be scheduled due to resource constraints combined with placement rules.

Why it happens: Dummy pods may not reflect actual resource consumption.

How to avoid: Use representative resource requests and limits in test pods or test in a staging environment with realistic workloads. Consider using kubectl apply --dry-run=server with actual pod specs to check admission.

Conclusion

Pod Affinity and Anti-Affinity are critical for controlling workload placement in Kubernetes, but they require careful planning and operation. By following the checklist in this article, you can avoid common pitfalls such as unschedulable pods, uneven distribution, and interference with cluster autoscaling.

Remember to start with a narrow, measurable pilot, validate configurations in a safe environment, and always have a rollback plan. Regularly review affinity rules as your cluster evolves to ensure they continue to meet operational and business requirements.

Incorporate these practices into daily operations to achieve greater reliability and efficiency in Kubernetes deployments. With a disciplined approach, you can leverage affinity and anti-affinity to optimize performance and availability without introducing unnecessary risk.