## Intro Kubernetes Pod Affinity and Anti-Affinity are scheduling features that control pod placement relative to other pods based on labels. While they are powerful for optimizing performance, ensuring high availability, and enforcing data locality or separation, misconfiguration can lead to unschedulable pods, uneven distribution, and operational headaches. This article provides a practical production operations checklist for managing pod affinity and anti-affinity. It is written for developers, DevOps engineers, and technical startup teams responsible for deploying and maintaining Kubernetes workloads. We cover environment prerequisites, safe configuration paths, verification techniques, failure modes, and a repeatable operations checklist. The goal is to help you avoid common pitfalls and keep your clusters healthy. Affinity and anti-affinity rules are not set-and-forget. They require planning, validation, and ongoing review. We include concrete commands, YAML snippets, and scenarios to illustrate each step. Whether you are new to these features or looking to refine your operational practices, this guide will help you manage scheduling constraints with confidence. ## Version and Environment Inventory Before modifying affinity rules, establish a clear picture of your Kubernetes environment. This inventory helps you understand the available scheduling domain and avoid misconfigurations due to version differences or missing labels. ### 1.1 Kubernetes Version Affinity and anti-affinity have been stable since Kubernetes 1.6, but newer versions have added improvements such as namespace selectors and more flexible matching. Check your cluster version with: ```bash kubectl version --short ``` Expected output: ``` Client Version: v1.27.3 Server Version: v1.27.3 ``` Ensure you run a supported version. As of this writing, Kubernetes 1.24 and later are commonly used in production. If you are on an older version, review the Kubernetes changelog for scheduling changes. ### 1.2 Topology Key Configuration The topology key is the node label that defines the scheduling domain for your affinity rule. Common topology keys include: - `kubernetes.io/hostname` (node-level) - `topology.kubernetes.io/zone` (availability zone) - `topology.kubernetes.io/region` (region) - Custom labels such as `rack`, `datacenter`, or `failure-domain.beta.kubernetes.io/zone` (deprecated). List node labels to see what is available: ```bash kubectl get nodes --show-labels ``` Example output: ``` NAME STATUS ROLES AGE VERSION LABELS node-1 Ready 10d v1.27.3 kubernetes.io/hostname=node-1,topology.kubernetes.io/zone=us-east-1a node-2 Ready 10d v1.27.3 kubernetes.io/hostname=node-2,topology.kubernetes.io/zone=us-east-1b ``` Ensure labels intended as topology keys are present and consistent across nodes. Inconsistent labels can lead to pods being unschedulable. ### 1.3 Cluster Topology and Capacity Understand cluster size and node distribution. Affinity rules constrain placement, so you need enough nodes in each topology domain to satisfy requirements. For example, if you require pod anti-affinity with `topologyKey: kubernetes.io/hostname` for three replicas, you need at least three nodes. Check node count and capacity: ```bash kubectl get nodes kubectl describe nodes | grep -A 5 "Allocatable" ``` Review current pod distribution to anticipate conflicts: ```bash kubectl get pods -o wide --all-namespaces ``` ### 1.4 Prerequisites for Production Use - **Access**: You need cluster-admin or appropriate RBAC permissions to modify workloads and view events. - **Observability**: Have monitoring in place (e.g., Prometheus, Grafana) to track pod scheduling and node resource usage. - **Backup**: Ensure you can roll back deployments easily using version control and deployment tools. - **Testing**: Use a staging cluster or namespace to test affinity changes before promoting to production. ## Safe Configuration Path Introducing affinity rules should be a controlled, incremental process. Follow these steps to minimize risk. ### 2.1 Start with a Narrow, Measurable Pilot Choose a single non-critical workload with clear placement requirements. For example, a cache service that should run on the same node as its dependent application, or replicas that should spread across zones. Define success metrics: number of pods scheduled, distribution across topology domains, and performance impact. ### 2.2 Define Clear Intent Use descriptive labels on pods to make affinity rules readable and maintainable. For example, label application pods with `app: myapp` and `tier: cache` or `tier: frontend`. Example label assignment in a Deployment spec: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: redis-cache spec: replicas: 2 selector: matchLabels: app: redis-cache template: metadata: labels: app: redis-cache tier: cache spec: containers: - name: redis image: redis:6.2 ``` Apply the manifest and verify labels: ```bash kubectl apply -f redis-cache.yaml kubectl get pods -l app=redis-cache --show-labels ``` ### 2.3 Choose Affinity Type There are three types: - `requiredDuringSchedulingIgnoredDuringExecution`: Hard requirement. If not satisfied, pod remains unscheduled. - `preferredDuringSchedulingIgnoredDuringExecution`: Soft preference. Scheduler tries to satisfy but will place pod anyway. - `requiredDuringSchedulingRequiredDuringExecution`: Planned but not yet available. Avoid using. Start with soft preferences (`preferred...`) when possible, especially for less critical constraints. Reserve hard requirements for essential placement rules and ensure capacity exists to satisfy them. Example of a pod anti-affinity soft rule to spread pods across nodes: ```yaml spec: affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - myapp topologyKey: kubernetes.io/hostname ``` ### 2.4 Use Namespace Selectors if Needed By default, `podAffinityTerm` only matches pods in the same namespace. If you need cross-namespace affinity, add a `namespaceSelector` to the term. This feature is stable in v1.24 and requires the `podAffinityNamespaceSelector` feature gate on older clusters. Example with namespace selector: ```yaml podAffinityTerm: labelSelector: matchLabels: app: myapp topologyKey: kubernetes.io/hostname namespaceSelector: matchLabels: team: data ``` ### 2.5 Validate Before Applying Use `kubectl apply --dry-run=client` or `--dry-run=server` to check syntax and admission. Also, use `kubectl diff` to see changes: ```bash kubectl apply -f deployment.yaml --dry-run=client kubectl diff -f deployment.yaml ``` ### 2.6 Gradual Rollout Apply affinity changes to a canary deployment or a subset of replicas. Monitor pod scheduling and application behavior before scaling out. Use deployment strategies like RollingUpdate to minimize impact. ## Verification and Diagnostics After applying affinity rules, verify that pods are scheduled as intended and diagnose any issues. ### 3.1 Check Pod Placement Use `kubectl get pods -o wide` to see which nodes the pods landed on: ```bash kubectl get pods -l app=myapp -o wide ``` Expected output: ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES myapp-6d4b7c9f5-abcde 1/1 Running 0 5m 10.244.1.5 node-1 myapp-6d4b7c9f5-fghij 1/1 Running 0 5m 10.244.2.7 node-2 ``` If anti-affinity is working, replicas should be on different nodes. If they are on the same node, check your topology key and labels. ### 3.2 Inspect Scheduler Events If a pod is pending, describe it to see why: ```bash kubectl describe pod ``` Look for events similar to: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m default-scheduler 0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules. ``` This indicates your rule is too restrictive. ### 3.3 Use kubectl get events for Cluster-wide View ```bash kubectl get events --sort-by=.lastTimestamp ``` Filter by reason: ```bash kubectl get events | grep FailedScheduling ``` ### 3.4 Validate Labels on Nodes and Pods Mislabeling is a common cause. Verify node labels: ```bash kubectl get nodes --show-labels ``` Verify pod labels: ```bash kubectl get pods -l --show-labels ``` ### 3.5 Test Scheduler with Simulated Pods Create a dummy pod with the same affinity rules to see if scheduling works without affecting real workload. Use a simple pod spec and check its status. ### 3.6 Check Topology Spread Use a simple script to count pods per node: ```bash kubectl get pods -l app=myapp -o wide | awk '{print $7}' | sort | uniq -c ``` This will show distribution. ## Failure Modes and Recovery Affinity rules can cause pods to remain unscheduled, affect cluster autoscaling, or cause overcommitment. Be prepared to diagnose and recover. ### 4.1 Unschedulable Pods **Symptom**: Pods stuck in Pending state. **Cause**: Hard affinity requirements cannot be met due to insufficient nodes or conflicting labels. **Recovery**: - Check events as described. - If necessary, relax the rule by switching from `required` to `preferred`. - Add more nodes with appropriate labels. - Correct pod labels so they match the selector. Example of relaxing: Change this: ```yaml affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - myapp topologyKey: kubernetes.io/hostname ``` to this: ```yaml affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - myapp topologyKey: kubernetes.io/hostname ``` Then update the deployment: ```bash kubectl apply -f deployment.yaml kubectl rollout status deployment/myapp ``` ### 4.2 Overly Broad Topology Key **Symptom**: Pods are spread too thinly, causing underutilization or excessive network latency. **Cause**: Using `topologyKey: topology.kubernetes.io/zone` when `kubernetes.io/hostname` would be better, or vice versa. **Recovery**: Adjust topology key to match the intended failure domain. Monitor application performance after change. ### 4.3 Affinity with Node Selector Conflicts Combining node selectors with pod affinity can create impossible constraints. For example, a node selector requires `disktype: ssd` but affinity requires co-location with pods on nodes labeled `disktype: hdd`. **Recovery**: Review all constraints together. Simplify by removing conflicting rules or combining them into a single node affinity. ### 4.4 Cluster Autoscaler Interference Hard anti-affinity may prevent the cluster autoscaler from scaling down nodes if pods cannot be moved. Conversely, soft rules may allow overpacking, causing resource pressure. **Recovery**: Use appropriate pod disruption budgets and consider node affinity in conjunction with pod anti-affinity to guide autoscaler. Test autoscaling scenarios in staging. ### 4.5 Rollback Strategy Always maintain versioned manifests. If a new affinity configuration causes issues, roll back to the previous version: ```bash kubectl rollout undo deployment/myapp ``` Or apply previous manifest: ```bash kubectl apply -f deployment-previous.yaml ``` After rollback, verify pod distribution returns to expected state. ## Operations Checklist Use this repeatable checklist whenever you plan to modify affinity rules or during periodic reviews. ### 5.1 Pre-Deployment Checklist - [ ] Confirm Kubernetes version supports required features (namespace selectors, etc.). - [ ] Verify node labels for topology keys are consistent and correct. - [ ] Ensure enough nodes in each topology domain to satisfy hard requirements. - [ ] Identify affected workloads and their current placement. - [ ] Define desired placement in terms of labels and topology keys. - [ ] Choose affinity type (required vs preferred) based on criticality. - [ ] Write or update manifests with clear comments. - [ ] Validate manifests with `kubectl apply --dry-run=client` and `kubectl diff`. - [ ] Prepare rollback plan and have previous manifests accessible. - [ ] Schedule change during maintenance window if risk is high. ### 5.2 Deployment Checklist - [ ] Apply changes to a canary or single replica first. - [ ] Monitor pod status and events immediately after apply. - [ ] Check pod distribution across expected topology domains. - [ ] Verify application health and performance (response times, error rates). - [ ] If canary is successful, gradually scale to full desired state. - [ ] Update documentation and runbooks with new affinity rules. ### 5.3 Post-Deployment Verification - [ ] Run `kubectl get pods -o wide -l ` and confirm node placement. - [ ] Run `kubectl describe pod` for any warnings. - [ ] Check cluster events for scheduling failures. - [ ] Confirm no nodes are under excessive resource pressure due to placement changes. - [ ] Validate that autoscaling (if used) behaves as expected. ### 5.4 Ongoing Maintenance - [ ] Periodically review affinity rules against current cluster topology and workload changes. - [ ] Update node labels when adding or removing nodes to maintain consistency. - [ ] Monitor for unschedulable pods using alerts on `FailedScheduling` events. - [ ] Keep manifests in version control and enforce review process. - [ ] Test disaster recovery scenarios: if a zone fails, can pods reschedule with current rules? ## Common Pitfalls and How to Avoid Them Beyond failure modes, several subtle pitfalls can trip up even experienced operators. ### 6.1 Ignoring Topology Key Semantics **Pitfall**: Using `topologyKey: kubernetes.io/hostname` for anti-affinity when you actually need zone-level spreading. This can lead to all replicas landing in the same zone if multiple nodes are in that zone. **Why it happens**: A misunderstanding of what the topology key represents. The key defines the scope of the rule. **How to avoid**: Document the intended failure domain for each workload. For high availability across zones, use `topology.kubernetes.io/zone` as the key. For node-level separation, use `kubernetes.io/hostname`. Review the actual node labels before setting the key. ### 6.2 Mixing Pod Affinity and Anti-Affinity Incorrectly **Pitfall**: Using both pod affinity and anti-affinity on the same pod without considering their interaction. For example, requiring affinity to a cache pod and anti-affinity from other replicas of the same app may be impossible if the cache pod runs on a node already occupied by a replica. **Why it happens**: Operators may add rules incrementally without a holistic view. **How to avoid**: Draw a diagram of desired placement and test with a small number of pods. Use `kubectl describe pod` to see scheduler decisions. ### 6.3 Forgetting to Update Labels **Pitfall**: Changing application labels (e.g., renaming `tier: backend` to `tier: api`) without updating existing affinity rules that reference the old label. **Why it happens**: Labels are often changed during refactoring or new deployments. **How to avoid**: Use a label management policy. When changing labels, search for affinity rules referencing them using `grep -r "old-label" manifests/`. Update rules and pods simultaneously. ### 6.4 Relying Solely on Hard Requirements **Pitfall**: Setting all affinity rules as `requiredDuringSchedulingIgnoredDuringExecution` without capacity planning. This can lead to pods being unschedulable if node count drops or labels drift. **Why it happens**: A desire for strict control, but without considering dynamic cluster changes. **How to avoid**: Prefer soft rules for non-critical constraints. If hard rules are necessary, ensure cluster autoscaling can add nodes with matching labels and that there is a monitoring alert for unschedulable pods. ### 6.5 Neglecting to Test with Realistic Workloads **Pitfall**: Testing affinity rules with dummy pods that have minimal resource requests, only to find that real pods with larger requests cannot be scheduled due to resource constraints combined with placement rules. **Why it happens**: Dummy pods may not reflect actual resource consumption. **How to avoid**: Use representative resource requests and limits in test pods or test in a staging environment with realistic workloads. Consider using `kubectl apply --dry-run=server` with actual pod specs to check admission. ## Conclusion Pod Affinity and Anti-Affinity are critical for controlling workload placement in Kubernetes, but they require careful planning and operation. By following the checklist in this article, you can avoid common pitfalls such as unschedulable pods, uneven distribution, and interference with cluster autoscaling. Remember to start with a narrow, measurable pilot, validate configurations in a safe environment, and always have a rollback plan. Regularly review affinity rules as your cluster evolves to ensure they continue to meet operational and business requirements. Incorporate these practices into daily operations to achieve greater reliability and efficiency in Kubernetes deployments. With a disciplined approach, you can leverage affinity and anti-affinity to optimize performance and availability without introducing unnecessary risk.