## Intro Kubernetes Pod Affinity and Anti-Affinity rules give you fine-grained control over pod scheduling based on other pods' labels. When combined with CI/CD automation, these rules help you enforce high availability, resource locality, and failure isolation in every deployment. This article provides a practical, command-driven workflow for integrating pod affinity and anti-affinity into your pipelines, from version validation to automated verification and rollback. This guide is intended for developers, DevOps consultants, and technical startup teams who manage Kubernetes clusters and want to make scheduling constraints part of their delivery process. We focus on operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. You will learn how to: - Check your Kubernetes version and current scheduling setup. - Define pod affinity and anti-affinity in deployment manifests. - Validate scheduling constraints locally before merging. - Automate deployment with safe rollout strategies. - Diagnose common failures such as unschedulable pods or topology mismatches. - Rollback safely when a change breaks pod placement. By the end, you will have a repeatable pipeline that makes scheduling behavior predictable and auditable. ## Version and Environment Inventory Before applying any affinity or anti-affinity rule, confirm that your cluster supports the features you intend to use. Pod affinity and anti-affinity have been stable since Kubernetes 1.6, but `topologyKey` behavior and `namespaceSelector` options have evolved. For production pipelines, use Kubernetes 1.24 or later where the feature is mature and well-documented. ### Check Kubernetes version and API availability Run the following command to check the server version: ```bash kubectl version --short ``` Expected output includes both client and server versions, for example: ``` Client Version: v1.29.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.7 ``` If the server version is below 1.24, test all affinity rules in a staging cluster first because some edge cases around `matchLabelKeys` and `mismatchLabelKeys` may behave differently. ### Verify scheduler and existing pod distribution Take a read-only snapshot of the current pod placement before any change. This helps you compare after deployment and detect unintended movement. Run: ```bash kubectl get pods -o wide --all-namespaces | tee pods-before.txt ``` The `-o wide` output includes node names and pod IPs, which are essential for understanding scheduling decisions. Save the output with a timestamp in the filename for auditability: ```bash kubectl get pods -o wide --all-namespaces > pods-before-$(date +%Y%m%d-%H%M%S).txt ``` If you have many namespaces, limit the output to your application namespace: ```bash kubectl get pods -o wide -n my-app > pods-before-$(date +%Y%m%d-%H%M%S).txt ``` ### Inspect existing affinity rules Check whether your current deployments already have affinity or anti-affinity configured: ```bash kubectl get deployments -n my-app -o custom-columns=NAME:.metadata.name,AFFINITY:.spec.template.spec.affinity ``` This prints a table like: ``` NAME AFFINITY web api worker map[nodeAffinity:map[...] podAntiAffinity:map[...]] ``` If a deployment already uses affinity, examine its YAML to understand its current constraints: ```bash kubectl get deployment worker -n my-app -o yaml | grep -A 50 affinity ``` ### List nodes and topology labels Affinity and anti-affinity rules often rely on node topology labels such as `kubernetes.io/hostname`, `topology.kubernetes.io/zone`, or `topology.kubernetes.io/region`. List your nodes and their labels to know what topology keys are available: ```bash kubectl get nodes --show-labels ``` Expected output excerpt: ``` NAME STATUS ROLES AGE VERSION LABELS node-1 Ready control-plane 21d v1.28.7 kubernetes.io/hostname=node-1,topology.kubernetes.io/zone=us-east-1a,topology.kubernetes.io/region=us-east-1 node-2 Ready 21d v1.28.7 kubernetes.io/hostname=node-2,topology.kubernetes.io/zone=us-east-1b,topology.kubernetes.io/region=us-east-1 node-3 Ready 21d v1.28.7 kubernetes.io/hostname=node-3,topology.kubernetes.io/zone=us-east-1c,topology.kubernetes.io/region=us-east-1 ``` For multi-zone clusters, using `topology.kubernetes.io/zone` as a topology key spreads pods across failure domains. For single-node development clusters, only `kubernetes.io/hostname` makes sense. ### Confirm required CRDs and admission controllers If you plan to use advanced scheduling features like pod topology spread constraints or `matchLabelKeys`, ensure the relevant feature gates are enabled. Check the scheduler configuration: ```bash kubectl describe pod -n kube-system -l component=kube-scheduler | grep -A 5 "Feature Gates" ``` Most managed Kubernetes services (EKS, GKE, AKS) enable these by default, but self-managed clusters may need manual flag toggling. ## Safe Configuration Path The goal is to define pod affinity and anti-affinity rules that match your availability and performance goals, without making the cluster impossible to schedule. Start with a minimal manifest, validate it locally, then expand scope. ### Understand affinity and anti-affinity types Kubernetes supports two kinds of pod affinity: - **requiredDuringSchedulingIgnoredDuringExecution**: The rule must be satisfied for the pod to be scheduled. If no node meets the rule, the pod remains unschedulable. - **preferredDuringSchedulingIgnoredDuringExecution**: The scheduler tries to satisfy the rule but does not guarantee it. This is useful for soft constraints. Similarly, anti-affinity rules can be required or preferred. Anti-affinity prevents pods from being co-located on the same topology domain (like the same node or zone). ### Example: spread web pods across nodes with anti-affinity Suppose you have a `web` deployment with 3 replicas, and you want to ensure each replica runs on a different node for resilience. Add the following `podAntiAffinity` under the pod spec: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: web namespace: my-app spec: replicas: 3 selector: matchLabels: app: web template: metadata: labels: app: web spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - web topologyKey: kubernetes.io/hostname containers: - name: nginx image: nginx:1.25 ports: - containerPort: 80 ``` Here, the rule says: do not schedule this pod on a node that already has a pod with label `app=web`. The `topologyKey: kubernetes.io/hostname` scopes the rule to a single node. ### Validate manifest syntax and dry-run Before applying, validate the YAML syntax and simulate the change with `--dry-run=client`: ```bash kubectl apply -f web-deployment.yaml --dry-run=client ``` Expected output: ``` deployment.apps/web created (dry run) ``` For a server-side dry-run that checks admission controllers, use `--dry-run=server`: ```bash kubectl apply -f web-deployment.yaml --dry-run=server ``` The server-side dry-run will catch issues like invalid topology keys or label selector mismatches. ### Apply and observe scheduling Apply the manifest and watch the pods being scheduled: ```bash kubectl apply -f web-deployment.yaml kubectl rollout status deployment/web -n my-app ``` Expected output: ``` deployment "web" successfully rolled out ``` Then check pod distribution: ```bash kubectl get pods -n my-app -l app=web -o wide ``` You should see one pod per node, like: ``` NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-5f7c9b6c7-abcde 1/1 Running 0 10s 10.244.1.5 node-1 web-5f7c9b6c7-fghij 1/1 Running 0 10s 10.244.2.7 node-2 web-5f7c9b6c7-klmno 1/1 Running 0 10s 10.244.3.9 node-3 ``` ### Use preferred anti-affinity for flexibility Required anti-affinity can cause scheduling failures if the cluster does not have enough nodes. For example, if you have only two nodes and 3 replicas, required anti-affinity on hostname will leave one pod pending. Use `preferredDuringSchedulingIgnoredDuringExecution` to allow soft spreading: ```yaml affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - web topologyKey: kubernetes.io/hostname ``` Prefer soft anti-affinity for most CI/CD use cases where availability matters more than strict placement. ### Add pod affinity to co-locate services Sometimes you want certain pods to run near each other, like a cache and its consumers. Use `podAffinity` to place pods on the same node or zone: ```yaml affinity: podAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - redis-cache topologyKey: kubernetes.io/hostname ``` This forces the new pod to schedule on a node where a pod with `app=redis-cache` already runs. ### Validate selector matches existing pods If the selector matches no pods, the affinity rule may never be satisfied. Check the labels of your intended target pods: ```bash kubectl get pods -n my-app -l app=redis-cache --show-labels ``` Output should show the label `app=redis-cache`. If not, adjust your manifest labels or selector. ### Use namespaceSelector for cross-namespace affinity By default, affinity only considers pods in the same namespace. To include pods from other namespaces, add `namespaceSelector`: ```yaml podAffinityTerm: labelSelector: matchLabels: app: shared-service namespaceSelector: matchLabels: environment: staging topologyKey: kubernetes.io/hostname ``` This rule looks for pods with `app=shared-service` in namespaces labeled `environment=staging`. Ensure the namespace label is set: ```bash kubectl label namespace shared env=staging --overwrite kubectl get namespace shared --show-labels ``` ## Verification and Diagnostics After deploying with affinity rules, verify that pods are scheduled as expected and that the rules did not introduce new failures. Use a combination of `kubectl` commands and event inspection. ### Check pod status and events First, list all pods in the namespace with wide output: ```bash kubectl get pods -n my-app -o wide ``` If any pod is `Pending`, investigate scheduling events: ```bash kubectl describe pod -n my-app ``` Look for events like: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 40s default-scheduler 0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules. ``` This tells you that the anti-affinity rule is too strict or the cluster lacks nodes. ### Verify that affinity rules are attached Confirm that the running pods have the expected affinity spec: ```bash kubectl get pod -n my-app -o jsonpath='{.spec.affinity}' | jq . ``` Expected output includes the `podAntiAffinity` or `podAffinity` block you defined. ### Test rollout in a staging namespace Never merge affinity changes directly to production. Use a staging namespace or a canary deployment to verify behavior with real traffic. For example, create a separate namespace `staging` and apply your manifest there: ```bash kubectl create namespace staging kubectl apply -f web-deployment.yaml -n staging ``` Then run the same checks in staging. If everything passes, propagate to production via your CI/CD pipeline. ### Automated verification with kubectl in CI In your CI pipeline, add a verification step that runs after `kubectl apply`. For example, in a GitHub Actions workflow: ```yaml - name: Verify pod distribution run: | kubectl get pods -n my-app -l app=web -o json | jq -e '.items | length == 3' kubectl get pods -n my-app -l app=web -o json | jq -e '[.items[].spec.nodeName] | unique | length == 3' ``` The first command checks that 3 replicas are running. The second checks that they are spread across 3 different nodes (assuming a 3-node cluster). If either fails, the step fails. ### Use pod topology spread constraints for additional control Pod topology spread constraints provide a more granular way to spread pods across nodes, zones, or other domains. Combine them with anti-affinity for robust placement. Example: ```yaml topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: web ``` This constraint ensures that the number of pods per zone differs by at most 1. If the cluster cannot satisfy it, the pod remains pending. ### Monitor scheduler logs for debugging If you suspect the scheduler is misbehaving, check its logs: ```bash kubectl logs -n kube-system -l component=kube-scheduler --tail=100 ``` Look for lines mentioning `predicate` or `priority` failures related to your pods. This requires access to the control plane, which may not be available on managed services. In that case, rely on pod events. ## Failure Modes and Recovery Even with careful planning, affinity and anti-affinity rules can cause deployment failures. Understand common failure modes and how to recover safely, using rollback strategies integrated into CI/CD. ### Failure mode 1: Pods stuck in Pending due to unsatisfiable anti-affinity **Symptom:** New pods remain `Pending` and events show `FailedScheduling` with `didn't match pod anti-affinity rules`. **Cause:** Required anti-affinity cannot be satisfied because the cluster lacks sufficient nodes or other pods occupy all eligible nodes. **Recovery:** 1. Check current node count and pod placement: ```bash kubectl get nodes kubectl get pods -n my-app -l app=web -o wide ``` 2. If the cluster has fewer nodes than the required replica count, you must either scale up nodes or relax the rule to `preferredDuringSchedulingIgnoredDuringExecution`. 3. Apply the relaxed manifest: ```bash kubectl apply -f web-deployment-relaxed.yaml -n my-app ``` 4. Verify that pods are now scheduled: ```bash kubectl rollout status deployment/web -n my-app ``` ### Failure mode 2: Affinity rule referencing a non-existent pod label **Symptom:** Pods are scheduled, but not co-located or spread as expected. No error events. **Cause:** The `labelSelector` in the affinity term does not match any existing pods, so the rule has no effect. This often happens due to a typo in the label key or value. **Diagnosis:** ```bash kubectl get pods -n my-app --show-labels ``` Compare with your manifest's `matchLabels`. If mismatched, fix the selector and re-apply. **Recovery:** Update the deployment manifest and roll out a new revision: ```bash kubectl apply -f web-deployment-fixed.yaml -n my-app kubectl rollout restart deployment/web -n my-app ``` ### Failure mode 3: Topology key invalid for cluster topology **Symptom:** Scheduler cannot find nodes with the specified topology key, or the key does not represent a meaningful failure domain. **Example:** Using `topology.kubernetes.io/zone` on a single-zone cluster, or using a custom label that nodes do not have. **Diagnosis:** List node labels: ```bash kubectl get nodes --show-labels | grep topology ``` If the key is absent, the rule will never match. Choose a valid key like `kubernetes.io/hostname` for single-zone clusters. **Recovery:** Modify the topology key in the manifest and re-apply. ### Failure mode 4: CI/CD pipeline applied changes without verification **Symptom:** A pipeline step applied a manifest, but no verification ran, and the production deployment is now broken. **Cause:** Missing automated checks after `kubectl apply`. **Recovery:** Rollback the deployment to the previous revision using `kubectl rollout undo`: ```bash kubectl rollout undo deployment/web -n my-app ``` Check rollout status: ```bash kubectl rollout status deployment/web -n my-app ``` This reverts to the last known good configuration, including previous affinity settings. ### Failure mode 5: Cross-namespace affinity failing due to namespace labels **Symptom:** Pods with `namespaceSelector` cannot be scheduled, and events show no matching namespaces. **Cause:** The target namespaces do not have the required labels. **Recovery:** Add the labels: ```bash kubectl label namespace shared env=staging --overwrite ``` Verify: ```bash kubectl get namespace shared --show-labels ``` Then re-apply the deployment. ### Automating rollback in CI/CD In your pipeline, add a rollback step that triggers automatically if verification fails. For example: ```yaml - name: Apply manifest run: kubectl apply -f web-deployment.yaml -n my-app - name: Verify rollout run: kubectl rollout status deployment/web -n my-app --timeout=120s - name: Rollback on failure if: failure() run: | echo "Rolling back to previous revision" kubectl rollout undo deployment/web -n my-app kubectl rollout status deployment/web -n my-app ``` This ensures that any failed deployment is automatically reverted. ### Document recovery runbooks For each deployment with complex scheduling, maintain a runbook that includes: - The command to capture current pod distribution. - The command to inspect scheduling events. - The exact rollback command. - Contact information for the on-call engineer. Store the runbook in your repository next to the manifests to keep it versioned. ## Operations Checklist Use this checklist before and after every CI/CD run that modifies pod affinity or anti-affinity. It enforces observation, verification, and recoverability. ### Pre-deployment checklist - [ ] Confirm Kubernetes server version supports the intended features: `kubectl version --short` - [ ] Snapshot current pod distribution: `kubectl get pods -o wide --all-namespaces > pods-before-$(date +%Y%m%d-%H%M%S).txt` - [ ] List available node topology labels: `kubectl get nodes --show-labels` - [ ] Validate manifest with server-side dry-run: `kubectl apply -f manifest.yaml --dry-run=server` - [ ] Check that label selectors match existing pods: `kubectl get pods -n my-app --show-labels` - [ ] Ensure topology keys are valid for cluster topology - [ ] Confirm that required anti-affinity does not exceed available capacity - [ ] Set up a staging namespace for pre-production testing - [ ] Define rollback command in the pipeline: `kubectl rollout undo deployment/` - [ ] Add verification steps to the pipeline that check pod placement and replica count ### Post-deployment checklist - [ ] Check rollout status: `kubectl rollout status deployment/ -n my-app` - [ ] Inspect pod events for scheduling warnings: `kubectl describe pod ` - [ ] Confirm pod distribution matches expectations: `kubectl get pods -o wide -n my-app` - [ ] Compare pod distribution to the pre-deployment snapshot - [ ] If any pods are Pending, diagnose with `kubectl describe pod` and fix or rollback - [ ] Record the new pod distribution snapshot for future comparison - [ ] Update the runbook with any new failure modes encountered - [ ] Notify the team of the deployment status ### Sample pre-deployment script Here is a combined script you can run locally before pushing changes: ```bash #!/bin/bash set -e echo "Kubernetes version:" kubectl version --short echo "Snapshotting current pods..." kubectl get pods -o wide --all-namespaces > pods-before-$(date +%Y%m%d-%H%M%S).txt echo "Listing node labels:" kubectl get nodes --show-labels echo "Validating manifest dry-run..." kubectl apply -f web-deployment.yaml --dry-run=server echo "All pre-deployment checks passed." ``` Run it and verify no errors before applying. ## Conclusion Kubernetes Pod Affinity and Anti-Affinity are powerful tools for controlling pod placement, but they must be integrated into CI/CD with the same rigor as any other infrastructure change. This article provided a practical path: start with version and environment checks, define safe configuration, verify automatically, and prepare for failures with documented rollback. Copying commands without checking prerequisites and expected output is not an operations procedure. Instead, choose one low-risk verification for your environment, record the current state, run the documented check, and compare the result with the expected signal. Review dependencies such as node affinity, the kube-scheduler, and the pods themselves to ensure the entire scheduling path is healthy. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By following the checklist and examples in this article, you can automate pod affinity and anti-affinity confidently in your CI/CD pipelines. Remember to keep your manifests versioned, your pipelines idempotent, and your runbooks up to date. Scheduling constraints are part of your application's resilience, and treating them as first-class citizens in your delivery process will pay off in stability and predictability.