Kubernetes Pod Affinity and Anti-Affinity: CI/CD Automation with Practical Examples

Intro

Kubernetes Pod Affinity and Anti-Affinity rules give you fine-grained control over pod scheduling based on other pods' labels. When combined with CI/CD automation, these rules help you enforce high availability, resource locality, and failure isolation in every deployment. This article provides a practical, command-driven workflow for integrating pod affinity and anti-affinity into your pipelines, from version validation to automated verification and rollback.

This guide is intended for developers, DevOps consultants, and technical startup teams who manage Kubernetes clusters and want to make scheduling constraints part of their delivery process. We focus on operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

You will learn how to:

• Check your Kubernetes version and current scheduling setup.
• Define pod affinity and anti-affinity in deployment manifests.
• Validate scheduling constraints locally before merging.
• Automate deployment with safe rollout strategies.
• Diagnose common failures such as unschedulable pods or topology mismatches.
• Rollback safely when a change breaks pod placement.

By the end, you will have a repeatable pipeline that makes scheduling behavior predictable and auditable.

Version and Environment Inventory

Before applying any affinity or anti-affinity rule, confirm that your cluster supports the features you intend to use. Pod affinity and anti-affinity have been stable since Kubernetes 1.6, but topologyKey behavior and namespaceSelector options have evolved. For production pipelines, use Kubernetes 1.24 or later where the feature is mature and well-documented.

Check Kubernetes version and API availability

Run the following command to check the server version:

kubectl version --short

Expected output includes both client and server versions, for example:

Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.7

If the server version is below 1.24, test all affinity rules in a staging cluster first because some edge cases around matchLabelKeys and mismatchLabelKeys may behave differently.

Verify scheduler and existing pod distribution

Take a read-only snapshot of the current pod placement before any change. This helps you compare after deployment and detect unintended movement. Run:

kubectl get pods -o wide --all-namespaces | tee pods-before.txt

The -o wide output includes node names and pod IPs, which are essential for understanding scheduling decisions. Save the output with a timestamp in the filename for auditability:

kubectl get pods -o wide --all-namespaces > pods-before-$(date +%Y%m%d-%H%M%S).txt

If you have many namespaces, limit the output to your application namespace:

kubectl get pods -o wide -n my-app > pods-before-$(date +%Y%m%d-%H%M%S).txt

Inspect existing affinity rules

Check whether your current deployments already have affinity or anti-affinity configured:

kubectl get deployments -n my-app -o custom-columns=NAME:.metadata.name,AFFINITY:.spec.template.spec.affinity

This prints a table like:

NAME        AFFINITY
web         <none>
api         <none>
worker      map[nodeAffinity:map[...] podAntiAffinity:map[...]]

If a deployment already uses affinity, examine its YAML to understand its current constraints:

kubectl get deployment worker -n my-app -o yaml | grep -A 50 affinity

List nodes and topology labels

Affinity and anti-affinity rules often rely on node topology labels such as kubernetes.io/hostname, topology.kubernetes.io/zone, or topology.kubernetes.io/region. List your nodes and their labels to know what topology keys are available:

kubectl get nodes --show-labels

Expected output excerpt:

NAME                 STATUS   ROLES           AGE   VERSION   LABELS
node-1               Ready    control-plane   21d   v1.28.7   kubernetes.io/hostname=node-1,topology.kubernetes.io/zone=us-east-1a,topology.kubernetes.io/region=us-east-1
node-2               Ready    <none>          21d   v1.28.7   kubernetes.io/hostname=node-2,topology.kubernetes.io/zone=us-east-1b,topology.kubernetes.io/region=us-east-1
node-3               Ready    <none>          21d   v1.28.7   kubernetes.io/hostname=node-3,topology.kubernetes.io/zone=us-east-1c,topology.kubernetes.io/region=us-east-1

For multi-zone clusters, using topology.kubernetes.io/zone as a topology key spreads pods across failure domains. For single-node development clusters, only kubernetes.io/hostname makes sense.

Confirm required CRDs and admission controllers

If you plan to use advanced scheduling features like pod topology spread constraints or matchLabelKeys, ensure the relevant feature gates are enabled. Check the scheduler configuration:

kubectl -n kube-system get pod kube-scheduler-<node-name> -o yaml | grep feature-gates

Most managed Kubernetes services (EKS, GKE, AKS) enable these by default, but self-managed clusters may need manual flag toggling.

Safe Configuration Path

The goal is to define pod affinity and anti-affinity rules that match your availability and performance goals, without making the cluster impossible to schedule. Start with a minimal manifest, validate it locally, then expand scope.

Understand affinity and anti-affinity types

Kubernetes supports two kinds of pod affinity:

• requiredDuringSchedulingIgnoredDuringExecution: The rule must be satisfied for the pod to be scheduled. If no node meets the rule, the pod remains unschedulable.
• preferredDuringSchedulingIgnoredDuringExecution: The scheduler tries to satisfy the rule but does not guarantee it. This is useful for soft constraints.

Similarly, anti-affinity rules can be required or preferred. Anti-affinity prevents pods from being co-located on the same topology domain (like the same node or zone).

Example: spread web pods across nodes with anti-affinity

Suppose you have a web deployment with 3 replicas, and you want to ensure each replica runs on a different node for resilience. Add the following podAntiAffinity under the pod spec:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - web
            topologyKey: kubernetes.io/hostname
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80

Here, the rule says: do not schedule this pod on a node that already has a pod with label app=web. The topologyKey: kubernetes.io/hostname scopes the rule to a single node.

Validate manifest syntax and dry-run

Before applying, validate the YAML syntax and simulate the change with --dry-run=client:

kubectl apply -f web-deployment.yaml --dry-run=client

Expected output:

deployment.apps/web created (dry run)

For a server-side dry-run that checks admission controllers, use --dry-run=server:

kubectl apply -f web-deployment.yaml --dry-run=server

The server-side dry-run will catch issues like invalid topology keys or label selector mismatches.

Apply and observe scheduling

Apply the manifest and watch the pods being scheduled:

kubectl apply -f web-deployment.yaml
kubectl rollout status deployment/web -n my-app

Expected output:

deployment "web" successfully rolled out

Then check pod distribution:

kubectl get pods -n my-app -l app=web -o wide

You should see one pod per node, like:

NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
web-5f7c9b6c7-abcde   1/1     Running   0          10s   10.244.1.5    node-1   <none>           <none>
web-5f7c9b6c7-fghij   1/1     Running   0          10s   10.244.2.7    node-2   <none>           <none>
web-5f7c9b6c7-klmno   1/1     Running   0          10s   10.244.3.9    node-3   <none>           <none>

Use preferred anti-affinity for flexibility

Required anti-affinity can cause scheduling failures if the cluster does not have enough nodes. For example, if you have only two nodes and 3 replicas, required anti-affinity on hostname will leave one pod pending. Use preferredDuringSchedulingIgnoredDuringExecution to allow soft spreading:

      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: app
                  operator: In
                  values:
                  - web
              topologyKey: kubernetes.io/hostname

Prefer soft anti-affinity for most CI/CD use cases where availability matters more than strict placement.

Add pod affinity to co-locate services

Sometimes you want certain pods to run near each other, like a cache and its consumers. Use podAffinity to place pods on the same node or zone:

      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - redis-cache
            topologyKey: kubernetes.io/hostname

This forces the new pod to schedule on a node where a pod with app=redis-cache already runs.

Validate selector matches existing pods

If the selector matches no pods, the affinity rule may never be satisfied. Check the labels of your intended target pods:

kubectl get pods -n my-app -l app=redis-cache --show-labels

Output should show the label app=redis-cache. If not, adjust your manifest labels or selector.

Use namespaceSelector for cross-namespace affinity

By default, affinity only considers pods in the same namespace. To include pods from other namespaces, add namespaceSelector:

          podAffinityTerm:
            labelSelector:
              matchLabels:
                app: shared-service
            namespaceSelector:
              matchLabels:
                environment: staging
            topologyKey: kubernetes.io/hostname

This rule looks for pods with app=shared-service in namespaces labeled environment=staging. Ensure the namespace label is set:

kubectl label namespace shared env=staging --overwrite
kubectl get namespace shared --show-labels

Verification and Diagnostics

After deploying with affinity rules, verify that pods are scheduled as expected and that the rules did not introduce new failures. Use a combination of kubectl commands and event inspection.

Check pod status and events

First, list all pods in the namespace with wide output:

kubectl get pods -n my-app -o wide

If any pod is Pending, investigate scheduling events:

kubectl describe pod <pending-pod-name> -n my-app

Look for events like:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  40s   default-scheduler  0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules.

This tells you that the anti-affinity rule is too strict or the cluster lacks nodes.

Verify that affinity rules are attached

Confirm that the running pods have the expected affinity spec:

kubectl get pod <pod-name> -n my-app -o jsonpath='{.spec.affinity}' | jq .

Expected output includes the podAntiAffinity or podAffinity block you defined.

Test rollout in a staging namespace

Never merge affinity changes directly to production. Use a staging namespace or a canary deployment to verify behavior with real traffic. For example, create a separate namespace staging and apply your manifest there:

kubectl create namespace staging
kubectl apply -f web-deployment.yaml -n staging

Then run the same checks in staging. If everything passes, propagate to production via your CI/CD pipeline.

Automated verification with kubectl in CI

In your CI pipeline, add a verification step that runs after kubectl apply. For example, in a GitHub Actions workflow:

- name: Verify pod distribution
  run: |
    kubectl get pods -n my-app -l app=web -o json | jq -e '.items | length == 3'
    kubectl get pods -n my-app -l app=web -o json | jq -e '[.items[].spec.nodeName] | unique | length == 3'

The first command checks that 3 replicas are running. The second checks that they are spread across 3 different nodes (assuming a 3-node cluster). If either fails, the step fails.

Use pod topology spread constraints for additional control

Pod topology spread constraints provide a more granular way to spread pods across nodes, zones, or other domains. Combine them with anti-affinity for robust placement. Example:

      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: web

This constraint ensures that the number of pods per zone differs by at most 1. If the cluster cannot satisfy it, the pod remains pending.

Monitor scheduler logs for debugging

If you suspect the scheduler is misbehaving, check its logs:

kubectl logs -n kube-system -l component=kube-scheduler --tail=100

Look for lines mentioning predicate or priority failures related to your pods. This requires access to the control plane, which may not be available on managed services. In that case, rely on pod events.

Failure Modes and Recovery

Even with careful planning, affinity and anti-affinity rules can cause deployment failures. Understand common failure modes and how to recover safely, using rollback strategies integrated into CI/CD.

Failure mode 1: Pods stuck in Pending due to unsatisfiable anti-affinity

Symptom: New pods remain Pending and events show FailedScheduling with didn't match pod anti-affinity rules.

Cause: Required anti-affinity cannot be satisfied because the cluster lacks sufficient nodes or other pods occupy all eligible nodes.

Recovery:

1. Check current node count and pod placement:

kubectl get nodes
kubectl get pods -n my-app -l app=web -o wide

1. If the cluster has fewer nodes than the required replica count, you must either scale up nodes or relax the rule to preferredDuringSchedulingIgnoredDuringExecution.

1. Apply the relaxed manifest:

kubectl apply -f web-deployment-relaxed.yaml -n my-app

1. Verify that pods are now scheduled:

kubectl rollout status deployment/web -n my-app

Failure mode 2: Affinity rule referencing a non-existent pod label

Symptom: Pods are scheduled, but not co-located or spread as expected. No error events.

Cause: The labelSelector in the affinity term does not match any existing pods, so the rule has no effect. This often happens due to a typo in the label key or value.

Diagnosis:

kubectl get pods -n my-app --show-labels

Compare with your manifest's matchLabels. If mismatched, fix the selector and re-apply.

Recovery: Update the deployment manifest and roll out a new revision:

kubectl apply -f web-deployment-fixed.yaml -n my-app
kubectl rollout restart deployment/web -n my-app

Failure mode 3: Topology key invalid for cluster topology

Symptom: Scheduler cannot find nodes with the specified topology key, or the key does not represent a meaningful failure domain.

Example: Using topology.kubernetes.io/zone on a single-zone cluster, or using a custom label that nodes do not have.

Diagnosis: List node labels:

kubectl get nodes --show-labels | grep topology

If the key is absent, the rule will never match. Choose a valid key like kubernetes.io/hostname for single-zone clusters.

Recovery: Modify the topology key in the manifest and re-apply.

Failure mode 4: CI/CD pipeline applied changes without verification

Symptom: A pipeline step applied a manifest, but no verification ran, and the production deployment is now broken.

Cause: Missing automated checks after kubectl apply.

Recovery: Rollback the deployment to the previous revision using kubectl rollout undo:

kubectl rollout undo deployment/web -n my-app

Check rollout status:

kubectl rollout status deployment/web -n my-app

This reverts to the last known good configuration, including previous affinity settings.

Failure mode 5: Cross-namespace affinity failing due to namespace labels

Symptom: Pods with namespaceSelector cannot be scheduled, and events show no matching namespaces.

Cause: The target namespaces do not have the required labels.

Recovery: Add the labels:

kubectl label namespace shared env=staging --overwrite

Verify:

kubectl get namespace shared --show-labels

Then re-apply the deployment.

Automating rollback in CI/CD

In your pipeline, add a rollback step that triggers automatically if verification fails. For example:

- name: Apply manifest
  run: kubectl apply -f web-deployment.yaml -n my-app

- name: Verify rollout
  run: kubectl rollout status deployment/web -n my-app --timeout=120s

- name: Rollback on failure
  if: failure()
  run: |
    echo "Rolling back to previous revision"
    kubectl rollout undo deployment/web -n my-app
    kubectl rollout status deployment/web -n my-app

This ensures that any failed deployment is automatically reverted.

Document recovery runbooks

For each deployment with complex scheduling, maintain a runbook that includes:

• The command to capture current pod distribution.
• The command to inspect scheduling events.
• The exact rollback command.
• Contact information for the on-call engineer.

Store the runbook in your repository next to the manifests to keep it versioned.

Operations Checklist

Use this checklist before and after every CI/CD run that modifies pod affinity or anti-affinity. It enforces observation, verification, and recoverability.

Pre-deployment checklist

• [ ] Confirm Kubernetes server version supports the intended features: kubectl version --short
• [ ] Snapshot current pod distribution: kubectl get pods -o wide --all-namespaces > pods-before-$(date +%Y%m%d-%H%M%S).txt
• [ ] List available node topology labels: kubectl get nodes --show-labels
• [ ] Validate manifest with server-side dry-run: kubectl apply -f manifest.yaml --dry-run=server
• [ ] Check that label selectors match existing pods: kubectl get pods -n my-app --show-labels
• [ ] Ensure topology keys are valid for cluster topology
• [ ] Confirm that required anti-affinity does not exceed available capacity
• [ ] Set up a staging namespace for pre-production testing
• [ ] Define rollback command in the pipeline: kubectl rollout undo deployment/<name>
• [ ] Add verification steps to the pipeline that check pod placement and replica count

Post-deployment checklist

• [ ] Check rollout status: kubectl rollout status deployment/<name> -n my-app
• [ ] Inspect pod events for scheduling warnings: kubectl describe pod <pod-name>
• [ ] Confirm pod distribution matches expectations: kubectl get pods -o wide -n my-app
• [ ] Compare pod distribution to the pre-deployment snapshot
• [ ] If any pods are Pending, diagnose with kubectl describe pod and fix or rollback
• [ ] Record the new pod distribution snapshot for future comparison
• [ ] Update the runbook with any new failure modes encountered
• [ ] Notify the team of the deployment status

Sample pre-deployment script

Here is a combined script you can run locally before pushing changes:

#!/bin/bash
set -e

echo "Kubernetes version:"
kubectl version --short

echo "Snapshotting current pods..."
kubectl get pods -o wide --all-namespaces > pods-before-$(date +%Y%m%d-%H%M%S).txt

echo "Listing node labels:"
kubectl get nodes --show-labels

echo "Validating manifest dry-run..."
kubectl apply -f web-deployment.yaml --dry-run=server

echo "All pre-deployment checks passed."

Run it and verify no errors before applying.

Conclusion

Kubernetes Pod Affinity and Anti-Affinity are powerful tools for controlling pod placement, but they must be integrated into CI/CD with the same rigor as any other infrastructure change. This article provided a practical path: start with version and environment checks, define safe configuration, verify automatically, and prepare for failures with documented rollback.

Copying commands without checking prerequisites and expected output is not an operations procedure. Instead, choose one low-risk verification for your environment, record the current state, run the documented check, and compare the result with the expected signal. Review dependencies such as node affinity, the kube-scheduler, and the pods themselves to ensure the entire scheduling path is healthy.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By following the checklist and examples in this article, you can automate pod affinity and anti-affinity confidently in your CI/CD pipelines.

Remember to keep your manifests versioned, your pipelines idempotent, and your runbooks up to date. Scheduling constraints are part of your application's resilience, and treating them as first-class citizens in your delivery process will pay off in stability and predictability.