## Intro Kubernetes Topology Spread Constraints are a powerful scheduling feature that ensures pods are evenly distributed across your cluster's topology domains such as zones, regions, or nodes. While the basic concept is straightforward, advanced usage involves nuanced interactions with node affinity, pod anti-affinity, unschedulable nodes, and cluster autoscaling. This article dives deep into the mechanics, provides practical examples with expected outputs, and equips you to troubleshoot common failure modes. By the end, you will understand how to configure constraints for high availability, avoid scheduling deadlocks, and verify your setup effectively. ## Version and Environment Inventory Before working with topology spread constraints, verify your Kubernetes version and environment capabilities. Topology spread constraints have been stable since v1.19 and continue to evolve. Check your server version: kubectl version --short Expected output (example): Client Version: v1.27.3 Kustomize Version: v5.0.1 Server Version: v1.27.3 Ensure the feature gate EvenPodsSpread is enabled (default since v1.18) and that your nodes have appropriate labels for topology keys. Common keys include: - topology.kubernetes.io/zone - topology.kubernetes.io/region - kubernetes.io/hostname List node labels to confirm: kubectl get nodes --show-labels Example snippet: NAME STATUS ROLES AGE VERSION LABELS node-1 Ready 10d v1.27.3 kubernetes.io/hostname=node-1,topology.kubernetes.io/zone=us-east-1a,topology.kubernetes.io/region=us-east-1 node-2 Ready 10d v1.27.3 kubernetes.io/hostname=node-2,topology.kubernetes.io/zone=us-east-1b,topology.kubernetes.io/region=us-east-1 node-3 Ready 10d v1.27.3 kubernetes.io/hostname=node-3,topology.kubernetes.io/zone=us-east-1c,topology.kubernetes.io/region=us-east-1 For a practical test, start with a minimal deployment and observe its scheduling behavior before adding constraints. ### Read-only Observation Before making changes, capture the current scheduling state: kubectl get pods -o wide Example output: NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-7d9b7456c4-abcde 1/1 Running 0 5m 10.244.1.5 node-1 nginx-7d9b7456c4-fghij 1/1 Running 0 5m 10.244.2.10 node-2 nginx-7d9b7456c4-klmno 1/1 Running 0 5m 10.244.3.7 node-3 This shows pods evenly distributed across nodes by chance, but without guarantees.

Quick check 1 of 2 What does the 'minDomains' field in a topology spread constraint specify? - The maximum number of pods that can be scheduled in a single domain - The minimum number of eligible domains that must have nodes matching the node selector - The minimum number of pods that must be placed in each domain - The maximum skew allowed between domains

Reference passage [5] states: 'minDomains indicates a minimum number of eligible domains. A domain is a particular instance of a topology. An eligible domain is a domain whose nodes match the node selector.'

## Safe Configuration Path ### Understanding the Constraint Object A topology spread constraint is defined in the pod spec. Here is a basic example spread across zones: apiVersion: apps/v1 kind: Deployment metadata: name: web spec: replicas: 3 selector: matchLabels: app: web template: metadata: labels: app: web spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: web containers: - name: nginx image: nginx:1.25 Apply it: kubectl apply -f web-deployment.yaml Check the distribution: kubectl get pods -o wide --selector=app=web Example output: NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES web-7d9b7456c4-abcde 1/1 Running 0 10s 10.244.1.6 node-1 web-7d9b7456c4-fghij 1/1 Running 0 10s 10.244.2.11 node-2 web-7d9b7456c4-klmno 1/1 Running 0 10s 10.244.3.8 node-3 Here, each pod landed in a different zone, satisfying maxSkew: 1 . ### Advanced Parameters maxSkew is the maximum allowed difference in pod counts between any two topology domains. It must be greater than zero. whenUnsatisfiable can be DoNotSchedule (hard) or ScheduleAnyway (soft). With DoNotSchedule , if the constraint cannot be met, the pod remains pending. With ScheduleAnyway , the scheduler places it anyway, but skew may be violated. minDomains (beta in v1.25) specifies a minimum number of eligible domains. The scheduler will not schedule if fewer domains are available, even if skew is satisfied. This is useful for ensuring availability during zone outages. Example with minDomains : topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule minDomains: 3 # requires at least 3 zones labelSelector: matchLabels: app: web ### Interaction with Node Affinity and Anti-affinity Topology spread constraints work alongside node affinity and pod anti-affinity, but the combined effect can be complex. The scheduler considers all constraints, and the most restrictive one dominates. Example combining with node affinity: Suppose you want pods spread across zones but only on nodes with SSD. Add node affinity: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: disktype operator: In values: - ssd Now pods will only be scheduled on nodes with disktype=ssd , and then spread among those nodes' zones. Interaction with pod anti-affinity: Pod anti-affinity can prevent pods from co-locating. If both spread and anti-affinity are set, anti-affinity rules may override spread constraints. For instance, if anti-affinity requires each pod on a different node, and spread wants maxSkew=1 across zones, the scheduler must satisfy both. Understand these priorities to avoid unexpected pending pods. ## Verification and Diagnostics ### Observing Spread with Commands After applying a deployment with spread constraints, verify the actual distribution: kubectl get pods -o wide -l app=web Count pods per node or zone: kubectl get pods -l app=web -o json | jq -r '.items[].spec.nodeName' | sort | uniq -c Example output: 1 node-1 1 node-2 1 node-3 To inspect the constraint on a running pod: kubectl get pod web-7d9b7456c4-abcde -o yaml | grep -A 10 topologySpreadConstraints ### Diagnosing Pending Pods If pods remain pending, describe them: kubectl describe pod web-7d9b7456c4-abcde Look for events like: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m default-scheduler 0/3 nodes are available: 3 node(s) didn't match topology spread constraints. This indicates the constraint cannot be satisfied with current node topology. Check scheduling decisions using the scheduler's debug logs if necessary: kubectl logs -n kube-system kube-scheduler- | grep -i spread Quick check 2 of 2 What happens if the intersection of multiple topology spread constraints returns an empty set? - The scheduler places the pod on the node with the least pods - The pod remains in Pending state - The scheduler evicts an existing pod to make room - The pod is scheduled with a relaxed maxSkew

Reference passage [4] shows that if the intersection of constraints returns an empty set, the scheduler cannot place the Pod, leaving it in Pending state.

## Failure Modes and Recovery ### Scenario 1: Insufficient Domains If you have fewer topology domains than required by minDomains or maxSkew cannot be met, pods will be pending. Example: you request spread across zones with minDomains: 3 , but only two zones exist. The scheduler will not place any pod. Recovery: - Reduce minDomains to 2 or remove it. - Increase maxSkew to allow uneven distribution. - Add nodes in new zones. - Temporarily change whenUnsatisfiable to ScheduleAnyway for a soft constraint. ### Scenario 2: Skew Violation After Scaling If you scale down and then up, the skew may exceed maxSkew . The scheduler attempts to rebalance, but it does not evict pods. New pods may be scheduled to reduce skew, but existing pods stay. Over time, skew may normalize. Example: Initial: 3 pods across 3 zones, skew=0. Scale down to 1 pod, which remains in zone-1. Scale up to 3: the scheduler sees zone-1 has 1 pod, others have 0. To keep maxSkew=1, it places one pod in zone-2 and one in zone-3. The result is one pod per zone again. If you scale to 4 replicas, with maxSkew=1, the scheduler can place two pods in one zone and one in each of the others, resulting in skew=1 (acceptable). ### Scenario 3: Interaction with Cluster Autoscaler Topology spread constraints can prevent the cluster autoscaler from scaling up correctly. If a pod is unschedulable due to spread constraints, the autoscaler may not know which node group to scale. To avoid this, ensure node groups correspond to topology zones and use topologySpreadConstraints with ScheduleAnyway or rely on pod anti-affinity. ### Scenario 4: Pod Topology Spread with Node Selector If a node selector restricts scheduling to a subset of nodes, the scheduler considers only those nodes for spread. This can lead to imbalance. For example, if node selector picks only nodes in zone A, all pods go there regardless of spread. Ensure selectors align with topology domains. ## Operations Checklist Use this checklist to safely implement and operate topology spread constraints:

Step	Action	Command / Verification	Example
1	Verify Kubernetes version and feature gates	`kubectl version --short`	Server v1.27+, `EvenPodsSpread` enabled
2	Inspect node topology labels	`kubectl get nodes --show-labels`	Nodes have `topology.kubernetes.io/zone` labels
3	Apply a test deployment without constraints	`kubectl apply -f test.yaml`	Pods schedule normally
4	Add spread constraints to a copy	Modify manifest, then `kubectl apply -f constrained.yaml`	Pods spread as expected
5	Verify actual distribution	`kubectl get pods -o wide -l app=test`	Pods evenly distributed across zones
6	Test failure scenarios	Scale down/up, simulate zone loss	Observe skew and pending behavior
7	Document rollback plan	Keep original manifest: `kubectl apply -f original.yaml`	Ready to revert if issues arise
8	Monitor over time	`kubectl get pods -o wide` periodically or use kube-state-metrics	Skew remains within bounds

### Concrete Example: Multi-Zone Web App Let's walk through a complete example for a production-like web app with 6 replicas spread across 3 zones, with maxSkew=1 and DoNotSchedule . - Label nodes (if not already): kubectl label node node-1 topology.kubernetes.io/zone=us-east-1a kubectl label node node-2 topology.kubernetes.io/zone=us-east-1b kubectl label node node-3 topology.kubernetes.io/zone=us-east-1c - Create a namespace: kubectl create namespace webapp - Define deployment manifest webapp-deployment.yaml : apiVersion: apps/v1 kind: Deployment metadata: name: webapp namespace: webapp spec: replicas: 6 selector: matchLabels: app: webapp template: metadata: labels: app: webapp spec: topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: webapp containers: - name: app image: nginx:1.25 ports: - containerPort: 80 - Apply and wait: kubectl apply -f webapp-deployment.yaml kubectl rollout status deployment/webapp -n webapp - Check distribution: kubectl get pods -n webapp -o wide -l app=webapp Expected output: 2 pods per zone, all Running. - Verify skew using a small script: kubectl get pods -n webapp -l app=webapp -o json | jq -r '.items[] | .spec.nodeName' | \ xargs -I {} kubectl get node {} --show-labels | grep topology.kubernetes.io/zone | \ awk -F'topology.kubernetes.io/zone=' '{print $2}' | cut -d',' -f1 | sort | uniq -c Example output: 2 us-east-1a 2 us-east-1b 2 us-east-1c - Simulate zone failure by cordoning one zone's nodes: kubectl cordon node-1 # assuming node-1 in us-east-1a Existing pods continue running. If you scale up to 9 replicas, the scheduler will place new pods in the remaining two zones, but may not exceed maxSkew of 1 if possible. However, with only two zones, the max skew may still be 1 if evenly distributed. - Uncordon to restore: kubectl uncordon node-1 This checklist and example demonstrate operational readiness. ## Conclusion Kubernetes Topology Spread Constraints provide fine-grained control over pod distribution, enhancing availability and resource utilization. By understanding the parameters and interactions with other scheduling features, you can design robust multi-zone deployments. Always verify with real commands, anticipate failure modes, and have a rollback plan. Start with a small test, observe the scheduler's behavior, and scale up with confidence.