Kubernetes Topology Spread Constraints: Advanced Concepts and Practical Implementation

Intro

Kubernetes Topology Spread Constraints are a powerful scheduling feature that ensures pods are evenly distributed across your cluster's topology domains such as zones, regions, or nodes. While the basic concept is straightforward, advanced usage involves nuanced interactions with node affinity, pod anti-affinity, unschedulable nodes, and cluster autoscaling. This article dives deep into the mechanics, provides practical examples with expected outputs, and equips you to troubleshoot common failure modes.

By the end, you will understand how to configure constraints for high availability, avoid scheduling deadlocks, and verify your setup effectively.

Version and Environment Inventory

Before working with topology spread constraints, verify your Kubernetes version and environment capabilities. Topology spread constraints have been stable since v1.19 and continue to evolve. Check your server version:

kubectl version --short

Expected output (example):

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

Ensure the feature gate EvenPodsSpread is enabled (default since v1.18) and that your nodes have appropriate labels for topology keys. Common keys include:

• topology.kubernetes.io/zone
• topology.kubernetes.io/region
• kubernetes.io/hostname

List node labels to confirm:

kubectl get nodes --show-labels

Example snippet:

NAME      STATUS   ROLES    AGE   VERSION   LABELS
node-1    Ready    <none>   10d   v1.27.3   kubernetes.io/hostname=node-1,topology.kubernetes.io/zone=us-east-1a,topology.kubernetes.io/region=us-east-1
node-2    Ready    <none>   10d   v1.27.3   kubernetes.io/hostname=node-2,topology.kubernetes.io/zone=us-east-1b,topology.kubernetes.io/region=us-east-1
node-3    Ready    <none>   10d   v1.27.3   kubernetes.io/hostname=node-3,topology.kubernetes.io/zone=us-east-1c,topology.kubernetes.io/region=us-east-1

For a practical test, start with a minimal deployment and observe its scheduling behavior before adding constraints.

Read-only Observation

Before making changes, capture the current scheduling state:

kubectl get pods -o wide

Example output:

NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
nginx-7d9b7456c4-abcde   1/1     Running   0          5m    10.244.1.5   node-1   <none>           <none>
nginx-7d9b7456c4-fghij   1/1     Running   0          5m    10.244.2.10  node-2   <none>           <none>
nginx-7d9b7456c4-klmno   1/1     Running   0          5m    10.244.3.7   node-3   <none>           <none>

This shows pods evenly distributed across nodes by chance, but without guarantees.

Safe Configuration Path

Understanding the Constraint Object

A topology spread constraint is defined in the pod spec. Here is a basic example spread across zones:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: web
      containers:
      - name: nginx
        image: nginx:1.25

Apply it:

kubectl apply -f web-deployment.yaml

Check the distribution:

kubectl get pods -o wide --selector=app=web

Example output:

NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
web-7d9b7456c4-abcde  1/1     Running   0          10s   10.244.1.6   node-1   <none>           <none>
web-7d9b7456c4-fghij  1/1     Running   0          10s   10.244.2.11  node-2   <none>           <none>
web-7d9b7456c4-klmno  1/1     Running   0          10s   10.244.3.8   node-3   <none>           <none>

Here, each pod landed in a different zone, satisfying maxSkew: 1.

Advanced Parameters

maxSkew is the maximum allowed difference in pod counts between any two topology domains. It must be greater than zero.

whenUnsatisfiable can be DoNotSchedule (hard) or ScheduleAnyway (soft). With DoNotSchedule, if the constraint cannot be met, the pod remains pending. With ScheduleAnyway, the scheduler places it anyway, but skew may be violated.

minDomains (beta in v1.25) specifies a minimum number of eligible domains. The scheduler will not schedule if fewer domains are available, even if skew is satisfied. This is useful for ensuring availability during zone outages.

Example with minDomains:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  minDomains: 3   # requires at least 3 zones
  labelSelector:
    matchLabels:
      app: web

Interaction with Node Affinity and Anti-affinity

Topology spread constraints work alongside node affinity and pod anti-affinity, but the combined effect can be complex. The scheduler considers all constraints, and the most restrictive one dominates.

Example combining with node affinity:

Suppose you want pods spread across zones but only on nodes with SSD. Add node affinity:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: disktype
          operator: In
          values:
          - ssd

Now pods will only be scheduled on nodes with disktype=ssd, and then spread among those nodes' zones.

Interaction with pod anti-affinity:

Pod anti-affinity can prevent pods from co-locating. If both spread and anti-affinity are set, anti-affinity rules may override spread constraints. For instance, if anti-affinity requires each pod on a different node, and spread wants maxSkew=1 across zones, the scheduler must satisfy both. Understand these priorities to avoid unexpected pending pods.

Verification and Diagnostics

Observing Spread with Commands

After applying a deployment with spread constraints, verify the actual distribution:

kubectl get pods -o wide -l app=web

Count pods per node or zone:

kubectl get pods -l app=web -o json | jq -r '.items[].spec.nodeName' | sort | uniq -c

Example output:

1 node-1
1 node-2
1 node-3

To inspect the constraint on a running pod:

kubectl get pod web-7d9b7456c4-abcde -o yaml | grep -A 10 topologySpreadConstraints

Diagnosing Pending Pods

If pods remain pending, describe them:

kubectl describe pod web-7d9b7456c4-abcde

Look for events like:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/3 nodes are available: 3 node(s) didn't match topology spread constraints.

This indicates the constraint cannot be satisfied with current node topology.

Check scheduling decisions using the scheduler's debug logs if necessary:

kubectl logs -n kube-system kube-scheduler-<master-node> | grep -i spread

Failure Modes and Recovery

Scenario 1: Insufficient Domains

If you have fewer topology domains than required by minDomains or maxSkew cannot be met, pods will be pending. Example: you request spread across zones with minDomains: 3, but only two zones exist. The scheduler will not place any pod.

Recovery:

• Reduce minDomains to 2 or remove it.
• Increase maxSkew to allow uneven distribution.
• Add nodes in new zones.
• Temporarily change whenUnsatisfiable to ScheduleAnyway for a soft constraint.

Scenario 2: Skew Violation After Scaling

If you scale down and then up, the skew may exceed maxSkew. The scheduler attempts to rebalance, but it does not evict pods. New pods may be scheduled to reduce skew, but existing pods stay. Over time, skew may normalize.

Example:

Initial: 3 pods across 3 zones, skew=0. Scale down to 1 pod, which remains in zone-1. Scale up to 3: the scheduler sees zone-1 has 1 pod, others have 0. To keep maxSkew=1, it places one pod in zone-2 and one in zone-3. The result is one pod per zone again.

If you scale to 4 replicas, with maxSkew=1, the scheduler can place two pods in one zone and one in each of the others, resulting in skew=1 (acceptable).

Scenario 3: Interaction with Cluster Autoscaler

Topology spread constraints can prevent the cluster autoscaler from scaling up correctly. If a pod is unschedulable due to spread constraints, the autoscaler may not know which node group to scale. To avoid this, ensure node groups correspond to topology zones and use topologySpreadConstraints with ScheduleAnyway or rely on pod anti-affinity.

Scenario 4: Pod Topology Spread with Node Selector

If a node selector restricts scheduling to a subset of nodes, the scheduler considers only those nodes for spread. This can lead to imbalance. For example, if node selector picks only nodes in zone A, all pods go there regardless of spread. Ensure selectors align with topology domains.

Operations Checklist

Use this checklist to safely implement and operate topology spread constraints:

Concrete Example: Multi-Zone Web App

Let's walk through a complete example for a production-like web app with 6 replicas spread across 3 zones, with maxSkew=1 and DoNotSchedule.

1. Label nodes (if not already):

   kubectl label node node-1 topology.kubernetes.io/zone=us-east-1a
   kubectl label node node-2 topology.kubernetes.io/zone=us-east-1b
   kubectl label node node-3 topology.kubernetes.io/zone=us-east-1c

1. Create a namespace:

   kubectl create namespace webapp

1. Define deployment manifest webapp-deployment.yaml:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: webapp
     namespace: webapp
   spec:
     replicas: 6
     selector:
       matchLabels:
         app: webapp
     template:
       metadata:
         labels:
           app: webapp
       spec:
         topologySpreadConstraints:
         - maxSkew: 1
           topologyKey: topology.kubernetes.io/zone
           whenUnsatisfiable: DoNotSchedule
           labelSelector:
             matchLabels:
               app: webapp
         containers:
         - name: app
           image: nginx:1.25
           ports:
           - containerPort: 80

1. Apply and wait:

   kubectl apply -f webapp-deployment.yaml
   kubectl rollout status deployment/webapp -n webapp

1. Check distribution:

   kubectl get pods -n webapp -o wide -l app=webapp

Expected output: 2 pods per zone, all Running.

1. Verify skew using a small script:

   kubectl get pods -n webapp -l app=webapp -o json | jq -r '.items[] | .spec.nodeName' | \
   xargs -I {} kubectl get node {} --show-labels | grep topology.kubernetes.io/zone | \
   awk -F'topology.kubernetes.io/zone=' '{print $2}' | cut -d',' -f1 | sort | uniq -c

Example output:

   2 us-east-1a
   2 us-east-1b
   2 us-east-1c

1. Simulate zone failure by cordoning one zone's nodes:

   kubectl cordon node-1   # assuming node-1 in us-east-1a

Existing pods continue running. If you scale up to 9 replicas, the scheduler will place new pods in the remaining two zones, but may not exceed maxSkew of 1 if possible. However, with only two zones, the max skew may still be 1 if evenly distributed.

1. Uncordon to restore:

   kubectl uncordon node-1

This checklist and example demonstrate operational readiness.

Conclusion

Kubernetes Topology Spread Constraints provide fine-grained control over pod distribution, enhancing availability and resource utilization. By understanding the parameters and interactions with other scheduling features, you can design robust multi-zone deployments. Always verify with real commands, anticipate failure modes, and have a rollback plan. Start with a small test, observe the scheduler's behavior, and scale up with confidence.