Kubernetes Topology Spread Constraints: A Practical Local Lab Setup

Intro

Kubernetes scheduling is flexible, but ensuring high availability across failure domains requires careful pod placement. Topology spread constraints let you control how pods are spread across your cluster's topology, such as nodes, zones, or regions. This guide provides a practical, step-by-step approach to setting up a safe local lab environment for testing and learning topology spread constraints. By experimenting locally, you can understand the behavior, troubleshoot issues, and build confidence before deploying to production.

A well-designed local lab reduces rework by letting you test scheduling policies in isolation. The first useful pilot should be narrow, measurable, and easy to inspect locally before deployment. This article walks through creating a multi-node cluster with kind (Kubernetes in Docker), labeling nodes to simulate zones, deploying a workload with topology spread constraints, and verifying the distribution.

In this article, you will:

• Set up a multi-node Kubernetes cluster using kind that simulates multiple zones.
• Label nodes with topology keys such as topology.kubernetes.io/zone.
• Deploy a sample application with topology spread constraints.
• Verify pod distribution and diagnose common issues.
• Learn failure modes and recovery procedures.

All commands and examples are designed to work on a typical developer workstation with Docker installed.

Version and Environment Inventory

Before diving into topology spread constraints, ensure your environment is properly set up. This section covers the prerequisites and the specific versions used in this guide.

Prerequisites

You need the following tools installed on your local machine:

• Docker: Required for kind. Install Docker Desktop (Windows/Mac) or Docker Engine (Linux).
• kubectl: Kubernetes command-line tool. Ensure it matches your cluster version within one minor version.
• kind: Kubernetes in Docker. Install from the official releases.

Version Matrix

The commands in this guide are tested with the following versions, but the concepts apply broadly:

If your versions differ, adjust accordingly. The key is that Kubernetes supports topologySpreadConstraints in Pod specs (introduced in v1.19, stable in v1.25).

Cluster Topology Design

For meaningful testing, we need multiple nodes with different topology labels. We'll create a kind cluster with three worker nodes, each simulating a different zone:

• worker1 in zone zone-a
• worker2 in zone zone-b
• worker3 in zone zone-c

Each node will also have a node-role.kubernetes.io/worker label for scheduling regular pods (optional). The cluster will have one control-plane node.

Setting Up the Cluster with kind

Create a configuration file kind-config.yaml:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
  labels:
    topology.kubernetes.io/zone: zone-a
- role: worker
  labels:
    topology.kubernetes.io/zone: zone-b
- role: worker
  labels:
    topology.kubernetes.io/zone: zone-c

Then create the cluster:

kind create cluster --name topo-lab --config kind-config.yaml

Expected output:

Creating cluster "topo-lab" ...
 ✓ Ensuring node image (kindest/node: v1.27.1) 🖼
 ✓ Preparing nodes 📦 📦 📦 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Joining worker nodes 🚜
Set kubectl context to "kind-topo-lab"
You can now use your cluster with:

kubectl cluster-info --context kind-topo-lab

Verify the nodes and their labels:

kubectl get nodes --show-labels

Expected output (abbreviated):

NAME                 STATUS   ROLES           AGE   VERSION   LABELS
kind-control-plane   Ready    control-plane   1m    v1.27.1   ...
kind-worker          Ready    <none>          1m    v1.27.1   topology.kubernetes.io/zone=zone-a,...
kind-worker2         Ready    <none>          1m    v1.27.1   topology.kubernetes.io/zone=zone-b,...
kind-worker3         Ready    <none>          1m    v1.27.1   topology.kubernetes.io/zone=zone-c,...

Note: kind names the workers as kind-worker, kind-worker2, kind-worker3 by default. We have successfully labeled them.

Understanding Topology Spread Constraints

A topology spread constraint specifies how pods should be distributed across a topology domain. The key fields in a Pod spec are:

• topologyKey: the label key of nodes that defines a domain (e.g., topology.kubernetes.io/zone).
• maxSkew: the maximum allowed difference in the number of matching pods between any two topology domains.
• whenUnsatisfiable: DoNotSchedule (hard constraint) or ScheduleAnyway (soft constraint).
• labelSelector: selects the pods to count for skew calculation (usually pods with the same labels as the pod itself).

A similar mechanism, pod topology spread constraints, evolved from podAntiAffinity rules but offers more fine-grained control over pod distribution. The scheduler attempts to honor the constraint while also considering other scheduling requirements like node resources, taints, and affinities.

Selecting a Pilot Scenario

We'll deploy a simple Deployment with three replicas, each with a topology spread constraint to spread across the three zones with maxSkew: 1. This is a common high-availability setup.

Create a file app.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      topologySpreadConstraints:
      - maxSkew: 1
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: web
      containers:
      - name: nginx
        image: nginx:1.25
        ports:
        - containerPort: 80

Apply the deployment:

kubectl apply -f app.yaml

Expected output:

deployment.apps/web created

Check pod distribution:

kubectl get pods -o wide

Expected output (similar):

NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
web-5c8b6f7c9d-abcde   1/1     Running   0          10s   10.244.1.2   kind-worker   <none>           <none>
web-5c8b6f7c9d-fghij   1/1     Running   0          10s   10.244.2.2   kind-worker2  <none>           <none>
web-5c8b6f7c9d-klmno   1/1     Running   0          10s   10.244.3.2   kind-worker3  <none>           <none>

We observe one pod per zone, satisfying the constraint.

Combining Multiple Constraints

In production, you might want to spread pods both across zones and across nodes within each zone. You can define multiple topology spread constraints. However, be cautious: increasing the number of constraints can make scheduling impossible if the cluster lacks enough diversity. For example, if you require maxSkew=1 on both zone and hostname with only three nodes total, some constraints will conflict. Use ScheduleAnyway if you want best-effort distribution.

Verification and Diagnostics

After deploying, verification is crucial. This section provides commands and expected outputs to confirm that topology spread constraints are working as intended.

Checking Pod Distribution

The primary verification is pod placement. Use kubectl get pods -o wide to see node assignment. To count pods per zone, you can use kubectl with custom columns or jq.

Example with kubectl and awk:

kubectl get pods -l app=web -o wide | awk 'NR>1 {print $7}' | sort | uniq -c

Expected output:

      1 kind-worker
      1 kind-worker2
      1 kind-worker3

This shows one pod on each node (and thus each zone).

Describing the Deployment for Events

Check the Deployment events for scheduling decisions:

kubectl describe deployment web

Look for the Events section. If there were any failures, they would appear here. For successful scheduling, you may see no relevant events.

Inspecting Pod Annotations

When a pod is scheduled, the scheduler records the topology spread constraints in the pod's annotations (since Kubernetes 1.20). You can inspect:

kubectl get pod <pod-name> -o jsonpath='{.metadata.annotations}'

Expected output includes an annotation like:

{"kubernetes.io/topology-spread-constraints":"[{\"maxSkew\":1,\"topologyKey\":\"topology.kubernetes.io/zone\",\"whenUnsatisfiable\":\"DoNotSchedule\",\"labelSelector\":{\"matchLabels\":{\"app\":\"web\"}}}]"}

This confirms the constraint was applied.

Testing Skew by Scaling

To verify skew enforcement, scale the deployment to 4 replicas:

kubectl scale deployment web --replicas=4

Then check pod distribution. With maxSkew: 1 and three zones, the scheduler tries to balance pods, but with 4 pods, one zone will have 2 pods, others 1 each. That's acceptable because the skew is at most 1. However, if you scale to 5, the scheduler may not be able to place all pods because the constraint cannot be satisfied (if whenUnsatisfiable: DoNotSchedule). The extra pod will remain Pending.

Check pending pods:

kubectl get pods -l app=web --field-selector=status.phase=Pending

Expected output shows a pending pod. Describe it for details:

kubectl describe pod <pending-pod-name>

In the Events, you'll see a message like:

0/4 nodes are available: 1 node(s) didn't match pod topology spread constraints, 3 node(s) had no free ports.

This indicates the constraint is being enforced, even though the message may mention ports due to node resource calculations.

Common Pitfalls and Avoidance

Understanding what can go wrong is half the battle. Here are common pitfalls with topology spread constraints and how to avoid them.

Pitfall 1: Missing or Incorrect Node Labels

Why it happens: Topology spread constraints rely on node labels to define domains. If nodes lack the required label (e.g., topology.kubernetes.io/zone), the scheduler may not consider them, or it may treat them as a single domain, leading to skewed distribution.

How to avoid: Always verify node labels before deploying. Use kubectl get nodes --show-labels and ensure all intended nodes have the correct labels. In production, use node pool automation to guarantee labeling.

Recovery: Add or correct labels with:

kubectl label node <node-name> topology.kubernetes.io/zone=zone-a --overwrite

Then delete pods to force rescheduling if necessary.

Pitfall 2: maxSkew Too Low for Cluster Size

Why it happens: If maxSkew is smaller than the unevenness you must tolerate, the scheduler cannot satisfy the constraint. For example, with 5 replicas and 3 zones, you cannot have even distribution (5/3 = 1.67). With maxSkew: 1, the best you can do is 2,2,1 or 2,1,2, which is allowed because the difference between max and min is 1. But if you tried to achieve perfect evenness (impossible with 5), the scheduler would leave pods pending.

How to avoid: Calculate the theoretical minimum and maximum per domain based on replicas and number of domains. Ensure maxSkew is at least ceil(replicas/domains) - floor(replicas/domains). For 5 replicas across 3 zones, that's ceil(5/3)=2 and floor(5/3)=1, difference 1, so maxSkew: 1 works. But if you try to spread across 2 zones with 3 replicas and maxSkew: 1, that's ceil(3/2)=2 and floor(3/2)=1, difference 1, also works. Generally, test before enforcing hard constraints.

Recovery: If pods are pending, relax the constraint by increasing maxSkew or switching whenUnsatisfiable to ScheduleAnyway, or add more nodes/domains.

Pitfall 3: Conflicting Constraints

Why it happens: Combining multiple topology spread constraints (e.g., zone and hostname) can make scheduling impossible if the cluster is too small. For instance, requiring maxSkew=1 on zone and maxSkew=1 on hostname across a 3-node, 3-zone cluster means each pod must be on a unique node and within a unique zone, which is fine for 3 pods, but scaling to 4 pods becomes impossible because adding a pod to any node would violate either the zone skew or hostname skew.

How to avoid: Review constraints holistically. Use minDomains (Kubernetes 1.25+) to specify a minimum number of domains for a constraint, which can influence scheduling decisions. Simulate or test in a lab first.

Recovery: Remove or relax one of the constraints, or add nodes.

Pitfall 4: Label Selector Mismatch

Why it happens: The labelSelector in the constraint must match the pod's own labels. If you set a selector that doesn't match the pod, the constraint may count zero pods or unintended pods, leading to unexpected scheduling.

How to avoid: Always set labelSelector.matchLabels to the same labels used in the pod template (e.g., app: web).

Recovery: Correct the selector and update the deployment:

kubectl patch deployment web --type='json' -p='[{"op": "replace", "path": "/spec/template/spec/topologySpreadConstraints/0/labelSelector/matchLabels/app", "value": "web"}]'

Pitfall 5: Interference with Node Affinity and Taints

Why it happens: Node selectors, affinity, or taints can restrict which nodes are eligible, reducing the number of domains and making the spread constraint unsatisfiable.

How to avoid: Ensure that pods can schedule on all nodes in the intended domains. For example, if using taints, add tolerations.

Recovery: Adjust affinity/taints or relax the constraint.

Safe Experimentation Path

Now that the cluster is ready, we can explore topology spread constraints safely. The goal is to have a controlled experiment that is narrow and measurable.

Start with a simple Deployment as shown above. Then experiment with these variations:

• Change whenUnsatisfiable to ScheduleAnyway to see how pods distribute when constraints cannot be fully met.
• Add a second constraint on kubernetes.io/hostname to spread across nodes within zones.
• Use minDomains to enforce a minimum number of zones even if not all have pods initially.
• Combine with node affinity to restrict placement to certain zones.

For each experiment, record the pod distribution and understand the scheduler's decisions.

Example: Soft Constraint with ScheduleAnyway

Modify the deployment to use ScheduleAnyway:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: ScheduleAnyway
  labelSelector:
    matchLabels:
      app: web

Apply and scale to 5 replicas. Observe that all 5 pods are running, but distribution is uneven, for example 2 pods in zone-a, 2 in zone-b, 1 in zone-c. The scheduler tried to minimize skew but didn't block scheduling.

Example: Adding Hostname Constraint

Add a second constraint:

topologySpreadConstraints:
- maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: web
- maxSkew: 1
  topologyKey: kubernetes.io/hostname
  whenUnsatisfiable: DoNotSchedule
  labelSelector:
    matchLabels:
      app: web

With 3 replicas, this will place one pod on each node (and zone). With 4 replicas, one pod will be pending because no node can host a second pod without violating hostname skew.

This demonstrates how constraints interact.

Failure Modes and Recovery

Understanding what can go wrong and how to recover is essential. This section covers common failure modes and step-by-step recovery.

Pods Stuck in Pending

As seen earlier, if the topology spread constraint cannot be satisfied, pods remain unscheduled. This can happen due to insufficient nodes, misconfigured labels, or conflicts.

Recovery steps:

1. Identify the pending pod:

kubectl get pods --field-selector=status.phase=Pending

1. Describe the pod to see the reason:

kubectl describe pod <pending-pod>

Look for events like "0/4 nodes are available..."

1. Decide on a fix:

• Scale down: kubectl scale deployment web --replicas=3
• Relax constraint: Change maxSkew to a higher value or whenUnsatisfiable to ScheduleAnyway.
• Add more nodes with appropriate labels.
• Remove conflicting constraints.

Example patch to relax:

kubectl patch deployment web -p '{"spec":{"template":{"spec":{"topologySpreadConstraints":[{"maxSkew":2,"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"ScheduleAnyway","labelSelector":{"matchLabels":{"app":"web"}}}]}}}}'

Then delete the pending pods to allow recreation:

kubectl delete pod -l app=web --field-selector=status.phase=Pending

Node Label Mismatches

If a node is missing the expected label, add it:

kubectl label node <node-name> topology.kubernetes.io/zone=zone-a

For mislabeled nodes, correct the label:

kubectl label node <node-name> topology.kubernetes.io/zone=zone-b --overwrite

After relabeling, you may need to delete pods to trigger rescheduling.

Rollback

To rollback a deployment to a known good state, you can use kubectl rollout undo if you have a previous revision:

kubectl rollout undo deployment/web

Or apply a previously saved YAML file.

Cluster Recreation

If the lab becomes unstable, delete and recreate the cluster:

kind delete cluster --name topo-lab
kind create cluster --name topo-lab --config kind-config.yaml

This resets everything.

Operations Checklist

Use this checklist for repeatable operations when working with topology spread constraints in a lab.

Pre-Deployment Checklist

• [ ] Verify cluster is running: kubectl cluster-info
• [ ] Verify nodes have correct labels: kubectl get nodes --show-labels
• [ ] Ensure the required topology keys are present on all intended nodes.
• [ ] Test scheduling with a simple pod (e.g., nginx) with no constraints to confirm basic scheduling works.
• [ ] Decide on whenUnsatisfiable based on requirements (hard vs. soft).
• [ ] Calculate expected skew based on replicas and domain count.
• [ ] Assign an owner for the deployment configuration review (e.g., Priya Shah, Engineering Lead), who must approve any changes to constraints.

Deployment Checklist

• [ ] Apply deployment with topology spread constraints: kubectl apply -f app.yaml
• [ ] Verify pods are scheduled as expected: kubectl get pods -o wide
• [ ] Count pods per topology domain using the awk command.
• [ ] Check for pending pods: kubectl get pods --field-selector=status.phase=Pending
• [ ] Review events if issues: kubectl describe deployment web
• [ ] Confirm the constraint annotation is present on a pod: kubectl get pod <pod-name> -o jsonpath='{.metadata.annotations}'

Post-Deployment Checklist

• [ ] Monitor pod distribution after scaling events (e.g., scale to 4 and 5 replicas).
• [ ] Test failure scenarios: cordon a node (kubectl cordon kind-worker) and see how pods reschedule; uncordon after test.
• [ ] Validate that skew remains within limits.
• [ ] Document any unexpected behavior in a lab notebook for future reference.
• [ ] Review the deployment weekly with the team to ensure constraints still align with availability goals.

Rollback Checklist

• [ ] Keep previous YAML manifests in version control.
• [ ] Use kubectl rollout undo if Deployment is managed via rolling updates.
• [ ] If necessary, scale down or delete deployment.
• [ ] Recreate cluster if needed.

Operational command examples:

# Check skew by counting pods per node
kubectl get pods -l app=web -o wide | awk 'NR>1 {print $7}' | sort | uniq -c

# Get events for a deployment
kubectl describe deployment web | tail -20

# Check pending pods
kubectl get pods --field-selector=status.phase=Pending

Conclusion

In this guide, you have set up a local Kubernetes lab to safely test topology spread constraints. You learned to:

• Create a multi-zone cluster using kind.
• Label nodes with topology keys.
• Deploy applications with topology spread constraints.
• Verify pod distribution and diagnose failures.
• Recover from common issues.

By following the narrow pilot approach, you can systematically explore Kubernetes scheduling behavior without risking production environments. The key takeaways are:

• Start with a clear, testable scenario.
• Use whenUnsatisfiable: ScheduleAnyway during initial exploration.
• Always verify with kubectl get pods -o wide and describe.
• Have a rollback plan.

Next steps:

• Experiment with multiple constraints and node affinities.
• Test soft constraints and observe skew under load.
• Explore minDomains parameter (Kubernetes 1.25+) for even more control.
• Integrate topology spread constraints into your CI/CD testing.

With this lab, you can build confidence and expertise to apply topology spread constraints effectively in production.