## Intro Node affinity in Kubernetes lets you control which nodes your pods land on based on node labels. It is more expressive than the older nodeSelector field and supports both hard requirements and soft preferences. This article explains the architecture behind node affinity, walks through practical examples, and shows how to verify, diagnose, and recover from common scheduling issues. You will learn how the kube-scheduler evaluates node affinity rules, how to write manifests with required and preferred terms, and how to combine affinity with taints, tolerations, and topology spread for robust scheduling. Every example includes commands and expected output so you can reproduce the scenarios in your own cluster. ## Version and Environment Inventory Before using node affinity, check your cluster version and the scheduling components involved. Node affinity has been stable since Kubernetes 1.6 and is available in all modern distributions. The examples in this article assume Kubernetes 1.25 or later and a working kubectl context. ### Identify your cluster version Run: kubectl version --short Expected output (example): Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.3 If your server version is older than 1.6, upgrade before using node affinity. Most current clusters satisfy this requirement. ### Understand the kube-scheduler role The kube-scheduler is the component that reads pod specifications and finds a suitable node. It runs as a static pod in the kube-system namespace on control plane nodes. Confirm it is running: kubectl get pods -n kube-system -l component=kube-scheduler Expected output shows a running scheduler pod, for example: NAME READY STATUS RESTARTS AGE kube-scheduler-control-plane-1 1/1 Running 0 12d The scheduler uses a scoring algorithm that considers node affinity rules during the filtering and scoring phases. Hard rules (requiredDuringSchedulingIgnoredDuringExecution) act as filters, eliminating nodes that do not match. Soft rules (preferredDuringSchedulingIgnoredDuringExecution) add weight during scoring, giving matching nodes a higher rank but not excluding others. ### Basic node labeling Node affinity matches pod requirements against node labels. List current labels on your nodes: kubectl get nodes --show-labels Example output: NAME STATUS ROLES AGE VERSION LABELS node-1 Ready control-plane 10d v1.28.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=node-1,node-role.kubernetes.io/control-plane= node-2 Ready 10d v1.28.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=node-2,disktype=ssd,zone=us-west-2a node-3 Ready 10d v1.28.3 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=node-3,disktype=hdd,zone=us-west-2b In this example, node-2 has the label disktype=ssd, which we can use for affinity. ## Safe Configuration Path Start with a read-only observation of the current scheduling state. Then make the smallest change that demonstrates node affinity and verify it before expanding. ### Step 1: Observe existing pods and their node placement Run: kubectl get pods -o wide --all-namespaces This shows where pods are currently running. If you have a specific namespace, use -n . ### Step 2: Design a minimal node affinity manifest Create a file named nginx-affinity.yaml : apiVersion: v1 kind: Pod metadata: name: nginx-affinity spec: containers: - name: nginx image: nginx:1.25 ports: - containerPort: 80 affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: disktype operator: In values: - ssd This pod requires a node with the label disktype=ssd . It will only be scheduled on node-2 in our example cluster. ### Step 3: Apply the manifest kubectl apply -f nginx-affinity.yaml Expected output: pod/nginx-affinity created ### Step 4: Verify scheduling Check the pod status and node: kubectl get pod nginx-affinity -o wide Example output: NAME READY STATUS RESTARTS AGE IP NODE nginx-affinity 1/1 Running 0 15s 10.244.2.5 node-2 The pod is running on node-2, which matches the label. If no node matches, the pod remains in Pending state. We will cover that in the failure modes section. ### Step 5: Inspect scheduling events Use kubectl describe pod nginx-affinity to see events: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 25s default-scheduler Successfully assigned default/nginx-affinity to node-2 Normal Pulling 24s kubelet Pulling image "nginx:1.25" Normal Started 23s kubelet Started container nginx Normal Created 23s kubelet Created container nginx The Scheduled event confirms the scheduler used the affinity rule. ## Verification and Diagnostics Now that a basic required rule works, expand to preferred rules and combine affinity with other scheduling features. ### Preferred node affinity example Create nginx-pref-affinity.yaml : apiVersion: v1 kind: Pod metadata: name: nginx-pref spec: containers: - name: nginx image: nginx:1.25 affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 preference: matchExpressions: - key: zone operator: In values: - us-west-2a - weight: 20 preference: matchExpressions: - key: disktype operator: In values: - ssd This pod prefers nodes in zone us-west-2a with a higher weight (80) and also prefers ssd nodes with weight 20. If no node matches, it still schedules on any node because there is no required rule. Apply and verify: kubectl apply -f nginx-pref-affinity.yaml kubectl get pod nginx-pref -o wide Example output: NAME READY STATUS RESTARTS AGE IP NODE nginx-pref 1/1 Running 0 10s 10.244.1.7 node-2 It runs on node-2, which matches both preferences. If node-2 were unavailable, it could run on node-1 or node-3. ### Using matchFields with node name You can target a specific node using its name, though this is less flexible than labels. Example: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchFields: - key: metadata.name operator: In values: - node-2 This hard requirement forces the pod onto node-2. Use sparingly because it reduces scheduling resilience. ### Combining node affinity with resource requests Add resource requests to influence scoring and prevent overcommit. For example, a memory-intensive pod: apiVersion: v1 kind: Pod metadata: name: mem-pod spec: containers: - name: mem-container image: polinux/stress resources: requests: memory: "512Mi" cpu: "250m" affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: disktype operator: Exists This pod requires a node with the disktype label (any value) and requests resources. The scheduler filters nodes by label and then by available capacity. ### Diagnostic commands for troubleshooting When a pod with node affinity is pending, use these commands: # Check pod events kubectl describe pod # Check node labels to see if any match kubectl get nodes --show-labels # Get scheduler logs (if you have access) kubectl logs -n kube-system kube-scheduler-control-plane-1 | grep -i affinity The scheduler logs may show why nodes were rejected. Look for messages like "node(s) didn't match node selector" or "0/3 nodes are available: 3 node(s) didn't match node affinity/selector." This is the standard event for failures. ## Failure Modes and Recovery Node affinity can cause pods to remain pending if requirements are too strict or labels are misconfigured. Here are common failure scenarios and how to recover. ### Scenario 1: No node matches required affinity Suppose you apply a pod requiring disktype=nvme , but no node has that label. The pod stays Pending. Check: kubectl get pod nvme-pod kubectl describe pod nvme-pod The describe output includes: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 12s default-scheduler 0/3 nodes are available: 3 node(s) didn't match node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. Recovery options: - Check if the label is missing or misspelled. Add the label to a node: kubectl label node node-3 disktype=nvme - Modify the pod spec to use a softer preference or remove the requirement. - If the pod is part of a Deployment, edit the deployment to fix the affinity, then rollout restart: kubectl edit deployment # change affinity, save and exit kubectl rollout status deployment/ ### Scenario 2: Affinity conflicts with taints A node may have the required label but also a taint that the pod does not tolerate. Example: node-2 has taint dedicated=special:NoSchedule . The pod with affinity for node-2 but no tolerance will not schedule. Diagnose: kubectl get node node-2 -o json | jq '.spec.taints' Example output: [ { "effect": "NoSchedule", "key": "dedicated", "value": "special" } ] Recovery: - Add a toleration to the pod spec: tolerations: - key: dedicated operator: Equal value: special effect: NoSchedule - Or remove the taint if it is not needed: kubectl taint nodes node-2 dedicated=special:NoSchedule- ### Scenario 3: Pod stuck terminating or unschedulable after node loss If a node that matches affinity goes down, pods with required affinity to that node will not reschedule elsewhere because no other node satisfies the rule. Use kubectl get pods -o wide to see node status. If the node is NotReady, the pods may be stuck in Terminating or Pending. Recovery: - Evict the pod manually: kubectl delete pod --force --grace-period=0 - Then either add the missing label to another node or change the affinity to a preferred rule. ## Operations Checklist Use this checklist before and after making node affinity changes in production. ### Before applying changes - [ ] Run kubectl get nodes --show-labels and record the current labels of candidate nodes. - [ ] Run kubectl get pods -o wide in the target namespace to see existing placement. - [ ] Check if the scheduler is healthy: kubectl get pods -n kube-system -l component=kube-scheduler - [ ] Review taints on nodes that should match: kubectl describe nodes | grep Taints - [ ] Decide if you need required or preferred affinity. Prefer preferred for high availability unless you have a strict hardware requirement. - [ ] Write the YAML manifest and validate it with kubectl apply --dry-run=client -f manifest.yaml ### After applying changes - [ ] Verify pods are scheduled: kubectl get pods -o wide - [ ] Check events for FailedScheduling warnings: kubectl describe pod - [ ] If using a Deployment, confirm rollout success: kubectl rollout status deployment/ - [ ] Test behavior by cordoning a matching node and seeing if pods reschedule according to affinity rules (if preferred) or stay pending (if required). - [ ] Document the change in your runbook with the exact labels and affinity terms used. ### Quick reference for common affinity operators Operator Meaning Example In Label value is in the list disktype In [ssd, nvme] NotIn Label value is not in the list zone NotIn [us-west-2b] Exists Label key exists regardless of value disktype Exists DoesNotExist Label key does not exist gpu DoesNotExist Gt Label value is greater than integer (for node affinity with matchFields? Not supported for label values) cpu-count Gt 4 (only in node selector? Not for affinity) Lt Label value is less than integer Similar to above Note: The Gt and Lt operators are not valid for label-based matchExpressions in node affinity; they are used in node selector terms with matchFields for metadata.name? Actually, they are not supported at all in node affinity. Stick to In, NotIn, Exists, DoesNotExist. ## Conclusion Node affinity gives you fine-grained control over pod placement while maintaining clarity in your manifests. By using labels, match expressions, and weights, you can build scheduling policies that reflect your infrastructure topology and application requirements. Always start with a small test, verify scheduling events, and prefer soft rules unless a hard constraint is necessary. Combine node affinity with tolerations, resource requests, and topology spread constraints to create resilient, efficient deployments. To deepen your understanding, experiment with different match expressions, practice with taints and tolerations, and monitor scheduler behavior using the metrics endpoint. Node affinity is a foundational tool in the Kubernetes scheduler, and mastering it will make you more effective at managing production clusters.