## Intro Kubernetes node affinity gives you fine-grained control over pod scheduling by declaring rules that attract pods to specific nodes. When those rules are misconfigured or misunderstood, pods can remain in Pending , evict unexpectedly, or land on the wrong hardware. This article walks through the most common node affinity errors, how to diagnose them with concrete commands, and how to fix them with practical examples. You will learn how to distinguish between requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution , how to read scheduler events from kubectl describe pod , and how to correct label mismatches, invalid operators, and topology key mistakes. Each fix includes copy-paste manifest snippets and verification steps so you can apply the solution safely in a development namespace before rolling it out to production. ## Version and Environment Inventory Before touching a manifest, confirm the Kubernetes version and the nodes in your cluster. Node affinity syntax is stable across recent releases, but label behavior and scheduler messages can vary slightly by version. Run: kubectl version --short kubectl get nodes --show-labels Expected output includes a server version line like Server Version: v1.28.2 and a list of nodes with their labels. For example: NAME STATUS ROLES AGE VERSION LABELS node-1 Ready control-plane 10d v1.28.2 kubernetes.io/hostname=node-1,disktype=ssd,region=us-east node-2 Ready 10d v1.28.2 kubernetes.io/hostname=node-2,disktype=hdd,region=us-west If your cluster is managed (EKS, GKE, AKS), use the cloud provider's CLI to check the control plane version, but node labels are still visible via kubectl . Record the exact label keys and values you plan to use in your affinity rule. A common error is assuming a label exists when it does not. Before adding disktype: ssd to your affinity, verify that at least one node actually has that label: kubectl get nodes -l disktype=ssd No output means no nodes match, and any requiredDuringSchedulingIgnoredDuringExecution rule using that label will never schedule.

Quick check 1 of 2 What is the key difference between requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution in node affinity? - Required rules are always enforced, while preferred rules are optional. - Preferred rules are enforced, while required rules are optional. - Required rules ignore label changes after scheduling, while preferred rules do not. - Preferred rules are only applied to existing pods.

According to the reference, requiredDuringSchedulingIgnoredDuringExecution means the scheduler can't schedule the Pod unless the rule is met, while preferredDuringSchedulingIgnoredDuringExecution means the scheduler tries to find a matching node but will still schedule the Pod if none is found.

## Safe Configuration Path Always test affinity changes in a non-production namespace first. The smallest safe test is a single pod with a simple affinity rule that mirrors your intended production logic. For example: apiVersion: v1 kind: Pod metadata: name: affinity-test spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: disktype operator: In values: - ssd containers: - name: nginx image: nginx:1.25 Apply it and immediately check the pod status: kubectl apply -f affinity-test.yaml kubectl get pod affinity-test -o wide If the pod is scheduled and running on a node with disktype=ssd , your rule works. If it is Pending , run kubectl describe pod affinity-test and look at the Events section: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 10s default-scheduler 0/2 nodes are available: 2 node(s) didn't match node selector. That message confirms the affinity rule excluded all nodes. Fix the label or the rule before moving on. ## Verification and Diagnostics Diagnosing node affinity failures requires looking at the scheduler's decision process. Start with the pod events, then inspect the node labels and the affinity expression itself. ### Step 1: Describe the pending pod kubectl describe pod Look for FailedScheduling events. They will tell you why no node matched. For example: 0/3 nodes are available: 1 node(s) had untolerated taint {dedicated: infrastructure}, 2 node(s) didn't match node affinity/selector. This separates taint issues from affinity issues. If taints are the problem, you need tolerations, not affinity changes. If affinity is the problem, proceed to check labels. ### Step 2: Check existing node labels kubectl get nodes --show-labels | grep disktype If no output, the label key does not exist on any node. Add it to the appropriate nodes: kubectl label node node-1 disktype=ssd Then verify: kubectl get node node-1 --show-labels Expected partial output: ... disktype=ssd ### Step 3: Validate the affinity expression Common mistakes include: - Using operator: Equal with multiple values (invalid; Equal accepts exactly one value). - Using operator: In but leaving values empty. - Misspelling requiredDuringSchedulingIgnoredDuringExecution or matchExpressions . - Forgetting that nodeSelectorTerms is an array and must contain at least one item. For example, this invalid snippet: matchExpressions: - key: disktype operator: Equal values: [ssd, nvme] will be rejected by the API server with an error like: The Deployment "my-deploy" is invalid: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values: Invalid value: []string{"ssd", "nvme"}: must be a single value for Equal operator Fix by using operator: In with multiple values or Equal with one value. ### Step 4: Watch the scheduler in real time If the pod remains pending, watch events: kubectl get events --field-selector involvedObject.name= --watch This streams scheduler messages as nodes become available or labels change. It helps confirm that your fix took effect without waiting for the next retry cycle. ## Failure Modes and Recovery Node affinity can fail in several predictable ways. Each has a specific signature and recovery procedure. ### Failure 1: No nodes satisfy required affinity Symptom: Pod stuck in Pending . Events show 0/N nodes are available: N node(s) didn't match node affinity/selector . Cause: The requiredDuringSchedulingIgnoredDuringExecution rule is too strict or labels are missing. Recovery: - Check node labels with kubectl get nodes --show-labels . - Add missing labels to at least one node. - Or modify the affinity to be less restrictive, e.g., change required... to preferred... if the requirement is not absolutely mandatory. Example of a preferredDuringSchedulingIgnoredDuringExecution rule that can replace a required one: affinity: nodeAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 preference: matchExpressions: - key: disktype operator: In values: - ssd This allows scheduling on nodes without disktype=ssd if none are available, but prefers them when present. ### Failure 2: Pod scheduled on wrong node because preferred rule ignored other constraints Symptom: Pod is running but on a node that violates a business expectation. Cause: preferred rules are soft. The scheduler may ignore them if hard constraints (like resource requests) force a different placement. Recovery: Convert the preferred rule to required, or add additional required rules. Verify with kubectl get pod -o wide to see the node. ### Failure 3: Scheduler crashes or becomes unavailable Symptom: All pending pods show no events, or kubectl describe pod shows no FailedScheduling event but pod remains pending. Cause: Scheduler pod is down or not running. Recovery: Check scheduler health: kubectl get pods -n kube-system | grep scheduler kubectl logs -n kube-system kube-scheduler- --tail=50 If the scheduler is healthy, the issue is elsewhere (e.g., no resources). If it is down, restart or investigate the underlying node. ### Failure 4: Node affinity combined with node selector conflicts Symptom: Pod pending despite nodes appearing to match. Cause: You set both nodeSelector and nodeAffinity . The scheduler ANDs all constraints, so a mismatch in either prevents scheduling. Recovery: Remove one or make them consistent. Check the full pod spec: kubectl get pod -o yaml | grep -A5 nodeSelector Then align the labels. ## Operations Checklist Use this checklist before and after applying node affinity changes to minimize risk. ### Pre-change - [ ] Run kubectl get nodes --show-labels and record the label set. - [ ] Identify at least one node that matches your intended affinity rule. - [ ] Confirm the Kubernetes version supports your affinity syntax (all stable versions do, but check if using beta features). - [ ] Back up the current deployment/pod spec: kubectl get deploy my-deploy -o yaml > deploy-backup.yaml . - [ ] Test the new affinity rule with a minimal pod in a development namespace. - [ ] Ensure you have permission to label nodes if your fix requires label changes. ### Post-change - [ ] Apply the changed manifest and watch rollout: kubectl rollout status deployment/my-deploy . - [ ] Verify pod placement: kubectl get pods -o wide shows expected node. - [ ] Check events for scheduling errors: kubectl describe pod . - [ ] If the pod is pending, capture the scheduler message before reverting. - [ ] Document the fix and label changes in your runbook. Example of a complete deployment with node affinity after a fix: apiVersion: apps/v1 kind: Deployment metadata: name: web spec: replicas: 3 selector: matchLabels: app: web template: metadata: labels: app: web spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/zone operator: In values: - us-east-1a - us-east-1b preferredDuringSchedulingIgnoredDuringExecution: - weight: 80 preference: matchExpressions: - key: disktype operator: In values: - ssd containers: - name: nginx image: nginx:1.25 This requires pods to run in zones us-east-1a or us-east-1b , and prefers nodes with disktype=ssd . Verify with: kubectl apply -f web-deploy.yaml kubectl get pods -o wide All pods should be Running and their NODE column should show nodes in the allowed zones. ## Conclusion Node affinity errors are almost always visible in the pod's events, and the fix usually involves aligning labels with your affinity expressions. By following the diagnostic steps in this article—check labels first, then validate the affinity syntax, then watch the scheduler—you can resolve most issues in minutes. Remember that requiredDuringSchedulingIgnoredDuringExecution is a hard constraint: if no node matches, the pod will never schedule. Use preferredDuringSchedulingIgnoredDuringExecution for soft preferences that should not block scheduling. Always test in a development namespace, keep a backup of your original manifest, and document any label changes. A reliable workflow makes failure visible, protects sensitive values, and defines recovery verification before an incident forces the decision. With the practical examples here, you can approach node affinity configuration and troubleshooting with confidence.