Kubernetes Node Affinity: Common Errors, Fixes, and Practical Examples

Intro

Kubernetes node affinity gives you fine-grained control over pod scheduling by declaring rules that attract pods to specific nodes. When those rules are misconfigured or misunderstood, pods can remain in Pending, evict unexpectedly, or land on the wrong hardware. This article walks through the most common node affinity errors, how to diagnose them with concrete commands, and how to fix them with practical examples.

You will learn how to distinguish between requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution, how to read scheduler events from kubectl describe pod, and how to correct label mismatches, invalid operators, and topology key mistakes. Each fix includes copy-paste manifest snippets and verification steps so you can apply the solution safely in a development namespace before rolling it out to production.

Version and Environment Inventory

Before touching a manifest, confirm the Kubernetes version and the nodes in your cluster. Node affinity syntax is stable across recent releases, but label behavior and scheduler messages can vary slightly by version. Run:

kubectl version --short
kubectl get nodes --show-labels

Expected output includes a server version line like Server Version: v1.28.2 and a list of nodes with their labels. For example:

NAME       STATUS   ROLES           AGE   VERSION   LABELS
node-1     Ready    control-plane   10d   v1.28.2   kubernetes.io/hostname=node-1,disktype=ssd,region=us-east
node-2     Ready    <none>          10d   v1.28.2   kubernetes.io/hostname=node-2,disktype=hdd,region=us-west

If your cluster is managed (EKS, GKE, AKS), use the cloud provider's CLI to check the control plane version, but node labels are still visible via kubectl. Record the exact label keys and values you plan to use in your affinity rule. A common error is assuming a label exists when it does not. Before adding disktype: ssd to your affinity, verify that at least one node actually has that label:

kubectl get nodes -l disktype=ssd

No output means no nodes match, and any requiredDuringSchedulingIgnoredDuringExecution rule using that label will never schedule.

Quick check 1 of 2

What is the key difference between requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution in node affinity?

According to the reference, requiredDuringSchedulingIgnoredDuringExecution means the scheduler can't schedule the Pod unless the rule is met, while preferredDuringSchedulingIgnoredDuringExecution means the scheduler tries to find a matching node but will still schedule the Pod if none is found.

Safe Configuration Path

Always test affinity changes in a non-production namespace first. The smallest safe test is a single pod with a simple affinity rule that mirrors your intended production logic. For example:

apiVersion: v1
kind: Pod
metadata:
  name: affinity-test
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd
  containers:
  - name: nginx
    image: nginx:1.25

Apply it and immediately check the pod status:

kubectl apply -f affinity-test.yaml
kubectl get pod affinity-test -o wide

If the pod is scheduled and running on a node with disktype=ssd, your rule works. If it is Pending, run kubectl describe pod affinity-test and look at the Events section:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  10s   default-scheduler  0/2 nodes are available: 2 node(s) didn't match node selector.

That message confirms the affinity rule excluded all nodes. Fix the label or the rule before moving on.

Verification and Diagnostics

Diagnosing node affinity failures requires looking at the scheduler's decision process. Start with the pod events, then inspect the node labels and the affinity expression itself.

Step 1: Describe the pending pod

kubectl describe pod <pod-name>

Look for FailedScheduling events. They will tell you why no node matched. For example:

0/3 nodes are available: 1 node(s) had untolerated taint {dedicated: infrastructure}, 2 node(s) didn't match node affinity/selector.

This separates taint issues from affinity issues. If taints are the problem, you need tolerations, not affinity changes. If affinity is the problem, proceed to check labels.

Step 2: Check existing node labels

kubectl get nodes --show-labels | grep disktype

If no output, the label key does not exist on any node. Add it to the appropriate nodes:

kubectl label node node-1 disktype=ssd

Then verify:

kubectl get node node-1 --show-labels

Expected partial output:

... disktype=ssd

Step 3: Validate the affinity expression

Common mistakes include:

Using operator: Equal with multiple values (invalid; Equal accepts exactly one value).
Using operator: In but leaving values empty.
Misspelling requiredDuringSchedulingIgnoredDuringExecution or matchExpressions.
Forgetting that nodeSelectorTerms is an array and must contain at least one item.

For example, this invalid snippet:

matchExpressions:
- key: disktype
  operator: Equal
  values: [ssd, nvme]

will be rejected by the API server with an error like:

The Deployment "my-deploy" is invalid: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].values: Invalid value: []string{"ssd", "nvme"}: must be a single value for Equal operator

Fix by using operator: In with multiple values or Equal with one value.

Step 4: Watch the scheduler in real time

If the pod remains pending, watch events:

kubectl get events --field-selector involvedObject.name=<pod-name> --watch

This streams scheduler messages as nodes become available or labels change. It helps confirm that your fix took effect without waiting for the next retry cycle.

Failure Modes and Recovery

Node affinity can fail in several predictable ways. Each has a specific signature and recovery procedure.

Failure 1: No nodes satisfy required affinity

Symptom: Pod stuck in Pending. Events show 0/N nodes are available: N node(s) didn't match node affinity/selector.

Cause: The requiredDuringSchedulingIgnoredDuringExecution rule is too strict or labels are missing.

Recovery:

Check node labels with kubectl get nodes --show-labels.
Add missing labels to at least one node.
Or modify the affinity to be less restrictive, e.g., change required... to preferred... if the requirement is not absolutely mandatory.

Example of a preferredDuringSchedulingIgnoredDuringExecution rule that can replace a required one:

affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      preference:
        matchExpressions:
        - key: disktype
          operator: In
          values:
          - ssd

This allows scheduling on nodes without disktype=ssd if none are available, but prefers them when present.

Failure 2: Pod scheduled on wrong node because preferred rule ignored other constraints

Symptom: Pod is running but on a node that violates a business expectation.

Cause: preferred rules are soft. The scheduler may ignore them if hard constraints (like resource requests) force a different placement.

Recovery: Convert the preferred rule to required, or add additional required rules. Verify with kubectl get pod -o wide to see the node.

Failure 3: Scheduler crashes or becomes unavailable

Symptom: All pending pods show no events, or kubectl describe pod shows no FailedScheduling event but pod remains pending.

Cause: Scheduler pod is down or not running.

Recovery: Check scheduler health:

kubectl get pods -n kube-system | grep scheduler
kubectl logs -n kube-system kube-scheduler-<control-plane-node> --tail=50

If the scheduler is healthy, the issue is elsewhere (e.g., no resources). If it is down, restart or investigate the underlying node.

Failure 4: Node affinity combined with node selector conflicts

Symptom: Pod pending despite nodes appearing to match.

Cause: You set both nodeSelector and nodeAffinity. The scheduler ANDs all constraints, so a mismatch in either prevents scheduling.

Recovery: Remove one or make them consistent. Check the full pod spec:

kubectl get pod <name> -o yaml | grep -A5 nodeSelector

Then align the labels.

Operations Checklist

Use this checklist before and after applying node affinity changes to minimize risk.

Pre-change

[ ] Run kubectl get nodes --show-labels and record the label set.
[ ] Identify at least one node that matches your intended affinity rule.
[ ] Confirm the Kubernetes version supports your affinity syntax (all stable versions do, but check if using beta features).
[ ] Back up the current deployment/pod spec: kubectl get deploy my-deploy -o yaml > deploy-backup.yaml.
[ ] Test the new affinity rule with a minimal pod in a development namespace.
[ ] Ensure you have permission to label nodes if your fix requires label changes.

Post-change

[ ] Apply the changed manifest and watch rollout: kubectl rollout status deployment/my-deploy.
[ ] Verify pod placement: kubectl get pods -o wide shows expected node.
[ ] Check events for scheduling errors: kubectl describe pod <new-pod>.
[ ] If the pod is pending, capture the scheduler message before reverting.
[ ] Document the fix and label changes in your runbook.

Example of a complete deployment with node affinity after a fix:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: topology.kubernetes.io/zone
                operator: In
                values:
                - us-east-1a
                - us-east-1b
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: disktype
                operator: In
                values:
                - ssd
      containers:
      - name: nginx
        image: nginx:1.25

This requires pods to run in zones us-east-1a or us-east-1b, and prefers nodes with disktype=ssd. Verify with:

kubectl apply -f web-deploy.yaml
kubectl get pods -o wide

All pods should be Running and their NODE column should show nodes in the allowed zones.

Conclusion

Node affinity errors are almost always visible in the pod's events, and the fix usually involves aligning labels with your affinity expressions. By following the diagnostic steps in this article—check labels first, then validate the affinity syntax, then watch the scheduler—you can resolve most issues in minutes.

Remember that requiredDuringSchedulingIgnoredDuringExecution is a hard constraint: if no node matches, the pod will never schedule. Use preferredDuringSchedulingIgnoredDuringExecution for soft preferences that should not block scheduling. Always test in a development namespace, keep a backup of your original manifest, and document any label changes.

A reliable workflow makes failure visible, protects sensitive values, and defines recovery verification before an incident forces the decision. With the practical examples here, you can approach node affinity configuration and troubleshooting with confidence.