Kubernetes Node Affinity Architecture Explained with Practical Examples

Intro

Node affinity in Kubernetes lets you control which nodes your pods land on based on node labels. It is more expressive than the older nodeSelector field and supports both hard requirements and soft preferences. This article explains the architecture behind node affinity, walks through practical examples, and shows how to verify, diagnose, and recover from common scheduling issues.

You will learn how the kube-scheduler evaluates node affinity rules, how to write manifests with required and preferred terms, and how to combine affinity with taints, tolerations, and topology spread for robust scheduling. Every example includes commands and expected output so you can reproduce the scenarios in your own cluster.

Version and Environment Inventory

Before using node affinity, check your cluster version and the scheduling components involved. Node affinity has been stable since Kubernetes 1.6 and is available in all modern distributions. The examples in this article assume Kubernetes 1.25 or later and a working kubectl context.

Identify your cluster version

Run:

kubectl version --short

Expected output (example):

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3

If your server version is older than 1.6, upgrade before using node affinity. Most current clusters satisfy this requirement.

Understand the kube-scheduler role

The kube-scheduler is the component that reads pod specifications and finds a suitable node. It runs as a static pod in the kube-system namespace on control plane nodes. Confirm it is running:

kubectl get pods -n kube-system -l component=kube-scheduler

Expected output shows a running scheduler pod, for example:

NAME                            READY   STATUS    RESTARTS      AGE
kube-scheduler-control-plane-1   1/1     Running   0             12d

The scheduler uses a scoring algorithm that considers node affinity rules during the filtering and scoring phases. Hard rules (requiredDuringSchedulingIgnoredDuringExecution) act as filters, eliminating nodes that do not match. Soft rules (preferredDuringSchedulingIgnoredDuringExecution) add weight during scoring, giving matching nodes a higher rank but not excluding others.

Basic node labeling

Node affinity matches pod requirements against node labels. List current labels on your nodes:

kubectl get nodes --show-labels

Example output:

NAME       STATUS   ROLES           AGE    VERSION   LABELS
node-1     Ready    control-plane   10d    v1.28.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=node-1,node-role.kubernetes.io/control-plane=
node-2     Ready    <none>          10d    v1.28.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=node-2,disktype=ssd,zone=us-west-2a
node-3     Ready    <none>          10d    v1.28.3   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=node-3,disktype=hdd,zone=us-west-2b

In this example, node-2 has the label disktype=ssd, which we can use for affinity.

Safe Configuration Path

Start with a read-only observation of the current scheduling state. Then make the smallest change that demonstrates node affinity and verify it before expanding.

Step 1: Observe existing pods and their node placement

Run:

kubectl get pods -o wide --all-namespaces

This shows where pods are currently running. If you have a specific namespace, use -n <namespace>.

Step 2: Design a minimal node affinity manifest

Create a file named nginx-affinity.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-affinity
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    ports:
    - containerPort: 80
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd

This pod requires a node with the label disktype=ssd. It will only be scheduled on node-2 in our example cluster.

Step 3: Apply the manifest

kubectl apply -f nginx-affinity.yaml

Expected output:

pod/nginx-affinity created

Step 4: Verify scheduling

Check the pod status and node:

kubectl get pod nginx-affinity -o wide

Example output:

NAME             READY   STATUS    RESTARTS   AGE   IP            NODE
nginx-affinity   1/1     Running   0          15s   10.244.2.5    node-2

The pod is running on node-2, which matches the label. If no node matches, the pod remains in Pending state. We will cover that in the failure modes section.

Step 5: Inspect scheduling events

Use kubectl describe pod nginx-affinity to see events:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  25s   default-scheduler  Successfully assigned default/nginx-affinity to node-2
  Normal  Pulling    24s   kubelet            Pulling image "nginx:1.25"
  Normal  Started    23s   kubelet            Started container nginx
  Normal  Created    23s   kubelet            Created container nginx

The Scheduled event confirms the scheduler used the affinity rule.

Verification and Diagnostics

Now that a basic required rule works, expand to preferred rules and combine affinity with other scheduling features.

Preferred node affinity example

Create nginx-pref-affinity.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: nginx-pref
spec:
  containers:
  - name: nginx
    image: nginx:1.25
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80
        preference:
          matchExpressions:
          - key: zone
            operator: In
            values:
            - us-west-2a
      - weight: 20
        preference:
          matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd

This pod prefers nodes in zone us-west-2a with a higher weight (80) and also prefers ssd nodes with weight 20. If no node matches, it still schedules on any node because there is no required rule.

Apply and verify:

kubectl apply -f nginx-pref-affinity.yaml
kubectl get pod nginx-pref -o wide

Example output:

NAME         READY   STATUS    RESTARTS   AGE   IP            NODE
nginx-pref   1/1     Running   0          10s   10.244.1.7    node-2

It runs on node-2, which matches both preferences. If node-2 were unavailable, it could run on node-1 or node-3.

Using matchFields with node name

You can target a specific node using its name, though this is less flexible than labels. Example:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchFields:
        - key: metadata.name
          operator: In
          values:
          - node-2

This hard requirement forces the pod onto node-2. Use sparingly because it reduces scheduling resilience.

Combining node affinity with resource requests

Add resource requests to influence scoring and prevent overcommit. For example, a memory-intensive pod:

apiVersion: v1
kind: Pod
metadata:
  name: mem-pod
spec:
  containers:
  - name: mem-container
    image: polinux/stress
    resources:
      requests:
        memory: "512Mi"
        cpu: "250m"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: Exists

This pod requires a node with the disktype label (any value) and requests resources. The scheduler filters nodes by label and then by available capacity.

Diagnostic commands for troubleshooting

When a pod with node affinity is pending, use these commands:

# Check pod events
kubectl describe pod <pod-name>

# Check node labels to see if any match
kubectl get nodes --show-labels

# Get scheduler logs (if you have access)
kubectl logs -n kube-system kube-scheduler-control-plane-1 | grep -i affinity

The scheduler logs may show why nodes were rejected. Look for messages like "node(s) didn't match node selector" or "0/3 nodes are available: 3 node(s) didn't match node affinity/selector." This is the standard event for failures.

Failure Modes and Recovery

Node affinity can cause pods to remain pending if requirements are too strict or labels are misconfigured. Here are common failure scenarios and how to recover.

Scenario 1: No node matches required affinity

Suppose you apply a pod requiring disktype=nvme, but no node has that label. The pod stays Pending.

Check:

kubectl get pod nvme-pod
kubectl describe pod nvme-pod

The describe output includes:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  12s   default-scheduler  0/3 nodes are available: 3 node(s) didn't match node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Recovery options:

1. Check if the label is missing or misspelled. Add the label to a node:

kubectl label node node-3 disktype=nvme

1. Modify the pod spec to use a softer preference or remove the requirement.
2. If the pod is part of a Deployment, edit the deployment to fix the affinity, then rollout restart:

kubectl edit deployment <deployment-name>
# change affinity, save and exit
kubectl rollout status deployment/<deployment-name>

Scenario 2: Affinity conflicts with taints

A node may have the required label but also a taint that the pod does not tolerate. Example: node-2 has taint dedicated=special:NoSchedule. The pod with affinity for node-2 but no tolerance will not schedule.

Diagnose:

kubectl get node node-2 -o json | jq '.spec.taints'

Example output:

[
  {
    "effect": "NoSchedule",
    "key": "dedicated",
    "value": "special"
  }
]

Recovery:

• Add a toleration to the pod spec:

tolerations:
- key: dedicated
  operator: Equal
  value: special
  effect: NoSchedule

• Or remove the taint if it is not needed:

kubectl taint nodes node-2 dedicated=special:NoSchedule-

Scenario 3: Pod stuck terminating or unschedulable after node loss

If a node that matches affinity goes down, pods with required affinity to that node will not reschedule elsewhere because no other node satisfies the rule. Use kubectl get pods -o wide to see node status. If the node is NotReady, the pods may be stuck in Terminating or Pending.

Recovery:

• Evict the pod manually:

kubectl delete pod <pod-name> --force --grace-period=0

• Then either add the missing label to another node or change the affinity to a preferred rule.

Operations Checklist

Use this checklist before and after making node affinity changes in production.

Before applying changes

• [ ] Run kubectl get nodes --show-labels and record the current labels of candidate nodes.
• [ ] Run kubectl get pods -o wide in the target namespace to see existing placement.
• [ ] Check if the scheduler is healthy: kubectl get pods -n kube-system -l component=kube-scheduler
• [ ] Review taints on nodes that should match: kubectl describe nodes | grep Taints
• [ ] Decide if you need required or preferred affinity. Prefer preferred for high availability unless you have a strict hardware requirement.
• [ ] Write the YAML manifest and validate it with kubectl apply --dry-run=client -f manifest.yaml

After applying changes

• [ ] Verify pods are scheduled: kubectl get pods -o wide
• [ ] Check events for FailedScheduling warnings: kubectl describe pod <pod-name>
• [ ] If using a Deployment, confirm rollout success: kubectl rollout status deployment/<deployment-name>
• [ ] Test behavior by cordoning a matching node and seeing if pods reschedule according to affinity rules (if preferred) or stay pending (if required).
• [ ] Document the change in your runbook with the exact labels and affinity terms used.

Quick reference for common affinity operators

Note: The Gt and Lt operators are not valid for label-based matchExpressions in node affinity; they are used in node selector terms with matchFields for metadata.name? Actually, they are not supported at all in node affinity. Stick to In, NotIn, Exists, DoesNotExist.

Conclusion

Node affinity gives you fine-grained control over pod placement while maintaining clarity in your manifests. By using labels, match expressions, and weights, you can build scheduling policies that reflect your infrastructure topology and application requirements.

Always start with a small test, verify scheduling events, and prefer soft rules unless a hard constraint is necessary. Combine node affinity with tolerations, resource requests, and topology spread constraints to create resilient, efficient deployments.

To deepen your understanding, experiment with different match expressions, practice with taints and tolerations, and monitor scheduler behavior using the metrics endpoint. Node affinity is a foundational tool in the Kubernetes scheduler, and mastering it will make you more effective at managing production clusters.