Kubernetes Taints and Tolerations: Upgrade and Migration with Practical Examples

Intro

Kubernetes taints and tolerations are the primary mechanism for controlling which pods can be scheduled onto which nodes. A taint on a node repels pods that do not tolerate it, while a toleration on a pod allows it to be scheduled onto nodes with matching taints. This article provides a practical, version-aware guide to upgrading and migrating taints and tolerations in a running Kubernetes cluster. Whether you are moving to a new Kubernetes version, restructuring node pools, or enforcing new scheduling policies, the steps below help you observe, plan, execute, verify, and roll back changes with minimal disruption.

We focus on real operational workflows: checking your current version and taint configuration, planning a migration that limits the blast radius, applying changes in a controlled way, validating that scheduling behaves as expected, and recovering if something goes wrong. Every section includes concrete commands, expected outputs, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before touching any taint or toleration, you need a complete inventory of your cluster version, node taints, pod tolerations, and relevant deployment topology. This establishes a baseline and helps you anticipate compatibility issues. Start by collecting the Kubernetes server version and checking the API deprecations that matter for taints and tolerations.

Check cluster and node versions

kubectl version --short

Expected output (example):

Client Version: v1.24.3
Server Version: v1.23.8

Taints and tolerations have been stable since Kubernetes v1.6, but newer versions may deprecate certain taint effects or add new ones. For example, the node.kubernetes.io/not-ready and node.kubernetes.io/unreachable taints have existed since early versions, but the node.kubernetes.io/unschedulable taint is automatically added by the scheduler. Always verify that your target version supports the taint effects you plan to use (NoSchedule, PreferNoSchedule, or NoExecute).

List current node taints

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Example output:

NAME       TAINTS
node-1     [map[effect:NoSchedule key:dedicated value:database]]
node-2     <none>
node-3     [map[effect:PreferNoSchedule key:gpu value:true]]

This shows which nodes have taints and what their key, value, and effect are. Note that the output is not always easy to read; for a cleaner view, use kubectl describe nodes and look for the Taints field.

List tolerations on existing pods

To understand which pods already tolerate which taints, inspect their YAML or use a JSONPath query:

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.spec.tolerations}{"\n"}{end}'

This may output something like:

default/nginx-deployment-7d4b8d6c9-abcde   [map[key:dedicated value:database effect:NoSchedule]]
kube-system/coredns-558bd4d5db-xyz12      <no tolerations>

A pod without tolerations cannot be scheduled onto a node with a NoSchedule taint. If you are introducing a new taint, you must ensure that all pods that need to run on that node have a matching toleration.

Check node conditions and events

Use kubectl describe node <node-name> to see conditions and recent events that may affect scheduling. For a problematic pod, use:

kubectl describe pod <pod-name> -n <namespace>

Look for events like:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  10s   default-scheduler  0/3 nodes are available: 1 node(s) had taint {dedicated: database}, that the pod didn't tolerate, 2 Insufficient cpu.

This tells you exactly which taint is blocking scheduling. Also check logs of the scheduler or kube-controller-manager if you suspect a problem with taint propagation.

Prerequisites checklist

• kubectl installed and configured with appropriate RBAC permissions (get, list, patch nodes and pods).
• jq for processing JSON output (optional but useful).
• A test namespace or cluster where you can validate changes before production.
• Backup of current node and pod configurations, especially if you manage them via GitOps.

Small local test

Before making any cluster-wide change, test in a local environment (e.g., kind, minikube, or a dedicated test namespace). Create a single node with a taint and a pod with a toleration to confirm the behavior:

# Create a single-node cluster with kind
kind create cluster --config kind-config.yaml
# where kind-config.yaml defines one control-plane node

Then apply a taint and a pod:

kubectl taint nodes kind-control-plane dedicated=database:NoSchedule
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  tolerations:
  - key: "dedicated"
    operator: "Equal"
    value: "database"
    effect: "NoSchedule"
  containers:
  - name: nginx
    image: nginx
EOF

Verify the pod is scheduled:

kubectl get pod test-pod

If the pod is Pending, run kubectl describe pod test-pod to see the taint error.

Safe Configuration Path

Once you have an inventory, plan your taint migration with a safe path that minimizes impact. The core principle: change one thing at a time, observe, and have a rollback plan. This section outlines a step-by-step procedure for upgrading taints and tolerations, including version compatibility considerations.

Step 1: Identify the target taint set

Decide what taints you need on which nodes. For example, you may want to dedicate a set of nodes to a specific workload by adding a taint workload=ml:NoSchedule. Or you may want to remove an obsolete taint. Document the desired state for each node group.

Step 2: Check version compatibility

If you are also upgrading Kubernetes, ensure the taint behavior is unchanged. Consult the Kubernetes release notes for any deprecations related to taints. For instance, taints with NoExecute effect can evict already-running pods that do not tolerate them; if you add such a taint, be prepared for eviction.

Step 3: Add tolerations before taints

The safest order is to add tolerations to the pods that should remain on a node before adding the taint to that node. This avoids a window where pods are unschedulable or evicted. For a Deployment, update the pod template:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ml-workload
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ml-workload
  template:
    metadata:
      labels:
        app: ml-workload
    spec:
      tolerations:
      - key: "workload"
        operator: "Equal"
        value: "ml"
        effect: "NoSchedule"
      containers:
      - name: trainer
        image: my-ml-image:latest

Apply the change and wait for the rollout:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/ml-workload

Expected output:

deployment "ml-workload" successfully rolled out

Step 4: Add taints to nodes incrementally

Start with one node in a node pool (if you have multiple nodes) to limit the blast radius. Use kubectl taint to add the taint:

kubectl taint nodes node-1 workload=ml:NoSchedule

Verify the taint was added:

kubectl get node node-1 -o jsonpath='{.spec.taints}'

Expected output:

[{"effect":"NoSchedule","key":"workload","value":"ml"}]

Now check that your intended pods are still running and new pods with tolerations can schedule. Use kubectl get pods -o wide to see node assignments.

Step 5: Monitor scheduling

If you added a NoExecute taint, Kubernetes will evict non-tolerating pods immediately. To avoid surprise evictions, you can first add the taint with NoSchedule, then later change to NoExecute after confirming no critical pods are affected. To change an existing taint, you must remove and re-add it:

kubectl taint nodes node-1 workload=ml:NoSchedule-
kubectl taint nodes node-1 workload=ml:NoExecute

This is a destructive sequence, so be cautious. Always have a backup of the original taint if you need to revert.

Step 6: Validate with read-only checks

Before declaring success, run a series of diagnostic commands (covered in the next section) to confirm that only intended pods are scheduled on tainted nodes and that no unintended evictions occurred.

Small local test for safe configuration

In your local cluster, simulate a migration: create two nodes, add a taint to one, deploy a pod without toleration and a pod with toleration, and observe scheduling. Use kubectl cordon and kubectl drain if you need to move pods before adding taints. For example, to drain a node before adding a taint:

kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

Then add the taint and uncordon:

kubectl taint nodes node-1 dedicated=test:NoSchedule
kubectl uncordon node-1

This ensures that no existing pods are stuck on the node when the taint takes effect.

Verification and Diagnostics

After applying taints or tolerations, you must verify that the scheduling behavior matches your expectations. This section provides a comprehensive set of commands and checks to diagnose issues.

Confirm taint application

List all node taints in a readable format:

kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, taints: .spec.taints}'

Example output:

{
  "name": "node-1",
  "taints": [
    {
      "effect": "NoSchedule",
      "key": "workload",
      "value": "ml"
    }
  ]
}
{
  "name": "node-2",
  "taints": null
}

Check pod scheduling

Use kubectl get pods -o wide to see which node each pod is on. For a specific pod, describe it to see events:

kubectl describe pod <pod-name>

Look for the Events section. A successful scheduling event shows:

Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  5s    default-scheduler  Successfully assigned default/ml-workload-abcde to node-1

If unscheduled, you will see a FailedScheduling event with the reason. For example:

Warning  FailedScheduling  2s   default-scheduler  0/3 nodes are available: 1 node(s) had taint {workload: ml}, that the pod didn't tolerate, 2 Insufficient memory.

Inspect tolerations on running pods

To verify that a pod has the correct toleration, use:

kubectl get pod ml-workload-abcde -o jsonpath='{.spec.tolerations}'

Expected output:

[{"effect":"NoSchedule","key":"workload","operator":"Equal","value":"ml"}]

Test with a dry-run

You can use kubectl apply --dry-run=server to validate manifest changes without actually applying them:

kubectl apply -f deployment.yaml --dry-run=server

This catches syntax errors or invalid toleration formats.

Use node labels alongside taints

Taints are about repelling pods; node labels are about attracting pods via node selectors or affinity. To ensure pods land on the right nodes, combine taints with node labels and pod node selectors. For example, label the node:

kubectl label nodes node-1 workload=ml

Then in the pod spec, add:

nodeSelector:
  workload: ml

This makes scheduling more deterministic.

Check for evictions

If you used NoExecute, monitor for evicted pods:

kubectl get events --all-namespaces --field-selector reason=Evicted

Example output:

NAMESPACE   LAST SEEN   TYPE      REASON    OBJECT                        MESSAGE
default     1m          Warning   Evicted   pod/old-pod                   The pod was evicted due to a taint

If you see unexpected evictions, immediately remove the taint or add tolerations to the affected pods.

Failure Modes and Recovery

Even with careful planning, taint changes can go wrong. This section describes common failure modes and how to recover from them.

Failure mode 1: Pod stuck in Pending due to taint

Symptom: A pod remains Pending with a FailedScheduling event indicating taint mismatch.

Diagnosis: Describe the pod and check events. Also verify the node taints and pod tolerations.

Recovery:

1. If the pod should tolerate the taint, add the appropriate toleration to the pod spec and reapply.
2. If the taint was added by mistake, remove it from the node:

kubectl taint nodes node-1 workload=ml:NoSchedule-

Note the minus sign at the end, which removes the taint.

1. If the pod is part of a controller (Deployment, StatefulSet), update the template and rollout.

Failure mode 2: Unintended eviction of critical pods

Symptom: After adding a NoExecute taint, critical system pods (e.g., kube-proxy, DNS) get evicted.

Diagnosis: Check kubectl get pods -n kube-system and events. System pods may lack tolerations for custom taints.

Recovery:

1. Immediately remove the taint:

kubectl taint nodes node-1 workload=ml:NoExecute-

1. Add necessary tolerations to critical pods or ensure they are in a daemonset that tolerates all taints.
2. Alternatively, use the PreferNoSchedule effect to avoid hard eviction.

Failure mode 3: Taint removal not working

Symptom: You run kubectl taint nodes node-1 key=value:Effect-, but the taint remains.

Diagnosis: Check the taint key and value exactly match what is on the node. Taints are exact match for key and value. If you omit the value for an Equal operator taint, it won't match.

Recovery:

1. Get the exact taint:

kubectl get node node-1 -o json | jq '.spec.taints'

1. Remove with the exact key, value, and effect:

kubectl taint nodes node-1 dedicated=database:NoSchedule-

Failure mode 4: Version incompatibility

Symptom: After upgrading Kubernetes, taints behave differently or errors occur.

Diagnosis: Check API versions and logs. Some taint features may have changed.

Recovery:

1. Roll back the Kubernetes upgrade if possible.
2. Adjust taints and tolerations to match the new version's expectations. For example, in newer versions, the node.kubernetes.io/not-ready taint has a different default toleration time for pods.
3. Consult the Kubernetes changelog for breaking changes.

General rollback procedure

Always keep a record of original taints and tolerations. If using GitOps, revert the commit. For manual changes, document each command and its inverse. For example, to roll back a taint addition, run the removal command. To roll back a toleration addition, edit the pod spec and remove the toleration.

If a change causes significant disruption, you can also cordon and drain affected nodes, remove the taint, and then uncordon.

kubectl cordon node-1
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
kubectl taint nodes node-1 workload=ml:NoSchedule-
kubectl uncordon node-1

This forces all pods off the node, removes the taint, and brings it back into rotation.

Operations Checklist

Use this checklist for every taint and toleration upgrade or migration. It summarizes the key steps and includes concrete verification commands.

Example validation script

You can create a simple script to validate that a set of nodes have the expected taints and pods have matching tolerations. Here is a snippet using bash and kubectl:

#!/bin/bash
# Check that all nodes with label workload=ml have taint workload=ml:NoSchedule
for node in $(kubectl get nodes -l workload=ml -o name); do
  taints=$(kubectl get $node -o jsonpath='{.spec.taints}')
  if [[ "$taints" != *"workload=ml:NoSchedule"* ]]; then
    echo "ERROR: $node missing taint"
  else
    echo "OK: $node has correct taint"
  fi
done

# Check that any pod with toleration for workload=ml is in Running state
for pod in $(kubectl get pods -A -o json | jq -r '.items[] | select(.spec.tolerations[]?.key=="workload") | .metadata.namespace+"/"+.metadata.name'); do
  ns=${pod%/*}
  name=${pod#*/}
  status=$(kubectl get pod $name -n $ns -o jsonpath='{.status.phase}')
  if [[ "$status" != "Running" ]]; then
    echo "WARNING: $pod in $ns is $status"
  else
    echo "OK: $pod running"
  fi
done

This script provides a quick automated check after changes.

Conclusion

Kubernetes taints and tolerations are powerful but require careful management during upgrades and migrations. By following the version-aware, observation-first approach outlined in this article, you can minimize risk and ensure your cluster's scheduling policies remain correct and effective.

Key takeaways:

• Always inventory your current taints, tolerations, and cluster version before making changes.
• Add tolerations to pods before adding taints to nodes to avoid scheduling failures.
• Roll out changes incrementally, starting with a test node.
• Use kubectl describe and events to diagnose scheduling issues.
• Have a clear rollback plan and document every command.
• Validate with scripts and periodic checks to catch drift.

As a next step, choose one low-risk verification for your own cluster: record the current taints on a node, add a toleration to a test pod, and confirm it schedules as expected. Then review dependencies such as node labels, node selectors, and pod anti-affinity, because they interact with taints and tolerations.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.