## Intro

Kubernetes taints and tolerations are the primary mechanism for controlling which pods can be scheduled onto which nodes. A taint on a node repels pods that do not tolerate it, while a toleration on a pod allows it to be scheduled onto nodes with matching taints. This article provides a practical, version-aware guide to upgrading and migrating taints and tolerations in a running Kubernetes cluster. Whether you are moving to a new Kubernetes version, restructuring node pools, or enforcing new scheduling policies, the steps below help you observe, plan, execute, verify, and roll back changes with minimal disruption.

We focus on real operational workflows: checking your current version and taint configuration, planning a migration that limits the blast radius, applying changes in a controlled way, validating that scheduling behaves as expected, and recovering if something goes wrong. Every section includes concrete commands, expected outputs, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

## Version and Environment Inventory

Before touching any taint or toleration, you need a complete inventory of your cluster version, node taints, pod tolerations, and relevant deployment topology. This establishes a baseline and helps you anticipate compatibility issues. Start by collecting the Kubernetes server version and checking the API deprecations that matter for taints and tolerations.

Check cluster and node versions

kubectl version --short 
 Expected output (example):

Client Version: v1.24.3
Server Version: v1.23.8 
 Taints and tolerations have been stable since Kubernetes v1.6, but newer versions may deprecate certain taint effects or add new ones. For example, the node.kubernetes.io/not-ready and node.kubernetes.io/unreachable taints have existed since early versions, but the node.kubernetes.io/unschedulable taint is automatically added by the scheduler. Always verify that your target version supports the taint effects you plan to use ( NoSchedule , PreferNoSchedule , or NoExecute ).

List current node taints

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints 
 Example output:

NAME TAINTS
node-1 [map[effect:NoSchedule key:dedicated value:database]]
node-2 <none>
node-3 [map[effect:PreferNoSchedule key:gpu value:true]] 
 This shows which nodes have taints and what their key, value, and effect are. Note that the output is not always easy to read; for a cleaner view, use kubectl describe nodes and look for the Taints field.

List tolerations on existing pods

To understand which pods already tolerate which taints, inspect their YAML or use a JSONPath query:

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}{"/"}{.metadata.name}{"\t"}{.spec.tolerations}{"\n"}{end}' 
 This may output something like:

default/nginx-deployment-7d4b8d6c9-abcde [map[key:dedicated value:database effect:NoSchedule]]
kube-system/coredns-558bd4d5db-xyz12 <no tolerations> 
 A pod without tolerations cannot be scheduled onto a node with a NoSchedule taint. If you are introducing a new taint, you must ensure that all pods that need to run on that node have a matching toleration.

Check node conditions and events

Use kubectl describe node <node-name> to see conditions and recent events that may affect scheduling. For a problematic pod, use:

kubectl describe pod <pod-name> -n <namespace> 
 Look for events like:

Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Warning FailedScheduling 10s default-scheduler 0/3 nodes are available: 1 node(s) had taint {dedicated: database}, that the pod didn't tolerate, 2 Insufficient cpu. 
 This tells you exactly which taint is blocking scheduling. Also check logs of the scheduler or kube-controller-manager if you suspect a problem with taint propagation.

Prerequisites checklist

- kubectl installed and configured with appropriate RBAC permissions (get, list, patch nodes and pods).

- jq for processing JSON output (optional but useful).

- A test namespace or cluster where you can validate changes before production.

- Backup of current node and pod configurations, especially if you manage them via GitOps.

Small local test

Before making any cluster-wide change, test in a local environment (e.g., kind , minikube , or a dedicated test namespace). Create a single node with a taint and a pod with a toleration to confirm the behavior:

# Create a single-node cluster with kind
kind create cluster --config kind-config.yaml
# where kind-config.yaml defines one control-plane node 
 Then apply a taint and a pod:

kubectl taint nodes kind-control-plane dedicated=database:NoSchedule
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
 name: test-pod
spec:
 tolerations:
 - key: "dedicated"
 operator: "Equal"
 value: "database"
 effect: "NoSchedule"
 containers:
 - name: nginx
 image: nginx
EOF 
 Verify the pod is scheduled:

kubectl get pod test-pod 
 If the pod is Pending , run kubectl describe pod test-pod to see the taint error.

<style>
.eno-quiz-widget{margin:2rem 0;padding:1.5rem;border-radius:12px;background:var(--eno-surface-lowest,#f7f7f8);border:1px solid var(--eno-border-soft,#e2e2e6);font-family:var(--eno-font-body,Inter,sans-serif)}
.eno-quiz-widget .eno-quiz-kicker{font-family:var(--eno-font-label,"JetBrains Mono",monospace);font-size:.75rem;letter-spacing:.05em;text-transform:uppercase;color:var(--eno-text-muted,#5f5f68);margin:0 0 .5rem}
.eno-quiz-widget .eno-quiz-question{font-family:var(--eno-font-heading,"Hanken Grotesk",sans-serif);font-size:1.0625rem;font-weight:600;margin:0 0 1rem;color:var(--eno-text-strong,#1a1a1f)}
.eno-quiz-widget .eno-quiz-options{list-style:none;margin:0;padding:0;display:flex;flex-direction:column;gap:.5rem}
.eno-quiz-widget .eno-quiz-option{display:block;width:100%;min-height:44px;text-align:left;padding:.625rem .875rem;border-radius:8px;border:1.5px solid var(--eno-border-soft,#e2e2e6);background:#fff;font-size:.9375rem;cursor:pointer;transition:border-color var(--eno-motion-base,180ms ease-out),background var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-option:hover{border-color:var(--eno-primary,#0059bb)}
.eno-quiz-widget .eno-quiz-option:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-option[aria-pressed="true"]{border-color:var(--eno-primary,#0059bb);background:rgb(0 89 187 / 0.06)}
.eno-quiz-widget .eno-quiz-option[data-correct="true"].eno-quiz-revealed{border-color:var(--eno-success,#17803a);background:rgb(23 128 58 / 0.08)}
.eno-quiz-widget .eno-quiz-option[data-correct="false"].eno-quiz-revealed.eno-quiz-was-selected{border-color:var(--eno-error,#ba1a1a);background:var(--eno-error-soft,#ffdad6)}
.eno-quiz-widget .eno-quiz-option-icon{display:inline-block;width:1.1em;margin-right:.4em;font-weight:700}
.eno-quiz-widget .eno-quiz-submit{margin-top:1rem;min-height:44px;padding:.5rem 1.25rem;border-radius:8px;border:none;background:var(--eno-primary,#0059bb);color:#fff;font-weight:600;font-size:.9375rem;cursor:pointer;transition:opacity var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-submit:disabled{opacity:.5;cursor:not-allowed}
.eno-quiz-widget .eno-quiz-submit:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-explanation{margin-top:1rem;padding:.875rem 1rem;border-radius:8px;font-size:.9375rem;line-height:1.5;display:none}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-visible{display:block}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-correct{background:rgb(23 128 58 / 0.08);color:var(--eno-success,#17803a)}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-incorrect{background:var(--eno-error-soft,#ffdad6);color:var(--eno-error,#ba1a1a)}
@media (prefers-reduced-motion: reduce){.eno-quiz-widget *{transition:none!important}}
</style><div class="eno-quiz-widget" role="group" aria-label="Quick check question">
Quick check 1 of 2

What is the effect of a `NoExecute` taint on pods that are already running on the node and do not tolerate it?

- They are evicted immediately.
- They remain bound forever.
- They remain bound for a specified time.
- They are not evicted.

<button type="button" class="eno-quiz-submit" disabled>Submit</button>
<div class="eno-quiz-explanation">Pods that do not tolerate a NoExecute taint are evicted immediately from the node.</div>
</div>
<script>
document.addEventListener('DOMContentLoaded', function () {
  document.querySelectorAll('.eno-quiz-widget:not([data-eno-quiz-bound])').forEach(function (widget) {
    widget.setAttribute('data-eno-quiz-bound', '1');
    var options = Array.prototype.slice.call(widget.querySelectorAll('.eno-quiz-option'));
    var submitBtn = widget.querySelector('.eno-quiz-submit');
    var explanation = widget.querySelector('.eno-quiz-explanation');
    var selected = null;
    options.forEach(function (opt) {
      opt.addEventListener('click', function () {
        if (widget.hasAttribute('data-eno-quiz-answered')) return;
        options.forEach(function (o) { o.setAttribute('aria-pressed', 'false'); });
        opt.setAttribute('aria-pressed', 'true');
        selected = opt;
        submitBtn.disabled = false;
      });
    });
    submitBtn.addEventListener('click', function () {
      if (!selected || widget.hasAttribute('data-eno-quiz-answered')) return;
      widget.setAttribute('data-eno-quiz-answered', '1');
      submitBtn.disabled = true;
      var correct = selected.getAttribute('data-correct') === 'true';
      options.forEach(function (o) {
        o.classList.add('eno-quiz-revealed');
        if (o === selected) o.classList.add('eno-quiz-was-selected');
        var icon = o.querySelector('.eno-quiz-option-icon');
        if (o.getAttribute('data-correct') === 'true') icon.textContent = '\u2713';
        else if (o === selected) icon.textContent = '\u2717';
      });
      explanation.classList.add('eno-quiz-visible', correct ? 'eno-quiz-correct' : 'eno-quiz-incorrect');
    });
  });
});
</script>

## Safe Configuration Path

Once you have an inventory, plan your taint migration with a safe path that minimizes impact. The core principle: change one thing at a time, observe, and have a rollback plan. This section outlines a step-by-step procedure for upgrading taints and tolerations, including version compatibility considerations.

### Step 1: Identify the target taint set

Decide what taints you need on which nodes. For example, you may want to dedicate a set of nodes to a specific workload by adding a taint workload=ml:NoSchedule . Or you may want to remove an obsolete taint. Document the desired state for each node group.

### Step 2: Check version compatibility

If you are also upgrading Kubernetes, ensure the taint behavior is unchanged. Consult the Kubernetes release notes for any deprecations related to taints. For instance, taints with NoExecute effect can evict already-running pods that do not tolerate them; if you add such a taint, be prepared for eviction.

### Step 3: Add tolerations before taints

The safest order is to add tolerations to the pods that should remain on a node before adding the taint to that node. This avoids a window where pods are unschedulable or evicted. For a Deployment, update the pod template:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: ml-workload
spec:
 replicas: 2
 selector:
 matchLabels:
 app: ml-workload
 template:
 metadata:
 labels:
 app: ml-workload
 spec:
 tolerations:
 - key: "workload"
 operator: "Equal"
 value: "ml"
 effect: "NoSchedule"
 containers:
 - name: trainer
 image: my-ml-image:latest 
 Apply the change and wait for the rollout:

kubectl apply -f deployment.yaml
kubectl rollout status deployment/ml-workload 
 Expected output:

deployment "ml-workload" successfully rolled out 

### Step 4: Add taints to nodes incrementally

 Start with one node in a node pool (if you have multiple nodes) to limit the blast radius. Use kubectl taint to add the taint:

kubectl taint nodes node-1 workload=ml:NoSchedule 
 Verify the taint was added:

kubectl get node node-1 -o jsonpath='{.spec.taints}' 
 Expected output:

[{"effect":"NoSchedule","key":"workload","value":"ml"}] 
 Now check that your intended pods are still running and new pods with tolerations can schedule. Use kubectl get pods -o wide to see node assignments.

### Step 5: Monitor scheduling

If you added a NoExecute taint, Kubernetes will evict non-tolerating pods immediately. To avoid surprise evictions, you can first add the taint with NoSchedule , then later change to NoExecute after confirming no critical pods are affected. To change an existing taint, you must remove and re-add it:

kubectl taint nodes node-1 workload=ml:NoSchedule-
kubectl taint nodes node-1 workload=ml:NoExecute 
 This is a destructive sequence, so be cautious. Always have a backup of the original taint if you need to revert.

### Step 6: Validate with read-only checks

Before declaring success, run a series of diagnostic commands (covered in the next section) to confirm that only intended pods are scheduled on tainted nodes and that no unintended evictions occurred.

### Small local test for safe configuration

In your local cluster, simulate a migration: create two nodes, add a taint to one, deploy a pod without toleration and a pod with toleration, and observe scheduling. Use kubectl cordon and kubectl drain if you need to move pods before adding taints. For example, to drain a node before adding a taint:

kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data 
 Then add the taint and uncordon:

kubectl taint nodes node-1 dedicated=test:NoSchedule
kubectl uncordon node-1 
 This ensures that no existing pods are stuck on the node when the taint takes effect.

## Verification and Diagnostics

After applying taints or tolerations, you must verify that the scheduling behavior matches your expectations. This section provides a comprehensive set of commands and checks to diagnose issues.

### Confirm taint application

List all node taints in a readable format:

kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, taints: .spec.taints}' 
 Example output:

{
 "name": "node-1",
 "taints": [
 {
 "effect": "NoSchedule",
 "key": "workload",
 "value": "ml"
 }
 ]
}
{
 "name": "node-2",
 "taints": null
} 

### Check pod scheduling

 Use kubectl get pods -o wide to see which node each pod is on. For a specific pod, describe it to see events:

kubectl describe pod <pod-name> 
 Look for the Events section. A successful scheduling event shows:

Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal Scheduled 5s default-scheduler Successfully assigned default/ml-workload-abcde to node-1 
 If unscheduled, you will see a FailedScheduling event with the reason. For example:

Warning FailedScheduling 2s default-scheduler 0/3 nodes are available: 1 node(s) had taint {workload: ml}, that the pod didn't tolerate, 2 Insufficient memory. 

### Inspect tolerations on running pods

 To verify that a pod has the correct toleration, use:

kubectl get pod ml-workload-abcde -o jsonpath='{.spec.tolerations}' 
 Expected output:

[{"effect":"NoSchedule","key":"workload","operator":"Equal","value":"ml"}] 

### Test with a dry-run

 You can use kubectl apply --dry-run=server to validate manifest changes without actually applying them:

kubectl apply -f deployment.yaml --dry-run=server 
 This catches syntax errors or invalid toleration formats.

### Use node labels alongside taints

Taints are about repelling pods; node labels are about attracting pods via node selectors or affinity. To ensure pods land on the right nodes, combine taints with node labels and pod node selectors. For example, label the node:

kubectl label nodes node-1 workload=ml 
 Then in the pod spec, add:

nodeSelector:
 workload: ml 
 This makes scheduling more deterministic.

### Check for evictions

If you used NoExecute , monitor for evicted pods:

kubectl get events --all-namespaces --field-selector reason=Evicted 
 Example output:

NAMESPACE LAST SEEN TYPE REASON OBJECT MESSAGE
default 1m Warning Evicted pod/old-pod The pod was evicted due to a taint 
 If you see unexpected evictions, immediately remove the taint or add tolerations to the affected pods.

<style>
.eno-quiz-widget{margin:2rem 0;padding:1.5rem;border-radius:12px;background:var(--eno-surface-lowest,#f7f7f8);border:1px solid var(--eno-border-soft,#e2e2e6);font-family:var(--eno-font-body,Inter,sans-serif)}
.eno-quiz-widget .eno-quiz-kicker{font-family:var(--eno-font-label,"JetBrains Mono",monospace);font-size:.75rem;letter-spacing:.05em;text-transform:uppercase;color:var(--eno-text-muted,#5f5f68);margin:0 0 .5rem}
.eno-quiz-widget .eno-quiz-question{font-family:var(--eno-font-heading,"Hanken Grotesk",sans-serif);font-size:1.0625rem;font-weight:600;margin:0 0 1rem;color:var(--eno-text-strong,#1a1a1f)}
.eno-quiz-widget .eno-quiz-options{list-style:none;margin:0;padding:0;display:flex;flex-direction:column;gap:.5rem}
.eno-quiz-widget .eno-quiz-option{display:block;width:100%;min-height:44px;text-align:left;padding:.625rem .875rem;border-radius:8px;border:1.5px solid var(--eno-border-soft,#e2e2e6);background:#fff;font-size:.9375rem;cursor:pointer;transition:border-color var(--eno-motion-base,180ms ease-out),background var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-option:hover{border-color:var(--eno-primary,#0059bb)}
.eno-quiz-widget .eno-quiz-option:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-option[aria-pressed="true"]{border-color:var(--eno-primary,#0059bb);background:rgb(0 89 187 / 0.06)}
.eno-quiz-widget .eno-quiz-option[data-correct="true"].eno-quiz-revealed{border-color:var(--eno-success,#17803a);background:rgb(23 128 58 / 0.08)}
.eno-quiz-widget .eno-quiz-option[data-correct="false"].eno-quiz-revealed.eno-quiz-was-selected{border-color:var(--eno-error,#ba1a1a);background:var(--eno-error-soft,#ffdad6)}
.eno-quiz-widget .eno-quiz-option-icon{display:inline-block;width:1.1em;margin-right:.4em;font-weight:700}
.eno-quiz-widget .eno-quiz-submit{margin-top:1rem;min-height:44px;padding:.5rem 1.25rem;border-radius:8px;border:none;background:var(--eno-primary,#0059bb);color:#fff;font-weight:600;font-size:.9375rem;cursor:pointer;transition:opacity var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-submit:disabled{opacity:.5;cursor:not-allowed}
.eno-quiz-widget .eno-quiz-submit:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-explanation{margin-top:1rem;padding:.875rem 1rem;border-radius:8px;font-size:.9375rem;line-height:1.5;display:none}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-visible{display:block}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-correct{background:rgb(23 128 58 / 0.08);color:var(--eno-success,#17803a)}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-incorrect{background:var(--eno-error-soft,#ffdad6);color:var(--eno-error,#ba1a1a)}
@media (prefers-reduced-motion: reduce){.eno-quiz-widget *{transition:none!important}}
</style><div class="eno-quiz-widget" role="group" aria-label="Quick check question">
Quick check 2 of 2

What command is used to add a taint to a node?

- kubectl add taint
- kubectl taint
- kubectl set taint
- kubectl patch node

<button type="button" class="eno-quiz-submit" disabled>Submit</button>
<div class="eno-quiz-explanation">The correct command to add a taint to a node is `kubectl taint nodes node1 key1=value1:NoSchedule`.</div>
</div>
<script>
document.addEventListener('DOMContentLoaded', function () {
  document.querySelectorAll('.eno-quiz-widget:not([data-eno-quiz-bound])').forEach(function (widget) {
    widget.setAttribute('data-eno-quiz-bound', '1');
    var options = Array.prototype.slice.call(widget.querySelectorAll('.eno-quiz-option'));
    var submitBtn = widget.querySelector('.eno-quiz-submit');
    var explanation = widget.querySelector('.eno-quiz-explanation');
    var selected = null;
    options.forEach(function (opt) {
      opt.addEventListener('click', function () {
        if (widget.hasAttribute('data-eno-quiz-answered')) return;
        options.forEach(function (o) { o.setAttribute('aria-pressed', 'false'); });
        opt.setAttribute('aria-pressed', 'true');
        selected = opt;
        submitBtn.disabled = false;
      });
    });
    submitBtn.addEventListener('click', function () {
      if (!selected || widget.hasAttribute('data-eno-quiz-answered')) return;
      widget.setAttribute('data-eno-quiz-answered', '1');
      submitBtn.disabled = true;
      var correct = selected.getAttribute('data-correct') === 'true';
      options.forEach(function (o) {
        o.classList.add('eno-quiz-revealed');
        if (o === selected) o.classList.add('eno-quiz-was-selected');
        var icon = o.querySelector('.eno-quiz-option-icon');
        if (o.getAttribute('data-correct') === 'true') icon.textContent = '\u2713';
        else if (o === selected) icon.textContent = '\u2717';
      });
      explanation.classList.add('eno-quiz-visible', correct ? 'eno-quiz-correct' : 'eno-quiz-incorrect');
    });
  });
});
</script>

## Failure Modes and Recovery

Even with careful planning, taint changes can go wrong. This section describes common failure modes and how to recover from them.

### Failure mode 1: Pod stuck in Pending due to taint

Symptom: A pod remains Pending with a FailedScheduling event indicating taint mismatch.

Diagnosis: Describe the pod and check events. Also verify the node taints and pod tolerations.

Recovery:

- If the pod should tolerate the taint, add the appropriate toleration to the pod spec and reapply.

- If the taint was added by mistake, remove it from the node:

kubectl taint nodes node-1 workload=ml:NoSchedule- 
 Note the minus sign at the end, which removes the taint.

- If the pod is part of a controller (Deployment, StatefulSet), update the template and rollout.

### Failure mode 2: Unintended eviction of critical pods

Symptom: After adding a NoExecute taint, critical system pods (e.g., kube-proxy, DNS) get evicted.

Diagnosis: Check kubectl get pods -n kube-system and events. System pods may lack tolerations for custom taints.

Recovery:

- Immediately remove the taint:

kubectl taint nodes node-1 workload=ml:NoExecute- 

- Add necessary tolerations to critical pods or ensure they are in a daemonset that tolerates all taints.

- Alternatively, use the PreferNoSchedule effect to avoid hard eviction.

### Failure mode 3: Taint removal not working

 Symptom: You run kubectl taint nodes node-1 key=value:Effect- , but the taint remains.

Diagnosis: Check the taint key and value exactly match what is on the node. Taints are exact match for key and value. If you omit the value for an Equal operator taint, it won't match.

Recovery:

- Get the exact taint:

kubectl get node node-1 -o json | jq '.spec.taints' 

- Remove with the exact key, value, and effect:

 kubectl taint nodes node-1 dedicated=database:NoSchedule- 

### Failure mode 4: Version incompatibility

 Symptom: After upgrading Kubernetes, taints behave differently or errors occur.

Diagnosis: Check API versions and logs. Some taint features may have changed.

Recovery:

- Roll back the Kubernetes upgrade if possible.

- Adjust taints and tolerations to match the new version's expectations. For example, in newer versions, the node.kubernetes.io/not-ready taint has a different default toleration time for pods.

- Consult the Kubernetes changelog for breaking changes.

### General rollback procedure

Always keep a record of original taints and tolerations. If using GitOps, revert the commit. For manual changes, document each command and its inverse. For example, to roll back a taint addition, run the removal command. To roll back a toleration addition, edit the pod spec and remove the toleration.

If a change causes significant disruption, you can also cordon and drain affected nodes, remove the taint, and then uncordon.

kubectl cordon node-1
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
kubectl taint nodes node-1 workload=ml:NoSchedule-
kubectl uncordon node-1 
 This forces all pods off the node, removes the taint, and brings it back into rotation.

## Operations Checklist

Use this checklist for every taint and toleration upgrade or migration. It summarizes the key steps and includes concrete verification commands.

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Step</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Action</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Command / Verification</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Expected Result</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">1</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Record current cluster version and node taints</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl version --short</code> and <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Version listed, taints visible</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">2</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Identify target taints and tolerations</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Document desired state</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Clear list of changes</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Add tolerations to pods before adding taints</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Edit Deployment or Pod spec, apply, <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl rollout status</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Rollout successful</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">4</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Add taint to a single test node</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl taint nodes &lt;node&gt; key=value:NoSchedule</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Taint appears in node spec</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">5</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Verify pod scheduling on test node</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl get pods -o wide</code> and <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl describe pod</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Pod scheduled on intended node</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">6</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Monitor for evictions</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">kubectl get events --all-namespaces --field-selector reason=Evicted</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">No unexpected evictions</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">7</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Roll out to remaining nodes incrementally</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Repeat taint addition for each node, checking between</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">All nodes tainted, no issues</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">8</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Update documentation and runbooks</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Record actual commands and outputs</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Documentation current</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">9</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Schedule periodic validation</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">e.g., weekly script to compare node taints and pod tolerations</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Drift detected early</td></tr></tbody>
</table>
</div>

### Example validation script

You can create a simple script to validate that a set of nodes have the expected taints and pods have matching tolerations. Here is a snippet using bash and kubectl:

#!/bin/bash
# Check that all nodes with label workload=ml have taint workload=ml:NoSchedule
for node in $(kubectl get nodes -l workload=ml -o name); do
 taints=$(kubectl get $node -o jsonpath='{.spec.taints}')
 if [[ "$taints" != *"workload=ml:NoSchedule"* ]]; then
 echo "ERROR: $node missing taint"
 else
 echo "OK: $node has correct taint"
 fi
done

# Check that any pod with toleration for workload=ml is in Running state
for pod in $(kubectl get pods -A -o json | jq -r '.items[] | select(.spec.tolerations[]?.key=="workload") | .metadata.namespace+"/"+.metadata.name'); do
 ns=${pod%/*}
 name=${pod#*/}
 status=$(kubectl get pod $name -n $ns -o jsonpath='{.status.phase}')
 if [[ "$status" != "Running" ]]; then
 echo "WARNING: $pod in $ns is $status"
 else
 echo "OK: $pod running"
 fi
done 
 This script provides a quick automated check after changes.

## Conclusion

Kubernetes taints and tolerations are powerful but require careful management during upgrades and migrations. By following the version-aware, observation-first approach outlined in this article, you can minimize risk and ensure your cluster's scheduling policies remain correct and effective.

Key takeaways:

- Always inventory your current taints, tolerations, and cluster version before making changes.

- Add tolerations to pods before adding taints to nodes to avoid scheduling failures.

- Roll out changes incrementally, starting with a test node.

- Use kubectl describe and events to diagnose scheduling issues.

- Have a clear rollback plan and document every command.

- Validate with scripts and periodic checks to catch drift.

As a next step, choose one low-risk verification for your own cluster: record the current taints on a node, add a toleration to a test pod, and confirm it schedules as expected. Then review dependencies such as node labels, node selectors, and pod anti-affinity, because they interact with taints and tolerations.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.