Kubernetes Taints and Tolerations: Production Operations Checklist and Practical Examples

Intro

Taints and tolerations are a core Kubernetes scheduling mechanism that lets you control which pods can be placed on which nodes. Taints are applied to nodes to repel pods that do not tolerate them, while tolerations are applied to pods to allow them to schedule onto tainted nodes. In production, a misconfigured taint or toleration can lead to pods stuck in Pending state, unschedulable workloads, or even accidentally scheduling critical pods onto unsuitable nodes.

This article provides a production-focused operations checklist for Kubernetes taints and tolerations. It covers version and environment inventory, safe configuration paths, verification and diagnostics, failure modes and recovery, and an end-to-end operations checklist. Each section includes concrete kubectl commands, expected outputs, and practical examples. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

This guide is intended for developers, DevOps consultants, and technical startup teams who need to manage taints and tolerations in production clusters. It assumes basic familiarity with Kubernetes concepts like pods, nodes, and Deployments.

Version and Environment Inventory

Before making any changes, you must know your Kubernetes version, cluster topology, and the exact state of your taints and tolerations. Taints and tolerations have evolved across Kubernetes versions; for example, the node.kubernetes.io/not-ready and node.kubernetes.io/unreachable taints behave differently depending on the version and the TaintBasedEvictions feature gate.

1. Check Kubernetes Version

Run the following command to check the server version:

kubectl version --short

Expected output (example):

Client Version: v1.25.3
Server Version: v1.25.3

Note: In newer versions, the --short flag is deprecated; use kubectl version without flags and read the output. If your client and server versions differ significantly, be aware that taint-related API fields might behave differently. For production, document the exact version in your runbook.

2. List All Nodes and Their Taints

To see all nodes and their taints:

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Expected output (example):

NAME            TAINTS
node-1          <none>
node-2          [{"effect":"NoSchedule","key":"dedicated","value":"database"}]
node-3          [{"effect":"NoExecute","key":"gpu","value":"true"}]

If a node shows <none> for taints, it has no taints and any pod without a matching toleration can schedule there. Look for taints with NoSchedule (pods without toleration will not be scheduled), PreferNoSchedule (soft preference), and NoExecute (evicts existing pods without toleration and prevents scheduling).

3. Describe a Specific Node

For detailed taint information and events:

kubectl describe node node-2

Look for the Taints: line in the output. Example:

Taints:             dedicated=database:NoSchedule

This tells you the taint key (dedicated), value (database), and effect (NoSchedule).

4. Check Existing Pods and Their Tolerations

To see which pods currently have tolerations:

kubectl get pods -A -o json | jq '.items[] | {name: .metadata.name, namespace: .metadata.namespace, tolerations: .spec.tolerations}'

Expected output (example):

{
  "name": "my-db-pod",
  "namespace": "default",
  "tolerations": [
    {
      "key": "dedicated",
      "operator": "Equal",
      "value": "database",
      "effect": "NoSchedule"
    }
  ]
}

If a pod does not list any tolerations, it will not schedule onto nodes with taints unless the taint has PreferNoSchedule effect (in which case the scheduler may still place it if no other nodes are available).

5. Environment Inventory Checklist

For production, record the following in your inventory:

• Kubernetes version (server and client)
• Node names and their taints
• Namespaces and workloads that rely on taints/tolerations
• Critical pods and their tolerations
• Any cluster autoscaler settings that interact with taints

Document this in a runbook before making changes.

Safe Configuration Path

When modifying taints or tolerations, always follow a safe configuration path: observe, plan, change one item, verify, and have a rollback plan.

1. Add a Taint to a Node

To add a taint to a node:

kubectl taint nodes node-1 dedicated=frontend:NoSchedule

Expected output:

node/node-1 tainted

This taint prevents any pod that lacks a matching toleration from being scheduled on node-1. After adding, verify with:

kubectl describe node node-1 | grep Taints

Expected output:

Taints:             dedicated=frontend:NoSchedule

2. Add a Toleration to a Pod Specification

Tolerations are added to the pod spec. For a Deployment, add them under spec.template.spec.tolerations. Example YAML snippet:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "frontend"
        effect: "NoSchedule"
      containers:
      - name: nginx
        image: nginx:1.21

Apply it with:

kubectl apply -f frontend-deployment.yaml

Then check rollout status:

kubectl rollout status deployment/frontend

Expected output:

deployment "frontend" successfully rolled out

3. Remove a Taint from a Node

To remove a taint, use the same key and effect with a minus sign at the end of the effect:

kubectl taint nodes node-1 dedicated=frontend:NoSchedule-

Expected output:

node/node-1 untainted

Verify with kubectl describe node node-1 and ensure the taint is gone.

4. Update Tolerations on an Existing Deployment

If you need to add tolerations to an existing deployment without recreating it, use kubectl patch:

kubectl patch deployment frontend --type='json' -p='[{"op": "add", "path": "/spec/template/spec/tolerations", "value": [{"key": "dedicated", "operator": "Equal", "value": "frontend", "effect": "NoSchedule"}]}]'

Then watch the rollout:

kubectl rollout status deployment/frontend

5. Safe Change Practices

• Always apply changes to a non-production environment first if possible.
• Use --dry-run=client to preview changes without applying them. For taints, you can simulate by using kubectl taint nodes node-1 dedicated=frontend:NoSchedule --dry-run=client (though taints are server-side mutations and dry-run may not fully apply; verify with kubectl auth can-i and test in a staging cluster).
• Document the exact command and its expected output before running.
• Have a rollback command ready (e.g., the untaint command).
• Limit the blast radius: change one node or one pod at a time, then observe.

Verification and Diagnostics

After making changes, you must verify that scheduling behaves as expected and diagnose any issues.

1. Check Pod Scheduling Status

List pods with their node assignments:

kubectl get pods -o wide

Expected output (example):

NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE
frontend-6f8c9d7b-abcde  1/1     Running   0          10m   10.244.1.5   node-1
frontend-6f8c9d7b-fghij  1/1     Running   0          10m   10.244.2.3   node-2

If pods are stuck in Pending, they may be unschedulable due to taints.

2. Describe a Pending Pod

Describe the pod to see scheduling events:

kubectl describe pod frontend-6f8c9d7b-abcde

Look for events like:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/3 nodes are available: 1 node(s) had taint {dedicated: frontend}, that the pod didn't tolerate, 2 node(s) had taint {gpu: true}, that the pod didn't tolerate.

This event indicates that the pod does not have the required tolerations for the taints on available nodes.

3. Verify Node Taints After Change

Use kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints again to confirm the taint is present or removed.

4. Check Logs for Startup Issues

If a pod is running but misbehaving, check its logs:

kubectl logs <pod-name> --previous

For example, if a pod crashed due to missing resources on a tainted node, logs might show errors.

5. Use Kubernetes Events

To see cluster-wide events related to scheduling:

kubectl get events --field-selector reason=FailedScheduling

This filters events for scheduling failures.

6. Simulate Scheduling with kubectl debug or Dry-Run

You can use the kubectl debug command to create a temporary pod with tolerations to test scheduling on a specific node. For example:

kubectl debug node/node-1 -it --image=busybox -- sh

This creates a debug pod on node-1, which may require tolerations if node-1 is tainted. Alternatively, use a dry-run manifest to see if the pod would be accepted:

kubectl apply -f test-pod.yaml --dry-run=client

This only validates the YAML, not scheduling. For actual scheduling simulation, use a tool like kube-scheduler-simulator or test in a staging cluster.

Failure Modes and Recovery

Misconfigured taints and tolerations can cause several failure modes. Here are common scenarios and recovery steps.

Failure Mode 1: Pod Stuck in Pending State Due to Missing Toleration

Symptom: kubectl get pods shows a pod with STATUS Pending for a long time. Describing the pod shows FailedScheduling events with messages about taints.

Recovery:

1. Identify the taints on nodes:

   kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

1. Determine which taint is blocking. Add the appropriate toleration to the pod spec. For a quick fix, patch the deployment:

   kubectl patch deployment <deployment-name> --type='json' -p='[{"op": "add", "path": "/spec/template/spec/tolerations", "value": [{"key": "<taint-key>", "operator": "Equal", "value": "<taint-value>", "effect": "<taint-effect>"}]}]'

1. Watch the new pods schedule:

   kubectl get pods -w

Failure Mode 2: Pod Evicted by NoExecute Taint

Symptom: A running pod suddenly terminates and is recreated repeatedly, or disappears from the node. Describing the pod may show an eviction event.

Recovery:

1. Check node taints: kubectl describe node <node-name>.
2. If a NoExecute taint was recently added, either add a toleration with tolerationSeconds or operator: Exists to the pod, or remove the taint from the node if it was unintended.
3. Remove the taint:

   kubectl taint nodes <node-name> <key>=<value>:NoExecute-

1. Verify the pod reschedules:

   kubectl get pods -o wide

Failure Mode 3: All Nodes Tainted and No Pods Can Schedule

Symptom: Every new pod remains Pending, and kubectl get nodes shows all nodes have taints without corresponding tolerations.

Recovery:

1. List taints:

   kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

1. Decide whether to remove taints from some nodes or add tolerations to critical workloads.
2. For critical system pods (e.g., kube-system), ensure they have appropriate tolerations; many system pods have built-in tolerations for common taints like node.kubernetes.io/not-ready.
3. Remove unnecessary taints or add a broad toleration:

   tolerations:
   - operator: "Exists"

Use with caution, as this allows the pod to schedule on any tainted node.

Failure Mode 4: Taint Removed but Pods Still Unschedulable

Symptom: After removing a taint, pods still cannot schedule on a node.

Possible Causes:

• The taint removal command had a typo or wrong effect.
• There are other taints on the node.
• The node is cordoned (kubectl cordon).
• Resource constraints (CPU/memory) prevent scheduling.

Recovery:

1. Verify node taints and cordon status:

   kubectl describe node <node-name> | grep -E 'Taints|Unschedulable'

1. If Unschedulable: true, uncordon the node:

   kubectl uncordon <node-name>

1. Check for other taints and remove them if appropriate.
2. Check resource requests and limits on pods and node allocatable capacity.

Rollback Strategies

Always have a rollback plan:

• For taints, keep the exact command to remove the taint (with trailing minus).
• For pod tolerations, keep the previous YAML or patch command to revert.
• Use version control for all manifests.

Operations Checklist

Use this checklist for routine taints and tolerations operations in production.

Before Making Changes

• [ ] Record current Kubernetes version: kubectl version --short
• [ ] Record current node taints: kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
• [ ] Identify affected workloads and their current tolerations
• [ ] Take a backup of relevant manifests or use Git version control
• [ ] Prepare rollback commands
• [ ] Assess blast radius: how many pods/nodes will be affected?

During Change

• [ ] Apply change to one node or one workload first
• [ ] Use exact commands and record output
• [ ] Monitor pods and nodes immediately: kubectl get pods -w and kubectl get nodes -w
• [ ] Check for unexpected evictions or scheduling failures

After Change

• [ ] Verify node taints: kubectl describe node <node-name> | grep Taints
• [ ] Verify pod scheduling: kubectl get pods -o wide
• [ ] Check deployment rollout status: kubectl rollout status deployment/<name>
• [ ] Review events: kubectl get events --sort-by=.metadata.creationTimestamp
• [ ] Update runbook with the change and outcome
• [ ] Ensure monitoring alerts are not firing due to the change

Example Runbook Entry

Consider a scenario where you need to dedicate a node for database pods. Here is a sample runbook entry:

Objective: Ensure only database pods run on node-2.

Prerequisites: Kubernetes v1.25, access to cluster, database deployment YAML.

Steps:

1. Check current taints on node-2:

   kubectl describe node node-2 | grep Taints

If none, proceed.

1. Add taint to node-2:

   kubectl taint nodes node-2 dedicated=database:NoSchedule

Output: node/node-2 tainted

1. Update database deployment to include toleration:

   kubectl patch deployment database --type='json' -p='[{"op": "add", "path": "/spec/template/spec/tolerations", "value": [{"key": "dedicated", "operator": "Equal", "value": "database", "effect": "NoSchedule"}]}]'

1. Wait for rollout:

   kubectl rollout status deployment/database

Output: deployment "database" successfully rolled out

1. Verify database pods are on node-2:

   kubectl get pods -o wide -l app=database

Ensure all pods show NODE=node-2.

1. Verify no other pods are on node-2 (if desired):

   kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=node-2

If other pods are running, they may need to be evicted or given appropriate tolerations (be careful with system pods).

Rollback:

• Remove taint: kubectl taint nodes node-2 dedicated=database:NoSchedule-
• Remove toleration from deployment if needed.

Verification:

• Confirm database pods are running and no scheduling errors.
• Check node taints again.

Conclusion

Taints and tolerations are powerful tools for controlling pod placement in Kubernetes, but they require careful management in production. By following the operations checklist in this article, you can avoid common pitfalls such as unschedulable pods, unexpected evictions, and node misconfiguration.

Remember to always observe before changing, limit the blast radius, use placeholders instead of secrets, verify results with concrete commands, and document recovery procedures before an incident forces the decision. Regularly review your taints and tolerations as your cluster evolves, and keep your runbooks up to date.

As a next step, choose one low-risk change in a staging environment, apply the checklist, and validate your understanding. Then, bring the same rigor to production.