## Intro Taints and tolerations are a core Kubernetes scheduling mechanism that lets you control which pods can be placed on which nodes. Taints are applied to nodes to repel pods that do not tolerate them, while tolerations are applied to pods to allow them to schedule onto tainted nodes. In production, a misconfigured taint or toleration can lead to pods stuck in Pending state, unschedulable workloads, or even accidentally scheduling critical pods onto unsuitable nodes. This article provides a production-focused operations checklist for Kubernetes taints and tolerations. It covers version and environment inventory, safe configuration paths, verification and diagnostics, failure modes and recovery, and an end-to-end operations checklist. Each section includes concrete `kubectl` commands, expected outputs, and practical examples. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. This guide is intended for developers, DevOps consultants, and technical startup teams who need to manage taints and tolerations in production clusters. It assumes basic familiarity with Kubernetes concepts like pods, nodes, and Deployments. ## Version and Environment Inventory Before making any changes, you must know your Kubernetes version, cluster topology, and the exact state of your taints and tolerations. Taints and tolerations have evolved across Kubernetes versions; for example, the `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable` taints behave differently depending on the version and the `TaintBasedEvictions` feature gate. ### 1. Check Kubernetes Version Run the following command to check the server version: ```bash kubectl version --short ``` Expected output (example): ``` Client Version: v1.25.3 Server Version: v1.25.3 ``` Note: In newer versions, the `--short` flag is deprecated; use `kubectl version` without flags and read the output. If your client and server versions differ significantly, be aware that taint-related API fields might behave differently. For production, document the exact version in your runbook. ### 2. List All Nodes and Their Taints To see all nodes and their taints: ```bash kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints ``` Expected output (example): ``` NAME TAINTS node-1 node-2 [{"effect":"NoSchedule","key":"dedicated","value":"database"}] node-3 [{"effect":"NoExecute","key":"gpu","value":"true"}] ``` If a node shows `` for taints, it has no taints and any pod without a matching toleration can schedule there. Look for taints with `NoSchedule` (pods without toleration will not be scheduled), `PreferNoSchedule` (soft preference), and `NoExecute` (evicts existing pods without toleration and prevents scheduling). ### 3. Describe a Specific Node For detailed taint information and events: ```bash kubectl describe node node-2 ``` Look for the `Taints:` line in the output. Example: ``` Taints: dedicated=database:NoSchedule ``` This tells you the taint key (`dedicated`), value (`database`), and effect (`NoSchedule`). ### 4. Check Existing Pods and Their Tolerations To see which pods currently have tolerations: ```bash kubectl get pods -A -o json | jq '.items[] | {name: .metadata.name, namespace: .metadata.namespace, tolerations: .spec.tolerations}' ``` Expected output (example): ```json { "name": "my-db-pod", "namespace": "default", "tolerations": [ { "key": "dedicated", "operator": "Equal", "value": "database", "effect": "NoSchedule" } ] } ``` If a pod does not list any tolerations, it will not schedule onto nodes with taints unless the taint has `PreferNoSchedule` effect (in which case the scheduler may still place it if no other nodes are available). ### 5. Environment Inventory Checklist For production, record the following in your inventory: - Kubernetes version (server and client) - Node names and their taints - Namespaces and workloads that rely on taints/tolerations - Critical pods and their tolerations - Any cluster autoscaler settings that interact with taints Document this in a runbook before making changes. ## Safe Configuration Path When modifying taints or tolerations, always follow a safe configuration path: observe, plan, change one item, verify, and have a rollback plan. ### 1. Add a Taint to a Node To add a taint to a node: ```bash kubectl taint nodes node-1 dedicated=frontend:NoSchedule ``` Expected output: ``` node/node-1 tainted ``` This taint prevents any pod that lacks a matching toleration from being scheduled on `node-1`. After adding, verify with: ```bash kubectl describe node node-1 | grep Taints ``` Expected output: ``` Taints: dedicated=frontend:NoSchedule ``` ### 2. Add a Toleration to a Pod Specification Tolerations are added to the pod spec. For a Deployment, add them under `spec.template.spec.tolerations`. Example YAML snippet: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: frontend spec: replicas: 2 selector: matchLabels: app: frontend template: metadata: labels: app: frontend spec: tolerations: - key: "dedicated" operator: "Equal" value: "frontend" effect: "NoSchedule" containers: - name: nginx image: nginx:1.21 ``` Apply it with: ```bash kubectl apply -f frontend-deployment.yaml ``` Then check rollout status: ```bash kubectl rollout status deployment/frontend ``` Expected output: ``` deployment "frontend" successfully rolled out ``` ### 3. Remove a Taint from a Node To remove a taint, use the same key and effect with a minus sign at the end of the effect: ```bash kubectl taint nodes node-1 dedicated=frontend:NoSchedule- ``` Expected output: ``` node/node-1 untainted ``` Verify with `kubectl describe node node-1` and ensure the taint is gone. ### 4. Update Tolerations on an Existing Deployment If you need to add tolerations to an existing deployment without recreating it, use `kubectl patch`: ```bash kubectl patch deployment frontend --type='json' -p='[{"op": "add", "path": "/spec/template/spec/tolerations", "value": [{"key": "dedicated", "operator": "Equal", "value": "frontend", "effect": "NoSchedule"}]}]' ``` Then watch the rollout: ```bash kubectl rollout status deployment/frontend ``` ### 5. Safe Change Practices - Always apply changes to a non-production environment first if possible. - Use `--dry-run=client` to preview changes without applying them. For taints, you can simulate by using `kubectl taint nodes node-1 dedicated=frontend:NoSchedule --dry-run=client` (though taints are server-side mutations and dry-run may not fully apply; verify with `kubectl auth can-i` and test in a staging cluster). - Document the exact command and its expected output before running. - Have a rollback command ready (e.g., the untaint command). - Limit the blast radius: change one node or one pod at a time, then observe. ## Verification and Diagnostics After making changes, you must verify that scheduling behaves as expected and diagnose any issues. ### 1. Check Pod Scheduling Status List pods with their node assignments: ```bash kubectl get pods -o wide ``` Expected output (example): ``` NAME READY STATUS RESTARTS AGE IP NODE frontend-6f8c9d7b-abcde 1/1 Running 0 10m 10.244.1.5 node-1 frontend-6f8c9d7b-fghij 1/1 Running 0 10m 10.244.2.3 node-2 ``` If pods are stuck in `Pending`, they may be unschedulable due to taints. ### 2. Describe a Pending Pod Describe the pod to see scheduling events: ```bash kubectl describe pod frontend-6f8c9d7b-abcde ``` Look for events like: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m default-scheduler 0/3 nodes are available: 1 node(s) had taint {dedicated: frontend}, that the pod didn't tolerate, 2 node(s) had taint {gpu: true}, that the pod didn't tolerate. ``` This event indicates that the pod does not have the required tolerations for the taints on available nodes. ### 3. Verify Node Taints After Change Use `kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints` again to confirm the taint is present or removed. ### 4. Check Logs for Startup Issues If a pod is running but misbehaving, check its logs: ```bash kubectl logs --previous ``` For example, if a pod crashed due to missing resources on a tainted node, logs might show errors. ### 5. Use Kubernetes Events To see cluster-wide events related to scheduling: ```bash kubectl get events --field-selector reason=FailedScheduling ``` This filters events for scheduling failures. ### 6. Simulate Scheduling with `kubectl debug` or Dry-Run You can use the `kubectl debug` command to create a temporary pod with tolerations to test scheduling on a specific node. For example: ```bash kubectl debug node/node-1 -it --image=busybox -- sh ``` This creates a debug pod on node-1, which may require tolerations if node-1 is tainted. Alternatively, use a dry-run manifest to see if the pod would be accepted: ```bash kubectl apply -f test-pod.yaml --dry-run=client ``` This only validates the YAML, not scheduling. For actual scheduling simulation, use a tool like `kube-scheduler-simulator` or test in a staging cluster. ## Failure Modes and Recovery Misconfigured taints and tolerations can cause several failure modes. Here are common scenarios and recovery steps. ### Failure Mode 1: Pod Stuck in Pending State Due to Missing Toleration **Symptom:** `kubectl get pods` shows a pod with STATUS `Pending` for a long time. Describing the pod shows `FailedScheduling` events with messages about taints. **Recovery:** 1. Identify the taints on nodes: ```bash kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints ``` 2. Determine which taint is blocking. Add the appropriate toleration to the pod spec. For a quick fix, patch the deployment: ```bash kubectl patch deployment --type='json' -p='[{"op": "add", "path": "/spec/template/spec/tolerations", "value": [{"key": "", "operator": "Equal", "value": "", "effect": ""}]}]' ``` 3. Watch the new pods schedule: ```bash kubectl get pods -w ``` ### Failure Mode 2: Pod Evicted by NoExecute Taint **Symptom:** A running pod suddenly terminates and is recreated repeatedly, or disappears from the node. Describing the pod may show an eviction event. **Recovery:** 1. Check node taints: `kubectl describe node `. 2. If a `NoExecute` taint was recently added, either add a toleration with `tolerationSeconds` or `operator: Exists` to the pod, or remove the taint from the node if it was unintended. 3. Remove the taint: ```bash kubectl taint nodes =:NoExecute- ``` 4. Verify the pod reschedules: ```bash kubectl get pods -o wide ``` ### Failure Mode 3: All Nodes Tainted and No Pods Can Schedule **Symptom:** Every new pod remains Pending, and `kubectl get nodes` shows all nodes have taints without corresponding tolerations. **Recovery:** 1. List taints: ```bash kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints ``` 2. Decide whether to remove taints from some nodes or add tolerations to critical workloads. 3. For critical system pods (e.g., kube-system), ensure they have appropriate tolerations; many system pods have built-in tolerations for common taints like `node.kubernetes.io/not-ready`. 4. Remove unnecessary taints or add a broad toleration: ```yaml tolerations: - operator: "Exists" ``` Use with caution, as this allows the pod to schedule on any tainted node. ### Failure Mode 4: Taint Removed but Pods Still Unschedulable **Symptom:** After removing a taint, pods still cannot schedule on a node. **Possible Causes:** - The taint removal command had a typo or wrong effect. - There are other taints on the node. - The node is cordoned (`kubectl cordon`). - Resource constraints (CPU/memory) prevent scheduling. **Recovery:** 1. Verify node taints and cordon status: ```bash kubectl describe node | grep -E 'Taints|Unschedulable' ``` 2. If `Unschedulable: true`, uncordon the node: ```bash kubectl uncordon ``` 3. Check for other taints and remove them if appropriate. 4. Check resource requests and limits on pods and node allocatable capacity. ### Rollback Strategies Always have a rollback plan: - For taints, keep the exact command to remove the taint (with trailing minus). - For pod tolerations, keep the previous YAML or patch command to revert. - Use version control for all manifests. ## Operations Checklist Use this checklist for routine taints and tolerations operations in production. ### Before Making Changes - [ ] Record current Kubernetes version: `kubectl version --short` - [ ] Record current node taints: `kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints` - [ ] Identify affected workloads and their current tolerations - [ ] Take a backup of relevant manifests or use Git version control - [ ] Prepare rollback commands - [ ] Assess blast radius: how many pods/nodes will be affected? ### During Change - [ ] Apply change to one node or one workload first - [ ] Use exact commands and record output - [ ] Monitor pods and nodes immediately: `kubectl get pods -w` and `kubectl get nodes -w` - [ ] Check for unexpected evictions or scheduling failures ### After Change - [ ] Verify node taints: `kubectl describe node | grep Taints` - [ ] Verify pod scheduling: `kubectl get pods -o wide` - [ ] Check deployment rollout status: `kubectl rollout status deployment/` - [ ] Review events: `kubectl get events --sort-by=.metadata.creationTimestamp` - [ ] Update runbook with the change and outcome - [ ] Ensure monitoring alerts are not firing due to the change ### Example Runbook Entry Consider a scenario where you need to dedicate a node for database pods. Here is a sample runbook entry: **Objective:** Ensure only database pods run on node-2. **Prerequisites:** Kubernetes v1.25, access to cluster, database deployment YAML. **Steps:** 1. Check current taints on node-2: ```bash kubectl describe node node-2 | grep Taints ``` If none, proceed. 2. Add taint to node-2: ```bash kubectl taint nodes node-2 dedicated=database:NoSchedule ``` Output: `node/node-2 tainted` 3. Update database deployment to include toleration: ```bash kubectl patch deployment database --type='json' -p='[{"op": "add", "path": "/spec/template/spec/tolerations", "value": [{"key": "dedicated", "operator": "Equal", "value": "database", "effect": "NoSchedule"}]}]' ``` 4. Wait for rollout: ```bash kubectl rollout status deployment/database ``` Output: `deployment "database" successfully rolled out` 5. Verify database pods are on node-2: ```bash kubectl get pods -o wide -l app=database ``` Ensure all pods show NODE=node-2. 6. Verify no other pods are on node-2 (if desired): ```bash kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=node-2 ``` If other pods are running, they may need to be evicted or given appropriate tolerations (be careful with system pods). **Rollback:** - Remove taint: `kubectl taint nodes node-2 dedicated=database:NoSchedule-` - Remove toleration from deployment if needed. **Verification:** - Confirm database pods are running and no scheduling errors. - Check node taints again. ## Conclusion Taints and tolerations are powerful tools for controlling pod placement in Kubernetes, but they require careful management in production. By following the operations checklist in this article, you can avoid common pitfalls such as unschedulable pods, unexpected evictions, and node misconfiguration. Remember to always observe before changing, limit the blast radius, use placeholders instead of secrets, verify results with concrete commands, and document recovery procedures before an incident forces the decision. Regularly review your taints and tolerations as your cluster evolves, and keep your runbooks up to date. As a next step, choose one low-risk change in a staging environment, apply the checklist, and validate your understanding. Then, bring the same rigor to production.