## Intro Taints and tolerations are a core Kubernetes scheduling mechanism. A taint on a node repels pods that do not tolerate it, while a toleration on a pod allows it to be scheduled onto tainted nodes. Unlike node selectors, taints and tolerations do not guarantee scheduling; they only influence it, working alongside affinity and anti-affinity rules. Operators often need to back up the taint configuration of their nodes and the tolerations defined in workload manifests. When a node is replaced, a cluster is restored from a disaster, or a rollout is rolled back, having a reliable backup and restore process prevents pods from landing on the wrong nodes or becoming unschedulable. This guide provides practical examples for backing up and restoring taints and tolerations, validating the result, and recovering from common failures. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before backing up taints and tolerations, know your Kubernetes version and cluster topology. The syntax and behavior of taints and tolerations are stable across recent versions, but some features, such as the `PreferNoSchedule` effect and the `kubernetes.io/` built-in taints, have evolved. Run `kubectl version` to check both client and server versions. For managed clusters (EKS, GKE, AKS), note the control plane version separately. Determine what you need to back up: - Node taints: attached to node objects, often added manually or by controllers. - Pod tolerations: defined in pod, deployment, statefulset, or daemonset manifests. ### Read-Only Observation First Collect the current state without modifying anything: ```bash kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints --show-labels ``` Expected output example: ``` NAME TAINTS node-1 [map[effect:NoSchedule key:dedicated value:database]] node-2 [map[effect:NoSchedule key:dedicated value:database]] node-3 ``` For a detailed JSON backup of all node taints: ```bash kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, taints: .spec.taints}' > node-taints-backup.json ``` This file serves as a restorable snapshot. ### Example Environment Assume a three-node cluster: - node-1 and node-2: tainted with `dedicated=database:NoSchedule` to reserve them for database workloads. - node-3: untainted, used for general workloads. Deployments include tolerations in their pod specs to allow scheduling on the dedicated database nodes. ## Safe Configuration Path ### Backing Up Taints To back up taints from all nodes in a human-readable format: ```bash kubectl get nodes -o yaml | kubectl neat > all-nodes.yaml ``` (Install `kubectl neat` for cleaned output; otherwise, use `grep -A 5 taints`.) For a targeted backup, use: ```bash kubectl get node node-1 -o yaml | grep -A 5 -B 5 taints > node-1-taints.yaml ``` Store these backups in version control. For automated backups, run a CronJob that periodically exports node taints and stores them in a Git repository or object storage. ### Backing Up Tolerations Tolerations live in workload manifests. Export the live configuration of all deployments: ```bash kubectl get deployments --all-namespaces -o yaml > deployments-backup.yaml ``` Extract only the tolerations for documentation: ```bash kubectl get deployments --all-namespaces -o json | jq '.items[] | {name: .metadata.name, namespace: .metadata.namespace, tolerations: .spec.template.spec.tolerations}' > tolerations-backup.json ``` This gives a compact view of which workloads tolerate which taints. ### Restoring Taints on a New Node After replacing a node, reapply the taints: ```bash kubectl taint nodes node-1-new dedicated=database:NoSchedule kubectl taint nodes node-2-new dedicated=database:NoSchedule ``` The `kubectl taint` command uses the syntax `key=value:effect` for taints with a value, or `key:effect` for existence-based taints. To remove a taint, append a minus sign: `kubectl taint nodes node-1 dedicated:NoSchedule-`. ### Restoring Tolerations If a deployment was accidentally modified or rolled back without tolerations, reapply the saved manifest: ```bash kubectl apply -f deployment-with-tolerations.yaml ``` Example deployment snippet with a toleration: ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: database-app spec: template: spec: tolerations: - key: "dedicated" operator: "Equal" value: "database" effect: "NoSchedule" ``` ## Verification and Diagnostics After restoring taints, verify they are present on the correct nodes: ```bash kubectl describe nodes | grep -i taint -A 2 ``` Expected output for node-1-new: ``` Taints: dedicated=database:NoSchedule ``` Check that pods with tolerations schedule as expected: ```bash kubectl get pods -o wide ``` Pods that tolerate `dedicated=database:NoSchedule` should be able to schedule on node-1-new and node-2-new. Pods without tolerations should be scheduled only on untainted nodes. ### Diagnosing Scheduling Failures If a pod remains pending, inspect events: ```bash kubectl describe pod | grep -A 10 Events ``` Look for messages like: ``` Warning FailedScheduling 0/3 nodes are available: 1 node(s) had taint {dedicated: database}, that the pod didn't tolerate, 2 node(s) had taint {dedicated: database}, that the pod didn't tolerate. ``` This indicates the pod lacks a required toleration. Verify a pod's tolerations: ```bash kubectl get pod -o jsonpath='{.spec.tolerations}' ``` Use `kubectl rollout status deployment/` to confirm a deployment has successfully rolled out after restoring tolerations. ## Failure Modes and Recovery Common failure modes when working with taints and tolerations: 1. **Missing toleration on pod**: A pod is scheduled on a node with a taint it does not tolerate. This is prevented before scheduling; the pod remains pending. - *Recovery*: Add the required toleration to the pod spec, or remove the taint from the node if it is not needed. 2. **Accidentally removing a taint**: If a taint is removed from a node, pods that should not run on that node may be scheduled there. - *Recovery*: Reapply the taint using `kubectl taint nodes =:`. 3. **Incorrect taint effect**: For example, using `NoExecute` instead of `NoSchedule` can evict existing pods that do not tolerate the taint. - *Recovery*: Update the taint to the correct effect, or add appropriate tolerations to the pods. 4. **Restoring from an old backup**: If the backup is outdated, newly added taints or tolerations may be lost. - *Recovery*: Keep backups up to date with automation, and diff current state against backup before restoring. 5. **Tolerations not applied due to YAML indentation errors**: YAML sensitivity can cause tolerations to be ignored. - *Recovery*: Use `kubectl apply --dry-run=client -f manifest.yaml` to validate syntax, and check the live pod spec with `kubectl get pod -o yaml`. ## Operations Checklist Use this checklist for every change involving taints and tolerations: | Step | Task | Owner | Frequency | |------|------|-------|-----------| | 1 | Export current node taints to a backup file | Priya Shah, Platform Engineer | Before any node change | | 2 | Export current pod tolerations from running workloads | Priya Shah, Platform Engineer | Before any workload update | | 3 | Store backups in Git and object storage | Marcus Chen, DevOps Engineer | Monthly review | | 4 | Review taint drift: compare live taints against desired state | Priya Shah, Platform Engineer | Weekly | | 5 | Validate restored taints with `kubectl describe nodes` | Jordan Lee, SRE | After each node replacement | | 6 | Verify pod scheduling after restore using `kubectl get pods -o wide` | Jordan Lee, SRE | After each incident | | 7 | Update backup procedures based on lessons learned | Team lead, Priya Shah | Quarterly | Each step should have a single accountable owner as indicated, not a group. The frequency column states how often the task or its outcome is revisited. ## Common Pitfalls 1. **Assuming tolerations are the same as node selectors**: Tolerations allow a pod to be scheduled on a tainted node, but they do not force scheduling there. If you want pods to prefer certain nodes, use node affinity in addition to tolerations. 2. **Forgetting to back up tolerations from statefulsets and daemonsets**: Many operators back up deployments but miss other controllers. Always include all workload types: `kubectl get statefulsets,daemonsets --all-namespaces -o yaml`. 3. **Using `kubectl taint` without `--overwrite`**: If a taint with the same key already exists, the command fails. Use `kubectl taint nodes =: --overwrite` to update it. 4. **Not handling built-in taints**: Nodes can have built-in taints like `node.kubernetes.io/not-ready` or `node.kubernetes.io/unreachable` with `NoExecute` effect. These are managed by the control plane and should not be backed up or restored manually. 5. **Restoring taints before the node is ready**: New nodes may still be provisioning. Apply taints only after the node is `Ready` to avoid interfering with node initialization. 6. **Relying solely on manual backups**: Human error is common. Automate backups with a CronJob and verify backup integrity regularly. ## Conclusion Backing up and restoring Kubernetes taints and tolerations is critical for maintaining a stable scheduling environment. This guide has shown how to inventory current state, export backups, restore configurations, verify results, and recover from failures. The key is to treat taints and tolerations as configuration that must be versioned and tested, just like application code. Start by implementing a simple backup process: export node taints and workload tolerations to a Git repository. Then, practice a restore in a non-production cluster to ensure the process works. Regularly review and update your backup and restore procedures as your cluster evolves. Remember: taints and tolerations are not a substitute for resource quotas, node selectors, or affinity rules. They are a scheduling mechanism that requires careful management. With proper backups and a recovery plan, you can avoid costly downtime and misconfigurations.