Kubernetes Taints and Tolerations Backup and Restore: A Practical Guide

Intro

Taints and tolerations are a core Kubernetes scheduling mechanism. A taint on a node repels pods that do not tolerate it, while a toleration on a pod allows it to be scheduled onto tainted nodes. Unlike node selectors, taints and tolerations do not guarantee scheduling; they only influence it, working alongside affinity and anti-affinity rules.

Operators often need to back up the taint configuration of their nodes and the tolerations defined in workload manifests. When a node is replaced, a cluster is restored from a disaster, or a rollout is rolled back, having a reliable backup and restore process prevents pods from landing on the wrong nodes or becoming unschedulable. This guide provides practical examples for backing up and restoring taints and tolerations, validating the result, and recovering from common failures.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before backing up taints and tolerations, know your Kubernetes version and cluster topology. The syntax and behavior of taints and tolerations are stable across recent versions, but some features, such as the PreferNoSchedule effect and the kubernetes.io/ built-in taints, have evolved. Run kubectl version to check both client and server versions. For managed clusters (EKS, GKE, AKS), note the control plane version separately.

Determine what you need to back up:

• Node taints: attached to node objects, often added manually or by controllers.
• Pod tolerations: defined in pod, deployment, statefulset, or daemonset manifests.

Read-Only Observation First

Collect the current state without modifying anything:

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints --show-labels

Expected output example:

NAME     TAINTS
node-1   [map[effect:NoSchedule key:dedicated value:database]]
node-2   [map[effect:NoSchedule key:dedicated value:database]]
node-3   <none>

For a detailed JSON backup of all node taints:

kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, taints: .spec.taints}' > node-taints-backup.json

This file serves as a restorable snapshot.

Example Environment

Assume a three-node cluster:

• node-1 and node-2: tainted with dedicated=database:NoSchedule to reserve them for database workloads.
• node-3: untainted, used for general workloads.

Deployments include tolerations in their pod specs to allow scheduling on the dedicated database nodes.

Safe Configuration Path

Backing Up Taints

To back up taints from all nodes in a human-readable format:

kubectl get nodes -o yaml | kubectl neat > all-nodes.yaml

(Install kubectl neat for cleaned output; otherwise, use grep -A 5 taints.)

For a targeted backup, use:

kubectl get node node-1 -o yaml | grep -A 5 -B 5 taints > node-1-taints.yaml

Store these backups in version control. For automated backups, run a CronJob that periodically exports node taints and stores them in a Git repository or object storage.

Backing Up Tolerations

Tolerations live in workload manifests. Export the live configuration of all deployments:

kubectl get deployments --all-namespaces -o yaml > deployments-backup.yaml

Extract only the tolerations for documentation:

kubectl get deployments --all-namespaces -o json | jq '.items[] | {name: .metadata.name, namespace: .metadata.namespace, tolerations: .spec.template.spec.tolerations}' > tolerations-backup.json

This gives a compact view of which workloads tolerate which taints.

Restoring Taints on a New Node

After replacing a node, reapply the taints:

kubectl taint nodes node-1-new dedicated=database:NoSchedule
kubectl taint nodes node-2-new dedicated=database:NoSchedule

The kubectl taint command uses the syntax key=value:effect for taints with a value, or key:effect for existence-based taints. To remove a taint, append a minus sign: kubectl taint nodes node-1 dedicated:NoSchedule-.

Restoring Tolerations

If a deployment was accidentally modified or rolled back without tolerations, reapply the saved manifest:

kubectl apply -f deployment-with-tolerations.yaml

Example deployment snippet with a toleration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: database-app
spec:
  template:
    spec:
      tolerations:
      - key: "dedicated"
        operator: "Equal"
        value: "database"
        effect: "NoSchedule"

Verification and Diagnostics

After restoring taints, verify they are present on the correct nodes:

kubectl describe nodes | grep -i taint -A 2

Expected output for node-1-new:

Taints:             dedicated=database:NoSchedule

Check that pods with tolerations schedule as expected:

kubectl get pods -o wide

Pods that tolerate dedicated=database:NoSchedule should be able to schedule on node-1-new and node-2-new. Pods without tolerations should be scheduled only on untainted nodes.

Diagnosing Scheduling Failures

If a pod remains pending, inspect events:

kubectl describe pod <pod-name> | grep -A 10 Events

Look for messages like:

Warning  FailedScheduling  0/3 nodes are available: 1 node(s) had taint {dedicated: database}, that the pod didn't tolerate, 2 node(s) had taint {dedicated: database}, that the pod didn't tolerate.

This indicates the pod lacks a required toleration.

Verify a pod's tolerations:

kubectl get pod <pod-name> -o jsonpath='{.spec.tolerations}'

Use kubectl rollout status deployment/<name> to confirm a deployment has successfully rolled out after restoring tolerations.

Failure Modes and Recovery

Common failure modes when working with taints and tolerations:

1. Missing toleration on pod: A pod is scheduled on a node with a taint it does not tolerate. This is prevented before scheduling; the pod remains pending.

• Recovery: Add the required toleration to the pod spec, or remove the taint from the node if it is not needed.

1. Accidentally removing a taint: If a taint is removed from a node, pods that should not run on that node may be scheduled there.

• Recovery: Reapply the taint using kubectl taint nodes <node> <key>=<value>:<effect>.

1. Incorrect taint effect: For example, using NoExecute instead of NoSchedule can evict existing pods that do not tolerate the taint.

• Recovery: Update the taint to the correct effect, or add appropriate tolerations to the pods.

1. Restoring from an old backup: If the backup is outdated, newly added taints or tolerations may be lost.

• Recovery: Keep backups up to date with automation, and diff current state against backup before restoring.

1. Tolerations not applied due to YAML indentation errors: YAML sensitivity can cause tolerations to be ignored.

• Recovery: Use kubectl apply --dry-run=client -f manifest.yaml to validate syntax, and check the live pod spec with kubectl get pod <pod> -o yaml.

Operations Checklist

Use this checklist for every change involving taints and tolerations:

Each step should have a single accountable owner as indicated, not a group. The frequency column states how often the task or its outcome is revisited.

Common Pitfalls

1. Assuming tolerations are the same as node selectors: Tolerations allow a pod to be scheduled on a tainted node, but they do not force scheduling there. If you want pods to prefer certain nodes, use node affinity in addition to tolerations.

1. Forgetting to back up tolerations from statefulsets and daemonsets: Many operators back up deployments but miss other controllers. Always include all workload types: kubectl get statefulsets,daemonsets --all-namespaces -o yaml.

1. Using kubectl taint without --overwrite: If a taint with the same key already exists, the command fails. Use kubectl taint nodes <node> <key>=<value>:<effect> --overwrite to update it.

1. Not handling built-in taints: Nodes can have built-in taints like node.kubernetes.io/not-ready or node.kubernetes.io/unreachable with NoExecute effect. These are managed by the control plane and should not be backed up or restored manually.

1. Restoring taints before the node is ready: New nodes may still be provisioning. Apply taints only after the node is Ready to avoid interfering with node initialization.

1. Relying solely on manual backups: Human error is common. Automate backups with a CronJob and verify backup integrity regularly.

Conclusion

Backing up and restoring Kubernetes taints and tolerations is critical for maintaining a stable scheduling environment. This guide has shown how to inventory current state, export backups, restore configurations, verify results, and recover from failures. The key is to treat taints and tolerations as configuration that must be versioned and tested, just like application code.

Start by implementing a simple backup process: export node taints and workload tolerations to a Git repository. Then, practice a restore in a non-production cluster to ensure the process works. Regularly review and update your backup and restore procedures as your cluster evolves.

Remember: taints and tolerations are not a substitute for resource quotas, node selectors, or affinity rules. They are a scheduling mechanism that requires careful management. With proper backups and a recovery plan, you can avoid costly downtime and misconfigurations.