## Intro Kubernetes has long discouraged swap on nodes, but newer versions support it with careful configuration for specific workloads. Managing swap memory effectively requires a clear backup and restore strategy to protect node configuration, workload state, and cluster stability. This guide provides practical, step-by-step methods for backing up, restoring, and validating Kubernetes swap memory configurations, enabling operators to recover quickly from misconfigurations or failures. We cover the full operational lifecycle: inventorying your environment, applying safe configuration changes, verifying outcomes, handling failure modes, and following an operations checklist. Every section includes concrete commands, expected outputs, and recovery decisions tailored for developers, DevOps engineers, and technical startup teams who need reliable, repeatable procedures. ## Version and Environment Inventory Before touching swap settings, establish a precise baseline. Identify your Kubernetes version, node operating system, kubelet configuration, and current swap status. This inventory prevents accidental changes to production nodes and ensures every subsequent step is version-compatible. Read-only observation first Run the following commands to capture the current state without modifying anything: # Kubernetes version kubectl version --short # Node status and details kubectl get nodes -o wide # Kubelet version on a specific node (replace NODE_NAME) kubectl get node NODE_NAME -o jsonpath='{.status.nodeInfo.kubeletVersion}{"\n"}' # Check swap usage on the node (requires SSH or node access) ssh NODE_NAME 'free -h && swapon --show' Expected output includes the Kubernetes server version, node list with roles and status, kubelet version, and swap summary. If swapon --show returns nothing, swap is disabled. If it returns a path like /swapfile , swap is active. Document environment details Create an inventory file with concrete values. Example: - Cluster version: v1.28.5 - Node OS: Ubuntu 22.04 LTS - Kubelet version: v1.28.5 - Swap status: enabled, 2GB swapfile at /swapfile - Cgroup driver: systemd - Container runtime: containerd 1.7.11 Check kubelet swap configuration Kubelet's swap behavior is controlled by the failSwapOn or memorySwap fields in its configuration. Retrieve the current setting: ssh NODE_NAME 'cat /var/lib/kubelet/config.yaml | grep -A5 swap' Expected output may show: memorySwap: swapBehavior: LimitedSwap failSwapOn: false If these fields are absent, kubelet uses defaults (swap disabled). Record this in your inventory. Verify node pressure and resource requests Understand how swap interacts with resource management. Check current resource requests and limits on a node: kubectl describe node NODE_NAME | grep -A10 "Allocated resources" This shows CPU, memory, and ephemeral storage allocations. Note if memory requests approach capacity, as swap can provide overflow under LimitedSwap behavior. Limit the blast radius Make changes one node at a time, ideally in a staging cluster first. Always cordon and drain a node before modifying its swap configuration: kubectl cordon NODE_NAME kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data After changes, uncordon: kubectl uncordon NODE_NAME Quick check 1 of 2 What is the default swap behavior in Kubernetes? - LimitedSwap - NoSwap - UnlimitedSwap - BestEffortSwap

According to the reference, the default swap behavior is NoSwap, meaning workloads do not use swap.

## Safe Configuration Path Configuring swap safely requires understanding kubelet's swap behavior options and applying changes through version-controlled files, not ad hoc edits. Understand swap behavior modes Kubelet offers three swap behaviors (starting from Kubernetes 1.28 with NodeSwap feature gate enabled and cgroup v2): - NoSwap: default, swap is disabled. - LimitedSwap: swap is allowed but kubelet limits its use based on the node's memory pressure and pod QoS classes. Only pods with Burstable or BestEffort QoS can use swap; Guaranteed pods cannot. - UnlimitedSwap: swap is allowed without kubelet restrictions (not recommended for production). Choose LimitedSwap for most workloads to balance performance and safety. Enable swap on the node Assuming swap file exists (if not, create one): # Create a 2GB swap file (if none) sudo fallocate -l 2G /swapfile sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile # Make swap permanent by adding to /etc/fstab echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab Update kubelet configuration Edit /var/lib/kubelet/config.yaml on the target node. Add or modify the memorySwap section: apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration memorySwap: swapBehavior: LimitedSwap failSwapOn: false Set failSwapOn: false to prevent kubelet from failing if swap is on. Restart kubelet sudo systemctl restart kubelet sudo systemctl status kubelet Verify kubelet is active and no errors appear. Verify swap is recognized by kubelet Check kubelet logs for swap-related messages: sudo journalctl -u kubelet -n 50 --no-pager | grep -i swap Expected output may include: I0214 10:00:00.123456 12345 kubelet.go:200] "Swap is enabled" swapBehavior="LimitedSwap" Test with a sample pod Deploy a Burstable pod that requests less than its limit, allowing it to use swap if needed: apiVersion: v1 kind: Pod metadata: name: swap-test spec: containers: - name: stress image: polinux/stress resources: requests: memory: "100Mi" limits: memory: "200Mi" command: ["stress"] args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"] Apply it: kubectl apply -f swap-test.yaml kubectl get pod swap-test Check the pod is Running. Then, inspect its memory usage on the node: ssh NODE_NAME 'watch -n1 free -h' You may see swap usage increase if the pod exceeds its memory request. This validates LimitedSwap working. Rollback plan If configuration causes issues, revert by setting swapBehavior back to NoSwap and failSwapOn: true , then restart kubelet. Keep a backup of the original config file: sudo cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak ## Verification and Diagnostics After configuration changes, thorough verification prevents silent failures. Use a systematic approach to confirm swap is working as intended and pods behave correctly. Check kubelet swap metrics Kubelet exposes swap-related metrics on its metrics endpoint. Access via curl on the node: ssh NODE_NAME 'curl -s http://localhost:10255/metrics | grep swap' Or if using the secure port with authentication, use kubectl proxy and query the kubelet API. Example output: # HELP kubelet_swap_used_bytes Current swap usage in bytes. # TYPE kubelet_swap_used_bytes gauge kubelet_swap_used_bytes 1073741824 This indicates about 1GB swap in use. Inspect pod resource usage Use kubectl top to see actual memory usage: kubectl top pod swap-test If kubectl top is not available, use kubectl describe to see resource requests and limits. Review node events Look for swap-related warnings or errors: kubectl describe node NODE_NAME | grep -i swap No output is good. If warnings appear, investigate further. Diagnose pod failures due to swap If a pod fails to start, check its events and logs: kubectl describe pod swap-test kubectl logs swap-test --previous Common failure signals: - Failed to create pod sandbox may indicate cgroup issues. - OOMKilled suggests memory constraints, potentially related to swap misconfiguration. - Pod stuck in ContainerCreating with swap-related messages in kubelet logs. Validate rollback readiness Simulate a rollback in a test environment. Apply the previous configuration and confirm pods restart cleanly. ## Failure Modes and Recovery Even with careful planning, failures occur. Know how to detect and recover from common swap-related issues. Failure: kubelet fails to start with swap enabled Symptom: systemctl status kubelet shows failure, logs contain failed to run Kubelet: running with swap on is not supported, please disable swap! Cause: failSwapOn is true in kubelet config or not set (default true) while swap is active. Recovery: Set failSwapOn: false in kubelet config, or disable swap temporarily ( sudo swapoff -a ), then restart kubelet. If using failSwapOn: false , ensure memorySwap.swapBehavior is set appropriately. Failure: Node becomes NotReady after enabling swap Symptom: kubectl get nodes shows NotReady for the node. Cause: Kubelet may have restarted and lost connection, or system resources misconfigured. Recovery: - Check kubelet status and logs. - Ensure kubelet has restarted properly. - Verify network connectivity and node conditions: kubectl describe node NODE_NAME - If swap caused kernel issues, disable swap and restart. Failure: Unexpected pod evictions Symptom: Pods are evicted despite available memory, with node condition MemoryPressure . Cause: Swap may be incorrectly accounted for. With LimitedSwap , kubelet may incorrectly calculate available memory, leading to evictions. Recovery: Adjust kubelet eviction thresholds. In kubelet config, set: evictionHard: memory.available: "200Mi" nodefs.available: "10%" Restart kubelet and monitor. Failure: Pods with Guaranteed QoS attempt to use swap Symptom: Pods with Guaranteed QoS (requests equal limits) may be terminated if they try to use swap. Cause: In LimitedSwap mode, Guaranteed pods are not allowed to use swap. If the workload requires swap, change its QoS class to Burstable. Recovery: Modify pod resource requests/limits to make it Burstable, or consider changing swap behavior if appropriate. Disaster Recovery: Full node restore If a node becomes unrecoverable, rebuild it from scratch using a configuration management tool or manual steps: - Provision a new node with same OS and specs. - Join it to the cluster (using kubeadm join or cloud provider). - Reapply swap configuration: create swap file, update kubelet config, restart kubelet. - Verify node readiness and pod scheduling. Backup strategy for swap configuration Treat node configuration as code. Store kubelet config, systemd unit files, and swap setup scripts in version control. Use tools like Ansible, Terraform, or cloud-init to rebuild nodes consistently. Periodically test restore procedures in a staging environment. Example backup script: #!/bin/bash # Backup kubelet config and swap settings mkdir -p ~/k8s-backup/$(date +%Y%m%d) cp /var/lib/kubelet/config.yaml ~/k8s-backup/$(date +%Y%m%d)/kubelet-config.yaml cp /etc/fstab ~/k8s-backup/$(date +%Y%m%d)/fstab swapon --show > ~/k8s-backup/$(date +%Y%m%d)/swap-status.txt ## Operations Checklist Use this checklist to ensure consistency and safety when managing swap on Kubernetes nodes. Before any change - Confirm Kubernetes version supports swap ( >=1.28 and cgroup v2). - Verify node operating system supports swap with cgroup v2. - Take a backup of kubelet config: cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak-$(date +%F) . - Record current swap status: swapon --show and free -h . - Cordone and drain node if making disruptive changes. During configuration - Apply one change at a time. - Use the minimal memorySwap configuration required. - Restart kubelet and verify status. - Check kubelet logs for errors. - Deploy a test pod and observe swap behavior. Verification - Run kubectl get nodes to confirm node is Ready. - Check kubectl describe node NODE_NAME for MemoryPressure conditions. - Inspect pod scheduling and resource usage. - Validate that rollback plan is executable. Documentation and rollback - Update inventory with new configuration. - Store backup copies in a secure location. - Document any troubleshooting steps taken. - Schedule a review of swap usage metrics. ## Conclusion Managing Kubernetes swap memory requires a well-defined backup, restore, and disaster recovery strategy. By inventorying your environment, applying safe configuration changes, verifying outcomes, and preparing for failures, you can leverage swap benefits without compromising cluster stability. The practical commands and examples in this guide provide a foundation for reliable operations. As a next step, implement a version-controlled configuration management system for node settings, including swap. Test your backup and restore procedures in a staging environment, and document your recovery runbook. With these practices, your Kubernetes clusters will be resilient and maintainable.