Kubernetes Swap Memory Management: Backup, Restore, and Disaster Recovery Guide

Intro

Kubernetes has long discouraged swap on nodes, but newer versions support it with careful configuration for specific workloads. Managing swap memory effectively requires a clear backup and restore strategy to protect node configuration, workload state, and cluster stability. This guide provides practical, step-by-step methods for backing up, restoring, and validating Kubernetes swap memory configurations, enabling operators to recover quickly from misconfigurations or failures.

We cover the full operational lifecycle: inventorying your environment, applying safe configuration changes, verifying outcomes, handling failure modes, and following an operations checklist. Every section includes concrete commands, expected outputs, and recovery decisions tailored for developers, DevOps engineers, and technical startup teams who need reliable, repeatable procedures.

Version and Environment Inventory

Before touching swap settings, establish a precise baseline. Identify your Kubernetes version, node operating system, kubelet configuration, and current swap status. This inventory prevents accidental changes to production nodes and ensures every subsequent step is version-compatible.

Read-only observation first

Run the following commands to capture the current state without modifying anything:

# Kubernetes version
kubectl version --short

# Node status and details
kubectl get nodes -o wide

# Kubelet version on a specific node (replace NODE_NAME)
kubectl get node NODE_NAME -o jsonpath='{.status.nodeInfo.kubeletVersion}{"\n"}'

# Check swap usage on the node (requires SSH or node access)
ssh NODE_NAME 'free -h && swapon --show'

Expected output includes the Kubernetes server version, node list with roles and status, kubelet version, and swap summary. If swapon --show returns nothing, swap is disabled. If it returns a path like /swapfile, swap is active.

Document environment details

Create an inventory file with concrete values. Example:

• Cluster version: v1.28.5
• Node OS: Ubuntu 22.04 LTS
• Kubelet version: v1.28.5
• Swap status: enabled, 2GB swapfile at /swapfile
• Cgroup driver: systemd
• Container runtime: containerd 1.7.11

Check kubelet swap configuration

Kubelet's swap behavior is controlled by the failSwapOn or memorySwap fields in its configuration. Retrieve the current setting:

ssh NODE_NAME 'cat /var/lib/kubelet/config.yaml | grep -A5 swap'

Expected output may show:

memorySwap:
  swapBehavior: LimitedSwap
failSwapOn: false

If these fields are absent, kubelet uses defaults (swap disabled). Record this in your inventory.

Verify node pressure and resource requests

Understand how swap interacts with resource management. Check current resource requests and limits on a node:

kubectl describe node NODE_NAME | grep -A10 "Allocated resources"

This shows CPU, memory, and ephemeral storage allocations. Note if memory requests approach capacity, as swap can provide overflow under LimitedSwap behavior.

Limit the blast radius

Make changes one node at a time, ideally in a staging cluster first. Always cordon and drain a node before modifying its swap configuration:

kubectl cordon NODE_NAME
kubectl drain NODE_NAME --ignore-daemonsets --delete-emptydir-data

After changes, uncordon:

kubectl uncordon NODE_NAME

Safe Configuration Path

Configuring swap safely requires understanding kubelet's swap behavior options and applying changes through version-controlled files, not ad hoc edits.

Understand swap behavior modes

Kubelet offers three swap behaviors (starting from Kubernetes 1.28 with NodeSwap feature gate enabled and cgroup v2):

• NoSwap: default, swap is disabled.
• LimitedSwap: swap is allowed but kubelet limits its use based on the node's memory pressure and pod QoS classes. Only pods with Burstable or BestEffort QoS can use swap; Guaranteed pods cannot.
• UnlimitedSwap: swap is allowed without kubelet restrictions (not recommended for production).

Choose LimitedSwap for most workloads to balance performance and safety.

Enable swap on the node

Assuming swap file exists (if not, create one):

# Create a 2GB swap file (if none)
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

# Make swap permanent by adding to /etc/fstab
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Update kubelet configuration

Edit /var/lib/kubelet/config.yaml on the target node. Add or modify the memorySwap section:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memorySwap:
  swapBehavior: LimitedSwap
failSwapOn: false

Set failSwapOn: false to prevent kubelet from failing if swap is on.

Restart kubelet

sudo systemctl restart kubelet
sudo systemctl status kubelet

Verify kubelet is active and no errors appear.

Verify swap is recognized by kubelet

Check kubelet logs for swap-related messages:

sudo journalctl -u kubelet -n 50 --no-pager | grep -i swap

Expected output may include:

I0214 10:00:00.123456   12345 kubelet.go:200] "Swap is enabled" swapBehavior="LimitedSwap"

Test with a sample pod

Deploy a Burstable pod that requests less than its limit, allowing it to use swap if needed:

apiVersion: v1
kind: Pod
metadata:
  name: swap-test
spec:
  containers:
  - name: stress
    image: polinux/stress
    resources:
      requests:
        memory: "100Mi"
      limits:
        memory: "200Mi"
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "150M", "--vm-hang", "1"]

Apply it:

kubectl apply -f swap-test.yaml
kubectl get pod swap-test

Check the pod is Running. Then, inspect its memory usage on the node:

ssh NODE_NAME 'watch -n1 free -h'

You may see swap usage increase if the pod exceeds its memory request. This validates LimitedSwap working.

Rollback plan

If configuration causes issues, revert by setting swapBehavior back to NoSwap and failSwapOn: true, then restart kubelet. Keep a backup of the original config file:

sudo cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak

Verification and Diagnostics

After configuration changes, thorough verification prevents silent failures. Use a systematic approach to confirm swap is working as intended and pods behave correctly.

Check kubelet swap metrics

Kubelet exposes swap-related metrics on its metrics endpoint. Access via curl on the node:

ssh NODE_NAME 'curl -s http://localhost:10255/metrics | grep swap'

Or if using the secure port with authentication, use kubectl proxy and query the kubelet API.

Example output:

# HELP kubelet_swap_used_bytes Current swap usage in bytes.
# TYPE kubelet_swap_used_bytes gauge
kubelet_swap_used_bytes 1073741824

This indicates about 1GB swap in use.

Inspect pod resource usage

Use kubectl top to see actual memory usage:

kubectl top pod swap-test

If kubectl top is not available, use kubectl describe to see resource requests and limits.

Review node events

Look for swap-related warnings or errors:

kubectl describe node NODE_NAME | grep -i swap

No output is good. If warnings appear, investigate further.

Diagnose pod failures due to swap

If a pod fails to start, check its events and logs:

kubectl describe pod swap-test
kubectl logs swap-test --previous

Common failure signals:

• Failed to create pod sandbox may indicate cgroup issues.
• OOMKilled suggests memory constraints, potentially related to swap misconfiguration.
• Pod stuck in ContainerCreating with swap-related messages in kubelet logs.

Validate rollback readiness

Simulate a rollback in a test environment. Apply the previous configuration and confirm pods restart cleanly.

Failure Modes and Recovery

Even with careful planning, failures occur. Know how to detect and recover from common swap-related issues.

Failure: kubelet fails to start with swap enabled

Symptom: systemctl status kubelet shows failure, logs contain failed to run Kubelet: running with swap on is not supported, please disable swap!

Cause: failSwapOn is true in kubelet config or not set (default true) while swap is active.

Recovery: Set failSwapOn: false in kubelet config, or disable swap temporarily (sudo swapoff -a), then restart kubelet. If using failSwapOn: false, ensure memorySwap.swapBehavior is set appropriately.

Failure: Node becomes NotReady after enabling swap

Symptom: kubectl get nodes shows NotReady for the node.

Cause: Kubelet may have restarted and lost connection, or system resources misconfigured.

Recovery:

1. Check kubelet status and logs.
2. Ensure kubelet has restarted properly.
3. Verify network connectivity and node conditions:

   kubectl describe node NODE_NAME

1. If swap caused kernel issues, disable swap and restart.

Failure: Unexpected pod evictions

Symptom: Pods are evicted despite available memory, with node condition MemoryPressure.

Cause: Swap may be incorrectly accounted for. With LimitedSwap, kubelet may incorrectly calculate available memory, leading to evictions.

Recovery: Adjust kubelet eviction thresholds. In kubelet config, set:

evictionHard:
  memory.available: "200Mi"
  nodefs.available: "10%"

Restart kubelet and monitor.

Failure: Pods with Guaranteed QoS attempt to use swap

Symptom: Pods with Guaranteed QoS (requests equal limits) may be terminated if they try to use swap.

Cause: In LimitedSwap mode, Guaranteed pods are not allowed to use swap. If the workload requires swap, change its QoS class to Burstable.

Recovery: Modify pod resource requests/limits to make it Burstable, or consider changing swap behavior if appropriate.

Disaster Recovery: Full node restore

If a node becomes unrecoverable, rebuild it from scratch using a configuration management tool or manual steps:

1. Provision a new node with same OS and specs.
2. Join it to the cluster (using kubeadm join or cloud provider).
3. Reapply swap configuration: create swap file, update kubelet config, restart kubelet.
4. Verify node readiness and pod scheduling.

Backup strategy for swap configuration

Treat node configuration as code. Store kubelet config, systemd unit files, and swap setup scripts in version control. Use tools like Ansible, Terraform, or cloud-init to rebuild nodes consistently. Periodically test restore procedures in a staging environment.

Example backup script:

#!/bin/bash
# Backup kubelet config and swap settings
mkdir -p ~/k8s-backup/$(date +%Y%m%d)
cp /var/lib/kubelet/config.yaml ~/k8s-backup/$(date +%Y%m%d)/kubelet-config.yaml
cp /etc/fstab ~/k8s-backup/$(date +%Y%m%d)/fstab
swapon --show > ~/k8s-backup/$(date +%Y%m%d)/swap-status.txt

Operations Checklist

Use this checklist to ensure consistency and safety when managing swap on Kubernetes nodes.

Before any change

• Confirm Kubernetes version supports swap (>=1.28 and cgroup v2).
• Verify node operating system supports swap with cgroup v2.
• Take a backup of kubelet config: cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak-$(date +%F).
• Record current swap status: swapon --show and free -h.
• Cordone and drain node if making disruptive changes.

During configuration

• Apply one change at a time.
• Use the minimal memorySwap configuration required.
• Restart kubelet and verify status.
• Check kubelet logs for errors.
• Deploy a test pod and observe swap behavior.

Verification

• Run kubectl get nodes to confirm node is Ready.
• Check kubectl describe node NODE_NAME for MemoryPressure conditions.
• Inspect pod scheduling and resource usage.
• Validate that rollback plan is executable.

Documentation and rollback

• Update inventory with new configuration.
• Store backup copies in a secure location.
• Document any troubleshooting steps taken.
• Schedule a review of swap usage metrics.

Conclusion

Managing Kubernetes swap memory requires a well-defined backup, restore, and disaster recovery strategy. By inventorying your environment, applying safe configuration changes, verifying outcomes, and preparing for failures, you can leverage swap benefits without compromising cluster stability. The practical commands and examples in this guide provide a foundation for reliable operations.

As a next step, implement a version-controlled configuration management system for node settings, including swap. Test your backup and restore procedures in a staging environment, and document your recovery runbook. With these practices, your Kubernetes clusters will be resilient and maintainable.