Kubernetes Swap Memory Management: Advanced Concepts and Practical Implementation for Production Clusters

Intro

Managing memory in Kubernetes often centers on requests and limits, but swap memory is an equally critical piece of the puzzle. Swap space allows a node to move inactive pages from RAM to disk when memory pressure builds. Historically, Kubernetes required swap to be disabled on nodes, but that has changed with recent versions and cgroup v2 support. Misconfiguring swap can lead to unstable pods, unexpected evictions, and performance degradation.

This article provides an advanced, practical guide to Kubernetes swap memory management for developers, DevOps engineers, SREs, and technical startup teams. We will cover the underlying architecture, kubelet configuration options, node-level controls using cgroup v2, and real-world scenarios. You will learn how to observe swap usage, configure swap safely, verify the configuration, and recover from common failure modes.

Operational safety is a core theme: observe before making changes, limit the blast radius, avoid exposing sensitive information in commands, verify outcomes with concrete signals, and always document a path back to a known-good state.

Version and Environment Inventory

Before touching swap configuration, you must understand your environment. This includes the Kubernetes version, the container runtime, the node operating system, whether cgroup v2 is enabled, and the current swap status. Start with read-only observations.

Identify your Kubernetes version:

kubectl version --short

Expected output shows client and server versions, for example:

Client Version: v1.28.2
Server Version: v1.28.2

Swap support in kubelet depends on the version. From Kubernetes 1.22, swap support is alpha and requires enabling the NodeSwap feature gate. In 1.28, it is still alpha but more stable. Check your kubelet feature gates:

ps aux | grep kubelet | grep feature-gates

If NodeSwap=true is present, swap support is enabled. Otherwise, you need to add it.

Check the node's swap status:

swapon --show

If swap is active, you'll see output like:

NAME      TYPE SIZE USED PRIO
/dev/sda2 partition 2G   0B   -2

If no output, swap is disabled. Also check total memory and swap with free -h.

Verify cgroup version:

stat -fc %T /sys/fs/cgroup/

If the output is cgroup2fs, you have cgroup v2. If tmpfs, you are on cgroup v1. Swap management via kubelet requires cgroup v2.

Gather this inventory before modifying anything. Document the values and timestamps. This establishes a baseline for comparison and recovery.

Safe Configuration Path

Once the environment is understood, you can proceed to configure swap. The goal is to enable swap on the node and tell kubelet how to handle it. This involves changes at the node level (systemd, fstab) and kubelet flags.

Step 1: Enable swap on the node

If swap is not active, create a swap file or partition. Here we use a swap file of 2GB.

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

To make it persistent across reboots, add to /etc/fstab:

/swapfile none swap sw 0 0

Verify with swapon --show and free -h.

Step 2: Configure kubelet to allow swap

Kubelet has several flags related to swap:

• --fail-swap-on=true|false: If true (default), kubelet will fail if swap is enabled on the node. Set to false to allow swap.
• --memory-swap=swapBehavior: Specifies how to handle swap. swapBehavior can be:
• UnlimitedSwap: default when NodeSwap feature is enabled; no limit on swap usage.
• LimitedSwap: swap usage limited by the container's memory request (or memory limit if set). The limit is calculated as containerMemoryRequest * (1 + 100/100) by default, where the percentage is set by --memory-swap-swappiness? Actually, the limit is defined by --swap-limit-percentage or --container-swap-limit-percentage. In kubelet config, it's memorySwap.swapBehavior and memorySwap.limitedSwap with swapLimitPercentage.

For example, to set memorySwap in the kubelet config file:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memorySwap:
  swapBehavior: LimitedSwap
  limitedSwap:
    swapLimitPercentage: 50
failSwapOn: false
featureGates:
  NodeSwap: true

Save this as /var/lib/kubelet/config.yaml and ensure kubelet is started with --config=/var/lib/kubelet/config.yaml. Restart kubelet:

sudo systemctl restart kubelet

Verify kubelet is running without errors:

sudo journalctl -u kubelet -f

Look for lines indicating swap is allowed.

Step 3: Test with a pod

Create a pod that requests memory and observes the cgroup's swap limit.

pod-swap-test.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: swap-test
spec:
  containers:
  - name: app
    image: alpine
    command: ["sh", "-c", "sleep 3600"]
    resources:
      requests:
        memory: "500Mi"

Apply it:

kubectl apply -f pod-swap-test.yaml

Find the node the pod is running on and inspect the container's memory and swap limits inside the cgroup. First, get the container ID:

kubectl get pod swap-test -o jsonpath='{.status.containerStatuses[0].containerID}'

Then on the node, check the cgroup files (adjust path as needed):

# Assuming cgroup v2 and container ID is something like docker://<id>
# For containerd, the cgroup path is /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod<uid>.slice/cri-containerd-<id>.scope
# For simplicity, search for the pod cgroup
find /sys/fs/cgroup -name '*swap-test*' -type d

Once you locate the cgroup directory, check the memory.swap.max file:

cat /sys/fs/cgroup/.../memory.swap.max

With LimitedSwap and swapLimitPercentage: 50, and memory request of 500Mi, the swap limit should be 250Mi (500 * 0.5). The output might be 262144000 (bytes). Verify it matches your expectation.

If everything matches, the configuration is successful.

Verification and Diagnostics

After configuration, continuous verification is necessary. Use read-only commands to monitor swap usage per pod and at the node level.

Node-level swap usage:

free -h
swapon --show

Per-pod or per-container swap usage:

Since cgroup v2 exposes memory.swap.current, you can read it from the container's cgroup. Automate with a script that iterates over pods on a node.

Example using kubectl and node access:

# On the node, for each cgroup in kubepods, read memory.swap.current
for cg in /sys/fs/cgroup/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod*.slice; do
  echo "$cg: $(cat $cg/memory.swap.current)"
done

For a more user-friendly view, use the kubectl top command (requires metrics-server):

kubectl top pod

Note that kubectl top pod shows memory usage but does not directly show swap. To get swap info, you may need to use cAdvisor metrics or node exporter.

Check kubelet logs for swap-related messages:

journalctl -u kubelet --since "1 hour ago" | grep -i swap

Look for warnings like "swap is enabled but kubelet is not configured to use it" or errors about memory limits.

Validate pod scheduling and eviction behavior:

If swap is enabled with LimitedSwap, kubelet considers swap when determining if a pod can be scheduled on a node. Test by creating a deployment with high memory requests and see if pods are scheduled.

Verification should include both positive and negative tests: confirm that allowed pods run and that disallowed configurations are rejected.

Failure Modes and Recovery

Even with careful configuration, things can go wrong. Here are common failure modes and how to recover.

Failure 1: kubelet fails to start after enabling swap

Symptom: kubelet service crashes with an error like:

failed to run Kubelet: failed to create cgroup manager: swap is enabled but cgroup v2 not available

Or:

invalid configuration: memorySwap is set but NodeSwap feature gate is disabled

Recovery:

1. Check cgroup version: stat -fc %T /sys/fs/cgroup/. If not cgroup2fs, you cannot use memorySwap. Either upgrade to cgroup v2 or set failSwapOn: true (default) and disable swap.
2. If cgroup v2 is available but feature gate missing, add --feature-gates=NodeSwap=true to the kubelet command line or add featureGates: {NodeSwap: true} to the config file.
3. Roll back the kubelet configuration change by restoring the previous config and restarting kubelet.

Failure 2: Pods are evicted despite memory requests

Symptom: Pods are killed with Evicted status even though node has free memory but swap is heavily used.

Recovery:

• Check kubelet eviction thresholds: --eviction-hard=memory.available<100Mi or similar. If swap is enabled, kubelet may include swap in the calculation of available memory. Adjust thresholds or set --eviction-soft appropriately.
• Consider setting --memory-swap=UnlimitedSwap if you want to rely solely on swap without eviction, but be aware of performance implications.
• Monitor memory and swap usage; if swap is causing disk I/O saturation, reduce swap usage or move workloads.

Failure 3: Swap limit not enforced

Symptom: Container uses more swap than the configured limit.

Recovery:

• Verify cgroup v2 is active.
• Confirm kubelet config is loaded: kubectl get --raw /api/v1/nodes/<node>/proxy/configz | jq . (if using configz endpoint). Otherwise, check kubelet logs for the effective configuration.
• Check that the container runtime supports swap accounting. Docker and containerd with runc do if cgroup v2 is used.
• Restart the container runtime and kubelet.

Failure 4: Node becomes unresponsive due to swap thrashing

Symptom: High disk I/O, high load average, but low CPU usage. Processes are constantly swapping.

Recovery:

• Reduce or disable swap temporarily: sudo swapoff -a.
• Investigate which pods are causing memory pressure: use kubectl top pods --sort-by=memory.
• Adjust memory limits to force containers to stay within physical RAM.
• Consider using LimitedSwap with a lower percentage or setting appropriate memory requests.

Always have a rollback plan: document the exact configuration and commands to revert to the previous state.

Operations Checklist

Use this checklist to ensure safe operation in production.

Before enabling swap on any node:

• [ ] Record node OS, kernel version, cgroup version.
• [ ] Record current kubelet configuration and flags.
• [ ] Verify Kubernetes version and feature gates.
• [ ] Test in a staging environment first.
• [ ] Create a rollback plan: revert kubelet config, disable swap, restart kubelet.

During configuration:

• [ ] Enable swap on the node and verify with swapon --show.
• [ ] Set failSwapOn: false in kubelet config.
• [ ] If using memorySwap, set appropriate swapBehavior and swapLimitPercentage.
• [ ] Add NodeSwap: true to feature gates.
• [ ] Restart kubelet and check logs for errors.

After configuration:

• [ ] Create a test pod with memory requests and verify swap limit in cgroup.
• [ ] Monitor node swap usage and pod memory usage.
• [ ] Test pod scheduling and eviction behavior.
• [ ] Document all changes and observed behavior.

Ongoing:

• [ ] Regularly audit swap usage.
• [ ] Keep kubelet and node OS updated.
• [ ] Review feature gate status as Kubernetes evolves (swap support is alpha and may change).

Conclusion

Kubernetes swap memory management is a nuanced topic that requires careful planning and execution. With the introduction of cgroup v2 and the NodeSwap feature gate, it is now possible to safely enable swap on nodes and control its usage per container. This guide has walked through the essential steps: understanding your environment, enabling swap, configuring kubelet, verifying the configuration, and handling failures.

Always start with observation, make minimal changes, verify each step, and keep a rollback path. Swap can be a useful tool to handle memory spikes, but it must be managed with clear limits and monitoring to avoid performance issues. By applying the practices outlined here, you can confidently manage swap memory in your Kubernetes clusters.

For further exploration, dive into the Kubernetes documentation on memory swap and cgroup v2, and experiment in a test cluster before production rollout.