Kubernetes Swap Memory Management: Monitoring, Metrics, Alerts, and Recovery

Intro

Kubernetes clusters often disable swap on nodes, but some environments enable it for development, edge deployments, or when required by local policies. Whether swap is enabled or disabled, effective swap memory management requires monitoring, alerts, dashboards, and incident response procedures that are based on concrete signals rather than guesswork.

This guide provides practical, operator-focused instructions for Kubernetes swap memory management. It covers version and environment inventory, safe configuration changes, verification and diagnostics, failure modes and recovery, and an operations checklist. Every section includes real commands, expected outputs, and failure signals. The target audience includes developers, DevOps consultants, and technical startup teams running self-managed or cloud Kubernetes clusters.

The operational philosophy is safe by design: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document recovery paths before an incident forces a decision.

Version and Environment Inventory

Before making any configuration change, document the Kubernetes components involved, their supported version ranges, prerequisites, a read-only observation approach, the smallest justified change, and the command or signal that verifies the outcome. For swap memory management, the primary components are the kubelet, the node operating system, container runtime, and possibly systemd units or cloud-init scripts.

Separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood.

Key concepts in this section intersect with Kubernetes monitoring, alerts, metrics, dashboards, and incident response. Areas such as Node, Kubelet, and Resource Requests and Limits are included only when they directly affect swap behavior or troubleshooting.

Practical inventory commands

Start with read-only commands to establish the baseline:

# Kubernetes version
kubectl version --short

# Node list with kubelet version
kubectl get nodes -o wide

# Detailed node info, including kubelet version and OS image
kubectl describe node <node-name>

Example expected output for kubectl get nodes -o wide:

NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node-1         Ready    control-plane   10d   v1.28.5   10.0.0.10      <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.13
node-2         Ready    <none>          10d   v1.28.5   10.0.0.20      <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.13

Check swap status on each node via SSH or a debug pod:

# On the node
swapon --show
free -h
cat /proc/swaps

If swap is active, you will see output similar to:

NAME      TYPE      SIZE   USED PRIO
/dev/sda3 partition 4G     0B   -2

For the kubelet configuration, inspect the active flags:

# On the node
ps aux | grep kubelet
# Or inspect the config file
cat /var/lib/kubelet/config.yaml | grep -i swap

In Kubernetes 1.28 with the NodeSwap feature gate enabled, a typical kubelet configuration includes:

memorySwap:
  swapBehavior: LimitedSwap

If swapBehavior is blank, the default is NoSwap, which means the kubelet will fail if swap is detected unless --fail-swap-on=false is set.

Version and feature gate matrix

Use the following concrete reference for common Kubernetes versions:

Always verify against your exact version by running kubectl version and reading the release notes.

Safe Configuration Path

Changing swap settings on a node can disrupt running pods if not done correctly. Always follow a safe configuration path: identify the component, supported version, prerequisites, read-only observation, smallest justified change, and verification command.

Prerequisites

• Access to the node (SSH or privileged pod) with sudo rights.
• Backup of the current kubelet configuration and system state.
• A maintenance window or the ability to cordon and drain the node.
• Knowledge of the node's role and workload sensitivity.

Read-only observation before change

Before altering swap, record the current state:

# Cordon and drain the node (if possible in a test environment)
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Backup kubelet config
sudo cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak

# Check current swap usage
free -h
swapon --show

Smallest justified change: enabling swap on a node

Suppose you need to enable a 2GB swap file on an Ubuntu node for limited swap support in Kubernetes 1.28.

Steps:

1. Create a swap file:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

1. Make it persistent (optional, but recommended for controlled environments):

echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

1. Configure kubelet to use LimitedSwap:

Edit /var/lib/kubelet/config.yaml and add:

memorySwap:
  swapBehavior: LimitedSwap

1. Restart kubelet:

sudo systemctl restart kubelet

1. Verify kubelet is running and swap is recognized:

sudo systemctl status kubelet
kubectl describe node <node-name> | grep -A5 "Conditions"

Expected output should show MemoryPressure condition as False initially.

Smallest justified change: disabling swap for compliance

If you need to disable swap permanently:

sudo swapoff -a
# Remove swap entry from /etc/fstab
sudo sed -i '/swap/d' /etc/fstab
sudo systemctl restart kubelet

Then verify with swapon --show (should output nothing) and kubectl get nodes to confirm the node is Ready.

Verification and Diagnostics

After any configuration change, verify that the system behaves as intended. Use read-only diagnostics first, then follow with specific tests.

Monitoring swap usage on a node

Use kubectl top if metrics-server is installed, but for swap-specific data use node-level commands or Prometheus node exporter.

If you have a privileged debug pod, run:

kubectl debug node/<node-name> -it --image=ubuntu -- /bin/bash
# Inside the debug pod
chroot /host
swapon --show
free -h
cat /proc/meminfo | grep -i swap

Expected output for a healthy node with swap:

SwapTotal:        2097148 kB
SwapFree:         2097148 kB

Prometheus metrics for swap

If using Prometheus with node_exporter, query these metrics:

• node_memory_SwapTotal_bytes
• node_memory_SwapFree_bytes
• node_memory_SwapCached_bytes
• node_vmstat_pswpin
• node_vmstat_pswpout

Example PromQL for swap usage percentage:

(node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes) / node_memory_SwapTotal_bytes * 100

For kubelet-specific swap metrics (Kubernetes 1.28+), use kubelet metrics endpoint:

curl -s http://localhost:10255/metrics | grep swap

(Note: kubelet read-only port may be disabled; use --kubelet-insecure-tls if needed.)

Pod-level diagnostics

Check if a pod is being affected by memory pressure:

kubectl describe pod <pod-name>

Look for events like:

Warning  Evicted    10m   kubelet  The node was low on resource: memory.

If swap is active and swapBehavior: LimitedSwap, pods may use swap up to the limit. Monitor pod memory usage with:

kubectl top pod <pod-name>

Example:

NAME        CPU(cores)   MEMORY(bytes)
pod-name    100m         512Mi

If memory exceeds requests and limits, the pod may be throttled or killed. Review the pod's QoS class and limits:

kubectl get pod <pod-name> -o yaml | grep -A5 resources

Failure Modes and Recovery

Even with careful planning, failures occur. Here are common failure modes related to swap management, their symptoms, and recovery steps.

Failure mode 1: Kubelet fails to start when swap is on and NoSwap is default

Symptom: Node becomes NotReady, and kubelet logs show:

F0128 10:00:00.000000   12345 server.go:275] failed to run Kubelet: running with swap on is not supported, please disable swap! or set --fail-swap-on=false

Recovery:

Option A: Disable swap immediately:

sudo swapoff -a
sudo systemctl restart kubelet

Option B: Set kubelet to allow swap (development only):

Edit /var/lib/kubelet/config.yaml:

failSwapOn: false

Or add --fail-swap-on=false to the kubelet command line.

Then restart kubelet.

Failure mode 2: Pods are evicted despite swap being available

Symptom: kubectl get pods shows pods in Evicted state, and node logs indicate memory pressure.

Root cause: With LimitedSwap, the kubelet still enforces memory limits. Swap only affects the node-level memory pressure threshold, not pod limits. So a pod exceeding its limit is killed regardless of swap.

Recovery:

• Increase pod memory limit if feasible: kubectl set resources deployment/<name> --limits=memory=1Gi
• Reduce memory usage in the application.
• Check node memory pressure: kubectl describe node <node-name> | grep -A5 Conditions

Failure mode 3: Swap file fills up and node becomes unresponsive

Symptom: High swap usage, sluggish node, and node_memory_SwapFree_bytes approaches zero.

Recovery:

• Identify heavy swap consumers using smem or top sorted by swap:

sudo smem -s swap -r

• Temporarily add more swap if disk space allows:

sudo fallocate -l 2G /swapfile2
sudo chmod 600 /swapfile2
sudo mkswap /swapfile2
sudo swapon /swapfile2

• Restart problematic pods or migrate workloads.

• Set an alert to detect high swap usage before it becomes critical (see Operations Checklist).

Operations Checklist

Use this daily/weekly checklist to maintain swap memory health.

Daily checks

• [ ] Run kubectl get nodes and ensure all nodes are Ready.
• [ ] Check node memory pressure conditions:

  kubectl get nodes -o custom-columns='NAME:.metadata.name,MemoryPressure:.status.conditions[?(@.type=="MemoryPressure")].status'

Expected output: False for all nodes.

• [ ] Review swap usage on nodes via monitoring dashboard or command:

  kubectl debug node/<node-name> -it --image=ubuntu -- chroot /host swapon --show

Weekly checks

• [ ] Verify kubelet configuration is consistent across nodes:

  kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}'

• [ ] Audit any recent changes to kubelet flags or swap settings.
• [ ] Test alerting rules by triggering a test alert (e.g., temporarily set a low threshold in a test environment).
• [ ] Review pod eviction events:

  kubectl get events --all-namespaces | grep -i evicted

Alerting rules examples (Prometheus)

Create an alert for high swap usage on a node:

groups:
- name: swap_alerts
  rules:
  - alert: NodeSwapUsageHigh
    expr: (node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes) / node_memory_SwapTotal_bytes > 0.8
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Swap usage above 80% on {{ $labels.instance }}"
      description: "Swap usage has been above 80% for 5 minutes."
  - alert: NodeSwapUsageCritical
    expr: (node_memory_SwapTotal_bytes - node_memory_SwapFree_bytes) / node_memory_SwapTotal_bytes > 0.95
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Swap usage above 95% on {{ $labels.instance }}"

For kubelet metrics (if available), alert on kubelet_swap_usage_bytes exceeding a threshold.

Dashboard recommendations

Build a Grafana dashboard with panels for:

• Swap usage percentage per node (time series).
• Swap in/out rate per node (node_vmstat_pswpin, node_vmstat_pswpout).
• Node memory pressure condition.
• Pod eviction count per namespace.

Example panel query for swap usage:

100 * (1 - (node_memory_SwapFree_bytes / node_memory_SwapTotal_bytes))

Set thresholds at 50% (yellow), 80% (orange), 95% (red).

Conclusion

Kubernetes swap memory management is often overlooked until it causes node instability or pod evictions. By following the structured approach in this guide, you can proactively monitor swap, configure it safely, detect failures early, and recover quickly.

As a next step, choose one low-risk verification from this article, such as checking swap status on a node, and record the current state. Then run the documented command and compare the output with the expected signal. Gradually implement the alerting rules and dashboard panels to build a comprehensive swap memory observability stack.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With these practices, you can manage swap memory in Kubernetes with confidence.