Kubernetes Node Pressure Eviction: A Production Operations Checklist

Intro

Node pressure eviction is one of the most disruptive events in a Kubernetes production cluster. When a node runs low on memory, disk, or inodes, the kubelet begins terminating pods to reclaim resources. Without a clear operational checklist, teams often scramble: they run ad-hoc commands, make risky changes, or misinterpret pod eviction signals as application failures.

This article provides a production operations checklist for Kubernetes node pressure eviction. It is written for developers, DevOps consultants, and technical startup teams who operate clusters or support workloads under resource stress. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Every recommendation in this checklist includes the relevant component, supported version range, prerequisites, a read-only observation step, the smallest justified change, and a verification command or signal. We avoid theory and instead give concrete examples, expected outputs, and failure signals.

Use this guide when:

• You see Evicted or OOMKilled pods in kubectl get pods.
• You receive alerts for node conditions like MemoryPressure or DiskPressure.
• You need to tune eviction thresholds or resource requests/limits.
• You want to systematically diagnose and recover from a node pressure event.

Throughout, we refer to related areas such as Node, Resource Requests and Limits, and Kubelet only when they affect prerequisites, compatibility, security, observability, or recovery.

Version and Environment Inventory

Before touching a production cluster, you must know exactly what you are working with. The Version and Environment Inventory step names the relevant component, supported version range, prerequisites, a read-only observation command, the smallest justified change, and the command or signal that verifies the outcome.

For Kubernetes node pressure eviction, the key components are:

• Kubernetes control plane and node version: Eviction behavior changes across releases. For example, the eviction-soft and eviction-hard flags have been stable since v1.12, but some defaults changed. Always check kubectl version.
• Kubelet version: The kubelet is the component that actually enforces eviction. Use kubectl get nodes -o wide and then kubectl describe node <node-name> to see kubelet version.
• Container runtime: The runtime (containerd, CRI-O, Docker) affects how memory and disk usage is reported. Check the node's runtime with kubectl get node <node-name> -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}'.
• Operating system and kernel: Some eviction signals like memory.available depend on cgroup v1 or v2. Check the node OS with kubectl get node <node-name> -o jsonpath='{.status.nodeInfo.osImage}'.

Prerequisites

• Cluster access with sufficient permissions (at least get and describe on nodes and pods).
• kubectl installed locally, matching the cluster minor version or within one minor version.
• A test namespace or cluster for safe experiments.
• Understanding of your node's allocatable resources and existing workload demands.

Read-only observation

Start with read-only commands to capture current state and timestamps:

kubectl get nodes -o wide
kubectl describe node <node-name> | grep -A 10 Conditions
kubectl get pods --all-namespaces -o wide | grep Evicted

Look for conditions such as MemoryPressure=True, DiskPressure=True, or PIDPressure=True. These are the first signals of imminent eviction.

Smallest justified change

Only after you have observed the state should you consider a change. A minimal intervention might be:

• Cordon the node to prevent new pods from being scheduled: kubectl cordon <node-name>.
• Add a temporary pod with generous requests to verify scheduling behavior (only in a test namespace).
• Adjust a single pod's resource requests/limits in a non-production environment first.

Verify the outcome

After any change, verify the result:

kubectl get node <node-name> -o jsonpath='{.status.conditions[?(@.type=="MemoryPressure")].status}'

If the condition changes from True to False, your change had the intended effect.

Always separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood.

Safe Configuration Path

Kubelet eviction thresholds are the primary configuration for node pressure eviction. The Safe Configuration Path ensures you understand the flags, their defaults, and how to change them without destabilizing the node.

Component and version range

• Kubelet configuration file: typically /var/lib/kubelet/config.yaml on the node, or managed by a ConfigMap if using kubeadm with kubelet-config.
• The eviction-related flags are --eviction-hard, --eviction-soft, --eviction-soft-grace-period, --eviction-max-pod-grace-period, --eviction-minimum-reclaim, and --system-reserved/--kube-reserved.
• These flags are stable in Kubernetes v1.12+ but verify your version's documentation because some defaults differ between cloud providers and on-premises.

Prerequisites

• Node access (SSH or a privileged pod) to view/edit the kubelet config, or permission to update the kubelet ConfigMap if using kubeadm.
• A backup of the current kubelet configuration.
• Understanding of your node's memory and disk capacity.

Read-only observation

View the current effective eviction thresholds:

kubectl proxy &
curl -s http://localhost:8001/api/v1/nodes/<node-name>/proxy/configz | jq '.kubeletconfig.evictionHard'

This returns a JSON object like:

{
  "memory.available": "100Mi",
  "nodefs.available": "10%",
  "nodefs.inodesFree": "5%",
  "imagefs.available": "15%"
}

If the configz endpoint is not enabled, check the kubelet flags on the node:

ps aux | grep kubelet | grep eviction

Or inspect the config file:

cat /var/lib/kubelet/config.yaml

Smallest justified change

Suppose you want to increase the hard eviction threshold for memory from 100Mi to 200Mi to reduce premature evictions. On a kubeadm cluster, edit the kubelet ConfigMap in the kube-system namespace:

kubectl edit cm -n kube-system kubelet-config-1.24

Find the evictionHard section and change:

evictionHard:
  memory.available: "200Mi"

Then restart the kubelet on the node (do this one node at a time):

sudo systemctl restart kubelet

Verify the outcome

After restart, confirm the new threshold is active:

curl -s http://localhost:8001/api/v1/nodes/<node-name>/proxy/configz | jq '.kubeletconfig.evictionHard'

Expected output shows "memory.available": "200Mi".

Always keep the local test small. Apply one manifest or config change, inspect the generated resources, and verify with read-only commands before moving to broader changes. For kubelet changes, consider using a canary node (a non-critical node) first.

Safety notes

• Never set evictionHard thresholds too low; they may cause immediate pod evictions.
• If you use soft eviction, set graceful periods long enough for applications to save state.
• Monitor node memory usage after changes with kubectl top node.

Verification and Diagnostics

Verification and Diagnostics focus on confirming that pressure exists, identifying the source, and understanding which pods are at risk. This section provides a sequence of read-only commands and the expected outputs.

Read-only observation

Start with cluster-wide pod status:

kubectl get pods --all-namespaces -o wide

Look for pods with status Evicted, OOMKilled, or CrashLoopBackOff. The reason is shown in the STATUS column or via kubectl describe.

For a specific pod:

kubectl describe pod <pod-name> -n <namespace>

In the Events section, look for messages like:

Warning  Evicted  ...  The node was low on resource: memory.

Or:

Warning  OOMKilled  ...  Container was OOM-killed.

Check node conditions:

kubectl describe node <node-name> | grep -A 10 Conditions

If MemoryPressure is True, the node is actively reclaiming memory.

Check current resource usage:

kubectl top node
kubectl top pod -n <namespace>

Compare MEMORY(bytes) usage to Allocatable from kubectl describe node.

Diagnostic example

Suppose you see a pod repeatedly evicted. Run:

kubectl get events --sort-by=.metadata.creationTimestamp | grep Evicted

Output might show:

3m         Warning   Evicted   pod/web-app-7b9f8c6d5-xyz   The node had condition: [MemoryPressure].

This tells you the node was under memory pressure, not that the pod itself was faulty.

Verify the outcome of a fix

If you add more memory to the node or reduce load, verify the pressure condition clears:

kubectl get node <node-name> -o jsonpath='{.status.conditions[?(@.type=="MemoryPressure")].status}'

If the output is False, the pressure has subsided.

For pod eviction, verify that no new evictions occur after your intervention by watching events for 10 minutes:

kubectl get events --watch | grep Evicted

Useful diagnostic commands

• kubectl describe pod <name> for scheduling and event details.
• kubectl logs <name> --previous for crash loops to distinguish application crashes from evictions.
• kubectl rollout status deployment/<name> before assuming a release succeeded; a rollout may be stuck due to node pressure.

Always separate observation from intervention. Capture current state and timestamps first, protect credentials and private material, then change one scoped item only when its blast radius and recovery path are understood.

Failure Modes and Recovery

Node pressure eviction can manifest in several failure modes. Understanding them helps you recover quickly and safely.

Failure mode 1: MemoryPressure eviction (OOM)

Signal

• Node condition MemoryPressure=True.
• Pods evicted with reason The node was low on resource: memory.
• Pods in OOMKilled state.

Recovery steps

1. Identify which pods are consuming the most memory: kubectl top pods --all-namespaces --sort-by=memory.
2. For each high-memory pod, check its memory requests and limits: kubectl get pod <pod> -n <ns> -o yaml | grep -A 5 resources.
3. If a pod has no memory limits, it may consume all node memory. Add a limit as a temporary mitigation:

resources:
  limits:
    memory: "512Mi"
  requests:
    memory: "256Mi"

1. If the node is permanently undersized, cordon and drain it to reschedule pods: kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data.
2. Investigate the root cause: application memory leak, insufficient requests scaling, or a sudden traffic spike.

Verification

After applying limits, ensure no more OOM kills:

kubectl get pods --all-namespaces -o wide | grep OOMKilled

Should return no results after some time.

Failure mode 2: DiskPressure eviction

Signal

• Node condition DiskPressure=True.
• Pods evicted with reason The node was low on resource: ephemeral-storage or nodefs.
• Node disk usage above 80-90% for the filesystem used by container logs or images.

Recovery steps

1. Check disk usage on the node (if accessible): df -h /var/lib/containerd /var/log.
2. Clean up unused images: crictl rmi --prune (on the node).
3. Remove old container logs: journalctl --vacuum-size=200M or truncate -s 0 /var/log/containers/*.log (careful with active logs).
4. If the pressure is due to ephemeral storage requests, set resources.requests.ephemeral-storage on pods to enforce capacity planning.
5. If the node disk is full, drain and re-provision after cleanup.

Verification

Check the condition:

kubectl get node <node-name> -o jsonpath='{.status.conditions[?(@.type=="DiskPressure")].status}'

Should return False.

Failure mode 3: PIDPressure eviction

Signal

• Node condition PIDPressure=True.
• Running pods show fork: retry: Resource temporarily unavailable in logs.
• System may be unresponsive due to process exhaustion.

Recovery steps

1. Identify process count on the node: ps -eLf | wc -l (if node access).
2. Reduce per-pod PID limits by setting resources.requests.pids or limits.pids in pod specs.
3. Restart leaking processes (application bug).
4. Increase pid_max on the node if it is set too low: sysctl kernel.pid_max=32768 (temporary).

Verification

Check the condition:

kubectl get node <node-name> -o jsonpath='{.status.conditions[?(@.type=="PIDPressure")].status}'

Should return False.

Failure mode 4: Eviction of critical system pods

Signal

• Kube-system pods such as kube-dns, kube-proxy, or CNI pods get evicted.
• Cluster functions degrade.

Recovery steps

1. Ensure critical pods have high priority classes. Check priority: kubectl get pod -n kube-system -o yaml | grep priorityClassName.
2. If not set, create a high-priority class and assign to critical pods.
3. Temporarily increase node resources or reduce load.
4. Investigate why system pods are under pressure; they may have insufficient requests.

Verification

Ensure system pods are running and not evicted after recovery:

kubectl get pods -n kube-system | grep -v Running

Should show only completed pods or none.

For all failure modes, always document the recovery procedure and expected outputs for future incidents.

Operations Checklist

This section consolidates the entire workflow into a concise, actionable checklist for production operations. Use it before, during, and after a node pressure event.

Pre-incident checklist

• [ ] Verify cluster version and kubelet version: kubectl version and kubectl get nodes -o wide.
• [ ] Review current eviction thresholds: curl -s http://localhost:8001/api/v1/nodes/<node>/proxy/configz | jq '.kubeletconfig.evictionHard'.
• [ ] Confirm monitoring for node conditions: set up alerts for MemoryPressure, DiskPressure, PIDPressure lasting more than 5 minutes.
• [ ] Ensure all production pods have resource requests and limits for CPU, memory, and ephemeral storage.
• [ ] Test high-priority critical pods have PriorityClasses.
• [ ] Document a baseline of normal node resource usage with kubectl top node.
• [ ] Practice a tabletop recovery: simulate a memory pressure by deploying a memory-hog pod in a test namespace and walk through the checklist.

Example tabletop exercise

In a test namespace, deploy:

apiVersion: v1
kind: Pod
metadata:
  name: memory-hog
spec:
  containers:
  - name: stress
    image: polinux/stress
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "500M", "--vm-hang", "1"]

Watch kubectl top pod memory-hog and observe node memory pressure. Then kill the pod and verify recovery.

During incident checklist

• [ ] Identify the affected node: kubectl get nodes -o wide and look for pressure conditions.
• [ ] Determine the pressure type: memory, disk, or PID.
• [ ] For memory: list top memory-consuming pods: kubectl top pods --all-namespaces --sort-by=memory.
• [ ] For disk: check node filesystem and clean images/logs if safe.
• [ ] Cordon the node if you need to prevent scheduling: kubectl cordon <node>.
• [ ] Drain the node if you must move workloads: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data.
• [ ] Do not manually delete pods unless necessary; eviction is the kubelet's job, and manual deletion may cause flapping.
• [ ] Communicate status to stakeholders with timestamps.

Post-incident checklist

• [ ] Verify pressure conditions are clear: kubectl get node <node> -o jsonpath='{.status.conditions[?(@.type=="MemoryPressure")].status}'.
• [ ] Check that all evicted pods have been rescheduled and are running: kubectl get pods --all-namespaces | grep -v Running.
• [ ] Review logs and events for root cause: kubectl get events --sort-by=.metadata.creationTimestamp | grep -E 'Evicted|OOM'.
• [ ] Adjust resource requests/limits or eviction thresholds based on findings.
• [ ] Update documentation and runbooks.
• [ ] Schedule a postmortem meeting and assign action items (e.g., "Priya Shah, Engineering Lead: Increase node memory by 25% within 2 weeks").

Practical Kubernetes checks

Start with kubectl get pods -o wide, then use kubectl describe pod <name> for scheduling and event details, kubectl logs <name> --previous for crash loops, and kubectl rollout status deployment/<name> before assuming a release succeeded.

Keep the local test small. Apply one manifest, inspect the generated resources, and verify traffic with kubectl port-forward or a local service type before moving to a cloud load balancer or ingress controller.

Conclusion

Kubernetes Node Pressure Eviction production operations checklist with practical examples is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure.

This article provided a comprehensive checklist covering version inventory, safe configuration, diagnostics, failure modes, and recovery. We emphasized read-only observation first, minimal changes, and verification after every step.

As a next step, choose one low-risk verification for Kubernetes Node Pressure Eviction production, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as Node, Resource Requests and Limits, and Kubelet.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.

Use this checklist as a living document: after every incident, update it with new observations and lessons learned. This iterative improvement will make your Kubernetes operations more resilient over time.