Kubernetes Node Pressure Eviction Configuration Mistakes with Practical Examples

Intro

Node pressure eviction is Kubernetes' way of reclaiming node resources when they run low. When a node's memory, disk, or inodes cross configured thresholds, the kubelet starts terminating pods to prevent the node from becoming unstable. A misconfiguration here can evict critical workloads, cause cascading failures, or leave the cluster silently degraded. This guide walks through the common mistakes operators make when configuring eviction thresholds and how to fix them with practical, verifiable steps.

We will cover how to inspect your current eviction setup, set safe thresholds, validate changes, diagnose eviction events, and roll back when things go wrong. Every recommendation includes commands, expected output, or example scenarios so you can follow along in your own cluster. The goal is operational safety: observe first, change one thing at a time, and verify before moving on.

Version and Environment Inventory

Before touching anything, know exactly what you are running. Node pressure eviction behavior depends on the Kubernetes version, the kubelet configuration, and the node's capacity. Start by collecting a read-only inventory.

Check the cluster and kubelet version:

kubectl version --short
kubectl get nodes -o wide

Example output:

Client Version: v1.29.1
Server Version: v1.29.1
NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node-1     Ready    control-plane   10d   v1.29.1   10.0.0.10     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11
node-2     Ready    <none>          10d   v1.29.1   10.0.0.11     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11

To see the current eviction configuration, check the kubelet's startup flags or its config file. On a systemd-based node, run:

systemctl status kubelet
ps aux | grep kubelet

The kubelet may be using a config file specified by --config. Look for that path, then inspect it:

sudo cat /var/lib/kubelet/config.yaml

Example snippet from a default config:

evictionHard:
  memory.available: "100Mi"
  nodefs.available: "10%"
  imagefs.available: "15%"
  nodefs.inodesFree: "5%"
evictionSoft:
  memory.available: "200Mi"
  nodefs.available: "20%"
evictionSoftGracePeriod:
  memory.available: "1m30s"
  nodefs.available: "1m30s"
evictionPressureTransitionPeriod: "5m"

If the config file is missing or no eviction section exists, the kubelet uses built-in defaults. Those defaults are:

• evictionHard: memory.available<100Mi, nodefs.available<10%, imagefs.available<15%, nodefs.inodesFree<5%
• evictionSoft: none
• evictionPressureTransitionPeriod: 5m

Confirm the node's actual capacity with:

kubectl describe node node-1 | grep -A5 "Capacity:"

This gives you the allocatable memory, ephemeral storage, and inodes so you can reason about thresholds relative to real sizes.

Now you have a baseline. Write down the current thresholds, capacity, and version. If something goes wrong later, you can compare against this snapshot.

Safe Configuration Path

Changing eviction thresholds is a node-level operation that affects scheduling and pod stability. Follow a careful, incremental path with a clear rollback plan.

Step 1: Choose the Right Threshold Type

Kubernetes supports two eviction policies: hard and soft.

• Hard eviction: When a resource falls below the threshold, the kubelet immediately evicts pods. No grace period.
• Soft eviction: When a resource falls below the threshold, the kubelet waits for a grace period before evicting. If the resource recovers above the threshold during the grace period, no eviction occurs.

Mistake: Using soft eviction with a long grace period can allow a node to spiral into memory pressure before pods are evicted. Using hard eviction with a very high threshold can cause sudden eviction storms.

Recommendation: Start with hard eviction for memory because memory pressure is immediate and can crash the node. Use soft eviction for disk pressure, where temporary spikes are common.

Step 2: Set Thresholds Based on Measured Baselines

Do not copy thresholds from a blog post. They must relate to your node size and workload profile. For a node with 16 GB RAM, a memory.available threshold of 100Mi is only 0.6% of total memory, which may be too low. A better starting point might be 500Mi or 5% of allocatable memory.

To measure current memory usage under load:

kubectl top node node-1

Example output:

NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-1   1200m        30%    11Gi            68%

If the node routinely runs at 75% memory, setting memory.available to 10% of total gives a buffer before eviction. For a 16 GiB node, 10% is about 1.6 GiB. The threshold string would be:

evictionHard:
  memory.available: "1.6Gi"

For disk, monitor nodefs.available and nodefs.inodesFree. Use df -h and df -i on the node:

df -h /
df -i /

Set thresholds that leave enough headroom for container logs and image pulls. A common mistake is setting nodefs.available too low, causing evictions when the container runtime needs space for new images.

Step 3: Apply the Configuration One Node at a Time

Never change the kubelet config on all nodes simultaneously. If the threshold is too aggressive, you could evict pods across the whole cluster at once.

To change the kubelet config on a node:

1. Edit the kubelet config file (e.g., /var/lib/kubelet/config.yaml).
2. Restart the kubelet:

sudo systemctl restart kubelet

1. Check that the kubelet is running and the node is still Ready:

systemctl status kubelet
kubectl get node node-1

If the node becomes NotReady or pods start evicting unexpectedly, revert the file and restart kubelet again.

Step 4: Use a DaemonSet or Managed Node Pool for Consistency

In managed Kubernetes (EKS, GKE, AKS), you often cannot edit the kubelet config directly. Instead, you modify the node pool's kubelet arguments or use a custom launch template. For example, on EKS with a self-managed node group, you can pass kubelet extra arguments in the user data script:

--eviction-hard=memory.available<500Mi,nodefs.available<10%

For GKE, you can configure node system configs via the kubelet-config in the node pool specification.

Always test on a single node pool or a canary node before rolling out to production.

Verification and Diagnostics

After changing eviction settings, you need to verify that the kubelet picked up the new configuration and that eviction behaves as expected. You also need to diagnose any eviction events that were already happening.

Check the Active Kubelet Configuration

You can confirm the active eviction thresholds by looking at the kubelet's logs during startup or by using the kubelet's /configz endpoint (if enabled). A simpler way is to check the node's status conditions:

kubectl describe node node-1 | grep -A10 "Conditions:"

Look for conditions like MemoryPressure, DiskPressure, PIDPressure. If any are True, the node is currently under pressure and evictions may be imminent.

You can also inspect kubelet logs for eviction-related messages:

journalctl -u kubelet --since "10 minutes ago" | grep -i evict

Example log line:

I0202 10:15:23.123456    1234 eviction_manager.go:340] "Eviction manager: attempting to reclaim" resourceName="memory.available"

This tells you the kubelet is actively trying to reclaim memory.

Simulate Pressure Safely

To test your thresholds without waiting for a real crisis, you can deliberately create pressure on a node using a pod that consumes resources. For memory pressure:

apiVersion: v1
kind: Pod
metadata:
  name: memory-eater
spec:
  containers:
  - name: memory-eater
    image: polinux/stress
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "2G", "--vm-hang", "0"]
    resources:
      requests:
        memory: "2G"
        cpu: "100m"
      limits:
        memory: "2G"
        cpu: "200m"

Schedule this pod on the target node using nodeSelector or nodeName. The pod requests 2G memory, which adds pressure. With a threshold of e.g. 500Mi available memory, you should see the kubelet start evicting pods. Watch the events:

kubectl get events --field-selector involvedObject.name=memory-eater --watch

If eviction does not happen when you expect it, check the actual thresholds in the kubelet config and the node's current memory usage. It may be that the node had enough headroom.

Diagnose Eviction Reasons and Order

When a pod is evicted, its status reason will be Evicted. The event will include a message like:

The node was low on resource: memory. Threshold quantity: 500Mi, available: 420Mi.

Use these messages to understand which resource triggered the eviction and how far below the threshold the node was.

To see which pods were evicted recently:

kubectl get pods --all-namespaces --field-selector=status.phase=Failed -o wide

Then inspect each failed pod:

kubectl describe pod <evicted-pod> -n <namespace> | grep -A10 "Events:"

The event will show the eviction reason and timestamp.

Also check pod priority. The kubelet evicts pods in a specific order: BestEffort pods first, then Burstable pods that use more than their requests, then Guaranteed pods (with some caveats). In newer versions, pod priority and QoS class both matter. A common mistake is to have critical system pods with no priority class, making them candidates for eviction. Always set appropriate priority classes for critical workloads.

Failure Modes and Recovery

Even with careful planning, eviction misconfigurations can cause serious problems. Here are the most common failure modes and how to recover from them.

Failure 1: Eviction Storms

Symptom: Many pods evicted suddenly across one or more nodes, causing service disruption.

Cause: Hard eviction threshold set too high (e.g., memory.available: 2Gi on an 8Gi node) causing immediate evictions as soon as memory drops below 25%. Or soft eviction with a very short grace period.

Recovery:

1. Immediately increase the threshold or switch to soft eviction with a longer grace period.
2. Restart kubelet on affected nodes.
3. Monitor pod rescheduling and node pressure.

Example revised config:

evictionSoft:
  memory.available: "500Mi"
evictionSoftGracePeriod:
  memory.available: "2m"
evictionHard:
  memory.available: "200Mi"

This gives pods a 2-minute grace period before hard eviction at 200Mi available.

Failure 2: Node Becomes NotReady Due to Resource Exhaustion

Symptom: Node flips to NotReady, kubelet may crash, pods on the node become inaccessible.

Cause: Eviction thresholds set too low, so the node runs out of memory or disk before the kubelet evicts anything. The kernel may OOM-kill critical processes, including the kubelet itself.

Recovery:

1. Free up resources manually: delete unnecessary pods, clean up old container images, or restart the node.
2. Once node is Ready, revise thresholds upward to give more headroom.
3. Consider setting system reserved resources (--system-reserved, --kube-reserved) so the kubelet accounts for system overhead.

Example kubelet flags:

--system-reserved=cpu=500m,memory=1Gi
--kube-reserved=cpu=500m,memory=1Gi

Failure 3: Critical System Pods Evicted

Symptom: CoreDNS, kube-proxy, or other system pods get evicted, breaking cluster networking or DNS.

Cause: No priority class set for system pods, so they are treated as BestEffort or Burstable and evicted early.

Recovery:

1. Ensure critical pods have high priority classes. For example, system-cluster-critical or system-node-critical.
2. If using a self-managed cluster, add priorityClassName: system-node-critical to the kube-system pods' manifests.
3. If evicted, the pods should be rescheduled quickly once node pressure subsides. If not, manually delete the pods to force recreation.

Failure 4: Disk Pressure from Image Pulls

Symptom: Nodes show DiskPressure condition True even though application data is small. Evictions may occur during new deployments.

Cause: imagefs.available threshold set too high or nodefs.available considers image layers. Container images fill up the filesystem.

Recovery:

1. Clean up unused images regularly using crictl rmi --prune or a cron job.
2. Increase the imagefs.available threshold to allow for temporary image pulls.
3. Use a dedicated filesystem for container images if possible (e.g., separate /var/lib/containerd partition).

Example cleanup command:

sudo crictl rmi --prune

Common Pitfalls and How to Avoid Them

The following pitfalls recur in real clusters. Each is described with why it happens, how to avoid it, and how to recover if you hit it.

Pitfall 1: Ignoring Node Capacity Differences

Why it happens: Teams copy the same eviction config to all nodes, assuming uniform resource size. But nodes in a heterogeneous cluster may have different memory or disk sizes. A 10% disk threshold on a 50GB node is 5GB, but on a 200GB node it is 20GB. The behavior differs.

How to avoid: Set thresholds as percentages for disk and absolute values for memory, or tailor absolute memory thresholds to each node group. Use taints or node pools to group nodes with similar capacity.

Recovery: If evictions happen only on certain nodes, check their capacity and adjust the threshold accordingly.

Pitfall 2: Overly Aggressive Soft Eviction Grace Periods

Why it happens: A short grace period (e.g., 10 seconds) gives pods almost no time to finish in-flight requests or clean up. This leads to abrupt terminations and client errors.

How to avoid: Set grace periods of at least 1-2 minutes for non-critical workloads. For stateful workloads that need longer cleanup, consider preStop hooks and longer grace periods.

Recovery: Lengthen the grace period and restart kubelet.

Pitfall 3: Not Monitoring Eviction Events

Why it happens: Eviction events are ephemeral and easy to miss. Without monitoring, you only notice when services fail.

How to avoid: Set up alerts on node conditions MemoryPressure, DiskPressure, PIDPressure, and on pod eviction events. Use tools like Prometheus with kube-state-metrics and node exporter.

Example Prometheus alert rule:

groups:
- name: node-pressure
  rules:
  - alert: NodeUnderMemoryPressure
    expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Node {{ $labels.node }} is under memory pressure"

Recovery: Once alerts are in place, you can react before evictions cascade.

Pitfall 4: Mixing Up evictionHard and evictionSoft Semantics

Why it happens: Operators set both hard and soft thresholds for the same signal without understanding that hard thresholds override soft. If evictionHard is lower than evictionSoft, the hard threshold will trigger first and evict immediately, making the soft threshold irrelevant.

How to avoid: Use hard thresholds as the last resort and soft thresholds as the early warning with grace. Ensure soft threshold is lower (i.e., triggers later) than hard threshold. For example, soft at 500Mi available, hard at 200Mi available.

Recovery: Fix the ordering and restart kubelet.

Pitfall 5: Not Considering Pod Disruption Budgets (PDBs)

Why it happens: Even with eviction thresholds correctly set, voluntary disruptions from evictions can violate PDBs, causing the Eviction API to block evictions or the kubelet to kill pods regardless, leading to unexpected downtime.

How to avoid: Define PDBs for critical applications and understand that node-pressure evictions are involuntary and ignore PDBs. Use priority classes to protect critical pods from eviction.

Recovery: If evictions caused downtime, check PDBs and priority classes. Adjust them to better protect workloads.

Operations Checklist

Use this checklist before and after making eviction configuration changes. Each item includes a concrete command or verification step, the responsible role, and how often to review.

Each owner is an individual, not a group. The review frequency ensures the configuration remains appropriate as workloads evolve.

Conclusion

Node pressure eviction is a safety net, but only if you configure it thoughtfully. The mistakes discussed here—overly aggressive thresholds, ignoring node capacity differences, weak monitoring, and misunderstanding hard vs soft eviction—are all avoidable with careful planning and verification. By inventorying your environment, making small changes, verifying with real signals, and having a rollback plan, you can prevent eviction-related outages.

Start with one low-risk action: check your current kubelet eviction config on one node and compare it against the node's actual memory and disk usage. If the thresholds seem too aggressive or too lax, adjust them using the safe path described above, and set up a simple alert for node pressure conditions. Then expand to all nodes gradually, documenting each step.

A robust eviction strategy is not static. Revisit thresholds quarterly or whenever you change node sizes, add new workload types, or observe new pressure patterns. With the right practices, you can keep your cluster stable and your services running even under resource strain.