## Intro

Node pressure eviction is Kubernetes' way of reclaiming node resources when they run low. When a node's memory, disk, or inodes cross configured thresholds, the kubelet starts terminating pods to prevent the node from becoming unstable. A misconfiguration here can evict critical workloads, cause cascading failures, or leave the cluster silently degraded. This guide walks through the common mistakes operators make when configuring eviction thresholds and how to fix them with practical, verifiable steps.

We will cover how to inspect your current eviction setup, set safe thresholds, validate changes, diagnose eviction events, and roll back when things go wrong. Every recommendation includes commands, expected output, or example scenarios so you can follow along in your own cluster. The goal is operational safety: observe first, change one thing at a time, and verify before moving on.

## Version and Environment Inventory

Before touching anything, know exactly what you are running. Node pressure eviction behavior depends on the Kubernetes version, the kubelet configuration, and the node's capacity. Start by collecting a read-only inventory.

Check the cluster and kubelet version:

```bash
kubectl version --short
kubectl get nodes -o wide
```

Example output:

```
Client Version: v1.29.1
Server Version: v1.29.1
NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node-1     Ready    control-plane   10d   v1.29.1   10.0.0.10     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11
node-2     Ready    <none>          10d   v1.29.1   10.0.0.11     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.7.11
```

To see the current eviction configuration, check the kubelet's startup flags or its config file. On a systemd-based node, run:

```bash
systemctl status kubelet
ps aux | grep kubelet
```

The kubelet may be using a config file specified by `--config`. Look for that path, then inspect it:

```bash
sudo cat /var/lib/kubelet/config.yaml
```

Example snippet from a default config:

```yaml
evictionHard:
  memory.available: "100Mi"
  nodefs.available: "10%"
  imagefs.available: "15%"
  nodefs.inodesFree: "5%"
evictionSoft:
  memory.available: "200Mi"
  nodefs.available: "20%"
evictionSoftGracePeriod:
  memory.available: "1m30s"
  nodefs.available: "1m30s"
evictionPressureTransitionPeriod: "5m"
```

If the config file is missing or no eviction section exists, the kubelet uses built-in defaults. Those defaults are:

- `evictionHard`: `memory.available<100Mi`, `nodefs.available<10%`, `imagefs.available<15%`, `nodefs.inodesFree<5%`
- `evictionSoft`: none
- `evictionPressureTransitionPeriod`: `5m`

Confirm the node's actual capacity with:

```bash
kubectl describe node node-1 | grep -A5 "Capacity:"
```

This gives you the allocatable memory, ephemeral storage, and inodes so you can reason about thresholds relative to real sizes.

Now you have a baseline. Write down the current thresholds, capacity, and version. If something goes wrong later, you can compare against this snapshot.

## Safe Configuration Path

Changing eviction thresholds is a node-level operation that affects scheduling and pod stability. Follow a careful, incremental path with a clear rollback plan.

### Step 1: Choose the Right Threshold Type

Kubernetes supports two eviction policies: hard and soft.

- **Hard eviction**: When a resource falls below the threshold, the kubelet immediately evicts pods. No grace period.
- **Soft eviction**: When a resource falls below the threshold, the kubelet waits for a grace period before evicting. If the resource recovers above the threshold during the grace period, no eviction occurs.

Mistake: Using soft eviction with a long grace period can allow a node to spiral into memory pressure before pods are evicted. Using hard eviction with a very high threshold can cause sudden eviction storms.

Recommendation: Start with hard eviction for memory because memory pressure is immediate and can crash the node. Use soft eviction for disk pressure, where temporary spikes are common.

### Step 2: Set Thresholds Based on Measured Baselines

Do not copy thresholds from a blog post. They must relate to your node size and workload profile. For a node with 16 GB RAM, a `memory.available` threshold of 100Mi is only 0.6% of total memory, which may be too low. A better starting point might be 500Mi or 5% of allocatable memory.

To measure current memory usage under load:

```bash
kubectl top node node-1
```

Example output:

```
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-1   1200m        30%    11Gi            68%
```

If the node routinely runs at 75% memory, setting `memory.available` to 10% of total gives a buffer before eviction. For a 16 GiB node, 10% is about 1.6 GiB. The threshold string would be:

```yaml
evictionHard:
  memory.available: "1.6Gi"
```

For disk, monitor `nodefs.available` and `nodefs.inodesFree`. Use `df -h` and `df -i` on the node:

```bash
df -h /
df -i /
```

Set thresholds that leave enough headroom for container logs and image pulls. A common mistake is setting `nodefs.available` too low, causing evictions when the container runtime needs space for new images.

### Step 3: Apply the Configuration One Node at a Time

Never change the kubelet config on all nodes simultaneously. If the threshold is too aggressive, you could evict pods across the whole cluster at once.

To change the kubelet config on a node:

1. Edit the kubelet config file (e.g., `/var/lib/kubelet/config.yaml`).
2. Restart the kubelet:

```bash
sudo systemctl restart kubelet
```

3. Check that the kubelet is running and the node is still Ready:

```bash
systemctl status kubelet
kubectl get node node-1
```

If the node becomes NotReady or pods start evicting unexpectedly, revert the file and restart kubelet again.

### Step 4: Use a DaemonSet or Managed Node Pool for Consistency

In managed Kubernetes (EKS, GKE, AKS), you often cannot edit the kubelet config directly. Instead, you modify the node pool's kubelet arguments or use a custom launch template. For example, on EKS with a self-managed node group, you can pass kubelet extra arguments in the user data script:

```bash
--eviction-hard=memory.available<500Mi,nodefs.available<10%
```

For GKE, you can configure node system configs via the `kubelet-config` in the node pool specification.

Always test on a single node pool or a canary node before rolling out to production.

## Verification and Diagnostics

After changing eviction settings, you need to verify that the kubelet picked up the new configuration and that eviction behaves as expected. You also need to diagnose any eviction events that were already happening.

### Check the Active Kubelet Configuration

You can confirm the active eviction thresholds by looking at the kubelet's logs during startup or by using the kubelet's `/configz` endpoint (if enabled). A simpler way is to check the node's status conditions:

```bash
kubectl describe node node-1 | grep -A10 "Conditions:"
```

Look for conditions like `MemoryPressure`, `DiskPressure`, `PIDPressure`. If any are `True`, the node is currently under pressure and evictions may be imminent.

You can also inspect kubelet logs for eviction-related messages:

```bash
journalctl -u kubelet --since "10 minutes ago" | grep -i evict
```

Example log line:

```
I0202 10:15:23.123456    1234 eviction_manager.go:340] "Eviction manager: attempting to reclaim" resourceName="memory.available"
```

This tells you the kubelet is actively trying to reclaim memory.

### Simulate Pressure Safely

To test your thresholds without waiting for a real crisis, you can deliberately create pressure on a node using a pod that consumes resources. For memory pressure:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: memory-eater
spec:
  containers:
  - name: memory-eater
    image: polinux/stress
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "2G", "--vm-hang", "0"]
    resources:
      requests:
        memory: "2G"
        cpu: "100m"
      limits:
        memory: "2G"
        cpu: "200m"
```

Schedule this pod on the target node using `nodeSelector` or `nodeName`. The pod requests 2G memory, which adds pressure. With a threshold of e.g. 500Mi available memory, you should see the kubelet start evicting pods. Watch the events:

```bash
kubectl get events --field-selector involvedObject.name=memory-eater --watch
```

If eviction does not happen when you expect it, check the actual thresholds in the kubelet config and the node's current memory usage. It may be that the node had enough headroom.

### Diagnose Eviction Reasons and Order

When a pod is evicted, its status reason will be `Evicted`. The event will include a message like:

```
The node was low on resource: memory. Threshold quantity: 500Mi, available: 420Mi.
```

Use these messages to understand which resource triggered the eviction and how far below the threshold the node was.

To see which pods were evicted recently:

```bash
kubectl get pods --all-namespaces --field-selector=status.phase=Failed -o wide
```

Then inspect each failed pod:

```bash
kubectl describe pod <evicted-pod> -n <namespace> | grep -A10 "Events:"
```

The event will show the eviction reason and timestamp.

Also check pod priority. The kubelet evicts pods in a specific order: BestEffort pods first, then Burstable pods that use more than their requests, then Guaranteed pods (with some caveats). In newer versions, pod priority and QoS class both matter. A common mistake is to have critical system pods with no priority class, making them candidates for eviction. Always set appropriate priority classes for critical workloads.

## Failure Modes and Recovery

Even with careful planning, eviction misconfigurations can cause serious problems. Here are the most common failure modes and how to recover from them.

### Failure 1: Eviction Storms

Symptom: Many pods evicted suddenly across one or more nodes, causing service disruption.

Cause: Hard eviction threshold set too high (e.g., `memory.available: 2Gi` on an 8Gi node) causing immediate evictions as soon as memory drops below 25%. Or soft eviction with a very short grace period.

Recovery:

1. Immediately increase the threshold or switch to soft eviction with a longer grace period.
2. Restart kubelet on affected nodes.
3. Monitor pod rescheduling and node pressure.

Example revised config:

```yaml
evictionSoft:
  memory.available: "500Mi"
evictionSoftGracePeriod:
  memory.available: "2m"
evictionHard:
  memory.available: "200Mi"
```

This gives pods a 2-minute grace period before hard eviction at 200Mi available.

### Failure 2: Node Becomes NotReady Due to Resource Exhaustion

Symptom: Node flips to NotReady, kubelet may crash, pods on the node become inaccessible.

Cause: Eviction thresholds set too low, so the node runs out of memory or disk before the kubelet evicts anything. The kernel may OOM-kill critical processes, including the kubelet itself.

Recovery:

1. Free up resources manually: delete unnecessary pods, clean up old container images, or restart the node.
2. Once node is Ready, revise thresholds upward to give more headroom.
3. Consider setting system reserved resources (`--system-reserved`, `--kube-reserved`) so the kubelet accounts for system overhead.

Example kubelet flags:

```bash
--system-reserved=cpu=500m,memory=1Gi
--kube-reserved=cpu=500m,memory=1Gi
```

### Failure 3: Critical System Pods Evicted

Symptom: CoreDNS, kube-proxy, or other system pods get evicted, breaking cluster networking or DNS.

Cause: No priority class set for system pods, so they are treated as BestEffort or Burstable and evicted early.

Recovery:

1. Ensure critical pods have high priority classes. For example, `system-cluster-critical` or `system-node-critical`.
2. If using a self-managed cluster, add `priorityClassName: system-node-critical` to the kube-system pods' manifests.
3. If evicted, the pods should be rescheduled quickly once node pressure subsides. If not, manually delete the pods to force recreation.

### Failure 4: Disk Pressure from Image Pulls

Symptom: Nodes show DiskPressure condition True even though application data is small. Evictions may occur during new deployments.

Cause: `imagefs.available` threshold set too high or `nodefs.available` considers image layers. Container images fill up the filesystem.

Recovery:

1. Clean up unused images regularly using `crictl rmi --prune` or a cron job.
2. Increase the `imagefs.available` threshold to allow for temporary image pulls.
3. Use a dedicated filesystem for container images if possible (e.g., separate `/var/lib/containerd` partition).

Example cleanup command:

```bash
sudo crictl rmi --prune
```

## Common Pitfalls and How to Avoid Them

The following pitfalls recur in real clusters. Each is described with why it happens, how to avoid it, and how to recover if you hit it.

### Pitfall 1: Ignoring Node Capacity Differences

Why it happens: Teams copy the same eviction config to all nodes, assuming uniform resource size. But nodes in a heterogeneous cluster may have different memory or disk sizes. A 10% disk threshold on a 50GB node is 5GB, but on a 200GB node it is 20GB. The behavior differs.

How to avoid: Set thresholds as percentages for disk and absolute values for memory, or tailor absolute memory thresholds to each node group. Use taints or node pools to group nodes with similar capacity.

Recovery: If evictions happen only on certain nodes, check their capacity and adjust the threshold accordingly.

### Pitfall 2: Overly Aggressive Soft Eviction Grace Periods

Why it happens: A short grace period (e.g., 10 seconds) gives pods almost no time to finish in-flight requests or clean up. This leads to abrupt terminations and client errors.

How to avoid: Set grace periods of at least 1-2 minutes for non-critical workloads. For stateful workloads that need longer cleanup, consider preStop hooks and longer grace periods.

Recovery: Lengthen the grace period and restart kubelet.

### Pitfall 3: Not Monitoring Eviction Events

Why it happens: Eviction events are ephemeral and easy to miss. Without monitoring, you only notice when services fail.

How to avoid: Set up alerts on node conditions `MemoryPressure`, `DiskPressure`, `PIDPressure`, and on pod eviction events. Use tools like Prometheus with kube-state-metrics and node exporter.

Example Prometheus alert rule:

```yaml
groups:
- name: node-pressure
  rules:
  - alert: NodeUnderMemoryPressure
    expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Node {{ $labels.node }} is under memory pressure"
```

Recovery: Once alerts are in place, you can react before evictions cascade.

### Pitfall 4: Mixing Up `evictionHard` and `evictionSoft` Semantics

Why it happens: Operators set both hard and soft thresholds for the same signal without understanding that hard thresholds override soft. If `evictionHard` is lower than `evictionSoft`, the hard threshold will trigger first and evict immediately, making the soft threshold irrelevant.

How to avoid: Use hard thresholds as the last resort and soft thresholds as the early warning with grace. Ensure soft threshold is lower (i.e., triggers later) than hard threshold. For example, soft at 500Mi available, hard at 200Mi available.

Recovery: Fix the ordering and restart kubelet.

### Pitfall 5: Not Considering Pod Disruption Budgets (PDBs)

Why it happens: Even with eviction thresholds correctly set, voluntary disruptions from evictions can violate PDBs, causing the Eviction API to block evictions or the kubelet to kill pods regardless, leading to unexpected downtime.

How to avoid: Define PDBs for critical applications and understand that node-pressure evictions are involuntary and ignore PDBs. Use priority classes to protect critical pods from eviction.

Recovery: If evictions caused downtime, check PDBs and priority classes. Adjust them to better protect workloads.

## Operations Checklist

Use this checklist before and after making eviction configuration changes. Each item includes a concrete command or verification step, the responsible role, and how often to review.

| # | Action | Command / Verification | Owner | Review Frequency |
|---|--------|-----------------------|-------|------------------|
| 1 | Record baseline node capacity and current eviction thresholds | `kubectl describe node <node> \| grep -A5 Capacity`, `cat /var/lib/kubelet/config.yaml` | Priya Shah, Platform Engineer | Upon any change, plus quarterly |
| 2 | Check current node pressure conditions | `kubectl describe node <node> \| grep -A5 Conditions` | Alex Chen, SRE | Weekly |
| 3 | Validate kubelet config syntax before restart | `kubelet --config=/var/lib/kubelet/config.yaml --validate` (if supported) or `kubelet --help` | Priya Shah | Before each change |
| 4 | Restart kubelet on one canary node | `sudo systemctl restart kubelet` | Jordan Lee, Node Operator | During change window |
| 5 | Confirm node returns Ready and no unexpected evictions | `kubectl get nodes`, `kubectl get events --all-namespaces \| grep Evicted` | Jordan Lee | Immediately after restart |
| 6 | Monitor eviction metrics for 24 hours | Prometheus: `kube_pod_evictions_total` or node exporter memory graphs | Maria Garcia, Monitoring Lead | After change, then daily for a week |
| 7 | Roll back if thresholds cause eviction storms or node instability | Restore previous config file, `sudo systemctl restart kubelet` | Jordan Lee | As needed, documented in runbook |
| 8 | Update documentation with new thresholds and rationale | Update runbook in wiki | Priya Shah | On every successful change |

Each owner is an individual, not a group. The review frequency ensures the configuration remains appropriate as workloads evolve.

## Conclusion

Node pressure eviction is a safety net, but only if you configure it thoughtfully. The mistakes discussed here—overly aggressive thresholds, ignoring node capacity differences, weak monitoring, and misunderstanding hard vs soft eviction—are all avoidable with careful planning and verification. By inventorying your environment, making small changes, verifying with real signals, and having a rollback plan, you can prevent eviction-related outages.

Start with one low-risk action: check your current kubelet eviction config on one node and compare it against the node's actual memory and disk usage. If the thresholds seem too aggressive or too lax, adjust them using the safe path described above, and set up a simple alert for node pressure conditions. Then expand to all nodes gradually, documenting each step.

A robust eviction strategy is not static. Revisit thresholds quarterly or whenever you change node sizes, add new workload types, or observe new pressure patterns. With the right practices, you can keep your cluster stable and your services running even under resource strain.