Kubernetes Node Pressure Eviction: Common Errors and Practical Fixes

Intro

When a Kubernetes node runs low on memory, disk, or inodes, the kubelet begins evicting pods to reclaim resources and keep the node stable. This automatic safeguard is called node pressure eviction. It protects the cluster but often surprises teams with pods in Evicted state, sudden restarts, or cascading failures.

This article is a practical guide for developers, DevOps engineers, and platform teams who need to move from an unexpected eviction to a verified fix. You will learn how eviction works, how to identify which resource pressure caused it, how to set sensible thresholds, and how to design workloads that avoid eviction altogether. Every recommendation includes real commands, expected output, and recovery checks so you can apply it safely in your own cluster.

By the end, you will be able to answer these questions:

• Why was my pod evicted even though the node looks healthy now?
• What do the kubelet eviction signals memory.available, nodefs.available, and nodefs.inodesFree actually mean?
• How do I tune eviction thresholds without making the node unstable?
• What resource requests and limits prevent eviction for my critical services?
• How do I get alerted before pressure evictions start happening?

How Node Pressure Eviction Works

The Kubelet's Decision Loop

The kubelet on every worker node monitors local resource usage at a configurable interval, by default 10 seconds. It compares usage against two sets of thresholds:

• Soft eviction thresholds: when exceeded, the kubelet waits a grace period before evicting pods. This gives the system time to recover on its own.
• Hard eviction thresholds: when exceeded, the kubelet evicts pods immediately, with no grace period.

If a resource remains above its soft threshold for the entire grace period, or crosses a hard threshold, the kubelet selects pods to evict. Selection is based on a priority order that considers pod QoS class, requested resources versus actual usage, and priority class.

Eviction is not deletion. An evicted pod is terminated and its containers stopped, but the Pod object remains in the namespace with a Failed phase and reason Evicted. You must clean up these pods manually or with a controller.

Resources Under Pressure

The kubelet can evict for three main local resources:

Each signal has a corresponding eviction-hard or eviction-soft flag in the kubelet configuration. For example, the default hard eviction threshold for memory is often set to memory.available<100Mi, meaning when available memory drops below 100Mi, the kubelet evicts pods.

Quality of Service Classes

Kubernetes assigns every pod a QoS class based on its container resource requests and limits:

• Guaranteed: every container has both requests and limits, and they are equal.
• Burstable: at least one container has a request or limit, but not all are equal.
• BestEffort: no resource requests or limits are set.

When evicting, the kubelet targets BestEffort pods first, then Burstable pods that are using more than their requests, then other pods. Guaranteed pods are evicted last. This means a pod with no memory request is an easy eviction victim under pressure.

Version and Environment Inventory

Before changing anything, confirm the Kubernetes version, kubelet configuration, and which node experienced eviction. Eviction thresholds and behavior change across versions, so checking the version matters.

Run the following read-only commands first:

# Cluster version
kubectl version --short

# List nodes and their conditions
kubectl get nodes -o wide

# Check kubelet config on a specific node (requires SSH access)
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/configz" | jq .

The configz endpoint returns the kubelet's effective configuration, including eviction thresholds. On managed clusters like GKE or EKS, you may need to use the cloud provider's CLI or console to view these settings.

Example output snippet from configz:

{
  "kubeletconfig": {
    "evictionHard": {
      "memory.available": "100Mi",
      "nodefs.available": "10%",
      "nodefs.inodesFree": "5%",
      "imagefs.available": "15%"
    },
    "evictionSoft": {},
    "evictionSoftGracePeriod": {},
    "evictionPressureTransitionPeriod": "5m0s"
  }
}

Note the thresholds and whether soft eviction is enabled. The evictionPressureTransitionPeriod is how long the node must stay below the threshold before the pressure condition clears.

Next, identify the pods that were evicted and their reason:

kubectl get pods --all-namespaces --field-selector=status.phase=Failed

For a specific pod, describe it to see the eviction message:

kubectl describe pod <pod-name> -n <namespace>

Look for an event like:

Warning  Evicted  ...  The node was low on resource: memory. Threshold quantity: 104857600, available: 94371840.

That message tells you exactly which resource triggered eviction and how far below the threshold it fell.

Safe Configuration Path: Setting Eviction Thresholds

Tuning eviction thresholds requires access to kubelet configuration, which is usually managed by your cluster provisioning tool (kubeadm, managed Kubernetes, etc.). The steps differ by cluster type.

For Self-Managed Clusters (kubeadm)

On each node, edit /var/lib/kubelet/config.yaml or the kubelet systemd unit flags. Look for evictionHard or evictionSoft and adjust to match your workload needs.

Example snippet:

evictionHard:
  memory.available: "200Mi"
  nodefs.available: "10%"
  nodefs.inodesFree: "5%"
  imagefs.available: "15%"
  pid.available: "100"
evictionSoft:
  memory.available: "300Mi"
  nodefs.available: "15%"
evictionSoftGracePeriod:
  memory.available: "1m30s"
  nodefs.available: "2m"
evictionPressureTransitionPeriod: "5m"

After editing, restart the kubelet:

sudo systemctl restart kubelet

Verify the new settings took effect:

kubectl get --raw "/api/v1/nodes/<node-name>/proxy/configz" | jq '.kubeletconfig.evictionHard'

For Managed Kubernetes (EKS, GKE, AKS)

Managed services hide kubelet flags. You often cannot set arbitrary eviction thresholds directly; instead, you manage eviction indirectly through node pool settings, taints, or autoscaling.

• On GKE, you can use node auto-provisioning and resource requests to influence eviction, but hard thresholds are fixed.
• On EKS, you can use a custom launch template with bootstrap arguments for self-managed node groups, but managed node groups do not allow changing kubelet flags.
• On AKS, you can use the --kubelet-config flag on the node pool, but the options are limited.

In all cases, the safer approach on managed clusters is to design pods with appropriate requests and limits and to add monitoring alerts rather than trying to change kubelet internals.

Rule of Thumb for Thresholds

Start with sensible defaults:

• Memory hard threshold: 100Mi to 200Mi. Going below 100Mi risks kubelet instability; above 500Mi wastes capacity for small nodes.
• Disk hard threshold: 10% of the filesystem, but on large disks 10% can be 100Gi, which is too high. Consider using absolute values like 5Gi for critical nodes.
• Soft thresholds: set about 1.5 to 2 times the hard threshold, with grace periods of 1-5 minutes.

Always test changes on one node first, then roll out gradually.

Verification and Diagnostics

After configuring thresholds or after an eviction event, you need to verify what happened and confirm the fix.

Inspect Node Conditions

Node conditions show if the node is under pressure. Run:

kubectl describe node <node-name> | grep -A 5 Conditions

Example output when memory pressure is active:

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   True    Thu, 25 May 2023 10:15:30 +0000   Thu, 25 May 2023 10:12:00 +0000   KubeletHasInsufficientMemory  kubelet has insufficient memory available
  DiskPressure     False   Thu, 25 May 2023 10:15:30 +0000   Thu, 24 May 2023 08:00:00 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 25 May 2023 10:15:30 +0000   Thu, 24 May 2023 08:00:00 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 25 May 2023 10:15:30 +0000   Thu, 25 May 2023 10:10:00 +0000   KubeletReady                 kubelet is posting ready status

The condition reason and message explain the immediate issue.

Analyse Evicted Pods

For each evicted pod, check events with:

kubectl get events --all-namespaces --field-selector reason=Evicted --sort-by='.lastTimestamp'

Output includes the threshold and available quantity, which helps you tune thresholds later.

Validate New Thresholds

After changing kubelet configuration and restarting the kubelet, confirm the new thresholds are active using configz as shown earlier. Then simulate pressure with a test workload to see if eviction behaves as expected.

Create a memory stress pod:

apiVersion: v1
kind: Pod
metadata:
  name: memory-stress
spec:
  containers:
  - name: stress
    image: polinux/stress
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "500M", "--vm-hang", "1"]
    resources:
      requests:
        memory: "100Mi"
      limits:
        memory: "600Mi"

Apply it:

kubectl apply -f memory-stress.yaml

Watch node conditions and pod status:

kubectl get nodes -w
kubectl get pods -w

You should see the node transition to MemoryPressure=True, then the stress pod (or other BestEffort pods) get evicted. Remember to delete the test pod afterward.

Common Errors and Their Fixes

Error 1: The node was low on resource: memory.

Full message: The node was low on resource: memory. Threshold quantity: 104857600, available: 94371840.

Why it happens: Available memory dropped below the hard threshold, usually because pods consumed more memory than their requests, or the node is oversized relative to its workload.

How to fix:

1. Check which pods were evicted and which pods are using the most memory:

kubectl top pods --all-namespaces --sort-by=memory

1. Set memory requests and limits on all pods, especially those in Burstable or BestEffort classes.
2. If legitimate spikes cause eviction, consider raising the hard threshold (e.g., to 200Mi) but ensure the node has enough total memory to absorb spikes.
3. Add node memory monitoring with alerts at 80% and 90% utilization.

Error 2: The node was low on resource: nodefs.

Full message: The node was low on resource: nodefs. Threshold quantity: 10737418240, available: 10485760000.

Why it happens: Disk space on the root filesystem (where container logs and ephemeral storage live) is nearly full. Common culprits are large container logs, image layers, or ephemeral volumes.

How to fix:

1. Find what is using disk space on the node:

ssh <node> 'df -h /var/lib/kubelet /var/lib/containerd /var/log'

1. Clean up unused images and stopped containers:

crictl rmi --prune

1. Set log rotation for containers via kubelet flags or container runtime config.
2. Increase disk size or move imagefs to a separate partition.

Error 3: Evicted pods accumulate and never get cleaned up.

Why it happens: Kubernetes does not garbage collect evicted pods automatically; a controller is needed.

How to fix: Deploy a cleanup job or use a tool like kubectl in a cron job:

kubectl get pods --all-namespaces --field-selector=status.phase=Failed -o json | jq -r '.items[] | select(.status.reason=="Evicted") | .metadata.namespace + " " + .metadata.name' | while read ns pod; do kubectl delete pod $pod -n $ns; done

Better: install a controller like descheduler or a simple cronjob to remove evicted pods periodically.

Failure Modes and Recovery

Node Becomes NotReady Under Extreme Pressure

If the kubelet cannot evict fast enough or the node runs out of critical resources, the node may stop reporting readiness and become NotReady. Workloads on other nodes should continue, but scheduling to this node stops.

Recovery steps:

1. Identify the node:

kubectl get nodes

1. Check node events and conditions:

kubectl describe node <node-name>

1. If the node is unresponsive, reboot it via the infrastructure provider.
2. Once it rejoins, inspect evicted pods and decide whether to reschedule or investigate root cause.

Soft Eviction Thresholds Not Firing

A common mistake is setting evictionSoft but forgetting to set evictionSoftGracePeriod for that signal. Without a grace period, the soft threshold is ignored.

Example broken config:

evictionSoft:
  memory.available: "300Mi"

Fixed config:

evictionSoft:
  memory.available: "300Mi"
evictionSoftGracePeriod:
  memory.available: "1m30s"

Verify by checking the effective kubelet config and simulating memory pressure.

Pods in Guaranteed QoS Class Still Evicted

Guaranteed pods are evicted last, but if all pods are Guaranteed and the node is still under pressure, even they will be evicted. This can happen if the sum of memory requests from all pods exceeds the node's allocatable memory.

Avoidance: Ensure total requests never exceed node capacity minus kubelet and system reservations. Use the Kubernetes scheduler's resource accounting to prevent overcommit.

Common Pitfalls and How to Avoid Them

Pitfall 1: Ignoring Requests and Limits

Why it happens: Teams rely on the scheduler without setting explicit requests, leaving pods as BestEffort.

Consequence: BestEffort pods are evicted first, causing sudden outages.

How to avoid: Set memory and CPU requests for all production workloads. Use a tool like Polaris or kube-score to flag missing requests.

Pitfall 2: Setting Excessive Hard Thresholds

Why it happens: Operators try to maximize node utilization by setting memory.available<50Mi.

Consequence: Node has almost no headroom for system daemons; kubelet or container runtime may crash.

How to avoid: Keep at least 100Mi of memory, 10% of disk, and 5% of inodes as hard thresholds. Monitor system daemon memory usage.

Pitfall 3: No Monitoring or Alerts for Node Pressure

Why it happens: Only reactive troubleshooting after evictions occur.

Consequence: Repeated evictions, degraded performance, and frustrated users.

How to avoid: Use Prometheus with node exporter to track node_memory_MemAvailable_bytes, node_filesystem_avail_bytes, and kubelet eviction metrics. Alert when resources approach soft thresholds.

Pitfall 4: Using Soft Thresholds Without Understanding Grace Periods

Why it happens: Copying a config snippet without reading docs.

Consequence: Soft thresholds act like hard thresholds if grace period is zero or omitted.

How to avoid: Always pair soft thresholds with explicit grace periods and test behavior.

Operations Checklist

Use this checklist when investigating or preventing eviction issues. Assign a single owner for each action and set a review frequency.

Each owner should document the changes, the verification result, and the next review date. The review frequency ensures the checks remain relevant as workloads evolve.

Conclusion

Node pressure eviction is a protective mechanism, but it becomes a problem when it surprises you. By understanding how the kubelet decides to evict pods, setting appropriate thresholds, and monitoring node resources, you can prevent most eviction-related incidents.

Start with observation: run the read-only commands to inventory your nodes and evicted pods. Then make one small change at a time, such as setting memory requests on a critical deployment, and verify the result. Track your thresholds and alerts, and review them regularly.

A reliable workflow makes failures visible before they cause downtime, protects sensitive values, limits changes to the intended resource, and defines recovery steps before an incident happens. With the practical examples in this article, you now have that workflow for Kubernetes node pressure eviction.