## Intro

Kubernetes schedules pods onto nodes based on resource requests and limits, but nodes have finite memory and disk. When a node's available resources fall below configured thresholds, the kubelet can evict pods to reclaim space and keep the node stable. This process is called node pressure eviction, and if it is not planned for, it can cause unexpected pod terminations, service degradation, and cascading failures in your cluster.

Effective capacity planning helps you size nodes appropriately, set safe eviction thresholds, and avoid these disruptions. In this guide, you will learn a step-by-step approach to estimate resource needs, tune kubelet eviction policies, monitor node health, test your configuration, and plan for growth. We will use a practical example of a web application cluster throughout to illustrate key concepts.

Node pressure eviction is one of several ways Kubernetes handles resource scarcity. Unlike the scheduler, which places pods based on requests, the kubelet actively monitors actual usage on each node. It tracks signals such as available memory, available disk space, and available inodes. When a signal crosses a threshold, the kubelet evicts pods in a specific order: first pods that exceed their requests, then pods with lower priority, and so on. By planning capacity and tuning thresholds, you can minimize evictions while still protecting node health.

This guide covers the full lifecycle: inventorying your environment, calculating safe thresholds, configuring the kubelet, verifying settings, simulating pressure to test behavior, understanding failure modes, and maintaining an ongoing operations checklist. We will also cover common pitfalls and how to avoid them, because a misconfigured eviction policy can be worse than none at all.

## Version and Environment Inventory

Before changing any eviction settings, you need a clear picture of your cluster. Eviction behavior has evolved across Kubernetes versions, so knowing your exact version is critical. For example, Kubernetes 1.23 introduced changes to soft eviction thresholds and grace periods. Newer versions may have different defaults or deprecations. Always check the release notes for your version.

Here is the environment we will use in this guide:

| Component | Version/Details |
|---|---|
| Kubernetes | v1.25.3 |
| Node OS | Ubuntu 22.04 LTS |
| Container runtime | containerd 1.6.9 |
| Kubelet | v1.25.3 |
| Cluster size | 3 control-plane, 5 worker nodes |
| Worker node specs | 8 vCPU, 32 GB RAM, 100 GB disk |

In addition to the platform versions, document the workloads running on the nodes. Our example cluster hosts a web application with the following components:

- Frontend pods: 3 replicas, each requesting 512 Mi memory and 0.5 CPU, limit 1 Gi memory and 1 CPU.
- Backend pods: 5 replicas, each requesting 1 Gi memory and 1 CPU, limit 2 Gi memory and 2 CPU.
- Database pods: 2 replicas (primary and replica), each requesting 4 Gi memory and 1 CPU, limit 8 Gi memory and 2 CPU.
- Cache pods: 2 replicas, each requesting 2 Gi memory and 0.5 CPU, limit 4 Gi memory and 1 CPU.

All pods have resource requests and limits set. Verify this with `kubectl describe pod <pod-name>` if unsure. Unset requests are a common cause of eviction: the scheduler cannot account for them, and the kubelet may evict pods that are using more than expected.

Check that monitoring is in place. For this guide, we assume the Kubernetes metrics-server is installed, which provides `kubectl top` commands. We also recommend kube-state-metrics for pod and node state metrics, and a time-series backend like Prometheus for historical analysis. These tools will be used later for verification and capacity planning.

Finally, locate the kubelet configuration file. On systemd-based nodes, it is typically at `/var/lib/kubelet/config.yaml`. On some distributions, it may be `/etc/kubernetes/kubelet.conf` or passed via command-line flags. Use `ps aux | grep kubelet` to see the running arguments. Document the path for each node, as we will edit it shortly.

### Prerequisites

- SSH access to all worker nodes with sudo privileges.
- Permission to modify kubelet configuration (usually root).
- metrics-server installed and functional.
- kubectl configured with cluster-admin or sufficient permissions.
- A staging or test node or cluster where you can safely simulate pressure before rolling out to production.

## Safe Configuration Path

Start with the default eviction settings and adjust gradually. The kubelet ships with default thresholds that are conservative for many environments, but they may not match your node size or workload pattern. The goal is to set thresholds that reflect actual node capacity while leaving enough headroom for normal fluctuations and spikes.

### Understanding Eviction Signals

Kubelet eviction uses two types of thresholds:

- **Hard eviction thresholds**: When a resource falls below the threshold, the kubelet immediately evicts pods. There is no grace period. This is a reactive measure to protect the node from imminent resource exhaustion.
- **Soft eviction thresholds**: When a resource falls below the threshold, the kubelet starts a timer. If the resource recovers above the threshold before the timer expires (the grace period), no eviction occurs. If the resource remains low for the entire grace period, the kubelet then evicts pods. Soft thresholds are preferable for production because they give workloads a chance to release resources naturally.

The kubelet monitors several signals, including:

- `memory.available`: Available memory in bytes (or a percentage of total memory, depending on configuration).
- `nodefs.available`: Available disk space on the node's main filesystem (usually the partition containing `/var/lib/kubelet`).
- `nodefs.inodesFree`: Available inodes on the main filesystem.
- `imagefs.available`: Available disk space for container images (if using a separate filesystem).
- `imagefs.inodesFree`: Available inodes for the image filesystem.

For our example, we focus on `memory.available` and `nodefs.available` because they are the most common triggers for eviction in production.

### Calculating Safe Thresholds

The first step is to determine node total memory and the memory that should be reserved for system daemons: the kubelet, container runtime, OS kernel, and other system services. A common practice is to reserve 20% of memory for system overhead, but this varies depending on node size. The formula is:

- System reserved = total memory x 0.20
- Memory available for pods = total memory - system reserved

For our 32 GB node:

- System reserved = 32 Gi x 0.20 = 6.4 Gi
- Available for pods = 32 Gi - 6.4 Gi = 25.6 Gi

But note: the kubelet does not automatically know about this reservation unless you configure it. The kubelet has flags like `--system-reserved` and `--kube-reserved` that tell it how much to subtract from the node's total capacity when making decisions. In Kubernetes, these are part of the kubelet configuration. We will set them as well.

Next, choose eviction thresholds. A common approach is to set a hard threshold for `memory.available` at 10% of total memory. For a 32 GB node, that is 3.2 GB. This means when available memory drops to 3.2 GB, the kubelet will immediately start evicting pods. Some operators prefer a higher threshold (e.g., 15-20%) to give more buffer, but that reduces the effective capacity for pods. Choose a threshold that balances safety and utilization.

For disk, set `nodefs.available` threshold at 10% of the disk size. On a 100 GB disk, that is 10 GB. If you have a separate image filesystem, set `imagefs.available` similarly. Inodes are also important; a threshold like `nodefs.inodesFree: "5%"` is often used, but the exact value depends on your workload's file creation rate.

In addition to hard thresholds, define soft thresholds with grace periods to allow graceful recovery. For example:

- Soft `memory.available`: 4 Gi with a grace period of 2 minutes.
- Soft `nodefs.available`: 15% with a grace period of 5 minutes.

These give the kubelet time to respond to short spikes without immediately evicting pods.

### Kubelet Configuration Example

The kubelet configuration is in YAML. Here is a snippet for `/var/lib/kubelet/config.yaml` on our example node:

```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
  memory.available: "3.2Gi"
  nodefs.available: "10%"
  nodefs.inodesFree: "5%"
evictionSoft:
  memory.available: "4Gi"
  nodefs.available: "15%"
evictionSoftGracePeriod:
  memory.available: "2m"
  nodefs.available: "5m"
evictionMaxPodGracePeriod: 60
systemReserved:
  memory: "4Gi"
  cpu: "500m"
kubeReserved:
  memory: "2Gi"
  cpu: "500m"
```

In this example:

- Hard eviction triggers at 3.2 GB available memory or 10% free disk or 5% free inodes.
- Soft eviction triggers at 4 GB available memory with a 2-minute grace, or 15% free disk with a 5-minute grace.
- `evictionMaxPodGracePeriod` is the maximum time the kubelet will wait for a pod to terminate gracefully during eviction (in seconds). 60 seconds is reasonable.
- System and kube reserved values tell the kubelet to subtract these amounts from the node's total capacity when calculating available resources for pods. This prevents the system daemons from being starved.

Adjust the numbers based on your node size and workload. The key is to ensure that the sum of system reserved + kube reserved + eviction threshold is less than total node resources, leaving enough for pods.

### Applying Kubelet Configuration

To apply the changes on a single node:

1. SSH to the worker node.
2. Back up the existing configuration: `sudo cp /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak`
3. Edit `/var/lib/kubelet/config.yaml` with your changes.
4. Restart kubelet: `sudo systemctl restart kubelet`
5. Check the service status: `sudo systemctl status kubelet`

Expected output: The status should show `active (running)`. If the service fails to start, check the logs with `journalctl -u kubelet -n 50` for errors. Common mistakes include YAML syntax errors or invalid threshold values (e.g., using `Gi` with integer-only fields).

For cluster-wide changes, avoid manually editing each node; instead, use a configuration management tool such as Ansible, Puppet, or a Kubernetes-native operator like kubeadm config. For kubeadm clusters, you can use `kubeadm upgrade node phase kubelet-config` to rotate configuration, but manual editing is acceptable for a single node test.

### Verification and Diagnostics

After applying the configuration, verify that the thresholds are active.

#### Check Effective Configuration

Use the kubelet's configz endpoint to see the effective configuration:

```bash
kubectl get --raw "/api/v1/nodes/<node-name>/proxy/configz" | jq .
```

Expected output includes the eviction thresholds you set. For example:

```json
{
  "kubeletconfig": {
    "evictionHard": {
      "memory.available": "3.2Gi",
      "nodefs.available": "10%",
      "nodefs.inodesFree": "5%"
    },
    "evictionSoft": {
      "memory.available": "4Gi",
      "nodefs.available": "15%"
    },
    "evictionSoftGracePeriod": {
      "memory.available": "2m",
      "nodefs.available": "5m"
    },
    "evictionMaxPodGracePeriod": 60
  }
}
```

If the output does not match, restart kubelet again and check for errors.

#### Monitor Node Conditions

The kubelet reports node conditions `MemoryPressure` and `DiskPressure`. Use `kubectl describe node <node-name>` to see them. Under normal conditions, they should be `False`. If they are `True`, the node is currently under pressure and evictions may be imminent.

#### Simulate Memory Pressure

To test eviction behavior, deploy a pod that consumes memory beyond its request. Create a file `memory-demo.yaml`:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: memory-demo
spec:
  containers:
  - name: memory-demo-ctr
    image: polinux/stress
    resources:
      limits:
        memory: "1Gi"
      requests:
        memory: "1Gi"
    command: ["stress"]
    args: ["--vm", "1", "--vm-bytes", "2G", "--vm-hang", "1"]
```

Apply it:

```bash
kubectl apply -f memory-demo.yaml
```

This pod requests 1 Gi but attempts to use 2 Gi. If the node's available memory is below 2 Gi at the time, the pod will be evicted when `memory.available` falls below the hard threshold (3.2 Gi in our example). Watch for eviction:

```bash
kubectl get pods --watch
```

The pod status will change to `Evicted` or `Terminating`. Check events:

```bash
kubectl describe pod memory-demo
```

Expected output includes something like: `The node was low on resource: memory. Threshold quantity: 3.2Gi, available: 2.8Gi.`

If the pod is not evicted, increase the memory stress or lower the threshold temporarily on a test node. Be careful: this test can disrupt other pods on the node. Always run it on a dedicated test node or a staging cluster.

#### Verify Grace Periods

For soft eviction, the pod may not be evicted immediately. After the grace period (2 minutes in our example), if memory remains low, the kubelet will evict the pod. You can observe the timing by noting the event timestamps. If the pod recovers above the threshold during grace, it will not be evicted.

To test this, you could create a pod that temporarily spikes memory and then releases it before the grace period expires. The kubelet should not evict it.

## Failure Modes and Recovery

Eviction misconfiguration can lead to excessive pod evictions or node instability. Here are common failure modes:

1. **Thresholds too high (e.g., memory.available at 50%)**: Evictions occur frequently under normal load, causing pods to be rescheduled constantly and degrading application performance. This happens when operators misunderstand the meaning of the threshold or try to be overly cautious. To avoid, calculate the node's real capacity and set thresholds based on that, not on fear.

2. **Thresholds too low (e.g., memory.available at 1%)**: The node may run out of memory completely before eviction triggers, leading to the kernel OOM killer terminating processes unpredictably, possibly including critical system services. This is more dangerous than high thresholds. To recover, increase thresholds immediately and investigate why available memory dropped so low.

3. **Insufficient system reserved**: If you do not reserve memory for system daemons, the kubelet or container runtime may be killed when memory is exhausted, causing the node to become `NotReady`. Always set `systemReserved` and `kubeReserved` appropriately.

4. **Misconfigured grace periods**: Too short a grace period evicts pods before they can gracefully shut down, causing connection resets. Too long a grace period allows the node to exhaust resources before eviction, leading to OOM kills. Choose grace periods that match your application's shutdown time (usually 30-120 seconds).

5. **Ignoring inode pressure**: Many operators focus only on memory and disk space, but inode exhaustion can cause `DiskPressure` and evictions even when disk space is fine. Monitor inode usage and set `nodefs.inodesFree` thresholds.

6. **Not accounting for daemonset pods**: DaemonSet pods like logging agents or monitoring agents consume resources on every node. If they are not accounted for in capacity planning, they can trigger evictions. Ensure DaemonSet pods have requests set and include them in your calculations.

### Rollback

If you need to revert kubelet configuration, restore the backup file and restart kubelet:

```bash
sudo cp /var/lib/kubelet/config.yaml.bak /var/lib/kubelet/config.yaml
sudo systemctl restart kubelet
```

Verify the node returns to `Ready` and conditions are `False`:

```bash
kubectl describe node <node-name> | grep -A5 Conditions
```

Because kubelet config is local to each node, always keep backups and use version control for configuration files if possible.

### Recovery Checks

After an eviction event, ensure pods are rescheduled and healthy:

```bash
kubectl get pods -o wide
```

Check node conditions and resource usage:

```bash
kubectl describe node <node-name> | grep -A5 Conditions
kubectl top nodes
kubectl top pods
```

If evictions persist, investigate pod resource requests vs. actual usage. Use `kubectl describe pod` to see requests and limits, and compare with `kubectl top pod`. Adjust requests if they are too low, or add more nodes to increase cluster capacity.

## Common Pitfalls and How to Avoid Them

Beyond failure modes, several subtle pitfalls can undermine your eviction planning:

- **Mixing units**: The kubelet accepts both absolute values (e.g., "2Gi") and percentages (e.g., "10%") for thresholds, but not all signals support both. `memory.available` can be either; disk signals often use percentages. Using the wrong unit can cause the kubelet to reject the configuration or behave unexpectedly. Always check the Kubernetes documentation for your version.

- **Not setting pod disruption budgets**: Evictions are considered voluntary disruptions. If you have a PodDisruptionBudget (PDB) for your deployment, the eviction API will respect it, but the kubelet's pressure eviction does not. It will evict pods even if it violates a PDB. To minimize impact, ensure your application can tolerate losing a pod, or over-provision resources.

- **Forgetting to update node capacity when scaling**: If you add larger nodes to your cluster, the old thresholds may be too conservative or too aggressive. Re-evaluate thresholds for each node size.

- **Assuming eviction order is predictable**: The kubelet ranks pods by priority, then by usage above requests. But this order can change with QoS classes and priority classes. Do not rely on a specific pod being evicted first; instead, design all pods to be resilient.

- **Neglecting to test in staging**: Always test eviction behavior in a staging environment before production. Simulate memory and disk pressure to see how your workloads react. This is the only way to validate your thresholds.

## Operations Checklist

Maintain a checklist for ongoing capacity planning and eviction management. Assign an owner for each item and set a frequency.

| Task | Frequency | Owner | Command/Tool |
|---|---|---|---|
| Check node resource usage | Daily | On-call SRE | `kubectl top nodes` |
| Check node conditions (MemoryPressure, DiskPressure) | Daily | On-call SRE | `kubectl describe nodes` |
| Review eviction events | Weekly | Platform Engineer | `kubectl get events --field-selector reason=Evicted` |
| Verify kubelet configuration consistency across nodes | Monthly | Platform Engineer | Compare node configs with desired state (e.g., using a script or ConfigMap) |
| Plan capacity based on growth trends | Monthly | Capacity Planner | Analyze historical metrics from Prometheus |
| Test eviction behavior in staging | Before any changes | QA Lead | Simulate memory/disk pressure with stress tools |
| Audit system reserved settings | Quarterly | Infrastructure Architect | Review kubelet reserved flags and actual OS usage |

In addition, set up monitoring alerts to notify when node memory or disk usage exceeds 80% of the eviction threshold. For example, if the memory eviction threshold is 3.2 Gi, alert when available memory drops below 4 Gi (or better, below 20% of total). Use Prometheus alert rules or a similar system. Dashboards should show historical trends for memory and disk usage per node, as well as eviction counts.

## Conclusion

Effective capacity planning for Kubernetes node pressure eviction requires understanding the eviction mechanisms, setting appropriate thresholds, and continuously monitoring node health. In this guide, you learned how to inventory your environment, calculate safe eviction thresholds, configure the kubelet, verify settings, simulate pressure to test behavior, handle failure modes, and maintain an operations checklist.

Start with a narrow pilot on a single node: choose one worker node, adjust thresholds, test eviction behavior with stress pods, and observe for a few days before rolling out to the entire cluster. This minimizes risk and allows you to tune thresholds based on real workload patterns.

Remember that capacity planning is not a one-time task. As your cluster grows and workloads change, revisit thresholds and capacity regularly. Use the checklist and monitoring alerts to stay ahead of pressure situations. By following these practices, you can minimize unexpected evictions and keep your applications running smoothly, even under heavy load.