Kubernetes Node Capacity Planning: A Practical Guide with Examples

Intro

Kubernetes node capacity planning is the process of determining the right size and number of nodes to run your workloads reliably and cost-effectively. Without a plan, you risk either overprovisioning (wasting money) or underprovisioning (causing pod evictions, scheduling failures, and downtime). This guide provides a practical, step-by-step approach to planning, sizing, and managing Kubernetes node capacity, with concrete commands, examples, and recovery strategies.

This article is intended for developers, DevOps consultants, and technical startup teams who operate Kubernetes clusters. We cover the key concepts of node scaling, resource requests and limits, node sizing, and autoscaling. Every recommendation is version-scoped, observable, and reversible where possible.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery paths when things go wrong.

Version and Environment Inventory

Before making any capacity decisions, you need a clear picture of your cluster's version, node pool composition, and the tools available. This inventory reduces risk and ensures that every subsequent step is compatible with your environment.

Identify Cluster and Node Versions

Run the following read-only commands to gather version information:

kubectl version --short
kubectl get nodes -o wide

Example output:

Client Version: v1.29.2
Server Version: v1.28.5
NAME                 STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node-pool-a-7f3d    Ready    <none>          30d   v1.28.5   10.0.1.10     <none>        Ubuntu 22.04.3 LTS   5.15.0-105-generic   containerd://1.7.13
node-pool-b-8a2e    Ready    <none>          30d   v1.28.5   10.0.1.11     <none>        Ubuntu 22.04.3 LTS   5.15.0-105-generic   containerd://1.7.13

Note the container runtime and OS image, as these affect resource overhead. For managed clusters (EKS, GKE, AKS), use the cloud provider's CLI or console to verify node pool configurations.

Check Available Capacity

Use kubectl describe nodes to see allocatable resources and current usage:

kubectl describe node <node-name> | grep -A 5 "Allocated resources"

Example:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                1250m (62%)   3400m (170%)
  memory             2.5Gi (50%)   6.8Gi (136%)
  ephemeral-storage  0 (0%)        0 (0%)
  hugepages-1Gi      0 (0%)        0 (0%)

This shows that CPU requests are at 62% of allocatable, but limits are at 170%, meaning the node is overcommitted. Overcommitment can lead to contention and throttling, but is sometimes acceptable for bursty workloads.

Prerequisites for Capacity Planning

Access to a Kubernetes cluster (version 1.21+ recommended for stable autoscaling APIs).
kubectl configured with appropriate RBAC permissions to list nodes, pods, and events.
Metrics Server installed for kubectl top commands (if not, install via kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml).
Cluster Autoscaler or cloud provider autoscaling enabled if you plan to scale nodes automatically.

Small Test Before Big Changes

When experimenting with capacity settings, apply changes to a single pod or namespace first. For example:

kubectl apply -f test-pod.yaml --dry-run=client
kubectl apply -f test-pod.yaml
kubectl get pod test-pod -o wide
kubectl logs test-pod

Verify the pod schedules and runs as expected before rolling out changes to all workloads.

Quick check 1 of 2

What does the Kubernetes scheduler check before placing a pod on a node?

The scheduler ensures that for each resource type, the sum of the resource requests of the scheduled containers is less than the capacity of the node.

Safe Configuration Path

The safe configuration path involves setting resource requests and limits appropriately on your pods. This is the foundation of capacity planning because the scheduler uses requests to decide where to place pods, and the kubelet enforces limits to prevent resource starvation.

Understanding Requests and Limits

Requests: The amount of CPU or memory guaranteed to a container. The scheduler sums requests to determine if a node has enough capacity.
Limits: The maximum amount a container can use. If exceeded, CPU is throttled, and memory may cause OOMKill.

Example Pod Specification

Below is a deployment manifest with realistic resource settings for a web application:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: webapp
  template:
    metadata:
      labels:
        app: webapp
    spec:
      containers:
      - name: web
        image: nginx:1.25
        resources:
          requests:
            cpu: "250m"
            memory: "256Mi"
          limits:
            cpu: "500m"
            memory: "512Mi"

Apply and verify:

kubectl apply -f webapp.yaml
kubectl get pods -l app=webapp -o wide
kubectl top pods -l app=webapp

Expected output shows pods scheduled and CPU/memory usage within limits.

Calculating Node Capacity

Assume a node with 4 vCPU and 16 GiB memory. The allocatable capacity is typically slightly less due to system overhead (e.g., 3.9 vCPU, 14.9 GiB).

With the above pod requests (250m CPU, 256Mi memory), you can fit approximately:

CPU: 3.9 vCPU / 0.25 vCPU = 15.6, so 15 pods per node.
Memory: 14.9 GiB / 0.256 GiB = 58.2, but memory is usually the tighter constraint for other workloads.

However, you should never fill a node to 100% requests. Reserve headroom for system pods, daemonsets, and burst capacity. A common target is 70-80% of allocatable for requests.

Setting Resource Quotas and Limit Ranges

To prevent a single team or namespace from consuming all node capacity, define ResourceQuotas and LimitRanges.

Example LimitRange to enforce default requests and limits:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-a
spec:
  limits:
  - default:
      cpu: "500m"
      memory: "512Mi"
    defaultRequest:
      cpu: "250m"
      memory: "256Mi"
    type: Container

Example ResourceQuota:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"

Apply and verify:

kubectl apply -f limitrange.yaml
kubectl apply -f resourcequota.yaml
kubectl describe resourcequota -n team-a

Right-Sizing Workloads

Use kubectl top over time to see actual usage and adjust requests accordingly. A common practice is to set requests close to average usage and limits at peak or burst level.

Example:

kubectl top pods -n team-a

If a pod averages 120m CPU but requests 500m, you can reduce the request to save capacity. If it spikes to 900m, set the limit to 1000m.

Verification and Diagnostics

After configuring resources, you must verify that the cluster behaves as expected and diagnose any issues. This section covers key commands and their interpretation.

Scheduling Verification

Check that pods are scheduled and running:

kubectl get pods -o wide
kubectl describe pod <pod-name> | grep -A 10 Events

If a pod is pending, events may show:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  12s   default-scheduler  0/3 nodes are available: 3 Insufficient cpu.

This indicates that no node has enough CPU to satisfy the pod's request. You may need to add nodes or reduce the pod's request.

Resource Usage Monitoring

Without Metrics Server, use kubectl top to see current usage:

kubectl top nodes
kubectl top pods -A

Example node output:

NAME                 CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-pool-a-7f3d    920m         23%    7.8Gi           52%
node-pool-b-8a2e    1800m        45%    12.1Gi          81%

High memory usage on node-b suggests it may be near eviction threshold.

Logs and Events for Troubleshooting

For pods that crash or restart:

kubectl logs <pod-name> --previous
kubectl describe pod <pod-name> | grep -A 5 "Last State"

Look for OOMKilled in the termination reason:

Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137

If a pod is OOMKilled, increase its memory limit or reduce its memory usage.

Cluster Autoscaler Verification

If you have Cluster Autoscaler (CA) enabled, check its logs to see scaling decisions:

kubectl logs -n kube-system deployment/cluster-autoscaler

Look for lines like:

Fast evaluation: node node-pool-a-7f3d may be scale down candidate
Adding node pool node-pool-a to scale up: 1

CA only scales up when a pod is unschedulable due to insufficient resources. It scales down nodes that are underutilized for a period (default 10 minutes).

Simulating Capacity Exhaustion

To safely test your diagnostic skills, create a pod with a large request that cannot be satisfied:

apiVersion: v1
kind: Pod
metadata:
  name: huge-pod
spec:
  containers:
  - name: busybox
    image: busybox
    command: ["sleep", "3600"]
    resources:
      requests:
        cpu: "100"
        memory: "100Gi"

Apply and observe:

kubectl apply -f huge-pod.yaml
kubectl get pod huge-pod
kubectl describe pod huge-pod | grep -A 5 Events

You should see a FailedScheduling event. Delete the pod afterward.

Failure Modes and Recovery

Capacity planning must account for failures. This section describes common failure modes, how to detect them, and recovery actions.

Node Pressure and Evictions

When node memory or disk becomes critically low, the kubelet evicts pods. Eviction thresholds are configurable but have defaults. Monitor node conditions:

kubectl describe node <node-name> | grep -A 5 Conditions

Look for:

Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   True    Thu, 01 Jan 2026 12:00:00 +0000   Thu, 01 Jan 2026 11:55:00 +0000   KubeletHasInsufficientMemory   kubelet has insufficient memory available

Recovery: add nodes, reduce memory usage, or adjust eviction thresholds (with caution).

Out-of-Disk

Disk pressure can prevent new pods from starting and cause image garbage collection. Check disk usage:

kubectl describe node <node-name> | grep -A 5 "DiskPressure"

Recovery: clean up unused images (crictl rmi --prune on the node), add disk space, or move ephemeral storage to a larger volume.

Pod Pending Due to Insufficient Resources

As seen earlier, a pod may remain pending. First, check if the cluster autoscaler can add nodes. If not, you may need to manually scale the node pool.

For cloud providers, use CLI to increase node count or size:

# AWS EKS example
aws eks update-nodegroup-config --cluster-name my-cluster --nodegroup-name my-nodegroup --scaling-config minSize=2,maxSize=5,desiredSize=3

For on-premises, add a new node to the cluster.

Overcommitment and CPU Throttling

If pods are being throttled, you may see high CPU usage but low application performance. Check CPU throttling metrics (if using Prometheus) or inspect pod status:

kubectl top pod <pod-name>

If usage equals limit and application is slow, increase the CPU limit or reduce parallel load.

Node Not Ready

If a node becomes NotReady, pods may be rescheduled elsewhere. Check node status and events:

kubectl get nodes
kubectl describe node <node-name> | grep -A 5 Conditions

Recovery: investigate the node's underlying infrastructure (disk, network, kubelet). If it cannot be fixed, drain and delete the node:

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
kubectl delete node <node-name>

Operations Checklist

Use this checklist to run capacity planning as a repeatable process. Replace the examples with your own values.

Inventory current state

Command: kubectl get nodes -o wide and kubectl describe nodes | grep -A 5 "Allocated resources"
Expected: List of nodes with versions, roles, and resource allocation percentages.
Owner: Platform engineer (e.g., Alex Chen, DevOps Lead)

Define SLOs and capacity targets

Example: Ensure 95% of pods scheduled within 30 seconds. Keep node CPU requests under 70%, memory requests under 80%.
Owner: Product owner and platform lead (e.g., Maria Lopez, SRE Manager)

Set resource requests and limits on all workloads

Apply LimitRange and ResourceQuota per namespace.
Command: kubectl apply -f policies.yaml
Verify: kubectl describe limitrange -n team-a

Monitor usage trends

Use Prometheus/Grafana or kubectl top weekly. Look for steady growth or spikes.
Example alert: if node memory requests exceed 75% for 1 hour, trigger review.

Test autoscaling

Simulate load increase (e.g., using hey -z 5m -c 50 http://service).
Verify Cluster Autoscaler scales up and HPA increases replicas.
Command: kubectl get hpa and kubectl logs -n kube-system cluster-autoscaler

Plan for node pool changes

Decide whether to add nodes of the same size or switch to different instance types.
Use capacity planning tooling (e.g., Kubernetes OOMKilled analyzer, KubeCost) to estimate.

Document recovery runbooks

For each failure mode (pending pods, evictions, node failure), have a step-by-step recovery.
Store in a wiki or version-controlled repo.
Review quarterly.

Conclusion

Kubernetes node capacity planning is not a one-time task but an ongoing practice. By inventorying your environment, setting safe resource configurations, verifying with diagnostics, and preparing for failures, you can maintain a reliable and cost-efficient cluster.

Start with one low-risk verification: choose a single deployment, inspect its current resource usage, adjust requests and limits to match observed needs, and monitor for 24 hours. Then expand to other workloads and implement quotas and autoscaling.

Remember: every change should be version-scoped, observable, and reversible. Document what you expect to see and what to do if reality differs. With these practices, you can prevent capacity-related incidents before they impact your users.