Kubernetes Resource Requests and Limits: Advanced Concepts Explained with Practical Examples

Intro

Kubernetes resource requests and limits are critical for cluster stability and efficiency. Requests guarantee a minimum amount of CPU and memory for a container, while limits cap its maximum consumption. Misconfiguring these can lead to pod evictions, node overcommitment, or unexpected throttling.

This article goes beyond the basics and explores advanced concepts: how the scheduler uses requests, how the kubelet enforces limits, the three Quality of Service (QoS) classes, and how ResourceQuotas and LimitRanges govern resource usage at the namespace level. We provide practical examples, commands, and configuration snippets to help you diagnose and resolve resource-related issues in your cluster.

By the end, you will understand how to set requests and limits effectively, interpret QoS classes, and enforce organizational policies with quotas and limits.

Version and Environment Inventory

Before troubleshooting or modifying resource settings, establish a clear picture of your environment. This section covers the components involved and how to gather relevant information.

Key components:

• Kubernetes cluster (version 1.28 or later recommended; the examples here are version-agnostic but avoid deprecated APIs).
• kubectl configured with appropriate permissions.
• A test namespace where you can safely apply changes.

Observation commands:

Start by checking the cluster version and your current context:

kubectl version --short
kubectl config current-context

List existing pods with wide output to see node assignment and IPs:

kubectl get pods -o wide --namespace=your-namespace

Describe a specific pod to inspect its requests, limits, and events:

kubectl describe pod your-pod-name --namespace=your-namespace

Check recent events for resource-related warnings:

kubectl get events --namespace=your-namespace --sort-by=.lastTimestamp

Example output snippet from kubectl describe pod:

Containers:
  my-app:
    Limits:
      cpu:     500m
      memory:  256Mi
    Requests:
      cpu:     250m
      memory:  128Mi

If requests or limits are absent, note that the pod is running without resource guarantees, which may affect scheduling and QoS.

Safe Configuration Path

Changing resource requests and limits can disrupt running workloads. Follow this safe path to minimize risk.

Step 1: Understand Current Configuration

Inspect existing deployments to see how resources are currently set:

kubectl get deployment my-deployment -o yaml --namespace=your-namespace

Look for the resources field in each container spec. If it is missing, the container has no requests or limits.

Step 2: Start with Read-Only Changes

Never modify a live deployment directly. Instead, use kubectl edit only after backing up the manifest:

kubectl get deployment my-deployment -o yaml > my-deployment-backup.yaml

Alternatively, use a dry run to test changes without applying them:

kubectl set resources deployment my-deployment --requests=cpu=200m,memory=256Mi --limits=cpu=500m,memory=512Mi --local -o yaml

This prints the modified object without sending it to the API server.

Step 3: Apply Changes Gradualy

Apply the resource changes using a patch or a modified manifest. For example, patch the deployment:

kubectl patch deployment my-deployment --patch '{"spec":{"template":{"spec":{"containers":[{"name":"my-app","resources":{"requests":{"cpu":"200m","memory":"256Mi"},"limits":{"cpu":"500m","memory":"512Mi"}}}]}}}}'

Then watch the rollout status:

kubectl rollout status deployment/my-deployment --namespace=your-namespace

If the rollout fails, rollback immediately:

kubectl rollout undo deployment/my-deployment --namespace=your-namespace

Step 4: Verify Pod Resource Assignment

After rollout, check the new pods:

kubectl get pods -l app=my-app --namespace=your-namespace
kubectl describe pod <new-pod-name> --namespace=your-namespace | grep -A5 Resources

Confirm requests and limits are correctly set.

Verification and Diagnostics

Once resources are configured, verify that the scheduler and kubelet are enforcing them correctly.

Check Scheduling Decisions

The scheduler places pods based on requests. If a pod is pending, inspect why:

kubectl describe pod pending-pod --namespace=your-namespace

Look for events like Insufficient cpu or Insufficient memory.

Example event:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  10s   default-scheduler  0/3 nodes are available: 3 Insufficient cpu.

This indicates that the pod's CPU request exceeds available capacity on all nodes.

Inspect Node Allocatable Resources

Check node capacity and allocatable resources:

kubectl describe node <node-name> | grep -A5 "Allocatable"

Allocatable is the amount available for pods after system overhead. Compare total requests on the node with allocatable to see if you are overcommitting.

Use a tool like kubectl top to see current usage:

kubectl top nodes
kubectl top pods --namespace=your-namespace

Diagnose Throttling and OOMKilled

If CPU limits are set too low, the container will be throttled. Check cgroup CPU stats inside the pod (requires access to the node or metrics server):

kubectl exec -it my-pod -- cat /sys/fs/cgroup/cpu.stat

Look for nr_throttled and throttled_time values.

For memory limits, if a container exceeds its memory limit, the kernel OOM killer terminates it. The pod status will show OOMKilled:

kubectl get pod my-pod --namespace=your-namespace

Output:

NAME     READY   STATUS      RESTARTS   AGE
my-pod   0/1     OOMKilled   1          5m

Then inspect previous logs:

kubectl logs my-pod --previous --namespace=your-namespace

Failure Modes and Recovery

Resource misconfigurations lead to specific failure modes. Here are common scenarios and how to recover.

Failure: Pod Pending Due to Insufficient Resources

Symptom: Pod stays in Pending state.

Cause: Requests exceed available node resources.

Recovery:

• Scale down other workloads or add nodes.
• Reduce pod requests if they were set too high.

kubectl scale deployment my-deployment --replicas=0
# adjust requests
kubectl scale deployment my-deployment --replicas=1

Failure: Container OOMKilled

Symptom: Pod restarts repeatedly with OOMKilled status.

Cause: Memory usage exceeds limit.

Recovery:

• Increase memory limit if legitimate.
• Optimize application memory usage.
• Ensure limit is not below baseline usage.

kubectl set resources deployment my-deployment --limits=memory=1Gi --requests=memory=512Mi

Failure: CPU Throttling

Symptom: Application is slow, but CPU usage is below limit.

Cause: CPU limit is set too low relative to bursts.

Recovery:

• Increase CPU limit or remove limit to allow bursting (but consider node stability).

kubectl set resources deployment my-deployment --limits=cpu=2

Failure: Pod Eviction Due to Memory Pressure

Symptom: Pod is evicted even though it is within its memory limit.

Cause: Node memory pressure; pods without memory requests or with lower QoS are evicted first.

Recovery:

• Set nonzero memory requests to avoid being in BestEffort QoS.
• Ensure requests are realistic.

resources:
  requests:
    memory: "128Mi"
  limits:
    memory: "256Mi"

Operations Checklist

Use this checklist to operate resource requests and limits safely in production.

• [ ] Cluster version and kubectl access verified.
• [ ] Current resource settings documented for all critical workloads.
• [ ] Resource requests and limits set for every container in production.
• [ ] Requests reflect realistic average usage; limits reflect peak plus buffer.
• [ ] No pods in Pending state due to insufficient resources.
• [ ] No recent OOMKilled events (check kubectl get pods --all-namespaces | grep OOMKilled).
• [ ] CPU throttling monitored and within acceptable range.
• [ ] ResourceQuotas defined for namespaces with multiple teams.
• [ ] LimitRanges set to provide defaults and prevent overcommit.
• [ ] Node allocatable resources compared with total requests to avoid overcommit beyond policy.
• [ ] Rollback procedure tested for resource changes.

Example of a ResourceQuota definition:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "10"
    requests.memory: "20Gi"
    limits.cpu: "20"
    limits.memory: "40Gi"

Apply with:

kubectl apply -f resource-quota.yaml

Example LimitRange:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-a
spec:
  limits:
  - default:
      cpu: 500m
      memory: 512Mi
    defaultRequest:
      cpu: 200m
      memory: 256Mi
    type: Container

Apply with:

kubectl apply -f limit-range.yaml

Conclusion

Kubernetes resource requests and limits are powerful controls that, when used correctly, ensure efficient scheduling and prevent resource contention. Advanced concepts like QoS classes, ResourceQuotas, and LimitRanges enable fine-grained governance.

Always verify current state before making changes, apply changes gradually with rollback plans, and monitor for throttling and OOM kills. By following the practical steps and checklists in this article, you can operate your cluster with confidence and avoid common resource-related failures.

As a next step, audit one of your deployments for missing requests and limits, apply appropriate values using a dry run, and then observe the pod's scheduling and runtime behavior.