Kubernetes Resource Requests and Limits: A Practical Guide to Monitoring and Alerting

Intro

Kubernetes resource requests and limits are fundamental to cluster stability, scheduling, and cost control. Requests guarantee a minimum amount of CPU and memory for a container, while limits cap its maximum usage. Misconfigured requests and limits lead to pod evictions, node pressure, unpredictable performance, and wasted capacity. Monitoring these settings and alerting on deviations is essential for any team running production workloads.

This guide provides a practical, operations-focused approach to monitoring and alerting for Kubernetes resource requests and limits. It is intended for developers, DevOps consultants, and technical startup teams who need actionable steps, not just theory. We cover version and environment inventory, safe configuration paths, verification and diagnostics, failure modes and recovery, and an operations checklist. Each section includes concrete commands, expected outputs, failure signals, and recovery decisions.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery procedures before an incident forces your hand.

Version and Environment Inventory

Before monitoring or alerting on resource requests and limits, establish a clear inventory of your environment. This includes the Kubernetes version, deployment topology, and the components you are inspecting. Knowing your version is critical because APIs and metrics names may differ between releases.

Identify Kubernetes Version and Topology

Run the following read-only commands to capture the current state:

kubectl version --short
kubectl get nodes -o wide
kubectl get pods -A -o wide

Expected output for kubectl version --short (example):

Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.6

The server version is what matters for compatibility. The kubectl get nodes output shows node names, status, roles, age, version, and resource capacity. For example:

NAME           STATUS   ROLES           AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
node-1         Ready    control-plane   21d   v1.27.6   10.0.0.11     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.28
node-2         Ready    <none>          21d   v1.27.6   10.0.0.12     <none>        Ubuntu 22.04.3 LTS   5.15.0-91-generic   containerd://1.6.28

Record these outputs in a text file or a runbook. This establishes a baseline and helps detect changes after configuration modifications.

Check Existing Resource Requests and Limits

To see current requests and limits for all pods in a namespace, use:

kubectl get pods -n your-namespace -o custom-columns=NAME:.metadata.name,REQUESTS_CPU:.spec.containers[*].resources.requests.cpu,LIMITS_CPU:.spec.containers[*].resources.limits.cpu,REQUESTS_MEM:.spec.containers[*].resources.requests.memory,LIMITS_MEM:.spec.containers[*].resources.limits.memory

Example output:

NAME          REQUESTS_CPU   LIMITS_CPU   REQUESTS_MEM   LIMITS_MEM
web-app       100m           200m         128Mi          256Mi
db            500m           1            512Mi          1Gi

If any container lacks requests or limits, that is a signal for intervention.

Prerequisites for Monitoring

Ensure you have the following tools and permissions:

• Kubernetes cluster version 1.19 or later (for stable metrics APIs).
• kubectl configured with appropriate RBAC permissions to list pods, nodes, and metrics.
• Metrics Server installed (if you plan to use kubectl top).
• Prometheus (or another monitoring system) for long-term metrics and alerting.

To check if Metrics Server is running:

kubectl get deployment metrics-server -n kube-system

Expected output if installed:

NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           10d

If not installed, you can install it via the official manifests, but be aware it may not be suitable for production without customization.

Safe Configuration Path

Changing resource requests and limits can disrupt running workloads. Follow a safe configuration path that separates observation from intervention and validates changes in a controlled manner.

Observe Before Changing

Before modifying any manifest, inspect the current deployment and its pod spec.

Example: check the deployment web-app:

kubectl get deployment web-app -n prod -o yaml

Look for the resources section under each container. If absent, note it. Then describe the running pods to see current resource usage (if metrics-server is available):

kubectl top pod -n prod -l app=web-app

Example output:

NAME                      CPU(cores)   MEMORY(bytes)
web-app-7c9f5b6d-abcde   45m          89Mi
web-app-7c9f5b6d-fghij   32m          78Mi

This shows actual usage, which should inform your requests and limits. A common practice is to set requests equal to average usage plus some headroom (e.g., 20-30%), and limits based on peak usage.

Smallest Justified Change

Make one change at a time. For example, add a memory limit to a container that is missing one. Use kubectl set resources or edit the deployment:

kubectl set resources deployment web-app -n prod --limits=memory=256Mi --requests=memory=128Mi

This adds both requests and limits for memory. Before applying, save the current manifest:

kubectl get deployment web-app -n prod -o yaml > web-app-backup.yaml

Apply the change, then verify the rollout:

kubectl rollout status deployment/web-app -n prod

Expected output:

deployment "web-app" successfully rolled out

If the rollout fails, revert using the backup:

kubectl apply -f web-app-backup.yaml

Test in a Staging Environment

For significant changes, test in a staging namespace or cluster. Use a copy of the production deployment with modified resources. Then simulate load and observe performance.

Example: create a staging namespace staging and apply a test manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-test
  namespace: staging
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web-app
  template:
    metadata:
      labels:
        app: web-app
    spec:
      containers:
      - name: app
        image: nginx:1.25
        resources:
          requests:
            memory: "64Mi"
            cpu: "100m"
          limits:
            memory: "128Mi"
            cpu: "200m"

After applying, monitor with kubectl top pod and check for throttling or OOM kills.

Verification and Diagnostics

Verification ensures that resource requests and limits are correctly applied and that workloads are healthy. Diagnostics helps identify issues when they are not.

Verify Pod Scheduling and Resource Allocation

After updating resources, verify that pods are scheduled and running:

kubectl get pods -n prod -l app=web-app -o wide

Example output:

NAME                      READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
web-app-7d4f8c9b5-9x2kj   1/1     Running   0          2m    10.244.1.5   node-2   <none>           <none>

If a pod is pending, describe it to see events:

kubectl describe pod <pod-name> -n prod

Look for events like:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  30s   default-scheduler  0/2 nodes are available: 2 Insufficient memory.

This indicates your requests are too high for available nodes. Adjust accordingly.

Diagnose Container Crashes

If a container crashes due to OOM (out of memory), check the pod's status and logs:

kubectl get pods -n prod
kubectl logs <pod-name> -n prod --previous

Example OOM kill log snippet:

container <names> consumed 268435456 bytes of memory, which exceeds the limit of 134217728 bytes

This tells you the memory limit is too low. Increase it after analysis.

Use Metrics for Deeper Diagnosis

Prometheus is the standard for collecting Kubernetes metrics. Key metrics related to resource requests and limits include:

• kube_pod_container_resource_requests (with labels for resource and container)
• kube_pod_container_resource_limits
• container_memory_working_set_bytes (from cAdvisor)
• container_cpu_usage_seconds_total
• container_memory_usage_bytes

Example PromQL query to find pods with no memory requests:

sum by (namespace, pod) (kube_pod_container_resource_requests{resource="memory"} == 0)

This returns a list of pods lacking memory requests. You can similarly find pods nearing their limits:

sum by (namespace, pod) (container_memory_working_set_bytes) / sum by (namespace, pod) (kube_pod_container_resource_limits{resource="memory"}) > 0.9

This shows pods using more than 90% of their memory limit.

Failure Modes and Recovery

Understanding common failure modes helps you prepare automatic or manual recovery procedures.

Pod Eviction Due to Node Pressure

When a node runs out of memory, the kubelet evicts pods, starting with those that exceed their requests. If a pod is evicted, you will see its status as Failed or Evicted. Check with:

kubectl get pods -n prod --field-selector=status.phase=Failed

Example output:

NAME                      READY   STATUS    RESTARTS   AGE
web-app-7d4f8c9b5-9x2kj   0/1     Evicted    0          5m

To recover, you can manually delete the evicted pod (it will be recreated by the controller), but more importantly, adjust requests to better reflect actual usage or add node capacity.

Container OOM Kill

As mentioned, if a container exceeds its memory limit, it gets OOM-killed. The pod restarts (if restart policy allows). Repeated OOM kills can lead to CrashLoopBackOff. Monitor with:

kubectl get pods -n prod -l app=web-app

If RESTARTS is high, investigate. Recovery: increase the memory limit, optimize the application, or both.

Node Resource Exhaustion

Nodes can become unresponsive if system processes lack resources. Monitor node conditions:

kubectl describe node node-2 | grep -A5 Conditions

Look for MemoryPressure or DiskPressure conditions being True. If so, cordon the node, drain it, and investigate.

kubectl cordon node-2
kubectl drain node-2 --ignore-daemonsets --delete-emptydir-data

After resolving, uncordon:

kubectl uncordon node-2

Recovery Verification

After any recovery action, verify that the system returns to a healthy state. For example, after increasing a memory limit, watch the pod restarts drop to zero and stay there for at least 10 minutes.

Operations Checklist

Use this checklist to ensure continuous monitoring and alerting for resource requests and limits.

Monitoring Setup Checklist

• [ ] Metrics Server installed and running in kube-system namespace.
• [ ] Prometheus deployed with a scrape config for Kubernetes nodes, pods, and kube-state-metrics.
• [ ] kube-state-metrics installed to expose kube_pod_container_resource_requests and kube_pod_container_resource_limits metrics.
• [ ] Grafana (or equivalent) dashboard created with panels for:
• Pod CPU usage vs request and limit.
• Pod memory usage vs request and limit.
• Pods without requests or limits (count by namespace).
• Nodes under memory or CPU pressure.
• [ ] Alert rules configured for:
• Pod memory usage > 90% of limit for 5 minutes.
• Pod CPU throttling > 25% for 10 minutes.
• Pod missing requests or limits.
• Node memory pressure condition true.

Alerting Rule Examples

Here are sample Prometheus alert rules (in alerting_rules.yml):

groups:
- name: resource-requests-limits
  rules:
  - alert: PodMemoryNearLimit
    expr: |
      sum by (namespace, pod) (container_memory_working_set_bytes) /
      sum by (namespace, pod) (kube_pod_container_resource_limits{resource="memory"}) > 0.9
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is using more than 90% of its memory limit"

  - alert: PodCPUThrottlingHigh
    expr: |
      sum by (namespace, pod) (rate(container_cpu_cfs_throttled_seconds_total[5m])) /
      sum by (namespace, pod) (rate(container_cpu_usage_seconds_total[5m])) > 0.25
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is being CPU throttled significantly"

  - alert: PodMissingRequestsOrLimits
    expr: |
      (sum by (namespace, pod) (kube_pod_container_resource_requests) == 0) or
      (sum by (namespace, pod) (kube_pod_container_resource_limits) == 0)
    for: 15m
    labels:
      severity: info
    annotations:
      summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} has containers without requests or limits"

Incident Runbook Highlights

For each alert, document the following runbook steps (example for PodMemoryNearLimit):

1. Identify affected pod: kubectl get pods -n <namespace> -l app=<label>
2. Check current memory usage: kubectl top pod <pod-name> -n <namespace>
3. Review recent logs: kubectl logs <pod-name> -n <namespace> --tail=100
4. Check if OOM kills occurred: kubectl describe pod <pod-name> -n <namespace> | grep -i oom
5. Determine if memory limit is appropriate; if not, increase after approval via change management.
6. Monitor for 30 minutes to ensure no further alerts.

Regular Review Cadence

• Weekly: review dashboards for any pods nearing limits or missing requests.
• Monthly: review alert rules to adjust thresholds based on workload changes.
• Quarterly: audit all deployments for resource settings; use a policy engine like OPA Gatekeeper to enforce defaults.

Conclusion

Monitoring Kubernetes resource requests and limits is not a one-time task but an ongoing discipline. By establishing a version-aware inventory, making incremental changes safely, verifying with concrete diagnostics, preparing for failure modes, and adhering to an operations checklist, you can prevent resource-related incidents and maintain cluster health.

Every recommendation in this guide is designed to be version-scoped, observable, and reversible where possible. Do not copy commands blindly; understand prerequisites, expected outputs, and recovery paths.

As a next step, choose one low-risk verification: record the current resource settings for a single deployment, run a read-only inspection command like kubectl get deployment <name> -o yaml, compare the output against your baseline, and set up a simple Prometheus alert for missing requests. Then expand to broader monitoring and alerting coverage.

A reliable technical workflow makes failures visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces a decision. With these practices, you can confidently manage Kubernetes resource requests and limits in production.