Kubernetes Resource Quota Troubleshooting: A Practical Field Guide

Intro

Kubernetes Resource Quotas are a critical control for multi-tenant and shared clusters. They limit the total resource consumption of a namespace, preventing a single team or workload from starving others. When a quota is exceeded, pods fail to schedule, services may degrade, and troubleshooting can become confusing if you do not know where to look.

This guide provides a practical, step-by-step approach to diagnosing and resolving Kubernetes Resource Quota issues. It is intended for developers, DevOps consultants, and technical startup teams who need to move from an observed problem to a verified result quickly and safely.

We will cover version and environment inventory, safe configuration paths, verification and diagnostics, failure modes, recovery procedures, and an operations checklist. Each section includes concrete commands, expected outputs, and decision guidance. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before troubleshooting any Resource Quota issue, you must know exactly what you are working with. Start by identifying the Kubernetes version, the quota configuration, and the current usage in the affected namespace. This establishes a baseline and prevents accidental changes to the wrong object.

Determine Cluster and Client Version

Run:

kubectl version --short

Expected output (example):

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.2

If your client is significantly older than the server, some Resource Quota fields may not be supported. Update kubectl to match the server minor version when possible.

Inspect Existing Resource Quotas

List all ResourceQuota objects in the namespace:

kubectl get resourcequota -n my-namespace

Example output:

NAME           AGE   REQUEST    LIMIT
compute-quota  5d    requests.cpu: 2/4, requests.memory: 2Gi/8Gi, limits.cpu: 4/8, limits.memory: 4Gi/16Gi

The REQUEST and LIMIT columns show current usage and hard limits. If usage is near or at the limit, you likely have a quota issue.

For detailed view:

kubectl describe resourcequota compute-quota -n my-namespace

Example excerpt:

Name:            compute-quota
Namespace:       my-namespace
Resource         Used    Hard
--------         ----    ----
requests.cpu     2       4
requests.memory  2Gi     8Gi
limits.cpu       4       8
limits.memory    4Gi     16Gi

Check Namespace Labels and Annotations

Resource Quotas can be scoped to specific priority classes or selectors. Verify namespace labels:

kubectl get namespace my-namespace --show-labels

Example:

NAME           STATUS   AGE   LABELS
my-namespace   Active   12d   team=backend, environment=production

Identify Pods and Their Resource Requests/Limits

A common cause of quota exhaustion is a pod that requests more resources than expected. List pods with their resource requests and limits using a custom column:

kubectl get pods -n my-namespace -o custom-columns='NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_LIM:.spec.containers[*].resources.limits.memory'

Example output:

NAME                    CPU_REQ   MEM_REQ   CPU_LIM   MEM_LIM
web-6c9d8f7b5d-abcde    250m      256Mi     500m      512Mi
worker-7f8b9c6d5e-fghij 500m      512Mi     1         1Gi

Compare these values with the quota usage. If a pod was recently created or updated, it may have pushed usage over the limit.

Safe Configuration Path

When you need to adjust quotas or resource requests, follow a safe configuration path. This minimizes disruption and allows for quick rollback if needed.

Never Edit Live Manifests Directly

Use version-controlled manifests. For example, a ResourceQuota definition might be:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: my-namespace
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi

Store this in Git and apply changes via a deployment pipeline or kubectl apply.

Make Incremental Changes

If you need to increase a quota, raise it by a small amount and observe. For example, to increase CPU requests from 4 to 5 cores, modify the manifest:

requests.cpu: "5"

Apply the change:

kubectl apply -f resourcequota.yaml

Verify:

kubectl get resourcequota compute-quota -n my-namespace

The REQUEST column should now show the new hard limit.

Use kubectl patch for Quick Adjustments

For a temporary change, you can patch directly:

kubectl patch resourcequota compute-quota -n my-namespace --type='json' -p='[{"op": "replace", "path": "/spec/hard/requests.cpu", "value": "5"}]'

Verify with:

kubectl get resourcequota compute-quota -n my-namespace -o jsonpath='{.spec.hard.requests.cpu}'

Expected output:

5

Adjusting Pod Resource Requests

If the issue is a pod requesting too much, reduce its requests rather than increasing the quota. Edit the deployment:

kubectl edit deployment web -n my-namespace

Find the container spec and change:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"

Save and exit. The deployment will roll out a new pod with lower requests, freeing quota for other workloads.

Verification and Diagnostics

After making changes, verify that the issue is resolved and understand why it happened. This section covers diagnostic commands and interpretation of results.

Check Recent Events

Kubernetes events often contain quota-related messages. List events sorted by time:

kubectl get events -n my-namespace --sort-by=.lastTimestamp

Example of a quota exceeded event:

LAST SEEN   TYPE      REASON             OBJECT                MESSAGE
2m          Warning   FailedCreate       replicaset/web-...    (combined from similar events): Error creating: pods "web-..." is forbidden: exceeded quota: compute-quota, requested: requests.cpu=250m,requests.memory=256Mi, used: requests.cpu=4000m,requests.memory=8192Mi, limited: requests.cpu=4,requests.memory=8Gi

This clearly shows the quota name, the resource requested, current usage, and the hard limit.

Check Pod Status and Descriptions

When a pod cannot be created due to quota, it will not appear in get pods because it was never admitted. Instead, check the replicaset or deployment status:

kubectl get deployment web -n my-namespace

Output may show:

NAME   READY   UP-TO-DATE   AVAILABLE   AGE
web    2/3     3            2           1h

Describe the deployment to see conditions:

kubectl describe deployment web -n my-namespace

Look for a condition like:

ReplicaFailure   True    FailedCreate

Then inspect the replicaset events as above.

Use kubectl describe on ResourceQuota

This gives a snapshot of usage vs hard limits:

kubectl describe resourcequota compute-quota -n my-namespace

If usage equals hard for a resource, that resource is saturated. You must either reduce requests or increase the quota.

Check API Server Logs for Admission Denials

If you have access to the Kubernetes API server logs, search for the namespace and quota name:

grep "exceeded quota" /var/log/kubernetes/kube-apiserver.log | tail -20

This can provide additional context, especially if events have aged out.

Failure Modes and Recovery

Understanding common failure modes helps you recover faster. Here are scenarios and recommended recovery steps.

Failure Mode 1: New Pods Fail to Schedule

Symptom: Deployment shows fewer replicas than desired, and events show exceeded quota.

Diagnosis: Quota for one or more resources is exhausted.

Recovery:

1. Identify the exhausted resource from events or describe quota.
2. If possible, reduce the resource requests of the new pod or other pods in the namespace to free up quota.
3. Alternatively, increase the quota hard limit for that resource, but only after confirming cluster capacity.
4. Verify the deployment scales up: kubectl rollout status deployment/web -n my-namespace

Failure Mode 2: Existing Pods Evicted or Terminated

Symptom: Pods are being killed, and the node shows pressure.

Diagnosis: This is usually not a ResourceQuota issue but a node-level resource pressure. However, if a quota is set for requests.storage and a persistent volume claim exceeds it, new PVCs will be rejected.

Recovery:

• Check node conditions: kubectl describe node <node-name> | grep -A5 Conditions
• If storage quota exceeded, delete unused PVCs or increase the quota.
• If node pressure, cordon and drain the node, or add capacity.

Failure Mode 3: Quota Not Enforced

Symptom: Resource usage exceeds quota limits without rejection.

Diagnosis: The ResourceQuota object might be misconfigured or not applied to the namespace. Check that the namespace has the correct labels if using quota selectors.

Recovery:

• Verify quota exists: kubectl get resourcequota -n my-namespace
• Check if quota has spec.scopeSelector or spec.scopes that exclude certain pods.
• Ensure the quota is not in a different namespace.
• Reapply the quota manifest if missing.

Failure Mode 4: Pods with Priority Classes

Symptom: High-priority pods are evicted or cannot be created.

Diagnosis: Quotas can be scoped to priority classes. If a quota has scopeSelector matching a high-priority class, it may be limited.

Recovery:

• Review the quota: kubectl get resourcequota compute-quota -n my-namespace -o yaml
• If necessary, adjust the quota or pod priority.

Recovery Verification

After any recovery action, always verify:

kubectl get resourcequota -n my-namespace
kubectl get pods -n my-namespace
kubectl get events -n my-namespace --sort-by=.lastTimestamp | tail -10

Ensure usage is within limits and no new quota errors appear.

Operations Checklist

Use this checklist to ensure a systematic approach to Resource Quota troubleshooting and recovery.

Conclusion

Kubernetes Resource Quota troubleshooting requires a methodical approach: understand your environment, make minimal changes, verify thoroughly, and be prepared to recover. By following the steps in this guide, you can quickly diagnose quota-related issues and restore service without causing additional disruption.

Always remember to observe before changing, protect sensitive values, and document your actions. With the right commands and a systematic checklist, Resource Quota problems become manageable rather than mysterious. As a next step, run the first three diagnostic commands in your own cluster to establish a baseline, and practice adjusting a quota in a non-production namespace.