## Intro Kubernetes Resource Quotas are a critical control for multi-tenant and shared clusters. They limit the total resource consumption of a namespace, preventing a single team or workload from starving others. When a quota is exceeded, pods fail to schedule, services may degrade, and troubleshooting can become confusing if you do not know where to look. This guide provides a practical, step-by-step approach to diagnosing and resolving Kubernetes Resource Quota issues. It is intended for developers, DevOps consultants, and technical startup teams who need to move from an observed problem to a verified result quickly and safely. We will cover version and environment inventory, safe configuration paths, verification and diagnostics, failure modes, recovery procedures, and an operations checklist. Each section includes concrete commands, expected outputs, and decision guidance. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before troubleshooting any Resource Quota issue, you must know exactly what you are working with. Start by identifying the Kubernetes version, the quota configuration, and the current usage in the affected namespace. This establishes a baseline and prevents accidental changes to the wrong object. ### Determine Cluster and Client Version Run: kubectl version --short Expected output (example): Client Version: v1.27.3 Kustomize Version: v5.0.1 Server Version: v1.27.2 If your client is significantly older than the server, some Resource Quota fields may not be supported. Update kubectl to match the server minor version when possible. ### Inspect Existing Resource Quotas List all ResourceQuota objects in the namespace: kubectl get resourcequota -n my-namespace Example output: NAME AGE REQUEST LIMIT compute-quota 5d requests.cpu: 2/4, requests.memory: 2Gi/8Gi, limits.cpu: 4/8, limits.memory: 4Gi/16Gi The REQUEST and LIMIT columns show current usage and hard limits. If usage is near or at the limit, you likely have a quota issue. For detailed view: kubectl describe resourcequota compute-quota -n my-namespace Example excerpt: Name: compute-quota Namespace: my-namespace Resource Used Hard -------- ---- ---- requests.cpu 2 4 requests.memory 2Gi 8Gi limits.cpu 4 8 limits.memory 4Gi 16Gi ### Check Namespace Labels and Annotations Resource Quotas can be scoped to specific priority classes or selectors. Verify namespace labels: kubectl get namespace my-namespace --show-labels Example: NAME STATUS AGE LABELS my-namespace Active 12d team=backend, environment=production ### Identify Pods and Their Resource Requests/Limits A common cause of quota exhaustion is a pod that requests more resources than expected. List pods with their resource requests and limits using a custom column: kubectl get pods -n my-namespace -o custom-columns='NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_LIM:.spec.containers[*].resources.limits.memory' Example output: NAME CPU_REQ MEM_REQ CPU_LIM MEM_LIM web-6c9d8f7b5d-abcde 250m 256Mi 500m 512Mi worker-7f8b9c6d5e-fghij 500m 512Mi 1 1Gi Compare these values with the quota usage. If a pod was recently created or updated, it may have pushed usage over the limit.
Quick check 1 of 2 What is the primary purpose of Kubernetes Resource Quotas? - To limit the number of nodes in a cluster - To limit the total resource consumption of a namespace - To restrict access to the Kubernetes API - To enforce network policies
Resource Quotas are used to manage resource usage of tenant workloads, limiting the total amount of resources a namespace can consume.
## Safe Configuration Path When you need to adjust quotas or resource requests, follow a safe configuration path. This minimizes disruption and allows for quick rollback if needed. ### Never Edit Live Manifests Directly Use version-controlled manifests. For example, a ResourceQuota definition might be: apiVersion: v1 kind: ResourceQuota metadata: name: compute-quota namespace: my-namespace spec: hard: requests.cpu: "4" requests.memory: 8Gi limits.cpu: "8" limits.memory: 16Gi Store this in Git and apply changes via a deployment pipeline or kubectl apply. ### Make Incremental Changes If you need to increase a quota, raise it by a small amount and observe. For example, to increase CPU requests from 4 to 5 cores, modify the manifest: requests.cpu: "5" Apply the change: kubectl apply -f resourcequota.yaml Verify: kubectl get resourcequota compute-quota -n my-namespace The REQUEST column should now show the new hard limit. ### Use kubectl patch for Quick Adjustments For a temporary change, you can patch directly: kubectl patch resourcequota compute-quota -n my-namespace --type='json' -p='[{"op": "replace", "path": "/spec/hard/requests.cpu", "value": "5"}]' Verify with: kubectl get resourcequota compute-quota -n my-namespace -o jsonpath='{.spec.hard.requests.cpu}' Expected output: 5 ### Adjusting Pod Resource Requests If the issue is a pod requesting too much, reduce its requests rather than increasing the quota. Edit the deployment: kubectl edit deployment web -n my-namespace Find the container spec and change: resources: requests: cpu: "250m" memory: "256Mi" Save and exit. The deployment will roll out a new pod with lower requests, freeing quota for other workloads. ## Verification and Diagnostics After making changes, verify that the issue is resolved and understand why it happened. This section covers diagnostic commands and interpretation of results. ### Check Recent Events Kubernetes events often contain quota-related messages. List events sorted by time: kubectl get events -n my-namespace --sort-by=.lastTimestamp Example of a quota exceeded event: LAST SEEN TYPE REASON OBJECT MESSAGE 2m Warning FailedCreate replicaset/web-... (combined from similar events): Error creating: pods "web-..." is forbidden: exceeded quota: compute-quota, requested: requests.cpu=250m,requests.memory=256Mi, used: requests.cpu=4000m,requests.memory=8192Mi, limited: requests.cpu=4,requests.memory=8Gi This clearly shows the quota name, the resource requested, current usage, and the hard limit. ### Check Pod Status and Descriptions When a pod cannot be created due to quota, it will not appear in get pods because it was never admitted. Instead, check the replicaset or deployment status: kubectl get deployment web -n my-namespace Output may show: NAME READY UP-TO-DATE AVAILABLE AGE web 2/3 3 2 1h Describe the deployment to see conditions: kubectl describe deployment web -n my-namespace Look for a condition like: ReplicaFailure True FailedCreate Then inspect the replicaset events as above. ### Use kubectl describe on ResourceQuota This gives a snapshot of usage vs hard limits: kubectl describe resourcequota compute-quota -n my-namespace If usage equals hard for a resource, that resource is saturated. You must either reduce requests or increase the quota. ### Check API Server Logs for Admission Denials If you have access to the Kubernetes API server logs, search for the namespace and quota name: grep "exceeded quota" /var/log/kubernetes/kube-apiserver.log | tail -20 This can provide additional context, especially if events have aged out.
Quick check 2 of 2 What does the `.status.allocatable` field on a Node describe? - The total capacity of the node - The amount of resources available to Pods on that node - The resource requests of all Pods on the node - The resource limits of all Pods on the node
The `.status.allocatable` field describes the amount of resources that are available to Pods on that node, accounting for system daemons.
## Failure Modes and Recovery Understanding common failure modes helps you recover faster. Here are scenarios and recommended recovery steps. ### Failure Mode 1: New Pods Fail to Schedule Symptom: Deployment shows fewer replicas than desired, and events show exceeded quota . Diagnosis: Quota for one or more resources is exhausted. Recovery: - Identify the exhausted resource from events or describe quota. - If possible, reduce the resource requests of the new pod or other pods in the namespace to free up quota. - Alternatively, increase the quota hard limit for that resource, but only after confirming cluster capacity. - Verify the deployment scales up: kubectl rollout status deployment/web -n my-namespace ### Failure Mode 2: Existing Pods Evicted or Terminated Symptom: Pods are being killed, and the node shows pressure. Diagnosis: This is usually not a ResourceQuota issue but a node-level resource pressure. However, if a quota is set for requests.storage and a persistent volume claim exceeds it, new PVCs will be rejected. Recovery: - Check node conditions: kubectl describe node | grep -A5 Conditions - If storage quota exceeded, delete unused PVCs or increase the quota. - If node pressure, cordon and drain the node, or add capacity. ### Failure Mode 3: Quota Not Enforced Symptom: Resource usage exceeds quota limits without rejection. Diagnosis: The ResourceQuota object might be misconfigured or not applied to the namespace. Check that the namespace has the correct labels if using quota selectors. Recovery: - Verify quota exists: kubectl get resourcequota -n my-namespace - Check if quota has spec.scopeSelector or spec.scopes that exclude certain pods. - Ensure the quota is not in a different namespace. - Reapply the quota manifest if missing. ### Failure Mode 4: Pods with Priority Classes Symptom: High-priority pods are evicted or cannot be created. Diagnosis: Quotas can be scoped to priority classes. If a quota has scopeSelector matching a high-priority class, it may be limited. Recovery: - Review the quota: kubectl get resourcequota compute-quota -n my-namespace -o yaml - If necessary, adjust the quota or pod priority. ### Recovery Verification After any recovery action, always verify: kubectl get resourcequota -n my-namespace kubectl get pods -n my-namespace kubectl get events -n my-namespace --sort-by=.lastTimestamp | tail -10 Ensure usage is within limits and no new quota errors appear. ## Operations Checklist Use this checklist to ensure a systematic approach to Resource Quota troubleshooting and recovery.
StepActionCommand / VerificationExpected Result
1Identify cluster versionkubectl version --shortClient and server versions compatible
2List quotas in namespacekubectl get resourcequota -n my-namespaceQuota object exists with hard limits
3Check current usagekubectl describe resourcequota compute-quota -n my-namespaceUsed values less than or equal to hard limits
4Inspect recent eventskubectl get events -n my-namespace --sort-by=.lastTimestampNo recent exceeded quota warnings
5Review pod resource requestskubectl get pods -n my-namespace -o custom-columns=...Sum of requests below quota hard limit
6If quota exhausted, identify which resourceFrom describe or eventsSpecific resource (cpu, memory, storage) identified
7Decide: reduce requests or increase quotaAnalyze workload requirementsMinimal change that resolves issue
8Apply change via version control or patchkubectl apply -f manifest.yaml or kubectl patch ...Change accepted without errors
9Verify quota updatedkubectl get resourcequota -n my-namespaceNew hard limit reflected
10Monitor deployment rolloutkubectl rollout status deployment/web -n my-namespaceDeployment successfully rolled out
11Confirm no new quota eventskubectl get events -n my-namespace --sort-by=.lastTimestampNo exceeded quota events in last 5 minutes
12Document change and rollback planUpdate runbook or incident notesRecovery steps recorded
## Conclusion Kubernetes Resource Quota troubleshooting requires a methodical approach: understand your environment, make minimal changes, verify thoroughly, and be prepared to recover. By following the steps in this guide, you can quickly diagnose quota-related issues and restore service without causing additional disruption. Always remember to observe before changing, protect sensitive values, and document your actions. With the right commands and a systematic checklist, Resource Quota problems become manageable rather than mysterious. As a next step, run the first three diagnostic commands in your own cluster to establish a baseline, and practice adjusting a quota in a non-production namespace.