## Intro Resource quotas in Kubernetes are critical for preventing a single namespace from consuming more than its fair share of cluster resources. They give platform teams a hard enforcement point for CPU, memory, storage, and object counts. Without them, a runaway deployment or a misconfigured autoscaler can starve other workloads and destabilize the entire cluster. This guide focuses on the operational commands you need to create, inspect, modify, and troubleshoot Kubernetes Resource Quotas. It is written for developers, DevOps consultants, and technical startup teams who manage multi-tenant clusters or shared environments. Every section includes concrete `kubectl` commands, example YAML manifests, expected output, and recovery steps for common failure modes. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. By the end, you will be able to enforce resource constraints confidently and diagnose quota-related issues in production. ## Version and Environment Inventory Before you run any quota command, confirm the Kubernetes environment version and the objects you are inspecting. Resource Quota behavior has evolved across releases, especially with the introduction of scopes and priority classes. Always capture the current state before making changes. ### Check Kubernetes Version Run the following to identify the cluster version and ensure compatibility with your quota definitions: ```bash kubectl version --short ``` Expected output looks similar to: ``` Client Version: v1.29.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.5 ``` If the client and server major versions differ by more than one minor version, upgrade your `kubectl` to avoid API compatibility problems. Resource Quota is available in all modern Kubernetes versions, but certain fields like `scopeSelector` and `matchScopes` require v1.24 or later. ### List All Resource Quotas in a Namespace To see existing quotas in a namespace, use: ```bash kubectl get resourcequota -n ``` For example, in a namespace called `team-a`, the output might be: ``` NAME AGE REQUEST LIMIT compute-quota 3d cpu: 2/10, memory: 1Gi/4Gi cpu: 4/10, memory: 2Gi/4Gi storage-quota 1d requests.storage: 100Gi/500Gi ``` The REQUEST column shows current usage and hard limit for request-type resources, while LIMIT shows the same for limit-type resources. This single command gives you a quick health check. ### Inspect Quota Details Use `kubectl describe` to get the full specification and current status of a quota: ```bash kubectl describe resourcequota compute-quota -n team-a ``` Expected output excerpt: ```yaml Name: compute-quota Namespace: team-a Resource Used Hard -------- ---- ---- cpu 2 10 memory 1Gi 4Gi pods 5 20 services 3 10 ``` If the quota does not appear, verify you are in the correct namespace and have appropriate RBAC permissions. The command `kubectl auth can-i list resourcequotas -n team-a` should return `yes`. ### Compare Quota Usage Across Namespaces For a multi-tenant overview, run: ```bash kubectl get resourcequota --all-namespaces ``` This lists all quotas cluster-wide, which is useful for identifying which teams are approaching their limits. You can pipe it through `grep` or `awk` to filter by resource type or utilization percentage. ## Safe Configuration Path Changing a Resource Quota can have immediate effects on running workloads. For example, lowering a hard limit below current usage will not terminate existing pods, but it will block new resource creation. Follow this safe path to minimize risk. ### Step 1: Export the Current Quota YAML Before modifying a quota, save a backup: ```bash kubectl get resourcequota compute-quota -n team-a -o yaml > compute-quota-backup.yaml ``` Inspect the backup file to understand the current spec. This backup is your recovery point if the new configuration causes problems. ### Step 2: Apply a Small, Well-Tested Change Instead of rewriting the entire quota, apply a minimal patch. For example, to increase the hard limit for CPU requests from 10 to 12, create a patch file or use a strategic merge patch: ```bash kubectl patch resourcequota compute-quota -n team-a --type merge -p '{"spec":{"hard":{"cpu":"12"}}}' ``` Expected output: ``` resourcequota/compute-quota patched ``` Always verify the patch with: ```bash kubectl describe resourcequota compute-quota -n team-a ``` Confirm the Hard column now shows `cpu: 12`. ### Step 3: Use a Dedicated GitOps Path for Production For production clusters, do not apply raw YAML changes with `kubectl patch`. Use a Git repository with a pull request flow. The following snippet shows a minimal Resource Quota manifest in Git: ```yaml apiVersion: v1 kind: ResourceQuota metadata: name: compute-quota namespace: team-a spec: hard: requests.cpu: "12" requests.memory: 4Gi limits.cpu: "20" limits.memory: 8Gi pods: "30" services: "10" ``` After merging the PR, a tool like Argo CD or Flux can sync the change. This approach provides an audit trail and easy rollback. ### Step 4: Verify That the Quota Is Being Enforced After a quota change, test that new resources are evaluated against the updated limits. For example, create a simple pod in the namespace: ```bash kubectl run test-pod --image=nginx -n team-a ``` If the quota is exceeded, the pod will not be created, and you will see an error like: ``` Error from server (Forbidden): pods "test-pod" is forbidden: exceeded quota: compute-quota, requested: cpu=1, used: cpu=12, limited: cpu=12 ``` If you need to delete the test pod, use `kubectl delete pod test-pod -n team-a`. ### Step 5: Monitor Quota Usage Over Time Set up a watch to observe quota changes as workloads scale: ```bash kubectl get resourcequota compute-quota -n team-a --watch ``` This updates the resource usage in real time. You can also integrate with Prometheus metrics like `kube_resourcequota` to build dashboards and alerts. ## Verification and Diagnostics Verification goes beyond checking the quota spec; it ensures that the quota is correctly applied to new workloads and that your namespace is healthy. Use the following commands and techniques. ### Verify a Quota Is Active Query the quota status with `kubectl get` and look at the REQUEST and LIMIT columns. A quota is active if it appears in the list and has non-zero hard values. ```bash kubectl get resourcequota compute-quota -n team-a ``` Output: ``` NAME AGE REQUEST LIMIT compute-quota 5d cpu: 3/12, memory: 1Gi/4Gi cpu: 5/20, memory: 2Gi/8Gi ``` If the REQUEST or LIMIT column is empty, the quota may not be tracking those resource types. Check the spec to confirm which resources are defined. ### Check Which Resource Is Blocking a Pod When a pod fails to schedule or create due to quota, the error message often includes the quota name and the exceeded resource. For more detail, describe the ReplicaSet or Deployment that is trying to create the pod: ```bash kubectl describe deployment my-app -n team-a ``` Look for events like: ``` Warning FailedCreate 5m replicaset-controller Error creating: pods "my-app-7b9f8b6c5-" is forbidden: exceeded quota: compute-quota, requested: memory=2Gi, used: memory=3.5Gi, limited: memory=4Gi ``` This tells you exactly which quota and resource is blocking. ### Use `kubectl describe quota` for Full Status The description output includes usage for all resources and, if scopes are used, the scope selector. Example: ```yaml Name: compute-quota Namespace: team-a Scopes: NotTerminating Resource Used Hard -------- ---- ---- cpu 3 12 memory 1Gi 4Gi pods 7 30 ``` If `Scopes` is listed, your quota only applies to pods with that scope. Misconfigured scopes are a common reason why a quota seems not to be enforced. ### Simulate a Quota-Exceeding Deployment To test your quota limits, deploy a workload that would exceed them. Create a file `stress-pod.yaml`: ```yaml apiVersion: v1 kind: Pod metadata: name: stress-pod namespace: team-a spec: containers: - name: stress image: polinux/stress resources: requests: memory: "10Gi" cpu: "5" limits: memory: "12Gi" cpu: "6" ``` Apply it and observe the rejection: ```bash kubectl apply -f stress-pod.yaml ``` Expected error: ``` Error from server (Forbidden): error when creating "stress-pod.yaml": pods "stress-pod" is forbidden: exceeded quota: compute-quota, requested: memory=10Gi, used: memory=1Gi, limited: memory=4Gi ``` This confirms the quota is enforced. ### Audit Quota Changes If you need to know when a quota was last modified, check the resource's metadata: ```bash kubectl get resourcequota compute-quota -n team-a -o jsonpath='{.metadata.managedFields[0].time}{"\n"}' ``` For a full audit trail, enable Kubernetes audit logging and filter for `resourcequotas`. ## Failure Modes and Recovery Resource Quota issues often manifest as failed pod creations, stuck deployments, or inability to create services. This section covers common failure modes and recovery steps. ### Failure: Pod Creation Fails with "Exceeded Quota" This is the most common failure. The error message will indicate the quota name and the resource that is over limit. To recover, either increase the quota (if justified) or reduce the resource requests of the pod. **Option 1: Increase the quota** ```bash kubectl patch resourcequota compute-quota -n team-a --type merge -p '{"spec":{"hard":{"memory":"6Gi"}}}' ``` **Option 2: Reduce the pod's memory request** Edit the deployment and change `resources.requests.memory` to a value that fits within the current quota. Then apply the change and the pod should create. ### Failure: Quota Does Not Seem to Be Enforced If pods are created despite exceeding the quota, check the following: - **Scopes**: A quota with `scopes: ["Terminating"]` only applies to pods with `activeDeadlineSeconds` set. A mismatch can cause the quota to be ignored for regular pods. - **Namespace**: Ensure the quota is in the same namespace as the pods. - **Resource Names**: The quota must specify the same resource type as the pod's resource field. For example, `requests.cpu` vs `cpu`. Use `kubectl describe resourcequota -n ` to review the spec. ### Failure: Cannot Delete a Namespace Because of Quota Sometimes a namespace gets stuck in `Terminating` state because a quota blocks finalizer cleanup. To resolve, first check if there are any remaining resources: ```bash kubectl get all -n stuck-ns ``` Delete any lingering pods or services. If the quota has finalizers, remove them after backing up: ```bash kubectl patch resourcequota -n stuck-ns -p '{"metadata":{"finalizers":[]}}' --type=merge ``` Then force delete the namespace: ```bash kubectl delete namespace stuck-ns --grace-period=0 --force ``` Be extremely cautious with force deletion; only do this after confirming no critical data remains. ### Failure: Quota Usage Not Updating If the usage numbers in `kubectl get resourcequota` do not change after deleting resources, it could be a delay in the controller or a bug. First, try waiting a few minutes. Then check the controller manager logs: ```bash kubectl logs -n kube-system kube-controller-manager- | grep -i quota ``` If the controller is healthy but usage is stale, you can force an update by restarting the controller (only if you have cluster admin rights). ### Recovery: Rollback a Quota Change If you applied a quota change that caused an outage, roll back using your backup YAML: ```bash kubectl apply -f compute-quota-backup.yaml ``` If you use GitOps, revert the commit in Git and let the controller sync. Always keep backups for at least 30 days. ## Operations Checklist Use this checklist before and after making Resource Quota changes in production. ### Pre-Change Checklist - Identify the namespace and quota name. - Record current usage with `kubectl describe resourcequota -n ` and save output to a file. - Back up the quota YAML with `kubectl get resourcequota -n -o yaml > backup.yaml`. - Determine the desired new limits based on projected workload growth. - Check RBAC permissions for the user making the change with `kubectl auth can-i update resourcequotas -n `. - Notify affected teams if the change might block their deployments. ### Change Execution Checklist - Apply the change via GitOps or `kubectl apply`/`kubectl patch`. - Immediately verify with `kubectl describe resourcequota -n ` and confirm the Hard column. - Test by creating a small pod (if appropriate) to ensure the quota behaves as expected. - Monitor `kubectl get resourcequota -n --watch` for a few minutes. ### Post-Change Validation Checklist - Check that existing workloads are still running: `kubectl get pods -n ` - Check for new quota violation events: `kubectl get events -n | grep quota` - Confirm that the quota usage is within acceptable bounds. - Update documentation and dashboards with new limits. - Ensure the backup is stored in a secure location. ### Monthly Audit Checklist - List all quotas across namespaces: `kubectl get resourcequota --all-namespaces` - Identify quotas that are too tight (usage > 80% of hard) or too loose (usage < 20% with no growth). - Review whether resource requests in deployments are properly set. - Check if any quotas are orphaned or not being used. - Plan adjustments based on capacity forecasts. ## Conclusion Kubernetes Resource Quotas are a powerful tool for maintaining fairness and stability in shared clusters. By mastering the commands in this guide, you can enforce resource limits proactively, diagnose quota-related failures quickly, and recover from mistakes safely. As a next step, choose one low-risk verification from this article, such as listing quotas in a test namespace or simulating an over-quota pod. Record the current state, run the documented commands, compare the results with the expected output, and practice the recovery process. Pay attention to related areas like Limit Ranges, namespace boundaries, and pod specifications, as they directly affect quota enforcement. A reliable operational workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With these practices, you can keep your Kubernetes clusters healthy and your development teams productive.