## Intro Kubernetes Resource Quotas are critical for preventing a single team or namespace from consuming excessive cluster resources. However, managing quotas manually across many namespaces is error-prone and slow. Automating quota management with CI/CD ensures consistent, auditable, and repeatable changes. This article provides a practical, hands-on guide to building a CI/CD pipeline for Resource Quotas, covering environment inventory, safe configuration, verification, failure recovery, and an operations checklist. We will use concrete commands, manifests, and examples so you can adapt the workflow to your own clusters. ## Version and Environment Inventory Before automating Resource Quota changes, you must understand your current environment. This section covers how to gather version, topology, and prerequisite information using read-only commands, then plan the smallest safe change. ### Prerequisites - Kubernetes cluster version 1.21+ (Resource Quota API stable since v1, but certain features like scope selectors may vary). Check with: kubectl version --short Expected output: Client Version: v1.27.0 Server Version: v1.27.0 - kubectl configured with appropriate RBAC permissions to get, list, and create ResourceQuota objects in target namespaces. - A Git repository for storing quota YAML manifests and pipeline definitions. - A CI/CD system (e.g., GitHub Actions, GitLab CI, Jenkins) with a service account that has limited cluster rights. ### Read-Only Observation Start by inspecting current Resource Quotas in a namespace without changing anything. For example, in namespace team-a : kubectl get resourcequota -n team-a If no quotas exist, you'll see: No resources found in team-a namespace. If quotas exist, output like: NAME AGE REQUEST LIMIT compute-quota 10d requests.cpu: 2/4, requests.memory: 2Gi/4Gi limits.cpu: 4/8, limits.memory: 4Gi/8Gi Get details of a specific quota: kubectl describe resourcequota compute-quota -n team-a Note the Used vs Hard values. This tells you how close you are to limits before any change. Also inspect namespace labels, as Resource Quotas can be scoped by namespace selector if using ResourceQuota with scopeSelector (more on that later). Check labels: kubectl get namespace team-a --show-labels ### Blast Radius and Smallest Change Changing a Resource Quota affects all pods in that namespace: new pods may be rejected if they exceed quota, but existing pods are not evicted automatically. The blast radius is namespace-wide. Therefore, the smallest justified change might be adding a new quota for a specific resource rather than modifying an existing one. For example, if you need to limit the number of persistent volume claims, create a separate quota object instead of altering the compute quota. Example of a minimal new quota YAML ( pvc-quota.yaml ): apiVersion: v1 kind: ResourceQuota metadata: name: pvc-quota namespace: team-a spec: hard: persistentvolumeclaims: "5" Apply this manually first in a test namespace to verify behavior, then automate. ### Verification Command After any change, verify the quota is active and correctly configured: kubectl get resourcequota pvc-quota -n team-a -o yaml Check the spec.hard and status.used fields. ## Safe Configuration Path This section details how to manage Resource Quota configurations safely within a CI/CD pipeline, ensuring that every change is reviewed, tested, and applied with a rollback plan. ### Repository Structure Organize your quota manifests in a Git repository with a clear directory structure. For example: quotas/ team-a/ compute-quota.yaml pvc-quota.yaml team-b/ compute-quota.yaml global/ default-quotas.yaml Each file should contain one or more ResourceQuota definitions with namespace specified in metadata. Alternatively, use Kustomize or Helm to manage namespace-specific overrides. ### Pipeline Stages A typical CI/CD pipeline for Resource Quotas has these stages: - Lint : Validate YAML syntax and Kubernetes schema. - Dry-run : Use kubectl apply --dry-run=client to check for API errors. - Apply to staging : Deploy to a staging cluster or namespace first. - Smoke test : Check that quotas are created and enforced as expected. - Approval : Require manual approval before production. - Apply to production : Deploy with --server-side optionally to avoid conflicts. - Record : Store applied manifests and outputs for audit. ### Example Pipeline (GitHub Actions) Create a workflow file .github/workflows/quota-deploy.yaml : name: Deploy Resource Quotas on: push: paths: - 'quotas/**' branches: - main workflow_dispatch: jobs: deploy: runs-on: ubuntu-latest environment: production steps: - uses: actions/checkout@v3 - name: Set up kubectl uses: azure/setup-kubectl@v3 with: version: 'v1.27.0' - name: Dry-run apply run: | set -e for file in $(find quotas -name '*.yaml'); do echo "Dry-run applying $file" kubectl apply -f $file --dry-run=client done - name: Apply to production run: | for file in $(find quotas -name '*.yaml'); do kubectl apply -f $file done In this example, the pipeline runs on every push to main affecting quota files. It first dry-runs all manifests, then applies them. You can add a manual approval step using GitHub Environments protection rules. ### Rollback Strategy Resource Quota changes are easy to rollback if you use Git tags or keep previous versions in the repository. To rollback, simply apply the previous manifest version: git revert kubectl apply -f quotas/team-a/compute-quota.yaml If the quota was deleted accidentally, reapply from Git. Because Resource Quota objects are declarative, applying the old manifest restores the desired state. ## Verification and Diagnostics After applying quota changes, you must verify not only that the quota object exists but also that it is enforced correctly. This section provides commands to diagnose issues. ### Verify Quota Creation Check that the quota is present: kubectl get resourcequota -n team-a Expected output shows the quota name and age. If missing, check the pipeline logs for errors. ### Verify Enforcement Create a test pod that exceeds the quota to see if it is rejected. For example, if you set requests.cpu: "4" in a quota, create a pod requesting 5 CPUs: # test-pod.yaml apiVersion: v1 kind: Pod metadata: name: test-overquota namespace: team-a spec: containers: - name: nginx image: nginx resources: requests: cpu: "5" Apply it: kubectl apply -f test-pod.yaml Expected error: Error from server (Forbidden): error when creating "test-pod.yaml": pods "test-overquota" is forbidden: exceeded quota: compute-quota, requested: requests.cpu=5, used: requests.cpu=2, limited: requests.cpu=4 If the pod is created, the quota is not enforced or is misconfigured. ### Diagnostic Commands - List events in the namespace to see quota-related errors: kubectl get events -n team-a --sort-by='.lastTimestamp' Look for FailedCreate or ExceededQuota messages. - Describe the quota for status: kubectl describe resourcequota compute-quota -n team-a - Check the API server logs if needed (requires cluster admin access). Quick check 2 of 2 According to the article, what is a common failure mode in Resource Quota CI/CD pipelines? - Cluster nodes running out of disk space - The pipeline applying to the wrong cluster - Resource quotas not being namespaced - Pod eviction on quota change

The article lists 'Pipeline applies to wrong cluster' as a common failure mode, explained as 'Kubeconfig context points to production when you intended staging.'

## Failure Modes and Recovery Even with automation, failures occur. This section covers common failure modes for Resource Quota CI/CD and how to recover. ### Common Failure Modes - Invalid manifest syntax : YAML indentation error or missing field. Pipeline should catch this in lint/dry-run stage. If not, apply fails with a parse error. Recovery: fix manifest and re-run pipeline. - RBAC insufficient : CI/CD service account lacks permission to create/update ResourceQuota in target namespace. Error: forbidden: User "system:serviceaccount:ci:deployer" cannot create resource "resourcequotas" . Recovery: grant appropriate RBAC role. - Namespace mismatch : Manifest specifies a namespace that does not exist. Error: namespaces "team-c" not found . Recovery: create namespace first or correct manifest. - Quota too restrictive : Developers cannot deploy pods, causing incident. Recovery: immediately increase quota by editing YAML and applying, or delete the quota temporarily (but with caution). - Pipeline applies to wrong cluster : Kubeconfig context points to production when you intended staging. Recovery: use separate contexts and environment protection in CI. ### Rollback Procedures For any failed change, rollback to the last known good state from Git: git log --oneline -- quotas/team-a/compute-quota.yaml # Identify previous commit hash git revert kubectl apply -f quotas/team-a/compute-quota.yaml If the pipeline itself is broken, manually apply the last good manifest: kubectl apply -f quotas/team-a/compute-quota.yaml ### Monitoring and Alerting Set up alerts for quota usage to proactively detect issues. For example, if using Prometheus, an alert rule: groups: - name: quota-alerts rules: - alert: QuotaAlmostFull expr: kube_resourcequota{resource="requests.cpu", type="used"} / kube_resourcequota{resource="requests.cpu", type="hard"} > 0.9 for: 5m labels: severity: warning annotations: summary: "Resource quota {{ $labels.resourcequota }} is over 90% used" ## Operations Checklist Use this checklist for every Resource Quota change through CI/CD to ensure safety and consistency.

Step	Action	Command / Tool	Expected Result
1	Validate YAML syntax	`kubectl apply --dry-run=client -f <file>`	No syntax errors
2	Check current quotas	`kubectl get resourcequota -n <namespace>`	List existing quotas
3	Compare desired vs current	`diff` between old and new manifest	Understand changes
4	Apply to staging namespace	`kubectl apply -f <file> -n staging`	Quota created/updated
5	Test enforcement in staging	Create pod exceeding quota	Pod rejected with Forbidden error
6	Approve for production	CI/CD manual gate	Approval granted
7	Apply to production	`kubectl apply -f <file>`	Quota updated
8	Verify production	`kubectl describe resourcequota <name> -n <namespace>`	Hard and used values correct
9	Record deployment	Git tag, CI log	Audit trail available
10	Monitor quota usage	Prometheus/ Grafana	No alerts firing

### Example Worked Scenario Let's walk through a concrete example: increasing the CPU request quota for team-a from 4 to 6 CPUs. - Current quota ( compute-quota.yaml ): apiVersion: v1 kind: ResourceQuota metadata: name: compute-quota namespace: team-a spec: hard: requests.cpu: "4" requests.memory: "4Gi" limits.cpu: "8" limits.memory: "8Gi" - Change requests.cpu to "6" in a feature branch. - Push branch , open PR. CI runs dry-run and lint. - Review and merge to main . Pipeline applies to production. - Verify : kubectl describe resourcequota compute-quota -n team-a Expect requests.cpu: 2/6 (if current usage is 2). - Test : deploy a pod requesting 5 CPUs, should succeed now. ## Conclusion Automating Kubernetes Resource Quota management with CI/CD brings reproducibility, auditability, and safety to namespace resource governance. By following the practices outlined in this article—inventorying your environment, designing a safe configuration path, verifying changes rigorously, preparing for failures, and using an operations checklist—you can prevent resource contention and misconfiguration. Start small: pick one namespace, define its quotas in Git, and create a simple pipeline with dry-run and apply stages. As you gain confidence, expand to all namespaces and add automated testing and alerting. Remember that the key is to observe before changing, limit the blast radius, and always have a rollback plan.