Kubernetes Resource Quota Monitoring and Alerts with Practical Examples

Intro

Kubernetes Resource Quotas are critical for controlling resource consumption in namespaces, but they only work if you can observe them. In this guide, you will learn what to monitor, which metrics matter, how to set up alerting with Prometheus, and how to build a dashboard in Grafana. We will also cover incident response workflows so you can act quickly when a quota is about to be exceeded.

Resource Quotas prevent a single team from exhausting cluster resources, but without monitoring, you may not notice until workloads are rejected. For example, a namespace hitting its CPU limit can cause pod scheduling failures. By monitoring quota usage and setting alerts, you can proactively manage capacity.

This article provides practical examples, including commands, configuration snippets, and expected outputs. You will also find a checklist to verify your setup and a recovery plan for common failure modes.

Version and Environment Inventory

Before you start, ensure you have a Kubernetes cluster with Resource Quotas enabled (they are enabled by default). The examples in this article assume:

• Kubernetes version 1.20 or later (API version v1 for ResourceQuota).
• Prometheus deployed in the cluster (e.g., via Prometheus Operator or Helm).
• Grafana for dashboards (version 7.0+).
• kubectl command-line tool configured.
• A namespace with an existing ResourceQuota or you will create one.

To check your Kubernetes version:

kubectl version --short

Expected output:

Client Version: v1.21.0
Server Version: v1.21.0

To verify that ResourceQuota is available:

kubectl api-resources | grep resourcequota

Expected output:

resourcequotas   quota   v1   true   ResourceQuota

If you see this, you are ready to proceed. Next, confirm that Prometheus is scraping metrics from kube-state-metrics, which exposes ResourceQuota metrics. If you do not have kube-state-metrics, install it (e.g., via Helm: helm install kube-state-metrics prometheus-community/kube-state-metrics).

Safe Configuration Path

Start with a single namespace to avoid noise. For example, create a namespace team-a and apply a ResourceQuota:

kubectl create namespace team-a
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-quota
  namespace: team-a
spec:
  hard:
    requests.cpu: "2"
    requests.memory: 4Gi
    limits.cpu: "4"
    limits.memory: 8Gi
    pods: "10"
EOF

Expected output:

resourcequota/compute-quota created

Now, check the quota status:

kubectl describe resourcequota compute-quota -n team-a

Expected output (abbreviated):

Name:            compute-quota
Namespace:       team-a
Resource         Used   Hard
--------         ----   ----
limits.cpu       0      4
limits.memory    0      8Gi
pods             0      10
requests.cpu     0      2
requests.memory  0      4Gi

Next, deploy a simple workload to consume resources. For example, a deployment with two replicas:

kubectl create deployment nginx --image=nginx --replicas=2 -n team-a

Check usage again:

kubectl describe resourcequota compute-quota -n team-a

Expected output shows pods used: 2.

Now, set up monitoring. Prometheus can scrape kube-state-metrics which exposes kube_resourcequota metrics. Key metrics include:

• kube_resourcequota{resource="pods", type="used"}
• kube_resourcequota{resource="pods", type="hard"}
• kube_resourcequota_created

A useful PromQL query to see usage percentage:

100 * (sum(kube_resourcequota{type="used", resource="pods"}) by (namespace, resource) / sum(kube_resourcequota{type="hard", resource="pods"}) by (namespace, resource))

This query returns the percentage of pod quota used per namespace. For alerts, define a Prometheus rule (e.g., in a ConfigMap or PrometheusRule):

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: resourcequota-alerts
  namespace: monitoring
spec:
  groups:
  - name: resourcequota.rules
    rules:
    - alert: ResourceQuotaAlmostFull
      expr: |
        100 * (kube_resourcequota{type="used"} / kube_resourcequota{type="hard"}) > 80
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Resource quota {{ $labels.resource }} in namespace {{ $labels.namespace }} is over 80% used"
        description: "The quota for {{ $labels.resource }} in namespace {{ $labels.namespace }} has exceeded 80% (current value: {{ $value }}%)"

Apply this rule and wait for Prometheus to evaluate it. This is the safe, scoped implementation: one namespace, one quota, simple alert threshold. Once verified, you can expand to more namespaces or resources.

Verification and Diagnostics

To verify that monitoring works, first check that Prometheus is scraping the metrics. Port-forward to Prometheus and run a query:

kubectl port-forward -n monitoring svc/prometheus-k8s 9090:9090

Then open http://localhost:9090 and enter the query:

kube_resourcequota{namespace="team-a", resource="pods"}

Expected result: a time series with labels type="hard" value 10 and type="used" value 2 (or current pod count).

You can also check via curl:

curl 'http://localhost:9090/api/v1/query?query=kube_resourcequota%7Bnamespace%3D%22team-a%22%2Cresource%3D%22pods%22%7D'

Expected JSON response (truncated):

{"status":"success","data":{"resultType":"vector","result":[{"metric":{"namespace":"team-a","resource":"pods","type":"hard"},"value":[1618000000,"10"]},{"metric":{"namespace":"team-a","resource":"pods","type":"used"},"value":[1618000000,"2"]}]}}

If you do not see metrics, check if kube-state-metrics is running and its service is discovered by Prometheus. Check the Prometheus targets page (http://localhost:9090/targets) for kube-state-metrics.

Next, verify the alert rule. In Prometheus UI, go to Alerts and look for ResourceQuotaAlmostFull. To trigger it, increase the used percentage by scaling the deployment beyond 80% of pod quota (8 pods). For example:

kubectl scale deployment nginx --replicas=8 -n team-a

Wait a few minutes (for: 5m) and check alerts. The alert should be in Pending or Firing state. You can also query the alert expression to see the current value.

For dashboards, create a Grafana panel with the PromQL query:

100 * (sum(kube_resourcequota{type="used"}) by (namespace, resource) / sum(kube_resourcequota{type="hard"}) by (namespace, resource))

Set the unit to percent and visualize as a table or bar gauge. This gives a quick overview of all quotas.

Failure Modes and Recovery

Common failure modes:

• Metrics missing: If kube-state-metrics is not running or not scraped, no alerts fire. Check Prometheus target health and kube-state-metrics logs. Fix by ensuring the service is up and the ServiceMonitor or scrape config is correct.
• False alerts due to division by zero: If a quota has hard value 0 (unlimited), the division in the alert expression may produce NaN or inf. Exclude such cases by adding kube_resourcequota{type="hard"} > 0 to the query.
• Alert storm: If many namespaces hit 80% simultaneously, you may get flooded. Use grouping and inhibition rules, or adjust thresholds.
• Quota misconfigured: A ResourceQuota with too low limits can break existing workloads. To rollback, delete the quota:

kubectl delete resourcequota compute-quota -n team-a

Or edit it:

kubectl edit resourcequota compute-quota -n team-a

Then increase the hard values. After rollback, verify the namespace returns to normal and workloads are running.

• Prometheus rule not firing: Check the rule evaluation. In Prometheus UI, go to Status > Rules and find the rule. Ensure the expression is valid and active. Check for syntax errors.

Recovery checklist:

• Identify the failing component (metrics, alerting, dashboard).
• Check logs: kubectl logs -n monitoring deployment/kube-state-metrics.
• Verify Prometheus targets.
• Adjust quota if necessary.
• Document the incident for future reference.

Common Pitfalls and How to Avoid Them

Beyond the failure modes above, several subtle issues trip up teams when monitoring ResourceQuotas. Understanding these pitfalls will save you hours of debugging.

1. Ignoring the difference between requests and limits. A quota may limit both requests.cpu and limits.cpu. Alerts that only look at one will miss overcommitment on the other. Always monitor both request and limit usage for compute resources. For example, if a namespace has a quota of 4 CPUs for limits but only 2 for requests, and a team sets high limits but low requests, the requests quota may be exhausted before the limits quota. Your alert rules should cover all resource types you care about, not just pods.

1. Using naive percentage thresholds for small quotas. For small quotas (e.g., 5 pods), an 80% threshold may fire when only one pod remains. This could be too late. Consider absolute thresholds in addition to percentages. For example, alert when used >= hard - 2 for pods, regardless of percentage. This gives earlier warning for small quotas.

1. Not accounting for multiple quotas in a namespace. Kubernetes allows multiple ResourceQuota objects in a namespace, and their effects are combined. If you have separate quotas for compute and object counts, a single alert rule that sums all quotas may produce misleading percentages. Instead, query per resource type across all quotas in the namespace. For instance, sum by (namespace, resource) (kube_resourcequota{type="used"}) will aggregate used across all quotas for the same resource in the namespace.

1. Overlooking the kube_resourcequota_created metric. This metric indicates when the quota was created and can help detect deleted or recreated quotas. If a quota is deleted and recreated, your alert rules may briefly miss it. Monitor for changes in this metric or use it in dashboards to show quota age.

1. Assuming kube-state-metrics is always up to date. kube-state-metrics may lag behind actual Kubernetes state, especially in large clusters. If you see discrepancies, check its scrape interval and resource consumption. In some cases, you may need to reduce its scrape interval or increase its resources.

1. Not testing alerts with realistic traffic. Simulate high usage in a staging cluster with the same quotas as production to ensure alerts fire as expected. This includes testing the notification channels and the runbook steps.

To avoid these pitfalls, we recommend a weekly review of alert rules and dashboard panels with at least one engineer from the platform team (for example, the Kubernetes platform lead, Priya Shah) as the accountable owner. The review should include checking for new namespaces, changed quotas, and false positives, with a specific agenda item to revisit thresholds based on recent usage trends.

Operations Checklist

Use this checklist for ongoing operations:

Example of a daily check command:

kubectl get resourcequota --all-namespaces

Expected output: list of quotas with used and hard values.

You can also create a script to alert on high usage.

Keep this checklist accessible to the team and update it as your environment evolves.

Conclusion

Monitoring Kubernetes Resource Quotas is essential for preventing resource exhaustion and ensuring fair usage. In this guide, you learned how to:

• Set up a ResourceQuota and consume resources.
• Configure Prometheus alerts for high quota usage.
• Verify monitoring with PromQL queries and expected outputs.
• Handle common failure modes and rollback changes.
• Use an operations checklist for ongoing management.

Next steps:

• Expand monitoring to all namespaces and resources you care about (CPU, memory, storage).
• Integrate alerts with notification channels like Slack or email.
• Build comprehensive Grafana dashboards with thresholds and color coding.
• Consider predictive alerts based on usage trends.

Remember, start small and iterate. A narrow, measurable pilot will help you validate your monitoring setup before scaling.