Automating Kubernetes Resource Requests and Limits with CI/CD: A Practical Guide

Introduction

Setting resource requests and limits is a core practice for running reliable workloads on Kubernetes. Without them, a single misbehaving pod can starve other services on the same node, cause unexpected evictions, or inflate your cloud bill. However, enforcing these settings consistently across teams is hard: developers forget them in YAML, reviewers miss them in pull requests, and production incidents expose the gap.

This article provides a practical guide to automating Kubernetes resource requests and limits using CI/CD pipelines. You will learn how to validate manifests before they reach the cluster, enforce policies with admission controllers, and set up safe rollback mechanisms when things go wrong. We focus on operational safety: observe the current state, make small reversible changes, and verify outcomes with concrete commands. Every section includes real kubectl commands, pipeline snippets, and expected outputs.

By the end, you will have a repeatable workflow for resource management that works for developers, DevOps consultants, and technical startup teams.

Prerequisites and Environment Inventory

Before automating anything, document your environment. This section covers the versions and tools you need, and how to inspect your cluster's current resource configuration.

Required Tooling

• Kubernetes cluster version 1.25 or later (for stable support of kubectl features and admission webhooks).
• kubectl client version matching the cluster within one minor version. Verify with:

  kubectl version --short

Expected output:

  Client Version: v1.27.3
  Kustomize Version: v5.0.1
  Server Version: v1.27.3

• A CI/CD platform (GitHub Actions, GitLab CI, Jenkins, etc.) with access to a container image registry and the cluster's kubeconfig (stored as a secret).
• Optional but recommended: a policy engine such as OPA Gatekeeper, Kyverno, or Kubernetes native ValidatingAdmissionPolicy.

Read-Only Observation Commands

Run these to assess the current state of resource requests and limits in your cluster:

kubectl get pods --all-namespaces -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIM:.spec.containers[*].resources.limits.memory

This shows any missing values. For a deeper look at a specific pod:

kubectl describe pod <pod-name> -n <namespace>

Look for events like FailedScheduling with messages such as 0/3 nodes are available: 3 Insufficient cpu.

To see resource quotas and limit ranges that may already be in place:

kubectl get resourcequota -A
kubectl get limitrange -A

Example output:

NAMESPACE   NAME           AGE
prod        prod-quota     10d

If quotas exist, inspect their constraints:

kubectl describe resourcequota prod-quota -n prod

Expected fields include:

Resource       Used  Hard
--------       ----  ----
requests.cpu   500m  10
limits.memory  1Gi   20Gi

Scoping the Automation

Before changing anything, ask:

• Which namespaces do you want to enforce? (e.g., prod, staging)
• Do you need different policies per namespace or workload type?
• What is the minimal viable policy? Start with: every container must have non-zero requests and limits for CPU and memory.

Record your current state and timestamps. For example:

date && kubectl get pods -o wide > pod_state_$(date +%Y%m%d).txt

Safe Configuration Path

This section explains how to build a safe automation pipeline that validates resource requests and limits before deployment. We will cover both CI/CD validation and runtime enforcement.

Step 1: Define the Policy as Code

Create a policy document that specifies the rules. For a CI/CD pipeline, you can use a simple script. Here is a Bash example using kubectl dry-run and jq to check for missing resources:

#!/bin/bash
# validate_resources.sh
set -euo pipefail

for file in ./manifests/*.yaml; do
  echo "Validating $file"
  kubectl apply --dry-run=client -f "$file" -o json | jq -e '
    .items[]?.spec.template.spec.containers[]? |
    (.resources.requests.cpu != null) and (.resources.requests.memory != null) and
    (.resources.limits.cpu != null) and (.resources.limits.memory != null)
  ' >/dev/null || {
    echo "ERROR: $file is missing resource requests or limits"
    exit 1
  }
done

This script fails if any container in a deployment lacks requests or limits.

Step 2: Integrate with CI/CD Pipeline

Example GitHub Actions workflow snippet:

name: Validate Kubernetes Manifests
on: [pull_request, push]
jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Setup kubectl
        uses: azure/setup-kubectl@v4
        with:
          version: 'v1.27.3'
      - name: Run resource validation
        run: ./validate_resources.sh

On a pull request, this check runs automatically and blocks merging if resources are missing.

Step 3: Add Runtime Enforcement with a Policy Engine

CI/CD only catches manifests that go through the pipeline. To enforce cluster-wide, use an admission controller. Kubernetes 1.26+ supports native ValidatingAdmissionPolicy. Here is an example policy (assuming the ValidatingAdmissionPolicy feature gate is enabled):

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-resource-limits
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments", "statefulsets", "daemonsets"]
  validations:
    - expression: "object.spec.template.spec.containers.all(c, has(c.resources) && has(c.resources.requests) && has(c.resources.limits))"
      message: "All containers must have resource requests and limits"

Apply it:

kubectl apply -f require-resource-limits.yaml

Test with an invalid deployment:

kubectl apply -f test-without-limits.yaml

Expected error:

Error from server (Forbidden): error when creating "test-without-limits.yaml": admission webhook "validating.admission.policy/require-resource-limits" denied the request: All containers must have resource requests and limits

Step 4: Use Resource Quotas and Limit Ranges

ResourceQuota limits aggregate resource consumption per namespace, while LimitRange sets defaults for pods that omit requests/limits.

Example LimitRange to default missing values:

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: prod
spec:
  limits:
  - default:
      cpu: 500m
      memory: 512Mi
    defaultRequest:
      cpu: 200m
      memory: 256Mi
    type: Container

Apply and verify:

kubectl apply -f limitrange.yaml
kubectl get limitrange -n prod

Now, if a pod is created without requests/limits in the prod namespace, it will receive the default values.

Step 5: Gradual Rollout

Start with a single namespace or a canary deployment. For example, enforce only on new deployments in staging, monitor for a week, then expand to prod. Never apply a policy change cluster-wide at once.

Verification and Diagnostics

After applying your policies, verify that the automation works correctly. This section covers commands to confirm resource settings and diagnose issues.

Verifying Resource Requests and Limits

Check a running pod's actual values:

kubectl get pod my-app-7d8f9b6c-abcde -n prod -o jsonpath='{.spec.containers[0].resources}'

Example output:

{"limits":{"cpu":"500m","memory":"512Mi"},"requests":{"cpu":"200m","memory":"256Mi"}}

To view all pods in a namespace with their resources:

kubectl get pods -n prod -o custom-columns=NAME:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,CPU_LIM:.spec.containers[*].resources.limits.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIM:.spec.containers[*].resources.limits.memory

Diagnosing Scheduling Failures

If a pod cannot schedule due to resource constraints, use:

kubectl describe pod <pod-name> -n <namespace>

Look for events like:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/3 nodes are available: 3 Insufficient cpu.

To see node capacity and allocatable resources:

kubectl describe node <node-name> | grep -A 5 "Allocated resources"

Example output:

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                1500m (75%)   5000m (250%)
  memory             2Gi (50%)     4Gi (100%)

Checking Policies and Quotas

Verify that the admission policy is active:

kubectl get validatingadmissionpolicies

Test the policy with a failing manifest and confirm the denial message.

Check resource quota usage:

kubectl describe resourcequota prod-quota -n prod

Monitor for Used approaching Hard limits.

CI/CD Pipeline Logs

In your CI system, inspect the logs of the validation job. For GitHub Actions, go to the Actions tab, select the run, and review the Run resource validation step. It should exit with code 0 on success; otherwise, an error message appears.

Failure Modes and Recovery

Even with automation, failures happen. This section covers common failure scenarios and how to recover safely.

Failure: Pipeline Validation Fails on Existing Manifests

If your CI/CD check starts failing because developers forgot to add resources to new manifests, that is expected. To handle legacy manifests, you can either:

• Update them manually to add resources, or
• Temporarily allow exceptions via an annotation (if your policy engine supports it).

For a ValidatingAdmissionPolicy, you can add a matchExclusions to skip certain namespaces or labels. Example to exclude a namespace:

matchExclusions:
- objectSelector:
    matchLabels:
      exempt: "true"

Then label the namespace:

kubectl label namespace legacy-apps exempt=true

But ensure you have a plan to bring those workloads into compliance.

Failure: Admission Policy Blocks a Critical Deployment

If a policy is too strict and blocks a legitimate deployment, you can temporarily disable the policy. For ValidatingAdmissionPolicy, set failurePolicy to Ignore, or delete the policy:

kubectl delete validatingadmissionpolicy require-resource-limits

Then deploy the workload, fix the manifest, and re-apply the policy.

Always have a rollback plan. In CI/CD, use version control to revert a policy change and re-run the pipeline.

Failure: Pod OOMKilled Despite Limits

If a pod is killed for exceeding memory limits (OOMKilled), check the pod status:

kubectl get pod my-app -n prod

Status will show OOMKilled in the RESTARTS column or in kubectl describe.

Then increase the memory limit appropriately:

resources:
  limits:
    memory: "1Gi"
  requests:
    memory: "512Mi"

Apply with:

kubectl apply -f deployment.yaml

Monitor:

kubectl top pod my-app -n prod

Failure: Cluster Running Out of Resources

If nodes are overloaded, you may see Evicted pods. Check events:

kubectl get events --all-namespaces | grep Evicted

Recovery steps:

• Scale down or remove resource-hungry workloads.
• Add more nodes.
• Adjust resource quotas to prevent overcommit.

Rollback Best Practices

Always keep the previous working version of your manifests in a Git repository. Use kubectl rollout undo for deployments:

kubectl rollout undo deployment/my-app -n prod

And verify:

kubectl rollout status deployment/my-app -n prod

For policy changes, revert the commit and re-run CI/CD. Document the recovery procedure in your runbooks.

Operations Checklist

Use this checklist to ensure your resource management automation is operationally sound.

1. Environment Inventory

• [ ] Run kubectl version --short and record client/server versions.
• [ ] List existing resource quotas: kubectl get resourcequota -A.
• [ ] List limit ranges: kubectl get limitrange -A.
• [ ] Capture current pod resource usage: kubectl get pods -o wide --all-namespaces > pod_state_$(date +%Y%m%d).txt.

1. Policy Definition

• [ ] Define minimum requirements: every container must have CPU and memory requests and limits.
• [ ] Create a validation script or use a policy engine.
• [ ] Test the policy locally with a sample invalid manifest.

1. CI/CD Integration

• [ ] Add the validation step to the pipeline (on pull requests and pushes).
• [ ] Ensure the pipeline fails on invalid manifests.
• [ ] Protect the main branch so that only passing checks can be merged.

1. Runtime Enforcement

• [ ] Apply admission policy or LimitRange to the target namespace.
• [ ] Test with a pod missing resources and confirm denial or defaulting.
• [ ] Set up resource quotas to cap total consumption.

1. Verification

• [ ] Deploy a test workload and verify resources with kubectl get pod -o jsonpath.
• [ ] Check scheduling: kubectl describe pod shows successful scheduling.
• [ ] Monitor CI/CD logs for successful runs.

1. Monitoring and Alerting

• [ ] Set up alerts for resource quota usage > 80%.
• [ ] Alert on pod evictions or OOMKilled events.
• [ ] Review dashboards for CPU/memory requests vs limits.

1. Rollback Plan

• [ ] Document how to revert policy changes.
• [ ] Use kubectl rollout undo for deployment regressions.
• [ ] Keep all manifests in Git with history.

1. Periodic Review

• [ ] Monthly, review resource usage and adjust requests/limits for efficiency.
• [ ] Quarterly, review policy effectiveness and update as needed.
• [ ] After incidents, update runbooks based on lessons learned.

Conclusion

Automating Kubernetes resource requests and limits with CI/CD is a powerful way to prevent resource exhaustion, reduce costs, and improve cluster stability. By combining pipeline validation, admission policies, and resource quotas, you create multiple layers of defense that catch issues early and enforce standards consistently.

The key is to proceed with operational safety: observe the current state, implement small reversible changes, verify with concrete commands, and have a rollback plan. Use the examples in this guide as a starting point, but adapt them to your specific environment and policies.

Start small: pick one namespace, define your minimum requirements, add a validation script to your CI pipeline, and monitor the results. As confidence grows, expand to more namespaces and add runtime enforcement. Remember that resource management is not a one-time setup; it requires ongoing review and adjustment as workloads evolve.

With these practices, you can ensure that every container running in your Kubernetes cluster has appropriate resource requests and limits, leading to predictable performance and a more resilient platform.