Automating Kubernetes Vertical Pod Autoscale with CI/CD: A Practical Guide

Intro

Kubernetes Vertical Pod Autoscale (VPA) adjusts the CPU and memory requests of running pods based on observed usage. This helps right-size workloads, reduce waste, and prevent out-of-memory kills. However, VPA changes can be disruptive if not rolled out carefully. Automating VPA with CI/CD pipelines lets you promote safe updates, validate changes in staging, and roll back quickly when a recommendation is wrong or a workload behaves unexpectedly.

This guide walks through a practical approach to integrating VPA into your CI/CD workflow. You will learn how to check your cluster version and compatibility, install VPA with version control, configure update policies, build a pipeline that stages changes, verify adjustments with real commands, and recover from common failures. The examples assume a working Kubernetes cluster and use standard kubectl and Helm commands.

By the end, you will have a repeatable process that limits risk, uses placeholders instead of secrets, and documents recovery steps before you need them.

Version and Environment Inventory

Before automating VPA, confirm that your cluster and tooling support it.

Prerequisites

• Kubernetes cluster version 1.24 or later (VPA is available in all supported versions, but some features change).
• Metrics Server installed and running (VPA needs metrics to make recommendations).
• kubectl and helm clients installed locally.
• Access to the cluster with permission to create CustomResourceDefinitions (CRDs) and deployments in the target namespace.

Check Current State

Run these read-only commands to capture the initial environment:

kubectl version --short
kubectl get nodes -o wide
kubectl get pods -n kube-system | grep metrics-server

Expected output is the Kubernetes version, node status, and a running metrics-server pod. If metrics-server is missing, install it before VPA. For example, on many clusters:

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

Identify the VPA Component

VPA consists of three components:

• Recommender: watches pod resource usage and suggests new requests.
• Updater: evicts pods that need new resource requests (if update mode allows).
• Admission Controller: sets initial resource requests when a pod is created.

Note the exact versions you plan to deploy. The current stable VPA release is v1.2.0 (as of this writing). You can check available releases with:

helm repo add vpa https://charts.fairwinds.com/stable
helm search repo vpa/vpa --versions

Use a specific version instead of latest for reproducibility.

Safe Configuration Path

A safe configuration path means starting with observation, applying a minimal change, and verifying before expanding. For VPA, this translates to installing with a conservative update policy, targeting one workload, and monitoring the behavior before wider adoption.

Install VPA with Helm

Create a values.yaml file to override defaults:

recommender:
  enabled: true
updater:
  enabled: true
admissionController:
  enabled: true

Install with Helm, pinning the version:

helm upgrade --install vpa vpa/vpa --version 1.2.0 -n vpa --create-namespace -f values.yaml

Verify that all three components are running:

kubectl get pods -n vpa -o wide
kubectl logs -n vpa deployment/vpa-updater --tail=20

Expect three pods with status Running and logs showing the updater is connected to the API server.

Create a VPA Object with a Conservative Update Mode

Start with updateMode: Off to get recommendations without making any changes. For a deployment named my-app:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
  namespace: default
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
    - containerName: "*"
      minAllowed:
        cpu: 50m
        memory: 100Mi
      maxAllowed:
        cpu: 2
        memory: 2Gi

Apply the VPA:

kubectl apply -f vpa-off.yaml
kubectl get vpa my-app-vpa -o yaml

Check the status.recommendation section. It should show target CPU and memory but the pod should not be restarted. Example output:

status:
  recommendation:
    containerRecommendations:
    - containerName: my-app
      lowerBound:
        cpu: 100m
        memory: 200Mi
      target:
        cpu: 150m
        memory: 300Mi
      upperBound:
        cpu: 1
        memory: 1Gi

Observe for a few hours or days before enabling updates.

Verification and Diagnostics

Verification ensures the VPA is working and the changes are safe. Use these commands to inspect recommendations, eviction events, and actual resource usage.

Inspect VPA Recommendations

Get the current recommendation with human-readable output:

kubectl describe vpa my-app-vpa

Look for the Recommendation block. If the target is much higher or lower than current requests, investigate the workload.

Check the actual resource usage of pods:

kubectl top pods -l app=my-app

Compare the CPU(cores) and MEMORY(bytes) columns with the VPA target. They should be close if the VPA is accurate.

Check VPA Component Logs

If recommendations are missing, check logs:

kubectl logs -n vpa deployment/vpa-recommender --tail=50
kubectl logs -n vpa deployment/vpa-updater --tail=50

Common errors include metrics server not ready or permission issues. Ensure the VPA service account has rights to read metrics and pods.

Validate a CI/CD Pipeline Step

A typical pipeline step for updating VPA in a GitOps workflow would be:

1. Developer updates the VPA manifest in the repository.
2. CI runs kubectl apply --dry-run=client -f vpa.yaml to validate syntax.
3. If dry-run succeeds, CD applies the manifest to the cluster.
4. After apply, run a verification job that waits for the VPA object to be ready and checks recommendations.

Example verification job using kubectl wait:

kubectl wait --for=condition=Available vpa/my-app-vpa --timeout=60s

If the VPA has no condition, use a custom script:

REC=$(kubectl get vpa my-app-vpa -o jsonpath='{.status.recommendation.containerRecommendations[0].target.cpu}')
if [ -z "$REC" ]; then
  echo "No recommendation yet"
  exit 1
else
  echo "Recommendation CPU: $REC"
fi

This ensures the VPA is functioning before you promote changes to production.

Failure Modes and Recovery

VPA can cause issues if misconfigured or applied too aggressively. Here are common failure modes and recovery steps.

Overly Aggressive Updates Cause Pod Evictions

If updateMode: Auto and the updater evicts too many pods, it can cause service disruption. To mitigate:

• Use updateMode: Initial or Recreate only after testing.
• Limit the update policy with minReplicas if using the updater's eviction controls.
• Set maxAllowed to prevent extreme changes.

Recovery: switch to Off immediately:

kubectl patch vpa my-app-vpa --type merge -p '{"spec":{"updatePolicy":{"updateMode":"Off"}}}'

Then scale the deployment to restore pods:

kubectl scale deployment my-app --replicas=3
kubectl rollout status deployment/my-app

VPA Recommends Too Low Resources

If the VPA recommends values that are too low, the pod may crash. Observe with:

kubectl describe pod <pod-name> | tail -20

Look for OOMKilled in the last state. To fix, manually increase requests in the deployment and adjust the VPA minAllowed:

minAllowed:
  cpu: 200m
  memory: 400Mi

Apply the change and restart the pod:

kubectl delete pod <pod-name>

VPA Components Fail

If recommender or updater pods crash, check logs and events:

kubectl describe pod -n vpa <pod-name>
kubectl logs -n vpa <pod-name> --previous

Common causes: insufficient permissions, API version mismatch, or missing CRDs. Reinstall CRDs if needed:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/autoscaler/vpa-release-1.2/vertical-pod-autoscaler/deploy/vpa-v1-crd-gen.yaml

Rollback to Previous State

If a CI/CD pipeline applied a bad VPA change, rollback via Git revert:

git revert <commit-hash>
git push origin main

The CD system will apply the previous manifest. Alternatively, manually apply the last known good VPA:

kubectl apply -f good-vpa.yaml

Verify that the VPA object is updated and the pods are stable.

Operations Checklist

Use this checklist before and after applying VPA changes in a CI/CD pipeline.

For every checklist item, record the timestamp, the command output, and the person who ran it. This audit trail is essential for debugging and compliance.

Conclusion

Automating Kubernetes Vertical Pod Autoscale with CI/CD stops guesswork and prevents resource waste. You must version your VPA installation, use a conservative update mode initially, verify recommendations with kubectl, and have a rollback plan before you enable automatic updates.

Start with one low-risk deployment in Off mode. Observe the VPA recommendations for at least a full business cycle. Then move to Auto in a non-production environment, test the pipeline, and monitor evictions. Only after that should you consider production rollout with gradual rollout strategies.

Remember: VPA is a tool to help you, not a replacement for alerting and capacity planning. Pair it with Horizontal Pod Autoscaler if you need to scale replicas based on load, and always keep resource requests and limits within sane bounds.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With the commands and examples in this guide, you can build that workflow for VPA today.