Kubernetes Vertical Pod Autoscale Monitoring and Alerts: A Practical Guide

Intro

Kubernetes Vertical Pod Autoscaler (VPA) automatically adjusts the CPU and memory requests and limits of pods based on historical and current usage. While VPA simplifies resource management, it requires careful monitoring to ensure it is working correctly, not causing instability, and not over- or under-provisioning. This guide provides a practical approach to monitoring VPA, collecting the right metrics, setting up alerts, building dashboards, and handling incidents.

We will cover the entire operational lifecycle: from the initial version and environment inventory, through safe configuration, verification, failure modes, and finally an operations checklist. Each section includes concrete commands, expected outputs, and decision criteria to help you run VPA in production with confidence.

The goal is operational safety: observe before changing, limit the blast radius, protect sensitive values, verify results, and document recovery paths. Whether you are a developer, DevOps engineer, or technical startup team, this guide will help you move from a detected problem to a verified resolution.

Version and Environment Inventory

Before monitoring VPA, you must understand what is installed and running. Start by identifying the VPA version, its deployment topology, prerequisites, and the specific components you will inspect. This inventory establishes a baseline and helps avoid misconfigurations.

Checking VPA Installation

Use the following command to list VPA pods and their namespaces:

kubectl get pods -n kube-system | grep vpa

Expected output (example):

vpa-admission-controller-6f7b8c9d-abcde   1/1     Running   0          2d
vpa-recommender-7b8c9d6f5-xyz12          1/1     Running   0          2d
vpa-updater-8c9d6f5e4-pqr34             1/1     Running   0          2d

The VPA consists of three main components:

• Recommender: Monitors resource usage and generates recommendations.
• Updater: Evicts pods if they need to be recreated with new resource requests.
• Admission Controller: Adjusts resource requests on new pods if configured.

If any component is missing or not running, VPA will not function correctly. Check the logs for each component:

kubectl logs -n kube-system deployment/vpa-recommender --tail=50

Look for errors such as Failed to list pods or Unable to fetch metrics. The expected output should be a series of log lines indicating successful scraping of metrics.

Verifying VPA API Version

Different VPA versions have different capabilities. Check the API version supported by your cluster:

kubectl api-versions | grep autoscaling.k8s.io

Expected output includes autoscaling.k8s.io/v1 (for VPA) and possibly autoscaling.k8s.io/v1beta2 depending on your version.

Document the exact version and any custom patches. This is critical when consulting documentation or filing issues.

Prerequisites

Ensure that:

• Metrics Server is installed and running (kubectl get deployment metrics-server -n kube-system).
• The VPA CRDs are present: kubectl get crd | grep verticalpodautoscalers.
• You have the necessary RBAC permissions to view VPA objects and events.

Observation vs. Intervention

At this stage, only gather information. Do not modify any resources. Capture current state with timestamps:

kubectl get vpa --all-namespaces -o yaml > vpa-inventory-$(date +%Y%m%d-%H%M%S).yaml

This inventory file is your rollback reference.

Safe Configuration Path

Once you have an inventory, you can make configuration changes in a controlled manner. The VPA is configured via a VerticalPodAutoscaler custom resource. A common use case is to enable VPA for a specific deployment in recommendation-only mode (also known as Off mode), so you can review recommendations before applying them automatically.

Creating a VPA Object in Off Mode

Here is a minimal VPA manifest for a deployment named my-app in namespace default, set to Off mode:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
  namespace: default
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Off"

Apply it:

kubectl apply -f vpa-off.yaml

Expected output:

verticalpodautoscaler.autoscaling.k8s.io/my-app-vpa created

Now the recommender will start generating recommendations but will not change pod resources.

Reviewing Recommendations

After some time (e.g., 24 hours to capture daily patterns), check the recommendations:

kubectl describe vpa my-app-vpa -n default

Look for the Recommendation section in the output:

Recommendation:
  Container Recommendations:
    Container Name:  my-app-container
    Lower Bound:
      Cpu:     100m
      Memory:  50Mi
    Target:
      Cpu:     250m
      Memory:  128Mi
    Upper Bound:
      Cpu:     500m
      Memory:  256Mi

These values indicate the range within which VPA suggests your container's requests should fall. The Target is the value VPA would apply if in Auto mode.

Switching to Auto Mode

After you are confident in the recommendations, you can switch to Auto mode by changing the updateMode field. First, edit the VPA:

kubectl edit vpa my-app-vpa -n default

Change updateMode: "Off" to updateMode: "Auto" and save.

Alternatively, use a patch:

kubectl patch vpa my-app-vpa -n default --type='json' -p='[{"op": "replace", "path": "/spec/updatePolicy/updateMode", "value":"Auto"}]'

Output:

verticalpodautoscaler.autoscaling.k8s.io/my-app-vpa patched

Safe Rollout

When switching to Auto, the VPA updater may evict pods to apply new resource requests. To minimize disruption:

• Apply changes during a low-traffic window.
• Ensure your deployment has multiple replicas (e.g., at least 3) to maintain availability.
• Monitor the rollout:

kubectl rollout status deployment/my-app -n default

Expected output:

deployment "my-app" successfully rolled out

If the rollout fails, you can revert by setting updateMode back to Off and manually adjusting resources.

Verification and Diagnostics

This section covers how to verify that VPA is working correctly and diagnose common issues.

Checking VPA Status

Use the following command to see the VPA status across your cluster:

kubectl get vpa --all-namespaces

Example output:

NAMESPACE   NAME         MODE   CPU    MEM       PROVIDED   AGE
default     my-app-vpa   Auto   250m   128Mi     True       1d

The PROVIDED column indicates whether the VPA has provided a recommendation. True means the recommender has enough data to make a recommendation. False could indicate missing metrics or insufficient history.

Inspecting Events

VPA generates events that can help diagnose problems:

kubectl describe vpa my-app-vpa -n default | grep -A10 Events

Common events include:

• Cannot obtain CPU/memory metrics for pod... - Metrics Server may be down or not scraping.
• Pod ... is not suitable for VPA - The pod may not have the correct labels or is owned by a controller not supported.

Checking Recommendations Over Time

You can view the historical recommendations using:

kubectl get vpa my-app-vpa -n default -o yaml | grep -A5 -B5 recommendation

Or, if you have Prometheus, query the metric vpa_recommendation_target to see trends.

Validating Applied Resources

If VPA is in Auto mode, verify that pods have been recreated with the recommended resources:

kubectl get pods -n default -l app=my-app -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[0].resources}{"\n"}{end}'

Example output:

my-app-6c7d8f9b-abcde   map[limits:map[cpu:500m memory:256Mi] requests:map[cpu:250m memory:128Mi]]

If the resources do not match the VPA target, check whether the admission controller is enabled and the pod was created after the VPA was activated.

Failure Modes and Recovery

VPA can fail in various ways. Understanding these failure modes helps you respond quickly.

Failure: VPA Recommender Not Running

Symptom: kubectl get pods -n kube-system | grep vpa shows no recommender pod or pod is CrashLoopBackOff.

Diagnosis:

kubectl logs -n kube-system deployment/vpa-recommender --tail=100

Look for errors like connection refused to metrics server or invalid configuration.

Recovery:

• Check Metrics Server status: kubectl get apiservice v1beta1.metrics.k8s.io -o yaml and ensure it is Available.
• If the recommender is misconfigured, fix the configuration (e.g., environment variables) and restart the deployment:

kubectl rollout restart deployment/vpa-recommender -n kube-system

Failure: VPA Updater Evicting Pods Repeatedly

Symptom: Pods for a deployment are constantly being evicted and recreated, causing restarts and disruption.

Diagnosis:

• Check VPA events:

kubectl describe vpa my-app-vpa -n default | grep -A20 Events

• Look for messages like Evicted pod ... due to resource update.

Recovery:

• Temporarily set VPA to Off mode to stop evictions:

kubectl patch vpa my-app-vpa -n default --type='json' -p='[{"op": "replace", "path": "/spec/updatePolicy/updateMode", "value":"Off"}]'

• Investigate why VPA keeps changing recommendations (e.g., noisy metrics, insufficient data) and consider adjusting minReplicas or using a custom recommender.

Failure: VPA Not Applying Recommendations

Symptom: VPA shows recommendations but pods are not updated.

Diagnosis:

• Ensure updateMode is Auto.
• Check if the admission controller webhook is registered:

kubectl get mutatingwebhookconfigurations | grep vpa

• If the webhook is missing, reinstall VPA admission controller.

Recovery:

• If admission controller is not working, you can manually update the deployment resources to match VPA target as a temporary fix.

General Recovery: Rollback Strategy

Always have a rollback plan. For example, if VPA causes instability, you can:

1. Set VPA to Off.
2. Manually set resource requests to a known good value (e.g., from before VPA was enabled).
3. Delete the VPA object if necessary:

kubectl delete vpa my-app-vpa -n default

Then monitor the deployment for stability.

Operations Checklist

Use this checklist for daily or weekly VPA operations. Replace the example values with ones specific to your environment.

Daily Checks

Example acceptable output: all rows show True; any False requires investigation.

kubectl get events --all-namespaces | grep -i vpa | tail -20. Look for eviction or recommendation errors.

• [ ] Run kubectl get vpa --all-namespaces and confirm all VPAs have PROVIDED=True.
• [ ] Check for VPA-related events in critical namespaces:
• [ ] Review Prometheus alerts for VPA metrics (if set up).

Weekly Checks

• [ ] Validate that VPA recommendations are within expected ranges for your applications. Compare current pod resource requests to VPA targets.
• [ ] Check for VPA components log errors:

  kubectl logs -n kube-system deployment/vpa-recommender --since=24h | grep -i error | tail -20

No errors should appear; if errors, investigate promptly.

• [ ] Backup VPA custom resources:

  kubectl get vpa --all-namespaces -o yaml > vpa-backup-$(date +%Y%m%d).yaml

Before Any Change

• [ ] Record the current state: kubectl get vpa <name> -n <ns> -o yaml > before.yaml.
• [ ] Determine the exact change to make and its impact.
• [ ] Notify relevant team members if the change may cause pod evictions.
• [ ] Have a rollback command ready (e.g., kubectl apply -f before.yaml).

After Any Change

• [ ] Verify the change: kubectl describe vpa <name> -n <ns>.
• [ ] Check pod status: kubectl get pods -n <ns>.
• [ ] Monitor application metrics for anomalies.

Conclusion

Monitoring Kubernetes Vertical Pod Autoscaler is essential for maintaining efficient and stable workloads. By following the practices in this guide—starting with a thorough inventory, configuring VPA safely, verifying its behavior, and knowing how to recover from failures—you can harness VPA's benefits without the risks.

Always remember the operational principles: observe before changing, limit blast radius, protect sensitive values, verify outcomes, and document recovery steps. As a next step, choose one low-risk application in your cluster, create a VPA in Off mode, and review its recommendations over a week. This hands-on experience will build your confidence and prepare you for broader adoption.

With proper monitoring and alerting, VPA can become a reliable component of your Kubernetes resource management strategy.