## Intro Kubernetes Vertical Pod Autoscaler (VPA) dynamically adjusts the CPU and memory requests and limits of pods based on actual usage. While powerful, VPA can introduce subtle issues: pods stuck in Pending, unexpected evictions, incorrect recommendations, and configuration conflicts. This guide covers the most common VPA errors you will encounter in production, how to diagnose them with practical commands, and how to apply fixes safely. We focus on developers, DevOps engineers, and technical startup teams running Kubernetes in production. Every section includes real-world scenarios, kubectl commands, expected outputs, and concrete YAML snippets. We emphasize operational safety: observe first, change one thing at a time, and verify the result before moving on. By the end of this article, you will be able to: - Identify whether VPA is installed and healthy in your cluster - Diagnose common VPA errors using kubectl and logs - Apply targeted fixes for resource constraints, misconfigurations, and conflicts - Use VPA with confidence alongside Horizontal Pod Autoscaler (HPA) ## Version and Environment Inventory Before troubleshooting VPA, you must know exactly what is installed and how it is configured. Start by checking the VPA version and deployment status. **Check if VPA is installed** ```bash kubectl get pods -n kube-system | grep vpa ``` Expected output if VPA is running: ``` vpa-admission-controller-6c7f8b9d4-abcde 1/1 Running 0 2d vpa-recommender-5f9c8b7d6-ghijk 1/1 Running 0 2d vpa-updater-7d8e9f0a1-lmnop 1/1 Running 0 2d ``` If you see no pods, VPA is not installed. Install it following the official documentation for your Kubernetes version. **Verify VPA API version** VPA uses a Custom Resource Definition (CRD). Check available API versions: ```bash kubectl api-versions | grep autoscaling.k8s.io ``` Expected output (for a recent cluster): ``` autoscaling.k8s.io/v1 ``` If you see only `v1beta2`, your VPA version may be older and have different behaviors. Align your troubleshooting steps with the API version. **Check VPA metrics source** VPA relies on metrics-server or Prometheus. Ensure metrics-server is running: ```bash kubectl get deployment metrics-server -n kube-system ``` If metrics-server is missing or unhealthy, VPA recommendations will be empty or delayed. Install or fix metrics-server first. **Key checklist for environment inventory:** - VPA components running in `kube-system` namespace - API version supports your use case - Metrics source accessible - Cluster has sufficient capacity for VPA-managed pods ## Safe Configuration Path A common error is creating a VPA object that conflicts with existing pod resource settings or has invalid parameters. Follow this safe configuration path to avoid misconfigurations. **Step 1: Understand VPA modes** VPA supports four update modes: - `Off`: only provides recommendations, no changes - `Initial`: sets resource requests at pod creation only - `Recreate`: evicts and recreates pods to apply new requests/limits (use with caution) - `Auto`: dynamically updates resource requests without eviction, but limits are not changed The most common error is using `Auto` mode and expecting limits to be adjusted, which does not happen. VPA only updates requests (and limits if `minAllowed` and `maxAllowed` are set and mode is `Auto` with `controlledResources` including `Limits`). Be clear about what VPA can and cannot do. **Step 2: Validate your VPA YAML** Here is a minimal VPA definition: ```yaml apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: my-app-vpa spec: targetRef: apiVersion: "apps/v1" kind: Deployment name: my-app updatePolicy: updateMode: "Auto" resourcePolicy: containerPolicies: - containerName: '*' minAllowed: cpu: 100m memory: 50Mi maxAllowed: cpu: 1 memory: 500Mi controlledResources: ["cpu", "memory"] ``` Common mistakes: - `targetRef.name` does not match an existing Deployment - `containerName` does not match the container in the pod spec - `minAllowed` or `maxAllowed` missing, causing unbounded recommendations - `controlledResources` omits a resource you expect to be managed **Step 3: Apply incrementally** Never apply VPA to a large deployment without testing. Start with a single replica and `Off` mode to observe recommendations before enabling updates. ```bash kubectl apply -f vpa.yaml kubectl get vpa my-app-vpa -o yaml ``` Look for `status.recommendation` to see what VPA suggests. **Step 4: Monitor events** After applying VPA, check events for anomalies: ```bash kubectl describe vpa my-app-vpa ``` Expected output includes events like: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Updated 10m vpa-updater Updated pod specs for my-app-1234 ``` If you see warnings about failed updates, investigate the vpa-updater logs. ## Verification and Diagnostics Once VPA is configured, you must verify it is working correctly and diagnose any errors. This section covers common diagnostic commands and interpretation of results. **Check VPA status and recommendations** ```bash kubectl get vpa my-app-vpa -o jsonpath='{.status.recommendation.containerRecommendations[0].target}' && echo ``` Expected output example: ``` map[cpu:200m memory:100Mi] ``` If the output is empty, VPA is not generating recommendations. This often indicates metrics-server is not providing data or the pod has no resource requests initially. **Inspect VPA events** ```bash kubectl describe vpa my-app-vpa | tail -20 ``` Look for error events such as: ``` Warning FailedGetResourceMetric 5m vpa-recommender unable to get metrics for resource cpu: no metrics returned from resource metrics API ``` This error points to a metrics collection issue. **Check vpa-recommender logs** ```bash kubectl logs -n kube-system deployment/vpa-recommender --tail=50 ``` Look for lines containing `ERROR` or `WARN`. Common errors: - `failed to get metrics` - `could not retrieve pod list` - `inconsistent labels` Each log line includes a pod name and reason, giving you a precise lead. **Check pod resource requests and limits** After VPA updates a pod, verify the new requests: ```bash kubectl get pod my-app-1234 -o jsonpath='{.spec.containers[0].resources}' && echo ``` Expected output: ``` map[limits:map[cpu:1 memory:500Mi] requests:map[cpu:200m memory:100Mi]] ``` If the requests are unchanged, VPA may not be updating due to mode `Off` or a policy constraint. **Troubleshooting flow** 1. Confirm VPA components are running. 2. Confirm metrics-server is available. 3. Confirm VPA object is targeted correctly. 4. Confirm update mode is not `Off`. 5. Confirm no conflict with HPA (explained later). ## Failure Modes and Recovery VPA can cause or be involved in several failure modes. This section details the most common errors, their symptoms, root causes, and recovery steps. ### Error 1: Pods stuck in Pending state **Symptom:** Pods managed by VPA remain in `Pending` with events like: ``` Warning FailedScheduling 10m default-scheduler 0/3 nodes are available: 3 Insufficient cpu. ``` **Root cause:** VPA has increased resource requests beyond the capacity of any node, or `maxAllowed` is set too high. **Diagnosis:** ```bash kubectl describe pod my-app-1234 | grep -A5 Events ``` Check the requested CPU/memory and node capacity: `kubectl get nodes -o custom-columns='NAME:.metadata.name,CPU:.status.allocatable.cpu,MEMORY:.status.allocatable.memory'` **Recovery:** Adjust `maxAllowed` to fit your cluster or add more nodes. You can also temporarily set `updateMode: "Off"` and manually lower requests to bring the pod back. ```bash kubectl patch vpa my-app-vpa --type='json' -p='[{"op": "replace", "path": "/spec/updatePolicy/updateMode", "value": "Off"}]' ``` Then edit the deployment to reduce requests manually. ### Error 2: VPA evicts pods too frequently **Symptom:** Pods are constantly being evicted and recreated due to VPA `Recreate` or `Auto` mode with aggressive updates. **Root cause:** VPA recommendation changes often because of fluctuating usage, and the update mode triggers evictions on every change. **Diagnosis:** ```bash kubectl get events --field-selector reason=Evicted -n my-namespace ``` Check the vpa-updater logs for eviction decisions. **Recovery:** Switch to `Auto` mode if not already, and consider increasing the `minReplicas` or using `Off` mode during peak variability. You can also tune the recommender interval (default 1 minute) by adjusting the recommender deployment flags. ### Error 3: HPA and VPA conflict **Symptom:** HPA scales pods based on CPU, and VPA also adjusts CPU requests, causing thrashing or unexpected scaling behavior. **Root cause:** Both HPA and VPA are managing the same resource metric. This is a known anti-pattern. **Diagnosis:** Check if both have targetRef to the same deployment: `kubectl get hpa,vpa -n my-namespace` **Recovery:** Do not use HPA on CPU/memory when VPA is active on those resources. If you must use both, set VPA to `Off` mode or limit HPA to custom metrics only. Alternatively, use HPA on external metrics and VPA on resource requests. ### Error 4: VPA recommendations are stale or empty **Symptom:** `kubectl get vpa` shows no recommendations or recommendations that never change. **Root cause:** Metrics-server not running, missing RBAC permissions, or VPA targetRef points to a non-existent workload. **Diagnosis:** - Check metrics-server: `kubectl top pods -n kube-system` - Check VPA CRD and controller pods: `kubectl logs -n kube-system deployment/vpa-recommender --tail=20` - Verify RBAC: `kubectl auth can-i get pods.metrics.k8s.io -n kube-system --as system:serviceaccount:kube-system:vpa-recommender` **Recovery:** Fix metrics-server or RBAC, or correct the targetRef. After fixing, wait a few minutes for new recommendations. ### Error 5: VPA does not update limits **Symptom:** Pod requests are updated but limits remain unchanged, causing OOM kills. **Root cause:** By default, VPA only updates requests. To update limits, you must explicitly set `controlledResources` to include `Limits` and provide `minAllowed` and `maxAllowed` for limits. **Diagnosis:** Inspect VPA configuration and pod limits. **Recovery:** Update VPA resource policy as follows: ```yaml resourcePolicy: containerPolicies: - containerName: '*' controlledResources: ["cpu", "memory", "limits.cpu", "limits.memory"] minAllowed: cpu: 100m memory: 50Mi limits.cpu: 200m limits.memory: 100Mi maxAllowed: cpu: 1 memory: 500Mi limits.cpu: 2 limits.memory: 1Gi ``` Then apply and verify. ## Operations Checklist Use this checklist before and after any VPA change to ensure safe operation. **Before enabling VPA on a workload:** - [ ] Confirm VPA components are healthy: `kubectl get pods -n kube-system | grep vpa` - [ ] Confirm metrics-server is available: `kubectl top pods -n my-namespace` - [ ] Check for existing HPA on the same deployment: `kubectl get hpa -n my-namespace` - [ ] Set VPA `updateMode` to `Off` first to observe recommendations without changes - [ ] Define `minAllowed` and `maxAllowed` to prevent out-of-range requests - [ ] Test on a low-impact deployment with one replica **After VPA is active:** - [ ] Monitor pod restarts: `kubectl get pods -w` - [ ] Check for evictions: `kubectl get events --field-selector reason=Evicted -n my-namespace` - [ ] Verify actual resource usage: `kubectl top pods -n my-namespace` - [ ] Review VPA recommendations: `kubectl get vpa -o yaml` - [ ] Ensure no node exhaustion: `kubectl describe nodes | grep -A5 'Allocated resources'` **Rollback plan:** - Delete the VPA object: `kubectl delete vpa my-app-vpa` - Manually set resource requests and limits in the deployment spec - Redeploy and verify pods are running ## Conclusion Kubernetes Vertical Pod Autoscaler can significantly improve resource utilization and application stability, but it must be configured and monitored carefully. This guide covered common errors such as pods stuck pending, frequent evictions, HPA conflicts, empty recommendations, and unchanged limits. For each, we provided practical commands, YAML examples, and recovery steps. Remember the core operational principles: - Observe before changing: always check current state and VPA logs - Limit blast radius: test with `Off` mode and small workloads - Verify after changes: confirm recommendations, pod resources, and cluster capacity - Have a rollback plan: know how to disable VPA and manually set resources As a next step, run through the Operations Checklist on one of your deployments. Start with `Off` mode, watch the recommendations for a day, then gradually switch to `Auto` with appropriate min/max bounds. Document your findings and share them with your team. With careful implementation, VPA will help you run a more efficient and resilient Kubernetes cluster.