## Intro Kubernetes clusters often run workloads with varying resource needs. Setting resource requests and limits too low causes throttling or eviction; setting them too high wastes capacity and increases cloud spend. The Vertical Pod Autoscaler (VPA) automatically adjusts CPU and memory requests and limits for pods based on observed usage. Unlike the Horizontal Pod Autoscaler (HPA), which changes the number of replicas, VPA modifies the pod spec to right-size resources. This guide focuses on the essential kubectl commands you need to operate VPA effectively, with practical examples, safe configuration patterns, and troubleshooting steps. VPA is not enabled by default in most clusters. It is typically installed as a set of controllers and a mutating admission webhook from the Kubernetes autoscaler repository. You interact with VPA through its custom resource definition (CRD): `VerticalPodAutoscaler`. Using basic kubectl commands, you can inspect recommendations, monitor updates, and diagnose problems. This article provides a hands-on reference for daily operations. ## Version and Environment Inventory Before using VPA commands, establish your cluster environment. You need to know: - Kubernetes cluster version. VPA supports Kubernetes 1.8+, but recent versions are strongly recommended for stability and bug fixes. - VPA version installed. You can find this by checking the controller image tags or the VPA deployment manifest. - Whether VPA is running in "recommendation only" mode or actively applying updates. This is controlled by the `updateMode` field in each VPA object. - Namespaces where VPA objects exist. Use `kubectl get vpa -A` to list all VPAs across all namespaces. First, verify cluster connectivity and kubectl version: ```bash kubectl version --short ``` Expected output includes both client and server versions, for example: ```text Client Version: v1.27.3 Kustomize Version: v5.0.1 Server Version: v1.27.3 ``` Next, check if the VPA CRDs are installed: ```bash kubectl get crd | grep verticalpodautoscalers ``` If VPA is installed, you should see the CRD `verticalpodautoscalers.autoscaling.k8s.io`. If not, install VPA before proceeding. Identify the VPA controller pods (typically running in `kube-system`): ```bash kubectl get pods -n kube-system | grep vpa ``` Example output: ```text vpa-admission-controller-6d5c8f6c9b-abcde 1/1 Running 0 10d vpa-recommender-6d5c8f6c9b-fghij 1/1 Running 0 10d vpa-updater-6d5c8f6c9b-klmno 1/1 Running 0 10d ``` The three components are: - **Recommender**: watches pod resource usage and computes recommended requests and limits. - **Updater**: evicts pods whose resources need updating (when `updateMode` allows it). - **Admission Controller**: mutates pod specs at creation time to apply VPA recommendations (if enabled). Understanding this architecture helps interpret command outputs and diagnose failures. ## Safe Configuration Path When configuring VPA, start with a narrow scope and avoid immediate pod disruption. The safest approach is to begin with `updateMode: Off` to observe recommendations without automatic updates. After validating, switch to `Recreate` mode for controlled updates. The following steps walk through the full lifecycle. ### Step 1: Create a VPA object for a specific workload Create a YAML file `myapp-vpa.yaml` with the following content (adjust selector and name for your application): ```yaml apiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: myapp-vpa namespace: default spec: targetRef: apiVersion: "apps/v1" kind: Deployment name: myapp updatePolicy: updateMode: "Off" # start with Off to only get recommendations resourcePolicy: containerPolicies: - containerName: '*' minAllowed: cpu: 100m memory: 100Mi maxAllowed: cpu: 2 memory: 2Gi controlledResources: ["cpu", "memory"] ``` This object targets a Deployment named `myapp`. The `containerName: '*'` applies to all containers. `minAllowed` and `maxAllowed` bound the recommendations. Apply it: ```bash kubectl apply -f myapp-vpa.yaml ``` Verify the object exists: ```bash kubectl get vpa myapp-vpa ``` ### Step 2: Inspect the VPA object details Use `kubectl describe vpa myapp-vpa` to see the status, including recommendations, current pod resource requests, and conditions: ```bash kubectl describe vpa myapp-vpa -n default ``` Example snippet from the `Status` section: ```text Status: Recommendation: Container Recommendations: Container Name: myapp-container Lower Bound: Cpu: 100m Memory: 100Mi Target: Cpu: 250m Memory: 200Mi Upper Bound: Cpu: 500m Memory: 400Mi Conditions: Last Transition Time: 2023-10-01T12:00:00Z Message: Recommendation is provided Reason: RecommendationProvided Status: True Type: RecommendationProvided ``` In this example, VPA recommends a target of 250m CPU and 200Mi memory, within the specified bounds. The current pod resources are also shown elsewhere in the output; compare them to the recommendation. ### Step 3: Transition to "Recreate" mode after validation Once recommendations look reasonable, you can allow VPA to update pods automatically. Edit the VPA object: ```bash kubectl edit vpa myapp-vpa ``` Change `updateMode: "Off"` to `updateMode: "Recreate"`. Alternatively, use a JSON patch: ```bash kubectl patch vpa myapp-vpa --type='json' -p='[{"op": "replace", "path": "/spec/updatePolicy/updateMode", "value": "Recreate"}]' ``` Verify the update: ```bash kubectl get vpa myapp-vpa -o yaml | grep updateMode ``` Output should show `updateMode: Recreate`. In Recreate mode, VPA will evict pods that need new resources. To minimize disruption, ensure your deployment has multiple replicas and a PodDisruptionBudget (PDB) in place. ## Verification and Diagnostics After VPA is active, monitor its behavior and verify that pod resources are being adjusted correctly. ### Check VPA recommendations Use `kubectl describe vpa ` to view the current recommendation summary. For example: ```bash kubectl describe vpa myapp-vpa | grep -A5 Recommendation ``` This shows the recommended target resources. Compare them with the actual pod spec by checking the deployment: ```bash kubectl get deployment myapp -o yaml | grep -A4 resources ``` For a running pod, get its actual resources: ```bash kubectl get pod -o jsonpath='{.spec.containers[0].resources}' ``` This prints the resource requests and limits as applied. If they differ from the deployment template, the admission controller may have mutated the pod. ### View VPA events Events provide insight into VPA actions, such as pod evictions and updates: ```bash kubectl get events --field-selector involvedObject.name=myapp-vpa ``` Or filter events in the namespace: ```bash kubectl get events -n default --sort-by=.metadata.creationTimestamp ``` Look for messages like `Pod myapp-xyz was evicted to apply new resource recommendations` or `Updated pod myapp-xyz with recommended resources`. ### List all VPA objects in a namespace ```bash kubectl get vpa -n default ``` Example output: ```text NAME MODE CPU MEMORY PROVIDED AGE myapp-vpa Recreate 250m 200Mi True 1h ``` ### Get VPA in YAML for full details ```bash kubectl get vpa myapp-vpa -o yaml ``` This is useful for auditing the full spec and status, including resource policies and conditions. Look for `status.conditions` to confirm the recommender is functioning. ### Check if the admission controller is modifying pod specs When a pod is created, the admission controller may modify its resource requests based on VPA recommendations. To see actual pod resources: ```bash kubectl get pod -o jsonpath='{.spec.containers[0].resources}' ``` If the output differs from the deployment template, VPA is actively mutating pods. Note that this only applies when the admission controller is deployed and the VPA update mode is `Recreate` or `InPlaceOrRecreate`; in `Off` mode, no mutation occurs. ## Failure Modes and Recovery VPA can encounter issues such as incorrect recommendations, unexpected pod evictions, or conflicts with other resource management tools. Knowing how to detect and recover from these situations is crucial. ### Common failure modes and remedies 1. **VPA recommends too low or too high resources due to insufficient historical data.** - **Detection**: Pod restarts, CPU throttling metrics, or memory OOMKilled events. Check pod events: `kubectl describe pod | grep OOMKilled`. - **Recovery**: Temporarily set `updateMode: Off` and adjust `minAllowed`/`maxAllowed` bounds in `resourcePolicy`. Or delete the VPA object to stop all recommendations. Ensure the workload has enough replicas to serve traffic while you tune. 2. **VPA evicting pods excessively causing application instability.** - **Detection**: Frequent pod restarts and VPA events showing evictions. Monitor `kubectl get events | grep Evicted`. - **Recovery**: Set `updateMode: Initial` to only apply recommendations at pod creation, or `Off` to stop updates. Tune `minReplicas` in a PodDisruptionBudget or ensure the application can handle rolling updates gracefully. Consider increasing replicas during the tuning period. 3. **Admission controller overriding explicit pod resource requests, leading to unexpected behavior.** - **Detection**: Pod spec resources differ from deployment spec. Use `kubectl get pod -o yaml` and compare with `kubectl get deployment -o yaml`. - **Recovery**: Remove the VPA for that workload, or exclude specific containers using `containerPolicies` with `mode: "Off"`. Reapply the desired resource requests to the deployment. 4. **VPA components not running (e.g., recommender pod crash-looping), resulting in no recommendations.** - **Detection**: `kubectl get pods -n kube-system | grep vpa` shows non-running pods; check logs for errors. - **Recovery**: Restart the failed pod or redeploy VPA. Check logs for root cause, such as insufficient permissions or misconfigured flags. ### Rollback steps To roll back VPA changes: 1. Delete the VPA object for the workload: ```bash kubectl delete vpa myapp-vpa ``` This stops recommendations and automatic updates. Existing pods keep their current resource values, but new pods will use the deployment template without VPA mutation. 2. Restore the original resource requests in the deployment if they were changed by VPA. Edit the deployment: ```bash kubectl edit deployment myapp ``` Manually set resources to known good values, or apply a previously saved manifest. 3. Verify pod resources return to expected values by checking a pod spec: ```bash kubectl get pod -o jsonpath='{.spec.containers[0].resources}' ``` ### Check VPA component logs for troubleshooting For recommender issues: ```bash kubectl logs -n kube-system deployment/vpa-recommender --tail=50 ``` For admission controller issues: ```bash kubectl logs -n kube-system deployment/vpa-admission-controller --tail=50 ``` For updater issues: ```bash kubectl logs -n kube-system deployment/vpa-updater --tail=50 ``` Look for error messages related to API calls, metric collection, or pod eviction failures. ## Common Pitfalls and How to Avoid Them Beyond outright failures, several subtle pitfalls can undermine VPA effectiveness. 1. **Ignoring the lower and upper bounds.** Without setting `minAllowed` and `maxAllowed`, VPA may recommend very low resources for idle applications, causing startup failures, or very high resources for spiky workloads, wasting cluster capacity. Always define bounds based on your application's baseline and expected peaks. For example, a web service that typically uses 200m CPU but bursts to 1 CPU during traffic spikes should have `minAllowed: cpu: 100m` and `maxAllowed: cpu: 1.5` to avoid throttling during bursts. 2. **Using VPA with HPA on the same metric.** VPA and HPA can conflict if both try to manage CPU or memory. If HPA scales replicas based on CPU and VPA adjusts CPU requests, the interplay can cause oscillations. The Kubernetes documentation recommends not using VPA with HPA on CPU or memory simultaneously. If you need both, use VPA for vertical scaling and HPA for a different metric (e.g., custom metrics) or use the Kubernetes Event-driven Autoscaling (KEDA) for event-driven scaling. To recover, temporarily disable one controller and observe stability. 3. **Forgetting about pod disruption budgets.** In `Recreate` mode, VPA may evict pods that need more resources. If your deployment has few replicas and no PDB, this can cause downtime. Always define a PDB to limit voluntary disruptions. Example PDB: ```yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: myapp-pdb spec: minAvailable: 2 selector: matchLabels: app: myapp ``` 4. **Applying VPA to stateful workloads without careful testing.** StatefulSets often have persistent identity and ordered pod management. VPA evictions can disrupt stateful applications. Test VPA in a staging environment first, and use `updateMode: Off` initially. For databases, consider using `updateMode: Initial` so resources are set only at pod creation, and manually reschedule pods during maintenance windows. 5. **Assuming VPA recommendations are instantly correct.** VPA uses historical usage data; it needs time to learn. New workloads may receive suboptimal recommendations until enough data accumulates (typically several days). Monitor recommendations over time and adjust bounds if needed. A healthy VPA condition `RecommendationProvided` with status `True` does not guarantee the recommendation is perfect; it only means the recommender ran. Always validate with your own metrics. ## Operations Checklist Use the following checklist for regular VPA operations. For each item, an owner is suggested; in a team setting, a single platform engineer or on-call SRE should be accountable, and the checklist is typically reviewed weekly unless otherwise stated. | Step | Command | Purpose | Owner | Frequency | |------|---------|---------|-------|-----------| | 1 | `kubectl get vpa -A` | List all VPAs in all namespaces | Platform Engineer | Weekly | | 2 | `kubectl describe vpa -n ` | View recommendations and status | Application Owner | Weekly | | 3 | `kubectl get events -n \| grep VPA` | Check for VPA-related events | On-call SRE | Daily | | 4 | `kubectl get pods -n kube-system \| grep vpa` | Verify VPA components are running | Platform Engineer | Daily | | 5 | `kubectl logs -n kube-system deployment/vpa-recommender --tail=20` | Check recommender logs for errors | Platform Engineer | Weekly | | 6 | `kubectl get vpa -o yaml \| grep updateMode` | Confirm update mode is as intended | Application Owner | Before any change | | 7 | `kubectl describe pod \| grep -A2 Requests` | Compare actual pod resources with recommendations | Application Owner | Weekly | | 8 | If issues, set `updateMode: Off` via patch | Immediately stop VPA updates | On-call SRE | As needed | Adjust the frequency based on your environment size and criticality. In production, daily automated checks with alerting are recommended. ## Conclusion Mastering the basic kubectl commands for Kubernetes Vertical Pod Autoscaler enables you to safely manage resource optimization in your clusters. Start with `kubectl get vpa`, `kubectl describe vpa`, and monitoring events to understand VPA behavior. Then progressively adopt automatic updates with careful validation. Always have a rollback plan: deleting the VPA object and restoring resource specs is straightforward. By following the safe configuration path, avoiding common pitfalls, and using the diagnostics commands outlined here, you can keep your workloads right-sized and your cluster efficient. For further reading, consult the official Kubernetes VPA documentation and experiment in a non-production namespace first.