Kubernetes Vertical Pod Autoscaler Commands: A Practical Operations Guide

Intro

Kubernetes clusters often run workloads with varying resource needs. Setting resource requests and limits too low causes throttling or eviction; setting them too high wastes capacity and increases cloud spend. The Vertical Pod Autoscaler (VPA) automatically adjusts CPU and memory requests and limits for pods based on observed usage. Unlike the Horizontal Pod Autoscaler (HPA), which changes the number of replicas, VPA modifies the pod spec to right-size resources. This guide focuses on the essential kubectl commands you need to operate VPA effectively, with practical examples, safe configuration patterns, and troubleshooting steps.

VPA is not enabled by default in most clusters. It is typically installed as a set of controllers and a mutating admission webhook from the Kubernetes autoscaler repository. You interact with VPA through its custom resource definition (CRD): VerticalPodAutoscaler. Using basic kubectl commands, you can inspect recommendations, monitor updates, and diagnose problems. This article provides a hands-on reference for daily operations.

Version and Environment Inventory

Before using VPA commands, establish your cluster environment. You need to know:

• Kubernetes cluster version. VPA supports Kubernetes 1.8+, but recent versions are strongly recommended for stability and bug fixes.
• VPA version installed. You can find this by checking the controller image tags or the VPA deployment manifest.
• Whether VPA is running in "recommendation only" mode or actively applying updates. This is controlled by the updateMode field in each VPA object.
• Namespaces where VPA objects exist. Use kubectl get vpa -A to list all VPAs across all namespaces.

First, verify cluster connectivity and kubectl version:

kubectl version --short

Expected output includes both client and server versions, for example:

Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

Next, check if the VPA CRDs are installed:

kubectl get crd | grep verticalpodautoscalers

If VPA is installed, you should see the CRD verticalpodautoscalers.autoscaling.k8s.io. If not, install VPA before proceeding.

Identify the VPA controller pods (typically running in kube-system):

kubectl get pods -n kube-system | grep vpa

Example output:

vpa-admission-controller-6d5c8f6c9b-abcde   1/1     Running   0          10d
vpa-recommender-6d5c8f6c9b-fghij           1/1     Running   0          10d
vpa-updater-6d5c8f6c9b-klmno               1/1     Running   0          10d

The three components are:

• Recommender: watches pod resource usage and computes recommended requests and limits.
• Updater: evicts pods whose resources need updating (when updateMode allows it).
• Admission Controller: mutates pod specs at creation time to apply VPA recommendations (if enabled).

Understanding this architecture helps interpret command outputs and diagnose failures.

Safe Configuration Path

When configuring VPA, start with a narrow scope and avoid immediate pod disruption. The safest approach is to begin with updateMode: Off to observe recommendations without automatic updates. After validating, switch to Recreate mode for controlled updates. The following steps walk through the full lifecycle.

Step 1: Create a VPA object for a specific workload

Create a YAML file myapp-vpa.yaml with the following content (adjust selector and name for your application):

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: myapp-vpa
  namespace: default
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: myapp
  updatePolicy:
    updateMode: "Off"   # start with Off to only get recommendations
  resourcePolicy:
    containerPolicies:
      - containerName: '*'
        minAllowed:
          cpu: 100m
          memory: 100Mi
        maxAllowed:
          cpu: 2
          memory: 2Gi
        controlledResources: ["cpu", "memory"]

This object targets a Deployment named myapp. The containerName: '*' applies to all containers. minAllowed and maxAllowed bound the recommendations. Apply it:

kubectl apply -f myapp-vpa.yaml

Verify the object exists:

kubectl get vpa myapp-vpa

Step 2: Inspect the VPA object details

Use kubectl describe vpa myapp-vpa to see the status, including recommendations, current pod resource requests, and conditions:

kubectl describe vpa myapp-vpa -n default

Example snippet from the Status section:

Status:
  Recommendation:
    Container Recommendations:
      Container Name: myapp-container
      Lower Bound:
        Cpu:     100m
        Memory:  100Mi
      Target:
        Cpu:     250m
        Memory:  200Mi
      Upper Bound:
        Cpu:     500m
        Memory:  400Mi
  Conditions:
    Last Transition Time:  2023-10-01T12:00:00Z
    Message:               Recommendation is provided
    Reason:                RecommendationProvided
    Status:                True
    Type:                  RecommendationProvided

In this example, VPA recommends a target of 250m CPU and 200Mi memory, within the specified bounds. The current pod resources are also shown elsewhere in the output; compare them to the recommendation.

Step 3: Transition to "Recreate" mode after validation

Once recommendations look reasonable, you can allow VPA to update pods automatically. Edit the VPA object:

kubectl edit vpa myapp-vpa

Change updateMode: "Off" to updateMode: "Recreate". Alternatively, use a JSON patch:

kubectl patch vpa myapp-vpa --type='json' -p='[{"op": "replace", "path": "/spec/updatePolicy/updateMode", "value": "Recreate"}]'

Verify the update:

kubectl get vpa myapp-vpa -o yaml | grep updateMode

Output should show updateMode: Recreate. In Recreate mode, VPA will evict pods that need new resources. To minimize disruption, ensure your deployment has multiple replicas and a PodDisruptionBudget (PDB) in place.

Verification and Diagnostics

After VPA is active, monitor its behavior and verify that pod resources are being adjusted correctly.

Check VPA recommendations

Use kubectl describe vpa <name> to view the current recommendation summary. For example:

kubectl describe vpa myapp-vpa | grep -A5 Recommendation

This shows the recommended target resources. Compare them with the actual pod spec by checking the deployment:

kubectl get deployment myapp -o yaml | grep -A4 resources

For a running pod, get its actual resources:

kubectl get pod <pod-name> -o jsonpath='{.spec.containers[0].resources}'

This prints the resource requests and limits as applied. If they differ from the deployment template, the admission controller may have mutated the pod.

View VPA events

Events provide insight into VPA actions, such as pod evictions and updates:

kubectl get events --field-selector involvedObject.name=myapp-vpa

Or filter events in the namespace:

kubectl get events -n default --sort-by=.metadata.creationTimestamp

Look for messages like Pod myapp-xyz was evicted to apply new resource recommendations or Updated pod myapp-xyz with recommended resources.

List all VPA objects in a namespace

kubectl get vpa -n default

Example output:

NAME        MODE    CPU    MEMORY   PROVIDED   AGE
myapp-vpa   Recreate    250m   200Mi    True       1h

Get VPA in YAML for full details

kubectl get vpa myapp-vpa -o yaml

This is useful for auditing the full spec and status, including resource policies and conditions. Look for status.conditions to confirm the recommender is functioning.

Check if the admission controller is modifying pod specs

When a pod is created, the admission controller may modify its resource requests based on VPA recommendations. To see actual pod resources:

kubectl get pod <pod-name> -o jsonpath='{.spec.containers[0].resources}'

If the output differs from the deployment template, VPA is actively mutating pods. Note that this only applies when the admission controller is deployed and the VPA update mode is Recreate or InPlaceOrRecreate; in Off mode, no mutation occurs.

Failure Modes and Recovery

VPA can encounter issues such as incorrect recommendations, unexpected pod evictions, or conflicts with other resource management tools. Knowing how to detect and recover from these situations is crucial.

Common failure modes and remedies

1. VPA recommends too low or too high resources due to insufficient historical data.

• Detection: Pod restarts, CPU throttling metrics, or memory OOMKilled events. Check pod events: kubectl describe pod <pod-name> | grep OOMKilled.
• Recovery: Temporarily set updateMode: Off and adjust minAllowed/maxAllowed bounds in resourcePolicy. Or delete the VPA object to stop all recommendations. Ensure the workload has enough replicas to serve traffic while you tune.

1. VPA evicting pods excessively causing application instability.

• Detection: Frequent pod restarts and VPA events showing evictions. Monitor kubectl get events | grep Evicted.
• Recovery: Set updateMode: Initial to only apply recommendations at pod creation, or Off to stop updates. Tune minReplicas in a PodDisruptionBudget or ensure the application can handle rolling updates gracefully. Consider increasing replicas during the tuning period.

1. Admission controller overriding explicit pod resource requests, leading to unexpected behavior.

• Detection: Pod spec resources differ from deployment spec. Use kubectl get pod <pod> -o yaml and compare with kubectl get deployment <deploy> -o yaml.
• Recovery: Remove the VPA for that workload, or exclude specific containers using containerPolicies with mode: "Off". Reapply the desired resource requests to the deployment.

1. VPA components not running (e.g., recommender pod crash-looping), resulting in no recommendations.

• Detection: kubectl get pods -n kube-system | grep vpa shows non-running pods; check logs for errors.
• Recovery: Restart the failed pod or redeploy VPA. Check logs for root cause, such as insufficient permissions or misconfigured flags.

Rollback steps

To roll back VPA changes:

1. Delete the VPA object for the workload:

   kubectl delete vpa myapp-vpa

This stops recommendations and automatic updates. Existing pods keep their current resource values, but new pods will use the deployment template without VPA mutation.

1. Restore the original resource requests in the deployment if they were changed by VPA. Edit the deployment:

   kubectl edit deployment myapp

Manually set resources to known good values, or apply a previously saved manifest.

1. Verify pod resources return to expected values by checking a pod spec:

   kubectl get pod <new-pod> -o jsonpath='{.spec.containers[0].resources}'

Check VPA component logs for troubleshooting

For recommender issues:

kubectl logs -n kube-system deployment/vpa-recommender --tail=50

For admission controller issues:

kubectl logs -n kube-system deployment/vpa-admission-controller --tail=50

For updater issues:

kubectl logs -n kube-system deployment/vpa-updater --tail=50

Look for error messages related to API calls, metric collection, or pod eviction failures.

Common Pitfalls and How to Avoid Them

Beyond outright failures, several subtle pitfalls can undermine VPA effectiveness.

1. Ignoring the lower and upper bounds. Without setting minAllowed and maxAllowed, VPA may recommend very low resources for idle applications, causing startup failures, or very high resources for spiky workloads, wasting cluster capacity. Always define bounds based on your application's baseline and expected peaks. For example, a web service that typically uses 200m CPU but bursts to 1 CPU during traffic spikes should have minAllowed: cpu: 100m and maxAllowed: cpu: 1.5 to avoid throttling during bursts.

1. Using VPA with HPA on the same metric. VPA and HPA can conflict if both try to manage CPU or memory. If HPA scales replicas based on CPU and VPA adjusts CPU requests, the interplay can cause oscillations. The Kubernetes documentation recommends not using VPA with HPA on CPU or memory simultaneously. If you need both, use VPA for vertical scaling and HPA for a different metric (e.g., custom metrics) or use the Kubernetes Event-driven Autoscaling (KEDA) for event-driven scaling. To recover, temporarily disable one controller and observe stability.

1. Forgetting about pod disruption budgets. In Recreate mode, VPA may evict pods that need more resources. If your deployment has few replicas and no PDB, this can cause downtime. Always define a PDB to limit voluntary disruptions. Example PDB:

   apiVersion: policy/v1
   kind: PodDisruptionBudget
   metadata:
     name: myapp-pdb
   spec:
     minAvailable: 2
     selector:
       matchLabels:
         app: myapp

1. Applying VPA to stateful workloads without careful testing. StatefulSets often have persistent identity and ordered pod management. VPA evictions can disrupt stateful applications. Test VPA in a staging environment first, and use updateMode: Off initially. For databases, consider using updateMode: Initial so resources are set only at pod creation, and manually reschedule pods during maintenance windows.

1. Assuming VPA recommendations are instantly correct. VPA uses historical usage data; it needs time to learn. New workloads may receive suboptimal recommendations until enough data accumulates (typically several days). Monitor recommendations over time and adjust bounds if needed. A healthy VPA condition RecommendationProvided with status True does not guarantee the recommendation is perfect; it only means the recommender ran. Always validate with your own metrics.

Operations Checklist

Use the following checklist for regular VPA operations. For each item, an owner is suggested; in a team setting, a single platform engineer or on-call SRE should be accountable, and the checklist is typically reviewed weekly unless otherwise stated.

Adjust the frequency based on your environment size and criticality. In production, daily automated checks with alerting are recommended.

Conclusion

Mastering the basic kubectl commands for Kubernetes Vertical Pod Autoscaler enables you to safely manage resource optimization in your clusters. Start with kubectl get vpa, kubectl describe vpa, and monitoring events to understand VPA behavior. Then progressively adopt automatic updates with careful validation. Always have a rollback plan: deleting the VPA object and restoring resource specs is straightforward. By following the safe configuration path, avoiding common pitfalls, and using the diagnostics commands outlined here, you can keep your workloads right-sized and your cluster efficient.

For further reading, consult the official Kubernetes VPA documentation and experiment in a non-production namespace first.