Kubernetes Vertical Pod Autoscale Production Operations: A Practical Checklist

Intro

Kubernetes Vertical Pod Autoscale (VPA) production operations should move operators from an observed problem to a verified result. This article provides a practical checklist for developers, DevOps consultants, and technical startup teams running VPA in production. It connects VPA operations, checklist items, best practices, and maintenance to concrete commands, expected output, failure signals, and recovery decisions.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery steps before an incident occurs.

VPA automatically adjusts resource requests and limits for pods based on historical and current usage. It is distinct from Horizontal Pod Autoscaler (HPA), which scales the number of replicas. VPA is useful for workloads with variable resource needs, such as batch jobs, stateful services, or applications that are difficult to tune manually. However, VPA in production requires careful planning because it can restart pods, evict workloads, and interact with HPA and cluster autoscaler.

This checklist covers key operational areas:

1. Version and Environment Inventory – know what is installed and its prerequisites.
2. Safe Configuration Path – deploy and adjust VPA safely.
3. Verification and Diagnostics – confirm that VPA is working and troubleshoot issues.
4. Failure Modes and Recovery – understand what can go wrong and how to recover.
5. Operations Checklist – a consolidated list for routine and incident operations.

Each section includes commands, expected outputs, and decision points.

Version and Environment Inventory

Before doing anything with VPA, identify the installed version, deployment topology, prerequisites, and exact component being inspected. VPA is composed of three components:

• Recommender: monitors resource usage and recommends target requests and limits.
• Updater: evicts pods that need new resource values and applies updates.
• Admission Controller: intercepts pod creation and rewrites resource requests.

Check if VPA is installed in your cluster:

kubectl get pods -n kube-system | grep vpa

Expected output:

vpa-admission-controller-xxxxx   1/1     Running   0          2d
vpa-recommender-xxxxx            1/1     Running   0          2d
vpa-updater-xxxxx                1/1     Running   0          2d

If the Admission Controller pod is missing, VPA cannot modify running pods, only recommend values. This is critical: the UpdateMode in a VPA object controls whether VPA can actually change resources or only provide recommendations.

Check the VPA version by examining the container image:

kubectl get deployment vpa-recommender -n kube-system -o jsonpath='{.spec.template.spec.containers[0].image}'

Example output:

registry.k8s.io/autoscaling/vpa-recommender:0.14.0

Prerequisites:

• Kubernetes 1.12+ (required for VPA, but newer versions have better integration).
• Metrics Server must be installed and running for VPA to collect resource metrics.
• If using HPA alongside VPA, be aware of conflicts (discussed later).
• Cluster must have sufficient capacity for pod evictions and rescheduling.

Read-only observation commands:

kubectl get vpa --all-namespaces
kubectl describe vpa <vpa-name> -n <namespace>

Check that metrics server is healthy:

kubectl get apiservice v1beta1.metrics.k8s.io -o yaml

Expected status: Available: True.

Record the current state with timestamps before making changes:

date -u +"%Y-%m-%dT%H:%M:%SZ"
kubectl get deployment -n kube-system | grep vpa
kubectl get pods -n kube-system | grep vpa

Keep a log of these outputs for audit and rollback.

Important: VPA requires the admission controller to be registered as a MutatingAdmissionWebhook. Verify:

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations | grep vpa

If the webhook is not registered, VPA will not apply recommendations to new pods.

Safe Configuration Path

Once the environment is understood, follow a safe path for configuring VPA. The key resource is the VerticalPodAutoscaler custom resource.

A minimal VPA definition:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
  namespace: default
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: "Auto"

UpdateMode can be:

• Off: VPA only recommends, never changes pods.
• Initial: VPA applies resource requests only when a pod is created, does not evict running pods.
• Recreate: VPA can evict and recreate pods to apply new recommendations.
• Auto: VPA uses Recreate for pods that can be safely evicted (respecting PodDisruptionBudget) and Initial otherwise.

For production, start with Off or Initial to observe recommendations before allowing changes.

Apply the VPA object:

kubectl apply -f vpa.yaml

Verify it was created:

kubectl get vpa my-app-vpa

Expected output:

NAME          MODE   CPU   MEM   PROVIDED   AGE
my-app-vpa    Off    100m  256Mi  true       10s

Check the detailed recommendations:

kubectl describe vpa my-app-vpa

Example snippet:

Recommendation:
  Container Recommendations:
    Container Name:  my-app
    Lower Bound:
      Cpu:     50m
      Memory:  100Mi
    Target:
      Cpu:     100m
      Memory:  256Mi
    Upper Bound:
      Cpu:     500m
      Memory:  1Gi

This tells you the recommended range and target values. Before enabling Auto, ensure that:

• Your application can tolerate restarts (if Recreate).
• PodDisruptionBudgets are configured so that evictions do not cause downtime.
• HPA is not configured on the same resource (CPU/memory) to avoid conflicts.

If you must use both HPA and VPA, use VPA for memory and HPA for CPU, or set VPA to Off mode for the HPA-controlled metric. Otherwise, VPA may overwrite HPA scaling decisions.

Safe configuration example with PDB:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-app-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: my-app

Apply PDB before enabling Auto.

To change update mode from Off to Auto, edit the VPA object:

kubectl patch vpa my-app-vpa --type='json' -p='[{"op": "replace", "path": "/spec/updatePolicy/updateMode", "value": "Auto"}]'

Always do a small change, verify, then proceed.

Verification and Diagnostics

Verification ensures VPA is operating as expected and helps diagnose issues.

Check VPA status:

kubectl get vpa my-app-vpa -o yaml

Look for status.conditions:

conditions:
- lastTransitionTime: "2024-03-15T10:00:00Z"
  status: "True"
  type: RecommendationProvided

If status is False, investigate the recommender logs:

kubectl logs -n kube-system deployment/vpa-recommender | tail -20

Common issue: no metrics available. Check Metrics Server:

kubectl top pods

If this fails, Metrics Server may not be installed or not collecting metrics.

Check if VPA is updating pods:

When update mode is Auto or Recreate, VPA Updater may evict pods. Check events:

kubectl get events -n default --field-selector involvedObject.name=my-app-vpa

Look for eviction events:

5m         Warning   EvictedByVPA   pod/my-app-1234   Pod was evicted by VPA Updater to apply resource recommendations.

Check if the pod restarted with new resources:

kubectl get pod my-app-1234 -o yaml | grep -A4 resources

If resources are not updated, ensure the Admission Controller is working. Check its logs:

kubectl logs -n kube-system deployment/vpa-admission-controller | tail -20

Verify that recommendations are being applied to new pods:

Create a test deployment and inspect the pod's resource requests after creation:

kubectl run test-pod --image=nginx --restart=Never
kubectl get pod test-pod -o jsonpath='{.spec.containers[0].resources}'

If VPA is configured for that deployment, the resources should be modified.

Diagnostic commands:

• Check VPA recommender metrics endpoint:

kubectl port-forward -n kube-system svc/vpa-recommender 8942:8942
curl http://localhost:8942/metrics | grep vpa_recommender

• Check updater metrics:

kubectl port-forward -n kube-system svc/vpa-updater 8943:8943
curl http://localhost:8943/metrics

Metrics such as vpa_recommender_recommendation_latest indicate activity.

Failure Modes and Recovery

VPA can fail in several ways. Understanding these helps in quick recovery.

1. VPA Evicts Critical Pods

Symptom: Unexpected pod restarts, application availability drops.

Cause: UpdateMode set to Auto or Recreate without proper PDBs.

Recovery:

• Immediately set VPA to Off:

kubectl patch vpa my-app-vpa --type='merge' -p '{"spec":{"updatePolicy":{"updateMode":"Off"}}}'

• Recreate the pod with original resources if needed.

• Add PDBs and consider Initial mode.

2. VPA Recommender Not Providing Recommendations

Symptom: kubectl describe vpa shows no recommendation, status.conditions type RecommendationProvided is False.

Cause: Metrics Server unavailable, or recommender cannot scrape metrics.

Diagnosis:

kubectl logs -n kube-system deployment/vpa-recommender --tail=50

Look for errors like failed to get metrics.

Recovery:

• Fix Metrics Server:

kubectl get deployment metrics-server -n kube-system
kubectl rollout restart deployment metrics-server -n kube-system

• If metrics are missing for a long period, VPA may turn off recommendations. After fixing, wait for a few minutes.

3. VPA Admission Controller Not Modifying Pods

Symptom: New pods are created with original resource requests even though VPA is in Auto mode.

Cause: Webhook not registered, admission controller not running, or VPA targetRef mismatch.

Diagnosis:

kubectl get mutatingwebhookconfigurations vpa-webhook-config -o yaml

Check that it exists and has the correct CA bundle.

Check admission controller logs:

kubectl logs -n kube-system deployment/vpa-admission-controller --tail=50

Recovery:

• Recreate the webhook configuration if corrupted.
• Ensure admission controller is running.
• Verify VPA object's targetRef matches the deployment.

4. VPA Conflict with Horizontal Pod Autoscaler

Symptom: HPA tries to scale replicas based on CPU, but VPA also adjusts CPU requests causing oscillation.

Cause: VPA and HPA both managing CPU.

Recovery:

• Set VPA to manage memory only, and HPA to manage CPU. This requires customizing VPA container policies:

spec:
  resourcePolicy:
    containerPolicies:
    - containerName: "*"
      controlledResources: ["memory"]

• Or disable HPA and rely on VPA for both.

5. VPA Causing Resource Starvation on Node

Symptom: Pods evicted due to node pressure, VPA recommends high values.

Cause: VPA overestimates due to spikes or noisy neighbors.

Recovery:

• Set upper bounds using minAllowed and maxAllowed:

spec:
  resourcePolicy:
    containerPolicies:
    - containerName: "*"
      minAllowed:
        cpu: "50m"
        memory: "100Mi"
      maxAllowed:
        cpu: "500m"
        memory: "1Gi"

• Adjust VPA recommender parameters (e.g., --recommendation-margin-fraction=0.15).

• Investigate actual usage with kubectl top pods.

Recovery Verification

After any recovery action, verify:

• VPA status returns to normal.
• Pods are running with expected resources.
• Application metrics are stable.
• No further evictions occur unnecessarily.

Document each incident with timestamp, symptom, cause, action, and result.

Operations Checklist

Use this concise checklist for routine and incident operations.

Pre-deployment

• [ ] Confirm VPA components are running in kube-system.
• [ ] Confirm Metrics Server is healthy.
• [ ] Confirm VPA webhook is registered.
• [ ] Identify workload and its current resource requests/limits.
• [ ] Check if HPA is already configured on the workload.
• [ ] Define VPA update mode based on workload tolerance for restarts.
• [ ] Set minAllowed and maxAllowed to prevent over/under provisioning.
• [ ] Create or verify PodDisruptionBudget if using Auto or Recreate.

Deployment

• [ ] Apply VPA manifest and verify creation.
• [ ] Check kubectl describe vpa for recommendations.
• [ ] If using Off mode, manually review recommendations.
• [ ] If enabling Auto, do it during low traffic and monitor.

Verification

• [ ] kubectl get vpa -o yaml shows RecommendationProvided=True.
• [ ] New pods receive updated resources (check with kubectl describe pod).
• [ ] No unexpected evictions in events.
• [ ] Application performance metrics within acceptable range.

Incident Response

• [ ] If pods are being evicted excessively, set VPA to Off immediately.
• [ ] Check recommender/updater logs for errors.
• [ ] Verify Metrics Server and webhook.
• [ ] Check for HPA conflict.
• [ ] Adjust resource policies if needed.
• [ ] Document incident and recovery.

Maintenance

• [ ] Periodically review VPA recommendations vs actual usage.
• [ ] Monitor VPA component versions and upgrade carefully.
• [ ] Test VPA behavior in staging before applying to production.
• [ ] Keep PDBs and resource policies up to date.
• [ ] Ensure metrics collection is reliable.

Conclusion

Kubernetes Vertical Pod Autoscale production operations require a disciplined, observable, and reversible approach. This checklist provides the essential commands, failure signals, and recovery decisions for safe VPA operation. Start with a read-only inventory, understand the VPA components and update modes, then gradually enable automation with safeguards.

As a next step, choose one low-risk verification: create a VPA in Off mode for a non-critical deployment, observe recommendations for a day, compare them with actual usage, and then decide whether to move to Auto with appropriate resource bounds and PDBs.

A reliable VPA workflow makes failure visible, protects sensitive values, limits changes to intended resources, and defines recovery verification before an incident forces the decision.