Kubernetes Horizontal Pod Autoscaler Backup and Restore: A Practical Guide

Intro

Kubernetes Horizontal Pod Autoscaler (HPA) backup and restore is often treated as an afterthought until a misconfiguration or accidental deletion forces a manual rebuild. A structured approach turns an observed problem into a verified result. This guide focuses on practical, repeatable procedures for backing up and restoring HPA configurations across clusters and namespaces.

We cover version and environment inventory, safe configuration paths, verification and diagnostics, failure modes and recovery, and an operations checklist. Each section includes concrete commands, expected outputs, and real-world examples. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify outcomes, and document recovery before an incident occurs.

This guide is for developers, DevOps consultants, and technical startup teams who manage Kubernetes workloads and want to protect autoscaling configurations as part of their disaster recovery strategy.

Version and Environment Inventory

Before backing up or restoring HPAs, you need a clear picture of your cluster version, HPA API versions, and current HPA objects. Start with read-only observations.

Identify Kubernetes and HPA API Versions

Run the following commands:

kubectl version --short
kubectl api-versions | grep autoscaling

Expected output includes lines like:

autoscaling/v1
autoscaling/v2beta2
autoscaling/v2

As of Kubernetes 1.23, autoscaling/v2 is stable and supports metric-based scaling. If your cluster shows only autoscaling/v1, you are limited to CPU-based scaling. This affects backup format: v2 HPAs may include multiple metrics and behavior configurations not representable in v1.

Capture Current HPA Inventory

List all HPAs across all namespaces:

kubectl get hpa --all-namespaces -o wide

Sample output:

NAMESPACE   NAME          REFERENCE            TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
prod        web-hpa      Deployment/web       cpu: 50%/80%    2         10        4          12d
dev         api-hpa      Deployment/api       memory: 60%     1         5         2          3d

Record this output along with a timestamp. For a complete backup, export each HPA as YAML:

kubectl get hpa web-hpa -n prod -o yaml > web-hpa-backup-$(date +%Y%m%d).yaml

Repeat for all HPAs. Store these files in version control, not just locally.

Prerequisites for Safe HPA Operations

• kubectl version compatible with your cluster (within one minor version).
• Permissions to get and list HPAs in target namespaces (read-only) and to create/update for restore.
• A backup location (git repository, object storage) that is accessible during a disaster.
• A clear understanding of dependent Deployment or Pod resource names, because HPA references them by name.

Smallest Justified Change

Before any backup or restore attempt, verify the target HPA exists and is functioning normally. Use:

kubectl describe hpa web-hpa -n prod

Look for events like:

Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  10m   horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target

If no recent events appear, the HPA may not be actively scaling. Compare with kubectl get hpa output to see current metrics.

Safe Configuration Path

A safe configuration path means making incremental, reversible changes to HPA settings, backed by a previous known-good version.

Backup Before Any Change

Always export the current HPA manifest before editing it. For example:

kubectl get hpa web-hpa -n prod -o yaml > web-hpa-2025-03-15.yaml

This file is your rollback point.

Example: Changing Min and Max Replicas

Suppose you want to increase maxReplicas from 10 to 15 for web-hpa in prod. Here's a minimal patch:

kubectl patch hpa web-hpa -n prod --patch '{"spec":{"maxReplicas":15}}'

Verify the change:

kubectl get hpa web-hpa -n prod -o yaml | grep maxReplicas

Expected output:

maxReplicas: 15

Check that the HPA still matches the deployment's pods:

kubectl get pods -l app=web -n prod

If the HPA tries to scale above the cluster's capacity, you will see events in kubectl describe hpa indicating failed rescale due to insufficient resources.

Using kubectl Apply with a Modified Manifest

Alternatively, edit the exported YAML file, change the desired field, and apply:

kubectl apply -f web-hpa-updated.yaml

Always keep the original backup file intact.

Maintaining a Versioned History

Store HPA manifests in git with meaningful commit messages. For example:

git add web-hpa-*.yaml
git commit -m "Backup HPA web-hpa before maxReplicas change to 15"

This gives you an auditable trail and easy rollback using kubectl apply -f with the previous version.

Verification and Diagnostics

After any HPA change or restore, verify that the HPA is working as expected. Do not assume success from the command exit code.

Check HPA Status and Events

Run:

kubectl describe hpa web-hpa -n prod

Look for the AbleToScale and ScalingActive conditions:

Conditions:
  Type            Status  Reason            Message
  ----            ------  ------            -------
  AbleToScale     True    ReadyForNewScale  recommended size matches current size
  ScalingActive   True    ValidMetricFound  the HPA was able to successfully calculate a replica count from cpu resource utilization

If ScalingActive is False, check the reason. Common issues include missing metrics or incorrect resource requests.

Validate Metric Sources

For CPU-based HPA, ensure your pods have CPU requests set:

kubectl get deployment web -n prod -o jsonpath='{.spec.template.spec.containers[0].resources}'

Output should include something like:

{"requests":{"cpu":"100m"},"limits":{"cpu":"500m"}}

For memory-based HPA, verify memory requests. For custom metrics, ensure the metrics server or Prometheus adapter is running:

kubectl get pods -n kube-system | grep metrics-server

Simulate Load to Trigger Scaling

To verify scaling behavior in a controlled way, temporarily increase load on the deployment. For example, use a load generation tool like hey or ab against a service endpoint. Then watch the HPA:

kubectl get hpa web-hpa -n prod --watch

After a few minutes, you should see REPLICAS increase. Then stop the load and observe scale-down.

Compare Current State with Backup

After restoring an HPA from backup, diff the current manifest with the backup file:

diff <(kubectl get hpa web-hpa -n prod -o yaml) web-hpa-backup.yaml

Any differences should be intentional. This is a crucial validation step.

Failure Modes and Recovery

HPAs can fail in several ways. Here are common failure modes, how to detect them, and how to recover using backups.

Failure Mode 1: Accidental Deletion of an HPA

If someone runs kubectl delete hpa web-hpa -n prod, the Deployment will continue running with its current replica count but will no longer scale automatically.

Detection:

kubectl get hpa -n prod

Output lacks web-hpa.

Recovery:

kubectl apply -f web-hpa-backup-2025-03-15.yaml

Verify:

kubectl get hpa web-hpa -n prod

Failure Mode 2: HPA Targets a Nonexistent Deployment

If the referenced deployment is deleted or renamed, the HPA will report a scaling error.

Detection:

kubectl describe hpa web-hpa -n prod | grep -A5 Events

Look for messages like:

Warning  FailedGetScale  horizontal-pod-autoscaler  deployments/scale.apps "web" not found

Recovery:

• Restore the Deployment if it was deleted.
• Or update the HPA's scaleTargetRef to the correct name using kubectl edit hpa or a patch.

Failure Mode 3: Invalid Metric Configuration

If you change the metric type or incorrectly specify a resource name, the HPA may fail to fetch metrics.

Detection:

kubectl get hpa web-hpa -n prod -o yaml | grep -A10 status

Look for conditions like:

conditions:
- type: ScalingActive
  status: "False"
  reason: FailedGetResourceMetric
  message: missing request for cpu

Recovery:

• Ensure the pods have CPU or memory requests as required.
• Or revert to the previous backup manifest that worked.

Failure Mode 4: HPA Conflicts with Cluster Autoscaler or Resource Quotas

If the cluster cannot provision new nodes, or a ResourceQuota limits pods, HPA scaling will fail.

Detection:

kubectl describe hpa web-hpa -n prod | tail -20

Look for FailedRescale events:

Warning  FailedRescale  horizontal-pod-autoscaler  New size: 6; reason: exceeded quota in namespace prod

Recovery:

• Adjust ResourceQuota or node capacity.
• Or temporarily reduce HPA maxReplicas to a feasible number using the backup as a reference.

General Recovery Procedure

1. Identify the failure from HPA events and conditions.
2. Retrieve the last known-good backup from version control.
3. Apply the backup with kubectl apply -f.
4. Verify with kubectl get hpa and kubectl describe hpa.
5. Document the incident and the fix.

Operations Checklist

Use the following checklist before and after any HPA backup, restore, or modification.

Before Change

• [ ] Run kubectl version --short and note cluster version.
• [ ] Run kubectl api-versions | grep autoscaling to confirm supported HPA API versions.
• [ ] List all HPAs: kubectl get hpa --all-namespaces -o wide
• [ ] Export current HPA YAML for the target namespace with a timestamp, e.g., hpa-backup-prod-web-2025-03-15.yaml.
• [ ] Store the backup file in a version-controlled repository.
• [ ] Check that the referenced Deployment exists and pods have resource requests if using resource metrics.
• [ ] Record current replica count and metrics.

During Change

• [ ] Apply changes with kubectl apply -f or kubectl patch, never with an inline edit that bypasses file backup.
• [ ] Use a single, scoped change at a time (e.g., only maxReplicas).
• [ ] Note the exact command used and expected outcome.

After Change or Restore

• [ ] Run kubectl get hpa <name> -n <namespace> to confirm object exists.
• [ ] Run kubectl describe hpa <name> -n <namespace> and verify conditions AbleToScale=True and ScalingActive=True.
• [ ] Check for error events in the HPA description.
• [ ] Diff the current manifest against the backup (if restoring).
• [ ] Monitor for at least one scaling cycle (e.g., 10-15 minutes) to ensure HPA can scale up and down.
• [ ] If any step fails, roll back to the previous backup manifest immediately.

Example Worked Scenario

Here is a complete backup and restore example for an HPA called web-hpa in namespace prod.

1. Backup:

   kubectl get hpa web-hpa -n prod -o yaml > web-hpa-2025-03-15.yaml
   git add web-hpa-2025-03-15.yaml && git commit -m "Backup HPA before change"

1. Make a change: increase minReplicas from 2 to 3.

   kubectl patch hpa web-hpa -n prod --patch '{"spec":{"minReplicas":3}}'

1. Verify immediate state:

   kubectl get hpa web-hpa -n prod -o yaml | grep minReplicas

Expected: minReplicas: 3

1. Simulate load and watch scaling:

   kubectl get hpa web-hpa -n prod --watch

1. If something goes wrong, roll back:

   kubectl apply -f web-hpa-2025-03-15.yaml

1. Verify rollback:

   kubectl get hpa web-hpa -n prod -o yaml | grep minReplicas

Expected: minReplicas: 2

Conclusion

Kubernetes HPA backup and restore is a critical part of cluster operations. By following a structured approach—inventorying versions, capturing HPA manifests, making small reversible changes, verifying with diagnostics, and having a clear recovery plan—you can prevent autoscaling outages and recover quickly from mistakes.

Always store HPA backups in version control, not just on a local disk. Test restore procedures in a non-production namespace before an actual failure. Make failure visible by monitoring HPA events and conditions. Protect sensitive values by using placeholders and secrets rather than hardcoding. Limit changes to the intended resource and verify outcomes.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Start with one low-risk HPA backup today, document the steps, and ensure your team knows how to restore it.