## Intro Kubernetes Horizontal Pod Autoscaler (HPA) backup and restore is often treated as an afterthought until a misconfiguration or accidental deletion forces a manual rebuild. A structured approach turns an observed problem into a verified result. This guide focuses on practical, repeatable procedures for backing up and restoring HPA configurations across clusters and namespaces. We cover version and environment inventory, safe configuration paths, verification and diagnostics, failure modes and recovery, and an operations checklist. Each section includes concrete commands, expected outputs, and real-world examples. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify outcomes, and document recovery before an incident occurs. This guide is for developers, DevOps consultants, and technical startup teams who manage Kubernetes workloads and want to protect autoscaling configurations as part of their disaster recovery strategy. ## Version and Environment Inventory Before backing up or restoring HPAs, you need a clear picture of your cluster version, HPA API versions, and current HPA objects. Start with read-only observations. ### Identify Kubernetes and HPA API Versions Run the following commands: ```bash kubectl version --short kubectl api-versions | grep autoscaling ``` Expected output includes lines like: ``` autoscaling/v1 autoscaling/v2beta2 autoscaling/v2 ``` As of Kubernetes 1.23, `autoscaling/v2` is stable and supports metric-based scaling. If your cluster shows only `autoscaling/v1`, you are limited to CPU-based scaling. This affects backup format: v2 HPAs may include multiple metrics and behavior configurations not representable in v1. ### Capture Current HPA Inventory List all HPAs across all namespaces: ```bash kubectl get hpa --all-namespaces -o wide ``` Sample output: ``` NAMESPACE NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE prod web-hpa Deployment/web cpu: 50%/80% 2 10 4 12d dev api-hpa Deployment/api memory: 60% 1 5 2 3d ``` Record this output along with a timestamp. For a complete backup, export each HPA as YAML: ```bash kubectl get hpa web-hpa -n prod -o yaml > web-hpa-backup-$(date +%Y%m%d).yaml ``` Repeat for all HPAs. Store these files in version control, not just locally. ### Prerequisites for Safe HPA Operations - `kubectl` version compatible with your cluster (within one minor version). - Permissions to get and list HPAs in target namespaces (read-only) and to create/update for restore. - A backup location (git repository, object storage) that is accessible during a disaster. - A clear understanding of dependent Deployment or Pod resource names, because HPA references them by name. ### Smallest Justified Change Before any backup or restore attempt, verify the target HPA exists and is functioning normally. Use: ```bash kubectl describe hpa web-hpa -n prod ``` Look for events like: ``` Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulRescale 10m horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target ``` If no recent events appear, the HPA may not be actively scaling. Compare with `kubectl get hpa` output to see current metrics. ## Safe Configuration Path A safe configuration path means making incremental, reversible changes to HPA settings, backed by a previous known-good version. ### Backup Before Any Change Always export the current HPA manifest before editing it. For example: ```bash kubectl get hpa web-hpa -n prod -o yaml > web-hpa-2025-03-15.yaml ``` This file is your rollback point. ### Example: Changing Min and Max Replicas Suppose you want to increase `maxReplicas` from 10 to 15 for `web-hpa` in `prod`. Here's a minimal patch: ```bash kubectl patch hpa web-hpa -n prod --patch '{"spec":{"maxReplicas":15}}' ``` Verify the change: ```bash kubectl get hpa web-hpa -n prod -o yaml | grep maxReplicas ``` Expected output: ```yaml maxReplicas: 15 ``` Check that the HPA still matches the deployment's pods: ```bash kubectl get pods -l app=web -n prod ``` If the HPA tries to scale above the cluster's capacity, you will see events in `kubectl describe hpa` indicating failed rescale due to insufficient resources. ### Using kubectl Apply with a Modified Manifest Alternatively, edit the exported YAML file, change the desired field, and apply: ```bash kubectl apply -f web-hpa-updated.yaml ``` Always keep the original backup file intact. ### Maintaining a Versioned History Store HPA manifests in git with meaningful commit messages. For example: ```bash git add web-hpa-*.yaml git commit -m "Backup HPA web-hpa before maxReplicas change to 15" ``` This gives you an auditable trail and easy rollback using `kubectl apply -f` with the previous version. ## Verification and Diagnostics After any HPA change or restore, verify that the HPA is working as expected. Do not assume success from the command exit code. ### Check HPA Status and Events Run: ```bash kubectl describe hpa web-hpa -n prod ``` Look for the `AbleToScale` and `ScalingActive` conditions: ```yaml Conditions: Type Status Reason Message ---- ------ ------ ------- AbleToScale True ReadyForNewScale recommended size matches current size ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization ``` If `ScalingActive` is `False`, check the reason. Common issues include missing metrics or incorrect resource requests. ### Validate Metric Sources For CPU-based HPA, ensure your pods have CPU requests set: ```bash kubectl get deployment web -n prod -o jsonpath='{.spec.template.spec.containers[0].resources}' ``` Output should include something like: ```json {"requests":{"cpu":"100m"},"limits":{"cpu":"500m"}} ``` For memory-based HPA, verify memory requests. For custom metrics, ensure the metrics server or Prometheus adapter is running: ```bash kubectl get pods -n kube-system | grep metrics-server ``` ### Simulate Load to Trigger Scaling To verify scaling behavior in a controlled way, temporarily increase load on the deployment. For example, use a load generation tool like `hey` or `ab` against a service endpoint. Then watch the HPA: ```bash kubectl get hpa web-hpa -n prod --watch ``` After a few minutes, you should see `REPLICAS` increase. Then stop the load and observe scale-down. ### Compare Current State with Backup After restoring an HPA from backup, diff the current manifest with the backup file: ```bash diff <(kubectl get hpa web-hpa -n prod -o yaml) web-hpa-backup.yaml ``` Any differences should be intentional. This is a crucial validation step. ## Failure Modes and Recovery HPAs can fail in several ways. Here are common failure modes, how to detect them, and how to recover using backups. ### Failure Mode 1: Accidental Deletion of an HPA If someone runs `kubectl delete hpa web-hpa -n prod`, the Deployment will continue running with its current replica count but will no longer scale automatically. Detection: ```bash kubectl get hpa -n prod ``` Output lacks `web-hpa`. Recovery: ```bash kubectl apply -f web-hpa-backup-2025-03-15.yaml ``` Verify: ```bash kubectl get hpa web-hpa -n prod ``` ### Failure Mode 2: HPA Targets a Nonexistent Deployment If the referenced deployment is deleted or renamed, the HPA will report a scaling error. Detection: ```bash kubectl describe hpa web-hpa -n prod | grep -A5 Events ``` Look for messages like: ``` Warning FailedGetScale horizontal-pod-autoscaler deployments/scale.apps "web" not found ``` Recovery: - Restore the Deployment if it was deleted. - Or update the HPA's `scaleTargetRef` to the correct name using `kubectl edit hpa` or a patch. ### Failure Mode 3: Invalid Metric Configuration If you change the metric type or incorrectly specify a resource name, the HPA may fail to fetch metrics. Detection: ```bash kubectl get hpa web-hpa -n prod -o yaml | grep -A10 status ``` Look for conditions like: ```yaml conditions: - type: ScalingActive status: "False" reason: FailedGetResourceMetric message: missing request for cpu ``` Recovery: - Ensure the pods have CPU or memory requests as required. - Or revert to the previous backup manifest that worked. ### Failure Mode 4: HPA Conflicts with Cluster Autoscaler or Resource Quotas If the cluster cannot provision new nodes, or a ResourceQuota limits pods, HPA scaling will fail. Detection: ```bash kubectl describe hpa web-hpa -n prod | tail -20 ``` Look for `FailedRescale` events: ``` Warning FailedRescale horizontal-pod-autoscaler New size: 6; reason: exceeded quota in namespace prod ``` Recovery: - Adjust ResourceQuota or node capacity. - Or temporarily reduce HPA `maxReplicas` to a feasible number using the backup as a reference. ### General Recovery Procedure 1. Identify the failure from HPA events and conditions. 2. Retrieve the last known-good backup from version control. 3. Apply the backup with `kubectl apply -f`. 4. Verify with `kubectl get hpa` and `kubectl describe hpa`. 5. Document the incident and the fix. ## Operations Checklist Use the following checklist before and after any HPA backup, restore, or modification. ### Before Change - [ ] Run `kubectl version --short` and note cluster version. - [ ] Run `kubectl api-versions | grep autoscaling` to confirm supported HPA API versions. - [ ] List all HPAs: `kubectl get hpa --all-namespaces -o wide` - [ ] Export current HPA YAML for the target namespace with a timestamp, e.g., `hpa-backup-prod-web-2025-03-15.yaml`. - [ ] Store the backup file in a version-controlled repository. - [ ] Check that the referenced Deployment exists and pods have resource requests if using resource metrics. - [ ] Record current replica count and metrics. ### During Change - [ ] Apply changes with `kubectl apply -f` or `kubectl patch`, never with an inline edit that bypasses file backup. - [ ] Use a single, scoped change at a time (e.g., only `maxReplicas`). - [ ] Note the exact command used and expected outcome. ### After Change or Restore - [ ] Run `kubectl get hpa -n ` to confirm object exists. - [ ] Run `kubectl describe hpa -n ` and verify conditions `AbleToScale=True` and `ScalingActive=True`. - [ ] Check for error events in the HPA description. - [ ] Diff the current manifest against the backup (if restoring). - [ ] Monitor for at least one scaling cycle (e.g., 10-15 minutes) to ensure HPA can scale up and down. - [ ] If any step fails, roll back to the previous backup manifest immediately. ### Example Worked Scenario Here is a complete backup and restore example for an HPA called `web-hpa` in namespace `prod`. 1. Backup: ```bash kubectl get hpa web-hpa -n prod -o yaml > web-hpa-2025-03-15.yaml git add web-hpa-2025-03-15.yaml && git commit -m "Backup HPA before change" ``` 2. Make a change: increase `minReplicas` from 2 to 3. ```bash kubectl patch hpa web-hpa -n prod --patch '{"spec":{"minReplicas":3}}' ``` 3. Verify immediate state: ```bash kubectl get hpa web-hpa -n prod -o yaml | grep minReplicas ``` Expected: `minReplicas: 3` 4. Simulate load and watch scaling: ```bash kubectl get hpa web-hpa -n prod --watch ``` 5. If something goes wrong, roll back: ```bash kubectl apply -f web-hpa-2025-03-15.yaml ``` 6. Verify rollback: ```bash kubectl get hpa web-hpa -n prod -o yaml | grep minReplicas ``` Expected: `minReplicas: 2` ## Conclusion Kubernetes HPA backup and restore is a critical part of cluster operations. By following a structured approach—inventorying versions, capturing HPA manifests, making small reversible changes, verifying with diagnostics, and having a clear recovery plan—you can prevent autoscaling outages and recover quickly from mistakes. Always store HPA backups in version control, not just on a local disk. Test restore procedures in a non-production namespace before an actual failure. Make failure visible by monitoring HPA events and conditions. Protect sensitive values by using placeholders and secrets rather than hardcoding. Limit changes to the intended resource and verify outcomes. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Start with one low-risk HPA backup today, document the steps, and ensure your team knows how to restore it.