## Intro Kubernetes Horizontal Pod Autoscaler (HPA) dynamically adjusts the number of pod replicas in a Deployment, StatefulSet, or other workload based on observed CPU utilization, memory usage, or custom metrics. Automating HPA configuration through CI/CD pipelines ensures that autoscaling behavior is versioned, tested, and consistently applied across environments. This guide walks through practical steps for automating HPA manifests, validating them in CI, deploying to Kubernetes, and recovering from common failures. The target audience includes DevOps engineers, platform teams, and developers managing production Kubernetes clusters. Before making any changes, understand your current environment: Kubernetes version, HPA API version, metrics server availability, and existing HPA objects. Always observe the current state, make small, reversible changes, and verify the result before proceeding. The examples use `kubectl`, `jq`, and common CI tools like GitHub Actions or GitLab CI, but the concepts apply to any pipeline. ## Version and Environment Inventory First, determine your Kubernetes cluster version and the HPA API version you should use. HPA is available in `autoscaling/v1` (CPU only) and `autoscaling/v2` (supports custom and multiple metrics). Most modern clusters use `autoscaling/v2`. Run the following read-only commands to gather information: ```bash kubectl version --short kubectl api-versions | grep autoscaling kubectl get hpa --all-namespaces kubectl get metrics-server --all-namespaces ``` Expected output for `kubectl api-versions` includes `autoscaling/v1` and `autoscaling/v2beta2` or `autoscaling/v2`. If metrics-server is not running, HPA cannot scale on CPU or memory. Install it if needed, but note that HPA based on custom metrics may require Prometheus or other adapters. Check current HPA objects with their targets and current metrics: ```bash kubectl describe hpa -n ``` Record the output. Identify the deployment controlled by HPA and its current replica count: ```bash kubectl get deployment -n -o wide ``` Understanding the baseline allows you to detect unintended changes after applying new HPA configurations. Keep a local text file with these outputs for comparison later. Use timestamps: ```bash date > hpa_baseline.txt kubectl get hpa -n -o yaml >> hpa_baseline.txt kubectl get deployment -n -o jsonpath='{.spec.replicas}' >> hpa_baseline.txt ``` Now you have a snapshot. Only proceed to modify HPA after confirming prerequisites: cluster supports the HPA version you plan, metrics source works, and you have permission to create/update HPA objects. ## Safe Configuration Path Think of the HPA configuration as code stored in your Git repository. The safe path is to manage HPA manifests under version control, review changes via pull requests, and apply them through CI/CD pipelines. Start with an existing HPA manifest or create a new one. Here is an example using `autoscaling/v2` based on CPU and memory: ```yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: webapp-hpa namespace: production spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: webapp minReplicas: 2 maxReplicas: 10 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 60 - type: Resource resource: name: memory target: type: Utilization averageUtilization: 70 behavior: scaleDown: stabilizationWindowSeconds: 300 policies: - type: Percent value: 50 periodSeconds: 60 scaleUp: stabilizationWindowSeconds: 60 policies: - type: Pods value: 2 periodSeconds: 60 - type: Percent value: 50 periodSeconds: 60 selectPolicy: Max ``` This HPA ensures the webapp deployment has between 2 and 10 replicas, targeting 60% CPU and 70% memory utilization. It scales up quickly (up to 2 pods or 50% of current replicas every minute) and scales down more conservatively (50% every minute after a 5-minute stabilization window). To safely test, first apply the HPA to a staging namespace or cluster. Use `kubectl apply --dry-run=client` to validate syntax without applying: ```bash kubectl apply -f webapp-hpa.yaml --dry-run=client ``` If no errors, apply to staging: ```bash kubectl apply -f webapp-hpa.yaml -n staging ``` Then monitor the HPA status: ```bash kubectl get hpa webapp-hpa -n staging --watch ``` Observe the TARGETS column eventually showing `50%/60%, 60%/70%` (current usage / target). If metrics are not available, it may show ``. Investigate metrics-server or adapter issues. Only after staging behavior matches expectations promote to production via CI/CD. ## CI/CD Pipeline Integration Automate the HPA deployment using a CI/CD pipeline. The pipeline should: - Validate HPA YAML syntax - Run dry-run against target cluster - Apply HPA to cluster - Verify HPA status and metrics - Rollback on failure Example GitHub Actions workflow: ```yaml name: Deploy HPA on: push: paths: - 'hpa/**' branches: [ main ] jobs: deploy-hpa: runs-on: ubuntu-latest environment: production steps: - name: Checkout code uses: actions/checkout@v4 - name: Setup kubectl uses: azure/setup-kubectl@v3 with: version: 'v1.28.0' - name: Set up kubeconfig run: | mkdir -p $HOME/.kube echo "${{ secrets.KUBE_CONFIG }}" > $HOME/.kube/config - name: Validate HPA manifests run: | for file in hpa/*.yaml; do kubectl apply -f $file --dry-run=client if [ $? -ne 0 ]; then echo "Validation failed for $file" exit 1 fi done - name: Apply HPA run: kubectl apply -f hpa/ - name: Verify HPA status run: | sleep 30 kubectl get hpa -n production -o yaml | tee /tmp/hpa_status.yaml # Check that HPA has metrics and is not in error state if kubectl get hpa -n production -o jsonpath='{.items[?(@.metadata.name=="webapp-hpa")].status.conditions[?(@.type=="AbleToScale")].status}' | grep -q False; then echo "HPA unable to scale; rolling back" kubectl rollout undo deployment/webapp -n production exit 1 fi ``` This pipeline applies HPA, waits 30 seconds, and checks a condition. If scaling is not possible, it rolls back the deployment (though HPA rollback is separate; see Failure Modes). Better verification: check that HPA has current metrics within a reasonable time. For example: ```bash kubectl get hpa webapp-hpa -n production -o jsonpath='{.status.currentMetrics}' | jq . ``` Expect a non-empty list of metric values. ## Verification and Diagnostics After deployment, verify the HPA is functioning correctly. Key commands: ```bash kubectl describe hpa webapp-hpa -n production ``` Look for events indicating successful scaling or errors. Common issues: - **Metrics not available**: HPA status shows `` for targets. Cause: metrics-server not running or no traffic. Fix: install/verify metrics-server, generate load. - **HPA cannot fetch metrics for resource**: Check if resource requests are set on pods. HPA uses requests for utilization percentage. Without requests, CPU utilization cannot be computed. - **Scale up not happening**: Check HPA's `status.conditions` for `ScalingActive` or `ScalingLimited`. Use `kubectl get hpa -o yaml` to inspect. - **Scale down too slow**: Adjust `behavior.scaleDown` policies. Generate test load to trigger scaling. For example, using a simple load generator: ```bash kubectl run -i --tty load-generator --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://webapp-service; done" ``` Then watch HPA: ```bash kubectl get hpa webapp-hpa -n production --watch ``` You should see current metrics exceed targets and replica count increase. Wait and then stop load; replica count should eventually decrease after stabilization window. Use `kubectl top pods` to see actual resource usage: ```bash kubectl top pods -n production -l app=webapp ``` If metrics-server is not installed, `kubectl top` fails. ## Failure Modes and Recovery Common HPA failures and how to recover: 1. **Invalid HPA configuration**: Syntax errors or referencing a non-existent deployment. Recovery: run `kubectl apply --dry-run=client` before applying. If already applied, fix manifest and re-apply. 2. **HPA scales to zero or max unexpectedly**: Due to wrong metric thresholds or missing resource requests. Recovery: adjust thresholds, set appropriate min/max, and verify resource requests. 3. **HPA not scaling due to metrics-server outage**: Recovery: fix metrics-server, then HPA resumes normal operation. 4. **Rolling back HPA changes**: Since HPA is not part of Deployment rollout, `kubectl rollout undo` does not revert HPA. Instead, maintain HPA manifests in Git and revert the commit/PR. Then re-apply the previous HPA config. 5. **HPA fights with manual scaling**: If you manually set `kubectl scale deployment`, HPA may override it. Avoid manual scaling or temporarily suspend HPA by setting `minReplicas` and `maxReplicas` to the desired count. Recovery verification: after fixing, check `kubectl get hpa` shows expected targets, and `kubectl describe hpa` shows no error events. Also confirm pod count matches expectations. ## Common Pitfalls Here are pitfalls many teams encounter: - **Not setting resource requests on pods**: HPA cannot compute utilization percentages. Always set requests for containers. If not set, HPA reports unknown targets. - **Using CPU only**: Memory-based or custom metrics may be needed. Consider using multiple metrics. - **Ignoring stabilization windows**: Without stabilization, HPA flapping can cause rapid scale up/down. Use `behavior` to stabilize. - **Hardcoding namespace in manifests**: Use templating (Helm, Kustomize) to parameterize namespace and other values. - **No CI/CD validation**: Manually applying HPA leads to drift. Integrate validation and deployment in pipeline. - **Forgetting to check cluster version**: HPA `v2` features may not be available in older Kubernetes. Ensure cluster supports desired API version. - **Not monitoring HPA events**: Set up alerts for HPA failure conditions. To avoid: include HPA manifests in code review, automate tests, and monitor metrics. ## Operations Checklist Use this checklist for each HPA deployment or update. Assign a single owner per item and track completion. | # | Task | Command / Tool | Expected Result | Owner | Review Cadence | |---|------|----------------|-----------------|-------|----------------| | 1 | Check Kubernetes and HPA API versions | `kubectl version`, `kubectl api-versions` | Cluster >=1.23, `autoscaling/v2` available | Platform Engineer (e.g., Alex Chen) | Quarterly | | 2 | Verify metrics-server running | `kubectl get pods -n kube-system \| grep metrics-server` | metrics-server pod Running | Platform Engineer | Monthly | | 3 | Validate HPA manifest syntax | `kubectl apply -f hpa.yaml --dry-run=client` | No error | DevOps Lead (e.g., Maria Garcia) | Every PR | | 4 | Apply HPA to staging | `kubectl apply -f hpa.yaml -n staging` | HPA created | DevOps Lead | Every PR | | 5 | Check HPA metrics in staging | `kubectl describe hpa -n staging` | Current metrics show values | DevOps Lead | After apply | | 6 | Generate load test | `kubectl run load-generator ...` | HPA scales up | QA Engineer (e.g., John Smith) | Weekly | | 7 | Promote to production via pipeline | `git push` (CI/CD) | Pipeline succeeds | DevOps Lead | On release | | 8 | Verify production HPA | `kubectl get hpa -n production` | Targets not `` | DevOps Lead | Immediately after deploy | | 9 | Monitor HPA events/alerts | Prometheus/Grafana alerts | No critical alerts | SRE (e.g., Emily Johnson) | Continuous | | 10 | Document HPA changes in runbook | Team wiki | Runbook updated | Tech Writer (or DevOps) | After each change | Each item should be revisited as indicated. For example, the platform engineer reviews API version compatibility quarterly or when cluster is upgraded. ## Conclusion Automating Kubernetes HPA with CI/CD brings consistency and safety to autoscaling configuration. By following the practices outlined—version inventory, safe configuration paths, pipeline integration, verification, and recovery—you can avoid common pitfalls and maintain reliable autoscaling. Start with a low-risk verification: pick one existing HPA, record its current state, run a dry-run on a copy with a minor change, and observe the impact in a staging environment. As confidence grows, integrate into your pipeline and expand to other workloads. Remember: operational safety means observing before changing, limiting blast radius, using placeholders for secrets, verifying results, and documenting recovery steps before they are needed.