Automating Kubernetes Horizontal Pod Autoscaler with CI/CD: A Practical Guide

Intro

Kubernetes Horizontal Pod Autoscaler (HPA) dynamically adjusts the number of pod replicas in a Deployment, StatefulSet, or other workload based on observed CPU utilization, memory usage, or custom metrics. Automating HPA configuration through CI/CD pipelines ensures that autoscaling behavior is versioned, tested, and consistently applied across environments. This guide walks through practical steps for automating HPA manifests, validating them in CI, deploying to Kubernetes, and recovering from common failures. The target audience includes DevOps engineers, platform teams, and developers managing production Kubernetes clusters.

Before making any changes, understand your current environment: Kubernetes version, HPA API version, metrics server availability, and existing HPA objects. Always observe the current state, make small, reversible changes, and verify the result before proceeding. The examples use kubectl, jq, and common CI tools like GitHub Actions or GitLab CI, but the concepts apply to any pipeline.

Version and Environment Inventory

First, determine your Kubernetes cluster version and the HPA API version you should use. HPA is available in autoscaling/v1 (CPU only) and autoscaling/v2 (supports custom and multiple metrics). Most modern clusters use autoscaling/v2. Run the following read-only commands to gather information:

kubectl version --short
kubectl api-versions | grep autoscaling
kubectl get hpa --all-namespaces
kubectl get metrics-server --all-namespaces

Expected output for kubectl api-versions includes autoscaling/v1 and autoscaling/v2beta2 or autoscaling/v2. If metrics-server is not running, HPA cannot scale on CPU or memory. Install it if needed, but note that HPA based on custom metrics may require Prometheus or other adapters.

Check current HPA objects with their targets and current metrics:

kubectl describe hpa <hpa-name> -n <namespace>

Record the output. Identify the deployment controlled by HPA and its current replica count:

kubectl get deployment <deployment-name> -n <namespace> -o wide

Understanding the baseline allows you to detect unintended changes after applying new HPA configurations. Keep a local text file with these outputs for comparison later. Use timestamps:

date > hpa_baseline.txt
kubectl get hpa -n <namespace> -o yaml >> hpa_baseline.txt
kubectl get deployment <deployment-name> -n <namespace> -o jsonpath='{.spec.replicas}' >> hpa_baseline.txt

Now you have a snapshot. Only proceed to modify HPA after confirming prerequisites: cluster supports the HPA version you plan, metrics source works, and you have permission to create/update HPA objects.

Safe Configuration Path

Think of the HPA configuration as code stored in your Git repository. The safe path is to manage HPA manifests under version control, review changes via pull requests, and apply them through CI/CD pipelines.

Start with an existing HPA manifest or create a new one. Here is an example using autoscaling/v2 based on CPU and memory:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: webapp-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: webapp
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60
      - type: Percent
        value: 50
        periodSeconds: 60
      selectPolicy: Max

This HPA ensures the webapp deployment has between 2 and 10 replicas, targeting 60% CPU and 70% memory utilization. It scales up quickly (up to 2 pods or 50% of current replicas every minute) and scales down more conservatively (50% every minute after a 5-minute stabilization window).

To safely test, first apply the HPA to a staging namespace or cluster. Use kubectl apply --dry-run=client to validate syntax without applying:

kubectl apply -f webapp-hpa.yaml --dry-run=client

If no errors, apply to staging:

kubectl apply -f webapp-hpa.yaml -n staging

Then monitor the HPA status:

kubectl get hpa webapp-hpa -n staging --watch

Observe the TARGETS column eventually showing 50%/60%, 60%/70% (current usage / target). If metrics are not available, it may show <unknown>. Investigate metrics-server or adapter issues.

Only after staging behavior matches expectations promote to production via CI/CD.

CI/CD Pipeline Integration

Automate the HPA deployment using a CI/CD pipeline. The pipeline should:

• Validate HPA YAML syntax
• Run dry-run against target cluster
• Apply HPA to cluster
• Verify HPA status and metrics
• Rollback on failure

Example GitHub Actions workflow:

name: Deploy HPA

on:
  push:
    paths:
      - 'hpa/**'
    branches: [ main ]

jobs:
  deploy-hpa:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Checkout code
        uses: actions/checkout@v4
      - name: Setup kubectl
        uses: azure/setup-kubectl@v3
        with:
          version: 'v1.28.0'
      - name: Set up kubeconfig
        run: |
          mkdir -p $HOME/.kube
          echo "${{ secrets.KUBE_CONFIG }}" > $HOME/.kube/config
      - name: Validate HPA manifests
        run: |
          for file in hpa/*.yaml; do
            kubectl apply -f $file --dry-run=client
            if [ $? -ne 0 ]; then
              echo "Validation failed for $file"
              exit 1
            fi
          done
      - name: Apply HPA
        run: kubectl apply -f hpa/
      - name: Verify HPA status
        run: |
          sleep 30
          kubectl get hpa -n production -o yaml | tee /tmp/hpa_status.yaml
          # Check that HPA has metrics and is not in error state
          if kubectl get hpa -n production -o jsonpath='{.items[?(@.metadata.name=="webapp-hpa")].status.conditions[?(@.type=="AbleToScale")].status}' | grep -q False; then
            echo "HPA unable to scale; rolling back"
            kubectl rollout undo deployment/webapp -n production
            exit 1
          fi

This pipeline applies HPA, waits 30 seconds, and checks a condition. If scaling is not possible, it rolls back the deployment (though HPA rollback is separate; see Failure Modes).

Better verification: check that HPA has current metrics within a reasonable time. For example:

kubectl get hpa webapp-hpa -n production -o jsonpath='{.status.currentMetrics}' | jq .

Expect a non-empty list of metric values.

Verification and Diagnostics

After deployment, verify the HPA is functioning correctly. Key commands:

kubectl describe hpa webapp-hpa -n production

Look for events indicating successful scaling or errors. Common issues:

• Metrics not available: HPA status shows <unknown> for targets. Cause: metrics-server not running or no traffic. Fix: install/verify metrics-server, generate load.
• HPA cannot fetch metrics for resource: Check if resource requests are set on pods. HPA uses requests for utilization percentage. Without requests, CPU utilization cannot be computed.
• Scale up not happening: Check HPA's status.conditions for ScalingActive or ScalingLimited. Use kubectl get hpa -o yaml to inspect.
• Scale down too slow: Adjust behavior.scaleDown policies.

Generate test load to trigger scaling. For example, using a simple load generator:

kubectl run -i --tty load-generator --rm --image=busybox --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://webapp-service; done"

Then watch HPA:

kubectl get hpa webapp-hpa -n production --watch

You should see current metrics exceed targets and replica count increase. Wait and then stop load; replica count should eventually decrease after stabilization window.

Use kubectl top pods to see actual resource usage:

kubectl top pods -n production -l app=webapp

If metrics-server is not installed, kubectl top fails.

Failure Modes and Recovery

Common HPA failures and how to recover:

1. Invalid HPA configuration: Syntax errors or referencing a non-existent deployment. Recovery: run kubectl apply --dry-run=client before applying. If already applied, fix manifest and re-apply.
2. HPA scales to zero or max unexpectedly: Due to wrong metric thresholds or missing resource requests. Recovery: adjust thresholds, set appropriate min/max, and verify resource requests.
3. HPA not scaling due to metrics-server outage: Recovery: fix metrics-server, then HPA resumes normal operation.
4. Rolling back HPA changes: Since HPA is not part of Deployment rollout, kubectl rollout undo does not revert HPA. Instead, maintain HPA manifests in Git and revert the commit/PR. Then re-apply the previous HPA config.
5. HPA fights with manual scaling: If you manually set kubectl scale deployment, HPA may override it. Avoid manual scaling or temporarily suspend HPA by setting minReplicas and maxReplicas to the desired count.

Recovery verification: after fixing, check kubectl get hpa shows expected targets, and kubectl describe hpa shows no error events. Also confirm pod count matches expectations.

Common Pitfalls

Here are pitfalls many teams encounter:

• Not setting resource requests on pods: HPA cannot compute utilization percentages. Always set requests for containers. If not set, HPA reports unknown targets.
• Using CPU only: Memory-based or custom metrics may be needed. Consider using multiple metrics.
• Ignoring stabilization windows: Without stabilization, HPA flapping can cause rapid scale up/down. Use behavior to stabilize.
• Hardcoding namespace in manifests: Use templating (Helm, Kustomize) to parameterize namespace and other values.
• No CI/CD validation: Manually applying HPA leads to drift. Integrate validation and deployment in pipeline.
• Forgetting to check cluster version: HPA v2 features may not be available in older Kubernetes. Ensure cluster supports desired API version.
• Not monitoring HPA events: Set up alerts for HPA failure conditions.

To avoid: include HPA manifests in code review, automate tests, and monitor metrics.

Operations Checklist

Use this checklist for each HPA deployment or update. Assign a single owner per item and track completion.

Each item should be revisited as indicated. For example, the platform engineer reviews API version compatibility quarterly or when cluster is upgraded.

Conclusion

Automating Kubernetes HPA with CI/CD brings consistency and safety to autoscaling configuration. By following the practices outlined—version inventory, safe configuration paths, pipeline integration, verification, and recovery—you can avoid common pitfalls and maintain reliable autoscaling.

Start with a low-risk verification: pick one existing HPA, record its current state, run a dry-run on a copy with a minor change, and observe the impact in a staging environment. As confidence grows, integrate into your pipeline and expand to other workloads.

Remember: operational safety means observing before changing, limiting blast radius, using placeholders for secrets, verifying results, and documenting recovery steps before they are needed.