Kubernetes Horizontal Pod Autoscaler Security Hardening: Practical Examples and Pitfalls

Intro

Securing the Kubernetes Horizontal Pod Autoscaler (HPA) is often overlooked because it seems like a purely operational component. Yet a misconfigured HPA can expose metrics, leak secrets through its configuration, or allow privilege escalation via overly broad RBAC rules. This guide walks through practical hardening steps, complete with commands, expected outputs, and recovery paths. It is written for developers, DevOps consultants, and technical startup teams who need to move from observing a problem to a verified, secure state.

Every recommendation is version-scoped, observable, and reversible where Kubernetes permits. We will start with environment inventory, then configure HPA securely, verify its behavior, prepare for failures, and close with an operations checklist and common pitfalls.

Version and Environment Inventory

Before changing anything, know what you have. Determine your Kubernetes version and the HPA API version in use. HPA has evolved across API versions: autoscaling/v1 supports only CPU metrics, while autoscaling/v2 and v2beta2 support custom and external metrics. Knowing the version avoids incompatible manifest errors.

First, check the cluster version:

kubectl version --short

Expected output resembles:

Client Version: v1.28.2
Server Version: v1.28.2

List the API resources available for autoscaling:

kubectl api-versions | grep autoscaling

You should see lines like:

autoscaling/v1
autoscaling/v2
autoscaling/v2beta2

If only v1 is listed, you cannot use memory or custom metrics in HPA manifests without upgrading or enabling feature gates.

Next, inspect existing HPA objects in all namespaces:

kubectl get hpa --all-namespaces

Example output:

NAMESPACE     NAME       REFERENCE             TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
default       web-hpa    Deployment/web        50%/80%         2         10        3          5d

Read the current state of one HPA without altering it:

kubectl describe hpa web-hpa -n default

Look for the Metrics: section to see which metrics are being used. If it lists something like resource cpu on pods, you are using the v1-style CPU-only scaling. If you see resource memory or external metric, v2 is active.

Also verify the HPA controller pod is running in kube-system:

kubectl get pods -n kube-system | grep hpa

Typical output:

kube-controller-manager-...   1/1     Running   0          10d

If the controller manager is not running, HPA decisions will not be applied. Check its logs:

kubectl logs -n kube-system kube-controller-manager-<pod-name> | grep -i hpa

Record all outputs with timestamps for later comparison. Never store secrets in plain text; use a secure notes system or environment-specific secret manager.

Safe Configuration Path

Once you have an inventory, harden the HPA itself. The primary attack surface is RBAC permissions and the metrics source. A least-privilege approach means the HPA controller should have only the permissions needed to scale resources, and your application's HPA should reference only allowed metrics.

RBAC for HPA Controller

By default, the kube-controller-manager uses a privileged service account. In managed clusters (EKS, GKE, AKS), this is abstracted, but in self-managed clusters, verify its role bindings. List the ClusterRoleBindings for the controller manager:

kubectl get clusterrolebinding -o wide | grep system:controller

Look for system:controller:horizontal-pod-autoscaler. If it is bound to a custom service account, inspect its permissions:

kubectl describe clusterrole system:controller:horizontal-pod-autoscaler

The role should allow get, list, watch on pods, replicationcontrollers, and deployments, and update on replicationcontrollers/scale and deployments/scale. If it grants broader permissions like on , tighten it immediately with a custom role.

Example custom role (hpa-controller-role.yaml):

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: restricted-hpa-controller
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources: ["deployments/scale"]
  verbs: ["get", "update"]
- apiGroups: [""]
  resources: ["replicationcontrollers/scale"]
  verbs: ["get", "update"]

Apply it and bind it to the controller's service account. In self-managed clusters, you may need to modify the controller manager manifest; in managed clusters, use the cloud provider's IAM roles if possible.

HPA Manifest Hardening

Your HPA manifest should avoid referring to external metric servers that could be compromised. If using custom metrics, ensure the metrics adapter has authentication and TLS. A typical secure HPA manifest for v2 with CPU and memory looks like:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-hpa
  namespace: default
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Resource
    resource:
      name: memory
      target:
        type: AverageValue
        averageValue: 500Mi

Do not embed secrets directly in HPA manifests. If using external metrics that require authentication, reference a Secret object rather than hardcoding tokens. For example, if your custom metric provider needs an API key, store it in a Secret and configure the adapter to read from it, not from the HPA manifest.

Implementation Sequence

Always test in a non-production namespace first. Create a mock deployment and HPA, then verify scaling behavior with a small load test. Example:

kubectl create namespace hpa-test
kubectl apply -f web-deployment.yaml -n hpa-test
kubectl apply -f web-hpa.yaml -n hpa-test

Check the HPA status:

kubectl get hpa -n hpa-test -w

Observe the replicas change as you generate load. If you see errors like unable to get metrics, check the metrics server availability:

kubectl get --raw /apis/metrics.k8s.io/v1beta1

A successful response returns JSON with metrics. If it times out, the metrics server is not installed or not reachable.

Once verified, move to production with a phased rollout.

Verification and Diagnostics

After applying hardening changes, verify that the HPA behaves as expected and that no security regressions occur.

Check HPA Status and Events

kubectl describe hpa web-hpa -n default

Look for Events at the bottom. Successful scaling events look like:

Normal  SuccessfulRescale  5m   horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target

If you see warnings like FailedGetResourceMetric or unable to fetch metrics, investigate the metrics pipeline.

Test RBAC Restrictions

Create a temporary service account with the exact permissions you intend for a user who should only read HPA objects. Then attempt to scale a deployment using that identity.

kubectl auth can-i update deployments/scale --as=system:serviceaccount:default:readonly-user -n default

Expected output:

no

If it returns yes, your RBAC is too permissive. Review role bindings.

Diagnose Metric Collection

If using the Kubernetes metrics server, ensure its pod is running:

kubectl get pods -n kube-system | grep metrics-server

Check its logs:

kubectl logs -n kube-system metrics-server-<pod>

Common issues include TLS certificate problems when using self-signed certs. In that case, configure the metrics server with --kubelet-insecure-tls only for non-production; in production, use proper certificates.

Verify Secrets Are Not Exposed

If your HPA references a Secret indirectly (e.g., for external metrics), confirm that the HPA object does not reveal the secret value:

kubectl get hpa web-hpa -o yaml | grep -i secret

The output should show a reference like secretKeyRef with a name and key, not the actual secret content. If you see the secret data in the manifest, remove it and use a Secret object.

Failure Modes and Recovery

Even with careful configuration, failures happen. Being prepared with recovery steps minimizes downtime and security exposure.

Common Failure Modes

1. Metrics server unavailable: HPA cannot get metrics, so it stops scaling. Symptoms: kubectl get hpa shows unknown in the TARGETS column, and events show unable to fetch metrics. Recovery: install or restart the metrics server, check network policies that might block the controller manager's access to the metrics API.

1. RBAC too restrictive: After tightening permissions, the HPA controller might lose the ability to update scales, causing scaling to fail with forbidden errors. Recovery: use kubectl auth can-i to test permissions, adjust the ClusterRole incrementally, and roll back if necessary.

1. Misconfigured metrics: If the HPA uses a custom metric that is not reported by any pod, the current replicas may be scaled down to minReplicas or scaled up to maxReplicas unpredictably. Recovery: verify metric values with kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1/... and check pod labels. Remove the misconfigured metric from the HPA until the metric provider is fixed.

1. Secret leakage in manifests: Accidentally committing a manifest with a hardcoded token exposes it in version control. Recovery: rotate the secret immediately, revoke the token, and replace the manifest with a Secret reference. Scan your repositories for similar leaks.

Recovery Verification

After any recovery, confirm the cluster is stable:

kubectl get hpa --all-namespaces
kubectl get events --sort-by='.metadata.creationTimestamp' | tail -20

Check that the HPA controller pod is running and no CrashLoopBackOff states exist. Run a load test again to see scaling resume.

Operations Checklist

Use this checklist before and after HPA security changes to prevent oversights.

• [ ] Confirm Kubernetes version and HPA API version supported (kubectl version, kubectl api-versions).
• [ ] Inventory all HPA objects and their metrics (kubectl get hpa --all-namespaces -o yaml).
• [ ] Review RBAC for HPA controller and any service accounts used by metric adapters (kubectl describe clusterrole ...).
• [ ] Ensure HPA manifests do not contain secrets; use Secret references.
• [ ] Verify metrics server is running and reachable from the controller (kubectl get pods -n kube-system | grep metrics-server, kubectl get --raw /apis/metrics.k8s.io/v1beta1).
• [ ] Test least-privilege access with kubectl auth can-i --as=<test-account>.
• [ ] Apply changes in a test namespace first and monitor scaling events.
• [ ] Record the exact commands and outputs before and after changes for audit.
• [ ] Define rollback procedure: save previous HPA and RBAC manifests in version control.
• [ ] After production rollout, verify scaling works under load.
• [ ] Schedule a monthly review of HPA security posture, including RBAC drift and metric source integrity.

Common Pitfalls and How to Avoid Them

Several recurring mistakes undermine HPA security. Here are the most common ones and how to prevent or recover from them.

Pitfall 1: Using Wildcards in RBAC for HPA

Why it happens: Developers want quick permission grants and use verbs: [""] or resources: [""] to avoid debugging permission errors.

How to avoid: Always define specific verbs and resources. Use kubectl auth can-i --list to see what a role actually grants, then trim. Test with a simulation of the workload.

Recovery: Audit existing ClusterRoles and Roles for wildcards related to autoscaling. Replace them with explicit rules.

Pitfall 2: Hardcoding Metrics Server Endpoints Without TLS

Why it happens: In development clusters, TLS is often skipped for convenience, and the configuration later gets copied to production.

How to avoid: Enforce TLS from the start. For metrics server, use signed certificates. For custom metrics adapters, require authentication and encryption.

Recovery: Update the metrics server deployment to use TLS, rotate any exposed keys, and test connectivity.

Pitfall 3: Ignoring HPA Events and Logs

Why it happens: Teams only look at HPA when scaling stops working; they do not monitor for security-relevant events like permission denials.

How to avoid: Set up alerts on HPA-related events in the namespace. Use tools like kubewatch or Prometheus alerts on kube_hpa_status_condition metrics.

Recovery: When an alert fires, immediately check kubectl describe hpa and controller logs to diagnose. Document the fix.

Pitfall 4: Allowing HPA to Scale Based on Untrusted Metrics

Why it happens: Custom metrics can come from user-controlled endpoints; if those endpoints are compromised, an attacker can force massive scaling, causing denial of service or resource exhaustion.

How to avoid: Validate metric sources, limit which metrics an HPA can use via admission policies, and set maxReplicas to a reasonable upper bound. A sudden spike to maxReplicas should trigger an alert.

Recovery: Temporarily delete the HPA or set minReplicas and maxReplicas to a safe range. Investigate the metric source.

Pitfall 5: Not Version-Controlling HPA and RBAC Manifests

Why it happens: Changes are made ad hoc with kubectl edit and lost.

How to avoid: Store all HPA and RBAC definitions in Git. Use GitOps tools like Argo CD or Flux to manage them. Every change gets reviewed.

Recovery: If a manual change broke something, revert to the last known good manifest from Git and redeploy.

Conclusion

Hardening the Kubernetes Horizontal Pod Autoscaler requires deliberate steps: know your environment, restrict permissions, validate configuration, monitor failures, and enforce operational discipline. Each recommendation in this guide is concrete and testable; the commands provided will show you current state and help you verify that your security measures work.

Start with one low-risk verification, such as checking RBAC permissions with kubectl auth can-i, and record the result. Then move to securing your HPA manifests and monitoring events. By making failure visible and protecting sensitive values, you ensure that your autoscaling remains both functional and secure. Revisit these practices monthly, or whenever you change Kubernetes versions or metric providers, to keep your cluster resilient against evolving threats.