Kubernetes Admission Controllers Security Hardening: Practical Examples and Operational Guide

Intro

Kubernetes Admission Controllers are a critical security boundary that validates and mutates requests before they are persisted. A misconfigured admission controller can allow privileged containers, insecure secrets, or non-compliant workloads into production. This guide provides a structured, practical approach to hardening admission controllers, moving from an observed problem to a verified result. It is written for developers, DevOps consultants, and technical startup teams who need to secure Kubernetes clusters without disrupting delivery.

We focus on the four core hardening areas: access control, secrets handling, permissions, and overall security posture. Every recommendation is tied to concrete commands, expected outputs, failure signals, and recovery decisions. The guiding principle is operational safety: observe before changing, limit blast radius, never expose secrets, verify each change, and document recovery paths before an incident occurs.

Throughout this guide, we assume a working Kubernetes cluster (version 1.25+ for most examples, though commands generally work on older versions), kubectl configured with appropriate access, and a pre-production namespace where you can safely test. Replace namespace: default with your own namespace as needed. Always run read-only commands first to capture the current state.

Version and Environment Inventory

Before any hardening, you must know exactly what admission controllers are active, which Kubernetes version you are running, and what configuration management system controls them. This inventory prevents accidental changes and ensures your commands are compatible with your cluster.

Step 1: Determine your Kubernetes version and API server details.

kubectl version --short

Expected output (example):

Client Version: v1.26.1
Kustomize Version: v4.5.7
Server Version: v1.26.1

If --short is deprecated, use kubectl version and look for the Server Version field.

Step 2: List currently enabled admission controllers.

The enabled admission controllers are set on the API server. You can often see them by inspecting the API server pod or your cluster's configuration. On a managed cluster (EKS, GKE, AKS), you may not have direct access to the API server flags, so consult your provider's documentation. On a self-managed cluster (kubeadm, kops, etc.), check the API server manifest:

# On a control plane node, if you have SSH access
sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep enable-admission-plugins

Example output:

- --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,ServiceAccount,DefaultStorageClass,ResourceQuota

If you do not have node access, you can infer some enabled controllers via kubectl api-versions or by testing specific behaviors. However, the most reliable method is to check the cluster configuration file used by your provisioning tool (e.g., kubeadm-config ConfigMap).

Step 3: Identify configuration management.

Determine whether admission controllers are configured via static pod manifests, kubeadm configuration, or a managed provider. This affects how you will make changes and roll back. For example, on kubeadm clusters, you can edit the /etc/kubernetes/manifests/kube-apiserver.yaml directly, but changes are overwritten on upgrades unless you also update the kubeadm-config. Managed clusters may require CLI flags or API settings.

Step 4: Capture current state and timestamps.

Before any change, record the current state of the API server and relevant resources. Use:

kubectl get pods -n kube-system -l component=kube-apiserver -o yaml > apiserver-backup-$(date +%Y%m%d%H%M%S).yaml

Also, check the cluster events for any recent issues:

kubectl get events --all-namespaces --sort-by=.metadata.creationTimestamp | tail -20

This establishes a baseline for troubleshooting if a change causes problems.

Step 5: Understand prerequisites and blast radius.

Each admission controller may require additional resources or configurations. For example, PodSecurityPolicy (deprecated in 1.21, removed in 1.25) requires defining policies and RBAC bindings. ResourceQuota requires creating quota objects. Changing an admission controller can affect all new workloads, so test in a non-production environment first.

Practical check: Before enabling a new admission controller, ensure its prerequisites are satisfied. For instance, if you plan to enable PodSecurity (the replacement for PodSecurityPolicy), you need to create Pod Security Admission labels on namespaces. Verify with:

kubectl get namespace my-namespace -o jsonpath='{.metadata.labels}' | jq

If the label pod-security.kubernetes.io/enforce is missing, the controller may not behave as expected.

Safe Configuration Path

The safe configuration path involves making the smallest possible change in a controlled manner, with clear rollback steps. Here we focus on two common hardening scenarios:

1. Enabling a recommended admission controller (e.g., NodeRestriction)
2. Tightening Pod Security via built-in Pod Security Admission (PSA)

Example 1: Enabling NodeRestriction Admission Controller

NodeRestriction ensures that kubelets can only modify their own Node and Pod objects, limiting the impact of a compromised node. It is enabled by default in many distributions, but verify and enable if missing.

Step 1: Check current state.

As described in the inventory, inspect the --enable-admission-plugins flag. If NodeRestriction is absent, proceed.

Step 2: Back up the API server manifest.

cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.backup

Step 3: Edit the manifest and add NodeRestriction to the list.

Using your preferred editor, add NodeRestriction to the --enable-admission-plugins flag, comma-separated. For example:

- --enable-admission-plugins=NodeRestriction,ServiceAccount,DefaultStorageClass,ResourceQuota

Step 4: Wait for the API server to restart.

The kubelet automatically restarts static pods when the manifest changes. Monitor the API server pod status:

kubectl get pods -n kube-system -l component=kube-apiserver -w

Wait until the new pod is Running and the old one is terminated.

Step 5: Verify the controller is active.

Indirectly verify by testing NodeRestriction behavior. For example, try to patch a Node object using the kubelet's credentials (if you have them). A less intrusive check is to look at the API server logs for a message indicating the plugin loaded:

kubectl logs -n kube-system <apiserver-pod-name> | grep NodeRestriction

Expected log line:

I0210 10:15:30.123456       1 plugins.go:158] Loaded 15 admission controller(s) successfully in the following order: ... NodeRestriction ...

Step 6: Rollback if needed.

If the API server fails to start, restore the backup:

cp /root/kube-apiserver.yaml.backup /etc/kubernetes/manifests/kube-apiserver.yaml

Example 2: Enforcing Pod Security Standards with Pod Security Admission

Pod Security Admission (PSA) is the modern replacement for PodSecurityPolicy. It enforces three policy levels: privileged, baseline, and restricted.

Step 1: Label a namespace to enforce the restricted level.

kubectl label namespace my-app pod-security.kubernetes.io/enforce=restricted pod-security.kubernetes.io/enforce-version=latest

Step 2: Test with a compliant and non-compliant pod.

Create a simple pod that meets the restricted policy (no privilege escalation, no host namespaces, etc.):

# compliant-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: compliant-pod
  namespace: my-app
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]

# non-compliant-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: non-compliant-pod
  namespace: my-app
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    securityContext:
      allowPrivilegeEscalation: true

Apply both and observe:

kubectl apply -f compliant-pod.yaml
kubectl apply -f non-compliant-pod.yaml

Expected for compliant pod:

pod/compliant-pod created

For non-compliant pod, you'll see an error like:

Error from server (Forbidden): error when creating "non-compliant-pod.yaml": pods "non-compliant-pod" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "nginx" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "nginx" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "nginx" must set securityContext.runAsNonRoot=true)

Step 3: Adjust policies as needed.

You can set warn and audit levels before enforcing to avoid breaking workloads:

kubectl label namespace my-app pod-security.kubernetes.io/warn=restricted pod-security.kubernetes.io/audit=restricted

Monitor audit logs and warnings before setting enforce.

Verification:

Check the namespace labels:

kubectl get namespace my-app --show-labels

Expected output includes the new labels.

Verification and Diagnostics

Verification is not just about confirming the change took effect; it's about ensuring the desired security posture is actually enforced. Use a combination of static checks and behavioral tests.

Check API Server Configuration

If you have access to API server logs or metrics, look for the list of enabled admission plugins. The log line from the previous section is definitive. Additionally, you can query the API server's --enable-admission-plugins flag via the kubeadm-config ConfigMap:

kubectl get configmap kubeadm-config -n kube-system -o jsonpath='{.data.ClusterConfiguration}' | grep enable-admission-plugins

Use the Kubernetes API to Test Admission Decisions

For controllers like ResourceQuota, you can test by attempting to create a resource that exceeds quota. For LimitRanger, test with a pod that exceeds limits.

Example: Validating ResourceQuota Enforcement

Create a quota for a namespace:

# quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: pod-quota
  namespace: my-app
spec:
  hard:
    pods: "2"

kubectl apply -f quota.yaml

Try to create three pods:

kubectl run pod1 --image=nginx --namespace=my-app
kubectl run pod2 --image=nginx --namespace=my-app
kubectl run pod3 --image=nginx --namespace=my-app

The third pod creation should fail with:

Error from server (Forbidden): pods "pod3" is forbidden: exceeded quota: pod-quota, requested: pods=1, used: pods=2, limited: pods=2

Inspecting Audit Logs for Admission Decisions

If audit logging is enabled, you can see admission controller decisions. The audit log records every request, including the admission phase. To find denied requests:

# On a node with audit log access
grep -A5 '"annotations".*admission' /var/log/kubernetes/audit/audit.log | grep '"allowed":false'

This helps identify workloads being blocked and adjust policies accordingly.

Troubleshooting Common Issues

• Admission controller not taking effect: Verify the API server pod restarted and is running the new version. Check the enabled plugins in the pod spec or logs.
• Requests failing after enabling a controller: Check the exact error message. It often indicates which policy is violated. Adjust the workload or the policy.
• RBAC permissions missing: Some admission controllers require additional RBAC roles. For example, the ImagePolicyWebhook controller needs the API server to have permissions to call the webhook service. Ensure the service account used by the API server has the necessary roles.

Use kubectl describe and kubectl logs to drill down:

kubectl describe pod <apiserver-pod> -n kube-system
kubectl logs <apiserver-pod> -n kube-system --previous

Failure Modes and Recovery

Even with careful planning, admission controller changes can cause outages. Here are common failure modes and recovery steps.

API Server Fails to Start

Symptom: After modifying the API server manifest, the API server pod is in CrashLoopBackOff or not starting.

Diagnosis: Check the API server pod logs:

kubectl logs -n kube-system <apiserver-pod> --previous

Look for errors like "invalid admission plugin" or syntax errors in the manifest.

Recovery: Restore the original manifest backup:

cp /root/kube-apiserver.yaml.backup /etc/kubernetes/manifests/kube-apiserver.yaml

Then wait for the API server to recover. Verify with:

kubectl get pods -n kube-system -l component=kube-apiserver

Workloads Being Rejected Unexpectedly

Symptom: New deployments or pods fail with Forbidden errors after enabling an admission controller (e.g., Pod Security Admission, ResourceQuota).

Diagnosis: Read the error message carefully. It indicates which policy is violated. Check the namespace labels and the resource quota/limit ranges.

Recovery: Depending on severity:

1. Temporary: Relax the enforcement level. For PSA, set enforce to a less restrictive level or remove the label:

   kubectl label namespace my-app pod-security.kubernetes.io/enforce-

1. Permanent: Fix the workload to comply with the policy. For example, for PSA restricted level, add the required security context fields. Then re-apply the enforce label.

Webhook Admission Controller Failure

Symptom: If you use a webhook admission controller (e.g., ValidatingWebhookConfiguration), and the webhook service is unavailable, all matching requests may fail.

Diagnosis: Check the webhook configuration and the service endpoints.

kubectl get validatingwebhookconfigurations my-webhook -o yaml
kubectl get endpoints -n webhook-namespace webhook-service

Look for failurePolicy in the webhook config. If set to Fail, any webhook error blocks the request. If Ignore, errors are ignored (less secure).

Recovery:

• Immediate: If the webhook is critical and misbehaving, temporarily delete the webhook configuration to unblock requests:

  kubectl delete validatingwebhookconfigurations my-webhook

• Permanent: Fix the webhook service, then re-create the configuration with appropriate failurePolicy (preferably Fail for security, but ensure high availability) and timeouts.

NodeRestriction Interfering with Node Updates

Symptom: Kubelet cannot update its own Node status or pods are not scheduled correctly.

Diagnosis: Check node conditions and kubelet logs. If the kubelet lacks proper credentials, it may be denied by NodeRestriction.

Recovery: Verify that the kubelet uses the correct client certificates and that the Node authorizer is enabled alongside NodeRestriction. If you must disable NodeRestriction temporarily, remove it from the enable list and restart API server as per rollback procedure.

Best Practice: Always test admission controller changes in a staging environment that mirrors production. Have a rollback plan documented before applying. Use canary namespaces or clusters to progressively roll out changes.

Operations Checklist

Use this checklist to ensure your admission controller hardening is systematic and safe.

Pre-Change Checklist

• [ ] Identify current Kubernetes version: kubectl version --short
• [ ] List enabled admission controllers: grep enable-admission-plugins /etc/kubernetes/manifests/kube-apiserver.yaml (or provider-specific method)
• [ ] Document the current configuration in version control.
• [ ] Backup API server manifest: cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.backup
• [ ] Ensure you have access to a non-production test cluster or namespace.
• [ ] Confirm RBAC permissions to make changes (cluster-admin or equivalent).
• [ ] Review prerequisite resources (e.g., Pod Security labels, ResourceQuota definitions).

During Change Checklist

• [ ] Apply the smallest change possible, one admission controller at a time.
• [ ] Use a canary namespace to test behavior before cluster-wide enforcement.
• [ ] Monitor API server pod health: kubectl get pods -n kube-system -l component=kube-apiserver -w
• [ ] Check API server logs for successful plugin loading.
• [ ] Run a set of test workloads (compliant and non-compliant) to verify enforcement.
• [ ] Record any errors and their resolutions.

Post-Change Verification Checklist

• [ ] Verify the admission controller is active (logs, config map, behavior test).
• [ ] Confirm that legacy workloads still function (or have been updated).
• [ ] Check audit logs for unexpected denials.
• [ ] Update documentation and runbooks with the new state.
• [ ] Notify the team about the change and potential impact.

Recovery Readiness Checklist

• [ ] Rollback plan documented: restore backup manifest or remove labels/configuration.
• [ ] Test rollback in staging.
• [ ] Define metrics or alerts for admission controller failures (e.g., increase in API server 4xx/5xx errors).
• [ ] Assign an owner for incident response related to admission controllers.

Conclusion

Kubernetes Admission Controllers are powerful gatekeepers that can significantly improve cluster security. However, they must be configured with care. By following the structured approach in this guide—inventory, safe configuration, verification, and recovery—you can harden your cluster without causing downtime or blocking legitimate workloads.

Remember these core principles:

• Observe before changing: Always capture current state and understand the impact.
• Smallest change: Modify one controller at a time and test incrementally.
• Verify with concrete tests: Use actual pods and requests to confirm enforcement.
• Have a rollback plan: Know how to revert quickly if something breaks.
• Protect secrets: Never expose credentials in manifests or commands.

As a next step, choose one low-risk verification from this guide—for example, enabling Pod Security Admission in audit mode on a test namespace. Record the current state, apply the change, run the verification tests, and document the results. Then gradually roll out to production namespaces, monitoring closely.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With these practices, you can confidently secure your Kubernetes clusters with admission controllers.