Kubernetes Admission Controllers Upgrade and Migration: Practical Examples for Safe Rollout

Introduction

Kubernetes admission controllers sit between authentication/authorization and object persistence, intercepting every API request to validate, mutate, or reject resources. Upgrading or migrating these controllers is a routine but high-stakes operation — whether you're moving from deprecated APIs like admissionregistration.k8s.io/v1beta1 to v1, replacing PodSecurityPolicy with the Pod Security Admission controller, or rolling out a custom webhook update. A poorly executed change can block cluster-wide workload creation, while a well-planned migration proceeds in controlled steps with instant rollback capability.

This guide provides a repeatable, production-tested workflow: inventory the current state, run a scoped pilot with safe defaults, verify behavior with concrete test cases, and maintain documented recovery procedures. Every step includes exact kubectl commands, YAML manifests, and expected output so you can execute confidently on your own clusters.

Version and Environment Inventory

Before touching any configuration, establish a precise baseline. Run these commands from a machine with cluster-admin access:

kubectl version --short
kubectl get --raw /version

Example output:

Client Version: v1.28.4
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.6

Confirm which admissionregistration API versions the cluster serves. The v1 API has been stable since Kubernetes 1.16; if your control plane is older, plan the control plane upgrade first.

kubectl api-versions | grep admissionregistration

Expected output on a modern cluster:

admissionregistration.k8s.io/v1

List all webhook configurations currently installed:

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

Example output:

NAME                           WEBHOOKS   AGE
pod-policy.example.com         1          45d
quota-enforcer.example.com     1          12d

For each webhook, inspect the full resource to capture apiVersion, admissionReviewVersions, failurePolicy, sideEffects, and namespaceSelector:

kubectl get validatingwebhookconfiguration pod-policy.example.com -o yaml

Key snippet to record:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-policy.example.com
webhooks:
- name: pod-policy.example.com
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail

For built-in admission plugins, examine the kube-apiserver static pod manifest. On kubeadm clusters:

grep -A2 'enable-admission-plugins' /etc/kubernetes/manifests/kube-apiserver.yaml

Example output:

- --enable-admission-plugins=NodeRestriction,NamespaceLifecycle,LimitRanger,ServiceAccount,PodSecurity
- --disable-admission-plugins=

Document all findings in a change log. Take an etcd snapshot before proceeding, and export every webhook configuration as YAML for immediate rollback:

mkdir -p /root/admission-backup/$(date +%F)
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml > /root/admission-backup/$(date +%F)/webhooks-backup.yaml

Safe Configuration Path: Scoped Pilot

Never apply a cluster-wide change in one step. Use a scoped pilot that limits the new configuration to a single test namespace, and start with failurePolicy: Ignore so a broken webhook cannot block legitimate workloads.

Webhook Migration Pilot

Create a migrated ValidatingWebhookConfiguration targeting the v1 API with a namespace selector:

# validatingwebhook-scoped.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-policy.example.com
webhooks:
- name: pod-policy.example.com
  clientConfig:
    service:
      name: webhook-service
      namespace: webhook-ns
      path: /validate
      port: 443
    caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t...  # replace with actual CA
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["*"]
    apiVersions: ["*"]
    resources: ["pods"]
    scope: Namespaced
  namespaceSelector:
    matchLabels:
      admission-pilot: enabled
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Ignore
  timeoutSeconds: 5

Apply it:

kubectl apply -f validatingwebhook-scoped.yaml

Expected output:

validatingwebhookconfiguration.admissionregistration.k8s.io/pod-policy.example.com configured

Create and label the pilot namespace:

kubectl create namespace pilot-ns
kubectl label namespace pilot-ns admission-pilot=enabled

Expected output:

namespace/pilot-ns created
namespace/pilot-ns labeled

Only resources created in pilot-ns will now hit this webhook. All other namespaces continue using the previous configuration (or no webhook if this is a new deployment).

Built-in Controller Migration: PodSecurityPolicy to Pod Security Admission

For the built-in migration, edit the kube-apiserver manifest to enable the PodSecurity plugin while preserving all existing required plugins:

# Backup first
cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/admission-backup/$(date +%F)/kube-apiserver.yaml.backup

# Edit the manifest
vim /etc/kubernetes/manifests/kube-apiserver.yaml

Locate the --enable-admission-plugins flag and ensure PodSecurity is included alongside the mandatory plugins:

- --enable-admission-plugins=NodeRestriction,NamespaceLifecycle,LimitRanger,ServiceAccount,PodSecurity

Do not remove NodeRestriction, NamespaceLifecycle, LimitRanger, or ServiceAccount unless you have validated their removal in a test cluster. The kubelet will automatically restart the API server when the manifest changes.

Now label the pilot namespace with Pod Security Admission modes. Use baseline for enforcement (blocks known privilege escalations) and restricted for warnings (alerts on highly privileged workloads):

kubectl label namespace pilot-ns pod-security.kubernetes.io/enforce=baseline
kubectl label namespace pilot-ns pod-security.kubernetes.io/warn=restricted
kubectl label namespace pilot-ns pod-security.kubernetes.io/audit=restricted

Expected output:

namespace/pilot-ns labeled
namespace/pilot-ns labeled
namespace/pilot-ns labeled

This scoped enforcement means only pilot-ns workloads are evaluated against the new policies. The rest of the cluster remains on the legacy PodSecurityPolicy (or unrestricted) until you expand the labels.

Verification and Diagnostics

Verification has two objectives: confirm the controller is active, and observe the exact accept/deny decisions for representative workloads.

Webhook Verification

Create a test Pod in the pilot namespace that should be denied by your policy (e.g., a container running as root when the policy requires runAsNonRoot: true):

# test-deny.yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-deny
  namespace: pilot-ns
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    securityContext:
      runAsNonRoot: false

Apply it:

kubectl apply -f test-deny.yaml

Expected denial output:

Error from server (Forbidden): error when creating "test-deny.yaml": admission webhook "pod-policy.example.com" denied the request: Pod does not meet security policy: runAsNonRoot must be true

Now create a compliant Pod that should be admitted:

# test-allow.yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-allow
  namespace: pilot-ns
spec:
  containers:
  - name: nginx
    image: nginx:1.25
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000

kubectl apply -f test-allow.yaml

Expected success output:

pod/test-allow created

If the webhook does not fire at all, troubleshoot in this order:

1. Namespace selector match: kubectl get namespace pilot-ns --show-labels — confirm admission-pilot=enabled is present.
2. Service reachability: kubectl get endpoints -n webhook-ns webhook-service — verify the endpoint IP matches a running webhook pod.
3. CA bundle validity: The caBundle in the webhook config must match the serving certificate. Decode and inspect: echo "<caBundle>" | base64 -d | openssl x509 -text -noout.
4. Webhook logs: kubectl logs -n webhook-ns -l app=webhook-service — look for incoming /validate requests and any TLS handshake errors.

Check recent events in the pilot namespace for admission-related entries:

kubectl get events -n pilot-ns --sort-by=.lastTimestamp

Example output:

LAST SEEN   TYPE      REASON        OBJECT           MESSAGE
12s         Warning   FailedCreate  pod/test-deny    admission webhook "pod-policy.example.com" denied the request: Pod does not meet security policy: runAsNonRoot must be true

Pod Security Admission Verification

Test the built-in controller with a privileged Pod that violates the baseline profile:

# test-privileged.yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-privileged
  namespace: pilot-ns
spec:
  containers:
  - name: test
    image: busybox:1.36
    command: ["sleep", "3600"]
    securityContext:
      privileged: true

kubectl apply -f test-privileged.yaml -n pilot-ns

Expected denial (baseline enforcement):

Error from server (Forbidden): pods "test-privileged" is forbidden: violates PodSecurity "baseline: v1.27": privileged (container "test" must not set securityContext.privileged=true)

A compliant Pod (no privileged, no hostPath, no hostNetwork, etc.) should create successfully.

Audit Log Confirmation

If API server audit logging is enabled, query for admission decisions:

grep 'admission webhook' /var/log/kubernetes/audit.log | tail -n 5

Example audit entry (truncated):

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","verb":"create","requestURI":"/api/v1/namespaces/pilot-ns/pods","user":{"username":"admin"},"responseStatus":{"code":403},"annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding"}}

A responseStatus.code of 200 or 201 means admitted; 403 means denied by an admission controller.

Confirm the live failure policy during the pilot:

kubectl get --raw /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/pod-policy.example.com | jq .webhooks[0].failurePolicy

Expected output during pilot:

"Ignore"

Failure Modes and Recovery

Admission controller changes fail in predictable ways. The table below maps each failure mode to its observable symptom and the exact recovery command.

Recovery Commands

1. Webhook endpoint failure — switch to Ignore immediately:

kubectl patch validatingwebhookconfiguration pod-policy.example.com \
  --type merge \
  -p '{"webhooks":[{"name":"pod-policy.example.com","failurePolicy":"Ignore"}]}'

Expected output:

validatingwebhookconfiguration.admissionregistration.k8s.io/pod-policy.example.com patched

2. kube-apiserver won't start due to plugin misconfiguration — restore manifest:

cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.broken
cp /root/admission-backup/$(date +%F)/kube-apiserver.yaml.backup /etc/kubernetes/manifests/kube-apiserver.yaml

The kubelet detects the file change and restarts the API server automatically. Avoid systemctl restart kubelet unless the kubelet itself is unhealthy.

3. Roll back a webhook migration — delete new config, apply backup:

kubectl delete validatingwebhookconfiguration pod-policy.example.com
kubectl apply -f /root/admission-backup/$(date +%F)/webhooks-backup.yaml

4. Disable Pod Security Admission enforcement on a namespace:

kubectl label namespace pilot-ns pod-security.kubernetes.io/enforce-
kubectl label namespace pilot-ns pod-security.kubernetes.io/warn-
kubectl label namespace pilot-ns pod-security.kubernetes.io/audit-

Expected output:

namespace/pilot-ns unlabeled
namespace/pilot-ns unlabeled
namespace/pilot-ns unlabeled

5. Post-recovery health check:

kubectl get nodes
kubectl get pods -A

Confirm all nodes are Ready and no system pods are in CrashLoopBackOff or Pending due to admission failures.

Operations Checklist

Use this checklist for every admission controller upgrade or migration. Check off each item and record the output in your change log.

• [ ] Confirm cluster version (kubectl version --short) and admissionregistration API version (kubectl api-versions | grep admissionregistration).
• [ ] Export all ValidatingWebhookConfiguration and MutatingWebhookConfiguration YAML to a timestamped backup directory.
• [ ] Take an etcd snapshot (etcdctl snapshot save /backup/etcd-$(date +%F).db) and verify it is restorable on a test cluster.
• [ ] Identify the exact webhook or built-in plugin to change, its current scope, and its failurePolicy.
• [ ] Create a pilot namespace with a unique label (e.g., admission-pilot=enabled).
• [ ] Apply the new configuration with failurePolicy: Ignore (webhooks) or scoped namespace labels (Pod Security Admission).
• [ ] Run positive test (compliant resource → admitted) and negative test (non-compliant resource → denied with expected message) in the pilot namespace.
• [ ] Capture the exact deny/allow output and paste into the change record.
• [ ] Check API server audit logs or webhook logs for admission decision entries.
• [ ] Once pilot passes, gradually expand the namespaceSelector (add labels to more namespaces) or switch failurePolicy to Fail.
• [ ] Document the rollback artifact location and the exact command to restore (e.g., kubectl apply -f /root/admission-backup/2024-01-15/webhooks-backup.yaml).
• [ ] Schedule a review 24–48 hours after full rollout to look for silent failures.

Post-rollout review command:

kubectl get events -A --field-selector reason=FailedCreate | grep -i admission

If this returns any events, investigate the webhook or policy logic before considering the migration complete.

Conclusion

A safe admission controller upgrade or migration follows a disciplined sequence: inventory the exact cluster state and current configuration, isolate the change behind a labeled pilot namespace with failurePolicy: Ignore, verify every accept and deny path with concrete test cases and captured output, and maintain documented, one-command rollback procedures. Start by collecting the cluster version, all webhook YAMLs, and the kube-apiserver plugin list. Apply changes only to the pilot namespace — whether through a webhook namespaceSelector or Pod Security Admission labels — and validate with both negative and positive test pods. Keep backup manifests and etcd snapshots accessible so you can revert in minutes if the API server fails to start or a webhook misbehaves. Repeat the operations checklist for each subsequent change, and audit events after full rollout to catch silent regressions. These steps transform a risky cluster-wide change into a controlled, repeatable migration that eliminates unplanned downtime and reduces rework.