## Introduction Kubernetes admission controllers sit between authentication/authorization and object persistence, intercepting every API request to validate, mutate, or reject resources. Upgrading or migrating these controllers is a routine but high-stakes operation — whether you're moving from deprecated APIs like admissionregistration.k8s.io/v1beta1 to v1 , replacing PodSecurityPolicy with the Pod Security Admission controller, or rolling out a custom webhook update. A poorly executed change can block cluster-wide workload creation, while a well-planned migration proceeds in controlled steps with instant rollback capability. This guide provides a repeatable, production-tested workflow: inventory the current state, run a scoped pilot with safe defaults, verify behavior with concrete test cases, and maintain documented recovery procedures. Every step includes exact kubectl commands, YAML manifests, and expected output so you can execute confidently on your own clusters. ## Version and Environment Inventory Before touching any configuration, establish a precise baseline. Run these commands from a machine with cluster-admin access: kubectl version --short kubectl get --raw /version Example output: Client Version: v1.28.4 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.6 Confirm which admissionregistration API versions the cluster serves. The v1 API has been stable since Kubernetes 1.16; if your control plane is older, plan the control plane upgrade first. kubectl api-versions | grep admissionregistration Expected output on a modern cluster: admissionregistration.k8s.io/v1 List all webhook configurations currently installed: kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations Example output: NAME WEBHOOKS AGE pod-policy.example.com 1 45d quota-enforcer.example.com 1 12d For each webhook, inspect the full resource to capture apiVersion , admissionReviewVersions , failurePolicy , sideEffects , and namespaceSelector : kubectl get validatingwebhookconfiguration pod-policy.example.com -o yaml Key snippet to record: apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: name: pod-policy.example.com webhooks: - name: pod-policy.example.com admissionReviewVersions: ["v1"] sideEffects: None failurePolicy: Fail For built-in admission plugins, examine the kube-apiserver static pod manifest. On kubeadm clusters: grep -A2 'enable-admission-plugins' /etc/kubernetes/manifests/kube-apiserver.yaml Example output: - --enable-admission-plugins=NodeRestriction,NamespaceLifecycle,LimitRanger,ServiceAccount,PodSecurity - --disable-admission-plugins= Document all findings in a change log. Take an etcd snapshot before proceeding, and export every webhook configuration as YAML for immediate rollback: mkdir -p /root/admission-backup/$(date +%F) kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml > /root/admission-backup/$(date +%F)/webhooks-backup.yaml ## Safe Configuration Path: Scoped Pilot Never apply a cluster-wide change in one step. Use a scoped pilot that limits the new configuration to a single test namespace, and start with failurePolicy: Ignore so a broken webhook cannot block legitimate workloads. ### Webhook Migration Pilot Create a migrated ValidatingWebhookConfiguration targeting the v1 API with a namespace selector: # validatingwebhook-scoped.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: name: pod-policy.example.com webhooks: - name: pod-policy.example.com clientConfig: service: name: webhook-service namespace: webhook-ns path: /validate port: 443 caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t... # replace with actual CA rules: - operations: ["CREATE", "UPDATE"] apiGroups: ["*"] apiVersions: ["*"] resources: ["pods"] scope: Namespaced namespaceSelector: matchLabels: admission-pilot: enabled admissionReviewVersions: ["v1"] sideEffects: None failurePolicy: Ignore timeoutSeconds: 5 Apply it: kubectl apply -f validatingwebhook-scoped.yaml Expected output: validatingwebhookconfiguration.admissionregistration.k8s.io/pod-policy.example.com configured Create and label the pilot namespace: kubectl create namespace pilot-ns kubectl label namespace pilot-ns admission-pilot=enabled Expected output: namespace/pilot-ns created namespace/pilot-ns labeled Only resources created in pilot-ns will now hit this webhook. All other namespaces continue using the previous configuration (or no webhook if this is a new deployment). ### Built-in Controller Migration: PodSecurityPolicy to Pod Security Admission For the built-in migration, edit the kube-apiserver manifest to enable the PodSecurity plugin while preserving all existing required plugins: # Backup first cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/admission-backup/$(date +%F)/kube-apiserver.yaml.backup # Edit the manifest vim /etc/kubernetes/manifests/kube-apiserver.yaml Locate the --enable-admission-plugins flag and ensure PodSecurity is included alongside the mandatory plugins: - --enable-admission-plugins=NodeRestriction,NamespaceLifecycle,LimitRanger,ServiceAccount,PodSecurity Do not remove NodeRestriction , NamespaceLifecycle , LimitRanger , or ServiceAccount unless you have validated their removal in a test cluster. The kubelet will automatically restart the API server when the manifest changes. Now label the pilot namespace with Pod Security Admission modes. Use baseline for enforcement (blocks known privilege escalations) and restricted for warnings (alerts on highly privileged workloads): kubectl label namespace pilot-ns pod-security.kubernetes.io/enforce=baseline kubectl label namespace pilot-ns pod-security.kubernetes.io/warn=restricted kubectl label namespace pilot-ns pod-security.kubernetes.io/audit=restricted Expected output: namespace/pilot-ns labeled namespace/pilot-ns labeled namespace/pilot-ns labeled This scoped enforcement means only pilot-ns workloads are evaluated against the new policies. The rest of the cluster remains on the legacy PodSecurityPolicy (or unrestricted) until you expand the labels. ## Verification and Diagnostics Verification has two objectives: confirm the controller is active, and observe the exact accept/deny decisions for representative workloads. ### Webhook Verification Create a test Pod in the pilot namespace that should be denied by your policy (e.g., a container running as root when the policy requires runAsNonRoot: true ): # test-deny.yaml apiVersion: v1 kind: Pod metadata: name: test-deny namespace: pilot-ns spec: containers: - name: nginx image: nginx:1.25 securityContext: runAsNonRoot: false Apply it: kubectl apply -f test-deny.yaml Expected denial output: Error from server (Forbidden): error when creating "test-deny.yaml": admission webhook "pod-policy.example.com" denied the request: Pod does not meet security policy: runAsNonRoot must be true Now create a compliant Pod that should be admitted: # test-allow.yaml apiVersion: v1 kind: Pod metadata: name: test-allow namespace: pilot-ns spec: containers: - name: nginx image: nginx:1.25 securityContext: runAsNonRoot: true runAsUser: 1000 kubectl apply -f test-allow.yaml Expected success output: pod/test-allow created If the webhook does not fire at all, troubleshoot in this order: - Namespace selector match : kubectl get namespace pilot-ns --show-labels — confirm admission-pilot=enabled is present. - Service reachability : kubectl get endpoints -n webhook-ns webhook-service — verify the endpoint IP matches a running webhook pod. - CA bundle validity : The caBundle in the webhook config must match the serving certificate. Decode and inspect: echo "" | base64 -d | openssl x509 -text -noout . - Webhook logs : kubectl logs -n webhook-ns -l app=webhook-service — look for incoming /validate requests and any TLS handshake errors. Check recent events in the pilot namespace for admission-related entries: kubectl get events -n pilot-ns --sort-by=.lastTimestamp Example output: LAST SEEN TYPE REASON OBJECT MESSAGE 12s Warning FailedCreate pod/test-deny admission webhook "pod-policy.example.com" denied the request: Pod does not meet security policy: runAsNonRoot must be true ### Pod Security Admission Verification Test the built-in controller with a privileged Pod that violates the baseline profile: # test-privileged.yaml apiVersion: v1 kind: Pod metadata: name: test-privileged namespace: pilot-ns spec: containers: - name: test image: busybox:1.36 command: ["sleep", "3600"] securityContext: privileged: true kubectl apply -f test-privileged.yaml -n pilot-ns Expected denial (baseline enforcement): Error from server (Forbidden): pods "test-privileged" is forbidden: violates PodSecurity "baseline: v1.27": privileged (container "test" must not set securityContext.privileged=true) A compliant Pod (no privileged, no hostPath, no hostNetwork, etc.) should create successfully. ### Audit Log Confirmation If API server audit logging is enabled, query for admission decisions: grep 'admission webhook' /var/log/kubernetes/audit.log | tail -n 5 Example audit entry (truncated): {"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"RequestResponse","verb":"create","requestURI":"/api/v1/namespaces/pilot-ns/pods","user":{"username":"admin"},"responseStatus":{"code":403},"annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding"}} A responseStatus.code of 200 or 201 means admitted; 403 means denied by an admission controller. Confirm the live failure policy during the pilot: kubectl get --raw /apis/admissionregistration.k8s.io/v1/validatingwebhookconfigurations/pod-policy.example.com | jq .webhooks[0].failurePolicy Expected output during pilot: "Ignore" ## Failure Modes and Recovery Admission controller changes fail in predictable ways. The table below maps each failure mode to its observable symptom and the exact recovery command.
Failure ModeSymptomRecovery Action
Webhook endpoint down with failurePolicy: FailAll matching pod creates hang then fail with timeoutSet failurePolicy: Ignore or delete webhook
Invalid caBundle or wrong service name"connection refused" or TLS errors in API server logsFix caBundle or delete webhook
Missing required v1 fields (sideEffects, admissionReviewVersions)kubectl apply returns field validation errorAdd missing fields and reapply
Typo in --enable-admission-pluginskube-apiserver fails to start, crashloopRestore manifest from backup
Namespace selector too broad (e.g., missing or matchExpressions too wide)Webhook rejects resources in production namespacesTighten matchLabels and reapply
### Recovery Commands 1. Webhook endpoint failure — switch to Ignore immediately: kubectl patch validatingwebhookconfiguration pod-policy.example.com \ --type merge \ -p '{"webhooks":[{"name":"pod-policy.example.com","failurePolicy":"Ignore"}]}' Expected output: validatingwebhookconfiguration.admissionregistration.k8s.io/pod-policy.example.com patched 2. kube-apiserver won't start due to plugin misconfiguration — restore manifest: cp /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/kube-apiserver.yaml.broken cp /root/admission-backup/$(date +%F)/kube-apiserver.yaml.backup /etc/kubernetes/manifests/kube-apiserver.yaml The kubelet detects the file change and restarts the API server automatically. Avoid systemctl restart kubelet unless the kubelet itself is unhealthy. 3. Roll back a webhook migration — delete new config, apply backup: kubectl delete validatingwebhookconfiguration pod-policy.example.com kubectl apply -f /root/admission-backup/$(date +%F)/webhooks-backup.yaml 4. Disable Pod Security Admission enforcement on a namespace: kubectl label namespace pilot-ns pod-security.kubernetes.io/enforce- kubectl label namespace pilot-ns pod-security.kubernetes.io/warn- kubectl label namespace pilot-ns pod-security.kubernetes.io/audit- Expected output: namespace/pilot-ns unlabeled namespace/pilot-ns unlabeled namespace/pilot-ns unlabeled 5. Post-recovery health check: kubectl get nodes kubectl get pods -A Confirm all nodes are Ready and no system pods are in CrashLoopBackOff or Pending due to admission failures. ## Operations Checklist Use this checklist for every admission controller upgrade or migration. Check off each item and record the output in your change log. - [ ] Confirm cluster version ( kubectl version --short ) and admissionregistration API version ( kubectl api-versions | grep admissionregistration ). - [ ] Export all ValidatingWebhookConfiguration and MutatingWebhookConfiguration YAML to a timestamped backup directory. - [ ] Take an etcd snapshot ( etcdctl snapshot save /backup/etcd-$(date +%F).db ) and verify it is restorable on a test cluster. - [ ] Identify the exact webhook or built-in plugin to change, its current scope, and its failurePolicy . - [ ] Create a pilot namespace with a unique label (e.g., admission-pilot=enabled ). - [ ] Apply the new configuration with failurePolicy: Ignore (webhooks) or scoped namespace labels (Pod Security Admission). - [ ] Run positive test (compliant resource → admitted) and negative test (non-compliant resource → denied with expected message) in the pilot namespace. - [ ] Capture the exact deny/allow output and paste into the change record. - [ ] Check API server audit logs or webhook logs for admission decision entries. - [ ] Once pilot passes, gradually expand the namespaceSelector (add labels to more namespaces) or switch failurePolicy to Fail . - [ ] Document the rollback artifact location and the exact command to restore (e.g., kubectl apply -f /root/admission-backup/2024-01-15/webhooks-backup.yaml ). - [ ] Schedule a review 24–48 hours after full rollout to look for silent failures. Post-rollout review command: kubectl get events -A --field-selector reason=FailedCreate | grep -i admission If this returns any events, investigate the webhook or policy logic before considering the migration complete. ## Conclusion A safe admission controller upgrade or migration follows a disciplined sequence: inventory the exact cluster state and current configuration, isolate the change behind a labeled pilot namespace with failurePolicy: Ignore , verify every accept and deny path with concrete test cases and captured output, and maintain documented, one-command rollback procedures. Start by collecting the cluster version, all webhook YAMLs, and the kube-apiserver plugin list. Apply changes only to the pilot namespace — whether through a webhook namespaceSelector or Pod Security Admission labels — and validate with both negative and positive test pods. Keep backup manifests and etcd snapshots accessible so you can revert in minutes if the API server fails to start or a webhook misbehaves. Repeat the operations checklist for each subsequent change, and audit events after full rollout to catch silent regressions. These steps transform a risky cluster-wide change into a controlled, repeatable migration that eliminates unplanned downtime and reduces rework.