Kubernetes Admission Controllers: Backup, Restore, and Disaster Recovery - A Practical Guide

Intro

Kubernetes admission controllers sit directly in the API request path. They intercept create, update, and delete operations, and they can reject a request before it is persisted. This makes them a powerful policy and safety layer—but also a fragile one. A misconfigured admission webhook can block all deployments, break namespaces, or take down a cluster's ability to schedule workloads.

Backup, restore, and disaster recovery for admission controllers are not afterthoughts. They are operational prerequisites. A cluster without a tested recovery path for its admission pipeline is one bad manifest away from an outage.

This article is for developers, DevOps consultants, and technical startup teams who operate real clusters. It covers:

• Version and environment inventory for admission controllers
• Safe configuration paths, including webhook manifests and certificate handling
• Verification and diagnostics with concrete commands and expected outputs
• Failure modes and recovery workflows
• An operations checklist you can actually run

Every section includes practical commands, manifest snippets, and failure signals. The goal is operational safety: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document recovery before you need it.

Version and Environment Inventory

Before touching an admission controller, know exactly what is running. Admission controllers come in two broad categories:

• Built-in admission controllers compiled into the Kubernetes API server, enabled with --enable-admission-plugins.
• Dynamic admission controllers (webhooks) registered via ValidatingWebhookConfiguration and MutatingWebhookConfiguration.

What to inventory

Capture:

• Kubernetes cluster version (kubectl version --short)
• API server flags on control plane nodes
• All webhook configurations
• The webhook services, deployments, and pods behind each webhook
• Webhook certificates and their expiry
• Any policies, custom resources, and namespaces involved

Read-only observation first

Run these commands before making any change:

kubectl version --short
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
kubectl get apiservices | grep -E 'v1beta1.admission|v1.admission'

Expected output example (truncated):

NAME                                   WEBHOOKS   AGE
opa-validating-webhook                 1          12d
image-policy-mutating-webhook          1          8d

For each webhook, inspect its rules, namespace selector, and client config:

kubectl describe validatingwebhookconfiguration opa-validating-webhook

Look for:

• rules: which resources and operations trigger the webhook
• namespaceSelector: which namespaces are affected
• clientConfig.service: the service name, namespace, path, and optional CA bundle
• failurePolicy: Fail or Ignore. If Fail, the API server rejects requests when the webhook is unavailable.

Check the webhook backend health

Webhook backends are just pods. Use the standard Kubernetes troubleshooting sequence:

kubectl get pods -n webhook-system -o wide
kubectl describe pod <pod-name> -n webhook-system
kubectl logs <pod-name> -n webhook-system --previous
kubectl rollout status deployment/<webhook-deployment> -n webhook-system

Example of a healthy webhook deployment:

$ kubectl rollout status deployment/opa-webhook -n webhook-system
deployment "opa-webhook" successfully rolled out

If the deployment is stuck, inspect events and logs before changing anything.

Certificates are part of the inventory

Dynamic admission webhooks require TLS. The CA bundle in the webhook configuration must match the certificate presented by the webhook server. Check certificate expiry:

kubectl get validatingwebhookconfiguration opa-validating-webhook -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -dates

Expected output:

notBefore=Mar  1 12:00:00 2024 GMT
notAfter=Mar  1 12:00:00 2025 GMT

If the cert is close to expiry, schedule rotation before it fails.

Record state before change

Create a snapshot file:

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml > admission-webhooks-snapshot-$(date +%F).yaml

This snapshot is your quick rollback path. Store it in version control alongside your cluster configs.

Safe Configuration Path

Changing admission controllers is high-risk because a mistake can block workload creation cluster-wide. Follow a controlled path.

Step 1: Define the change and blast radius

Before editing, document:

• Which webhook or flag is changing?
• Which resources and namespaces are affected?
• What is the failurePolicy?
• How will you verify success?
• What is the rollback command?

Step 2: Use a copy, not the live object

Always work from a copy of the configuration:

kubectl get validatingwebhookconfiguration opa-validating-webhook -o yaml > opa-webhook-backup.yaml
cp opa-webhook-backup.yaml opa-webhook-new.yaml

Edit opa-webhook-new.yaml with your changes.

Step 3: Pre-flight validation

Use kubectl apply --dry-run=client or --dry-run=server to validate syntax and server-side admission:

kubectl apply -f opa-webhook-new.yaml --dry-run=client

For more thorough validation, test with a minimal admission request using kubectl create dry-run on a dummy resource that triggers the webhook.

Step 4: Apply and immediately verify

Apply the change:

kubectl apply -f opa-webhook-new.yaml

Then check the webhook is still working:

kubectl get validatingwebhookconfiguration opa-validating-webhook -o yaml
kubectl get pods -n webhook-system
kubectl logs <webhook-pod> -n webhook-system --tail=20

Create a test resource that should be admitted and one that should be rejected (if your policy has a denylist). For example, use a dry-run to avoid side effects:

kubectl create deployment test-admission --image=nginx --dry-run=client -o yaml | kubectl apply --dry-run=server -f -

If the request is rejected unexpectedly, check webhook logs and metrics.

Step 5: Keep local testing minimal

For new webhook development, use a local cluster or a dedicated namespace with namespaceSelector limiting the webhook to that namespace. Example namespace selector in the webhook configuration:

namespaceSelector:
  matchLabels:
    admission-webhook: enabled

Then label only the test namespace:

kubectl label namespace test-ns admission-webhook=enabled

Verify traffic locally with kubectl port-forward to the webhook service before exposing it via a load balancer or ingress.

Verification and Diagnostics

Verification is not "it applied without error". You must confirm the admission controller is actually intercepting requests and making correct decisions.

1. Check webhook configuration is active

kubectl get validatingwebhookconfigurations opa-validating-webhook -o jsonpath='{.metadata.name} {.webhooks[0].failurePolicy} {.webhooks[0].rules}'

Expected output shows rules and failurePolicy.

2. Test admission decisions with dry-run

Create a manifest that should pass and one that should fail. Use kubectl apply --dry-run=server to avoid persisting the resource.

Pass example:

cat <<EOF | kubectl apply --dry-run=server -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-pass
spec:
  containers:
  - name: nginx
    image: nginx:1.21
EOF

Fail example (assuming policy requires specific label):

cat <<EOF | kubectl apply --dry-run=server -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-fail
spec:
  containers:
  - name: nginx
    image: nginx:latest
EOF

Expected failure message:

Error from server (Forbidden): error when creating "STDIN": admission webhook "opa-validating-webhook.example.com" denied the request: image tag 'latest' not allowed

If the failure is not as expected, the webhook is not working correctly.

3. Inspect webhook logs and metrics

Most webhooks log admission review requests. Enable verbose logging if needed.

kubectl logs -n webhook-system deploy/opa-webhook --tail=50

Look for decisions and timestamps. For OPA, use opa eval or check the decision log.

Many webhook frameworks expose Prometheus metrics. Query metrics like apiserver_admission_webhook_request_total from the API server to see webhook calls:

kubectl get --raw /metrics | grep apiserver_admission_webhook_request_total

This shows counts per webhook and result.

4. Test API server flags for built-in admission controllers

If you changed --enable-admission-plugins, check the API server logs and current flags:

kubectl -n kube-system logs kube-apiserver-control-plane | grep admission

Or check the static pod manifest on the control plane node.

5. Validate certificate chain

If webhook TLS is misconfigured, API server logs show errors. Check:

kubectl -n kube-system logs kube-apiserver-control-plane | grep "x509"

Verify the CA bundle matches the webhook server cert using openssl.

Failure Modes and Recovery

Admission controller failures can be silent or loud. Know the common failure modes and how to recover.

Failure mode 1: Webhook service unavailable

Symptom: all API requests that match the webhook's rules fail with timeout or connection refused, but only if failurePolicy: Fail. If failurePolicy: Ignore, requests are allowed through silently.

Error example:

Error from server (InternalError): error when creating "deployment.yaml": Internal error occurred: failed calling webhook "opa-validating-webhook.example.com": Post "https://opa-webhook.webhook-system.svc:443/validate?timeout=10s": dial tcp 10.96.0.42:443: connect: connection refused

Recovery:

1. Check if the webhook deployment is running:

   kubectl get pods -n webhook-system

1. If not, scale up or restart:

   kubectl scale deployment opa-webhook --replicas=1 -n webhook-system

1. If the service selector is wrong, fix it.
2. As an emergency escape hatch, temporarily set failurePolicy: Ignore or delete the webhook configuration:

   kubectl delete validatingwebhookconfiguration opa-validating-webhook

Warning: deleting the webhook removes policy enforcement. Restore from backup as soon as possible.

Failure mode 2: Certificate expiry or mismatch

Symptom: API server logs show x509: certificate signed by unknown authority or certificate has expired. All matching requests fail.

Recovery:

1. Generate new cert/key pair for the webhook server.
2. Update the webhook deployment's TLS secret.
3. Update the caBundle in the webhook configuration with the new CA.
4. Apply the updated configuration.

Example to update CA bundle from a file:

CA_BUNDLE=$(cat ca.crt | base64 | tr -d '\n')
kubectl patch validatingwebhookconfiguration opa-validating-webhook --type='json' -p="[{'op': 'replace', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${CA_BUNDLE}'}]"

Then verify with a dry-run admission test.

Failure mode 3: Overly broad webhook rules block all resources

Symptom: after applying a new webhook, everything fails, including system components.

Recovery:

1. Immediately delete the webhook configuration:

   kubectl delete validatingwebhookconfiguration <name>

1. Restore the previous version from your snapshot:

   kubectl apply -f admission-webhooks-snapshot-YYYY-MM-DD.yaml

1. Investigate the rules and namespaceSelector to narrow scope.
2. Reapply a fixed version.

Failure mode 4: Mutating webhook corrupts resources

A mutating webhook can inject invalid patches, breaking applications. If you notice pods failing after a mutation, check:

kubectl get pods <name> -o yaml | grep -A5 'annotations'

Look for mutations that changed required fields.

Recovery:

1. Disable the mutating webhook.
2. Redeploy affected workloads.
3. Fix the mutation logic and re-enable in a test namespace.

Rollback workflow

Always keep a known-good snapshot. Apply rollback with:

kubectl apply -f admission-webhooks-known-good.yaml

Then monitor API server logs and admission metrics to confirm the baseline is restored.

Operations Checklist

Use this checklist before and after any change to admission controllers.

Before change

• [ ] Cluster version and webhook list captured in a document
• [ ] Current webhook configurations exported to YAML and stored in git
• [ ] Certificate expiry dates noted
• [ ] Webhook backend deployment and service health verified (kubectl get pods, kubectl describe)
• [ ] Test namespace labeled for scoped rollout if possible
• [ ] Dry-run of the new configuration completed without errors
• [ ] Rollback snapshot accessible
• [ ] On-call notification or maintenance window arranged

During change

• [ ] Apply one webhook change at a time
• [ ] Watch API server logs for errors
• [ ] Run admission dry-run tests for both allow and deny cases
• [ ] Monitor apiserver_admission_webhook_request_total metric

After change

• [ ] Verify webhooks are still registered
• [ ] Run full admission test suite
• [ ] Check application creation in affected namespaces
• [ ] Record the change in runbook with timestamp and outcome
• [ ] Ensure rollback document is updated if needed

Emergency response quick reference

Conclusion

Kubernetes admission controllers are powerful, but they require operational discipline. Backup and restore are not just about configuration files—they include certificates, webhook backend deployments, and test procedures.

This guide provides a practical workflow:

1. Inventory your admission controllers and their state
2. Make changes through a safe, scoped path with dry-runs and backups
3. Verify behavior with concrete tests and metrics
4. Prepare for common failures with explicit recovery steps
5. Use an operations checklist to prevent mistakes

Start with one low-risk verification in a test namespace. Record current state, run the check, compare the result, and review dependencies. A reliable workflow makes failure visible, protects sensitive values, limits changes, and defines recovery before an incident forces the decision.

Next steps: pick a webhook you already run, export its configuration, review its failurePolicy and namespace selectors, and test a rollback with your snapshot. That exercise will show you exactly how prepared you are.