## Intro

Kubernetes admission controllers sit directly in the API request path. They intercept create, update, and delete operations, and they can reject a request before it is persisted. This makes them a powerful policy and safety layer—but also a fragile one. A misconfigured admission webhook can block all deployments, break namespaces, or take down a cluster's ability to schedule workloads.

Backup, restore, and disaster recovery for admission controllers are not afterthoughts. They are operational prerequisites. A cluster without a tested recovery path for its admission pipeline is one bad manifest away from an outage.

This article is for developers, DevOps consultants, and technical startup teams who operate real clusters. It covers:

- Version and environment inventory for admission controllers

- Safe configuration paths, including webhook manifests and certificate handling

- Verification and diagnostics with concrete commands and expected outputs

- Failure modes and recovery workflows

- An operations checklist you can actually run

Every section includes practical commands, manifest snippets, and failure signals. The goal is operational safety: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document recovery before you need it.

## Version and Environment Inventory

Before touching an admission controller, know exactly what is running. Admission controllers come in two broad categories:

- Built-in admission controllers compiled into the Kubernetes API server, enabled with --enable-admission-plugins .

- Dynamic admission controllers (webhooks) registered via ValidatingWebhookConfiguration and MutatingWebhookConfiguration .

### What to inventory

Capture:

- Kubernetes cluster version ( kubectl version --short )

- API server flags on control plane nodes

- All webhook configurations

- The webhook services, deployments, and pods behind each webhook

- Webhook certificates and their expiry

- Any policies, custom resources, and namespaces involved

### Read-only observation first

Run these commands before making any change:

kubectl version --short
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
kubectl get apiservices | grep -E 'v1beta1.admission|v1.admission' 
 Expected output example (truncated):

NAME WEBHOOKS AGE
opa-validating-webhook 1 12d
image-policy-mutating-webhook 1 8d 
 For each webhook, inspect its rules, namespace selector, and client config:

kubectl describe validatingwebhookconfiguration opa-validating-webhook 
 Look for:

- rules : which resources and operations trigger the webhook

- namespaceSelector : which namespaces are affected

- clientConfig.service : the service name, namespace, path, and optional CA bundle

- failurePolicy : Fail or Ignore . If Fail , the API server rejects requests when the webhook is unavailable.

### Check the webhook backend health

Webhook backends are just pods. Use the standard Kubernetes troubleshooting sequence:

kubectl get pods -n webhook-system -o wide
kubectl describe pod <pod-name> -n webhook-system
kubectl logs <pod-name> -n webhook-system --previous
kubectl rollout status deployment/<webhook-deployment> -n webhook-system 
 Example of a healthy webhook deployment:

$ kubectl rollout status deployment/opa-webhook -n webhook-system
deployment "opa-webhook" successfully rolled out 
 If the deployment is stuck, inspect events and logs before changing anything.

### Certificates are part of the inventory

Dynamic admission webhooks require TLS. The CA bundle in the webhook configuration must match the certificate presented by the webhook server. Check certificate expiry:

kubectl get validatingwebhookconfiguration opa-validating-webhook -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -dates 
 Expected output:

notBefore=Mar 1 12:00:00 2024 GMT
notAfter=Mar 1 12:00:00 2025 GMT 
 If the cert is close to expiry, schedule rotation before it fails.

### Record state before change

Create a snapshot file:

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml > admission-webhooks-snapshot-$(date +%F).yaml 
 This snapshot is your quick rollback path. Store it in version control alongside your cluster configs.

<style>
.eno-quiz-widget{margin:2rem 0;padding:1.5rem;border-radius:12px;background:var(--eno-surface-lowest,#f7f7f8);border:1px solid var(--eno-border-soft,#e2e2e6);font-family:var(--eno-font-body,Inter,sans-serif)}
.eno-quiz-widget .eno-quiz-kicker{font-family:var(--eno-font-label,"JetBrains Mono",monospace);font-size:.75rem;letter-spacing:.05em;text-transform:uppercase;color:var(--eno-text-muted,#5f5f68);margin:0 0 .5rem}
.eno-quiz-widget .eno-quiz-question{font-family:var(--eno-font-heading,"Hanken Grotesk",sans-serif);font-size:1.0625rem;font-weight:600;margin:0 0 1rem;color:var(--eno-text-strong,#1a1a1f)}
.eno-quiz-widget .eno-quiz-options{list-style:none;margin:0;padding:0;display:flex;flex-direction:column;gap:.5rem}
.eno-quiz-widget .eno-quiz-option{display:block;width:100%;min-height:44px;text-align:left;padding:.625rem .875rem;border-radius:8px;border:1.5px solid var(--eno-border-soft,#e2e2e6);background:#fff;font-size:.9375rem;cursor:pointer;transition:border-color var(--eno-motion-base,180ms ease-out),background var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-option:hover{border-color:var(--eno-primary,#0059bb)}
.eno-quiz-widget .eno-quiz-option:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-option[aria-pressed="true"]{border-color:var(--eno-primary,#0059bb);background:rgb(0 89 187 / 0.06)}
.eno-quiz-widget .eno-quiz-option[data-correct="true"].eno-quiz-revealed{border-color:var(--eno-success,#17803a);background:rgb(23 128 58 / 0.08)}
.eno-quiz-widget .eno-quiz-option[data-correct="false"].eno-quiz-revealed.eno-quiz-was-selected{border-color:var(--eno-error,#ba1a1a);background:var(--eno-error-soft,#ffdad6)}
.eno-quiz-widget .eno-quiz-option-icon{display:inline-block;width:1.1em;margin-right:.4em;font-weight:700}
.eno-quiz-widget .eno-quiz-submit{margin-top:1rem;min-height:44px;padding:.5rem 1.25rem;border-radius:8px;border:none;background:var(--eno-primary,#0059bb);color:#fff;font-weight:600;font-size:.9375rem;cursor:pointer;transition:opacity var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-submit:disabled{opacity:.5;cursor:not-allowed}
.eno-quiz-widget .eno-quiz-submit:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-explanation{margin-top:1rem;padding:.875rem 1rem;border-radius:8px;font-size:.9375rem;line-height:1.5;display:none}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-visible{display:block}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-correct{background:rgb(23 128 58 / 0.08);color:var(--eno-success,#17803a)}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-incorrect{background:var(--eno-error-soft,#ffdad6);color:var(--eno-error,#ba1a1a)}
@media (prefers-reduced-motion: reduce){.eno-quiz-widget *{transition:none!important}}
</style><div class="eno-quiz-widget" role="group" aria-label="Quick check question">
Quick check 1 of 2

What are the two broad categories of admission controllers mentioned?

- Built-in admission controllers and dynamic admission controllers (webhooks)
- Validating and mutating admission controllers
- Policy-based and rule-based admission controllers
- Local and remote admission controllers

<button type="button" class="eno-quiz-submit" disabled>Submit</button>
<div class="eno-quiz-explanation">The article states that admission controllers come in two broad categories: Built-in admission controllers compiled into the Kubernetes API server, and dynamic admission controllers (webhooks) registered via ValidatingWebhookConfiguration and MutatingWebhookConfiguration.</div>
</div>
<script>
document.addEventListener('DOMContentLoaded', function () {
  document.querySelectorAll('.eno-quiz-widget:not([data-eno-quiz-bound])').forEach(function (widget) {
    widget.setAttribute('data-eno-quiz-bound', '1');
    var options = Array.prototype.slice.call(widget.querySelectorAll('.eno-quiz-option'));
    var submitBtn = widget.querySelector('.eno-quiz-submit');
    var explanation = widget.querySelector('.eno-quiz-explanation');
    var selected = null;
    options.forEach(function (opt) {
      opt.addEventListener('click', function () {
        if (widget.hasAttribute('data-eno-quiz-answered')) return;
        options.forEach(function (o) { o.setAttribute('aria-pressed', 'false'); });
        opt.setAttribute('aria-pressed', 'true');
        selected = opt;
        submitBtn.disabled = false;
      });
    });
    submitBtn.addEventListener('click', function () {
      if (!selected || widget.hasAttribute('data-eno-quiz-answered')) return;
      widget.setAttribute('data-eno-quiz-answered', '1');
      submitBtn.disabled = true;
      var correct = selected.getAttribute('data-correct') === 'true';
      options.forEach(function (o) {
        o.classList.add('eno-quiz-revealed');
        if (o === selected) o.classList.add('eno-quiz-was-selected');
        var icon = o.querySelector('.eno-quiz-option-icon');
        if (o.getAttribute('data-correct') === 'true') icon.textContent = '\u2713';
        else if (o === selected) icon.textContent = '\u2717';
      });
      explanation.classList.add('eno-quiz-visible', correct ? 'eno-quiz-correct' : 'eno-quiz-incorrect');
    });
  });
});
</script>

## Safe Configuration Path

Changing admission controllers is high-risk because a mistake can block workload creation cluster-wide. Follow a controlled path.

### Step 1: Define the change and blast radius

Before editing, document:

- Which webhook or flag is changing?

- Which resources and namespaces are affected?

- What is the failurePolicy?

- How will you verify success?

- What is the rollback command?

### Step 2: Use a copy, not the live object

Always work from a copy of the configuration:

kubectl get validatingwebhookconfiguration opa-validating-webhook -o yaml > opa-webhook-backup.yaml
cp opa-webhook-backup.yaml opa-webhook-new.yaml 
 Edit opa-webhook-new.yaml with your changes.

### Step 3: Pre-flight validation

Use kubectl apply --dry-run=client or --dry-run=server to validate syntax and server-side admission:

kubectl apply -f opa-webhook-new.yaml --dry-run=client 
 For more thorough validation, test with a minimal admission request using kubectl create dry-run on a dummy resource that triggers the webhook.

### Step 4: Apply and immediately verify

Apply the change:

kubectl apply -f opa-webhook-new.yaml 
 Then check the webhook is still working:

kubectl get validatingwebhookconfiguration opa-validating-webhook -o yaml
kubectl get pods -n webhook-system
kubectl logs <webhook-pod> -n webhook-system --tail=20 
 Create a test resource that should be admitted and one that should be rejected (if your policy has a denylist). For example, use a dry-run to avoid side effects:

kubectl create deployment test-admission --image=nginx --dry-run=client -o yaml | kubectl apply --dry-run=server -f - 
 If the request is rejected unexpectedly, check webhook logs and metrics.

### Step 5: Keep local testing minimal

For new webhook development, use a local cluster or a dedicated namespace with namespaceSelector limiting the webhook to that namespace. Example namespace selector in the webhook configuration:

namespaceSelector:
 matchLabels:
 admission-webhook: enabled 
 Then label only the test namespace:

kubectl label namespace test-ns admission-webhook=enabled 
 Verify traffic locally with kubectl port-forward to the webhook service before exposing it via a load balancer or ingress.

## Verification and Diagnostics

Verification is not "it applied without error". You must confirm the admission controller is actually intercepting requests and making correct decisions.

### 1. Check webhook configuration is active

kubectl get validatingwebhookconfigurations opa-validating-webhook -o jsonpath='{.metadata.name} {.webhooks[0].failurePolicy} {.webhooks[0].rules}' 
 Expected output shows rules and failurePolicy.

### 2. Test admission decisions with dry-run

Create a manifest that should pass and one that should fail. Use kubectl apply --dry-run=server to avoid persisting the resource.

Pass example:

cat <<EOF | kubectl apply --dry-run=server -f -
apiVersion: v1
kind: Pod
metadata:
 name: test-pass
spec:
 containers:
 - name: nginx
 image: nginx:1.21
EOF 
 Fail example (assuming policy requires specific label):

cat <<EOF | kubectl apply --dry-run=server -f -
apiVersion: v1
kind: Pod
metadata:
 name: test-fail
spec:
 containers:
 - name: nginx
 image: nginx:latest
EOF 
 Expected failure message:

Error from server (Forbidden): error when creating "STDIN": admission webhook "opa-validating-webhook.example.com" denied the request: image tag 'latest' not allowed 
 If the failure is not as expected, the webhook is not working correctly.

### 3. Inspect webhook logs and metrics

Most webhooks log admission review requests. Enable verbose logging if needed.

kubectl logs -n webhook-system deploy/opa-webhook --tail=50 
 Look for decisions and timestamps. For OPA, use opa eval or check the decision log.

Many webhook frameworks expose Prometheus metrics. Query metrics like apiserver_admission_webhook_request_total from the API server to see webhook calls:

kubectl get --raw /metrics | grep apiserver_admission_webhook_request_total 
 This shows counts per webhook and result.

### 4. Test API server flags for built-in admission controllers

If you changed --enable-admission-plugins , check the API server logs and current flags:

kubectl -n kube-system logs kube-apiserver-control-plane | grep admission 
 Or check the static pod manifest on the control plane node.

### 5. Validate certificate chain

If webhook TLS is misconfigured, API server logs show errors. Check:

kubectl -n kube-system logs kube-apiserver-control-plane | grep "x509" 
 Verify the CA bundle matches the webhook server cert using openssl .

<style>
.eno-quiz-widget{margin:2rem 0;padding:1.5rem;border-radius:12px;background:var(--eno-surface-lowest,#f7f7f8);border:1px solid var(--eno-border-soft,#e2e2e6);font-family:var(--eno-font-body,Inter,sans-serif)}
.eno-quiz-widget .eno-quiz-kicker{font-family:var(--eno-font-label,"JetBrains Mono",monospace);font-size:.75rem;letter-spacing:.05em;text-transform:uppercase;color:var(--eno-text-muted,#5f5f68);margin:0 0 .5rem}
.eno-quiz-widget .eno-quiz-question{font-family:var(--eno-font-heading,"Hanken Grotesk",sans-serif);font-size:1.0625rem;font-weight:600;margin:0 0 1rem;color:var(--eno-text-strong,#1a1a1f)}
.eno-quiz-widget .eno-quiz-options{list-style:none;margin:0;padding:0;display:flex;flex-direction:column;gap:.5rem}
.eno-quiz-widget .eno-quiz-option{display:block;width:100%;min-height:44px;text-align:left;padding:.625rem .875rem;border-radius:8px;border:1.5px solid var(--eno-border-soft,#e2e2e6);background:#fff;font-size:.9375rem;cursor:pointer;transition:border-color var(--eno-motion-base,180ms ease-out),background var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-option:hover{border-color:var(--eno-primary,#0059bb)}
.eno-quiz-widget .eno-quiz-option:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-option[aria-pressed="true"]{border-color:var(--eno-primary,#0059bb);background:rgb(0 89 187 / 0.06)}
.eno-quiz-widget .eno-quiz-option[data-correct="true"].eno-quiz-revealed{border-color:var(--eno-success,#17803a);background:rgb(23 128 58 / 0.08)}
.eno-quiz-widget .eno-quiz-option[data-correct="false"].eno-quiz-revealed.eno-quiz-was-selected{border-color:var(--eno-error,#ba1a1a);background:var(--eno-error-soft,#ffdad6)}
.eno-quiz-widget .eno-quiz-option-icon{display:inline-block;width:1.1em;margin-right:.4em;font-weight:700}
.eno-quiz-widget .eno-quiz-submit{margin-top:1rem;min-height:44px;padding:.5rem 1.25rem;border-radius:8px;border:none;background:var(--eno-primary,#0059bb);color:#fff;font-weight:600;font-size:.9375rem;cursor:pointer;transition:opacity var(--eno-motion-base,180ms ease-out)}
.eno-quiz-widget .eno-quiz-submit:disabled{opacity:.5;cursor:not-allowed}
.eno-quiz-widget .eno-quiz-submit:focus-visible{outline:none;box-shadow:0 0 0 3px rgb(0 89 187 / 0.15)}
.eno-quiz-widget .eno-quiz-explanation{margin-top:1rem;padding:.875rem 1rem;border-radius:8px;font-size:.9375rem;line-height:1.5;display:none}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-visible{display:block}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-correct{background:rgb(23 128 58 / 0.08);color:var(--eno-success,#17803a)}
.eno-quiz-widget .eno-quiz-explanation.eno-quiz-incorrect{background:var(--eno-error-soft,#ffdad6);color:var(--eno-error,#ba1a1a)}
@media (prefers-reduced-motion: reduce){.eno-quiz-widget *{transition:none!important}}
</style><div class="eno-quiz-widget" role="group" aria-label="Quick check question">
Quick check 2 of 2

What is the purpose of the failurePolicy field in a webhook configuration?

- It determines the HTTP status code to return on failure.
- It controls whether the API server rejects requests when the webhook is unavailable.
- It specifies the number of retries before the webhook is considered failed.
- It defines the timeout for webhook calls.

<button type="button" class="eno-quiz-submit" disabled>Submit</button>
<div class="eno-quiz-explanation">The reference says: &#39;failurePolicy: Fail or Ignore. If Fail, the API server rejects requests when the webhook is unavailable.&#39;</div>
</div>
<script>
document.addEventListener('DOMContentLoaded', function () {
  document.querySelectorAll('.eno-quiz-widget:not([data-eno-quiz-bound])').forEach(function (widget) {
    widget.setAttribute('data-eno-quiz-bound', '1');
    var options = Array.prototype.slice.call(widget.querySelectorAll('.eno-quiz-option'));
    var submitBtn = widget.querySelector('.eno-quiz-submit');
    var explanation = widget.querySelector('.eno-quiz-explanation');
    var selected = null;
    options.forEach(function (opt) {
      opt.addEventListener('click', function () {
        if (widget.hasAttribute('data-eno-quiz-answered')) return;
        options.forEach(function (o) { o.setAttribute('aria-pressed', 'false'); });
        opt.setAttribute('aria-pressed', 'true');
        selected = opt;
        submitBtn.disabled = false;
      });
    });
    submitBtn.addEventListener('click', function () {
      if (!selected || widget.hasAttribute('data-eno-quiz-answered')) return;
      widget.setAttribute('data-eno-quiz-answered', '1');
      submitBtn.disabled = true;
      var correct = selected.getAttribute('data-correct') === 'true';
      options.forEach(function (o) {
        o.classList.add('eno-quiz-revealed');
        if (o === selected) o.classList.add('eno-quiz-was-selected');
        var icon = o.querySelector('.eno-quiz-option-icon');
        if (o.getAttribute('data-correct') === 'true') icon.textContent = '\u2713';
        else if (o === selected) icon.textContent = '\u2717';
      });
      explanation.classList.add('eno-quiz-visible', correct ? 'eno-quiz-correct' : 'eno-quiz-incorrect');
    });
  });
});
</script>

## Failure Modes and Recovery

Admission controller failures can be silent or loud. Know the common failure modes and how to recover.

### Failure mode 1: Webhook service unavailable

Symptom: all API requests that match the webhook's rules fail with timeout or connection refused, but only if failurePolicy: Fail . If failurePolicy: Ignore , requests are allowed through silently.

Error example:

Error from server (InternalError): error when creating "deployment.yaml": Internal error occurred: failed calling webhook "opa-validating-webhook.example.com": Post "https://opa-webhook.webhook-system.svc:443/validate?timeout=10s": dial tcp 10.96.0.42:443: connect: connection refused 
 Recovery:

- Check if the webhook deployment is running:

kubectl get pods -n webhook-system 

- If not, scale up or restart:

 kubectl scale deployment opa-webhook --replicas=1 -n webhook-system 

- If the service selector is wrong, fix it.

- As an emergency escape hatch, temporarily set failurePolicy: Ignore or delete the webhook configuration:

 kubectl delete validatingwebhookconfiguration opa-validating-webhook 
 Warning : deleting the webhook removes policy enforcement. Restore from backup as soon as possible.

### Failure mode 2: Certificate expiry or mismatch

Symptom: API server logs show x509: certificate signed by unknown authority or certificate has expired . All matching requests fail.

Recovery:

- Generate new cert/key pair for the webhook server.

- Update the webhook deployment's TLS secret.

- Update the caBundle in the webhook configuration with the new CA.

- Apply the updated configuration.

Example to update CA bundle from a file:

CA_BUNDLE=$(cat ca.crt | base64 | tr -d '\n')
kubectl patch validatingwebhookconfiguration opa-validating-webhook --type='json' -p="[{'op': 'replace', 'path': '/webhooks/0/clientConfig/caBundle', 'value':'${CA_BUNDLE}'}]" 
 Then verify with a dry-run admission test.

### Failure mode 3: Overly broad webhook rules block all resources

Symptom: after applying a new webhook, everything fails, including system components.

Recovery:

- Immediately delete the webhook configuration:

kubectl delete validatingwebhookconfiguration <name> 

- Restore the previous version from your snapshot:

 kubectl apply -f admission-webhooks-snapshot-YYYY-MM-DD.yaml 

- Investigate the rules and namespaceSelector to narrow scope.

- Reapply a fixed version.

### Failure mode 4: Mutating webhook corrupts resources

 A mutating webhook can inject invalid patches, breaking applications. If you notice pods failing after a mutation, check:

kubectl get pods <name> -o yaml | grep -A5 'annotations' 
 Look for mutations that changed required fields.

Recovery:

- Disable the mutating webhook.

- Redeploy affected workloads.

- Fix the mutation logic and re-enable in a test namespace.

### Rollback workflow

Always keep a known-good snapshot. Apply rollback with:

kubectl apply -f admission-webhooks-known-good.yaml 
 Then monitor API server logs and admission metrics to confirm the baseline is restored.

## Operations Checklist

Use this checklist before and after any change to admission controllers.

### Before change

- [ ] Cluster version and webhook list captured in a document

- [ ] Current webhook configurations exported to YAML and stored in git

- [ ] Certificate expiry dates noted

- [ ] Webhook backend deployment and service health verified ( kubectl get pods , kubectl describe )

- [ ] Test namespace labeled for scoped rollout if possible

- [ ] Dry-run of the new configuration completed without errors

- [ ] Rollback snapshot accessible

- [ ] On-call notification or maintenance window arranged

### During change

- [ ] Apply one webhook change at a time

- [ ] Watch API server logs for errors

- [ ] Run admission dry-run tests for both allow and deny cases

- [ ] Monitor apiserver_admission_webhook_request_total metric

### After change

- [ ] Verify webhooks are still registered

- [ ] Run full admission test suite

- [ ] Check application creation in affected namespaces

- [ ] Record the change in runbook with timestamp and outcome

- [ ] Ensure rollback document is updated if needed

### Emergency response quick reference

<div class="my-stack-md overflow-x-auto">
<table class="min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Situation</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Immediate action</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">All resources being rejected</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Delete webhook configuration or set failurePolicy=Ignore</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Webhook backend down</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Scale up or restart deployment</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Certificate expired</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Rotate certs and update caBundle</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Mutating webhook corruption</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Disable mutating webhook, redeploy workloads</td></tr></tbody>
</table>
</div>

## Conclusion

Kubernetes admission controllers are powerful, but they require operational discipline. Backup and restore are not just about configuration files—they include certificates, webhook backend deployments, and test procedures.

This guide provides a practical workflow:

- Inventory your admission controllers and their state

- Make changes through a safe, scoped path with dry-runs and backups

- Verify behavior with concrete tests and metrics

- Prepare for common failures with explicit recovery steps

- Use an operations checklist to prevent mistakes

Start with one low-risk verification in a test namespace. Record current state, run the check, compare the result, and review dependencies. A reliable workflow makes failure visible, protects sensitive values, limits changes, and defines recovery before an incident forces the decision.

Next steps: pick a webhook you already run, export its configuration, review its failurePolicy and namespace selectors, and test a rollback with your snapshot. That exercise will show you exactly how prepared you are.