## Intro Admission webhooks are a critical control point in Kubernetes clusters. They intercept API requests for resources such as Pods, Deployments, and custom resources, and can mutate or validate them before they are persisted. When a webhook fails, it can block deployments, break auto-scaling, or silently weaken security policy. A structured operations checklist turns ad-hoc debugging into repeatable, low-risk procedures. This article is for developers, DevOps consultants, and technical startup teams who run or maintain admission webhooks in production. It covers version and environment inventory, safe configuration changes, verification and diagnostics, failure modes and recovery, and an operational checklist you can adapt to your cluster. Every step pairs a read-only observation with a concrete command, expected output, and a recovery decision. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before touching a webhook, you need to know exactly what is installed, which API version it uses, and how it is deployed. Start by finding all admission webhook configurations in the cluster. The read-only command is: ```bash kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations ``` Example output (abbreviated): ``` NAME WEBHOOKS AGE cert-manager-webhook 1 42d kyverno-resource-validating-webhook-cfg 2 12d pod-policy.example.com 1 3h ``` For each webhook configuration, inspect its YAML to see the webhook client configuration, failure policy, match rules, and the service reference. For example: ```bash kubectl get validatingwebhookconfiguration pod-policy.example.com -o yaml ``` Look for these key fields: - `failurePolicy`: `Fail` or `Ignore`. In production, `Fail` is often preferred for security, but it increases outage risk if the webhook is down. - `matchPolicy`: `Exact` or `Equivalent`. This affects whether rules apply to API groups and versions. - `timeoutSeconds`: often 10 seconds default. If your webhook is slow, requests may fail. - `clientConfig.service`: namespace, name, and port of the webhook service. Verify that the service exists and selects the correct pods. Check the deployed webhook server version and image. If your webhook runs as a Deployment: ```bash kubectl get deployment -n webhook-namespace pod-policy-webhook -o jsonpath='{.spec.template.spec.containers[*].image}' ``` Example output: ``` registry.example.com/team/pod-policy-webhook:v1.4.2 ``` Compare this with the version that is documented as supported for your Kubernetes cluster. Admission webhooks rely on the `admissionregistration.k8s.io` API, which is stable in v1 since Kubernetes 1.16. Most webhook frameworks require Kubernetes 1.19 or later for features like matchConditions (introduced in 1.28 as a beta). Check the webhook server logs to see if it is receiving requests and handling them without errors: ```bash kubectl logs -n webhook-namespace deployment/pod-policy-webhook --tail=50 ``` Pay attention to: - TLS errors, especially if the webhook service uses a self-signed certificate that has expired. - Panic stack traces or repeated 500 errors. - Slow request latencies, which may indicate the webhook is doing too much work synchronously. If the webhook uses a ValidatingWebhookConfiguration or MutatingWebhookConfiguration that references a service, confirm that the service endpoints are populated: ```bash kubectl get endpoints -n webhook-namespace pod-policy-webhook ``` Example output: ``` NAME ENDPOINTS AGE pod-policy-webhook 10.244.2.15:8443 42d ``` If ENDPOINTS is empty, the service selector does not match any pods, or the pods are not ready. For a complete environment inventory, record: - Kubernetes control plane version (`kubectl version --short`) - Webhook configuration names and failure policies - Webhook server image versions - TLS certificate expiry dates for the webhook serving certificate - Any relevant ResourceQuotas or LimitRanges that might affect webhook pod scheduling Practical checklist for Version and Environment Inventory: - [ ] List all mutating and validating webhook configurations with their failure policies. - [ ] Record the Kubernetes server and client versions. - [ ] Identify the webhook server image and confirm it is within the supported version range. - [ ] Verify the webhook service exists and has active endpoints. - [ ] Check the webhook server logs for TLS errors, panics, or repeated HTTP 500s. - [ ] Note the expiry date of the TLS certificate used by the webhook. - [ ] Document the namespace and deployment topology of the webhook server. ## Safe Configuration Path Changing a webhook configuration can have immediate cluster-wide impact. Follow a safe path: observe current state, stage the change in a local file, validate it, apply with a dry-run, and then gradually roll out. Start by exporting the current configuration to a file. This serves as a backup and a baseline for comparison. ```bash kubectl get validatingwebhookconfiguration pod-policy.example.com -o yaml > pod-policy-webhook-original.yaml ``` Make a copy to edit: ```bash cp pod-policy-webhook-original.yaml pod-policy-webhook-new.yaml ``` Edit `pod-policy-webhook-new.yaml` with your intended change. For example, suppose you want to tighten the failure policy from `Ignore` to `Fail` and increase the timeout from 10 to 15 seconds. Change only those fields, and leave everything else identical. Before applying, validate the YAML syntax and Kubernetes schema with `kubectl apply --dry-run=client`: ```bash kubectl apply -f pod-policy-webhook-new.yaml --dry-run=client ``` If there is a syntax error, you will see a message like: ``` error: error validating "pod-policy-webhook-new.yaml": error validating data: ValidationError(ValidatingWebhookConfiguration.webhooks[0].timeoutSeconds): invalid type for io.k8s.api.admissionregistration.v1.Webhook.timeoutSeconds: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false ``` Fix the error and repeat the dry-run until it succeeds. Then use `--dry-run=server` to simulate the API server's validation without persisting the change: ```bash kubectl apply -f pod-policy-webhook-new.yaml --dry-run=server ``` If the server accepts it, you will see: ``` validatingwebhookconfiguration.admissionregistration.k8s.io/pod-policy.example.com configured (server dry run) ``` Now you can apply the change. But to reduce risk, consider using a canary namespace or a subset of resources first. For example, if you are adding a new validation rule, you can scope it to a test namespace by adding a `namespaceSelector` or `objectSelector`. In the webhook YAML, add under the webhook: ```yaml namespaceSelector: matchLabels: kubernetes.io/metadata.name: test-namespace ``` Apply the change: ```bash kubectl apply -f pod-policy-webhook-new.yaml ``` Immediately verify that the webhook configuration was updated and that the API server can still process requests. Check the webhook server logs for new requests and any errors: ```bash kubectl logs -n webhook-namespace deployment/pod-policy-webhook --tail=20 ``` If you need to roll back, restore the original configuration: ```bash kubectl apply -f pod-policy-webhook-original.yaml ``` Then check that the rollback is effective by examining the configuration again. Always keep a copy of the last known good configuration in version control. For production webhooks, use a GitOps flow where changes are reviewed and applied automatically, but even manual changes should be traceable. Practical checklist for Safe Configuration Path: - [ ] Export the current webhook configuration to a local backup file. - [ ] Make a single, well-defined change in a copy of the file. - [ ] Validate with `kubectl apply --dry-run=client` and fix any syntax errors. - [ ] Validate with `kubectl apply --dry-run=server` to catch semantic issues. - [ ] If possible, roll out to a test namespace or subset using selectors. - [ ] Apply the change and immediately watch webhook server logs for errors. - [ ] Keep the previous configuration available for quick rollback. ## Verification and Diagnostics After applying a webhook change (or even before, when investigating a suspected problem), you need to verify that the webhook is operating correctly and diagnose any deviations. First, check that the webhook server is healthy. If your webhook exposes a health endpoint, query it directly from inside the cluster or via port-forward. For example, if your webhook uses the common `/healthz` endpoint: ```bash kubectl port-forward -n webhook-namespace service/pod-policy-webhook 8443:8443 & curl -k https://localhost:8443/healthz ``` Expected output if healthy: ``` OK ``` If the health check fails or the service is unresponsive, the problem is likely in the webhook server itself, not the configuration. Next, test admission behavior with a controlled object. Create a minimal Pod specification that you expect to be accepted or rejected based on the webhook's policy. For example, if your webhook requires a `team` label on every Pod, you can test with and without the label. Create a file `test-pod-no-label.yaml`: ```yaml apiVersion: v1 kind: Pod metadata: name: test-no-label spec: containers: - name: nginx image: nginx:1.25 ``` Attempt to create it: ```bash kubectl apply -f test-pod-no-label.yaml ``` If the webhook is working correctly with `failurePolicy: Fail` and the rule rejects missing labels, you will see an error like: ``` Error from server: admission webhook "pod-policy.example.com" denied the request: Pod "test-no-label" is invalid: missing required label 'team' ``` Now test with the label: ```yaml apiVersion: v1 kind: Pod metadata: name: test-with-label labels: team: platform spec: containers: - name: nginx image: nginx:1.25 ``` ```bash kubectl apply -f test-with-label.yaml ``` Expected output: ``` pod/test-with-label created ``` Clean up the test pods after verification: ```bash kubectl delete pod test-no-label test-with-label ``` To diagnose timeouts or slow responses, check the webhook's own metrics if it exposes them. Many webhook frameworks export Prometheus metrics. For example, if your webhook exposes metrics on port 8080: ```bash kubectl port-forward -n webhook-namespace deployment/pod-policy-webhook 8080:8080 & curl http://localhost:8080/metrics | grep webhook_request_duration ``` Look for high percentiles or errors. Also check the Kubernetes API server audit logs to see webhook call timing and status codes. If you have access to the control plane, search for lines containing `admission webhook` or `webhook` in the audit log. If the webhook uses a self-signed certificate, verify that the CA bundle in the webhook configuration matches the serving certificate. The `clientConfig.caBundle` field must contain the PEM-encoded CA certificate that signed the webhook's serving certificate. If the webhook server's certificate is rotated, the webhook configuration must be updated with the new CA bundle. You can check the certificate details using `openssl`: ```bash kubectl get validatingwebhookconfiguration pod-policy.example.com -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -text | grep -E 'Subject:|Not After' ``` Example output: ``` Subject: CN = pod-policy-webhook.webhook-namespace.svc Not After : May 5 12:00:00 2025 GMT ``` If the certificate is expired or the CA bundle does not match, the Kubernetes API server will refuse to call the webhook and will log error messages like `x509: certificate signed by unknown authority` or `certificate has expired`. Practical checklist for Verification and Diagnostics: - [ ] Confirm the webhook server health endpoint returns OK. - [ ] Test admission with a known good and known bad resource against the policy. - [ ] Observe the exact error message from the API server for denied requests. - [ ] Check webhook server metrics for high latency or error rates. - [ ] Review Kubernetes API server audit logs for webhook call failures. - [ ] Verify the CA bundle in the webhook configuration matches the server certificate and is not expired. ## Failure Modes and Recovery Admission webhooks can fail in several ways, each with its own symptoms and recovery steps. Understanding these failure modes helps you design a resilient system and react quickly. **1. Webhook server down or unreachable** Symptom: When creating or updating resources, the API server returns an error like: ``` Error from server (InternalError): error when creating "pod.yaml": Internal error occurred: failed calling webhook "pod-policy.example.com": failed to call webhook: Post "https://pod-policy-webhook.webhook-namespace.svc:8443/validate?timeout=10s": dial tcp 10.96.0.15:8443: connect: connection refused ``` Recovery: - Check if the webhook pods are running and ready: ```bash kubectl get pods -n webhook-namespace -l app=pod-policy-webhook ``` If no pods are ready, investigate the Deployment, StatefulSet, or DaemonSet that manages the webhook. - If the failure policy is `Fail`, temporarily switching it to `Ignore` can restore cluster operations while you fix the webhook. However, this bypasses validation or mutation and should be done only as an emergency measure with proper approval. **2. Webhook timeout** Symptom: API requests fail with a timeout error, such as: ``` Error from server: Timeout: request did not complete within requested timeout ``` Or the webhook server logs show requests taking longer than the configured `timeoutSeconds`. Recovery: - Increase the `timeoutSeconds` in the webhook configuration if the webhook legitimately needs more time, but be aware that this can block API requests longer. - Optimize the webhook code to reduce latency. For example, cache frequently used data or move expensive operations to asynchronous controllers. - If the webhook is under high load, scale up its replicas. **3. TLS certificate failure** Symptom: API server logs show TLS errors, and requests are denied. The webhook server may be running fine, but the API server cannot establish a trusted connection. Recovery: - Rotate the webhook serving certificate and update the `caBundle` in the webhook configuration with the new CA certificate. - If you use cert-manager or a similar tool, ensure the Certificate resource is valid and renews automatically. - Temporarily set failurePolicy to `Ignore` if you must allow requests while fixing the certificate, but this is a security risk. **4. Webhook returns an error or denies unexpectedly** Symptom: Resource creation is denied with a webhook-specific error message. The API server passes through the error from the webhook. Recovery: - Read the webhook server logs to identify the rule or code path that returned the error. - If the rejection is due to a bug in the webhook policy, fix the policy and redeploy the webhook. - Test with different resources to isolate whether the issue is specific to certain kinds or namespaces. **5. Webhook misconfiguration** Symptom: The webhook matches resources it should not, or fails to match resources it should. This can happen due to incorrect `rules`, `namespaceSelector`, or `objectSelector`. Recovery: - Review the webhook configuration against the intended scope. - Use `kubectl describe` on the webhook configuration to see a human-readable summary: ```bash kubectl describe validatingwebhookconfiguration pod-policy.example.com ``` Example output snippet: ``` Name: pod-policy.example.com Namespace: Labels: Annotations: API Version: admissionregistration.k8s.io/v1 Kind: ValidatingWebhookConfiguration Webhooks: Name: pod-policy.webhook.example.com Client Config: Ca Bundle: Service: Name: pod-policy-webhook Namespace: webhook-namespace Port: 8443 Rules: API Groups: [""] API Versions: ["v1"] Operations: ["CREATE"] Resources: ["pods"] Scope: "Namespaced" ``` - Update the rules or selectors and re-apply with the safe configuration path described above. Recovery strategies should be documented in your runbooks. Include the exact commands to roll back a webhook change, restore a previous certificate, or disable a webhook in an emergency. For example, to temporarily disable a webhook without deleting it, you can set its `failurePolicy` to `Ignore` and remove all its rules, or simply delete the webhook configuration (which stops all admission calls). Emergency recovery snippet: ```bash kubectl delete validatingwebhookconfiguration pod-policy.example.com ``` This will remove the webhook entirely, allowing all requests to proceed. You can later re-create it from your version-controlled manifest. Practical checklist for Failure Modes and Recovery: - [ ] Identify the failure mode from API server errors and webhook logs. - [ ] If webhook pods are down, restart them or scale up. - [ ] If timeout is the issue, consider increasing `timeoutSeconds` or optimizing the webhook. - [ ] If TLS fails, rotate certificates and update CA bundle. - [ ] If misconfiguration is suspected, review rules and selectors. - [ ] Have a documented emergency rollback or disable procedure. - [ ] Test the recovery procedure in a non-production cluster first. ## Operations Checklist Use this daily or weekly operations checklist to keep admission webhooks healthy. Adapt the commands to your webhook names and namespaces. 1. **List all webhooks and their statuses** - Command: `kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations` - Expected: all webhooks listed with no error; note any webhook whose AGE is unexpectedly recent (may indicate a recent change). 2. **Check webhook server health** - Command: `kubectl get pods -n webhook-namespace -l app=pod-policy-webhook -o wide` - Expected: all pods Running and Ready (1/1 or more). If any pod is not ready, investigate with `kubectl describe pod` and `kubectl logs`. 3. **Verify service endpoints** - Command: `kubectl get endpoints -n webhook-namespace pod-policy-webhook` - Expected: at least one endpoint IP:port. Empty endpoints mean no pods are serving. 4. **Inspect webhook configuration for drift** - Command: `kubectl get validatingwebhookconfiguration pod-policy.example.com -o yaml | diff - pod-policy-webhook-original.yaml` - Expected: no differences unless a change was authorized. If differences exist, review and decide whether to accept or roll back. 5. **Check webhook logs for new errors** - Command: `kubectl logs -n webhook-namespace deployment/pod-policy-webhook --tail=100 | grep -i -E 'error|panic|timeout'` - Expected: no output or only known benign errors. Investigate any new error lines. 6. **Verify certificate expiry** - Command: `kubectl get validatingwebhookconfiguration pod-policy.example.com -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -enddate` - Expected: a future date, ideally more than 30 days away. If less, schedule certificate rotation. 7. **Test admission with a sample resource** - Command: apply a test Pod with expected accept/reject, then delete it. - Expected: webhook denies when policy is violated and allows when compliant. 8. **Review metrics for anomalies** - If metrics are exposed, query for request rate, latency, and error rate, and compare with baseline. - Expected: no sudden spikes in 5xx errors or latency. 9. **Check resource usage of webhook pods** - Command: `kubectl top pods -n webhook-namespace -l app=pod-policy-webhook` - Expected: CPU and memory within limits. High memory may indicate a memory leak. 10. **Ensure GitOps or config management is in sync** - Compare the live webhook configuration with the source of truth in your repo. - Expected: no uncommitted changes in production. For each item, record the timestamp, the person performing the check, and any anomalies found. This creates an audit trail and helps spot trends. ## Conclusion Admission webhooks are powerful but risky if not operated carefully. This checklist gives you a systematic way to inventory, change, verify, and recover webhook configurations in production. Every step is version-scoped, observable, and reversible where the technology permits. Start with one low-risk verification: list your webhooks, check their health, and test one admission rule with a sample resource. Record the current state, run the documented checks, compare results with expected signals, and review dependencies such as service endpoints, TLS certificates, and RBAC permissions. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By following these practices, you keep your admission webhooks stable, secure, and maintainable.