## Intro Admission webhooks are a powerful Kubernetes extension point that lets you intercept API requests before objects are persisted. They enable validation, mutation, and policy enforcement that built-in admission controllers cannot handle. Yet many operators struggle to move from a vague idea to a verified production setup. This article is a practical deep dive into advanced admission webhook concepts. You will learn how to deploy a webhook server, configure it to validate or mutate resources, handle failure modes, and troubleshoot common problems. We cover internals, architecture, and real examples with commands and expected output. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. ## Version and Environment Inventory Before touching webhooks, confirm your cluster version and capabilities. Admission webhooks are GA since Kubernetes 1.9 and work on any conformant cluster. Check the API server version and enabled admission plugins: ```bash kubectl version --short # Example output: # Client Version: v1.28.2 # Server Version: v1.28.2 ``` ```bash kubectl get --raw /apis/admissionregistration.k8s.io/v1 | jq . # This should return API resources including validatingwebhookconfigurations and mutatingwebhookconfigurations ``` The webhook server can run inside the cluster or outside. If inside, it must be reachable by the API server via a Service. Check that the Service and its endpoints exist: ```bash kubectl get svc webhook-service -n webhook-system # NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE # webhook-service ClusterIP 10.96.100.10 443/TCP 5m kubectl get endpoints webhook-service -n webhook-system # NAME ENDPOINTS AGE # webhook-service 172.17.0.5:8443 5m ``` If the endpoint is empty, your webhook server pods are not ready or labels do not match the Service selector. Inspect pods: ```bash kubectl get pods -n webhook-system -o wide # NAME READY STATUS RESTARTS AGE IP NODE # webhook-server-6f6d5b7c9-abcde 1/1 Running 0 5m 172.17.0.5 kind-worker ``` If a pod is pending, describe it to see scheduling issues: ```bash kubectl describe pod webhook-server-6f6d5b7c9-abcde -n webhook-system # Events: # Type Reason Age From Message # ---- ------ ---- ---- ------- # Warning FailedScheduling 2m default-scheduler 0/3 nodes are available: 3 Insufficient cpu. ``` Before applying any webhook configuration, take a snapshot of existing configurations so you can roll back: ```bash kubectl get validatingwebhookconfigurations -o yaml > validating-backup.yaml kubectl get mutatingwebhookconfigurations -o yaml > mutating-backup.yaml ``` Store these backups outside the cluster if possible. If your webhook breaks the cluster, you may need to delete the configuration urgently (see Failure Modes). ## Webhook Server Setup A webhook server is just an HTTPS server that handles AdmissionReview requests. It must present a TLS certificate signed by a CA trusted by the API server. The `caBundle` field in the webhook configuration must contain that CA certificate in PEM format, base64-encoded. A straightforward approach is to use a self-signed CA and generate certificates for the server. You can use `openssl` or `cert-manager`. For testing, use a local tool like `kind` or `minikube` where you can create a cert and patch the webhook config easily. Here is a minimal Go webhook server skeleton: ```go package main import ( "encoding/json" "fmt" "io/ioutil" "net/http" admissionv1 "k8s.io/api/admission/v1" ) func handleValidate(w http.ResponseWriter, r *http.Request) { body, _ := ioutil.ReadAll(r.Body) var admissionReview admissionv1.AdmissionReview json.Unmarshal(body, &admissionReview) // Build response admissionResponse := &admissionv1.AdmissionResponse{ UID: admissionReview.Request.UID, Allowed: true, } response := admissionv1.AdmissionReview{ TypeMeta: admissionv1.TypeMeta{ APIVersion: "admission.k8s.io/v1", Kind: "AdmissionReview", }, Response: admissionResponse, } respBytes, _ := json.Marshal(response) w.Write(respBytes) } func main() { http.HandleFunc("/validate", handleValidate) http.ListenAndServeTLS(":8443", "/tls/tls.crt", "/tls/tls.key", nil) } ``` This server allows every request. A real server will inspect the request object, apply policies, and return an `Allowed` boolean and optional `Warnings` or `Status` message. ## Configuring a ValidatingWebhook A ValidatingWebhookConfiguration tells the API server which requests to send to your webhook. Here is an example that validates that every pod in the `production` namespace has an `app` label: ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: name: require-app-label webhooks: - name: require-app-label.example.com clientConfig: service: namespace: webhook-system name: webhook-service path: "/validate" port: 443 caBundle: rules: - operations: ["CREATE", "UPDATE"] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"] scope: "Namespaced" namespaceSelector: matchExpressions: - key: kubernetes.io/metadata.name operator: In values: ["production"] admissionReviewVersions: ["v1"] sideEffects: None failurePolicy: Fail ``` Apply it: ```bash kubectl apply -f validating-webhook.yaml ``` Then test with a pod lacking the label: ```bash kubectl run nginx --image=nginx -n production # Error from server (Forbidden): error when creating "STDIN": admission webhook "require-app-label.example.com" denied the request: missing app label ``` If the webhook server is down and `failurePolicy` is `Fail`, all pod creations in the production namespace will be blocked. That is why we set `failurePolicy: Ignore` in many production setups, but that can allow bad objects. Always consider the tradeoff. ## Configuring a MutatingWebhook A MutatingWebhookConfiguration modifies objects before they are persisted. A common use is to inject default values or sidecars. Here is an example that adds a `team` label to every pod if missing: ```yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: add-team-label webhooks: - name: add-team-label.example.com clientConfig: service: namespace: webhook-system name: webhook-service path: "/mutate" port: 443 caBundle: rules: - operations: ["CREATE"] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"] namespaceSelector: matchLabels: webhook-enabled: "true" admissionReviewVersions: ["v1"] sideEffects: None reinvocationPolicy: IfNeeded failurePolicy: Ignore ``` In your mutating webhook code, you decode the incoming object, modify it, and return a JSONPatch: ```go patch := []map[string]interface{}{ { "op": "add", "path": "/metadata/labels/team", "value": "platform", }, } patchBytes, _ := json.Marshal(patch) admissionResponse.Patch = patchBytes admissionResponse.PatchType = new(admissionv1.PatchType) *admissionResponse.PatchType = admissionv1.PatchTypeJSONPatch ``` After applying, create a pod in a namespace labeled `webhook-enabled=true`: ```bash kubectl create ns test kubectl label ns test webhook-enabled=true kubectl run nginx --image=nginx -n test kubectl get pod nginx -n test --show-labels # NAME READY STATUS RESTARTS AGE LABELS # nginx 1/1 Running 0 5s team=platform ``` ## Advanced Concepts: Matching, Timeout, and Side Effects Webhook configurations support fine-grained matching using `objectSelector` to filter based on object labels. For example, to only validate pods with label `env=prod`: ```yaml objectSelector: matchLabels: env: prod ``` The `timeoutSeconds` field (default 10, max 30) controls how long the API server waits for your webhook response. A slow webhook can delay all matching requests. Set it explicitly: ```yaml timeoutSeconds: 5 ``` Test with a slow webhook: simulate a 10-second delay. You will see API server logs like: ``` E1110 10:00:00.123456 1 dispatcher.go:200] failed calling webhook "slow.example.com": Post "https://webhook-service.webhook-system.svc:443/validate": context deadline exceeded ``` Set `sideEffects` properly. `None` means the webhook does not make outbound changes that need to be considered when dry-running. If your webhook calls external systems, use `sideEffects: Some` or `Unknown`. The API server uses this to decide whether to allow dry-run requests. ## Verification and Diagnostics Verifying webhook behavior is critical. Here is a systematic approach. First, confirm the webhook configuration was created correctly: ```bash kubectl get validatingwebhookconfiguration require-app-label -o yaml # Check that caBundle is populated, rules match, and failurePolicy is as expected. ``` Check the webhook server logs for incoming requests: ```bash kubectl logs deployment/webhook-server -n webhook-system --tail=50 # Look for admission requests with UID and object info. ``` If the webhook is not being called, verify the namespace selector and object selector. You can use `kubectl api-resources` to check API groups and resources. Enable audit logging on the API server if possible. Admission webhook calls appear in audit logs with `annotations` including `admissionwebhook` information. For a denied request, the API server returns a clear error message from the webhook's `status.message`. If the message is not helpful, improve your webhook's error responses. Use `kubectl auth can-i` to simulate permissions? Not directly for webhooks, but you can create a test object with a dry-run: ```bash kubectl apply -f test-pod.yaml --dry-run=server # If dry-run fails due to webhook, the webhook was invoked successfully. ``` ## Failure Modes and Recovery Webhooks can break your cluster if misconfigured. Common failure modes include: 1. **Webhook server unavailable**: If the server is down and `failurePolicy: Fail`, requests to matching resources are blocked. Recovery: temporarily set `failurePolicy: Ignore` or delete the webhook configuration. ```bash kubectl delete validatingwebhookconfiguration require-app-label ``` But be careful: deleting the webhook allows invalid objects. Have a fix ready. 2. **CA certificate mismatch or expiry**: If the `caBundle` is wrong or expired, the API server cannot verify the webhook server certificate. Check API server logs for `x509: certificate signed by unknown authority`. Renew the certificate and update the `caBundle` in all webhook configurations. 3. **Timeout**: If the webhook takes too long, requests fail with timeout errors. Increase `timeoutSeconds` or optimize the webhook code. Monitor webhook latency using metrics. 4. **Missing RBAC for webhook server**: If the webhook server needs to read resources, ensure its ServiceAccount has proper RBAC. Test by checking webhook server logs for `forbidden` errors. 5. **Port mismatch**: Ensure the Service targetPort matches the container port and the webhook config port. Use `kubectl describe` to verify. 6. **Namespace selector misconfiguration**: If using `namespaceSelector`, test with a namespace that matches. Use `kubectl label namespace` to add labels. Recovery steps should be planned before deployment. Always back up existing configurations, and practice deleting a problematic webhook to restore cluster functionality quickly. ## Operations Checklist Before deploying a webhook to production, go through this checklist: - [ ] Webhook server runs as a Deployment with at least 2 replicas for high availability. - [ ] PodDisruptionBudget is set to ensure availability during node drains. - [ ] NetworkPolicy allows API server to reach the webhook Service (port 443). - [ ] TLS certificate validity is monitored (e.g., via cert-manager or custom alerts). - [ ] `failurePolicy` is explicitly set and documented. For critical validation, use `Fail`; for non-critical mutation, use `Ignore`. - [ ] `timeoutSeconds` is tuned to avoid unnecessary request delays. - [ ] Webhook configuration is versioned in Git and applied via CI/CD. - [ ] Rollback procedure is tested: deleting the webhook configuration restores normal API behavior within seconds. - [ ] Logs and metrics are collected from the webhook server for troubleshooting. - [ ] Admission webhook is included in disaster recovery drills. After deployment, verify with a series of test requests, both valid and invalid, to ensure the webhook behaves as expected. ## Common Pitfalls and How to Avoid Them Many teams hit the same issues with webhooks. Here are the top pitfalls and solutions. **Pitfall 1: Blocking all resources due to broad rules.** If you set `rules` with `resources: ["*"]` and `failurePolicy: Fail`, a webhook failure can make the cluster unusable, even preventing you from deleting the webhook configuration itself. Avoid broad rules. Instead, target specific resources and namespaces. If you must use broad rules, set `failurePolicy: Ignore` and monitor errors. **Pitfall 2: Forgetting to update `caBundle` after certificate rotation.** Certificates expire. Set up automated renewal and update the webhook configurations. Use tools like cert-manager with a CA injector, or a controller that patches webhook configs. Otherwise, you will see `x509: certificate expired` errors. **Pitfall 3: Mutation loops.** If a mutating webhook modifies an object in a way that triggers the same webhook again, you can create an infinite loop. Kubernetes has a safeguard: `reinvocationPolicy` and it avoids re-invoking the same webhook on the same object if the object has not changed? Actually, the API server tracks mutations and may re-invoke if other webhooks changed the object, but a webhook that keeps modifying the same object can cause repeated patches. Ensure your webhook is idempotent: check if the desired state already exists before patching. **Pitfall 4: Ignoring dry-run support.** If your webhook has side effects, such as calling an external API, you must set `sideEffects: Some` and handle dry-run requests. Otherwise, dry-run operations could cause real side effects. In your webhook code, check `admissionReview.Request.DryRun` and skip side-effecting actions if true. **Pitfall 5: Not setting resource requests/limits on the webhook server.** The webhook server sits in the request path. If it is starved for CPU or memory, admission latency increases. Set appropriate requests and limits, and monitor with HPA. **Pitfall 6: Using self-signed certs without proper CA distribution.** In production, use a proper PKI. If you use self-signed, ensure the CA is securely stored and included in webhook config. Avoid copying CA from a local machine without versioning. By anticipating these pitfalls, you can design a webhook system that is robust and maintainable. ## Conclusion Admission webhooks are essential for enforcing Kubernetes policies and automating resource mutation. This article walked through setting up a webhook server, configuring validating and mutating webhooks, handling failure modes, and avoiding common mistakes. As a next step, choose one low-risk validation rule, deploy a webhook in a test cluster, and run through the verification checklist. Record the current state, apply the webhook, test with both allowed and denied resources, and ensure you can roll back quickly. A reliable webhook implementation makes failure visible, protects sensitive data, limits changes to the intended resources, and defines recovery steps before an incident forces the decision.