## Intro Kubernetes admission webhooks are powerful extension points that let you validate or mutate resources before they are persisted. However, testing them directly on a shared cluster is risky: a misconfigured webhook can block deployments, cause cascading failures, or even make the cluster partially inoperable. A local lab environment gives you a safe, repeatable sandbox to experiment with webhook logic, understand failure modes, and build confidence before rolling out to production. In this guide, we will set up a local lab using lightweight tools like kind or minikube, create both a validating and a mutating admission webhook with practical examples, verify their behavior, and cover recovery steps. By following this guide, you will have a fully functional environment to develop and test admission webhooks without risking shared infrastructure. A clear setup reduces rework by separating the stages: environment preparation, certificate management, webhook deployment, and testing. We will start with a narrow, measurable example to ensure every step is observable. ## Version and Environment Inventory Before starting, confirm the versions and tools you will use. For this guide, we assume the following: | Component | Version | Purpose | |-----------|---------|---------| | Kubernetes | v1.27+ | Local cluster API server | | kind | v0.20.0+ | Lightweight local cluster | | or minikube | v1.31.0+ | Alternative local cluster | | OpenSSL | 3.0+ | Generate TLS certificates | | Go | 1.20+ | Write the webhook server | | curl | any recent | Test webhook endpoints | Prerequisites: - A machine with at least 2 CPUs and 4GB RAM available. - Docker or a container runtime supported by kind/minikube. - Basic familiarity with Kubernetes manifests and command-line tools. Topology: We will create a single-node cluster where the API server runs inside a container (kind) or a VM (minikube). The webhook server will be deployed as a pod inside the same cluster and exposed via a service. The API server will call the webhook over HTTPS using a self-signed certificate. Ensure your local cluster is up: ```bash kind create cluster --name webhook-lab ``` Expected output: ``` Creating cluster "webhook-lab" ... ✓ Ensuring node image (kindest/node: v1.27.3) 🖼 ✓ Preparing nodes 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 Set kubectl context to "kind-webhook-lab" ``` If using minikube: ```bash minikube start --driver=docker --kubernetes-version=v1.27.3 ``` Verify cluster access: ```bash kubectl cluster-info ``` ## Safe Configuration Path Now we will implement two webhooks: a mutating webhook that adds a label to every pod, and a validating webhook that rejects pods with a specific label value. This scoped approach lets us verify each piece independently. ### Step 1: Generate TLS Certificates The API server must trust the webhook server. Since we are in a lab, we can use a self-signed CA and issue a server certificate. The webhook configuration will specify the CA bundle. Run these commands to generate the certificates (using a temporary directory): ```bash mkdir -p /tmp/webhook-certs && cd /tmp/webhook-certs # Generate CA key and certificate openssl genrsa -out ca.key 2048 openssl req -x509 -new -nodes -key ca.key -subj "/CN=webhook-ca" -days 365 -out ca.crt # Generate server key and CSR openssl genrsa -out server.key 2048 openssl req -new -key server.key -subj "/CN=webhook-server.default.svc" -out server.csr # Sign the server certificate with our CA openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365 ``` The certificate's CN must match the service DNS name: `webhook-server.default.svc`. Create a Kubernetes secret with the server certificate and key: ```bash kubectl create secret tls webhook-server-cert --cert=server.crt --key=server.key ``` You will also need the base64-encoded CA certificate for the webhook configuration. We will use it when creating the webhook configuration. ### Step 2: Deploy the Webhook Server Create a simple HTTP server that responds to admission review requests. Below is an example in Go. Save it as `main.go`: ```go package main import ( "encoding/json" "fmt" "io/ioutil" "net/http" "crypto/tls" admissionv1 "k8s.io/api/admission/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) func mutatePods(w http.ResponseWriter, r *http.Request) { var admissionReview admissionv1.AdmissionReview body, _ := ioutil.ReadAll(r.Body) if err := json.Unmarshal(body, &admissionReview); err != nil { http.Error(w, "could not parse admission review", http.StatusBadRequest) return } // Build the patch to add a label patchType := admissionv1.PatchTypeJSONPatch pod := corev1.Pod{} if err := json.Unmarshal(admissionReview.Request.Object.Raw, &pod); err != nil { http.Error(w, "could not unmarshal pod", http.StatusBadRequest) return } // If labels are nil, create an empty map if pod.Labels == nil { pod.Labels = map[string]string{} } pod.Labels["webhook-lab"] = "mutated" // Create patch patch := []map[string]interface{}{ {"op": "add", "path": "/metadata/labels/webhook-lab", "value": "mutated"}, } patchBytes, _ := json.Marshal(patch) admissionResponse := &admissionv1.AdmissionResponse{ UID: admissionReview.Request.UID, Allowed: true, PatchType: &patchType, Patch: patchBytes, } response := admissionv1.AdmissionReview{ TypeMeta: metav1.TypeMeta{ APIVersion: "admission.k8s.io/v1", Kind: "AdmissionReview", }, Response: admissionResponse, } respBytes, _ := json.Marshal(response) w.Header().Set("Content-Type", "application/json") w.Write(respBytes) } func main() { http.HandleFunc("/mutate", mutatePods) server := &http.Server{ Addr: ":8443", TLSConfig: &tls.Config{ Certificates: []tls.Certificate{loadX509KeyPair("/etc/webhook/certs/tls.crt", "/etc/webhook/certs/tls.key")}, }, } fmt.Println("Starting webhook server on :8443") server.ListenAndServeTLS("", "") } func loadX509KeyPair(certFile, keyFile string) tls.Certificate { cert, err := tls.LoadX509KeyPair(certFile, keyFile) if err != nil { panic(err) } return cert } ``` Build the Docker image and load it into kind: ```bash docker build -t webhook-server:latest . kind load docker-image webhook-server:latest --name webhook-lab ``` Create a deployment and service: ```yaml # webhook-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: webhook-server namespace: default spec: replicas: 1 selector: matchLabels: app: webhook-server template: metadata: labels: app: webhook-server spec: containers: - name: server image: webhook-server:latest imagePullPolicy: IfNotPresent ports: - containerPort: 8443 volumeMounts: - name: certs mountPath: /etc/webhook/certs readOnly: true volumes: - name: certs secret: secretName: webhook-server-cert ``` ```yaml # webhook-service.yaml apiVersion: v1 kind: Service metadata: name: webhook-server namespace: default spec: selector: app: webhook-server ports: - port: 443 targetPort: 8443 ``` Apply: ```bash kubectl apply -f webhook-deployment.yaml kubectl apply -f webhook-service.yaml ``` Verify the pod is running: ```bash kubectl get pods -l app=webhook-server ``` Expected output: ``` NAME READY STATUS RESTARTS AGE webhook-server-79c4f4c9b7-9x2kq 1/1 Running 0 5s ``` ### Step 3: Create the MutatingWebhookConfiguration Create a configuration that tells the API server to call our webhook when pods are created. Replace `CA_BUNDLE` with the base64 of `ca.crt`: ```bash CA_BUNDLE=$(cat /tmp/webhook-certs/ca.crt | base64 | tr -d '\n') ``` ```yaml # mutating-webhook.yaml apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: pod-mutator webhooks: - name: pod-mutator.default.svc clientConfig: service: name: webhook-server namespace: default path: "/mutate" port: 443 caBundle: ${CA_BUNDLE} rules: - operations: ["CREATE"] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"] admissionReviewVersions: ["v1"] sideEffects: None failurePolicy: Fail ``` Apply: ```bash envsubst < mutating-webhook.yaml | kubectl apply -f - ``` ### Step 4: Test the Mutating Webhook Create a simple pod: ```bash kubectl run test-pod --image=nginx ``` Check labels: ```bash kubectl get pod test-pod --show-labels ``` Expected output includes `webhook-lab=mutated`. ### Step 5: Add a Validating Webhook Now add a validating webhook that rejects pods if the label `webhook-lab` is set to `mutated` (to demonstrate denial). You can extend the server or create a separate one. For brevity, assume a similar server with a `/validate` endpoint. The configuration: ```yaml # validating-webhook.yaml apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: name: pod-validator webhooks: - name: pod-validator.default.svc clientConfig: service: name: webhook-server namespace: default path: "/validate" port: 443 caBundle: ${CA_BUNDLE} rules: - operations: ["CREATE"] apiGroups: [""] apiVersions: ["v1"] resources: ["pods"] admissionReviewVersions: ["v1"] sideEffects: None failurePolicy: Fail ``` Apply and test: the pod creation should be denied with a message. ## Verification and Diagnostics To verify the webhooks are working correctly, perform these checks: - Check webhook configurations exist: ```bash kubectl get mutatingwebhookconfigurations, validatingwebhookconfigurations ``` - Inspect webhook logs: ```bash kubectl logs deployment/webhook-server ``` Expected to see incoming admission review requests. - Use `kubectl apply --dry-run=server` to see if the webhook is called without persisting the object. - Test with a dry-run: ```bash kubectl run test-pod --image=nginx --dry-run=server ``` - Use `kubectl describe` on the webhook configuration to see the CA bundle and rules. If the webhook is not being called, check the API server logs (for kind, see container logs) for error messages like `x509: certificate signed by unknown authority` or `connection refused`. ## Failure Modes and Recovery Admission webhooks can cause problems. Common failure modes: - **Webhook server unavailable:** If `failurePolicy` is `Fail`, all matching requests will be rejected. To recover, you can either delete the webhook configuration or scale the deployment to restore availability. If the API server cannot be reached to delete the webhook (unlikely but possible), you may need to edit the API server manifest to disable admission plugins. - **Incorrect CA bundle:** The API server will reject the webhook call, and with `failurePolicy: Fail`, resource creation fails. Fix the CA bundle in the webhook configuration. - **Timeout:** If the webhook server does not respond within the timeout (default 10 seconds), the API server will apply the failure policy. Recovery steps: ```bash # Temporarily disable webhook by deleting its configuration kubectl delete mutatingwebhookconfiguration pod-mutator kubectl delete validatingwebhookconfiguration pod-validator ``` Then fix the underlying issue and reapply. Rollback example: If a new webhook causes issues, roll back to a previous version by reapplying the old configuration or using `kubectl apply` with the known good manifest. ## Common Pitfalls and How to Avoid Them Beyond the failure modes already covered, several other pitfalls can trip you up when working with admission webhooks. Being aware of these will save you debugging time. - **Mismatched `admissionReviewVersions`:** The API server sends admission requests in the versions listed in the webhook configuration. If your webhook server only understands v1beta1 but the configuration lists v1, the request will fail. Always declare at least v1 and ensure your server handles it. Test with a simple echo endpoint to see what version is sent. - **Forgetting `sideEffects: None`:** If your webhook has no side effects (it does not modify external state), set `sideEffects: None`. This allows the API server to call it during dry-run requests and prevents it from being skipped when the dry-run feature is used. If your webhook does have side effects, document them and consider `sideEffects: NoneOnDryRun`. - **Overly broad rules:** Matching all resources with `*/*` can cause the webhook to interfere with system components. Scope your rules to the minimum needed, such as specific resources, groups, and operations. For example, instead of matching all pods, match only pods with a specific label or in a particular namespace. - **Missing namespace in clientConfig:** If you omit `namespace` in `clientConfig.service`, the API server assumes the same namespace as the webhook configuration, which is often not where your service lives. Always specify the namespace explicitly. - **Certificate renewal:** Self-signed certificates have an expiration date. If the certificate expires, the webhook will fail. In production, use a certificate management solution like cert-manager with automatic renewal. In the lab, note the expiration date and regenerate when needed. - **Ignoring webhook timeouts:** The default timeout is 10 seconds, but some operations may take longer. Set an appropriate `timeoutSeconds` in the webhook configuration and ensure your webhook responds within that time. If your webhook needs more time, adjust the timeout, but be cautious about slowing down all matching requests. ## Operations Checklist Use this checklist for your local lab and future production use. Each task has a single accountable owner and a review cadence. In a production setting, assign an owner such as the platform engineering lead. | Task | Command / Action | Expected Result | Owner | Review Frequency | |------|------------------|-----------------|-------|------------------| | Verify webhook pods running | `kubectl get pods -l app=webhook-server` | 1/1 Running | DevOps engineer on call | Every deployment, or weekly | | Check webhook configurations | `kubectl get mutatingwebhookconfigurations, validatingwebhookconfigurations` | List shows configurations | Platform lead | Monthly | | Test mutating webhook | Create pod and inspect labels | Label added | Developer | Per change | | Test validating webhook | Create pod with forbidden label | Request rejected | Developer | Per change | | Monitor webhook logs | `kubectl logs deployment/webhook-server` | Admission requests logged | SRE | Daily during incident, weekly otherwise | | Check API server logs for errors | Varies by cluster | No TLS or connection errors | Cluster admin | Weekly | | Test failure policy | Scale webhook deployment to 0, create pod | Pod creation fails with error | QA engineer | Before each release | | Restore webhook | Scale back to 1 | Pod creation succeeds | QA engineer | After failure test | ## Conclusion You now have a working local lab for Kubernetes admission webhooks. You learned how to generate certificates, deploy a webhook server, create mutating and validating webhook configurations, verify behavior, and recover from failures. Use this environment to safely develop and test more complex admission logic before deploying to production. Start with small, measurable changes, and always test failure scenarios to understand the impact. The skills you practiced here—TLS configuration, webhook registration, and failure recovery—are directly transferable to production environments, where careful testing and a solid rollback plan are essential.