Kubernetes Admission Webhooks: Local Lab Setup with Practical Examples

Intro

Kubernetes admission webhooks are powerful extension points that let you validate or mutate resources before they are persisted. However, testing them directly on a shared cluster is risky: a misconfigured webhook can block deployments, cause cascading failures, or even make the cluster partially inoperable. A local lab environment gives you a safe, repeatable sandbox to experiment with webhook logic, understand failure modes, and build confidence before rolling out to production.

In this guide, we will set up a local lab using lightweight tools like kind or minikube, create both a validating and a mutating admission webhook with practical examples, verify their behavior, and cover recovery steps. By following this guide, you will have a fully functional environment to develop and test admission webhooks without risking shared infrastructure.

A clear setup reduces rework by separating the stages: environment preparation, certificate management, webhook deployment, and testing. We will start with a narrow, measurable example to ensure every step is observable.

Version and Environment Inventory

Before starting, confirm the versions and tools you will use. For this guide, we assume the following:

Prerequisites:

• A machine with at least 2 CPUs and 4GB RAM available.
• Docker or a container runtime supported by kind/minikube.
• Basic familiarity with Kubernetes manifests and command-line tools.

Topology: We will create a single-node cluster where the API server runs inside a container (kind) or a VM (minikube). The webhook server will be deployed as a pod inside the same cluster and exposed via a service. The API server will call the webhook over HTTPS using a self-signed certificate.

Ensure your local cluster is up:

kind create cluster --name webhook-lab

Expected output:

Creating cluster "webhook-lab" ...
 ✓ Ensuring node image (kindest/node: v1.27.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-webhook-lab"

If using minikube:

minikube start --driver=docker --kubernetes-version=v1.27.3

Verify cluster access:

kubectl cluster-info

Safe Configuration Path

Now we will implement two webhooks: a mutating webhook that adds a label to every pod, and a validating webhook that rejects pods with a specific label value. This scoped approach lets us verify each piece independently.

Step 1: Generate TLS Certificates

The API server must trust the webhook server. Since we are in a lab, we can use a self-signed CA and issue a server certificate. The webhook configuration will specify the CA bundle. Run these commands to generate the certificates (using a temporary directory):

mkdir -p /tmp/webhook-certs && cd /tmp/webhook-certs

# Generate CA key and certificate
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -subj "/CN=webhook-ca" -days 365 -out ca.crt

# Generate server key and CSR
openssl genrsa -out server.key 2048
openssl req -new -key server.key -subj "/CN=webhook-server.default.svc" -out server.csr

# Sign the server certificate with our CA
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 365

The certificate's CN must match the service DNS name: webhook-server.default.svc.

Create a Kubernetes secret with the server certificate and key:

kubectl create secret tls webhook-server-cert --cert=server.crt --key=server.key

You will also need the base64-encoded CA certificate for the webhook configuration. We will use it when creating the webhook configuration.

Step 2: Deploy the Webhook Server

Create a simple HTTP server that responds to admission review requests. Below is an example in Go. Save it as main.go:

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
    "crypto/tls"

    admissionv1 "k8s.io/api/admission/v1"
    corev1 "k8s.io/api/core/v1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

func mutatePods(w http.ResponseWriter, r *http.Request) {
    var admissionReview admissionv1.AdmissionReview
    body, _ := ioutil.ReadAll(r.Body)
    if err := json.Unmarshal(body, &admissionReview); err != nil {
        http.Error(w, "could not parse admission review", http.StatusBadRequest)
        return
    }

    // Build the patch to add a label
    patchType := admissionv1.PatchTypeJSONPatch
    pod := corev1.Pod{}
    if err := json.Unmarshal(admissionReview.Request.Object.Raw, &pod); err != nil {
        http.Error(w, "could not unmarshal pod", http.StatusBadRequest)
        return
    }

    // If labels are nil, create an empty map
    if pod.Labels == nil {
        pod.Labels = map[string]string{}
    }
    pod.Labels["webhook-lab"] = "mutated"

    // Create patch
    patch := []map[string]interface{}{
        {"op": "add", "path": "/metadata/labels/webhook-lab", "value": "mutated"},
    }
    patchBytes, _ := json.Marshal(patch)

    admissionResponse := &admissionv1.AdmissionResponse{
        UID:       admissionReview.Request.UID,
        Allowed:   true,
        PatchType: &patchType,
        Patch:     patchBytes,
    }

    response := admissionv1.AdmissionReview{
        TypeMeta: metav1.TypeMeta{
            APIVersion: "admission.k8s.io/v1",
            Kind:       "AdmissionReview",
        },
        Response: admissionResponse,
    }
    respBytes, _ := json.Marshal(response)
    w.Header().Set("Content-Type", "application/json")
    w.Write(respBytes)
}

func main() {
    http.HandleFunc("/mutate", mutatePods)
    server := &http.Server{
        Addr: ":8443",
        TLSConfig: &tls.Config{
            Certificates: []tls.Certificate{loadX509KeyPair("/etc/webhook/certs/tls.crt", "/etc/webhook/certs/tls.key")},
        },
    }
    fmt.Println("Starting webhook server on :8443")
    server.ListenAndServeTLS("", "")
}

func loadX509KeyPair(certFile, keyFile string) tls.Certificate {
    cert, err := tls.LoadX509KeyPair(certFile, keyFile)
    if err != nil {
        panic(err)
    }
    return cert
}

Build the Docker image and load it into kind:

docker build -t webhook-server:latest .
kind load docker-image webhook-server:latest --name webhook-lab

Create a deployment and service:

# webhook-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webhook-server
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: webhook-server
  template:
    metadata:
      labels:
        app: webhook-server
    spec:
      containers:
      - name: server
        image: webhook-server:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8443
        volumeMounts:
        - name: certs
          mountPath: /etc/webhook/certs
          readOnly: true
      volumes:
      - name: certs
        secret:
          secretName: webhook-server-cert

# webhook-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: webhook-server
  namespace: default
spec:
  selector:
    app: webhook-server
  ports:
  - port: 443
    targetPort: 8443

Apply:

kubectl apply -f webhook-deployment.yaml
kubectl apply -f webhook-service.yaml

Verify the pod is running:

kubectl get pods -l app=webhook-server

Expected output:

NAME                            READY   STATUS    RESTARTS   AGE
webhook-server-79c4f4c9b7-9x2kq 1/1     Running   0          5s

Step 3: Create the MutatingWebhookConfiguration

Create a configuration that tells the API server to call our webhook when pods are created. Replace CA_BUNDLE with the base64 of ca.crt:

CA_BUNDLE=$(cat /tmp/webhook-certs/ca.crt | base64 | tr -d '\n')

# mutating-webhook.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: pod-mutator
webhooks:
- name: pod-mutator.default.svc
  clientConfig:
    service:
      name: webhook-server
      namespace: default
      path: "/mutate"
      port: 443
    caBundle: ${CA_BUNDLE}
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail

Apply:

envsubst < mutating-webhook.yaml | kubectl apply -f -

Step 4: Test the Mutating Webhook

Create a simple pod:

kubectl run test-pod --image=nginx

Check labels:

kubectl get pod test-pod --show-labels

Expected output includes webhook-lab=mutated.

Step 5: Add a Validating Webhook

Now add a validating webhook that rejects pods if the label webhook-lab is set to mutated (to demonstrate denial). You can extend the server or create a separate one. For brevity, assume a similar server with a /validate endpoint. The configuration:

# validating-webhook.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-validator
webhooks:
- name: pod-validator.default.svc
  clientConfig:
    service:
      name: webhook-server
      namespace: default
      path: "/validate"
      port: 443
    caBundle: ${CA_BUNDLE}
  rules:
  - operations: ["CREATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail

Apply and test: the pod creation should be denied with a message.

Verification and Diagnostics

To verify the webhooks are working correctly, perform these checks:

• Check webhook configurations exist:

kubectl get mutatingwebhookconfigurations, validatingwebhookconfigurations

• Inspect webhook logs:

kubectl logs deployment/webhook-server

Expected to see incoming admission review requests.

• Use kubectl apply --dry-run=server to see if the webhook is called without persisting the object.

• Test with a dry-run:

kubectl run test-pod --image=nginx --dry-run=server

• Use kubectl describe on the webhook configuration to see the CA bundle and rules.

If the webhook is not being called, check the API server logs (for kind, see container logs) for error messages like x509: certificate signed by unknown authority or connection refused.

Failure Modes and Recovery

Admission webhooks can cause problems. Common failure modes:

• Webhook server unavailable: If failurePolicy is Fail, all matching requests will be rejected. To recover, you can either delete the webhook configuration or scale the deployment to restore availability. If the API server cannot be reached to delete the webhook (unlikely but possible), you may need to edit the API server manifest to disable admission plugins.
• Incorrect CA bundle: The API server will reject the webhook call, and with failurePolicy: Fail, resource creation fails. Fix the CA bundle in the webhook configuration.
• Timeout: If the webhook server does not respond within the timeout (default 10 seconds), the API server will apply the failure policy.

Recovery steps:

# Temporarily disable webhook by deleting its configuration
kubectl delete mutatingwebhookconfiguration pod-mutator
kubectl delete validatingwebhookconfiguration pod-validator

Then fix the underlying issue and reapply.

Rollback example: If a new webhook causes issues, roll back to a previous version by reapplying the old configuration or using kubectl apply with the known good manifest.

Common Pitfalls and How to Avoid Them

Beyond the failure modes already covered, several other pitfalls can trip you up when working with admission webhooks. Being aware of these will save you debugging time.

• Mismatched admissionReviewVersions: The API server sends admission requests in the versions listed in the webhook configuration. If your webhook server only understands v1beta1 but the configuration lists v1, the request will fail. Always declare at least v1 and ensure your server handles it. Test with a simple echo endpoint to see what version is sent.

• Forgetting sideEffects: None: If your webhook has no side effects (it does not modify external state), set sideEffects: None. This allows the API server to call it during dry-run requests and prevents it from being skipped when the dry-run feature is used. If your webhook does have side effects, document them and consider sideEffects: NoneOnDryRun.

• Overly broad rules: Matching all resources with / can cause the webhook to interfere with system components. Scope your rules to the minimum needed, such as specific resources, groups, and operations. For example, instead of matching all pods, match only pods with a specific label or in a particular namespace.

• Missing namespace in clientConfig: If you omit namespace in clientConfig.service, the API server assumes the same namespace as the webhook configuration, which is often not where your service lives. Always specify the namespace explicitly.

• Certificate renewal: Self-signed certificates have an expiration date. If the certificate expires, the webhook will fail. In production, use a certificate management solution like cert-manager with automatic renewal. In the lab, note the expiration date and regenerate when needed.

• Ignoring webhook timeouts: The default timeout is 10 seconds, but some operations may take longer. Set an appropriate timeoutSeconds in the webhook configuration and ensure your webhook responds within that time. If your webhook needs more time, adjust the timeout, but be cautious about slowing down all matching requests.

Operations Checklist

Use this checklist for your local lab and future production use. Each task has a single accountable owner and a review cadence. In a production setting, assign an owner such as the platform engineering lead.

Conclusion

You now have a working local lab for Kubernetes admission webhooks. You learned how to generate certificates, deploy a webhook server, create mutating and validating webhook configurations, verify behavior, and recover from failures. Use this environment to safely develop and test more complex admission logic before deploying to production. Start with small, measurable changes, and always test failure scenarios to understand the impact. The skills you practiced here—TLS configuration, webhook registration, and failure recovery—are directly transferable to production environments, where careful testing and a solid rollback plan are essential.