Kubernetes Admission Webhooks Production Operations Checklist with Practical Examples

Intro

Admission webhooks are a critical control point in Kubernetes clusters. They intercept API requests for resources such as Pods, Deployments, and custom resources, and can mutate or validate them before they are persisted. When a webhook fails, it can block deployments, break auto-scaling, or silently weaken security policy. A structured operations checklist turns ad-hoc debugging into repeatable, low-risk procedures.

This article is for developers, DevOps consultants, and technical startup teams who run or maintain admission webhooks in production. It covers version and environment inventory, safe configuration changes, verification and diagnostics, failure modes and recovery, and an operational checklist you can adapt to your cluster. Every step pairs a read-only observation with a concrete command, expected output, and a recovery decision.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before touching a webhook, you need to know exactly what is installed, which API version it uses, and how it is deployed. Start by finding all admission webhook configurations in the cluster. The read-only command is:

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations

Example output (abbreviated):

NAME                                          WEBHOOKS   AGE
cert-manager-webhook                          1          42d
kyverno-resource-validating-webhook-cfg       2          12d
pod-policy.example.com                        1          3h

For each webhook configuration, inspect its YAML to see the webhook client configuration, failure policy, match rules, and the service reference. For example:

kubectl get validatingwebhookconfiguration pod-policy.example.com -o yaml

Look for these key fields:

• failurePolicy: Fail or Ignore. In production, Fail is often preferred for security, but it increases outage risk if the webhook is down.
• matchPolicy: Exact or Equivalent. This affects whether rules apply to API groups and versions.
• timeoutSeconds: often 10 seconds default. If your webhook is slow, requests may fail.
• clientConfig.service: namespace, name, and port of the webhook service. Verify that the service exists and selects the correct pods.

Check the deployed webhook server version and image. If your webhook runs as a Deployment:

kubectl get deployment -n webhook-namespace pod-policy-webhook -o jsonpath='{.spec.template.spec.containers[*].image}'

Example output:

registry.example.com/team/pod-policy-webhook:v1.4.2

Compare this with the version that is documented as supported for your Kubernetes cluster. Admission webhooks rely on the admissionregistration.k8s.io API, which is stable in v1 since Kubernetes 1.16. Most webhook frameworks require Kubernetes 1.19 or later for features like matchConditions (introduced in 1.28 as a beta).

Check the webhook server logs to see if it is receiving requests and handling them without errors:

kubectl logs -n webhook-namespace deployment/pod-policy-webhook --tail=50

Pay attention to:

• TLS errors, especially if the webhook service uses a self-signed certificate that has expired.
• Panic stack traces or repeated 500 errors.
• Slow request latencies, which may indicate the webhook is doing too much work synchronously.

If the webhook uses a ValidatingWebhookConfiguration or MutatingWebhookConfiguration that references a service, confirm that the service endpoints are populated:

kubectl get endpoints -n webhook-namespace pod-policy-webhook

Example output:

NAME                  ENDPOINTS           AGE
pod-policy-webhook    10.244.2.15:8443    42d

If ENDPOINTS is empty, the service selector does not match any pods, or the pods are not ready.

For a complete environment inventory, record:

• Kubernetes control plane version (kubectl version --short)
• Webhook configuration names and failure policies
• Webhook server image versions
• TLS certificate expiry dates for the webhook serving certificate
• Any relevant ResourceQuotas or LimitRanges that might affect webhook pod scheduling

Practical checklist for Version and Environment Inventory:

• [ ] List all mutating and validating webhook configurations with their failure policies.
• [ ] Record the Kubernetes server and client versions.
• [ ] Identify the webhook server image and confirm it is within the supported version range.
• [ ] Verify the webhook service exists and has active endpoints.
• [ ] Check the webhook server logs for TLS errors, panics, or repeated HTTP 500s.
• [ ] Note the expiry date of the TLS certificate used by the webhook.
• [ ] Document the namespace and deployment topology of the webhook server.

Safe Configuration Path

Changing a webhook configuration can have immediate cluster-wide impact. Follow a safe path: observe current state, stage the change in a local file, validate it, apply with a dry-run, and then gradually roll out.

Start by exporting the current configuration to a file. This serves as a backup and a baseline for comparison.

kubectl get validatingwebhookconfiguration pod-policy.example.com -o yaml > pod-policy-webhook-original.yaml

Make a copy to edit:

cp pod-policy-webhook-original.yaml pod-policy-webhook-new.yaml

Edit pod-policy-webhook-new.yaml with your intended change. For example, suppose you want to tighten the failure policy from Ignore to Fail and increase the timeout from 10 to 15 seconds. Change only those fields, and leave everything else identical.

Before applying, validate the YAML syntax and Kubernetes schema with kubectl apply --dry-run=client:

kubectl apply -f pod-policy-webhook-new.yaml --dry-run=client

If there is a syntax error, you will see a message like:

error: error validating "pod-policy-webhook-new.yaml": error validating data: ValidationError(ValidatingWebhookConfiguration.webhooks[0].timeoutSeconds): invalid type for io.k8s.api.admissionregistration.v1.Webhook.timeoutSeconds: got "string", expected "integer"; if you choose to ignore these errors, turn validation off with --validate=false

Fix the error and repeat the dry-run until it succeeds. Then use --dry-run=server to simulate the API server's validation without persisting the change:

kubectl apply -f pod-policy-webhook-new.yaml --dry-run=server

If the server accepts it, you will see:

validatingwebhookconfiguration.admissionregistration.k8s.io/pod-policy.example.com configured (server dry run)

Now you can apply the change. But to reduce risk, consider using a canary namespace or a subset of resources first. For example, if you are adding a new validation rule, you can scope it to a test namespace by adding a namespaceSelector or objectSelector. In the webhook YAML, add under the webhook:

namespaceSelector:
  matchLabels:
    kubernetes.io/metadata.name: test-namespace

Apply the change:

kubectl apply -f pod-policy-webhook-new.yaml

Immediately verify that the webhook configuration was updated and that the API server can still process requests. Check the webhook server logs for new requests and any errors:

kubectl logs -n webhook-namespace deployment/pod-policy-webhook --tail=20

If you need to roll back, restore the original configuration:

kubectl apply -f pod-policy-webhook-original.yaml

Then check that the rollback is effective by examining the configuration again.

Always keep a copy of the last known good configuration in version control. For production webhooks, use a GitOps flow where changes are reviewed and applied automatically, but even manual changes should be traceable.

Practical checklist for Safe Configuration Path:

• [ ] Export the current webhook configuration to a local backup file.
• [ ] Make a single, well-defined change in a copy of the file.
• [ ] Validate with kubectl apply --dry-run=client and fix any syntax errors.
• [ ] Validate with kubectl apply --dry-run=server to catch semantic issues.
• [ ] If possible, roll out to a test namespace or subset using selectors.
• [ ] Apply the change and immediately watch webhook server logs for errors.
• [ ] Keep the previous configuration available for quick rollback.

Verification and Diagnostics

After applying a webhook change (or even before, when investigating a suspected problem), you need to verify that the webhook is operating correctly and diagnose any deviations.

First, check that the webhook server is healthy. If your webhook exposes a health endpoint, query it directly from inside the cluster or via port-forward. For example, if your webhook uses the common /healthz endpoint:

kubectl port-forward -n webhook-namespace service/pod-policy-webhook 8443:8443 &
curl -k https://localhost:8443/healthz

Expected output if healthy:

OK

If the health check fails or the service is unresponsive, the problem is likely in the webhook server itself, not the configuration.

Next, test admission behavior with a controlled object. Create a minimal Pod specification that you expect to be accepted or rejected based on the webhook's policy. For example, if your webhook requires a team label on every Pod, you can test with and without the label.

Create a file test-pod-no-label.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: test-no-label
spec:
  containers:
  - name: nginx
    image: nginx:1.25

Attempt to create it:

kubectl apply -f test-pod-no-label.yaml

If the webhook is working correctly with failurePolicy: Fail and the rule rejects missing labels, you will see an error like:

Error from server: admission webhook "pod-policy.example.com" denied the request: Pod "test-no-label" is invalid: missing required label 'team'

Now test with the label:

apiVersion: v1
kind: Pod
metadata:
  name: test-with-label
  labels:
    team: platform
spec:
  containers:
  - name: nginx
    image: nginx:1.25

kubectl apply -f test-with-label.yaml

Expected output:

pod/test-with-label created

Clean up the test pods after verification:

kubectl delete pod test-no-label test-with-label

To diagnose timeouts or slow responses, check the webhook's own metrics if it exposes them. Many webhook frameworks export Prometheus metrics. For example, if your webhook exposes metrics on port 8080:

kubectl port-forward -n webhook-namespace deployment/pod-policy-webhook 8080:8080 &
curl http://localhost:8080/metrics | grep webhook_request_duration

Look for high percentiles or errors. Also check the Kubernetes API server audit logs to see webhook call timing and status codes. If you have access to the control plane, search for lines containing admission webhook or webhook in the audit log.

If the webhook uses a self-signed certificate, verify that the CA bundle in the webhook configuration matches the serving certificate. The clientConfig.caBundle field must contain the PEM-encoded CA certificate that signed the webhook's serving certificate. If the webhook server's certificate is rotated, the webhook configuration must be updated with the new CA bundle. You can check the certificate details using openssl:

kubectl get validatingwebhookconfiguration pod-policy.example.com -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -text | grep -E 'Subject:|Not After'

Example output:

Subject: CN = pod-policy-webhook.webhook-namespace.svc
Not After : May  5 12:00:00 2025 GMT

If the certificate is expired or the CA bundle does not match, the Kubernetes API server will refuse to call the webhook and will log error messages like x509: certificate signed by unknown authority or certificate has expired.

Practical checklist for Verification and Diagnostics:

• [ ] Confirm the webhook server health endpoint returns OK.
• [ ] Test admission with a known good and known bad resource against the policy.
• [ ] Observe the exact error message from the API server for denied requests.
• [ ] Check webhook server metrics for high latency or error rates.
• [ ] Review Kubernetes API server audit logs for webhook call failures.
• [ ] Verify the CA bundle in the webhook configuration matches the server certificate and is not expired.

Failure Modes and Recovery

Admission webhooks can fail in several ways, each with its own symptoms and recovery steps. Understanding these failure modes helps you design a resilient system and react quickly.

1. Webhook server down or unreachable

Symptom: When creating or updating resources, the API server returns an error like:

Error from server (InternalError): error when creating "pod.yaml": Internal error occurred: failed calling webhook "pod-policy.example.com": failed to call webhook: Post "https://pod-policy-webhook.webhook-namespace.svc:8443/validate?timeout=10s": dial tcp 10.96.0.15:8443: connect: connection refused

Recovery:

• Check if the webhook pods are running and ready:

kubectl get pods -n webhook-namespace -l app=pod-policy-webhook

If no pods are ready, investigate the Deployment, StatefulSet, or DaemonSet that manages the webhook.

• If the failure policy is Fail, temporarily switching it to Ignore can restore cluster operations while you fix the webhook. However, this bypasses validation or mutation and should be done only as an emergency measure with proper approval.

2. Webhook timeout

Symptom: API requests fail with a timeout error, such as:

Error from server: Timeout: request did not complete within requested timeout

Or the webhook server logs show requests taking longer than the configured timeoutSeconds.

Recovery:

• Increase the timeoutSeconds in the webhook configuration if the webhook legitimately needs more time, but be aware that this can block API requests longer.
• Optimize the webhook code to reduce latency. For example, cache frequently used data or move expensive operations to asynchronous controllers.
• If the webhook is under high load, scale up its replicas.

3. TLS certificate failure

Symptom: API server logs show TLS errors, and requests are denied. The webhook server may be running fine, but the API server cannot establish a trusted connection.

Recovery:

• Rotate the webhook serving certificate and update the caBundle in the webhook configuration with the new CA certificate.
• If you use cert-manager or a similar tool, ensure the Certificate resource is valid and renews automatically.
• Temporarily set failurePolicy to Ignore if you must allow requests while fixing the certificate, but this is a security risk.

4. Webhook returns an error or denies unexpectedly

Symptom: Resource creation is denied with a webhook-specific error message. The API server passes through the error from the webhook.

Recovery:

• Read the webhook server logs to identify the rule or code path that returned the error.
• If the rejection is due to a bug in the webhook policy, fix the policy and redeploy the webhook.
• Test with different resources to isolate whether the issue is specific to certain kinds or namespaces.

5. Webhook misconfiguration

Symptom: The webhook matches resources it should not, or fails to match resources it should. This can happen due to incorrect rules, namespaceSelector, or objectSelector.

Recovery:

• Review the webhook configuration against the intended scope.
• Use kubectl describe on the webhook configuration to see a human-readable summary:

kubectl describe validatingwebhookconfiguration pod-policy.example.com

Example output snippet:

Name:         pod-policy.example.com
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  admissionregistration.k8s.io/v1
Kind:         ValidatingWebhookConfiguration
Webhooks:
  Name: pod-policy.webhook.example.com
  Client Config:
    Ca Bundle:  <base64-encoded>
    Service:
      Name:        pod-policy-webhook
      Namespace:   webhook-namespace
      Port:        8443
  Rules:
    API Groups:   [""]
    API Versions: ["v1"]
    Operations:   ["CREATE"]
    Resources:    ["pods"]
    Scope:        "Namespaced"

• Update the rules or selectors and re-apply with the safe configuration path described above.

Recovery strategies should be documented in your runbooks. Include the exact commands to roll back a webhook change, restore a previous certificate, or disable a webhook in an emergency. For example, to temporarily disable a webhook without deleting it, you can set its failurePolicy to Ignore and remove all its rules, or simply delete the webhook configuration (which stops all admission calls).

Emergency recovery snippet:

kubectl delete validatingwebhookconfiguration pod-policy.example.com

This will remove the webhook entirely, allowing all requests to proceed. You can later re-create it from your version-controlled manifest.

Practical checklist for Failure Modes and Recovery:

• [ ] Identify the failure mode from API server errors and webhook logs.
• [ ] If webhook pods are down, restart them or scale up.
• [ ] If timeout is the issue, consider increasing timeoutSeconds or optimizing the webhook.
• [ ] If TLS fails, rotate certificates and update CA bundle.
• [ ] If misconfiguration is suspected, review rules and selectors.
• [ ] Have a documented emergency rollback or disable procedure.
• [ ] Test the recovery procedure in a non-production cluster first.

Operations Checklist

Use this daily or weekly operations checklist to keep admission webhooks healthy. Adapt the commands to your webhook names and namespaces.

1. List all webhooks and their statuses

• Command: kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
• Expected: all webhooks listed with no error; note any webhook whose AGE is unexpectedly recent (may indicate a recent change).

1. Check webhook server health

• Command: kubectl get pods -n webhook-namespace -l app=pod-policy-webhook -o wide
• Expected: all pods Running and Ready (1/1 or more). If any pod is not ready, investigate with kubectl describe pod and kubectl logs.

1. Verify service endpoints

• Command: kubectl get endpoints -n webhook-namespace pod-policy-webhook
• Expected: at least one endpoint IP:port. Empty endpoints mean no pods are serving.

1. Inspect webhook configuration for drift

• Command: kubectl get validatingwebhookconfiguration pod-policy.example.com -o yaml | diff - pod-policy-webhook-original.yaml
• Expected: no differences unless a change was authorized. If differences exist, review and decide whether to accept or roll back.

1. Check webhook logs for new errors

• Command: kubectl logs -n webhook-namespace deployment/pod-policy-webhook --tail=100 | grep -i -E 'error|panic|timeout'
• Expected: no output or only known benign errors. Investigate any new error lines.

1. Verify certificate expiry

• Command: kubectl get validatingwebhookconfiguration pod-policy.example.com -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | base64 -d | openssl x509 -noout -enddate
• Expected: a future date, ideally more than 30 days away. If less, schedule certificate rotation.

1. Test admission with a sample resource

• Command: apply a test Pod with expected accept/reject, then delete it.
• Expected: webhook denies when policy is violated and allows when compliant.

1. Review metrics for anomalies

• If metrics are exposed, query for request rate, latency, and error rate, and compare with baseline.
• Expected: no sudden spikes in 5xx errors or latency.

1. Check resource usage of webhook pods

• Command: kubectl top pods -n webhook-namespace -l app=pod-policy-webhook
• Expected: CPU and memory within limits. High memory may indicate a memory leak.

1. Ensure GitOps or config management is in sync

• Compare the live webhook configuration with the source of truth in your repo.
• Expected: no uncommitted changes in production.

For each item, record the timestamp, the person performing the check, and any anomalies found. This creates an audit trail and helps spot trends.

Conclusion

Admission webhooks are powerful but risky if not operated carefully. This checklist gives you a systematic way to inventory, change, verify, and recover webhook configurations in production. Every step is version-scoped, observable, and reversible where the technology permits.

Start with one low-risk verification: list your webhooks, check their health, and test one admission rule with a sample resource. Record the current state, run the documented checks, compare results with expected signals, and review dependencies such as service endpoints, TLS certificates, and RBAC permissions.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By following these practices, you keep your admission webhooks stable, secure, and maintainable.