Kubernetes Admission Controllers Configuration Mistakes with Practical Examples: A Practical Implementation Guide

Intro

Kubernetes Admission Controllers are a powerful but often misunderstood control point in a cluster. When configured incorrectly, they can silently reject valid workloads, allow insecure resources, or even lock out administrators. This practical implementation guide focuses on turning observed configuration problems into verified, safe fixes.

This article is written for developers, DevOps consultants, and technical startup teams who manage Kubernetes clusters. It connects common admission controller configuration mistakes, validation techniques, rollback procedures, and troubleshooting steps to concrete commands, expected outputs, failure signals, and recovery decisions.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before touching any admission controller, inventory your environment. Knowing the exact Kubernetes version, the admission controllers enabled by default, and your cluster topology prevents misdiagnosis and unnecessary changes.

Kubernetes Version

Use kubectl version to see both client and server versions. Admission controller behavior and defaults vary between versions. For example, the PodSecurity admission controller replaced PodSecurityPolicy in Kubernetes 1.25. If you assume a controller is active without checking, you may attempt to configure something that no longer exists.

kubectl version --short

Expected output for a client and server:

Client Version: v1.29.1
Server Version: v1.28.3

List Enabled Admission Controllers

The kube-apiserver process specifies enabled admission controllers with the --enable-admission-plugins flag. On a managed cluster (EKS, GKE, AKS), you may not have direct access to this flag. Instead, inspect the API server pods or configuration.

For clusters where you can list the API server pods:

kubectl get pods -n kube-system | grep kube-apiserver

If the pod is visible (e.g., kubeadm clusters), describe it to see the command line flags:

kubectl describe pod kube-apiserver-control-plane -n kube-system

Look for the --enable-admission-plugins line in the Containers section. Example:

--enable-admission-plugins=NodeRestriction,NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeClaimResize,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota

This shows exactly which built-in controllers are enabled. If a controller you intend to configure is missing, you must add it (if possible) or use a different mechanism.

Check for Dynamic Admission Controllers

Dynamic admission controllers (webhooks) are implemented as MutatingWebhookConfiguration and ValidatingWebhookConfiguration resources. List them to see what custom logic is active:

kubectl get mutatingwebhookconfigurations
kubectl get validatingwebhookconfigurations

Example output:

NAME                       WEBHOOKS   AGE
pod-policy.example.com     1          3d

For each webhook, describe it to see the rules and failure policy:

kubectl describe validatingwebhookconfiguration pod-policy.example.com

Pay attention to failurePolicy. If set to Fail, an unavailable webhook service will block all matching requests. This is a common mistake that takes down a cluster. We'll address this later.

Prerequisites for Safe Configuration

• A test cluster or namespace where you can simulate failures without affecting production.
• kubectl configured with appropriate permissions.
• Access to cluster event logs (kubectl get events).
• Backup of any existing admission controller configurations before changes.

Practical check: Start with kubectl get pods -o wide, then use kubectl describe pod <name> for scheduling and event details, kubectl logs <name> --previous for crash loops, and kubectl rollout status deployment/<name> before assuming a release succeeded.

For local testing, keep the scope small. Apply one manifest at a time, inspect the generated resources, and verify traffic with kubectl port-forward or a local service type before moving to a cloud load balancer or ingress controller.

Safe Configuration Path

When modifying admission controllers, follow a disciplined path to minimize risk. The high-level steps are:

1. Observe current behavior and record baseline.
2. Design the intended configuration change.
3. Apply the change in a test environment.
4. Validate with positive and negative tests.
5. Roll out to production with a rollback plan.

Common Mistake: Changing Multiple Controllers at Once

A frequent mistake is enabling several admission controllers simultaneously or changing multiple webhook rules without isolated testing. This makes it difficult to identify which change caused a failure.

Correct approach: Change one controller or one rule at a time. For built-in controllers, if you need to enable several, do so incrementally in a test cluster.

Example: Suppose you want to enable PodSecurity with the baseline level and also enable ResourceQuota. Do not change both in the same kube-apiserver restart. First enable PodSecurity, verify, then enable ResourceQuota.

For dynamic webhooks, update one webhook configuration at a time. After applying, check the webhook status and test both allowed and disallowed requests.

Concrete Example: Configuring a ValidatingWebhook for Pod Security

Let's walk through a practical example of configuring a validating admission webhook that enforces a simple policy: all pods must have a label owner.

Step 1: Create the webhook server

Create a simple HTTPS server that validates pod creation requests. For brevity, we'll outline the logic; the full code is beyond this article. The server must respond to AdmissionReview requests with allowed or denied.

Step 2: Deploy the webhook server

Deploy the server as a service in your cluster. Ensure it has a certificate signed by a CA trusted by the API server.

apiVersion: v1
kind: Service
metadata:
  name: webhook-service
  namespace: webhook
spec:
  selector:
    app: webhook-server
  ports:
    - port: 443
      targetPort: 8443

Step 3: Create the ValidatingWebhookConfiguration

Create a configuration that calls the webhook for pod creation:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-label-validator
webhooks:
  - name: validate-pod-label.example.com
    clientConfig:
      service:
        name: webhook-service
        namespace: webhook
        path: "/validate"
      caBundle: <base64-encoded-CA-cert>
    rules:
      - operations: ["CREATE"]
        apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
    admissionReviewVersions: ["v1"]
    sideEffects: None
    failurePolicy: Fail  # This is dangerous; we'll discuss later

Step 4: Apply and test

Apply the configuration:

kubectl apply -f validatingwebhook.yaml

Verify it exists:

kubectl get validatingwebhookconfiguration pod-label-validator

Test with a pod that lacks the label:

kubectl run test-pod --image=nginx

Expected result: the pod creation is denied with an error similar to:

Error from server: admission webhook "validate-pod-label.example.com" denied the request: pod missing required label 'owner'

Now test with a compliant pod:

kubectl run test-pod --image=nginx --labels=owner=team-a

This should succeed.

Important: Use failurePolicy: Ignore in Development

The above configuration used failurePolicy: Fail. This means if the webhook service is down or unreachable, all pod creations are blocked. This can take down your cluster. A safer default for new webhooks is failurePolicy: Ignore, which allows requests if the webhook fails. Once you have confidence, you can switch to Fail for critical enforcement.

To change, patch the configuration:

kubectl patch validatingwebhookconfiguration pod-label-validator --type='json' -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"}]'

Verify:

kubectl get validatingwebhookconfiguration pod-label-validator -o yaml | grep failurePolicy

Expected output:

failurePolicy: Ignore

Verification and Diagnostics

After configuring admission controllers, thorough verification is essential. This section covers the commands and signals to confirm correct operation and diagnose issues.

Check Admission Controller Activity

Kubernetes does not provide a direct log of every admission decision by default. However, you can enable audit logging or use the API server logs to see admission controller behavior.

For built-in controllers, the API server logs include lines indicating admission failures. For example, enabling NamespaceLifecycle will log errors when a request targets a terminating namespace.

Access API server logs (if allowed):

kubectl logs -n kube-system kube-apiserver-control-plane | grep -i admission

For dynamic webhooks, you can check the webhook server logs for each request.

Use Dry-Run to Test Without Persisting

kubectl supports dry-run, which simulates the request without persisting. This is invaluable for testing admission webhooks.

Example:

kubectl run test-pod --image=nginx --dry-run=server

If the admission controller rejects the request, you'll see the error without actually creating the pod. The output will be similar to the non-dry-run denial.

Inspect Events for Failures

Admission controller rejections often generate events. Use kubectl get events to see recent warnings.

kubectl get events --field-selector type=Warning

Look for messages mentioning admission webhooks or policy violations.

Debugging Webhook Connectivity

If a webhook is not being called, check the following:

• The webhook configuration references the correct service and namespace.
• The service selector matches the webhook server pods.
• The webhook server is running and listening on the correct port.
• The CA bundle in clientConfig.caBundle is correct and trusted.
• Network policies or firewall rules allow API server to reach the webhook service.

Test the service endpoint from within the cluster using a temporary pod:

kubectl run curl-pod --image=curlimages/curl --rm -it -- sh
# Inside the pod:
curl -k https://webhook-service.webhook.svc:443/health

If the webhook server has a health endpoint, it should return a success response.

Verifying Rollout of Admission Controller Changes

For changes to built-in controllers via kube-apiserver flags, you must restart the API server. This is disruptive. On managed clusters, the cloud provider handles this. For self-managed kubeadm clusters, you typically edit the API server manifest:

sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml

Add or modify the --enable-admission-plugins flag, save. The kubelet will restart the API server pod automatically. Monitor the API server pod:

kubectl get pods -n kube-system -w

Wait until the API server pod is RUNNING and READY. Check the cluster is functional:

kubectl get nodes

If the API server fails to start, revert the manifest change immediately. This is why having a backup of the original manifest is critical.

Failure Modes and Recovery

Admission controller misconfigurations can cause cluster-wide outages or subtle policy bypasses. Understanding common failure modes prepares you for quick recovery.

Failure Mode 1: Webhook Down with failurePolicy: Fail

Symptom: All requests of the matched type (e.g., pod create) are rejected with error "failed calling webhook".

Diagnosis: Check webhook service availability:

kubectl get pods -n webhook
kubectl get svc -n webhook

If pods are crashing or not ready, the webhook is unavailable.

Recovery options:

1. Patch the webhook configuration to set failurePolicy: Ignore as an immediate mitigation:

kubectl patch validatingwebhookconfiguration pod-label-validator --type='json' -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value": "Ignore"}]'

1. Delete or edit the webhook configuration to remove the failing webhook entirely if it is not yet critical:

kubectl delete validatingwebhookconfiguration pod-label-validator

1. Fix the webhook server (restart pods, address configuration) and then restore failurePolicy: Fail.

Failure Mode 2: NamespaceSelector Too Broad

A webhook with no namespaceSelector or a broad selector may inadvertently block resources in critical namespaces like kube-system or kube-public. This can prevent the cluster from healing or scaling.

Symptom: System components cannot be created or updated; cluster operations fail.

Diagnosis: Look at the webhook configuration's namespaceSelector:

kubectl get validatingwebhookconfiguration pod-label-validator -o yaml | grep -A5 namespaceSelector

If missing or set to match everything, the webhook applies to all namespaces.

Recovery: Patch the configuration to restrict namespaces. For example, to only apply to namespaces with the label policy=enforced:

kubectl patch validatingwebhookconfiguration pod-label-validator --type='json' -p='[{"op": "add", "path": "/webhooks/0/namespaceSelector", "value": {"matchLabels": {"policy": "enforced"}}}]'

Then label the desired namespaces:

kubectl label namespace myapp policy=enforced

System namespaces remain unaffected.

Failure Mode 3: Missing RBAC for Webhook Service

The API server must be able to authenticate and authorize to the webhook service. If RBAC is misconfigured, calls fail.

Symptom: Webhook requests denied with authorization errors in API server logs or webhook server logs.

Diagnosis: Check webhook server logs for 403 Forbidden.

Recovery: Ensure the service account used by the webhook server has appropriate RBAC permissions to process the admission review. Typically, the webhook server does not need cluster permissions, but it must be reachable. For the API server side, ensure clientConfig includes correct credentials if needed (though usually service reference is enough).

Failure Mode 4: Misconfigured MutatingWebhook Causes Invalid Mutations

A mutating webhook might inject sidecars or modify resources in a way that makes them invalid or violates other policies.

Symptom: Pods fail to start with cryptic errors, or subsequent validating webhooks reject the mutated resource.

Diagnosis: Inspect the mutated object. You can use kubectl get pod <name> -o yaml to see final state, or if the pod is not created, use dry-run with a dummy mutating webhook that logs the object.

Recovery: Disable the mutating webhook temporarily by setting failurePolicy: Ignore won't help because the mutation is the issue. Instead, you can set the webhook's matchPolicy or rules to exclude the problematic resources, or patch the webhook logic.

General Recovery Checklist

• Keep backups of all webhook configurations and API server manifests.
• Have a quick way to disable a webhook: a command like kubectl delete validatingwebhookconfiguration <name> can be a lifesaver.
• Use failurePolicy: Ignore for non-critical webhooks.
• Test all admission controller changes in a staging cluster first.
• Monitor cluster health after changes: check kubectl get nodes, kubectl get pods --all-namespaces, and API server logs.

Operations Checklist

Use this checklist to ensure safe and effective admission controller management.

Pre-Change Checklist

• [ ] Record current Kubernetes version and enabled admission controllers.
• [ ] List existing mutating and validating webhook configurations.
• [ ] Identify the exact resource types and namespaces affected by the change.
• [ ] Take backups of every YAML configuration that will be modified.
• [ ] Prepare a test scenario in a non-production environment.
• [ ] Define expected positive and negative test results.

During Change Checklist

• [ ] Apply changes one at a time.
• [ ] Use kubectl apply --dry-run=server where possible to test before persisting.
• [ ] Monitor live with kubectl get events --watch.
• [ ] Check webhook service health immediately after applying configuration.
• [ ] Have a rollback command ready (e.g., kubectl delete -f <file> or patch to failurePolicy: Ignore).

Post-Change Verification Checklist

• [ ] Run positive tests: applications that should be allowed are allowed.
• [ ] Run negative tests: applications that should be denied are denied with the expected error message.
• [ ] Confirm no unintended resources are blocked: create a pod or other resource in a namespace that should be unaffected.
• [ ] Check system namespaces are healthy: kubectl get pods -n kube-system.
• [ ] Verify rollout status of any deployment involved in the webhook or controller.
• [ ] Document the change and any observed quirks.

Example Checklist Entry with Concrete Values

Conclusion

Kubernetes Admission Controllers configuration mistakes can have severe consequences, but with careful planning and verification, they are manageable. This guide provided practical examples of inventorying your environment, applying safe configuration changes, validating behavior, diagnosing failures, and recovering from common missteps.

Every recommendation in this article is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure.

As a next step, choose one low-risk verification for your admission controller setup. Record the current state, run a documented check, compare the result with the expected signal, and review dependencies such as namespaces, roles, and webhook configurations. Start with a non-critical validating webhook in a test namespace, experiment with failure policies, and observe how it affects pod creation. Then gradually introduce it to production with a rollback plan.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. With these practices, you can harness admission controllers to enforce security and compliance without sacrificing cluster stability.