Kubernetes Admission Controllers: Common Errors and Practical Fixes

Intro

Kubernetes Admission Controllers are plugins that govern and enforce how the cluster is used. They can be thought of as a gatekeeper that intercepts authenticated API requests and can change the request object or deny it altogether. While they are a powerful tool for enforcing security, resource management, and configuration policies, they are also a common source of frustration when something goes wrong. A misconfigured admission controller can block workload deployments, prevent pods from starting, or cause unexpected mutations to resources, leading to production outages.

This article focuses on the most common errors encountered with Kubernetes Admission Controllers and provides actionable, practical fixes. It is written for developers, DevOps engineers, and technical startup teams who need to move from an observed problem to a verified result quickly. We will cover the core concepts, typical error scenarios, effective debugging techniques, and step-by-step recovery procedures. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify results, and document recovery procedures.

Understanding Admission Controllers

Before diving into errors, it's essential to understand how admission controllers work. Admission controllers are pieces of code that intercept requests to the Kubernetes API server after authentication and authorization but before the object is persisted in etcd. They can be divided into two categories:

• Built-in Admission Controllers: These are compiled into the kube-apiserver binary and enabled via the --enable-admission-plugins flag. Examples include NamespaceLifecycle, LimitRanger, PodSecurity, and ResourceQuota.
• Dynamic Admission Controllers: These are webhooks that can be configured at runtime using ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects. They call external services (HTTP callbacks) to make admission decisions.

Common admission controller errors can originate from either category. However, dynamic webhooks are often the culprit because they involve external dependencies and network calls.

Version and Environment Inventory

Before troubleshooting any admission controller issue, you must establish a clear inventory of your environment. This includes the Kubernetes version, the admission controllers enabled, and the configuration of any webhooks. Knowing the version is critical because certain admission controllers are removed or deprecated in different releases, and webhook API versions evolve.

1. Determine Your Kubernetes Version

Run the following command to get the server version:

kubectl version --short

Example output:

Client Version: v1.28.0
Server Version: v1.28.3

If you are using a managed Kubernetes service (EKS, GKE, AKS), the version might be listed in the cluster overview. Note the minor version because admission controller behavior can vary.

2. List Enabled Admission Controllers

To see which admission controllers are enabled on your kube-apiserver, you can inspect the pod specification for the kube-apiserver if you're using a self-managed cluster with static pods (e.g., kubeadm). If you have access to the control plane node, run:

ps aux | grep kube-apiserver

Look for the --enable-admission-plugins flag. Alternatively, if your kube-apiserver runs as a pod, use:

kubectl get pod -n kube-system -l component=kube-apiserver -o yaml | grep enable-admission-plugins

Example output:

- --enable-admission-plugins=NodeRestriction,PodSecurity,ResourceQuota,DefaultStorageClass

In managed clusters (EKS, GKE), you may not have direct access, but you can infer enabled controllers by attempting to create resources that they would reject.

3. List Webhook Configurations

For dynamic admission controllers, list the webhook configurations:

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

Example output:

NAME                          WEBHOOKS   AGE
istio-validator               1          5d
cert-manager-webhook          1          10d

Then examine the details of the relevant webhook:

kubectl describe validatingwebhookconfiguration istio-validator

This will show the rules, namespace selector, and the service or URL that the webhook calls.

4. Check Webhook Services and Endpoints

For webhooks that use a service reference (the most common pattern), ensure the service exists and has ready endpoints. For example:

kubectl get svc -n istio-system istiod
kubectl get endpoints -n istio-system istiod

If the endpoints list is empty, the webhook will fail because the API server cannot reach any backend pod.

Common Errors and Fixes

Error 1: Webhook Not Available (Timeout or Connection Refused)

Symptom: When creating a resource, you get an error like:

Error from server (InternalError): error when creating "pod.yaml": Internal error occurred: failed calling webhook "webhook.example.com": Post "https://istiod.istio-system.svc:443/validate?timeout=10s": dial tcp 10.0.0.10:443: connect: connection refused

Or you might see a timeout:

Error from server: error when creating "pod.yaml": Internal error occurred: failed calling webhook "webhook.example.com": Post "https://istiod.istio-system.svc:443/validate?timeout=10s": context deadline exceeded

Cause: The webhook service is not running, not ready, or the API server cannot reach it due to network policies or misconfigured TLS.

Fix:

1. Check if the webhook backend pod is running:

kubectl get pods -n istio-system -l app=istiod

1. Check the endpoints of the service:

kubectl get endpoints -n istio-system istiod

If there are no endpoints, the service selector doesn't match any running pods. Adjust the service selector or scale up the deployment.

1. Verify that the API server can reach the service. If you have network policies, ensure they allow traffic from the API server namespace (usually kube-system) to the webhook service on the required port.

1. If TLS is misconfigured, check the CA bundle in the webhook configuration. The caBundle field must contain the CA certificate that signed the webhook server's certificate. You can update it using:

kubectl patch validatingwebhookconfiguration my-webhook --type='json' -p='[{"op": "replace", "path": "/webhooks/0/clientConfig/caBundle", "value": "<base64-encoded-CA>"}]'

Error 2: Webhook Rejects All Requests Due to Policy Misconfiguration

Symptom: You get a clear denial from the webhook, e.g.:

Error from server: admission webhook "namespace.dns.required" denied the request: namespace must have label 'team'

Cause: The webhook's policy is too strict or the object you're creating does not comply with the policy.

Fix:

1. Review the webhook's rules and policy. You can inspect the webhook's configuration and possibly its logs.

1. If the policy is correct, adjust your object to comply. For example, add the required label to the namespace:

kubectl label namespace my-namespace team=myteam

1. If the policy is too broad or you need to temporarily bypass it, you can:

• Change the webhook's namespaceSelector to exclude certain namespaces.
• Or, in an emergency, delete or disable the webhook (not recommended for long-term, but can unblock you):

kubectl delete validatingwebhookconfiguration my-webhook

Note: Always document the deletion and recreate it with a fix as soon as possible.

Error 3: Mutating Webhook Changes Resources Unexpectedly

Symptom: Resources are created with unexpected annotations, labels, or sidecar containers (e.g., Istio sidecar, Linkerd proxy). You might see unexpected behavior in your applications.

Cause: A mutating admission webhook is modifying your resources. This could be intentional (e.g., automatic sidecar injection) but sometimes a misconfigured webhook could apply changes wrongly.

Fix:

1. Identify which mutating webhook is active:

kubectl get mutatingwebhookconfigurations

1. Describe the webhook to see its rules and namespaceSelector:

kubectl describe mutatingwebhookconfiguration istio-sidecar-injector

1. If the webhook is targeting namespaces that it shouldn't, adjust the namespaceSelector. For example, to exclude a namespace, ensure it does not match the selector. If the selector is matchLabels: {istio-injection: enabled}, then any namespace without that label will not be targeted.

To disable injection for a specific namespace, remove the label:

kubectl label namespace my-namespace istio-injection-

1. To test the effect, you can dry-run a deployment with and without the label to see the changes. Use kubectl apply --dry-run=server -f deployment.yaml -o yaml (requires Kubernetes 1.18+).

Error 4: Admission Controller Missing from API Server Flags

Symptom: You expect a built-in admission controller to enforce a policy, but nothing happens. For example, you have a ResourceQuota defined, but pods can still be created beyond the limits.

Cause: The admission controller is not enabled in the kube-apiserver flags. For example, ResourceQuota is not in the --enable-admission-plugins list.

Fix:

1. Check the enabled plugins as described earlier.

1. Modify the kube-apiserver manifest (e.g., /etc/kubernetes/manifests/kube-apiserver.yaml) to include the missing controller in the list. For managed clusters, you may need to check the cloud provider's configuration options.

1. After updating, the kube-apiserver pod will restart automatically. Monitor its status:

kubectl get pods -n kube-system -l component=kube-apiserver

1. Verify that the controller is now active by attempting a request that should be rejected. For ResourceQuota, try creating a pod that exceeds the quota and observe if it's forbidden.

Error 5: Admission Webhook Configuration Has Invalid Rules or Match Conditions

Symptom: Webhook is not called for certain resources, or it's called for resources it shouldn't be.

Cause: The rules field in the webhook configuration may be misconfigured. For example, missing an API group or resource, or using * incorrectly.

Fix:

1. Review the rules:

kubectl get validatingwebhookconfiguration my-webhook -o yaml

Look for the rules section:

rules:
  - operations: ["CREATE"]
    apiGroups: ["apps"]
    apiVersions: ["v1"]
    resources: ["deployments"]
    scope: "Namespaced"

1. Ensure that apiGroups matches the resource's group. For core resources, use "" (empty string). For example, to match pods, set apiGroups: [""].

1. Adjust the rules as needed and apply the updated configuration.

Verification and Diagnostics

Once you've made a fix, it's crucial to verify that the issue is resolved without introducing new problems.

1. Dry-Run Tests

Use kubectl apply --dry-run=server or kubectl create --dry-run=server to simulate the request. This will validate the object against admission controllers without persisting it.

Example:

kubectl apply --dry-run=server -f deployment.yaml

If the command returns deployment.apps/myapp created (server dry run), the admission is successful. If there's an error, it will be shown.

2. Check Events

After applying a resource, check the events in the namespace to see if there are any admission-related warnings:

kubectl get events -n my-namespace --sort-by='.lastTimestamp'

3. Inspect API Server Logs

If the issue is with the kube-apiserver or webhook call failures, check the API server logs. For self-managed clusters, you can often access them on the control plane node:

journalctl -u kube-apiserver -f

Or view the pod logs:

kubectl logs -n kube-system kube-apiserver-master -f

Search for lines containing admission or webhook.

4. Use kubectl describe for Resources

When a resource fails to be created, sometimes the error is only visible in the resource's events if it's partially created. Use kubectl describe on the resource type or check if a configuration object exists.

5. Monitor Webhook Backend Logs

Check the logs of the webhook backend pods to see if they received the requests and what decisions they made.

kubectl logs -n istio-system -l app=istiod --tail=50

Look for error messages or admission review logs.

Failure Modes and Recovery

It's important to plan for failure modes when admission controllers block critical operations. Here are some common failure scenarios and recovery steps.

Webhook Service Down Completely

If the webhook service is down and you cannot bring it back quickly, you may need to temporarily disable the webhook to allow deployments. However, this can compromise security or policy. Assess the risk.

To temporarily disable a webhook, you can either:

• Delete the webhook configuration (and recreate later).
• Set the failurePolicy to Ignore if it's currently Fail. This allows requests to proceed even if the webhook fails. However, this requires modifying the webhook configuration, which you can do if you have API access.

kubectl patch validatingwebhookconfiguration my-webhook --type='merge' -p '{"webhooks":[{"name":"webhook.example.com","failurePolicy":"Ignore"}]}'

Note: This is a temporary measure; the webhook's policy will not be enforced while it's down. Document the change and revert once the service is restored.

Webhook Blocking Node Deletion

Some webhooks may block deletion of nodes or pods, especially if they have finalizers. Ensure that the webhook's rules are scoped correctly to avoid interfering with system resources.

Cluster Upgrade Breaks Admission Controllers

When upgrading Kubernetes, built-in admission controllers may change. For example, PodSecurityPolicy was removed in v1.25 and replaced by Pod Security Admission. Check the release notes and update your configurations accordingly.

After an upgrade, verify that all webhooks are still reachable and their TLS certificates are valid. Certificates may expire or the CA bundle may need updating.

Operations Checklist

Use this checklist to systematically address admission controller issues:

1. Identify the failing request: Note the exact command, resource type, and error message.
2. Check Kubernetes version and enabled admission plugins: Use kubectl version and inspect kube-apiserver flags.
3. List all webhook configurations: kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations.
4. Determine which webhook is involved: The error message usually names the webhook.
5. Check webhook backend health: Ensure pods are running and endpoints are populated.
6. Review webhook configuration: Look at rules, namespaceSelector, caBundle, and failurePolicy.
7. Test with dry-run: Simulate the request to see if it passes without side effects.
8. Inspect logs: Check API server logs and webhook backend logs for detailed causes.
9. Apply minimal fix: Change one item at a time (e.g., add label, adjust selector, update CA).
10. Verify the fix: Retry the original request and monitor.
11. Document the incident and resolution: Update runbooks and note any temporary changes.

Conclusion

Kubernetes Admission Controllers are powerful but can be a source of operational pain when misconfigured. By following a systematic approach—starting with environment inventory, understanding the specific error, diagnosing with the right commands, and applying targeted fixes—you can resolve most issues quickly and safely. Remember to always observe before changing, limit the blast radius, and verify results. Document your recovery procedures so that future incidents are handled even faster.

As a next step, implement a baseline monitoring check for your admission webhooks: regularly test that they are reachable and that their certificates are valid. Additionally, establish a policy for managing temporary webhook failures, including when to set failurePolicy to Ignore and how to track and revert such changes.

With these practices, you can maintain the security and policy benefits of admission controllers while minimizing their potential to disrupt your cluster operations.