Kubernetes Admission Controllers: Commands, Examples, and Daily Operations

Intro

Admission controllers in Kubernetes act as gatekeepers for API requests, intercepting them after authentication and authorization but before persistence. They enforce policies, validate resources, and mutate objects. This article provides a practical command-line reference for developers, DevOps consultants, and technical startup teams working with admission controllers.

You'll learn how to inspect enabled controllers, test configurations, diagnose failures, and recover from common issues. By following the examples, you'll move from observing a problem to verifying a fix with clear commands and expected outputs.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery steps.

Version and Environment Inventory

Before interacting with admission controllers, gather environment details. Run these read-only commands to understand your cluster's current state.

Check cluster version

kubectl version --short

Expected output includes client and server versions (e.g., Server Version: v1.28.2). Admission controller behavior can vary by version, so note the server version.

List enabled admission controllers Admission controllers are configured in the API server's flags. On control plane nodes, check the kube-apiserver process:

ps aux | grep kube-apiserver

Look for the --enable-admission-plugins flag. It lists enabled controllers in order, e.g., --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,ResourceQuota,ValidatingAdmissionWebhook,MutatingAdmissionWebhook.

Alternatively, on managed clusters (EKS, GKE, AKS), you may not have direct access. Instead, inspect the cluster configuration via cloud provider CLI or web console.

List current admission configurations If using ValidatingWebhookConfiguration or MutatingWebhookConfiguration:

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations

Example output:

NAME                                   WEBHOOKS   AGE
pod-policy.example.com                 1          5d

Inspect details

kubectl describe validatingwebhookconfiguration pod-policy.example.com

Look for Rules, NamespaceSelector, and FailurePolicy fields.

Prerequisites

• kubectl installed and configured with cluster admin or permissions to view webhook configurations.
• Access to control plane logs if debugging built-in controllers.
• In managed clusters, permissions to use provider commands (e.g., aws eks describe-cluster).

Safe Configuration Path

Making changes to admission controllers can affect cluster-wide behavior. Follow a cautious path to minimize risk.

Step 1: Observe current configuration Always capture the existing state before changes:

kubectl get validatingwebhookconfiguration pod-policy.example.com -o yaml > /tmp/original-webhook.yaml

This backup can be restored later if needed.

Step 2: Test in a development namespace Create a test namespace if one does not exist:

kubectl create namespace admission-test

Apply a sample pod manifest to see if it passes admission:

cat <<EOF | kubectl apply -n admission-test -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: nginx
    image: nginx:latest
EOF

If admission rejects, you'll see an error like:

Error from server (Forbidden): error when creating "STDIN": admission webhook "pod-policy.example.com" denied the request: Pod must have label 'team'

This tells you the policy is active and what it requires.

Step 3: Modify configuration with minimal change Example: Update a webhook's failure policy from Fail to Ignore to test behavior without blocking all requests. Edit the resource:

kubectl edit validatingwebhookconfiguration pod-policy.example.com

Change failurePolicy: Fail to failurePolicy: Ignore, save, and exit. The API server will update the configuration.

Step 4: Verify the change Run the test pod again and check it is created (if policy passed) or still rejected (if policy still active).

kubectl apply -n admission-test -f test-pod.yaml
kubectl get pods -n admission-test

If the pod is created, the change took effect. Monitor API server logs for webhook call errors if something goes wrong.

Step 5: Clean up test resources

kubectl delete namespace admission-test

Always remove test artifacts.

Verification and Diagnostics

After changes, verify that admission controllers perform as expected. Use these commands to check status and troubleshoot.

Check API server health Admission controllers run as part of the API server. If API server is unhealthy, admission requests fail. On a control plane node, check the API server pod (if using static pods):

kubectl get pods -n kube-system | grep kube-apiserver

Or check the process:

curl -k https://localhost:6443/healthz?verbose

Expected output includes ok for all checks, including admission if applicable.

View admission controller logs API server logs contain admission decisions and potential errors. Access logs using journalctl on control plane nodes:

journalctl -u kube-apiserver -f | grep admission

For managed clusters, use cloud provider log services (e.g., CloudWatch for EKS).

Test with a known-bad request Create a pod that violates a policy to see rejection:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: bad-pod
  namespace: default
spec:
  containers:
  - name: nginx
    image: nginx
EOF

If a policy requires labels, the output will show the webhook's denial message. This confirms enforcement.

Inspect webhook configurations Ensure webhook service endpoints are reachable:

kubectl get endpoints -n webhook-namespace

If no endpoints, webhook service may not be running. Check pods:

kubectl get pods -n webhook-namespace

If the webhook service is down and failurePolicy is Fail, all matching requests will be rejected. Change to Ignore or fix the service.

Use dry-run to test without persisting Kubernetes supports dry-run for some resources. For example:

kubectl apply -f test-pod.yaml --dry-run=server

This sends the request through admission without persisting, showing whether it would be accepted. Useful for testing policies.

Failure Modes and Recovery

Admission controllers can fail, causing API requests to be rejected or hung. Recognize these modes and recover quickly.

Failure mode 1: Webhook service unavailable If a validating webhook's service is down and failurePolicy is Fail, requests matching its rules will be rejected with an error:

Error from server (InternalError): error when creating "pod.yaml": Internal error occurred: failed calling webhook "pod-policy.example.com": Post "https://webhook-service.webhook-namespace.svc:443/validate?timeout=10s": dial tcp 10.0.0.5:443: connect: connection refused

Recovery: Either restore the webhook service or temporarily set failurePolicy to Ignore:

kubectl patch validatingwebhookconfiguration pod-policy.example.com --type='json' -p='[{"op": "replace", "path": "/webhooks/0/failurePolicy", "value":"Ignore"}]'

Then fix the service and restore Fail.

Failure mode 2: Webhook timeout Webhook calls have a timeout (default 10 seconds). If the webhook takes too long, requests fail. Check webhook logs for slow processing, and increase timeout in the webhook configuration if needed:

webhooks:
- name: pod-policy.example.com
  timeoutSeconds: 30

Apply with kubectl edit. Also scale webhook replicas to handle load.

Failure mode 3: Misconfigured webhook rules If webhook rules match unexpected resources, it may block unrelated operations. For example, a rule matching all pods in all namespaces. Adjust namespaceSelector or rules to limit scope. Use labels or namespace selectors to target only intended objects.

Recovery from bad configuration If a new admission controller configuration breaks the cluster (e.g., all pod creations fail), revert to previous configuration:

kubectl apply -f /tmp/original-webhook.yaml

Or, if API server is not starting due to invalid admission plugin flag, edit the kube-apiserver manifest on the control plane node (/etc/kubernetes/manifests/kube-apiserver.yaml) and restart kubelet. For managed clusters, contact provider support.

Operations Checklist

Use this checklist to routinely verify admission controller health and policy compliance.

• [ ] Daily: Check API server health (kubectl get --raw='/healthz?verbose').
• [ ] Weekly: Review webhook configurations for unexpected changes (kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml | diff - previous-config/).
• [ ] Monthly: Audit admission logs for denied requests that might indicate policy misconfiguration or attack attempts. Use cloud log services or API server logs.
• [ ] Before any change: Backup current webhook configurations and test in a sandbox namespace.
• [ ] After any change: Verify that both conforming and non-conforming requests behave as expected (dry-run or test pod).

Assign a single owner for admission controller policies, e.g., "Platform Engineering Lead", responsible for reviewing changes and ensuring documentation is up to date. Revisit the checklist quarterly to adjust for new policies or cluster changes.

Common Pitfalls

Here are frequent mistakes when managing admission controllers and how to avoid them.

Pitfall 1: Setting failurePolicy to Fail without a reliable webhook

• Why: If webhook service is down, all matching API requests fail, causing outages.
• Avoid: Use failurePolicy=Ignore for non-critical policies or ensure high availability of webhook service.
• Recover: Patch failurePolicy to Ignore temporarily and fix service.

Pitfall 2: Overly broad webhook rules

• Why: Rules match all resources or namespaces, interfering with system operations.
• Avoid: Use namespace selectors or object labels to limit scope. For example, only match pods with label app: myapp.
• Recover: Edit rules to narrow scope.

Pitfall 3: Ignoring timeout settings

• Why: Default 10s timeout might be too short for complex webhooks, causing intermittent failures.
• Avoid: Set timeoutSeconds appropriately after load testing webhook.
• Recover: Increase timeout in webhook configuration.

Pitfall 4: Not testing in a development namespace

• Why: Applying a new webhook directly to production can break existing workloads if policy rejects them.
• Avoid: Always test in a separate namespace with a subset of resources before deploying cluster-wide.
• Recover: Roll back webhook configuration if production is affected.

Pitfall 5: Forgetting to update webhook certificates

• Why: Webhooks often use TLS. Expired certificates cause API server to reject webhook calls.
• Avoid: Use cert-manager or automated renewal and monitor expiration dates.
• Recover: Update certificate and restart webhook pods.

Conclusion

Managing Kubernetes admission controllers requires careful observation, precise changes, and robust recovery procedures. By using the commands and examples in this article, you can inspect enabled controllers, test policies, diagnose failures, and recover from common issues.

As a next step, choose one low-risk verification: inspect your cluster's webhook configurations, backup them, and test a policy in a development namespace. Record the current state, run the documented checks, and compare results with expected behavior. Review dependencies such as namespaces and roles to ensure policies are correctly scoped.

A reliable operational workflow makes failures visible, protects sensitive values, limits changes to intended resources, and defines recovery before an incident forces a decision.