## Introduction

Admission controllers are a critical gatekeeper in the Kubernetes API request pipeline. They intercept every create, update, and delete request, enforce policies, and can mutate resources before they are persisted. When they are misconfigured or overloaded, they become a bottleneck that slows down deployments, breaks CI/CD pipelines, and causes mysterious timeouts. This guide walks you through a systematic process to tune admission controller performance: from initial observation and version inventory to safe configuration changes, verification, failure recovery, and an operational checklist. Each section includes concrete commands, expected outputs, and practical examples for developers, DevOps consultants, and technical startup teams.

We will focus on two main types of admission controllers:

1. **Built-in admission controllers** (e.g., ResourceQuota, LimitRanger, PodSecurity) compiled into the Kubernetes API server.
2. **Dynamic admission webhooks** (e.g., ValidatingWebhookConfiguration, MutatingWebhookConfiguration) that call external HTTP services.

Performance problems can arise from either type, but dynamic webhooks are the most common source of latency because they add network round-trips to every API request. Throughout this article, we will use a typical scenario: a team notices that `kubectl apply` takes 10 seconds instead of 1 second, and pods stay in `Pending` state longer than expected.

## Version and Environment Inventory

Before touching any configuration, gather precise information about your Kubernetes version, the admission controllers currently enabled, and the webhooks installed. This baseline is essential for comparing before and after performance and for ensuring that any changes are compatible with your cluster.

### Check Kubernetes Version

Run the following command to get the server version:

```bash
kubectl version --short
```

Expected output (example):

```
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3
```

Note the server version; for admission controller tuning, versions 1.24 and later have consistent webhook timeout defaults (10 seconds) and support for `failurePolicy`.

### List Enabled Admission Controllers

You can see the enabled admission controllers on the API server by checking its flags. If you have access to the control plane nodes or the `kube-apiserver` manifest, look at the `--enable-admission-plugins` flag. For example:

```bash
ps aux | grep kube-apiserver | grep enable-admission-plugins
```

If you are using a managed Kubernetes service (EKS, GKE, AKS), you may not have direct access. Instead, you can infer some plugins by checking for specific resources or behaviors. For example, to see if `ResourceQuota` is enabled, create a namespace and try to apply a quota. However, the most reliable way for managed clusters is to consult the provider documentation.

Alternatively, use `kubectl api-versions` to see if certain APIs are available, which can hint at enabled admission controllers. For instance, if `policy/v1` is available and you can retrieve `PodDisruptionBudgets`, the `PodDisruptionBudget` admission controller is likely active.

### List Configured Admission Webhooks

Dynamic webhooks are defined in `ValidatingWebhookConfiguration` and `MutatingWebhookConfiguration` objects. List them:

```bash
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
```

Example output for validating webhooks:

```
NAME                          WEBHOOKS   AGE
gatekeeper-validating-webhook   1         30d
pod-policy.example.com          1         10d
```

For each webhook, inspect its configuration to see which resources it intercepts, its timeout, and failure policy:

```bash
kubectl get validatingwebhookconfiguration gatekeeper-validating-webhook -o yaml
```

The crucial fields are:

- `timeoutSeconds`: maximum time the API server waits for a webhook response (default is 10 seconds).
- `failurePolicy`: either `Fail` (request is rejected if webhook fails) or `Ignore` (request proceeds).
- `rules`: which API resources and operations trigger the webhook.

Example snippet:

```yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: gatekeeper-validating-webhook
webhooks:
- name: validation.gatekeeper.sh
  timeoutSeconds: 3
  failurePolicy: Fail
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["*"]
    apiVersions: ["*"]
    resources: ["pods", "deployments"]
```

### Check Webhook Endpoint Health

Webhooks are typically exposed as services in the cluster or as external URLs. Verify that the backing pods are running and responsive. For a service in the `gatekeeper-system` namespace:

```bash
kubectl get pods -n gatekeeper-system -o wide
kubectl logs -n gatekeeper-system <pod-name> --tail=50
```

If the webhook service has a health endpoint, test it with `kubectl port-forward` or `curl`. For example:

```bash
kubectl port-forward -n gatekeeper-system svc/gatekeeper-webhook-service 8443:443 &
curl -k https://localhost:8443/healthz
```

Expected output: `ok`.

## Safe Configuration Path

The next step is to make targeted changes to improve performance while minimizing risk. Always start with read-only observation, then apply the smallest change that addresses the bottleneck, and always have a rollback plan.

### Identify the Bottleneck

The first diagnostic is to measure the latency of API requests and determine whether admission webhooks are the culprit. Enable audit logging or use the Kubernetes API server metrics.

**1. Use Kubernetes API server metrics (if Prometheus is available)**

The API server exposes metrics at `/metrics`. Key metrics for admission:

- `apiserver_admission_controller_admission_duration_seconds`
- `apiserver_admission_webhook_admission_duration_seconds`

Query Prometheus for the 99th percentile latency:

```promql
histogram_quantile(0.99, rate(apiserver_admission_webhook_admission_duration_seconds_bucket[5m]))
```

If this value exceeds 1 second, webhooks are likely contributing to slowdowns.

**2. Use audit logs**

If audit logging is enabled, look for request stages and latencies. For example, in the audit log, find entries with `"stage":"ResponseComplete"` and inspect the `"annotations"` field for webhook latencies.

**3. Perform a manual latency test**

Create a test pod and time the request:

```bash
time kubectl run test-pod --image=nginx --restart=Never
```

Then delete it:

```bash
kubectl delete pod test-pod --wait=false
```

Compare the time with and without a particular webhook (by temporarily disabling it, if possible).

### Tuning Built-in Admission Controllers

Built-in admission controllers are generally efficient, but some can cause overhead. For example, `ResourceQuota` and `LimitRanger` perform additional lookups and calculations. However, they rarely are the main performance bottleneck. If you suspect a built-in controller, you can measure its duration via metrics:

```promql
apiserver_admission_controller_admission_duration_seconds{name="ResourceQuota", quantile="0.99"}
```

If the value is high, consider whether you can reduce the number of quotas or limit ranges, or whether the controller's scope can be narrowed (e.g., applying quotas to fewer namespaces). But in most cases, the issue is with dynamic webhooks.

### Tuning Dynamic Webhooks

The primary knobs for webhook performance are:

1. **Reduce `timeoutSeconds`**: This sets an upper bound, but it does not make the webhook faster. However, lowering it (e.g., from 10 to 3 seconds) can fail faster and avoid long waits when the webhook is down. However, if the webhook is slow but eventually returns within 3 seconds, lowering the timeout may cause more failures.

2. **Adjust `failurePolicy` to `Ignore`**: If a webhook is non-critical and experiences intermittent failures, setting `failurePolicy: Ignore` allows requests to proceed without the webhook's approval. This is a trade-off between availability and policy enforcement. Use only for advisory or audit-style webhooks.

3. **Narrow the webhook's scope**: The more resources and operations a webhook intercepts, the more requests it processes. Limit `rules` to only the necessary resources. For example, if a webhook only validates pods, do not include `deployments` in the rules (since deployments create pods, the pod creation will still trigger the webhook; intercepting deployments would cause duplicate calls).

4. **Improve the webhook service's performance**: Ensure the webhook backend has enough replicas, is horizontally scalable, and has low internal latency. Check CPU and memory usage:

```bash
kubectl top pods -n <webhook-namespace>
```

If the pods are resource-constrained, increase limits or replicas.

5. **Use object selectors**: If the webhook only needs to process a subset of objects (e.g., specific namespaces or labels), use `namespaceSelector` or `objectSelector` to filter at the API server side, reducing unnecessary webhook calls.

### Example: Tuning a Slow Validation Webhook

Assume you have a validating webhook that checks pod security policies and takes an average of 4 seconds per request. The current configuration looks like:

```yaml
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: pod-security-webhook
webhooks:
- name: pod-security.example.com
  timeoutSeconds: 10
  failurePolicy: Fail
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: ["*"]
    apiVersions: ["*"]
    resources: ["pods", "deployments", "statefulsets", "daemonsets", "jobs", "cronjobs"]
  namespaceSelector:
    matchLabels:
      pod-security: enabled
```

Issues: the webhook intercepts many resource types and has no timeout reduction. Additionally, the backend may be under-provisioned.

**Safe changes:**

1. Scope rules to only `pods` (since all other workload controllers create pods, the pod creation event is sufficient) or use `objectSelector` if possible.
2. Reduce `timeoutSeconds` to 3 seconds.
3. Scale the webhook deployment to more replicas.

Apply the changes:

```bash
kubectl apply -f updated-webhook.yaml
```

Then test with a pod creation:

```bash
time kubectl run test-pod --image=nginx --restart=Never
```

Observe the new latency and ensure the pod is created successfully.

## Verification and Diagnostics

After making changes, verify that performance has improved and that the admission controllers are functioning correctly. Use a combination of metrics, logs, and real-world tests.

### Verify Webhook Response Time

Use API server metrics to confirm the webhook latency has dropped. If Prometheus is not available, you can enable audit logging temporarily or use the `kubectl --v=6` flag to see detailed request timing.

For example, run:

```bash
kubectl get pod test-pod -v=6
```

The output includes timing for each phase, including admission:

```log
I1015 10:30:00.123456   12345 round_trippers.go:553] GET https://api-server:6443/api/v1/namespaces/default/pods/test-pod 200 OK in 123 milliseconds
```

The total time is not broken down by admission, but if the total time is close to your timeout setting, webhooks are still slow.

### Check Webhook Success/Failure Rates

If the webhook service exports metrics, monitor them. Otherwise, check the webhook service's logs for errors or timeouts. For example:

```bash
kubectl logs -n pod-security-webhook -l app=webhook --tail=100 | grep -i error
```

Also, check the API server logs for webhook timeouts. On a managed cluster, you may not have access, but on a self-managed cluster, check the kube-apiserver container logs:

```bash
journalctl -u kube-apiserver | grep -i webhook
```

Expected errors like:

```
E1015 10:35:00.234567   12345 dispatcher.go:173] failed calling webhook "pod-security.example.com": failed to call webhook: Post https://pod-security-webhook.default.svc:443/validate?timeout=3s: context deadline exceeded
```

### Validate that Policies are Still Enforced

Performance tuning should not compromise policy enforcement. Test that the admission controller still rejects invalid requests. For example, if the webhook enforces a label requirement, try creating a pod without the required label and expect a rejection:

```bash
kubectl run bad-pod --image=nginx --restart=Never
```

Expected output includes an error message from the webhook:

```
Error from server (Forbidden): admission webhook "pod-security.example.com" denied the request: missing required label 'app'
```

If the request is accepted, the webhook may be misconfigured (e.g., `failurePolicy: Ignore` causing it to skip on errors) or the rules no longer match.

### Performance Test at Scale

To simulate load, use a tool like `kubectl apply` in a loop or a load testing tool. For example, create 100 pods in batches and measure total time:

```bash
time for i in $(seq 1 100); do kubectl run test-$i --image=nginx --restart=Never --labels=app=test-$i; done
```

Then clean up:

```bash
kubectl delete pods -l app --selector 'app in (test-1,test-2,...)'
```

Compare the time before and after tuning. A significant reduction in total time indicates improvement.

## Failure Modes and Recovery

Even with careful tuning, admission controllers can fail and cause widespread outages. Understand the common failure modes and how to recover quickly.

### Webhook Timeout Failures

If a webhook exceeds its `timeoutSeconds`, the API server treats it as a failure according to `failurePolicy`. If `failurePolicy: Fail`, the request is rejected with an error like:

```
Error from server (InternalError): an error on the server ("") has prevented the request from succeeding
```

This can cause all matching requests to fail. To recover quickly:

1. **Temporarily disable the webhook** by setting `failurePolicy: Ignore` or removing the webhook configuration. For a temporary fix, you can edit the webhook:

```bash
kubectl edit validatingwebhookconfiguration pod-security-webhook
```

Change `failurePolicy` to `Ignore` and save. Requests will then proceed without validation.

2. **Scale up the webhook backend** to handle the load.

3. **Investigate the root cause** (e.g., network issues, resource exhaustion) and fix it.

### Webhook Service Unavailable

If the webhook service's endpoints are not ready (e.g., pods are crash-looping), the API server will also fail requests. Check the service endpoints:

```bash
kubectl get endpoints -n pod-security-webhook
```

If there are no endpoints, the service selector may not match the pods, or pods are not ready. Check pod status and logs.

### Misconfigured Webhook Rules

A webhook with overly broad rules can intercept more requests than intended, causing performance degradation and unexpected denials. For example, a webhook meant for pods but configured to intercept all resources (`resources: ["*"]`) will be called for every object. To recover, narrow the rules to specific resources.

### Built-in Admission Controller Issues

Although less common, built-in admission controllers can also cause problems. For example, if the `ResourceQuota` controller is slow, it might be due to a large number of quotas in the cluster. In extreme cases, you may need to disable the admission controller temporarily. This requires editing the API server manifest and is not recommended for production without thorough testing.

### Rollback Strategy

Always keep a backup of the original webhook configurations before making changes. Use `kubectl get <resource> -o yaml > backup.yaml` to save them. If a change causes issues, restore from backup:

```bash
kubectl apply -f backup.yaml
```

Additionally, version control your configurations using GitOps tools like Argo CD or Flux, which allow easy rollbacks.

## Operations Checklist

Use this checklist to ensure you have covered all aspects of admission controller performance tuning:

- [ ] Identify the Kubernetes version and enabled admission controllers.
- [ ] List all validating and mutating webhook configurations.
- [ ] Review each webhook's `timeoutSeconds` and `failurePolicy` settings.
- [ ] Measure baseline latency of API requests with and without webhooks.
- [ ] Analyze API server metrics for admission duration, if available.
- [ ] Identify the most time-consuming webhooks.
- [ ] Narrow webhook rules to minimum necessary resources and operations.
- [ ] Reduce `timeoutSeconds` to a value that balances availability and failure detection.
- [ ] Consider setting `failurePolicy: Ignore` for non-critical webhooks.
- [ ] Use `namespaceSelector` or `objectSelector` to filter webhook invocation.
- [ ] Scale webhook backend deployments appropriately.
- [ ] Monitor webhook backend resource usage (CPU, memory) and adjust limits.
- [ ] Verify that policies are still enforced after tuning.
- [ ] Conduct a performance test under realistic load.
- [ ] Document the changes and ensure rollback procedures are in place.
- [ ] Set up alerting on webhook latency and failure rates if not already present.

## Conclusion

Tuning Kubernetes admission controllers is a balance between performance and policy enforcement. By systematically measuring, making targeted changes, and verifying the results, you can eliminate bottlenecks and prevent outages. Remember to always have a rollback plan and to monitor the effects of your changes over time. With the practical steps in this guide, you can ensure that admission controllers do their job without slowing down your cluster.