## Intro

Kubernetes events are the primary record of why something happened in your cluster. When a pod fails, a deployment scales, or a node becomes unready, events capture the reason from controllers, schedulers, kubelets, and the API server. They are essential for debugging, auditing, and automation. However, in large or busy clusters, events can become a performance problem themselves: high event volume can overwhelm the API server, slow down `kubectl get events`, fill etcd storage, and hide critical warnings in a flood of noise.

This article is a practical guide for Kubernetes operators, DevOps engineers, and platform teams who need to tune event handling for performance and reliability. It focuses on identifying event latency and bottlenecks, optimizing event generation and retention, and verifying that your changes reduce load without losing critical visibility. We will cover version and environment inventory, safe configuration paths, verification and diagnostics, failure modes and recovery, and an operations checklist. Each section includes concrete commands, expected outputs, and decision criteria.

The goal is operational safety: observe before changing, limit blast radius, use placeholders instead of secrets, verify results, and document how to recover if the expected state is not reached.

## Version and Environment Inventory

Before tuning Kubernetes events, you need to know your cluster version, event sources, and current event rate. The behavior of event handling has changed across Kubernetes versions, particularly around the `Event` API and the newer `events.k8s.io` API group. Start by identifying what you are working with and measuring the baseline.

### Identify Kubernetes Version and Event API Support

Run the following command to check the server version:

```bash
kubectl version --short
```

Expected output resembles:

```
Client Version: v1.29.2
Server Version: v1.28.5
```

If your server version is 1.19 or later, the `events.k8s.io/v1` API is available. This newer API provides more structured events and is used by some components. Check if any events are being created in that API group:

```bash
kubectl get events --all-namespaces --field-selector type!=Normal
```

You can also list event API resources:

```bash
kubectl api-resources | grep events
```

Expected output:

```
events                            events.k8s.io/v1                true         Event
events                            v1                              true         Event
```

### Inventory Event Sources and Rate

Determine which controllers and nodes generate the most events. Use `kubectl get events` with sorting and field selectors to identify noisy sources.

List all events in the last hour (if supported by your cluster's event retention):

```bash
kubectl get events --all-namespaces --sort-by=.metadata.creationTimestamp
```

Count events by source component:

```bash
kubectl get events --all-namespaces -o json | jq -r '.items[].source.component' | sort | uniq -c | sort -nr
```

Example output:

```
  834 kubelet
  156 deployment-controller
   89 scheduler
   34 replicaset-controller
   12 node-controller
```

This tells you which components generate the most events. In this example, kubelet is the top source, which often indicates frequent pod restarts or probe failures.

Check event count by namespace:

```bash
kubectl get events --all-namespaces -o json | jq -r '.items[].metadata.namespace' | sort | uniq -c | sort -nr
```

Example output:

```
  456 default
  210 kube-system
   98 production-app
   23 monitoring
```

High event rates in a namespace may point to application misconfiguration, such as crash-looping pods.

### Check etcd Size and Event Retention

Events are stored in etcd and have a default TTL of one hour (set by `--event-ttl` on the kube-apiserver). If event volume is high, etcd can grow significantly. Check etcd size (if you have access) or monitor via metrics.

For a managed cluster like EKS or GKE, use the cloud provider's monitoring console. For self-managed, you can inspect etcd metrics:

```bash
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --write-out=table
```

Look for `DB SIZE`. If it exceeds your capacity expectations and events are a major contributor, consider tuning event TTL or aggregation.

### Prerequisites and Access Requirements

To perform these checks, you need:

- `kubectl` configured with cluster-admin or at least read access to events across all namespaces.
- `jq` installed for JSON processing (optional but helpful).
- For etcd inspection on self-managed clusters, SSH access to control plane nodes and etcd client certificates.
- In managed clusters, access to the cloud provider's monitoring dashboards and logs.

## Safe Configuration Path

The primary levers for event performance are the kube-apiserver flags `--event-ttl`, `--max-mutating-requests-inflight`, and `--max-requests-inflight`, plus event aggregation settings. Changing these can have broad blast radius, so follow a careful, reversible approach.

### Understand the Default Settings

The kube-apiserver has several relevant parameters:

- `--event-ttl`: Time to retain events, default 1h.
- `--max-requests-inflight`: Maximum non-mutating requests, default 400.
- `--max-mutating-requests-inflight`: Maximum mutating requests, default 200.
- `--audit-log-maxbackup`: Not directly related but affects logging load.

Event creation is a mutating request. If the API server is saturated with event writes, other mutating requests (pod creations, updates) can be delayed. Raising `--max-mutating-requests-inflight` can increase throughput but also increases memory usage and etcd load.

### Tuning Event TTL

If your etcd is bloated with events, reducing `--event-ttl` can free space. For example, to set TTL to 15 minutes:

Edit the kube-apiserver manifest on each control plane node (typically `/etc/kubernetes/manifests/kube-apiserver.yaml`) and add or modify:

```yaml
spec:
  containers:
  - command:
    - kube-apiserver
    - --event-ttl=15m
```

Then the kubelet will restart the apiserver automatically. Verify the flag is active:

```bash
kubectl get pod -n kube-system -l component=kube-apiserver -o yaml | grep event-ttl
```

Expected output:

```
    - --event-ttl=15m
```

Impact: Events older than 15 minutes are deleted, reducing etcd size but also historical debugging information. Choose a value balanced for your needs.

### Enabling Event Aggregation

Kubernetes has an event aggregation mechanism in the kube-apiserver (not enabled by default in older versions, but available as a feature gate). In newer versions (1.13+), event aggregation is on by default. It groups similar events to reduce noise. You can verify by checking the apiserver logs for aggregation messages:

```bash
kubectl logs -n kube-system kube-apiserver-control-plane -c kube-apiserver | grep -i aggregate
```

If you need to adjust aggregation parameters, they are hard-coded in the source and not exposed via flags. In that case, focus on reducing source events.

### Reducing Event Generation at the Source

Often, the best tuning is to stop generating unnecessary events. Common causes:

- Pods with failing probes causing constant restarts.
- Deployments with rolling update failures.
- Nodes with flapping conditions.

Fix the underlying issue and event volume drops. Use `kubectl describe pod <name>` and `kubectl logs <pod> --previous` to diagnose.

### Applying Changes Safely

When modifying kube-apiserver flags:

1. Backup the original manifest file:
   ```bash
   cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.bak
   ```
2. Make the change on one control plane node first.
3. Observe the API server restart and cluster stability.
4. Verify with `kubectl get nodes` and `kubectl get --raw='/readyz?verbose'`.
5. If healthy, apply to other control plane nodes one at a time.

## Verification and Diagnostics

After making any tuning change, verify that event handling improves and the cluster remains stable. Use metrics, logs, and practical commands to confirm.

### Monitor API Server Request Latency and Errors

Check API server metrics for event-related request duration:

```bash
kubectl get --raw /metrics | grep apiserver_request_duration_seconds | grep events
```

Look for high p99 or increasing error rates. Compare before and after your change.

Alternatively, use `kubectl top` or monitoring dashboards (Prometheus/Grafana) to visualize.

### Check Event Rate and Latency

Measure how quickly you can list events:

```bash
time kubectl get events --all-namespaces --sort-by=.lastTimestamp
```

If it completes in under a few seconds, that's acceptable. If it takes tens of seconds, the API server may be overloaded or etcd slow.

Count events over a period to see the rate:

```bash
watch -n 5 'kubectl get events --all-namespaces --no-headers | wc -l'
```

Observe the count change; if it's increasing rapidly, event sources are still generating noise.

### Verify etcd Size Reduction (if TTL changed)

If you reduced TTL, wait for the old events to expire (based on the old TTL) and then check etcd size again with the endpoint status command. Expect a decrease if events were a significant portion.

### Test Event Creation Under Load

Generate a test event to ensure the API accepts event creation promptly:

```bash
kubectl create event test-event --namespace=default --type=Normal --reason=TestReason --message="Testing event latency"
```

Then immediately list it:

```bash
kubectl get event test-event -n default
```

Check that it appears within a second. If delayed, investigate API server performance.

### Diagnose Event Source Issues

For noisy sources, dig into specific pods or nodes:

```bash
kubectl describe pod <noisy-pod> -n <namespace>
```

Look for repeated events like `BackOff`, `FailedScheduling`, or `ProbeWarning`. Use `kubectl logs <pod> --previous` to see crash reasons.

## Failure Modes and Recovery

Tuning event handling can lead to new failures. Know what to watch for and how to recover quickly.

### Failure: API Server Unstable or Unresponsive After Flag Change

If changing `--max-mutating-requests-inflight` or other flags causes the API server to crash loop or become unresponsive:

- Revert the manifest to the backup.
- If the API server is completely down, access the control plane node directly.
- Restore the backup file:
  ```bash
  cp /root/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml
  ```
- The kubelet will restart the API server with the original settings.
- Verify cluster health.

### Failure: Events Disappear Too Quickly (TTL Too Short)

If you set `--event-ttl` too low, you may lose events needed for debugging. A user reports they cannot see events from a failure 20 minutes ago. Recovery:

- Increase TTL back to a longer period, e.g., 1 hour.
- Apply the change as described and verify.
- Consider implementing external event export (e.g., to Elasticsearch or Loki) for long-term retention instead of storing in etcd.

### Failure: Increased Memory Usage on API Server

Raising inflight request limits may increase memory consumption. Monitor API server memory via `kubectl top pod -n kube-system`. If it approaches limits, reduce the values or add more resources to the control plane nodes.

### Failure: Event Aggregation Hides Important Events

If aggregation is too aggressive (older versions), critical events may be merged and hard to find. In that case, you may need to disable aggregation (not recommended) or adjust your event monitoring to capture all events before aggregation using an event exporter like `event-exporter` from Kubernetes or third-party tools.

### Recovery Verification

Always verify recovery:

```bash
kubectl get --raw='/readyz?verbose'
```

Should output `ok` for all checks.

Check event creation:

```bash
kubectl create event recovery-test --namespace=default --type=Normal --reason=RecoveryCheck --message="Recovery verified"
```

And ensure it appears.

## Operations Checklist

Use this checklist before, during, and after tuning Kubernetes events to ensure safe operations.

### Pre-Change Checklist

- [ ] Confirm Kubernetes version and event API availability.
- [ ] Measure current event rate and source distribution.
- [ ] Check etcd size and event contribution.
- [ ] Backup kube-apiserver manifest on all control plane nodes.
- [ ] Document current flag values:
  ```bash
  kubectl get pod -n kube-system -l component=kube-apiserver -o json | jq '.items[].spec.containers[].command'
  ```
- [ ] Ensure you have direct access to control plane nodes for emergency recovery.

### Change Execution Checklist

- [ ] Apply change to one control plane node at a time.
- [ ] Wait for API server to restart and become ready.
- [ ] Run `kubectl get --raw='/readyz?verbose'` to verify API server health.
- [ ] Observe event listing performance.
- [ ] Monitor API server resource usage (CPU, memory).

### Post-Change Verification Checklist

- [ ] Compare event rate before and after.
- [ ] Confirm event latency is acceptable (e.g., create and list test event).
- [ ] Verify etcd size trend if TTL changed.
- [ ] Check that essential events (e.g., pod failures) are still visible.
- [ ] Roll out to remaining control plane nodes if stable.

### Ongoing Operations

- [ ] Set up monitoring alerts for API server request latency and error rates.
- [ ] Implement event export if long-term retention is required.
- [ ] Periodically review top event sources and fix root causes.
- [ ] Document all tuning changes in a runbook.

## Conclusion

Kubernetes event performance tuning is about balancing visibility and resource usage. By measuring event rates, identifying noisy sources, and carefully adjusting API server settings such as event TTL, you can reduce etcd load and API pressure without losing critical operational data. Always follow a systematic approach: inventory, safe configuration, verification, and recovery planning.

As a next step, start with the non-invasive checks: identify your cluster version, measure event sources with the provided jq commands, and assess whether event volume is causing problems. Then consider applying the least risky change, such as fixing a noisy pod or adjusting event TTL, and verify the impact with concrete metrics. Remember to document your baseline and recovery path so that any tuning remains safe and reversible.