## Intro

Kubernetes has become the de facto standard for container orchestration, but as clusters grow and workloads become more demanding, performance bottlenecks can emerge. These bottlenecks often manifest as slow API responses, scheduling delays, network latency, or resource contention. Tuning the Kubernetes architecture itself - not just the applications running on it - is essential to maintain reliability and efficiency.

This article provides a practical guide to identifying and resolving common Kubernetes performance issues. We will cover how to inventory your cluster, make safe configuration changes, verify the impact, and recover if something goes wrong. The goal is to help you move from reactive troubleshooting to proactive optimization with concrete commands and examples.

## Version and Environment Inventory

Before making any changes, you need to establish a clear picture of your Kubernetes environment. This includes the exact versions of the control plane components and nodes, the cluster topology, and the prerequisites for tuning.

### Check Cluster Version and Component Versions

Start by checking the version of the Kubernetes control plane and nodes. Use `kubectl version` to see both client and server versions:

```bash
kubectl version --short
```

Expected output should show the server version, for example:

```
Client Version: v1.21.0
Server Version: v1.21.0
```

If you have access to the control plane node, you can also check the individual component versions by examining the running processes or their container images. For example, for the API server:

```bash
kubectl get pods -n kube-system -l component=kube-apiserver -o jsonpath='{.items[0].spec.containers[0].image}'
```

This will output the image tag, such as:

```
k8s.gcr.io/kube-apiserver:v1.21.0
```

### Understand Topology

Document the topology: how many master nodes and worker nodes, their roles, and any high availability configuration. Use `kubectl get nodes` with the `-o wide` flag to see IP addresses, OS, and container runtime versions:

```bash
kubectl get nodes -o wide
```

Example output:

```
NAME       STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master-1   Ready    control-plane,master   30d   v1.21.0   10.0.0.1      <none>        Ubuntu 20.04.2 LTS   5.4.0-72-generic   docker://20.10.7
worker-1   Ready    <none>                 30d   v1.21.0   10.0.0.2      <none>        Ubuntu 20.04.2 LTS   5.4.0-72-generic   docker://20.10.7
worker-2   Ready    <none>                 30d   v1.21.0   10.0.0.3      <none>        Ubuntu 20.04.2 LTS   5.4.0-72-generic   docker://20.10.7
```

### Verify Prerequisites

Ensure that you have the necessary permissions to make changes. You should have cluster-admin or at least write access to the configurations you plan to modify. Also confirm that etcd is healthy, as it is central to Kubernetes performance. On a control plane node with etcdctl installed, you can check health:

```bash
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health
```

Expected output indicates each endpoint is healthy:

```
https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.1ms
```

If any of these checks reveal issues, resolve them before proceeding with tuning.

## Safe Configuration Path

Making changes directly to a running cluster can be risky. The safe approach is to make scoped, reversible changes one at a time, ideally in a test environment first. This section outlines common tuning areas and how to adjust them safely.

### API Server Tuning

The API server is the entry point for all Kubernetes operations. Common issues include request latency under load and excessive memory usage. Key parameters to consider are `--max-requests-inflight` and `--max-mutating-requests-inflight`, which limit the number of simultaneous read and write requests, respectively. Increasing these can improve throughput if the API server has spare CPU and memory, but too high a value can overwhelm it.

To change these, you typically edit the API server manifest on the control plane node. For a kubeadm cluster, the file is usually `/etc/kubernetes/manifests/kube-apiserver.yaml`. Add or modify the spec for the container:

```yaml
spec:
  containers:
  - command:
    - kube-apiserver
    - --max-requests-inflight=400
    - --max-mutating-requests-inflight=200
```

After saving, the kubelet will restart the API server automatically. Verify the new process arguments with:

```bash
ps aux | grep kube-apiserver | grep max-requests-inflight
```

Expected output includes the new values.

### Controller Manager and Scheduler Tuning

The controller manager and scheduler run as static pods. Their performance can be tuned by adjusting the `--kube-api-qps` and `--kube-api-burst` flags, which control the rate of API requests they make. Increasing these can speed up reconciliation but also increases load on the API server. For example, in the controller manager manifest (`/etc/kubernetes/manifests/kube-controller-manager.yaml`), you can add:

```yaml
spec:
  containers:
  - command:
    - kube-controller-manager
    - --kube-api-qps=50
    - --kube-api-burst=100
```

Similarly for the scheduler in `/etc/kubernetes/manifests/kube-scheduler.yaml`.

### Kubelet Tuning

The kubelet on each node manages pod lifecycles. Tuning its `--max-pods` can help if you have many small pods. The default is 110 pods per node. If your nodes have ample resources, you can increase this. The setting is in the kubelet configuration file, typically `/var/lib/kubelet/config.yaml`:

```yaml
maxPods: 110
```

Change to a higher value, like 250, and restart the kubelet:

```bash
systemctl restart kubelet
```

Check the kubelet logs to confirm the new value is loaded:

```bash
journalctl -u kubelet | grep maxPods
```

### Kube Proxy Tuning

Kube proxy is responsible for network routing. Its mode (iptables or IPVS) has a big impact on performance. IPVS is generally more efficient for large clusters. To change from iptables to IPVS, you need to ensure the IPVS kernel modules are loaded, then edit the kube-proxy configuration. For kubeadm, the config is in a ConfigMap in the kube-system namespace. First, verify current mode:

```bash
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode
```

Expected output: `mode: ""` (which defaults to iptables). To change to IPVS, edit the ConfigMap:

```bash
kubectl edit configmap kube-proxy -n kube-system
```

Set `mode: "ipvs"` under the appropriate section, save, and then restart the kube-proxy pods:

```bash
kubectl delete pods -n kube-system -l k8s-app=kube-proxy
```

The pods will be recreated with the new configuration.

### Safe Implementation Practices

- Always make one change at a time, then observe before making another.
- Keep backups of original manifest files. For example:

```bash
cp /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml.bak
```

- Use a version control system for configuration changes.
- Test in a staging cluster if possible.

## Verification and Diagnostics

After making changes, you need to verify that they are working as expected and not causing regressions. This section covers key metrics and checks.

### API Server Latency and Throughput

You can use kubectl to measure API server response times. For example, time a list request:

```bash
time kubectl get pods --all-namespaces
```

Compare the time before and after tuning. A well-tuned API server should respond in well under a second for a small number of objects, but this depends on scale.

For more detailed metrics, if you have Prometheus set up, query for `apiserver_request_duration_seconds` percentiles. If not, you can check the API server logs for slow requests:

```bash
kubectl logs -n kube-system kube-apiserver-master-1 | grep 'apiserver_request_total' | tail
```

### Scheduler Throughput

The scheduler processes pending pods. Check for pending pods that are not being scheduled quickly. Use `kubectl get pods --all-namespaces --field-selector=status.phase=Pending` to see if any are stuck. Also, look at the scheduler logs:

```bash
kubectl logs -n kube-system kube-scheduler-master-1 | tail
```

If you increased the scheduler's API QPS, you should see fewer delays in log timestamps between scheduling attempts.

### Node Resource Pressure

Check node conditions for resource pressure, such as MemoryPressure or DiskPressure:

```bash
kubectl describe nodes | grep -A5 Conditions
```

Expected output shows `MemoryPressure False` and `DiskPressure False`. If any become True, the kubelet may evict pods, causing disruption.

### Network Performance

If you switched to IPVS, verify that kube-proxy is using IPVS by checking the logs:

```bash
kubectl logs -n kube-system kube-proxy-xxxxx | grep 'Using ipvs Proxier'
```

You should see a line like `Using ipvs Proxier`. Additionally, you can check the IPVS rules on the node:

```bash
sudo ipvsadm -L -n | head
```

Expected output shows the service IPs and backends.

### Etcd Health

Performance of Kubernetes heavily depends on etcd. Monitor etcd metrics such as `etcd_disk_wal_fsync_duration_seconds` and `etcd_network_peer_round_trip_time_seconds`. If you have etcdctl, you can check the endpoint status:

```bash
ETCDCTL_API=3 etcdctl endpoint status --cluster -w table
```

Example output:

```
+------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT         | ID               | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------------+------------------+---------+---------+-----------+-----------+------------+
| 10.0.0.1:2379   | 8e9e05c52164694d | 3.4.13  | 25 MB   | true      | 3         | 12345      |
+------------------+------------------+---------+---------+-----------+-----------+------------+
```

Deviations from normal (e.g., high fsync duration) indicate potential issues.

## Failure Modes and Recovery

Even with careful planning, tuning changes can cause unexpected problems. This section describes common failure modes and how to recover.

### API Server CrashLoopBackOff

If you increase `--max-requests-inflight` too high, the API server might run out of memory and crash. This can lead to a CrashLoopBackOff. Symptoms include the API server pod restarting repeatedly and the cluster becoming unresponsive to kubectl commands. To recover, revert the manifest to the original values. Since the API server is down, you need to access the control plane node directly and restore the backup:

```bash
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml
```

The kubelet will restart the API server with the previous configuration. After recovery, check the API server logs for out-of-memory errors and adjust the values more conservatively.

### Scheduler Failing to Schedule Pods

If you change scheduler flags incorrectly, the scheduler may fail to start or crash. This results in new pods staying in Pending state indefinitely. To diagnose, check the scheduler pod status:

```bash
kubectl get pods -n kube-system -l component=kube-scheduler
```

If it is not running, inspect the logs (if possible) or the manifest file for syntax errors. Correct the manifest and save; the scheduler will be restarted. If the scheduler is running but still not scheduling, check its logs for errors like API connection failures and verify the `--kube-api-qps` and `--kube-api-burst` values are not set to zero.

### Kubelet Not Starting After maxPods Change

If you set `maxPods` to a value that exceeds the node's capability (e.g., too high for the pod CIDR range), the kubelet may fail to start. The node will show as NotReady. Check the kubelet status:

```bash
systemctl status kubelet
```

If failed, examine the logs for the specific error:

```bash
journalctl -u kubelet -n 50 --no-pager
```

Common error: `failed to reserve container for pod: invalid maxPods value`. To recover, edit `/var/lib/kubelet/config.yaml` and set `maxPods` back to a valid value (e.g., 110), then restart kubelet and verify the node becomes Ready:

```bash
kubectl get nodes
```

### Kube Proxy Issues After IPVS Change

If IPVS modules are not loaded, kube-proxy may fail to start, leaving services unreachable. Check the kube-proxy pod logs for errors like `can't initialize ipvs module`. To recover, either load the necessary kernel modules or revert the mode to iptables in the ConfigMap. For quick recovery, edit the ConfigMap back to `mode: ""` and restart kube-proxy:

```bash
kubectl edit configmap kube-proxy -n kube-system
kubectl delete pods -n kube-system -l k8s-app=kube-proxy
```

### General Rollback Strategy

Always keep backups of any file or ConfigMap you modify. For files on nodes, use `cp` to create a `.bak` file. For Kubernetes objects, you can export the original YAML before modification:

```bash
kubectl get configmap kube-proxy -n kube-system -o yaml > kube-proxy-original.yaml
```

If a change causes issues, restore the original using `kubectl apply -f kube-proxy-original.yaml` or by copying the backup file back. Document any change in a change log with the date, what was changed, and why, to aid in troubleshooting.

## Common Pitfalls and How to Avoid Them

Tuning Kubernetes can be tricky, and many operators fall into the same traps. Here are some frequent mistakes and how to steer clear of them.

### Making Multiple Changes at Once

When you modify several parameters simultaneously, it becomes difficult to identify which change caused an improvement or a regression. For example, increasing API server inflight requests while also switching kube-proxy to IPVS could mask a problem with one change. To avoid this, make one change at a time, test thoroughly, and observe for at least a few hours before moving on.

### Ignoring Resource Headroom

Setting limits too high without ensuring the control plane nodes have enough CPU and memory can lead to crashes. For instance, raising `--max-requests-inflight` to 1000 on a node with only 2 GB of RAM available to the API server will likely cause out-of-memory kills. Always check current resource usage with `kubectl top nodes` and `kubectl top pods -n kube-system` before adjusting limits, and use conservative increments.

### Not Testing in a Staging Environment

Applying tuning directly to production is risky. A configuration that works in a small test cluster may fail under production load. For example, IPVS may work fine with 10 services but hit issues with 1000 services due to kernel limits. If possible, replicate your production topology in a staging cluster and run load tests before rolling out changes.

### Skipping Backups

Failing to back up manifests and ConfigMaps means a bad change could leave you scrambling. Always create a backup file or export the original object before editing. For manifests, use `cp file.yaml file.yaml.bak`; for Kubernetes objects, use `kubectl get -o yaml > original.yaml`. Store backups in a version control system as well.

### Not Monitoring After Changes

After a change, some operators immediately move on without verifying the impact. This can allow performance regressions to go unnoticed. For example, after increasing scheduler QPS, you should monitor scheduler latency and pending pod counts for at least a day. Set up alerts for key metrics like API server request duration and etcd fsync duration to catch issues early.

## Operations Checklist

To keep your Kubernetes architecture performing well, follow this checklist on a regular basis. Assign an owner for each item and set a review frequency. The owner should be a single accountable individual, such as a platform engineer or cluster administrator.

| Item | Owner | Frequency |
|---|---|---|
| Verify cluster and component versions are consistent and supported | Platform Engineer (e.g., Priya Shah) | Monthly |
| Check etcd health and performance metrics (fsync duration, leader changes) | Database Administrator (e.g., Carlos Gomez) | Weekly |
| Monitor API server request duration and error rates; alert on high latency | SRE (e.g., Alex Chen) | Daily (automated alerts) |
| Review scheduler and controller manager logs for repeated errors or delays | Platform Engineer | Weekly |
| Check node conditions and resource usage (CPU, memory, disk) for pressure | SRE | Daily (automated alerts) |
| Validate network performance; if using IPVS, ensure rules are correct | Network Engineer (e.g., Maria Rossi) | Monthly |
| Test disaster recovery procedures for each component (e.g., restore API server manifest) | Platform Engineer | Quarterly |
| Keep configuration files under version control and maintain backups | Platform Engineer | Continuous |
| Schedule performance tests after any significant cluster change or upgrade | Performance Engineer (e.g., David Kim) | After each change |
| Document all tuning changes with rationale and observed effects | Platform Engineer | Continuous |

Owners should review their items at the specified frequency. For example, the Platform Engineer reviews the node conditions weekly and adjusts alerts as needed. This ensures accountability and keeps the cluster healthy.

## Conclusion

Tuning Kubernetes architecture performance is a continuous process of measuring, adjusting, and verifying. By following the practical steps in this guide, you can systematically identify bottlenecks and apply safe, reversible changes. Remember to start with a thorough inventory, change one parameter at a time, and always have a rollback plan. The operations checklist will help you maintain optimal performance as your cluster evolves. With careful tuning, you can achieve lower latency, higher throughput, and a more reliable platform for your workloads.