Intro
Kubernetes has become the de facto standard for container orchestration, but as clusters grow and workloads become more demanding, performance bottlenecks can emerge. These bottlenecks often manifest as slow API responses, scheduling delays, network latency, or resource contention. Tuning the Kubernetes architecture itself - not just the applications running on it - is essential to maintain reliability and efficiency.
This article provides a practical guide to identifying and resolving common Kubernetes performance issues. We will cover how to inventory your cluster, make safe configuration changes, verify the impact, and recover if something goes wrong. The goal is to help you move from reactive troubleshooting to proactive optimization with concrete commands and examples.
Version and Environment Inventory
Before making any changes, you need to establish a clear picture of your Kubernetes environment. This includes the exact versions of the control plane components and nodes, the cluster topology, and the prerequisites for tuning.
Check Cluster Version and Component Versions
Start by checking the version of the Kubernetes control plane and nodes. Use kubectl version to see both client and server versions:
kubectl version --short
Expected output should show the server version, for example:
Client Version: v1.21.0
Server Version: v1.21.0
If you have access to the control plane node, you can also check the individual component versions by examining the running processes or their container images. For example, for the API server:
kubectl get pods -n kube-system -l component=kube-apiserver -o jsonpath='{.items[0].spec.containers[0].image}'
This will output the image tag, such as:
k8s.gcr.io/kube-apiserver:v1.21.0
Understand Topology
Document the topology: how many master nodes and worker nodes, their roles, and any high availability configuration. Use kubectl get nodes with the -o wide flag to see IP addresses, OS, and container runtime versions:
kubectl get nodes -o wide
Example output:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
master-1 Ready control-plane,master 30d v1.21.0 10.0.0.1 <none> Ubuntu 20.04.2 LTS 5.4.0-72-generic docker://20.10.7
worker-1 Ready <none> 30d v1.21.0 10.0.0.2 <none> Ubuntu 20.04.2 LTS 5.4.0-72-generic docker://20.10.7
worker-2 Ready <none> 30d v1.21.0 10.0.0.3 <none> Ubuntu 20.04.2 LTS 5.4.0-72-generic docker://20.10.7
Verify Prerequisites
Ensure that you have the necessary permissions to make changes. You should have cluster-admin or at least write access to the configurations you plan to modify. Also confirm that etcd is healthy, as it is central to Kubernetes performance. On a control plane node with etcdctl installed, you can check health:
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health
Expected output indicates each endpoint is healthy:
https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.1ms
If any of these checks reveal issues, resolve them before proceeding with tuning.
Safe Configuration Path
Making changes directly to a running cluster can be risky. The safe approach is to make scoped, reversible changes one at a time, ideally in a test environment first. This section outlines common tuning areas and how to adjust them safely.
API Server Tuning
The API server is the entry point for all Kubernetes operations. Common issues include request latency under load and excessive memory usage. Key parameters to consider are --max-requests-inflight and --max-mutating-requests-inflight, which limit the number of simultaneous read and write requests, respectively. Increasing these can improve throughput if the API server has spare CPU and memory, but too high a value can overwhelm it.
To change these, you typically edit the API server manifest on the control plane node. For a kubeadm cluster, the file is usually /etc/kubernetes/manifests/kube-apiserver.yaml. Add or modify the spec for the container:
spec:
containers:
- command:
- kube-apiserver
- --max-requests-inflight=400
- --max-mutating-requests-inflight=200
After saving, the kubelet will restart the API server automatically. Verify the new process arguments with:
ps aux | grep kube-apiserver | grep max-requests-inflight
Expected output includes the new values.
Controller Manager and Scheduler Tuning
The controller manager and scheduler run as static pods. Their performance can be tuned by adjusting the --kube-api-qps and --kube-api-burst flags, which control the rate of API requests they make. Increasing these can speed up reconciliation but also increases load on the API server. For example, in the controller manager manifest (/etc/kubernetes/manifests/kube-controller-manager.yaml), you can add:
spec:
containers:
- command:
- kube-controller-manager
- --kube-api-qps=50
- --kube-api-burst=100
Similarly for the scheduler in /etc/kubernetes/manifests/kube-scheduler.yaml.
Kubelet Tuning
The kubelet on each node manages pod lifecycles. Tuning its --max-pods can help if you have many small pods. The default is 110 pods per node. If your nodes have ample resources, you can increase this. The setting is in the kubelet configuration file, typically /var/lib/kubelet/config.yaml:
maxPods: 110
Change to a higher value, like 250, and restart the kubelet:
systemctl restart kubelet
Check the kubelet logs to confirm the new value is loaded:
journalctl -u kubelet | grep maxPods
Kube Proxy Tuning
Kube proxy is responsible for network routing. Its mode (iptables or IPVS) has a big impact on performance. IPVS is generally more efficient for large clusters. To change from iptables to IPVS, you need to ensure the IPVS kernel modules are loaded, then edit the kube-proxy configuration. For kubeadm, the config is in a ConfigMap in the kube-system namespace. First, verify current mode:
kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode
Expected output: mode: "" (which defaults to iptables). To change to IPVS, edit the ConfigMap:
kubectl edit configmap kube-proxy -n kube-system
Set mode: "ipvs" under the appropriate section, save, and then restart the kube-proxy pods:
kubectl delete pods -n kube-system -l k8s-app=kube-proxy
The pods will be recreated with the new configuration.
Safe Implementation Practices
- Always make one change at a time, then observe before making another.
- Keep backups of original manifest files. For example:
cp /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml.bak
- Use a version control system for configuration changes.
- Test in a staging cluster if possible.
Verification and Diagnostics
After making changes, you need to verify that they are working as expected and not causing regressions. This section covers key metrics and checks.
API Server Latency and Throughput
You can use kubectl to measure API server response times. For example, time a list request:
time kubectl get pods --all-namespaces
Compare the time before and after tuning. A well-tuned API server should respond in well under a second for a small number of objects, but this depends on scale.
For more detailed metrics, if you have Prometheus set up, query for apiserver_request_duration_seconds percentiles. If not, you can check the API server logs for slow requests:
kubectl logs -n kube-system kube-apiserver-master-1 | grep 'apiserver_request_total' | tail
Scheduler Throughput
The scheduler processes pending pods. Check for pending pods that are not being scheduled quickly. Use kubectl get pods --all-namespaces --field-selector=status.phase=Pending to see if any are stuck. Also, look at the scheduler logs:
kubectl logs -n kube-system kube-scheduler-master-1 | tail
If you increased the scheduler's API QPS, you should see fewer delays in log timestamps between scheduling attempts.
Node Resource Pressure
Check node conditions for resource pressure, such as MemoryPressure or DiskPressure:
kubectl describe nodes | grep -A5 Conditions
Expected output shows MemoryPressure False and DiskPressure False. If any become True, the kubelet may evict pods, causing disruption.
Network Performance
If you switched to IPVS, verify that kube-proxy is using IPVS by checking the logs:
kubectl logs -n kube-system kube-proxy-xxxxx | grep 'Using ipvs Proxier'
You should see a line like Using ipvs Proxier. Additionally, you can check the IPVS rules on the node:
sudo ipvsadm -L -n | head
Expected output shows the service IPs and backends.
Etcd Health
Performance of Kubernetes heavily depends on etcd. Monitor etcd metrics such as etcd_disk_wal_fsync_duration_seconds and etcd_network_peer_round_trip_time_seconds. If you have etcdctl, you can check the endpoint status:
ETCDCTL_API=3 etcdctl endpoint status --cluster -w table
Example output:
+------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------------+------------------+---------+---------+-----------+-----------+------------+
| 10.0.0.1:2379 | 8e9e05c52164694d | 3.4.13 | 25 MB | true | 3 | 12345 |
+------------------+------------------+---------+---------+-----------+-----------+------------+
Deviations from normal (e.g., high fsync duration) indicate potential issues.
Failure Modes and Recovery
Even with careful planning, tuning changes can cause unexpected problems. This section describes common failure modes and how to recover.
API Server CrashLoopBackOff
If you increase --max-requests-inflight too high, the API server might run out of memory and crash. This can lead to a CrashLoopBackOff. Symptoms include the API server pod restarting repeatedly and the cluster becoming unresponsive to kubectl commands. To recover, revert the manifest to the original values. Since the API server is down, you need to access the control plane node directly and restore the backup:
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml
The kubelet will restart the API server with the previous configuration. After recovery, check the API server logs for out-of-memory errors and adjust the values more conservatively.
Scheduler Failing to Schedule Pods
If you change scheduler flags incorrectly, the scheduler may fail to start or crash. This results in new pods staying in Pending state indefinitely. To diagnose, check the scheduler pod status:
kubectl get pods -n kube-system -l component=kube-scheduler
If it is not running, inspect the logs (if possible) or the manifest file for syntax errors. Correct the manifest and save; the scheduler will be restarted. If the scheduler is running but still not scheduling, check its logs for errors like API connection failures and verify the --kube-api-qps and --kube-api-burst values are not set to zero.
Kubelet Not Starting After maxPods Change
If you set maxPods to a value that exceeds the node's capability (e.g., too high for the pod CIDR range), the kubelet may fail to start. The node will show as NotReady. Check the kubelet status:
systemctl status kubelet
If failed, examine the logs for the specific error:
journalctl -u kubelet -n 50 --no-pager
Common error: failed to reserve container for pod: invalid maxPods value. To recover, edit /var/lib/kubelet/config.yaml and set maxPods back to a valid value (e.g., 110), then restart kubelet and verify the node becomes Ready:
kubectl get nodes
Kube Proxy Issues After IPVS Change
If IPVS modules are not loaded, kube-proxy may fail to start, leaving services unreachable. Check the kube-proxy pod logs for errors like can't initialize ipvs module. To recover, either load the necessary kernel modules or revert the mode to iptables in the ConfigMap. For quick recovery, edit the ConfigMap back to mode: "" and restart kube-proxy:
kubectl edit configmap kube-proxy -n kube-system
kubectl delete pods -n kube-system -l k8s-app=kube-proxy
General Rollback Strategy
Always keep backups of any file or ConfigMap you modify. For files on nodes, use cp to create a .bak file. For Kubernetes objects, you can export the original YAML before modification:
kubectl get configmap kube-proxy -n kube-system -o yaml > kube-proxy-original.yaml
If a change causes issues, restore the original using kubectl apply -f kube-proxy-original.yaml or by copying the backup file back. Document any change in a change log with the date, what was changed, and why, to aid in troubleshooting.
Common Pitfalls and How to Avoid Them
Tuning Kubernetes can be tricky, and many operators fall into the same traps. Here are some frequent mistakes and how to steer clear of them.
Making Multiple Changes at Once
When you modify several parameters simultaneously, it becomes difficult to identify which change caused an improvement or a regression. For example, increasing API server inflight requests while also switching kube-proxy to IPVS could mask a problem with one change. To avoid this, make one change at a time, test thoroughly, and observe for at least a few hours before moving on.
Ignoring Resource Headroom
Setting limits too high without ensuring the control plane nodes have enough CPU and memory can lead to crashes. For instance, raising --max-requests-inflight to 1000 on a node with only 2 GB of RAM available to the API server will likely cause out-of-memory kills. Always check current resource usage with kubectl top nodes and kubectl top pods -n kube-system before adjusting limits, and use conservative increments.
Not Testing in a Staging Environment
Applying tuning directly to production is risky. A configuration that works in a small test cluster may fail under production load. For example, IPVS may work fine with 10 services but hit issues with 1000 services due to kernel limits. If possible, replicate your production topology in a staging cluster and run load tests before rolling out changes.
Skipping Backups
Failing to back up manifests and ConfigMaps means a bad change could leave you scrambling. Always create a backup file or export the original object before editing. For manifests, use cp file.yaml file.yaml.bak; for Kubernetes objects, use kubectl get -o yaml > original.yaml. Store backups in a version control system as well.
Not Monitoring After Changes
After a change, some operators immediately move on without verifying the impact. This can allow performance regressions to go unnoticed. For example, after increasing scheduler QPS, you should monitor scheduler latency and pending pod counts for at least a day. Set up alerts for key metrics like API server request duration and etcd fsync duration to catch issues early.
Operations Checklist
To keep your Kubernetes architecture performing well, follow this checklist on a regular basis. Assign an owner for each item and set a review frequency. The owner should be a single accountable individual, such as a platform engineer or cluster administrator.
| Item | Owner | Frequency |
|---|---|---|
| Verify cluster and component versions are consistent and supported | Platform Engineer (e.g., Priya Shah) | Monthly |
| Check etcd health and performance metrics (fsync duration, leader changes) | Database Administrator (e.g., Carlos Gomez) | Weekly |
| Monitor API server request duration and error rates; alert on high latency | SRE (e.g., Alex Chen) | Daily (automated alerts) |
| Review scheduler and controller manager logs for repeated errors or delays | Platform Engineer | Weekly |
| Check node conditions and resource usage (CPU, memory, disk) for pressure | SRE | Daily (automated alerts) |
| Validate network performance; if using IPVS, ensure rules are correct | Network Engineer (e.g., Maria Rossi) | Monthly |
| Test disaster recovery procedures for each component (e.g., restore API server manifest) | Platform Engineer | Quarterly |
| Keep configuration files under version control and maintain backups | Platform Engineer | Continuous |
| Schedule performance tests after any significant cluster change or upgrade | Performance Engineer (e.g., David Kim) | After each change |
| Document all tuning changes with rationale and observed effects | Platform Engineer | Continuous |
Owners should review their items at the specified frequency. For example, the Platform Engineer reviews the node conditions weekly and adjusts alerts as needed. This ensures accountability and keeps the cluster healthy.
Conclusion
Tuning Kubernetes architecture performance is a continuous process of measuring, adjusting, and verifying. By following the practical steps in this guide, you can systematically identify bottlenecks and apply safe, reversible changes. Remember to start with a thorough inventory, change one parameter at a time, and always have a rollback plan. The operations checklist will help you maintain optimal performance as your cluster evolves. With careful tuning, you can achieve lower latency, higher throughput, and a more reliable platform for your workloads.