E-NO
Kubernetes Architecture performance 11 Min Read

Kubernetes Architecture Performance Tuning with Practical Examples

calendar_today Published: 2026-11-17
update Last Updated: 2026-11-17
analytics SEO Efficiency: 100%
Technical guide illustration for Kubernetes Architecture Performance Tuning with Practical Examples.

Intro

Kubernetes has become the de facto standard for container orchestration, but as clusters grow and workloads become more demanding, performance bottlenecks can emerge. These bottlenecks often manifest as slow API responses, scheduling delays, network latency, or resource contention. Tuning the Kubernetes architecture itself - not just the applications running on it - is essential to maintain reliability and efficiency.

This article provides a practical guide to identifying and resolving common Kubernetes performance issues. We will cover how to inventory your cluster, make safe configuration changes, verify the impact, and recover if something goes wrong. The goal is to help you move from reactive troubleshooting to proactive optimization with concrete commands and examples.

Version and Environment Inventory

Before making any changes, you need to establish a clear picture of your Kubernetes environment. This includes the exact versions of the control plane components and nodes, the cluster topology, and the prerequisites for tuning.

Check Cluster Version and Component Versions

Start by checking the version of the Kubernetes control plane and nodes. Use kubectl version to see both client and server versions:

kubectl version --short

Expected output should show the server version, for example:

Client Version: v1.21.0
Server Version: v1.21.0

If you have access to the control plane node, you can also check the individual component versions by examining the running processes or their container images. For example, for the API server:

kubectl get pods -n kube-system -l component=kube-apiserver -o jsonpath='{.items[0].spec.containers[0].image}'

This will output the image tag, such as:

k8s.gcr.io/kube-apiserver:v1.21.0

Understand Topology

Document the topology: how many master nodes and worker nodes, their roles, and any high availability configuration. Use kubectl get nodes with the -o wide flag to see IP addresses, OS, and container runtime versions:

kubectl get nodes -o wide

Example output:

NAME       STATUS   ROLES                  AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master-1   Ready    control-plane,master   30d   v1.21.0   10.0.0.1      <none>        Ubuntu 20.04.2 LTS   5.4.0-72-generic   docker://20.10.7
worker-1   Ready    <none>                 30d   v1.21.0   10.0.0.2      <none>        Ubuntu 20.04.2 LTS   5.4.0-72-generic   docker://20.10.7
worker-2   Ready    <none>                 30d   v1.21.0   10.0.0.3      <none>        Ubuntu 20.04.2 LTS   5.4.0-72-generic   docker://20.10.7

Verify Prerequisites

Ensure that you have the necessary permissions to make changes. You should have cluster-admin or at least write access to the configurations you plan to modify. Also confirm that etcd is healthy, as it is central to Kubernetes performance. On a control plane node with etcdctl installed, you can check health:

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

Expected output indicates each endpoint is healthy:

https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.1ms

If any of these checks reveal issues, resolve them before proceeding with tuning.

Quick check 1 of 2

What are two key parameters for tuning the API server?

The article states that key parameters for API server tuning are --max-requests-inflight and --max-mutating-requests-inflight, which limit simultaneous read and write requests.

Safe Configuration Path

Making changes directly to a running cluster can be risky. The safe approach is to make scoped, reversible changes one at a time, ideally in a test environment first. This section outlines common tuning areas and how to adjust them safely.

API Server Tuning

The API server is the entry point for all Kubernetes operations. Common issues include request latency under load and excessive memory usage. Key parameters to consider are --max-requests-inflight and --max-mutating-requests-inflight, which limit the number of simultaneous read and write requests, respectively. Increasing these can improve throughput if the API server has spare CPU and memory, but too high a value can overwhelm it.

To change these, you typically edit the API server manifest on the control plane node. For a kubeadm cluster, the file is usually /etc/kubernetes/manifests/kube-apiserver.yaml. Add or modify the spec for the container:

spec:
  containers:
  - command:
    - kube-apiserver
    - --max-requests-inflight=400
    - --max-mutating-requests-inflight=200

After saving, the kubelet will restart the API server automatically. Verify the new process arguments with:

ps aux | grep kube-apiserver | grep max-requests-inflight

Expected output includes the new values.

Controller Manager and Scheduler Tuning

The controller manager and scheduler run as static pods. Their performance can be tuned by adjusting the --kube-api-qps and --kube-api-burst flags, which control the rate of API requests they make. Increasing these can speed up reconciliation but also increases load on the API server. For example, in the controller manager manifest (/etc/kubernetes/manifests/kube-controller-manager.yaml), you can add:

spec:
  containers:
  - command:
    - kube-controller-manager
    - --kube-api-qps=50
    - --kube-api-burst=100

Similarly for the scheduler in /etc/kubernetes/manifests/kube-scheduler.yaml.

Kubelet Tuning

The kubelet on each node manages pod lifecycles. Tuning its --max-pods can help if you have many small pods. The default is 110 pods per node. If your nodes have ample resources, you can increase this. The setting is in the kubelet configuration file, typically /var/lib/kubelet/config.yaml:

maxPods: 110

Change to a higher value, like 250, and restart the kubelet:

systemctl restart kubelet

Check the kubelet logs to confirm the new value is loaded:

journalctl -u kubelet | grep maxPods

Kube Proxy Tuning

Kube proxy is responsible for network routing. Its mode (iptables or IPVS) has a big impact on performance. IPVS is generally more efficient for large clusters. To change from iptables to IPVS, you need to ensure the IPVS kernel modules are loaded, then edit the kube-proxy configuration. For kubeadm, the config is in a ConfigMap in the kube-system namespace. First, verify current mode:

kubectl get configmap kube-proxy -n kube-system -o yaml | grep mode

Expected output: mode: "" (which defaults to iptables). To change to IPVS, edit the ConfigMap:

kubectl edit configmap kube-proxy -n kube-system

Set mode: "ipvs" under the appropriate section, save, and then restart the kube-proxy pods:

kubectl delete pods -n kube-system -l k8s-app=kube-proxy

The pods will be recreated with the new configuration.

Safe Implementation Practices

  • Always make one change at a time, then observe before making another.
  • Keep backups of original manifest files. For example:
cp /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml.bak
  • Use a version control system for configuration changes.
  • Test in a staging cluster if possible.

Verification and Diagnostics

After making changes, you need to verify that they are working as expected and not causing regressions. This section covers key metrics and checks.

API Server Latency and Throughput

You can use kubectl to measure API server response times. For example, time a list request:

time kubectl get pods --all-namespaces

Compare the time before and after tuning. A well-tuned API server should respond in well under a second for a small number of objects, but this depends on scale.

For more detailed metrics, if you have Prometheus set up, query for apiserver_request_duration_seconds percentiles. If not, you can check the API server logs for slow requests:

kubectl logs -n kube-system kube-apiserver-master-1 | grep 'apiserver_request_total' | tail

Scheduler Throughput

The scheduler processes pending pods. Check for pending pods that are not being scheduled quickly. Use kubectl get pods --all-namespaces --field-selector=status.phase=Pending to see if any are stuck. Also, look at the scheduler logs:

kubectl logs -n kube-system kube-scheduler-master-1 | tail

If you increased the scheduler's API QPS, you should see fewer delays in log timestamps between scheduling attempts.

Node Resource Pressure

Check node conditions for resource pressure, such as MemoryPressure or DiskPressure:

kubectl describe nodes | grep -A5 Conditions

Expected output shows MemoryPressure False and DiskPressure False. If any become True, the kubelet may evict pods, causing disruption.

Network Performance

If you switched to IPVS, verify that kube-proxy is using IPVS by checking the logs:

kubectl logs -n kube-system kube-proxy-xxxxx | grep 'Using ipvs Proxier'

You should see a line like Using ipvs Proxier. Additionally, you can check the IPVS rules on the node:

sudo ipvsadm -L -n | head

Expected output shows the service IPs and backends.

Etcd Health

Performance of Kubernetes heavily depends on etcd. Monitor etcd metrics such as etcd_disk_wal_fsync_duration_seconds and etcd_network_peer_round_trip_time_seconds. If you have etcdctl, you can check the endpoint status:

ETCDCTL_API=3 etcdctl endpoint status --cluster -w table

Example output:

+------------------+------------------+---------+---------+-----------+-----------+------------+
| ENDPOINT         | ID               | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
+------------------+------------------+---------+---------+-----------+-----------+------------+
| 10.0.0.1:2379   | 8e9e05c52164694d | 3.4.13  | 25 MB   | true      | 3         | 12345      |
+------------------+------------------+---------+---------+-----------+-----------+------------+

Deviations from normal (e.g., high fsync duration) indicate potential issues.

Failure Modes and Recovery

Even with careful planning, tuning changes can cause unexpected problems. This section describes common failure modes and how to recover.

API Server CrashLoopBackOff

If you increase --max-requests-inflight too high, the API server might run out of memory and crash. This can lead to a CrashLoopBackOff. Symptoms include the API server pod restarting repeatedly and the cluster becoming unresponsive to kubectl commands. To recover, revert the manifest to the original values. Since the API server is down, you need to access the control plane node directly and restore the backup:

sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml

The kubelet will restart the API server with the previous configuration. After recovery, check the API server logs for out-of-memory errors and adjust the values more conservatively.

Scheduler Failing to Schedule Pods

If you change scheduler flags incorrectly, the scheduler may fail to start or crash. This results in new pods staying in Pending state indefinitely. To diagnose, check the scheduler pod status:

kubectl get pods -n kube-system -l component=kube-scheduler

If it is not running, inspect the logs (if possible) or the manifest file for syntax errors. Correct the manifest and save; the scheduler will be restarted. If the scheduler is running but still not scheduling, check its logs for errors like API connection failures and verify the --kube-api-qps and --kube-api-burst values are not set to zero.

Kubelet Not Starting After maxPods Change

If you set maxPods to a value that exceeds the node's capability (e.g., too high for the pod CIDR range), the kubelet may fail to start. The node will show as NotReady. Check the kubelet status:

systemctl status kubelet

If failed, examine the logs for the specific error:

journalctl -u kubelet -n 50 --no-pager

Common error: failed to reserve container for pod: invalid maxPods value. To recover, edit /var/lib/kubelet/config.yaml and set maxPods back to a valid value (e.g., 110), then restart kubelet and verify the node becomes Ready:

kubectl get nodes

Kube Proxy Issues After IPVS Change

If IPVS modules are not loaded, kube-proxy may fail to start, leaving services unreachable. Check the kube-proxy pod logs for errors like can't initialize ipvs module. To recover, either load the necessary kernel modules or revert the mode to iptables in the ConfigMap. For quick recovery, edit the ConfigMap back to mode: "" and restart kube-proxy:

kubectl edit configmap kube-proxy -n kube-system
kubectl delete pods -n kube-system -l k8s-app=kube-proxy

General Rollback Strategy

Always keep backups of any file or ConfigMap you modify. For files on nodes, use cp to create a .bak file. For Kubernetes objects, you can export the original YAML before modification:

kubectl get configmap kube-proxy -n kube-system -o yaml > kube-proxy-original.yaml

If a change causes issues, restore the original using kubectl apply -f kube-proxy-original.yaml or by copying the backup file back. Document any change in a change log with the date, what was changed, and why, to aid in troubleshooting.

Quick check 2 of 2

According to the article, what is the default value for maxPods in the kubelet configuration?

The article mentions that the default for --max-pods is 110 pods per node.

Common Pitfalls and How to Avoid Them

Tuning Kubernetes can be tricky, and many operators fall into the same traps. Here are some frequent mistakes and how to steer clear of them.

Making Multiple Changes at Once

When you modify several parameters simultaneously, it becomes difficult to identify which change caused an improvement or a regression. For example, increasing API server inflight requests while also switching kube-proxy to IPVS could mask a problem with one change. To avoid this, make one change at a time, test thoroughly, and observe for at least a few hours before moving on.

Ignoring Resource Headroom

Setting limits too high without ensuring the control plane nodes have enough CPU and memory can lead to crashes. For instance, raising --max-requests-inflight to 1000 on a node with only 2 GB of RAM available to the API server will likely cause out-of-memory kills. Always check current resource usage with kubectl top nodes and kubectl top pods -n kube-system before adjusting limits, and use conservative increments.

Not Testing in a Staging Environment

Applying tuning directly to production is risky. A configuration that works in a small test cluster may fail under production load. For example, IPVS may work fine with 10 services but hit issues with 1000 services due to kernel limits. If possible, replicate your production topology in a staging cluster and run load tests before rolling out changes.

Skipping Backups

Failing to back up manifests and ConfigMaps means a bad change could leave you scrambling. Always create a backup file or export the original object before editing. For manifests, use cp file.yaml file.yaml.bak; for Kubernetes objects, use kubectl get -o yaml > original.yaml. Store backups in a version control system as well.

Not Monitoring After Changes

After a change, some operators immediately move on without verifying the impact. This can allow performance regressions to go unnoticed. For example, after increasing scheduler QPS, you should monitor scheduler latency and pending pod counts for at least a day. Set up alerts for key metrics like API server request duration and etcd fsync duration to catch issues early.

Operations Checklist

To keep your Kubernetes architecture performing well, follow this checklist on a regular basis. Assign an owner for each item and set a review frequency. The owner should be a single accountable individual, such as a platform engineer or cluster administrator.

ItemOwnerFrequency
Verify cluster and component versions are consistent and supportedPlatform Engineer (e.g., Priya Shah)Monthly
Check etcd health and performance metrics (fsync duration, leader changes)Database Administrator (e.g., Carlos Gomez)Weekly
Monitor API server request duration and error rates; alert on high latencySRE (e.g., Alex Chen)Daily (automated alerts)
Review scheduler and controller manager logs for repeated errors or delaysPlatform EngineerWeekly
Check node conditions and resource usage (CPU, memory, disk) for pressureSREDaily (automated alerts)
Validate network performance; if using IPVS, ensure rules are correctNetwork Engineer (e.g., Maria Rossi)Monthly
Test disaster recovery procedures for each component (e.g., restore API server manifest)Platform EngineerQuarterly
Keep configuration files under version control and maintain backupsPlatform EngineerContinuous
Schedule performance tests after any significant cluster change or upgradePerformance Engineer (e.g., David Kim)After each change
Document all tuning changes with rationale and observed effectsPlatform EngineerContinuous

Owners should review their items at the specified frequency. For example, the Platform Engineer reviews the node conditions weekly and adjusts alerts as needed. This ensures accountability and keeps the cluster healthy.

Conclusion

Tuning Kubernetes architecture performance is a continuous process of measuring, adjusting, and verifying. By following the practical steps in this guide, you can systematically identify bottlenecks and apply safe, reversible changes. Remember to start with a thorough inventory, change one parameter at a time, and always have a rollback plan. The operations checklist will help you maintain optimal performance as your cluster evolves. With careful tuning, you can achieve lower latency, higher throughput, and a more reliable platform for your workloads.

Related Research

Article Quality Score

Reader usefulness 100%
  • check_circle Reader-ready guide
  • check_circle Practical examples included
  • check_circle Clean SEO article URL