## Intro
The Kube API Server is the front door to a Kubernetes cluster. When it fails, kubectl commands hang, controllers stop reconciling, and workloads become unmanageable. This guide collects the most common Kube API Server errors, their causes, and concrete fixes with commands, expected outputs, and recovery steps.
It is written for developers, DevOps consultants, and technical startup teams who operate Kubernetes clusters running versions 1.26 through 1.30. The focus is on kube-apiserver itself; related control plane components like kubelet, kube-controller-manager, and RBAC are mentioned only when they affect the API server's prerequisites, security, or observability.
The workflow follows operational safety principles:
- Observe before changing.
- Capture the current state with read-only commands.
- Use placeholders instead of real credentials, tokens, or private keys in examples.
- Limit changes to one scoped item at a time.
- Verify the result after every intervention.
- Have a tested recovery path before making a change.
Each section names the relevant component, supported versions, prerequisites, a read-only observation, the smallest justified change, and the verification signal.
## Version and Environment Inventory
Before troubleshooting, establish the exact version and topology. The kube-apiserver binary is usually deployed as a static Pod managed by the kubelet on the control plane nodes, or as a systemd service in non-containerized setups.
### Identify version and deployment
Read-only observation
kubectl version --short
Expected output on a working cluster:
Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.3
If the API server is down, kubectl cannot connect. In that case, check the kubectl client version only:
kubectl version --client
Or inspect the binary directly on the control plane node:
/usr/local/bin/kube-apiserver --version
Expected output:
Kubernetes v1.28.3
To determine the deployment type, check if the kube-apiserver Pod is visible (this requires a working API server):
kubectl get pods -n kube-system | grep kube-apiserver
If the API server is down, search for the static Pod manifest file:
ls -la /etc/kubernetes/manifests/kube-apiserver.yaml
If this file exists, the kubelet manages the kube-apiserver as a static Pod. Otherwise, look for a systemd unit:
systemctl status kube-apiserver
Prerequisites
- Access to a control plane node via SSH or console.
- Read permission to /etc/kubernetes/manifests/ or systemd unit files.
- kubectl configured for the cluster (when the API server is up).
Blast radius
These commands are read-only and safe to run.
### Verify API server health
Once you know the version and deployment, check the health endpoint.
Read-only observation
From a control plane node, run:
curl -k https://localhost:6443/healthz?verbose
Expected output when healthy:
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/priority-and-fairness-config-consumer ok
[+]poststarthook/priority-and-fairness-filter ok
[+]poststarthook/storage-object-count-tracker-hook ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/start-kube-apiserver-identity-lease-controller ok
[+]poststarthook/start-deprecated-alpha-api-gateway-trickster ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
healthz check passed
If any check fails, note the failing component; it often points directly to the root cause (e.g., etcd failure means the API server cannot reach its backend database).
Smallest justified change
No change yet. Record the current state and move to diagnostics.
## Safe Configuration Path
Most kube-apiserver misconfigurations come from edits to its manifest file ( /etc/kubernetes/manifests/kube-apiserver.yaml ) or systemd unit ( /etc/systemd/system/kube-apiserver.service ). Follow this safe path to avoid making things worse.
### Before editing: capture baseline
Read-only observation
- Copy the current manifest or unit file to a timestamped backup:
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /var/tmp/kube-apiserver.yaml.$(date +%Y%m%d%H%M%S)
- Record the current Pod status and recent logs:
kubectl get pods -n kube-system -l component=kube-apiserver -o wide
kubectl logs -n kube-system -l component=kube-apiserver --tail=50
If the API server is down, check the kubelet logs or systemd journal:
journalctl -u kubelet -n 50 --no-pager
or
journalctl -u kube-apiserver -n 50 --no-pager
Prerequisites
- Write access to a safe backup location (e.g., /var/tmp).
- Sudo privileges on the control plane node.
Blast radius
Backup and log capture are read-only and safe.
### Edit configuration safely
Example: change the advertise address
Suppose the API server advertises the wrong IP because the node's IP changed. You need to update --advertise-address .
- Open the manifest with a text editor:
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml
- Find the container args and modify:
spec:
containers:
- command:
- kube-apiserver
- --advertise-address=192.168.1.10 # change to new IP
- --etcd-servers=https://127.0.0.1:2379
...
- Save and exit. The kubelet will detect the change and restart the kube-apiserver automatically within a few seconds.
Verification
Check the API server endpoint and health:
kubectl get --raw /healthz
Expected output:
ok
And verify the advertised address:
kubectl get endpoints kubernetes -n default -o yaml
Expected snippet:
subsets:
- addresses:
- ip: 192.168.1.10
ports:
- name: https
port: 6443
protocol: TCP
Recovery path
If the API server fails to start after the change, restore the backup:
sudo cp /var/tmp/kube-apiserver.yaml. /etc/kubernetes/manifests/kube-apiserver.yaml
The kubelet will restart the API server with the old configuration. Then investigate the root cause.
## Verification and Diagnostics
When you suspect a problem, run a systematic set of diagnostics. This section covers common checks and what their outputs mean.
### Check API server logs
Read-only observation
On a static Pod deployment:
kubectl logs -n kube-system -l component=kube-apiserver --tail=100
On systemd:
journalctl -u kube-apiserver -n 100 --no-pager
Look for repeated errors such as:
- etcdserver: request timed out – connectivity issue to etcd.
- x509: certificate signed by unknown authority – certificate trust problem.
- Unauthorized – authentication or RBAC issue.
- the server is currently unable to handle the request – API server overloaded.
### Check API server metrics
The API server exposes Prometheus metrics on port 6443 (or a separate metrics port if configured).
Read-only observation
curl -k https://localhost:6443/metrics | grep apiserver_request_total | head
Example output:
apiserver_request_total{client="kubelet", code="200", content_type="application/vnd.kubernetes.protobuf", dry_run="", group="", resource="pods", scope="cluster", subresource="", verb="GET", version="v1"} 1200
Key metrics to watch:
- apiserver_request_duration_seconds_bucket – high latency indicates overload.
- apiserver_request_total – error rates per verb/resource.
- etcd_request_duration_seconds_bucket – etcd latency, which often impacts API server performance.
### Check API server performance
If requests are slow, check the API priority and fairness settings and admission webhooks.
Read-only observation
List admission webhooks:
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
If any webhook has a high timeoutSeconds or is unreachable, it can block requests. Check webhook logs or test connectivity.
Query priority levels:
kubectl get prioritylevelconfigurations
If a priority level is saturated, you may need to adjust its assuredConcurrencyShares .
Smallest justified change
If a webhook is misbehaving, temporarily disable it by setting failurePolicy: Ignore , but do not delete it. Then verify performance improves.
Verification
After disabling, run a simple kubectl command and measure time:
time kubectl get pods
Expected: command returns within a second (unless cluster is huge).
Recovery
Re-enable the webhook by setting failurePolicy: Fail and fixing the underlying service.
## Failure Modes and Recovery
This section lists concrete failure scenarios with symptoms, diagnosis, and recovery steps.
### Scenario 1: API server cannot start due to invalid flag
Symptom
kubectl shows The connection to the server 192.168.1.10:6443 was refused - did you specify the right host or port?
Diagnosis
Check kubelet logs or systemd journal for the API server container crash:
journalctl -u kubelet -n 100 | grep kube-apiserver
Example error:
Error: unknown flag: --advertise-addresss
Cause
Typo in flag name or invalid value.
Recovery
- Restore the previous manifest from backup.
- Correct the flag.
- Verify with kubectl get --raw /healthz .
### Scenario 2: API server cannot reach etcd
Symptom
API server starts but logs repeated errors:
etcdserver: request timed out
Diagnosis
Check etcd health from the control plane node:
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health
Expected if healthy:
https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.345678ms
If unhealthy, check etcd logs and network connectivity.
Recovery
- Ensure etcd service is running.
- Check firewall rules between API server and etcd (default port 2379).
- Check certificates validity: openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -dates .
- If etcd is down, restore from backup or restart etcd.
### Scenario 3: Certificate errors on client requests
Symptom
kubectl commands fail with:
Unable to connect to the server: x509: certificate signed by unknown authority
Diagnosis
Check the certificate used by the API server:
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -issuer -dates
Expected:
subject=CN=kube-apiserver
issuer=CN=kubernetes
notBefore=...
notAfter=...
Check client kubeconfig certificate info:
kubectl config view --raw
Cause
- Client certificate expired or not trusted.
- API server certificate expired or not signed by the cluster CA.
Recovery
- Renew API server certificates using kubeadm: kubeadm certs renew apiserver
- For client certificate, regenerate kubeconfig or use a valid certificate.
- If using a custom CA, ensure the client kubeconfig has the correct certificate-authority-data .
### Scenario 4: API server denies requests due to RBAC
Symptom
A user or workload gets:
Error from server (Forbidden): pods is forbidden: User "system:serviceaccount:default:my-sa" cannot list resource "pods" in API group "" in the namespace "default"
Diagnosis
Check the roles and role bindings for the user/service account:
kubectl auth can-i list pods --as=system:serviceaccount:default:my-sa -n default
Expected: no
Cause
Missing RoleBinding or ClusterRoleBinding.
Recovery
Create the necessary RBAC resources, e.g., a Role that allows listing pods:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
namespace: default
name: pod-reader
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: read-pods
namespace: default
subjects:
- kind: ServiceAccount
name: my-sa
namespace: default
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
Apply and verify:
kubectl auth can-i list pods --as=system:serviceaccount:default:my-sa -n default
Expected: yes
## Operations Checklist
Use this checklist for routine maintenance and incident response.
### Pre-change checklist
- [ ] Record current version: kubectl version --short
- [ ] Check health endpoint: curl -k https://localhost:6443/healthz?verbose
- [ ] Backup manifest or unit file: sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /var/tmp/backup-$(date +%s).yaml
- [ ] Note current API server Pod status: kubectl get pods -n kube-system -l component=kube-apiserver
- [ ] Identify change scope and expected impact.
### During change
- [ ] Edit only one configuration item at a time.
- [ ] Use version control for manifest changes if possible (e.g., GitOps).
- [ ] Save a copy of the new file before restarting.
### Post-change verification
- [ ] Check health: kubectl get --raw /healthz returns ok .
- [ ] Verify core resources: kubectl get nodes returns node list.
- [ ] Check logs for new errors: kubectl logs -n kube-system -l component=kube-apiserver --tail=50
- [ ] If failure occurs, restore backup and document the incident.
### Recovery checklist
- [ ] Stop the change, restore previous configuration.
- [ ] Verify API server is back to healthy.
- [ ] Investigate root cause using logs and metrics before reapplying.
- [ ] Test the change in a staging cluster if possible.
## Conclusion
Kube API Server troubleshooting requires a methodical approach: identify the environment, observe symptoms, diagnose with focused commands, apply a minimal fix, and verify recovery. This guide provided concrete commands for common error scenarios, from configuration typos to etcd connectivity and certificate issues.
As a next step, choose one low-risk verification from the Operations Checklist, run it against your cluster, and record the expected vs. actual output. That practice will build muscle memory for handling the next kube-apiserver incident. Remember: never change more than one thing at a time, always have a backup, and verify before declaring victory.