## Intro

The Kubernetes API server is the front door to your cluster. Every kubectl command, controller, and component interaction flows through it. When it fails, the whole control plane can become unusable. Troubleshooting the API server requires a systematic approach: identify the version and topology, read the logs, check certificates and configuration, and verify connectivity. This guide provides practical examples and commands to diagnose and recover from common API server issues.

We will focus on scenarios most relevant to developers, DevOps engineers, and technical startup teams. The goal is operational safety: observe before changing, limit the blast radius, and verify the result. All examples use explicit placeholders and read-only commands first, so you can diagnose without disrupting production.

## Version and Environment Inventory

Before troubleshooting, gather the essential facts. Run these read-only commands to identify the Kubernetes version, API server deployment method, and node status.

### Identify Kubernetes Version and API Server Deployment

Use kubectl to get the server version:

```bash
kubectl version --client
```

Expected output includes both client and server versions. For example:

```text
Client Version: v1.28.2
Server Version: v1.28.2
```

If the server version is empty or the command times out, the API server may be unreachable. In that case, check the kubeconfig and network connectivity.

Determine how the API server is deployed. On a kubeadm cluster, it runs as a static pod managed by the kubelet. Check with:

```bash
kubectl get pods -n kube-system | grep kube-apiserver
```

On a managed cluster (EKS, GKE, AKS), you cannot access the API server pod directly; instead, you rely on cloud provider logs and metrics.

### Cluster Topology and Health Overview

Get the list of nodes and their status:

```bash
kubectl get nodes
```

Sample output:

```text
NAME           STATUS   ROLES           AGE   VERSION
control-plane  Ready    control-plane   10d   v1.28.2
worker-1       Ready    <none>          10d   v1.28.2
worker-2       NotReady <none>          10d   v1.28.2
```

If nodes are NotReady, the API server may still be running, but kubelet communication or CNI issues could cause symptoms that look like API server problems. Always check node status first.

Check the control plane component statuses (if accessible):

```bash
kubectl get componentstatuses
```

Note: This command is deprecated and may not show all components in newer versions. Instead, check the pods in kube-system:

```bash
kubectl get pods -n kube-system
```

Look for kube-apiserver, kube-controller-manager, kube-scheduler, and etcd. All should be Running and not in a CrashLoopBackOff.

### Prerequisites and Access

Ensure you have:
- kubectl installed and configured.
- Cluster admin permissions, or at least read access to kube-system namespace and node logs.
- SSH access to control plane nodes if the API server is not responding.

## Safe Configuration Path

When the API server is misbehaving, avoid making arbitrary changes. Instead, follow a safe, incremental path.

### Backup Current Configuration

Before modifying any API server manifest or configuration, back it up. On a control plane node, the static pod manifest is typically at /etc/kubernetes/manifests/kube-apiserver.yaml. Copy it:

```bash
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /root/kube-apiserver.yaml.backup-$(date +%Y%m%d)
```

Also save the current kubelet logs or API server logs for comparison:

```bash
sudo journalctl -u kubelet --since "1 hour ago" > /root/kubelet-before.log
```

### Read-Only Inspection

Check the API server pod logs without restarting anything:

```bash
kubectl logs -n kube-system kube-apiserver-control-plane --tail=100
```

If the pod is crash-looping, you may need to inspect previous logs:

```bash
kubectl logs -n kube-system kube-apiserver-control-plane --previous
```

Look for error lines such as certificate issues, etcd connection failures, or invalid flags.

### Smallest Justified Change

Common safe changes include:
- Correcting a typo in an admission webhook URL.
- Rotating an expired certificate.
- Adjusting resource limits.

Apply one change at a time. For example, if the API server is OOMKilled, increase its memory limit in the manifest:

```yaml
resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"  # increased from 512Mi
```

After editing, save the file. The kubelet will automatically restart the pod.

### Verification

Verify the pod restarts and becomes Running:

```bash
kubectl get pod -n kube-system kube-apiserver-control-plane
```

Then check logs for successful startup messages:

```bash
kubectl logs -n kube-system kube-apiserver-control-plane --tail=20
```

Expected output includes lines like:

```text
"Serving securely on [::]:6443"
```

If the pod fails again, revert to the backup.

## Verification and Diagnostics

This section covers key diagnostic commands and what they reveal.

### API Server Endpoint Health

Check the healthz endpoint. From the control plane node:

```bash
curl -k https://localhost:6443/healthz
```

Expected: `ok`

For a more detailed check, use:

```bash
curl -k https://localhost:6443/livez?verbose
```

This returns a list of checks and their status.

From outside the cluster, use kubectl to check:

```bash
kubectl get --raw='/readyz?verbose'
```

### Authentication and Authorization Issues

If you get `403 Forbidden` or `401 Unauthorized`, check the kubeconfig and user permissions.

View your current context:

```bash
kubectl config current-context
```

Check if your certificate is valid:

```bash
kubectl config view --raw -o jsonpath='{.users[0].user.client-certificate-data}' | base64 -d | openssl x509 -noout -dates
```

This shows the certificate's start and expiry dates.

Test access with a simple read:

```bash
kubectl auth can-i list pods --namespace=default
```

Expected: `yes` if authorized.

### Certificate Checks

API server certificates commonly cause failures. Check the serving certificate:

```bash
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
```

Ensure the certificate is not expired and that the SANs include the control plane IP, hostname, and load balancer DNS.

If using kubeadm, check certificate expiration:

```bash
kubeadm certs check-expiration
```

Sample output:

```text
CERTIFICATE                EXPIRES                  RESIDUAL TIME
apiserver                  Jan 01, 2025 12:00 UTC   364d
apiserver-etcd-client      Jan 01, 2025 12:00 UTC   364d
...
```

### Etcd Connectivity

The API server depends on etcd. Test connectivity from the control plane:

```bash
curl -k https://127.0.0.1:2379/health
```

Etcd may require client certificates. Use:

```bash
curl --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key https://127.0.0.1:2379/health
```

Expected: `{\"health\":\"true\"}`

If the API server logs show errors like \"etcdserver: request timed out\", investigate etcd cluster health.

## Failure Modes and Recovery

Understanding typical failure modes helps speed up recovery.

### API Server CrashLoopBackOff

Causes:
- Missing or invalid flags in the manifest.
- Misconfigured admission webhooks.
- Insufficient resources.
- Expired certificates.

Recovery steps:

1. Check logs:

```bash
kubectl logs -n kube-system kube-apiserver-control-plane --previous
```

2. Examine the manifest for errors:

```bash
sudo cat /etc/kubernetes/manifests/kube-apiserver.yaml
```

3. Validate YAML syntax:

```bash
sudo python3 -c 'import yaml, sys; print(yaml.safe_load(open(\"/etc/kubernetes/manifests/kube-apiserver.yaml\")))'
```

No output means YAML is valid (or adjust command).

4. Fix the issue. For example, if an admission webhook is unreachable, either remove it from the `--enable-admission-plugins` flag or ensure the webhook service is running.

5. The pod should restart automatically. Verify:

```bash
kubectl get pod -n kube-system kube-apiserver-control-plane
```

### API Server Not Responding

If kubectl commands hang, the API server may be down or unreachable.

Check if the process is running:

```bash
sudo crictl ps | grep kube-apiserver
```

Or

```bash
sudo docker ps | grep kube-apiserver
```

If no container, check kubelet status:

```bash
sudo systemctl status kubelet
```

Review kubelet logs:

```bash
sudo journalctl -u kubelet -n 100 --no-pager
```

If kubelet cannot pull the API server image, check network and image registry access.

### Certificate Expired

API server certificate expiration is a common cause of failure.

Renew certificates using kubeadm:

```bash
sudo kubeadm certs renew apiserver
```

Then restart the API server (or let kubelet pick up changes):

```bash
sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/
```

Or simply:

```bash
sudo kill -s SIGHUP $(pidof kube-apiserver)
```

Verify:

```bash
curl -k https://localhost:6443/healthz
```

###etcd Failure

If etcd is down, the API server cannot persist state. Check etcd cluster health:

```bash
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health
```

If etcd is unhealthy, refer to etcd troubleshooting guides. Then restart etcd if necessary.

## Operations Checklist

Use this checklist to methodically work through API server issues. Each item includes a command or action and expected result.

| Step | Action | Command / Check | Expected Result |
|------|--------|-----------------|-----------------|
| 1 | Verify cluster version | `kubectl version --client` | Server version known |
| 2 | Check API server pod status | `kubectl get pods -n kube-system` | Pod Running, not CrashLoop |
| 3 | Read logs | `kubectl logs -n kube-system kube-apiserver-<node>` | No fatal errors |
| 4 | Test healthz | `curl -k https://localhost:6443/healthz` | Returns `ok` |
| 5 | Validate certificates | `kubeadm certs check-expiration` | Not expired |
| 6 | Check etcd connectivity | `curl --cacert ... https://127.0.0.1:2379/health` | `{\"health\":\"true\"}` |
| 7 | Verify node status | `kubectl get nodes` | All nodes Ready |
| 8 | Confirm user authorization | `kubectl auth can-i list pods` | `yes` |
| 9 | Inspect manifest | `cat /etc/kubernetes/manifests/kube-apiserver.yaml` | Valid flags and config |
| 10 | Backup before changes | `cp kube-apiserver.yaml backup` | Backup exists |

Update this checklist after each incident to include checks for newly discovered failure modes. Assign a single owner (e.g., the on-call SRE) to maintain it and review it monthly.

## Common Pitfalls and Mistakes

### 1. Modifying the Manifest Without Backup

Why it happens: Pressure to fix quickly leads to direct edits.

How to avoid: Always copy the manifest to a timestamped backup before editing. If the API server fails to start, restore immediately.

### 2. Using kubectl to Diagnose When API Server Is Down

Why it happens: Habit or lack of awareness of node-level tools.

How to avoid: If kubectl is unresponsive, switch to node-level tools: crictl, docker, journalctl, and direct curl commands. Use SSH to access the control plane.

### 3. Ignoring Certificate Expiry

Why it happens: Certificates expire infrequently, so teams forget to monitor.

How to avoid: Set up monitoring alerts for certificate expiration (e.g., using Prometheus and a cert-exporter). Run `kubeadm certs check-expiration` regularly.

### 4. Not Checking Etcd Health

Why it happens: API server errors often point to itself, but root cause may be etcd.

How to avoid: Always include etcd health checks in your diagnostic routine. If etcd is down, API server cannot function.

### 5. Applying Multiple Changes at Once

Why it happens: Attempting to fix several suspected issues simultaneously.

How to avoid: Make one change at a time and verify. If the change fails, revert. This isolates the cause.

### 6. Overlooking Admission Webhooks

Why it happens: Misconfigured webhooks can block API requests, but logs may not explicitly say so.

How to avoid: Check API server logs for webhook timeout or connection refused errors. Temporarily disable suspicious webhooks to test.

## Conclusion

Troubleshooting the Kubernetes API server requires a structured approach. Start with read-only observations, gather version and topology details, inspect logs, and verify certificates and etcd connectivity. When a change is needed, make the smallest possible adjustment, back up configuration, and verify the result. The failure modes and checklist in this article provide a practical path to restore service quickly.

By following these steps and avoiding common pitfalls, you can reduce downtime and maintain a healthy control plane. Regularly update your runbooks and monitor for certificate expiration and etcd health to prevent issues before they occur.