## Intro

The Kubernetes API server is the front door to your cluster. Every kubectl command, controller action, and scheduler decision flows through it. When it fails or behaves unexpectedly, the whole control plane can grind to a halt. Knowing the right commands to inspect it is not optional for anyone who operates Kubernetes in production.

This article gives you a practical command reference for the kube-apiserver. You will learn how to check its health, inspect its configuration, view its logs, test authentication and authorization, and recover from common failure modes. Every command includes an example with expected output where useful, plus notes on prerequisites and blast radius. We will also cover common mistakes that operators make when working with the API server and how to avoid them.

We focus on commands that are safe to run first: read-only observations that tell you what is happening without changing anything. Then we move to controlled changes with verification steps and recovery paths. Whether you are a developer debugging access issues, a DevOps engineer responding to an incident, or a platform team documenting runbooks, these examples will help you work more confidently.

## Version and Environment Inventory

Before running any command, know your cluster version and how the API server is deployed. Different Kubernetes distributions and versions can change command paths, flags, and endpoints. Start with a few read-only commands to establish context.

### Check Cluster Version

Use kubectl to get both client and server versions.

```bash
kubectl version --short
```

Expected output:

```
Client Version: v1.28.2
Server Version: v1.28.2
```

If the server version does not appear, your kubeconfig may not have valid credentials or the API server is unreachable.

### Identify API Server Pod

In a kubeadm or self-managed cluster, the kube-apiserver runs as a static pod on control plane nodes. List control plane pods in the kube-system namespace.

```bash
kubectl get pods -n kube-system -l component=kube-apiserver
```

Expected output:

```
NAME                    READY   STATUS    RESTARTS   AGE
kube-apiserver-node1   1/1     Running   0          4d
```

In managed clusters like EKS, AKS, or GKE, you will not see the API server pod because the control plane is managed by the provider. You interact with the API server via the managed endpoint.

### Check API Server Endpoints

The Kubernetes service in the default namespace points to the API server.

```bash
kubectl get endpoints kubernetes
```

Expected output:

```
NAME         ENDPOINTS             AGE
kubernetes   192.168.1.10:6443     10d
```

That IP and port are where kubectl sends requests. Verify your kubeconfig points to the same address with `kubectl config view`.

### View API Server Container Arguments

If you can access the pod, inspect its command line arguments. These flags define how the API server behaves: authentication, authorization, admission control, and more.

```bash
kubectl get pod kube-apiserver-node1 -n kube-system -o jsonpath='{.spec.containers[0].command}' | tr ' ' '\n'
```

You will see flags like `--authorization-mode=Node,RBAC`, `--enable-admission-plugins=...`, and `--tls-cert-file=...`. Save this output for troubleshooting later; do not share it publicly because paths may reveal internal naming.

## Safe Configuration Path

Changing API server configuration is delicate. A bad flag or malformed certificate can break the entire control plane. Always follow this sequence:

1. Observe current state.
2. Back up the manifest or config file.
3. Make one small change.
4. Verify the API server restarts successfully.
5. Have a rollback plan ready.

### Locate the Static Pod Manifest

In kubeadm clusters, the API server manifest is at `/etc/kubernetes/manifests/kube-apiserver.yaml` on the control plane node. The kubelet watches this directory and automatically restarts the pod when the file changes.

```bash
sudo ls -l /etc/kubernetes/manifests/kube-apiserver.yaml
```

Expected output:

```
-rw------- 1 root root 2438 Jan 12 10:22 /etc/kubernetes/manifests/kube-apiserver.yaml
```

### Back Up the Manifest

Before editing, create a timestamped backup.

```bash
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml.bak.$(date +%Y%m%d%H%M%S)
```

### Edit the Manifest Safely

Use a non-destructive editor and save carefully. For example, to add an audit log path, you would add `--audit-log-path=/var/log/kube-audit.log` to the command list. But first check if the log directory exists and is writable by the API server container.

```bash
sudo mkdir -p /var/log/kube-audit
sudo chown root:root /var/log/kube-audit
```

Then edit the manifest and add the flag with the appropriate volume mount. After saving, the kubelet will attempt to restart the API server pod. Watch the pod status.

```bash
kubectl get pods -n kube-system -l component=kube-apiserver -w
```

If the pod fails to start, check logs (see below) and revert to the backup manifest.

## Verification and Diagnostics

Once the API server is running, you need to verify it is healthy and serving requests correctly. These commands help you diagnose issues without making changes.

### Health Endpoints

The API server exposes health check endpoints for liveness and readiness.

```bash
curl -sk https://<api-server-ip>:6443/healthz?verbose
```

On a control plane node, you can use localhost.

```bash
curl -sk https://localhost:6443/healthz?verbose
```

Expected output when healthy:

```
[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
...
healthz check passed
```

If any check fails, the output shows the failing component. That points you to the next area to investigate (e.g., etcd connectivity).

### Check API Server Metrics

The API server exposes Prometheus metrics at `/metrics`. You can fetch them with curl if you have the right client certificate, or use kubectl proxy for convenience.

```bash
kubectl proxy --port=8001 &
curl -s http://localhost:8001/metrics | grep -E '^apiserver_request_total|^apiserver_current_inflight_requests'
```

Example line:

```
apiserver_current_inflight_requests{requestKind="readOnly",resource="pods"} 1
```

High inflight requests may indicate slow backend or overloaded API server.

### View API Server Logs

For static pod clusters, use kubectl logs.

```bash
kubectl logs -n kube-system kube-apiserver-node1
```

For non-static pods or systemd-managed API server, use journalctl.

```bash
journalctl -u kube-apiserver -f
```

Look for repeated error messages, TLS handshake failures, or etcd timeouts. Timestamp and correlation with other control plane components (etcd, controller manager) is key.

### Test Authentication and Authorization

Use `kubectl auth can-i` to verify what a user or service account can do.

```bash
kubectl auth can-i create deployments --as=system:serviceaccount:default:deployer
```

Expected output:

```
yes
```

If you get `no`, check RBAC roles and bindings. This command does not require changing anything and is safe to run.

## Failure Modes and Recovery

Even with careful management, the API server can fail. Here are common failure modes, how to detect them, and recovery steps.

### API Server Pod CrashLoopBackOff

If the pod is not ready and restarting, inspect the logs immediately.

```bash
kubectl describe pod kube-apiserver-node1 -n kube-system
```

Check the Events section for messages like `Back-off restarting failed container` or specific errors such as `Error: --etcd-servers must be specified`.

Common causes:

- Misconfigured etcd endpoints (flag `--etcd-servers`)
- Incorrect TLS certificate paths or expired certificates
- Invalid admission plugin configuration

Recovery: fix the manifest, allow the kubelet to restart the pod, and monitor with `kubectl get pods -n kube-system -w`.

### etcd Unreachable

The API server depends on etcd. If etcd is down, the API server may become unhealthy but still run.

Check etcd health from a control plane node.

```bash
sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health
```

If etcd is down, first recover etcd, then restart the API server if needed.

### Certificate Expiry

Expired serving certificates prevent clients from connecting. Check certificate expiration dates.

```bash
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates
```

Expected output:

```
notBefore=Jan 12 10:00:00 2024 GMT
notAfter=Jan 11 10:00:00 2025 GMT
```

If the date has passed, renew the certificate using your cluster's renewal process (e.g., `kubeadm certs renew apiserver`).

### Admission Webhook Failure

A misbehaving admission webhook can block all resource creation or updates. If you see errors like `Internal error occurred: failed calling webhook`, check the webhook configuration.

```bash
kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
```

In an emergency, you can temporarily delete the faulty webhook configuration to unblock the API, but this should be a last resort and done with awareness of the security implications.

## Operations Checklist

Use this checklist before and after making changes to the API server or when diagnosing issues. Each item includes the responsible role and how often to revisit it.

| Check | Command or Action | Expected Result | Owner | Review Frequency |
|-------|-------------------|-----------------|-------|------------------|
| API server pods healthy | `kubectl get pods -n kube-system -l component=kube-apiserver` | All pods Running and Ready 1/1 | Platform Engineer: Alex Chen | Daily during on-call handoff |
| etcd health | `etcdctl endpoint health` | `is healthy` for all endpoints | Infrastructure Lead: Maria Garcia | Weekly or before any control plane change |
| Certificate expiry | `openssl x509 -in apiserver.crt -noout -dates` | notAfter date > 30 days away | Security Officer: Priya Shah | Monthly, and 30 days before expiry |
| Audit log enabled | Check `--audit-log-path` flag in manifest | Flag present and logs being written | Compliance Lead: John Kim | Quarterly audit review |
| RBAC policy tests | `kubectl auth can-i --list --as=<user>` | Expected permissions only | Platform Engineer: Alex Chen | After any role change or quarterly |
| Metrics and alerts | Query monitoring system for API server request latency and errors | Latency within SLO, no spike in 5xx errors | SRE Lead: Sara Ali | Weekly review |

Run this checklist at least weekly in production clusters. Store results in a shared log for trend analysis. When a check fails, assign an owner and a due date for remediation.

## Common Pitfalls

Even experienced operators make mistakes with the API server. Here are frequent pitfalls and how to avoid them.

### Running Destructive Commands Before Observing

It is tempting to restart the API server or delete a pod to fix a problem, but that can hide the root cause. Always run read-only diagnostics first and capture logs and metrics. Only then make a change.

### Ignoring Version Differences

Flags and file paths change between Kubernetes versions. A command from a blog post for version 1.18 may not work on 1.28. Always consult the official documentation for your specific version.

### Editing Static Pod Manifests Without Backup

A typo in the manifest can take down the API server. Without a backup, recovery is slow. Always copy the manifest to a safe location before editing. Use version control for manifests if possible.

### Using kubectl Proxy Without Restricting Access

`kubectl proxy` exposes the API server on localhost by default, but if you bind it to 0.0.0.0 or use port forwarding carelessly, you could expose your cluster. Always use the default localhost binding and close the proxy when done.

### Overlooking Certificate Expiry

Expired certificates are a common cause of sudden API server failure. Set up monitoring and alerts for certificate expiration at least 30 days in advance. Automate renewal where possible.

### Misconfigured Admission Webhooks

Admission webhooks can silently break deployments if they are down or misconfigured. Always test webhook changes in a staging cluster and have a rollback plan. Monitor webhook latency and availability.

## Conclusion

Operating the Kubernetes API server requires a careful balance of observation and intervention. With the commands in this article, you can systematically check health, inspect configuration, diagnose failures, and recover safely. Remember to always start with read-only commands, back up before changes, verify after changes, and document recovery steps.

Next, pick one low-risk verification from the checklist and run it in your cluster today. Record the current state, compare it with expected output, and investigate any differences. Review your RBAC policies and certificate expiration dates soon. A well-practiced operations routine will make future incidents less stressful and faster to resolve.