## Intro The Kubernetes API server (kube-apiserver) is the front-end for the Kubernetes control plane. It exposes the Kubernetes API and acts as the gateway for all internal and external interactions with the cluster. Understanding its architecture, configuration, and operational behavior is essential for anyone responsible for running Kubernetes in production. This guide provides a practical, hands-on approach to kube-apiserver architecture and operations. It is written for developers, DevOps consultants, and startup technical teams who need to move beyond theory and into real-world command-line work. We will cover: - Core components of the API server and how they fit together - How a request flows through the API server, from admission to storage - Safe configuration paths and common tuning parameters - Verification and diagnostic techniques - Common failure modes and recovery steps - An operations checklist for day-to-day management Throughout, we emphasize operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and know how to recover if something goes wrong. ## Version and Environment Inventory Before making any changes, you must know exactly what you are working with. This section covers how to identify the installed Kubernetes version, deployment topology, and key configuration details. ### Identify Cluster Version and Topology Use `kubectl version` to see both the client and server versions: ```bash kubectl version --short ``` Expected output (example): ```text Client Version: v1.29.2 Server Version: v1.29.2 ``` In a managed cluster (e.g., EKS, GKE, AKS), the API server is not directly accessible; you interact via the cloud provider's control plane endpoint. In a self-managed cluster (e.g., kubeadm, kops, manual), the API server runs as a static pod or systemd service on control plane nodes. To determine how the API server is deployed, check for static pod manifests or systemd unit files: ```bash # For static pod (common with kubeadm) ls /etc/kubernetes/manifests/kube-apiserver.yaml # For systemd service (common with manual installation) systemctl status kube-apiserver ``` ### Read-Only Observation Always start with read-only commands to gather current state. For example, to see the API server pod details: ```bash kubectl get pods -n kube-system -l component=kube-apiserver -o wide ``` Expected output (example): ```text NAME READY STATUS RESTARTS AGE IP NODE kube-apiserver-master 1/1 Running 1 10d 10.0.0.10 master-1 ``` Note the IP address, node, and restart count; these are clues for troubleshooting. To check the API server's configured flags and their values without modifying anything, use `ps` (if running as a process) or `kubectl describe` (for static pods): ```bash # If running as a systemd service ps aux | grep kube-apiserver # If running as a static pod kubectl describe pod -n kube-system kube-apiserver-master ``` The `describe` output includes the command and arguments, which show all flags. Look for critical flags such as `--etcd-servers`, `--service-cluster-ip-range`, `--authorization-mode`, and `--enable-admission-plugins`. **Note on control plane logs:** The official Kubernetes troubleshooting guide for control plane nodes references node-level log files such as `/var/log/kube-apiserver.log` (and similar files for other components). If the API server runs as a static pod (common with kubeadm), `kubectl logs -n kube-system ` also works and shows the same output. For non-kubeadm or systemd-managed deployments, use the node log file directly. ### Smallest Justified Change After collecting baseline information, identify a specific configuration item to change. For example, suppose you need to increase the API server's request timeout or enable a feature gate. Only change one flag at a time, and always back up the original manifest or unit file. Example: To enable the `PodSecurity` admission plugin (if not already enabled), you would edit the API server manifest and add the plugin to the list. But first, verify the current list: ```bash kubectl describe pod -n kube-system kube-apiserver-master | grep enable-admission-plugins ``` If the output shows `--enable-admission-plugins=NodeRestriction`, you would append `,PodSecurity` to that flag. The next section covers safe configuration changes in detail. ### Prerequisites and Blast Radius Before changing the API server configuration, ensure you have: - Access to the control plane node(s) (SSH or equivalent) - Backup of the current manifest or configuration file (e.g., `/etc/kubernetes/manifests/kube-apiserver.yaml`) - A maintenance window or understanding of the impact: restarting the API server briefly interrupts API access but does not affect running workloads (they continue to run, but new scheduling or API calls fail until it's back) - Ability to roll back quickly (restore the backup and let the kubelet restart the pod) Blast radius: The API server is a single point of failure for control plane operations. Changes that cause the API server to fail to start will make the cluster unmanageable until fixed. However, existing pods and services continue to run; you just can't control them. ### Verification Step After a change, verify the API server is healthy: ```bash kubectl get --raw='/readyz?verbose' | grep -A2 'readyz' ``` Expected output: `ok` for the overall readiness. If there is a problem, the output will show which check failed (e.g., `poststarthook/start-kube-apiserver-admission-initializer` failed). Also check the API server pod status: ```bash kubectl get pods -n kube-system -l component=kube-apiserver ``` If the pod is `CrashLoopBackOff`, inspect logs using either the node log file or the static pod logs: ```bash # Node log file (referenced by official control plane troubleshooting): sudo tail -n 50 /var/log/kube-apiserver.log # Static pod alternative (kubeadm): kubectl logs -n kube-system kube-apiserver-master --tail=50 ``` ### Recovery Path If the API server fails to start, revert to the backup manifest: ```bash cp /etc/kubernetes/manifests/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml ``` The kubelet will detect the file change and restart the pod automatically. Monitor the pod status until `Running` and `Ready`. ## Safe Configuration Path This section provides a structured approach to modifying the API server configuration safely. The API server has many flags and configuration files; we will focus on common operational changes. ### Understanding the Configuration Mechanism The API server is configured via command-line flags in most self-managed deployments. These flags are defined in the pod manifest (static pod) or systemd unit file. In newer Kubernetes versions (1.19+), there is a move toward configuration files (`--config` flag), but many flags remain. Example of a typical `kube-apiserver.yaml` manifest (excerpt): ```yaml apiVersion: v1 kind: Pod metadata: name: kube-apiserver namespace: kube-system spec: containers: - command: - kube-apiserver - --advertise-address=10.0.0.10 - --allow-privileged=true - --authorization-mode=Node,RBAC - --client-ca-file=/etc/kubernetes/pki/ca.crt - --enable-admission-plugins=NodeRestriction - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key - --etcd-servers=https://127.0.0.1:2379 - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key - --service-account-key-file=/etc/kubernetes/pki/sa.pub - --service-cluster-ip-range=10.96.0.0/12 - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key image: registry.k8s.io/kube-apiserver:v1.29.2 ... ``` ### Observation Before Change Always record the current state of flags using the commands from the previous section. Additionally, note the API server's current resource usage and logs: ```bash # Check resource usage of the API server container docker stats kube-apiserver-master # if using Docker # or crictl stats kube-apiserver-master # if using containerd ``` Logs can show warnings or errors that might indicate misconfiguration. Use the node log file or the static pod logs: ```bash # Node log file (official control plane troubleshooting location): sudo tail -n 100 /var/log/kube-apiserver.log # Static pod alternative: kubectl logs -n kube-system kube-apiserver-master --tail=100 ``` ### Smallest Justified Change: Example - Increasing Request Timeout Suppose you notice that long-running requests (e.g., large `kubectl get` of many resources) sometimes time out. The default `--request-timeout` is 1 minute (60 seconds). You can increase it to 2 minutes. Steps: 1. Backup the current manifest: ```bash cp /etc/kubernetes/manifests/kube-apiserver.yaml /etc/kubernetes/manifests/kube-apiserver.yaml.bak ``` 2. Edit the manifest (e.g., with `vi`) and add or modify the flag: ``` - --request-timeout=2m ``` 3. Save and exit. The kubelet automatically restarts the API server pod. 4. Verify the pod restarts and becomes ready: ```bash kubectl get pods -n kube-system -l component=kube-apiserver -w ``` 5. Check the new flag is in effect: ```bash kubectl describe pod -n kube-system kube-apiserver-master | grep request-timeout ``` Expected output: `--request-timeout=2m` 6. Test a long-running request to ensure it works: ```bash time kubectl get pods --all-namespaces ``` It should complete without timeout errors. ### Changing Admission Controllers Admission controllers are a common area for customization. For example, to enable the `NamespaceLifecycle` admission controller (if not already present), you would add it to the `--enable-admission-plugins` flag. **Important**: The order of admission controllers matters because they are executed in order. Refer to the Kubernetes documentation for the recommended order. Example: Add `NamespaceLifecycle` at the beginning: Before: `--enable-admission-plugins=NodeRestriction,PodSecurity` After: `--enable-admission-plugins=NamespaceLifecycle,NodeRestriction,PodSecurity` After the change, restart and verify the API server starts and the plugins are active. ### Configuring Audit Logging Audit logging is often not enabled by default, but it's critical for security and compliance. Adding audit configuration requires more than just a flag; you need an audit policy file and a log backend. Example: Enable audit logging to a file with a basic policy. 1. Create an audit policy file (e.g., `/etc/kubernetes/audit-policy.yaml`): ```yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata ``` 2. Add flags to the API server manifest: ``` - --audit-policy-file=/etc/kubernetes/audit-policy.yaml - --audit-log-path=/var/log/kubernetes/audit.log - --audit-log-maxage=30 - --audit-log-maxbackup=10 - --audit-log-maxsize=100 ``` 3. Ensure the directory `/var/log/kubernetes` exists and is writable by the API server (usually the container runs as root in static pods, but in some hardened setups it may be non-root). 4. Restart the API server and verify the audit log file is created and receives entries. ### Verification and Recovery For every change, have a rollback plan. The backup file is your primary recovery mechanism. After any change, monitor the API server for a few minutes to ensure stability. Check the API server logs for errors and the cluster's overall health: ```bash kubectl get nodes kubectl get pods --all-namespaces | grep -v Running ``` If the API server enters a crash loop, restore the backup immediately. ## Verification and Diagnostics Once the API server is running, you need to be able to verify its health and diagnose issues. This section covers key diagnostics commands and what to look for. ### Readiness and Liveness Probes The API server exposes health endpoints on its secure port (default 6443). Use `kubectl get --raw` to check them: ```bash # Overall readiness kubectl get --raw='/readyz?verbose' # Overall liveness kubectl get --raw='/livez?verbose' ``` A healthy server returns `ok` for the top-level check. The verbose output shows individual checks, e.g.: ```text [+]ping ok [+]log ok [+]etcd ok [+]poststarthook/start-kube-apiserver-admission-initializer ok ... readyz check passed ``` If any check fails, it is marked with `[-]` and a failure reason. For example, if etcd is unreachable, you might see `[-]etcd failed: reason withheld`. ### Checking API Server Metrics The API server exposes Prometheus metrics at the `/metrics` endpoint. You can fetch them with `kubectl get --raw`: ```bash kubectl get --raw='/metrics' | head -n 20 ``` Useful metrics include: - `apiserver_request_total` - total number of API requests - `apiserver_request_duration_seconds` - request latency - `apiserver_current_inflight_requests` - number of requests currently being processed - `etcd_request_duration_seconds` - latency of etcd requests These metrics can be scraped by Prometheus and visualized in Grafana. For quick diagnosis, you can use `kubectl top` to see resource usage of the control plane components (if metrics server is installed): ```bash kubectl top pod -n kube-system -l component=kube-apiserver ``` ### Log Analysis API server logs are essential for troubleshooting. The official troubleshooting guide points to node log files such as `/var/log/kube-apiserver.log` for the API server. For static pods (e.g., kubeadm), `kubectl logs` on the pod also works. Use the option that fits your deployment. Retrieve logs with either command: ```bash # Node log file (preferred for control plane components): sudo tail -n 100 /var/log/kube-apiserver.log # Static pod alternative: kubectl logs -n kube-system kube-apiserver-master --tail=100 ``` Common log messages: - `http: TLS handshake error` - often indicates client certificate issues. - `etcdserver: request timed out` - problems connecting to etcd. - `Authentication failed` - invalid credentials or expired tokens. - `forbidden` - RBAC permissions missing. Use `grep` to filter for specific errors. For node log file: ```bash sudo grep -i error /var/log/kube-apiserver.log ``` Or for the static pod alternative: ```bash kubectl logs -n kube-system kube-apiserver-master | grep -i error ``` ### Tracing a Request To understand the request flow, enable audit logging with a detailed policy to capture request metadata. Then you can see each request's stages: authentication, authorization, admission, and persistence. This is useful for debugging API latency or permission issues. Example audit policy to capture all requests at Metadata level for a specific namespace: ```yaml apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata namespaces: ["my-app"] - level: Request resources: - group: "" resources: ["secrets"] ``` After enabling, search the audit log for a specific request using `grep` on the user or resource. ## Failure Modes and Recovery Even with careful management, the API server can fail. This section describes common failure scenarios and steps to recover. ### Scenario 1: API Server Pod CrashLoopBackOff **Symptom**: `kubectl get pods -n kube-system` shows the API server pod in `CrashLoopBackOff`. **Diagnosis**: 1. Check pod status and events: ```bash kubectl describe pod -n kube-system kube-apiserver-master ``` Look for events like `Back-off restarting failed container`. 2. Check logs. Use the node log file or the static pod logs: ```bash # Node log file: sudo tail -n 50 /var/log/kube-apiserver.log # Static pod alternative (previous container logs): kubectl logs -n kube-system kube-apiserver-master --previous ``` Common causes: invalid flag value, missing certificate files, etcd unreachable. **Recovery**: - If the issue is due to a recent configuration change, revert to the backup manifest: ```bash cp /etc/kubernetes/manifests/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml ``` - If certificates are missing or expired, restore from a reliable backup or regenerate them (using `kubeadm` or manual process). - If etcd is unreachable, fix etcd connectivity first (check etcd pod/process, network, firewalls). ### Scenario 2: API Server Unresponsive or High Latency **Symptom**: `kubectl` commands hang or time out. The API server pod may be running but not processing requests. **Diagnosis**: 1. Check API server pod CPU/memory usage: ```bash kubectl top pod -n kube-system kube-apiserver-master ``` 2. Check metrics: ```bash kubectl get --raw='/metrics' | grep apiserver_current_inflight_requests ``` High values (e.g., >1000) may indicate overload. 3. Check logs for slow requests or errors. Use the node log file or the static pod logs as described earlier. 4. Check etcd performance; etcd is often the bottleneck. **Recovery**: - If resource limits are set too low, increase CPU/memory in the manifest and restart. - If the API server is flooded with requests, identify the source (e.g., a misbehaving controller) and fix it. - If etcd is slow, consider scaling etcd or optimizing its disk. ### Scenario 3: Certificate Expiry **Symptom**: API server logs show TLS errors, or `kubectl` fails with certificate errors. **Diagnosis**: 1. Check certificate expiration: ```bash openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -dates ``` For kubeadm clusters, use `kubeadm certs check-expiration`. **Recovery**: - Renew certificates using the appropriate tool (e.g., `kubeadm certs renew all` for kubeadm clusters). - After renewal, restart the API server (and other control plane components as needed). ### Scenario 4: etcd Data Corruption **Symptom**: API server fails to start or returns errors about etcd. **Diagnosis**: - Check etcd logs and health: ```bash # From API server node crictl logs --tail=50 # or journalctl -u etcd ``` - Use etcdctl (if available) to check cluster health: ```bash ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health ``` **Recovery**: - Restore etcd from a snapshot if available. This is a critical operation; consult the Kubernetes documentation for detailed steps. - If no snapshot, disaster recovery procedures may be needed. ## Operations Checklist Use this checklist for routine operations and maintenance of the kube-apiserver. ### Daily/Weekly Checks - [ ] Verify API server pod is Running and Ready: ```bash kubectl get pods -n kube-system -l component=kube-apiserver ``` - [ ] Check API server health endpoints: ```bash kubectl get --raw='/readyz?verbose' kubectl get --raw='/livez?verbose' ``` - [ ] Review API server logs for errors (use `sudo grep -i error /var/log/kube-apiserver.log` for node logs, or `kubectl logs -n kube-system kube-apiserver-master | grep -i error` for static pods). - [ ] Monitor API server resource usage (CPU, memory) via `kubectl top` or monitoring dashboards. - [ ] Check etcd health and disk usage (since API server depends on etcd). - [ ] Ensure audit logs (if enabled) are being written and rotated properly. ### Change Management Checklist Before making any configuration change: - [ ] Backup the current manifest or configuration file. - [ ] Document the change and expected outcome. - [ ] Identify rollback steps. - [ ] Schedule a maintenance window if the change may cause downtime. - [ ] After change, monitor pod status, logs, and health endpoints. - [ ] Verify cluster functionality (e.g., run `kubectl get nodes`, `kubectl run test-pod`). ### Periodic Tasks - [ ] Review certificate expiration dates and renew before expiry (e.g., use `kubeadm certs check-expiration` monthly). - [ ] Test etcd backup and restore procedures. - [ ] Review audit policy and adjust as needed. - [ ] Review API server flags against Kubernetes version release notes (some flags may be deprecated). - [ ] Perform load testing to ensure the API server handles expected request volumes. ### Security Checklist - [ ] Ensure TLS certificates are valid and not using weak ciphers. - [ ] Verify authorization mode is set to `RBAC` (or `Node,RBAC`). - [ ] Audit RBAC roles and bindings to follow least privilege. - [ ] Enable audit logging if required for compliance. - [ ] Restrict access to the API server's secure port (firewall rules, network policies). - [ ] Use strong authentication mechanisms (e.g., OIDC, client certificates). - [ ] Keep the API server version up-to-date with security patches. ## Conclusion Understanding and operating the Kubernetes API server requires a systematic approach. In this guide, we covered: - How to inventory your cluster's version and deployment topology. - Safe configuration practices with concrete examples (timeout changes, admission controllers, audit logging). - Verification and diagnostic techniques using health endpoints, metrics, and logs. - Common failure modes and recovery steps. - An operations checklist for routine and change management. The key to operational safety is to observe before acting, make small reversible changes, verify outcomes, and always have a rollback plan. With these practices, you can maintain a healthy and secure Kubernetes control plane. As a next step, choose one low-risk verification task from the checklist (e.g., checking the `/readyz` endpoint), execute it on your cluster, and review the results. Then, consider enabling audit logging if not already in place. Understanding dependencies like etcd, kubelet, and the controller manager will deepen your operational knowledge and prepare you for more advanced troubleshooting. Remember: a reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.