## Intro

Troubleshooting a Kubernetes cluster bootstrapped with kubeadm means moving from an observed symptom to a verified fix without making things worse. The fastest path is almost never a blind `kubeadm reset`. It is a sequence of read-only checks that confirm the exact version, topology, and component state before you change a single file.

This article is for developers, DevOps consultants, and small platform teams who operate self-managed kubeadm clusters. It connects common kubeadm failure modes to concrete commands, expected output, log interpretation, and safe recovery steps. Every example uses placeholders for hostnames, paths, and versions so you can adapt it to your environment without exposing production secrets.

The guiding rules are: observe before changing, limit the blast radius, verify after every modification, and document the recovery path before you need it.

## Version and Environment Inventory

Before touching anything, capture the current state. You need to know exactly which Kubernetes version, kubeadm version, container runtime, and operating system you are dealing with. Then record the node topology and the current kubeadm configuration.

Start with read-only commands:

```bash
# Kubernetes client and server versions
kubectl version --output=yaml

# kubeadm and kubelet versions
kubeadm version -o json
kubelet --version

# Operating system and kernel
cat /etc/os-release
uname -r

# Container runtime (example: containerd)
ctr version
```

Expected output for `kubeadm version -o json` on a healthy node looks similar to:

```json
{
  "clientVersion": {
    "major": "1",
    "minor": "30",
    "gitVersion": "v1.30.1",
    "gitCommit": "ac3b1c5e035d74a5c1e6f7d4d1abf0e2a8f5e7b1",
    "gitTreeState": "clean",
    "buildDate": "2024-05-15T12:00:00Z",
    "goVersion": "go1.22.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}
```

If `kubeadm version` fails with `command not found`, then kubeadm is not installed on this node. That is a different problem from a version skew. Install the correct package for your OS, but only after you document that this node is not yet part of the control plane.

Now check the cluster topology and node status:

```bash
# From a working control plane node
kubectl get nodes -o wide

# Detailed node information
kubectl describe node control-plane-1
```

Look for node `NotReady` conditions. `kubectl describe node` shows the reason under `Conditions`. Common reasons are `KubeletNotReady`, `ContainerRuntimeUnhealthy`, or `NetworkUnavailable`.

Next, record the active kubeadm configuration. The cluster configuration generated during `kubeadm init` is stored in the `kube-system/kubeadm-config` ConfigMap. Get it read-only:

```bash
kubectl -n kube-system get configmap kubeadm-config -o yaml
```

Pay attention to `ClusterConfiguration.kubernetesVersion`, `ClusterConfiguration.controlPlaneEndpoint`, and `ClusterConfiguration.networking.podSubnet`. For example:

```yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.30.1
controlPlaneEndpoint: "kube-apiserver.example.com:6443"
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
```

Do not edit this ConfigMap directly. Copy it to a local file for reference only.

Finally, record timestamps for later correlation. `date -u` prints the current UTC time. Add all of this information to your incident notes.

## Safe Configuration Path

Kubeadm stores cluster configuration in three places that you may need to modify: the kubeadm configuration file on disk, the kubeadm-config ConfigMap, and the kubelet configuration. Only change one of them at a time, and always keep a backup.

### Back up before changing

For files on disk, use `cp -a` to preserve permissions and ownership:

```bash
sudo cp -a /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.bak-$(date +%Y%m%d)
sudo cp -a /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak-$(date +%Y%m%d)
```

For the kubeadm-config ConfigMap, export a clean copy:

```bash
kubectl -n kube-system get configmap kubeadm-config -o yaml > kubeadm-config-backup.yaml
```

Store backups outside `/etc/kubernetes` so a failed reset does not delete them.

### Change kubelet configuration safely

The kubelet configuration file `/var/lib/kubelet/config.yaml` controls runtime behavior such as `maxPods`, `systemReserved`, and `evictionHard`. For example, to increase the maximum number of pods per node from the default 110 to 150:

Original file snippet:

```yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 110
```

Edit only the `maxPods` field to `150`. Then restart kubelet:

```bash
sudo systemctl restart kubelet
sudo systemctl status kubelet --no-pager
```

Verify the active configuration took effect by checking the kubelet process flags:

```bash
ps -ef | grep kubelet | grep -o 'config.yaml'
# Expected output contains: --config=/var/lib/kubelet/config.yaml
```

Also confirm the node still reports Ready:

```bash
kubectl get node worker-1
# Expected: NAME       STATUS   ROLES    AGE   VERSION
#          worker-1   Ready    <none>   12d   v1.30.1
```

If the node becomes NotReady or kubelet fails to start, revert the change by restoring the backup and restarting kubelet again.

### Update kubeadm cluster configuration

Cluster-wide upgrades or feature gate changes are done with `kubeadm upgrade apply` or `kubeadm upgrade node`. Do not modify the kubeadm-config ConfigMap by hand except in emergencies, and even then document while doing it.

To preview an upgrade without changing anything:

```bash
sudo kubeadm upgrade plan
```

Expected output shows both the current and target Kubernetes version, for example:

```text
COMPONENT   CURRENT   TARGET
kubelet     v1.29.2   v1.30.1
kubeadm     v1.29.2   v1.30.1
kubectl     v1.29.2   v1.30.1
```

Never apply an upgrade without reading the release notes for breaking changes specific to your environment.

## Verification and Diagnostics

Once you understand the current state, use targeted diagnostics to identify the failing component. The most common sources of kubeadm problems are kubelet failing to start, the control plane static pods not becoming ready, and network plugin issues.

### Check kubelet status and logs

The kubelet is the first thing to fail when certificates expire, container runtime is unavailable, or the kubelet configuration is invalid.

```bash
sudo systemctl status kubelet --no-pager -l
```

A healthy kubelet shows `active (running)`. If it is in `failed` state, check the logs:

```bash
sudo journalctl -u kubelet -n 50 --no-pager -l
```

Common error messages and their likely causes:

- `failed to run Kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs"` -- cgroup driver mismatch. Fix by setting `cgroupDriver: systemd` in kubelet config or container runtime config.
- `error: failed to start container "kubelet": Error response from daemon: ...` -- container runtime not running or misconfigured. Check `sudo systemctl status containerd` or `sudo systemctl status docker`.
- `x509: certificate signed by unknown authority` -- often means the cluster CA has changed or `kubelet.conf` is corrupted. Regenerate with `kubeadm init phase kubeconfig kubelet` or copy from another node, but be extremely careful with secrets.

### Inspect control plane static pods

Control plane components run as static pods in `/etc/kubernetes/manifests`. If they crash, the cluster loses API access.

List the static pods:

```bash
ls -la /etc/kubernetes/manifests/
```

Expected files: `etcd.yaml`, `kube-apiserver.yaml`, `kube-controller-manager.yaml`, `kube-scheduler.yaml`.

Check the logs of the API server pod even if kubectl is not responding:

```bash
# Using crictl because kubelet manages static pods via CRI
sudo crictl ps -a | grep kube-apiserver
```

Get the container ID and then:

```bash
sudo crictl logs <container-id> --tail=100
```

Watch for errors like `etcdserver: request timed out` (etcd down), `connection refused` (networking), or `failed to create listener: failed to listen on 0.0.0.0:6443` (port already in use).

### Verify the API server endpoint

From a control plane node, test the local API server:

```bash
curl -k https://127.0.0.1:6443/healthz?verbose
```

Expected output includes `[+]ping ok` and `[+]log ok`. If you get `connection refused`, the API server is not running. If you get a certificate error, may be time or CA mismatch.

### Check cluster networking

Pod networking failure shows up as `CNI plugin not initialized` in kubelet logs or nodes stuck NotReady after join. Confirm your CNI plugin pods are running:

```bash
kubectl get pods -n kube-system | grep -E 'calico|flannel|weave|cilium'
```

For Flannel, expected pods are `kube-flannel-ds-xxxxx` with `1/1 Running`. If they are `CrashLoopBackOff`, check their logs:

```bash
kubectl -n kube-system logs ds/kube-flannel-ds --tail=50
```

Common Flannel error: `failed to find any valid interface to use` when multiple interfaces exist. Fix by adding `--iface=eth1` to the flannel daemonset or setting `--iface-regex`.

## Failure Modes and Recovery

When something is already broken, follow a structured recovery path. The order matters: restore service first, then investigate root cause, then apply prevention.

### Failure mode 1: kubelet certificate rotation failed

**Symptom**: Nodes become NotReady, `kubectl get nodes` shows `SchedulingDisabled` or `NotReady` for all workers, kubelet logs show `certificate has expired or is not yet valid`.

**Cause**: kubelet client certificates are valid for 1 year by default and may fail to rotate if the certificate signing request (CSR) approval is not automatic.

**Recovery**:

1. Check for pending CSRs:

```bash
kubectl get csr
```

Expected output shows pending CSRs with `Pending` status for each node.

2. Approve all pending CSRs:

```bash
kubectl get csr --no-headers | awk '{print $1}' | xargs -I {} kubectl certificate approve {}
```

3. Verify nodes return to Ready:

```bash
kubectl get nodes
```

If automatic CSR approval is missing, implement a controller like `kubelet-csr-approver` or adjust your `kube-controller-manager` flags.

### Failure mode 2: etcd data directory corrupted

**Symptom**: `kubectl get nodes` returns `The connection to the server 192.168.1.10:6443 was refused - did you specify the right host or port?` and API server pod logs show `etcdmain: cannot access data directory`.

**Cause**: Disk full on etcd data directory or accidental deletion of `/var/lib/etcd`.

**Recovery**:

1. Confirm etcd pod is crash looping:

```bash
sudo crictl ps -a | grep etcd
```

2. Check disk space:

```bash
df -h /var/lib/etcd
```

3. If disk is full, stop etcd pod by moving the manifest temporarily:

```bash
sudo mv /etc/kubernetes/manifests/etcd.yaml /tmp/etcd.yaml.bak
```

4. Free space or restore from etcd backup. If no backup, you must rebuild the cluster. This is why etcd backups are non-negotiable.

5. Restore etcd from snapshot (only if you have one):

```bash
sudo ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot.db --data-dir /var/lib/etcd-restore
sudo mv /var/lib/etcd-restore /var/lib/etcd
sudo mv /tmp/etcd.yaml.bak /etc/kubernetes/manifests/etcd.yaml
```

6. Wait for API server to come back, then verify with `kubectl get nodes`.

### Failure mode 3: worker node join failed with token error

**Symptom**: On a new worker, `kubeadm join` exits with `error execution phase preflight: couldn't validate the identity of the API Server: expected a 32byte but got 0` or `token not found`.

**Cause**: Join token expired (default 24 hours) or CA hash mismatch.

**Recovery**:

1. On the control plane, create a new token:

```bash
sudo kubeadm token create --print-join-command
```

Expected output:

```text
kubeadm join 10.0.0.10:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:1a2b3c...
```

2. Run that exact command on the new worker with sudo.

3. Verify the node joins:

```bash
kubectl get nodes
```

If the CA hash is still mismatched, you may have multiple clusters or an outdated `ca.crt`. Inspect the certificate hash manually:

```bash
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
```

Compare that with the hash in the join command.

## Operations Checklist

Use this checklist during any kubeadm troubleshooting session. It is ordered from least invasive to most invasive.

| Step | Action | Command | Expected Result | Recovery if Fail |
| --- | --- | --- | --- | --- |
| 1 | Check versions | `kubeadm version -o json` | Version JSON output | Install or upgrade kubeadm |
| 2 | Check node status | `kubectl get nodes` | All nodes Ready | Diagnose kubelet, runtime, or network |
| 3 | Check kubelet status | `sudo systemctl status kubelet` | `active (running)` | Restart kubelet, inspect logs |
| 4 | Check API health | `curl -k https://127.0.0.1:6443/healthz?verbose` | `ok` for all checks | Inspect API server pod logs |
| 5 | Check static pods | `sudo crictl ps -a \| grep -E 'etcd\|kube-api\|kube-controller\|kube-scheduler'` | All running | Restart kubelet to re-pull manifests |
| 6 | Check pending CSRs | `kubectl get csr` | No Pending CSRs | Approve CSRs or install approver |
| 7 | Check disk space | `df -h /var/lib/etcd /var/lib/kubelet` | > 20% free | Clean up or expand disk |
| 8 | Check pod logs | `kubectl -n kube-system logs <pod-name> --tail=50` | No repeated errors | Fix underlying issue, then restart pod |

Go through the checklist top to bottom. Do not skip to step 8 without doing steps 1-7 first. The checklist is also useful as a post-incident review document.

## Conclusion

Kubeadm troubleshooting is practical only when every step is version-scoped, observable, and reversible where the technology allows. Copying a command from a forum without understanding the expected output and recovery path is not an operations procedure; it is a gamble.

As a next step, pick one low-risk diagnostic from this article, such as checking `kubeadm version -o json` or `kubectl get csr`. Record the current state, run the command, compare the result with the expected signal, and write down what you would do if it failed.

A reliable workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Keep these habits in place, and your kubeadm clusters will spend far less time in mysterious failure states.