Kubeadm Troubleshooting with Practical Examples

Intro

Troubleshooting a Kubernetes cluster bootstrapped with kubeadm means moving from an observed symptom to a verified fix without making things worse. The fastest path is almost never a blind kubeadm reset. It is a sequence of read-only checks that confirm the exact version, topology, and component state before you change a single file.

This article is for developers, DevOps consultants, and small platform teams who operate self-managed kubeadm clusters. It connects common kubeadm failure modes to concrete commands, expected output, log interpretation, and safe recovery steps. Every example uses placeholders for hostnames, paths, and versions so you can adapt it to your environment without exposing production secrets.

The guiding rules are: observe before changing, limit the blast radius, verify after every modification, and document the recovery path before you need it.

Version and Environment Inventory

Before touching anything, capture the current state. You need to know exactly which Kubernetes version, kubeadm version, container runtime, and operating system you are dealing with. Then record the node topology and the current kubeadm configuration.

Start with read-only commands:

# Kubernetes client and server versions
kubectl version --client --output=yaml

# kubeadm and kubelet versions
kubeadm version -o json
kubelet --version

# Operating system and kernel
cat /etc/os-release
uname -r

# Container runtime (example: containerd)
ctr version

Expected output for kubeadm version -o json on a healthy node looks similar to:

{
  "clientVersion": {
    "major": "1",
    "minor": "30",
    "gitVersion": "v1.30.1",
    "gitCommit": "ac3b1c5e035d74a5c1e6f7d4d1abf0e2a8f5e7b1",
    "gitTreeState": "clean",
    "buildDate": "2024-05-15T12:00:00Z",
    "goVersion": "go1.22.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}

If kubeadm version fails with command not found, then kubeadm is not installed on this node. That is a different problem from a version skew. Install the correct package for your OS, but only after you document that this node is not yet part of the control plane.

Now check the cluster topology and node status:

# From a working control plane node
kubectl get nodes -o wide

# Detailed node information
kubectl describe node control-plane-1

Look for node NotReady conditions. kubectl describe node shows the reason under Conditions. Common reasons are KubeletNotReady, ContainerRuntimeUnhealthy, or NetworkUnavailable.

Next, record the active kubeadm configuration. The cluster configuration generated during kubeadm init is stored in the kube-system/kubeadm-config ConfigMap. Get it read-only:

kubectl -n kube-system get configmap kubeadm-config -o yaml

Pay attention to ClusterConfiguration.kubernetesVersion, ClusterConfiguration.controlPlaneEndpoint, and ClusterConfiguration.networking.podSubnet. For example:

apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.30.1
controlPlaneEndpoint: "kube-apiserver.example.com:6443"
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"

Do not edit this ConfigMap directly. Copy it to a local file for reference only.

Finally, record timestamps for later correlation. date -u prints the current UTC time. Add all of this information to your incident notes.

Safe Configuration Path

Kubeadm stores cluster configuration in three places that you may need to modify: the kubeadm configuration file on disk, the kubeadm-config ConfigMap, and the kubelet configuration. Only change one of them at a time, and always keep a backup.

Back up before changing

For files on disk, use cp -a to preserve permissions and ownership:

sudo cp -a /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.bak-$(date +%Y%m%d)
sudo cp -a /var/lib/kubelet/config.yaml /var/lib/kubelet/config.yaml.bak-$(date +%Y%m%d)

For the kubeadm-config ConfigMap, export a clean copy:

kubectl -n kube-system get configmap kubeadm-config -o yaml > kubeadm-config-backup.yaml

Store backups outside /etc/kubernetes so a failed reset does not delete them.

Change kubelet configuration safely

The kubelet configuration file /var/lib/kubelet/config.yaml controls runtime behavior such as maxPods, systemReserved, and evictionHard. For example, to increase the maximum number of pods per node from the default 110 to 150:

Original file snippet:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
maxPods: 110

Edit only the maxPods field to 150. Then restart kubelet:

sudo systemctl restart kubelet
sudo systemctl status kubelet --no-pager

Verify the active configuration took effect by checking the kubelet process flags:

ps -ef | grep kubelet | grep -o 'config.yaml'
# Expected output contains: --config=/var/lib/kubelet/config.yaml

Also confirm the node still reports Ready:

kubectl get node worker-1
# Expected: NAME       STATUS   ROLES    AGE   VERSION
#          worker-1   Ready    <none>   12d   v1.30.1

If the node becomes NotReady or kubelet fails to start, revert the change by restoring the backup and restarting kubelet again.

Update kubeadm cluster configuration

Cluster-wide upgrades or feature gate changes are done with kubeadm upgrade apply or kubeadm upgrade node. Do not modify the kubeadm-config ConfigMap by hand except in emergencies, and even then document while doing it.

To preview an upgrade without changing anything:

sudo kubeadm upgrade plan

Expected output shows both the current and target Kubernetes version, for example:

COMPONENT   CURRENT   TARGET
kubelet     v1.29.2   v1.30.1
kubeadm     v1.29.2   v1.30.1
kubectl     v1.29.2   v1.30.1

Never apply an upgrade without reading the release notes for breaking changes specific to your environment.

Verification and Diagnostics

Once you understand the current state, use targeted diagnostics to identify the failing component. The most common sources of kubeadm problems are kubelet failing to start, the control plane static pods not becoming ready, and network plugin issues.

Check kubelet status and logs

The kubelet is the first thing to fail when certificates expire, container runtime is unavailable, or the kubelet configuration is invalid.

sudo systemctl status kubelet --no-pager -l

A healthy kubelet shows active (running). If it is in failed state, check the logs:

sudo journalctl -u kubelet -n 50 --no-pager -l

Common error messages and their likely causes:

• failed to run Kubelet: misconfiguration: kubelet cgroup driver: "systemd" is different from docker cgroup driver: "cgroupfs" -- cgroup driver mismatch. Fix by setting cgroupDriver: systemd in kubelet config or container runtime config.
• error: failed to start container "kubelet": Error response from daemon: ... -- container runtime not running or misconfigured. Check sudo systemctl status containerd or sudo systemctl status docker.
• x509: certificate signed by unknown authority -- often means the cluster CA has changed or kubelet.conf is corrupted. Regenerate with kubeadm init phase kubeconfig kubelet or copy from another node, but be extremely careful with secrets.

Inspect control plane static pods

Control plane components run as static pods in /etc/kubernetes/manifests. If they crash, the cluster loses API access.

List the static pods:

ls -la /etc/kubernetes/manifests/

Expected files: etcd.yaml, kube-apiserver.yaml, kube-controller-manager.yaml, kube-scheduler.yaml.

Check the logs of the API server pod even if kubectl is not responding:

# Using crictl because kubelet manages static pods via CRI
sudo crictl ps -a | grep kube-apiserver

Get the container ID and then:

sudo crictl logs <container-id> --tail=100

Watch for errors like etcdserver: request timed out (etcd down), connection refused (networking), or failed to create listener: failed to listen on 0.0.0.0:6443 (port already in use).

Verify the API server endpoint

From a control plane node, test the local API server:

curl -k https://127.0.0.1:6443/healthz?verbose

Expected output includes [+]ping ok and [+]log ok. If you get connection refused, the API server is not running. If you get a certificate error, may be time or CA mismatch.

Check cluster networking

Pod networking failure shows up as CNI plugin not initialized in kubelet logs or nodes stuck NotReady after join. Confirm your CNI plugin pods are running:

kubectl get pods -n kube-system | grep -E 'calico|flannel|weave|cilium'

For Flannel, expected pods are kube-flannel-ds-xxxxx with 1/1 Running. If they are CrashLoopBackOff, check their logs:

kubectl -n kube-system logs ds/kube-flannel-ds --tail=50

Common Flannel error: failed to find any valid interface to use when multiple interfaces exist. Fix by adding --iface=eth1 to the flannel daemonset or setting --iface-regex.

Failure Modes and Recovery

When something is already broken, follow a structured recovery path. The order matters: restore service first, then investigate root cause, then apply prevention.

Failure mode 1: kubelet certificate rotation failed

Symptom: Nodes become NotReady, kubectl get nodes shows SchedulingDisabled or NotReady for all workers, kubelet logs show certificate has expired or is not yet valid.

Cause: kubelet client certificates are valid for 1 year by default and may fail to rotate if the certificate signing request (CSR) approval is not automatic.

Recovery:

1. Check for pending CSRs:

kubectl get csr

Expected output shows pending CSRs with Pending status for each node.

1. Approve all pending CSRs:

kubectl get csr --no-headers | awk '{print $1}' | xargs -I {} kubectl certificate approve {}

1. Verify nodes return to Ready:

kubectl get nodes

If automatic CSR approval is missing, implement a controller like kubelet-csr-approver or adjust your kube-controller-manager flags.

Failure mode 2: etcd data directory corrupted

Symptom: kubectl get nodes returns The connection to the server 192.168.1.10:6443 was refused - did you specify the right host or port? and API server pod logs show etcdmain: cannot access data directory.

Cause: Disk full on etcd data directory or accidental deletion of /var/lib/etcd.

Recovery:

1. Confirm etcd pod is crash looping:

sudo crictl ps -a | grep etcd

1. Check disk space:

df -h /var/lib/etcd

1. If disk is full, stop etcd pod by moving the manifest temporarily:

sudo mv /etc/kubernetes/manifests/etcd.yaml /tmp/etcd.yaml.bak

1. Free space or restore from etcd backup. If no backup, you must rebuild the cluster. This is why etcd backups are non-negotiable.

1. Restore etcd from snapshot (only if you have one):

sudo ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot.db --data-dir /var/lib/etcd-restore
sudo mv /var/lib/etcd-restore /var/lib/etcd
sudo mv /tmp/etcd.yaml.bak /etc/kubernetes/manifests/etcd.yaml

1. Wait for API server to come back, then verify with kubectl get nodes.

Failure mode 3: worker node join failed with token error

Symptom: On a new worker, kubeadm join exits with error execution phase preflight: couldn't validate the identity of the API Server: expected a 32byte but got 0 or token not found.

Cause: Join token expired (default 24 hours) or CA hash mismatch.

Recovery:

1. On the control plane, create a new token:

sudo kubeadm token create --print-join-command

Expected output:

kubeadm join 10.0.0.10:6443 --token abcdef.0123456789abcdef --discovery-token-ca-cert-hash sha256:1a2b3c...

1. Run that exact command on the new worker with sudo.

1. Verify the node joins:

kubectl get nodes

If the CA hash is still mismatched, you may have multiple clusters or an outdated ca.crt. Inspect the certificate hash manually:

openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'

Compare that with the hash in the join command.

Operations Checklist

Use this checklist during any kubeadm troubleshooting session. It is ordered from least invasive to most invasive.

Go through the checklist top to bottom. Do not skip to step 8 without doing steps 1-7 first. The checklist is also useful as a post-incident review document.

Conclusion

Kubeadm troubleshooting is practical only when every step is version-scoped, observable, and reversible where the technology allows. Copying a command from a forum without understanding the expected output and recovery path is not an operations procedure; it is a gamble.

As a next step, pick one low-risk diagnostic from this article, such as checking kubeadm version -o json or kubectl get csr. Record the current state, run the command, compare the result with the expected signal, and write down what you would do if it failed.

A reliable workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Keep these habits in place, and your kubeadm clusters will spend far less time in mysterious failure states.