## Intro Kubernetes etcd is the source of truth for your cluster. When etcd misbehaves, the entire control plane can become unavailable, and applications may stop scheduling or scaling. Troubleshooting etcd configuration and upgrades requires a methodical approach: observe the current state, understand the deployment topology, verify prerequisites, and make one scoped change at a time. This guide provides a practical, command-driven workflow for Kubernetes etcd configuration and upgrade troubleshooting. It is written for developers, DevOps consultants, and technical startup teams who need to diagnose issues quickly and safely. The goal is operational safety. We will follow a pattern: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document recovery steps before an incident forces a decision. Each section includes concrete commands, expected output, and failure signals for common etcd problems. ## Version and Environment Inventory Before touching etcd, you must know what you are working with. Gather version information, deployment topology, and cluster health. Run these commands from a control plane node or a machine with kubectl access to the cluster. First, check the Kubernetes version and the etcd image version running in the cluster: kubectl version --short kubectl get pods -n kube-system -l component=etcd -o jsonpath='{.items[*].spec.containers[0].image}{"\n"}' Expected output: Client Version: v1.27.3 Server Version: v1.27.3 registry.k8s.io/etcd:3.5.9-0 If etcd is running as a static pod managed by kubelet, you can also inspect the manifest directly: sudo cat /etc/kubernetes/manifests/etcd.yaml | grep image: Expected output: image: registry.k8s.io/etcd:3.5.9-0 Next, identify the etcd cluster topology. For a stacked control plane (etcd runs on the same nodes as the Kubernetes control plane), list the etcd members: sudo crictl ps | grep etcd Expected output shows one etcd container per control plane node. Alternatively, if you have etcdctl available: ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list Expected output lists members with their IDs, peer URLs, and client URLs. For example: 8211f1d0f64f3269, started, node1, https://192.168.1.10:2380, https://192.168.1.10:2379, false 91bc3c398fb3c146, started, node2, https://192.168.1.11:2380, https://192.168.1.11:2379, false fd422379fda50e48, started, node3, https://192.168.1.12:2380, https://192.168.1.12:2379, false Check the health of the etcd cluster: ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health Expected output for a healthy cluster: https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.1ms If the output shows unhealthy endpoints or timeouts, note the exact error. Common failures include certificate mismatches, network partitions, or insufficient disk space. Practical Kubernetes check for Version and Environment Inventory: start with kubectl get pods -n kube-system -o wide to see etcd pod status. Then use kubectl describe pod -n kube-system for events, and kubectl logs -n kube-system --previous for crash loop details. If the etcd pod is crash-looping, inspect the kubelet logs on the control plane node: sudo journalctl -u kubelet -n 100 --no-pager Look for lines mentioning etcd, such as failed to start etcd or etcd exited with status 1 . Keep the local test small. If you suspect etcd misconfiguration, verify connectivity from the kube-apiserver to etcd. On the control plane node, run: sudo ss -tunlp | grep 2379 Expected output shows etcd listening on port 2379 (client) and 2380 (peer). If not, check the etcd static pod manifest and ensure the --listen-client-urls and --listen-peer-urls flags are correct. ## Safe Configuration Path Changing etcd configuration is risky. Always backup first, then change one setting at a time, and verify after each change. The etcd data directory (default /var/lib/etcd ) contains the database. Before any change, take a snapshot. Create a snapshot using etcdctl: sudo ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key snapshot save /tmp/etcd-snapshot-$(date +%Y%m%d%H%M).db Expected output: Snapshot saved at /tmp/etcd-snapshot-20250320.db Verify the snapshot: sudo ETCDCTL_API=3 etcdctl snapshot status /tmp/etcd-snapshot-20250320.db Expected output shows hash, revision, total keys, and total size. Now, let's consider a common configuration change: adjusting etcd's quota backend bytes to handle a growing dataset. The default is 2 GB. Suppose your etcd is hitting the quota and returning etcdserver: mvcc: database space exceeded . You decide to increase the quota. Locate the etcd static pod manifest: sudo ls /etc/kubernetes/manifests/etcd.yaml Make a backup of the manifest before editing: sudo cp /etc/kubernetes/manifests/etcd.yaml /tmp/etcd.yaml.backup Open the manifest with a text editor (e.g., sudo vi /etc/kubernetes/manifests/etcd.yaml ) and add or modify the --quota-backend-bytes flag. For example, set it to 4 GB: - --quota-backend-bytes=4294967296 Save the file. The kubelet automatically detects the change and restarts the etcd pod. Wait a few seconds and check the pod status: kubectl get pods -n kube-system -l component=etcd Expected output: NAME READY STATUS RESTARTS AGE etcd-node1 1/1 Running 1 2m Check etcd health again: ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health If the pod fails to start, check the logs: kubectl logs etcd-node1 -n kube-system Common errors include invalid flag values or permissions. If the change causes repeated failures, restore the backup manifest and restart kubelet: sudo cp /tmp/etcd.yaml.backup /etc/kubernetes/manifests/etcd.yaml sudo systemctl restart kubelet Another common configuration issue is certificate rotation. If etcd fails to start after certificates are renewed, verify certificate validity: sudo openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -noout -dates Expected output: notBefore=Mar 20 10:00:00 2025 GMT notAfter=Mar 20 10:00:00 2026 GMT If the certificate is expired, regenerate it using kubeadm or your certificate management tool. ## Verification and Diagnostics After any change or when diagnosing a problem, you need to verify etcd health and diagnose issues. etcd provides several diagnostic endpoints and commands. Check the etcd member list and status: ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --write-out=table Expected output includes endpoint, ID, version, db size, leader, raft term, etc. Monitor the etcd leader status. In a healthy cluster, there should be exactly one leader. Run: ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status | grep -i leader If you see multiple leaders or none, that indicates a raft consensus problem. Check etcd metrics via the metrics endpoint. Enable metrics if not already. On a control plane node, curl the metrics endpoint: sudo curl -s https://127.0.0.1:2379/metrics --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key | grep etcd_server_leader_changes_seen_total Expected output: etcd_server_leader_changes_seen_total 2 A high number of leader changes indicates instability. Investigate performance issues. etcd might be slow if disk latency is high. Check disk I/O: sudo iostat -x 1 5 Look for high await or %util. etcd requires fast disks (SSD). If disks are slow, consider moving etcd data to faster storage. Check for database fragmentation: ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key defrag --cluster Expected output: Finished defragmenting etcd member[https://127.0.0.1:2379] Run defragmentation during maintenance windows to avoid performance impact. Practical Kubernetes check for Verification and Diagnostics: use the following sequence to systematically verify etcd. - Check pod status: kubectl get pods -n kube-system -l component=etcd - Check logs: kubectl logs -n kube-system etcd-node1 --tail=100 - Check events: kubectl describe pod etcd-node1 -n kube-system | grep Events -A 10 - Check etcd endpoint health as shown above. - Check kube-apiserver connectivity to etcd: kubectl get --raw /healthz?verbose and look for etcd-related messages. Expected output for a healthy API server includes etcd: ok or similar. ## Failure Modes and Recovery etcd can fail in various ways. Let's examine common failure modes and recovery procedures. Failure Mode 1: etcd pod crash loop due to invalid configuration If etcd crashes immediately after a configuration change, check logs. kubectl logs etcd-node1 -n kube-system --previous Typical error: invalid value for --quota-backend-bytes: ... or flag provided but not defined . Recovery: restore the previous manifest from backup and restart kubelet. Failure Mode 2: etcd data directory corruption If etcd fails to start and logs show panic: failed to open data directory or corrupt file , you may need to restore from a snapshot. First, stop etcd and kube-apiserver (if static pod, move the manifest files temporarily): sudo mv /etc/kubernetes/manifests/etcd.yaml /tmp/ sudo mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/ Restore the snapshot: sudo ETCDCTL_API=3 etcdctl snapshot restore /tmp/etcd-snapshot-20250320.db --data-dir=/var/lib/etcd-restore --name=node1 --initial-cluster=node1=https://192.168.1.10:2380 --initial-advertise-peer-urls=https://192.168.1.10:2380 Then replace the data directory: sudo mv /var/lib/etcd /var/lib/etcd.broken sudo mv /var/lib/etcd-restore /var/lib/etcd Move the manifests back: sudo mv /tmp/etcd.yaml /etc/kubernetes/manifests/ sudo mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/ Wait for pods to restart and verify health. Failure Mode 3: etcd cluster quorum lost If more than half of etcd members fail, the cluster loses quorum. The cluster cannot serve reads or writes. Recovery requires re-establishing quorum by removing failed members and possibly restoring from snapshot. First, identify the surviving members. On a healthy node, run: ETCDCTL_API=3 etcdctl --endpoints=https://192.168.1.10:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key member list If this fails due to no quorum, you may need to force a new cluster from a snapshot on a single member. For example, to recover using node1 as the source of truth, stop etcd on all nodes, restore snapshot on node1 with --initial-cluster containing only node1, then start etcd on node1. Once it is up, add other members back. This is complex; refer to Kubernetes documentation for detailed steps. Failure Mode 4: etcd running out of disk space If etcd logs show etcdserver: mvcc: database space exceeded , you must free space. Options: - Defragment the database: etcdctl defrag - Compact old revisions: etcdctl compact - Increase quota and resize disk. Compact to a revision (e.g., last 1000 changes): revision=$(ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint status --write-out="json" | jq -r '.[0].Status.header.revision') compact_to=$((revision-1000)) ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key compact $compact_to Then defrag. Practical Kubernetes check for Failure Modes and Recovery: always have a recent snapshot. Schedule regular snapshots using a cron job. Test restore procedure in a non-production environment. ## Operations Checklist Use this checklist to ensure you cover all bases when troubleshooting etcd configuration and upgrades. ### Before making changes - [ ] Confirm etcd version and Kubernetes version compatibility. For example, Kubernetes 1.27 supports etcd 3.5.x. - [ ] Take an etcd snapshot: sudo ETCDCTL_API=3 etcdctl snapshot save /tmp/etcd-snapshot-$(date +%Y%m%d%H%M).db - [ ] Record current etcd member list and health status. - [ ] Backup etcd manifest: sudo cp /etc/kubernetes/manifests/etcd.yaml /tmp/etcd.yaml.backup - [ ] Ensure you have console access to control plane nodes in case kubectl fails. ### During changes - [ ] Change one configuration parameter at a time. - [ ] After each change, wait for etcd pod to become ready: kubectl wait --for=condition=Ready pod/etcd-node1 -n kube-system --timeout=60s - [ ] Check etcd health immediately: etcdctl endpoint health - [ ] Monitor logs: kubectl logs -f etcd-node1 -n kube-system ### After changes - [ ] Verify all etcd members are healthy. - [ ] Verify kube-apiserver can communicate with etcd: kubectl get --raw /healthz?verbose - [ ] Check that cluster operations work: kubectl get nodes , kubectl create deployment test --image=nginx , then delete it. - [ ] Remove temporary backups only after a few days of stability. ### Upgrade-specific checks When upgrading etcd as part of a Kubernetes upgrade: - [ ] Read the Kubernetes upgrade notes for etcd changes. - [ ] Backup etcd and all control plane manifests. - [ ] Upgrade one control plane node at a time. - [ ] After upgrading etcd on a node, verify that the member rejoins the cluster and health is good before proceeding to the next node. - [ ] Do not proceed if any member is unhealthy. Practical Kubernetes check for Operations Checklist: automate these checks where possible using scripts. For example: #!/bin/bash # etcd-health-check.sh ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key endpoint health kubectl get pods -n kube-system -l component=etcd ## Conclusion etcd troubleshooting and configuration changes require careful planning and execution. By following the structured approach in this guide—inventory, safe configuration, verification, and recovery—you can minimize downtime and prevent data loss. Always start with observation, make small reversible changes, and verify each step with concrete commands and expected outputs. Remember to keep backups, document your recovery procedures, and test them before you need them. With these practices, you can maintain a healthy etcd cluster and keep your Kubernetes control plane reliable.