## Intro

etcd is the consistent and highly-available key-value store that powers Kubernetes. It holds the entire cluster state, including every object, configuration, and secret. A healthy etcd is essential for a stable cluster, and misconfiguration or a botched upgrade can take down your entire control plane.

This article provides a hands-on guide to etcd operations for developers, DevOps engineers, and technical teams managing Kubernetes clusters. We will cover the essential commands and procedures you need to inspect, configure, upgrade, and recover etcd safely.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Throughout, replace all placeholder values like <pod-name> or <etcd-pod> with your actual pod names. For a typical kubeadm cluster, the etcd pod often runs as a static pod named `etcd-<master-node-name>` in the `kube-system` namespace.

## Understanding etcd in Kubernetes

In a Kubernetes cluster, etcd is the source of truth for the control plane. The API server reads from and writes to etcd, and every `kubectl` command ultimately interacts with this key-value store. Therefore, etcd performance and health directly impact cluster responsiveness.

etcd typically runs as a static pod managed by the kubelet on each control plane node. In a highly available setup, you may have an external etcd cluster or multiple etcd members spanning control plane nodes. Knowing your etcd topology is the first step before any operation.

Key concepts:

- **Static pod**: A pod defined by a YAML file in `/etc/kubernetes/manifests/` on the control plane node. The kubelet automatically manages it.
- **etcd member**: A single etcd instance. A cluster consists of an odd number of members (usually 3 or 5) for quorum.
- **Raft consensus**: The algorithm etcd uses to maintain a consistent log across members. A majority (quorum) is required for writes.
- **Snapshots**: Consistent backups of the etcd data directory, used for disaster recovery.

Understanding these basics helps you interpret commands and diagnose issues.

## Version and Environment Inventory

Before making any changes, gather information about your etcd installation. This includes the version, cluster health, and configuration.

### Check etcd Version

Use the `etcdctl` binary inside the etcd container or pod.

```bash
# Exec into the etcd pod (assuming it's in kube-system)
kubectl exec -n kube-system etcd-master-node -- etcdctl version
```

Expected output example:

```
etcdctl version: 3.5.10
API version: 3.5
```

If you are on the node with `etcdctl` installed locally, you can run `etcdctl version` directly, but ensure you have the correct certificates and endpoint.

### Verify Cluster Health

List endpoints and check the health of each member.

```bash
# On the control plane node, using etcdctl with the v3 API and peer certificates
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/peer.crt \
  --key=/etc/kubernetes/pki/etcd/peer.key \
  endpoint health
```

For convenience, you can set environment variables or use a wrapper script. Often, kubeadm setups already have the certs at those paths. If etcdctl is not available on the control plane node, run it inside the etcd container using `crictl run`, not via `kubectl exec`. Adjust the endpoint if your etcd listens on a different address.

Expected healthy output:

```
https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 2.5ms
```

### View Cluster Members

```bash
# Run on the control plane node with the v3 API
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  member list
```

Output might look like:

```
8211f1d0a64f3269, started, master-node, https://192.168.1.10:2380, https://192.168.1.10:2379, false
```

This shows the member ID, status, name, peer address, client address, and whether it is a learner.

### Understand Your etcd Deployment

Determine if etcd is embedded as a static pod or external. On a control plane node, check for the manifest:

```bash
ls -l /etc/kubernetes/manifests/etcd.yaml
```

If present, etcd is a static pod. The file contains the pod spec, including the etcd image version, command arguments, and volume mounts for certificates. Review this file to understand configuration details such as data directory, listen URLs, and initial cluster token.

Example snippet from `/etc/kubernetes/manifests/etcd.yaml`:

```yaml
spec:
  containers:
  - command:
    - etcd
    - --advertise-client-urls=https://192.168.1.10:2379
    - --cert-file=/etc/kubernetes/pki/etcd/server.crt
    - --data-dir=/var/lib/etcd
    - --initial-advertise-peer-urls=https://192.168.1.10:2380
    - --initial-cluster=master-node=https://192.168.1.10:2380
    - --key-file=/etc/kubernetes/pki/etcd/server.key
    - --listen-client-urls=https://127.0.0.1:2379,https://192.168.1.10:2379
    - --listen-peer-urls=https://192.168.1.10:2380
    - --name=master-node
    - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt
    - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key
    - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    - --snapshot-count=10000
    - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt
    image: registry.k8s.io/etcd:3.5.10-0
    volumeMounts:
    - mountPath: /var/lib/etcd
      name: etcd-data
    - mountPath: /etc/kubernetes/pki/etcd
      name: etcd-certs
```

This inventory gives you context for any subsequent action.

## Safe Configuration Path

Modifying etcd configuration should be done with extreme caution. A mistake can cause quorum loss and cluster downtime. Follow these principles:

- **Backup first**: Always take a snapshot before changing configuration.
- **Change one setting at a time**: It is easier to isolate problems.
- **Test in a non-production environment first**: If possible, replicate the cluster and try changes there.
- **Document the change and rollback plan**: You should know exactly how to revert.

### Common Configuration Changes

#### Adjusting etcd Log Level

Logging verbosity can be changed to diagnose issues. For example, to increase debug logging temporarily on a static pod, you would edit the manifest file to add `--log-level=debug` to the etcd command arguments. The kubelet restarts the pod automatically.

However, editing static pod manifests directly is risky because the kubelet may not validate the change and could crash the etcd pod repeatedly. A safer approach is to use `kubectl edit` if the etcd is managed by an operator, or to follow a controlled process.

For a kubeadm cluster, the typical process is:

1. **Back up the existing manifest**:
   ```bash
   cp /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests/etcd.yaml.bak
   ```
2. **Edit the manifest** with your desired change:
   ```bash
   vi /etc/kubernetes/manifests/etcd.yaml
   ```
   Add `--log-level=debug` to the `command` list.
3. **Save and exit**. The kubelet detects the change and restarts the etcd pod.
4. **Monitor the pod** to ensure it becomes ready:
   ```bash
   kubectl get pods -n kube-system -w | grep etcd
   ```
   Wait for STATUS to be `Running` and READY `1/1`.
5. If the pod fails to start, revert to the backup:
   ```bash
   cp /etc/kubernetes/manifests/etcd.yaml.bak /etc/kubernetes/manifests/etcd.yaml
   ```
   The kubelet will restore the original pod.

#### Updating Certificates

etcd certificates expire and need rotation. Kubeadm provides a way to renew certs, but the etcd member may need to restart to load new certs. The procedure involves:

- Renewing certs with `kubeadm certs renew etcd-server` and related commands.
- Restarting etcd (often by moving the manifest file temporarily).

We will cover certificate renewal in more detail in the upgrade section.

#### Networking Changes

Changing the etcd listen URL or peer URL is complex and rarely needed. It usually involves reconfiguring the etcd cluster with new member URLs and potentially reinitializing. This should be done only with careful planning and after consulting the etcd documentation.

A safer way is to add a new member with the desired URL and remove the old member.

### Using etcdctl for Configuration

Although etcd itself has few runtime configuration knobs, you can inspect and change certain cluster settings via the etcd API. For example, you can set cluster-wide flags like `--max-request-bytes` at boot time, but not dynamically. So configuration is mostly managed through startup flags.

Always ensure that changes are consistent across all members to avoid misconfiguration.

## Verification and Diagnostics

After any change, verify the etcd cluster is healthy. Routine diagnostics also help catch problems before they escalate.

### Health Checks

Run endpoint health and member list as shown earlier. Additionally, check the etcd logs for errors:

```bash
kubectl logs -n kube-system etcd-master-node --tail=50
```

Look for warnings such as "failed to send out heartbeat on time" or "leader changed" patterns, which may indicate network issues or heavy load.

### Checking Quorum

If the cluster has multiple members, ensure a majority are healthy. For a 3-member cluster, at least 2 members must be up. Use:

```bash
kubectl exec -n kube-system etcd-master-node -- etcdctl endpoint status --cluster -w table
```

This prints a table with endpoint, ID, version, db size, leader, raft term, etc. Verify that all endpoints report healthy and that one leader is elected.

Example output:

```
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|          ENDPOINT           |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.1.10:2379   | 8211f1d0a64f3269 |  3.5.10 |   25 MB |      true |      false |         2 |       4567 |               4567 |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
```

### Monitoring Metrics

etcd exposes Prometheus metrics on the client port. You can scrape these to track performance. Useful metrics include:

- `etcd_server_has_leader` (should be 1)
- `etcd_server_leader_changes_seen_total` (frequent changes indicate instability)
- `etcd_disk_backend_commit_duration_seconds` (high latency may suggest disk I/O issues)
- `etcd_network_peer_round_trip_time_seconds` (slow peer RTTs can affect quorum)

Example query to check leader:

```bash
curl -s https://127.0.0.1:2379/metrics | grep etcd_server_has_leader
```

Expected output:

```
etcd_server_has_leader 1
```

### Testing Performance

Use `etcdctl check perf` to run a simple performance test against the cluster. This command writes and reads a number of keys and reports latency.

```bash
kubectl exec -n kube-system etcd-master-node -- etcdctl check perf --endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key
```

Sample output:

```
60 / 60 Booooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooooo! 100.00%1m0s
PASS: Throughput is 150 writes/s
Slowest request took too long (1.2s)
```

This helps identify if etcd is underperforming.

## Failure Modes and Recovery

etcd failures can be catastrophic if not handled correctly. Common failure modes include:

- **Quorum loss**: More than (n-1)/2 members down, cluster becomes read-only or unavailable.
- **Data directory corruption**: File system errors or disk failure.
- **Certificate expiration**: TLS errors prevent connectivity.
- **Operator error**: Accidental deletion of keys or bad configuration.

### Backup and Restore

Regular backups are your safety net. Use `etcdctl snapshot save` to create a point-in-time backup.

```bash
ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/lib/etcd/snapshot.db
```

This saves a snapshot to the specified file on the control plane node. You should copy it to a safe location outside the cluster.

To restore from a snapshot, you need to stop the etcd pod, restore the data directory, and optionally start a new cluster. This is typically done on a single-member etcd or when rebuilding a cluster.

For a static pod, you can move the manifest to stop the pod:

```bash
mkdir -p /etc/kubernetes/manifests-backup
mv /etc/kubernetes/manifests/etcd.yaml /etc/kubernetes/manifests-backup/
```

Wait for the pod to terminate. Then, on the node, run `etcdctl snapshot restore` with appropriate flags to regenerate the data directory. Be careful to specify a new data directory or clean the existing one.

Example restore command (run on the node as root with etcdctl binary):

```bash
etcdctl snapshot restore /path/to/snapshot.db \
  --name master-node \
  --initial-cluster master-node=https://192.168.1.10:2380 \
  --initial-advertise-peer-urls https://192.168.1.10:2380 \
  --data-dir /var/lib/etcd-restore
```

Then update the etcd manifest to point `--data-dir` to the new directory and restore the manifest.

**Caution**: Restoring from snapshot will overwrite the current cluster state and should be done only when necessary.

### Recovering from Quorum Loss

Quorum loss can happen if multiple nodes go down simultaneously. To recover, you may need to force a new cluster using the members that are still available. This involves running `etcd --force-new-cluster` on a single member to create a new single-node cluster, then adding the other members back.

Example of forcing a new cluster (on a surviving member):

```bash
# Stop etcd and run manually with force-new-cluster flag (simplified)
etcd --data-dir=/var/lib/etcd \
  --force-new-cluster \
  --name=master-node \
  --initial-cluster master-node=https://192.168.1.10:2380 \
  --initial-advertise-peer-urls https://192.168.1.10:2380 \
  --listen-peer-urls https://192.168.1.10:2380 \
  --listen-client-urls https://127.0.0.1:2379,https://192.168.1.10:2379 \
  --advertise-client-urls https://192.168.1.10:2379
```

After the member is up and running as a single-node cluster, you can add new members and restore redundancy.

### Handling Certificate Expiration

If etcd certificates expire, the etcd pod may crash loop. To fix, renew the certificates using kubeadm if applicable:

```bash
kubeadm certs renew etcd-server
kubeadm certs renew etcd-peer
kubeadm certs renew etcd-healthcheck-client
```

Then restart etcd by moving the manifest and moving it back:

```bash
mv /etc/kubernetes/manifests/etcd.yaml /tmp/
mv /tmp/etcd.yaml /etc/kubernetes/manifests/
```

Wait for the pod to become ready.

If kubeadm is not available, you need to manually generate certificates using your CA and update the etcd configuration.

## Upgrading etcd

Upgrading etcd should be carefully planned. In Kubernetes, the etcd version is tied to the control plane version but may be upgraded independently in some cases.

### Pre-upgrade Checklist

- Ensure all etcd members are healthy.
- Take a snapshot.
- Check the current etcd version and target version compatibility.
- Review the Kubernetes release notes for etcd changes.
- Plan for downtime if rolling upgrade is not possible.

### Upgrade Process for kubeadm Managed Clusters

For clusters deployed with kubeadm, etcd is often upgraded as part of the Kubernetes upgrade process. The standard steps are:

1. Upgrade the kubeadm tool.
2. Run `kubeadm upgrade plan` to see available versions.
3. Run `kubeadm upgrade apply vX.Y.Z` which upgrades the control plane components, including etcd.

However, you can manually upgrade etcd by editing the etcd manifest to change the image version. For example, to upgrade from 3.5.10 to 3.5.11:

```bash
# On the control plane node
export ETCD_VERSION=3.5.11-0
sed -i "s/registry.k8s.io\/etcd:.*/registry.k8s.io\/etcd:${ETCD_VERSION}/" /etc/kubernetes/manifests/etcd.yaml
```

The kubelet will pull the new image and restart etcd automatically. Monitor the pod status and etcd health after the restart.

### Rolling Upgrade of etcd Members

If you have an external etcd cluster or multiple members, you can perform a rolling upgrade. The process is:

1. Upgrade one member at a time.
2. For each member, stop it, replace the binary or update the container image, and start it.
3. Wait for the member to rejoin the cluster and become healthy before moving to the next.
4. Verify the cluster health after each member.

This minimizes downtime and allows the cluster to maintain quorum during the upgrade.

### Post-upgrade Verification

After upgrading, run the health checks and version commands to confirm the new version is active and the cluster is healthy. Also, check the Kubernetes API server logs to ensure it can communicate with the new etcd.

## Common Pitfalls and How to Avoid Them

Working with etcd can be tricky. Here are some common pitfalls and how to avoid or recover from them.

### Pitfall 1: Using Wrong Certificate Paths

Many operators copy commands from documentation without adjusting the certificate paths to their environment. This leads to authentication errors.

**How to avoid**: Always verify the actual paths inside the etcd pod. You can list the mounted volumes:

```bash
kubectl exec -n kube-system etcd-master-node -- ls /etc/kubernetes/pki/etcd
```

Typical files: `ca.crt`, `server.crt`, `server.key`, `peer.crt`, `peer.key`. If in doubt, inspect the pod's volume mounts from the manifest.

**Recovery**: If you get TLS errors, check if the certificates are expired or missing. Renew if necessary.

### Pitfall 2: Snapshot Restore on a Live Cluster

Restoring a snapshot while etcd is still running can corrupt data and cause split-brain. The restore process assumes the data directory is not in use.

**How to avoid**: Always stop the etcd member before restoring. For static pods, move the manifest file away and ensure the pod is terminated. Back up the current data directory before overwriting.

**Recovery**: If you accidentally restored on a live member, stop it immediately and re-restore from a clean snapshot.

### Pitfall 3: Ignoring Quorum Requirements

Assuming a 3-member etcd can tolerate losing two members. In fact, a 3-member cluster can only survive one member failure. Losing two means no quorum.

**How to avoid**: Understand the quorum formula: a cluster of N members requires (N/2)+1 members to be available. For 3, that's 2. For 5, it's 3. Plan your topology accordingly and never stop more than one member at a time during maintenance.

### Pitfall 4: Not Backing Up Regularly

Many teams only think about backups after a disaster. Regularly automated backups are essential.

**How to avoid**: Schedule `etcdctl snapshot save` via cron job or use a Kubernetes CronJob to take periodic snapshots and store them outside the cluster (e.g., in object storage). Test your restore process periodically to ensure backups are valid.

### Pitfall 5: Changing Configuration Without Testing

Editing the etcd manifest directly on a production cluster without testing can cause repeated crash loops and extended downtime.

**How to avoid**: Use a staging cluster or at least a development environment to test changes. Document the exact change and rollback procedure. If possible, use configuration management tools to apply changes uniformly.

## Operations Checklist

This checklist is a quick reference for safe etcd operations. Assign each item to a specific owner (not a team) and revisit the checklist monthly or after any major cluster event.

### Pre-Operation

- [ ] **Owner**: Priya Shah, Engineering Lead - Verify etcd cluster health with `etcdctl endpoint health`.
- [ ] **Owner**: Alex Chen, DevOps Engineer - Take a snapshot with `etcdctl snapshot save` and store it securely outside the cluster.
- [ ] **Owner**: Alex Chen - Check current versions and note compatibility.
- [ ] **Owner**: Priya Shah - Document the change, expected outcome, and rollback steps in a runbook.
- [ ] **Owner**: Alex Chen - Notify stakeholders if downtime is possible.

### During Operation

- [ ] **Owner**: Alex Chen - Execute the change on one member at a time (if applicable).
- [ ] **Owner**: Alex Chen - Monitor etcd pod logs for errors: `kubectl logs -n kube-system etcd-<node> -f`
- [ ] **Owner**: Priya Shah - Verify cluster health after each step.
- [ ] **Owner**: Alex Chen - If any step fails, follow the rollback plan immediately.

### Post-Operation

- [ ] **Owner**: Alex Chen - Confirm all members are healthy and quorum is intact.
- [ ] **Owner**: Priya Shah - Run `etcdctl endpoint status --cluster` to check versions and leader.
- [ ] **Owner**: Alex Chen - Verify Kubernetes API server is responsive and cluster workloads are scheduling normally.
- [ ] **Owner**: Priya Shah - Record the operation details, actual outcomes, and any deviations in the runbook.
- [ ] **Owner**: Alex Chen - Schedule a review of the operation in one week to catch any delayed issues.

### Quarterly Review

The checklist itself should be reviewed quarterly by the Engineering Lead (Priya Shah) to incorporate lessons learned and changes to the cluster environment.

## Conclusion

etcd is the heart of Kubernetes, and operating it safely requires a methodical approach. By following the procedures outlined in this article, you can confidently inspect, configure, upgrade, and recover etcd without causing unnecessary downtime.

Remember: always observe before changing, take backups, limit the blast radius, verify health at every step, and have a recovery plan. With these practices, you will keep your Kubernetes control plane stable and your data safe.

As a next step, pick one low-risk verification like checking `etcdctl endpoint health` and doing a snapshot save, and put it into action. Document your cluster's specific parameters into an operational runbook for your team.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.