Kubernetes Backup and Restore with Practical Examples

A reliable Kubernetes backup and restore strategy protects not only stateless manifests, but also persistent data and, when you manage the control plane, etcd state. This article provides a practical path you can execute safely: build a precise environment inventory so your choices fit your cluster, start with a narrow pilot in a single namespace, prove it works, and expand. Back up manifests, persistent volumes, and optionally etcd with concrete, tested steps. Restore in the right order, verify success, and handle failure modes with confidence. Use a short checklist to make this part of regular operations. The goal is predictable restores with minimal downtime and no surprises.

Version and Environment Inventory

Before you run the first backup, write down your versions, topology, and storage capabilities. This informs compatible tools and the restore sequence.

• Kubernetes version and distribution (managed cloud vs. self-managed):

  kubectl version --short
  kubectl cluster-info

• Nodes and architecture:

  kubectl get nodes -o wide

• Namespaces and high-level footprint:

  kubectl get ns
  kubectl get deploy,sts,ds,svc,ingress -A --no-headers | wc -l

• API resources and CRDs (needed to restore custom resources):

  kubectl api-resources | sort
  kubectl get crd | wc -l
  kubectl get crd -o name | head -n 10

• StorageClasses and CSI snapshot support (for PV snapshots):

  kubectl get storageclass
  kubectl get volumesnapshotclass.snapshot.storage.k8s.io 2>/dev/null || echo "No CSI snapshot classes detected"

• If you manage etcd (self-managed control plane): record etcd version and TLS paths on control-plane nodes:

  sudo ETCDCTL_API=3 etcdctl version
  ls /etc/kubernetes/pki/etcd

Use the following compact inventory as a reference during design and restore.

Safe Configuration Path

Backups must cover three layers, chosen per your environment:

1. Manifests and configuration: Namespaces, Deployments, StatefulSets, Services, Ingress, ConfigMaps, Secrets, RBAC, and custom resources.
2. Persistent data: PVC-backed volumes, ideally through CSI snapshots or application-aware dumps when snapshots are not available.
3. Control-plane state (optional): etcd snapshots for self-managed clusters.

Start with a pilot in one namespace that is narrow, measurable, and easy to inspect. Expand only after you can restore it reliably.

Pilot 1: Namespace Manifest Export (No Data)

Target: a single namespace with stateless or externally stored data (e.g., cache-only). Expected duration: minutes.

NS=myapp
# Namespace definition
kubectl get namespace "$NS" -o yaml > ns-$NS.yaml

# Core namespaced resources
kubectl get all -n "$NS" -o yaml > $NS-manifests.yaml

# Common non-"all" resources
kubectl get cm,secret,role,rolebinding,serviceaccount,ingress -n "$NS" -o yaml >> $NS-manifests.yaml

# If your app uses CRDs, add them explicitly (example kinds)
# kubectl get kafka,elasticsearch,redis -n "$NS" -o yaml >> $NS-manifests.yaml || true

# Cluster-scoped dependencies (only if you own them)
kubectl get crd -o yaml > cluster-crds.yaml

Store the files with a timestamp and integrity hash:

tar -czf backup-$NS-$(date +%F).tgz ns-$NS.yaml $NS-manifests.yaml cluster-crds.yaml
sha256sum backup-$NS-*.tgz > backup-$NS-SHA256SUMS.txt

Pilot 2: Add Persistent Volume Snapshots (CSI)

If your StorageClass supports CSI snapshots, create VolumeSnapshot objects in the same namespace. Replace values to match your environment.

Create a VolumeSnapshot:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: myapp-pvc-snap-20260101
  namespace: myapp
spec:
  volumeSnapshotClassName: csi-snapclass
  source:
    persistentVolumeClaimName: myapp-data

Apply and verify:

kubectl apply -f myapp-pvc-snapshot.yaml
kubectl -n myapp get volumesnapshot
kubectl -n myapp describe volumesnapshot myapp-pvc-snap-20260101 | sed -n '/Status/,$p'

ReadyToUse should become true. Record the snapshot handle if the driver exposes it.

Restore from the snapshot into a test namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: myapp-restore
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: myapp-data-restore
  namespace: myapp-restore
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: gp2
  resources:
    requests:
      storage: 20Gi
  dataSource:
    name: myapp-pvc-snap-20260101
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

Apply and confirm the PVC binds:

kubectl apply -f myapp-restore-ns-and-pvc.yaml
kubectl -n myapp-restore get pvc myapp-data-restore -w

Then deploy a test Pod or Deployment that mounts the restored PVC and run a simple data check.

Optional: etcd Snapshot (Self-Managed Control Plane)

If you operate your own control plane, take an etcd snapshot from a control-plane node. Paths may differ by distribution.

sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /backup/etcd-$(date +%F-%H%M%S).db

sudo ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-*.db -w table

Store snapshots off the node. Restoring etcd replaces cluster state; practice only on non-production first.

All-in-One Alternative: Velero Backup and Restore

Velero can orchestrate manifest backups and PV snapshots with provider plugins. Example (adjust provider, plugin, and storage configuration):

velero install \
  --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.8.0 \
  --bucket my-velero-bucket \
  --backup-location-config region=us-east-1 \
  --snapshot-location-config region=us-east-1

# Create a namespace-scoped backup including volume snapshots
velero backup create myapp-20260101 \
  --include-namespaces myapp \
  --ttl 168h \
  --snapshot-volumes

# Inspect details
velero backup describe myapp-20260101 --details

Restore from a Velero backup:

velero restore create --from-backup myapp-20260101
velero restore get
velero restore describe <restore-name>

Constructed Pilot Plan (Example)

The following table is an illustrative plan with hypothetical numbers for sizing and timing.

Verification and Diagnostics

Prove backups and restores with objective checks before relying on them.

Validate Manifest Backups

• Server-side dry-run against a test cluster or the same cluster:

  kubectl apply -f ns-myapp.yaml --dry-run=server
  kubectl apply -f myapp-manifests.yaml --dry-run=server

• Count resources before and after restore:

  # Before (capture expected counts)
  kubectl -n myapp get deploy,sts,svc,cm,secret,pvc --no-headers | wc -l
  # After restore into myapp-restore (compare)
  kubectl -n myapp-restore get deploy,sts,svc,cm,secret,pvc --no-headers | wc -l

Validate PV Snapshots and Data

• Ensure VolumeSnapshot ReadyToUse is true:

  kubectl -n myapp get volumesnapshot myapp-pvc-snap-20260101 -o jsonpath='{.status.readyToUse}{"\n"}'

• Create a test Pod mounting the restored PVC and verify file presence or checksums:

  kubectl -n myapp-restore exec deploy/api -- sh -c 'ls -al /data | head -n 20'
  # Optionally, compare hashes for a small set of files
  kubectl -n myapp-restore exec deploy/api -- sh -c 'sha256sum /data/*.idx 2>/dev/null'

Health and Events

• Pod readiness and restarts:

  kubectl -n myapp-restore get pods -o wide
  kubectl -n myapp-restore get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.containerStatuses[0].restartCount}{"\n"}{end}'

• Recent events for failures:

  kubectl -n myapp-restore get events --sort-by=.lastTimestamp | tail -n 30

Velero Diagnostics (If Used)

velero backup logs myapp-20260101 | tail -n 50
velero restore describe <restore-name> --details

Failure Modes and Recovery

Plan for what breaks. Below are common issues, symptoms, and recoveries.

Symptom: apply errors like "no matches for kind X in version Y". Fix: apply CRDs first, then custom resources. Keep a separate CRD backup file and restore it before namespaced manifests.

• Missing CRDs before CRs

Symptom: resources fail because the Namespace does not exist yet, or existing namespace contains conflicting objects. Fix: create Namespace first. For test restores, use a fresh namespace (e.g., myapp-restore). To roll back, delete only the test namespace.

• Namespace collisions or wrong order

Symptom: PVC from snapshot stays Pending; events show no suitable volume found. Fix: use a StorageClass compatible with the snapshot driver. Verify VolumeSnapshotClass and StorageClass mapping; consult your CSI driver docs for cross-zone or type constraints.

• Snapshot class or StorageClass mismatch

Symptom: pods start but fail to connect to dependencies; env or files differ from production. Fix: ensure secrets/config are included in the backup, or regenerate them in the restore namespace. For generated secrets, back up their source or recreate securely.

• Secret and ConfigMap drift

Symptom: controllers fail with Forbidden errors after restore. Fix: restore Role, RoleBinding, ClusterRole, ClusterRoleBinding, and ServiceAccount in the right scope. Confirm subjects and names are unchanged.

• RBAC gaps

Symptom: app errors after restore though the PVC mounted; logical corruption. Fix: prefer application-consistent backups (pre/post hooks or database-native dumps) if your storage snapshots are not crash-consistent for the workload.

• Application consistency on writable databases

Symptom: cluster-wide regressions if etcd restored to mismatched cluster config. Fix: practice on non-production first. Match name, initial-cluster, peer URLs, and data-dir with your original static pod manifests.

• etcd restore hazards (self-managed only)

Rollback and Recovery Patterns

If a restore test causes issues, delete only the test namespace:

• Namespaced restore rollback

  kubectl delete ns myapp-restore --wait=false

No changes leak into the original namespace.

If a restored PVC is wrong, delete the PVC (not the VolumeSnapshot) and recreate from the snapshot with corrected StorageClass or size.

• PVC rollback

Keep the previous /var/lib/etcd directory as a backup before restore. If the restore fails, stop etcd, swap back the directory, and start again.

• etcd rollback (self-managed)

Controlled etcd Restore (Outline)

Only for self-managed control planes; follow your distribution specifics.

# On a control-plane node
sudo systemctl stop kube-apiserver etcd
sudo mv /var/lib/etcd /var/lib/etcd.bak.$(date +%s)

# Restore from a known-good snapshot (fill placeholders to match your cluster)
sudo ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-YYYYMMDDHHMM.db \
  --data-dir=/var/lib/etcd \
  --name=<etcd-node-name> \
  --initial-cluster=<etcd-node-name>=https://127.0.0.1:2380 \
  --initial-advertise-peer-urls=https://127.0.0.1:2380

sudo systemctl start etcd kube-apiserver

Confirm API health after etcd starts:

kubectl get --raw=/healthz

Operations Checklist

Run this list on a schedule (e.g., weekly) and after significant changes.

• Inventory
• Record kubectl version, cluster-info, nodes.
• Export current StorageClasses and VolumeSnapshotClasses.
• List CRDs you depend on.

• Back up
• Export namespace manifests and common resources to YAML.
• If applicable, back up CRDs (cluster-scoped) you own.
• For PVCs, create CSI VolumeSnapshots for key volumes.
• If self-managed control plane, take an etcd snapshot.

• Verify
• Run kubectl apply --dry-run=server on exported YAML.
• Ensure snapshots show ReadyToUse=true.
• Log backup artifact names, hashes, and locations.

• Store
• Push artifacts to durable, access-controlled storage.
• Keep at least one offsite or cross-region copy.

• Test restore
• Use a fresh namespace (e.g., myapp-restore).
• Apply CRDs, then namespace and manifests.
• Restore PVCs from snapshots and mount in test pods.
• Confirm pods Ready, low restarts, and basic data checks.

• Review and improve
• Capture time-to-restore, errors, and missing dependencies.
• Expand scope to additional namespaces only after clean tests.

Conclusion

You now have a practical, low-risk path to Kubernetes backups and restores. Start small with a namespace manifest export, prove you can restore it, and add persistent data via CSI snapshots where available. If you manage your own control plane, include etcd snapshots and practice restores in non-production. Verify with dry-runs, counts, events, and simple data checks. Treat test restores as regular drills. Document failure modes and rollback steps so that a bad restore is easy to undo. Next steps: schedule the checklist, automate exports and snapshot creation, and expand coverage to additional namespaces and critical volumes. As your confidence grows, document a full disaster recovery plan that includes cluster bootstrap, CRD installation, and ordered restoration of workloads and data. With these steps in place, you can meet recovery objectives predictably and repeatedly.