Ceph Backup and Restore with Practical Examples: A Complete Implementation Guide

Introduction

Ceph backup and restore operations demand precision, version awareness, and a clear recovery path before any command touches production data. This guide walks through practical implementations for backing up and restoring Ceph clusters, covering RBD images, CephFS volumes, and cluster configuration state. You will find version-specific commands, expected output patterns, failure signals, and tested recovery procedures.

The audience includes developers running Ceph on Kubernetes via Rook, DevOps consultants managing Proxmox-backed Ceph clusters, and technical startup teams operating self-hosted storage. Every example uses explicit placeholders instead of real identifiers, states prerequisites and blast radius, and includes a verification step with a documented rollback path.

Operational safety principles apply throughout: observe before changing, limit blast radius, protect credentials, verify results, and document recovery before an incident forces the decision.

Version and Environment Inventory

Before any backup or restore operation, capture the exact cluster state. This inventory becomes your reference point for validation and rollback.

Identify Ceph Version and Deployment Topology

Run the following read-only commands on a monitor node or from an admin client with cephadm access:

ceph version
ceph orch host ls
ceph orch ls
ceph osd tree
ceph df

Record the output with timestamps. Example expected output for a Ceph 18.2.2 (Reef) cluster:

ceph version 18.2.2 (6f5c3a8b7c) reef (stable)
HOST          ADDR          LABELS  STATUS
ceph-mon-1    10.0.1.11     mon     running
ceph-mon-2    10.0.1.12     mon     running
ceph-mon-3    10.0.1.13     mon     running
ceph-osd-1    10.0.1.21     osd     running
ceph-osd-2    10.0.1.22     osd     running
ceph-osd-3    10.0.1.23     osd     running

Document Pool and Crush Map State

Capture pool configuration and CRUSH map for restore reference:

ceph osd dump -o /backup/osd_dump_$(date +%F).json
ceph osd getcrushmap -o /backup/crushmap_$(date +%F).bin
crushtool -d /backup/crushmap_$(date +%F).bin -o /backup/crushmap_$(date +%F).txt

Store these artifacts in a separate backup domain (object storage, NFS, or another Ceph cluster). Verify file sizes are non-zero and JSON parses cleanly:

jq empty /backup/osd_dump_$(date +%F).json && echo "OSD dump valid"

Prerequisites Checklist

• Admin keyring (/etc/ceph/ceph.client.admin.keyring) accessible
• cephadm shell or podman/docker for containerized CLI access
• Sufficient space in backup target (estimate 2-3x raw data for full RBD exports)
• Network connectivity between client and monitor/OSD nodes
• Maintenance window approved for any operation that may increase load

Safe Configuration Path: Backup Procedures

This section covers three backup strategies: RBD image export, CephFS snapshot + rsync, and cluster configuration backup. Choose based on RPO/RTO requirements.

RBD Image Backup with rbd export

For block storage (VM disks, databases), rbd export creates a portable image file. Use --no-progress for automation.

# Prerequisites: pool exists, image exists, client has read access
# Blast radius: single RBD image; cluster remains online
# Recovery: rbd import restores to same or different pool

rbd export --no-progress \
  <pool-name>/<image-name> \
  /backup/rbd/<pool-name>_<image-name>_$(date +%F_%H%M).img

# Verify
rbd info <pool-name>/<image-name> | grep -E 'size|objects|order'
ls -lh /backup/rbd/<pool-name>_<image-name>_$(date +%F_%H%M).img

Expected verification: exported file size matches rbd info reported size (within 1% for sparse images). Failure signal: rbd: error: image not found or permission denied on backup mount.

Incremental backup option (Ceph 16.2+): Use rbd export-diff with a prior snapshot:

rbd snap create <pool>/<image>@snap_$(date +%F)
rbd export-diff --from-snap snap_20240115 <pool>/<image>@snap_$(date +%F) /backup/rbd/diff_<image>_$(date +%F).diff

CephFS Backup via Snapshot and Rsync

CephFS supports native snapshots. Combine with rsync for file-level backup to external storage.

# Create snapshot (requires Ceph 15.2+)
mkdir -p /mnt/cephfs/.snap/backup_$(date +%F_%H%M)
# Snapshot appears instantly under .snap directory

# Rsync to backup target (NFS, S3 via rclone, another CephFS)
rsync -avHAX --delete --numeric-ids \
  /mnt/cephfs/.snap/backup_$(date +%F_%H%M)/ \
  /backup/cephfs/backup_$(date +%F_%H%M)/

# Verify
du -sh /mnt/cephfs/.snap/backup_$(date +%F_%H%M)
du -sh /backup/cephfs/backup_$(date +%F_%H%M)

Blast radius: read-only snapshot creation; rsync load depends on change rate. Recovery: rsync back to restored CephFS mount. For Proxmox environments, this integrates with pvesm backup jobs targeting CephFS storage.

Cluster Configuration Backup

Back up mon database, keyrings, and ceph.conf for full disaster recovery:

# On each monitor node
tar -czf /backup/ceph-mon-$(hostname)-$(date +%F).tar.gz \
  /etc/ceph/ \
  /var/lib/ceph/mon/ceph-$(hostname)/store.db \
  /var/lib/ceph/mgr/ceph-$(hostname)/

# Verify
tar -tzf /backup/ceph-mon-$(hostname)-$(date +%F).tar.gz | head -20

Store monitor backups from at least two monitors. For cephadm deployments, also export the spec:

ceph orch ls --export > /backup/cephadm_spec_$(date +%F).yaml

Verification and Diagnostics

Every backup must be verified. This section defines validation procedures and diagnostic commands for common failure modes.

RBD Export Verification

# Check image integrity via rbd info on exported file (requires import to temp pool)
rbd import --no-progress \
  /backup/rbd/<pool>_<image>_$(date +%F_%H%M).img \
  <verify-pool>/verify-<image>-$(date +%F)

rbd info <verify-pool>/verify-<image>-$(date +%F)
rbd diff <verify-pool>/verify-<image>-$(date +%F) | wc -l

# Cleanup
rbd rm <verify-pool>/verify-<image>-$(date +%F)

Expected result: object count matches source, diff shows zero differences from source snapshot. Failure signal: import fails with checksum mismatch or truncated file.

CephFS Rsync Verification

# Compare file counts and sizes
find /mnt/cephfs/.snap/backup_$(date +%F_%H%M) -type f | wc -l
find /backup/cephfs/backup_$(date +%F_%H%M) -type f | wc -l

# Sample checksum verification (1% of files)
find /mnt/cephfs/.snap/backup_$(date +%F_%H%M) -type f -print0 | \
  shuf -z -n $(($(find /mnt/cephfs/.snap/backup_$(date +%F_%H%M) -type f | wc -l) / 100)) | \
  xargs -0 sha256sum > /tmp/src_checksums.txt

find /backup/cephfs/backup_$(date +%F_%H%M) -type f -print0 | \
  xargs -0 sha256sum > /tmp/dst_checksums.txt

# Compare (requires same relative paths)
diff -u /tmp/src_checksums.txt /tmp/dst_checksums.txt

Cluster Health Diagnostics

Run before and after any backup/restore operation:

ceph -s
ceph health detail
ceph pg dump_stuck inactive unclean stale undersized degraded
ceph osd df tree | grep -E 'osd\.[0-9]+' | awk '{if ($5 > 85) print $0}'

Failure signals: HEALTH_ERR, any PGs stuck > 5 minutes, OSD utilization > 85%. These indicate cluster instability that may corrupt backups or prevent restore.

Failure Modes and Recovery

This section maps common failure scenarios to specific recovery procedures with commands.

Scenario 1: RBD Image Corruption or Accidental Deletion

Symptoms: rbd ls <pool> shows missing image; VM reports read errors; rbd info returns "image not found".

Recovery from full export:

# 1. Create target pool if needed (match original pg_num, pgp_num)
ceph osd pool create <restore-pool> <pg_num> <pgp_num> replicated
ceph osd pool set <restore-pool> allow_ec_overwrites true

# 2. Import image
rbd import --no-progress \
  /backup/rbd/<pool>_<image>_<timestamp>.img \
  <restore-pool>/<image>

# 3. Verify
rbd info <restore-pool>/<image>
rbd map <restore-pool>/<image>  # Test mount
rbd unmap /dev/rbd<X>

Recovery from incremental chain (if full export is stale):

# Import base full export
rbd import --no-progress /backup/rbd/<pool>_<image>_base.img <restore-pool>/<image>

# Apply each diff in chronological order
for diff in /backup/rbd/diff_<image>_*.diff; do
  rbd import-diff "$diff" <restore-pool>/<image>
done

Rollback: rbd rm <restore-pool>/<image> if verification fails.

Scenario 2: CephFS Data Loss (Accidental rm -rf)

Symptoms: Directory tree missing; application errors; snapshot still exists.

Recovery:

# 1. Identify latest clean snapshot
ls -lt /mnt/cephfs/.snap/ | head -5

# 2. Restore via rsync from snapshot to active filesystem
rsync -avHAX --delete \
  /mnt/cephfs/.snap/backup_20240115_0200/ \
  /mnt/cephfs/restored_data/

# 3. Verify and swap (atomic rename if same filesystem)
mv /mnt/cephfs/lost_data /mnt/cephfs/lost_data.corrupt
mv /mnt/cephfs/restored_data /mnt/cephfs/lost_data

Blast radius: single directory tree. Cluster remains online.

Scenario 3: Monitor Database Corruption / Quorum Loss

Symptoms: ceph -s hangs or shows mon: 1 daemons, quorum <only-one>; ceph mon dump shows epoch gaps.

Recovery (requires backed-up monitor store.db from healthy peer):

# On failed monitor node (cephadm example)
systemctl stop ceph-mon@<hostname>

# Restore store.db from backup
tar -xzf /backup/ceph-mon-<peer-host>-<date>.tar.gz -C / \
  var/lib/ceph/mon/ceph-<hostname>/store.db

# Fix ownership
chown -R ceph:ceph /var/lib/ceph/mon/ceph-<hostname>/

# Restart
systemctl start ceph-mon@<hostname>

# Verify quorum
ceph -s
ceph mon dump | grep -A5 "monmap"

Critical: Restore from a monitor that was in quorum at backup time. Never restore all monitors from same backup simultaneously — stagger by 30 seconds.

Scenario 4: Full Cluster Disaster Recovery (New Hardware)

Prerequisites: Backed-up osd_dump.json, crushmap.txt, monitor tarballs, cephadm_spec.yaml, RBD/CephFS exports.

Procedure:

1. Deploy new OS nodes with same hostname/IP scheme
2. Bootstrap first monitor from backup:

   cephadm bootstrap --mon-ip <new-ip> --config /backup/ceph.conf \
     --registry-url <registry> --initial-dashboard-user admin \
     --initial-dashboard-password <from-secret-store>

1. Restore monitor store.db on bootstrapped node (as Scenario 3)
2. Apply CRUSH map:

   crushtool -c /backup/crushmap_<date>.txt -o /tmp/crushmap_new.bin
   ceph osd setcrushmap -i /tmp/crushmap_new.bin

1. Recreate pools from osd_dump.json (parse pool_name, pg_num, pgp_num, type)
2. Import RBD images and restore CephFS via rsync
3. Reapply cephadm_spec.yaml for daemons:

   ceph orch apply -i /backup/cephadm_spec_<date>.yaml

Verification: ceph -s shows HEALTH_OK, all PGs active+clean, all daemons running.

Operations Checklist

Use this checklist before, during, and after backup/restore operations.

Pre-Backup

• [ ] Cluster health HEALTH_OK or HEALTH_WARN (no HEALTH_ERR)
• [ ] Backup target mount verified writable with sufficient space (df -h /backup)
• [ ] Admin keyring accessible; ceph -s succeeds
• [ ] Maintenance window communicated; no conflicting operations scheduled
• [ ] Placeholder values replaced in all commands (pool, image, hostname, dates)

During Backup

• [ ] Capture start timestamp in backup log
• [ ] Run backup command with --no-progress for automation
• [ ] Monitor cluster health every 5 minutes (watch -n 300 ceph -s)
• [ ] Verify each artifact immediately after creation (size, checksum, parse test)
• [ ] Record end timestamp and duration

Post-Backup

• [ ] Copy artifacts to secondary domain (off-site, different failure zone)
• [ ] Verify secondary copy integrity
• [ ] Update backup inventory (spreadsheet or CMDB) with locations, sizes, timestamps
• [ ] Prune backups older than retention policy (keep minimum 3 full cycles)
• [ ] Document any anomalies in runbook

Pre-Restore

• [ ] Confirm restore target cluster version matches or exceeds backup version
• [ ] Verify backup artifact integrity (checksums, rbd info, tar -tzf)
• [ ] Isolate target namespace/pool (prevent client writes during restore)
• [ ] Document rollback plan: rbd rm, rsync --delete, monitor restart

Post-Restore

• [ ] Run full verification (Section 3 procedures)
• [ ] Validate application connectivity and data integrity
• [ ] Monitor cluster health for 30 minutes post-restore
• [ ] Update runbook with actual timings and any deviations
• [ ] Notify stakeholders of completion

Conclusion

Ceph backup and restore reliability comes from version-scoped procedures, observable verification steps, and tested recovery paths — not from copying commands without context. This guide covered RBD export and incremental diff chains, CephFS snapshot-plus-rsync workflows, cluster configuration capture, and full disaster recovery on new hardware. Each procedure includes prerequisites, blast radius assessment, expected output, failure signals, and a rollback command.

As a next step, select one low-risk verification: export a single non-production RBD image, import it to a verification pool, compare rbd info output, and clean up. Record the duration and any deviations. Then extend the same pattern to CephFS snapshot validation and monitor store.db restore drills on a staging cluster. A reliable workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision.