Ceph RBD for Proxmox VM storage explained with practical examples: practical implementation guide

Ceph RADOS Block Device (RBD) gives you network-backed block volumes. In Proxmox, each VM disk on a Ceph RBD storage is an RBD image in a pool. This brings thin provisioning, fast snapshots and clones, and shared storage for live migration. This guide explains the moving parts, shows how to create and use pools and images, and gives practical commands to verify health and debug issues.

What you will learn:

How Proxmox maps VM disks to RBD images and pools
How to create a pool, add it to Proxmox, and provision VM disks
How to use RBD snapshots, clones, and rollback safely
Performance basics to avoid common pitfalls
Failure modes to expect and how to check for them before users notice

RBD and Proxmox: the quick mental model

Ceph cluster: monitors (MONs) maintain cluster maps; OSDs store data. Clients talk to MONs for maps, then to OSDs for data.
Pool: a namespace with replication settings and placement rules. VM images live inside a pool.
RBD image: a virtual block device. Proxmox names them like vm-<VMID>-disk-<N>.
Snapshots: point-in-time metadata of an RBD image. Clones can be created from protected snapshots.
Proxmox integration: QEMU/KVM accesses images via librbd (krbd optional). Live migration works because all nodes see the same storage.

Workflow Overview

Plan networking and failure domains

At minimum, isolate Ceph cluster network from client/public traffic (two VLANs or two NICs). Low latency and consistent MTU across nodes matter more than raw bandwidth.
Keep MONs on odd count (3 or 5). Place OSDs across hosts and chassis to avoid correlated failures.

Create a pool for VM images

Choose replicated first (e.g., size=3) for simplicity. Erasure coding can come later.
Start with a modest PG count appropriate for your OSD count. For small labs, 64 or 128 can be fine.

Example commands:

# View cluster health and OSDs
ceph -s
ceph osd tree

# Create a replicated pool for VMs (adjust pg_num to your cluster)
ceph osd pool create vms 64

# Set replication size and confirm
ceph osd pool set vms size 3
ceph osd pool get vms size

Add Ceph RBD storage to Proxmox

If ceph.conf and keyring are deployed to nodes, add RBD storage via GUI or storage.cfg.

storage.cfg example:

# /etc/pve/storage.cfg
rbd: ceph-vms
    pool vms
    content images
    krbd 0
    monitor_host 10.0.0.11;10.0.0.12;10.0.0.13
    username admin
    keyring /etc/pve/priv/ceph.client.admin.keyring

Create a VM and place its disk on the RBD storage

In the Proxmox UI, pick the Ceph storage for the disk. Proxmox creates images like vms/vm-100-disk-0.
Verify from a node:

rbd ls -p vms
rbd info vms/vm-100-disk-0

Use snapshots and clones safely

Snapshots are near-instant. Protect a snapshot before cloning it.

# Create and list snapshots
rbd snap create vms/vm-100-disk-0@pre-upgrade
rbd snap ls vms/vm-100-disk-0

# Roll back if needed (VM must not be running)
rbd snap rollback vms/vm-100-disk-0@pre-upgrade

# Clone from a snapshot
rbd snap protect vms/vm-100-disk-0@gold
rbd clone vms/vm-100-disk-0@gold vms/vm-101-disk-0

Monitor, operate, and maintain

Keep an eye on health, space, and recovery state.

ceph -s
ceph df
ceph health detail
pvesm status

During planned maintenance, consider:

# Prevent rebalancing if you intentionally stop OSDs (use sparingly)
ceph osd set noout
# ...maintenance...
ceph osd unset noout

Practical examples

Example 1: Provision a new pool and attach to Proxmox

Goal: get a working vms pool for non-critical VMs.

# Create pool and set size
ceph osd pool create vms 64
ceph osd pool set vms size 3

# Check placement groups
ceph -s

Add to Proxmox via storage.cfg or GUI. Confirm Proxmox sees the storage: Datacenter -> Storage -> ceph-vms.
Create a small test VM with its disk on ceph-vms. Confirm:

rbd ls -p vms
rbd info vms/vm-101-disk-0

Example 2: Snapshot before a risky change

rbd snap create vms/vm-101-disk-0@before-change
# If rollback needed (power off or ensure disk not in use)
rbd snap rollback vms/vm-101-disk-0@before-change

Example 3: Golden image to many VMs with clones

Convert a prepared VM disk into a golden snapshot, then clone fast:

# Ensure the base image is powered off and consistent
rbd snap create vms/vm-200-disk-0@gold
rbd snap protect vms/vm-200-disk-0@gold
# Create clones for new VMs
rbd clone vms/vm-200-disk-0@gold vms/vm-201-disk-0
rbd clone vms/vm-200-disk-0@gold vms/vm-202-disk-0

After clones are provisioned, consider flattening heavy hitters later to reduce dependency on the parent image:

rbd flatten vms/vm-201-disk-0

Example 4: Quick performance sanity checks

Baseline the pool with a simple rados bench while no VMs are busy:

rados bench -p vms 60 write --no-cleanup
rados bench -p vms 60 seq
rados bench -p vms 60 rand

Check client-side cache usage for an image:

rbd status vms/vm-101-disk-0

Verify network and OSD load distribution:

ceph osd df
ceph osd tree

Performance basics that matter

Network matters first: keep latency low and consistent; verify MTU end-to-end if using jumbo frames.
Replication size: size=3 is common. Never set min_size so low that you risk serving data without sufficient redundancy.
Avoid pool sprawl. Fewer pools with correct PG counts are usually better.
Start with librbd (krbd=0). Consider krbd only for special cases; librbd enables features like live migration smoothly.
Watch for pool fullness: nearfull and full flags will stall writes. Keep headroom (for example, avoid running above ~80% without a plan).
Benchmark with realistic patterns. rados bench is a cluster-level test; for VM-like patterns, fio inside a guest can help, but do not test during production peaks.

Common failure modes and what to look for

OSD down or flapping
Symptoms: degraded data, recovery/backfill active, lower IOPS.
Checks: ceph -s, ceph osd tree, journal or device health.

Monitor quorum loss
Symptoms: clients cannot get cluster maps; severe errors.
Checks: ceph quorum_status, ensure odd MON count and network stability.

Pool full or nearfull
Symptoms: stalled writes; VMs hang on IO.
Checks: ceph df, ceph health detail for full/nearfull. Free space urgently or add capacity.

Network partitions or MTU mismatch
Symptoms: intermittent timeouts, slow requests, many osdmap churn events.
Checks: consistent MTU, ping and iperf between nodes, monitor logs.

Mis-sized PG count
Symptoms: hotspots, uneven OSD usage.
Checks: ceph pg dump, ceph osd df. Adjust with care and plan for backfill impact.

Client mapping issues
Symptoms: VM cannot start or attach disk; AUTH errors.
Checks: keyring permissions, storage.cfg correctness, monitor_host reachability.

Safe operational checks before and after changes

Run this compact checklist before risky changes (like upgrades or taking down nodes):

Health and capacity

ceph -s
ceph df
ceph health detail

Pool redundancy

ceph osd pool get vms size
ceph osd pool get vms min_size

Recovery backlog

ceph -s  # ensure no large recovery/backfill in progress

Proxmox storage visibility

pvesm status | grep ceph-vms

Optional maintenance guardrails

# If you will stop OSDs intentionally
ceph osd set noout
# ... do the work ...
ceph osd unset noout

Local Pilot Plan

Start small, measure, and expand only after passing exit criteria.

Scope

3 Proxmox nodes that also host Ceph MONs and 1-2 OSDs each
1 replicated pool vms with size=3
2 test VMs: one general-purpose, one IO-heavy

Success signals

VM disks run on RBD with stable latency under typical dev load
Live migration completes within your target time window
Snapshots and rollback work and are documented

Measurement plan

Baseline: rados bench on an idle cluster
Load: fio in a guest with simple 4k randrw and 1M seq write tests
Observe: ceph -s during tests, note slow requests or recovery

Exit criteria

No HEALTH_WARN under normal load for at least 48 hours
Pool below 70% usage during tests
Live migration succeeds for both test VMs

Rollout next steps

Add capacity or tune networks if you miss targets
Document standard operating procedures for snapshots and maintenance

Conclusion

Ceph RBD integrates cleanly with Proxmox to provide shared, thin-provisioned VM disks plus fast snapshots and clones. Start with a simple replicated pool, wire it into Proxmox, and validate with a small pilot before scaling out. Keep networks predictable, watch health and capacity, and use snapshots deliberately.

Next actions

Build the vms pool and attach it to Proxmox
Create a test VM, snapshot before changes, and practice rollback
Run the safe operational checks before any maintenance
Expand capacity only after your pilot meets its exit criteria