Ceph storage cluster basics with modern best practices and hands-on examples

Intro

Ceph is a distributed, software-defined storage system that delivers block (RBD), file (CephFS), and object (RGW) services with strong fault tolerance and horizontal scale. This practical guide focuses on the core building blocks you actually use: monitors, managers, OSDs, pools, placement groups, and replication. You will also see how to apply current best practices, test real failure scenarios, and integrate Ceph with Proxmox VE. A small pilot plan is included so you can prove resiliency and performance before scaling.

Who this helps:

Developers who need durable, low-maintenance backing storage for CI and services
DevOps engineers building small-to-medium clusters
Startup teams that want resilient storage without a SAN

What you will learn:

The roles of MON, MGR, OSD, pools, PGs, and CRUSH
How data is placed, protected, and rebalanced
Modern defaults: msgr2, BlueStore, autoscaling PGs, and device classes
A step-by-step 3-node lab example with commands
Failure walk-throughs (OSD, host, MON, network, and full-disk conditions)
How to plug Ceph into Proxmox VE for VM storage

Core building blocks (and what matters in practice)

Monitors (MON): Maintain the cluster maps and require quorum. Run three MONs to tolerate one MON failure.
Managers (MGR): Expose metrics, the dashboard, and service modules. Run two MGRs (active/standby).
OSDs: Object Storage Daemons that store data on disks, handle replication or erasure coding, and recover after failures. Use BlueStore (default) with optional SSD/NVMe for DB/WAL to accelerate HDD OSDs.
CRUSH: A deterministic algorithm that places data across failure domains (host, rack, room). Use rules that keep replicas on different hosts at minimum.
Pools: Logical namespaces with policies (replication size, PG count, autoscaler). For VM disks, use a replicated pool with application set to rbd.
Placement Groups (PGs): Shards of a pool that spread data across OSDs. Let the PG autoscaler manage counts in most cases.
Replication and min_size: A common default for VM pools is size=3, min_size=2 (writes continue with one replica missing).

Notes for practitioners:

Prefer replicated pools for VM workloads due to latency; consider erasure coding only for colder data or backups.
Enable the PG autoscaler and the balancer (upmap mode) to keep placement efficient without over-sharding.
Use device classes (hdd/ssd/nvme) and CRUSH rules to keep latency-sensitive pools on faster media.

Plan the cluster and networks

Workloads: Decide which services you need: RBD for VMs, CephFS for shared POSIX files, RGW for S3-compatible objects.
Capacity and performance targets: Estimate usable TB after replication; plan IOPS/latency goals.
Networks: Use a public network for client traffic and a cluster network for OSD replication/backfill. 10 GbE minimum works; 25 GbE is recommended for modern builds. Keep latency low and MTU consistent (jumbo frames only if end-to-end).
Hosts and disks: Keep NTP in sync, hostnames consistent, and firmware current. For HDD OSDs, pair each with a small NVMe partition for BlueStore DB/WAL. Uniform drive sizes simplify balancing.
Services: Start with 3 MONs, 2 MGRs, and at least 3 hosts so CRUSH can spread replicas by host.

Deploy with modern defaults

The following command snippets assume a shell with admin privileges on the cluster. Adjust names as needed.

Key checks:

ceph -s
ceph health detail
ceph osd tree
ceph osd df

Recommended modules and autoscaling:

ceph mgr module enable pg_autoscaler
ceph mgr module enable balancer
ceph balancer mode upmap
ceph balancer on

Create a replicated RBD pool with autoscaler:

# Create a pool and let the autoscaler pick PGs
ceph osd pool create vms
# Set replication policy
ceph osd pool set vms size 3
ceph osd pool set vms min_size 2
# Enable RBD application for features like layering and deep-flatten
ceph osd pool application enable vms rbd
rbd pool init vms

CRUSH and device classes:

# Label devices with classes (typically automatic). Example:
ceph osd crush set-device-class ssd osd.1 osd.2
# Create a host-failure-domain rule over SSDs (if needed)
ceph osd crush rule create-replicated vms-ssd default host ssd
ceph osd pool set vms crush_rule vms-ssd

Tip on PG sizing: Let autoscaler manage PGs. Optionally hint capacity with target ratios so PGs match data size across pools:

ceph osd pool set vms target_size_ratio 0.5

Practical example: a resilient 3-node lab for VMs

Baseline:

3 nodes, each with 2 data devices (6 OSDs total)
3 MONs (one per node), 2 MGRs, 6 OSDs
Separate public and cluster networks if possible

Create and verify:

# Health and topology
ceph -s
ceph osd tree
ceph osd df

# Pool (as above)
ceph osd pool create vms
ceph osd pool set vms size 3
ceph osd pool set vms min_size 2
ceph osd pool application enable vms rbd
rbd pool init vms

# Confirm autoscaler is active
ceph osd pool autoscale-status

Rule of thumb: For small labs, aim for about 100–200 total PGs per OSD across all pools; the autoscaler will converge near safe values as you add data. Avoid manually forcing very high PG counts that increase memory/CPU overhead.

Failure scenarios you should practice

The safest time to learn Ceph’s behavior is in a lab. The commands below create controlled faults and the checks to confirm healthy recovery.

OSD failure (single disk)

Action:

# Identify an OSD ID (e.g., 2) on a non-critical node
systemctl stop ceph-osd@2
# Optional: mark it out if you want data to move quickly off it
ceph osd out 2

What to expect: ceph -s shows HEALTH_WARN with degraded/undersized PGs. Writes continue because size=3, min_size=2.
Recovery: Start the OSD and clear the out flag if you set it.

systemctl start ceph-osd@2
ceph osd in 2
ceph -s

Tuning during recovery (to protect client latency):

ceph config set osd osd_max_backfills 2
ceph config set osd osd_recovery_max_active 2

Full host failure (node offline)

Action: Power off or disconnect one node.
Expectation: Pool stays writable; replicas remain on surviving hosts thanks to failure-domain host. ceph -s shows reduced availability for some PGs until the node returns.
After restoring power, watch backfill complete and ensure HEALTH_OK.

Monitor failure and quorum

Action:

systemctl stop ceph-mon@$(hostname -s)

Expectation: With three MONs, quorum holds (2/3). ceph quorum_status shows which MONs are in quorum. Do not run an even number of MONs.
Recovery: Restart the MON and confirm it rejoins.

Network partition

Action: Disconnect the cluster network interface of one node while keeping the public network up.
Expectation: Client IO may continue, but replication/backfill will stall for that node; health shows stalled peering. This validates the value of separate networks and low-latency links.

Nearfull/full conditions

Thresholds: Typical defaults are nearfull ~85% and full ~95%.
Checks and remediation:

ceph df
ceph health detail
# Add capacity or remove data. Do not ignore FULL states.

Safe maintenance flags

During planned reboots:

ceph osd set noout
# Reboot one node at a time
ceph osd unset noout

Only use norecover/nobackfill temporarily and clear them promptly after work is done.

Using Ceph with Proxmox VE (RBD for VMs)

Proxmox VE integrates tightly with Ceph, letting you run compute and storage on the same nodes while keeping HA and live migration.

Recommended approach:

Install Ceph on each Proxmox node using the Proxmox Ceph management UI.
Create 3 MONs across your nodes and 2 MGRs.
Prepare OSDs (via the UI) using whole disks; for HDD OSDs, choose an NVMe device for DB/WAL if available.
Create a replicated pool named vms with size=3, min_size=2, and enable the PG autoscaler.
In Proxmox, add a storage of type RBD (Ceph) pointing at the vms pool. Enable cephx and msgr2 (default).
Choose the access method: librbd (userspace) is common for snapshots and live migration; kernel RBD (krbd) can be used if required by your environment.

Validation steps:

Create a small test VM on the Ceph-backed storage.
Live-migrate the VM between nodes.
Stop one OSD and confirm the VM continues to run and IO remains available.

Tips:

Keep the Proxmox cluster network and Ceph public/cluster networks distinct with predictable MTU and routing.
Use a CRUSH rule with failure-domain host to survive a full node failure.
Watch the Proxmox Ceph dashboard for latency, OSD usage, and recovery progress.

Local pilot plan (safe, measurable)

Objective: Prove that your 3-node Ceph cluster keeps VMs online through common failures and meets baseline latency.

Scope:

3 Proxmox VE nodes running a Ceph cluster for a single RBD pool vms
1–2 test VMs to exercise read/write paths

Steps:

Build 3 nodes with at least 2 data drives each. Verify NTP, networks, and hostnames.
Deploy 3 MONs and 2 MGRs. Prepare and add all data drives as OSDs.
Create pool vms with autoscaler enabled; set size=3 and min_size=2; enable rbd application.
Add RBD storage in Proxmox pointing to vms and create a small 10 GB test VM.
Run a short IO test inside the VM:

# Example fio inside the VM
fio --name=randrw --filename=/dev/sda \
    --rw=randrw --bs=4k --iodepth=16 --numjobs=1 \
    --time_based --runtime=60 --group_reporting

Stop one OSD service and confirm the VM remains responsive and writes succeed:

systemctl stop ceph-osd@<id>
ceph -s

Restart the OSD and verify recovery completes and HEALTH_OK returns.

Metrics and exit criteria:

Availability: Writes succeed with one OSD down.
Health: Cluster returns to clean state after recovery.
Latency: 4k random read/write latency is acceptable for your workload (track avg and 99th percentile in the dashboard).
Balance: OSD utilization is roughly even; enable the balancer and upmap if skewed.

Risk controls:

Use disposable data and test VMs.
For any host maintenance, set noout before reboot and unset it after.

Conclusion

You have the essentials to build, test, and operate a small but resilient Ceph cluster:

Monitors and managers provide quorum, visibility, and automation.
OSDs store data, heal after failures, and scale horizontally.
CRUSH rules, pools, and PGs determine how data is placed and protected.
Replicated pools with size=3 and min_size=2 are a robust baseline for VM workloads.
The PG autoscaler and balancer reduce manual tuning as you grow.
Proxmox VE integrates cleanly with Ceph to enable shared storage, live migration, and HA.

Validate a small pilot first, capture health and latency baselines, then scale methodically. As you add capacity, keep networks low-latency, enforce clear failure domains, and periodically rehearse failure drills so recovery behavior matches your expectations.