Ceph storage cluster basics explained with practical examples: practical implementation guide

Intro

Ceph is a distributed, software-defined storage system that provides block, file, and object storage with fault tolerance and horizontal scale. This guide focuses on the practical core: monitors, managers, OSDs, pools, placement groups, and replication. You will also see how Ceph fits with Proxmox VE for virtual machine storage and get a safe pilot plan you can run in a small lab.

Who this helps:

• Developers who need reliable backing storage for CI and services
• DevOps consultants who stand up small-to-medium clusters
• Startup teams building an affordable, resilient storage layer

What you will learn:

• The roles of MON, MGR, OSD, pools, PGs, and replication
• How data is placed and protected
• Simple commands that verify health and create pools
• How to integrate Ceph with Proxmox VE
• A local, low-risk pilot plan you can measure and iterate

Workflow Overview

A clear path reduces surprises and helps you deliver predictable results.

1. Set goals and constraints

• Target workloads: VMs (RBD), files (CephFS), or objects (RGW)
• Capacity, performance, and failure domain expectations

1. Plan the networks

• Public network for client to OSD traffic
• Cluster network for OSD replication and backfill (recommended)
• Aim for at least 10 GbE where possible; keep latency low

1. Prepare nodes and disks

• Uniform disks simplify balancing
• Use dedicated devices or partitions for OSDs
• Ensure NTP, consistent hostnames, and clean drives

1. Deploy monitors (MON) and managers (MGR)

• Typically 3 MONs for quorum in small clusters
• A pair of MGRs for active-standby

1. Create OSDs

• One OSD per data device
• Verify they appear in the OSD tree

1. Define CRUSH rules

• Choose failure domain host or chassis
• Ensure replicas do not land on the same host

1. Create pools and tune PGs

• Start with replicated pools for VM workloads
• Begin with 32 to 128 PGs per pool depending on OSD count

1. Connect clients or Proxmox VE

• For VMs, use RBD-backed storage

1. Test failure and recovery

• Stop an OSD or shut down a host to validate resiliency

1. Observe and iterate

• Watch ceph -s, OSD utilization, latency, and recovery durations

Ceph Core Building Blocks

• Monitors (MON): Maintain the cluster maps (OSD map, monitor map, etc.) and require quorum. With 3 MONs, the cluster can tolerate 1 MON failure.
• Managers (MGR): Provide metrics, dashboards, and service modules. Run at least two; one active, one standby.
• OSDs: Object Storage Daemons that store actual data and metadata on disks, replicate or encode data, and recover when devices fail.
• CRUSH: A placement algorithm that deterministically maps data to OSDs using rules that respect failure domains (host, rack, room).
• Pools: Logical namespaces with policies like replication size and PG count. You can have separate pools for VMs, images, and backups.
• Placement Groups (PGs): Shards of a pool that balance data across OSDs. The CRUSH algorithm maps each PG to a set of OSDs.
• Replication and min_size: Replicated pools keep multiple copies (size). A common starting point is size=3 with min_size=2 so writes can continue during a single OSD or host failure.

Notes for practitioners:

• Replicated pools are preferred for VM workloads due to latency characteristics.
• Erasure coding can reduce capacity overhead for colder data but has higher CPU and write amplification. Use carefully for VM boot disks.
• Avoid oversharding with too many PGs; each PG consumes memory and CPU.

Practical Examples

Example 1: Small 3-node lab for VMs

• Hardware: 3 nodes, each with 2 data drives for OSDs
• Services: 1 MON per node (3 total), 2 MGRs total, 6 OSDs total
• Networks: Separate public and cluster networks if possible

Key checks

• ceph status
• ceph osd tree
• ceph osd df

Create a replicated pool for VMs (RBD)

• Start with 6 OSDs and size=3
• A practical PG count is 64 or 128 for the first pool

Example commands

• Check health: ceph -s
• Create pool with 64 PGs: ceph osd pool create vms 64
• Set replication: ceph osd pool set vms size 3
• Set min_size: ceph osd pool set vms min_size 2
• Confirm placement: ceph osd pool get vms size and ceph osd pool get vms pg_num

PG sizing rule of thumb

• For small clusters, start at 32 to 128 PGs per pool
• Revisit when you add OSDs or pools

Failure testing

• Stop one OSD service on a node and watch ceph -s
• The pool should remain writable with size=3, min_size=2
• After restarting the OSD, confirm recovery completes and health returns to OK

Example 2: Spreading replicas by host

• Ensure your CRUSH rule uses failure domain host
• Verify with ceph osd tree that replicas of the same PG are on different hosts

Using Ceph with Proxmox VE

Proxmox VE integrates well with Ceph to provide shared storage for virtual machines without a separate SAN.

Recommended approach

• Install Ceph on each Proxmox node
• Create 3 MONs across the nodes and at least 2 MGRs
• Prepare disks as OSDs using the Proxmox UI or CLI
• Create a replicated pool, for example vms, with size=3 and min_size=2
• In Proxmox, add a storage of type RBD (Ceph), reference the pool, and verify connectivity

Tips

• Use a CRUSH rule with failure domain host to survive a full node failure
• Keep the Proxmox cluster network and the Ceph public or cluster networks distinct, with predictable MTU and routing
• For live migration and HA, place VM disks on the Ceph RBD storage

Validation steps in Proxmox

• Create a small test VM on the Ceph-backed storage
• Live migrate the VM between nodes
• Simulate an OSD failure and check that the VM keeps running and storage remains available

Local Pilot Plan

Objective

• Deliver a narrow, measurable pilot you can inspect locally before any production move.

Scope

• 3 Proxmox VE nodes running a Ceph cluster for a single RBD pool named vms
• One or two test VMs that exercise read and write paths

Steps

1. Build 3 nodes with 2 data drives each for OSDs. Ensure NTP and consistent networking.
2. Create 3 MONs and 2 MGRs. Prepare and add all data drives as OSDs.
3. Create pool vms with pg_num=64, size=3, min_size=2.
4. Add RBD storage in Proxmox pointing to the vms pool.
5. Create a small VM (for example 10 GB disk) on the Ceph storage.
6. Run a short FIO test in the VM to generate read and write IO.
7. Stop one OSD service. Confirm the VM stays responsive and writes succeed.
8. Restart the OSD. Verify recovery completes and ceph -s returns to HEALTH_OK.

Metrics and exit criteria

• Availability: Writes succeed with one OSD down
• Health: Cluster returns to clean state after recovery
• Latency: Basic read and write latency is acceptable for your workload
• Utilization: OSDs have balanced usage and no single node is overloaded

Risk controls

• Use disposable VMs and data
• Keep clear rollback: remove RBD storage from VMs before changing pool policies

Conclusion

You now have the essentials for a practical Ceph rollout:

• Monitors and managers provide control and visibility
• OSDs hold data and heal from failures
• Pools and PGs define how data is sharded and placed
• Replication with size=3 and min_size=2 is a solid starting point for VMs
• Proxmox VE pairs well with Ceph to enable live migration and HA

Next steps

• Validate your pilot, then scale OSD count and rebalance PGs
• Refine CRUSH rules and network separation as you grow
• Consider erasure coding for colder data where capacity matters

A small, verified pilot builds confidence and informs sizing for production. From there, iterate with measured changes and keep an eye on health, latency, and recovery behavior.