Ceph local lab setup with practical examples: practical implementation guide

A local Ceph lab is the safest place to learn distributed storage hands-on, rehearse changes, and troubleshoot without risking production data. In this guide, you will build a practical three-node Ceph cluster on local virtual machines using a package-based installation (no containers), create RBD and CephFS for concrete exercises, verify cluster health with observable checks, and practice recovery from common failure modes. The end result is a repeatable workflow you can rebuild and extend as your understanding grows.

Constructed example scope used in this guide:

• Three VMs (ceph1, ceph2, ceph3) with one system disk and one data disk each
• Replicated pools with size 3 to see real quorum and placement behavior
• RBD and CephFS examples to practice block and filesystem workflows

Version and Environment Inventory

Choose simple, consistent versions and a topology that fits a laptop or workstation. Keep the first pilot small, measurable, and easy to inspect.

Constructed example (adjust to your host capacity):

• OS: Ubuntu Server 22.04 LTS on all nodes
• Ceph release: example Reef 18.x (package-based, no containers)
• CPU and RAM per VM: 2 vCPU, 4-8 GB RAM
• Disks per VM: 1 x OS (40 GB), 1 x data (50-200 GB)
• Network: one private network segment (example 192.168.56.0/24)
• Hostnames and IPs (example): ceph1 192.168.56.11, ceph2 192.168.56.12, ceph3 192.168.56.13

Constructed example lab inventory:

Notes:

• Three monitors (one per node) form a proper quorum. Avoid single-mon labs if you want to practice real failover.
• Use dedicated data disks for OSDs. Do not place OSD data on the OS disk.
• Time sync is mandatory. Enable NTP or chrony on all nodes.

Safe Configuration Path

This section provides a clear, reversible path to a working lab. All destructive commands are labeled; double-check device names before running anything that alters disks.

Prerequisites on all three nodes (run as a user with sudo):

• Set hostnames and ensure forward and reverse name resolution (or static /etc/hosts entries) across nodes.
• Ensure time sync:
• sudo apt update && sudo apt install -y chrony
• sudo systemctl enable --now chrony
• Disable swap (Ceph strongly prefers no swap):
• sudo swapoff -a
• Comment any swap lines in /etc/fstab
• Open firewall or disable it within the lab network. For quick labs on Ubuntu, you can allow common Ceph traffic or temporarily disable UFW within the isolated lab network:
• sudo ufw disable

Prepare an admin node

• Use ceph1 as the admin node. Generate an SSH key and copy it to ceph1, ceph2, ceph3 for passwordless access:
• ssh-keygen -t ed25519 -C "ceph-admin"
• ssh-copy-id user@ceph1; ssh-copy-id user@ceph2; ssh-copy-id user@ceph3
• Create a working directory:
• mkdir -p ~/ceph-lab && cd ~/ceph-lab
• Install ceph-deploy on the admin node (package-based install, no containers):
• sudo apt update && sudo apt install -y ceph-deploy

Initialize the cluster configuration (constructed example addresses)

• Create a new cluster with three monitors:
• ceph-deploy new ceph1 ceph2 ceph3
• Edit the generated ceph.conf in ~/ceph-lab and add lines that match your network:
• public_network = 192.168.56.0/24
• osd_pool_default_size = 3
• osd_pool_default_min_size = 2
• mon_allow_pool_delete = true # lab only; do not use in production

Install Ceph packages on all nodes

• ceph-deploy install ceph1 ceph2 ceph3
• Optionally pin a release supported by your distro packages.

Create the initial monitors and gather keys

• ceph-deploy mon create-initial
• Distribute admin keys so all nodes can run ceph commands:
• ceph-deploy admin ceph1 ceph2 ceph3
• On each node, verify /etc/ceph/ceph.client.admin.keyring exists and is readable by your user or via sudo.

Deploy a manager daemon

• ceph-deploy mgr create ceph1

Prepare and create OSDs (DESTRUCTIVE: zaps target device)

• Ensure each node has a dedicated empty disk (example /dev/sdb). Double-check with lsblk.
• For each node, run one of the following from the admin directory:
• ceph-deploy osd create --data /dev/sdb ceph1
• ceph-deploy osd create --data /dev/sdb ceph2
• ceph-deploy osd create --data /dev/sdb ceph3

Optional: Deploy a metadata server for CephFS later

• ceph-deploy mds create ceph1

At this point, the cluster should have 3 mons, 1 mgr, and 3 OSDs up/in.

Create a small RBD for hands-on testing

• Create a pool and enable the application:
• sudo ceph osd pool create rbd 32 32 # constructed example PG counts
• sudo ceph osd pool application enable rbd rbd
• Create and map an image:
• rbd create --size 512M rbd/labimg
• sudo rbd map rbd/labimg --name client.admin
• sudo mkfs.ext4 /dev/rbd0
• sudo mkdir -p /mnt/rbd && sudo mount /dev/rbd0 /mnt/rbd
• Quick write test and cleanup:
• dd if=/dev/zero of=/mnt/rbd/testfile bs=4M count=16 status=progress
• ls -lh /mnt/rbd

Create a minimal CephFS for directory-based testing

• Create data and metadata pools and a filesystem (constructed example PG counts):
• sudo ceph osd pool create cephfs_data 32 32
• sudo ceph osd pool create cephfs_metadata 8 8
• sudo ceph fs new cephfs cephfs_metadata cephfs_data
• Mount via kernel client (uses mon addresses or names resolvable in your lab):
• sudo mkdir -p /mnt/cephfs
• sudo mount -t ceph ceph1, ceph2, ceph3:/ /mnt/cephfs -o name=admin, secretfile=/etc/ceph/ceph.client.admin.keyring
• Quick directory test:
• sudo mkdir -p /mnt/cephfs/lab && sudo sh -c 'echo hello > /mnt/cephfs/lab/greeting.txt'
• sudo ls -l /mnt/cephfs/lab

Verification and Diagnostics

Use these checks to confirm the cluster is healthy and that data paths work as expected.

Constructed example verification matrix:

Expected results (constructed examples):

• ceph -s prints something like: health: HEALTH_OK; mon: 3 daemons, quorum ceph1, ceph2, ceph3; osd: 3 up, 3 in; mgr: active: ceph1.
• dd to /mnt/rbd completes and file appears with ~64 MiB size when bs=4M count=16.
• /mnt/cephfs/lab/greeting.txt contains "hello".

Useful diagnostics during experiments:

• Track placement group state during writes:
• watch -n2 "ceph pg stat"
• Inspect OSD logs for an individual daemon (paths vary by distro, constructed example):
• sudo journalctl -u ceph-osd@0 -f
• Confirm MDS state for CephFS:
• ceph fs status

Failure Modes and Recovery

Practice these in the lab to build intuition. Always note the cluster state before and after.

1. OSD disk failure (simulated by stopping an OSD or powering off a node)

• Trigger: shut down ceph2 or stop its OSD:
• sudo systemctl stop ceph-osd@<id> on ceph2
• Observation:
• ceph -s shows degraded data redundancy, PGs remapped, backfilling may start.
• Recovery path:
• If transient, start the OSD: sudo systemctl start ceph-osd@<id>
• If disk lost, mark OSD out, remove, replace disk, and recreate OSD:
• ceph osd out <id>
• ceph osd purge <id> --yes-i-really-mean-it
• Prepare replacement disk and run ceph-deploy osd create --data /dev/newdisk ceph2
• Verification:
• ceph -s returns to HEALTH_OK and PGs active+clean.

1. Monitor quorum loss (do not do this with less than 3 mons)

• Trigger: stop two monitor daemons (for demonstration only). Expect the cluster to become unavailable for writes.
• Safe practice approach:
• Stop only one mon to see quorum shrink to 2/3:
• sudo systemctl stop ceph-mon@ceph2
• Observation:
• mon: 2 daemons in quorum; clients continue to work.
• Recovery path:
• Restart the mon and verify quorum returns to 3:
• sudo systemctl start ceph-mon@ceph2
• Verification:
• ceph quorum_status

1. Nearfull or full cluster (constructed example; be careful not to actually fill your host disk)

• Trigger: write large data to /mnt/rbd until nearfull warning appears.
• Observation:
• ceph health detail shows nearfull/full warnings and blocked writes when full.
• Recovery path:
• Delete test data, optionally increase capacity by adding an OSD, or lower replication temporarily in the lab (never compromise production durability):
• ceph osd pool set rbd size 2 # lab only, revert to 3 after testing
• Verification:
• Health returns to OK; writes succeed again.

1. Misplaced or undersized PGs after topology changes

• Trigger: add or remove an OSD.
• Observation:
• ceph -s shows remapped, backfilling, recovering PGs; performance dips temporarily.
• Recovery path:
• Wait for backfill to complete. Avoid restarting daemons mid-recovery.
• Verification:
• PGs reach active+clean state.

1. Client auth or mount failures

• Trigger: wrong key or missing ceph.conf on the client.
• Observation:
• Mount or rbd map fails with permission or not found errors.
• Recovery path:
• Copy /etc/ceph/* from admin node to client and ensure correct key permissions.
• Verification:
• Mount and rbd map succeed.

Rollback and reset procedures (lab only)

• Remove a mistaken OSD (DESTRUCTIVE):
• ceph osd out <id>
• ceph osd purge <id> --yes-i-really-mean-it
• Manually wipe the disk before reusing: sudo sgdisk --zap-all /dev/sdb && sudo wipefs -a /dev/sdb
• Purge the entire lab (DESTRUCTIVE):
• From admin dir: ceph-deploy purge ceph1 ceph2 ceph3
• ceph-deploy purgedata ceph1 ceph2 ceph3
• Remove any remaining /etc/ceph on nodes.
• Rebuild cleanly using the steps in Safe Configuration Path.

Operations Checklist

Use this short list to repeat the lab consistently.

Planning and inventory

• Pick OS and Ceph versions for all nodes; record them.
• Allocate 3 VMs with one extra data disk each; verify time sync.
• Ensure name resolution and SSH key-based access among nodes.

Cluster bring-up

• Create a work directory and run ceph-deploy new ceph1 ceph2 ceph3.
• Define public_network and pool defaults in ceph.conf.
• ceph-deploy install all nodes.
• ceph-deploy mon create-initial; ceph-deploy admin all nodes.
• ceph-deploy mgr create ceph1.
• ceph-deploy osd create --data /dev/sdb on each node.

Functional tests

• Create rbd pool; enable application; create and map a 512M image.
• Write a testfile to /mnt/rbd and list it.
• Create cephfs_data and cephfs_metadata; ceph fs new; mount CephFS.

Verification

• ceph -s shows HEALTH_OK; 3 mons; 3 osds up/in; 1 mgr active.
• ceph df shows pools with expected usage; rbd ls shows labimg.
• mount shows /mnt/rbd and /mnt/cephfs if mounted.

Exercise failures safely

• Stop a single mon and observe quorum; restart it.
• Stop a single OSD and verify degraded then healed state when restarted.
• Optionally simulate nearfull with caution, then clean up.

Reset and documentation

• When finished, unmount clients and remove test data.
• If rebuilding, ceph-deploy purge and purgedata, then repeat bring-up.
• Capture your exact commands, versions, and observations for repeatability.

Conclusion

You now have a compact, realistic Ceph lab you can build and rebuild locally to practice core workflows: quorum formation, OSD provisioning, RBD creation and mapping, and CephFS setup and mounting. You verified health with concrete checks, exercised common failure modes, and used safe rollback procedures to reset the environment. Keep the first pilot narrow and observable, then extend gradually: add an additional OSD, test different pool sizes, or try a client on a separate VM. With a consistent inventory, a safe configuration path, and a short runbook, your team can learn and troubleshoot Ceph with confidence before changing anything in more complex environments.