Ceph administration demands precision. A single misapplied command can cascade into data unavailability or prolonged recovery. This guide provides version-aware, production-tested workflows for Reef (18.2.x), Squid (19.2.x), and 20.x releases. It replaces generic cheat sheets with concrete procedures: inventory baselines, scoped configuration changes, diagnostic deep-dives, and documented recovery paths for monitor quorum loss, OSD replacement, PG inconsistency, and capacity pressure. Every command includes expected output signals, verification steps, and rollback actions. ## Prerequisites and Environment Inventory ### Supported Release Matrix
SeriesVersion Rangecephadm DefaultContainer RuntimeMinimum KernelPython
Reef18.2.xYespodman / docker5.15+3.8+
Squid19.2.xYespodman / docker6.1+3.10+
20.x20.0.xYespodman / docker6.5+3.11+
Package-based deployments (apt, yum, dnf) use systemctl for daemon management; cephadm clusters run all daemons in containers. Never mix orchestration methods on the same cluster. ### Admin Access Requirements - Keyring : /etc/ceph/ceph.client.admin.keyring (mode 0600, owned by root:root) or CEPH_KEYRING environment variable. - Config : /etc/ceph/ceph.conf or CEPH_CONF pointing to a valid config with mon_host entries. - Privileges : sudo or root; cephadm requires passwordless sudo for the cephadm user. - Network : TCP 6789/3300 (mon), 6800–7300 (OSD/MDS/MGR/RGW) reachable between all nodes; DNS or /etc/hosts resolution for monitor hostnames. - Quorum Health : ceph -s must show mon: daemons, quorum before any write operation. ### Baseline Capture Commands Run these before any change and store outputs with timestamped filenames: ceph -s -f json ceph versions -f json cephadm ls -f json ceph config dump -f json > baseline-config-$(date +%F-%H%M).json ceph auth ls -f json ceph osd tree -f json > baseline-topology-$(date +%F-%H%M).json Commit these to version control or attach to your change ticket for rollback reference. ## Architecture Quick-Reference ### Daemon Roles and Key Commands
DaemonPurposeHealth CheckCommon CLI
Monitor (mon)Cluster map, quorum, authceph mon dump, ceph tell mon.* versionceph mon stat
Manager (mgr)Metrics, modules, dashboard, orchestratorceph mgr stat, ceph mgr module lsceph tell mgr.* perf dump
OSDData storage, replication, recoveryceph osd tree, ceph osd df, ceph osd perfceph daemon osd.<id> perf dump
MDSCephFS metadata, journalceph fs status, ceph mds statceph tell mds.* session ls
RGWS3/Swift object gatewayceph orch ps --daemon-type rgwradosgw-admin realm list
### CRUSH Hierarchy and Device Classes Default hierarchy: root=default datacenter=dc1 room=room1 row=row1 rack=rack1 chassis=chassis1 host=storage-01 osd.0 (class=hdd) osd.1 (class=nvme) Device classes ( hdd , ssd , nvme ) drive PG placement via crush rules. List classes with ceph osd crush class ls . ### PG State Cheat Sheet
StateMeaningAction Required
active+cleanHealthy, all replicas currentNone
active+remappedActing set ≠ up set; backfill in progressMonitor
degradedMissing replicas, below min_sizeInvestigate OSD down
peering / recovery / backfillTransient rebuild statesWait, verify progress
incompleteInsufficient OSDs to satisfy min_sizeAdd OSDs or adjust min_size
stalePG not updated by mon; possible mon issueCheck mon quorum
undersizedFewer copies than size but ≥ min_sizePlan OSD replacement
inconsistentScrub detected data mismatchceph pg repair (see warnings)
## Version and Environment Inventory (Concrete) ### Core Inventory Commands # Full version report (JSON includes overall, mon, mgr, osd, mds, rgw) ceph versions -f json # cephadm-managed daemon inventory cephadm ls -f json # Orchestrator host/daemon view ceph orch host ls -f json ceph orch ps -f json --format=json-pretty # Capacity and utilization ceph df -f json ceph osd df -f json ceph pg stat -f json ### Expected JSON Fields and Health Indicators - ceph versions : "overall": "18.2.4" , "mon": ["18.2.4", "18.2.4", "18.2.4"] , "mgr": "18.2.4" , "osd": ["18.2.4" ...] . Mixed versions → plan upgrade. - cephadm ls : Each entry shows "style": "cephadm:v18.2.4" , "name": "osd.0" , "container_image_name": "docker.io/ceph/ceph:v18.2.4" , "status": "running" . - ceph df : "stats": {"total_bytes": 10995116277760, "total_used_bytes": 3298534883328, "total_avail_bytes": 7696581394432} . total_avail_bytes below 30% → rebalance needed. - ceph pg stat : "pgmap": {"num_pgs": 1024, "num_objects": 50000, "state": ["active+clean": 1020, "active+remapped": 4]} . ## Safe Configuration Path ### Read-Only Audit First # Full effective config (all sections, all daemons) ceph config dump -f json # Single key for a specific daemon or global ceph config get global osd_pool_default_size ceph config get osd.0 osd_memory_target ceph config get mgr mgr/cephadm/container_image_base ### Scoped Changes with Defined Blast Radius Change Command Blast Radius Prerequisites Enable PG autoscaler (preferred over manual pg_num) ceph config set global osd_pool_default_pg_autoscale_mode on All new pools; existing pools unchanged until ceph osd pool set pg_autoscale_mode on Mgr active, PG autoscaler module enabled Set default replica count ceph config set global osd_pool_default_size 3 New pools only min_size ≤ size ; quorum healthy Allow pool deletion (dangerous) ceph config set global mon_allow_pool_delete true Cluster-wide; enables destructive ops Maintenance window, backup verified Pin cephadm container image ceph config set mgr mgr/cephadm/container_image_base docker.io/ceph/ceph Next redeploy/upgrade Image exists in registry, matches release series ### Verification After Change # Show diff from on-disk ceph.conf and monitor DB ceph config diff -f json # Confirm pool-level autoscaler state ceph osd pool ls detail -f json | jq '.[] | {pool_name, pg_autoscale_mode, pg_num, pgp_num}' # Health and PG state ceph -s ceph pg stat ### Rollback Procedures # Remove a config key (reverts to compiled default or parent scope) ceph config rm global osd_pool_default_pg_autoscale_mode # Restore previous explicit value ceph config set global osd_pool_default_size 3 # Redeploy affected daemons to pick up config (cephadm only) cephadm redeploy --limit osd --dry-run cephadm redeploy --limit osd ⚠️ Blast Radius : cephadm redeploy restarts daemons; schedule during low I/O. Verify ceph -s returns HEALTH_OK within 5 minutes. ## Verification and Diagnostics ### Health Deep-Dive # Detailed health with codes and descriptions ceph health detail -f json # Mute a specific check with TTL (e.g., during planned OSD maintenance) ceph health mute OSD_DOWN 1h --sticky # PGs by state (filter for actionable states) ceph pg ls bystate degraded,inconsistent,incomplete,stale,undersized -f json # OSD tree with weights and device classes ceph osd tree -f json # OSD performance counters (latency, throughput) ceph osd perf -f json ### Slow Operations and OSD Internals # Historic slow ops on a specific OSD (requires mgr active) ceph daemon osd.0 dump_historic_ops -f json # Heap stats across all OSDs (memory pressure) ceph tell osd.* heap stats -f json # Perf dump from OSD daemon (latency histograms, queue depths) ceph daemon osd.0 perf dump -f json | jq '.osd_op_latency, .commit_latency, .apply_latency' ### Network and Clock Diagnostics # Monitor clock skew check (requires mon quorum) ceph mon_clock_skew_check -f json # Mon version and map epoch ceph tell mon.* version -f json # Network connectivity from admin node ceph tell mon.* ping -f json ### Log Collection # cephadm logs (last 100 lines) ceph log last cephadm 100 # Systemd journal for a specific daemon (package or cephadm) journalctl -u ceph-@osd.0 -n 200 --no-pager # Ceph daemon log via socket (works for both deployments) ceph daemon osd.0 log flush ceph daemon osd.0 log reopen ## Failure Modes and Recovery ### Monitor Quorum Loss Symptoms : ceph -s shows mon: 1 daemons, quorum , HEALTH_ERR , client I/O errors. Prerequisites : At least one mon running, access to its DB ( /var/lib/ceph/mon/ceph-/store.db ). Recovery : # 1. Dump monmap from surviving mon ceph mon dump -f json > monmap-backup.json # 2. Stop all mons, inject monmap on each systemctl stop ceph-mon@ # or cephadm: ceph orch daemon stop mon. ceph-mon -i --inject-monmap monmap-backup.json # 3. Start mons sequentially, verify quorum systemctl start ceph-mon@ ceph -s Disaster Recovery : If all mons lost, rebuild from ceph-mon --mkfs -i --monmap --keyring using a saved monmap. ### OSD Down / Out / Replacement Symptoms : ceph osd tree shows down , out , or destroyed ; PGs degraded / recovery . Workflow : # 1. Mark OSD out (starts rebalance) ceph osd out 12 # 2. Wait for PGs to reach active+clean (monitor) ceph -w # or: while ceph pg ls degraded -f json | grep -q degraded; do sleep 10; done # 3. Destroy OSD (removes from crush, deletes auth) ceph osd destroy 12 --yes-i-really-mean-it # 4. Zap device (package or cephadm) ceph-volume lvm zap /dev/sdX --destroy # 5. Add replacement (cephadm) ceph orch daemon add osd storage-01:/dev/nvme0n1 # Package: ceph-volume lvm create --bluestore --data /dev/nvme0n1 ⚠️ Warning : Never ceph osd destroy without prior ceph osd out and clean rebalance. Data loss risk if PGs not active+clean . ### PG Inconsistent / Stuck Symptoms : ceph health detail shows PG_INCONSISTENT , ceph pg ls inconsistent returns PG IDs. Prerequisites : Identify affected PGs, verify backups, understand ceph pg repair may discard data. Recovery : # 1. List inconsistent PGs with details ceph pg ls inconsistent -f json # 2. Repair single PG (run on primary OSD if possible) ceph pg repair 1.5 # 3. Monitor repair state ceph pg ls 1.5 -f json -w # watch for "repair" state to clear # 4. If repair fails, deep-scrub first ceph pg deep-scrub 1.5 ceph pg repair 1.5 🔄 Rollback : No automated rollback; restore from backup if repair causes data loss. ### Full / Nearfull Cluster Symptoms : HEALTH_ERR / HEALTH_WARN FULL / NEARFULL , ceph df shows > 85% / 95% used. Recovery : # 1. Identify utilization outliers ceph osd df -f json | jq '.nodes[] | select(.type=="osd") | {id, utilization, kb_avail}' # 2. Temporarily adjust full ratios (buys time) ceph config set global mon_osd_full_ratio 0.95 ceph config set global mon_osd_nearfull_ratio 0.85 # 3. Reweight overfull OSDs (gradual) ceph osd crush reweight osd.0 0.8 # 4. Add capacity (preferred) ceph orch daemon add osd : # 5. Trigger balancer ceph balancer on ceph balancer status -f json ### cephadm Bootstrap / Upgrade Failures Symptoms : cephadm bootstrap hangs, cephadm upgrade reports upgrade_plan errors. Diagnostics : # Dry-run before any change cephadm bootstrap --dry-run --config /etc/ceph/ceph.conf cephadm upgrade --dry-run --image docker.io/ceph/ceph:v19.2.0 # Inspect upgrade plan cephadm upgrade --dry-run --image docker.io/ceph/ceph:v19.2.0 -f json | jq '.upgrade_plan' # Redeploy specific daemon type after failed upgrade cephadm redeploy --limit mgr --dry-run cephadm redeploy --limit mgr Recovery : Rollback container image via ceph config set mgr mgr/cephadm/container_image_base then cephadm redeploy --limit mgr . ## Operations Checklist (Runbook Style) ### Pre-Change Validation - [ ] ceph -s → HEALTH_OK or known HEALTH_WARN (documented) - [ ] ceph config dump -f json > pre-change-$(date +%F-%H%M).json - [ ] ceph df → capacity headroom > 20% - [ ] Maintenance window approved, stakeholders notified - [ ] Rollback commands drafted and tested in dry-run mode ### Per-Task Execution Template
FieldExample
Commandceph osd out 12
PrerequisitesQuorum healthy, PGs active+clean, OSD 12 up / in
Expected Outputmarked osd.12 out
Verificationceph pg stat shows recovery → active+clean within 30 min
Rollbackceph osd in 12
Timeout45 minutes; escalate if > 10% PGs stuck recovery
Operator / Ticketops-2024-0423, alice@example.com
### Post-Change Verification - [ ] ceph -s → HEALTH_OK - [ ] ceph pg stat → all active+clean (zero degraded , inconsistent , stale ) - [ ] ceph osd perf → commit_latency < 10 ms, apply_latency < 5 ms (adjust for workload) - [ ] Dashboard alerts clear; no new ceph health detail warnings - [ ] ceph config diff matches intended change only ### Documentation Requirements - Ticket reference, operator name, start/end timestamps - ceph config diff output attached - ceph -s and ceph pg stat before/after snapshots - Any anomalies observed and mitigation steps taken ## Realistic Technical Scenario ### Scenario: Add Three NVMe OSDs to storage-04 in a Reef 18.2.4 cephadm Cluster Goal : Expand capacity by 3 × 3.84 TB NVMe on storage-04 , verify rebalance completes without client-visible latency spike. ### Prerequisites - Cluster HEALTH_OK , ceph versions shows all daemons 18.2.4 - storage-04 labeled: ceph orch host label add storage-04 nvme - Devices visible: lsblk -d -o NAME,SIZE,TYPE,MODEL /dev/nvme[0-2]n1 - Admin keyring and ceph.conf on control node ### Execution Steps # 1. Add OSDs (cephadm orchestrates ceph-volume lvm internally) ceph orch daemon add osd storage-04:/dev/nvme0n1 ceph orch daemon add osd storage-04:/dev/nvme1n1 ceph orch daemon add osd storage-04:/dev/nvme2n1 # 2. Watch PG rebalance in real time ceph -w # press 'q' to quit; look for "pgmap" updates # 3. Monitor PG stats until settled while true; do ceph pg stat -f json | jq '.pgmap | {active_clean: .state_counts[] | select(.name=="active+clean") | .count, recovering: .state_counts[] | select(.name=="recovery") | .count}' sleep 30 done # 4. Verify OSD performance during rebalance ceph osd perf -f json | jq '.osd_perf_infos[] | select(.id==12 or .id==13 or .id==14) | {id, commit_latency_ms, apply_latency_ms}' # 5. Check balancer status (auto-enabled in Reef+) ceph balancer status -f json # 6. Confirm even utilization ceph osd df -f json | jq '.nodes[] | select(.type=="osd") | {id, utilization, kb_avail}' | sort -k2 -n ### Verification Criteria - All PGs active+clean within 2 hours (adjust for data volume) - ceph osd perf latency on new OSDs ≤ existing OSD median + 20% - ceph df shows utilization variance ### Rollback Plan # If rebalance stalls or latency spikes: ceph orch daemon rm osd.12 osd.13 osd.14 ceph-volume lvm zap /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 --destroy ceph osd crush rm osd.12 osd.13 osd.14 # if crush entries remain Document final state with ceph config dump > post-expansion-$(date +%F).json . ## Conclusion Operational safety in Ceph demands version-scoped commands, observed baselines, and tested rollback paths for every change. This guide replaces generic cheat sheets with concrete workflows: inventory Reef, Squid, and 20.x releases via ceph versions and cephadm ls ; diagnose with ceph health detail and ceph daemon perf dump ; execute scoped changes like PG autoscaler enablement or OSD replacement with explicit verification using ceph pg stat and ceph osd perf ; and recover from quorum loss, PG inconsistency, or capacity pressure using documented procedures. Always capture ceph config dump and ceph -s snapshots before and after; mute health checks with TTL during planned maintenance; distinguish cephadm container workflows from package-based systemctl operations. The next step is to select one read-only verification from this guide, run it against your cluster, compare output to the expected signals documented here, and record the result in your runbook.