Ceph CLI Operations Guide: Version-Scoped Workflows for Safe Administration

Ceph administration demands precision. A single misapplied command can cascade into data unavailability or prolonged recovery. This guide provides version-aware, production-tested workflows for Reef (18.2.x), Squid (19.2.x), and 20.x releases. It replaces generic cheat sheets with concrete procedures: inventory baselines, scoped configuration changes, diagnostic deep-dives, and documented recovery paths for monitor quorum loss, OSD replacement, PG inconsistency, and capacity pressure. Every command includes expected output signals, verification steps, and rollback actions.

Prerequisites and Environment Inventory

Supported Release Matrix

Package-based deployments (apt, yum, dnf) use systemctl for daemon management; cephadm clusters run all daemons in containers. Never mix orchestration methods on the same cluster.

Admin Access Requirements

• Keyring: /etc/ceph/ceph.client.admin.keyring (mode 0600, owned by root:root) or CEPH_KEYRING environment variable.
• Config: /etc/ceph/ceph.conf or CEPH_CONF pointing to a valid config with mon_host entries.
• Privileges: sudo or root; cephadm requires passwordless sudo for the cephadm user.
• Network: TCP 6789/3300 (mon), 6800–7300 (OSD/MDS/MGR/RGW) reachable between all nodes; DNS or /etc/hosts resolution for monitor hostnames.
• Quorum Health: ceph -s must show mon: <N> daemons, quorum <ids> before any write operation.

Baseline Capture Commands

Run these before any change and store outputs with timestamped filenames:

ceph -s -f json
ceph versions -f json
cephadm ls -f json
ceph config dump -f json > baseline-config-$(date +%F-%H%M).json
ceph auth ls -f json
ceph osd tree -f json > baseline-topology-$(date +%F-%H%M).json

Commit these to version control or attach to your change ticket for rollback reference.

Architecture Quick-Reference

Daemon Roles and Key Commands

CRUSH Hierarchy and Device Classes

Default hierarchy:

root=default
  datacenter=dc1
    room=room1
      row=row1
        rack=rack1
          chassis=chassis1
            host=storage-01
              osd.0 (class=hdd)
              osd.1 (class=nvme)

Device classes (hdd, ssd, nvme) drive PG placement via crush rules. List classes with ceph osd crush class ls.

PG State Cheat Sheet

Version and Environment Inventory (Concrete)

Core Inventory Commands

# Full version report (JSON includes overall, mon, mgr, osd, mds, rgw)
ceph versions -f json

# cephadm-managed daemon inventory
cephadm ls -f json

# Orchestrator host/daemon view
ceph orch host ls -f json
ceph orch ps -f json --format=json-pretty

# Capacity and utilization
ceph df -f json
ceph osd df -f json
ceph pg stat -f json

Expected JSON Fields and Health Indicators

• ceph versions: "overall": "18.2.4", "mon": ["18.2.4", "18.2.4", "18.2.4"], "mgr": "18.2.4", "osd": ["18.2.4" ...]. Mixed versions → plan upgrade.
• cephadm ls: Each entry shows "style": "cephadm:v18.2.4", "name": "osd.0", "container_image_name": "docker.io/ceph/ceph:v18.2.4", "status": "running".
• ceph df: "stats": {"total_bytes": 10995116277760, "total_used_bytes": 3298534883328, "total_avail_bytes": 7696581394432}. total_avail_bytes below 30% → rebalance needed.
• ceph pg stat: "pgmap": {"num_pgs": 1024, "num_objects": 50000, "state": ["active+clean": 1020, "active+remapped": 4]}.

Safe Configuration Path

Read-Only Audit First

# Full effective config (all sections, all daemons)
ceph config dump -f json

# Single key for a specific daemon or global
ceph config get global osd_pool_default_size
ceph config get osd.0 osd_memory_target
ceph config get mgr mgr/cephadm/container_image_base

Scoped Changes with Defined Blast Radius

Verification After Change

# Show diff from on-disk ceph.conf and monitor DB
ceph config diff -f json

# Confirm pool-level autoscaler state
ceph osd pool ls detail -f json | jq '.[] | {pool_name, pg_autoscale_mode, pg_num, pgp_num}'

# Health and PG state
ceph -s
ceph pg stat

Rollback Procedures

# Remove a config key (reverts to compiled default or parent scope)
ceph config rm global osd_pool_default_pg_autoscale_mode

# Restore previous explicit value
ceph config set global osd_pool_default_size 3

# Redeploy affected daemons to pick up config (cephadm only)
cephadm redeploy --limit osd --dry-run
cephadm redeploy --limit osd

⚠️ Blast Radius: cephadm redeploy restarts daemons; schedule during low I/O. Verify ceph -s returns HEALTH_OK within 5 minutes.

Verification and Diagnostics

Health Deep-Dive

# Detailed health with codes and descriptions
ceph health detail -f json

# Mute a specific check with TTL (e.g., during planned OSD maintenance)
ceph health mute OSD_DOWN 1h --sticky

# PGs by state (filter for actionable states)
ceph pg ls bystate degraded,inconsistent,incomplete,stale,undersized -f json

# OSD tree with weights and device classes
ceph osd tree -f json

# OSD performance counters (latency, throughput)
ceph osd perf -f json

Slow Operations and OSD Internals

# Historic slow ops on a specific OSD (requires mgr active)
ceph daemon osd.0 dump_historic_ops -f json

# Heap stats across all OSDs (memory pressure)
ceph tell osd.* heap stats -f json

# Perf dump from OSD daemon (latency histograms, queue depths)
ceph daemon osd.0 perf dump -f json | jq '.osd_op_latency, .commit_latency, .apply_latency'

Network and Clock Diagnostics

# Monitor clock skew check (requires mon quorum)
ceph mon_clock_skew_check -f json

# Mon version and map epoch
ceph tell mon.* version -f json

# Network connectivity from admin node
ceph tell mon.* ping -f json

Log Collection

# cephadm logs (last 100 lines)
ceph log last cephadm 100

# Systemd journal for a specific daemon (package or cephadm)
journalctl -u ceph-<FSID>@osd.0 -n 200 --no-pager

# Ceph daemon log via socket (works for both deployments)
ceph daemon osd.0 log flush
ceph daemon osd.0 log reopen

Failure Modes and Recovery

Monitor Quorum Loss

Symptoms: ceph -s shows mon: 1 daemons, quorum <single>, HEALTH_ERR, client I/O errors. Prerequisites: At least one mon running, access to its DB (/var/lib/ceph/mon/ceph-<id>/store.db).

Recovery:

# 1. Dump monmap from surviving mon
ceph mon dump -f json > monmap-backup.json

# 2. Stop all mons, inject monmap on each
systemctl stop ceph-mon@<id>  # or cephadm: ceph orch daemon stop mon.<host>
ceph-mon -i <id> --inject-monmap monmap-backup.json

# 3. Start mons sequentially, verify quorum
systemctl start ceph-mon@<id>
ceph -s

Disaster Recovery: If all mons lost, rebuild from ceph-mon --mkfs -i <id> --monmap <file> --keyring <admin-keyring> using a saved monmap.

OSD Down / Out / Replacement

Symptoms: ceph osd tree shows down, out, or destroyed; PGs degraded / recovery.

Workflow:

# 1. Mark OSD out (starts rebalance)
ceph osd out 12

# 2. Wait for PGs to reach active+clean (monitor)
ceph -w  # or: while ceph pg ls degraded -f json | grep -q degraded; do sleep 10; done

# 3. Destroy OSD (removes from crush, deletes auth)
ceph osd destroy 12 --yes-i-really-mean-it

# 4. Zap device (package or cephadm)
ceph-volume lvm zap /dev/sdX --destroy

# 5. Add replacement (cephadm)
ceph orch daemon add osd storage-01:/dev/nvme0n1
# Package: ceph-volume lvm create --bluestore --data /dev/nvme0n1

⚠️ Warning: Never ceph osd destroy without prior ceph osd out and clean rebalance. Data loss risk if PGs not active+clean.

PG Inconsistent / Stuck

Symptoms: ceph health detail shows PG_INCONSISTENT, ceph pg ls inconsistent returns PG IDs. Prerequisites: Identify affected PGs, verify backups, understand ceph pg repair may discard data.

Recovery:

# 1. List inconsistent PGs with details
ceph pg ls inconsistent -f json

# 2. Repair single PG (run on primary OSD if possible)
ceph pg repair 1.5

# 3. Monitor repair state
ceph pg ls 1.5 -f json -w  # watch for "repair" state to clear

# 4. If repair fails, deep-scrub first
ceph pg deep-scrub 1.5
ceph pg repair 1.5

🔄 Rollback: No automated rollback; restore from backup if repair causes data loss.

Full / Nearfull Cluster

Symptoms: HEALTH_ERR / HEALTH_WARN FULL / NEARFULL, ceph df shows > 85% / 95% used.

Recovery:

# 1. Identify utilization outliers
ceph osd df -f json | jq '.nodes[] | select(.type=="osd") | {id, utilization, kb_avail}'

# 2. Temporarily adjust full ratios (buys time)
ceph config set global mon_osd_full_ratio 0.95
ceph config set global mon_osd_nearfull_ratio 0.85

# 3. Reweight overfull OSDs (gradual)
ceph osd crush reweight osd.0 0.8

# 4. Add capacity (preferred)
ceph orch daemon add osd <host>:<device>

# 5. Trigger balancer
ceph balancer on
ceph balancer status -f json

cephadm Bootstrap / Upgrade Failures

Symptoms: cephadm bootstrap hangs, cephadm upgrade reports upgrade_plan errors.

Diagnostics:

# Dry-run before any change
cephadm bootstrap --dry-run --config /etc/ceph/ceph.conf
cephadm upgrade --dry-run --image docker.io/ceph/ceph:v19.2.0

# Inspect upgrade plan
cephadm upgrade --dry-run --image docker.io/ceph/ceph:v19.2.0 -f json | jq '.upgrade_plan'

# Redeploy specific daemon type after failed upgrade
cephadm redeploy --limit mgr --dry-run
cephadm redeploy --limit mgr

Recovery: Rollback container image via ceph config set mgr mgr/cephadm/container_image_base <old_image> then cephadm redeploy --limit mgr.

Operations Checklist (Runbook Style)

Pre-Change Validation

• [ ] ceph -s → HEALTH_OK or known HEALTH_WARN (documented)
• [ ] ceph config dump -f json > pre-change-$(date +%F-%H%M).json
• [ ] ceph df → capacity headroom > 20%
• [ ] Maintenance window approved, stakeholders notified
• [ ] Rollback commands drafted and tested in dry-run mode

Per-Task Execution Template

Post-Change Verification

• [ ] ceph -s → HEALTH_OK
• [ ] ceph pg stat → all active+clean (zero degraded, inconsistent, stale)
• [ ] ceph osd perf → commit_latency < 10 ms, apply_latency < 5 ms (adjust for workload)
• [ ] Dashboard alerts clear; no new ceph health detail warnings
• [ ] ceph config diff matches intended change only

Documentation Requirements

• Ticket reference, operator name, start/end timestamps
• ceph config diff output attached
• ceph -s and ceph pg stat before/after snapshots
• Any anomalies observed and mitigation steps taken

Realistic Technical Scenario

Scenario: Add Three NVMe OSDs to storage-04 in a Reef 18.2.4 cephadm Cluster

Goal: Expand capacity by 3 × 3.84 TB NVMe on storage-04, verify rebalance completes without client-visible latency spike.

Prerequisites

• Cluster HEALTH_OK, ceph versions shows all daemons 18.2.4
• storage-04 labeled: ceph orch host label add storage-04 nvme
• Devices visible: lsblk -d -o NAME,SIZE,TYPE,MODEL /dev/nvme[0-2]n1
• Admin keyring and ceph.conf on control node

Execution Steps

# 1. Add OSDs (cephadm orchestrates ceph-volume lvm internally)
ceph orch daemon add osd storage-04:/dev/nvme0n1
ceph orch daemon add osd storage-04:/dev/nvme1n1
ceph orch daemon add osd storage-04:/dev/nvme2n1

# 2. Watch PG rebalance in real time
ceph -w  # press 'q' to quit; look for "pgmap" updates

# 3. Monitor PG stats until settled
while true; do
  ceph pg stat -f json | jq '.pgmap | {active_clean: .state_counts[] | select(.name=="active+clean") | .count, recovering: .state_counts[] | select(.name=="recovery") | .count}'
  sleep 30
done

# 4. Verify OSD performance during rebalance
ceph osd perf -f json | jq '.osd_perf_infos[] | select(.id==12 or .id==13 or .id==14) | {id, commit_latency_ms, apply_latency_ms}'

# 5. Check balancer status (auto-enabled in Reef+)
ceph balancer status -f json

# 6. Confirm even utilization
ceph osd df -f json | jq '.nodes[] | select(.type=="osd") | {id, utilization, kb_avail}' | sort -k2 -n

Verification Criteria

• All PGs active+clean within 2 hours (adjust for data volume)
• ceph osd perf latency on new OSDs ≤ existing OSD median + 20%
• ceph df shows utilization variance < 15% across OSDs

Rollback Plan

# If rebalance stalls or latency spikes:
ceph orch daemon rm osd.12 osd.13 osd.14
ceph-volume lvm zap /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 --destroy
ceph osd crush rm osd.12 osd.13 osd.14  # if crush entries remain

Document final state with ceph config dump > post-expansion-$(date +%F).json.

Conclusion

Operational safety in Ceph demands version-scoped commands, observed baselines, and tested rollback paths for every change. This guide replaces generic cheat sheets with concrete workflows: inventory Reef, Squid, and 20.x releases via ceph versions and cephadm ls; diagnose with ceph health detail and ceph daemon perf dump; execute scoped changes like PG autoscaler enablement or OSD replacement with explicit verification using ceph pg stat and ceph osd perf; and recover from quorum loss, PG inconsistency, or capacity pressure using documented procedures. Always capture ceph config dump and ceph -s snapshots before and after; mute health checks with TTL during planned maintenance; distinguish cephadm container workflows from package-based systemctl operations. The next step is to select one read-only verification from this guide, run it against your cluster, compare output to the expected signals documented here, and record the result in your runbook.