Ceph Advanced Concepts Explained with Practical Examples

Introduction

Ceph is a distributed storage system designed for massive scalability and reliability. Understanding its advanced concepts—such as CRUSH maps, placement groups, and the monitor quorum—is crucial for operating a production cluster. However, theoretical knowledge alone is insufficient; you need practical examples that connect each concept to real commands, expected outputs, and recovery steps.

This article is for developers, DevOps consultants, and technical startup teams who want to deepen their Ceph expertise. It focuses on operational safety: observe before changing, limit blast radius, protect secrets, verify results, and document recovery paths. We will walk through version inventory, safe configuration changes, verification, failure modes, and an operations checklist—each with concrete examples.

Version and Environment Inventory

Before touching a Ceph cluster, you must know exactly what you are working with. Start by identifying the installed version and your deployment topology.

Check the version:

ceph --version

Expected output: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989d7973a8a7d2e0a) quincy (stable)

Inventory your monitors and OSDs:

ceph mon dump
ceph osd tree

These read-only commands show the monitor map and OSD hierarchy, giving you a snapshot of your cluster's current state. Record the output with timestamps; this is your baseline.

Prerequisites:

• Access to a Ceph node with admin keyring (or sudo)
• Network connectivity to all cluster nodes
• Understanding of your cluster's version (e.g., Octopus, Pacific, Quincy)

Blast radius of observation: None—these commands are read-only and safe.

Always separate observation from intervention. Never run a command that modifies state without first capturing the read-only baseline.

Safe Configuration Path

When you need to change a Ceph setting, follow a structured path to minimize risk. This applies to both ceph config commands and manual config file edits.

Identify the component and scope:

• Which daemon? MON, OSD, MDS, or client?
• Is the change cluster-wide or per-daemon?

Example: Set a cluster-wide debug level temporarily

1. Observe current setting:

ceph config get mon debug_mon

Record the current value (e.g., 20/20).

1. Change with a scope limit:

ceph config set global debug_mon 10/10

This is a small, justified change to increase logging for troubleshooting.

1. Verify:

ceph config get mon debug_mon

Expected: 10/10.

1. Recovery (if needed):

ceph config set global debug_mon 20/20

Or restart the monitor if the change was in a config file.

Blast radius: This change affects all monitors and can increase log volume, but it is reversible. Always document the change and its purpose.

For persistent changes, edit the config file (e.g., ceph.conf) and push with ceph config assimilate-conf or restart daemons. Use ceph config dump to review the entire configuration.

Verification and Diagnostics

Verification is not just checking that a command succeeds; it is confirming that the system behaves as expected. Ceph provides several diagnostic tools to inspect health and performance.

Health checks:

ceph health detail

This shows detailed health status, including warnings like PG_AVAILABILITY or OSD_DOWN.

Placement group status:

ceph pg stat
ceph pg dump | head -20

These commands show PG states (active+clean, degraded, etc.) and distribution.

Example: Diagnosing slow requests

1. Observe the problem:

ceph daemon osd.0 ops

Look for ops_in_flight or slow_ops.

1. Identify the cause:

ceph daemon osd.0 dump_ops_in_flight

This shows operation details, target OSDs, and timing.

Normally, ops complete quickly. If you see ops stuck, check network or disk latency.

1. Compare with expected baseline:

1. Recovery action:

• Temporary: ceph osd set norebalance to stop rebalancing and reduce load.
• Permanent: fix the underlying issue (e.g., bad disk, network congestion).

Blast radius: Read-only commands are safe. Setting norebalance is reversible with ceph osd unset norebalance.

Remember to use version-appropriate commands. For example, ceph daemon osd.0 ops might differ in older versions.

Failure Modes and Recovery

Ceph is resilient, but it can fail in predictable ways. Understanding common failure modes helps you respond quickly and safely.

Common failure: OSD down

1. Observe:

ceph health detail

This shows which OSDs are down and how many PGs are affected.

1. Assess:

ceph osd tree

Identify the down OSD's ID and its host.

1. Action:

• If the OSD process crashed, restart it:

systemctl start ceph-osd@<id>

• If the disk is failed, mark it out and remove it:

ceph osd out <id>
ceph osd purge <id> --yes-i-really-mean-it

Then replace the disk and re-add the OSD.

1. Verify recovery:

ceph health
ceph pg status

Wait for PGs to reach active+clean.

Blast radius: Removing an OSD reduces cluster capacity and triggers data rebalancing. Ensure you have sufficient free space and redundancy (e.g., replication factor 3) before proceeding.

Another failure: Monitor clock skew

Monitors require synchronized clocks. Check with:

ceph mon stat

If you see clock skew errors, sync clocks using NTP or chrony on all nodes. After fixing, verify with ceph health.

Always plan for recovery before making changes. Document the exact commands and expected outcomes.

Operations Checklist

Use this checklist for any routine or maintenance operation on a Ceph cluster.

1. Inventory:

• Record Ceph version, topology, and current state (health, OSD tree, config dump).

1. Scope change:

• Define the smallest change needed. State the purpose and expected outcome.

1. Preflight checks:

• Verify prerequisites: network, disk space, clock sync.
• Ensure you have recovery steps documented.

1. **Execute change:

• Apply the change in a controlled manner (e.g., one OSD at a time for reboots).

1. Verify:

• Run health checks, PG status, and performance monitoring.
• Compare results with baseline.

1. Rollback plan:

• Have commands ready to revert the change if needed.

1. Document:

• Record what was changed, when, and why. Include any troubleshooting steps.

Example: Performing a rolling OSD restart

ceph osd set noout
for osd in $(ceph osd ls); do
  systemctl restart ceph-osd@${osd}
  sleep 30
  ceph health detail  # ensure cluster is healthy
  sleep 120          # let PGs recover
done
ceph osd unset noout

This approach limits blast radius by controlling the number of OSDs down at any time.

Conclusion

Ceph advanced concepts become practical when you apply them with an operational mindset. Version-scoped commands, observable baselines, and reversible changes are the backbone of safe Ceph operations. Start by picking one low-risk verification—like checking your cluster's health with ceph health detail—and practice recording the state before making any change. As you gain confidence, you can explore more complex tasks like pool tuning or CRUSH map modifications, always with a clear recovery path. Remember: an operator who understands failure modes and can verify recovery is the most valuable asset in any storage environment.