Ceph Troubleshooting with Practical Examples

Ceph troubleshooting does not need to be guesswork. With a clear inventory, a safe configuration approach, and a focused diagnostic sequence, most issues can be localized quickly and recovered with low risk. This guide provides:

• A quick inventory to baseline versions and topology
• A safe path for configuration and operational changes
• Diagnostic commands with expected signals and constructed examples
• Step-by-step recovery workflows for common failures
• Rollback and verification guidance after each fix
• A concise operations checklist you can run daily or during incidents

The goal is a practical, repeatable flow that reduces downtime and rework while keeping data safe.

Version and Environment Inventory

Before taking action, capture what you have now. This protects you from blind spots and gives you a measurable before/after.

Prerequisites

• Shell access to all monitor, manager, OSD, and metadata servers
• Sudo privileges on nodes that run Ceph daemons
• Ceph CLI configured (keyrings present under /etc/ceph or environment)
• A maintenance window if data movement may occur

Cluster Inventory Commands

Overall status and versions:

ceph -s
ceph --version
ceph versions
ceph fsid

Topology and capacity:

ceph osd stat
ceph osd tree
ceph osd df tree
rados df
ceph osd pool ls detail

Core services:

ceph mon stat
ceph mgr stat
ceph mds stat  # if using CephFS

Time and system basics on each node:

hostnamectl
chronyc sources -v  # or: ntpq -p
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT

Constructed Example: Sample ceph -s Output

cluster:
  id: 3c1a6c92-aaaa-bbbb-cccc-3f5c9a1d0001
  health: HEALTH_WARN 1 osds down; 2 pgs degraded

services:
  mon: 3 daemons, quorum mon-a, mon-b, mon-c (age 2h)
  mgr: mgr-a(active), standbys: mgr-b
  osd: 12 osds: 11 up, 11 in

data:
  pools: 3 pools, 128 pgs
  objects: 1.2M objects, 1.5 TiB used, 8.7 TiB / 10.2 TiB avail
  pgs: 2 degraded, 126 active+clean

Interpretation: One OSD is down, two PGs are degraded but the cluster is mostly healthy. Recovery likely completes after bringing the OSD up or replacing it. Note the 15% utilization -- well below the default 85% nearfull threshold.

Safe Configuration Path

When production data is involved, change discipline is essential. Use these guardrails:

• Change one thing at a time. Verify between steps with ceph -s and targeted checks.
• Use cluster flags sparingly and clear them promptly:
• noout avoids data movement during brief maintenance windows. Clear it as soon as possible.
• norecover and norebalance pause recovery and rebalancing; use only to protect latency during an urgent fix and unset when done.
• Prefer targeted reweight over hard out when isolating a noisy disk. Reweight reduces impact and is easy to roll back.
• Do not run destructive tools on BlueStore data paths unless you have explicit backups and vendor guidance.
• Keep a time sync budget under 50 ms across nodes; fix clock skew first if present.

Maintenance Flags Quick Reference

Verification and Diagnostics

Start with cluster-wide signals, then narrow to components. Confirm each hypothesis with a specific check.

Core Health and Time

ceph health detail
chronyc tracking && chronyc sources -v  # or: ntpstat; ntpq -p

• Expected healthy: No clock skew reported in ceph health detail. If you see CLOCK_SKEW, fix time sync first.

Placement Groups and OSDs

ceph pg stat
ceph pg dump pgs_brief | head -n 30
ceph osd perf
ceph osd df tree

• Expected healthy: Most PGs in active+clean; small number briefly in remapped+backfill after changes.
• Watch ceph osd perf: Commit and apply latency should stay under 10 ms on healthy SSDs; sustained values above 20 ms often precede flapping.

Logs

journalctl -u ceph-osd@<id> -n 200 --no-pager
journalctl -u ceph-mon@<name> -n 200 --no-pager
journalctl -u ceph-mgr@<name> -n 200 --no-pager
journalctl -u ceph-mds@<name> -n 200 --no-pager  # if using CephFS

• Look for device I/O errors, auth failures, clock jumps, and repeated crashes.

Networking Sanity

ip -br addr
ip -br link
ethtool -S <iface> | egrep 'err|drop'
ping -c3 <other-node-mgmt-ip>

• Expected: No RX/TX errors; consistent MTU (typically 1500 or 9000) across the Ceph network.

PG Introspection (Targeted)

ceph pg <pgid> query | jq '.state, .acting, .up, .info.last_scrub_stamp, .peering_blocked_by'

• Expected: State shows active+clean or active+recovery; empty peering_blocked_by when healthy.

Constructed Example: Fast Health Taxonomy

Failure Modes and Recovery

The following workflows cover common issues. For each, observe, act, and verify before moving on. All examples are constructed.

1) OSD Down or Flapping

Symptoms

• ceph -s shows 1 or more OSDs down
• ceph osd tree indicates devices in down or out state
• journalctl shows repeated OSD restarts or I/O errors

Steps

1. Identify the OSD id and host

   ceph osd tree
   ceph osd find <id>

Expected: You see the host and device path for the OSD (e.g., osd.7 on host3 using /dev/nvme1n1).

1. Inspect the service and logs on that host

   sudo systemctl status ceph-osd@<id>
   sudo journalctl -u ceph-osd@<id> -n 200 --no-pager
   sudo smartctl -a /dev/<device> | egrep 'SMART overall|Reallocated|Pending|Error'

Expected: If the disk is failing, SMART shows Reallocated_Sector_Ct or Current_Pending_Sector > 0; otherwise, look for filesystem or permission issues.

1. Short maintenance window: prevent churn if you must restart

   ceph osd set noout

1. Attempt a clean restart

   sudo systemctl restart ceph-osd@<id>
   sleep 10; ceph osd stat

Expected: OSD returns to up+in; PGs begin recovering. If it flaps again, continue.

1. If the device is noisy but not dead, reduce weight to shift load off it

   ceph osd crush reweight osd.<id> 0.60

Observe: ceph -s and ceph osd df tree to confirm movement. If the device is failing, mark it out and plan replacement.

1. If the device is failed, mark out and remove from CRUSH

   ceph osd out <id>
   # After data is drained and safe, remove from CRUSH and LVM/device mapping per your deployment

Verification and Rollback

• After restart or reweight: ceph -s should trend toward active+clean. If performance degrades, rollback reweight (e.g., 1.00) or re-enable the OSD with ceph osd in <id>.
• Clear flags once stable: ceph osd unset noout

2) PGs Stuck Degraded, Peering, or Inactive

Symptoms

• ceph pg stat shows PGs in states like degraded, peering, stale, or inactive
• ceph health detail lists specific PG ids and blocked peering reasons

Steps

1. Inspect PG detail

   ceph pg <pgid> query | jq '.state, .up, .acting, .peering_blocked_by, .recovery_state'

1. Check that all acting set OSDs are up

   ceph osd stat
   ceph osd tree | grep -E 'osd\(<id1>|<id2>|<id3>\)'

1. If peering is blocked by unfound objects, give recovery headroom and avoid bandwidth caps you control

   # Temporarily pause rebalancing only if it contends with recovery
   ceph osd set norebalance
   # Ensure norecover is NOT set unless you intentionally paused it
   ceph osd dump | grep flags

1. If a specific OSD is slow, temporarily reweight it down to unblock peering

   ceph osd crush reweight osd.<id> 0.80

1. For a single inconsistent PG, attempt a repair

   ceph pg repair <pgid>

Expected: PG transitions to active+recovering, then active+clean.

Rollback

• Undo reweights to 1.00 if they cause imbalance.
• Clear norebalance: ceph osd unset norebalance

3) MON Quorum or Clock Skew Issues

Symptoms

• ceph -s shows mon daemons not in quorum
• ceph health detail shows CLOCK_SKEW warnings

Steps

1. Verify time sync on all MON hosts

   chronyc tracking; chronyc sources -v

Expected: Offset remains within tens of milliseconds. A Last offset above 50 ms warrants immediate correction.

1. Check MON status and logs

   ceph mon stat
   sudo journalctl -u ceph-mon@<name> -n 200 --no-pager

1. If a MON is wedged, restart that MON

   sudo systemctl restart ceph-mon@<name>

1. If quorum is still not formed, ensure network reachability between MONs and that the mon_host addresses are correct for your environment.

Verification and Rollback

• ceph -s should show a stable quorum (e.g., mon: 3 daemons, quorum mon-a, mon-b, mon-c). If restarting a MON worsens stability, revert the restart and investigate network or host time further.

4) Nearfull, Backfillfull, or Full Cluster

Symptoms

• HEALTH_WARN or HEALTH_ERR with nearfull/backfillfull/full flags
• Writes blocked in full condition (cluster stops accepting writes at 95% by default)

Steps

1. Confirm utilization and pool distribution

   rados df
   ceph osd df tree
   ceph osd pool ls detail | egrep 'size|min_size|pg_num'

1. Free space safely

• Delete unneeded snapshots or objects in low-risk pools.
• Temporarily reduce hot pool pressure (application-side throttling) while you add capacity.
• Reweight a few large OSDs down to spread data if imbalance is the cause:

     ceph osd crush reweight osd.<id> 0.95  # small adjustments across several OSDs

1. Add capacity as the durable fix. After adding OSDs, verify that backfill proceeds and flags clear.

Verification and Rollback

• ceph -s clears nearfull/backfillfull/full; pool free space increases.
• Rollback reweights to 1.00 after balance is restored.

5) Inconsistent PGs and Scrubbing

Symptoms

• ceph health detail lists PG inconsistent
• Scrub or deep-scrub reveals mismatched objects

Steps

1. Identify the PGs and relevant OSDs

   ceph health detail | grep -i inconsistent -A3

1. Trigger targeted repair during a low-traffic window

   ceph pg repair <pgid>

1. If many PGs are inconsistent, review scrub schedule to avoid bursty deep-scrub contention. Stagger deep-scrubs during off-peak hours (e.g., set osd_scrub_begin_hour = 22 and osd_scrub_end_hour = 4 with randomized offsets per OSD).

Verification and Rollback

• ceph -s shows no inconsistent PGs. If repair increases latency too much, pause and resume during a quieter window.

6) RBD Client Errors (Timeouts or IO Failures)

Symptoms

• Clients (e.g., Proxmox or Linux rbd map) report IO errors or timeouts
• ceph -s may show slow ops or degraded PGs

Steps

1. Confirm client and cluster versions

   ceph versions
   rbd --version

1. Check the specific image and pool health

   rbd info <pool>/<image>
   ceph pg stat

1. Confirm the client can reach monitors and OSDs

• Validate ms_bind network and firewall ports (6789 for monitors, 6800-7300 for OSDs).
• Test with ceph -s from the client node using its keyring.

1. If only one client path is impacted, remap the image on that client after confirming the cluster is healthy.

Constructed Example: Remap on a Linux Client

sudo rbd unmap /dev/rbd0
sudo rbd map <pool>/<image>

Verification and Rollback

• Application IO resumes without timeouts. If the remap does not help, revert and continue server-side diagnostics.

7) CephFS MDS Issues (Laggy or Stuck Failover)

Symptoms

• ceph mds stat shows fewer active MDS than expected or laggy states
• Clients see metadata timeouts

Steps

1. Confirm MDS daemons and standbys

   ceph mds stat
   journalctl -u ceph-mds@<name> -n 200 --no-pager

1. If the active MDS is unhealthy, fail over to a standby by restarting the unhealthy daemon

   sudo systemctl restart ceph-mds@<name>

1. If no standby exists, add one before planned maintenance to avoid single points of failure.

Verification and Rollback

• ceph mds stat shows one active and at least one standby (e.g., mds.fs-a:1 {0=mds-host1=up:active} 1 up:standby-replay mds-host2). If restart worsens behavior, revert and examine the underlying host and network.

Expected Results, Checks, and Failure Signals

Use these quick confirmations after each change:

• Health trend: ceph -s transitions from WARN/ERR to HEALTH_OK or fewer errors.
• PG recovery: ceph pg stat shows decreasing degraded/remapped counts over time.
• Service stability: systemctl status remains active (running); no repeated restarts in journalctl.
• Latency: Application-side metrics (IOPS/latency) improve or at least do not regress.

Escalate if you see:

• PGs stuck inactive for more than a few minutes without peering progress.
• Increasing inconsistent PGs after repair attempts.
• MON quorum flapping despite time and network stability.
• Full condition persisting after reclamation and reweighting.

Rollback and Recovery Guidance

Keep recovery reversible where possible:

• Reweights: Always note previous value; rollback with ceph osd crush reweight osd.<id> 1.00.
• Flags: List and clear as you exit maintenance.

  ceph osd dump | grep flags
  ceph osd unset noout
  ceph osd unset norecover
  ceph osd unset norebalance

• Restarts: If a daemon restart worsens quorum or stability, revert by restarting a previously healthy peer and rechecking time and network.
• Repairs: If ceph pg repair increases load, pause and retry in a quieter window; do not batch-repair many PGs simultaneously in a busy cluster.

Practical Examples (Constructed)

Example A: Single OSD Down After Host Reboot

1. Observe

   ceph -s  # shows 1 osd down, 8 pgs degraded

1. Act

   ceph osd set noout
   sudo systemctl restart ceph-osd@7
   sleep 15; ceph osd stat
   ceph osd unset noout

1. Verify

• ceph -s shows 0 osds down; degraded PGs count drops to 0 within minutes.

Example B: Inconsistent PG After Deep Scrub

1. Observe

   ceph health detail | grep inconsistent -A2

1. Act

   ceph pg repair 1.23a

1. Verify

• ceph -s clears the inconsistent warning; application I/O unaffected.

Operations Checklist

Daily

• ceph -s and ceph health detail are clean or trending to clean.
• rados df shows comfortable headroom (>20% free); plan capacity when below 20% free.
• chronyc sources -v reveals stable time sources on all nodes (offset < 50 ms).
• Review a small sample of logs on each role (mon, mgr, osd, mds) for anomalies.

During Incidents

• Capture baseline: ceph -s, ceph versions, ceph osd tree.
• Localize: pg query, osd perf, and relevant daemon logs.
• Protect: set noout only when needed and clear it promptly.
• Make a single, reversible change (restart, reweight, targeted repair) and verify.
• Avoid pausing recovery longer than necessary; unset flags.

Weekly or After Changes

• Confirm scrub scheduling is staggered and not colliding with peak hours.
• Review the largest OSDs and pool distribution; fix imbalances with small reweights (±0.05 increments).
• Test failover for one MDS or non-critical OSD to ensure procedures are current.

Conclusion

You now have a practical, defensible approach to Ceph troubleshooting: inventory first, make minimal and reversible changes, verify with clear signals, and roll back quickly if needed. Start with a narrow, measurable pilot such as stabilizing OSD flapping on one host, confirm that health indicators improve, and then extend the workflow to additional nodes and services. With this methodical flow, you can reduce downtime, lower risk, and keep your storage cluster predictable under pressure.