Ceph OSD and monitor troubleshooting for Proxmox environments: practical implementation guide

Ceph underpins many Proxmox environments, but when OSDs go down, monitors lose quorum, or placement groups (PGs) degrade, the blast radius can grow quickly. This guide gives you a clear, repeatable way to triage and fix the most common issues: OSD down/out, monitor troubles, health warnings, degraded or stuck PGs, and disk or network problems that sit underneath.

What you will get:

• A field-tested troubleshooting workflow
• Command snippets you can run on any node
• Guardrails to avoid data churn during maintenance
• A small, safe pilot you can run locally before wider rollout

Workflow Overview

Use this sequence to limit impact and shorten time to green:

1. Confirm the state and impact

• ceph -s and ceph health detail
• Identify affected pools and clients. If impact is low and you plan maintenance, consider temporarily setting noout; otherwise, preserve default behavior so Ceph can heal.

1. Map the scope

• ceph osd tree and ceph osd stat to list down OSDs
• ceph pg stat and ceph pg ls-by-state degraded, undersized, peering
• ceph mon stat and ceph quorum_status --format json-pretty for monitor health

1. Stabilize before changes

• For planned work, ceph osd set noout to prevent unnecessary rebalance; clear it with ceph osd unset noout as soon as possible
• Avoid changing multiple variables at once

1. Node and device checks

• systemctl status ceph-osd@<id>, ceph-mon@<host>
• journalctl -u ceph-osd@<id> and journalctl -u ceph-mon@<host>
• ceph-volume lvm list to map OSDs to devices
• smartctl -a /dev/<disk> or nvme smart-log /dev/<nvme>
• ip link, ip addr, ethtool -S <iface> for drops and errors

1. Fix cause, then recover

• Restart failed services if the cause is transient
• Replace failing disks and re-add OSDs
• Reweight or mark out only when you are confident about the direction
• Monitor recovery speed and client latency; tune recovery throttle temporarily if needed

1. Post-incident

• Clear temporary flags (noout, nobackfill, norecover)
• Recheck ceph health detail and logs
• Document what worked and set alerts for early detection

Quick Health Triage

Start with global health, then pivot to the subsystem that is red.

Commands:

• ceph -s
• ceph health detail
• ceph osd tree
• ceph mon stat

Interpretation guide:

• OSD_DOWN or OSD_HOST_DOWN: One or more OSD daemons or a whole node is not reporting. Check power, services, and disks.
• PG_DEGRADED or PG_UNDERSIZED: Data redundancy is below target. Investigate down OSDs and pool size/min_size.
• MON_DOWN or low monitor quorum: Client ops may continue but cluster changes and recovery can stall. Restore quorum quickly.
• Clock skew warnings: Fix NTP/chrony and ensure consistent time across nodes.

If you must do maintenance on a healthy cluster, use ceph osd set noout just before the work and ceph osd unset noout immediately after. Leaving noout set for long periods can hide real failures and increase risk.

OSD Down/Out Diagnostics

1. Verify process and logs

• systemctl status ceph-osd@<id>
• journalctl -u ceph-osd@<id> --since -2h
• ceph osd tree to see host and CRUSH placement

1. Map OSD to device

• ceph-volume lvm list | less
• lsblk -f to confirm filesystem and device health

1. Disk health

• smartctl -a /dev/<disk> or nvme smart-log /dev/<nvme>
• dmesg | grep -i -E 'blk|i/o|nvme|sata|scsi' for kernel I/O errors

1. Bring an OSD back if the disk is healthy

• systemctl restart ceph-osd@<id>
• Watch: ceph -s and ceph osd stat

1. If the disk is failing

• Mark the OSD out to trigger backfill: ceph osd out <id>
• Replace the disk, prepare and create a new OSD (use your standard provisioning method), then:
• Verify it joins: ceph osd tree and ceph osd df
• Reweight if needed: ceph osd reweight <id> <0.0-1.0>

1. Avoid churn during short outages

• For brief, planned restarts, use noout to prevent unnecessary movement
• For real failures, do not leave the OSD in; mark it out so recovery can proceed

Monitor and Quorum Recovery

Monitors maintain the cluster map and quorum. Loss of quorum blocks many operations.

1. Assess monitor health

• ceph mon stat
• ceph quorum_status --format json-pretty
• ceph mon dump

1. Fix the obvious first

• Ensure time sync is healthy (chrony/ntpd)
• Check network reachability between monitors
• systemctl status ceph-mon@<host> and journalctl -u ceph-mon@<host>

1. Restore quorum

• If a mon process is down but the host is fine, restart it
• If the host is down, recover the node or move the role to a healthy node according to your operational standard
• Aim for an odd number of monitors (commonly 3) on reliable nodes

1. Remove or replace a dead monitor only when certain

• After you confirm permanent loss, remove it from the monmap using the supported CLI for your deployment
• Add a replacement monitor and confirm quorum stabilizes

Recheck: ceph -s should show monmap healthy and quorum present.

Degraded PGs and Recovery

When PGs are degraded or undersized, data is at higher risk. Stabilize quickly.

1. Inspect PGs

• ceph pg stat
• ceph pg ls-by-state degraded, undersized, peering, backfill_wait, backfilling
• For a specific PG: ceph pg <pgid> query

1. Find missing or slow OSDs

• Look at the acting and up sets in the PG query
• Cross-check those OSDs with ceph osd tree and service status

1. Encourage recovery, safely

• Ensure enough OSDs are up to meet pool min_size
• If needed, temporarily raise recovery effort during low traffic windows (example):
• ceph tell osd.* injectargs '--osd_max_backfills 2 --osd_recovery_max_active 3 --osd_recovery_op_priority 63'
• Observe client latency; revert to defaults after recovery

1. PG repair (last resort for specific corruption cases)

• ceph pg repair <pgid>
• Use sparingly and only after confirming the underlying OSDs are healthy

Recovery is complete when PGs return to active+clean and ceph health is OK.

Disk and Network Checks

Ceph depends on predictable disks and a clean, loss-free network.

Disk checks

• smartctl -a /dev/<disk> or nvme smart-log /dev/<nvme>
• Compare media errors, wear, and temperature across peers
• dmesg for I/O resets, timeouts, or controller faults

Network checks

• ip link show and ip addr show to confirm interfaces and VLANs
• ethtool -S <iface> for drops, CRC errors, fifo overflows
• Ensure consistent MTU end-to-end on Ceph public and cluster networks
• Validate bonding/LACP settings if used

Proxmox specifics

• Verify that Ceph public and cluster traffic are on reliable networks
• Ensure Proxmox host time sync and stable management networking; noisy networks can cascade into Ceph instability

Local Pilot Plan

Start small, measure, and make it easy to inspect locally.

Goal

• Validate OSD restart behavior and PG recovery without risking production

Pilot steps

1. Create a small test pool

• ceph osd pool create pilot-pool 32
• Set size=3, min_size=2 on a replication rule appropriate for your hardware

1. Create a small RBD and write data

• rbd create pilot-pool/pilot-vol --size 64M
• Map and write a few hundred MB to generate PG activity

1. Simulate a single OSD restart during a low-traffic window

• ceph osd set noout
• systemctl restart ceph-osd@<id>
• Watch ceph -s and ceph pg stat until stable
• ceph osd unset noout

1. Capture metrics

• Time to recovery, max client latency, number of moved PGs
• Decide acceptable thresholds for production

1. Practice a controlled disk replacement on a spare

• Mark an OSD out, replace a spare disk, re-add an OSD, and confirm recovery speed and cleanliness

Success criteria

• PGs return to active+clean within your target window
• No unexpected client I/O errors
• Temporary flags are cleared immediately after use

Conclusion

A fast, disciplined approach to Ceph troubleshooting keeps Proxmox environments stable even during failures. Start with ceph -s and health details, map scope, stabilize with minimal flags, fix root causes at the node or disk level, then recover and verify. Use targeted recovery tuning only while you watch client impact. Finally, validate everything with a small local pilot before applying changes broadly.

Next checks

• Confirm no lingering flags: ceph osd dump | grep flags
• Ensure monitors are in quorum and time sync is healthy
• Verify PGs are active+clean and pool min_size is safe
• Document steps and thresholds so the next incident is faster