Ceph pools and placement groups troubleshooting explained: practical implementation guide

Ceph reliability hinges on how pools and placement groups (PGs) map data to OSDs. When PGs are degraded or undersized, the cluster can serve IO but with reduced durability. This guide explains how pools and PGs work together, how to interpret PG states, how replication size and min_size affect availability and safety, and which commands to run first to diagnose and recover without making things worse. It targets developers, DevOps consultants, and startup teams that operate Ceph directly or through platforms that embed it.

What you will take away:

• A clear workflow to triage and fix PG problems safely.
• Practical checks that surface the root cause fast.
• Tactical playbooks for common recovery scenarios.

Workflow Overview

Use a repeatable path to reduce risk and speed resolution:

1. Read health and scope the blast radius

• ceph -s and ceph health detail to learn what is wrong and how widespread it is.
• Note counts of degraded/undersized PGs and affected pools.

1. Map pools to PGs and OSDs

• ceph osd pool ls detail to see pool settings.
• ceph pg dump pgs_brief to see state and OSD sets.

1. Verify OSD and host health

• ceph osd tree and ceph osd df for status and capacity.
• systemctl status ceph-osd@<id> and journalctl -u ceph-osd@<id> -n 200 when an OSD is down or flapping.

1. Check replication size and min_size

• ceph osd pool get <pool> size and ceph osd pool get <pool> min_size.
• Confirm settings match your failure domain and risk tolerance.

1. Identify exact PG states

• ceph pg stat for totals; ceph pg ls-by-pool <pool> degraded to focus.
• ceph pg <pgid> query for details on stuck PGs.

1. Choose the least disruptive fix

• Prefer restoring failed OSDs or hosts over changing pool settings.
• If you must change settings, do it on a narrow scope with capacity headroom.

1. Monitor recovery to clean

• Watch ceph -s and ceph health detail until PGs are active+clean.
• Track backfill and recovery speed; verify no nearfull conditions.

1. Prevent repeats

• Validate autoscaling or manual PG counts per pool.
• Align size and min_size with expected failures and SLAs.

Ceph Pools and PGs: Key Concepts

• Pools: Logical namespaces with their own replication size, min_size, and CRUSH rule. Different workloads should have different pools.
• Placement groups (PGs): The unit of placement, peering, and recovery. Each PG maps to a set of OSDs based on the pool's CRUSH rule and replication setting.
• Replication size: Target number of replicas per PG. More replicas increase durability at the cost of capacity and recovery traffic.
• min_size: Minimum replicas required to accept writes safely. Writes are blocked below min_size to protect data integrity.
• PG count: Affects data distribution and recovery parallelism. Too few PGs creates hotspots; too many wastes memory and slows peering and recovery. Use autoscaling when available, or plan manual PG counts carefully.

Common PG States and Actions

• active+clean: Desired state. No action needed.
• active+degraded: Some replicas are missing but the PG can serve IO. Action: confirm affected OSDs are up and in; let recovery backfill missing replicas.
• undersized+degraded: Fewer OSDs than required by size. Action: restore OSDs or hosts. Change size only if absolutely necessary and after impact review.
• peering: OSDs are exchanging maps to agree on the PG's history. Action: let it finish; if stuck, inspect OSD logs and network between peers.
• backfilling/recovering: Data is moving to restore redundancy. Action: monitor progress; avoid heavy maintenance until it completes.
• stale/inactive: Monitors are not receiving timely reports. Action: restore communication or the OSDs hosting those PGs.
• inconsistent: Scrubbing found mismatches. Action: schedule ceph pg repair <pgid> during low load and monitor.

Replication Size and min_size

Size and min_size directly control durability and availability:

• size: How many copies exist when healthy. Raising size increases durability but triggers backfill; ensure capacity and bandwidth first.
• min_size: The floor for serving writes. Lowering min_size can restore write availability during failures but raises risk. Prefer bringing OSDs back and only adjust min_size with clear limits and rollback.

Safe adjustments:

• Read current: ceph osd pool get <pool> size and ceph osd pool get <pool> min_size.
• Plan headroom: verify ceph osd df for capacity and cluster load.
• Change narrowly: adjust a single non-critical pool first, measure impact, then proceed.

Safe Commands and Checks

Start with read-only diagnostics:

• Cluster health: ceph -s, ceph health detail
• PG overview: ceph pg stat, ceph pg dump pgs_brief
• Focus a pool: ceph pg ls-by-pool <pool> degraded, ceph pg <pgid> query
• OSD layout and capacity: ceph osd tree, ceph osd df
• Service status and logs: systemctl status ceph-osd@<id>, journalctl -u ceph-osd@<id> -n 200
• Pool settings and autoscaling: ceph osd pool ls detail, ceph osd pool get <pool> size, ceph osd pool get <pool> min_size, ceph osd pool autoscale-status

Maintenance safeguards:

• Temporarily prevent rebalancing while investigating a known issue: ceph osd set noout
• Remove the safeguard immediately after: ceph osd unset noout

Avoid:

• Destroying or removing OSDs during troubleshooting.
• Large, simultaneous changes to size, min_size, or PG counts on multiple pools.

Recovery Playbooks

Single OSD down

1. Confirm scope: ceph -s and ceph osd tree. Note affected PGs.
2. Check the service: systemctl status ceph-osd@<id>; read recent logs with journalctl.
3. Fix the cause: storage device, mount, permissions, or network.
4. Verify recovery: ceph -s until PGs return to active+clean.

Multiple OSDs or a host down

1. Check host power, NIC links, and time sync.
2. If the outage is temporary and you are actively fixing it, set ceph osd set noout to avoid churn; unset as soon as the host is back.
3. Once OSDs are up, monitor backfill speed and client impact.

Degraded after maintenance

1. Ensure ceph osd unset noout is applied.
2. Confirm all intended OSDs are in and up.
3. Track ceph pg stat for backfill completion.

Long-running backfill or recovery

1. Check capacity headroom with ceph osd df; nearfull slows movement.
2. Look for overloaded OSDs in ceph osd df and slow request logs.
3. Defer heavy client IO if possible to let recovery finish faster.

Inconsistent PGs after scrub

1. Schedule repair during low load: ceph pg repair <pgid>.
2. Verify that the PG returns to active+clean; recheck health detail.

PG count alignment

1. Prefer autoscaling when available: ceph osd pool autoscale-status.
2. If manual changes are necessary, adjust PG counts gradually and during low load only.

Local Pilot Plan

Keep the first implementation narrow, measurable, and easy to inspect locally before any broad rollout.

Pilot steps

1. Create a non-critical test pool with a small PG count and default replication size.
2. Write sample data; capture baselines: ceph -s, ceph pg stat, and client IO throughput.
3. In a lab or maintenance window, briefly stop a single OSD process to observe PG transitions to degraded and back as it recovers.
4. Raise the test pool size by +1 and confirm backfill starts and completes without unexpected impact.
5. Restore original settings and verify the cluster returns to active+clean.

Success criteria

• PG states follow expected transitions and recover within a defined time window.
• No impact to critical pools or client SLAs.
• Operators can read, run, and explain each step confidently.

Conclusion

Operational safety in Ceph comes from a disciplined sequence: read health first, map pool and PG impact, fix the smallest thing that restores safety, and verify recovery to active+clean. Keep replication size and min_size aligned with risk tolerance, prefer read-only diagnostics before changes, and validate any adjustment in a narrow pilot before applying it more widely. This approach reduces rework and builds confidence in your cluster's behavior under stress.