Ceph Capacity Planning with Practical Examples

Capacity planning in Ceph is an operational discipline that aligns durability methods, failure domains, workload patterns, placement group sizing, and safe utilization margins. It is not simply a matter of counting disks. This guide shows how to estimate usable capacity, choose safety headroom, read scaling signals, verify results with live commands, and recover safely when limits are reached. You will walk through constructed examples with clear math that you can adapt to your environment.

What You Will Get

• A clear inventory to gather before sizing.
• Safe, step-by-step decisions for durability, placement groups, and headroom.
• Two constructed examples (replicated and erasure-coded) with exact computations.
• Verification commands and expected results to confirm the plan.
• Common failure modes with rollback and recovery guidance.
• A simple, repeatable checklist for ongoing operations.

Version and Environment Inventory

Before sizing, gather the facts you will rely on during calculations and verification.

Prerequisites:

• Basic Linux administration.
• Shell access to a Ceph monitor node with the ceph CLI.
• Cluster admin permissions.

Inventory checklist:

• Ceph version.
• Network fabric (1/10/25/40/100 GbE).
• Node count and failure domains (chassis, rack, host).
• OSD media (HDD, SSD, NVMe) and sizes; BlueStore DB/WAL placement.
• Target use (RBD, CephFS, RGW) and typical object size distribution.
• Expected data durability (replicated 3x, replicated 2x for dev, or erasure coding k+m).
• Pools and expected data mix per pool.
• Service SLOs (recovery time after host loss, upgrade windows).

Useful commands to capture inventory:

ceph versions
ceph status
ceph osd tree
ceph osd crush tree
ceph df detail
ceph osd df tree
ceph osd pool ls detail
ceph osd pool autoscale-status

Safe Configuration Path

Use a scoped, stepwise path that avoids overcommitting capacity and constrains risk.

Step 1. Choose Durability and Failure Domain

• Replicated pools: Simple, good for small objects and write-heavy workloads. Common replication size is 3.
• Erasure-coded pools: Higher efficiency for large objects and archival data. Example: 8+2 (k=8 data, m=2 parity). Writes require a replicated cache tier or careful client tuning for small objects.
• Failure domain: Use host as the minimum. For erasure coding, ensure k+m stripes can be distributed across the failure domain so that one host loss does not take multiple stripes from the same set.

Step 2. Convert Raw to Usable Capacity

• Replicated R: usable_fraction = 1 / R. Example: 3x => 0.333.
• Erasure-coded k+m: usable_fraction = k / (k + m). Example: 8+2 => 0.8.
• Apply filesystem and BlueStore overhead. Plan a 10% reserve for metadata, WAL/DB, and alignment.

Step 3. Reserve Operational Headroom

• Target steady-state fill: 60–75%. This preserves performance and gives room to absorb failures and recovery. Many operators choose ~70% as a planning limit.
• Do not size clusters such that a single host loss pushes any OSD above nearfull thresholds.

Step 4. Placement Groups (PGs)

• Enable PG autoscaler for new clusters and pools.

ceph osd pool set <pool> pg_autoscale_mode on

• For mixed pools, set relative shares with target_size_ratio.

ceph osd pool set <pool> target_size_ratio <0.0-1.0>

• Healthy total PGs per OSD typically fall in a modest range. For planning, aim around 50–150 total PGs per OSD across all pools, letting autoscaler choose specifics. Avoid extremes that create peering storms.

Step 5. Configure Safe Full-Ratio Thresholds and Alerting

• Inspect current ratios:

ceph config get mon mon_osd_nearfull_ratio
ceph config get mon mon_osd_backfillfull_ratio
ceph config get mon mon_osd_full_ratio

• Plan your capacity so steady-state remains well below nearfull. If defaults are tight for your risk posture, set alerting thresholds in your monitoring stack accordingly rather than changing cluster hard limits prematurely.

Step 6. Validate Network and Recovery Budgets

• Choose a target recovery time after a host loss (for example, recover to clean within 4–8 hours).
• Ensure network bandwidth, disk throughput, and backfill limits can meet that target at your planned utilization. If recovery estimates are marginal, increase headroom or add capacity before load.

Practical Sizing Examples

These are constructed, hypothetical examples to show the math you can adapt.

Example A: Small 3-Host Replicated RBD Cluster

• Hosts: 3
• OSDs per host: 6 × 8 TB HDD (raw 48 TB/host)
• Total raw: 3 × 6 × 8 TB = 144 TB
• Planning overhead (metadata, BlueStore): 10% => effective raw 129.6 TB
• Replication: 3x => usable = 129.6 / 3 = 43.2 TB
• Operational headroom: 70% target fill => data-at-rest budget = 43.2 × 0.70 = 30.24 TB

Interpretation: Plan to store about 30 TB of data to keep the cluster fast and safe, even though naive math might suggest 48 TB.

What if one host fails? You lose 48 TB raw temporarily. Ensure that remaining capacity at 70% fill can accept backfill without crossing nearfull thresholds. Simulate by estimating the largest pool share backfill impact and check headroom before go-live.

Example B: 6-Host Cluster with Erasure-Coded Archival Pool (8+2) and Small Replicated Metadata Pool

• Hosts: 6
• OSDs per host: 10 × 12 TB HDD (raw 120 TB/host)
• Total raw: 6 × 10 × 12 TB = 720 TB
• Planning overhead: 10% => effective raw 648 TB
• Pools:
• EC pool (8+2) for large objects, expected 90% of data
• Efficiency 0.8 => usable = 648 × 0.9 × 0.8 = 466.56 TB
• Replicated pool (size 3) for metadata, 10% of data
• Efficiency 0.333 => usable = 648 × 0.1 × 0.333 = 21.58 TB
• Combined usable = 488.14 TB
• Operational headroom: 70% target fill => data-at-rest budget = 488.14 × 0.70 = 341.70 TB

Interpretation: With mixed pools, always weight by expected data share and each pool's durability efficiency. If your actual data share drifts, revisit target_size_ratio and capacity planning.

Durability Scheme Comparison (Planning Guardrails)

Scaling Signals and Planning Limits (Constructed Guardrails)

Capacity Summary for the Examples

Verification and Diagnostics

After planning and initial implementation, verify your assumptions with live data. These checks are safe to run online.

Check Versions, Health, and Utilization

ceph versions
ceph status
ceph health detail
ceph df detail
rados df

Inspect OSD Capacity, Fullness, and Distribution

ceph osd df tree
ceph osd perf
ceph osd tree

Confirm Pool Durability and Autoscaling Settings

ceph osd pool ls detail
ceph osd pool autoscale-status
# Example: ensure autoscale is on
ceph osd pool set mypool pg_autoscale_mode on
# Example: set relative data share for autoscaler
ceph osd pool set bigobjects target_size_ratio 0.8
ceph osd pool set metadata target_size_ratio 0.2

Inspect Fullness Thresholds and Confirm Alerting

ceph config get mon mon_osd_nearfull_ratio
ceph config get mon mon_osd_backfillfull_ratio
ceph config get mon mon_osd_full_ratio

Expected Results When Within Plan

• ceph status shows HEALTH_OK or only transient warnings during scrubs.
• ceph df shows total used below your target steady-state (for example, ≤ 70%).
• No OSDs are nearfull; distribution across hosts is even (watch for outliers in ceph osd df tree).
• Pools have the intended size or k+m profile, and autoscaler status is not suggesting extreme PG changes.
• No persistent slow requests; recovery operations are not throttled during quiet periods.

If any check fails, prioritize: fullness and durability first, then latency, then PG balance.

Failure Modes and Recovery

Common failure modes, risks, and safe recovery actions.

1. Nearfull or Full OSDs

• Signal: ceph health detail shows nearfull/full warnings.
• Risks: Writes blocked, degraded redundancy, prolonged recovery windows.
• Actions (in order):
• Pause growth: stop data onboarding.
• Add capacity: safest option; add OSDs/nodes and let backfill restore balance.
• Create temporary high-water alerts lower than nearfull to buy time.
• As a last resort, move heavy pools or set target_size_ratio to nudge autoscaler/CRUSH rebalance.
• Avoid: lowering hard full ratios to hide symptoms.

2. Single Host Failure Extends Recovery Beyond Budget

• Signal: ceph status remains DEGRADED/RECOVERING for too long; client latency rises.
• Risks: Exposure to second failure, client timeouts.
• Actions:

# Temporarily increase backfill bandwidth if safe for clients
ceph config set osd osd_recovery_max_active 4
ceph config set osd osd_max_backfills 2

• Monitor client impact (latency, IOPS) while recovery runs.
• Long-term: add OSDs or reduce steady-state fill target (e.g., from 75% to 65%).
• Rollback: return recovery limits to defaults after heal to avoid chronic contention.

3. PG Mis-sizing or Imbalanced Pools

• Signal: Autoscaler status suggests large PG increases; some OSDs carry too many PGs.
• Risks: Peering storms during changes, uneven data movement.
• Actions:
• Enable autoscaler on all pools and set realistic target_size_ratio per pool.
• Stagger PG changes during quiet periods; watch ceph -w.
• Rollback: if changes overshoot, reduce target_size_ratio gradually and let autoscaler settle.

4. Hot Objects or Skewed Data Placement

• Signal: A few OSDs show much higher usage or latency.
• Risks: Localized full OSDs despite global headroom.
• Actions:
• Rebalance CRUSH weights slightly to even usage; verify with ceph osd df tree.
• For RGW or CephFS, review sharding and directory layout.
• Rollback: restore previous CRUSH weights if imbalance reverses.

5. BlueStore DB/WAL Spillover to Slow Media

• Signal: ceph health detail mentions BlueFS spillover; OSD latency rises.
• Risks: Latency-sensitive pools slow down under load.
• Actions:
• Add or resize DB/WAL on faster media; migrate selected OSDs.
• Reduce compaction and recovery pressure during migration windows.
• Rollback: if performance does not improve, revert the affected OSDs to prior layout and investigate workload patterns before retrying.

Guardrails for Any Risky Change

• Take only one significant step at a time.
• Keep noout and norebalance flags off by default; if you must use them briefly, document start time and clear them immediately after work.
• Verify health after each step; proceed only when the cluster stabilizes.

Operations Checklist

Use this repeatable checklist monthly or before major growth events.

Plan and Review

• Recompute usable capacity with current OSD count, pool mix, and the chosen headroom (e.g., 70%).
• Confirm that a single host loss does not drive any OSD near nearfull thresholds.
• Review expected data mix per pool; adjust target_size_ratio if shares drift.

Validate Health and Performance

• ceph status and ceph health detail are clean; investigate recurring warnings.
• ceph df detail shows fill below planning target; OSD distribution is even.
• ceph osd df tree shows no outliers in fullness or latency.
• Verify recovery performance by simulating constraints during a low-traffic window (for example, temporarily setting lower recovery throttles and observing).

Check Scaling Signals

• If steady-state fill > 70% or recovery edges past your time budget, schedule node or OSD additions.
• If PGs per OSD approach extreme values, let autoscaler converge or split workloads across pools.

Document and Decide

• Record calculations, assumptions (durability, headroom, recovery budget), and observed metrics.
• Decide: add capacity, reduce growth rate temporarily, or re-tier some data to a more efficient pool.

Conclusion

Ceph capacity planning is an ongoing practice: choose a durability scheme that matches workload characteristics, convert raw to usable with realistic overhead, reserve operating headroom, and watch scaling signals that inform when to add capacity. Start with a narrow, measurable pilot to validate your math, then grow incrementally. Verify with ceph df, autoscaler status, and recovery timing after a host loss. When in doubt, prioritize safety: add capacity before you hit nearfull, and keep enough free space to heal quickly. With these practices and the example calculations in this guide, you can scale Ceph predictably while protecting performance and durability.