Ceph capacity planning with practical examples: practical implementation guide

Intro

Ceph rewards good planning. Get it right and you onboard workloads smoothly, absorb failures without drama, and scale predictably. Get it wrong and you fight rebalancing storms, long recoveries, and surprise full states.

This guide gives you a practical workflow to size Ceph clusters, with simple formulas, concrete examples, and the operational signals that tell you when to add nodes. You will learn how to translate usable capacity goals into raw storage, choose replication or erasure coding, set safe headroom, and validate your plan with a small pilot you can inspect locally.

Workflow Overview

Use this repeatable sizing process for new clusters or expansions:

1. Define the workload

• Data services: RBD (block), CephFS (files), RGW (object)
• Capacity goal (usable TB) and growth horizon (months)
• IO profile: read/write mix, typical object or block size, throughput vs IOPS priority

1. Choose data protection

• Replication (e.g., 3x) for simplicity and small-block write performance
• Erasure coding (k+m, e.g., 8+2) for better capacity efficiency on large objects/streams
• Failure domain for placement: host or rack

1. Set operational headroom

• Baseline free raw headroom: 20-40%. A practical default is 30%
• Extra headroom if you expect frequent node maintenance or bursty writes

1. Convert usable to raw

• Replication: Raw = Usable * R / (1 - headroom)
• EC k+m: Raw = Usable * ((k+m)/k) / (1 - headroom)

1. Map to nodes and OSDs

• Pick drive size and media (HDD, SSD, NVMe)
• Plan OSD count per node, CPU, RAM, and network
• Ensure enough hosts for the failure domain and coding scheme

1. Plan PGs and pools

• Target 50-200 PGs per OSD; 100 is a common starting point
• Distribute PGs across pools and round to a power of two (e.g., 1024, 2048)

1. Check scaling signals and thresholds

• Define warn/nearfull/full ratios and alerting
• Define recovery time SLO after a node loss
• Document trigger points to add capacity

1. Validate with a small pilot

• Deploy a minimal cluster that exercises your protection scheme
• Run realistic workloads and a failure test
• Measure recovery impact and tune backfill/recovery throttles

Core sizing formulas

Key concepts

• Raw capacity: sum of all drive capacities in the cluster
• Usable capacity: what applications can store, after data protection overhead
• Headroom: reserved raw free space to absorb growth, rebalancing, and failures

Replication R

• Usable = Raw / R
• Raw needed = Usable_target * R / (1 - headroom)

Erasure coding k+m

• Overhead factor = (k+m)/k
• Usable = Raw * k/(k+m)
• Raw needed = Usable_target * ((k+m)/k) / (1 - headroom)

Failure domain requirements

• Replication R across hosts: need at least R hosts
• EC k+m across hosts: to maintain host-level fault tolerance, plan at least k+m hosts so each shard can land on a different host

Headroom guidance

• Start at 30% raw headroom. Increase for HDD-heavy write clusters or frequent maintenance windows
• Keep per-OSD free space healthy; very full OSDs rebalance slowly and fragment more

Practical examples

Example A: 100 TB usable, RBD, 3x replication, 30% headroom

• Raw needed = 100 TB * 3 / 0.70 = 428.6 TB. Round up to 430 TB+
• Node layout option: 6 nodes, each 12 x 8 TB HDD = ~96 TB raw per node, 576 TB total
• Failure domain: host. With 3x replication and 6 hosts, replicas land on different hosts
• PG planning: suppose 72 OSDs (6 nodes x 12). Target 100 PGs/OSD => ~7200 PGs total. If you run 3 pools, ~2400 PGs/pool; round to 2048 or 2560 based on preference
• Resource sketch per node: HDD OSDs benefit from SSD for RocksDB/WAL; plan 2-4 GB RAM per OSD plus base for services; plan CPU so each 3-5 HDD OSDs has ~1 core available; dual 25 GbE or better for public+cluster traffic is a solid baseline

Example B: 300 TB usable, large-object workload, EC 8+2, 30% headroom

• Overhead factor = (8+2)/8 = 1.25
• Raw needed = 300 TB * 1.25 / 0.70 = 535.7 TB. Round up to 540 TB+
• Failure domain: host. To place 10 EC shards on distinct hosts, plan at least 10 hosts
• Node layout option: 10 nodes, each 12 x 6 TB HDD = ~72 TB raw per node, 720 TB total. The extra raw covers headroom and growth
• PG planning: with 120 OSDs (10 x 12) and target 100 PGs/OSD => ~12,000 PGs total. If you run 2 pools, ~6000 PGs/pool; round to 6144
• Throughput note: EC decoding adds CPU; plan more CPU per OSD vs replication. Keep a fast network fabric (25/40/100 GbE) and test recovery impact

Scaling signals and limits

Add capacity or rebalance when you see:

• Free raw headroom trends below 30% within your planning horizon
• Nearfull or full warnings on OSDs or pools
• Recovery after a host loss exceeds your SLO (for example, cannot restore full redundancy within your acceptable window)
• PGs per OSD drift far from target (for example, much higher than 100) or total PGs approach your comfort ceiling
• OSD latency grows during normal load (client I/O starts competing with constant recovery or backfill)

Common practical limits and checks

• Hosts: keep at least R hosts for replication and at least k+m hosts for EC at the chosen failure domain. More hosts smooth rebalancing
• OSD density: HDD nodes often run 8-24 OSDs comfortably; NVMe OSDs drive higher CPU and network, so size cores and NICs accordingly
• CPU: budget more CPU for EC (encoding/decoding) and for SSD/NVMe OSDs
• Memory: plan a healthy base for system and monitors/managers, then add 2-4 GB per OSD as a practical starting point
• Network: ensure enough east-west bandwidth for backfill and recovery without starving clients; separate or prioritize cluster traffic if needed
• PG count: avoid extreme PG counts per OSD; rebalance when expanding so PG targets remain in range

Operational safety margins

Headroom policy

• Baseline 30% raw headroom. For heavy-write HDD clusters or frequent maintenance, use 35-40%

Failure absorption

• Plan to lose one host without crossing nearfull thresholds. A simple test: subtract one host worth of raw and verify remaining raw usable still stays below your warn/nearfull ratios after rebuild

Rebalancing and maintenance

• Throttle recovery and backfill to protect client latency during business hours; allow faster recovery during maintenance windows
• Keep OSDs reasonably balanced by weight and drive health; uneven weights cause hot spots and early nearfull states

Data protection choices

• Replication for small-block latency and simpler ops; EC for capacity efficiency with large objects or sequential workloads. Mixed clusters are common, but assign pools carefully

Placement scope

• If you need rack-level failure tolerance, set the failure domain to rack and ensure enough racks for R or k+m

Upgrade and expansion

• Maintain spare capacity to drain a host for hardware replacement or OS updates without entering nearfull

Local Pilot Plan

Goal: validate sizing and recovery behavior safely on a small footprint you can inspect locally.

Pilot scope

• 3 or 4 hosts, each with a modest number of drives
• Protection: 3x replication for a first pass. If you plan EC in production, add a second pilot with EC on large-object pools
• Target: 5-10 TB usable so you can run fill, steady-state, and failure tests in hours, not weeks

Pilot steps

1. Define targets: usable capacity, headroom (30%), warn/nearfull/full ratios, and a recovery time SLO after one host loss
2. Deploy the cluster with host as failure domain; create one RBD pool for small-block tests and, if relevant, one EC pool for large objects
3. Set PGs so you land near 100 PGs/OSD across your pools
4. Run a realistic write workload to 60-70% of target usable and observe client latency and OSD memory usage
5. Induce a failure: stop one host. Observe recovery start, bandwidth, and client impact. Adjust recovery/backfill limits to meet your SLO
6. Recheck capacity math with one host absent to confirm you remain under nearfull during rebuild
7. Document thresholds and the trigger to add nodes (for example, forecasted headroom < 30% within 90 days or recovery time SLO risk)

Success criteria

• Recovery time and client impact meet your SLO
• Alerts fire at the expected thresholds and are actionable
• PG targets remain in range after adding or removing a host

Conclusion

Ceph capacity planning is straightforward when you follow a clear workflow:

• Characterize workloads and choose data protection
• Convert usable to raw with a realistic headroom policy
• Map capacity to hosts and OSDs, set PGs, and verify failure-domain requirements
• Watch the right scaling signals and define clear add-capacity triggers
• Validate everything with a small, measurable pilot you can inspect locally

Next steps: build a simple sizing sheet with the formulas in this guide, run the pilot to calibrate recovery and thresholds, and schedule regular capacity reviews so you always add nodes before headroom gets tight.