Ceph performance tuning with practical examples: practical implementation guide

Ceph performance tuning is not a one-time tweak; it is an iterative, observable practice. Whether you run RBD for virtual machines on Proxmox, RGW for object workloads, or CephFS for shared files, your cluster's throughput and latency depend on a chain: client queues, network, OSD device behavior, CRUSH placement, recovery and scrubbing, and background compaction. This guide shows a safe way to inventory versions, topology, and device capabilities; identify bottlenecks with concrete checks; apply reversible configuration changes with practical examples; verify results, understand failure modes, and roll back; and keep a short, repeatable checklist for day-2 operations. Our recommendation throughout: prove improvements with a narrow, measurable pilot before broad rollout. The examples below use constructed numbers to illustrate expectations; adjust to your environment.

Version and Environment Inventory

Before tuning, make sure the cluster is healthy and you know what you are changing.

Record versions and topology:

ceph versions
ceph -s
ceph osd tree
ceph osd df tree
ceph mon stat
ceph osd pool autoscale-status

Confirm placement and pool settings:

• Replicated vs erasure coded pools.
• PG autoscaler mode (on, off, warn) per pool.
• Device classes (HDD, SSD, NVME) and CRUSH rules mapping pools to classes.

ceph osd crush tree
ceph osd crush class ls
ceph osd crush rule dump

Baseline latency and throughput (constructed examples):

• Client-facing: a small fio test for 4k random read/write IOPS and 128k sequential throughput.
• In-cluster: rados bench for object-level ops.

Example commands (use a non-production client or a quiet window):

# RADOS write then sequential read without cleanup (synthetic, object-level)
rados bench -p <pool> 60 write --no-cleanup
rados bench -p <pool> 60 seq

# RBD block-level test using a mapped device (constructed example)
rbd create --size 10G <pool>/tune-test
rbd map <pool>/tune-test
mkfs.xfs -f /dev/rbd/<id>
mount /dev/rbd/<id> /mnt/rbdtest
fio --name=randread --filename=/mnt/rbdtest/f --rw=randread --bs=4k \
 --iodepth=32 --numjobs=4 --size=2G --time_based=1 --runtime=60 --direct=1
fio --name=seqwrite --filename=/mnt/rbdtest/f2 --rw=write --bs=128k \
 --iodepth=32 --numjobs=2 --size=2G --time_based=1 --runtime=60 --direct=1
umount /mnt/rbdtest
rbd unmap /dev/rbd/<id>
rbd rm <pool>/tune-test

Check OSD op latency and busy disks:

ceph osd perf
ceph osd dump | grep noscrub -n || true

If health is not OK, fix health first. Tuning while the cluster is degraded often masks the real issue.

Key metrics and how to collect them

Safe Configuration Path

A safe path prevents cluster-wide regressions and preserves the ability to revert quickly.

1. Formulate a single hypothesis

Examples:

• Increasing client queue depth will raise throughput for sequential writes until the OSD device saturates.
• Lowering recovery/backfill limits during peak hours will reduce client tail latency.
• Enabling jumbo frames end-to-end will increase RADOS throughput for large I/O.

1. Apply a scoped change

• Prefer a single client or a single OSD for pilot changes.
• For Ceph daemon settings, test on one OSD using a temporary override; only then persist cluster-wide.

Temporary override on a single OSD (constructed example OSD.3):

# Example: reduce active recovery ops to lower client contention
ceph tell osd.3 injectargs '--osd_recovery_max_active 1 --osd_max_backfills 1'

Persist after successful pilot:

# Persist cluster-wide only if pilot helps
ceph config set osd osd_recovery_max_active 1
ceph config set osd osd_max_backfills 1

Rollback:

# Remove persistent keys or restore previous values
ceph config rm osd osd_recovery_max_active
ceph config rm osd osd_max_backfills

1. Measure, compare, decide

• Re-run the exact same benchmark and workload slice.
• Check Ceph health and OSD latencies for side effects.
• If it helps and does not harm, persist; otherwise revert.

Practical Tuning Examples

The examples below are scoped and reversible. Replace placeholders with your pool, device, and host names.

1) Throttle recovery to protect client latency during peak

When recovery/backfill and client I/O compete on the same OSDs, p95/p99 latency spikes. You can lower recovery intensity during business hours and relax it off-peak.

Pilot on one OSD:

ceph tell osd.<id> injectargs '--osd_recovery_max_active 1 --osd_max_backfills 1 --osd_recovery_op_priority 1'

Observe for 15-30 minutes:

ceph osd perf
ceph -s
rados bench -p <pool> 60 seq

Expected (constructed example): client p99 latency improves by 20-40% during peaks; recovery slows proportionally.

If positive, persist cluster-wide:

ceph config set osd osd_recovery_max_active 1
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_op_priority 1

Rollback:

ceph config rm osd osd_recovery_max_active
ceph config rm osd osd_max_backfills
ceph config rm osd osd_recovery_op_priority

Tradeoff: recovery windows lengthen; plan off-peak higher limits or a scheduled toggle.

2) Right-size OSD memory target (BlueStore)

On BlueStore, cache and RocksDB/BlueFS need memory headroom. Too small and you thrash; too big and the kernel OOM killer risks terminating daemons.

Pilot on one OSD (constructed values):

# Example: 8 GiB per OSD on a 128 GiB host running 8 OSDs
ceph tell osd.<id> injectargs '--osd_memory_target=8589934592'

Observe resident set size (RSS), compaction stalls, and OSD op latency:

ceph osd perf
ceph daemon osd.<id> perf dump | jq '.bluefs | .[]?'

If stable and latency improves, persist:

ceph config set osd osd_memory_target 8589934592

Rollback:

ceph config rm osd osd_memory_target

Notes:

• Avoid setting memory targets so high that total per-host OSD memory exceeds physical RAM minus OS overhead.
• Keep bluestore_cache_autotune enabled unless you have a strong reason to micro-manage.

3) Client-side RBD queue depth for throughput testing

For sequential throughput, client queue depth and parallelism matter. With a mapped RBD device:

# Constructed example: test higher depth for 128k writes
fio --name=seqwrite --filename=/mnt/rbdtest/f --rw=write --bs=128k \
 --iodepth=128 --numjobs=4 --size=4G --time_based=1 --runtime=120 --direct=1

Compare against iodepth=32 or fewer jobs. Watch for increasing latency without throughput gains; that indicates either the device or network is saturated.

Rollback: none needed; this is a client-only benchmark change.

4) NVMe and HDD scheduler and CPU governor

Fast NVMe often benefits from the none or mq-deadline scheduler; HDDs may prefer mq-deadline.

Check and set (per device):

# Inspect
cat /sys/block/nvme0n1/queue/scheduler

# Set (runtime test; persist via udev rule if desired)
echo none > /sys/block/nvme0n1/queue/scheduler

Set performance CPU governor to minimize frequency scaling latency:

# Inspect current
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
# Set for all CPUs (runtime)
for c in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do echo performance > "$c"; done

Verify OSD latencies and fio throughput. Rollback by restoring previous values (e.g., ondemand governor and prior scheduler).

5) Network MTU validation and sysctls

Large I/O benefits from lower per-packet overhead if jumbo frames are consistently enabled end-to-end.

Validate MTU across the path (constructed example MTU 9000):

ping -M do -s 8972 <peer-ip>

If successful cluster-wide, set interface MTU on all Ceph hosts and switches. If any hop cannot support jumbo frames, keep MTU at 1500 to avoid fragmentation and blackholes.

Optional runtime sysctls (test-first):

sysctl -w net.core.rmem_max=134217728
sysctl -w net.core.wmem_max=134217728
sysctl -w net.core.netdev_max_backlog=250000

Persist by writing to /etc/sysctl.d/99-ceph-net.conf only after pilot success. Rollback by restoring the prior file or using sysctl -w with previous values.

6) Pool PG autoscaling and device classes

Ensure pools use appropriate device classes (e.g., NVME for hot RBD) and reasonable PG counts.

Inspect:

ceph osd pool autoscale-status
ceph osd crush class ls
ceph osd crush tree

If a pool is on the wrong device class, create a CRUSH rule that targets the correct class and apply it during a low-traffic window. This triggers data movement; plan for recovery bandwidth.

Enable autoscaling on pools that change in size frequently:

ceph osd pool set <pool> pg_autoscale_mode on

Disable autoscaling only if you have a specific reason and a manual PG plan.

Verification and Diagnostics

Verify both that the intended metric improved and that no new problems appeared.

1. Health and latency checks

ceph -s
ceph osd perf
ceph health detail

1. Compare synthetic benchmarks

• Repeat the exact same rados bench and fio invocations.
• Track p95/p99 latency and throughput deltas (constructed example: +15% sequential write throughput with stable latency).

1. Watch recovery and scrubbing

ceph -s | grep -E 'recovery|backfill|scrub'

Ensure recovery is not starved indefinitely. If you lowered recovery, schedule windows to restore defaults off-peak.

1. Check OSD-level signals

# Example: check compaction and BlueFS stats on a single OSD
ceph daemon osd.<id> perf dump | jq '{bluefs: .bluefs, bluestore: .bluestore}'

Look for compaction stalls or unusually high kv_sync_lat or state transitions that coincide with regressions.

Safe tunables summary

Failure Modes and Recovery

Knowing what can break makes you faster at recovery.

1. MTU mismatches

Symptoms: timeouts, variable throughput, or stalls on large transfers.

• Detection: ping -M do -s 8972 fails randomly.
• Fix: standardize MTU across servers and switches; if not feasible, revert to 1500 everywhere.

1. Over-throttled recovery

Symptoms: PGs remain in recovering/backfilling for too long; prolonged data movement windows.

• Detection: ceph -s shows persistent recovery; OSDs are idle for client ops but data not rebalanced.
• Fix: increase osd_recovery_max_active and osd_max_backfills gradually; schedule off-peak rebalancing.

1. Too-low OSD memory target

Symptoms: rising op latency, compaction stalls, or OOM if memory is overcommitted elsewhere.

• Detection: ceph osd perf degrades; dmesg shows memory pressure.
• Fix: raise osd_memory_target or reduce concurrent workload; ensure host RAM headroom.

1. PG changes trigger heavy backfill

Symptoms: large data movement, degraded client performance.

• Detection: ceph -s indicates remapping; network/disk saturated by backfill.
• Fix: apply PG changes in steps during low-traffic windows; temporarily relax recovery throttles for faster completion, then restore.

1. Misleading benchmarks

Symptoms: fio numbers look excellent during first run but drop later.

• Detection: missing --direct=1 leads to page cache effects.
• Fix: always use --direct=1 for synthetic disk tests; clear caches only in test hosts where safe.

Rollback and recovery checklist

• Ceph config keys: ceph config rm to remove; or restore known-good values.
• Sysctls: restore prior /etc/sysctl.d file and sysctl --system, or use sysctl -w with previous values.
• Schedulers/governors: write prior values back to /sys files or revert udev rules.
• CRUSH rule or pool changes: pause, schedule for a low-traffic window, or revert to the previous rule (expect data movement).
• Re-enable scrubbing if you disabled it for testing: ceph osd unset noscrub; ceph osd unset nodeep-scrub.

Operations Checklist

Use this as a practical, repeatable procedure.

1. Inventory and baseline

• Capture ceph versions, ceph -s, ceph osd perf.
• Note pool types, autoscale modes, device classes.
• Run quick rados bench and a small fio suite with direct=1.

1. Hypothesize and scope

• Choose exactly one bottleneck hypothesis (e.g., recovery vs client contention, network overhead, scheduler choice).
• Pick the smallest pilot (one OSD or one client) and a 30-60 minute observation window.

1. Change safely

• Apply a temporary override (e.g., injectargs on one OSD) or client-only test parameters.
• Avoid cluster-wide updates until pilot success.

1. Verify

• Re-run identical tests; compare p95/p99 latency and throughput.
• Check ceph -s, ceph osd perf, and PG states.

1. Decide and persist

• If improvement is clear and side effects are acceptable, persist via ceph config set or OS config files.
• Document new defaults and when to toggle (e.g., peak vs off-peak recovery settings).

1. Rollback if needed

• Use ceph config rm or restore known-good values.
• Revert sysctls and scheduler/governor changes.

1. Schedule routine reviews

• Re-run the baseline quarterly or after hardware/workload shifts.
• Trim tunings that no longer help.

Conclusion

Ceph performance improves fastest when changes are small, safe, and measurable. Start by inventorying your versions, topology, and workloads; establish a clean health baseline; and test one knob at a time on a narrow pilot. Use temporary overrides to prove a gain, then persist only what helps and document clear rollback steps. Over time, your cluster will converge on well-understood defaults for your hardware and workloads, and you will have a lightweight routine to keep it that way. Consistent, evidence-driven tuning beats speculative changes every time, and the discipline of pilot-then-persist protects production workloads while still delivering real gains.