Ceph Configuration Mistakes: A Practical Guide to Safe Changes, Validation, and Rollback

Intro

Ceph is a powerful distributed storage system that offers remarkable flexibility, but that same flexibility can lead to configuration mistakes that cause performance degradation, data unavailability, or even data loss. This article provides practical examples of common mistakes and shows how to validate changes safely, implement them with minimal risk, and roll back when necessary. By following these guidelines, you can avoid costly errors and maintain a healthy Ceph cluster.

Version and Environment Inventory

Before making any configuration changes, you must understand your environment. This includes the Ceph version, cluster topology, hardware specifications, and current configuration settings. A solid inventory prevents mistakes caused by version-specific behavior or incompatible settings.

First, check the Ceph version on all nodes:

ceph --version

Expected output example:

ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)

Next, review the cluster status and topology:

ceph status
ceph osd tree

These commands show the cluster health, monitor quorum, OSD layout, and placement groups. Document this baseline.

Also, inspect the current configuration:

ceph config dump

This shows all configuration options, their sources, and values. Pay attention to overrides in the central config database versus local ceph.conf files.

Finally, verify prerequisites: ensure time synchronization (NTP) is working, network latency between nodes is low, and sufficient disk space is available for OSDs and monitors. These factors affect configuration behavior.

Safe Configuration Path

A safe configuration path involves making changes in a controlled, scoped manner. Avoid modifying multiple settings at once. Instead, change one parameter, validate its effect, and then proceed.

Use the Ceph configuration database for centralized changes. For example, to set the OSD recovery priority to a lower value to reduce impact during recovery:

ceph config set osd osd_recovery_op_priority 2

This command changes the setting for all OSDs. To limit the change to a specific OSD, use a mask:

ceph config set osd.0 osd_recovery_op_priority 2

Always test changes on a non-production or less critical subset first. For instance, if you have a test OSD or a canary node, apply the change there.

Document every change: record the original value, the new value, the reason, and the date. This documentation is essential for rollback and troubleshooting.

For changes in ceph.conf on individual nodes, edit the file and restart the relevant daemon. However, prefer the central config database to avoid inconsistency.

Example of a common mistake: changing osd_pool_default_size (replication factor) without considering the impact on data durability. A safe approach is to first create a test pool, change the setting for that pool only, and observe behavior.

ceph osd pool create testpool 128 128
ceph osd pool set testpool size 2

Then verify the pool's settings with ceph osd pool get testpool size.

Verification and Diagnostics

After making a configuration change, verify that it took effect and that the cluster remains healthy. Use the following commands:

• Check the effective configuration for a daemon:

ceph config get osd.0 osd_recovery_op_priority

Expected output:

2

• Monitor cluster health:

ceph -s

Look for HEALTH_OK. If warnings appear, investigate before proceeding.

• Check performance metrics. For example, observe recovery progress if you changed recovery settings:

ceph pg stat

Or use ceph osd perf to see OSD latency.

• Examine logs for errors. Ceph logs are in /var/log/ceph/. Use grep to filter for the daemon and time range.

Example: if you changed a monitor setting, check the monitor log:

tail -n 50 /var/log/ceph/ceph-mon.$(hostname -s).log

• For performance-sensitive changes, run benchmarks before and after. For RBD, you can use rbd bench or fio with the RBD kernel module.

Example RBD bench:

rbd bench --io-type write --io-size 4096 --io-threads 4 --io-total 1G myimage

Compare the results to the baseline. If performance degrades, consider rolling back the change.

Failure Modes and Recovery

Despite precautions, configuration mistakes can cause failures. Common failure modes include:

• Setting a value too low or too high, causing OSDs to fail or become unresponsive.
• Misconfiguring network settings, leading to mon or OSD communication failures.
• Changing pool settings that result in data unavailability or lost redundancy.

To recover, first identify the change that caused the issue. Check the configuration history if using a configuration management system, or review your change log.

Roll back the change using the same method you used to apply it. For central config database changes:

ceph config rm osd osd_recovery_op_priority

This removes the override, reverting to the default. If you set a specific value, set it back to the original:

ceph config set osd.0 osd_recovery_op_priority <original value>

If the cluster is in a bad state, you may need to restart affected daemons:

systemctl restart [email protected]

For monitors, ensure quorum is maintained. If a mon fails due to configuration, you can remove it from the monmap and re-add it after fixing the config.

Example recovery scenario: You set osd_max_backfills to 0 to stop backfilling, but then OSDs are stuck in a degraded state. Roll back by setting it to a sane value like 1:

ceph config set osd osd_max_backfills 1

Then monitor recovery with ceph -s.

Always have a backup of critical configuration files and the cluster's monmap. Test rollback procedures in a staging environment.

Operations Checklist

Use this checklist before and after making configuration changes to minimize mistakes and ensure quick recovery.

Following this checklist ensures a systematic approach to configuration management and reduces the risk of prolonged outages.

Common Pitfalls and How to Avoid Them

Even experienced administrators can make mistakes. Here are some common pitfalls and strategies to avoid them.

1. Changing multiple settings at once: It becomes difficult to identify which change caused a problem. Always change one parameter at a time and test thoroughly before moving to the next.

1. Ignoring version-specific behavior: Some configuration options have different defaults or semantics across Ceph releases. Always consult the documentation for your specific version and test in a staging environment.

1. Not backing up the configuration: Without a backup, rolling back becomes guesswork. Use ceph config dump to save the current configuration before making changes.

1. Applying changes globally without testing: A setting that works on one OSD may not be appropriate for all. Use daemon-specific masks to test on a subset first.

1. Overlooking hardware limitations: Configuration changes may have different effects depending on hardware. For example, increasing osd_max_backfills on a cluster with slow disks can overwhelm them. Always consider hardware capabilities.

1. Forgetting to update documentation: Failing to document changes leads to confusion later. Maintain a change log with original values, reasons, and dates.

Conclusion

Ceph configuration mistakes are common but avoidable with careful planning, validation, and rollback procedures. By inventorying your environment, making scoped changes, verifying effects, and preparing for failure, you can maintain a stable and performant Ceph cluster. Use the operations checklist as a standard practice, and always document changes for future reference. With these habits, you can confidently manage Ceph configurations and quickly recover from any missteps.