## Ceph Architecture Explained with Practical Examples ### Introduction Ceph is an open-source, distributed storage system that provides object, block, and file storage in a single unified platform. Its architecture is designed for self-healing, self-managing behavior with no single point of failure, making it a popular choice for cloud infrastructure, Kubernetes clusters, and large-scale data analytics. For DevOps consultants and technical startup teams, understanding the core components—monitors (MONs), managers (MGRs), object storage daemons (OSDs), metadata servers (MDSs), and clients—is essential for successful deployment and long-term operations. This article dives deep into Ceph's architecture with practical examples, including configuration snippets, verification commands, and failure recovery steps that you can immediately apply in your environment. ### Version and Environment Inventory Before you start, establish your Ceph version and environment. Ceph evolves rapidly; knowing your exact release helps avoid surprises and ensures compatibility with the latest features. This guide assumes Ceph Reef (v18.2.0) on Ubuntu 22.04 LTS, but the principles apply to other versions and distributions. A minimal cluster for testing requires at least three nodes: one node for MON/MGR and two nodes for OSDs. In production, you should separate these roles and run multiple monitors and managers for high availability. The following table summarizes the required components and their minimum counts:

Component	Minimum Count	Role
Monitor (MON)	1 (3 for HA)	Maintains the cluster map and state, provides consensus
Manager (MGR)	1 (2 for HA)	Provides extra monitoring, management, and dashboard functions
OSD	3 (for replication)	Stores data and handles replication, recovery, and rebalancing
Metadata Server (MDS)	1 (for CephFS only)	Provides file metadata for CephFS

Ensure your OS, kernel, and network meet the prerequisites. Ceph recommends 10GbE or faster networking for decent performance. Use ceph --version to verify the installed version. For example: ceph --version Expected output: ceph version 18.2.0 (5dd24139a4e2a36d3b2e1f3f8b3e7b0f2b0e1a9c) reef (stable) ### Safe Configuration Path Implementing Ceph requires careful configuration to avoid disrupting existing workloads. Start with a narrow pilot to minimize risk. For instance, if you plan to migrate from an existing storage solution, begin with a single RBD image on a test pool. This approach allows you to validate performance, features, and recovery processes without affecting production data. First, create a dedicated pool with a defined number of placement groups (PGs). Use the ceph osd pool create command. The number of PGs depends on the number of OSDs and the desired data distribution. A common rule of thumb is about 100 PGs per OSD for large clusters, but for a small test cluster, 32 or 64 is fine. Example: ceph osd pool create testpool 32 32 # pool name, pg_num, pgp_num Expected output: pool 'testpool' created Next, configure the pool's size (replication factor) and placement group autoscaling. For a test, set the size to 2 to save space and still maintain redundancy: ceph osd pool set testpool size 2 Ensure you have at least two OSDs up to meet the size requirement. Verify with ceph osd tree : ceph osd tree Example output: ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 0.02939 root default -3 0.00980 host node1 0 hdd 0.00980 osd.0 up 1.00000 1.00000 -5 0.00980 host node2 1 hdd 0.00980 osd.1 up 1.00000 1.00000 Now, create a block device image and map it on a client node. On the client, install the ceph-common package and retrieve the keyring from the cluster. Create the image: rbd create testimage --size 1G --pool testpool Then map it: rbd map testpool/testimage If successful, the device /dev/rbd/rbd/testpool/testimage (or /dev/rbd0 ) will appear. Format and mount it: mkfs.ext4 /dev/rbd0 mkdir -p /mnt/ceph-test mount /dev/rbd0 /mnt/ceph-test This demonstrates a basic Ceph block device setup. For production, consider using CRUSH maps to control data placement and failure domains, ensuring that replicas land on different hosts or racks. ### Verification and Diagnostics Verification is critical to ensure your Ceph cluster is healthy. Use the following commands to observe the cluster's status, performance, and data integrity. First, check overall health: ceph -s Expected output shows HEALTH_OK or HEALTH_WARN with details. For example: cluster: id: 4b5c8ccb-6a5e-4bf0-9e3b-9a2c8b1e5f3a health: HEALTH_OK services: mon: 1 daemons, quorum name (age 5m) mgr: 1 daemons active (age 5m) osd: 2 osds: 2 up (since 5m), 2 in (since 5m) data: pools: 1 pools, 32 pgs objects: 0 objects, 0 B usage: 0 B used, 0 B / 0 B avail pgs: 32 active+clean Inspect the placement group distribution and state: ceph pg stat Should show active+clean states. Example: 32 pgs: 32 active+clean; 0 B used, 0 B / 0 B avail Monitor disk usage and performance: ceph osd df This shows per-OSD usage. Example: ID CLASS WEIGHT RAW USED DATA OMAP META USED% AVAIL RAW USE 0 hdd 0.00980 1.0 GiB 1.0 GiB 0 B 1.0 MiB 10.20% 8.8 GiB 1.0 GiB 1 hdd 0.00980 1.0 GiB 1.0 GiB 0 B 1.0 MiB 10.20% 8.8 GiB 1.0 GiB TOTAL 0.0196 2.0 GiB 2.0 GiB 0 B 2.0 MiB 10.20% 17.6 GiB 2.0 GiB For network and client performance, use ceph osd perf : ceph osd perf Example output: osd fs_commit_latency_ms fs_apply_latency_ms 0 0.000000 0.000000 1 0.000000 0.000000 Verify RBD image operations: rbd info testpool/testimage This shows the image size and features. Expected output: rbd image 'testimage': size 1 GiB features: layering, exclusive-lock, object-map, fast-diff, deep-flatten ... Also, test end-to-end I/O with a simple write and read: echo "Hello Ceph" > /mnt/ceph-test/test.txt cat /mnt/ceph-test/test.txt Expected output: Hello Ceph ### Failure Modes and Recovery Understanding failure modes is essential for maintaining a reliable Ceph cluster. Common risks include OSD failure, network partitions, and monitor issues. Let's explore each and how to recover. ### OSD Failure If an OSD crashes, Ceph automatically re-replicates data from remaining copies to new OSDs to maintain the replication factor. Use ceph osd tree to see the status. If an OSD is down , restart it: systemctl restart ceph-osd@ If it cannot recover, you may need to replace the disk. For testing, simulate an OSD failure by stopping the systemd service. For example, to stop OSD 0: systemctl stop ceph-osd@0 Check health with ceph -s ; the status will show DEGRADED and pg states like degraded or undersized . Once you restart the OSD, the cluster should recover. Example: systemctl start ceph-osd@0 Then verify with ceph -s until HEALTH_OK is restored. ### Monitor Failure If a monitor fails, the cluster may lose quorum, which stops all I/O. Ensure you have an odd number of monitors (3 or more) in production. If a monitor is down, check its status with: ceph mon stat If it's not running, restart it: systemctl restart ceph-mon@ If the monitor's data is corrupted, you may need to rebuild it from other monitors using ceph-mon --cluster ceph --mkfs and re-add it, but that's a more advanced recovery scenario. ### Network Partition Ceph uses a heartbeat mechanism between monitors and OSDs. If there is a network disconnect, OSDs may be marked down due to missed heartbeats. Ensure your network is stable and properly configured (e.g., no packet loss, appropriate MTU). If a network partition occurs, Ceph may mark OSDs as down even if they are healthy. Once the network is restored, Ceph will automatically bring the OSDs back up and rebalance. You can check the OSD status with ceph osd tree . ### Recovery Steps In case of a serious issue, you can prevent Ceph from rebalancing mislabeled OSDs by setting a flag: ceph osd set noout This tells Ceph not to rebalance data out of OSDs that are marked out . After fixing the underlying issue, unset the flag: ceph osd unset noout For complex failures, always keep a backup of /etc/ceph/ceph.conf and keyrings. Regular monitoring with ceph health detail helps detect issues early. For example: ceph health detail This provides specific warnings and suggestions. ### Operations Checklist Repeatable operations start with a checklist. Use this to ensure your Ceph cluster remains healthy and performant. ### Daily - Check ceph -s for health status. - Review OSD and MON logs for errors: journalctl -u ceph-osd@* - Verify disk space: ceph df For example, to check logs for OSD 0: journalctl -u ceph-osd@0 --since "today" ### Weekly - Run ceph pg repair if any PGs are misplaced or degraded. - Check for slow requests: ceph daemon osd. ops (e.g., ceph daemon osd.0 ops ) Example of ceph pg repair : ceph pg repair 1.0 ### Monthly - Update Ceph packages after testing in a staging environment: sudo apt update && sudo apt upgrade (or zypper / dnf on other distros). - Review configuration for deprecated parameters: ceph config dump Here is a table summarizing key commands for each task:

Task	Command	Expected Result
Health check	`ceph -s`	`HEALTH_OK`
Pool detail	`ceph osd pool ls detail`	Lists pools and their settings (size, pg_num, etc.)
RBD image list	`rbd list --pool testpool`	Lists RBD images in the pool
Client performance	`ceph tell mon.* injectargs`	No errors; used to adjust runtime config

Always document changes and verify backups. Use maintenance windows for major upgrades to minimize impact. ### Conclusion Ceph's architecture provides a robust foundation for scalable storage, but it requires careful understanding and ongoing operations. We have explored the core components, implemented a test block device, verified cluster health, and discussed failure recovery. Next steps: start with a small pilot in a lab, use the checklist to monitor health, and gradually expand to production workloads. Keep learning and stay current with Ceph releases to benefit from improvements and new features. With the right approach, Ceph can become a reliable and efficient storage backbone for your infrastructure.