Ceph Architecture Explained with Practical Examples

Ceph Architecture Explained with Practical Examples

Introduction

Ceph is an open-source, distributed storage system that provides object, block, and file storage in a single unified platform. Its architecture is designed for self-healing, self-managing behavior with no single point of failure, making it a popular choice for cloud infrastructure, Kubernetes clusters, and large-scale data analytics. For DevOps consultants and technical startup teams, understanding the core components—monitors (MONs), managers (MGRs), object storage daemons (OSDs), metadata servers (MDSs), and clients—is essential for successful deployment and long-term operations. This article dives deep into Ceph's architecture with practical examples, including configuration snippets, verification commands, and failure recovery steps that you can immediately apply in your environment.

Version and Environment Inventory

Before you start, establish your Ceph version and environment. Ceph evolves rapidly; knowing your exact release helps avoid surprises and ensures compatibility with the latest features. This guide assumes Ceph Reef (v18.2.0) on Ubuntu 22.04 LTS, but the principles apply to other versions and distributions. A minimal cluster for testing requires at least three nodes: one node for MON/MGR and two nodes for OSDs. In production, you should separate these roles and run multiple monitors and managers for high availability. The following table summarizes the required components and their minimum counts:

Ensure your OS, kernel, and network meet the prerequisites. Ceph recommends 10GbE or faster networking for decent performance. Use ceph --version to verify the installed version. For example:

ceph --version

Expected output:

ceph version 18.2.0 (5dd24139a4e2a36d3b2e1f3f8b3e7b0f2b0e1a9c) reef (stable)

Safe Configuration Path

Implementing Ceph requires careful configuration to avoid disrupting existing workloads. Start with a narrow pilot to minimize risk. For instance, if you plan to migrate from an existing storage solution, begin with a single RBD image on a test pool. This approach allows you to validate performance, features, and recovery processes without affecting production data.

First, create a dedicated pool with a defined number of placement groups (PGs). Use the ceph osd pool create command. The number of PGs depends on the number of OSDs and the desired data distribution. A common rule of thumb is about 100 PGs per OSD for large clusters, but for a small test cluster, 32 or 64 is fine. Example:

ceph osd pool create testpool 32 32   # pool name, pg_num, pgp_num

Expected output:

pool 'testpool' created

Next, configure the pool's size (replication factor) and placement group autoscaling. For a test, set the size to 2 to save space and still maintain redundancy:

ceph osd pool set testpool size 2

Ensure you have at least two OSDs up to meet the size requirement. Verify with ceph osd tree:

ceph osd tree

Example output:

ID  CLASS  WEIGHT   TYPE NAME              STATUS  REWEIGHT  PRI-AFF
-1         0.02939  root default
-3         0.00980      host node1
 0    hdd  0.00980          osd.0              up      1.00000  1.00000
-5         0.00980      host node2
 1    hdd  0.00980          osd.1              up      1.00000  1.00000

Now, create a block device image and map it on a client node. On the client, install the ceph-common package and retrieve the keyring from the cluster. Create the image:

rbd create testimage --size 1G --pool testpool

Then map it:

rbd map testpool/testimage

If successful, the device /dev/rbd/rbd/testpool/testimage (or /dev/rbd0) will appear. Format and mount it:

mkfs.ext4 /dev/rbd0
mkdir -p /mnt/ceph-test
mount /dev/rbd0 /mnt/ceph-test

This demonstrates a basic Ceph block device setup. For production, consider using CRUSH maps to control data placement and failure domains, ensuring that replicas land on different hosts or racks.

Verification and Diagnostics

Verification is critical to ensure your Ceph cluster is healthy. Use the following commands to observe the cluster's status, performance, and data integrity.

First, check overall health:

ceph -s

Expected output shows HEALTH_OK or HEALTH_WARN with details. For example:

  cluster:
    id:     4b5c8ccb-6a5e-4bf0-9e3b-9a2c8b1e5f3a
    health: HEALTH_OK

  services:
    mon: 1 daemons, quorum name (age 5m)
    mgr: 1 daemons active (age 5m)
    osd: 2 osds: 2 up (since 5m), 2 in (since 5m)

  data:
    pools:   1 pools, 32 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     32 active+clean

Inspect the placement group distribution and state:

ceph pg stat

Should show active+clean states. Example:

32 pgs: 32 active+clean; 0 B used, 0 B / 0 B avail

Monitor disk usage and performance:

ceph osd df

This shows per-OSD usage. Example:

ID  CLASS  WEIGHT   RAW USED  DATA     OMAP  META     USED%  AVAIL   RAW USE
 0    hdd  0.00980    1.0 GiB  1.0 GiB  0 B  1.0 MiB  10.20%  8.8 GiB  1.0 GiB
 1    hdd  0.00980    1.0 GiB  1.0 GiB  0 B  1.0 MiB  10.20%  8.8 GiB  1.0 GiB
TOTAL          0.0196    2.0 GiB  2.0 GiB  0 B  2.0 MiB  10.20%  17.6 GiB  2.0 GiB

For network and client performance, use ceph osd perf:

ceph osd perf

Example output:

osd fs_commit_latency_ms fs_apply_latency_ms
 0               0.000000              0.000000
 1               0.000000              0.000000

Verify RBD image operations:

rbd info testpool/testimage

This shows the image size and features. Expected output:

rbd image 'testimage':
	size 1 GiB
	features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
	...

Also, test end-to-end I/O with a simple write and read:

echo "Hello Ceph" > /mnt/ceph-test/test.txt
cat /mnt/ceph-test/test.txt

Expected output:

Hello Ceph

Failure Modes and Recovery

Understanding failure modes is essential for maintaining a reliable Ceph cluster. Common risks include OSD failure, network partitions, and monitor issues. Let's explore each and how to recover.

OSD Failure

If an OSD crashes, Ceph automatically re-replicates data from remaining copies to new OSDs to maintain the replication factor. Use ceph osd tree to see the status. If an OSD is down, restart it:

systemctl restart ceph-osd@<osd-id>

If it cannot recover, you may need to replace the disk. For testing, simulate an OSD failure by stopping the systemd service. For example, to stop OSD 0:

systemctl stop ceph-osd@0

Check health with ceph -s; the status will show DEGRADED and pg states like degraded or undersized. Once you restart the OSD, the cluster should recover. Example:

systemctl start ceph-osd@0

Then verify with ceph -s until HEALTH_OK is restored.

Monitor Failure

If a monitor fails, the cluster may lose quorum, which stops all I/O. Ensure you have an odd number of monitors (3 or more) in production. If a monitor is down, check its status with:

ceph mon stat

If it's not running, restart it:

systemctl restart ceph-mon@<mon-id>

If the monitor's data is corrupted, you may need to rebuild it from other monitors using ceph-mon --cluster ceph --mkfs and re-add it, but that's a more advanced recovery scenario.

Network Partition

Ceph uses a heartbeat mechanism between monitors and OSDs. If there is a network disconnect, OSDs may be marked down due to missed heartbeats. Ensure your network is stable and properly configured (e.g., no packet loss, appropriate MTU). If a network partition occurs, Ceph may mark OSDs as down even if they are healthy. Once the network is restored, Ceph will automatically bring the OSDs back up and rebalance. You can check the OSD status with ceph osd tree.

Recovery Steps

In case of a serious issue, you can prevent Ceph from rebalancing mislabeled OSDs by setting a flag:

ceph osd set noout

This tells Ceph not to rebalance data out of OSDs that are marked out. After fixing the underlying issue, unset the flag:

ceph osd unset noout

For complex failures, always keep a backup of /etc/ceph/ceph.conf and keyrings. Regular monitoring with ceph health detail helps detect issues early. For example:

ceph health detail

This provides specific warnings and suggestions.

Operations Checklist

Repeatable operations start with a checklist. Use this to ensure your Ceph cluster remains healthy and performant.

Daily

• Check ceph -s for health status.
• Review OSD and MON logs for errors: journalctl -u ceph-osd@*
• Verify disk space: ceph df

For example, to check logs for OSD 0:

journalctl -u ceph-osd@0 --since "today"

Weekly

• Run ceph pg repair if any PGs are misplaced or degraded.
• Check for slow requests: ceph daemon osd.<id> ops (e.g., ceph daemon osd.0 ops)

Example of ceph pg repair:

ceph pg repair 1.0

Monthly

• Update Ceph packages after testing in a staging environment: sudo apt update && sudo apt upgrade (or zypper/dnf on other distros).
• Review configuration for deprecated parameters: ceph config dump

Here is a table summarizing key commands for each task:

Always document changes and verify backups. Use maintenance windows for major upgrades to minimize impact.

Conclusion

Ceph's architecture provides a robust foundation for scalable storage, but it requires careful understanding and ongoing operations. We have explored the core components, implemented a test block device, verified cluster health, and discussed failure recovery. Next steps: start with a small pilot in a lab, use the checklist to monitor health, and gradually expand to production workloads. Keep learning and stay current with Ceph releases to benefit from improvements and new features. With the right approach, Ceph can become a reliable and efficient storage backbone for your infrastructure.