## Intro A Ceph production operations checklist helps operators move from an observed problem to a verified result. This guide focuses on practical, repeatable steps for developers, DevOps consultants, and technical startup teams managing Ceph clusters. It connects operations, checklists, best practices, and maintenance with concrete commands, expected outputs, failure signals, and recovery decisions. The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached. Each section below follows the same pattern: identify the relevant component and supported version, capture read-only state, define the smallest justified change, and verify the outcome. Commands use explicit placeholders like or ; never run them with real production values without first understanding the impact. ## Version and Environment Inventory Before any change, know exactly what you are running. This prevents applying advice meant for a different Ceph release or deployment model. Capture the cluster version, deployment tool, and health status with read-only commands. Observe first Run these commands from a monitor node or any client with the appropriate admin keyring. They are read-only and safe in production: ceph --version ceph -s ceph versions Expected output example: ceph version 17.2.7 (b12291d110049b2f35e32e0de30d70e9a4a3f04c) quincy (stable) cluster: id: 12345678-1234-1234-1234-123456789abc health: HEALTH_OK services: mon: 3 daemons, quorum a,b,c (age 2h) mgr: 2 daemons, active c osd: 24 osds: 24 up (since 3d), 24 in (since 10d) Also identify the deployment method: ceph orch status # if using cephadm systemctl status ceph-mon@ # if using traditional packages Define prerequisites and expected results For example, if you plan to change a pool configuration, list the pool name, current settings, and desired settings. Before the change, record the current state with a timestamp: date -u +%Y-%m-%dT%H:%M:%SZ ceph osd pool get all Version-specific consideration Ceph releases differ in command syntax and defaults. For example, ceph osd pool set pg_autoscale_mode on works in Nautilus and later, but older releases require manual PG calculation. Always check the official documentation for your exact version. Blast radius and recovery Changing a pool setting affects only that pool's placement and performance, but can impact all clients using it. To recover, revert the setting with the same command and the original value. For example, if you changed size from 3 to 4 and saw unexpected latency, set it back: ceph osd pool set size 3 ## Safe Configuration Path Configuration changes are a common source of incidents. Follow a controlled path: observe, record, change one item, verify, and know how to roll back. Example: adjust OSD memory target Suppose you want to increase osd_memory_target from 4 GB to 8 GB to reduce OSD cache flushes. Steps: - Check current value across all OSDs: ceph config get osd osd_memory_target Expected output: 4294967296 (bytes). If the command returns (unset) , the default applies. Find the default with ceph config get osd osd_memory_target --format=json or consult docs. - Change the value cluster-wide using the central config database (Nautilus and later): ceph config set osd osd_memory_target 8589934592 - Verify the change: ceph config get osd osd_memory_target Expected output: 8589934592 . - Monitor for a few hours; check OSD memory usage: ceph daemon osd. perf dump | jq '.mempool' Blast radius and rollback This change affects all OSDs and may increase overall memory pressure. If the nodes have limited RAM, OSDs could be OOM-killed. Rollback: ceph config set osd osd_memory_target 4294967296 For older releases using ceph.conf , edit the [osd] section on all OSD nodes and restart OSDs one by one, verifying health between restarts. Use explicit placeholders Never put real hostnames, IPs, or keyring paths in documentation. Use , , or . ## Verification and Diagnostics Verification confirms that a change produced the expected state without side effects. Diagnostics help locate the cause when things go wrong. Cluster health verification After any change, start with overall health: ceph -s Look for HEALTH_OK . If not, note the specific warning or error. For example, if you see HEALTH_WARN: 1 pools have many more objects per pg than average , investigate the pool's PG counts. Performance diagnostics Use ceph osd perf to see latency and commit times per OSD: ceph osd perf Sample output: osd commit_latency(ms) apply_latency(ms) 0 2.5 3.1 1 5.2 4.8 ... If one OSD shows much higher latency, check its disk health: smartctl -a /dev/sdX PG state diagnostics When clients report slow I/O, examine placement groups: ceph pg stat ceph pg dump | grep -E 'active\+clean|down|peering|recovering' If some PGs are stuck peering, identify the acting OSDs: ceph pg query Look for the "state": "peering" and check the OSDs' up/out status: ceph osd tree Verification script Create a one-liner to verify health after maintenance: ceph -s | grep -q HEALTH_OK && echo "Cluster healthy" || echo "Cluster needs attention" ## Failure Modes and Recovery Plan for common failures and know the recovery steps in advance. This section covers disk failure, OSD down, monitor problems, and network partitions. OSD or disk failure If a disk fails or an OSD crashes: - Check OSD status: ceph osd tree A down OSD shows down in the status column. - If the OSD is down but the disk is healthy, try bringing it back: systemctl start ceph-osd@ - If the disk has failed permanently, remove the OSD: ceph osd out ceph osd down ceph osd rm ceph osd crush remove osd. ceph auth del osd. Then physically replace the disk and add a new OSD with the same ID or a new one. Monitor quorum loss If you lose one monitor, the cluster still works but degraded. If you lose quorum (e.g., 2 of 3 monitors down), the cluster stops serving writes. Recovery steps: - Check monitor status: ceph mon stat - If a monitor node is reachable, restart the monitor: systemctl restart ceph-mon@ - If the monitor daemon cannot start due to corrupt data, remove and re-add it according to official documentation, ensuring you do not break quorum further. Network partition A split-brain can cause OSDs to fail heartbeats and mark each other down. Verify network connectivity between OSD nodes: ping -c 4 Check for firewall rules blocking ports 6800-7300 (Ceph OSD range). After restoring network, OSDs should automatically rejoin and start recovery. Replacement example with placeholders Suppose OSD 5 on host storage03 has a failed disk. The recovery path: - Mark OSD 5 out and remove it as above. - Replace disk /dev/sdb on storage03 . - Bootstrap the new OSD: ceph-volume lvm create --data /dev/sdb - Verify the new OSD starts and is marked in : ceph osd tree | grep osd.5 Expected: osd.5 up 1.00000 1.00000 with no down flag. ## Operations Checklist Use this checklist before any maintenance window or change. Each item includes the command to verify current state, the action, the expected result, and the owner (for teams). The owner is the single accountable person who must verify the action; they should revisit the checklist quarterly to ensure it matches the current cluster version and topology.
CheckCommand / ActionExpected ResultOwnerReview Frequency
Cluster healthceph -sHEALTH_OKPriya Shah, Storage LeadBefore every change
Version consistencyceph versionsAll daemons same versionDevOps Engineer (on-call)Monthly
Free capacityceph dfRaw usage < 70%Capacity PlannerWeekly
PG stateceph pg statAll PGs active+clean (or only expected recovering)Storage LeadDaily during operations
OSD hardware healthsmartctl -a /dev/sdX on each OSD diskNo SMART errorsSysadminMonthly
Backup of configceph config dump > ceph-config-$(date +%Y%m%d).confFile saved off-clusterConfiguration ManagerBefore and after changes
Authentication keysceph auth lsOnly expected clients, no unauthorizedSecurity OfficerQuarterly
Monitoring alertsCheck alertmanager / dashboardNo active critical alertsMonitoring AdminContinuous with weekly review
Network connectivityping between all nodes; check firewall rulesNo packet loss; ports 6800-7300 openNetwork AdminMonthly
For each item, define the failure signal and recovery step. For example, if free capacity exceeds 70% raw usage, add OSDs or delete unnecessary data; if PGs are stuck, investigate with ceph pg query and restart or reweight OSDs as needed. ## Common Pitfalls and How to Avoid Them Even experienced operators make mistakes. Here are the most frequent ones and how to recover. Pitfall 1: Changing multiple settings at once Why it happens: Time pressure or believing the changes are independent. How to avoid: Change one parameter, verify health and performance, then proceed. Use a maintenance window and record each change in a log. Recovery: Roll back to the previous configuration with ceph config set or ceph osd pool set using the recorded values. Pitfall 2: Ignoring cluster health warnings Why it happens: The cluster seems to work despite warnings, so teams postpone fixing them. How to avoid: Treat every HEALTH_WARN as actionable. Schedule time to resolve warnings; some are precursors to data loss (e.g., PG_DEGRADED ). Recovery: Address warnings one by one. For example, if you see OSD_DOWN , bring the OSD back or remove it if permanently failed. Pitfall 3: Not testing recovery procedures Why it happens: Recovery is only needed in an emergency, so teams skip drills. How to avoid: Test OSD removal and re-addition on a staging cluster or a non-production pool. Document the exact steps. Recovery: If an untested recovery fails in production, follow the official disaster recovery guide, which often involves manual steps like modifying CRUSH maps or using ceph-objectstore-tool . Pitfall 4: Using incorrect PG counts Why it happens: Autoscaling disabled or manually set pools without calculation. How to avoid: Enable PG autoscaling for new pools ( ceph osd pool set pg_autoscale_mode on ). For existing pools, monitor with ceph osd pool autoscale-status and adjust manually if necessary. Recovery: If a pool has too few PGs, increase gradually to avoid performance impact; if too many, decrease after ensuring data distribution. Pitfall 5: Running commands as root without understanding Why it happens: Copy-pasting from tutorials without checking the command's effect. How to avoid: Always read the command's documentation and verify its scope. Use --dry-run options where available. Test on a non-critical resource first. Recovery: If a command damages the cluster, stop all changes and consult the Ceph community or vendor support immediately. ## Conclusion A Ceph production operations checklist is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure. As a next step, choose one low-risk verification from the checklist, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as Proxmox, Linux, and storage clusters. For example, start with ceph -s and ensure the cluster is healthy before diving deeper. A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. Implement these practices gradually: start with the version and environment inventory, then adopt safe configuration paths, and finally incorporate failure drills into your regular operations.