Ceph Monitoring and Alerts: Practical Implementation Guide

Intro

Ceph monitoring and alerts with practical examples should help operators move from an observed problem to a verified result. Start by identifying the installed version, deployment topology, prerequisites, and the exact component being inspected.

This article focuses on Ceph monitoring for developers, DevOps consultants and technical startup teams. It connects Ceph alerts, Ceph metrics, Ceph dashboard and Ceph incident response to commands, expected output, failure signals, and recovery decisions that match the selected technology.

The goal is operational safety: observe before changing, limit the blast radius, use placeholders instead of secrets, verify the result, and document how to recover if the expected state is not reached.

Version and Environment Inventory

Before monitoring or alerting on a Ceph cluster, you need to know exactly what you are running. The inventory captures the Ceph version, deployment tool, operating system, cluster topology, and the key services that must be observed. This information is the foundation for every subsequent command because flags, metric names, and alert rules differ between Ceph releases.

Run these read-only commands on any monitor node to collect the environment facts. The commands do not modify state and are safe for production.

# Ceph version and build
ceph --version
# Expected output: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)

# Cluster status (compact)
ceph -s
# Expected output (example):
# cluster:
#   id:     a1b2c3d4-5678-90ab-cdef-1234567890ab
#   health: HEALTH_WARN
#            1 MDSs report slow metadata IOs
#            1 nearfull osd(s)
#   services:
#     mon: 3 daemons, quorum host1,host2,host3 (age 5d)
#     mgr: host1(active, since 2h), standbys: host2, host3
#     osd: 12 osds: 12 up (since 5d), 12 in (since 6d)
#   data:
#     pools:   3 pools, 257 pgs
#     objects: 45.12k objects, 168 GiB
#     usage:   502 GiB used, 3.5 TiB / 4.0 TiB avail
#     pgs:     257 active+clean

# Deployment tool (choose one)
# For cephadm:
ceph orch status
# For ceph-deploy or manual:
ls /etc/ceph/

Prerequisites for these commands:

• SSH access to a monitor node with a user that can read /etc/ceph/ceph.conf and the admin keyring.
• The ceph CLI package installed, version matching the cluster.
• Network access to the Ceph monitor port (default 6789).

Blast radius: None. All commands are query-only.

Verification: The ceph -s output must show the expected number of monitors, OSDs, and pools. If any service is missing or the health is not HEALTH_OK, resolve that before adding monitoring, because an unhealthy baseline will produce noisy alerts.

Recovery: If ceph --version fails with permission denied, ensure your user can read the admin keyring, typically at /etc/ceph/ceph.client.admin.keyring. If the command is not found, install ceph-common from your distribution repository.

Record the version and topology in a shared operations document. Example entry:

Cluster ID: a1b2c3d4-5678-90ab-cdef-1234567890ab
Ceph version: 17.2.6 (quincy) deployed with cephadm on Ubuntu 22.04
Monitor hosts: mon01, mon02, mon03
Manager host: mon01
OSD count: 12, all HDD, bluestore
Pools: rbd (replicated 3x), cephfs_data (erasure 4+2), cephfs_metadata (replicated)

Safe Configuration Path

Monitoring and alerting often require configuration changes: enabling the Prometheus module, adjusting scrape intervals, or adding alert rules. The safe configuration path separates observation from intervention. Never change a setting without first capturing the current state and understanding how to revert.

Enabling the Prometheus module

The Ceph Manager (mgr) includes a Prometheus exporter module that serves metrics on port 9283. This is the recommended way to expose Ceph metrics to Prometheus.

1. Observe current manager modules:

ceph mgr module ls
# Look for "prometheus" in the enabled_modules list. Expected output snippet:
# "enabled_modules": ["balancer", "dashboard", "iostat", "prometheus", "restful"]

1. Enable the module if absent:

ceph mgr module enable prometheus
# Expected output: module 'prometheus' is enabled (always on)

1. Verify the module is active and the endpoint is listening:

ceph mgr services
# Expected output should include:
# "prometheus": "http://mon01:9283/"

curl -s http://mon01:9283/metrics | head -n 5
# Expected output starts with Prometheus metric lines, e.g.:
# # HELP ceph_health_status Health status of Cluster, can vary only between 3 states (err:2, warn:1, ok:0)
# # TYPE ceph_health_status gauge
# ceph_health_status 1.0

Prerequisites:

• Ceph cluster running Nautilus (14.x) or later.
• The manager (mgr) daemon is active and reachable.
• Firewall allows access to port 9283 from the Prometheus server.

Blast radius: Low. Enabling the module does not affect data plane operations. It adds a metric endpoint on the manager.

Recovery path: To revert, disable the module with ceph mgr module disable prometheus. The metric endpoint disappears immediately.

Configuring the Prometheus server

Add a scrape job to your Prometheus configuration file (usually prometheus.yml):

scrape_configs:
  - job_name: 'ceph-mgr'
    static_configs:
      - targets: ['mon01:9283', 'mon02:9283', 'mon03:9283']
        labels:
          cluster: 'production-ceph'
    metrics_path: /metrics
    scrape_interval: 15s
    scrape_timeout: 10s

Verification: After reloading Prometheus, check the target status at http://<prometheus>:9090/targets. The Ceph targets should show UP in green. If a target shows DOWN, verify network connectivity and that the manager's Prometheus module is active.

Blast radius: Changing Prometheus configuration does not affect Ceph itself, but a misconfigured scrape can overload the manager if too many targets or too frequent intervals are used. Keep scrape interval at 15s or higher.

Recovery path: Restore the previous prometheus.yml and restart Prometheus.

Avoiding unsafe changes

Never modify Ceph configurations like osd_max_backfills, mon_osd_down_out_interval, or pool sizes without a rollback plan. For example, reducing mon_osd_down_out_interval can cause OSDs to be marked out too quickly during transient network issues, leading to data movement and performance degradation.

If you must change such a setting, first capture the current value:

ceph config get mon mon_osd_down_out_interval
# Expected output: 300 (seconds)

Then make the change and record the command to revert:

ceph config set mon mon_osd_down_out_interval 600
# To revert:
ceph config set mon mon_osd_down_out_interval 300

Verification: Run ceph config get mon mon_osd_down_out_interval to confirm the new value is applied. Then monitor ceph -s for any unexpected OSD rebalancing.

Verification and Diagnostics

Once monitoring is in place, you need to verify that it reflects cluster reality. This section covers diagnostic commands and how to interpret typical outputs to detect issues early.

Checking cluster health

The primary health command is ceph health detail. It provides more information than the summary in ceph -s.

ceph health detail
# Expected output example:
# HEALTH_WARN 1 MDSs report slow metadata IOs; 1 nearfull osd(s)
# [WRN] MDS_SLOW_METADATA_IO: 1 MDSs report slow metadata IOs
#     mds.cephfs.host1.up:active reported 13 slow metadata IOs in the last 60s
# [WRN] OSD_NEARFULL: 1 nearfull osd(s)
#     osd.5 is near full (85%)

Interpretation:

• MDS_SLOW_METADATA_IO may indicate insufficient metadata pool performance or a busy filesystem client. Investigate with ceph daemon mds.<id> perf dump or the dashboard's MDS performance graph.
• OSD_NEARFULL means an OSD is above the nearfull threshold (default 85%). Immediate action: add capacity or rebalance. Do not wait for full (95%), as writes will be blocked.

Verification of monitoring: After seeing OSD_NEARFULL in the CLI, check that Prometheus records the metric ceph_osd_nearfull and that an alert is triggered. For example, the default alert rule for nearfull might be:

- alert: CephOSDNearFull
  expr: ceph_osd_nearfull > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Ceph OSD near full"
    description: "OSD {{ $labels.ceph_osd }} is above 85% capacity."

Digging into performance

Use ceph osd perf to identify slow OSDs.

ceph osd perf
# Expected output (example):
# osd commit_latency(ms) apply_latency(ms)
#   0                  2                  3
#   1                 25                 30
#   2                  3                  4
#   ...

A high latency (tens of milliseconds or more) on a single OSD suggests a failing disk, network issue, or overloaded OSD. Correlate with smartctl and iostat on the OSD host.

Diagnostic command to verify data distribution:

ceph osd df tree
# Expected output shows per-OSD usage, e.g.:
# ID CLASS WEIGHT  REWEIGHT SIZE    RAW USE DATA    OMAP    META     AVAIL   %USE  VAR  PGS STATUS
#  0   hdd 1.00000  1.00000 500 GiB  120 GiB 118 GiB   2 MiB 1.2 GiB 380 GiB 24.00 0.96  64     up
#  1   hdd 1.00000  1.00000 500 GiB  400 GiB 398 GiB   5 MiB 2.0 GiB 100 GiB 80.00 1.60  64     up

A high variance (VAR > 1.5) indicates uneven data distribution. Use the balancer module to correct it:

ceph balancer status
# Expected output: {"active": true, "last_optimize_duration": "0:00:07.123", "mode": "upmap", "no_optimization_needed": false}
ceph balancer eval
# Expected output: "current cluster score: 0.1523 (lower is better)"

If the score is high (e.g., > 0.5), run ceph balancer optimize plan to see what changes would be made, then ceph balancer execute plan to apply.

Failure Modes and Recovery

Monitoring must help you respond to failures. This section describes common Ceph failure modes, how to detect them, and how to recover.

Monitor quorum loss

Symptoms:

• ceph -s hangs or reports mon client: couldn't connect to cluster.
• Prometheus target for the monitor goes down.

Detection:

ceph quorum_status
# If quorum is lost, the command fails.
# If quorum exists, output includes a list of monitors and their ranks.

Recovery:

1. Check monitor processes on each host: systemctl status ceph-mon@<hostname>.
2. Restart a failed monitor: systemctl restart ceph-mon@<hostname>.
3. If a majority of monitors are down, you may need to start monitors manually with ceph-mon -i <id>.
4. After quorum is restored, verify with ceph -s and ceph quorum_status.

Prevention: Monitor alerts with Prometheus:

- alert: CephMonDown
  expr: ceph_mon_quorum_count < 2
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Ceph monitor quorum lost"
    description: "Only {{ $value }} monitors in quorum."

OSD failure

Symptoms:

• ceph -s shows HEALTH_WARN with osd.X is down.
• Applications report I/O errors or timeouts.

Detection:

ceph osd tree
# Look for OSDs with status 'down' or 'out'.
# Example output shows osd.3 down:
# ID CLASS WEIGHT  TYPE NAME      STATUS REWEIGHT PRI-AFF
# -1       0.09766 root default
# -3       0.04883     host host2
#  3   hdd 0.04883         osd.3    down  1.00000 1.00000

Recovery:

1. Investigate the OSD host: systemctl status ceph-osd@3.
2. If the disk has failed, replace it. For cephadm, remove the OSD:

ceph orch device zap host2 /dev/sdX --force
ceph orch osd rm 3 --force

1. Add a new OSD if a replacement disk is available: ceph orch apply osd --all-available-devices (use cautiously).
2. Wait for rebalancing and verify health returns to HEALTH_OK.

Prevention: Alert on OSD down:

- alert: CephOSDDown
  expr: ceph_osd_down > 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Ceph OSD down"
    description: "OSD {{ $labels.ceph_osd }} is down."

Full OSD or pool

Symptoms:

• Writes fail with -ENOSPC or pool is full.
• ceph -s shows HEALTH_ERR with full or nearfull OSDs.

Detection:

ceph df
# Expected output includes per-pool usage:
# POOL            ID  PGS  STORED  OBJECTS  USED  %USED  MAX AVAIL
# rbd             1   64   80 GiB   20.48k  240 GiB  80.00     60 GiB

If a pool's %USED nears 100%, it is full.

Recovery:

1. Immediately identify and delete unnecessary data if possible: rados -p <pool> rm <object>.
2. Expand the cluster by adding OSDs.
3. Increase the pool's quota if allowed: ceph osd pool set-quota <pool> max_bytes <larger_value>.
4. If the cluster is truly full, you may need to temporarily set ceph osd set full and ceph osd set pause to prevent further writes while you free space.

Prevention: Set alerts for pool usage:

- alert: CephPoolFull
  expr: (ceph_pool_percent_used > 85)
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Ceph pool above 85% usage"
    description: "Pool {{ $labels.name }} is {{ $value }}% full."

Operations Checklist

Use this checklist daily or weekly to ensure Ceph monitoring and alerting remain effective.

Daily checks:

• Run ceph -s on a monitor. Confirm health is HEALTH_OK or that all warnings are known and tracked.
• Check Prometheus targets at http://prometheus:9090/targets. All Ceph targets should be UP.
• Review active alerts in Alertmanager (or your alerting system). Investigate any new alerts.
• Verify dashboard login works and shows cluster status: https://mon01:8443.

Weekly checks:

• Review ceph df for pool usage trends. Plan expansion if any pool exceeds 70%.
• Check OSD capacity balance with ceph osd df tree. Run balancer if variance > 1.5.
• Review slow requests: ceph daemon osd.0 perf dump | jq '.osd.op_latency' or ceph osd perf.
• Validate backup and restore procedures for Ceph configuration and keyrings.
• Test alert notification delivery by sending a test alert from Alertmanager.

Monthly checks:

• Update the environment inventory document if any changes occurred.
• Review and prune old alert rules that are noisy or no longer relevant.
• Check disk health on OSD hosts using smartctl -a /dev/sdX.
• Perform a failure drill: simulate an OSD down by stopping the OSD service and verify alerts fire and recovery steps work.

Each checklist item should have an owner. For example:

Conclusion

Ceph monitoring and alerts with practical examples is useful only when each recommendation is version-scoped, observable, and reversible where the technology permits. Copying a command without checking prerequisites and expected output is not an operations procedure.

As a next step, choose one low-risk verification for Ceph monitoring, record the current state, run the documented check, compare the result with the expected signal, and review dependencies such as Prometheus, the Ceph dashboard, and the underlying Linux hosts.

A reliable technical workflow makes failure visible, protects sensitive values, limits changes to the intended resource, and defines recovery verification before an incident forces the decision. By applying the commands, configuration snippets, and alert rules in this guide, you can build a Ceph monitoring system that provides early warning and enables rapid recovery.