Ceph Common Errors and Fixes: A Practical Field Guide

Intro

Ceph is a powerful distributed storage system, but its complexity can lead to cryptic error messages that frustrate even experienced administrators. This guide focuses on practical, hands-on fixes for the most common Ceph errors: OSDs marked down, monitors failing to form a quorum, placement group (PG) issues, and authentication problems. You'll learn how to diagnose these issues using standard Ceph commands, apply safe configuration changes, verify recovery, and roll back if needed.

By following the examples and checklists, you can reduce downtime and maintain a healthy cluster. The guidance is based on real-world troubleshooting patterns, emphasizing narrow, measurable interventions that you can inspect locally before broader rollout.

A key principle throughout this guide: always verify the current state before making changes, apply changes in a scoped and reversible manner, and confirm the cluster returns to a healthy state afterward. Each section includes concrete commands with example outputs so you can compare against your own cluster.

Version and Environment Inventory

Before troubleshooting any error, gather essential information about your Ceph cluster. This ensures that fixes are compatible and that you understand the topology. Rushing into a fix without knowing the baseline often makes problems worse.

First, check the Ceph version on all nodes:

ceph --version

Expected output example:

ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)

Record the versions of MON, OSD, and MDS daemons on every host. Mixing major versions can cause unexpected behavior; during an upgrade, some daemons may run a different version transiently, but long-term version skew should be avoided. If you find version mismatches, note them and consult the Ceph upgrade documentation for your release before making other changes.

Next, verify cluster health and status:

ceph status

Look for the health summary line, which may be HEALTH_OK, HEALTH_WARN, or HEALTH_ERR. The output also includes the number of monitors, OSDs, and placement groups, plus any active warnings. Here is a sample healthy output:

  cluster:
    id:     a1b2c3d4-5678-90ab-cdef-1234567890ab
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph-mon-1,ceph-mon-2,ceph-mon-3 (age 2h)
    mgr: ceph-mgr-1(active, since 2h), standbys: ceph-mgr-2
    osd: 12 osds: 12 up (since 2h), 12 in (since 3d)

  data:
    pools:   4 pools, 256 pgs
    objects: 1.23M objects, 4.5 TiB
    usage:   13 TiB used, 40 TiB / 53 TiB avail
    pgs:     256 active+clean

If the health is not HEALTH_OK, the ceph status output will list specific warnings or errors under health: such as OSD_DOWN, PG_DEGRADED, or MON_DOWN. Capture those details before proceeding.

Document the network layout, including public and cluster networks. Ensure all nodes can resolve hostnames and reach each other on required ports: 6789 for monitors, 6800-7300 for OSDs (depending on configuration). Use ping and nc -zv <host> <port> or telnet <host> <port> to test connectivity.

Finally, confirm that authentication is properly configured and that you have the necessary admin keyring. Test your access with:

ceph -s

If this fails with a message like [errno 13] RADOS permission denied or Error connecting to cluster: PermissionDeniedError, you may have a configuration or keyring issue before any daemon-specific problem. Check that /etc/ceph/ceph.client.admin.keyring exists and contains a valid key, and that ceph.conf points to the correct monitors.

Safe Configuration Path

When applying configuration changes, always use the Ceph configuration database (ceph config) and make scoped changes. Avoid editing ceph.conf directly on individual nodes, as this can lead to inconsistencies between hosts and make rollback harder. The configuration database stores settings centrally and applies them to daemons automatically.

A safe workflow for any configuration change:

1. Identify the exact setting and its current value.
2. Test the change on a limited set of daemons (e.g., one OSD or one monitor) using a daemon-specific override.
3. Observe the effect for a period (minutes to hours) while watching cluster health.
4. If the change is beneficial, broaden the scope to all relevant daemons.
5. If the change causes issues, remove the override to revert to the previous state.

Example: Adjust OSD heartbeat grace to accommodate transient network delays.

The osd_heartbeat_grace parameter controls how long an OSD waits before declaring a peer OSD down. The default is 20 seconds. If you have frequent network blips (e.g., packet loss on a congested switch), you might see OSDs flapping up and down. Temporarily increasing this value can stabilize the cluster while you fix the network.

Step-by-step with commands:

# View current osd_heartbeat_grace value for all OSDs
ceph config get osd osd_heartbeat_grace

Expected output:

20

# Set a temporary override for a specific OSD (osd.3)
ceph config set osd.3 osd_heartbeat_grace 30

# Verify the setting for that OSD
ceph config get osd.3 osd_heartbeat_grace

Expected output:

30

Now monitor the cluster health and OSD behavior for a few minutes. If the flapping stops and no other issues appear, you may decide to apply the setting to all OSDs:

ceph config set osd osd_heartbeat_grace 30

To remove the override entirely and revert to the compiled-in or central default:

ceph config rm osd osd_heartbeat_grace

For daemon-specific overrides, the syntax is ceph config set <daemon_type>.<daemon_id> <key> <value>. For example, ceph config set mds.ceph-mds-1 mds_cache_memory_limit 4294967296 sets the MDS cache memory limit to 4 GiB for a single MDS.

Always test on a subset of daemons if possible. A bad configuration value can cause daemons to crash or behave unpredictably, so a scoped rollout is critical.

Verification and Diagnostics

Effective diagnostics rely on observing cluster state and interpreting log output. The following commands are your primary tools. Run them when the cluster is healthy to establish a baseline, and again when troubleshooting to spot differences.

1. Check overall cluster health

ceph status

Interpret the output as described in the inventory section. Pay attention to the health: block for specific error codes.

2. Inspect OSD status

ceph osd tree

This shows a tree of hosts and OSDs with their status, weight, and location. Example output:

ID  CLASS  WEIGHT   TYPE NAME        STATUS  REWEIGHT  PRI-AFF
-1         0.09799  root default
-3         0.03299      host ceph-osd-1
 0    ssd  0.01100          osd.0        up   1.00000  1.00000
 1    ssd  0.01100          osd.1        up   1.00000  1.00000
 2    ssd  0.01099          osd.2      down   1.00000  1.00000
-5         0.03299      host ceph-osd-2
 3    ssd  0.01100          osd.3        up   1.00000  1.00000
 4    ssd  0.01100          osd.4        up   1.00000  1.00000
 5    ssd  0.01099          osd.5        up   1.00000  1.00000

Look for OSDs with status down or out. An OSD that is up but out has been manually marked out of the cluster and will not receive data until marked in.

3. Examine placement group states

ceph pg dump_stuck

This shows PGs stuck in states like stale, inactive, undersized, or degraded. By default, it lists PGs stuck for 60 seconds or more; you can adjust with --threshold.

To get a summary of PG states:

ceph pg stat

Example output:

256 pgs: 256 active+clean; 12 TiB data, 40 TiB used, 53 TiB / 93 TiB avail

If you see PGs in inactive, undersized, or degraded, investigate which OSDs are missing or down using ceph pg <pgid> query.

4. View recent cluster log events

ceph log last 100

This prints the last 100 events from the cluster log. Search for relevant error messages such as osd.X failed, mon.X is down, or auth: unable to find a keyring.

5. Locate a specific OSD

ceph osd find <osd-id>

Example:

ceph osd find 0

Output shows the host, CRUSH location, and IP address of the OSD.

6. Query monitor quorum

ceph quorum_status

This returns JSON with the list of monitors and their quorum status. Expected output includes all monitors in the quorum array. Example:

{
  "election_epoch": 42,
  "quorum": [0, 1, 2],
  "quorum_names": ["ceph-mon-1", "ceph-mon-2", "ceph-mon-3"],
  "quorum_leader_name": "ceph-mon-1",
  "monmap": {
    "epoch": 3,
    "mons": [
      {"name": "ceph-mon-1", "rank": 0, "addr": "10.0.0.11:6789/0"},
      {"name": "ceph-mon-2", "rank": 1, "addr": "10.0.0.12:6789/0"},
      {"name": "ceph-mon-3", "rank": 2, "addr": "10.0.0.13:6789/0"}
    ]
  }
}

If a monitor is missing from the quorum array, investigate that monitor's network connectivity and service status.

For deeper analysis, examine daemon logs in /var/log/ceph/. Each daemon logs to a file named ceph-<daemon-type>-<id>.log (e.g., ceph-osd.0.log). Use journalctl -u ceph-osd@0 on systems with systemd to see recent service logs. You can also query a running daemon directly with ceph daemon <daemon-name> status, but this local socket interface is deprecated in newer releases in favor of the ceph tell command, e.g., ceph tell osd.0 version.

Failure Modes and Recovery

When things go wrong, having a clear recovery plan is crucial. This section covers the four most common failure modes: OSD down/out, monitor quorum loss, inactive placement groups, and authentication errors. For each, you'll find diagnostic steps, concrete recovery commands, and verification checks.

OSD Down or Out

An OSD can be marked down if the monitor cannot reach it within the heartbeat grace period. This can happen due to network issues, a crash, or high disk latency causing the OSD to miss heartbeats.

Diagnosis:

ceph osd tree

Look for down status. Check the OSD's own logs for crash reasons:

journalctl -u ceph-osd@<id> | tail -50

Also test network connectivity from the monitor to the OSD host on port 6800-7300.

Recovery:

First, try restarting the OSD service:

systemctl restart ceph-osd@<id>

If the OSD comes back up, the cluster will start recovering. Monitor with ceph -w (watch mode) until health returns to OK.

If the node is unreachable and cannot be quickly brought back, mark the OSD out manually to allow data re-replication:

ceph osd out <osd-id>

This tells CRUSH to redistribute data that was on this OSD to other OSDs. Once the OSD is back online and healthy, bring it back in:

ceph osd in <osd-id>

After an OSD is marked out and then in, the data will rebalance. This can take a long time depending on data amount; monitor progress with ceph status and ceph pg stat.

Verification:

• ceph osd tree shows OSD up and in.
• ceph status shows no OSD_DOWN warnings.
• ceph pg stat shows PGs returning to active+clean.

Monitor Quorum Lost

Ceph monitors use the Paxos algorithm to maintain a consistent cluster map. If fewer than half of the monitors are reachable, quorum is lost, and the cluster stops responding to changes (though existing I/O may continue if OSDs have current maps).

Diagnosis:

ceph quorum_status

If this command hangs or returns an error, the monitors are not in quorum. Check individual monitor status:

systemctl status ceph-mon@<hostname>

Check network connectivity between monitors (port 6789) and look at monitor logs:

journalctl -u ceph-mon@<hostname> | tail -100

Common causes: network partition, firewall misconfiguration, clock skew between monitors (more than 0.05 seconds can cause issues), or disk full on a monitor.

Recovery:

1. Fix the underlying cause: restore network, correct time synchronization (use NTP), free disk space.
2. If a single monitor has failed and cannot be recovered quickly, you can remove it from the monmap and continue with an even number of monitors temporarily, but an odd number is required for a stable quorum. Better to add a new monitor to replace the failed one.
3. As a last resort, if all monitors have lost data or the monmap is corrupted, you can recover the monitor store from a backup using ceph-monstore-tool:

ceph-monstore-tool /var/lib/ceph/mon/ceph-<hostname> rebuild

This rebuilds the monitor store from OSDs; it is a complex procedure that should be performed with care and preferably with vendor or community support.

Verification:

• ceph quorum_status shows all expected monitors in quorum.
• ceph status shows HEALTH_OK or only warnings.

Placement Groups Inactive

Placement groups (PGs) are the internal units of data distribution. They normally reside in active+clean state. If a PG is inactive, it cannot serve I/O for the objects it contains.

Diagnosis:

ceph pg dump_stuck inactive

Then drill into a specific stuck PG:

ceph pg <pgid> query

This returns JSON with details about the PG's peering state, acting set, and any errors. Look for "state": "inactive" and "peering_blocked_by" fields that indicate what is preventing peering.

Common causes:

• Not enough OSDs from the PG's acting set are up and in.
• The PG is missing a required log or object due to OSD loss.
• A configuration error in CRUSH rules results in too few OSDs.

Recovery:

1. Ensure the required OSDs are up and in. Use ceph osd tree and ceph osd in <id> if needed.
2. If OSDs were down and are now back, the PGs may peer automatically. Check with ceph pg stat after a few minutes.
3. If the PG remains inactive because it believes data is lost, you can force it to accept the current state and attempt to peer. This should only be done after ensuring that the data is not actually lost, as it may cause data inconsistency. Use:

ceph pg force_create_pg <pgid>

This command tells the monitor to create the PG if it doesn't exist, which can kickstart peering. For more severe cases, you may need to use ceph osd force-create-pg or ceph pg mark_unfound_lost revert|delete, but those are advanced and potentially destructive; consult documentation and community before using.

Verification:

• ceph pg dump_stuck inactive returns no PGs.
• ceph pg stat shows all PGs active+clean eventually.

Authentication Errors

Clients may fail to connect to the cluster with errors like error connecting to the cluster or bad auth. This usually indicates missing or incorrect keyrings, or insufficient capabilities.

Diagnosis:

Test client access:

ceph -s --id <client-name>

If you see [errno 13] RADOS permission denied, the client's keyring or capabilities are wrong. Check the client's keyring file (usually /etc/ceph/ceph.client.<name>.keyring) and verify the key matches the one in the cluster:

ceph auth get <client-name>

Compare the key value with the one in the client's keyring.

Recovery:

1. Ensure the client keyring exists and has the correct key. If missing, recreate it from the cluster:

ceph auth get-or-create client.<name> mon 'allow r' osd 'allow rw pool=<pool-name>' -o /etc/ceph/ceph.client.<name>.keyring

1. Adjust capabilities if needed. For example, to grant read-only access to a specific pool:

ceph auth caps client.<name> mon 'allow r' osd 'allow r pool=<pool-name>'

1. Regenerate keys if they are compromised:

ceph auth caps client.<name> mon 'allow r' osd 'allow rw pool=<pool-name>'
ceph auth get-or-create client.<name>

Then distribute the new key to the client.

Verification:

• Client can run ceph -s --id <client-name> without auth errors.
• ceph auth list shows the client with correct capabilities.

Always have a rollback plan: document all changes, keep backups of configuration and keyrings, and test recovery procedures in a non-production environment first.

Operations Checklist

Use this checklist for routine operations and when troubleshooting. Running these commands regularly (e.g., daily) helps you detect problems before they escalate.

For each issue found, document the diagnosis and fix. Always test changes in a scoped manner before applying cluster-wide. Here is an example of checking disk usage:

ceph df

Example output:

--- RAW STORAGE ---
CLASS  SIZE     AVAIL    USED     RAW USED  %RAW USED
ssd    93 TiB   53 TiB   40 TiB    40 TiB      43.01
TOTAL  93 TiB   53 TiB   40 TiB    40 TiB      43.01

--- POOLS ---
POOL        ID  PGS   STORED   OBJECTS  USED     %USED  MAX AVAIL
rbd         1   64    8.5 TiB   2.1M    25 TiB   80.00    6.5 TiB
cephfs_data 2   64    1.2 TiB   300k     3.6 TiB  40.00    8.8 TiB

If any OSD or pool usage is above 80-85%, plan to add capacity or rebalance to avoid OSD_NEARFULL or OSD_FULL warnings that can block writes.

Conclusion

You have learned practical strategies for diagnosing and fixing common Ceph errors. Start by gathering an accurate environment inventory, then apply configuration changes safely using the Ceph configuration database with scoped overrides. Verify each fix with observable checks, and always have a rollback plan. Use the operations checklist to maintain cluster health and catch issues early.

The key takeaways:

• Know your baseline: Record versions, health status, and topology before problems arise.
• Change incrementally: Use ceph config with daemon-specific overrides and test before broad rollout.
• Diagnose systematically: Use ceph status, ceph osd tree, ceph pg dump_stuck, and ceph quorum_status to pinpoint issues.
• Recover safely: Follow the specific recovery steps for OSD, monitor, PG, and auth failures, and always verify recovery.
• Keep a checklist: Run routine checks and document changes for faster troubleshooting.

Next steps: apply this guide to a specific problem in your own cluster, beginning with the diagnostics commands. For deeper automation, consider integrating these checks into your monitoring system (e.g., Prometheus with Ceph exporter) or configuration management (e.g., Ansible) to enforce consistency. Remember to test any non-trivial recovery procedure in a staging environment first.

Ceph is robust but unforgiving of hasty actions. With the disciplined approach outlined here, you can keep your cluster running smoothly and recover quickly when issues arise.