Ceph Upgrade and Migration: Practical Implementation Guide with Examples

Intro

Ceph underpins block, file, and object services across Linux environments, Proxmox clusters, and mixed fleets. Upgrading and migrating Ceph can improve stability, performance, and features, but it must be surgical: a misstep can trigger rebalancing storms, stuck placement groups (PGs), or client I/O stalls. This guide shows a low-risk, practitioner path to plan, execute, validate, and if necessary roll back Ceph upgrades and migrations. You will see concrete steps, verification checks, failure modes, and recovery procedures you can reuse.

Version and Environment Inventory

Establish ground truth before you touch production. Capture the cluster state in a change record so you can both plan the upgrade path and quickly compare after the change.

Prerequisites and inventory commands:

• Cluster health must be clean or at most advisory warnings. Investigate and resolve any degraded states.
• ceph -s
• ceph health detail
• Record current daemon versions and the target version.
• ceph versions
• ceph -v
• Confirm topology, device classes, and CRUSH layout.
• ceph osd tree
• ceph osd crush tree
• ceph osd df tree
• Review MON quorum and MGR status.
• ceph mon stat
• ceph -s (mgr map shown in status)
• Check pool and PG counts, target utilization, and autoscaler settings.
• ceph df
• ceph osd pool autoscale-status
• Record current OSD map flags so you can restore them later.
• ceph osd dump | grep flags
• Verify client compatibility (RBD, CephFS, RGW). For Proxmox and Linux nodes, ensure kernel and librbd support the target Ceph release features you plan to use.
• Back up configurations and maps.
• Save /etc/ceph/* from at least one admin host.
• ceph mon getmap -o /root/monmap.backup
• Optional: monmaptool --print /root/monmap.backup (for human check)
• Ensure you have access to package repositories or images for both the new and current Ceph versions so a downgrade is possible during the change window.

General path constraints:

• Plan to move one major release at a time unless the official release notes state otherwise for your path.
• Postpone enabling any new cluster-wide feature flags until all daemons are on the target version and stable. This preserves downgrade options.

Safe Configuration Path

Use a staged, controlled approach that limits blast radius and keeps data availability.

Upgrade order and rationale:

• MON (1st) — Establishes a control plane that understands the new OSDMap and features
• MGR (2nd) — Metrics and orchestration features align with MON map
• OSD (3rd) — Data path upgrade after control plane is ready
• MDS (CephFS) (4th) — Metadata services after core data path is stable
• RGW (Object) (5th) — API services last to avoid client disruption
• Clients (RBD/CephFS) (6th) — Upgrade clients after the cluster is stable

Maintenance controls:

• During rolling OSD restarts, reduce churn:
• ceph osd set noout
• ceph osd set noscrub
• ceph osd set nodeep-scrub
• For major OSD rotations, consider temporarily reducing recovery or backfill concurrency if needed for service quality. Revert to defaults after.
• Always undo flags when the step ends:
• ceph osd unset nodeep-scrub; ceph osd unset noscrub; ceph osd unset noout

Pilot first, then scale:

• Start with a canary scope: one host with a single OSD and no collocated critical services. Observe for at least one scrub cycle or a defined client workload window.
• Define a rollback path that is realistic for that scope before touching it.

Two execution paths are common:

• cephadm/orchestrated clusters: control upgrades with ceph orch.
• Package-managed clusters (apt, yum, dnf) with systemd units.

Practical Upgrade and Migration Examples

The examples below use constructed hostnames, IDs, and pool names. Adjust to your environment.

Example A: Minor upgrade with cephadm (constructed)

Goal: Move from 17.2.5 to 17.2.6 with minimal disruption.

1. Prechecks

• ceph -s should be HEALTH_OK or show only safe warnings.
• ceph versions should show a single major line (for example, all daemons reporting 17.2.x).

1. Prepare a canary host

• Pick host osd-03 with 2 OSDs.
• Pause automatic changes if you use them: ceph orch pause (optional)
• Enter maintenance and drain actively placed workloads if feasible: ceph orch host maintenance enter osd-03

1. Start the minor upgrade

ceph orch upgrade start --version 17.2.6

ceph orch upgrade status ceph -s

• Trigger the orchestrator upgrade:
• Monitor status:

1. Verify canary

• ceph versions should show osd processes on osd-03 now at 17.2.6.
• ceph pg stat should remain stable; no excessive backfill.
• Exit maintenance on success: ceph orch host maintenance exit osd-03

1. Roll the cluster

• Upgrade MONs and MGRs first if not already done by cephadm.
• Stagger remaining hosts by availability zones or racks.
• Verify and clear maintenance for each host before moving on.

1. Cleanup

• Ensure ceph osd unset noscrub; unset nodeep-scrub; unset noout.
• Confirm all daemons show 17.2.6 via ceph versions.

Expected result: Smooth rotation with no rebalance storms and a single minor version across all daemons.

Example B: Major upgrade with OS packages (constructed)

Goal: Move from 16.x to 17.x on package-managed hosts.

1. Prechecks and staging

• Ensure repository for 17.x is available on all nodes. Cache 16.x packages locally.
• ceph -s is clean; take monmap backup.

1. Upgrade MONs

ceph osd set noout; ceph osd set noscrub; ceph osd set nodeep-scrub

apt-get update && apt-get install -y ceph-mon (or dnf/yum) systemctl restart ceph-mon@mon-1

• On mon-1, mark OSD maintenance flags to avoid accidental data movement during any incidental OSD restarts:
• Update packages and restart MON on mon-1:
• Confirm quorum remains healthy: ceph mon stat
• Repeat for mon-2, mon-3, one at a time.

1. Upgrade MGRs

apt-get install -y ceph-mgr systemctl restart ceph-mgr@mgr-1

• Update packages and restart mgr on mgr-1, then mgr-2:
• Validate: ceph -s shows active mgr on target version.

1. Upgrade OSDs (rolling)

apt-get install -y ceph-osd systemctl restart ceph-osd@<id> Wait for osd.<id> to be up and in: ceph osd tree; ceph -s

• For each OSD host, one at a time:
• Rotate through all OSD hosts. Keep an eye on PG states and client latencies.

1. Upgrade MDS and RGW

• For any MDS: apt-get install -y ceph-mds && systemctl restart ceph-mds@<fs>
• For RGW: apt-get install -y radosgw && systemctl restart ceph-radosgw@<zone>

1. Post-upgrade

ceph osd require-osd-release reef (example only; use your target)

ceph osd unset nodeep-scrub; ceph osd unset noscrub; ceph osd unset noout

• Ensure all daemons are at 17.x: ceph versions
• Only now consider bumping require-osd-release to the new major if advised and after stability soak:
• Unset maintenance flags and re-enable scrubs:

Expected result: Stable quorum, consistent daemon versions, and no forced backfills outside of planned restarts.

Example C: Migrate a pool to faster devices via CRUSH rules (constructed)

Goal: Move pool vmdata from HDD to SSD without downtime.

ceph osd crush rule create-replicated ssd-rule default host ssd

1. Create a CRUSH rule for SSDs (device class ssd is assumed)

ceph osd pool set vmdata crush_rule ssd-rule

1. Apply rule to the pool

1. Monitor backfill and recovery

• ceph -s; ceph pg stat
• Expect PGs to remap gradually to SSD OSDs.

1. Verify placement

• ceph osd map vmdata (spot-checks)
• ceph osd df tree (utilization should rise on SSD OSDs)

Expected result: Gradual data migration to SSDs while the pool stays available.

Example D: Replace legacy FileStore OSDs with BlueStore (constructed)

Goal: Move data off FileStore OSDs to new BlueStore OSDs on the same host.

ceph-volume lvm create --bluestore --data /dev/nvme1n1

1. Add a new BlueStore OSD on a free device

ceph osd out <old_id>

1. Mark one old FileStore OSD out

ceph -s; ceph pg stat

1. Wait for backfill to finish

systemctl stop ceph-osd@<old_id> ceph osd purge <old_id> --force --yes-i-really-mean-it

1. Remove the old OSD

1. Repeat for remaining FileStore OSDs, one at a time

Expected result: All OSDs run BlueStore, with data migrated via normal CRUSH backfill.

Verification and Diagnostics

Validation is not a single check; combine state, version, and I/O tests.

Core checks:

ceph osd pool create testpool 32 32 rbd create testpool/testimg --size 64M

• Health and PGs
• ceph -s should be HEALTH_OK or show only transient recovery states during controlled moves.
• ceph pg stat should return all active+clean after each step.
• Version convergence
• ceph versions should show only target versions for each daemon type when complete.
• Capacity and placement
• ceph df and ceph osd df tree should show healthy headroom and balanced utilization.
• OSD behavior
• ceph osd tree shows all OSDs up and in.
• ceph osd perf can reveal outliers.
• Client I/O smoke tests (RBD)
• Create a small test pool and image:
• Map on a Linux client: rbd map testpool/testimg --pool testpool
• Write/read a file system worth of data (mkfs.ext4, mount, dd 32M, md5sum)
• Unmap: rbd unmap /dev/rbd/testpool/testimg
• Remove test artifacts after validation.

Pilot validation matrix (constructed):

• Health — ceph -s — HEALTH_OK or only transient recovery during canary
• Versions — ceph versions — Canary daemons show target version only
• PGs — ceph pg stat — No increase in undersized+degraded beyond canary window
• OSD restart — systemctl restart ceph-osd@<id> — OSD returns up+in within minutes
• RBD I/O — rados bench -p testpool 30 write — Stable latency; no client timeouts

Diagnostics tips:

• If PGs are slow to clean, check slow requests in logs and ceph health detail.
• If rebalancing is too aggressive, temporarily reduce recovery thread counts or backfill max.
• If clients see timeouts, verify msgr2 network, MTU consistency, and that only one feature set is active across librbd and kernel clients.

Failure Modes and Recovery

Issues happen; prepare realistic reversibility.

Common failure modes and remedies:

1. MON quorum loss after a restart

systemctl stop ceph-mon@<id> monmaptool --inject /root/monmap.backup /var/lib/ceph/mon/ceph-<id> systemctl start ceph-mon@<id>

• Symptom: ceph -s shows no quorum or degraded quorum.
• Actions:
• Ensure at least a majority of MONs are running and reachable.
• Restart a stuck MON: systemctl restart ceph-mon@<id>
• If MON map corruption is suspected, restore from backup:

1. OSD flaps or fails to come up after upgrade

• Symptom: ceph osd tree shows down OSDs, repeated restarts.
• Actions:
• Inspect logs for BlueStore or rocksdb errors.
• If this is a canary host on a major upgrade and minor rollback is possible, downgrade packages on that host and restart the OSD.
• Keep the OSD out until stable; do not trigger cluster-wide recovery churn.

1. Stuck PGs in peering or backfill

• Symptom: ceph pg stat shows prolonged peering or backfill_wait.
• Actions:
• Verify OSD availability on all replica targets.
• Check full/nearfull flags; reweight OSDs if one is saturated: ceph osd reweight <id> 0.9
• Ensure network path and MTU are correct between OSD peers.

1. RBD client incompatibility after upgrade

• Symptom: VM I/O stalls, mapping errors.
• Actions:
• Align client librbd and kernel modules to compatible versions.
• For Proxmox or other hypervisors, migrate one host first and test a non-critical VM disk before broad rollout.

Rollback and containment strategy:

• Prefer rolling forward within the same major to a fixed minor if bugs are found. Keep the previous minor packages cached.
• Across majors, do not set or bump cluster-wide features (for example, require-osd-release) until all daemons are upgraded and validated. This keeps a limited-time path to downgrade a subset of nodes if required.
• For package-managed clusters:
• Downgrade a canary node: reinstall previous packages; restart the affected services.
• Validate health and client I/O locally.
• If you must roll back more, proceed host-by-host, keeping quorum and majority health at every step.
• For cephadm clusters:
• ceph orch upgrade revert is not always available across majors; instead, pin an image or version for a given host or daemon and restart it.
• Keep the old image available locally or in your registry mirror.

Recovery checks after rollback:

• ceph -s returns to HEALTH_OK.
• ceph versions shows a consistent, supported mix (avoid split-brain mixes across major releases for long).
• Client I/O resumes without timeouts.

Quick triage signals (constructed):

• Quorum flaps — Likely area: MON — First check: ceph mon stat; systemctl status ceph-mon@*
• Sudden rebalance — Likely area: OSD flags — First check: ceph osd dump | grep flags; verify noout/noscrub
• High client latency — Likely area: OSD recovery — First check: ceph -s; tune recovery/backfill concurrency temporarily
• Mapping failures — Likely area: Client features — First check: librbd/kernel compat; msgr2 path

Operations Checklist

Use this as a repeatable runbook. Adjust to your environment.

Planning

• Define scope: target version, daemons in scope, and migration goals.
• Inventory versions, topology, and client compatibility.
• Back up /etc/ceph, monmap, and capture ceph -s and ceph versions snapshots.
• Confirm package or image availability for both new and current versions.
• Draft a pilot plan and rollback for the pilot.

Execution (pilot)

• Set maintenance flags: ceph osd set noout; ceph osd set noscrub; ceph osd set nodeep-scrub.
• Upgrade one host or a minimal set of daemons.
• Verify health, versions, PGs, and client smoke tests.
• If issues arise, roll back the pilot and investigate.

Execution (cluster)

• Upgrade MONs one-by-one; keep quorum.
• Upgrade MGRs.
• Roll OSD hosts sequentially; wait for up+in each time.
• Upgrade MDS and RGW as applicable.
• Upgrade or validate client hosts after the cluster is stable.

Post-change

• Clear flags: ceph osd unset nodeep-scrub; ceph osd unset noscrub; ceph osd unset noout.
• Confirm ceph versions and HEALTH_OK.
• Do a brief performance and I/O validation (rados bench or a known workload pattern).
• Consider bumping require-osd-release only after a stability soak.
• Update documentation and diagrams to reflect the new state.

Conclusion

A reliable Ceph upgrade or migration starts with an accurate inventory, uses a narrow pilot to learn safely, and proceeds in a controlled sequence with clear maintenance flags and verification gates. By validating after each step and preserving a realistic rollback for the scoped change, you reduce risk and keep storage services available. Use the examples and checklist here as a base runbook, then adapt it to your versions, hardware, and client mix. With a measured approach, you can deliver the benefits of newer Ceph releases while maintaining confidence in data integrity and service continuity.