Ceph CI/CD automation with practical examples: practical implementation guide

Ceph underpins critical storage for VMs, containers, and services. Treating Ceph changes as code, reviewed and verified through a CI/CD flow, helps you make small, safe, and auditable adjustments. The payoff is fewer regressions, faster recovery when something goes wrong, and a clear trail of what changed and why.

This guide shows how to build practical Ceph CI/CD automation with a narrow, verifiable pilot. You will:

• Capture a reliable environment inventory.
• Set up restricted access for automation.
• Implement a safe configuration path with explicit guardrails.
• Build a minimal pipeline that validates, canaries, and applies.
• Verify outcomes with observable checks.
• Handle common failure modes with rollback guidance.

All examples are constructed for illustration. Adjust values to your topology and risk tolerance.

Version and Environment Inventory

Before automating any change, lock down a consistent view of the cluster. The inventory becomes your baseline artifact and your first pipeline stage.

Run these commands from an admin host with ceph CLI access and store outputs under a timestamped directory (for example, inventory/2025-01-15/):

set -euo pipefail
TS="$(date +%F-%H%M%S)"
OUT="inventory/$TS"
mkdir -p "$OUT"

ceph -v | tee "$OUT/ceph-version.txt"
ceph versions | tee "$OUT/component-versions.json"
ceph status --format json-pretty | tee "$OUT/status.json"
ceph health detail | tee "$OUT/health.txt"
ceph osd tree | tee "$OUT/osd-tree.txt"
ceph osd df tree | tee "$OUT/osd-df-tree.txt"
ceph osd crush rule dump | tee "$OUT/crush-rules.json"
ceph osd getcrushmap -o "$OUT/crushmap.bin"
ceph df | tee "$OUT/df.txt"
ceph osd pool ls detail --format json-pretty | tee "$OUT/pools.json"
rados df --format json-pretty | tee "$OUT/rados-df.json"
ceph auth ls --format json-pretty | tee "$OUT/auth.json"

Keep a short human-readable digest for quick reviews.

Store these artifacts with your configuration repository so each pipeline run references the same facts. Diffing inventories between runs makes drift and upgrades obvious.

Prerequisites and Access

You need dependable tooling and tightly scoped credentials.

• Tools: ceph CLI and rados CLI installed on your runner host(s).
• Time sync: NTP/chrony working on your runner and cluster nodes.
• Network: runner can reach monitors (6789/3300) and managers as required.
• Secrets: two keyrings, one read-only for validation and one restricted write for controlled changes.

Create a read-only key for validation:

# Run with admin privileges once
ceph auth get-or-create client.cicd_ro mon 'allow r' osd 'allow r' mgr 'allow r' \
  | tee /etc/ceph/ceph.client.cicd_ro.keyring

Create a restricted maintenance key for controlled writes. Scope capabilities to change only what you intend to automate initially. For example, for limited pool operations and config keys:

# Example capabilities; tighten for your policy
ceph auth get-or-create client.cicd_maint \
  mon 'allow rw' osd 'allow rw pool=<pilot_pool>' mgr 'allow rw' \
  | tee /etc/ceph/ceph.client.cicd_maint.keyring

Store both keyrings securely on the runner host. Do not embed keys in scripts. Use file permissions 0600 and a dedicated system user.

Conventionally set environment variables in your runner job to point to the keyring:

export CEPH_ARGS="--name client.cicd_ro --keyring /etc/ceph/ceph.client.cicd_ro.keyring"
# For apply stages, switch to maint user explicitly per step

Safe Configuration Path

Start with changes that are:

• Low risk to availability and data safety.
• Fast to verify.
• Easy to revert.

Early, high-signal targets:

• Read-only gates: health must be OK (or a defined set of allowable warnings), monitors in quorum, no nearfull/full states.
• Pool metadata hygiene: ensure pool application labels are present; verify quotas for new pools.
• Non-disruptive parameters with bounded effect, changed gradually (for example, adjusting a management setting with no IO-path impact).

Avoid in the first pilot:

• CRUSH map edits that move lots of data.
• Size/min_size changes on production pools.
• OSD/Monitor package upgrades.
• Flags like noout, norecover, noscrub unless you have explicit maintenance flow and timebox.

Use hold points: a job that emits a summary and requires an explicit human approval before proceeding to any cluster-wide apply.

Pilot Pipeline Example

This example shows a minimal three-stage flow:

1. validate: Read-only checks against production with client.cicd_ro. Fails fast if unsafe.
2. canary: Create a short-lived canary pool, perform an object write/read/delete, then remove the pool. This exercises control path with client.cicd_maint without touching existing workloads.
3. apply_config: Optionally apply a small, reversible configuration change gated by health and a manual approval.

Directory layout (constructed example):

repo/
  scripts/
    ceph-validate.sh
    ceph-canary.sh
    ceph-apply-config.sh
  plans/
    config-plan.yaml

scripts/ceph-validate.sh

#!/usr/bin/env bash
set -euo pipefail

# Use read-only key
export CEPH_ARGS="--name client.cicd_ro --keyring /etc/ceph/ceph.client.cicd_ro.keyring"

# 1) Cluster health gates
ceph status
HEALTH=$(ceph health -f plain)
if [[ "$HEALTH" != "HEALTH_OK" ]]; then
  echo "Refusing to proceed: $HEALTH"
  exit 1
fi

# 2) Mon quorum must be >= 3 (constructed threshold)
MONS=$(ceph quorum_status --format json | jq -r '.quorum | length')
if (( MONS < 3 )); then
  echo "Quorum size $MONS below threshold"
  exit 1
fi

# 3) No nearfull/full pools
if ceph df | grep -E 'NEARFULL|FULL' >/dev/null; then
  echo "Refusing: nearfull/full condition present"
  exit 1
fi

# 4) No unsafe cluster flags
FLAGS=$(ceph osd dump | grep 'flags')
if echo "$FLAGS" | grep -E 'noout|norecover|nobackfill|norebalance|pause' >/dev/null; then
  echo "Unsafe flags present: $FLAGS"
  exit 1
fi

echo "Validation passed."

scripts/ceph-canary.sh

#!/usr/bin/env bash
set -euo pipefail

# Use maintenance key for write actions but keep scope narrow
export CEPH_ARGS="--name client.cicd_maint --keyring /etc/ceph/ceph.client.cicd_maint.keyring"

CANARY_POOL="ci_canary_$(date +%s)"

# Create small canary pool (constructed parameters; adjust to your policy)
ceph osd pool create "$CANARY_POOL" 8 8 replicated
ceph osd pool application enable "$CANARY_POOL" rados --yes-i-really-mean-it

# Write, read, and delete a test object
TMPDIR=$(mktemp -d)
echo "hello" > "$TMPDIR/obj"
rados -p "$CANARY_POOL" put canary_obj "$TMPDIR/obj"
rados -p "$CANARY_POOL" get canary_obj "$TMPDIR/obj.out"
diff "$TMPDIR/obj" "$TMPDIR/obj.out"
rados -p "$CANARY_POOL" rm canary_obj

# Remove canary pool
ceph osd pool delete "$CANARY_POOL" "$CANARY_POOL" --yes-i-really-really-mean-it
rm -rf "$TMPDIR"

echo "Canary succeeded."

scripts/ceph-apply-config.sh

#!/usr/bin/env bash
set -euo pipefail

# Use maintenance key, with a config backup and a manual hold point.
export CEPH_ARGS="--name client.cicd_maint --keyring /etc/ceph/ceph.client.cicd_maint.keyring"

# Example plan: change one low-risk config key gradually
# Plans/config-plan.yaml (constructed example):
# changes:
#   - scope: global
#     key: mgr_stats_period
#     value: "10"
#     previous: "20"

PLAN_FILE="plans/config-plan.yaml"
KEY=$(yq '.changes[0].key' -r "$PLAN_FILE")
SCOPE=$(yq '.changes[0].scope' -r "$PLAN_FILE")
VAL=$(yq '.changes[0].value' -r "$PLAN_FILE")
PREV=$(yq '.changes[0].previous' -r "$PLAN_FILE")

# Backup existing value if present
EXISTING=$(ceph config get "$SCOPE" "$KEY" || true)
echo "Existing $KEY=$EXISTING"

# Hold point: require explicit env var set by approver in the runner
: "${APPROVED:?Set APPROVED=yes to continue}"

# Apply change
ceph config set "$SCOPE" "$KEY" "$VAL"

# Post-change quick gate
sleep 5
if [[ "$(ceph health -f plain)" != "HEALTH_OK" ]]; then
  echo "Health degraded after config change; reverting"
  if [[ -n "$EXISTING" ]]; then
    ceph config set "$SCOPE" "$KEY" "$EXISTING"
  else
    ceph config rm "$SCOPE" "$KEY"
  fi
  exit 1
fi

# Record previous value for rollback
echo "$SCOPE $KEY $EXISTING" > ".rollback-$KEY.txt"

echo "Applied $KEY=$VAL"

You can orchestrate these scripts in your preferred runner as three sequential jobs with an approval gate before ceph-apply-config.sh.

A compact view of the stages and gates:

Verification and Diagnostics

Verification is both expected outputs and the absence of new warnings. After each stage:

• validate
• ceph status shows HEALTH_OK.
• ceph quorum_status shows expected quorum length.
• ceph df shows no nearfull/full pools.
• ceph osd dump flags do not contain noout, norecover, nobackfill, norebalance, pause.

• canary
• rados put/get/diff returns success.
• ceph health detail remains unchanged before/after.
• ceph osd pool ls detail does not list the canary pool after cleanup.

• apply_config
• ceph config get global mgr_stats_period shows the new value (constructed key example).
• ceph health is still HEALTH_OK 5-30 minutes post-change.

Useful diagnostics if something looks odd:

# Degraded PGs
ceph pg stat
ceph pg dump_stuck stale

# OSD flaps
ceph osd tree
ceph osd dump | grep 'up|in'

# Network/clock suspicion
ceph time-sync-status

# Config history (constructed by your process)
ls -1 .rollback-*.txt

Tip: Always include a short dwell time after apply_config. Some effects surface seconds later (e.g., monitoring frequency), and you want health gates to catch them.

Failure Modes and Recovery

Plan for the common ways a Ceph automation can fail. Treat every change as reversible.

Rollback playbook examples:

• Config change rollback

# From recorded rollback file
read SCOPE KEY PREV < .rollback-mgr_stats_period.txt || true
if [[ -n "${PREV:-}" ]]; then
  ceph config set "$SCOPE" "$KEY" "$PREV"
else
  ceph config rm "$SCOPE" "$KEY"
fi
ceph health detail

• Pool property rollback (if you changed a property by plan)

# Example: revert a quota or app label (constructed example)
POOL="example_pool"
ceph osd pool set-quota "$POOL" max_bytes 0
ceph osd pool application disable "$POOL" rados --yes-i-really-mean-it || true

• CRUSH rollback (only after a verified backup; avoid in first pilot)

# Restore a previously saved crushmap
ceph osd setcrushmap -i inventory/<ts>/crushmap.bin
ceph health detail

• Unsafe flags cleanup

# Clear maintenance flags only if you set them intentionally and finished work
for f in noout norecover nobackfill norebalance pause; do
  ceph osd unset "$f" || true
done
ceph osd dump | grep flags

Recovery confirmation checklist:

• ceph health returns HEALTH_OK for a sustained period (for example, 15 minutes).
• No stuck PGs: ceph pg dump_stuck returns empty.
• IO tests on a non-critical pool succeed: rados bench -p <pool> 10 seq (optional, read-only where safe).

Operations Checklist

Run this checklist whenever you propose or apply a change via the pipeline.

• Propose
• Describe the intent, blast radius, and rollback in the plan file.
• Choose an off-peak window.

• Validate
• Run scripts/ceph-validate.sh.
• Confirm HEALTH_OK, quorum >= 3, no nearfull/full, no unsafe flags.

• Canary
• Run scripts/ceph-canary.sh.
• Confirm canary pool is cleaned up and no new warnings appear.

• Approval
• Review diff of inventory artifacts and the change plan.
• Set APPROVED=yes in the runner only if you accept the risk.

• Apply
• Run scripts/ceph-apply-config.sh.
• Record previous values automatically in .rollback-*.txt.

• Observe
• Wait 15 minutes; watch ceph health detail.
• Check osd tree for unexpected changes.

• Rollback (if needed)
• Use recorded previous values or backup crushmap.
• Confirm recovery gates return to green.

• Postmortem
• If anything degraded, write a short note and adjust gates to catch it earlier next time.

Practical Extensions

Once the pilot is stable and boring, expand safely:

• Add a linter for your plan files to prevent typos and out-of-bounds values.
• Gradual adjustments with progressive holds: for example, 20 -> 15 -> 10 over days, each with dwell and verification.
• Read-only policy checks: verify every pool has an application label; alert on inconsistent replica sizes.
• Preflight gates for upgrades: compare ceph versions output to ensure homogeneity before moving any package changes into scope.
• Topology-aware checks: block risky actions if a failure domain has fewer than N hosts.

Conclusion

A small, well-instrumented Ceph CI/CD pilot delivers quick wins: predictable outcomes, confident rollbacks, and faster iteration. Start with read-only gates and a canary that exercises control without touching existing data. Add a single, reversible configuration change with an approval hold and clear rollback steps. Keep inventories and rollbacks as first-class artifacts.

As you gain confidence, grow the surface gradually: more checks, more parameters, then cautious, topology-aware changes. The key is to keep each step narrow, measurable, and easy to inspect locally before any broad rollout. That approach reduces rework, improves safety, and makes your Ceph operations steadily more reliable.