Ceph security hardening with practical examples: practical implementation guide

Ceph is a powerful, distributed storage system. Power invites risk if defaults are left untouched. This practical guide shows how to harden Ceph in small, safe steps:

• Use cephx with least-privilege caps
• Store and protect secrets correctly
• Shrink the network attack surface
• Enable encrypted transport on the wire
• Lock down the dashboard
• Rotate credentials safely
• Run repeatable checks to verify hardening

You can apply the steps incrementally. A small, measurable pilot is included so you can test locally before rolling out cluster-wide.

Workflow Overview

The following production workflow is designed for clarity and safety. Apply each step, verify, and only then proceed.

1. Inventory and baseline

• List versions and health:

ceph versions
ceph -s
ceph health detail

• Snapshot effective config for traceability:

ceph config dump -f json-pretty > ceph-config-baseline.json

1. Lock down authentication and authorization (cephx)

• Ensure insecure global id reclaim is disabled (required for secure messaging behavior):

ceph config get mon auth_allow_insecure_global_id_reclaim
ceph config set mon auth_allow_insecure_global_id_reclaim false

• Create a least-privilege client for RBD access to a single pool (example pool: vms):

sudo ceph auth get-or-create client.rbduser \
  mon 'profile rbd' \
  osd 'profile rbd pool=vms' \
  mgr 'allow r' \
  -o /etc/ceph/ceph.client.rbduser.keyring

• Tighten caps for an existing client (update-in-place):

ceph auth caps client.rbduser \
  mon 'profile rbd' \
  osd 'profile rbd pool=vms' \
  mgr 'allow r'

• For CephFS path-based caps (example):

ceph fs authorize cephfs client.webuser /var/www r

1. Secrets at rest: protect keyrings and admin material

• Restrict filesystem permissions and ownership of keyrings:

sudo chown ceph: ceph /etc/ceph/ceph.client.rbduser.keyring
sudo chmod 600 /etc/ceph/ceph.client.rbduser.keyring

• Keep admin keyrings off application hosts. Use the restricted client identity on app nodes.

1. Reduce network exposure

• Separate networks in ceph.conf (example values):

# /etc/ceph/ceph.conf
[global]
 public_network = 10.10.0.0/24
 cluster_network = 10.20.0.0/24
 ms_bind_msgr2 = true
 auth_allow_insecure_global_id_reclaim = false

• Apply host firewall rules. Allow only required ports from trusted subnets. Example with iptables:

# Allow monitors on public network (msgr2 3300, legacy 6789) from trusted subnet
sudo iptables -A INPUT -p tcp -s 10.10.0.0/24 --dport 3300 -j ACCEPT
sudo iptables -A INPUT -p tcp -s 10.10.0.0/24 --dport 6789 -j ACCEPT

# Allow OSD range only on the cluster network
sudo iptables -A INPUT -p tcp -s 10.20.0.0/24 --dport 6800:7300 -j ACCEPT

# Drop unsolicited traffic to Ceph ports (order matters; place after explicit ACCEPT rules)
sudo iptables -A INPUT -p tcp --dport 3300 -j DROP
sudo iptables -A INPUT -p tcp --dport 6789 -j DROP
sudo iptables -A INPUT -p tcp --dport 6800:7300 -j DROP

• Persist firewall rules using your distro's tooling.

1. Enable encrypted transport (Messenger v2)

• Ensure msgr2 is bound and reachable (port 3300) and legacy msgr1 is not required by your clients.
• The earlier config toggle ensures insecure reclaim is disabled:

ceph config get mon auth_allow_insecure_global_id_reclaim

• Verify that clients and daemons prefer v2 addresses:

ceph mon dump | grep v2:
ceph osd dump | grep v2:

1. Restrict and encrypt the Ceph dashboard

• Enable HTTPS for the dashboard with your cert and key:

ceph dashboard set-ssl-certificate -i dashboard.crt
ceph dashboard set-ssl-private-key -i dashboard.key
ceph config set mgr mgr/dashboard/ssl true

• Create a dedicated admin role for day-to-day work (avoid overusing the built-in admin):

ceph dashboard ac-user-create secadmin 'StrongP4ssw0rd!' administrator

• Expose the dashboard only on trusted interfaces and via a reverse proxy if desired.

1. Credential rotation without downtime

• Create a new principal, switch apps, then retire the old one:

# Create a new client with the same caps
sudo ceph auth get-or-create client.rbduser_v2 \
  mon 'profile rbd' \
  osd 'profile rbd pool=vms' \
  mgr 'allow r' \
  -o /etc/ceph/ceph.client.rbduser_v2.keyring

# Update app config to use client.rbduser_v2, then verify access

# Remove the old identity when safe
ceph auth rm client.rbduser

1. Validation checks (repeatable)

• Access scoped correctly:

ceph auth list | grep client.rbduser
ceph auth get client.rbduser -f json-pretty

• Network exposure minimal:

ss -tulpn | grep -E 'ceph|3300|6789|6800|7300'

• On-wire security enforced:

ceph config get mon auth_allow_insecure_global_id_reclaim
ceph mon dump | grep -E 'v2:'

• Cluster health steady after each change:

ceph -s
ceph health detail

Local Pilot Plan

Objective: safely test least-privilege access and transport hardening for one application using a single pool, and tighten host firewalling on one node.

Scope

• Storage: pool vms
• App: a single VM host mapping one RBD image
• Nodes touched: one OSD host (firewall), monitors (config flags), and the app host (client identity)

Steps

1. Prepare and snapshot current state

ceph -s
ceph config dump -f json-pretty > pilot-pre-config.json

1. Create a restricted client for the app

sudo ceph auth get-or-create client.vmapp \
  mon 'profile rbd' \
  osd 'profile rbd pool=vms' \
  mgr 'allow r' \
  -o /etc/ceph/ceph.client.vmapp.keyring
sudo chown ceph: ceph /etc/ceph/ceph.client.vmapp.keyring
sudo chmod 600 /etc/ceph/ceph.client.vmapp.keyring

1. Map and use an RBD image with the new identity

# On the app host
rbd -n client.vmapp --keyring /etc/ceph/ceph.client.vmapp.keyring \
  map vms/test-image

# Simple IO test
sudo dd if=/dev/zero of=/dev/rbd/vms/test-image bs=1M count=64 oflag=direct

1. Enforce on-wire security

ceph config set mon auth_allow_insecure_global_id_reclaim false
# Confirm v2 endpoints are in use
ceph mon dump | grep v2:

1. Lock down firewall on one OSD host (pilot)

# Allow cluster network OSD traffic only from cluster subnet
sudo iptables -A INPUT -p tcp -s 10.20.0.0/24 --dport 6800:7300 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 6800:7300 -j DROP

1. Validate and measure

• Function: app IO completes, no auth errors in logs
• Security: msgr2 endpoints advertised, insecure reclaim disabled
• Exposure: only expected ports open

ceph -s
journalctl -u ceph-* | tail -n 100
ss -tulpn | grep -E '3300|6789|6800|7300'

Rollback (if needed)

• Revert firewall rules by deleting the new DROP rules
• Temporarily re-enable old client identity on the app
• Restore config snapshot if a toggle caused issues:

# Example of removing a DROP rule (adjust rule numbers to match your system)
sudo iptables -L INPUT --line-numbers
sudo iptables -D INPUT <rule_number>

Success criteria

• The app functions using only client.vmapp
• Unauthorized pools are inaccessible
• Only necessary ports are open on the pilot host

Conclusion

Hardening Ceph is most effective when you:

• Enforce cephx with least-privilege caps
• Protect keyrings with strict file permissions and limited distribution
• Limit network exposure to known subnets and required ports
• Prefer messenger v2 and disable insecure behaviors
• Restrict and encrypt the dashboard
• Rotate credentials with a side-by-side cutover
• Verify with simple, repeatable checks after each change

Quick checks before wider rollout

# Auth must be least-privilege and correct
ceph auth get client.vmapp -f json-pretty

# On-wire security must be in effect
ceph config get mon auth_allow_insecure_global_id_reclaim
ceph mon dump | grep v2:

# Only necessary ports must be reachable
ss -tulpn | grep -E 'ceph|3300|6789|6800|7300'

# Cluster must be healthy
ceph -s

Adopt the pilot, adjust for your topology, and expand in measured steps across pools and nodes. Document each decision so future maintenance and incident response are faster and safer.