Ceph Security Hardening: Practical Implementation Guide

Ceph is a powerful distributed storage system, but default settings leave gaps an attacker can exploit. This guide walks you through hardening a Ceph cluster in small, verifiable steps — authentication, secrets, network exposure, transport encryption, dashboard access, and credential rotation — with a local pilot you can run before rolling out cluster-wide.

Workflow Overview

Apply each step, verify, then proceed. The workflow is designed for production safety and clarity.

1. Inventory and Baseline

Capture current state so you can trace changes and roll back if needed.

# List versions and health
ceph versions
ceph -s
ceph health detail

# Snapshot effective config for traceability
ceph config dump -f json-pretty > ceph-config-baseline.json

2. Lock Down Authentication and Authorization (cephx)

Disable the insecure global ID reclaim behavior — required for secure Messenger v2 messaging.

ceph config get mon auth_allow_insecure_global_id_reclaim
ceph config set mon auth_allow_insecure_global_id_reclaim false

Create a least-privilege client for RBD access to a single pool (example pool: vms).

sudo ceph auth get-or-create client.rbduser \
  mon 'profile rbd' \
  osd 'profile rbd pool=vms' \
  mgr 'allow r' \
  -o /etc/ceph/ceph.client.rbduser.keyring

Tighten caps for an existing client in place:

ceph auth caps client.rbduser \
  mon 'profile rbd' \
  osd 'profile rbd pool=vms' \
  mgr 'allow r'

For CephFS, use path-based caps. This example grants read-only access to /var/www:

ceph fs authorize cephfs client.webuser /var/www r

3. Secrets at Rest: Protect Keyrings and Admin Material

Restrict filesystem permissions so only the ceph user can read keyrings.

sudo chown ceph:ceph /etc/ceph/ceph.client.rbduser.keyring
sudo chmod 600 /etc/ceph/ceph.client.rbduser.keyring

Keep admin keyrings off application hosts. Deploy only the restricted client identity on app nodes.

4. Reduce Network Exposure

Separate public and cluster networks in ceph.conf and bind Messenger v2.

# /etc/ceph/ceph.conf
[global]
public_network = 10.10.0.0/24
cluster_network = 10.20.0.0/24
ms_bind_msgr2 = true
auth_allow_insecure_global_id_reclaim = false

Apply host firewall rules. Allow only required ports from trusted subnets. This iptables example assumes the subnets above; adjust for your topology.

# Allow monitors on public network (msgr2 3300, legacy 6789) from trusted subnet
sudo iptables -A INPUT -p tcp -s 10.10.0.0/24 --dport 3300 -j ACCEPT
sudo iptables -A INPUT -p tcp -s 10.10.0.0/24 --dport 6789 -j ACCEPT

# Allow OSD range only on the cluster network
sudo iptables -A INPUT -p tcp -s 10.20.0.0/24 --dport 6800:7300 -j ACCEPT

# Drop unsolicited traffic to Ceph ports (place after explicit ACCEPT rules)
sudo iptables -A INPUT -p tcp --dport 3300 -j DROP
sudo iptables -A INPUT -p tcp --dport 6789 -j DROP
sudo iptables -A INPUT -p tcp --dport 6800:7300 -j DROP

Persist rules using your distribution's tooling (iptables-save, nftables, firewalld, or ufw).

5. Enable Encrypted Transport (Messenger v2)

Ensure msgr2 is bound on port 3300 and legacy msgr1 is not required by your clients.

# Verify insecure reclaim is disabled
ceph config get mon auth_allow_insecure_global_id_reclaim

# Confirm clients and daemons advertise v2 endpoints
ceph mon dump | grep v2:
ceph osd dump | grep v2:

If any daemon still lists only v1: addresses, check ms_bind_msgr2 and client library versions.

6. Restrict and Encrypt the Ceph Dashboard

Enable HTTPS with your own certificate and key.

ceph dashboard set-ssl-certificate -i dashboard.crt
ceph dashboard set-ssl-private-key -i dashboard.key
ceph config set mgr mgr/dashboard/ssl true

Create a dedicated admin role for day-to-day operations instead of overusing the built-in admin user.

ceph dashboard ac-user-create secadmin 'StrongP4ssw0rd!' administrator

Expose the dashboard only on trusted interfaces; place a reverse proxy (nginx, HAProxy) in front if you need additional auth or rate limiting.

7. Credential Rotation Without Downtime

Rotate keys with a side-by-side cutover: create a new principal, switch applications, then retire the old one.

# Create a new client with identical caps
sudo ceph auth get-or-create client.rbduser_v2 \
  mon 'profile rbd' \
  osd 'profile rbd pool=vms' \
  mgr 'allow r' \
  -o /etc/ceph/ceph.client.rbduser_v2.keyring

# Update app config to use client.rbduser_v2, then verify access
# Remove the old identity when safe
ceph auth rm client.rbduser

8. Validation Checks (Repeatable After Each Change)

Run these quick checks to confirm hardening is effective.

# Access scoped correctly
ceph auth list | grep client.rbduser
ceph auth get client.rbduser -f json-pretty

# Network exposure minimal
ss -tulpn | grep -E 'ceph|3300|6789|6800|7300'

# On-wire security enforced
ceph config get mon auth_allow_insecure_global_id_reclaim
ceph mon dump | grep -E 'v2:'

# Cluster health steady
ceph -s
ceph health detail

Local Pilot Plan

Objective: Safely test least-privilege access and transport hardening for one application using a single pool, and tighten host firewalling on one node.

Scope

• Storage: pool vms
• App: a single VM host mapping one RBD image
• Nodes touched: one OSD host (firewall), monitors (config flags), and the app host (client identity)

Steps

1. Prepare and snapshot current state

ceph -s
ceph config dump -f json-pretty > pilot-pre-config.json

1. Create a restricted client for the app

sudo ceph auth get-or-create client.vmapp \
  mon 'profile rbd' \
  osd 'profile rbd pool=vms' \
  mgr 'allow r' \
  -o /etc/ceph/ceph.client.vmapp.keyring
sudo chown ceph:ceph /etc/ceph/ceph.client.vmapp.keyring
sudo chmod 600 /etc/ceph/ceph.client.vmapp.keyring

1. Map and use an RBD image with the new identity

# On the app host
rbd -n client.vmapp --keyring /etc/ceph/ceph.client.vmapp.keyring \
  map vms/test-image

# Simple IO test
sudo dd if=/dev/zero of=/dev/rbd/vms/test-image bs=1M count=64 oflag=direct

1. Enforce on-wire security

ceph config set mon auth_allow_insecure_global_id_reclaim false
# Confirm v2 endpoints are in use
ceph mon dump | grep v2:

1. Lock down firewall on one OSD host (pilot)

# Allow cluster network OSD traffic only from cluster subnet
sudo iptables -A INPUT -p tcp -s 10.20.0.0/24 --dport 6800:7300 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 6800:7300 -j DROP

1. Validate and measure

• Function: app IO completes, no auth errors in logs
• Security: msgr2 endpoints advertised, insecure reclaim disabled
• Exposure: only expected ports open

ceph -s
journalctl -u ceph-* | tail -n 100
ss -tulpn | grep -E '3300|6789|6800|7300'

Rollback (if needed)

• Revert firewall rules by deleting the new DROP rules
• Temporarily re-enable the old client identity on the app
• Restore config snapshot if a toggle caused issues

# Example: remove a DROP rule (adjust rule number to match your system)
sudo iptables -L INPUT --line-numbers
sudo iptables -D INPUT <rule_number>

Success Criteria

• The app functions using only client.vmapp
• Unauthorized pools are inaccessible
• Only necessary ports are open on the pilot host

Conclusion

Hardening Ceph is most effective when you:

• Enforce cephx with least-privilege caps
• Protect keyrings with strict file permissions and limited distribution
• Limit network exposure to known subnets and required ports
• Prefer Messenger v2 and disable insecure behaviors
• Restrict and encrypt the dashboard
• Rotate credentials with a side-by-side cutover
• Verify with simple, repeatable checks after each change

Quick Checks Before Wider Rollout

# Auth must be least-privilege and correct
ceph auth get client.vmapp -f json-pretty

# On-wire security must be in effect
ceph config get mon auth_allow_insecure_global_id_reclaim
ceph mon dump | grep v2:

# Only necessary ports must be reachable
ss -tulpn | grep -E 'ceph|3300|6789|6800|7300'

# Cluster must be healthy
ceph -s

Adopt the pilot, adjust for your topology, and expand in measured steps across pools and nodes. Document each decision so future maintenance and incident response are faster and safer.