Ceph is a powerful distributed storage system, but default settings leave gaps an attacker can exploit. This guide walks you through hardening a Ceph cluster in small, verifiable steps — authentication, secrets, network exposure, transport encryption, dashboard access, and credential rotation — with a local pilot you can run before rolling out cluster-wide.

## Workflow Overview

Apply each step, verify, then proceed. The workflow is designed for production safety and clarity.

### 1. Inventory and Baseline

Capture current state so you can trace changes and roll back if needed.

# List versions and health
ceph versions
ceph -s
ceph health detail

# Snapshot effective config for traceability
ceph config dump -f json-pretty > ceph-config-baseline.json 

### 2. Lock Down Authentication and Authorization (cephx)

 Disable the insecure global ID reclaim behavior — required for secure Messenger v2 messaging.

ceph config get mon auth_allow_insecure_global_id_reclaim
ceph config set mon auth_allow_insecure_global_id_reclaim false 
 Create a least-privilege client for RBD access to a single pool (example pool: vms ).

sudo ceph auth get-or-create client.rbduser \
 mon 'profile rbd' \
 osd 'profile rbd pool=vms' \
 mgr 'allow r' \
 -o /etc/ceph/ceph.client.rbduser.keyring 
 Tighten caps for an existing client in place:

ceph auth caps client.rbduser \
 mon 'profile rbd' \
 osd 'profile rbd pool=vms' \
 mgr 'allow r' 
 For CephFS, use path-based caps. This example grants read-only access to /var/www :

ceph fs authorize cephfs client.webuser /var/www r 

### 3. Secrets at Rest: Protect Keyrings and Admin Material

 Restrict filesystem permissions so only the ceph user can read keyrings.

sudo chown ceph:ceph /etc/ceph/ceph.client.rbduser.keyring
sudo chmod 600 /etc/ceph/ceph.client.rbduser.keyring 
 Keep admin keyrings off application hosts. Deploy only the restricted client identity on app nodes.

### 4. Reduce Network Exposure

Separate public and cluster networks in ceph.conf and bind Messenger v2.

# /etc/ceph/ceph.conf
[global]
public_network = 10.10.0.0/24
cluster_network = 10.20.0.0/24
ms_bind_msgr2 = true
auth_allow_insecure_global_id_reclaim = false 
 Apply host firewall rules. Allow only required ports from trusted subnets. This iptables example assumes the subnets above; adjust for your topology.

# Allow monitors on public network (msgr2 3300, legacy 6789) from trusted subnet
sudo iptables -A INPUT -p tcp -s 10.10.0.0/24 --dport 3300 -j ACCEPT
sudo iptables -A INPUT -p tcp -s 10.10.0.0/24 --dport 6789 -j ACCEPT

# Allow OSD range only on the cluster network
sudo iptables -A INPUT -p tcp -s 10.20.0.0/24 --dport 6800:7300 -j ACCEPT

# Drop unsolicited traffic to Ceph ports (place after explicit ACCEPT rules)
sudo iptables -A INPUT -p tcp --dport 3300 -j DROP
sudo iptables -A INPUT -p tcp --dport 6789 -j DROP
sudo iptables -A INPUT -p tcp --dport 6800:7300 -j DROP 
 Persist rules using your distribution's tooling ( iptables-save , nftables , firewalld , or ufw ).

### 5. Enable Encrypted Transport (Messenger v2)

Ensure msgr2 is bound on port 3300 and legacy msgr1 is not required by your clients.

# Verify insecure reclaim is disabled
ceph config get mon auth_allow_insecure_global_id_reclaim

# Confirm clients and daemons advertise v2 endpoints
ceph mon dump | grep v2:
ceph osd dump | grep v2: 
 If any daemon still lists only v1: addresses, check ms_bind_msgr2 and client library versions.

### 6. Restrict and Encrypt the Ceph Dashboard

Enable HTTPS with your own certificate and key.

ceph dashboard set-ssl-certificate -i dashboard.crt
ceph dashboard set-ssl-private-key -i dashboard.key
ceph config set mgr mgr/dashboard/ssl true 
 Create a dedicated admin role for day-to-day operations instead of overusing the built-in admin user.

ceph dashboard ac-user-create secadmin 'StrongP4ssw0rd!' administrator 
 Expose the dashboard only on trusted interfaces; place a reverse proxy (nginx, HAProxy) in front if you need additional auth or rate limiting.

### 7. Credential Rotation Without Downtime

Rotate keys with a side-by-side cutover: create a new principal, switch applications, then retire the old one.

# Create a new client with identical caps
sudo ceph auth get-or-create client.rbduser_v2 \
 mon 'profile rbd' \
 osd 'profile rbd pool=vms' \
 mgr 'allow r' \
 -o /etc/ceph/ceph.client.rbduser_v2.keyring

# Update app config to use client.rbduser_v2, then verify access
# Remove the old identity when safe
ceph auth rm client.rbduser 

### 8. Validation Checks (Repeatable After Each Change)

 Run these quick checks to confirm hardening is effective.

# Access scoped correctly
ceph auth list | grep client.rbduser
ceph auth get client.rbduser -f json-pretty

# Network exposure minimal
ss -tulpn | grep -E 'ceph|3300|6789|6800|7300'

# On-wire security enforced
ceph config get mon auth_allow_insecure_global_id_reclaim
ceph mon dump | grep -E 'v2:'

# Cluster health steady
ceph -s
ceph health detail 

## Local Pilot Plan

 Objective : Safely test least-privilege access and transport hardening for one application using a single pool, and tighten host firewalling on one node.

### Scope

- Storage: pool vms

- App: a single VM host mapping one RBD image

- Nodes touched: one OSD host (firewall), monitors (config flags), and the app host (client identity)

### Steps

- Prepare and snapshot current state

ceph -s
ceph config dump -f json-pretty > pilot-pre-config.json 

- Create a restricted client for the app

 sudo ceph auth get-or-create client.vmapp \
 mon 'profile rbd' \
 osd 'profile rbd pool=vms' \
 mgr 'allow r' \
 -o /etc/ceph/ceph.client.vmapp.keyring
sudo chown ceph:ceph /etc/ceph/ceph.client.vmapp.keyring
sudo chmod 600 /etc/ceph/ceph.client.vmapp.keyring 

- Map and use an RBD image with the new identity

 # On the app host
rbd -n client.vmapp --keyring /etc/ceph/ceph.client.vmapp.keyring \
 map vms/test-image

# Simple IO test
sudo dd if=/dev/zero of=/dev/rbd/vms/test-image bs=1M count=64 oflag=direct 

- Enforce on-wire security

 ceph config set mon auth_allow_insecure_global_id_reclaim false
# Confirm v2 endpoints are in use
ceph mon dump | grep v2: 

- Lock down firewall on one OSD host (pilot)

 # Allow cluster network OSD traffic only from cluster subnet
sudo iptables -A INPUT -p tcp -s 10.20.0.0/24 --dport 6800:7300 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 6800:7300 -j DROP 

- Validate and measure

- Function: app IO completes, no auth errors in logs

- Security: msgr2 endpoints advertised, insecure reclaim disabled

- Exposure: only expected ports open

 ceph -s
journalctl -u ceph-* | tail -n 100
ss -tulpn | grep -E '3300|6789|6800|7300' 

### Rollback (if needed)

- Revert firewall rules by deleting the new DROP rules

- Temporarily re-enable the old client identity on the app

- Restore config snapshot if a toggle caused issues

 # Example: remove a DROP rule (adjust rule number to match your system)
sudo iptables -L INPUT --line-numbers
sudo iptables -D INPUT <rule_number> 

### Success Criteria

- The app functions using only client.vmapp

- Unauthorized pools are inaccessible

- Only necessary ports are open on the pilot host

## Conclusion

 Hardening Ceph is most effective when you:

- Enforce cephx with least-privilege caps

- Protect keyrings with strict file permissions and limited distribution

- Limit network exposure to known subnets and required ports

- Prefer Messenger v2 and disable insecure behaviors

- Restrict and encrypt the dashboard

- Rotate credentials with a side-by-side cutover

- Verify with simple, repeatable checks after each change

### Quick Checks Before Wider Rollout

# Auth must be least-privilege and correct
ceph auth get client.vmapp -f json-pretty

# On-wire security must be in effect
ceph config get mon auth_allow_insecure_global_id_reclaim
ceph mon dump | grep v2:

# Only necessary ports must be reachable
ss -tulpn | grep -E 'ceph|3300|6789|6800|7300'

# Cluster must be healthy
ceph -s 
 Adopt the pilot, adjust for your topology, and expand in measured steps across pools and nodes. Document each decision so future maintenance and incident response are faster and safer.