Ceph performance and stability depend on fast, reliable network paths between monitors, managers, OSDs, metadata servers, and clients. When networking breaks, symptoms range from slow ops and degraded placement groups to MON quorum loss. This guide gives you a safe, stepwise method to diagnose and fix DNS, ports, routing, firewalls, and MTU issues with concrete commands and predictable outcomes.

Scope note: All IPs, hostnames, and subnets in this article are constructed examples for illustration. Replace them with your environment values.

## Intro

You will:

- Establish a clean inventory of versions and network topology.

- Validate DNS and name resolution without risking data movement.

- Confirm that required ports are open and listening.

- Test connectivity and MTU end to end across public and cluster networks.

- Identify routing and firewall issues and fix them safely with rollbacks.

- Verify success with observable Ceph health and network checks.

Who should use this: developers, DevOps consultants, and startup teams running or supporting Ceph clusters on Linux (including on platforms like Proxmox) who need a practical, low-risk troubleshooting flow.

## Version and Environment Inventory

Before you change anything, capture the current state. This lets you compare before/after and recover quickly if needed.

Prerequisites:

- Shell access to at least one monitor node and one OSD node with sudo.

- Basic familiarity with Ceph CLI and Linux networking commands.

- A maintenance window if you plan to change firewall, MTU, or routing.

Record these on each node:

- Ceph versions and health

ceph -s
 ceph health detail
 ceph versions
 ceph mon dump -f json-pretty | jq '.mons[] | {name, public_addrs}' 
 Expected:

- HEALTH_OK or clear WARN messages you can triage.

- Monitor addresses that match your intended public network.

- Host identity and DNS

hostname -f
 hostname -I
 getent hosts $(hostname -f)
 getent ahosts $(hostname -f) 
 Expected:

- FQDN resolves to the correct IPv4/IPv6 used by Ceph.

- Interfaces, IPs, and MTU

ip -br a
 ip -br link
 ethtool -k eth0 | grep -E 'tso|gso|gro' 
 Expected:

- Correct addresses on public and (if used) cluster networks.

- MTU values consistent across peering nodes.

- Routing

ip route
 ip rule
 sysctl net.ipv4.conf.all.rp_filter
 sysctl net.ipv4.ip_forward 
 Expected:

- Deterministic routes to peer subnets; rp_filter=1 can drop asymmetric return paths.

Use the tool your distro manages:

- Firewalls

# firewalld
 sudo firewall-cmd --state || true
 sudo firewall-cmd --list-all || true

 # ufw
 sudo ufw status verbose || true

 # nftables
 sudo nft list ruleset || true 
 Expected:

- Explicit allowances for Ceph ports between cluster nodes.

- Time sync (affects networking symptoms like timeouts)

timedatectl
 chronyc sources -v || true 
 Expected:

- Synchronized clocks across nodes.

### Ceph service ports (reference)

Use this table to confirm listening sockets and firewall allowances. Adjust for your deployment specifics.

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Service</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Protocol</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Default ports</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Notes</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">MON (msgr v2)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">TCP</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">3300</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Modern default; ensure open cluster-wide</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">MON (msgr v1)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">TCP</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">6789</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Legacy; may be disabled or dual-stack with v2</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">OSD</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">TCP</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">6800-7300</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Range depends on OSD count and binding</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">MGR</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">TCP</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">6800-7300</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Shares range with OSD/MDS on host</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">MDS</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">TCP</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">6800-7300</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">CephFS metadata servers</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">RGW (radosgw)</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">TCP</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">7480 or 80/443</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Frontend may vary by proxy/ingress</td></tr></tbody>
</table>
</div>

## Safe Configuration Path

Keep the first iteration narrow, measurable, and easy to inspect locally before deployment. The goal is to isolate networking layers without triggering unnecessary data movement.

If you expect brief OSD flaps during tests, set a guardrail and remember to remove it:

- Freeze unwanted data movement (temporary)

ceph osd set noout
 # Optional if you expect heavy link changes; use sparingly
 # ceph osd set norecover
 # ceph osd set nobackfill 
 Verification:

ceph osd dump | grep flags 
 Rollback:

ceph osd unset noout
 # ceph osd unset norecover
 # ceph osd unset nobackfill 

- Back up configs locally

 sudo cp -a /etc/ceph/ceph.conf /etc/ceph/ceph.conf.bak-$(date -Iseconds)
 sudo cp -a /etc/sysctl.conf /etc/sysctl.conf.bak-$(date -Iseconds) || true 

- Scope changes

- Test DNS and port reachability first. No service restarts required.

- Change one variable at a time (e.g., MTU), verify, then proceed.

- Prefer additive firewall allows over broad policy flips.

- Plan a quick rollback

- Keep terminal notes of each command you run.

- For network settings, know the previous MTU/gateway and how to revert.

- Test on a single non-critical OSD node before cluster-wide changes.

## Verification and Diagnostics

 This section provides concrete checks, expected outputs, and how to interpret failures.

### DNS and name resolution

Consistent hostname resolution avoids subtle misroutes, especially on dual-stack clusters.

- Resolve FQDN to A/AAAA records

dig +short A mon1.ceph.example.com
 dig +short AAAA mon1.ceph.example.com 
 Expected:

- IPs match what ceph mon dump shows for the monitor.

- Reverse DNS (optional but helpful)

dig +short -x 10.10.10.11 
 Expected:

- Returns the FQDN you use consistently in configs.

- Systemd-resolved or nswitch path

resolvectl dns
 resolvectl query mon1.ceph.example.com
 getent hosts mon1.ceph.example.com 
 Expected:

- Same answer path on all nodes. If nodes disagree, align resolvers or /etc/nsswitch.conf ordering.

Fixes:

- Standardize /etc/hosts only for emergency testing; do not leave permanent host-only overrides that diverge from DNS.

- Ensure search domains do not cause accidental short-name collisions.

### Port listening and reachability

- Confirm listeners on each node

ss -ltnp | grep -E ':(3300|6789|68[0-9]{2,3}|7480)' 
 Expected:

- MONs listen on 3300 (and possibly 6789).

- OSD/MGR/MDS listen in the 6800-7300 range.

- Test from peer nodes

# From mon2 to mon1
 nc -vz mon1.ceph.example.com 3300
 nc -vz mon1.ceph.example.com 6789 || true

 # From osd3 to osd5 (constructed example port)
 nc -vz osd5.ceph.example.com 6802 
 Expected:

- Connection succeeded messages. Timeouts or refusals indicate firewall or listener issues.

- Firewall validation

- firewalld example (allow across a trusted zone):

sudo firewall-cmd --zone=trusted --add-port=3300/tcp --permanent
 sudo firewall-cmd --zone=trusted --add-port=6789/tcp --permanent
 sudo firewall-cmd --zone=trusted --add-port=6800-7300/tcp --permanent
 sudo firewall-cmd --reload 

- ufw example :

 sudo ufw allow 3300/tcp comment 'Ceph MON v2'
 sudo ufw allow 6789/tcp comment 'Ceph MON v1'
 sudo ufw allow 6800:7300/tcp comment 'Ceph OSD/MGR/MDS' 
 Verification:

sudo nft list ruleset | grep -E '3300|6789|6800' 

### Routing and asymmetric paths

- Trace the path

 # IPv4
 mtr -wrc 10 mon1.ceph.example.com
 # IPv6
 mtr -wrc 10 -6 mon1.ceph.example.com 
 Expected:

- Stable path with low loss. Wildly different forward/return paths can trigger rp_filter drops.

- Check reverse path filtering

sysctl net.ipv4.conf.all.rp_filter
 sysctl net.ipv4.conf.default.rp_filter 
 Interpretation:

- Value 1 (strict) can drop asymmetrically routed replies. If you intentionally route asymmetrically on storage networks, consider 2 (loose) for the specific interface:

echo 'net.ipv4.conf.eth1.rp_filter=2' | sudo tee /etc/sysctl.d/60-ceph-rpfilter.conf
 sudo sysctl --system 
 Rollback:

sudo rm /etc/sysctl.d/60-ceph-rpfilter.conf
 sudo sysctl -w net.ipv4.conf.eth1.rp_filter=1 

### MTU and fragmentation

 Mismatch in MTU can cause OSD heartbeats and client ops to stall.

- Verify MTU on peers

ip -br link | awk '{print $1, $3}' 

- Probe path MTU (IPv4 example for 9000 MTU)

 ping -c 3 -M do -s 8972 osd5.ceph.example.com 
 Expected:

Note: 8972 = 9000 MTU minus 28 bytes for IP/ICMP headers.

- No fragmentation required. If it fails, align MTU across the entire path or use standard 1500.

### Ceph health symptoms that point to network

Run on a monitor node:

ceph -s
ceph health detail
ceph osd tree 
 Common network-linked messages include:

- OSD_DOWN or OSD_TIMEOUT : often link, MTU, firewall, or route.

- MON_DOWN or election storms: monitor port reachability, time sync, DNS.

- SLOW_OPS or degraded PGs with many remapped: intermittent packet loss.

### Packet-level inspection (last resort during a window)

Use only when you cannot isolate the issue with previous steps.

sudo tcpdump -ni eth1 tcp port 3300 or tcp port 6789 or portrange 6800-7300 
 Expected:

- SYN/SYN-ACK completes. Repeated SYNs without SYN-ACK indicate filtering or no listener.

## Failure Modes and Recovery

Use this mapping to go from symptom to fix quickly.

<div class="my-stack-md overflow-x-auto">
<table class="w-full min-w-[42rem] border-collapse text-left">
<thead><tr><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Symptom</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Likely cause</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Quick check</th><th scope="col" class="border border-outline-variant bg-surface-container-low px-4 py-3 text-left font-label-md font-semibold text-on-surface">Safe fix</th></tr></thead>
<tbody><tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">OSDs flapping in/out</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">MTU mismatch or packet loss</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">ping -M do -s 8972 peer</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Align MTU end to end or set 1500 uniformly</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Cannot form MON quorum</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Port blocked or wrong DNS</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">nc -vz monX 3300</code> and <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">dig A/AAAA monX</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Open 3300/tcp and fix DNS records</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Slow ops with spikes</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Asymmetric routing, rp_filter</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">mtr</code> both ways; <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">rp_filter</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Set rp_filter=2 on storage iface or fix routes</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Some nodes IPv6 only</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Incomplete dual-stack config</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">ceph mon dump</code> vs <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">ip -6 a</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Align cluster to IPv4 or full dual-stack</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Client auth hangs</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Firewall at edge</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">nc -vz rgwX 7480</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Allow RGW ports or proxy correctly</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Intermittent timeouts</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">LACP hashing imbalance</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">cat /proc/net/bonding/bond0</code></td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Use layer3+4 hash or single NIC for Ceph traffic</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">NAT in path</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Ceph peers behind NAT</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant"><code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">traceroute</code> shows NAT</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Remove NAT between cluster nodes</td></tr>
<tr><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Wrong network roles</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">public_network/cluster_network mixed</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Inspect <code class="font-mono text-[0.9em] bg-surface-container px-1 py-0.5 rounded">ceph.conf</code> and routes</td><td class="border border-outline-variant px-4 py-3 align-top text-body-md text-on-surface-variant">Correct networks and restart affected daemons</td></tr></tbody>
</table>
</div>

### Recovery recipes

- Fix DNS safely

- Correct A/AAAA records or host entries on an isolated test node first.

- Verify with dig and resolvectl across all nodes.

- No daemon restarts are needed if monmap uses IPs. If you changed monitor names or addresses in configuration-only clusters, schedule a controlled restart of monitors one at a time.

- Reopen ports

- Apply targeted firewall rules as shown above.

- Verify with nc -vz from all peers.

- Persist rules and reload.

- Align MTU

- If using jumbo MTU, ensure switches and NICs match. Otherwise, set all storage interfaces to 1500 for baseline consistency:

sudo ip link set dev eth1 mtu 1500 

- Make it persistent via your distro network config tool.

- Verify with ping -M do -s 1472 .

- Correct routing

- Add an explicit route for the cluster subnet if traffic was taking a default gateway:

 sudo ip route add 10.20.0.0/24 dev eth1 proto static 

- For policy routing with multiple interfaces, ensure symmetrical return paths with ip rule and per-interface tables.

- If rp_filter blocked traffic, set loose mode on the storage interface only and document it.

- Rollback plan

- If a change worsens health, immediately revert the last step: restore the prior MTU/gateway, remove the last firewall rule, or delete the test sysctl drop-in and reload.

- Confirm with ceph -s and targeted reachability tests.

## Operations Checklist

 Use this during incidents or post-change validation.

- Baseline

- [ ] ceph -s is reachable and reports health messages.

- [ ] ceph mon dump shows expected addresses.

- [ ] Time is synchronized ( timedatectl ).

- DNS

- [ ] dig A/AAAA host returns intended IPs.

- [ ] getent hosts consistent on all nodes.

- [ ] No stale /etc/hosts overrides left behind.

- Ports and listeners

- [ ] ss -ltnp shows MON on 3300 (and optionally 6789).

- [ ] OSD/MGR/MDS listen in 6800-7300.

- [ ] nc -vz succeeds between peers on required ports.

- Firewall

- [ ] Rules allow 3300, 6789, 6800-7300 between cluster nodes.

- [ ] Rules persisted and documented.

- Routing and MTU

- [ ] mtr shows stable, symmetric paths.

- [ ] rp_filter appropriate for your routing.

- [ ] MTU aligned end to end; ping -M do passes.

- Ceph verification

- [ ] No new OSD flaps during and after changes.

- [ ] ceph health detail free of network-related warnings.

- [ ] If used, remove guardrails: ceph osd unset noout (and others).

## Practical examples

These end-to-end examples show how to apply the steps. All values are constructed examples.

### Example 1: MON quorum lost after DNS change

Symptoms: ceph -s from a surviving node shows 1/3 mons in quorum; others oscillate.

Steps:

- Check resolution:

dig +short A mon2.ceph.example.com
 getent hosts mon2.ceph.example.com 
 Observed: mon2 now resolves to 10.10.20.15, but monmap shows 10.10.10.15.

- Fix DNS record to 10.10.10.15. Purge caches ( resolvectl flush-caches if applicable).

- Verify reachability:

nc -vz mon2.ceph.example.com 3300 

- Confirm quorum recovery:

 ceph -s 
 Expected: All monitors rejoin quorum; health improves.

Rollback: If DNS propagation is delayed, use a temporary /etc/hosts entry on affected nodes, then remove it once DNS is corrected.

### Example 2: OSD flaps due to MTU mismatch

Symptoms: Repeated OSD_DOWN/UP events and slow ops; switches recently enabled jumbo frames.

Steps:

- Check MTU on peers:

ip -br link | grep eth1 
 Observed: Some nodes at 9000, others at 1500.

- Choose a safe baseline (1500) to restore stability:

sudo ip link set dev eth1 mtu 1500 

- Verify end-to-end with ping -M do -s 1472 osdX .

- Confirm stability:

 ceph -s
 ceph health detail 
 Expected: OSDs stop flapping; health stabilizes.

Rollback: If performance regresses later, reintroduce jumbo MTU only after validating every hop supports it and updating all nodes consistently.

### Example 3: Asymmetric routing breaks OSD heartbeats

Symptoms: OSDs on subnet 10.20.0.0/24 intermittently timeout; return traffic takes default gateway on some nodes.

Steps:

- Confirm path asymmetry:

mtr -wrc 10 osd8.ceph.example.com
 mtr -wrc 10 osd3.ceph.example.com 

- Add an explicit route on affected nodes:

 sudo ip route add 10.20.0.0/24 dev eth1 proto static 

- If still dropping, set interface-specific loose rp_filter:

 echo 'net.ipv4.conf.eth1.rp_filter=2' | sudo tee /etc/sysctl.d/60-ceph-rpfilter.conf
 sudo sysctl --system 

- Verify with mtr again and watch Ceph health.

 Rollback: Remove the route or sysctl drop-in if it introduces unexpected flows.

## Conclusion

Reliable Ceph networking comes from disciplined inventory, focused tests, and small, reversible changes. Start by confirming DNS and required ports, then verify routing symmetry and MTU alignment. Use targeted firewall rules and per-interface sysctl changes rather than broad policy shifts. After each change, validate with nc -vz , mtr , ping -M do , and ceph -s . If a step degrades health, roll it back immediately and reassess the last variable changed.

By following the inventory, diagnostics, and recovery steps here, you can restore monitor quorum, stop OSD flapping, and remove slow ops caused by the network with confidence. Keep the operations checklist handy, and standardize these checks as part of your routine maintenance so that future incidents resolve faster and more safely.