Ceph Networking Troubleshooting: A Practical Guide with Examples

Ceph performance and stability depend on fast, reliable network paths between monitors, managers, OSDs, metadata servers, and clients. When networking breaks, symptoms range from slow ops and degraded placement groups to MON quorum loss. This guide gives you a safe, stepwise method to diagnose and fix DNS, ports, routing, firewalls, and MTU issues with concrete commands and predictable outcomes.

Scope note: All IPs, hostnames, and subnets in this article are constructed examples for illustration. Replace them with your environment values.

Intro

You will:

• Establish a clean inventory of versions and network topology.
• Validate DNS and name resolution without risking data movement.
• Confirm that required ports are open and listening.
• Test connectivity and MTU end to end across public and cluster networks.
• Identify routing and firewall issues and fix them safely with rollbacks.
• Verify success with observable Ceph health and network checks.

Who should use this: developers, DevOps consultants, and startup teams running or supporting Ceph clusters on Linux (including on platforms like Proxmox) who need a practical, low-risk troubleshooting flow.

Version and Environment Inventory

Before you change anything, capture the current state. This lets you compare before/after and recover quickly if needed.

Prerequisites:

• Shell access to at least one monitor node and one OSD node with sudo.
• Basic familiarity with Ceph CLI and Linux networking commands.
• A maintenance window if you plan to change firewall, MTU, or routing.

Record these on each node:

1. Ceph versions and health

   ceph -s
   ceph health detail
   ceph versions
   ceph mon dump -f json-pretty | jq '.mons[] | {name, public_addrs}'

Expected:

• HEALTH_OK or clear WARN messages you can triage.
• Monitor addresses that match your intended public network.

1. Host identity and DNS

   hostname -f
   hostname -I
   getent hosts $(hostname -f)
   getent ahosts $(hostname -f)

Expected:

• FQDN resolves to the correct IPv4/IPv6 used by Ceph.

1. Interfaces, IPs, and MTU

   ip -br a
   ip -br link
   ethtool -k eth0 | grep -E 'tso|gso|gro'

Expected:

• Correct addresses on public and (if used) cluster networks.
• MTU values consistent across peering nodes.

1. Routing

   ip route
   ip rule
   sysctl net.ipv4.conf.all.rp_filter
   sysctl net.ipv4.ip_forward

Expected:

• Deterministic routes to peer subnets; rp_filter=1 can drop asymmetric return paths.

Use the tool your distro manages:

1. Firewalls

   # firewalld
   sudo firewall-cmd --state || true
   sudo firewall-cmd --list-all || true

   # ufw
   sudo ufw status verbose || true

   # nftables
   sudo nft list ruleset || true

Expected:

• Explicit allowances for Ceph ports between cluster nodes.

1. Time sync (affects networking symptoms like timeouts)

   timedatectl
   chronyc sources -v || true

Expected:

• Synchronized clocks across nodes.

Ceph service ports (reference)

Use this table to confirm listening sockets and firewall allowances. Adjust for your deployment specifics.

Safe Configuration Path

Keep the first iteration narrow, measurable, and easy to inspect locally before deployment. The goal is to isolate networking layers without triggering unnecessary data movement.

If you expect brief OSD flaps during tests, set a guardrail and remember to remove it:

1. Freeze unwanted data movement (temporary)

   ceph osd set noout
   # Optional if you expect heavy link changes; use sparingly
   # ceph osd set norecover
   # ceph osd set nobackfill

Verification:

   ceph osd dump | grep flags

Rollback:

   ceph osd unset noout
   # ceph osd unset norecover
   # ceph osd unset nobackfill

1. Back up configs locally

   sudo cp -a /etc/ceph/ceph.conf /etc/ceph/ceph.conf.bak-$(date -Iseconds)
   sudo cp -a /etc/sysctl.conf /etc/sysctl.conf.bak-$(date -Iseconds) || true

1. Scope changes

• Test DNS and port reachability first. No service restarts required.
• Change one variable at a time (e.g., MTU), verify, then proceed.
• Prefer additive firewall allows over broad policy flips.

1. Plan a quick rollback

• Keep terminal notes of each command you run.
• For network settings, know the previous MTU/gateway and how to revert.
• Test on a single non-critical OSD node before cluster-wide changes.

Verification and Diagnostics

This section provides concrete checks, expected outputs, and how to interpret failures.

DNS and name resolution

Consistent hostname resolution avoids subtle misroutes, especially on dual-stack clusters.

1. Resolve FQDN to A/AAAA records

   dig +short A mon1.ceph.example.com
   dig +short AAAA mon1.ceph.example.com

Expected:

• IPs match what ceph mon dump shows for the monitor.

1. Reverse DNS (optional but helpful)

   dig +short -x 10.10.10.11

Expected:

• Returns the FQDN you use consistently in configs.

1. Systemd-resolved or nswitch path

   resolvectl dns
   resolvectl query mon1.ceph.example.com
   getent hosts mon1.ceph.example.com

Expected:

• Same answer path on all nodes. If nodes disagree, align resolvers or /etc/nsswitch.conf ordering.

Fixes:

• Standardize /etc/hosts only for emergency testing; do not leave permanent host-only overrides that diverge from DNS.
• Ensure search domains do not cause accidental short-name collisions.

Port listening and reachability

1. Confirm listeners on each node

   ss -ltnp | grep -E ':(3300|6789|68[0-9]{2,3}|7480)'

Expected:

• MONs listen on 3300 (and possibly 6789).
• OSD/MGR/MDS listen in the 6800-7300 range.

1. Test from peer nodes

   # From mon2 to mon1
   nc -vz mon1.ceph.example.com 3300
   nc -vz mon1.ceph.example.com 6789 || true

   # From osd3 to osd5 (constructed example port)
   nc -vz osd5.ceph.example.com 6802

Expected:

• Connection succeeded messages. Timeouts or refusals indicate firewall or listener issues.

1. Firewall validation

• firewalld example (allow across a trusted zone):

     sudo firewall-cmd --zone=trusted --add-port=3300/tcp --permanent
     sudo firewall-cmd --zone=trusted --add-port=6789/tcp --permanent
     sudo firewall-cmd --zone=trusted --add-port=6800-7300/tcp --permanent
     sudo firewall-cmd --reload

• ufw example:

     sudo ufw allow 3300/tcp comment 'Ceph MON v2'
     sudo ufw allow 6789/tcp comment 'Ceph MON v1'
     sudo ufw allow 6800:7300/tcp comment 'Ceph OSD/MGR/MDS'

Verification:

   sudo nft list ruleset | grep -E '3300|6789|6800'

Routing and asymmetric paths

• Trace the path

  # IPv4
  mtr -wrc 10 mon1.ceph.example.com
  # IPv6
  mtr -wrc 10 -6 mon1.ceph.example.com

Expected:

• Stable path with low loss. Wildly different forward/return paths can trigger rp_filter drops.

• Check reverse path filtering

  sysctl net.ipv4.conf.all.rp_filter
  sysctl net.ipv4.conf.default.rp_filter

Interpretation:

• Value 1 (strict) can drop asymmetrically routed replies. If you intentionally route asymmetrically on storage networks, consider 2 (loose) for the specific interface:

    echo 'net.ipv4.conf.eth1.rp_filter=2' | sudo tee /etc/sysctl.d/60-ceph-rpfilter.conf
    sudo sysctl --system

Rollback:

  sudo rm /etc/sysctl.d/60-ceph-rpfilter.conf
  sudo sysctl -w net.ipv4.conf.eth1.rp_filter=1

MTU and fragmentation

Mismatch in MTU can cause OSD heartbeats and client ops to stall.

1. Verify MTU on peers

   ip -br link | awk '{print $1, $3}'

1. Probe path MTU (IPv4 example for 9000 MTU)

   ping -c 3 -M do -s 8972 osd5.ceph.example.com

Expected:

Note: 8972 = 9000 MTU minus 28 bytes for IP/ICMP headers.

• No fragmentation required. If it fails, align MTU across the entire path or use standard 1500.

Ceph health symptoms that point to network

Run on a monitor node:

ceph -s
ceph health detail
ceph osd tree

Common network-linked messages include:

• OSD_DOWN or OSD_TIMEOUT: often link, MTU, firewall, or route.
• MON_DOWN or election storms: monitor port reachability, time sync, DNS.
• SLOW_OPS or degraded PGs with many remapped: intermittent packet loss.

Packet-level inspection (last resort during a window)

Use only when you cannot isolate the issue with previous steps.

sudo tcpdump -ni eth1 tcp port 3300 or tcp port 6789 or portrange 6800-7300

Expected:

• SYN/SYN-ACK completes. Repeated SYNs without SYN-ACK indicate filtering or no listener.

Failure Modes and Recovery

Use this mapping to go from symptom to fix quickly.

Recovery recipes

1. Fix DNS safely

• Correct A/AAAA records or host entries on an isolated test node first.
• Verify with dig and resolvectl across all nodes.
• No daemon restarts are needed if monmap uses IPs. If you changed monitor names or addresses in configuration-only clusters, schedule a controlled restart of monitors one at a time.

1. Reopen ports

• Apply targeted firewall rules as shown above.
• Verify with nc -vz from all peers.
• Persist rules and reload.

1. Align MTU

• If using jumbo MTU, ensure switches and NICs match. Otherwise, set all storage interfaces to 1500 for baseline consistency:

     sudo ip link set dev eth1 mtu 1500

• Make it persistent via your distro network config tool.
• Verify with ping -M do -s 1472.

1. Correct routing

• Add an explicit route for the cluster subnet if traffic was taking a default gateway:

     sudo ip route add 10.20.0.0/24 dev eth1 proto static

• For policy routing with multiple interfaces, ensure symmetrical return paths with ip rule and per-interface tables.
• If rp_filter blocked traffic, set loose mode on the storage interface only and document it.

1. Rollback plan

• If a change worsens health, immediately revert the last step: restore the prior MTU/gateway, remove the last firewall rule, or delete the test sysctl drop-in and reload.
• Confirm with ceph -s and targeted reachability tests.

Operations Checklist

Use this during incidents or post-change validation.

1. Baseline

• [ ] ceph -s is reachable and reports health messages.
• [ ] ceph mon dump shows expected addresses.
• [ ] Time is synchronized (timedatectl).

1. DNS

• [ ] dig A/AAAA host returns intended IPs.
• [ ] getent hosts consistent on all nodes.
• [ ] No stale /etc/hosts overrides left behind.

1. Ports and listeners

• [ ] ss -ltnp shows MON on 3300 (and optionally 6789).
• [ ] OSD/MGR/MDS listen in 6800-7300.
• [ ] nc -vz succeeds between peers on required ports.

1. Firewall

• [ ] Rules allow 3300, 6789, 6800-7300 between cluster nodes.
• [ ] Rules persisted and documented.

1. Routing and MTU

• [ ] mtr shows stable, symmetric paths.
• [ ] rp_filter appropriate for your routing.
• [ ] MTU aligned end to end; ping -M do passes.

1. Ceph verification

• [ ] No new OSD flaps during and after changes.
• [ ] ceph health detail free of network-related warnings.
• [ ] If used, remove guardrails: ceph osd unset noout (and others).

Practical examples

These end-to-end examples show how to apply the steps. All values are constructed examples.

Example 1: MON quorum lost after DNS change

Symptoms: ceph -s from a surviving node shows 1/3 mons in quorum; others oscillate.

Steps:

1. Check resolution:

   dig +short A mon2.ceph.example.com
   getent hosts mon2.ceph.example.com

Observed: mon2 now resolves to 10.10.20.15, but monmap shows 10.10.10.15.

1. Fix DNS record to 10.10.10.15. Purge caches (resolvectl flush-caches if applicable).

1. Verify reachability:

   nc -vz mon2.ceph.example.com 3300

1. Confirm quorum recovery:

   ceph -s

Expected: All monitors rejoin quorum; health improves.

Rollback: If DNS propagation is delayed, use a temporary /etc/hosts entry on affected nodes, then remove it once DNS is corrected.

Example 2: OSD flaps due to MTU mismatch

Symptoms: Repeated OSD_DOWN/UP events and slow ops; switches recently enabled jumbo frames.

Steps:

1. Check MTU on peers:

   ip -br link | grep eth1

Observed: Some nodes at 9000, others at 1500.

1. Choose a safe baseline (1500) to restore stability:

   sudo ip link set dev eth1 mtu 1500

1. Verify end-to-end with ping -M do -s 1472 osdX.

1. Confirm stability:

   ceph -s
   ceph health detail

Expected: OSDs stop flapping; health stabilizes.

Rollback: If performance regresses later, reintroduce jumbo MTU only after validating every hop supports it and updating all nodes consistently.

Example 3: Asymmetric routing breaks OSD heartbeats

Symptoms: OSDs on subnet 10.20.0.0/24 intermittently timeout; return traffic takes default gateway on some nodes.

Steps:

1. Confirm path asymmetry:

   mtr -wrc 10 osd8.ceph.example.com
   mtr -wrc 10 osd3.ceph.example.com

1. Add an explicit route on affected nodes:

   sudo ip route add 10.20.0.0/24 dev eth1 proto static

1. If still dropping, set interface-specific loose rp_filter:

   echo 'net.ipv4.conf.eth1.rp_filter=2' | sudo tee /etc/sysctl.d/60-ceph-rpfilter.conf
   sudo sysctl --system

1. Verify with mtr again and watch Ceph health.

Rollback: Remove the route or sysctl drop-in if it introduces unexpected flows.

Conclusion

Reliable Ceph networking comes from disciplined inventory, focused tests, and small, reversible changes. Start by confirming DNS and required ports, then verify routing symmetry and MTU alignment. Use targeted firewall rules and per-interface sysctl changes rather than broad policy shifts. After each change, validate with nc -vz, mtr, ping -M do, and ceph -s. If a step degrades health, roll it back immediately and reassess the last variable changed.

By following the inventory, diagnostics, and recovery steps here, you can restore monitor quorum, stop OSD flapping, and remove slow ops caused by the network with confidence. Keep the operations checklist handy, and standardize these checks as part of your routine maintenance so that future incidents resolve faster and more safely.