Proxmox cluster quorum troubleshooting with practical examples: practical implementation guide

Intro

Quorum is the safety belt of a Proxmox VE cluster. It ensures that only a majority partition can schedule workloads, preventing data corruption and split brain when the cluster is partitioned. Under the hood, Proxmox relies on corosync for membership and quorum. When quorum is lost, the Proxmox cluster filesystem (pmxcfs at /etc/pve) becomes read-only and high availability actions pause by design.

This guide explains quorum and votes, common failure modes, the exact commands to inspect corosync state, and safe recovery techniques. You will find practical examples for 3-node and 5-node clusters, advice for 2-node designs, and a small pilot you can run locally to validate your approach before changing production.

Workflow Overview

Use this step-by-step workflow anytime you suspect quorum issues or see nodes flapping between online and unknown:

1. Map the intended quorum design

• Count voting participants. By default, each node has 1 vote. Aim for an odd number of voters (3, 5, 7...).
• For 2-node clusters, add a tie-breaker (QDevice) or a third lightweight node. Running 2 nodes without a tie-breaker is fragile.
• Document the corosync rings (ring0 and optional ring1) and the dedicated network(s) they use.

Run on any node:

1. Check current quorum status

pvecm status

Key lines:

• Quorate: Yes/No
• Expected votes, Total votes, Quorum (required count)
• Membership list and which nodes are seen

Also check corosync directly:

corosync-quorumtool -s

This confirms whether corosync considers the cluster quorate and lists vote counts.

Example: 3-node cluster healthy

# pvecm status
Quorate:          Yes
Expected votes:   3
Total votes:      3
Quorum:           2
Membership:
    Nodeid      Votes Name
0x00000001          1 10.0.0.11 (local)
0x00000002          1 10.0.0.12
0x00000003          1 10.0.0.13

1. Inspect transport and ring health

• Show ring health and interface status:

systemctl status corosync
journalctl -u corosync -b --no-pager | tail -n 200

• Confirm the active configuration:

cat /etc/pve/corosync.conf

Typical snippet:

totem {
  version: 2
  cluster_name: pve
}

nodelist {
  node {
    name: pve1
    nodeid: 1
    ring0_addr: 10.10.10.11
    ring1_addr: 172.16.10.11
  }
  node {
    name: pve2
    nodeid: 2
    ring0_addr: 10.10.10.12
    ring1_addr: 172.16.10.12
  }
  node {
    name: pve3
    nodeid: 3
    ring0_addr: 10.10.10.13
    ring1_addr: 172.16.10.13
  }
}

• Network checks for ring0/ring1:
• Ping between every pair of nodes on ring0 and ring1 subnets.
• Verify MTU is consistent end-to-end. If you use jumbo frames on one side, use them everywhere on that path.
• Keep corosync on a quiet, dedicated VLAN or interface separate from large east-west VM traffic.
• Ensure DNS/hosts names in corosync.conf resolve identically on all nodes, or prefer IPs.

1. Understand votes, expected votes, and split brain risk

• Quorum requires a strict majority of expected votes.
• If a 3-node cluster loses 1 node, remaining 2 keep quorum (2 of 3). If it loses 2 nodes, quorum is lost.
• Never run guests that write to shared storage on a partition that lacks quorum. That is how split brain happens.

1. Safe remediation patterns

• Prefer restoring connectivity over overrides. Fix the network first.
• If a single node is isolated: repair its ring NIC/VLAN, check switch ports, cabling, and host firewall.
• After recovering a node, ensure corosync is running:

systemctl restart corosync

• Use quorum overrides only in a controlled maintenance window where you are certain no other partition can run the same guests or access the same shared storage. Example (temporary and risky):

# Temporarily reduce expected votes to 1 (do not use during a live split)
pvecm expected 1

This is a last resort to regain write access to /etc/pve for repairs when you have physically isolated all other nodes. Remove the override by restarting corosync after repairs.

1. Practical examples

• Example A: 3-node cluster, one node down
• Expected votes: 3, Quorum: 2. With 2 nodes up, Quorate: Yes. No action required; investigate the failed node while the cluster runs.

• Example B: 5-node cluster, two nodes down
• Expected votes: 5, Quorum: 3. With 3 nodes up, Quorate: Yes. Capacity is reduced; plan repair but services remain safe.

• Example C: 2-node cluster without a tie-breaker
• When either node fails or the link flaps, quorum frequently drops. Add a QDevice on a third host to provide an extra vote, or convert to a 3-node design.

1. Proxmox, HA, and storage cautions

• If HA is enabled, do not force start services on a node without quorum.
• With Ceph or shared storage, avoid any action that could create two primaries writing concurrently. Validate quorum and storage health first.
• Editing /etc/pve/corosync.conf requires quorum; when quorum is lost, this filesystem is read-only by design. Recover quorum first or use a planned maintenance override with strict isolation.

Local Pilot Plan

A small, local pilot lets you prove your approach before touching production.

Scope

• Build a 3-node lab cluster (nested or spare hosts). Target two corosync rings on dedicated VLANs, plus a separate bridge for VM traffic.

Success criteria

• Losing 1 node in 3 remains quorate (2 of 3).
• Losing ring0 or ring1 alone does not drop quorum (knet should use the surviving ring).
• Recovery time from a single-node failure is under your SLO (for example, detect within 10 seconds, stable within 60 seconds).

Setup checklist

• Install Proxmox VE on all nodes with identical versions.
• Ensure consistent time sync (NTP) and matching MTU across ring NICs.
• Join nodes with pvecm and confirm:

pvecm status
corosync-quorumtool -s

• Configure optional ring1 for redundancy, then verify both paths are used.

Failure drills (observe and record)

1. Pull the ring0 cable from one node

• Expected: corosync switches to ring1, cluster stays quorate.
• Validate with pvecm status.

1. Power off one node

• Expected: Remaining 2 nodes stay quorate. Note detection and stabilization times from logs.

1. Introduce MTU mismatch on one link

• Expected: Packet loss causes corosync warnings; fix MTU and confirm stability returns.

Measurements to capture

• pvecm status and corosync-quorumtool -s before, during, after each drill.
• journalctl -u corosync timestamps for detection and recovery.

Exit criteria

• All drills meet success criteria twice in a row without operator intervention beyond the intended repair. Document the exact commands and observations for your production runbook.

Conclusion

Quorum problems are usually network or configuration issues around corosync. Start with a clear view of expected votes, confirm quorum state with pvecm status and corosync-quorumtool -s, and fix connectivity rather than reaching for overrides. Keep corosync on stable, low-latency paths, prefer odd numbers of voters, and add a tie-breaker to any 2-node design.

Next steps

• Add the checks in this guide to your standard operating procedures.
• Run the local pilot and capture timings as a baseline.
• Review ring design for redundancy, MTU consistency, and isolation from VM traffic.
• Rehearse failure drills quarterly so on-call engineers know exactly what to do.

With a disciplined approach and small, measurable pilots, you can diagnose quorum issues quickly and keep your Proxmox clusters safe and predictable.