Consensus and recovery
Layer 2. Core. Core
SHC’s control plane uses rqlited on mesh0. The raft and database listeners
bind the node’s mesh address on TCP 4002 and 4001; mesh0 uses WireGuard on
UDP 4244 (ce core/nodeports/ports.go). The Linux container fabric, shc0 on UDP 4243, has its own keys and
underlay endpoints. See cluster join.
Voter policy
Section titled “Voter policy”Voters come from eligible members of the control-plane fence. Host
capabilities constrain membership: Linux supplies voter; macOS supplies
mesh and joins raft as a non-voter.
VoterTarget in ce modules/cluster/voters.go chooses the largest odd
number no greater than the eligible member count, capped at five:
| Eligible members | 0 | 1 | 2 | 3 | 4 | 5 or more |
|---|---|---|---|---|---|---|
| Target voters | 0 | 1 | 1 | 3 | 3 | 5 |
Enrolment claims a unique node.voter_slot through GrantVoterSlot up to
that target. The suffrage reconciler in modules/cluster/suffrage.go
changes one member at a time, removing it from raft and rejoining it with
the requested suffrage while retaining its data directory. A down member
keeps its vote until removal.
How many voters can you lose?
Section titled “How many voters can you lose?”Quorum is floor(configured_voters / 2) + 1. Fault tolerance is
reachable_voters - quorum; an unreachable voter still counts in the
configured set.
| Configured voters | Reachable voters | Quorum | Further losses tolerated |
|---|---|---|---|
| 1 | 1 | 1 | 0 |
| 3 | 3 | 2 | 1 |
| 3 | 2 | 2 | 0 |
| 3 | 1 | 2 | Quorum lost |
| 5 | 5 | 3 | 2 |
A majority permits commits. A minority cannot elect a second committing leader under the same membership. Membership changes also need quorum, so removing dead voters cannot repair a cluster that has already lost it.
Single-writer containers depend on a live node lease. The node agent stops them when it expires; the independent watchdog enforces that stop even if shcd hangs. See the watchdog contract.
Removal guards
Section titled “Removal guards”The ce cluster service checks both configured and reachable voters before
removal. X409201CLSRMQ refuses an insufficient configured set;
X409202CLSRMR refuses insufficient reachable survivors. --force does
not bypass these quorum guards. Draining workloads does not itself remove
a raft member.
5. Recovery
Section titled “5. Recovery”5.1 Preferred: bring the voters back
Section titled “5.1 Preferred: bring the voters back”Restore enough configured voters and their mesh0 connectivity to regain
quorum before changing membership. Inspect shc cluster status and
shc cluster mesh diagnose from the affected node.
5.2 rqlite force recovery
Section titled “5.2 rqlite force recovery”shc cluster repair --force requests single-node recovery through shcd.
Confirm the other voters are shut down or destroyed before attempting it.
A partition can leave them alive with a committing majority; rewriting
membership on an isolated survivor can then produce two independent
control planes.
Service.ForceLeader in ce modules/cluster/service.go checks the local
raft identity and roster before invoking the recoverer:
| Refusal | Condition |
|---|---|
X409405CLSLDR | Another leader is reachable |
X409403CLSPRT | The liveness check finds other voters that are not provably dead |
X409407CLSRVU | The local identity, roster or liveness signal cannot establish the preconditions |
Service.ForceLeaderPlan runs the same checks in the same order and stops
before the recoverer. It is shc cluster repair --force --dry-run, which
prints the node the recovery would seat and the members it would drop, or
the refusal, and writes nothing.
SHC wires rqliteRecoverer in subsystems/rqlite_engine.go. The
implementation in subsystems/rqlite_recoverer.go requires an existing
raft directory, writes a single-voter raft/peers.json with the engine’s
own identity and raft address, then schedules an engine restart. Recovery
can lose uncommitted writes. A successful single-voter recovery has zero
fault tolerance until membership is rebuilt.