Skip to content
SHC Docs

Consensus and recovery

Layer 2. Core. Core

SHC’s control plane uses rqlited on mesh0. The raft and database listeners bind the node’s mesh address on TCP 4002 and 4001; mesh0 uses WireGuard on UDP 4244 (ce core/nodeports/ports.go). The Linux container fabric, shc0 on UDP 4243, has its own keys and underlay endpoints. See cluster join.

Voters come from eligible members of the control-plane fence. Host capabilities constrain membership: Linux supplies voter; macOS supplies mesh and joins raft as a non-voter.

VoterTarget in ce modules/cluster/voters.go chooses the largest odd number no greater than the eligible member count, capped at five:

Eligible members012345 or more
Target voters011335

Enrolment claims a unique node.voter_slot through GrantVoterSlot up to that target. The suffrage reconciler in modules/cluster/suffrage.go changes one member at a time, removing it from raft and rejoining it with the requested suffrage while retaining its data directory. A down member keeps its vote until removal.

Quorum is floor(configured_voters / 2) + 1. Fault tolerance is reachable_voters - quorum; an unreachable voter still counts in the configured set.

Configured votersReachable votersQuorumFurther losses tolerated
1110
3321
3220
312Quorum lost
5532

A majority permits commits. A minority cannot elect a second committing leader under the same membership. Membership changes also need quorum, so removing dead voters cannot repair a cluster that has already lost it.

Single-writer containers depend on a live node lease. The node agent stops them when it expires; the independent watchdog enforces that stop even if shcd hangs. See the watchdog contract.

The ce cluster service checks both configured and reachable voters before removal. X409201CLSRMQ refuses an insufficient configured set; X409202CLSRMR refuses insufficient reachable survivors. --force does not bypass these quorum guards. Draining workloads does not itself remove a raft member.

Restore enough configured voters and their mesh0 connectivity to regain quorum before changing membership. Inspect shc cluster status and shc cluster mesh diagnose from the affected node.

shc cluster repair --force requests single-node recovery through shcd. Confirm the other voters are shut down or destroyed before attempting it. A partition can leave them alive with a committing majority; rewriting membership on an isolated survivor can then produce two independent control planes.

Service.ForceLeader in ce modules/cluster/service.go checks the local raft identity and roster before invoking the recoverer:

RefusalCondition
X409405CLSLDRAnother leader is reachable
X409403CLSPRTThe liveness check finds other voters that are not provably dead
X409407CLSRVUThe local identity, roster or liveness signal cannot establish the preconditions

Service.ForceLeaderPlan runs the same checks in the same order and stops before the recoverer. It is shc cluster repair --force --dry-run, which prints the node the recovery would seat and the members it would drop, or the refusal, and writes nothing.

SHC wires rqliteRecoverer in subsystems/rqlite_engine.go. The implementation in subsystems/rqlite_recoverer.go requires an existing raft directory, writes a single-voter raft/peers.json with the engine’s own identity and raft address, then schedules an engine restart. Recovery can lose uncommitted writes. A successful single-voter recovery has zero fault tolerance until membership is rebuilt.