shc cluster
The shc cluster commands provide unified management of the service mesh, abstracting the mesh0 WireGuard device and raft engine (rqlited) internals into a simple interface.
Cluster Topology
Section titled “Cluster Topology”graph TB subgraph CLUSTER["SHC Cluster"] subgraph QUORUM["Voting Nodes (Quorum)"] N1["node01<br/>10.42.0.1<br/>Leader + Voter"] N2["node02<br/>10.42.0.2<br/>Voter"] N3["node03<br/>10.42.0.3<br/>Voter"] end
subgraph CLIENTS["Client Nodes"] N4["node04<br/>10.42.0.4<br/>Client"] N5["node05<br/>10.42.0.5<br/>Client"] end end
N1 <-->|"Raft consensus"| N2 N2 <-->|"Raft consensus"| N3 N1 <-->|"Raft consensus"| N3
N1 -.->|"replicate"| N4 N1 -.->|"replicate"| N5 N2 -.->|"gossip"| N4 N3 -.->|"gossip"| N5
classDef leader fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px classDef voter fill:#e3f2fd,stroke:#1565c0,stroke-width:2px classDef client fill:#fff8e1,stroke:#f57f17
class N1 leader class N2,N3 voter class N4,N5 clientNode Roles:
- Leader + Voter: Handles write operations, coordinates consensus
- A voter participates in leader election and Raft consensus. A node with the
voterhost capability in the control-plane fence takes a slot up to the voter target: the largest odd number no greater than the eligible members, capped at five. - A non-voter reads from the cluster and forwards writes to the leader. Every Mac joins as a non-voter.
Node Join Flow
Section titled “Node Join Flow”sequenceDiagram participant NEW as New Node participant LEADER as Leader Node participant RAFT as Raft Engine (rqlited) participant MESH as mesh0 (WireGuard)
Note over NEW,RAFT: shc node join <token>
NEW->>NEW: Decode LAN token (shc-lan-v1.)<br/>(cluster_id, bootstrap_token,<br/>join addresses, CA cert) NEW->>LEADER: POST /api/join/node.enroll on HTTPS 443 through Traefik<br/>(TLS against the token's CA: bootstrap token, node info) LEADER->>LEADER: Validate bootstrap token LEADER->>LEADER: Register node, grant mesh0 under the<br/>joiner's public key, mint<br/>internal-API certificates LEADER-->>NEW: Return node identity, cert material,<br/>raft join addresses, mesh0 grant,<br/>signed with the cluster key the token pins NEW->>MESH: Bring up mesh0 from the grant
NEW->>RAFT: Join raft cluster (voter or non-voter) RAFT-->>NEW: Replicating cluster state Note over NEW,RAFT: Node is now part of the cluster
Note over LEADER,MESH: The first join mints the mesh CA and brings the leader onto the mesh, then converge on the leader pushes every map change to the membersCluster Status
Section titled “Cluster Status”shc cluster status
Section titled “shc cluster status”Show overall cluster health and status:
shc cluster statusOutput (text/tty renders the shared one-row detail table; use the
global -o json/-o yaml for the structured envelope):
CLUSTER_NAME TOTAL_NODES HEALTHY_NODES DEGRADED_NODES UNREACHABLE_NODES VOTERS REQUIRED_VOTERS LEADER_NAME QUORUM_HEALTHYCluster status reports SHC membership and the rqlite quorum. rqlite is the only quorum on the new line.
Node Listing
Section titled “Node Listing”shc cluster nodes
Section titled “shc cluster nodes”List all nodes in the cluster with mesh IPs and roles:
shc cluster nodesOutput:
NAME ID HOSTNAME CLUSTER_STATE IS_LEADERnode01 abc123def456 node01 healthy truenode02 def456abc789 node02 healthy falsenode03 789abc123def node03 healthy falseVerbose mode:
shc cluster nodes -vAdds INTERNAL_ADDRESS, MESH_ADDRESS, RUNTIME, and IS_VOTER columns.
Column selection and structured output come from the global flags
(--fields, -o json, -o yaml).
Connectivity Testing
Section titled “Connectivity Testing”shc cluster ping
Section titled “shc cluster ping”Test mesh0 connectivity. The node is optional: with none named, every node in the cluster is probed.
shc cluster ping # every nodeshc cluster ping node02 # one nodeOutput: one row per node probed, including peers that did not
answer (those carry error instead of rtt_ms):
nodes: - node_id: 3f1c... node_name: node02 status: ok rtt_ms: 0.44 - node_id: 91ab... node_name: node05 status: unreachable error: "no route to host"status is per node. There is no top-level success field, so read each
row rather than the envelope.
Options:
| Flag | Default | Description |
|---|---|---|
-c, --count | 3 | Number of pings to send |
-T, --timeout | 5 | Timeout in seconds |
Examples:
# Ping every node in the clustershc cluster ping
# Ping by node nameshc cluster ping node02
# Ping by mesh IP directlyshc cluster ping 10.42.0.2
# Custom ping count and timeoutshc cluster ping node02 -c 10 -T 2Log Viewing
Section titled “Log Viewing”shc cluster logs
Section titled “shc cluster logs”View cluster service logs (rqlited, the daemon itself):
# View local node logsshc cluster logs
# View logs for a specific nodeshc cluster logs node01Options:
| Flag | Default | Description |
|---|---|---|
-S, --service | all | Unit whose log to read: all, shc, or another unit’s name |
-n, --tail | 50 | Number of lines to show from the end of the logs |
The store runs inside the daemon, so its lines are in the daemon’s log,
-S shc.
Examples:
# View the last 100 lines of the daemon's logshc cluster logs -n 100 -S shcNode Removal
Section titled “Node Removal”shc cluster rm
Section titled “shc cluster rm”Remove a node from the cluster:
shc cluster rm node03Pre-removal checks:
- Node must have no deployments
- Use
--forceto override (not recommended)
Options:
| Flag | Description |
|---|---|
-y, --yes | Skip confirmation prompt |
-f, --force | Force removal even with deployments (dangerous) |
Examples:
# Remove with confirmationshc cluster rm node03
# Skip confirmationshc cluster rm node03 -y
# Force removal (use with caution)shc cluster rm node03 --forceSuccess output:
removed node "node03", mesh IP 10.42.0.3 reclaimedFailure output (has deployments):
success: falseerror: "Node 'node03' has 2 deployment(s)"deployments: - myapp - postgreshint: "Move deployments to another node before removal, or use --force"On removal, SHC automatically:
- Reclaims the mesh IP for reuse
- Removes its mesh0 peer from every remaining node
- Removes node from the raft voter set (if applicable)
Recovery
Section titled “Recovery”shc cluster repair
Section titled “shc cluster repair”Repair cluster state. The single operator entry point for fixing a degraded or wedged cluster.
Default (tidy):
shc cluster repairSweeps confirmed-dead / stale nodes and removes them through the
guarded removal funnel. The non-bypassable quorum-floor guard still
protects the cluster: if removing a dead voter would lose quorum the
action is refused with the hint to escalate to --force.
| Flag | Description |
|---|---|
-v, --verbose | Show detailed repair actions |
--dry-run | Show what would be done without making changes |
--force (destructive quorum-loss recovery):
shc cluster repair --forceUse ONLY when quorum is lost and the other voters are truly dead.
This rewrites raft membership to make the local node the sole leader,
forcing an immediate election. It prompts for confirmation (skip with
--yes) and reuses the shc.cluster.force_leader endpoint.
Data-loss warning.
--forcemay lose uncommitted writes from the missing voters and will split-brain the cluster if those nodes are actually alive. After it succeeds, treat the node as a new single-node cluster and rejoin the others fresh. See Disaster recovery.
| Flag | Description |
|---|---|
--force | Destructive quorum-loss recovery (rewrite raft membership) |
-y, --yes | Skip the confirmation prompt |
Command Summary
Section titled “Command Summary”| Command | Description |
|---|---|
shc cluster status | Show cluster health, leader, quorum |
shc cluster nodes | List nodes with mesh IPs and roles |
shc cluster ping [node] | Test mesh0 connectivity; every node when none is named |
shc cluster logs [node] | Tail cluster service logs |
shc cluster repair | Tidy cluster state; --force for quorum-loss recovery |
shc cluster rm <node> | Remove a node from cluster |
See Also
Section titled “See Also”- Node Commands - Node join, leave, and management
- Mesh Architecture - Service mesh overview