Skip to content
SHC Docs

shc cluster

The shc cluster commands provide unified management of the service mesh, abstracting the mesh0 WireGuard device and raft engine (rqlited) internals into a simple interface.

graph TB
subgraph CLUSTER["SHC Cluster"]
subgraph QUORUM["Voting Nodes (Quorum)"]
N1["node01<br/>10.42.0.1<br/>Leader + Voter"]
N2["node02<br/>10.42.0.2<br/>Voter"]
N3["node03<br/>10.42.0.3<br/>Voter"]
end
subgraph CLIENTS["Client Nodes"]
N4["node04<br/>10.42.0.4<br/>Client"]
N5["node05<br/>10.42.0.5<br/>Client"]
end
end
N1 <-->|"Raft consensus"| N2
N2 <-->|"Raft consensus"| N3
N1 <-->|"Raft consensus"| N3
N1 -.->|"replicate"| N4
N1 -.->|"replicate"| N5
N2 -.->|"gossip"| N4
N3 -.->|"gossip"| N5
classDef leader fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px
classDef voter fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
classDef client fill:#fff8e1,stroke:#f57f17
class N1 leader
class N2,N3 voter
class N4,N5 client

Node Roles:

  • Leader + Voter: Handles write operations, coordinates consensus
  • A voter participates in leader election and Raft consensus. A node with the voter host capability in the control-plane fence takes a slot up to the voter target: the largest odd number no greater than the eligible members, capped at five.
  • A non-voter reads from the cluster and forwards writes to the leader. Every Mac joins as a non-voter.
sequenceDiagram
participant NEW as New Node
participant LEADER as Leader Node
participant RAFT as Raft Engine (rqlited)
participant MESH as mesh0 (WireGuard)
Note over NEW,RAFT: shc node join <token>
NEW->>NEW: Decode LAN token (shc-lan-v1.)<br/>(cluster_id, bootstrap_token,<br/>join addresses, CA cert)
NEW->>LEADER: POST /api/join/node.enroll on HTTPS 443 through Traefik<br/>(TLS against the token's CA: bootstrap token, node info)
LEADER->>LEADER: Validate bootstrap token
LEADER->>LEADER: Register node, grant mesh0 under the<br/>joiner's public key, mint<br/>internal-API certificates
LEADER-->>NEW: Return node identity, cert material,<br/>raft join addresses, mesh0 grant,<br/>signed with the cluster key the token pins
NEW->>MESH: Bring up mesh0 from the grant
NEW->>RAFT: Join raft cluster (voter or non-voter)
RAFT-->>NEW: Replicating cluster state
Note over NEW,RAFT: Node is now part of the cluster
Note over LEADER,MESH: The first join mints the mesh CA and brings the leader onto the mesh, then converge on the leader pushes every map change to the members

Show overall cluster health and status:

Terminal window
shc cluster status

Output (text/tty renders the shared one-row detail table; use the global -o json/-o yaml for the structured envelope):

CLUSTER_NAME TOTAL_NODES HEALTHY_NODES DEGRADED_NODES UNREACHABLE_NODES VOTERS REQUIRED_VOTERS LEADER_NAME QUORUM_HEALTHY

Cluster status reports SHC membership and the rqlite quorum. rqlite is the only quorum on the new line.

List all nodes in the cluster with mesh IPs and roles:

Terminal window
shc cluster nodes

Output:

NAME ID HOSTNAME CLUSTER_STATE IS_LEADER
node01 abc123def456 node01 healthy true
node02 def456abc789 node02 healthy false
node03 789abc123def node03 healthy false

Verbose mode:

Terminal window
shc cluster nodes -v

Adds INTERNAL_ADDRESS, MESH_ADDRESS, RUNTIME, and IS_VOTER columns. Column selection and structured output come from the global flags (--fields, -o json, -o yaml).

Test mesh0 connectivity. The node is optional: with none named, every node in the cluster is probed.

Terminal window
shc cluster ping # every node
shc cluster ping node02 # one node

Output: one row per node probed, including peers that did not answer (those carry error instead of rtt_ms):

nodes:
- node_id: 3f1c...
node_name: node02
status: ok
rtt_ms: 0.44
- node_id: 91ab...
node_name: node05
status: unreachable
error: "no route to host"

status is per node. There is no top-level success field, so read each row rather than the envelope.

Options:

FlagDefaultDescription
-c, --count3Number of pings to send
-T, --timeout5Timeout in seconds

Examples:

Terminal window
# Ping every node in the cluster
shc cluster ping
# Ping by node name
shc cluster ping node02
# Ping by mesh IP directly
shc cluster ping 10.42.0.2
# Custom ping count and timeout
shc cluster ping node02 -c 10 -T 2

View cluster service logs (rqlited, the daemon itself):

Terminal window
# View local node logs
shc cluster logs
# View logs for a specific node
shc cluster logs node01

Options:

FlagDefaultDescription
-S, --serviceallUnit whose log to read: all, shc, or another unit’s name
-n, --tail50Number of lines to show from the end of the logs

The store runs inside the daemon, so its lines are in the daemon’s log, -S shc.

Examples:

Terminal window
# View the last 100 lines of the daemon's log
shc cluster logs -n 100 -S shc

Remove a node from the cluster:

Terminal window
shc cluster rm node03

Pre-removal checks:

  • Node must have no deployments
  • Use --force to override (not recommended)

Options:

FlagDescription
-y, --yesSkip confirmation prompt
-f, --forceForce removal even with deployments (dangerous)

Examples:

Terminal window
# Remove with confirmation
shc cluster rm node03
# Skip confirmation
shc cluster rm node03 -y
# Force removal (use with caution)
shc cluster rm node03 --force

Success output:

removed node "node03", mesh IP 10.42.0.3 reclaimed

Failure output (has deployments):

success: false
error: "Node 'node03' has 2 deployment(s)"
deployments:
- myapp
- postgres
hint: "Move deployments to another node before removal, or use --force"

On removal, SHC automatically:

  1. Reclaims the mesh IP for reuse
  2. Removes its mesh0 peer from every remaining node
  3. Removes node from the raft voter set (if applicable)

Repair cluster state. The single operator entry point for fixing a degraded or wedged cluster.

Default (tidy):

Terminal window
shc cluster repair

Sweeps confirmed-dead / stale nodes and removes them through the guarded removal funnel. The non-bypassable quorum-floor guard still protects the cluster: if removing a dead voter would lose quorum the action is refused with the hint to escalate to --force.

FlagDescription
-v, --verboseShow detailed repair actions
--dry-runShow what would be done without making changes

--force (destructive quorum-loss recovery):

Terminal window
shc cluster repair --force

Use ONLY when quorum is lost and the other voters are truly dead. This rewrites raft membership to make the local node the sole leader, forcing an immediate election. It prompts for confirmation (skip with --yes) and reuses the shc.cluster.force_leader endpoint.

Data-loss warning. --force may lose uncommitted writes from the missing voters and will split-brain the cluster if those nodes are actually alive. After it succeeds, treat the node as a new single-node cluster and rejoin the others fresh. See Disaster recovery.

FlagDescription
--forceDestructive quorum-loss recovery (rewrite raft membership)
-y, --yesSkip the confirmation prompt
CommandDescription
shc cluster statusShow cluster health, leader, quorum
shc cluster nodesList nodes with mesh IPs and roles
shc cluster ping [node]Test mesh0 connectivity; every node when none is named
shc cluster logs [node]Tail cluster service logs
shc cluster repairTidy cluster state; --force for quorum-loss recovery
shc cluster rm <node>Remove a node from cluster