Skip to content
SHC Docs

shc node

Generate a join token for adding new nodes to the cluster:

Terminal window
shc node token
# Output: eyJ...base64-encoded-token...

The token includes:

  • Leader address
  • Bootstrap address list
  • CA certificate
  • Encrypted join credentials

--ttl sets how long the token stays valid (default 1h).

Join an existing cluster using a token:

Terminal window
shc node join eyJ...token...

Options:

FlagDescription
--nicNetwork interface for ALL cluster-internal traffic: enroll address, raft (rqlited), internal API bind, mesh0 underlay (auto-detected by default)

NIC selection (auto-detect):

  • Deterministic: candidates sorted by interface name
  • Private (RFC1918) addresses preferred over public
  • Within the preferred class, the default-route interface wins
  • A bridge the node’s container runtime made (docker0, br-…, podman0, incusbr0), SHC’s own devices and a link that carries no signal are passed over while any other interface is up
  • Use --nic to override auto-detection (a nic that fails to resolve aborts the join)
Terminal window
# Auto-detect nic
shc node join eyJ...token...
# Pin the cluster-internal nic explicitly
shc node join eyJ...token... --nic eth1

On join, the node:

  1. Receives cluster CA and node certificate
  2. Joins the raft cluster (as voter or non-voter)
  3. Starts SHC server with mTLS-authenticated internal API

On a mesh-enabled cluster the new node joins the mesh by itself: the node holding the mesh CA signs its certificate and extends every node’s static host map within about 30 seconds of the join.

Gracefully leave the cluster:

Terminal window
shc node leave

This:

  1. Checks and drains workloads
  2. Syncs pending writes
  3. Notifies other nodes
  4. On a mesh cluster, takes mesh0 down; the leader then removes its peer from every other node
  5. Converts back to SQLite

View cluster status:

Terminal window
shc node status

Shows:

  • Cluster mode (single/cluster)
  • Leader node
  • All node states
  • Quorum status

List nodes:

Terminal window
shc node ls
shc node ls -v # Verbose output with extra details
shc node ls --output json

View node details:

Terminal window
shc node info node01

Shows:

  • Node ID and address
  • Role (leader/voter/non-voter)
  • Health status
  • Resource usage (CPU, memory, disk)

Check the health of a specific node:

Terminal window
shc node health node01

Shows:

  • Node connectivity status
  • Resource usage
  • Service health
Terminal window
shc node drain <node> [--force]
shc node drain <node> --plan

Drain stops new placement on a node and relocates its workloads. It asks the placement engine where each deployment goes, with the drained node excluded: the target matches the deployment’s --where and the constraints over it, and the scorer ranks the peers that are left. A volume’s current reach does not bind the target, because the relocation carries the volume (a backup and restore when it cannot reattach there). A store’s reach does bind.

Deployments move one at a time, highest scheduler.priority first and the platform’s own stacks in their recovery order. A global replica is never moved: it stops last, once everything else has left the node.

A deployment whose --where, or a constraint over it, holds it to the source node with a node= term is pinned there (ce modules/fence/node_fence.go, NodeFence): drain refuses before changing state (X400401CLSDRP). A row the engine cannot place (its --where empties its eligible set, or no peer is ready) is a failed row that names why; --force stops it in place instead.

Two gates stop the run before it changes anything. A deployment with a stored group of two or more members is highly available. Drain protects its last serving copy (X400402CLSDRB). A deployment whose deploy.disruption is never blocks the drain, which names it (X400423CLSDRM), and a setting that could not be read refuses too (X503425CLSDBU). A rolling deployment, the default, moves one replica of a service at a time, each ready on its new node before the next is touched. --force explicitly accepts the outage. Its journal records committed and unresolved moves so retries do not repeat a cutover, and partial results retain their per-stack detail.

--plan asks the same question and changes nothing. It prints one row per stack: STACK, ACTION (move, stop, stay or blocked), TARGET (the node it would go to) and BLOCKER (why it can’t move, in the words a drain would refuse with). When a gate would refuse the whole drain, the summary says so and the refusal follows the rows. It exits 0 whether or not a row is blocked. Each stack is placed against the cluster as it is now, while a drain reads it again after every move, and the move’s own admission isn’t asked, so a planned move can still be refused for cpu, memory or start room.

Terminal window
shc node undrain <node>

Restores SHC availability to active and closes the open drain attempt. It does not move workloads back or restart deployments stopped by a forced drain.

Remove an unreachable node:

Terminal window
shc node forget node01

Use when a node cannot gracefully leave.

A removal that would break raft quorum refuses with X409201CLSRMQ or X409202CLSRMR, including when --force is set. Removing the final node tears down the cluster.

Initialize a new SHC cluster on the first node:

Terminal window
shc init

Options:

FlagDescription
--nicUnderlay network interface (auto-detected by default); raft uses mesh0

NIC selection (auto-detect):

  • Deterministic: candidates sorted by interface name
  • Private (RFC1918) addresses preferred over public
  • Within the preferred class, the default-route interface wins
  • A bridge the node’s container runtime made (docker0, br-…, podman0, incusbr0), SHC’s own devices and a link that carries no signal are passed over while any other interface is up
  • Use --nic to override auto-detection
Terminal window
# Auto-detect nic
shc init
# Pin the cluster-internal nic explicitly (e.g., for dual-homed servers)
shc init --nic eth1

On init, SHC:

  1. Generates cluster CA and node certificate
  2. Initializes the state database
  3. Starts SHC server with internal API
  4. Installs system tenant apps (Traefik, OpenObserve)