SHC Cluster Architecture
Layer 1. Mesh. Mesh
This document describes the cluster architecture for SHC, enabling secure communication, service discovery, and cross-node integrations.
Architecture Overview
Section titled “Architecture Overview”Each node runs shcd and one container runtime. In a cluster, rqlited carries replicated state over mesh0, while nodes use shc0 for container traffic.
flowchart LR A["shcd and rqlited on node A"] <-->|"mesh0: UDP 4244"| B["shcd and rqlited on node B"] C["Containers on node A"] <-->|"shc0: UDP 4243"| D["Containers on node B"]Runtime Modes
Section titled “Runtime Modes”| Runtime | Execution | Fabric attachment |
|---|---|---|
docker | Docker | shc0 on Linux, each container a routed /32 on a veth of its own |
podman | Podman through the compose CLI | shc0 on Linux, each container a routed /32 on a veth of its own. shc init warns that a podman cluster is single-node only |
incus | incus-compose | shc0 on Linux, each instance a routed /32 on a routed NIC |
apple | Apple’s container CLI on macOS | shc0 on a utun, each replica on a /30 vmnet network of its own |
All four runtimes are linked into shcd. Linux supplies the mesh, fabric and
voter capabilities. macOS supplies mesh, adds fabric once its shc0 key is
published after its pf policy loads (SHC-1623, SHC-1642), and joins as a
non-voter.
Sources: CE daemon/imports.go, core/hostplat and
modules/fabric/types.go.
A Linux cluster node with the shc fabric and service VIPs, or a Mac node
on the apple runtime, runs each replica as its own compose project. Other
nodes run the deployment’s one project. Each replica runs on the node it is
bound to: the leader finishes its project for that node and that node’s
runner starts it (SHC-1426). See
Runtime support and
Traffic across runtimes.
Core Components
Section titled “Core Components”rqlite (Distributed Database)
Section titled “rqlite (Distributed Database)”In cluster mode the daemon supervises an external rqlited process
(core/rqlited): SQLite replicated over Raft:
| Feature | Port | Purpose |
|---|---|---|
| HTTP API | 4001/tcp | Database queries |
| Raft | 4002/tcp | Consensus protocol |
Cluster topology:
- A joiner with the
voterhost capability in the control-plane fence takes a voter slot while the cluster is below its voter target: the largest odd number no greater than the eligible members, capped at five (ce/modules/cluster/voters.go). Every other joiner, including every Mac, enrolls as a non-voter (rqlited -raft-non-voter). Nodes are keyed by the string node id in{state}/node.id. - Leader elected via Raft consensus
- Strong consistency for all writes
Data replicated through the raft engine:
- Deployment, service and container state
- Configuration (cascading scopes)
- Vault secrets (encrypted)
- Audit events
Leadership is derived from the raft leader itself (see below), not from lock rows in the database.
Internal API (mTLS)
Section titled “Internal API (mTLS)”Secure inter-node communication:
| Port | Purpose |
|---|---|
| 4003/tcp | mTLS-authenticated internal API |
| 4004/tcp | Peer status port: mTLS, at the node’s underlay address |
The peer status port answers one read-only function, cluster.peer_status:
whether this node is the raft leader, its raft term, and the era and
generation of the signed membership table it holds. It exists for the one
caller that cannot reach the internal API, a node held off mesh0 (raft and
the internal API both bind mesh0), which asks it of every other node before
shc cluster repair rebuilds its mesh0 cache. The handshake requires a node
certificate under the cluster CA, the function refuses a caller that is not a
node of its signed table, and each source address is rate-limited.
Capabilities:
- Container exec (WebSocket proxy)
- Log streaming (SSE proxy)
- Health checks
- Deployment operations on remote nodes
Traefik (Ingress)
Section titled “Traefik (Ingress)”HTTP routing and TLS termination:
- Runs on nodes that need external access
- Routes based on hostname rules
- Automatic HTTPS via Let’s Encrypt
- Supports HTTP-01 and DNS-01 challenges
Service Discovery
Section titled “Service Discovery”On the shc fabric, shcd answers service names using the resolver’s current routing table. The renderer records provider names for integrations in the deployment’s stored spec; credentials and names come from plugs and sockets. A connection does not grant network permission across tenants.
Sources: CE modules/resolver, modules/app/fabric_render.go and
modules/fabric/policy/doc.go.
Networking
Section titled “Networking”mesh0 carries the node plane. shc0 carries container traffic between nodes.
Every container is a routed /32 from its node’s /26 chunks, and the
nftables reachability rules read its tenant as a label. Each tunnel dials
the peer’s underlay address with its own keys.
See Network fabric for the address plan and policy.
Node Management
Section titled “Node Management”Joining a Cluster
Section titled “Joining a Cluster”# On leader node, generate join tokenshc node token
# On new nodeshc node join <token>The join process:
- The token carries the cluster endpoints, a single-use bootstrap secret and the cluster CA trust bundle.
- The new node calls
POST /api/join/node.enrollthrough Traefik on HTTPS 443 with that token and its locally generated mesh0 public key. - The leader registers the node and returns a signed response containing its mesh0 grant, internal-API certificate and mesh raft join addresses.
- The joiner verifies the response, brings up mesh0 and starts rqlited on its mesh address. The leader grants a voter slot only to an eligible control-plane member; a Mac joins as a non-voter.
Sources: SHC modules/cluster/enroll_rpc.go, enroll_handler.go and
modules/node/commands/local_join.go.
Node Roles
Section titled “Node Roles”| Role | Description | Raft Role |
|---|---|---|
| Leader | Runs background tasks, coordinates operations | Voter |
| Worker | Runs containers, responds to leader | Voter if eligible and granted a slot; otherwise non-voter |
Node Removal
Section titled “Node Removal”shc cluster rm <node-id>Requirements:
- Node must have no running deployments
- Relocate its deployments first by changing where they may run (
shc update --where), or drain it
Event Propagation
Section titled “Event Propagation”Events flow through the SSE broker and internal API:
sequenceDiagram participant D as Docker Daemon participant C as Event Collector participant SSE as SSE Broker participant INT as Internal API participant N2 as Node 2 SSE
D->>C: Container event (start/stop/die) C->>C: Enrich event with node_id C->>SSE: Publish to events:{tenant} channel
Note over INT,N2: If clients on Node 2 need event INT->>N2: Forward via internal API N2->>N2: Publish to local SSELeader Election
Section titled “Leader Election”SHC leadership follows the raft leader; there is no separate lock
table. Each node’s leader_hub subsystem polls the raft engine’s
reported leader id and compares it against the local node id
({state}/node.id); the matching node runs the leader-only work
(ce subsystems/impl/leader_hub.go).
Single-node installs are their own standing leader (no poller runs).
sequenceDiagram participant HUB as leader_hub (each node) participant DB as rqlited
loop Every leader_poll_interval HUB->>DB: current raft leader id? DB-->>HUB: leader id HUB->>HUB: leader id == {state}/node.id → leader end
Note over HUB: On a become/lose-leader edge the hub fans<br/>OnLeader out to every leader-aware subsystemCertificate Management
Section titled “Certificate Management”| Certificate | Storage | Purpose |
|---|---|---|
| Cluster CA | database (encrypted, raft-replicated) | Signs node certificates |
| Node certificates | Filesystem | mTLS authentication |
Automatic rotation:
- Node certificates rotate 30 days before expiry
- Zero-downtime via graceful reload
Network Ports
Section titled “Network Ports”Required Ports
Section titled “Required Ports”| Port | Protocol | Purpose | Exposed To |
|---|---|---|---|
| 4001 | TCP | rqlite HTTP API | Cluster nodes over mesh0 |
| 4002 | TCP | rqlite Raft | Cluster nodes over mesh0 |
| 4003 | TCP | SHC Internal API (mTLS) | Cluster nodes over mesh0 |
| 4004 | TCP | SHC peer status (mTLS) | Node underlay addresses |
| 8000 | TCP | SHC HTTP API, Traefik’s backend | This node, its system ingress, and operator mesh clients over mesh0 |
| 443 | TCP | Traefik HTTPS, including enrolment | Node underlay addresses and public clients |
| 4244 | UDP | mesh0 (node plane) | Node underlay addresses |
A join needs 443 between nodes. A joining node enrols at the bootstrap’s
POST /api/join/node.enroll on 443 before mesh0 exists, so a firewall between
nodes that opens 4001 to 4004 and not 443 forms the first node and times every
join out with X500023JNFAIL. The cloud modules’ inter-node firewall opens
443 with the plane ports.
Optional Ports
Section titled “Optional Ports”| Port | Protocol | Purpose | Exposed To |
|---|---|---|---|
| 80 | TCP | Traefik HTTP | Internet |
| 4243 | UDP | shc0 (required when the fabric is on) | Node underlay addresses |
Node plane (mesh0)
Section titled “Node plane (mesh0)”The node plane is mesh0 on UDP 4244, coordinated by shcd: kernel WireGuard on Linux, or userspace WireGuard on a utun on macOS. Each node holds one /32 from the cluster’s mesh range. Raft, the internal API and VIPs ride mesh0. Container traffic rides the separate shc0 fabric on UDP 4243. See Two WireGuard tunnels.
Security Model
Section titled “Security Model”| Tier | What it reaches |
|---|---|
| Internet | Only reaches Traefik (80/443); mesh0 and shc0 are between nodes |
| Cluster | mTLS-authenticated communication between nodes |
| Local | Docker socket never exposed to network |
Key principles:
- mTLS for inter-node communication - All internal API calls are authenticated
- Raft for consensus - Distributed state with strong consistency (rqlite)
- No external dependencies - Cluster operates without internet access
- Private keys generated locally - Never transmitted
Deployment Targeting
Section titled “Deployment Targeting”A deployment never names a node. Its --where says where it may run:
# Hold a deployment to one nodeshc install postgres --where 'node=hel-02'
# Relocate it by replacing the expression; the update prints its plan firstshc update postgres --where 'node=hel-03'Placement uses node labels and label expressions. --where limits eligible
nodes and --prefer ranks them. shc node selector manages named node selectors;
shc constraint sets constraints at cluster, tenant or deployment scope.
Sources: CE modules/fence/commands (selector.go, constraint.go).
Every install, update and uninstall goes through shcd. An unreachable
daemon refuses the operation with X503738DMNREQ.