Skip to content
SHC Docs

The mesh under it all

Two WireGuard tunnels run between every pair of nodes. mesh0 is the node plane: the store’s replication, the internal API and the node DNS travel on it. shc0 is the fabric from the last page, where containers travel. Both dial the peer’s underlay address, each with its own key and UDP port, so neither runs inside the other. shc cluster mesh status is the node plane as this node sees it, one row per node with its mesh address and whether its tunnel is installed and running:

home01
$ shc cluster mesh status
mesh: up
NODE NAME MESH ADDRESS IS LOCAL INSTALLED RUNNING NODE ID
...

shc cluster mesh typed with no subcommand prints the same status:

explained, not walked: it prints what shc cluster mesh status printed above
$ shc cluster mesh

Each node’s mesh address is in the mesh range, 10.192.0.0/11 on the default address block, handed out in join order: home01 is 10.192.0.1, far01 10.192.0.2. shc node ls -v shows it in the INTERNAL column, with the port the internal API answers on, beside the underlay address the tunnel dials:

home01
$ shc node ls -v
NAME ID HOST INTERNAL UNDERLAY RUNTIME CORES MEMORY FREE CORES FREE MEMORY OUTSIDE RESERVED STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD LEADER VOTER
far01 node-fc10df0b46b8 far01 10.192.0.2:4003 65.21.55.73 docker 2 7.6G 2.0 7.6G - - healthy ready active learn learn/far ok 0.1.2 no no
home01 node-666d6f4dc640 home01 10.192.0.1:4003 167.233.221.15 docker 2 7.6G 2.0 7.6G - - healthy ready active learn learn/home ok 0.1.2 yes yes

The devices on the node, and the ports they listen on:

home01
$ ip -4 addr show mesh0
7: mesh0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
inet 10.192.0.1/11 brd 10.223.255.255 scope global mesh0
valid_lft forever preferred_lft forever
$ ip -4 addr show shc0
...
inet 10.128.16.1/32 scope global shc0
...
$ ss -lun
State Recv-Q Send-Q Local Address:Port Peer Address:Port
UNCONN 0 0 10.192.0.1:53 0.0.0.0:*
...
UNCONN 0 0 0.0.0.0:4243 0.0.0.0:*
UNCONN 0 0 0.0.0.0:4244 0.0.0.0:*
...

4244 is mesh0, 4243 is shc0, and port 53 on the mesh address is the node DNS. shc cluster mesh diagnose has this node probe every peer over the mesh and report the round trip:

home01
$ shc cluster mesh diagnose
FROM MESH ADDRESS TUN PEER PEER IP REACHABLE RTT MS
...

The node DNS answers the cluster’s private names: every node under nodes.shc.internal, every service under its tenant. Ask far01’s daemon for home01:

far01
$ dig +short @10.192.0.2 home01.nodes.shc.internal
10.192.0.1

A node’s mesh0 and shc0 keys can be rotated while the cluster runs. Each rotation is prepared on its node, every peer installs the new key beside the old one, and then the node switches; shc cluster mesh status shows it until it settles. With nothing named, this node’s mesh0 key rotates and then its shc0 key.

home01
$ shc cluster mesh rotate
✔ rotation active: node-666d6f4dc640 mesh0 epoch 1
✔ rotation active: node-666d6f4dc640 shc0 epoch 1

A node’s name rotates that node’s keys from here, and a device after a colon rotates that one alone:

home01
$ shc cluster mesh rotate far01:shc0
✔ rotation active: node-fc10df0b46b8 shc0 epoch 1

The leader also rotates every key once it is 30 days old, one key an hour, and cluster.key_rotation sets the age, or turns the schedule off:

explained, not walked: the schedule acts over days
$ shc config set cluster.key_rotation 7d

Run the status command again until the rotation line is gone, then probe the peers once more.

home01
$ shc cluster mesh status
mesh: up
NODE NAME MESH ADDRESS IS LOCAL INSTALLED RUNNING NODE ID
...
$ shc cluster mesh diagnose
FROM MESH ADDRESS TUN PEER PEER IP REACHABLE RTT MS
...

Every peer reaches home01 on its new key. shc cluster mesh client enrols an operator’s Mac on the mesh the same way. The cluster lists the clients it has:

home01
$ shc cluster mesh client ls
no clients in tenant "prod"

The rest runs on the Mac: enrol it under a name, bring its mesh0 up, read or move the gateway it dials, and revoke it when it goes:

explained, not walked: these run on an operator's Mac, and the rig has only Linux nodes
$ shc cluster mesh client enrol --name laptop
$ shc cluster mesh client up
$ shc cluster mesh client peers --name laptop --current home01
$ shc cluster mesh client revoke laptop --yes

The cluster keeps everything this ladder installed: web with two replicas, hello, site and www with their routes and records, dns and the connection.

Next: Your own ingress.