Skip to content
SHC Docs

SHC Network Fabric

Layer 9. Fabric. Fabric

The shc fabric carries container traffic between nodes. shcd owns its address allocation and reconciles shc0 with the cluster store. Every container is one routed /32, its pin, on a link of its own to the host. No two containers share a link, and no tenant has a bridge.

Sources: CE modules/fabric/types.go, posture.go, chunk.go, pins.go and subsystems/impl/fabric_reconcile.go.

The shc fabric is the workload network. shc init with no --fabric sets it up wherever the node’s runtime can attach it: docker, podman and incus on Linux, and apple on a Mac. --fabric none is refused there (X400389FABOFF). Where the runtime cannot attach it, docker or podman on a Mac, the cluster is formed without a fabric. Init says so in its output and shc doctor reports it under fabric.posture. --fabric shc on such a node is refused (X400395FABATT). --fabric swarm is refused (ce modules/fabric/types.go, X400371FABSWM). --runtime swarm is refused (ce modules/cluster/service.go, X400422CLSRUN).

--address-block selects the IPv4 block SHC carves its own ranges from, chosen at genesis. It is not tied to the fabric mode: mesh addresses and service VIPs come out of it too, so it is recorded whether the fabric is shc or none. Unset is 10.64.0.0/16. A block is a /8 at the largest and a /16 at the smallest, written as its own network address, and refused (X400349ADRBLK) when it overlaps the docker address pool 172.16.0.0/12.

core/addrblock carves the genesis block by fixed fractions, whatever its size. The first eighth is the operator’s own: SHC never addresses it and never checks a network in it. Service VIPs take the third sixteenth and mesh addresses the fourth. Deployment networks take the second quarter, a /25 each: a docker, podman or apple node carves the whole quarter for its own bridges, and the members of an Incus cluster carve it together, because an Incus bridge is every member’s. On Incus a network holds one /25 on every member it runs on, and keeps it through a move, and once the quarter is full the members take /25s from the blocks the cluster added. The fabric takes the second half, which nodes take in /26 chunks. On the default block that is these ranges:

  • 10.64.0.0/19 the operator’s own networks
  • 10.64.32.0/20 service VIPs
  • 10.64.48.0/20 mesh addresses
  • 10.64.64.0/18 deployment networks
  • 10.64.128.0/17 the fabric

A site picks a block whose four ranges are clear of every network any node is on, VPNs included. shc init and shc node join refuse a node with a network or route inside one of them (X400450ADROVL at init, X409450ADROVL from the leader at join), naming the interface and the range. A site that keeps its own LANs low in a /16 can lay the block over them: with its LANs inside 10.236.0.0/19, --address-block 10.236.0.0/16 leaves them in the operator’s eighth and gives service VIPs 10.236.32.0/20, mesh addresses 10.236.48.0/20, deployment networks 10.236.64.0/18 and the fabric 10.236.128.0/17. Do not use 100.64.0.0/10: carrier-grade NAT and VPNs live there. shc cluster fabric status prints the block in its own cell and the four ranges as rows under it; -o json carries them as block and ranges.

The genesis block keeps its carve for the life of the cluster. A /16 block’s fabric half, a /17, holds 512 chunks of 64 addresses, and the first is never handed out. Every node holds at least one, so the default block has room for 511 nodes. A /14 block’s /15 fabric holds 2,048 chunks, and a /8 block’s /9 holds 131,072. A cluster that needs more room adds a block: shc cluster block add <cidr> records one more IPv4 block, a /8 to a /26, as one pool of /26 chunks. A /24 added block is four chunks. An added block is not carved. Node fabric chunks, service VIPs and mesh addresses are drawn from it once the genesis ranges are full. The add is refused when the block shares an address with the genesis block, a block already added, the docker pool, or a network or route on any node, and while any node cannot be asked for its networks. A block is never removed or shrunk. shc cluster block list prints the genesis block, then every added block with how many chunks it holds, how many are held and how many are left. Sources: CE core/addrblock/addrblock.go, pool.go, modules/fabric/blocks.go and modules/cluster/block.go.

A docker or podman node’s engine network shc_fabric holds every fabric range as a subnet, the genesis fabric and each added block, so the engine takes a pin of any of them. Neither engine adds a subnet to a network that exists, so an added block reaches a node as shc_fabric made again, one node at a time. A node whose network lacks a range takes the cluster’s one turn, a store row that lapses after ten minutes if its holder goes quiet. It stops every container on the network and every container in one’s network namespace, takes each off, makes the network again over every range, puts each back at its pin with the gateway priority and aliases it had, starts what was running, and gives the turn up once those run and pass their health checks. A journal under the state home names what it moves before anything stops, so a daemon that ends part way finishes the move at its next sweep. A node that is down when the block is added takes its turn when its daemon is back. shc cluster block add prints each node’s progress and returns once every node holds the block; shc cluster block list and the doctor check fabric.network-ranges say where each node stands. Incus attaches a pin as a routed NIC and apple gives each pin a network of its own, so neither holds a range and neither takes a turn. ip shc_fabric_nat covers every range; no rule of SHC’s in iptables names one (SHC-3322). Sources: CE modules/fabric/network.go, network_roll.go, network_recreate.go and glue_linux.go.

A node takes /26 chunks of the fabric as it needs them: one when it enrols, and one more each time the chunks it holds are full. Its first chunk’s .1 is the node’s fabric address, which shc0 holds as a /32. Every other address of its chunks except each chunk’s first and last can be a pin: 62 of a chunk’s 64 addresses, 61 of the first.

A pin is a /32 from its node’s chunks, whatever its tenant. On docker and podman the container joins the node’s engine network shc_fabric, whose driver shc-fabric is SHC’s own: docker’s remote network and IPAM driver, which shcd serves, and the netavark plugin podman runs. It is no bridge. On incus the executor turns the attachment into a routed NIC with no parent. Each way the container’s end carries the pin as a /32, with a route to the node’s fabric address on the link and the default route through it. The host’s end is a veth of its own, named shr and the pin’s four bytes in hex (10.84.7.130 is shr0a540782). It carries the node’s fabric address and holds the host’s route to the pin. No bridge sits anywhere on the path, and the tenant holds nothing on the node: it is a label the node’s policy reads from the pin ledger.

On a Mac shc0 is the daemon’s userspace WireGuard device on a utun, and the Mac publishes its key only once its pf policy anchor com.apple/shc/policy is loaded. Apple takes no address it is given, so each pinned replica runs on a vmnet network of its own, a /30 of the node’s chunks, 14 to a chunk once the /30s holding its first and last address are left out. Its pin is the one address vmnet hands out there, and the host’s bridge is the network’s other end. One network per replica keeps every container on a link of its own with the host, as a veth does on Linux.

Pins are rows of the store’s pin ledger, fabric_pin. A service keeps its pin across renders on the same node, and two allocators racing for one address collapse on the ledger’s unique node and address. A released pin’s address stays out of allocation for 10 minutes, and every node refuses traffic to it for the first five, so a client that still holds it fails at once. A chunk nothing sits in any more goes back. The chunk holding the node’s address stays, and so does one taken in the last 10 minutes, which its allocator is about to fill. A given-back chunk, or the chunks of a removed node, go to another node only after cluster.address_quarantine (1 h) and once every live node has said its shc0 config no longer names them.

Addresses come in per-node /26 chunks of one shared fabric range because shc0 routes each peer’s chunks. A peer’s WireGuard AllowedIPs are the chunks it holds, so the peer state on every node is a few stable /26s that change only when a node takes or gives back a chunk, never one route per container rewritten across the cluster. A new container is one pin drawn from its own node’s chunks and changes no peer’s configuration. WireGuard also drops every packet whose source is outside the sending peer’s AllowedIPs, so a node can send only from its own chunks.

A chunk is a /26 so that a block’s fabric is cut fine enough for many nodes. At /24 the default block’s fabric held 128 chunks, and a cluster stopped near 128 nodes however few containers each ran. At /26 it holds 512. A node that runs more than 61 containers takes another chunk, which adds one entry to each peer’s AllowedIPs.

A container’s address is a /32 on a routed NIC so that nothing is switched. No two containers share a link, so every packet a pin sends or receives passes the node’s policy on that pin’s own veth, a co-located pair included. No L2 is shared, so nothing reaches another container except as a routed packet the policy has judged.

The trade is that an address belongs to its node. A container that moves to another node gets a new pin from the new node’s chunks, and its old address goes into quarantine. The stable handle for a service is its VIP and its DNS name. The node’s DNS answers the new address from the next revision, the VIP dispatches to whichever replica pins are ready, and no connected deployment re-renders.

shc0 listens on UDP 4243. It routes the fabric range over the device, and every block the cluster added beside it, and WireGuard’s AllowedIPs pick the peer. It dials every peer at its underlay address, never inside mesh0. It starts at MTU 1280, the least an IPv6 link carries and the least an Incus NIC takes, and every pin’s veth or routed NIC is made at shc0’s MTU. A pair of nodes runs a larger per-route MTU only while both ends’ readiness rows are fresh and a probe has proven the path, bounded by each end’s link MTU and the proven path MTU, less WireGuard’s 80. The floor needs an underlay link of 1360: shc init and shc node join refuse a node whose link carries less (X400390FABMTU).

flowchart LR
A["Container on node A, its pin a /32"] --> B["Its own veth on A"]
B --> C["shc0 on A"]
C -->|"WireGuard over underlay: UDP 4243"| D["shc0 on B"]
D --> E["Its own veth on B"]
E --> F["Container on node B, its pin a /32"]

The nft table ip shc_fabric_nat keeps a fabric-to-fabric flow’s real source. Its chain hooks postrouting just ahead of the engines’ masquerades and maps the flow to its own source, so the receiving node sees the container’s pin. The same chain masquerades a pin’s flow to anything outside the fabric: a pin’s veth is no engine’s bridge, so no engine’s masquerade covers it. One fabric-to-fabric flow is masqueraded anyway: a replica that dials its own service VIP. Once DNATed it runs from its pin to the same pin, so the postrouting_hairpin chain of ip shc_vip hooks ahead of the exemption and masquerades it to the node’s address on the replica’s interface. Sources: CE modules/fabric/wireguard.go, endpoints.go, natexempt.go and modules/vip/compile/compile.go.

SHC writes no rule into iptables on any engine. A pin is a routed /32 on a veth of its own, so dockerd’s chains, its masquerade and docker 28’s raw drops, all keyed to a docker bridge or its subnet, never match one, and the identity mapping in ip shc_fabric_nat decides a fabric flow’s NAT before any engine’s masquerade runs. What is left is dockerd’s ip filter FORWARD policy drop, which takes every packet dockerd did not accept, a pin’s among them. That is node setup, not a rule SHC fights: shc init and shc node join put "ip-forward-no-drop": true in /etc/docker/daemon.json (Docker 28 and newer), checked with dockerd --validate and put back as it was if dockerd rejects it. The key acts at dockerd’s start, so a drop dockerd set before it stays until the next boot, or until an operator runs iptables -P FORWARD ACCEPT. Podman needs nothing in iptables with either netavark firewall driver: netavark never sets a FORWARD policy, and its rules, iptables or inet netavark, name its own bridges and subnets, never a pin’s veth. A daemon upgraded from v0.1.16 takes out what it kept in iptables (SHC-FABRIC, its FORWARD jump, the nat and raw head rules, SHC-MESH-CLIENT and its masquerade) on its first fabric pass, and only those. SHC’s own rules live in its own nft tables, the operator-client verdict in ip shc_mesh_client among them. A program that drops a bridge’s own traffic at the forward hook (dockerd before setup, a dockerd beside Incus, ufw, firewalld) is not patched around. The doctor check fabric.forward-path probes the path with two throwaway endpoints on a throwaway bridge, on every engine, and names the program and its fix, and a job that times out reaching its peer carries the same reading. A node runs one container engine. Sources: CE modules/fabric/glue_linux.go, iptables_release_linux.go, meshclient_nft.go, forward_path_doctor.go, core/fwdhook and core/hostactivate/fabric_host_setup.go.

The host’s networking is shared with every other program on it, so SHC sets its part up once and reports what changes afterwards instead of putting it back (SHC-3323). Every host setting the fabric sweep or the VIP pass used to re-assert, and where each lives now:

SettingKindWhy
net.bridge.bridge-nf-call-iptables = 1, with br_netfilter loadedsetupthe host’s, shared with every engine; without it a frame one bridge switches between two of its ports never meets SHC’s policy table
net.ipv4.conf.all.accept_source_route = 0setupthe host’s; with it on, a destination read at the forward hook is a packet’s first hop, not where it goes
net.ipv4.ip_forward = 1setupthe host’s; every pin flow to another node, a relay and a mesh gateway are forwarded. It was re-asserted by the relay and the mesh-client glue
dockerd’s "ip-forward-no-drop": truesetupdockerd’s own configuration; without it dockerd drops forwarded traffic it did not make when it turns forwarding on
SHC-FABRIC, its FORWARD jump, the nat RETURN, the raw ACCEPT, the pins’ egress and relay acceptsgonethey lived in docker’s tables and were hoisted back to the head every sweep and on every docker network event; nothing needs them on a set-up node
SHC-MESH-CLIENT and its masquerade in iptablesgonethe same verdict lives in ip shc_mesh_client
ip shc_fabric_nat, ip shc_mesh_client, ip shc_fabric_loopback, the crossing guard and SHC’s policy tablessweepSHC’s own tables, no other program writes them; each follows the cluster’s fabric ranges, blocks, pins and tenants, which change while the node runs
route_localnet on shc0 and each pin’s vethsweepSHC’s own devices; a pin’s veth exists only while its container runs
the node’s fabric network on its enginesweepSHC’s own network; it is remade when the cluster adds a block

shc init and shc node join write the kernel settings to /etc/sysctl.d/90-shc-fabric.conf and br_netfilter to /etc/modules-load.d/shc-fabric.conf, and apply them to the running kernel. The daemon applies the kernel settings once when it starts, which carries a node set up by an older release, and never again in that process. No watcher runs on a docker or podman node’s networks. When an operator or another program changes a setting, it stays as they left it, and the doctor check fabric.host-setup fails naming the setting, the value it found, the value setup put there and the command that puts it back. It warns when a drop-in is missing or differs, and on a docker node when daemon.json does not set ip-forward-no-drop. Sources: CE core/hostnet, core/hostactivate/fabric_host_setup.go, modules/fabric/host_setup_linux.go and host_setup_doctor.go.

Apps declare services in compose. SHC translates or refuses deploy: keys according to Compose keys.

Rendered elementFabric behaviour
Fabric networkEvery pinned service joins shc_fabric as an external network, with its pin as ipv4_address and its default route there (gw_priority)
Authored networksDeployment networks: a /25 of the deployment range each, at MTU 1500, recorded in networks.json. Node-local on docker, podman and apple; one /25 for every member on Incus
Service addressesPins from the fabric ledger
Provider namesRecorded in the deployment’s stored spec; every node’s DNS answers them with the provider’s live addresses
Ingress upstreamService VIP for replica projects; ready fabric pin for a deployment project

The render joins no consumer to a provider’s own network: a same-tenant provider is reached at its pin, and Traefik dials a routed service on its pin. The renderer writes no extra_hosts entries for provider discovery. The node’s DNS answers from live state. Sources: CE modules/app/fabric_render.go, modules/depnet/depnet.go, modules/resolver/source.go, modules/app/ingress_fabric_upstream.go and ingress_vip_upstream.go.

Enrolment gives a node its fabric address and first chunk only when it can attach the fabric. shcd reconciles the device and peers from membership. mesh0 enrolment and raft startup are described in Cluster join.

Install and update draw pins and render fabric attachments. Node removal tombstones the node’s row, releases every live pin on it and gives its chunks back; every peer drops them from shc0 on its next reconcile and says so. Sources: CE modules/fabric/repo.go, chunk.go, pins.go and modules/app/fabric_render.go.

The compiler builds per-node nftables rules from deployments and their declared ports. The deployment is the policy unit, and a pin’s tenant is the label it is filed under.

TrafficRule
Within a deploymentAll ports allowed
Within a tenantOther deployments reach a service only on its declared container ports
Across tenantsBlocked except for the derived system roles
System ingressReaches routed backends on their route ports, and the service an armed tcp_ingress binding names on its declared port
To the system ingressEvery tenant’s containers reach its pins on the ports it publishes, keeping their own address, unless a published port’s listener took the connection
System collector and shared providerReachable on their declared ports
Other fabric trafficDropped

A plug supplies connection details; it does not grant permission. A cross-tenant plug is refused with X400372FABXTN. A service with no ports declared by ports: or expose: is private to its deployment. Published ports have separate listener rules, requested by --expose.

The compiler checks same-node traffic as well as traffic through shc0. Two pins on one node are two veths, so their traffic is routed and meets the same forward hook. fabric_sources binds each pin to its own veth and the node’s chunks to its pins’ veths, so a packet that does not wear the pin of the veth it entered on is dropped, whatever the container can do inside its own namespace. fabric_entry drops a packet entering a pin’s veth from any local bridge: a deployment’s own network, docker0, a podman or incus network. A node whose uplink is itself a Linux bridge (br0, vmbr0, a bridge over a VLAN) names it to the compiler, so those checks read it as the network the node faces and not as a container’s bridge. fabric_outside takes a packet for the node’s chunks from a source outside them that did not enter on shc0: a host on the uplink, a VPN or a local bridge. It passes the replies of accepted flows, an ICMP error about a flow a pin here opened, a listener’s flow to a published port and an incus node’s collector export, and drops the rest, so a host that routes the pin range through a node reaches no pin (SHC-3484). Linux’s inet shc_node_plane table keeps containers off control listeners, with explicit allowances such as the node DNS resolver. These controls do not make a node a boundary for hostile tenants; see Security.

Sources: CE modules/fabric/policy/doc.go, policy/compile.go and policy/hostplane.go.

mesh0 carries raft and the internal API on node mesh addresses. shc0 carries container traffic. Both tunnels dial the peer’s underlay address with separate WireGuard keys and UDP ports, so neither tunnel runs inside the other. On Linux both are kernel devices. A Mac runs both as the daemon’s userspace WireGuard devices on utuns and joins raft as a non-voter. See Two WireGuard tunnels, sections 2 and 13.

Each device’s key rotates the same way, by shc cluster mesh rotate [<node>][:<device>] and on the leader’s schedule (cluster.key_rotation, 30 days): every peer installs the pending key beside the old one, the node switches, and a peer moves the node’s addresses on the first handshake under the new key. shc0’s key is on the node’s fabric grant; a Mac’s is not rotated. The policy’s forward chain clamps the MSS of every SYN in or out of shc0 to rt mtu, so a container’s TCP fits the peer’s route without an ICMP reaching it.

An apple node holds its containers off the node plane with the pf child anchor com.apple/shc/node-plane, compiled from the same HostInput as the Linux nft table inet shc_node_plane, and loaded by shcd while the Mac is a member. Its policy anchor com.apple/shc/policy holds each replica network’s bridge to its one address, in place of fabric_sources. The design is in Security, mesh networking.