The address block
Every address SHC hands out comes from the genesis block, chosen with
shc init --address-block <cidr> and carved for the life of the cluster,
or from a block the cluster adds later.
Unset is 10.64.0.0/16. A block is a /8 at the largest and a /16 at the
smallest, written as its own network address. It must stay clear of the
docker address pool 172.16.0.0/12 (X400349ADRBLK).
The four ranges
Section titled “The four ranges”SHC carves the genesis block by fixed fractions, whatever its size. The first eighth is left to the operator’s own networks: SHC never addresses it and never checks a network in it. Service VIPs take the third sixteenth and mesh addresses the fourth. Deployment networks take the second quarter and the fabric the second half. On the default block that gives:
| Range | Used for |
|---|---|
10.64.0.0/19 | nothing: the operator’s own networks |
10.64.32.0/20 | service VIPs |
10.64.48.0/20 | mesh addresses (mesh0) |
10.64.64.0/18 | deployment networks (a /25 each) |
10.64.128.0/17 | the fabric (shc0, /26 chunks per node) |
The deployment-network quarter is one budget per bridge owner. A docker,
podman or apple node’s bridges are its own, so each such node carves the
whole quarter and the default /16 holds 128 deployment networks on every
one of them. An Incus bridge is every member’s, so the members of an Incus
cluster carve the quarter together: the default /16 holds 128 deployment
networks for the whole Incus cluster. A network takes one /25 however
many members its replicas run on, and keeps it when a replica moves to
another member.
A node takes /26 chunks of the fabric as it needs them, and each of its
containers is a routed /32 from them. A chunk is 64 addresses, 62 of them
pins. The default /16’s fabric holds 512 chunks, the first never handed
out, and every node holds at least one, so it has room for 511 nodes. A
/14 holds 2,048.
A node whose own LAN or underlay sits inside one of the last four would lose
that network once SHC’s rules converge. The VIP table rejects every packet
into the VIP range that it did not rewrite, so a LAN there stops answering.
So shc init and shc node join refuse such a node before anything is
written: X400450ADROVL at init, X409450ADROVL from the leader. The error
names the interface, the network and the range, and suggests a block that
fits. shc doctor reports an existing node that overlaps under
fabric.addressblock.
A network the node only routes to counts the same way: a site behind a
WireGuard or VPN gateway has no address on any interface, and it would be
cut off just the same. Every routing table is read, so a VPN client’s own
table counts, and the error names the route: route 10.64.35.0/24 via 192.168.1.254 dev eth0. The default route never counts, and neither do the
routes SHC installs. A node that rejoins with shc node join --force while
its containers run still holds their host ends, each with the node’s fabric
address and the route to one container. The leader finds those in the store,
so they do not count either; any other network on the fabric range still
does.
A site that already uses those ranges
Section titled “A site that already uses those ranges”Pick a block clear of every network any node is on, overlays included, and
pass it at genesis. A site with a LAN on 10.64.40.x sits in the default
VIP range, so that site runs:
shc init --address-block 10.80.0.0/16That gives service VIPs 10.80.32.0/20, mesh addresses 10.80.48.0/20,
deployment networks 10.80.64.0/18 and the fabric 10.80.128.0/17, with
10.80.0.0/19 left alone. Any /8 to /16 the site doesn’t use works,
192.168.0.0/16 included. Don’t use 100.64.0.0/10: carrier-grade NAT and
the overlays nodes already run live there.
The block is recorded at genesis and every joiner takes it from the leader. Nothing renumbers a live cluster.
Adding a block
Section titled “Adding a block”A cluster that runs out of room adds a block:
shc cluster block add 10.81.0.0/16An added block is one IPv4 block, a /8 to a /26, and it is not carved.
It is one pool of /26 chunks that node fabric chunks, service VIPs and
mesh addresses are drawn from once the genesis ranges are full. An Incus
cluster’s deployment networks draw on it too once the genesis
deployment-network quarter is full, a /25 (two chunks) at a time, so an
Incus cluster whose quarter is full installs and moves again after an add.
A docker or podman node’s fabric network holds every added block, and an
engine refuses a deployment network inside a network it already has, so
those nodes keep to the genesis quarter’s 128 per node. The add is
refused when the block shares an address with the genesis block, a block
already added, the docker pool, or a network or route on any node, and the
refusal names the overlap. Every node is asked for its networks, so the add
is refused while a node cannot be reached. A block is never removed or
shrunk, so pick one no site LAN, VPN or cloud network will want.
shc cluster block list prints the genesis block, then every added block
with how many chunks it holds, how many are held and how many are left.
shc doctor’s fabric.addressblock and shc cluster fabric status read a
pool amber at 80% held and red when full. A genesis pool that draws on the
added blocks, the fabric, service VIPs, mesh node addresses and an Incus
cluster’s deployment networks, is neither while an added block still has
room: the next address comes from there, and the added block’s own line
says how much is left. Mesh client addresses and a docker or podman node’s
deployment networks never leave the genesis block. On an Incus cluster the
deployment line counts the members’ networks together.
Terraform sets it with address_block on shc_cluster, the bootstrap
shc_node, the shc_cloudinit data source (role cluster) and the core
module’s bootstrap.address_block.
A cluster formed before 0.1.5
Section titled “A cluster formed before 0.1.5”0.1.5 changed the carve. A cluster formed by an earlier release holds mesh
addresses outside the mesh range its block gives now, so it cannot be
upgraded in place. Form a new cluster with shc init and move the apps
over.