Multi-site estates and fences (design)
Status: open design (SHC-1050). Fences are built and admission reads them. Node labels have replaced operator-made fences (SHC-1355, finished): the operator writes
shc node label,shc label,shc constraintandshc selectorwith--whereand--prefer, andshc fenceonly refuses, naming them. So on next §3.1’scustomorigin islabel, §3.2’snode.provideris thecloudcolumn, §3.17’s explain isshc constraint explain -s <deployment>, and §3.18’sshc fenceandshc ruleverbs,shc move --groupand the terraformshc_fenceandshc_ruledon’t exist (terraform hasshc_constraint), and--ruleand--fenceonly refuse. Every recommendation this document made was approved, and the owner added decisions of his own (Swarm retired, Nebula always on, fabric for container traffic, no backwards compatibility, fences and rules). They are listed in the decisions. Scheduler mechanics (rescheduling, replicas and spread, failure handling, taints and tolerations) are a separate design, SHC-1058. Moving the bitspur estate onto the new line is SHC-1061; promanager stays on its current line until the owner decides otherwise.
Later the same day the owner refined four points on SHC-1058, and this document follows them: Traefik and the private resolver run on every node, fed from the replicated control plane; a stack renders as one compose project per node it touches, with names resolved live instead of baked in; every Swarm compose key is honoured or refused; and a node that comes back starts nothing until the control plane confirms what it still owns. The four questions that left open were then decided by agent on the owner’s standing delegation, reversibly (decisions 24 to 28).
Written against shc corp/main ce4b456f9 and ce corp/main 3225a49d, and
updated against shc 3af98e0f2 and ce 68926d62. Every
statement about today’s behaviour cites the file it was read from; ce paths
are in the ce repo, everything else is in shc. Where a doc and the code
disagree, the code wins and the doc is listed under
stale docs.
A word on vocabulary. The ticket says “region”. This document says site in
prose for “a place nodes live”, and it deliberately has no built-in region,
zone or metro. The only places SHC knows by itself are a node, its provider
and the provider’s own location token. Everything coarser is a fence the
operator defines, with whatever kind name suits the estate. The examples use
custom kinds (site, metro, compliance) on purpose.
1. Vision and end goal
Section titled “1. Vision and end goal”The owner wants one SHC cluster to span sites and providers, starting with Hetzner hel1 plus a GorillaServers Proxmox VM, instead of a separate cluster per site. One control plane, one console, one set of tenants. The operator says where things may run; SHC’s one placement engine enforces what physics forbids, keeps chatty integrations together and explains every choice. Nodes run Docker Compose or Podman as executors, Nebula addresses every node, the fabric carries container traffic, and every private name resolves and routes from everywhere.
The end state, as checks a stranger can run:
shc install nextcloud --rule fence=site:ogdenlands nextcloud insidesite:ogden, and its postgres binding either lands next to it or is refused by name with the command that would satisfy it.- A deployment’s block volume comes from the volume connection whose reach covers the node it lands on. Two volume connections stop being an ambiguity.
shc moveinside a volume’s reach rebinds; outside it, copies. A move that leaves the deployment’s fence is refused by name. A move that breaks an integration’s colocation is refused by name and offers the group move.shc node drainand volume failover pick targets only inside each deployment’s eligible set, and never move voter duty out of the control-plane fence.- Every private hostname resolves from any node, container and NetBird device to a mesh address reachable from all of them, from the resolver on the asker’s own node (a NetBird device asks a node’s).
- Every public hostname resolves to the edge of the location the engine placed the app in, and follows the app when it moves.
- Cut the WAN between sites: the home site keeps full function, the far site keeps serving what it runs, and nothing split-brains. Today that does not hold for far-site writers on SHC-managed volumes, which stop themselves (5.8); decision 24 exempts far-location volumes.
shc fence explain <deployment>says why it runs where it runs, or why a placement was refused, naming every fence involved.
Out of scope: active-active apps across sites, a database stretched across sites, automatic cross-site failover of stateful apps, a multi-master control plane, per-site tenancy, and backwards compatibility with today’s clusters (6.3).
2. Architecture
Section titled “2. Architecture”2.1 The starting point
Section titled “2.1 The starting point”What exists today and what this design builds on or replaces:
| piece | state today | where |
|---|---|---|
| node location | node.location holds the provider’s location token, set at self-register from SHC_NODE_LOCATION or the cloud metadata chain (Hetzner, EC2, GCE, Azure, DigitalOcean). Read only by the rebind gate. Nothing in terraform sets it. | ce ent/schema/node.go:68-76, subsystems/impl/node_self_register.go:548-557,765-781 |
| volume location | stamped on the volume row at bind through the optional VolumeLocationReporter | ce daemon/volume_binder.go:234-247, modules/storage/volumeapi/types.go |
| move strategy | ephemeral / rebind / backup+restore, the minimum over every mount. The fast paths relocate only on Swarm; a compose or podman relocation always copies (MOVE-5). The entry point always takes the archive (DNS-2). | ce modules/clone/move_strategy.go |
| rebind gate | a block volume rebinds when its volume.location equals the target’s node.location; a proven shared-filesystem mount and a same-node mount also rebind; everything else copies | ce daemon/clone_move_adapters.go:268-350,620-660 |
| volume provider choice | exactly one volume-capable connection auto-selects; two or more is X400327VOLAMB, a warning and a host-bind, “no node-location tie-break is wired yet”. Only storage.volumes.provider settles it: the ambiguity branch returns before --storage is read. | ce daemon/volume_binder.go:684-748, modules/storage/storageapi/errors.go:596-611 |
| placement engine | the EE scorer ranks nodes for a compose first install (spread, memory headroom) and the install dispatcher runs it there. Swarm gets no score (“SHC owns intent, swarm owns placement”). The scorer “cannot refuse”. | modules/placement/advisor.go; ce modules/app/placement_seam.go, modules/app/install_dispatch.go, daemon/services_install_placement.go:49-58, modules/stack/placement_seam.go |
--node | Swarm: a node.hostname== constraint. Compose and podman: a foreign node is refused (X400956NDLOCL). | ce modules/app/swarm_placement.go, modules/app/runtime_guard.go:150-200 |
| explain | the scorer’s Decision keeps every candidate’s terms “so an operator can be told WHY”, but ChooseNode returns only the node id, so nothing shows it | modules/placement/placement.go:211-224, modules/placement/advisor.go |
| location in scoring | ”node.location exists on the row and the scorer ignores it” | docs/proposals/2026-08-11-placement-policy.md:802-805 |
| drain | filters candidates per deployment through the constraint matcher and refuses by name when nothing satisfies a pin; reads constraints from docker service inspect and labels from docker node inspect | ce modules/cluster/placement.go; modules/cluster/drain.go:1188-1210, daemon/wiring/cluster_drain_constraints.go:73-110 |
| control plane | rqlite. The voter target defaults to three and is settable (shc cluster set-voter-target); later nodes join as non-voters; suffrage is fixed at join by a voter_slot claim. Swarm managers target a constant three. | ce modules/cluster/voters.go, voter_grant.go, types.go:666, swarm_managers.go:81-97 |
| Nebula | EE, off by default, 10.42.0.0/16, UDP 4242, every node a lighthouse, CA key on the enabling node’s disk, lighthouse pins in replicated KV, mesh_address persisted on the node row. Nothing SHC dials rides it. | modules/nebula/, modules/cluster/mesh_orchestrator_impl.go, daemon/wiring/services_extra_formation.go:66; ce ent/schema/node.go:62; docs/guide/operations/cluster-setup.md:45-48 |
| fabric | cluster choice swarm / shc / none, default the Swarm overlay. The shc fabric (WireGuard shc0, IPAM, renderer, nftables compiler) is built; ce runs podman stacks and admits them on --fabric shc. Peers dial internal_address. A Traefik host on the shc fabric is not built. | docs/architecture/cluster/network-fabric.md:150-180,872-882; ce core/runtimecmd/runtimecmd.go, modules/fabric/types.go:98-106, subsystems/impl/fabric_reconcile.go:463-467 |
| ingress | one system Traefik, a compose project with 80/443 published on one node; every managed record points there or at one edge VIP | docs/architecture/cluster/traffic-across-runtimes.md:294-297,790-800; ce modules/clone/move_ingress.go, modules/app/dns_target.go:299-379 |
| edges | an L4 proxy plane with pfSense HAProxy, Hetzner LB and eight other cloud LB backends; one edge serves the system Traefik; a second web connection is X200981DNSAMB | ce modules/proxy/proxyapi/types.go, modules/proxy/providers/, modules/app/dns_target.go:328-354 |
| DNS | a record in every connection zone that covers the host, each with its plane’s address; private zones minted on demand when exactly one private connection is eligible; private providers are pfSense Unbound, BIND over RFC 2136 and PowerDNS-local; the overlay plane is defined and empty; default TTL 300 s | ce modules/dns/dual.go, purpose.go, autozone.go, dnsapi/types.go:281-310, modules/app/dns_private.go:105-128, modules/app/dns_target.go:711-749 |
| resolver | SHC runs none. Where an estate has a split-DNS resolver today it was installed by hand, outside the product. | ce addons/ holds keycloak, openobserve, postgres and traefik only |
| SSO | one Keycloak per cluster in the master tenant, a realm per tenant; the forward-auth sidecar lives in the Traefik project, asks the local daemon, caches an allow for 30 s and fails closed | ce daemon/services_seams.go:836-841,913-915, addons/traefik/compose.yaml:967-974; forward-auth/cmd/forward-auth/authz_cache.go |
| backups | backup.repo resolves per scope through the tenant/environment/stack cascade, “a single global value” in practice; capture runs on the owning node; schedules are keyed by scope, not node; a nightly off-provider mirror exists | docs/guide/operations/enable-backups.md:609-633; ce modules/backup/config_reload.go:112-113, ent/schema/backup_schedule.go |
| provider LBs | a Hetzner LB cannot attach across network zones (X400974PRXLBZ); the Hetzner module’s network is single-zone, “a multi-zone estate is not expressible here at all” | ce modules/proxy/errors.go:106-110; terraform/modules/shc/hetzner/variables.tf:333-338 |
Much of this is EE code in this repo: Nebula, drain, the placement scorer,
volume rebind (X403324VOLEEO in CE) and volume failover.
2.2 The target architecture
Section titled “2.2 The target architecture” people and devices ── NetBird (+ pfSense VPN) ──┐ routes 10.42.0.0/16 │ site A (home) site B (far)│ ┌───────────────────────────────┐ ┌─────────────┴──────────────┐ │ edge A (pfSense HAProxy / LB) │ │ edge B (LB) │ │ Traefik + resolver per node │ │ Traefik + resolver per node │ │ n1 n2 n3 rqlite voters │ Nebula │ n4 n5 non-voters │ │ compose / podman executors │<=======>│ compose / podman executors │ │ keycloak postgres apps │ UDP4242 │ postgres apps │ └───────────────────────────────┘ └─────────────────────────────┘ control-plane fence = site A fabric (shc0) rides Nebula- Executors. Every node runs Docker Compose or Podman. Docker Swarm is retired. Nodes schedule nothing themselves.
- One placement engine. SHC places every deployment, at install, move, drain and failover, inside its fences, and explains the choice. Its scheduling mechanics are SHC-1058.
- Node addressing. Nebula, on in every multi-node cluster from the first join, is the single plane nodes address each other on. Inside a site Nebula keeps the path on the LAN.
- Containers. The
shcWireGuard fabric, riding Nebula. - People. NetBird and the pfSense VPN, never in the cluster boot path.
- Control plane. rqlite voters live in the control-plane fence.
- Per site, an edge. It fronts every healthy Traefik in its location.
- Per node, a Traefik and a private resolver. Both read what shcd materialises from the replicated control plane onto the node (SHC-1058’s cluster config directory), so they need no sync with each other.
2.3 Fences are the placement primitive
Section titled “2.3 Fences are the placement primitive”A fence is a named set of nodes. A node can belong to any number of fences. A rule is attached to an object and says either “stay inside that set” or “share a set of this kind with that partner” (3.3). A deployment’s eligible nodes are the intersection of every rule that applies to it:
eligible(D) = ⋂ fences named by absolute rules on D's scopes, estate to deployment ⋂ reach fence of every volume and bound resource of D ⋂ fences compiled from the relative rules of D's integrations ⋂ nodes that are healthy/degraded and active (today's candidate filter)An empty intersection is a refusal that names the fence that emptied it.
3. The parts
Section titled “3. The parts”3.1 Fences: kinds and origins
Section titled “3.1 Fences: kinds and origins”Every fence is a node set with a kind (a family) and a name. Its id is
<kind>:<name>.
| origin | kinds | membership comes from | who edits | used by |
|---|---|---|---|---|
| built-in, derived | node | each node, a singleton | nobody | any |
| built-in, derived | provider | node.provider (new, 3.2) | nobody | any |
| built-in, derived | location | node.location, qualified by provider: location:hetzner/hel1 | nobody | any |
| resource | volume, storage, lb, ip, endpoint | the resource’s own reach | nobody | resource rules (HARD) |
| system | the one control-plane fence | defaults to the genesis node’s location fence | operator | voter placement |
| custom | any kind the operator names: site, metro, continent, compliance, customer, tier | the operator | operator | operator rules |
Built-ins are derived from facts SHC knows and cannot be removed. location
is the provider’s own attach scope (a Hetzner location, an EC2 availability
zone), which is the unit block volumes care about, so it is kept as the
provider reports it rather than renamed “region”. Qualifying it by provider
answers X400327VOLAMB’s objection that clouds use heterogeneous location
vocabularies: location:hetzner/hel1 and location:vultr/hel1 cannot
collide.
A node whose location is unknown (a Proxmox VM with no metadata and no
declaration) belongs to location:unknown. A single-site estate that never
declares locations then has one location fence and colocation is trivially
satisfied. On a multi-site estate, shc fence ls and explain flag unlocated
nodes, because two of them in different places would look colocated.
Resource fences are the only HARD fences:
- a block volume with a location: the nodes of that location (the rebind gate
computes this today, ce
daemon/clone_move_adapters.go:620-660); - a local volume or host bind: its node (placement ruling P2,
docs/decisions/2026-08-14-placement-rulings.md:10); - a shared filesystem: the nodes with a proven
store_nodelink (X400340LNKEVDrefuses a hand-made link); - a provider LB or floating IP: the nodes it can attach to (
X400974PRXLBZfor Hetzner network zones); - a public endpoint: the nodes holding a public address. A NAT’d node cannot
be reached from the internet, and the DNS ladder already refuses a private
address for internet exposure (ce
modules/app/dns_target.go).
This is the “storage domain” of
docs/proposals/2026-08-11-swarm-native-rebalance.md §3.3 (“Local → one node.
Cloud volume → the nodes in its AZ. Shared → the whole estate”), generalised to
every resource and made a first-class object.
Custom kinds are free-form and may overlap: a node can sit in metro:us-west
and metro:pacific at once. There is no hierarchy object. Nesting emerges
from membership: fence A is effectively inside fence B when every node of A is
in B, so site:ogden inside metro:us-west inside continent:na needs no
declaration beyond membership.
Runtime and architecture requirements stay Phase-1 refusals (rulings P6 and P11). They could become derived fences later; nothing here needs that.
3.2 Node facts, and how they are set
Section titled “3.2 Node facts, and how they are set”| fact | today | decided |
|---|---|---|
node.location | SHC_NODE_LOCATION, else cloud metadata (ce subsystems/impl/node_self_register.go:765-781) | also declared from terraform (shc_node.location, modules/core nodes[].location) and shc node join --location |
node.provider | not stored. The Hetzner probe fills CloudMetadata.Cloud, which is not persisted (node_self_register.go:537-546) | a new column: SHC_NODE_PROVIDER, else the probe that answered, else declared by terraform or --provider |
| custom fence membership | n/a | shc fence add-node, the shc_fence terraform resource, or a join-time --fence list |
Declared beats detected, the order both existing facts already use.
3.3 Rules, and where they attach
Section titled “3.3 Rules, and where they attach”Fences say where nodes are. Rules say where deployments may run. Every rule is attached to a concrete object and travels with it; there are no estate-wide placement “policies” floating free of any object, because those act at a distance and nobody can tell which one moved an app. (This rationale is the only place the word appears.)
A rule is absolute, “stay in fence F”, or relative, “share a fence of kind K with X”.
Rules come from exactly four sources:
| source | the object | set by | form |
|---|---|---|---|
| resource | a volume, floating IP, LB or public endpoint | its own reach, derived | absolute |
| integration | a plug-to-socket binding | the socket’s default, adjusted by the plug (3.5) | relative |
| deployment | one deployment | --rule at install, bind or move; a rules: entry in its deployfile; rules on shc_deployment | absolute, or a relative override for one of its integrations |
| scope | the tenant, and the estate, environment and stack on the same ladder | the operator, on that object | absolute |
The owner first named the fourth source “tenant”, then refined it the same
day: rules attach along the scope ladder SHC already uses for config, vault
paths and backup schedules. That ladder is core/scope (global → tenant →
environment → stack → deployment, ce core/scope/scope.go; one ladder by
ADR-0003). Reusing it brings the stack rung along with the ones the owner
named, and integrations and resources hang below the deployment.
Rules narrow; they never override. Config walks scope.Cascade and takes
the first hit, so the most specific value wins. Placement rules walk the same
cascade and apply every hit: the eligible set is the intersection of all
of them. A lower scope can narrow and can never escape a higher scope’s
fence, and inheritance runs only along that ownership ladder.
Loosening is explicit, recorded and authorised. A loosening names the rule
it relaxes, carries a reason, and is recorded with its author. Only someone
with authority at the scope that set the rule may loosen it: a tenant admin
cannot relax an estate rule. Resource rules cannot be loosened by anyone.
Today config writes are platform-admin only at every tier (ce
modules/config/router.go:35-47), so rules need a finer check than config
has. Decision 27 maps rungs to roles: platform:manage at every rung, a
tenant’s tenant:manage on that tenant and below.
The sources fall into three classes, by how far their rules bend:
| class | sources | overridable | examples |
|---|---|---|---|
| HARD | resource | never | block volume location; local disk = its node; shared FS = linked nodes; public endpoint = nodes with a public address; provider LB / floating IP attach scope |
| DEFAULT | integration | yes, explicitly and recorded, because it is performance and not correctness | postgres, mysql, redis/valkey, mongodb, clickhouse sockets: consumers share location. oidc, saml, smtp, imap, webhook, s3-download, llm sockets: none |
| OPERATOR | deployment, scope | by whoever has authority at that rung | anything |
The DEFAULT class is product behaviour: catalog guidance every estate gets. Which sites exist and which fences and rules an estate uses is estate configuration, never a product default (the owner’s 2026-08-26 ruling that an estate’s operating choices never become product defaults).
3.4 Relative rules
Section titled “3.4 Relative rules”The two ends of an integration must share at least one fence of kind K. The
socket sets the default, the plug adjusts it, and the operator can override it
for one deployment with a recorded --rule colocate=<plug>:<kind>. Relative
rules are judged on each end’s eligible set, not its current node, so no
later placement decision can break one silently. Resolution, replicas,
symmetry and transitivity are in section 5.
3.5 Where colocation rules live
Section titled “3.5 Where colocation rules live”The socket declares the default, next to schema and data, naming the
fence KIND a consumer must share:
placement: colocate: location # node | location | noneCatalog sockets may name only built-in kinds (node, location) or none.
Custom kinds differ per estate, and a catalog default naming site would mean
nothing where no site fence exists. Custom kinds enter through plug
overrides in an estate’s own packages and operator rules at the tenant,
environment, stack and deployment scopes.
The plug may adjust it, and only for the services it lists:
services: # existing field: the consumer services that dial the socket - defectdojo-uwsgi - defectdojo-celery-workerplacement: colocate: node # tighten# or, to loosen, a reason is mandatory:# placement:# colocate: none# reason: only the one-shot initializer reads it- The rule binds only the services the plug lists in
services:. The field already exists and already limits which consumer services join the provider network (cemodules/app/integration_render.go:244-252). - A plug used only by a provisioning or one-shot job has no runtime
colocation: when every listed service is a one-shot, the effective rule is
nonewithout the plug saying so. - A plug may tighten freely and may loosen only with
reason:. - The effective rule is the strictest of socket and plug unless the plug
loosened explicitly. Among catalog values,
node>location>none. - The operator overrides at install or bind, either direction, any kind
including custom ones:
--rule colocate=postgres:metro. The override is recorded on the integration edge and shown by explain.
External sockets. apps/standalone/external-postgres and its siblings
front RDS, Cloud SQL or Neon. The provider is not a node, so the socket
declares where it lives as a fence id from a var, or none:
placement: colocate: location location: "{{ vars.location }}" # e.g. site:aws-us-east-1, or emptyFor an external provider the rule is a scored preference, never a refusal: consumers prefer nodes inside that fence. An absolute rule makes it hard.
Catalog lint. The catalog build refuses:
- a socket for a data-store interface (postgres, mysql, redis, valkey,
mongodb, clickhouse) with no
placement.colocate; - a catalog socket or plug naming a kind other than
node,locationornone; - a plug that loosens without
reason:; - a plug on a colocated interface with no
services:list; - a
services:entry that is not a service in the package’s compose; - an
external-*socket withoutlocationor withcolocateother thannoneorlocation.
No socket or plug file carries a locality field today, so the lint lands with the catalog sweep that satisfies it.
3.6 Location-aware volume provisioning
Section titled “3.6 Location-aware volume provisioning”Today the binder resolves one provider before it knows where the deployment
lands, and two volume-capable connections decline to host-bind with
X400327VOLAMB.
Decided:
- Each volume-capable connection reports its reach before provisioning: a
location-scoped provider its configured provisioning location, a shared
filesystem its linked nodes.
VolumeLocationReporterallows this (“may ignore handle and return their single configured provisioning location”, cemodules/storage/volumeapi/types.go), and eight of its nine implementations already answer without a handle. Cinder looks the volume up first (cemodules/storage/providers/cinder/provider.go:260-264) and needs a no-handle arm. - The engine picks the node from
eligible(D) ∩ ⋃ reach(connection)when the package declaresx-shc-volumes. - The binder keeps only the connections whose reach contains that node. One
is selected, zero keeps the host-bind path, and two or more is still
X400327VOLAMB, now a true tie thatstorage.volumes.providersettles.
Two Hetzner locations are two connections, each with its own location.
3.7 One placement engine on Compose and Podman
Section titled “3.7 One placement engine on Compose and Podman”Today’s fabric matrix (network-fabric.md:150-180) has swarm (Compose +
Swarm), shc (Compose + Podman) and none (one node). The decided line keeps
only the shc fabric: every node runs Compose or Podman, and SHC’s one
placement engine places every deployment. Neither runtime has cross-node
placement of its own, so nothing competes with the engine.
What the engine does, and what it inherits:
- Install. It filters candidates to the eligible set, the scorer ranks
inside it (
modules/placement/advisor.go), and the dispatcher runs the deploy there (cemodules/app/install_dispatch.go). The scorer’s contract stays “it returns a node or nothing; it cannot refuse” (cemodules/stack/placement_seam.go); refusing an empty eligible set is a Phase-1 admission gate before the QUEUED row. --nodeand--rule.X400956NDLOCLwas written because nothing could run a compose deploy on another node. The dispatcher can, so both flags become placement inputs and the refusal goes.- Move. MOVE-5 exists for the same reason: only Swarm could relocate by re-rendering. With the dispatcher, a compose or podman relocation is stop on source, rebind if the volume allows, deploy on target, so the ephemeral and rebind fast paths apply to both runtimes.
- Record.
recorded_node_idstays where the bytes are,pinned_node_idstays the operator’s explicit--node, and the rules attached to the deployment and its scopes are the recorded intent every later move and drain reads. Writing a scored node into the pin slot is the mistakemodules/stack/placement_seam.gowarns about. - Explain. Every decision keeps the scorer’s full
Decisionand the fence provenance (3.17). - Scheduling mechanics (rescheduling on failure, replicas and spread, taints and tolerations) are SHC-1058. This design supplies its inputs: the eligible set per service and the move group.
Retiring Swarm removes the Swarm runtime and --fabric swarm, the Swarm
placement renderers (ApplyDeployPlacement, ApplyVolumePlacement,
ApplyVoterExcludes in ce modules/app/swarm_placement.go), the unread
storage.<provider> label helper (ce modules/storage/swarm_label.go),
converge-swarm, and drain’s docker service inspect constraint reads. No
Swarm shim survives (6.3).
The Swarm keys in package compose files stay, with their meaning (ruled on
SHC-1058). Any key that had an effect under Swarm keeps it under SHC or is
refused at render, never silently dropped. Today the fabric doc accepts the
opposite, “compose already ignores swarm-only deploy: keys”
(network-fabric.md:262-268). Two of those keys are placement, so they land
here:
deploy.placement.constraintstranslate into rules on the deployment:node.hostname==Xtofence=node:X,node.labels.K==Vto a custom fenceK:V. A constraint with no fence form is refused at render. No catalog package declares one today (a grep ofapps/,addons-overlay/and ceaddons/finds none), so this is for estate and third-party packages.deploy.replicasandmode: globalbecome placement inside the eligible set: N copies spread by SHC-1058’s rules, or one per eligible node.
The rest (configs, secrets, resources, update and rollback, restart) is
SHC-1058’s.
3.8 Move, move groups, drain and failover
Section titled “3.8 Move, move groups, drain and failover”Move. For shc move D --node N or --rule fence=F:
| condition | outcome |
|---|---|
| N outside a fence named by one of D’s absolute rules | refused by name, naming the rule and the scope that set it |
| N breaks a colocation rule with a partner | refused by name, offering shc move --group D --rule fence=<fence> |
| N inside every volume’s reach | ephemeral or rebind, on either runtime (3.7) |
| N outside a volume’s reach | backup+restore, as the rebind gate already decides (ce daemon/clone_move_adapters.go:323-350); the volume is re-provisioned from the connection whose reach covers N (3.6) |
Move group. The connected component of colocation edges (5.4). shc move --group D --rule fence=F checks every member’s rules against F, then moves
members through the single-deployment Move in dependency order: stop
consumers, move providers, move consumers. Nothing is re-rendered for
addressing: cross-node names resolve live through each node’s resolver
(3.12), which replaces the fabric’s render-time extra_hosts and the
re-render ripple it accepted (network-fabric.md:326-331,854-862). It keeps
a write-ahead journal like
the drain’s (modules/cluster/drain_journal.go), so a failure resumes rather
than guesses. A failed member stops the group, leaves the rest at the source,
and prints that the colocation is violated with the command that finishes or
reverses it. SHC never moves a group on its own initiative.
Drain. The EE drain already refuses by name when nothing satisfies a pin
(modules/cluster/drain.go:1188-1210, modules/cluster/drain_pin_gate.go).
It reads each deployment’s eligible set from the fence store instead of
docker service inspect, refuses a deployment with no in-fence target by
name, and never moves voter duty out of the control-plane fence.
Failover. The EE volume failover ticker (off by default,
subsystems/volume_failover.go) already stays inside the volume’s location.
It also stays inside the deployment’s eligible set; no target means the
deployment stays down and the stall diagnosis names the fence. What triggers a
failover and how aggressively is SHC-1058, which reuses the dead-source move
rather than building a second one. shc move --presumed-dead already
relocates off a dead node from replicated state: it needs the node
unreachable in the registry and a fresh TCP probe failing, rebinds surviving
block volumes behind a generation bump, and refuses local-only mounts unless
--discard-local (ce modules/clone/move_presumed_dead.go). The probe cannot
tell dead from partitioned; the self-fence covers that (5.8).
3.9 The control-plane fence
Section titled “3.9 The control-plane fence”Today the voter set is the first nodes to join, up to the voter target, with
no location awareness. rqlite fixes suffrage in the join request and “exposes
no online role change” (ce modules/cluster/voters.go); the only change is
leave-and-rejoin (PromoteBeforeDeparture, ce voter_rebalance.go). The
voter slot is claimed at enrol (ce modules/cluster/voter_grant.go). With
Swarm retired there are no managers to converge.
Decided:
- A built-in
control-planefence. Its default membership is the genesis node’s location fence. The operator can change membership; the fence cannot be deleted. - A node outside the fence always joins as a non-voter, whatever the voter count. The check runs at enrol, in the voter-slot claim, because suffrage cannot change later without a rejoin.
- Drain, failover and
PromoteBeforeDeparturenever pick a candidate outside it, and leader-only work therefore always runs inside it. shc cluster statusshows voters against the fence, warns when the fence holds fewer than three healthy nodes, and notes when it has no spare (fewer than four).- Refined by Clay on 2026-09-30: the voter count is automatic and odd, the largest odd number up to the fence’s member nodes, capped at five, with automatic promotion and demotion as nodes join or leave (new-line simplification §4.9).
Losing nodes is in 5.9.
3.10 Nebula, the node-addressing plane
Section titled “3.10 Nebula, the node-addressing plane”What stops the mixed shape today. Measured in code:
- Nothing SHC dials rides Nebula. Raft and the rqlite API bind the
--nicinterface (“Raft does not ride the Nebula mesh”,docs/guide/operations/cluster-setup.md:45-48); peers dialInternalAddrFor, “the only dialable address column node rows carry” (cemodules/cluster/peer_seam.go:110-114); fabric peers dialinternal_address(cesubsystems/impl/fabric_reconcile.go:463-467). - Nebula dials
internal_addresstoo, so a NAT’d LAN node publishes an unroutable endpoint: “the overlay cannot bootstrap the nodes it exists to bridge” (docs/proposals/2026-08-02-node-addressing-model.md:449-462). - The Nebula CA key lives only on the enabling node’s disk.
GetCAreads{state}/nebula/ca.key(modules/nebula/service.go:376-395), “the leader is the only place with the nebula CA key” (modules/cluster/mesh_orchestrator_impl.go:40-41), andEnsureCAExistsmints a new CA when none is on disk. The cluster mTLS CA, by contrast, is escrowed in the vault (docs/guide/operations/node-rotation.md:316-319). - Every node is a lighthouse advertising its internal address. A
lighthouse pin exists in replicated KV
(
daemon/wiring/services_extra_formation.go:66) with no terraform surface. - Proxmox nodes are not staged with
nebula-cert, the mixed example’s own KNOWN GAP (terraform/examples/mixed-proxmox-hetzner/main.tf:27-37). - The shipped config is WAN-hostile: MTU 1300,
punchyon, nopreferred_ranges, no relays (modules/nebula/templates.go). Until SHC-1049 it also shippedtx_queue: 500and default socket buffers: one TCP stream over a 150 ms link reached 20 Mbit/s with those and 147 Mbit/s with 16 MiB socket buffers andtx_queue: 5000, which now ship (network layering decision). The same cell cut everything but UDP 4242: the far node lost quorum while its mesh pings answered, and healed about 50 s after the path reopened (Swarm about 2.5 min), which is item 1 seen from a rig. - Latent: NIC auto-detect excludes an overlay device named
nebula1(cecore/nic/nic.go:357-361), but the device SHC creates isnebula0(modules/nebula/templates.go:131).
Decided shape.
- Always on. Every multi-node cluster runs Nebula from the first join; a
one-node estate has no peer to address. It is the single node-addressing
plane: raft (
-raft-adv-addr,docs/proposals/bind-advertise-split.md), the rqlite API, the internal API and fabric peers all dial mesh addresses. There is no LAN-addressed mode beside it. - The addressing model gains a fifth role. Its rule “every SHC-owned
overlay dials the internal address plus its own port” cannot hold when the
internal address is the overlay.
underlayis what Nebula dials (the LAN address inside a site, the public or declared edge address across sites);internalis the mesh address. - Bootstrap over the underlay, run over the overlay. A joiner enrols
against a lighthouse’s underlay address (the internal API is mTLS), receives
its Nebula cert, the CA and the lighthouse list in the enrol response, starts
Nebula, and addresses everything by mesh from then on. Joins already ride
“the LAN token over any routable path” (
modules/node/commands/join.go:39-47). - The Nebula CA lives in the vault, like the cluster CA, so any voter can sign.
- Lighthouses and relays are the nodes with a public address, at least one per site that has one and at least two in total, pinned from terraform. LAN-only nodes are members that punch out through them; relays cover symmetric NAT.
preferred_rangesis each node’s site LAN, so same-site traffic stays on the LAN path.- WAN tuning of buffers and
tx_queuefrom SHC-1049’s final numbers.
3.11 The fabric for container traffic
Section titled “3.11 The fabric for container traffic”The shc fabric carries container traffic: WireGuard shc0 per node, a /24
per node, pinned service addresses, and today cross-node names baked into
extra_hosts at render time (network-fabric.md:1-40,970-989). Its peers
dial mesh addresses, so it rides Nebula; its MTU drops from 1420 to 1220
(Nebula’s 1300 less WireGuard’s 80).
Ruled on SHC-1058: a stack renders as one compose project per node it
touches, holding only the services placed there, and cross-node names resolve
live through each node’s resolver (3.12) instead of extra_hosts. A move no
longer re-renders the stacks connected to the moved one.
Two things must land before Swarm can go, because today the Swarm overlay does them:
- Traefik on the fabric. “A Traefik host on the shc fabric is slice 2
work” (
network-fabric.md:878), and the renderer bakes no ingress host entry, so on--fabric shc“the firewall permits traffic to a target the proxy has no way to name” (traffic-across-runtimes.md:352-380). With Traefik’s routes fed from the control plane (3.13) its backends are fabric addresses, so nothing has to be baked. - Fabric slices 2 and 3, tracked as SHC-1055 (nftables filtering and default-drop) and
SHC-1056 (Podman adapter, depends on SHC-1055). The fabric doc disagrees
with itself about how much of both landed (stale docs, 6.6), and ce already
carries an nftables compiler (
modules/fabric/policy/) and runs podman stacks, so both tickets start by stating exactly what is missing.
The fabric doc’s “Retire the Nebula module” and “WAN membership over the
fabric” follow-ons (network-fabric.md:1083-1098) are withdrawn: Nebula
addresses nodes, the fabric carries containers.
3.12 Split DNS
Section titled “3.12 Split DNS”Which provider owns which zone. A domain’s public zone lives at one public
DNS connection (Hetzner, Cloudflare and so on). A domain resolves privately
only if it is on a private connection’s zone list; SHC then mints the private
zone (autozone.go) and publishes a record in every zone that covers the
host, each with its own plane’s address (dual.go). Records are
provider-created per connection, never by hand.
Decided answers.
- The public answer is the edge of the location the engine placed the app in (3.13): that location’s LB or pfSense VIP. A location with no edge answers with one healthy node that holds a public address, re-pointed by the engine when it fails (decision 28).
- The private answer for a routed hostname is the mesh addresses of the
healthy Traefiks in that location. It is the same answer everywhere, and it
is reachable from every node over Nebula, from every container through its
host, and from every NetBird device through routing peers that advertise
10.42.0.0/16. - A service name between containers answers with the fabric addresses of the healthy replicas (SHC-1058).
When an app moves between locations. The engine re-runs the DNS ladder for
the moved deployment and the public record re-points at the new location’s
edge. Clients cache the old answer for up to the record TTL, 300 s by default
and settable per scope with dns.ttl (ce modules/app/dns_target.go:711-749).
That window costs nothing: every Traefik carries every route (3.13) and
reaches every backend over the fabric, so the old edge keeps serving across
the WAN until caches expire. Certificates are already on every Traefik (3.13),
so the new edge needs no fresh issuance. Lowering dns.ttl before a planned
move shortens the hairpin. The private answers change as soon as the new
placement is materialised on each node.
A resolver on every node. Each node runs a private resolver that answers from what shcd materialises out of the replicated control plane onto that node, the cluster config directory of SHC-1058. There is no zone transfer and no sync between resolvers. The node’s host resolver asks it, and containers reach it through their runtime’s embedded resolver, which forwards to the host’s. During a partition it keeps answering from the last materialised state, so a cut-off site still resolves its own names. NetBird devices reach the node resolvers’ mesh addresses through a NetBird nameserver group. The resolver is a container in the builtin Traefik project beside forward-auth (decision 26).
The private DNS connections stay. They still decide which domains are private, and pfSense Unbound, BIND and PowerDNS-local keep the LAN-plane records SHC publishes into them for clients that ask them directly, such as a house LAN behind a pfSense edge.
| state | |
|---|---|
| works today | dual publication writes every matching zone with its plane’s address (dual.go); private zones are minted on demand (autozone.go); LAN-plane records carry dns.lan_address, else the public answer, for a site’s LAN clients (dns_private.go:105-128); the public record is always kept, so an off-VPN client gets a clear refusal rather than NXDOMAIN (purpose.go); dns.ttl is a scoped knob; moving the entry point re-points every managed record (ce modules/clone/move_ingress.go) |
| wrong | a private name has one answer, reachable from one site’s LAN only; nothing serves the overlay plane, and dns_private.go’s note that “the node table carries no overlay address” is stale for Nebula, which persists node.mesh_address; two private connections make autozone refuse (X200923DNSAZA, autozone.go:27-31), which an estate with a pfSense edge in each site trips |
| missing | a public target per location (today one Traefik and one edge); a resolver on every node answering mesh and fabric addresses from replicated state; node and container resolver configuration; NetBird nameserver groups, which nothing manages (SHC-277 is the future NetBird DNS lane, ce modules/connections/capabilities.go:297-298) |
3.13 Edges and load balancers
Section titled “3.13 Edges and load balancers”Decided: one edge per location fence, a pfSense HAProxy or a provider LB, in front of every healthy Traefik in that location. The edge reaches them over the location’s LAN, which is why it is per location.
Ingress: a Traefik on every node (ruled on SHC-1058). There is no
Traefik-to-Traefik sync. Hostnames, healthy backends and their fabric
addresses live in the replicated control plane; shcd materialises each node’s
share into the cluster config directory, and the local Traefik reads it
through its file provider, which the builtin already runs
(--providers.file.directory=/etc/traefik/dynamic; the daemon sends a SIGHUP
after writing, because the provider’s watch skips subdirectories, ce
addons/traefik/compose.yaml:638-647, modules/app/traefik_reload.go). The
Docker-label provider (:624-628) goes. Every Traefik carries every route and reaches every
backend over the fabric, preferring backends in its own location, so DNS
alone decides which location a client enters and a re-point never waits for a
re-render. Readiness gating and draining cover the seconds a node’s copy lags.
The forward-auth sidecar comes with each Traefik, since it is part of the
Traefik project. The fabric’s nftables compiler already models a per-node
Ingress class (ce modules/fabric/policy/model.go).
Certificates are issued once and distributed through the control plane;
no Traefik runs ACME. Today Traefik on the one ingress node does ACME, and the
cert-delivery broker reads its store to hand certificates to apps that
terminate TLS themselves (ce modules/certbroker/doc.go,
modules/certbroker/acme_source.go). The broker’s source seam already names
the replacement: “a daemon-side ACME client is a second implementation behind
this same seam” (ce modules/certbroker/broker.go:14-15). With one, the
issued material flows to every Traefik and to the broker’s app deliveries
alike. The entry-point move (ce modules/clone/move_ingress.go) has nothing
left to move and retires.
Apps with replicas in several locations. Three shapes:
- One global entry point. The app’s public name points at one location’s edge; its Traefiks proxy to replicas in every location over the fabric. Nothing new is needed beyond the per-location edge. Losing that location loses the entry point.
- Health-checked DNS. SHC probes each location’s edge and withdraws a dead one’s records, so a client fails over within a TTL. SHC’s DNS module has no health checks today.
- GeoDNS. Answers by client location. Provider-specific and usually a paid feature; SHC has no support for it.
The v1 answer is one global entry point, because colocation keeps stateful apps in one location anyway; health-checked DNS is the follow-up, and GeoDNS is not planned.
Hard fences. A provider LB attaches only within its own scope: a Hetzner LB
cannot cross network zones (X400974PRXLBZ), and the Hetzner module cannot
express a multi-zone network (terraform/modules/shc/hetzner/variables.tf:333-338).
Each LB is a resource fence of its location. SHC manages no movable
floating IP today (the OpenStack module’s per-node floating IPs are NAT
addresses, terraform/modules/shc/openstack/main.tf:207-221); when it does, each is a resource fence with its provider’s attach scope.
Edge failover. A provider LB is highly available inside its location, and that is the provider’s job. A pfSense edge is one box: SHC has no CARP or HA pair support (nothing in ce mentions CARP, VRRP or keepalived). When an edge dies but its location lives, the other location’s edge can take the public names and reach the apps over the fabric; when the location dies, backups are the recovery path (3.16). Until health-checked DNS exists, that re-point is a runbook step.
| state | |
|---|---|
| works today | the L4 proxy plane: pfSense HAProxy, Hetzner LB, AWS NLB, GCP TCP LB, Azure, DigitalOcean, Linode, Vultr, Octavia and CloudStack backends (ce modules/proxy/providers/), shared-VIP and per-app cardinality, full-pool reconcile, the VIP handed to DNS (proxyapi/types.go); PROXY protocol with the edge’s own addresses kept out of LAN allowlists (proxyapi/edge.go); Hetzner LB targets resolved to servers and attached to the private network (providers/hcloudlb/provider.go) |
| wrong | a second web connection on the system Traefik is X200981DNSAMB, so there is one edge per cluster (ce modules/app/dns_target.go:328-354); a compose Traefik publishes on one node, so the edge’s pool narrows to that node (Forward.Pool, proxyapi/types.go) |
| missing | an edge per location fence, with DNS choosing by the app’s location; Traefik on every node with routes fed from the control plane; a daemon-side ACME source for the cert broker; health-checked DNS; floating IPs; pfSense edge HA |
3.14 Cross-site integration guidance
Section titled “3.14 Cross-site integration guidance”The rule of thumb is calls per user request times round-trip time. At 150 ms RTT a page that makes 50 sequential database queries waits 7.5 s.
| integration | default | why |
|---|---|---|
| postgres, mysql, redis/valkey, mongodb, clickhouse | location | per-query round trips on the request path |
| the app’s own files | HARD, via its volume | the volume’s reach fence |
| oidc, saml | none | a few round trips per login, none per request |
| smtp, imap | none | asynchronous or per-session |
| webhook | none | fire-and-forget |
| s3 downloads, llm | none | few, large requests |
| app-to-app APIs on the page path (webdav, document editors, PDF tools) | the socket author’s call, usually location | chatty in practice |
3.15 SSO per site
Section titled “3.15 SSO per site”Today one Keycloak serves every realm (ce daemon/services_seams.go:913-915).
The forward-auth sidecar in each Traefik project asks the local daemon per
session, caches an allow for 30 s, fails closed, and serves stale answers only
on a 429 (forward-auth/cmd/forward-auth/authz_cache.go).
Decided for v1: one Keycloak in the home site, OIDC colocation none. A
far-site login pays a few WAN round trips. During a partition, far-site gated
apps fail closed within about 30 s, because the daemon’s answer needs
leader-forwarded reads (5.8) and Keycloak is across the cut. A follow-up lets
the daemon answer forward-auth from its local replica with bounded staleness,
so established sessions survive a partition; new logins still need Keycloak.
Keycloak per site means Keycloak cross-site replication and is out of scope.
3.16 Backups under fences
Section titled “3.16 Backups under fences”Decided.
- Targets are chosen per location, off-site. Each location fence carries
its own backup target as an attribute of the fence object (
shc fence set location:hetzner/hel1 --backup-repo …). A deployment’s archives go to the target of the location it runs in unless a narrower config scope setsbackup.repo, and each location’s target lives outside that location, so losing a location never loses its backups. - A second copy on a second provider. The nightly off-provider mirror
(
enable-backups.md:609-633) points at Backblaze B2 or another provider that is not the first repo’s. The estates have no true second provider today: their object storage is Hetzner only. - Restore-as-move is the copy path. A move that leaves a volume’s reach is
backup+restore already (
SelectMoveStrategy, MOVE-2 and MOVE-6); it carries the re-provisioning of 3.6. - Backups are the recovery path when a location is lost. Each lost
deployment is restored from its off-site repo into a surviving location with
shc move --presumed-dead(cemodules/clone/commands/main.go:298-315) orshc recover, with its fence pointed at the new location. The recovery point is the schedule interval. - The schedule follows the app. Schedules are keyed by scope (tenant,
stack, environment; ce
ent/schema/backup_schedule.go), not by node, so a move changes nothing about when backups run. Where they go follows the new location’s repo; each backup row records its archive’s location (backup.location, ceent/schema/backup.go), so history in the old repo stays restorable.
| state | |
|---|---|
| works today | per-scope backup.repo; capture on the node that owns the volume; the verify sweep; the off-provider mirror mechanism; shc recover and --presumed-dead moves; backup+restore as the cross-reach move; scope-keyed schedules; per-archive location on the backup row |
| wrong | backup.repo is “a single global value: one provider, one credential, holding every archive the estate has” (enable-backups.md:611-614); there is no site axis, so a far-site capture crosses the WAN to one bucket; the mirror is armed on the leader only; no true second provider on the estates |
| missing | per-location off-site repo resolution; a per-location mirror; a location-loss runbook proven on a rig |
3.17 Explain
Section titled “3.17 Explain”shc fence explain <deployment> prints the eligible set and every rule that
shaped it, each with the object and scope it came from, the move group, the chosen node with the scorer’s
terms (modules/placement/placement.go:211-224), and for a refusal the first
fence that emptied the set:
$ shc fence explain acme/nextcloud/proddeployment acme/nextcloud/prod composerunning on gor-02 scored 812/1000 (spread 640, memory 930)
eligible gor-01 gor-02 resource volume nc-data stay in storage:ceph-ogden gor-01 gor-02 gor-03 integration postgres plug -> acme/postgres/prod share location (socket default) location:gorillaservers/ogden tenant acme stay in site:us-west (2026-10-02) gor-01 gor-02 hil-01 deployment acme/nextcloud/prod no rulesmove group acme/nextcloud/prod, acme/postgres/prod (location)Every placement refusal (install, move, drain, fence edits) prints the same lines.
3.18 Surfaces
Section titled “3.18 Surfaces”- CLI:
shc fence create <kind>:<name>,ls [--kind],show,add-node,remove-node,rm,explain.rmrefuses built-ins and fences a deployment depends on;remove-noderefuses when a deployment on that node would leave its eligible set, naming it. - Rules at a scope:
shc rule add <rule>/ls/rmwith the same scope flagsshc config settakes (--tenant,--environment,--stack), plus--reasonfor a loosening. - Rules on one deployment:
--rule(repeatable) on install, update, bind and move, in two forms,--rule fence=<fence>and--rule colocate=<plug>:<kind|none>; arules:entry in a deployfile.--rule fence=location:hetzner/hel1is the explicit form the ticket called--region.shc move --groupmoves a move group. - Membership:
--location,--providerand--fenceonshc node join(the fences the node joins). - Terraform: a
shc_fenceresource (kind, name, nodes); ashc_ruleresource at a scope;rulesonshc_deployment;locationandprovideronshc_nodeandmodules/corenodes[]; lighthouse pins onshc_mesh. - API: the same through the daemon; the openapi document regenerates.
4. Scenarios
Section titled “4. Scenarios”S1. Joining a GorillaServers VM to hel1. Terraform declares location = "ogden", provider = "gorillaservers" and fences = ["site:ogden"]. The VM
enrols over a hel1 lighthouse’s public address, receives its Nebula identity,
and joins as a non-voter because it is outside the control-plane fence
(location:hetzner/hel1). shc fence ls shows
location:gorillaservers/ogden with one node.
S2. An app in Ogden whose database is in hel1. shc install wiki --rule fence=site:ogden --plug postgres=acme/postgres/prod is refused: the postgres socket
colocates location, the provider’s eligible set is location:hetzner/hel1,
and no location fence holds both. The refusal offers three ways out: a
postgres in Ogden, a group move, or --rule colocate=postgres:none with a
reason (recorded, shown by explain).
S3. Moving a database with its consumers. shc move acme/postgres/prod --rule fence=site:ogden is refused, naming its three consumers. shc move --group acme/postgres/prod --rule fence=site:ogden moves all four by backup+restore (every
volume leaves its location), consumers stopped first and started last.
Their public records re-point to the Ogden edge; hel1’s edge serves the
stragglers for one TTL.
S4. Keycloak in hel1, apps everywhere. Allowed. OIDC colocates none, so
no edge forms and Keycloak does not glue the estate into one move group.
S5. A fence spanning two sites. The operator creates metro:us-west with
the Ogden and Hillsboro nodes and adds the rule fence=metro:us-west to
tenant acme. Every acme deployment inherits it. A stateless acme app may run
in either location; how its replicas spread is SHC-1058. Its redis
binding (location) puts the app and redis in one location inside the metro.
S6. Draining an Ogden node. Each deployment’s candidates are the other Ogden nodes in its eligible set. A deployment with none is refused by name, and the output names the group move that would evacuate it.
S7. Two Hetzner locations, two volume connections. An install lands on a
fsn1 node, and the binder keeps the connection whose reach is
location:hetzner/fsn1. No X400327VOLAMB.
S8. The WAN link drops. See 5.8.
S9. A control-plane node dies. See 5.9.
S10. The Ogden location is lost. Its deployments are restored from their
off-site repos into hel1 with --presumed-dead, their fences re-pointed, and
DNS follows the new placement.
5. Mechanisms
Section titled “5. Mechanisms”5.1 Effective placement
Section titled “5.1 Effective placement”For deployment D, per service s:
eligible(D, s) = ⋂ { F : F an absolute fence of D } ⋂ { reach(v) : v a volume or bound resource of s } ⋂ { F_K(g) : g the move group of D, K a colocation kind binding s } ⋂ healthyActivehealthyActive is today’s candidate filter (cluster_state healthy or
degraded, availability active; modules/placement/advisor.go). By default
the services of one stack share an implicit node colocation, so the stack
lands on one node and its per-service sets intersect into one. Only a service
that declares replicas or its own placement is split out (5.7).
Every rule applies at once. Resource rules are never loosened. An integration
rule can be loosened only by the plug with a reason, or by a recorded
deployment-scope --rule colocate. An operator rule can be loosened only by
someone with authority at the scope that set it. Everything else only
narrows: an operator rule that would widen a resource fence simply intersects
down to it, and explain says so.
5.2 Resolving a relative rule
Section titled “5.2 Resolving a relative rule”“C and P share a fence of kind K” holds when some fence F of kind K contains every eligible node of both ends. Eligible here means the base set, before any derived fence is added. The engine compiles the rule to an absolute fence:
- Candidates are the K fences that contain
eligible(P)and intersecteligible(C). - None: refused by name. If
eligible(P)itself spans several K fences, the refusal offers an absolute rule on P or a group move. - One or more: the smallest by node count wins, then by id. The choice is recorded on the move group as a derived fence with its provenance (“from C → P, kind K”) and shown by explain.
- The derived fence joins both ends’ eligible sets.
none is the whole-cluster answer. It is trivially satisfied and forms no
edge.
5.3 Replicas
Section titled “5.3 Replicas”Every replica must share a K fence with the partner’s placement. Because the rule compiles to one fence F that is intersected into both ends’ eligible sets, every replica of both ends lands inside F. A three-replica app colocated with a single-writer database runs all three in the database’s location. Spreading them inside it (maximum skew), a stable identity and volume per replica for clustered stateful apps, a fabric address per replica and Traefik’s list of healthy replicas are SHC-1058.
5.4 Symmetry, transitivity, the move group
Section titled “5.4 Symmetry, transitivity, the move group”- Symmetric. Moving either end re-checks the rule for both, because the derived fence belongs to the group, not to one end.
- Transitive. A
B and BC (bothlocation) put A, B and C in one location. The engine builds groups with union-find per kind over the binding edges, one derived fence per kind per component. - Mixed kinds. A
B (C (location) and Bmetro) give a location for {A, B} and a metro for {B, C}; B sits in both, and the resolver checks that the two choices intersect on B. - The move group is the connected component over edges of every kind
except
none.
5.5 Nesting and overlap
Section titled “5.5 Nesting and overlap”Nothing special. “Share a fence of kind K” is evaluated on membership. If every location sits inside some metro, sharing a location implies sharing a metro, because the node sets say so. A node in two metros satisfies a metro rule with a partner in either. A node in no fence of kind K cannot satisfy a K rule, and explain names the missing membership.
5.6 Data model
Section titled “5.6 Data model”All in the replicated store, all in the new line’s single initial migration (6.3):
node.provider, nullable text.node.locationexists.fence(id, kind, name, origin, description, created_at); origin isderived,resource,systemorcustom.fence_member(fence_id, node_id, source); built-in and resource rows are written by a reconciler from node and resource facts, custom rows by the operator.placement_rule(id, scope, scope_key, tenant_name, stack_name, environment, form, fence_id, partner, fence_kind, loosens, reason, actor, created_at), keyed on the scope columns andscope_keyexactly asbackup_scheduleis (ceent/schema/backup_schedule.go), so rules reusecore/scoperather than a parallel ladder. Integration rules live on the edge (below); resource rules are derived and never stored.- A location fence’s backup target, an attribute of the fence (3.16).
- On the persisted integration edge:
colocate_override,override_reason, and the derived fence id with its provenance. placement.colocate,placement.locationandplacement.reasonin the socket and plug schemas.
5.7 Rendering
Section titled “5.7 Rendering”The engine decides a node per service, one node per stack by default (5.1). For each node a stack touches, SHC renders a compose project holding only the services and replicas placed there, under the stack’s own project and network names, and the dispatcher runs it on that node (ruled on SHC-1058). Pinned fabric addresses stay; partners’ names are not baked in, because each node’s resolver answers them live (3.12). Swarm placement keys arrive as rules (3.7). There is no scheduler to render constraints for.
5.8 WAN partition
Section titled “5.8 WAN partition”Voters all sit in the home site (5.9). rqlite forwards weak and strong
reads to the leader (core/db/rqlitedriver/discovery.go:47-58); SHC’s DSN
sets no read level (dialDSN, same file), so reads run at gorqlite’s default,
weak (gorqlite conn.go:259 at the pinned version). A far-site daemon
therefore loses reads as well as writes.
| home site | far site | |
|---|---|---|
| control-plane writes | work | refused: no leader reachable |
| control-plane reads | work | fail, forwarded to an unreachable leader |
| running containers | run | deployments on host binds and local disks run; a deployment holding an SHC-managed RWO volume stops itself, all of its containers (below) |
| placement | normal, but no far node is a candidate | frozen |
| ingress and edge | serve their hosts | serve their hosts from the last materialised routes; route changes frozen |
| public DNS | unchanged (records live at the providers) | unchanged |
| private DNS | each node’s resolver answers | each node’s resolver answers from its last materialised state |
| SSO-gated apps | work | fail closed within about 30 s (3.15) |
| backups | run | cannot record; resume after the heal |
| volume failover | finds no candidate for a far deployment, because home nodes are outside its volume’s location fence | nothing runs |
| heal | nothing to reconcile in raft | non-voters catch up from the log; a stopped writer starts only after the control plane confirms the node still owns it and its lease is re-acquired (SHC-1058) |
SHC-1049 measured the heal at about 50 s for SHC, and about 2.5 min for Swarm, after a mesh-only cut (network layering decision).
The self-fence stops far-site writers. Every node runs a self-fence loop,
whatever the auto-failover flag says (ce
subsystems/impl/volume_self_fence.go:14-22,44-47). Every 5 s it renews a
lease on each RWO volume attached to it, and a renew is a quorum write. After
two failed ticks (:68-82) it stops every container carrying the owning
deployment’s labels, lazily unmounts and flushes (:19-20,576-583;
volume_self_fence_node.go:134-177). A far node cut off from the leader
cannot commit a renew, so every far-site deployment holding an SHC-managed RWO
volume stops within seconds, although no home node could attach its
location-bound volume anyway. A healthy renew only clears the loop’s mark
(:258-265); the loop restarts nothing it stopped. Host binds and local disks
carry no attachment and keep running. Decision 24 closes this gap with end
state 7: a volume whose reach holds no node of the control-plane fence is
exempt from the self-fence and is taken over only on a STONITH confirm or an
operator’s --presumed-dead.
The same loop is what makes the EE automatic failover safe against a node that
is only partitioned: that engine waits a rebind dwell of at least 45 s (ce
modules/storage/storageapi/errors.go:552-556), and the self-fence finishes
well inside it. --presumed-dead has no dwell. It needs the poller’s
unreachable stamp, three failed polls on a 2 s tick (ce
modules/health/node_poller.go:287-288,396-397), and a failing probe, so when
SHC-1058 reuses it for automatic failover it has to wait out the self-fence
first (decision 25). On return, SHC-1058 adds the other half: runtime restart policies no
longer start SHC-managed containers at boot, and the daemon starts them only
after the control plane confirms what the node still owns.
Nothing split-brains: non-voters cannot be elected, so the far site cannot
form a second leader. The unsafe verb is shc cluster repair --force (it posts
to the force_leader route, modules/cluster/commands/admin_fleet.go:98-123), an
operator act against nodes confirmed destroyed. Whether the engine ever fails
a far deployment over into the home site on its own is SHC-1058’s call; the
fences already forbid it for anything bound to a far volume.
5.9 The control-plane fence losing a node
Section titled “5.9 The control-plane fence losing a node”V voters need floor(V/2) + 1 alive. With three in the fence:
| voters lost | live voters | needed | state |
|---|---|---|---|
| 0 | 3 | 2 | healthy, tolerates one loss |
| 1 | 2 | 2 | writes work, zero fault tolerance (NoFaultTolerance) |
| 2 | 1 | 2 | wedged: no writes, and raft cannot remove the dead voters (ce modules/cluster/voters.go, “The quorum-loss deadlock”) |
Recovery from one loss: with a healthy non-voter already in the fence,
removing the dead voter with --rebalance runs PromoteBeforeDeparture, 3 →
4 → 3, never below target. A fence with only three nodes has no spare, so the
removal needs --accept-deficit (a plain removal below target is refused,
X409205CLSRMD); quorum 2 of 2 still commits it, the slot frees, and a
replacement joining inside the fence takes it as a voter. That window has zero
fault tolerance, which is why status notes a fence with fewer than four nodes.
Superseded on the new line (Clay, 2026-09-30): with the automatic odd
voter count, removing the dead voter is a plain shc cluster rm, the two
survivors become one voter and one non-voter, and --accept-deficit goes
(new-line simplification §4.9).
Recovery from two losses is shc cluster repair --force on the survivor, only
after the lost nodes are confirmed destroyed.
Losing the whole home site leaves the far site with no voter. Whether repair --force can seat a non-voter as the single voter is not verified; a site-loss
runbook proven on a rig is a build step.
Two other layouts were weighed. One voter per site across three sites survives any single site loss and pays WAN latency on every write. A 2 + 1 split across two sites keeps quorum in the home site through a partition, as 3 + 0 does, and gives the far site a voter to recover from if the home site is destroyed; its cost is that raft may elect the far voter, after which every write crosses the WAN and leader-only work runs in the far site. 3 + 0 was chosen until the runbook proves recovery from a non-voter.
6. Implementation
Section titled “6. Implementation”6.1 Naming, and the lease rename
Section titled “6.1 Naming, and the lease rename”fence is the placement primitive. The volume-attachment I/O fencing is
renamed lease: the fence generation becomes the lease generation,
self-fence becomes lease loss. “Lease” already names nearby things, so the
renamed concept is always the attachment lease. volume_attachment already
carries owner_ref and leased_until for backup leases on the same row,
which fits; deployment_lease, volume_mount_lease and crons_lease are
unrelated and keep their names. The STONITH agent keeps its industry name
(Fencer becomes a STONITH agent, without the word fence).
The rename lands whole, with no aliases, in the clean line (6.3). Its surface, measured on both mains:
| surface | what changes |
|---|---|
| ce Go | 126 files contain “fence” (case-insensitive), 1,145 occurrences; 78 of them are “defence” and a few more are unrelated senses |
| shc Go | about 50 files use the volume sense (FenceGeneration, BumpFenceAndAttach, self-fence, fence_pending, Fencer, volume_fence*, assert-node-fenced) |
| shc docs | 32 files outside evidence/ and handoff/ name the volume-sense identifiers, not counting this one (64 contain “fence” in any sense) |
| schema | volume_attachment.fence_generation (ce migrations/20260819001215_auto.sql:1035); the fence_pending value of node.cluster_state (ce ent/schema/node.go:60), which the scorer’s candidate filter names |
| CLI and API | shc volume assert-node-fenced and /api/method/shc.storage.volume.assert-node-fenced; fence_generation in shc volume ls -o json (ce modules/volume/commands/list.go:36); openapi VolumeAssertNodeFenced{Request,Response} (docs/reference/api/openapi.json:14838-14891) |
| error codes | X409035VOLFNC (“volume … fence refused”, ce modules/storage/storageapi/errors.go:418) |
| subsystems and files | volume_self_fence, volume_self_fence_node, volume_fence_override.go; the FENCE-0..3 labels in move_strategy.go |
6.2 Decisions, ruled 2026-09-29
Section titled “6.2 Decisions, ruled 2026-09-29”The owner approved every recommendation below and added decisions 16 to 23. Decisions 9 and 10 were refined the same day on SHC-1058.
- Adopt the node addressing model (
2026-08-02-node-addressing-model.md) as the address vocabulary, with its Q2 answered “declare the address” and one amendment, the fifth roleunderlay(3.10). Its Q10 (voter removal) is answered by--rebalance; its Q12 (terraform suffrage) by the control-plane fence. Decided: adopted, with the amendment. - What does the cluster plane dial between sites? Mesh addresses, rather than site-to-site routing through gateway nodes or public addresses behind firewalls. Decided: mesh addresses, always (decision 17).
- Where does the Nebula CA live? Decided: in the vault; any voter signs.
- Do the fabric doc’s “retire Nebula” and “WAN membership over the fabric” follow-ons stand? Decided: withdrawn.
- Which kinds are built in? Decided:
node,provider,location; an unknown location islocation:unknown. - Where do colocation defaults live? Decided: on the socket; the plug
adjusts for its listed services; strictest wins unless loosened with a
reason; catalog files name only
node,locationornone; the six lint rules of 3.5. - What do relative rules mean? Decided: judged on eligible sets,
compiled to one derived fence per kind per move group, binding every
replica;
noneforms no edge. - Where do voters live? 3 + 0 in the home site, against 2 + 1 or one per site. Decided: the control-plane fence, defaulting to the genesis location, enforced at enrol.
- How many ingresses? First decided as one Traefik per location fence. Refined on SHC-1058: an edge per location fence in front of a Traefik on every node; routes and certificates come from the control plane; every Traefik carries every route; DNS follows placement.
- Which private resolver? First decided as a PowerDNS-local catalog app fenced to each site, with pfSense Unbound where a site has a pfSense edge. Refined: a resolver on every node answering mesh and fabric addresses from the replicated control plane, running in the builtin Traefik project (decision 26).
- SSO? Decided: one Keycloak in the home site for v1; far-site gated apps fail closed during a partition; a local-read forward-auth answer follows.
- Backups? Decided: a backup target per location fence, off-site, plus a second copy on a second provider.
- Group moves? Decided: explicit (
shc move --group) and offered by refusals; never automatic. X400956NDLOCL? Decided: lifted;--nodeand--rulego through the dispatcher.- Multi-location apps? Decided: one global entry point in v1, health-checked DNS next, no GeoDNS.
- Retire Docker Swarm. SHC’s one placement engine is the only scheduler; nodes run Compose or Podman as executors. Scheduler mechanics are SHC-1058.
- Nebula always on as the single node-addressing plane.
- The fabric carries container traffic; NetBird is for people only.
- No backwards compatibility. A clean line: one squashed initial migration, no in-place upgrade inside the product, no Swarm or LAN-addressing shims (6.3).
- Fences and rules, no free-floating placement settings. Every rule is
attached to a concrete object and travels with it. There are four
sources: resource, integration, deployment, and the scopes above the
deployment (tenant, refined to the whole
core/scopeladder). Rules narrow downward: all of them apply, a lower scope can never escape a higher scope’s fence, and loosening is explicit, recorded and allowed only with authority at the scope that set the rule. Explain prints every rule with its source (3.3). - Swarm compose keys keep their meaning (SHC-1058). Each is honoured or refused at render, never ignored; placement constraints become rules (3.7).
- Per-node compose projections (SHC-1058). A stack renders as one project per node it touches, and names resolve live through each node’s resolver (3.11, 5.7).
- Failover reuses the dead-source move, and a returning node starts nothing on its own (SHC-1058). The daemon starts SHC-managed containers only after the control plane confirms ownership and the lease is re-acquired (3.8, 5.8).
Decisions 24 to 28 were the questions the SHC-1058 follow-up left open. They were decided by agent on Clay’s standing delegation, 2026-09-29, reversible. Each took the recommended option; the rejected options are kept so a reversal knows what it is choosing between.
- Far-location volumes keep running through a WAN cut. A volume whose
reach holds no node of the control-plane fence is exempt from the
self-fence and is never taken over on lease expiry. A takeover needs a
STONITH confirm (the Hetzner fencer powers the server off and reads it
back, ce
modules/storage/providers/hcloud/fencer.go) or an operator’s--presumed-dead. There is no automatic failover inside a far site. Rejected: keeping today’s stop, which takes far-site stateful apps down for every cut (5.8), and a lease authority per site, a second consensus group per site that the 3 + 0 layout exists to avoid. - Automatic failover waits for the old node to stop itself. When
SHC-1058’s automatic failover reuses
--presumed-dead, it takes over a volume only after the self-fence window has passed.--presumed-deadalone has no dwell (5.8); the EE engine’s rebind dwell of at least 45 s is the model. - The per-node resolver is a container in the builtin Traefik project,
beside forward-auth, reading the cluster config directory. The builtin set
stays at four. Rejected: inside shcd, where every daemon restart or roll
drops DNS for every container on the node, and a fifth builtin, which
reopens the set ruled closed at four
(
docs/decisions/2026-08-18-builtin-vs-deploy-placement.md). - Rule authority.
platform:managesets and loosens rules at every rung. A tenant’stenant:managesets rules on that tenant and its environments, stacks and deployments, and loosens only rules set at those rungs. Nobody loosens a resource rule. Both roles exist in Keycloak today (cemodules/auth/keycloak_admin.go:2064-2066). - Public DNS for a location with no edge answers with one healthy node that holds a public address, chosen by the engine and re-pointed when it fails, until health-checked DNS lands. Rejected: an A record per public node, where a dead node fails a share of clients until it is withdrawn.
6.3 The clean break
Section titled “6.3 The clean break”The new line carries zero backwards compatibility. The rqlite and SQLite
control-plane migrations are squashed into one initial migration that already
holds the fence tables, node.provider and the lease rename. The product
cannot upgrade a cluster from today’s line in place, and it carries no shim
for Swarm or for LAN-shape addressing.
The migration covers bitspur only for now. The owner ruled on 2026-09-29 that promanager stays on its current line until he decides otherwise, so nothing in this plan touches the promanager estate. bitspur moves with a one-off, throwaway migration script that lives outside the product (not in shcd, and no compat code anywhere in it), runs once, and is deleted once bitspur is across and the owner has settled promanager’s path. It is tracked as SHC-1061. Two ways to write it:
- A. Convert in place. Stop the control plane, read today’s rqlite state, write it into the squashed schema, lay down the new line’s packages, bring Nebula and the fabric up, and re-place every stack through the new engine. Fast and needs no second set of servers. The risky part is the conversion itself: every table’s semantics has to map, and a half-converted cluster has no product to fall back on.
- B. Build a fresh cluster and move every app. Stand up the new line on new nodes (or on nodes drained out of the old cluster one at a time), then script backup on the old cluster and restore on the new one, app by app, in move-group order, re-pointing DNS as each lands. Slower and needs capacity, but the old cluster keeps serving until the last app is across.
Either way the script:
- has a dry run that prints every step, every stack in move-group order and every DNS change, and changes nothing;
- takes a verified backup of every stack and of the control-plane state before its first write, and refuses to start if any verify fails;
- has a rollback. For B it is re-pointing DNS back at the old cluster, which is still running. For A it is restoring the control-plane snapshot and the old packages, which is why A’s dry run must also prove the snapshot restores on a scratch node first.
B is the lower-risk path and matches the owner’s first statement of the break (fresh cluster, backup and restore). A is worth writing only if capacity for a second cluster is the blocker.
6.4 Phased build plan
Section titled “6.4 Phased build plan”The order below is superseded by the delete-first order in the new-line simplification (SHC-1092): 7.5 moves to the front, 0.1 shrinks to a baseline reset, 3.6 merges into SHC-1058’s E1, 7.3 into 8.1, and 3.5 into SHC-1058’s D3. The steps’ content stands.
Each numbered step is one ticket and one MR. Steps inside a phase are independent unless a dependency is named.
Phase 0. The clean line.
- 0.1 One squashed initial migration for the new line, carrying everything in 5.6 and the lease rename’s schema half.
- 0.2 The lease rename across ce and shc: identifiers, CLI verb, API route, JSON field, error text, docs (6.1). No aliases.
- 0.3 Node facts:
node.provider,--location/--provideron init and join, terraformlocation/provideronshc_nodeandmodules/core. - 0.4 Exclude
nebula0(anynebula*) from NIC auto-detect. - 0.5 Correct the stale docs (6.6).
Phase 1. Nebula as the addressing plane. Depends on 0.1.
- 1.1 The Nebula CA in the vault; any voter signs.
- 1.2 Nebula on from the first join; enrol over a lighthouse’s underlay address and hand out cert, CA and lighthouse list in the enrol response.
- 1.3 The cluster plane on mesh addresses: rqlite advertise, internal API,
fabric peers. The
underlayandinternalroles of the addressing model (its slices 2 and 3, amended). - 1.4 Lighthouses and relays on public-address nodes, pinned from terraform
(
shc_mesh, the addressing model’s slice 5);preferred_rangesper site LAN. - 1.5 WAN tuning from SHC-1049’s final numbers.
- 1.6
nebula-certstaging for Proxmox nodes.
Phase 2. Fences as data. Depends on 0.3.
- 2.1 Fence tables, the built-in and resource reconciler,
shc fence ls/show. - 2.2 Custom fences:
create,add-node,remove-node,rm,set, API, terraformshc_fence, join-time--fence. - 2.4 The rule store on
core/scope:placement_rule,shc rule add/ls/rm, terraformshc_rule, narrowing across the cascade, and the loosening authority check per scope. - 2.3
shc fence explain, reading the scorer’s fullDecision.
Phase 3. One placement engine. Depends on phase 2.
- 3.1
--ruleon install and update, deployfilerules:,rulesonshc_deployment, and the Phase-1 admission gate that refuses an empty eligible set by name. - 3.2 The scorer filters by the eligible set;
--nodeand--rulethrough the dispatcher;X400956NDLOCLremoved. - 3.3 Compose and Podman relocation through the dispatcher; the MOVE-5 clamp removed.
- 3.4
--ruleon move, refusals by name naming the rule’s scope. - 3.5 Drain reads eligible sets from the fence store.
- 3.6 Swarm placement keys (
deploy.placement.constraints,replicas,mode: global) become rules and placement at render; a constraint with no fence form is refused by name. - Scheduler mechanics are SHC-1058, which consumes this phase.
Phase 4. Storage. Depends on phase 3.
- 4.1 Pre-provision reach per volume connection (Cinder’s no-handle arm); the binder picks by target node.
- 4.2 A move leaving a volume’s reach auto-selects the target connection.
- 4.3 Volume failover respects the eligible set.
- 4.4 Volumes whose reach lies outside the control-plane fence are exempt
from the self-fence and from lease-expiry takeover; takeover only on a
STONITH confirm or
--presumed-dead(decision 24). Automatic failover waits out the self-fence before it takes over (decision 25).
Phase 5. Colocation. Depends on phase 3.
- 5.1
placementin the socket and plug schemas, the six lint rules, and the catalog sweep that satisfies them. - 5.2 Relative rules: derived fences, union-find move groups, refusals by
name,
--rule colocaterecorded on the edge. - 5.3
shc move --groupwith a write-ahead journal. - 5.4 External sockets:
placement.locationas a scored preference.
Phase 6. Control plane. Depends on phase 2.
- 6.1 The
control-planefence; enrol forces non-voter outside it. - 6.2
shc cluster statusreports voters against the fence. - 6.3 Site-loss runbook, including whether
repair --forcecan seat a non-voter, proven on a rig.
Phase 7. The fabric, and Swarm’s retirement. Depends on phase 1.
- 7.1 SHC-1055: fabric slice 2, nftables filtering and default-drop.
- 7.2 SHC-1056: fabric slice 3, the Podman adapter (depends on SHC-1055).
- 7.3 Traefik on the fabric: backends are fabric addresses from the control plane, read through the file provider; the Docker-label provider goes.
- 7.4 Fabric MTU 1220 under Nebula.
- 7.5 Remove the Swarm runtime,
--fabric swarm, the Swarm placement renderers, the Swarm label helper,converge-swarmand drain’s Swarm reads. Depends on 7.3 and phase 3.
Phase 8. Edges and DNS. Depends on phases 3 and 7 and on SHC-1058’s cluster config directory.
- 8.1 A Traefik on every node, fed from the cluster config directory (SHC-1058’s materialiser); the entry-point move removed.
- 8.2 A daemon-side ACME source behind the cert broker’s seam; certificates reach every Traefik and the broker’s app deliveries from it.
- 8.3 An edge per location fence: a proxy connection bound to a location, its pool that location’s healthy Traefik nodes; the DNS ladder picks the edge by the app’s location instead of refusing a second web connection.
- 8.4 The per-node resolver answering mesh and fabric addresses from the materialised state, a container in the builtin Traefik project beside forward-auth (decision 26).
- 8.5 Host and container resolver configuration forwarding the private zones to the node’s resolver; the NetBird routing-peer and nameserver-group runbook.
- 8.6 Health-checked DNS for edges.
- 8.7 Public DNS for a location with no edge: one healthy public node, re-pointed by the engine on failure (decision 28).
Phase 9. SSO and backups.
- 9.1
backup.repoper fence, off-site; a mirror per location; B2 as bitspur’s second provider. - 9.2 Forward-auth answered from the local replica with bounded staleness.
- 9.3 Location-loss restore runbook, proven on a rig.
Phase 10. Across. SHC-1061: the one-off migration script (6.3) and bitspur moved. promanager stays on its current line until the owner decides otherwise; the script is deleted once that is settled. A two-site proof rig (pve01 VMs and a far network with netem, the SHC-1049 shape) runs S1 to S10 end to end, including the WAN cut, and stays as a lane.
6.5 What stays out
Section titled “6.5 What stays out”Scheduler mechanics (SHC-1058), GeoDNS, Keycloak cross-site replication, pfSense edge HA, and any stretched database.
6.6 Stale docs this design found
Section titled “6.6 Stale docs this design found”- Fixed by SHC-1049:
docs/guide/operations/cluster-setup.mddescribed a Nebula-based join token and a temporary Nebula during join; the Go join is LAN-token only (modules/node/commands/join.go:39-47). - Fixed by SHC-1049:
docs/architecture/modules/nebula.mdsaid non-raft cluster traffic flows through Nebula; nothing SHC dials does. docs/guide/operations/node-rotation.md:419-449says there is no voter rebalance;--rebalance/PromoteBeforeDepartureandshc cluster set-voter-targetexist.docs/architecture/cluster/network-fabric.mddisagrees with itself: its header (:8-10) says slices 2 and 3 are design, its body (:35-38) says both landed, and its build inventory (:882) lists the Podman adapter as not built while ce runs podman stacks and admits them on--fabric shc(core/runtimecmd/runtimecmd.go,modules/fabric/types.go:98-106).docs/architecture/cluster/network-fabric.md:1083-1098, the Nebula retirement follow-on, is withdrawn by decision 4.docs/architecture/cluster/traffic-across-runtimes.mdcitescompose.yaml:410-412for the published ports; they are now at ceaddons/traefik/compose.yaml:732-733.docs/reference/configuration/config-keys.md:540still says split-horizon DNS was rejected; cemodules/dns/purpose.gorecords the 2026-08-20 re-ruling.- ce
modules/app/dns_private.go:105-128says the node table carries no overlay address;node.mesh_addressexists. - ce
modules/cluster/voters.gocalls the unsafe escapeshc cluster force_leader; the verb isshc cluster repair --force.
Superseded rather than stale, by the SHC-1058 refinements:
docs/architecture/cluster/network-fabric.md:262-268accepts that compose ignores Swarm-onlydeploy:keys; decision 21 honours or refuses each.docs/architecture/cluster/network-fabric.md:326-331,859bakes cross-node names at render time and re-renders connected stacks on a move; decision 22 resolves them live.docs/architecture/cluster/traffic-across-runtimes.md:797, “One Traefik, one node, one address”, gives way to a Traefik on every node (decision 9).
References
Section titled “References”- node addressing model: roles, provenance, the mixed shape, suffrage.
- swarm-native rebalance: the storage domain this generalises.
- placement scoring: the scorer and the location gap it names.
- storage constraint and move cost
- placement rulings 2026-08-14
- network fabric and traffic across runtimes
- node rotation and cluster setup
- enable backups
- bind/advertise split
- builtin vs deploy placement
- SHC-1049 (Nebula underlay proof), SHC-1055 and SHC-1056 (fabric slices 2 and 3), SHC-1058 (scheduler), SHC-1061 (moving bitspur across).