The new line, deleting first (design)
Status: open design (SHC-1092). This is the delete-first build order (section 7), and the epic is still in progress. Steps 1 and 2 are done: Swarm is deleted and
runtime: swarmis refused by name, and the baseline was reset, though step 2’s edit-in-place rule became an append-only rule on 2026-10-03 (cemigrations/embed.go), with the squash left for v0.1.0. Step 7 went the other way round: Nebula is deleted on next and mesh0 and shc0 replaced it before the cutover. Not built on next: forward-auth deciding from files (9.2, SHC-1163; it still asks shcd),shc forwardthrough the public route to a replica on another node (it resolves only on the daemon it reaches and answers not found, X404921FWDCNT; SHC-1692), and the cutover (step 14, SHC-1165). 3.14’s--hostand--portare refused on next (X400226FLGOLD) and--exposeis the one publication. It changes the ORDER of the build in the fences design (2026-09-29-multi-region-fences.md, SHC-1050, “§6.4”) and the scheduler design (2026-09-29-shc-scheduler.md, SHC-1058, “§8”), and shrinks or merges steps. It changes none of their rulings. Where a simplification brushes a ruling it’s listed in section 9 as a question, and the ruling stands until Clay answers.
Clay’s architecture rulings of 2026-09-30 and 2026-10-01 are folded in (SHC-1198). Each is stated as a rule with its date where it lands, and section 8 lists them all. Section 9 keeps only what is still open, with a recommended answer beside each.
The owner’s direction for the new line (2026-09-30): “Build it how you would build it and no legacy.” “Maximum simplicity. Simple interfaces, straightforward implementation, building it how you would build it if you’re building it from scratch.” No build-then-rip-out. And: simplification loses no feature. “I still want backups, I still want the repos.” Deleting Swarm removes a mechanism, never a capability; section 5 holds every capability to that.
Written against shc corp/next ab1d460e5 and ce corp/next 0b647c53 (both
equal to their mains at the cut). ce paths are in the ce repo; every other
path is in shc. LOC figures are cloc code lines (blank and comment lines
excluded) unless a row says otherwise. Since the cut, next has merged main
(shc 183e4edda, ce 4ede059c), which brings SHC-1079’s fence store (ce
fence and fence_member tables, modules/fence with its derive, explain
and fenceapi types, migration 20260930001929_fences.sql) and SHC-974’s
lease fixes. Step 2.1 (SHC-1105) builds on that store instead of beside it,
the contracts in section 4 reuse its types where they name the same thing,
and the baseline reset in step 2 folds its migration in.
1. The branch, and what it frees
Section titled “1. The branch, and what it frees”next was cut from main in shc and ce. main stays the old line and takes
fixes only, for the live estates. At cutover next becomes main and the old
main becomes release/v0.0. promanager is never touched.
The claim that a next push publishes nothing holds. Measured, not read
from comments:
- shc.
workflow:creates a pipeline for any ref (- when: always,.gitlab-ci.yml:283-306), but every job in the file and in.gitlab/ci/pulp-publish.ymlrules itself to the default branch, a tag or a schedule. A rule simulator over the merged job graph (both files and the pinnedcicdtemplatetemplates/shc-deployfile.yml@fa4fee78), with extends and!referenceresolved, finds 0 jobs for a push tonext, against 17 for a push tomainwith corp’s variables (15 unconditional, plus the Pulp and Pages jobs corp’sPULP_URLandSHC_PAGES_ENABLEDarm). No deb, image, terraform artifact, apps index, Pulp copy or site. - MR pipelines into
next. The template’sshc:generateandshc:deployalso matchmerge_request_event, but only withSHC_DEPLOY_ENABLED == "true", which corp’s shc project doesn’t set (6 project variables, none of them that one), and on an MR the generator deploys only areviewenvironment, which neither deployfile declares (deploy/marketplace.shc.yaml:269,deploy/license-broker.shc.yaml:176, bothprodonly).nextisn’t a protected branch either, so protected variables never reach it. - ce. Publishes nothing on any ref (its header says so and it has no
publish job). Its one job,
consumer/shc, runs on every branch push and compiles shc against the pushed ce. It defaults toSHC_REF: main, so a cenextpush compiles shcmain, which goes red the moment cenextdrops a symbol old shc uses. That isn’t a publish hazard, but every cenextpipeline would lie. Fix:SHC_REF: nexton cenext(one line, step N0 below). A branch with an open MR runs as an MR pipeline instead (ce.gitlab-ci.yml:64-68), with the same default. The DCO sign-off check in the same job reads MR commits, so cenextMRs needSigned-off-by:.
What next frees:
- No step waits for SHC-1093 or the
release/v0.0cut. The line-aware CI is still needed, but at cutover, not before the first new-line commit. - The squash is a command. ce already has a reset doctrine
(
migrations/embed.go: “Resetting the history isrm -rf migrations/*.sql && make migrate”), and the history is three files, 1,097 SQL lines, one of them the 1,057-line baseline. - The Swarm code can go first, because nothing on
nexthas to keep a live Swarm estate running. - The rig builds debs from
nextlocally (make package). If agents want a pipeline-built deb for the rig, SHC-1093 adds one job onnextthat keeps its deb as a job artifact and publishes nothing.
2. What gets deleted
Section titled “2. What gets deleted”Measured on next. “Whole” means the file exists only for Swarm or legacy and
goes entirely; “part” means the lines were counted from the functions or
blocks that go. shc keeps ce’s tests in tests/_ce/ (ce ships none), so ce
test lines are counted there once and not again under shc.
ce (production Go; ce ships no tests). Swarm-named files are 44 files, 7,803 lines, on their own.
| bucket | what | whole files | part (est. ±15%) | total |
|---|---|---|---|---|
| A | Swarm verbs, compose-loader reconciliation, file-config lift, Swarm backup capture/quiesce/replay (daemon/services_swarm_verbs.go, modules/app/swarm_compat.go, swarm_file_configs.go, daemon/backup_swarm_*.go, less about 150 lines of streaming helpers the compose capture shares, which move) | 4,417 | 1,216 | 5,633 |
| B | service and task reads (subsystems/impl/swarm_recorder.go, core/swarmclass, Swarm health projection and one-shots) | 1,560 | 256 | 1,816 |
| C | routing mesh and ingress (swarm_mesh, ingress_runtime) | 169 | 134 | 303 |
| D | overlay networks (swarm_network, shared-network shadow, provider network attach) | 427 | 252 | 679 |
| E | managers, join, quorum (modules/cluster/swarm_managers.go, duplicate reap, suffrage census, core/swarminfo) | 1,221 | 502 | 1,723 |
| F | placement translation (swarm_mobility, swarm_placement, placement replay, volume guard, storage/swarm_label.go) | 1,031 | 18 | 1,049 |
| H | one-shot upgrade shims (payloadref backfill, vdeployment dir, OIDC plug, stack name, package trust, cluster host) 1,091; compose-versus-Swarm orphan detection 295; scattered compat arms about 220 | 1,606 | ||
| ce total | 12,809 |
Plus 552 generated ent/ lines from the three Swarm columns on node. The
migration history (G) is 3 files and 851 code lines; the reset regenerates the
baseline, so it removes history rather than schema.
shc (code lines; the Go the ce-overlay tests exercise is counted under “ce’s tests” below).
| bucket | Go | Go tests | sh/bats | other |
|---|---|---|---|---|
A docker stack deploy (modules/app/swarm_driver.go:47, swarm_waves.go, daemon/wiring/services_swarm_edition.go) | 990 | 852 | 813 | |
B docker service/node inspect reads in drain and cordon (daemon/wiring/cluster_drain_constraints.go, drain_pin_gate.go, drain.go:1101) | 408 | 1,255 | 72 | |
C Swarm VIP upstream in traefik/render | 8 | 8 | ||
D overlay (TF provider fabric.go, overlay-dup-ip-check.sh) | 45 | 87 | 202 | |
E managers, join tokens, converge-swarm (modules/cluster/swarm_converge.go, swarm_removal.go, modules/node/swarm_tokens.go, enroll_handler.go:1156) | 1,053 | 2,914 | 138 | |
F Swarm pool capacity, advisor arm (modules/quota/enforcement.go:200, modules/placement/advisor.go:90) | 66 | 68 | ||
generated pkg/apiclient Swarm fields | 110 | |||
| terraform (Swarm ports, runtime enum in 14 modules), catalog Jinja branches (5), TS, apps bats | 42 | 170 | ||
H legacy: terraform state upgraders and migration boundary, the provider/CLI skew gate ssh_compat.go, legacy-shape-probe.sh | 1,197 | 1,670 | 186 | |
| shc total | 3,877 | 6,854 | 1,453 | 170 |
ce’s tests, which live in shc (tests/_ce, tests/unit): 23,725 lines
exercise the ce code above (77 whole files in tests/_ce) and go with it.
Totals. About 17,000 lines of production code (ce 12,809, shc 3,877
Go plus 170 other), 32,000 lines of tests (shc 6,854 Go and 1,453 sh/bats,
ce’s 23,725), and 552 generated ent/ lines. Against about 481,000 lines of
production Go (ce about 370,000 outside ent/, moving by a few hundred with
how nested ent directories are excluded; shc 111,091), that’s 3.5%. Deleting
is cheap; the cascade in section 3 is where the simplicity comes from.
What the measure found besides:
- The catalog has no Swarm-only keys. 104 of 116 compose files carry
deploy:, and it’sresourceseverywhere plus onereplicas(pulp). The five short-form port publishers were SHC-1081’s sweep, withdrawn on 2026-10-01:ports:in compose now only declares (3.14). docker stack deployis in shc, not ce (ce plugs it in throughSetSwarmDeployLane), so step 1 is two coordinated MRs, ce first.- The adapters stay. ce
modules/adapter(22,672 lines) imports estates from Portainer, CapRover, Dokploy and Easypanel, and 16 of its files read the source platform’s Swarm. That’s a product feature reading someone else’s Swarm, not SHC running one. - The lease rename is two senses. The attachment-ownership sense (about
450 occurrences in 57 ce files, 30 in 10 shc files) and the node-fencing
sense SHC-1050 §6.1 also renames (
fence_pending,assert-node-fenced,Fencer; about 380 in 36 ce files, 495 in 35 shc files). “Lease” already names the self-fence lease (LeasedUntil,LeaseExpired, about 350 uses in shc), which is the same object, so the rename converges on it; the unrelateddeployment_lease,volume_mount_leaseandcrons_leasekeep their names.
3. The cascade
Section titled “3. The cascade”Deleting Swarm doesn’t only remove its adapter. Each of these is code that exists because two schedulers, two network planes or two runtimes had to be reconciled, and it collapses when one side is gone.
3.1 Four runtimes, one executor
Section titled “3.1 Four runtimes, one executor”Today a deployment’s runtime is compose, swarm, podman or "", and
ce core/runtimecmd maps each to a binary and a label dialect (docker stack deploy writes com.docker.stack.namespace and never
com.docker.compose.project, so every project selector needs a Swarm arm).
With Swarm gone the remaining runtimes speak one dialect (podman-compose sets
the same compose labels, ce core/runtimecmd/runtimecmd.go:14-17), so the
switch becomes one Executor interface with one implementation per runtime
(4.7).
Ruled 2026-09-30 and 2026-10-01: the runtimes are docker, podman,
incus and apple, named for the engine that takes the compose file, and
all four are the architecture now. The Docker one is not called
compose. incus is driven through lxc/incus-compose (SHC-1199); apple is
its own executor over Apple’s container (SHC-1200). GPU, microVM
isolation and architecture are node capabilities, facts that become fences
and placement requirements, never runtimes; each executor turns a
capability into its own mechanism. No Docker API shim and no Proxmox
executor get built: Proxmox stays a node provider, SHC decides placement
and uses Proxmox APIs only as mechanisms, and Proxmox HA never manages an
SHC guest. pve02 and pve03 become bare Incus hosts (Debian 13 + Incus 7)
only after the bitspur cutover (ruled 2026-10-01); until then they stay
Proxmox and the incus runtime is proven in a VM.
The runtime evidence ladder goes with it. ce core/swarmclass,
core/swarminfo, modules/app/ingress_runtime.go:95 (RuntimeFromEvidence),
EffectiveRuntime and the
“runtime unsaid” policy exist so a "" row can’t fail open against a
Swarm-owned deployment (ce core/runtimecmd/runtimecmd.go:55-80: 33 of 83 rows on the measured
estate carried ""). On a fresh baseline the column is NOT NULL, there’s no
Swarm-owned deployment to protect, and there’s nothing to infer.
Runtime becomes a node fact (a node runs docker, podman, incus or apple;
SHC-1056’s mixed cluster), and --runtime docker|podman|incus|apple on
install becomes a placement requirement the fit filter checks, not a choice
of code path (Q2, ruled 2026-09-30).
3.2 Ingress: a Traefik per node, no mesh, no label provider
Section titled “3.2 Ingress: a Traefik per node, no mesh, no label provider”Today one system Traefik publishes 80/443 on one node. app.AmITraefikHost
gates the ingress sweep to that node (ce modules/app/ingress_materialize.go:88),
the entry-point move relocates it (ce modules/clone/move_ingress.go), the
edge’s pool narrows to one node, and Traefik reads Docker labels besides the
file provider. Under Swarm, published ports ride the routing mesh.
In the new line every node runs the per-node platform set (Traefik,
forward-auth, resolver) as a global deployment, and Traefik reads only its
own clusterfs section, traefik/current/dynamic (per-section since
2026-10-01, 4.4; SHC-1050 §3.13, SHC-1058 §3.16). What
goes: the Docker-label provider, the Traefik-host singleton and
AmITraefikHost, the entry-point move, the one-node pool, the ingress network,
and every mesh workaround. Published ports come from --port on the
deployment, never from the compose file (3.14, ruled 2026-10-01; this
supersedes the mode: host shape of SHC-1058 S8 and the SHC-1081 sweep).
Why a Traefik per node works now when the old line ran one on one node (Clay asked, 2026-10-01): the singleton existed because Traefik read Docker labels from the local engine, and Swarm’s routing mesh carried the published 80 and 443 to whichever node held it. On the new line Traefik reads only clusterfs files that are identical on every node, its backends are fabric pins reachable from any node, and certificates are issued once by the daemon-side ACME source (8.2) and delivered to every node. No instance is special, so every node can serve and losing one loses nothing.
Built once. SHC-1050 7.3 (“Traefik on the fabric”, still one Traefik) and 8.1 (“a Traefik on every node”) would build the singleton on the fabric and then replace it. The new line builds 8.1 directly (section 7).
3.3 Drain without docker service inspect
Section titled “3.3 Drain without docker service inspect”Drain reads constraints from docker service inspect and labels from docker node inspect (daemon/wiring/cluster_drain_constraints.go:38,51,95, consumed
by modules/cluster/drain_pin_gate.go:129, drain.go:1101 and
cordon.go:315) because Swarm owned placement. In the new line placement is SHC’s, so drain is: cordon the node,
then ask the engine to re-place every replica assigned there (Place with the
node excluded), in priority order, under the disruption budget. The journal,
the pin gate and the --ha budget stay. The constraint matcher and both
inspect reads go.
3.4 clusterfs replaces the materialiser family
Section titled “3.4 clusterfs replaces the materialiser family”SHC has “no file-sync mechanism, on purpose” (modules/license/distribution.go)
and instead grew one pattern many times: a replicated row, then each node
writes its own local file. The licence, the cluster CA and mTLS leaf, the
ingress route and cert files, the cert broker’s certserve homes, Nebula’s
static host map and lighthouse pins, and the fabric’s render-time
extra_hosts each have their own writer, their own atomicity and their own
reload.
clusterfs (SHC-1058 §3.24) makes that one mechanism: one materialiser, one
revision counter, an atomic swap per section with a root REVISION written
after every section is in (per-section since 2026-10-01), and each consumer
is a Section that writes its subtree (4.4). Each existing writer becomes a
section or goes. The fabric’s extra_hosts go outright, because the
resolver answers those names live (SHC-1050 decision 22).
3.5 Join is enrol
Section titled “3.5 Join is enrol”Today a join carries a Swarm join token beside the cluster token, Swarm
managers converge to three (converge-swarm, ce
modules/cluster/swarm_managers.go, modules/cluster/swarm_converge.go), node rows
carry swarm_node_id, swarm_role and is_swarm_manager, and Nebula is an
EE toggle (shc cluster mesh enable|disable) with a LAN-addressed mode beside
it.
In the new line a join is one enrol over the underlay: mTLS, the voter-slot
claim (forced non-voter outside the control-plane fence), and the Nebula cert,
CA and lighthouse list in the response (SHC-1050 §3.10). rqlite is the only
quorum. On the two-tunnel target (3.11) the response carries the node’s
WireGuard keys, fabric grant and endpoint candidates as one signed rqlite
row in place of the cert and lighthouse list (noted 2026-09-30). What goes:
Swarm tokens (shc node swarm-tokens), managers, converge-swarm and the
Swarm node facts in step 1; the mesh on/off toggle and the LAN-addressing
mode in 1.2 and 1.3.
3.6 Placement: an engine that can refuse
Section titled “3.6 Placement: an engine that can refuse”The EE scorer “returns a node or nothing; it cannot refuse”
(ce modules/stack/placement_seam.go:95-105), “Swarm does not get a score”,
the dispatcher falls back to “the node that answered”, --node naming another
node on compose is refused (X400956NDLOCL), a compose move always copies (MOVE-5), and deployment_node
exists because “an unpinned swarm install stamps no placement constraint at
all” (ce ent/schema/deployment_node.go:10-14).
In the new line Place returns bindings, pending, or a refusal naming the
fence that emptied the eligible set (4.1). assignment rows replace
deployment_node. X400956NDLOCL, the MOVE-5 clamp, the “no opinion” path
and the Swarm placement renderers (ApplyDeployPlacement,
ApplyVolumePlacement, ApplyVoterExcludes) go.
3.7 One backup capture path
Section titled “3.7 One backup capture path”Swarm-owned volumes need their own capture, quiesce and replay (ce
daemon/backup_swarm_capture*.go, backup_swarm_quiesce.go,
backup_swarm_replay*.go, backup_swarm_volume_probe.go) because a Swarm
task can run anywhere. In the new line the assignment names the node holding
each volume, so capture always runs there through the one compose/podman path
(ce daemon/backup_volume_capture.go), and SHC-1058 §3.25 adds the
stopped-app path. One catch for step 1: the compose capture path calls the
streaming helpers that live in backup_swarm_capture_stream.go
(captureStreamVerbs, pullCaptureStream, captureChunkRPC,
captureEndRPC; backup_volume_capture.go:59,175,309), so step 1 moves
them into a neutral file before it deletes the rest. Backups, restore, verify, browse, the mirror and the second provider are
untouched as features.
3.8 Restarts belong to the node agent
Section titled “3.8 Restarts belong to the node agent”Today every service without restart: gets unless-stopped (ce
modules/app/restart_default.go), so dockerd restarts containers at boot
before ownership is known, and the leader-only reconcile cron can repair only
its own node (X200924RSTRMT). In the new line the render stamps restart: "no" and the node agent restarts crashes (SHC-1058 S1). Swarm’s restart
policy semantics come back natively (deploy.restart_policy read by the
agent), and the per-node reconcile gap disappears because every node repairs
itself.
3.9 The schema is a baseline, edited in place until v0.1.0
Section titled “3.9 The schema is a baseline, edited in place until v0.1.0”The history is already one baseline plus two small files. The new line
resets it once (after the Swarm delete drops the Swarm columns) and then
every schema step edits the baseline in place (rm migrations/*.sql && make migrate) until v0.1.0 ships. No new-line database exists outside the rig
before cutover, and the rig re-inits freely. So step 0.1 no
longer has to carry every table of SHC-1050 §5.6 and SHC-1058 §6 before the
code that uses them exists: each step brings its own tables into the baseline.
After v0.1.0 migrations append normally. An old-line database is refused by
name, never migrated; the one-off script (SHC-1061) reads the old schema from
outside the product.
3.10 Compose keys: implemented where the feature lives, Swarm vocabulary translated or refused
Section titled “3.10 Compose keys: implemented where the feature lives, Swarm vocabulary translated or refused”The Compose Specification owns deploy: (replicas, mode, resources,
restart_policy, update_config, rollback_config, endpoint_mode,
labels). Those aren’t Swarm legacy; they’re the app format. Each is
read by the step that implements its feature: replicas by replicas (B4),
mode: global by global mode (B5), restart_policy by the node agent (A4),
update_config/rollback_config by rolling updates (D2), endpoint_mode: dnsrr by the resolver (8.4), deploy.labels by the render as container
labels (traefik.* ones become file routes), top-level configs/secrets
by staging (E2). There’s no separate translation layer.
What’s left is vocabulary that only means something to Swarm’s scheduler or
network: deploy.placement.constraints (node.hostname, node.labels.*,
node.role), placement.preferences, max_replicas_per_node,
endpoint_mode: vip, networks with driver: overlay, and device or
generic_resources reservations. One render lint handles all of it, with
SHC-1058 §3.23’s outcome for each key: vip and devices are refused; the
placement keys and driver: overlay have a native form
(rules: [fence=node:X], spread=K, spread=node,max=n, a deployment
network on the fabric) and §3.23 translates them. Ruled 2026-09-30 (Q1):
they are translated, and the render warns, naming the native form to
write. No catalog app uses any of them. Ports left this list on 2026-10-01:
this section refused short-form and mode: ingress ports and counted on
SHC-1081 to sweep five apps to mode: host; now every ports: form only
declares and --port on the deployment is the one publication (3.14), and
the sweep is withdrawn. The catalog sweep over all 106 apps stays as the
proof nothing is dropped silently (SHC-1050 decision 21: honoured or
refused, never ignored).
3.11 One fabric
Section titled “3.11 One fabric”ce cluster_fabric.mode is swarm | shc | none. The new line has one
fabric, so the mode column, --fabric and the Swarm overlay branch go. This
section first said “nothing on one node”; since 2026-10-01 the per-tenant
bridges and the tenant policy apply on every node, a single-node estate
included (3.13), and one node simply has no cross-node tunnel. Today the
fabric rides Nebula (MTU 1220), and that carrier can take the cutover.
Ruled 2026-09-30: the target is two kernel WireGuard devices per node,
coordinated by shcd, never double wrapped. mesh0 carries the node plane
(rqlite, the mesh listener); shc0 carries container traffic with each
peer’s fabric grant as its AllowedIPs. Separate keys and ports, one
handshake each, both dialling the same discovered underlay endpoint, so
each packet is wrapped once. SHC coordinates and writes no tunnel: shcd
owns the peer table in rqlite (keys, grants, endpoint candidates ranked
same-site LAN, then public, then relay through a public node), a local
cache of that table for boot since raft rides the mesh, the estate’s DNS
name as the fallback for public nodes, and a per-peer MTU from an RFC 8899
style probe with an MSS clamp on shc0. A Mac runs the same control code
over embedded wireguard-go. No new protocol, no Rust, no eBPF, no GPU
crypto. Nebula goes at stage 2 (SHC-1201). Ruled 2026-10-01: the mesh lands
after the bitspur cutover, which rides today’s Nebula plus fabric at MTU
1220; before it only the design, the rig measurement and a default-off
stage 1 may merge, Nebula work in flight finishes, and no new Nebula scope
is added.
3.12 Three doors into shcd, and no fourth
Section titled “3.12 Three doors into shcd, and no fourth”Ruled 2026-09-30. shcd listens on three doors and nothing else. What collapses: forward-auth’s sidecar secret and its realm calls, and every “module credential” that would have let a container into the daemon.
| door | who | proof |
|---|---|---|
the unix socket, /run/shc/shc.sock (0660 root:shc; SHC_SOCKET_PATH overrides it) | the local shc CLI: root or the shc group | the socket file’s permission |
| the mesh listener, on the node’s mesh address | other nodes: the leader relay, rqlite, node RPC | the tunnel binds a node’s address to its key (a Nebula cert today, the WireGuard key on mesh0); headers carry only the principal a node relays |
| the public listener | the Mac CLI, Terraform and the Console, over NetBird today and the mesh client later; forward-auth’s gate call | a Keycloak bearer; the gate call carries the user’s own JWT and nothing else |
The rules that keep the doors honest:
- No container reaches the node plane. The mesh address, rqlite, the mesh listener and the unix socket are unreachable from any container in any tenant; the policy compiler emits that as a device match, never as a per-app rule (3.13).
- No container reaches the internal API, ever. It is node RPC with node identity. Clay asked twice whether modules should use it to simplify; the answer stayed no.
- Reads are files. What a module needs arrives in its own clusterfs
section, bind-mounted read-only: the resolver gets
resolver/, Traefiktraefik/, forward-authauth/, never the whole tree, with a revision directory per section so a bind mount never pins a stale revision (ruled 2026-10-01). forward-auth’s realm-for-host and the gating declarations become files, so its sidecar secret goes. - The gate decision. Target: forward-auth links the Go decision package ce uses and decides locally from the JWT, validated against Keycloak’s keys, and the declarations file, zero calls per request, through daemon restarts and partitions. Until then its call goes to the local public listener carrying only the user’s JWT (SHC-1163).
- No module credential; identity is the pin. A caller’s identity is its
source address from the pin ledger, protected by the bridge antispoof
guard; the API port is dropped when it arrives over
shc0, so a source is always a local pin. Master’s platform environment, which is the system tier, has full scope by position; a tenant deployment that declares a broker use gets its verbs by position; the firewall opens the API port only to those. With forward-auth on files that set is empty on today’s estate.
3.13 Policy: the tenant is the boundary
Section titled “3.13 Policy: the tenant is the boundary”Ruled 2026-09-30. The policy unit is the deployment (tenant, environment, stack); nothing finer, no per-container, per-label or L7 rule. What collapses: the cross-tenant socket and plug edges today’s compiler still allows, and the idea that a connection grants reachability.
| flow | verdict |
|---|---|
| inside a deployment | open: it is one network |
same tenant, to a service that declares ports (ports: or expose:) | open, on exactly those container ports, by default |
| same tenant, to a service with no port declaration | blocked: private to its deployment |
| across tenants | blocked, and a plug across tenants is refused when declared, never compiled |
| the system tier: Traefik and forward-auth to routed backends; any deployment to the OpenObserve collector and to the per-node resolver | open, as derived accept classes |
| any container to the node plane | blocked (3.12) |
The system tier is master’s platform environment. Connections (sockets and plugs) are wiring: names, credentials, config injection. They are not a permission. Policy is derived from declarations; there is no firewall to author, and the compiler stays nftables on Linux with a pf output for a Mac node. Every ported service gets a fabric pin, so a node’s grant is sized for its ported services. The accepted trade: a compromised app can reach every ported service in its own tenant, which then rests on those services’ own credentials. Making “fully blocked” true on one node is the structural work: a fabric bridge per tenant per node, which under an open-within-tenant default is simply the tenant’s network, and ingress-bridge-keyed rules, built under SHC-1202.
The address plan (ruled 2026-10-01). The fabric lives in
10.128.0.0/10. Each node gets a fabric bridge per tenant. The per-node
grant is a genesis knob that defaults to /20 (32 tenants of 126 pins on a
node, 1,024 nodes). Node-local deployment networks come from 10.64.0.0/10
and the control plane (mesh addresses, VIPs) from 10.192.0.0/10. New
clusters only: the rig’s 10.84.0.0/16 is not renumbered.
No service mesh (closed, 2026-09-30). No sidecars, ambient proxies, per-workload certificates, per-request L7 policy or automatic service-to-service tracing, and no Istio, Linkerd, Consul Connect or Cilium mesh. Retries, circuit breaking and weighted rollout exist only as Traefik middlewares on the ingress path. Never re-propose.
3.14 Publication: --host and --port, never the compose file
Section titled “3.14 Publication: --host and --port, never the compose file”Ruled 2026-10-01. In an app’s compose, ports: and expose: both mean
only “this service listens on these container ports”; a ports: entry is
treated as expose: and its host side is not honoured. An app never
publishes a host port by itself. An exposed port becomes reachable three
ways, none of them in the authored compose:
- inside the tenant, by policy (3.13): a pin, a resolver name and an nftables accept on those ports;
--hoston the deployment binds a service to a public hostname through Traefik, as a route file;--porton the deployment publishes on the node, or through a proxy-capable connection’s load balancer withconn@(cemodules/app/port_mappings.go, grammar[<conn>@]<external>[:<service>|:<container>[:<container>]][/udp], a trailing!removes). This is the only case that renders aports:entry, long formmode: host, on the replica’s compose entry.
A service with no declared port has no pin and is private to its
deployment. What collapses: the SHC-1081 shape (five apps swept to
long-form mode: host in compose) and S8’s reading of mode: host as host
publication; the ten catalog apps with ports: blocks stop
self-publishing. Not ruled: where an app’s default mappings live, and
--port on a service with more than one replica (section 9). Neither is
built until ruled.
4. The new core’s interfaces
Section titled “4. The new core’s interfaces”Small on purpose: each contract is what one spine owns and the others call. Types are sketches; the spine that builds one may rename a field, not widen the method set without a note on SHC-1092. Package homes follow SHC-1058 S9: fence store, reconciler, explain and the single-node engine in ce; the multi-node engine loop, rescheduling and failover in shc.
Shared vocabulary (ce core/placement, extending the existing leaf):
type NodeID stringtype FenceID stringtype DeploymentRef struct{ Tenant, Environment, Stack string }type ReplicaRef struct { Deployment DeploymentRef Service string Ordinal int}type NodeSet []NodeID // sorted, unique: no map iteration anywhere in placementtype Generation uint64The words (ruled 2026-09-30 and 2026-10-01): the catalog thing is an
app (apps/<name>/, app.yaml), never a package. One app installed
for one tenant in one environment is a deployment, the running unit and
its own network; stack names only the axis (DeploymentRef.Stack,
-s/--stack, the clusterfs path). The ladder is app, deployment, service,
replica (deployment, service, ordinal), compose entry
<service>-r<ordinal> in a node’s projection, container. Existing code is
not mass-renamed for it; new prose, docs, CLI help and error text use these
words.
4.1 Placement engine
Section titled “4.1 Placement engine”Pure and deterministic, like today’s scorer: no I/O, no clock, no map
iteration in the path. Install, update, scale, move, drain, failover and
volume provisioning all call Place. Explain takes the same snapshot and
the same demand and runs the same code, so it can’t disagree with what
Place did. For a running deployment the caller rebuilds the demand from its
retained deployment_spec (A5) and the last placement_event.
// ce core/placementtype Engine interface { Place(s Snapshot, d Demand) Decision Explain(s Snapshot, d Demand) Explanation // Decision.Trace rendered, plus the move group}
// Snapshot is one consistent read at one control-plane revision.type Snapshot interface { Revision() uint64 Nodes() []Node // condition, capacity, committed requests, runtime, location, provider Members(f FenceID) NodeSet Rules(ref DeploymentRef) []Rule // every rule with its source: resource, integration, deployment, scope rung Assignments(ref DeploymentRef) []Assignment}
type Demand struct { Deployment DeploymentRef Replicas []ReplicaDemand // service, ordinal, requests, mode, single writer, runtime requirement Exclude NodeSet // drain and failover exclude the source Priority int}
type Decision struct { Bind []Binding // replica -> node, with the generation it replaces Pending []Pending // fits nowhere now: per-node reason Refusal *Refusal // eligible set empty: the rule, its scope and the fence that emptied it Trace Trace // eligible set per replica, rules with provenance, scorer terms}CE implements Engine with the fence filter and one node; shc implements it
with the scorer (modules/placement.Score, weights unchanged). The existing
PlacementAdvisor seam goes.
4.2 Assignment store with epoch
Section titled “4.2 Assignment store with epoch”The only writer of desired placement. A new engine takes an epoch. A bind is one guarded statement batch that commits only while its epoch is current, each assignment’s generation still matches, and every target node still fits once the batch’s requests are added to the commitments it re-reads inside the same batch (SHC-1058 §2.2, §3.3). So a deposed engine’s write matches no row, and two decisions about different replicas can’t both spend the same free capacity.
// ce modules/schedulertype Epoch uint64
type Assignments interface { Lead(ctx context.Context, self NodeID) (Epoch, error) Bind(ctx context.Context, e Epoch, b []Binding) error // ErrDeposed, ErrGenerationMoved, ErrNoFit ForNode(ctx context.Context, n NodeID, linearizable bool) ([]Assignment, error) Observe(ctx context.Context, n NodeID, obs []Observed) error // what ran, never what should}4.3 Node agent
Section titled “4.3 Node agent”Runs on every node, CE included. Reads clusterfs, renders nothing, and owns
restarts, the boot gate and the node lease. The boot gate is Confirm: after
a boot the agent starts nothing, the per-node platform set included, until it
holds its node lease for this boot, clusterfs has been re-materialised from
the control plane (not the revision left on disk), a linearizable read of its
own assignments (ForNode(..., true)) has succeeded, and every attachment
lease it needs is re-acquired at its expected generation (SHC-1050 decision
23, SHC-1058 §3.9 and S17). Until then Apply stops what
the control plane says moved and starts nothing.
// ce modules/nodeagenttype Agent interface { Confirm(ctx context.Context) error // the boot gate; ErrPartitioned keeps it closed Apply(ctx context.Context, rev clusterfs.Revision) error // converge to rev; idempotent; ErrUnconfirmed before Confirm Report(ctx context.Context) (Report, error)}
type Report struct { Node NodeID BootID string Confirmed bool // the boot gate has opened for this BootID Lease NodeLease // renewed by quorum write; TTL 35 s Conditions []Condition // pressure reasons, runtime health Observed []Observed // per replica incarnation: state, readiness, restarts Revision uint64 // the clusterfs revision last applied}Before the agent runs any entry, four checks hold (ruled 2026-10-01): the
revision is whole (every section written and swapped, REVISION written
last, 4.4); the entry’s hash matches the spec digest on its assignment, so a
corrupted disk, a materialiser bug and a hand edit all stop it; the replica
is still this node’s at the current generation and engine epoch, with the
lease held for anything exclusive; and every cross-node depends_on target
is ready. Afterwards it reports the revision it applied and each replica’s
state, which is what shc update waits on.
4.4 clusterfs materialiser
Section titled “4.4 clusterfs materialiser”One writer of node-local files. A build reads the revision counter, builds
every section from one read, re-checks the counter (a change discards the
build), writes each section’s <section>/rev-<n>/ whole with its own
REVISION file, fsyncs, swaps each section’s current, and writes the root
REVISION last, the completion marker the node agent reads. This replaced one
rev-<n>/ tree behind one current on 2026-10-01: a platform container
bind-mounts only its own section’s stable parent, so it never pins a stale
revision and never sees another section (3.12).
// ce modules/clusterfstype Revision struct { ID uint64 Root string // /var/lib/shc/clusterfs; section s of this revision is <Root>/<s>/rev-<ID>}
type Section interface { Name() string // "node", "traefik", "certs", "resolver", "auth", "fabric", "nebula", "licence" Render(ctx context.Context, tx Reader, node NodeID, w Writer) error}
type Materialiser interface { Build(ctx context.Context) (Revision, error) Current() (Revision, error) OnSwap(func(Revision)) // Traefik SIGHUP, resolver reload, Agent.Apply}Three words, ruled 2026-10-01. Render is Jinja over the app’s files and
the deployment’s settings, and it happens once, on the leader, at install
or update; the result is a per-replica spec stored in rqlite with a digest
(SHC-1113’s deployment_spec is its home). The Jinja context gains a
replica object (ordinal, location, node); compose ${VAR} is not
used for per-replica values, because it resolves once per file.
Enrichment is the part of that render that adds what the author did
not write: the pin, dns and dns_search, labels, the SHC_* variables,
CA trust, telemetry, the volume suffix. Materialising is each node
writing the stored specs and the derived Traefik and resolver sections into
its own revision directory. A Section.Render call here derives files from
rows; it never runs Jinja and never enriches. Nodes never render. So a
daemon upgrade changes no projection until an explicit update; a spec that
uses only replica.ordinal moves between nodes as an assignment change
with no re-render, and one that uses replica.node or replica.location
is re-rendered by the leader for the new placement. clusterfs is never an
input: nothing that decides or renders reads it. Any voter must be able to
fetch an app’s files at the pinned version, because a new leader renders
the next update (section 9).
4.5 Resolver
Section titled “4.5 Resolver”A small binary in the builtin Traefik project beside forward-auth (SHC-1050
decision 26). Answers from its own section, resolver/current/zones, only,
never dials the control plane, and keeps answering from the last revision
during a partition.
// shc cmd/shc-resolver (Go side of the container)type Resolver interface { Resolve(q Question) (Answer, error) // <svc>.<deployment zone>, <node>.nodes, routed hostnames}type Answer struct { Addrs []netip.Addr // ready replicas' fabric addresses, or mesh addresses for nodes TTL time.Duration Revision uint64}Ruled 2026-09-30: a few hundred lines of Go on miekg/dns, one per node, not
CoreDNS. It answers <service>.<stack>.<environment>.<tenant>.shc.internal
with the ready replicas’ pins, nearest first, TTL 5 s;
<node>.nodes.shc.internal with a node’s mesh address; and a routed
hostname with the ready edges of the app’s location, empty where the
location has none (ce modules/resolver, zone shc.internal). Everything
else is forwarded. Each container gets it as dns plus dns_search [<deployment zone>, <environment zone>], so web and proxy.postgres both
resolve; wiring that into containers per runtime belongs to the executor
(4.7).
4.6 Fabric policy
Section titled “4.6 Fabric policy”Everything one node needs to join the container fabric at one revision. The
applier exists (ce modules/fabric/wireguard.go) and so does the nftables
compiler (ce modules/fabric/policy), whose model needs more than placement:
the integration edges, the port declarations, the ingress backends, and the
pin ledger (ce modules/fabric/policy/model.go:105,133). The input says so.
Since 2026-09-30 an edge is wiring, never a permission: the ruleset is
derived from tenancy and port declarations (3.13), and the data plane is
kernel WireGuard, routes, bridges and nftables with no per-packet code of
ours (3.11).
// ce modules/fabrictype Input interface { placement.Snapshot Edges() []Edge // socket/plug edges: wiring (names, credentials), never permission (3.13) Backends() []Backend // routed hosts -> ready replicas, for the per-node Ingress class Pins() []Pin // pinned service addresses per node grant, with the ports each declares}
type Policy interface { Plan(in Input, n NodeID) (NodePlan, error) // grant, per-tenant bridges, peers, pins, compiled ruleset}type Applier interface { Apply(ctx context.Context, p NodePlan) error}The fabric’s plan reaches the node as the fabric clusterfs section. Since
2026-10-01 the node’s grant is the genesis knob of 3.13 (default /20; this
sketch first said /24), the plan carries one bridge per tenant on the node,
and its peers are mesh addresses on today’s Nebula carrier and discovered
underlay endpoints on shc0 at stage 2 (3.11).
4.7 Runtime executor
Section titled “4.7 Runtime executor”One interface, one implementation per runtime, chosen by the node’s
runtime: docker (docker compose), podman (podman compose), incus
(lxc/incus-compose with its healthd sidecar, SHC-1199) and apple (Apple’s
container, which has no compose tool, so the executor does the
translation itself, SHC-1200). The node agent is its only caller. Ruled
2026-09-30 and 2026-10-01: the interface stays runtime-neutral and nothing
above it may assume docker. Its input is the projection: one compose entry
per replica, rendered and enriched once by the leader (4.4), carrying the
author’s healthcheck, volumes and networks, the enriched dns and
dns_search, and a ports: entry only where --port publishes on this
node (authored ports are declarations, 3.14). Turning each into the
runtime’s own mechanism (the engine’s probe, a CDI or Incus gpu device, a
container exec on the declared interval for apple) is the executor’s job,
never the render’s.
// ce core/executor (replaces ce core/runtimecmd)type Executor interface { Up(ctx context.Context, p Project) error // restart: "no" already stamped Down(ctx context.Context, ref ProjectRef, o DownOpts) error Ps(ctx context.Context, ref ProjectRef) ([]Container, error) Exec(ctx context.Context, c ContainerRef, cmd []string, io IO) (int, error) Events(ctx context.Context) (<-chan Event, error) // die, health; feeds the restart supervisor}Exec never puts a job’s environment on an argv (SHC-1176): values reach the process through a mode-0600 env file in a private temp dir, deleted after the call, or on stdin, never as -e K=v on the engine CLI or export lines in an sh -c script, since /proc/<pid>/cmdline is world-readable.
4.8 Volume reach and attachment lease
Section titled “4.8 Volume reach and attachment lease”Reach answers before provisioning which nodes a volume connection can serve
(SHC-1050 §3.6; VolumeLocationReporter already answers without a handle for
eight of nine providers). The attachment lease is today’s self-fence, renamed
(SHC-1050 §6.1).
// ce modules/storage/volumeapitype Reach interface { Reach(ctx context.Context, conn ConnectionID) (NodeSet, error)}
type AttachmentLease interface { // Acquire with the holder's own generation re-acquires and keeps it (the // boot gate). A different node takes over only by naming the generation it // expects and a Takeover proof: a STONITH confirm, --presumed-dead, or a // lease expired past the single-writer stop window on a watchdog node // (SHC-1050 decisions 24, 25; SHC-1058 §3.10, S16). A takeover bumps it. Acquire(ctx context.Context, v VolumeID, n NodeID, expect Generation, proof *Takeover) (Generation, error) Renew(ctx context.Context, v VolumeID, g Generation) error // every 5 s, TTL 15 s Release(ctx context.Context, v VolumeID, g Generation) error // Lost fires after two failed renews, and the agent stops the owner's // containers. It never fires for a far-location volume, one whose reach // holds no control-plane fence node (decision 24): that one keeps running // through a WAN cut and is never taken over on expiry alone. Lost() <-chan VolumeID}The stop after Lost does not depend on the engine answering (SHC-1182).
The node keeps a ledger of every container it saw start (id, deployment
labels, cgroup) under its state home. When the engine cannot list or stop a
volume’s writers inside the stop’s budget, the stop kills their recorded
cgroups itself: cgroup.kill, else SIGKILL of every pid in the cgroup,
repeated until none is left. It reports done only once no process remains
in them and no running container on the node is missing from the ledger, so
a failed kill leaves the volume unstopped and the next tick retries. The
kill fires only once the lease has lapsed and waits out a renewal still in
flight until the start bound. Neither step calls the engine or reads the
cluster (ce subsystems/impl/volume_lease_kill.go). The ledger is fed for
docker-class runtimes and for podman through the docker API client (events
and inspect; podman on its docker-compatible socket, its containers under
machine.slice/libpod-<id>.scope), and for incus through the executor: a
start is recorded at once and a floor pass lists every instance, with the
cgroup cut at the top-level lxc.payload.<name> on unified cgroup v2 only.
Apple has no host cgroup a lease-loss stop can kill. Every binder-bound
volume is leased, so binding one there, including read-only, is refused
with X409050VOLNKP before the volume is provisioned.
Ruled 2026-10-01: no replicated volumes in the first cut. Local volumes, restic, and failover by restore; LINSTOR comes later as a per-site capability, and reach and the attachment lease stay in the design for it.
4.9 Control-plane suffrage
Section titled “4.9 Control-plane suffrage”The voter count is automatic and odd (Clay, 2026-09-30): the largest odd
number no greater than the member nodes of the control-plane fence, capped at
five. One eligible node gives one voter; two give one voter and one
non-voter; three and four give three; five or more give five. Nodes are
promoted and demoted automatically as they join or leave, one change at a
time and never below a quorum of the current voters. A node that’s down but
still a member keeps its vote until it’s removed, because shrinking the voter
set during a failure would cut tolerance rather than add it. rqlite fixes
suffrage at join, so a promotion or demotion is a remove and rejoin with the
new suffrage, the move PromoteBeforeDeparture already makes (ce
modules/cluster/voter_rebalance.go:179). The runtime target (ce
modules/cluster/voter_target.go) and shc cluster set-voter-target go;
the slot ledger (voter_grant.go) stays as the mechanism.
This replaces SHC-1050 §5.9’s recovery from one lost voter. There, a
three-node fence needs --accept-deficit to remove the dead voter
(X409205CLSRMD refuses a plain removal below target) and then runs on two
voters. Here the target follows membership, so the removal is a plain
shc cluster rm: two members remain, Reconcile demotes one survivor, and
the fence runs on one voter and one non-voter, which tolerates losing the
non-voter where two voters tolerate nothing. A replacement joining inside the
fence brings it back to three. --accept-deficit and X409205CLSRMD go.
Recovery from two lost voters is unchanged (repair --force).
// ce modules/clusterfunc VoterTarget(eligible int) int // largest odd <= eligible, capped at 5; 0 for 0
type Suffrage interface { Reconcile(ctx context.Context) (Change, error) // at most one promotion or demotion per call}The consequence, plainly: with two nodes and one voter, losing the voter stops every lease-held (stateful) app on the survivor, because attachment and node leases renew through the control plane and a lone non-voter has no quorum to renew against. The one exception is a far-location volume (decision 24, 4.8), whose owner keeps running. Stateless containers on the survivor keep running too, but nothing reschedules or changes until the voter returns. A third node in the control-plane fence makes it three voters and gives automatic tolerance of any single node failure, with no datacenter requirement beyond fence membership (SHC-1050 §3.9 decides eligibility; the count inside the fence is no longer a knob). Where the voters sit matters less than it seems: raft commits on a majority, so while the leader and the other near voter are up, a far third voter adds no commit latency. If raft elects the far node leader, every write crosses the WAN until leadership moves back (SHC-1050 §5.9’s 2 + 1 trade-off).
Nine contracts, 14 Go interfaces and 36 methods in all, counting the
helpers (Snapshot, Section, fabric Input, Applier). Nothing else in
the new core is an interface until a second implementation exists.
5. Feature parity inventory
Section titled “5. Feature parity inventory”Every user-facing capability ce and shc have on next today, found from the
CLI tree (docs/reference/cli, 311 shc_*.md pages), the daemon API
(docs/reference/api/openapi.json, 324 routes), the terraform provider (19
resources and data sources), the ent schema (45 files) and the catalog (106
apps). S means today’s implementation rides a Swarm mechanism; I
means it doesn’t. Every S row names what replaces the mechanism and the step
that delivers it.
The code column says what happens to the implementation. REUSED: carries
over unchanged, or loses only a Swarm arm or a name. MODIFIED: the
existing code stays and changes where it’s fed from or where it runs (routed
by assignment, read from clusterfs, filtered by fences). REWRITTEN: the
behaviour needs code that doesn’t exist today, or replaces a Swarm or dockerd
behaviour outright. Of 60 rows, 20 are REUSED, 33 MODIFIED and 7 REWRITTEN,
and the 7 are all scheduler behaviour Swarm used to supply: replicas, rolling
updates, restarts, rescheduling, global mode, service names and per-node
reconcile. By lines, about 96% of today’s production Go survives the delete, and the
edits land in a small part of it: install, render and ingress in ce
modules/app, drain, the health reconciler, the move path and the scorer.
Backups (ce modules/backup, 41,235 lines), storage (16,923), vault
(11,563), DNS (10,099), auth (9,715), proxies (5,473) and the adapters
(22,672) carry over as they are, less the Swarm arms step 1 removes. The new
code is the scheduler Swarm used to be, an estimated 20,000 to 30,000 lines
across the engine loop, assignments, node agent, clusterfs, resolver,
replicas, global mode and rollouts.
Apps and their lifecycle
| capability | entry points | today | code | new line provides it by | step |
|---|---|---|---|---|---|
| install, update, uninstall, start, stop, restart | shc install, update, uninstall, start, stop, restart; shc.stack.* | S for Swarm-run deployments (docker stack deploy), compose driver otherwise | MODIFIED | Place, then the node agent runs the node’s projection through Executor | 1, A3, 3.2, B3 |
dry run, validate, config, inspect, services, status, shc ls | shc validate, inspect, status, ls; shc.stack.dry_run, inspect, services, status | S for Swarm-run deployments (service and task reads) | MODIFIED | assignments plus the agents’ Observed | A3, A1 |
| logs, exec, shell, port-forward, container restart and mount | shc logs, exec, shell, forward, container; ce modules/portforward/swarm_resolver.go, daemon/services_forward_swarm_route.go | S: Swarm task resolution across nodes | MODIFIED | routed to the replica’s node from its assignment; not built for shc forward through the public route, which resolves only on the daemon it reaches (SHC-1692); the node-local route proxies to a named node | A3 |
| scale and replicas | --scale on shc install and shc update (three shapes; shc scale goes, ruled 2026-10-01), deploy.replicas | S: Swarm spreads tasks; compose puts every replica on one node | REWRITTEN | replicas across nodes, one compose entry per replica, spread derived from the shape | B4 |
| rolling update and rollback | shc rollback, rollback.list; deploy.update_config | S: Swarm’s updater; compose recreates together | REWRITTEN | rollout row, start-first incarnations, rollback to the retained spec | A5, D2 |
| restart a crashed container | restart:, deploy.restart_policy | S: Swarm task restart; dockerd unless-stopped | REWRITTEN | the node agent’s restart supervisor | A4 |
| run elsewhere when a node dies | none (automatic) | S: Swarm reschedules swarm tasks; nothing for compose | REWRITTEN | grace clocks, rescheduling, dead-source failover of a replica with a volume, and single-writer takeover on a STONITH confirm or the lease proof from a watchdog node (SHC-1150) | C1, C2, C3 |
| one copy per node | deploy.mode: global (no catalog app) | S | REWRITTEN | global mode | B5 |
| health, readiness, liveness | healthcheck (93 apps) | I (compose), S for swarm tasks | MODIFIED | readiness feeds endpoints; liveness restart | D1 |
depends_on across services | 51 apps, 50 with conditions | I inside one node | MODIFIED | held across nodes by the agent | B3 |
| move, dead-source move, repoint, clone, rename, protect, reclaim, recover, interrupted operations | shc move [--presumed-dead], repoint, clone, protect, unprotect, reclaim, recover, interrupted, finish-forward | I, with a Swarm-only relocation fast path (MOVE-5) | MODIFIED | the dispatcher relocates either runtime; --rule; group moves | 3.3, 3.4, 5.3 |
| choose a node | --node | S: a node.hostname== constraint; compose takes only the local node and refuses another (X400956NDLOCL, ce modules/app/runtime_guard.go:170) | MODIFIED | Place with fence=node:X | 3.2 |
| placement in an app | deploy.placement.constraints (no catalog app) | S | MODIFIED | translated into rules with a warning naming the native form (Q1, ruled 2026-09-30) | E1 |
| deployfiles and the CI template | shc deploy, deploy/*.shc.yaml, cicd | I | MODIFIED | unchanged, plus rules: | 3.1 |
| vapps | shc install <host>/<site>, vdeployment | I | REUSED | unchanged | none |
| jobs, hooks, cron, schedules, tasks | shc job, cron, schedule, task; ce modules/job | S partly: job contexts resolve Swarm tasks (modules/job/context_service.go) | MODIFIED | run on the node the assignment names; jobs leave a lost node | A3, C5 |
| configs and secrets in deployments | top-level configs (12 apps), secrets | S for Swarm-run deployments (file binds lifted into Swarm configs, ce modules/app/swarm_file_configs.go); compose binds them on its one node | MODIFIED | staged per node; secrets from the vault into RAM-only mounts, with a per-node sealed cache of external-provider values only (ruled 2026-10-01) | E2 |
| reconcile | shc reconcile, shc cluster reconcile | I, leader’s own node only | REWRITTEN | every node’s agent converges itself | A4 |
Tenancy, config, identity, licence
| capability | entry points | today | code | new line provides it by | step |
|---|---|---|---|---|---|
| tenants, environments, scopes, 5-level config, contexts | shc tenant, config, ctx; core/scope | I | REUSED | unchanged; rules reuse core/scope | none, 2.4 |
| vault, secrets, key rotation, cold start | shc vault (13 verbs) | I | REUSED | unchanged; the Nebula CA joins the vault | 1.1 |
| quotas and usage | shc quota; shc.quota.* | I; capacity gates have no caller | MODIFIED | gates called by Place | 3.2 |
| users, roles, grants, launcher layout, service clients, login | shc user, auth; /api/user/* | I | REUSED | unchanged; rule authority on platform:manage / tenant:manage | 2.4 |
| licence | shc license (9 verbs) | I, own materialiser | MODIFIED | a clusterfs section | A2 |
| branding, console, portal, launcher, marketplace, MCP | shc branding, portal, mcp; console UI | I | REUSED | unchanged | none |
| HUD | shc hud (x9s) | I | MODIFIED | gains the :fences, :rules and :placement views | F1 |
Repos and catalog
| capability | entry points | today | code | new line provides it by | step |
|---|---|---|---|---|---|
| repos | shc repo add, ls, sync, update, rm | I | REUSED | unchanged | none |
| catalog and addons | shc catalog, addon | I | REUSED | unchanged | none |
| import from Portainer, CapRover, Dokploy, Easypanel | shc adapter load, ls, migrate; ce modules/adapter (22,672 lines) | I: 16 files read the SOURCE platform’s Swarm, not SHC’s | REUSED | unchanged; step 1 must leave these reads alone | none |
| publish the apps index, debs, provider, site | CI on main and tags | I | REUSED | unchanged until cutover, then split by line | SHC-1093 |
| integrations: sockets and plugs | shc integration, *.socket.yaml, *.plug.yaml | I; cross-node names ride the Swarm overlay or extra_hosts (S) | MODIFIED | resolver names; colocation rules; wiring only, never a permission, and a cross-tenant plug is refused at declaration (3.13) | 8.4, B3, 5.1, 5.2, SHC-1202 |
Storage and backups
| capability | entry points | today | code | new line provides it by | step |
|---|---|---|---|---|---|
| storage connections, CephFS, NFS | shc storage (20 verbs), shc connection | I | MODIFIED | reach per connection; the binder picks by node | 4.1, 4.2 |
| volumes, snapshots, rebind, volume mounts | shc volume (10 verbs), shc.volume.* | I | REUSED | lease rename | 3 |
| volume failover and STONITH | EE ticker, Hetzner STONITH agent | I | MODIFIED | inside the eligible set; far-location exemption; the EE volume ticker, off by default, still rebinds a volume through its STONITH seam, and the replica failover pass reads the same agent per node (SHC-1150) | 4.3, 4.4, C2 |
| backups: create, schedule, retention, verify, browse, mount, restore, export, import, groups, keys | shc backup (27 verbs), retention, schedule | I, with a Swarm-only capture, quiesce and replay path | REUSED | one capture path on the volume’s node; stopped-app backups | 1, E3 |
| mirror and a second provider | nightly off-provider mirror | I, leader only | MODIFIED | per-location off-site repo and mirror; B2 | 9.1 |
| recover and restore-as-move | shc recover, backup+restore move | I | REUSED | unchanged | none |
| backup repository sync and relocation | shc backup sync, shc backup relocate | I | REUSED | unchanged | none |
| writable backup mounts | shc backup mount, shc.mount.* sessions with commit and discard (ce modules/backup/mount) | I | REUSED | unchanged | none |
| deployment export and import | shc export, shc import (ce modules/export, exporter-based) | I | REUSED | unchanged | none |
Networking, DNS, SSO, certificates
| capability | entry points | today | code | new line provides it by | step |
|---|---|---|---|---|---|
| node-to-node addressing | --nic, internal_address | I, LAN; Nebula optional | MODIFIED | mesh0, the two-tunnel target, carries the cluster plane on next; Nebula is deleted (3.11) | 1.1 to 1.4, SHC-1201 |
| container traffic across nodes | overlay networks | S: the Swarm overlay | MODIFIED | the shc fabric on shc0 (3.11) | 7.1, 7.4, B3, SHC-1201 |
| service names across nodes | service DNS | S: Swarm VIP and DNS | REWRITTEN | the per-node resolver answering one VIP per service (SHC-1238; S7’s refusal was overturned on 2026-10-02) | 8.4 |
| HTTP and TLS ingress | builtin Traefik | S for Swarm-run deployments (Swarm provider, mesh); one Traefik on one node | MODIFIED | a Traefik per node from clusterfs, an edge per location | 8.1, 8.3 |
| published TCP and UDP ports | --host and --port on the deployment; ports: blocks in 10 apps | S for Swarm-run deployments (the routing mesh); compose publishes on its one node | MODIFIED | --expose is the one publication on next and --port is refused (X400226FLGOLD); ports: in compose only declares (3.14, ruled 2026-10-01) | E1 |
| edges and load balancers | shc.provider.*, ce modules/proxy (10 backends) | I, one edge per cluster | MODIFIED | an edge per location | 8.3 |
| mesh status, diagnose, lighthouses, cert rotation | shc cluster mesh, shc nebula | I | REUSED | shc cluster mesh over mesh0; the leader-signed peer table replaced lighthouses and certs (SHC-1201, before the cutover) and shc nebula is gone | 1.2, 1.4 |
| fabric status | shc cluster fabric status | I | REUSED | unchanged, one mode | 7.1 |
| DNS: public and private, dual planes, autozone, reap, takeover, validate | shc dns; /api/dns/* | I | MODIFIED | the public answer follows the location; the private answer is every Traefik’s mesh address | 8.3, 8.7 |
| NetBird and pfSense connections | shc connection | I | REUSED | nameserver groups and routing peers runbook | 8.5 |
| SSO, Keycloak, OIDC, gating, LAN posture | /api/auth/*, ipAllowList | I | REUSED | unchanged; the gating declarations are also written as a clusterfs file for forward-auth (3.12) | 9.2 |
| forward-auth | the sidecar in the Traefik project (forward-auth/) | I, one sidecar on the one Traefik node | MODIFIED | a sidecar beside every Traefik, fed realms and gating declarations as files, its gate call carrying only the user’s JWT (3.12) | 8.1, 9.2 |
| certificates: ACME, cert delivery, edge certs, CA, internal rotation | shc cert, shc ca, shc cluster cert | I, ACME on the one Traefik | MODIFIED | daemon-side ACME feeding every Traefik and the broker | 8.2 |
Cluster and observability
| capability | entry points | today | code | new line provides it by | step |
|---|---|---|---|---|---|
| init, join, leave, rm, forget, repair, preflight, host set | shc cluster, shc node | S: a join is also a Swarm join; managers converge | MODIFIED | one enrol onto mesh0 with a leader-signed peer table | 1, 1.2 |
| voters | shc cluster status, set-voter-target, voter-target | I | MODIFIED | automatic odd count inside the control-plane fence; the target knob goes (4.9) | 6.1, 6.2 |
| drain, undrain, cordon, uncordon | shc node drain | S: docker service inspect and docker node inspect | MODIFIED | drain calls Place with the node excluded | 1, D3 |
| node health and conditions | shc node health, status, ls | S partly: the Swarm health observer | MODIFIED | the node lease and the agent’s Report | A1 |
| voter rebalance on departure | shc cluster leave --rebalance, shc cluster rm --rebalance | I | MODIFIED | candidates only inside the control-plane fence | 6.1 |
| events, audit, notifications, metrics, telemetry, collector, doctor, conditions, support bundle, search | shc events, audit, notification, doctor, support, search | I | REUSED | unchanged; placement events | 2.3 |
| terraform | 15 resources, 4 data sources | I | MODIFIED | shc_fence, shc_rule, rules on shc_deployment, location/provider on shc_node, lighthouse pins on shc_mesh | 0.3, 1.4, 2.2, 2.4, 3.1 |
| leader | shc leader | I | MODIFIED | the engine rides the leader (1 s poll) | A3 |
Nothing in this inventory is dropped. Mechanism-only surfaces go with their
mechanism, because they manage Swarm itself rather than serve a user:
shc cluster converge-swarm, shc node swarm-tokens, the
shc.network.swarm_status route, and --runtime swarm / --fabric swarm.
Removing them loses no capability.
The cutover gate carries parity. SHC-1058 §7 gains a thirteenth condition:
every row of this inventory works on the rig. That gate (SHC-1165) is what
next has to pass before it becomes main.
6. The one-time migration
Section titled “6. The one-time migration”The migration lives outside the product, so ce and shc stay free to make
breaking changes with nothing compat baked in (SHC-1050 §6.3, decision 19).
It’s a throwaway script in its own repo (or bitspur/ops on gitlab.com), run
once per estate and deleted once the estate is across. Today that’s bitspur
only; promanager isn’t touched. It’s SHC-1061.
The only thing that crosses the line is app data: database dumps, volume archives and the restic snapshots that hold them. Nothing reads the old control-plane schema except the old line’s own CLI and API, and nothing in the new line reads anything the old line wrote except that data.
Read side. Through the old-line CLI and API only, as JSON: tenants and environments, scope config, repos (definitions and credentials) and every app payload a deployment pins, private and local ones included, every deployment with its full spec (app digest, image digests, vars, overrides, port and host mappings, posture), connections and grants, socket and plug edges, vault entries key by key, DNS records and their owners, backup repos, schedules and retention, users’ launcher layouts, the licence.
Write side. Through new-line CLI and API calls only: shc tenant,
shc config set, shc repo add and the payloads, shc connection connect,
shc vault writes, installs pinned to the same app digest and image
digests with the same vars and plug bindings (the same app_version isn’t
enough: an app’s contents can change under one upstream version, ce
ent/schema/deployment.go:54), shc backup schedule set, and a data restore
into the freshly installed deployment. The first update onto a new-layout
package renames 509 var keys (SHC-1360, SHC-1748), so the script translates each
persisted override through
var-renames-shc-1360.tsv before
that update; an old name is otherwise kept and read by nothing.
The archive is what crosses. A sealed backup is one .bak in the restic
repo (ce modules/backup/types.go:751) holding backup.tar in the canonical
layout: .shc/manifest.json, app/, deployment/, secrets.vlt,
data/<mount>/, data/@<converter>/ for database dumps, and hostbind/*.tar
kept tar-in-tar so uid, gid and mode survive (ce modules/backup/archive.go:26).
Restore today needs a backup row and its manifest (ce
modules/backup/restore.go:210). The .bak is an ar container of two members,
an outer manifest.json and backup.tar (ce modules/backup/archive.go:522),
and import reads the outer one (ce modules/backup/actions.go:1049,1250). So
per deployment the script pulls the .bak from the old repo, rewrites both
manifests, the outer one and .shc/manifest.json inside backup.tar, into
the shape the new line accepts (runtime, node and stack-record fields; volume capacity, access
mode and host ownership carried over, ce modules/backup/types.go:329; for a
pod with x-shc-scale.stateful: true, the manifest’s ordinals and the
<volume>-r<ordinal> data names, without which restore refuses with
X409896BKPORD),
drops deployment/ and secrets.vlt (the new line renders its own and the
vault moved key by key), reseals, and hands it to the new line’s
shc backup import, then restores it into the deployment it installed. The
product never reads an old-line manifest; the script owns that conversion.
Order. A dry run first: it renders every deployment under the new line, lists every refusal (Swarm vocabulary, an empty eligible set), prints move groups in order and every DNS change, changes nothing, and stops on any refusal. Then a verified backup of every deployment and of the control plane; any failed verify stops the run. Then, on the fresh cluster, the master tenant first (postgres, keycloak, openobserve), then tenants in move-group order. The shape, ruled 2026-10-01: the fresh cluster is VMs on both Gorilla boxes, pve02 and pve03, under Proxmox, docker runtime at cutover; the move is app by app with both estates live; each app’s data arrives by restoring a fresh restic backup taken right before its move, never by copying volumes off compose01, so the move proves the backups; compose01 stays untouched as the rollback for two weeks after the last app moves, switching back being a DNS change plus starting the old deployment, and is retired after that, never upgraded in place. Still SHC-1061’s to settle: the order of apps, how the shared databases and mailcow move, and what happens to the rig VMs on those boxes. For each deployment:
- Install and restore the first backup while the old copy keeps serving, then a health gate on the new copy through the new cluster’s own Traefik (a Host-header probe), before any DNS moves.
- Write barrier. Cut the old copy’s ingress (its routes answer 503) and
let in-flight work drain, then take the final backup through the old
line’s normal running-app path, whose hooks and converters need the
containers up (ce
modules/backup/cli_runner.go:184-198; stopped-app capture is new-line work, E3). Verify it, stop the old copy, and restore the backup into the new copy. This is the deployment’s only downtime. An app with background writers that ingress doesn’t stop (crons, workers) is listed by the dry run and gets its own quiesce step. - Health gate again, then DNS handover and the edge re-point for its hosts.
- The old copy stays stopped, not removed, until the soak ends: two weeks after the last app moves (ruled 2026-10-01).
DNS handover. The new cluster is built as a new generation of the same
cluster name (same slug and heritage), which is the case the existing
takeover exists for: it adopts a sibling generation’s ownership claim and
refuses a foreign one (ce modules/dns/takeover.go:46-88). It changes the
ownership claim only, never the targets, so the script calls it and then the
new line’s ordinary DNS reconcile re-points the records. There’s no CLI verb
for it today (ce modules/app/commands/dns.go:61 registers ls only), so the
script calls the API (/api/dns/takeover).
Rollback. Before a deployment’s handover it’s one command: start the old
copy and restore its ingress; nothing public changed. After it, the script
stops the new copy first, so only one writer exists, then takes the names
back with the old cluster’s takeover (it’s a sibling generation too). Because
a takeover moves ownership and never targets (ce modules/dns/takeover.go:63),
the script then runs the old cluster’s DNS reconcile to point the records
back, re-points the edge, and starts the old copy. Writes the new copy took since the handover are lost
unless the script restores a new-line backup into the old copy, which holds
only while the archive format is unchanged. Hence the soak before the old
cluster stops, with its disks kept until Clay signs off.
Format assumptions that must hold, because they’re the whole interface between the lines:
- The restic repository format and the
.bak/backup.tarlayout above, including the converter directories and the tar-in-tar host binds. If a new-line step changes any of it, the script converts, never the product. - Each converter’s dump format is what the new line’s restore reads
(
pg_dumpandmysqldumpoutput), with the same engine major on both sides, and apps install at the same app digest and upgrade afterwards, so each app’s own data schema matches. - The manifest fields the script can’t derive from the old API: volume capacity, access mode and host ownership.
- The old repo’s restic password reaches the script from the old vault; it becomes a new-line backup key only if the repo is kept.
- The DNS ownership claim (
heritage=shc,shc/owner=..., generation) keeps its shape, so the new generation’s takeover recognises the old claim. - The licence is domain-bound, so the new cluster activates the same domain’s token.
Tickets. SHC-1061 carries the script, its dry run and the bitspur move. Its criteria already cover the dry run, verified backups, master tenant first, move-group order, health gates, the edge re-point, the soak, the script living outside the product and promanager untouched; this section adds the write barrier, the manifest rewrite, the digest-pinned installs, the takeover-then-reconcile DNS handover and the rollback to its scope. SHC-1123’s rig rehearses the script against a small old-line cluster before bitspur. SHC-1165 condition 9 proves every bitspur deployment’s backup restores on a new-line scratch node, which is the format check above. No new ticket is needed.
7. Build order, deleting first
Section titled “7. Build order, deleting first”7.1 What goes, merges or shrinks
Section titled “7.1 What goes, merges or shrinks”| step (ticket) | fate | why |
|---|---|---|
| 7.5 remove Swarm (SHC-1166) | moves to the front as step 1; its eight dependencies drop | nothing on next runs Swarm; deleting first means later steps never write Swarm arms |
| 0.1 squash (SHC-1094) | shrinks to a baseline reset after step 1, plus the edit-in-place rule | the reset is one command (3.9); each step brings its own tables |
| 0.2 lease rename (SHC-1095) | stays, after step 1 | renames only code that survives |
| 3.6 constraints become rules (SHC-1120) | merged into E1, canceled | E1’s render handles the Swarm placement keys with the rest of the Swarm vocabulary, in one place (3.10, Q1) |
| E1 honour or refuse every Swarm key (SHC-1158) | shrinks to the Swarm-vocabulary lint and the 106-app sweep; translate and warn (Q1, ruled 2026-09-30) | each compose-spec key is read by the step that implements its feature (3.10) |
| 7.3 Traefik on the fabric (SHC-1125) | merged into 8.1, canceled | building the singleton on the fabric and then replacing it is the churn Clay ruled out (3.2) |
| 3.5 drain reads the fence store (SHC-1119) | merged into D3, canceled | step 1 deletes the inspect reads; drain then calls Place once, in D3 (3.3) |
| 2.3 explain (SHC-1108) and B2 explain v2 (SHC-1136) | merged; B2 canceled, 2.3 builds the one explain | explain v1 then v2 is build-then-replace; Explain is part of the engine (4.1) |
| 3.2 scorer filters (SHC-1116) and B1 fit (SHC-1135) | merged; B1 canceled, 3.2 builds Place | filter, fit, score, bind is one function (4.1) |
| B5 global mode (SHC-1146) and 8.1 Traefik per node (SHC-1137) | dependency inverted | the platform set is a global deployment from day one, so B5 comes first and 8.1 feeds its Traefiks |
| 7.2 Podman adapter (SHC-1056) | shrinks to the Podman Executor on the fabric’s pinned subnets | the render-time extra_hosts wiring in its criteria is replaced by the resolver (decision 22) |
| 1.2a join extends the mesh (SHC-1078) | on next, merged at a8af897c8, and the base of 1.2 | its converge, cert ledger and leave-revokes stay: 1.1 moves the CA and ledger to the vault and nebula_mesh, 1.2 adds the enrol-time cert and deletes the toggle; the old line never ran the opt-in mesh, so no backport |
| 6.1 control-plane fence (SHC-1121) | grows: the voter count becomes automatic and odd, with automatic promotion and demotion; the target knob goes | Clay, 2026-09-30 (4.9) |
| B4 replicas (SHC-1145), B5 global mode (SHC-1146) | split in time: engine half in step 6, working across nodes after step 9 | cross-node names need the resolver and projections; the platform set needs the per-node Traefik |
| C2 failover (SHC-1150) | dependency added on 4.1 and 4.4 | failover picks targets by reach and must honour the far-location exemption |
| §7 gate (SHC-1165) | becomes the cutover gate for next to main, with the parity condition | |
| line-aware CI (SHC-1093) | moves to cutover; plus ce SHC_REF: next now (N0) | next publishes nothing (section 1) |
| SHC-1081 port sweep | withdrawn as a shape | ports: in compose only declares; --port on the deployment is the one publication (3.14, ruled 2026-10-01) |
| incus and apple executors (SHC-1199, SHC-1200), the two-tunnel mesh (SHC-1201), the tenant policy model and per-tenant bridges (SHC-1202) | added 2026-10-01 | the rulings of 3.1, 3.11 and 3.13; each lands beside the spine it extends (7.2) |
Everything else in §6.4 and §8 stays as written, one ticket each.
7.2 The revised order
Section titled “7.2 The revised order”Spines as the epic defines them. Numbers in brackets are the old step names.
N0. next hygiene. ce consumer/shc compiles shc next on a ce next
push (SHC_REF: next). Part of SHC-1093.
Delete (one lane, back to back, before any spine writes code):
- [7.5] Delete Swarm (SHC-1166). ce and shc: the Swarm runtime and
docker stack deploy, everydocker service/node/swarmread and write, the Swarm fabric mode and overlay, join tokens, managers andconverge-swarm, the placement renderers and label helper, the Swarm compose loader and file-config lift and mobility pin, Swarm backup capture, the Swarm health observer and route forwarder, the runtime evidence ladder, their CLI verbs, API routes, terraform arms, bats suites and docs. It keeps what the compose path shares (the backup streaming helpers, 3.7) and the adapters’ reads of other platforms’ Swarm. Aruntime: swarmrequest is refused by name. Two MRs, ce first, then shc on the new pin. Tree compiles, unit tests green, a one-node compose install still works. - [0.1] Baseline reset (SHC-1094).
rm migrations/*.sql && make migrateafter step 1 drops the Swarm columns; the edit-in-place rule written intomigrations/embed.go; an old-line database refused by name. The one-shot upgrade shims go with the history they upgraded (cecore/payloadref/backfill.go, the vdeployment directory, OIDC plug, stack-name, package-trust and cluster-host migrations), because a fresh baseline has no old rows for them to fix. - [0.2] Lease rename (SHC-1095). Whole, no aliases, on the smaller tree.
- [0.3, 0.4, 0.5] node facts (SHC-1096),
nebula*NIC skip (SHC-1097), stale docs that survive step 1 (SHC-1098). Parallel with step 3.
Build (spines start once step 2 is on next; each owns its interfaces from
section 4):
- Node truth. A2 clusterfs (SHC-1110) first, because 8.x and the agent
read it; A1 node lease (SHC-1109); A3 assignments with epoch and the node
agent running single-replica installs through the
Executor(SHC-1111); A4 restarts and boot gate (SHC-1112); A5deployment_spec(SHC-1113), the home of the per-replica rendered specs (render once, 4.4); A6 watchdog (SHC-1114). - Fences and engine. 2.1 (SHC-1105), 2.2 (SHC-1106), 2.4 (SHC-1107);
3.1 admission and
--rule(SHC-1115); 3.2Placewith filter, fit and score, absorbing B1, binding through A3’s store (SHC-1116); 2.3Explain, absorbing B2 (SHC-1108); 3.3 relocation (SHC-1117); 3.4--ruleon move (SHC-1118); 6.1 the control-plane fence with the automatic odd voter count (SHC-1121, 4.9) and 6.2 status against it (SHC-1122); then the engine half of B4 replicas (SHC-1145: one compose entry per replica and the three--scaleshapes, SHC-1058 §3.6) and B5 global mode (SHC-1146), B6 (SHC-1147), B7 (SHC-1148). B4 and B5 bind and place here; they work across nodes only once step 9 delivers the resolver and projections (B4) and the per-node Traefik (B5’s platform set), and their tickets close then. - Nebula. 1.1 CA in the vault (SHC-1099), 1.2 on from genesis (SHC-1100), 1.3 cluster plane on mesh addresses (SHC-1101), 1.4 lighthouses and relays (SHC-1102), 1.5 (SHC-1103), 1.6 (SHC-1104). The two-tunnel mesh (SHC-1201) lands after the cutover; before it only its design, the rig measurement and a default-off stage 1 merge, and no new Nebula scope is added (3.11, ruled 2026-10-01).
- Fabric. 7.1 default-drop (SHC-1055), 7.2 Podman executor (SHC-1056),
7.4 MTU (SHC-1126); the incus executor (SHC-1199) and the apple executor
(SHC-1200), the runtime-name change landing first (3.1); the tenant
policy model, the per-tenant bridges and the address plan in
modules/fabric/policy(SHC-1202, 3.13). - Edges and DNS. 8.1 Traefik per node from clusterfs, absorbing 7.3
(SHC-1137, after B5’s engine half and A2); 8.2 daemon ACME (SHC-1138); 8.3 edge per
location (SHC-1139); 8.4 resolver (SHC-1140, miekg/dns, 4.5); 8.5 forwarding and NetBird
runbook (SHC-1141); B3 per-node projections (SHC-1144, after 8.4 and 8.5;
one entry per replica,
dnsanddns_searchenriched at render); 8.6, 8.7 (SHC-1142, SHC-1143). - Render. E1 Swarm-vocabulary lint and the 106-app sweep, absorbing
3.6 (SHC-1158; translate and warn, Q1; the lint can land any time after
step 1, the sweep closes once B4 and B5 honour
replicasandmode: global);ports:read as a declaration and--portas the one publication (3.14; SHC-1081’s sweep is withdrawn); E2 configs and secrets with the per-node sealed cache of external-provider values (SHC-1159, ruled 2026-10-01); E3 stopped-app backups (SHC-1160). - Storage. Phase 4 (SHC-1127 to SHC-1130): reach and the binder’s choice by node (4.1, 4.2), failover inside the eligible set (4.3), and the far-location exemption (4.4, after 6.1). No replicated volumes in the first cut (4.8, ruled 2026-10-01).
- Failure and rollout. C1 (SHC-1149); C2 (SHC-1150, after 4.1, 4.4, A5 and A6); C3 to C5 (SHC-1151 to SHC-1153); D1, D2, D3 drain on the engine absorbing 3.5, D4 (SHC-1154 to SHC-1157).
- Colocation, surfaces, SSO, backups. Phase 5 (SHC-1131 to SHC-1134), F1 (SHC-1079), F2 (SHC-1161), 9.1 to 9.3 (SHC-1162 to SHC-1164; 9.2 is forward-auth on files, 3.12), 6.3 (SHC-1124).
- Rig and cutover. The rig (SHC-1123) from step 7’s 1.2 on; the cutover
gate (SHC-1165) with the parity condition; SHC-1093’s line-aware CI; then
nextbecomesmain, oldmainbecomesrelease/v0.0, and SHC-1061 moves bitspur by path B onto fresh VMs on both Gorilla boxes, app by app from restic, with compose01 kept untouched as the rollback for two weeks and then retired (section 6, ruled 2026-10-01). After that cutover pve02 and pve03 become bare Incus hosts (3.1) and the two-tunnel mesh lands (3.11).
The rig milestones on the epic (M1 to M5) keep their meaning; M3 now proves the per-node Traefik directly instead of a fabric singleton first.
8. Rulings this respects
Section titled “8. Rulings this respects”- Fences are the placement primitive; every rule is attached to an object
(SHC-1050 decisions 5 to 7, 20).
Snapshot.Rulescarries provenance. - Nebula is the node-addressing plane, always on; the fabric carries containers; NetBird is for people only (decisions 16 to 18). 1.2 and 1.3 delete the mesh toggle and the LAN mode instead of keeping them beside Nebula. Superseded as the target on 2026-09-30: two kernel WireGuard tunnels coordinated by shcd, Nebula gone at stage 2, after the cutover (3.11); Nebula stays today’s carrier, so 1.2 and 1.3 still delete the toggle and the LAN mode.
- One placement engine with explain, used by every placement decision (4.1).
- The control-plane fence, enforced at enrol, with the voter count automatic and odd, 1, 3 or 5 (decision 8, refined by Clay on 2026-09-30; 4.9).
- No backwards compatibility, one squashed migration, a one-off migration script outside the product (decision 19, §6.3; 3.9 above).
- Swarm compose keys honoured or refused, never ignored (decision 21; 3.10 above; Q1 ruled 2026-09-30: translate and warn, naming the native form).
- Per-node projections, a Traefik and a resolver on every node, clusterfs (decisions 9, 10, 22, 26; SHC-1058 §3.24).
- A returning node starts nothing it can’t prove it owns (decision 23, S17).
- promanager isn’t touched.
Clay’s rulings of 2026-09-30 and 2026-10-01 (SHC-1198), each a rule, with the section that carries it:
- Deployment, not stack; app, not package. The ladder is app, deployment,
service, replica, compose entry
<service>-r<ordinal>, container (2026-09-30, 2026-10-01; section 4). - The runtimes are
docker,podman,incusandapple, all four now; a runtime is the engine that takes a compose file; GPU, microVM and architecture are node capabilities; no Docker API shim, no Proxmox executor, and Proxmox HA never manages an SHC guest;--runtime docker|podman|incus|appleis a placement requirement; pve02 and pve03 become bare Incus hosts only after the cutover (2026-09-30, 2026-10-01; 3.1, 4.7, Q2). - Three doors into shcd: the unix socket, the mesh listener, the public listener. No container reaches the node plane or the internal API; reads are files from a module’s own section; the gate call carries only the user’s JWT; no module credential, identity is the pin (2026-09-30; 3.12).
- The policy unit is the deployment; same tenant open on declared ports;
cross-tenant blocked and refused at declaration; the system tier is
master’s platform environment; connections are wiring (2026-09-30;
3.13, 4.6). The address plan: fabric
10.128.0.0/10, a bridge per tenant per node, a /20 default grant,10.64.0.0/10node-local,10.192.0.0/10control plane (2026-10-01; 3.13). - No service mesh, closed (2026-09-30; 3.13).
ports:andexpose:only declare; an app never self-publishes;--hostand--porton the deployment are the only publication (2026-10-01; 3.14).- Two kernel WireGuard tunnels,
mesh0andshc0, never double wrapped, coordinated by shcd; Nebula is today’s carrier and goes at stage 2, after the cutover (2026-09-30, 2026-10-01; 3.11, 3.5). - The cutover lands bitspur on fresh VMs on both Gorilla boxes under Proxmox, docker runtime, app by app with both estates live, each app restored from a fresh restic backup, compose01 untouched as the rollback for two weeks and then retired; no replicated volumes in the first cut (2026-10-01; section 6, 4.8).
- One
--scaleflag with three shapes on install and update; noshc scale, nohome, nofailover; flags, never a path; spread, regions and apart are engine-internal (2026-10-01; section 5, SHC-1058 §3.6). - One compose entry per replica, always,
<service>-r<ordinal>; compose never scales;x-shc-scale.stateful: truemarks distinct members and replacesidentity: stablewith no alias (2026-10-01; SHC-1058 §3.6). - Render once on the leader with a
replicaobject; nodes never render or enrich; enrichment is the part of the render the author did not write; materialising is each node writing stored specs (2026-10-01; 4.4). - Swarm keys with a native form are translated with a warning naming it (2026-09-30; 3.10, Q1).
- Four pre-run checks on a node (2026-10-01; 4.3).
- The resolver is a small Go binary on miekg/dns per node;
dnsplusdns_searchper container; per-runtime wiring in the executor (2026-09-30; 4.5). - Health is defined once in compose, run by the runtime, read by the node agent, recorded per replica; no healthcheck means running is ready; dampened and coalesced (2026-10-01; SHC-1058 §3.13).
- Lease margins per SHC-1182 (2026-10-01; SHC-1058 §3.10).
- The per-node sealed secret cache is for external providers only (2026-10-01, SHC-1159): the node agent caches only values resolved from external vault providers (aws, 1password, hashicorp and the rest) for the replicas this node currently runs, in a root-only per-node file and never in rqlite; each value is sealed with the vault’s existing envelope scheme (a fresh AES-256-GCM DEK wrapped by the in-memory Argon2id KEK from unlock), so a node that cannot unlock its vault cannot read its cache, and an entry drops when its replica leaves the node. Local-provider secrets are not cached: every node already holds them, encrypted, in its rqlite copy. A platform container mounts only its own clusterfs section, with per-section revision directories (2026-10-01; 3.12, SHC-1058 §3.24).
- The six gotchas carry verdicts (2026-10-01; SHC-1058 §3.26), and the open questions are one table with a recommended answer each (section 9).
9. Questions for the owner
Section titled “9. Questions for the owner”Q1 and Q2 were ruled on 2026-09-30; the record stays here so every build lane reads one answer.
- Q1. Swarm-only vocabulary with a native equivalent. Ruled 2026-09-30:
translate and warn. The render translates
deploy.placement.constraintsinto rules,placement.preferencesintospread=K,max_replicas_per_nodeintospread=node,max=nanddriver: overlaynetworks onto the fabric, as SHC-1050 decision 21 and SHC-1058 §3.23 say, and prints a warning that names the native form to write. The proposal to refuse instead is not taken; no catalog app uses any of them, so the only readers are third-party apps written for Swarm. E1 (SHC-1158) builds it this way. - Q2.
--runtimeon install. Ruled 2026-09-30: runtime is a node fact (3.1) and--runtime docker|podman|incus|appleis a placement requirement the fit filter checks, so a mixed cluster can still pin an app to one runtime’s nodes. No fence per runtime.
Still open. One table, the recommended answer beside each; nothing ruled is in it. Clay decides; agents build neither until he does.
| question | recommended answer |
|---|---|
Where an app’s default port mappings live, now that ports: in compose never publishes (3.14) | In app.yaml, as default --port and --host mappings a deployment can override or remove, so shc install keeps working with no flags for the ten apps that self-published. |
--port on a service with more than one replica | Publish on each node that runs a replica; a published port is an exclusive per-node resource, so one such replica per node and a conflict is refused at placement; conn@ load balancers target those nodes. |
| When NAT traversal (hole punching, relay) is built for the two-tunnel mesh | Before any external estate, not before the cutover; our nodes have public addresses or a pfSense port-forward. |
Not open: the foundation layer is built in ce and carries into the new
line unchanged (Clay, 2026-10-01): the vault, the CA, events, the task
queue, internal RPC, crons, user schedules and jobs (ce modules/vault,
ca, events, task, internal_rpc, cron, schedule, job). Events
are durable, an event row broadcast to every peer with retry and
recovery on start, not best-effort hints. What is open in that layer is
only how it meets the new line, each owed a section or a ticket: which node
runs a job when a deployment spans nodes (task affinity, SHC-1058 §3.21),
which replica a backup captures, rolling shcd upgrades with stored specs
(4.4), and any voter fetching app content by digest (4.4). The migration
script’s detail stays SHC-1061’s.
References
Section titled “References”- Multi-site estates and fences (SHC-1050) and the SHC scheduler (SHC-1058): the rulings; this document reorders their build.
- Legacy and breaking changes: the earlier legacy sweep this one finishes.
- ce
migrations/embed.go: the reset doctrine. - Epic SHC-1092: the build map, the spines and the rig milestones.