Skip to content
SHC Docs

The new line, deleting first (design)

Status: open design (SHC-1092). This is the delete-first build order (section 7), and the epic is still in progress. Steps 1 and 2 are done: Swarm is deleted and runtime: swarm is refused by name, and the baseline was reset, though step 2’s edit-in-place rule became an append-only rule on 2026-10-03 (ce migrations/embed.go), with the squash left for v0.1.0. Step 7 went the other way round: Nebula is deleted on next and mesh0 and shc0 replaced it before the cutover. Not built on next: forward-auth deciding from files (9.2, SHC-1163; it still asks shcd), shc forward through the public route to a replica on another node (it resolves only on the daemon it reaches and answers not found, X404921FWDCNT; SHC-1692), and the cutover (step 14, SHC-1165). 3.14’s --host and --port are refused on next (X400226FLGOLD) and --expose is the one publication. It changes the ORDER of the build in the fences design (2026-09-29-multi-region-fences.md, SHC-1050, “§6.4”) and the scheduler design (2026-09-29-shc-scheduler.md, SHC-1058, “§8”), and shrinks or merges steps. It changes none of their rulings. Where a simplification brushes a ruling it’s listed in section 9 as a question, and the ruling stands until Clay answers.

Clay’s architecture rulings of 2026-09-30 and 2026-10-01 are folded in (SHC-1198). Each is stated as a rule with its date where it lands, and section 8 lists them all. Section 9 keeps only what is still open, with a recommended answer beside each.

The owner’s direction for the new line (2026-09-30): “Build it how you would build it and no legacy.” “Maximum simplicity. Simple interfaces, straightforward implementation, building it how you would build it if you’re building it from scratch.” No build-then-rip-out. And: simplification loses no feature. “I still want backups, I still want the repos.” Deleting Swarm removes a mechanism, never a capability; section 5 holds every capability to that.

Written against shc corp/next ab1d460e5 and ce corp/next 0b647c53 (both equal to their mains at the cut). ce paths are in the ce repo; every other path is in shc. LOC figures are cloc code lines (blank and comment lines excluded) unless a row says otherwise. Since the cut, next has merged main (shc 183e4edda, ce 4ede059c), which brings SHC-1079’s fence store (ce fence and fence_member tables, modules/fence with its derive, explain and fenceapi types, migration 20260930001929_fences.sql) and SHC-974’s lease fixes. Step 2.1 (SHC-1105) builds on that store instead of beside it, the contracts in section 4 reuse its types where they name the same thing, and the baseline reset in step 2 folds its migration in.


next was cut from main in shc and ce. main stays the old line and takes fixes only, for the live estates. At cutover next becomes main and the old main becomes release/v0.0. promanager is never touched.

The claim that a next push publishes nothing holds. Measured, not read from comments:

  • shc. workflow: creates a pipeline for any ref (- when: always, .gitlab-ci.yml:283-306), but every job in the file and in .gitlab/ci/pulp-publish.yml rules itself to the default branch, a tag or a schedule. A rule simulator over the merged job graph (both files and the pinned cicd template templates/shc-deployfile.yml@fa4fee78), with extends and !reference resolved, finds 0 jobs for a push to next, against 17 for a push to main with corp’s variables (15 unconditional, plus the Pulp and Pages jobs corp’s PULP_URL and SHC_PAGES_ENABLED arm). No deb, image, terraform artifact, apps index, Pulp copy or site.
  • MR pipelines into next. The template’s shc:generate and shc:deploy also match merge_request_event, but only with SHC_DEPLOY_ENABLED == "true", which corp’s shc project doesn’t set (6 project variables, none of them that one), and on an MR the generator deploys only a review environment, which neither deployfile declares (deploy/marketplace.shc.yaml:269, deploy/license-broker.shc.yaml:176, both prod only). next isn’t a protected branch either, so protected variables never reach it.
  • ce. Publishes nothing on any ref (its header says so and it has no publish job). Its one job, consumer/shc, runs on every branch push and compiles shc against the pushed ce. It defaults to SHC_REF: main, so a ce next push compiles shc main, which goes red the moment ce next drops a symbol old shc uses. That isn’t a publish hazard, but every ce next pipeline would lie. Fix: SHC_REF: next on ce next (one line, step N0 below). A branch with an open MR runs as an MR pipeline instead (ce .gitlab-ci.yml:64-68), with the same default. The DCO sign-off check in the same job reads MR commits, so ce next MRs need Signed-off-by:.

What next frees:

  • No step waits for SHC-1093 or the release/v0.0 cut. The line-aware CI is still needed, but at cutover, not before the first new-line commit.
  • The squash is a command. ce already has a reset doctrine (migrations/embed.go: “Resetting the history is rm -rf migrations/*.sql && make migrate”), and the history is three files, 1,097 SQL lines, one of them the 1,057-line baseline.
  • The Swarm code can go first, because nothing on next has to keep a live Swarm estate running.
  • The rig builds debs from next locally (make package). If agents want a pipeline-built deb for the rig, SHC-1093 adds one job on next that keeps its deb as a job artifact and publishes nothing.

Measured on next. “Whole” means the file exists only for Swarm or legacy and goes entirely; “part” means the lines were counted from the functions or blocks that go. shc keeps ce’s tests in tests/_ce/ (ce ships none), so ce test lines are counted there once and not again under shc.

ce (production Go; ce ships no tests). Swarm-named files are 44 files, 7,803 lines, on their own.

bucketwhatwhole filespart (est. ±15%)total
ASwarm verbs, compose-loader reconciliation, file-config lift, Swarm backup capture/quiesce/replay (daemon/services_swarm_verbs.go, modules/app/swarm_compat.go, swarm_file_configs.go, daemon/backup_swarm_*.go, less about 150 lines of streaming helpers the compose capture shares, which move)4,4171,2165,633
Bservice and task reads (subsystems/impl/swarm_recorder.go, core/swarmclass, Swarm health projection and one-shots)1,5602561,816
Crouting mesh and ingress (swarm_mesh, ingress_runtime)169134303
Doverlay networks (swarm_network, shared-network shadow, provider network attach)427252679
Emanagers, join, quorum (modules/cluster/swarm_managers.go, duplicate reap, suffrage census, core/swarminfo)1,2215021,723
Fplacement translation (swarm_mobility, swarm_placement, placement replay, volume guard, storage/swarm_label.go)1,031181,049
Hone-shot upgrade shims (payloadref backfill, vdeployment dir, OIDC plug, stack name, package trust, cluster host) 1,091; compose-versus-Swarm orphan detection 295; scattered compat arms about 2201,606
ce total12,809

Plus 552 generated ent/ lines from the three Swarm columns on node. The migration history (G) is 3 files and 851 code lines; the reset regenerates the baseline, so it removes history rather than schema.

shc (code lines; the Go the ce-overlay tests exercise is counted under “ce’s tests” below).

bucketGoGo testssh/batsother
A docker stack deploy (modules/app/swarm_driver.go:47, swarm_waves.go, daemon/wiring/services_swarm_edition.go)990852813
B docker service/node inspect reads in drain and cordon (daemon/wiring/cluster_drain_constraints.go, drain_pin_gate.go, drain.go:1101)4081,25572
C Swarm VIP upstream in traefik/render88
D overlay (TF provider fabric.go, overlay-dup-ip-check.sh)4587202
E managers, join tokens, converge-swarm (modules/cluster/swarm_converge.go, swarm_removal.go, modules/node/swarm_tokens.go, enroll_handler.go:1156)1,0532,914138
F Swarm pool capacity, advisor arm (modules/quota/enforcement.go:200, modules/placement/advisor.go:90)6668
generated pkg/apiclient Swarm fields110
terraform (Swarm ports, runtime enum in 14 modules), catalog Jinja branches (5), TS, apps bats42170
H legacy: terraform state upgraders and migration boundary, the provider/CLI skew gate ssh_compat.go, legacy-shape-probe.sh1,1971,670186
shc total3,8776,8541,453170

ce’s tests, which live in shc (tests/_ce, tests/unit): 23,725 lines exercise the ce code above (77 whole files in tests/_ce) and go with it.

Totals. About 17,000 lines of production code (ce 12,809, shc 3,877 Go plus 170 other), 32,000 lines of tests (shc 6,854 Go and 1,453 sh/bats, ce’s 23,725), and 552 generated ent/ lines. Against about 481,000 lines of production Go (ce about 370,000 outside ent/, moving by a few hundred with how nested ent directories are excluded; shc 111,091), that’s 3.5%. Deleting is cheap; the cascade in section 3 is where the simplicity comes from.

What the measure found besides:

  • The catalog has no Swarm-only keys. 104 of 116 compose files carry deploy:, and it’s resources everywhere plus one replicas (pulp). The five short-form port publishers were SHC-1081’s sweep, withdrawn on 2026-10-01: ports: in compose now only declares (3.14).
  • docker stack deploy is in shc, not ce (ce plugs it in through SetSwarmDeployLane), so step 1 is two coordinated MRs, ce first.
  • The adapters stay. ce modules/adapter (22,672 lines) imports estates from Portainer, CapRover, Dokploy and Easypanel, and 16 of its files read the source platform’s Swarm. That’s a product feature reading someone else’s Swarm, not SHC running one.
  • The lease rename is two senses. The attachment-ownership sense (about 450 occurrences in 57 ce files, 30 in 10 shc files) and the node-fencing sense SHC-1050 §6.1 also renames (fence_pending, assert-node-fenced, Fencer; about 380 in 36 ce files, 495 in 35 shc files). “Lease” already names the self-fence lease (LeasedUntil, LeaseExpired, about 350 uses in shc), which is the same object, so the rename converges on it; the unrelated deployment_lease, volume_mount_lease and crons_lease keep their names.

Deleting Swarm doesn’t only remove its adapter. Each of these is code that exists because two schedulers, two network planes or two runtimes had to be reconciled, and it collapses when one side is gone.

Today a deployment’s runtime is compose, swarm, podman or "", and ce core/runtimecmd maps each to a binary and a label dialect (docker stack deploy writes com.docker.stack.namespace and never com.docker.compose.project, so every project selector needs a Swarm arm). With Swarm gone the remaining runtimes speak one dialect (podman-compose sets the same compose labels, ce core/runtimecmd/runtimecmd.go:14-17), so the switch becomes one Executor interface with one implementation per runtime (4.7).

Ruled 2026-09-30 and 2026-10-01: the runtimes are docker, podman, incus and apple, named for the engine that takes the compose file, and all four are the architecture now. The Docker one is not called compose. incus is driven through lxc/incus-compose (SHC-1199); apple is its own executor over Apple’s container (SHC-1200). GPU, microVM isolation and architecture are node capabilities, facts that become fences and placement requirements, never runtimes; each executor turns a capability into its own mechanism. No Docker API shim and no Proxmox executor get built: Proxmox stays a node provider, SHC decides placement and uses Proxmox APIs only as mechanisms, and Proxmox HA never manages an SHC guest. pve02 and pve03 become bare Incus hosts (Debian 13 + Incus 7) only after the bitspur cutover (ruled 2026-10-01); until then they stay Proxmox and the incus runtime is proven in a VM.

The runtime evidence ladder goes with it. ce core/swarmclass, core/swarminfo, modules/app/ingress_runtime.go:95 (RuntimeFromEvidence), EffectiveRuntime and the “runtime unsaid” policy exist so a "" row can’t fail open against a Swarm-owned deployment (ce core/runtimecmd/runtimecmd.go:55-80: 33 of 83 rows on the measured estate carried ""). On a fresh baseline the column is NOT NULL, there’s no Swarm-owned deployment to protect, and there’s nothing to infer.

Runtime becomes a node fact (a node runs docker, podman, incus or apple; SHC-1056’s mixed cluster), and --runtime docker|podman|incus|apple on install becomes a placement requirement the fit filter checks, not a choice of code path (Q2, ruled 2026-09-30).

3.2 Ingress: a Traefik per node, no mesh, no label provider

Section titled “3.2 Ingress: a Traefik per node, no mesh, no label provider”

Today one system Traefik publishes 80/443 on one node. app.AmITraefikHost gates the ingress sweep to that node (ce modules/app/ingress_materialize.go:88), the entry-point move relocates it (ce modules/clone/move_ingress.go), the edge’s pool narrows to one node, and Traefik reads Docker labels besides the file provider. Under Swarm, published ports ride the routing mesh.

In the new line every node runs the per-node platform set (Traefik, forward-auth, resolver) as a global deployment, and Traefik reads only its own clusterfs section, traefik/current/dynamic (per-section since 2026-10-01, 4.4; SHC-1050 §3.13, SHC-1058 §3.16). What goes: the Docker-label provider, the Traefik-host singleton and AmITraefikHost, the entry-point move, the one-node pool, the ingress network, and every mesh workaround. Published ports come from --port on the deployment, never from the compose file (3.14, ruled 2026-10-01; this supersedes the mode: host shape of SHC-1058 S8 and the SHC-1081 sweep).

Why a Traefik per node works now when the old line ran one on one node (Clay asked, 2026-10-01): the singleton existed because Traefik read Docker labels from the local engine, and Swarm’s routing mesh carried the published 80 and 443 to whichever node held it. On the new line Traefik reads only clusterfs files that are identical on every node, its backends are fabric pins reachable from any node, and certificates are issued once by the daemon-side ACME source (8.2) and delivered to every node. No instance is special, so every node can serve and losing one loses nothing.

Built once. SHC-1050 7.3 (“Traefik on the fabric”, still one Traefik) and 8.1 (“a Traefik on every node”) would build the singleton on the fabric and then replace it. The new line builds 8.1 directly (section 7).

Drain reads constraints from docker service inspect and labels from docker node inspect (daemon/wiring/cluster_drain_constraints.go:38,51,95, consumed by modules/cluster/drain_pin_gate.go:129, drain.go:1101 and cordon.go:315) because Swarm owned placement. In the new line placement is SHC’s, so drain is: cordon the node, then ask the engine to re-place every replica assigned there (Place with the node excluded), in priority order, under the disruption budget. The journal, the pin gate and the --ha budget stay. The constraint matcher and both inspect reads go.

3.4 clusterfs replaces the materialiser family

Section titled “3.4 clusterfs replaces the materialiser family”

SHC has “no file-sync mechanism, on purpose” (modules/license/distribution.go) and instead grew one pattern many times: a replicated row, then each node writes its own local file. The licence, the cluster CA and mTLS leaf, the ingress route and cert files, the cert broker’s certserve homes, Nebula’s static host map and lighthouse pins, and the fabric’s render-time extra_hosts each have their own writer, their own atomicity and their own reload.

clusterfs (SHC-1058 §3.24) makes that one mechanism: one materialiser, one revision counter, an atomic swap per section with a root REVISION written after every section is in (per-section since 2026-10-01), and each consumer is a Section that writes its subtree (4.4). Each existing writer becomes a section or goes. The fabric’s extra_hosts go outright, because the resolver answers those names live (SHC-1050 decision 22).

Today a join carries a Swarm join token beside the cluster token, Swarm managers converge to three (converge-swarm, ce modules/cluster/swarm_managers.go, modules/cluster/swarm_converge.go), node rows carry swarm_node_id, swarm_role and is_swarm_manager, and Nebula is an EE toggle (shc cluster mesh enable|disable) with a LAN-addressed mode beside it.

In the new line a join is one enrol over the underlay: mTLS, the voter-slot claim (forced non-voter outside the control-plane fence), and the Nebula cert, CA and lighthouse list in the response (SHC-1050 §3.10). rqlite is the only quorum. On the two-tunnel target (3.11) the response carries the node’s WireGuard keys, fabric grant and endpoint candidates as one signed rqlite row in place of the cert and lighthouse list (noted 2026-09-30). What goes: Swarm tokens (shc node swarm-tokens), managers, converge-swarm and the Swarm node facts in step 1; the mesh on/off toggle and the LAN-addressing mode in 1.2 and 1.3.

The EE scorer “returns a node or nothing; it cannot refuse” (ce modules/stack/placement_seam.go:95-105), “Swarm does not get a score”, the dispatcher falls back to “the node that answered”, --node naming another node on compose is refused (X400956NDLOCL), a compose move always copies (MOVE-5), and deployment_node exists because “an unpinned swarm install stamps no placement constraint at all” (ce ent/schema/deployment_node.go:10-14).

In the new line Place returns bindings, pending, or a refusal naming the fence that emptied the eligible set (4.1). assignment rows replace deployment_node. X400956NDLOCL, the MOVE-5 clamp, the “no opinion” path and the Swarm placement renderers (ApplyDeployPlacement, ApplyVolumePlacement, ApplyVoterExcludes) go.

Swarm-owned volumes need their own capture, quiesce and replay (ce daemon/backup_swarm_capture*.go, backup_swarm_quiesce.go, backup_swarm_replay*.go, backup_swarm_volume_probe.go) because a Swarm task can run anywhere. In the new line the assignment names the node holding each volume, so capture always runs there through the one compose/podman path (ce daemon/backup_volume_capture.go), and SHC-1058 §3.25 adds the stopped-app path. One catch for step 1: the compose capture path calls the streaming helpers that live in backup_swarm_capture_stream.go (captureStreamVerbs, pullCaptureStream, captureChunkRPC, captureEndRPC; backup_volume_capture.go:59,175,309), so step 1 moves them into a neutral file before it deletes the rest. Backups, restore, verify, browse, the mirror and the second provider are untouched as features.

Today every service without restart: gets unless-stopped (ce modules/app/restart_default.go), so dockerd restarts containers at boot before ownership is known, and the leader-only reconcile cron can repair only its own node (X200924RSTRMT). In the new line the render stamps restart: "no" and the node agent restarts crashes (SHC-1058 S1). Swarm’s restart policy semantics come back natively (deploy.restart_policy read by the agent), and the per-node reconcile gap disappears because every node repairs itself.

3.9 The schema is a baseline, edited in place until v0.1.0

Section titled “3.9 The schema is a baseline, edited in place until v0.1.0”

The history is already one baseline plus two small files. The new line resets it once (after the Swarm delete drops the Swarm columns) and then every schema step edits the baseline in place (rm migrations/*.sql && make migrate) until v0.1.0 ships. No new-line database exists outside the rig before cutover, and the rig re-inits freely. So step 0.1 no longer has to carry every table of SHC-1050 §5.6 and SHC-1058 §6 before the code that uses them exists: each step brings its own tables into the baseline. After v0.1.0 migrations append normally. An old-line database is refused by name, never migrated; the one-off script (SHC-1061) reads the old schema from outside the product.

3.10 Compose keys: implemented where the feature lives, Swarm vocabulary translated or refused

Section titled “3.10 Compose keys: implemented where the feature lives, Swarm vocabulary translated or refused”

The Compose Specification owns deploy: (replicas, mode, resources, restart_policy, update_config, rollback_config, endpoint_mode, labels). Those aren’t Swarm legacy; they’re the app format. Each is read by the step that implements its feature: replicas by replicas (B4), mode: global by global mode (B5), restart_policy by the node agent (A4), update_config/rollback_config by rolling updates (D2), endpoint_mode: dnsrr by the resolver (8.4), deploy.labels by the render as container labels (traefik.* ones become file routes), top-level configs/secrets by staging (E2). There’s no separate translation layer.

What’s left is vocabulary that only means something to Swarm’s scheduler or network: deploy.placement.constraints (node.hostname, node.labels.*, node.role), placement.preferences, max_replicas_per_node, endpoint_mode: vip, networks with driver: overlay, and device or generic_resources reservations. One render lint handles all of it, with SHC-1058 §3.23’s outcome for each key: vip and devices are refused; the placement keys and driver: overlay have a native form (rules: [fence=node:X], spread=K, spread=node,max=n, a deployment network on the fabric) and §3.23 translates them. Ruled 2026-09-30 (Q1): they are translated, and the render warns, naming the native form to write. No catalog app uses any of them. Ports left this list on 2026-10-01: this section refused short-form and mode: ingress ports and counted on SHC-1081 to sweep five apps to mode: host; now every ports: form only declares and --port on the deployment is the one publication (3.14), and the sweep is withdrawn. The catalog sweep over all 106 apps stays as the proof nothing is dropped silently (SHC-1050 decision 21: honoured or refused, never ignored).

ce cluster_fabric.mode is swarm | shc | none. The new line has one fabric, so the mode column, --fabric and the Swarm overlay branch go. This section first said “nothing on one node”; since 2026-10-01 the per-tenant bridges and the tenant policy apply on every node, a single-node estate included (3.13), and one node simply has no cross-node tunnel. Today the fabric rides Nebula (MTU 1220), and that carrier can take the cutover.

Ruled 2026-09-30: the target is two kernel WireGuard devices per node, coordinated by shcd, never double wrapped. mesh0 carries the node plane (rqlite, the mesh listener); shc0 carries container traffic with each peer’s fabric grant as its AllowedIPs. Separate keys and ports, one handshake each, both dialling the same discovered underlay endpoint, so each packet is wrapped once. SHC coordinates and writes no tunnel: shcd owns the peer table in rqlite (keys, grants, endpoint candidates ranked same-site LAN, then public, then relay through a public node), a local cache of that table for boot since raft rides the mesh, the estate’s DNS name as the fallback for public nodes, and a per-peer MTU from an RFC 8899 style probe with an MSS clamp on shc0. A Mac runs the same control code over embedded wireguard-go. No new protocol, no Rust, no eBPF, no GPU crypto. Nebula goes at stage 2 (SHC-1201). Ruled 2026-10-01: the mesh lands after the bitspur cutover, which rides today’s Nebula plus fabric at MTU 1220; before it only the design, the rig measurement and a default-off stage 1 may merge, Nebula work in flight finishes, and no new Nebula scope is added.

Ruled 2026-09-30. shcd listens on three doors and nothing else. What collapses: forward-auth’s sidecar secret and its realm calls, and every “module credential” that would have let a container into the daemon.

doorwhoproof
the unix socket, /run/shc/shc.sock (0660 root:shc; SHC_SOCKET_PATH overrides it)the local shc CLI: root or the shc groupthe socket file’s permission
the mesh listener, on the node’s mesh addressother nodes: the leader relay, rqlite, node RPCthe tunnel binds a node’s address to its key (a Nebula cert today, the WireGuard key on mesh0); headers carry only the principal a node relays
the public listenerthe Mac CLI, Terraform and the Console, over NetBird today and the mesh client later; forward-auth’s gate calla Keycloak bearer; the gate call carries the user’s own JWT and nothing else

The rules that keep the doors honest:

  • No container reaches the node plane. The mesh address, rqlite, the mesh listener and the unix socket are unreachable from any container in any tenant; the policy compiler emits that as a device match, never as a per-app rule (3.13).
  • No container reaches the internal API, ever. It is node RPC with node identity. Clay asked twice whether modules should use it to simplify; the answer stayed no.
  • Reads are files. What a module needs arrives in its own clusterfs section, bind-mounted read-only: the resolver gets resolver/, Traefik traefik/, forward-auth auth/, never the whole tree, with a revision directory per section so a bind mount never pins a stale revision (ruled 2026-10-01). forward-auth’s realm-for-host and the gating declarations become files, so its sidecar secret goes.
  • The gate decision. Target: forward-auth links the Go decision package ce uses and decides locally from the JWT, validated against Keycloak’s keys, and the declarations file, zero calls per request, through daemon restarts and partitions. Until then its call goes to the local public listener carrying only the user’s JWT (SHC-1163).
  • No module credential; identity is the pin. A caller’s identity is its source address from the pin ledger, protected by the bridge antispoof guard; the API port is dropped when it arrives over shc0, so a source is always a local pin. Master’s platform environment, which is the system tier, has full scope by position; a tenant deployment that declares a broker use gets its verbs by position; the firewall opens the API port only to those. With forward-auth on files that set is empty on today’s estate.

Ruled 2026-09-30. The policy unit is the deployment (tenant, environment, stack); nothing finer, no per-container, per-label or L7 rule. What collapses: the cross-tenant socket and plug edges today’s compiler still allows, and the idea that a connection grants reachability.

flowverdict
inside a deploymentopen: it is one network
same tenant, to a service that declares ports (ports: or expose:)open, on exactly those container ports, by default
same tenant, to a service with no port declarationblocked: private to its deployment
across tenantsblocked, and a plug across tenants is refused when declared, never compiled
the system tier: Traefik and forward-auth to routed backends; any deployment to the OpenObserve collector and to the per-node resolveropen, as derived accept classes
any container to the node planeblocked (3.12)

The system tier is master’s platform environment. Connections (sockets and plugs) are wiring: names, credentials, config injection. They are not a permission. Policy is derived from declarations; there is no firewall to author, and the compiler stays nftables on Linux with a pf output for a Mac node. Every ported service gets a fabric pin, so a node’s grant is sized for its ported services. The accepted trade: a compromised app can reach every ported service in its own tenant, which then rests on those services’ own credentials. Making “fully blocked” true on one node is the structural work: a fabric bridge per tenant per node, which under an open-within-tenant default is simply the tenant’s network, and ingress-bridge-keyed rules, built under SHC-1202.

The address plan (ruled 2026-10-01). The fabric lives in 10.128.0.0/10. Each node gets a fabric bridge per tenant. The per-node grant is a genesis knob that defaults to /20 (32 tenants of 126 pins on a node, 1,024 nodes). Node-local deployment networks come from 10.64.0.0/10 and the control plane (mesh addresses, VIPs) from 10.192.0.0/10. New clusters only: the rig’s 10.84.0.0/16 is not renumbered.

No service mesh (closed, 2026-09-30). No sidecars, ambient proxies, per-workload certificates, per-request L7 policy or automatic service-to-service tracing, and no Istio, Linkerd, Consul Connect or Cilium mesh. Retries, circuit breaking and weighted rollout exist only as Traefik middlewares on the ingress path. Never re-propose.

3.14 Publication: --host and --port, never the compose file

Section titled “3.14 Publication: --host and --port, never the compose file”

Ruled 2026-10-01. In an app’s compose, ports: and expose: both mean only “this service listens on these container ports”; a ports: entry is treated as expose: and its host side is not honoured. An app never publishes a host port by itself. An exposed port becomes reachable three ways, none of them in the authored compose:

  1. inside the tenant, by policy (3.13): a pin, a resolver name and an nftables accept on those ports;
  2. --host on the deployment binds a service to a public hostname through Traefik, as a route file;
  3. --port on the deployment publishes on the node, or through a proxy-capable connection’s load balancer with conn@ (ce modules/app/port_mappings.go, grammar [<conn>@]<external>[:<service>|:<container>[:<container>]][/udp], a trailing ! removes). This is the only case that renders a ports: entry, long form mode: host, on the replica’s compose entry.

A service with no declared port has no pin and is private to its deployment. What collapses: the SHC-1081 shape (five apps swept to long-form mode: host in compose) and S8’s reading of mode: host as host publication; the ten catalog apps with ports: blocks stop self-publishing. Not ruled: where an app’s default mappings live, and --port on a service with more than one replica (section 9). Neither is built until ruled.


Small on purpose: each contract is what one spine owns and the others call. Types are sketches; the spine that builds one may rename a field, not widen the method set without a note on SHC-1092. Package homes follow SHC-1058 S9: fence store, reconciler, explain and the single-node engine in ce; the multi-node engine loop, rescheduling and failover in shc.

Shared vocabulary (ce core/placement, extending the existing leaf):

type NodeID string
type FenceID string
type DeploymentRef struct{ Tenant, Environment, Stack string }
type ReplicaRef struct {
Deployment DeploymentRef
Service string
Ordinal int
}
type NodeSet []NodeID // sorted, unique: no map iteration anywhere in placement
type Generation uint64

The words (ruled 2026-09-30 and 2026-10-01): the catalog thing is an app (apps/<name>/, app.yaml), never a package. One app installed for one tenant in one environment is a deployment, the running unit and its own network; stack names only the axis (DeploymentRef.Stack, -s/--stack, the clusterfs path). The ladder is app, deployment, service, replica (deployment, service, ordinal), compose entry <service>-r<ordinal> in a node’s projection, container. Existing code is not mass-renamed for it; new prose, docs, CLI help and error text use these words.

Pure and deterministic, like today’s scorer: no I/O, no clock, no map iteration in the path. Install, update, scale, move, drain, failover and volume provisioning all call Place. Explain takes the same snapshot and the same demand and runs the same code, so it can’t disagree with what Place did. For a running deployment the caller rebuilds the demand from its retained deployment_spec (A5) and the last placement_event.

// ce core/placement
type Engine interface {
Place(s Snapshot, d Demand) Decision
Explain(s Snapshot, d Demand) Explanation // Decision.Trace rendered, plus the move group
}
// Snapshot is one consistent read at one control-plane revision.
type Snapshot interface {
Revision() uint64
Nodes() []Node // condition, capacity, committed requests, runtime, location, provider
Members(f FenceID) NodeSet
Rules(ref DeploymentRef) []Rule // every rule with its source: resource, integration, deployment, scope rung
Assignments(ref DeploymentRef) []Assignment
}
type Demand struct {
Deployment DeploymentRef
Replicas []ReplicaDemand // service, ordinal, requests, mode, single writer, runtime requirement
Exclude NodeSet // drain and failover exclude the source
Priority int
}
type Decision struct {
Bind []Binding // replica -> node, with the generation it replaces
Pending []Pending // fits nowhere now: per-node reason
Refusal *Refusal // eligible set empty: the rule, its scope and the fence that emptied it
Trace Trace // eligible set per replica, rules with provenance, scorer terms
}

CE implements Engine with the fence filter and one node; shc implements it with the scorer (modules/placement.Score, weights unchanged). The existing PlacementAdvisor seam goes.

The only writer of desired placement. A new engine takes an epoch. A bind is one guarded statement batch that commits only while its epoch is current, each assignment’s generation still matches, and every target node still fits once the batch’s requests are added to the commitments it re-reads inside the same batch (SHC-1058 §2.2, §3.3). So a deposed engine’s write matches no row, and two decisions about different replicas can’t both spend the same free capacity.

// ce modules/scheduler
type Epoch uint64
type Assignments interface {
Lead(ctx context.Context, self NodeID) (Epoch, error)
Bind(ctx context.Context, e Epoch, b []Binding) error // ErrDeposed, ErrGenerationMoved, ErrNoFit
ForNode(ctx context.Context, n NodeID, linearizable bool) ([]Assignment, error)
Observe(ctx context.Context, n NodeID, obs []Observed) error // what ran, never what should
}

Runs on every node, CE included. Reads clusterfs, renders nothing, and owns restarts, the boot gate and the node lease. The boot gate is Confirm: after a boot the agent starts nothing, the per-node platform set included, until it holds its node lease for this boot, clusterfs has been re-materialised from the control plane (not the revision left on disk), a linearizable read of its own assignments (ForNode(..., true)) has succeeded, and every attachment lease it needs is re-acquired at its expected generation (SHC-1050 decision 23, SHC-1058 §3.9 and S17). Until then Apply stops what the control plane says moved and starts nothing.

// ce modules/nodeagent
type Agent interface {
Confirm(ctx context.Context) error // the boot gate; ErrPartitioned keeps it closed
Apply(ctx context.Context, rev clusterfs.Revision) error // converge to rev; idempotent; ErrUnconfirmed before Confirm
Report(ctx context.Context) (Report, error)
}
type Report struct {
Node NodeID
BootID string
Confirmed bool // the boot gate has opened for this BootID
Lease NodeLease // renewed by quorum write; TTL 35 s
Conditions []Condition // pressure reasons, runtime health
Observed []Observed // per replica incarnation: state, readiness, restarts
Revision uint64 // the clusterfs revision last applied
}

Before the agent runs any entry, four checks hold (ruled 2026-10-01): the revision is whole (every section written and swapped, REVISION written last, 4.4); the entry’s hash matches the spec digest on its assignment, so a corrupted disk, a materialiser bug and a hand edit all stop it; the replica is still this node’s at the current generation and engine epoch, with the lease held for anything exclusive; and every cross-node depends_on target is ready. Afterwards it reports the revision it applied and each replica’s state, which is what shc update waits on.

One writer of node-local files. A build reads the revision counter, builds every section from one read, re-checks the counter (a change discards the build), writes each section’s <section>/rev-<n>/ whole with its own REVISION file, fsyncs, swaps each section’s current, and writes the root REVISION last, the completion marker the node agent reads. This replaced one rev-<n>/ tree behind one current on 2026-10-01: a platform container bind-mounts only its own section’s stable parent, so it never pins a stale revision and never sees another section (3.12).

// ce modules/clusterfs
type Revision struct {
ID uint64
Root string // /var/lib/shc/clusterfs; section s of this revision is <Root>/<s>/rev-<ID>
}
type Section interface {
Name() string // "node", "traefik", "certs", "resolver", "auth", "fabric", "nebula", "licence"
Render(ctx context.Context, tx Reader, node NodeID, w Writer) error
}
type Materialiser interface {
Build(ctx context.Context) (Revision, error)
Current() (Revision, error)
OnSwap(func(Revision)) // Traefik SIGHUP, resolver reload, Agent.Apply
}

Three words, ruled 2026-10-01. Render is Jinja over the app’s files and the deployment’s settings, and it happens once, on the leader, at install or update; the result is a per-replica spec stored in rqlite with a digest (SHC-1113’s deployment_spec is its home). The Jinja context gains a replica object (ordinal, location, node); compose ${VAR} is not used for per-replica values, because it resolves once per file. Enrichment is the part of that render that adds what the author did not write: the pin, dns and dns_search, labels, the SHC_* variables, CA trust, telemetry, the volume suffix. Materialising is each node writing the stored specs and the derived Traefik and resolver sections into its own revision directory. A Section.Render call here derives files from rows; it never runs Jinja and never enriches. Nodes never render. So a daemon upgrade changes no projection until an explicit update; a spec that uses only replica.ordinal moves between nodes as an assignment change with no re-render, and one that uses replica.node or replica.location is re-rendered by the leader for the new placement. clusterfs is never an input: nothing that decides or renders reads it. Any voter must be able to fetch an app’s files at the pinned version, because a new leader renders the next update (section 9).

A small binary in the builtin Traefik project beside forward-auth (SHC-1050 decision 26). Answers from its own section, resolver/current/zones, only, never dials the control plane, and keeps answering from the last revision during a partition.

// shc cmd/shc-resolver (Go side of the container)
type Resolver interface {
Resolve(q Question) (Answer, error) // <svc>.<deployment zone>, <node>.nodes, routed hostnames
}
type Answer struct {
Addrs []netip.Addr // ready replicas' fabric addresses, or mesh addresses for nodes
TTL time.Duration
Revision uint64
}

Ruled 2026-09-30: a few hundred lines of Go on miekg/dns, one per node, not CoreDNS. It answers <service>.<stack>.<environment>.<tenant>.shc.internal with the ready replicas’ pins, nearest first, TTL 5 s; <node>.nodes.shc.internal with a node’s mesh address; and a routed hostname with the ready edges of the app’s location, empty where the location has none (ce modules/resolver, zone shc.internal). Everything else is forwarded. Each container gets it as dns plus dns_search [<deployment zone>, <environment zone>], so web and proxy.postgres both resolve; wiring that into containers per runtime belongs to the executor (4.7).

Everything one node needs to join the container fabric at one revision. The applier exists (ce modules/fabric/wireguard.go) and so does the nftables compiler (ce modules/fabric/policy), whose model needs more than placement: the integration edges, the port declarations, the ingress backends, and the pin ledger (ce modules/fabric/policy/model.go:105,133). The input says so. Since 2026-09-30 an edge is wiring, never a permission: the ruleset is derived from tenancy and port declarations (3.13), and the data plane is kernel WireGuard, routes, bridges and nftables with no per-packet code of ours (3.11).

// ce modules/fabric
type Input interface {
placement.Snapshot
Edges() []Edge // socket/plug edges: wiring (names, credentials), never permission (3.13)
Backends() []Backend // routed hosts -> ready replicas, for the per-node Ingress class
Pins() []Pin // pinned service addresses per node grant, with the ports each declares
}
type Policy interface {
Plan(in Input, n NodeID) (NodePlan, error) // grant, per-tenant bridges, peers, pins, compiled ruleset
}
type Applier interface {
Apply(ctx context.Context, p NodePlan) error
}

The fabric’s plan reaches the node as the fabric clusterfs section. Since 2026-10-01 the node’s grant is the genesis knob of 3.13 (default /20; this sketch first said /24), the plan carries one bridge per tenant on the node, and its peers are mesh addresses on today’s Nebula carrier and discovered underlay endpoints on shc0 at stage 2 (3.11).

One interface, one implementation per runtime, chosen by the node’s runtime: docker (docker compose), podman (podman compose), incus (lxc/incus-compose with its healthd sidecar, SHC-1199) and apple (Apple’s container, which has no compose tool, so the executor does the translation itself, SHC-1200). The node agent is its only caller. Ruled 2026-09-30 and 2026-10-01: the interface stays runtime-neutral and nothing above it may assume docker. Its input is the projection: one compose entry per replica, rendered and enriched once by the leader (4.4), carrying the author’s healthcheck, volumes and networks, the enriched dns and dns_search, and a ports: entry only where --port publishes on this node (authored ports are declarations, 3.14). Turning each into the runtime’s own mechanism (the engine’s probe, a CDI or Incus gpu device, a container exec on the declared interval for apple) is the executor’s job, never the render’s.

// ce core/executor (replaces ce core/runtimecmd)
type Executor interface {
Up(ctx context.Context, p Project) error // restart: "no" already stamped
Down(ctx context.Context, ref ProjectRef, o DownOpts) error
Ps(ctx context.Context, ref ProjectRef) ([]Container, error)
Exec(ctx context.Context, c ContainerRef, cmd []string, io IO) (int, error)
Events(ctx context.Context) (<-chan Event, error) // die, health; feeds the restart supervisor
}

Exec never puts a job’s environment on an argv (SHC-1176): values reach the process through a mode-0600 env file in a private temp dir, deleted after the call, or on stdin, never as -e K=v on the engine CLI or export lines in an sh -c script, since /proc/<pid>/cmdline is world-readable.

Reach answers before provisioning which nodes a volume connection can serve (SHC-1050 §3.6; VolumeLocationReporter already answers without a handle for eight of nine providers). The attachment lease is today’s self-fence, renamed (SHC-1050 §6.1).

// ce modules/storage/volumeapi
type Reach interface {
Reach(ctx context.Context, conn ConnectionID) (NodeSet, error)
}
type AttachmentLease interface {
// Acquire with the holder's own generation re-acquires and keeps it (the
// boot gate). A different node takes over only by naming the generation it
// expects and a Takeover proof: a STONITH confirm, --presumed-dead, or a
// lease expired past the single-writer stop window on a watchdog node
// (SHC-1050 decisions 24, 25; SHC-1058 §3.10, S16). A takeover bumps it.
Acquire(ctx context.Context, v VolumeID, n NodeID, expect Generation, proof *Takeover) (Generation, error)
Renew(ctx context.Context, v VolumeID, g Generation) error // every 5 s, TTL 15 s
Release(ctx context.Context, v VolumeID, g Generation) error
// Lost fires after two failed renews, and the agent stops the owner's
// containers. It never fires for a far-location volume, one whose reach
// holds no control-plane fence node (decision 24): that one keeps running
// through a WAN cut and is never taken over on expiry alone.
Lost() <-chan VolumeID
}

The stop after Lost does not depend on the engine answering (SHC-1182). The node keeps a ledger of every container it saw start (id, deployment labels, cgroup) under its state home. When the engine cannot list or stop a volume’s writers inside the stop’s budget, the stop kills their recorded cgroups itself: cgroup.kill, else SIGKILL of every pid in the cgroup, repeated until none is left. It reports done only once no process remains in them and no running container on the node is missing from the ledger, so a failed kill leaves the volume unstopped and the next tick retries. The kill fires only once the lease has lapsed and waits out a renewal still in flight until the start bound. Neither step calls the engine or reads the cluster (ce subsystems/impl/volume_lease_kill.go). The ledger is fed for docker-class runtimes and for podman through the docker API client (events and inspect; podman on its docker-compatible socket, its containers under machine.slice/libpod-<id>.scope), and for incus through the executor: a start is recorded at once and a floor pass lists every instance, with the cgroup cut at the top-level lxc.payload.<name> on unified cgroup v2 only. Apple has no host cgroup a lease-loss stop can kill. Every binder-bound volume is leased, so binding one there, including read-only, is refused with X409050VOLNKP before the volume is provisioned.

Ruled 2026-10-01: no replicated volumes in the first cut. Local volumes, restic, and failover by restore; LINSTOR comes later as a per-site capability, and reach and the attachment lease stay in the design for it.

The voter count is automatic and odd (Clay, 2026-09-30): the largest odd number no greater than the member nodes of the control-plane fence, capped at five. One eligible node gives one voter; two give one voter and one non-voter; three and four give three; five or more give five. Nodes are promoted and demoted automatically as they join or leave, one change at a time and never below a quorum of the current voters. A node that’s down but still a member keeps its vote until it’s removed, because shrinking the voter set during a failure would cut tolerance rather than add it. rqlite fixes suffrage at join, so a promotion or demotion is a remove and rejoin with the new suffrage, the move PromoteBeforeDeparture already makes (ce modules/cluster/voter_rebalance.go:179). The runtime target (ce modules/cluster/voter_target.go) and shc cluster set-voter-target go; the slot ledger (voter_grant.go) stays as the mechanism.

This replaces SHC-1050 §5.9’s recovery from one lost voter. There, a three-node fence needs --accept-deficit to remove the dead voter (X409205CLSRMD refuses a plain removal below target) and then runs on two voters. Here the target follows membership, so the removal is a plain shc cluster rm: two members remain, Reconcile demotes one survivor, and the fence runs on one voter and one non-voter, which tolerates losing the non-voter where two voters tolerate nothing. A replacement joining inside the fence brings it back to three. --accept-deficit and X409205CLSRMD go. Recovery from two lost voters is unchanged (repair --force).

// ce modules/cluster
func VoterTarget(eligible int) int // largest odd <= eligible, capped at 5; 0 for 0
type Suffrage interface {
Reconcile(ctx context.Context) (Change, error) // at most one promotion or demotion per call
}

The consequence, plainly: with two nodes and one voter, losing the voter stops every lease-held (stateful) app on the survivor, because attachment and node leases renew through the control plane and a lone non-voter has no quorum to renew against. The one exception is a far-location volume (decision 24, 4.8), whose owner keeps running. Stateless containers on the survivor keep running too, but nothing reschedules or changes until the voter returns. A third node in the control-plane fence makes it three voters and gives automatic tolerance of any single node failure, with no datacenter requirement beyond fence membership (SHC-1050 §3.9 decides eligibility; the count inside the fence is no longer a knob). Where the voters sit matters less than it seems: raft commits on a majority, so while the leader and the other near voter are up, a far third voter adds no commit latency. If raft elects the far node leader, every write crosses the WAN until leadership moves back (SHC-1050 §5.9’s 2 + 1 trade-off).

Nine contracts, 14 Go interfaces and 36 methods in all, counting the helpers (Snapshot, Section, fabric Input, Applier). Nothing else in the new core is an interface until a second implementation exists.


Every user-facing capability ce and shc have on next today, found from the CLI tree (docs/reference/cli, 311 shc_*.md pages), the daemon API (docs/reference/api/openapi.json, 324 routes), the terraform provider (19 resources and data sources), the ent schema (45 files) and the catalog (106 apps). S means today’s implementation rides a Swarm mechanism; I means it doesn’t. Every S row names what replaces the mechanism and the step that delivers it.

The code column says what happens to the implementation. REUSED: carries over unchanged, or loses only a Swarm arm or a name. MODIFIED: the existing code stays and changes where it’s fed from or where it runs (routed by assignment, read from clusterfs, filtered by fences). REWRITTEN: the behaviour needs code that doesn’t exist today, or replaces a Swarm or dockerd behaviour outright. Of 60 rows, 20 are REUSED, 33 MODIFIED and 7 REWRITTEN, and the 7 are all scheduler behaviour Swarm used to supply: replicas, rolling updates, restarts, rescheduling, global mode, service names and per-node reconcile. By lines, about 96% of today’s production Go survives the delete, and the edits land in a small part of it: install, render and ingress in ce modules/app, drain, the health reconciler, the move path and the scorer. Backups (ce modules/backup, 41,235 lines), storage (16,923), vault (11,563), DNS (10,099), auth (9,715), proxies (5,473) and the adapters (22,672) carry over as they are, less the Swarm arms step 1 removes. The new code is the scheduler Swarm used to be, an estimated 20,000 to 30,000 lines across the engine loop, assignments, node agent, clusterfs, resolver, replicas, global mode and rollouts.

Apps and their lifecycle

capabilityentry pointstodaycodenew line provides it bystep
install, update, uninstall, start, stop, restartshc install, update, uninstall, start, stop, restart; shc.stack.*S for Swarm-run deployments (docker stack deploy), compose driver otherwiseMODIFIEDPlace, then the node agent runs the node’s projection through Executor1, A3, 3.2, B3
dry run, validate, config, inspect, services, status, shc lsshc validate, inspect, status, ls; shc.stack.dry_run, inspect, services, statusS for Swarm-run deployments (service and task reads)MODIFIEDassignments plus the agents’ ObservedA3, A1
logs, exec, shell, port-forward, container restart and mountshc logs, exec, shell, forward, container; ce modules/portforward/swarm_resolver.go, daemon/services_forward_swarm_route.goS: Swarm task resolution across nodesMODIFIEDrouted to the replica’s node from its assignment; not built for shc forward through the public route, which resolves only on the daemon it reaches (SHC-1692); the node-local route proxies to a named nodeA3
scale and replicas--scale on shc install and shc update (three shapes; shc scale goes, ruled 2026-10-01), deploy.replicasS: Swarm spreads tasks; compose puts every replica on one nodeREWRITTENreplicas across nodes, one compose entry per replica, spread derived from the shapeB4
rolling update and rollbackshc rollback, rollback.list; deploy.update_configS: Swarm’s updater; compose recreates togetherREWRITTENrollout row, start-first incarnations, rollback to the retained specA5, D2
restart a crashed containerrestart:, deploy.restart_policyS: Swarm task restart; dockerd unless-stoppedREWRITTENthe node agent’s restart supervisorA4
run elsewhere when a node diesnone (automatic)S: Swarm reschedules swarm tasks; nothing for composeREWRITTENgrace clocks, rescheduling, dead-source failover of a replica with a volume, and single-writer takeover on a STONITH confirm or the lease proof from a watchdog node (SHC-1150)C1, C2, C3
one copy per nodedeploy.mode: global (no catalog app)SREWRITTENglobal modeB5
health, readiness, livenesshealthcheck (93 apps)I (compose), S for swarm tasksMODIFIEDreadiness feeds endpoints; liveness restartD1
depends_on across services51 apps, 50 with conditionsI inside one nodeMODIFIEDheld across nodes by the agentB3
move, dead-source move, repoint, clone, rename, protect, reclaim, recover, interrupted operationsshc move [--presumed-dead], repoint, clone, protect, unprotect, reclaim, recover, interrupted, finish-forwardI, with a Swarm-only relocation fast path (MOVE-5)MODIFIEDthe dispatcher relocates either runtime; --rule; group moves3.3, 3.4, 5.3
choose a node--nodeS: a node.hostname== constraint; compose takes only the local node and refuses another (X400956NDLOCL, ce modules/app/runtime_guard.go:170)MODIFIEDPlace with fence=node:X3.2
placement in an appdeploy.placement.constraints (no catalog app)SMODIFIEDtranslated into rules with a warning naming the native form (Q1, ruled 2026-09-30)E1
deployfiles and the CI templateshc deploy, deploy/*.shc.yaml, cicdIMODIFIEDunchanged, plus rules:3.1
vappsshc install <host>/<site>, vdeploymentIREUSEDunchangednone
jobs, hooks, cron, schedules, tasksshc job, cron, schedule, task; ce modules/jobS partly: job contexts resolve Swarm tasks (modules/job/context_service.go)MODIFIEDrun on the node the assignment names; jobs leave a lost nodeA3, C5
configs and secrets in deploymentstop-level configs (12 apps), secretsS for Swarm-run deployments (file binds lifted into Swarm configs, ce modules/app/swarm_file_configs.go); compose binds them on its one nodeMODIFIEDstaged per node; secrets from the vault into RAM-only mounts, with a per-node sealed cache of external-provider values only (ruled 2026-10-01)E2
reconcileshc reconcile, shc cluster reconcileI, leader’s own node onlyREWRITTENevery node’s agent converges itselfA4

Tenancy, config, identity, licence

capabilityentry pointstodaycodenew line provides it bystep
tenants, environments, scopes, 5-level config, contextsshc tenant, config, ctx; core/scopeIREUSEDunchanged; rules reuse core/scopenone, 2.4
vault, secrets, key rotation, cold startshc vault (13 verbs)IREUSEDunchanged; the Nebula CA joins the vault1.1
quotas and usageshc quota; shc.quota.*I; capacity gates have no callerMODIFIEDgates called by Place3.2
users, roles, grants, launcher layout, service clients, loginshc user, auth; /api/user/*IREUSEDunchanged; rule authority on platform:manage / tenant:manage2.4
licenceshc license (9 verbs)I, own materialiserMODIFIEDa clusterfs sectionA2
branding, console, portal, launcher, marketplace, MCPshc branding, portal, mcp; console UIIREUSEDunchangednone
HUDshc hud (x9s)IMODIFIEDgains the :fences, :rules and :placement viewsF1

Repos and catalog

capabilityentry pointstodaycodenew line provides it bystep
reposshc repo add, ls, sync, update, rmIREUSEDunchangednone
catalog and addonsshc catalog, addonIREUSEDunchangednone
import from Portainer, CapRover, Dokploy, Easypanelshc adapter load, ls, migrate; ce modules/adapter (22,672 lines)I: 16 files read the SOURCE platform’s Swarm, not SHC’sREUSEDunchanged; step 1 must leave these reads alonenone
publish the apps index, debs, provider, siteCI on main and tagsIREUSEDunchanged until cutover, then split by lineSHC-1093
integrations: sockets and plugsshc integration, *.socket.yaml, *.plug.yamlI; cross-node names ride the Swarm overlay or extra_hosts (S)MODIFIEDresolver names; colocation rules; wiring only, never a permission, and a cross-tenant plug is refused at declaration (3.13)8.4, B3, 5.1, 5.2, SHC-1202

Storage and backups

capabilityentry pointstodaycodenew line provides it bystep
storage connections, CephFS, NFSshc storage (20 verbs), shc connectionIMODIFIEDreach per connection; the binder picks by node4.1, 4.2
volumes, snapshots, rebind, volume mountsshc volume (10 verbs), shc.volume.*IREUSEDlease rename3
volume failover and STONITHEE ticker, Hetzner STONITH agentIMODIFIEDinside the eligible set; far-location exemption; the EE volume ticker, off by default, still rebinds a volume through its STONITH seam, and the replica failover pass reads the same agent per node (SHC-1150)4.3, 4.4, C2
backups: create, schedule, retention, verify, browse, mount, restore, export, import, groups, keysshc backup (27 verbs), retention, scheduleI, with a Swarm-only capture, quiesce and replay pathREUSEDone capture path on the volume’s node; stopped-app backups1, E3
mirror and a second providernightly off-provider mirrorI, leader onlyMODIFIEDper-location off-site repo and mirror; B29.1
recover and restore-as-moveshc recover, backup+restore moveIREUSEDunchangednone
backup repository sync and relocationshc backup sync, shc backup relocateIREUSEDunchangednone
writable backup mountsshc backup mount, shc.mount.* sessions with commit and discard (ce modules/backup/mount)IREUSEDunchangednone
deployment export and importshc export, shc import (ce modules/export, exporter-based)IREUSEDunchangednone

Networking, DNS, SSO, certificates

capabilityentry pointstodaycodenew line provides it bystep
node-to-node addressing--nic, internal_addressI, LAN; Nebula optionalMODIFIEDmesh0, the two-tunnel target, carries the cluster plane on next; Nebula is deleted (3.11)1.1 to 1.4, SHC-1201
container traffic across nodesoverlay networksS: the Swarm overlayMODIFIEDthe shc fabric on shc0 (3.11)7.1, 7.4, B3, SHC-1201
service names across nodesservice DNSS: Swarm VIP and DNSREWRITTENthe per-node resolver answering one VIP per service (SHC-1238; S7’s refusal was overturned on 2026-10-02)8.4
HTTP and TLS ingressbuiltin TraefikS for Swarm-run deployments (Swarm provider, mesh); one Traefik on one nodeMODIFIEDa Traefik per node from clusterfs, an edge per location8.1, 8.3
published TCP and UDP ports--host and --port on the deployment; ports: blocks in 10 appsS for Swarm-run deployments (the routing mesh); compose publishes on its one nodeMODIFIED--expose is the one publication on next and --port is refused (X400226FLGOLD); ports: in compose only declares (3.14, ruled 2026-10-01)E1
edges and load balancersshc.provider.*, ce modules/proxy (10 backends)I, one edge per clusterMODIFIEDan edge per location8.3
mesh status, diagnose, lighthouses, cert rotationshc cluster mesh, shc nebulaIREUSEDshc cluster mesh over mesh0; the leader-signed peer table replaced lighthouses and certs (SHC-1201, before the cutover) and shc nebula is gone1.2, 1.4
fabric statusshc cluster fabric statusIREUSEDunchanged, one mode7.1
DNS: public and private, dual planes, autozone, reap, takeover, validateshc dns; /api/dns/*IMODIFIEDthe public answer follows the location; the private answer is every Traefik’s mesh address8.3, 8.7
NetBird and pfSense connectionsshc connectionIREUSEDnameserver groups and routing peers runbook8.5
SSO, Keycloak, OIDC, gating, LAN posture/api/auth/*, ipAllowListIREUSEDunchanged; the gating declarations are also written as a clusterfs file for forward-auth (3.12)9.2
forward-auththe sidecar in the Traefik project (forward-auth/)I, one sidecar on the one Traefik nodeMODIFIEDa sidecar beside every Traefik, fed realms and gating declarations as files, its gate call carrying only the user’s JWT (3.12)8.1, 9.2
certificates: ACME, cert delivery, edge certs, CA, internal rotationshc cert, shc ca, shc cluster certI, ACME on the one TraefikMODIFIEDdaemon-side ACME feeding every Traefik and the broker8.2

Cluster and observability

capabilityentry pointstodaycodenew line provides it bystep
init, join, leave, rm, forget, repair, preflight, host setshc cluster, shc nodeS: a join is also a Swarm join; managers convergeMODIFIEDone enrol onto mesh0 with a leader-signed peer table1, 1.2
votersshc cluster status, set-voter-target, voter-targetIMODIFIEDautomatic odd count inside the control-plane fence; the target knob goes (4.9)6.1, 6.2
drain, undrain, cordon, uncordonshc node drainS: docker service inspect and docker node inspectMODIFIEDdrain calls Place with the node excluded1, D3
node health and conditionsshc node health, status, lsS partly: the Swarm health observerMODIFIEDthe node lease and the agent’s ReportA1
voter rebalance on departureshc cluster leave --rebalance, shc cluster rm --rebalanceIMODIFIEDcandidates only inside the control-plane fence6.1
events, audit, notifications, metrics, telemetry, collector, doctor, conditions, support bundle, searchshc events, audit, notification, doctor, support, searchIREUSEDunchanged; placement events2.3
terraform15 resources, 4 data sourcesIMODIFIEDshc_fence, shc_rule, rules on shc_deployment, location/provider on shc_node, lighthouse pins on shc_mesh0.3, 1.4, 2.2, 2.4, 3.1
leadershc leaderIMODIFIEDthe engine rides the leader (1 s poll)A3

Nothing in this inventory is dropped. Mechanism-only surfaces go with their mechanism, because they manage Swarm itself rather than serve a user: shc cluster converge-swarm, shc node swarm-tokens, the shc.network.swarm_status route, and --runtime swarm / --fabric swarm. Removing them loses no capability.

The cutover gate carries parity. SHC-1058 §7 gains a thirteenth condition: every row of this inventory works on the rig. That gate (SHC-1165) is what next has to pass before it becomes main.


The migration lives outside the product, so ce and shc stay free to make breaking changes with nothing compat baked in (SHC-1050 §6.3, decision 19). It’s a throwaway script in its own repo (or bitspur/ops on gitlab.com), run once per estate and deleted once the estate is across. Today that’s bitspur only; promanager isn’t touched. It’s SHC-1061.

The only thing that crosses the line is app data: database dumps, volume archives and the restic snapshots that hold them. Nothing reads the old control-plane schema except the old line’s own CLI and API, and nothing in the new line reads anything the old line wrote except that data.

Read side. Through the old-line CLI and API only, as JSON: tenants and environments, scope config, repos (definitions and credentials) and every app payload a deployment pins, private and local ones included, every deployment with its full spec (app digest, image digests, vars, overrides, port and host mappings, posture), connections and grants, socket and plug edges, vault entries key by key, DNS records and their owners, backup repos, schedules and retention, users’ launcher layouts, the licence.

Write side. Through new-line CLI and API calls only: shc tenant, shc config set, shc repo add and the payloads, shc connection connect, shc vault writes, installs pinned to the same app digest and image digests with the same vars and plug bindings (the same app_version isn’t enough: an app’s contents can change under one upstream version, ce ent/schema/deployment.go:54), shc backup schedule set, and a data restore into the freshly installed deployment. The first update onto a new-layout package renames 509 var keys (SHC-1360, SHC-1748), so the script translates each persisted override through var-renames-shc-1360.tsv before that update; an old name is otherwise kept and read by nothing.

The archive is what crosses. A sealed backup is one .bak in the restic repo (ce modules/backup/types.go:751) holding backup.tar in the canonical layout: .shc/manifest.json, app/, deployment/, secrets.vlt, data/<mount>/, data/@<converter>/ for database dumps, and hostbind/*.tar kept tar-in-tar so uid, gid and mode survive (ce modules/backup/archive.go:26). Restore today needs a backup row and its manifest (ce modules/backup/restore.go:210). The .bak is an ar container of two members, an outer manifest.json and backup.tar (ce modules/backup/archive.go:522), and import reads the outer one (ce modules/backup/actions.go:1049,1250). So per deployment the script pulls the .bak from the old repo, rewrites both manifests, the outer one and .shc/manifest.json inside backup.tar, into the shape the new line accepts (runtime, node and stack-record fields; volume capacity, access mode and host ownership carried over, ce modules/backup/types.go:329; for a pod with x-shc-scale.stateful: true, the manifest’s ordinals and the <volume>-r<ordinal> data names, without which restore refuses with X409896BKPORD), drops deployment/ and secrets.vlt (the new line renders its own and the vault moved key by key), reseals, and hands it to the new line’s shc backup import, then restores it into the deployment it installed. The product never reads an old-line manifest; the script owns that conversion.

Order. A dry run first: it renders every deployment under the new line, lists every refusal (Swarm vocabulary, an empty eligible set), prints move groups in order and every DNS change, changes nothing, and stops on any refusal. Then a verified backup of every deployment and of the control plane; any failed verify stops the run. Then, on the fresh cluster, the master tenant first (postgres, keycloak, openobserve), then tenants in move-group order. The shape, ruled 2026-10-01: the fresh cluster is VMs on both Gorilla boxes, pve02 and pve03, under Proxmox, docker runtime at cutover; the move is app by app with both estates live; each app’s data arrives by restoring a fresh restic backup taken right before its move, never by copying volumes off compose01, so the move proves the backups; compose01 stays untouched as the rollback for two weeks after the last app moves, switching back being a DNS change plus starting the old deployment, and is retired after that, never upgraded in place. Still SHC-1061’s to settle: the order of apps, how the shared databases and mailcow move, and what happens to the rig VMs on those boxes. For each deployment:

  1. Install and restore the first backup while the old copy keeps serving, then a health gate on the new copy through the new cluster’s own Traefik (a Host-header probe), before any DNS moves.
  2. Write barrier. Cut the old copy’s ingress (its routes answer 503) and let in-flight work drain, then take the final backup through the old line’s normal running-app path, whose hooks and converters need the containers up (ce modules/backup/cli_runner.go:184-198; stopped-app capture is new-line work, E3). Verify it, stop the old copy, and restore the backup into the new copy. This is the deployment’s only downtime. An app with background writers that ingress doesn’t stop (crons, workers) is listed by the dry run and gets its own quiesce step.
  3. Health gate again, then DNS handover and the edge re-point for its hosts.
  4. The old copy stays stopped, not removed, until the soak ends: two weeks after the last app moves (ruled 2026-10-01).

DNS handover. The new cluster is built as a new generation of the same cluster name (same slug and heritage), which is the case the existing takeover exists for: it adopts a sibling generation’s ownership claim and refuses a foreign one (ce modules/dns/takeover.go:46-88). It changes the ownership claim only, never the targets, so the script calls it and then the new line’s ordinary DNS reconcile re-points the records. There’s no CLI verb for it today (ce modules/app/commands/dns.go:61 registers ls only), so the script calls the API (/api/dns/takeover).

Rollback. Before a deployment’s handover it’s one command: start the old copy and restore its ingress; nothing public changed. After it, the script stops the new copy first, so only one writer exists, then takes the names back with the old cluster’s takeover (it’s a sibling generation too). Because a takeover moves ownership and never targets (ce modules/dns/takeover.go:63), the script then runs the old cluster’s DNS reconcile to point the records back, re-points the edge, and starts the old copy. Writes the new copy took since the handover are lost unless the script restores a new-line backup into the old copy, which holds only while the archive format is unchanged. Hence the soak before the old cluster stops, with its disks kept until Clay signs off.

Format assumptions that must hold, because they’re the whole interface between the lines:

  • The restic repository format and the .bak/backup.tar layout above, including the converter directories and the tar-in-tar host binds. If a new-line step changes any of it, the script converts, never the product.
  • Each converter’s dump format is what the new line’s restore reads (pg_dump and mysqldump output), with the same engine major on both sides, and apps install at the same app digest and upgrade afterwards, so each app’s own data schema matches.
  • The manifest fields the script can’t derive from the old API: volume capacity, access mode and host ownership.
  • The old repo’s restic password reaches the script from the old vault; it becomes a new-line backup key only if the repo is kept.
  • The DNS ownership claim (heritage=shc,shc/owner=..., generation) keeps its shape, so the new generation’s takeover recognises the old claim.
  • The licence is domain-bound, so the new cluster activates the same domain’s token.

Tickets. SHC-1061 carries the script, its dry run and the bitspur move. Its criteria already cover the dry run, verified backups, master tenant first, move-group order, health gates, the edge re-point, the soak, the script living outside the product and promanager untouched; this section adds the write barrier, the manifest rewrite, the digest-pinned installs, the takeover-then-reconcile DNS handover and the rollback to its scope. SHC-1123’s rig rehearses the script against a small old-line cluster before bitspur. SHC-1165 condition 9 proves every bitspur deployment’s backup restores on a new-line scratch node, which is the format check above. No new ticket is needed.


step (ticket)fatewhy
7.5 remove Swarm (SHC-1166)moves to the front as step 1; its eight dependencies dropnothing on next runs Swarm; deleting first means later steps never write Swarm arms
0.1 squash (SHC-1094)shrinks to a baseline reset after step 1, plus the edit-in-place rulethe reset is one command (3.9); each step brings its own tables
0.2 lease rename (SHC-1095)stays, after step 1renames only code that survives
3.6 constraints become rules (SHC-1120)merged into E1, canceledE1’s render handles the Swarm placement keys with the rest of the Swarm vocabulary, in one place (3.10, Q1)
E1 honour or refuse every Swarm key (SHC-1158)shrinks to the Swarm-vocabulary lint and the 106-app sweep; translate and warn (Q1, ruled 2026-09-30)each compose-spec key is read by the step that implements its feature (3.10)
7.3 Traefik on the fabric (SHC-1125)merged into 8.1, canceledbuilding the singleton on the fabric and then replacing it is the churn Clay ruled out (3.2)
3.5 drain reads the fence store (SHC-1119)merged into D3, canceledstep 1 deletes the inspect reads; drain then calls Place once, in D3 (3.3)
2.3 explain (SHC-1108) and B2 explain v2 (SHC-1136)merged; B2 canceled, 2.3 builds the one explainexplain v1 then v2 is build-then-replace; Explain is part of the engine (4.1)
3.2 scorer filters (SHC-1116) and B1 fit (SHC-1135)merged; B1 canceled, 3.2 builds Placefilter, fit, score, bind is one function (4.1)
B5 global mode (SHC-1146) and 8.1 Traefik per node (SHC-1137)dependency invertedthe platform set is a global deployment from day one, so B5 comes first and 8.1 feeds its Traefiks
7.2 Podman adapter (SHC-1056)shrinks to the Podman Executor on the fabric’s pinned subnetsthe render-time extra_hosts wiring in its criteria is replaced by the resolver (decision 22)
1.2a join extends the mesh (SHC-1078)on next, merged at a8af897c8, and the base of 1.2its converge, cert ledger and leave-revokes stay: 1.1 moves the CA and ledger to the vault and nebula_mesh, 1.2 adds the enrol-time cert and deletes the toggle; the old line never ran the opt-in mesh, so no backport
6.1 control-plane fence (SHC-1121)grows: the voter count becomes automatic and odd, with automatic promotion and demotion; the target knob goesClay, 2026-09-30 (4.9)
B4 replicas (SHC-1145), B5 global mode (SHC-1146)split in time: engine half in step 6, working across nodes after step 9cross-node names need the resolver and projections; the platform set needs the per-node Traefik
C2 failover (SHC-1150)dependency added on 4.1 and 4.4failover picks targets by reach and must honour the far-location exemption
§7 gate (SHC-1165)becomes the cutover gate for next to main, with the parity condition
line-aware CI (SHC-1093)moves to cutover; plus ce SHC_REF: next now (N0)next publishes nothing (section 1)
SHC-1081 port sweepwithdrawn as a shapeports: in compose only declares; --port on the deployment is the one publication (3.14, ruled 2026-10-01)
incus and apple executors (SHC-1199, SHC-1200), the two-tunnel mesh (SHC-1201), the tenant policy model and per-tenant bridges (SHC-1202)added 2026-10-01the rulings of 3.1, 3.11 and 3.13; each lands beside the spine it extends (7.2)

Everything else in §6.4 and §8 stays as written, one ticket each.

Spines as the epic defines them. Numbers in brackets are the old step names.

N0. next hygiene. ce consumer/shc compiles shc next on a ce next push (SHC_REF: next). Part of SHC-1093.

Delete (one lane, back to back, before any spine writes code):

  1. [7.5] Delete Swarm (SHC-1166). ce and shc: the Swarm runtime and docker stack deploy, every docker service / node / swarm read and write, the Swarm fabric mode and overlay, join tokens, managers and converge-swarm, the placement renderers and label helper, the Swarm compose loader and file-config lift and mobility pin, Swarm backup capture, the Swarm health observer and route forwarder, the runtime evidence ladder, their CLI verbs, API routes, terraform arms, bats suites and docs. It keeps what the compose path shares (the backup streaming helpers, 3.7) and the adapters’ reads of other platforms’ Swarm. A runtime: swarm request is refused by name. Two MRs, ce first, then shc on the new pin. Tree compiles, unit tests green, a one-node compose install still works.
  2. [0.1] Baseline reset (SHC-1094). rm migrations/*.sql && make migrate after step 1 drops the Swarm columns; the edit-in-place rule written into migrations/embed.go; an old-line database refused by name. The one-shot upgrade shims go with the history they upgraded (ce core/payloadref/backfill.go, the vdeployment directory, OIDC plug, stack-name, package-trust and cluster-host migrations), because a fresh baseline has no old rows for them to fix.
  3. [0.2] Lease rename (SHC-1095). Whole, no aliases, on the smaller tree.
  4. [0.3, 0.4, 0.5] node facts (SHC-1096), nebula* NIC skip (SHC-1097), stale docs that survive step 1 (SHC-1098). Parallel with step 3.

Build (spines start once step 2 is on next; each owns its interfaces from section 4):

  1. Node truth. A2 clusterfs (SHC-1110) first, because 8.x and the agent read it; A1 node lease (SHC-1109); A3 assignments with epoch and the node agent running single-replica installs through the Executor (SHC-1111); A4 restarts and boot gate (SHC-1112); A5 deployment_spec (SHC-1113), the home of the per-replica rendered specs (render once, 4.4); A6 watchdog (SHC-1114).
  2. Fences and engine. 2.1 (SHC-1105), 2.2 (SHC-1106), 2.4 (SHC-1107); 3.1 admission and --rule (SHC-1115); 3.2 Place with filter, fit and score, absorbing B1, binding through A3’s store (SHC-1116); 2.3 Explain, absorbing B2 (SHC-1108); 3.3 relocation (SHC-1117); 3.4 --rule on move (SHC-1118); 6.1 the control-plane fence with the automatic odd voter count (SHC-1121, 4.9) and 6.2 status against it (SHC-1122); then the engine half of B4 replicas (SHC-1145: one compose entry per replica and the three --scale shapes, SHC-1058 §3.6) and B5 global mode (SHC-1146), B6 (SHC-1147), B7 (SHC-1148). B4 and B5 bind and place here; they work across nodes only once step 9 delivers the resolver and projections (B4) and the per-node Traefik (B5’s platform set), and their tickets close then.
  3. Nebula. 1.1 CA in the vault (SHC-1099), 1.2 on from genesis (SHC-1100), 1.3 cluster plane on mesh addresses (SHC-1101), 1.4 lighthouses and relays (SHC-1102), 1.5 (SHC-1103), 1.6 (SHC-1104). The two-tunnel mesh (SHC-1201) lands after the cutover; before it only its design, the rig measurement and a default-off stage 1 merge, and no new Nebula scope is added (3.11, ruled 2026-10-01).
  4. Fabric. 7.1 default-drop (SHC-1055), 7.2 Podman executor (SHC-1056), 7.4 MTU (SHC-1126); the incus executor (SHC-1199) and the apple executor (SHC-1200), the runtime-name change landing first (3.1); the tenant policy model, the per-tenant bridges and the address plan in modules/fabric/policy (SHC-1202, 3.13).
  5. Edges and DNS. 8.1 Traefik per node from clusterfs, absorbing 7.3 (SHC-1137, after B5’s engine half and A2); 8.2 daemon ACME (SHC-1138); 8.3 edge per location (SHC-1139); 8.4 resolver (SHC-1140, miekg/dns, 4.5); 8.5 forwarding and NetBird runbook (SHC-1141); B3 per-node projections (SHC-1144, after 8.4 and 8.5; one entry per replica, dns and dns_search enriched at render); 8.6, 8.7 (SHC-1142, SHC-1143).
  6. Render. E1 Swarm-vocabulary lint and the 106-app sweep, absorbing 3.6 (SHC-1158; translate and warn, Q1; the lint can land any time after step 1, the sweep closes once B4 and B5 honour replicas and mode: global); ports: read as a declaration and --port as the one publication (3.14; SHC-1081’s sweep is withdrawn); E2 configs and secrets with the per-node sealed cache of external-provider values (SHC-1159, ruled 2026-10-01); E3 stopped-app backups (SHC-1160).
  7. Storage. Phase 4 (SHC-1127 to SHC-1130): reach and the binder’s choice by node (4.1, 4.2), failover inside the eligible set (4.3), and the far-location exemption (4.4, after 6.1). No replicated volumes in the first cut (4.8, ruled 2026-10-01).
  8. Failure and rollout. C1 (SHC-1149); C2 (SHC-1150, after 4.1, 4.4, A5 and A6); C3 to C5 (SHC-1151 to SHC-1153); D1, D2, D3 drain on the engine absorbing 3.5, D4 (SHC-1154 to SHC-1157).
  9. Colocation, surfaces, SSO, backups. Phase 5 (SHC-1131 to SHC-1134), F1 (SHC-1079), F2 (SHC-1161), 9.1 to 9.3 (SHC-1162 to SHC-1164; 9.2 is forward-auth on files, 3.12), 6.3 (SHC-1124).
  10. Rig and cutover. The rig (SHC-1123) from step 7’s 1.2 on; the cutover gate (SHC-1165) with the parity condition; SHC-1093’s line-aware CI; then next becomes main, old main becomes release/v0.0, and SHC-1061 moves bitspur by path B onto fresh VMs on both Gorilla boxes, app by app from restic, with compose01 kept untouched as the rollback for two weeks and then retired (section 6, ruled 2026-10-01). After that cutover pve02 and pve03 become bare Incus hosts (3.1) and the two-tunnel mesh lands (3.11).

The rig milestones on the epic (M1 to M5) keep their meaning; M3 now proves the per-node Traefik directly instead of a fabric singleton first.


  • Fences are the placement primitive; every rule is attached to an object (SHC-1050 decisions 5 to 7, 20). Snapshot.Rules carries provenance.
  • Nebula is the node-addressing plane, always on; the fabric carries containers; NetBird is for people only (decisions 16 to 18). 1.2 and 1.3 delete the mesh toggle and the LAN mode instead of keeping them beside Nebula. Superseded as the target on 2026-09-30: two kernel WireGuard tunnels coordinated by shcd, Nebula gone at stage 2, after the cutover (3.11); Nebula stays today’s carrier, so 1.2 and 1.3 still delete the toggle and the LAN mode.
  • One placement engine with explain, used by every placement decision (4.1).
  • The control-plane fence, enforced at enrol, with the voter count automatic and odd, 1, 3 or 5 (decision 8, refined by Clay on 2026-09-30; 4.9).
  • No backwards compatibility, one squashed migration, a one-off migration script outside the product (decision 19, §6.3; 3.9 above).
  • Swarm compose keys honoured or refused, never ignored (decision 21; 3.10 above; Q1 ruled 2026-09-30: translate and warn, naming the native form).
  • Per-node projections, a Traefik and a resolver on every node, clusterfs (decisions 9, 10, 22, 26; SHC-1058 §3.24).
  • A returning node starts nothing it can’t prove it owns (decision 23, S17).
  • promanager isn’t touched.

Clay’s rulings of 2026-09-30 and 2026-10-01 (SHC-1198), each a rule, with the section that carries it:

  • Deployment, not stack; app, not package. The ladder is app, deployment, service, replica, compose entry <service>-r<ordinal>, container (2026-09-30, 2026-10-01; section 4).
  • The runtimes are docker, podman, incus and apple, all four now; a runtime is the engine that takes a compose file; GPU, microVM and architecture are node capabilities; no Docker API shim, no Proxmox executor, and Proxmox HA never manages an SHC guest; --runtime docker|podman|incus|apple is a placement requirement; pve02 and pve03 become bare Incus hosts only after the cutover (2026-09-30, 2026-10-01; 3.1, 4.7, Q2).
  • Three doors into shcd: the unix socket, the mesh listener, the public listener. No container reaches the node plane or the internal API; reads are files from a module’s own section; the gate call carries only the user’s JWT; no module credential, identity is the pin (2026-09-30; 3.12).
  • The policy unit is the deployment; same tenant open on declared ports; cross-tenant blocked and refused at declaration; the system tier is master’s platform environment; connections are wiring (2026-09-30; 3.13, 4.6). The address plan: fabric 10.128.0.0/10, a bridge per tenant per node, a /20 default grant, 10.64.0.0/10 node-local, 10.192.0.0/10 control plane (2026-10-01; 3.13).
  • No service mesh, closed (2026-09-30; 3.13).
  • ports: and expose: only declare; an app never self-publishes; --host and --port on the deployment are the only publication (2026-10-01; 3.14).
  • Two kernel WireGuard tunnels, mesh0 and shc0, never double wrapped, coordinated by shcd; Nebula is today’s carrier and goes at stage 2, after the cutover (2026-09-30, 2026-10-01; 3.11, 3.5).
  • The cutover lands bitspur on fresh VMs on both Gorilla boxes under Proxmox, docker runtime, app by app with both estates live, each app restored from a fresh restic backup, compose01 untouched as the rollback for two weeks and then retired; no replicated volumes in the first cut (2026-10-01; section 6, 4.8).
  • One --scale flag with three shapes on install and update; no shc scale, no home, no failover; flags, never a path; spread, regions and apart are engine-internal (2026-10-01; section 5, SHC-1058 §3.6).
  • One compose entry per replica, always, <service>-r<ordinal>; compose never scales; x-shc-scale.stateful: true marks distinct members and replaces identity: stable with no alias (2026-10-01; SHC-1058 §3.6).
  • Render once on the leader with a replica object; nodes never render or enrich; enrichment is the part of the render the author did not write; materialising is each node writing stored specs (2026-10-01; 4.4).
  • Swarm keys with a native form are translated with a warning naming it (2026-09-30; 3.10, Q1).
  • Four pre-run checks on a node (2026-10-01; 4.3).
  • The resolver is a small Go binary on miekg/dns per node; dns plus dns_search per container; per-runtime wiring in the executor (2026-09-30; 4.5).
  • Health is defined once in compose, run by the runtime, read by the node agent, recorded per replica; no healthcheck means running is ready; dampened and coalesced (2026-10-01; SHC-1058 §3.13).
  • Lease margins per SHC-1182 (2026-10-01; SHC-1058 §3.10).
  • The per-node sealed secret cache is for external providers only (2026-10-01, SHC-1159): the node agent caches only values resolved from external vault providers (aws, 1password, hashicorp and the rest) for the replicas this node currently runs, in a root-only per-node file and never in rqlite; each value is sealed with the vault’s existing envelope scheme (a fresh AES-256-GCM DEK wrapped by the in-memory Argon2id KEK from unlock), so a node that cannot unlock its vault cannot read its cache, and an entry drops when its replica leaves the node. Local-provider secrets are not cached: every node already holds them, encrypted, in its rqlite copy. A platform container mounts only its own clusterfs section, with per-section revision directories (2026-10-01; 3.12, SHC-1058 §3.24).
  • The six gotchas carry verdicts (2026-10-01; SHC-1058 §3.26), and the open questions are one table with a recommended answer each (section 9).

Q1 and Q2 were ruled on 2026-09-30; the record stays here so every build lane reads one answer.

  • Q1. Swarm-only vocabulary with a native equivalent. Ruled 2026-09-30: translate and warn. The render translates deploy.placement.constraints into rules, placement.preferences into spread=K, max_replicas_per_node into spread=node,max=n and driver: overlay networks onto the fabric, as SHC-1050 decision 21 and SHC-1058 §3.23 say, and prints a warning that names the native form to write. The proposal to refuse instead is not taken; no catalog app uses any of them, so the only readers are third-party apps written for Swarm. E1 (SHC-1158) builds it this way.
  • Q2. --runtime on install. Ruled 2026-09-30: runtime is a node fact (3.1) and --runtime docker|podman|incus|apple is a placement requirement the fit filter checks, so a mixed cluster can still pin an app to one runtime’s nodes. No fence per runtime.

Still open. One table, the recommended answer beside each; nothing ruled is in it. Clay decides; agents build neither until he does.

questionrecommended answer
Where an app’s default port mappings live, now that ports: in compose never publishes (3.14)In app.yaml, as default --port and --host mappings a deployment can override or remove, so shc install keeps working with no flags for the ten apps that self-published.
--port on a service with more than one replicaPublish on each node that runs a replica; a published port is an exclusive per-node resource, so one such replica per node and a conflict is refused at placement; conn@ load balancers target those nodes.
When NAT traversal (hole punching, relay) is built for the two-tunnel meshBefore any external estate, not before the cutover; our nodes have public addresses or a pfSense port-forward.

Not open: the foundation layer is built in ce and carries into the new line unchanged (Clay, 2026-10-01): the vault, the CA, events, the task queue, internal RPC, crons, user schedules and jobs (ce modules/vault, ca, events, task, internal_rpc, cron, schedule, job). Events are durable, an event row broadcast to every peer with retry and recovery on start, not best-effort hints. What is open in that layer is only how it meets the new line, each owed a section or a ticket: which node runs a job when a deployment spans nodes (task affinity, SHC-1058 §3.21), which replica a backup captures, rolling shcd upgrades with stored specs (4.4), and any voter fetching app content by digest (4.4). The migration script’s detail stays SHC-1061’s.


  • Multi-site estates and fences (SHC-1050) and the SHC scheduler (SHC-1058): the rulings; this document reorders their build.
  • Legacy and breaking changes: the earlier legacy sweep this one finishes.
  • ce migrations/embed.go: the reset doctrine.
  • Epic SHC-1092: the build map, the spines and the rig milestones.