Skip to content
SHC Docs

The SHC scheduler (design)

Status: open design (SHC-1058). Most of this is built on next: the assignment store and its epoch (ce modules/scheduler), the leader’s engine loop and its passes (shc registers the failover and rollout passes), grace, node return, global mode and rollouts. Failover moves a lost node’s stateful replicas and single writers too: shc’s failover pass wires ce clone’s dead-source move, the configured volume backend’s STONITH agent read per node, and the watchdog fact each node records on its row (SHC-1150, 3.9, 3.10, S16); unit specs prove it and the node-loss rig run is still to come. The cutover gate (SHC-1165) has not passed. §4.2’s JSON, §4.3’s reasons and §6’s columns differ from the code (fenceapi.ExplainDoc, ce ent/schema), which is the reference, and §5.1’s shc fence verbs became shc label, shc constraint and shc selector. Its seventeen open questions were decided by agent on Clay’s standing delegation, 2026-09-30, reversible (section 9). It answers the question list on SHC-1058 and builds on the ruled fences design, 2026-09-29-multi-region-fences.md (SHC-1050), cited below as “SHC-1050 §n”. It contradicts none of it: where SHC-1050 handed a question to SHC-1058, this document answers it, and where this document refines a SHC-1050 surface (the shc fence verbs), it says so.

Written against shc corp/main 7f484505c and ce corp/main 6e194b44b. ce paths are in the ce repo; every other path is in shc. Where a comment and the code disagree, the code wins and the comment is listed in section 10.

Clay’s architecture rulings of 2026-09-30 and 2026-10-01 are folded in (SHC-1198) and listed in section 11. Where one supersedes a line below, the line is replaced and dated. The open questions live in one table in the new-line simplification §9.

The owner’s rulings this design is bound by (SHC-1050 decisions 16 to 28, and the SHC-1058 thread of 2026-09-29):

  • Swarm is retired. SHC is the only scheduler. Nodes run a compose runtime as a dumb executor: docker, podman, incus or apple, all four the architecture now (2026-09-30, 2026-10-01).
  • Nebula addresses nodes today and the shc fabric carries container traffic; the target, ruled 2026-09-30, is two kernel WireGuard tunnels, mesh0 for nodes and shc0 for containers, never double wrapped, landing after the cutover (2026-10-01).
  • A fence is a named node set and means placement only. The old volume attachment fencing is renamed lease (SHC-1050 §6.1). This document uses the new names throughout: “attachment lease”, “lease generation”, “lease loss”.
  • One placement engine, with explain, used by install, move, drain, failover, scaling and volume provisioning.
  • 3 or 5 voters in the control-plane fence; since 2026-09-30 the count is automatic and odd, 1, 3 or 5, with automatic promotion and demotion (new-line simplification §4.9).
  • A resolver on every node and a read-only cluster config directory on every node, materialised from the control plane.
  • One compose project per node a deployment touches (a per-node projection), one entry per replica (2026-10-01); a Traefik on every node fed through the file provider.
  • No backwards compatibility. bitspur moves with a one-off script outside the product (SHC-1061). promanager is not touched.
  • Rules come from the resource (hard), the integration (socket default, plug override), the deployment, and the scope ladder. No free-floating policies.
  • A move works on a dead app as long as its data is reachable and a target satisfies the rules. Unreachable data waits for its node, or is removed by force as orphaned.
  • A node that reboots starts nothing it lost ownership of.
  • Automatic failover reuses the dead-source move (--presumed-dead) and waits for the old node to stop itself first (decision 25).
  • Every Swarm compose key keeps its effect or is refused at render. None is silently dropped. A key with a native form is translated with a warning that names it (2026-09-30).

Vocabulary used below:

wordmeaning
appthe authored thing in the catalog (apps/<name>/), never “package” (2026-10-01)
deploymentone app for one tenant in one environment; the running unit and its own network; never “stack” for the running thing, which names only the axis (2026-09-30)
serviceone authored compose service inside a deployment: template, desired replicas, healthcheck, DNS name
enginethe one placement engine. Runs on the raft leader. Writes assignments.
replicaone running copy of one service of one deployment: (deployment, service, ordinal)
assignmentthe row that binds a replica to a node, with a monotonic generation
node agentthe part of shcd on every node that runs that node’s assignments
projectionthe compose project the node agent runs for one deployment on one node: one entry per replica, named <service>-r<ordinal>
clusterfsthe read-only cluster config directory, /var/lib/shc/clusterfs
node leasea per-node liveness lease renewed by a quorum write (new)
attachment leasetoday’s per-volume self-fence lease, renamed by SHC-1050 §6.1

Checks a stranger can run on a three-node estate with no Swarm:

  1. shc install wiki with no --node lands on the node the engine picked, and shc fence explain -s wiki prints why, down to the scorer’s terms.
  2. shc install wiki on an estate with no node that fits prints pending, not a local install, and explain names the nodes and what each lacked.
  3. Power a node off. Its stateless deployments run elsewhere within about 95 s (35 s lease + 60 s grace). Its reattachable stateful deployments run elsewhere within about 5 min 35 s, on their own volumes, and never while the old node could still write (3.10). Its local-disk deployments show down, stay assigned to it, and start when it returns, unless an operator fails one over by restoring its backup elsewhere (3.9).
  4. Power the node back on. It starts nothing until it has confirmed ownership with the control plane, removes the copies that moved, and starts the ones it still owns. Nothing moves back on its own.
  5. shc update -s wiki -e prod --scale web=3 on a stateless service puts one replica on each node, and Traefik on every node load-balances across the three. Kill one node: its replica leaves every Traefik and resolver answer within the lease TTL and is replaced after the grace.
  6. shc update -s wiki -e prod with deploy.update_config: {parallelism: 1, order: start-first} replaces one replica at a time, and a replica that never reports healthy stops the rollout (failure_action: pause) or reverses it (rollback).
  7. shc node drain n2 moves each deployment inside its eligible set, keeps --ha deployments serving throughout, and leaves the per-node platform set (Traefik, forward-auth, resolver) running until the node is empty.
  8. Every compose key Swarm honoured either works under SHC or is refused at render by name. A catalog sweep renders all 106 apps under apps/<tier>/<app>/compose.yaml with zero silent drops.
  9. shc fence ls, shc fence show, shc fence explain and the :fences, :rules and :placement HUD views answer the same questions from the same data.

piecetodaywhere
desired statethe replicated deployment row; its node is recorded_node_id (where the bytes are) and pinned_node_id (the operator’s --node)ce ent/schema/deployment.go:104,131
who placesSwarm, for the swarm runtime; for compose/podman, the EE scorer through the install dispatcher, which falls back to “the node that answered” when it has no opinionce modules/app/placement_seam.go, modules/app/install_dispatch.go, daemon/services_install_placement.go:48; modules/placement/advisor.go:80-137
scorerdeterministic, integer, spread-weighted; returns every candidate’s terms; cannot refusemodules/placement/placement.go:27-50,175,221-240
capacity gatesEnforceNodeCapacity, EnforceSwarmPoolCapacity, EnforceLargestReplicaFitsNode exist and no production code calls themmodules/quota/enforcement.go:168,200,246
node livenessthe leader polls every node every 2 s and stamps cluster_state=unreachable after 3 failed pollsce modules/health/crons.go:81,194, modules/health/node_poller.go:396-397,876-930
node intentavailability = active, draining, paused (cordon); drain adds drained, partially_drainedce modules/node/types.go:73-83, modules/cluster/service.go:2970-2973, modules/cluster/availability.go:46-60; modules/cluster/drain.go:121
self-healDocker’s restart: unless-stopped, stamped on every compose and podman service that declares no restart:; the leader-only reconcile cron runs compose up -d for its own node’s deployments and logs X200924RSTRMT for everyone else’sce modules/app/restart_default.go:26, modules/health/reconcile/reconcile.go:252-271
single writerper-volume attachment lease renewed every 5 s, TTL 15 s; after 2 failed renews the node stops the owner’s containers, lazily unmounts and flushesce subsystems/impl/volume_lease.go:68-82
failoverEE ticker, off by default, rebinds a volume (never the workload) on lease expiry after three consecutive qualifying 15 s ticks, or on a STONITH confirm. The constants describe a 45 s dwell; the code counts ticks, so the real floor is about 30 s after the first qualifying ticksubsystems/volume_failover.go:36-58,80,89,465-500; daemon/wiring/volume_rebind_ops.go:42-60
dead-source moveshc move --presumed-dead rebuilds a deployment on a target from replicated state; needs unreachable and a failing probece modules/clone/move_presumed_dead.go:13-51, modules/clone/types.go:129-135
replicasdeploy.replicas and --scale feed compose, so every replica lands on the one node the project runs once modules/app/deploy_overrides.go:28-83, modules/app/services_writer.go:181-189
rolling updatecompose up -d --wait recreates changed services together; a failed converge restarts what it stopped; no image rollbackce modules/app/compose_driver.go:79-120, modules/app/converge_restore.go, modules/app/converge_journal.go
drainEE, journalled, per-deployment relocation, --ha read as a disruption budget, pin gatemodules/cluster/drain.go:300,361,1201-1340, drain_disruption.go, drain_pin_gate.go
ingressone system Traefik on one node, Docker-label provider plus a file provider the daemon writes and SIGHUPsce addons/traefik/compose.yaml:624-651, subsystems/impl/ingress_materialize.go, modules/app/ingress_materialize.go:88,324
leaderrqlite raft; LeaderHub polls the leader id every 5 s and fans OnLeader out; CE has no electorce subsystems/impl/leader_hub.go:246-257, subsystems/impl/leader_elector.go
replicated control plane (rqlite)
deployments, rules, fences, assignments, node leases, revision
▲ writes (API, quorum) │ reads
│ ▼
┌─────┴──────────── leader ───────────────────────┐
│ engine: filter (fences) → fit (capacity) │
│ → spread/apart → score → bind │
│ writes assignment(gen+1) + placement_event │
│ render: Jinja once per replica → spec + digest │
└─────────────────────────────────────────────────┘
│
every node: materialiser (DB → clusterfs, atomic; copies, never renders)
▼
/var/lib/shc/clusterfs (read-only mounts) ── node agent ── executor
assignments, projections, routes, runs its projection, (docker, podman,
certs, resolver zones restarts crashes, incus, apple)
reports observed state

Three parts in the loop, two of them new in kind (the two node-side workers that keep the mesh and the firewall are listed after them):

  1. The engine runs only on the leader, like every other leader-only subsystem (LeaderOnly: true, ce subsystems/lifecycle/subsystem.go:29-45). It holds no state of its own between ticks. Its input is the replicated tables; its output is assignment rows. rqlite forwards any member’s write to the current leader (core/db/rqlitedriver/discovery.go:44-56), so a deposed engine’s write would still commit. Every bind is therefore fenced by an engine epoch: a new engine increments scheduler.epoch when it takes leadership, and each bind statement is guarded by epoch = <mine> as well as a compare-and-swap on the assignment generation. A deposed engine’s binds match no row. It runs on events (install, scale, update, node condition change, fence edit) and on a 5 s safety tick. The leader also renders, once (ruled 2026-10-01): at install or update it evaluates Jinja for every replica, the deployment context plus replica.ordinal, replica.location and replica.node, and stores each replica’s spec in rqlite with a digest (deployment_spec, section 6). Nothing else ever renders, so a daemon upgrade changes no projection until an explicit update, and a new leader holds every stored spec through raft before it serves.
  2. The node agent runs on every node. It reads its own assignments from clusterfs, renders nothing, and runs the projection shcd materialised for it. It owns container restarts and the boot gate (3.9) and the readiness report (3.13). Today’s per-node container recorder (ce subsystems/impl/container_recorder_node.go) is its observation half.
  3. The materialiser runs on every node and writes clusterfs from the control plane (3.24). Today’s ingress sweep, which writes route and cert files into a node-local Traefik directory and SIGHUPs Traefik (ce modules/app/ingress_materialize.go:324, modules/app/traefik_reload.go), is its first consumer and moves onto it. It copies stored specs into projections and derives the Traefik and resolver sections from readiness rows; it never renders and never enriches (3.24).

This is the Kubernetes split (scheduler binds, kubelet runs) with SHC’s fences in place of labels and a file tree in place of the watch API. It is not Swarm’s: Swarm’s managers push tasks to workers over a dispatcher stream, and a worker that loses its manager keeps running tasks with no ownership check at restart.

shcd on a node is four workers (2026-10-01). None of them touches a packet; the data plane is the kernel.

workerreadswrites or does
materialiserSHC’s rqlitethis node’s clusterfs revision: stored specs become projections, readiness becomes the Traefik and resolver sections. It copies; it never renders
node agentits clusterfsruns the four checks below, hands projections to the executor, restarts containers, reads health, reports readiness and the revision it applied to rqlite
mesh coordinatorthe peer table in rqlite, cached locally for bootkeeps mesh0 and shc0 configured over netlink: peers, keys, grants, the best endpoint per peer, a per-route MTU from its probes. Control plane only, no packet path
policy applierthe compiled per-node ruleset in its revisionloads it with one atomic nft -f, verifies the loaded version hash, re-applies on drift. On a Mac the same ruleset comes out as pf

Four checks before the agent runs anything (ruled 2026-10-01), each with a concrete test:

  1. The revision is whole. The materialiser built it aside, verified the store did not move while it read, swapped each section’s current with a rename, and wrote REVISION last (3.24).
  2. The entry’s hash matches the stored spec digest on its assignment (deployment, service, ordinal, generation, digest). A mismatch does not run and reports why; that catches a corrupted disk, a materialiser bug and a hand-edited file alike. The revision carries a digest of its own.
  3. The replica is still this node’s at the current generation and engine epoch, with the lease held for anything exclusive; a file saying “run pg-r1 here” is never enough on its own.
  4. Its cross-node dependencies are ready. A depends_on on a service running elsewhere holds the start until that replica reports ready.

Correct here means intact and still owned, not newest. A node acting on a revision a second old is fine; checks 2 and 3 make sure it can only run what is its own, and the next materialise removes what a newer revision took away.

Replica and assignment. The engine places replicas, not deployments. A deployment with one service and no replicas: has one replica, and the default implicit node colocation of a deployment’s services (SHC-1050 §5.1) puts every one of its replicas on one node, so the common case is today’s shape.

assignment state machine:

pending ──bind──▶ assigned ──agent starts──▶ starting ──ready──▶ ready
▲ │ │ │
│ node lost past its grace, movable unready ◀────────────┘
│ ▼
└──────────── rescheduling ──bind(gen+1)──▶ assigned (new node)
node lost past its grace, unmovable ──▶ down (stays on its node; starts on return)
data unreachable, operator forces ──▶ orphaned ──▶ removed

Node conditions. Two columns exist, cluster_state (membership and health, ce ent/schema/node.go:46-60) and availability (operator intent, :66). The engine reads one derived condition, shown in shc node ls and the HUD:

conditionwhennew replicasrunning replicas
readynode lease fresh, cluster_state healthyyeskept
pressurednode lease fresh, cluster_state degraded with a resource reason (disk, inode, memory)nokept
cordonedavailability=pausednokept
draining, drainedavailabilitynomoved by drain; global replicas stop last
rebootingannounced shutdown or maintenance window, with an expected_back_bynoheld, nothing rescheduled before the deadline
lostnode lease expired (TTL 35 s)noremoved from every endpoint list; single writers there have been told to stop (3.10); each deployment’s reschedule clock runs from the expiry
downlost past every grace that applies to its work (3.8), or a STONITH agent confirmed power offnothe movable ones already rescheduled; the pinned ones shown down

The condition gates new replicas only. A running replica leaves a node through drain, rescheduling, a fence edit (refused while it would strand one, 5.1), or its deployment’s removal, never because the node was cordoned or pressured. That is Kubernetes’ NoSchedule behaviour, and it is what keeps the per-node platform set running on a cordoned node (3.17).

The candidate query admits degraded today (modules/placement/advisor.go:62-67); under this design a degraded node with a pressure reason takes no new work, as a Kubernetes node with a pressure taint does.


Each part says what Kubernetes and Swarm do, what SHC does, what exists, and what is missing.

Kubernetes stores desired objects and every controller converges actual toward them. Swarm does the same for services and tasks inside its raft.

SHC. The desired state is the deployment row, its rules, and its assignments. The engine converges assignments toward the deployment (how many replicas, where). Each node agent converges containers toward its assignments. Neither needs the other to be up: assignments survive a leader change, and a node agent keeps running what it was last told.

existsmissing
the deployment row as desired state; the leader-only reconcile cron runs compose up -d every minute for broken deployments on the leader only and logs X200924RSTRMT for the rest (ce modules/health/reconcile/reconcile.go:252-271, modules/health/reconcile/errors.go:105-128); the leader-only container reconciler reaps orphan rows every 60 s (ce subsystems/impl/container_reconciler.go); a boot pass settles rows parked mid-operation (ce subsystems/impl/parked_stack_reconcile.go)the assignment table; a per-node reconciler, so a broken deployment on a non-leader node is repaired by its own node; the engine loop

Kubernetes filters nodes with predicates, scores the survivors with weighted plugins, and reports “0/5 nodes are available” with a reason per node. Swarm filters by constraints, resources, platform and ports, then spreads by task count.

SHC. Four stages, in order, for each replica:

  1. Eligible set from fences, exactly SHC-1050 §5.1: absolute rules on every scope, resource reach, compiled colocation, and the node condition ready.
  2. Fit: reservation against free capacity (3.3). A hard filter, not a score.
  3. Spread and apart: hard spread and anti-affinity rules (3.4, 3.6) remove nodes; soft ones become score terms.
  4. Score with today’s scorer, unchanged in contract: it ranks and never refuses. Its Decision keeps every term for explain.

A replica with no survivor is pending, with the reason per node that removed it (section 4). An empty eligible set from rules alone is a refusal at admission, before the QUEUED row, as SHC-1050 §3.7 rules. Pending is for “nothing fits now”; refused is for “nothing can ever fit these rules”.

existsmissing
the scorer with memory, cpu, count, tenant and app terms, deterministic tie-break, unknown axes abstaining (modules/placement/placement.go:175,240-300); candidate filter on condition and availability (advisor.go:62-67); memory headroom filter that falls back to all candidates when every node is full (advisor.go:107-131); the keep-off-voters filter (advisor.go:244)fence filtering (SHC-1050 phase 3); fit as a filter that yields pending instead of “no opinion”; per-node reasons; spread as a filter

Kubernetes schedules on requests, enforces limits in the kubelet, and subtracts system-reserved from allocatable. Swarm schedules on reservations and ignores limits for placement.

SHC. Placement uses the reservation; compose applies the limit. Both come from today’s quota chain: one quota per service, limit = quota × deploy.limit_ratio (default 1.5), reservation = quota × deploy.reservation_ratio (default 1.0) (ce modules/app/resource_quota.go:13-14,200, modules/app/resource_reservations.go:125). A node’s allocatable is its total minus a system reserve for shcd, the runtime and the per-node platform set. A node’s committed figure stays derived, never materialised, but it is derived from assignments instead of from the deployment anchor. Today’s ledger charges every service’s full desired replica count to the node the deployment is recorded on (modules/quota/commitment.go:120-132), which is right only while every replica sits on that node. Under the engine:

  • a node is charged, per assigned replica, that replica’s reservation;
  • a pending replica is charged to its tenant’s quota (3.22), never to a node;
  • a surge copy during a start-first update or drain is charged on its node for as long as it exists, so a surge that does not fit waits;
  • a global replica is charged on every node it runs on.

The engine is the only writer of assignments and binds one replica at a time, and each bind is one statement batch that re-reads the target’s committed figure, checks fit, and writes the assignment under the epoch guard (2.2). Two binds cannot both admit against the same free memory, the race usage once lost to eight concurrent charges (the header of modules/quota/commitment.go).

Unknown capacity stays unknown: a node whose memory_bytes is NULL is kept as a candidate and flagged by explain, never read as zero, which is today’s rule (ce ent/schema/node.go:115 comment, modules/quota/enforcement.go header).

existsmissing
per-node cpu_cores and memory_bytes (ce ent/schema/node.go:91,115); the quota chain and reservations derived from limits (ce modules/app/resource_quota.go, resource_reservations.go); the commitment ledger, per deployment anchor (modules/quota/commitment.go); the node gates (modules/quota/enforcement.go:168), uncalleda caller: fit as stage 2 of the engine; a system reserve per node (config scheduler.system_reserved, default 1 GiB + 5 % of memory, 500 m CPU); the commitment derived from assignments, surge and global replicas included; the fit check inside the bind

3.4 Affinity and anti-affinity as fence rules

Section titled “3.4 Affinity and anti-affinity as fence rules”

Kubernetes has node affinity (required and preferred), pod affinity and pod anti-affinity over a topology key. Swarm has placement constraints and spread preferences.

SHC. Every one of these is a rule on an object, in the SHC-1050 vocabulary. SHC-1050 defined two forms; this design adds four. Five of the six narrow and never widen (SHC-1050 §3.3); admit is the toleration of 3.5 and only lifts a reserved fence out of the base set:

formspellingKubernetes analoguejudged on
stay infence=<kind>:<name> (SHC-1050)required node affinitynode membership
stay outavoid=<kind>:<name>required node anti-affinity (NotIn)node membership
colocatecolocate=<plug>:<kind|none> (SHC-1050)pod affinitythe partner’s eligible set, compiled to one derived fence (SHC-1050 §5.2)
apartapart=<deployment>[/<service>]:<kind>required pod anti-affinitythe partner’s current assignments
spreadspread=<kind>[,skew=<n>][,max=<n>][,hard]topology spread constraintthis deployment’s own replicas (3.6)
admitadmit=<kind>:<name>tolerationreserved fences only (3.5)

Ruled 2026-10-01: operators never type spread, apart or a region. The engine derives both from the --scale shape (3.6) and from stateful: true; they stay as its internal vocabulary and in explain. An operator states a rule with one flag per form, each taking <kind>:<name>: --fence, --avoid, --colocate and --admit, plus --location <name> for --fence location:<name> (SHC-1237, ruled 2026-10-02; --rule is gone).

apart is judged on assignments, not eligible sets, because “never share a location” cannot be compiled to one fence ahead of time; it is symmetric, so moving either end re-checks it. Preferred (soft) node affinity is not a rule: the scorer’s tenant and app spread terms already are the soft form, and SHC-1050’s external-socket preference (§3.5) is the one scored location preference.

existsmissing
Swarm constraints rendered by SHC (node.hostname==, voter excludes, RWO pins: ce modules/app/swarm_placement.go:33,90,171); drain’s constraint matcher (modules/cluster/drain.go:1201)the rule store (SHC-1050 phase 2.4); the avoid, apart, spread and admit forms; apart evaluation in the engine

3.5 Taints and tolerations: reserved fences

Section titled “3.5 Taints and tolerations: reserved fences”

Kubernetes taints a node so that only pods that tolerate it land there (NoSchedule), prefers not to (PreferNoSchedule), or evicts (NoExecute).

SHC. This maps cleanly onto one fence attribute, so it lands; the words “taint” and “toleration” do not. A fence can be reserved (shc fence set <fence> --reserved). A node in a reserved fence is outside every deployment’s base candidate set unless one of the deployment’s rules, at any rung, names that fence with fence= (enter it and stay in it) or admit= (may enter it, may also run elsewhere). admit= is the toleration; fence= is a toleration plus node affinity, as a Kubernetes pod that both tolerates a taint and selects the tainted nodes.

  • NoSchedule is --reserved.
  • PreferNoSchedule has no SHC form: a preference that nothing enforces is the free-floating policy the owner ruled out. An operator who wants it writes avoid= on the scopes that should stay off.
  • NoExecute is drain.

Reservation is part of the base candidate set, like the node condition, not a rule, so naming a reserved fence at the deployment rung does not loosen an estate rule (SHC-1050 §3.3 stays true). The keep-workloads-off-voters filter (modules/placement/advisor.go:244) becomes shc fence set control-plane --reserved. A system-priority deployment (3.7) is admitted to every reserved fence without a rule; that is what lets the per-node platform set run on every node, reserved or not (3.17).

existsmissing
filterVoterTier, the only taint-like filter (advisor.go:244)the reserved fence attribute, its base-set rule, admit=, the system-priority admission

Kubernetes spreads by topology spread constraints (max skew, per topology key, DoNotSchedule or ScheduleAnyway) and gives StatefulSet pods a stable name and their own volume claim. Swarm spreads replicas by node and by placement.preferences.

SHC. Replicas across nodes, per the owner’s 2026-09-29 answer:

  • deploy.replicas: N (or --scale <service>=N on shc install or shc update) asks for N replicas of a service. The engine places each one independently inside the service’s eligible set. There is no shc scale verb (ruled 2026-10-01).
  • Scale is one flag with three shapes (ruled 2026-10-01), persisted as a deployment override and capped by the app’s x-shc-scale.max: --scale web=3 is three replicas, one per node where possible, anywhere the deployment is allowed; --scale pg=eu:2,us:1 is exact counts per location, with the total, the allowed locations and the spread by location implied; --scale proxy=per:location is one in every value of that fence, following the estate as it grows. Fence kinds are location and provider; “region” is a name someone gives a location. No home key and no failover key: a primary stays where it was last promoted, nothing fails back on its own, and which side may promote automatically is quorum arithmetic on the counts (eu:2,us:1 recovers on its own when us is lost and needs a manual promote when eu is; three locations make it automatic everywhere). A deployment is addressed with -s/--stack, -e/--environment and the tenant flag or shc ctx, never a tenant/environment/stack path. Lands in SHC-1145 and SHC-1146.
  • One compose entry per replica, always (ruled 2026-10-01). The projection carries one entry per replica on that node, named <service>-r<ordinal>, each a single container. Compose’s own replicas: is never used: it cannot give each copy its own pin, environment or volume, and the runtimes disagree about it. The bare service name is a local alias only when every replica of the service is on this node; otherwise it falls through to the resolver (3.15). Each service numbers its replicas from 0. One copy on one node is the same code path as nine across three locations.
  • Spread. For kind K, the domains are the K fences that meet the eligible set. Skew is the largest replica count in a domain minus the smallest. spread=K,skew=1 keeps skew at or under 1: as a score term by default, as a filter with ,hard. max=<n> caps the replicas in any one K domain and is always hard; it is the only way to say “one per node”, because a skew of 1 still allows three replicas on two nodes as 2 + 1. A node in no K fence is not a candidate under a hard K spread or a cap, and explain names it. A replicated service with no spread rule gets an implicit soft spread=node,skew=1.
  • Members. A service declaring x-shc-scale: {stateful: true} gets StatefulSet semantics: replica <service>-r<ordinal>, its own volume per ordinal, its own pinned fabric address per ordinal, ordinals started in order, and an implicit spread=node,max=1: one ordinal per node, so a 3-ordinal service needs three eligible nodes. Replication between them stays the app’s job. Ruled 2026-10-01: stateful: true replaces this design’s identity: stable, with no alias; since every ordinal sticks anyway (above), the flag means distinct members: own volume, own name, one per node, started in order. The key is gone since: distinct members are a {% members %} group (groups proposal), whose members prefer separate nodes through an implicit soft spread=node,skew=1 counted over the group. A group whose block says separate="required" takes spread=node,max=1 over the group instead: no node takes two members, on install, update, failover, a move or a rebalance, and a member with no node of its own is pending with reason separate. An install that asks for more members than the eligible nodes is refused, not left pending (X409184FNCSEP).
  • Volumes. A replicated service may mount RWX storage (a shared filesystem). An RWO volume with replicas > 1 and no stateful: true is refused at render, which keeps today’s rule that RWO is single writer (ce modules/app/swarm_placement.go:150-170). No replicated volumes in the first cut (ruled 2026-10-01): local volumes, restic, failover by restore; LINSTOR comes later as a per-site capability.
  • Ceilings. x-shc-scale.max stays the one ceiling every replica change consults (ce modules/app/replica_ceiling.go, modules/app/deploy_overrides.go:28-50).
existsmissing
deploy.replicas and --scale, all on one node (ce modules/app/services_writer.go:181-189, deploy_overrides.go:80-83); the x-shc-scale schema with max, backup, restore (ce modules/app/scale_extension.go); 97 services in 73 catalog apps declare x-shc-scale.max: 1; one catalog service uses replicas (apps/standalone/pulp/compose.yaml:322)per-replica assignments and one entry per replica; the three --scale shapes; spread; stateful: true with per-ordinal volumes and pins; the pin table keyed by ordinal (today (deployment, service), ce modules/fabric/pins.go:17-19,86-101)

Kubernetes evicts lower-priority pods to fit a pending higher-priority one. Swarm has neither.

SHC. Priority, no preemption. A deployment carries priority: system (the builtins), high, normal (default) or low. Beyond admitting system to reserved fences (3.5), it only orders work:

  • after a node goes down, rescheduling runs in priority order, then by today’s system-first order (ce modules/health/reconcile/system_first.go);
  • drain moves in priority order;
  • a pending queue is retried in priority order.

Preemption is refused because an SHC move is a stateful cutover (backup, ship, restore, health gate; 38 to 51 s measured per relocation, modules/cluster/drain_disruption.go:14-16), not a pod restart. Evicting a database to fit a web app is never the right trade on these estates. Explain prints what would fit if something moved, as a hint only.

existsmissing
system-first reconcile orderthe priority column and its three uses

3.8 Node health: heartbeat, lease, grace; dead, restarting or powered off

Section titled “3.8 Node health: heartbeat, lease, grace; dead, restarting or powered off”

Kubernetes: each kubelet renews a Lease every 10 s; the node goes Unknown after node-monitor-grace-period (50 s by default); pods get the default 300 s not-ready/unreachable toleration before eviction. Swarm: workers heartbeat the dispatcher every 5 s; a node is down after missing a few, and its tasks are rescheduled at once.

SHC. Two signals, two jobs, and a grace per class of work.

  • The node lease (new) is the liveness truth. Every node’s shcd renews one node_lease row every 5 s with a quorum write, TTL 35 s, the same cadence as today’s attachment lease (ce subsystems/impl/volume_lease.go:68-82). A node that cannot commit its renew knows it is cut off; a leader that sees the lease expire knows the node is cut off or dead. Neither can tell which, which is why the self-stop exists (3.10).
  • The poller keeps its job: health detail (disk, inode, memory, certs) and the fast unreachable stamp (3 failed polls on a 2 s tick, ce modules/health/node_poller.go:396-397, modules/health/crons.go:81). It no longer decides liveness for the scheduler: a poll can fail on a path the node’s raft traffic does not use.

Dead, restarting, powered off.

what happenedwhat SHC seeswhat the engine does
announced reboot (shc node maintenance, or shcd stopping during a system shutdown)the node writes rebooting with expected_back_by before it goesholds everything until the deadline (default scheduler.reboot_window, 5 min); reschedules nothing
unannounced crash or rebootlease expires; the node is lostreschedules each deployment by movability at lease expiry + its grace
powered off, confirmeda STONITH agent reads the power state back as off (the Hetzner agent does this today, ce modules/storage/providers/hcloud/stonith.go)down at once; the grace is skipped because the old node cannot write
alive but partitionedlease expires at the leader; the node itself fails its renewsthe node starts stopping its single writers after two failed renews (3.10); the engine treats it as lost and runs the same clocks

Grace. One clock per deployment, started at the node’s lease expiry. When it runs out the deployment is rescheduled at once; there is no second wait. The node’s condition turns down when the longest clock on it has run out, or at once on a STONITH confirm.

classdefaultconfig keywhy
stateless (no persistent data)60 sscheduler.grace.statelessa replacement is a compose up elsewhere; other replicas usually serve meanwhile
stateful, reattachable5 minscheduler.grace.statefula move is a rebind or a backup+restore, and most reboots finish inside 5 min; Kubernetes’ default toleration is the same 300 s. Never shorter than the single-writer stop window of 3.10, which the engine enforces

The keys resolve through the scope cascade (core/scope), so a deployment can shorten its own grace. They are settings, not placement rules, so the owner’s no-free-floating-policies ruling does not reach them.

existsmissing
poller and unreachable stamp; joining/healthy/degraded/unreachable/leaving/removed/lease_pending (ce ent/schema/node.go:60); attachment lease; STONITH seam with the Hetzner agent; containers marked unknown once per death episode (ce modules/health/node_poller.go:144-160)the node lease; the rebooting state and expected_back_by; lost and down conditions; grace keys; an announced-shutdown hook in shcd; shc node maintenance

3.9 Rescheduling, and a node that comes back

Section titled “3.9 Rescheduling, and a node that comes back”

Kubernetes recreates a Deployment’s pods elsewhere after the toleration; StatefulSet pods are never recreated until the old pod is confirmed gone. Swarm reschedules every task of a down node at once and, when the node returns, shuts down the stale tasks it finds there.

SHC. When a node’s lease expires, each deployment on it is handled by its movability, which the engine derives and explain prints. “Clock” is the deployment’s grace from 3.8, started at the lease expiry:

movabilitywhenwhen its clock runs out
freeno persistent mountsnew assignments inside the eligible set; nothing moves, because there is nothing to carry
reattachits data is on RWO block volumes whose reach holds another eligible node, or on a shared filesystemmoved by the dead-source move (--presumed-dead flow), which rebinds the surviving volumes behind a lease-generation bump; the stateful clock is never shorter than the single-writer stop window (3.10, decision 25)
pinneda local volume or host bind, a fence=node:<n> rule, a volume whose reach is that one node, or inputs the control plane does not hold (below)stays assigned, shown down, starts when the node returns
fara volume whose reach holds no node of the control-plane fence (decision 24)stays assigned; taken over only on a STONITH confirm or an operator’s --presumed-dead

Failover by restore (2026-10-01). With no replicated volumes in the first cut, a deployment on local volumes fails over by restore: its latest verified restic backup restored onto an eligible node through restore-as-move (shc recover, 3.25), losing the writes since that backup. pinned means the engine never does that on its own, because the data sits on the lost node and the owner’s rule is that unreachable data waits for its node; it does not mean the deployment cannot be recovered. Explain prints a pinned deployment’s last backup, its age and the restore command, and an operator starts it.

The dead-source move stays the one mover. The engine calls the same PresumedDeadProvider the CLI path uses (ce modules/clone/types.go:1145-1149), with the target it chose, instead of building a second flow. Its two gates stay: the node is lost past the clock (the node lease replaces the unreachable stamp as gate 1’s first half) and a fresh probe fails; local mounts are refused, so a pinned deployment never reaches it. Two things it lacks today:

  • A replica operand. The provider plans and installs a whole deployment and repoints its node (ce modules/clone/move_presumed_dead.go:135-160). That is right for the default shape, every service on one node. A stateless replica of a split deployment needs no move at all, only a new assignment. A stable-identity ordinal on its own volume needs the move scoped to that ordinal, so its healthy siblings’ volumes are not rebound.
  • Lossless inputs. Today’s reconstruction states that it loses published --port mappings and install-time --var overrides (ce daemon/clone_dead_move_adapters.go:342-345). An unattended move must not silently come back on app defaults. The new line stores the full effective specification in the replicated store (6); until a deployment’s inputs are all there, the engine classes it pinned and explain says which input is missing.

No flip-flop. Today an interrupted move is reverted by default, and one that already tore its source down is refused for unattended recovery; only a rename finishes forward (ce modules/clone/move_journal.go:598-612). An interrupted dead-source move leaves the rebound volumes on the target and asks for a retry (ce modules/clone/move_presumed_dead.go:321). So the engine records the chosen target on the assignment before it starts the move, and a new engine after a leader change retries the same move to the same target until it lands. A node that returns mid-move is not a reason to stop: its lease generation is already stale. When a node returns:

  1. Its movable deployments already live elsewhere. The node agent finds their containers without a current assignment and removes them (compose down of that projection, volumes kept for shc reclaim).
  2. Its pinned and far deployments are still assigned to it and start after the boot gate below.
  3. Its freed capacity takes new work. Nothing moves back unless an operator moves it or runs a rebalance (3.19).

The boot gate (the owner’s safety requirement). No SHC-managed container may be started by the runtime’s own restart policy at boot, because the node cannot yet know what it still owns. So:

  • The runtime must see no restart policy from either source. Compose lets deploy.restart_policy override the top-level restart: (ce modules/app/restart_effective.go:1-30), so the render stamps restart: "no" and removes deploy.restart_policy from the runtime’s document, keeping today’s rule that strips the Swarm stanza. The authored policy from both keys, resolved as X200932RSTDIA resolves a conflict today (ce modules/app/errors_reboot.go:12-31), is recorded as labels for the node agent. This reverses ApplyRestartDefault, which supplies unless-stopped to every compose and podman service that declares no restart: (ce modules/app/restart_default.go:26-47).
  • The node agent owns crash restarts: on a container die event (ce subsystems/impl/docker_events.go already streams them) it applies the recorded policy, and only while clusterfs still assigns the replica to this node and, for a single writer, its lease is live. Compose’s restart: values keep their meaning, with Docker’s own backoff (100 ms doubling to 1 min, reset after 10 s up). deploy.restart_policy keeps its fields: condition (none, on-failure, any), delay as the first backoff step, max_attempts as the retry cap, and window as the time a restart must stay up to count as a success.
  • On Podman, podman-restart.service stays disabled; nothing in either repo enables it today.
  • At boot the node agent starts nothing, the per-node platform set included, until it has (1) re-read its assignments from the leader with a linearizable read and re-materialised clusterfs, (2) acquired its node lease, (3) re-acquired each assigned RWO volume’s attachment lease at the current lease generation. This is decision 23 read literally: the platform set holds no data, but whether this node is still a member and still assigned the platform set is exactly what it cannot know yet. A bumped generation means another node took the volume over, so the local copy is removed, not started. shc node status on that node says what it is waiting for. Every start after the gate still passes the four checks of 2.2.

This also retires two workarounds that exist because the runtime’s restart policy runs unsupervised: the operator step “run docker compose down on the returned node” (ce modules/clone/move_presumed_dead.go:43-51) and the reason compose stop in a backup needs a journal to come back (ce modules/backup/stop_journal.go:1-30), which becomes a node-agent restart.

existsmissing
the dead-source move and its gates; the move journal; boot hydration that skips rows homed elsewhere (ce modules/clone/move_presumed_dead.go:43-45); Docker events on every node; the rule that strips deploy.restart_policy under compose (ce modules/app/restart_effective.go)movability derivation; engine-driven dead-source moves with roll-forward; a replica-scoped dead-source move; the replicated full specification; stale-copy removal; the boot gate; node-agent restarts; restart: "no" at render

Kubernetes StatefulSets wait for the old pod to be deleted, which a partitioned kubelet never confirms, so an operator force-deletes or a fencing agent confirms. Swarm has no guarantee: a partitioned worker keeps its task while the manager starts another.

SHC. Three layers. The first two are today’s code, extended; the third is new, because the first two cannot prove a stop on their own.

  1. The node stops itself. The self-fence loop, renamed lease loss, starts stopping every container of a deployment holding an RWO attachment lease after two failed renews (ce subsystems/impl/volume_lease.go:14-22,68-82). The stops are sequential, bounded at 10 s per container, and a failure only logs (ce subsystems/impl/volume_lease_node.go:134-177). This design extends the same stop to every single-writer service: one with an RWO volume, or one whose x-shc-scale.max is 1, the data-safety ceiling 97 catalog services already declare. A single writer on host binds has no attachment lease to lose, so the node lease stands in: failing to renew it twice stops it. The attachment leases stay the authority for volumes; the node lease is the authority for liveness and for volume-less single writers. Decision 24’s far-location exemption still applies.
  2. A watchdog outside shcd. The loop above runs inside shcd, so a stopped, hung or suspended shcd renews nothing and stops nothing while its containers keep writing. A small systemd unit on every node, separate from shcd, reads the lease expiry shcd writes to a local file after each committed renew. When that expiry passes, it stops the node’s single-writer containers itself, escalating to a kill, and records what it did. It needs no network and no control plane.
  3. The engine waits for the stop, and prefers proof. A replacement for a single writer is bound on a STONITH confirm, or only after the old node’s lease has been expired for longer than TTL + grace + the stop budget (10 s times the node’s single-writer containers, plus the watchdog’s escalation), measured as elapsed time from the expiry, not as a count of ticks. Today’s compile-time assertion checks only dwell > TTL + grace (subsystems/volume_failover.go:108), and the dwell is three counted ticks rather than an elapsed interval (:465-500); both are fixed as part of this. The attachment lease generation bump is what refuses a stale remount if the old node was only slow (ce modules/clone/move_presumed_dead.go:36-41).

Lease margins (ruled 2026-10-01, the SHC-1182 fixes): the holder stops well before anyone else may take over, on a monotonic clock, so a paused VM that wakes past expiry has already lost; a lease lifetime is longer than a leader election, so an election stops nothing; a new leader honours every existing lease for one full lifetime. Proven on the rig before the cutover.

What remains is one case no timer closes: a node frozen whole (a suspended VM) resumes after the takeover with its containers mid-write, before its watchdog notices. Only a STONITH confirm excludes that. So a STONITH agent is used wherever the provider has one (Hetzner today), and a lease-expiry takeover is allowed only where it does not, and only with the watchdog installed (decision S16).

Stateless replicated services get no such wait; two copies of a stateless web process for a few seconds is harmless.

existsmissing
the lease-loss loop and its timings; the failover dwell and its compile-time assertion; lease-generation-gated mounts; the Hetzner STONITH agentself-stop for single writers without a volume; the watchdog; an elapsed-time wait that includes the stop budget

3.11 High availability: stateless and stateful

Section titled “3.11 High availability: stateless and stateful”
statelessstateful, single writerstateful, clustered
shapeN replicas, soft spread by nodeone replicaN replicas with stateful: true, one per node
node lossthe survivors serve at once (endpoints drop the lost replica within the lease TTL); a replacement 60 s after the lease expireddown for the grace plus the move (5 min + rebind), never while the old node could still write (3.10)the app’s own replication fails over; SHC replaces the lost ordinal on its own volume only if the volume is reattachable, else waits
updaterolling, start-first allowedstop-first onlyrolling by ordinal, stop-first
drainsurge on the target, then stop on the sourcemove; --ha refuses a gap (today’s budget)one ordinal at a time

“SHC has no HA” is true today and written into the scorer’s comment (modules/placement/placement.go:160); this table is what replaces it.

3.12 Partitions, leader election, split brain, controller HA

Section titled “3.12 Partitions, leader election, split brain, controller HA”

Leader election is raft’s (rqlite). The LeaderHub polls the leader id every 5 s and fans OnLeader out (ce subsystems/impl/leader_hub.go:246-257); the engine is one more leader-only subsystem. On a leader change the new engine re-derives everything from the tables; there is no in-memory queue to lose. The pending queue is a view over assignment.state = pending.

Split brain. Voters live in the control-plane fence (SHC-1050 §3.9); non-voters cannot be elected; a minority cannot commit. A deposed engine on the majority side could still reach the leader, because rqlite forwards writes, so every bind carries the engine epoch (2.2): the new engine’s epoch bump makes the old engine’s binds match no row. Epoch and generation together mean one engine binds at a time and never on a stale read of the assignment. A partitioned node cannot renew its lease, stops its single writers (except decision 24’s far volumes), and keeps serving stateless work from its last clusterfs; its routes are frozen until the heal (SHC-1050 §5.8). That the minority side stops its stateful work is intentional (2026-10-01): the price of never running two primaries, and it means a two-node estate is not highly available for stateful work. shc doctor warns when stateful work runs with fewer than three voters; a witness node with no workloads is the cheap third.

Controller HA is therefore the control plane’s HA: with 3 voters, one loss keeps the engine running after one election plus one LeaderHub poll.

existsmissing
raft, the LeaderHub, leader-only subsystems, the control-plane fence designthe engine epoch and the epoch-guarded bind; the LeaderHub poll lowered to 1 s once the engine rides it (decision S12)

3.13 Health checks: startup, readiness, liveness

Section titled “3.13 Health checks: startup, readiness, liveness”

Kubernetes has three probes. Docker has one healthcheck with a start_period; plain Docker never restarts an unhealthy container, Swarm replaces it.

SHC maps the one healthcheck onto all three:

  • startup is start_period: failures inside it do not count.
  • readiness is healthy: a replica joins Traefik backends and resolver answers only when healthy, and leaves them when unhealthy. A service with no healthcheck is ready once it is running (ruled 2026-10-01; this replaces the earlier rule that it waited out the update monitor period). 93 of the 106 catalog apps declare a healthcheck.
  • liveness keeps Swarm’s effect: a container unhealthy for x-shc-health.restart_after (default 5 min, 0 disables) is restarted by the node agent. The default is longer than Swarm’s immediate replacement on purpose: the one measured case on these estates was a container unhealthy for 13.8 hours that nothing restarted (ce subsystems/impl/container_unhealth_watch.go:1-12), and a database crash-looping on a slow recovery is the failure a short timer invites.

Ruled 2026-10-01: the healthcheck is defined once, in compose; run by the runtime (on apple, which has none, the node agent runs the test command with container exec on the declared interval); read by the node agent; recorded per replica and summed per service. Readiness flips are dampened (several failures to go unready, several successes to come back), coalesced to at most one revision a second, and Traefik and the resolver reload only when their section’s digest changed; a replica that flaps repeatedly is held out with a visible condition (3.26). Ruled 2026-10-18: a replica flaps once its readiness flips 4 times in 10 minutes. It is held out only while another replica of its service that does not flap is ready, so the only ready replica of a service keeps serving, and it comes back after 5 minutes of ready reads, an unready read starting them again. Held out or not, shc doctor names it (deployments.replicas-flapping, condition deployment.ReplicaFlapping) and its service’s health reads flapping.

existsmissing
compose up --wait gating install on health, podman emulated (ce modules/app/compose_driver.go:79-120); the unhealth watcher, which reports and does nothing else (ce subsystems/impl/container_unhealth_watch.go, modules/app/unhealth.go); readiness dampened, coalesced to a revision a second and reloaded on a changed section digest, and a flapping replica held out (ce modules/app/readiness_damper.go, modules/app/holdout, modules/clusterfs/materialiser.go)readiness feeding endpoints; the liveness restart; x-shc-health

Kubernetes: maxSurge, maxUnavailable, readiness-gated, kubectl rollout undo. Swarm: update_config (parallelism 1, delay 0 s, failure_action: pause, monitor 5 s, max_failure_ratio: 0, order stop-first by default) and rollback_config.

SHC honours update_config and rollback_config with Swarm’s defaults. An update is a durable rollout row: deployment, from-spec, to-spec, the batch in progress, state. The engine drives it replica by replica: move a batch of parallelism assignments to the new spec, wait for each to be ready and to stay ready for monitor, wait delay, next batch. A leader change resumes the rollout from its row. A rollout that cannot finish because a node is down stays pending, which is correct; hanging is not (2026-10-01). The command returns after a timeout saying which replicas moved and which node has not reported, and the rollout resumes on its own when the node returns or rolls back on request.

  • order: start-first creates the new replica as a second incarnation of the same (deployment, service, ordinal) beside the old one, charged on its node (3.3), and retires the old incarnation once the new one is ready. It is refused at render for a single writer.
  • A replica that never becomes ready fails after its healthcheck’s start_period plus monitor, bounded by timeouts.compose_converge (900 s today, ce core/timeouts/timeouts.go:343).
  • On failure past max_failure_ratio, pause stops and reports, rollback re-applies the from-spec to the batches already done, and continue carries on.
  • shc update <deployment> --rollback[=<digest>] is the rollback on request. It makes a kept spec current again, the one the current spec replaced or the one the digest names, and writes the deployment row’s settings back from that spec in the same transaction. A rollout still running or paused is turned around under rollback_config; after a finished one, a new rollout takes the replicas to the kept spec under update_config. It puts the spec back, not data: volumes keep what the newer spec wrote, and the plan says so. shc backup restore is the data path. A bare rollback with nowhere to go, a deployment that keeps one spec included, is refused X404547DSPRBG naming --rollback=<digest>.

The from-spec has to be kept, and today nothing keeps it: the converge journal’s before-image is the list of containers that were running, not the specification that produced them (ce modules/app/converge_journal.go:90-114), and a converge overwrites the rendered artifacts (ce modules/app/converge_restore.go:40-50). The new line stores each deployment’s effective specification by digest in the replicated store and keeps the last two, which is also what makes the dead-source move lossless (3.9). A single-replica deployment on one node, the common case, updates exactly as today: one batch, stop-first.

existsmissing
compose up -d --wait over the whole project; restart of containers a failed converge stopped (ce modules/app/converge_restore.go); the container before-image journal; settings rollback for an abandoned deploy (ce modules/app/service.go:927); update_config and rollback_config are accepted keys (ce modules/app/package_trust_compose.go:293-296) and used by no catalog appthe rollout row; batching by replica; incarnations for start-first; the monitor window; retained specifications and rollback to them

3.15 Service discovery when a replica moves

Section titled “3.15 Service discovery when a replica moves”

Kubernetes updates EndpointSlices; clients reach a stable ClusterIP. Swarm gives each service a VIP on the overlay, or DNS round-robin with endpoint_mode: dnsrr.

SHC. The per-node resolver (SHC-1050 decision 26) answers a service name with the fabric addresses of that service’s ready replicas, across nodes, from clusterfs. A reschedule changes the answer as soon as the new assignment is ready and materialised; a lost node’s replicas drop out within the lease TTL. Answers carry a 5 s TTL.

Names: the resolver serves the names the fabric bakes into extra_hosts today (ce modules/fabric/pins.go:17-19), so nothing an app references changes. A projection sets dns to the node resolver and dns_search to the deployment’s zone and the environment’s zone (<stack>.<environment>.<tenant>.shc.internal, <environment>.<tenant>.shc.internal), so a bare web and a plug’s proxy.postgres both resolve, and a name the compose network does not hold falls through the runtime’s embedded resolver to the node resolver. Compose adds a service’s own name as a network alias whatever the authored aliases say, so every entry is named <service>-r<ordinal> (always, ruled 2026-10-01; S11 said only when replicas span nodes) and the bare name is a local alias only when every replica of the service is on this node. The local alias never hides remote replicas. Only the SHC resolver filters by readiness; the local alias does not, so a local copy that is unready still answers on it.

The resolver itself (ruled 2026-09-30): a few hundred lines of Go on miekg/dns, one per node, not CoreDNS. It answers <service>.<stack>.<environment>.<tenant>.shc.internal with the ready replicas’ pins, nearest first, TTL 5 s; <node>.nodes.shc.internal with a node’s mesh address; a routed hostname with the ready edges of the app’s location, empty where the location has none (ce modules/resolver, zone shc.internal). Everything else is forwarded to the host’s nameservers from the same section. It never dials shcd and keeps answering from its last revision through a partition. Wiring it into containers (dns, dns_search) per runtime belongs to the executor.

A stable virtual IP per service (a per-node L4 proxy) is the later option the owner named. Until it exists, the resolver’s shape is dnsrr.

existsmissing
pinned fabric addresses per service; render-time extra_hoststhe resolver (SHC-1050 phase 8.4); readiness-filtered answers; dns_search in projections

Swarm publishes a port on every node and routes it to a task anywhere (mode: ingress). Kubernetes uses a Service plus an ingress controller.

SHC. HTTP and TLS go through a Traefik on every node (SHC-1050 §3.13), whose routes and backends come from clusterfs: for each host, the fabric addresses of the ready replicas, preferring the Traefik’s own location. The engine is the source of the backend list; readiness and drain remove a backend before its container stops. Traefik’s own loadBalancer.healthCheck stays on as a second guard for the seconds a node’s copy lags. A Traefik per node works now because it reads only clusterfs files that are identical on every node, its backends are pins any node reaches, and certificates are issued once by the daemon-side ACME source and delivered everywhere; the old singleton existed because Traefik read local Docker labels and the routing mesh carried 80 and 443 to it (2026-10-01; new-line simplification §3.2).

Published ports (10 catalog apps carry ports: blocks: mailcow, gitea’s ssh, netbird’s signal and relay, kubo, ethereum, keycloak’s ldap, the traefik edge). Under Swarm a published port, short form included, rides the routing mesh and answers on every node. No SHC mechanism forwards one port from every node, so reading the short form as host publication would drop that reachability without saying so, which decision 21 forbids.

Ruled 2026-10-01, superseding the three bullets this section carried (long-form mode: host honoured as host publication, short form and mode: ingress refused, the five-app sweep of SHC-1081): an app never publishes a host port by itself. ports: and expose: in compose both declare only the container ports a service listens on; a ports: entry is treated as expose: and its host side is not honoured. To keep decision 21 (nothing dropped in silence), the render warns when it discards an authored host side and names --port; that warning is this design’s reading, not part of the ruling. Publication is a deployment decision: --host binds a service to a hostname through Traefik as a route file; --port publishes on the node, or through a load-balancer connection with conn@, and is the only thing that renders a ports: entry (long form mode: host) on the replica’s compose entry, with the edge and DNS for the name following the placement. Tenant reach is by policy. Not ruled: where an app’s default mappings live, and --port on a service with more than one replica (new-line simplification §9).

existsmissing
one Traefik, Docker-label provider plus file provider (ce addons/traefik/compose.yaml:624-651); the fabric’s per-node Ingress class (ce modules/fabric/policy/model.go:117-147); --host and --port on the deployment (ce modules/app/port_mappings.go)Traefik on every node from clusterfs (SHC-1050 phase 8.1); backend lists from assignments; ports: read as a declaration with its warning; --port rendering the mode: host entry on the replica

Kubernetes DaemonSets and Swarm mode: global run one copy per eligible node and follow nodes in and out.

SHC. deploy.mode: global is one replica per node of the eligible set. Creating and keeping are separate questions, as they are for every replica (2.3). The engine creates a global replica on a node that is in the rule-derived set and ready. It removes one only when the node leaves the rule-derived set (a fence edit, a node removed) or is drained. Cordon, pressure and a lost lease do not remove it, so cordoning a node never takes its ingress or resolver away. Drain does not move a global replica: it stops last, when the node is empty, as kubectl drain --ignore-daemonsets leaves DaemonSet pods.

The per-node platform set (Traefik, forward-auth, resolver; decision 26 keeps the builtin set at four apps) is a global deployment in the master tenant with priority: system, admitted to every reserved fence (3.5). It waits at boot like everything else (3.9). The other builtins (keycloak, openobserve, postgres) are ordinary deployments; keycloak and postgres belong in the control-plane fence’s location (SHC-1050 §3.15).

global-job is a global service whose replicas each run to one clean exit, a node that joins later included, and replicated-job the same on its deploy.replicas replicas. Neither is a job (3.21).

existsmissing
a single Traefik host (ce modules/app/ingress_materialize.go:88); Swarm global mode, which no catalog app usesglobal mode in the engine; the platform set as a global deployment

3.18 Drain, maintenance, disruption budgets

Section titled “3.18 Drain, maintenance, disruption budgets”

Kubernetes: cordon, kubectl drain with PodDisruptionBudgets. Swarm: node availability drain evicts every task at once.

SHC. Today’s EE drain keeps its shape (journal, pin gate, --ha as the disruption budget, serial moves, resume) and changes its source of truth:

  • targets come from the engine, per replica, inside each eligible set, not from docker service inspect (SHC-1050 phase 3.5);
  • a stateless replicated service drains with surge: the new replica is ready on the target before the old one stops;
  • replicated services get an implicit budget of max_unavailable: 1, declarable as x-shc-disruption.max_unavailable; --ha keeps its meaning for single-replica deployments;
  • global replicas stay until the node is empty.

Maintenance. shc node maintenance <node> --window 30m [--drain] cordons the node and announces it rebooting with expected_back_by = now + window, so the engine reschedules nothing from it inside the window. With --drain it drains first. shc node maintenance <node> --end lifts it; the node’s return (3.9) ends it too.

existsmissing
drain, undrain, cordon, the budget, the pin gate, the journal (modules/cluster/drain.go, drain_disruption.go, drain_pin_gate.go, drain_journal.go, cordon.go)engine targets; surge drain; max_unavailable; shc node maintenance

Kubernetes has none by default (the descheduler is an add-on). Swarm never rebalances running tasks.

SHC. None automatic, as the rebalancer v1 ruling set it (docs/decisions/2026-08-11-rebalancer-v1-foundations.md §0) and the owner repeated on SHC-1058: nothing moves back except by operator request. shc rebalance --plan prints the moves that would reduce skew and free the most constrained nodes, each with its explain; --apply runs them through the ordinary move. A rebalance rule attached to a scope is a later option (decision S10).

The one exception: group members move apart (Clay, 2026-10-01). The leader’s separation pass (members.separate, shc modules/separation) makes the first moves a rebalance would make, one member off each node that holds more than one member of a group, without an operator.

  • What arms it. A node joins, a replica leaves a ready node, or a node comes back after a failover moved a replica off it. Nothing else does. The pass’s first read is a baseline, so a new leader, an shcd restart and a daemon upgrade arm nothing. A group already sharing a node stays put and reads “not redundant” until one of those events or an operator’s shc cluster rebalance.
  • What holds a move. The cluster must have been settled 10 minutes (no node joined, left or changed readiness), the freeze must be open and the move limiter must have room. No rollout may be under way, and every other member of the group must be ready. A member whose durable data stops it while it copies moves on its own only up to 20 GiB. Bigger copies need the operator’s rebalance, which shows the plan and its downtime.
  • Pacing. One separation at a time across the cluster, announced 15 minutes ahead in shc status and shc doctor (“db-2 moves to n3 at T”). A group waits 10 minutes after a move, and a failed move backs off an hour.
  • Leader change. It drops the armed pass and the announced move. The next event re-arms it, and doctor’s “not redundant” line covers the gap.

The kubelet evicts pods under memory, disk or PID pressure. Swarm does not.

SHC. No eviction. A node under pressure becomes pressured (2.3’s table) and takes no new work; the kernel’s OOM killer enforces memory limits, and the disk crons keep pruning (ce modules/health/crons.go:198-200, CronDiskAutoPrune, CronImagePrune). Evicting by priority would be a preemption by another name (3.7).

Today. The leader’s schedule dispatcher enqueues task rows (ce subsystems/impl/schedule_dispatcher.go); workers claim them with a compare-and-swap that honours affinity_node_id (ce modules/task/service.go:434-447, ent/schema/task.go:78); a stale claim is released after 60 s (ce modules/task/worker.go:77-78); a container job runs docker compose run --rm on the claiming node (ce modules/job/context_container.go). Backups run on the node that holds the volume.

SHC. A job is placed by the engine like a replica and gets the chosen node as its affinity_node_id. A job of a deployment colocates with the deployment’s node by default, since that is where its volumes are. A task whose affinity names a node that has been lost past the stateless grace is re-placed instead of waiting forever (today it waits). Swarm’s replicated-job and global-job run as replicas, not jobs (3.17). Schedules stay keyed by scope, not node (SHC-1050 §3.16), so a reschedule changes nothing about when they fire.

existsmissing
dispatcher, task affinity, stale-claim recovery, container contextengine placement for jobs; re-placement off a down node; the two job modes

Today. Caps per stack, environment and tenant, enforced before the runtime moves, charged on render as quota × replicas (ce modules/app/quota_seam.go:41-67,117, modules/quota/).

SHC. Unchanged in shape. The charge is the desired replica count, charged at admission, so a pending replica is charged; a global deployment is charged per node it runs on. A replica refused by a tenant cap is refused, not pending (reason quota).

3.23 Swarm compose keys: honoured or refused

Section titled “3.23 Swarm compose keys: honoured or refused”

The rule (owner, 2026-09-29): a key that had an effect under Swarm keeps it or is refused at render, by name. Ruled 2026-09-30 (new-line simplification Q1): a key with a native form is translated and the render warns, naming the native form; a key with none is refused. Counts are over the 106 apps at apps/<tier>/<app>/compose.yaml.

keycatalog useunder SHC
deploy.replicas1 appreplicas across nodes (3.6)
deploy.mode: replicated / global0engine placement / one per eligible node (3.17)
deploy.mode: replicated-job / global-job0replicas that each run to one clean exit: deploy.replicas of them / one per eligible node (3.17)
deploy.placement.constraints0rules: node.hostname==X → fence=node:X, node.labels.K==V → fence=K:V, != → avoid=, each with a warning naming the native form; anything else refused (SHC-1050 §3.7). node.role==manager is refused too: there are no managers, and the control-plane fence holds non-voters as well as voters (SHC-1050 §3.9), so fence=control-plane would not mean the same thing
deploy.placement.preferences (spread: node.labels.K)0spread=K (soft), with a warning naming it
deploy.placement.max_replicas_per_node0spread=node,max=<n> (3.6), with a warning naming it
deploy.resources.limitsmost appscompose applies them
deploy.resources.reservations4 apps (onlyoffice, openarchiver, pulp, stirling-pdf)capacity (3.3); where absent, derived from the limit as today (ce modules/app/resource_reservations.go)
deploy.resources.reservations.devicesjellyfin, ollama (as runtime: nvidia or commented out)an NVIDIA GPU (capabilities: [gpu], driver nvidia or none) is the device fence: the replica needs a node whose capabilities record gpu-nvidia, and incus and podman get it as a CDI device; any other device is refused (NODEFACTS)
deploy.resources.reservations.generic_resources0refused: no fence for it
deploy.restart_policy0the node agent’s restart policy, with condition, delay, max_attempts and window (3.9); removed from the runtime’s document so it cannot override restart: "no"; restart: still governs when both are set (ce X200932RSTDIA, modules/app/errors_reboot.go:12-31)
deploy.update_config, rollback_config0rolling updates (3.14)
deploy.endpoint_mode: dnsrr0honoured: the resolver’s shape
deploy.endpoint_mode: vip0refused until the per-node L4 VIP exists
deploy.labels0container labels; traefik.* ones become file routes like labels
top-level configs12 appsstaged onto the chosen node; a content change rolls the service through the SHC-1040 per-service digest (ce modules/app/mount_digest.go)
top-level secrets0rendered from the vault into a RAM-only mount at start, never an on-disk bind
networks with driver: overlay0the fabric, with a warning naming the native form
ports, any form10 appsdeclares the container ports only, as expose: does, and the host side is not honoured (ruled 2026-10-01); the render warns, naming --port (3.16)
ports long form mode: host0 todaythe same declaration; publication comes only from --port on the deployment, which renders this form on the replica’s entry (3.16)
healthcheck93 appsstartup, readiness, liveness (3.13)
depends_on with conditions50 appsinside one projection, compose honours it as today. Across nodes the node agent holds the consumer’s start until the provider’s assignment reaches the state the condition names: service_started running, service_healthy ready, service_completed_successfully a one-shot that exited 0

What goes away: the Swarm loader reconciliation (ce modules/app/swarm_compat.go, its four constructs exist only because docker stack deploy uses a narrower loader), the file-bind lift into Swarm configs (ce modules/app/swarm_file_configs.go), and the node pin that made deployments with rendered files immobile (ce modules/app/swarm_mobility.go). The rule that strips deploy.restart_policy so compose cannot prefer it (ce modules/app/restart_effective.go) stays: the boot gate depends on it.

The owner’s pmxcfs-like design (SHC-1058, 2026-09-29): a materialiser in shcd, not FUSE. Who writes it (2026-10-01): every node’s own materialiser, from rqlite; not the leader and not a shared filesystem. On the rig home01 and home02 carry the same revision digest, written at different times into differently numbered local directories.

/var/lib/shc/clusterfs/
REVISION control-plane revision, its digest and write
time; written last, once every section is in;
the node agent's completion marker
node/
current -> rev-000184213 swapped atomically after a full write
rev-000184213/
REVISION every section carries its own copy
assignments.json this node's replicas: deployment, service,
ordinal, generation, spec digest, state
projections/<tenant>/<environment>/<stack>/compose.yaml
traefik/ the Traefik container mounts only this
current -> rev-000184213
rev-000184213/
dynamic/*.yaml every route, backends = ready replicas
certs/ issued once, from the cert broker (0700)
resolver/ the resolver mounts only this
current -> rev-000184213
rev-000184213/zones/*.json names → ready replica fabric addresses,
node names → mesh addresses
auth/ forward-auth mounts only this
current -> rev-000184213
rev-000184213/ realm per host, gating declarations

The first layout was one rev-<n>/ tree holding every section behind one current. Superseded 2026-10-01 by a revision directory per section (Mounts, below).

  • Writes go only through the API into the replicated store. Nothing writes into clusterfs but the materialiser.
  • Render, enrich, materialise (ruled 2026-10-01). Render is Jinja over the app’s files and the deployment’s settings, once, on the leader, at install or update, with a per-replica replica object (ordinal, location, node); the result is a per-replica spec stored in rqlite with a digest. Compose ${VAR} is not used for per-replica values; it resolves once per file. Enrichment is the part of that render that adds what the author did not write: the pin, dns and dns_search, labels, the SHC_* variables, CA trust, telemetry, the volume suffix. Materialising is this node writing the stored specs and the derived Traefik and resolver sections into its revision directory. Nodes never render or enrich, so a daemon upgrade changes no projection until an explicit update; a spec that uses only replica.ordinal moves as an assignment change with no re-render, and one that uses replica.node or replica.location is re-rendered by the leader for the new placement. clusterfs is never an input: nothing that decides or renders reads it.
  • Consistency. The materialiser reads the revision counter, reads what it needs, and reads the counter again; a changed counter discards the build and starts over, so one revision directory never mixes two states. Each section’s directory is written whole and fsynced, then that section’s current is swapped with a rename, and REVISION is written last. A reader of a section sees one revision of it or the next, never half of one. The last three revisions of each section are kept.
  • Freshness. A revision counter row bumps with every write that changes what a node would materialise. The materialiser polls it every second and is kicked on the event bus, the shape today’s ingress sweep already uses (ce subsystems/impl/ingress_materialize.go:37, Kick() on an exposure change). The poll and the kick are only how the materialiser notices a new revision; events themselves are durable, an event row broadcast to every peer with retry and recovery on start (ce modules/events). Each section carries current/REVISION with its revision, digest and write time, so a container that mounts only its section can say how stale it is; the node agent reads the root REVISION completion marker; shc node status prints both.
  • Mounts. A bind mount of a current link would pin the revision it resolved at container start. This design first had every consumer mount the stable parent, /var/lib/shc/clusterfs, read-only, and read through current/ inside the container, the relative symlink resolving at each access. Superseded 2026-10-01: a platform container mounts only its own section, read-only (the resolver resolver/, Traefik traefik/, forward-auth auth/), never the whole tree, so each section gets its own revision directory and its own current and the mount is that section’s stable parent; a bind mount still never pins a stale revision. Traefik’s file provider points at current/dynamic inside its section and gets a SIGHUP after each swap, as the daemon already sends after its writes today (ce addons/traefik/compose.yaml:634-651, modules/app/traefik_reload.go); the resolver reloads on the same signal, and either reload fires only when that section’s digest changed (3.13).
  • Small files only. App payloads keep the staging path.
  • Secrets never land here in plaintext. Projections carry ref+vault:// references; the node agent resolves them at start into the RAM-only mount of 3.23. Today’s rendered compose.yaml interpolates plaintext secrets (ce modules/app/secret_file.go, the list of credential-bearing artifacts); the projection must not. Traefik’s private keys are the one exception, readable only by root and Traefik. Ruled 2026-10-01 (SHC-1159, 3.26): the node agent caches only values resolved from external vault providers (aws, 1password, hashicorp and the rest) for the replicas this node currently runs, in a root-only per-node file and never in rqlite; each value is sealed with the vault’s existing envelope scheme (a fresh AES-256-GCM DEK wrapped by the in-memory Argon2id KEK from unlock), so a node that cannot unlock its vault cannot read its cache, and an entry drops when its replica leaves the node. Local-provider secrets are not cached: every node already holds them, encrypted, in its rqlite copy. This narrows the cache stated earlier the same day, which covered every resolved secret.
  • Partitions. A cut-off node keeps its last revision and keeps serving from it (SHC-1050 §5.8).

The owner asked that every app’s backup run against a stopped or crashed app, because failover and restore-as-move depend on it. Today every database SHC ships quiesces itself in a pre_backup hook run inside the running container (ce modules/backup/hook_bracket.go:1-20), which a crashed app cannot answer.

SHC. A backup of a deployment that is not running skips the hook bracket and captures its volumes as a crash-consistent file archive, and the archive records consistency: crash. Every database app recovers from a crash image on start (postgres and mysql replay their logs). A catalog test runs backup and restore against each database app stopped, and an app that cannot be restored from a crash image declares it, which makes it pinned for failover and makes explain say why.

Raised 2026-10-01 while walking the loop of 2.2; each carries Clay’s verdict and what the design does about it.

gotchaverdictwhat the design does
Lease timing. A paused VM wakes believing it holds an expired lease; an election longer than a lease stops every stateful replica for nothingfix (SHC-1182)the holder stops well before anyone may take over, on a monotonic clock; lease lifetime exceeds election time; a new leader honours existing leases for one lifetime. A frozen VM that resumes after a takeover is excluded only by a STONITH confirm; without one, the residual risk of 3.10 remains, which is why a lease-expiry takeover needs the watchdog (S16). Proven on the rig before the cutover
A daemon upgrade restarts everything. If nodes rendered, a new shcd rendering one label differently would recreate every container on every nodefixed by render oncethe leader renders at install or update and stores each spec with a digest; nodes write stored specs; nothing changes until an explicit update (2.2, 3.24)
The minority side stops its stateful work. A cut-off node cannot renew and stops its databases although it is fineintentionalthe price of never two primaries. shc doctor warns when stateful work runs with fewer than three voters; a witness node is the cheap third (3.12)
A flapping healthcheck churns the estate. Every readiness flip is a revision every node writes and may reload ondampen and coalesceseveral failures to go unready and several successes to return; at most one revision a second; Traefik and the resolver reload only on a changed section digest; a repeat flapper is held out visibly (3.13)
Secrets at restart during a partition. Projections carry vault references the agent resolves at start; a cut-off node cannot resolve an external-provider ref to restart a containerruled on (2026-10-01, SHC-1159): external providers onlya root-only per-node file of the values resolved from external vault providers for current replicas, never in rqlite, each sealed with the vault’s envelope scheme (fresh AES-256-GCM DEK wrapped by the Argon2id KEK from unlock), dropped when a replica leaves. Local-provider secrets need no cache: every node holds them encrypted in its rqlite copy. The trade is external values resting on disk, sealed, and a rotation unseen until the provider is reachable again (3.24)
Waiting for acknowledgement forever. A node down mid-update would hang shc updatethe rollout is a stored, resumable objectthe command returns after a timeout with which replicas moved and which node has not reported; the rollout resumes when the node returns or rolls back on request (3.14)

shc fence explain (SHC-1050 §3.17) is the scheduler’s one explain. This design adds the replica table, the pending and refused reasons, the capacity step, spread, the failover plan and the history.

A placed deployment:

$ shc fence explain -s nextcloud -t acme -e prod
deployment acme/nextcloud/prod docker priority normal movability reattach
status placed 1/1 ready
replicas
SERVICE REPLICA NODE GEN STATE READY SINCE
app 0 gor-02 4 ready yes 2026-10-02 14:03
eligible gor-01 gor-02
resource volume nc-data stay in storage:ceph-ogden gor-01 gor-02 gor-03
integration postgres -> acme/postgres/prod
share location (socket default) location:gorillaservers/ogden
tenant acme stay in site:us-west (2026-10-02) gor-01 gor-02 hil-01
deployment acme/nextcloud/prod no rules
condition ready gor-01 gor-02
capacity needs 2.0 GiB, 500m fits gor-01 gor-02
move group acme/nextcloud/prod, acme/postgres/prod (location)
scored gor-02 812/1000 memory 930 cpu 700 count 640 tenant 900 app 1000
gor-01 640/1000 memory 610 cpu 690 count 500 tenant 900 app 1000
failover 5m0s after its node's lease expires: dead-source move inside storage:ceph-ogden (rebind)
history
2026-10-02 14:02 rescheduled app/0 gor-03 -> gor-02 5m0s after gor-03's node lease expired (13:57)

A pending one names every node and what removed it, the Kubernetes “0/5 nodes are available” line with the fence that did it:

status pending since 14:05 (3m) app/0
0/5 nodes fit
2 outside site:ogden tenant acme hil-01 hil-02
1 cordoned gor-03
2 insufficient memory: needs 4.0 GiB gor-01 (1.2 GiB free) gor-02 (0.8 GiB free)
nothing is retried until capacity or rules change

A refusal prints the SHC-1050 lines up to the first fence that emptied the set, then the commands that would satisfy it.

-o json is the document the HUD :placement view and the API return:

{
"deployment": {"tenant": "acme", "stack": "nextcloud", "environment": "prod",
"runtime": "docker", "priority": "normal", "movability": "reattach"},
"status": {"state": "placed", "since": "2026-10-02T14:03:10Z",
"ready": 1, "desired": 1, "reason": null},
"replicas": [{"service": "app", "ordinal": 0, "node": "gor-02", "generation": 4,
"state": "ready", "ready": true, "since": "2026-10-02T14:03:10Z"}],
"eligible": {
"nodes": ["gor-01", "gor-02"],
"steps": [{"source": "resource", "object": "volume nc-data",
"rule": "fence=storage:ceph-ogden", "scope": null,
"fence": "storage:ceph-ogden", "kept": ["gor-01", "gor-02", "gor-03"],
"removed": ["hil-01"], "loosened": null}],
"emptied_by": null
},
"move_group": {"members": ["acme/nextcloud/prod", "acme/postgres/prod"],
"derived": [{"kind": "location", "fence": "location:gorillaservers/ogden",
"from": "acme/nextcloud/prod -> acme/postgres/prod"}]},
"spread": [{"service": "app", "kind": "node", "skew": 1, "hard": false,
"domains": {"node:gor-01": 0, "node:gor-02": 1}}],
"score": {"weights": "spread",
"candidates": [{"node": "gor-02", "score": 812, "runtime_match": true,
"terms": [{"name": "memory_headroom", "normalised": 930,
"weight": 40, "abstained": false, "reason": ""}]}]},
"pending": [],
"failover": {"movability": "reattach", "after_lease_expiry": "5m0s", "path": "presumed-dead rebind"},
"history": [{"at": "2026-10-02T14:02:41Z", "verb": "rescheduled", "service": "app",
"ordinal": 0, "from": "gor-03", "to": "gor-02", "reason": "node_down",
"detail": "5m0s after gor-03's node lease expired (13:57)"}]
}

score.candidates[].terms is today’s placement.Term, unchanged (modules/placement/placement.go:188-200).

One vocabulary for pending[].reason, status.reason and history[].reason, so a script can match on it:

outside_fence, avoided_fence, reserved_fence, volume_reach, colocation, apart, spread, max_per_node, separate, condition (with the node’s condition), insufficient_cpu, insufficient_memory, port_in_use, architecture, runtime, quota, node_down, node_returned, operator_move, drain, rebalance, update.

capacity_unknown is not a reason to remove a node; it is a flag on a kept one.


Today there is no shc fence command; SHC-1050 §3.18 listed its verbs. This is their exact surface, built by SHC-1079 with the HUD views of 5.3. Every verb takes the global -o tty|text|json|yaml and --fields. Fence ids are <kind>:<name>; <node> is a node name or id, as every shc node verb takes it.

shc fence ls [--kind <kind>] [--origin derived|resource|system|custom] [--node <node>] [--all]

Lists fences. Without --all it hides the node: singletons and resource fences, which number one per node and one per volume. --node lists the fences a node is in, which includes them.

FENCE KIND ORIGIN NODES READY RESERVED RULES DEPLOYMENTS
control-plane control-plane system 3 3 no 0 2
location:hetzner/hel1 location derived 3 3 no 1 14
location:unknown location derived 1 1 no 0 0
provider:hetzner provider derived 3 3 no 0 0
site:ogden site custom 2 2 no 3 6

-v adds MEMBERS, BACKUP-REPO, DESCRIPTION, CREATED. Warnings print under the table: an unlocated node on a multi-site estate (SHC-1050 §3.1), a control-plane fence with fewer than three ready nodes or no spare (SHC-1050 §3.9). JSON rows carry id, kind, name, origin, description, reserved, backup_repo, members[] {node, source, condition}, rule_count, deployment_count, created_at, warnings[].

shc fence show <fence>

The fence’s attributes, then three tables: MEMBERS (node, condition, availability, source: derived, terraform, cli or join), RULES naming it (scope, object, form, set by, when, reason), and DEPLOYMENTS it shapes (deployment, the rule that brings the fence in, where it runs).

shc fence create <kind>:<name> [--node <node>]... [--description <text>] [--reserved] [--dry-run]

Creates a custom fence. kind matches [a-z][a-z0-9-]{0,31}, name [a-z0-9][a-z0-9.-]{0,62}. Refused by name: a built-in kind (node, provider, location, control-plane), a resource kind (volume, storage, lb, ip, endpoint), an id that exists, an unknown node. --reserved refuses when a deployment already on a listed node does not name the fence, naming each one. Prints the created fence as show does.

shc fence rm <fence> [-y] [--dry-run]

Deletes a custom fence. Refuses a non-custom origin, and refuses while any rule names the fence, listing each rule with the shc rule rm that removes it. Confirms unless -y.

shc fence add-node <fence> <node>... [--dry-run] shc fence remove-node <fence> <node>... [--dry-run]

Membership of a custom fence, or of control-plane. Both are idempotent. Either refuses, naming the deployment and the rule, when a deployment running on the node would leave its eligible set: removing the node from a fence the deployment is held in, or adding it to a reserved fence the deployment does not name. The fix is to move or drain first; there is no --force, because a fence edit that strands a running deployment is the action at a distance fences exist to prevent. remove-node on control-plane refuses a node that holds a voter slot and warns below three members.

shc fence set <fence> [--description <text>] [--reserved|--no-reserved] [--backup-repo <url>|--no-backup-repo]

Attributes. --reserved applies to custom, location, provider and control-plane fences and refuses as create --reserved does. --backup-repo applies to location fences only (SHC-1050 §3.16).

shc fence explain -s <stack> [-t <tenant>] [-e <environment>] [--service <service>] [--history <n>] [--what-if-node <node>] [--what-if-rule <rule>]...

Section 4. The deployment is addressed the way every deployment verb addresses it, by the scope flags -s, -e and -t (or shc ctx), never a tenant/environment/stack path (2026-10-01); the header prints the tenant/stack/environment triple. --what-if-node explains one node’s verdict as if it were the only candidate; --what-if-rule evaluates the set with extra rules added and changes nothing. --history bounds the history rows (default 10).

Authority. Reads are open to any authenticated operator; explain follows the deployment’s tenant authorisation, so a tenant admin can explain their own deployments and sees fence ids but not other tenants’ rules. Fence writes need platform:manage (SHC-1050 decision 27).

Where it lives. The fence store, its reconciler, the CLI and the explain renderer are in ce, because a single-node CE estate has built-in fences and explain answers there too. The multi-node engine loop, rescheduling and failover are EE in shc, where the scorer, drain and volume failover already are.

commandchange
shc install, update, clone--fence, --avoid, --colocate and --admit take <kind>:<name>, and --location <name> is --fence location:<name>; --rule is gone (SHC-1237, 2026-10-02). spread and apart are engine-derived (3.4, 2026-10-01). There is no move or bind verb. --scale on install and update takes the three shapes of 3.6
shc rule add|ls|rmSHC-1050 §3.18, with the new forms
shc scalegoes (ruled 2026-10-01); --scale on shc install and shc update places the replicas across nodes and prints the explain replica table
shc node lsCONDITION column (2.3)
shc node statuson the node: boot gate state, clusterfs revision and age
shc node maintenance <node> --window <d> [--drain], --end3.18
shc rebalance --plan|--apply3.19
shc cluster statusengine state: pending replicas, nodes lost or down, grace clocks running

Today the HUD has no fence type: shc hud is x9s with the provider in pkg/x9sprov (cli/hud.go), and every type is a resource returned by catalogResources() (pkg/x9sprov/catalog.go:276-318). Three types are added, each in its own type_<name>.go with its test, so the shared files change by one registration line each (the house rule for parallel HUD lanes). Each follows the rules of docs/design/hud-types.md §1: every verb is a CLI verb first, every key comes from a declared action or relation, and the confirm dialog prints the exact shc command.

:fences (alias fence). Platform-wide (Scoped: false), like :nodes. Lists what shc fence ls --all lists.

  • Narrow: FENCE (name role) · KIND · ORIGIN · NODES · READY · RESERVED · RULES · DEPLOYMENTS.
  • Wide: MEMBERS · BACKUP-REPO · DESCRIPTION · AGE (age role).
  • Relations (v): n member nodes (Drill: Enter), r rules naming it, d deployments it shapes.
  • Keys: Enter → :nodes filtered to members. ctrl-n create (shc fence create), a form with kind, name, description, reserved, and one toggle per node, the shape :storage’s node form already uses (pkg/x9sprov/type_storage_nodes.go:100-115). e edit: the same form prefilled; the confirm prints one shc fence set plus one add-node or remove-node per toggled node. ctrl-d delete (shc fence rm <fence> -y), Danger, offered only where ORIGIN is custom (the action’s When).
  • d is the shc fence show -o json document.

:rules (alias rule). Scoped to the active tenant; estate-rung rules show in every tenant with SCOPE global, read-only for a tenant admin.

  • Narrow: SCOPE · TARGET (the scope key) · FORM · RULE (the rule as its flag without the dashes, fence storage:fast) · LOOSENS · SET-BY · AGE.
  • Wide: REASON · FENCE · ID.
  • Relations: f the fence it names, d the deployments it binds.
  • Keys: ctrl-n add (shc rule add --fence <kind>:<name> with the scope flags; a deployment-rung rule prints shc update -s <stack> --fence <kind>:<name>); ctrl-d remove (shc rule rm <id>). ctrl-j on FENCE → :fences.

:placement (hidden, drill-only, read-only). The explain view.

  • Reached by v p on a :deployments row (p is free in that picker: pkg/x9sprov/catalog.go:401-414 uses s c b h j v k g i n e a w r).
  • Rows are explain’s lines: STEP (replica, rule, condition, capacity, spread, score, failover, history) · SOURCE · OBJECT · RULE · FENCE · NODE · LEFT (nodes remaining) · VERDICT (kept, removed, chosen, pending, refused).
  • d is the shc fence explain -o json document. ctrl-j follows FENCE to :fences and NODE to :nodes.

Existing types gain: :nodes a v f relation (the fences the node is in; f is free, the picker uses d v t w, catalog.go:1183-1186) and a CONDITION column; :deployments a wide PLACEMENT column (placed, pending, refused, down, moving) and the v p relation; :tenants a v p relation to :rules at that tenant (the picker uses d u q k g c r, catalog.go:1573-1579).

No global key changes: every key above is already in the verb table with the same meaning (docs/design/hud-types.md §2), so the collision test holds. hud-types.md gains §3.43 to §3.45; testdata/hud_commands.txt is regenerated after checking each new printed line is a real command (TestEveryHUDExplainParses), and each new daemon call names its tenant the way the route reads it (SHC-973).

Routes in the existing /api/method/shc.<module>.<verb> shape: shc.fence.list, get, create, delete, add-node, remove-node, set, explain; shc.rule.list, add, delete. Fence routes are platform-wide and read no tenant; shc.fence.explain and the rule routes read the tenant the deployment or scope belongs to. The openapi document regenerates.

SHC-1050 §3.18 (shc_fence, shc_rule, rules on shc_deployment, location, provider, fences on nodes), plus reserved and backup_repo on shc_fence.


All in the new line’s single initial migration (SHC-1050 §6.3), beside SHC-1050 §5.6:

  • assignment(deployment_id, service, ordinal, incarnation, node_id, generation, state, spec_digest, target_node_id, previous_node_id, reason, updated_at), unique on (deployment_id, service, ordinal, incarnation). Two incarnations of one ordinal exist only during a start-first update or a surge drain (3.14, 3.18).
  • recorded_node_id keeps its SHC-1050 §3.7 meaning, where the bytes are: the node holding the deployment directory and its persistent volumes. For the default shape, every service on one node, that is also where every replica runs. For a split deployment it is the node of its stateful services, and each replica’s node is on its assignment. The readers that assume every container sits on recorded_node_id (logs, exec, deployment inspect, the container recorder’s per-deployment status) read assignments instead; backup capture already goes to the node holding each volume.
  • deployment_spec(deployment_id, digest, spec, created_at): the full effective specification (app digest, vars with secrets as vault references, --host and --port mappings, integrations, posture), the last two kept per deployment. Rollback (3.14) and the lossless dead-source move (3.9) read it. It is also the home of the per-replica rendered specs and their digests, which the leader writes at render and the materialiser copies (render once, 2026-10-01).
  • rollout(deployment_id, from_digest, to_digest, batch, state, updated_at).
  • scheduler(epoch, holder_node_id, since): one row, bumped by each new engine (2.2).
  • node_lease(node_id, boot_id, leased_until, renewed_at).
  • node: cluster_state gains rebooting; new expected_back_by, reserved_cpu_millicores, reserved_memory_bytes. lost and down are derived from node_lease and the grace clocks, never stored.
  • deployment: priority.
  • placement_event(deployment_id, service, ordinal, at, verb, from_node, to_node, reason, detail), bounded to the last 100 per deployment.
  • revision(id, value, updated_at): one row, bumped by every write a node would materialise.
  • fence.reserved (bool) and the avoid, apart, spread and admit forms in placement_rule.form.
  • Compose extensions: x-shc-scale.stateful (was identity; renamed 2026-10-01, no alias), x-shc-health.restart_after, x-shc-disruption.max_unavailable.

An estate may drop Swarm when all of these hold on it, each proven on a rig first:

  1. Nebula on every node as the addressing plane (SHC-1050 phase 1). The two-tunnel mesh (SHC-1201) lands after the cutover (ruled 2026-10-01).
  2. Fabric slices 2 and 3 and Traefik on the fabric (SHC-1050 7.1 to 7.3).
  3. The engine places every install through the dispatcher inside fences (SHC-1050 phase 3; this design’s B phase).
  4. Node leases, conditions, the boot gate and node-agent restarts (A phase).
  5. Rescheduling by movability and the node-return path (C phase).
  6. Per-node projections and the resolver answer every cross-node name the estate’s deployments use (B3, SHC-1050 8.4).
  7. The per-node platform set on every node and an edge per location (SHC-1050 8.1, 8.3).
  8. The render honours or refuses every Swarm key, and a dry-run render of every deployment installed on the estate refuses nothing (E phase).
  9. Every deployment’s backup restores from a stopped app on a scratch node (3.25).
  10. Drain works with no docker service inspect read (SHC-1050 3.5).
  11. Every deployment’s full specification is in the replicated store, so no automatic move comes back on app defaults (3.9).
  12. The single-writer watchdog runs on every node, and every node whose provider has a STONITH agent has it configured (3.10). It stops single writers on docker, podman, incus and apple, and on macOS a launchd job (org.selfhostedcloud.shc-watchdog) runs it.
  13. Every capability in the feature parity inventory works on the rig (new-line simplification §5). Deleting Swarm removes a mechanism, never a capability.

With the next branch these thirteen conditions are the cutover gate for next becoming main (SHC-1165), not a gate on deleting Swarm code, which the new line does first.

bitspur moves by SHC-1061’s path B (fresh cluster, app by app; SHC-1050 §6.3):

  1. Dry run of the one-off script: inventory (shc ls --all-tenants -o json), every deployment rendered under the new line with refusals listed, move groups in order, every DNS change. Stop on any refusal.
  2. Verified backups of every deployment and of the control plane; stop on any failed verify.
  3. Stand up the new-line cluster as fresh VMs on both Gorilla boxes (pve02, pve03) under Proxmox, docker runtime, with terraform: Nebula, the control-plane fence, site fences. Each app’s data arrives by restoring its restic backup, never by copying volumes off compose01, and compose01 is never upgraded in place (ruled 2026-10-01; this replaces the earlier hel1 target). After the cutover pve02 and pve03 become bare Incus hosts.
  4. Restore the master tenant first (postgres, keycloak, openobserve), then tenants in move-group order, app by app with both estates live, each one: a fresh backup right before the move, restore, health gate, DNS re-point, old copy left running until the new one passes.
  5. Point the edge (pfSense HAProxy) at the new Traefiks.
  6. Soak: compose01 stays untouched as the rollback for two weeks after the last app moves (ruled 2026-10-01), switching back being a DNS change plus starting the old deployment; then retire it, keep its disks until the owner signs off, and delete the script.

promanager is not touched. It stays on its current line until the owner decides otherwise.


The order below is superseded by the delete-first order in the new-line simplification (SHC-1092), which merges B1 into 3.2, B2 into 2.3, inverts B5 and SHC-1050 8.1, and moves G to the front. The steps’ content stands.

One numbered step is one ticket and one MR. The letters are this design’s phases; the numbers in brackets are SHC-1050’s.

A. Node truth. Depends on SHC-1050 phase 0.

  • A1 node_lease and its renew loop on every node; node conditions; CONDITION in shc node ls.
  • A2 The clusterfs materialiser: revision row, consistent builds, atomic revision directories per section, each platform container mounting only its own section (2026-10-01); the ingress sweep moves onto it.
  • A3 assignment rows with generations and incarnations; the scheduler epoch and the epoch-guarded bind; the dispatcher executes assignments for single-replica installs; the readers of recorded_node_id that mean “where the containers are” move to assignments.
  • A4 Node-agent restarts and the boot gate: restart: "no" and no deploy.restart_policy in the runtime’s document, the restart supervisor on Docker events, podman-restart kept off, the stop journal and converge restore reworked onto it.
  • A5 deployment_spec: the full effective specification replicated, the last two kept.
  • A6 The single-writer watchdog, a systemd unit outside shcd. It stops single writers on docker, podman, incus and apple, and on macOS a launchd job (org.selfhostedcloud.shc-watchdog) runs it.

B. The engine. Depends on A3 and SHC-1050 steps 3.1 to 3.5. SHC-1050 step 3.6 keeps turning deploy.placement.constraints into rules; its replicas and mode: global half moves to B4 and B5 here, so neither design waits on the other.

  • B1 Fit as a filter: the capacity gates called, the system reserve, pending instead of “no opinion”.
  • B2 Explain v2: replica table, reasons, placement_event, JSON (section 4).
  • B3 Per-node projections with dns_search (needs SHC-1050 8.4).
  • B4 Replicas across nodes, one compose entry per replica, the three --scale shapes, spread, stateful: true, per-ordinal volumes and pins.
  • B5 Global mode; the per-node platform set as a global deployment (needs SHC-1050 8.1).
  • B6 Reserved fences and the avoid, apart and admit forms; the system-priority admission.
  • B7 Priority.

C. Failure. Depends on A and B1.

  • C1 Movability derivation; grace clocks from lease expiry; stateless replicas re-assigned when their clock runs out.
  • C2 Reattachable failover through the dead-source move, with roll-forward after a leader change and a replica-scoped variant, after the single-writer stop window measured as elapsed time (decision 25, 3.10); single-writer self-stop without a volume. Depends on A5 and A6.
  • C3 Node return: stale-copy removal, pinned restart, no flip-flop.
  • C4 rebooting: the announced-shutdown hook, shc node maintenance, the STONITH-confirmed shortcut.
  • C5 Jobs off a lost node.

D. Rollout. Depends on B.

  • D1 Readiness feeding endpoints; the liveness restart.
  • D2 Rolling updates: the rollout row, incarnations for start-first, rollback to the retained specification. Depends on A5.
  • D3 Drain on the engine: per replica, surge, max_unavailable, global replicas last.
  • D4 shc rebalance --plan|--apply.

E. Swarm keys. Depends on B4 and B5.

  • E1 The render translates or refuses every key in 3.23, with the Q1 warning naming the native form; ports: read as a declaration and --port as the one publication (3.16; the SHC-1081 sweep to mode: host is withdrawn, 2026-10-01); a catalog sweep test renders all 106 apps with no silent drop.
  • E2 Configs staged per node; secrets from the vault into RAM-only mounts.
  • E3 The stopped-app backup path and its catalog test (3.25).

F. Surfaces.

  • F1 shc fence and the HUD fence views (SHC-1079): the CLI verbs of 5.1 exactly, their API routes, and the :fences, :rules and :placement HUD types of 5.3. It builds on SHC-1050 2.1 and 2.2 (the fence tables, the reconciler, terraform) and takes over the CLI halves of those steps and of 2.3 (explain rendering), so the verbs land once. :placement renders the explain JSON of B2 once B2 lands, and SHC-1050’s shorter explain before that.
  • F2 shc cluster status engine summary.

G. Swarm out. SHC-1050 7.5 removes the Swarm runtime once section 7 holds; SHC-1061 moves bitspur.


The seventeen questions this design left open were decided by agent on Clay’s standing delegation, 2026-09-30, reversible, the way SHC-1050 decisions 24 to 28 were. Each took the recommended option. The rejected options are kept so a reversal knows what it is choosing between. They are numbered S1 to S17 so they cannot be mistaken for SHC-1050’s decisions 1 to 28, which they build on and do not change.

  • S1. shcd restarts crashed containers; the runtime’s restart policy is off. The render stamps restart: "no", removes deploy.restart_policy from the runtime’s document, and the node agent applies the authored policy (3.9). Cost: while shcd is down a crash waits for it; systemd restarts shcd in seconds. Rejected: keeping unless-stopped and rewriting container config before dockerd starts, which depends on Docker’s private on-disk format and has no Podman equivalent; keeping the runtime’s policy as is, which starts containers at boot before ownership is known and breaks decision 23.

  • S2. Grace: 60 s for stateless work, 5 min for reattachable stateful work, from lease expiry, as scoped config a deployment may shorten, never below the single-writer stop window (3.8). Rejected: one grace for everything, which either strands stateless apps for minutes or moves stateful data on every reboot; Swarm’s immediate rescheduling, which moves a database for a two-minute reboot; Kubernetes’ flat 300 s.

  • S3. A single writer is a service with an RWO volume or x-shc-scale.max: 1, the ceiling 97 catalog services already declare as a data-safety directive (3.10). Rejected: RWO volumes only, which misses single writers on host binds; a new explicit key, which is a catalog sweep for a fact the catalog already states.

  • S4. A far node that reboots during a WAN cut waits for the heal. Its far-location volumes may keep running through a cut (decision 24) but are not started after a reboot without a confirmed ownership. An operator break-glass verb (shc node start-local) comes only if a real outage asks for it. Rejected: starting far-location apps from the last clusterfs at boot, which breaks decision 23; a lease authority per site, which decision 24 already rejected.

  • S5. No preemption. Priority orders rescheduling, drain and the pending queue (3.7); its one other effect is that system priority is admitted to every reserved fence (3.5), which is how the per-node platform set reaches every node. Rejected: Kubernetes-style priority preemption, which would evict a stateful app with a full cutover to fit another.

  • S6. No node-pressure eviction. A pressured node takes no new work and keeps what it runs (3.20). Rejected: kubelet-style eviction by priority, which is preemption by another name.

  • S7. deploy.endpoint_mode: vip is refused at render until the per-node L4 VIP exists; dnsrr is honoured (3.15, 3.23). No catalog app sets either. Rejected: reading vip as dnsrr, which changes its meaning silently against decision 21; building the L4 VIP now, which nothing in the catalog needs.

  • S8. Short-form and mode: ingress published ports are refused at render, and the five catalog apps that publish ports are swept to long-form mode: host (3.16; the sweep is SHC-1081). Under Swarm those ports ride the routing mesh; under compose today they already publish on one node, so the sweep changes nothing that runs. Rejected: reading short form as host publication, which drops mesh reachability without saying so and breaks decision 21; rebuilding the routing mesh, a second forwarding plane beside Traefik. Superseded 2026-10-01: ports: in compose only declares, an app never self-publishes, and --port on the deployment is the one publication (3.16); the sweep shape is withdrawn.

  • S9. The fence store, its reconciler, the shc fence CLI and explain live in ce; the multi-node engine loop, rescheduling and failover live in shc, as drain and volume failover do today (5.1). Rejected: all in shc, which leaves single-node CE estates with no built-in fences and no explain; all in ce, which moves the multi-node HA engine across the edition split.

  • S10. Rebalancing is explicit only in v1 (shc rebalance --plan|--apply, 3.19). Rejected: a rebalance rule attached to a scope now, which no estate has asked for; automatic rebalancing, which the rebalancer v1 ruling excludes. Amended for group members (Clay, 2026-10-01, mem shc-scale-and-cli-scope-model): members of a group that share a node move apart on their own when a node frees or joins (3.19, the separation pass). Only that separation runs on its own; load moves stay an operator’s request.

  • S11. Cross-node names resolve through dns_search to the deployment’s zone, and a service split across nodes carries a per-replica name inside each projection (3.15). Rejected: baking remote names as aliases or extra_hosts, the render-time binding decision 22 removed; a per-node L4 VIP now (S7). Widened 2026-10-01: every entry is <service>-r<ordinal>, always, not only when a service spans nodes (3.6, 3.15).

  • S12. The LeaderHub polls the leader every 1 s once the engine rides it (3.12). Rejected: keeping 5 s, which adds up to 5 s to every failover decision after an election.

  • S13. Default spread is soft spread=node for stateless replicas and spread=node,max=1 for stateful: true (3.6; the key was identity: stable until 2026-10-01). Rejected: one per node for every replicated service, which refuses --scale web=3 on a two-node estate; a soft spread for members, which lets two ordinals of a clustered database share a node.

  • S14. Liveness restarts a container unhealthy for 5 min (x-shc-health.restart_after, 0 disables; 3.13). Rejected: Swarm’s immediate replacement, which crash-loops a database on a slow recovery; never restarting, which is how a container stayed unhealthy for 13.8 hours on corp.

  • S15. :rules is its own HUD type (5.3). Rejected: folding rules into :fences: rules attach to scopes and deployments, not fences, and a tenant admin needs them tenant-scoped while fences are platform-wide.

  • S16. A single writer is taken over on a STONITH confirm wherever the provider has a STONITH agent (Hetzner today). Elsewhere a lease-expiry takeover is allowed only on a node running the watchdog, after the elapsed stop window, and explain states the residual risk of a frozen VM resuming (3.10). Rejected: no automatic single-writer takeover without STONITH, which leaves Proxmox and bare-metal nodes with manual failover only; lease-expiry takeover without the watchdog, where a hung shcd leaves its writers running.

  • S17. Decision 23 stays strict. A rebooted node starts nothing, the per-node platform set included, until the control plane confirms what it owns and its leases are re-acquired (3.9). A node that reboots inside a partition therefore serves nothing until the heal. Rejected: starting the platform set from the last clusterfs at boot, which serves routes for a node that may no longer be a member or may have lost the platform assignment.


Stale today:

  • ce modules/app/resource_reservations.go:13-19 says no catalog deploy.resources block writes a reservation; four apps do now (onlyoffice, openarchiver, pulp, stirling-pdf).
  • modules/quota/enforcement.go:37 says reservation_ratio defaults to 0.25; ce modules/app/resource_reservations.go:125 defaults it to 1.0.
  • modules/quota/enforcement.go presents three gates as the admission refusal; none has a production caller.
  • subsystems/volume_failover.go:80-89 describes a 45 s dwell; the dwell counts three qualifying ticks (:465-500), so it can authorize about 30 s after the first, and its compile-time assertion (:108) omits the stop time.

Superseded by this design when it lands:

  • modules/placement/placement.go:160 “SHC has no HA”, and the scorer’s “SWARM DOES NOT GET A SCORE” (modules/placement/advisor.go:84, ce modules/stack/placement_seam.go).
  • ce modules/app/restart_default.go (unless-stopped for every service that declares no restart:).
  • ce modules/clone/move_presumed_dead.go:43-51, the manual docker compose down on a returned node.
  • ce modules/health/reconcile/errors.go:105-128, X200924RSTRMT’s “nothing will repair it”: the owning node’s agent does.
  • docs/decisions/2026-08-11-rebalancer-v1-foundations.md §0 “no failover”: C1 and C2 add it, still with no autonomous rebalance.

Clay’s rulings, folded in by SHC-1198. Each is a rule; the section that carries it is in brackets, and the full text of each lives in the memory files named on SHC-1198 and on the one-page summary.

  • Deployment, not stack; app, not package (2026-09-30, 2026-10-01). The catalog thing is an app. One app for one tenant in one environment is a deployment, the running unit and its own network. “Stack” names only the axis. The ladder is app, deployment, service, replica, compose entry <service>-r<ordinal>, container. Existing code is not mass-renamed [vocabulary].
  • Runtimes are docker, podman, incus and apple, all four the architecture now; a runtime is the engine that takes a compose file; GPU, microVM and architecture are node capabilities; no Docker API shim, no Proxmox executor, and Proxmox HA never manages an SHC guest; pve02 and pve03 become bare Incus hosts only after the cutover (2026-09-30, 2026-10-01) [2.2, 3.13, 7].
  • Three doors into shcd (2026-09-30): the unix socket for the local CLI, the mesh listener for nodes, the public listener for the Mac CLI, Terraform and the Console. No container reaches the node plane or the internal API [new-line simplification §3.12].
  • The policy unit is the deployment (2026-09-30): same tenant open to services that declare ports, on those ports; cross-tenant blocked and refused at declaration; the system tier is master’s platform environment; connections are wiring, not permission. The address plan (2026-10-01): fabric 10.128.0.0/10, a bridge per tenant per node, a /20 default grant, 10.64.0.0/10 node-local, 10.192.0.0/10 control plane [new-line simplification §3.13].
  • ports: and expose: only declare; an app never self-publishes; --host and --port on the deployment are the only publication (2026-10-01) [3.16, 3.23, S8].
  • Identity is the pin; modules read files (2026-09-30): a platform container reads its own clusterfs section, with per-section revision directories (2026-10-01), and never dials the control plane; forward-auth’s gate call carries only the user’s JWT on the local public listener until it decides from files; no module credential [3.24, new-line simplification §3.12].
  • No service mesh (2026-09-30, closed). No sidecars, no L7 mesh, no per-request policy; Traefik middlewares are the only retries and circuit breaking. Never re-propose.
  • Two kernel WireGuard tunnels, never double wrapped (2026-09-30): mesh0 for the node plane, shc0 for containers, coordinated by shcd from a peer table in rqlite; Nebula is today’s carrier and goes at stage 2, after the cutover (2026-10-01, SHC-1201) [rulings list, 7].
  • The cutover lands bitspur on fresh VMs on both Gorilla boxes under Proxmox, docker runtime, app by app with both estates live, each app restored from a fresh restic backup, compose01 untouched as the rollback for two weeks and then retired; no replicated volumes in the first cut (2026-10-01) [3.6, 7].
  • One --scale flag, three shapes, no shc scale verb, no home, no failover (2026-10-01): web=3, pg=eu:2,us:1, proxy=per:location on install and update; deployments addressed by flags, never a path; spread, regions and apart are engine-internal [1, 3.4, 3.6, 5.2].
  • One compose entry per replica, always, named <service>-r<ordinal> (2026-10-01); compose never scales; the bare name is a local alias only when every replica is local; x-shc-scale.stateful: true marks distinct members and replaces identity: stable with no alias [3.6, 3.15, 6, S11, S13].
  • Render once on the leader; nodes never render (2026-10-01): Jinja with a replica object at install or update, stored per replica with a digest; the materialiser copies; a daemon upgrade changes nothing until an explicit update [2.2, 3.24, 6].
  • Swarm keys with a native form are translated with a warning that names it (2026-09-30, Q1) [3.23].
  • Four pre-run checks on a node (2026-10-01): the revision is whole, the entry hash matches the stored digest, the replica is still this node’s at the current generation and epoch with its lease, cross-node dependencies are ready [2.2, 3.9].
  • Resolver (2026-09-30): a small Go binary on miekg/dns per node, reads resolver/zones, never dials shcd, answers ready replicas nearest first with TTL 5 s; containers get dns plus dns_search [<deployment zone>, <environment zone>]; per-runtime wiring belongs to the executor [3.15].
  • Health (2026-10-01): defined once in compose, run by the runtime (the node agent on apple), read by the node agent, recorded per replica; no healthcheck means running is ready; dampen, coalesce to one revision a second, reload only on a changed section digest [3.13].
  • Leases (2026-10-01, SHC-1182): the holder stops well before anyone may take over, monotonic clock, lifetime longer than an election, a new leader honours existing leases for one lifetime [3.10].
  • The per-node sealed secret cache is for external providers only (2026-10-01, SHC-1159): the node agent caches only values resolved from external vault providers (aws, 1password, hashicorp and the rest) for the replicas this node currently runs, in a root-only per-node file and never in rqlite; each value is sealed with the vault’s existing envelope scheme (a fresh AES-256-GCM DEK wrapped by the in-memory Argon2id KEK from unlock), so a node that cannot unlock its vault cannot read its cache, and an entry drops when its replica leaves the node. Local-provider secrets are not cached: every node already holds them, encrypted, in its rqlite copy [3.24, 3.26].
  • The six gotchas and their verdicts (2026-10-01) [3.26].
  • Open questions live in one table with a recommended answer beside each, and nothing ruled is in it [new-line simplification §9].