Rebalancer v1 foundations: four rulings, and what was measured wrong on the way
Superseded by New-line runtime. The text below is the record as of its date.
Status: RULED and BUILT. Everything on this page is landed code with a
guard, or a ruling recorded where the question was asked. Nothing here is a
proposal. The design pages it builds from are
docs/proposals/2026-08-11-{placement-ownership-fork,capacity-substrate,placement-policy,rebalance-trigger-and-safety}.md;
where this page and one of those disagree, this page is what was
measured and the proposal is the earlier draft.
Baselines. shc main 2a5800aaf, ce main ba0252a. Every figure
below was read out of those trees or produced by a test run in this
session.
0. What v1 is, and the line it does not cross
Section titled “0. What v1 is, and the line it does not cross”v1 is placement plus the substrate underneath it. It chooses where a first install lands, and it builds the facts a future rebalancer would need to read. It contains:
- no autonomous trigger,
- no rebalance timer,
- nothing that moves a live deployment,
- no failover.
The reason for the line is not caution in general, it is one specific asymmetry. A bad first placement costs a deployment sitting on a node the operator would not have picked, and it is undone by the next install. A bad rebalance costs a full stateful cutover (stop, archive, ship, restore, health gate, data gate), and the revert window closes at the commit point, after which failures roll forward. The two are not comparable risks and they do not belong in one release.
1. R1: RULED: runtime: swarm is LEGACY / SECOND-CLASS
Section titled “1. R1: RULED: runtime: swarm is LEGACY / SECOND-CLASS”Recorded where it was asked, in
docs/architecture/cluster/swarm-contention-map.md §8. The evidence,
re-measured in this session rather than inherited:
| claim | measured |
|---|---|
| catalog compose files | 99 |
carrying depends_on conditions Swarm cannot express (service_healthy / service_completed_successfully) | 37 |
| ce builtins carrying the same | 2, and one of them is builtin/traefik, the system ingress |
| Swarm-runtime stacks on either estate | 0 |
| Swarm-runtime installs verified end to end, ever | 0 |
grep -rl 'service_healthy\|service_completed_successfully' apps --include=compose.yaml | wc -l → 37 of 99.
A runtime that cannot host 37% of the catalogue or the ingress, has no production instance, and has never been proven end to end is not a supported production configuration whatever the code says. Swarm is not deleted and not broken; it is where new capability does not go.
★ The ruling is acted on in code, in both directions. The install-time placement scorer declines to have an opinion when the runtime is swarm; Swarm has a scheduler, SHC now feeds it the reservations it needs to use one, and two control planes choosing over one fleet is not a better scheduler. Meanwhile the capacity substrate is deliberately runtime-neutral: every figure is computed from SHC’s own replicated tables and its own mesh, so none of it requires Swarm to exist.
R2 of the same map (“should SHC write deploy.resources.reservations?
no code should be written before this is ruled”) is marked OVERTAKEN
there: SHC has been writing them since resource_reservations.go landed.
2. The node capacity column did not need re-adding. It needed READING.
Section titled “2. The node capacity column did not need re-adding. It needed READING.”The brief for this lane, and scheduling_source.go’s own comment, both
said the column was dropped 2026-07-22 for want of a writer and that
“a future resource-aware scheduler re-adds the column together with a
writer”.
MEASURED: the columns came back over a week ago, with three writers each, and the scheduling source never learned about them.
| column | migration | writers |
|---|---|---|
node.cpu_cores | 20260730032524_node_cpu_cores.sql | joiner at enroll (modules/cluster/service.go:977), boot upsert (subsystems/impl/node_self_register.go:187), per-node fact-refresh tick (:332) |
node.memory_bytes | 20260804022217_auto.sql | same three (service.go:982, node_self_register.go:192, :363) |
So there is no migration in this lane and there must not be one. The
migration doctrine (make migrate, never flatten) was the recorded reason
to be careful here, and the v0.0.26 brick is why: a table-rebuild
migration inside a transaction hits FK 1811 and the daemon refuses to boot
on every node. That risk was never on this path.
2.1 What the un-read column was publishing
Section titled “2.1 What the un-read column was publishing”ListNodes scanned three columns and left capacity at its zero value.
SchedulingMetricsCollector is wired unconditionally
(ce subsystems/impl/collectors.go:341), so for every node whose row was
not being read:
shc_node_cpu_capacity_millicores= 0shc_node_cpu_scheduled_millicores= a real non-zero sumshc_node_cpu_schedulable_millicores=0 − scheduled= a published NEGATIVE quantity of schedulable CPU- both
*_schedule_utilization_ratiogauges = never emitted, because theircapacity > 0guard was the only thing standing between the fabricated zero and a division by it
★ The gauge that would have exposed the contradiction was the one the contradiction suppressed.
Fixed. The columns are read; NULL stays NULL through the scan;
NodeSchedule grows CPUCapacityKnown / MemoryCapacityKnown /
DiskCapacityKnown; and everything derived from a node’s SIZE is withheld
unless that size was measured. What SHC knows from its own rows, what a
node has been promised, how many stacks sit on it, still publishes
unconditionally. A measured oversubscription still publishes its
negative schedulable figure, because that one is true and is the thing a
rebalancer exists to fix.
The local-node runtime probe is now explicitly local-only. A remote node must never inherit this box’s core count.
3. The commitment operand is the RESERVATION x REPLICAS
Section titled “3. The commitment operand is the RESERVATION x REPLICAS”collectScheduled summed CPULimitMillicores / MemoryLimitBytes, once
per service. Wrong twice:
- The limit is the deliberately-oversold CEILING. The commitment is
the reservation, which is also the only figure Swarm’s own scheduler
consults, and the one
enforcement.go’s header already argued for on the request side. Summing ceilings and calling the difference “schedulable” reports the intended overcommit as a breach. - It ignored
desired_replicas, while the quota charge over the very same table multiplies by it. A 3-replica service was committing a third of what it holds.
Both fixed, with a guard that names each wrong answer separately so a regression cannot land quietly on either.
4. The multipliers: RULED, and the brief’s numbers were not the shipped ones
Section titled “4. The multipliers: RULED, and the brief’s numbers were not the shipped ones”The brief stated the model as reservation = quota x 1.0,
limit = quota x 1.5 and described that as what ships. MEASURED, ce
main shipped reservation = quota x 0.25, limit = quota x 1.0, the
owner’s shape with a competing design’s values, the residue of a
derivation that read the app author’s limit as its input.
The owner’s numbers are restored. What that changes, numerically:
| shipped (0.25 / 1.0) | ruled (1.0 / 1.5) | |
|---|---|---|
| committed memory, per unit of granted quota | 25% | 100% |
| the live estates’ ~20.68 GiB of authored ceilings, as commitment, on a 30 GiB node | ~5.17 GiB (17%) | ~20.68 GiB (69%) |
shc_node_memory_schedulable_bytes on that node | ~24.83 GiB | ~9.32 GiB |
| rendered container ceiling, vs authored | 1.0x | 1.5x |
| that node’s applied ceilings | 20.68 GiB | 31.02 GiB |
| tenant accounting charge | the quota | the quota (unchanged; the charge is the quota, always) |
★ It is a 4x swing on the commitment axis, not the 50% the brief assumed, and it is the difference between a capacity gate that can never fire and one that means something.
★ The two constants are ONE decision. Raising the reservation to 1.0
while leaving the limit at 1.0 renders reservation == limit on every
service; docker accepts it, and it tells the kernel never to reclaim a
container below its own hard cap. A floor jammed against its own ceiling
is not a floor. That state is neither model; it is the gap between them.
Blast radius and rollback, stated rather than glossed. Every service’s
effective ceiling is 50% larger on its next render, install or
update; a daemon roll re-renders nothing, so this arrives one deploy later
per stack. Ceilings are not allocations and both axes were already
oversubscribed by authorship alone (18.20 cores on 16). A ceiling going UP
refuses nothing and OOM-kills nothing that fits today; what it costs is
that a runaway container can take 50% more RAM before the kernel stops it.
Rollback is one key: shc config set deploy.limit_ratio 1.0 at global
scope.
4.1 Two defects the change surfaced
Section titled “4.1 Two defects the change surfaced”- A refused limit ratio was clamping to the DEFAULT, not the minimum.
Invisible while both were 1.0. At 1.5 it hands an operator who typed
0.5, plainly asking for tighter caps, a ceiling 50% looser than the minimum, as the consequence of refusing their input, while X200948QTALRT says the value “was clamped to 1.0”. Fixed with an explicitminLimitRatio, and the transform now distinguishes the zero value (unset → default) from a stated-but-illegal ratio (→ minimum). - ★ The whole-catalogue blast-radius guard was SELF-CONSISTENT on both axes. It computed its expected ceiling by calling the very scaler it was testing, so a mutation making limits ignore the ratio entirely passed over all 329 services. That hole predates this change; it was invisible while the multiplier was 1 and both numbers agreed. The multiply is now done in the test.
TestCatalogueEffectiveLimitsDoNotMove is re-pointed, not deleted, and
the new form is strictly stronger: every service’s ceiling must equal its
authored figure times exactly defaultLimitRatio. “Does not move”
caught movement in either direction only while the multiplier was 1;
“moves by exactly the multiplier” catches drift at any multiplier, and
catches a service scaled twice.
5. Total-vs-free: three gates, and they do NOT get the same answer
Section titled “5. Total-vs-free: three gates, and they do NOT get the same answer”The question was open in
docs/decisions/2026-08-04-physical-capacity-gates.md, and
enforcement.go’s header said why it could not be closed there: the fix
“needs an operand no signature here accepts, a call-site change, not a
change in this file.” The operand now exists.
| gate | answer | why |
|---|---|---|
EnforceNodeCapacity | FREE (total − committed) | “Is there room right now.” Against a total, the first deployment and the fortieth got the same verdict, and a node carrying 15 of its 16 cores admitted a 15-core request. |
EnforceLargestReplicaFitsNode | TOTAL, unchanged | ”Is any node physically big enough for ONE replica.” A 16 GiB service does not fit an 8 GiB node whether that node is empty or full. Subtracting usage could only make it refuse what would fit once something stops. It is also the only one of the three that catches a failure with no other detector. |
EnforceSwarmPoolCapacity | TOTAL, unchanged, understood as a FLOOR | ”The fleet cannot hold this at all.” A necessary condition, not a sufficient one. |
5.1 The ledger
Section titled “5.1 The ledger”Derived, not materialised, and that is the design rather than a shortcut:
every input is already replicated and three of the four CASCADE from
deployment, so an uninstall releases the commitment by deleting the
rows it was computed from. Nothing to credit back, nothing to drift, no
release path to get wrong. The alternative was measured before it was
rejected: a materialised ledger is usage on a second axis, and usage
needed its check and its charge folded into one statement batch after
eight concurrent charges all read used = 0 and all admitted, plus a
reconcile cron and a repair verb.
Three properties, each with its own mutation-proven guard:
- reservation x replicas, never the limit;
- the ANCHOR set UNION’d with
recorded_node_id(so a live deployment can never occupy nothing), andUNION, notUNION ALL, because the ordinary compose deployment has both halves and counting it twice reports every compose node at exactly double; - no status filter. A stopped deployment still holds its bytes and
wants its memory back the instant it starts. This is the
deployment_nodeprune rule with the sign of the danger flipped: there, a derived-empty set produced a PERMIT; here it produces an ADMIT. Both are the unsafe direction, so the rule is the same: absence of observation must never reduce a commitment.
5.2 The third unknown state
Section titled “5.2 The third unknown state”★ total known + committed unknown is not committed = 0. That is the
unknown-as-zero collapse one layer out and it fails worse: it admits
everything instead of refusing everything. An axis is judgeable only when
the total is known and the commitment is. A zero commitment across
zero deployments stays a measurement (the node is empty); a zero across
deployments whose reservation columns were never written does not.
5.3 What is deliberately NOT built
Section titled “5.3 What is deliberately NOT built”A fourth gate: a system headroom floor. total − committed >= request admits a node to exactly 100% reserved, and every real scheduler
keeps a floor for the daemon, rqlite, the docker engine and the OS. It is
not written because its numbers are not answerable from a desk: it cannot
be a percentage (10% of a cpx32 and 10% of a cx53 are different
amounts of daemon, and the daemon’s footprint does not scale with the
box), so it must be an absolute floor, and an absolute floor invented
rather than measured is a refusal threshold nobody can defend. Measuring
the daemon’s own footprint on both estates is the prerequisite.
6. Placement: the scorer left the drain orchestrator
Section titled “6. Placement: the scorer left the drain orchestrator”pickTarget (shc modules/cluster/drain.go) was the entirety of SHC’s
placement intelligence: sort by the count of running deployment ROWS,
tie-break by node id, prefer a runtime match. Unexported, reachable from
one caller, never consulted at install.
It counted the wrong thing. Over this catalogue the largest app declares 32.62 GiB of authored memory limits and the smallest 0.125, a 261x spread, so “one gitlab” and “one link shortener” weigh the same to a counter.
shc modules/placement is that function extracted and made
capacity-aware. Every property of pickTarget worth keeping is kept and
the tests say so by name: deterministic id tie-break,
runtime-match-beats-everything, a pure function over a slice, and (the
one whose loss would be the whole regression) the batch tally folded
back in between picks, so placing N deployments spreads them instead of
stacking all N on whichever node was quietest when the batch started.
Three rules that are easy to get backwards:
- An unknown axis ABSTAINS and the remaining weights renormalise. For
a scorer the unknown-as-zero failure is inverted from a gate’s and
just as bad: a node whose
memory_bytesis NULL scored as zero headroom never receives a deployment and silently leaves the fleet; scored as infinite it receives everything. A node that can answer nothing falls back to the counts, which is exactly today’spickTarget, so the degraded behaviour is the current behaviour rather than a new one. - ★ An abstaining candidate must not set the SCALE. Found while mutation-testing: the placeholder zero an abstaining node carries was reaching the min/max computation, so one unmeasured node rescaled a nearly-full node and an empty one from 1000 apart to 32 apart. The scorer kept working and quietly stopped discriminating. Same class as a pool sum that sums only the members who answered.
- It cannot refuse. A scorer that can refuse is a gate, and the two verbs have opposite costs of refusal: a drain that refuses strands deployments on the node the operator is emptying. Admission refusal is the capacity gates’ job, on a separate operand with a separate posture.
Default posture is SPREAD, weights memory 40 / cpu 20 / count 20 / tenant 10 / app 10. Memory outweighs CPU two to one because over-limit memory KILLS and over-limit CPU throttles. Bin-pack is the same function with the three resource weights negated: one scorer, one explain output, one test surface. The weights are a starting point and are not tuned; tuning them without production placement data would be fitting noise.
6.1 Install-time placement, and the door one line away from the right one
Section titled “6.1 Install-time placement, and the door one line away from the right one”resolveInstallTarget synthesises a descriptor with no NodeID for a stack
that does not exist yet, and the back-fill fills it from the request:
absent --node, the daemon that received the API call. That was the whole
policy: the machine you typed the command on.
The window is two conditions and both matter: the operator named no
node (an explicit --node is an instruction, not a hint) and the
stack does not exist yet (a stack that exists already has bytes
somewhere, and re-placing it here would be an unannounced move with no
data behind it).
★ The result goes in the ROUTING slot, never the PIN slot. Writing the
scored node into req.NodeID instead of target.NodeID lands it in
StackRepoRow.PinnedNode and, through app.PlacementIntent, in
deployment.pinned_node_id. On compose that looks harmless because it is
what already happens; on swarm it manufactures a hard constraint the
operator never asked for, the exact distinction StatedNodeID exists
to keep expressible, and one an earlier two-way fold in that file
destroyed once already. The existing fold is left alone and the derived
node is applied after it, to target.NodeID only; recordStackRow’s
existing firstNonEmpty(req.NodeID, target.NodeID) picks it up for the
Node column while PinnedNode stays empty.
The advisor also declines on a single-candidate fleet: one candidate is not a decision, and a scorer that “chooses” on a single-node estate starts logging placement decisions where there is no placement to decide.
Kill switch: deploy.placement_scoring ∈ on | off, default on,
through the ordinary 5-level cascade. off means do not score, not
“score and ignore”; a kill switch that still runs the code is a kill
switch that can still be crashed by the code. An unreadable knob leaves
scoring ON, because a disabled feature and a broken one look identical
from outside.
7. Guards
Section titled “7. Guards”Every guard in this lane asserts a non-empty corpus before it asserts anything else, and every one was mutation-proven: the mutation was applied, the test was watched to fail naming the defect, and the tree was restored.
| guard | mutation that must kill it | caught |
|---|---|---|
TestListNodesReadsCapacityColumns | stop scanning cpu_cores | yes |
TestNodeScheduleNeverPublishesNegativeSchedulable | same | yes |
TestCollectScheduledSumsReservationTimesReplicas | revert to the limit operand | yes |
TestCommitmentDoesNotDoubleCountTheAnchorAndTheScalar | UNION ALL | yes |
TestCommitmentCountsStoppedDeployments | add a status = 'running' filter | yes |
TestEnforceNodeCapacityComparesAgainstFree | compare against total | yes |
TestBatchSpreadsAcrossNodes | make Commit a no-op | yes |
TestUnmeasuredNodeIsNeitherStarvedNorFlooded | zero-fill an abstained axis | yes |
TestAbstainingCandidateDoesNotSetTheScale | let the abstained placeholder into min/max | yes |
TestFirstInstallTakesTheScoredNodeAsROUTINGNotAsAPIN | write the scored node into req.NodeID | yes |
TestAnExistingStackIsNeverRePlaced | drop the “does not exist yet” condition | yes |
TestAnExplicitNodeIsNeverOverridden | drop the operator-pin condition | yes |
TestAdvisorSkipsDrainingAndQuarantinedNodes | drop availability; drop the cluster_state allowlist | yes (both) |
TestAdvisorRefusesToScoreSwarm | drop the swarm guard | yes |
TestAdvisorUnreadableKnobLeavesScoringOn | invert the kill-switch default | yes |
TestCatalogueEffectiveLimitsMoveByExactlyTheRatio | limits ignore the ratio; ratio applied twice; reservation scaler halved | yes (all three, over 329 services) |
★ Two mutations that did NOT kill their guard first time, both fixed rather than explained away: the catalogue guard asked the code under test what it should have done (self-consistent on both axes), and the scorer let an abstaining candidate set the normalisation scale.
8. What this lane deliberately did not do
Section titled “8. What this lane deliberately did not do”- No autonomous trigger, no timer, no move. §0.
- No fourth gate. §5.3; its numbers need a measurement nobody has taken.
- No durable utilization digest. The scorer reads intent (replicated, answerable while a node is unreachable), never a live sample. A rebalancer that plans on instantaneous samples thrashes; that axis is phase 2 and no gate needs it.
- No per-node disk fact.
nodehas no disk column and the local probe answers only for the local node, soshc_node_disk_capacity_bytesis now SKIPPED for remote nodes rather than published as a zero. The move’s staging preflight still measures only the executor’s disk and cannot ask a remote node; that capability is owed to the move lane. - No
shc placement explaincommand. The data is carried out of the scorer (Decision.Allholds every candidate’s score and every abstained term with its reason); nothing renders it yet.
Related
Section titled “Related”docs/architecture/cluster/swarm-contention-map.md§8, where R1 is asked and now answereddocs/decisions/2026-08-04-physical-capacity-gates.md: the three gates and the total-vs-free question this page closesshc modules/quota/commitment.go,enforcement.go: the ledger and the gatesshc modules/placement/: the scorer and the live advisorce modules/stack/placement_seam.go: the install-time seamce modules/app/resource_quota.go,resource_reservations.go: the multipliers