Skip to content
SHC Docs

What triggers a move, and how it cannot run away

Status: open design. Automatic rebalance is not built. A move runs only when an operator asks for one.

★ SUPERSEDED IN PART: read docs/decisions/2026-08-11-rebalancer-v1-foundations.md first

Section titled “★ SUPERSEDED IN PART: read docs/decisions/2026-08-11-rebalancer-v1-foundations.md first”

This page is the DESIGN DRAFT. The lane that built from it landed on 2026-08-11 and measured several of its claims differently. Where this page and that decision disagree, the decision is what was measured and this page is kept for its reasoning, not for its facts.

The corrections that matter here:

  • NONE of this page is built, deliberately. v1 is placement plus the substrate: no autonomous trigger, no rebalance timer, nothing that moves a live deployment, no failover. The asymmetry is the reason; a bad first placement is undone by the next install, while a bad rebalance is a full stateful cutover whose revert window closes at the commit point.
  • What this page’s R1..R6 target contract will be able to READ, once somebody builds it, is now real: the per-node commitment ledger (shc modules/quota/commitment.go) and the scorer (shc modules/placement). The commit_ratio its thresholds are expressed in is computable today.
  • Its threshold numbers were written against a commitment operand of quota x 0.25. That operand is now quota x 1.0, so every ratio in §R1..R6 is 4x what it would have been and the low/high bands need re-deriving before anything gates on them.

Everything else on this page stands, including the parts the implementation did not reach; they are named in §8 of the decision.

As written: PROPOSED. Nothing here is built. This is lane 4 of the automatic deployment-distribution design: the trigger, the safety envelope, the relation to shc node drain, the per-app standby declaration, and the timer the move reconciler still lacks. Six rulings are owed (§9). Two claims in the brief this lane was handed are corrected against HEAD (§1.2), and one measurement puts a hard ceiling on what the feature can do on the estates that exist today (§2.6); the ceiling is not a defect and the design does not try to route around it.

Baselines. shc main c0b009ba5, ce main 6a3dcc5, both read on 2026-08-11. Every figure below is marked MEASURED (I read it in those trees or computed it over the catalogue in this checkout), RECORDED (a source comment states a measurement I did not repeat), or PROPOSED (a design choice with no evidence behind it yet).

The one-line recommendation. Build the evaluator, the decision record and the dry-run planner first and let them run for weeks with no actuator at all; make the trigger a pressure-relief valve, never an evener; make the per-deployment cooldown the outer bound on runaway; and refuse to auto-move any app that has not declared a standby mode.


1. What exists, and what the brief got wrong

Section titled “1. What exists, and what the brief got wrong”

MEASURED. The move primitive is largely there and it is careful. The durable node-local journal (ce/modules/clone/move_journal.go), the daemon-start reconciler (ce/subsystems/impl/move_journal_recovery.go, priority 73, ModeAny, 15-minute ceiling), the container-observation health gate (move_health.go) and the byte-shortfall data gate (move_data_gate.go) all exist, all fail closed, and the destroy guard already refuses a move against a protected deployment (protection.go).

MEASURED. Nothing schedules. The only node-scoring code in either repo is pickTarget at shc/modules/cluster/drain.go:512: lowest count of running deployment rows, ties broken by node id, runtime match preferred. It is unexported and its sole caller is Drain. No CPU, memory, disk, quota or affinity term appears in it.

MEASURED. shc node drain is EE-gated. Fleet verbs register through registerFleetCommands; the CE binary gets refusing stubs (ce/modules/node/commands/main.go:142-148, code X403028CLIEEO). Whatever this lane builds inherits that gate, because it is strictly more than a drain.

CORRECTION 1: commitments are summed, per node, on every metrics tick. The brief says “totals exist, nothing sums commitments”. MEASURED: ce/subsystems/impl/scheduling_source.go implements metrics/collectors.SchedulingSource and is wired at ce/subsystems/impl/collectors.go:341. Per node it produces CPUCapacity, MemoryCapacity, DiskCapacity, CPUScheduled, MemoryScheduled and StackCount. The commitment sum exists and runs today.

It is not usable as-is, for four reasons, each of which is a stage-1 work item:

defectevidence
it sums limits, not reservations (sv.CPULimitMillicores, sv.MemoryLimitBytes in collectScheduled)docs/decisions/2026-08-04-physical-capacity-gates.md §3 measured the limit operand to be the wrong one, and measured it refusing installs that succeed today
a stack whose Node is empty is bucketed under the literal key "local", which matches no node rowscheduling_source.go collectScheduled
swarm-runtime stacks aggregate cluster-wide rather than per nodesame function; correct for swarm, but a mixed fleet gets an un-attributable bucket
the output goes only to OTLP gaugesce/modules/metrics/router.go:13: “there is NO /metrics scrape endpoint; push-only”. No decision path can read the number back

CORRECTION 2: cross-node live utilization exists on the leader. The brief says “StatsCache is per-process”. MEASURED: ce/modules/metrics/collectors/peer_nodes.go is a leader-only fan-in that asks every peer for its local NodeStats over the cluster mTLS internal API and merges the answers into the leader’s own cache; it is wired at collectors.go:368 and fires on the metrics tick, default 2s (resolveTickInterval, config timing.tick_interval). NodeStats carries CPUPercent, MemoryPercent, DiskPercent, DiskUsedBytes, DiskTotalBytes, Load1m/5m/15m and ContainersRunning per node.

What remains true: the cache is in-memory, unpersisted, and empty before the first tick after a restart. An evaluator must therefore refuse to run against a cold cache rather than read zeros as calm.

Related, and softer than the brief states: per-node free disk is not “no fact anywhere”. ce/modules/clone/preflight_space.go:134 has a real DiskFreeBytes (Bavail x Bsize) probe used by clone preflight, NodeStats carries used and total per node cluster-wide on the leader, and a get_disk_usage internal RPC exists. What is missing is a durable column, not a measurement.

1.3 Two numbers the metric has to be built around

Section titled “1.3 Two numbers the metric has to be built around”

MEASURED, over the 104 apps/**/compose.yaml files in this checkout:

  • 94 declare deploy.resources.limits. Zero declare deploy.resources.reservations. (A plain grep reservations: finds one hit, in apps/standalone/ollama; it is inside a comment showing how to request a GPU. Parsing the resources: blocks properly returns nothing.)
  • 81 carry an x-shc-scale block; of the 74 my parse read cleanly, zero declare a backup: or restore: window. This agrees with ce’s own comment in daemon/backup_scaler.go (“all are max-only”).

MEASURED, quota derivation at HEAD (ce/modules/app/resource_quota.go:13, resource_reservations.go:105): limit = quota x deploy.limit_ratio, default 1.0, range [1, inf); reservation = quota x deploy.reservation_ratio, default 0.25, range (0, 1], with per-axis overrides deploy.cpu_reservation_ratio / deploy.memory_reservation_ratio. The brief’s “reservation_multiplier (1.0), limit_multiplier (1.5)” matches neither the key names nor the defaults in this tree.

The consequence, and it is the honest limit of this whole feature. The capacity decision says the operand must be the reservation. But no app in the catalogue declares one. Every reservation in either estate is quota x 0.25, derived, written into the rendered artifact and then persisted into service.cpu_request_millicores. So a rebalancer that balances committed reservations is balancing a policy number the operator invented, scaled by a constant; arithmetically it is the limits sum times a quarter, carrying no independent information about what any app needs. That is why the trigger in §2 requires an observed signal as well, and why §2.2 keeps the two terms separate instead of blending them into one score.

MEASURED, the persistence that makes the join cheap: ent service already carries cpu_quota_millicores, cpu_request_millicores, cpu_limit_millicores, memory_quota_bytes, memory_request_bytes, memory_limit_bytes and desired_replicas; ent deployment carries recorded_node_id and pinned_node_id. A per-node committed-reservation sum is one grouped query. No new column is needed for it.


2.1 The shape of the decision, before any formula

Section titled “2.1 The shape of the decision, before any formula”

A rebalancer can be one of two things, and the choice governs everything else:

  • an evener, which drives the fleet toward a flat utilization curve; or
  • a pressure-relief valve, which acts only when a node is genuinely in trouble and otherwise leaves an uneven fleet alone.

PROPOSED: SHC builds the valve. An evener moves stateful stacks on a healthy fleet to improve a number nobody is suffering from, and each of those moves costs a real outage window (journalPhaseStop opens the downtime window; MEASURED, move_journal.go). The Kubernetes analogy the owner reached for breaks here: the descheduler evicts and lets the scheduler re-place, cheaply, because pods are disposable. SHC has no scheduler to re-place into, and a relocation is a backup, a ship and a restore. An uneven fleet where every node is comfortable needs no moves.

The imbalance number the brief asks for therefore still gets defined, but it is a reported figure, not a trigger. See §2.3.

The evaluator reads four things. Each carries a freshness stamp and a DERIVED | MEASURED flag into the decision record (§3.4), because a threshold crossed by a derived number and one crossed by a measurement are different claims.

inputsourceflag
capacity(n): cpu millicores, memory bytes, disk bytesent node.cpu_cores x 1000, node.memory_bytes, node.resources.disk_total_bytes; NULL = unknownMEASURED (probe), may be unknown
committed(n): sum of reservations of rows recorded on ngrouped query over service joined to deployment on recorded_node_id, Σ cpu_request_millicores x desired_replicasDERIVED for every app in the catalogue (quota x 0.25)
observed(n): cpu / memory / disk percentthe leader’s fan-in StatsCache, sampled at timing.tick_intervalMEASURED
free_disk(n)DiskFreeBytes per node, durably recorded (stage 1)MEASURED

An unknown axis is never a breach. The per-axis unknown guard already landed in shc/modules/quota/enforcement.go (knowsCPU / knowsMem, which also read a pointer-to-zero as unknown); the evaluator must use the same disposition. A node whose memory total is NULL is not a node under memory pressure; it is a node the fleet cannot answer for, and it is excluded as both source and target with UNKNOWN_CAPACITY in the record.

capacity(n, axis) per the table above
committed(n) Σ over deployments d recorded on n,
over services s of d,
s.<axis>_request x s.desired_replicas
commit_ratio(n, axis) = committed(n, axis) / capacity(n, axis)
sustained(n, axis, w) = the 10th percentile of the observed samples for
(n, axis) over window w
-- i.e. the level that 90% of the window exceeded
pressure(n) = max over axes of max( commit_ratio, sustained )
spread = max pressure(n) - min pressure(n), over active nodes

spread is the imbalance metric, and it never triggers anything. It is what shc node capacity and shc rebalance plan print, what an operator watches, and what a future ruling could promote into a trigger once it has been observed on a real fleet for a quarter. Promoting it is a one-line default change; demoting it after it has caused a move is not.

The trigger is a two-gate test, and both gates must pass.

Gate 1: PRESSURE. A node s is a candidate source only if, for at least one axis, pressure(s, axis) > high where high defaults to 0.90. sustained is a windowed 10th percentile, so the “node hit 71% for ten seconds” case cannot arm the gate at all; 90% of a 30-minute window has to be above the line. That is the hysteresis, built into the metric rather than bolted beside it.

Gate 2: BENEFIT. For a candidate deployment d on s and a candidate target t, all six must hold:

R1 commit_ratio'(t) <= low (low = 0.75)
R2 commit_ratio'(s) < high (the move actually relieves)
R3 max(p'(s), p'(t)) <= max(p(s), p(t)) - min_improvement (0.10)
R4 free_disk(t) >= size(d) x disk_headroom (2.0 -- the heavy strategy
stages an archive AND the
restored tree)
R5 t is active, not draining, not in target cooldown, and t != s
R6 every one-hop connected peer of d is already on t, or d has no peers

where p' is pressure recomputed with d’s reservation subtracted from s and added to t.

R4 reuses clone/preflight_space.go rather than re-deriving a headroom rule. Two space calculations that must agree are one calculation.

R6 is the restraint that matters most, and §2.6 explains why it will refuse nearly everything on today’s estates.

2.4 Cadence: three clocks, only one of them new work

Section titled “2.4 Cadence: three clocks, only one of them new work”
  • Sampler: unchanged. The 2s metrics tick already fills the leader’s cluster-wide cache. Add a bounded ring buffer per (node, axis) so the window quantile is cheap; add no new poll. PROPOSED: a 30-minute window at 2s is 900 samples per series, which at three axes and a ten-node fleet is 27,000 float64, trivial, and deliberately in memory, so a restart clears it and the evaluator refuses until the window refills. A rebalancer that acts on four minutes of history after a daemon restart is a rebalancer acting on the restart.
  • Evaluator: a new leader-gated cron, every 5 minutes (crons.rebalance_evaluate). It writes a plan and a decision record. It never moves anything, in any mode.
  • Actuator: a second leader-gated cron, every 15 minutes (crons.rebalance_actuate). It picks at most one approved item and calls the existing move path.

Why 5 and 15, and not tighter. RECORDED in move_journal_recovery.go: measured crowdsec relocations completed end to end in 38–51s. A gitlab-class stack is minutes plus a backup. An actuator that fires faster than the work it starts is a queue wearing a control loop’s clothes. Splitting evaluate from actuate is also what makes stage 2 shippable at zero risk; the evaluator is the whole feature minus the danger.

Registration mechanics, MEASURED. Both crons register through core/registry.CronsAll and are fired by subsystems/impl/cron_runner.go, which already guards overlapping runs of the same cron (inflight map, skip warn X503087CRNSKP) and already resolves a per-cron config override under crons.<name>. Schedules must use the prose form ("every 5 minutes"); schedule.Parse rejects "@every 5m" and the runner then drops the job silently, RECORDED at ce/modules/backup/crons.go:267, where the same trap would have deleted a crash-recovery net.

2.5 Hysteresis: six dampers, each against a distinct failure

Section titled “2.5 Hysteresis: six dampers, each against a distinct failure”
  1. The windowed quantile (§2.3). Against transient spikes.
  2. Dual thresholds, arm at high 0.90, disarm at low 0.75. Against a node oscillating across one line.
  3. Per-deployment cooldown, default 7 days. A deployment that moved is not a candidate again until it expires, whatever the metric says. ★ This is the single most important rule in the document, because it is the only one that bounds the worst case independently of whether the policy is correct: a given stack can move at most once a week even if every other gate is wrong.
  4. Per-node target cooldown, default 1 hour. A node that just received a stack has metrics that do not yet describe it.
  5. Settle period, default 30 minutes with no evaluation after any install, uninstall, drain, node join or leave, or daemon restart anywhere in the fleet. A fleet mid-change is a fleet whose numbers lie.
  6. Plan stability. The actuator executes an item only if the identical (deployment, target) pair appeared in the last 3 consecutive evaluations, 15 minutes of agreement. A flapping metric produces a flapping plan, and a flapping plan executes nothing. This is the damper that survives a policy bug, because it does not depend on the policy being right, only on it being stable.

2.6 ★ The ceiling: measured movability, and why the honest answer is “almost nothing”

Section titled “2.6 ★ The ceiling: measured movability, and why the honest answer is “almost nothing””

RECORDED, in ce/modules/clone/service.go:1408-1418 and the block at :1200-1225, measured on promanager 2026-08-09:

  • keycloak’s one-hop neighbourhood is 12 deployments;
  • keycloak’s transitive closure is 31 of 33 deployments;
  • before --closure existed, 31 of 33 deployments were unmovable by any combination of flags;
  • moveClosureMaxStacks = 8, chosen as a policy statement so --closure stays a relocation flag rather than an estate-migration button.

The automatic rebalancer must never pass --closure, and must never pass --all-connections either. R6 in §2.3 restricts it to deployments whose one-hop set is empty or already co-located with the target. On an estate where one shared provider reaches 31 of 33 rows, that will refuse nearly every candidate.

State it plainly: on the two estates that exist today, this feature will find almost nothing it is allowed to move. That is the correct behaviour and not a bug to route around. Its value lands in three places instead:

  1. the decision record, which tells an operator why the fleet is stuck; “31 rows rejected CONNECTED, naming keycloak” is a far better artifact than a rebalancer that moved them;
  2. a future fleet of many small independent stacks, where R6 passes;
  3. the drain path, which shares the machinery and is not restricted the same way because an operator asked for it explicitly.

An honest design says the ceiling out loud rather than widening R6 to look useful. Widening R6 is how a rebalancer relocates thirty deployments nobody asked about.


Three surfaces, because one is not enough, and the split respects the config-versus-state ruling (a setter verb must not store state in config).

  1. rebalance.mode: the policy knob. Scoped config, three values: off (the evaluator does not run), observe (evaluate and record, never act), act. Default observe from stage 2 onward, act never by default. Resolved through the ordinary 5-level cascade, so a single tenant can be opted in first.
  2. <state>/rebalance.stop: the state-side hard stop. A node-local file. Presence refuses every new move on that node. It is a file for the same reason move_journal.go is a file (its own header argues this): it must work when the replicated store has no quorum, and quorum trouble is exactly what a misbehaving rebalancer causes. Checked before each item, not once per pass.
  3. shc rebalance stop writes the file, fans out to every reachable node, and reports which nodes it reached and which it did not. A kill switch that reports partial coverage is the only honest kind. Because the actuator is leader-gated, an unreachable follower cannot start a move while it is unreachable; the residual risk is a follower that later becomes leader, and the message should say that rather than imply totality. Making the stop durable is a separate, explicit shc config set rebalance.mode=off.

★ The switch never interrupts a move mid-phase. It makes the next phase boundary refuse, and the move journal’s existing disposition (revert by default, refuse after teardown) is what runs. A kill switch that aborts a relocation at an arbitrary instant is a data-loss button with a reassuring name.

★ Asymmetry, stated because getting it backwards is fatal: the stop file blocks new moves and never blocks a resume or a revert. A rebalancer halted mid-move still needs its recovery to complete.

knobdefaultenforced where
rebalance.max_in_flight1leader, fleet-wide
rebalance.max_in_flight_node1 as source, 1 as targetthe node itself
rebalance.max_in_flight_tenant1leader

A node that would be both source and target of two concurrent moves is refused outright.

The count is taken from the move-journal directory, not from an in-memory counter; the journal is what survives the restart a counter does not.

★ The honest weakness: a fleet-wide count needs a fan-out, and a partitioned node can start a move the leader cannot see. So the per-node cap is authoritative and fail-closed (each node counts its own journal directory and refuses on an unreadable one), and the fleet-wide cap is advisory. With a per-node cap of one and a leader-gated actuator, the true worst case is one concurrent move per node, not per fleet. Design for that number rather than claiming the smaller one.

Where it lives: the scoped config cascade, as rebalance.never_move (boolean), resolvable at deployment, stack, environment, tenant and global. Not a new column and not a new table; the migration doctrine and the v0.0.26 brick both argue against a migration for a feature that ships off. The same cascade already carries deploy.reservation_ratio, and its *Source reporting is the precedent for naming which scope decided (§3.4).

Setting it true at global scope is a third kill switch, and that is deliberate.

Four conditions imply never-move and cannot be overridden:

  • pinned_node_id is non-empty; an explicit --node is an operator’s placement decision (MEASURED: PlacementIntent writes it only from an explicit --node);
  • the deployment is protected; protection.go already refuses move outright, fail-closed on an undeterminable answer (X409624CLNPRU);
  • the runtime is swarm; swarm owns placement;
  • the app declares no standby mode (§5).

Fail closed. An unreadable annotation is never-move, matching protection.go’s disposition exactly.

★ A naming trap worth avoiding: shc protect refuses the operator’s move as well as an automatic one. rebalance.never_move must refuse only the automatic mover. They answer different questions and must not collapse into one flag; the existing overload of protect is visible from here and is flagged as an adjacent ruling in §9.

3.4 ★ The decision record: the artifact that makes this debuggable instead of spooky

Section titled “3.4 ★ The decision record: the artifact that makes this debuggable instead of spooky”

Written by the evaluator, once per pass, including on the pass that decided to do nothing, which is the commonest outcome and the one an operator most wants to interrogate (“why is it not fixing my hot node?”).

decision_id uuid
evaluated_at RFC3339
evaluator_node the leader that ran the pass
policy every resolved rebalance.* value AND, per key, the scope
that set it (precedent: the quota *Source fields)
policy_hash so two records can be compared at a glance
metric per node, per axis:
commit_ratio, capacity, committed,
sustained(window), samples_in_window,
samples_above_threshold, threshold,
freshness_seconds, DERIVED|MEASURED
spread the reported imbalance figure (§2.3)
candidates every deployment on every pressured node, each with:
tenant/stack/environment, size_bytes, reservation,
observed usage, one_hop_peers[]
rejected per rejected candidate, ONE reason code plus its evidence:
NEVER_MOVE(scope that set it)
PINNED(node)
PROTECTED
NO_STANDBY(the app declares none)
CONNECTED(the peers that would be stranded, by name)
COOLDOWN(last_moved_at, eligible_at)
UNKNOWN_CAPACITY(axis)
NO_TARGET(per target: which of R1..R6 refused, with the
two numbers that failed the comparison)
NO_BENEFIT(projected p'(s), p'(t), improvement, minimum)
TOO_BIG(size_bytes, free_bytes per target)
selected (deployment, target) or NONE, with the reason NONE
projection p(s), p(t) before; p'(s), p'(t) after; improvement
stability how many consecutive passes have agreed on this item

Storage, split deliberately. One row in the existing hash-chained audit table per decision, carrying decision_id and the one-line verdict; that table is append-only with prev_hash / hash (MEASURED, ce/ent/schema/audit.go), which is the right home for the tamper-evident claim and the wrong home for a 200-candidate dump. The full record lands as JSON under <state>/rebalance/decisions/<id>.json with a retention.

Operator surfaces:

  • shc rebalance plan: run an evaluation now, print the record, change nothing. This is the estate dry-run planner the brief says does not exist, and it is the deliverable of stage 2.
  • shc rebalance why <stack>: the most recent record that mentions this stack, selected or rejected, with the reason code and its numbers. ★ The record must be as complete for a rejection as for a selection. A system that only records what it did cannot answer “why didn’t it”, and “why didn’t it” is the question an operator actually asks.
  • shc rebalance history: decisions and the outcome of each acted item.

4. (c) Drain semantics, and exactly how the rebalancer relates to it

Section titled “4. (c) Drain semantics, and exactly how the rebalancer relates to it”

Same machinery below the picker; different machinery above it. The difference is the safety story, so it must be structural rather than a comment.

Shared, unchanged: DeploymentMigrator.Migrate and everything under it: the move primitive, its journal, the health gate, the data gate, the rollback. The rebalancer is a caller of the move path and adds no new way to relocate a deployment. That is not a convenience; a second relocation path is a second set of gates to keep correct.

Not shared, and the reasons:

shc node drainthe rebalancer
triggeran operator names a nodea metric crosses a sustained threshold
scope of one runevery running row on the nodeat most one deployment per cycle
target contractany active peer beats leaving the row where it isa target must be an improvement (R1–R6) or there is no move
failureends the run, node left draining, per-stack reasons on result.Failed (MEASURED, drain.go:269)one failed item; nothing is left half-flagged
deadlineone budget for the whole run, default 300s via timeouts.drainper item
--forcestops rows in place when no target existsno analogue, ever

★ --force deserves its own line. Drain’s force flag asks the migrator to stop a row in place. That is a reasonable answer when an operator is taking a node away. It is never a reasonable automatic answer to load: “the fleet was busy so I stopped your app” is an outage nobody requested. The rebalancer must not gain this flag, and the refusal should be written into the code rather than left to reviewer memory.

PROPOSED, and small: promote pickTarget and candidateTargets out of drain.go into a placement package with two named strategies: StrategyEvacuate (drain’s rule: any active peer, prefer least-loaded, prefer runtime match) and StrategyImprove (the rebalancer’s: refuse unless the benefit gate passes). Same candidate construction, two explicit verdicts, nothing about drain’s behaviour changes. The alternative (one picker with a mode flag) is how drain’s “any target beats none” rule leaks into the rebalancer on some future refactor.

Two interlocks between them:

  • The rebalancer refuses to evaluate while any node is draining. Two relocators on one fleet is a fight, and drain is the one that was asked for.
  • Drain flips availability to draining so that “schedulers stop placing new work on the node mid-flight” (MEASURED, drain.go:170), a comment that today describes a scheduler that does not exist. When this lands the comment becomes true, and honouring availability != active as a hard target filter is mandatory. Reusing candidateTargets gets it for free.

One efficiency note. candidateTargets calls ListByNode once per candidate node, per drain (MEASURED). That is fine at drain scale and wrong at a five-minute cadence over a fleet; the evaluator needs one grouped query, not N.


5.1 The three-state cutover already exists in this codebase, under another name

Section titled “5.1 The three-state cutover already exists in this codebase, under another name”

MEASURED. The backup quiesce bracket is the design lane 4 was asked to invent:

  • ce/modules/backup/hook_bracket.go: openBackupBracket fires the pre half and returns a closer the caller defers, so the undo runs on every exit path, and an unarmed bracket closes as a no-op.
  • ce/modules/backup/quiesce_journal.go: journal-then-arm, sha256 filename, an in-process liveQuiesceHolders sync.Map so a sweep can tell “abandoned” from “in flight”, and a clear only after a verified undo.
  • backup_quiesce_resume cron, every 1 minute, on every node, plus a boot kick fired when the backup service is published.

And the persistent toggles it manages are exactly the standby vocabulary: postgres and supabase ALTER SYSTEM SET default_transaction_read_only = on (written to postgresql.auto.conf, surviving container and host restart), mysql and mariadb SET GLOBAL super_read_only = ON, mongodb db.fsyncLock(), nextcloud php occ maintenance:mode --on.

★ And its own header states the cost in the right words: “a quiesce is not backpressure; it does not make a client WAIT, it makes it FAIL”. A standby window is an outage that the application’s clients see as 5xx, logged in the app and not in SHC. Any operator-facing text for this feature must say so.

So standby is not a new mechanism. It is the quiesce bracket, generalised, with a journal whose shape is already proven.

5.2 The declaration: two layers, both of which already exist

Section titled “5.2 The declaration: two layers, both of which already exist”

Layer 1: app.yaml hooks: a new pre_standby / post_standby pair, alongside the existing pre_backup / post_backup. Same job-file mechanism, same bracket object, same journal. For most apps that already quiesce, the pre_standby job is the existing one; nextcloud’s would be enable-maintenance-mode, which its app.yaml already names.

Layer 2: x-shc-scale gains a standby: field per service (a fourth field beside max, backup, restore), meaning “run this many replicas of this service while the deployment is in standby”. standby: 0 on mautic-worker is the declaration that stops the worker.

Layer 2 is nearly free, and that is the argument for it. MEASURED: daemon/backup_scaler.go already scales named services to a window count and back through app.ScaleService; app.ScaleWindowOps already reads the block from the rendered compose (post-Jinja, always parseable, RECORDED in scale_window.go: the source template failed a plain unmarshal for 29 of the 48 apps declaring when that note was written, which is why the render now preserves the marker); extension_lint.go’s xshcRegistry holds a closed field set for x-shc-scale with a test that derives the expected set by walking the Go sources, so a typo’d standbyy: is a refusal rather than a silent no-op. Adding a field is one registry entry, one checker arm and one reader.

Why both layers, and this is the honest cost. Layer 2 handles services SHC can see. MEASURED over the catalogue: only 13 of 104 compose files declare a service whose name looks like a background worker (dify, windmill, medusa, monica, pulp, hyperswitch, keygen, mautic, frappe, journiv, plane, and both defectdojo variants). That count undercounts the problem badly: nextcloud’s compose has four services and none is a cron; its cron runs inside the nextcloud container; gitlab-omnibus runs sidekiq inside one container. Name detection cannot find those writers, and neither can any other automatic inspection. That is precisely why the declaration has to be authored, and why layer 1 exists.

An app author who declares standby asserts three things, and the app-development doc must say them in these words:

  1. after pre_standby completes, no process in this deployment writes to its own data or to any database it consumes;
  2. post_standby returns it to normal, and running post_standby against a deployment that never entered standby is a no-op;
  3. the standby state either survives a container restart, or the app says it does not, in which case it is not auto-movable. Postgres’s default_transaction_read_only survives; a stopped worker under restart: unless-stopped does not.

Current declaration coverage, MEASURED over the 104 app.yaml files in this checkout: 26 declare pre_backup; 8 declare post_backup; 8 declare both (nextcloud, uptime-kuma, supabase, postgres, mail, mariadb, mongodb, mysql). The 18 with a pre and no post are dump-style hooks, not quiesce brackets. So the paired enter/leave shape exists on 8 of 104 apps today; that is the adoption baseline, and it is small.

5.4 ★ What happens to an app that does not declare standby

Section titled “5.4 ★ What happens to an app that does not declare standby”

Recommendation: it is simply not auto-movable. It appears in the decision record as NO_STANDBY and never as a selection.

One line of justification, which is the whole argument: a fleet that silently double-runs cron against one live database is far worse than a fleet that moves fewer things.

The pleasant side effect is that the feature advertises its own adoption path. An operator who runs shc rebalance why gitlab and reads NO_STANDBY(gitlab declares no pre_standby hook) has been handed the exact work item, by name, instead of a shrug.

5.5 The -db companion, stated more precisely than the brief does

Section titled “5.5 The -db companion, stated more precisely than the brief does”

The brief frames the danger as “moving the app but not the DB double-runs cron against one live database”. Sharpened:

  • The move primitive as it exists today does not double-run anything. It stops the source before recovering the target; journalPhaseStop is labelled “THE DOWNTIME WINDOW OPENS HERE” (MEASURED). Source and target are never both serving.
  • Moving an app away from its database is already refused. X400600CLNCON names the peers that would be stranded, and peersOutsideTheMove refuses a one-hop set that is not closed (MEASURED). R6 in §2.3 keeps the rebalancer on the safe side of that refusal rather than reaching for --closure.
  • So the double-run risk is specific to a future low-downtime cutover, the finish-forward verb that “is reserved as an operator verb that does not exist yet”, or any design that starts the target before stopping the source.

Therefore: standby is not required by today’s move primitive. It is required the instant anyone implements an overlapping cutover. Declaring it now is cheap and reviewable; retrofitting it after someone has shipped an overlap is an incident. That is the argument for building stage 3 before stage 5, not the other way round.

5.6 The standby leak, and why it needs no new thinking

Section titled “5.6 The standby leak, and why it needs no new thinking”

A move killed between pre_standby and post_standby leaves nextcloud serving a maintenance page forever. This is the same failure the quiesce journal exists for, and it has the same answer: a standby_journal with the quiesce journal’s exact shape: journal-then-arm, sha256 filename, a liveStandbyHolders map, a standby_resume cron every 1 minute on every node, and a boot kick.

Do not write a third variation. quiesce_journal.go opens with “THIS FILE IS A DELIBERATE COPY OF stop_journal.go, not a variation on it”; that sentence is an instruction to the next author, and this is the next author.


6. (e) What the move reconciler still needs

Section titled “6. (e) What the move reconciler still needs”

MEASURED, the current state: MoveJournalRecovery runs at daemon start only. ModeAny, priority 73, deps {container_runtime, ent_client}, a 15-minute pass ceiling, work done in a goroutine so boot is not held open, returns nil on every path so an unresolvable record cannot stop the daemon. There is no timer.

MEASURED, its two siblings both have one: backup_quiesce_resume and backup_stop_resume, both every 1 minute, both on every node, both with a boot kick and an in-process holder map. The move journal is the odd one out, and the gap is visible: an interrupted move currently waits for a daemon restart that may not come for weeks.

PROPOSED: a move_journal_resume cron, built as the sibling rather than as a new shape.

  • Cadence: every 1 minute, eventually. The quiesce cron’s own argument applies verbatim; every tick a deployment waits is downtime nobody asked for, and the sweep is one readdir of a directory that is empty on almost every node almost always, so the cost of the cadence is a syscall. Ship it at every 5 minutes first (§8, stage 0) and tighten only after a proving run shows zero spurious reverts. Use the prose form; "@every 1m" is silently dropped.
  • Node-local, ModeAny, not leader-gated. Same structural reason the boot recovery is ModeAny: shc move forwards to the source’s home node before any phase runs, so the record lives on a node that is very often not the leader. A leader-gated sweep would scan the node least likely to hold anything and report clean.

Five interlocks, and the first two are what make a timer safe where a boot kick needed neither:

  1. In-process holder map. A liveMoveHolders sync.Map keyed on operation id, so the sweep skips a record a live orchestrator in this process still owns. Without it the timer reverts moves that are running. The boot kick never needed this; a boot proves the old process is gone.
  2. An age floor. Do not act on a record whose UpdatedAt is newer than max(2 x the phase's own timeout, 5 minutes). A young record is not abandoned; it belongs to a process that may simply be slow. Again, only the timer needs this.
  3. Overlap. CronRunner already guards overlapping runs of the same cron. Inherit it; do not add a second guard.
  4. The 15-minute pass ceiling stays, and a pass that hits it leaves the record for the next pass. Unchanged disposition: nothing is abandoned by a timeout.
  5. The kill switch, asymmetrically. The stop file blocks new moves and never blocks this sweep. A halted rebalancer still needs its reverts.

★ The honest cost of adding the timer: it changes the blast radius of a resume bug from “once per daemon start” to “once per minute”. Interlocks 1 and 2 are what buy that back, and they must exist before the cadence is tightened.


PROPOSED: the existing scoped-config cascade, no new table, no migration.

rebalance.mode string off|observe|act default observe
rebalance.never_move bool false
rebalance.max_in_flight int 1
rebalance.max_in_flight_node int 1
rebalance.max_in_flight_tenant int 1
rebalance.commit_high float 0.90
rebalance.commit_low float 0.75
rebalance.observe_high float 0.90
rebalance.window dur 30m
rebalance.min_improvement float 0.10
rebalance.stability_evals int 3
rebalance.cooldown_deployment dur 168h
rebalance.cooldown_node dur 1h
rebalance.settle dur 30m
rebalance.disk_headroom float 2.0
crons.rebalance_evaluate spec "every 5 minutes"
crons.rebalance_actuate spec "every 15 minutes"
crons.move_journal_resume spec "every 5 minutes" -> "every 1 minute"

Every value resolves through the ordinary five-level walk (global, tenant, environment, stack, deployment). There is deliberately no node scope, per the standing ruling that node is an infrastructure axis rather than a config scope, and the metric does not need one, because a node’s thresholds are fleet policy, not per-node policy.

Rejected: a rebalance_policy ent table. Cost: a schema migration for a feature that ships off by default. The migration doctrine and the v0.0.26 brick both say no, and the cascade already carries comparable knobs (deploy.reservation_ratio) including per-key provenance reporting, which the decision record needs anyway.


Section titled “8. Recommended rollout: six stages, each independently valuable and abortable”

Stage 0: the reconciler timer. §6, alone, with no rebalancer anywhere. Value: closes a real gap today. Abort: delete one registration. Exit test: a proving run in which a SIGKILL mid-move is recovered by the timer rather than by a restart, and a week of passes with zero spurious reverts.

Stage 1: inputs, honestly. Fix the four defects in SchedulingSourceAdapter (§1.2), give it a read path that is not OTLP, record per-node free disk durably, and ship shc node capacity showing per node: capacity, committed reservation, sustained utilization, free disk, and the freshness plus DERIVED|MEASURED flag on every figure. Value: an operator can answer “is this fleet balanced” by hand for the first time. Abort: it is a read-only verb. Note: this is the natural moment to settle the still-open docs/decisions/2026-08-04-physical-capacity-gates.md, same numbers, same seam, and stage 1 needs the answer anyway.

Stage 2: ★ the decision record and the planner, with no actuator. The evaluator cron, the policy object, the full record (§3.4), shc rebalance plan / why / history. Nothing moves, in any mode. Value: a continuously-running estate dry-run planner at zero risk, which is where the thresholds get calibrated against a real fleet instead of against the guesses in §2.3. Abort: turn off one cron. Exit test: four weeks of records on both estates, and a review that says the thresholds would have fired when an operator agrees they should have. ★ This is the stage the whole document exists for. A rebalancer whose policy was never observed before it was armed is the runaway.

Stage 3: standby declarations, with nothing consuming them. pre_standby / post_standby, x-shc-scale.standby, the standby journal and its resume cron, and shc standby enter|leave <stack> as an operator verb. Declare it on a handful of apps. Value on its own: an operator can safely quiesce a stack for maintenance, and the backup path may want the same verb. Abort: declarations are inert without a consumer. Gate to stage 4: shc rebalance plan on a real estate must report a non-zero count of candidates passing every gate including standby. If it reports zero, stage 4 has nothing to do and should not be built; see §2.6.

Stage 4: the actuator, off by default, one move at a time. Mode act, max_in_flight 1, the stop file, the cooldowns, the stability requirement. First estate exposure is rebalance.mode=act scoped to one tenant. Value: the feature. Abort: the kill switch, which by then has been in the tree since stage 2 and been exercised in anger.

Stage 5: only if measured to be needed: the low-downtime cutover. The overlapping three-state flip, where standby stops being belt-and-braces and becomes load-bearing. Do not begin until stage 4 has run on an estate for a quarter.


  1. Scope: fabric only, or compose-runtime multi-node too? Both estates are 100% compose today; the repo’s own architecture pages call the fabric the strategic multi-node path while noting it is a data plane, not a scheduler, and the two are mutually exclusive at shc init. The rebalancer as designed acts on compose rows and is runtime-agnostic below the picker, but the answer changes which estate can ever exercise it.
  2. The physical capacity gates (docs/decisions/2026-08-04-physical-capacity-gates.md) are still open, and stage 1 needs them settled. The reservation-versus-limit operand question is the same question §2.2 is answering.
  3. never_move default: opt-in or opt-out? Recommendation: opt-in for the first two estates (nothing moves unless annotated movable), opt-out as the eventual default. It is one default value, separable from all the code.
  4. Does a rebalance move count against a tenant’s usage or metering? It consumes a backup, a transfer and an outage window that the tenant did not request.
  5. Tier. Drain is EE; this is strictly more. Which tier, and is the planner (stage 2, read-only) worth exposing more widely than the actuator?
  6. Adjacent, and visible from here: should shc protect keep refusing the operator’s move, now that a separate automatic-mover guard exists? The overload is defensible today because there was no other guard; after rebalance.never_move it is two questions on one flag.

  • It cannot rebalance swarm-runtime stacks. Swarm owns placement, SHC stamps no constraint unless --node, and the brief reports that SHC writes no deploy.resources.reservations into swarm services (which I did not verify), so swarm’s scheduler would have nothing to honour anyway. Swarm rebalancing is swarm’s job.
  • It cannot move a stack whose connected set is not already co-located, and on today’s estates that is nearly everything (§2.6).
  • It cannot know an application’s readiness semantics. The health gate observes containers and healthchecks; it is explicitly not a smoke test, and the data gate is a shortfall detector rather than a correctness check. Both files say so about themselves.
  • It cannot promise the reservation it balances is real. No catalogue app declares one; every reservation is quota x 0.25, derived. The observed term in §2.3 is the only measured half of the metric.
  • It cannot act during a partition. The leader-gated evaluator stops; the per-node cap keeps a partitioned node honest; nothing converges until quorum returns.
  • It does not provide HA. The owner’s framing was “atomic is required, HA is desirable but optional”. This document delivers restraint around an atomic primitive. Nothing here makes anything highly available.
  • It cannot be calibrated from this document. Every threshold in §2.3 and §7 is a guess with a plausible story attached. Stage 2 exists because those numbers have to be earned against a real fleet before anything is armed.