Skip to content
SHC Docs

The placement policy and the scorer: what chooses a node, at install and at rebalance

Status: open design. The first-placement scorer is built, as the decision named below records. Rebalance scoring and signals S6 and S7 are not.

★ SUPERSEDED IN PART: read docs/decisions/2026-08-11-rebalancer-v1-foundations.md first

Section titled “★ SUPERSEDED IN PART: read docs/decisions/2026-08-11-rebalancer-v1-foundations.md first”

This page is the DESIGN DRAFT. The lane that built from it landed on 2026-08-11 and measured several of its claims differently. Where this page and that decision disagree, the decision is what was measured and this page is kept for its reasoning, not for its facts.

The corrections that matter here:

  • §1.4’s correction is itself now stale in one direction: the retired keys are retired, but the SHIPPED default multipliers were 0.25 / 1.0, not the owner’s 1.0 / 1.5. Both are now the owner’s numbers.
  • §2.4’s signal table is built for S1, S2, S3, S4 and S5. S6 (free disk) and S7 (live utilization) are not, and S7 is still recommended against for the reason §8.3 gives.
  • §2.5’s weights shipped verbatim (memory 40 / cpu 20 / count 20 / tenant 10 / app 10, bin-pack as the same function with the first three negated), and so did §2.6’s five determinism properties and §2.7’s abstain-and-renormalise rule.
  • ★ §2.7 has a hole this page did not name, found by mutation-testing the implementation. An abstaining candidate carries a placeholder zero into the value slice, and letting that zero into the min/max computation makes ONE unmeasured node rescale every measured node toward the top of the range: a nearly-full node at 1 GiB free and an empty one at 31 GiB go from 1000 apart to 32 apart. Abstention must exclude the candidate from the SCALE, not merely from the weighted sum.
  • §4.2’s warning about the wrong door was exactly right and is now a mutation-proven guard.
  • §4.4 (shc placement explain) is NOT built. The data is carried out of the scorer (every candidate’s score and every abstained term with its reason), but nothing renders it.
  • §4.5’s kill switch shipped as deploy.placement_scoring, default on rather than off, with off meaning do-not-score exactly as specified.

Everything else on this page stands, including the parts the implementation did not reach; they are named in §8 of the decision.

As written: PROPOSED. Nothing in this document is built, and nothing in it should be built before the four rulings in §9 are made. Three of those rulings are cheap; one (§9.1, bin-pack vs spread) is the policy question the owner has to answer himself because it trades money against blast radius and only he prices both.

Baselines. shc docs/rebalancer-policy off main c0b009ba5; ce main 6a3dcc5. Every figure below was measured against those two trees, on this box, with no estate contact. Claims I did not measure myself are marked (inherited) and name their source.

The recommendation in one line: build the scorer as a pure function over durable rows, ship it at install time only, default it to spread, and do not connect it to anything that moves data until it has placed a few hundred installs and an operator can read why it chose each one.


The rebalancer is the expensive, dangerous half of the control plane: it needs the atomic move, the standby contract, the concurrency cap, the kill switch. The scorer is none of that. It is a function from (candidate nodes, deployment being placed, current commitments) to a node id. It moves nothing, destroys nothing, and it is exercised on every install rather than on the rare rebalance.

That asymmetry is the whole argument of §c: the scorer should ship, and be proven, before a rebalancer exists to consume it. A rebalancer that inherits an unproven scorer will move a deployment somewhere wrong and the operator will never trust either component again.


1. Ground truth, and four corrections to the brief

Section titled “1. Ground truth, and four corrections to the brief”
ClaimVerified atVerdict
pickTarget is the only node-scoring code in the productshc modules/cluster/drain.go:508-533confirmed: lowest Count, tie-break by ID, prefers runtime match, never called outside drain
Install places on “whoever handled the request”ce modules/stack/service.go:1027-1043: a fresh install synthesises a descriptor with an empty NodeIDconfirmed
PlacementConstraint / NodeSatisfies are a drain-time FILTER onlyce modules/cluster/placement.go; consumed at shc modules/cluster/drain.go:456-464confirmed
Three physical capacity gates have zero non-test callersshc modules/quota/enforcement.go; docs/decisions/2026-08-04-physical-capacity-gates.mdconfirmed
No node labels anywhere durablece ent/schema/node.go has no labels field; labels come only from docker node inspect (shc daemon/wiring/cluster_drain_constraints.go:82-100)confirmed, and worse than stated; see §3.2
No per-node free-disk factget_health returns DiskFree (ce modules/metrics/rpc_utility.go:183) but has zero non-test callers and no durable columnconfirmed
Zero of the catalogue declares x-shc-volumes or deploy.placement104 app.yaml files under shc/apps; grep -rl returns 0 for bothconfirmed (the brief says 99 apps; I count 104 at this commit)

1.2 ★ Correction 1: SHC does write deploy.resources.reservations at ce HEAD

Section titled “1.2 ★ Correction 1: SHC does write deploy.resources.reservations at ce HEAD”

The brief says “SHC never writes deploy.resources.reservations, so Swarm’s scheduler has nothing to honour.” That was true; it is not true at 6a3dcc5. ApplyResourceQuotas injects both keys into the rendered artifact (ce modules/app/resource_quota.go:455-460), effectiveCPUReservation / effectiveMemoryReservation are the single writers (:488-540), and ParseComposeServices reads them straight back out (ce modules/app/services_writer.go:117,121).

This is the single most load-bearing correction in this document, because it means the operand a real scorer needs is already durable:

service.cpu_request_millicores ce ent/schema/service.go:56
service.memory_request_bytes ce ent/schema/service.go:60
service.desired_replicas ce ent/schema/service.go:52

per service, per deployment, in the replicated table, written at every install and update. The committed-reservation-per-node sum the brief calls missing is a two-join query over rows that already exist. Nothing sums them (that part of the brief is right), but the summands are there and no new probe, no new column, and no new tick is needed to compute the figure.

Not verified: whether the estates are running a daemon new enough to have written non-zero request columns. That needs estate contact, which this lane does not have. Treat “the columns are populated on the estates” as unproven and make the scorer tolerate zeros (§2.7).

1.3 ★ Correction 2: cross-node install dispatch already exists

Section titled “1.3 ★ Correction 2: cross-node install dispatch already exists”

The brief implies install-time placement needs new machinery. It does not. makeRoutingStackInstaller (ce daemon/services_stack_routing.go:100-113) already POSTs the whole install request to /internal/rpc/stack.install on target.NodeID whenever that isn’t the local node. This is the path shc install --node other takes today, and its header comment records the bug it was written to fix.

So the entire execution path for “install this somewhere else” is built and in production use. A scorer’s only job is to supply the value that --node supplies. That is the strongest single argument for shipping install-time placement first, and I did not expect it going in.

1.4 ★ Correction 3: the quota model in the brief names retired keys

Section titled “1.4 ★ Correction 3: the quota model in the brief names retired keys”

The brief states reservation = quota × reservation_multiplier (1.0) and limit = quota × limit_multiplier (1.5). That is the owner’s stated architecture (inherited, memory quota_derivation_model) and its shape is intact at HEAD: quota is the input, both figures are derived, both resolve through the ordinary cascade, there is no node scope. The key names and defaults are not:

limit = quota × deploy.limit_ratio default 1.0 ce modules/app/resource_quota.go:174
reservation = quota × deploy.reservation_ratio default 0.25 ce modules/app/resource_reservations.go:102

quota.limit_multiplier and quota.reservation_multiplier were removed as a declared breaking change (shc CHANGELOG.md:8685) on the grounds that they “never did anything.” website/src/content/docs/architecture/modules/quota.md:90-92 still documents the removed pair and is stale; worth a one-line fix by whoever touches that page next, but out of scope here.

Why this matters to a scorer: the default overcommit factor is 4×, not 1.5×. A scorer that thinks reservations are 100% of quota will pack four times too conservatively.

1.5 ★ Correction 4: pinned_node_id is set on every compose deployment

Section titled “1.5 ★ Correction 4: pinned_node_id is set on every compose deployment”

PlacementIntent returns the resolved node for any host-bound runtime (ce modules/app/placement_intent.go:60-68), and the schema comment is explicit that this is intended: “A host-bound runtime has exactly one place its bytes can be, so the node resolved at install IS the placement and lands here” (ce ent/schema/deployment.go:102-105).

Both estates are 100% compose (inherited, brief + swarm-contention-map.md). Therefore pinned_node_id is non-NULL for every deployment on both estates, and a rebalancer that reads it as “the operator chose this node, do not move it” would refuse to move 100% of the fleet.

The distinction does exist in flight: StackInstallRequest.StatedNodeID (ce modules/stack/types.go:212) captures what the request named before the back-fill erases it, but it is never persisted. At rest, the schema cannot answer “did a human choose this node?” for a compose deployment. §3.4 and §10 treat that as a prerequisite, not a detail.


// shc modules/cluster/drain.go:512
func pickTarget(candidates []*candidateNode, runtime string) *candidateNode

Sort by Count ascending, tie-break by ID ascending, then return the first candidate whose Runtime matches, else the first overall. Count is the number of rows from DeploymentLister.ListByNode whose status is running, starting, or empty (drain.go:470-506, isRunningStatus at :548). The caller bumps picked.Count++ after each successful migration “so the next pick spreads” (drain.go:255).

Three properties are worth keeping and one is a bug in waiting:

  • Keep: deterministic tie-break by ID.
  • Keep: the running tally mutates within a batch, so a plan of N placements spreads rather than stacking N deployments on the same node.
  • Keep: it is a pure function over a slice, trivially unit-testable.
  • Fix: isRunningStatus("") returns true, so a row with an unset status counts as running. That is the right conservative choice for a drain (better to over-count than to leave something behind) and the wrong one for a scorer (it inflates a node’s apparent load). A scorer needs its own predicate.

2.2 What count-only scoring gets wrong: measured

Section titled “2.2 What count-only scoring gets wrong: measured”

The catalogue’s authored memory limits, summed per app over services.*.deploy.resources.limits.memory:

Σ authored memory limits
security/greenbone32.62 GiB
devops/gitlab16.25 GiB
ai/dify15.56 GiB
standalone/ollama12.00 GiB
……
email/mailbox-reconciler0.125 GiB

94 of 104 apps declare a memory limit. Largest / smallest = 261×. Median 2.06 GiB, mean 3.33 GiB. The gitlab figure reproduces docs/decisions/2026-08-04-physical-capacity-gates.md §3.3 exactly (16.25 GiB), which is the cross-check that my parse is reading the same thing the earlier measurement read.

So the brief’s framing (“one deployment is gitlab and another is a 40MB link shortener”) is real and its magnitude is two and a half orders of magnitude. tools/kutt, the actual link shortener in this catalogue, declares 0.125 GiB.

But the honest counter-argument, which I ran into by testing it: at large N, count-only scoring is fine. Placing all 82 non-fixture apps across 3 nodes in alphabetical order, least-count produced a 1.16× memory imbalance and least-committed-memory produced 1.015×. The law of large numbers does most of the work.

The defect lives at small N, which is the size SHC actually runs. 3000 random draws, 3 nodes, reservation operand (limits × 0.25), imbalance measured as busiest-node-commitment ÷ quietest:

deploymentspickermedianp95worst
9least-COUNT2.80×9.14×n/a
9least-RESERVATION1.63×4.00×n/a
15least-COUNT2.28×5.39×n/a
15least-RESERVATION1.33×2.58×n/a

At the fleet size both estates run, count-only scoring leaves the busiest node carrying roughly 2.8× the committed memory of the quietest, and one draw in twenty leaves it carrying 9×. Weight-awareness roughly halves the median and more than halves the tail.

And a finding that cuts against building an admission gate: with the reservation operand and a cpx32-class node (15.6 GiB, inherited from the capacity-gates doc), the busiest node exceeded physical memory in 0.0% of 9-deployment draws and 0.3% of 15-deployment draws. With the limit operand it exceeded in 49% and 75% respectively, which is exactly the §3.4 argument of the capacity-gates decision, reproduced independently here.

So: weight-aware scoring is worth building for balance and burst headroom, not for admission. At current fleet sizes a physical-fit gate would essentially never fire. That is an argument for scoring and against wiring the three gates at refuse; it does not change the recommendation in that decision doc, it supports it.

2.3 The replacement: filter, score, commit

Section titled “2.3 The replacement: filter, score, commit”

Three phases, in this order, and the order is the design:

1. FILTER hard constraints. A node that fails any is not a candidate.
Produces a candidate set, possibly empty.
2. SCORE every surviving candidate gets an integer score. Highest wins.
3. COMMIT the winner's tally is mutated in the caller's in-memory ledger
before the next placement in the same batch is scored.

Phase 3 is pickTarget’s picked.Count++ generalised: after a placement, the winner’s committed CPU and memory rise by this deployment’s reservation and its deployment count rises by one. Without it, a batch of N placements all see the same starting state and all pick the same node. This is the property the drain picker already has and it must not be lost in the rewrite.

An empty candidate set after phase 1 is not a scoring failure and must not fall back to “score everything anyway.” It is a refusal with a message naming which constraint eliminated the last candidate, the shape drain.go:232-238 already uses to distinguish “no active peer” from “no node satisfies the constraints.”

Every signal below reads a durable, replicated row. Nothing reads a probe, a cache, or a live container. That is deliberate: a scorer that depends on StatsCache (per-process, 15s, empty before the first tick, inherited, capacity-gates §4) is a scorer whose answer depends on which daemon you asked and how recently it booted.

#SignalSourceAvailable today?
S1Free memory reservation = node.memory_bytes − Σ committedce ent/schema/node.go memory_bytes; service.memory_request_bytes × desired_replicas joined through deployment.recorded_node_idyes, no new column
S2Free CPU reservation = node.cpu_cores × 1000 − Σ committedsame, cpu_request_millicoresyes
S3Deployment countdeployment_node occupancy set, anchored = trueyes, and this is a better count than drain’s, see below
S4Tenant spread: how many of this tenant’s deployments the node already holdsjoin deployment.tenant_nameyes
S5App spread: how many instances of this app the node already holdsjoin deployment.app_nameyes
S6Free disk✗ nothing durableno: §10 P3
S7Live utilization✗ get_health exists, uncalled, per-node, not storedno, and I recommend against it, §8.3

On S3 and why it should read deployment_node, not ListByNode. The occupancy set is one row per (deployment, node) whose bytes or tasks are on that node, with anchored and last_observed_at as two independent facts (ce ent/schema/deployment_node.go). Its own header records that recorded_node_id was unsound in both directions for exactly this class of question. The drain picker’s count predates it. A scorer should read the set and count anchored rows; that is “how much has SHC put on this box,” which is the question.

On S4/S5 and why they are scores, not constraints. “Don’t put both replicas of the same app on one node” is a preference at three nodes and a guarantee nobody can honour at one node. Expressing it as a score degrades gracefully: with room, it spreads; without room, it places anyway and the operator can see the penalty in the explain output. Expressing it as a hard constraint means a single-node estate cannot install a second instance of anything. §3.3 covers the case where the operator genuinely wants the hard version.

Scores are integers in per-mille, not floats. Float comparison makes tie detection architecture-dependent and makes the audit trail unreproducible; integers make score(a) == score(b) exact and the tie-break deterministic.

score(node) = w_mem × headroom_mem_permille(node)
+ w_cpu × headroom_cpu_permille(node)
+ w_cnt × count_permille(node)
+ w_tnt × tenant_spread_permille(node)
+ w_app × app_spread_permille(node)

where each term is normalised to [0, 1000] across the candidate set, so no term can dominate merely by having larger units. Proposed defaults:

weightspread (default)bin-packrationale
w_mem40−40memory is the axis that actually kills a box
w_cpu20−20CPU degrades, memory OOMs
w_cnt20−20keeps behaviour recognisable to anyone who has read pickTarget
w_tnt1010tenant blast radius is a real product concern; not negated under bin-pack
w_app1010ditto

Bin-pack is the same function with the first three weights negated; it prefers the node with the least headroom that still satisfies the filters. It is not a second code path, and that is the point: one scorer, one explain output, one test surface.

These weights are a starting point and I have not tuned them. The honest statement is that any weighting in this family beats count-only at the estate sizes measured in §2.2, and that fine-tuning them without production placement data would be fitting noise. The weights should be constants first and become config only if an operator produces a case they get wrong (§8.5).

The scorer must be reproducible: same inputs, same output, on any node, on any architecture, in any Go version.

  1. Integer arithmetic only. No floats in the score. Ratios computed as (numerator * 1000) / denominator with integer division.
  2. Sort on (score DESC, node_id ASC) with sort.SliceStable, mirroring pickTarget’s existing tie-break so the behaviour is familiar.
  3. Candidate order is normalised before scoring: sort by node id up front, so a registry that returns rows in a different order cannot change the answer even transiently.
  4. The batch tally is part of the input. Placing a plan of N is fold(scorer, initial_tally, deployments), and the deployments must be iterated in a stated order (by deployment id) so a replan produces the same plan.
  5. No clock, no randomness, no map iteration in the scoring path.

Property 4 gives the audit trail for free: replaying the fold from the recorded initial tally reproduces every decision, which is what makes shc placement explain (§4.4) honest rather than a re-derivation that might disagree with what actually happened.

shc modules/quota/enforcement.go already states the rule this file must obey, and it was learned the hard way: “an unknown must never collapse into a zero. A zero is a measurement; an unknown is the absence of one. A gate that confuses them does not degrade to unchecked, it degrades to refuse-everything.”

For a scorer the failure mode is inverted but just as bad. A node whose memory_bytes is NULL, treated as zero, scores as having no headroom and never receives a deployment; it silently drops out of the fleet. Treated as infinite, it scores best on every axis and receives everything.

The rule:

  • A node with an unknown axis abstains that axis: its term is omitted and the remaining weights are renormalised over the terms it can answer.
  • A node with no answerable axis falls back to w_cnt alone, which is exactly today’s pickTarget, i.e. the degraded behaviour is the current behaviour, not a new one.
  • Σ committed of zero is a legitimate measurement (an empty node) and must not be conflated with “the request columns were never written.” Distinguish them by asking whether the node holds any deployment_node rows at all: zero commitments across zero deployments is empty; zero commitments across nine deployments is unmeasured, and the memory axis abstains.

That last bullet is the one that protects against §1.2’s unproven half: an estate whose daemon predates the reservation writer degrades to count-only scoring rather than to a wrong answer.


3.1 Extend PlacementConstraint, do not invent a second vocabulary

Section titled “3.1 Extend PlacementConstraint, do not invent a second vocabulary”

ce modules/cluster/placement.go already parses docker’s constraint syntax into typed values, matches them against a NodeAttrs, skips unrecognised keys rather than rejecting, and exposes HasHardNodePin. It is 130 lines, it has a test file, and it is already the filter in front of the drain picker.

Use it verbatim as phase 1 of the scorer. Concretely: filterCandidates moves out of drain.go into a shared placement package, candidateNode grows the two commitment fields, and drain calls the shared scorer with the bin-pack-off weights. Drain’s behaviour is preserved exactly if w_cnt is the only non-zero weight, which is the migration test to write first.

Three additions to the vocabulary, all in docker’s own dialect so an operator who knows swarm needs no new syntax:

additionsyntaxwhy
operator constraints on a deploymentshc install foo --constraint 'node.labels.disk==ssd'today constraints exist only on the rendered artifact, so a compose deployment cannot express one at all
soft preference--prefer 'node.labels.zone==fsn1'scored, not filtered, worth a large positive term, never a refusal
anti-affinitysee §3.3not expressible in this grammar at all

3.2 ★ The label gap is worse than “no labels column”

Section titled “3.2 ★ The label gap is worse than “no labels column””

daemonNodeAttrs populates Labels from docker node inspect (shc daemon/wiring/cluster_drain_constraints.go:82-100), which requires a swarm_node_id. And daemonDeploymentConstraints returns nil immediately for any non-swarm deployment (:41-43).

On a 100%-compose estate, both halves of the constraint pathway return nothing. The parse and match code is live and unit-tested; the data is structurally unavailable. So the existing constraint machinery is not “wired but unused”; on the estates that exist, it is wired to a source that cannot answer.

This is a prerequisite, not a caveat: node labels need a durable home (node.labels as a JSON column, or, better, given the config module already exists, a node-keyed config row) before any label-bearing constraint means anything outside swarm. Until then, --constraint node.hostname==X and node.id==Y work (those come off the registry row) and node.labels.* silently matches nothing.

★ And the silence is the danger. SatisfiesConstraint returns true for an unrecognised key by design: “skip keys we can’t evaluate rather than reject the node.” But node.labels.gpu==true is a recognised key; it resolves to the empty string for a node with no labels (placement.go:86-89) and therefore fails the == test. So on a compose estate a GPU constraint eliminates every candidate and the operator gets “no node satisfies the placement constraints” with no hint that the label source was never populated. The scorer must detect “this constraint references a label and no node in the fleet has any labels at all” and say that instead.

3.3 Anti-affinity is not expressible in this grammar, and pretending otherwise is the trap

Section titled “3.3 Anti-affinity is not expressible in this grammar, and pretending otherwise is the trap”

NodeSatisfies(constraints, node) evaluates one node against literal values. Anti-affinity (“these two deployments must not share a node”) is a relation between a candidate and the rest of the placement, which no (key, op, value) triple can express. Bolting it in by inventing node.deployments!=foo would create a constraint whose truth depends on state the matcher does not receive, and the matcher’s purity is why it is testable.

Two honest options:

Option A (recommended): derive constraints per placement. Before phase 1, the caller resolves the deployment’s anti-affinity group into a concrete exclusion list against the current tally, and emits ordinary node.id!=<id> constraints. NodeSatisfies stays pure; the relation is resolved by the caller that already holds the tally. Cost: the derivation must re-run for every placement in a batch (cheap: it is a map lookup), and the audit trail must record the derived constraints, not just the authored ones, or an operator reading the explain output sees an exclusion with no stated cause.

Option B: a second filter phase. filterByRelations(candidates, tally, deployment) runs after NodeSatisfies. More direct, but it splits “why was this node eliminated” across two mechanisms with two message formats, and the first thing an operator asks is exactly that question.

Option A. It costs one derivation step and keeps one refusal vocabulary.

Where the group comes from: a deployment-scoped config key placement.anti_affinity_group (a string; deployments sharing a value repel). Soft by default (a large negative score term) with placement.anti_affinity: required promoting it to a hard exclusion. Soft by default is not timidity: at one node, hard anti-affinity makes the second member uninstallable, and one node is a supported SHC topology.

3.4 never-move: copy deployment.protected exactly

Section titled “3.4 never-move: copy deployment.protected exactly”

The owner named a per-deployment never-move annotation. The shape to copy is already in the tree and its header is a written record of the two ways this goes wrong (ce modules/app/protected.go):

  • a deployment-scoped config row through the same 5-level cascade: no ent migration, and deployment-scoped config is state, so it rides backup/restore for free;
  • mirrored onto the manifest sidecar, because the config resolver is nil in the shc CLI process and a guard that only exists daemon-side is a guard the daemonless path walks straight through;
  • fail closed: unreadable is UNKNOWN and UNKNOWN reads as protected;
  • the write reports which halves landed, because “armed” that armed nothing readable is the specific lie that file exists to stop telling.

So: deployment.never_move, bool, same cascade, same mirror, same fail-closed read, same two-store write report. Fail-closed for never-move means an unreadable flag reads as never_move = true; the rebalancer declines to move something it cannot ask about, which costs availability-of-optimisation and never costs data.

3.5 ★ protect already refuses move, and the rebalancer inherits that

Section titled “3.5 ★ protect already refuses move, and the rebalancer inherits that”

ce modules/clone/protection.go refuses shc move outright on a protected deployment, up front, before any phase runs, because the protection flag cannot reliably follow a move across a rename/tenant/environment change, and “a silently disarmed destroy guard is worse than a refusal.”

Consequences a rebalancer must be built knowing:

  1. Every protected deployment is already never-move. The rebalancer gets that for free and must not fight it.
  2. protected and never_move are different questions and must stay two flags. protected means “do not destroy”; never_move means “do not relocate automatically.” An operator may well want a deployment that a human can move but a scheduler may not; that is the entire point of the annotation.
  3. The rebalancer’s immobility inventory is protected ∪ never_move ∪ hard-pinned ∪ (RWO volume whose location the candidate cannot reach), and it should be reportable before anyone builds a mover, because if it covers most of the fleet the mover is not worth building. The brief’s inherited figure (31 of 33 deployments immobile pre---closure) is exactly this inventory, taken once.

The scorer must never write a swarm placement constraint for a node it merely scored. ce ent/schema/deployment.go:104-107 states the rule: “under swarm ONLY an explicit --node lands here… SHC must never add a placement constraint the operator did not ask for.” A scored placement is SHC’s opinion, not the operator’s request. §4.3 is where that bites.


4.1 Install-time placement should ship before any rebalancer. I agree with the brief, for four reasons, one of which is not the obvious one.

Section titled “4.1 Install-time placement should ship before any rebalancer. I agree with the brief, for four reasons, one of which is not the obvious one.”
  1. It moves no data. The worst outcome of a bad first placement is a deployment on a node the operator would not have picked. The worst outcome of a bad rebalance is a stopped app, or worse. These are not comparable risks.
  2. It exercises the entire scoring stack under load. Every install runs it. A rebalancer runs it rarely, which is the worst possible way to gain confidence in a decision function.
  3. The transport already exists and is in production use (§1.3). This is the reason I did not expect: install-time placement is not a new capability, it is supplying an argument to a capability that already ships. shc install --node other proves the whole path today.
  4. It produces the corpus the rebalancer needs. After a few hundred scored installs there is a record of what the scorer chose and what happened next. Without that, the first rebalance is the first time anyone finds out whether the weights are sane.

The argument against, stated fairly: install-time placement delivers no benefit on a single-node estate, and a scorer that only ever has one candidate is untested by its own production use. That is real. The mitigation is that the scorer is a pure function and its tests are cheap; the production exercise is a bonus, not the proof.

One more argument against, which I take more seriously: install-time placement changes where things land, and “where things land” is currently predictable (“wherever I ran the command”). An operator who has built habits around that predictability will experience the scorer as the machine putting my app somewhere weird. That is a documentation-and-explain-output problem, and §4.4 is the answer, but it is a real cost and it is why shc placement explain is in phase 1 rather than phase 3.

The right door. ce modules/stack/service.go:409-436, Service.Install. After resolveInstallTarget and after StatedNodeID is captured, before CheckRuntimeMatchesNode:

if req.StatedNodeID == "" && target.NodeID == "" {
target.NodeID = <scorer's choice> // routing only
}

Everything downstream then behaves as it does for --node today: makeRoutingStackInstaller dispatches, StampPlacement stamps, the anchor and the pin are written by the intent functions.

★ The wrong door, and it is one line away. Setting req.NodeID instead of target.NodeID (or letting the existing back-fill at :425-427 do it for you) writes the scored node into StackRepoRow.PinnedNode (modules/stack/service.go:865), which PlacementIntent turns into deployment.pinned_node_id. On compose that is harmless-looking (it is what already happens). On swarm it manufactures a hard pin the operator never asked for, which is the exact violation deployment.go:104-107 forbids and which StatedNodeID was introduced to make expressible.

The fold at :425-427 currently has two cases (--node given / not given) and a scorer needs a third (derived). Either add req.ScoredNodeID alongside StatedNodeID, or make the back-fill three-way. Do not reuse req.NodeID. That is the highest-value sentence in this section; the same class of fold destroyed the same distinction once already, and the comments at :417-421 and :857-864 are the record of it.

4.3 The swarm asymmetry: install-time placement is a compose feature

Section titled “4.3 The swarm asymmetry: install-time placement is a compose feature”

On swarm, SHC should not choose a node. Swarm’s scheduler exists, and at ce HEAD SHC now feeds it the reservations it needs to make an informed choice (§1.2). Overriding it means SHC and swarm both scheduling, and the doctrine already says SHC loses that argument (“swarm owns placement; SHC owns intent”, shc modules/cluster/drain.go:135-141).

So: the scorer runs when SchedulerOwnedRuntime(runtime) is false. On swarm it does nothing, and the honest framing is that SHC’s contribution to swarm placement is the reservation figures, not a node choice.

Given both estates are 100% compose (inherited) and no swarm-runtime install has ever been verified end to end (inherited), this is not a limitation. It is a statement that install-time placement lands exactly where the fleet is.

4.4 shc placement explain is phase 1, not a nicety

Section titled “4.4 shc placement explain is phase 1, not a nicety”

The owner named “an audit trail answering why did it move this” as a prerequisite. The same requirement applies to why did it put this here, and it is much cheaper to build against the pure function than to reconstruct later.

$ shc placement explain gitlab
target: node-b (score 612)
candidate mem-free cpu-free deps tenant app score note
node-a 2.1 GiB 3200m 7 4 0 388
node-b 9.4 GiB 6400m 3 1 0 612 <- chosen
node-c unknown unknown 4 2 0 411 memory axis
abstained
(node.memory_bytes
is NULL)
node-d excluded:
node.labels.ssd==true
(no node in this
fleet carries any
labels; see
`shc node label`)

Two requirements that are easy to get wrong:

  • The explain output must be a replay of the recorded decision, not a re-derivation. A re-derivation run a day later reads a different tally and disagrees with what happened, which destroys exactly the trust the feature exists to build. Persist the score vector and the candidate set with the decision; §2.6’s determinism is what makes the persisted form small (the inputs, not the outputs).
  • Exclusions must name their cause, including derived anti-affinity exclusions (§3.3) and abstained axes (§2.7). A candidate that vanished with no stated reason is how operators conclude the scheduler is haunted.

placement.scoring: off | on resolved through the ordinary cascade, and off means exactly today’s behaviour: the request runs on the daemon that received it. Not “score but ignore”: do not score. The distinction matters because a kill switch that still runs the code is a kill switch that can still be crashed by the code.

Because the seam is one assignment inside one if, this is genuinely cheap now and genuinely expensive later, which is the owner’s own instinct applied to the scorer rather than to the mover.


Four reasons, in ascending order of force:

  1. SHC has no HA. The owner’s own framing: “HA is desirable but optional” (inherited, atomic_move_design). A packed node’s failure takes down everything on it and nothing fails over. Density’s payoff is fewer nodes; its cost is a bigger crater, and SHC has no parachute.
  2. Recovery is expensive here. Moving a deployment off a failed or overloaded node is a backup-and-restore for every deployment on both estates (zero block volumes, zero storage providers, inherited). Spread is cheap insurance against an operation that is measured in minutes per stack.
  3. Bin-packing needs an admission gate to be safe, and there isn’t one. Packing deliberately drives nodes toward their limit; the three gates that would catch overshoot have zero callers and their own decision doc recommends warn, not refuse. Packing without admission is packing without a floor.
  4. The customer is a self-hoster. The population that runs SHC has already bought the hardware. Density optimises a cost they have mostly already paid, while blast radius is a cost they pay every time.

Spread. Default. Not configurable off in phase 1.

placement.strategy: spread | pack, one key, resolved through the ordinary 5-level cascade: global → tenant → environment → stack → deployment (shc docs/reference/quota-model-state.md:110, docs/reference/cli/config.md:72).

There is no node level in that cascade and this proposal does not ask for one. The owner’s ruling is structural and correct: node is an infrastructure axis, not a config scope, and the cascade is the tenant/environment/stack tree (inherited, memory reservation_ratio_ruling).

And the ruling composes with this key more neatly than it does with the reservation ratio, which is worth stating. placement.strategy is resolved at the scope of the deployment being placed, and it describes how SHC should treat the fleet on that deployment’s behalf. “This tenant’s workloads should spread; that tenant’s dev environment can pack” is coherent, resolvable, and needs no node scope to express. A per-node strategy would be incoherent anyway: strategy is a property of the decision, and one decision ranges over all nodes, so a per-node value would have no single reader.

The mechanism is §2.5’s negated weights: one function, one explain output.

  • No per-node knob, per the ruling.
  • No placement.strategy in phase 1. Ship spread as a constant. A knob shipped alongside its first implementation is a knob nobody has evidence to set, and it doubles the behaviour surface before anyone has seen the single behaviour work.
  • No third strategy (“balanced”, “cost-aware”). Two opposites and a weight vector already cover the space; a third name would be a preset, and presets are how a policy surface becomes unexplainable.

Phase 1: the scorer, install-time, compose only. Pure function; signals S1–S5; spread weights as constants; PlacementConstraint reused as phase 1 of the filter; placement.scoring: off|on kill switch defaulting off for one release, then on; shc placement explain; drain migrated onto the shared scorer with count-only weights so its behaviour is provably unchanged. No data moves. No new schema.

Phase 2: the durable inputs. node.labels (or node-keyed config rows) so label constraints stop silently matching nothing; a durable free-disk fact (S6); deployment.never_move in the protected shape; persistence of placement provenance so pinned_node_id stops being a lie about operator intent on compose (§1.5).

Phase 3: constraints and anti-affinity. --constraint / --prefer on install; placement.anti_affinity_group soft by default; derived exclusions in the explain output.

Phase 4: the knob. placement.strategy: spread|pack, only if an operator has produced a case that spread gets wrong.

Phase 5 (and only then): the rebalancer consumes the scorer. By this point the scorer has placed hundreds of deployments, its explain output is something an operator reads without surprise, and the immobility inventory (§3.5) is a known number rather than a guess.

What I would not do: ship a rebalancer that carries its own scoring logic “for now.” Two scorers is how the fleet ends up with two answers to where a deployment belongs, and the drain picker is already the first of them.


Stated plainly, because it will.

  1. It scores reservations, and reservations are a derivation, not a measurement. The figure is quota × 0.25, where quota is usually the app author’s authored limit. An app that over-declares gets scored as heavy forever; an app that under-declares gets scored as light and then eats the node. The scorer will be confidently wrong about any app whose declaration is wrong, and nothing in the design detects that: the loop from “what it actually used” back to “what we scored it as” does not exist and this proposal does not build it.
  2. It will pick badly at N=2. With two candidates and one large deployment, any headroom-weighted scorer puts the big thing on the emptier node and the next big thing on the other, which is right; but a third big thing has no good answer and the scorer will pick one and look arbitrary. Small fleets have small numbers of good answers.
  3. It does not know about disk at all until S6 exists, so it will happily place a 200 GiB app on the node with the most RAM and the least free disk. Given the catalogue’s largest apps are the ones with the largest data directories, this is not a hypothetical.
  4. It does not know about network topology, location, or egress cost. node.location exists on the row and the scorer ignores it. A fleet spanning two Hetzner locations will get placements that are memory-optimal and latency-hostile.
  5. It cannot see co-tenancy through a plug. Two deployments joined by a plug (an app and its -db companion) want to be together, and the scorer’s spread terms actively push them apart. The --closure work knows how to compute that relation; the scorer does not consume it. This is the most likely source of a genuinely bad first placement and it should be the first thing measured after phase 1 ships.
  6. The explain output will be read as a promise. An operator who sees “score 612 vs 411” will assume the difference is meaningful. It is a weighted sum of normalised terms with hand-picked constants; the ordering is meaningful and the magnitude is not, and no amount of documentation fully fixes that.
  7. It changes where installs land, which will be experienced as a regression by anyone whose runbook says “ssh to the node, run the install.” That is a real cost paid by real operators and the kill switch is why it is survivable.

8. Rejected alternatives, with their costs

Section titled “8. Rejected alternatives, with their costs”

8.1 Keep pickTarget and just add a memory term. Cheapest possible change, and it would capture most of the §2.2 benefit. Rejected because it leaves the scorer inside drain.go, unexported and uncallable from install, which is the actual defect. The scoring maths is the easy part; the location is the change. Cost of rejecting: a few hundred more lines than the minimum.

8.2 Wire the three physical capacity gates and call that placement. They answer “does this fit an empty box,” never “is there room now,” and §2.2 measured that at estate scale they would fire in ~0% of cases. Wiring them is not placement and would produce a feature that appears to work because it never refuses. Cost of rejecting: the swarm preflight lane still has nothing to refuse an unschedulable stack with, unchanged by this document, and still governed by docs/decisions/2026-08-04-physical-capacity-gates.md.

8.3 Score on live utilization (get_health / StatsCache). Rejected on three grounds: the cache is per-process and 15 seconds and empty before the first tick (inherited), so the answer depends on which daemon you asked; live utilization is not reproducible, which kills the audit trail; and scoring on current load creates a feedback loop where a node that gets quiet attracts placements until it is loud. Reservations are stable, replicated, and already written. Cost of rejecting: the scorer cannot tell a node that is committed heavy but actually idle from one that is genuinely busy; see §7.1.

8.4 Let swarm do it (adopt swarm as the multi-node path). Swarm’s scheduler is real, and at ce HEAD SHC finally feeds it reservations. Rejected because swarm is a second-class runtime, the fabric is the strategic path, the two are mutually exclusive at shc init with no in-place flip (inherited, swarm-contention-map.md:467, swarm-quorum-and-health.md), 37 of 73 catalogue apps could not docker stack deploy at all when last measured (inherited, M27), and both estates are 100% compose. Cost of rejecting: SHC writes a scheduler it could in principle have borrowed.

8.5 Make the weights config from day one. Rejected: five knobs no operator has evidence to set, each of which must then be honoured forever. Constants first; a knob when someone produces a case. Cost of rejecting: an operator with an unusual fleet has no recourse but the kill switch until phase 4.

8.6 Score at every install including swarm. Rejected per §4.3: it manufactures placement constraints the operator did not ask for and puts SHC in contention with a scheduler that owns the decision. Cost of rejecting: swarm installs keep whatever swarm decides, which is the correct outcome anyway.


9.1 Bin-pack or spread as the product default? §5 recommends spread and gives four reasons, but this trades hardware spend against blast radius and only the owner prices both. If the answer is pack, §2.5’s negated weights are the whole implementation, but §5.1’s third reason then becomes load-bearing and the capacity gates need a posture before packing ships.

9.2 May a scored placement be recorded as a pin? Today pinned_node_id is set for every compose deployment (§1.5), so the column already cannot express operator intent on the runtime both estates use. Options: (i) leave it, and add a separate provenance signal a rebalancer reads; (ii) change the compose semantics so only an explicit --node pins, a behaviour change with a migration; (iii) accept that a rebalancer must treat compose pins as advisory. I recommend (i) (it is additive and it does not touch a column three subsystems already read), but it is a schema decision and belongs to the owner.

9.3 Is anti-affinity soft or hard by default? §3.3 recommends soft because one node is a supported topology and hard anti-affinity makes the second member uninstallable there. An owner who considers co-location of two members a correctness violation rather than a preference would rule the other way.

9.4 Does placement.scoring default on or off on first release? §6 proposes off for one release, then on. The argument for defaulting on immediately is that a feature nobody enables gets no production exercise, which is the whole point of phase 1. The argument for off is §7.7. This is a release-posture call.


P1. The shared placement package. Move candidateNode, candidateTargets, filterCandidatesByConstraints and pickTarget out of shc modules/cluster/drain.go into a package both drain and install can import. Blocks everything. First test: drain’s existing behaviour is byte-identical with count-only weights.

P2. The commitment query. Σ service.cpu_request_millicores × desired_replicas and Σ service.memory_request_bytes × desired_replicas, grouped by node via deployment_node (anchored rows). No new columns (§1.2). Blocks S1/S2, i.e. everything that makes the scorer better than pickTarget.

P3. The install seam. A third case in the req.NodeID / target.NodeID fold at ce modules/stack/service.go:425-427 that can express “derived” without writing a pin (§4.2). Blocks install-time placement. This is the one place a mistake is expensive, and the mistake is one character of variable name.

P4. shc placement explain + decision persistence. Blocks operator trust, which blocks turning the kill switch on.

P5. Node labels with a durable home. Blocks every node.labels.* constraint outside swarm, which is to say, blocks them on both estates (§3.2).

P6. deployment.never_move in the protected shape. Blocks any rebalancer. Independent of P1–P5; can be built in parallel by whoever is doing the mover.

P7. A free-disk fact with a durable home. Blocks S6. Lowest priority of the seven and the easiest to defer, but see §7.3 for what deferring costs.


All figures in §2.2 come from parsing shc/apps/**/compose*.y*ml with yaml.safe_load after stripping jinja ({{…}} → 0, {%…%} → “), summing services.*.deploy.resources.limits.memory per app directory, excluding paths containing /tests/ and fixtures/. The reservation operand is that figure × 0.25 (deploy.reservation_ratio, §1.4). The simulation places k apps sampled without replacement onto 3 nodes, picking min(key, node_index) per app, seed 20260811, 3000 draws. Node size 15.6 GiB is the cpx32 MemTotal from docs/decisions/2026-08-04-physical-capacity-gates.md §3.3 (inherited; I did not measure a node).

The validation that the parse reads the right thing: it independently reproduces that document’s gitlab figure of 16.25 GiB.

What the simulation is not. It assumes apps are placed one at a time in random order onto an empty fleet, which is how a fresh estate fills up and is not how an existing one evolves. It uses authored limits as a proxy for real consumption, which §7.1 already flags as the design’s central weakness. Read the ratios between pickers as the finding; the absolute imbalance numbers are illustrative.


  • shc modules/cluster/drain.go: pickTarget, candidateTargets, the count bump
  • ce modules/cluster/placement.go: PlacementConstraint, NodeSatisfies, HasHardNodePin
  • ce modules/stack/service.go:409-436: the install seam, and the fold to be careful with
  • ce daemon/services_stack_routing.go: the cross-node install dispatch that already exists
  • ce modules/app/protected.go: the two-store, fail-closed annotation shape to copy
  • ce modules/clone/protection.go: why protect already refuses move
  • ce ent/schema/deployment_node.go: the occupancy set the count should read
  • ce ent/schema/deployment.go:96-123: recorded_node_id vs pinned_node_id
  • ce modules/app/resource_quota.go: where reservations are derived and written
  • shc modules/quota/enforcement.go: the unknown-is-not-zero rule
  • shc docs/decisions/2026-08-04-physical-capacity-gates.md: the three inert gates, still open
  • shc docs/architecture/cluster/swarm-contention-map.md: M27, M33, and the second-class-runtime framing
  • shc docs/reference/quota-model-state.md: the derivation ladder and the five scope tiers