Skip to content
SHC Docs

The three physical capacity gates: wire, warn, or delete

Superseded by New-line runtime. The text below is the record as of its date.

Status: open owner decision. Cost of delay: cosmetic today (the gates are inert, so nothing is broken by waiting); blocks-work for the swarm preflight lane, which has nothing else to refuse an unschedulable stack with.

Baselines. shc origin/main 6a559b526, ce origin/main 8ab2081. Every figure below was measured against those two trees, not inferred.

★ UPDATE 2026-08-04: the two blocking prerequisites are DONE. The posture question is unchanged and still open.

Steps (1) and (2) of option D have landed. Nothing is wired; no call site was added; the codes remain unreachable.

  • The per-axis unknown guard is fixed (shc modules/quota/enforcement.go). hasResources() is gone, replaced by per-axis knowsCPU() / knowsMem() predicates that also read a pointer-to-0 as unknown. §4’s measured refusals no longer reproduce: the 1-core/1-GiB request on a 48-core pool with memory unknown is admitted by all three gates. The known axis still enforces; modules/quota/enforcement_unknown_axis_test.go covers known-enforces, unknown-skips and the mixed case per gate, and every assertion was verified red against this page’s baseline tree first.
  • node.memory_bytes exists (ce, migration 20260804022217_auto.sql, additive, Int64 Nillable, NULL = unknown). It is written by the same three writers as cpu_cores (the genesis upsert at shc init, node_self_register’s boot upsert, and the per-node fact-refresh tick), so an upgraded estate backfills itself within one tick and a resized host re-converges without a daemon restart. The probe (cluster.PhysicalMemoryBytes) returns 0 for unmeasurable, never a guess, and 0 persists as NULL.

Still open, and still the owner’s call: the posture (A/B/C/D-then-B/E), the ce CapacityAdvisor seam, and the call sites in §2. Still true and now recorded in the code: the operand must be the reservation (§3), and docs/architecture/flows/install.md §10–§12 still documents these gates as things the install path does (§2, last paragraph) and is still wrong.

EnforceNodeCapacity (shc modules/quota/enforcement.go:117), EnforceSwarmPoolCapacity (:138) and EnforceLargestReplicaFitsNode (:184) are well-written, unit-tested, and have zero non-test callers in either repo. Their codes X403010NODCPU, X403011NODMEM, X403012SWMPOL, X403013SWMREP are unreachable. This page states where they would be called, what they would refuse, and why the answer is not yet rather than yes or delete.

The recommendation in one line: do not wire them this release; land the per-axis unknown-guard fix (zero production risk, no callers), add a durable node memory column, and only then wire them at warn, with the reservation as the operand, never the limit.


1. What each gate asserts, and in what units

Section titled “1. What each gate asserts, and in what units”

All three compare a request against a node’s TOTAL capacity. None of them subtracts what is already committed; there is no usage read in the file. So they are not oversubscription gates; they are does this even physically fit an empty box gates. The error templates say so in as many words: “node {node} only has {capacity} millicores total”.

GateSignatureAssertsNil-safe?
EnforceNodeCapacity(node NodeCapacity, cpuMillicores, memoryBytes int64)the request fits one named nodeyes; was per-field nil-guarded already; now also treats a pointer-to-0 as unknown
EnforceSwarmPoolCapacity(members []NodeCapacity, totalCPUMillicores, totalMemoryBytes int64)the request fits the sum of the poolwas no, now yes; see §4
EnforceLargestReplicaFitsNode(members []NodeCapacity, perReplicaCPUMillicores, perReplicaMemoryBytes int64)some single node satisfies CPU and memory togetherwas no, now yes; see §4

Units: CPU in millicores (NodeCapacity.CPUMillicores = cpu_cores × 1000), memory in bytes. NodeCapacity’s fields are pointers so “unknown” reads as nil rather than as a legitimate zero.

Gate 3 exists for the case gate 2 cannot catch: the aggregate pool is large enough but no single node satisfies both axes at once, so swarm accepts the service and leaves the task Pending forever.


The install/update render chain is ce modules/app/install_runner.go. Ordering at ce 8ab2081, and the ordering is the whole answer:

LineStep
~1025ApplyResourceQuotas: mints per-service quota; writes limits = quota × limit_ratio and reservations = quota × reservation_ratio
1068chargeRenderQuotas: the tenant-cap charge (see §3 on its operand)
~1204StampPlacement → renderedPlacement PlacementDecision{Node, Pins}; placement is decided here
1256SwarmLoaderRefusals: swarm-only preflight refusal
~1268deployDir := DeploymentDir(...): first filesystem write

So the correct window is (1204, 1268), after placement exists, before anything is created, which is what lets a verdict ride the existing pre-deploy refusal contract (install_refusal.go: nothing created, the QUEUED intent row deleted rather than stamped FAILED).

  • EnforceSwarmPoolCapacity and EnforceLargestReplicaFitsNode: immediately after StampPlacement, inside the existing if opts.Runtime == string(RuntimeSwarm) branch that already hosts SwarmLoaderRefusals at :1256. Swarm-only by nature: on compose there is one node and swarm’s scheduler is not making the decision.
  • EnforceNodeCapacity: the same window, on the pinned/host-bound branch. renderedPlacement.Pins names, per service, the node(s) the rendered artifact pins to; for a host-bound runtime the target is opts.PlacementNode.

All three need a new seam. ce cannot import shc/modules/quota. The shape already exists and should be copied, not invented: QuotaEnforcer in ce modules/app/quota_seam.go, wired daemon-side by shc daemon/wiring/quota_wire.go (eeWireQuotaEnforcer), fail-open on nil. A CapacityAdvisor alongside it is the minimal addition.

Nobody reading the docs would know they are unwired, which is likely how they stayed inert this long. docs/architecture/flows/install.md §10–§12 describes pool capacity, largest-replica-fits and EnforceNodeCapacity as things the install path does, with line numbers, and names X403014SWMFRG, CheckSwarmCanPlaceReplicas, EnforceStackCreationQuota, ExtractComposeResources and PreRenderCompose. All five have zero occurrences in either Go repo; that page documents the Python original as if it were the Go path. It should be corrected whichever way this is ruled.


3. The operand: reservations, not limits, and the call site has the wrong one

Section titled “3. The operand: reservations, not limits, and the call site has the wrong one”

The figure sitting in scope at the charge is renderQuotaTotals(quotas, replicas), Σ (quota × replicas). At a limit_ratio of 1.0 (the default then; it is 1.5 now) the quota is the limit, so that figure is the sum of the LIMITS. It is the correct operand for the accounting charge (the charge is always the quota; quota_render_charge.go) and the wrong operand for a physical gate. Four reasons, in ascending order of teeth:

  1. Swarm’s scheduler never consults limits. reservations is a hard admission constraint; limits is a post-placement cgroup setting the scheduler ignores. A gate on limits does not predict the Pending-forever failure it exists to catch.

  2. The overcommit is the design, and its size is a declared knob. limit_ratio ≥ 1 (default 1.5) with reservation_ratio defaulting to 1.0 means the intended overcommit factor is 1.5×. (This said 0.25 and 4× when it was written; ce 358b0f081 raised the reservation default to 1.0 and the limit default is 1.5.) Summing limits measures the deliberately-overcommitted number and reports it as a capacity breach.

  3. Measured, on compose01’s real shape (16 cores / 30 GiB), with both axes known:

    app (catalogue figures)Σ quotaquota operandreservation operand
    greenbone34000m / 32.62 GiBREFUSEADMIT
    mailcow8350m / 7.56 GiBADMITADMIT
    gitlab2250m / 16.25 GiBADMITADMIT
    huly7000m / 9.99 GiBADMITADMIT

    And gate 3, per-replica, on the smallest node either estate could use (Hetzner cpx32, 8 vCPU / ~15.6 GiB MemTotal): gitlab’s largest single service (16 GiB) is REFUSE on the quota operand and ADMIT on the reservation operand. Both are installs that succeed today.

  4. The estates are already red on the limit sum before anyone installs anything. docs/reference/quota-model-state.md:325 measures 20.68 GiB of memory limits on a 30 GiB node and 18.20 cores on 16. A gate on limits starts refusing on contact.

The reservation is available, from the right place. ParseComposeServices already returns CPURequestMillicores and MemoryRequestBytes, parsed from deploy.resources.reservations (ce modules/app/services_writer.go:118-121), and renderReplicaCounts already calls it. Reading the reservation off the rendered artifact (rather than recomputing quota × ratio) is what keeps the gate and the service row from disagreeing, and it is the only source that respects derive only what is absent: an authored reservation kept verbatim, or one clamped under its ceiling, is a figure quota × ratio would get wrong.

Note ServiceQuota deliberately does not carry the derived reservation (resource_quota.go:206-213); it did, and it was deleted as a written-by-one-read-by-none field. Do not re-add it; read the document.


4. The blocking precondition: the memory axis has no durable source, and two gates refuse everything without it

Section titled “4. The blocking precondition: the memory axis has no durable source, and two gates refuse everything without it”

FIXED 2026-08-04. Both halves of this section are closed; see the update box at the top. The measurements below are retained because they are the evidence for the ruling, and because they are the shape any future capacity consumer must not re-create.

There was no node memory column. ce’s node table carried cpu_cores (migrations/20260730032524_node_cpu_cores.sql, Nillable, NULL = unknown, never zero) and nothing for memory. The only memory-total source in the product was shc modules/metrics’ StatsCache → NodeSnapshot.MemoryTotalBytes, an in-memory cache with a 15-second staleness threshold (shc modules/hud/builder/builder.go:33) that a consumer on another node cannot read at all. node.memory_bytes now exists alongside cpu_cores with the same NULL-is-unknown contract and the same three writers.

And the unknown-resources guard could not survive a half-known node. hasResources() was cpu > 0 || mem > 0 (:35-37), and the skip was evaluated pool-wide, across both axes at once (:82-84, :121-123). CPU known + memory unknown ⇒ hasAnyResources == true ⇒ the memory comparison runs against poolMem == 0 ⇒ every memory-bearing request is refused. Measured against a 3-node pool of 16 cores each with memory unknown (i.e. exactly what the node table can supply), asking for 1 core and 1 GiB:

GATE1 EnforceNodeCapacity: admitted (per-field nil guard holds)
GATE2 EnforceSwarmPoolCapacity refused: [X403012SWMPOL] swarm pool capacity
exceeded, requires 1000 millicores and 1073741824 bytes but pool has
48000 millicores and 0 bytes across 3 nodes
GATE3 EnforceLargestReplicaFitsNode refused: [X403013SWMREP] no single swarm
node can host one replica, requires 1000 millicores and 1073741824 bytes
but largest node has 16000 millicores and 0 bytes across 3 nodes

So the naive-wiring failure was worse than oversubscription: not a policy error that refuses a few large stacks, but a refusal of 100% of memory-bearing installs on every estate, including single-node ones, with a message blaming the pool for having 0 bytes.

The fix, landed: the guard is per-axis. knowsCPU() / knowsMem() replace hasResources(), and each axis is judged only when it has a usable total (a nil pointer and a pointer-to-0 both read as unknown). Two details fell out of doing it honestly rather than mechanically:

  • Gate 2’s pool sum needs EVERY member to report the axis, not just one. A sum over a set containing an unknown is unknown; adding up only the members that answered under-states the pool and would refuse a stack that fits, the same unknown-as-zero error, localised to one member. One unmeasured node abstains the axis for the whole pool.
  • Gate 3’s per-node test must treat an unknown axis as SATISFIED, because the question is “does some node fit” and a node cannot be excluded unless it is provably too small. A refusal now means every member failed on an axis that was actually measured.

Because there are zero production callers, that fix carried no production risk, which is why it landed on its own ahead of any posture ruling.


PostureVerdict
AWire and refuseNo. Refuses installs that work today even with the right operand on a right source. (It no longer refuses everything; that was the guard defect, now fixed, but the reason to say no was never only that.)
BWire advisory warn-only, codedViable. X200944 is the one free code below the quota warn band (X200945–X200948; X200949/X200950 are taken), emitted via corelog.Warn exactly as X200946QTASRC and X200948QTALRT already are. Warn-vs-refuse is still the smallest part of the work, but the remaining part is now just the seam; the source and the guard fix are done.
CWire behind an off-by-default scoped knob: deploy.capacity_admission ∈ off|warn|refuse, default off, through the same 5-level cascade the other deploy.* keys useViable, and lets one estate opt into refuse after measuring.
DDo not wire; fix preconditions in order: (1) per-axis unknown guard done; (2) durable node.memory_bytes, NULL = unknown, written at enroll/self-register and re-converged by the existing per-node fact-refresh, mirroring cpu_cores done; (3) the ce seam; (4) then B or CRecommended, and steps 1–2 have landed, so this now reads “do (3), then rule on (4)”.
EDelete all threeNo; see below.

D, then C at warn. Specifically:

  • Not this release. (As written 2026-08-04 before the fixes: the memory axis had no durable source, and the two gates that would consume it refused everything when it was absent; §4.) Both are now closed, so the break-a-working-fleet objection is gone; what remains is the operand and the posture. A physical gate that is wrong once gets switched off permanently rather than fixed, which is still the argument for warn.
  • Land the per-axis guard fix first and alone. Done 2026-08-04, together with the durable memory column. Zero production callers, measured behaviour, no posture implications; the two landmines for whoever wires next are cleared. This was the only part of this page that was pure engineering; everything left is a decision.
  • Keep them; do not delete. On swarm they are the only thing between an operator and a task Pending forever, and the swarm path already needs more preflight, not less; 37 of 73 catalogue apps cannot docker stack deploy at all (M27, docs/architecture/cluster/swarm-contention-map.md). M33 on that page offers exactly this fork: “give them a call site and a node-capacity source, or delete them.” The source is the missing half, and it is buildable.
  • When wired, default warn, and never refuse on a first roll. Not caution in general; the specific stack of derivations: the operand is read off a document SHC itself rewrote three transforms earlier, compared against a capacity figure from a 15-second cache, on a fleet whose declared overcommit was 4× then (1.5× at today’s defaults).
  • The reservation is the operand. If the owner rules refuse, this is the load-bearing detail, because the limit sum is red on both estates before anything is installed (§3.4).

7. Two questions this surfaces and does not answer

Section titled “7. Two questions this surfaces and does not answer”
  1. Does a tenant lowering reservation_ratio buy themselves more physical admission? The 2026-08-03 ruling makes the ratio tenant-overridable (bounded by quota). Under any reservation-operand gate, a tenant who lowers their ratio shrinks the number the gate measures; they can admit more physical load by declaring they need less of it. Whether that is intended is a policy call.
  2. Should the compose runtime get a gate at all? These are physical-fit checks against total capacity, and compose has one node whose capacity the daemon is running on. Gate 1 is the only one that makes sense there, and what it would catch is narrow: a single service whose reservation exceeds the whole box.
  • shc modules/quota/enforcement.go: the three functions
  • shc docs/reference/quota-model-state.md §4: the inert-gates statement and the measured 20.68 GiB / 18.20 cores commitment
  • shc docs/architecture/cluster/swarm-contention-map.md M33: wire-or-delete
  • ce modules/app/quota_seam.go: the seam shape to copy
  • ce modules/app/install_runner.go:1204-1268: the call-site window
  • shc docs/decisions/2026-08-04-owner-docket.md: the rest of tonight’s docket