Skip to content
SHC Docs

The capacity substrate: what a rebalancer has to be able to read

Status: open design. Built in part on 2026-08-11, and the decision named below says which claims changed. The other steps are not built.

★ SUPERSEDED IN PART: read docs/decisions/2026-08-11-rebalancer-v1-foundations.md first

Section titled “★ SUPERSEDED IN PART: read docs/decisions/2026-08-11-rebalancer-v1-foundations.md first”

This page is the DESIGN DRAFT. The lane that built from it landed on 2026-08-11 and measured several of its claims differently. Where this page and that decision disagree, the decision is what was measured and this page is kept for its reasoning, not for its facts.

The corrections that matter here:

  • §9.1 is ANSWERED. The reservation multiplier is 1.0 and the limit multiplier 1.5, the owner’s model. This page was written to be correct under either value and it is; the value is now 1.0, so a node’s committed sum is 4x what it was, and the live estates read ~69% committed rather than ~17%.
  • §1’s Q1 row and §5.4 both understate what already exists. The node capacity columns have three writers each and had them before this page was drafted; the defect was that scheduling_source.go never scanned them. Fixed with no migration.
  • §5.4’s fourth gate (EnforceNodeHeadroom) is NOT built, and §9.3 is the reason: its numbers need the daemon’s own footprint measured on both estates first, and an absolute floor invented rather than measured is a refusal threshold nobody can defend. The PROPOSED placeholder of 1000m / 2 GiB was deliberately not shipped.
  • Steps 0, 1, 4 and 5 of §6 are NOT built. The utilization axis, the free-disk fact, the node-addressed free-space read and the durable digest all remain owed. Step 2 (the commitment reader) and the gate half of step 3 are built; the ce-side CapacityAdvisor seam is not; what landed instead is a PLACEMENT seam on the same install path.

Everything else on this page stands, including the parts the implementation did not reach; they are named in §8 of the decision.

As written: PROPOSED. Design only; nothing on this page is built. Steps 0-2 of §6 are safe to build before any ruling; step 3 is blocked on §9.1 and §9.3. This is lane 2 of the automatic-distribution design: not the scheduler, not the move, but the layer underneath both: what does node N have, and what is already promised on it. A scheduler cannot exist without it and a rebalancer that guesses at it is a random walk with a commit point.

Baselines. shc main c0b009ba5, ce main 6a3dcc5. Every claim tagged MEASURED was read out of those two trees during this session and the file and line are given. Claims tagged PROPOSED, ASSUMED or RULING OWED are exactly that. Claims taken from another page without re-verification are tagged INHERITED and name their source.


0. Four corrections to the ground truth this lane was handed

Section titled “0. Four corrections to the ground truth this lane was handed”

The brief for this lane carried four statements that the trees do not support. Three of them make the job smaller than it looked; one makes it larger. All four change the design, so they come first.

0.1 Cross-node live utilization already exists, and it is already wired

Section titled “0.1 Cross-node live utilization already exists, and it is already wired”

The brief said the StatsCache is “per-process, 15s, empty before the first health tick, and a consumer on another node cannot read it at all.” The first three clauses hold. The fourth does not.

MEASURED. ce modules/metrics/collectors/peer_nodes.go is PeerNodeStatsCollector: a fan-in that asks every peer for its local StatsCache view over the cluster mTLS internal API and merges the result into the caller’s own cache. It is:

  • wired: ce subsystems/impl/collectors.go:368, unconditionally, in the collectors subsystem’s start path;
  • running on every node, not just the leader; LeaderOnly() returns false and the comment says why (“every node aggregates its own cluster-wide view so the HUD reflects all nodes no matter which node it connects to”);
  • served by a real endpoint: GET /internal/_node_stats, registered at ce modules/metrics/router.go:33, returning {"nodes":{…},"containers":{…}};
  • dialled over the mesh peer client: ce subsystems/impl/peer_node_stats_adapter.go, peers enumerated from the node registry by internal_address, local node excluded;
  • ticking at 2 seconds; resolveTickInterval (collectors.go:703) reads timing.tick_interval, whose shipped value is "every 2 seconds" (ce core/config/default.yaml:244) and whose unparseable fallback is 2s.

So the transport question for (b) is not “build one”. It is “the one that exists carries three defects that make it unsafe for a scheduler to read”: §3.

0.2 A per-node disk fact exists too, and it is the wrong number, measured six ways

Section titled “0.2 A per-node disk fact exists too, and it is the wrong number, measured six ways”

The brief said “per-node free disk; no fact exists anywhere.”

MEASURED. NodeStats (ce modules/metrics/stats_cache.go:32-34) carries DiskTotalBytes, DiskUsedBytes and DiskPercent, produced by readDiskStats(s.statfsPath) where statfsPath defaults to /var/lib/shc and falls back to / when that does not exist (ce modules/metrics/collectors/system_tick_source.go:92-97). It rides the fan-in in §0.1. It reaches every node’s cache every 2 seconds.

What does not exist is a usable free figure. readDiskStats computes used = total − Bfree (system_tick_source.go:504-517), so any consumer deriving free = total − used recovers Bfree, which counts the root reserve, the 5% of an ext4 filesystem an unprivileged writer cannot touch. The two places in the product that actually gate on free space both use Bavail, and say in their own comments that the root reserve is not budget.

MEASURED: six syscall.Statfs call sites, three definitions of “free”:

sitepathfield
ce modules/clone/preflight_space.go:134 DiskFreeBytesstaging dir, walks to nearest existing ancestorBavail
ce modules/backup/preflight_space.go:169 diskFreeBytesbackups dir, same ancestor walkBavail, a byte-identical second copy of the function above
ce modules/metrics/collectors/system_tick_source.go:504 readDiskStats/var/lib/shc || /Bfree
ce modules/health/internal_router.go:288/, hardcodedBfree
ce modules/health/maintenance.go:150 defaultDiskUsageProbecaller-suppliedBfree
ce daemon/services_seams_maint.go:317 maintDiskUsageProbecaller-suppliedBfree, a second copy of the probe above

Two of the six are literal duplicates of another two. Nothing reconciles them. A node’s disk pressure therefore reads differently depending on which surface asked, and the two surfaces that refuse work disagree with the four that report health.

0.3 The reservation is already summed per node, and the sum is wrong three ways

Section titled “0.3 The reservation is already summed per node, and the sum is wrong three ways”

The brief said “nothing sums commitments.”

MEASURED. ce subsystems/impl/collectors.go:1046-1072 (aggregateSourceAdapter.Stacks) walks every stack, sums sm.CPURequestMillicores and sm.MemoryRequestBytes across its services, and labels the row with a NodeID. Those become the OTel gauges shc_stack_cpu_request_millicores / shc_stack_memory_request_bytes (ce modules/metrics/collectors/aggregate.go:142), and the per-service variants carry node_id too (aggregate.go:116).

It is not a commitment ledger, for three independent reasons:

  1. It does not multiply by replicas. row.CPURequest += float64(sm.CPURequestMillicores); no desired_replicas factor, while the quota charge on the very same table does multiply (ce modules/app/quota_render_charge.go:73, quota × desired_replicas). Two summations of one table disagree on the replica factor and the metric one is the wrong one.
  2. Its node attribution is the routing scalar. NodeID comes from s.Node (deployment.recorded_node_id) or, when a container row lands first, from that container’s node. recorded_node_id names whoever handled the install request; under swarm it is not even a member of the set of nodes running the tasks. That is the exact confusion the deployment_node set was created to end (ce ent/schema/deployment_node.go, whole header).
  3. Nothing in the product can read it back. ce modules/metrics/router.go states it plainly: “Per B9 there is NO /metrics scrape endpoint; push-only via the OTLP exporters”. The numbers leave for a telemetry backend and never return.

So the ingredients for a commitment ledger are all persisted and one wrong aggregation of them already ships. That makes §2 a smaller job than it looked and a more delicate one: the failure mode is not an empty page, it is a plausible number.

0.4 The reservation multiplier is contested, and it moves every figure on this page by 4×

Section titled “0.4 The reservation multiplier is contested, and it moves every figure on this page by 4×”

This is the correction that makes the job larger.

The brief stated the owner’s model as reservation = quota × 1.0 and limit = quota × 1.5. MEASURED, ce main disagrees on both constants:

  • defaultReservationRatio = 0.25: ce modules/app/resource_reservations.go:102
  • defaultLimitRatio = 1.0: ce modules/app/resource_quota.go:174
  • and the chain in the shipped header (resource_quota.go:12-14) is limit = quota × limit_ratio / reservation = quota × reservation_ratio, i.e. the owner’s shape with different values.

INHERITED (memory quota_derivation_model, owner 2026-08-03): the owner stated reservation_multiplier default 1.0 and limit_multiplier default 1.5, and explicitly rejected deriving the reservation from an app-authored limit. The shipped code derives from the quota (the shape objection is resolved), but ships 0.25 and 1.0.

Why this lane cares more than the quota lane does. The quota charge is the quota, always, so the ratios do not move the bill. The commitment ledger’s operand is the reservation, so:

committed sum, per unit of granted quota
ratio 0.25 (shipped)25%
ratio 1.0 (owner’s stated model)100%

At 0.25 a node is “full” only when four times its RAM has been sold on it. At 1.0 it is full when its RAM has been sold once. Every threshold, every headroom number and the entire question of whether a capacity gate ever fires turns on this. RULING OWED (§9.1). This page is written so the ledger is correct under either value, and it does not pick one.


1. The four questions the substrate has to answer

Section titled “1. The four questions the substrate has to answer”

Everything below is in service of exactly these. A consumer that needs a fifth thing is out of scope for this lane.

#questionanswered fromtoday
Q1how big is node Nnode.cpu_cores, node.memory_bytesexists, NULL = unknown, three writers
Q2how much of N is already promisedΣ reservation × replicas over what occupies Ndoes not exist (§0.3 is not it)
Q3how loaded is N right nowNodeStats via the peer fan-inexists, unsafe to read (§3)
Q4how much writable disk does N have, per path-classstatfsexists, wrong field, one path (§0.2)

★ Q1 and Q2 are claims about intent and are answerable from replicated tables at any time, including while a node is unreachable. Q3 and Q4 are measurements and are only as good as the last sample. Keeping the two classes apart is the single structural decision on this page, and every disposition below falls out of it: a gate may be built on intent; a gate built on a measurement has to say what it does when the measurement is missing.


2.1 The operand, and where every piece of it already lives

Section titled “2.1 The operand, and where every piece of it already lives”

committed(N) is the sum, over everything occupying node N, of reservation × replicas. Both factors are persisted columns on service (ce ent/schema/service.go:52-61):

service.cpu_request_millicores int NOT NULL DEFAULT 0
service.memory_request_bytes int64 NOT NULL DEFAULT 0
service.desired_replicas int NOT NULL

MEASURED. The request pair is written from the rendered artifact; ParseComposeServices reads deploy.resources.reservations (ce modules/app/services_writer.go:81) and UpsertService persists it (ce modules/app/container_repo.go:187-191). That is the source docs/decisions/2026-08-04-physical-capacity-gates.md §3 ruled correct, and it is correct for a reason worth restating: it respects derive only what is absent. An authored reservation kept verbatim, or one clamped down under its quota ceiling (X200947QTACLM), is a figure that quota × ratio computes wrong.

★ And SHC does write reservations now. The brief said it never does, and that Swarm therefore has nothing to honour. ce modules/app/resource_reservations.go exists precisely to inject them, and its header carries first-hand measurements against docker 29.4.1 / compose v5.1.3: reservations.cpus is ignored by compose entirely and read only by swarm’s scheduler; reservations.memory becomes HostConfig.MemoryReservation, a soft reclaim bias that cannot kill (measured: a 64M reservation allowed a 300M allocation, exit 0). On swarm both keys are hard admission constraints. So the operand this ledger sums is also the operand swarm’s own scheduler would consult, which is the property that makes the ledger predictive rather than decorative.

2.2 The attribution problem, and why deployment_node is necessary but not sufficient

Section titled “2.2 The attribution problem, and why deployment_node is necessary but not sufficient”

The ledger needs a map from deployment to node. Three candidates exist and only one combination works.

  • deployment.recorded_node_id: the routing scalar. Measured wrong for occupancy: it names whoever handled the request, and an unpinned swarm install stamps no placement constraint at all. Using it is the defect that let the node-removal gate permit removing a node running live tasks (ce ent/schema/deployment_node.go header).
  • container.node_id: first-hand per-node attribution (ce ent/schema/container.go:50). Correct while it exists. It stops existing: the leader reaper deletes container rows 120s after the containers go, and an unreachable node’s rows are cleared during the exact outage in which the answer matters most.
  • deployment_node(deployment_id, node_id, anchored, last_observed_at): the set, with anchored written by the same call that writes recorded_node_id so it is never empty, and last_observed_at monotone.

★ The set answers whether, not how much. A 3-replica service spread across three nodes commits one third of its reservation to each, and deployment_node has three rows that all say the same thing. Counting the whole deployment’s reservation once per occupied node triple-counts it. So deployment_node alone cannot be the ledger’s key either.

PROPOSED.

committed(N) = Σ over services s of reservation(s) × placed(s, N)
placed(s, N) = |{ containers of s observed on N }| ... when any
container
of s is
observed
anywhere
= s.desired_replicas if deployment_node(D(s), N).anchored ... otherwise
= 0 ... otherwise

In words: count real replicas where we can see them; fall back to the whole declared shape on the anchor when we cannot see any. The fallback covers the three cases the deployment_node header already enumerated (a stopped deployment, an unreachable node, and the window before a fresh install’s first recorder pass), and it covers the ordinary compose case, where the anchor and the runner are the same node and the fallback is exact.

2.4 The deployment_node lesson applies, with the sign of the danger flipped

Section titled “2.4 The deployment_node lesson applies, with the sign of the danger flipped”

The recorded rule there is prune on positive evidence, never on absence. It applies here, and the reason it applies is worth writing out because the two gates fail in opposite-looking directions that turn out to be the same direction.

  • For the removal gate, a derived-empty set produced a permit; a node with live tasks looked removable.
  • For the ledger, a derived-empty sum produces “this node is free”, an admit.

Both are the unsafe direction, so the rule is the same: absence of observation must never reduce a commitment. A stopped deployment still holds its disk and will hold its memory again the instant it starts. An unreachable node’s containers are not gone; they are unobserved.

★ The difference, named rather than glossed: the removal gate’s failure mode is data loss and irreversible; the ledger’s is one deployment too many on a node that then swaps. The ledger can therefore afford a cheaper correctness posture than the removal gate (it does not need its own reconciler), but it must fail in the same direction. Unknown counts as committed, never as free.

2.5 Derived on demand, not materialised, and the argument that decides it

Section titled “2.5 Derived on demand, not materialised, and the argument that decides it”

PROPOSED: derive it, from one indexed query, with no new table and no migration.

  1. Every input is already replicated and already correct. service, deployment, deployment_node and container all live in the same store, and three of the four already CASCADE from deployment. An uninstall therefore releases the commitment by deleting the rows the commitment was computed from. There is nothing to credit back, nothing to drift, and no release path to get wrong. That is the single strongest argument and it is worth more than the query cost.
  2. A materialised ledger is usage again, on a second axis. MEASURED, the cost of that shape: usage needed its check and its charge folded into one statement batch (shc modules/quota/enforcement.go, UpdateUsageOnScale) after a measured race in which eight concurrent 500m charges under a 2000m cap all read used = 0 and all admitted, and a mutex could not have fixed it, because the racing charges come from different daemons against one shared store. It also needs a reconcile cron and a repair verb. A node ledger would need all three again, for a fact that is derivable.
  3. The read rate does not justify it. The ledger is read at admission and at rebalance-planning time: tens of reads a minute, not thousands a second. INHERITED: both estates run 33 deployments (memory atomic_move_design). The join is over (deployments × services).

★ This does not dodge the “derived-only cannot be a gate” ruling; it honours it one layer down. What could not be derived there was the set. Here the set is an input, not the output; the output is a byte figure derived from a set that is itself durable and never empty. The whole argument rests on the anchor row being written unconditionally. If a path is ever allowed to create a deployment without an anchor row, this design is wrong and the ledger has to be materialised.

PROPOSED, in ce package app, alongside the existing repo readers:

type NodeCommitment struct {
NodeID string
CPUMillicores int64
MemoryBytes int64
// Deployments counted, and how each was attributed, so a caller can
// say "3 of 11 were counted from the anchor because no container of
// theirs was observed" instead of publishing a bare number.
Deployments int
AnchorFallbackCount int
// UndeclaredServices counts services whose reservation columns are 0.
// A 0 is NOT a measurement here (see below), so a consumer that wants
// to be a gate can refuse to be one when this is a large fraction.
UndeclaredServices int
TotalServices int
ObservedAt time.Time
}
type NodeCommitmentReader interface {
CommitmentFor(ctx context.Context, nodeID string) (NodeCommitment, error)
CommitmentForAll(ctx context.Context) ([]NodeCommitment, error)
}

★ UndeclaredServices exists because a zero here is ambiguous and the substrate must not pretend otherwise. MEASURED: service.cpu_request_millicores is Default(0) and NOT NULL (ce ent/schema/service.go:56), and the schema comment says why: “an existing row predates them and 0 is the honest reading: nothing declared.” But nothing declared and declared zero are the same bit pattern, and the whole hostfacts package exists in this codebase to stop exactly that collapse. In practice ApplyResourceQuotas writes a reservation on every render, so any deployment installed or updated since the column landed carries a real figure, but a fleet is not all one vintage, and a ledger that silently under-counts a pre-column deployment reports a node as emptier than it is. Counting them and reporting the count is the cheap honest move. RULING OWED (§9.2) on whether to backfill or make the column nillable.

2.7 Behaviour under move, uninstall and failure

Section titled “2.7 Behaviour under move, uninstall and failure”
eventledger behaviourcorrect?
uninstallinputs CASCADE away with the deploymentyes, by construction
move, mid-flightsource anchor persists until the uninstall half; target anchor written at restore ⇒ both counted★ yes, and deliberately: during the window both nodes really do hold the bytes, and the target really will need the memory. Double-counting is the conservative answer and it self-clears at the commit point
move, reverted after the target anchor was writtentarget over-counted until the anchor row is prunedacceptable; the existing prune rule (positive evidence, node reachable) owns it, and the error is in the safe direction
node unreachableobservation stops; the anchor floor holds; committed does not fallyes; this is §2.4
daemon restartnothing to warm; the ledger is a queryyes, and it is the reason this axis, unlike §3, is readable during boot

3. (b) Live utilization, readable cross-node

Section titled “3. (b) Live utilization, readable cross-node”

PROPOSED: do not build a transport. Promote the one in §0.1, and fix the three things that make it unsafe to gate on.

3.1 Defect one: the freshness clock restarts at the merging node

Section titled “3.1 Defect one: the freshness clock restarts at the merging node”

MEASURED. StatsCache.UpdateNode stamps stats.UpdatedAt = time.Now() unconditionally on write (ce modules/metrics/stats_cache.go:128-133). The fan-in calls exactly that method for every peer entry (peer_nodes.go, c.cache.UpdateNode(id, ns)). NodeStats.UpdatedAt has no json tag, so it marshals under its Go field name and does round-trip; the peer’s own timestamp arrives and is then discarded.

Consequence: a peer whose local tick source has wedged keeps serving its last-good NodeStats from its own cache; the merging node re-stamps it fresh every 2 seconds; NodeAge reports 2s for data that is an hour old. The one staleness signal in the system reads green for the failure it exists to catch.

★ This is read out of the source, not proven against a running daemon. Marked MEASURED-FROM-SOURCE; a live repro is one wedged tick source away and should be part of whatever lands the fix.

Fix (PROPOSED): carry the peer’s UpdatedAt across the merge (UpdateNodeAt(id, stats, observedAt)), and leave UpdateNode’s behaviour alone for the local entry, which genuinely is fresh at write time.

3.2 Defect two: unmeasured exists on one payload and not the other

Section titled “3.2 Defect two: unmeasured exists on one payload and not the other”

MEASURED. ce core/hostfacts is the package whose entire purpose is that “unmeasured” is a value and not a zero, and /internal/health’s SystemMetricsPayload carries UnmeasuredFacts accordingly (ce modules/health/internal_router.go:80). /internal/_node_stats’s NodeStats does not. And SystemTickSource.ReadNode does not go through hostfacts at all; it has its own readMemInfo / readCPUPercent / readLoadAvg (system_tick_source.go:386-500), a second set of /proc readers in the same binary, none of which can express a failed read.

So the fan-in delivers MemoryTotalBytes: 0 for a node that could not measure memory, and a consumer cannot tell it from a node with no RAM. That is the same class as the defect the physical-capacity gates decision measured in §4 of its own page (an unknown collapsing into a zero) reaching a scheduler instead of an installer.

Fix (PROPOSED): NodeStats.UnmeasuredFacts []string, populated in ReadNode through core/hostfacts, retiring the duplicate /proc readers. This is a prerequisite for everything else and is independently useful today: the HUD publishes those zeroes now.

3.3 Defect three: no durability, which is fine, and one consequence that is not

Section titled “3.3 Defect three: no durability, which is fine, and one consequence that is not”

The cache survives nothing. After a daemon restart the node map is empty for one tick plus one peer round-trip. NodeAge returns 24 hours for a missing entry (stats_cache.go:159), so a fail-closed consumer correctly refuses during that window.

PROPOSED: accept it, and do not replicate the hot path. Writing every node’s utilization into rqlite every 2 seconds is a write per node per tick through raft, forever, for a fact that expires in twenty seconds. It is the “put the hot loop in the consensus log” mistake, and it is the same instinct the move journal deliberately refused when it chose a node-local file over a table.

★ One durable thing is worth writing, and it is not the sample. Phase 2, not required for the first gate: a digest, one row per node per minute, last-write-wins, holding a 5-minute EWMA of CPU and memory plus free bytes per path-class and a measured_at. One write per node per minute. It exists because utilization is a claim about an interval, and the 2-second sample cannot make one. An instantaneous sample says a node is busy now; it cannot say a node is busy. A rebalancer that plans on instantaneous samples thrashes; it moves work onto a node that was idle for the 2 seconds it was asked about. The EWMA is not a refinement; it is the difference between a scheduler and a coin flip.

3.4 The staleness budget, and what happens past it

Section titled “3.4 The staleness budget, and what happens past it”

PROPOSED: 20 seconds for the utilization axis, stated as a named constant adjacent to the tick interval, and derived rather than picked: 2s tick × 8 missed rounds, which is the HUD’s existing 15s (shc modules/hud/builder/builder.go:33) plus one peer round-trip. A sample older than that means the reader has not heard from that node in ten ticks.

★ What a scheduler does past it: REFUSE TO MOVE. Fail-safe is refuse, and the asymmetry is the whole argument.

  • Not moving costs an imbalance that persists to the next planning cycle. It is undone by doing nothing.
  • Moving on stale data costs a full stateful cutover (stop, archive, ship, restore, health gate, data gate), possibly onto the node that is actually the hot one. INHERITED (memory atomic_move_design): the revert window closes at the commit point, after which failures roll forward. So a move started on a wrong premise cannot be taken back by regretting it.

The two are not symmetric, so the rule is not symmetric.

★ But stale utilization must never block an install. Admission and rebalancing are different verbs with different costs of refusal. An install refused because the daemon restarted three seconds ago is a product defect; a move refused for the same reason is prudence. The dispositions therefore split by verb, not by axis:

verbQ3/Q4 missing or staleQ2 unreadableQ1 unknown on an axis
install / admissionproceed; warnproceed; warnskip that axis (the existing per-axis rule)
rebalance / choose a move targetrefuse that node as a targetrefuserefuse that axis; refuse outright if both axes unknown
drain (operator-initiated)proceed, warnproceed, warnproceed

★ Drain is the third case and it is the interesting one. The deployments have to go somewhere; the source node is being removed. Refusing on stale data strands them on the node the operator is emptying, which is worse than placing them imperfectly. So drain keeps today’s disposition (place anyway; today that means pickTarget’s lowest-row-count) and gains a warning, never a gate. A capacity substrate that makes shc node drain refuse is a regression even when its reasoning is impeccable.


PROPOSED, step 1. Promote clone.DiskFreeBytes (Bavail, ancestor-walking, already written and already the right shape) into ce core/ next to hostfacts, as the single free-space probe, returning (bytes, ok) so an unreadable filesystem is not a zero. Then:

  • delete backup.diskFreeBytes (a literal copy) and call the promoted one;
  • collapse maintDiskUsageProbe and defaultDiskUsageProbe (a second literal pair) onto it;
  • point readDiskStats and internal_router’s statfs at it.

★ This is the “reuse rather than duplicate” the brief asked for. The sibling lane already wrote the function; the work is making it the only one.

★ And one fleet-visible hazard has to land with it. Switching readDiskStats from Bfree to Bavail raises every node’s reported DiskPercent by roughly the root reserve (about 5% on ext4) and nodeStateFromMetrics thresholds at > 90 → unhealthy and > 80 → degraded (ce modules/health/internal_router.go:114-124, system_tick_source.go:154-161). A node reading 87% today reads about 92% after, and flips to unhealthy on contact. That is the same shape as the limits-operand problem in the physical-capacity-gates decision: a correct fix that starts refusing on the day it lands.

PROPOSED resolution: do not move DiskPercent. Leave it on Bfree for wire stability and thresholds, and add FreeBytes (Bavail) as a new field. The percentage stays a health signal; the byte count becomes the budget. They are different questions and it is fine for them to have different answers, as long as the field names say which is which.

4.2 The fact, per path-class rather than per path

Section titled “4.2 The fact, per path-class rather than per path”

PROPOSED. NodeStats.Filesystems map[string]FilesystemFact, keyed by role, not by path:

type FilesystemFact struct {
Role string // "deployments" | "staging" | "backups"
MountPoint string // so a reader can see three roles on one mount
TotalBytes int64
FreeBytes int64 // Bavail
Measured bool
MeasuredAt time.Time
}

Roles, not paths, because the questions are role-shaped (“can the target hold the restored tree”, “can the executor hold the archive”) and because the mount point is what tells a reader whether two roles are competing for the same bytes. On a stock estate all three resolve under /var/lib/shc and carry one mount point; the day anyone mounts a separate volume for backups, the single figure in use today is wrong for two of the three and nothing notices.

Cadence: not 2 seconds. Three statfs calls per node per tick plus the fan-in is noise for a number that moves slowly. PROPOSED: refresh the filesystem facts every 30 seconds inside the existing tick source (a counter, not a second loop), and stamp MeasuredAt per entry so the staleness of this axis is independent of the CPU/memory axis.

4.3 What it is for, and the one capability that does not exist yet

Section titled “4.3 What it is for, and the one capability that does not exist yet”

Use 1: the move’s staging preflight, which is already built for the local side. preflightStagingSpace (ce modules/clone/preflight_space.go) refuses X507632CLNSPC when the staging disk cannot hold the estimate. It is the right gate with the right disposition (unmeasurable skips with a warning, because this gate protects a disk rather than licensing a destruction, the opposite of the data gate, and its header says so).

★ The gap: it measures the executor’s staging dir and nothing else. MEASURED: FilesystemProvider.FreeBytes(path string) takes no node argument (ce modules/clone/types.go:868), while its sibling MeasureDeploymentTree(ctx, tenant, stack, environment, node) does (types.go:963) and the full daemon bundle serves it over the mesh peer client. So a move can measure a remote tree and cannot measure a remote disk.

A rebalancer choosing node N as a move target must know N can hold the restored deployment. Nothing can ask. This is the concrete capability lane 2 owes the move lane, and once §4.2 lands it is a cache read rather than a new RPC:

FreeBytesOn(ctx context.Context, nodeID, role string) (bytes int64, measured bool, err error)

served from the fan-in, with measured=false when that node’s entry is stale past the §3.4 budget or names the role unmeasured. The move’s preflight then checks both ends and keeps its skip-with-a-warning disposition on the target side too.

Use 2: placement. A node whose deployments filesystem sits at 92% is not a move target. ★ And a refuse on disk is defensible in a way a refuse on CPU is not: CPU exhaustion is a slowdown, ENOSPC is an outage, and the measured incident that motivated preflight_space.go was a backup that “filled the staging filesystem MID-CAPTURE… and the failure took the whole node’s disk down with it; the daemon, the databases and every co-resident deployment share that filesystem.” Disk is the one axis where this substrate should be allowed to say no before any ruling on the others.


5.1 The shape, copied from QuotaEnforcer exactly

Section titled “5.1 The shape, copied from QuotaEnforcer exactly”

MEASURED, the pattern to match (ce modules/app/quota_seam.go): an interface in package app; a package-level var quotaEnforcer QuotaEnforcer; SetQuotaEnforcer(q) that no-ops on nil; unexported helpers (enforceQuota, recordQuotaCreate, …) that no-op when unwired; wired daemon-side from shc daemon/wiring/quota_wire.go (eeWireQuotaEnforcer); fail-OPEN on nil for the caps, with recordQuotaRender the one deliberate fail-closed member.

PROPOSED, ce modules/app/capacity_seam.go:

type CapacityRequest struct {
Runtime string
Node string // pinned / host-bound target; "" on unpinned swarm
Members []string // swarm pool member node ids; nil on compose
// Σ reservation × replicas over the render, and the largest single
// replica, the two operands the three existing gates already take.
TotalCPUMillicores int64
TotalMemoryBytes int64
LargestReplicaCPUMillicores int64
LargestReplicaMemoryBytes int64
}
type CapacityVerdict struct {
Fits bool
Reasons []string // one per failing axis, operator-readable
Code string // the coded refusal a refusing posture would return
Skipped []string // axes NOT judged, and why: "memory: node w1 reports no total"
}
type CapacityAdvisor interface {
Advise(ctx context.Context, req CapacityRequest) (CapacityVerdict, error)
}

5.2 Two deliberate departures from QuotaEnforcer, and why

Section titled “5.2 Two deliberate departures from QuotaEnforcer, and why”

★ It is Advise, not Enforce, and the name is the design. QuotaEnforcer.Enforce returns error: allowed, or a coded 403. One bit. A physical capacity check has three outcomes that matter (fits, does not fit, could not tell), and collapsing the third into either of the other two is precisely the defect the gates decision measured: an unknown read as a zero refused 100% of memory-bearing installs on every estate, single-node ones included, while blaming the pool for having 0 bytes. A verdict struct makes “could not tell” expressible across the repo boundary, which is the only place it can be handled correctly.

★ The advisor returns a verdict; the CALLER picks the posture. That is the opposite of the quota seam and it is deliberate. Quota posture is not configurable; a cap is a cap. Capacity posture is: deploy.capacity_admission ∈ off | warn | refuse, default off, option C of the gates decision. Putting the knob at ce’s call site rather than inside the shc functions means it resolves through the ordinary 5-level cascade at the one point where tenant, environment and stack are all in hand, and means the three gates never have to know a policy exists.

INHERITED from docs/decisions/2026-08-04-physical-capacity-gates.md §2: the window is after placement is decided and before the first filesystem write, so a refusal rides the existing pre-deploy refusal contract: nothing created, the QUEUED intent row deleted rather than stamped FAILED.

MEASURED, re-located at ce 6a3dcc5 (that page’s line numbers were taken at 8ab2081 and have moved):

linestep
install_runner.go:1275ApplyResourceQuotas: mints quota, writes derived limits + reservations
:1318chargeRenderQuotas: the tenant-cap charge
:1455StampPlacement → renderedPlacement: placement is decided here
:1507SwarmLoaderRefusals: the existing swarm-only preflight
:1520deployDir := DeploymentDir(...): first filesystem write

The window is (1455, 1520), and :1507 is an existing refusal already sitting inside it, so the new call has a neighbour with the right disposition to sit beside.

5.4 The three existing gates: adopt two, re-sign one, add a fourth

Section titled “5.4 The three existing gates: adopt two, re-sign one, add a fourth”

PROPOSED.

  • EnforceLargestReplicaFitsNode: adopt verbatim. Its question is “is any node physically big enough for one replica”, and the answer does not depend on what is running: a 16 GiB service does not fit an 8 GiB node whether that node is empty or full. Knowing usage cannot improve it. It is also the only one of the three that catches a failure with no other detection (swarm accepting the service and leaving the task Pending forever), and its per-node unknown handling (an unmeasured axis counts as satisfied, because a node cannot be excluded unless provably too small) is already right.

  • EnforceSwarmPoolCapacity: adopt verbatim, understood as a floor. It is a necessary condition, not a sufficient one: it catches “the whole fleet cannot hold this stack”, which no per-node check catches because the stack could legitimately be spread. Its stricter unknown rule (an axis is comparable only when every member reports it) is correct for a sum and should not be loosened.

  • EnforceNodeCapacity: re-sign it. This is the one that must change, and the change is a signature change, not a policy change. Today it compares a request against TOTAL and reads no usage. ★ Its own doc comment is the specification: “Turning them into oversubscription checks means subtracting committed reservations, which needs an operand no signature here accepts; a call-site change, not a change in this file.” So:

    EnforceNodeCapacity(ctx, node NodeCapacity, committed NodeCommitment,
    cpuMillicores, memoryBytes int64) error

    comparing against total − committed. ★ And the unknown handling has to grow a third state, which that comment does not mention. Total known plus committed unknown is not “committed = 0”; that is the unknown-as-zero error one layer out, and it admits everything instead of refusing everything. The per-axis predicates generalise: an axis is judgeable only when knowsTotal(axis) && knowsCommitted(axis), and an unjudgeable axis lands in CapacityVerdict.Skipped with its reason.

  • The fourth gate, which does not exist: headroom. total − committed ≥ request admits a node to exactly 100% reserved. Every real scheduler keeps a floor for the system itself: the daemon, rqlite, the docker engine, the collector, the OS. PROPOSED EnforceNodeHeadroom, subtracting a system allowance before any deployment is counted. ★ Not a percentage: 10% of a cpx32 and 10% of a cx53 are different amounts of daemon, and the daemon’s footprint does not scale with the box. An absolute floor, resolved through the ordinary cascade as deploy.node_system_reserve.cpu / .memory. RULING OWED (§9.3) on the numbers; I did not measure the daemon’s own footprint on either estate, and that measurement is a prerequisite to choosing them, not a detail to be filled in later.

5.5 What the gates still cannot do after all of this

Section titled “5.5 What the gates still cannot do after all of this”

They judge a request against a node. They do not choose the node. Scoring, spread, anti-affinity, concurrency caps and the rebalancing decision itself belong to another lane. This substrate’s job is to be the thing that lane reads, and to refuse cleanly when it cannot answer.

★ MEASURED, and worth stating because it is the whole reason lane 2 exists: the only node-scoring code in the product is pickTarget (shc modules/cluster/drain.go:512): lowest count of running deployment rows, tie-break by node id, prefer matching runtime, filtered to availability=active, with the count supplied by DeploymentLister.ListByNode (drain.go:493-502). No CPU, no memory, no disk, no quota, no affinity. It is not exported, and it is never consulted at install. A row count is a proxy for load that is exactly wrong in the case that matters: one gitlab is not one test-simple.


6. The minimum viable substrate, in build order

Section titled “6. The minimum viable substrate, in build order”

Ordered by dependency. Each step is independently landable, independently useful, and independently revertible.

Step 0: the honesty fixes. No dependencies, no new consumers, ship first. NodeStats.UnmeasuredFacts through core/hostfacts (§3.2), and preserve the peer’s UpdatedAt across the fan-in merge (§3.1). ★ Ship this before anything else, because it is the step whose absence makes every later step silently wrong: a scheduler reading a fabricated zero or a re-stamped stale sample is worse than one with no data, since it acts confidently.

Step 1: one free-space function, and one new field. Promote clone.DiskFreeBytes to core; delete the two duplicates; add FreeBytes (Bavail) to NodeStats without moving DiskPercent (§4.1). Add the per-role Filesystems map at 30s cadence (§4.2).

Step 2: the commitment reader. app.NodeCommitmentReader, one indexed query, the anchor-floor rule, the UndeclaredServices count. No new table, no migration. Read-only; nothing consumes it yet, so it can land and be measured against both estates before anything depends on it.

Step 3: the seam and the gates. app.CapacityAdvisor + SetCapacityAdvisor

  • the ce call site in the (1455, 1520) window at posture warn, wired shc-side in daemon/wiring/. EnforceNodeCapacity re-signed to take the commitment; gates 2 and 3 adopted unchanged; EnforceNodeHeadroom added once §9.3 is ruled.

Step 4: the node-addressed free-space read. FreeBytesOn(ctx, node, role) served from the fan-in, and the move’s staging preflight extended to check the target as well as the executor (§4.3). This is the step the move lane is actually waiting on.

Step 5: phase 2, not required for admission: the durable digest. One row per node per minute: EWMA CPU/memory, free bytes per role, measured_at. This is what a planner reads across restarts and across time; no gate needs it.

Deliberately not in the minimum: anti-affinity, spread, per-node ratios, a replicated per-tick utilization table, and a scheduler.


7. Rejected alternatives, with their costs

Section titled “7. Rejected alternatives, with their costs”
  1. A materialised node_commitment table, charged at install and released at uninstall. Rejected; §2.5. It is usage on a second axis, and usage needed a measured race fix, a reconcile cron and a repair verb to stay honest. Cost of rejecting: every read is a join rather than a lookup. Immaterial at 33 deployments; revisit at four figures.
  2. Gossip. Rejected. There is already an mTLS mesh with a working peer client and a working fan-out; adding an eventually-consistent membership protocol to distribute a fact that expires in 20 seconds buys nothing the 2s pull does not, and costs a partition-time divergence mode in a product whose linearizable substrate is already present. Cost of rejecting: the pull is O(N²); every node polls every peer every 2s, because LeaderOnly() is false by choice. At 3–10 nodes that is nothing. At 100 nodes it is 10,000 requests per 2 seconds and the fan-in must become leader-only. The flag to do that already exists; the threshold at which to flip it does not, and should be measured before any estate approaches it.
  3. Read utilization back out of the OTel pipeline. Rejected. The right numbers with the right labels are already exported (§0.3), but B9 removed the scrape endpoint deliberately, and making placement depend on an external TSDB makes placement unavailable whenever telemetry is, on a product whose premise is self-hosting without external dependencies. Cost of rejecting: we recompute in-process what is already computed for export.
  4. Ask every node synchronously at decision time. Rejected. It turns a placement decision into an N-way network fan-out on the install path, with the slowest node setting the latency and an unreachable node forcing a policy choice under time pressure. Cost of rejecting: decisions run on data up to 20 seconds old, which is exactly why the utilization axis is advisory at install and gating only at rebalance (§3.4).
  5. Wire the three existing gates as-is at refuse. Rejected; option A of the gates decision, and it stays rejected for the reason measured there plus one this page adds: against TOTAL they answer the wrong question, and a gate that answers the wrong question correctly is worse than no gate, because it gets switched off and takes the right question’s answer with it.
  6. Delete the three gates. Rejected. EnforceLargestReplicaFitsNode has no substitute and no other detector.
  7. A node-scoped reservation ratio. Rejected, and not by this page; the owner ruled it out 2026-08-03 (INHERITED, memory reservation_ratio_ruling): node is an infrastructure axis, not a config scope, and a per-node ratio papers over a wrongly-sized limit. The per-node concern is real and belongs in this substrate’s headroom knob (§5.4), which is a property of the node, not of the tenant’s derivation.

Stated plainly, because each of these will otherwise be discovered by a consumer that assumed otherwise.

  • It cannot tell a declared-zero reservation from an undeclared one; §2.6. Mitigated by counting them, not solved.
  • It cannot split a stopped multi-node swarm deployment’s reservation across its nodes. The anchor floor covers whether; the per-node split needs the container table, and the container table is reaped 120s after the containers stop. Named, not solved. In practice it does not bite today: INHERITED, both estates are 100% compose-runtime with zero swarm-runtime stacks, and no swarm-runtime install has been verified end to end.
  • It has no memory. Every figure is either instantaneous or intent-derived. “Was this node hot yesterday” is unanswerable until step 5, and the first rebalancer built on this will thrash if it is written before step 5 lands.
  • ★ It measures SHC-managed load only. A hand-run container, a co-tenant process, the operator’s own compose file, all invisible to committed and visible to utilization. That is a good reason the two axes are not interchangeable and a better reason the utilization axis has to exist at all: the ledger knows what SHC promised, and only the measurement knows what the machine is actually doing.
  • It cannot make swarm honour any of it beyond what swarm already reads. SHC does write deploy.resources.reservations (§2.1), and swarm treats both keys as hard admission constraints, so the substrate and swarm’s scheduler would at least be reading the same numbers. But INHERITED (the two docs/architecture/cluster/ pages, “landed understanding, not proposals”): swarm is a second-class runtime, the fabric is the strategic multi-node path, the fabric is a data plane and not a scheduler, and the two are mutually exclusive at shc init with no in-place flip. This substrate is deliberately runtime-neutral for that reason: everything in §2 and §3 is computed from SHC’s own tables and SHC’s own mesh, and none of it requires swarm to exist.
  • It does not decide anything. No scoring, no spread, no anti-affinity, no concurrency cap, no never-move annotation, no kill switch. Those are the prerequisites the owner named, and they are a different lane’s list.

9.1: the reservation multiplier (§0.4). Blocks: the meaning of every number this substrate produces. Shipped is reservation = quota × 0.25, limit = quota × 1.0. The owner’s stated model is reservation = quota × 1.0, limit = quota × 1.5. The commitment ledger is correct under either, but a node’s committed sum differs by 4× between them, which decides whether any capacity gate ever fires and what a headroom number means. This is not a quota question (the charge is the quota either way); it is a capacity question, and it should be answered before step 3.

9.2: undeclared reservations (§2.6). service.cpu_request_millicores is Default(0) NOT NULL, so “declared zero” and “row predates the column” are indistinguishable. Backfill from the render, make the column nillable, or accept the ambiguity and gate on the UndeclaredServices fraction. Interim behaviour is proposed (count and report); the durable answer is owed.

9.3: the system headroom allowance (§5.4). How much CPU and memory a node reserves for the daemon, rqlite, docker and the OS before any deployment is counted. ★ Not answerable from a desk: it needs the daemon’s own footprint measured on both estates first. PROPOSED placeholder 1000m / 2 GiB, explicitly unmeasured.

9.4: the admission posture (carried, unchanged). off | warn | refuse for deploy.capacity_admission. The gates decision recommends D-then-C-at-warn and that recommendation is unchanged by this page. What this page adds is that disk deserves a different answer from CPU and memory (§4.3): ENOSPC is an outage rather than a slowdown, the incident that motivated the existing free-space preflight was exactly that, and disk is the one axis where refuse is defensible on a first roll.


  • docs/decisions/2026-08-04-physical-capacity-gates.md: the three gates, their operand, their unknown-handling fix, and the posture question this page inherits
  • shc modules/quota/enforcement.go: the three gates and the NodeCapacity unknown predicates
  • ce modules/app/quota_seam.go: the seam shape §5 copies
  • ce modules/app/resource_quota.go, resource_reservations.go: where the operand comes from and what the runtimes do with it
  • ce ent/schema/deployment_node.go: the occupancy set and the prune rule §2.4 reasons from
  • ce modules/clone/preflight_space.go: the free-space preflight §4 reuses rather than duplicates
  • ce modules/metrics/collectors/peer_nodes.go, ce subsystems/impl/peer_node_stats_adapter.go: the fan-in §3 promotes
  • ce core/hostfacts: the “unmeasured is a value, not a zero” package §3.2 extends to the second payload
  • docs/architecture/cluster/swarm-quorum-and-health.md, swarm-contention-map.md: the runtime fork this substrate stays neutral across