shc update must say what it did
Status: built; this note is still its only description. Implemented (ce
fix/update-noop-honesty, extended by §8 in celane/three-honesty-defects), except §5 which is a recommendation with a measurement behind it.
Origin: two live measurements on the night of 2026-08-10/11, on two estates and two different apps.
| estate | command | printed | actually happened |
|---|---|---|---|
| promanager | shc update traefik -t master | updated stack "traefik", exit 0 | container unchanged at 15h uptime; rendered compose byte-identical; the release’s traefik plugin never reached the node |
| bitspur | shc update windmill -t corp -e prod | updated stack "windmill", exit 0 | containers unchanged at 42h uptime; moved only when handed 1.784.0+0 explicitly, and only after shc repo sync shc (index was stale, 769 → 784 apps) |
Two lanes spent a night on the gap between those two columns.
1. The mechanism
Section titled “1. The mechanism”Three distinct defects share one symptom. They are listed in the order the command hits them.
1a. A bare update replays the deployment’s pin (correctly) and never says so
Section titled “1a. A bare update replays the deployment’s pin (correctly) and never says so”shc update <stack> with no version is defined as “re-render this
deployment as it is configured”. resolveUpdateAppVersion
(ce/modules/app/system_app_pin.go) replays deployment.app_version, and
ExecuteCompose re-renders that same package. When nothing else moved, the
compose comes out byte-identical and docker compose up recreates nothing.
That is correct behaviour. The defect is that the command cannot tell you
it happened. The completion payload carried {"success": true}, which
means no error was raised, and the CLI’s result sentence
(ce/modules/app/commands/operation_sse.go) printed the past tense of the
verb the operator typed. A no-op and a successful upgrade were byte-identical
on stdout and in the exit code.
This is the whole of the bitspur windmill case.
1b. A version-string edit strands every deployment pinned to the old string
Section titled “1b. A version-string edit strands every deployment pinned to the old string”resolveAppDirVersionedProv (ce/modules/app/source_qualifier.go) resolves a
pinned version to the node’s shipped tree only on an exact string match:
if dir := firstStaticAppDir(appName, baseDir, stateHome); dir != "" && declaredAppVersion(dir) == version { return dir, ResolvedProvenance{}, nil}ce/builtin/traefik/app.yaml declared version: "3.6.6" from 2026-07-26
until 2026-08-08, when commit 65eb0e7 corrected it to "3.6.25", a
manifest fix, because no build of the package had ever shipped a 3.6.6
Traefik. Every system traefik installed before that date carries the pin
3.6.6.
From 65eb0e7 onward the match fails, resolution leaves the shipped tree, and
the deb’s drop is never read again by that deployment, however many
releases ship. shc update --check reports CURRENT because both halves of
its comparison resolve the same non-deb package, and the deb’s drop being
correct on disk is not a contradiction of that: it is the diagnosis.
The class is general. Any catalogue app_version string edit does this to
every deployment pinned to the old string.
1c. A pin the catalogue cannot serve was silently substituted
Section titled “1c. A pin the catalogue cannot serve was silently substituted”The last branch of resolveAppDirVersionedProv (“not in the catalog at
all”) delegated to the unversioned resolver, which walks the static roots
and returns the shipped tree. Read the reachability: the exact-match
short-circuit above it already returned when the shipped tree declares the
pin, so this branch can only be entered when the shipped tree declares
something else. Its one possible effect was therefore to render a version
the operator had pinned away from, under the pin’s name, writing nothing
anywhere. The comment justified it with the daemonless-CLI case, which is
unreachable here; a nil CatalogAppDirResolver short-circuits at line one.
2. The honesty fix (the one that matters most)
Section titled “2. The honesty fix (the one that matters most)”A verb must report its effect, not its attempt.
ce/modules/app/deploy_effect.go records what a render actually did, and
ComposeRunOptions.EffectOut carries it out of ExecuteCompose:
| fact | taken |
|---|---|
AppDir | the package directory the render actually read |
PriorPackageDigest / PackageDigest | apppkg identity before / after |
PriorComposeSHA256 / ComposeSHA256 | rendered compose bytes before / after |
Verdict() is tri-state (changed / no-op / unknown) and routes
the digest comparison through apppkg.Comparable, which refuses absence.
Two empty digests compare equal, so the natural spelling of this comparison
would report a confident “nothing changed” precisely when the render knew the
least. That is the same anti-absence rule package_identity.go already
enforces, applied to a second comparison.
Both facts are compared, because either alone lies: same package + different vars is a real change the package digest misses; a doc-only package edit is not a change to the running stack.
The verdict rides the operation’s terminal frame and the audit row, and the CLI prints it, in the live TTY view too, where there used to be no result sentence at all:
traefik: NO CHANGE: nothing was updated; the re-render read the same apppackage and produced the same compose the deployment already had; packageapppkg-v1:5dd22040… read from /usr/lib/shc/apps/traefik. A bare updatereplays the deployment's own version pin, so if you meant to take a NEWERpackage, name the version (`shc update <stack> <version>`); if the versionyou want is not offered, this node's catalogue index may be stale;`shc repo sync` is read-only and re-renders nothing.The exit code is unchanged. A no-op is not a failure, and converge-shaped callers (terraform, the provider converge, CI) must keep working. What changed is that the command says what it did.
An absent verdict (a daemon predating this change) keeps the old sentence. Reading absence as “no-op” would print “nothing was updated” for every update against an older node, the same anti-absence rule at the wire.
Failed resolution now fails loudly
Section titled “Failed resolution now fails loudly”X409042APPVSUB replaces the silent substitution of §1c: a pin the catalogue
cannot serve, on a node shipping a different version, is refused naming
both versions, with both remedies (shc repo sync first; it is read-only).
3. The resolution fix: a system app re-resolves to the deb’s drop
Section titled “3. The resolution fix: a system app re-resolves to the deb’s drop”applySystemAppVersionRefresh (ce/modules/app/system_app_pin.go) is
applySystemAppVersionPin’s update-lane twin. On a bare update of a
canonical system stack it moves the pin onto the version this node’s own
package ships, announced by X200972SYSDEB naming both versions.
This honours the [[system-pin-ruling]] rather than reversing it. The ruling’s stated purpose is that a re-render must not silently start a new-major server against an old-major data directory. It is a rule about silence and about the data, not about freezing a string:
- it is not silent:
X200972SYSDEBnames both versions before the render; - it is not the catalogue head: the version taken is
ShippedAppVersion, static roots only, no network: the artifactapt install shcjust placed, which is what every system seam is built against; - the data-directory guard is untouched and still upstream of any harm;
this lane fires
pre_update, which is wherepostgres-guardrefuses; - it is the system stacks only.
IsCanonicalSystemStack’s four clauses keep an operator’s ownshc install postgres -t master --name mydbuntouched, as the standing unpinned means unpinned, no auto-pin ever ruling requires. An unpinned system stack also stays unpinned; the newest selector already walks the static roots first, so it already lands on the deb’s drop and a new auto-pin would have no defect to justify it.
Leaving the pin frozen is not the neutral option. It strands the control plane on a package the node no longer carries, reports that as CURRENT, and makes every future platform fix hollow, which is the outcome the ruling exists to prevent, arrived at from the other side.
4. shc update --check now names the tree it read
Section titled “4. shc update --check now names the tree it read”PackageDrift.ResolvedFrom carries the directory the catalogue half
resolved, and --check prints it on every verdict:
traefik: app package is CURRENT; `shc update` would re-render the same package app: traefik 3.6.25 running: apppkg-v1:5dd22040… catalogue: apppkg-v1:5dd22040… read from: /usr/lib/shc/apps/traefik“CURRENT” and “the deb dropped new bytes” read as a contradiction without that line, and as a diagnosis with it. It is one line and it is the single measurement that would have ended the promanager investigation immediately.
5. Should a bare shc update <app> imply a catalog sync? No. Recommended instead: report, don’t fetch.
Section titled “5. Should a bare shc update <app> imply a catalog sync? No. Recommended instead: report, don’t fetch.”The cost, measured
Section titled “The cost, measured”| measurement | |
|---|---|
public index (selfhostedcloud.org/apps/index.yaml) | 744 KB, 90 apps, ~1.2 s fetch (3 runs: 1257 / 1229 / 1169 ms from a warm box) |
a real sync | phase 1 of syncHTTPIndexRepo calls fetchHeadArchives(ctx, &m, repoURL, idx.Apps); it downloads every app’s head .app archive before writing a row |
| DB cost | phase 2 issues applicationDeletes(repoID) and reinserts the repo’s entire application row set; on the estates that is an rqlite cluster write of ~800 rows |
So the index fetch is the cheap part and it is not the operation. A sync is 90 archive downloads plus a full catalogue rewrite, per update.
The correctness argument, which is the stronger one
Section titled “The correctness argument, which is the stronger one”A bare shc update <stack> means “re-render this deployment as it is
configured”. An implicit sync would make the same command mean different
things depending on what the index published in the last minute. For an
unpinned deployment it would literally mean “take whatever head is now”,
the silent acquisition the codebase already refuses to do quietly
(X200971UPDHED exists to warn about exactly that). Making a mutation’s
outcome depend on a network fetch the operator did not ask for is a larger
footgun than the stale index, and it is one that fires on production data.
What to do instead
Section titled “What to do instead”- Say the no-op out loud: done (§2). The bitspur operator’s actual
problem was not the stale index; it was that the command told them
nothing. The no-op sentence now names both remedies, and names
shc repo syncas read-only so it is safe to reach for first. shc update --checkresolves the catalogue without mutating and now names the directory (§4). That is the read-only “what would happen” verb, and it is the right place to spend the network.- Optional follow-up, not built here: an opt-in
shc update --syncand a per-repo index-age warning on the update lane (the repo row already carriesSyncedAt; the seam needed is aRepoSyncAgelookup insidemodules/app). Recommended as a small, separate change; it is additive and does not alter what a bare update means.
6. Tests
Section titled “6. Tests”ce/modules/app/update_says_what_it_did_test.go and
ce/modules/app/commands/operation_change_line_test.go. Every assertion
names the mutation it dies on; all were run against a deliberately broken
tree:
| mutation | reds |
|---|---|
Verdict compares digests with == instead of apppkg.Comparable | 4 rows of TestTheVerdictRefusesAbsence (absence became agreement) |
| the router drops the effect fields from the terminal frame | all 3 rows of TestANoOpUpdateIsReportedAsANoOp |
the silent delegation is restored in resolveAppDirVersionedProv | TestAPinTheCatalogueCannotServeIsRefusedNotSubstituted |
the applySystemAppVersionRefresh call is deleted from Service.update | TestTheUpdateDoorWritesTheRefreshedPin |
the refresh’s IsCanonicalSystemStack clause is dropped | 2 rows of TestASystemStackRepinsToTheDebsDrop (a user’s deployment gets auto-moved) |
the CLI’s no-op arm is removed | TestTheResultSentenceNamesTheEffectNotTheAttempt, TestTheSentenceFallsBackToAStackNameItAlwaysHas |
the CLI folds unknown into the no-op arm | TestTheResultSentenceNamesTheEffectNotTheAttempt |
the CLI reads an absent verdict as no-op | 2 rows of the same |
the prior compose hash is taken after WriteRenderedCompose | TestTheBeforeHalfIsTakenBeforeTheOverwrite |
The last one is a source-level ordering guard on purpose: ExecuteCompose
needs a docker daemon, so no unit test in the package can drive it, and
reading the prior hash after the overwrite would make every update report
no-op, silently. A guard that only fires under a docker daemon is a guard
that never fires.
7. What is still open
Section titled “7. What is still open”- The promanager traefik deployment’s actual state. §1b is the mechanism
the evidence fits and it is now closed in code, but the estate-side
confirmation (which directory that node’s daemon resolved) was not taken
by this lane. With §4 shipped it is one command:
shc update traefik --check -t masternow printsread from:. system.apps.<app>.vars.images.*config rows destroy the persisted deployment var on daemon-driven recreates. A hand-set image override on promanager’s traefik would pin the container image regardless of which package resolves, and would show up here aschangedon the package with no change to the running container. Out of scope for this branch; it is a separate defect on the same estate.
8. The third fact: a var that feeds a job (2026-08-15)
Section titled “8. The third fact: a var that feeds a job (2026-08-15)”Status: implemented (ce lane/three-honesty-defects). This EXTENDS §2; it
is the same doctrine reaching a change the two original facts are structurally
blind to.
The measurement
Section titled “The measurement”shc update mailcow --var ssoOnly=truemailcow: NO CHANGE: nothing was updated; the re-render read the same apppackage and produced the same compose the deployment already had …exit 0, and the deployment’s manifest came out carrying ssoOnly: "true".
Why the two facts of §2 could not see it
Section titled “Why the two facts of §2 could not see it”ssoOnly feeds the oidc consumer plug’s jobs.post_deployment env and
nothing else (apps/flagship/mailcow/oidc.plug.yaml). It never reaches
compose.yaml, and it does not change which app package is read. So both
halves of the §2 verdict (package identity and compose bytes) are equal by
construction on the very run the operator cares about, and “both known, both
unchanged” fired.
This is the doctrine’s own anti-absence rule turned up one level. §2 refuses to read a missing comparison as agreement. Here both comparisons are present and both agree, and the verdict is still false, because the state that moved is in neither of them. A verdict that reads only the compose is a verdict about docker, not about the deployment.
The result was worse than the silence §2 replaced: the deployment was DECLARED ARMED and REPORTED UNARMED. The operator re-types the command or gives up, while the persisted override means a later, unrelated reconcile arms the flag unattended.
The fix
Section titled “The fix”DeployEffect.ChangedVarKeys is the third fact, recorded at the ONE line
where both sets exist: install_runner.go’s
MergeVarOverrides(priorState.VarOverrides, …) seam, immediately after the
merge (before it, opts.VarOverrides is still the request’s set and every
replayed override reads as moved). KEYS, not values: the list rides the wire
to the terminal, the audit row and -o json.
Verdict() consults it FIRST, ahead of both existing halves, because it is
positive evidence. The two halves below it are comparisons that can come
out UNKNOWN; a recorded var move is a fact this render established about
state it just persisted. Ordering it after them would ask the two facts that
cannot see the change to rule on it.
A bare update replays the persisted set unchanged, so nothing is recorded and the §2 no-op sentence keeps firing for the run that genuinely did nothing.
The sentence, because “CHANGED” alone is the other wrong answer
Section titled “The sentence, because “CHANGED” alone is the other wrong answer”An operator handed a bare updated stack "mailcow" goes looking for the
container recreate, and, finding none, reaches the same conclusion the false
NO CHANGE produced. So the var-only move is the one CHANGED run that carries
a reason:
mailcow: CHANGED: var ssoOnly moved and the deployment now carries it. Theapp package and the compose bytes are UNCHANGED; this var reaches thedeployment through its integration jobs, not through compose, so no containerwas recreated and none needed to be. If the reconcile that consumes itfailed, that failure is named on this same lineThe CLI arm fires on the reason’s PRESENCE, not on a second flag: an ordinary changed run sends no reason and keeps its existing sentence, and a daemon predating the reason keeps the old sentence too, the same anti-absence rule at the output surface. The last clause is not decoration: §2’s failed-job names ride this same frame, so the two halves of “what did this run do” print together.
The exit code is unchanged, as in §2.
And the reconcile is NOT short-circuited
Section titled “And the reconcile is NOT short-circuited”The obvious optimisation (“the compose is identical, so skip the consumer
plug reconcile”) is exactly what would make a var-only change a
declared-armed, never-armed deployment. ExecuteCompose therefore never
branches on its own change verdict: it RECORDS the effect and runs the
reconcile unconditionally, and judging what that adds up to belongs to the
surface that reports it. TestTheReconcileIsNotGatedOnTheChangeVerdict is a
source-level guard on precisely that (same reasoning as §6’s last row; the
function needs a docker daemon, so a runtime guard here is a guard that never
fires).
ce/modules/app/var_only_change_is_a_change_test.go and the new row in
ce/modules/app/commands/operation_change_line_test.go.
| mutation | reds |
|---|---|
the len(e.ChangedVarKeys) > 0 arm is deleted from Verdict | TestAVarOnlyChangeIsAChangeNotANoOp, TestTheMovedVarKeysRideTheWire |
RecordVarOverrideMoves drops its equality skip | 4 rows of TestRecordVarOverrideMovesNamesOnlyWhatMoved (a bare update’s replay becomes a change) |
| the sort is dropped | the multi-key row of the same |
WireFields drops the changed_var_keys fold | TestTheMovedVarKeysRideTheWire |
WireFields returns the effect’s own slice | the aliasing assertion of the same |
| the record is moved ABOVE the merge | TestTheVarMovesAreRecordedAtTheMergeSeam |
the reconcile is wrapped in if …Verdict() != EffectNoOp | TestTheReconcileIsNotGatedOnTheChangeVerdict |
the CLI’s changed arm is removed | TestTheChangedSentenceNamesTheVarThatMoved |
that arm’s reason == "" guard is dropped | TestTheResultSentenceNamesTheEffectNotTheAttempt, TestTheChangedSentenceNamesTheVarThatMoved |
The two sibling defects reported with this one, both already closed
Section titled “The two sibling defects reported with this one, both already closed”Named here so the next reader does not re-open them:
- A failed integration job swallowed by a green update: closed by §2’s
FailedIntegrationJobs(deploy_effect.go), the CLI’soperationJobFailureLine, andoperationJobFailureError, which makes the exit code non-zero. That last part is the deliberate asymmetry: a no-op is not a failure, an unconverged edge is. On the update door only; see §9. shc adapter migrate --from <tenant>/<stack>@<env>ignoring the@env: closed ince/modules/adapter/commands/migrate_from.go:stackSourcecuts the@envsuffix and threads that environment through toPause, and the CLI pauser routes throughnode.ComposePauseVerified, which reports an EMPTY container set as failure rather than the exit-0docker compose pausegives it. Covered byTestStackSourceEnvSuffixAndFallbackandTestComposePauseVerifiedFailsWhenTheProjectHasNoContainers.
9. The other door: shc install (2026-08-16)
Section titled “9. The other door: shc install (2026-08-16)”Status: implemented (ce lane/install-door-says-what-it-did). This EXTENDS
§2 the way §8 did, same doctrine, same fields, the door §2 did not reach.
The measurement is §8’s own closing note, re-read
Section titled “The measurement is §8’s own closing note, re-read”§8 records the failed-integration-job swallow as closed. Verified by content, it was closed on the update door only:
| door | builds a DeployEffect | terminal frame carries the failed jobs | exit code |
|---|---|---|---|
shc update | RunUpdateWithObserver → (*DeployEffect, error) | effect.WireFields() folded in updateHandler | non-zero |
shc install | none | nothing | 0 |
RunInstallWithObserver returned a bare error and built no effect, so
ExecuteCompose ran the entire consumer-plug reconcile with
opts.EffectOut == nil. RecordFailedIntegrationJob is nil-safe by design
(the daemonless and test paths carry no effect) so every failure on the
install door was recorded onto that nil and discarded without a word. The
terminal frame was a bare {"success": true, "stack": …, "environment": …},
and the CLI printed installed stack "x" and exited 0.
The CLI half needed nothing: verbNounOr already maps "installed" →
"install", and operationJobFailureLine / operationJobFailureError
already fire off failed_integration_jobs regardless of verb. The frame
was the only thing missing.
Why this door is the worse one to lose
Section titled “Why this door is the worse one to lose”An update over a failed reconcile leaves the previous working configuration
in place; the estate keeps whatever it had. A first install over a
failed reconcile leaves an app that has never had its OIDC client secret
written at all, and reports it as installed. It is the bitspur
frappe-set-social-login-key shape with no prior good state to mask it.
The fix
Section titled “The fix”RunInstallWithObserver now returns (*DeployEffect, error), allocating the
effect up front and returning it on every path, mirroring
RunUpdateWithObserver, for the reason that function already states. The
dispatched-elsewhere path returned an empty effect: the work happened on
another node, and claiming this one’s silence as a clean bill would be the
anti-absence rule broken across a network hop. Since SHC-2426 it returns
the effect that node answered, and the empty one only when the answer
carried none.
RunInstall is the discarding wrapper, exactly as RunUpdate is for the
update lane, and carries the same warning. It routes through the
InstallDeploy package var rather than calling the runner directly, so the
placement lane’s test seam still describes a call that happens.
JobFailureWireFields, and why the install door does not take WireFields
Section titled “JobFailureWireFields, and why the install door does not take WireFields”The install door folds only the failed-job half. An install has no prior
package digest and no prior compose by construction, so Verdict() is
unknown on every successful install, and folding the whole projection
would print
CHANGE UNKNOWN: this run cannot say whether anything was installedafter every shc install on the estate. That is the anti-absence rule
pointed at the wrong absence: it exists for a measurement a run FAILED to
take, and a missing prior on an install is not that; it is what the word
means. The failed-job fact carries no such assumption (a job either failed
or it did not, on either door), so it is the half that crosses.
It adds no key when nothing failed, because an install that reconciled
cleanly must send no opinion rather than an empty claim, the same rule
operationJobFailureLine reads at the far end, where an absent key and an
empty list both mean “named no failed job”. An empty list here would make
every clean install exit non-zero.
Both install doors take it: the stack door (installHandler) and the
vdeployment door (installVdeploymentAsync), which binds its own plug edges
and can fail the same way.
success: true is unchanged on every arm, including the failed one. The
stack really is up; the asymmetry lives in the CLI’s exit code, taken off the
job names, not in that flag.
ce/modules/app/install_door_reports_failed_jobs_test.go.
| mutation | reds |
|---|---|
the JobFailureWireFields fold is dropped from installHandler | TestTheInstallDoorReportsAFailedIntegrationJob (frame returns to the bare map[environment stack success]) |
EffectOut: effect is deleted from RunInstallWithObserver | TestTheInstallDoorBuildsAnEffectToRecordOnto |
JobFailureWireFields returns e.WireFields() | TestJobFailureWireFieldsCarriesOnlyTheFailure (a clean install starts sending changed:unknown) |
the len(e.FailedIntegrationJobs) == 0 clause is dropped | TestJobFailureWireFieldsCarriesOnlyTheFailure and the clean-install row of TestTheInstallDoorReportsAFailedIntegrationJob |
TestTheInstallDoorBuildsAnEffectToRecordOnto is source-level for §6’s
stated reason (ExecuteCompose needs a docker daemon) and it windows the
scan to RunInstallWithObserver’s own body: the update lane’s
EffectOut: effect is in the same file, so an anchor that merely proved
“EffectOut appears somewhere” would stay green with the install door blind.
What is still open
Section titled “What is still open”- CLOSED by SHC-2426: a placement-DISPATCHED install reported green over a
failed reconcile. Every install typed off the leader is dispatched, so
this was every follower’s
shc install, and it lost theexposure:line and the kept groups the same way. The receiving node now runsInstallDeployand puts its effect in its answer (app.HolderEffectReply), and the dispatching door returns that effect with the holder’s settle (app.ReportHolderEffect), as the update door always has.InstallDispatcherstill answers(dispatched bool, err error).