Backup engine convergence: the two engines are wrong in opposite directions
Status. Half landed. BOTH HALVES LANDED, and the second ships in
v0.0.19. The encrypted engine gained its freeze bracket in v0.0.18 (ce
modules/backup/service.go, test modules/backup/restic_quiesce_test.go).
The scheduler’s destination converged in ce de396a7, pinned for the
v0.0.19 cut, but not by the re-route this document deferred, and §3b is
kept below because its measurements are why.
★ THE DESTINATION CONVERGED WITHOUT THE ENGINE CONVERGING. The capture
stays orchestrated: same phases, same freeze, same escalation to a STOP,
byte-identical .bak. FINALIZE then seals that .bak into backup.repo
under backup.password, verifies it readable there at exactly its
on-disk size, and only then removes the local plaintext; the row becomes
restic:<repo>:<snapshot>. That answers all three §3b blockers without
paying any of their costs: the two “captures strictly less” payload gaps and
the pre-uninstall hard-refuse simply never arise, because the engine that
captures is the one that always did. §3c step 2 (“teach the encrypted engine
the payloads it is missing”) is therefore not owed for this outcome; it is
only owed if step 4’s engine deletion is ever pursued.
★ AND THE RETENTION LEAK WAS NOT A BLOCKER. IT WAS A LIVE DEFECT. §3b
frames it as a consequence of re-routing. It was already happening: Apply
deleted the SQL row whether or not the bytes went away, so every restic:
row shc backup create produced and retention expired had its os.Remove
fail, its error swallowed, and its row deleted anyway. Fixed in the same ce
commit (§3c step 3, in full: the ref-carrying ArchiveDeleter, all three
daemon wiring sites, keep-the-row-on-failure, the restic-location refusal in
the default deleter, BytesFreed from what was freed, one prune per sweep,
and the backup_cache twin).
What is still owed is the live pause measurement (§4).
Baselines. ce origin/main 8ab2081, shc origin/main d934af851, shc’s
ce pin v0.0.18-0.20260804002903-5a52e9042337. Everything below was read at
those refs, not from a working checkout.
1. What each engine actually does
Section titled “1. What each engine actually does”There are three capture paths, not two. The third is daemonless and is included because it is reachable from the same verb.
| restic-direct | orchestrated | daemonless tar | |
|---|---|---|---|
| entry point | Service.CreateBackup (modules/backup/service.go:937) | Service.CreateOrchestratedBackup → createWithID → phases.go | RunCLICreate (modules/backup/cli_runner.go:94) |
| quiesce bracket | now yes (this change): bracketPause held across restic backup | yes: PREPARE → QUIESCE → CAPTURE → RESUME → FINALIZE, resume() deferred | no |
| pause escalation to STOP | no (pause_hold.go is orchestrated-only) | yes | no |
| destination | backup.repo, resolved per-deployment through EnsureResticConfigured (scope tenant/stack/env) | PathsResolver.BackupsDir(tenant, stack) = {state_home}/backups/<tenant>/<stack>/. Not configurable, and backup.repo cannot redirect it; TRUE at the baseline, FALSE at the v0.0.19 pin: the capture still lands there, then FINALIZE seals it into backup.repo and removes the local copy (see Status) | {state_home}/backups/... sidecar tar |
| encryption | yes: every blob under backup.password; row Location = restic:<repo>:<snap> | no: !<arch> AR container, archive.go’s only crypto import is sha256. The one sealed member is the manifest’s inline Fernet EncryptedBundle (vault secrets) | no |
| key gate | refuses without backup.password (X500414RSTPAS) and without backup.repo (X500415RSTBKN) | none: constructs no *Restic, reads no key | none |
| manifest | full BackupManifest staged and captured inside the snapshot (docs/proposals/backup-manifest-in-repo.md); local row copy adds Checksum | full BackupManifest written in FINALIZE, checksummed, packed as manifest.json member of the .bak | thin state.json-class sidecar, not a BackupManifest (group_cli.go says so) |
| captures | the requested Paths (default: the deployment dir), minus backup.yaml exclude.mounts | per-mount tars plus host-bind sources outside the deployment dir, plus unbound named volumes (swarm), plus app converters (pg_dumpall) | compose named volumes only |
| single-flight lease | no (LiveBackupLockHolder is called only from beginStackBackup) | yes | no |
| retention | defaultArchiveDeleter os.Removes the Location → fails, row deleted anyway ⇒ snapshot orphaned in the repo. TRUE at the baseline (this was a live defect, not a hypothetical) and FIXED at the v0.0.19 pin | os.Remove on a filesystem path ⇒ works | works |
Which trigger reaches which engine
Section titled “Which trigger reaches which engine”restic-direct: shc backup create with a wired in-process Service
(register.go:288); shc backup create over the wire (router.go:253, the
public POST /api/method/shc.backup.create); the async create endpoint
(router.go:322); shc backup create --file (register.go:467 →
createFileViaService / createFileViaDaemon); --all-connected /
--with-dependants from the CLI (CreateConnectedGroupForStack →
CreateConnectedBackupGroup → per-member CreateBackup, grouping.go:286);
the backup mount overlay’s snapshot (mount/overlay.go:856).
orchestrated: the backup schedule cron
(subsystems/impl/backup_scheduler.go:290, via ExecuteBackupScope); the
pre-uninstall safety gate (daemon/services_backup_config.go:210, wired as
app.SetPreUninstallBackup); the pre-restore rollback snapshot
(rollback.go:69); restore reentry (restore.go:337); clone / move
(daemon/services_seams_clone.go:51; clone.BackupProvider.CreateBackup is a
seam name, its adapter calls CreateOrchestratedBackup); the group driver
CreateBackupGroupFromRequest (grouping.go:89 → beginStackBackup).
CreateBackupAsync (orchestrator.go:76) has no production caller.
daemonless tar: shc backup create and create --file with neither a
wired Service nor a configured daemon endpoint (register.go:266,339).
That shc has no vault, and since 2026-10-19 no mint store on the disk
either: a vault ref it meets is refused (X404905SECREQ, no vault
reachable), and the credential --var heal fails closed there, so its
capture never holds a secret it minted or sealed itself.
Two consequences of the destination split, both verified live tonight
Section titled “Two consequences of the destination split, both verified live tonight”backup.repois configured and the orchestrated archives ignore it. On promanager,backup.repo=s3:https://sos-de-fra-1.exo.io/shc-backup-promanager-hel1, and/var/lib/shc/backups/holds 4.bakfiles totalling 658 MB. So the framing “a schedule would write plaintext credentials into S3” is not what happens. It is worse for durability and better for confidentiality than that: the orchestrated engine cannot reach S3 at all, so a schedule writes plaintext to the leader’s local disk, next to the data it exists to survive the loss of.warnArchiveMissedRepo/X200079BKPLOCsays so once per archive, at WARN.- The plaintext is plaintext. Read read-only off promanager with no key:
head -c 600of/var/lib/shc/backups/default/s3front/20260729T180234Z_7f3a3f9feb60cf27.bakis!<arch>\n,manifest.json/, then the manifest in the clear:tenant,stack,app,node, the full stack record. The manifest also carries install-time--varvalues as bare strings.
The unreachable skip in the uninstall gate
Section titled “The unreachable skip in the uninstall gate”app.SetPreUninstallBackup treats X500415RSTBKN (“no backup repository
configured”) as a clean skip with a loud WARN. That code is raised only from
restic-touching call sites (actions.go:911, service.go:808,1085,
keyrotate.go, mount/subproc/manager.go:194, EnsureResticConfigured at
seams.go:346). CreateOrchestratedBackup reaches none of them. So the branch
is unreachable: on an estate with no repo configured the pre-uninstall backup
still runs and still writes a plaintext local .bak, and the “NO SAFETY BACKUP”
warning never fires. Not a defect in itself (a backup did happen), but the
operator’s mental model of that warning is wrong.
2. The ruling both engines violated
Section titled “2. The ruling both engines violated”Owner, 2026-08-02: backup stops the stack rather than risk an untrusted
archive: “we can never ever have an untrusted backup; I’d actually literally
rather our service go down.” DegradedStackStopped in types.go quotes it.
Before this change:
- the engine that protects the data (orchestrated: freeze, escalation to STOP, host-binds, named volumes, converters) wrote it unencrypted to a path the operator cannot move;
- the engine that encrypts it (restic-direct:
backup.password,backup.repo, per-deployment scope) read the volumes live.
So neither satisfied the ruling, and the practical consequence stands: an owner told to “set a backup schedule” would be scheduling unencrypted archives onto local disk on a cron.
3. The convergence, and what the prior got wrong
Section titled “3. The convergence, and what the prior got wrong”The strong prior going in was: route the scheduler to the encrypted engine, and give the encrypted engine the quiesce bracket. Half of that survives measurement.
3a. LANDED: the encrypted engine gains the freeze
Section titled “3a. LANDED: the encrypted engine gains the freeze”It did not need new machinery. quiesce.go bracketPause was already the shared
primitive: the orchestrated per-stack flow decomposes it into phases, and two
other callers (quiesceToStatic, the routed single-mount RPC CaptureMountTar)
use it whole. CreateBackup now calls the same function. Nothing was
duplicated, and the pause/unpause classification, the fresh-context release, the
ErrQuiesceBroken / ErrStackLeftPaused sentinels and the three-valued outcome
all came along for free.
Shape, in CreateBackup:
pre_backup hooks (stack live) → pre_pause hooks → PAUSE ← bracketPause → stamp manifest → stage manifest → restic backup → UNPAUSE → post_pause hooks ← deferred release→ post_backup hooks- Defer-safe from the first line. The release is armed by
deferin the same statement group that acquires the freeze, so a staging error, a restic failure, a cancelled ctx or a panic all still unpause. This is deliberate coordination with the sibling lane removing the non-deferredrunBackupPostJobsbug from this same function: that block is untouched and sits entirely below the bracket, so the two changes compose. Do not introduce a second copy of the bug being removed. - Only three things run under the freeze: stamp, write the few-KB manifest, run restic. The whole manifest (stack record, vault bundle, deployment config, volume inventory) is still built with the stack live.
- The stamp is split, and the split is forced. The manifest rides inside
the snapshot, so the in-repo copy must be final before restic runs and can
only carry the freeze’s t0 answer. The release’s t1 answer (a container
replaced mid-capture, or one left frozen) is folded into the local row
copy, which is the copy
backup ls/backup inspectproject from and which already diverges by carrying the snapshot id. Folding only ever weakens, so the re-stamp can downgrade a claim and never upgrade one. - The projection is now one function.
applyQuiesceStampinpointintime.gowritesPointInTime+not_quiesced+stack_left_paused, andphases.goFINALIZE was moved onto it. Two engines making the same claim from two hand-written copies is exactly how an archive ends up claiming a point-in-time it never had. - A failed freeze does not abort.
QuiesceFailedstampslive+not_quiescedand proceeds. That matches the rulingErrQuiesceUnsupportedalready encodes (an honest live archive beats no archive) and matches invariant (c), which aborts only in the whole-tar mode. Whether the restic path should instead escalate to a STOP likepause_hold.godoes is an owner call and is listed as owed below.
Proof. modules/backup/restic_quiesce_test.go. The assertion is a
sample, not a call order: the fake pauser keeps a live paused flag and the
fake restic runner reads it at the moment the backup subcommand runs, so a
pause acquired and released around the snapshot fails. Three derived cases:
freeze lands + capture succeeds (frozen during the read, resumed after, stamped
assured); freeze lands + capture fails (resumed anyway); freeze fails +
capture succeeds (release still attempted, stamped live + not_quiesced).
Verified adversarially, not just green:
| mutation | result |
|---|---|
| release only on the success path (the sibling lane’s bug shape) | TestCreateBackupResumesAfterAFailedCapture FAILS |
| release immediately after the pause (not held across the capture) | TestCreateBackupHoldsTheStackFrozenForTheSnapshot FAILS |
pristine origin/main service.go, new test file | all three FAIL |
3b. DEFERRED: the scheduler re-route, on three measured blockers
Section titled “3b. DEFERRED: the scheduler re-route, on three measured blockers”Read this section as the measurement that chose the seal over the re-route, not as pending work. All three blockers below are accurate at the trees named in Baselines, and the first one was accurate for
shc backup createtoo. The scheduler’s destination converged in cede396a7without re-routing the engine, which is why none of these had to be paid; the retention leak was fixed outright rather than worked around. The one line that is now FALSE as a forward-looking claim is “Today the scheduler writes filesystem paths, so retention works”; a sealed schedule writesrestic:rows, and it is the retention fix that makes that safe.
Retention would leak every scheduled archive, forever. The only wired
ArchiveDeleter is retention/service.go’s defaultArchiveDeleter, which does
os.Remove(location). Nothing in the daemon supplies another
(grep 'Archives:' finds only an unrelated struct in
adapter/commands/migrate.go). Apply swallows the delete error and deletes
the SQL row regardless (service.go:178-189, and the comment says so). Today
the scheduler writes filesystem paths, so retention works. Re-route it and every
expired row gets forgotten from the DB while its snapshot stays in the restic
repo: an unbounded, un-enumerable repo, on the estates most likely to have a
schedule. Fixing this is not a one-liner: it needs a restic-aware deleter doing
forget + prune with the key, and prune is a repo-wide lock.
The encrypted engine captures strictly LESS. captureHostBinds,
CaptureNamedVolumes and the converter run exist only in phases.go
FINALIZE. Re-routing the scheduler today would silently stop capturing:
absolute compose bind sources outside the deployment dir; unbound
docker-managed named volumes (which is how a swarm deployment holds its
data); and every app converter artifact (postgres’ pg_dumpall). That is a
data-loss-class regression dressed as a security improvement, and it is the
single strongest argument against the naive re-route. grouping.go:249 already
notes the mirror-image problem in the other direction: the orchestrated tar
hard-errors on container-owned mount data an unprivileged daemon cannot read,
where restic tolerates it.
Restore is already red fleet-wide. X400867RSTEMPT (restore/recover replay
failing, proven pre-existing on origin/main by control experiment, 20+ apps)
means the artifact-shape change a re-route implies cannot be verified end to end
tonight. Changing which engine writes the archive while the read path is broken
is untestable by construction.
Secondary, but real: the restic engine refuses without backup.password
(X500414RSTPAS). Re-routing turns the pre-uninstall safety backup from
always-runs into hard-refuses on any estate with no key; the gate treats a
non-X500415RSTBKN error as fatal, so shc uninstall would start failing.
3c. The convergence design, for the ruling
Section titled “3c. The convergence design, for the ruling”Target: one engine, freezing, encrypting, redirectable. The honest order, smallest first:
- (landed) the encrypted engine freezes, and both engines stamp their claim through one projection.
- Teach the encrypted engine the payloads it is missing (host-binds,
unbound named volumes, converters) by lifting them out of
phases.goFINALIZE into helpers both engines call. This is the real work, and it is pure addition: the orchestrated engine keeps behaving identically. - Wire a restic-aware
ArchiveDeleter(forget+prune, holding the key) and makeretention.Applystop deleting rows whose archive delete failed. Both halves are needed; either alone still leaks. - Then re-route the scheduler, the pre-uninstall gate and the rollback snapshot to the encrypted engine, and delete the orchestrated capture rather than leave two.
- Owner call, independent of the above: should the restic path escalate a
held freeze to a STOP the way
pause_hold.godoes for the orchestrated path? The ruling’s own words point at yes.
3d. Migration concern for the archives that already exist
Section titled “3d. Migration concern for the archives that already exist”Existing plaintext .bak files are not convertible in place: the row’s
Location prefix is the predicate every read path branches on
(IsResticLocation, consumed by restore.go, actions.go export/verify/mount,
service.go delete), so a .bak cannot become a restic: row without being
re-ingested, and ImportBackupFromFile is the only tool that could, per
archive, needing the vault master to re-seal the inline bundle. Whatever step 4
does, it must:
- leave existing filesystem-located rows readable (do not narrow the read branch);
- not silently re-key or move them (a rotate orphans every prior archive and
SHC has no key-add verb; see
backup_encryption_two_engines); - state plainly that pre-cutover archives are unencrypted, which
backup ls’sEncryptioncolumn now does per row (encryption.go).
4. Landed vs deferred
Section titled “4. Landed vs deferred”Landed (ce, this change).
modules/backup/service.go:CreateBackupholds a defer-safebracketPauseacross the restic capture;pre_pause/post_pausebracket hooks fire; stack resolution hoisted so the bracket and the manifest share one lookup; the local row manifest re-stamped with the folded verdict.modules/backup/pointintime.go:applyQuiesceStamp(+appendDegradedOnce), the single projection of aQuiesceVerdictonto a manifest.modules/backup/phases.go: FINALIZE moved ontoapplyQuiesceStamp.modules/backup/restic_quiesce_test.go: the three-case guard, mutation- verified and control-verified against pristineorigin/main.
Landed (ce de396a7, shipping in v0.0.19).
- The orchestrated
.bakis sealed intobackup.repounderbackup.passwordin FINALIZE, verified readable at its exact size, and the local plaintext removed only then. Row location becomesrestic:<repo>:<snapshot>, sobackup lsreports it encrypted because it is. Applies to the scheduler, the pre-uninstall safety capture and the pre-restore rollback snapshot, every trigger §1 lists as orchestrated. - The verify is load-bearing: at the seal the local
.bakis the only copy, so a failed verify KEEPS it, keeps the row’s filesystem location, and forgets the half-trusted snapshot (X200075BKPSEL names the reason, X200079BKPLOC still fires). Sealing is conditional on repo + key and never applies to an explicit--file/ clone-staging output path. backup.reposet with nobackup.passwordis refused atschedule set(X400077BKPSCD); a schedule is the one backup verb nobody watches, and that combination can only ever write plaintext locally. An estate with no repo at all is NOT refused: local plaintext is the only destination it has.- Reads needed no second implementation: a sealed archive IS the
.bak, so it is streamed out of the repo and handed to each reader as a file-backed row (restore, export, rollback, delete/verify/forget).backup mountrefuses and points atbackup export. - Retention no longer orphans: §3c step 3 in full: the ref-carrying
ArchiveDeleter, all three daemon wiring sites, keep-the-row on a failed delete (X500918RETARC + the X500920RETKPT rollup), the default deleter REFUSING a restic location instead of pretending (X500921RETNRS),BytesFreedcounting what was freed, onerestic pruneper sweep rather than per archive, and thebackup_cachetwin swept (X500922RETCAC). - Existing plaintext
.bakrows are untouched and no read branch narrowed, which is §3d’s first requirement.
Still deferred, with the reason.
scheduler / pre-uninstall / rollback re-route: moot: the destination converged via the seal, so the engine never had to move and the payload gaps were never paid.restic-aware: landed above.ArchiveDeleter+retention.Applyrow-delete-on-failure- lifting host-bind / named-volume / converter captures into shared helpers: now owed only for §3c step 4 (deleting the orchestrated capture), not for encrypting a scheduled archive.
- pause-hold escalation on the restic path: owner call.
CreateBackuptaking the single-flight lease the orchestrated path takes.
Owed.
a ce pin bump: DONE. The v0.0.18 release lane bumped the pin to ce852c768(v0.0.18-0.20260804030128-852c768ca9b1), so everything under Landed above ships in v0.0.18 and does change estate behaviour from the roll onward.- a live proof on an estate, after the pin bump:
shc backup createagainst a real deployment, samplingdocker inspectforPausedduring the capture, the same measurement that showed 0 of 300 samples paused before this change. STILL OWED, and it is the load-bearing one: the pre-change measurement is the only pause figure this project has for the restic path, and the window now covers a restic walk rather than the orchestrated engine’s double-pass rsync of a delta, so the 0.60–0.76 s orchestrated numbers must not be read across.
The sibling lane landed too, and the two composed as designed. ce 8111eb3
removed the non-deferred runBackupPostJobs bug this document anticipated at
§3a; the pre_backup/post_backup pair is now one openBackupBracket object
whose closer the caller defers, so all eight early returns between the hooks
pay the undo, and a failed undo reports X500459BKPUNQ instead of being
unreportable. That block sat entirely below this change’s freeze bracket, which
is why neither commit re-introduced what the other removed. Both are in the
v0.0.18 pin.