Skip to content
SHC Docs

Backup engine convergence: the two engines are wrong in opposite directions

Status. Half landed. BOTH HALVES LANDED, and the second ships in v0.0.19. The encrypted engine gained its freeze bracket in v0.0.18 (ce modules/backup/service.go, test modules/backup/restic_quiesce_test.go). The scheduler’s destination converged in ce de396a7, pinned for the v0.0.19 cut, but not by the re-route this document deferred, and §3b is kept below because its measurements are why.

★ THE DESTINATION CONVERGED WITHOUT THE ENGINE CONVERGING. The capture stays orchestrated: same phases, same freeze, same escalation to a STOP, byte-identical .bak. FINALIZE then seals that .bak into backup.repo under backup.password, verifies it readable there at exactly its on-disk size, and only then removes the local plaintext; the row becomes restic:<repo>:<snapshot>. That answers all three §3b blockers without paying any of their costs: the two “captures strictly less” payload gaps and the pre-uninstall hard-refuse simply never arise, because the engine that captures is the one that always did. §3c step 2 (“teach the encrypted engine the payloads it is missing”) is therefore not owed for this outcome; it is only owed if step 4’s engine deletion is ever pursued.

★ AND THE RETENTION LEAK WAS NOT A BLOCKER. IT WAS A LIVE DEFECT. §3b frames it as a consequence of re-routing. It was already happening: Apply deleted the SQL row whether or not the bytes went away, so every restic: row shc backup create produced and retention expired had its os.Remove fail, its error swallowed, and its row deleted anyway. Fixed in the same ce commit (§3c step 3, in full: the ref-carrying ArchiveDeleter, all three daemon wiring sites, keep-the-row-on-failure, the restic-location refusal in the default deleter, BytesFreed from what was freed, one prune per sweep, and the backup_cache twin).

What is still owed is the live pause measurement (§4).

Baselines. ce origin/main 8ab2081, shc origin/main d934af851, shc’s ce pin v0.0.18-0.20260804002903-5a52e9042337. Everything below was read at those refs, not from a working checkout.


There are three capture paths, not two. The third is daemonless and is included because it is reachable from the same verb.

restic-directorchestrateddaemonless tar
entry pointService.CreateBackup (modules/backup/service.go:937)Service.CreateOrchestratedBackup → createWithID → phases.goRunCLICreate (modules/backup/cli_runner.go:94)
quiesce bracketnow yes (this change): bracketPause held across restic backupyes: PREPARE → QUIESCE → CAPTURE → RESUME → FINALIZE, resume() deferredno
pause escalation to STOPno (pause_hold.go is orchestrated-only)yesno
destinationbackup.repo, resolved per-deployment through EnsureResticConfigured (scope tenant/stack/env)PathsResolver.BackupsDir(tenant, stack) = {state_home}/backups/<tenant>/<stack>/. Not configurable, and backup.repo cannot redirect it; TRUE at the baseline, FALSE at the v0.0.19 pin: the capture still lands there, then FINALIZE seals it into backup.repo and removes the local copy (see Status){state_home}/backups/... sidecar tar
encryptionyes: every blob under backup.password; row Location = restic:<repo>:<snap>no: !<arch> AR container, archive.go’s only crypto import is sha256. The one sealed member is the manifest’s inline Fernet EncryptedBundle (vault secrets)no
key gaterefuses without backup.password (X500414RSTPAS) and without backup.repo (X500415RSTBKN)none: constructs no *Restic, reads no keynone
manifestfull BackupManifest staged and captured inside the snapshot (docs/proposals/backup-manifest-in-repo.md); local row copy adds Checksumfull BackupManifest written in FINALIZE, checksummed, packed as manifest.json member of the .bakthin state.json-class sidecar, not a BackupManifest (group_cli.go says so)
capturesthe requested Paths (default: the deployment dir), minus backup.yaml exclude.mountsper-mount tars plus host-bind sources outside the deployment dir, plus unbound named volumes (swarm), plus app converters (pg_dumpall)compose named volumes only
single-flight leaseno (LiveBackupLockHolder is called only from beginStackBackup)yesno
retentiondefaultArchiveDeleter os.Removes the Location → fails, row deleted anyway ⇒ snapshot orphaned in the repo. TRUE at the baseline (this was a live defect, not a hypothetical) and FIXED at the v0.0.19 pinos.Remove on a filesystem path ⇒ worksworks

restic-direct: shc backup create with a wired in-process Service (register.go:288); shc backup create over the wire (router.go:253, the public POST /api/method/shc.backup.create); the async create endpoint (router.go:322); shc backup create --file (register.go:467 → createFileViaService / createFileViaDaemon); --all-connected / --with-dependants from the CLI (CreateConnectedGroupForStack → CreateConnectedBackupGroup → per-member CreateBackup, grouping.go:286); the backup mount overlay’s snapshot (mount/overlay.go:856).

orchestrated: the backup schedule cron (subsystems/impl/backup_scheduler.go:290, via ExecuteBackupScope); the pre-uninstall safety gate (daemon/services_backup_config.go:210, wired as app.SetPreUninstallBackup); the pre-restore rollback snapshot (rollback.go:69); restore reentry (restore.go:337); clone / move (daemon/services_seams_clone.go:51; clone.BackupProvider.CreateBackup is a seam name, its adapter calls CreateOrchestratedBackup); the group driver CreateBackupGroupFromRequest (grouping.go:89 → beginStackBackup). CreateBackupAsync (orchestrator.go:76) has no production caller.

daemonless tar: shc backup create and create --file with neither a wired Service nor a configured daemon endpoint (register.go:266,339). That shc has no vault, and since 2026-10-19 no mint store on the disk either: a vault ref it meets is refused (X404905SECREQ, no vault reachable), and the credential --var heal fails closed there, so its capture never holds a secret it minted or sealed itself.

Two consequences of the destination split, both verified live tonight

Section titled “Two consequences of the destination split, both verified live tonight”
  1. backup.repo is configured and the orchestrated archives ignore it. On promanager, backup.repo = s3:https://sos-de-fra-1.exo.io/shc-backup-promanager-hel1, and /var/lib/shc/backups/ holds 4 .bak files totalling 658 MB. So the framing “a schedule would write plaintext credentials into S3” is not what happens. It is worse for durability and better for confidentiality than that: the orchestrated engine cannot reach S3 at all, so a schedule writes plaintext to the leader’s local disk, next to the data it exists to survive the loss of. warnArchiveMissedRepo / X200079BKPLOC says so once per archive, at WARN.
  2. The plaintext is plaintext. Read read-only off promanager with no key: head -c 600 of /var/lib/shc/backups/default/s3front/20260729T180234Z_7f3a3f9feb60cf27.bak is !<arch>\n, manifest.json/, then the manifest in the clear: tenant, stack, app, node, the full stack record. The manifest also carries install-time --var values as bare strings.

The unreachable skip in the uninstall gate

Section titled “The unreachable skip in the uninstall gate”

app.SetPreUninstallBackup treats X500415RSTBKN (“no backup repository configured”) as a clean skip with a loud WARN. That code is raised only from restic-touching call sites (actions.go:911, service.go:808,1085, keyrotate.go, mount/subproc/manager.go:194, EnsureResticConfigured at seams.go:346). CreateOrchestratedBackup reaches none of them. So the branch is unreachable: on an estate with no repo configured the pre-uninstall backup still runs and still writes a plaintext local .bak, and the “NO SAFETY BACKUP” warning never fires. Not a defect in itself (a backup did happen), but the operator’s mental model of that warning is wrong.


Owner, 2026-08-02: backup stops the stack rather than risk an untrusted archive: “we can never ever have an untrusted backup; I’d actually literally rather our service go down.” DegradedStackStopped in types.go quotes it.

Before this change:

  • the engine that protects the data (orchestrated: freeze, escalation to STOP, host-binds, named volumes, converters) wrote it unencrypted to a path the operator cannot move;
  • the engine that encrypts it (restic-direct: backup.password, backup.repo, per-deployment scope) read the volumes live.

So neither satisfied the ruling, and the practical consequence stands: an owner told to “set a backup schedule” would be scheduling unencrypted archives onto local disk on a cron.


3. The convergence, and what the prior got wrong

Section titled “3. The convergence, and what the prior got wrong”

The strong prior going in was: route the scheduler to the encrypted engine, and give the encrypted engine the quiesce bracket. Half of that survives measurement.

3a. LANDED: the encrypted engine gains the freeze

Section titled “3a. LANDED: the encrypted engine gains the freeze”

It did not need new machinery. quiesce.go bracketPause was already the shared primitive: the orchestrated per-stack flow decomposes it into phases, and two other callers (quiesceToStatic, the routed single-mount RPC CaptureMountTar) use it whole. CreateBackup now calls the same function. Nothing was duplicated, and the pause/unpause classification, the fresh-context release, the ErrQuiesceBroken / ErrStackLeftPaused sentinels and the three-valued outcome all came along for free.

Shape, in CreateBackup:

pre_backup hooks (stack live)
→ pre_pause hooks → PAUSE ← bracketPause
→ stamp manifest → stage manifest → restic backup
→ UNPAUSE → post_pause hooks ← deferred release
→ post_backup hooks
  • Defer-safe from the first line. The release is armed by defer in the same statement group that acquires the freeze, so a staging error, a restic failure, a cancelled ctx or a panic all still unpause. This is deliberate coordination with the sibling lane removing the non-deferred runBackupPostJobs bug from this same function: that block is untouched and sits entirely below the bracket, so the two changes compose. Do not introduce a second copy of the bug being removed.
  • Only three things run under the freeze: stamp, write the few-KB manifest, run restic. The whole manifest (stack record, vault bundle, deployment config, volume inventory) is still built with the stack live.
  • The stamp is split, and the split is forced. The manifest rides inside the snapshot, so the in-repo copy must be final before restic runs and can only carry the freeze’s t0 answer. The release’s t1 answer (a container replaced mid-capture, or one left frozen) is folded into the local row copy, which is the copy backup ls / backup inspect project from and which already diverges by carrying the snapshot id. Folding only ever weakens, so the re-stamp can downgrade a claim and never upgrade one.
  • The projection is now one function. applyQuiesceStamp in pointintime.go writes PointInTime + not_quiesced + stack_left_paused, and phases.go FINALIZE was moved onto it. Two engines making the same claim from two hand-written copies is exactly how an archive ends up claiming a point-in-time it never had.
  • A failed freeze does not abort. QuiesceFailed stamps live + not_quiesced and proceeds. That matches the ruling ErrQuiesceUnsupported already encodes (an honest live archive beats no archive) and matches invariant (c), which aborts only in the whole-tar mode. Whether the restic path should instead escalate to a STOP like pause_hold.go does is an owner call and is listed as owed below.

Proof. modules/backup/restic_quiesce_test.go. The assertion is a sample, not a call order: the fake pauser keeps a live paused flag and the fake restic runner reads it at the moment the backup subcommand runs, so a pause acquired and released around the snapshot fails. Three derived cases: freeze lands + capture succeeds (frozen during the read, resumed after, stamped assured); freeze lands + capture fails (resumed anyway); freeze fails + capture succeeds (release still attempted, stamped live + not_quiesced).

Verified adversarially, not just green:

mutationresult
release only on the success path (the sibling lane’s bug shape)TestCreateBackupResumesAfterAFailedCapture FAILS
release immediately after the pause (not held across the capture)TestCreateBackupHoldsTheStackFrozenForTheSnapshot FAILS
pristine origin/main service.go, new test fileall three FAIL

3b. DEFERRED: the scheduler re-route, on three measured blockers

Section titled “3b. DEFERRED: the scheduler re-route, on three measured blockers”

Read this section as the measurement that chose the seal over the re-route, not as pending work. All three blockers below are accurate at the trees named in Baselines, and the first one was accurate for shc backup create too. The scheduler’s destination converged in ce de396a7 without re-routing the engine, which is why none of these had to be paid; the retention leak was fixed outright rather than worked around. The one line that is now FALSE as a forward-looking claim is “Today the scheduler writes filesystem paths, so retention works”; a sealed schedule writes restic: rows, and it is the retention fix that makes that safe.

Retention would leak every scheduled archive, forever. The only wired ArchiveDeleter is retention/service.go’s defaultArchiveDeleter, which does os.Remove(location). Nothing in the daemon supplies another (grep 'Archives:' finds only an unrelated struct in adapter/commands/migrate.go). Apply swallows the delete error and deletes the SQL row regardless (service.go:178-189, and the comment says so). Today the scheduler writes filesystem paths, so retention works. Re-route it and every expired row gets forgotten from the DB while its snapshot stays in the restic repo: an unbounded, un-enumerable repo, on the estates most likely to have a schedule. Fixing this is not a one-liner: it needs a restic-aware deleter doing forget + prune with the key, and prune is a repo-wide lock.

The encrypted engine captures strictly LESS. captureHostBinds, CaptureNamedVolumes and the converter run exist only in phases.go FINALIZE. Re-routing the scheduler today would silently stop capturing: absolute compose bind sources outside the deployment dir; unbound docker-managed named volumes (which is how a swarm deployment holds its data); and every app converter artifact (postgres’ pg_dumpall). That is a data-loss-class regression dressed as a security improvement, and it is the single strongest argument against the naive re-route. grouping.go:249 already notes the mirror-image problem in the other direction: the orchestrated tar hard-errors on container-owned mount data an unprivileged daemon cannot read, where restic tolerates it.

Restore is already red fleet-wide. X400867RSTEMPT (restore/recover replay failing, proven pre-existing on origin/main by control experiment, 20+ apps) means the artifact-shape change a re-route implies cannot be verified end to end tonight. Changing which engine writes the archive while the read path is broken is untestable by construction.

Secondary, but real: the restic engine refuses without backup.password (X500414RSTPAS). Re-routing turns the pre-uninstall safety backup from always-runs into hard-refuses on any estate with no key; the gate treats a non-X500415RSTBKN error as fatal, so shc uninstall would start failing.

3c. The convergence design, for the ruling

Section titled “3c. The convergence design, for the ruling”

Target: one engine, freezing, encrypting, redirectable. The honest order, smallest first:

  1. (landed) the encrypted engine freezes, and both engines stamp their claim through one projection.
  2. Teach the encrypted engine the payloads it is missing (host-binds, unbound named volumes, converters) by lifting them out of phases.go FINALIZE into helpers both engines call. This is the real work, and it is pure addition: the orchestrated engine keeps behaving identically.
  3. Wire a restic-aware ArchiveDeleter (forget + prune, holding the key) and make retention.Apply stop deleting rows whose archive delete failed. Both halves are needed; either alone still leaks.
  4. Then re-route the scheduler, the pre-uninstall gate and the rollback snapshot to the encrypted engine, and delete the orchestrated capture rather than leave two.
  5. Owner call, independent of the above: should the restic path escalate a held freeze to a STOP the way pause_hold.go does for the orchestrated path? The ruling’s own words point at yes.

3d. Migration concern for the archives that already exist

Section titled “3d. Migration concern for the archives that already exist”

Existing plaintext .bak files are not convertible in place: the row’s Location prefix is the predicate every read path branches on (IsResticLocation, consumed by restore.go, actions.go export/verify/mount, service.go delete), so a .bak cannot become a restic: row without being re-ingested, and ImportBackupFromFile is the only tool that could, per archive, needing the vault master to re-seal the inline bundle. Whatever step 4 does, it must:

  • leave existing filesystem-located rows readable (do not narrow the read branch);
  • not silently re-key or move them (a rotate orphans every prior archive and SHC has no key-add verb; see backup_encryption_two_engines);
  • state plainly that pre-cutover archives are unencrypted, which backup ls’s Encryption column now does per row (encryption.go).

Landed (ce, this change).

  • modules/backup/service.go: CreateBackup holds a defer-safe bracketPause across the restic capture; pre_pause / post_pause bracket hooks fire; stack resolution hoisted so the bracket and the manifest share one lookup; the local row manifest re-stamped with the folded verdict.
  • modules/backup/pointintime.go: applyQuiesceStamp (+ appendDegradedOnce), the single projection of a QuiesceVerdict onto a manifest.
  • modules/backup/phases.go: FINALIZE moved onto applyQuiesceStamp.
  • modules/backup/restic_quiesce_test.go: the three-case guard, mutation- verified and control-verified against pristine origin/main.

Landed (ce de396a7, shipping in v0.0.19).

  • The orchestrated .bak is sealed into backup.repo under backup.password in FINALIZE, verified readable at its exact size, and the local plaintext removed only then. Row location becomes restic:<repo>:<snapshot>, so backup ls reports it encrypted because it is. Applies to the scheduler, the pre-uninstall safety capture and the pre-restore rollback snapshot, every trigger §1 lists as orchestrated.
  • The verify is load-bearing: at the seal the local .bak is the only copy, so a failed verify KEEPS it, keeps the row’s filesystem location, and forgets the half-trusted snapshot (X200075BKPSEL names the reason, X200079BKPLOC still fires). Sealing is conditional on repo + key and never applies to an explicit --file / clone-staging output path.
  • backup.repo set with no backup.password is refused at schedule set (X400077BKPSCD); a schedule is the one backup verb nobody watches, and that combination can only ever write plaintext locally. An estate with no repo at all is NOT refused: local plaintext is the only destination it has.
  • Reads needed no second implementation: a sealed archive IS the .bak, so it is streamed out of the repo and handed to each reader as a file-backed row (restore, export, rollback, delete/verify/forget). backup mount refuses and points at backup export.
  • Retention no longer orphans: §3c step 3 in full: the ref-carrying ArchiveDeleter, all three daemon wiring sites, keep-the-row on a failed delete (X500918RETARC + the X500920RETKPT rollup), the default deleter REFUSING a restic location instead of pretending (X500921RETNRS), BytesFreed counting what was freed, one restic prune per sweep rather than per archive, and the backup_cache twin swept (X500922RETCAC).
  • Existing plaintext .bak rows are untouched and no read branch narrowed, which is §3d’s first requirement.

Still deferred, with the reason.

  • scheduler / pre-uninstall / rollback re-route: moot: the destination converged via the seal, so the engine never had to move and the payload gaps were never paid.
  • restic-aware ArchiveDeleter + retention.Apply row-delete-on-failure: landed above.
  • lifting host-bind / named-volume / converter captures into shared helpers: now owed only for §3c step 4 (deleting the orchestrated capture), not for encrypting a scheduled archive.
  • pause-hold escalation on the restic path: owner call.
  • CreateBackup taking the single-flight lease the orchestrated path takes.

Owed.

  • a ce pin bump: DONE. The v0.0.18 release lane bumped the pin to ce 852c768 (v0.0.18-0.20260804030128-852c768ca9b1), so everything under Landed above ships in v0.0.18 and does change estate behaviour from the roll onward.
  • a live proof on an estate, after the pin bump: shc backup create against a real deployment, sampling docker inspect for Paused during the capture, the same measurement that showed 0 of 300 samples paused before this change. STILL OWED, and it is the load-bearing one: the pre-change measurement is the only pause figure this project has for the restic path, and the window now covers a restic walk rather than the orchestrated engine’s double-pass rsync of a delta, so the 0.60–0.76 s orchestrated numbers must not be read across.

The sibling lane landed too, and the two composed as designed. ce 8111eb3 removed the non-deferred runBackupPostJobs bug this document anticipated at §3a; the pre_backup/post_backup pair is now one openBackupBracket object whose closer the caller defers, so all eight early returns between the hooks pay the undo, and a failed undo reports X500459BKPUNQ instead of being unreportable. That block sat entirely below this change’s freeze bracket, which is why neither commit re-introduced what the other removed. Both are in the v0.0.18 pin.