Do we use the saga pattern? And what we should do instead
Status: proposal, awaiting ruling. Nothing in this document changes behaviour.
Scope: every multi-step destructive operation in shc + ce.
Evidence: three read-only survey rungs (app install/update/uninstall · backup/restore/recover ·
tenant/cluster/terraform), cite-checked against ce a66b5b3 (2026-08-16 13:54Z) and shc e2eaf657a.
Cite-check note. The three reports disagree on one line number (
RollbackFailedInstall’s update early-return: 2653 vs 2691 vs 2694). All three are correct for the tree each read; the working copy at2fd655b2is 6003 lines,a66b5b3is 6109. Every load-bearing claim below was re-read ata66b5b3and holds. Line numbers here area66b5b3.
1. The straight answer
Section titled “1. The straight answer”No, we do not use the saga pattern, with one exception, and it is excellent.
clone.Move’s heavy path is a real saga: a write-ahead journal that is fail-closed
(move_journal.go), two independent evidence gates before the destroy (health, then bytes), a
compensation that quarantines rather than deletes, a daemon-start resume, and a refusal (never a
guess) once the source has been torn down. It is one verb out of roughly thirty.
What the rest of the system actually guarantees today, by class:
| class | operations | the guarantee |
|---|---|---|
| additive | backup create, Clone | Safe by nature. A failure leaks bytes, never destroys them. |
| converge-on-retry | install | Real. Re-running the same install over a live row converges; SettleDeployStatus stamps the verdict from an engine probe, not from the absence of an error. |
| destructive-in-place | update, restore, rollback, group restore, recover, clone.Rename | None. No compensation, no checkpoint, no resume. The destructive act (compose up recreate; removeAllBestEffort per mount) happens before the failure is knowable. |
| one-way doors | uninstall, tenant delete, tf node destroy, presumed-dead move | A pre-commit refusal, implemented anywhere from excellent (presumed-dead move’s two gates; node destroy’s occupancy refusal) to decorative (the uninstall backup gate) to absent (tenant delete; daemonless uninstall). |
Two disciplines the tree already has right, stated once so the rest reads as signal: completion on
evidence (SettleDeployStatus, seal.go:verifySealedMember, VerifyRestoreSourceInventory), and
node-local crash journals for the backup quiesce bracket plus lease_boot.go’s boot reconcile.
Every one of these exists in exactly one module. The defect is not absence of the idea. It is
that no rule is applied uniformly.
2. Reachable half-finished states, ranked by what a customer loses
Section titled “2. Reachable half-finished states, ranked by what a customer loses”Data loss beats a stuck row beats a leaked byte.
| # | state | how it is reached | what is lost |
|---|---|---|---|
| 1 | shc uninstall with the daemon down destroys volumes with no archive and no warning. GateUninstallBackup returns nil on its first line when preUninstallBackup == nil, and SetPreUninstallBackup is called only from daemon/, never from cmd/shc. RunCLIUninstall then runs ExecuteComposeDown{RemoveData: true} (modules/app/register.go:533-553). The adapter’s own comment concedes it: “nil in the daemonless CLI … so this is a no-op there”. | “the daemon is wedged, let me clean this up” | All of one deployment’s data. Silently. Permanently. |
| 2 | Daemon uninstall proceeds on an archive that does not exist. The gate keys on err != nil and never reads the returned row; phases.go could return (completed, Location: ""). | master-archive assembly fails; teardown continues | Same as #1, with a green row claiming otherwise. ce!22 closes the producer half. |
| 3 | Recover leaves a live, healthy-looking, EMPTY stack. install succeeded, restore failed, postgres initdb’d a fresh cluster, nothing tore it down and nothing marks it. | DR replay with one failed member | The entire point of DR, discovered late, by a user. |
| 4 | Restore stranded mid-wipe. The inventory proof is whole-archive; the wipe is per-mount (removeAllBestEffort). Group restore takes no per-member safety backup at all, and its doc comment says “atomically restores”. | any failure after mount 1 of N | Partial, unrecoverable data, unless a pre-restore backup exists and has a location. |
| 5 | clone.Rename half-rename, permanently. Dirs and volumes at target, DB row at source, stack down. Retry 404s on GetStack(source). The compensation (restartSourceBestEffort) provably cannot work past step 3; by then the tree has moved. | ordinary shc move within one tenant/env/node | No bytes destroyed; the deployment is unreachable and needs hand surgery. |
| 6 | A failed update leaves the stack down. compose up recreates destructively, aborts, leaves the replacement in created; restart: policies govern containers that exit, so nothing acts. RollbackFailedInstall returns immediately on update. Four occurrences on 2026-08-16, four causes, one consequence: the estate’s ingress dark until a human typed docker start. | any failed update | Availability, estate-wide when it is the ingress. ce!20 closes the in-process half. |
| 7 | Tenant delete silently undone. The tombstone is written at service.go:602, after the row drop at :566 and the scoped sweep at :583. Crash in that window → the realm reconcile finds no tombstone and re-creates the tenant within 60s, on top of a half-swept row set. | crash mid-delete | A deliberate delete reverted, with its secrets and connections already gone. |
| 8 | Rows parked in updating / installing / removing / queued forever. grep over modules/ daemon/ finds only writers and the type table; there is no boot reconciler for any transient stack status. Contrast modules/backup/lease_boot.go, which exists for exactly this. | daemon death mid-operation | Operator time; a stack that refuses further operations. |
| 9 | Leaked bytes and estate objects. 26.1 GB across 114 unreferenced files measured on bitspur; orphan Keycloak realms and overlay networks after a failed tenant delete; leaked voter slots and fabric /24s on the unlicensed enroll path; restore.go:1516 leaves a full second copy of every restic restore forever. | assorted | Money and capacity. |
Read the top four together: they are all durability and verification defects, not compensation defects. That observation is the whole of section 3.
3. The proposal: four layers, not four options
Section titled “3. The proposal: four layers, not four options”Prescribing sagas for #1–#4 would add an orchestrator and fix nothing. These are layers; each defect needs the cheapest layer that closes it.
Layer 0: verified completion. Never record success without reading the result back.
Section titled “Layer 0: verified completion. Never record success without reading the result back.”Cost: near zero. Runs on every path, therefore constantly exercised, therefore trustworthy.
Closes #2, most of #3, and the whole “optimistic completion” class. The rule: a command’s exit
status is a claim; the row must carry a fact. The tree already does this correctly three times
(SettleDeployStatus, verifySealedMember, VerifyRestoreSourceInventory) and incorrectly in the
places that bit us. This is the answer to most of the owner’s question.
Layer 1: refuse at the one-way door.
Section titled “Layer 1: refuse at the one-way door.”Cost: one if and an error code. Fails safe by construction. A wrong refusal costs an operator
a flag; a wrong unwind costs data. Closes #1, hardens #2, #3, #7.
The rule to adopt verbatim, because move’s data gate already states it: “I could not look” must
never share an answer with “it is fine.” Currently violated by GateUninstallBackup (nil hook →
nil), by node_join_recovery.recoverAlreadyClustered (membership unreadable → proceed with the
destructive --force wipe anyway), and by rollback.go:69 (err == nil discards a failed
checkpoint and continues into a replace-restore).
Layer 2: write-ahead intent + idempotent resume.
Section titled “Layer 2: write-ahead intent + idempotent resume.”Cost: a durable record per operation, a boot reconciler, and idempotency on each step. Closes
#5, #6’s crash half, #7, #8. Its virtue over compensation is decisive: if the steps are
idempotent, the resume path is the same code as the normal path, so it is exercised by ordinary
use rather than only by disaster.
Two correct implementations already exist to copy: move_journal.go (write-ahead, fail-closed,
“over-record is free, under-record is the bug”) and lease_boot.go (boot reconcile on
node-identity proof, 6.82s vs the old 16m). The two journals in the tree disagree on their own
core rule: drain_journal.go is write-after and deliberately swallows a failed write. Pick one
rule (move’s) and make the other conform.
Layer 3: full saga: compensation per step, orchestrator, reverse unwind.
Section titled “Layer 3: full saga: compensation per step, orchestrator, reverse unwind.”Recommend against as a general pattern. Two reasons, both mechanical rather than aesthetic:
- The underlying tools have no inverse.
docker compose uprecreates destructively and offers no rollback;removeAllBestEffortunlinks. “Compensating” the destructive-in-place class would mean restore-from-backup, a second destructive operation with its own failure modes, run on the rarest path in the product. That is a worse trade than refusing. - Compensations are the least-tested code we own, and this tree proves it (section 4).
Where compensation is right, it is narrow and forward-only: docker start on what a converge
stopped (ce!20), ForgetSnapshot on an unverifiable seal, quarantine-instead-of-delete in
rollbackFailedMove. Adopt that form. Do not build an orchestrator.
Only two operations are genuinely saga-shaped: recover (#3) and group restore (#4), where
step N’s failure really does invalidate step N−1 across members. Even there the right compensation
is mark and refuse, not unwind: an empty recovered stack should be stamped unusable and kept out
of the wiring phase, not auto-uninstalled. Auto-uninstall is a destructive act triggered by an
error path we cannot test.
4. What it costs, and where “refuse and require a human” is the honest answer
Section titled “4. What it costs, and where “refuse and require a human” is the honest answer”A compensation runs only when its forward step failed. That is why every compensation in this tree that is not routinely exercised is wrong, and we can show it rather than assert it:
Rename’srestartSourceBestEffortcannot work after step 3; it restarts a name the registry no longer holds, or a tree that has moved. No test caught it; nothing exercises the path.RollbackFailedInstall’s update early-return rests on a premise that is false for the failures that actually occur (“the stack is still live”). Four measured incidents disproved it in one day.- Against those:
move’srollbackFailedMoveis correct, and the reason is structural:ResumeInterruptedMovesruns it at every daemon boot and keeps failed rollback records for retry. The compensation sits on a hot path by construction.
The rule this implies: a compensation you cannot arrange to run routinely must be replaced by a refusal. Three acceptable exercise mechanisms, best first:
- Make it the resume path: the boot reconciler runs it on ordinary restarts (
move). - Make it forward-only and idempotent, so it is safe to run twice and can be an operator verb
(
docker start,ForgetSnapshot). - Fault-injection tests that kill between named steps. Expensive; reserve it for the two saga-shaped operations only.
Operations where the answer is refuse and require a human, and we should say so in the product:
- uninstall: no inverse exists. Verify the archive; refuse without one.
- cluster leave past storage detach: the code already says a refusal here cannot be undone.
- presumed-dead move: rebinding to a dead node is never correct. Already refuses; correct.
- recover with a failed restore: mark unusable, refuse to wire it in. Do not auto-uninstall.
- move past
SourceTornDown: already refuses, and correctly. But the--finish-forwardverb the refusal points the operator at does not exist. A refusal with no exit verb is an operator trap; that is a cheap, high-value fix.
5. Migration order, and the ruling on the two changes in flight
Section titled “5. Migration order, and the ruling on the two changes in flight”Ordered by safety bought per unit of risk taken.
| # | change | layer | risk |
|---|---|---|---|
| 0 | Merge ce!22 as-is. | 0 | low; it makes a green row rarer, never a red one greener |
| 1 | Merge ce!20 with adjustment A1 (below). | 0 + narrow compensation | low |
| 2 | Close the uninstall gate, both halves. (a) daemon: refuse on Location == "" and on DegradedNoDataCaptured, one if. (b) daemonless: preUninstallBackup == nil must be a loud refusal naming --no-backup, not a silent return nil. | 1 | ~5 lines; the single highest-value change in this report |
| 3 | One journal rule, one boot reconciler. Make drain_journal write-ahead like move_journal. Add a stack-status reconciler that adopts updating/installing/removing/queued rows at boot and settles them by re-observation; it must not re-run the operation. | 2 | medium; closes #8 and #6’s crash half |
| 4 | Put clone.Rename on the move journal and delete the compensation that cannot work. | 2 | medium; closes #5 |
| 5 | Refusals: rollback.go:69 aborts like restore.go:348; node_join_recovery refuses when membership is unreadable; tenant delete writes the tombstone before the row drop; recover marks-and-refuses. | 1 | low each |
| 6 | Leaks: restore.go:1516 defer os.RemoveAll; release-defers on the unlicensed enroll path. | n/a | low |
| 7 | Only now, if ever, consider real orchestration for recover + group restore. | 3 | high |
Is ce!22 (backup read-back verification) consistent with this shape?
Section titled “Is ce!22 (backup read-back verification) consistent with this shape?”Yes, it is the reference implementation of Layer 0. Merge unchanged. Three things it gets right that the rest of the estate should copy:
- It fixes the producer, not the consumer. One producer, three consumers that currently disagree about what an empty location means. Fixing the gate alone would have left the other two.
verifyCapturedSnapshotdistinguishes DISPROVEN (forget the snapshot, fail the row) from INTERRUPTED (leave it, say so); that is exactly the Layer-1 rule, applied at Layer 0.- The empty-location refusal at
finalize(X500473BKPNOA) makes the top of the ranking’s precondition unrepresentable going forward, rather than merely detected.
One follow-up, not a blocker: ce!22 does not subsume migration item 2. It stops new
empty-location completed rows; legacy rows written before the merge still exist, the gate still
keys on err != nil, and the daemonless hole (#1) is untouched by it entirely. Keep item 2 on the
list after ce!22 merges.
Is ce!20 (traefik converge restore) consistent with this shape?
Section titled “Is ce!20 (traefik converge restore) consistent with this shape?”Yes in shape, with one adjustment I would ask for before merge, and one immediately after.
What it gets right, and should be treated as the template for narrow compensation: it is
forward-only (docker start, never a re-converge), idempotent, verified by re-observation rather
than by exit code, and it never changes the verdict; the update still reports FAILED. It also
correctly leaves RollbackFailedInstall’s update early-return alone: the answer to a failed update
is not to tear down a live stack.
- A1: adjust before merge: the before-image is in-process only.
preUpis a local variable inExecuteCompose(install_runner.go:2223on the MR branch). The four measured incidents were converges the daemon survived. A daemon killed mid-compose up(which is precisely the failure that already parks rows inupdatingforever (#8)) leaves the identical dark ingress with nobody left to run the restore. Ask: persist the before-image beside the rendered compose (one small file), so the boot reconciler in migration item 3 can invoke the sameRestoreAfterFailedConverge. Without it we ship a fix that covers the observed incidents but not the mechanism, and the follow-up gets much more expensive once the code has shipped in-process-only. - A2: land in ce!20 or immediately after:
DeriveContainerStatusmaps"created"→ContainerStatusStarting(container_status.go:52). So even with ce!20 merged, when the restore itself fails (X500117CVGRSFis a WARN), the health rollup still reports the stack as coming up rather than broken; our own status surface continues to lie about the exact state ce!20 exists to fix. A container increatedwith no start time is notstarting; it is degraded.
6. The one-line ruling being asked for
Section titled “6. The one-line ruling being asked for”Do not adopt the saga pattern. Adopt three rules and apply them uniformly: record completion only from a read-back; refuse at every one-way door, and never let “I could not look” answer the same as “it is fine”; write intent ahead of the act and resume it at boot. Reserve compensation for the narrow, forward-only, routinely-exercised cases we already have. The correct implementation of each of these already exists in this codebase, in exactly one module apiece.