Integration teardown: wake the 22 dead jobs, then collapse three runners into one
Status: open design. Steps 3, 5, and 7 are not done.
Revision 4 (status only, 2026-08-05). Steps 4 and 6 LANDED on ce main 2026-08-04:
988d4c2(PersistedEdgeState written at bind),63e44b6(the uninstall leg runsjobs.unintegrationagainst the frozen edge),45aae15(the-i <ref>!disconnect leg, snapshot-before-deploy / run-after),1c8aba6(thesso→oidcsocket rename with the node-local migration that keeps already-wired edges revocable). The design text below is unchanged and still the argument of record; only the step table’s status column moved. Still open: step 3, step 5’score/lifetimecascade (resolveEdgeLifetimestays pinned toexternal, so every destroy is closed by construction), and step 7.Revision 3. The six echo-only
cleanup-accesssocket jobs have been deleted, so the dead-job count is 28 → 22 and every remaining dead job undoes real state. The table below is a fresh measurement, not an edit of revision 2’s. One prerequisite is added that revision 2 did not know about: theintegrationtable is never written on the install path, so step 6 cannot key teardown off a DB row (see The registry the rows would come from is unwired).Revision 2. Rewritten after an adversarial review returned ship with changes. What moved: the dead-job count is remeasured across both catalog trees (26 → 28); the gating mechanism is now the
core/lifetimescope parameter, not an operator flag and not an app-authoredcleanup@boolean; the rollout reflects the renames that have already landed; and the project owner’s vApp constraint is recorded as a hard constraint rather than an open question.
Why / current state
Section titled “Why / current state”Twenty-two job blocks in the catalog were written by app authors, are parsed by nothing, and have never executed. They are not stale leftovers; they are the teardown halves of integrations that ship today.
The count, remeasured
Section titled “The count, remeasured”The catalog lives in two trees. ce/builtin/<app> is the source of truth for
the four built-in apps (keycloak, openobserve, postgres, traefik); it is
gitignored in this repo and staged into apps/ at build time. The first
scan walked apps/ only and therefore missed it. Counts below are a YAML parse
of every jobs.<phase> list in the CE repo’s builtin/ and this repo’s apps/,
re-run at revision 3 rather than carried forward; a count nobody re-measures is
a count that silently drifts, which is what happened to socket/discovery
(revision 2 recorded 28 in apps/; a job has been added since).
| surface | phase | ce/builtin | apps | total | status |
|---|---|---|---|---|---|
*.plug.yaml | unintegration | 0 | 15 | 15 | dead |
*.socket.yaml | unintegration | 1 | 5 | 6 | dead |
*.vsocket.yaml | post_deployment | 1 | 0 | 1 | dead |
| 22 | |||||
*.plug.yaml | post_deployment | 1 | 42 | 43 | runs |
*.socket.yaml | discovery | 2 | 29 | 31 | runs |
*.socket.yaml | integration | 2 | 21 | 23 | runs |
*.vsocket.yaml | integration | 2 | 10 | 12 | runs |
*.vsocket.yaml | cleanup | 2 | 10 | 12 | runs |
| all authored jobs | 11 | 132 | 143 |
2 of the 22 dead jobs live in the staged tree, which is why a scan of apps/
alone cannot see them:
ce/builtin/keycloak/sso.socket.yaml :: delete-oidc-client(unintegration)ce/builtin/postgres/database.vsocket.yaml :: vapp-install-extensions(post_deployment)
The 15 dead plug jobs span 10 apps (defectdojo, fleet, gitea, immich×2, mailcow×2, mautic×2, nextcloud, sonarqube, windmill×2, wordpress×2). The 6 dead socket jobs span 6 apps (crowdsec, keycloak, minio, pmm, s3, wazuh).
The socket row fell from 12 to 6 because rollout step 0 deleted the six
cleanup-access jobs (hockeypuck, immich, paperless, peertube, penpot, postiz)
whose entire body was echo "<app> integration cleaned up". Every one of the
remaining 21 unintegration jobs undoes state a real teardown has to undo:
which is the property that makes waking the phase worth doing and deleting it
unacceptable.
Lesson for the audit itself. Any catalog measurement that reads
apps/without stagingce/builtinis wrong by construction, and will stay wrong silently because the missing apps are the platform apps, the ones every other app plugs into. Every future catalog assertion must run against the staged tree.
What parses what
Section titled “What parses what”The runners parse, exhaustively:
plugJobsDoc(plug_jobs.go:43-47):post_deploymentONLY.SocketJobsDoc(socket_jobs.go:45-50):discovery,integration.vsocketDoc(vapp_runner.go:495-507):discovery,integration,cleanup.
So cleanup executes on a vsocket and is silently ignored on a socket and a
plug, and post_deployment is the mirror image: it executes on a plug and
is silently ignored on a vsocket. The same word, in the same YAML dialect, in
the same directory, with opposite behaviour, in both directions.
plug_jobs.go:41-42 tells the reader the other phases are “handled elsewhere”;
they are not handled anywhere. integration_service.go:176-179 is honest about
it (“the unintegration + cleanup job phases are out of scope here”) and
IntegrationService.Disconnect (integration_service.go:180) has zero
callers in the tree.
None of this fails loudly because yaml.Unmarshal into a typed struct discards
unknown keys. An author writes a phase, the parser drops it, nothing logs.
The consequences are live, not theoretical
Section titled “The consequences are live, not theoretical”- Uninstalling a consumer never revokes its provider-side access. Ten apps wrote
SSO/SMTP unwiring (
mailcow,nextcloud,gitea,wordpress, …) that has never run once. A removed Nextcloud leaves its Keycloak client, its SMTP relay credential and its bucket IAM user live. - The only teardown that DOES run is the vApp path, and it runs
unconditionally:
UninstallVapploopsvsocket.Jobs.Cleanupwith no gate of any kind (vapp_runner.go:437-445). Ten of the twelve vsocket cleanups destroy data:postgres/mariadb/mysqlDROP DATABASE+DROP USER,mongodbdropDatabase,minio/s3bucketrb --force,keycloakDELETE /realms/<realm>,redisSCAN+DELover the whole namespace,frappeandnginxrm -rf. (The remaining two,wordpressandnextclouduninstall-plugin, remove an app, not a dataset.) - The migration case that prompted the audit: a bucket holding 55.6 GiB of replicated Nextcloud data, adopted rather than provisioned, destroyed on teardown with no opt-out.
Of the dead jobs, three destroy data on the edge surface:
apps/minio/s3.socket.yaml, apps/minio-classic/s3.socket.yaml and
apps/s3/s3.socket.yaml, each rb --force. The rest release access, deregister
an agent, unwire a config file, or, in the lone vsocket post_deployment case,
install postgres extensions. That asymmetry is what makes waking the phases
tractable: the blast radius of “parse unintegration” is three jobs wide, and
all three are identified.
This count has already drifted once. It was written as “two, of 22” and
became three when apps/minio-classic/s3.socket.yaml was added, a copy of
apps/minio’s socket, destroy job included, with nothing in the tree that
noticed. A hand-maintained inventory in a proposal is not a control, so the
predicate is now enforced instead of counted:
tests/unit/catalog/unintegration_destroy_gate_test.go walks every committed
.plug.yaml / .socket.yaml, fails on ANY unintegration job that removes a
bucket without lifetime: managed, and pins the inventory by name so the next
provider cannot join silently. Treat the number in this paragraph as history and
that test as the source of truth.
The persistence gap is the cause, not a coincidence
Section titled “The persistence gap is the cause, not a coincidence”This is the single most important fact in the proposal, and it is why the dead phases cannot be fixed by adding a struct field.
VappState persists the entire rendered context (PlugData, VVars,
Discovery, Preset/PresetResolvedVars (vapp_runner.go:70-110)) with the
reason stated inline at :91-94:
“PlugData / VVars are persisted so the cleanup jobs can render against the same context the install used. Without this the
{{ integration.plug.data.domain }}ref in the uninstall job has nothing to bind against and the cleanup runs blank.”
PersistedIntegration (install_runner.go:2925-2946) is five bare strings:
Plug, ProviderStackName, ProviderTenant, ProviderEnvironment,
ProviderConnection. No plug data, no socket data, nothing rendered.
The vApp surface persists its bind-time context and its teardown phase works. The
integration-edge surface does not persist its bind-time context and its teardown
phase does not exist. That is not two independent gaps; it is one gap and its
consequence. A teardown job that says mc rb s3/$${PLUG_BUCKET} has nothing to
render PLUG_BUCKET against.
Parsing the phase without fixing persistence is worse than leaving it dead:
the jobs would run with blank variables. mc rb s3/ and
DROP DATABASE IF EXISTS "" are not no-ops in every implementation, and
rm -rf "$${DOC_ROOT}" with an empty DOC_ROOT is a catastrophe.
The registry the rows would come from is unwired
Section titled “The registry the rows would come from is unwired”Revision 2 assumed the integration table was the natural place to hang
teardown state; it already carries a connection_values JSON column, and
DisconnectRemovedIntegrations (install_async.go) already deletes its rows on
-i <ref>!. Neither assumption survives contact with the tree.
IntegrationService.Connect refuses any edge whose consumer plug and provider
socket are not declared to the repository, and the ONLY writer of those
declarations is IntegrationRepository.RegisterStack
(integration_repository.go:134). Nothing outside _test.go calls it, and
nothing outside the repository’s own zero-value initialiser constructs a
PlugSpec. So on the install path the declarations are always empty, Connect
can only return X404007PLGNOF, and no install ever writes an integration
row. Three consequences follow, and step 6 must be designed around all three:
DisconnectRemovedIntegrationsis a no-op in production. It looks like the natural hook for teardown; it is a walk over an empty table.daemonRecoveryActions.WireIntegrations(daemon/services_recovery.go:312) is the one production caller ofConnect, and it returns the first error; so a recover carrying edges fails on a lookup that cannot succeed. That is a separate bug, but it shares this root and should be fixed with it.ListIntegrationsalready ships astate.jsonfallback for exactly this reason, with the cause named inline: “the thin install path records integration edges only in the consumer’s state.json; it never writes the deployment / integration rows the DB-backed registry reads”. The fallback is what makesshc integration lslook correct while the table is empty.
state.json is therefore the only surface that knows an edge exists, which
is where PersistedEdgeState (step 4) has to live. It is also the surface
ExecuteComposeDown already reads before it tears anything down (st.Integrations
feeds revokeConnectionEdges), so step 6’s uninstall wiring needs no new source
of truth, only the rendered context step 4 adds to it.
Why re-resolving at teardown is not a substitute for persisting
Section titled “Why re-resolving at teardown is not a substitute for persisting”BuildIntegrationsContextForDeployment (integration_render.go:1505) rebuilds
the whole integrations: namespace from state.json plus the deployment’s
app/ copy, and export and backup both use it. It is the obvious shortcut, and
it must not be taken on the teardown path:
- It re-renders plug data against vars as they are NOW, not as they were at bind. That is the drift the Alternatives considered section already rejects.
- For a connection-backed edge (
-i <plug>=@<conn>) it routes throughActiveConnectionMinter.Mint(integration_render.go:691). Calling a cloud mint API is defensible on a render path; on an uninstall path it provisions a scoped credential moments beforerevokeConnectionEdgesrevokes one. - The mint-and-return channel is not replayable at all. Values a socket job
emits as
::shc-connection-value::KEY=VALUE::are folded into the render context in memory (MergeJobMintedValues) and never persisted, so any key that exists only because a job minted it renders empty on a re-resolve. No liveunintegrationjob reads such a key today;apps/crowdsecis the one emitter in the catalog, and its ownremove-bouncerreads plug data, but nothing in the tree stops the next one from doing so.
Design
Section titled “Design”0. Constraint of record
Section titled “0. Constraint of record”The project owner’s constraint, recorded verbatim:
vApps destroy by default, with no opt-out.
This is a hard constraint on the design, not a default to be revisited. Its consequences run through everything below:
- The
core/lifetimeparameter described in §2 governs integration edges only.runVappJobnever consults it. - A
lifetime:key on a*.vsocket.yamljob is a validation error, not an ignored field. There must be no syntax that looks like a vApp opt-out. - The ten destructive vsocket cleanups stay unguarded, and that is now the documented intent rather than an oversight. Revision 1 proposed guarding them; that proposal is withdrawn.
- Adoption of a pre-existing resource is therefore an edge operation, never a
vApp. A vApp instance is its resource:
postgres/databasemeans “this database exists because this vApp exists”. If an operator needs to point at a bucket or database they already own and keep, that is-i <plug>=<provider>withlifetime: external, or-i <plug>=@<connection>. The 55.6 GiB bucket belongs on the edge path, and the fix for that incident is to make the edge path exist and be safe, not to weaken the vApp contract.
1. Two lifecycle events, two names, kept
Section titled “1. Two lifecycle events, two names, kept”They are genuinely different and the distinction is worth preserving:
cleanup: this thing is being uninstalled. The resource is going away. Fires for a vApp instance and for an app.unintegration: an edge is breaking, both endpoints survive. Fires on disconnect and on consumer uninstall.
Under that rule the catalog was misnamed in one direction only: the socket
cleanup blocks (remove-bouncer, remove-agent, and the six cleanup-access
jobs step 0 has since deleted) and the 15 plug cleanup blocks all fire when a
CONSUMER goes away, edge teardown. They become unintegration. The 12 vsocket
cleanup blocks are correct as-is. This rename has landed; see the rollout
below.
The remaining misname is the one the recount exposed: post_deployment on a
vsocket. ce/builtin/postgres/database.vsocket.yaml :: vapp-install-extensions
means “after the vApp’s database exists”; post_deployment on a plug means
“after the CONSUMER’s containers are up and healthy”. Same word, different
event. The job is a second step of provisioning and belongs in jobs.integration
after vapp-create-database, which already runs in declaration order. One file,
one move.
And the parser must stop swallowing this class of bug. yaml.Unmarshal into
a typed struct silently drops unknown keys, which is how 28 jobs (the 22 that
remain plus the 6 step 0 deleted) got authored into the void. The fix is not Decoder.KnownFields(true); the job docs are
partial views of files that also carry schema:, data:, services:, so
strict decoding would reject valid catalog. The fix is to decode jobs into
map[string]yaml.Node first and hard-error on any key outside the surface’s
known phase set. Cheap, exhaustive, and it makes “authored but never executed”
unrepresentable rather than merely tested-for.
2. core/lifetime: a scope parameter, not a flag and not an app boolean
Section titled “2. core/lifetime: a scope parameter, not a flag and not an app boolean”Destruction is gated by one platform-owned parameter, lifetime, resolved
through the existing SHC scope ladder in core/scope (core/scope/scope.go:24-38):
Deployment → Stack → Environment → Tenant → Global. Hence core/lifetime: the
core/ namespace marks it as platform-defined; the platform owns its name, its
value domain, its default and its validation, so no app can misspell it and no
app has to re-implement its truthiness.
Value domain: a closed enum, validated at parse time:
| value | meaning |
|---|---|
external | The resource pre-existed this edge or outlives it. Teardown releases access only. Default. |
managed | SHC provisioned this resource for this edge and may destroy it when the edge breaks. |
Resolution, once, at bind time:
- explicit
--lifetime managedon the install / connect invocation; core.lifetimethrough the scope cascade:shc config set core.lifetime managed --tenant devmakes a scratch tenant reclaim everything, while production keeps the globalexternal;- the plug’s declared default in
<plug>.plug.yaml; external.
The resolved value is persisted on the edge (§3) and that persisted value is what teardown reads. Resolving at bind and freezing it is the same argument as persisting plug data: the disposition of a resource is a fact about how it came into existence, and a config change six months later must not silently convert “retain” into “destroy”.
How a job declares itself destructive. The author marks the job; the platform decides whether it runs:
jobs: unintegration: # Always runs: release the per-consumer credential. - name: revoke-access context: container command: | mc admin policy detach s3 "policy-$${PLUG_BUCKET}" --user "$${PLUG_ACCESS_KEY}" || true mc admin user remove s3 "$${PLUG_ACCESS_KEY}" || true
# Runs only when the edge resolved to lifetime: managed. - name: delete-bucket lifetime: managed context: container command: mc rb s3/$${PLUG_BUCKET} --forceTwo properties follow that the previous designs did not have:
- Release-access and destroy-resource stop being the same job. The
pre-split
delete-s3-bucketdid both in one command, so any guard that skipped the destroy also skipped the credential revocation, the exact leak the whole proposal exists to close. Splitting them is only possible because the gate is per-job. (Landed for all three s3 providers; the shape below is now what the catalogue ships, not an illustration.) - Shell truthiness leaves the blast radius. The runner compares an enum it
parsed; no catalog job ever tests a boolean in
shagain. This repo has been bitten by that class twice in the last week alone: the jinja fix atcore/jinja/jinja.go:865-887(a stringified"False"was truthy, so huly, affine and gitlab addressed a plaintext S3 endpoint over TLS) and its shell mirror,b9c50518“a YAML bool never equals the stringtruein a shell test”. The in-flightcleanup@booleanguards already had to writecase "$${PLUG_CLEANUP}" in true|True|TRUE|1)to survive it. A destructive guard is the last place that should depend on which of five spellings of true a template happened to emit.
Default is retain, and the failure modes are the argument. An orphaned bucket
costs pennies and is swept later; a destroyed one costs whatever was in it. An
edge whose persisted state predates this design, or was lost to a partial
restore, resolves to external, so an unknown edge behaves like one that
declined. That is what makes waking these phases safe rather than merely
possible.
Why the KeepIntegrations flag failed
Section titled “Why the KeepIntegrations flag failed”A KeepIntegrations operator flag was prototyped and has been reverted. It is
worth being precise about why. The first reason is the one that would have
shipped a data-loss bug; the other three apply to any flag-shaped fix, so a
narrower flag in a better place would still have been the wrong answer.
-
It was structurally in the wrong place. The gate sat inside
ExecuteComposeDown, below the vApp early-return atinstall_runner.go:1440-1442:if (strings.Contains(opts.AppName, "/") && !IsExplicitInstallSource(opts.AppName)) ||strings.Contains(opts.StackName, "/") {return UninstallVapp(ctx, opts)}Every vApp uninstall returns there. The vApp path is the only path in the product that currently runs destructive teardown (
vapp_runner.go:437-445). So the flag guarded exactly the code paths that had nothing to guard, and never once fired on the path that did, while reading, in the CLI help and in review, as though it protected everything. A flag that is silently inert on its most important case is worse than no flag: it converts an unguarded destroy into an unguarded destroy that someone has already signed off on. -
Per-invocation disposition is the wrong lifetime. A flag is a property of one command. Whether a bucket survives then depends on how the last uninstall happened to be typed. Worse, several teardown paths have no human at the keyboard at all: the lease reaper (
crons_lease.go:199→DefaultLeaseReapTriggerat:348), the clone/rollback seams (daemon/clone_adapters.go:525, 547,daemon/services_seams_clone.go:389, 404) and the post-teardownPurgeStackRowtail all reach uninstall with a synthesised request. A flag they cannot set is a flag that defaults them into whichever behaviour the code happened to pick, and the reaper, which removes deployments nobody is watching, is precisely the caller you least want guessing. -
It asked the wrong actor at the wrong time. The platform does not know whether a resource was created or adopted. Only the consumer’s operator knows, and they knew it at bind time, possibly months earlier, possibly a different person. Asking at teardown time is asking someone to recall a fact nobody wrote down.
-
It left no trace. Nothing was persisted, so nothing could be audited, replayed, reported by
shc ls, or reasoned about before running the uninstall. There was no way to answer “what happens if I remove this?” other than removing it.
core/lifetime inverts all four: it is resolved at bind (2, 3), by the party who
knows (3), persisted on the edge (4), and consumed by the job runner rather than
by the uninstall entry point, so there is no early-return it can sit below (1).
Because it is a scope parameter rather than a flag, it also gains the tier
resolution a flag structurally cannot have: an estate-wide posture, a
per-tenant override for scratch environments, and a per-edge exception, all in
one key.
Why not the app-authored cleanup@boolean convention
Section titled “Why not the app-authored cleanup@boolean convention”Revision 1 proposed a per-plug cleanup@boolean schema field guarded in each
job’s shell. It is strictly better than the flag and it is what the in-flight
guards on apps/minio/s3.socket.yaml and apps/s3/s3.socket.yaml currently
implement. It is still the wrong long-term mechanism:
- It is re-implemented per app, in shell, against jinja’s stringified booleans:
see the truthiness argument above. Every copy of that five-way
casestatement is another chance to write it wrong once, in the destroy path, in a job nobody runs until the day it matters. - It has no scope. There is no way to express “this whole tenant is scratch” or “this estate never destroys” without editing every plug.
- It is invisible to the platform:
shccannot report it, validate it, or refuse a destroy, because it is just another opaque string inplug.data. - It conflates the two halves of teardown, as shown above.
The cleanup@boolean guards were not wasted work; they were the correct
interim state (see rollout step 2), and they marked exactly which jobs got
lifetime: managed. They now sit INSIDE those jobs as a second condition that
can only add retention, so consumer intent and provenance both have to hold.
3. Persist the bind-time plug context; re-render socket data
Section titled “3. Persist the bind-time plug context; re-render socket data”Give integration edges the same bind-time replay VappState already gives vApps:
but not by symmetry alone. The two halves of the rendered context need opposite
treatment, and the reason is a security boundary the vApp case does not have.
Plug data is replayed. Re-rendering it at teardown reads the consumer’s
deployed app/ tree and its vars as they are at uninstall, not as they were at
bind. If vars.postgres.database drifted between the two, the teardown job
targets a different database than the one provisioned; it does not fail, it
succeeds against the wrong resource. Bucket names, database names and usernames
must come from the bind-time snapshot. This is the same contract port mappings
and var overrides already have.
Socket data is re-rendered, deliberately. The provider’s admin credential is
exactly the field you want current at teardown: a rotated superuser password
must authenticate, and a replayed one will not. And an edge can be cross-tenant,
where integration_render.go:507-520 already blanks every @secret socket field
in the consumer-facing copy. Persisting provider secrets into the consumer’s
state.json would hand the consumer’s tenant the very values that redaction
exists to withhold. The provider is by definition still running when a consumer
is being removed (if it is not, the teardown job could not execute anyway) so
re-rendering from the live provider is both safe and required.
Where it goes. Not in PersistedIntegration. That struct is one of three
that are converted positionally: IntegrationSpec
(integration_render.go:50-72), PersistedIntegration
(install_runner.go:2043-2064) and DiskIntegration (disk_lookup.go:297-310),
each with a comment insisting ProviderConnection stay last because
IntegrationSpec(pi) / PersistedIntegration(spec) / DiskIntegration(in) are
direct conversions. Adding a map to one silently mis-maps the others, and
IntegrationSpec is the type -i parses into; it has no business carrying a
rendered blob.
Instead, add a sibling record modelled on VappState, stored as its own key in
the deployment’s state.json (JSON, keyed by name, no positional coupling):
// PersistedEdgeState is to an integration edge what VappState is to a// vapp instance: the bind-time context, replayed at teardown.type PersistedEdgeState struct { Plug string `json:"plug"` Lifetime string `json:"lifetime"` // resolved core/lifetime, frozen at bind PlugData map[string]any `json:"plug_data"`}The three positional structs are untouched. PersistedEdgeState inherits the
same at-rest posture the vApp state file already has for the same reason (plug
data carries consumer-side secrets such as clientSecret and the vApp DB
password); nothing new is exposed, and the provider’s secrets are never written
into it at all.
An edge with no PersistedEdgeState (pre-existing deployments, or a state file
lost to a partial restore) renders integration.plug.data from the consumer’s
deployed app/ tree as a fallback and is forced to lifetime: external. This
is not a compatibility shim; it is the permanent rule for any edge whose
bind-time context is unavailable. Unknown provenance never destroys.
4. One runner
Section titled “4. One runner”runVappJob (vapp_runner.go:586-709), runOneSocketJob
(socket_jobs.go:402-500) and runOnePlugJob (plug_jobs.go:143-211) are three
retypings of one skeleton: default context → validate → render → $$→$
unescape → renderHookEnv → dispatch. The $$ unescape alone is duplicated
verbatim at vapp_runner.go:668, socket_jobs.go:423, plug_jobs.go:164.
They converge only in the last four inches: ExecuteResolvedJob,
renderHookEnv, ResolveVaultAutogenInVars, LoadVars, RenderMapStrings.
There is no shared “build a job template context” function, and the drift shows:
- Different defaults for an omitted
context:: vappservice(vapp_runner.go:597-600), socketcontainer(socket_jobs.go:409-411), plugservice(plug_jobs.go:150-153). Unifying naively changes the meaning of every catalog job that omits the key. - Robustness on one side only. Socket-only:
context: httpprobes (socket_jobs.go:360),max_retries/retry_interval(socket_jobs.go:81-87, 298-301),dockerNetworkAvailablefail-fast withX500963NETUNR(socket_jobs.go:473-474),SocketServiceHostfallback (socket_jobs.go:484), mint-and-return via::shc-connection-value::(socket_jobs.go:239),@secretredaction on cross-tenant edges (integration_render.go:507-520). The vapp runner rejectshttpoutright (vapp_runner.go:602) and has no retry concept at all, which is why a vApp teardown against a half-stopped parent just loses. discoverymeans opposite things. Socket discovery is a readiness probe whose stdout is DISCARDED (socket_jobs.go:307). Vapp discovery is value-producing: stdout is parsed, published as{{ discovery.<job> }}, and it triggers a SECOND render of the vplugdata:block (vapp_runner.go:302, 329).pickJobContainerNetworkis defined invapp_runner.go:730and borrowed by both the socket runner (socket_jobs.go:458) and the render path (integration_render.go:814), whilesocket_jobs.go:10claims it “mirrorsvapp_runner.go::runVappJob”. The dependency runs backwards from the documentation, and the helper hardcodesDockerProjectName("default", …)(vapp_runner.go:757), a single-tenant vApp assumption sitting in the cross-tenant path, with the comment at:752-756admitting it.
Target: one job runner and one context builder, parameterised by lifecycle event
and provider identity. cleanup vs unintegration becomes two values, not two
code paths, and lifetime is checked in exactly one place.
5. The trap that decides the whole refactor
Section titled “5. The trap that decides the whole refactor”vars resolves four ways, two of them ninety lines apart in one file:
| surface | vars is | winner | anchor |
|---|---|---|---|
vplug data: | vvars (+ --var) | the instance | vapp_runner.go:289 |
vsocket data: | SocketVars(parentVars, vvars); vvars fill gaps, parent overwrites | the parent | vapp_runner.go:264, fn at :542-551 |
| vapp job command | parentVars merged under vvars | the instance | vapp_runner.go:630-641, bound at :649 |
| socket / plug job | provider vars for the socket render, consumer vars for the plug render; never merged | n/a | integration_render.go:359-364 / :461-466 |
So inside ONE .vsocket.yaml, {{ vars.password }} is the parent superuser
in data: and the per-vapp user in jobs.*[].command. This inversion is
deliberate and load-bearing. ce/builtin/postgres/database.vsocket.yaml:17
publishes adminPassword: "{{ vars.password }}", and
ce/builtin/postgres/database.vvars.yaml declares its own
password: ref+vault://vapp_db_password?autogen, the per-vapp DB user’s secret.
The comment at vapp_runner.go:534-541 spells out why parent must win there: a
vapp’s vvars password must not shadow the parent superuser password the socket
publishes. Invert it and every vApp provisions its database with the wrong
credential, succeeding, silently, until something tries to log in.
Three consequences, all of which bind work already in flight:
- A shared context builder that picks one merge order silently mis-provisions credentials. Any unification must keep merge order a per-surface property, with a table test pinning all four before a line of the collapse is written.
- The parent-wins order is pinned in two places: the install render
(
vapp_runner.go:264) and the teardown replay (vapp_runner.go:429). Both callSocketVars. A refactor that fixes one and not the other makes install and uninstall disagree about which database they are talking about. - The in-flight
vars.→vvars.rename sweep (17 refs) must not be applied blanket. In postgres and s3,vars.username/vars.passwordinside a vsocketdata:block mean the parent’s admin credentials by design; renaming them tovvars.swaps admin for instance and inverts exactly the case the comment warns about.
Migration / rollout
Section titled “Migration / rollout”Ordered so each step is separately revertable and none silently destroys data. Status reflects the working trees as of this revision.
| # | step | status |
|---|---|---|
| 0 | Delete the six echo-only unintegration jobs (cleanup-access in hockeypuck, immich, paperless, peertube, penpot, postiz). Their whole body was an echo; keeping them means step 6 must reason about jobs that undo nothing, and every count in this proposal overstates the work. The 21 that undo real state stay, as does unintegration in modules/job/service.go’s phaseList; the inventory entry is what keeps shc job ls honest about jobs that exist but do not run. | DONE (uncommitted), 6 files under apps/, 42 lines, no other reference in the tree |
| 1 | Rename the 23 misnamed blocks (8 socket + 15 plug cleanup → unintegration). Catalog-only, still inert. Clean break, no alias. | DONE (uncommitted), 23 catalog files renamed under apps/, plus one stale doc-comment fix in apps/crowdsec/wazuh.plug.yaml. apps/minio, apps/pmm, apps/s3 and ce/builtin/keycloak already shipped unintegration, so nothing to rename there. |
| 2 | Guard the destructive edge jobs. The interim cleanup@boolean convention on the rb --force jobs. | DONE, and superseded by step 5’s catalogue half. Landed at THREE files, not two; apps/minio-classic/s3.socket.yaml arrived after this row was written. The cleanup allow-list survives inside each destroy job as a second, retain-only condition; it is no longer the gate. |
| 3 | Move vapp-install-extensions from jobs.post_deployment to jobs.integration in ce/builtin/postgres/database.vsocket.yaml, and hard-error on unknown jobs.<phase> keys in all three parsers. | TODO, one catalog line, one parser change; closes the class. |
| 4 | Add PersistedEdgeState (plug data + resolved lifetime) and write it at bind. No behaviour change; new deployments start carrying it. Edges without it resolve to external. | DONE, ce 988d4c2 (modules/app/edge_state.go, written into state.json’s edge_states by every successful deploy). Identifiers freeze rendered; @secret fields are held as their RAW templates and re-rendered at teardown; socket data is never persisted (cross-tenant redaction). Mutation-audited: forcing BuildPersistedEdgeStates to nil reddens the install-persists test by name; forcing the re-derive path reddens seven tests with “revoked the WRONG resource, successfully”. |
| 5 | Land core/lifetime: the scope key, the --lifetime flag, the per-job lifetime: marker and its runner gate, and the vsocket validation error that forbids the key there. Convert the step-2 cleanup@boolean guards to lifetime: managed jobs, splitting release-access from destroy-resource. | PARTIALLY DONE. The per-job marker + runner gate are in ce (modules/app/unintegration_runner.go, X400074UNILFT / X200925UNISKP), and the catalogue conversion has landed: all three s3 sockets now ship delete-bucket with lifetime: managed, and minio/minio-classic ship an ungated revoke-bucket-access beside it (apps/s3 mints no credential, so it has no revoke half). Still TODO: the core/lifetime scope key, the --lifetime flag, resolveEdgeLifetime (still pinned to external, so every destroy is skipped by construction), and the vsocket validation error forbidding lifetime: there. |
| 6 | Parse and run unintegration on socket and plug, from the uninstall path while containers are still up, and from Disconnect (which gets its first caller). | DONE, ce 63e44b6 (uninstall leg: RunUnintegrationJobs sited after pre_uninstall, before the compose verb, in ExecuteComposeDown) + 45aae15 (disconnect leg: SnapshotDisconnectedEdges before ExecuteCompose, DisconnectedEdges.Run after it succeeds, before the DB row drop). Plug jobs before socket jobs, per-job authored timeouts, pipeline/steps: shapes refused loudly (X500089UNISHP). A failed job never fails the teardown: it lands on the retained-cleanup channel as X500927UNILEK and the uninstall completes. |
| 7 | Collapse the three runners, with the four vars merge orders pinned by tests first. | TODO |
Steps 3–6 are small and independently useful. Step 7 is the real refactor and should not be bundled with them.
Ordering constraints that are not negotiable:
- 5 before 6. No phase is parsed until the gate that governs it exists.
- 4 before 6. No teardown job runs until there is a bind-time context to render it against, and a rule for what happens when there isn’t.
- 4 lands in
state.json, not theintegrationtable. The table has no install-path writer (see The registry the rows would come from is unwired), so an edge state hung off a row would be written for no edge that exists. - 6 runs before
composeDown, not after. Nine of the 21 live jobs arecontext: serviceagainst the CONSUMER’s own container (nextcloud’socc,gitea’sadmin auth delete,mautic’sconsole cache:clear), and three more arecontext: containerjobs that curl the consumer over the stack network (immich×2,sonarqube). The only point inExecuteComposeDownwhere the consumer is still up is thepre_uninstallhook site; anywhere below the compose verb, each of those twelve silently takes the “could not authenticate, skipping” branch its own author wrote, and exits 0. - 7 last. Collapsing runners while the surfaces still disagree about
varsis how the credential inversion ships.
Risks & mitigations
Section titled “Risks & mitigations”| risk | mitigation |
|---|---|
| Waking the jobs starts destroying data | Only three destroy anything, and all three now carry lifetime: managed, converted BEFORE the phase is parsed (step 5 before 6), with resolveEdgeLifetime still pinned to external so the destroy is closed by construction. tests/unit/catalog/unintegration_destroy_gate_test.go keeps the claim true as the corpus grows. |
| A woken job runs but silently does nothing, and the runner reports success | Three of the 21 live unintegration jobs use shapes NEITHER existing runner parses: context: pipeline + steps: (apps/fleet/smtp.plug.yaml) and volumes: (apps/wordpress/{smtp,sso}.plug.yaml). Dropping volumes: is the dangerous one: wp core is-installed against an unmounted /var/www/html exits non-zero and the job’s own guard then exit 0s, so a no-op is indistinguishable from a clean teardown. Step 6 asserts the RESOLVED ResolvedJobSpec per surviving job, not just that the phase was reached |
| Teardown jobs run with blank vars on old edges | Step 4 first; an edge with no PersistedEdgeState falls back to the deployed app/ tree AND is forced to external, so a blank render can never reach a destroy |
core/lifetime resolves differently at teardown than at bind | Resolved once at bind and frozen in PersistedEdgeState; teardown reads the persisted value, never the live cascade |
A lifetime: key gets added to a vsocket job, creating a de-facto vApp opt-out | Validation error, not an ignored field; the owner’s constraint is enforced by the parser, not by convention |
| A failing teardown job blocks uninstall | Never abort; log per job with X500009INTJOB and continue; the deployment must still be removable when the provider is half-down. Matches the existing best-effort disposition at vapp_runner.go:440-443 |
| Silent no-op looks like success | Loud on failure, and loud on skip: "lifetime=external, retaining bucket <name>", with the resolved lifetime reported before the uninstall runs |
| Release-access is skipped along with destroy | Split into separate jobs (landed); only the destroy job carries lifetime:, and a test asserts the revoke sibling never does |
Unified runner flips an omitted context: default | Pin the three current defaults per surface in tests before collapsing |
Unified context builder picks one vars merge order | Test all four orders first; keep merge order a per-surface property; check both SocketVars call sites |
A future catalog scan misses ce/builtin again | Every catalog assertion runs against the staged tree, and CI fails if the assertion runs against an unstaged apps/ |
Test plan
Section titled “Test plan”- Parser exhaustiveness, as an error not a test. Every
jobs.<phase>key outside a surface’s known set hard-fails at load. The accompanying test asserts the catalog parses clean; run against the staged tree, soce/builtinis included by construction. - Table test pinning the four
varsmerge orders, including the vsocketdata:vs job-command inversion, with thepostgres/databaseadmin-vs-instance password case named explicitly, and bothSocketVarscall sites (install render and teardown replay) covered. - Lifetime resolution: each tier of the cascade wins over the next; the
--lifetimeflag beats the cascade; the plug default beats the built-in; absent everything yieldsexternal; an unknown value is a parse error, not a fallback. - Lifetime freezing: change
core.lifetimeafter bind, tear down, assert the bind-time value governed. - vApp constraint: a vsocket job carrying
lifetime:fails validation; a vApp uninstall runs its destructive cleanup withcore.lifetime=externalset at every tier. - Teardown behaviour: runs on consumer uninstall while containers are up;
releases access unconditionally; destroys only at
lifetime: managed; continues past a failing job; forcesexternaland says so whenPersistedEdgeStateis absent. - Round-trip: install → uninstall with
lifetime: manageddestroys; withexternalthe resource survives and a reinstall re-adopts it, with the same bucket/database name the first bind used. - Disconnect gets its first test alongside its first caller.
Alternatives considered
Section titled “Alternatives considered”Parse unintegration only, leave the vsocket post_deployment dead. Fixes
21 of 22 and leaves one word still meaning two things on two surfaces. Rejected:
the one remaining case is a single catalog line.
Make cleanup the single name everywhere. Fewer renames (12 vsocket files
already use it), but it collapses two genuinely different lifecycle events (“I
am being removed” and “an edge to me broke”) into one word, and a provider must
distinguish them: remove-bouncer unwires one consumer, delete-realm destroys
everything.
KeepIntegrations operator flag. Prototyped and reverted. Four reasons,
detailed above; the decisive one is that its gate sat below the vApp early-return
at install_runner.go:1440-1442, so it never fired on the only path that
currently destroys anything.
App-authored cleanup@boolean in plug data. Better than the flag, and the
correct interim state, but it re-implements a destroy gate in shell once per
destructive job against jinja’s stringified booleans, has no scope, is
invisible to the platform,
and cannot split release-access from destroy-resource. Superseded by
core/lifetime; the in-flight guards convert.
Guard the vsocket destroyers too. Revision 1 proposed this. Withdrawn; it contradicts the owner’s constraint that vApps destroy by default with no opt-out.
Re-render plug data at teardown instead of persisting it. Cheaper, and it is
what a narrow fix does, but it binds teardown to vars as they are at uninstall
rather than at bind, and silently targets the wrong resource after var drift.
Retained only as the forced-external fallback for edges with no persisted
state.
Persist socket data alongside plug data, mirroring VappState wholesale.
Rejected on two counts: a rotated provider credential must be current at teardown
or the job cannot authenticate, and on a cross-tenant edge it would write the
provider secrets that integration_render.go:507-520 deliberately redacts into
the consumer’s own state file.
Deferred
Section titled “Deferred”pre_uninstall / post_uninstall app hooks. Both are fully wired
(types.go:97-98, hooks.go:29-30, install_runner.go:1471, 1550,
service.go:654, 659) and used by zero apps in either tree. Revision 1
proposed folding them into an app-level cleanup. Deferred: they carry a real
before/after-teardown ordering distinction that cleanup does not, they are not
part of the integration surface, and folding two working-but-unused hooks is
churn that would enlarge an already-large change. Revisit once the first app
needs one.