Swarm start-ordering: should the refusal be revisited?
Archived, superseded by the new line (SHC-1092), which removed Swarm.
Swarm start-ordering: should the refusal be revisited?
Section titled “Swarm start-ordering: should the refusal be revisited?”STATUS: PROPOSAL. Nothing here is built. It answers one question: the refusal in
ce/modules/app/swarm_compat.goblocks 38 catalogue packages from the swarm runtime; is that still the right call, and would an SHC-injected wait-wrapper be an honest way out?RECOMMENDATION: keep the refusal, reject the in-container wait-wrapper, and build a daemon-side WAVE DEPLOY instead, behind the acceptance bar in §7. See §6 for why the wrapper cannot be made honest and §5 for the swarm-native primitive that changed since the refusal was written.
Every count below was re-measured on 2026-08-11 against
shc@0e5621a79/ce@0796595, docker 29.4.1 (CLIdocker/[email protected], engine API 1.54). Claims are marked MEASURED, PROPOSED, or PROOF-OWED.
1. The claim that started this, and the correction
Section titled “1. The claim that started this, and the correction”The owner challenged a flat statement that 37 of 91 apps cannot run on swarm:
“why are you saying none of my apps support the swarm runtime? I thought our enrichment thing makes most of them actually work correctly on swarm.”
He is substantially right and the flat statement was wrong. SHC does
translate. ce/modules/app/swarm_compat.go’s ApplySwarmLoaderCompat
rewrites the translatable half of the docker stack deploy reject set and
re-runs until the set closes at zero: $-escaping, long-form
depends_on whose every condition is service_started → list form,
configs.<n>.content → a materialised file:, and a pull_policy drop.
Four of the six constructs that used to make a swarm install die at its
last step are gone.
What is left is not a missing feature. It is a deliberate refusal of exactly two constructs, and the file argues its own case:
service_healthy: “swarm has no start-ordering primitive, so a dependant would start against an unready service”service_completed_successfully: “swarm cannot express run-once-to-completion ordering, so a dependant would start before the one-shot finished”- and the design note that governs the whole file: “a translation that quietly weakens a guarantee would replace a loud loader refusal with a stack that deploys green and is wrong, the same defect this file exists to remove, moved one step later.”
That note is the bar any proposal here has to clear. “It deployed green” is precisely the evidence the current design distrusts, so it cannot be the evidence that retires the refusal.
2. The corrected counts
Section titled “2. The corrected counts”MEASURED. Catalogue at HEAD is 102 packages: 98 under shc/apps/**
(three of them fixtures) plus the 4 built-ins that live in ce/builtin/.
Not 91. The “37 of 91” figure was stale on both halves.
38 packages are refused on the swarm runtime. Counted independently
here (grep for condition: values, then set arithmetic over package dirs)
and cross-checked against the pinned ledger swarmUntranslatable in
shc/tests/unit/modules/app/catalog_swarm_loader_test.go: the two
agree exactly, 38 = 38, with no package on either side only. That
agreement is the reason to trust the number.
2a. By blocking condition
Section titled “2a. By blocking condition”| Blocked by | Packages |
|---|---|
service_healthy only | 7 |
service_completed_successfully only | 20 |
| both | 11 |
| Total refused | 38 |
service_healthyonly (7):graphiti,matrix,blockscout,postiz,journiv,kubo,kutt.service_completed_successfullyonly (20):agent-canvas,dify,librechat,windmill,affine,jellyfin,netbird,frappe,plane,ciso-assistant,defectdojo,defectdojo-classic,fleet,midpoint,midpoint-classic,gitlab-runner,hockeypuck,pmm,traefik,postgres.- Both (11):
superset,medusa,pulp,mailcow,hyperswitch,keygen,immich,openreplay,supabase,greenbone,wazuh.
The two in bold are ce/builtin/. A swarm-runtime estate cannot stand
up its own ingress or its own system store: traefik gates on
traefik-init, which writes the static config file traefik’s own
--configFile points at; postgres gates on postgres-guard, the
fail-closed half of the major-version ladder. That is the sharp end of the
whole finding and it is why this is worth solving rather than documenting.
2b. Edge counts, and what a wrapper would have to satisfy
Section titled “2b. Edge counts, and what a wrapper would have to satisfy”MEASURED, over shc/apps/** (script: parse service blocks, walk every
condition: line, resolve the depended-on service, track Jinja {% if %}
depth):
- 47
condition: service_healthyedges. 0 of them sit inside a Jinja conditional; every one is unconditional. - 76
condition: service_completed_successfullyedges (+2 ince/builtin= 78 total). 3 sit inside a Jinja{% if %}. - 47 of 47
service_healthytargets declare ahealthcheck:block in the same file. There is no edge whose dependency has nothing to poll.
That last number is the strongest fact in favour of a wrapper, and it is why the option deserved a real design rather than a dismissal.
2c. How many would an ordering fix ACTUALLY unlock?
Section titled “2c. How many would an ordering fix ACTUALLY unlock?”Fewer than 38, and the difference matters.
| Packages | |
|---|---|
| Refused today | 38 |
− also carry platform: (untranslatable for a different reason) | −4 |
− mailcow, which declares compose-only on its own merits | −1 |
− jellyfin, refused first by the host-bind mount policy | −1 |
| Genuinely unlocked by a working ordering mechanism | 32 |
- The 4
platform:carriers areblockscout,defectdojo,defectdojo-classic,fleet. MEASURED:platform:never appears alone; all four also carry an ordering condition, so it contributes zero extra refusals but keeps four packages refused after any ordering fix. jellyfinis the one package in both ledgers (swarmUntranslatableandhostBindRefused); the mount policy refuses it before swarm gets a say.- The third refusal branch (
configs.<n>.contentcarrying a live${VAR}) matches 0 catalogue packages, asswarm_compat.goalready says of itself.
So: 32, and two of those 32 are the system ingress and the system store. No package in the catalogue is blocked by something other than ordering alone.
3. A sharper problem sitting next to this one
Section titled “3. A sharper problem sitting next to this one”MEASURED, and not previously recorded anywhere: docker stack deploy
silently discards 16 service properties and warns about 2 more.
docker/[email protected]’s cli/compose/types.UnsupportedProperties is
build, cgroupns_mode, cgroup_parent, devices, domainname, external_links, ipc, links, mac_address, network_mode, pid, privileged, restart, security_opt, shm_size, userns_mode; DeprecatedProperties adds
container_name and expose.
The warning is printed by cli/command/stack/loader.go on the deploy
path only. MEASURED: docker stack config on a file carrying
devices: + privileged: true + network_mode: host prints the keys
back out on stdout and emits NOTHING on stderr. SHC’s stack-config
preflight (X500062SWMCFG, swarm_driver.go) therefore cannot see this
class at all.
Catalogue exposure, service-level keys only:
| Key | Packages | Reading |
|---|---|---|
restart | 79 | benign; swarm’s deploy.restart_policy default is any, and ApplyRestartDefault already skips swarm |
expose | 79 | benign, informational under compose too |
security_opt | 9 | sharp, incl. ce/builtin/traefik and ce/builtin/keycloak |
devices | 1 | sharp, jellyfin (hardware transcode) |
shm_size | 1 | sharp, gitlab |
An app whose security_opt or devices line is dropped deploys green
and is wrong, the exact defect class the ordering refusal exists to
prevent, currently unguarded and one layer down. This is out of scope for
this proposal but should be its own ledger, built the same way
swarmUntranslatable was.
4. Option A: the SHC-injected wait-wrapper
Section titled “4. Option A: the SHC-injected wait-wrapper”The shape asked for: an entrypoint SHC injects into every dependant
container that blocks until its dependency is ready, then execs the
original command.
Where it would be injected. Not the image (SHC does not build app
images) and not an init container (swarm has no such concept; there is no
pod). The only seam is the compose render, in the same slot
swarm_compat.go already occupies: rewrite entrypoint: to a static
shc-wait binary bind-mounted or delivered as a swarm config, with the
original entrypoint/command as its argv tail, and the dependency set
passed as flags derived from the depends_on block being removed.
What it costs.
- A binary in every dependant container. Delivering it as a swarm
configis workable (ApplySwarmLoaderCompatalready materialises config assets), but it must be static, arch-correct on a mixed cluster, and must not collide with images that have no shell. Entrypoint rewriting also breaks images whoseENTRYPOINT/CMDsplit carries meaning, and SHC would be guessing that split from the compose file, not from the image manifest. - A new failure mode: the wait that never satisfies. Needs a deadline, and on expiry the honest action is to fail the task, which swarm will then restart, forever, producing a converge timeout that names nothing. That is the original symptom the refusal replaced.
- ★ It cannot reproduce either guarantee. This is the disqualifier.
Why it cannot reproduce the guarantee.
service_healthymeans “the dependency’s own healthcheck has passed”. A wrapper in a different container cannot run that healthcheck; the check is defined against the dependency’s filesystem and process (pg_isready,mas-cli config check, awgettolocalhost). The wrapper can only poll a proxy: a TCP connect, or an HTTP probe it invented. A listening port is not health; Postgres accepts connections during recovery, andsuperset-initfails on a driver set that is installed-but-not-importable. That is a weakened guarantee, deploying green, wrong. Objection unanswered.service_completed_successfullymeans “the one-shot exited 0”. A wrapper cannot observe another container’s exit code without the daemon API. The catalogue’s one-shots are schema migrations (midpointninja.sh,openreplaypg/ch-migration,medusa db:migrate), volume-ownership chowns (agent-canvas,dify), and fail-closed guards (postgres-guard). Approximating “it finished” by polling for a sentinel file or a schema row is per-app, not generic, and the guard case is worse than approximate: a wrapper that cannot distinguish “guard passed” from “guard crashed” turns a fail-closed guard into a no-op, on a live data directory.- The only faithful implementation needs the daemon API, and that is
already forbidden. Handing every dependant container
/var/run/docker.sockis host root across every tenant on the node; the refusalce/core/volumeenforces and thehostBindRefusedledger pins. A wait-wrapper’s honest form is banned by an existing security policy.
Verdict: REJECT. The wrapper is either unfaithful (port-poll) or prohibited (socket). It repeats the defect the refusal was written to prevent.
5. Option B: what swarm can actually do at docker 29.4.1
Section titled “5. Option B: what swarm can actually do at docker 29.4.1”The refusal was reasoned about swarm’s 2016 answer. Re-measured against the pinned version, three of four facts still hold and one has changed.
| Construct | Result | Evidence |
|---|---|---|
long-form depends_on (any condition, incl. service_started, incl. required:) | REJECTED by the loader | MEASURED: docker stack config → services.web.depends_on must be a list |
list-form depends_on: [db] | Loaded, then DISCARDED | MEASURED: stack config echoes it, but DependsOn appears nowhere in cli/compose/convert/, and swarm.ServiceSpec (moby api v1.54) has no dependency field of any kind |
deploy.update_config.order / --update-order / --rollback-order | not cross-service | MEASURED: within-service task replacement order only |
deploy.mode: replicated-job / global-job | ACCEPTED and CONVERTED | MEASURED: stack config emits it, and cli/compose/convert/service.go:615-626 maps it to swarm.ReplicatedJob / swarm.GlobalJob |
★ The last row is the news. Swarm does have a run-once-to-completion
primitive now, it is expressible in a stack file at the pinned version,
and a job’s terminal task state is complete. swarm_compat.go’s wording
(“swarm cannot express run-once-to-completion ordering”) is right
about the ordering and now wrong about the primitive. That single
correction is what makes Option C possible.
Two further seams, both MEASURED:
docker stack deploy --detach=falsewaits:deploy_composefile.go:73→waitOnServices→service.WaitOnService→ per-service progress untiltask.Status.State == swarm.TaskStateRunningfor every replica.swarm.TaskStatecarries a distinctstartingstate ahead ofrunning(moby api v1.54types/swarm/task.go:26), which is the state a healthchecked container occupies before its check passes.
PROOF-OWED: that swarmkit holds a healthchecked task in starting
until the container reports healthy, i.e. that running is
health-gated, is documented upstream behaviour but was not re-measured
here: swarm is inactive on the dev box (docker info → Swarm: inactive) and this lane does not touch an estate. Option C’s
correctness for the service_healthy half depends entirely on this fact,
so it is item 1 of the acceptance bar.
PAID, MEASURED 2026-08-14 on the bitspur estate (docker 26.1.5+dfsg1, 3-node swarm, the engine production actually runs, not the dev box’s 29.4.1): a probe service whose healthcheck failed for ~86s was observed
Pending→Startingon all 46 consecutive 2-second observations andrunningon none; its task’sStatus.Stateflipped torunningat 20:52:06.971Z, 73ms after the container’s first passing health probe (20:52:06.898Z, from the engine’s own health log). Bar item 1 is GREEN. Evidence:evidence/wave-deploy/bar1-health-gate-bitspur.txt(+bar1-ANALYSIS.txt) in the 2026-08-14 lane records.
6. Option C: the daemon-side WAVE DEPLOY (recommended)
Section titled “6. Option C: the daemon-side WAVE DEPLOY (recommended)”Move the waiting out of the container and into the one actor that already
has a trustworthy view: the SHC daemon. It already runs the deploy, it
already has the docker API, and swarm_driver.go already observes swarm
convergence.
Mechanism (PROPOSED).
- Before
ApplySwarmLoaderCompatstrips it, read the original long-formdepends_ongraph and topologically sort the services into waves. - Rewrite each
restart: "no"one-shot that is the target of aservice_completed_successfullyedge todeploy.mode: replicated-job, a translation that strengthens rather than weakens: it tells swarm the truth about what the service is, and gives its completion a terminal state the daemon can read. - Issue N
docker stack deploycalls against the same stack name, one per wave. Non-job services are cumulative (waves 1..k). A rewritten one-shot appears in its own wave’s artifact only, MEASURED on bitspur’s docker 26.1.5: an identical redeploy of an unchangedreplicated-jobputs a secondCompletetask in the same slot (the job re-executes), while regular services no-op. Carrying an earlier wave’s one-shot forward would re-run a migration while its server boots. Deploy without--prune; absent-from-file means unmanaged, not removed. - Between waves, block on the daemon’s observation:
completefor job-mode tasks, health-gatedrunningfor healthchecked ones. Not onstack deployexit, on the task state. --pruneonly on the final wave.
Why this answers the objection the wrapper could not. Nothing in the
compose file is weakened. The dependency’s own healthcheck stays the
predicate (it is the thing swarm gates running on); the one-shot’s own
exit code stays the predicate (it is the thing swarm’s job completion
records). SHC does not invent a proxy for either. The guarantee is
reproduced by deferring the dependant’s creation, which is the same
thing compose does; compose likewise cannot make swarm order anything; it
simply does not start the dependant yet.
What it costs, honestly.
- Install stops being one atomic
stack deploy. A failed wave leaves a partial stack, though compose already leaves a partial project, and the existing rollback path already handles a partial deployment dir. - Update and rollback need the same wave logic or they will race what install carefully ordered.
- A wave that never converges needs a deadline and a coded error that
names the wave and the blocking service.
X200074SWMPRE’s shape is the precedent. - It does not help the 4
platform:carriers orjellyfin. Ceiling is 32 packages. - Job-mode conversion changes how a one-shot appears in
shc ps/ status roll-ups;deploy_verdict.go’s one-shot folding andswarm_driver.go’s health roll-up both need a job-aware branch, or every migrated stack reportsdegradedforever.
7. Acceptance bar
Section titled “7. Acceptance bar”The refusal stays until all six are green. “It deployed green” is not on this list, deliberately.
- The health gate is real. On a live swarm, a service with a
healthcheck that fails for 60s must be observed in
starting, neverrunning, for that whole window. If this is false, theservice_healthyhalf of Option C is unfounded and only thecompleted_successfullyhalf survives. (This is the PROOF-OWED item from §5.), GREEN, MEASURED 2026-08-14 on the bitspur estate; see the PAID note in §5. - Observable equivalence, positive. A synthetic app: dependency
becomes healthy at T+30s / exits 0 at T+30s; the dependant records the
wall-clock instant of its first line of work and the dependency’s
state at that instant. Under compose,
first_work > dependency_ready. Under swarm-with-waves, the same inequality, over 20 consecutive from-empty runs, zero retries (the standing reliability bar). GREEN, MEASURED 2026-08-14/15 on bitspur after the job-omit fix (v1 failed because wave 3 re-raninit; v3: 20/20 PASS, zero retries). Evidence:evidence/wave-deploy/bar2-waves-20runs-bitspur-v3.txt. - Mutation control. With the wave gate disabled, test 2 must FAIL.
A guard that passes with its mechanism removed proves nothing; this
house has shipped that defect before. GREEN, MEASURED 2026-08-14
on bitspur: with the wave gate disabled the inequality failed
(
init_doneafterfirst_work). Evidence:evidence/wave-deploy/bar3-mutation-control-bitspur.txt. - The two system builtins, by name.
traefikon swarm must serve with a static config whose file mtime precedes the traefik task’s start timestamp.postgreson swarm must be shown refusing a skipped-major upgrade; the guard’s fail-closed behaviour has to survive the translation, not merely run. - The nastiest real app.
midpointon swarm: the server task must not reachrunningbeforemidpoint-init’s job task reachescomplete, proven from task timestamps, not from the stack looking healthy. It is the chosen subject because a midPoint booted against a schema-less database does not fail fast; it sits there retrying, which is exactly the “green and wrong” shape. - The ledger keeps its teeth.
swarmUntranslatableshrinks only by packages that passed 2–5 individually. Any package not proven keeps itsX400073SWMXLTrefusal by name. The ledger is never converted into a permission list.
8. Recommendation
Section titled “8. Recommendation”- Keep the refusal. It is correct today and it is the only thing standing between an operator and a silently-wrong swarm estate.
- Do not build the in-container wait-wrapper. It cannot reproduce either guarantee without the docker socket, and the socket is already refused for good reason.
- Correct
swarm_compat.go’s wording onservice_completed_successfully: swarm does have a run-once-to-completion primitive (replicated-job, MEASURED at the pinned version); what it lacks is declarative ordering. The refusal is unchanged, its stated reason is not. - Build the wave deploy (§6) behind the bar in §7 if swarm-runtime parity is wanted. The prize is 32 packages including the system ingress and system store, but it is worth nothing if item 3 of the bar is skipped.
- Open a separate ledger for §3, the 16 silently-dropped properties. It is the same defect class, it is unguarded, and the stack-config preflight provably cannot see it.
9. Related: the requires.runtimes ruling (2026-08-11)
Section titled “9. Related: the requires.runtimes ruling (2026-08-11)”Landed alongside this proposal, and relevant because it is the field an operator would use to say “this app prefers compose”:
requires.runtimes:is an ORDERED preference list. First entry preferred, later entries acceptable fallbacks; a single entry is a restriction as well as a preference; an app that declares nothing gets the estate default.
MEASURED 2026-08-11: nothing reads the key today: neither
requires.runtimes nor the legacy top-level runtimes:, in either repo.
Three packages declare one (ivatar [compose, swarm],
dependency-track [compose], mailcow legacy [compose]) and four
documents claimed install enforced it; those claims are now corrected. The
canonical reading is app.RequiredRuntimes / app.PreferredRuntime in
ce/modules/app/required_runtimes.go, tested but not yet wired. See
app-definition: requirements.