The apple runtime (design)
Status: open design (SHC-1200). The executor is built in ce and linked into every shcd (ce
daemon/imports.go), and a node runs it only on macOS: the gate isexecutor.RunsOn(cecore/executor/executor.go, SHC-1211). A Mac node also brings up mesh0 and shc0 on utun devices, guards its node plane with pf and runs replica projects (SHC-1287, SHC-1623, SHC-1642, SHC-1557), which sections 5.2 and 7.2 once placed after the cutover. A restore that names--runtime applelands on a node that runs the apple executor (SHC-3155). The healthcheck runner still lives in the executor althoughmodules/nodeagentnow exists. Clay ruled on 2026-10-01 that apple is part of the architecture now, beside docker, podman and incus (memory: shc-runtimes-docker-podman-incus). This note is the contract for cecore/executor/appleand the port of shcd to a macOS node. It does not edit the simplification proposal; the deltas it owes that document are in section 9.
Written against apple/container 1.5.0 (2026-09-29): its docs and Sources/
at that tag, read 2026-10-01. Every flag below was checked there. Upstream
issue and PR states were read on 2026-10-01.
1. What apple/container is, as it bears on an executor
Section titled “1. What apple/container is, as it bears on an executor”Each container is its own Linux micro-VM under the Virtualization
framework, Apple Silicon only, macOS 26 or newer. The CLI is container;
its state lives in an apiserver that launchd supervises, rooted at an app
root (--app-root, CONTAINER_APP_ROOT).
What exists (1.5.0):
container run -d --name <id> -l k=v --env-file <path> --network <name>[,mac=..] -p [ip:]host:ctr[/proto] -v src:dst[:ro] --mount type=bind|volume|tmpfs,...[,size=,mode=][,readonly] --tmpfs <path[:opts]> --dns <ip> --dns-search <d> --dns-option <o> -u <user> -w <dir> --read-only --shm-size --init --platform --cpus <n> --memory <size> --entrypoint <cmd> -i -t <image> [args]. Also--cap-add,--cap-dropand--ulimit, applied inside the container’s own VM, and--runtime <handler>, which picks the runtime helper the host apiserver launches.container network create [--subnet <cidr>] [--subnet-v6 <cidr>] [--internal] [--label k=v] <name>. Networks are isolated from each other, every network carries IPv4 and IPv6 (a v6 subnet is allocated when none is given), and the host reaches every container address.container network list --format jsonprints each network’sconfiguration(modenat or hostOnly,ipv4Subnet,ipv6Subnet,labels).container volume create [--label k=v] <name>: an ext4 image, mounted as a block device into the VM.container list --all --format json,container exec [-i] [-u] [-w] [--env-file] <id> <cmd...>(answers the process’s exit code),container stop [-s SIG] -t <seconds> <ids>,container delete [--force] <ids>,container start <id>.container runfetches a missing image itself.- Label keys must match docker’s grammar: lowercase, digits, dots and
hyphens. Container names match
[a-zA-Z0-9][a-zA-Z0-9_.-]+, at most 63; network names are lowercase.
What does not exist:
- No compose. The executor translates the projection itself.
- No event stream (
container eventsis not a command). - No healthcheck (#1918), no restart policy (PR #1258, tracking #2142), no
exit code:
RuntimeStatusis running, stopped, stopping or unknown. The executor records the exit itself (section 2.7). - No static address (#282; PR #751 “Specify ip in --network” and PR #1740 “sticky IPs” are open).
- No
--hostname,--add-host,--sysctl,--privileged, no host or shared network namespace, no device passthrough, no annotations. - Headless start (#2008):
container system startbootstraps the apiserver plist intogui/<uid>oruser/<uid>and discards launchctl’s failure, so as root outside a login session it fails with a misleading XPC error. The reported workaround islaunchctl bootstrap system <plist>and a secondcontainer system start. PR #2045 only surfaces the bootstrap failure; bootstrapping into the system domain as root outside Aqua is PR #2058. Both are open.
2. The executor contract
Section titled “2. The executor contract”core/executor/apple implements executor.Executor (simplification §4.7).
It registers under the runtime name apple through executor.Register, so
executor.New("apple") answers once the package is linked; the node agent,
the interface’s only caller, blank-imports it.
2.1 Up: a projection becomes network, volume and run calls
Section titled “2.1 Up: a projection becomes network, volume and run calls”The projection is the compose file the leader rendered for this node: one
entry per replica named <service>-r<ordinal>, restart: "no" stamped.
Up first runs the refusal (section 6.1), then converges:
- Every network the file declares is created when absent:
container network create --subnet <ipam subnet> [--subnet-v6 ..] [--internal] --label com.docker.compose.network=<key> [--label com.docker.compose.project=<project>] <name>. A network without aname:is<project>_<key>and carries the project label; one with aname:is shared with other deployments on the node, carries no project label and is never removed by Down. The implicitdefaultis created when a service names it or declares no networks. An existing network is reused only if it carries the declared subnets and mode (internal is hostOnly, otherwise nat); anything else fails Up withX400775APLCMPbefore a container starts, because reusing it would break the pin. - Every top-level volume is created when absent:
container volume create --label com.docker.compose.project=<project> --label com.docker.compose.volume=<key> <project>_<key>, so Down can remove by label. A volume is exclusive to one entry: an apple volume is an ext4 image that one VM mounts at a time, so a projection where two entries mount the same volume is refused, and so is a volume with its ownname:, which another deployment could mount. 32 of the 107 catalog apps share a volume between services today; on apple such a volume has to reach the projection as a host directory bind (virtiofs is shareable across VMs). That is the render’s change to make, and until it does, placement keeps those apps off Mac nodes. Each container records the volumes it mounts inshc.volumes, and before Up starts any entry it removes the orphans and every container that holds a volume another entry now mounts, so a volume renamed or moved between entries is never attached to two VMs. - Entries start in
depends_onorder. Each container is named<project>-<entry>and carriescom.docker.compose.project,com.docker.compose.service, the user’s labels, the healthcheck asshc.health.*labels, the stop grace and signal asshc.stop.*labels, andshc.spec.digest, a sha256 of everything the engine is handed. A compose label with any of those keys is refused: it would move a container, network or volume into another project’s ownership. Each start or recreate Up (or the agent’sStart) makes is a new generation of the container, separate from its digest (arestart: truerecreate keeps the digest). The generation begins before the old container is touched: its word is discarded, its runner is stopped, and no runner is admitted for that container until the new one has started and a listing begun after that has seen it, so no probe of the old container, and no row describing it, can count for the new generation. - Up is idempotent by that digest. A running container with the same
digest is left alone. A stopped one is
container started, except a one-shot (shc.lifecycle=one-shot) with the same digest, which has run for this revision and is left alone (section 3.2). A changed one iscontainer delete --forced and run again. A container with the project label the projection no longer declares is removed (compose’s--remove-orphans). - Before an entry starts, each
condition: service_healthydependency must report healthy: Up runs that dependency’s healthcheck throughcontainer execwith the runner of section 3.1 until it is healthy, and an unhealthy dependency fails Up before the dependant starts, asdocker compose updoes. Only a word about the dependency’s current generation counts, so a dependency Up just recreated is probed again. An entry whoserestart: truedependency Up started in this pass is recreated. Arequired: falsedependency the projection does not carry is skipped, and one that turns unhealthy releases its dependant, as compose does. - Environment never rides an argv (SHC-1176):
environmentandenv_file(one path, a list, or{path, required, format}mappings) merge into a mode-0600 file in a mode-0700 private directory, handed as--env-fileand deleted after the call. Env files are read with compose’s dotenv rules (export, single quotes literal, double quotes with\n \t \\ \" \$, unquoted trimmed and cut at#) or verbatim withformat: raw. Inlineenvironmentvalues and every label (on a service, a network or a volume) are read by the same rule: a value that would interpolate a variable is refused by name before anything runs, since the projection is already rendered, and$$is a literal$, as docker hands it over. A bareKEYwith no value is refused; the daemon’s environment is never copied in. A bareNAMEline in an env file, which compose reads as “unset”, is refused too: the CLI can set a variable but never remove one the image declares. A value with a line break or NUL cannot ride an env file and is refused before any engine call.
Every refusal Up can hit is also what apple.Refuse answers: it runs the
key classification and the whole translation (everything but reading env
files), so placement refuses an empty entrypoint or command (apple
cannot clear an image’s), a mistyped size or an undeclared volume, instead
of Up failing on the node.
Memory is rounded up to whole mebibytes (--memory 512M), CPUs up to whole
vCPUs (--cpus 1 for cpus: "0.5"); a VM takes nothing finer.
2.2 Ports and expose
Section titled “2.2 Ports and expose”Ruling 2026-10-01: ports: and expose: in an app’s compose only declare
the container ports a service listens on. The executor publishes a port
only from a long-form entry with mode: host and published:, which is
what the render writes for --port or a load-balancer connection:
-p [host_ip:]published:target[/protocol]. Any other ports: entry in a
projection (a short form, or a long form without mode: host and
published) means the render left an app’s own port in, which docker
would publish; apple refuses it by name rather than ignore it. An app’s
own ports reach the projection as expose:.
2.3 Down, Ps, Exec
Section titled “2.3 Down, Ps, Exec”Down needs no compose file. It lists the project’s containers by label,
stops them grouped by stop signal and grace (container stop [-s SIG] -t <grace> ids, grace from shc.stop.grace, default 10 s, DownOpts.Timeout
overriding every grace), deletes them, deletes the project-labelled
networks and, with Volumes, the project-labelled volumes.
Ps lists by label. State is apple’s own word. For a container that is
not running, ExitCode and FinishedAt are what the exec helper recorded
when its first process ended (section 2.7), and ExitUnknown (-1) and no
time where it left no record, because the CLI reports neither; nothing
above the executor may read a stopped apple container with no record as
“exited 0”. Health is
"" for a container without a healthcheck or one not running, otherwise
the runner’s last word, starting until the runner has probed.
Exec is container exec [-i] [-u user] [-w dir] [--env-file path] <id> /.shc-bin/exec-env --exec-keep-env --cap-shed CAP_NET_ADMIN --cap-shed CAP_SYS_ADMIN -- cmd..., env through the same private file. The helper
stands in front of the command only to give up the capability every run
adds (section 2.8) and the one a named service’s run adds (section 2.9),
and then becomes the command with the environment the CLI gave it.
2.4 Events: polling, deaths, and the healthcheck runner
Section titled “2.4 Events: polling, deaths, and the healthcheck runner”Events polls container list --all --format json every
Options.PollInterval (default 2 s). A container seen running that is no
longer running is an EventDie, and so is a container first seen stopped
after the subscription began: it started and stopped between two polls. A
container is followed run by run: one that reads another startedDate or
another spec digest than it did is another run, whatever was seen of the
one before, so a run that began and ended between two polls of a container
already seen stopped is a death too, and a run first seen stopping is
followed to its end. A run replaced while it was up sends no death: its
container is up again under the same name. A
death carries the exit code and the time the helper recorded, 137 for a
container a stop of the executor is killing at that moment, and
ExitUnknown without either. Every running container that carries shc.health.*
labels gets a runner (section 3.1) whose transitions are EventHealth with
docker’s words, counting start_period from the container’s own
startedDate. A container recreated under the same name (a new digest) or
restarted by Up or Start (a new generation) gets a new runner from
starting, admitted only from a listing begun after the replacement
finished. A runner is
stopped, and its last report finished or dropped, before its container’s
death is sent, so no health word follows a death. One window remains: a
word the old runner validated just before a replacement can still reach
the channel after it, always followed by the new runner’s starting,
which a consumer takes as the container’s state; the dependency gate
never reads events, only the generation-scoped word.
The channel closes when a listing fails or the context ends; the caller subscribes again and re-reads Ps. Polling can still miss a container that died and was recreated between two polls, so the node agent does not rely on events alone: on every resubscribe, and on a periodic resync, it reconciles the desired replicas against Ps (missing, stopped, wrong digest) and acts on the difference.
2.5 How the fabric pin is realised
Section titled “2.5 How the fabric pin is realised”The CLI cannot pin an address, so on an apple node the subnet is the pin,
and the address is observed. The leader reserves each network’s subnet
out of the node’s grant (ruling 2026-10-01: node-local deployment networks
from 10.64.0.0/10, the fabric from 10.128.0.0/10 with a bridge per tenant
per node) and writes it as the network’s ipam subnet. apple assigns each
container’s address inside it; the list row reports it, and the node agent
carries it into the node report (the one field executor.Container needs
for it is added by the agent’s spine, not here).
What needs only the subnet is rendered from the reservation: routes and the
peer’s AllowedIPs. What needs the exact address is rendered from node
reports: the resolver’s zone, Traefik’s backends, and the policy
compiler’s per-service rules (section 5.4), because a tenant’s replicas
share one subnet and “port 5432 on that subnet” would open the port on
every replica, including services that never declared it. When an observed
address changes (a recreate), the node report changes, and the resolver,
Traefik and the pf anchor are re-rendered, coalesced to at most one
revision a second as the health ruling requires. That window fails closed
(section 5.4): an address has no permission until a revision names it,
and a retired address loses its rules and its pf states before a new
container may start on that network, so a reused address never inherits
another service’s permissions. The pin ledger row for a
replica on an apple node is filled by the node, not chosen by the leader.
ipv4_address on a service is refused by name so a render that still
stamps it fails loudly.
2.6 Refusal at placement
Section titled “2.6 Refusal at placement”An executor that cannot run every compose key implements
executor.Refuser. executor.Refusals(runtimes, project, compose) asks
each runtime present in the snapshot and returns, per refusing runtime, the
refusal’s text. The demand builder puts it on
placement.ReplicaDemand.Refused, and Place narrows every node of that
runtime out with a runtime reason that names the keys. A cap_add
replica therefore lands on a docker node, or pends with “service app-r0
cap_add: …” against each Mac, instead of failing at Up. Up runs the same
refusal again before touching the engine.
2.7 Exit status: the exec helper is every container’s entrypoint
Section titled “2.7 Exit status: the exec helper is every container’s entrypoint”The CLI reports no exit code and no finish time, and that kept jobs,
restart: on-failure and service_completed_successfully off Mac nodes.
The executor records both itself (SHC-1300), with the static exec helper
the incus runtime uses (core/executor/incus/execenv).
- The helper. A linux/arm64 build ships in the Mac install at
lib/shc/exec-env/linux-arm64/exec-envunder the install prefix (a container is a Linux micro-VM, so the build is linux/arm64 whichever Mac it is). Its directory is shared read-only into every container at/.shc-bin(--mount type=bind,...,readonly; apple shares directories, not single files) and the helper is the container’s--entrypoint. A Mac whose install lacks it starts nothing. The mount sits directly under/and not under/.shcbecause every exec runs this path while it still holds a capability the helper is there to take away (section 2.8): a root process in a container can rename/.shcand put another directory in its place, while a mount point cannot be renamed, unmounted, written or covered from inside (measured on apple 1.5.0). - The argv. The CLI can replace an image’s entrypoint but cannot put a
program in front of it, so the executor reads what the image declares
(
container image inspect, the variant for the entry’splatform, linux/arm64 without one, and that variant exactly, its variant word included: linux/arm64/v9 is not the v8 the node holds, and the one equivalence is OCI’s own, arm64 with no variant is v8. An image the node does not hold, or holds for another platform only, is pulled with--platformfirst, and one with no such variant is refused) and hands the helper the argv docker would run: the compose entrypoint and command, each the image’s where the compose file gives none, and the image’s command dropped where the compose file gives an entrypoint. With a compose entrypoint the image is not read at all. The run iscontainer run ... --entrypoint /.shc-bin/exec-env --cap-add CAP_NET_ADMIN <image> --supervise --keep-env --cap-drop CAP_NET_ADMIN -- <argv>(the capability and its drop are section 2.8’s). No compose sits between the projection and the engine here, so the executor does compose’s part itself: an entrypoint or command written as one string is split exactly as compose splits it (executor.SplitWords, compose’s own parser kept rule for rule),$$is a literal$, and a word that would interpolate a variable is refused. - The environment is the runtime’s own: the image’s
ENVwith the--env-fileover it.--keep-envtells the helper to hand its child exactly the environment it was started with and to read no staged file (on incus it builds docker’s environment from one, because Incus gives a first process its own). - What the helper does: starts the argv as its child, passes every signal
it gets on to it, reaps whatever dies under it (it is process 1, so
orphans are its to reap), waits, writes
<exit code> <finish time>to/.shc/status/exitthrough a rename, and exits with that code: the command’s own, 128 plus the signal for one a signal ended, 126 or 127 for one it could not start.container stop -s <signal>signals the container’s first process, which is the helper, so the entry’sstop_signaland its whole grace reach the app. It keeps two signals for itself: SIGURG, which its own runtime uses, and SIGPIPE, which its own writes raise when an attach client has hung up (docker’s signal proxy leaves both out), sostop_signal: SIGURGandSIGPIPEare refused (SHC-3428). A container the CLI reads as stopping is still inside its grace: its death is sent once it has stopped. - The record’s home is a directory of the container’s own on the Mac,
<state>/apple/status/<container>, shared read-write at/.shc/status. It is mode 0777 under a 0700 parent: a shared directory carries the Mac’s owner into the VM and the entry’s user is whoever the image or the compose file names, while nothing else on the Mac can reach it. macOS has no cheap way to cap a shared directory’s size, so unlike incus’s page of RAM this one is bounded only by what the executor reads from it: one regular file, opened without following a link, no further than a record is long. The executor empties the directory before every start it makes (a run, the start of a stopped container, the agent’sStart), the helper clears the record again as it starts, and a record whose file was written before the container’s last start does not read: that is the file’s own time on the Mac, not the time in the record, which is the VM’s clock. The directory goes with the container. A compose mount at or under/.shcor/.shc-binis refused. - The last start. The runtime lists a container’s
startedDateonly until its API service next restarts: it reads its containers back as stopped with no start date (ContainersService loads them withstartedDate: nil, and its exit handler never clears one), which is every container after the Mac reboots. The record is on disk and outlives that, so the executor keeps each container’s last start itself, in<state>/apple/status/.started/<container>(the 0700 parent, no container’s name): every start it makes, noted before it makes it, and any later one a listing shows. A record is read against the later of the listed start and the kept one. With neither, no record is read: whether one is this run’s cannot be told, and that is an unknown exit. So a job that finished before a reboot still reads its exit after it, and a dependent that waits on it starts. What is left open: a container started by hand past the executor, whose helper was killed before it cleared the earlier record, on a runtime whose API service restarted before the executor listed that start, reads the earlier run’s exit. - No record is an unknown exit, never a clean one: a helper that was
killed writes none. A
container stopkills what outlives its grace, so after one the executor writes 137 for a container that left no record, as docker reads a killed container. - Process 1: the trade-off, stated. The app was process 1 in its
container and is now the helper’s child (with
init: trueit already was not). A signal it does not handle therefore takes its default action instead of being ignored as process 1’s are, which is docker’s--initbehaviour and usually what an app wants; an app that inspects process 1 or relies on being it changes. There is no opt-out key. An image whose entrypoint insists on being process 1 (s6-overlay v3, which every linuxserver image is built on) exits at once under the helper; the executor watches a started container for a second, in Up and in the agent’sStart, and one that ended with a failure and “can only run as pid 1” incontainer logsis refused by name: X400788APLPID, “insists on being process 1 … it runs only on docker or podman”. Catalog apps known to need process 1 get an ordinary constraint keeping them off incus and apple once selectors land. - Not proven on a real Mac (nothing is installed on the build Mac; the
stubbed CLI and a live test that skips are the proof here): the shape of
container image inspect’s JSON (read asvariants[].platformwith itsos,architectureandvariant, andvariants[].config.config.Entrypoint/Cmd), that a shared directory is writable by a non-root entry user and that a file written through it carries the Mac’s clock, not the VM’s, as its time, that a stopped container still lists itsstartedDate, thatcontainer logscarries the first process’s stderr, and what the helper’s exit does to the VM. Section 7.1 lists the checks.
2.8 TX checksum offload is off in every container
Section titled “2.8 TX checksum offload is off in every container”A service VIP on a Mac is translated by pf (section 5.4): a dial is
redirected to the replica’s pin and, on the replica’s own network, given the
network’s host as its source. The reply comes back through both states. pf
patches the TCP checksum field for each rewritten address as if the field
held a whole checksum. A container’s virtio device offloads TX checksums, so
the reply leaves the replica with only the pseudo-header sum in that field,
nothing completes it on the way, and the client drops what arrives
(InCsumErrors). Measured on a real Mac with tcpdump -vv inside both
replicas (SHC-1989): the field received is exactly pf’s patch of the
pseudo-header sum, and with the offload off in the guests every VIP and
every name answers. A dial by pin is bridged or routed with no rewrite and
was never affected.
So the executor turns the offload off itself, in every container, whoever its entry runs as:
- Every
container runaddsCAP_NET_ADMIN(beside the helper’s own flags), unless the entry’scap_addalready holds it orALL. - Right after every start it makes (a run, the start of a stopped
container in Up, the agent’s
Start) the executor runs the helper as root in the container:container exec -u 0 <name> /.shc-bin/exec-env --tx-checksum-off. The helper turns TX checksum offload off on every interface but loopback (SIOCETHTOOL,ETHTOOL_STXCSUM) and reads it back. The entry’s own user may be anyone, and a user holds no capability in an apple container, which is why this is the executor’s exec and not the first process’s work. A container that has already ended is no failure of its start; one still running with its offload on is, with the helper’s words. - The capability is taken back. As the container’s first process the
helper drops it from its bounding, ambient, permitted, effective and
inheritable sets before it starts the entry (
--cap-drop). The CLI gives every exec whatever the run added and has no flag to hold one back, so every Exec runs through the helper (section 2.3), which drops it the same way unless the helper that supervises the entry is root and still has it in its bounding set: the entry that declared it. That helper is found in/procby its argv, process 1 or the child of the runtime’s init underinit: true; only its bounding set is trusted, which no process can grow. - A container run before this lacks the capability, so its spec digest differs and Up recreates it.
What this leaves, stated:
- An entry whose first process is not root holds no capability, and
cannot shrink its bounding set (that takes
CAP_SETPCAP). ThereCAP_NET_ADMINstays reachable through a set-uid binary in the image, asCAP_NET_RAWalready is in every container. - An exec that starts in the instant before the supervising helper has dropped the capability keeps it.
- TCP segmentation offload goes off with the checksum offload, so a bulk send from a Mac replica costs more CPU than it did.
2.9 A declared hostname is set with a capability the helper gives up
Section titled “2.9 A declared hostname is set with a capability the helper gives up”The CLI has no --hostname, so the helper names the VM before the entry’s
process: sethostname(2) for hostname, setdomainname(2) for
domainname. Both need CAP_SYS_ADMIN, and Apple’s VM gives a container’s
first process none unasked: on a real Mac a service with hostname: exited
125 in a loop, “the hostname cannot be set: operation not permitted”
(SHC-1964). Measured on apple 1.5.0: root holds a capability the run adds
in its permitted, effective and bounding sets, a user holds nothing
whatever the run adds, and an exec as root holds what the run added.
- The run of a service that declares
hostnameordomainnameaddsCAP_SYS_ADMINbeside the helper’s own flags, unless the service’scap_addalready holds it orALL. A service that declares neither is run without it. - The helper drops it once the VM is named, the way it drops section 2.8’s
capability (
--cap-drop), so the entry’s process holds what the service asked for and nothing more. Every Exec sheds it too (--cap-shed), unless the entry holds it. - Root only. A first process that is not root cannot name the VM, so the
service is refused by name, X400774APLKEY: a compose
userbefore placement, an image’s user at start.cap_addandextra_hostsare refused the same way for the same reason. - A named container run before this never started. Its spec digest differs, so Up runs it afresh.
What this leaves is what section 2.8 leaves for its capability: an exec that
starts in the instant before the supervising helper has dropped
CAP_SYS_ADMIN keeps it, and under init: true Apple’s own init keeps it.
The third case there does not arise here, since a first process that is
not root is refused.
3. What the node agent does that the runtime does not
Section titled “3. What the node agent does that the runtime does not”Ruling: health is defined once in compose, run by the runtime, read by the node agent. apple has no runtime health, so the agent runs it.
3.1 Healthcheck through container exec
Section titled “3.1 Healthcheck through container exec”core/executor/healthcheck is runtime-neutral. A Spec is parsed from the
compose healthcheck: against its closed key set (test, interval,
timeout, start_period, start_interval, retries, disable; any other
key is refused). CMD, CMD-SHELL and string forms are read; disable: true
and [NONE] mean none. Zero values read as docker’s defaults: interval
30 s, timeout 30 s, start_interval 5 s, retries 3. The spec rides the
container as labels, so a runner that finds a running container after a
daemon restart knows what to probe without the compose file.
The Runner follows moby’s daemon/health.go: the next probe is due one
interval after the previous one finished, start_interval instead while the
check is starting inside start_period; a failure whose probe began inside
start_period does not count while starting; retries consecutive failures
after that mean unhealthy; one success means healthy at any time. The apple
executor’s Events owns the runners today. modules/nodeagent exists on
next, and the runner is still to move there, unchanged.
Gap, stated: docker runs an image’s own HEALTHCHECK when compose declares
none, and a compose healthcheck: without test inherits the image’s test.
apple exposes neither to the executor today, so a compose healthcheck
without test is refused, and a service with no compose healthcheck is
ready when running even if its image declares one. Follow-up: read the
image config through container image inspect and run it.
3.2 Restarts and one-shots
Section titled “3.2 Restarts and one-shots”restart: "no" is stamped by the render. The agent restarts a replica on
EventDie or on a reconcile that finds it stopped. The exit code is the
one the exec helper recorded (section 2.7), so a one-shot
(shc.lifecycle=one-shot) that finished is told from one that failed, and
restart: on-failure from a clean exit, as on docker. Up itself still
leaves a stopped one-shot whose digest is current alone, whatever it exited
with: rerunning a job is the agent’s decision, and a new revision (a new
digest) recreates and reruns it. A stopped long-running container is a
death.
3.3 depends_on
Section titled “3.3 depends_on”Up gates starts the way docker compose up does (section 2.1, step 5):
service_started orders, service_healthy waits for a healthy probe,
restart: true recreates. service_completed_successfully waits until the
dependency has stopped and takes only an exit of 0 from its record; one
that failed or left no record fails the up, unless the dependency is
required: false. A
dependency on another node is not the executor’s: the projection only
carries this node’s entries, and the agent’s pre-run check (cross-node
depends_on targets are ready) gates the start.
4. Networks on a Mac node
Section titled “4. Networks on a Mac node”Ruling: a fabric bridge per tenant per node. On apple the mapping is one
vmnet network per tenant per node, the tenant’s slice of the node’s grant
as its subnet, every deployment of that tenant on the node attached to it.
vmnet isolates networks from each other, which gives “cross-tenant blocked”
between networks. Port scoping inside a tenant’s network is pf’s job
(section 5.4) and is unproven: whether VM-to-VM traffic on one vmnet
bridge passes a pf-filtered hook at all is the open question. The render
decides the mapping by what it declares in networks:; the executor
creates what is declared.
The system tier (Traefik and forward-auth, the OpenObserve collector, the per-node resolver) must reach every tenant’s network. The host reaches container addresses, so a system-tier process on the host does; a system-tier container on its own vmnet network reaches another network only through host routing, which is unproven. Proposed, for a ruling: on a Mac node the system tier runs as host processes supervised by shcd, not as containers, until host routing between vmnet networks is proven.
5. The darwin node port
Section titled “5. The darwin node port”5.1 shcd and the apiserver under launchd
Section titled “5.1 shcd and the apiserver under launchd”shcd runs as a system launch daemon:
/Library/LaunchDaemons/com.selfhostedcloud.shcd.plist, RunAtLoad,
KeepAlive, root, state under /var/lib/shc as on Linux, logs to
/var/log/shc. shc node install writes the plist and runs launchctl bootstrap system on it; launchctl kickstart -k system/com.selfhostedcloud.shcd is the restart.
The apiserver has one root-owned app root, /var/lib/shc-apple, and
runs in the system domain only. The app root sits beside SHC’s state home,
never inside it: shc init on a node with no identity, and shc init --force, remove /var/lib/shc whole, and an app root under it loses its
kernel, images and network plists to the first genesis (SHC-1859). The node
never starts a per-user instance: a gui/<uid> instance would answer the
user’s own Mach lookups first and shadow the system one. The sequence
(section 8 spells out the commands):
container system start --app-root /var/lib/shc-appleas root writes/var/lib/shc-apple/apiserver/apiserver.plistwithCONTAINER_APP_ROOTset, then its own bootstrap fails (#2008).- That plist is installed as
/Library/LaunchDaemons/ com.apple.container.apiserver.plist, root:wheel 0644, and bootstrapped withlaunchctl bootstrap systemfrom there. A plist bootstrapped from anywhere else survives logout but is not loaded at boot; one in/Library/LaunchDaemonsis. container system start --app-root /var/lib/shc-apple --enable-kernel-installruns again; its lookup reaches the system instance, and it installs the init filesystem and the kernel.- Verification reads the identity, not just liveness:
launchctl print system/com.apple.container.apiservershows it running withCONTAINER_APP_ROOT=/var/lib/shc-apple, and nogui/<uid>/com.apple.container.apiserveris loaded.
shcd does not set CONTAINER_APP_ROOT: the CLI reaches the system-domain
apiserver over its Mach service, and the app root is the one the
apiserver’s plist names. shcd checks container system status on start,
and refuses work until it answers. When #2058 lands, container system start bootstraps the system domain itself, which removes the failing first
bootstrap; installing the plist under /Library/LaunchDaemons stays,
since #2058 does not install anything that starts at boot.
5.2 The mesh on utun
Section titled “5.2 The mesh on utun”Ruling (shc-mesh-two-wireguard-tunnels): the darwin shcd embeds
wireguard-go in-process, behind the same control code as the Linux netlink
applier. On darwin wireguard-go only accepts utun or utunN as an
interface name, so mesh0 and shc0 are the two tunnels’ roles; each
maps to the utunN the kernel allocates when shcd opens it, and every
route and pf rule is rendered against the allocated name, re-read on each
start. The ruling that the two-tunnel mesh lands after the cutover, with a
Mac node a single-node cluster until then, was overtaken: the mesh is on
next and a Mac node joins over it, as built below. Section 7.1 is still its
single-node acceptance.
As built (SHC-1287, ce modules/fabric): UserspaceApplier is the same
fabric.Applier the netlink applier is, over the darwin host. The peer
diff and the order of an apply are shared with Linux; what differs is
where the writes land.
- The address is set point to point to itself, which is how a utun takes one. macOS gives such an interface no route for its prefix, so the applier adds the pool’s route onto the utun itself. An address with no peer then dies in the device, as on Linux, instead of leaving on the default route.
- Routes are written on a PF_ROUTE socket with the utun as gateway. A per-peer MTU is a locked route MTU; macOS rewrites an unlocked one to the interface’s MTU whenever that changes.
- macOS compares a packet with the interface MTU and reads a route’s MTU
only for TCP’s segment size (xnu
ip_output). So the utun runs at the smallest peer MTU, where Linux runs the device at the largest and lets each route cap its own peer. A Mac node sends every peer packets sized for its narrowest path. - A route is the applier’s when the applier added it, at the MTU it added it. The utun and its routes cannot outlive shcd, so that memory is the whole set; a route an operator puts on the utun is left alone.
- macOS routes a packet for the node’s own mesh address into the utun. shcd hands it straight back, so the node reaches its own listeners (rqlited, the internal API) on its mesh address.
- Nothing finds the device by interface name.
mesh0is found by the address inside the mesh range, or by asking the applier. - The tunnels live inside shcd. When shcd exits, for any reason, the
kernel removes the utun and every route on it; nothing is left to clean
up. On start shcd brings
mesh0back from its cache before the store answers, and peers handshake again within seconds.
5.3 Routing from a utun into a vmnet network: UNPROVEN, the multi-node gate
Section titled “5.3 Routing from a utun into a vmnet network: UNPROVEN, the multi-node gate”A peer’s packet for a replica arrives on the shc0 utun with a destination
inside a vmnet subnet. macOS must forward it: net.inet.ip.forwarding=1
and a route for the subnet onto the vmnet bridge interface. Whether vmnet’s
NAT network accepts forwarded traffic from a utun, whether the reply routes
back through the utun rather than the NAT, and whether --internal
(host-only) networks behave differently, has not been measured. Nothing
multi-node works on apple without it (section 7.2). If it fails, the
fallback is a per-node userspace proxy on the host for inbound fabric
traffic, which this design does not want.
5.4 pf as the policy compiler’s second output
Section titled “5.4 pf as the policy compiler’s second output”ce modules/fabric/policy compiles the tenant policy to an nftables
ruleset. On darwin the same model compiles to a pf ruleset loaded into one
anchor:
/etc/pf.confcarriesanchor "com.shc"after Apple’s ownanchor "com.apple/*";shc node installadds it and loads the main ruleset withpfctl -f /etc/pf.conf, and shcd re-checks the line on every start, since a macOS update can rewrite/etc/pf.conf.- shcd enables pf with
pfctl -Eand holds the reference token it prints for its lifetime (pfctl -X <token>on stop), so it neither depends on nor disturbs anyone else’s enable. - Each revision is one file in the clusterfs
fabricsection, loaded withpfctl -a com.shc -f <file>(the rules go directly intocom.shc, so the main ruleset’sanchor "com.shc"evaluates them). - The rules are the nftables model’s accept classes, by observed address (section 2.5): within a tenant, a replica’s declared ports only; cross-tenant blocked except the system tier; the node plane (mesh, rqlite, internal API) unreachable from any vmnet interface. The anchor is default-deny for vmnet traffic, so an address no revision names has no permission.
- Retiring an address is ordered: when a replica leaves, the revision
without it is loaded and its states are killed (
pfctl -k <addr>andpfctl -k 0.0.0.0/0 -k <addr>) before the agent starts another container on that network, because loading new rules does not end states already established. - A revision that removes an allowance between addresses that both stay
(a port no longer declared, a policy change) kills the states between
those two hosts (
pfctl -k <src> -k <dst>) once it is loaded. macOS pfctl kills by host pair, not by port, so connections the new revision still allows are cut too and re-establish. - IPv6: every vmnet network carries it, and node reports carry IPv4 only today, so the anchor blocks all IPv6 to and from vmnet interfaces, link-local to the node included, until reports and policy are dual-stack.
Unproven, and not promised until measured: that pf sees and filters
traffic between two VMs on the same vmnet bridge. Isolation between vmnet
networks alone would make a cross-tenant test pass without pf, so the
proof (section 7.1) uses same-tenant allowed and denied ports, and reads
the pf rule counters (pfctl -a com.shc -vsr) to show pf, not vmnet, made
the decision. Over IPv6 the anchor blocks everything, so there the proof
is that nothing at all gets through.
5.5 CA trust and the resolver
Section titled “5.5 CA trust and the resolver”The node’s CA goes into the system keychain: security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain <ca.crt>, run by shc node install, so the host’s own clients trust estate hostnames.
Containers get the resolver as --dns <ip> with --dns-search the stack
and environment zones, as on every runtime. The resolver listens on a host
address every vmnet network routes to; which address (each network’s
gateway, or a host alias) is proven in section 7.1.
5.6 Volumes and backup
Section titled “5.6 Volumes and backup”An apple named volume is an ext4 image under the app root, mounted as a block device inside one VM; the host cannot read it, and mounting the image in a second VM while the first has it mounted is unsafe concurrent access to one filesystem. So a capture needs exclusive access. Volumes are exclusive to one entry (section 2.1), so that entry’s replica is the only holder:
- Default: stop the replica, run the restic helper (
container run --rm -v <vol>:/data <restic image> ...) against the volume, start the replica. The whole sequence runs under a maintenance hold on the replica that the node agent honours: no restart on its death, no reconcile and no Up for it, until the helper has exited and the replica is started again. The backup window is a stop, like the stopped-backup path elsewhere. - To prove, for a capture without a stop: freeze the filesystem inside the
owning VM (
container exec <id> fsfreeze -f <mount>), APFS-clone the image file (cp -c), thaw, and run the helper against the clone.
Bind mounts of host directories work (virtiofs) and are what the clusterfs section and app files use; the host reads them directly.
6. What a macOS node cannot do
Section titled “6. What a macOS node cannot do”- Replicated volumes: no LINSTOR or DRBD; out of the first cut anyway
(ruling 2026-10-01). Local volumes and restic. A restore that names
--runtime applelands only on a node that runs the apple executor, and is refused elsewhere (X400955RTMDRV). A replica on a local volume stays pinned to its node when that node is lost, as on Linux. - GPU: nothing passes a GPU into the VM.
- Anything needing the Linux host kernel: the refused keys below, kernel WireGuard (wireguard-go instead, slower), nftables (pf instead), host networking, device passthrough, cgroup tuning.
- Exit codes and events: a death is learned by polling, without a code.
- An image’s own
HEALTHCHECK(section 3.1). - amd64 images run under Rosetta; only arm64 Macs on macOS 26 or newer are nodes at all.
- Vote in raft. A Mac node joins as a non-voter (ruling 2026-10-02, pending Clay’s confirmation on SHC-1287): its tunnel dies with every shcd restart, so a Mac voter would leave raft on each daemon restart or upgrade. Promoting one is refused with a coded error.
- Keep its tunnels across a daemon restart.
mesh0andshc0are down from shcd’s exit until it has rebuilt them from the cache, so the node is off the node plane and its replicas lose fabric reach for those seconds. A Linux node’s kernel devices stay up through a restart.
6.1 Compose keys, classified by full path
Section titled “6.1 Compose keys, classified by full path”apple.Refuse(project, compose) answers X400774APLKEY naming every
refused path with its reason, and then runs the translation, whose
refusals are X400775APLCMP and X400772EXEENV. Placement reaches it
through executor.Refusals (section 2.6), and Up runs it before touching
the engine. A path is in exactly one class; a path in none is refused as
unknown, so a new compose key is honoured on purpose or not at all.
Checked against compose-spec schema/compose-spec.json at commit
914ec15d1fa498969c0df5c1d672306db3256089: every service, network, volume,
mount, port, deploy and top-level key it defines is in one class below.
Honoured by the executor: image, command (not empty), entrypoint (not
empty), environment, env_file (path, list, {path, required, format: raw}), labels (not the reserved keys), ports (only long form with
mode: host and published; keys target, published, host_ip, protocol,
mode, name, app_protocol), expose, volumes (short form with ro/rw;
long form type bind, volume or tmpfs with source, target, read_only,
tmpfs.size, tmpfs.mode written in octal), tmpfs, dns,
dns_search, dns_opt, user, working_dir, read_only, shm_size,
init, platform, stdin_open, tty, cpus, mem_limit,
deploy.resources.limits.cpus, deploy.resources.limits.memory,
networks (list or map; per attachment mac_address), mac_address,
stop_grace_period, stop_signal, healthcheck (section 3.1),
depends_on (condition service_started or service_healthy, required,
restart as a boolean), pull_policy always, missing or if_not_present.
Networks: name, ipam.config[].subnet (one IPv4 and one IPv6),
ipam.driver: default, internal, labels, driver: bridge,
attachable: true, enable_ipv4: true, enable_ipv6: true. Volumes:
labels, driver: local. Top level: services, networks, volumes,
name, version, and x-* everywhere.
Owned elsewhere: of the deploy paths the render’s closed set lets
through, every one but the two honoured limits above, namely deploy.replicas
(placement), deploy.mode: replicated (the render),
deploy.endpoint_mode: dnsrr (the resolver),
deploy.restart_policy.condition (the node agent), deploy.update_config
and deploy.rollback_config (the agent’s rollout), and
deploy.resources.reservations.cpus and .memory (placement). Also
restart (the node agent; the render stamps “no”), attach and
profiles (mean nothing on a node). A deploy path neither honoured
above nor listed here is refused below.
Refused because the CLI has no such capability:
| path | why |
|---|---|
sysctls, privileged, devices, device_cgroup_rules, security_opt, userns_mode, cgroup, cgroup_parent, pid, ipc, uts, oom_kill_disable, oom_score_adj, pids_limit, cpu_period, cpu_quota, cpu_shares, cpu_count, cpu_percent, cpu_rt_period, cpu_rt_runtime, cpuset, mem_reservation, mem_swappiness, memswap_limit, blkio_config, storage_opt, isolation, gpus, group_add, deploy.resources.limits.pids, .reservations.devices, .reservations.generic_resources | a host-kernel or device setting the CLI has no flag for |
deploy.labels, deploy.placement, deploy.mode other than replicated, deploy.endpoint_mode other than dnsrr, deploy.restart_policy keys but condition, any other deploy key | the render refuses them; a projection entry is one replica on one node |
network_mode (any value) | a container cannot share another’s network namespace or the host’s; a deployment is its network |
hostname, domainname on a service whose first process is not root | the helper names the VM, and only root holds the capability that takes (section 2.9) |
extra_hosts, links, external_links, attachment aliases | no --add-host, no alias; the resolver answers names |
attachment ipv4_address, ipv6_address, link_local_ips, gw_priority; ipam.config[].ip_range, .gateway, .aux_addresses | apple assigns addresses inside the declared subnet; the subnet is the pin |
attachment interface_name, priority, driver_opts; network driver_opts, ipam.options, any other driver | vmnet takes none of them |
network attachable: false, enable_ipv4: false, enable_ipv6: false | every apple network is attachable and carries both families |
volumes_from; network and volume external | a VM cannot mount another VM’s volumes; networks are isolated, a shared one is declared by name in both |
volume driver_opts, any driver but local; mount bind.propagation, bind.create_host_path, bind.recursive, bind.selinux, volume.nocopy, volume.subpath, volume.labels, image, consistency, any type but bind, volume, tmpfs; short-form options but ro/rw | virtiofs and ext4 images have none of them; the render creates host paths |
volume name; a volume two entries mount | an ext4 image takes one VM at a time (section 2.1) |
ports entries but long form mode: host with published | the render leaves only those in a projection (section 2.2) |
logging | apple keeps container logs itself; there is no log driver |
annotations | the CLI has no annotation flag |
post_start, pre_stop, pre_start | lifecycle hooks and init containers; the executor runs none |
models, provider (and top-level models) | need Docker’s model runner or provider plugins |
credential_spec | a Windows container setting |
develop | compose watch is a development loop |
use_api_socket | no container reaches an engine socket |
pull_policy: never | run fetches a missing image and cannot be stopped from doing so |
pull_policy refresh, daily, weekly, every_<d>; pull_refresh_after | apple has no pull schedule |
command: [] or "" without an entrypoint, entrypoint: [] or "" | an entry with no command or entrypoint of its own runs its image’s; clearing one is not built |
an entrypoint or command word that would interpolate a variable | the projection is already rendered; $$ is a literal $ |
a volume or tmpfs at or under /.shc or /.shc-bin | the executor shares the exec helper and the entry’s exit record there (section 2.7) |
stop_signal: SIGURG or SIGPIPE | the helper keeps these two for itself (section 2.7) |
an image with no variant for the entry’s platform (linux/arm64 without one) | another platform’s entrypoint is not this one’s (section 2.7) |
| an image whose entrypoint insists on being process 1 (s6-overlay v3, every linuxserver image) | the helper that records the exit holds that place; refused at start, X400788APLPID (section 2.7) |
| an environment value with a line break or NUL | an env file carries one line per variable |
a bare NAME line in an env file | compose reads it as unset, and the CLI cannot remove an image’s variable |
service_healthy on a service with no healthcheck; a required dependency the projection does not carry | nothing to wait for; a dependency on another node is the agent’s pre-run check |
healthcheck without test; any key outside its set | section 3.1 |
env_file format other than raw; an env file, inline environment or label value that interpolates | the projection is already rendered |
labels com.docker.compose.project, .service, .network, .volume, shc.spec.digest, shc.stop.*, shc.health.*, shc.volumes | the executor writes them |
Refused by SHC’s choice, not for lack of a flag:
| path | why |
|---|---|
cap_add, cap_drop, ulimits | the CLI has --cap-add, --cap-drop and --ulimit, applied inside the container’s own VM rather than on a shared host kernel. SHC-1200’s acceptance criteria refuse them; honouring each is a one-line flag if Clay rules so |
runtime | names a docker OCI runtime; apple’s --runtime takes its own handler, and the node’s runtime is placement’s |
pull_policy: build, build, container_name, configs, secrets, extends, label_file, scale; top-level configs, secrets, include, jobs | the render’s: they never reach a projection |
7. To prove on a real Mac, in this order
Section titled “7. To prove on a real Mac, in this order”7.1 A single-node Mac (the acceptance before any mesh)
Section titled “7.1 A single-node Mac (the acceptance before any mesh)”- The bootstrap script of section 8 on Clay’s Mac, then the live
harness: a catalog app (whoami) and an nginx replica with a healthcheck
come up through the executor, the probe reports healthy through
container exec, the host reaches whoami at its observed address inside the declared subnet, and Down removes everything. This closes acceptance criterion 5. - The apiserver bootstrapped into the system domain answers
container system statusin a non-login session and after a reboot, with no user instance loaded. - The resolver, as a host process, answers from inside a container on a tenant network, at the address chosen in section 5.5.
- The system tier as host processes (section 4) reaches every tenant network: Traefik routes to a replica, the collector scrapes one.
- pf (section 5.4): the anchor loads; on one tenant network a declared
port is reachable and an undeclared one is not, with the
com.shcrule counters moving; a second tenant’s network and the node plane are unreachable from a container. That is over IPv4; over IPv6 every destination is unreachable from a container (another replica, another tenant, the node plane, the node’s link-local address), as the anchor blocks it all. Then address reuse: remove a replica holding an established connection, confirm the connection is cut and its state gone, and start a different service that is given the same address; it must hold none of the old service’s permissions. - The healthcheck runner’s
container execcost at a 5 s interval with twenty replicas. 6a. Exit status (section 2.7), with the linux/arm64 helper from the Mac archive in place.SHC_CE_TEST_REAL_APPLE=1runs three live tests intests/_ce/core/executor/hosttest:TestAppleIsAHost(the conformance run, with exit codes),TestAppleRecordsHowEachEntryEnded(exits 3, 5 as a non-root user, 7 from a two-line command, 0 from a job a dependent waits on, 42 on the entry’s own stop signal, 137 for one killed at the end of its grace, read through Ps, Labelled and Events) andTestAppleRefusesAnImageThatInsistsOnBeingProcessOne(a linuxserver image fails its start with X400788APLPID). Then by hand: whatcontainer image inspect alpine:3.22prints (the executor readsvariants[].platformandvariants[].config.config.Entrypoint/Cmd), which process is 1 inside a container started with and withoutinit: true, that the status directory shared from the Mac’s disk is writable by a non-root entry user, and that the VM stops when the helper exits. - The volume image path under
/var/lib/shc-apple, the stop-and-capture restic backup, then the freeze-and-clone capture.
7.2 The multi-node gate (with the mesh)
Section titled “7.2 The multi-node gate (with the mesh)”mesh0on a real utun (SHC-1287). One command, as root, on a Mac that routes nothing overlapping 10.192.0.0/11; it refuses to start otherwise and removes everything it made:tests/_ce/modules/fabric/realmac_macos_test.go, built as the user and run withSHC_MESH_REAL_MAC=1. It proves, against a WireGuard peer in the same process over loopback: the utun is up at the smallest peer MTU with the mesh address; the pool’s route and each peer’s /32 are on it; a peer route’s MTU survives a change of the interface MTU; a matching apply writes nothing; the Mac’s kernel reaches the peer with the mesh address as source and nothing larger than the interface MTU arrives; the peer reaches a socket on the Mac; the Mac reaches its own mesh address; Remove takes the interface and its routes; and a process that dies without removing the device leaves nothing behind.- A Mac node against Linux nodes: it enrols, takes its mesh address,
joins as a non-voter and answers internal RPC over
mesh0;shc cluster mesh diagnoseon a Linux node shows a fresh handshake with it. Restart shcd on the Mac and measure the gap until the handshake is fresh again. - Published ports on a Mac node (SHC-1364): the steps are in that ticket’s design note, and need a container network to forward to.
- Routing from a utun into a vmnet network and back (section 5.3), for a nat network and an internal one.
8. Install and bootstrap on a Mac (Clay’s step)
Section titled “8. Install and bootstrap on a Mac (Clay’s step)”Installing apple/container needs an administrator, so the agent does not
do it. Both steps are fail-stop POSIX sh scripts: save each and run it
with sh. The bootstrap is for 1.5.0 and the #2008 workaround, with one
root-owned app root and no per-user instance; it installs the kernel and
verifies as root from an ordinary shell, proves the instance by running a
disposable container, and can be rerun: an already loaded system service is
reused only when its LaunchDaemon plist is in place, root:wheel 0644 and
identical to the one under the app root, its app root is exactly this
one, and the loaded program is exactly the plist’s. The app root is
/var/lib/shc-apple, outside the state home that shc init removes, so
the bootstrap may run before or after the node’s genesis.
No step runs under launchctl bsexec 1. On macOS 27.0 a process started
that way has no launchd manager (launchctl managername prints “Could not
get manager name.” and exits 153), so container system start fails,
container system status reports the running service as not registered,
and container run fails with “XPC connection error: Connection invalid”.
A root process in an ordinary shell reaches the system-domain apiserver’s
Mach service, and the apiserver refuses any other uid (“unauthorized
request”). container system status asks the caller’s own launchd domain,
so from a shell it is not a check of the system instance;
launchctl print system/com.apple.container.apiserver is. Measured on a
real Mac on 2026-10-07 (SHC-1200).
cat > ~/shc1200-bootstrap.sh <<'SCRIPT'#!/bin/shset -eulabel=com.apple.container.apiserverroot=/var/lib/shc-appleuid=$(id -u)daemon=/Library/LaunchDaemons/$label.plistfield() { sudo launchctl print "system/$label" | sed -n "s/^[[:space:]]*$1 =>* *//p" | head -n 1; }
# 1. install the signed package (Apple Silicon, macOS 26+)curl -fsSLo /tmp/container-1.5.0-installer-signed.pkg \ https://github.com/apple/container/releases/download/1.5.0/container-1.5.0-installer-signed.pkgsudo installer -pkg /tmp/container-1.5.0-installer-signed.pkg -target /bin=$(command -v container)
# 2. no per-user instance may exist anywhere; it would shadow the system onecontainer system stop >/dev/null 2>&1 || truefor d in "gui/$uid" "user/$uid" gui/0 user/0; do if sudo launchctl print "$d/$label" >/dev/null 2>&1; then echo "STOP: $label is loaded in $d; sudo launchctl bootout $d/$label, then rerun" >&2 exit 1 fidonesudo mkdir -p "$root"sudo chmod 700 "$root"
# 3. the system-domain service: reuse one already loaded with this app root,# else write the plist (its own bootstrap fails, #2008), clear whatever that# registered, install it as a LaunchDaemon and bootstrap the system domainif sudo launchctl print "system/$label" >/dev/null 2>&1; then stale="sudo launchctl bootout system/$label, then rerun" sudo cmp -s "$daemon" "$root/apiserver/apiserver.plist" || { echo "STOP: $daemon is missing or differs from the app root's plist; $stale" >&2; exit 1; } [ "$(stat -f '%Su:%Sg %Lp' "$daemon")" = "root:wheel 644" ] || { echo "STOP: $daemon is not root:wheel 0644; $stale" >&2; exit 1; } [ "$(field CONTAINER_APP_ROOT)" = "$root" ] || { echo "STOP: system/$label runs with another app root; $stale" >&2; exit 1; } prog=$(sudo plutil -extract ProgramArguments.0 raw "$daemon") [ "$(field program)" = "$prog" ] || { echo "STOP: system/$label does not run $prog; $stale" >&2; exit 1; } echo "reusing system/$label"else sudo "$bin" system start --app-root "$root" --disable-kernel-install --timeout 5 || true for d in "gui/$uid" "user/$uid" gui/0 user/0; do sudo launchctl bootout "$d/$label" 2>/dev/null || true; done sudo install -m 644 -o root -g wheel "$root/apiserver/apiserver.plist" "$daemon" sudo launchctl bootstrap system "$daemon"fi
# 4. install the kernel through the system instance (never `system start` again:# from a shell it registers a second copy in user/0)sudo "$bin" system kernel set --recommended
# 5. verify: system domain, this app root, no user instance, and a container runs headlesssudo launchctl print "system/$label" | grep -E 'state = running' >/dev/null[ "$(field CONTAINER_APP_ROOT)" = "$root" ]sudo cmp -s "$daemon" "$root/apiserver/apiserver.plist"[ "$(stat -f '%Su:%Sg %Lp' "$daemon")" = "root:wheel 644" ]for d in "gui/$uid" "user/$uid" gui/0 user/0; do if sudo launchctl print "$d/$label" >/dev/null 2>&1; then echo "SHADOWED by $d" >&2; exit 1; fidonesudo "$bin" list --all >/dev/nullsudo "$bin" run --rm docker.io/library/alpine:3.20 /bin/trueecho "BOOTSTRAP OK"SCRIPTsh ~/shc1200-bootstrap.shThe live proof runs the build-tagged harness
tests/_ce/core/executor/apple/live_test.go as root in the same system
bootstrap context, through a wrapper the script writes for go test -exec, from worktrees at the ce and shc merge commits named on SHC-1200.
It fails unless the log holds both --- PASS: TestLiveAppleRuntime and
make exit 0; a missing PASS line means the test did not run.
cat > ~/shc1200-live.sh <<'SCRIPT'#!/bin/shset -euce_ref=$1shc_ref=$2base=$HOME/Projects/bitspur/selfhosted-cloudlog=$HOME/shc1200-live.logwrap=$HOME/shc1200-exec.shgit -C "$base/ce" fetch corpgit -C "$base/shc" fetch corptree() { want=$(git -C "$1" rev-parse --verify "$3^{commit}") if [ -d "$2" ]; then repo=$(cd "$1" && cd "$(git rev-parse --git-common-dir)" && pwd -P) have=$(cd "$2" && cd "$(git rev-parse --git-common-dir)" && pwd -P) self=$(cd "$2" && pwd -P) top=$(cd "$(git -C "$2" rev-parse --show-toplevel)" && pwd -P) head=$(git -C "$2" rev-parse HEAD) dirty=$(git -C "$2" status --porcelain) if [ "$have" != "$repo" ] || [ "$top" != "$self" ] || [ "$head" != "$want" ] || [ -n "$dirty" ]; then echo "STOP: $2 exists but is not a clean checkout of $3 from $1" >&2 exit 1 fi else git -C "$1" worktree add --detach "$2" "$want" fi}tree "$base/ce" "$HOME/wt/SHC-1200-live-ce" "$ce_ref"tree "$base/shc" "$HOME/wt/SHC-1200-live-shc" "$shc_ref"test -f "$HOME/wt/SHC-1200-live-ce/core/executor/apple/apple.go"test -f "$HOME/wt/SHC-1200-live-shc/tests/_ce/core/executor/apple/live_test.go"printf '#!/bin/sh\nexec sudo /usr/bin/env PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin CONTAINER_APP_ROOT=/var/lib/shc-apple "$@"\n' > "$wrap"chmod 755 "$wrap"cd "$HOME/wt/SHC-1200-live-shc"sudo -vrc=0SHC_CE_DIR=$HOME/wt/SHC-1200-live-ce SHC_CE_STORE_BASELINE_REF=corp/next \ CE_PKGS=./core/executor/apple CE_RUN='^TestLiveAppleRuntime$' \ CE_GOFLAGS="-tags=applelive -exec=$wrap -v -timeout=20m" make test/ce > "$log" 2>&1 || rc=$?echo "make exit $rc" >> "$log"if [ "$rc" -ne 0 ] || ! grep -q -- '--- PASS: TestLiveAppleRuntime' "$log"; then echo "PROOF FAILED, see $log" >&2 exit 1fiecho "PROOF OK: $log"SCRIPTsh ~/shc1200-live.sh <ce merge sha> <shc merge sha>~/shc1200-live.log goes on SHC-1200: every container view, the Ps rows,
the health events and whoami’s answer are in it.
9. Deltas owed to the simplification proposal
Section titled “9. Deltas owed to the simplification proposal”For the SHC-1198 owner of 2026-09-30-new-line-simplification.md:
- §3.1 “Runtime becomes a node fact (a node runs Docker or Podman)”: a node runs docker, podman, incus or apple.
- §4.7 “One interface, two implementations”: four;
executor.Newconsults a registry so a runtime in its own package can answer, and an executor may implementexecutor.Refuser. - §4.7
Container.ExitCode: apple reports what the exec helper recorded, andExitUnknownonly where it left no record (section 2.7). - §4.1 placement:
ReplicaDemand.Refusedkeeps a replica off a runtime that refuses its compose keys, with the refusal as the node’s reason. - §3.8 restarts: on apple the agent reconciles desired replicas against Ps rather than trusting polled events alone (sections 2.4 and 3.2); a finished one-shot is told from a crash by the exit the helper recorded.
- §3.10 compose keys: on apple a volume is exclusive to one entry; the render projects a volume several services share as a host bind there.
- §5 feature parity: “healthcheck” is agent-run on apple; an image’s own
HEALTHCHECKis a gap there until the follow-up in section 3.1.