Skip to content
SHC Docs

A tenant container runtime: what "this app spawns containers" actually needs

Status: built; this note is still its only description. The verb is shc.container.run in ce modules/containerbroker. Written 2026-08-08 while taking gitlab-runner off the host Docker socket, so that the capability gap that work exposed is recorded once, with its real cost, instead of being re-derived badly the next time an app package reaches for /var/run/docker.sock.

Read the status line before quoting anything below as a fact about SHC.

This document is the answer to the question docs/proposals/2026-08-08-in-stack-exec-broker.md §5 explicitly declines to answer. That proposal covers “run a declared verb in a named sibling service”. This one covers the other shape: create containers that do not exist yet, from images chosen at run time. They are different problems and neither document should be cited for the other.


1. The gap, stated as narrowly as it actually is

Section titled “1. The gap, stated as narrowly as it actually is”

An application’s core function is starting containers, from images it does not know until the work arrives, with configuration the platform did not author.

Two packages in the catalogue are this shape:

packagewhat it wants the runtime for
gitlab-runnerthe docker executor: one container per CI job, image named in .gitlab-ci.yml
windmillwindmill-worker docker-mode jobs: hands the engine to user-authored scripts

Everything else that reached for the socket wanted something narrower and has since been moved off it without any new capability: matrix (exec into a distroless sibling → context: service + argv-direct), mailcow (ofelia-mailcow read it and never ran; dockerapi-mailcow only pinged at import).

What is emphatically NOT the gap: the socket itself. Measured on gitlab-runner, three ways:

  • executor = "shell": job ran to Job succeeded with no container runtime present at all, as uid 999, and could not read the 0600 config.toml carrying the runner’s auth token.
  • executor = "docker" + host = "tcp://…": the full docker executor against a sibling engine. Job reached its script and succeeded; the bind-watcher on the host daemon recorded zero job containers. host is a first-class runner config key. The socket was never required to reach a daemon, only to reach this node’s daemon.

So the package could ship without the socket, and now does. What remains unavailable is narrower and worth stating exactly:

per-job container isolation, for a TENANT, on a SHARED SHC node.


2. Why the socket cannot be the answer, restated with numbers

Section titled “2. Why the socket cannot be the answer, restated with numbers”

Not a principle: a measurement. One trivial CI job (GIT_STRATEGY=none) routed through a logging unix-socket proxy produced 225 Docker Engine API calls. Among them, on every job, before any user script runs:

GET /networks → then GET /networks/<id> for EVERY network

On the box this was measured, that walked 60 networks: every other stack’s network, inspected individually, twice per job. GET /networks/<id> returns each attached container’s name and IP. There is no opt-out; it is how the executor decides what to attach to.

The compiled client surface is the whole API: ContainerCreate/Start/Exec/ Kill/Remove, ImagePull/Load/Import/Tag, Network*, Volume*. ImageLoad alone is “push an arbitrary tarball into the host image store”.

A socket bind is therefore not “docker access”. It is read access to every tenant’s topology on the node, plus write access to every tenant’s containers, which is host root by a shorter path than the usual -v /:/host argument. :ro buys nothing: the API is request/response over the socket.

A docker-socket-proxy does not fix this either. The proxy container still needs the raw bind, which the sandbox refuses for a tenant; and filtering HTTP verbs on an API whose object namespace is the whole node still leaves the object namespace the whole node.


A deployment-level declaration: x-shc-runtime: nested in app.yaml, or a package-level capabilities: [nested-runtime]. SHC, not the package author, materialises the sidecar: picks the dind image, sets privileged/security_opt/devices, puts it on an SHC-minted network reachable only from that deployment, never publishes its port, and injects DOCKER_HOST=tcp://<alias>:2375 into the consuming service.

The package never writes privileged: itself. That key stays refused everywhere, which is the point: the privilege is granted by the platform, to a container the platform authored, or not at all.

On a stock kernel a privileged dind container is host root. Rootless dind was tested as the way out and is not one:

docker:28-dind-rootless, no --privileged, with
--security-opt seccomp=unconfined --security-opt apparmor=unconfined
--device /dev/fuse --device /dev/net/tun
daemon boots clean: API listen on [::]:2375, docker version → 28.5.2,
uid=1000(rootless), Privileged=false
first container: runc create failed: error mounting "proc" to rootfs at
"/proc": flags=MS_NOSUID|MS_NODEV|MS_NOEXEC:
operation not permitted

Three retries, same error. SHC nodes have no userns-remap (docker info SecurityOptions: apparmor, seccomp=builtin, cgroupns), so rootless dind cannot start a single container.

Therefore granting this to a tenant on a shared node is equivalent to giving them the socket, minus the SHC daemon’s own API. It is narrowly better in exactly one respect: the 60-network enumeration and the ability to stop neighbours’ containers go away, because the tenant reaches a different daemon.

That one respect is not enough on its own. The capability is only defensible combined with placement:

nested-runtime must imply NODE EXCLUSIVITY: a deployment holding it is pinned to a node hosting no other tenant.

That is the real cost and it should be priced as such: this capability sells a node, not a feature.

Not a var. A tenant-scoped entitlement flipped by a platform admin (the master tenant), recorded in the audit log at grant time and again at every install that consumes it. Node exclusivity enforced at placement, not documented. doctor reds a node carrying a nested-runtime deployment alongside another tenant’s stack.

userns-remap on the SHC daemon, or a sysbox / gVisor runtime class. Neither exists on SHC nodes today. Until one does, nested-runtime is honestly described as “host root, confined to a node we agreed to burn.”


4. Capability 2: an ephemeral-container broker

Section titled “4. Capability 2: an ephemeral-container broker”

The general answer, and the one worth building.

A daemon API (POST /api/method/shc.container.run) accepting (image, argv, env, mounts from a declared set, network = this deployment’s network, timeout) and returning a handle supporting attach / exec / wait / kill. It is context: container (ce:modules/job/context_container.go, today an internal hook executor that already synthesises a temp compose project and runs docker compose run --rm) promoted to an authenticated, tenant-scoped, quota-charged API. The daemon keeps sole possession of the engine endpoint; the caller never sees a socket.

Why gitlab-runner specifically can consume it

Section titled “Why gitlab-runner specifically can consume it”

The runner ships a custom executor, confirmed compiled into the pinned image (gitlab-runner/executors/custom), with config keys config_exec / prepare_exec / run_exec / cleanup_exec (+ *_timeout), the job environment delivered as CUSTOM_ENV_<KEY>, and BUILD_FAILURE_EXIT_CODE / SYSTEM_FAILURE_EXIT_CODE conventions.

Four small SHC-authored scripts in the runner image translate those four stages into four broker calls. No fork, no patch, an upstream-supported extension point.

Bounded by what the broker refuses:

  • image-pull policy per tenant;
  • mounts only from ResolveVolume-approved sources, so @output/../../.. and /var/run/docker.sock are refused by the same one policy, not a second one that has to be kept in sync;
  • attaches only to the deployment’s own network, so GET /networks is not expressible;
  • CPU / memory / count charged against the tenant quota; concurrency and lifetime hard-capped.

A compromised runner gets “spawn containers inside my own blast radius” (which is the actual requirement), not “administer the node”.

A capability on the deployment, granted at install by a platform admin and carried on the deployment row. Checked on every call against the calling deployment’s identity as the daemon sees the peer, not a bearer token in a config file a job script could read. Every call audited with image + argv digest.

Not a socket proxy. A proxy filters verbs on an API whose object namespace is the whole node; the broker’s object namespace is the deployment. That difference is the entire security argument.


  • Swarm. The broker has to place the ephemeral container somewhere. On a swarm estate “the deployment’s own network” is an overlay and the placement node is not the caller’s node. Unresolved, and it is the same unresolved question in-stack-exec-broker.md §4 has.
  • Quota accounting for a container that outlives the request. attach/wait implies a lifetime longer than one HTTP call. Whose quota is charged while it runs, and what reclaims it if the caller dies?
  • Is custom-executor CI actually good enough? Every job stage becomes an RPC round trip. Nobody has measured the overhead against the docker executor on a real pipeline.
  • Demand. Two packages want it (gitlab-runner, windmill), and both ship today without it: the runner on shell, windmill with docker-mode jobs simply unavailable. That is a real but not urgent case. Build it when a third consumer appears, or when a paying estate needs per-job isolation.

gitlab-runner is off the socket and installable, without any of the above. Both declarations are deleted (volumes: entry and the [runners.docker] volumes string in the config heredoc), the default executor is shell, and executor: docker without an operator-supplied dockerHost is refused at install with a message that names this document.

Nothing in this document is required for that, and nothing in this document has been built.