Skip to content
SHC Docs

Cert-delivery broker: handing a deployment the edge certificate for a host it terminates itself

Status: built; this note is still its only description. shc cert delivery shows what was delivered; the certs: declaration, the reload actions and the mount contract are described only here.

Status: PARTLY BUILT, and this document is NOT a description of the shipped surface. The broker LANDED and is WIRED (ce modules/certbroker, 2026-08-07); four of this document’s design decisions were REVERSED or DROPPED on the way in, and it has ZERO adopters. Read the per-section table below before you draft a verification step from anything here.

Originally written 2026-08-06 as a pre-build proposal, when a deliberately unwired skeleton sat on ce branch lane/cert-delivery-broker. Statuses re-verified 2026-08-07 against ce df1e7e4, the pin shc v0.0.27 carries, and shc origin/main. The prose below is left as authored, because it is the record of what was argued; this header is the authority where the two disagree. ce modules/certbroker/doc.go carries the same departures at their site.

This document saysActuallyStatus
A certs: declaration with per-service reload actionsShipped exactly (declaration.go, ParseCertsDecl)BUILT
/etc/shc/tls.serve/<host> read-only platform mount; <StateHome>/certserve/ homes; serial dirs + current symlinkShipped (materializer.go, core/paths/platform_mounts.go)BUILT
posture.json per delivery; requested-vs-delivered classes in the cert_edge vocabularyShipped (posture.go)BUILT
shc cert delivery renders both posture columnsShipped (ce modules/ca/commands/delivery.go)BUILT
The module is an unwired skeleton on a branchLanded and wired 2026-08-07: modules/app builds the grants and supplies the Source/Reloader, subsystems/impl drives Reconcile from the ingress sweepSUPERSEDED
§ Honesty: “Default: staging material is WITHHELD”, and the X409473CRTPOS refusal (lines ~264, ~297)REVERSED by owner ruling. Staging is DELIVERED, labelled, never withheld; “a workload with no certificate is worse than one with a correctly labelled staging certificate”. SEQ 473 was re-minted as X200473CRTSTG, a warn, not a refusal. posture.go states plainly that nothing in the classifier can refuse. Open question 3 (“staging default”) is answered, not open.REVERSED
certdeliver.allow_staging estate config key, and allowStaging: true in the declarationNEVER IMPLEMENTED. No such config key exists in ce; ParseCertsDecl reads only certs[].reload and has no allowStaging branch. The only trace is certbroker.HostGrant.AllowStaging, a struct field that is never set and never read, dead code carried over from this draft.DROPPED
A host: field in the certs: entry (the grammar block, ~line 131)NEVER IMPLEMENTED. CertDecl has one field, reload. The MOUNT is the request; authorisation comes from the ingress intent set, which a package cannot edit. A host: key in an app.yaml is silently ignored.DROPPED
The mount is the request: material goes to a deployment that binds /etc/shc/tls.serve/<host>Until SHC-2414 the sweep delivered every routed host of every deployment with a replica on the node, bound or not. It now grants a host only when a replica there binds its home (modules/app/cert_grant_bound.go). A home already on disk that reads as unbound goes after 15 minutes of reading so.BUILT
shc cert delivery rm <host> (force-remove)Not built. shc cert carries ls, edge, delivery and nothing else.NOT BUILT
Landing notes: “the delivery ledger … is a new ent schema; shc’s DDL mirror must move in the same train”NO ENT LEDGER. Delivered material is node-local disk state; rotation compares source serial to delivered serial. No shc DDL-mirror move is owed.DROPPED
Adoption map: mailcow “First adopter”ZERO ADOPTERS. No app under apps/ declares a certs: block or binds /etc/shc/tls.serve/…, mailcow included. fusionpbx, the ldap slot and mail are likewise unadopted.NOT DONE

★ What shc cert delivery printing no certificate deliveries means. It means the broker has nothing to deliver because nothing declared. That is the UN-ADOPTED state, not a pass. On both estates the command is empty for that reason, and a verification doc that treats the empty list as green is asserting a property nobody has yet armed. The first real signal will come from the first adopter commit, not from this surface.

Traefik owns :80/:443 and the estate’s ACME stores. Any app that terminates TLS on its own published ports never sees the real leaf:

  • mailcow’s postfix (465/587) and dovecot (993/995) serve the SELF-SIGNED pair mailcow-ssl-init writes into /etc/ssl/mail. Measured on bitspur: :443 verifies clean with the LE leaf while 465/993/995 present issuer=CN=mail.corp.bitspur.com, verify code 18. Every real mail client warns, and so does any SHC consumer that verifies the STARTTLS endpoint the smtp/imap sockets advertise (they hand out the PUBLIC hostname).
  • The long comment block at apps/mailcow/compose.yaml (SKIP_LETS_ENCRYPT) states the gap and names the three shortcuts that fail on inspection:
    1. bind-mounting traefik’s ACME store into the stack crosses a trust boundary (the store holds the estate’s ACME account key and every host’s private key) and assumes co-location with the traefik host;
    2. a copied leaf is silently a STAGING leaf on any estate under a staging validation posture; a quiet wrong cert is worse than a loud self-signed one;
    3. traefik TCP routers with a certResolver move 465/993/995 off the stack’s published ports, an ingress-model change, not a package change.

Mail is not the only self-terminating app: FreeSWITCH terminates SIP-TLS (apps/fusionpbx, sipTls: 5061), the ldap capability slot’s future provider will terminate ldaps:636, and anything speaking raw TLS on a published port has the same shape. The comment’s own conclusion is the assignment here: “the right home is a daemon-side cert-delivery surface next to shc cert edge.”

What exists today (survey of the machinery this design composes)

Section titled “What exists today (survey of the machinery this design composes)”

All references are ce origin/main unless marked shc.

  • Traefik is the estate’s ONLY ACME client. go.mod carries no lego / x/crypto/acme; issuance happens inside traefik, keyed by the acme-* certResolvers the render writes, against the providers in acme.providers.<name> (modules/acme/reader.go, proposal 2026-07-16-acme-providers.md). The ACME store JSONs live in a docker volume (“unreadable across runtimes/nodes”, modules/app/traefik_acme_reconcile.go).
  • The daemon already materializes serving certs for traefik. The 60s ingress sweep (modules/app/ingress_materialize.go) signs SHC-CA leaves locally on the traefik-host node and writes cert+key files under <StateHome>/system/traefik/dynamic, records them in the leaf_certificate ledger (ent/schema/leaf_certificate.go), renews within 30d of expiry, and removes orphans for deployments that left the intent set. Private keys on the daemon’s StateHome disk are established practice.
  • Provenance vocabulary exists. shc cert edge (modules/app/cert_edge.go, modules/ca/commands/edge.go) classifies requested-vs-served per host: production / staging / private / other / traefik-default / unprobed. The staging-detection rules (issuer contains “(STAGING)”, directory host acme-staging-*) are the honesty mechanism this broker must reuse, not reinvent.
  • Cross-node cert distribution exists. EdgeCATrust (modules/app/edge_ca.go) derives edge trust anchors on the traefik-host node only and serves them to every other node over the internal mTLS RPC (edge_ca.anchors). The broker’s cross-node path is the same shape.
  • Platform-mediated file delivery exists. The reserved /etc/shc/... container namespace (core/paths/platform_mounts.go): a recipe writes the same string on both sides of the bind, core/volume rewrites the source to the node’s real backing file, unknown entries are REJECTED (fail-closed). Today it carries tls.ca / tls.p12, inert trust material only; the host side (PlatformMountDir) is explicitly forbidden to hold secrets, which is exactly why this proposal does NOT put leaf keys there.
  • In-stack change delivery exists. The mailcow restart: true idiom (shc apps/mailcow/compose.yaml, nginx-mailcow ← mailcow-ssl-init): compose restarts a dependent when its init dependency is recreated, so a changed artifact is actually re-read. Delivery is not done until the consumer re-reads; the broker’s reload contract encodes the same rule.

The contract: a certs: declaration in app.yaml, not a tls socket

Section titled “The contract: a certs: declaration in app.yaml, not a tls socket”

Two candidate grammars were on the table: a cert plug/socket (a platform-served socket, protocol-named tls) or a declaration in the app package. This proposal chooses the declaration. The socket case first, because it is genuinely respectable:

Steelman for the socket: the naming test half-passes. “The publicly-trusted leaf for host H” could be served by a second implementation (an external cert-manager, an operator-pasted pair; an external-tls shim is nameable), and a socket would give consumers typed metadata through the existing integrations.<stem>.* render namespace.

Why the declaration wins anyway:

  1. Renewal does not fit the integration phase model. Integration jobs run on connect and on backup; disconnect runs none (shc docs/app-development/plugs-and-sockets.md). Certificates rotate every ~60–90 days and the delivery must CONVERGE on rotation with no operator step. That is a sweep-reconcile problem, the shape ingress_materialize.go already is, not a connect-time job. A tls socket would bolt a renewal loop onto machinery that has none.
  2. There is no pairing choice to express. The socket grammar’s point is -i plug=provider selection among peer stacks. The edge issuer is a singleton platform capability (the system traefik + acme.providers); there is never a second provider stack to pick. Where a second SOURCE exists (external cert-manager, operator-supplied pair), that is a broker source selected by config, the same way DNS credentials are config, not a socket.
  3. Secret file delivery is not socket vocabulary. Sockets inject render DATA (strings); the deliverable here is a rotating private-key file pair with mount + reload semantics. The estate’s precedent for platform-issued trust material is the platform mount (tls.ca), and the standing counter-example is telemetry: the otel socket was deleted precisely because a direct app↔store pipe bypassed platform mediation (“Not everything cross-cutting is a socket”, same doc). A tls socket would hand an app a capability edge where the platform must stay in the middle.
  4. The durable edge already exists. Deliveries are per (tenant, host, deployment) rows next to the leaf_certificate ledger; an IntegrationModel row would be a second, drifting record of the same fact.

The consumer-visible posture metadata a socket would have carried rides the delivery itself (posture.json, below) and the existing smtp/imap socket consumers benefit with zero schema change: the endpoint they already verify simply starts presenting the real leaf.

DROPPED, both fields. The shipped certs: entry has exactly one key, reload. There is no host: and no allowStaging:; ParseCertsDecl parses neither, and writing either into an app.yaml does nothing. See the status table at the top.

app.yaml
certs:
- host: "{{ vars.hostname }}" # the host this stack terminates itself
# Per-service reload contract, fired AFTER new material is fully
# visible. `restart` bounces the container; a list is exec'd in it.
reload:
postfix-mailcow: restart
dovecot-mailcow: ["doveadm", "reload"]
# allowStaging: true # explicit opt-in only; see honesty §
  • host is jinja over the app’s vars, like socket data: blocks.
  • Omitted reload defaults to restart for every service whose compose binds the granted mount path, the safe reading of the mailcow restart: true doctrine (delivery isn’t done until the consumer re-reads). Declaring an exec is the opt-IN to something gentler.
  • Multiple entries allowed (an app may terminate several hosts).

Delivery mechanism (renewal-safe by construction)

Section titled “Delivery mechanism (renewal-safe by construction)”

Host home. Per-host serving material lives at <StateHome>/certserve/<tenant>/<host>/, deliberately NOT PlatformMountDir (inert-only, node-shared) and NOT the deployment dir (backup capture snapshots deployment dirs wholesale; delivered leaves are regenerable and must never ride a backup onto another estate, the same argument that moved the CA union out of <deployDir>/files/). Precedent for keys on StateHome disk: <StateHome>/system/traefik/dynamic already holds every SHC-leaf private key. Dir 0700, files 0600, root-owned.

Reserved mount. The container-side interface is a new fail-closed entry in the platform-mount namespace:

volumes:
- /etc/shc/tls.serve/{{ vars.hostname }}:/etc/shc/tls.serve/{{ vars.hostname }}:ro

core/volume rewrites the source to the host home, but ONLY when the deployment’s certs: declaration claims that host AND the host is bound to that deployment (ownership check below). Any other /etc/shc/tls.serve/... source is rejected exactly like a typo’d tls.ca is today. Read-only is load-bearing, as with every platform mount.

Layout + atomic rotation. Inside the granted dir:

serial-<N>/fullchain.pem
serial-<N>/privkey.pem
serial-<N>/posture.json
current -> serial-<N> # flipped by rename(2), never edited in place

Consumers configure paths through current/. A rotation writes the new serial-<N+1>/ completely (fsync), flips the symlink atomically, THEN fires the reload contract, then prunes all but one previous serial. A reader can never observe a mismatched cert/key pair, and a reload that races the flip still lands on a complete pair.

Convergence. A broker reconcile runs at the ingress sweep’s cadence (60s) on the traefik-host node: for every granted (deployment, host), read the source’s current material; if the leaf serial/fingerprint changed, rotate as above. Renewal therefore converges within one sweep tick of traefik’s own renewal, with zero manual steps. Fail-soft like edge_ca.go: a wedged source logs a coded warn and leaves the last-delivered material in place; it never blocks a render, an install, or the sweep.

First install / fallback. The broker is a convergence surface, not an install dependency. An app keeps its packaged fallback (mailcow’s mailcow-ssl-init self-signed pair) and prefers /etc/shc/tls.serve/<host>/current/ when populated. Install completes before first delivery; the reload contract delivers the upgrade moments later. No install-order coupling, no new failure mode on an estate with no ACME provider (the grant dir simply stays empty and the app keeps its current, honest self-signed posture).

Cross-node. Material is extracted where the source is (the traefik-host node). A node hosting a granted deployment elsewhere pulls per-host material over a new internal mTLS RPC (certbroker.material), the exact edge_ca.anchors shape. The RPC enforces the same grant check server-side (the requesting node must currently host the granted deployment; node identity comes from the mTLS peer), verifies the chain parses and the key matches the leaf, then materializes into the local host home. Co-location assumption: gone.

The source: traefik’s ACME store, extracted daemon-side

Section titled “The source: traefik’s ACME store, extracted daemon-side”

The v1 source reads traefik’s ACME store JSONs (docker volume, traefik-host node, via the docker runtime; one cached read per sweep tick, invalidated on store mtime) and extracts, per granted host, ONLY that host’s leaf chain and private key.

Why this is not shortcut #1 wearing a suit. The rejected move was handing the APP the store. What the app receives here is its own host’s pair and nothing else; the store never leaves the daemon. And the daemon reading the store grants the daemon nothing it lacks: it already writes traefik’s entire routing config (it could route any host anywhere), holds the SHC CA keys, and fronts the vault. The trust boundary that matters, app ≤ its own host, is enforced at the mount grant and at the RPC, both fail-closed.

Why not a second, daemon-side ACME client (lego/x-crypto-acme, DNS-01 via the dns module): it would double the estate’s ACME accounts, add a heavyweight dependency, interact with duplicate-certificate rate limits, and create two issuance paths whose posture surfaces (shc cert edge, the stall watcher, the acme-providers model) would each need a twin. Traefik staying the ONLY ACME client is a feature. The Source interface in the skeleton keeps the door open; an external cert-manager or a daemon-ACME source is a second implementation behind the same seam, config- selected, but v1 does not build one.

Hosts traefik doesn’t route (mail-only host, ldaps host). The broker registers a cert-claim for the host into the ingress intent set: a synthetic low-priority router entry naming the deployment’s certResolver, routing nowhere useful; its only job is to arm traefik’s issuance for that SNI. The claim participates in host-uniqueness exactly like a real ingress host (one deployment per host), and public_cert_host.go’s attainability verdict applies unchanged, so an uncertifiable name degrades loudly instead of wedging the stall watcher.

Host ownership check. A certs: claim for host H is granted iff H is already one of the deployment’s own ingress hosts, or H is unclaimed and becomes this deployment’s cert-claim. A claim on another stack’s host is refused (X403471CRTSCP). This is what makes the mount rewrite safe to automate.

★ REVERSED BY OWNER RULING: the whole withholding half of this section is dead. Staging material is DELIVERED, labelled by the posture record and a warn (X200473CRTSTG), never withheld. X409473CRTPOS does not exist; certdeliver.allow_staging and allowStaging: do not exist. The requested-vs-delivered posture record and shc cert delivery, the rest of this section, are BUILT and correct as written.

The estate’s ACME posture must be VISIBLE to the consumer, and a staging leaf must never arrive quietly (rejected shortcut #2).

  • Every delivery carries posture.json: the requested resolver/provider/ directory class AND the delivered leaf’s issuer classification, both in the cert_edge.go vocabulary (production/staging/private/other), plus issuer string, notAfter, serial, extraction time.
  • Default: staging material is WITHHELD. If the classified leaf is staging and neither the estate (certdeliver.allow_staging: true) nor the declaration (allowStaging: true) opted in, the broker refuses the rotation (X409473CRTPOS, logged), keeps the last production material if any, else delivers nothing; the app stays on its loud self-signed fallback. A quiet wrong cert is worse than a loud self-signed one; the default encodes that sentence.
  • shc cert delivery (below) renders requested-vs-delivered per host, the exact REQUESTED/SERVED split shc cert edge renders for the edge.
  • Uninstall / host moved: the grant vanishes from the intent set; the next sweep removes <StateHome>/certserve/<tenant>/<host>/ (orphan cleanup mirroring the route-file orphan removal in ingress_materialize.go) and drops the cert-claim route if one existed. removeRenderedArtifacts never touches the host home (it only walks deployment dirs), so the broker owns its own teardown symmetrically.
  • Compromise: rotating the upstream cert (traefik re-issue; actual ACME revocation remains an operator/traefik concern, out of scope here) converges to consumers within one sweep tick via the ordinary rotation path. shc cert delivery rm <host> (admin) force-removes delivered material ahead of the sweep.
  • Backups: the host home is outside every capture path by construction; a restore on another node/estate re-converges from that estate’s own source. Nothing to exclude, nothing to migrate.
# core/config default.yaml: new block
certdeliver:
# Estate-wide opt-in to delivering STAGING-classified leaves to
# declared consumers. Default false: staging is withheld, loudly.
allow_staging: false

Deliberately minimal. No per-provider knobs (the acme.providers model already owns issuance), no enable flag (a declaration IS the enable).

  • shc cert delivery: sibling of shc cert ls / shc cert edge under the existing shc cert subtree (modules/ca/commands/main.go NewCertCommand). Columns: HOST, DEPLOYMENT, REQUESTED, DELIVERED, SERIAL, EXPIRES, RELOAD. Thin dispatcher to GET /api/method/shc.cert.delivery (master tenant, like shc.cert.edge).
  • shc cert delivery rm <host>: force-remove delivered material.

NOT DONE, every row. As of shc origin/main no app under apps/ declares a certs: block or binds /etc/shc/tls.serve/…, mailcow included. “First adopter” below is an assignment, not a record. shc cert delivery is empty on both estates because of this, and that emptiness is the un-adopted state, not a passing check.

AppSelf-terminated TLS todayAdoption
mailcow (shc apps/mailcow)postfix 465/587, dovecot 993/995 out of /etc/ssl/mail (self-signed pair from mailcow-ssl-init)First adopter. Declare certs: [{host: vars.hostname}]; entrypoints prefer tls.serve material over assets/ssl; keep the self-signed fallback; delete none of the honesty comments until the broker delivers on a live estate. smtp/imap socket consumers start verifying clean with no socket change.
fusionpbx (apps/fusionpbx)FreeSWITCH SIP-TLS on published sipTls: 5061, self-generated certs in the volumeSecond adopter; softphone clients verify the SIP domain.
ldap slot (decision 2026-08-05-ldap-socket.md)future provider terminates ldaps:636Broker is the prerequisite that makes a public-hostname ldaps provider shippable.
mail (apps/mail, mailpit)no STARTTLS todayOptional later: broker material would let the dev-mail stack offer STARTTLS honestly.
wazuhagent transport 1514/1515 under wazuh’s own enrollment PKINo; agents pin the wazuh CA; web PKI is wrong there. Dashboard is behind traefik already.
greenboneinternal nginx self-signs, expose: 9392 onlyNo, not a published edge; traefik fronts it.
gitea / gitlab SSHSSH host keysNo, not X.509.
matrixfully traefik-fronted (SYNAPSE_NO_TLS: "1")No current need.

What the ce skeleton contains (branch lane/cert-delivery-broker)

Section titled “What the ce skeleton contains (branch lane/cert-delivery-broker)”

modules/certbroker/: package compiles, shape-tested, NOT WIRED: no registry/subsystem/daemon registration, no route, no CLI verb, no ent schema. Contents:

  • doc.go: the not-wired banner and design pointer (this file).
  • types.go: HostGrant, Material, Posture, ReloadSpec, Delivery; posture-class vocabulary shared by spelling with modules/app/cert_edge.go.
  • broker.go: the Source / Materializer / Broker interfaces and NewUnwired() (every method returns X501476CRTNIW).
  • errors.go: coded factories X400470CRTDCL, X403471CRTSCP, X404472CRTHST, X409473CRTPOS, X500474CRTSRC, X503475CRTPND, X501476CRTNIW, X200477CRTROT (SEQ 470–477 verified free against the registry on origin/main), registered via the caErrorSpecs init idiom.
  • broker_test.go: shape tests: codes register, statuses parse, the unwired broker refuses coherently, vocabulary matches cert_edge’s spellings.

Landing notes (for the eventual, owner-approved train)

Section titled “Landing notes (for the eventual, owner-approved train)”

The train ran. The ent-ledger note DROPPED out of it: delivered material is node-local disk state and the rotation trigger compares source serial to delivered serial, so there is no ent schema and no shc DDL-mirror move is owed. The CE/EE note stands (mechanism is CE, no entitlement gate). The adoption note stands and is still outstanding; see the adoption map.

  • Ent + shc DDL mirror move together. The delivery ledger (one row per tenant/host/deployment: serial, not_after, posture class) is a new ent schema; shc’s hand-written DDL mirror must move in the same train and the migration is append-only (make migrate, never flatten).
  • CE/EE: mechanism proposed as CE, matching the fabric precedent (the WireGuard overlay mechanism is CE); no entitlement gate on serving a correct certificate.
  • Adoption commits (mailcow, fusionpbx) ride shc’s apps catalog on its own clock, after the daemon surface exists on an estate.

Concentrated in three places: the ACME-store read (a traefik store-format change breaks extraction, mitigated by fail-soft + the posture surface making breakage visible), the mount-grant check in core/volume (the one security-critical branch; fail-closed and small), and the cert-claim route interaction with the stall watcher (a claim on an unattainable host must take the public_cert_host.go downgrade, or it becomes a watcher-fed restart loop; the attainability verdict is applied to claims for exactly this reason).

Questions 1, 2, 3 and 5 are ANSWERED; the surface shipped on the answers. 1: the declaration + platform mount won. 2: the ACME store is read from disk on the compose runtime (the proposal’s “unreadable” premise holds only for swarm). 3: deliver-always-with-posture, not withhold; see the reversal note above. 5: the names all stand except certdeliver.allow_staging, which was never created. Question 6 (scope of v1 adoption) is the one still genuinely open, and it is open at zero: nothing has adopted.

  1. Contract blessing: declaration + platform mount (this design) vs a platform-served tls socket. The socket’s naming test half-passes; the phase-model and secret-delivery arguments are why this document says declaration. Fork decided here shapes everything downstream.
  2. Source posture: is daemon-side extraction from traefik’s ACME volume acceptable, or should traefik be taught to export per-host material itself (a traefik-side change SHC has so far avoided)?
  3. Staging default: withhold-unless-opted-in (proposed) vs deliver-always-with-posture. The LE-staging validation posture estates are the ones that feel this.
  4. Cert-claim routes: comfortable with the broker registering HTTP-less hosts into the ingress intent set to arm issuance?
  5. Names: certs: (app.yaml), /etc/shc/tls.serve/<host>, <StateHome>/certserve/, shc cert delivery, module certbroker, config certdeliver.allow_staging.
  6. Scope of v1 adoption: mailcow only, or mailcow + fusionpbx in the same catalog train?