Subsystem lifecycle management
Layer 2. Core. Core
The SubsystemManager is the central orchestrator for every
background process inside the SHC server. It provides one place to
register, start, and stop subsystems with explicit ordering,
dependency declarations, rollback on startup failure, and signal-
driven shutdown.
The single-writer watchdog runs outside shcd so it can stop labelled containers after the node lease expires even when shcd is hung. See Runtime support.
- Unified lifecycle: one place to understand all background things.
- Explicit ordering:
Priority+Depsenforce startup order; reverse for shutdown. - Fail-fast: a required subsystem failing at start triggers rollback and crashes the server (systemd restarts).
- No in-place restart (by design): per B8 there is no runtime
restart of an individual subsystem; the recovery unit is the whole
process (systemd restarts it cleanly). Leader-only work is gated in
place via
LeaderAware.OnLeader, not by stopping/starting the subsystem. - Composition root, not service locator: the daemon’s module
services are owned by a single composition root,
daemon.Services(daemon/composition.go), built in dependency order beforelifecycle.Wireand injected into consumers (HTTP/CLI builders viaregistry.Deps.Services; subsystems via the lifecycleResolver).
Architecture
Section titled “Architecture”Subsystem hierarchy (current)
Section titled “Subsystem hierarchy (current)”The registration list lives in
subsystems/register.go;
each subsystem’s Priority() is declared in its own impl/*.go
constructor (the numbers below are read from those).
Selected subsystems register under the Lifecycle orchestrator
(subsystems/lifecycle):
| Subsystem | Priority | svc | Role |
|---|---|---|---|
Bootstrap | -50 | first-run setup | |
Vault | -30 | vault unlock | |
ContainerRuntime | -20 | svc | docker / runtime detection |
Collectors | -10 | svc | OTel exporters + tick cron |
RqliteEngine | 1 | svc | rqlited supervisor + pool swap (no-op without the engine marker) |
SSEBroker | 1 | svc | |
ConfigLoader | 1 | load DB configuration | |
EventBus | 2 | svc | |
StackReconcile | 4 | reconcile a deployment’s desired state (subsystems/impl/stack_reconcile.go) | |
Migrations | 5 | atlas migrations against the post-swap pool | |
EntClient | 6 | svc | opens ent.Client against the post-swap pool |
NodeSelfRegister | 6 | ModeAny: leader upserts its own row; follower fills worker fields | |
BackupMount | 6 | svc | FUSE backup-mount manager |
CAGenesis | 7 | cluster CA and app-root CA reconciliation | |
Queue | 8 | svc | |
DockerEvents | 9 | svc | |
ContainerRecorderNode | 11 | svc | |
UnixSocket | 20 | svc | /run/shc/shc.sock listener |
HTTP | 30 | svc | TCP API listener |
InternalAPI | 40 | svc | mTLS internal listener (cluster mode) |
InternalCertRotation | 45 | svc | per-node :4003 x509 leaf renew (ModeAny) |
LeaderHub | 50 | svc | polls raft leadership, fans out OnLeader |
ContainerReconciler | 51 | svc | |
VolumeSizeSweep | 56 | svc | once per leadership gain, records missing sizes of live block volumes |
IngressMaterialize | 65 | svc | ModeAny: self-gates on Traefik-host placement |
ProxyReconcile | 66 | svc | leader-only L4 proxy-plane reconcile |
BackupScheduler | 70 | svc | leader-only cron-driven backups |
OperationRegistry | 70 | svc | SSE-streamed long-running op store |
ScheduleDispatcher | 71 | svc | |
CronRunner | 72 | svc | owns every registry.CronJob (deps queue, leader_hub) |
VolumeSizeSweep runs once on each leadership gain, including startup as leader.
It reads all tenants, skips kept or releasing volumes, and asks providers sequentially
for missing sizes. Leadership loss cancels the pass; the next gain retries rows
that still have no size. Every pass emits one X200311VOLSWP summary, including
zero counts. It has no periodic timer.
Legend: svc = service-style subsystem (owns a goroutine until Stop).
The Priority column is the Priority() each subsystem declares,
and the lifecycle resolver topologically sorts by priority + Deps()
at Wire time, so registration order is incidental.
Subsystem types
Section titled “Subsystem types”| Type | Description | Examples |
|---|---|---|
| One-shot | Runs once during Start, then markStarted() and exits | Bootstrap, Migrations, NodeSelfRegister, ConfigLoader, StackReconcile, CAGenesis |
| Service | Long-running goroutine driven by ctx | SSEBroker, EventBus, Queue, DockerEvents, LeaderHub, Collectors, CronRunner |
| Listener | HTTP / WS / UDS listeners | UnixSocket, HTTP, InternalAPI |
| Leader-only | LeaderOnly: true. Stays Running on every node. Only its leader-gated work toggles via OnLeader. (Modes separately gates a subsystem off entirely on the wrong cluster role.) | BackupScheduler, ProxyReconcile |
Priority ranges (guidelines)
Section titled “Priority ranges (guidelines)”-50 to -10: bootstrap, vault, runtime, collectors1: rqlite engine (no-op without the engine marker; the pool swap to the raft engine must land before migrations)1-11: config + event infrastructure + reconcile + migrations + ent client + node self-register + backup mount + cluster CA + queue- docker events + container recorder
20-29: local API (Unix socket)30-49: network listeners (HTTP, internal API) + per-node cert rotation50-59: leader detection + container reconciliation60-72: cluster cert rotation, ingress materialize, proxy reconcile, backup scheduler, operation registry, schedule dispatcher, cron runner
Voter membership is not a subsystem. The leader cron voter_suffrage
runs every minute and adjusts voter roles one member at a time. A down
member keeps its vote until an operator removes it. Source: CE
modules/cluster/register.go and modules/cluster/suffrage.go.
Subsystem protocol
Section titled “Subsystem protocol”type Subsystem interface { Name() string Modes() Mode // bitmask of run modes (ModeAny when always-on) Deps() []string // names that must be running first Priority() int LeaderOnly() bool
Start(ctx context.Context, deps any) error // Idempotent. deps is built by the Registration's DepsBuilder. Stop(ctx context.Context) error // Idempotent. HealthCheck(ctx context.Context) error // nil = healthy; non-nil = Degraded.}LeaderAware is an optional second interface: a subsystem that
implements OnLeader(ctx, isLeader) is driven by the leader hub on
every transition (and once at Wire time with the initial state).
Each subsystem is installed through a lifecycle.Registration
(subsystems/lifecycle/registry.go) carrying:
Priority: startup orderDeps: list of subsystem names that must be running firstModes: bitmask of run modes (ModeAnyfor always-on; panics if zero)LeaderOnly: gates the subsystem’s leader-work behind raft leadershipStartIf: optionalfunc(*Env) boolevaluated against the Env at Wire time; if it returnsfalse, the subsystem is skippedDepsBuilder:func(*Resolver) anythat constructs the typed deps value handed toStart(this is where a subsystem pullsResolver.Services, the composition root, and the already-started subsystems it depends on)
Lifecycle
Section titled “Lifecycle”Normal startup
Section titled “Normal startup”cmd/shcd/main.godelegates todaemon.Run(daemon/daemon.go), which loads config, initialises logging + OTel, opens the database pool, and callssubsystems.RegisterAll().daemon.Runthen builds the composition root withnewServices()and populates it viawireModuleServices+wireExtraModuleServiceswireModuleSeams(all indaemon/services*.go, ce module), passing amoduleServiceDeps{pool, cfg, logger, paths, services}. Each per-module setter stores its service onServices. This runs BEFORElifecycle.Wire, because theunix_socketsubsystem walksregistry.Routers()at Wire time and each router closure captures its service then.
lifecycle.Wire(ctx, WireOptions{..., Services: services})returns the*Lifecycleafter every subsystem is registered; the composition root rides through onWireOptions.Servicesand lands on the resolver asResolver.Services.- Subsystems carry priority +
Deps(); the resolver topologically sorts them at Wire-time. - SIGINT/SIGTERM are bound in
cmd/shcd/main.goviasignal.NotifyContext; the resulting cancellable ctx is threaded intodaemon.Run.lifecycle.Wirethen starts subsystems in priority groups (the resolver enforcesDeps()); the first group member to fail cancels its siblings, rolls everything already started back in reverse-priority order, and returns the error. lc.Wait(ctx)blocks until that ctx cancels (the SIGTERM/SIGINT path) or aFatalHealthErrorsurfaces.
Normal shutdown (SIGTERM / SIGINT)
Section titled “Normal shutdown (SIGTERM / SIGINT)”- ctx cancellation makes
lc.Wait(ctx)return;daemon.Runcallslc.Stop(stopCtx)with a 60 s bounded context. - Subsystems stopped in reverse priority order.
- If a subsystem doesn’t yield within the stop deadline it’s considered stopped and the orchestrator moves on.
daemon.Runreturns; OTel + DB pool shutdown run indeferblocks.
Failure during startup
Section titled “Failure during startup”Every subsystem is required; there is no optional/continue-on-error
start path. Any Start (or DepsBuilder) failure marks the subsystem
StateErrored, cancels its priority-group siblings, rolls back every
already-started subsystem in reverse-priority order, and returns the
wrapped coded error from daemon.Run → the process exits → systemd
restarts it.
Cluster vs solo mode
Section titled “Cluster vs solo mode”Subsystems are gated on cluster state through three independent
knobs: Modes (off entirely on the wrong cluster role), LeaderOnly
(stays Running but quiesces leader-work off the leader), and StartIf
(an Env predicate evaluated at Wire time):
| Subsystem | Active when |
|---|---|
RqliteEngine | always-on (ModeAny, a no-op without the {state}/engine.json marker; supervises the external rqlited process on promoted nodes) |
NodeSelfRegister | ModeAny: the leader self-registers its own row from on-disk bootstrap files; a follower fills in the worker-side fields |
InternalAPI | always registered; StartIf gates on Env.InternalAPIOn (NOT HasCertsOnDisk. Certs may post-date vault unlock); binds the node mesh address with mTLS |
Accessing subsystems
Section titled “Accessing subsystems”There is deliberately NO global service-locator for the lifecycle
(B8 bans the GetSubsystemManager() pattern). The daemon holds the
*Lifecycle handle returned by lifecycle.Wire and may consult it:
lc, err := lifecycle.Wire(ctx, opts)// ...state := lc.State("rqlite_engine") // one subsystem's Stateall := lc.States() // name -> State snapshotnames := lc.Running() // currently-running namessub, ok := lc.GetSubsystem("event_bus")Cross-subsystem dependencies flow through the typed DepsBuilder
(lifecycle.Resolver) at Start time.
Module services: the composition root
Section titled “Module services: the composition root”Module services (as distinct from subsystems) are owned by the
composition root daemon.Services (daemon/composition.go).
Services are built in dependency order and injected into consumers.
- HTTP / CLI builders read
registry.Deps.Services(type-erased toanyto avoid aregistry → daemonimport cycle; a builder type-asserts the small accessor interface it needs, e.g.d.Services.(interface{ UserService() *user.Service })). - Subsystems read
Resolver.Servicesinside theirDepsBuilder(alsoany, to keeplifecyclefree of a daemon import) and thread it intoStart.
Two publishing shapes coexist on the root:
- Daemon-built leaves (W1a/W3): services built inside a
wireXfunction in the daemon package (user,schedule,audit,app,clone, …) are stored via the UNEXPORTED setters (setUser,setApp, …) called directly. - Subsystem-published (W1b): services constructed inside a
lifecycle subsystem in
subsystems/impl(eventsfrom sse_broker/event_bus,dockerfrom container_runtime,taskfrom queue,configfrom config_loader,vault,cluster,backup,health,metrics,leader) reach the root through EXPORTEDSet<X>Servicemethods.implcannot name*daemon.Services(impl → daemonwould cycle), so each subsystem declares the small publisher interface it needs and theSet<X>Servicemethod satisfies it (Dependency Inversion).
No runtime restart (by design)
Section titled “No runtime restart (by design)”The lifecycle verbs are Wire and Stop. A subsystem signalling
FatalHealthError makes Wait return and shuts down the daemon; recovery
is a process restart. Sources: CE subsystems/lifecycle/lifecycle.go and
daemon/daemon.go.
- Leadership changes:
lc.LeaderTransition(ctx, isLeader)broadcasts to everyLeaderAwaresubsystem’sOnLeader, which starts/cancels internal goroutines in place (seecontainer_reconciler). - Config changes: subsystems watch
*config.Config.OnChangeor re-read config per tick; the config_loader reloads the live tree when DB-backed rows change. - Self-healing: health checks flip
Running <-> Degradedwithout goroutine bounces; terminal failures are fatal on purpose.
Subsystem state
Section titled “Subsystem state”| State | Description |
|---|---|
StateRegistered | Declared in init(), not yet started |
StateStarting | Start() in flight |
StateRunning | Start() returned nil; HealthCheck green |
StateDegraded | HealthCheck failing, goroutines still owned |
StateStopping | Stop() in flight |
StateStopped | Stop() returned cleanly |
StateErrored | Terminal: Start() failed or FatalHealthError |
Unless WireOptions.SkipHealth is set (test mode), lifecycle.Wire
spawns one healthLoop per subsystem that ticks HealthCheck on a
cadence: a non-nil result flips Running → Degraded (and back on
recovery) without bouncing goroutines, and a FatalHealthError flips
Errored, closes the fatal channel, and makes Wait return so the
daemon shuts down (fail-fast); systemd restarts it.
Leader-only services
Section titled “Leader-only services”The leader_hub subsystem (subsystems/impl/leader_hub.go) polls the
raft leader id against {state}/node.id at
config.leader_poll_interval, and on
each become-leader / lose-leader edge drives the transition through
Lifecycle.LeaderTransition, which fans OnLeader(ctx, isLeader) out
to every subsystem implementing the lifecycle.LeaderAware interface.
In single-node mode the hub still starts but IsLeader returns true
unconditionally.
flowchart TB hub["leader_hub<br/>polls raft leadership"] -->|"become/lose-leader edge"| lt["Lifecycle.LeaderTransition"] lt --> walk["lifecycle walks every LeaderAware subsystem"] walk -->|"OnLeader(ctx, isLeader)"| gated["BackupScheduler / ProxyReconcile<br/>(LeaderOnly)"] walk -->|"OnLeader(ctx, isLeader)"| aware["ContainerReconciler, ...<br/>(any LeaderAware subsystem)"]To make a subsystem leader-aware, set LeaderOnly: true on its
Registration and implement LeaderAware.OnLeader:
func (s *MySubsystem) OnLeader(ctx context.Context, isLeader bool) error { if isLeader { return s.startLeaderWork(ctx) } return s.stopLeaderWork()}OnLeader is also called once at Wire time with the initial
leadership state, so a subsystem that started before the hub settled
catches up. The subsystem stays Running on every node; only its
leader-gated internal work toggles.
File structure
Section titled “File structure”subsystems/├── lifecycle/ # Subsystem protocol, Lifecycle orchestrator, State enum│ ├── lifecycle.go # Wire / Stop / LeaderTransition / startGroup / health│ ├── registry.go # Registration, Resolver (incl. Services), Register│ ├── subsystem.go # Subsystem + LeaderAware interfaces, baseSubsystem│ ├── topo.go # priority + Deps topological sort│ ├── interfaces.go│ ├── bridge.go│ ├── errors.go│ └── types.go # Mode, State enum├── impl/ # Concrete subsystems│ ├── bootstrap.go # Bootstrap│ ├── vault.go # Vault│ ├── container_runtime.go # ContainerRuntime│ ├── collectors.go # Collectors (tick cron + metric collectors)│ ├── rqlite_engine.go # RqliteEngine (rqlited supervisor + pool swap)│ ├── sse_broker.go # SSEBroker│ ├── config_loader.go # ConfigLoader│ ├── event_bus.go # EventBus│ ├── stack_reconcile.go # StackReconcile│ ├── migrations.go # Migrations (atlas runner)│ ├── ent_client.go # EntClient (opens ent.Client on post-swap pool)│ ├── node_self_register.go # NodeSelfRegister│ ├── backup_mount.go # BackupMount│ ├── ca_genesis.go # CAGenesis│ ├── queue.go # Queue (task.Worker)│ ├── docker_events.go # DockerEvents (event collector)│ ├── container_recorder_node.go # ContainerRecorderNode│ ├── unix_socket.go # UnixSocket listener│ ├── http.go # HTTP listener│ ├── internal_api.go # InternalAPI (mTLS)│ ├── internal_cert_rotation.go # InternalCertRotation (per-node :4003 leaf)│ ├── leader_hub.go # LeaderHub│ ├── container_reconciler.go # ContainerReconciler (LeaderAware)│ ├── ingress_materialize.go # IngressMaterialize│ ├── proxy_reconcile.go # ProxyReconcile│ ├── backup_scheduler.go # BackupScheduler│ ├── schedule_dispatcher.go # ScheduleDispatcher│ ├── cron_runner.go # CronRunner (owns every registry.CronJob)│ ├── operation_registry.go # OperationRegistry│ └── *_publisher.go # consumer-defined publisher interfaces → daemon.Services└── register.go # RegisterAll: installs every subsystem above
daemon/├── composition.go # Services (the DI composition root) + per-service accessors/setters├── services.go # wireModuleServices + moduleServiceDeps (W1a leaves)├── services_extra.go # wireExtraModuleServices├── services_seams*.go # wireModuleSeams + the cross-module resolver seams└── daemon.go # Run: RegisterAll → newServices → wireX → lifecycle.Wirecmd/shcd/main.go # entry point: cobra, then daemon.RunBest practices
Section titled “Best practices”- Register in priority order in code for readability.
- Make
Start()andStop()idempotent (embedbaseSubsystem). - Long-lived loops must NOT derive their context from the Start ctx;
startGroupcancels the group’s start context once the priority group is up. Root loops incontext.Background()(or the OnLeader ctx) and letStopown the cancel. - Use
StartIf(afunc(*Env) boolon the Registration) for runtime-state-dependent subsystems (certs on disk, internal API enabled, …) andModesfor leader/follower gating. - Periodic work can live either as a subsystem loop or as a
registry.CronJob. Thecron_runnersubsystem owns every registeredCronJob: leader-gated jobs (rpc-crons always; task-crons whenLeaderOnly) tick only while this node is leader, while node-local jobs tick on every node for the whole Start..Stop window. - Leader-gated work: set
LeaderOnly: trueand implementLeaderAware.OnLeader(the leader_hub subsystem drives the transitions); the subsystem stays Running on every node; only its internal work toggles.
Related
Section titled “Related”- Communication patterns: how the running services talk to each other (RPC, task queue, WebSocket data plane, SSE).
taskmodule: read-side over the queue subsystem.healthmodule: registered health checks.