Skip to content
SHC Docs

Config versus state, the migration (proposal)

Status: open design. The first migration, backup.schedule to the backup_schedule table, is built. The discriminator and the verdict table are not ruled.

As written: the discriminator and the verdict table are a proposal awaiting owner ruling. The first migration (backup.schedule → the backup_schedule table) is BUILT AND LANDED, and is described here as built, not as proposed.

Written 2026-08-07. The backup.schedule migration landed on ce b2322ca; the verdict table was compiled against ce 7ee33bc..8b0b8b4 and shc 819becc62, and every file:line citation below was read in a worktree at those commits. No production estate was contacted; the one measured estate fact quoted (zero backup.schedule rows on both estates) is carried from the lane that measured it.


“If a command is gonna set state and you’re storing that state in config, you’re doing it wrong. Either have it be fully driven by config where you don’t actually have a special command that sets it, or you store the state separately from config, just in the database. It could still be scoped.”

And, when the alternative of validating harder inside the config write path was put to the owner:

“If you have to guard config because some other command is using it, doesn’t that mean it should be some kind of state in the database instead?”

Needing a guard is the tell. A guard requirement is not answered by adding a guard.


The ruling turns on what a key’s checks are ASKING. There are only two kinds, and the difference is not one of severity; it is one of what information the question needs in order to be answered.

the check asksexamplewhat it needs to answerverdict
is this a well-formed VALUE?the cron parses; the enum member exists; the retention count is a non-negative integer; the port is in rangethe value, and nothing elseTYPE CHECK. Belongs in the key’s schema, where it protects every writer at once. The key STAYS CONFIG.
is this consistent with something ELSE?the destination exists; a sibling key is also set; the application being hidden was crawled; the deployment named here is realfacts about other keys, other rows, or the live estateCROSS-FACT INVARIANT. No key’s schema can carry it, because it is not a statement about this value. The key is STATE and moves to a table.

The reason the second class cannot live in config is structural, not a matter of effort. Config has four writers and no common gate:

  1. shc config set against a live daemon
  2. the daemonless overlay (ce/modules/config/local_store.go:222)
  3. POST /api/method/shc.config.set (ce/modules/config/router.go)
  4. the terraform shc_config resource, and shc install/update -c

A validation attached to any one of them is a validation the other two route around. A validation attached to Service.Set itself could only see the value, which is precisely the information a cross-fact invariant does not have. So the invariant has no owner, and a key that needs one has no business being a config row.

There is a third property that keeps showing up alongside the second, and it is worth naming because it is diagnostic: a key whose scope is narrowed on read is doubly a state candidate. A reader that empties the Ctx makes finer-scope rows inert, so the store silently accepts writes it will never honour. A dedicated table has no visibility walk to opt out of.


Verdicts are per key, with the evidence that decides them. Paths are ce unless marked.

keyowning writerthe cross-fact invariantevidence
backup.schedule ✅ MIGRATEDshc backup schedule set → POST shc.backup.schedule.setCheckScheduleDestination asks whether backup.password is set for the backup.repo this scope resolves to. A repo with no password can never seal, so every fire lands plaintext on local disk, on a cron, forever, unwatched.check: modules/backup/seal.go:246; refusal: X400077BKPSCD, modules/backup/errors.go:1713
lease.reap_attemptsthe lease-retry cron, no operator surface at allIt is a running tally the daemon increments. Nobody authors it; there is no value an operator could type that would be correct. Every increment is a config-change event that wakes every config watcher on the estate.modules/app/crons_lease_retry.go:66, written at :187
lease.reap_next_attemptsame cronAn RFC3339 backoff stamp, bookkeeping about a retry that is in progress.crons_lease_retry.go:70, written at :204
lease.reap_abandonedsame cronThe terminal marker set when the retry budget is exhausted. A record of what happened, not a setting.crons_lease_retry.go:76, written at :196
lease.expires_atthe lease stamp on installNames a deployment that must exist, at deployment scope, and is written by the system rather than authored. It is the lease.modules/app/lease.go:31
deployment.protectedshc protect / shc unprotect★ The flag lives in TWO stores (a deployment-scoped config row and a node-local sidecar mirror), and the command already has to report which half landed, because a deployment placed on another node arms the row and nothing this node can read. Two-store agreement is the definition of a cross-fact invariant.modules/app/protected.go:102; the ProtectionStores type and its comment at :112-140 document the split in the code itself
dns.owner_idminted by shc cluster init (mintDNSOwnerID)The cluster’s identity for DNS claim attribution. Two clusters sharing a zone are separated by it, and a destroy-and-rebuild must not silently reuse one. Identity is not a setting.modules/dns/service.go:34; X503038DNSOID, modules/dns/dnsapi/errors.go:170

STAYS-CONFIG, but the type check moves into the schema

Section titled “STAYS-CONFIG, but the type check moves into the schema”

These keys are genuine operator inputs. The ruling’s other half applies: the check they need is a type check, so it belongs in the schema where every writer gets it, rather than in the one command that happens to have it today.

keytoday’s checkwhere it lives nowwhere it belongs
backup.retention.{daily,weekly,monthly,yearly,bundle_days}validateFields: range checks on integersmodules/retention/service.go:372, reachable only through shc retention setthe schema. shc config set backup.retention.daily -5 bypasses it today.
crons.<name>the cron expression must parsethe cron runner, at read time; a bad override is a warn nobody readsthe schema, plus the existing registry-derived ReadClaim that already catches an unregistered job name (modules/config/keyspace.go:133)
backup.repononen/athe schema: a destination URL has a form (s3:, b2:, an absolute path)
backup.password, dns.<provider>.<credential>, backup.{aws,b2,azure,google}_*nonen/asecret-class handling, not a type check. Value-shaped: a credential is whatever the provider says it is.
lease.protected_environmentsnonen/aa list of environment names, layered over a hardcoded set that config can never remove (modules/app/lease.go:37, :59). Pure operator policy.
system.apps.<name>.host / .base_path / .vars.*nonen/aan app’s own variable namespace; AnyMemberRead is correct here for the reason the keyspace doc gives
catalog.hidden.<qualified>nonen/a★ the interesting near-miss. “Does that application exist?” IS a cross-fact question, but hiding an application that has not been crawled yet is legitimate, so absence of the referent is not an error. It stays config, and the honest response to an unknown qualified name is the warning it already gets, not a refusal.

quota.* is not a config family in ce; quota is derived from the authored compose figure rather than stored. Nothing to classify.

Independent confirmation, arrived at from the other end

Section titled “Independent confirmation, arrived at from the other end”

While this table was being written, a sibling lane landed ce 3d8b2f5, which adds modules/config/command_owned.go, a marker declaring a key “command-owned state”, enforced at the shc install/update -c passthrough. It registers exactly three paths:

  • backup.schedule (modules/backup/config_reload.go)
  • lease.expires_at and deployment.protected (modules/app/register.go)

All three are on the IS-STATE list above, reached by a different route and by a different author. That lane’s own comment on backup.schedule states the limit of a marker and predicts this document’s conclusion:

“Per the owner’s ruling that is the tell that a schedule is not config at all; it wants its own table. Until it has one, the registration stops -c backup.schedule=… becoming a SECOND way to arm a schedule… shc config set backup.schedule remains the FIRST way, and remains a known defect: closing it means moving the storage.”

The storage moved. backup.schedule’s command-owned registration is replaced by the retirement in this landing; the marker mechanism stays and now covers lease.expires_at and deployment.protected, which is a good description of what it is for: the holding position for a key that is state and has not been migrated yet. Read the two registries together and the remaining migration queue is legible from the code.


  1. backup.schedule: first, and already done. It is the only key on the list whose bad state arms a recurring action: a config row written raw registered a cron with none of the owning command’s checks. Every other candidate stores a wrong fact; this one stored a wrong fact that then did something, repeatedly, unattended.
  2. lease.reap_attempts / reap_next_attempt / reap_abandoned: next, and cheapest by a wide margin. There is no operator surface to preserve, no intent to migrate, and no compatibility story: the daemon writes them and the daemon reads them. They should land in one deployment_lease table together. Doing them second proves the pattern on a case with zero migration risk.
  3. lease.expires_at: rides into the same table as (2). Split out here only because it has an operator-visible origin (--lease) and therefore a real value to carry.
  4. deployment.protected: after the pattern is proven twice, because it is the one with a second store. The table becomes the truth and the sidecar becomes an explicitly-named cache, which lets shc protect stop reporting two-valued success.
  5. dns.owner_id: last of the state set. Smallest change, largest blast radius: it is cluster identity, and the write-once mechanism (modules/config/write_once.go) may be the better answer for it than a table. Worth its own ruling.

The STAYS-CONFIG schema work is independent of all five and can proceed in parallel; it has no migration and no ordering constraint.


backup_schedule (ce ent/schema/backup_schedule.go), created by migrations/20260807170745_auto.sql. Still scoped, as the ruling allows: scope carries the tier, tenant_name / stack_name / environment carry the target.

Two decisions worth stating:

  • scope_key is the uniqueness key, not the scope tuple. SQLite and rqlite treat NULLs as distinct inside a composite UNIQUE, which is why the config table’s own composite constraint is inert for global rows and why its upsert has to be a hand-written guarded INSERT (modules/config/service.go:379). The scope columns here default to '' and scope_key carries the canonical backup.ScopeKey() rendering, so one-registration-per-scope is enforced by the database on every backend.
  • It is a NEW table, not a reuse of schedule. The existing schedule table is the per-stack JOB scheduler: its rows name a <job>.job.yaml inside a deployed stack and its fires become task rows. Its tenant_name / stack_name are NOT NULL and its unique key is (tenant, stack, environment, job). A backup schedule is scoped at five tiers including global, where there is no stack, so the global and tenant registrations would collide on that key. Overloading it would have meant a sentinel-stack convention plus two columns meaningless to job rows.

make migrate generated the DDL; the backfill is hand-appended beneath it, which is the established shape here (see 20260805004250_application_table.sql, whose header explains why Atlas cannot sequence a data-carrying transition).

The config rows are copied and left in place. Two independent reasons:

  1. The daemon-side loader takes the union of the table and any leftover backup.schedule rows, so a row the backfill could not see (an older CLI against a newer daemon, or a schedule armed concurrently with this landing) still fires. Deleting here would make that safety net unreachable for exactly the rows it exists to catch.
  2. A migration cannot see a row written after it runs.

Three classes are deliberately not carried, and each stays in config where it remains visible and honoured: a value that is not valid JSON (json_extract would abort the migration and take daemon boot with it); a value with no cron in either accepted shape (it could never have registered); and a row whose scope columns are malformed: a stack or environment recorded with no tenant, which scope.Classify reads as GLOBAL and which would therefore collide with the real global row. Nothing is silently dropped.

Measured: both estates carry zero backup.schedule rows, so the backfill is empty in practice. It is written as though it were not.

modules/config/retired.go adds RegisterRetiredPath, a sibling to the existing RegisterWriteOncePath and to RegisterCommandOwnedPath. Service.Set and the daemonless overlay both refuse a retired path with X409073CFGRTD, naming the table and the command that replaces it.

The asymmetry with write-once is deliberate and is the opposite way round:

write-onceretired
SETcreate and identical re-state allowed; change refusedalways refused
DELETErefused: delete-then-set would be a flip through the back doorallowed: a leftover row must be removable, or it is stranded forever
READallowedallowed: the loader is still honouring it, so calling it inert would be a lie

One more surface needed the tolerant form: the deployment-config restore replay (daemon/services_backup_config.go:74) now SKIPS a retired path with a warning instead of failing. An archive is a record of the past; a key that has since moved stores will be in any archive taken before the move, and letting that fail a restore would mean a schema migration retroactively invalidated the estate’s backups.

Both are consequences of the same root cause: a check that lived inside an optional live-apply rather than in front of the write.

  • The cron parse was conditional on a scheduler existing. It rode on Scheduler.RegisterScope inside if sch != nil, so a request served before the backup_scheduler subsystem published its instance (a partial boot, a follower, a test root) persisted an unparseable cron and returned 200. It is now backup.ValidateCron, called unconditionally in front of the write. This is the type check the discriminator says belongs in front of every writer, and naming it as one is what made the bug visible.
  • shc backup schedule ls rendered nothing on a follower. The list came from the in-memory cron map, and the subsystem is LeaderOnly, so any node that was not the leader reported “no backup schedules” for an estate that had them. It now reads the table; next_run stays best-effort, because when a schedule next fires genuinely is a property of the node that would fire it.
  • migrations/backup_schedule_carry_test.go: applies the real catalogue in two halves, seeds one config row per tier plus the three skip classes, and asserts the carry, the skips, that the config row count is unchanged, and that the SQL’s scope-key rendering equals what backup.ScopeKey returns for the same scope.
  • subsystems/impl/backup_schedule_loader_test.go: table-only, config-only (must still fire), and both-for-one-scope (must register once, and the table must win).
  • modules/config/retired_test.go: SET refused, READ and DELETE still work, exact-path granularity.
  • modules/backup/schedule_store_test.go: round-trip at all five tiers, upsert replaces rather than duplicates, and backup.schedule is registered retired.

  1. Is dns.owner_id a table or a write-once config row? It is identity, not configuration, so the ruling points at a table, but write_once.go already exists for facts whose CHANGE needs machinery that does not, and identity may be exactly that shape.
  2. Should the retired-path refusal ever expire? Today a retired path refuses writes forever and the loader honours leftovers forever. An alternative is a release in which the loader stops reading them and the rows become ordinary dead config. That is a decision about how long an upgrade window is, and it is not ours to pick.
  3. catalog.hidden.<qualified> for an application that does not exist: confirmed as a warn, not a refusal? The table above rules it stays config on the grounds that hiding something not yet crawled is legitimate. If that is wrong it becomes the sixth migration.