Init-time decisions
The question this page answers: what do you set at shc init that you
can never take back?
Some of it is genuinely stuck. Most of it is not: it only looks stuck, because the decision is reversible and SHC never grew the verb. Telling those two apart is the point.
The worked example is the backup repository password. restic encrypts
data under a repo-wide master key; the password only unwraps a key
file holding it, so rotation is O(1) with no data re-encrypted:
measured, with a byte-identical tree outside keys/ and a pre-existing
snapshot still restorable under the new key. This one is now CLOSED:
shc backup key rotate re-keys the repository and updates the key SHC
presents in one operation, and shc init --backup-password seals a key
at genesis. It is kept here because it is the cleanest illustration of
the class: the decision only ever looked permanent, and what was
missing was the verb, not a property of restic.
The counter-example is raft suffrage changed in place. rqlite stores voter or non-voter in the join request and has no promote API. SHC changes it by removing the member from raft and rejoining it. The reconciler does that toward the voter target. See suffrage.
The honest headline. Of the 41 decisions below, 7 are HARD. Most of those are permanent for a reason: the trial clock, the licence signature, the node id, an app’s own seed-derived credentials. 10 are SOFT: reversible in principle, with no operator verb. 15 are already changeable today. The remaining 9 are LOSSY, and those deserve more caution than the HARD ones do: a change works, reports success, and quietly strands something. Read those and the defaulted decisions, permanence an estate acquires because nobody was asked, before anything else on this page.
The four classes
Section titled “The four classes”| Class | Meaning |
|---|---|
| HARD | Genuinely irreversible. The engine or the data model forbids it. |
| SOFT | Reversible in principle; SHC exposes no verb, so it presents as irreversible. Each one is a small feature; see the SOFT list. |
| CHANGEABLE | Already changeable today. Named so the record does not scare you away from something that is fine. |
| LOSSY | Changeable, but doing it destroys or strands something specific. |
The table
Section titled “The table”Ordered by how expensive it is to get wrong, worst first, not alphabetically, and not by class. The first six rows are the ones worth slowing down for before you type the command.
| Decision | Set by | Class | What changing costs |
|---|---|---|---|
| Estate serving host | shc init --ingress-host, or SHC_HOST in the unit env | LOSSY | Every public certificate re-issued; the system Traefik recreated; every Keycloak redirect URI re-provisioned (the Keycloak URL is derived as https://<SHC_HOST>/keycloak and is deliberately not operator-configurable). Then the two that bite: the licence is domain-bound, so the estate silently discards its own paid licence and falls back to the trial clock until a re-issued token arrives from the broker; and this string is the dns.owner_id slug, so every record the cluster ever published becomes permanently un-adoptable. No verb: shc node update accepts only --public-address. |
dns.owner_id (DNS ownership identity) | Minted once at genesis as <host-slug>-<8 hex> (modules/cluster/commands/init.go SeedDNSOwnerID) | LOSSY | Mechanically an ordinary config row. But foreignness is decided on the slug, not the whole id: a claim whose slug matches and whose suffix differs is adopted as this cluster’s predecessor (X403046DNSTKO), and a claim whose slug differs stays foreign forever. Since the slug is the serving host, a rebuild at the same host recovers and a change of host orphans everything, reported as X403044DNSRPL, never deleted. See the live precedent. |
| Backup repository password | shc init --backup-password / --backup-password-file (else generated at genesis and sealed into the vault at backup/password), read through the backup.password config row; SHC_BACKUP_PASSWORD in the unit env still works. Keyed at the first restic init. | SOFT, verb landed | O(1), through shc backup key rotate, which re-keys the repository and the stored key together. See the SOFT list. Note the separate live problem in defaulted decisions: a repository created by a daemon built before the fallback constant was removed stays keyed by that constant, and deleting it from source does not re-key a repository already created under it. |
| Node name | shc init --name / shc node join --name; auto-generated node-<8 hex> otherwise | SOFT | No rename verb anywhere. The only route today is the terraform one, and name carries RequiresReplace, so tofu apply destroys and recreates the VM, which mints a new node id and pays every cost in the row below. On Hetzner the name is load-bearing outside SHC: the hcloud volume provider attaches by resolving the SHC node name against the server name. |
Node id (node-<12 hex>) | Minted by crypto/rand at shc init / enroll; persisted to <state>/node.id | HARD | Not changeable, only replaceable. It is four identities at once: the raft member id passed straight through as rqlited’s -node-id, the fabric allocation key, the voter_slot row key, and a SAN on the node’s :4003 leaf. To each of those a changed id is a different node. The only path is shc init --force, which wipes local cluster state and rejoins as a new member. |
| Raft suffrage of an existing member (voter vs non-voter) | The suffrage reconciler, toward VoterTarget | CHANGEABLE | rqlite stores suffrage in the join request and has no online role change. SHC removes the member from raft and rejoins it, data directory intact, one node at a time. A member that is down keeps its vote. See suffrage. |
The voter_slot grant | Claimed at enroll, up to VoterTarget | CHANGEABLE | VoterTarget is the largest odd count of eligible control-plane members, capped at five (modules/cluster/voters.go). The reconciler releases a grant it demotes. See voter slots. |
Fabric mode (shc or none) | shc init --fabric; an empty value selects shc where the runtime can attach it | SOFT | Joiners inherit the genesis posture. shc node join has no --fabric. No verb renumbers a live cluster. none is refused where the runtime can attach the fabric (X400389FABOFF). |
| Address block | shc init --address-block (default 10.64.0.0/16) | SOFT | Every SHC range is carved from it: deployment networks, the fabric, mesh addresses and service VIPs. Changing it moves every shc0 address, every deployment network, every fabric_pin row, every mesh address and VIP, and every peer’s AllowedIPs at once, with the data plane down. No renumber verb exists. shc cluster block add adds room later: one more block, a pool of /26 chunks for node fabric chunks, VIPs and mesh addresses, never removed or shrunk. The genesis block keeps its carve. Init and join refuse a node whose networks overlap the block’s ranges. See the address block. |
| A node’s fabric chunks | One /26 at enroll, one more each time the node’s chunks are full | CHANGEABLE | A chunk nothing sits in goes back, and a removed node’s chunks go back with it. Another node takes one only after cluster.address_quarantine (1 h) and once every live node has said its applied shc0 config no longer names it, so a stale AllowedIPs entry never routes a tenant’s traffic to the wrong node. The fabric running out of chunks is X500353FABFUL; shc cluster block add gives it more. Dormant unless the shc fabric is selected. |
SHC_STATE_HOME | shc.service (/var/lib/shc) | LOSSY | The variable is a one-line edit. The estate is not: the value is baked absolutely into every rendered compose.yaml bind source, and nothing re-renders outside install/update. Safe route: move the bytes and leave a symlink. Unsafe route: move the state home, and until a full shc update of every deployment lands, a restarted container binds a directory docker creates empty: the app comes up with no data and reports healthy. Also strands what lives outside the state home: /etc/shc/, the vault auto-unlock drop-in, the tmpfiles ownership grants. |
| Vault auto-unlock drop-in (genesis node only) | Written by the terraform provider at Create, never by shc init | LOSSY | shc vault rotate does not rewrite it and nothing warns. After a rotation the genesis node’s drop-in carries the dead password, so the next daemon restart comes up locked with no auto-recovery, while terraform state records a password that no longer opens the vault. Any rotation must rewrite this file in the same step. |
kek_salt (cluster-wide Argon2id KEK salt) | 16 bytes minted once into the replicated vault_config table at the first envelope unlock | SOFT | Rotation is O(rows) × two small GCM operations, the same shape as a password rotation, which already ships. Losing the salt is a different thing and is genuinely unrecoverable: every v2: row’s DEK is wrapped under Argon2id(master, salt), so a lost or clobbered salt orphans every secret regardless of the password. Do not confuse the loss with the operation. |
| Vault master password | shc init --vault-password[-file] / SHC_VAULT_PASSWORD | CHANGEABLE | shc vault rotate proves the old password by round-trip, re-keys in one transaction, flips the replicated verifier in the same batch, and broadcasts to peers. An envelope row re-wraps its DEK only: the payload ciphertext is never touched. The two unadvertised costs are the drop-in row above and the archive bundle row below. |
| The seal over each archive’s secrets bundle | Every orchestrated archive is sealed under the cluster vault master by default | LOSSY | Rotate the vault password and every archive taken before the rotation stays sealed under the retired master: the rows are re-encrypted, the archives are not. Restoring one needs that old master passed back in with --secrets-password / --secrets-password-file, honoured on shc backup restore --id, --file and shc recover alike. So keep every master you retire. The restore no longer hides this: it runs to completion (data replayed, jobs and hooks fired, stack started) and then reports failed with X400871VLTRSF naming the tenant/stack and the remedy. It used to log a WARN and report completed: a green row and a stack that could not boot. The designed exit is a global-scope cluster archive key that rotation re-wraps in place, leaving archives untouched; it is not built, and it must ship with a custody surface, because a key that lives only as a vault row dies with the cluster it was protecting. |
Container runtime (docker, podman, incus, apple) | shc init --runtime, or detection; one runtime per node. shc node join --runtime defaults to docker | SOFT | node_bootstrap.json is re-read on every boot, so a sidecar edit plus a daemon restart converges the chain with no machine recreate. No verb writes it, and the terraform provider marks runtime as RequiresReplace and destroys the VM. |
--nic pin | shc init --nic / shc node join --nic | SOFT | Asymmetric. A DHCP address change already heals itself: the interface name is re-resolved to its current IPv4 on every boot. Changing the interface is a sidecar edit plus a restart locally, but it moves the node’s internal address, which every peer’s raft join list, the leaf’s IP SANs and the fabric peer map all carry, so it is not a single-node operation. |
Raft (:4002) mTLS posture | Hardcoded true on a from-empty genesis; joiners inherit the leader’s file value | SOFT | A cluster initialised before the field existed decodes as plaintext and stays plaintext through every binary upgrade. Flipping it is an edit of every node’s sidecar plus an ordered restart; the material needs no minting, since raft reuses the certificates the :4003 listener already serves. A node that flipped alone mid-roll could not dial its still-plaintext peers, so it is an all-nodes operation. |
| Cluster SSH break-glass keypair | tls_private_key.cluster, minted once per estate root | LOSSY | The public half reaches nodes through cloud-init at create time and nothing re-pushes authorized_keys to a running node. Replacing the resource mints a key no live node trusts while destroying the only one that does: a simultaneous lockout of every node. This key and the vault password are the only two artefacts that survive node loss, which makes it the highest-consequence rotation in the estate and the one with no safe procedure. |
| Trial start instant | shc license trial start, or the license.accept_trial config row | HARD | Deliberately one-way, and the cleanest in the codebase. The recorded start may only ever move earlier: a monotone lattice, so any number of nodes in any order converge and never write again. A later start is what a reset looks like and is refused from the row, from the local marker, and from a second trial start. Monotonicity is why it needs no signature: an earlier start only ever costs its holder entitlement. The sanctioned escapes are fair: a genuinely new estate gets a fresh trial, a restore resumes the same window, and a spent trial keeps the full free tier forever. |
| Licence domain binding (per file) | Minted by the broker at activation against the estate’s domain | HARD | The binding is inside the Ed25519-signed payload, and adoption verifies signature and domain before a byte reaches disk. No operator action re-binds an existing token. At the estate level this is not a wall (shc license activate / refresh mint a newly-bound token), but the vendor-side slot may need freeing separately, and shc license delink only drops the local copy. |
| Vendor licence signing keys | A primary and a reserve Ed25519 public key baked into every EE binary at build time | HARD | Verification is fully offline, so revocation cannot exist; a licence signed by a retired key stops verifying on every binary in the field. Rotation is a binary release plus a fleet-wide re-roll. This is the one HARD door whose mitigation was designed in advance: any embedded key verifies, so a compromised primary is answered by promoting the reserve the same day. |
Immutable ?autogen app secrets (the APP_SEED class) | Minted once at deployment scope on first install, marked shc:immutable non-rotatable | HARD | The constraint belongs to the foreign app, not to SHC: the app persists seed-derived credentials at first init and derives others from them. Rotating locks the app out of its own datastore, and SHC cannot detect it: the vault write succeeds, the redeploy succeeds, the app is broken. Restore correctness depends on replaying the identical value, which is why these ride the archive secrets bundle and inherit its rotation loss. |
| Node mTLS leaf CommonName | Always the node name | SOFT | A derived mirror, not an independent door: the leaf is re-signable on demand, which is what shc cluster cert rotate --type mtls does. It is stuck exactly as far as the node name is stuck, and no further. |
| Cluster CA / global root | Cluster CA minted at genesis; the global root arrives through a loader seam | CHANGEABLE | shc cluster cert rotate --type cluster-ca|global-root|mtls|due is live, with a resumable phase machine a new leader can pick up, --rollback, and --algo. --algo is refused for anything but cluster-ca, so the global root’s key algorithm has no operator-chosen upgrade path. |
mesh0 range, shc0 device name, cluster ports | Compile-time constants (10.192.0.0/10, shc0, 4002/4003/4243/4244) | HARD | No flag, no config key, no environment override reaches any of them. The ports are already encoded in durable artefacts (join tokens, enroll responses, every node row’s internal address), so changing one is a fleet re-enrolment, not a restart. |
| Storage engine (plain sqlite ⇄ rqlite) | Not an operator choice at any point; derived from node count | CHANGEABLE | Flips itself automatically in both directions, each exactly-once guarded, at the cost of one daemon restart per flip and no data loss either way. The demote is deliberately paranoid so a freshly promoted leader is not demoted back and its joiner stranded. The real limitation is the absence of a knob: you cannot run rqlite on one node, nor plain sqlite on three. |
| A deployment’s stack name | shc install <app> [name] | LOSSY | A deployment that runs as replica projects is renamed by a move: a backup, a recover under the new name (a fresh render), then the source’s uninstall, which keeps its data for shc reclaim (SHC-2319). One that runs as one compose project is renamed in place by shc rename, which does not re-render. Three losses: vault rows in an external backend are stranded (the bulk-rename is a no-op on every external provider and the failure is warned, not fatal); the stop/start bracket runs compose under a different project name than the containers were created under; and the moved compose.yaml still names the old absolute bind paths. Run shc update <newname> on the daemon path after any rename. |
| A deployment’s tenant / environment | --tenant / -e at install | CHANGEABLE | Neither takes the fast path, so both run the heavy backup → uninstall → recover pipeline, which is the safer outcome, because the heavy path re-renders. Costs: downtime proportional to the data; the source teardown must set KeepDNS or it releases the records the target now lives behind; and volume rows keyed to the source are marked orphaned, not deleted. Non-default environments are entitlement-gated on create, never on adopt. |
| A tenant’s own name | shc tenant add <name> | SOFT | No shc tenant rename. Nothing forbids one: it is a bounded set of table updates, a directory move per node, vault rows at four scope tiers, restic snapshot tags, and a Keycloak realm rename. Today the only route is a heavy shc clone --target-tenant per deployment. The name is not burned by deletion: an operator-driven tenant add clears the tombstone. |
backup.repo | SHC_BACKUP_REPO, or the backup.repo config row; defaults to <state_home>/backups | LOSSY | Moving is cheap; keeping the old archives addressable is the cost. A scoped cache sync after a move prunes every pointer row it did not see at the new location: the bytes stay restic-readable, but shc backup ls no longer knows they exist. Three routes out: shc backup export first, or set backup.repo at a narrower scope pointing back for the duration of a restore, or point restic at the old path by hand. |
| Orchestrated archive staging root | Always <state_home>/backups/<tenant>/<stack> | HARD | The resolver takes only (tenant, stack); there is no seam through which a configured repo could redirect it. So an S3 or B2 backup.repo still needs local free space equal to the largest archive, and still writes it to the disk being backed up. On a default-repo estate the staging root and the restic repository are the same directory. |
| ACME provider | shc init --acme-provider / --acme-directory / --acme-email, or the terraform acme block | CHANGEABLE | Looks install-only from three directions and is not: --acme-provider is bound on both shc install and shc update, the update path persists the selection, and the next ingress sweep asks the daemon’s ACME source for each host’s certificate from the new provider; Traefik is never recreated. Costs are CA-side only: fresh issuance and rate limits, and the old provider’s certificate stays in the store until it lapses. The staging-first rule is a policy, not a code restriction. |
| Keycloak master-realm admin password | Injected at genesis through modules.keycloak.vars.adminPassword | CHANGEABLE | Reset over REST with the current password. A REST-only reset leaves the container’s env stale and kcadm then fails 400, and the cure is a redeploy, not another reset. The terraform output is genesis-time truth, not current truth. |
Ordinary ?autogen app secrets | Same mint path as the immutable class, without the marker | CHANGEABLE | shc vault set <key> <value> at the right scope, then redeploy. Listed so the immutable class does not scare you off the ordinary class, which is the majority. Live trap: shc vault set with no -t lands at tenant default, and another tenant’s deployment never sees it. |
| Vault at-rest encryption format | Envelope, always | FIXED: there is no choice to make. | The crypto_format write epoch and the bidirectional shc vault migrate verb are removed (v0.0.16). One format, no reader for any other, and the retirement is a one-way make migrate step that REFUSES to apply while any retired-format row survives. |
A node’s public_address | shc init --public-address / the enroll request | CHANGEABLE | One command: shc node update <node> --public-address <v>, idempotent, no reinstall and no rejoin. Clearing it makes internet DNS fail closed rather than publish a stale address. Recorded because almost everything else on the node row is stuck. |
<state>/cluster.id | Not at init; minted lazily on the first shc node token | CHANGEABLE | Nothing keys to it. It rides the join token’s claims, is decoded on the join side, and is then never compared to anything. Delete the file and the next mint writes a new one. An archive taken under one cluster id restores into a different cluster without complaint. |
Startup-only config prefixes (dev, vault, collector, rate_limit) | The shc config set refusal X400050CFGSTP | CHANGEABLE | Not init-only and not unchangeable: the refusal text is the remedy. It is scoped to the database-backed writer only; the keys are fully settable in /etc/shc/config.yaml or /etc/shc/config.d/*.yaml plus a restart. The genuinely confusing case is the opposite one; see corrections. |
| Timezone and locale | Nothing. No SHC-owned setting exists at any point. | CHANGEABLE | Nothing on the SHC side. The host’s timezone is the host’s business; a container that wants one gets it from a host bind or a per-app variable. Recorded so the answer is a cited no rather than an absence. |
shc init --force | The escape hatch that resets most of this list | LOSSY | The one verb that clears node identity, the raft data dir, the certificates, the vault, every deployment record and the system-apps marker in a single move. It exists because a partly-initialised node otherwise has no cure. It is the wrong tool for every question on this page, and reaching for it to change one setting costs that node’s whole control plane, plus the cluster’s fault tolerance, because the re-join comes back a non-voter. |
The SOFT list: reversible decisions with no verb
Section titled “The SOFT list: reversible decisions with no verb”Each of these is a small feature that removes a permanence that is not real. They are ordered by value, not by size.
shc backup key rotate: ✅ LANDED
Section titled “shc backup key rotate: ✅ LANDED”Removed: “changing the backup password loses every backup.”
restic encrypts under a repo-wide master key that the password merely
unwraps, so rotation is O(1) with no data re-encrypted. Measured against
a real repository: a snapshot taken under the old key stays listable and
restorable under the new one, the old key is then refused, and every file
outside keys/ is byte-identical before and after.
shc backup key rotate performs both halves or neither: the repository
re-key, and the update of the key SHC presents. Doing only one of them is
the failure this verb exists to prevent, and it is exactly what a
hand-run restic key passwd produces. Companions: shc backup key ls
lists the keys that open the repository, and shc backup key has a
single-key retirement verb for clearing one a partial rotation left
behind.
Two properties worth knowing before running it:
- The new key is added BEFORE the old one is removed, so an interrupted rotation leaves the repository openable by both keys rather than neither. A re-run converges, retiring every key present at its start.
--repois an assertion, not an override. SHC stores one backup key with no repository dimension, so a value that differs from the configured repository is refused (X500455BKPKRP) rather than honoured: re-keying an overridden repository would overwrite the key for the configured one, which was never touched.
If the removal half fails, the rotation reports X500456BKPKPT naming
the keys that still open the repository. That is a failure, not a
warning on a green result: the credential the operator asked to revoke
was not revoked.
X500414RSTPAS no longer prescribes a raw restic invocation. It now says
to put the original key back and change it with shc backup key rotate.
Suffrage follows the voter target
Section titled “Suffrage follows the voter target”The reconciler in ce modules/cluster/suffrage.go promotes and demotes
one node at a time so the voter set meets VoterTarget: the largest odd
count of eligible control-plane members, capped at five. rqlite records
suffrage in the join request and has no online role change, so each
change removes the member from raft and rejoins it with its data
directory intact. A member that is down keeps its vote. Promotion runs
before demotion. There is a window with the member out of raft.
Voter slots follow the same target
Section titled “Voter slots follow the same target”Join claims a voter_slot up to VoterTarget
(modules/cluster/voter_grant.go). The column is UNIQUE, so two
joiners racing for the last slot are split by the database. The suffrage
reconciler releases a grant it demotes (ReleaseVoterSlot). A node
outside the control-plane fence is not granted a slot.
shc vault rekey-salt
Section titled “shc vault rekey-salt”Removes: “the KEK salt can never be changed.”
The re-wrap machinery already exists and is already parameterised on two KEKs: a password rotation derives an old and a new KEK from one salt and re-wraps each envelope row’s DEK, never touching the payload. A salt rotation is the same loop with the salt varied instead of the password.
Implementation: O(rows) × two small GCM operations, reusing the rotation mutex, the verifier flip and the peer broadcast the password rotation already has. This is the weakest classification on this page; read that section before acting on it.
shc node rename / a runtime and nic writer
Section titled “shc node rename / a runtime and nic writer”Removes: “a node’s name, runtime or interface is fixed for the life of the machine.”
shc node update accepts --public-address and nothing else. The runtime
in particular is cheap: node_bootstrap.json is re-read on every boot by
both consumers, so a sidecar edit plus a daemon restart converges the
chain, yet the terraform provider destroys and rebuilds the VM for it.
Implementation: for runtime and nic, a sidecar writer plus a
restart. For name, re-sign the leaf and rewrite the registry row. Coordinate
with the cloud server name where the volume provider resolves against it.
shc tenant rename
Section titled “shc tenant rename”Removes: “a tenant name is permanent.”
Bounded work: the registry row, the tombstone key, deployments/<tenant>/
on every node, vault rows at four scope tiers, the backup cache column,
restic snapshot tags, and the Keycloak realm.
A fabric posture writer, and an address block renumber
Section titled “A fabric posture writer, and an address block renumber”Removes: “the data plane and its address space are chosen once.”
The posture is one replicated row that Ensure overwrites; nothing points
a verb at it. The renumber is genuinely larger: every shc0 address,
every deployment network, every pin row, every mesh address and VIP, every
peer’s AllowedIPs, and the compiled fabric policy, with the data plane
down. Growing is not renumbering: shc cluster block add adds a block of
/26 chunks beside the genesis block.
A rotation-aware vault password path
Section titled “A rotation-aware vault password path”Removes: two silent losses that ride an encouraged hygiene action.
shc vault rotate works. What it does not do is rewrite the genesis
node’s auto-unlock drop-in (so the next restart comes up locked), or warn
that every archive taken before the rotation now holds a secrets bundle
nobody can open. Both belong in the rotation transaction: rewrite the
drop-in, and either re-seal existing bundles or refuse until the operator
acknowledges them.
Decisions you get by default
Section titled “Decisions you get by default”An irreversible decision nobody made is worse than one made badly. These are the ones an estate acquires without anybody choosing.
The fabric. An empty fabric omits --fabric. shc init then
selects shc where the node’s runtime can attach the fabric, and forms
the cluster without one where it cannot (ce modules/cluster/service.go).
none is refused where the runtime can attach (X400389FABOFF). A chunk a
node gives back goes to another node only after the address quarantine and
once every live node has dropped it from shc0.
The backup repository password. Absent from every layer on both
estates: backup.password unset, SHC_BACKUP_PASSWORD absent from the
daemon environment. Before the current pin, that fell through to a
constant compiled into the public repository, and a read-only check
on one estate confirmed the live repository opens under it. The pin
removed the constant (resolveBootBackupRepo now returns nothing without
SHC_BACKUP_PASSWORD, on the reasoning that “a key the daemon can derive
is a key its operator does not hold”). Two things follow, and the order
matters:
- Removing the default from source does not re-key a repository already created under it. Rotate the live repository before rolling.
- Rolling to the pin without first setting
SHC_BACKUP_PASSWORDorbackup.passwordleaves the daemon with no password at all, and every restic backup surface refuses (X503107BKPNOK).
The backup repository location. backup.repo is unset on both
estates, so both write to <state_home>/backups, the same filesystem as
everything the backups exist to restore. On a default-repo estate that
directory is simultaneously the restic repository and the per-tenant
staging tree, which is the most confusing on-disk state in this record.
Raft suffrage. The reconciler sets it from the control-plane fence. Terraform does not. See suffrage.
The auto-unlock capability. Only the genesis node can cold-start the
vault. Followers come up locked and pull the unlock from an
already-unlocked peer, correct as a security posture, since the password
never lands on a follower’s disk. The unwritten consequence: the pull
needs an already-unlocked peer. If every node restarts at once and the
genesis node is gone, no node can cold-start the vault and the control
plane waits at X503906VLTLCK for a manual shc vault unlock -p.
Retiring the genesis node silently removes a cluster capability, and
terraform has no way to move it.
What terraform exposes, and what it hides
Section titled “What terraform exposes, and what it hides”An invisible irreversible decision is the worst kind. Enumerated from the provider’s schema blocks:
Visible on shc_cluster: package, fabric,
address_block, host_domain, modules, vault_password, acme.
Visible on shc_node: bootstrap, force_genesis, host,
package, ssh, adopt, name, nic, runtime, public_address,
private_ip, host_domain, vault_password, plus the outputs
node_id, ca_pem, endpoint, admin_token, join_token, healthy,
ready.
Invisible entirely:
| Hidden decision | Class | Why it matters |
|---|---|---|
| The backup repository password | SOFT | No first-class attribute exists; neither estate sets one anywhere. The single most consequential defaulted decision in the estate. |
dns.owner_id | LOSSY | The provider only reads it. Settable solely as a generic shc_config row. |
| The vault auto-unlock drop-in | LOSSY | A side effect of Create with no attribute at all, and nothing rewrites it on rotation. |
kek_salt | SOFT | Never surfaced. Losing it is unrecoverable. |
Raft suffrage and the voter_slot grant | CHANGEABLE | The reconciler keeps the voter set at VoterTarget. Terraform does not set either. |
| The raft mTLS posture | SOFT | Inherited from the genesis node’s file, invisible in state. |
| The trial start instant | HARD | Reachable only through the license.accept_trial config row. |
<state>/cluster.id | CHANGEABLE | Zero references in the provider. |
| So of the decisions that cost the most to get wrong, terraform surfaces | ||
| the serving host, the fabric and the node name, and hides the backup key | ||
| and the DNS ownership identity completely. |
Two more traps in what is visible. shc_node.admin_token is a
sensitive output named like a root credential, emitted by the bootstrap
node alongside genuine genesis artefacts: it merely echoes the provider
block’s own token and is null when none was set. And name, nic and
runtime all carry RequiresReplace, so terraform’s remedy for three
SOFT decisions is to destroy the machine.
Corrections to the record
Section titled “Corrections to the record”Claims elsewhere in the repository that are not true at HEAD.
shc init --backup-password DID not exist; it does now, and so does
shc backup key. This entry stays as a record of what was corrected.
The claim was accurate at the time: the flag was absent from the
registration in modules/cluster/commands/init.go, the backup key was
not minted at genesis, and no verb could change it. As of the backup-key
lane, --backup-password / --backup-password-file are registered and
honoured at genesis only, an unsupplied key is generated with
crypto/rand and sealed into the vault at backup/password, and
backup.password is pointed at that row unless the operator had already
set it to something else. What is still NOT surfaced is terraform: the
provider has no attribute for it, so a tf-bootstrapped estate takes the
generated key, and reading it back means reading the vault row.
The terraform provider’s vault claim is false, in two places.
node_resource.go:908-912 asserts that “no in-place rotation exists …
nothing in the provider (or the daemon’s HTTP surface) re-keys a live
vault”, and cluster_resource.go:366-376 repeats it. Both halves are
wrong: shc vault rotate is a live CLI verb and
POST /api/method/shc.vault.rotate is a wired handler that re-keys every
row in one transaction. The honest constraints are narrower and neither
is stated: terraform cannot track a rotation it did not perform, and the
auto-unlock drop-in goes stale. Meanwhile the prescribed remedy, tainting
the bootstrap node, is a full cluster re-genesis.
CA rotation is implemented, not proposed.
pki-cluster-ca-succession.md
opens with SUPERSEDED and “do not implement from this doc”; rotation
shipped with a different design.
shc cluster cert rotate --type cluster-ca|global-root|mtls|due
is live, with a resumable phase machine, --rollback and --algo.
Likewise vault-envelope-encryption.md
opens with LANDED, which is why a vault re-key re-wraps DEKs instead of
re-encrypting payloads.
Voter grants over the target are released. ReleaseVoterSlot runs
from the suffrage reconciler and from voter reconcile (ce
modules/cluster/suffrage.go, voter_reconcile.go). See voter
slots.
“acme_provider is install-time only” is a policy, not a code
restriction. --acme-provider is bound on both shc install and
shc update, the update path persists the selection, and a reconcile
recreates Traefik so new resolvers activate. The schema comment
“recreate to change CA” describes the CA-side consequence, not a refusal.
“Startup-only” is not “init-only”, and is not the confusing case. The
dev / vault / collector / rate_limit refusal is scoped to the
database-backed writer; the file layers accept them. The genuinely
confusing case is a key not on that list: server.tcp.bind is
accepted by shc config set, written, and
then ignored until a restart, because the reload fan-out rebinds no
listener. Loud refusal and silent no-op are different failures, and only
one announces itself.
Reset the trial costs less than the code claims. The trial row’s own
comment says a reset now costs “every deployment record, tenant, config
row and user in the estate.” The adoption path treats a missing row and
a missing marker as “never trialled”, and the row is unsigned, so the
real floor is one targeted delete plus a marker removal per node. Still
far above the original mv and restart, and still not worth doing, but
the record should not overstate it.
A note for anyone re-checking these citations. The local ce
checkout is routinely behind the pin in go.mod. Every ce claim on
this page was verified with git show <pin>:<path>. Reading the working
tree instead reaches different conclusions, most visibly about the
backup password constant, which the pin removed.
The weakest point in this record
Section titled “The weakest point in this record”kek_salt classified as SOFT. Every other reclassification on this
page moves with the code’s own comments; this one moves against a
warning banner that says regenerating the salt makes every envelope value
permanently undecryptable.
The argument for SOFT: that banner describes salt loss or blind regeneration, not rotation. Rotation knows both salts, and the machinery is already there and already parameterised on two KEKs: a password rotation derives an old and a new KEK and re-wraps each row’s DEK without touching the payload. Varying the salt instead of the password is the same loop.
The argument for caution: nobody has written or run that loop. The classification is a strong code reading, not an execution proof, and it contradicts a deliberate warning.
What would settle it: write the salt-rotation loop against a scratch vault holding a mix of legacy and envelope rows, rotate, and prove every value still decrypts, the same treatment that settled restic. Until then, treat “the salt cannot be lost” as certain and “the salt can be rotated” as untested.
Two further gaps worth naming, both about proof rather than reasoning:
- The fast-path stack rename is derived, not executed. The directory
move is a plain rename, the pipeline has no re-render step, and the
compose project name changes, but nobody has run
shc renameon a stack with data and watched whether the containers come back empty, come back under the old project, or fail loudly. See the deployment binding model and the storage model, which both turn on the same mount-classification derivation. - The moved state home is likewise read from code. The claim that a
pre-existing deployment’s mounts reclassify as ephemeral host binds
after the move (and that a subsequent
shc updatefully repairs them) has not been executed end to end.
Where these claims come from
Section titled “Where these claims come from”Derived by walking the code at the ce pin recorded in go.mod and at
shc HEAD, plus the two production estates’ terraform roots, which the
no-manual-applies doctrine makes authoritative for their declared
configuration.
The restic O(1) rotation is measured, not inferred: a scratch
repository, a snapshot under the first password, key add + key remove,
then key passwd, with a checksum of the repository’s data/ tree taken
at each step.
The live-estate facts quoted here (that the default backup repository
opens under the source constant, and the two estates’ dns.owner_id
values) were established read-only by a separate audit pass and are
second-hand in this page. No secret value appears anywhere on it:
credentials are described by presence, shape and provenance only.
See also
Section titled “See also”shc init: the command and its complete flag set- Cluster setup: the walkthrough these decisions are made inside
- Node rotation: replacing a node, and what it burns
- File locations: what lives inside the state home and what does not
- Config keys:
backup.repo,backup.password,dns.owner_id,acme.* - The node addressing model: proposal; the bootstrap-node asymmetry and the suffrage wording
- The deployment binding model: proposal; the sixteen sites that encode the mount path by string
- The storage model: proposal; why mount classification stays a derivation
- Network fabric: the fabric this page says is chosen once
- New-line runtime support