Skip to content
SHC Docs

Portal Terraform runner: secret injection and the run ledger (design)

Status: built, with owner rulings on 2026-08-15 and 2026-11-01. Customer credentials and harvested secrets use Frappe Password fields. State lives in a separate B2 object bucket, encrypted per cluster with AES-256-GCM; only its data key lives in a Password field. vault_password defaults to generated (class B). Sections 1-4 retain the original design discussion; §4b and §5.1 state the binding custody and mode rulings. This elaborates §6 and §10.5-10.6 of 2026-08-14-portal-architecture.md.

A portal-run terraform apply touches three kinds of secret, and they have different lifetimes, different owners, and must not share a path.

classexamplewho owns itwhere it may appear
A: customer cloud credentialHCLOUD_TOKEN, AWS access/secret, Exoscale API keythe customerprocess env of the runner, for the life of one run
B: cluster secret Terraform GENERATESssh_private_key (module output, sensitive = true), vault_password, the cluster CA materialnobody yet, born during applystate, then harvested to OpenBao
C: cluster secret Terraform CONSUMESACME eab_hmac, registry auththe portal mints it per-clusterpassed by reference, never by value

Class B is why the ruling says custody of state is custody of the cluster: terraform/modules/shc/*/outputs.tf exposes ssh_private_key as a sensitive output, so the private key is in the state file by construction.

Class A: environment only. Cloud credentials are exported into the runner process environment (HCLOUD_TOKEN, AWS_ACCESS_KEY_ID, …) for the duration of one run and never written to disk. Specifically: no .tfvars file, no -var on the command line (argv is world-readable via /proc), no Frappe DocType field, no bench log. The runner fetches them from OpenBao at job start, holds them in memory, and the job’s process death is the expiry. Where the provider supports short-lived credentials (AWS STS AssumeRole), the runner prefers minting a scoped session over holding the long-lived key.

Class C: ref+vault:// indirection, which the provider already speaks. terraform/provider/internal/provider/cluster_resource.go:213 documents eab_hmac as “(ref+vault:// ok)”, and config_masked_readback_ssh_test.go pins the deliberate rule that a ref+vault:// pointer is never masked (never_redact_value_prefixes); the reference stays legible in state and plan output while the value only ever exists inside the cluster’s vault. That is exactly the primitive the portal wants: the portal writes ref+vault://cluster_vault_password?autogen into the plan, and the secret itself is born in, and never leaves, the target cluster.

RULED 2026-11-01: vault_password is class B. It is generated during apply and lands in state, as the owner’s 2026-08-16 ruling intended. render.default_vault_password_mode is "generated"; "reference" remains an implemented alternative. The separate encrypted bucket in §4b protects state, and every run records which mode it used.

Class B: harvest then encrypt. After a successful apply the runner reads the sensitive outputs, writes them into the customer’s OpenBao compartment (this is what the portal hands the customer when they ask for their SSH key), and the state itself is stored encrypted at rest in OpenBao, never on the bench filesystem and never in the Frappe DB. State is fetched per run, locked for the duration, written back encrypted, and the plaintext working copy lives in a per-run tmpfs directory destroyed with the job, including on failure and on kill.

Blast-radius rule. One OpenBao compartment per customer, so a portal app-server compromise does not yield every customer’s infrastructure. The runner’s own OpenBao token is scoped to the single customer path for the single job, minted at job start and revoked at job end.

Every provisioning action is a Terraform Run record. This is the surface the owner asked for: “the logs of terraform so i can review them for issues.”

Each run carries: cluster, customer, cloud, action (plan/apply/destroy), who or what triggered it, module + provider versions, start/end, exit status, the resource-change counts, and the full captured stdout/stderr, streamed live to the dashboard and retained after.

Redaction is a pipeline stage, not a hope. Three defenses, because any one of them alone fails:

  1. TF_LOG is never DEBUG/TRACE in the portal runner. Terraform’s debug logging prints provider request/response bodies; that is a bulk credential leak into the log store. Default TF_LOG unset; a support engineer may raise it only for a run explicitly marked diagnostic = true, whose log is quarantined to a shorter retention and flagged in the UI.
  2. Value-based scrubbing on the way in. The runner knows every class A and class C secret it injected for that run; it scrubs those exact byte sequences (and their base64/JSON-escaped forms) out of the stream before the log is persisted. This catches the case Terraform’s own (sensitive value) masking misses: a secret echoed by a provider that never marked it sensitive.
  3. A guard test that proves it. A mutation-proven test injects a known canary secret into a fake run whose fake provider echoes it, and asserts the persisted log does not contain the canary. Removing the scrubber must red that test by name. Without this, the redactor rots silently.

State never enters the log. Not on error, not in a crash dump. A failed apply reports resource addresses and diagnostics, never the state document.

Retention and access: run records are customer-visible for their own clusters (that is the review surface), staff-visible under the same support-bundle consent modes ruled in §5: a one-off code or a standing grant, not ambient staff access to every customer’s infrastructure log.

A failed apply must not leak billing. The runner’s failure path is reap-on-failure (§10.6): on a non-zero apply the runner attempts destroy of what that run created, and if the destroy also fails it opens a leak record naming the cloud, the run, and the resource addresses, a loud, tracked artifact rather than a silent orphan. Two independent findings from the 2026-08-14 campaign motivate this: a Linode volume that billed silently because product-created volumes carry no estate tag, and perfchaos nodes that were invisible to the documented cleanup selector because the campaign tag was written as a label value instead of a label key. The portal must tag every provisioned resource in the house form, and the leak record is the backstop when tagging or teardown fails.

Per the C1 amendment, the reaper is campaign insurance and is explicitly not the portal’s answer; the portal relies on the product teardown path plus this leak record.

4b. RULED 2026-08-15: Frappe password fields, not OpenBao (owner)

Section titled “4b. RULED 2026-08-15: Frappe password fields, not OpenBao (owner)”

Owner ruling, verbatim: “use the frappe db to store the secrets using password field which i think is encrypted”. OpenBao is OUT for phase 1.

He is right that it is encrypted: a DocField of fieldtype Password is stored in the __Auth table (not the doctype table), encrypted with Fernet (AES-128-CBC + HMAC) keyed on the site’s encryption_key from site_config.json, and read back only through get_password(). It never appears in frappe.db.get_value, in a list view, in the API’s normal document payload, or in the desk UI. That is genuinely better than a plain field and it is the framework’s own supported answer.

What this ruling changes: §2’s class A (customer cloud credentials) and class B (harvested SSH keys) live in Password fields on the customer’s Frappe record. ref+vault:// (class C) is unaffected; those never touched the portal anyway. The per-run injection rules in §2 stand unchanged: read via get_password() at job start, into the process environment only, never a tfvars file, never argv, never a log.

Three properties the framework does not give us, recorded so they are chosen rather than discovered:

  1. One key for the whole site. encryption_key decrypts every customer’s secrets, and it sits in site_config.json on the same host as the database. There is no per-customer separation; a single host compromise or a leaked backup pair (DB + site config) exposes every customer at once. Mitigation while staying on Frappe: keep encryption_key out of the DB backup path, hold it in the container’s environment/secret mount rather than the on-disk file where the platform allows, and treat DB+config as a single custody boundary in the runbook.
  2. bench backup is now a secret export. Any routine backup, replica, or staging restore carries __Auth. Mitigation: the portal bench’s backups are encrypted at rest and access-controlled like the secrets they contain, and no prod→staging data copy is permitted without scrubbing __Auth. This must be written into the operational runbook, not assumed.
  3. No lease, no revocation, no read audit. get_password() leaves no trail. Mitigation: the §3 run ledger becomes the audit surface; every run records which customer’s credentials it read, so “which secrets did that job touch” stays answerable at the job level even though the framework cannot answer it at the secret level.

RULED 2026-11-01: state has its own custody boundary. Terraform state leaves __Auth. It lives in its own bucket on the existing bitspur B2 account, using the S3-compatible API. Each cluster gets a random 32-byte data key in Terraform State.data_key, a Password field. AES-256-GCM encrypts each state object with the cluster name as authenticated additional data; its key is <prefix>/<cluster>.tfstate.enc. A DB backup holds keys without state, and the bucket holds ciphertext without keys. Neither alone recovers the state.

There is no fallback to Password chunks. The runner refuses before taking a state lock or creating a workdir when the bucket configuration is missing, and names the missing configuration keys in the failed run’s redacted log. The old chunk DocType is removed by a migration patch. Credentials and harvested cluster secrets retain the 2026-08-15 Password-field ruling.

Implemented at frappe/apps/marketplace/marketplace/provisioning/, with the operating half at docs/guide/operations/portal-secret-custody.md. Everything in §2, §3 and §4 above is code with a guard test behind it, and every §4b mitigation is implemented rather than noted: custody of the site encryption_key is stamped on each run row, scripts/portal-backup-guard.sh refuses a prod→non-prod restore still carrying __Auth, and secrets_read on each run row is the job-level read audit.

Two implementation notes worth carrying forward because they are not obvious from the design text:

  • The provider was already published. 38 versions in the project’s generic registry tracking tags to 0.0.56, plus the 0.0.0-latest rolling channel, behind an immutability guard. There was never a publishing prerequisite; §10.6 of the architecture doc is stale on that point. What there IS is a choice, and the runner pins an exact release: the rolling channel is re-uploaded on every main push, so pinning it would let a bad push silently change what every customer provision does, with no release and nothing to roll back to.
  • State is encrypted per cluster in B2. The 2026-11-01 ruling replaces the Password-chunk implementation. The database keeps the data key and object metadata, never the state document.
  1. vault_password: RULED 2026-11-01, generated (class B). This makes explicit the owner’s 2026-08-16 ruling that the vault password lives in state. render.default_vault_password_mode is "generated"; the runner harvests it into a Password field after apply. State’s separate bucket and per-cluster encryption provide its custody boundary. "reference" remains implemented as the alternative, and every run records its mode.
  2. Does the customer ever see raw cloud credentials again after entry? (Write-only-then-opaque is the safer default; it costs a re-entry when a key rotates.)
  3. Log retention window, and whether a customer may delete their own run logs (auditability vs. their data rights).
  4. Concurrency: one run at a time per cluster is assumed (state locking demands it); confirm the queue is per-cluster, not global.
  5. Managed phase: when we hold the cloud account (§6), class A collapses into our own credential; the compartment-per-customer rule still applies to state and class B, but the injection story changes.