Portal secret custody: operating a bench that holds customer cloud credentials
Applies to: the Portal bench (my.selfhostedcloud.org) once the terraform
runner is live. Design of record:
docs/proposals/2026-08-15-portal-terraform-runner.md.
The owner ruled on 2026-08-15 that customer cloud credentials live in Frappe
Password fields rather than OpenBao. That ruling is sound: fieldtype
Password lands in the __Auth table, Fernet-encrypted (AES-128-CBC + HMAC) on
the site’s encryption_key, readable only through get_password(), invisible to
frappe.db.get_value, to list views, to the REST document payload and to the
desk UI.
It also changes three things about how this bench must be operated. The runner implements what code can implement; this document is the rest, and it is not optional; proposal §4b records these as chosen properties, which only holds if somebody actually operates them.
1. One key decrypts every customer
Section titled “1. One key decrypts every customer”encryption_key is site-wide. It decrypts every customer’s cloud
credentials and every cluster’s harvested SSH key, and by default it sits in
site_config.json on the same host as the database. A single host compromise,
or a leaked backup pair (DB dump + site config), exposes the whole book of
business at once, not one customer.
Operate it this way:
-
Hold
encryption_keyin the container’s environment / secret mount (FRAPPE_ENCRYPTION_KEY), not in the on-disksite_config.json, wherever the platform allows it. -
Treat database + site config as ONE custody boundary. They are never backed up to the same destination, never handed to the same operator for a restore, and never both present on a laptop.
-
The runner stamps
encryption_key_custody(env/site_config/unknown) on every Terraform Run row. That field is the estate’s actual posture, queryable, per run. If it sayssite_configin production, that is a finding, not a default.select encryption_key_custody, count(*)from `tabTerraform Run`where creation > date_sub(now(), interval 30 day)group by 1;
2. bench backup is now a secret export
Section titled “2. bench backup is now a secret export”Every routine backup, every replica, every “let me copy prod into staging to
reproduce it” carries __Auth. That tarball is not a database backup any more;
it is a credential archive.
Operate it this way:
-
Portal bench backups are encrypted at rest and access-controlled like the secrets they contain, not like a database backup.
-
No prod → staging/dev data copy without scrubbing
__Auth. The check isfrappe/apps/marketplace/scripts/portal-backup-guard.sh:Terminal window # refuse a dump that still carries __Auth into a non-prod sitesh scripts/portal-backup-guard.sh dump.sql.gz staging# produce a restorable dump with the __Auth ROWS removed (schema kept)sh scripts/portal-backup-guard.sh --scrub dump.sql.gz > scrubbed.sqlA staging portal that can reach a customer’s cloud account is a staging portal that can bill a customer’s cloud account.
-
Restore drills use the scrubbed dump. If a drill needs real secrets to pass, the drill is testing the wrong thing.
3. There is no read audit, no lease, no revocation
Section titled “3. There is no read audit, no lease, no revocation”get_password() leaves no trail. The framework cannot tell you who read which
secret when, and it cannot revoke one.
The mitigation is the run ledger, and it works at the job level:
-
every Terraform Run row records
secrets_read: the credential field names the run read, never values; -
so “which secrets did that job touch?” is answerable, and “which jobs touched this customer’s credentials?” is a query:
select name, action, status, started_at, secrets_readfrom `tabTerraform Run`where customer = %(customer)sorder by started_at desc; -
revocation is manual and it is the customer’s cloud console, not ours. When a credential is suspected exposed, rotate it at the cloud provider and re-enter it in the portal.
Portal Cloud Credential.write_onlydefaults on, so the portal never renders it back; a rotation costs a re-entry, which is the trade §5.2 of the proposal names.
What the code already enforces (do not re-check by hand)
Section titled “What the code already enforces (do not re-check by hand)”| Property | Where | Guard test |
|---|---|---|
| Credentials only in the process environment | provisioning/runenv.py | test_secret_injection.py |
| Never a tfvars file, never argv | provisioning/runenv.py, runner.py | test_secret_injection.py |
TF_LOG never DEBUG/TRACE/JSON outside a diagnostic run | provisioning/runenv.py | test_log_policy.py |
| Injected secrets scrubbed from the log in raw/base64/JSON form | provisioning/redact.py | test_redaction_canary.py |
| Per-run workdir on tmpfs, destroyed on success/failure/kill | provisioning/workdir.py | test_workdir.py |
| Failed apply reaps; failed reap opens a leak record | provisioning/runner.py | test_reap_and_tags.py |
| Every resource tagged in house KEY form; selector derived from it | provisioning/tags.py | test_reap_and_tags.py |
| Provider + module pinned to an exact release | provisioning/versions.py | test_ledger_and_versions.py |
CI runs the Go-driven scenarios in tests/unit/portal/clouds_test.go and
provisioning_test.go. The Python drivers use fake HTTP and subprocesses;
the state driver exercises real AES-256-GCM. The older marketplace Python
unittests remain available locally and are not reintroduced into CI.
4. State has its own bucket (RULED 2026-11-01)
Section titled “4. State has its own bucket (RULED 2026-11-01)”Terraform state holds the SSH key and vault password together. It is stored
in a dedicated bucket in the existing bitspur B2 account, through its
S3-compatible API. Each cluster has a random 32-byte data key in the
Terraform State.data_key Password field. State is encrypted with
AES-256-GCM using the cluster name as AAD; the object key is
<prefix>/<cluster>.tfstate.enc. The row also keeps serial, byte count and
lock metadata.
The DB backup carries keys without state. The bucket carries ciphertext
without keys. Keep the bucket and bench backup access separate. The existing
credential and harvested-secret backup precautions above still apply.
There is no __Auth state fallback and no chunked state DocType.
Configure portal_state_bucket, portal_state_endpoint,
portal_state_region and portal_state_prefix in site config. Supply
PORTAL_STATE_KEY_ID and PORTAL_STATE_APP_KEY through vault-backed worker
environment variables. Clay supplies their values; deployment does not mint
them. A run with missing bucket configuration fails before a lock, workdir or
Terraform command and names the missing keys in its redacted log.
5. Provisioning posture
Section titled “5. Provisioning posture”The marketplace stack’s provisioning variables write the non-secret site
configuration during bench-configure. Provisioning is closed by default:
portal_provisioning_open must be enabled, or the caller must be a System
Manager or administer an org in portal_provisioning_orgs. Connecting a cloud
remains open to org admins. Set portal_cluster_domain,
portal_provision_shc_version and portal_cluster_portal_url before a provision.
The domain suffix is still an open-design placeholder.
OAuth client IDs live in portal_oauth_clients; the three
PORTAL_OAUTH_<CLOUD>_CLIENT_SECRET values come from the deployment’s vault
rows portal_oauth_<cloud>_client_secret, mounted on the web and worker
containers as PORTAL_OAUTH_<CLOUD>_CLIENT_SECRET_FILE. Leave the
provisioning.<cloud>ClientId variables empty until the clients exist. PORTAL_PROVISION_NETBIRD_SETUP_KEY is optional and seeded
only in dev. Its presence opts into NetBird enrol; the environment name is
not a gate. Registration always uses shc portal register - over SSH
after apply, with the code carried on stdin rather than in any argv.
vault_password uses render.default_vault_password_mode = "generated", the
2026-11-01 class-B ruling. "reference" remains implemented as an alternative.
The run ledger records the chosen mode, preflight, refresh and enrol outcome.
An enrol failure preserves the created cluster for diagnosis.
The current CLI has a registration prerequisite outside this backend:
modules/portal/commands.go calls an in-process agent, while
daemon/wiring/portal_wire.go publishes that agent only in the daemon.
cmd/shc/main.go does not wire it. Until that CLI path is connected, a real
post-apply registration can refuse with X503145PTLNYI. The runner records the
enrol failure and keeps the cluster. The offline SSH fake verifies the
invocation and secret handling, not successful live registration.