Reaching estate nodes over SSH
Estate access and shell access are two different things. The CLI reaches
the daemon API over HTTPS through the device flow, so shc verbs keep
working with no shell at all: a Permission denied (publickey) against
every node tells you nothing about whether the estate is healthy. This
page is about the second kind: getting a login shell on a node.
The short version:
| Question | Answer |
|---|---|
| Which user? | admin, on every node. Never root. |
| Which key? | The cluster key the estate’s terraform root mints. |
| Where does the private half live? | The estate’s tofu state. Nowhere else. |
| How many keys authorize the admin user? | Usually exactly one. |
| Can a converge add a second one in place? | No; see below. |
The cluster key is a terraform resource, not an operator input
Section titled “The cluster key is a terraform resource, not an operator input”The substrate modules take the node keypair one of two ways, selected by
create_ssh_key:
create_ssh_key = true: the module mintstls_private_key.clusteritself. In this modessh_public_keyis optional and becomes a second, break-glass key (<cluster>-operator) so a human can still reach nodes when terraform or its state is the thing that broke.create_ssh_key = false(the default) means BYO:ssh_public_keyis required and backs every node’s one shared key slot.
The estate roots in this project use the second mode, with a
tls_private_key.cluster declared in the root module and handed to
the substrate module as ssh_public_key. That shape has a consequence
worth stating plainly, because it is easy to read the variable
documentation and assume otherwise:
In BYO mode the break-glass key does not exist. The
hcloud_ssh_key.operatorresource is countedcreate_ssh_key && ssh_public_key != null, so withcreate_ssh_keyfalse the count is zero no matter whatssh_public_keyis set to. The second key slot the variable describes is reachable only in mint mode.
So: one key per estate, and the private half of it lives in the state.
Retrieving it
Section titled “Retrieving it”Two routes. Prefer the first; the second is for when you have no working
directory, no provider cache, and no appetite for a tofu init.
From an estate working directory
Section titled “From an estate working directory”Every estate root exports the key as a sensitive output:
cd <estate>/<region>/infratofu init # the state is remote; init is what attaches to itumask 077tofu output -raw ssh_private_key > ./cluster.keychmod 600 ./cluster.keyssh -i ./cluster.key -o IdentitiesOnly=yes admin@<node-ip>Sibling outputs on the same root cover the other genesis secrets:
vault_password, keycloak_admin_password, cluster_ssh_hosts,
node_ips.
From the state over the GitLab API
Section titled “From the state over the GitLab API”The estates keep state in GitLab’s managed terraform state backend, one
state per estate layer, named for the estate
(infra.<region>.<cloud>.<domain>). The current state reads back over the
REST API with a token carrying api scope:
umask 077curl -sL --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \ "https://gitlab.com/api/v4/projects/<ops-project-id>/terraform/state/<state-name>" \ > state.jsonThe key is the root-module tls_private_key named cluster; take
instances[0].attributes.private_key_openssh from it. The same object
carries public_key_fingerprint_sha256, which is what you compare against
ssh-keygen -l -f <node's authorized_keys> to tell a live key from a
stale one.
Two notes that save time:
- List the state names for a project with the GraphQL
project.terraformStatesfield. There is no REST listing. - The per-version REST endpoint (
…/state/<name>/versions/<serial>) answers404on this instance. If your recovery plan assumes you can fetch an older state version to recover a superseded key, test that assumption before you need it.
Handling
Section titled “Handling”The state holds the vault master password and the Keycloak admin password next to the key, so treat the whole file as key material:
umask 077before you write anything, andchmod 600after.- Write to a directory only you can traverse. A shared temp directory is not one.
- Delete both the key and the state dump when you are done. A copy left behind in a world-traversable path is the reason this page exists.
- Never commit either. See the next section for what happens when someone does.
Why the copy in your checkout is probably dead
Section titled “Why the copy in your checkout is probably dead”Estate working directories tend to accumulate a convenience copy of the
key: a .ssh/cluster file next to the root module. Those copies are
meant to be gitignored, and nothing regenerates them. No CI job, no
tofu apply, no make target refreshes one.
Any event that recreates tls_private_key.cluster mints a fresh keypair.
A node rebuild is such an event: destroy-and-recreate takes the key with
it, the new nodes come up authorizing the new public half, and the file in
your checkout keeps the old one. The failure is silent: you learn about
it the next time you try to log in, which may be weeks later.
Two rules follow:
- The state is the only source of truth. If a key file and the state disagree, the state is right. Compare fingerprints rather than guessing.
- A committed copy is worse than a stale one. A private key in git
history stays in git history after it goes dead, and cannot be verified
as dead by anyone reading the repo later. Check that your estate’s
.gitignoreactually covers the path; an ignore rule that only names.terraform/providers/does not cover.ssh/cluster.
Why you cannot add a key with a converge
Section titled “Why you cannot add a key with a converge”Both channels that put a key on a node run once, at create time:
- the cloud key slot (
hcloud_server.ssh_keysand its equivalents), and - cloud-init
user_data, which is whereextra_authorized_keysand the admin user’sssh_authorized_keysare rendered.
Changing either replaces the server. extra_authorized_keys documents
this itself: “changing it later REPLACES servers (append by hand over an
existing key instead).”
There is also no other door to fall back on. On a converged node:
disable_rootplus aPermitRootLogin nosshd drop-in refuse root logins, and the cloud-side key that lands onrootcarries the image’s forced-command wrapper anyway.ssh_pwauth: false: no password authentication.- The image’s own default user is never created. The module’s
cloud-init
users:list namesadminand does not include- default, and cloud-init treats an explicit list as the whole list. On a Debian image there is nodebianuser on the node at all. (This is the same mechanism as the earlier bug where ausers:list dropped the default user’sNOPASSWDsudo: an explicit entry overrides the default, it does not merge with it.)
So the set of credentials that can reach a running estate is fixed at node creation, and for these estates it has exactly one member.
The two ways this bites
Section titled “The two ways this bites”Losing the state is losing the estate’s shell
Section titled “Losing the state is losing the estate’s shell”One key, one copy, one custodian. If the state is deleted, or the token that reads it is revoked, or the project holding it goes away, then shell access to a healthy, running estate is gone permanently. The nodes cannot be re-keyed in place, so the only remaining route is the provider’s rescue console.
The daemon API keeps working throughout, which makes this easy to under-rate: the estate looks fine, and stays fine, right up to the first task that needs a shell.
A partial converge can orphan access silently
Section titled “A partial converge can orphan access silently”This one is the sharper edge. Consider protected = true, the posture
the variable’s own documentation recommends (“flip true once the estate is
live”), which sets the cloud’s native delete and rebuild protection on
every node.
Now suppose anything replaces tls_private_key.cluster: a taint, a
-replace, an algorithm change, a state restore that lost the resource.
The apply proceeds in dependency order, and the keypair has no
dependencies, so:
- tofu mints the new keypair first and writes it to state. The old private half is not retained.
- The node replacement that would carry the new public key onto the nodes then fails, because the nodes are rebuild-protected.
- The apply ends partially applied: the state now holds a key that is authorized nowhere, and the nodes authorize a key that is no longer in the state.
That is a permanent lockout produced by a converge that got halfway. It is quiet: the error you see is about protection, not about access, and the estate keeps serving traffic.
With protected = false the same trigger instead replaces every node,
which is destructive but loud. Neither posture is good; they differ in
whether you find out immediately.
What to do about it
Section titled “What to do about it”None of this is applied automatically; re-keying an estate is a consented change, not a cleanup. The options, in the order they are worth doing:
- Get a second key onto the nodes. Append an operator public key to
/home/admin/.ssh/authorized_keysby hand over an existing session (this is whatextra_authorized_keystells you to do), and add the same key toextra_authorized_keysso the next rebuild carries it without a hand step. The by-hand append is not drift:user_datais never re-read on a live node. - Prefer mint mode with a break-glass key on new estates. With
create_ssh_key = trueandssh_public_keyset to an operator key, the estate comes up with two independent credentials by construction. - Verify your state recovery route before you need it, including whether you can actually fetch a previous state version.
- Never set
protected = truewhile the estate has one key, unless you have first confirmed that no pending plan touches the keypair.
See also
Section titled “See also”- Rotating nodes in and out: what a node replacement costs, which is what re-keying an estate amounts to.
- Disaster recovery: quorum loss, which is a different failure from this one and needs shell access to fix.
- Vault: the other genesis secret that lives in the same state.