Skip to content
SHC Docs

Reaching estate nodes over SSH

Estate access and shell access are two different things. The CLI reaches the daemon API over HTTPS through the device flow, so shc verbs keep working with no shell at all: a Permission denied (publickey) against every node tells you nothing about whether the estate is healthy. This page is about the second kind: getting a login shell on a node.

The short version:

QuestionAnswer
Which user?admin, on every node. Never root.
Which key?The cluster key the estate’s terraform root mints.
Where does the private half live?The estate’s tofu state. Nowhere else.
How many keys authorize the admin user?Usually exactly one.
Can a converge add a second one in place?No; see below.

The cluster key is a terraform resource, not an operator input

Section titled “The cluster key is a terraform resource, not an operator input”

The substrate modules take the node keypair one of two ways, selected by create_ssh_key:

  • create_ssh_key = true: the module mints tls_private_key.cluster itself. In this mode ssh_public_key is optional and becomes a second, break-glass key (<cluster>-operator) so a human can still reach nodes when terraform or its state is the thing that broke.
  • create_ssh_key = false (the default) means BYO: ssh_public_key is required and backs every node’s one shared key slot.

The estate roots in this project use the second mode, with a tls_private_key.cluster declared in the root module and handed to the substrate module as ssh_public_key. That shape has a consequence worth stating plainly, because it is easy to read the variable documentation and assume otherwise:

In BYO mode the break-glass key does not exist. The hcloud_ssh_key.operator resource is counted create_ssh_key && ssh_public_key != null, so with create_ssh_key false the count is zero no matter what ssh_public_key is set to. The second key slot the variable describes is reachable only in mint mode.

So: one key per estate, and the private half of it lives in the state.

Two routes. Prefer the first; the second is for when you have no working directory, no provider cache, and no appetite for a tofu init.

Every estate root exports the key as a sensitive output:

Terminal window
cd <estate>/<region>/infra
tofu init # the state is remote; init is what attaches to it
umask 077
tofu output -raw ssh_private_key > ./cluster.key
chmod 600 ./cluster.key
ssh -i ./cluster.key -o IdentitiesOnly=yes admin@<node-ip>

Sibling outputs on the same root cover the other genesis secrets: vault_password, keycloak_admin_password, cluster_ssh_hosts, node_ips.

The estates keep state in GitLab’s managed terraform state backend, one state per estate layer, named for the estate (infra.<region>.<cloud>.<domain>). The current state reads back over the REST API with a token carrying api scope:

Terminal window
umask 077
curl -sL --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
"https://gitlab.com/api/v4/projects/<ops-project-id>/terraform/state/<state-name>" \
> state.json

The key is the root-module tls_private_key named cluster; take instances[0].attributes.private_key_openssh from it. The same object carries public_key_fingerprint_sha256, which is what you compare against ssh-keygen -l -f <node's authorized_keys> to tell a live key from a stale one.

Two notes that save time:

  • List the state names for a project with the GraphQL project.terraformStates field. There is no REST listing.
  • The per-version REST endpoint (…/state/<name>/versions/<serial>) answers 404 on this instance. If your recovery plan assumes you can fetch an older state version to recover a superseded key, test that assumption before you need it.

The state holds the vault master password and the Keycloak admin password next to the key, so treat the whole file as key material:

  • umask 077 before you write anything, and chmod 600 after.
  • Write to a directory only you can traverse. A shared temp directory is not one.
  • Delete both the key and the state dump when you are done. A copy left behind in a world-traversable path is the reason this page exists.
  • Never commit either. See the next section for what happens when someone does.

Why the copy in your checkout is probably dead

Section titled “Why the copy in your checkout is probably dead”

Estate working directories tend to accumulate a convenience copy of the key: a .ssh/cluster file next to the root module. Those copies are meant to be gitignored, and nothing regenerates them. No CI job, no tofu apply, no make target refreshes one.

Any event that recreates tls_private_key.cluster mints a fresh keypair. A node rebuild is such an event: destroy-and-recreate takes the key with it, the new nodes come up authorizing the new public half, and the file in your checkout keeps the old one. The failure is silent: you learn about it the next time you try to log in, which may be weeks later.

Two rules follow:

  1. The state is the only source of truth. If a key file and the state disagree, the state is right. Compare fingerprints rather than guessing.
  2. A committed copy is worse than a stale one. A private key in git history stays in git history after it goes dead, and cannot be verified as dead by anyone reading the repo later. Check that your estate’s .gitignore actually covers the path; an ignore rule that only names .terraform/providers/ does not cover .ssh/cluster.

Both channels that put a key on a node run once, at create time:

  • the cloud key slot (hcloud_server.ssh_keys and its equivalents), and
  • cloud-init user_data, which is where extra_authorized_keys and the admin user’s ssh_authorized_keys are rendered.

Changing either replaces the server. extra_authorized_keys documents this itself: “changing it later REPLACES servers (append by hand over an existing key instead).”

There is also no other door to fall back on. On a converged node:

  • disable_root plus a PermitRootLogin no sshd drop-in refuse root logins, and the cloud-side key that lands on root carries the image’s forced-command wrapper anyway.
  • ssh_pwauth: false: no password authentication.
  • The image’s own default user is never created. The module’s cloud-init users: list names admin and does not include - default, and cloud-init treats an explicit list as the whole list. On a Debian image there is no debian user on the node at all. (This is the same mechanism as the earlier bug where a users: list dropped the default user’s NOPASSWD sudo: an explicit entry overrides the default, it does not merge with it.)

So the set of credentials that can reach a running estate is fixed at node creation, and for these estates it has exactly one member.

Losing the state is losing the estate’s shell

Section titled “Losing the state is losing the estate’s shell”

One key, one copy, one custodian. If the state is deleted, or the token that reads it is revoked, or the project holding it goes away, then shell access to a healthy, running estate is gone permanently. The nodes cannot be re-keyed in place, so the only remaining route is the provider’s rescue console.

The daemon API keeps working throughout, which makes this easy to under-rate: the estate looks fine, and stays fine, right up to the first task that needs a shell.

A partial converge can orphan access silently

Section titled “A partial converge can orphan access silently”

This one is the sharper edge. Consider protected = true, the posture the variable’s own documentation recommends (“flip true once the estate is live”), which sets the cloud’s native delete and rebuild protection on every node.

Now suppose anything replaces tls_private_key.cluster: a taint, a -replace, an algorithm change, a state restore that lost the resource. The apply proceeds in dependency order, and the keypair has no dependencies, so:

  1. tofu mints the new keypair first and writes it to state. The old private half is not retained.
  2. The node replacement that would carry the new public key onto the nodes then fails, because the nodes are rebuild-protected.
  3. The apply ends partially applied: the state now holds a key that is authorized nowhere, and the nodes authorize a key that is no longer in the state.

That is a permanent lockout produced by a converge that got halfway. It is quiet: the error you see is about protection, not about access, and the estate keeps serving traffic.

With protected = false the same trigger instead replaces every node, which is destructive but loud. Neither posture is good; they differ in whether you find out immediately.

None of this is applied automatically; re-keying an estate is a consented change, not a cleanup. The options, in the order they are worth doing:

  1. Get a second key onto the nodes. Append an operator public key to /home/admin/.ssh/authorized_keys by hand over an existing session (this is what extra_authorized_keys tells you to do), and add the same key to extra_authorized_keys so the next rebuild carries it without a hand step. The by-hand append is not drift: user_data is never re-read on a live node.
  2. Prefer mint mode with a break-glass key on new estates. With create_ssh_key = true and ssh_public_key set to an operator key, the estate comes up with two independent credentials by construction.
  3. Verify your state recovery route before you need it, including whether you can actually fetch a previous state version.
  4. Never set protected = true while the estate has one key, unless you have first confirmed that no pending plan touches the keypair.
  • Rotating nodes in and out: what a node replacement costs, which is what re-keying an estate amounts to.
  • Disaster recovery: quorum loss, which is a different failure from this one and needs shell access to fix.
  • Vault: the other genesis secret that lives in the same state.