Skip to content
SHC Docs

Vultr

Operator guide for standing up an SHC cluster on Vultr cloud compute. Built from a live reference estate (region ewr, first applied 2026-07); every quirk below was found on real hardware, not guessed.

ConcernValue
Terraform moduleterraform/modules/shc/vultr (front door)
Example rootterraform/examples/vultr-shc
Substrate providervultr/vultr ~> 2.0
SHC connection typevultr, CredFields api_key, region
DNS approachCloudflare-parent (Vultr DNS refuses subdomain zones; see gotchas)
DNS connection typecloudflare, CredFields token
Volume providervultr-volume (block storage; region-scoped)
Load-balancer providervultr-lb (SharedVIP; tcp/http/https only)
Node imageDebian 13 x64 (trixie) (x86 / amd64 plans only)
Credentials (env)VULTR_API_KEY, plus a Cloudflare token for DNS

One Vultr API key drives both the substrate (the vultr Terraform provider) and the daemon’s vultr connection (dns/proxy/volume). A Vultr API key is account-level (it cannot be scoped into per-surface sub-keys), so the daemon reuses the account key exactly as the DigitalOcean estate reuses its account token. There is no scoped daemon sub-credential the way AWS/GCP have.

  1. Vultr customer portal → Account → API.

  2. Enable API access and copy the Personal Access Token.

  3. Turn off the IP allow-list (portal: “Access Control”); see the gotchas below. The default allow-list contains only your current IP, which silently 403s any CI runner or shared build host.

  4. Export it:

    Terminal window
    export VULTR_API_KEY=... # substrate + the vultr connection
    export TF_VAR_vultr_api_key="$VULTR_API_KEY"

    The vultr Terraform provider reads VULTR_API_KEY natively; the estate also wires the provider block from var.vultr_api_key so the same key feeds the SHC vultr connection with one source of truth.

Because Vultr DNS cannot host subdomain zones, managed DNS runs through Cloudflare on the parent zone. Mint a token scoped Zone:DNS:Edit on the one zone you will serve records under:

Terminal window
export TF_VAR_dns_token=... # Cloudflare API token, Zone:DNS:Edit

Keep both out of *.tfvars files and out of Terraform state variables; pass them as TF_VAR_* / environment only.

Two connections light the platform. The estate wires them as first-class shc_connection resources at system scope (provider ≥ 0.0.322), so the tenant-less cluster-host DNS surface resolves them directly:

# Proxy (vultr-lb) + volume (vultr-volume). Its DNS surface goes UNUSED:
# Vultr DNS can't host subdomain zones (see gotchas).
resource "shc_connection" "vultr" {
name = "vultr"
type = "vultr"
credentials = {
api_key = var.vultr_api_key # the same account key
region = module.shc.region # required by vultr-lb AND vultr-volume
}
}
# Managed DNS on the parent zone.
resource "shc_connection" "cloudflare" {
name = "cloudflare"
type = "cloudflare"
credentials = { token = var.dns_token }
zones = ["example.com"] # YOUR Cloudflare-hosted parent zone
}

Then the post-init defaults, one YAML document (provider ≥ 0.3.0):

resource "shc_config" "platform" {
yaml = <<-EOF
dns:
provider: cloudflare # the connection NAME
public_address: ${local.bootstrap_ip}
storage:
volumes:
provider: vultr # vultr-volume dispatch (carries region)
acme:
email: [email protected] # arms public Let's Encrypt (DNS-01)
EOF
depends_on = [shc_connection.vultr, shc_connection.cloudflare]
}

region is required in the vultr connection credentials: both vultr-lb and vultr-volume are region-scoped and provision in it.

The obvious design (delegate vultr.example.com into Vultr DNS with a vultr_dns_domain + parent NS records) does not work. Vultr DNS accepts only registrable domains; a subdomain zone is rejected at create:

Error: error while creating domain :
{"error":"Unable to create domain: Subdomains are not permitted","status":400}

So every SHC estate on Vultr hosts its *.<sub>.example.com records as ordinary rows inside the Cloudflare parent zone, the same shape the Hetzner and DigitalOcean estates use. The vultr connection still lights vultr-lb and vultr-volume; only its DNS surface goes unused.

SHC writes only records for hosts it claims, each guarded by a companion ownership TXT: it never touches records it did not create, so pointing it at a zone you also use for other things is safe.

With dns.provider set and bootstrap.host seeded, the daemon self-registers the cluster-host A record (cloud.<sub>.example.com) every leader tick (~60s) and fails closed rather than publish a LAN address; there is no hand-managed shc_dns_record resource to maintain.

  • vultr-volume: block storage, region-scoped. Set storage.volumes.provider: vultr (as above); the daemon provisions, attaches (by node name), formats, and mounts a volume for any app declaring x-shc-volumes. Recreate caveat: Vultr assigns each node’s VPC plane IP itself (no cidrhost pinning on this substrate), so treat node replacement as remove-node + join-node, not an in-place recreate. Detach caveat: Vultr often refuses a live detach from a running instance (“Unable to unmount block device”, for as long as it is asked) and takes one from a stopped instance at once. A non-forced detach from a running node therefore fails with X409351VLTLIV and never reloads the node; a forced detach (the node is asserted gone) follows a refusal with Vultr’s reloading detach, which restarts that instance (SHC-1800).
  • vultr-lb: a SharedVIP Load Balancer in front of the system Traefik. It is off by default; flip it on with an shc_stack_ports attach (vultr@80 / vultr@443). vultr-lb is tcp/http/https only: a UDP conn@ mapping is rejected at install. For a single-node or direct-ingress cluster you don’t need it: Traefik on 80/443 on the bootstrap node, with the cluster host pointed at that node’s IP, serves the edge directly.

Full worked root: terraform/examples/vultr-shc. It mirrors the live estate: one module "shc" (infra + bootstrap) plus the vultr + cloudflare connections and shc_config, with the DNS provider, base domain, and name prefix all exposed as variables so you can point it at your own single Cloudflare zone.

Terminal window
cd terraform/examples/vultr-shc
export VULTR_API_KEY=...
export TF_VAR_vultr_api_key="$VULTR_API_KEY"
export TF_VAR_dns_token=... # Cloudflare Zone:DNS:Edit token
tofu init # needs the provider network-mirror config (~/.tofurc); see below
tofu apply \
-var dns_zone=example.com \
-var cluster_name=shc-vultr \

tofu apply alone provisions the instances + VPC + firewall groups, installs the .deb and runs shc init / shc node join over the provider’s SSH transport, wires both connections and the platform config, and (via the demo shc_deployment in the example) installs a public whoami app that gets a real Let’s Encrypt certificate through Cloudflare DNS-01.

Verify:

Terminal window
tofu output -raw ssh_private_key > /tmp/vultr-key && chmod 600 /tmp/vultr-key
ssh -i /tmp/vultr-key admin@"$(tofu output -json node_management_ipv4s | jq -r '.[0]')" \
'sudo shc cluster status; sudo shc connection ls'
curl -fsS https://whoami.example.com/ # 200 + request dump
echo | openssl s_client -connect whoami.example.com:443 \
-servername whoami.example.com 2>/dev/null | openssl x509 -noout -issuer # Let's Encrypt

Provider mirror: terraform-provider-shc is not on the public registry. tofu init needs TF_CLI_CONFIG_FILE / ~/.tofurc pointing at the SHC network mirror (https://selfhostedcloud.org/terraform/0.1/); a bare tofu init fails with registry.opentofu.org does not have a provider named .../shc. See the repo’s PUBLISHING.md.

Each of these was found on a real Vultr apply. Symptom first, then the fix.

2. Vultr DNS refuses subdomain zones → Cloudflare-parent

Section titled “2. Vultr DNS refuses subdomain zones → Cloudflare-parent”

Symptom: creating a vultr_dns_domain for sub.example.com fails with 400 "Subdomains are not permitted".

Fix: don’t delegate a subdomain into Vultr DNS. Host the records in the parent zone’s own provider (Cloudflare) and point the daemon at a cloudflare connection instead; see DNS above. This is the documented default estate shape on Vultr.

3. vultr_ssh_key trailing-newline perpetual diff

Section titled “3. vultr_ssh_key trailing-newline perpetual diff”

Symptom: every tofu plan shows a benign in-place update to the vultr_ssh_key resource, forever, even with no change.

Fix: trimspace() the key material. tls_private_key’s public_key_openssh carries a trailing \n, but the Vultr API returns the stored key normalized without it, so the values never compare equal. The module already wraps ssh_key in trimspace(); do the same in any hand-rolled vultr_ssh_key.

4. The API key is IP-allow-listed by default

Section titled “4. The API key is IP-allow-listed by default”

Symptom: Terraform / the daemon get 403 from the Vultr API from a CI runner or shared build host, even though the key is valid; it works from your laptop but not from automation.

Fix: in the Vultr portal (Account → API → Access Control) set the allow-list to allow-all (0.0.0.0/0), or add the runner’s egress ranges. Shared CI runners have non-allowlistable, rotating egress IPs, so allow-all is the practical setting for automation. (Rotate/retire the key when the estate is torn down.)

5. Plane NIC names and no plane-IP pinning

Section titled “5. Plane NIC names and no plane-IP pinning”

Symptom (minor): the VPC interface is enp8s0 (public is enp1s0) on Debian 13; set nic_name if your plan/image differs. And because Vultr assigns the VPC address (no cidrhost pinning), a node recreate may hand it a different plane IP that stored membership won’t follow.

Fix: the module derives enp8s0/enp1s0 automatically (override via var.nic_name). Treat node replacement as remove + rejoin, not in-place recreate.

Backups to Vultr Object Storage are not wired by the module: a known gap. Vultr Object Storage is S3-compatible, so restic can target s3:<region>.vultrobjects.com/<bucket>/shc with backup.aws_access_key_id / backup.aws_secret_access_key set to a Vultr Object Storage key pair. But those keys are minted per-object-store in the Vultr console, not through the Terraform provider, and the same keys are needed just to create the bucket, so there is nothing to wire from Terraform. Provision the bucket + key by hand and set the backup.* config if you need off-cluster backups.

Vultr is the substrate that named a whole class of bug: tofu destroy removes every instance, exits, and leaves the network tier (VPC, firewall groups, firewall rules, SSH key) standing in the cloud and described in state. It happened twice on the same estate (promanager ewr, 2026-07-14 and 2026-07-19) and both times the teardown was recorded as successful.

DELETE /v2/instances/<id> returns while the instance is still in destroying, and the provider’s instance delete does not wait that out; it detaches the VPC, issues the delete, and returns. tofu destroy walks dependents-first, so the network tier destroys last, which means its deletes fire the instant the instance delete returns: straight into the purge window, against objects Vultr still considers attached. vultr_firewall_group’s delete carries no retry at all and vultr_vpc’s retry engages on one exact API message, so those deletes simply fail.

Both halves are now fixed in terraform/modules/shc/vultr:

  • time_sleep.instance_purge sits between the two tiers. Its depends_on names the whole network tier and vultr_instance.node depends on it, so on destroy the order is instances -> 120s settle -> reserved-IP / firewall / VPC / SSH-key deletes. Create-side cost is zero (only destroy_duration is set).
  • The settle’s membership rule is enforced, not remembered. Every resource the instance block references must be behind the settle: a referenced resource is one the instance is attached to, so its delete is the delete the half-purged instance blocks. TestCloudModuleSettleCoversEveryAttachedResource (tests/unit/ci/cloud_module_teardown_settle_test.go) derives that set from the .tf source for every cloud module and fails if anything is missing. It was written because the list had already fallen behind twice: vultr shipped the settle with the SSH keys omitted (promanager-vultr-key was found stranded beside the VPC the settle had been written for), and scaleway shipped it with scaleway_instance_ip.node (the server’s own ip_id) omitted.

The lane shells used to treat the destroy’s exit code as the only evidence, so a destroy that deleted the instances and stranded the tier reported CLEAN. teardown_verdict now reads state back (teardown_state_residue in tests/lib/tf-harness.sh): an address only leaves state when its Delete succeeded, so anything still listed after a “successful” destroy is something the destroy did not delete, and the lane exits INCOMPLETE.

That is still only a state check. State and cloud can disagree in both directions (drop_platform_from_state divorces them deliberately), so for a real teardown verdict the cloud has to be read through the provider API:

Terminal window
tests/clouds/lib/reap-run.sh --cloud vultr --prefix <cluster-name>

tests/clouds/hostinger/scripts/destroy.sh has the shape to copy: it queries the provider API after the destroy and fails the lane on what it finds, rather than on what the destroy claimed.

The ewr residue, and how to clear it by hand

Section titled “The ewr residue, and how to clear it by hand”

Two tofu states in promanager-ops still describe an estate that no longer exists. Cloud pruning is deferred by owner ruling, so this is the sequence, not a thing that has been run:

stateserialwhat it still describes
infra.ewr.vultr.promanager.vip1413 resource blocks / 31 instances: 1 vultr_vpc, 2 vultr_firewall_group, 24 vultr_firewall_rule (4 ssh + 4 icmp + 16 public), 1 vultr_ssh_key, plus tls_private_key.cluster, random_password.vault and local_sensitive_file.cluster_key. No vultr_instance block at all.
deployments.ewr.vultr.promanager.vip43 shc_deployment (gitea, postgres, whoami), 1 shc_user, 1 shc_repo, 1 random_password, all against a cluster with no nodes

Two things to understand before touching either.

The state carries live secrets. tls_private_key.cluster, random_password.vault and local_sensitive_file.cluster_key are stored in plaintext in the state JSON. The nodes they unlock are gone, but vultr_ssh_key.this still holds the matching public half in the Vultr account, so the pair is not inert until the key is gone from the account too. Treat the state as a credential, and revoke the Vultr API key used to create the estate when you are done.

The 2026-07-31 sweep already deleted 8 Vultr objects: two VPCs, four firewall groups and two SSH keys, covering both generations. So the cloud is expected to be empty and the state is expected to be a stale description of nothing. Expected is not verified: nothing on the dev box holds a Vultr API credential, so this has not been checked against the Vultr API.

The sequence:

  1. Read the cloud first, through the API, not through tofu. With a Vultr API key that can list (Account -> API, and mind the IP allow-list; see First-live gotchas #4):

    Terminal window
    for p in instances vpcs firewalls ssh-keys reserved-ips; do
    printf '== %s\n' "$p"
    curl -sS -H "Authorization: Bearer $VULTR_API_KEY" \
    "https://api.vultr.com/v2/$p?per_page=500" | jq -r '..|.label?//.description?//.name?|strings'
    done

    Anything matching the estate’s cluster_name is real residue. Nothing matching means the state is describing objects that are already gone.

  2. If the cloud is empty, do not run tofu destroy to “clean up” the state. A destroy against absent objects depends on every provider delete being 404-tolerant, and the one thing this module has proven is that its deletes are not uniformly forgiving. terraform state rm the addresses, or delete the state outright; the estate root is already removed from promanager-ops, so there is nothing left to plan against:

    Terminal window
    # GitLab-backed http backend; the state name is the dotted form above
    curl -sS --request DELETE \
    --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
    "https://gitlab.com/api/v4/projects/<id>/terraform/state/infra.ewr.vultr.promanager.vip"

    Delete deployments.ewr.vultr.promanager.vip the same way. Do the deployments state first: it holds a data.terraform_remote_state.infra pointing at the other one.

  3. If the cloud is not empty, delete the objects in dependents-first order (firewall rules, then firewall groups, then the VPC, then the SSH key and any reserved IPs), re-reading the API after each step. A DELETE returning 204 is not evidence either; absence in a follow-up GET is.

  4. Verify by absence, then remove the state. Re-run step 1 and confirm it returns nothing before deleting the state, or you lose the only record of what is left.

The general rule this section exists to state: a teardown is verified by reading the provider, not by reading an exit code. Every deletion in the 2026-07-31 sweep was confirmed by absence in a follow-up read, and that is why its ledger can be trusted.