Skip to content
SHC Docs

Google Cloud

SHC runs on GCE as a private-plane cluster: a custom-mode VPC network + one subnetwork in a single zone, N Compute Engine instances, and tag-targeted firewall rules, all folded behind one module "shc" front door (terraform/modules/shc/gcp, or the published shc-gcp-<ver>.tgz tarball). One tofu apply builds the substrate (Layer 1), installs the shc deb and runs init/join over the provider’s SSH transport (Layer 2), and, if you wire the platform block, lights DNS + Let’s Encrypt off a single connection.

The live reference estate is a gcp/us-central1 root kept in a private ops repo (a real, running cluster). This page documents what that estate actually does, plus the runnable terraform/examples/gcp-shc example and the tests/clouds/gcp smoke-test lane.

GCE is the one substrate with no cloud-init: provisioning is a startup-script shell translation of the cloud-init recipe. Most of the first-live findings below trace back to that fact.


There are two distinct credential sets, and they must not be confused.

The google provider reads these natively from the environment, never from a tfvars file or state:

Env varMeaning
GOOGLE_APPLICATION_CREDENTIALSPath to a service-account key JSON. Typical for local runs.
GOOGLE_CREDENTIALSThe key JSON contents (or a path). Typical for CI variables.
GOOGLE_PROJECT / CLOUDSDK_CORE_PROJECTDefault project, if you don’t pass project explicitly.

The admin identity needs enough IAM to create instances, networks, firewalls, a Cloud DNS zone (if you use the native-DNS path), and a scoped service account: roles/compute.admin + roles/dns.admin + roles/iam.serviceAccountAdmin covers it, or roles/owner on a throwaway project.

The SHC daemon does not reuse your admin key. The estate mints a dedicated, least-privilege service account as a terraform resource and hands only its key to the gcp connection:

resource "google_service_account" "shc_daemon" {
account_id = "shc-daemon-gcp"
display_name = "SHC daemon (scoped platform credentials)"
}
# One project role per provider slot the daemon uses:
# roles/dns.admin -> Cloud DNS provider (zone list + record writes)
# roles/compute.loadBalancerAdmin -> gcp-tcp-lb proxy provider (target pools, forwarding rules)
# roles/compute.instanceAdmin.v1 -> gcepd volume provider (disk create/attach/detach)
resource "google_service_account_key" "daemon" { service_account_id = google_service_account.shc_daemon.name }

The connection then carries base64decode(google_service_account_key.daemon.private_key) as its credentials_json; see §2. GCP IAM grants take ~a minute to propagate; if the connection’s connect-time prober fails on the first apply, just re-apply.

ValueHowNotes
Admin GCP credsGOOGLE_APPLICATION_CREDENTIALS / GOOGLE_CREDENTIALS envnever a TF_VAR
ProjectTF_VAR_project or GOOGLE_PROJECT
Cloudflare DNS token (cloudflare-mix path)TF_VAR_cloudflare_tokenZone:DNS:Edit on your test domain
Vault passwordminted in-root (random_password.vault)never an input
Cluster SSH keyminted in-root (tls_private_key.cluster)read back via tofu output -raw ssh_private_key

Connection type: gcp (modules/connections/capabilities.go).

FieldRequired?Consumed by
projectyesall three providers
credentials_jsonyesall three providers (the SA key JSON, inline)
regionride-if-suppliedgcp-tcp-lb proxy (regional target pools / forwarding rules)
zoneride-if-suppliedgcepd volume (required to provision: disks are zonal; X400344GPDCFG without it)
disk_typeoptionalgcepd volume (defaults pd-balanced)

One gcp connection lights three provider slots:

SlotImplNotes
dnsgcp (Cloud DNS)project-wide zone list + record writes
proxy / LBgcp-tcp-lbpassthrough TCP network load balancer; fixed node port (cannot remap ports)
volumegcepdGCE persistent disk; zone-scoped, mirrors awsebs

Create it at system scope (provider ≥ 0.0.322 does this automatically), so the tenant-less cluster-host DNS surface resolves it directly and default-tenant apps ride it with no per-tenant duplicate:

resource "shc_connection" "gcp" {
name = "gcp"
type = "gcp"
credentials = {
project = local.project
credentials_json = base64decode(google_service_account_key.daemon.private_key)
region = module.shc.region
zone = module.shc.zone
}
zones = ["gcp.example.com"] # connect-time prober validates zone visibility
cluster = { ssh = local.app_ssh }
}

DNS approach: two shapes, pick by whether you have free zone hosting

Section titled “DNS approach: two shapes, pick by whether you have free zone hosting”

GCP is one of the few substrates that bundle managed-zone hosting for free (Cloud DNS), so it supports the native path. Both are valid; the right one depends on your domain.

  • Native Cloud DNS, NS-delegated: what the live reference estate uses. You stand up a google_dns_managed_zone (gcp.example.com), delegate it out of a parent zone with four NS records pointing at the Cloud DNS name servers, and point dns.provider at the gcp connection. The daemon’s scoped SA writes records straight into that zone. Why: you get a real, isolated subdomain zone at no extra cost, and the daemon self-registers the cluster host into it. Best for a durable estate under a subdomain you control.

  • Cloudflare-parent-mix: the default for the example + test lane. A single cloudflare connection (type = "cloudflare", cred token) writes records straight into your one Cloudflare-hosted zone. No Cloud DNS zone, no NS delegation. Why: simplest path for a tester who has exactly one Cloudflare domain and doesn’t want to provision or delegate a Cloud DNS zone. This is the convergent “cloudflare-mix” estate shape the multi-cloud campaign settled on for substrates without free bundled zone hosting; GCP supports it too.

NS delegation is a Cloud DNS fact worth repeating: Cloud DNS returns name servers with a trailing dot (ns-cloud-a1.googledomains.com.). trimsuffix(...) it before writing the parent-zone NS records: Cloudflare (and most parents) want the bare name.


Using terraform/examples/gcp-shc (cloudflare-mix DNS, one public demo app):

Terminal window
cd terraform/examples/gcp-shc
# admin GCP creds + the shc provider mirror
export GOOGLE_APPLICATION_CREDENTIALS=~/keys/gcp-admin.json
export TF_CLI_CONFIG_FILE="$PWD/registry.tfrc" # teaches tofu the shc provider mirror
# inputs (never in tfvars)
export TF_VAR_project=my-gcp-project
export TF_VAR_dns_zone=example.com # your Cloudflare-hosted domain
export TF_VAR_cloudflare_token=... # Zone:DNS:Edit on example.com
export TF_VAR_acme_email=ops@example.com
tofu init
tofu apply
# verify
tofu output -raw ssh_private_key > /tmp/cluster.key && chmod 600 /tmp/cluster.key
ssh -i /tmp/cluster.key admin@$(tofu output -json node_management_ipv4s | jq -r '.[0]') \
'sudo shc cluster status; sudo shc connection ls'
dig +short git.example.com
openssl s_client -connect git.example.com:443 -servername git.example.com </dev/null 2>/dev/null \
| openssl x509 -noout -issuer # expect: issuer=...Let's Encrypt...

The example pins module "shc" to the shc-gcp-0.3.0.tgz Pages tarball (the SSH user is admin on that release), exactly like the estate. The tests/clouds/gcp lane instead builds the in-repo module + provider from the working tree (SSH user admin too, module pins >= the admin-user rename); see the test README.


Every item below was observed on a real GCP boot during the cross-cloud campaign. Each is a one-layer fix, already landed.

a. GCE has no cloud-init → startup-script + apt/dpkg lock hygiene

Section titled “a. GCE has no cloud-init → startup-script + apt/dpkg lock hygiene”

Symptom. google-startup-scripts.service runs late in boot, after apt-daily/unattended-upgrades has already grabbed the dpkg lock. The startup-script died with apt exit 100: Could not get lock /var/lib/dpkg/lock-frontend, held by apt-get, leaving docker installed but the compose plugin + git/gnupg/restic never installed, so the daemon’s docker compose -p … then failed with unknown shorthand flag: 'p' (compose v1 vs v2).

Fix. GCE Debian images run google-guest-agent, not cloud-init, so the recipe is a shell translation (modules/shc/gcp/templates/startup-script.sh.tftpl). Before any apt-get it stops and masks unattended-upgrades + the apt-daily[-upgrade] services/timers (a disable alone leaves the in-flight run holding the lock), and passes -o DPkg::Lock::Timeout=300 on every apt-get so a lock it didn’t win is waited for (up to 5 min), not failed on. set -e guarantees the /var/lib/shc-node-prepared marker is touched only after docker compose version succeeds, so a half-provisioned node never looks ready.

Symptom. The connection registry declares the cred field project, but the DNS factory’s buildGCP used to read project_id, a name mismatch that was papered over by a dns-local connCredAliases rename table (audit over-fit #7). A gcp connection carrying only project produced an empty project on the DNS path.

Fix (562a81811). buildGCP now reads project first, falling back to project_id only for older inline configs. The estate connection drops the project_id duplicate entirely. If you see empty-project zone misses on an old cluster, re-apply shc_connection to rewrite it at the current shape.

c. DNS-01 cert issuance: SHC impl name ≠ lego provider code

Section titled “c. DNS-01 cert issuance: SHC impl name ≠ lego provider code”

Symptom. Traefik solves the ACME DNS-01 challenge with lego, whose Google provider code is gcloud, not gcp. Rendering the SHC impl name (gcp) verbatim into dnsChallenge.provider made lego reject it and cert issuance silently fail (audit over-fit #2). Separately, the pre-0.3.0 estate had to SSH-upload the SA key file into the traefik ACME mount and set a credentials_file, which is brittle.

Fix (0fe8033c2 + 562a81811). A LegoProvider() method maps gcp → gcloud (also powerdns → pdns, azure → azuredns), so the resolver renders the code lego expects. And the gcp DNS provider’s TraefikEnv() now forwards credentials_json directly as lego’s GCE_SERVICE_ACCOUNT (inline JSON, no file); the terraform_data.traefik_dns01_sa_key file-upload workaround is dead code. Note: this only bites the native gcp-DNS path; the cloudflare-mix path uses lego code cloudflare (identity), so it’s unaffected.

d. gcp.<domain> NS delegation via googledomains name servers

Section titled “d. gcp.<domain> NS delegation via googledomains name servers”

Symptom / behaviour. In the native-DNS estate, gcp.example.com is a real Cloud DNS zone delegated out of the Cloudflare-hosted parent. Cloud DNS assigns exactly four name servers under *.googledomains.com, and returns them with a trailing dot.

Fix / recipe. Create four NS records in the parent (use count = 4, not for_each: the name-server list is computed and unknown at plan time), and trimsuffix(..., ".") each name server before writing it. The daemon then self-registers the cluster host into the delegated zone, not into Cloudflare.

The whole cluster is pinned to one zone on purpose: gcepd provisions zonal persistent disks and refuses without a zone (X400344GPDCFG). Multi-zone node groups would strand volumes. The gcp connection’s zone cred is the placement pin (the EBS availability_zone analogue). Default regional static-IP quota is 8/region: a 4-node cluster with assign_static_ips uses 4.

f. metadata_startup_script is create-time-only → key changes replace nodes

Section titled “f. metadata_startup_script is create-time-only → key changes replace nodes”

Unlike EC2 user-data (first-boot-only but adoptable), GCE’s metadata_startup_script is baked at create time. Editing ssh_public_key, extra_authorized_keys, or flipping create_ssh_key on a live cluster replaces every node. To authorize a new key without a rebuild, append it by hand over an existing key (see the module README’s rotation recipe), and mirror it in terraform only alongside a planned roll.

The shc deb is amd64. Use x86 machine families (e2/n2/c3/…), never Arm (t2a/c4a). image_family defaults to debian-13; fall back to debian-12 if debian-cloud hasn’t published 13 in your project. The plane NIC is ens4 (virtio-net); verify with ip a and override nic_name if a gVNIC-only family differs. Both defaults are now live-verified: a debian-13 lookup in debian-cloud resolves (Debian 13 trixie, 6.12.101+deb13-cloud-amd64) and the plane NIC comes up ens4.


4b. First live execution of the volume + LB providers (2026-08-14)

Section titled “4b. First live execution of the volume + LB providers (2026-08-14)”

Section 4’s gotchas came from the native-DNS reference estate, which exercises the Cloud DNS path. The gcepd volume backend and the gcp-tcp-lb proxy backend had never been executed against GCP at all: both packages shipped with an explicit UNVERIFIED-LIVE note. They have now been run against a real project (one e2-micro node from terraform/modules/shc/gcp, manage_platform = false), by tests/live/gcp.

gcepd passes. Provision (incl. the find-before-create adopt path), Attach (and its idempotent re-attach), the on-node /dev/disk/by-id/google-<handle> symlink, VolumeLocation, Detach and Delete (both absent-tolerant on a second call) all behave as documented.

gcp-tcp-lb does not work as shipped. Four defects, each reproduced with a literal API refusal:

h. gcp-tcp-lb forwarding rules are built with ports: GCP refuses

Section titled “h. gcp-tcp-lb forwarding rules are built with ports: GCP refuses”

reconcileForwardingRule inserts a ForwardingRule with Ports: [...] and Target: <targetPool>. GCP accepts ports only for backend-service or internal targets; a classic target-pool passthrough NLB takes a single portRange:

Error 400: Invalid value for field 'resource.ports[0]': '5432'.
The ports field must only be used with backend services or internal target instances.

The same insert with portRange: "5432" returns 200. Consequence: the backend can never provision a rule, so no gcp-tcp-lb deployment has ever had a VIP. Note the shape change is not a one-line swap: a target-pool rule carries one contiguous range, so a forward set of discrete ports needs one rule per port (and equalPortSets must then compare PortRange, not Ports, or drift detection never converges).

i. gcp-tcp-lb pool members are sent as node IPs, not instance URLs

Section titled “i. gcp-tcp-lb pool members are sent as node IPs, not instance URLs”

PoolFromLiveNodes emits NodeTarget{Host: <internal IP>}; reconcileMembers passes that straight into TargetPools.addInstance:

Error 400: Invalid value for field 'instances[0].instance': '10.10.1.11'.
The URL is malformed.

Every sibling backend (vultrlb, dolb, hcloudlb) resolves the pool host to a provider instance id first; this one does not.

j. gcp-tcp-lb member diff empties the pool on the SECOND reconcile

Section titled “j. gcp-tcp-lb member diff empties the pool on the SECOND reconcile”

Even when the desired host IS a valid instance reference, reconcileMembers compares the desired strings against TargetPool.Instances, which GCP returns as full https://… selfLinks. A partial URL never equals a selfLink, so an unchanged pool computes as “add the desired, remove the current”: the same instance is added and then removed inside one reconcile. Measured on two identical back-to-back reconciles:

AFTER-RECONCILE-1: pool members n=1 [https://www.googleapis.com/compute/v1/projects/…/instances/app01-…]
AFTER-RECONCILE-2: pool members n=0 []

An idempotent no-op tick silently strips every backend off the LB.

k. ensureTargetPool does not wait for its own insert

Section titled “k. ensureTargetPool does not wait for its own insert”

serviceClient deliberately drops the returned *Operation, but reconcileMembers runs immediately after ensureTargetPool in the SAME Reconcile, against a pool that is still being created:

Error 400: The resource 'projects/…/regions/us-central1/targetPools/shc-<tenant>-<stack>'
is not ready, resourceNotReady

The first reconcile of any new deployment therefore always fails; only a later tick can converge. (Contrast gcepd, which waits every zonal operation to DONE.)

proxyapi teardown is spelled “reconcile with an empty forward set”, which deletes the forwarding rules. The target pool has no delete path in the backend at all, so it outlives the deployment as regional residue; it had to be swept by hand at the end of the live run.

m. gcepd resolves a node by GCE instance name, not the SHC node name

Section titled “m. gcepd resolves a node by GCE instance name, not the SHC node name”

gcepd.Attach does a direct instances.get with the node string the daemon hands it, documented as “GCE instance names == SHC node names by the substrate-module contract”. The substrate module does not honour that: it keeps the dotted FQDN as the SHC node identity and hands GCE the sanitized form (local.safe_name, dots → dashes), because GCE names are RFC1035. Any cluster whose cluster_name is a dotted FQDN (the documented recommended form) fails every attach:

Attach(node="app01.shctest-r4-gcp") ->
[X502344GPDAPI] … get instance failed googleapi: Error 400:
Invalid value for field 'instance': 'app01.shctest-r4-gcp'.
Must be a match of regex 'a-z?|[1-9][0-9]{0,19}'

Attach(node="app01-shctest-r4-gcp") (the sanitized name) succeeds. Either the provider must sanitize the same way the module does, or the module must stop diverging: a design call, not a patch.

The module’s own teardown settle (time_sleep.instance_purge, 120 s between the instance delete and the network tier) works: tofu destroy removed all 8 resources with no resourceInUseByAnotherResource, and a full re-enumerate of the project returned to the exact pre-run baseline: zero leaks. The gcp Cloud DNS backend passes end to end, including a dig against the zone’s own *.googledomains.com nameserver.


ConcernNameNotes
Connection typegcpCredFields project, credentials_json (+ ride-along region/zone/disk_type)
DNSgcp (Cloud DNS)lego code gcloud; live-verified 2026-08-14
VolumegcepdGCE persistent disk, zone-scoped, mirrors awsebs; live-verified 2026-08-14, except the node-name resolution of §4b.m
LB / proxygcp-tcp-lbpassthrough TCP NLB, fixed node port; NON-FUNCTIONAL live; see §4b.h–l
DNS (test/example default)cloudflaresingle-domain cloudflare-mix, cred token

See also: the module README (GCP-vs-aws/hetzner delta table + first-boot verify checklist) and the live reference estate in your ops repo.