Skip to content
SHC Docs

Moving bitspur to the new line (design)

Status: open design (SHC-1061). Nothing here is built and nothing has moved. It answers what the simplification design §6 left to SHC-1061: the order of apps, how the shared databases and mailcow move, and what happens to the rig VMs on the Gorilla boxes. It keeps §6’s mechanics (the archive is what crosses, the manifest rewrite, the write barrier, takeover then reconcile for DNS) and changes its rollback (section 5.8). No app moves before the cutover gate, scheduler §7 (SHC-1165), passes on the rig.

The shape, ruled by Clay on 2026-10-01:

  1. Fresh nodes. compose01 is retired, never upgraded in place.
  2. App by app, with both estates live. A deployment is stopped on the old estate, brought up on the new one, verified, and its DNS switched.
  3. Each app’s data arrives by restoring its restic backup, taken fresh right before its move. Volumes are never copied off compose01. The move proves the backups.
  4. The new nodes are VMs on both Gorilla boxes under Proxmox, docker runtime. Bare Incus on those hosts comes after the cutover; so does the two-tunnel mesh (SHC-1201). The move rides today’s Nebula plus fabric.
  5. compose01 stays untouched for two weeks after the last app moves. Switching back is a DNS change plus starting the old deployment.

The migration is a one-off script outside the product. promanager is frozen and not touched.

Evidence: a read-only inventory of bitspur taken 2026-10-01 19:19Z to 19:58Z, attached to SHC-1061 (inventory.md, inventory.json, the raw listings). Every number below comes from it unless it names another source. A Codex judge pass (high effort) read this design against that evidence and blocked the first draft on 18 findings; every one is answered below.


deployments68 (corp 61, master 4, shc 3), plus 5 plugins with no data of their own
data in restic88.0 GiB in the newest scheduled snapshots, every one dated 2026-09-30 or 2026-10-01
data on compose01’s disk84.2 GiB under /var/lib/shc/deployments; swarm01 and swarm02 refuse our key, so GitLab and ollama (swarm01) and penpot and pokemon-kc (swarm02) are measured through their backups only
data outside restic26 Hetzner hel1 buckets bound by 9 deployments (Nextcloud’s primary store was 41,030 objects, 57.1 GiB on 09-30; GitLab’s registry, artifacts, LFS and packages; peertube, pulp, huly, penpot, openreplay, affine, supabase)
memorycompose01’s containers hold 9.4 GiB resident and 29.3 GiB in swap; with GitLab, ollama and penpot the estate’s working set is about 50 GB
sources47 from the catalog, 4 deb-shipped builtins in master, 16 path-pinned (Nextcloud, nc-db, wazuh, keygen, keygen-db, license-broker, ivatar, onlyoffice, immich, postiz, marketplace, ses, openhands, gitlab-huly, mpo-storybook, pokemon-kc) and 1 git-pinned to the shc repo on git.corp (pokemon-web), each carried as a local package
DNS70 A records through the cloudflare connection, all 2.29.3.100, TTL 300; 66 belong to a deployment, 4 do not (auth.corp.bitspur.com, shc.hel1.hetzner.bitspur.com, traefik.corp.bitspur.com, and docs.corp.bitspur.com, which is not desired); MX of bitspur.com and flyingfoxlabs.in is 10 mail.bitspur.com and is not an SHC record
vaultabout 5,600 entries across 74 deployment scopes (09-30 count), copied key by key, never re-minted

Not covered by any backup, or not proven to be:

  • The 26 buckets. corp/hz-s3 has backup.buckets empty on purpose (arming it once filled compose01’s disk). They are bound again, not copied, and nothing protects them against deletion or a provider loss, before or after the move. GitLab’s registry is one of them (SHC-812).
  • LibreChat’s conversations. corp/librechat-db holds 456 MiB of MongoDB on disk and its backups are under 1 MiB, and the librechat template says its own cross-boundary mongodump was removed. Until a rehearsal restore shows LibreChat’s users and known conversations, treat them as not backed up; librechat does not move before that.
  • corp/mysql backs up 17 MiB against 2.6 GiB on disk. Its four consumers’ databases are in that dump or nowhere; the rehearsal restore has to show each one’s tables and rows. (sonarqube’s 704 MiB is its Elasticsearch index, which rebuilds; its database is in corp/postgres.)
  • 57 anonymous Docker volumes attached to running containers. None holds data that matters: Frappe sites/assets (rebuilt), pulp’s /var/lib/containers (a cache), image-declared VOLUMEs of client containers whose real data sits in a bind mount (checked for every postgres, mysql and mongo container), jitsi and coturn config.
  • shc/ffx-frappe:based93, the image ffx runs, exists only in compose01’s Docker store with no registry digest. A fresh node cannot pull it, so it is pushed to the corp registry and pinned by digest before wave 0’s rehearsal (SHC-691).
  • Control-plane state (tenants, config, connections, grants, schedules, retention, DNS ownership, the licence) is not app data. The script recreates it on the new cluster from the old API, per §6.

Not moved, by recommendation (open question 7): corp/nc-db has no consumer left since Nextcloud moved to the shared postgres, and ffx’s mount bench.pre-based93 (2.7 GiB) is a stale copy beside the live bench. Both stay on compose01 and go with it. docs.corp.bitspur.com is already not desired and is not moved. traefik.corp.bitspur.com is the old Traefik’s dashboard (dashboardHost); the new line’s per-node Traefiks have no shared dashboard name, so it is retired with compose01 by decision, not because the record is stale.


nodehostVMIDvCPURAMdiskaddressrole
shc01.odg.bitspur.compve021101628 GiB + 16 GiB swap500 GB146.71.124.78/29voter, workloads, mail
shc02.odg.bitspur.compve031201628 GiB + 16 GiB swap500 GB146.71.124.68/29voter, workloads
shc03.odg.bitspur.compve0313024 GiB40 GB146.71.124.69/29voter only

One workload VM per box. A box is the failure domain, so a workload node per box keeps “one replica per node” honest: two workload VMs on one box would let the engine believe two replicas are apart when one power cut takes both. It also fits the addresses: pve02 has only .78 left while the rig holds .75 to .77, pve03 has .68 to .70 (all four measured silent on 10-01).

Sizes. The estate’s working set is about 50 GB; two workload nodes at 28 GiB plus swap carry it with room, and both boxes keep the rig running beside them (pve02 44 of 60 GB with the rig’s three 4 GB nodes, pve03 50 of 60 GB with shc03, far01 and the SHC-1199 and SHC-1202 proof VMs). After the rig goes, each workload node grows to 44 GiB, one at a time with a drain, which is also what lets one box carry the whole estate while the other is reinstalled as a bare Incus host. Disk: 84 GiB of deployment data plus images and restore staging fits 500 GB on each, with 370 GB of each box left free.

Three voters on two boxes, and what that does not buy. With two nodes the control plane has one voter, and losing it stops every lease-held app on the survivor (simplification §4.9). So shc03 is a small voter-only VM in a reserved fence no app tolerates: it runs rqlite, shcd and the per-node platform set and nothing else. The ruling puts the new nodes on pve02 and pve03, so shc03 sits on pve03 beside shc02. That tolerates losing pve02 or any one VM, and does not tolerate losing pve03: two of three voters go with it, lease-held apps on shc01 stop renewing, and recovery is repair --force on shc01 or pve03 coming back. Before pve03 is ever taken down on purpose (the Incus reinstall), the voter-only role moves first: a voter-only VM joins on pve02, then shc03 is removed while all three voters are up, so pve02 holds two of the three voters when pve03 goes down. Adding a fourth node alone would not do it: four members still give three voters, and the engine may leave two of them on pve03. A voter-only VM on pve01, the third GorillaServers box in the same datacenter (0.5 ms from pve02, measured), would make any single box survivable; that needs Clay to widen the ruling (open question 1).

Fences.

fencekindmembers
control-planecontrol-planeshc01, shc02, shc03
gorillaservers/ogdenlocationshc01, shc02, shc03
gorillaserversprovidershc01, shc02, shc03
voter-onlyreservedshc03

One location, because both boxes sit in one datacenter. With one workload node per box, “one per node” already spreads across boxes; naming a box a location would mislead every later estate.

Cluster identity. The new cluster is a new generation of the same cluster: slug shc-hel1-hetzner-bitspur-com, DNS heritage shc/owner=shc-hel1-hetzner-bitspur-com-a995f257. The takeover only adopts a sibling generation’s claim, and the licence is bound to shc.hel1.hetzner.bitspur.com and auth.corp.bitspur.com (installed on the new cluster by file, its fallback carrier). During the move the old console keeps shc.hel1.hetzner.bitspur.com; the new cluster is driven by the Mac CLI at shc.odg.bitspur.com, a new record the new cluster owns, and the console name flips in the last wave. The hel1 names are renamed after the move, with a re-minted licence (open question 5).

The SHC-1123 rig (VMs 200 to 203). It stays through the move. The cutover gate is proven there, and each wave is rehearsed there first against its own small old-line cluster. It uses at most half of each box and does not collide with production (VMIDs 200 to 209, addresses .75 to .77 and .67). After the last wave, and after SHC-1201’s mesh rehearsal, which needs a standing two-site cluster, the rig is torn down (lock released, module block deleted) and the production nodes grow into the room.


The inventory holds 119 plug edges. What decides the order is whether a consumer reaches its provider by a public hostname or privately (a socket field such as internalUrl, hostname, apiUrl, a database socket, or a container network alias). A private edge cannot cross the estates, and in most templates the socket value wins over any var, so it cannot be pointed at a public hostname either.

edgehow it is reachedconsequence
29 oidc, 3 saml to master/keycloakthe public issuer https://auth.corp.bitspur.com, except identity-broker’s API calls (socket.hostname, port; overridable by vars.oidc.apiUrl) and both mailbox reconcilers (http://${KC_HOSTNAME}:${KC_PORT}, no override)29 cross by hostname; identity-broker gets oidc.apiUrl; the reconcilers pause (3.2)
10 smtp, 3 imap to mailcowmail.bitspur.com:587/993, which mailcow’s network aliases onto postfix and dovecotpublic from either estate once the old containers are stopped (the alias goes with them)
3 mailcow admin API, matrix, matrixAppservice, matrixAdmin, 4 nextcloud via internalUrl, affine, fizzy, identityBroker, gitlab, llm, frappe, keygen, goalert, clickhouse, redis, mongodb, every mysql/postgres edgeprivatethe two ends share a move unit
Nextcloud and officeoffice calls Nextcloud back through a private tenant alias (its template says so) and Nextcloud calls officeone unit
3 webhook to hookshothookshot has no hostname; GitLab, glitchtip and wazuh post to it privatelyglitchtip joins hookshot’s unit; GitLab and wazuh lose notifications for a day or two (3.4)
nextcloud from bit, ffxread by a Frappe addon, not the composewhich URL it uses is read on the rig; if private, wave 6 runs the night after wave 4 and DAV sync pauses that day
9 s3 to hz-s3the hel1 endpoint, publicno constraint

A move unit is a connected component of private edges. Its members share one write barrier: every member stops (consumers first), every member is backed up, every backup is restored, providers start before consumers, every member is verified, and only then do their DNS records switch together. A unit is never left half-moved: if anything fails the whole unit rolls back (5.8). Most units are one deployment.

A deployment whose private providers sit in two different units stays stopped from the first of those units’ barriers until the last has moved, then moves with the last. Two need it: gitlab-fizzy (fizzy in wave 4, GitLab in wave 5) and both mailbox reconcilers (Keycloak in wave 1, mailcow in wave 4). While they are stopped, GitLab-to-Fizzy sync stops and new Keycloak users get their mailbox by hand. A paused deployment’s fresh backup is the one taken at its pause, through the running-app path right before it stops; it does not change while stopped, so that verified capture stands in for the scheduled-backup freshness check of section 7, with the script’s journal and the container state as proof it stayed stopped.

The reconcilers also cannot resume as they are: on the new line Keycloak is in another tenant, so their private http://${KC_HOSTNAME}:${KC_PORT} is refused. They move at a new package version that reads a public API base (vars.oidc.apiUrl, the override identity-broker already has), filed as a catalog change before wave 4 and proven in rehearsal. It is the one deliberate version change in the move.

Only by public hostname. There is no tunnel, Nebula link or shared network between the estates. 47 deployments are LAN-only (exposure: lan), Keycloak among them, and a request from one estate to the other’s public address arrives from that estate’s egress, which neither LAN allowlist admits today. So each side’s tenant lanAllowlistCidrs admits the other side’s node addresses for the duration (old: 2.29.3.100, 2.29.2.229, 2.29.10.60; new: 146.71.124.78, 146.71.124.68, 146.71.124.69). It is written on the old cluster before wave 1 and stays through the two weeks, because a rollback needs it.

A wave is a set of units moved one unit at a time in one window, in the order listed. Window estimates use tonight’s capture times, a pull of about 1 GiB a minute from hel1 (17 MB/s single stream measured from pve02; restic runs several streams), and 5 minutes fixed per deployment; the rig rehearsal replaces them with measured numbers. Section 7 lists when no window may run.

waveunits, in order (one unit per brace)why herewindow
0 buildnothing moves: cluster, platform set, connections, tenants, config, vault, repos and local packages, the licence; empty engines corp/postgres, corp/mysql, corp/mariadb at the old majors and image digests; the hz-s3 and ses adapters; every image by digest; every certificate by DNS-01; bucket versioning; every unit’s rehearsal restore in isolation (5.1)no traffic, no DNS changedays, no downtime
1 identity{master/postgres, master/keycloak, the record auth.corp.bitspur.com}; {master/openobserve}the new cluster’s shcd, forward-auth and every restored app need the identity provider it administers (6.3)about 20 min of no logins; openobserve about 75 min of no new logs
2 self-contained{wellknown}, {lookout}, {mpo-storybook}, {openhands}, {crowdsec}, {uptime}, {marketplace}, {workflows-redis, workflows}, {shc/keygen-db, keygen, license-broker}, {pokemon-db, pokemon-kc, pokemon-bench, pokemon-web}, {ollama, librechat-db, librechat} once LibreChat’s data is provenno private edge leaves its unit; small units prove the runbook; license-broker also answers promanager’s six-hourly licence reverify, so its unit’s window stays under 15 minutesabout 3 h over two nights
3 single consumers{ciso}, {docuseal}, {dtrack}, {ivatar}, {openreplay-ch, openreplay}, {peertube}, {penpot}, {photos}, {postiz}, {pulp}, {sonarqube}, {superset}, {supabase}, {blog with its two plugins}, {fleet}, {mautic}each reaches only its databases, Keycloak, mail and S3about 5 h over two or three nights
4 mail and collaborationone unit: {mailcow, mailbox-reconciler, mailbox-reconciler-ffx, identity-broker, matrix, hookshot, glitchtip, nextcloud with its plugins mail and integration_paperless, office, pdf, projects, affine, fizzy, crm}one component of private edges (identity-broker reaches mailcow, matrix and Nextcloud; projects reaches matrix, Nextcloud and identity-broker; office and Nextcloud call each other; Stirling is wired to Nextcloud by its plug)about 3 h, every member down for all of it
5 GitLab{gitlab, gitlab-huly, gitlab-fizzy}GitLab alone is 12.9 GiB plus its database; the bridges reach it privatelyabout 1.5 h, CI paused
6 Frappe and SIEM{bit, ffx with ffx/live}; {wazuh, oncall}bit and ffx share the mariadb engine; wazuh posts alerts to oncall privatelyabout 2 h
7 VPN and console{netbird}; the record shc.hel1.hetzner.bitspur.commoving NetBird re-points the VPN path to LAN-only apps; last, so every LAN app is already where the VPN landsabout 45 min
holdnothing stops on compose01, the old engines included, for two weeks; then the old cluster stops with its disks kept until Clay signs offrollback needs every old provider running

A wave may stop after any unit and resume the next night.


engineconsumers, with every database each one uses
corp/postgres23 consumers, 25 databases: affine (affine_rs), ciso (ciso_assistant), docuseal, dtrack, gitlab (gitlabhq_production, role gitlab), glitchtip, identity-broker (broker), ivatar, matrix (synapse and synapse_mas), netbird, nextcloud, oncall (goalert), openreplay, peertube, penpot, photos (immich), pokemon-kc (postgres, role postgres), postiz (postiz and postiz_temporal), projects (huly), pulp, sonarqube, superset, workflows (activepieces)
corp/mysqlblog, crm (monica), fleet, mautic
corp/mariadbbit, ffx
single-consumer enginesmaster/postgres (keycloak), corp/pokemon-db (pokemon-bench, frappe), corp/librechat-db (librechat, LibreChat), corp/workflows-redis (workflows), shc/keygen-db (keygen), corp/openreplay-ch (openreplay)

Recommendation: stand up each shared engine empty in wave 0 and move its consumers in their own units, each with its own databases. Do not move an engine with all its consumers in one wave.

Moving corp/postgres whole means the engine and all 23 consumers down in one window (the 09-30 dry run put 26 deployments, 92.5 GiB and a 148-minute write barrier in that group): the big bang the ruling rules out, and rolling back any one of them rolls back all of them.

It costs nothing new, because the engine’s backup already holds one dump per database: the postgres addon writes databases/<n>.sql.gz with a .name beside it and globals.sql for roles (ce addons/postgres/backup.yaml), and the mysql and mariadb apps write the same layout with system-mysql.sql.gz for users and grants. For each unit:

  1. Write barrier on the unit (3.2).
  2. shc backup create postgres on the old cluster (5.2 minutes for corp/postgres tonight), once per unit whatever the number of consumers in it. The unit is stopped, so its databases are quiet.
  3. The script pulls that .bak from the restic repository, extracts every database of every consumer in the unit (all of them, from the table above) and their roles from globals.sql, creates the roles and databases on the new engine and replays them, then checks each table count against its dump, the check the addon’s own restore makes. pokemon-kc lives in the maintenance database postgres as the superuser postgres: its dump replays into the existing postgres database with no create, and the role already exists with the password the new engine carries from the copied vault. (A consumer on the superuser is its own follow-up.)
  4. Each consumer’s own backup, also taken in the barrier, is restored into its new deployment, and it starts against the new engine.

The data still arrives from restic, so the ruling holds, and every unit proves more of the engine’s dump. The old engine keeps serving the consumers that have not moved and keeps running through the two weeks, so a rolled-back consumer finds its old database where it left it. Engines match majors on both sides (the restore refuses otherwise) and upgrade after the move. Single-consumer engines move with their consumer, whole.


Every unit, in its wave, through the script (section 7):

  • Rehearsal in isolation. Each unit’s newest backup is restored once into a throwaway tenant rehearsal on the new cluster, with its hosts unrouted, its S3 and SMTP relay credentials blanked, no plug job run against any production provider (Keycloak, mailcow, the buckets), and outbound mail and webhooks dropped. It proves the archive restores and the data is there (row and file counts against the source), not that the app runs against production. The rehearsal tenant is deleted after. This is gate condition 9 for bitspur.
  • Images on the new nodes by digest; certificates for the unit’s hostnames issued by DNS-01; local packages uploaded; vault keys copied.
  • Images pulled ahead of their wave are unused images. On a docker node an install counts the bytes unused images hold as room and removes the least recently used of them before it pulls (SHC-2621), and image_prune_cron removes any older than a day. An image is safe only while a container, running or stopped, is on it, a deployment on the node names it or keeps it in its rollback reserve, or the node cannot pull it back. So pull a unit’s images in that unit’s own prepare, or install the unit stopped first. Never clear images by hand (docker image prune -a, docker rmi). shc/ffx-frappe:based93 is pushed and pinned by digest before it reaches a new node (section 1); a copy that cannot be pulled back is kept.
  • Bucket versioning on every bucket the unit binds, proven by deleting and recovering a test object, so a rollback can return the bucket to its state at the barrier. The buckets are in no backup, so a bucket that cannot be versioned makes its unit forward-only from 5.4: no rollback once the new copy has started, and Clay is told before that unit’s window.
  • The new deployment is installed and left stopped, at the same package and image digests, vars, overrides, --host and --port mappings, and the same plugs except SSO (section 8). Plug jobs that would change something a running old copy also uses (a Keycloak client’s redirect URIs, a mailbox, a bucket policy) are deferred to after the barrier; the copied vault means the rest find their credentials and change nothing.

Every record is already TTL 300 (the DNS module’s default) and the old cluster’s DNS reconcile would rewrite a hand-lowered TTL, so it stays 300. Clients with a cached address reach the old ingress for up to five minutes after the switch and get 503 for that host.

Cut the old copies’ ingress, drain (for office, open documents force-save first), quiesce background writers the dry run lists, take each member’s final backup through the old line’s running-app path plus the engine backup of section 4, verify them, stop the old copies.

Into the new deployments (manifest rewrite, shc backup import, restore, per §6), then run the deferred plug jobs and start providers before consumers. This is the first write on the new side; from here a rollback is no longer free (5.8).

Through the new cluster, before DNS moves, from test clients that resolve the unit’s hostnames to the new node (curl --resolve, a browser profile with host-resolver rules, extra_hosts on a test runner), and confirm in the new Traefik’s access log that the new side answered. Apps on the new side that call a same-unit public name (Nextcloud to office, any app to mail.bitspur.com in wave 4, anything to auth.corp.bitspur.com in wave 1) would still resolve the old address, so for 5.5 the script pins the unit’s hostnames to the new node for the unit’s own members only and drops the pin at 5.6. Every other deployment, moved or not, keeps resolving public DNS and reaches nothing of the unit until the switch. Until then the new ingress admits the unit’s hosts only from the test clients’ addresses and from the unit’s own members, whose pin points at the node’s ingress over the fabric, so ingress sees their deployment’s fabric sources, not a node address every deployment shares. No production request reaches a new copy before 5.6:

classcheck
web appits health path; a login through SSO completes, callbacks included
database consumerper-database table counts equal the dumps’; the app reads a record created before the barrier
object-store boundlists a known object and round-trips a test file
identitydiscovery document and JWKS; a token for a test client; a gated app’s login
mailbanners on 25, 465, 587 and 993 with the mail.bitspur.com certificate; IMAP login; a message sent straight to the new node’s 25 is delivered; one sent out leaves through SES (its Received chain); DKIM verifies at the receiver
gitgit ls-remote over HTTPS and over SSH on 2222, both with the address pinned; docker pull of a known tag; one CI job after the switch, in the watch
Frappebench --site <site> doctor; a login; a document opens with its CSS
VPNa peer connects; LAN-only apps answer through the route; split DNS answers the new addresses

The new cluster takes over every record of the unit’s deployments, plus any platform record the unit carries (auth.corp.bitspur.com with Keycloak, shc.hel1.hetzner.bitspur.com in wave 7), and its DNS reconcile points them at the new node.

24 hours: errors in OpenObserve, the Uptime Kuma monitor, the logs. That night the new cluster takes each member’s first new-line backup and verifies it; the unit is not done until those backups exist.

Always the whole unit, never one member, and back to the state each member had before the unit’s barrier: a deployment that was already paused (3.2) stays paused. Every old provider a rolled-back unit needs is still running, because nothing on compose01 stops during the move or the two weeks except the units being moved and the paused ones. Every rollback also restores the split-DNS answers for the unit’s own hosts (6.8), from the values the script journalled, and checks a VPN client resolves and reaches the old copy. The NetBird routes to the new nodes are shared by every moved unit, so a rollback never removes them; they go only when no moved unit is left on a node.

  1. Before 5.4 (no write on the new side): start the old copies, restore their ingress. Nothing public changed. Free.
  2. After 5.4, before 5.6, for a unit with no background work that acts on the outside world: stop the new copies, return each bound bucket to its state at the barrier from its versions, start the old copies. Only the new copies’ own migrations and verification traffic are discarded. A unit whose apps process on their own once started (mailcow’s queue, GitLab’s CI and mirrors, Huly, wazuh, workflows, postiz, the Frappe schedulers; the dry run lists them, as it lists background writers for 5.3) has no step 2: from its first start its state is production state, and rollback is step 3. A forward-only unit (5.1) has no step 2 or 3.
  3. After 5.6 (users wrote to the new side): rollback is a reverse restore: quiesce the new unit, take its new-line backups, restore every database and volume of it into the old copies (the archive format is the same on both lines during the move, which wave 0 rehearses once per app class), leave each bucket at its current versions (the restored databases are current too, so the objects they name exist), then switch DNS back by the old cluster’s own takeover and reconcile, then start the old copies. If the reverse restore has not been rehearsed for that class, or fails, there is no rollback after the switch: the fix goes forward on the new side. Starting the old copy on its pre-move data while users have written to the new one is never done, because it silently drops their writes.

Mail must not bounce. Moving mailcow changes one record, the A record of mail.bitspur.com; MX stays 10 mail.bitspur.com for both domains, and SPF’s mx follows the A record by itself.

  • No bounce. While the old postfix is stopped and before the record flips, a sender gets a refused connection on 25, a temporary failure: it queues and retries for days. Nothing answers 5xx. The five-minute TTL tail lands on the stopped old postfix and retries the same way.
  • The relay credentials trap. Outbound goes through Amazon SES (relayHost, relayUsername, relayPassword). They are persisted overrides now, with mailDomain, keycloak.version, hostname and apppass.audience, and the new install carries all seven. A mailcow install without them falls back to direct delivery and reports healthy; 5.5 reads the Received chain for SES.
  • The certificate on raw TLS ports. 465, 993, 995, 587 and 143 serve CN=mail.bitspur.com from Let’s Encrypt (issued 09-29, read on the live ports). The new cluster issues it by DNS-01 in wave 0, mailcow’s certificate sync places it for dovecot and postfix, and 5.5 reads it with openssl s_client against 146.71.124.78 on every port.
  • IP reputation. 146.71.124.78 is not on Spamhaus ZEN, Barracuda, SORBS or PSBL (checked 10-01 from the Mac; rechecked from the node’s own resolver before wave 4). Outbound relays through SES, so the address only matters if the relay is lost. Its PTR is set to mail.bitspur.com with GorillaServers anyway (today’s PTR is Hetzner’s generic name). Outbound 25 from pve02 is open (measured).
  • Placement. Its ports are published on the node that runs it and mail.bitspur.com must resolve to exactly that node, so mailcow is pinned to shc01 until multi-replica --port is ruled.
  • After the switch mail lands on the new side only. Rolling back after that is a reverse restore of the whole mailcow backup (mailboxes, database, configuration and queue) into the old mailcow, never imapsync alone. This is the least reversible step of the migration.
  • Data. 12.9 GiB in restic (repositories through gitlab-backup, run with SKIP=db), gitlabhq_production in corp/postgres (section 4), and ten hel1 buckets: artifacts, packages, LFS, uploads, pages, terraform state, secure files, dependency proxy, external diffs and the registry (about 100 GiB, in no backup, SHC-812), bound again, never copied.
  • Single writer on the buckets. The old GitLab is stopped before the new one starts, always in that order, and the rehearsal never boots GitLab against them (5.1). Two GitLabs on one registry bucket can garbage-collect each other’s blobs.
  • The backup that freezes it. The old line’s capture pauses the whole stack while gitlab-backup-create runs (13 minutes tonight). In the move that pause is the write barrier. Do not restart GitLab while it is frozen.
  • SSH. Today Traefik publishes 2222 and routes it to GitLab’s sshd. On the new line the gitlab app must declare 22 (it declares only 80 and 5050 at next) and the deployment gets --port 2222:gitlab:22. That catalog change is filed before wave 5.
  • Webhooks. Its hooks to gitlab-huly and gitlab-fizzy move in its own unit. Its hooks to hookshot fail between waves 4 and 5; GitLab disables a hook after repeated failures, so 5.5 re-enables any hook with disabled_until set and checks one delivery.
  • CI. build01’s runner on pve01 reaches git.corp.bitspur.com; it is paused for the window. After the move GitLab and its runner share a datacenter.
  • Images. Nine deployments run images from registry.git.corp.bitspur.com (fizzy, fleet, gitlab-fizzy, mailcow, marketplace, nextcloud, pokemon-kc, pokemon-web, pulp), and pokemon-web’s package is a git URL on the same GitLab. Wave 0 pulls every image by digest and uploads every package, so no install depends on GitLab being up; nothing re-pulls during wave 5.

Every gated deployment depends on it. It moves first (wave 1), with master/postgres, its only database, and the platform record auth.corp.bitspur.com, which no deployment owns (Keycloak has no host binding; its name comes from system.apps.keycloak.host). The script takes that one name over explicitly in 5.6 and gives it back explicitly in a rollback.

  • The new cluster’s shcd reaches Keycloak through its node-aware dialer, its forward-auth decides from tokens Keycloak signs, and each restored app’s SSO client must exist in the Keycloak its cluster administers. Running the new cluster against the old Keycloak would keep compose01 in every login path for the whole move.
  • 29 of 32 consumers use the public issuer, so after the switch the old side’s apps follow DNS to the new Keycloak. The realm’s signing keys are in the restored database, so tokens validate on both sides.
  • identity-broker’s API calls get oidc.apiUrl=https://auth.corp.bitspur.com before wave 1 (an update that replays its persisted overrides). The two mailbox reconcilers build their URL from the socket’s private hostname with no override, so they stop at wave 1’s barrier and move with mailcow (3.2). They use their own confidential clients, so nothing else changes.
  • The risk this order carries: the old line’s shcd dials Keycloak for its own backchannel (forward-auth’s code exchange, JWKS, the console) through the same node-aware dialer (ce modules/auth/service.go). If that dialer keeps sending auth.corp.bitspur.com to compose01’s own ingress once the old Keycloak is stopped, the 17 gated deployments still on the old side lose login until they move. The rig proves this first, against an old-line cluster. If it fails, Keycloak moves last instead: the new side uses the old Keycloak by its public hostname (admitted per 3.3), and wave 1 is openobserve alone (open question 2).
  • It runs a path-pinned package (/var/lib/shc/_ncimport/load/adapted/nc, plugins mail and integration_paperless from the same tree). The script uploads that tree as a local package, so the new deployment runs the same bytes. Repointing it onto the catalog app comes after the move.
  • Its primary storage is cloud-bitspur-com in hel1, bound again: 57 GiB of objects do not move, and every file read crosses 157 ms to hel1 (measured from pve02) until the bucket moves (open question 3).
  • Its local data (5.7 GiB in restic) and its database cross normally, in wave 4’s unit with everything that reaches it privately: office (both directions), Stirling (pdf, wired by Nextcloud’s stirling plug, not by a plug edge), crm, fizzy, identity-broker and projects.
  • The files_stirling plug job is run by hand on the path-pinned deployment; 5.5 opens a PDF in Stirling and saves it back.
  • cloud.corp, cloud.bitspur.com and cloud.hel1.hetzner move together.
  • projects runs the huly fork (images on registry.gitlab.com), its database huly in corp/postgres, blobs in huly-corp-bitspur-com, 839 MiB in restic; the fulltext index rebuilds.
  • It reaches matrix, Nextcloud and identity-broker privately, so it is in wave 4’s unit. gitlab-huly reaches Huly by its public URL and GitLab privately, so it moves with GitLab in wave 5.
  • Huly is the tracker every agent writes to. During wave 4 no ticket write lands; wave 4 runs when no agent wave does.
  • bit and ffx run the catalog Frappe app with image overrides (images.frappe), their databases in the shared corp/mariadb, and /home/frappe/frappe-bench as a host bind mount that the backup carries as a host-bind tar (2.65 GiB for bit, 2.57 GiB for ffx). The image matters only at seed time; the restored bench is the truth.
  • ffx’s image shc/ffx-frappe:based93 is pushed and digest-pinned before wave 0 (section 1).
  • sites/assets are anonymous volumes and are rebuilt; 5.5 checks a page’s CSS.
  • marketplace (path-pinned, _w34) bundles its own MariaDB and moves whole in wave 2. pokemon-bench’s database is the separate deployment corp/pokemon-db, in its unit.

It resolves from the catalog now (2.0+34); the _platform path pin is gone. What still matters from its import is the seven vault secrets seeded from its source estate: a fresh install would mint credentials that cannot open its data. The vault copy carries them key by key, and 5.5 checks that /auth/v1/ and /rest/v1/ answer with key-auth and Studio sits behind the gate. Its bucket is bound again.

NetBird is how VPN users reach LAN-only apps, so moving it can cut that access, never ours: every node of both estates keeps public SSH from the admin addresses, the out-of-band path for every wave.

  • Today the route estate-lan advertises 10.236.16.0/20 through the peer on swarm01, with masquerade, and dnsmasq on swarm01 answers *.corp.bitspur.com with compose01’s private 10.236.16.8. From wave 1, as each unit switches, dnsmasq answers its hosts with the new node’s address, and the route gains 146.71.124.78/32, .68/32 and .69/32 through the same peer, so VPN traffic to a moved app arrives from swarm01’s egress, which 3.3 admits. Those are edits on swarm01’s hand-made dnsmasq and in NetBird’s own settings, not on compose01, and the script journals each previous answer so a rollback restores it (5.8). The routes are added once per node and stay while any moved unit runs there.
  • Wave 7 moves the management, signal (--port 33073:netbird-signal:80) and relay (--port 33080:netbird-relay:33080), their database and volumes. Peers reconnect to vpn.corp.bitspur.com and api.vpn.corp.bitspur.com after the switch. A routing peer on shc01 takes over the routes and the split DNS, and swarm01’s peer is removed. vpn and vpn-api stay public in the split DNS, or a client cannot reach its own control plane on reconnect.
  • The second VPN path, the pfSense in hel1, keeps working because its clients’ egress is admitted by 3.3; where it lives after compose01 goes is open question 9.
  • The source. Everything is restored from shc-backup-primary-bitspur-com in hel1. The script reads it with the old estate’s restic password and S3 key, taken from the old vault at run time, never written down. Only the old cluster’s own schedules write to it.
  • The new cluster’s repository is a new bucket, never the old one and never the old copy bucket: pointing a new repository or a mirror at a bucket holding another generation is how the copy bucket’s history was destroyed on 08-22. Recommendation: a new hel1 bucket on the estate’s existing credential for the move, so no new secret is needed, and a US repository later (open question 4). Each unit’s first new-line backups are taken in its watch night, which spreads the first full upload (about 90 GiB at the 6 MiB/s pve01 measured to hel1) across the waves.
  • The final pre-move archive of each deployment is also imported into the new repository by the move itself (§6’s shc backup import), so it outlives the old repository’s retention.
  • The old schedules keep firing on compose01 and will try the stopped deployments every night. Untouched means they are left alone; the alerts for stopped deployments are expected noise for the two weeks, and retention (7 daily, 4 weekly, 12 monthly) keeps the pre-move snapshots.

It extends what exists: bitspur/ops terraform/cluster/migration/ (migrate.sh, merged 09-30 as !198), which today runs the read-only dry run against compose01 through an allowlist of reads and refuses any promanager host. It stays outside shc and ce and is deleted once bitspur is across.

It does:

  • inventory, render, plan, dry-run as today, the plan rebuilt from this design: the move units (3.2), the waves and their order (3.4), every database of every unit (section 4), the platform records each unit carries, and the preconditions below.
  • prepare <wave>: section 5.1 for the wave’s units.
  • move <unit>: 5.3 to 5.6, then records the watch.
  • rollback <unit>: 5.8, from whichever phase the unit reached.
  • status: one line per unit with its phase.

The only writes it makes on the old cluster: stopping and starting the units it moves and cutting and restoring their ingress; taking their backups; the LAN allowlist cross-admission (3.3); oidc.apiUrl on identity-broker (6.3); stopping gitlab-fizzy and the two reconcilers until their unit moves (3.2). Outside compose01 it also edits swarm01’s dnsmasq and NetBird’s routes as units switch (6.8); swarm01 refuses the Mac’s key, so that needs the cluster key from the ops Terraform state. It never deletes anything on either side, copies a volume off compose01, moves a bucket, mints, rotates or prints a secret (it reads them from the old vault at run time), or touches promanager.

When it refuses to run. No move starts before the SHC-1165 gate is recorded as passed. No barrier may overlap a backup on either side: the old estate’s corp tenant run (03:30 to about 05:20 UTC tonight), master’s openobserve (04:00), postgres (04:30) and keycloak (05:00), traefik on Sundays (05:30) and the shc tenant (23:00 to about 23:05), each widened by 30 minutes and by its last measured duration. The script checks for a running backup job on both clusters and refuses a unit whose estimated window would reach the next scheduled run.

Inputs. ssh to compose01 ([email protected], the estate key) and the old CLI there; the new cluster’s CLI context on the Mac (shc.odg.bitspur.com); the plan file it generates; ce and shc checkouts at the commit the new cluster runs, for the render check.

Dry run. Every command takes --dry-run, which prints each API call and remote command it would run and the checks it would make, and runs only the reads. move refuses unless the unit’s preconditions hold: rehearsal restore passed, images present, certificates issued, the new installs exist and are stopped, every member’s last scheduled backup is verified and under 26 hours old (or, for a paused deployment, its verified pause-time capture, 3.2), its wave’s earlier units are done, and the window is clear.

State and resume. One JSON file per unit under ~/.drover/state/shc-1061-migration/units/<unit>.json (mode 600, outside the repo): its members, phase (prepared, barrier, backed-up, restored, verified, switched, watching, done, rolled-back), each member’s old final snapshot id, the engine snapshot id, the new backup ids, the records moved, the rollback prerequisites (old providers that must be running), and every check’s result with its time. Each phase is idempotent and is written before the next starts, so a killed run resumes with the same move command from the last completed phase, and a unit can never be marked done with a member behind.


Every connection on bitspur whose two ends are in different tenants:

consumerplugprovider
corp/affine, bit, blog, ciso, dtrack, ffx, fizzy, gitlab, glitchtip, identity-broker, ivatar, librechat, mailbox-reconciler, mailbox-reconciler-ffx, mailcow, matrix, netbird, nextcloud, oncall, openreplay, peertube, penpot, photos, postiz, projects, supabase, superset, wazuh, workflowsoidcmaster/keycloak
corp/fleet, mautic, sonarqubesamlmaster/keycloak

32 edges, all corp to master/keycloak. Nothing else crosses: no corp deployment uses master/postgres or openobserve’s store directly, and the shc tenant only reaches inside itself. At the network level 32 corp containers sit on master_default_keycloak_public and 51 on master_default_traefik_public (ingress, which is the system tier).

The new line refuses a plug across tenants where it is declared (X400372FABXTN, SHC-1202, merged on next), with one carve-out: the shared keycloak at master/default is the one cross-tenant edge the check and the fabric let through (Clay, 2026-10-02; ce modules/app/integration_tenant.go, integration_magic.go). These 32 are all edges to that keycloak, so as the code stands they reinstall as plugs through it. The resolution below was written before the carve-out. Named resolution: on the new line SSO stops being a plug. A deployment’s SSO client becomes a platform auth registration (/api/auth, the module that already provisions clients), which renders the same values the templates read today (clientId, clientSecret, issuer, realm, the SAML entity) without a plug declaration, and the script carries each client’s id, secret, redirect URIs and grants from the restored Keycloak and the copied vault. The network path is the public issuer through ingress, so no fabric accept is needed. This has to be ruled and built before wave 0 (open question 8); it is product work, not script work.


Under the publication ruling the host side of ports: is not honoured, so each needs --port at install. Grammar [<conn>@]<external>[:<service>][:<container>][/udp] (ce modules/app/port_mappings.go).

deploymenttoday--port at install
corp/mailcow25, 465, 587 on postfix-mailcow; 110, 143, 993, 995, 4190 on dovecot-mailcow--port 25:postfix-mailcow:25 --port 465:postfix-mailcow:465 --port 587:postfix-mailcow:587 --port 110:dovecot-mailcow:110 --port 143:dovecot-mailcow:143 --port 993:dovecot-mailcow:993 --port 995:dovecot-mailcow:995 --port 4190:dovecot-mailcow:4190
corp/netbird33080 on netbird-relay; 33073 to netbird-signal’s 80--port 33080:netbird-relay:33080 --port 33073:netbird-signal:80
corp/gitlab2222 through Traefik’s raw TCP entrypoint to sshd--port 2222:gitlab:22, once the app declares 22 (6.2)
master/traefik80, 443not moved: the new line’s per-node Traefik binds them
shc-collector4318 on docker0not moved: the per-node collector is the new line’s own

All three run single-replica on a pinned node until multi-replica --port is ruled, and their hostnames resolve to that node only.


10. Risks, what cannot be undone, and windows

Section titled “10. Risks, what cannot be undone, and windows”

Cannot be undone without a reverse restore (5.8 step 3):

  • Mail delivered to the new mailcow (6.1).
  • Writes on the new side after a switch: GitLab pushes and merges, Huly tickets, Nextcloud files (their objects sit in the shared bucket, their rows in the new database), Keycloak password and user changes, Frappe documents.
  • Anything in a forward-only unit (an unversioned bucket, 5.1) after 5.4.

Cannot be undone at all: Matrix notifications GitLab and wazuh sent to hookshot while it sat in another wave (a day or two), and the GitLab-to-Fizzy sync and mailbox provisioning skipped while their bridges were stopped (3.2). Data in GitLab, wazuh and oncall is unaffected.

Top risks:

  1. Keycloak first strands old-side SSO if the old shcd’s dialer pins to its own ingress (6.3). Proven on the rig before wave 1; the fallback order is written.
  2. The cross-tenant refusal blocks every SSO app until SSO is a platform registration (section 8). Product work on the critical path.
  3. LibreChat’s conversations and corp/mysql’s databases may not be in their backups (section 1). The rehearsal decides; nothing moves on belief.
  4. pve03 holds two voters. Losing that box stops lease-held apps until a manual repair (section 2).
  5. The ocean. Every bucket and the backup repository stay in hel1, 157 ms away. Restores pull at about 1 GiB a minute; Nextcloud, the GitLab registry and peertube serve every object across that link until the buckets move.
  6. compose01 is fragile (91% disk, 29 GiB swap). Each fresh backup stages on /mnt/shc-staging; no barrier runs beside another heavy job, and the units leaving take load off it.
  7. Single-writer buckets. Old copies stop before new ones start, and rehearsals never touch them.
  8. Wave 4 is one long barrier: mail and every collaboration app down together for about three hours, and Huly’s downtime stops agent ticket writes.
  9. The licence and the takeover need the old cluster’s identity; a rename before the move breaks both.

Windows: wave 1 about 20 minutes of no logins; waves 2 and 3 a few minutes per unit, over four or five nights; wave 4 about three hours; wave 5 about 1.5 hours of GitLab; wave 6 about two hours; wave 7 about 45 minutes of VPN. Roughly ten windows over two to three weeks, then the two-week hold.


#questionrecommended answer
1The ruling puts every new node on pve02 and pve03, so the third voter shares pve03 and losing pve03 loses quorum. Allow a voter-only VM elsewhere?Yes: a 2 vCPU, 4 GiB voter-only VM on pve01 (same datacenter, its own box), taken from the lane farm’s budget. Then any one box can fail.
2Keycloak first or last?First (wave 1), if the rig shows the old line’s shcd reaches it by public DNS once the old copy stops; last otherwise.
3Do the 26 buckets stay in hel1?Yes for the move, bound again, no copy. Move them to US object storage afterwards, Nextcloud’s first, one app at a time.
4Where does the new cluster’s backup repository live?A new bucket in hel1 on the existing credential for the move, never an old bucket. A US provider afterwards, with credentials you mint.
5Keep the cluster’s identity (slug, owner heritage, shc.hel1.hetzner.bitspur.com, the licence) through the move?Yes: the takeover and the licence depend on it. Rename afterwards with a re-minted licence, with the other *.hel1.hetzner.* names.
6Restore openobserve’s 24 GiB of logs, or start it empty?Restore it: every app moves from its backup, and it costs one 75-minute window with nothing depending on it.
7Move corp/nc-db (no consumer) and ffx’s stale bench.pre-based93?No. Both stay on compose01 and go with it.
8SSO on the new line: a platform auth registration instead of a cross-tenant plug?Yes, with the refusal kept unconditional. Rule it now; it gates wave 0.
9Where does the pfSense VPN path land after compose01 goes?Keep the hel1 pfSense through the move with its egress admitted; afterwards pfsense01 on pve01 takes it over, so both paths end in Ogden.
10SHC-1061 still has the criterion “the pfSense HAProxy edge points at the new cluster’s Traefiks”.Drop it. That HAProxy is empty and the public names resolve straight to the nodes; the DNS switch is the re-point.
11The writes on compose01 this design needs (LAN allowlist cross-admission, identity-broker’s oidc.apiUrl, gitlab-fizzy and the reconcilers stopped between waves)?Yes, all before wave 1, kept through the two weeks.
12Wave 4 as one three-hour barrier for mail and every collaboration app?Yes. Splitting it needs public endpoints for six private edges whose templates have no override, which is catalog work on six apps for one night’s saving.

  • The new line, deleting first, §3.13 policy, §3.14 publication, §4.9 suffrage, §6 the one-time migration.
  • The SHC scheduler, §3.5 reserved fences, §3.25 stopped-app backups, §7 the cutover gate.
  • Multi-region fences.
  • bitspur/ops terraform/cluster/migration/ (the dry run and its 09-30 SUMMARY) and terraform/gorillaservers/ogden/ (pve01 to pve03 and the rig).
  • SHC-1061 (this), SHC-1165 (the gate), SHC-1123 (the rig), SHC-1202 (policy), SHC-1201 (mesh), SHC-812 (registry backup), SHC-691 (ffx image).