A node failing
Remove the members group first. This page follows one deployment. The uninstall removes the copy on each of the three nodes, in whatever order they answer.
$ shc uninstall members -yuninstalling deployment "members"[members] Complete 3/3...✔ uninstalled deployment "members"A label and a location limited placed to learn/home.
Change the rule to learn/far. far01 is the only node there, so the database
has to move, and its volume with it: backup.yaml marks pgdata critical,
and a move must carry critical data. --plan shows the move and what it
carries before anything happens.
$ shc update placed --where location=learn/far --plan...$ shc update placed --where location=learn/far --yesrelocation plan for prod/placed/prod placement: far01 is the only node its rules, condition and capacity leave replicas: postgres r0 moves home01 -> far01 volumes: pgdata durable, copied to the target, 61.9 MiB durable data: 61.9 MiB to carry downtime: about 34swarning: placed landed with 64703721 bytes in 1287 files against a source of 64960703 bytes in 1286 files, 99.6% of the bytes and 100.1% of the files. Expected if its backup.yaml excludes caches, worth checking otherwisewarning: prod/prod/placed was checked by size only: this node cannot take a per-file manifest of a deployment on node node-ec8c46fe25e0, so a target of the right size holding the wrong content would not have been detectedwarning: placed's pre-move data was KEPT at /var/lib/shc/deployments/prod/placed/prod.moved-3d2a9b26 on home01. Verify the target is serving, then run `shc reclaim` on home01warning: [X200628CLNARC] the pre-move backups are kept at /var/lib/shc/backups/prod/placed, 1 in all, and each comes under normal retention once the move releases it, or stays in the move's staging until the move is recovered✔ relocated deployment "placed" to far01$ shc ls --where tier=learnSTACK TENANT ENV APP STATUS NODEplaced prod prod notes running far01$ shc exec -s placed -S postgres -- psql -U postgres -tAc 'select body from notes'kept by the backupThe row the backup page kept came along. The move also kept the data it
left on home01, until shc reclaim there clears it. Now stop SHC on far01. This stops
the node’s daemon, not its containers: placed’s postgres keeps running on
far01, but the rest of the cluster can no longer reach it.
$ systemctl stop shcThe other nodes notice. Run shc node ls again until far01 changes state.
$ shc node lsNAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD LEADER VOTERfar01 unreachable down active learn learn/far ok 0.1.0 no nohome01 healthy ready active learn learn/home ok 0.1.0 yes yeshome02 healthy ready active learn learn/home ok 0.1.0 no no$ shc ls --where tier=learnSTACK TENANT ENV APP STATUS NODE REASONplaced prod prod notes unknown far01 node far01 unreachable since 2026-10-10T20:17:01Z; failover held since 2026-10-10T20:17:01Z: none: it stays assigned and starts when its node returns; postgres: local volume pgdatawarning: 1 of 1 deployments are not running: 1 unknownwarning: node far01 is unreachable since 2026-10-10T20:17:01.036034159Z and holds 1 deployment$ shc cluster statuscluster name: shc-clusterleader: home01nodes: 2 healthy of 3, 1 unreachablevoters: 1quorum: healthy, 1 voter requiredwarning: only 1 raft voter: losing one stops the control plane committing, and then every deployment with a single-writer volume stops its writer within about a minute, on the surviving nodes too, unless the volume is far, with no control-plane fence node in its reachwarning: control-plane fence has 2 members, fewer than three, so it cannot hold three voters until `shc node update <node> --control-plane` brings a node into it, and the voter count followsleader-only work, recorded by home01 32s ago: acme last swept 58s ago backup_scheduler running 6m58s, no sweep yet container_reconciler last swept 57s ago mesh_snapshot_sign last swept 37s ago operation_registry last swept 58s ago proxy_reconcile last swept 57s ago schedule_dispatcher last swept 57s ago scheduler last swept 37s ago service_vip_alloc last swept 32s agofar01 does not vote: home01 is the one voter, so the cluster keeps deciding
without far01. placed keeps its data on a local volume, so SHC holds it on
far01 and starts it there when far01 returns, instead of starting an empty
copy somewhere else.
Start SHC on far01 again.
$ systemctl start shcRun shc node ls again until far01 reads healthy, then read the row the
backup page kept.
$ shc node lsNAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD LEADER VOTERfar01 healthy ready active learn learn/far ok 0.1.0 no nohome01 healthy ready active learn learn/home ok 0.1.0 yes yeshome02 healthy ready active learn learn/home ok 0.1.0 no no$ shc exec -s placed -S postgres -- psql -U postgres -tAc 'select body from notes'kept by the backupRead Underneath a node failing, then Render once on the leader; each reads what this page left running.