Skip to content
SHC Docs

A node failing

Remove the members group first. This page follows one deployment. The uninstall removes the copy on each of the three nodes, in whatever order they answer.

home01
$ shc uninstall members -y
uninstalling deployment "members"
[members] Complete 3/3
...
✔ uninstalled deployment "members"

A label and a location limited placed to learn/home. Change the rule to learn/far. far01 is the only node there, so the database has to move, and its volume with it: backup.yaml marks pgdata critical, and a move must carry critical data. --plan shows the move and what it carries before anything happens.

home01
$ shc update placed --where location=learn/far --plan
...
$ shc update placed --where location=learn/far --yes
relocation plan for prod/placed/prod
placement: far01 is the only node its rules, condition and capacity leave
replicas: postgres r0 moves home01 -> far01
volumes: pgdata durable, copied to the target, 61.9 MiB
durable data: 61.9 MiB to carry
downtime: about 34s
warning: placed landed with 64703721 bytes in 1287 files against a source of 64960703 bytes in 1286 files, 99.6% of the bytes and 100.1% of the files. Expected if its backup.yaml excludes caches, worth checking otherwise
warning: prod/prod/placed was checked by size only: this node cannot take a per-file manifest of a deployment on node node-ec8c46fe25e0, so a target of the right size holding the wrong content would not have been detected
warning: placed's pre-move data was KEPT at /var/lib/shc/deployments/prod/placed/prod.moved-3d2a9b26 on home01. Verify the target is serving, then run `shc reclaim` on home01
warning: [X200628CLNARC] the pre-move backups are kept at /var/lib/shc/backups/prod/placed, 1 in all, and each comes under normal retention once the move releases it, or stays in the move's staging until the move is recovered
✔ relocated deployment "placed" to far01
$ shc ls --where tier=learn
STACK TENANT ENV APP STATUS NODE
placed prod prod notes running far01
$ shc exec -s placed -S postgres -- psql -U postgres -tAc 'select body from notes'
kept by the backup

The row the backup page kept came along. The move also kept the data it left on home01, until shc reclaim there clears it. Now stop SHC on far01. This stops the node’s daemon, not its containers: placed’s postgres keeps running on far01, but the rest of the cluster can no longer reach it.

far01
$ systemctl stop shc

The other nodes notice. Run shc node ls again until far01 changes state.

home01
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD LEADER VOTER
far01 unreachable down active learn learn/far ok 0.1.0 no no
home01 healthy ready active learn learn/home ok 0.1.0 yes yes
home02 healthy ready active learn learn/home ok 0.1.0 no no
home01
$ shc ls --where tier=learn
STACK TENANT ENV APP STATUS NODE REASON
placed prod prod notes unknown far01 node far01 unreachable since 2026-10-10T20:17:01Z; failover held since 2026-10-10T20:17:01Z: none: it stays assigned and starts when its node returns; postgres: local volume pgdata
warning: 1 of 1 deployments are not running: 1 unknown
warning: node far01 is unreachable since 2026-10-10T20:17:01.036034159Z and holds 1 deployment
$ shc cluster status
cluster name: shc-cluster
leader: home01
nodes: 2 healthy of 3, 1 unreachable
voters: 1
quorum: healthy, 1 voter required
warning: only 1 raft voter: losing one stops the control plane committing, and then every deployment with a single-writer volume stops its writer within about a minute, on the surviving nodes too, unless the volume is far, with no control-plane fence node in its reach
warning: control-plane fence has 2 members, fewer than three, so it cannot hold three voters until `shc node update <node> --control-plane` brings a node into it, and the voter count follows
leader-only work, recorded by home01 32s ago:
acme last swept 58s ago
backup_scheduler running 6m58s, no sweep yet
container_reconciler last swept 57s ago
mesh_snapshot_sign last swept 37s ago
operation_registry last swept 58s ago
proxy_reconcile last swept 57s ago
schedule_dispatcher last swept 57s ago
scheduler last swept 37s ago
service_vip_alloc last swept 32s ago

far01 does not vote: home01 is the one voter, so the cluster keeps deciding without far01. placed keeps its data on a local volume, so SHC holds it on far01 and starts it there when far01 returns, instead of starting an empty copy somewhere else.

Start SHC on far01 again.

far01
$ systemctl start shc

Run shc node ls again until far01 reads healthy, then read the row the backup page kept.

home01
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.0 no no
home01 healthy ready active learn learn/home ok 0.1.0 yes yes
home02 healthy ready active learn learn/home ok 0.1.0 no no
home01
$ shc exec -s placed -S postgres -- psql -U postgres -tAc 'select body from notes'
kept by the backup

Read Underneath a node failing, then Render once on the leader; each reads what this page left running.