Skip to content
SHC Docs

Underneath a node failing

Read this right after A node failing. Every command here reads; none changes the cluster.

The leader logs by node id. far01’s is in the first column here:

home01
$ shc node info far01
ID HOSTNAME RUNTIME AVAILABILITY INTERNAL ADDRESS UNDERLAY ADDRESS PUBLIC ADDRESS CLOUD LOCATION IS VOTER FABRIC GRANT FABRIC TENANT SLOTS FABRIC FULLEST BRIDGE
node-3845076fd5b8 far01 docker active 10.192.0.4:4003 65.21.55.73 65.21.55.73 learn learn/far false 10.128.64.0/20 2 of 31 master, 3 of 125 addresses

The leader’s journal for the last half hour, the lines about losing far01, holding its deployments and getting it back. If far01 joined less than half an hour ago, the window also holds a short unavailable and reachable pair from just after the join, when home01’s daemon restarted to replicate the cluster’s database; the ... stands for it.

home01
$ journalctl --no-pager -u shc --since '30 min ago' -o cat -g 'X503014NODNAV|V200872FOVHLD|node reachable again'
...
time=2026-10-09T20:22:59.039Z level=WARN msg="node node-729da85a624c unavailable" code=X503014NODNAV http_status=503 node_id=node-729da85a624c
time=2026-10-09T20:23:26.772Z level=WARN msg="master/traefik/default on lost node node-729da85a624c is held: none: it stays assigned and starts when its node returns; traefik: local volume static" code=V200872FOVHLD deployment=master/traefik/default from=node-729da85a624c movability=pinned action=held detail="none: it stays assigned and starts when its node returns; traefik: local volume static"
time=2026-10-09T20:23:26.772Z level=WARN msg="prod/placed/prod on lost node node-729da85a624c is held: none: it stays assigned and starts when its node returns; postgres: local volume pgdata" code=V200872FOVHLD deployment=prod/placed/prod from=node-729da85a624c movability=pinned action=held detail="none: it stays assigned and starts when its node returns; postgres: local volume pgdata"
time=2026-10-09T20:23:43.094Z level=INFO msg="node reachable again" node=node-729da85a624c suppressed=21 was_unavailable_for=44.054416917s

The leader marked far01 unavailable when it stopped answering. A deployment on a lost node moves only when its data can go with it. placed keeps its data in a local volume on far01, and so do far01’s copy of traefik and any other database the scheduler had put there, so moving one would start it empty on another node. SHC held each where it was, and they ran again when far01 came back.

Next: Render once on the leader.