Skip to content
SHC Docs

Upgrading the daemon

SHC is upgraded by installing the newer package on each node and restarting the daemon there. There is no shc upgrade: the package manager installs, the daemon runs its schema migrations when it starts, and the cluster carries on. Every node runs the same version, so a cluster upgrade is a rolling one: one node at a time, never more than one voter away, the leader last.

This page installs the package you kept at ~/shc.deb on every machine. It is the version already running, so the version does not change; a newer package goes through exactly these steps.

home01
$ shc version
shc version 0.1.2
build: embedded-tui
UNLICENSED/trial watermarked, see `shc license status`

The leader writes a snapshot of the control-plane database every night into /var/lib/shc/snapshots/control-plane; it is what a failed upgrade is rolled back from. Read when the next one runs. A cluster younger than a day has none yet.

home01
$ shc cron ls
...

Take one now rather than wait for the night. shc db snapshot create writes a snapshot through the nightly’s own dump into the same directory. It runs on the leader only, home02 here; on any other node it refuses and names the leader. shc db snapshot ls lists the snapshots the node it runs on holds, newest first, the on-demand one beside any nightly, with its time and size.

home02
$ shc db snapshot create
...
$ shc db snapshot ls
...

Start with far01, the worker. A maintenance window tells the scheduler the node is away on purpose, so nothing fails over while the daemon restarts.

home01
$ shc node maintenance far01 --window 10m
warning: node far01 is cordoned; this binds SHC's own placement only: it is excluded from install-time scoring and from drain relocation targets, but a compose deploy typed against it directly with `shc install` on that node or `--node far01` still lands there
✔ node "far01" is in maintenance until 2026-10-11T05:51:29Z, availability cordoned: nothing is rescheduled from it before then
far01
$ dpkg -i ~/shc.deb
...
$ systemctl restart shc
$ shc version
shc version 0.1.2
build: embedded-tui
UNLICENSED/trial watermarked, see `shc license status`

Wait for far01 to read healthy, then end the window.

home01
$ shc node ls --where node=far01
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD LEADER VOTER
far01 healthy cordoned cordoned learn learn/far ok 0.1.0 no no
home01
$ shc node maintenance far01 --end
✔ maintenance on node "far01" lifted, availability active

The voters next, one at a time. With three voters, two must answer at every moment, so wait for each to be healthy before the next.

home01
$ shc node maintenance home03 --window 10m
warning: node home03 is cordoned; this binds SHC's own placement only: it is excluded from install-time scoring and from drain relocation targets, but a compose deploy typed against it directly with `shc install` on that node or `--node home03` still lands there
✔ node "home03" is in maintenance until 2026-10-11T05:51:53Z, availability cordoned: nothing is rescheduled from it before then
home03
$ dpkg -i ~/shc.deb
...
$ systemctl restart shc
home01
$ shc cluster status
cluster name: shc-cluster
leader: home02
nodes: 4 healthy of 4
voters: 3
quorum: healthy, 2 voters required
cordoned: 1 of 4 nodes
leader-only work, recorded by home02 1m4s ago:
...
home01
$ shc node maintenance home03 --end
✔ maintenance on node "home03" lifted, availability active
$ shc node maintenance home01 --window 10m
warning: node home01 is cordoned; this binds SHC's own placement only: it is excluded from install-time scoring and from drain relocation targets, but a compose deploy typed against it directly with `shc install` on that node or `--node home01` still lands there
✔ node "home01" is in maintenance until 2026-10-11T05:57:11Z, availability cordoned: nothing is rescheduled from it before then
$ dpkg -i ~/shc.deb
...
$ systemctl restart shc
home01
$ shc cluster status
cluster name: shc-cluster
leader: home02
nodes: 4 healthy of 4
voters: 3
quorum: healthy, 2 voters required
cordoned: 1 of 4 nodes
leader-only work, recorded by home02 16s ago:
...
$ shc node maintenance home01 --end
✔ maintenance on node "home01" lifted, availability active

The leader last. While its daemon restarts the other two voters elect a leader between them; the restarted node comes back as a follower.

home01
$ shc node maintenance home02 --window 10m
warning: node home02 is cordoned; this binds SHC's own placement only: it is excluded from install-time scoring and from drain relocation targets, but a compose deploy typed against it directly with `shc install` on that node or `--node home02` still lands there
✔ node "home02" is in maintenance until 2026-10-11T06:00:33Z, availability cordoned: nothing is rescheduled from it before then
home02
$ dpkg -i ~/shc.deb
...
$ systemctl restart shc
home01
$ shc cluster status
cluster name: shc-cluster
leader: home01
nodes: 4 healthy of 4
voters: 3
quorum: healthy, 2 voters required
cordoned: 1 of 4 nodes
leader-only work, recorded by home01 5s ago:
...
home01
$ shc node maintenance home02 --end
✔ maintenance on node "home02" lifted, availability active

Every node reports the version it runs. Check them all.

home01
$ shc version
shc version 0.1.2
build: embedded-tui
UNLICENSED/trial watermarked, see `shc license status`
home02
$ shc version
shc version 0.1.2
build: embedded-tui
UNLICENSED/trial watermarked, see `shc license status`
home03
$ shc version
shc version 0.1.2
build: embedded-tui
UNLICENSED/trial watermarked, see `shc license status`
far01
$ shc version
shc version 0.1.2
build: embedded-tui
UNLICENSED/trial watermarked, see `shc license status`

An upgrade is also when the platform’s modules are re-ensured. shc cluster reconcile installs any module that init skipped and now fits; --modules names the set, here the one the daemon already has. openobserve still needs a node with three CPUs, and this cluster has none, so reconcile skips it again and says so.

home01
$ shc cluster reconcile --modules postgres,keycloak,openobserve,traefik
...
$ shc reconcile --modules postgres,keycloak,openobserve,traefik
...

Every deployment ran through the whole roll.

home01
$ shc ls --where owner=learn
STACK TENANT ENV APP STATUS NODE
a1 prod prod /root/learn/cluster/hello running home01
a2 prod prod /root/learn/cluster/hello running far01
a3 prod prod /root/learn/cluster/hello running home02
a6 prod prod /root/learn/cluster/hello running home02

A package older than the one running goes back the same way, node by node, as long as the newer one added no schema migration: a daemon whose database holds a migration it does not ship refuses to start. Keep the package you are replacing; the package manager does not.

The control-plane database keeps every event. shc db purge reports what its rules would take and takes nothing without --yes; --before bounds it. A cluster a day old has little to take.

home01
$ shc db purge
...
$ shc db purge --before 30d
...

--yes performs the purge, --batch-size bounds each delete, and --audit takes the compliance records with it, which no second copy holds.

explained, not walked: a day-old cluster has nothing to take and an audit chain to keep
$ shc db purge --before 30d --batch-size 500 --yes
$ shc db purge --audit --before 30d --yes

An image stays on a node after nothing there runs it. shc node prune removes those on the nodes it names, or on every node with --all. It keeps an image a container on the node runs, one a deployment there names, one a rollback would need and one the node could not pull back, at any age. It prints per node what it removed, what it kept and why. --dry-run prints the same and removes nothing.

home01
$ shc node prune --all --dry-run
...

The platform’s own Keycloak has an admin password the genesis minted. An operator who lost it resets it on the node that holds the keycloak stack; the recorded seed follows and the stack redeploys. Without --password a new one is generated and printed once. The access ladder’s Users and SSO page types it and logs in with the new password, so it is not typed here.

explained, not walked: the access ladder's users page types it
$ shc system keycloak reset-admin-password --password P@ssw0rd

Next: Repositories.