Rolling a daemon back
Upgrading SHC covers going forward. This page covers going back, and it exists because the interesting half of “can we roll back?” is not the binary; it is the state the new binary already touched.
The short version:
| Question | Answer |
|---|---|
Does dpkg keep the deb it replaced? | No. Nothing retains it. Keep it yourself, before you roll. |
Will dpkg -i install an older deb over a newer one? | Yes, silently. No --force-downgrade needed. |
| Do the maintainer scripts fight a downgrade? | No. prerm/postinst are direction-agnostic. |
| Does the daemon refuse to boot on a schema it does not know? | Yes. A ledger row newer than every shipped file stops the boot with X409991MIGNEW and exit 78 before any subsystem starts, and the unit does not restart it. See a downgrade across a migration refuses to boot. |
| Can a schema migration be undone? | No. There are zero down-migrations. |
| So can I roll back? | Only within a window that crosses no schema migration. Outside it, the binary goes back and the database does not. |
Before you roll forward: the two-minute pre-flight
Section titled “Before you roll forward: the two-minute pre-flight”Do these before the roll, not after. Both are cheap and both are unrecoverable if skipped.
1. Keep the deb you are replacing
Section titled “1. Keep the deb you are replacing”A node rolled by scp + dpkg -i has no apt repository in the picture.
That means:
dpkg -idoes not populate/var/cache/apt/archives; checked on a live node, there is noshc_*.debthere.dpkgkeeps no copy of the payload it replaced.apt-get install --reinstall shc=<oldversion>has nothing to talk to, unless the node reads the Based apt repo and the old version was a release: releases from 0.1.1 on are there asshc=1:<ver>, and every release’s deb stays athttps://downloads.selfhostedcloud.org/shc/<ver>/. A snapshot build is in neither.
If you do not save the outgoing deb, your rollback artifact does not exist. Save it off-node:
# On each node, BEFORE installing the new debdpkg-query -W -f='${Version}\n' shc # record thisand keep the deb you last installed, plus the CI artifact it came from. Every
package/snapshot job publishes dist/shc_*_linux_amd64.deb, so the outgoing
version is re-fetchable from its pipeline as long as the artifact has not
expired; verify that before relying on it.
1a. Do not read anything into the deb’s version number
Section titled “1a. Do not read anything into the deb’s version number”Snapshot debs are versioned 1:0.0.<n>~SNAPSHOT-<gitsha>. dpkg splits that at
the last hyphen, so the part that actually orders two builds of the same
0.0.<n> line is the git sha, compared as a Debian version string. Git shas
are not ordered, so neither are the debs:
$ dpkg --compare-versions '1:0.0.57~SNAPSHOT-5d42fa6ee8a' lt '1:0.0.57~SNAPSHOT-e573822b6ef' && echo lowerlowerThat pair is a genuine roll forward (5d42fa6 is 17 commits ahead of
e573822), and dpkg considers it a downgrade. The reverse pair would
compare the other way purely on the first hex digit.
Consequences:
-
dpkg -idoes not care, which is why both directions work. On a genuine version-number downgrade it warns and proceeds. Measured:Terminal window dpkg: warning: downgrading shc (1:0.0.57~SNAPSHOT-e573822b6ef) to (1:0.0.1~SNAPSHOT-e573822)Unpacking shc (1:0.0.1~SNAPSHOT-e573822) over (1:0.0.57~SNAPSHOT-e573822b6ef) ...Setting up shc (1:0.0.1~SNAPSHOT-e573822) ...No
--force-downgradeis needed. Treat the warning as information, not a failure. -
Anything that does care is unreliable here:
apt,unattended-upgrades,--refuse-downgrade, and any script gating ondpkg --compare-versions. Never gate a roll or a rollback on version ordering; gate on the git sha you intended. -
shc versionprints the upstream version with~rendered as-and no epoch, so1:0.0.57~SNAPSHOT-5d42fa6ee8areports as0.0.57-SNAPSHOT-5d42fa6ee8a. Normalise both when comparing.
Locally built debs (make deb) use a different upstream version (0.0.1) than
CI snapshots (0.0.57), so a CI deb outranks a local one regardless of which is
newer in git. Do not mix the two sources across a roll without checking what is
actually installed.
2. Confirm the control-plane snapshot is current
Section titled “2. Confirm the control-plane snapshot is current”This is the only thing that can undo a migration. The leader-only
db_snapshot_cron writes one every night at 02:00 UTC to
/var/lib/shc/snapshots/control-plane/. shc db snapshot create takes one
now, through the same dump into the same directory. Take one before a roll
or a hand change to vault rows, placement or users.
shc db snapshot create # on the leader; any other node refuses and names itshc db snapshot ls # on-demand and nightly snapshots, with time and sizeshc db snapshot ls lists the directory of the node it runs on, and a
snapshot sits on the node that led when it was taken. Retention is
maintenance.control_plane_snapshot_retention (default 7) for each kind on
its own: an on-demand snapshot prunes only older on-demand ones, never a
nightly, and the nightly never prunes an on-demand one.
Is this roll reversible? Decide before you start
Section titled “Is this roll reversible? Decide before you start”A roll is reversible by binary alone iff it crosses no migration. The store’s
migration lives in the ce module (<ce>/migrations/, one generated file until
v1, embedded via migrations/embed.go), and shc takes it through its go.mod
pin. So the question reduces to one diff:
# In the shc checkout, OLD = the version currently deployed, NEW = the roll targetgit diff OLD..NEW -- go.mod go.sum- Empty → the
cepin did not move → the migration file cannot have changed → the roll is binary-reversible. - Non-empty → extract both
cepins and diff the migration catalogue:
git show OLD:go.mod | grep selfhosted-cloud/cegit show NEW:go.mod | grep selfhosted-cloud/ce# then, in a ce checkout:git diff --name-status <oldce>..<newce> -- migrations/A changed file there is a one-way boundary: the older build refuses a store the newer file made.
A downgrade across a migration refuses to boot
Section titled “A downgrade across a migration refuses to boot”assertNoDrift in <ce>/core/db/migrate.go still carries this doc comment:
This catches the “daemon downgraded against newer DB” case so the boot fails fast rather than silently running against a schema it does not know.
That check alone would not: its scan is bounded by the old binary’s own
catalogue window, and a migration newer than anything the old binary knows sorts
above it. The guard that does catch it runs first. The migrations subsystem
calls db.AssertLedgerCovered over every catalogue the build ships before any
SQL applies (<ce>/subsystems/impl/migrations.go), and a ledger row no
catalogue ships refuses the boot. A row newer than every file the build ships
reads X409991MIGNEW (“this store’s ledger records N migrations newer than
every file this build ships, …”). A daemon downgraded across a migration
therefore stops at boot with that error. It doesn’t come up and fail later.
That newer row is asked first, before any subsystem starts: shcd reads the
node’s own copy of the store (the raft engine’s rqlite/db.sqlite under the
state home, read only, or the single-node database), prints the one
X409991MIGNEW line and exits 78. shc.service carries
RestartPreventExitStatus=78, so systemctl status shc reads failed with
status=78 and the node stays down until it runs the newer build. A node
whose copy has not yet caught up is refused by the migrations subsystem
instead, once the store is up, with the same line and the same exit.
So the order in this page holds for a reason: restore the control-plane
snapshot taken before the migration, whose ledger the older build covers, then
install the older deb. A database from the old line (main, release/v0.0)
under a new-line daemon is refused too, with X409990MIGRST, and there is no
rollback across the lines in place; see
upgrades.
Still exercise a real read path after any rollback, see verification.
Migrations are raft-replicated
Section titled “Migrations are raft-replicated”In cluster mode the migration runner submits through the rqlite engine’s atomic request executor, so DDL is replicated through raft. Two consequences:
- Downgrading one node’s deb does not put that node on the old schema. The schema is cluster state, not node state.
- Wiping a node and rejoining it replays the migration from the raft log; it does not escape it.
There is no per-node schema. There is one schema, and the roll moved it.
Procedure: roll back one node
Section titled “Procedure: roll back one node”Use this when a single node is misbehaving and the cluster schema has not moved.
# 1. Put the old deb on the nodescp shc_<OLD>_linux_amd64.deb admin@<node>:/tmp/shc-rollback.deb
# 2. Wait out any dpkg lock, then install. dpkg -i accepts an older version# silently; no --force-downgrade is required.ssh admin@<node> ' while sudo fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 3; done sudo dpkg -i /tmp/shc-rollback.deb sudo systemctl restart shc'
# 3. Confirm it answers before touching another nodessh admin@<node> 'curl -s -m 3 -o /dev/null -w "%{http_code}\n" http://localhost:8000/api/capabilities'# any HTTP status means the daemon is alive; the unauthenticated probe# legitimately returns 401, so do not use `curl -f` here
ssh admin@<node> 'shc version'postinst runs systemctl try-restart shc, which acts only on an already-active
unit, and self-heals a node that has node_bootstrap.json but an inactive
daemon. prerm deliberately does nothing on an upgrade or a downgrade (dpkg
passes the action word upgrade for both), so the daemon is never left stopped
by the package itself.
Running containers are not affected. Docker keeps them up and the daemon
reattaches via container labels. The name set from docker ps should be
identical either side of the roll; that is the “installed apps undamaged” proof.
Procedure: roll back an estate
Section titled “Procedure: roll back an estate”Reverse of the roll. Go one node at a time, primary last, and wait for each daemon to answer before touching the next; that is what keeps raft quorum intact.
# node-3, then node-2, then node-1 (primary)for ip in <node3> <node2> <node1>; do scp shc_<OLD>_linux_amd64.deb admin@$ip:/tmp/shc-rollback.deb ssh admin@$ip ' while sudo fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 3; done sudo dpkg -i /tmp/shc-rollback.deb && sudo systemctl restart shc' ssh admin@$ip 'i=0; while [ "$i" -lt 45 ]; do curl -s -m 3 -o /dev/null http://localhost:8000/api/capabilities && exit 0 sleep 2; i=$((i + 1)); done; exit 1' || { echo "$ip did not come back"; break; }doneWith 3 nodes never have more than 1 offline; with 5, at most 2. Leadership moving mid-roll is expected and fine.
The repo already automates exactly this shape for the forward direction, and it
is direction-agnostic: it installs whatever DEB_FILE points at and asserts
every node ends on that deb’s version:
DEB_FILE=/path/to/shc_<OLD>_linux_amd64.deb tests/proxmox/scripts/upgrade.shVerification after a rollback
Section titled “Verification after a rollback”“The daemon started” is not verification on its own: a daemon that boots has passed the ledger check, not a read of your data. Run all of these:
# 1. Every node is on the expected versionfor ip in <nodes>; do ssh admin@$ip 'shc version | head -1'; done
# 2. All nodes healthy, exactly one leadersudo shc node ls
# 3. A real read path through the tables a migration would have renamed.sudo shc lssudo shc status
# 4. App containers never flinchedssh admin@<node> 'sudo docker ps --format "{{.Names}}" | sort' # compare to pre-roll
# 5. A deployment still cyclessudo shc stop <stack> && sudo shc start <stack>
# 6. Ingress still resolvescurl -s -o /dev/null -w '%{http_code}\n' http://<estate>/If shcd does not come back, exited 78 and its journal shows X409991MIGNEW
naming a migration the older build doesn’t ship, you have rolled the binary back across
a migration. The binary is not the problem and reinstalling it again will not
help; go to when the schema moved.
When the schema moved
Section titled “When the schema moved”If the roll crossed a migration, the binary rollback is not sufficient and not
safe on its own. There are no down-migrations: the loader skips .down.sql
and none are authored.
Your options, in order of preference:
- Roll forward instead. Fix the defect on the new version. This is the supported path and the reason the policy is “backup first, forward-only in production”.
- Restore the control-plane snapshot, then install the old deb. This is the only true rollback, and it costs you every control-plane write since the snapshot was taken. Stop the daemon on every node first; the schema is raft state, so a partial restore will be overwritten by the surviving majority.
- Do nothing and escalate. A half-rolled-back estate (old binaries, new schema) is worse than either end state, because it fails late and silently.
Known one-way boundaries
Section titled “Known one-way boundaries”Until v1 the store has one migration file, and every change to it is a one-way
boundary: the older build refuses a store the newer file made with
X409991MIGNEW, so crossing it backwards requires a snapshot restore.
State that does NOT roll back with the binary
Section titled “State that does NOT roll back with the binary”Say this part out loud to whoever authorises the roll, because none of it is covered by reinstalling a deb:
- Sidecar
manifest.jsonadd-on keys.PersistedState.UnmarshalJSONreads both the old and new spellings, but the struct has no retired json tag, so any write emits the new keys only. A singleshc updateunder the new binary permanently strips the old spelling; a rolled-back binary then re-renders that deployment without its add-on, silently. - Terraform provider state.
shc_deploymentv2→v3 andshc_vdeploymentv0→v1. Unlike the daemon, the plugin framework does hard-refuse a stored state version newer than the provider. Oncetofu applyhas run, that state file is one-way: roll the provider back and it will refuse to load the state.
Mixed versions during the roll
Section titled “Mixed versions during the roll”A sequential roll means mixed versions exist mid-flight, so it is worth being precise about what is and is not enforced:
- Nothing refuses a peer on version. There is no version gate on join, no
raft protocol-version check, and no per-node schema version.
RQLITE_VERSIONhas been pinned at 10.2.7 across all recorded history, so there is no raft store-format skew either. - The “version skew across nodes is not supported” line in
upgrades.md is policy, not a mechanism; nothing enforces
it. Compatibility rests entirely on
omitemptyplus strict-decode discipline in the wire types. - Because DDL is raft-replicated, the moment the first upgraded node applies a migration, every node’s schema has moved, including the ones still running the old binary. A mixed-version window across a schema-bearing roll is therefore the dangerous case, and it is dangerous from the first node, not the last.
Keep the window short, and do not pause a schema-bearing roll half-way.
See also
Section titled “See also”- Upgrading SHC: the forward direction
- Disaster recovery: quorum loss, dead leaders
- Refreshing app packaging: why a daemon roll does not touch running apps