Skip to content
SHC Docs

Rolling a daemon back

Upgrading SHC covers going forward. This page covers going back, and it exists because the interesting half of “can we roll back?” is not the binary; it is the state the new binary already touched.

The short version:

QuestionAnswer
Does dpkg keep the deb it replaced?No. Nothing retains it. Keep it yourself, before you roll.
Will dpkg -i install an older deb over a newer one?Yes, silently. No --force-downgrade needed.
Do the maintainer scripts fight a downgrade?No. prerm/postinst are direction-agnostic.
Does the daemon refuse to boot on a schema it does not know?Yes. A ledger row newer than every shipped file stops the boot with X409991MIGNEW and exit 78 before any subsystem starts, and the unit does not restart it. See a downgrade across a migration refuses to boot.
Can a schema migration be undone?No. There are zero down-migrations.
So can I roll back?Only within a window that crosses no schema migration. Outside it, the binary goes back and the database does not.

Before you roll forward: the two-minute pre-flight

Section titled “Before you roll forward: the two-minute pre-flight”

Do these before the roll, not after. Both are cheap and both are unrecoverable if skipped.

A node rolled by scp + dpkg -i has no apt repository in the picture. That means:

  • dpkg -i does not populate /var/cache/apt/archives; checked on a live node, there is no shc_*.deb there.
  • dpkg keeps no copy of the payload it replaced.
  • apt-get install --reinstall shc=<oldversion> has nothing to talk to, unless the node reads the Based apt repo and the old version was a release: releases from 0.1.1 on are there as shc=1:<ver>, and every release’s deb stays at https://downloads.selfhostedcloud.org/shc/<ver>/. A snapshot build is in neither.

If you do not save the outgoing deb, your rollback artifact does not exist. Save it off-node:

Terminal window
# On each node, BEFORE installing the new deb
dpkg-query -W -f='${Version}\n' shc # record this

and keep the deb you last installed, plus the CI artifact it came from. Every package/snapshot job publishes dist/shc_*_linux_amd64.deb, so the outgoing version is re-fetchable from its pipeline as long as the artifact has not expired; verify that before relying on it.

1a. Do not read anything into the deb’s version number

Section titled “1a. Do not read anything into the deb’s version number”

Snapshot debs are versioned 1:0.0.<n>~SNAPSHOT-<gitsha>. dpkg splits that at the last hyphen, so the part that actually orders two builds of the same 0.0.<n> line is the git sha, compared as a Debian version string. Git shas are not ordered, so neither are the debs:

Terminal window
$ dpkg --compare-versions '1:0.0.57~SNAPSHOT-5d42fa6ee8a' lt '1:0.0.57~SNAPSHOT-e573822b6ef' && echo lower
lower

That pair is a genuine roll forward (5d42fa6 is 17 commits ahead of e573822), and dpkg considers it a downgrade. The reverse pair would compare the other way purely on the first hex digit.

Consequences:

  • dpkg -i does not care, which is why both directions work. On a genuine version-number downgrade it warns and proceeds. Measured:

    Terminal window
    dpkg: warning: downgrading shc (1:0.0.57~SNAPSHOT-e573822b6ef) to (1:0.0.1~SNAPSHOT-e573822)
    Unpacking shc (1:0.0.1~SNAPSHOT-e573822) over (1:0.0.57~SNAPSHOT-e573822b6ef) ...
    Setting up shc (1:0.0.1~SNAPSHOT-e573822) ...

    No --force-downgrade is needed. Treat the warning as information, not a failure.

  • Anything that does care is unreliable here: apt, unattended-upgrades, --refuse-downgrade, and any script gating on dpkg --compare-versions. Never gate a roll or a rollback on version ordering; gate on the git sha you intended.

  • shc version prints the upstream version with ~ rendered as - and no epoch, so 1:0.0.57~SNAPSHOT-5d42fa6ee8a reports as 0.0.57-SNAPSHOT-5d42fa6ee8a. Normalise both when comparing.

Locally built debs (make deb) use a different upstream version (0.0.1) than CI snapshots (0.0.57), so a CI deb outranks a local one regardless of which is newer in git. Do not mix the two sources across a roll without checking what is actually installed.

2. Confirm the control-plane snapshot is current

Section titled “2. Confirm the control-plane snapshot is current”

This is the only thing that can undo a migration. The leader-only db_snapshot_cron writes one every night at 02:00 UTC to /var/lib/shc/snapshots/control-plane/. shc db snapshot create takes one now, through the same dump into the same directory. Take one before a roll or a hand change to vault rows, placement or users.

Terminal window
shc db snapshot create # on the leader; any other node refuses and names it
shc db snapshot ls # on-demand and nightly snapshots, with time and size

shc db snapshot ls lists the directory of the node it runs on, and a snapshot sits on the node that led when it was taken. Retention is maintenance.control_plane_snapshot_retention (default 7) for each kind on its own: an on-demand snapshot prunes only older on-demand ones, never a nightly, and the nightly never prunes an on-demand one.


Is this roll reversible? Decide before you start

Section titled “Is this roll reversible? Decide before you start”

A roll is reversible by binary alone iff it crosses no migration. The store’s migration lives in the ce module (<ce>/migrations/, one generated file until v1, embedded via migrations/embed.go), and shc takes it through its go.mod pin. So the question reduces to one diff:

Terminal window
# In the shc checkout, OLD = the version currently deployed, NEW = the roll target
git diff OLD..NEW -- go.mod go.sum
  • Empty → the ce pin did not move → the migration file cannot have changed → the roll is binary-reversible.
  • Non-empty → extract both ce pins and diff the migration catalogue:
Terminal window
git show OLD:go.mod | grep selfhosted-cloud/ce
git show NEW:go.mod | grep selfhosted-cloud/ce
# then, in a ce checkout:
git diff --name-status <oldce>..<newce> -- migrations/

A changed file there is a one-way boundary: the older build refuses a store the newer file made.


A downgrade across a migration refuses to boot

Section titled “A downgrade across a migration refuses to boot”

assertNoDrift in <ce>/core/db/migrate.go still carries this doc comment:

This catches the “daemon downgraded against newer DB” case so the boot fails fast rather than silently running against a schema it does not know.

That check alone would not: its scan is bounded by the old binary’s own catalogue window, and a migration newer than anything the old binary knows sorts above it. The guard that does catch it runs first. The migrations subsystem calls db.AssertLedgerCovered over every catalogue the build ships before any SQL applies (<ce>/subsystems/impl/migrations.go), and a ledger row no catalogue ships refuses the boot. A row newer than every file the build ships reads X409991MIGNEW (“this store’s ledger records N migrations newer than every file this build ships, …”). A daemon downgraded across a migration therefore stops at boot with that error. It doesn’t come up and fail later.

That newer row is asked first, before any subsystem starts: shcd reads the node’s own copy of the store (the raft engine’s rqlite/db.sqlite under the state home, read only, or the single-node database), prints the one X409991MIGNEW line and exits 78. shc.service carries RestartPreventExitStatus=78, so systemctl status shc reads failed with status=78 and the node stays down until it runs the newer build. A node whose copy has not yet caught up is refused by the migrations subsystem instead, once the store is up, with the same line and the same exit.

So the order in this page holds for a reason: restore the control-plane snapshot taken before the migration, whose ledger the older build covers, then install the older deb. A database from the old line (main, release/v0.0) under a new-line daemon is refused too, with X409990MIGRST, and there is no rollback across the lines in place; see upgrades.

Still exercise a real read path after any rollback, see verification.


In cluster mode the migration runner submits through the rqlite engine’s atomic request executor, so DDL is replicated through raft. Two consequences:

  • Downgrading one node’s deb does not put that node on the old schema. The schema is cluster state, not node state.
  • Wiping a node and rejoining it replays the migration from the raft log; it does not escape it.

There is no per-node schema. There is one schema, and the roll moved it.


Use this when a single node is misbehaving and the cluster schema has not moved.

Terminal window
# 1. Put the old deb on the node
scp shc_<OLD>_linux_amd64.deb admin@<node>:/tmp/shc-rollback.deb
# 2. Wait out any dpkg lock, then install. dpkg -i accepts an older version
# silently; no --force-downgrade is required.
ssh admin@<node> '
while sudo fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 3; done
sudo dpkg -i /tmp/shc-rollback.deb
sudo systemctl restart shc
'
# 3. Confirm it answers before touching another node
ssh admin@<node> 'curl -s -m 3 -o /dev/null -w "%{http_code}\n" http://localhost:8000/api/capabilities'
# any HTTP status means the daemon is alive; the unauthenticated probe
# legitimately returns 401, so do not use `curl -f` here
ssh admin@<node> 'shc version'

postinst runs systemctl try-restart shc, which acts only on an already-active unit, and self-heals a node that has node_bootstrap.json but an inactive daemon. prerm deliberately does nothing on an upgrade or a downgrade (dpkg passes the action word upgrade for both), so the daemon is never left stopped by the package itself.

Running containers are not affected. Docker keeps them up and the daemon reattaches via container labels. The name set from docker ps should be identical either side of the roll; that is the “installed apps undamaged” proof.

Reverse of the roll. Go one node at a time, primary last, and wait for each daemon to answer before touching the next; that is what keeps raft quorum intact.

Terminal window
# node-3, then node-2, then node-1 (primary)
for ip in <node3> <node2> <node1>; do
scp shc_<OLD>_linux_amd64.deb admin@$ip:/tmp/shc-rollback.deb
ssh admin@$ip '
while sudo fuser /var/lib/dpkg/lock-frontend >/dev/null 2>&1; do sleep 3; done
sudo dpkg -i /tmp/shc-rollback.deb && sudo systemctl restart shc'
ssh admin@$ip 'i=0; while [ "$i" -lt 45 ]; do
curl -s -m 3 -o /dev/null http://localhost:8000/api/capabilities && exit 0
sleep 2; i=$((i + 1)); done; exit 1' || { echo "$ip did not come back"; break; }
done

With 3 nodes never have more than 1 offline; with 5, at most 2. Leadership moving mid-roll is expected and fine.

The repo already automates exactly this shape for the forward direction, and it is direction-agnostic: it installs whatever DEB_FILE points at and asserts every node ends on that deb’s version:

Terminal window
DEB_FILE=/path/to/shc_<OLD>_linux_amd64.deb tests/proxmox/scripts/upgrade.sh

“The daemon started” is not verification on its own: a daemon that boots has passed the ledger check, not a read of your data. Run all of these:

Terminal window
# 1. Every node is on the expected version
for ip in <nodes>; do ssh admin@$ip 'shc version | head -1'; done
# 2. All nodes healthy, exactly one leader
sudo shc node ls
# 3. A real read path through the tables a migration would have renamed.
sudo shc ls
sudo shc status
# 4. App containers never flinched
ssh admin@<node> 'sudo docker ps --format "{{.Names}}" | sort' # compare to pre-roll
# 5. A deployment still cycles
sudo shc stop <stack> && sudo shc start <stack>
# 6. Ingress still resolves
curl -s -o /dev/null -w '%{http_code}\n' http://<estate>/

If shcd does not come back, exited 78 and its journal shows X409991MIGNEW naming a migration the older build doesn’t ship, you have rolled the binary back across a migration. The binary is not the problem and reinstalling it again will not help; go to when the schema moved.


If the roll crossed a migration, the binary rollback is not sufficient and not safe on its own. There are no down-migrations: the loader skips .down.sql and none are authored.

Your options, in order of preference:

  1. Roll forward instead. Fix the defect on the new version. This is the supported path and the reason the policy is “backup first, forward-only in production”.
  2. Restore the control-plane snapshot, then install the old deb. This is the only true rollback, and it costs you every control-plane write since the snapshot was taken. Stop the daemon on every node first; the schema is raft state, so a partial restore will be overwritten by the surviving majority.
  3. Do nothing and escalate. A half-rolled-back estate (old binaries, new schema) is worse than either end state, because it fails late and silently.

Until v1 the store has one migration file, and every change to it is a one-way boundary: the older build refuses a store the newer file made with X409991MIGNEW, so crossing it backwards requires a snapshot restore.

State that does NOT roll back with the binary

Section titled “State that does NOT roll back with the binary”

Say this part out loud to whoever authorises the roll, because none of it is covered by reinstalling a deb:

  • Sidecar manifest.json add-on keys. PersistedState.UnmarshalJSON reads both the old and new spellings, but the struct has no retired json tag, so any write emits the new keys only. A single shc update under the new binary permanently strips the old spelling; a rolled-back binary then re-renders that deployment without its add-on, silently.
  • Terraform provider state. shc_deployment v2→v3 and shc_vdeployment v0→v1. Unlike the daemon, the plugin framework does hard-refuse a stored state version newer than the provider. Once tofu apply has run, that state file is one-way: roll the provider back and it will refuse to load the state.

A sequential roll means mixed versions exist mid-flight, so it is worth being precise about what is and is not enforced:

  • Nothing refuses a peer on version. There is no version gate on join, no raft protocol-version check, and no per-node schema version. RQLITE_VERSION has been pinned at 10.2.7 across all recorded history, so there is no raft store-format skew either.
  • The “version skew across nodes is not supported” line in upgrades.md is policy, not a mechanism; nothing enforces it. Compatibility rests entirely on omitempty plus strict-decode discipline in the wire types.
  • Because DDL is raft-replicated, the moment the first upgraded node applies a migration, every node’s schema has moved, including the ones still running the old binary. A mixed-version window across a schema-bearing roll is therefore the dangerous case, and it is dangerous from the first node, not the last.

Keep the window short, and do not pause a schema-bearing roll half-way.