Release lines and backports
SHC has two lines (SHC-1093, decided on SHC-1092). release/v0.0 in shc and ce
is the old line. The live estates take debs, providers and the catalog only from
it. main is the new line and versions as 0.1. One .gitlab-ci.yml serves
both, and SHC_LINE in its variables: block says which line a tree is.
What publishes where
Section titled “What publishes where”| Pipeline | Line | Writes |
|---|---|---|
push to release/v0.0, tag v0.0.* | old | generic/*/0.0.0-latest, selfhostedcloud.org/apps/index.yaml and apps/infra/index.yaml, the Pulp copies at downloads.selfhostedcloud.org/shc-apps, every image’s :latest. The site sync writes only the catalog, the provider mirrors and the frozen modules, never the docs. |
push to main with SHC_LINE: '0.1', tag v0.1.* | new | generic/*/0.1.0-latest, selfhostedcloud.org/apps/0.1/, every image’s :0.1.0-latest, snapshots versioned 0.1.0-SNAPSHOT-<sha>, and the docs. The site sync holds back the old line’s paths. No Pulp, no deploy trigger, and no app packages or catalog yet: generic/<app> is one namespace, and the old line’s prune and index would read new-line revisions. |
push to main with SHC_LINE: '0.0' | old | everything, as before the split. Only valid until release/v0.0 exists. |
publish/line-guard runs before publish/host-guard on every publishing
pipeline and refuses one whose resolved names cross lines. It also refuses main
while main still says SHC_LINE: '0.0' and release/v0.0 exists, so a
forgotten flip stops publishing instead of overwriting the estates’ channels.
tests/unit/ciline checks the same rules statically.
Backporting a fix
Section titled “Backporting a fix”-
Fix it on
mainfirst when the code still exists there, through the usuallane/SHC-<n>-<slug>MR. -
After it merges, cherry-pick it onto the old line on a lane named for the same ticket with a
-v0.0suffix, and open the MR againstrelease/v0.0:Terminal window git fetch corpgit worktree add ~/wt/SHC-<n>-<slug>-v0.0 -b lane/SHC-<n>-<slug>-v0.0 corp/release/v0.0cd ~/wt/SHC-<n>-<slug>-v0.0git cherry-pick -x <main-sha>...git push -o merge_request.create -o merge_request.target=release/v0.0 corp lane/SHC-<n>-<slug>-v0.0 -
Code the new line deleted is fixed on
release/v0.0directly, with the same lane naming and target. Say in the MR that there is nothing to fix on main. -
A ce fix goes the same way in ce. On an shc
-v0.0lane,make ce/latestpins the head of ce’srelease/v0.0, never ce main.
Never merge release/v0.0 into main or the reverse. The two lines share
history only up to the cut.
Cutting a line
Section titled “Cutting a line”The CI needs no edit at the cut. The cut is: protect the branch, push it in ce
and shc, move the daily deploy schedule to it, seed the new line’s builder
tags, and merge the one-line flip of SHC_LINE on main. The exact commands for
release/v0.0 are on SHC-1093.
The release gate starts at the landing
Section titled “The release gate starts at the landing”ce’s schedule 26 is the release gate: it runs every suite on ce next’s tip against the shc next commit that pins it. The lane that lands a merge train plays it the moment it declares next green, on the landed pair:
curl -fsS -X POST -H "PRIVATE-TOKEN: $GITLAB_TOKEN" \ https://git.corp.bitspur.com/api/v4/projects/10/pipeline_schedules/26/playThe release lane’s R1 then only confirms that pipeline ran on the final pair, and plays the schedule again if anything landed after it. Started at R1, the gate waited for the release lane to get there: 55 minutes between the landing and the gate for v0.1.18 (SHC-3369).
Past guard, build and lint, the gate runs the Go suites (next/test/shc,
next/test/unit, next/test/ce), make test/scripts’s bats suites
(next/test/scripts) and the image check. R1g stops the cut on any job that
is not green.
App suites on cells from the released deb
Section titled “App suites on cells from the released deb”Every release must run the app suites after R7 reads PUBLISHED and before any
estate rolls to it. The release lane runs them by hand until the new line is
done, then the Mac takes them back as a CI runner (FZ-7289). A not ok row
stops the roll until a fix release or main’s written ruling on that row.
Use the release lane’s existing cell kit to create one docker node and one
incus node, named shc-test-* and labelled. Record their ids in a ledger at
create and reap them by those recorded ids when the suites end.
Install the released deb from Based’s apt repository, never a lane deb, with
Recommends so bats comes with it. A --no-install-recommends node has no bats
and every suite dies at once with X500010BATSNF. The kit’s prep initialises
each node as a single node; then enable the container broker. The suites’
direct rows run in job containers and refuse X403985CBROFF without that
opt-in.
Run the suites on both nodes from a checkout of the release tag. The deb ships
only the built-in postgres app under /usr/lib/shc/apps. Set ver to the
released version and node to the ledger’s node label; keep a TAP file per
suite and node:
apt-get install --install-recommends "shc=1:$ver"shc config set container_broker.enabled trueshc test run /usr/lib/shc/apps/postgres >"postgres-$node.tap"shc test run apps/standalone/mail >"mail-$node.tap"shc test run apps/flagship/mailcow >"mailcow-$node.tap"shc test run apps/flagship/netbird >"netbird-$node.tap"LANG=en_US.UTF-8 shc test run apps/integrated/dependency-track >"dependency-track-$node.tap"Attach every TAP file, the deb version from dpkg-query -W shc, and the node
ids to the release card. Jellyfin needs a GPU node to deploy, so its suite is
outside this set: its rows skip on these nodes, and a skip proves nothing.
These runs prove FZ-6662, FZ-6344, FZ-6410, FZ-6442, FZ-6417 and FZ-6399 at
every release.
An estate’s roll MR moves the provider with the deb
Section titled “An estate’s roll MR moves the provider with the deb”Each estate rolls to a release in one ops MR. That MR moves four things to the release together:
- the deb,
local.versionin the root’sshc.tf; - module core;
- the
selfhosted-cloud/shcprovider pin in the root’sversions.tf; - that provider’s entry in the root’s
.terraform.lock.hcl.
A provider older than the shcd it talks to can misread an answer whose shape
changed, and then every plan of the root fails refresh. Ogden’s v0.1.22 roll
(bitspur/ops!554) moved the deb and module core but held the provider at
0.1.21. Provider 0.1.21 reads shc -o json backup schedule ls as
{rows, refused}, shcd 0.1.22 prints a bare array, and every untargeted plan
failed on the root’s six backup schedules until bitspur/ops!563 moved the pin
(SHC-3667).
A change a release holds back doesn’t hold the provider back with it: the tag
builds its own provider, and the roll takes that one.
Lock the provider’s entry alone. Delete the
provider "registry.opentofu.org/selfhosted-cloud/shc" block from the root’s
.terraform.lock.hcl, then run this from the ops checkout (Ogden’s platform
root here):
export TF_CLI_CONFIG_FILE=$PWD/gorillaservers/ogden/platform/registry.tfrctofu -chdir=gorillaservers/ogden/platform providers lock \ -net-mirror=https://selfhostedcloud.org/terraform/0.1/ \ -platform=darwin_arm64 -platform=linux_amd64 \ registry.opentofu.org/selfhosted-cloud/shcThe lock’s diff is that one entry: the new version and its hashes. With the
old entry still in place, providers lock refuses the new version, and
tofu init -upgrade gets past that only by adding hashes to every other
provider’s entry too.