Skip to content
SHC Docs

Upgrade

The cluster record and every shc_node carry package, and its version is the shc release the node installs. Changing it plans an in-place upgrade. The provider installs the new version by apt on each node over SSH and holds it there, one node at a time. The leader goes last, whichever node leads when the roll starts, so the cluster elects a new leader once, at the end. After each node the roll waits until that node reads healthy at the new build, quorum is healthy and shc doctor run shows no new failure. Only then does the next node restart, so quorum holds throughout.

The cluster runs 0.1.2. Change shc_version to 0.1.3, the release of the modules and the provider.

~/learn/cluster/terraform.tfvars
home_region = "nbg1"
far_region = "hel1"
home_count = 4
server_type = "cx23"
shc_version = "0.1.3"
work
$ tofu -chdir=learn/cluster plan
...
# module.shc.module.core[0].shc_cluster.main[0] will be updated in-place
...
# module.shc.module.core[0].shc_license.estate[0] will be updated in-place
...
# module.shc.module.core[0].shc_node.bootstrap[0] will be updated in-place
...
# module.shc.module.core[0].shc_node.workers["far01.learn"] will be updated in-place
...
# module.shc.module.core[0].shc_node.workers["home02.learn"] will be updated in-place
...
# module.shc.module.core[0].shc_node.workers["home03.learn"] will be updated in-place
...
# module.shc.module.core[0].shc_node.workers["home04.learn"] will be updated in-place
...
Plan: 0 to add, 7 to change, 0 to destroy.
...

No replacement: a node keeps its identity, its disk and its place in the cluster. The cluster record and the five nodes change their package. The licence is in the plan because it reads the bootstrap node’s endpoint, which is known only after the apply; it does not change, so the apply counts six. Apply and the roll runs.

work
$ tofu -chdir=learn/cluster apply -auto-approve
...
Apply complete! Resources: 0 added, 6 changed, 0 destroyed.
...
work
$ ssh -F learn/cluster/ssh/config home01 sudo shc version
shc version 0.1.3
...
$ ssh -F learn/cluster/ssh/config far01 sudo shc version
shc version 0.1.3
...
$ ssh -F learn/cluster/ssh/config home01 sudo shc node ls --fields name,state,condition,build,voter
NAME STATE CONDITION BUILD VOTER
far01.learn healthy ready 0.1.3 no
home01.learn healthy ready 0.1.3 yes
home02.learn healthy ready 0.1.3 yes
home03.learn healthy ready 0.1.3 yes
home04.learn healthy ready 0.1.3 no
$ ssh -F learn/cluster/ssh/config home01 apt-mark showhold
shc

home01 led when the roll began, so it restarted last and one of the other two voters leads now. Which one won the election varies from run to run.

work
$ ssh -F learn/cluster/ssh/config home01 sudo shc cluster status
cluster name: shc-cluster
leader: home02.learn
nodes: 5 healthy of 5
voters: 3
quorum: healthy, 2 voters required
leader-only work, recorded by home02.learn 34s ago:
...

Every node runs 0.1.3, apt holds shc there so an apt upgrade never moves it, and the cluster never lost quorum. The provider’s log shows the order and each node’s health gate: run the apply with TF_LOG_PROVIDER=INFO and look for shc upgrade lines with phase=roll-order and phase=gate-pass. A node that fails its gate stops the roll there, and the apply names it and the check it failed. A node that answers over SSH but stays not ready for ten minutes fails the upgrade; a node that stops answering SSH while its daemon restarts gets twenty minutes before the roll reports that it could not confirm the upgrade. Both budgets are package attributes, ready_timeout and transport_timeout.

Next: Destroy.