Groups and HA
This page writes a group: copies of a service that are not alike. A service
scales by replicas that are all the same, but a database with a primary and
replicas is not: copy 2 is not copy 3. A group is a {% members %} block
whose body SHC renders once per member, each member with its own number,
services and volumes. The app ladder’s Postgres members across
locations installs the catalogue’s group. Write one.
More nodes
Section titled “More nodes”A group earns its keep only across machines: members on one node all go down with it. This is the first page that needs more than home01, so join three more now.
Prepare three machines like home01: Debian 13, SHC installed, 2 CPUs and
8 GB of memory, a root shell. Name them home02 and home03, in location
home, and far01, in location far. Root on each must be able to
ssh home01 with a key and no prompt: a join reads its token from home01
that way.
Each node joins from its own shell. The join asks home01 for a fresh token
over ssh and hands it to shc node join -, which reads it from standard
input, so the token is never copied or typed. The join also names the node,
its cloud and its location.
$ ssh home01 shc node token --fields token -o text | shc node join - --name home02 --cloud learn --location home✔ joined cluster as node "home02" at 192.168.231.12$ ssh home01 shc node token --fields token -o text | shc node join - --name home03 --cloud learn --location home✔ joined cluster as node "home03" at 192.168.231.13$ ssh home01 shc node token --fields token -o text | shc node join - --name far01 --cloud learn --location far✔ joined cluster as node "far01" at 192.168.231.14The cluster takes a few minutes to settle. The first join moves home01’s store to the replicated one and restarts its daemon, and the two new home nodes become voters about a minute apart. Run these again until they read like this.
$ shc node lsNAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD STORE LEADER VOTERfar01 healthy ready active learn learn/far ok 0.1.47 replica no nohome01 healthy ready active learn learn/home ok 0.1.47 voter yes yeshome02 healthy ready active learn learn/home ok 0.1.47 voter no yeshome03 healthy ready active learn learn/home ok 0.1.47 voter no yes$ shc cluster statuscluster name: shc-clusterleader: home01nodes: 4 healthy of 4voters: 3quorum: healthy, 2 voters requiredleader-only work, recorded by home01 19s ago: acme last swept 28s ago app_refinish last swept 58s ago backup_scheduler running 10m58s, no sweep yet container_reconciler last swept 57s ago mesh_snapshot_sign last swept 28s ago operation_registry last swept 28s ago proxy_reconcile last swept 56s ago schedule_dispatcher last swept 28s ago scheduler last swept 23s ago service_vip_alloc last swept 23s agoThree voters, so the cluster now survives losing one node. Nothing you
installed moved: an app moves when a rule or a command says so. The vault’s
cold start, which home01 has held alone since the vault page, can move now
too. coldstart replant hands that duty to another node.
$ shc vault coldstart replant --node home02...$ shc vault coldstart replant --node home01...A group
Section titled “A group”scores is a Redis replica set: member 1 is the primary and every other
member replicates it. Write it on home01 as a compose file.
name: scoresversion: "1.0.0"title: Scoresdescription: A Redis replica set and a witnesscategory: demolicense: id: MITx-shc-service-selectors: kv: role=master:1services:{% members "kv", default=3, max=5 as m %} kv-{{ m.n }}: image: redis:7-alpine command: ["sh", "-c", "if [ {{ m.n }} = 1 ]; then exec redis-server --appendonly yes; else exec redis-server --appendonly yes --replicaof kv-1 6379; fi"] expose: - "6379" volumes: - data-{{ m.n }}:/data environment: MEMBER_LOCATION: "{{ m.location }}" x-shc-live-labels: role: test: ["CMD-SHELL", "redis-cli role | head -n 1"] interval: 5s timeout: 3s{% endmembers %} watcher: image: redis:7-alpine command: ["sh", "-c", "while true; do redis-cli -h kv role | head -n 1; sleep 10; done"] x-shc-scale: witness: kv{% members "kv", default=3, max=5 as m %}is groupkv, three members unless the install says otherwise and never more than five.m.nis the member’s number andm.locationthe location it was given.data-{{ m.n }}is one volume per member, which SHC declares itself.x-shc-live-labelsrunsredis-cli rolein each member every 5 seconds and keeps the first line it prints as that member’s labelrole:masterorslave. SHC does nothing with a label by itself. You read it, and so do a node selector and a service selector. A few label names are SHC’s own,serviceandnodeamong them, and a live label that names one is refused.x-shc-service-selectors, at the top of the file, is the names the deployment answers under besides its services’ own. Each is a name and a selector over the labels of the deployment’s replicas.kv: role=master:1sends the namekvto the member whose label readsmaster. The:1means exactly one: with no master, or with two,kvanswers nothing rather than reach the wrong member. Without:1a name goes to every member that matches, sokv-ro: role=slavewould spread over the replicas.x-shc-scale.witnessasks forwatcheron a node that holds no member ofkv, so losing one node never takes a member and the witness together.
Every rule of the block is in Groups.
Install it with a count per location: --scale kv=2@home,1@far puts two
members in home and one in far. A bare --scale kv=3 would set the count
and leave the places to SHC.
$ shc validate ~/learn/scores✔ validated "/root/learn/scores"$ shc install ~/learn/scores -s scores --scale kv=2@home,1@farinstalling deployment "scores"[scores] Healthy 4/4✔ kv-1 healthy✔ kv-2 healthy✔ kv-3 healthy✔ watcher healthy✔ installed deployment "scores"shc status -s scores shows the node each replica runs on, and its Groups
section lists the group’s members with the location each was given and the
member the service selector kv points at. kv-3 is on far01, the only node in far,
the witness is on the home node that holds no member, and each member carries
its role label: master on kv-1, slave on the others. Its Service selectors
section shows the service selector kv with the replica it points at.
$ shc status -s scores...prod scores prod kv-1-r0 home02 ready role=masterprod scores prod kv-2-r0 home03 ready role=slaveprod scores prod kv-3-r0 far01 ready role=slaveprod scores prod watcher-r0 home01 ready -...GroupsGROUP MEMBER LOCATION READY ALIAS SERVICESkv 1 learn/home ready kv kv-1kv 2 learn/home ready kv-2kv 3 learn/far ready kv-3...Write on member 1 and read it from member 2 and through the service selector kv.
$ shc exec -s scores -S kv-1 -- redis-cli set top 100OK$ shc exec -s scores -S kv-2 -- redis-cli get top100$ shc exec -s scores -S watcher -- redis-cli -h kv get top100$ shc logs watcher -s scores...Scale the group, never its services. A member keeps its number, and a member scaled away keeps its volume until it comes back. Put a file on kv-3’s volume, then scale the far member away.
$ shc exec -s scores -S kv-3 -- touch /data/mark$ shc update scores --scale kv=2@home --yes...$ shc status -s scores...GroupsGROUP MEMBER LOCATION READY ALIAS SERVICESkv 1 learn/home ready kv kv-1kv 2 learn/home ready kv-2...Two members, both in home. Bring the far one back.
$ shc update scores --scale kv=2@home,1@far --yes...$ shc status -s scores...prod scores prod kv-3-r0-i1 far01 ready role=slave...$ shc exec -s scores -S kv-3 -- ls /data/mark/data/markkv-3 is back on far01 with its number and its volume: the file is still
there. The -i1 on its name says this is the replica’s second making.