Skip to content
SHC Docs

Postgres members across locations

The catalogue Postgres can run more than one member: one writes, the others stream from it and can take over. Members are worth having when they sit on different machines in more than one place, so that losing one machine or one place leaves the others. home01 is one machine in one location, so this page adds two: home02 in location home and far01 in location far.

Prepare two more Debian 13 machines the way home01 was prepared, with 2 CPUs, 8 GB of memory and SHC installed, named home02 and far01. They must reach home01 and each other, and root on each must be able to ssh home01 with a key and no prompt: a join reads its token from home01 that way.

Each node joins from its own shell. The join asks home01 for a fresh token over ssh and hands it to shc node join -, which reads it from standard input, so the token is never copied or typed. The join also names the node, its cloud and its location; a join without them takes a random name and whatever the machine’s cloud reports.

home02
$ ssh home01 shc node token --fields token -o text | shc node join - --name home02 --cloud learn --location home
✔ joined cluster as node "home02" at 91.99.184.50
far01
$ ssh home01 shc node token --fields token -o text | shc node join - --name far01 --cloud learn --location far
✔ joined cluster as node "far01" at 65.21.55.73

Run these commands again if a node is still vault-locked or not ready.

Alone, home01 kept the cluster’s database as one SQLite file. When the first node joins, SHC promotes that file into a replicated database and restarts home01’s daemon to do it, within a minute of the join. A command typed on home01 in that moment can fail; type it again.

home01
$ shc node ls
NAME STATE CONDITION AVAIL CLOUD LOCATION ROOM BUILD LEADER VOTER
far01 healthy ready active learn learn/far ok 0.1.0 no no
home01 healthy ready active learn learn/home ok 0.1.0 yes yes
home02 healthy ready active learn learn/home ok 0.1.0 no no
$ shc cluster status
cluster name: shc-cluster
leader: home01
nodes: 3 healthy of 3
voters: 1
quorum: healthy, 1 voter required
warning: only 1 raft voter: losing one stops the control plane committing, and then every deployment with a single-writer volume stops its writer within about a minute, on the surviving nodes too, unless the volume is far, with no control-plane fence node in its reach
warning: control-plane fence has 2 members, fewer than three, so it cannot hold three voters until `shc node update <node> --control-plane` brings a node into it, and the voter count follows
leader-only work, recorded by home01 17s ago:
acme last swept 22s ago
backup_scheduler running 2m22s, no sweep yet
container_reconciler last swept 21s ago
mesh_snapshot_sign last swept 22s ago
operation_registry last swept 22s ago
proxy_reconcile last swept 22s ago
schedule_dispatcher last swept 22s ago
scheduler last swept 17s ago
service_vip_alloc last swept 17s ago

Three nodes in two locations, and one voter. home01 and home02 are the control plane’s fence, and two fence members keep one voter, as the warnings say; the cluster ladder grows the fence to three. The deployments of the earlier pages stay on home01: a node joining moves nothing.

Install the group with --scale:

home01
$ shc install postgres -s members --scale postgres=2@home,1@far
installing deployment "members"
[members] Healthy 6/6
✔ etcd-1 healthy
✔ etcd-2 healthy
✔ etcd-3 healthy
✔ postgres-1 healthy
✔ postgres-2 healthy
✔ postgres-3 healthy
✔ installed deployment "members"
$ shc ls
STACK TENANT ENV APP STATUS NODES
...
members prod prod postgres running far01, home01, home02
...
$ shc exec -s members -S postgres-1 -- psql -U postgres -d postgres -tAc 'select pg_is_in_recovery()'
f
$ shc exec -s members -S postgres-1 -- psql -U postgres -d postgres -tAc 'select application_name, state from pg_stat_replication order by 1'
postgres-2|streaming
postgres-3|streaming
$ shc exec -s members -S postgres-2 -- psql -U postgres -d postgres -tAc 'select pg_is_in_recovery()'
t
$ shc exec -s members -S postgres-3 -- psql -U postgres -d postgres -tAc 'select pg_is_in_recovery()'
t

--scale postgres=3 sets a count without choosing locations. Here 2@home,1@far places two members in learn/home and one in learn/far. The member services are postgres-1, postgres-2 and postgres-3. The service name postgres reaches the writer. shc ls lists the deployment; shc status -s members lists its group, each member with its location, readiness and services, and the member the postgres address points at.

The members come from the catalogue’s compose.yaml. It writes a member’s services once, between members and endmembers, and SHC repeats them for each member:

home01
$ grep -E '^\{% (end)?members|^ [a-z]+-\{\{ m.n \}\}:' /usr/lib/shc/apps/postgres/compose.yaml
{% members "postgres", default=1, max=5 as m %}
postgres-{{ m.n }}:
etcd-{{ m.n }}:
{% endmembers %}

m.n is the member’s number, so member 2 is postgres-2 and etcd-2. default=1 is why db has one member, and max=5 is the most --scale can ask for. The etcd-N services run only from two members on. See Groups.

Read Underneath the members group before going on; it reads what this page left running.

Next: A node failing.