Skip to content
SHC Docs

Groups

Layer 6. Services. Services

New line, in development. This page describes SHC as it is on the dev branch, not the released product. See The new line.

A service scales by interchangeable replicas. A group scales by numbered members, and each member is its own set of services with its own names and volumes. Use a group when the copies are not interchangeable: a database with one primary and two secondaries, a quorum, anything where copy 2 holds different data from copy 3.

WordMeaning
groupThe thing you scale, named in the template (rs). --scale names it.
memberOne stamped copy of the group’s services, known by its number (rs member 2). The number never changes while the member exists.
members block{% members %} ... {% endmembers %} inside services:, the only way to write a group. There is no x-shc-groups and no x-shc-member; both are refused as unknown directives (X400330XSHUNK).

The full syntax and every refusal code are in the design, Groups and the members block. This page walks through one example. mongodb ships the block: one member by default, --scale mongodb=N up to seven, an arbiter at two, and its socket fronted by the service selector named for the group. mongodb-replicaset and librechat-db are retired into it (SHC-1792). Checked against ce db676cf4aa0b, pinned by shc 3c88a263038b (core/jinja/members.go, core/pod/members.go, modules/app/groups.go).

# compose.yaml of an app called replicaset (shape only)
services:
{% members "rs", default=3, max=7 as m %}
mongo-{{ m.n }}:
image: percona/percona-server-mongodb:7.0
command: ["mongod", "--replSet", "rs0", "--bind_ip_all"]
volumes:
- data-{{ m.n }}:/data/db
expose:
- "27017"
environment:
MEMBER_LOCATION: "{{ m.location }}"
healthcheck:
test: ["CMD", "mongosh", "--quiet", "--eval", "db.adminCommand('ping').ok"]
interval: 10s
deploy:
resources:
limits:
cpus: "1"
memory: 1G
{% endmembers %}

With no --scale, a first install renders three members, because default=3:

MemberServiceVolume
1mongo-1data-1
2mongo-2data-2
3mongo-3data-3

SHC renders the block’s body once per member, in number order, with m.n the member’s number and m.location the location it was given ("" when the scale names none). It declares data-1, data-2 and data-3 at the top level itself (modules/app/install_runner.go AutoDeclareVolumes), so write no top-level volumes: for them. Each member’s data lives in its own named volume. A durable volume on local disk holds that member on its node; numbering alone does not make data portable.

Auth is left out above to keep the group visible. A real replica set also needs a keyfile and a step that runs rs.initiate and adds each member: that is the app’s work, not the group’s.

What a member is called and how it is reached

Section titled “What a member is called and how it is reached”

A member’s services are ordinary services named as the body wrote them: mongo-2 is member 2’s database.

  • Inside the deployment, any service reaches it by that name: mongo-2:27017.
  • The deployment’s containers, and an integration consumer’s, also reach it by its scoped name, <environment>.<service>.<stack>: prod.mongo-2.mongo-rs for a deployment called mongo-rs in prod. On the shc fabric the node resolver answers that name with the service’s VIP to every asker; off the fabric it’s a network alias SHC puts on each of the service’s network entries (install_runner.go serviceNameAliases).
  • Its full name in the internal zone is mongo-2.mongo-rs.prod.<tenant>.shc.internal (modules/resolver ServiceName), answered by the node resolver, which the render sets as a service’s dns when the node has one. Other tenants are refused, and a service that declares no port is private to its deployment.
  • On a cluster on the shc fabric each service, a member’s included, also gets its own virtual IP (modules/vip Active).
  • shc logs mongo-2 -s mongo-rs and shc restart -s mongo-rs -S mongo-2 act on one member’s service.

A client lists the members it knows, the usual MongoDB way: mongodb://mongo-1:27017,mongo-2:27017,mongo-3:27017/?replicaSet=rs0.

Inside SHC, member 2’s services make up one pod named rs-2 (core/pod/members.go MemberPod). That name is what placement and the replica records use; you never write it. See Replicas and pods for how SHC renders a pod and which nodes run its replica projects.

Scale the group, never its services:

Terminal window
shc install ./replicaset -s mongo-rs --scale rs=5
shc update -s mongo-rs --scale rs=3
  • Numbers are stable. They are a stored list per deployment (core/pod/members.go ResolveMembers). Going from 3 to 5 adds members 4 and 5 and leaves 1 to 3 exactly as they were, with the same rendered services.
  • Going down keeps the lowest numbers: 5 to 3 retires members 5 and 4. Their volumes are kept, and scaling back up brings them back by number.
  • Retiring a member stops its services; what that means to the data is the app’s. An engine that keeps a membership list needs a pre_update hook that removes the retiring members first, or its set keeps a primary only while the running members are a majority of the largest count it ever ran: 5 to 3 is safe, 5 to 2 leaves no primary while every healthcheck passes. mongodb’s handover job does that removal (SHC-1792).
  • max is the ceiling: --scale rs=9 with max=7 is refused (X400093GRPMAX).
  • Only --scale rs=... changes the group. An update that names no value for it keeps every member it has, and raising default in a new app version does not grow a deployment that already runs the group. The exception: a block inside an {% if %} that renders false keeps no member list, so when it renders again the group starts over from its --scale value, else its default (modules/app/groups.go AttributeGroups).
  • --scale mongo-2=... is refused (X400095GRPSVC): scale the group.
  • A member’s service may not set deploy.replicas or scale above 1 (X400099GRPREP). One copy per member for now.
Terminal window
shc update -s mongo-rs --scale rs=2@home,1@far

This holds two members to home and one to far. Members already in a location stay there; new members take the lowest free numbers. A plain count (--scale rs=5) clears every member’s location; to keep them, scale with counts per location. A location that is a level (incus holding incus/home and incus/far) gives each member a leaf under it, spread over its children, and keeps it there: a member holds data and never floats inside a level. A group takes no <n>/<kind> shape (X400094GRPSHP: a count or counts per place, not that value). The member sees its location as m.location, which the example passes to the container.

Placement records a node for every member and prefers a node no other member of the group holds; with too few nodes, members share rather than wait, and SHC reports the sharing as not redundant (core/placement/engine.go SharedGroupMembers). A member with a location is placed only there.

An app whose members must never share a node says so on the block:

{% members "rs", default=3, max=7, separate="required" as m %}

separate is "preferred" (the default) or "required". A required group never puts two members on one node, on any placement: install, update, failover, a move or a rebalance (core/placement/spread.go serviceSpreads, a hard node,max=1 over the group). Only the app declares it; no deployment flag turns it on or off.

  • An install, update or --scale that asks for more members than the cluster has eligible nodes is refused, with both numbers: X409184FNCSEP, “the cluster has fewer eligible nodes (1) than group rs has members (3)”. A single-node estate can’t run a required group of more than one member.
  • With enough nodes but no room on the ones a member could take, the install waits like any other (X409179PLCPND). An update never finishes such a member on the deployment’s own node beside a sibling.
  • A member whose node is lost waits for a node holding no other member. Its failover hold says needs a node holding no other member of rs (separate="required").
  • An update that makes a group required while two of its members share a node is refused (X409185FNCSHR). The error names the members, the node and shc cluster rebalance --apply, which moves them apart. Nothing moves on its own because an app version says so.
  • shc doctor reports a required group’s members sharing a node as an error; a preferred group’s sharing stays a warning.

The render stores the declaration beside the member lists (group_separate in the stored spec’s effective state), and every reader takes both through pod.ReadStoredGroups. A spec stored before SHC knew separate, as every v0.1.21 spec is, reads as preferred.

On a fresh install, admission binds each member to a node. The render finishes that member’s replica project for the bound node’s id, location and runtime, including its fabric pins and deployment networks (modules/app/replica_placement.go, replica_projects.go). A dispatched install carries those bindings too (daemon/services_install_placement.go). The runner on the bound node starts the project and refuses one finished for another node (modules/pods/runner.go). Cross-node member placement is built (SHC-1426).

This requires a Linux node on the shc fabric with service VIPs, or a Mac node on the apple runtime (SHC-1557). A node that declines replica projects runs the deployment’s one compose document instead. shc doctor check pods.replica-projects reports the decision (modules/pods/doctor.go).

An update keeps members where they run. Admission places nothing on an ordinary update, so the render finishes each member for the node its live assignment row binds it to, its highest live incarnation, instead of the node that renders (modules/app/replica_placement.go heldReplicas, SHC-1628). A replica with no row and no binding is finished for the render node. DNS stays the render node’s resolver. Global services separately expand into projects for held bindings and eligible ready nodes with room (expandGlobalReplicas).

Install and update render on the cluster leader (SHC-1618): an update and an install from a catalog or git source are dispatched there (daemon/install_leader.go renderLeaderNode, modules/app/update_dispatch.go updateOnHolder), and an install from a path or an archive renders on its placed node, which holds the bytes (daemon/services_install_placement.go). Every render also stores the deployment’s compose before any of it is finished for a node, and the leader finishes that stored render for another node with nothing rendering there (modules/app/stored_render.go FinishStoredRender, SHC-1616). That path carries a failed-over replica to its new node (daemon/scheduler_deliver.go), a move to its target (FinishStoredMove, SHC-1618) and a node joining a global deployment (finishGlobalRerender, SHC-1146).

A vote outside the members block declares which group it witnesses:

services:
{% members "db", default=2 as m %}
pg-{{ m.n }}:
image: postgres:17
{% endmembers %}
postgres-witness:
image: etcd
x-shc-scale:
witness: db

witness takes one group name declared by this app. A service inside a members block cannot declare it. Lint refuses a missing group or a member declaring a witness, naming the service and group. In shc.yaml it is scale: {witness: db} on the service.

Placement prefers a node holding none of that group’s members when one has room. With only two nodes, or no room on the third, the witness still places and SHC reports the group as not redundant, naming the witness and member sharing a node. The witness does not change how data members spread. Existing bindings stay put until a move or reschedule; the stored spec preserves the declaration for that placement too.

This is an app property. There is no operator flag. Database apps should declare it on postgres-witness, the MongoDB arbiter and the Redis sentinel services, outside their members blocks.

shc status -s <deployment> prints the deployment’s groups under a Groups title, one line per member: group, member, location, ready, the service selector named for the group on the member it points at, and services (modules/health/commands/status_groups.go). They come from shc.status.get’s groups, which the health service stands from the stored spec (app.GroupLayouts) and the replicas of the same answer. A member is ready when every replica it runs as reads ready. When the deployment declares a service selector under the group’s name (x-shc-service-selectors), the section names it on the member it points at: the only member of a one-member group while that member is ready, otherwise the member whose first replica the service selector picks and reads ready, and none while no member or more than one does, which the section says under its table. -o json carries the same under groups; shc status with no deployment named lists no members. shc ls prints no member table, and its -o json keeps each deployment’s groups as the wire carries them.

Every render carries groups.<name>, the group’s members in number order, each with n and location (ce core/jinja/members.go GroupsKey, SHC-1790). It is readable inside the block, outside it, and in the hook jobs, socket jobs and other jobs rendered for the same deployment. A pre_update job sees the list the update is moving to, so on a scale-down it can tell which members are about to retire while they still run. A render with no stored list, such as a lint or a catalogue sweep, sees members 1 to default.

services:
{% members "db", default=1, max=5 as m %}
pg-{{ m.n }}:
image: postgres:17
{% endmembers %}
{% if groups.db|length == 2 %}
witness:
image: etcd
x-shc-scale: {witness: db}
{% endif %}

Here the witness exists only while the group has exactly two members. Keep a member’s own services free of groups: a member whose rendered bytes change with the count is restarted on every scale change, because the node takes a changed pod down before it brings it up (ce modules/pods/runner.go). If a member’s server really has to change when the group gains HA, let it change only at the edge between one member and two. Database apps, HA by scale uses this for every catalogue database.

A group’s own name, and any other name an app wants, is a service selector: a name plus a selector over the labels of the deployment’s replicas (ce modules/vip, SHC-2459). Declare the service selectors at the top level of the compose file and a live label on each member for them to read:

x-shc-service-selectors:
postgres: role=primary:1
postgres-ro: role=replica
services:
{% members "postgres", default=1, max=7 as m %}
postgres-{{ m.n }}:
image: postgres:17
expose:
- "5432"
x-shc-live-labels:
role:
test: ["CMD-SHELL", "psql -U postgres -Atc \"select case when pg_is_in_recovery() then 'replica' else 'primary' end\""]
interval: 5s
timeout: 3s
{% endmembers %}

x-shc-live-labels maps a label key to a command. SHC runs it inside the container every interval, and the first line it prints is the label’s value on that member’s replica: here role=primary or role=replica. A run that exits 0 and prints nothing a label may hold makes the label absent. A run that times out, cannot start or exits nonzero could not ask, and the label keeps its value. interval is 30s unless stated and never under 5s, and timeout 5s unless stated and below the interval. SHC never knows what primary or replica means. The app measures it, and the app’s own service selectors act on it.

x-shc-service-selectors maps a name to a selector in the node selector grammar, with &, |, != and parentheses. It reads service facts only: service, ordinal, group, member, the app’s compose labels and the live labels. location and node are node selector terms and are refused here (X400454ALSDCL). A trailing :1 is a count: exactly one replica may answer (x-shc directives has the whole grammar). Each name answers as one virtual IP on the shc fabric:

  • postgres from any service of the deployment;
  • postgres.<stack>.<environment>.<tenant>.shc.internal from the tenant’s containers;
  • the socket’s hostname from a consumer, when the app’s socket says service: postgres or is named after it.
The service selectorWhere it goes
role=replicaEvery ready replica whose labels match, spread like a service’s replicas.
role=primary:1The one ready replica whose labels match.
no replica matchesNowhere. The connection is refused, as for any VIP with no ready replica, and shc status -s names the service selector and says no replica matches role=primary.
:1, and two replicas matchNowhere, for as long as both do. It never reaches two, and shc status -s names the replicas that both match.

Each member keeps its own name and its own virtual IP beside them. The same shape gives a writer and its read replicas, an etcd or Kafka controller’s leader, or a Redis Sentinel master: the app prints what the member is, and the service selector picks by it. A service selector is not tied to a group. web-canary: service=web & track=canary is one too, where track is a compose label.

What to know before you rely on it:

  • A service selector may have the group’s name, and that is how a socket named after the group follows its primary. It may not have the name of a service or of a member’s pod (postgres-2), and the render refuses one that does (X400454ALSDCL).
  • Name each member’s main service <group>-{{ m.n }}. A member answers a service selector through the service its pod is named for, and a member’s pod is <group>-<n>.
  • A service selector reads every replica of the deployment. If a service outside the group prints the same label, hold the service selector to the group: group=postgres & role=primary:1.
  • Print the label on one plain member too. At one member postgres prints primary whenever it accepts connections, so postgres answers from the first install. A service selector with no match is refused, so a label that only prints once the HA pieces run leaves the service selector refused until they do. At one member postgres-ro has no replica and is refused.
  • A label holds down a change. A new answer is reported once two passes in a row give it, so a value that flaps between passes moves nothing and one failed or slow run takes no label away. A run’s first answer is reported at once. After the engine has moved its primary, expect the service selector to follow within two intervals plus a few seconds. Between the old primary’s last primary and the new one’s first, a service selector with :1 sees two or none and is refused.
  • A node the cluster no longer sees ready counts for nothing. Its replica is not a backend and its last answer does not hold a service selector with :1 closed, so the member the engine promoted takes over.
  • A scale-down that drops the primary closes a service selector with :1 for the length of the update. The update’s member list is in force before pre_update runs. If a member that list dropped still says primary, the service selector is refused from then until the update retires that member, after your hand-over has moved the primary to a kept one. Hand the primary to the lowest member before you scale down if the gap matters.
  • SHC does not cut connections that are already open. The engine does that when it demotes a member.
  • The group has no container. A context: service job names a member’s service (service: postgres-1) and dials a service selector from there.

role is the app’s own label. SHC derives no role, so a backup’s from: and any selector read the value the app printed. A backup that reads one member of a group prefers a member the group-named service selector with :1 does not pick.

  • The body of one pass must read on its own as a mapping of whole services, and the file must read without the block. Write each member’s services whole inside the block. A pass may merge an anchor written outside the block before it (<<: *common); an anchor written inside the block is for that member only (X400097GRPEMT).
  • A member’s service may share a namespace or volumes_from only with services of its own member (X400098GRPLNK).
  • The group’s name may not be a service’s name, and a member pod name (rs-2) may not be a service outside the group (X400096GRPDUP).
  • The block may sit inside an {% if %}, never inside another members block, a {% for %} or a macro, and a group is written once per template (X400512JNJMBR).
  • Replicas inside a member, the read-only copies of one member behind its own virtual IP: the groups design, section 9.
  • Dropping one member by number, and an fqdn() helper: not ruled.

A service stays its replicas plus one VIP. Every other name is a service selector the app declares (above).