Store replication tiers for clusters of thousands of nodes (design)
Status: accepted design (SHC-2829, MAIN CALL 11:27Z of 10-11; parent SHC-2813 “what breaks first at 1,000 nodes”: the second scale wall). Nothing here builds until v0.1.5 has shipped and Ogden is live with the corp tenant. Then the two cuts that pay at every size go first, then the tiers, each its own layer-2 ticket.
Where the new line is
Section titled “Where the new line is”Every node runs rqlited (shc subsystems/rqlite_engine.go), and shcd talks to its own copy over HTTP. The raft membership is up to 5 voters (VoterTarget, ce modules/cluster/voters.go:31: the largest odd count no greater than the eligible nodes, capped at 5) and every other node is a non-voter. Raft has no relay: the leader sends every log entry to every member itself, voter or not.
shcd uses the store three ways:
- Writes go to the local rqlited, which forwards them to the leader. All writes through one node serialise on one connection (ce
core/dbPool, writeDB MaxOpenConns 1). - Default reads run at rqlite’s weak level, which a follower also forwards to the leader. So a plain read from any node is answered by the leader.
- Local reads (ce
core/db.WithLocalRead) run at level none with a freshness bound, answered from the node’s own copy and refused as stale past the bound (shccore/db/rqlitedriver/local_read.go). Clusterfs is the main user: every node reads the revision counter once a second from its own copy (5 s bound) and rebuilds the config directory when it moves, or every minute (cemodules/clusterfs/loop.go).
Client-side leader discovery is off on purpose (shc core/db/rqlitedriver/discovery.go, defect X1): the driver dials one URL and never asks /nodes, because one blackholed member held every discovery call for 30 s.
What the store carries today
Section titled “What the store carries today”Measured on cell podcfg-m29 (three Scaleway DEV1-L, docker, shc next pinned to ce next 7bdab6c08e, only the system deployments), 600 s per window, read from each node’s rqlited /status, /debug/vars and the raft port’s sockets (lanes/PODCFG/m2829):
| 1 node | 2 nodes | 3 nodes | |
|---|---|---|---|
| raft entries a second | (plain SQLite) | 1.83 | 2.01 |
| bytes an entry | 350 | 354 | |
| writes each follower sends the leader | 0.24/s | 0.23/s | |
| reads each follower sends the leader | 12.4/s | 12.3/s | |
| leader’s raft stream to each follower | 4.4 KB/s | 4.4 KB/s | |
| leader’s answers to each follower’s forwarded calls | 12 KB/s | 14.6 KB/s | |
| store rows changed a minute | 18 | 28 | 36 |
A node with no deployments of its own writes about 0.23 times a second. The code says what that is:
| writer | each node | where |
|---|---|---|
| task worker heartbeat | one entry every 10 s | ce modules/task/worker.go:82, service.go:1382 |
| node-local cron run rows, written at start and end | ~4 a minute (CronHostDNS and CronBackupRecoverySweep each minute) | ce daemon/cron_runs_store.go:59 |
| container freshness touch, batched | one entry every 30 s | ce subsystems/impl/container_recorder_node.go:576 |
| node facts, outside report | every 5 min, every 30 s | ce subsystems/impl/node_self_register.go:551 |
| node lease refresh | none: the leader batches them, at most one row per node per 17.5 s, in one entry a second | ce modules/nodelease/nodelease.go |
On top of that, each node writes one mesh_endpoint row per peer per device every 5 minutes: 8 rows at 2 nodes, 18 at 3, 2N² in all. That is the parent design’s first wall (SHC-2823) and not this design’s, but every number below assumes it is fixed.
The surprise is reads. A follower forwards 12.3 reads a second to the leader at weak level and answers only about 4 a second from its own copy. The task claim poll alone is 2 a second per node (every 500 ms, ce modules/task/worker.go:73), and its SELECT only nominates a candidate the CAS update then claims, so a stale answer costs nothing. The leader’s answers to forwarded calls are three times its raft stream.
The numbers at 100 and 1,000 nodes
Section titled “The numbers at 100 and 1,000 nodes”Entries a second W(N) ≈ 1.5 + 0.23·N, 354 bytes each, plus about 3.7 KB/s of heartbeats and framing per raft stream. Forwarded reads 12.3 a second per node, about 1.2 KB each.
| today, 100 nodes | today, 1,000 nodes | tiers, 100 nodes | tiers, 1,000 nodes | |
|---|---|---|---|---|
| raft streams from the leader | 99 | 999 | 7 | 25 |
| entries a second | 25 | 232 | 25 | 232 |
| raft egress at the leader | 1.3 MB/s | 86 MB/s (690 Mbit/s) | 0.09 MB/s | 2.2 MB/s |
| reads the leader answers | 1,200/s | 12,300/s | own only, ~70/s | own only, ~70/s |
| reads each replica answers | ~600/s | ~600/s |
Today breaks at 1,000 nodes twice over: 690 Mbit/s of raft egress from one node, and 12,300 reads a second through the leader’s one SQLite. At 100 nodes it still holds (1.3 MB/s, 1,200 reads a second), so 50 is a comfortable line rather than a cliff.
Two cuts make the write side smaller still, and the design takes them: rows only their own node reads stay off the replicated store (the cron run rows, written to the node’s own disk), and the task worker heartbeat rides the node lease the leader already refreshes. That takes a node from 0.23 to about 0.06 writes a second, so 1,000 nodes write about 60 entries a second.
The design: three tiers
Section titled “The design: three tiers”A node holds exactly one store role, and the role is a property with its own command (shc node show prints it, shc node update <node> --store voter|replica|client pins it; unpinned, the leader assigns it).
| role | runs rqlited | how many | serves |
|---|---|---|---|
| voter | yes, voting | VoterTarget, at most 5 | raft quorum, and reads like a replica |
| replica | yes, -raft-non-voter | one per store.replica_every nodes (a cluster setting, default 50), at least one per location, 0 below that | its clients’ reads and writes, through store.query and store.execute |
| client | no | everything else | nothing; it calls its replica’s RPCs |
Below 50 nodes nothing changes: every node is a voter or a replica, as today. Past it, the leader’s raft fan-out is voters plus replicas, 5 + ceil(N/50): 7 at 100 nodes, 25 at 1,000, instead of 99 and 999.
Replica slots are voter slots’ twin. The leader already grants voters through unique voter_slot claims (ce modules/cluster/voter_grant.go) and the suffrage reconciler demotes past the cap. Replica slots use the same table shape and the same reconciler: target ceil(N/50), spread so each location holds at least one, a node’s own pin first. A node with no slot is a client.
A client reaches the store through two named internal RPCs on its replica, never raw rqlite HTTP: every node-to-node call is a named RPC, with one auth and one error shape (Clay, 10-01). store.query carries a statement and the caller’s read mark; store.execute carries a statement list. Both are registered and described like every other internal RPC. The replica’s handler runs them on its own rqlited: an execute is forwarded by rqlited to the leader and commits as one raft entry, as the batch executor does now; a query marked WithLocalRead is answered from the replica’s copy at level none under the caller’s freshness bound, and an unmarked one at weak level by the leader. On the client, core/db gets an RPC-backed driver in place of the rqlite one, so nothing above core/db changes: a module can’t tell a client from a replica.
Each client has an ordered list, not a discovery call. The leader writes each client’s list into clusterfs (its home replica, then the other replicas of its location, then the rest by location distance, voters last), so the client reads it from local disk and never asks a member who else exists (X1 stays closed). Home replica choice is by location, then by fewest clients.
Which reads must be local
Section titled “Which reads must be local”A node must keep running what it runs with no store at all. That’s already true: the per-replica specs the leader renders (render once, on the leader) and every module’s config reach a node as clusterfs files on its own disk, and a node serves its containers from current/ whatever the store does. So no read has to be answered from a copy on the node itself, and a client keeps no copy.
What a client loses is cheap reads. The clusterfs counter poll becomes one HTTPS request a second to its replica, answered from the replica’s copy at level none.
The reads that matter are the forwarded ones: 12.3 a second per node, all of which go to the leader today, and would still go to the leader through a replica if nothing else changed. So every periodic read a node makes moves to WithLocalRead with a freshness bound, answered by its replica, unless its correctness needs the leader. A poll whose answer only nominates (the task claim SELECT before its CAS) or only decides when to look again needs no leader. A guard test lists every periodic reader and fails on one that reads at weak level without saying why. Reads a person asks for (shc ls, the console) stay weak and go to the leader; they’re rare.
When the tier is unreachable
Section titled “When the tier is unreachable”A client tries its list in order, each member with a 2 s budget, and stays on the first that answers. It goes back to its home replica after 60 s of that replica answering store.query again, so clients don’t pile onto one member after a blip.
When no member on the list answers, the node is cut off from the store, the same state as a voter on the minority side of a partition today, and it does the same things: containers keep running from current/, writes fail with their coded error and the caller’s retry, leases it holds lapse at their TTL so a lease-gated writer stops where it would today, and the leader’s fence treats the node as lost after the node TTL. Nothing new is invented for the cut-off case.
A replica that falls behind the leader past the freshness bound refuses local reads (“stale read”), and its clients’ local reads move down the list exactly as on any other failure.
How a node changes tier
Section titled “How a node changes tier”Moves are leader-driven, one at a time, and frozen while any store queue reports pressure (the placement freeze ruling).
- client to replica: the leader grants the slot; the node starts rqlited as a non-voter joining the leader, waits until its applied index is within 1 s of the leader’s, then points its own driver at loopback and tells the leader, which writes the new client lists.
- replica to client: the leader writes new client lists without it and waits until no client names it first; the node points its driver at its new home replica, the leader removes it from raft (
DELETE /remove, shccore/rqlited/status.go:175), and the node stops rqlited and keeps its data directory for 24 h before deleting it. - replica to voter and back: unchanged, through the existing voter slots.
A replica lost outright is a slot freed: the reconciler grants a new one in the same location and the clients’ lists already name the next member.
Proving the bound
Section titled “Proving the bound”- A named test on the reconciler: for 1,000 members across 4 locations it grants at most 5 voters and 20 replicas, at least one replica per location, and a raft configuration of 25 members. It fails if anyone raises the ratio or drops the location spread.
- A rig read at 12 nodes with the cluster setting
store.replica_everyat 4 (the same rule at a smaller ratio):/statuson the leader lists 5 + 3 raft peers, its raft port carries 8 streams, and a client’s forwarded reads (proxy.remote_querieson its replica) stay at the leader’s own rate.
Effort, one lane
Section titled “Effort, one lane”| piece | size | notes |
|---|---|---|
| store role property and replica slots | M, 3 to 5 days | store_slot beside voter_slot, the reconciler’s target ceil(N/50) with one per location, shc node update --store, the role in shc node show and shc node ls |
| client mode | M, about 1 week | store.query and store.execute on every voter and replica; the RPC-backed core/db driver on a client, which starts no rqlited; the member list as a clusterfs section; the 2 s per-member budget and the 60 s return home in that driver |
| tier moves | M, about 1 week | client to replica behind the catch-up gate, replica to client behind the list drain and DELETE /remove, both frozen under pressure |
| periodic reads local | M, 3 to 5 days | each periodic reader on WithLocalRead with a freshness bound unless it needs the leader, and the guard test that lists them |
| writes a node makes for itself | S, 2 to 3 days | node-local cron run rows on the node’s own disk, the task worker heartbeat folded into the node lease |
| doctor and pressure | S, 2 days | clients per replica, each replica’s lag behind the leader, a client off its home replica, one pressure queue per replica |
| proof | S, 2 days | the named test and the rig read above |
About four weeks for one lane. The two read and write pieces pay off on their own before any tier exists: they take the leader’s forwarded reads and a node’s writes down at every size. It rides after the endpoint rows go node-local (the first wall), since that cut sets how much the store carries at all.
Decided (MAIN CALL 11:27Z of 10-11)
Section titled “Decided (MAIN CALL 11:27Z of 10-11)”- The tiers as designed: voter, replica and client as a node property with its own command, leader-driven moves one at a time, frozen under pressure.
- Up to 5 voters, and one replica per 50 nodes with at least one per location, as the default of the cluster setting
store.replica_every. - A client reaches the store through a registered internal RPC on its replica, never raw rqlite HTTP.
- The role names stand: voter, replica, client.
- Order: nothing builds until v0.1.5 has shipped and Ogden is live with the corp tenant; then the periodic-reads cut and the node-only-writes cut, then the tiers.