Skip to content
SHC Docs

Failover delivery

Layer 11. Lifecycle. Layer page.

Failover delivery is the leader finishing a deployment’s stored render for the node a bind named. The node does not render.

sequenceDiagram
participant Leader
participant Store
participant Node
Leader->>Store: take the deployment relocation lease
Leader->>Leader: FinishStoredRender for the bound node
Leader->>Store: store the spec and bind replica rows
Node->>Store: materialise clusterfs
Node->>Node: start replicas from the stored spec
Leader->>Store: anchor the deployment on the node

The daemon wires that delivery in ce daemon/scheduler_deliver.go. deliverStoredRender holds the relocation lease, calls FinishStoredRender, and then anchors the deployment on that node (deployment_node) while the replica is still bound to it at the generation the leader read. FinishStoredRender is refused off the leader (ce modules/app/stored_render.go).

The scheduler tick, while this node leads, converges deployment bindings and replica rows from the stored spec, then runs registered passes (ce subsystems/impl/scheduler.go). On this pin nothing calls scheduler.Deliver or Store.WriteHolds. The delivery function is wired. The words a failover pass would write are defined (ce modules/scheduler/holds.go) and read onto listing rows (ce modules/app/failover_hold.go).

Stacks.Undelivered is a deployment whose replica is bound away from the node that holds its bytes, with no move under way (ce modules/scheduler/stack.go). StaleCopies is a node whose lease is not lost and which still holds containers after the deployment is bound and recorded elsewhere and runs there: every replica row its operator wants running reads ready or completed. Until then the copy is kept, so a replacement that never starts leaves the deployment failed on its new node with the old copy still up. A lost node’s copy is not read.

Ruled, not on this pin: an automatic volume-failover engine. ce modules/storage/volume_failover_edition.go says this build ships none. shc volume rebind is the operator move that remains.

Sources: ce daemon/scheduler_deliver.go, modules/app/stored_render.go, subsystems/impl/scheduler.go, modules/scheduler/stack.go, modules/scheduler/holds.go, modules/storage/volume_failover_edition.go.