Skip to content
SHC Docs

Internal RPC reference

Generated file. Do not edit by hand. Regenerate from the repo root with make docs/generate.

A call one node makes to another is a named function, sent as POST /internal/rpc/{function} on the internal mTLS listener with a JSON body {"args": [], "kwargs": {...}}.

A request refused on its shape or its caller gets a real HTTP status: 403 on the wrong listener or for a capability the caller lacks, 400 with no function name or a body that cannot be read or parsed, 413 for a body over the size limit. Every other answer is HTTP 200, {"result": ...} or a coded error the caller raises again as the same typed error. That covers what the function answers and four refusals made before it runs: a function name nobody registered, a node already serving its limit of calls from this peer, staged blobs past the function’s transfer bound, and a binary argument that cannot be resolved, whether its reference is malformed or its transfer cannot be pulled.

Callers go through internal_rpc.Call, never a hand-built request, so every call gets the per-peer breaker, the busy backoff and the byte transfer.

Arguments are the keys of kwargs. A blob is binary data: small ones ride the call, large ones are staged and pulled as raw ranged reads of GET /internal/transfer/{id}, the one route under RPC. Each function bounds the bytes of staged blobs one call may make the callee pull, 4 MiB unless a blob’s notes say otherwise. The callee checks the sizes the call declares before it pulls anything and refuses a call past the bound with X413147RPCBND.

Not everything is a function. The live streams (log follow, interactive exec, port-forward) keep WebSocket routes, because a function answers once. Enrol keeps its route. tests/unit/internalroutes pins that whole set of routes.

Total: 166 functions.

Runs, on this node, which holds the deployment directory, the post hooks the deployment’s last update left for its next start because they needed services its operator had stopped (FZ-6993), once the deployment reads ready.

Arguments:

KeyTypeRequiredNotes
tenantstringyesthe deployment’s tenant
stackstringyesthe deployment’s name
environmentstringyesthe deployment’s environment

Result:

KeyTypeNotes
replystringa DeferredHookReport as JSON, null when the update left nothing

Answers whether this node’s disk holds the deployment’s directory by the uninstall’s test: any directory, a vdeployment’s parent’s. A stat that fails other than absent is an error, never an answer. With it, whether the directory holds a render or only a replica’s mounts, and the render mark it carries, which tells the current render’s directory from an older one’s.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstringyes
envstringthe environment

Result:

KeyTypeNotes
presentbool
renderedboolthe directory holds a render (its manifest), not only the mounts a replica’s volumes need
markstringthe digest of the render the directory’s state is; empty for a directory no render marked

Runs the deploy half of an install on this node, the one the deployment was placed on.

Arguments:

KeyTypeRequiredNotes
requestInstallRequestyesthe install as the public API received it
request.tenant_namestring
request.app_namestringyes
request.app_versionstring
request.stack_namestring
request.environmentstring
request.node_idstring
request.repo_namestring
request.variablesmap[string]string
request.remove_vars[]string
request.integrations[]string
request.profiles[]string
request.wherestring
request.preferstring
request.where_services[]string
request.prefer_services[]string
request.connections[]string
request.config[]string
request.vappstring
request.resource_overridesmap[string]any
request.hosts[]string
request.ports[]string
request.dns_providerstring
request.storagestring
request.volume_storagemap[string]VolumeStorage
request.acme_providerstring
request.domainstring
request.naming_exceptionstring
request.runtimestring
request.operation_idstring
request.no_rollbackbool
request.no_waitbool
request.scale_overridesmap[string]string
request.addonstring
request.addon_versionstring
request.accessstring
request.exposurestring
request.sourceSourceData
request.app_archivestring
request.tokenstring
request.authstring
request.within_tenantbool
request.ttlstring
request.forcebool
request.protectedbool
request.restorebool
request.installed_bystring
request.labels[]string
seeded_integrationsboolthe asking node already seeded the integrations
input_var_keys[]stringoperator variable keys before defaults and persisted overrides are folded, empty for platform writes
held_bindings_knownbooltrue when held_bindings is what the deployment held before this install; without it held_bindings is not read
held_bindings[]PersistedIntegrationthe bindings the deployment held before this install
held_bindings[].plugstring
held_bindings[].provider_stack_namestring
held_bindings[].provider_tenant_namestring
held_bindings[].provider_environmentstring
held_bindings[].provider_connectionstring
step_nodestringthe node that handed the deploy here and follows its operation; each step is sent to its app.operation_step
step_tokenstringthe relay that node opened for this deploy; without both no step is sent

Result:

KeyTypeNotes
deployedbool
settledstringthe deployment’s status once the deploy has settled
settle_errorstringa coded error envelope as JSON, when the status settled on a failure

Answers whether this node’s filesystem can hard-cap a deployment’s node-disk volume directory, which need not exist yet. The render asks the node holding a volume before it gives the volume a size. Writes nothing.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstringyes
envstringthe environment
volumestringyesthe volume’s name, its directory under the deployment’s mounts

Result:

KeyTypeNotes
hardboolthe filesystem refuses writes past a cap SHC sets on the directory
filesystemstring
whystringwhy the cap would not be hard; empty when it would

Changes the system OpenObserve’s admin password on this node, which runs OpenObserve: the live password through its users API, then the vault record, then the deployment’s update. A node whose host cannot reach the store’s service address hands the change here. The change goes on after the caller stops waiting.

Arguments:

KeyTypeRequiredNotes
passwordstringthe new admin password; empty generates one

Result:

KeyTypeNotes
replystringas JSON: result (password when generated, generated, redeployed), or refusal (code, message, kwargs, details) for a refusal that carries details, which an RPC error does not

Takes one step of a deploy this node handed to another node, for the operation that is followed here. The node running the deploy sends its steps one at a time and in order, and answers its own call after the last.

Arguments:

KeyTypeRequiredNotes
tokenstringyesthe relay this node opened for the deploy
stepstringyesthe step, a DeploymentPayload as JSON

Result:

KeyTypeNotes
takenboolfalse when the relay is closed, which ends the sender’s relay and nothing else

Runs a whole uninstall on this node, which holds the deployment directory: the safety backup, the teardown and the row’s purge. A second uninstall of the same deployment is refused with X409949UNIRUN while this one runs.

Arguments:

KeyTypeRequiredNotes
requeststringyesthe UninstallRequest, as JSON; stack_name must be set
runstringyesthe uninstall’s operation, the node holding that operation and the node running it, as JSON: operation, origin, holder, detach
step_nodestringthe node that sent the uninstall here and follows its operation; each step, and each progress line of the safety backup, is sent to its app.uninstall_step
step_tokenstringthe relay that node opened for this uninstall; without both no step is sent

Result:

KeyTypeNotes
outcomestringas JSON: retained, the cleanup jobs the teardown left behind; unasked_nodes; purged, the rows the deployment owned that the uninstall removed

Takes this node’s copy of a deployment down and removes its directory, data included, for an uninstall that ran on the node holding the deployment. Local only: no hook, unintegration job or DNS, credential or role release runs. Refused X409194CPYLIV unless the deployment’s row is stamped removing and names another node, or names this node and this node holds no render of the deployment. A directory with a volume still mounted under mounts/ is kept and refused. A node that holds no directory of the deployment takes down the replica projects it runs of it, and answers once they are down.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstringyesa deployment’s name; a vdeployment is refused
envstringthe environment

Result:

KeyTypeNotes
removedbool

Takes one step of an uninstall this node sent to the node holding the deployment, or one progress line of its safety backup, for the operation that is followed here. The holder sends them one at a time and in order.

Arguments:

KeyTypeRequiredNotes
tokenstringyesthe relay this node opened for the uninstall
stepstringthe step’s name, as the uninstall emits it; empty for a progress line
percentintthe step’s percent
messagestringthe step’s words
detailstringa progress line of the safety backup: what it has done so far, in words

Result:

KeyTypeNotes
takenboolfalse when the relay is closed, which ends the sender’s relay and nothing else

Runs an update’s deploy on this node, which holds the deployment directory, and settles the deployment’s status after it.

Arguments:

KeyTypeRequiredNotes
requeststringyesthe UpdateRequest, as JSON
currentstringthe deployment’s row, as JSON
empty_mapsstringcomma-separated names of override maps that were empty rather than absent, which JSON does not carry
held_bindingsstringthe bindings the deployment held before this update, a list of PersistedIntegration as JSON; sending it marks them known
placement_nodestringthe node a move’s or a rebuild’s re-render places the deployment on
step_nodestringthe node that handed the deploy here and follows its operation; each step is sent to its app.operation_step
step_tokenstringthe relay that node opened for this deploy; without both no step is sent

Result:

KeyTypeNotes
replystringan UpdateDeployReply, as JSON: effect, settled, settle_error

Resolves the pin a bare or latest update moves to on this node, which holds the deployment directory. Writes nothing.

Arguments:

KeyTypeRequiredNotes
requeststringyesthe UpdateRequest, as JSON
currentstringyesthe deployment’s row, as JSON

Result:

KeyTypeNotes
pinnedboolfalse when the row ends with no pin
app_versionstringthe pin, empty when pinned is false

Plans an update on this node, the render leader, and writes nothing to the deployment: what the update would change.

Arguments:

KeyTypeRequiredNotes
requeststringyesthe UpdateRequest, as JSON
shipped_app_digeststringthe app archive digest of a path deployment’s folder, interned by the node the plan was typed on; the plan renders it instead of resolving the path

Result:

KeyTypeNotes
planstringan UpdatePlan, as JSON

Starts the leader’s forward-auth redirect-uri restamp for a follower whose completed-login miss detector fired; a node that does not lead runs nothing and relays nothing.

Arguments:

KeyTypeRequiredNotes
reasonstringthe trigger the follower saw, completed_login_miss

Result:

KeyTypeNotes
acceptedboolfalse when this node does not lead

Answers whether this node’s capture of a deployment’s named mounts is one atomic snapshot, which a group backup tells the member’s quiesce before it fires.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstringyes
envstringthe environment
capture_mountsarrayyesthe names the capture reads

Result:

KeyTypeNotes
atomicbooltrue when one snapshot here covers every named mount

Runs a backup converter the caller rendered on this node, the one that holds the mounts it claims, over a static copy made under this node’s own freeze of the deployment, and answers its artifact as a staged archive.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstring
envstringthe environment
converterstringyesthe converter’s name
renderedstringyesthe converter as JSON, rendered by the caller; this node renders nothing
modestringthe run’s mode: fail aborts the copy when the freeze outlasts timeouts.quiesce_hold, anything else stops the deployment here and copies it halted
backup_idstringthe run’s id, for the converter, the stop journal and the journal’s lines
unfrozenboolthe caller froze what is read by other means: nothing is paused here

Result:

KeyTypeNotes
outcomestringproduced, not_armed or lost; the caller reads anything else as lost
errorstringwhy the converter was not armed or was lost
existsbool
archiveblobthe converter’s output as a tar, when produced
databasesstringa JSON list of the databases the artifact carries, when the converter names them
quiescestringhow this node froze the deployment for the copy: assured, unsupported or failed; the caller reads a missing one as failed
left_pausedboolthe release could not confirm every container it froze is running again
thaw_revivedstringcomma-separated replicas that read stopped after the thaw and were started again
thaw_downstringcomma-separated replicas that read stopped after the thaw and are not running
stack_stoppedboolthe freeze outlasted its hold and the deployment was stopped here for the copy
stack_stop_secondsfloat64how long that stop kept it down; 0 while its start has not verified

Runs an operator’s backup create on this node, which holds the deployment directory, and answers the archive it recorded.

Arguments:

KeyTypeRequiredNotes
requeststringyesthe BackupCreate, as JSON; stack_name must be set
relative_pathsstringa JSON list of capture paths relative to the deployment directory, which this node roots in its own
relative_excludesstringa JSON list of exclusions relative to the deployment directory

Result:

KeyTypeNotes
backupstringthe recorded BackupModel, as JSON; for a one-off, the archive’s model, which no row records
archivebloba one-off’s .bak, which the caller stages for its export; absent otherwise

Answers whether this node holds the deployment’s directory, by the same test the scheduler applies to its own disk.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstringyes
envstringthe environment

Result:

KeyTypeNotes
presentbool

Exports this node’s copy of a backup and answers it as a staged archive the caller pulls through the transfer layer.

Arguments:

KeyTypeRequiredNotes
backup_idstringyes

Result:

KeyTypeNotes
archiveblob

Answers a backup create --file archive this node stages for its export: its tenant and stack, or with take the archive itself, which then leaves this node.

Arguments:

KeyTypeRequiredNotes
backup_idstringyes
takeboolhand the archive over instead of describing it

Result:

KeyTypeNotes
tenantstringwithout take
stackstringwithout take
environmentstringwithout take
archiveblobwith take

Runs an orchestrated backup on this node, which holds the deployment directory, and records the archive. A directory missing here answers X500466BKPPLC.

Arguments:

KeyTypeRequiredNotes
requeststringyesthe CreateBackupRequest, as JSON; stack_name must be set

Result:

KeyTypeNotes
backupstringthe recorded BackupModel, as JSON

Restores the group member of a deployment this node holds: stops its replicas here, replays each directory, empties the held ones the archive does not carry, and starts them again.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstring
envstringthe environment
strategystringreplace (the default), merge or skip
archivesobjecteach ordinal directory’s tree as a tar blob, by directory name
ownersstringJSON: per directory, the owners and modes the archive captured
empty[]stringheld ordinal directories the archive does not carry

Result:

KeyTypeNotes
successbool

Runs a recover another node placed on this one (SHC-2584): receives its archives into this node’s spool, points the request at them and runs the whole recover here, data replay included.

Arguments:

KeyTypeRequiredNotes
requeststringJSON: the recover request as the placing node received it, its archives named by their paths there
archivesobjecteach archive the request names as a blob, by its path on the placing node

Result:

KeyTypeNotes
resultstringJSON: the recover’s result

Answers which replicas of a deployment this node holds and what each binds under the deployment’s mounts directory, read from the replicas’ own documents.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstringyes
envstringthe environment

Result:

KeyTypeNotes
replicasstringa JSON list of {service, ordinal, mounts}; empty when this node holds no replica of it

Runs an operator’s restore on this node, which holds the deployment directory: stops its replicas here, replays, routes a group member another node holds to that node, and answers what it restored.

Arguments:

KeyTypeRequiredNotes
requeststringyesthe RestoreRequest, as JSON; stack_name must be set

Result:

KeyTypeNotes
outcomestringthe RestoreOutcome, as JSON

Restores a deployment this node holds the directory of from a .bak another node shipped: spools the archive and runs the same restore from file a node holding it runs, and answers the backup it recorded.

Arguments:

KeyTypeRequiredNotes
requeststringyesthe RestoreFromFileRequest without its path, as JSON; stack_name must be set
pathstringthe archive’s path on the sending node, which the backup row names once this node’s spool copy is gone
archiveblobyesthe .bak; only the transfer store’s admission bounds its size

Result:

KeyTypeNotes
backupstringthe recorded BackupModel, as JSON

Runs one backup-schedule operation on the leader for a follower that was asked for it.

Arguments:

KeyTypeRequiredNotes
opstringyesthe operation, one of the schedule ops in modules/backup/schedule_leader_forward.go
paramsobjectthat operation’s own arguments

Returns: what the named operation answers on the leader.

Runs a whole orchestrated backup of a deployment on this node, which holds it, for a scheduler on another node.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstringyes
envstringthe environment
consistency_idstringgroups this capture with others taken together
namestringthe backup’s name
scheduledbooltrue when a schedule fired it
pre_uninstallbooltake an uninstall’s safety backup instead: name, consistency_id and scheduled are not read, and the answer is captured and verdict

Result:

KeyTypeNotes
capturedbool
backup_idstringpresent when a backup row was written
statusstringthat row’s status
verdictstringwith pre_uninstall, in place of backup_id and status: an UninstallBackupVerdict, as JSON

Holds, confirms or releases a stopped deployment’s starts on this node while a backup reads it as a crash image, and answers whether its containers here are stopped.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstringyes
envstringyesthe environment
opstringyeshold, confirm or release
hold_idstringon hold, the caller’s request id, so a repeat takes no second hold; on confirm and release, the id the hold call answered
markstringon confirm, the mark hold answered; a different one fails the confirmation

Result:

KeyTypeNotes
hold_idstring
markstringwhat identifies this node’s stopped containers of the deployment
stoppedboolevery container of the deployment here is stopped; on confirm, also that the hold held and the mark matches

Verifies a backup against this node’s own disk, for a node asked about an archive whose repository or file is a path on this node. It asks no other node.

Arguments:

KeyTypeRequiredNotes
backup_idstringyes
deepboolread the snapshot’s bytes back

Result:

KeyTypeNotes
reportstringthe VerifyReport, as JSON

Answers the last deep-verify sweep report this node holds, for a node whose backup.verify_sweep reads the newest one in the cluster. It asks no other node.

Result:

KeyTypeNotes
pathstringwhere this node keeps the report
reportstringthe VerifySweepReport, as JSON; absent when no sweep has run on this node
errorstringwhy the report could not be read; absent when it could

Takes no arguments.

Tars a mount of a deployment on this node, where its volume is attached or its replica runs, under this node’s own freeze of the deployment, and answers it as a staged archive.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstring
envstringthe environment
mountstringyesthe mount’s name
modestringthe run’s mode: fail aborts the capture when the freeze outlasts timeouts.quiesce_hold, anything else stops the deployment here and captures it halted
backup_idstringthe run’s id, for the stop journal and the journal’s lines
unfrozenboolthe caller froze what is read by other means: nothing is paused here

Result:

KeyTypeNotes
existsbool
archiveblob
quiescestringhow this node froze the deployment for the capture: assured, unsupported or failed; the caller reads a missing one as failed
left_pausedboolthe release could not confirm every container it froze is running again
thaw_revivedstringcomma-separated replicas that read stopped after the thaw and were started again
thaw_downstringcomma-separated replicas that read stopped after the thaw and are not running
stack_stoppedboolthe freeze outlasted its hold and the deployment was stopped here for the capture
stack_stop_secondsfloat64how long that stop kept it down; 0 while its start has not verified

Unpacks a restored archive into a volume-backed mount on this node, where the volume is attached.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstring
envstringthe environment
mountstringyesthe mount’s name
archiveblobyesthe volume’s tree as a tar; only the transfer store’s admission bounds its size

Result:

KeyTypeNotes
successbool

Answers the certificate deliveries on this node’s disk, one per granted host of the deployments it runs, as their source, classes, issuer and expiry, never key material.

Result:

KeyTypeNotes
entriesarraythe rows shc.cert.delivery lists for this node

Takes no arguments.

Answers this node’s cluster-mTLS leaf (node.pem) as its fingerprint and expiry, never key material.

Result:

KeyTypeNotes
fingerprintstring
not_afterstringRFC 3339

Takes no arguments.

Lists this node’s certificate files with their presence, expiry, subject, issuing key id and fingerprints.

Result:

KeyTypeNotes
entries[]CertEntry
entries[].namestring
entries[].pathstring
entries[].presentbool
entries[].not_afterstring
entries[].days_remainingint
entries[].subjectstring
entries[].authority_key_idstring
entries[].cert_fingerprints[]string

Takes no arguments.

Makes this node renew its own internal-listener leaf now and reload it, the step a CA rotation fans out to every node.

Result:

KeyTypeNotes
successbool
messagestring

Takes no arguments.

Signs a fresh leaf for the calling node against the current cluster CA; the caller is the node named by the certificate the call arrived on.

Result:

KeyTypeNotes
ca_certstring
node_certstring
node_keystring

Takes no arguments.

Runs a clone on this node, the source deployment’s home, for the node that was asked for it.

Arguments:

KeyTypeRequiredNotes
requestStackCloneRequestyesthe clone as the public API received it
request.source_stackstringyes
request.target_namestringyes
request.environmentstring
request.wherestring
request.preferstring
request.source_environmentstring
request.source_tenantstring
request.tenant_namestring
request.all_connectionsbool
request.closurebool
request.rename_mapmap[string]string
request.dry_runbool
request.operation_idstring
request.runtimestring
request.storagestring
callerobjectthe asking user’s scope, so the clone runs as them
caller.within_tenantbool
caller.installed_bystring

Result:

KeyTypeNotes
successbool
source_scopeScopeInfo
source_scope.tenant_namestring
source_scope.environmentstring
source_scope.nodestring
target_scopeScopeInfo
target_scope.tenant_namestring
target_scope.environmentstring
target_scope.nodestring
stacks_cloned[]StackRenameInfo
stacks_cloned[].source_namestring
stacks_cloned[].target_namestring
backup_ids[]string
warnings[]string

Finishes a move’s stored render on the leader under the executor’s relocation lease.

Arguments:

KeyTypeRequiredNotes
deploymentDeploymentRefyesthe deployment being relocated
deployment.Tenantstring
deployment.Environmentstring
deployment.Stackstring
nodestringyesthe target node
relocationarrayyesthe executor’s relocation claims

Result:

KeyTypeNotes
finishedbool

Has the leader hold a moved deployment’s committed render, put back from the store.

Arguments:

KeyTypeRequiredNotes
deploymentDeploymentRefyesthe deployment that moved
deployment.Tenantstring
deployment.Environmentstring
deployment.Stackstring

Result:

KeyTypeNotes
heldbool

Measures a deployment’s tree on this node, or one of its mounts: its bytes and file count.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstringyes
environmentstring
mountstringone mount’s name; empty measures the whole tree

Result:

KeyTypeNotes
bytesint64
filesint

Runs a move on this node, the source deployment’s home, for the node that was asked for it.

Arguments:

KeyTypeRequiredNotes
requestStackMoveRequestyesthe move as the public API received it
request.source_stackstringyes
request.target_namestringyes
request.nodestring
request.wherestring
request.preferstring
request.where_services[]string
request.prefer_services[]string
request.environmentstring
request.source_environmentstring
request.source_tenantstring
request.tenant_namestring
request.all_connectionsbool
request.closurebool
request.rename_mapmap[string]string
request.dry_runbool
request.operation_idstring
request.runtimestring
request.storagestring
request.expect_source_nodestring
request.planned_bystring
request.presumed_deadbool
request.surgebool
request.downtime_secondsint64
request.no_backupbool
request.restore_from_backupbool
request.reclaimstring
callerobjectthe asking user’s scope, so the move runs as them
caller.within_tenantbool
caller.installed_bystring

Result:

KeyTypeNotes
successbool
source_scopeScopeInfo
source_scope.tenant_namestring
source_scope.environmentstring
source_scope.nodestring
target_scopeScopeInfo
target_scope.tenant_namestring
target_scope.environmentstring
target_scope.nodestring
stacks_moved[]StackRenameInfo
stacks_moved[].source_namestring
stacks_moved[].target_namestring
backup_ids[]string
used_fast_pathbool
warnings[]string
strategystring
volumes[]RelocationVolume
volumes[].servicestring
volumes[].namestring
volumes[].durablebool
volumes[].actionstring
volumes[].bytesint64
volumes[].measuredbool

Restores the archive the call carries on this node as the named deployment, renders it and brings it up. The archive is pulled before the restore starts; a pull that fails restores nothing.

Arguments:

KeyTypeRequiredNotes
archiveblobyesthe backup archive; only the transfer store’s admission bounds its size
tenantstring
stackstringthe target deployment’s name
environmentstring
runtimestring
storagestring
recreate_on_startbool
relocation[]objectthe relocation claims this restore runs under
relocation[].deploymentstringyes
relocation[].holderstringyes
package_within_tenantboolwhether the source renders a package from inside the tenant; absent means the sender did not say
caller_within_tenantboolthe asking user’s scope, so the restore runs as them
caller_installed_bystringthe asking user
stoppedboolthe deployment was stopped before the move began; the restore lands it and starts nothing
finish_storedboolthe deployment moves keeping its identity: the leader finishes its stored render for this node, which holds the restored mounts and replicas and no render

Result:

KeyTypeNotes
successbool

Lands the deployment tree the call carries on this node, its new home. The tree is pulled before anything lands; a pull that fails lands nothing and the answer says so.

Arguments:

KeyTypeRequiredNotes
archiveblobyesthe packed deployment tree; only the transfer store’s admission bounds its size
tenantstring
stackstring
environmentstring
runtimestring
stated_nodestringthe node the caller believes this is
present[]stringthe mounts the move relies on finding on this node; the landing refuses while one holds nothing here
stoppedboolthe deployment was stopped before the move began; the landing starts nothing

Result:

KeyTypeNotes
successbool

Renders a cloned, recovered or rehomed deployment on the leader for the node it lands on, from the tree its restore laid there.

Arguments:

KeyTypeRequiredNotes
renderclonedRenderyesthe render, as the landing node would have run it
render.tenantstring
render.stackstring
render.environmentstring
render.app_namestring
render.runtimestring
render.nodestring
render.placement_nodestring
render.hosts[]string
render.accessstring
render.exposurestring
render.held[]PersistedIntegration
render.held_knownbool
render.storagestring
treeblobyesthe landing node’s deployment directory, less its mounts, mark and seal
relocationarrayyesthe landing node’s relocation claims
callerobjectthe asking user’s scope, so the render runs as them

Result:

KeyTypeNotes
renderedbool
boundarray

Copies the stopped replica’s durable local volumes and carried bind directories through the clone blob transfer plane to fresh labelled target volumes and marked target directories.

Arguments:

KeyTypeRequiredNotes
planobjectyes
targetstringyes
generationuint64yes
operationstringyes
claimsarrayyes

Result:

KeyTypeNotes
successbool

Sets this node’s directory of a deployment aside as

.moved-<operation_id>, where shc reclaim on this node lists it, once a move to another name has taken its row. It renames and never removes.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstringyes
environmentstring
operation_idstringyesthe move’s id, one path element

Result:

KeyTypeNotes
pathstringwhere the tree was kept; empty when this node held nothing to keep

Answers the last lines of a service’s log on this node, its log file or else its unit’s journal: the read shc cluster logs <node> routes to the node it names. No lines is X404024CLSLOG naming this node.

Arguments:

KeyTypeRequiredNotes
servicestring
linesint

Result:

KeyTypeNotes
servicestring
nodestring
lines[]string

Answers this node’s attached networks and routes, its underlay, and the bridges its runtime lists as SHC’s own: what shc cluster block add checks a block against on every node.

Result:

KeyTypeNotes
networks[]EnrollNetwork
networks[].interfacestring
networks[].cidrstring
networks[].ownbool
networks[].routebool
networks[].viastring

Takes no arguments.

Answers whether this node is the raft leader, its raft term and the era and generation of the signed membership table it holds. Served only on the peer status port (TCP 4004 at the node’s underlay address, mTLS with a node leaf under the cluster CA, each source rate-limited), never on the internal listener, so a node held off mesh0 can ask before shc cluster repair rebuilds its mesh0 cache. A caller whose leaf names no node of this node’s table is refused.

Result:

KeyTypeNotes
leaderbool
termuint64
membershipMembershipGeneration
membership.erauint64
membership.generationuint64

Takes no arguments.

Answers the raft leader and membership as this node’s engine sees them, the view a store client reads from its list.

Result:

KeyTypeNotes
leader_idstring
leader_addrstring
members[]RaftViewMember
members[].idstring
members[].addressstring
members[].rolestring
members[].reachablebool
members[].reachability_knownbool

Takes no arguments.

Rewrites this node’s cluster_join.json so its next start runs no rqlited and reaches the store through the members sent, in order. A refusal is success=false, never an error.

Arguments:

KeyTypeRequiredNotes
members[]Member
members[].nodestring
members[].addrstring

Result:

KeyTypeNotes
successbool
changedbool
messagestring

Rewrites this node’s cluster_join.json so its next start runs rqlited as a non-voter joining the addresses sent. A refusal is success=false, never an error.

Arguments:

KeyTypeRequiredNotes
join_addrs[]string

Result:

KeyTypeNotes
successbool
changedbool
messagestring

Answers the store role this node’s engine runs: voter, replica or client, empty before the engine has started.

Result:

KeyTypeNotes
tierstring

Takes no arguments.

Rewrites this node’s cluster_join.json so its next raft start joins as a non-voter. A refusal is success=false, never an error.

Arguments:

KeyTypeRequiredNotes
join_addrs[]string

Result:

KeyTypeNotes
successbool
changedbool
messagestring

Schedules the raft engine restart that applies a rewritten join marker, and answers before the restart lands.

Result:

KeyTypeNotes
successbool
changedbool
messagestring

Takes no arguments.

Rewrites this node’s cluster_join.json so its next raft start joins as a voter. A refusal is success=false, never an error.

Arguments:

KeyTypeRequiredNotes
join_addrs[]string

Result:

KeyTypeNotes
successbool
changedbool
messagestring

Wakes this node’s clusterfs revision loop in the background; concurrent wakes coalesce.

Takes no arguments.

Creates an empty file inside a container.

Arguments:

KeyTypeRequiredNotes
container_idstringyes
pathstringyesabsolute path inside the container
modeuint32permission bits

Result:

KeyTypeNotes
okbool

Removes a path inside a container: a file, or a directory with everything under it.

Arguments:

KeyTypeRequiredNotes
container_idstringyes
pathstringyesabsolute path inside the container

Result:

KeyTypeNotes
okbool

Creates a directory inside a container.

Arguments:

KeyTypeRequiredNotes
container_idstringyes
pathstringyesabsolute path inside the container
modeuint32permission bits

Result:

KeyTypeNotes
okbool

Reads a byte range of a file inside a container. A read past the inline limit streams through the transfer layer without buffering.

Arguments:

KeyTypeRequiredNotes
container_idstringyes
pathstringyesabsolute path inside the container
offsetint64
lengthint64

Result:

KeyTypeNotes
datablob

Lists a directory inside a container on this node.

Arguments:

KeyTypeRequiredNotes
container_idstringyes
pathstringyesabsolute path inside the container

Result:

KeyTypeNotes
entries[]FSDirEntry
entries[].namestring
entries[].typestring
entries[].sizeint64
entries[].modeuint32

Renames a path inside a container.

Arguments:

KeyTypeRequiredNotes
container_idstringyes
old_pathstringyesabsolute
new_pathstringyesabsolute

Result:

KeyTypeNotes
okbool

Changes a path’s mode, its owner or its group inside a container. Each of mode, uid and gid is applied when sent.

Arguments:

KeyTypeRequiredNotes
container_idstringyes
pathstringyesabsolute path inside the container
modeuint32permission bits
uiduint32the owner, alone or with gid
giduint32the group, alone or with uid

Result:

KeyTypeNotes
okbool

Stats one path inside a container on this node.

Arguments:

KeyTypeRequiredNotes
container_idstringyes
pathstringyesabsolute path inside the container

Result:

KeyTypeNotes
typestring
sizeint64
modeuint32
mtimeint64
uiduint32
giduint32
symlink_targetstring

Creates a symlink inside a container.

Arguments:

KeyTypeRequiredNotes
container_idstringyes
link_pathstringyesabsolute
targetstringyes

Result:

KeyTypeNotes
okbool

Truncates a file inside a container to a length.

Arguments:

KeyTypeRequiredNotes
container_idstringyes
pathstringyesabsolute path inside the container
lengthint64yes

Result:

KeyTypeNotes
okbool

Writes bytes into a file inside a container at an offset, streaming the blob onto the write.

Arguments:

KeyTypeRequiredNotes
container_idstringyes
pathstringyesabsolute path inside the container
offsetint64
datablobyes32 MiB at most; a larger write is several calls at increasing offsets

Result:

KeyTypeNotes
writtenint64

Restarts a container on this node after inspecting its deployment labels and checking the node agent’s stopped hold.

Arguments:

KeyTypeRequiredNotes
container_idstringyesthe container on this node

Result:

KeyTypeNotes
Successbool
ExitCodeint
Stdoutstring
Stderrstring
StdoutBytesbytes
StderrBytesbytes
Replicas[]string
Members[]string
Waitingbool

Runs a broker container for a deployment on this node, which holds one of its replicas, for the node that was asked for it, through the same gates as a local run.

Arguments:

KeyTypeRequiredNotes
requeststringyesthe RunRequest, as JSON: stack, environment, tenant, image, command, env and timeout_seconds

Result:

KeyTypeNotes
truncatedbool
exit_codeint64
outputstring
handlestring
statusstring

Answers the leader’s cron ledger, each leader-gated cron’s last fire, for a follower’s shc cron ls.

Result:

KeyTypeNotes
runsarrayone object per cron: module, name, last_run_at (RFC 3339), last_status, last_error

Takes no arguments.

Runs this node’s node-scoped doctor checks for the leader’s condition sweep and answers their results.

Result:

KeyTypeNotes
resultsarrayone doctor result per node-scoped check: name, category, verdict and findings

Takes no arguments.

Runs this node’s node-scoped doctor checks, diagnose-only, for shc doctor run on another node and answers their results.

Arguments:

KeyTypeRequiredNotes
categorystringrun only the checks in this category; empty runs every node-scoped check

Result:

KeyTypeNotes
resultsarrayone doctor result per node-scoped check: name, category, verdict, findings and the fix it carries

Answers the edge CA anchors and the hosts they cover from the node that runs the ingress. A node that holds none answers an empty object.

Result:

KeyTypeNotes
anchors_pemstring
hosts[]string

Takes no arguments.

Hands this node an event another node published, for its live stream and broadcast handlers. An event from this node itself is dropped.

Arguments:

KeyTypeRequiredNotes
eventstringyesthe event as a JSON string, so its payload is never read as a binary reference
source_hostnamestringyesthe node that published it
source_node_idstringthe id of the node that published it, which decides whether it is this node’s own

Result:

KeyTypeNotes
deliveredbool

Answers the frames this node published to its own bus alone (healthcheck probes, coalesced occurrences) after a cursor, in a stream’s scope, so the node serving the stream can merge them.

Arguments:

KeyTypeRequiredNotes
cursorint64the cursor the last answer gave; negative answers the ring’s end and no frames
limitint64at most this many frames, capped at 500
recentboolanswer the newest limit frames instead, for a bounded reading
tenantstringthe stream’s tenant
all_tenantsboolthe stream reads every tenant
stack_scopedboolthe caller may read only stack_names
stack_namesstringcomma-separated stacks the caller may read
stack_namestringthe stream’s stack
nodestringthe stream’s node, by name
codestringthe one event code the stream keeps
typesstringcomma-separated event classes
errors_onlyboolerror frames only
quietboolleave out the exec probe triplet

Result:

KeyTypeNotes
eventsstring
cursorint64
missedint64
source_hostnamestring

Runs one rendered export or import job on this node, which holds the replica the job’s service runs in, over a directory this node makes for the run, and answers the job’s outcome.

Arguments:

KeyTypeRequiredNotes
jobstringyesthe rendered ExporterJobDefinition, as JSON, with @output and @input left for this node to resolve
directionstringyesexport or import
tenantstringyes
stackstringyes
environmentstringyes
external_networksstringa JSON list of the integration networks the deployment’s render names
inputblobwith import: the archive of what the job reads from @input
outputblobwith export: the archive of what the run’s earlier jobs wrote to @output, when they wrote anything

Result:

KeyTypeNotes
statusstringsuccess or failed
exit_codeint
stdoutstring
stderrstring
errorstring
outputblobwith export, when the job succeeded: the archive of @output as the job left it

Runs an operator’s export or import on this node, which holds the deployment’s render. An export keeps its artifact in this node’s own artifacts directory and answers the result and the artifact; an import reads the artifact that came with the call, or this node’s own artifacts directory when none came.

Arguments:

KeyTypeRequiredNotes
directionstringyesexport or import
requeststringyesthe ExportRequest or ImportRequest, as JSON, held to the HTTP door’s validation; output_dir, input_dir and wrap are not read
tenantstringthe tenant the operator’s request was scoped to
artifactblobwith import: the .exp the operator named on the node typed on

Result:

KeyTypeNotes
resultstringthe ExportResult or ImportResult, as JSON
namestringwhen an export succeeded: the artifact’s file name
artifactblobwhen an export succeeded: the artifact

Runs a fence reconcile pass on the leader for a follower that was asked for one.

Result:

KeyTypeNotes
reconciledbool

Takes no arguments.

Answers the leader’s node ledger, each row saying which node observed it, for a follower to display.

Result:

KeyTypeNotes
observed_bystring
last_pollTime
nodesmap[string]NodeHealth
nodes{}.namestring
nodes{}.statestring
nodes{}.cpu_percentfloat64
nodes{}.memory_percentfloat64
nodes{}.disk_percentfloat64
nodes{}.inodes_percentfloat64
nodes{}.disk_pathstring
nodes{}.disk_total_bytesuint64
nodes{}.load_averagefloat64
nodes{}.last_checkTime
nodes{}.docker_okbool
nodes{}.disk_checksmap[string]float64
nodes{}.fitness_issuestring
nodes{}.certificateNodeCertHealth
nodes{}.certificate.not_afterTime
nodes{}.certificate.read_errorstring
nodes{}.certificate.renew_duebool
nodes{}.certificate.consecutive_renew_failuresint
nodes{}.certificate.last_renew_errorstring
nodes{}.certificate.last_renew_successTime
nodes{}.certificate.renewer_runningbool
nodes{}.certificate.last_renew_checkTime
nodes{}.certificate_issuestring
nodes{}.pressure[]Sample
nodes{}.pressure[].queuestring
nodes{}.pressure[].unitstring
nodes{}.pressure[].depthint64
nodes{}.pressure[].capacityint64
nodes{}.pressure[].oldest_ageint64
nodes{}.pressure[].in_per_secondfloat64
nodes{}.pressure[].out_per_secondfloat64
nodes{}.pressure[].drops_per_secondfloat64
nodes{}.pressure[].in_totaluint64
nodes{}.pressure[].out_totaluint64
nodes{}.pressure[].dropped_totaluint64
nodes{}.pressure[].leveluint8
nodes{}.pressure[].sinceTime
nodes{}.pressure[].reasonstring
nodes{}.pressure[].errorstring
nodes{}.pressure[].peerstring
nodes{}.pressure[].retired_totaluint64
nodes{}.pressure[].retired_reasonstring
nodes{}.pressure[].stuck_forint64
nodes{}.pressure[].still_forint64
nodes{}.pressure[].stalebool
nodes{}.pressure[].amber_depthint64
nodes{}.pressure[].red_depthint64
nodes{}.pressure[].amber_ageint64
nodes{}.pressure[].red_ageint64
nodes{}.pressure[].atTime
nodes{}.unmeasured_facts[]string

Takes no arguments.

Runs a leader-resolved hook job on this replica’s node, without rendering its definition.

Arguments:

KeyTypeRequiredNotes
specResolvedJobSpecyesthe resolved job
spec.Namestring
spec.Kindstring
spec.Contextstring
spec.Imagestring
spec.Servicestring
spec.Commandstring
spec.Argv[]string
spec.Env[]string
spec.Userstring
spec.WorkingDirstring
spec.Tenantstring
spec.Stackstring
spec.Environmentstring
spec.Runtimestring
spec.TargetDeploymentRef
spec.OwnerDeploymentRef
spec.NamesDeploymentDeploymentRef
spec.FabricFabricRenderInput
spec.Networks[]string
spec.Volumes[]string
spec.MountsRootstring
spec.VolumePrep[]PrepOp
spec.Entrypoint[]string
spec.ServiceDocumentbytes
spec.ServiceEntrystring
spec.TimeoutSecondsint
spec.AdmitSecondsint
spec.Reach[]string
spec.Preparedbool
skip_empty_serviceboolskip a pre-update service job when this node reports no container; a count error still runs it

Result:

KeyTypeNotes
stdoutstringthe job’s stdout, streamed back to the leader
stderrstring
skippedbool
errorobjecta coded envelope when the job failed
failurestringan integration job’s failure with no code of its own, which the asking runner’s code carries

Runs a socket’s service job inside its provider’s running container on this node, the node that runs the provider’s service.

Arguments:

KeyTypeRequiredNotes
specResolvedJobSpecyesthe resolved job; Target names the provider
spec.Namestring
spec.Kindstring
spec.Contextstring
spec.Imagestring
spec.Servicestring
spec.Commandstring
spec.Argv[]string
spec.Env[]string
spec.Userstring
spec.WorkingDirstring
spec.Tenantstring
spec.Stackstring
spec.Environmentstring
spec.Runtimestring
spec.TargetDeploymentRef
spec.OwnerDeploymentRef
spec.NamesDeploymentDeploymentRef
spec.FabricFabricRenderInput
spec.Networks[]string
spec.Volumes[]string
spec.MountsRootstring
spec.VolumePrep[]PrepOp
spec.Entrypoint[]string
spec.ServiceDocumentbytes
spec.ServiceEntrystring
spec.TimeoutSecondsint
spec.AdmitSecondsint
spec.Reach[]string
spec.Preparedbool

Result:

KeyTypeNotes
stdoutstringthe job’s stdout, streamed back to the node that asked
stderrstring
skippedboolalways false
errorobjecta coded envelope when the job failed, X501060SVCNCT when the provider has no running container here
failurestringan integration job’s failure with no code of its own, which the asking runner’s code carries

Readies this replica’s node for a leader-resolved hook job held at its phase’s gate: the job’s preparation and nothing after it, so its run later spends only its command.

Arguments:

KeyTypeRequiredNotes
specResolvedJobSpecyesthe resolved job
spec.Namestring
spec.Kindstring
spec.Contextstring
spec.Imagestring
spec.Servicestring
spec.Commandstring
spec.Argv[]string
spec.Env[]string
spec.Userstring
spec.WorkingDirstring
spec.Tenantstring
spec.Stackstring
spec.Environmentstring
spec.Runtimestring
spec.TargetDeploymentRef
spec.OwnerDeploymentRef
spec.NamesDeploymentDeploymentRef
spec.FabricFabricRenderInput
spec.Networks[]string
spec.Volumes[]string
spec.MountsRootstring
spec.VolumePrep[]PrepOp
spec.Entrypoint[]string
spec.ServiceDocumentbytes
spec.ServiceEntrystring
spec.TimeoutSecondsint
spec.AdmitSecondsint
spec.Reach[]string
spec.Preparedbool
skip_empty_serviceboolanswer skipped when this node reports no container for the service

Result:

KeyTypeNotes
skippedbool
errorobjecta coded envelope when the preparation failed

Registers the jobs declared by this node’s deployment in the shared job table.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstringyes
environmentstring

Result:

KeyTypeNotes
registeredint

Answers a snapshot of a deployment’s logs from this node’s containers, merged by timestamp, for display.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
environmentstringyes
stackstringyes
servicestringone service, or all when empty
runtimestring
sincestring
timestampsbool
tailintlines from the end

Returns: the log text, a string.

On the leader, previews the peer table a confirmed restore would sign, or confirms one: preview asks for the plan, and from_era is the era the plan named. A follower does not run it.

Arguments:

KeyTypeRequiredNotes
previewbool
from_erauint64

Result:

KeyTypeNotes
current_erauint64
new_erauint64
generationuint64
mesh_nodesint
clientsint
grantsint
return[]string
vanish[]string
clients_return[]string
clients_vanish[]string
unanswered[]string
unverified[]string

On the leader, signs the peer table now and returns it as JSON text. A follower that answers enrol asks for this, because only the leader holds the snapshot signer. The table is text so its bytes are not lifted into RPC blobs.

Result:

KeyTypeNotes
snapshotstring

Takes no arguments.

Returns this node’s held leader-signed peer table and fresh mesh0 endpoints to an authenticated member. Served only through the public join route on HTTPS 443; the join route signs the answer.

Arguments:

KeyTypeRequiredNotes
requestobjectyesclustersign envelope over node_id, issued_at and a 32-character lowercase hex nonce, signed by the node identity key under shc-mesh-peer-table-request-v1 followed by a newline
chainstringyesthe caller’s node.pem certificate chain

Returns: a join-signed envelope containing snapshot (the held SignedSnapshot) and endpoints (at most 1024 fresh mesh0 rows).

The pair owner coordinates a direct endpoint trial through its applied relay. The relay requires fresh kernel handshakes to both ends, stamps their observed mappings and forwards the owner proof. Each end updates only its direct peer endpoint at the requested instant; relay AllowedIPs remain until the existing direct-handshake streak releases them.

Arguments:

KeyTypeRequiredNotes
devicestringyesmesh0 or shc0
astringyeslower node id, the owner
cstringyeshigher node id
relaystringyes
generationint64yesthe currently applied relay generation
atint64yesunix milliseconds; at most 10 seconds ahead or 30 seconds old
issued_atint64yesunix seconds; owner proof freshness is two minutes
signedstringyesbase64 owner signature, context shc mesh punch v1
certstringyesbase64 DER owner certificate
endpoint_astringstamped by the relay on the forwarded call
endpoint_cstringstamped by the relay on the forwarded call

Result:

KeyTypeNotes
atint64
endpoint_astring
endpoint_cstring

Sets or releases the relay of one pair of nodes on a device. The end with the lower id signs the decision and sends it to the relay, which checks it reaches both ends, records the pair and passes the call on to the other end. An end applies a strictly higher generation and runs its sweep, and the same generation renews the relay it holds: the end that is not the owner goes direct once a relay has gone unrenewed for the relay lease (5 minutes), as the relay stops forwarding it. The answer is the generation the node holds for the pair.

Arguments:

KeyTypeRequiredNotes
devicestringyesmesh0 or shc0
astringyesone end’s node id
cstringyesthe other end’s node id
relaystringthe relay’s node id; empty releases the pair
generationint64yes
issued_atint64yesunix seconds; refused when more than two minutes from the node’s clock
viastringthe relay the call travels through
signedstringyesbase64 signature of the lower end over the decision
certstringyesbase64 DER certificate of the signer

Result:

KeyTypeNotes
generationint64

Aborts the caller node’s open mesh0 epoch without changing its active key.

Arguments:

KeyTypeRequiredNotes
node_idstring
devicestring
epochint
pending_public_keystring
key_digeststring
bystring

Acknowledges the exact pending key in an applied signed table as the caller’s node leaf.

Arguments:

KeyTypeRequiredNotes
node_idstring
devicestring
epochint
pending_public_keystring
key_digeststring
bystring

Records this authenticated node’s prepared mesh0 key on the leader.

Arguments:

KeyTypeRequiredNotes
node_idstring
devicestring
epochint
pending_public_keystring
key_digeststring
bystring

Relays an operator rotation request to the leader.

Arguments:

KeyTypeRequiredNotes
node_idstring

The leader asks the target to persist its next key and prepare a rotation. The target answers once the rotation has ended: its key is active, or a voter never installed the pending key and nothing switched.

Result:

KeyTypeNotes
node_idstring
devicestring
epochint
public_keystring
preparedbool

Takes no arguments.

Answers with the signed peer table this node holds. The asker sends the table it holds (held, era, generation, digest, issued_at; held false asks for the envelope whatever this node holds). held in the answer is false when this node has no store or no table. The envelope is the signed table itself, sent only when this node holds a newer pair, the same pair under another digest, or a later signing of the same table; the same table signed no later, or an older pair, is answered by era, generation and digest alone. The mesh0 sweep asks every peer it reaches every minute, so a cluster that agrees moves no tables.

Arguments:

KeyTypeRequiredNotes
heldbool
erauint64
generationuint64
digeststring
issued_atint64

Result:

KeyTypeNotes
node_idstring
heldbool
erauint64
generationuint64
digeststring
envelopeSignedSnapshot
envelope.payloadbytes
envelope.algstring
envelope.key_idstring
envelope.signaturebytes
envelope.certificatebytes

Verifies and applies a signed peer table without reading the store, and answers only once it is applied, with the public key mesh0 runs under on this node. A leader activates a rotation on that answer, so a table that tells this node its rotation is switching is answered under the new key.

Arguments:

KeyTypeRequiredNotes
payloadbytes
algstring
key_idstring
signaturebytes
certificatebytes

Result:

KeyTypeNotes
appliedbool
public_keystring

Runs this node’s mesh0 sweep now and answers once it is done, so a peer applies a row another node just changed. A node with no sweeper, one not on mesh0 or whose mesh reconcile has stopped, runs nothing and answers swept false with the reason.

Result:

KeyTypeNotes
sweptbooltrue once a sweep ran, false when none did
reasonstringwhy no sweep ran, set only when swept is false

Takes no arguments.

Takes part in a coordinated trial of the caller’s pair on a device (two-tunnel design section 4). At the instant this node sets the endpoint it holds for the caller, drops every UDP packet to the device’s port that is not from that tuple for the window, and sends three ICMP echoes to the caller’s address on the device. It answers once the window has closed. The caller must be the node named from, and the pair must be direct.

Arguments:

KeyTypeRequiredNotes
devicestringyesmesh0 or shc0
fromstringyesthe calling node’s id
atint64yesunix milliseconds, within the next 10 seconds
window_msint64yes1 to 400

Result:

KeyTypeNotes
okboolan echo was answered inside the window
endpointstringthe endpoint this node held for the caller

Lists the metric definitions this node’s collectors registered, with their cardinality-drop counts.

Result:

KeyTypeNotes
countint
metrics[]IntrospectEntry
metrics[].namestring
metrics[].typestring
metrics[].helpstring
metrics[].labels[]string
metrics[].buckets[]float64
metrics[].delta_wrappedbool
metrics[].leader_onlybool
metrics[].droppedint

Takes no arguments.

This node’s cached node and container resource stats, the per-node input of the HUD.

Result:

KeyTypeNotes
nodesmap[string]NodeStats
nodes{}.node_idstring
nodes{}.cpu_percentfloat64
nodes{}.cpu_coresint
nodes{}.memory_percentfloat64
nodes{}.memory_used_bytesint64
nodes{}.memory_total_bytesint64
nodes{}.memory_available_bytesint64
nodes{}.disk_percentfloat64
nodes{}.disk_used_bytesint64
nodes{}.disk_total_bytesint64
nodes{}.load_1mfloat64
nodes{}.load_5mfloat64
nodes{}.load_15mfloat64
nodes{}.docker_okbool
nodes{}.vault_okbool
nodes{}.is_leaderbool
nodes{}.uptime_secondsint64
nodes{}.containers_totalint
nodes{}.containers_runningint
nodes{}.statestring
nodes{}.subsystemsmap[string]int
nodes{}.unmeasured[]string
nodes{}.UpdatedAtTime
containersmap[string]ContainerCacheEntry
containers{}.container_idstring
containers{}.namestring
containers{}.servicestring
containers{}.statusstring
containers{}.healthstring
containers{}.node_idstring
containers{}.tenant_namestring
containers{}.stack_namestring
containers{}.environmentstring
containers{}.imagestring
containers{}.projectstring
containers{}.exit_codeint
containers{}.restart_countint
containers{}.oom_killedbool
containers{}.cpu_percentfloat64
containers{}.memory_bytesint64
containers{}.memory_limit_bytesint64
containers{}.network_rx_bytesint64
containers{}.network_tx_bytesint64
containers{}.disk_read_bytesint64
containers{}.disk_write_bytesint64
containers{}.raw_statsmap[string]any
containers{}.host_configmap[string]any
containers{}.started_atstring
containers{}.UpdatedAtTime
containers{}.stderr_tailstring

Takes no arguments.

Enrols a joining node with a single-use bootstrap token: mints its cluster certificate, records its row and grants its mesh and fabric addresses. Served only on the public join route, POST /api/join/node.enroll through Traefik on 443; the answer is signed with the cluster key.

Arguments:

KeyTypeRequiredNotes
bootstrap_tokenstringyesthe token shc node token minted; consumed by this call
node_namestringyes
physical_ipstringmust be the address the request came from, unless public_address is
public_addressstringan IP the request came from registers and signs that address instead of physical_ip (a node behind NAT)
locationstring
providerstring
runtimestring
fabric_public_keystring
cpu_coresint64
architecturestring

Returns: a signed envelope: payload (the enrol answer as JSON, base64), alg, key_id, signature; the joiner verifies it against the cluster CA its token pins.

This node’s liveness answer: leader status, engine health and a system-metrics snapshot. Exempt from the per-peer in-flight cap, because it is liveness.

Result:

KeyTypeNotes
node_idstring
statestring
uptime_secondsfloat64
timestampstring
is_leaderbool
leader_addrstring
cluster_modebool
docker_okbool
system_metricsSystemMetricsPayload
system_metrics.cpu_percentfloat64
system_metrics.cpu_coresint
system_metrics.memory_percentfloat64
system_metrics.memory_used_bytesuint64
system_metrics.memory_total_bytesuint64
system_metrics.disk_percentfloat64
system_metrics.disk_used_bytesuint64
system_metrics.disk_total_bytesuint64
system_metrics.disk_pathstring
system_metrics.inodes_useduint64
system_metrics.inodes_totaluint64
system_metrics.inodes_percentfloat64
system_metrics.load_averagefloat64
system_metrics.disk_checksmap[string]float64
system_metrics.unmeasured_facts[]string
system_metrics.memoryMeasuredMemory
system_metrics.memory.total_bytesint64
system_metrics.memory.available_bytesint64
system_metrics.memory.swap_total_bytesint64
system_metrics.memory.swap_used_bytesint64
system_metrics.memory.atTime
system_metrics.memory.exhaustedbool
system_metrics.memory.sinceTime
vault_okbool
certificateNodeCertHealth
certificate.not_afterTime
certificate.read_errorstring
certificate.renew_duebool
certificate.consecutive_renew_failuresint
certificate.last_renew_errorstring
certificate.last_renew_successTime
certificate.renewer_runningbool
certificate.last_renew_checkTime
pressure[]Sample
pressure[].queuestring
pressure[].unitstring
pressure[].depthint64
pressure[].capacityint64
pressure[].oldest_ageint64
pressure[].in_per_secondfloat64
pressure[].out_per_secondfloat64
pressure[].drops_per_secondfloat64
pressure[].in_totaluint64
pressure[].out_totaluint64
pressure[].dropped_totaluint64
pressure[].leveluint8
pressure[].sinceTime
pressure[].reasonstring
pressure[].errorstring
pressure[].peerstring
pressure[].retired_totaluint64
pressure[].retired_reasonstring
pressure[].stuck_forint64
pressure[].still_forint64
pressure[].stalebool
pressure[].amber_depthint64
pressure[].red_depthint64
pressure[].amber_ageint64
pressure[].red_ageint64
pressure[].atTime

Takes no arguments.

Runs this node’s image prune for room before an install pulls, or for an operator’s shc node prune, and answers its report.

Arguments:

KeyTypeRequiredNotes
requeststringyesa JSON object: keep, the images the install runs; reclaim, the bytes to give back, 0 for every unused image; dry_run, report and remove nothing

Result:

KeyTypeNotes
reportstringthe prune’s report as JSON: what it removed, or would, and the unused images it kept by reason

Answers this node’s room for a deployment’s images: the bytes free where its runtime unpacks them, the bytes its unused images would give back, and which of the images it already holds, for the admission that places the deployment.

Arguments:

KeyTypeRequiredNotes
imagesstringyesthe image refs, as a JSON array

Result:

KeyTypeNotes
heldstringthe images this node holds, as a JSON array
freestringthe free bytes, absent when the runtime’s store could not be read
reclaimablestringthe bytes this node’s image prune would give back keeping the deployment’s images, beside free

Starts this node’s departure from the cluster in the background and answers at once.

Arguments:

KeyTypeRequiredNotes
forcebool

Result:

KeyTypeNotes
statusstring
messagestring

Checks the notice names this node, answers, and stands this node down behind the answer: another node removed it from the cluster.

Arguments:

KeyTypeRequiredNotes
node_idstringyes

Result:

KeyTypeNotes
statusstring
messagestring

Deletes the named node’s record on this node, best effort.

Arguments:

KeyTypeRequiredNotes
node_idstringyes

Result:

KeyTypeNotes
statusstring

Renews the calling node’s lease on the leader. Exempt from the per-peer in-flight cap, because it is liveness.

Arguments:

KeyTypeRequiredNotes
nodestringyesthe renewing node’s id
boot_idstringthis boot of the renewing node
epochuint64the lease epoch the node last held

Result:

KeyTypeNotes
epochuint64
ttl_msint64how long the granted lease lasts
next_msint64when to renew next

Cancels every step of an operation this node runs under its id, tenant and stack, for the node a shc.operation.cancel landed on. Each step ends failed with X499052OPRCNL naming who cancelled it. It asks no other node.

Arguments:

KeyTypeRequiredNotes
operation_idstringyes
kindstringthe operation’s kind, as X499052OPRCNL names it
tenantstringthe operation’s tenant; a step held under another tenant, or under one when this names none, is left running
stackstringthe operation’s stack, as X499052OPRCNL names it; it scopes the cancel too, so a step held under another stack is left running
actorstringwho asked for the cancel, as X499052OPRCNL names them

Result:

KeyTypeNotes
environmentsarraythe environment of each step this node cancelled; empty when it runs none under that id, tenant and stack, as for an id it never held

Sends an operation this node holds to the node that asked, each frame through that node’s operation.frame, and answers after its terminal frame or once that node takes no more. The call holds one of the asking node’s in-flight slots here for as long as the follow lasts, as app.update_deploy does for a deploy.

Arguments:

KeyTypeRequiredNotes
operation_idstringyes
sourcestringyesas operation.state answered it
tenantstringas operation.state answered it; another operation under the same id is not sent
stackstringas operation.state answered it
afterintwith registry: the last frame the follower has
nodestringyesthe node following the operation
tokenstringyesthe relay that node opened for it

Result:

KeyTypeNotes
foundboolfalse when this node no longer holds that operation

Takes one frame of an operation another node holds, for the stream following it on this node.

Arguments:

KeyTypeRequiredNotes
tokenstringyesthe relay operation.follow named
framestringyesthe frame as JSON: sequence, type and data

Result:

KeyTypeNotes
takenboolfalse once the stream has ended; the holder then stops sending

Answers every operation this node holds in its memory under an id, for a node a read of it landed on that holds none the caller may read. It asks no other node.

Arguments:

KeyTypeRequiredNotes
operation_idstringyes

Result:

KeyTypeNotes
operationsarrayempty when this node holds nothing under that id; the operations store’s entry comes before the operation registry’s
operations[].sourcestringstore for the operations store, registry for the operation registry
operations[].tenantstring
operations[].stackstring
operations[].statestringthe state as shc.operation.state writes it, as JSON

Queues one OTLP export on this node, which holds OpenObserve, for delivery to it. A node that does not hold OpenObserve refuses with X409115OTRMOV and names the holder.

Arguments:

KeyTypeRequiredNotes
signalstringyesmetrics, traces or logs
bodyblobyesthe OTLP request body, 32 MiB at most
content_typestring
content_encodingstring
sourcestringthe node that exported it

Result:

KeyTypeNotes
queuedbool

Runs an automatic move, a drain’s or a rebalance’s, on the leader for the follower that was asked for it, under the leader’s freeze gate and move budget. A node that does not lead refuses X503798FRZLDR and sends it nowhere.

Arguments:

KeyTypeRequiredNotes
deploymentDeploymentSummaryyesthe deployment to move, as the asking node listed it
deployment.idstring
deployment.namestring
deployment.tenant_namestring
deployment.stackstring
deployment.envstring
deployment.statusstring
deployment.runtimestring
deployment.anchoredbool
deployment.observedbool
deployment.nodestring
targetstringyesthe node id it moves to
servicestringa replica move’s service; empty moves the whole deployment
ordinalinta replica move’s ordinal
plannerstringthe planner whose plan the move must still stand on; empty for none
callerobjectthe asking user’s scope, so the move runs as them
caller.within_tenantbool
caller.installed_bystring

Result:

KeyTypeNotes
movedbool

Inspects, names the directories it binds under the deployment’s mounts, stops with read-back, starts at a fenced generation, or receives and removes one replica’s local volumes and bind directories.

Arguments:

KeyTypeRequiredNotes
verbstringyes
planobjectyes
generationuint64
operationstring
volumestring
bindstringa directory under the deployment’s mounts the plan carries, in place of volume
archiveblob
claimsarray

Result:

KeyTypeNotes
successbool
binds[]stringinspect and binds: the directories the replica’s project binds under this node’s mounts root

Answers how many containers of a deployment’s replica projects have not exited on this node, for a failed install’s rollback that waits for them to stop.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
environmentstringyes
stackstringyes

Result:

KeyTypeNotes
upintcontainers that have not exited; an engine that cannot be asked is an error, never zero

Starts or stops every local replica holding a hook action’s service and returns the first coded refusal.

Arguments:

KeyTypeRequiredNotes
refDeploymentRefyesthe deployment
ref.Tenantstring
ref.Environmentstring
ref.Stackstring
servicestringyesthe logical service
stopboolyes

Result:

KeyTypeNotes
appliedbool

Answers the leader’s read of a load balancer’s VIPs for a node that is not the leader; a node that does not lead refuses with X503984PRXLDR and reads nothing.

Arguments:

KeyTypeRequiredNotes
tenantstringthe tenant of the load balancer’s deployment
stackstringthe deployment’s stack
connstringthe proxy connection
freshboolskip the leader’s kept ready answer

Result:

KeyTypeNotes
vipsarraythe VIPs, empty when not ready
readyboolfalse when the box is not ready
not_readystringthe box’s reason when ready is false

Caps the directories of the deployments named at the bytes given, through the filesystem’s own quota where it has one (an xfs project on a prjquota mount, a zfs dataset of its own, a btrfs subvolume with quotas on), and answers how each was capped. Zero bytes takes off only a cap SHC set. The disk sweep sends each node the caps of the deployments it holds, with the sizes of their node-disk volumes from the node whose manifest gives them, so a node holding the volumes without the render caps them too.

Arguments:

KeyTypeRequiredNotes
capsstringa JSON array of {tenant, environment, deployment, bytes, volumes, volumes_known}

Result:

KeyTypeNotes
resultsstringa JSON array of {path, bytes, applied, hard, how}, one entry per directory capped

Measures this node’s own deployment tree and answers the bytes each deployment holds here, without crossing into another filesystem, so a block volume bound onto a mount is left out. The disk sweep asks every node and sums the answers per tenant, environment and deployment.

Result:

KeyTypeNotes
deploymentsstringa JSON array of {tenant, environment, deployment, bytes, volumes, rendered}, one entry per deployment directory on this node; volumes are the node-disk volume sizes its manifest gives, rendered says this node holds that manifest

Takes no arguments.

A node reports the live label values that changed for replicas it runs; the leader writes them.

Arguments:

KeyTypeRequiredNotes
changes[]Change
changes[].tenantstring
changes[].environmentstring
changes[].stack_namestring
changes[].service_namestring
changes[].ordinalint
changes[].keystring
changes[].valuestring
changes[].absentbool
changes[].node_idstring
changes[].routedbool

Answers the id of a running container for a deployment’s service on this node, for a peer that will bridge an interactive exec there.

Arguments:

KeyTypeRequiredNotes
tenantstring
stackstringyes
environmentstring
servicestringempty means the deployment’s first running container on this node

Result:

KeyTypeNotes
container_idstring

Runs a compose command against a deployment on this node, which holds it.

Arguments:

KeyTypeRequiredNotes
targetStackTargetDescriptorthe deployment and the node it is routed to
target.Namestring
target.TenantNamestring
target.Environmentstring
target.NodeIDstring
target.AppNamestring
target.Guardedbool
target.OneNodebool
target.wherestring
argv[]stringyesthe compose arguments, at least one
bytesboolan exec answers its streams as StdoutBytes and StderrBytes, with Stdout and Stderr empty; without it, as the strings alone, which a caller before 0.1.24 reads

Result:

KeyTypeNotes
Successbool
ExitCodeint
Stdoutstring
Stderrstring
StdoutBytesbytes
StderrBytesbytes
Replicas[]string
Members[]string
Waitingbool

Answers where a streamed shc exec of a deployment runs on this node, which holds its replica: the container, the runtime that runs it and the command, for the node the client reached, which then runs it here through the interactive exec route.

Arguments:

KeyTypeRequiredNotes
targetStackTargetDescriptorthe deployment and the node it is routed to
target.Namestring
target.TenantNamestring
target.Environmentstring
target.NodeIDstring
target.AppNamestring
target.Guardedbool
target.OneNodebool
target.wherestring
argv[]stringyesthe exec as compose takes it: exec -T [--user U] [--workdir W] SERVICE CMD…, SERVICE empty for the deployment’s first running container

Result:

KeyTypeNotes
container_idstring
runtimestring
command[]string
userstring
workdirstring
missingstring

Answers shc inspect for a deployment from this node, which holds its directory, redacted here against the files it read.

Arguments:

KeyTypeRequiredNotes
targetstringyesthe StackTargetDescriptor, as JSON

Result:

KeyTypeNotes
inspectstringthe StackInspectResponse, as JSON

Lists this node’s containers of a deployment for the inspect fan-out that follows assignment rows.

Arguments:

KeyTypeRequiredNotes
targetstringyesthe StackTargetDescriptor, as JSON

Result:

KeyTypeNotes
containersstringthe []StackContainerView, as JSON

Installs a deployment on this node for the node that routed the request here.

Arguments:

KeyTypeRequiredNotes
targetStackTargetDescriptorthe deployment and the node it is routed to
target.Namestring
target.TenantNamestring
target.Environmentstring
target.NodeIDstring
target.AppNamestring
target.Guardedbool
target.OneNodebool
target.wherestring
requestStackInstallRequestthe install as the public API received it; the deployment is named by target
request.stack_namestring
request.environmentstring
request.tenant_namestring
request.node_idstring
request.app_namestring
request.runtimestring
request.composestring
request.timeout_secint
request.operation_idstring
request.integrations[]string
request.variablesmap[string]string
request.app_dirstring
request.stored_app_digeststring
request.ports[]string
request.profiles[]string
request.app_versionstring
request.repo_urlstring
request.labelsmap[string]string
request.domainstring
request.dns_providerstring
request.acme_providerstring
request.recoverybool
request.effective_varsstring
request.backed_up_varsmap[string]string
request.groupsmap[string][]Member
request.package_from_archivebool
request.stated_node_idstring

Result:

KeyTypeNotes
successbool
messagestring
exit_codeint
stdoutstring
stderrstring
deployment_dirstring
bound[]StackBinding
bound[].plugstring
bound[].provider_stack_namestring
bound[].provider_tenant_namestring
bound[].provider_environmentstring
bound[].provider_connectionstring
scratch_notes[]string
set_aside_pathstring
set_aside_errorstring
proxied_ports[]StackProxiedPort
proxied_ports[].connectionstring
proxied_ports[].externalint
proxied_ports[].protocolstring
proxied_ports[].node_portint
proxied_ports[].hostnamestring
proxied_ports[].addressstring
proxied_ports[].proxy_protocolbool
published_ports[]Port
published_ports[].servicestring
published_ports[].containerint
published_ports[].portint
published_ports[].protocolstring
published_ports[].connectionstring
published_ports[].listenstring
published_ports[].spreadbool
published_ports[].hostnamestring
published_ports[].localbool
published_ports[].privatebool
published_ports[].proxy_protocolbool
published_ports[].published_onlybool

Answers the deployment’s descriptor if this node holds it.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstring
environmentstringdefault when empty

Result:

KeyTypeNotes
stack_namestring
tenant_namestring
environmentstring
node_idstring
app_namestring
runtimestring

Returns: null when this node does not hold the deployment.

Tears a deployment down on this node for the node that routed the request here.

Arguments:

KeyTypeRequiredNotes
targetStackTargetDescriptorthe deployment and the node it is routed to
target.Namestring
target.TenantNamestring
target.Environmentstring
target.NodeIDstring
target.AppNamestring
target.Guardedbool
target.OneNodebool
target.wherestring
requestStackUninstallRequestthe uninstall as the public API received it; the deployment is named by target
request.stack_namestring
request.environmentstring
request.tenant_namestring
request.remove_volumesbool
request.drop_databool
request.operation_idstring
request.node_idstring
request.keep_recordbool
request.keep_dnsbool
request.scratch_volumes[]string
request.set_asidestring
request.keep_renderbool
require_local_stateboolrefuse when this node holds no deployment directory, instead of tearing down nothing

Result:

KeyTypeNotes
successbool
messagestring
exit_codeint
stdoutstring
stderrstring
deployment_dirstring
bound[]StackBinding
bound[].plugstring
bound[].provider_stack_namestring
bound[].provider_tenant_namestring
bound[].provider_environmentstring
bound[].provider_connectionstring
scratch_notes[]string
set_aside_pathstring
set_aside_errorstring
proxied_ports[]StackProxiedPort
proxied_ports[].connectionstring
proxied_ports[].externalint
proxied_ports[].protocolstring
proxied_ports[].node_portint
proxied_ports[].hostnamestring
proxied_ports[].addressstring
proxied_ports[].proxy_protocolbool
published_ports[]Port
published_ports[].servicestring
published_ports[].containerint
published_ports[].portint
published_ports[].protocolstring
published_ports[].connectionstring
published_ports[].listenstring
published_ports[].spreadbool
published_ports[].hostnamestring
published_ports[].localbool
published_ports[].privatebool
published_ports[].proxy_protocolbool
published_ports[].published_onlybool
retained_cleanup[]RetainedCleanupJobcleanup jobs the teardown left behind, when there are any
retained_cleanup[].stackstring
retained_cleanup[].jobstring
retained_cleanup[].verbstring
retained_cleanup[].codestring
retained_cleanup[].detailstring

Unmounts CephFS on this node. The caller’s certificate must name target_node.

Arguments:

KeyTypeRequiredNotes
target_nodestringyes

Result:

KeyTypeNotes
messagestring

Mounts the cluster CephFS on this node. The caller’s certificate must name target_node.

Arguments:

KeyTypeRequiredNotes
mount_pointstringyes
monitors[]stringyes
client_namestringyes
secretstringyes
fsidstring
target_nodestringyes

Result:

KeyTypeNotes
messagestring

Installs the cephadm public key the leader sends, so this node can be added as a Ceph host.

Arguments:

KeyTypeRequiredNotes
pub_keystringyes

Result:

KeyTypeNotes
okbool
hostnamestring

Tears down this node’s NFS server or client role. The caller’s certificate must name target_node.

Arguments:

KeyTypeRequiredNotes
rolestring
target_nodestringyes

Result:

KeyTypeNotes
messagestring

Mounts the cluster NFS share on this node. The caller’s certificate must name target_node.

Arguments:

KeyTypeRequiredNotes
server_addressstringyes
export_pathstringyes
mount_pointstringyes
target_nodestringyes

Result:

KeyTypeNotes
messagestring

Sets up the NFS kernel server on this node and exports the share. The caller’s certificate must name target_node.

Arguments:

KeyTypeRequiredNotes
export_pathstringyes
allowed_networks[]stringyes
take_overbool
bindstring
target_nodestringyes

Result:

KeyTypeNotes
messagestring
bind_ipstring

Reports this node’s NFS state: installed, running, exporting, mounted.

Result:

KeyTypeNotes
is_serverbool
is_clientbool
is_installedbool
is_runningbool
exports_countint
mount_activebool
errorstring

Takes no arguments.

Runs one shared-storage operation on the leader for a follower that was asked for it.

Arguments:

KeyTypeRequiredNotes
opstringyesthe operation, one of the SharedOp names in modules/storage/leader_forward.go
paramsobjectthat operation’s own arguments

Returns: what the named operation answers on the leader.

Grows the filesystem on a block volume attached to this node after its provider grew the volume, after checking the caller still holds the volume’s lease generation. The caller’s certificate must name target_node.

Arguments:

KeyTypeRequiredNotes
lease_generationint64 as a decimal string
volume_idstringyes
size_bytesint64 as a decimal string
target_nodestringyes

Result:

KeyTypeNotes
size_bytesint64 as a decimal string

Mounts an attached block volume on this node, after checking the caller still holds the volume’s lease generation. The caller’s certificate must name target_node.

Arguments:

KeyTypeRequiredNotes
device_pathstringyes
mountpointstringyes
filesystemstring
lease_generationint64 as a decimal string
volume_idstring
target_nodestringyes
createdbool

Result:

KeyTypeNotes
messagestring

Unmounts a block volume on this node. The caller’s certificate must name target_node.

Arguments:

KeyTypeRequiredNotes
mountpointstringyes
volume_idstring
target_nodestringyes

Result:

KeyTypeNotes
messagestring

Answers which containers of a deployment have not exited on this node, before a takeover moves its volume away.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstring
environmentstring

Result:

KeyTypeNotes
up[]stringcontainer ids, when the node could tell
undeterminedstringthe reason, when it could not

Runs a store client’s write on this node’s store, which forwards it to the leader: one statement alone, several as one atomic batch.

Arguments:

KeyTypeRequiredNotes
statements[].sqlstringyes
statements[].args[]anywritten as store.query’s args are

Result:

KeyTypeNotes
results[].rows_affectedint64
results[].last_insert_idint64

Runs a store client’s read on this node’s store: from this node’s copy under the caller’s bound when the read is marked local, else at weak level by the leader.

Arguments:

KeyTypeRequiredNotes
sqlstringyes
args[]anyeach nil, raw bytes, or a string whose first byte names its type
boundstringhow far behind the leader this node’s copy may be, empty for a weak read

Result:

KeyTypeNotes
columns[]string
rows[][]anycells written as args are

Drops a staged transfer the caller has finished pulling, so the holder frees its disk before the handle expires.

Arguments:

KeyTypeRequiredNotes
idstringthe transfer handle id; an empty or unknown id is a no-op

Lists the cluster’s members as they are now, so a joining node can try a handshake with each of them in both directions before it enrols (shc node join). Served only on the public join route, POST /api/join/underlay.members through Traefik on 443; the answer is signed with the cluster key.

Arguments:

KeyTypeRequiredNotes
bootstrap_tokenstringa joiner’s: the token shc node token minted, checked and not consumed
requestobjecta member’s: the signed request underlay.probe.arm takes, made for this function
chainstringa member’s node.pem certificate chain, with request

Returns: a signed envelope: node, the member that answered, and members (node, underlay, public, devices (device, port, public_key), reached: the devices on which the member that answered holds a handshake with it inside three minutes).

Holds a UDP probe’s throwaway WireGuard public keys as peers of this node’s mesh0 and shc0, with no allowed IPs, for at most 15 seconds, so the caller can try a handshake on each device’s real port before a join (shc node join) or in doctor (underlay.udp). Served only on the public join route, POST /api/join/underlay.probe.arm through Traefik on 443; the answer is signed with the cluster key.

Arguments:

KeyTypeRequiredNotes
bootstrap_tokenstringa joiner’s: the token shc node token minted, checked and not consumed
requestobjecta member’s: clustersign envelope over node_id, issued_at, a 32-character lowercase hex nonce, function and the fields below, signed by the node identity key under shc-underlay-probe-request-v1 followed by a newline
chainstringa member’s node.pem certificate chain, with request
keysobjectyesdevice name (mesh0, shc0) to the probe’s base64 public key for it; inside request for a member
dialobjecta joiner’s only: device name to key and endpoint, a second base64 public key this node holds as a peer and the joiner’s underlay address and port, which this node’s device sends its own handshake to

Returns: a signed envelope: id, node, devices (device, port, public_key, dialing) held, and refused (device to the reason this node cannot hold a key on it).

Arms another member for a joiner whose own dial to that member’s join route does not answer: this node checks the token, then asks the member over mesh0 to hold the probe’s keys as underlay.probe.arm would (shc node join). Served only on the public join route, POST /api/join/underlay.probe.relay.arm through the edge on 443; the answer is signed with the cluster key.

Arguments:

KeyTypeRequiredNotes
bootstrap_tokenstringyesthe joiner’s token shc node token minted, checked and not consumed; a relay carries a joiner’s probe only
nodestringyesthe member to arm, by node id, one of the cluster’s live mesh0 rows
keysobjectyesdevice name (mesh0, shc0) to the probe’s base64 public key for it
dialobjectdevice name to key and endpoint the member’s device sends its own handshake to, as underlay.probe.arm takes it

Returns: a signed envelope: the member’s underlay.probe.arm answer.

Reads a relayed probe’s result from the member it armed, and releases the keys. Served only on the public join route, POST /api/join/underlay.probe.relay.result through the edge on 443; the answer is signed with the cluster key.

Arguments:

KeyTypeRequiredNotes
bootstrap_tokenstringyesthe joiner’s token shc node token minted, checked and not consumed; a relay carries a joiner’s probe only
nodestringyesthe member to arm, by node id, one of the cluster’s live mesh0 rows
idstringyesthe id underlay.probe.relay.arm answered

Returns: a signed envelope: the member’s underlay.probe.result answer.

Holds a joiner’s probe keys that the member it joins through relays here, as underlay.probe.arm does for the joiner itself.

Arguments:

KeyTypeRequiredNotes
callerstringyeswho the probe is held for: the relaying member and a digest of the joiner’s token
keysobjectyesdevice name to the probe’s base64 public key
dialobjectdevice name to key and endpoint, as underlay.probe.arm takes it

Returns: the underlay.probe.arm answer.

Reads and releases a relayed probe, for the member that armed it.

Arguments:

KeyTypeRequiredNotes
callerstringyesthe caller the probe was armed for
idstringyesthe id underlay.probe.relayed.arm answered

Returns: the underlay.probe.result answer.

Reads the bytes each device received from and sent to a probe’s key and whether a handshake completed, then releases the keys. Served only on the public join route, POST /api/join/underlay.probe.result through Traefik on 443; the answer is signed with the cluster key.

Arguments:

KeyTypeRequiredNotes
bootstrap_tokenstringa joiner’s: the token shc node token minted, checked and not consumed
requestobjecta member’s: clustersign envelope over node_id, issued_at, a 32-character lowercase hex nonce, function and the fields below, signed by the node identity key under shc-underlay-probe-request-v1 followed by a newline
chainstringa member’s node.pem certificate chain, with request
idstringyesthe id underlay.probe.arm answered; inside request for a member

Returns: a signed envelope: id and devices (device, port, received, sent, handshake, and dialing, back_received, back_sent, back_handshake for a device that dialled).

Writes this node’s auto-unlock drop-in, once the pushed password matches the cluster’s replicated verifier.

Arguments:

KeyTypeRequiredNotes
passwordstringyes

Result:

KeyTypeNotes
successbool
messagestring

Removes this node’s auto-unlock drop-in.

Result:

KeyTypeNotes
successbool
messagestring

Takes no arguments.

Moves this node’s in-memory vault key to the rotated password, after validating it against the replicated verifier. A rejected re-key is an error, never a reply.

Arguments:

KeyTypeRequiredNotes
new_passwordstringyes
adopt_drop_inbool

Result:

KeyTypeNotes
successbool
messagestring

The knock: a locked node asks this unlocked node to push the unlock straight back to the address it names.

Arguments:

KeyTypeRequiredNotes
node_idstringyes
addressstringyes

Result:

KeyTypeNotes
successbool
messagestring

Unlocks this node’s vault with the pushed password. An already unlocked vault answers success and does not re-derive its key.

Arguments:

KeyTypeRequiredNotes
passwordstringyes

Result:

KeyTypeNotes
successbool
messagestring

Dials each published port on this node’s own address once its fabric policy carries the revision, and says which answer.

Arguments:

KeyTypeRequiredNotes
targets[]ProbeTargetyesthe published TCP ports this node listens for
targets[].portint
targets[].addressstring
targets[].policybool
revisionuint64the clusterfs revision the ports were recorded at; zero asks nothing of the fabric policy

Result:

KeyTypeNotes
answers[]ProbeAnswerone per target, in order
answers[].portint
answers[].addressstring
answers[].answersbool
answers[].unprobedbool
answers[].policybool
answers[].reasonstring

Reads a file, or the byte range a Range header names, from a deployment volume this node holds.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstringyes
envstringyes
volumestringyesthe mount name the app declares
blockboola block volume, read only while it is mounted here
pathstringyesslash path inside the volume
rangestringan HTTP Range header, bytes=a-b

Result:

KeyTypeNotes
datablob
entryobject
offsetint64
lengthint64
partialbool

Says whether this node’s disk lays out a deployment volume.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstringyes
envstringyes
volumestringyesthe mount name the app declares
blockboola block volume, read only while it is mounted here

Result:

KeyTypeNotes
heldbool

Lists one page of a folder of a deployment volume this node holds, sorted whole first.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstringyes
envstringyes
volumestringyesthe mount name the app declares
blockboola block volume, read only while it is mounted here
pathstringyesslash path inside the volume
offsetint64where the page starts in the sorted folder
limitint64the most entries the page holds, at most 5000

Result:

KeyTypeNotes
tenantstring
stackstring
environmentstring
volumestring
pathstring
nodestring
node_namestring
entries[]VolumeFileEntry
entries[].namestring
entries[].pathstring
entries[].kindstring
entries[].sizeint64
entries[].modifiedTime
entries[].modestring
entries[].targetstring
totalint
offsetint
nextint
truncatedbool

Names the volumes a deployment lays out on this node’s disk.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstringyes
envstringyes

Result:

KeyTypeNotes
volumes[]string

Renames a file or folder inside a deployment volume this node holds.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstringyes
envstringyes
volumestringyesthe mount name the app declares
blockboola block volume, read only while it is mounted here
fromstringyes
tostringyes

Result:

KeyTypeNotes
namestring
pathstring
kindstring
sizeint64
modifiedTime
modestring
targetstring

Writes a file into a deployment volume this node holds, renamed into place once whole.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstringyes
envstringyes
volumestringyesthe mount name the app declares
blockboola block volume, read only while it is mounted here
pathstringyesslash path inside the volume
overwritebool
datablobyes

Result:

KeyTypeNotes
namestring
pathstring
kindstring
sizeint64
modifiedTime
modestring
targetstring

Deletes a file or folder from a deployment volume this node holds.

Arguments:

KeyTypeRequiredNotes
tenantstringyes
stackstringyes
envstringyes
volumestringyesthe mount name the app declares
blockboola block volume, read only while it is mounted here
pathstringyesslash path inside the volume
recursivebool

Result:

KeyTypeNotes
pathstring