Internal RPC reference
Generated file. Do not edit by hand. Regenerate from the repo root with
make docs/generate.
A call one node makes to another is a named function, sent as
POST /internal/rpc/{function} on the internal mTLS listener with a
JSON body {"args": [], "kwargs": {...}}.
A request refused on its shape or its caller gets a real HTTP status: 403
on the wrong listener or for a capability the caller lacks, 400 with no
function name or a body that cannot be read or parsed, 413 for a body over
the size limit. Every other answer is HTTP 200, {"result": ...} or
a coded error the caller raises again as the same typed error. That covers
what the function answers and four refusals made before it runs: a
function name nobody registered, a node already serving its limit of calls
from this peer, staged blobs past the function’s transfer bound, and a
binary argument that cannot be resolved, whether its reference is
malformed or its transfer cannot be pulled.
Callers go through internal_rpc.Call, never a hand-built request, so
every call gets the per-peer breaker, the busy backoff and the byte
transfer.
Arguments are the keys of kwargs. A blob is binary data: small
ones ride the call, large ones are staged and pulled as raw ranged reads
of GET /internal/transfer/{id}, the one route under RPC. Each
function bounds the bytes of staged blobs one call may make the callee
pull, 4 MiB unless a blob’s notes say otherwise. The callee checks the
sizes the call declares before it pulls anything and refuses a call past
the bound with X413147RPCBND.
Not everything is a function. The live streams (log follow, interactive
exec, port-forward) keep WebSocket routes, because a function answers
once. Enrol keeps its route. tests/unit/internalroutes pins that whole set of routes.
Total: 166 functions.
app: 12 functionsauth: 1 functionbackup: 19 functionscert: 2 functionscerts: 3 functionsclone: 10 functionscluster: 10 functionsclusterfs: 1 functioncontainer: 12 functionscontainerbroker: 1 functioncron: 1 functiondoctor: 2 functionsedge_ca: 1 functionevents: 2 functionsexport: 2 functionsfence: 1 functionhealth: 1 functionjob: 4 functionslogs: 1 functionmesh: 14 functionsmetrics: 2 functionsnode: 7 functionsnodelease: 1 functionoperation: 4 functionsotel: 1 functionplacement: 1 functionpods: 3 functionsproxy: 1 functionquota: 2 functionsreplica_label: 1 functionshell: 1 functionstack: 7 functionsstorage: 12 functionsstore: 2 functionstransfer: 1 functionunderlay: 7 functionsvault: 5 functionsvip: 1 functionvolume: 7 functions
app.deferred_hooks
Section titled “app.deferred_hooks”Runs, on this node, which holds the deployment directory, the post hooks the deployment’s last update left for its next start because they needed services its operator had stopped (FZ-6993), once the deployment reads ready.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | the deployment’s tenant |
stack | string | yes | the deployment’s name |
environment | string | yes | the deployment’s environment |
Result:
| Key | Type | Notes |
|---|---|---|
reply | string | a DeferredHookReport as JSON, null when the update left nothing |
app.deployment_dir
Section titled “app.deployment_dir”Answers whether this node’s disk holds the deployment’s directory by the uninstall’s test: any directory, a vdeployment’s parent’s. A stat that fails other than absent is an error, never an answer. With it, whether the directory holds a render or only a replica’s mounts, and the render mark it carries, which tells the current render’s directory from an older one’s.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | yes | |
env | string | the environment |
Result:
| Key | Type | Notes |
|---|---|---|
present | bool | |
rendered | bool | the directory holds a render (its manifest), not only the mounts a replica’s volumes need |
mark | string | the digest of the render the directory’s state is; empty for a directory no render marked |
app.install_deploy
Section titled “app.install_deploy”Runs the deploy half of an install on this node, the one the deployment was placed on.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | InstallRequest | yes | the install as the public API received it |
request.tenant_name | string | ||
request.app_name | string | yes | |
request.app_version | string | ||
request.stack_name | string | ||
request.environment | string | ||
request.node_id | string | ||
request.repo_name | string | ||
request.variables | map[string]string | ||
request.remove_vars | []string | ||
request.integrations | []string | ||
request.profiles | []string | ||
request.where | string | ||
request.prefer | string | ||
request.where_services | []string | ||
request.prefer_services | []string | ||
request.connections | []string | ||
request.config | []string | ||
request.vapp | string | ||
request.resource_overrides | map[string]any | ||
request.hosts | []string | ||
request.ports | []string | ||
request.dns_provider | string | ||
request.storage | string | ||
request.volume_storage | map[string]VolumeStorage | ||
request.acme_provider | string | ||
request.domain | string | ||
request.naming_exception | string | ||
request.runtime | string | ||
request.operation_id | string | ||
request.no_rollback | bool | ||
request.no_wait | bool | ||
request.scale_overrides | map[string]string | ||
request.addon | string | ||
request.addon_version | string | ||
request.access | string | ||
request.exposure | string | ||
request.source | SourceData | ||
request.app_archive | string | ||
request.token | string | ||
request.auth | string | ||
request.within_tenant | bool | ||
request.ttl | string | ||
request.force | bool | ||
request.protected | bool | ||
request.restore | bool | ||
request.installed_by | string | ||
request.labels | []string | ||
seeded_integrations | bool | the asking node already seeded the integrations | |
input_var_keys | []string | operator variable keys before defaults and persisted overrides are folded, empty for platform writes | |
held_bindings_known | bool | true when held_bindings is what the deployment held before this install; without it held_bindings is not read | |
held_bindings | []PersistedIntegration | the bindings the deployment held before this install | |
held_bindings[].plug | string | ||
held_bindings[].provider_stack_name | string | ||
held_bindings[].provider_tenant_name | string | ||
held_bindings[].provider_environment | string | ||
held_bindings[].provider_connection | string | ||
step_node | string | the node that handed the deploy here and follows its operation; each step is sent to its app.operation_step | |
step_token | string | the relay that node opened for this deploy; without both no step is sent |
Result:
| Key | Type | Notes |
|---|---|---|
deployed | bool | |
settled | string | the deployment’s status once the deploy has settled |
settle_error | string | a coded error envelope as JSON, when the status settled on a failure |
app.node_disk_cap
Section titled “app.node_disk_cap”Answers whether this node’s filesystem can hard-cap a deployment’s node-disk volume directory, which need not exist yet. The render asks the node holding a volume before it gives the volume a size. Writes nothing.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | yes | |
env | string | the environment | |
volume | string | yes | the volume’s name, its directory under the deployment’s mounts |
Result:
| Key | Type | Notes |
|---|---|---|
hard | bool | the filesystem refuses writes past a cap SHC sets on the directory |
filesystem | string | |
why | string | why the cap would not be hard; empty when it would |
app.openobserve_reset_admin_password
Section titled “app.openobserve_reset_admin_password”Changes the system OpenObserve’s admin password on this node, which runs OpenObserve: the live password through its users API, then the vault record, then the deployment’s update. A node whose host cannot reach the store’s service address hands the change here. The change goes on after the caller stops waiting.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
password | string | the new admin password; empty generates one |
Result:
| Key | Type | Notes |
|---|---|---|
reply | string | as JSON: result (password when generated, generated, redeployed), or refusal (code, message, kwargs, details) for a refusal that carries details, which an RPC error does not |
app.operation_step
Section titled “app.operation_step”Takes one step of a deploy this node handed to another node, for the operation that is followed here. The node running the deploy sends its steps one at a time and in order, and answers its own call after the last.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
token | string | yes | the relay this node opened for the deploy |
step | string | yes | the step, a DeploymentPayload as JSON |
Result:
| Key | Type | Notes |
|---|---|---|
taken | bool | false when the relay is closed, which ends the sender’s relay and nothing else |
app.uninstall
Section titled “app.uninstall”Runs a whole uninstall on this node, which holds the deployment directory: the safety backup, the teardown and the row’s purge. A second uninstall of the same deployment is refused with X409949UNIRUN while this one runs.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | string | yes | the UninstallRequest, as JSON; stack_name must be set |
run | string | yes | the uninstall’s operation, the node holding that operation and the node running it, as JSON: operation, origin, holder, detach |
step_node | string | the node that sent the uninstall here and follows its operation; each step, and each progress line of the safety backup, is sent to its app.uninstall_step | |
step_token | string | the relay that node opened for this uninstall; without both no step is sent |
Result:
| Key | Type | Notes |
|---|---|---|
outcome | string | as JSON: retained, the cleanup jobs the teardown left behind; unasked_nodes; purged, the rows the deployment owned that the uninstall removed |
app.uninstall_copy
Section titled “app.uninstall_copy”Takes this node’s copy of a deployment down and removes its directory, data included, for an uninstall that ran on the node holding the deployment. Local only: no hook, unintegration job or DNS, credential or role release runs. Refused X409194CPYLIV unless the deployment’s row is stamped removing and names another node, or names this node and this node holds no render of the deployment. A directory with a volume still mounted under mounts/ is kept and refused. A node that holds no directory of the deployment takes down the replica projects it runs of it, and answers once they are down.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | yes | a deployment’s name; a vdeployment is refused |
env | string | the environment |
Result:
| Key | Type | Notes |
|---|---|---|
removed | bool |
app.uninstall_step
Section titled “app.uninstall_step”Takes one step of an uninstall this node sent to the node holding the deployment, or one progress line of its safety backup, for the operation that is followed here. The holder sends them one at a time and in order.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
token | string | yes | the relay this node opened for the uninstall |
step | string | the step’s name, as the uninstall emits it; empty for a progress line | |
percent | int | the step’s percent | |
message | string | the step’s words | |
detail | string | a progress line of the safety backup: what it has done so far, in words |
Result:
| Key | Type | Notes |
|---|---|---|
taken | bool | false when the relay is closed, which ends the sender’s relay and nothing else |
app.update_deploy
Section titled “app.update_deploy”Runs an update’s deploy on this node, which holds the deployment directory, and settles the deployment’s status after it.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | string | yes | the UpdateRequest, as JSON |
current | string | the deployment’s row, as JSON | |
empty_maps | string | comma-separated names of override maps that were empty rather than absent, which JSON does not carry | |
held_bindings | string | the bindings the deployment held before this update, a list of PersistedIntegration as JSON; sending it marks them known | |
placement_node | string | the node a move’s or a rebuild’s re-render places the deployment on | |
step_node | string | the node that handed the deploy here and follows its operation; each step is sent to its app.operation_step | |
step_token | string | the relay that node opened for this deploy; without both no step is sent |
Result:
| Key | Type | Notes |
|---|---|---|
reply | string | an UpdateDeployReply, as JSON: effect, settled, settle_error |
app.update_pin
Section titled “app.update_pin”Resolves the pin a bare or latest update moves to on this node, which holds the deployment directory. Writes nothing.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | string | yes | the UpdateRequest, as JSON |
current | string | yes | the deployment’s row, as JSON |
Result:
| Key | Type | Notes |
|---|---|---|
pinned | bool | false when the row ends with no pin |
app_version | string | the pin, empty when pinned is false |
app.update_plan
Section titled “app.update_plan”Plans an update on this node, the render leader, and writes nothing to the deployment: what the update would change.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | string | yes | the UpdateRequest, as JSON |
shipped_app_digest | string | the app archive digest of a path deployment’s folder, interned by the node the plan was typed on; the plan renders it instead of resolving the path |
Result:
| Key | Type | Notes |
|---|---|---|
plan | string | an UpdatePlan, as JSON |
auth.forward_auth_restamp
Section titled “auth.forward_auth_restamp”Starts the leader’s forward-auth redirect-uri restamp for a follower whose completed-login miss detector fired; a node that does not lead runs nothing and relays nothing.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
reason | string | the trigger the follower saw, completed_login_miss |
Result:
| Key | Type | Notes |
|---|---|---|
accepted | bool | false when this node does not lead |
backup
Section titled “backup”backup.capture_atomic
Section titled “backup.capture_atomic”Answers whether this node’s capture of a deployment’s named mounts is one atomic snapshot, which a group backup tells the member’s quiesce before it fires.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | yes | |
env | string | the environment | |
capture_mounts | array | yes | the names the capture reads |
Result:
| Key | Type | Notes |
|---|---|---|
atomic | bool | true when one snapshot here covers every named mount |
backup.converter_capture
Section titled “backup.converter_capture”Runs a backup converter the caller rendered on this node, the one that holds the mounts it claims, over a static copy made under this node’s own freeze of the deployment, and answers its artifact as a staged archive.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | ||
env | string | the environment | |
converter | string | yes | the converter’s name |
rendered | string | yes | the converter as JSON, rendered by the caller; this node renders nothing |
mode | string | the run’s mode: fail aborts the copy when the freeze outlasts timeouts.quiesce_hold, anything else stops the deployment here and copies it halted | |
backup_id | string | the run’s id, for the converter, the stop journal and the journal’s lines | |
unfrozen | bool | the caller froze what is read by other means: nothing is paused here |
Result:
| Key | Type | Notes |
|---|---|---|
outcome | string | produced, not_armed or lost; the caller reads anything else as lost |
error | string | why the converter was not armed or was lost |
exists | bool | |
archive | blob | the converter’s output as a tar, when produced |
databases | string | a JSON list of the databases the artifact carries, when the converter names them |
quiesce | string | how this node froze the deployment for the copy: assured, unsupported or failed; the caller reads a missing one as failed |
left_paused | bool | the release could not confirm every container it froze is running again |
thaw_revived | string | comma-separated replicas that read stopped after the thaw and were started again |
thaw_down | string | comma-separated replicas that read stopped after the thaw and are not running |
stack_stopped | bool | the freeze outlasted its hold and the deployment was stopped here for the copy |
stack_stop_seconds | float64 | how long that stop kept it down; 0 while its start has not verified |
backup.create
Section titled “backup.create”Runs an operator’s backup create on this node, which holds the deployment directory, and answers the archive it recorded.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | string | yes | the BackupCreate, as JSON; stack_name must be set |
relative_paths | string | a JSON list of capture paths relative to the deployment directory, which this node roots in its own | |
relative_excludes | string | a JSON list of exclusions relative to the deployment directory |
Result:
| Key | Type | Notes |
|---|---|---|
backup | string | the recorded BackupModel, as JSON; for a one-off, the archive’s model, which no row records |
archive | blob | a one-off’s .bak, which the caller stages for its export; absent otherwise |
backup.deployment_dir
Section titled “backup.deployment_dir”Answers whether this node holds the deployment’s directory, by the same test the scheduler applies to its own disk.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | yes | |
env | string | the environment |
Result:
| Key | Type | Notes |
|---|---|---|
present | bool |
backup.export
Section titled “backup.export”Exports this node’s copy of a backup and answers it as a staged archive the caller pulls through the transfer layer.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
backup_id | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
archive | blob |
backup.export_staged
Section titled “backup.export_staged”Answers a backup create --file archive this node stages for its export: its tenant and stack, or with take the archive itself, which then leaves this node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
backup_id | string | yes | |
take | bool | hand the archive over instead of describing it |
Result:
| Key | Type | Notes |
|---|---|---|
tenant | string | without take |
stack | string | without take |
environment | string | without take |
archive | blob | with take |
backup.orchestrated
Section titled “backup.orchestrated”Runs an orchestrated backup on this node, which holds the deployment directory, and records the archive. A directory missing here answers X500466BKPPLC.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | string | yes | the CreateBackupRequest, as JSON; stack_name must be set |
Result:
| Key | Type | Notes |
|---|---|---|
backup | string | the recorded BackupModel, as JSON |
backup.ordinal_restore
Section titled “backup.ordinal_restore”Restores the group member of a deployment this node holds: stops its replicas here, replays each directory, empties the held ones the archive does not carry, and starts them again.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | ||
env | string | the environment | |
strategy | string | replace (the default), merge or skip | |
archives | object | each ordinal directory’s tree as a tar blob, by directory name | |
owners | string | JSON: per directory, the owners and modes the archive captured | |
empty | []string | held ordinal directories the archive does not carry |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool |
backup.recover_here
Section titled “backup.recover_here”Runs a recover another node placed on this one (SHC-2584): receives its archives into this node’s spool, points the request at them and runs the whole recover here, data replay included.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | string | JSON: the recover request as the placing node received it, its archives named by their paths there | |
archives | object | each archive the request names as a blob, by its path on the placing node |
Result:
| Key | Type | Notes |
|---|---|---|
result | string | JSON: the recover’s result |
backup.replica_mounts
Section titled “backup.replica_mounts”Answers which replicas of a deployment this node holds and what each binds under the deployment’s mounts directory, read from the replicas’ own documents.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | yes | |
env | string | the environment |
Result:
| Key | Type | Notes |
|---|---|---|
replicas | string | a JSON list of {service, ordinal, mounts}; empty when this node holds no replica of it |
backup.restore
Section titled “backup.restore”Runs an operator’s restore on this node, which holds the deployment directory: stops its replicas here, replays, routes a group member another node holds to that node, and answers what it restored.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | string | yes | the RestoreRequest, as JSON; stack_name must be set |
Result:
| Key | Type | Notes |
|---|---|---|
outcome | string | the RestoreOutcome, as JSON |
backup.restore_file
Section titled “backup.restore_file”Restores a deployment this node holds the directory of from a .bak another node shipped: spools the archive and runs the same restore from file a node holding it runs, and answers the backup it recorded.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | string | yes | the RestoreFromFileRequest without its path, as JSON; stack_name must be set |
path | string | the archive’s path on the sending node, which the backup row names once this node’s spool copy is gone | |
archive | blob | yes | the .bak; only the transfer store’s admission bounds its size |
Result:
| Key | Type | Notes |
|---|---|---|
backup | string | the recorded BackupModel, as JSON |
backup.schedule_op
Section titled “backup.schedule_op”Runs one backup-schedule operation on the leader for a follower that was asked for it.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
op | string | yes | the operation, one of the schedule ops in modules/backup/schedule_leader_forward.go |
params | object | that operation’s own arguments |
Returns: what the named operation answers on the leader.
backup.stack_capture
Section titled “backup.stack_capture”Runs a whole orchestrated backup of a deployment on this node, which holds it, for a scheduler on another node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | yes | |
env | string | the environment | |
consistency_id | string | groups this capture with others taken together | |
name | string | the backup’s name | |
scheduled | bool | true when a schedule fired it | |
pre_uninstall | bool | take an uninstall’s safety backup instead: name, consistency_id and scheduled are not read, and the answer is captured and verdict |
Result:
| Key | Type | Notes |
|---|---|---|
captured | bool | |
backup_id | string | present when a backup row was written |
status | string | that row’s status |
verdict | string | with pre_uninstall, in place of backup_id and status: an UninstallBackupVerdict, as JSON |
backup.stopped_hold
Section titled “backup.stopped_hold”Holds, confirms or releases a stopped deployment’s starts on this node while a backup reads it as a crash image, and answers whether its containers here are stopped.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | yes | |
env | string | yes | the environment |
op | string | yes | hold, confirm or release |
hold_id | string | on hold, the caller’s request id, so a repeat takes no second hold; on confirm and release, the id the hold call answered | |
mark | string | on confirm, the mark hold answered; a different one fails the confirmation |
Result:
| Key | Type | Notes |
|---|---|---|
hold_id | string | |
mark | string | what identifies this node’s stopped containers of the deployment |
stopped | bool | every container of the deployment here is stopped; on confirm, also that the hold held and the mark matches |
backup.verify
Section titled “backup.verify”Verifies a backup against this node’s own disk, for a node asked about an archive whose repository or file is a path on this node. It asks no other node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
backup_id | string | yes | |
deep | bool | read the snapshot’s bytes back |
Result:
| Key | Type | Notes |
|---|---|---|
report | string | the VerifyReport, as JSON |
backup.verify_sweep_report
Section titled “backup.verify_sweep_report”Answers the last deep-verify sweep report this node holds, for a node whose backup.verify_sweep reads the newest one in the cluster. It asks no other node.
Result:
| Key | Type | Notes |
|---|---|---|
path | string | where this node keeps the report |
report | string | the VerifySweepReport, as JSON; absent when no sweep has run on this node |
error | string | why the report could not be read; absent when it could |
Takes no arguments.
backup.volume_capture
Section titled “backup.volume_capture”Tars a mount of a deployment on this node, where its volume is attached or its replica runs, under this node’s own freeze of the deployment, and answers it as a staged archive.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | ||
env | string | the environment | |
mount | string | yes | the mount’s name |
mode | string | the run’s mode: fail aborts the capture when the freeze outlasts timeouts.quiesce_hold, anything else stops the deployment here and captures it halted | |
backup_id | string | the run’s id, for the stop journal and the journal’s lines | |
unfrozen | bool | the caller froze what is read by other means: nothing is paused here |
Result:
| Key | Type | Notes |
|---|---|---|
exists | bool | |
archive | blob | |
quiesce | string | how this node froze the deployment for the capture: assured, unsupported or failed; the caller reads a missing one as failed |
left_paused | bool | the release could not confirm every container it froze is running again |
thaw_revived | string | comma-separated replicas that read stopped after the thaw and were started again |
thaw_down | string | comma-separated replicas that read stopped after the thaw and are not running |
stack_stopped | bool | the freeze outlasted its hold and the deployment was stopped here for the capture |
stack_stop_seconds | float64 | how long that stop kept it down; 0 while its start has not verified |
backup.volume_merge
Section titled “backup.volume_merge”Unpacks a restored archive into a volume-backed mount on this node, where the volume is attached.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | ||
env | string | the environment | |
mount | string | yes | the mount’s name |
archive | blob | yes | the volume’s tree as a tar; only the transfer store’s admission bounds its size |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool |
cert.deliveries
Section titled “cert.deliveries”Answers the certificate deliveries on this node’s disk, one per granted host of the deployments it runs, as their source, classes, issuer and expiry, never key material.
Result:
| Key | Type | Notes |
|---|---|---|
entries | array | the rows shc.cert.delivery lists for this node |
Takes no arguments.
cert.node_leaf
Section titled “cert.node_leaf”Answers this node’s cluster-mTLS leaf (node.pem) as its fingerprint and expiry, never key material.
Result:
| Key | Type | Notes |
|---|---|---|
fingerprint | string | |
not_after | string | RFC 3339 |
Takes no arguments.
certs.info
Section titled “certs.info”Lists this node’s certificate files with their presence, expiry, subject, issuing key id and fingerprints.
Result:
| Key | Type | Notes |
|---|---|---|
entries | []CertEntry | |
entries[].name | string | |
entries[].path | string | |
entries[].present | bool | |
entries[].not_after | string | |
entries[].days_remaining | int | |
entries[].subject | string | |
entries[].authority_key_id | string | |
entries[].cert_fingerprints | []string |
Takes no arguments.
certs.renew
Section titled “certs.renew”Makes this node renew its own internal-listener leaf now and reload it, the step a CA rotation fans out to every node.
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
message | string |
Takes no arguments.
certs.rotate
Section titled “certs.rotate”Signs a fresh leaf for the calling node against the current cluster CA; the caller is the node named by the certificate the call arrived on.
Result:
| Key | Type | Notes |
|---|---|---|
ca_cert | string | |
node_cert | string | |
node_key | string |
Takes no arguments.
clone.clone
Section titled “clone.clone”Runs a clone on this node, the source deployment’s home, for the node that was asked for it.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | StackCloneRequest | yes | the clone as the public API received it |
request.source_stack | string | yes | |
request.target_name | string | yes | |
request.environment | string | ||
request.where | string | ||
request.prefer | string | ||
request.source_environment | string | ||
request.source_tenant | string | ||
request.tenant_name | string | ||
request.all_connections | bool | ||
request.closure | bool | ||
request.rename_map | map[string]string | ||
request.dry_run | bool | ||
request.operation_id | string | ||
request.runtime | string | ||
request.storage | string | ||
caller | object | the asking user’s scope, so the clone runs as them | |
caller.within_tenant | bool | ||
caller.installed_by | string |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
source_scope | ScopeInfo | |
source_scope.tenant_name | string | |
source_scope.environment | string | |
source_scope.node | string | |
target_scope | ScopeInfo | |
target_scope.tenant_name | string | |
target_scope.environment | string | |
target_scope.node | string | |
stacks_cloned | []StackRenameInfo | |
stacks_cloned[].source_name | string | |
stacks_cloned[].target_name | string | |
backup_ids | []string | |
warnings | []string |
clone.finish_stored_render
Section titled “clone.finish_stored_render”Finishes a move’s stored render on the leader under the executor’s relocation lease.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
deployment | DeploymentRef | yes | the deployment being relocated |
deployment.Tenant | string | ||
deployment.Environment | string | ||
deployment.Stack | string | ||
node | string | yes | the target node |
relocation | array | yes | the executor’s relocation claims |
Result:
| Key | Type | Notes |
|---|---|---|
finished | bool |
clone.hold_committed_render
Section titled “clone.hold_committed_render”Has the leader hold a moved deployment’s committed render, put back from the store.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
deployment | DeploymentRef | yes | the deployment that moved |
deployment.Tenant | string | ||
deployment.Environment | string | ||
deployment.Stack | string |
Result:
| Key | Type | Notes |
|---|---|---|
held | bool |
clone.measure_tree
Section titled “clone.measure_tree”Measures a deployment’s tree on this node, or one of its mounts: its bytes and file count.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | yes | |
environment | string | ||
mount | string | one mount’s name; empty measures the whole tree |
Result:
| Key | Type | Notes |
|---|---|---|
bytes | int64 | |
files | int |
clone.move
Section titled “clone.move”Runs a move on this node, the source deployment’s home, for the node that was asked for it.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | StackMoveRequest | yes | the move as the public API received it |
request.source_stack | string | yes | |
request.target_name | string | yes | |
request.node | string | ||
request.where | string | ||
request.prefer | string | ||
request.where_services | []string | ||
request.prefer_services | []string | ||
request.environment | string | ||
request.source_environment | string | ||
request.source_tenant | string | ||
request.tenant_name | string | ||
request.all_connections | bool | ||
request.closure | bool | ||
request.rename_map | map[string]string | ||
request.dry_run | bool | ||
request.operation_id | string | ||
request.runtime | string | ||
request.storage | string | ||
request.expect_source_node | string | ||
request.planned_by | string | ||
request.presumed_dead | bool | ||
request.surge | bool | ||
request.downtime_seconds | int64 | ||
request.no_backup | bool | ||
request.restore_from_backup | bool | ||
request.reclaim | string | ||
caller | object | the asking user’s scope, so the move runs as them | |
caller.within_tenant | bool | ||
caller.installed_by | string |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
source_scope | ScopeInfo | |
source_scope.tenant_name | string | |
source_scope.environment | string | |
source_scope.node | string | |
target_scope | ScopeInfo | |
target_scope.tenant_name | string | |
target_scope.environment | string | |
target_scope.node | string | |
stacks_moved | []StackRenameInfo | |
stacks_moved[].source_name | string | |
stacks_moved[].target_name | string | |
backup_ids | []string | |
used_fast_path | bool | |
warnings | []string | |
strategy | string | |
volumes | []RelocationVolume | |
volumes[].service | string | |
volumes[].name | string | |
volumes[].durable | bool | |
volumes[].action | string | |
volumes[].bytes | int64 | |
volumes[].measured | bool |
clone.recover_one
Section titled “clone.recover_one”Restores the archive the call carries on this node as the named deployment, renders it and brings it up. The archive is pulled before the restore starts; a pull that fails restores nothing.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
archive | blob | yes | the backup archive; only the transfer store’s admission bounds its size |
tenant | string | ||
stack | string | the target deployment’s name | |
environment | string | ||
runtime | string | ||
storage | string | ||
recreate_on_start | bool | ||
relocation | []object | the relocation claims this restore runs under | |
relocation[].deployment | string | yes | |
relocation[].holder | string | yes | |
package_within_tenant | bool | whether the source renders a package from inside the tenant; absent means the sender did not say | |
caller_within_tenant | bool | the asking user’s scope, so the restore runs as them | |
caller_installed_by | string | the asking user | |
stopped | bool | the deployment was stopped before the move began; the restore lands it and starts nothing | |
finish_stored | bool | the deployment moves keeping its identity: the leader finishes its stored render for this node, which holds the restored mounts and replicas and no render |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool |
clone.relocate_one
Section titled “clone.relocate_one”Lands the deployment tree the call carries on this node, its new home. The tree is pulled before anything lands; a pull that fails lands nothing and the answer says so.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
archive | blob | yes | the packed deployment tree; only the transfer store’s admission bounds its size |
tenant | string | ||
stack | string | ||
environment | string | ||
runtime | string | ||
stated_node | string | the node the caller believes this is | |
present | []string | the mounts the move relies on finding on this node; the landing refuses while one holds nothing here | |
stopped | bool | the deployment was stopped before the move began; the landing starts nothing |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool |
clone.render_on_leader
Section titled “clone.render_on_leader”Renders a cloned, recovered or rehomed deployment on the leader for the node it lands on, from the tree its restore laid there.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
render | clonedRender | yes | the render, as the landing node would have run it |
render.tenant | string | ||
render.stack | string | ||
render.environment | string | ||
render.app_name | string | ||
render.runtime | string | ||
render.node | string | ||
render.placement_node | string | ||
render.hosts | []string | ||
render.access | string | ||
render.exposure | string | ||
render.held | []PersistedIntegration | ||
render.held_known | bool | ||
render.storage | string | ||
tree | blob | yes | the landing node’s deployment directory, less its mounts, mark and seal |
relocation | array | yes | the landing node’s relocation claims |
caller | object | the asking user’s scope, so the render runs as them |
Result:
| Key | Type | Notes |
|---|---|---|
rendered | bool | |
bound | array |
clone.replica_transfer
Section titled “clone.replica_transfer”Copies the stopped replica’s durable local volumes and carried bind directories through the clone blob transfer plane to fresh labelled target volumes and marked target directories.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
plan | object | yes | |
target | string | yes | |
generation | uint64 | yes | |
operation | string | yes | |
claims | array | yes |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool |
clone.set_aside
Section titled “clone.set_aside”Sets this node’s directory of a deployment aside as
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | yes | |
environment | string | ||
operation_id | string | yes | the move’s id, one path element |
Result:
| Key | Type | Notes |
|---|---|---|
path | string | where the tree was kept; empty when this node held nothing to keep |
cluster
Section titled “cluster”cluster.logs
Section titled “cluster.logs”Answers the last lines of a service’s log on this node, its log file or else its unit’s journal: the read shc cluster logs <node> routes to the node it names. No lines is X404024CLSLOG naming this node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
service | string | ||
lines | int |
Result:
| Key | Type | Notes |
|---|---|---|
service | string | |
node | string | |
lines | []string |
cluster.networks
Section titled “cluster.networks”Answers this node’s attached networks and routes, its underlay, and the bridges its runtime lists as SHC’s own: what shc cluster block add checks a block against on every node.
Result:
| Key | Type | Notes |
|---|---|---|
networks | []EnrollNetwork | |
networks[].interface | string | |
networks[].cidr | string | |
networks[].own | bool | |
networks[].route | bool | |
networks[].via | string |
Takes no arguments.
cluster.peer_status
Section titled “cluster.peer_status”Answers whether this node is the raft leader, its raft term and the era and generation of the signed membership table it holds. Served only on the peer status port (TCP 4004 at the node’s underlay address, mTLS with a node leaf under the cluster CA, each source rate-limited), never on the internal listener, so a node held off mesh0 can ask before shc cluster repair rebuilds its mesh0 cache. A caller whose leaf names no node of this node’s table is refused.
Result:
| Key | Type | Notes |
|---|---|---|
leader | bool | |
term | uint64 | |
membership | MembershipGeneration | |
membership.era | uint64 | |
membership.generation | uint64 |
Takes no arguments.
cluster.raft_view
Section titled “cluster.raft_view”Answers the raft leader and membership as this node’s engine sees them, the view a store client reads from its list.
Result:
| Key | Type | Notes |
|---|---|---|
leader_id | string | |
leader_addr | string | |
members | []RaftViewMember | |
members[].id | string | |
members[].address | string | |
members[].role | string | |
members[].reachable | bool | |
members[].reachability_known | bool |
Takes no arguments.
cluster.store_client
Section titled “cluster.store_client”Rewrites this node’s cluster_join.json so its next start runs no rqlited and reaches the store through the members sent, in order. A refusal is success=false, never an error.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
members | []Member | ||
members[].node | string | ||
members[].addr | string |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
changed | bool | |
message | string |
cluster.store_replica
Section titled “cluster.store_replica”Rewrites this node’s cluster_join.json so its next start runs rqlited as a non-voter joining the addresses sent. A refusal is success=false, never an error.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
join_addrs | []string |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
changed | bool | |
message | string |
cluster.store_tier
Section titled “cluster.store_tier”Answers the store role this node’s engine runs: voter, replica or client, empty before the engine has started.
Result:
| Key | Type | Notes |
|---|---|---|
tier | string |
Takes no arguments.
cluster.suffrage_nonvoter
Section titled “cluster.suffrage_nonvoter”Rewrites this node’s cluster_join.json so its next raft start joins as a non-voter. A refusal is success=false, never an error.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
join_addrs | []string |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
changed | bool | |
message | string |
cluster.suffrage_restart
Section titled “cluster.suffrage_restart”Schedules the raft engine restart that applies a rewritten join marker, and answers before the restart lands.
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
changed | bool | |
message | string |
Takes no arguments.
cluster.suffrage_voter
Section titled “cluster.suffrage_voter”Rewrites this node’s cluster_join.json so its next raft start joins as a voter. A refusal is success=false, never an error.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
join_addrs | []string |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
changed | bool | |
message | string |
clusterfs
Section titled “clusterfs”clusterfs.build
Section titled “clusterfs.build”Wakes this node’s clusterfs revision loop in the background; concurrent wakes coalesce.
Takes no arguments.
container
Section titled “container”container.fs.create
Section titled “container.fs.create”Creates an empty file inside a container.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | |
path | string | yes | absolute path inside the container |
mode | uint32 | permission bits |
Result:
| Key | Type | Notes |
|---|---|---|
ok | bool |
container.fs.delete
Section titled “container.fs.delete”Removes a path inside a container: a file, or a directory with everything under it.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | |
path | string | yes | absolute path inside the container |
Result:
| Key | Type | Notes |
|---|---|---|
ok | bool |
container.fs.mkdir
Section titled “container.fs.mkdir”Creates a directory inside a container.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | |
path | string | yes | absolute path inside the container |
mode | uint32 | permission bits |
Result:
| Key | Type | Notes |
|---|---|---|
ok | bool |
container.fs.read
Section titled “container.fs.read”Reads a byte range of a file inside a container. A read past the inline limit streams through the transfer layer without buffering.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | |
path | string | yes | absolute path inside the container |
offset | int64 | ||
length | int64 |
Result:
| Key | Type | Notes |
|---|---|---|
data | blob |
container.fs.readdir
Section titled “container.fs.readdir”Lists a directory inside a container on this node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | |
path | string | yes | absolute path inside the container |
Result:
| Key | Type | Notes |
|---|---|---|
entries | []FSDirEntry | |
entries[].name | string | |
entries[].type | string | |
entries[].size | int64 | |
entries[].mode | uint32 |
container.fs.rename
Section titled “container.fs.rename”Renames a path inside a container.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | |
old_path | string | yes | absolute |
new_path | string | yes | absolute |
Result:
| Key | Type | Notes |
|---|---|---|
ok | bool |
container.fs.setattr
Section titled “container.fs.setattr”Changes a path’s mode, its owner or its group inside a container. Each of mode, uid and gid is applied when sent.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | |
path | string | yes | absolute path inside the container |
mode | uint32 | permission bits | |
uid | uint32 | the owner, alone or with gid | |
gid | uint32 | the group, alone or with uid |
Result:
| Key | Type | Notes |
|---|---|---|
ok | bool |
container.fs.stat
Section titled “container.fs.stat”Stats one path inside a container on this node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | |
path | string | yes | absolute path inside the container |
Result:
| Key | Type | Notes |
|---|---|---|
type | string | |
size | int64 | |
mode | uint32 | |
mtime | int64 | |
uid | uint32 | |
gid | uint32 | |
symlink_target | string |
container.fs.symlink
Section titled “container.fs.symlink”Creates a symlink inside a container.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | |
link_path | string | yes | absolute |
target | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
ok | bool |
container.fs.truncate
Section titled “container.fs.truncate”Truncates a file inside a container to a length.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | |
path | string | yes | absolute path inside the container |
length | int64 | yes |
Result:
| Key | Type | Notes |
|---|---|---|
ok | bool |
container.fs.write
Section titled “container.fs.write”Writes bytes into a file inside a container at an offset, streaming the blob onto the write.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | |
path | string | yes | absolute path inside the container |
offset | int64 | ||
data | blob | yes | 32 MiB at most; a larger write is several calls at increasing offsets |
Result:
| Key | Type | Notes |
|---|---|---|
written | int64 |
container.restart
Section titled “container.restart”Restarts a container on this node after inspecting its deployment labels and checking the node agent’s stopped hold.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
container_id | string | yes | the container on this node |
Result:
| Key | Type | Notes |
|---|---|---|
Success | bool | |
ExitCode | int | |
Stdout | string | |
Stderr | string | |
StdoutBytes | bytes | |
StderrBytes | bytes | |
Replicas | []string | |
Members | []string | |
Waiting | bool |
containerbroker
Section titled “containerbroker”containerbroker.run
Section titled “containerbroker.run”Runs a broker container for a deployment on this node, which holds one of its replicas, for the node that was asked for it, through the same gates as a local run.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | string | yes | the RunRequest, as JSON: stack, environment, tenant, image, command, env and timeout_seconds |
Result:
| Key | Type | Notes |
|---|---|---|
truncated | bool | |
exit_code | int64 | |
output | string | |
handle | string | |
status | string |
cron.runs
Section titled “cron.runs”Answers the leader’s cron ledger, each leader-gated cron’s last fire, for a follower’s shc cron ls.
Result:
| Key | Type | Notes |
|---|---|---|
runs | array | one object per cron: module, name, last_run_at (RFC 3339), last_status, last_error |
Takes no arguments.
doctor
Section titled “doctor”doctor.node_checks
Section titled “doctor.node_checks”Runs this node’s node-scoped doctor checks for the leader’s condition sweep and answers their results.
Result:
| Key | Type | Notes |
|---|---|---|
results | array | one doctor result per node-scoped check: name, category, verdict and findings |
Takes no arguments.
doctor.node_run
Section titled “doctor.node_run”Runs this node’s node-scoped doctor checks, diagnose-only, for shc doctor run on another node and answers their results.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
category | string | run only the checks in this category; empty runs every node-scoped check |
Result:
| Key | Type | Notes |
|---|---|---|
results | array | one doctor result per node-scoped check: name, category, verdict, findings and the fix it carries |
edge_ca
Section titled “edge_ca”edge_ca.anchors
Section titled “edge_ca.anchors”Answers the edge CA anchors and the hosts they cover from the node that runs the ingress. A node that holds none answers an empty object.
Result:
| Key | Type | Notes |
|---|---|---|
anchors_pem | string | |
hosts | []string |
Takes no arguments.
events
Section titled “events”events.deliver
Section titled “events.deliver”Hands this node an event another node published, for its live stream and broadcast handlers. An event from this node itself is dropped.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
event | string | yes | the event as a JSON string, so its payload is never read as a binary reference |
source_hostname | string | yes | the node that published it |
source_node_id | string | the id of the node that published it, which decides whether it is this node’s own |
Result:
| Key | Type | Notes |
|---|---|---|
delivered | bool |
events.tail
Section titled “events.tail”Answers the frames this node published to its own bus alone (healthcheck probes, coalesced occurrences) after a cursor, in a stream’s scope, so the node serving the stream can merge them.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
cursor | int64 | the cursor the last answer gave; negative answers the ring’s end and no frames | |
limit | int64 | at most this many frames, capped at 500 | |
recent | bool | answer the newest limit frames instead, for a bounded reading | |
tenant | string | the stream’s tenant | |
all_tenants | bool | the stream reads every tenant | |
stack_scoped | bool | the caller may read only stack_names | |
stack_names | string | comma-separated stacks the caller may read | |
stack_name | string | the stream’s stack | |
node | string | the stream’s node, by name | |
code | string | the one event code the stream keeps | |
types | string | comma-separated event classes | |
errors_only | bool | error frames only | |
quiet | bool | leave out the exec probe triplet |
Result:
| Key | Type | Notes |
|---|---|---|
events | string | |
cursor | int64 | |
missed | int64 | |
source_hostname | string |
export
Section titled “export”export.job
Section titled “export.job”Runs one rendered export or import job on this node, which holds the replica the job’s service runs in, over a directory this node makes for the run, and answers the job’s outcome.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
job | string | yes | the rendered ExporterJobDefinition, as JSON, with @output and @input left for this node to resolve |
direction | string | yes | export or import |
tenant | string | yes | |
stack | string | yes | |
environment | string | yes | |
external_networks | string | a JSON list of the integration networks the deployment’s render names | |
input | blob | with import: the archive of what the job reads from @input | |
output | blob | with export: the archive of what the run’s earlier jobs wrote to @output, when they wrote anything |
Result:
| Key | Type | Notes |
|---|---|---|
status | string | success or failed |
exit_code | int | |
stdout | string | |
stderr | string | |
error | string | |
output | blob | with export, when the job succeeded: the archive of @output as the job left it |
export.run
Section titled “export.run”Runs an operator’s export or import on this node, which holds the deployment’s render. An export keeps its artifact in this node’s own artifacts directory and answers the result and the artifact; an import reads the artifact that came with the call, or this node’s own artifacts directory when none came.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
direction | string | yes | export or import |
request | string | yes | the ExportRequest or ImportRequest, as JSON, held to the HTTP door’s validation; output_dir, input_dir and wrap are not read |
tenant | string | the tenant the operator’s request was scoped to | |
artifact | blob | with import: the .exp the operator named on the node typed on |
Result:
| Key | Type | Notes |
|---|---|---|
result | string | the ExportResult or ImportResult, as JSON |
name | string | when an export succeeded: the artifact’s file name |
artifact | blob | when an export succeeded: the artifact |
fence.reconcile
Section titled “fence.reconcile”Runs a fence reconcile pass on the leader for a follower that was asked for one.
Result:
| Key | Type | Notes |
|---|---|---|
reconciled | bool |
Takes no arguments.
health
Section titled “health”health.node_ledger
Section titled “health.node_ledger”Answers the leader’s node ledger, each row saying which node observed it, for a follower to display.
Result:
| Key | Type | Notes |
|---|---|---|
observed_by | string | |
last_poll | Time | |
nodes | map[string]NodeHealth | |
nodes{}.name | string | |
nodes{}.state | string | |
nodes{}.cpu_percent | float64 | |
nodes{}.memory_percent | float64 | |
nodes{}.disk_percent | float64 | |
nodes{}.inodes_percent | float64 | |
nodes{}.disk_path | string | |
nodes{}.disk_total_bytes | uint64 | |
nodes{}.load_average | float64 | |
nodes{}.last_check | Time | |
nodes{}.docker_ok | bool | |
nodes{}.disk_checks | map[string]float64 | |
nodes{}.fitness_issue | string | |
nodes{}.certificate | NodeCertHealth | |
nodes{}.certificate.not_after | Time | |
nodes{}.certificate.read_error | string | |
nodes{}.certificate.renew_due | bool | |
nodes{}.certificate.consecutive_renew_failures | int | |
nodes{}.certificate.last_renew_error | string | |
nodes{}.certificate.last_renew_success | Time | |
nodes{}.certificate.renewer_running | bool | |
nodes{}.certificate.last_renew_check | Time | |
nodes{}.certificate_issue | string | |
nodes{}.pressure | []Sample | |
nodes{}.pressure[].queue | string | |
nodes{}.pressure[].unit | string | |
nodes{}.pressure[].depth | int64 | |
nodes{}.pressure[].capacity | int64 | |
nodes{}.pressure[].oldest_age | int64 | |
nodes{}.pressure[].in_per_second | float64 | |
nodes{}.pressure[].out_per_second | float64 | |
nodes{}.pressure[].drops_per_second | float64 | |
nodes{}.pressure[].in_total | uint64 | |
nodes{}.pressure[].out_total | uint64 | |
nodes{}.pressure[].dropped_total | uint64 | |
nodes{}.pressure[].level | uint8 | |
nodes{}.pressure[].since | Time | |
nodes{}.pressure[].reason | string | |
nodes{}.pressure[].error | string | |
nodes{}.pressure[].peer | string | |
nodes{}.pressure[].retired_total | uint64 | |
nodes{}.pressure[].retired_reason | string | |
nodes{}.pressure[].stuck_for | int64 | |
nodes{}.pressure[].still_for | int64 | |
nodes{}.pressure[].stale | bool | |
nodes{}.pressure[].amber_depth | int64 | |
nodes{}.pressure[].red_depth | int64 | |
nodes{}.pressure[].amber_age | int64 | |
nodes{}.pressure[].red_age | int64 | |
nodes{}.pressure[].at | Time | |
nodes{}.unmeasured_facts | []string |
Takes no arguments.
job.execute_resolved
Section titled “job.execute_resolved”Runs a leader-resolved hook job on this replica’s node, without rendering its definition.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
spec | ResolvedJobSpec | yes | the resolved job |
spec.Name | string | ||
spec.Kind | string | ||
spec.Context | string | ||
spec.Image | string | ||
spec.Service | string | ||
spec.Command | string | ||
spec.Argv | []string | ||
spec.Env | []string | ||
spec.User | string | ||
spec.WorkingDir | string | ||
spec.Tenant | string | ||
spec.Stack | string | ||
spec.Environment | string | ||
spec.Runtime | string | ||
spec.Target | DeploymentRef | ||
spec.Owner | DeploymentRef | ||
spec.NamesDeployment | DeploymentRef | ||
spec.Fabric | FabricRenderInput | ||
spec.Networks | []string | ||
spec.Volumes | []string | ||
spec.MountsRoot | string | ||
spec.VolumePrep | []PrepOp | ||
spec.Entrypoint | []string | ||
spec.ServiceDocument | bytes | ||
spec.ServiceEntry | string | ||
spec.TimeoutSeconds | int | ||
spec.AdmitSeconds | int | ||
spec.Reach | []string | ||
spec.Prepared | bool | ||
skip_empty_service | bool | skip a pre-update service job when this node reports no container; a count error still runs it |
Result:
| Key | Type | Notes |
|---|---|---|
stdout | string | the job’s stdout, streamed back to the leader |
stderr | string | |
skipped | bool | |
error | object | a coded envelope when the job failed |
failure | string | an integration job’s failure with no code of its own, which the asking runner’s code carries |
job.execute_socket
Section titled “job.execute_socket”Runs a socket’s service job inside its provider’s running container on this node, the node that runs the provider’s service.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
spec | ResolvedJobSpec | yes | the resolved job; Target names the provider |
spec.Name | string | ||
spec.Kind | string | ||
spec.Context | string | ||
spec.Image | string | ||
spec.Service | string | ||
spec.Command | string | ||
spec.Argv | []string | ||
spec.Env | []string | ||
spec.User | string | ||
spec.WorkingDir | string | ||
spec.Tenant | string | ||
spec.Stack | string | ||
spec.Environment | string | ||
spec.Runtime | string | ||
spec.Target | DeploymentRef | ||
spec.Owner | DeploymentRef | ||
spec.NamesDeployment | DeploymentRef | ||
spec.Fabric | FabricRenderInput | ||
spec.Networks | []string | ||
spec.Volumes | []string | ||
spec.MountsRoot | string | ||
spec.VolumePrep | []PrepOp | ||
spec.Entrypoint | []string | ||
spec.ServiceDocument | bytes | ||
spec.ServiceEntry | string | ||
spec.TimeoutSeconds | int | ||
spec.AdmitSeconds | int | ||
spec.Reach | []string | ||
spec.Prepared | bool |
Result:
| Key | Type | Notes |
|---|---|---|
stdout | string | the job’s stdout, streamed back to the node that asked |
stderr | string | |
skipped | bool | always false |
error | object | a coded envelope when the job failed, X501060SVCNCT when the provider has no running container here |
failure | string | an integration job’s failure with no code of its own, which the asking runner’s code carries |
job.prepare_resolved
Section titled “job.prepare_resolved”Readies this replica’s node for a leader-resolved hook job held at its phase’s gate: the job’s preparation and nothing after it, so its run later spends only its command.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
spec | ResolvedJobSpec | yes | the resolved job |
spec.Name | string | ||
spec.Kind | string | ||
spec.Context | string | ||
spec.Image | string | ||
spec.Service | string | ||
spec.Command | string | ||
spec.Argv | []string | ||
spec.Env | []string | ||
spec.User | string | ||
spec.WorkingDir | string | ||
spec.Tenant | string | ||
spec.Stack | string | ||
spec.Environment | string | ||
spec.Runtime | string | ||
spec.Target | DeploymentRef | ||
spec.Owner | DeploymentRef | ||
spec.NamesDeployment | DeploymentRef | ||
spec.Fabric | FabricRenderInput | ||
spec.Networks | []string | ||
spec.Volumes | []string | ||
spec.MountsRoot | string | ||
spec.VolumePrep | []PrepOp | ||
spec.Entrypoint | []string | ||
spec.ServiceDocument | bytes | ||
spec.ServiceEntry | string | ||
spec.TimeoutSeconds | int | ||
spec.AdmitSeconds | int | ||
spec.Reach | []string | ||
spec.Prepared | bool | ||
skip_empty_service | bool | answer skipped when this node reports no container for the service |
Result:
| Key | Type | Notes |
|---|---|---|
skipped | bool | |
error | object | a coded envelope when the preparation failed |
job.register
Section titled “job.register”Registers the jobs declared by this node’s deployment in the shared job table.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | yes | |
environment | string |
Result:
| Key | Type | Notes |
|---|---|---|
registered | int |
logs.snapshot
Section titled “logs.snapshot”Answers a snapshot of a deployment’s logs from this node’s containers, merged by timestamp, for display.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
environment | string | yes | |
stack | string | yes | |
service | string | one service, or all when empty | |
runtime | string | ||
since | string | ||
timestamps | bool | ||
tail | int | lines from the end |
Returns: the log text, a string.
mesh.confirm_restore
Section titled “mesh.confirm_restore”On the leader, previews the peer table a confirmed restore would sign, or confirms one: preview asks for the plan, and from_era is the era the plan named. A follower does not run it.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
preview | bool | ||
from_era | uint64 |
Result:
| Key | Type | Notes |
|---|---|---|
current_era | uint64 | |
new_era | uint64 | |
generation | uint64 | |
mesh_nodes | int | |
clients | int | |
grants | int | |
return | []string | |
vanish | []string | |
clients_return | []string | |
clients_vanish | []string | |
unanswered | []string | |
unverified | []string |
mesh.enrol.sign
Section titled “mesh.enrol.sign”On the leader, signs the peer table now and returns it as JSON text. A follower that answers enrol asks for this, because only the leader holds the snapshot signer. The table is text so its bytes are not lifted into RPC blobs.
Result:
| Key | Type | Notes |
|---|---|---|
snapshot | string |
Takes no arguments.
mesh.peer_table
Section titled “mesh.peer_table”Returns this node’s held leader-signed peer table and fresh mesh0 endpoints to an authenticated member. Served only through the public join route on HTTPS 443; the join route signs the answer.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | object | yes | clustersign envelope over node_id, issued_at and a 32-character lowercase hex nonce, signed by the node identity key under shc-mesh-peer-table-request-v1 followed by a newline |
chain | string | yes | the caller’s node.pem certificate chain |
Returns: a join-signed envelope containing snapshot (the held SignedSnapshot) and endpoints (at most 1024 fresh mesh0 rows).
mesh.punch
Section titled “mesh.punch”The pair owner coordinates a direct endpoint trial through its applied relay. The relay requires fresh kernel handshakes to both ends, stamps their observed mappings and forwards the owner proof. Each end updates only its direct peer endpoint at the requested instant; relay AllowedIPs remain until the existing direct-handshake streak releases them.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
device | string | yes | mesh0 or shc0 |
a | string | yes | lower node id, the owner |
c | string | yes | higher node id |
relay | string | yes | |
generation | int64 | yes | the currently applied relay generation |
at | int64 | yes | unix milliseconds; at most 10 seconds ahead or 30 seconds old |
issued_at | int64 | yes | unix seconds; owner proof freshness is two minutes |
signed | string | yes | base64 owner signature, context shc mesh punch v1 |
cert | string | yes | base64 DER owner certificate |
endpoint_a | string | stamped by the relay on the forwarded call | |
endpoint_c | string | stamped by the relay on the forwarded call |
Result:
| Key | Type | Notes |
|---|---|---|
at | int64 | |
endpoint_a | string | |
endpoint_c | string |
mesh.relay.set
Section titled “mesh.relay.set”Sets or releases the relay of one pair of nodes on a device. The end with the lower id signs the decision and sends it to the relay, which checks it reaches both ends, records the pair and passes the call on to the other end. An end applies a strictly higher generation and runs its sweep, and the same generation renews the relay it holds: the end that is not the owner goes direct once a relay has gone unrenewed for the relay lease (5 minutes), as the relay stops forwarding it. The answer is the generation the node holds for the pair.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
device | string | yes | mesh0 or shc0 |
a | string | yes | one end’s node id |
c | string | yes | the other end’s node id |
relay | string | the relay’s node id; empty releases the pair | |
generation | int64 | yes | |
issued_at | int64 | yes | unix seconds; refused when more than two minutes from the node’s clock |
via | string | the relay the call travels through | |
signed | string | yes | base64 signature of the lower end over the decision |
cert | string | yes | base64 DER certificate of the signer |
Result:
| Key | Type | Notes |
|---|---|---|
generation | int64 |
mesh.rotation.abort
Section titled “mesh.rotation.abort”Aborts the caller node’s open mesh0 epoch without changing its active key.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
node_id | string | ||
device | string | ||
epoch | int | ||
pending_public_key | string | ||
key_digest | string | ||
by | string |
mesh.rotation.ack
Section titled “mesh.rotation.ack”Acknowledges the exact pending key in an applied signed table as the caller’s node leaf.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
node_id | string | ||
device | string | ||
epoch | int | ||
pending_public_key | string | ||
key_digest | string | ||
by | string |
mesh.rotation.prepare
Section titled “mesh.rotation.prepare”Records this authenticated node’s prepared mesh0 key on the leader.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
node_id | string | ||
device | string | ||
epoch | int | ||
pending_public_key | string | ||
key_digest | string | ||
by | string |
mesh.rotation.request
Section titled “mesh.rotation.request”Relays an operator rotation request to the leader.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
node_id | string |
mesh.rotation.start
Section titled “mesh.rotation.start”The leader asks the target to persist its next key and prepare a rotation. The target answers once the rotation has ended: its key is active, or a voter never installed the pending key and nothing switched.
Result:
| Key | Type | Notes |
|---|---|---|
node_id | string | |
device | string | |
epoch | int | |
public_key | string | |
prepared | bool |
Takes no arguments.
mesh.snapshot
Section titled “mesh.snapshot”Answers with the signed peer table this node holds. The asker sends the table it holds (held, era, generation, digest, issued_at; held false asks for the envelope whatever this node holds). held in the answer is false when this node has no store or no table. The envelope is the signed table itself, sent only when this node holds a newer pair, the same pair under another digest, or a later signing of the same table; the same table signed no later, or an older pair, is answered by era, generation and digest alone. The mesh0 sweep asks every peer it reaches every minute, so a cluster that agrees moves no tables.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
held | bool | ||
era | uint64 | ||
generation | uint64 | ||
digest | string | ||
issued_at | int64 |
Result:
| Key | Type | Notes |
|---|---|---|
node_id | string | |
held | bool | |
era | uint64 | |
generation | uint64 | |
digest | string | |
envelope | SignedSnapshot | |
envelope.payload | bytes | |
envelope.alg | string | |
envelope.key_id | string | |
envelope.signature | bytes | |
envelope.certificate | bytes |
mesh.snapshot.apply
Section titled “mesh.snapshot.apply”Verifies and applies a signed peer table without reading the store, and answers only once it is applied, with the public key mesh0 runs under on this node. A leader activates a rotation on that answer, so a table that tells this node its rotation is switching is answered under the new key.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
payload | bytes | ||
alg | string | ||
key_id | string | ||
signature | bytes | ||
certificate | bytes |
Result:
| Key | Type | Notes |
|---|---|---|
applied | bool | |
public_key | string |
mesh.sweep
Section titled “mesh.sweep”Runs this node’s mesh0 sweep now and answers once it is done, so a peer applies a row another node just changed. A node with no sweeper, one not on mesh0 or whose mesh reconcile has stopped, runs nothing and answers swept false with the reason.
Result:
| Key | Type | Notes |
|---|---|---|
swept | bool | true once a sweep ran, false when none did |
reason | string | why no sweep ran, set only when swept is false |
Takes no arguments.
mesh.trial
Section titled “mesh.trial”Takes part in a coordinated trial of the caller’s pair on a device (two-tunnel design section 4). At the instant this node sets the endpoint it holds for the caller, drops every UDP packet to the device’s port that is not from that tuple for the window, and sends three ICMP echoes to the caller’s address on the device. It answers once the window has closed. The caller must be the node named from, and the pair must be direct.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
device | string | yes | mesh0 or shc0 |
from | string | yes | the calling node’s id |
at | int64 | yes | unix milliseconds, within the next 10 seconds |
window_ms | int64 | yes | 1 to 400 |
Result:
| Key | Type | Notes |
|---|---|---|
ok | bool | an echo was answered inside the window |
endpoint | string | the endpoint this node held for the caller |
metrics
Section titled “metrics”metrics.introspect
Section titled “metrics.introspect”Lists the metric definitions this node’s collectors registered, with their cardinality-drop counts.
Result:
| Key | Type | Notes |
|---|---|---|
count | int | |
metrics | []IntrospectEntry | |
metrics[].name | string | |
metrics[].type | string | |
metrics[].help | string | |
metrics[].labels | []string | |
metrics[].buckets | []float64 | |
metrics[].delta_wrapped | bool | |
metrics[].leader_only | bool | |
metrics[].dropped | int |
Takes no arguments.
metrics.node_stats
Section titled “metrics.node_stats”This node’s cached node and container resource stats, the per-node input of the HUD.
Result:
| Key | Type | Notes |
|---|---|---|
nodes | map[string]NodeStats | |
nodes{}.node_id | string | |
nodes{}.cpu_percent | float64 | |
nodes{}.cpu_cores | int | |
nodes{}.memory_percent | float64 | |
nodes{}.memory_used_bytes | int64 | |
nodes{}.memory_total_bytes | int64 | |
nodes{}.memory_available_bytes | int64 | |
nodes{}.disk_percent | float64 | |
nodes{}.disk_used_bytes | int64 | |
nodes{}.disk_total_bytes | int64 | |
nodes{}.load_1m | float64 | |
nodes{}.load_5m | float64 | |
nodes{}.load_15m | float64 | |
nodes{}.docker_ok | bool | |
nodes{}.vault_ok | bool | |
nodes{}.is_leader | bool | |
nodes{}.uptime_seconds | int64 | |
nodes{}.containers_total | int | |
nodes{}.containers_running | int | |
nodes{}.state | string | |
nodes{}.subsystems | map[string]int | |
nodes{}.unmeasured | []string | |
nodes{}.UpdatedAt | Time | |
containers | map[string]ContainerCacheEntry | |
containers{}.container_id | string | |
containers{}.name | string | |
containers{}.service | string | |
containers{}.status | string | |
containers{}.health | string | |
containers{}.node_id | string | |
containers{}.tenant_name | string | |
containers{}.stack_name | string | |
containers{}.environment | string | |
containers{}.image | string | |
containers{}.project | string | |
containers{}.exit_code | int | |
containers{}.restart_count | int | |
containers{}.oom_killed | bool | |
containers{}.cpu_percent | float64 | |
containers{}.memory_bytes | int64 | |
containers{}.memory_limit_bytes | int64 | |
containers{}.network_rx_bytes | int64 | |
containers{}.network_tx_bytes | int64 | |
containers{}.disk_read_bytes | int64 | |
containers{}.disk_write_bytes | int64 | |
containers{}.raw_stats | map[string]any | |
containers{}.host_config | map[string]any | |
containers{}.started_at | string | |
containers{}.UpdatedAt | Time | |
containers{}.stderr_tail | string |
Takes no arguments.
node.enroll
Section titled “node.enroll”Enrols a joining node with a single-use bootstrap token: mints its cluster certificate, records its row and grants its mesh and fabric addresses. Served only on the public join route, POST /api/join/node.enroll through Traefik on 443; the answer is signed with the cluster key.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
bootstrap_token | string | yes | the token shc node token minted; consumed by this call |
node_name | string | yes | |
physical_ip | string | must be the address the request came from, unless public_address is | |
public_address | string | an IP the request came from registers and signs that address instead of physical_ip (a node behind NAT) | |
location | string | ||
provider | string | ||
runtime | string | ||
fabric_public_key | string | ||
cpu_cores | int64 | ||
architecture | string |
Returns: a signed envelope: payload (the enrol answer as JSON, base64), alg, key_id, signature; the joiner verifies it against the cluster CA its token pins.
node.health
Section titled “node.health”This node’s liveness answer: leader status, engine health and a system-metrics snapshot. Exempt from the per-peer in-flight cap, because it is liveness.
Result:
| Key | Type | Notes |
|---|---|---|
node_id | string | |
state | string | |
uptime_seconds | float64 | |
timestamp | string | |
is_leader | bool | |
leader_addr | string | |
cluster_mode | bool | |
docker_ok | bool | |
system_metrics | SystemMetricsPayload | |
system_metrics.cpu_percent | float64 | |
system_metrics.cpu_cores | int | |
system_metrics.memory_percent | float64 | |
system_metrics.memory_used_bytes | uint64 | |
system_metrics.memory_total_bytes | uint64 | |
system_metrics.disk_percent | float64 | |
system_metrics.disk_used_bytes | uint64 | |
system_metrics.disk_total_bytes | uint64 | |
system_metrics.disk_path | string | |
system_metrics.inodes_used | uint64 | |
system_metrics.inodes_total | uint64 | |
system_metrics.inodes_percent | float64 | |
system_metrics.load_average | float64 | |
system_metrics.disk_checks | map[string]float64 | |
system_metrics.unmeasured_facts | []string | |
system_metrics.memory | MeasuredMemory | |
system_metrics.memory.total_bytes | int64 | |
system_metrics.memory.available_bytes | int64 | |
system_metrics.memory.swap_total_bytes | int64 | |
system_metrics.memory.swap_used_bytes | int64 | |
system_metrics.memory.at | Time | |
system_metrics.memory.exhausted | bool | |
system_metrics.memory.since | Time | |
vault_ok | bool | |
certificate | NodeCertHealth | |
certificate.not_after | Time | |
certificate.read_error | string | |
certificate.renew_due | bool | |
certificate.consecutive_renew_failures | int | |
certificate.last_renew_error | string | |
certificate.last_renew_success | Time | |
certificate.renewer_running | bool | |
certificate.last_renew_check | Time | |
pressure | []Sample | |
pressure[].queue | string | |
pressure[].unit | string | |
pressure[].depth | int64 | |
pressure[].capacity | int64 | |
pressure[].oldest_age | int64 | |
pressure[].in_per_second | float64 | |
pressure[].out_per_second | float64 | |
pressure[].drops_per_second | float64 | |
pressure[].in_total | uint64 | |
pressure[].out_total | uint64 | |
pressure[].dropped_total | uint64 | |
pressure[].level | uint8 | |
pressure[].since | Time | |
pressure[].reason | string | |
pressure[].error | string | |
pressure[].peer | string | |
pressure[].retired_total | uint64 | |
pressure[].retired_reason | string | |
pressure[].stuck_for | int64 | |
pressure[].still_for | int64 | |
pressure[].stale | bool | |
pressure[].amber_depth | int64 | |
pressure[].red_depth | int64 | |
pressure[].amber_age | int64 | |
pressure[].red_age | int64 | |
pressure[].at | Time |
Takes no arguments.
node.image_reclaim
Section titled “node.image_reclaim”Runs this node’s image prune for room before an install pulls, or for an operator’s shc node prune, and answers its report.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
request | string | yes | a JSON object: keep, the images the install runs; reclaim, the bytes to give back, 0 for every unused image; dry_run, report and remove nothing |
Result:
| Key | Type | Notes |
|---|---|---|
report | string | the prune’s report as JSON: what it removed, or would, and the unused images it kept by reason |
node.image_room
Section titled “node.image_room”Answers this node’s room for a deployment’s images: the bytes free where its runtime unpacks them, the bytes its unused images would give back, and which of the images it already holds, for the admission that places the deployment.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
images | string | yes | the image refs, as a JSON array |
Result:
| Key | Type | Notes |
|---|---|---|
held | string | the images this node holds, as a JSON array |
free | string | the free bytes, absent when the runtime’s store could not be read |
reclaimable | string | the bytes this node’s image prune would give back keeping the deployment’s images, beside free |
node.leave
Section titled “node.leave”Starts this node’s departure from the cluster in the background and answers at once.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
force | bool |
Result:
| Key | Type | Notes |
|---|---|---|
status | string | |
message | string |
node.removed
Section titled “node.removed”Checks the notice names this node, answers, and stands this node down behind the answer: another node removed it from the cluster.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
node_id | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
status | string | |
message | string |
node.unenroll
Section titled “node.unenroll”Deletes the named node’s record on this node, best effort.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
node_id | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
status | string |
nodelease
Section titled “nodelease”nodelease.renew
Section titled “nodelease.renew”Renews the calling node’s lease on the leader. Exempt from the per-peer in-flight cap, because it is liveness.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
node | string | yes | the renewing node’s id |
boot_id | string | this boot of the renewing node | |
epoch | uint64 | the lease epoch the node last held |
Result:
| Key | Type | Notes |
|---|---|---|
epoch | uint64 | |
ttl_ms | int64 | how long the granted lease lasts |
next_ms | int64 | when to renew next |
operation
Section titled “operation”operation.cancel
Section titled “operation.cancel”Cancels every step of an operation this node runs under its id, tenant and stack, for the node a shc.operation.cancel landed on. Each step ends failed with X499052OPRCNL naming who cancelled it. It asks no other node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
operation_id | string | yes | |
kind | string | the operation’s kind, as X499052OPRCNL names it | |
tenant | string | the operation’s tenant; a step held under another tenant, or under one when this names none, is left running | |
stack | string | the operation’s stack, as X499052OPRCNL names it; it scopes the cancel too, so a step held under another stack is left running | |
actor | string | who asked for the cancel, as X499052OPRCNL names them |
Result:
| Key | Type | Notes |
|---|---|---|
environments | array | the environment of each step this node cancelled; empty when it runs none under that id, tenant and stack, as for an id it never held |
operation.follow
Section titled “operation.follow”Sends an operation this node holds to the node that asked, each frame through that node’s operation.frame, and answers after its terminal frame or once that node takes no more. The call holds one of the asking node’s in-flight slots here for as long as the follow lasts, as app.update_deploy does for a deploy.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
operation_id | string | yes | |
source | string | yes | as operation.state answered it |
tenant | string | as operation.state answered it; another operation under the same id is not sent | |
stack | string | as operation.state answered it | |
after | int | with registry: the last frame the follower has | |
node | string | yes | the node following the operation |
token | string | yes | the relay that node opened for it |
Result:
| Key | Type | Notes |
|---|---|---|
found | bool | false when this node no longer holds that operation |
operation.frame
Section titled “operation.frame”Takes one frame of an operation another node holds, for the stream following it on this node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
token | string | yes | the relay operation.follow named |
frame | string | yes | the frame as JSON: sequence, type and data |
Result:
| Key | Type | Notes |
|---|---|---|
taken | bool | false once the stream has ended; the holder then stops sending |
operation.state
Section titled “operation.state”Answers every operation this node holds in its memory under an id, for a node a read of it landed on that holds none the caller may read. It asks no other node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
operation_id | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
operations | array | empty when this node holds nothing under that id; the operations store’s entry comes before the operation registry’s |
operations[].source | string | store for the operations store, registry for the operation registry |
operations[].tenant | string | |
operations[].stack | string | |
operations[].state | string | the state as shc.operation.state writes it, as JSON |
otel.relay
Section titled “otel.relay”Queues one OTLP export on this node, which holds OpenObserve, for delivery to it. A node that does not hold OpenObserve refuses with X409115OTRMOV and names the holder.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
signal | string | yes | metrics, traces or logs |
body | blob | yes | the OTLP request body, 32 MiB at most |
content_type | string | ||
content_encoding | string | ||
source | string | the node that exported it |
Result:
| Key | Type | Notes |
|---|---|---|
queued | bool |
placement
Section titled “placement”placement.automatic_move
Section titled “placement.automatic_move”Runs an automatic move, a drain’s or a rebalance’s, on the leader for the follower that was asked for it, under the leader’s freeze gate and move budget. A node that does not lead refuses X503798FRZLDR and sends it nowhere.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
deployment | DeploymentSummary | yes | the deployment to move, as the asking node listed it |
deployment.id | string | ||
deployment.name | string | ||
deployment.tenant_name | string | ||
deployment.stack | string | ||
deployment.env | string | ||
deployment.status | string | ||
deployment.runtime | string | ||
deployment.anchored | bool | ||
deployment.observed | bool | ||
deployment.node | string | ||
target | string | yes | the node id it moves to |
service | string | a replica move’s service; empty moves the whole deployment | |
ordinal | int | a replica move’s ordinal | |
planner | string | the planner whose plan the move must still stand on; empty for none | |
caller | object | the asking user’s scope, so the move runs as them | |
caller.within_tenant | bool | ||
caller.installed_by | string |
Result:
| Key | Type | Notes |
|---|---|---|
moved | bool |
pods.replica_move
Section titled “pods.replica_move”Inspects, names the directories it binds under the deployment’s mounts, stops with read-back, starts at a fenced generation, or receives and removes one replica’s local volumes and bind directories.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
verb | string | yes | |
plan | object | yes | |
generation | uint64 | ||
operation | string | ||
volume | string | ||
bind | string | a directory under the deployment’s mounts the plan carries, in place of volume | |
archive | blob | ||
claims | array |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
binds | []string | inspect and binds: the directories the replica’s project binds under this node’s mounts root |
pods.replicas_up
Section titled “pods.replicas_up”Answers how many containers of a deployment’s replica projects have not exited on this node, for a failed install’s rollback that waits for them to stop.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
environment | string | yes | |
stack | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
up | int | containers that have not exited; an engine that cannot be asked is an error, never zero |
pods.service_verb
Section titled “pods.service_verb”Starts or stops every local replica holding a hook action’s service and returns the first coded refusal.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
ref | DeploymentRef | yes | the deployment |
ref.Tenant | string | ||
ref.Environment | string | ||
ref.Stack | string | ||
service | string | yes | the logical service |
stop | bool | yes |
Result:
| Key | Type | Notes |
|---|---|---|
applied | bool |
proxy.vips
Section titled “proxy.vips”Answers the leader’s read of a load balancer’s VIPs for a node that is not the leader; a node that does not lead refuses with X503984PRXLDR and reads nothing.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | the tenant of the load balancer’s deployment | |
stack | string | the deployment’s stack | |
conn | string | the proxy connection | |
fresh | bool | skip the leader’s kept ready answer |
Result:
| Key | Type | Notes |
|---|---|---|
vips | array | the VIPs, empty when not ready |
ready | bool | false when the box is not ready |
not_ready | string | the box’s reason when ready is false |
quota.node_cap
Section titled “quota.node_cap”Caps the directories of the deployments named at the bytes given, through the filesystem’s own quota where it has one (an xfs project on a prjquota mount, a zfs dataset of its own, a btrfs subvolume with quotas on), and answers how each was capped. Zero bytes takes off only a cap SHC set. The disk sweep sends each node the caps of the deployments it holds, with the sizes of their node-disk volumes from the node whose manifest gives them, so a node holding the volumes without the render caps them too.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
caps | string | a JSON array of {tenant, environment, deployment, bytes, volumes, volumes_known} |
Result:
| Key | Type | Notes |
|---|---|---|
results | string | a JSON array of {path, bytes, applied, hard, how}, one entry per directory capped |
quota.node_disk
Section titled “quota.node_disk”Measures this node’s own deployment tree and answers the bytes each deployment holds here, without crossing into another filesystem, so a block volume bound onto a mount is left out. The disk sweep asks every node and sums the answers per tenant, environment and deployment.
Result:
| Key | Type | Notes |
|---|---|---|
deployments | string | a JSON array of {tenant, environment, deployment, bytes, volumes, rendered}, one entry per deployment directory on this node; volumes are the node-disk volume sizes its manifest gives, rendered says this node holds that manifest |
Takes no arguments.
replica_label
Section titled “replica_label”replica_label.report
Section titled “replica_label.report”A node reports the live label values that changed for replicas it runs; the leader writes them.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
changes | []Change | ||
changes[].tenant | string | ||
changes[].environment | string | ||
changes[].stack_name | string | ||
changes[].service_name | string | ||
changes[].ordinal | int | ||
changes[].key | string | ||
changes[].value | string | ||
changes[].absent | bool | ||
changes[].node_id | string | ||
changes[].routed | bool |
shell.resolve_container
Section titled “shell.resolve_container”Answers the id of a running container for a deployment’s service on this node, for a peer that will bridge an interactive exec there.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | ||
stack | string | yes | |
environment | string | ||
service | string | empty means the deployment’s first running container on this node |
Result:
| Key | Type | Notes |
|---|---|---|
container_id | string |
stack.compose
Section titled “stack.compose”Runs a compose command against a deployment on this node, which holds it.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
target | StackTargetDescriptor | the deployment and the node it is routed to | |
target.Name | string | ||
target.TenantName | string | ||
target.Environment | string | ||
target.NodeID | string | ||
target.AppName | string | ||
target.Guarded | bool | ||
target.OneNode | bool | ||
target.where | string | ||
argv | []string | yes | the compose arguments, at least one |
bytes | bool | an exec answers its streams as StdoutBytes and StderrBytes, with Stdout and Stderr empty; without it, as the strings alone, which a caller before 0.1.24 reads |
Result:
| Key | Type | Notes |
|---|---|---|
Success | bool | |
ExitCode | int | |
Stdout | string | |
Stderr | string | |
StdoutBytes | bytes | |
StderrBytes | bytes | |
Replicas | []string | |
Members | []string | |
Waiting | bool |
stack.exec_target
Section titled “stack.exec_target”Answers where a streamed shc exec of a deployment runs on this node, which holds its replica: the container, the runtime that runs it and the command, for the node the client reached, which then runs it here through the interactive exec route.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
target | StackTargetDescriptor | the deployment and the node it is routed to | |
target.Name | string | ||
target.TenantName | string | ||
target.Environment | string | ||
target.NodeID | string | ||
target.AppName | string | ||
target.Guarded | bool | ||
target.OneNode | bool | ||
target.where | string | ||
argv | []string | yes | the exec as compose takes it: exec -T [--user U] [--workdir W] SERVICE CMD…, SERVICE empty for the deployment’s first running container |
Result:
| Key | Type | Notes |
|---|---|---|
container_id | string | |
runtime | string | |
command | []string | |
user | string | |
workdir | string | |
missing | string |
stack.inspect
Section titled “stack.inspect”Answers shc inspect for a deployment from this node, which holds its directory, redacted here against the files it read.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
target | string | yes | the StackTargetDescriptor, as JSON |
Result:
| Key | Type | Notes |
|---|---|---|
inspect | string | the StackInspectResponse, as JSON |
stack.inspect_containers
Section titled “stack.inspect_containers”Lists this node’s containers of a deployment for the inspect fan-out that follows assignment rows.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
target | string | yes | the StackTargetDescriptor, as JSON |
Result:
| Key | Type | Notes |
|---|---|---|
containers | string | the []StackContainerView, as JSON |
stack.install
Section titled “stack.install”Installs a deployment on this node for the node that routed the request here.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
target | StackTargetDescriptor | the deployment and the node it is routed to | |
target.Name | string | ||
target.TenantName | string | ||
target.Environment | string | ||
target.NodeID | string | ||
target.AppName | string | ||
target.Guarded | bool | ||
target.OneNode | bool | ||
target.where | string | ||
request | StackInstallRequest | the install as the public API received it; the deployment is named by target | |
request.stack_name | string | ||
request.environment | string | ||
request.tenant_name | string | ||
request.node_id | string | ||
request.app_name | string | ||
request.runtime | string | ||
request.compose | string | ||
request.timeout_sec | int | ||
request.operation_id | string | ||
request.integrations | []string | ||
request.variables | map[string]string | ||
request.app_dir | string | ||
request.stored_app_digest | string | ||
request.ports | []string | ||
request.profiles | []string | ||
request.app_version | string | ||
request.repo_url | string | ||
request.labels | map[string]string | ||
request.domain | string | ||
request.dns_provider | string | ||
request.acme_provider | string | ||
request.recovery | bool | ||
request.effective_vars | string | ||
request.backed_up_vars | map[string]string | ||
request.groups | map[string][]Member | ||
request.package_from_archive | bool | ||
request.stated_node_id | string |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
message | string | |
exit_code | int | |
stdout | string | |
stderr | string | |
deployment_dir | string | |
bound | []StackBinding | |
bound[].plug | string | |
bound[].provider_stack_name | string | |
bound[].provider_tenant_name | string | |
bound[].provider_environment | string | |
bound[].provider_connection | string | |
scratch_notes | []string | |
set_aside_path | string | |
set_aside_error | string | |
proxied_ports | []StackProxiedPort | |
proxied_ports[].connection | string | |
proxied_ports[].external | int | |
proxied_ports[].protocol | string | |
proxied_ports[].node_port | int | |
proxied_ports[].hostname | string | |
proxied_ports[].address | string | |
proxied_ports[].proxy_protocol | bool | |
published_ports | []Port | |
published_ports[].service | string | |
published_ports[].container | int | |
published_ports[].port | int | |
published_ports[].protocol | string | |
published_ports[].connection | string | |
published_ports[].listen | string | |
published_ports[].spread | bool | |
published_ports[].hostname | string | |
published_ports[].local | bool | |
published_ports[].private | bool | |
published_ports[].proxy_protocol | bool | |
published_ports[].published_only | bool |
stack.resolve
Section titled “stack.resolve”Answers the deployment’s descriptor if this node holds it.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | ||
environment | string | default when empty |
Result:
| Key | Type | Notes |
|---|---|---|
stack_name | string | |
tenant_name | string | |
environment | string | |
node_id | string | |
app_name | string | |
runtime | string |
Returns: null when this node does not hold the deployment.
stack.uninstall
Section titled “stack.uninstall”Tears a deployment down on this node for the node that routed the request here.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
target | StackTargetDescriptor | the deployment and the node it is routed to | |
target.Name | string | ||
target.TenantName | string | ||
target.Environment | string | ||
target.NodeID | string | ||
target.AppName | string | ||
target.Guarded | bool | ||
target.OneNode | bool | ||
target.where | string | ||
request | StackUninstallRequest | the uninstall as the public API received it; the deployment is named by target | |
request.stack_name | string | ||
request.environment | string | ||
request.tenant_name | string | ||
request.remove_volumes | bool | ||
request.drop_data | bool | ||
request.operation_id | string | ||
request.node_id | string | ||
request.keep_record | bool | ||
request.keep_dns | bool | ||
request.scratch_volumes | []string | ||
request.set_aside | string | ||
request.keep_render | bool | ||
require_local_state | bool | refuse when this node holds no deployment directory, instead of tearing down nothing |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
message | string | |
exit_code | int | |
stdout | string | |
stderr | string | |
deployment_dir | string | |
bound | []StackBinding | |
bound[].plug | string | |
bound[].provider_stack_name | string | |
bound[].provider_tenant_name | string | |
bound[].provider_environment | string | |
bound[].provider_connection | string | |
scratch_notes | []string | |
set_aside_path | string | |
set_aside_error | string | |
proxied_ports | []StackProxiedPort | |
proxied_ports[].connection | string | |
proxied_ports[].external | int | |
proxied_ports[].protocol | string | |
proxied_ports[].node_port | int | |
proxied_ports[].hostname | string | |
proxied_ports[].address | string | |
proxied_ports[].proxy_protocol | bool | |
published_ports | []Port | |
published_ports[].service | string | |
published_ports[].container | int | |
published_ports[].port | int | |
published_ports[].protocol | string | |
published_ports[].connection | string | |
published_ports[].listen | string | |
published_ports[].spread | bool | |
published_ports[].hostname | string | |
published_ports[].local | bool | |
published_ports[].private | bool | |
published_ports[].proxy_protocol | bool | |
published_ports[].published_only | bool | |
retained_cleanup | []RetainedCleanupJob | cleanup jobs the teardown left behind, when there are any |
retained_cleanup[].stack | string | |
retained_cleanup[].job | string | |
retained_cleanup[].verb | string | |
retained_cleanup[].code | string | |
retained_cleanup[].detail | string |
storage
Section titled “storage”storage.cephfs_disable
Section titled “storage.cephfs_disable”Unmounts CephFS on this node. The caller’s certificate must name target_node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
target_node | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
message | string |
storage.cephfs_enable_client
Section titled “storage.cephfs_enable_client”Mounts the cluster CephFS on this node. The caller’s certificate must name target_node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
mount_point | string | yes | |
monitors | []string | yes | |
client_name | string | yes | |
secret | string | yes | |
fsid | string | ||
target_node | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
message | string |
storage.cephfs_enroll_host
Section titled “storage.cephfs_enroll_host”Installs the cephadm public key the leader sends, so this node can be added as a Ceph host.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
pub_key | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
ok | bool | |
hostname | string |
storage.nfs_disable
Section titled “storage.nfs_disable”Tears down this node’s NFS server or client role. The caller’s certificate must name target_node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
role | string | ||
target_node | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
message | string |
storage.nfs_enable_client
Section titled “storage.nfs_enable_client”Mounts the cluster NFS share on this node. The caller’s certificate must name target_node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
server_address | string | yes | |
export_path | string | yes | |
mount_point | string | yes | |
target_node | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
message | string |
storage.nfs_enable_server
Section titled “storage.nfs_enable_server”Sets up the NFS kernel server on this node and exports the share. The caller’s certificate must name target_node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
export_path | string | yes | |
allowed_networks | []string | yes | |
take_over | bool | ||
bind | string | ||
target_node | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
message | string | |
bind_ip | string |
storage.nfs_status
Section titled “storage.nfs_status”Reports this node’s NFS state: installed, running, exporting, mounted.
Result:
| Key | Type | Notes |
|---|---|---|
is_server | bool | |
is_client | bool | |
is_installed | bool | |
is_running | bool | |
exports_count | int | |
mount_active | bool | |
error | string |
Takes no arguments.
storage.shared_op
Section titled “storage.shared_op”Runs one shared-storage operation on the leader for a follower that was asked for it.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
op | string | yes | the operation, one of the SharedOp names in modules/storage/leader_forward.go |
params | object | that operation’s own arguments |
Returns: what the named operation answers on the leader.
storage.volume_grow
Section titled “storage.volume_grow”Grows the filesystem on a block volume attached to this node after its provider grew the volume, after checking the caller still holds the volume’s lease generation. The caller’s certificate must name target_node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
lease_generation | int64 as a decimal string | ||
volume_id | string | yes | |
size_bytes | int64 as a decimal string | ||
target_node | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
size_bytes | int64 as a decimal string |
storage.volume_mount
Section titled “storage.volume_mount”Mounts an attached block volume on this node, after checking the caller still holds the volume’s lease generation. The caller’s certificate must name target_node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
device_path | string | yes | |
mountpoint | string | yes | |
filesystem | string | ||
lease_generation | int64 as a decimal string | ||
volume_id | string | ||
target_node | string | yes | |
created | bool |
Result:
| Key | Type | Notes |
|---|---|---|
message | string |
storage.volume_unmount
Section titled “storage.volume_unmount”Unmounts a block volume on this node. The caller’s certificate must name target_node.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
mountpoint | string | yes | |
volume_id | string | ||
target_node | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
message | string |
storage.volume_writers
Section titled “storage.volume_writers”Answers which containers of a deployment have not exited on this node, before a takeover moves its volume away.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | ||
environment | string |
Result:
| Key | Type | Notes |
|---|---|---|
up | []string | container ids, when the node could tell |
undetermined | string | the reason, when it could not |
store.execute
Section titled “store.execute”Runs a store client’s write on this node’s store, which forwards it to the leader: one statement alone, several as one atomic batch.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
statements[].sql | string | yes | |
statements[].args | []any | written as store.query’s args are |
Result:
| Key | Type | Notes |
|---|---|---|
results[].rows_affected | int64 | |
results[].last_insert_id | int64 |
store.query
Section titled “store.query”Runs a store client’s read on this node’s store: from this node’s copy under the caller’s bound when the read is marked local, else at weak level by the leader.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
sql | string | yes | |
args | []any | each nil, raw bytes, or a string whose first byte names its type | |
bound | string | how far behind the leader this node’s copy may be, empty for a weak read |
Result:
| Key | Type | Notes |
|---|---|---|
columns | []string | |
rows | [][]any | cells written as args are |
transfer
Section titled “transfer”transfer.release
Section titled “transfer.release”Drops a staged transfer the caller has finished pulling, so the holder frees its disk before the handle expires.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
id | string | the transfer handle id; an empty or unknown id is a no-op |
underlay
Section titled “underlay”underlay.members
Section titled “underlay.members”Lists the cluster’s members as they are now, so a joining node can try a handshake with each of them in both directions before it enrols (shc node join). Served only on the public join route, POST /api/join/underlay.members through Traefik on 443; the answer is signed with the cluster key.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
bootstrap_token | string | a joiner’s: the token shc node token minted, checked and not consumed | |
request | object | a member’s: the signed request underlay.probe.arm takes, made for this function | |
chain | string | a member’s node.pem certificate chain, with request |
Returns: a signed envelope: node, the member that answered, and members (node, underlay, public, devices (device, port, public_key), reached: the devices on which the member that answered holds a handshake with it inside three minutes).
underlay.probe.arm
Section titled “underlay.probe.arm”Holds a UDP probe’s throwaway WireGuard public keys as peers of this node’s mesh0 and shc0, with no allowed IPs, for at most 15 seconds, so the caller can try a handshake on each device’s real port before a join (shc node join) or in doctor (underlay.udp). Served only on the public join route, POST /api/join/underlay.probe.arm through Traefik on 443; the answer is signed with the cluster key.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
bootstrap_token | string | a joiner’s: the token shc node token minted, checked and not consumed | |
request | object | a member’s: clustersign envelope over node_id, issued_at, a 32-character lowercase hex nonce, function and the fields below, signed by the node identity key under shc-underlay-probe-request-v1 followed by a newline | |
chain | string | a member’s node.pem certificate chain, with request | |
keys | object | yes | device name (mesh0, shc0) to the probe’s base64 public key for it; inside request for a member |
dial | object | a joiner’s only: device name to key and endpoint, a second base64 public key this node holds as a peer and the joiner’s underlay address and port, which this node’s device sends its own handshake to |
Returns: a signed envelope: id, node, devices (device, port, public_key, dialing) held, and refused (device to the reason this node cannot hold a key on it).
underlay.probe.relay.arm
Section titled “underlay.probe.relay.arm”Arms another member for a joiner whose own dial to that member’s join route does not answer: this node checks the token, then asks the member over mesh0 to hold the probe’s keys as underlay.probe.arm would (shc node join). Served only on the public join route, POST /api/join/underlay.probe.relay.arm through the edge on 443; the answer is signed with the cluster key.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
bootstrap_token | string | yes | the joiner’s token shc node token minted, checked and not consumed; a relay carries a joiner’s probe only |
node | string | yes | the member to arm, by node id, one of the cluster’s live mesh0 rows |
keys | object | yes | device name (mesh0, shc0) to the probe’s base64 public key for it |
dial | object | device name to key and endpoint the member’s device sends its own handshake to, as underlay.probe.arm takes it |
Returns: a signed envelope: the member’s underlay.probe.arm answer.
underlay.probe.relay.result
Section titled “underlay.probe.relay.result”Reads a relayed probe’s result from the member it armed, and releases the keys. Served only on the public join route, POST /api/join/underlay.probe.relay.result through the edge on 443; the answer is signed with the cluster key.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
bootstrap_token | string | yes | the joiner’s token shc node token minted, checked and not consumed; a relay carries a joiner’s probe only |
node | string | yes | the member to arm, by node id, one of the cluster’s live mesh0 rows |
id | string | yes | the id underlay.probe.relay.arm answered |
Returns: a signed envelope: the member’s underlay.probe.result answer.
underlay.probe.relayed.arm
Section titled “underlay.probe.relayed.arm”Holds a joiner’s probe keys that the member it joins through relays here, as underlay.probe.arm does for the joiner itself.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
caller | string | yes | who the probe is held for: the relaying member and a digest of the joiner’s token |
keys | object | yes | device name to the probe’s base64 public key |
dial | object | device name to key and endpoint, as underlay.probe.arm takes it |
Returns: the underlay.probe.arm answer.
underlay.probe.relayed.result
Section titled “underlay.probe.relayed.result”Reads and releases a relayed probe, for the member that armed it.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
caller | string | yes | the caller the probe was armed for |
id | string | yes | the id underlay.probe.relayed.arm answered |
Returns: the underlay.probe.result answer.
underlay.probe.result
Section titled “underlay.probe.result”Reads the bytes each device received from and sent to a probe’s key and whether a handshake completed, then releases the keys. Served only on the public join route, POST /api/join/underlay.probe.result through Traefik on 443; the answer is signed with the cluster key.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
bootstrap_token | string | a joiner’s: the token shc node token minted, checked and not consumed | |
request | object | a member’s: clustersign envelope over node_id, issued_at, a 32-character lowercase hex nonce, function and the fields below, signed by the node identity key under shc-underlay-probe-request-v1 followed by a newline | |
chain | string | a member’s node.pem certificate chain, with request | |
id | string | yes | the id underlay.probe.arm answered; inside request for a member |
Returns: a signed envelope: id and devices (device, port, received, sent, handshake, and dialing, back_received, back_sent, back_handshake for a device that dialled).
vault.coldstart_plant
Section titled “vault.coldstart_plant”Writes this node’s auto-unlock drop-in, once the pushed password matches the cluster’s replicated verifier.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
password | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
message | string |
vault.coldstart_uproot
Section titled “vault.coldstart_uproot”Removes this node’s auto-unlock drop-in.
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
message | string |
Takes no arguments.
vault.rekey
Section titled “vault.rekey”Moves this node’s in-memory vault key to the rotated password, after validating it against the replicated verifier. A rejected re-key is an error, never a reply.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
new_password | string | yes | |
adopt_drop_in | bool |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
message | string |
vault.request_unlock
Section titled “vault.request_unlock”The knock: a locked node asks this unlocked node to push the unlock straight back to the address it names.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
node_id | string | yes | |
address | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
message | string |
vault.unlock
Section titled “vault.unlock”Unlocks this node’s vault with the pushed password. An already unlocked vault answers success and does not re-derive its key.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
password | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
success | bool | |
message | string |
vip.published_probe
Section titled “vip.published_probe”Dials each published port on this node’s own address once its fabric policy carries the revision, and says which answer.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
targets | []ProbeTarget | yes | the published TCP ports this node listens for |
targets[].port | int | ||
targets[].address | string | ||
targets[].policy | bool | ||
revision | uint64 | the clusterfs revision the ports were recorded at; zero asks nothing of the fabric policy |
Result:
| Key | Type | Notes |
|---|---|---|
answers | []ProbeAnswer | one per target, in order |
answers[].port | int | |
answers[].address | string | |
answers[].answers | bool | |
answers[].unprobed | bool | |
answers[].policy | bool | |
answers[].reason | string |
volume
Section titled “volume”volume.files.get
Section titled “volume.files.get”Reads a file, or the byte range a Range header names, from a deployment volume this node holds.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | yes | |
env | string | yes | |
volume | string | yes | the mount name the app declares |
block | bool | a block volume, read only while it is mounted here | |
path | string | yes | slash path inside the volume |
range | string | an HTTP Range header, bytes=a-b |
Result:
| Key | Type | Notes |
|---|---|---|
data | blob | |
entry | object | |
offset | int64 | |
length | int64 | |
partial | bool |
volume.files.held
Section titled “volume.files.held”Says whether this node’s disk lays out a deployment volume.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | yes | |
env | string | yes | |
volume | string | yes | the mount name the app declares |
block | bool | a block volume, read only while it is mounted here |
Result:
| Key | Type | Notes |
|---|---|---|
held | bool |
volume.files.ls
Section titled “volume.files.ls”Lists one page of a folder of a deployment volume this node holds, sorted whole first.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | yes | |
env | string | yes | |
volume | string | yes | the mount name the app declares |
block | bool | a block volume, read only while it is mounted here | |
path | string | yes | slash path inside the volume |
offset | int64 | where the page starts in the sorted folder | |
limit | int64 | the most entries the page holds, at most 5000 |
Result:
| Key | Type | Notes |
|---|---|---|
tenant | string | |
stack | string | |
environment | string | |
volume | string | |
path | string | |
node | string | |
node_name | string | |
entries | []VolumeFileEntry | |
entries[].name | string | |
entries[].path | string | |
entries[].kind | string | |
entries[].size | int64 | |
entries[].modified | Time | |
entries[].mode | string | |
entries[].target | string | |
total | int | |
offset | int | |
next | int | |
truncated | bool |
volume.files.mounts
Section titled “volume.files.mounts”Names the volumes a deployment lays out on this node’s disk.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | yes | |
env | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
volumes | []string |
volume.files.mv
Section titled “volume.files.mv”Renames a file or folder inside a deployment volume this node holds.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | yes | |
env | string | yes | |
volume | string | yes | the mount name the app declares |
block | bool | a block volume, read only while it is mounted here | |
from | string | yes | |
to | string | yes |
Result:
| Key | Type | Notes |
|---|---|---|
name | string | |
path | string | |
kind | string | |
size | int64 | |
modified | Time | |
mode | string | |
target | string |
volume.files.put
Section titled “volume.files.put”Writes a file into a deployment volume this node holds, renamed into place once whole.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | yes | |
env | string | yes | |
volume | string | yes | the mount name the app declares |
block | bool | a block volume, read only while it is mounted here | |
path | string | yes | slash path inside the volume |
overwrite | bool | ||
data | blob | yes |
Result:
| Key | Type | Notes |
|---|---|---|
name | string | |
path | string | |
kind | string | |
size | int64 | |
modified | Time | |
mode | string | |
target | string |
volume.files.rm
Section titled “volume.files.rm”Deletes a file or folder from a deployment volume this node holds.
Arguments:
| Key | Type | Required | Notes |
|---|---|---|---|
tenant | string | yes | |
stack | string | yes | |
env | string | yes | |
volume | string | yes | the mount name the app declares |
block | bool | a block volume, read only while it is mounted here | |
path | string | yes | slash path inside the volume |
recursive | bool |
Result:
| Key | Type | Notes |
|---|---|---|
path | string |