Skip to content
SHC Docs

Status

shc status is the first thing to type when something seems wrong, and the last thing to type after a change. It answers for the tenant you have selected, prod, and lists the platform’s own deployments in master beside yours. Run it again if a queue is not green yet: a queue that held deliveries while obs01 joined clears within a few minutes.

home01
$ shc status
last polled 2026-10-09 06:53:04 UTC
NODE STATE CPU% MEM% DISK% LOAD
home01 healthy 47.5 37.6 17.9 1.82
obs01 healthy 4.2 5.4 2.5 0.14
all 284 queues green on 2 nodes
TENANT STACK ENVIRONMENT STATE SERVICES NODES LABELS NO HEALTHCHECK
master keycloak default healthy 1/1 home01 app=keycloak environment=default stack=keycloak tenant=master -
master openobserve default healthy 1/1 obs01 app=openobserve environment=default stack=openobserve tenant=master 1
master postgres default healthy 2/2 home01 app=postgres environment=default stack=postgres tenant=master -
master traefik default healthy 15/15 home01, obs01 app=traefik environment=default stack=traefik tenant=master -
prod cloud prod healthy 4/4 home01 app=nextcloud environment=prod stack=cloud tenant=prod 1
prod db prod healthy 2/2 home01 app=postgres environment=prod stack=db tenant=prod -
prod pgweb prod healthy 1/1 home01 app=/root/learn/pgweb environment=prod stack=pgweb tenant=prod -
prod placed prod healthy 1/1 home01 app=notes environment=prod stack=placed tenant=prod tier=learn -
prod store prod healthy 1/1 home01 app=/root/learn/versitygw environment=prod stack=store tenant=prod -
prod web prod healthy 1/1 home01 app=/root/learn/nginx environment=prod stack=web tenant=prod 1
Published ports
master/default/traefik traefik 80/tcp listen=node=@(service=traefik) nodes: home01 local, obs01 local
master/default/traefik traefik 443/tcp listen=node=@(service=traefik) nodes: home01 local, obs01 local
...
2 nodes, 10 deployments

Three tables, the nodes, the queues and the deployments, then the ports the deployments publish and a summary line.

The first table is the nodes. STATE is what the leader’s last probe found: healthy, degraded when the node is short of disk or memory, unhealthy when it stopped answering, and the row stays until the node is forgotten. CPU%, MEM%, DISK% and LOAD are the node’s own last readings. The last polled line above the table is when the leader last gathered them; a stale time means the leader’s poll is not running.

Between the nodes and the deployments come the queues. Every queue in the daemon grades itself green, amber or red against its own thresholds. When all are green one line says so and counts them; a queue that is not green gets a row in a table here, naming the node, the queue, its level, its depth, the reason, its rates and what it dropped. A node’s event broadcast queue reads amber for a while after a peer joined and dropped a delivery, then clears on its own. shc doctor raises the same queues as findings.

The deployments table is next. STATE is the rollup of the deployment’s services: healthy when every service runs and passes its healthcheck, degraded when some do, unhealthy when none do, unknown when the node that runs it cannot be reached. SERVICES is the healthy count over the total. A service that declares no healthcheck counts as healthy and is counted again in a NO HEALTHCHECK column when the deployment has one; a running container is not a liveness claim. A service that keeps restarting is excluded from the healthy count and the deployment reads degraded with the reason beside it.

Published ports lists every port a deployment publishes, which nodes answer on it and whether each node serves it locally or crosses the fabric to the node that runs it. The summary line counts what the tables showed.

Name a deployment and the answer is about that one, with its replicas, the constraints that placed it, the nodes it may run on and where it could move.

home01
$ shc status db
last polled 2026-10-11 02:41:01 UTC
NODE STATE CPU% MEM% DISK% LOAD
home01 healthy 37.1 25.0 13.6 2.67
obs01 healthy 18.0 6.1 2.2 0.50
all 108 queues green on 2 nodes
TENANT STACK ENVIRONMENT STATE SERVICES NODE LABELS
prod db prod healthy 1/1 home01 app=postgres environment=prod stack=db tenant=prod
TENANT STACK ENVIRONMENT REPLICA NODE STATE LABELS
prod db prod postgres-1-r0 home01 ready role=primary
Groups
GROUP MEMBER LOCATION READY ALIAS SERVICES
postgres 1 learn/home ready postgres postgres-1
Aliases
TENANT STACK ENVIRONMENT ALIAS SELECTS REPLICAS REASON
prod db prod postgres role=primary:1 postgres-1-r0 -
prod db prod postgres-ro role=replica - no replica matches role=replica
runs on home01; can move to obs01
2 nodes, 1 deployment

The replica table is one row per copy of each service. role=primary is the live label Postgres measures on the member that takes writes. The Groups section lists the deployment’s groups, one row a member, and names the member the alias under each group’s name points at. The Aliases section lists the names the deployment answers under besides its services’ own: postgres goes to the one replica that says primary, and postgres-ro has no replica to go to at one member, which its row says.

-a is every deployment of every tenant, and -t narrows it to one. --all-tenants is the wider count shc ls --all-tenants gives, every tenant and every environment.

home01
$ shc status -a
last polled 2026-10-09 06:53:04 UTC
NODE STATE CPU% MEM% DISK% LOAD
home01 healthy 60.9 37.9 17.9 1.62
obs01 healthy 15.0 5.3 2.5 0.63
all 284 queues green on 2 nodes
TENANT STACK ENVIRONMENT STATE SERVICES NODES LABELS NO HEALTHCHECK
master keycloak default healthy 1/1 home01 app=keycloak environment=default stack=keycloak tenant=master -
master openobserve default healthy 1/1 obs01 app=openobserve environment=default stack=openobserve tenant=master 1
master postgres default healthy 2/2 home01 app=postgres environment=default stack=postgres tenant=master -
master traefik default healthy 15/15 home01, obs01 app=traefik environment=default stack=traefik tenant=master -
prod cloud prod healthy 4/4 home01 app=nextcloud environment=prod stack=cloud tenant=prod 1
prod db prod healthy 2/2 home01 app=postgres environment=prod stack=db tenant=prod -
prod pgweb prod healthy 1/1 home01 app=/root/learn/pgweb environment=prod stack=pgweb tenant=prod -
prod placed prod healthy 1/1 home01 app=notes environment=prod stack=placed tenant=prod tier=learn -
prod store prod healthy 1/1 home01 app=/root/learn/versitygw environment=prod stack=store tenant=prod -
prod web prod healthy 1/1 home01 app=/root/learn/nginx environment=prod stack=web tenant=prod 1
Published ports
master/default/traefik traefik 80/tcp listen=node=@(service=traefik) nodes: home01 local, obs01 local
master/default/traefik traefik 443/tcp listen=node=@(service=traefik) nodes: home01 local, obs01 local
...
2 nodes, 10 deployments
$ shc status --all-tenants
...
2 nodes, 10 deployments
$ shc status -a -t master
...
2 nodes, 4 deployments

--resources adds a table of what each deployment is using right now, measured on its containers, not the reservation its quota holds. MEASURED counts the containers the reading covers; a deployment reads unmeasured when the leader had no reading from the node that runs it.

home01
$ shc status --resources
last polled 2026-10-09 06:53:04 UTC
NODE STATE CPU% MEM% DISK% LOAD
home01 healthy 95.2 37.7 17.9 1.49
obs01 healthy 5.2 5.5 2.5 0.66
all 284 queues green on 2 nodes
TENANT STACK ENVIRONMENT STATE SERVICES NODES LABELS NO HEALTHCHECK
master keycloak default healthy 1/1 home01 app=keycloak environment=default stack=keycloak tenant=master -
master openobserve default healthy 1/1 obs01 app=openobserve environment=default stack=openobserve tenant=master 1
master postgres default healthy 2/2 home01 app=postgres environment=default stack=postgres tenant=master -
master traefik default healthy 15/15 home01, obs01 app=traefik environment=default stack=traefik tenant=master -
prod cloud prod healthy 4/4 home01 app=nextcloud environment=prod stack=cloud tenant=prod 1
prod db prod healthy 2/2 home01 app=postgres environment=prod stack=db tenant=prod -
prod pgweb prod healthy 1/1 home01 app=/root/learn/pgweb environment=prod stack=pgweb tenant=prod -
prod placed prod healthy 1/1 home01 app=notes environment=prod stack=placed tenant=prod tier=learn -
prod store prod healthy 1/1 home01 app=/root/learn/versitygw environment=prod stack=store tenant=prod -
prod web prod healthy 1/1 home01 app=/root/learn/nginx environment=prod stack=web tenant=prod 1
STACK ENVIRONMENT CPU MEMORY MEASURED
keycloak default 0.46% 1.04 GiB 1/1
openobserve default 0.37% 466.7 MiB 1/1
postgres default 0.35% 167.5 MiB 2/2
traefik default 4.13% 837.4 MiB 15/15
cloud prod 2.19% 1.14 GiB 4/4
db prod 1.10% 90.3 MiB 2/2
pgweb prod 0.00% 7.9 MiB 1/1
placed prod 0.86% 77.6 MiB 1/1
store prod 0.05% 86.3 MiB 1/1
web prod 0.00% 3.1 MiB 1/1
Published ports
master/default/traefik traefik 80/tcp listen=node=@(service=traefik) nodes: home01 local, obs01 local
master/default/traefik traefik 443/tcp listen=node=@(service=traefik) nodes: home01 local, obs01 local
...
2 nodes, 10 deployments

shc ls is the inventory, not the health. Two of its flags read beside shc status: --status keeps the deployments in one status, and --posture adds who may connect to each and how far it reaches.

home01
$ shc ls --status running
STACK TENANT ENV APP STATUS NODES
keycloak master default keycloak running home01
openobserve master default openobserve running obs01
postgres master default postgres running home01
traefik master default traefik running home01, obs01
cloud prod prod nextcloud running home01
db prod prod postgres running home01
pgweb prod prod /root/learn/pgweb running home01
placed prod prod notes running home01
store prod prod /root/learn/versitygw running home01
web prod prod /root/learn/nginx running home01
$ shc ls --posture
STACK TENANT ENV APP STATUS NODES ACCESS EXPOSURE
keycloak master default keycloak running home01 open by default lan by default
openobserve master default openobserve running obs01 open by default lan by default
postgres master default postgres running home01 open by default lan by default
traefik master default traefik running home01, obs01 open by default lan by default
cloud prod prod nextcloud running home01 open by default lan by default
db prod prod postgres running home01 open by default lan by default
pgweb prod prod /root/learn/pgweb running home01 open by default lan by default
placed prod prod notes running home01 open by default lan by default
store prod prod /root/learn/versitygw running home01 open by default lan by default
web prod prod /root/learn/nginx running home01 open by default lan by default

A node answers for itself. shc node status is the row of the node you are on; shc node health <name> asks the daemon about one node; shc node info is the registry row, with the id the leader logs by.

obs01
$ shc node status
NODE ID HOSTNAME CLUSTER STATE AVAILABILITY IS LEADER RUNTIME CLUSTERFS REVISION CLUSTERFS AGE
node-194800e1a3c5 obs01 healthy active false docker 262 2m10s
home01
$ shc node health obs01
NAME STATUS HEALTH STATUS CHECKED AT
obs01 active healthy 2026-10-08T16:20:31Z
...
$ shc node info obs01
ID HOSTNAME RUNTIME AVAILABILITY INTERNAL ADDRESS UNDERLAY ADDRESS PUBLIC ADDRESS CLOUD LOCATION IS VOTER FABRIC GRANT FABRIC TENANT SLOTS FABRIC FULLEST BRIDGE
node-194800e1a3c5 obs01 docker active 10.192.0.2:4003 2.31.25.111 2.31.25.111 learn learn/home false 10.128.32.0/20 1 of 31 master, 4 of 125 addresses

The leader is the node that decides. shc leader status says whether the node you are on is the leader and which node is. shc leader health is the leader’s record of its last probe of every node, and a follower relays the leader’s answer, so it reads the same on any node.

home01
$ shc leader status
ROLE LEADER OBSERVED AT
leader node-15826b16f572 2026-10-11T02:41:48Z
$ shc leader health
NODE ID STATE OBSERVED AT ISSUE
node-15826b16f572 healthy 2026-10-11T02:41:49Z -
node-194800e1a3c5 healthy 2026-10-11T02:41:49Z -
obs01
$ shc leader status
ROLE LEADER OBSERVED AT
follower node-15826b16f572 2026-10-11T02:41:53Z
$ shc leader health
NODE ID STATE OBSERVED AT ISSUE
node-15826b16f572 healthy 2026-10-11T02:41:55Z -
node-194800e1a3c5 healthy 2026-10-11T02:41:55Z -

shc cluster status is the control plane in five lines: the leader, how many nodes answer, how many vote and whether the voters can still reach quorum. Under its warnings it lists the work only the leader does, each job with how long ago it last swept.

home01
$ shc cluster status
cluster name: shc-cluster
leader: home01
nodes: 2 healthy of 2
voters: 1
quorum: healthy, 1 voter required
warning: only 1 raft voter: losing one stops the control plane committing, and then every deployment with a single-writer volume stops its writer within about a minute, on the surviving nodes too, unless the volume is far, with no control-plane fence node in its reach
warning: control-plane fence has 2 members, fewer than three, so it cannot hold three voters until `shc node update <node> --control-plane` brings a node into it, and the voter count follows
leader-only work, recorded by home01 51s ago:
acme last swept 56s ago
backup_scheduler running 2m56s, no sweep yet
container_reconciler last swept 55s ago
mesh_snapshot_sign last swept 56s ago
operation_registry last swept 56s ago
proxy_reconcile last swept 55s ago
schedule_dispatcher last swept 56s ago
scheduler last swept 56s ago
service_vip_alloc last swept 56s ago

Two nodes keep one voter, since a second voter would make losing either node stop the cluster. Both warnings say what that costs: the one voter is the control plane, and it takes a third node to hold three. The cluster ladder grows the control plane; this ladder lives with one voter.

Some of the daemon’s work runs in the background as tasks, and every node runs a worker that takes them. shc task alone says what the group does. shc task ls lists the running and pending tasks of your tenant: -a is every status, -S picks statuses by name, --dead keeps the tasks that ran out of retries, --node the ones a node claimed, and -n bounds the listing. A cluster with nothing in flight has none. shc task workers lists the workers, with how long ago each last reported.

home01
$ shc task
...
$ shc task ls
no tasks in tenant "prod"
$ shc task ls -a -n 5
no tasks in tenant "prod"
$ shc task ls -S completed --node home01
no tasks in tenant "prod"
$ shc task ls --dead
no tasks in tenant "prod"
$ shc task workers
ID NODE ID HEARTBEAT AGO STARTED AT
...

shc task show prints one task whole, and shc task cancel withdraws a pending one: --reason is recorded on the task and --yes skips the question.

explained, not walked: a learn cluster with nothing in flight queues no task, so there is no id to show or cancel
$ shc task show 42
$ shc task cancel 42 --reason 'superseded by a newer run' --yes

This page changed nothing. Every deployment of the app ladder is still running where it was, and obs01 holds openobserve.

Next: Doctor.