shc storage
The shc storage commands manage the storage provider inventory: which shared filesystems exist, which nodes can reach them, and the lifecycle of the two SHC-managed kinds. SHC supports three provider kinds:
| Kind | Shared across nodes | Managed by SHC | Typical path |
|---|---|---|---|
local | No | No | /var/lib/shc or custom |
nfs | Yes | Yes (singleton) | /mnt/shc/nfs |
cephfs | Yes | Yes (singleton) | /mnt/shc/cephfs |
Registering a provider does not move any deployment onto it. Deployment files always live under the node’s own state home (
<state_home>/deployments/...); no provider is consulted for that path.shc install --storage <name>records a label on the deployment, and only the clone/move classifiers and the block-volume binder read it back; see Using a provider in a deployment below and the storage module architecture.
Concepts
Section titled “Concepts”- Managed providers are brought up and torn down by SHC. There is at most one of each kind (
nfs,cephfs) per cluster, and their names are reserved. - External providers are
local-kind entries that point at an existing mount. They are the only kind you create withshc storage add. - Links associate a provider with specific nodes. Managed providers link themselves; external providers are linked via
shc storage link.
Inventory
Section titled “Inventory”shc storage ls
Section titled “shc storage ls”List all storage providers known to the cluster.
shc storage lsshc storage ls --type cephfsshc storage ls --node node01shc storage ls --managedshc storage ls --externalOptions:
| Flag | Short | Description |
|---|---|---|
--type | Filter by provider kind (local, nfs, cephfs) | |
--node | -n | Filter to providers linked to a node |
--managed | Show only SHC-managed providers | |
--external | Show only external providers |
Default columns are name, type, filesystem, managed, quota, snapshot, nodes (count of linked nodes), and path. Output format follows the global -o/--output flag (tty|text|json|yaml).
shc storage show
Section titled “shc storage show”Show full details for one provider.
shc storage show local-ssdReturns the provider id, path, filesystem, quota and snapshot capability, config blob, and the list of linked nodes.
External Providers
Section titled “External Providers”shc storage add
Section titled “shc storage add”Register an existing mount as an external local-kind provider.
shc storage add local-ssd --path /var/lib/shc-ssdshc storage add local-ssd --path /var/lib/shc-ssd --node node01Options:
| Flag | Short | Description |
|---|---|---|
--path | -p | Mount path on the target node (required) |
--node | -n | Node to link; defaults to no link (link later with shc storage link) |
Rules:
- The name must not be one of the reserved managed-provider names
local,nfs,cephfs. - For managed providers, use
shc storage nfs enableorshc storage cephfs enableinstead.
shc storage rm
Section titled “shc storage rm”Remove a provider from the inventory.
shc storage rm local-ssdshc storage rm local-ssd --forceBy default the delete is refused if any stack still uses the provider. --force skips that check.
shc storage link / shc storage unlink
Section titled “shc storage link / shc storage unlink”Attach or detach an external provider to one or more nodes.
shc storage link local-ssd node01 node02shc storage unlink local-ssd node01Both commands take the provider name followed by one or more node ids. Managed providers manage their own links.
NFS (Managed)
Section titled “NFS (Managed)”Managed NFS is a single kernel NFS server exported from one cluster node, mounted by every other node at /mnt/shc/nfs. It is created and torn down by SHC.
shc storage nfs enable
Section titled “shc storage nfs enable”Bring up managed NFS on the cluster.
shc storage nfs enableshc storage nfs enable --export-path /mnt/shc/nfs --network 10.42.0.0/24Options:
| Flag | Default | Description |
|---|---|---|
--export-path | /mnt/shc/nfs | Base export path on the server node |
--network | (auto) | CIDR allowed to mount the export |
shc storage nfs disable
Section titled “shc storage nfs disable”Tear down managed NFS. Refused (X409033STGUSE) while any deployment’s storage column or any volume row still names the provider (the same in-use join shc storage rm runs). The refusal names the stacks holding it.
shc storage nfs disableshc storage nfs disable --force # tear down anyway; those stacks lose the share--force orphans the referencing rows: nothing reaps them, and the deployments keep pointing at a provider that no longer exists. Move or uninstall them first unless you are recovering a cluster whose consumers are already gone.
shc storage nfs status
Section titled “shc storage nfs status”Show NFS server role, export path, and per-node mount state.
shc storage nfs statusshc storage nfs status -vshc storage nfs join
Section titled “shc storage nfs join”Pin the NFS server role onto a node.
shc storage nfs join # pin local nodeshc storage nfs join --node node02 # pin a specific nodeshc storage nfs leave
Section titled “shc storage nfs leave”Release the NFS server role from the pinned node. SHC will re-elect another voter as server.
shc storage nfs leaveshc storage nfs leave --node node02CephFS (Managed)
Section titled “CephFS (Managed)”Managed CephFS gives the cluster a replicated, distributed filesystem mounted at /mnt/shc/cephfs. Each node that contributes capacity runs one or more Ceph OSDs backed by a block device. Monitors and the MDS are placed on voter nodes.
shc storage cephfs enable
Section titled “shc storage cephfs enable”Enable CephFS across the cluster. Requires at least one voter node.
shc storage cephfs enableNote: multi-node enable (key distribution, monmap bootstrap, MDS/MGR deployment) is not yet automated. The command is wired up end-to-end and will succeed once the deployment path lands.
shc storage cephfs disable
Section titled “shc storage cephfs disable”Tear down CephFS, stopping local monitors, MDS, and OSDs and removing the mount. Refused (X409033STGUSE) while any deployment’s storage column or any volume row still names the provider (the same in-use join shc storage rm runs).
shc storage cephfs disableshc storage cephfs disable --force # tear down anyway; those stacks lose the mountshc storage cephfs status
Section titled “shc storage cephfs status”Show cluster health, OSD counts, and per-node daemon state.
shc storage cephfs statusshc storage cephfs join
Section titled “shc storage cephfs join”Contribute a local block device as a new OSD.
shc storage cephfs join --device /dev/sdbshc storage cephfs join --device /dev/sdb --node node03Options:
| Flag | Short | Description |
|---|---|---|
--device | Block device to consume (required, e.g. /dev/sdb) | |
--node | -n | Node to add the OSD on; defaults to local |
shc storage cephfs leave
Section titled “shc storage cephfs leave”Remove an OSD from the cluster, waiting for rebalance by default.
shc storage cephfs leave --osd-id 3shc storage cephfs leave --osd-id 3 --forceOptions:
| Flag | Short | Description |
|---|---|---|
--osd-id | OSD id to remove (required) | |
--force | -f | Skip the wait for rebalance |
shc node leave automatically drains and removes every local OSD before the node exits the cluster, so manual cephfs leave is only needed for capacity management.
Using a provider in a deployment
Section titled “Using a provider in a deployment”--storage is repeatable on shc install, and each value takes one of three
forms:
| Value | Meaning |
|---|---|
<backend> or node | every volume of the deployment goes there; node is the node’s own disk |
<volume>=<size> | one volume’s size, on the deployment’s backend |
<volume>=<backend>:<size> | one volume’s backend and size |
shc install nextcloud --storage cephfsshc install nextcloud --storage local-ssdshc install nextcloud --storage nfs --stack cloud --environment prodshc install postgres --storage pgdata=50Gishc install postgres --storage hcloud --storage pgdata=50Gishc install postgres --storage pgdata=hcloud:50Gi --storage wal=node:10Gishc install postgres --storage node --storage pgdata=hcloud:50GiA backend is node, the configured block-volume provider, a volume-capable
connection, or a provider shc storage ls lists. Any other name is refused
with X400305STGBKE, which lists the backends the cluster has. A size takes Ki,
Mi, Gi, Ti, K, M, G or T, the units an app’s x-shc-volumes size takes, and a
size for a volume the app does not declare is refused with X400304STGVOL.
Each volume binds on the backend its value names, so one deployment can hold a
volume on hcloud, another on ceph and others on the node disk. With
--storage node, every volume not named on another backend stays on the node
disk. The deployment records each volume’s backend, and a later render keeps
the volume there. A backend the cluster has but that cannot bind a block
volume, such as a provider the cluster’s volume configuration pins away from,
is refused with X400307STGBND. A volume on node stays on the node disk even
when the app declares x-shc-volumes for it. A volume the deployment already
keeps on a block volume is refused with X409305VOLNOD instead: a render does
not move its bytes.
Sizes and the disk quota
Section titled “Sizes and the disk quota”A volume with no size on the command line takes the app’s x-shc-volumes
size. A block volume is provisioned at its size. A volume on the node disk
takes a size only where the node holding it can hold the volume to it: an xfs
filesystem mounted with prjquota, a zfs dataset of its own or a btrfs
subvolume with quotas on (the quota module
says where). Before any volume is bound or prepared, the render asks that
node, which may not be the node rendering, and refuses a size it cannot cap
with X409311VOLQTA, naming the volume, the node and the filesystem. A size it
takes is recorded on the deployment and becomes the volume’s hard cap at the
next disk sweep.
shc: [X409311VOLQTA] volume gitlab-logs cannot be capped at 10Gi on node incus03 (ext4): ext4 has no directory quotaBefore any image is pulled, admission charges every new volume’s size to the disk quota of the deployment, its environment and its tenant, and refuses an install that does not fit:
shc: [X403008DSKASK] disk quota of deployment pg exceeded, 40Gi in use, 50Gi asked, 64Gi limitSizes on update
Section titled “Sizes on update”shc update takes --storage <volume>=<size> only, and a volume keeps its
backend.
shc update postgres --storage pgdata=80GiA volume never shrinks. A larger size grows a block volume: its provider resizes it, the node it is attached to grows the filesystem on it online (ext4, xfs and btrfs) once the disk reads the new size, and then the volume’s record takes the size the provider reports. Hetzner volumes grow in whole GB, rounded up. Every size is checked before any volume grows, so one volume that cannot grow refuses the whole update with nothing changed. Its own size is no change and a smaller one is refused with X409303VOLSHR. A provider that cannot resize a volume refuses with X409306VOLRSZ, and a volume attached to no node with X409308VOLDET. A volume on the node disk takes the new size where the node holding it can cap it, and its cap moves to it at the next disk sweep. A size below the bytes it holds is refused with X409303VOLSHR before anything changes, and a size its node cannot cap with X409311VOLQTA before the render binds or prepares any volume. Sizes print in the units the flag takes, so a size in a refusal can be typed back as it reads.
shc: [X409303VOLSHR] volume pgdata is 80Gi, an update cannot shrink it to 50Gishc: [X409306VOLRSZ] volume pgdata is on ceph, which cannot grow a volumeAn app update that raises a block volume’s x-shc-volumes size grows the
volume the same way. Where the volume cannot grow, the update goes ahead, the
volume stays at its size, and the update warns X200303VOLSIZ: the app now sizes volume pgdata at 80Gi, it stays at 50Gi.
If a block volume has no recorded size, the update reads it from the provider and records it before deciding whether to grow. If the size cannot be recorded, the update continues and warns X200304VOLUNK with the volume’s requested size and the reason.
On each leadership gain, the leader sweeps live block volumes with no recorded
size and asks their providers one at a time, leaving kept or released volumes
alone. Doctor’s storage.volume-size-recorded check lists any live volumes
still without a size; a provider that cannot report one needs
shc update --storage <volume>=<size>.
Where the backend is recorded
Section titled “Where the backend is recorded”The deployment’s backend is stored on the deployment: the deployment.storage
column, and the storage field of the deployment’s manifest.json sidecar.
node is stored as local, the store’s name for the node disk. Two consumers
read it back:
- Relocation (
shc update --where, live or off a dead node) reads it, and since the Slice 0 fix it no longer believes it on its own. Annfsorcephfslabel buys a rebindable (live) or shared (dead-node) verdict only when a persistedVolumerow also names that provider; nothing else at HEAD can say where a deployment’s bytes actually are. An unbacked label degrades to not-rebindable on the live path, which forces backup+restore and so carries the bytes; on the dead-node path a label with no row is treated as lost and refused unless--restore-from-backupbrings the data back from the latest backup. Both say in the plan warnings that the claim could not be verified, so the slower answer reads as deliberate rather than as a fault. - The block-volume binder, on the daemon-served
shc installonly, declines when--storagenames something other than the resolved block-volume provider, which leaves the deploy on the host-bind path. This has an effect only on an estate that has a block-volume backend at all; with none configured the binder is absent either way and--storagechanges nothing. That decline is not a substitute for checking the persisted volume records. Install, update and uninstall require a reachable shcd.
The deployment’s files land in <state_home>/deployments/... regardless: no
value of --storage moves them, and no provider is consulted for that path.
Omitting --storage leaves the column empty. The relocation classifiers
interpret an empty value as local.
Command summary
Section titled “Command summary”| Command | Description |
|---|---|
shc storage ls | List providers |
shc storage show <name> | Show one provider |
shc storage add <name> --path ... | Register an external local provider |
shc storage rm <name> | Remove a provider |
shc storage link <name> <node>... | Link external provider to nodes |
shc storage unlink <name> <node>... | Unlink external provider from nodes |
shc storage nfs enable | Bring up managed NFS |
shc storage nfs disable | Tear down managed NFS |
shc storage nfs status | Show NFS status |
shc storage nfs join | Pin NFS server role to a node |
shc storage nfs leave | Release NFS server role from a node |
shc storage cephfs enable | Bring up managed CephFS |
shc storage cephfs disable | Tear down managed CephFS |
shc storage cephfs status | Show CephFS status |
shc storage cephfs join --device | Contribute a disk as an OSD |
shc storage cephfs leave --osd-id | Remove an OSD |
See Also
Section titled “See Also”- File Locations: where provider data lives on disk
- App Commands:
--storageflag onshc install - Node Commands: join/leave and OSD drain behavior
- Data Model: deployment state layout
- Storage module: what
--storagedoes and does not do