Skip to content
SHC Docs

Storage providers

SHC supports three storage provider types for stack volumes and backups:

ProviderTypical useShared across nodesSnapshotsQuotas
localSingle-node, or per-node local fast storageNoZFS/Btrfs onlyZFS/XFS/Btrfs only
nfsMulti-node shared storage, simple setupYesrsync-basedNo
cephfsMulti-node with native snapshottingYesNativeLimited

An SHC cluster can have multiple providers, e.g. local for per-node fast volumes plus nfs for shared backups. Each stack picks which provider to use based on the app’s requires.storage whitelist and operator-chosen preferences.

Source of truth (ce module): modules/storage/storageapi/types.go (types, ProviderConfig, the Capabilities set), modules/storage/providers/ (the provider implementations), modules/storage/commands/ (CLI).

For the per-stack resolution logic (how SHC picks which provider for a given stack) see architecture/modules/storage.md.

interactive: drag or click to explore

Cloud block volumes are a fourth thing, and they have their own rule

Section titled “Cloud block volumes are a fourth thing, and they have their own rule”

The three providers above are where a stack’s volumes and backups are served from. Separately, an estate whose cloud sells attachable disks (Hetzner, AWS EBS, Linode, Vultr, Azure, Exoscale) can put a deployment’s durable mounts on real block devices, and on such an estate that is the default rather than an opt-in.

Which mounts are durable is not a new judgment: backup.yaml already carries it. See Block storage by default for the decision procedure, how to declare a volume ephemeral, why object storage and block storage answer different questions, and why an existing deployment is flagged rather than switched.

Every SHC installation starts with a local provider pointing at the daemon’s state directory (typically /var/lib/shc/storage). No configuration needed; shc init sets it up.

Capabilities are auto-detected by detection.go based on the filesystem backing the state directory:

  • ZFS → supports_quotas=true, supports_snapshots=true, native snapshots, native quotas
  • XFS (with project quotas enabled) → supports_quotas=true, snapshot via rsync
  • Btrfs → supports_snapshots=true (CoW snapshots), supports_quotas=true (qgroups)
  • ext4 → supports_quotas=false, supports_snapshots=false (rsync fallback for snapshots works, but expensive)

To see what was detected:

Terminal window
shc storage ls
shc storage show local
  • Production single-node, heavy IO: ZFS with a dedicated dataset. Best snapshot + quota + IO story SHC supports.
  • Production multi-node with local-mode stacks: Btrfs subvolumes, or ZFS if you have it. Fall through to NFS / CephFS for shared volumes.
  • Development or small deployments: ext4 is fine. You won’t get instant snapshots but rsync-based backup will still work, just slower.

You can add a second local provider pointing at a different mount, e.g. a fast SSD for DB data, keeping the default provider on slower storage:

Terminal window
shc storage add fast-nvme --path /mnt/nvme/shc --node node-a
shc storage link fast-nvme node-a node-b # attach to specific nodes

Options:

  • --path: mount path. Must be writable by the shc user.
  • --node: single-node scope. Omit for cluster-wide.

A node is named as shc node ls shows it, or by its id. shc storage show and shc storage ls print the linked nodes by name.

Commands:

  • shc storage add <name> --path <path> [--node <node>]: commands/add.go
  • shc storage link <name> <node> [<node>...]: attach a provider to specific nodes (commands/link.go)
  • shc storage unlink <name> <node>: detach
  • shc storage rm <name>: remove a provider (refuses if stacks are using it)

Reserved names that you cannot use for a user-added provider: local, nfs, cephfs: these are the managed singletons created via their dedicated enable commands.


  • Multi-node cluster where the same stack can schedule to any node and needs the same data
  • Shared backup repository (backup.repo in default.yaml)
  • You want the simplest possible shared-storage setup: NFS “just works” over a LAN

SHC ships a managed NFS singleton: one NFS server across the cluster, with SHC handling exports, client mounts, and reconnects.

Terminal window
shc storage nfs enable --export-path /mnt/shc/nfs --network 10.0.0.0/24

What happens:

  1. SHC designates one node as the NFS server (the one you ran the command on, unless overridden) and installs nfs-kernel-server.
  2. Exports <export-path> to the given network CIDR.
  3. Installs nfs-common on every other cluster node.
  4. Mounts the export on each client node under /mnt/shc/nfs-mount (or similar; see shc storage show nfs).
  5. Registers an nfs provider in the control-plane database pointing at the mount.

Source: modules/storage/commands/managed.go (the shc storage nfs verbs), modules/storage/shared_manager.go + modules/storage/nfs/ (exports, cleanup).

  • Shared access: yes. That’s the whole point.
  • Snapshots: rsync-based. SHC stages snapshots by rsync’ing to a .snapshots directory on the NFS export. Works on any backing filesystem but isn’t atomic.
  • Quotas: no. NFS doesn’t model per-directory quotas in a way SHC can enforce.
  • Locking: SHC uses application-level locking (database-backed) for concurrency, so NFS’s own locking quirks don’t bite you.
  • Performance: fine for most web-app workloads. Not suitable for databases that fsync heavily; run those on local storage and only back them up to NFS.
Terminal window
shc storage nfs enable # create the singleton (run once per cluster)
shc storage nfs status # check server & client mount health
shc storage nfs join -n home02 # pin home02 as the NFS server, this node without -n
shc storage nfs leave # hand the NFS server role from this node to another
shc storage nfs disable # tear down the singleton (CAREFUL: affects all nodes)

join moves the server role, not the data. The new server stops mounting the share and serves its own /mnt/shc/nfs; the files stay on the old host. A share mount the new server still has in use refuses the join until whatever holds it open is stopped.

If you already have an NFS server you want SHC to consume (instead of letting SHC manage one), add it as an external provider:

Terminal window
# Mount the NFS export yourself (via /etc/fstab or autofs),
# then register it as a local-type external provider:
shc storage add team-nfs --path /mnt/team-nfs

SHC will treat it like a local provider backed by whatever filesystem the mount exposes.


CephFS: shared storage with native snapshots

Section titled “CephFS: shared storage with native snapshots”
  • You already run Ceph for other workloads and want SHC to participate
  • You need native atomic snapshots of shared storage (NFS snapshots aren’t atomic)
  • Larger clusters where you want to decouple storage from SHC-managed infrastructure

CephFS is configured by connecting SHC to an existing Ceph cluster; SHC does not provision Ceph for you.

Terminal window
shc storage cephfs enable [--cluster-network CIDR] [--public-network CIDR]

--cluster-network and --public-network are optional. Omit both when Ceph should use the host’s default networks.

SHC will:

  1. Validate the Ceph credentials can reach a monitor.
  2. Create the SHC subdirectory on the CephFS filesystem if needed.
  3. Mount it on every cluster node via mount -t ceph ....
  4. Register a cephfs provider in the control-plane database.

Source: modules/storage/commands/managed.go (the shc storage cephfs verbs), modules/storage/ceph/manager.go.

  • Shared access: yes
  • Snapshots: native, atomic. Faster and safer than NFS+rsync
  • Quotas: limited. Ceph supports directory quotas; SHC exposes them where the subdirectory is set up with quota attributes
  • Setup complexity: you need a working Ceph cluster first. If you don’t already run Ceph, NFS is the pragmatic choice.

Beyond the filesystem providers above, SHC can back an individual compose mount with a provider-managed block volume: a real cloud disk (Hetzner Cloud volume, OpenStack Cinder, AWS EBS, GCE PD, Azure Disk, DigitalOcean, Linode, Vultr, CloudStack, UpCloud, Oracle Cloud, Alibaba Cloud) that is created, attached, and mounted for that one mount, instead of a directory on the node’s filesystem.

An app opts a mount in with the x-shc-volumes compose extension:

x-shc-volumes:
pgdata:
size: 10Gi # Ki/Mi/Gi/Ti (binary), K/M/G/T (decimal), or bare bytes
access: rwo # rwo = single-node read-write (default), rox = read-only multi-attach
volumes:
pgdata:

What you get compared to a plain directory:

  • Survives the node. The disk is a first-class cloud resource; a deployment move re-attaches it to the target node (same location) instead of copying files.
  • Sized and enforced per mount: size is the disk’s real capacity, not a quota approximation.
  • Backup-reconstructable. Backups record the volume identity, and a restore can re-provision + re-attach the disk from the backup when the original is gone.
  • Leased single-writer. rwo volumes carry a monotonic attach lease, so a partitioned node can never keep writing after the cluster rebinds the volume elsewhere.

rwo and rox are the only accepted access modes. An unsupported value returns X400328VOLACC at install. Read-write-many data belongs on a shared filesystem such as CephFS or NFS.

The cloud credentials come from the connection registry (shc connection), the same place the cluster’s provisioning credentials live: the storage module resolves the node’s provider (for example hcloud on a Hetzner node) and calls the cloud API with the registered connection. Nodes without a block-volume provider fall back to the app’s whitelisted filesystem providers.

shc volume mount <id> binds a volume’s data read-only under the daemon’s state directory. A block volume’s data is the filesystem on its device, and the directory its deployment mounts it at is an empty directory on the node’s own disk whenever the volume is not mounted there, so the daemon binds that directory only while the volume is mounted at it:

  • Attached to the daemon’s node and mounted there, it is bound.
  • Attached to the daemon’s node but not mounted, say on a node back from a power cut before its remount, it is refused with X409886VLMNMT.
  • Attached to another node, it is refused with X409896VLMHLD. Run the mount on that node.
  • Detached, as an uninstall that kept it or a failed install’s rollback leaves it, it is attached to the daemon’s node through the same steps an install takes, its device mounted, then bound. The output says it attached the volume. Releasing the mount detaches it again, whether by shc volume umount, the lease running out, or the reap after a restart. A device another node holds open is refused with X409896VLMHLD before anything changes, and an attach that fails is undone and refused with X500897VLMATT.

A block volume reaches only the nodes whose recorded cloud and location match the volume’s. Each volume provider belongs to one cloud:

Volume providerNode cloud
hcloudhetzner
aws-ebs-volumeaws
gcepdgcp
azure-diskazure
dovolumedigitalocean
vultr-volumevultr
linode-volumelinode
cinder-volumeopenstack
cloudstack-volumecloudstack
incus-volumeincus
upcloud-storageupcloud
oci-volumeoci
alibaba-diskalibaba
linstorany: every node inside the connection’s location

The location is the place the provider creates the volume in, and a node is inside it when its location is that place or a path under it, read either as the node’s own --location or as the path shc node ls prints, under its cloud: a node at --cloud hetzner --location fsn1 is inside fsn1, hetzner/fsn1 and hetzner. A volume in use also reaches the node it is attached on, whatever its location says. For upcloud-storage it is the connection’s zone (fi-hel1), for oci-volume the availability domain’s label in lower case (phx-ad-1), for alibaba-disk the connection’s zone id (cn-hangzhou-h).

A node reads its cloud and location from the cloud’s metadata service on Hetzner, AWS, GCP, Azure, UpCloud and DigitalOcean. On any other cloud, Oracle Cloud and Alibaba Cloud among them, say both at shc init or shc node join, or later with shc node update:

Terminal window
$ shc node update <node> --cloud oci --location phx-ad-1

shc node ls shows what each node records. A node with no cloud, or with a location outside the volume’s, is outside the volume, and a deployment whose volume reaches no node stays pending. The per-cloud pages have the exact values: UpCloud, Oracle Cloud, Alibaba Cloud.

LINSTOR: volumes on the nodes’ own disks, copied across hosts

Section titled “LINSTOR: volumes on the nodes’ own disks, copied across hosts”

When the SHC nodes are themselves LINSTOR hosts, as on an Incus cluster where shcd runs on each host, a linstor connection puts a mount on a DRBD resource with copies on several hosts:

Terminal window
$ shc connection connect --type linstor --name ogden-linstor \
--endpoint http://10.22.0.11:3370,http://10.22.0.12:3370,http://10.22.0.13:3370 \
--cred resource_group=shc-volumes --cred location=ogden

endpoint lists the controllers. LINSTOR runs one at a time and the one that answers is used. resource_group is the group a volume is spawned from, and its place count is how many hosts hold a copy. Connect refuses a group the controller does not have (X400291LNSCFG). location is the location path the hosts sit under. A volume reaches every node inside it, whatever cloud the node records. Connect takes a location no node is in yet, for nodes that join later, and storage.connection-reach warns while it holds none. ca_cert and tls_server_name pin an https controller.

A mount is attached on the node that runs its replica: a node with a copy uses it, and any other node gets a diskless resource that reads and writes the copies over the network. DRBD lets one node at a time open the device, and only while that node keeps quorum. A move, or a move off a lost host, attaches the same resource on the target and copies nothing then.

A diskless node pays the network on every read and waits on two peers for every write, so SHC sets LINSTOR’s DrbdOptions/auto-diskful 1 and DrbdOptions/auto-diskful-allow-cleanup true on each volume it provisions or attaches. Once a diskless resource has been primary for a minute, LINSTOR makes a copy on its node, and once that copy is UpToDate it deletes a spare one, so the volume holds the group’s place count of copies before, during and after. After a move the new node gets the copy the same way. A key set on the resource definition, the resource group or the controller is left as the operator set it. The new node needs a storage pool of the resource group to hold a copy.

DRBD promotes the node that opens the device for writing, inside the open, and every peer must agree. A peer refuses while it holds the device open read-only, and an LVM scan (lvs, lvdisplay, Incus’s LVM driver) opens every block device it is not told to skip. On every LINSTOR host, keep LVM off the DRBD devices in /etc/lvm/lvm.conf:

devices { global_filter = [ "r|^/dev/drbd|" ] }

lvmconfig devices/global_filter shows it, and shc doctor run --category storage fails a host with drbd loaded whose filter lets LVM open /dev/drbd* (storage.lvm-drbd-filter).

A refused open never leaves a volume read-only. mount without -w answers a device it cannot open for writing with a read-only mount and exit 0, so SHC mounts with -w, and the volume’s fstab entry carries x-systemd.rw-only for boot. Before it reads, formats or mounts a DRBD device, the node waits for drbdsetup events2 to say it may promote, and it tries a refused promote again with backoff. Each wait logs X503425DRBWAT, a mount that went through after one logs X200425VOLRWM, and after two minutes the install fails with X503425VOLRWF, which names the refusal and the resource’s state; the peer’s kernel log names what held the device open. A volume root found mounted read-only is unmounted and mounted again read-write.

On a host whose LINSTOR storage pool is an LVM thin pool, LVM runs thin_check before it activates the pool. A hard power-off with writes in flight leaves leaked blocks, which thin_check reports as an error, and LVM then keeps the pool inactive, with every copy on it, until someone repairs it. Let leaked blocks through in /etc/lvm/lvmlocal.conf:

global {
thin_check_options = [ "-q", "--clear-needs-check-flag", "--ignore-non-fatal-errors" ]
}

lvmconfig global/thin_check_options shows it. The initrd activates the host’s volume groups at boot with its own copy of /etc/lvm, so rebuild it with update-initramfs -u -k all after the change.

LVM logs nothing to the journal by default, and the LINSTOR satellite keeps lvchange’s errors in its own error reports. For thin_check’s status to reach the journal, append this line to /etc/lvm/lvm.conf. LVM ignores logging keys in lvmlocal.conf.

log { syslog = 1 level = 3 }

shc doctor run --category storage warns on a node whose thin pools hold volumes without the option, and names any pool LVM refused, with thin_check’s exit status when the journal records it: 64 is leaked blocks, anything else wants lvconvert --repair. The node logs X503366THNPOL when a pool is refused and once more when its status becomes known, and shc events ls -T error lists it.

A node that comes back from a power cut mounts its volumes again by itself. At boot the fstab mount of a DRBD device can run while DRBD is still attaching its disk and fail with “Can’t open blockdev”. Once the node’s lease loop renews its lease on the volume, shcd mounts the volume’s root and then its bind through the same fstab entries, logs X200368VOLRMD, and the node agent starts the deployment. A mount that keeps failing is logged as X503368VOLRMT with the mount’s own error, tried again every 15 seconds, and named by storage.volume-mounts. A lease-loss restart mounts every volume of the deployment it can and fails naming each one it could not, rather than stopping at the first.

Every writer of /etc/fstab on a node (block volumes, NFS, CephFS) holds one lock and an flock on the table from its read to its write, and writes it in place, whole. A line holding NUL bytes from an older torn write is taken out by the next rewrite and logged as X200088FSTTRN. When a volume’s root or bind line is missing and the volume is not mounted, the lease loop writes the line again from the attach, but only once the attach has settled (the attachment reads attached, no mount or unmount of it is running) and the line has been missing for two minutes. It logs X200089FSTRGN with the line it wrote.


SHC’s storage module (modules/storage/, ce module) picks a provider at install time based on:

  1. App requirement: requires.storage: [local, nfs] in the app’s app.yaml whitelists which providers are acceptable.
  2. Per-stack override: shc install <app> --storage <provider> at install time forces a specific provider (must still be in the whitelist).
  3. Node affinity: the stack can only use providers that are linked to its assigned node.
  4. Capability requirements: if the app or install flow needs snapshots or quotas, providers without those capabilities are filtered out.
  5. Operator default: if multiple providers qualify, SHC picks the cluster default (configurable in default.yaml).

If no provider qualifies, install fails with X400937QNOSUP (quotas not supported) / X400942SNOSUP (snapshots not supported) / X404914STGNOF (storage provider not found).


Terminal window
shc storage ls
shc doctor run --category storage # structured diagnostics

The storage module registers a storage.local diagnosis: it walks every registered provider and checks that the path exists, is a directory, and is a mount point. A missing or non-directory path is an error; an existing path that is not a mount point is a warning. Source: modules/storage/doctor.go in the ce module.

storage.thin-pools runs on every node and reads its LVM: an error for each thin pool LVM left inactive while it holds volumes, and a warning when global/thin_check_options lacks --ignore-non-fatal-errors. Source: modules/storage/thin_pools.go.

storage.volume-mounts runs on every node and fails for each block volume attached to it that is not mounted where its deployment expects it, naming the deployment, the mountpoint and why (X503368VOLRMT). Source: modules/storage/volume_mounts.go.

storage.fstab runs on every node and fails for each line of /etc/fstab that mount skips as a parse error (NUL bytes, fewer than three fields) and for each block volume attached to the node with no fstab line for its root or its bind, naming the volume and its deployment. It warns for a day on each lost line the lease loop wrote again. Source: modules/storage/fstab_doctor.go.

storage.volume-locality runs once for the cluster and warns on each block volume in use on a node that holds no copy of it, a LINSTOR diskless resource, naming the volume, its deployment and mount, the LINSTOR resource and the node. One named for more than a few minutes has no copy coming: linstor resource-definition list-properties <volume> shows the auto-diskful options and linstor error-reports list why LINSTOR did not make the copy. Source: modules/storage/volume_locality.go.

storage.connection-reach runs once for the cluster and warns on each volume connection whose location holds no node, naming the connection, its tenant and the location as its credential writes it. A volume it makes reaches only the node it is attached on, so its deployment cannot move and each consumer colocated with it is held to that node. Source: modules/storage/connection_reach.go.

“Provider is accessible but stacks fail with X400942SNOSUP on backup.” → The backing filesystem doesn’t support snapshots. Install stacks on a provider whose capabilities include snapshots, or accept the rsync-fallback path (slower, non-atomic).

“NFS mount keeps dropping on reboot.” → Check that shc storage nfs join has registered the mount in the SHC-managed systemd unit. Distro-level /etc/fstab entries bypass SHC and should be removed if present.

“CephFS mount requires ceph kernel module.” → Load it (modprobe ceph) and add to /etc/modules-load.d/. SHC’s cephfs-enable checks for this and refuses if missing.

“Adding a provider fails with ‘name is reserved’.” → local, nfs, and cephfs are reserved for managed singletons. Use shc storage nfs enable / cephfs enable instead of storage add nfs.


Deleting a provider refuses if any stack still uses it. A relocation keeps a deployment’s storage provider, so take each stack off it another way: shc clone <stack> <new-name> --storage <other> and uninstall the original, or reinstall it. Then:

Terminal window
shc storage rm <name>

For the managed singletons, use the disable commands; they tear down the server/mount infrastructure as well:

Terminal window
shc storage nfs disable # CAREFUL: removes the cluster-wide mount
shc storage cephfs disable # unmounts Ceph on all nodes

Disable refuses if any stacks are still using the provider.