Skip to content
SHC Docs

Common issues

Detailed, symptom-first walkthroughs. For quick symptom→page routing see the troubleshooting index. For every error code’s exact definition see the error reference.

The commands below run in a tenant. Select yours once with shc ctx -t <tenant>, or add -t <tenant> to each command.


Symptom: X503738DMNREQ, shc install requires shcd

Section titled “Symptom: X503738DMNREQ, shc install requires shcd”

shc install, shc update and shc uninstall send their work to shcd, which renders the deployment and keeps its state. When the CLI cannot reach shcd they refuse before anything changes; there is no local fallback (SHC-1328, ce modules/app/commands/install.go). Start it with systemctl start shc, or on a Mac with sudo launchctl kickstart system/org.selfhostedcloud.shcd, run shc doctor, and if it will not stay up read Daemon won’t start.

Symptom: X504740DMNTKN, shcd took the request and did not answer in time

Section titled “Symptom: X504740DMNTKN, shcd took the request and did not answer in time”

shcd took the request of shc install, shc update or shc uninstall and gave no answer within the CLI’s wait, which is 30 seconds. It goes on with what it took, so the deployment may be coming up, may have failed, or may never have been recorded. Read it with the shc status command the error names before you run the verb again. A node far from the leader or under heavy load is the usual cause (SHC-2159).

The command named no tenant, and nothing picks one for you. Name it with -t, $SHC_TENANT or shc ctx -t. A cluster made through the portal has default; one you installed yourself has only master until you add one (Tenants):

Terminal window
shc tenant ls # the tenants this cluster has
shc ctx -t default # select it for every later command
shc install <app> -s <name>

Symptom: the app installed but its address shows 404

Section titled “Symptom: the app installed but its address shows 404”

shc install succeeded and the app is healthy, but a browser gets Traefik’s 404 and shc ingress ls shows no route for it. An app gets a web address only from --expose <host>; SHC never invents one. The install says so with warning X200951HSTNOA. Give the installed app its address with shc update:

Terminal window
shc update <stack> --expose <stack>.example.com
shc ingress ls # the route is there now

Symptom: X400903INVAPP (“invalid app”)

Section titled “Symptom: X400903INVAPP (“invalid app”)”

Install aborted validating the app definition. Causes, in rough order of likelihood:

  1. Required variable not provided. The app declares vars.admin_password@secret! but you didn’t pass --var admin_password=.... Check which fields are ! (required) in the app’s vars: block or vars.yaml.
  2. Required plug has no viable socket. An app with required: true on a plug refuses to install unless a matching socket exists. Either install the provider stack first, or pass -i <provider>.
  3. Integration spec doesn’t resolve. Error detail says “no matching sockets found” or “multiple plugs match”: use the explicit -i plug=provider.socket form. See plugs-and-sockets: CLI spec.
  4. Malformed app.yaml or vars.yaml. YAML parse error or a field of the wrong type. Run shc validate <path-to-app-dir> to get a precise schema error.

Symptom: X409003STKEXS (“stack already exists”)

Section titled “Symptom: X409003STKEXS (“stack already exists”)”

A stack with that (tenant, stack, environment) tuple is already installed.

Terminal window
shc ls --all-tenants # find the conflicting stack
shc uninstall <stack> -e <env> # remove it, OR…
shc install <app> -s <name> -e <env> # pick a different stack/env

Symptom: Stack starts but immediately fails

Section titled “Symptom: Stack starts but immediately fails”

Install reached compose up but containers exit. The install flow waits for a healthcheck window (configurable per-app via timeouts.start) and then rolls back.

  1. shc logs -s <stack>: raw container logs
  2. shc events --follow: daemon lifecycle events
  3. shc events: structured events
  4. Check X500907STKFAL detail in the logs; it includes which service’s healthcheck failed

Common root causes:

  • Missing integration variable: template rendered DATABASE_URL=postgres://user:@host/db (empty password) → provider integration hadn’t finished when compose started. Re-run with -i <provider> to wire the integration, or install the provider first.
  • Image architecture mismatch: running on arm64 but the app’s image is amd64-only. Check requires.architectures in app.yaml and the image’s manifest.
  • Port already in use: something else on the host is bound to a public_ports target. ss -tlnp | grep <port>.
  • Volume mount failed: the storage provider refused the mount path; check shc doctor run --category storage.

Symptom: Install hangs on “pulling image”

Section titled “Symptom: Install hangs on “pulling image””

Docker pull is slow or unreachable from the target node.

Terminal window
# On the target node:
docker pull <image> # reproduce directly
shc node ls -v # find which node it targets
shc inspect deployment <stack> # see assignment

If pulling works manually but SHC’s pull hangs, the issue is usually Docker daemon health. Restart Docker on the node.

Symptom: X404002APPNOF (“app not found”)

Section titled “Symptom: X404002APPNOF (“app not found”)”

The app name isn’t in any registered repo.

Terminal window
shc repo ls # check repos are configured
shc repo sync # force-refresh
shc search <partial-name> # confirm the app exists

If it’s a local file path repo, make sure shc has read access to the directory.


Symptom: X400903INVAPP (“no matching sockets found”)

Section titled “Symptom: X400903INVAPP (“no matching sockets found”)”

Auto-match failed because the consumer’s plugs don’t match any of the provider’s socket names.

Terminal window
# See the consumer's plugs:
shc inspect app <app> # lists plugs + their target socket names
# See the provider's sockets:
shc inspect stack <provider-stack>

If the names don’t match (e.g. your plug wants postgres but the provider exports database), use explicit syntax:

Terminal window
shc install myapp -i postgres=provider-stack.database

Plug or socket name in your -i spec doesn’t exist on the named stack. Typo or stale spec. Verify names with shc inspect stack.

Symptom: Consumer renders a literal {{ integrations.X.Y }}

Section titled “Symptom: Consumer renders a literal {{ integrations.X.Y }}”

The integration didn’t register before compose rendered, so the template variable was unset. This almost always means the integration phase job failed silently.

Terminal window
shc integration ls <stack> # verify the edge exists
shc job history # find the failed integration-phase job
shc job show <run-id> # get full stdout/stderr

Re-run the integration:

Terminal window
shc update <stack> -i <plug>! # detach the broken edge
shc update <stack> -i <plug>=<provider> # re-wire it

Symptom: X400954XNDSCK (cross-node socket blocked)

Section titled “Symptom: X400954XNDSCK (cross-node socket blocked)”

The provider’s socket has local_only: true, meaning it can only serve consumers on the same node. Your consumer is on a different node.

Options:

  • Install the consumer on the same node (shc install <app> --where 'node=<same-as-provider>')
  • Remove the local_only restriction in the provider app’s socket YAML (only appropriate if the exposed service is actually mesh-reachable)

A pre_backup or post_backup hook (or a backup converter) failed. SHC logs the job error but does not abort the backup, so this is a warning, not a fatal error, unless the backup evidently lacks expected content.

Terminal window
shc job history # find the failed hook run
shc job show <run-id> # full output

Then:

Backup failed before capture because restic couldn’t access its repo.

  1. Confirm backup.repo in your config points at a reachable location, and that backup.password is set; they are required together, and either one alone is refused
  2. If backup.password holds a ref+vault:// reference, confirm it resolves: an unresolvable reference (locked vault, absent key) refuses rather than falling back. Unlock with shc vault unlock, then read the path back with shc vault get backup/password
  3. For NFS/local shared repos, confirm the mount is present on every node (the node doing the backup is the node the stack lives on). See storage-providers: NFS.

Symptom: Restore succeeds but stack won’t start

Section titled “Symptom: Restore succeeds but stack won’t start”

Restore completed (metadata, volumes, vault secrets, integrations) but the restored stack’s containers fail to come up.

Most common causes:

  • Integrations didn’t replay. Restore re-creates integration rows but may skip re-running the integration phase jobs if the provider is unavailable at restore time. Check shc integration ls <stack>: if edges exist but wait_for jobs aren’t present in provider DBs, re-issue the edge so the integration phase runs again:
    Terminal window
    shc update <stack> -i <plug>=<provider>
  • Vault scope mismatch. Restoring into a different tenant or environment means ref+vault://...?scope=deployment references resolve to nothing until you set values in the new scope. See vault: scopes.
  • Image version drift. The app was installed at v1 but now resolves to v2, and the restored data is v1-compatible. Pin versions in app.yaml or use a repo at a fixed ref.

Symptom: shc node join hangs at “enrolling”

Section titled “Symptom: shc node join hangs at “enrolling””

The joiner needs HTTPS 443 to the cluster’s public enrolment route through Traefik, then UDP 4244 between node underlay addresses for mesh0.

Terminal window
# On the joining node, check TCP connectivity to the HTTPS entrypoint:
nc -zv <cluster-host> 443
# On the leader:
ss -tlnp | grep ':443' # HTTPS entrypoint
ss -ulnp | grep ':4244' # mesh0 UDP listener
shc cluster mesh status
shc cluster mesh diagnose # peer internal APIs over mesh0

A listening UDP socket alone does not prove reachability. Inspect mesh handshake status and diagnose output after enrolment.

See cluster join flow for the full handshake.

Symptom: shc node ls shows a node as unhealthy forever

Section titled “Symptom: shc node ls shows a node as unhealthy forever”

Either the node is actually dead (see disaster recovery: Scenario 3) or the healthcheck probe can’t reach it.

Terminal window
# On any healthy voter:
shc cluster status # node counts, voter/quorum state, leader
shc cluster ping <node> # probe that node directly
shc cluster mesh diagnose # this node probes peer internal APIs over mesh0
# Run this check from the "unhealthy" node itself if you have shell:
shc node ls # if it also sees itself as lonely,
# mesh is broken from its side

Symptom: Writes return 503 / X500021CLSINT

Section titled “Symptom: Writes return 503 / X500021CLSINT”

Raft lost quorum. Recover per disaster recovery: Scenario 2.

Logs show cert expiry warnings or rotation errors.

Terminal window
shc cluster cert ls # report cert ages across the cluster
shc cluster cert rotate # force-rotate everything that's due
shc cluster cert rotate --type mtls # rotate only one category

See disaster recovery: Scenario 5.


Symptom: an operation fails with “vault is locked”

Section titled “Symptom: an operation fails with “vault is locked””

X503903VLTPWM. This reports a runtime lock state, not a configuration fault: the vault exists and is initialized, but no master key is held in memory right now. Nothing is missing from your config.

On a cluster follower this is usually normal and self-healing. Followers deliberately hold no copy of the master password. They boot locked and pull the unlock from an already-unlocked peer within seconds. A burst of X503903VLTPWM in the first moments after systemctl restart shc is the expected shape; the pull loop logs vault unlock pull: loop armed → knocking peers → peer accepted push-back request, then goes quiet. Silence is the success signal. A command typed on the follower in those seconds waits for the unlock, up to 20 s, instead of being refused X503906VLTLCK. The refusal comes at once only on a node no peer can unlock: a single node, or one whose peers are all locked.

Confirm rather than assume: shc vault status prints a two-row table whose header contains the word LOCKED, so a naive grep reports a false positive:

Terminal window
shc vault status

If a node stays locked past a minute:

  • Follower: it has no unlocked peer to pull from, or the peer transport is down. Check that another node reports locked:false, then look for the pull-loop lines above. Do not “fix” this by putting SHC_VAULT_PASSWORD on the follower, keeping the master password off every non-genesis node is deliberate.

  • Genesis / single node: this one does carry the password, because at genesis there is no unlocked peer to pull from. It is sealed with systemd-creds in /etc/systemd/system/shc.service.d/vault-autounlock.cred, which shcd opens at boot. On launchd it is a System keychain item (io.selfhostedcloud.shc.vault-unlock). If the credential is absent, or shcd’s log says it did not open (the host key or the TPM it was sealed to changed, or the keychain item could not be read), unlock manually. The unlock also seals a fresh credential, so the next restart auto-unlocks (see vault: the sealed auto-unlock credential). The vault.autounlock_at_rest row of shc doctor run -c vault says which of these the node is in. On a node whose systemd is older than 250 there is no credential at all (X200052VLTSKP), and every restart needs the manual unlock. If shcd was started with SHC_VAULT_PASSWORD set it refuses with X400975VLTRFE and does not fall back; move the node with the steps in that section. The rewrite of the secret file is the operator’s. Let the command prompt (echo is suppressed on a terminal), or pipe the password in. Never pass it as an argument, which leaks it into shell history and ps:

    Terminal window
    sudo shc vault unlock # prompts, no echo
    cat /path/to/password-file | sudo shc vault unlock

    -p / --password takes the password value, not a path to one: -p /path/to/file tries to unlock with the literal string /path/to/file and fails. There is no file form of the flag.

See vault: unlocking.

Symptom: X500961VLTCRP (vault crypto failure)

Section titled “Symptom: X500961VLTCRP (vault crypto failure)”

The stored blob can’t be decrypted with the current master password. The {op} token names the step that gave up (base64 decode, ciphertext too short, bad fernet key), so quote it when reporting. A value that is not in the envelope format at all fails earlier and differently, with X400980VLTFMT.

  • You recently rotated and the rotation didn’t finish → restart daemon, retry
  • You restored a backup from a different cluster → rotate to the backup’s original password, then rotate again to your new one
  • Hardware corruption of the state database (/var/lib/shc/shc.db, or /var/lib/shc/rqlite in cluster mode) → restore from a clean backup

Symptom: ref+vault://key renders unresolved

Section titled “Symptom: ref+vault://key renders unresolved”

The resolver returned None for the key. Checks:

  1. Does the value exist? shc vault exists <key> at the exact (scope, tenant, stack, environment) you expect.
  2. Is the scope in the URI what you think? ?scope=tenant vs ?scope=deployment resolve against different keys.
  3. Is the vault healthy? shc doctor run --category vault reports lock state and flags broken (empty-value) rows. An unreachable external provider surfaces as X503902VLTNAV (vault not available) in the daemon log.
  4. Did you autogen? ?autogen only generates on first render; if the first render failed (vault write error), the key may still be absent on retry. Check for X500901OPFAIL in logs.

Terminal window
# 1. Is there actually a process?
systemctl status shc
# 2. What did it log?
journalctl -u shc -n 200
# 3. Can it reach its state dir?
ls -la /var/lib/shc/
sudo -u shc test -w /var/lib/shc && echo OK
# 4. Are required ports free?
ss -tlnp | grep -E '4001|4002|4003'
ss -ulnp | grep 4244
# 5. Is the raft engine's data corrupt? (cluster mode)
ls -la /var/lib/shc/rqlite/
# Look for a fresh 'raft.db' and old logs. If this is a cluster
# member whose peers are gone, see disaster-recovery Scenario 2.
  • “address already in use”: another process owns the port. Find it with ss -tlnp. SHC’s internal API is TCP 4003 by default; change via internal_api.port in config.
  • “permission denied” on /var/lib/shc/...: systemd user/group mismatch. Verify with stat /var/lib/shc and id shc. The state dir must be owned by the shc user.
  • “vault is locked” (X503903VLTPWM): a runtime lock state, not a missing setting; on a follower it is normally transient. See vault section above.
  • “raft: failed to open snapshot”: the raft engine’s snapshot is corrupt. See disaster recovery: Option C, restore from backup.

If none of the above helps, turn up the logging and re-run:

Terminal window
# Run the daemon in foreground with debug logging:
systemctl stop shc
sudo -u shc SHC_LOG_LEVEL=debug shcd

…and inspect the full structured log (not just human messages) via shc logs --follow (on a separate shell) while reproducing the issue.

For production incidents, gather the diagnostic surfaces into one place before opening an issue:

Terminal window
shc doctor run > doctor.txt # every registered diagnosis
shc version > version.txt # version + build info
shc cluster status > cluster.txt # voter/quorum state, leader
shc node ls > nodes.txt # membership + node health
shc events --audit -n 200 > audit.log # recent audited actions
journalctl -u shc --since -2h > daemon.log # structured daemon log