Security model
Layer 2. Core. Core
How SHC authenticates callers, how it authorizes actions against tenants
and resources, how tenants are isolated from each other, and how
cluster-internal traffic is secured. This page is the canonical
reference; the core/api/auth.go,
modules/auth/types.go, and
modules/auth/middleware.go modules are the source
of truth.
1. Authentication
Section titled “1. Authentication”Every request resolves a caller through Service.CurrentUser in
modules/auth/service.go,
using the auth precedence extracted by ExtractAuth in
core/api/auth.go.
There are exactly two transports that can authenticate a caller:
1.1 Unix Domain Socket (local admin)
Section titled “1.1 Unix Domain Socket (local admin)”Requests arriving over the UDS listener (default
/run/shc/shc.sock) are treated as local admin requests and minted by
NewLocalUser (modules/auth/types.go)
as a synthetic *User with roles ["platform:manage"] and a wildcard
permission {Resource: "*", Scopes: ["*"]} (wildcard tenant access
comes from IsPlatformAdmin() / CanAccessTenant, not a tenants
set). Access to
the socket file is gated by filesystem permissions; the socket is
owned by the shc user, and its group is shc; members of that group
(set with usermod -aG shc <user>) can open it. Everything else
cannot.
The detection lives in the udsMarkerMiddleware that tags the request
with Transport=uds at ingress; CurrentUser mints the local admin
when that marker is present.
Authorization bypass (see §2.4) keys off the unexported isLocal
bool flag on *User (set only by NewLocalUser), not a claim string
or a Go type assertion, so a
Keycloak user whose sub happens to equal "local-user" cannot
escalate.
1.2 Keycloak (everyone else)
Section titled “1.2 Keycloak (everyone else)”Every other transport on the public listener, HTTP(S) and
WebSockets, must present a Bearer token
that validate_keycloak_token(token) accepts. The token is validated
against the Keycloak realm configured in authentication.keycloak.*
(see config keys). On
success, the returned User includes the sub, roles, tenant
claims, and resolved permissions (§2.2).
Failures raise X401900NOAUTH (no credentials) or X401901* (token
invalid). Failed authentications emit A401000AUTHFL audit events
with a reason tag.
1.3 Public client API: TLS
Section titled “1.3 Public client API: TLS”The Keycloak-protected, tenant REST API (what clients and the CLI use
over the network) reaches shcd only through Traefik on 443. shcd’s
public listener is plaintext on port 8000 at three addresses and never a
wildcard: 127.0.0.1 for callers on the node, the container gateway
where shc:host-gateway sends this node’s system ingress (the node’s
fabric address on Linux, each system-ingress pin’s vmnet gateway on a
Mac), and the mesh address for operator mesh clients (the http
listener subsystem in subsystems/impl/http.go).
The node plane drops 8000 on every interface but lo, mesh0 and the
system ingress’s own veth, so a host that routes one of those addresses
through the node gets no answer either; a Mac’s pf anchor holds the same
drop. server.tcp.bind replaces the three with one address for dev
servers and the e2e harness, and SHC_PORT moves a test daemon to its
own loopback port. shc doctor’s api.public-listener reads what the
listener holds and is red on a wildcard or an uplink address however it
got there. The public host is SHC_HOST.
TLS is not terminated inside SHC. Traefik (installed as a system
app) fronts this listener and handles TLS and routing by host+path.
Traefik runs no ACME client. It serves the certificates the daemon
writes: a leaf from the tenant’s internal CA, or one the daemon’s ACME
source ordered (see Networking).
2. Authorization
Section titled “2. Authorization”Once a caller is authenticated, every endpoint that changes state (and
most that read it) is wrapped with RequirePermissions(...) from
modules/auth/middleware.go. The middleware runs
four checks in order.
2.1 Context present
Section titled “2.1 Context present”RequirePermissions pulls the Ctx object from the request context. If no Ctx is available the
request is rejected with X403000AUTHZF. This is a programming error,
not an auth failure, and indicates a route whose gate was built with no
auth Service to resolve, or a request with no context.
2.2 User present
Section titled “2.2 User present”Non-UDS requests must carry a resolved user. Missing user ⇒
X403903NOUSER.
2.3 Tenant membership (IDOR guard)
Section titled “2.3 Tenant membership (IDOR guard)”Before any scope check, ctx.user.can_access_tenant(ctx.tenant_name)
must return True. The tenant on the Ctx was derived from an
untrusted source (the X-SHC-Tenant HTTP header, or the
--tenant CLI flag), so this check is mandatory. Without it, a user
with apps:write on tenant A could set the header to tenant B and
write apps in B.
CanAccessTenant (modules/auth/types.go)
considers three cases (the User model carries a single
TenantName string field; there is no multi-valued tenants set):
- Empty tenant (
tenantName == "", cluster-wide probe) → allow; the per-scope check is the real gate. - Platform admin (
IsPlatformAdmin(), holdsplatform:manage, or is the syntheticLocalUser) → allow. user.TenantName == requested→ allow. This single-tenant equality is the primary match for Keycloak users.
Denial raises X403904NOTENT and emits an A403002TENTDNY audit
event with the attempted tenant and the user’s sub.
2.4 Scope & resource check
Section titled “2.4 Scope & resource check”The required scopes are the arguments to RequirePermissions(...), e.g.
RequirePermissions(resolve, "stack:read"). The current resource is
derived from the Ctx by resolveResource:
- If the
Ctxhas no stack and no environment → resource is"tenant". - If the
Ctxhas a stack → resource is"stack:<stack_name>". - If it only has an environment → resource is
"environment:<name>". - Otherwise →
"tenant".
HasPermission(userPermissions, resource, requiredScopes) then
walks the user’s Permission objects:
// pseudocodefor _, perm := range user.Permissions { if matchResource(perm.Resource, currentResource) { for _, userScope := range perm.Scopes { for _, required := range requiredScopes { if matchScope(userScope, required) { return allow } } } }}return denyDenial raises X403900NOPERM(scope=…, resource=…) and emits an
A403001PERMDNY audit event.
LocalUser instances short-circuit this entire pipeline. That’s
the bypass from §1.1.
2.5 Scope syntax
Section titled “2.5 Scope syntax”Scopes are "<noun>:<verb>":
match_scope(user, required)allows:"*"→ matches everything.- Exact match (
stack:write=stack:write). - Noun wildcard: user has
*:read, required isstack:read→ allow. - Verb wildcard: user has
stack:*, required isstack:delete→ allow.
match_resource(user, required) uses fnmatch globs, so a policy
with resource="stack:prod-*" will match stack:prod-web but not
stack:dev-web.
2.6 Canonical scope vocabulary
Section titled “2.6 Canonical scope vocabulary”Most modules gate access via CanAccessTenant + a request-scoped
tenant check rather than a per-noun RequirePermissions vocabulary. The
explicit RequirePermissions(...) middleware guards only a handful of
routers with these scopes.
Source: rg "RequirePermissions\(" over the ce module and this repo’s trees (non-test call sites).
| Scope | Used by |
|---|---|
platform:manage | dns, cluster, storage, tenant, task, backup, auth, node, config routers (admin gate) |
audit:read | audit |
events:read | events |
stack:read | logs |
3. Policies, roles, and caching
Section titled “3. Policies, roles, and caching”Users don’t carry permissions directly in their token; they carry
roles. The server translates roles → permissions by evaluating
Policy objects (typically sourced from Keycloak custom policies;
see PolicyFromKeycloak). Each Policy carries:
nameroles: if the user has any of these, the policy matches.resources: which resource patterns (stack:*,stack:prod-web,tenant, …) the policy grants access to.scopes: which scope strings (stack:read,backup:write, …) are granted on those resources.
get_permissions_for_roles(policies, roles) flattens matching
policies into the Permission list that the user object carries into
RequirePermissions.
Policy evaluation is expensive (Keycloak round-trip for policy
lookup), so results are cached per-client in PolicyCache with a TTL
taken from keycloak.policy_cache_ttl (default 60 seconds). The
cache is an in-process singleton; if a policy changes in Keycloak, it
takes up to the TTL to propagate. Call PolicyCache.invalidate() to
force a refresh.
4. Tenant isolation
Section titled “4. Tenant isolation”Tenants are the top-level namespace in SHC. Every stack, environment, schedule, backup, audit row, and vault secret is rooted at a tenant. Isolation is enforced at four layers:
-
Authorization (above): the IDOR guard in
RequirePermissionsblocks cross-tenant requests before the handler runs. -
Database: every table that carries user data has a
tenant_namecolumn; every query includes it in theWHEREclause (see thequeries.gofiles in each module). -
Filesystem: per-tenant directories under
paths.TenantsDir(tenant)(see core/paths/paths.go). Compose project names (viabuild_docker_project_name(tenant, stack, env)) carry the tenant so container names don’t collide. -
Networks: on the shc fabric every container is one routed
/32, its pin, on a link of its own to the host. No tenant has a bridge and no two pins share a link: a pin’s tenant is a label the node’s compiled nftables policy reads. That policy is default-drop between fabric addresses on every node, for a co-located pair as for a cross-node one. A deployment reaches itself on every port, another deployment of its tenant reaches a service on the ports it declares, and another tenant reaches it only through the system tier’s roles. A deployment’s own compose networks are node-local, and the render joins no consumer to a provider’s network.The same-node line is drawn by interface.
fabric_sourcesdrops a packet on a pin’s veth that does not wear that pin, and one wearing an address of the node’s chunks that entered on no pin’s veth, so a source cannot be forged, by a container holdingCAP_NET_ADMINincluded.fabric_entrydrops a packet entering a pin’s veth from any local bridge: a deployment’s own network,docker0, a podman or incus network.fabric_outsidekeeps every host outside the fabric off the pins: a packet for a node’s chunks from a source outside them that did not enter on shc0 passes only as a reply, an ICMP error about a flow a pin opened, a listener’s flow to a published port or an incus collector’s export, so a LAN host that routes the pin range through a node gets nothing and reaches an app only through its publish or its Traefik route (SHC-3484). The crossing guard (cemodules/fabric/crossing.go,ip shc_fabric_crossing) drops a pin’s packet leaving on a local bridge for a destination outside the fabric. It spares the node’s uplink where that is a bridge (br0,vmbr0, a bridge over a VLAN), since a packet leaving there leaves the node, and on an incus node it lets a pin answer a flow the node’s collector network opened, in the reply direction only. On a Mac the pf policy anchor holds each replica network’s bridge to its one address in place offabric_sources.The authority for what holds and what does not is ce
modules/fabric/policy/doc.go, under “What is a boundary and what is not”.
What is not isolated: host kernel resources, the control plane process, and the underlying docker daemon. SHC is a multi-tenant-within-one-trust-boundary system, not a hostile- multi-tenant system. Operators should not run untrusted-code tenants on the same node.
5. Cluster-internal traffic
Section titled “5. Cluster-internal traffic”Cluster-internal traffic falls into two kinds:
5.1 Node-to-node RPC (internal API)
Section titled “5.1 Node-to-node RPC (internal API)”The internal API (/internal/*, mounted by buildRouter in
modules/cluster/router.go) is used for leader →
follower operations, cross-node shc exec, log streaming, etc.
It is authenticated with mTLS: each node presents a client
certificate signed by the cluster CA
(modules/cluster/cert_provider.go).
Certificates are minted at cluster-join time using a short-lived join
token (§5.3) and rotated by CE subsystems/impl/internal_cert_rotation.go.
Nodes call each other’s internal API on TCP 4003 at the mesh0 address. The listener binds only this node’s mesh0 address, never the underlay. Cluster members authenticate internal calls with their cluster-CA-signed client certificates. These certificates are separate from mesh0’s WireGuard keys.
A joiner reaches POST /api/join/node.enroll through Traefik on HTTPS 443
into the public API, authenticated by its single-use join token. The joiner
verifies the signed response against the trust bundle in that token before
using the returned identity and mesh0 grant. Sources: SHC
modules/cluster/enroll_rpc.go, join_client.go; two-tunnel design §7.
Allow UDP 4244 for mesh0 and UDP 4243 for shc0 between node underlay addresses. rqlite (TCP 4001/4002) binds and dials mesh0 addresses, and nodes dial the internal API (TCP 4003) on mesh0, so none of these ports needs a public firewall opening. The internal API listener binds only the node’s mesh0 address.
5.2 Raft consensus (rqlite)
Section titled “5.2 Raft consensus (rqlite)”Durable cluster state (tenants, stacks, schedules, quotas, audit
log, …) lives in rqlite, raft-replicated
SQLite, run as an external rqlited process the daemon supervises.
Raft replication runs on :4002 between members; members join as
voters by default, and a non-voter member replicates read-only.
Transport security (#58 raft mTLS). The raft plane is gated by the
RaftTLS posture recorded in node_bootstrap.json:
- Genesis default-on: a from-empty
shc initalways writesRaftTLS: true(modules/cluster/service.go), so every fresh cluster runs raft replication (:4002) and the rqlited HTTP API (:4001) under mutual TLS:rqlitedgets-node-cert/-node-key/-node-ca-cert+-node-verify-clientand-http-cert/-http-key/-http-ca-cert(core/rqlited/manager.go), using the per-node cluster certs under{state}/certs/(ca.pem,node.pem,node-key.pem). - Leader-inherited: a joiner never decides its own posture; it
adopts the leader’s
RaftTLSfrom the enroll response (modules/cluster/enroll_handler.go), so a cluster is always uniformly TLS or uniformly plaintext. - Legacy plaintext: a
node_bootstrap.jsonwritten before the field existed decodesRaftTLSas false, and that cluster stays plaintext through binary upgrades; flipping it to TLS is a deliberate all-nodes operation, not an upgrade side effect. For those clusters, firewalling :4002 between members remains the only protection. - File-gated bring-up: if the posture says TLS but the cert
material is missing, genesis falls back with a logged warning
(
X503200RFTTLW) while a joiner refuses to start (X500211RFTTLF) rather than silently joining plaintext.
5.3 Cluster join tokens
Section titled “5.3 Cluster join tokens”Short-lived join tokens are minted by the leader (shc node token)
and presented by a joining node to bootstrap mTLS. Token generation
lives in
modules/cluster/token_router.go (the shc node token
route) backed by the single-use ledger in
modules/cluster/meshtoken_repo.go;
the handshake itself is in
modules/cluster/enroll_handler.go.
Tokens are single-use and expire quickly (default: minutes).
5.4 Mesh networking (mesh0)
Section titled “5.4 Mesh networking (mesh0)”The node plane is mesh0 on UDP 4244, coordinated by shcd: kernel WireGuard
on Linux, or userspace WireGuard on a utun on macOS. Raft, the internal API
and VIPs ride mesh0. Each node keeps its own private key; the public key is
recorded in mesh_peer.
The leader signs the peer table, and revocation tombstones the row. shc0
carries containers over separate underlay endpoints on UDP 4243. See
Two WireGuard tunnels.
Built (SHC-1481). Apple containers are held off the node plane by the pf
child anchor com.apple/shc/node-plane, compiled from the same HostInput
as Linux’s nft table inet shc_node_plane. The rules match source addresses
from every apple network subnet and protect control ports 4001/4002/4003,
the node’s own addresses and mesh prefixes, with the same allowances. Stock
/etc/pf.conf evaluates anchor "com.apple/*", so shcd never edits it.
shcd loads the rules with pfctl -a com.apple/shc/node-plane -f -, holds a
pfctl -E reference token for its lifetime and releases it with
pfctl -X <token> on stop. A leave empties the anchor.
6. Secrets at rest (vault)
Section titled “6. Secrets at rest (vault)”User-facing secrets (ref+vault://… references in compose templates,
app vars marked @secret) are stored via the
vault module, which is itself multi-provider:
Local (SQLite with a cluster-distributed KEK), AWS Secrets Manager,
HashiCorp Vault, Azure Key Vault, GCP Secret Manager, 1Password.
Secrets are resolved lazily at read time by
core/config/secrets.go:
Config.GetResolved parses a ref+vault://… value via ParseSecretRef
and dispatches it through the provider Dispatcher() resolver. They are
never committed to
the rendered compose file that Docker sees. They are injected
directly as environment variables or mounted as tmpfs files at
container start.
7. Audit trail
Section titled “7. Audit trail”Every authorization decision that matters emits an audit event:
| Event | Meaning |
|---|---|
A401000AUTHFL | Authentication failed (bad/missing token). |
A403001PERMDNY | Authorization denied: user didn’t have the required scope on the resource. |
A403002TENTDNY | Tenant IDOR attempt: user tried to target a tenant they aren’t a member of. |
The full reference lives at errors.md. Audit
rows are hash-chained (see audit module) so
tampering is detectable.
8. What SHC is NOT
Section titled “8. What SHC is NOT”- Not a Kubernetes / Coolify replacement for hostile multi-tenancy. One trust boundary per node. Tenant isolation is for organizational separation, not adversarial separation.
- Not a full IAM system. Role and policy management happens in Keycloak; SHC only consumes the resulting token. Role design is the operator’s responsibility.
- Not a secret manager. Vault providers are external systems; the
vaultmodule is a thin, uniform facade.
9. Related reading
Section titled “9. Related reading”- Architecture: vault module
- Architecture: cluster module
- Architecture: node module
- Architecture: audit module
- Error reference:
X40*/X50*auth errors - Config keys:
authentication.*andauthorization.*
Image vulnerability scanning
Section titled “Image vulnerability scanning”The EE scan module resolves deployment images from the daemon inspect response. Catalog app names resolve through the installer on an authenticated daemon route; local app directories use pkg/appimages.ResolveRuntime, shared with the runtime image pin guard. Tag-only images remain in the report as unscanned.
The CLI sends a private JSON request to its sibling shc-scan process. Only the helper links Syft and Grype, avoiding a collision between their SQLite driver and the daemon driver. The helper pulls digests directly from registries with Docker credentials and selects linux/amd64 for indexes. Two image workers share a validated vulnerability database under the CLI cache home at scan/db. A database older than five days must refresh from its recorded source before scanning.
The CLI renders one CycloneDX 1.6 VDR, SARIF 2.1.0, or a text table. CycloneDX preserves package references and unscanned reasons. Exit 1 means failure or incomplete coverage; exit 4 means a complete scan met the requested severity threshold. See scan command reference.
Grype matches the packages inside an image, not the app the image ships, so it misses GitLab’s own CVEs in the omnibus image. The helper also matches each app’s version against the sources its package declares in advisories.yaml: an OSV package (osv: Bitnami/gitlab) and, for GitLab, a feed naming the patch-release Atom feed and the product to filter on (GitLab CE). An app with no source says why in none:. The reader refuses a missing or empty file, bad YAML, an unknown key, sources together with none:, a blank none:, and a source with no repository: or package: to read the version from, and each refusal names the file and the fix. The catalog test runs the same reader over every package the publisher would ship, at any depth. SHC holds the feed parsers and no app names. A deployment scan reads the declaration from the deployment’s catalog app. The app is an application component, and its findings share the image findings’ vulnerability shape and --fail-on rules.
The version comes from the package the declaration names in the scanned SBOM, then the tag of its repository image, then app.yaml when it is a patch version. app.yaml usually names a release line such as 18.11, and an incus container reference has no tag, so the SBOM package comes first. An unknown patch version or a failed source leaves the app unscanned.