Fabric policy follows the tenant model
Status: built; this note is still its only description (SHC-1202). The bridge allocator it once left for a follow-up is built too (SHC-1282), and so is later compiler work this note does not describe: the publication class for published ports (SHC-1432), routed veths for incus pins, relay transit on the host plane and the pf renderer for a Mac node.
SHC-1202, 2026-10-01. New line (next). Extends fabric slice 2 (SHC-1055).
Rulings it implements: shc-network-policy-tenant-boundary (2026-09-30),
shc-module-api-access-model (2026-09-30), shc-expose-and-publication-model
and the address-plan row of shc-newline-rulings-20261001-table (2026-10-01).
The model, as compiled
Section titled “The model, as compiled”The unit is the deployment. Nothing finer.
| flow | verdict | where it is decided |
|---|---|---|
| inside a deployment | open, any port | ClassDeployment |
| same tenant, to a service that declares ports | open on exactly those container ports | ClassTenant |
| same tenant, to a service that declares none | dropped, the service is private to its deployment | default-drop |
| across tenants | dropped in every direction, declared port or not | default-drop, plus the bridge line on a per-tenant-bridge node |
| any deployment to the collector | open on its declared ports | ClassSystem |
| any deployment to the platform Keycloak (master/default) | open on its declared ports, nothing else in master | ClassSystem (shared provider) |
| any container to cluster DNS (shcd on the node’s mesh address, 53) | open | shc_node_plane |
| ingress (Traefik, forward-auth) to a routed backend | open on the route port, every tenant | ClassIngress |
| any container to the node plane | dropped | shc_node_plane |
ports: and expose: both only declare. The render reads the container side
of each (expose entries, short and long ports: forms) and the pin ledger
stores them as proto/port (fabric_pin.ports), a range as tcp/8000-8002,
which the compiler opens whole and no wider. Every service still gets a pin,
so a private service is still a source.
There is no connection class. A plug is wiring (names, credentials) and grants
nothing, in the render as well as in the compiler: on the fabric no consumer
joins a provider’s own network (that would reach every port of every service
on it, across the tenant line for a magic provider), and no tenant deployment
joins the system Traefik’s discovery network, the one segment every routed
service of every tenant used to share; Traefik dials a route on the service’s
pin. A same-tenant provider is
reached at its pin through extra_hosts on whichever node it runs, carrying
the alias strings the consumer would have resolved on a shared network; a
headless provider’s endpoint (<env>.<service>.<stack>, the s3 dispatcher’s)
maps to that service’s pin. A
provider reference that is not one plain name (../globex/pg) is refused
before any lookup. A plug whose provider sits in another tenant is refused where it is
declared, at install (ValidateIntegrationProviders), at render
(ResolveIntegrations) and at Connect, with X400372FABXTN naming both
tenants and both ends. The one legal cross-tenant plug is the magic allowlist
(KEYCLOAK), because its provider lives in the system tier.
The system tier is derived
Section titled “The system tier is derived”The system tier is master’s platform environment, master/default. That is
where the builtin Traefik deployment (SystemIngressTenant/Stack/
Environment) and every magic provider (MagicProviderEnvironment) already
live, so membership is a scope check, never a list. fabric restates the scope
because modules/app imports fabric; a test pins the two equal. Pins of
master/default/traefik carry the ingress role. A role on a pin outside the
tier is refused (X400364FABPOL).
Pins of master/default/collector carry the collector role (SHC-1915): the
telemetry collector is a pinned deployment there, one replica on every node.
There is no resolver role: shcd answers cluster DNS itself on the node’s mesh
address (SHC-1213), so that accept is the node plane’s, not a pin’s.
The OTLP carve-out is retired
Section titled “The OTLP carve-out is retired”Until SHC-1915 the collector was a per-node host-published workload on
docker’s default bridge. An app exported to host-gateway:4318; docker’s
PREROUTING DNAT rewrote the destination before filter, so the forward hook saw
fabric bridge to docker0, the shape of a bridge crossing, and the crossing
guard and the iptables glue each kept it alive with an accept keyed on
pre-DNAT identity (ct status dnat ct original proto-dst 4318). The collector
is a pinned deployment in master/default now: every tenant reaches it
through ClassSystem on its declared ports, nothing is published on a host,
and both accepts are gone. The crossing guard is one drop, and the glue
deletes the old rule from a node that still holds it. A pin dialling a port
the engine publishes on its node for a container on another bridge, 4318
included, is dropped as a crossing.
Conntrack admits replies only
Section titled “Conntrack admits replies only”Every conntrack head of the address chains and the node plane accepts ct state established,related ct direction reply and nothing else. The bridge
chain’s head is narrower still: only the reply of a DNAT’d connection (a
published port), so a forged reply of a masqueraded or
cross-node flow sent from another tenant’s network meets the bridge drop
instead of ending the base chain with an accept (measured, with the generic
head as the mutation that lets it through). An original-direction packet is judged by
the rules on every packet, so a grant that goes away (an edge the old
connection class allowed, a container’s open connection to the internal API
when the node plane first lands) stops at the next apply instead of living as
long as the flow, and a forged tuple of somebody else’s tracked flow is never
waved past the bridge line. The kernel lab measures both halves: an open
connection to rqlite stops, one to an unclosed port keeps flowing.
The DNAT-reply head alone would still take a forged reply of a DNAT’d flow:
a container on another network can emit the DNAT target’s reply tuple (to the
bridge gateway the engine masqueraded the flow to). So the source chain,
jumped first for every forwarded packet on every node, ends in a strict
reverse-path drop for bridge ingress (fib saddr . iif oif missing): a source
that does not route back through the bridge it came in on is dropped before
any accept sees it. The lab forges both replies from a second non-fabric
network: the reverse-path rule counts them, and with that rule removed the
forged DNAT reply arrives at the exporting container.
The node plane needs the same guard on its own. Its chains open with a conntrack head too, and a locally delivered packet never meets the policy table’s source chain, so a container could forge the reply of a connection the node itself opened (to a mesh peer on raft, to anything) and the head would accept it into the node’s socket. Both node-plane chains therefore open with the reverse-path drop, per family, ahead of the head; only the unspecified source (DAD, DHCP) is spared, since it is no reply to anything. The lab forges a mesh peer’s reply to a UDP flow the node opened: dropped and counted, and with the pair removed it reaches the node’s socket.
Data quality and refusals
Section titled “Data quality and refusals”A bad row never stalls the cluster: the builder leaves out a pin row the
compiler would refuse (an unparseable address, one outside its node’s grant,
the device address, every row of an address two rows claim, and on a node
with a bridge per tenant one outside its tenant’s bridge or of a tenant with
no bridge there) with X200376FABQRT, and hands the compile the canonical
address it validated. A node that carves a bridge per tenant must carve one for
every tenant placed on it, or the compile refuses (X400365FABTEN); a missing
bridge never falls back to the shared-bridge line, and no broker grant is
issued there without a usable system bridge, nor when the node row is
duplicated or its inventory fails the compiler’s own bridge checks
(policy.NodeBridgesValid), because the node plane is applied even when the
compile refuses. The render refuses instead of
falling back to the shared networks the fabric replaces: a node with no grant
(X409374FABNGR), a same-tenant provider or headless endpoint with no pin for
the service the consumer reaches (X409375FABUNP).
The node plane
Section titled “The node plane”A second owned table, inet shc_node_plane, hooks forward and input at -10.
It is inet because the listeners bind every family: its port drops carry no
address and hold for IPv6, and a peer’s IPv6 address is closed like its IPv4
one.
- forward: a packet that entered on any bridge and is bound for the mesh prefix is dropped; one bound for a peer’s underlay or public address is dropped on a control or API port and passes on any other (a published port on a peer is a legitimate path).
- input: rqlite HTTP (4001), raft (4002) and the internal API (4003) are accepted from mesh addresses arriving on the mesh device; from any bridge and from the fabric device they and the public API (8000) are dropped. A container may ask shcd for cluster DNS (53, udp and tcp) on the mesh address; anything else from a bridge to the mesh address is dropped.
- the public API opens from a bridge only for broker sources: addresses the
system ingress provably owns (one IPv4 address, one pin anywhere, inside the
node’s grant and the system bridge, in no other workload’s source space;
the pin ledger does not enforce a unique IP). On a node with a bridge per
tenant the accept is bound to the system tier’s bridge, because a locally
delivered packet never meets the policy table’s source chain. The builder
derives exactly one broker use today: the system ingress, because Traefik
routes the shc-api to
http://shc:8000and forward-auth calls the gate there. That use goes when forward-auth decides from files (SHC-1163) and the API route leaves Traefik. The ruling expected this set to be empty; it is not, yet, and closing 8000 to ingress now would take every gated app down.
The table is built from the node rows (mesh IP, internal, underlay and public addresses) and applied before the policy table, so it holds while a refused input keeps the policy at its last version. It only drops what containers send; a non-container source on the underlay is the host firewall’s business.
Decision: a fabric bridge per tenant per node (adopted)
Section titled “Decision: a fabric bridge per tenant per node (adopted)”Ruled 2026-10-01 (address plan row): fabric in 10.128.0.0/10, a bridge per
tenant per node, the per-node grant a genesis knob defaulting to /20, new
clusters only. This note records why it holds up against the alternatives and
what it costs.
With one shared fabric bridge per node every tenant’s pins sit on one L2
segment and the same-node tenant line is address-keyed. Two things follow and
both were measured: a forged source is only stopped by the bridge-family
anti-spoof guard, and on podman nothing drops a scope-bridge container
reaching into a co-located pin (netavark ships no inter-network isolation).
With a bridge per tenant the line becomes a drop keyed on the interface a
packet entered on (fabric_bridges): entry into a tenant’s bridge is allowed
only from that bridge and from the system tier’s bridge. A forged source then
reaches the forger’s own tenant at most, CAP_NET_ADMIN included, and the
podman inbound half closes for every scope bridge.
Weighed against:
- The address plan. A node grant must hold tenants times pins. A /20 grant carved into /25 tenant bridges is 32 slots of 125 pins (slot 0 holds the fabric device address and carries no tenant, so 31 tenants), and the /10 holds 1,024 such nodes. The grant size and the slot size stay parameters (the compiler takes any grant and any bridge inside it); the plan’s default is the ruling’s, not this note’s.
- docker and podman pools. The tenant bridges are created with an
explicit subnet, so neither engine picks their space, but either engine can
still hand fabric space to a deployment-local network from its own pool.
docker’s stock pools (172.17-31/16, then 192.168/16) stay clear. podman’s do
not: the stock
default_subnet_poolsend with10.128.0.0/9, which covers the whole fabric and the control-plane quarter (measured on Debian 13, podman 5.4.2). A new-line node must set both engines’ pools to the plan’s deployment-local quarter,10.64.0.0/10, before the first deployment; the docker address-pool runbook’s 10.128/9 candidate is superseded by the plan. - Incus. Incus builds a bridge per project anyway; a tenant maps to a project and its bridge is the tenant bridge. Nothing in the compiler assumes an engine: it names interfaces, and the executor owns how they are made.
- Uplinks that are bridges. The bridge line drops entry from any bridge
that is not the tenant’s own or the system tier’s. A node whose uplink is
itself a Linux bridge (a bare Incus host with
br0, a bridge over a VLAN that a VM shares) names it to the compiler: the policy pass reads the node plane’s own choice of uplinks, keeps the bridges and sets them on itspolicy.Node.BridgeUplinks. The bridge line and the source chain’s reverse-path drop both spare those names, and the verifier re-derives both with them. Until that was built such a node dropped its published ports on docker and podman, and a client routed in from another network met the reverse-path drop (SHC-3022). A node with a plain uplink names none and compiles what it did.
What this change builds and what it leaves
Section titled “What this change builds and what it leaves”Built and merged with this note:
- the compiler side:
policy.Node.Bridges, thefabric_bridgeschain with a DNAT-reply head, thefabric_sourceschain jumped first in forward (a packet leaving a tenant’s bridge with a source outside its subnet drops, and so does one wearing an address of the node’s grant from any other interface, so a forged source stays in the forger’s tenant on this node and across the fabric), validation (one bridge per tenant, inside the grant, disjoint, never covering the device address, every pin and source of a tenant inside its bridge), goldens for the plan, and the verifier re-deriving both chains from the node’s bridges rule for rule; - the builder seam:
PolicyReaders.Bridges, the read of the tenant-bridge ledger (cemodules/fabric/policyinput.go); - the kernel proof on docker and netavark shapes, both directions, five
forgeries included, across the fabric and into the system tier’s bridge
(
tenantbridge_kernel_linux_test.go).
Built since by the allocator lane (SHC-1282, new clusters only, the rig’s
10.84.0.0/16 is not renumbered): the grant-size genesis knob
(shc cluster init --fabric-grant), the tenant-bridge ledger per node (ce
modules/fabric/tenantbridge.go), one engine network per tenant, the render
joining a deployment to its tenant’s network (BridgeNetworkName), and pins
written live only inside a slot their tenant holds.
Also left, each its own follow-up:
- TCP entrypoint routes (
tcp_ingress, GitLab’s SSH) carry no ingress grant: the builder readshost_mappingonly, as next did before this change, so a tenant’s TCP route through Traefik drops on the fabric until the armed bindings feedIngressBackends; - inside
master/defaultthe system deployments Traefik fronts (keycloak, openobserve) still share its discovery network with each other, wider than their declared ports; it goes when system routes dial fabric upstreams too.
What is still not a boundary
Section titled “What is still not a boundary”- the kernel and the engine, shared by every tenant on a node;
- CAP_NET_ADMIN on a shared-bridge node (on a per-tenant-bridge node it is confined to the forger’s tenant);
network_mode: hostand macvlan or ipvlan containers, which never enter a bridge, so neither table sees them;- a job container the daemon runs for an integration (the s3 dispatcher’s) still joins the provider’s network outside the render;
- IPv6 on a deployment’s own network: the policy table is ip-family and the fabric is IPv4; the guard keeps IPv6 off the fabric bridges and a tenant’s IPv6 network is its own deployment’s;
- multicast and broadcast discovery over the fabric bridge, inside one deployment included: the guard drops it, and it never crossed nodes;
- a non-container source on the underlay reaching the control ports: the node plane only polices containers, and the internal listener moves to the mesh address once enrol leaves it (SHC-1207);
- a co-located consumer reaching another tenant’s published port through in-kernel hairpin NAT is dropped by the bridge line, while the same request from another node passes; the userland proxy hides the difference on docker;
- the labs model the engines’ rule layouts in user namespaces; a run against a live node with the fabric up is still owed.
Magic providers dialled internally (ruled 2026-10-02)
Section titled “Magic providers dialled internally (ruled 2026-10-02)”The keycloakAdmin socket hands consumers http://default.keycloak.keycloak:8080
and the OIDC socket’s hostname supports backchannel: internal. Those are
internal, cross-tenant dials. Clay ruled that the only traffic between tenants
is to the master tenant’s default-environment system services, the platform
Keycloak among them. So the magic allowlist is the fabric’s shared-provider
list: modules/app registers it with the fabric at init, the builder gives every
pin of master/default/<magic stack> the shared-provider role, and every
deployment of every tenant reaches it on its declared ports (ClassSystem) and
nothing else in master. The render bakes that provider’s pin into a consumer in
any tenant, as it does a same-tenant provider’s. The plug refusal is unchanged:
the magic allowlist was already its one legal cross-tenant plug.
Same tenant, same node, one bridge
Section titled “Same tenant, same node, one bridge”Every service gets a pin, a portless one too: the fabric identifies a caller by
its source pin and refuses masqueraded traffic, so a service with no declared
port is still a source (TestFabricTargetPinsAndDefaultRoute). Two deployments
of a tenant on one node share that tenant’s bridge, so their frames are
switched, never routed, and meet the policy table only through bridge
netfilter (br_netfilter, bridge-nf-call-iptables=1). Podman never switches
it on and docker cannot be relied on to, so the fabric reconcile converges it
every tick and warns X200377FABBNF when it cannot. The tenant-bridge lab
measures it on both engine shapes: same deployment open on an undeclared port,
same tenant open on a declared port and dropped on an undeclared one and on a
service that declares nothing, and with the switch at 0 the undeclared port
connects until EnsureBridgeNetfilter brings the drop back.
Bridge netfilter hands the ip hooks untagged IPv4 only, and the policy table
selects the co-located pair by both addresses, so five kinds of frame cross a
bridge without meeting a drop there. The bridge-family guard
(shc_fabric_guard, the anti-spoof table) drops the first four before the
bridge switches them; the fifth never meets a bridge hook and is described
after the list:
- a tagged or otherwise encapsulated frame from a container port (an 802.1Q tag with VID 0 is enough, and the receiver strips it): anything that is not IPv4, IPv6 or ARP;
- IPv6 entering a fabric bridge: a netavark container keeps a link-local address on every interface;
- IPv4 to the limited broadcast address or a multicast group entering a fabric bridge: the bridge floods it to every wildcard UDP socket;
- IPv4 the bridge would switch to an address outside its own subnet: a
container answers for every address it holds, so a frame sent to a
neighbour’s MAC and addressed to the neighbour’s address on its deployment’s
own network arrives there. A frame not addressed to the bridge itself
(
pkttype host, what a container sends its gateway to be routed) must be addressed inside the bridge’s subnet.
The last three are keyed on the bridge (meta ibrname), so a container is
covered from its first frame. A fabric bridge therefore carries ARP and
unicast IPv4 between its own addresses and nothing else, which costs multicast
or broadcast discovery between containers on it, inside one deployment
included. The replies to the last three never came back (the source chain
drops a source that does not belong to the bridge), so each was a one-way
datagram into an undeclared UDP port. The fifth kind is a
frame to a reserved link-local MAC (LLDP’s), which the bridge hands up the
stack on the container’s own port with no bridge hook at all: the policy
table’s source chain and the node plane drop any packet that arrives on a veth
with a source that does not route back through it, which on an enslaved port
is every packet. The kernel’s own rp_filter already refuses that frame in
any mode but off; the rule is what holds on a host that runs with it off. The
tenant-bridge and node-plane labs measure all five with raw frames, each
arriving without the rule and counted by it.
Two more things follow from how the kernel treats a packet rather than from
the bridge. Conntrack relates an ICMP error to a flow by the packet the error
quotes, whoever sends it, so no conntrack head accepts related: a head
admits the reply direction of an established flow, and an error is accepted
by its own rule only from the pin the flow is addressed to (matched as the
packet’s source and as the flow’s original destination) or, arriving over the
fabric device, from a peer node’s device address. The node plane accepts none.
What this costs: an error raised by anything else between two deployments is
dropped, which today is nothing the fabric produces. And every chain reads a
destination at the forward hook, which is where the packet is going only
while the kernel refuses IPv4 source-route options, so the fabric reconcile
keeps net.ipv4.conf.all.accept_source_route at 0 (the kernel default) and
warns X200378FABSRT when it cannot.