Private names through the node resolver
shcd answers DNS itself on every node. It listens on the node’s mesh
address (the mesh0 interface, in the mesh range: 10.64.48.0/20 on the
default block, see the address block),
UDP and TCP port 53,
and answers from the node’s clusterfs revision, so it keeps answering from
the last state it saw when the node is cut off. No container is involved:
name lookup comes up with shcd, before anything it installs. It answers
eight kinds of name:
| Name | Answer |
|---|---|
<node>.nodes.shc.internal | the node’s mesh address, to every asker but a tenant’s containers |
<service>.<stack>.<environment>.<tenant>.shc.internal | to a container it is in view of, the service’s VIP while the cluster runs VIPs, else the fabric addresses of its ready replicas; to the node itself, always the replicas’ addresses, because the kernel resets a packet the host sends to a VIP; TTL 5 s |
<service>-r<ordinal>.<stack>.<environment>.<tenant>.shc.internal | one replica’s fabric address, for a replica pinned under its own entry; never a VIP |
a routed hostname, such as git.corp.example.com | the ready nodes in the app’s location, whose Traefiks all carry every route, at the address their published ports take: their underlay addresses, never a mesh address; to a tenant’s container the system Traefik service’s VIP instead, and without one the nodes’ public addresses; TTL 30 s |
a name an app calls an integration provider by, such as prod.postgres.mydb or postgres | the same answer as the provider’s service, to the consumer’s own containers only |
collector.shc.internal | the system collector deployment’s VIP; nothing while it has no replica |
ingress-sources.shc.internal | every node’s ingress address, the one fabric address of each node that only the system Traefik holds, to every asker; NXDOMAIN while no node has one. An app trusts X-Forwarded-For from these |
_replicas.<service> | to a container of the service’s own deployment, the fabric addresses of the service’s ready replicas, never its VIP; also asked as _replicas.<service>.<stack>.<environment>.<tenant>.shc.internal; NXDOMAIN to every other asker; TTL 5 s |
A service’s name also answers SRV, one record per port the service
declares, asked and answered in view like its A record. The bare name gives
the TCP ports; _tcp. or _udp. before it picks the protocol. Each record
names the service itself, with its addresses in the additional section. A
port range has no SRV record:
dig +short SRV postgres.mydb.prod.corp.shc.internal0 0 5432 postgres.mydb.prod.corp.shc.internal.dig +short SRV _udp.statsd.metrics.prod.corp.shc.internal0 0 8125 statsd.metrics.prod.corp.shc.internal.Every other name, including other records under a routed domain, goes to the node’s own upstream nameservers.
Who is answered what
Section titled “Who is answered what”The names under shc.internal are answered by who asks. shcd reads who
that is from the address the question comes from, never from the name
asked.
| Asker | Names under shc.internal it is answered |
|---|---|
| the node itself (its own mesh address, or loopback) | every name |
| a container, known by its fabric address | its own tenant’s services, the platform’s shared providers (Keycloak in master/default), the providers its deployment is plugged into, _replicas.<service> of its own deployment’s services, collector.shc.internal and ingress-sources.shc.internal |
| any other address | the node names, collector.shc.internal and ingress-sources.shc.internal |
A name outside the asker’s view is NXDOMAIN, the same answer a name that
does not exist gets, so a tenant cannot learn another tenant’s
deployments, services or addresses by asking for them. Node names are not
answered to a tenant’s containers. Routed hostnames are answered to
everyone, as they are public names, but not with the same addresses.
mesh0 feeds no published port, so a routed hostname never answers a
mesh address. A tenant’s container is answered the system Traefik
service’s VIP, which it reaches wearing its own fabric address, so a
LAN-only route of its own tenant answers it and another tenant’s does
not (see
published ports).
While the node routes no such VIP it is answered the nodes’ public
addresses only, and nothing where they have none. The node itself, the
system tier’s containers and every other asker are answered the nodes’
underlay addresses.
A container is known only by the fabric address shcd has filed under its
deployment. One that asks from another address is answered as a stranger.
One case asks as the node and gets its view: a service with
network_mode: host. Another node’s address is a stranger’s too.
An install or update on the fabric has the node’s DNS take up its pins
before it starts a container, and fails with X503273RSVNAM when the node
cannot, so a new container is never answered as whoever held its address
before. For the same reason a node whose answers have fallen behind the
control plane for more than five minutes (half the fabric’s pin
quarantine), because its clusterfs cannot build or its DNS cannot load
the revision, answers its containers as strangers until it catches up. A
node cut off from the control plane sees nothing newer and keeps its
answers.
Each node lists what it answers whom in
/var/lib/shc/clusterfs/resolver/current/views.json.
To see what a container is answered, ask from inside it. dig on the
node shows the node’s view, which is everything.
The private zones are shc.internal plus the zones of every private DNS
connection: the ones it declares, and each public zone auto-zone mirrored
into it so a host gets a private record beside its public one. A mirror
lasts as long as a public connection owns its zone. When the public
connection drops the zone, every private plane drops it too, and each node
stops routing it within a minute. Each node lists them in
/var/lib/shc/clusterfs/resolver/current/domains.json.
Restarts, floods and pressure
Section titled “Restarts, floods and pressure”The DNS sockets, like shcd’s unix socket and API listeners, are handed to
systemd’s file descriptor store (FileDescriptorStoreMax= on
shc.service). A restart never closes them: queries that arrive while
shcd restarts wait in the kernel and the next shcd answers them, late but
not refused. The listener binds with free bind, so it holds the mesh
address from boot, before mesh0 brings the link up. A node without
systemd (the Mac) binds directly and retries until the address exists;
X503272RSVBND in the log names an address it could not bind yet.
Each source address may send 2,000 queries a second, with bursts to
4,000. Past that its UDP queries are dropped and its TCP queries go
unanswered, so one noisy container cannot crowd out the others or the
control plane. Queries wait in a bounded queue that reports into the
node’s pressure registry as resolver.dns, like every other queue: depth,
oldest age, rates and refusals, exported as the shc_pressure_* metrics.
A query a quarter of a second old grades the queue amber and one a second
old red; refusals grade it amber.
Check that a node answers:
dig +short @<mesh address> <node>.nodes.shc.internalUntil a node answers, shcd routes nothing to it, and hosts and containers keep their own nameservers.
shcd routes the private zones to its own listener through systemd-resolved, once a minute on every node:
resolvectl dns mesh0 <mesh address>resolvectl domain mesh0 ~shc.internal ~corp.example.comresolvectl default-route mesh0 falseOnly the private zones take that route; every other name keeps the node’s
own nameservers. It is applied again when the mesh link comes back after a
mesh0 restart, which drops those settings. If the listener stops
answering, shcd runs resolvectl revert mesh0 and the node resolves the
private zones through its own nameservers until it answers again.
A node without systemd-resolved is left alone. Install systemd-resolved
on it to have the private zones routed.
Check a node:
resolvectl status mesh0resolvectl query n1.nodes.shc.internalX500271RSVHST in the daemon log means systemd-resolved refused the
change. The node keeps its own nameservers and shcd retries a minute later.
Containers
Section titled “Containers”When an app is installed or updated on a node whose listener answers, each
service that has its own network namespace and declares no dns: gets
dns: set to the node’s mesh address, followed by the node’s own
nameservers as a fallback. Docker’s embedded resolver still answers the
compose network’s own names first and forwards the rest there. A service
with network_mode: or an authored dns: is left as it is. Apps
installed before the listener answered pick it up on their next update.
A container asks its own deployment’s services by the bare service name.
The node’s DNS looks a name with no dot up in the asking container’s
deployment zone first, so db-1 from a replica of pgha answers
db-1.pgha.<environment>.<tenant>.shc.internal, whatever the image’s libc
and ndots. SHC states no ndots, so on docker the embedded resolver keeps
its own ndots:0 and forwards such a name from the container’s own
address. A name with no dot that the deployment has no service of goes on
as asked, to the names its plugs gave it and then to the node’s own
nameservers, so a LAN host such as nas keeps resolving.
It asks them by <environment>.<service>.<stack> too, the name
{{ hosts.<service> }} renders in an app’s templates. From a container of
that environment and stack the node’s DNS answers it exactly as it answers
the bare name, on every runtime, so prod.valkey.wiki from a replica of
wiki in prod is valkey.wiki.prod.<tenant>.shc.internal. Another
deployment’s containers are not answered it unless a plug gave them the
name. On a Docker node the embedded resolver answers it first, with the
replica’s own address on the node’s bridge, when a replica of the service
runs on the same node.
SHC writes no extra_hosts into an app’s compose. The ones an app’s
author wrote stay as written.
An app’s jobs (hooks, socket and integration jobs run in a one-off
container) ask the same DNS. On the SHC fabric each run joins the node’s
fabric network with a pin of its own, filed under the job’s deployment,
with dns: set to the node’s mesh address and the deployment’s zone as its
search domain. The node answers that address what it answers the
deployment’s own containers, including the names its plugs call providers
by. The address is published to the node’s DNS before the container starts,
and goes into quarantine like any released address when the run ends. Two
runs of one job get two addresses. While the node’s DNS is not answering, a
job of a deployment that calls providers by name is refused with
X503275RSVDWN, the way its install would be.
An app plugged into a provider (-i db=mydb) keeps the hostnames its
config has always carried: the socket’s hostname, the
<environment>.<service>.<stack> name and the bare service name. On the
SHC fabric the app is not joined to its provider’s network, so the node’s
DNS answers these names the way it answers the provider’s service name:
with the service’s VIP while the cluster runs VIPs, else with the fabric
addresses of the provider’s ready replicas, whether the provider runs on
the same node or another (the bare name has the limit described below).
Which provider service each name means is written when the app is rendered,
in the deployment’s stored spec, which every node’s DNS reads, so it
follows the app’s -i flags and its plugs’ services: pins. The node
answers the new names before the install or update starts a container, and
keeps answering the names the previous render gave until compose-up has
replaced its containers. A VIP stays the provider’s address while its
service exists, through a move, a restart or a stop, and the kernel takes
each new connection to a ready replica or resets it when there is none.
Without VIPs, a provider that moves or stops changes the next answer within
seconds. Either way the app is not re-rendered. These names carry no
tenant, so the node answers them by who asks: only the consumer’s own
containers, recognised by their fabric addresses, get the provider’s
address. Anyone else asking the same name gets the upstream’s answer. An
app rendered before its node answered these names gets them on its next
update.
While the node’s DNS is not answering, an install or update whose
services call providers by these names is refused with X503275RSVDWN.
Nothing starts and a running app keeps its containers. shc doctor
shows resolver.revision until the DNS answers again; retry then. Every
node answers these names from the deployment’s stored spec, wherever its
replicas run, so a replica that moves to another node keeps them. If the
store cannot be read, the node keeps answering every name from its last
revision until it can.
The bare service name is a known limit on a Docker node. Docker’s embedded
DNS answers a bare name itself, before the node’s DNS sees the question,
when any container on a network the asker shares has that service name, and
every deployment on a node joins the node’s fabric network. A consumer
whose provider is on another node should use the socket’s hostname or the
<environment>.<service>.<stack> name.
Podman nodes
Section titled “Podman nodes”Podman’s own resolver, aardvark-dns, answers a container on any network
that has DNS enabled and forwards every other name to the container’s
dns: servers from the host. The node’s DNS would then see the node’s own
address and answer the node’s view, every tenant’s names.
So a Podman node of the SHC fabric has no aardvark-dns. That is the guarantee, and it does not depend on how a compose document is written or read: with none on the node, nothing forwards a container’s question to the node’s DNS from the node’s address.
- Remove the package before
shc initorshc node join:apt-get remove aardvark-dns(dnf remove aardvark-dns). Both commands refuse a node whosepodman inforeports a DNS backend, and one where an aardvark-dns process still runs, withX400398FABAVK. A process started before the package was removed keeps answering every container on its networks, new ones included, until the last container on those networks stops. Then it exits and nothing starts it again. - Every start asks again: an install, an update, a job,
shc start,shc restart, a rollback, the health reconcile, a replica project and the restart of one container. A package manager that brings the package back as a recommended dependency makes the next start refuse withX400398FABAVK, andshc doctor runreports it asfabric.enginebefore that. - Your own Podman containers on that node, outside SHC, lose aardvark’s names too. The node is SHC’s.
The rest is what gives a container a resolver at all, not the guarantee.
On a network with DNS enabled and no aardvark-dns, a container’s
resolv.conf points at the network’s gateway, where nothing answers. So
the engine’s resolver is off on every network a container joins:
- SHC creates its own networks (the node’s fabric network, the shared
ingress network, a deployment’s networks on the node) with
--disable-dns. - A network podman-compose creates carries
x-podman.disable_dns: true. podman-compose reads that key from 1.4.0. An older one drops it without an error, soshc initandshc node joinrefuse it by name (X400396FABPDC) andshc doctor runreports it asfabric.engine. Debian 13’s podman-compose is 1.3.0. - A start is refused with
X409429PODDNSwhile a network its compose could name still has the resolver on, withX503429PODDNQwhen podman cannot list its networks, and withX400396FABPDCwhen the node’s podman-compose is older than 1.4.0 by then, on the fabric or off it. podman-compose versions name a network differently, so every name it could have is asked about: its key, itsname:, the name underexternal:, and the project’s name joined to the key. - podman-compose 1.4.x keeps the resolver off but ignores
--no-recreate: an update that changes one service creates every container of the deployment again, and an up over containers that exist fails on the first name it finds in use.shc init,shc node joinand every start refuse it withX400397FABPRC, andfabric.enginereports it as an error naming the version found and 1.5.0. Containers that already run are left as they are. - A document rendered before SHC wrote the key is refused with
X409430PODDNR: runshc updatefor the deployment. One that cannot be read is refused withX503430PODDOC. One with a top-levelinclude:is refused withX409431PODINC, because a network an included file creates would not carry the key: inline what it includes. - Nothing turns the resolver off on an existing network. Remove a
deployment’s network once no container names it (
podman network rm) and install again.
Where SHC and podman-compose still read a document differently, the result is a container that cannot resolve names. It is never a container answered another tenant’s names.
With the resolver off, a container’s resolv.conf is the node’s DNS and
the deployment’s search domain, and it asks from its own fabric address,
as on Docker. What this takes away: the names only the engine knows. On a
Podman node a compose container name (wiki_db_1) and an app-authored
network aliases: entry do not resolve. A service name still does, through
the search domain, with the answer the node’s DNS gives everywhere: the
service’s VIP, or its ready replicas. On a Docker node the embedded
resolver still answers container names and aliases, so the two runtimes
differ there until the node’s DNS answers aliases itself.
A single Podman node resolves no service names at all. The node’s DNS listens on the mesh address, and a node gets one when a second node joins. Until then nothing answers on the node, containers keep the host’s nameservers, and with Podman’s resolver off a deployment’s containers cannot find each other by name. A single Docker node is not affected: the embedded resolver answers the compose network’s names.
An app with -i otel=OTEL exports to http://collector.shc.internal:4318.
While the system collector deployment (master/default/collector) has a
ready replica, every node answers that name with the deployment’s VIP, to
every container, whatever the node’s runtime. The deployment runs one
replica on each ready node, so the export goes to one of them, and the fabric
lets every tenant reach its two OTLP ports and nothing else of it.
Until that deployment has a replica, no node answers the name: a node’s own
collector project reads that node’s logs and metrics and takes no OTLP. An
install or update of an otel-integrated deployment then emits
X200182OTLNIL once. The app still runs; its OTLP export has no destination
until shc install collector -t master. See the
telemetry integration
for the system collector and the warning text.
A service that sets its own dns: asks its own nameservers and sees none
of this. Forwarding shc.internal from them to the node’s mesh address
brings back the cluster names. It does not bring back the provider
names: the node answers those by the asker’s fabric address, and a
forwarder asks from its own, so the service has to ask the node’s DNS
directly for them.
NetBird devices
Section titled “NetBird devices”People’s devices reach the resolvers over NetBird. Two objects in the NetBird management console carry it: a network route to the mesh through routing peers, and a nameserver group for the private zones. SHC manages neither, so set them up by hand once per estate and again when you add a location.
-
Routing peers. Run the NetBird agent on at least one node per location and add those peers to a group, for example
shc-routers. Two per location keeps the route up while one reboots. -
Network route. Under Network Routes, add the cluster’s mesh range with the
shc-routersgroup as routing peers, masquerade on, and distribute it to the group your people’s devices are in. Add an access control policy that lets that group reachshc-routers. The mesh range is the address block’s fourth sixteenth:10.64.48.0/20on the default10.64.0.0/16,10.80.48.0/20on a10.80.0.0/16block (see the address block). On any node,ip -4 addr show mesh0prints the node’s mesh address with the range’s prefix length, for example10.64.48.1/20. -
Nameserver group. Under DNS → Nameservers, add a group whose nameservers are the mesh addresses of the routing nodes, port 53, UDP. Set the match domains to every entry in
domains.json, with search domains off, and distribute it to the same people’s group. Every node’s resolver gives the same answer, so any node’s address works. -
Verify from a device. First the route, straight at a routing node’s mesh address. The first node of a cluster on the default block holds
10.192.0.1:Terminal window dig +short @10.192.0.1 n1.nodes.shc.internalThen the nameserver group, through the device’s own resolver.
digwithout@reads/etc/resolv.conf, which a match-domain nameserver never appears in, so ask the system resolver instead:Terminal window dscacheutil -q host -a name git.corp.example.com # macOSresolvectl query git.corp.example.com # LinuxResolve-DnsName git.corp.example.com # WindowsEach should return mesh addresses.
With masquerade on, Traefik sees a device’s requests coming from the routing peer’s mesh address. An app restricted to the LAN admits them only if its allowlist covers that address; see Gated apps. To the resolvers a device is a stranger whichever address its question arrives from: it is answered the routed hostnames and the node names, and no tenant’s services. Ask from the node for those.
A private zone added later, for example a new private DNS connection,
appears in domains.json within a minute. Add it to the nameserver
group’s match domains.