Local dev servers
The console, marketplace, zkpoker, storybook and the daemon they talk to, run on a workstation and reachable from other machines on the private network. Audience: whoever is trying to test the UI and finding nothing listening.
Short version: they are launchd jobs now. Start them with
scripts/devservers.sh install # first time, or after changing the scriptscripts/devservers.sh status # is it up, on what bind, answering whatWhat was actually wrong
Section titled “What was actually wrong”Three separate faults, diagnosed 2026-08-13. Only the first one matches the “it was serving twenty minutes ago and now there is no listener at all” symptom; the other two were found while fixing it.
1. Every dev server was a child of an editor session
Section titled “1. Every dev server was a child of an editor session”Nobody started these from a terminal they owned. They were started from AI agent tool calls, which gives every one of them this parent chain:
one dev --port 4600 └─ pnpm --filter @app/console exec one dev --port 4600 └─ /bin/zsh -c ... (the agent harness shell) └─ Cursor Helper (Plugin): extension-host └─ Cursor.appWhen the agent turn ends, the harness reaps that zsh and the whole process group goes with it. This is why the failure had no fingerprints:
- no crash report:
~/Library/DiagnosticReportshad nothing for them; - no error in the app log:
/tmp/zkpoker-web.logends mid-SSR-render on an ordinary line and simply stops; - no OOM: the kernel log had zero jetsam events and the box was at 97% free memory;
- not a crash loop: a crash loop keeps a listener most of the time. This left no listener at all.
It reproduced while being investigated: the console listening on 4600 as pid 54851 at 17:23 was gone by 17:27, replaced by pid 7535 from a different agent’s shell. Restarting by hand loses this race every time, which is why “restart it and confirm it works” was never a fix.
A second-order version of the same problem: several agent lanes each started their own copy of the same app on the same port, so they took turns evicting each other.
2. A hand-rolled supervisor was actively killing the daemon
Section titled “2. A hand-rolled supervisor was actively killing the daemon”An earlier lane left this running:
while :; do SHC_MODULES=... make dev/run >> /tmp/shc-daemon-supervised.log 2>&1 sleep 3; donemake dev/run begins with make stop, and make stop does:
kill $(readlink .shc/run/shc.pid)rm -f .shc/run/shc.pid .shc/run/shc.sockSo every three seconds that loop deleted the running daemon’s unix socket and
then failed to start its own replacement with bind: address already in use.
The daemon reported the damage honestly and nobody was reading it:
X503048USKVNS unix_socket filesystem entry vanished, rebinding, every 30
seconds for hours. Downstream, every /api call through the dev proxy
returned 502 because the socket path had no filesystem entry to connect to.
The supervised daemon now keeps its pid and socket in
~/.local/state/shc-dev/, outside the worktree, so make stop from any lane,
test harness or editor task is a no-op against it rather than a kill.
3. Port 8000 was not the daemon: that is the 502
Section titled “3. Port 8000 was not the daemon: that is the 502”shcd was healthy the whole time. It was not what answered.
A frappe bench belonging to a different checkout
(frappe/multiplatform.one, started 07:33, orphaned to pid 1) held IPv4
*:8000. shcd came up later, and its bare :8000 listen produced a
dual-stack IPv6 socket that coexists with the IPv4 one, but on BSD the more
specific IPv4 socket wins every IPv4 connection. The result:
| Request | Reached |
|---|---|
the host’s LAN address on :8000 (IPv4) | the frappe bench |
http://127.0.0.1:8000/ (IPv4) | the frappe bench |
http://[::1]:8000/ (IPv6) | shcd, answering 401 X401900NOAUTH correctly |
That bench reverse-proxies to a One dev server on :3456, which was itself an
agent-session orphan and had died. A proxy with a dead upstream returns
502, so curling the host’s LAN address on :8000 returned 502 while
shcd was perfectly fine one address family away.
The bench was stopped (it had no clients and no supervisor), and shcd now
binds an explicit SHC_SERVER_TCP_BIND=0.0.0.0:8000 rather than a bare
:8000, so a future squatter collides loudly at startup instead of silently
shadowing the daemon on one address family.
A 401 from :8000 is the healthy answer. The public TCP listener requires
a Keycloak bearer. Use :8003 for an authorized API in dev (below).
Newly visible: the console runs out of heap
Section titled “Newly visible: the console runs out of heap”With real log files in place, a fourth failure showed up immediately: the
console dev server died of FATAL ERROR: Ineffective mark-compacts near heap limit after about seven minutes of SSR churn. V8 caps old-space at ~4 GB
regardless of machine size; this workstation has 512 GB. The node services now
run with --max-old-space-size=8192. Unlike the faults above, this one was
always a genuine crash; it was just invisible, because nothing was capturing
stderr.
How they run now
Section titled “How they run now”Each service is a per-user LaunchAgent, com.clayrisser.shcdev.<service>,
written by scripts/devservers.sh into ~/Library/LaunchAgents/.
launchd is the point. It is the only parent on macOS that outlives every
editor, terminal and session, and with KeepAlive it restarts a job within
seconds whether it crashed, was kill -9’d, or lost a race with another lane.
RunAtLoad brings everything back at login after a reboot. nohup/setsid
would have survived the session but not a reboot, and would not restart on
exit, and macOS ships no setsid anyway.
| Service | Port | Bind | What it is |
|---|---|---|---|
shcd | 8000 | 0.0.0.0 | the daemon; 401 unauthenticated is correct |
uds-proxy | 8003 | 127.0.0.1 | unix socket bridged to TCP as local admin |
console | 4600 | 0.0.0.0 | apps/console, proxies /api → :8003 |
marketplace | 4700 | 0.0.0.0 | apps/marketplace |
zkpoker | 4456 | 0.0.0.0 | crypto-outlaws/zkpoker-ui/apps/zkpoker |
storybook | 6006 | 0.0.0.0 | frappe/multiplatform.one/apps/storybook |
Commands
Section titled “Commands”scripts/devservers.sh install # write agents, reap strays, start allscripts/devservers.sh install console # one servicescripts/devservers.sh status # listener + HTTP table, checked on the host's real IPscripts/devservers.sh restart consolescripts/devservers.sh stop consolescripts/devservers.sh start consolescripts/devservers.sh logs console # tail -fscripts/devservers.sh uninstall # remove all agentsstatus deliberately probes this host’s LAN address, not localhost. A
loopback bind passes a localhost check and still fails the person testing from
another machine, which is exactly how the :8003 bind bug survived.
stop is a launchctl bootout, not launchctl stop; with KeepAlive a
plain stop is undone immediately.
~/Library/Logs/shc-dev/<service>.logstdout and stderr both land there. Before this, the dev servers wrote to
scattered /tmp/*.log files or to a terminal buffer that vanished with the
session, and the console’s OOM crash had never once been seen by anyone.
Bind addresses, and what is not safe to widen
Section titled “Bind addresses, and what is not safe to widen”Every HTTP service here except uds-proxy binds 0.0.0.0. This box sits on
a private /20 and is driven from other machines on it, so a loopback-only
bind on those services means they are not testable at all. That is a
deliberate and appropriate choice for a dev workstation on a trusted
private network.
uds-proxy is the exception. The generator defaults it to 127.0.0.1
(SHC_DEV_UDS_PROXY_BIND) and ignores $BIND_HOST, because the unix
socket behind it authenticates every caller as the synthetic local admin.
uds-proxyon 8003 has no authentication. It bridges the daemon’s unix socket. A wide bind hands platform-admin to anything that can reach the port, with no token and no login. Loopback is the default so a host on the /20 cannot get there. Do not copy this service onto a shared, routed, or internet-facing listener.
The console and marketplace dev servers proxy /api to the daemon from
the server side (apps/console/vite.config.ts), so a browser on another
machine reaches the API through :4600 without ever touching :8003. That
is the supported remote path. To widen the proxy anyway (you almost never
want this):
SHC_DEV_UDS_PROXY_BIND=0.0.0.0 scripts/devservers.sh install uds-proxySHC_DEV_BIND_HOST still narrows every other service at once. It does not
move uds-proxy.
What is not supervised
Section titled “What is not supervised”zkpoker talks to backends that are still started by hand, and it will render but misbehave if they are down:
| Port | What | Where |
|---|---|---|
| 8001 | frappe bench, site zkpoker.localhost | zkpoker-ui/apps/frappe |
| 9001 | socket.io realtime | screen -r zkpoker-socketio |
| 8280 | keycloak (realm zkpoker) | docker |
| 8545 | hardhat localnet | screen -r zkpoker-hardhat |
Gotchas
Section titled “Gotchas”-
make dev/runandmake stopstill exist and still fight you. They target.shc/run/shc.pid, which the supervised daemon no longer writes, so they can no longer kill it, butmake dev/runwill fail withbind: address already in usewhile the supervised daemon holds 8000. Usescripts/devservers.sh restart shcdinstead, orstop shcdfirst if you genuinely want a foreground daemon. -
The
shcCLI looks for the socket in the default location. The supervised daemon’s socket is out of the worktree, so point the CLI at it:Terminal window SHC_SOCKET_PATH=~/.local/state/shc-dev/shc.sock bin/shc lsWithout it the CLI reports
X503931DMNDWN the shc daemon is not reachable on this node, with a hint aboutsystemctlthat does not apply on macOS. -
A broad
pkill -f shcdfrom another lane still reaches this daemon, and there are several othershcdbinaries running on this box from other worktrees, so that pattern gets used. It is no longer fatal: launchd brings the daemon back within seconds andruns = Ninlaunchctl print gui/$UID/com.clayrisser.shcdev.shcdtells you it happened. -
Do not start these from an agent shell or an editor terminal again. That is the whole bug. If a service needs different flags, change
scripts/devservers.shand re-runinstall, so the change is versioned and the process still belongs to launchd.