Skip to content
SHC Docs

SHC Engineering Standards

This document defines the quality bar for the Self-Hosted Cloud (SHC) project. The long-term ambition is to be to self-hosted infrastructure what Linux is to operating systems: a foundation others can build on for decades. That ambition sets the bar.

These are the rules. They apply to the maintainer first, then to every contributor.


The Go rewrite addressed the issues identified in the original Python codebase audit. The standards below now apply to the Go implementation.


The CLI, HTTP API, config schema, and on-disk layout change when the design changes. The new line carries no backwards compatibility.

  • Every public surface gets a version.
  • No aliases, no compatibility shims, no legacy read paths and no deprecation cycle. The old shape is deleted, not kept beside the new one.
  • The one thing carried is the in-place upgrade from the previous release, with no manual step.
  • Broad recover() is forbidden except at well-defined top-level boundaries (CLI entry, HTTP handler, background task runner, RPC transports, callback dispatch loops). At every such site the recover must:
    1. Use a // Broad recover: … comment stating why the breadth is correct.
    2. Either re-panic, surface a typed error to the caller, or log at WARN or above with a typed error code (see 2.16.1).
  • Handle the specific errors you intend to handle. Everything else propagates.
  • Never return nil as a stand-in for failure without a typed sentinel and a documented contract.
  • Background jobs must record structured failure state. “Unknown failed unknown” is a bug.
  • defer func() { recover() }() without logging is specifically forbidden. If you don’t know what operation was being attempted, you cannot tell the operator reading the log either.
  • If it is not imported, it is deleted.
  • If it is legacy, it is deleted, not kept behind a shim or in a parallel directory.
  • shc-old/ must be removed before the next release.
  • No commented-out code. Git remembers.
  • Functions: target under 50 lines, hard ceiling at 100 except in generated code or parsers.
  • Files: target under 1000 lines, hard ceiling at 2000.
  • Classes: one responsibility. If you need “and” to describe what a class does, split it.
  • Nesting: four levels is a smell, five is a bug.
  • Integration tests exercise real Docker, real filesystems, real networks.
  • Mocks are allowed only at true external boundaries (paid APIs, nondeterministic third parties).
  • Every bug fix ships with a test that would have caught it.
  • “The test passes” is not evidence the feature works; evidence is running the feature.

2.5.1 Real-unit tests with in-memory SQLite

Section titled “2.5.1 Real-unit tests with in-memory SQLite”

For unit tests that exercise SQL (service methods, CRUD, graph traversal, DB query predicates) use an in-memory SQLite instance with the relevant schema. Each test file typically defines a newTestDB(t *testing.T) *sql.DB helper that creates the database, applies the necessary schema DDL, and registers cleanup via t.Cleanup.

func TestSkipIfPendingTaskExists(t *testing.T) {
db := newTestDB(t)
_, err := db.Exec(`INSERT INTO task (name, trigger, status) VALUES (?, ?, ?)`,
"my_cron_fn", "cron", "pending")
require.NoError(t, err)
// ...run production query...
require.NotNil(t, existing)
}

When to use real DB vs. mocks:

  • Use real DB when the test’s correctness depends on the SQL doing the right thing: a WHERE clause filter, an ORDER BY, a UNIQUE constraint, a DELETE/UPDATE affecting specific rows, tenant/env isolation, cycles in a graph, foreign-key cascades.
  • Keep mocking at external-system boundaries: Docker/subprocess, restic binary, HTTP webhooks, RPC proxies, external KV providers.
  • Don’t test mocks. Testing that a mock returns what you told it to is not a test of production code.

Seed helpers: tests that need related rows (e.g., DeploymentModel needs NodeModel FK) should factor seeding into a private helper inside the test file.

Performance: an in-memory SQLite with the SHC schema costs ~5 ms per test. A test file with 20-30 tests typically runs in under a second.

  • Get the types right first. If the types are wrong, the code is wrong.
  • Prefer plain data (Go structs, typed constants) over interfaces with methods that just shuffle state.
  • One source of truth per piece of data. Duplicated state is a bug waiting to happen.
  • Abstract only when you have two concrete call sites that demand it.
  • Interfaces with one implementation are deleted.
  • Plugin systems used once are inlined.
  • “We might want to swap this out later” is not a reason to build a proxy today.
  • golangci-lint run must pass with no per-file ignores. If a rule is wrong for the project, disable it globally in .golangci.yml and explain why.
  • gofmt -d must produce zero diffs.
  • Formatting is automated. Style arguments are not allowed.
  • Names explain what. Comments explain why.
  • If a comment restates the code, delete it.
  • Non-obvious constraints, invariants, workarounds, and references to specific bugs or RFCs are the only reasons to comment.
  • No build artifacts, debug dumps, or scratch notes in version control.
  • fix.md, report.txt, shc.db move out or get gitignored.
  • Source tree reflects the project, not its history.
  • “No, rewrite it” is a valid and frequent answer.
  • Praise and harshness are both calibrated to the code, not the contributor.
  • The code outlives every contributor. Protect it.

SHC runs workloads, holds secrets, and exposes network surface. Security is not a feature; it’s a precondition.

  • Threat model is documented. Every subsystem with a network or filesystem boundary declares what it trusts and what it doesn’t. If the threat model isn’t written down, the code hasn’t been designed.
  • Secrets never touch disk unencrypted. The vault is the only place secrets live. No .env files in repos, no plaintext in logs, no secrets in error messages.
  • Rotation is routine, not exceptional. Every credential has a rotation path. If a credential cannot be rotated without downtime, that’s a bug.
  • Principle of least privilege. Containers run non-root by default. Volumes get the tightest permissions that work. Capabilities are dropped unless explicitly needed.
  • Dependencies are audited. go.sum is committed. Known-vulnerable versions fail CI (govulncheck).
  • Security fixes jump the queue. A CVE in a dependency or a reported vulnerability is a drop-everything event.
  • Semantic versioning, strictly. MAJOR breaks contracts, MINOR adds, PATCH fixes. No “we’ll bump major when it feels right.”
  • v0.x is the prototype phase. During v0, contracts can still change but require a migration note. After v1.0, breaking changes require a full deprecation cycle.
  • LTS branches live for two years minimum. When you cut an LTS, you commit to patching it. If you can’t commit, don’t cut it.
  • Release notes are written by the maintainer, not generated. Every release explains what changed, what broke, and what to do about it.
  • “Stable” means stable. A release tagged stable has been exercised in production, not just tested in CI.
  • Every public CLI command, HTTP endpoint, config key, and extension point is documented before it merges.
  • Documentation lives in the repo, next to the code. External wikis drift and die.
  • Examples in documentation are tested. Broken examples are worse than no examples.
  • A feature without documentation is a feature that doesn’t exist, and cannot be merged.
  • One name per thing. The CLI, the docs and the console use one word for a concept. When the daemon’s word is wrong for a user, the console translates it once, in one place, and every surface reads that (was UX-CP-07 and CX-W-07).
  • Mockups mark what doesn’t ship. Every mockup, spec rendering, prototype and design review carries a visible PROPOSED marker on each element that isn’t implemented, as a SHIPPED / PARTIAL / PROPOSED chip per claim (was UX-CP-10). A rendering’s own build constraints (single-file HTML, no JavaScript, one root <section>) are artifact constraints and never bind the product (was UX-SC-06).

Infrastructure software that gets slow gets replaced. Performance is a feature.

  • Startup: shc cold-start under 500 ms for list/status commands. Measured in CI.
  • Memory: core daemon under 200 MB resident at idle.
  • Backup throughput: snapshot and restore track disk speed within 20%. Regressions fail the build.
  • API latency: p50 under 50 ms, p99 under 500 ms for read endpoints.
  • Console: on a Pixel 6a or a desktop browser with a 4x CPU throttle, scrolling the springboard, the events feed and any list holds 60 fps with no frame over 16 ms. On Slow 4G (1.6 Mbps down, 150 ms round trip) the first interaction works inside 2 s, and each route ships its own chunk (was CX-P-01 and CX-P-09).
  • Every subsystem declares its budget. If a subsystem has no budget, it has no contract and can’t be depended on.

Numbers above are starting targets; refine them as the project matures, but once set, regressions block merges.

Every subsystem emits the same shape of signal. Operators should not need to learn a new telemetry dialect per module.

  • Structured logs only. JSON or key-value. No printf debugging in main branch.
  • Every log at WARN or above carries a typed error code (the existing X200924SYSINS-style codes). Free-text-only warnings are rejected.
  • Metrics for every boundary: request counts, latency histograms, error rates. Exposed in a standard format (Prometheus-compatible).
  • Traces for every cross-subsystem call. Context propagates through background jobs, not just HTTP requests.
  • Log levels mean what they say. ERROR means a human needs to look. WARN means something unusual happened. Nothing routine is WARN or above.

2.16.1 Logger API: the only shapes allowed

Section titled “2.16.1 Logger API: the only shapes allowed”

SHC logs through the typed core/log (corelog) wrapper; raw slog.Warn / slog.Error are lint-banned in production paths. The standard shapes are:

logger.Debug("free-text message", "key", value) // developer diagnostics (slog-style OK below WARN)
logger.Info("free-text message", "key", value) // routine operator signal
corelog.Warn(ctx, X200003OPFAIL(op, output, err)) // something unusual happened: typed coded error
corelog.Error(ctx, X500907STKFAL(op, err), err) // a human needs to look: coded error + cause

Guidelines:

  • WARN and above take a *coded.CodedError, never a free-text string; the human message lives in the factory’s registered template, and a string argument is a compile error
  • ERROR means a human needs to investigate
  • WARN means something unusual happened but was handled
  • Generic “suppressed exception” messages tell the reader nothing; name the operation that failed
  • Every WARN/ERROR therefore carries an X-code from the module’s errors.go by construction

A platform not in the matrix is not supported. A platform in the matrix is tested in CI.

  • Host OS: Debian stable, Ubuntu LTS. Others are community best-effort.
  • Container runtime: docker (Docker), podman, incus, and apple.
  • Go: the version pinned in go.mod. One version at a time.
  • Architecture: amd64 first-class, arm64 supported, others opportunistic.
  • Adding a platform is a commitment. Dropping a platform requires a deprecation cycle.

Who decides what. Explicit, not implicit.

  • Benevolent dictator model. The maintainer has final say on what merges, what ships, and what gets rejected. This is the model until the project outgrows it.
  • Disputes resolve in public. Technical disagreements are hashed out on the issue tracker or mailing list, not in private. The decision and its reasoning are recorded.
  • Succession plan. If the maintainer is unavailable for more than 30 days, a named deputy has merge authority for security and critical fixes only. Feature work waits.
  • Bus factor is tracked. Every subsystem has at least one other person who has read it and could patch it in an emergency. If only one person understands a subsystem, that’s a risk logged as a bug.
  • The model can change. When the project outgrows benevolent dictatorship, the replacement model is proposed, debated in public, and documented here.

Mechanics of getting code in.

  • One logical change per patch. Commits are atomic. “Fix X and also refactor Y” is two commits.
  • Commit messages follow the kernel style. Subject line imperative, under 72 chars. Body explains why, not what. Reference the issue, bug, or RFC that motivates the change.
  • Developer Certificate of Origin. Every commit is signed off (Signed-off-by:). No CLA required, but provenance must be clear.
  • Reviews are technical, not social. Reviewers comment on the code. Contributors respond to the comments. Nobody merges their own PR without an approving review, including the maintainer.
  • CI is green before review starts. Don’t waste reviewer time on broken patches.

Standard 2.1 carries one thing across a release: the in-place upgrade from the previous release. This section says how we prove it.

  • Migration paths are tested, not inferred. Upgrading from vN to vN+1 is a CI job, not a hope.
  • A release reads the previous release’s data. The upgrade needs no manual step. Nothing older is carried.

SHC moves data. Losing user data is the worst thing this project can do. Given the backup history, this deserves its own wall.

  • Destructive operations require explicit confirmation. Restore, wipe, volume delete, stack destroy: all prompt for confirmation by default. Non-interactive mode requires an explicit flag (--yes-really).
  • Dry-run is a first-class mode. Every destructive command supports --dry-run and prints exactly what it would do.
  • Audit log for destructive ops. Who ran what, when, with what arguments. Persisted outside the data the operation might destroy.
  • Backups are verified. A backup that cannot be restored is not a backup. The backup subsystem periodically runs restore-to-scratch to prove the archive is readable.
  • Irreversible ops have a cooling-off period where possible. Deleted volumes move to a trash namespace for 24 hours before actual removal.
  • Tests for data-loss paths are mandatory. Every code path that can delete or overwrite user data ships with a test that proves the guard holds.

Bugs ship. What happens next is what separates infrastructure software from hobby projects.

  • Severity classification. SEV1: data loss or security breach in the wild. SEV2: outage or unrecoverable failure. SEV3: degraded behavior with a workaround. SEV4: cosmetic.
  • SEV1 drops everything. The maintainer and all active contributors stop feature work until the fix is shipped and users are notified.
  • Postmortem within one week of every SEV1 or SEV2. Public, blameless, and linked from the release notes. Template: what happened, timeline, root cause, contributing factors, what prevented detection, what changes land as a result.
  • CVE process. Security bugs get a CVE if they qualify. Disclosure timeline is coordinated with reporters and follows a standard 90-day window unless actively exploited.
  • User notification. Breaking incidents are announced on a known channel (release notes, mailing list, security advisory feed). Silence is not an option.
  • Every postmortem produces at least one test or rule change. If a bug could happen twice, the postmortem didn’t finish.

Projects die from accretion, not from malice. Things get added and never removed. This section defines when something leaves.

  • Subsystems: a subsystem with no active maintainer for two releases, no users (measurable by telemetry or explicit removal of the feature flag), or a critical unfixed bug for more than 90 days is a candidate for deprecation.
  • Platforms: a platform whose CI fails repeatedly without a fix, or whose user base drops below a meaningful threshold, moves to best-effort and then to unsupported on a published schedule.
  • Features: a feature that can be replaced by a simpler one is deprecated, not left parallel. Two ways to do the same thing is one too many.
  • Dependencies: a dependency that is unmaintained upstream for more than a year gets replaced or vendored with explicit ownership.
  • Deprecation cycle is public and dated. “Deprecated in v1.4, removed in v1.6.” Not “deprecated eventually.”
  • Saying no is the maintainer’s most important skill. Features that don’t fit the project’s scope are rejected early, before they accumulate investment.

Order of operations. Do these before adding new features.

  1. Clean up legacy code. Remove any deprecated or orphaned code paths. One commit. Today.
  2. Error handling audit. Replace every broad recover() with specific error handling or let it propagate. Add top-level handlers only where needed.
  3. Define the public API. Pin the CLI surface, HTTP routes, and config schema. Write the compatibility test suite. Tag a v0 that commits to them.
  4. Split the god modules. app/service.go and backup/service.go along existing seams (backup: snapshot vs restore).
  5. Replace the top 10 most-mocked tests with real integration tests. Real Docker, real volumes, real network.
  6. Turn on golangci-lint strict mode and clean all exemptions in CI. Fail the build on violations.
  7. Write CONTRIBUTING.md. Point it here. Enforce it on yourself first.
  8. Document the threat model and rotation paths for every subsystem that holds credentials or exposes network surface.
  9. Write the observability contract and retrofit existing subsystems to emit structured logs, metrics, and traces consistently.
  10. Set and publish performance budgets for each subsystem. Wire regression checks into CI.
  11. Publish the governance model and succession plan. Name the deputy. Log the bus-factor-one subsystems as bugs.
  12. Adopt the contribution workflow. Enable DCO sign-off enforcement. Document the commit message rules. Require reviews even for maintainer PRs.
  13. Build the backwards-compatibility test harness. Snapshot the first tagged release’s public surface. Run it against every subsequent release in CI.
  14. Harden every destructive code path. Add --dry-run, confirmation prompts, and audit logging. Write the data-loss-path tests.
  15. Write the incident response playbook. Define severity levels. Create the postmortem template. Set up the security advisory channel.
  16. Draft the deprecation policy. Pick the first subsystem, platform, or feature to deprecate as a test run of the process.

Do these and SHC is at the starting line of Linux-grade discipline. Not there, at the starting line.


The maintainer holds this bar for themselves before holding anyone else to it. Every PR the maintainer writes gets reviewed against this document. Every PR the maintainer merges gets reviewed against this document. No exceptions for convenience, deadlines, or mood.

If the maintainer cannot hold the bar, the bar is wrong and the document gets updated, not quietly ignored.


5. Architecture: the canonical patterns and how they are enforced

Section titled “5. Architecture: the canonical patterns and how they are enforced”

A codebase grows spaghetti when the same concept gets implemented many ways because no contributor (human or agent) holds the whole model: architectural drift. SHC fights it with three things: one canonical pattern per concern, written down; CI checks that fail the build on regression; and a visible backlog of the deviations not yet normalized.

5.1 Before you implement: reconnaissance first

Section titled “5.1 Before you implement: reconnaissance first”

Before adding a service, a wiring path, an error type, an HTTP client, a scope, or a test fixture: find the existing pattern and reuse it. Do not invent a parallel one. If you genuinely need a new pattern, you change the canonical doc + the fitness check in the same change and say why. “I didn’t know it existed” is the drift this section exists to prevent.

Each is recorded in docs/decisions/ with context, the decision, and consequences:

  • ADR-0001: Pure DI at a composition root (daemon/composition.go), no container. The four transform classes (W1a/W1b/W1c/W2) are named there; injected resolvers are per-call closures, never wire-time captures. Modules/subsystems must never import daemon.
  • ADR-0002: Stack state: the DB = identity/status authority, manifest.json = offline SPEC, docker = observed STATUS, the in-memory map = the offline and test store, not a cache of the DB. One authority per role.
  • ADR-0003: One data-ladder core/scope.Scope; auth.AuthScope is the separate RBAC marker.
  • ADR-0004: Hermetic default make test/unit (pure Go, no cgo); live-engine rqlited tests skip unless a rqlited binary is resolvable (SHC_RQLITED_BIN / make vendor/rqlite).
  • ADR-0005: the enforcement mechanism itself.

The cross-module wiring pattern is documented in docs/architecture/patterns/cross-module.md; the package DAG in import-dag.md. (Docs that predate the DI/scope/ent refactors and are known-stale are catalogued in the drift register below; trust the register over an unmarked older doc.)

5.3 Enforcement: make lint runs the fitness functions

Section titled “5.3 Enforcement: make lint runs the fitness functions”

scripts/arch-fitness.sh (wired into make lint) fails the build on a regression of a converged pattern. ratchet checks may only decrease (a migration lowers the ceiling, never raises it); hard checks must be zero:

CheckKindForbids
di-singletonsratchet → 0a new SetRuntimeService package global
di-test-fixturesratchet → 0a new X.SetRuntimeService(fake) test swap
uds-dialer-copiesratchet → 1a new inline unix-socket DialContext (use core/daemonclient)
one-scope-typehard-zeroa new type Scope string outside core/scope
no-cgo-sqlitehard-zeroa mattn/go-sqlite3 import
layer-boundaryhard-zeromodules or subsystems importing daemon; ce’s half is check 4b (see below)

The di-singletons / one-scope-type rows above scan shc’s own trees. Their ce-side twins are separate checks over a separate corpus, because the two trees have two different verdicts (shc is at hard zero on all of them; ce carries seven grandfathered runtime bridges):

CheckKindForbids
ce-modules-import-daemonhard-zeroce modules/subsystems importing ce daemon
ce-core-imports-moduleshard-zeroce core importing ce modules
ce-di-singletonshard-zeroa func (Set|Get)RuntimeService package global in ce
ce-di-runtime-familiesratchet → 0 (at 7)a new func (Set|Get)RuntimeX global in ce
ce-one-scope-typehard-zeroa new type Scope string outside ce core/scope

They resolve a ce tree (checkout first, module-cache pin as fallback) and REFUSE rather than pass when they cannot see one. make lint runs them; make ci/checks/ce-smoke delegates to the same implementation with sh scripts/arch-fitness.sh ce. In CI they run in shc’s next/lint job, which runs the same script on every push to dev and every merge request into dev.

To add a pattern: write its rule into arch-fitness.sh, set the baseline, record an ADR, and reference it here.

docs/architecture/REGISTER.md is the prioritized backlog of every code deviation and every stale-doc claim found in the architecture audit, plus the pending fitness rules (greppable checks ready to be promoted into the script as each class is normalized). It is the normalization worklist for the from-scratch convergence.