App metrics and the trust score (2026-08-18)
Status: open design. No trust score is implemented, and no metric row is written. This answers the owner’s question of 2026-08-18 in his own framing: “I need you to figure out what all data points we should be collecting”, with OpenRouter’s model pages and Snyk’s npm metrics as the stated reference. It carries the ranked list of data points, what each costs, what is measurable today, what is not, and what is refused.
★ This is the “scoring lane” that scripts/gen-app-capabilities-md.go was
built to feed. Makefile:629 says of that generator: “-format json is the
artifact the scoring lane consumes. This program emits no score, no weight and
no total: the two axes are INPUTS to a model that lives elsewhere.” This
document is that model. §2 is written against its JSON.
Companions:
2026-08-14-marketplace-architecture.md
(the store this renders in) ·
2026-08-16-app-landing-pages.md (the page
these metrics fill) ·
../app-development/app-capabilities.md
and ../app-development/app-features.md
(the two capability axes, shipped 2026-08-18 in !292) ·
../architecture/sso-redirect-coverage.md
and ../architecture/vendor-resource-floors.md
(two fitness registers that already exist and are the model for the rest).
Every count below was measured on this checkout at 7cdaba2da on 2026-08-18.
Denominators are 90 store-facing packages: apps/<bucket>/<app>/, the same
set gen-app-capabilities-md.go counts; the six top-level built-ins
(branding, keycloak, license, openobserve, postgres, traefik) are
excluded, as are the 10 apps/tests/ fixtures. Method is stated per number.
Anything not measured is marked INFERENCE.
0. Three corrections to the working brief, before anything is built on it
Section titled “0. Three corrections to the working brief, before anything is built on it”★ AN INSTALL COUNTER ALREADY EXISTS. The brief that opened this lane said
“NO INSTALL COUNTER EXISTS ANYWHERE”. That is stale.
App Listing.install_cluster_count and install_count_basis are on the doctype
(frappe/apps/marketplace/marketplace/marketplace/doctype/app_listing/app_listing.json:34),
recomputed by an hourly bench sweep (marketplace/hooks.py:15,
marketplace/community/api.py:739), counting DISTINCT REGISTERED CLUSTERS from
two bases: settled Install Dispatch rows and Cluster.app_inventory
heartbeats. A k-anonymity floor of 5 is applied at write time
(community/installs.py:110) because the row is Guest-readable, and the label
is fixed in code as “installs from registered clusters”
(community/installs.py:112). The reader never returns 0
(features/marketplace/src/listing/community.ts:45). It is better
privacy-engineered than anything this document would have proposed; nothing here
reopens it.
★ THE SHC RUNNER IS BACK. The brief said shc CI was dead with ten
consecutive runner_system_failure. Measured against the GitLab API today:
pipeline 2768223652 (schedule, main, 07:07Z) ran four jobs green:
marketplace/seed-dry, apps/verify-pins, catalog/freshness, shc:generate.
Push pipelines on main are still red, but on script_failure and
job_execution_timeout in apps/index, images and test/baked, real
failures in real scripts, not a missing docker bridge. Nothing in this
document is blocked on the runner, and one thing in it (§4) now has a working
home it did not have this morning.
★ THE CAPABILITY AXES LANDED THREE HOURS AGO. !292 (c552e3f82, merged
7cdaba2da at 10:07 local) added a closed 16-key features: block to
app.yaml, a derived SHC-integration axis, make catalog/capabilities, and 479
lines of generated page. It changes this document’s shape rather than adding to
it; see §2.0. Any earlier draft of this lane that predates it is describing a
catalogue that no longer exists.
1. ★ THE DIFFERENTIATOR, AND IT SURVIVED THE ANALYSIS
Section titled “1. ★ THE DIFFERENTIATOR, AND IT SURVIVED THE ANALYSIS”OpenRouter can tell you a model’s latency because OpenRouter calls the model. Snyk can tell you an npm package’s CVEs because npm is one registry at total coverage. Neither can tell you whether the thing works when you run it, because neither runs it.
SHC is the only party obliged to make these apps work together, on someone else’s hardware, and keep them working. That obligation leaves a residue (files, verdicts, registers) that SHC already owns and nobody else can compute:
| what we can say | who else can say it | where it already lives |
|---|---|---|
| this app’s backup was restored and the contents read back | nobody | shc-backup-verify.service, weekly PROVEN/DISPROVEN/SKIPPED/UNVERIFIABLE/ORPHANED |
| SSO actually redirects at this pin, or here is the author check it fails | nobody | sso-redirect-coverage.md:26-28: 24 wired / 6 pending / 10 structurally excluded |
| the vendor asks for 4 GB, we ship a 2 GB cap, and here is the vendor’s page | nobody | vendor-resource-floors.md, hand-researched, source per row, four BREACH→fixed rows |
| every image in this stack still pulls, checked today | nobody | cmd/verify-image-pins (2,732 lines) on the daily schedule, computed and thrown away |
| this app’s whole stack passes an end-to-end suite | nobody | 89 of 90 packages ship tests/tests.bats |
| what this app plugs into, what plugs into it, and where WE have not caught up | nobody | 214 plugs, 62 sockets, 77 add-ons, and !292’s public 8-gap / 3-substitution / 2-inert lists |
The refinement worth stating plainly: “we run them” sounds like it points at runtime telemetry from customer estates. It does not, and it must not; that is the one thing the privacy line forbids, and it is also the weakest of these signals. Every row above is a packaging artifact: deterministic, un-gameable by a vendor, zero privacy cost, and 60–80% already written. Fleet telemetry ranks last here not out of caution but because it is worth less.
If only one signal beyond the star rating ever ships, ship integration depth: “SSO wired · Backup declared · metrics exported · 7 plugs · 3 sockets · 43 add-ons”. That is not a fact about Nextcloud. It is a fact about how well SHC packaged Nextcloud, which is the actual product being sold.
2. THE DATA POINTS, RANKED BY VALUE PER COST
Section titled “2. THE DATA POINTS, RANKED BY VALUE PER COST”Five privacy classes, extending the vocabulary community/installs.py already
established:
- A: Catalogue. Read from files in this repo. No network, no estate, no person.
- B: Lab. Measured on our own machines. Our data about our own runs.
- C: Receipt. A transaction the customer initiated with us; they pressed Install on our storefront and we carried it.
- D: Volunteered. A person chose to publish it (a review, a comment).
- E: Refused. Never collected. §7.
2.0: ★ Rung 0: the two capability axes, shipped today, already Class A
Section titled “2.0: ★ Rung 0: the two capability axes, shipped today, already Class A”!292 did the hardest part of this lane before the lane started, and it did it in
the shape this document needs. Measured from
docs/app-development/app-capabilities.md:
| count | |
|---|---|
| packages | 90 |
with a features: block | 60 |
with none (every value unknown) | 30 |
| feature keys in the closed vocabulary | 16 |
| …with an SHC integration surface (derivable) | 9 |
| …with no SHC surface at all (declared or nothing) | 7 |
| GAPS: app has it, SHC wires nothing | 8 |
| SUBSTITUTIONS: app lacks it, SHC supplies it anyway | 3 |
| INERT INTEGRATIONS: SHC ships a plug the app cannot consume | 2 |
Four properties of that design are binding constraints on any score built on top of it, and each is already argued in tree:
- The two axes are kept apart on purpose. “Does it do SSO” is not the
same question as “does the SHC package ship an oidc plug”. Reading the
second as the first “punishes a vendor for our backlog” (
c552e3f82). - ★ A gap is never a mark against the app.
app-capabilities.md:20. The 8-gap list is SHC’s to-do list; it must reduce our Operability sub-score and never the app’s. unknownis a value, not an absence, and it is the DEFAULT, because on the day it shipped, 30 of 90 packages declared nothing and “a schema where silence reads as failure marks the whole catalogue down for existing”. This is exactly the evidence-class doctrine §5 needs, already shipped.- Derived beats declared wherever the filesystem can answer, because “a declared answer to a question the filesystem already answers is an answer that rots”. Everything in Rung 1 below is derived for that reason.
The inert-integration finding is the strongest single argument for building
this lane at all. blockscout and monica each ship an oidc.plug.yaml
against an application measured to have no SSO entry of any kind: the plug mints
a Keycloak client nobody reads. Neither axis finds that alone. A trust model
is not a decoration on top of two lists; crossing the lists is where the
defects are.
Rung 1: Catalogue facts · Class A · available TODAY · 90/90 coverage
Section titled “Rung 1: Catalogue facts · Class A · available TODAY · 90/90 coverage”Zero network. Zero telemetry. One importer extension plus the capabilities JSON. Every number in this table was produced by reading these files, which is the proof they are readable.
| id | data point | measured now | source | why a shopper cares |
|---|---|---|---|---|
| PKG-01 | pin discipline: images pinned to a digest vs floating on a tag | 258 of 321 refs pinned (80.4%); 232 unique, 175 pinned; 72 of 90 packages fully pinned; 18 have ≥1 floating; 1 empty ref | apps/**/vars.yaml images: | what you install next month is what we tested this month; a tag can be re-pushed, a digest cannot |
| PKG-02 | health-check coverage | 80 of 90 composes carry ≥1 healthcheck: | apps/**/compose.yaml | whether the platform can tell “started” from “working” |
| PKG-03 | backup declared, and what it captures | 70 of 90 | apps/**/backup.yaml | the single most-asked self-hosting question |
| PKG-04 | SSO wired by SHC (derived, ≠ “app supports SSO”) | 41 oidc.plug.yaml + 3 saml.plug.yaml; the capability page counts 50 wired incl. 3 gate substitutions | apps/**/*.plug.yaml | one login across the estate, or 41 |
| PKG-05 | upgrade path: migrations and lifecycle jobs | 5 packages carry migrations/; 107 *.job.yaml | apps/** | whether an upgrade is a button or a weekend |
| PKG-06 | package test suite present | 89 of 90 | apps/**/tests/tests.bats | somebody wrote an end-to-end test for this whole stack |
| PKG-07 | ★ integration depth | 214 plugs, 62 sockets, 77 add-ons; metrics exported by 37, import/export by 70 | apps/** + capabilities JSON | the differentiator (§1) |
| PKG-08 | privilege posture | 8 packages declare privileged_waivers | apps/** | this container asks for more than the others |
| PKG-09 | declared footprint vs published vendor floor | already a sourced table | vendor-resource-floors.md | will it fit on the box I have |
| PKG-10 | licence, and whether we verified it | 68 of 90 say unverified, only 22 verified | app.yaml license.id | can I ship this at work |
| PKG-11 | declared runtime / architecture support | 4 of 90 declare requires.runtimes; 1 of 90 requires.architectures | app.yaml | almost entirely absent; Rung 2 measures it instead of asking |
| PKG-12 | image count as attack surface | median 2, max 21 (flagship/openreplay) | vars.yaml | a 21-container stack is a 21-container problem |
| PKG-13 | packaging cadence: dated writes to version: | 90/90 have ≥1; 55/90 have ≥2 | git log -L /^version:/,+1: | our clock, and it must be labelled as ours, §5 |
| PKG-14 | upstream capability (16 keys, tri-state + unknown) | 60 of 90 declared | app.yaml features: | the only axis that answers “can the APP do it” |
| PKG-15 | ★ the SHC gap list | 8 gaps, 3 substitutions, 2 inert | capabilities JSON | our debt, in public; §5 makes it cost us, not the app |
★ Ship the uncomfortable rows with the flattering ones. PKG-01’s 63 floating
refs, PKG-10’s 68 unverified licences, PKG-15’s gap list, and the existing
apps/VERSION_IMAGE_DIVERGENCE and apps/FRESHNESS_UNVERIFIABLE registers are
findings against SHC, not against the apps. A store that publishes its own
defect registers is believed about everything else, and every one of those
numbers shrinks once it is visible. !292 already did this with the gap list;
extend the habit rather than inventing it.
Rung 2: Registry facts · Class A · available TODAY · one anonymous request per digest
Section titled “Rung 2: Registry facts · Class A · available TODAY · one anonymous request per digest”cmd/verify-image-pins already enumerates every ref in every package,
authenticates anonymously against docker.io / ghcr.io / quay.io / Harbor /
Gitea, handles rate limits and declared registry outages, and sends the
manifest-index Accept header. Lift collect / parseRef / token into an
internal package and add a sibling command: ~14 KB of network per ref.
| id | data point | source | why a shopper cares |
|---|---|---|---|
| IMG-01 | ★ still pullable, checked today: the verdict apps/verify-pins already computes daily and discards | manifest HEAD | the honest SHC analogue of OpenRouter’s uptime chart. greenbone’s registry garbage-collects superseded manifests within hours; a freshly-installed stack has gone unpullable the same evening |
| IMG-02 | architectures actually published | manifest index | flagship/ivatar/app.yaml:136-140 hand-documents amd64-only in a prose comment. One request answers it for all 232 unique refs, and closes PKG-11’s 1-of-90 hole by measurement |
| IMG-03 | compressed download size, layer count | manifest | ”how much am I about to pull” |
| IMG-04 | ★ image build date | config blob created | the freshness signal that replaces the NULL released_on with no backfill and no new collection. Answers the owner’s requirement 5 today. (scripts/import_catalog.py:1654 leaves released_on to the bench default; it is NULL on all 90) |
| IMG-05 | runs as root / declared USER | config blob | a real security signal, actionable, 14 KB |
| IMG-06 | base distro | config blob | report factually; let SEC-01 carry the consequence (§6) |
Rung 3: Install receipts · Class C · half available TODAY, unread
Section titled “Rung 3: Install receipts · Class C · half available TODAY, unread”Install Dispatch already records state ∈ {queued, delivered, running, done, failed, refused}, an error code, a detail, and three timestamps:
issued_on, delivered_on, settled_on
(frappe/apps/portal/portal/portal/doctype/install_dispatch/install_dispatch.json).
The marketplace reads only settled rows. Every failure and every refusal, with
its reason and its timing, is already on disk and unread. Highest value per
hour in this document, and it collects nothing new.
| id | data point | available | why a shopper cares |
|---|---|---|---|
| INST-01 | install success rate: state mix per app | today, from existing rows | ”94% of store installs of this app succeeded” is a claim no competitor can make |
| INST-02 | failure reason mix: the code histogram | today | tells us which packages to fix; tells a shopper what usually goes wrong |
| INST-03 | time to running: issued→delivered→settled | today | ”typically running in four minutes” |
| INST-04 | installs from registered clusters | LIVE (§0) | adoption. Never a sort key, §5 |
Rung 4: Our own lab · Class B · computed TODAY, stored nowhere
Section titled “Rung 4: Our own lab · Class B · computed TODAY, stored nowhere”| id | data point | state |
|---|---|---|
| LAB-01 | backup restore PROVEN / DISPROVEN | the weekly sweep runs and emits a per-deployment verdict; nothing stores it per app |
| LAB-02 | SSO actually redirects | sso-redirect-coverage.md is a dated, researched matrix, in prose. It needs a machine-readable companion beside it, not a rewrite |
| LAB-03 | vendor floor breaches | same shape, same need (vendor-resource-floors.md) |
| LAB-04 | end-to-end smoke verdict per app | 89 suites exist; nothing records pass/fail per package per release |
| LAB-05 | upgrade success across a version pair | not built |
★ LAB-01, LAB-02 and LAB-04 are the flagship metrics of this store, and none of them touches a customer. Two of the three are already researched and already dated; they are invisible to shoppers only because they live in prose.
Rung 5: The scanner · Class A · needs the service (§4)
Section titled “Rung 5: The scanner · Class A · needs the service (§4)”| id | data point | note |
|---|---|---|
| SEC-01 | fixable CVE counts by severity, per digest | ★ score on fixable, never raw. Measured: nextcloud:34.0.2-apache carries 36 CRITICAL with 0 fixable; gitlab/gitlab-ce:18.11.7 carries 20 CRITICAL with 20 fixable. A raw count ranks those identically and is therefore worthless |
| SEC-02 | SBOM component count | the anti-opacity floor. busybox (shipped by 16+ packages) makes Trivy exit 0 with "Results": null; a naive consumer renders a green tick. Below a component floor the verdict is unknown, never “0 findings” |
| SEC-03 | ecosystem / language mix | falls out of the scan for free; reported, never scored (§6) |
| SEC-04 | exposure trend over time | the owner’s requirement 4. ~0.3 s per image per night from a cached SBOM, zero registry traffic |
Rung 6: Community · Class D · needs the bench migration first
Section titled “Rung 6: Community · Class D · needs the bench migration first”App Review / App Comment / App Comment Vote / App Content Report are
merged, with Wilson lower-bound ranking (community/ranking.py:95, 95%, no time
decay). The anti-gaming lever exists and is unused: community/api.py:674
_verification_for proves a reviewer’s own registered cluster runs the app, so
each fake vote costs a cluster registration plus an install. Show two
numbers side by side, always: all reviews, and verified-install reviews, and
rank on the second.
The owner’s ruling that a star requires a review is already the shipped shape: ratings live inside the comment system, not as a one-click widget.
3. WHAT IS AVAILABLE WHEN: the plain answer
Section titled “3. WHAT IS AVAILABLE WHEN: the plain answer”| availability | data points |
|---|---|
| Today, no new collection, no network | PKG-01…PKG-15 (15 metrics, 90/90 coverage, two of them shipped this morning), INST-01…INST-03 (already on disk), LAB-01…LAB-04 (already computed, not stored) |
| Today, one anonymous registry request per digest | IMG-01…IMG-06 |
| Needs the install-reporting infrastructure to grow | INST-04’s coverage. The counter is live; it renders nowhere until a portal_dispatch_key is minted. Not blocked on design |
| Needs the scanner service | SEC-01…SEC-04 |
| Needs the community doctypes migrated onto the live bench | COM-01…COM-03. ★ Merged in git, absent in production: a field query for review_count against api.marketplace.selfhostedcloud.org returns DataError: Field not permitted in query (measured read-only 2026-08-18). The review system cannot receive its first review until this ships |
| Off-limits, refused | §7 |
4. THE SCANNER: Trivy produces; our own packaged apps optionally consume
Section titled “4. THE SCANNER: Trivy produces; our own packaged apps optionally consume”Trivy is the authority the storefront renders. One static binary, no daemon;
--image-src remote scans straight from a registry with no Docker involved;
emits CycloneDX and SPDX; covers OS and language ecosystems in one pass.
integrated/dependency-track (5.0.4, already packaged) and
enterprise/defectdojo (3.2, already packaged) are optional deep surfaces off
the critical path; they ingest what Trivy produces. Dependency-Track cannot
be the scanner: it has no container image analyzer. Keeping them off the
critical path is what stops a DT outage blanking the shop window; that is
dogfooding without making the shop depend on the merchandise.
Four tiers, because the costs differ by ~40,000×:
| tier | what | when | cost |
|---|---|---|---|
| 0 | inventory: reuse cmd/verify-image-pins collect() verbatim | on apps/** change | free |
| 1 | config facts: manifest + config blob (IMG-02…IMG-06) | on apps/** change | 14 KB per ref; ~42 s for the catalogue at concurrency 8 |
| 2 | SBOM, generated once per digest and kept forever (a digest is immutable) | when a new digest appears | ~8.6 s/image cold; 40–80 GB one-time transfer; ~192 MB stored |
| 3 | vulnerability match from the cached SBOM | nightly, every stored digest | ~0.3 s/image, zero registry network, ~12 s for the catalogue |
Tier 3 is the time series. The expensive part is digest-keyed and paid once; the part that changes daily is the match against a moving vuln DB. A digest whose counts move while the package did not change is exactly the trend the owner asked for, and it costs seconds.
★ WHERE IT RUNS: the finding that changed today. The repo already has the
right lane. .gitlab-ci.yml:1951 runs catalog/freshness and :1851
apps/verify-pins on a daily schedule described in that file’s own header
as “the CHECKS-ON-THE-WORLD only”, and that schedule pipeline was green at
07:07Z today. It is not a CI guard: it gates no merge and reds no MR, so
it satisfies the owner’s no-guards ruling exactly, and it is already the home of
two probes of precisely this shape. make catalog/scan belongs beside them.
That is strictly better than a cron on the owner’s laptop: it exists, it has
credentials, and it already reports.
★ unknown must be a first-class value, and the failure it prevents is
real. Two reproductions from the prototype: busybox exits 0 with
"Results": null, and a live ghcr.io blob failure mid-run would have
overwritten a real immich result (29 CRITICAL / 111 HIGH) with “0
vulnerabilities” had the state guard not held. Therefore state ∈ {scanned, empty, unresolved, failed}; counts are NULL, never 0, for every state but
scanned; coverage is a published field on every rollup; any derived grade
degrades to a literal unknown rather than a number. A consumer that forgets to
branch gets a blank or a type error, never a green tick it did not earn. This
is the same rule !292 already applies to features:, one layer down.
One cheap thing first: repin the 63 floating refs. They are 20% of all refs,
one is an empty string, and greenbone alone accounts for 19 (seven with no tag
at all, i.e. :latest), huly 14, dify 9, penpot 4. Every scan result attached to
a floating ref decays silently, and apps/VERSION_IMAGE_DIVERGENCE already
records that greenbone’s registry garbage-collects superseded manifests within
hours. apps/verify-pins already probes them, so the work is bounded.
5. ★ THE TRUST SCORE: the verdict
Section titled “5. ★ THE TRUST SCORE: the verdict”Ship a ledger with a denominator and four named sub-scores. Do not ship a composite number yet. Do not leave it undecided.
The ledger is the product
Section titled “The ledger is the product”One doctype, App Fitness Check: (app, app_version, check, verdict, value, source, measured_at). Every number on every page is a view over rows, and
clicking a number opens the row that produced it. verdict uses the
vocabulary this repo has already converged on four times independently:
cmd/verify-image-pins→PULLABLE/UNPULLABLE/UNVERIFIED/UNREACHABLE, where UNVERIFIED is explicitly never a pass- the backup sweep →
PROVEN/DISPROVEN/SKIPPED/UNVERIFIABLE/ORPHANED apps/FRESHNESS_UNVERIFIABLE→ structurally unmeasurable subjects, named with a reason, itemised in every run’s output, suppressing the failure but never the report- !292’s
features:→supported/paid/absent/unknownas the default
The doctrine underneath all four, in FRESHNESS_UNVERIFIABLE’s own words: “a
probe which cannot see is not a green.” The ledger is that doctrine given a
schema.
source is mandatory and visible: probe:verify-image-pins@2026-08-18,
derive:catalog-capabilities@7cdaba2da, or manual:clay@2026-08-18.
Hand-baked inputs are allowed and labelled; the precedent is already in
tree, since vendor-resource-floors.md is entirely hand-researched, cites a
vendor doc per row, and nobody mistakes it for a probe. The rule is that a
hand-entered input must never silently count as measured.
Four sub-scores, each with its own coverage
Section titled “Four sub-scores, each with its own coverage”| sub-score | shopper-facing question | inputs | coverage today |
|---|---|---|---|
| Operability | how well did SHC package this? | PKG-02…PKG-08, PKG-15, LAB-01, LAB-04 | 90/90, measurable now |
| Security | what am I exposing? | SEC-01, SEC-02, PKG-01 pin gate, PKG-08 waivers, IMG-05 | 0/90, needs §4 |
| Freshness | is anyone still working on this? | IMG-04 image build date, PKG-01 pin lag, PKG-13 packaging cadence | partial, IMG-04 needs Rung 2 |
| Community | what do operators say? | COM-01, COM-02 | 0/167 listings, needs the bench migration |
Capability (PKG-14) is a FILTER, not a score. !292 was explicit that a gap is
never a mark against the app; folding the 16 keys into a number would do exactly
that by the back door, because 30 of 90 packages are entirely unknown and any
weighting of unknown is an invention. Render the chips, let shoppers filter,
and keep them out of the arithmetic.
Each sub-score carries one of three evidence classes, and this is how absence and confidence are expressed:
- Measured: every input present → number and bar.
- Partial: number computed over the present inputs only, with the missing ones named on the page. A missing input never scores as average and never as zero; it removes itself and says so.
- Insufficient: no number at all. An empty hatched bar and the reason.
Print the denominator beside every score: “7 of 9 checks measured.” An app with a lower score and full coverage should outrank one with a higher score and three blanks, and only a printed denominator makes that legible.
★ A gap costs US, not the app
Section titled “★ A gap costs US, not the app”The single scoring rule that keeps this honest, and it is already ruled in
app-capabilities.md:20: when the app supports a capability and SHC wires
nothing, Operability falls and the app’s own chips are untouched. A
SUBSTITUTION (SHC supplies what the app lacks, the forward-auth gate in front
of uptime-kuma, s3manager, agent-canvas) raises Operability. An INERT
INTEGRATION (blockscout, monica: an oidc plug against an app with no SSO
entry) is a package defect and must reduce Operability harder than a plain
gap, because a gap is honest absence and an inert plug is a claim that does not
hold.
Why no composite yet: the argument is measured, not aesthetic
Section titled “Why no composite yet: the argument is measured, not aesthetic”Coverage of the inputs a composite would need, today: Operability 90/90, pin hygiene 90/90, capability declared 60/90, CVE 0/90, upstream cadence 0/90, ratings 0/167, installs rendering nowhere, licence verified 22/90. A weighted sum written today resolves arithmetically to Operability while printing itself under a heading that says trust. His own cited reference (OpenRouter) ships no composite: it ships measurements with units. Snyk’s composite is defensible because Snyk’s inputs come from one registry at near-total coverage. On coverage we are OpenRouter, not Snyk.
Meanwhile the one strip cell !264 asked for carries Operability, not a fake
composite. features/marketplace/src/listing/ListingFacts.tsx on !264’s
branch (design/marketplace-product-page; the comment is not on main) says in
its own header that whoever merges the two branches “owes this cell a decision
rather than a rebase”. Measured at 390×844 the four-cell case already wraps to
two rows, so it is a one-cell budget. Spend it on the number that is real.
The composite revisit gate: three testable conditions
Section titled “The composite revisit gate: three testable conditions”Publish a composite the day all three hold, and not before:
- the scanner covers ≥95% of digest-pinned refs with results <14 days old;
- ≥30% of app listings carry ≥5 published reviews;
- release recency is populated for ≥80% of apps, from IMG-04 or a real
released_onbackfill.
When they hold, Snyk’s shape becomes defensible because the coverage that justifies it finally exists. Until then a composite is a costume.
The hand-baked input: VETO-ONLY
Section titled “The hand-baked input: VETO-ONLY”Name it Vendor Note. It may suppress a listing from a shelf, attach a
warning, or force a sub-score to Insufficient. It may never raise a computed
score. The asymmetry is the safeguard: the owner is the best source alive for
“this upgrade path eats data” and structurally not a trustworthy source for
“better than its measurements say”, which is indistinguishable from a thumb on
the scale, including to him. Mandatory fields: a closed flag vocabulary
(upgrade-hazard, upstream-abandoned, not-production-ready,
restore-verified, licence-changed), set_by, set_on, expires_on (max 12
months; an unrenewed note stops rendering), and a one-sentence reason that
prints in the ledger. A closed vocabulary, exactly as features: is closed,
for exactly the same reason.
Sorting
Section titled “Sorting”Operability may sort the catalogue. It is measured at 90/90, and absence there is genuine absence: a package that declares no backup is a package with no backup, which is a measurement, not a blank.
Installs may never sort anything. The existing Trending/Popular refusal
(features/marketplace/src/catalog/types.ts:140-158) stands verbatim, and
install_cluster_count does not overturn it: it counts only registered
clusters, is withheld below a privacy floor so most listings have no value at
all to sort by, and “sorting a shelf by it would put the apps we happen to have
registrations for at the top and call that popularity.” Adoption is a fact, not
a quality judgement.
6. LANGUAGE DOES NOT ENTER THE SCORE
Section titled “6. LANGUAGE DOES NOT ENTER THE SCORE”His own instinct was right: “maybe that just comes down to CVEs”.
- Already priced in. Memory-safety shows up as fewer CVEs at the digest. Counting the language as well counts the same evidence twice and calls it independent.
- Not measurable from what we ship. The catalogue ships containers, and !292’s closed 16-key vocabulary, chosen because every key earned its place by varying across the 90 packages, contains no language key. It would be hand-entered: an opinion dressed as a measurement.
- It penalises the catalogue’s own spine on a correlation (nextcloud PHP, midPoint Java, frappe Python) while saying nothing about those pinned digests.
- It puts “written in PHP, therefore less trustworthy” on a page selling that vendor’s software.
Report the base image factually (distro, size, root or non-root (IMG-05, IMG-06)) because a shopper can act on that. Let SEC-01 carry the consequence.
7. ★ WHAT WE REFUSE TO COLLECT, AND WHY IT IS WRITTEN DOWN
Section titled “7. ★ WHAT WE REFUSE TO COLLECT, AND WHY IT IS WRITTEN DOWN”The per-version install histogram is FORBIDDEN. AppEntry.Version already
rides the heartbeat into Cluster.app_inventory
(modules/portal/wire/wire.go:113). Nothing today breaks the install count down
by version, and nothing today stops someone doing it; it is one GROUP BY
away. Joined to a published CVE posture it stops being a metric and becomes a
sized, prioritised target list: here are the estates running the version
with the unpatched RCE. The refusal belongs in code, beside
min_display_count in community/installs.py, in that file’s own voice, so
the next person to reach for it reads the argument before writing the query.
Also refused:
- Any usage signal. Not a launch, a request, a session, a page view, a login, a user count, or a byte of what anybody did with the app. The owner named this line himself: “we can’t really collect number of users that people use when they install it. So I have to be careful.”
- Real memory from customer estates: deferred, probably permanently. It is the only proposal that would widen the heartbeat wire, and the wire’s narrowness is a security property:
AppEntryhas no field that could carry a usage fact, so nobody can add one quietly. Declared footprint (PKG-09, Class A) plus OOM diagnosis on our own estates (Class B) answer the shopper’s question without touching anyone’s data. - Anything derived from an unregistered estate. There is nothing to derive: it has no
Clusterrow and sends no heartbeat. It is not counted as zero; it is not counted.
★ A privacy hole that must close before installs feed any surface.
portal.enabled defaults off, good; not being counted is the default and
the majority path. But a registered estate currently cannot decline the
inventory: daemon/wiring/portal_wire.go:147 wires Apps: localAppInventory
unconditionally. Today it is all-or-nothing. Do not let a trust score start
depending on install data before that toggle exists; a score that needs the
number is a reason to keep collecting it.
8. THE PRIVACY POSTURE: one paragraph, written for a customer to read
Section titled “8. THE PRIVACY POSTURE: one paragraph, written for a customer to read”Almost everything on this page is measured from the app’s own package files and its public container images. We read the files we wrote, and we ask public registries public questions; that requires nothing from you and nothing from your servers. A few figures come from our own test estates (whether a backup restored, whether single sign-on redirected) and those are our machines, not yours. The install count is different, and we say so wherever it appears: it counts registered clusters, from estates that chose to connect to our portal, and it is shown only above a floor of five, so no single estate can be picked out of it. We never collect usage. Not a login, not a session, not a page view, not a user count; there is no field in the channel that could carry one. If you never register, you are not counted as zero; you are simply not counted, and everything on this page still works.
9. THE ORDERED WORK
Section titled “9. THE ORDERED WORK”Each rung is days, not weeks, and each ships a real number before the next starts.
- Publish the catalogue facts (PKG-01…PKG-15). One importer extension plus
make catalog/capabilities -format json; no network, 90/90 coverage. This alone gives a real body to the listings that currently have none. Ship the defect registers with it: the 63 floating refs, the 68 unverified licences,VERSION_IMAGE_DIVERGENCE, the 8-gap list, and SSO-01’s ten structural exclusions. - Read the install outcomes already on disk (INST-01…INST-03). Highest
value per hour here; collects nothing new. Needs the
frappe/apps/**subtree republish ceremony (scripts/publish-frappe-apps.sh, guarded bytests/unit/ci/frappe_apps_subtree_test.go), or a guard reds main. - Land the fitness ledger (
App Fitness Check) now that four real producers exist to prove the shape. Not first; the shape is only credible once something fills it. - Machine-read the two curated registers (LAB-02, LAB-03). Two of the three most differentiating facts in the store, already researched, invisible only because they are prose.
- Repin the 63 floating refs. Bounded work; makes everything downstream honest.
- Add the registry probe (IMG-01…IMG-06) by lifting
collect/parseRef/tokenout ofcmd/verify-image-pins. Brings IMG-04, the freshness signal that unblocks the third gate condition in §5. - Build the scan service (§4) as
make catalog/scan, on the existing daily schedule besidecatalog/freshness. - Migrate the community doctypes onto the live bench. A prerequisite, not a nicety; the review system cannot receive its first review until it lands.
- Ship the portal inventory opt-out before installs feed any surface.
- Store the per-app smoke verdict (LAB-04), then restore proofs (LAB-01), then upstream cadence last.
★ One thing to be careful about: install_cluster_count already works and is
well designed. Do not extend it. The temptation to add a version breakdown, a
trend line or a growth rate is exactly the pressure §7 exists to resist.
10. WHAT NEEDS THE OWNER’S RULING
Section titled “10. WHAT NEEDS THE OWNER’S RULING”- Four named sub-scores with printed coverage, and NO composite until the §5 gate opens. Accept, or overrule. If overruled, the composite must still print its four components beside it; that half is non-negotiable and Snyk agrees.
- A gap costs SHC, an inert plug costs it more, a substitution earns credit, and none of the three touches the app’s own chips. This is the scoring rule that makes !292’s two axes safe to build on. Confirm.
- Capability (
features:) is a filter, never a score input. 30 of 90 packages are entirelyunknown; any weighting ofunknownis an invention. - The one strip cell in !264 carries Operability, not a composite. !264’s own header asks whoever merges main into it to decide this rather than rebase past it. At 390×844 the four-cell case already wraps, a one-cell budget.
- Vendor Note is veto-only (may suppress, warn, or force
Insufficient; may never raise a score). Accept, or name a separately-labelled boost path, but never a silent one. - The per-version install histogram is forbidden in code, beside
min_display_count. Confirm. - Operability may sort the catalogue; installs may never sort anything. Rule explicitly before anything ranks.
- Repin the 63 floating refs, or ship a per-listing “as of digest X, observed <date>” caveat instead? Recommend repinning.
- The scanner runs on the existing daily schedule,
catalog/freshness’s lane, a check-on-the-world, not a merge guard, and not on a laptop cron. Confirm. - Grade bands on any exposure metric are a product call, not a measurement. Nothing ships publicly until you set them.
- 68 of 90 packages declare
license: {id: unverified}. Not a scoring input either way, but “unverified” on three quarters of a store’s pages is its own trust problem. Is that a backfill project? - Two cells !264 refused for lack of data (“Available architectures” and “Download size”) become available the day the Rung-2 probe lands. Reopen them then, or leave them closed?