Pulp npm metadata overlay
SHC-730 adds an optional image for pulp-content. images.pulpContent defaults
to empty and falls back to images.pulp. API, workers and initialization keep
their existing image. Enabling the overlay does not migrate the database or
publish packages.
The image requires pulpcore 3.114.2, pulp-npm 0.9.0, semantic-version 2.10.0 and
Python 3.11. Its base is images.pulp by the same tag and digest, so the
content service keeps the fork’s pulpcore and plugins and differs from the API
and workers only by the overlay. TestPulpContentOverlayBaseIsServerImage
fails a re-pin of images.pulp that leaves the Dockerfile behind. The build verifies
both npm source hashes before replacing only NpmDistribution.content_handler
with a delegate. Source drift fails the build. The inherited runtime command
and user are retained. Image labels record the base, source hashes and revision.
Each advertised version comes from the stored gzip artifact’s canonical,
regular package/package.json. Dependencies and other manifest fields survive;
server-owned identity, distribution URL and digests replace packed values.
Conditional exports and imports retain their packed declaration order,
including nested conditions and array fallbacks, on cold and cached responses.
Reordering these keys can change which target a consumer resolves. An image
update does not remove old shared response-cache entries; targeted invalidation
and public metadata/cold-install verification remain rollout requirements.
hasInstallScript includes the implicit binding.gyp installation case, and
_hasShrinkwrap records an actual regular shrinkwrap file. SHA-256 must match
the Pulp Artifact; SHA-512 integrity and SHA-1 shasum are computed from those
same bytes. There is no remote registry or source-publication fallback.
Both JSON and install-v1 requests receive the complete JSON document. Version
times are the actual Package.pulp_created instants. time.created is the
earliest represented creation, and time.modified is the selected repository
version’s creation. These are Pulp ingestion times, not original npm publication
dates. GET, cache misses and re-association never change a version’s time.
latest is the greatest stable strict SemVer. A prerelease-only package uses
the greatest prerelease. Equal precedence uses the original version string as
a deterministic tie break. Arbitrary dist-tag CRUD isn’t provided.
Pinned repository versions remain pinned. Queries constrain both package domain fields and repository membership, and require one stored artifact in the same domain. Hosted metadata refuses repositories or distributions with remotes. Scoped package paths accept percent encoding. Tarball paths, including prerelease/build names, return to the existing core artifact handler. Content guards remain upstream of the delegate. URLs use the configured HTTPS content origin, domain and base path plus the actual ContentArtifact relative path.
Limits
Section titled “Limits”Per archive: 64 MiB compressed, 256 MiB expanded, 20,000 members and 1 MiB per manifest or tar extension. Recorded Artifact.size and the configured storage object size must be positive integers, agree and fit the compressed limit before opening the file. This preflight precedes the S3 adapter’s eager body download. The byte reader and decompression limits still apply; no files are extracted. Invalid gzip/tar/PAX, sparse archives, ambiguous manifest aliases and identity mismatches fail closed. Only the canonical package root is supported. Per document: 1,024 versions and 16 MiB of serialized versions.
The process-local LRU stores at most 128 entries and 32 MiB of serialized
artifact-derived data. Its key includes schema, domain, artifact identity and
SHA-256. Failed validation isn’t cached. Returned objects are detached;
repository membership, URLs and timestamps are rebuilt for every request.
Cache misses are serialized per process to bound concurrent archive work.
Metadata failures raise JSON aiohttp HTTP errors, which escape Pulp’s cache
entry method. Ordinary error responses would be cached even with no-store.
Successful responses retain core caching; invalidation after publishing and
edge cache behavior still require verification during rollout.
The reviewed 34-package 7.8.0 archive fixture passed complete manifest and digest
comparison. Its cached metadata occupied 88,066 bytes. Local filesystem timings
are not S3 performance measurements. A newly published non-exempt package must
wait for its true creation time plus the consumer’s release-age interval.
Preserve minimumReleaseAge: 7200 and the existing exclusion list, including
@multiplatform.one/*. mp.one is not covered by the exact multiplatform.one
exclusion. Never backdate content or weaken the policy to make an install pass.
Local verification
Section titled “Local verification”From the repository root:
uv run --no-project --python 3.11 --with pytest --with semantic-version==2.10.0 --with aiohttp pytest docker/pulp-npm-metadata/tests/test_metadata.pyuv run --no-project --with semantic-version==2.10.0 --with aiohttp python docker/pulp-npm-metadata/tests/cold_install_probe.py --consumer /path/to/consumer --output /tmp/pulp-cold-uniqueuv run --no-project --with semantic-version==2.10.0 --with aiohttp python docker/pulp-npm-metadata/tests/verify_archives.py --train /path/to/reviewed/train.json --output /tmp/pulp-archive-proof.jsonmake -C docker build/pulp-npm-metadata BAKE_OUTPUT=docker DOCKER_REGISTRY=shc-local DOCKER_TAG=SHC-730CGO_ENABLED=0 go test -C tests -count=1 ./unit/catalog -run Pulpbats tests/scripts/images-publish.batsThe image-only test_s3_storage.py and test_core_cache.py regressions use the
installed Django S3 adapter and Pulp cache methods with recording/in-memory
transports. Run these alongside test_metadata.py inside the built image with
pytest available and networking disabled. They require no Redis or credentials
beyond the existing static synthetic S3 fixture values.
The cold harness requires pnpm 10.32.1 selected in the consumer checkout. It copies only the effective age/exclusion/resolution policy and uses separate user/global npm config paths, stores and caches. It never changes HOME. Its loopback HTTP fixture rewrites only its own synthetic HTTPS tarball origin for local transport. Controls cover missing timestamps, complete dependency/peer/ optional metadata, exports, corrupt tarballs, missing dependencies, release age and engine refusal. Scripts stay disabled. Explicit highest/time-based controls are separate from the normal-policy run.
docker/pulp-npm-metadata/tests/compose.yaml provides an isolated Postgres,
Versity S3 Gateway and Redis stack with the overlay only on content. It requires
PULP_TEST_ENCRYPTION_KEY pointing to an existing isolated test key, mounted
read-only. Do not use an estate key. No key generator is included. All other
credentials are static synthetic fixture values. Ports bind only to loopback.
PULP_TEST_ENCRYPTION_KEY=/path/to/existing/test.key docker compose -p shc730-test -f docker/pulp-npm-metadata/tests/compose.yaml up -duv run --no-project --with semantic-version==2.10.0 --with aiohttp python docker/pulp-npm-metadata/tests/integration_probe.py --output /tmp/pulp-integration.jsonThe integration probe refuses non-loopback targets and uploads synthetic archives through the publisher’s multipart REST endpoint. It checks actual metadata and S3 tarball delivery, pinned/latest membership and a warm-cache subsequent upload. It leaves fixture resources for inspection. This probe has not yet completed: the implementation run stopped at Pulp’s missing test encryption-key precondition. Real content-guard refusal, edge caching and cold pnpm against the patched Pulp stack remain acceptance work.
Scoped metadata ingress
Section titled “Scoped metadata ingress”The catalog’s ungated_npm_metadata declaration binds GET/HEAD metadata reads
to the pulp-web service and the exact 34 @multiplatform.one package names
under /pulp/content/multiplatform.one/. It names no host: CE renders the
sibling on every host the deployment routes to pulp-web, so the hosts come
from the deployment’s own --expose (SHC-1769). An entry that still carries a
host key is ignored. The declaration is independent of package versions. Adding another name is a
catalog policy change; an arbitrary package or another distribution does not
inherit the exemption.
CE reads the declaration during both install and ingress reconciliation.
It emits a sibling at priority 89001 using the existing service and TLS
configuration. The scope/name separator accepts / or %2F, including lowercase
encoding, and the scope marker accepts @ or %40. A full-path match excludes
tarballs and suffixes. The original ungated_paths traversal guard stays
unchanged, including its encoded-slash refusal.
The new sibling inherits the renderer’s gate-cookie stripping and LAN access policy. It carries no forward-auth gate or response injector. Pulp still owns distribution authorization and content guards; this route neither grants a role nor turns a private distribution public. Decoded non-read requests can still reach the original content-prefix router, where Pulp enforces its method contract. No credentials are added to public discovery.
Traefik 3.6.25 normalizes unreserved escapes and dot segments before routing. Normalization-equivalent canonical public names are accepted. Encoded percent remains reserved, so double-encoded separators do not gain this exemption. Destination isolation is the contract; original wire spelling is not retained for a later router to inspect.
Validation uses the supported CE overlay runner against an isolated CE checkout:
SHC_CE_DIR=/path/to/reviewed-ce sh scripts/ce-test.sh -run 'TestNPMMetadata' ./modules/appCGO_ENABLED=0 go -C tests test -mod=readonly -count=1 ./unit/catalog -run TestPulpNPMMetadataIngressDeclarationTestNPMMetadataPinnedTraefik is opt-in and skips without
CE_NPM_TRAEFIK_BINARY. Supply an independently checksum-verified upstream
Traefik 3.6.25 executable to run the actual entrypoint/router and this
checkout’s cookie-strip plugin against loopback content/gate fixtures:
SHC_CE_DIR=/path/to/reviewed-ce \CE_NPM_TRAEFIK_BINARY=/path/to/verified-traefik-3.6.25 \sh scripts/ce-test.sh -v -run '^TestNPMMetadataPinnedTraefik$' ./modules/appThe fixture uses the real catalog allowlist and tests all names with encoded and decoded separators, non-read methods, neighboring paths and the original OpenReplay traversal cases. Its upstream 404/403 and gate 401/302 responses are synthetic markers. They prove routing and response preservation, not actual Pulp content-guard execution or live edge behavior. The parent must confirm the deployed edge version and retain the separate Pulp guard/publication proof.
This declaration requires the CE renderer revision that understands it. Release that revision and advance SHC’s CE dependency pins before deploying the catalog; an older daemon ignores the new field and encoded metadata still receives 401. The developer CE overlay is not a release pin. Removing the declaration and reconciling removes only this sibling and restores the original route set. Do not patch generated ingress YAML or relax the shared traversal guard.
Rollout and rollback
Section titled “Rollout and rollback”The parent owns publication and deployment after independent review. Capture
the existing deployment image and repository/distribution state first. Publish
through the existing images/core Bake pipeline, then set only
images.pulpContent to the reviewed immutable image digest. Review the rendered
service diff and run the real train’s metadata/tarball and cold-client checks.
Also verify the existing apt/file supply paths, which share the content process.
Invalidate only proven stale entries for the affected npm distribution, never
the whole Redis cache.
The images.pulpContent override is persistent deployment state, so a later
re-pin of images.pulp doesn’t reach the content service on its own. Re-point
the override to the overlay image built from the re-pinning commit in the same
update.
Rollback restores the captured images.pulpContent override, empty if it was
previously absent, and reconverges content only. There is no data migration to
reverse. Rollback also restores the original npm metadata limitation, so leave
consumer cutover blocked until registry acceptance passes.