Skip to content
SHC Docs

Pulp mirror targets

The pulp app takes any number of s3 connections as mirror targets:

Terminal window
shc -t corp update pulp -C mirror.apt-basedlinux=cloudflare -C mirror.bin-basedlinux=cloudflare

Each -C mirror.<name>=<connection> is one target. The target name is the bucket name. Two kinds of connection satisfy it:

ConnectionWhat happens at bind
cloudflare (R2)the daemon creates bucket <name> on R2, mints a token scoped to it, attaches the domain the grant names
s3 (a held key: MinIO, Hetzner, a pasted R2 key)nothing is created; bucket <name> must already exist

Artifact storage is untouched: the s3 plug and vars.s3.* remain the one STORAGES backend. A mirror target is a copy the fork’s pulp_mirror plugin (SHC-845) writes on top of it: the chosen distributions as a static apt/file tree, so public artifacts serve without Pulp in the path.

Every bound target becomes per-name settings in the shared pulp environment, which dynaconf folds into settings.MIRROR_TARGETS[<name>]:

PULP_MIRROR_TARGETS__<name>__endpoint https://<account>.r2.cloudflarestorage.com:443
PULP_MIRROR_TARGETS__<name>__bucket <name>, or the bucket the mint chose
PULP_MIRROR_TARGETS__<name>__region auto / us-east-1 / …
PULP_MIRROR_TARGETS__<name>__access_key
PULP_MIRROR_TARGETS__<name>__secret_key
PULP_MIRROR_TARGETS__<name>__path_style true

The plugin upserts one MirrorTarget row per name from these on start; which distributions go to which target is the plugin’s own rule set, driven from its API and the dashboard toggle.

Before any pulp process starts, the pulp-mirror-probe start job HeadBuckets every target with its own key. It is read-only (no mb, rb, cp, sync, rm, put or delete) and it fails the compose up, and with it the install or update, loudly:

CodeMeaning
X5055MIRNOBthe bucket does not exist at the endpoint; the probe never creates one
X5056MIRDENthe key was rejected for that bucket
X5057MIRUNRthe endpoint did not answer in 30 attempts

With no target bound the probe prints one line and exits 0, so an install without mirrors is unchanged.

Rebind the name to another connection and the old mint is revoked before the new one is made. Uninstalling the deployment revokes every target’s mint; the connection can be removed after that (shc connection rm refuses while a mint is held).

The app’s bats suite binds an s3 connection to a hosted provider, named by the SHC_TEST_S3_* variables (-C mirror.based=…), asserts the rendered settings on pulp-api, and reads the probe’s exit status and log. The probe command itself runs against a stub aws for the refusal paths, the same way external-s3’s discovery probe is tested.

shc’s own artifacts: the rules (SHC-853)

Section titled “shc’s own artifacts: the rules (SHC-853)”

Release tags publish the images in scripts/pulp-images.txt through pulp/shc-images to registry.selfhostedcloud.org/shc/<name>, at both the release tag and the commit tag. This includes Keycloak behind images/keycloak-gate. Pulp waits for that job before syncing any image. Core’s image-publish-record.txt and Keycloak’s keycloak-publish/image-publish-record.txt are separate artifacts; the publisher reads both with repeated --record arguments.

scripts/pulp-carried-images.txt lists catalogue images SHC CI does not build, each with its upstream tag and the digest the catalogue pins. pulp/shc-images passes this list with --carry, using the job’s Pulp credential for the same sync and mirror rule, then reading every digest and tag back anonymously; a tag that no longer names the pinned digest fails the job. pulp/shc-cli clears PULP_CARRY, so the list runs once per release tag.

The gate pulls the recorded Keycloak digest for linux/amd64 and linux/arm64, verifies each layer digest, and scans every layer, including files deleted by later layers. It refuses tls.key paths and PEM private-key blocks with a body in regular files up to 96 MiB. Every keystore found in a layer is checked with that platform image’s own keytool, as root. A PrivateKeyEntry, SecretKeyEntry, unreadable keystore, missing platform, unreadable layer or digest mismatch fails the gate. Bare PEM headers pass. Other .key and .pem paths are reported for review. The job log records rules, layers, paths, sizes and hash prefixes; it never prints key material. The runner needs both platform emulators and its Docker socket available.

The gate proves the recorded digest. The Pulp remote still syncs tags, so release and commit tags must remain immutable between scanning and syncing. Public digest read-back detects a replacement after mirroring; it does not prevent that race.

Clay’s ruling on shape: one R2 bucket per public hostname, Pulp base paths as key prefixes, split by protocol and never by product. shc’s artifacts reach the bucket the same way Based’s do: CI publishes into pulp_file repositories, pulp_mirror copies the chosen distributions, R2 fronts the name. CI never writes the bucket.

.gitlab/ci/pulp-publish.yml fills three repositories through scripts/pulp-publish-shc.py, each behind a guard-free distribution whose base path is its name. Every job needs PULP_URL and PULP_PASSWORD on the corp project (SHC-854), and names the environment pulp, the scope PULP_PASSWORD holds so no other job receives it (SHC-1746). The new line’s tree, shc, is written by a v0.1+ release tag and nothing else (SHC-2557); shc-binaries and shc-apps are the old line’s, written from release/v0.0 and v0.0.* tags, and promanager reads them:

distributionwhat CI lands therefrom
shc<version>/ mirrors generic/shc/<version>/ (deb, rpm, apk, darwin tar.gz, checksums) plus that release’s install.sh; RELEASE names the newest release not on BAD_RELEASES; install.sh at the root is the installer of the release RELEASE namespulp/shc-release (package/release)
shcfrappe/<app>/<version>/<app>-<sha12>.tar, every bench app tarball frappe/docker/Dockerfile pins, checked against its pinned sha256pulp/shc-frappe (frappe/apps)
shc-binaries<version>/ mirrors generic/shc/<version>/ (deb, rpm, apk, darwin tar.gz, checksums); latest/ mirrors the moving channel generic/shc/0.0.0-latest/ (shc_latest_linux_<arch>.deb, shc_latest_COMMIT_SHA); RELEASE names the newest release not on BAD_RELEASESpackage/release, package/snapshot
shc-appsinstall/latest/install.sh and install/<version>/install.sh; apps/latest/index.yaml and index-infra.yaml with every archive: rewritten to the downloads host; the archives themselves at apps/<name>/<version>/<file>publish/installer, apps/index

Every directory carries a SHA256SUMS in sha256sum format.

The bucket is downloads-selfhostedcloud, bound to the pulp app as -C mirror.downloads-selfhostedcloud=cloudflare (the daemon creates it on R2 and attaches the hostname the grant names). The plugin upserts the MirrorTarget from the rendered settings; the fields it carries, as MirrorTargetSerializer declares them:

{
"name": "downloads-selfhostedcloud",
"endpoint_url": "https://<account>.r2.cloudflarestorage.com",
"bucket": "downloads-selfhostedcloud",
"region": "auto",
"path_style": true,
"object_acl": "",
"key_prefix": "",
"credential_name": "downloads-selfhostedcloud",
"public_url": "https://downloads.selfhostedcloud.org"
}

object_acl stays empty on R2 (the bucket policy decides), key_prefix stays empty so the base path is the first key segment, and public_url is the hostname the bucket fronts.

One MirrorRule per distribution, all to that target. pulp/shc-release creates or converges the shc rule itself on every run (it refuses when the target is not bound); the old line’s two were made by hand. Fields as MirrorRuleSerializer declares them; a rule sets exactly one of distribution and base_path_glob, and the plugin refuses both or neither:

Terminal window
curl -u admin -X POST https://artifacts.bitspur.com/pulp/api/v3/mirror/rules/ \
-H 'Content-Type: application/json' -d '{
"name": "shc-binaries",
"distribution": "<href of the file distribution shc-binaries>",
"enabled": true, "dry_run": false, "public": true,
"key_prefix": "",
"targets": [{"target": "<href of downloads-selfhostedcloud>"}]
}'
ruleselectortargetkey_prefixserved at
shcdistribution = file distribution shcdownloads-selfhostedcloud""https://downloads.selfhostedcloud.org/shc/…
shc-binariesdistribution = file distribution shc-binariesdownloads-selfhostedcloud""https://downloads.selfhostedcloud.org/shc-binaries/…
shc-appsdistribution = file distribution shc-appsdownloads-selfhostedcloud""https://downloads.selfhostedcloud.org/shc-apps/…
nextcloud-apps (later)base_path_glob = nextcloud-apps*downloads-selfhostedcloudnextcloudhttps://downloads.selfhostedcloud.org/nextcloud/…

The edge’s public_url, public and grace_seconds are left null so the rule’s and the target’s values apply. A key on the bucket is <target.key_prefix>/<rule.key_prefix>/<base_path>/<relative_path> with empty prefixes dropped, so shc-binaries/latest/shc_latest_linux_amd64.deb in Pulp is shc-binaries/latest/shc_latest_linux_amd64.deb on R2. The nextcloud prefix is the one place a rule prefix is used, because those distributions carry a protocol of their own under one path.

Runs are queued by the plugin on every completed publication (the CI jobs cut one each), on a rule save, on the MIRROR_INTERVAL_SECONDS timer, and by POST /pulp/api/v3/mirror/rules/sync/. Files a publication dropped are retired for the grace period, then pruned; for shc-apps that is what retention on the registry turns into on the bucket, because the repository version is rebuilt to exactly what the index names.

The new line serves every artifact from the shc tree, and its debs from the Based apt repo as well (SHC-2763): pulp/shc-apt puts each release’s amd64 and arm64 debs in the set based-d13 and the Based suite d13, signed by that repo’s key, so a box with the Based repo runs apt install shc, or apt install shc=<ver> to pin. corp’s registry is CI’s internal origin and answers 403 outside, so nothing a customer reads names it:

whatURL
the installerhttps://downloads.selfhostedcloud.org/shc/install.sh
the newest releasehttps://downloads.selfhostedcloud.org/shc/RELEASE
a release debhttps://downloads.selfhostedcloud.org/shc/<ver>/shc_<ver>_linux_amd64.deb
a release deb by apthttps://apt.basedlinux.com/deb/based, suite d13, component main, package shc
a release darwin archivehttps://downloads.selfhostedcloud.org/shc/<ver>/shc_<ver>_darwin_arm64.tar.gz
its checksumshttps://downloads.selfhostedcloud.org/shc/<ver>/shc_<ver>_checksums.txt

The old line, for the estates that still run it: each registry URL has one mirror URL. The registry stays the durable origin; the mirror is what a user can reach with the estate down.

whatregistry (corp project 22)mirror
moving deb, deb_url default…/packages/generic/shc/0.0.0-latest/shc_latest_linux_amd64.debhttps://downloads.selfhostedcloud.org/shc-binaries/latest/shc_latest_linux_amd64.deb
a release deb…/packages/generic/shc/<ver>/shc_<ver>_linux_amd64.debhttps://downloads.selfhostedcloud.org/shc-binaries/<ver>/shc_<ver>_linux_amd64.deb
a release darwin archive…/packages/generic/shc/<ver>/shc_<ver>_darwin_arm64.tar.gzhttps://downloads.selfhostedcloud.org/shc-binaries/<ver>/shc_<ver>_darwin_arm64.tar.gz
the newest releasethe Releases APIhttps://downloads.selfhostedcloud.org/shc-binaries/RELEASE
the installer…/packages/generic/shc-install/0.0.0-latest/install.shhttps://downloads.selfhostedcloud.org/shc-apps/install/latest/install.sh
the apps index…/packages/generic/shc-apps-index/0.0.0-latest/index.yamlhttps://downloads.selfhostedcloud.org/shc-apps/apps/latest/index.yaml

… is https://git.corp.bitspur.com/api/v4/projects/22. The knobs:

  • scripts/install.sh reads only a static tree: RELEASE, then the package and checksum URLs it constructs under SHC_INSTALL_BASE. Unset, that is the new line’s https://downloads.selfhostedcloud.org/shc; set to https://downloads.selfhostedcloud.org/shc-binaries it installs an old-line release. Same denied-release list either way.
  • terraform’s bootstrap.package.deb_url (and every example’s deb_url default) is already a plain URL; the mirror’s latest/ line above is the drop-in alternate.
  • shc repo add shc <index url> takes the mirror’s apps/latest/index.yaml; its archive: lines already point at the mirror.

Flip a consumer only after GET https://downloads.selfhostedcloud.org/shc-binaries/latest/shc_latest_COMMIT_SHA answers 200: the plugin reaches the estate with the SHC-707 daemon roll, and until then the Pulp copies exist at https://artifacts.bitspur.com/pulp/content/shc-binaries/ and …/shc-apps/.

Taking Pulp private: artifacts.corp.bitspur.com (SHC-862)

Section titled “Taking Pulp private: artifacts.corp.bitspur.com (SHC-862)”

Once the mirrors above carry every public artifact, Pulp itself withdraws behind the VPN. On the bitspur estate the corp name is already the deployment’s first host (access: open, with pulp guarding every surface itself) and artifacts.bitspur.com is the second, public one, so the move is a withdrawal, not a rename. It is the LAST step of a strict order:

  1. the daemon roll that makes pulp_mirror live on the estate (SHC-707);
  2. bind the targets (-C mirror.<bucket>=cloudflare) and their rules, let the buckets fill, and prove apt-get update against apt.basedlinux.com with pulp-content stopped;
  3. flip the consumers to the mirrors: BASED_APT_HOST and BASED_DOWNLOADS_HOST on Based CI, SHC_INSTALL_BASE for the installer, the omarchy update-channel script, the docs;
  4. only then, the move below.

Two facts decide what “private” buys. With s3.directDownloads on, every /pulp/content GET answers 302 to a presigned URL on the artifact store, so a private Pulp restricts who can obtain that redirect and nothing about the store: clients behind the VPN still need egress to it, and the store’s own exposure is its own question. And exposure: lan enforces nothing while lanEnforcement is off (bitspur runs it off today): the allowlist has to be armed, and the runners that publish into Pulp have to be on it.

Terminal window
# 1. arm the LAN allowlist and admit the two CI runners' source addresses
# (build01's egress and the Mac runner's), or their VPN addresses.
# -t is the root tenant flag, for this command only.
shc -t corp config set lanAllowlistCidrs '<runner-1>/32,<runner-2>/32' --scope tenant
shc -t corp config set lanEnforcement allowlist --scope tenant
# 2. the content origin every distribution's base_url follows
shc -t corp update pulp --var contentOrigin=https://artifacts.corp.bitspur.com
# 3. the route set becomes the corp name alone. A bare '!' clears every route
# and every published port, so a port this command does not restate is
# removed. The public host's managed
# DNS record goes with the route, and so does its ungated npm metadata
# router, which ce renders per routed host (app.yaml names none)
shc -t corp update pulp --expose '!' --expose artifacts.corp.bitspur.com=pulp-web:8080 --private
# 4. producers follow one variable each: PULP_URL=https://artifacts.corp.bitspur.com
# on the based, shc, packages and cody CI projects; BASED_APT_HOST=
# https://artifacts.corp.bitspur.com/pulp/content on the Based image builds
# (os/mirrors.yaml follows it); PULP_API for pulpview

Verify: an off-estate GET https://artifacts.corp.bitspur.com/pulp/content/ answers 403; the same GET from each runner answers 302 to the store; one publish job on each runner is green. Rollback is the reverse and each step is its own: shc -t corp config set lanEnforcement off --scope tenant opens the router on the next 60 s sweep, --expose artifacts.bitspur.com=pulp-web:8080 re-adds the public route and its record, --var contentOrigin= returns the default, and the CI variables go back.

What stays on Pulp after the move, and therefore behind the VPN: the dev channels, the rpm channels, the OBS build-time mirror, the npm metadata route, and the API itself.