Zoltan Kochan
e5988d13b1
Dependency side effects and `pnpm pipeline` task results now share one artifact format, one trust model, and two transports. `pnpm pipeline` stores cacheable task results as `workspace-task` artifacts of the shared artifact protocol: P-256 signed envelopes whose manifest mirrors a local cache entry (`meta.json` and `outputs/`). A remote artifact is verified against `trustedKeys`, written into the local cache, and restored through the same checks as a local entry. Results of tasks that ran are published in the background when `publish` is on, and the run waits for the uploads before it ends. `ArtifactStore` in pnpm-pnpr-client is either a pnpr server or a server that speaks the Turborepo Remote Cache API. Both answers go through the same client-side verification. On the Turborepo API an artifact is the signed envelope stored under a hash of the owner, the input key, and the compatibility scope from `compatibility_scopes`, and each blob once per owner under a hash of its integrity, mirroring pnpr's resolve and blob routes. The API cannot create conditionally, so a later publication replaces an earlier one. A failed per-key lookup is a miss. Both transports refuse credentials for a server that is neither HTTPS nor loopback. The remote side-effects cache now goes through `ArtifactStore`, so it can use a Turborepo server too. pnpr-client's candidate filter now permits `workspace-task` candidates, which the pnpr server already accepted. `ArtifactSigner` replaces the duplicated signing setup in deps-restorer and the pipeline, and `decode_trusted_keys` moves into the protocol crate. The new `remoteCache` setting holds `url`, `team`, `token`, `org`, and the signing fields, with `PNPM_REMOTE_CACHE_*` overrides. Every field except `org` is machine-only and refused in pnpm-workspace.yaml with ERR_PNPM_WORKSPACE_REMOTE_CACHE_TRUST: `token` is not URL-scoped, so a repository that could name `url` could have the token sent anywhere. `sideEffectsCache.remote` keeps `packages`; its signing fields remain a fallback through `Config::remote_cache_settings()`. Only `remoteCache.publish` publishes pipeline tasks, so a machine that publishes dependency builds does not start uploading task outputs and logs. The section is boxed on `Config` to keep CLI futures under clippy's large_futures limit. Each artifact kind has its own size limits: a dependency build keeps 64 MiB per file and per artifact, and a task result may hold files of up to 4 GiB and 16 GiB in all. Blobs travel between files and servers as streams, so no result is held in memory. On pnpr, `PUT /-/pnpr/v0/artifacts/blob` stores one blob ahead of its envelope and commits it only once its bytes match the integrity, and `HEAD` tells a publisher which blobs to skip. `artifacts.quota` makes pnpr's storage bounds configurable. pnpm-testing-utils gains an in-memory Turborepo Remote Cache server for tests. On pnpr, `artifacts.orgs` declares who may read and who may publish what an organization owns: its signed artifacts, its compiler cache, and its pipeline run records. It replaces the username-equals-organization rule for artifacts, `artifacts.compilerCaches`, and `pipeline.workspaces`. The shared-artifact store asks an `OrgAccess` policy the server builds from it, and `CallerOrgs` replaces the duplicated authorize functions. Pipeline run records are keyed `<org>/<workspace>/<run id>` under a new `.pipeline-runs/v1` namespace; `pnpm pipeline --report` sends `remoteCache.org`.
5.2 KiB
id, title
| id | title |
|---|---|
| compiler-cache | Cargo compilation cache |
Added in: v0.1.0-alpha.11
pnpr's artifact service speaks sccache's
WebDAV backend, so a Cargo compilation cache can be shared between CI and
developer machines. No pnpm workspace or package.json is involved.
Declaring a cache
A compiler cache belongs to an organization the server declares under
artifacts.orgs,
and is named after it:
artifacts:
enabled: true
orgs:
acme:
access: [ci-builder, alice, bob]
publish: ci-builder
Each caller supplies its own pnpr token. access is required for reads and
writes alike; publish additionally gates writes.
This is what lets CI publish while developers only read, even where their tokens otherwise permit registry writes. Read-only token restrictions are enforced as well.
Configuring sccache
Install sccache 0.17.0 or newer with
WebDAV support (sccache --help lists the enabled backends), then set the
environment before starting sccache or invoking Cargo:
export RUSTC_WRAPPER=sccache
export CARGO_INCREMENTAL=0
export SCCACHE_MULTILEVEL_CHAIN=disk,webdav
export SCCACHE_WEBDAV_ENDPOINT=https://cache.example.com/-/pnpr/v0/compiler-cache/acme
export SCCACHE_WEBDAV_TOKEN="$PNPR_TOKEN"
export SCCACHE_WEBDAV_RW_MODE=READ_ONLY
cargo build
sccache --show-stats
PNPR_TOKEN holds that caller's token; keep it in developer credentials or CI
secrets. CI uses its builder account's token and sets
SCCACHE_WEBDAV_RW_MODE=READ_WRITE. The pnpr policy enforces publication
permission independently of that client setting.
SCCACHE_MULTILEVEL_CHAIN=disk,webdav checks disk first, then pnpr, and
backfills disk on a remote hit. SCCACHE_DIR and SCCACHE_CACHE_SIZE configure
the local cache. With sccache's default multilevel write-error policy, a remote
write failure does not fail a successful disk write. A running sccache daemon
keeps its startup configuration, so restart it with sccache --stop-server
after changing these settings, and use separate daemons for projects that need
different cache settings concurrently.
What determines a hit
Reuse depends on sccache's compilation keys: the compiler, target, features,
flags, source inputs, and dependencies. Rust keys in sccache 0.17.0 also include
the absolute compilation directory, so CI and developer builds need matching
source paths, for example a common /workspace mount inside their build
containers, with consistent Cargo registry source paths. SCCACHE_BASEDIRS does
not lift that Rust restriction.
Align toolchains and build profiles for useful hit rates. Rust incremental compilation must be off. System linking is not cached, and procedural macros with undeclared filesystem inputs have correctness limitations. See sccache's Rust support.
sccache 0.17.0 also treats a multilevel cache as read-only when any tier is read-only. Developers can read shared entries, and those hits backfill disk, but newly compiled misses are not added to disk in that mode. Builds still succeed. To cache new local compilations, use a disk-only daemon or a cache the developer may publish to. This is an upstream multilevel-cache limitation.
Storage, limits, and trust
Entries use the configured filesystem or S3 artifact store and the same quota
accounting as side effects, in a separate
compiler-cache namespace. A cache key is immutable: the first successful PUT
wins. SCCACHE_WEBDAV_KEY_PREFIX can select a fresh namespace, though that does
not reclaim old entries.
The experimental limit is 256 MiB per compiler entry. Entries count toward the
artifacts.quota
of their cache name and of the whole artifact store. A compiler cache sharing a name with
a side-effects owner shares its quota. Live compiler entries are not evicted
automatically yet.
Each server accepts at most two concurrent compiler uploads, acquiring capacity
before reading request bodies. Further uploads get 503 with Retry-After: 1.
HEAD inspects object metadata without downloading the entry; GET always
verifies the full content digest.
The WebDAV endpoint trusts pnpr and the accounts allowed to publish. Use HTTPS outside localhost, and grant publication only to trusted builds, keeping untrusted pull-request jobs out of the shared writer credentials. Stock sccache does not verify pnpm's signed side-effects envelopes. pnpr verifies stored compiler entries against a digest of their cache scope, key, and bytes before serving them, which detects storage corruption, not a malicious server or an authorized publisher.
The endpoint implements GET, HEAD, and PUT at
/-/pnpr/v0/compiler-cache/<cache>/<key>, plus PROPFIND for virtual parent
directories. It is the subset sccache uses, not a general WebDAV filesystem.
Local builds keep using ordinary Cargo commands. The pnpm
sideEffectsCache setting governs npm
dependency side effects and does not configure this integration.