Zoltan Kochan
7fb2bfab4a
Part of https://github.com/pnpm/tasks/issues/191. Wrong claims: - cli: drop the nonexistent `add-user --password` flag and document the terminal/stdin password read; `oci-gc --min-age` and `cache-gc --max-age` are the flags (the `-secs` forms are aliases); `--from` defaults to the `storage` key, not an internal field; `info` and `http` log the same access records; only global and server options have PNPR_* variables. - configuration: `packumentTtl` defaults to 5m; the minimal config now sets `auth.htpasswd.file`, and the auth section says accounts live in memory without it or a backend; grouped hosted registries default their org to `<ecosystem>~<name>`; the handshake is served for any surface; the pipeline surface also satisfies "something must be served"; add `package.update_maintainers` to the audit list. - tls-and-reverse-proxy: remove the invalid `publicUrl:` YAML key and the nonexistent `rate_limiting` setting. - backup-and-restore: `pnpr verify` hashes by default (`--check-hashes` does not exist, `--skip-hashes` skips only the hash comparison); `migrate-storage --verify` compares bytes. - webhooks: audit events do not cover publishes or token creation; the example uses a real action and the `actor` field. - cli: the `RUST_LOG` example keeps `pnpr::audit=info`. - install-acceleration: the `--public-url` host is exempt from the private-address refusal. - shared-side-effects-cache: `pnpm pipeline` publishes workspace-task artifacts. Missing facts: `routes.allowedPrivateNetworks`, the upstream `search` key and canonical `maxAge`, built-in `npm.jsr.io` route, relative path resolution, `log` defaults and `type`, `${VAR:-default}` substitution and the full secret-field list, `oci` defaults, `artifacts.quota` bounds, `migrate-storage` destination rules, `--osv false`, `--packument-ttl` default, S3 `forcePathStyle`/`allowHttp` defaults, grouped org layout, OIDC key table, container image section, `backend:` ignoring the htpasswd and token files. Duplicates and links: health probe bodies, reload procedure, and the resolver allowlist now link to their one home; broken `#secret` anchor fixed; links into the pnpm docs use absolute pnpm.io URLs. Stale history: drop "pnpr v1" and the per-paragraph "Since alpha.N" / "Since pnpm 12.x" prefixes, and the contributor-only extensibility text. Also corrects the `as_filter_directive` doc comment, which claimed `http` quiets the rest of the crate.
119 lines
5.2 KiB
Markdown
119 lines
5.2 KiB
Markdown
---
|
|
id: compiler-cache
|
|
title: Cargo compilation cache
|
|
---
|
|
|
|
Added in: v0.1.0-alpha.11
|
|
|
|
pnpr's artifact service speaks [sccache](https://github.com/mozilla/sccache)'s
|
|
WebDAV backend, so a Cargo compilation cache can be shared between CI and
|
|
developer machines. No pnpm workspace or `package.json` is involved.
|
|
|
|
## Declaring a cache
|
|
|
|
A compiler cache belongs to an organization the server declares under
|
|
[`artifacts.orgs`](configuration.md#registry-resolver-and-artifact-surfaces),
|
|
and is named after it:
|
|
|
|
```yaml title="pnpr.yaml"
|
|
artifacts:
|
|
enabled: true
|
|
orgs:
|
|
acme:
|
|
access: [ci-builder, alice, bob]
|
|
publish: ci-builder
|
|
```
|
|
|
|
Each caller supplies its own pnpr token. `access` is required for reads and
|
|
writes alike; `publish` additionally gates writes.
|
|
|
|
This is what lets CI publish while developers only read, even where their tokens
|
|
otherwise permit registry writes. Read-only token restrictions are enforced as
|
|
well.
|
|
|
|
## Configuring sccache
|
|
|
|
Install [sccache 0.17.0 or newer](https://github.com/mozilla/sccache) with
|
|
WebDAV support (`sccache --help` lists the enabled backends), then set the
|
|
environment before starting sccache or invoking Cargo:
|
|
|
|
```sh
|
|
export RUSTC_WRAPPER=sccache
|
|
export CARGO_INCREMENTAL=0
|
|
export SCCACHE_MULTILEVEL_CHAIN=disk,webdav
|
|
export SCCACHE_WEBDAV_ENDPOINT=https://cache.example.com/-/pnpr/v0/compiler-cache/acme
|
|
export SCCACHE_WEBDAV_TOKEN="$PNPR_TOKEN"
|
|
export SCCACHE_WEBDAV_RW_MODE=READ_ONLY
|
|
|
|
cargo build
|
|
sccache --show-stats
|
|
```
|
|
|
|
`PNPR_TOKEN` holds that caller's token; keep it in developer credentials or CI
|
|
secrets. CI uses its builder account's token and sets
|
|
`SCCACHE_WEBDAV_RW_MODE=READ_WRITE`. The pnpr policy enforces publication
|
|
permission independently of that client setting.
|
|
|
|
`SCCACHE_MULTILEVEL_CHAIN=disk,webdav` checks disk first, then pnpr, and
|
|
backfills disk on a remote hit. `SCCACHE_DIR` and `SCCACHE_CACHE_SIZE` configure
|
|
the local cache. With sccache's default multilevel write-error policy, a remote
|
|
write failure does not fail a successful disk write. A running sccache daemon
|
|
keeps its startup configuration, so restart it with `sccache --stop-server`
|
|
after changing these settings, and use separate daemons for projects that need
|
|
different cache settings concurrently.
|
|
|
|
## What determines a hit
|
|
|
|
Reuse depends on sccache's compilation keys: the compiler, target, features,
|
|
flags, source inputs, and dependencies. Rust keys in sccache 0.17.0 also include
|
|
the absolute compilation directory, so CI and developer builds need matching
|
|
source paths, for example a common `/workspace` mount inside their build
|
|
containers, with consistent Cargo registry source paths. `SCCACHE_BASEDIRS` does
|
|
not lift that Rust restriction.
|
|
|
|
Align toolchains and build profiles for useful hit rates. Rust incremental
|
|
compilation must be off. System linking is not cached, and procedural macros
|
|
with undeclared filesystem inputs have correctness limitations. See
|
|
[sccache's Rust support](https://github.com/mozilla/sccache/blob/v0.17.0/docs/Rust.md).
|
|
|
|
sccache 0.17.0 also treats a multilevel cache as read-only when any tier is
|
|
read-only. Developers can read shared entries, and those hits backfill disk, but
|
|
newly compiled misses are not added to disk in that mode. Builds still succeed.
|
|
To cache new local compilations, use a disk-only daemon or a cache the developer
|
|
may publish to. This is an upstream
|
|
[multilevel-cache limitation](https://github.com/mozilla/sccache/blob/v0.17.0/src/cache/multilevel.rs).
|
|
|
|
## Storage, limits, and trust
|
|
|
|
Entries use the configured filesystem or S3 artifact store and the same quota
|
|
accounting as [side effects](shared-side-effects-cache.md), in a separate
|
|
compiler-cache namespace. A cache key is immutable: the first successful `PUT`
|
|
wins. `SCCACHE_WEBDAV_KEY_PREFIX` can select a fresh namespace, though that does
|
|
not reclaim old entries.
|
|
|
|
The experimental limit is 256 MiB per compiler entry. Entries count toward the
|
|
[`artifacts.quota`](configuration.md#registry-resolver-and-artifact-surfaces)
|
|
of their cache name and of the whole artifact store. A compiler cache sharing a name with
|
|
a side-effects owner shares its quota. Live compiler entries are not evicted
|
|
automatically yet.
|
|
|
|
Each server accepts at most two concurrent compiler uploads, acquiring capacity
|
|
before reading request bodies. Further uploads get `503` with `Retry-After: 1`.
|
|
`HEAD` inspects object metadata without downloading the entry; `GET` always
|
|
verifies the full content digest.
|
|
|
|
The WebDAV endpoint trusts pnpr and the accounts allowed to publish. Use HTTPS
|
|
outside localhost, and grant publication only to trusted builds, keeping
|
|
untrusted pull-request jobs out of the shared writer credentials. Stock sccache
|
|
does not verify pnpm's signed side-effects envelopes. pnpr verifies stored
|
|
compiler entries against a digest of their cache scope, key, and bytes before
|
|
serving them, which detects storage corruption, not a malicious server or an
|
|
authorized publisher.
|
|
|
|
The endpoint implements `GET`, `HEAD`, and `PUT` at
|
|
`/-/pnpr/v0/compiler-cache/<cache>/<key>`, plus `PROPFIND` for virtual parent
|
|
directories. It is the subset sccache uses, not a general WebDAV filesystem.
|
|
Local builds keep using ordinary Cargo commands. The pnpm
|
|
[`sideEffectsCache`](https://pnpm.io/settings/build#sideeffectscache) setting governs npm
|
|
dependency side effects and does not configure this integration.
|