mirror of
https://github.com/pnpm/pnpm.git
synced 2026-06-28 09:55:39 -04:00
76587d3def6fabf2ec59458a2d00a56069a29e5b
11750 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
dependabot[bot]
|
76587d3def |
chore(cargo): bump reqwest from 0.13.3 to 0.13.4 (#12215)
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.13.3 to 0.13.4. - [Release notes](https://github.com/seanmonstar/reqwest/releases) - [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md) - [Commits](https://github.com/seanmonstar/reqwest/compare/v0.13.3...v0.13.4) --- updated-dependencies: - dependency-name: reqwest dependency-version: 0.13.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Zoltan Kochan
|
089484aca8 |
perf(pnpr): resolve server-side and fetch tarballs directly (#12232)
## Summary Reworks pnpr from an install/file accelerator into a resolve-only accelerator: - `POST /v1/resolve` resolves against the client-supplied registries and returns a gzipped JSON lockfile response - pacquet/pnpm clients then fetch tarballs normally from registries with their own credentials and existing parallel fetch/integrity paths - pnpr no longer serves package file bytes or store-index rows, so the server-side file diff, file-frame response, grant table, and public-package byte-gating code are removed The follow-up resolution fast paths are included on the new measured path: - repeated public no-lockfile resolves use a bounded in-memory TTL cache - fresh frozen input lockfiles skip the server-side lockfile-only pacquet resolve after verification proves the lockfile is usable - input lockfile verification and the verdict cache are preserved ## Benchmark Integrated benchmark on Linux shows small improvements in all pnpr rows, with the clearest movement in hot restore. This should be treated as an incremental win rather than a large install-speed change. | Scenario | `pnpr@HEAD` | `pnpr@main` | Change | | --- | ---: | ---: | ---: | | fresh restore, cold cache + cold store | `1.677 s ± 0.090` | `1.686 s ± 0.070` | ~0.6% faster | | fresh restore, hot cache + hot store | `492.5 ms ± 18.1` | `521.9 ms ± 33.4` | ~5.6% faster | | fresh install, cold cache + cold store | `1.997 s ± 0.025` | `2.003 s ± 0.038` | ~0.3% faster | | fresh install, hot cache + hot store | `1.211 s ± 0.024` | `1.236 s ± 0.038` | ~2.0% faster | ## Trade-off Going registry-direct means pnpr no longer gates tarball bytes itself. Private package access is enforced by the upstream registry when the client fetches tarballs. Resolution policy still runs server-side: lockfile verification, release-age policy, trust policy, and resolved package selection continue to happen before the client fetches bytes. |
||
|
Zoltan Kochan
|
50cb7af337 |
feat(lockfile): write pnpm-lock.yaml byte-for-byte identical to pnpm (#12223)
* feat(lockfile): write pnpm-lock.yaml byte-for-byte identical to pnpm
pacquet's lockfile writer now produces bytes identical to the pnpm CLI.
The format that pnpm emits comes from `@zkochan/js-yaml`, a fork of
js-yaml with lockfile-specific rendering rules that no general-purpose
Rust YAML serializer reproduces, so this ports the relevant dumper
behavior directly.
- Add `yaml_emit.rs`: a port of the `@zkochan/js-yaml` dumper used for
`pnpm-lock.yaml` — blank lines between top-level / `packages:` /
`importers:` / `snapshots:` entries, single-line flow for
`cpu`/`engines`/`os`/`libc`/`resolution` (block for variations/binary),
single-quote scalar style, `lineWidth:-1`, and the scalar-style chooser
with the implicit-type resolvers that decide when a plain scalar must be
quoted. `serialize_yaml::to_string` now delegates here.
- Port pnpm's `sortLockfileKeys` (deep priority + lexical key sort) into
the emitter, so byte order is independent of struct field order
(`react-dom@x` sorts before `react@x`, matching pnpm's `lexCompare`).
- Emit registry packages as `resolution: {integrity}` only, dropping the
reconstructible tarball URL — port of `toLockfileResolution` plus
`get-npm-tarball-url`, applied at lockfile-build time.
- Stop writing the importer `specifiers:` block; v9 inlines
`{specifier, version}`.
- Populate `engines`/`cpu`/`os`/`libc`/`hasBin`: the registry
`PackageVersion` struct now keeps the picked manifest's extra fields via
a `#[serde(flatten)]` catch-all instead of dropping them.
- Order importer blocks `dependencies` -> `devDependencies` ->
`optionalDependencies`.
Verified byte-identical against `pnpm install --lockfile-only` on
fixtures covering registry deps, scoped names, dev/optional groups,
platform packages, and peer dependencies.
* style(lockfile): satisfy Perfectionist dylint in yaml_emit
Rename single-letter params/closures to descriptive names, use raw
strings for the escape table, add trailing commas on multi-line macro
invocations, and move the inline test module into `yaml_emit/tests.rs`.
* fix(install): detect host node version on fresh hoisted install path
Populating `engines` in the lockfile (so it matches pnpm) means the
hoisted dep-graph walker now runs the installability engine check. The
fresh-install hoisted path hard-coded `host_node: None`, which left
`current_node_version` empty and made `check_engine` throw
`ERR_PNPM_INVALID_NODE_VERSION` for any package declaring `engines.node`.
Detect the host node version (as the frozen path does) whenever a package
carries an installability constraint.
Also update tests for the now-correct pnpm-format output:
- `package_integrity` parses the single-line `resolution: {integrity: ...,
tarball: ...}` flow map (tarball deps render flow, not block).
- the catalog test expects pnpm's single-quoted `specifier: 'catalog:'`.
- accept the `node_modules/.pnpm/node_modules/.bin` snapshot additions --
now created because `hasBin` is populated, mirroring pnpm's bin linking.
* test(tarball): anchor integrity lookup to the YAML key token
Match `integrity:` only when it sits at a key position (preceded by
start-of-line, indentation, `{`, or `,`) so a tarball URL or path that
happens to contain the substring can't masquerade as the field and mask a
genuinely missing `integrity`. Addresses a review note from CodeRabbit.
|
||
|
dependabot[bot]
|
04473e027c |
chore(deps): bump the github-actions group across 1 directory with 10 updates (#12220)
Bumps the github-actions group with 10 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github/codeql-action](https://github.com/github/codeql-action) | `4.35.5` | `4.36.0` | | [docker/setup-qemu-action](https://github.com/docker/setup-qemu-action) | `4.0.0` | `4.1.0` | | [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) | `4.0.0` | `4.1.0` | | [docker/login-action](https://github.com/docker/login-action) | `4.1.0` | `4.2.0` | | [docker/build-push-action](https://github.com/docker/build-push-action) | `7.1.0` | `7.2.0` | | [taiki-e/install-action](https://github.com/taiki-e/install-action) | `2.78.1` | `2.79.14` | | [crate-ci/typos](https://github.com/crate-ci/typos) | `1.46.1` | `1.47.0` | | [codecov/codecov-action](https://github.com/codecov/codecov-action) | `6.0.0` | `6.0.1` | | [cbrgm/mastodon-github-action](https://github.com/cbrgm/mastodon-github-action) | `2.2.0` | `2.2.1` | | [zizmorcore/zizmor-action](https://github.com/zizmorcore/zizmor-action) | `0.5.5` | `0.5.6` | Updates `github/codeql-action` from 4.35.5 to 4.36.0 - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits]( |
||
|
Zoltan Kochan
|
9f002da43f | chore: update lockfile and pnpm (#12211) | ||
|
Juan Picado
|
ae212c8e1b |
feat(pnpr): forward uplink auth token and custom headers to upstreams (#12186)
* feat(pnpr): forward uplink auth token and custom headers to upstreams pnpr now attaches a per-uplink `Authorization` header (derived from the verdaccio-shaped `auth:` block) plus any operator-supplied custom `headers:` on every packument and tarball request it makes to an upstream registry. This unblocks proxying private upstreams (CodeArtifact, GitHub Packages, authed npm Enterprise, private verdaccio) that previously returned 401/403. `auth` supports `type: bearer | basic`, an inline `token`, and verdaccio's `token_env` (`true` -> `NPM_TOKEN`, or a named var); an inline `token` takes priority. A custom `headers.Authorization` overrides the auth-derived one, matching verdaccio's merge order. Tokens/headers resolve once at config load through the existing `EnvVar` seam, so a missing token or invalid header value fails fast as an `InvalidConfig` error rather than a silent unauthenticated request. Implements the "Forward auth.token / custom headers to uplinks" item under Uplinks & caching in the pnpr-verdaccio-parity tracking issue. Ref: https://github.com/pnpm/pnpm/issues/11973 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(pnpr): cover uplink auth/header resolution error branches Add tests for the previously-uncovered branches in resolve_uplink and from_yaml_str flagged by Codecov on the uplink-auth patch: - token_env false resolves no token (config error) - an auth token that is not a valid header value (config error) - an invalid custom header name (config error) - an invalid custom header value (config error) - an unresolved env-var reference is tolerated, not an error Brings config.rs from 92.05% to 96.69% line coverage; the patch's nine missing lines are now fully covered. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pnpr): satisfy perfectionist lints and rustdoc on uplink auth code CI's stricter dylint/perfectionist pass and `cargo doc` (which plain clippy does not run) flagged the uplink-auth changes: - collapse the split `reqwest::` and `crate::` imports to one `use` per crate root (perfectionist::import-granularity = crate) - add the trailing comma to the multi-line `format!` in resolve_uplink (perfectionist::macro-trailing-comma) - drop the intra-doc links from the public `UplinkConfig` docs to the private `UplinkFile`/`resolve_uplink` items, which broke `cargo doc --document-private-items` under -D warnings No behavior change. Verified locally with `cargo dylint --all`, `cargo doc --document-private-items`, clippy, and the config tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(pnpr): redact uplink header values in Debug output UplinkConfig and Upstream both hold a resolved HeaderMap that can carry an Authorization credential (or a secret in a custom header). The derived Debug printed those values verbatim, so a debug log or span could leak them. Replace the derives with hand-written impls that route the map through a RedactedHeaders wrapper, which lists header names with every value rendered as <redacted>. * fix: reject empty uplink auth tokens --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
70554b8677 |
feat(pnpr): config-selectable networked-SQLite auth backend (#12199 phase 3) (#12206)
## What Implements the **auth half** of [#12199](https://github.com/pnpm/pnpm/issues/12199) (phase 3) — making pnpr's remaining per-instance state pluggable so the registry can run as stateless, horizontally-scaled replicas. ### Auth records behind config-selected backends Users + tokens now sit behind narrow async `UserBackend` / `TokenBackend` traits, built once at startup into `Arc<dyn …>` handles (the same build-once pattern #12198 used for the hosted store). Three implementations: - **Local** (default) — today's htpasswd file + SQLite token DB, or in-memory when no file is configured. Unchanged behavior. - **Networked SQLite (libsql / Turso)** — `LibsqlAuth` stores **both** records in one shared database, so several stateless replicas observe a consistent set of users and tokens. The `tokens` table DDL is shared verbatim with the local backend (a DB can migrate between them); users — which the local backend keeps in htpasswd — move into a `users` table. Selected via a new top-level YAML block: ```yaml backend: libsql: url: ${PNPR_LIBSQL_URL} authToken: ${PNPR_LIBSQL_TOKEN} # optional embedded replica for local-fast hot-path reads: replicaPath: ./auth-replica.db syncIntervalSecs: 60 ``` When the block is absent, auth stays on local disk exactly as before. ### Embedded-replica read acceleration Token lookups are on the request hot path, so against a remote primary every read would be a network round-trip. With `replicaPath` set, `LibsqlAuth` builds a libsql **embedded replica**: reads hit a local file that libsql keeps current in the background; writes go to the primary. `syncIntervalSecs` is the freshness knob that bounds token-revocation lag. ### Async access path `identify` / `enforce_access` are now async (a networked lookup is async). `enforce_access` is split into an async `resolve_identity` + a sync `authorize`, so the search endpoint resolves the caller once and authorizes each candidate synchronously (no async-in-`retain`). ### Concurrent-publish guard (cross-cutting follow-up from the issue) Closes the same-instance lost-update window in the three read-modify-write packument flows (publish, dist-tag change, partial-unpublish): a striped per-package lock serializes same-package writers on one instance while letting different packages proceed in parallel. The **cross-replica** half (S3 `If-Match` / ETag CAS) is documented in-code as the remaining piece — the issue files it under "fix when we get there," and it belongs with the multi-writer S3 publish work, not this auth branch. ## Tests All green — `cargo test -p pnpr`: - **176 lib unit tests** incl. new `LibsqlAuth` tests (run against an in-memory libsql DB — same driver + SQL, no server) and `backend.libsql` config-parsing tests (incl. the replica options). - New `concurrent_publishes_of_distinct_versions_all_survive` integration test for the publish guard. - Existing auth_persistence / auth_user_endpoints / auth_publish / server / s3_backend suites pass. - Clean under `cargo fmt`, `clippy`, `RUSTDOCFLAGS=-D warnings cargo doc`, **Dylint perfectionist**, and `taplo`. ## Docs `backend.libsql` (incl. embedded replica) documented in the bundled `config.yaml` and the `pnpr` npm README, mirroring how the S3 backend was documented in #12198. |
||
|
Zoltan Kochan
|
4b4d38361c | chore(release): 11.5.2 (#12207) v11.5.2 | ||
|
Zoltan Kochan
|
cbfeeef328 |
fix: avoid partial package materialization under concurrent installs (#12204)
* fix(importer): avoid partial package materialization under concurrent installs The fast import path destructively emptied the shared virtual-store directory before writing files directly into it. When multiple pnpm installs ran against the same workspace concurrently, one importer could wipe files another had already written; if the survivors included the package.json completion marker, every later install treated the broken directory as complete and never repaired it. The regular path now imports directly only when it can create the target directory exclusively (proving no concurrent importer), and otherwise builds the package in a private temp dir and atomically renames it into place. Close #12197 * fix(importer): clean up exclusively-created dir on failed import When the auto importer probes clone then falls back to hardlink/copy, the failed clone attempt left an empty exclusively-created directory behind, so the retry saw EEXIST and diverted to the staging path instead of writing directly. Since the directory was created exclusively (no other process writes into it), remove the partial result on failure so subsequent same-process method fallbacks can fast-path again. |
||
|
Zoltan Kochan
|
02e1e7760e |
test(registry-mock): read uplinked packuments from pnpr's proxy cache dir (#12205)
#12195 separated pnpr's proxied upstream cache from hosted packages, moving proxied packuments to a `.pnpr-cache` subdirectory of the storage root. The `getIntegrity` test helper still only read the hosted `<storage>/<pkg>/package.json` path, so tests that resolve integrity for packages uplinked from the real npm registry (e.g. store add express, store prune is-negative) failed with ENOENT. Try the hosted location first, then `<storage>/.pnpr-cache/<pkg>/package.json`, keeping the existing retry for the lazy/in-flight cache write. |
||
|
Ruben Nogueira
|
4e740d5562 |
fix(gvs): run dependency build scripts under the global virtual store (#11987)
* fix: unavailable dep * fix(gvs): re-link lifecycle-script bins into the GVS projection The post-lifecycle bin re-link pass used the classic virtualStoreDir path even under the global virtual store, so bins created by the build scripts this fix runs were never re-linked into the GVS projection. Use the same pkgModulesDir helper as the rest of the rebuild path. Also thread enableGlobalVirtualStore through the (recursive) rebuild command opts explicitly, and list pnpm in the changeset. * chore: add prebuild to cspell dictionary * test(gvs): cover rebuilding a shared dependency across workspace projects Adds a multi-project recursive rebuild test under the global virtual store with per-project lockfiles (sharedWorkspaceLockfile: false), which routes through recursiveRebuild's per-project concurrent branch. Both projects depend on the same package, deduped into one shared GVS projection, so the concurrent passes select the same projection directory and exercise the per-projection build lock. Asserts the projection is deduped to one directory and built exactly once. * test(gvs): reword comment to satisfy spellcheck * docs: trim verbose comments to 2-3 lines --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
a6682244cd |
feat(pacquet): port the catalogMode auto-cataloging half (saveCatalogName / catalog writes) (#12202)
* feat(package-manager): port catalogMode auto-cataloging to pacquet Implements the auto-cataloging half of pnpm's `catalogMode` (the gate was ported in #11706): under `catalogMode: strict`/`prefer` (or `--save-catalog[-name]`), `add`/`update` write `catalog:`/`catalog:<name>` to `package.json`, insert the entry into `pnpm-workspace.yaml`, and record the resolved snapshot in `pnpm-lock.yaml`'s `catalogs:`. - config: add `saveCatalogName`; move `save-catalog-name` out of the parity NOT_PORTED list - new crate `pacquet-workspace-manifest-writer`: format-preserving (comment/blank-line/key-order/quote-style) catalog write-back, byte-for-byte with pnpm, on top of yamlpatch/yamlpath/yaml_serde - lockfile: add `catalogs:` snapshot (CatalogSnapshots/ResolvedCatalogEntry) - catalog_mode: `decide_catalog` per-dep decision (gate + auto-catalog) - add: parse `pkg@version`; `--save-catalog`/`--save-catalog-name` flags - update: route the catalog gate through the decision core, handling `--latest` on catalog deps Closes #12196 * fix(package-manager): address review feedback on catalog auto-cataloging - update: gate the `pnpm-workspace.yaml` write on `save` so `--no-save` persists nothing to disk, matching pnpm's `if (opts.save !== false)` - workspace-manifest-writer: detect each block's child indent dynamically instead of assuming two spaces - tests: assert the lockfile `catalogs:` snapshot in the named-catalog `update --latest` e2e test; add `--no-save` and four-space-indent cases |
||
|
Zoltan Kochan
|
69354b288f |
chore(release): key changeset-released ledger by target branch (#12201)
Releases now land via a PR from an ad-hoc branch rather than a commit pushed straight to the target, so `bump.ts` keyed the ledger off the ephemeral PR branch and scattered each release into its own file instead of accumulating in `main.txt`. Recover the target from a `release-pr/<target>` branch name, add a manually-dispatched workflow that creates such a PR, and drop the accumulated per-PR ledger files (verified inert: no overlap with pending changesets, no live branch reintroduces a released changeset on merge). |
||
|
Zoltan Kochan
|
8e5e764037 |
feat(pnpr): store hosted packages in an S3-compatible object store (#12198)
## What
Lets pnpr store its **hosted** packages (the ones published to it, plus static-served content) in an **S3-compatible object store** instead of a local directory. Because the same code targets any S3-compatible endpoint, this also covers **Cloudflare R2**, MinIO, Backblaze B2, Wasabi, etc.
The local `tokio::fs` path remains the default — nothing changes unless you add the new `s3:` config block.
## Why
The hosted store is pnpr's source of truth: durable, must be backed up, and can't be regenerated. That's exactly what belongs in object storage:
- The provider handles durability/replication, so there's no single-node volume to back up.
- Multiple **stateless pnpr replicas** can share one hosted store.
- R2 is the S3 API, so a configurable `endpoint` gets it (and the other S3-compatibles) for free.
The disposable proxy cache and the install-accelerator SQLite stores deliberately **stay on local disk** — they're ephemeral, latency-sensitive, and streamed/locked in filesystem-shaped ways.
## How
- New `s3.rs` module: `S3Settings` (the YAML `s3:` block), a client builder (`object_store` crate; AWS-env credentials with explicit override, plus R2/MinIO/path-style/HTTP knobs), and an `S3Store` adapter (packument get/put, streaming tarball get, staged upload, delete, prefix-scoped list).
- `storage.rs`: a `HostedStore { Fs | S3 }` backend enum routes the hosted ops; the `cached` store stays fs-only. Publish stages the decoded+verified tarball to local scratch, then finalize either renames (fs) or uploads (S3). `open_tarball` now returns a streaming response body so S3 reads stream straight through.
- `config.rs`: parses `s3:` and builds the client once at config-load time (the only fallible step), so `Storage` construction stays infallible.
- `search.rs`: local search now lists package names through the storage abstraction, so it works against a bucket too.
- Documented (commented) in the bundled `config.yaml`.
### Example: Cloudflare R2
```yaml
storage: ./storage # still backs the local proxy cache + upload staging
s3:
bucket: my-pnpr-packages
region: auto
endpoint: https://<account-id>.r2.cloudflarestorage.com
accessKeyId: ${PNPR_S3_ACCESS_KEY_ID}
secretAccessKey: ${PNPR_S3_SECRET_ACCESS_KEY}
```
|
||
|
shiminshen
|
e7e99f04e4 |
fix: do not crash when a catalog specifier is a range (#11706)
## Summary `pnpm update --recursive --lockfile-only <pkg>@<version>` crashed with `Invalid Version: <range>` when the catalog entry for `<pkg>` was a range (e.g. `^21.2.10`) and `catalogMode` was `strict` or `prefer`. This is the exact command Renovate's pnpm artifact updater runs; monorepos using `catalog:` with any range specifier were blocked from Renovate-driven lockfile updates. **Root cause:** in `installSome`, the catalog-match short-circuit guards `semver.eq(wantedDep.bareSpecifier, catalogDepSpecifier)` with `semver.validRange` on both sides. `validRange` returns truthy for ranges, but `semver.eq` constructs `new SemVer(...)` internally and throws on a range. **Fix:** use `semver.valid` instead of `semver.validRange` on both sides of the equality guard. Range specifiers now fall through to the existing mismatch handling (`CatalogVersionMismatchError` in `strict` mode, warn-and-use-direct in `prefer` mode) instead of crashing. Behavior for concrete-on-both-sides is unchanged. Closes #11570 ## Behavior after the fix This turns a crash into pnpm's normal catalog-mismatch handling; it does **not** make a strict-mode update succeed when the catalog is a range: - **`catalogMode: strict`** — rejects with `ERR_PNPM_CATALOG_VERSION_MISMATCH` (clean, actionable error instead of a stack trace). - **`catalogMode: prefer`** — warns and uses the direct version. - **concrete-vs-concrete** — unchanged (`semver.eq` still runs). ## pacquet parity The TypeScript fix patches a crash inside pnpm's `catalogMode` mismatch gate — a feature pacquet had not ported at all (`catalog-mode` was in the config parity test's `NOT_PORTED` list). Rather than just the one-liner, this PR ports that gate to pacquet so the two stacks match: - **config:** new `CatalogMode { Manual, Strict, Prefer }` enum (default `manual`), `Config.catalog_mode`, wired through `pnpm-workspace.yaml` (`catalogMode:`) and the env overlay; `catalog-mode` moved from `NOT_PORTED` to a mapped row in the `pnpm_default_parity` contract test. - **package-manager:** `check_catalog_mode` + `CatalogVersionMismatchError` (`ERR_PNPM_CATALOG_VERSION_MISMATCH`), invoked from `update` before the manifest is mutated. The comparison only treats both sides as equal when each parses as a concrete semver version, so a ranged catalog entry falls through to the mismatch path instead of reaching an exact-version comparison — the Rust analogue of the `semver.valid` guard above. The crash itself can't occur in pacquet (Rust's `node-semver` returns a `Result` rather than throwing); the port is the *feature* with the range-correct comparison built in, so pacquet behaves like fixed pnpm. **Not ported** (the surrounding pieces pacquet still lacks, so wiring them would diverge from pnpm rather than match it): the `add`-path cataloging that relies on `defaultCatalog` rewriting, and the `saveCatalogName` → `pnpm-workspace.yaml` auto-cataloging half. The gate is therefore wired into `update <pkg>@<version>` / `--latest` (the Renovate scenario), not `add`. |
||
|
Zoltan Kochan
|
43ad0941a1 |
feat(pnpr): separate the proxied upstream cache from published packages (#12195)
## What Splits pnpr's on-disk storage into two physically separate roots so the disposable proxy cache and the authoritative **hosted** packages no longer share a lifecycle. Closes #12194. ### Before Proxied upstream packuments/tarballs and locally-published packages were written to the same `<storage>/<pkg>/` tree through a single `Cache` abstraction, with no marker distinguishing them. Consequences: - No safe way to clear the proxy cache — deleting a package dir removed hosted packages too. - Hosted packages shared a lifecycle with disposable cache; a naive "clear the cache" could wipe the source of truth. - Backups and upgrades had to treat the entire (reconstructible) mirror as precious data. ### After | | `storage` (hosted) | `cache` (proxy) | |---|---|---| | Holds | packages this server hosts directly (published via its API) + static-served content | proxied upstream mirror + install-accelerator store | | Durability | source of truth, never overwritten by an upstream refresh | safe to wipe anytime; self-heals on next request | | Default | `./storage` | `<storage>/.pnpr-cache` | | Override | `storage:` / `--storage` | `cache:` / `--cache` (point at a separate ephemeral volume) | - A new `Storage` type wraps two `Store` roots — `hosted` (authoritative) and `cached` (disposable). Reads prefer the hosted store; a hosted/static packument is served as-is and never refreshed over. - Publish, unpublish, packument PUTs and dist-tag changes write to the hosted store; upstream refreshes write only to the cache. - Removal clears both stores — full unpublish *and* partial (single-tarball) unpublish — so a stale proxied copy can't resurface via the tarball-read fallback. - The install-accelerator CAS + verdict DB move under the cache root. ### Naming - Internally the two roots are the `hosted` / `cached` fields of the `Storage` type (`hosted`/`proxy` is the registry-server convention, e.g. Sonatype Nexus). - The user-facing YAML keys stay `storage:` / `cache:` (nouns that name directories, the clearest register for a setting), which also keeps verdaccio-shaped configs working. ### Server / deployment - Put `storage` on a durable, backed-up volume and `cache` on scratch/ephemeral disk (or just leave the default subdir). - **Upgrades retain hosted packages trivially**: point the new server at the same `storage`; the cache can start cold. - **DR**: back up only `storage`. |
||
|
Zoltan Kochan
|
5192edf40e |
feat(pnpr): forward credentials and add per-user access grants for external private registries (#12184) (#12189)
Closes #12184 (part 2). #12181 shipped the per-caller access gate on `POST /v1/install`, which authorizes every served package against pnpr's own `packages:` policy — the complete answer **while pnpr fetches anonymously**. This PR adds the remaining piece: forwarding the caller's per-registry credentials so the accelerator can resolve/fetch **external private** content as the caller, and gating that content per user against the registry that actually owns it. ## Credential forwarding (issue steps 1–2) - **Wire:** `POST /v1/install` gains an `authHeaders` body map (`{ "//host/path/": "Bearer …" }`, the shape `AuthHeaders::from_map` consumes / `getAuthHeadersFromCreds` produces) plus an HTTP `Authorization` header. The body map carries the *upstream* registry tokens; the header identifies the caller to pnpr's own gate and keys the grant table. - **pacquet plumbing:** a request-scoped `Arc<AuthHeaders>` is threaded via a new `Install.auth_override` field and an `auth_override` param on `build_resolution_verifiers`, so resolution/verification run as the caller **without** baking per-user auth into the interned `&'static Config` (which would leak one config per user). - **Server:** `handle_install` builds the per-request `AuthHeaders` and threads it through resolve, verify, and `fetch_uncached` (which now returns the freshly-fetched set). - **Clients:** pacquet `pnpr-client` and `@pnpm/pnpr.client` send `registry` / `namedRegistries` / `authHeaders` + `Authorization`; the TS path sources them from the caller's registry credentials via `@pnpm/network.auth-header` (`getAuthHeadersFromCreds` is newly re-exported). `@pnpm/worker` is unchanged — downloads happen server-side. - **Credential scope:** both clients forward the caller's *full* credential map, not a subset scoped to the declared registries. The registries a dependency graph touches aren't knowable up front — a transitive package can be scope-routed to another registry or pinned to a tarball URL on a host that's in `.npmrc` but isn't a declared registry — so pnpr attaches the right token per fetched URL exactly as a local install does. These are package-fetch credentials going to the very service the caller configured to fetch its packages. ## Per-user grant table (issue steps 3–4) Externally-resolved private content carries no pnpr policy, so the store's possession of the bytes must not authorize a user the upstream never cleared. A served package is dispatched by **whether a forwarded credential was used to fetch it**: - **No forwarded cred → pnpr-as-authority:** the existing local `packages:` policy check, unchanged. - **Forwarded cred → upstream-as-authority:** gated against a persistent `(user, name@version)` grant table (SQLite, modeled on `VerdictCache`). Freshly fetched this request ⇒ record + allow (the upstream just accepted the token). Cache hit with a standing grant ⇒ allow, no upstream trip. Cache hit, no grant ⇒ re-verify against the owning registry with the caller's credential — record on success; **clear-on-discovery** (purge the user's grants for the package) + deny on `401`/`403`. TTL is the `installAccelerator.grantTtl` config knob (default: permanent). ## Public vs private (no per-user gating for public packages) A forwarded credential matching a registry doesn't mean a package is *private* — in a mixed proxy (one registry serving a company's private packages **and** public ones), the token matches everything, and gating public content per user would cost a grant row and a re-verify round trip per user for bytes anyone may read. So before the per-user path, a not-yet-classified cache hit is probed **anonymously**: a `2xx` classifies the package public in a global set (no user pays for it again, no grant, no further round trip); a `401`/`403` means it's genuinely private and falls through to the grant / re-verify path above. Public packages thus cost **one anonymous probe across the whole fleet**, not one per user. ## Tests - pnpr: grant-table + public-set mechanics, regime dispatch, the upstream-authorization paths (fresh-fetch, granted cache hit, private re-verify-and-record, denied-clears-grants, public-classified-once-then-free), and forwarded-cred-routes-around-local-policy. - pacquet `pnpr-client`: a test asserting `authHeaders` + `Authorization` travel on the wire. - Full suites green: `pnpr` (237), `pacquet-package-manager` (389), `pacquet-pnpr-client` (12), `pacquet-network`/`config` (325); clippy `-D warnings`, `cargo fmt`, rustdoc `-D warnings --document-private-items`, `typos`, and the TS compile all clean. ## Scoped follow-ups (not in this PR) - Clear-on-discovery fires at the re-verify hook only. A `401`/`403` during the cold resolve aborts the request anyway (nothing is served); threading the offending package out of the deep resolve error to also clear stale grants for *future* requests needs structured auth errors. - Per-scope external registries route via the default registry, since pacquet doesn't yet surface `@scope:registry` routing in `collect_packages`. |
||
|
Zoltan Kochan
|
2bcd69fd8c | chore: update lockfile, Node.js, pnpm, and pacquet versions (#12157) | ||
|
Zoltan Kochan
|
a358ee09ab |
fix: don't catalog runtime: dependencies under strict catalog mode (#12188)
A `runtime:` specifier (e.g. node from `devEngines.runtime` or `pnpm runtime set`) round-trips to `devEngines.runtime`, which only recognizes the `runtime:` protocol. Under `catalogMode` strict/prefer the auto-save loop promoted it into a catalog and rewrote the manifest entry to `catalog:`, which broke that round-trip and stranded it in `devDependencies`. Skip `runtime:` specifiers in that loop. |
||
|
Zoltan Kochan
|
3b76b8eaed |
fix(pnpr): access-gate install-accelerator files and remove unauthenticated /v1/files (#12181)
* fix(pnpr): authorize served packages against pnpr's policy in /v1/install A content-addressed digest in the install-accelerator store is shared across packages and says nothing about access, so the store's possession of a package's bytes is not a capability to receive them. `/v1/install` served files for any package found in the store, including ones reached only on the cache-hit / frozen-lockfile path where no access check happened — letting a caller who knows a private package's digest pull bytes the registry routes would 401 on. Check every served package against pnpr's own `packages:` policy before serving — the same decision `serve_packument` / `serve_tarball` make, in process, with no network round trip (so a warm shared server keeps its resolution advantage). `serve_install` resolves the caller's identity from `Authorization`; `deny_unauthorized_packages` denies the install (401 anonymous / 403 authenticated-but-outside-the-allowed-set) when any served package is not readable by the caller. This authorizes against pnpr's own surface, the authority for everything the store can hold today (pnpr fetches anonymously, so cached content is pnpr-hosted or publicly fetchable). When credential forwarding lands, packages the client resolved from external registries under its own token carry no pnpr policy and will need per-caller re-verification against the owning registry (TTL-cached) — noted at the check and tracked in #12184. The raw `/v1/files` endpoint is still unauthenticated; removing it (it is superseded by the inline single-response path) is a follow-up (#12184) that also ports the TS `@pnpm/pnpr.client` + worker off the two-trip path. --- Written by an agent (Claude Code, claude-opus-4-8). * fix(pnpr): remove the unauthenticated /v1/files endpoint `POST /v1/files` served any CAFS file by digest with no authentication and no package identity, so the access gate on `/v1/install` (which is per package) couldn't cover it — it had to be removed, not gated. It was already superseded by the single-response inline path (#12178). * Server: `/v1/install` always answers with the inline gzipped body (lockfile + stats + store-index entries + the missing files' contents); the NDJSON two-trip path, the `/v1/files` route, `handle_files`, and the `FilesRequest`/`is_valid_sha512_hex` helpers are gone. * TS client + worker: `@pnpm/pnpr.client` now does the one inline request and hands the file frames to `@pnpm/worker`'s `writeCafsFiles`, which writes them to the CAFS; the `fetchAndWriteCafsFiles` /v1/files fetcher is replaced. Error bodies are decompressed before being surfaced, since the server also gzips its JSON error responses (e.g. an access denial). Verified end to end by `pnpm/test/install/pnpmRegistry.ts` (11 tests: install / add / remove / workspace through a real pnpr server). Closes the second half of the install-accelerator access work (#12184); file-bearing responses are now both inline-only and access-gated. |
||
|
Zoltan Kochan
|
f06ab5e201 |
perf(pnpr): collapse the install-accelerator cold path to one round trip (#12178)
Reduce the cost of a pnpr install-accelerator install against a remote
server, where latency — not bandwidth — dominates.
The cold path was three sequential round trips: a GET /-/pnpr handshake,
a POST /v1/install that returned the lockfile + missing-file digests as
NDJSON, and a POST /v1/files that fetched the file contents. At 100ms RTT
most of an install is spent waiting on trips, not transferring data.
- A new `inlineFiles` request flag makes POST /v1/install return a single
gzipped body: a length-prefixed JSON header (lockfile, stats,
store-index entries, or verification violations) followed by the
missing files' contents as the same binary frames /v1/files serves.
The pacquet client drops the handshake, sends the flag, and writes the
inlined files straight to its CAFS — no follow-up fetch. The legacy
NDJSON + /v1/files path is unchanged for clients that don't set it.
- File-bearing responses now compress at gzip level 6 (was level 1),
~16% smaller payload at negligible CPU.
Measured at 50ms one-way latency (warm server, 5-run average): a
single-round-trip install drops from ~381ms to ~135ms (≈2.8x).
Two further experiments from the issue were evaluated and not adopted:
server-side fetch/transfer streaming (the public npm registry is
CDN-backed and latency-bound, not bandwidth-throttled, so there is
nothing to overlap — it measured slower under an artificial bandwidth
cap) and bloom-filter integrity uploads (real but niche, and false
positives would require a correctness-critical re-fetch fallback).
Closes #12165
|
||
|
Zoltan Kochan
|
69cfcb7417 |
perf(ci): cache benchmark binaries per commit instead of rebuilding (#12173)
The integrated-benchmark "Precompile benchmark revisions" step took ~14 minutes every run. Two causes: 1. The "Cache Rust builds" step cached the multi-GB `bench-work-env/*/pacquet/target` dirs under a 1-minute restore timeout. A restore that large never finished in 60s, so (with `continue-on-error`) the cache silently missed and every run built all four targets cold. 2. `pacquet@HEAD` and `pnpr@HEAD` resolve to the same commit but built in separate clones, compiling the `pacquet` binary twice (same for main). Cache the compiled binaries per *resolved commit* instead: - Orchestrator: a `--reuse-prebuilt-binaries` flag skips the clone + `cargo build` for a target whose output binary is already present (i.e. restored from cache). Targets are built pnpr-first; since a `pnpr@<rev>` build also produces the `pacquet` client binary, a same-revision `pacquet@<rev>` reuses it by copy rather than recompiling the commit. - Workflow: resolve the HEAD/main SHAs, then cache the two `pnpr@<rev>` binaries keyed on the commit (they cover all four targets via the dedup-copy). `main` is a near-certain hit on PRs (stable SHA) and a same-HEAD re-run hits HEAD too, so only a fresh HEAD compiles. Drop the giant `bench-work-env/*/pacquet/target` cache (the small binary caches restore in seconds, with no eviction risk) and keep a cargo-deps + orchestrator-target cache with a realistic 3-minute timeout. A fresh-HEAD run now compiles one workspace once (~half the old work); re-runs and main reuse cached binaries and skip compilation entirely. |
||
|
Zoltan Kochan
|
3492bb8f4a |
feat(pnpr): gzip-compress package metadata over the wire (#12170)
Neither side used gzip for package metadata: pacquet fetched packuments uncompressed from every registry (unlike pnpm-TS, which gets gzip via undici), and pnpr served them uncompressed. Packuments are the largest payloads pulled during resolution and gzip ~5-10x, so this was a real resolution-time cost and a divergence from how a CDN-fronted registry behaves. Closes #12169. Both halves are needed and land together: - Client (pacquet): enable reqwest`s `gzip` feature and set `.gzip(true)` explicitly on the network client builder, so it sends `Accept-Encoding: gzip` and transparently decompresses. Tarballs are unaffected (served as `application/octet-stream` with no `Content-Encoding`, so reqwest leaves them alone and store-integrity verification is unchanged). It also transparently handles the install accelerator's already-gzipped `/v1/files` stream — the client's existing magic-byte check covers the now-auto-decompressed case. - Server (pnpr): add a `tower-http` `CompressionLayer`, scoped to JSON via `NotForContentType` so it compresses packuments / version manifests / dist-tags / search but never re-gzips an already-compressed payload: tarballs (`application/octet-stream`), the file stream (`application/x-pnpm-install`), or the resolve NDJSON (`application/x-ndjson`). pnpr is commonly hit directly with no CDN or nginx in front, so the application is the only layer that can compress; where a proxy/CDN is present, the `Content-Encoding: gzip` is passed through (no double compression). Tests assert a packument is gzipped when `Accept-Encoding: gzip` is sent, served plain otherwise, and a tarball is never re-gzipped. |
||
|
Zoltan Kochan
|
65c9bef283 |
ci(pnpr): inject network latency into the install-accelerator benchmark (#12166)
## What The pnpr install accelerator is a **remote** server, but the integrated benchmark ran it on **loopback** (RTT ≈ 0), which hides the round-trip cost that dominates a real install — and that pnpr exists to reduce. This injects network latency so the benchmark measures pnpr as the remote service it is in production. ## How A dependency-free, synchronous latency-injecting TCP proxy (`latency_proxy`) plus two knobs on `integrated-benchmark`: - **`--pnpr-latency-ms`** — fronts each `pnpr@<rev>` server, so the client↔server link pays the given round trip (half each direction). - **`--registry-latency-ms`** — fronts the registry for the direct (`pacquet`/`pnpm`/`--with-pnpm`) targets, so a direct install crosses the same network. `pnpr@<rev>` targets keep a **direct (fast) registry link** — that models a warm, colocated server, so pnpr's advantage shows up as **fewer round trips, not a faster backend**: ``` direct target: client → [latency proxy] → registry pnpr target: client → [latency proxy] → pnpr server → (direct) → registry ``` The workflow sets both equal (`50ms`) so the in-run pnpr-vs-direct ratio is fair and the `pnpr` Bencher testbed (pnpr@HEAD vs pnpr@main) becomes **sensitive to protocol round-trip-count changes** — which is what makes the upcoming protocol work (collapsing the 3-round-trip handshake/install/files flow) measurable on main. See #12165 for that plan. ## Notes - **Latency only, no bandwidth cap.** The public registry is CDN-backed and CI runners are fast, so install time is latency/round-trip bound, not throughput bound — a bandwidth cap would be overly pessimistic. A high-ceiling, opt-in bandwidth knob can follow if a slow-link scenario is ever wanted. - Both flags **default to `0`** (current behavior unchanged); the registry proxy is also skipped in `--registry=npm` mode (already remote). - The proxy is unit-tested (a round trip through it reflects the injected latency). `cargo check`/`clippy`/`fmt`/`dylint` clean. - One caveat the proxy does **not** model: TLS-handshake round trips and HTTP/2 multiplexing of a real CDN — it reproduces propagation delay, the dominant and relevant factor here, not a byte-exact replica of registry.npmjs.org. |
||
|
Zoltan Kochan
|
e1648a6ca0 |
perf(pnpr): coarsen packument time precision to shrink abbreviated responses (#12168)
pnpr serves abbreviated packuments uncompressed, so the verbatim `time` map is pure wire cost. Drop precision the resolvers don't need: seconds come off every entry, and entries older than a week lose the time-of-day entirely (down to a bare date). The reserved `unpublished` object and any non-RFC-3339 value pass through untouched. Timestamps are rounded *up* (next minute / next day, leaving values already on the boundary untouched), so a coarsened value is never earlier than the real publish time. `minimumReleaseAge`, the abbreviated-modified shortcut, and the trust checks can therefore only ever read a version as newer than it is — the fail-safe direction; a too-new version is never coarsened into looking mature. Both reduced forms are accepted by pnpm's lenient `new Date(...)`. pacquet parsed strict RFC 3339 only, so add a shared `parse_packument_timestamp` in resolving-resolver-base that also accepts minute precision (`2024-03-15T09:42Z`) and bare dates (`2024-03-15`, read as midnight UTC), and route every existing publish-timestamp parse site through it. |
||
|
Zoltan Kochan
|
6305e955c6 |
perf(pnpr): shrink the abbreviated packuments served to clients (#12163)
## What When pnpr proxies a registry, trim the `application/vnd.npm.install-v1+json` packument down to only the fields the pnpm and pacquet resolvers actually read, so clients download, parse, and cache less metadata. Only the abbreviated path (`Accept: application/vnd.npm.install-v1+json`) is affected; full-document clients are untouched. ### Dropped (never read during resolution) - **top-level:** `readme`, `readmeFilename`, `_id`, `_rev` — `readme` is the dominant per-packument bloat (full README text) and npm's own abbreviated format never carried it. - **per-version:** `funding`, `devDependencies`, `acceptDependencies`, `_hasShrinkwrap`. A dependency's `devDependencies` are never installed, so the resolver has no use for them. - **per-version `dist["npm-signature"]`** — npm's deprecated PGP detached signature. npm stopped populating it years ago and nothing in pnpm or pacquet reads it. - **per-version `dist.fileCount`** — read nowhere in pnpm or pacquet. - **per-version `dist.unpackedSize`** — read only by `pnpm view`, which fetches the full metadata document (`fullMetadata: true`) that pnpr serves unstripped. - **per-version `dist.shasum` when `dist.integrity` is present.** Both pnpm (`getIntegrity`) and pacquet prefer SRI `integrity` and only fall back to the legacy sha1 `shasum` when `integrity` is absent, so shipping both is a redundant hash on every version. `shasum` is **kept** when `integrity` is absent (pre-2017 publishes) so the `getIntegrity` fallback still has a hash. ### Deliberately kept - **`time`** (top-level publish timestamps). npm's own abbreviated form omits it, but pnpr retains it because pnpm/pacquet read it for the `minimumReleaseAge` check. - **`dist.signatures`** (ECDSA registry signatures). It binds `name@version:integrity` to the upstream registry's signing key and survives pnpr's `dist.tarball` rewriting (the signature covers the triple, not the URL). Nothing verifies it at install time today — `pnpm audit signatures` fetches its own *full* metadata — but keeping it leaves the door open to an optional client-side install-time registry-signature check, which is most valuable precisely on the pnpr path (an extra trust hop). - **`dist.attestations`** — read by pacquet's `trustPolicy` verifier. ### Fixed along the way Per-version **`libc`** is now forwarded alongside `os`/`cpu` — it was previously stripped. pnpm reads `libc` for optional-dependency platform filtering ([#9950](https://github.com/pnpm/pnpm/issues/9950)), so omitting it produced wrong installs through pnpr. ## Impact (measured) Compact JSON size of the abbreviated packument, before vs. after this PR, on real registry metadata: | Package | Versions | Before | After | Reduction | |---|---:|---:|---:|---:| | typescript | 3760 | 8379 KB | 1985 KB | **76%** | | webpack | 875 | 1974 KB | 824 KB | **58%** | | @types/node | 2336 | 2249 KB | 995 KB | **55%** | | chalk | 43 | 46 KB | 21 KB | **53%** | | react | 2817 | 2735 KB | 1509 KB | **44%** | | express | 288 | 331 KB | 240 KB | **27%** | | lodash | 117 | 67 KB | 49 KB | **26%** | | **aggregate** | | **15.4 MB** | **5.5 MB** | **64%** | Where the savings come from (react, % of the before size): | Field dropped | Share | |---|---:| | `dist["npm-signature"]` | 36.0% | | `dist.shasum` (integrity present) | 5.2% | | `dist.unpackedSize` | 2.1% | | `dist.fileCount` | 1.4% | | `funding` / `devDependencies` / etc. | <0.1% | `dist["npm-signature"]` dominates. (`dist.signatures`, kept, is a further ~18% that this PR intentionally leaves in place.) **Methodology note:** the table simulates the per-version and `dist` trims against npm's *already-abbreviated* metadata, which does not contain top-level `readme`/`_id`/`_rev`. pnpr fetches the *full* upstream document and abbreviates it, so the real reduction is at least the figures above **plus** the full README text that is additionally dropped at the top level. The win lands on the proxy/registry path (where the client still resolves locally); it does not affect the `/v1/install` accelerator path, where the client never receives a packument. ## Follow-up Dropping `dist.tarball` (a further ~8% on react) is tracked separately in [#12164](https://github.com/pnpm/pnpm/issues/12164) — it first needs pnpm and pacquet to reconstruct the URL when it is absent. ## Safety - Verified against the pnpm TS resolver: none of the dropped fields are read during resolution; `getIntegrity` returns `integrity` outright when present; `dist.signatures` / `npm-signature` / `unpackedSize` are consumed only by commands (`audit signatures`, `view`) that fetch their own full metadata, which pnpr serves unstripped. - pacquet's `PackageVersion` has `#[serde(default)]` on `dev_dependencies` and ignores `shasum` / signature / size fields, so a pacquet-as-client deserialization stays valid. |
||
|
Zoltan Kochan
|
a017bf3394 |
refactor: rename the agent client and agent setting to pnpr (#12155)
* refactor: rename the agent client + setting to pnpr The pnpm-side client and its config setting still carried the old "agent" name after the server moved to pnpr. Align both with pnpr (and with pacquet, which already uses `pnprServer`): - Move `agent/client` → `pnpr/client` and rename the package `@pnpm/agent.client` → `@pnpm/pnpr.client` (exported `AgentProject` type → `PnprProject`). - Rename the config setting `agent` → `pnprServer` (`--pnpr-server` CLI flag), matching pacquet's setting name. - Rename the internal install-path symbols and the user-facing log / error strings that mentioned "pnpm agent" to "pnpr". No behavioral change — only names. The e2e suite now drives `--config.pnprServer`. * fix: forward optionalDependencies to the pnpr server `PnprProject` and the install-request body only carried `dependencies` and `devDependencies`, so a project's `optionalDependencies` were dropped on the way to the pnpr server — it resolved as if they didn't exist, producing a different lockfile than the local resolver. Thread `optionalDependencies` through the client request shape, the deps-installer single-project and workspace request builders, and the pnpr server (`InstallRequestProject` / `InstallRequest` + the throwaway manifest it writes for resolution). Adds an e2e case asserting an optional dependency is resolved through `pnprServer`. |
||
|
Zoltan Kochan
|
930c9d7718 |
ci(pnpr): benchmark the install accelerator (new Bencher pnpr testbed) (#12154)
* ci(pnpr): add pnpr@<rev> target + Bencher testbed for the install accelerator
Measures the pnpr-accelerated install path end to end. A new `pnpr@<rev>`
target in the integrated-benchmark orchestrator builds both the `pacquet`
client and the `pnpr` server from the revision's monorepo clone, boots a
per-target pnpr server with an isolated `--storage`, and points the client
at it via `PNPR_SERVER`.
Reusing the existing multi-target hyperfine model gives both comparisons:
- `pnpr@HEAD pacquet@HEAD` -> pnpr-vs-direct ratio in one run (same client,
with and without the accelerator).
- `pnpr@HEAD pnpr@main` -> regression delta tracked in a new Bencher `pnpr`
testbed.
Two CI workflows mirror the fork-safe two-stage pacquet pattern, triggered
on pnpr/**, pacquet/crates/pnpr-client/**, and pacquet/crates/config/**
(the pnprServer plumbing), running the hot-cache/hot-store restore and
fresh-install scenarios that model a warm long-running server.
* ci(pnpr): fold the install-accelerator bench into the pacquet workflow
The pnpr server is built from the pacquet resolver/store/tarball crates,
so any pacquet change can move the pnpr-accelerated numbers as much as the
direct ones. That means the two benchmarks share a trigger surface and
should co-run — so rather than a separate pnpr workflow posting a second
comment on every pacquet PR, measure both in one run.
The pacquet integrated-benchmark workflow now also runs `pnpr@<rev>`
targets in the two hot-cache/hot-store scenarios (a warm long-running
server is pnpr's realistic shape), emits one combined report/comment, and
uploads to two Bencher testbeds: `pacquet` (direct, all scenarios) and
`pnpr` (accelerated, hot scenarios). The trigger gains `pnpr/**`.
Deletes the standalone pnpr-integrated-benchmark{,-comment}.yml added
earlier in this branch.
* ci(pnpr): also benchmark pnpr with a cold client store
Run the pnpr targets in the cold-cache/cold-store scenarios too, not just
the hot ones. Those scenarios already wipe the client store between
iterations while the per-target pnpr server store stays warm, so this
measures pnpr's cold-client-vs-warm-server shape — the realistic CI case
(empty local store hitting a warm shared server) — alongside the existing
hot-client numbers.
Both tools now run all four scenarios, so the report tables and both
Bencher testbeds (pacquet, pnpr) cover cold and hot. Collapses the two
target-list env vars into one and bumps the cold-step timeouts for the
extra commands. Table rendering is unchanged.
* ci(pnpr): address PR review feedback
- work_env: wrap the spawned pnpr child in its PnprServer guard before the
readiness wait and .pnpr-env write, so an early panic kills the process
on unwind instead of leaking an orphaned server (Copilot).
- cli_args: document pnpr@<rev> in the `targets` --help text (CodeRabbit).
- workflows: guard each bencher upload on its file existing, so a missing
optional results file logs a notice instead of failing the step (Copilot).
|
||
|
Zoltan Kochan
|
2b788d53fd |
refactor: replace the experimental pnpm-agent server with pnpr (#12151)
The experimental TypeScript `pnpm-agent` install-accelerator server is superseded by the `pnpr` server, which implements the same protocol. Remove `agent/server` and route the agent e2e test through pnpr. The pnpm TypeScript client (`@pnpm/agent.client`) is kept and made compatible with pnpr. The wire protocol carries the on-disk lockfile format, while pnpm keeps an in-memory `LockfileObject` in process: - Incoming: the agent's response lockfile is converted to the in-memory shape via `convertToLockfileObject`. - Outgoing: the existing lockfile is read in its on-disk shape with the new `readWantedLockfileFile` and forwarded as-is — no in-memory round-trip. pnpr now resolves multi-project workspaces by reconstructing the workspace on disk (root manifest + `pnpm-workspace.yaml` + member manifests) and letting pacquet's install path discover every importer. Member dirs are written as quoted YAML scalars; importer dirs are validated against path traversal (rejecting absolute, `..`, backslash, and slashes-only inputs) and de-duplicated; synthetic manifest names map injectively from dirs. The CI test job builds the `pnpr` server from source (cached on the Rust sources) so the agent e2e tests run against the current server. The published `@pnpm/pnpr` is dropped as a test dependency: running the suite already requires building `pnpr-prepare` from source (no npm fallback), so the toolchain to build `pnpr` is always present, and the published binary can predate the server protocol the tests exercise. |
||
|
suzunn
|
8bc25f3c26 |
fix: retry pacquet metadata fetches with a shared, config-driven retry policy (#12029)
## Problem `pacquet` metadata fetches made a single registry request, so a transient network failure or a retryable HTTP response (`408`/`429`/`5xx`) aborted resolution — even though the tarball path already followed pnpm's retry policy. pnpm wraps **every** registry request, metadata and tarball, in `@zkochan/retry` under one `fetch-retries` budget; pacquet only retried tarballs, and the metadata retry that was added in the first cut duplicated the tarball budget and ignored the user's config. Fixes #11841. ## Solution A single retry primitive now lives in `pacquet-network` and is driven by config on the install paths, matching pnpm exactly. - **One shared `RetryOpts`.** The retry budget (`fetch-retries` / `-factor` / `-mintimeout` / `-maxtimeout`) plus its exponential-backoff math moves into `pacquet-network`, next to `should_retry_status` and a generic `send_with_retry(client, url, opts, build_request) -> (guard, Response)` helper. `pacquet-tarball` re-exports `RetryOpts`; the metadata fetchers use `send_with_retry`. This removes the duplicate `MetadataRetryOpts`. - **No parked sockets/permits during backoff.** `send_with_retry` acquires the network permit *per attempt* and drops the response and its guard before each backoff sleep, so a flapping registry can't pin sockets or hold a concurrency permit while it waits. On the winning attempt the guard rides out with the `Response` so the caller keeps the permit through body streaming. - **Config-driven retries on every metadata path.** A config-sourced `RetryOpts` (via the existing `retry_opts_from_config`) is threaded through the resolution verifier, `NpmResolver`, `NamedRegistryResolver`, and `PickPackageContext`, so the metadata and verify paths honor the user's `fetch-retries*` settings instead of a hardcoded default — the same way the tarball path already did. ### Parity bug fixed Because the verifier and resolver hardcoded the default retry budget, a `5xx` flap on the main resolve/verify paths hung for **~70 s** (`10 s + 60 s` default backoff) regardless of the user's `fetch-retries` setting — a user who set `fetch-retries=0` to fail fast would still wait. Driving the budget from config fixes this; test harnesses use a zero-retry budget so failure-path cases don't wait out the backoff. ### Not pnpr `pnpr` is intentionally untouched. Retry is an install-time concern: pnpr's `/v1/install` accelerator already retries through pacquet's resolver and tarball fetcher, while the verdaccio-style registry-**proxy** path must fail fast (and serve stale on failure) rather than block a client request behind a 70 s backoff. The shared `RetryOpts` re-export keeps the accelerator's existing tarball-download retry compiling unchanged. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Sharmila
|
1c73e8303c |
fix(deps-resolver): prefer locked peer contexts during resolution by default (#12083)
## Summary Preserve compatible peer contexts already recorded in the lockfile during a writable re-resolution. A fresh install still resolves peers normally. When a lockfile already records multiple valid peer contexts, pnpm keeps those contexts instead of collapsing them into one compatible context and rewriting unrelated lockfile entries. ## Why [#12075](https://github.com/pnpm/pnpm/pull/12075) fixed optional-peer candidate selection: pnpm no longer discards a compatible optional-peer version merely because it came from the lockfile. This PR addresses a separate source of lockfile churn. A writable install could still replace one valid peer context with another valid peer context even when the existing provider remained present and satisfied the peer range. Public reproduction: <https://github.com/sharmila-oai/pnpm-optional-peer-lockfile-repro> The nested reproduction starts with two valid `vitest@3.2.4` contexts: ```text context-low -> vitest@3.2.4(jsdom@26.1.0) context-high -> vitest@3.2.4(jsdom@27.4.0) ``` Running a writable lockfile regeneration should retain both contexts: ```sh ./reproduce-nested-context.sh ``` ## Behavior pnpm reuses a locked peer provider only when: - The provider is still present in the current dependency graph. - The provider still satisfies the peer range. Current manifest choices remain authoritative. In particular, pnpm does not replace: - A newly added direct peer provider. - An explicitly updated direct peer provider. - A changed nested provider. - A direct provider installed through an alias. The reuse pass runs only when the dependency tree contains locked peer contexts, so fresh installs do not pay for a second peer-resolution pass. ## Tradeoff This change favors lockfile stability over reducing the number of peer contexts. A writable install may retain multiple compatible peer contexts where a fresh install would select one. ## Implementation The resolver performs its normal peer-resolution pass first. When the dependency tree contains locked peer contexts, it performs a second pass that may reuse compatible provider paths from the lockfile while respecting current manifest choices. pacquet now mirrors this behavior. Its lockfile-reuse path rebuilds child dependencies from the package manifest and skips peer dependencies recorded in the snapshot, so the peer pass derives each dependency instance's peer context. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
f429f93e8b |
feat(pnpr): support --lockfile-only on the pnpr install path (resolve-only mode) (#12148)
* feat(pnpr): support --lockfile-only on the pnpr install path Add a server-side resolve-only mode so `--lockfile-only` (and the `lockfileOnly` setting) is honored when a pnprServer / agent is configured: resolve and write the lockfile, fetch nothing, link nothing. The pnpr server skips the tarball fetch and file diff when the request sets `lockfileOnly`, returning just the lockfile. The pnpr client and the TypeScript agent client forward the flag and ignore any file/index lines an older server still streams, keeping the store untouched. The pacquet CLI no longer errors and stops after writing the lockfile. Closes #12146 * fix(agent.client): observe fileDownloads in lockfile-only path; correct minimumReleaseAge unit doc Address PR review: avoid a possible unhandled rejection if the NDJSON stream errors after the L frame in lockfile-only mode, and fix the `minimumReleaseAge` JSDoc (minutes, not seconds) to match the resolver and the Rust client. |
||
|
Khải
|
622c056bbc |
feat(pacquet/cli): initial implementations of run, exec, dlx (#11938)
> [!WARNING] > **Scope note.** Per [`pacquet/CONTRIBUTING.md`](https://github.com/pnpm/pnpm/blob/d4a2b0364c/pacquet/CONTRIBUTING.md), pacquet's current focus is Stage 1 (the headless installer); `exec` and `dlx` are new top-level commands, so this PR sits outside Stage 1 and is opened for review/discussion ([roadmap pnpm/pacquet#299](https://github.com/pnpm/pacquet/issues/299)). ## Summary Ports of `run`, `exec`, and `dlx` from the TypeScript pnpm CLI. - **`run`**: runs scripts through a new foreground `run_script` in `pacquet-executor` (sets up `node_modules/.bin` on `PATH` + the `npm_*` env). Handles `pre`/`post` scripts under `enablePrePostScripts`, arg shell-quoting (with the Windows `cmd /d /s /c` verbatim `raw_arg` path), script listing, hidden (`.`-prefixed) scripts, `--if-present`, the `start`→`server.js` fallback (with the NO_SCRIPT_OR_SERVER guard, including empty-string `start`), the `[ELIFECYCLE]` failure line (with the `test`-stage and signal-killed variants), and exit-code propagation. The recursive runner's scaffolding (`--resume-from` / `--report-summary`) landed separately on `main` via [#12093](https://github.com/pnpm/pnpm/pull/12093); this PR dispatches to it when `-r` is set and hardens it to match pnpm — per-project `pre`/`post`, the `PNPM_SCRIPT_SRC_DIR` recursion guard, pnpm's per-stage no-op guards, hidden-script handling, and `--no-bail`. - **`exec`**: runs a command with `node_modules/.bin` + `extraBinPaths` on `PATH` (resolved via `which`), stamps `npm_config_user_agent` / `PNPM_PACKAGE_NAME` / `NODE_OPTIONS`, supports `--shell-mode` (the joined command goes through the shared `push_script_arg` helper, so the Windows `cmd /d /s /c` verbatim path uses `raw_arg` and embedded quoting survives), rejects delimiter-containing dirs (`ERR_PNPM_BAD_PATH_DIR`). The **recursive variant** (`-r`) runs the command in every workspace project, topologically sorted and sequential, with `--resume-from` / `--report-summary` / `--no-bail` and pnpm's error codes (`ERR_PNPM_RECURSIVE_EXEC_NO_PACKAGE` / `ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL` / `ERR_PNPM_RECURSIVE_FAIL`). The workspace-graph / summary machinery is shared with recursive `run` through a new `cli_args::recursive` module. - **`dlx`**: installs the package(s) into a TTL cache dir (reusing the install pipeline, anchored at the cache dir, with a *fresh* per-install build-script allow-list — caller's `allow_builds`/`dangerouslyAllowAllBuilds` don't leak in), then runs the resolved bin in the process cwd. Supports `--package`, `--allow-build`, `--shell-mode` (same `push_script_arg` verbatim path as exec), `--cpu`/`--os`/`--libc` architecture overrides (folded into the per-axis `supportedArchitectures` of the dlx install **and** into the cache key, so different overrides don't share a cached install), `dlxCacheMaxAge`; same PATH guard as exec. New `Config` settings: `enablePrePostScripts`, `scriptShell`, `nodeOptions`, `dlxCacheMaxAge` (wired into `pnpm-workspace.yaml` + the `PNPM_CONFIG_*` overlay). Their defaults match pnpm and are asserted by the `pnpm_default_parity` contract test — this PR moves `enablePrePostScripts` (which pnpm defaults to `true`, a breaking change in [#7634](https://github.com/pnpm/pnpm/pull/7634) shipped in v9) and `dlxCacheMaxAge` into its mapped rows. `extraBinPaths` is kept as a computed field (empty until workspace support lands), matching pnpm — it is not a user-settable key. ## Deferred (documented in code) - **`--filter` and `--workspace-concurrency`.** Recursive `run` and `exec` run every workspace project sequentially; the `--filter` package-selector subsystem and `--workspace-concurrency` parallelism are not ported yet (the global `--filter` / `--recursive` flags are accepted via clap but `--filter` is not consumed). `dlx` stays single-package by design (matches pnpm). - `run`: the `/regexp/` script selector and the fuzzy "did you mean" hint are not ported (no regex/levenshtein dep); `scriptsPrependNodePath: always` can't prepend the node dir (pacquet resolves no node execpath anywhere yet). - `dlx`: the cache key uses raw specs (not resolved ids); no `approve-builds` prompt. |
||
|
Zoltan Kochan
|
32c07bfee0 |
feat(pnpr): offload lockfile verification to the server (#12139) (#12144)
Closes #12139. ## What When a `pnpr` server is configured, the client no longer runs `verifyLockfileResolutions` locally. It sends its on-disk lockfile plus its **full verification policy** to `/v1/install`; pnpr verifies that *input* lockfile under the **client's** policy *before* resolving, and streams back any violations so the client aborts with the identical `ERR_PNPM_*` diagnostic the local gate would have produced. This is faster (pnpr's packument cache is warm + shared) and removes the client's own registry-reachability requirement — it adds no new trust (the client already trusts pnpr to resolve and serve bytes). All three phases from the issue, delivered together. **Rust-only**: `pacquet` client + `pnpr` server. The TS agent server is deprecated and the TS client already skips local verification, so no TS changes were needed. ## How **Phase 1 — send lockfile + policy; pnpr verifies; client skips local verify** - Protocol (`install_accelerator/protocol.rs`, mirrored in `pnpr-client`): `/v1/install` now carries `lockfile`, `frozenLockfile`, and the full policy (`minimumReleaseAge[Exclude|IgnoreMissingTime]`, `trustPolicy[Exclude|IgnoreAfter]`). - `handle_install` verifies the input lockfile via `build_resolution_verifiers` + `collect_resolution_policy_violations` (under the client policy threaded into the server `config_for`) **before** resolving. On a violation it streams a `200` NDJSON `E` line of rendered violations; the client rebuilds the identical `VerifyError` (`PnprClientError::Verification`). - The pacquet CLI sends `state.lockfile` + policy, drops the `trustPolicy: no-downgrade` guard (pnpr enforces it now — input-lockfile verifier for reused entries + resolver pick-time check for new ones), and sets `trust_lockfile: true` on the local materialization so it never re-verifies or touches the local `lockfile-verified.jsonl`. **Phase 2 — `frozenLockfile` governs resolution reuse** - `resolve.rs` seeds resolution from the input lockfile (frozen → as-is; non-frozen → reuse pins + resolve new). **Phase 3 — SQLite whole-lockfile verdict cache on pnpr** - New `install_accelerator/verdict_cache.rs`: SQLite-backed (reuses the existing `rusqlite` dep), keyed by `(lockfile hash, merged policy snapshot)`, hit = all verifiers `can_trust_past_check`. Only *passes* are cached (monotonic age + hash pins versions → time-correct without a cutoff, same property as the local cache); LRU cap, no TTL. |
||
|
Zoltan Kochan
|
6d17b669b4 |
fix: verify lockfile tarball URL matches registry metadata (#12134)
## What The lockfile resolution verifier now confirms that a registry entry pinning an explicit `tarball` URL points at the artifact the registry's own metadata lists for that `name@version`. A mismatch — or any entry that can't be confirmed against the registry — is rejected with `ERR_PNPM_TARBALL_URL_MISMATCH`. ## Why Follow-up to the design discussion on #12122. The verifier checked the age/trust of `name@version` against the registry packument but never bound the lockfile's `tarball` URL to it. For the non-standard entries pnpm preserves a tarball URL for (npm Enterprise, GitHub Packages — see `toLockfileResolution`), pnpm fetches straight from that URL. So a **tampered lockfile could pair a trusted `name@version` with an attacker-chosen tarball URL** (plus a matching integrity for the attacker's bytes); verification passed against the legitimate version while the install fetched the attacker's bytes. Defending a checked-in lockfile is explicitly in this feature's threat model. ## How - For a registry-keyed entry that pins an explicit `tarball`, fetch the packument and assert the URL equals `versions[v].dist.tarball`. The comparison canonicalizes away benign differences — http/https scheme, default ports (`:443`/`:80`), and `%2f` scope-separator encoding (case-insensitive) — so only real mismatches are flagged. The packument is fetched from the user's configured registry (the lockfile's tarball host can't redirect it), and named-registry routing uses the same canonicalization so a scheme/`%2f`-only difference doesn't route to the wrong packument. - **The binding is unconditional.** It runs regardless of `minimumReleaseAge`/`trustPolicy` and is **not** narrowed by their exclude lists, because it guards *integrity*, not *maturity/trust*. Disabling the age/trust policies must not silently disable anti-tamper. (`createNpmResolutionVerifier` therefore always returns a verifier.) - **It is fail-closed.** An entry passes only when the registry metadata affirmatively lists the version with a matching tarball URL. If the metadata can't be fetched, doesn't list the version, or omits `dist.tarball`, the entry is rejected — otherwise a tampered lockfile could smuggle a malicious URL past the check by pointing it at a `name@version` the registry can't vouch for. - **Behavior change:** as a result, an install that re-verifies a lockfile (its content changed since the last verified run, so the verification cache no longer short-circuits) now requires the configured registry to be reachable. `trustLockfile` is the opt-out for environments that treat the on-disk lockfile as already trusted. - **Verification cache.** The policy snapshot records a `tarballUrlBinding` marker and `canTrustPastCheck` requires it, so a cache record written before this rule existed is re-verified rather than trusted (closing an upgrade-time bypass). - Entries with no explicit `tarball` reconstruct the URL from name+version+registry and are inherently bound (no check). `file:`/git-hosted resolutions stay out of scope (#12122). - Threads `nonSemverVersion` to the verifier so URL-keyed tarball deps (a remote `https:` tarball that carries a semver `version` copied from its manifest) are recognized as deliberate non-registry deps and skipped — also fixing a latent release-age over-match on them. The candidate dedupe key includes `nonSemverVersion` so a registry snapshot and a URL-keyed snapshot sharing a `name@version` and serialized resolution stay distinct. Mirrored in pacquet (`create_npm_resolution_verifier`). The dedupe-key change is TS-only: pacquet's candidate `version` comes from the lockfile key suffix, so the two shapes never share a key there. ## Tests - TS: confirmed mismatch → violation; non-standard URL matching metadata → pass; default-port/scheme difference → pass; URL-keyed dep → skipped; URL binding runs (and fails closed) with no age/trust policy configured; `canTrustPastCheck` rejects a cache record lacking the binding marker. Regression-verified (the mismatch test fails when the check is disabled). - pacquet: mirror tests + the no-policy / `minimumReleaseAge: 0` / `trustPolicy: off` cases, default-port/scheme equivalence, and the missing-`tarballUrlBinding` cache rejection. A few install-dispatch / resolution-reuse tests that pin a deliberately bogus tarball URL (or run against an unreachable registry to prove resolution reuse) now set `trustLockfile`, since the always-on fail-closed tarball-URL check would otherwise flag the fixture before the path under test runs. - `clippy --deny warnings`, `fmt`, and `dylint` clean. |
||
|
Zoltan Kochan
|
ae6251ca7d |
chore: extend update-lockfile workflow to bump Node.js, pnpm, and pacquet (#12135)
* chore: also bump Node.js, pnpm, and pacquet in update-lockfile workflow * chore: address PR review feedback on update-lockfile workflow - Base the update branch on an explicitly fetched origin/main - Don't persist the write token during install; push with explicit URL - Detect open PRs via gh --json instead of grepping table output - Add a concurrency guard to serialize dispatch + scheduled runs |
||
|
Zoltan Kochan
|
172d5e56e6 |
chore: update pnpm-lock.yaml (#12014)
* chore: update pnpm-lock.yaml * chore: sync node.js runtime version in scripts with devEngines via meta-updater * chore: sync node.js version in CI workflows with devEngines via meta-updater |
||
|
Marvin Hagemeister
|
5b95e08c50 | perf(pacquet): avoid per-entry tarball allocations (#12131) | ||
|
btea
|
722b9cda24 |
fix: skip lockfile minimumReleaseAge/trustPOlicy verification for non-registry tarball (#12122)
|
||
|
Zoltan Kochan
|
c0368f473f | chore: update pacquet to 0.2.13 (#12130) v11.5.1 | ||
|
Zoltan Kochan
|
ba2bacd9cf |
fix(pacquet): dedupeDirectDeps default + per-importer workspace link resolution, with defaults contract test (#12128)
* fix(config): default dedupeDirectDeps to false to match pnpm
pacquet defaulted `dedupe_direct_deps` to `true`, but pnpm's
config-reader default is `false` (config/reader/src/index.ts:139,
`'dedupe-direct-deps': false`). With the wrong default, a direct
dependency that both the workspace root and a non-root importer
declare gets dropped from the non-root importer's `node_modules/`.
This broke the v11.5.1 release (.github#214). The release installs
the monorepo through pacquet's frozen-lockfile path, then runs
`pnpm publish` on `@pnpm/exe`, which rewrites its `workspace:*`
devDependency on `@pnpm/jest-config` by reading
`exe/node_modules/@pnpm/jest-config/package.json`. The workspace
root also depends on `@pnpm/jest-config`, so pacquet's erroneous
dedupe removed the per-importer symlink and publish failed with
`ERR_PNPM_CANNOT_RESOLVE_WORKSPACE_PROTOCOL`.
The dedupe-on integration tests already in this file relied on the
old default; they now set `dedupeDirectDeps: true` explicitly,
mirroring upstream's `testDefaults({ ..., dedupeDirectDeps: true })`.
A new test pins the default-off behavior against a pnpm-written
frozen lockfile, reproducing the release scenario.
* fix(deps-resolver): key workspace link resolutions by importer
A non-injected workspace dependency (`workspace:*`, `link:<path>`)
resolves to a `link:<path>` whose path is computed relative to the
*consuming importer's* directory. pacquet's workspace-wide
`resolved_by_wanted` cache keyed only `(alias, specifier, optional,
injected, pick_lowest, published_by)`, so the first importer to
resolve a shared workspace dep seeded the cache with its own relative
path and every other importer reused it verbatim.
With a root and a `packages/app` both depending on `@scope/lib`, the
root resolved `link:packages/lib` and `packages/app` got that same
string back — a dangling `packages/app/packages/lib`. pnpm writes
`link:../lib` for `packages/app`.
Add the consuming importer's `project_dir` to the cache key for
project-relative specifiers (`link:` / `file:` / `workspace:`) only;
registry specifiers stay importer-independent and keep sharing one
cache slot across importers, preserving the cross-importer dedup.
* test(config): pin pacquet defaults to pnpm CLI defaults
A contract test reads pnpm's `defaultOptions` literal from
config/reader/src/index.ts at test time and asserts every pacquet
default that maps to a pnpm setting equals pnpm's value. Drift on
either side fails here — the `dedupeDirectDeps` default that broke the
v11.5.1 release would have been caught.
`every_pnpm_default_is_classified` partitions all 80 pnpm settings into
mapped / non-literal (env, platform, CPU) / not-ported, so a new pnpm
setting that's neither mapped nor skipped fails the test until someone
classifies it.
|
||
|
Zoltan Kochan
|
15e0576893 |
feat(pacquet): wire pnpmfile context.log to the pnpm:hook channel (#12125)
* feat(pacquet): wire pnpmfile context.log to the pnpm:hook channel Forward each pnpmfile hook's context.log(...) call to a new LogEvent::Hook reporter variant (name: 'pnpm:hook', with from/hook/ message/prefix payload parity with pnpm's hookLogger). The readPackage and afterAllResolved call sites previously dropped these on a no-op closure. Rather than make resolve_dependency_tree generic over Reporter, the install layer pre-binds the reporter, project prefix, and pnpmfile path into a LogFn closure and threads it through WorkspaceResolveOptions into the shared WorkspaceTreeCtx, keeping the resolver reporter-agnostic. Ports pnpm's 'pnpmfile: pass log function to readPackage hook', 'run afterAllResolved hook', and 'run async afterAllResolved hook' (pnpm/test/install/hooks.ts), plus a reporter wire-shape test. Part of #12118. * refactor(pacquet): thread read_package_log through the standalone resolver entry point resolve_dependency_tree() accepts a pnpmfile_hook but built its TreeCtx without the read_package_log sink, so a caller of this entry point would silently drop the hook's context.log(...) output. Add the field to ResolveDependencyTreeOptions and a TreeCtx::with_read_package_log passthrough so both resolver entry points behave identically. The production install path goes through resolve_workspace (already wired); this only affects the standalone entry point, today used by the resolver's unit tests. |
||
|
Sharmila
|
60a1eeca28 |
fix(cli): avoid Windows crash after network requests (#12121)
* fix: avoid Windows crash after network requests * fix: destroy dispatchers before Windows error exits * test: cover destroyDispatchers and document its shutdown contract Add a JSDoc to destroyDispatchers() explaining it is for process shutdown, a unit test verifying it destroys the global and cached dispatchers, and bump @pnpm/network.fetch to minor for the new public export. * fix(network.fetch): destroy the active global dispatcher on shutdown Include getGlobalDispatcher() so destroyDispatchers() also closes the currently-active dispatcher if it was replaced via setGlobalDispatcher(). --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
0f509d055f | chore(release): 11.5.1 (#12126) | ||
|
Zoltan Kochan
|
eec417b74f |
fix(lockfile): emit lockfile maps in canonical key order (#12120)
pacquet serialized the `importers`, `packages`, `snapshots`, and per-importer/per-snapshot dependency maps in `std::HashMap` iteration order — a per-instance random seed — so two installs of the same resolution could emit byte-different `pnpm-lock.yaml` files. This diverges from pnpm, whose lockfile is canonically ordered, and produces spurious git diffs on no-op re-installs (#12117). Sort every lockfile map by its rendered key string at emit time via two `serialize_with` helpers in `serialize_yaml`, matching pnpm's `sortLockfileKeys`/`lexCompare`. Sorting by the rendered key (not a field-wise `Ord`) is load-bearing: the `@` separating `name@version` and the leading `@` of a scoped name both order differently as struct fields than as the concatenated string pnpm compares. `overrides` is the one map pnpm leaves unsorted (declaration order), so it moves from `HashMap` to `IndexMap` to preserve insertion order rather than being sorted — deterministic and faithful to pnpm. The reuse-vs-fresh equivalence test now asserts byte-for-byte parity (was a parsed-struct comparison working around the old non-determinism), and a new test guards that a no-op re-install leaves the lockfile bytes unchanged. |
||
|
Sharmila
|
122ab0a1ed |
fix(deps-resolver): preserve locked optional peer candidates (#12075)
## Summary Preserve compatible optional-peer versions already recorded in the lockfile when pnpm re-resolves a workspace. ## Reproduction Public reproduction: <https://github.com/sharmila-oai/pnpm-optional-peer-lockfile-repro> The workspace contains: ```text packages/uses-vitest -> vitest@3.2.4 packages/older -> jsdom@26.1.0 ``` Vitest declares `jsdom` as an optional peer dependency. The committed lockfile was generated when another workspace package also depended on `jsdom@27.4.0`. At that time, pnpm selected the higher compatible version for Vitest: ```text vitest@3.2.4(jsdom@27.4.0) ``` That additional direct dependency was then removed from its `package.json`, without regenerating the lockfile. This simulates a normal manifest edit. Running: ```sh pnpm install --lockfile-only --no-frozen-lockfile ``` unnecessarily rewrites Vitest's still-valid optional-peer context: ```diff - version: 3.2.4(jsdom@27.4.0) + version: 3.2.4(jsdom@26.1.0) ``` Both versions satisfy Vitest's optional peer range. The existing `27.4.0` resolution remains valid and should not be discarded while pnpm updates the lockfile. ## Cause Preferred versions loaded from the wanted lockfile are stored as weighted selectors: ```ts { selectorType: 'version', weight: EXISTING_VERSION_SELECTOR_WEIGHT } ``` `getHoistableOptionalPeers()` only recognized the plain string form: ```ts specType === 'version' ``` As a result, it ignored the compatible locked `27.4.0` candidate. It only saw `26.1.0`, which was rediscovered from `packages/older/package.json`, and rewrote the peer context. ## Fix Normalize the selector before checking its type, matching the handling already used by `hoistPeers()` for required peers: ```ts const specType = typeof selector === 'string' ? selector : selector.selectorType ``` This restores lockfile-seeded versions to the candidate set. It does not add a new preference rule or force pnpm to keep every locked version. Optional-peer auto-installation continues to choose the highest version satisfying every recorded peer range. The equivalent fix is included in pacquet, pnpm's Rust port. ## Validation - Added matching TypeScript and Rust regression tests. - Verified the public reproduction against `pnpm@11.4.0` and the patched CLI. - Ran the focused TypeScript resolver checks and pacquet test, clippy, format, and `cargo nextest` checks. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Alessio Attilio
|
5c669d7387 |
feat(pacquet): add pnpmfile hooks support (#12044)
Implements Tier 4 pnpmfile hooks for pacquet (#11633, point 4.1): pacquet now discovers and runs a project `.pnpmfile` during dependency management, matching pnpm. ## What it does - **Discovery** — looks for `.pnpmfile.mjs` then `.pnpmfile.cjs` (dotted names only, `.mjs` preferred), matching pnpm's `requireHooks`. Only actual files are accepted (`is_file()`). - **`readPackage`** — wired into resolution. Mirrors pnpm's `requirePnpmfile` contract: the four dependency fields are defaulted to `{}` before the hook runs, and the returned manifest is validated (must be a non-null object whose dependency fields, when present, are objects rather than arrays). A throwing/syntactically-invalid pnpmfile, a missing `require`, or a hook that returns nothing aborts the install (`PNPMFILE_FAIL`) instead of being silently ignored. - **`afterAllResolved`** — wired into the lockfile write. The resolved lockfile is passed to the hook and its return value is what gets written to `pnpm-lock.yaml`. The round-trip goes through `serde_json::Value` (the workspace already enables `preserve_order`) so hook-added keys the typed `Lockfile` cannot represent survive to disk; the round-trip only runs when a hook is present, so unmodified installs write byte-identical lockfiles. A throwing hook aborts the install. - **`preResolution`** — wired. Receives the resolution context (wanted/current lockfile, `existsCurrentLockfile`, `existsNonEmptyWantedLockfile`, lockfile dir, store dir, registries) over stdin. - **`filterLog`** — implemented in the bridge but not yet routed through the reporter (pacquet's reporter is a stateless synchronous emitter); deferred, see follow-ups. ## How hooks run Hooks are served by a long-lived Node.js worker, spawned lazily once per pnpmfile. Requests and responses are newline-delimited JSON over the worker's stdin/stdout, multiplexed by a monotonic request id so the concurrent `readPackage` calls the resolver makes (it resolves dependencies in parallel) share one process. This removes the per-package `node` startup cost on the resolution hot path and avoids interpolating payloads into a `node -e` argument (no `E2BIG` risk for large lockfiles). Each `context.log(...)` a hook emits is forwarded back to the call's log callback. `preResolution` keeps a one-shot `node` invocation since it runs once per install and needs an `info`/`warn` logger. ## Tests - Unit (hooks crate): readPackage validation (returns nothing / non-object / array dependency fields), manifest-field normalization, syntax-error and missing-module failures, worker request-id multiplexing under concurrency, and `context.log` forwarding. - Integration (package-manager): a `readPackage` hook pins a transitive dependency version; a hook that returns nothing aborts the install; a pnpmfile syntax error aborts the install; an `afterAllResolved` hook's mutation is written to `pnpm-lock.yaml`; a throwing `afterAllResolved` aborts the install. ## Scope The remaining pnpmfile-hook surface pnpm has but pacquet does not yet implement — wiring `filterLog` and the `pnpm:hook` log channel into the reporter, the `--pnpmfile` / `--global-pnpmfile` / `--ignore-pnpmfile` flags, pnpmfile checksum invalidation, `updateConfig`, and finders/resolvers/fetchers — is tracked in #12118. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
73a294e6dd |
perf(pacquet): reuse lockfile resolutions on re-resolution (#12113)
During a non-frozen install, reuse the prior pnpm-lock.yaml's resolution and transitive subtree for dependencies still satisfied and not being updated, instead of re-resolving everything from manifests. Faithful port of pnpm's getInfoFromLockfile / resolvedDependencies / parentPkg.updated machinery, adapted to pacquet's resolver as a hybrid resolve: snapshot-walk unchanged subtrees, fresh-resolve new/changed/update-targeted deps, with an `updated` flag propagated down. Semver-satisfies reuse gate (pnpm parity). Conservative by construction — registry resolutions only, gated on the whole subtree being synthesizable; `pacquet update` targets, packageExtensions drift, and dependency cycles all fall through to a fresh resolve. Follow-up to #12096 (which covered only remote tarballs). Lockfile canonical-ordering tracked separately as #12117. |
||
|
Zoltan Kochan
|
ae6e07705d |
feat(pacquet): implement the outdated command (#12112)
## What
Ports pnpm's `outdated` command to pacquet. It reports the direct dependencies whose newest published version (or highest in-range version under `--compatible`) is newer than the lockfile-pinned version, and exits with code `1` when any outdated dependency is found.
This widens pacquet's command surface beyond `install`/`add`/`update`/`remove`.
## How
- **Detection core** — `collect_outdated` / `OutdatedQuery` / `TargetVersion` in `pacquet/crates/cli/src/cli_args/outdated.rs`. This is shared with `update --interactive`, which already computed the same "what has a newer version" list inline; that code now calls the shared collector. The two callers differ only in the `TargetVersion` they compare against (`outdated` → `latest` tag / highest in-range under `--compatible`; `update` → the version a bump moves to). Packument fetches fan out concurrently via `futures::join_all`, mirroring pnpm's `Promise.all` (bounded by the HTTP client's per-registry concurrency limit).
- **Flags** — `--compatible`, `--long`, `--format table|list|json` (plus `--no-table` / `--json` shorthands), `-P/--prod` (`--production`), `-D/--dev`, `--no-optional` (following pnpm's `only`-normalization include-set), `--sort-by name` (default sort is by semver-change size then name), and positional name patterns (`@pnpm/config.matcher`).
- **Exit code** — `OutdatedArgs::run` returns an `OutdatedOutcome`; the single `process::exit(1)` lives in the top-level CLI dispatch, keeping the command composable.
- **`--global` and `--recursive` are rejected** with a "not supported yet" message, matching `pacquet update` (single-project scope).
- **Empty-manifest short-circuit** — a dependency-free manifest reports empty (exit 0) *before* the no-lockfile check, matching pnpm's `packageHasNoDeps` behavior so an empty, never-installed project doesn't error.
- **Rendering** — adds `tabled` (`Style::modern()` reproduces pnpm's `@zkochan/table` full-grid borders) and `owo-colors` (auto-disables on non-TTY, like chalk, so piped/JSON output is escape-free). Both are MIT (allowed by `deny.toml`). JSON output is keyed by package name with `current` / `latest` / `wanted` / `isDeprecated` / `dependencyType` (+ `latestManifest` under `--long`), matching pnpm's `renderOutdatedJSON`.
- Adds an optional `homepage` field to the registry `Package` for the `--long` details column.
## Parity notes / scope
pacquet loads a single (wanted) lockfile, so `current == wanted` and pnpm's "missing (wanted X)" state does not arise. `OUTDATED_NO_LOCKFILE` is raised when a manifest declares dependencies but no lockfile exists. Deprecated-but-current packages are still reported.
Deferred with their surrounding features (not yet ported to pacquet), each tracked by an upstream test that does not yet translate:
- `--long` **homepage** needs full registry metadata; the abbreviated fetch omits it, so it appears only when the registry serves it (deprecation details work fully).
- `minimumReleaseAge` / `minimumReleaseAgeExclude`, `pnpm.updateConfig.ignoreDependencies`, `catalog:` protocol replacement, `-g`/global packages, recursive workspace listing, and `runtime:` (node/deno/bun) dependencies.
## Tests
Ported the translatable pnpm `outdated` tests (`deps/inspection/outdated/test/*` and `deps/inspection/commands/test/outdated/*`):
- **10 unit tests** — semver-change classification, include-set normalization (default / `--prod` / `--dev` / `--no-optional`), default sort order, `renderLatest` (deprecated / not), and JSON shape (with/without `--long`).
- **14 integration tests** against the mocked registry — newer-version report, `--compatible` discrimination, JSON, JSON-empty `{}`, list (`--no-table`) format, up-to-date → exit 0, name-pattern filter, prod/dev filtering, npm-alias real-name reporting, deprecated package, `--long` deprecation details, no-deps/no-lockfile → empty exit 0, no-lockfile-with-deps error, and `--recursive` rejection.
|
||
|
Zoltan Kochan
|
12b7201a85 |
feat(pacquet): port the remove command (#12110)
* feat(cli): port the `remove` command to pacquet Ports pnpm's `remove`/`rm`/`uninstall`/`un`/`uni` command to pacquet, mirroring installing/commands/src/remove.ts and removeDeps.ts. - PackageManifest::available_dependency_names + remove_dependencies port pnpm's getAllDependenciesFromManifest validation set and removeDeps (peerDependencies and dependenciesMeta always cleared). - New Remove subroutine: validates with ERR_PNPM_MUST_REMOVE_SOMETHING / ERR_PNPM_CANNOT_REMOVE_MISSING_DEPS, mutates the manifest, then runs a fresh re-resolve install (partial uninstallSome mutation). - RemoveArgs with --save-prod/--save-dev/--save-optional (getSaveType precedence) and --lockfile-only; subcommand registered with the rm/uninstall/un/uni aliases. * refactor(package-manager): address review feedback on remove - Use HashSet for available-deps dedup and removal-target lookup (O(n) instead of O(n^2)/O(m*n)). - Fix grammar in remove's SaveManifest error message. - Add a test for the "project has no dependencies of any kind" branch. * fix(package-manager): correct grammar in add's SaveManifest error * test(package-manager): port pnpm's remove handler validation tests Extract `validate_removable` (the up-front MUST_REMOVE_SOMETHING / CANNOT_REMOVE_MISSING_DEPS guards, which fire before any install) and port every message/hint sub-case from pnpm's installing/commands/test/remove/remove.ts: the four "project has no '<field>'" variants, the per-save-type "no such dependencies found in '<field>'" + hint cases, and the singular no-field case with the dev->prod->optional hint ordering. * fix(package-manager): split out RemoveValidationError to satisfy result_large_err The extracted sync `validate_removable` returned `Result<(), RemoveError>`, and RemoveError carries the large InstallError variant. On Windows that pushed the Err variant over clippy's large-error-threshold (`clippy::result_large_err`), failing CI. Validation can only ever produce MUST_REMOVE_SOMETHING / CANNOT_REMOVE_MISSING_DEPS, so give it a small dedicated RemoveValidationError; RemoveError now wraps it transparently, preserving the rendered error codes. |