mirror of
https://github.com/pnpm/pnpm.git
synced 2026-06-28 09:55:39 -04:00
8928aed6a9a0e0271dc541d8140fbcbaaaf65521
111 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
dependabot[bot]
|
a34bfc942e |
chore(cargo): bump insta from 1.47.2 to 1.48.0 (#12524)
Bumps [insta](https://github.com/mitsuhiko/insta) from 1.47.2 to 1.48.0. - [Release notes](https://github.com/mitsuhiko/insta/releases) - [Changelog](https://github.com/mitsuhiko/insta/blob/master/CHANGELOG.md) - [Commits](https://github.com/mitsuhiko/insta/compare/1.47.2...1.48.0) --- updated-dependencies: - dependency-name: insta dependency-version: 1.48.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Zoltan Kochan
|
bd5390e24e |
feat(pnpr): add pluggable SQL auth backends (#12547)
Add feature-gated pnpr auth backends so deployments can build with libsql, PostgreSQL, or MySQL support instead of locking the registry to one database driver. The auth state still resolves to the existing UserBackend and TokenBackend trait objects. Configuration now accepts backend.libsql, backend.postgres/backend.postgresql, or backend.mysql and rejects selecting multiple shared databases. PostgreSQL and MySQL use a shared SQLx-backed auth implementation with driver-specific placeholder syntax. The shared auth schema uses common column types, and pnpr avoids SQLite-only upsert syntax in auth and verdict-cache writes. |
||
|
Alessio Attilio
|
247201eb19 |
feat(pacquet): implement cat-file command (#12550)
Adds `pnpm cat-file` functionality to the `pacquet` CLI. This extracts the base64-encoded file hash from an integrity string (e.g. `sha512-...`), decodes it, and reads the corresponding file from the CAFS store directory at `files/XX/YYYY...`. Includes a new e2e test verifying correct lookup and read operations. |
||
|
Zoltan Kochan
|
0d24a60236 |
feat(pnpr): block vulnerable versions with local OSV index (#12506)
Add local OSV npm database support to pnpr. When enabled, pnpr loads an OSV npm dump (an `all.zip` or an extracted JSON directory) from disk at startup and fails before serving requests if the configured database is missing, not a regular file, empty of npm advisories, or otherwise invalid. Resolution then uses the in-memory index rather than querying OSV during package selection. Package names are matched case-insensitively, OSV range events are sorted before evaluation, and per-record reads are size-bounded. Add a resolver-time package version guard to pacquet's npm resolver. The guard can reject a concrete package version, after which the picker filters that version out of the packument and tries the normal selection flow again. pnpr uses this hook to avoid vulnerable versions while preserving existing semver selection semantics. When every matching version is rejected, the resolver returns a distinct AllVersionsBlockedError rather than reporting the spec as unsupported. Check frozen, cached, verified, and freshly produced lockfiles against the local OSV index, gating tarball entries on the tarball URL rather than the tamper-prone gitHosted flag. The pnpr lockfile-verdict cache records a content-based OSV database fingerprint in its policy snapshot, so a changed vulnerability database invalidates previous cached verification passes and forces rechecking under the new data. Add fallible try_router / try_router_with_auth constructors so callers that build the router directly can handle an invalid OSV database recoverably instead of panicking. Add tests for OSV range matching (including out-of-order events), exact-version and case-insensitive matching, withdrawn handling, zip and directory loading, empty/non-regular-file rejection, pnpr config parsing, guarded re-pick and all-versions-blocked behavior, and tarball OSV-checkability. |
||
|
dependabot[bot]
|
c12a15f766 |
chore(cargo): bump ignore from 0.4.25 to 0.4.26 (#12525)
Bumps [ignore](https://github.com/BurntSushi/ripgrep) from 0.4.25 to 0.4.26. - [Release notes](https://github.com/BurntSushi/ripgrep/releases) - [Changelog](https://github.com/BurntSushi/ripgrep/blob/master/CHANGELOG.md) - [Commits](https://github.com/BurntSushi/ripgrep/compare/ignore-0.4.25...ignore-0.4.26) --- updated-dependencies: - dependency-name: ignore dependency-version: 0.4.26 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
c59ae0a477 |
chore(cargo): bump which from 8.0.2 to 8.0.3 (#12523)
Bumps [which](https://github.com/harryfei/which-rs) from 8.0.2 to 8.0.3. - [Release notes](https://github.com/harryfei/which-rs/releases) - [Changelog](https://github.com/harryfei/which-rs/blob/master/CHANGELOG.md) - [Commits](https://github.com/harryfei/which-rs/compare/8.0.2...8.0.3) --- updated-dependencies: - dependency-name: which dependency-version: 8.0.3 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
John-David Dalton
|
2dd79e372c |
feat(pacquet): honor NODE_EXTRA_CA_CERTS for custom CA trust (#12508)
* feat(pacquet): honor NODE_EXTRA_CA_CERTS for custom CA trust pacquet's reqwest client trusts only its bundled webpki roots plus the .npmrc ca/cafile material — it ignores NODE_EXTRA_CA_CERTS. pnpm running on Node picks that variable up transitively via Node's TLS runtime, so a native port has to read it explicitly to keep real-world parity for users behind a corporate MITM proxy or a self-signed registry. Load the named PEM bundle as additional trust roots in default_client_builder (the single chokepoint every client routes through), keeping the .npmrc-derived TlsConfig env-free. Additive and lowest-priority; a missing, unreadable, or malformed file is silently ignored, matching pnpm's silent treatment of a missing cafile. Documented as the one deliberate exception to the TlsConfig no-env-vars parity note. * perf(pacquet): load NODE_EXTRA_CA_CERTS once per for_installs Addresses review on the NODE_EXTRA_CA_CERTS change: - Read and parse the bundle once in for_installs via load_node_extra_ca_certs(), then clone the parsed certs into the default client and each per-registry client. Previously default_client_builder re-read and re-parsed the file on every call, i.e. once per per-registry override. - Use the existing EnvGuard test helper (process-wide lock + restore on drop, panic-safe) instead of a hand-rolled lock with manual restore. The test now also asserts a valid bundle parses to one root and a missing file yields none. * test(pacquet): align NODE_EXTRA_CA_CERTS test with aube's Make the pacquet and aube tests mirror each other in form and coverage: exercise the same four cases in the same order — empty value, valid bundle (asserting one parsed root plus a successful client build), a readable non-PEM file, and a missing file — each asserting load_node_extra_ca_certs() yields the expected roots. Also align the env-var read in load_node_extra_ca_certs to the same let-else + filter form aube uses (no behavior change). * docs(pacquet): fix broken intra-doc links in load_node_extra_ca_certs The free function's doc used [`Self::for_installs`], but `Self` only resolves in impl/trait contexts, so rustdoc flagged it as an unresolved intra-doc link and the Rust CI Doc job (RUSTDOCFLAGS=-D warnings) failed. Reference [`ThrottledClient::for_installs`] instead. |
||
|
Maël Nison
|
0474a9c3b1 |
feat: add support for package maps (#12430)
Generate a Node.js package map at `node_modules/.package-map.json` on every isolated or hoisted install, including under the global virtual store, so that third-party tooling can start experimenting with package maps. The file is serialized compactly. Two settings control how the map is consumed: - `node-experimental-package-map` (default: off): inject `--experimental-package-map` into `NODE_OPTIONS` for the Node.js scripts pnpm runs — dependency lifecycle scripts, `pnpm exec`, and `pnpm run` (including recursive runs). - `node-package-map-type` (`standard` | `loose`): choose between a strict map and one that tolerates hoisting-like access. Covered by both the pnpm CLI and the pacquet (Rust) implementation. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
bee4bf41ca |
fix: reject path-traversal config dependency names from the env lockfile (#12470)
Config dependency names and versions are read from the committed env lockfile (pnpm-lock.yaml) and the legacy inline-integrity format in pnpm-workspace.yaml, and both become path segments of the directories pnpm creates during install (node_modules/.pnpm-config/<name> and the global virtual store's <name>/<version>/<hash>). They were used unvalidated, so a malicious repository could commit a traversal-shaped name (../../PWNED) or version (../../../PWNED) and make `pnpm install` create symlinks or write package files outside those roots — triggered on install, even with --ignore-scripts. Add verifyEnvLockfile, an offline structural gate that validates every config dependency and optional-subdependency name (must be a valid npm package name) and version (must be an exact semver version) before any path is built from it. It runs at the install boundary and, through a single writeVerifiedEnvLockfile seam, before the env lockfile is ever persisted, so an invalid entry is rejected with no write side effect. __proto__ names are rejected too (the validation accumulators use null-prototype objects so the key can't slip past Object.keys). The same fix and structure land in pacquet to keep the two stacks in sync. Fixes GHSA-qrv3-253h-g69c. |
||
|
dependabot[bot]
|
23c8efeffc |
chore(cargo): bump dialoguer from 0.11.0 to 0.12.0 (#12355)
Bumps [dialoguer](https://github.com/console-rs/dialoguer) from 0.11.0 to 0.12.0. - [Release notes](https://github.com/console-rs/dialoguer/releases) - [Changelog](https://github.com/console-rs/dialoguer/blob/main/CHANGELOG-OLD.md) - [Commits](https://github.com/console-rs/dialoguer/compare/v0.11.0...v0.12.0) --- updated-dependencies: - dependency-name: dialoguer dependency-version: 0.12.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
de74c58f58 |
chore(cargo): bump object_store from 0.12.5 to 0.13.2 (#12354)
* chore(cargo): bump object_store from 0.12.5 to 0.13.2 Bumps [object_store](https://github.com/apache/arrow-rs-object-store) from 0.12.5 to 0.13.2. - [Changelog](https://github.com/apache/arrow-rs-object-store/blob/main/CHANGELOG-old.md) - [Commits](https://github.com/apache/arrow-rs-object-store/compare/v0.12.5...v0.13.2) --- updated-dependencies: - dependency-name: object_store dependency-version: 0.13.2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * fix(pnpr): import ObjectStoreExt for object_store 0.13 method move object_store 0.13 moved get/put/delete off the ObjectStore trait onto the new ObjectStoreExt trait. Import it so the S3 hosted-store backend keeps compiling. --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
sijie-Z
|
3d1fd202f9 |
fix(bins.linker): skip redundant .exe warning when hardlink is already correct (#12284)
Closes [pnpm/pnpm#12203](https://github.com/pnpm/pnpm/issues/12203). On Windows, `node` is linked by hardlinking `node.exe` directly rather than through a cmd-shim. The early-exit in `linkBin()` only recognized an existing cmd-shim, so on every warm install the existing (already correct) `node.exe` was treated as a conflict: pnpm warned `The target bin directory already contains an exe called node` and then removed and re-linked it. Because many commands re-link `node`, the warning was spammed. ### `@pnpm/bins.linker` Before warning and replacing an existing `.exe`, check whether it already refers to the link target via a new `isSameFile()` helper: - Compares the OS file identity (inode/device), read as `BigInt` to avoid the precision loss NTFS 64-bit file IDs suffer when cast to a `Number`. A zero/unreliable inode (common on Windows) is not treated as a match. - Falls back to a streaming, chunked (64 KB) content comparison when identity can't be established, so a byte-identical copy still counts as the same file without ever buffering a whole executable. A read error during the comparison is treated as "not the same file" so it can never abort bin linking. The early-return is scoped to the `node.exe` path, so non-`node` commands still fall through to the existing warn + remove + `cmdShim()` regeneration and never end up with a partially populated bins directory. ### pacquet pacquet never emitted this warning, but its Windows `link_node_bin` unconditionally removed and re-linked `node.exe` on every install. Ported the same same-file early-return to `cmd-shim` so warm installs skip that churn, using `same_file::Handle` for the cheap identity check (promoted from a transitive to a workspace dependency) with the same chunked content fallback. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
70f1343d9a |
feat(ecosystem-e2e): ecosystem test harness across pnpm/pacquet and node_modules layouts (#12439)
* feat(ecosystem-e2e): add harness installing real JS stacks across pnpm/pacquet and layouts
Adds `pacquet/tasks/ecosystem-e2e`, a Yarn-PnP-style ecosystem test that
installs, builds, and serves real framework scaffolds (Next.js, Vite,
Angular, Astro, SvelteKit, Nuxt, React Router 7) across the cross product
of {pnpm, pacquet} x {isolated, global virtual store}.
The binary axis catches pnpm/pacquet parity gaps; the layout axis catches
breakage introduced by the global virtual store. Each cell scaffolds the
project once, installs with the binary under test, runs the build, then
boots the production server and probes it over HTTP — so a green cell means
the produced node_modules works at runtime, not just at bundle time.
Runs on a daily cron (one job per stack), not per-PR: the installs are slow
and track upstream framework releases, so a red cell is investigated rather
than treated as a merge blocker.
* style(ecosystem-e2e): use ASCII ellipsis in comments for dylint
* fix(ecosystem-e2e): address PR review findings
- workflow: wrap `pnpm/bin/pnpm.cjs` for the `--pnpm` shim; the built bundle
is `dist/pnpm.mjs`, so the previous `dist/pnpm.cjs` path never existed and
the shim would have failed to launch the repo-built pnpm.
- keep: `--keep` now reuses an already-scaffolded template instead of
re-running the generator into a non-empty directory (which fails).
- serve: retry HTTP 4xx/5xx until the deadline rather than failing on the
first one — dev servers can briefly answer error statuses while warming up.
- serve: the probe reads only the status line, not the whole response body.
- stacks: pin the create-astro, sv, nuxi, and create-react-router generators
to a major version instead of tracking the latest tag, matching the stated
reproducibility intent for the scheduled run.
* fix(ecosystem-e2e): scrub subprocess env, bound CI job, fix README fence
- security: scaffold/install/build/serve run third-party lifecycle code, so
they now launch via `sandboxed_command`, which clears the inherited
environment down to a small allowlist (PATH, HOME, proxy/cert/locale vars,
plus PORT/HOST for servers). An unattended dependency build script can no
longer read ambient CI secrets such as the workflow token.
- workflow: add a 60-minute job timeout so a hung build or serve subprocess
can't pin a runner indefinitely.
- docs: give the README's grid code block a language identifier (MD040).
* fix(ecosystem-e2e): spawn serve argv directly and reset cell log per run
- serve: run the tokenized serve command directly off the project's
node_modules/.bin instead of joining tokens into a `sh -c` string, so
argument boundaries survive and a token that needs quoting can't break the
launch. The spawned PID is the server itself, so teardown still works.
- log: truncate cell.log at the start of each run so reruns (notably under
--keep) don't interleave with stale output.
|
||
|
Zoltan Kochan
|
2b81344605 |
feat(default-reporter): render pnpm-identical visual install output in pacquet (#12431)
* feat(default-reporter): render pnpm-identical visual install output Add a `pacquet-default-reporter` crate that renders the same terminal output as pnpm's `@pnpm/cli.default-reporter` for install/add/update/ remove — the live progress line, the packages diff, lifecycle script output, and the "Done in ..." footer — and make it pacquet's default reporter. The reporter folds the existing `pnpm:*` log events into a mutex-guarded state machine that recomputes the terminal frame, replacing pnpm's RxJS graph (which only exists because pnpm's reporter is a separate process). It renders in place on a TTY and append-only otherwise. A new `pnpm:execution-time` channel drives the "Done in" footer. The footer reads "using pacquet v<version>" rather than "using pnpm v..." so pacquet does not misreport the tool identity; everything else matches pnpm's output. * fix(default-reporter): address PR review — fast-path footer, throttling, append-only flag - Emit `pnpm:execution-time` on the up-to-date fast path too, so the `Done in ...` footer shows there as well as on the full install path. - Throttle high-volume progress redraws (200ms in place, 1s append-only), mirroring pnpm's `throttleProgress`; non-progress events still render immediately and the final frame is always forced. - Add `--reporter=append-only`, forcing append-only rendering on a TTY, matching pnpm's reporter values. - Borrow in the `is_install_family` `matches!` and clarify the `contains_path` doc (it intentionally mirrors pnpm's substring `String.includes`, not segment matching). * fix(package-manager): only emit pnpm:root added for newly-linked direct deps `link_one_importer` emitted a `pnpm:root added` event for every direct dependency it symlinked, including ones whose symlink already pointed at the target. With the default reporter on, that made `pacquet add <new>` list every already-installed dependency in the install summary instead of only the new one. pnpm skips the event for reused symlinks (`if ((await symlinkDependency(...)).reused) return` in linkDirectDeps.ts). Thread the `reused` flag that `force_symlink_dir` already returns out through `symlink_package` and gate the emit on it, so the summary lists only dependencies whose symlink was actually created this run. |
||
|
Zoltan Kochan
|
5f63458644 |
fix: bound descendant-process lookup on error exit to avoid a Windows hang (#12403)
## Problem On Windows, **any failed `pnpm` command hangs 20–46 seconds before exiting.** The error handler (`pnpm/src/errorHandler.ts`) enumerates descendant processes via `pidtree` to terminate them on every error exit. On Windows `pidtree` shells out to `wmic` and, where wmic has been removed, a PowerShell `Get-CimInstance Win32_Process` fallback — a process listing that takes tens of seconds on busy CI runners. This also broke Windows CI: the `verifyDepsBeforeRun/*` e2e suites are full of intentional-failure assertions (e.g. `pnpm start` with `--config.verify-deps-before-run=error` when deps aren't installed). Each failure paid the ~23 s error-handler tax, so the suite blew past the 70-minute cap. `pnpm install` and success paths never hit the error handler, which is why only failures were slow. Diagnosed by sampling `process.getActiveResourcesInfo()` during the hang: it showed a lingering `ProcessWrap` (a spawned child), and hooking `child_process.spawn` named it (`wmic` → `powershell … Get-CimInstance Win32_Process`, exiting after ~23–46 s). ## Fix Race the descendant-process lookup against a 2 s timeout. If it doesn't return in time, skip the kill and exit — `exit()` calls `process.exit`, which abandons the still-running (harmless, read-only) process query instead of blocking on it. The fast path (Unix, fast Windows) is unchanged. Confirmed on Windows CI: the failing `start` invocations dropped from **~23 s to ~2.7 s**, and `multiProjectWorkspace.ts` went from **716 s to 124 s**. ## Also included The CI pnpr-binary cache is split into `restore` + an explicit `save` step that runs right after the build, so a failing test step no longer discards the ~20-minute Rust build (the combined `actions/cache` only saved in a post-job step that gets skipped on failure). |
||
|
Zoltan Kochan
|
681b593eb2 |
fix: support scope-specific registry auth tokens (#12392)
pnpm can now use different auth tokens for different package scopes, even when those scopes use the same registry URL. Previously, auth was selected only by registry URL. If `@org-a` and `@org-b` both used `https://npm.pkg.github.com/`, they had to share the same token. This caused problems for registries that issue tokens per organization or per scope. Configure a scope-specific token by adding the package scope after the registry URL in the auth key: ```ini @org-a:registry=https://npm.pkg.github.com/ @org-b:registry=https://npm.pkg.github.com/ //npm.pkg.github.com/:@org-a:_authToken=${ORG_A_TOKEN} //npm.pkg.github.com/:@org-b:_authToken=${ORG_B_TOKEN} //npm.pkg.github.com/:_authToken=${FALLBACK_TOKEN} ``` `pnpm login --registry=https://npm.pkg.github.com --scope=@org-a` writes the token to the same scope-specific auth key. When installing or publishing `@org-a/*`, pnpm uses `ORG_A_TOKEN`. For `@org-b/*`, pnpm uses `ORG_B_TOKEN`. Packages without a matching scope continue to use the registry-wide fallback token. |
||
|
kimulaco
|
2da044434e |
fix(pacquet): preserve exec bit on copy fallback (#12385)
## Summary Fixes #12171. pacquet stores executable CAFS entries under paths ending in `-exec`, but the copy fallback tier in `link_file.rs` materialized them without the executable bit. When hardlink/reflink isn't available and the install falls back to `fs::copy` (e.g. overlayfs on CI), a native binary such as `@esbuild/linux-x64/bin/esbuild` lands at `0o644` and fails to spawn with `EACCES`. This routes all copy-tier imports through one `copy_file` helper. On Unix, when the CAS source path ends with `-exec`, it OR-s the exec bits onto the copied file, mirroring `git-fetcher`'s existing `materialize_into`. Non-executable files are left exactly as `fs::copy` produced them. #12177 (closed) took the same direction but re-read and re-applied the source's full mode to every copied file. This version only touches files the store marked executable (`-exec` suffix), so it never widens a restrictive mode (e.g. `0o600` → `0o711`) and adds no `set_permissions` syscall on the non-exec majority. Rather than duplicate that logic, the `-exec` check now lives in a shared `pacquet_fs::file_mode::cas_path_is_executable` that both copy paths call, replacing the private copy in `cas_io.rs`. This is a pacquet-only bug. pnpm preserves the exec bit on its own copy path, so no pnpm-side change is needed. |
||
|
Zoltan Kochan
|
29981f663b |
fix(pacquet): refresh lockfile when catalogs change (#12382)
## Summary - detect catalog drift between `pnpm-workspace.yaml` and the lockfile `catalogs:` snapshot - record and compare catalogs in pacquet's workspace-state fast path so plain `pacquet install` does not skip after catalog edits - add lockfile, repeat-install, and CLI regression coverage for catalog changes |
||
|
dependabot[bot]
|
80af95a6f2 |
chore(cargo): bump chrono from 0.4.44 to 0.4.45 (#12353)
Bumps [chrono](https://github.com/chronotope/chrono) from 0.4.44 to 0.4.45. - [Release notes](https://github.com/chronotope/chrono/releases) - [Changelog](https://github.com/chronotope/chrono/blob/main/CHANGELOG.md) - [Commits](https://github.com/chronotope/chrono/compare/v0.4.44...v0.4.45) --- updated-dependencies: - dependency-name: chrono dependency-version: 0.4.45 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Zoltan Kochan
|
d2b42c2dfc |
fix(pacquet): per-level preferred-version fold + all-importers hoist rounds (#12357)
## Summary Two parity changes for pacquet's resolver, continuing https://github.com/pnpm/pnpm/issues/12266. Measured on the monorepo (fresh state, `install --lockfile-only`, back-to-back vs **pnpm 11.6.0**), the real-lockfile document diff drops from **128 to 5 changed lines** (re-measured after rebasing over #12361/#12362: **132 → 11**, where 8 of the 11 are a divergence the pacquet side of #12362 itself introduced — see the analysis on pnpm/pnpm#12266 — and 3 are the known cycle-closing-edge gap). ### 1. Per-level preferred-version fold pnpm extends the preferred-versions map per resolution level: after a package's direct dependencies settle, their `(name, version)` pairs join the map the *children's* subtree resolutions pick against ([resolveDependencies.ts#L717-L746](https://github.com/pnpm/pnpm/blob/ce9c096e8e/installing/deps-resolver/src/resolveDependencies.ts#L717-L746)). So `signed-varint`'s `varint@~5.0.0` dedupes to the `varint@5.0.0` its parent pinned as a sibling instead of drifting to `5.0.2`. pacquet picked against a static seed only; besides `varint`/`es-abstract`, this turned out to drive the remaining `jest`/`@types/node` duplicate variants too. - The walk resolves a whole sibling level before any child subtree starts (upstream's postponed-resolution barrier): `resolve_node` splits into `resolve_node_seed` + `walk_node_children`. - Each level layers its versions onto a new `PreferredVersionsOverlay` (O(1) `Arc`-chained layers in `resolver-base`); the npm picker folds the per-name view in as plain `version` selectors at both registry seams. - The overlay's per-name view joins the per-wanted dedup cache key; lockfile-reuse subtrees keep the no-overlay path (exact pins). ### 2. Hoist rounds across all importers (deterministic barrier, same logic as pnpm) pnpm resolves **every importer's initial wave before any peer hoist**, then repeats global hoist rounds (per round: each importer's required-peer loop to a fixpoint, then one optional-peer hoist) until no importer hoists ([resolveDependencies.ts#L335-L445](https://github.com/pnpm/pnpm/blob/ce9c096e8e/installing/deps-resolver/src/resolveDependencies.ts#L335-L445)). pacquet ran each importer's whole hoist loop before the next importer's initial wave, so an early importer's optional-peer pick couldn't see versions a later importer resolves — `@cyclonedx`'s `spdx-expression-parse` hoisted `3.0.1` where pnpm's barrier-visible map picks `4.0.0`. `resolve_importer_with_workspace` is now an `ImporterHoistState` (`init` / `run_required_round` / `hoist_optional_round`) driven by `resolve_workspace` in upstream's exact phase order. Both implementations are deterministic here; the rule is identical. ## Verification - New regression test `child_resolution_prefers_parent_level_sibling_versions` (fails with the fold disabled) + full `resolving-*`, `package-manager`, `cli` suites: 1,242 tests pass; clippy `--deny warnings`, rustfmt, typos clean. - Whole-monorepo diff vs fresh pnpm 11.6.0: 128 → 5 changed lines; consecutive pacquet runs byte-identical. |
||
|
Zoltan Kochan
|
43b5bf7520 |
perf(pacquet): cache build metadata during installs (#12360)
* perf(pacquet): cache build metadata during installs * fix(pacquet): satisfy clippy for build helpers * fix(pacquet): reduce prefetch row type complexity |
||
|
dependabot[bot]
|
52148e6916 |
chore(cargo): bump tabled from 0.20.0 to 0.21.0 (#12352)
Bumps [tabled](https://github.com/zhiburt/tabled) from 0.20.0 to 0.21.0. - [Changelog](https://github.com/zhiburt/tabled/blob/master/CHANGELOG.md) - [Commits](https://github.com/zhiburt/tabled/commits) --- updated-dependencies: - dependency-name: tabled dependency-version: 0.21.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Alessio Attilio
|
a9d2ec8817 |
feat(pacquet): support pnpmfile custom resolver hooks (adapter, IPC, chain integration, force-refresh) (#12153)
Port pnpm's custom resolver hooks to the Rust pacquet engine: a pnpmfile can export a top-level `resolvers` array whose entries override built-in dependency resolution and force re-resolution when needed. See pnpm/pnpm#10389 for the TypeScript-side feature request that motivated this port. ## What's included - **Hook contract** — `CustomResolver` trait (`canResolve` / `resolve` / `shouldRefreshResolution`) mirroring `hooks/types/src/index.ts`. All three methods are optional upstream, so the Node worker reports per-resolver capability flags in one IPC round trip and pacquet skips calls a resolver doesn't implement (mirrors pnpm's `if (!customResolver.canResolve || !customResolver.resolve) continue` and `checkCustomResolverForceResolve`'s hook filter). - **Node IPC** — the long-lived pnpmfile worker gained `resolvers` (capabilities) and `resolver` (method invocation) requests. Methods are invoked with `this` bound to the resolver object, like pnpm. Pending-request cleanup is cancellation-safe via an RAII guard. - **Adapter & chain integration** — `CustomResolverAdapter` bridges the JSON hook contract to the typed `Resolver` trait. Custom resolvers are built into the inner resolver chain ahead of the built-ins (upstream chain priority), inside the prefetching/observing wrappers so their tarball results get resolve-time prefetch and pnpr streaming. `canResolve` results are memoized keyed `alias@bareSpecifier`, exactly like pnpm's `getCustomResolverCacheKey`. A resolver-returned `manifest` passes through (pnpm spreads the whole hook result). Payloads match upstream: `prevSpecifier`, and resolve opts carry `lockfileDir` / `projectDir` / `preferredVersions` / `currentPkg`. - **`shouldRefreshResolution` semantics** — port of `checkCustomResolverForceResolve`: the hook receives the merged packages+snapshots entry (pnpm's in-memory `PackageSnapshot`), checks run concurrently with first-true/first-error short-circuit, and a throwing hook aborts the install (`PNPMFILE_FAIL`). A `true` verdict defeats both up-to-date optimizations, as documented in the hook's contract: - the prefer-frozen dispatch consults the hook (pnpm: `forceResolutionFromHook` → `needsFullResolution` blocks `isFrozenInstallPossible`) and routes to the fresh-resolve path with lockfile reuse disabled (`UpdateReuseScope::None`); - the optimistic repeat-install fast path now ports the pnpmfile branch of `patchesOrHooksAreModified`: the workspace state records the loaded pnpmfile list, and an added/removed/edited pnpmfile invalidates the mtime check. - **`CurrentPkg`** — added to `ResolveOptions`, matching upstream's `currentPkg` shape `{id, name?, version?, resolution, publishedAt?}` (camelCase). ## Tests - Adapter unit tests: missing `id`/`resolution`, invalid shapes, `canResolve` memoization, payload shapes, manifest passthrough. - `check_custom_resolver_force_resolve` unit tests: port of upstream's `checkCustomResolverForceResolve.ts` suite (capability filter, true/false/error propagation, merged snapshot payload). - Node IPC integration tests against a real pnpmfile: capabilities, `this` binding, round trips, error propagation, cancellation cleanup. - CLI e2e tests against the mock registry: custom resolver precedence over the npm resolver, `shouldRefreshResolution` re-resolving past an up-to-date lockfile, and a throwing hook failing the install. |
||
|
Zoltan Kochan
|
f648e9b7c4 |
fix: contain hoisted dependency aliases (GHSA-fr4h-3cph-29xv) (#12343)
* fix: contain hoisted dependency aliases (GHSA-fr4h-3cph-29xv) The `nodeLinker: hoisted` install restores its dependency graph straight from the lockfile via `lockfileToHoistedDepGraph`, which joins each dependency alias under a `node_modules` directory and imports the package files there. On a frozen / up-to-date lockfile, resolution is skipped entirely, so the alias validation added for the resolution path never runs. A crafted lockfile alias such as `../../../escape` could therefore escape the install root, and reserved aliases such as `.bin`, `.pnpm`, or `node_modules` could overwrite pnpm-owned layout. Validate every alias at the hoisted-graph directory sink. The shared `safeJoinModulesDir` helper now rejects aliases that are not valid npm package names (path-traversal, absolute, and reserved names) in addition to its containment check, and the hoisted graph routes its `dep.name` sink through it. Pacquet mirrors the boundary: `safe_join_modules_dir` validates the hoister's `dep.0.name` before adding the graph node or recursing, reusing the same dependency-name rule it already applies to direct-dependency aliases. Both stacks surface `ERR_PNPM_INVALID_DEPENDENCY_NAME`. --- Written by an agent (Claude Code, claude-fable-5). * fix: reject invalid dependency aliases at the lockfile verification gate Add an always-on, policy-independent structural check to verifyLockfileResolutions that rejects any importer or package-snapshot dependency alias that is not a valid npm package name. A dependency alias becomes a `node_modules/<alias>` directory at link time, so an alias with path-traversal segments or a reserved name (`.bin`, `.pnpm`, `node_modules`) could escape the install root or overwrite pnpm-owned layout. This complements the linker-sink guards: the verifier runs before any fetch or filesystem work and covers every node linker at once, while the sink guards still protect the `trustLockfile` path the verifier skips. The check runs before the cache lookup so a record written by a version that predates the rule cannot fast-path around it, and before the `packages` guard so a tampered importer alias is caught even when nothing is installed. `isValidDependencyAlias` is now exported from `@pnpm/installing.deps-resolver` and reused here. Pacquet mirrors the gate in its lockfile-verification crate with a matching `ERR_PNPM_INVALID_DEPENDENCY_NAME` verdict. --- Written by an agent (Claude Code, claude-fable-5). * docs(package-manager): drop redundant explicit intra-doc link target `is_valid_dependency_alias` is in scope via `use`, so the bare intra-doc link resolves on its own. The explicit path target tripped `rustdoc::redundant-explicit-links` under the CI Doc job's `cargo doc --document-private-items` (the local pre-push hook runs `cargo doc` without that flag, so it didn't surface). --- Written by an agent (Claude Code, claude-fable-5). * refactor(lockfile-verification): fold the alias check into the single candidate pass The dependency-alias check ran as its own full traversal of the lockfile in addition to collectCandidates' existing pass over every package snapshot. Fold it into that pass instead: collectCandidates now also validates each importer and snapshot dependency alias and returns the invalid ones alongside the resolution-shape violations, so the lockfile is walked once per verification rather than twice. Because collectCandidates runs after the verification-cache lookup, the alias check is now covered by the cache the same way the resolution-shape check is: a new dependencyAliasCheck cache identity makes a record written before this rule existed fail canTrustPastCheck, forcing a re-verification. The shared helper is renamed withOfflineCheckCacheIdentities and appends both offline-structural-check identities. No behavior change for valid lockfiles; the same ERR_PNPM_INVALID_DEPENDENCY_NAME is thrown for invalid aliases. Mirrored in pacquet's lockfile-verification crate. --- Written by an agent (Claude Code, claude-fable-5). * refactor: declare pushInvalidAliases after its caller, trim duplicated comments Move `pushInvalidAliases` below `collectCandidates`, following the repo's declare-after-use convention. Collapse the repeated "an alias becomes a node_modules directory, so a traversal/reserved name escapes or overwrites layout" explanation that was copied across the verifier, the hoisted-graph error, and the pacquet mirror down to a single reference each — the full rationale lives once in the validating sink (`safeJoinModulesDir` / `safe_join_modules_dir`) and the user-facing error hints. --- Written by an agent (Claude Code, claude-fable-5). |
||
|
Zoltan Kochan
|
66a9078060 |
fix: route pacquet scoped packages through scoped registries (#12340)
* fix: route pacquet scoped packages through scoped registries * fix: satisfy pacquet add test clippy * test: seed scoped registry in modules yaml fixture * test: stabilize latency proxy slow start timing |
||
|
Victor Sumner
|
61810aa684 |
feat: add --frozen-store for installs against a read-only store (#12190)
## What
Adds an opt-in `frozenStore` / `--frozen-store` setting (default `false`) that lets `pnpm install --offline --frozen-lockfile` run against a package store that lives on a **read-only filesystem** — a Nix store, a read-only bind mount, an OCI layer.
## Why
A normal install fails against such a store **not** because it writes package content, but because it unconditionally:
1. opens the SQLite `index.db` in WAL mode, which needs to create `-shm`/`-wal` sidecars in the store directory; and
2. writes a project-registry entry under the store.
Both fail with `attempt to write a readonly database` / `EROFS` on a read-only store directory, even when the store is complete and the lockfile is frozen. This blocks any deployment that wants an immutable, content-addressed store.
## How
When `frozenStore` is enabled, pnpm opens `index.db` through the SQLite **`immutable=1`** URI — which tells SQLite the file cannot change underneath it, so it bypasses the WAL/`-shm` sidecar machinery entirely and reads the raw file with zero sidecar creation — and suppresses every store-write path. Pair it with `--offline --frozen-lockfile` against a fully-populated store. It is incompatible with two settings that would write into the store, and each throws a clear config-conflict error before any network or store access: **`--force`** (which bypasses the no-write-on-hit skip → `ERR_PNPM_CONFIG_CONFLICT_FROZEN_STORE_WITH_FORCE`) and a configured **pnpr server** (which would fetch and write packages into the store → `ERR_PNPM_FROZEN_STORE_INCOMPATIBLE_WITH_PNPR`).
> Plain `SQLITE_OPEN_READ_ONLY` is **not** sufficient: opening a WAL-mode db read-only still tries to create the `-shm` sidecar, which fails on a read-only *directory*. `immutable=1` is the load-bearing piece.
> **Node.js requirement:** `node:sqlite` only passes `SQLITE_OPEN_URI` to SQLite (so the `immutable=1` query is honored rather than treated as part of a literal filename) starting in **v22.15.0** (22.x line), **v23.11.0**, and every **v24+**. pnpm's `engines` floor is `>=22.13`, so on a runtime older than that the frozen open is detected up front and fails with a clear `ERR_PNPM_FROZEN_STORE_UNSUPPORTED_NODE` instead of SQLite's cryptic "unable to open database file". (pacquet uses rusqlite with an explicit `SQLITE_OPEN_URI` flag, so it has no such floor.)
> The `immutable=1` URI path is also percent-encoded (`%`→`%25`, `?`→`%3f`, `#`→`%23`, in that order, leaving `/` literal) so a store path containing those characters doesn't truncate the path or inject a spurious query parameter — applied identically in both stacks.
### Build backstop under the global virtual store
Under the global virtual store (default), a package's directory lives **inside** the store (`{storeDir}/links/...`). Applying a patch or running an allowlisted lifecycle script writes into that directory — so on a frozen store it would crash mid-build with a raw `EROFS`. A fully-seeded store never reaches the build step (patched/built packages are imported from the side-effects cache and filtered out by the `isBuilt` gate), so any residual build candidate means the seed is **missing that package's build output**.
`buildModules` now refuses up front with `ERR_PNPM_FROZEN_STORE_NEEDS_BUILD` and actionable guidance ("rebuild the seed with their scripts enabled, or remove them from `onlyBuiltDependencies`") instead of failing cryptically once a script starts. The check is gated on the global virtual store — under the isolated linker, slot directories live in the writable project-local store, so builds there are fine. Non-allowlisted scripts never run, so they are not treated as a blocking write.
Bin-linking has its own read-only-store edge under the global virtual store. On a **warm** checkout (the project's `.bin/<name>` already points at the seed target) `linkBin` returns before touching the store, so it is write-free. But on a **fresh** checkout it (re)creates the bin and calls `fixBin`, whose `chmod` targets the bin's **source file inside the store** (`{storeDir}/links/...`) — which is refused with `EPERM`/`EACCES` on a read-only store, even though a complete seed already ships that bin executable (so the `chmod` is redundant). `@pnpm/bins.linker` now wraps that call in `ensureExecutable`: it swallows the refusal when the target is already executable and rethrows otherwise, so bin-linking is write-free against a frozen store on a cold checkout too, while a genuinely non-executable bin (a broken seed) still surfaces as an error.
The blocking predicate distinguishes the two write kinds: a **patch** is applied regardless of `ignoreScripts`, so a patched package is always blocked; a **lifecycle script** is suppressed under `ignoreScripts`, so an allowlisted build-requiring package is *not* blocked when scripts are off (it would write nothing). This avoids falsely rejecting a valid `--ignore-scripts` frozen install. **Optional dependencies are exempt**: a build or patch failure on an optional dependency is non-fatal at runtime, so a seed missing an optional package's build output skips that build (emitting the `skipped-optional-dependency` log) instead of blocking the install — in both stacks.
### Both stacks (parity rule)
**TypeScript pnpm CLI**
- Config plumbing (`@pnpm/config.reader`): `frozen-store` type, config-file key, default, `Config.frozenStore`.
- The read-only open branch (`@pnpm/store.index`): `immutable=1`, read-only statements, throwing mutators.
- Wiring through `@pnpm/store.controller` and `@pnpm/store.connection-manager` to the sole `StoreIndex` construction site.
- Gating the project-registry write (`@pnpm/installing.context`).
- The `--force` / pnpr-server conflict guards (`@pnpm/installing.deps-installer`) and CLI surface (`@pnpm/installing.commands`).
- **After-install rebuild** (`@pnpm/building.after-install`): the post-install rebuild opens its `StoreIndex` immutably under the flag, so re-reading the store for a rebuild never attempts a writable open against the frozen store.
- **Worker fix:** `@pnpm/worker` opens its *own* writable `StoreIndex` on every `readPkgFromCafs` cache hit, so a pure read crashed on a frozen store. `frozenStore` is threaded through to `getStoreIndex` and keyed into its connection cache.
- **Build backstop** (`@pnpm/building.during-install`): `buildModules` throws `ERR_PNPM_FROZEN_STORE_NEEDS_BUILD` for a GVS slot that would build/patch on a frozen store, honoring `ignoreScripts`; threaded from `@pnpm/installing.deps-installer`.
- **Bin-linking on a read-only store** (`@pnpm/bins.linker`): `linkBin` wraps the `fixBin` chmod in `ensureExecutable`, which tolerates `EPERM`/`EACCES` when the bin's store-resident source is already executable (a complete seed) and rethrows otherwise — so a fresh checkout against a frozen store links bins without crashing on the redundant chmod. Catch-on-failure keeps the writable hot path at zero added syscalls.
**Rust pacquet**
- A dedicated `open_immutable` / `shared_immutable_in` opens via `immutable=1`, selected only under the flag. Plain `open_readonly` keeps the ordinary `SQLITE_OPEN_READ_ONLY` open (WAL locking intact) because normal installs read the index while the same process's `StoreIndexWriter` writes it concurrently — an immutable connection skips all locking and change detection, so a concurrent writer would make those reads undefined.
- `--frozen-store` CLI flag + `frozenStore` workspace-yaml setting.
- The store-index writer is replaced with a drain-and-drop stub (`spawn_disabled`) and `init_store_dir_best_effort` is skipped under the flag.
- **Build backstop:** `build_modules` returns `BuildModulesError::FrozenStoreNeedsBuild` (`ERR_PNPM_FROZEN_STORE_NEEDS_BUILD`) under the same GVS + frozen-store condition, threaded from `config.frozen_store`. The gate keys off `should_run_scripts` (which already folds the allow-build policy), so it is correct without an explicit ignore-scripts branch — pacquet has no configurable ignore-scripts mode yet.
pacquet already separated read-only index access (`shared_readonly_in`, or `shared_immutable_in` under the flag) from writes, so it never had the worker-conflation bug; the flag makes the "no writes attempted" contract explicit and gates the remaining best-effort write attempts.
## Testing
- **TS:** `store.index` frozen-mode-on-`0555`-directory test (reads work, writes throw `ERR_PNPM_FROZEN_STORE_WRITE`) plus a path-with-`?` open test — both gated on the runtime's immutable-URI support, with a complementary test asserting `ERR_PNPM_FROZEN_STORE_UNSUPPORTED_NODE` fires where that support is absent (the CI Node 22.13.0 path); `config.reader` round-trip; `deps-installer` `--force` and pnpr-server conflict guards; `worker`/`package-requester` unit tests. End-to-end on a `chmod -R 0555` store: install succeeds, `node_modules` materializes, no `-shm`/`-wal`/`-journal` sidecars; negative control without the flag fails as expected; incomplete store → clean offline error.
- **pacquet:** `open_immutable_reads_wal_db_on_readonly_directory` unit test plus the `immutable_sqlite_uri` encoding test and a path-with-`?` open test; yaml + CLI fold tests; **integration test** `frozen_store_installs_against_a_read_only_store` — primes a store, `chmod 0555` the tree, runs `install --frozen-lockfile --frozen-store --offline`, asserts success + materialized `node_modules` + zero sidecars. Confirmed load-bearing by reverting the `immutable=1` fix (test then fails).
- **Build backstop (both stacks):** `building/during-install` unit tests — approved-build-not-cached and patched-not-cached refuse; cached, non-allowlisted, and non-GVS cases pass through; **approved-build-under-`ignoreScripts` passes through while patched-under-`ignoreScripts` still refuses** — and the matching pacquet `build_modules` tests (`frozen_store_gvs_patch_not_seeded_refuses` + GVS-off / frozen-off controls). Each confirmed load-bearing by disabling the relevant guard and watching the corresponding test fail.
- **Bin-linking (`bins/linker`):** `ensureExecutable` tests with `fixBin` mocked to reject with `EPERM` — an already-executable bin source resolves (and `fixBin` is asserted called, so it isn't the warm skip-guard passing), a non-executable one rethrows `EPERM`. Confirmed end-to-end by running the built `linkBins` against a real `chflags uchg`-immutable store: the executable-seed case resolves and links the bin, the non-executable-seed control throws `EPERM`.
A changeset is included with `"pnpm": minor` and `"@pnpm/bins.linker": patch` (the read-only-store bin-linking fix).
|
||
|
Zoltan Kochan
|
cb18695c5b |
perf(pacquet): lazy packument hydration, sharded meta cache, and an indexed metadata mirror (#12322)
## Why Profiling a warm babylon resolve (metadata mirrors hot, ~520 MB across 1,476 packuments) showed the dominant cost was not I/O but **hydration**: every version of every packument was deserialized into typed manifests — maps, strings, and `serde_json::Value` trees built, hashed, and dropped — even though a pick consults only the version *strings* plus the handful of manifests it actually considers. typescript ships 3,800 versions; a pick needs one. The `#[serde(flatten)]` catch-all on `PackageVersion` compounded it by routing the whole struct through serde's buffering deserializer. The same hydration cost was paid on cold resolves inside the `spawn_blocking` parses from pnpm/pnpm#12318. ## What Three changes, applied in profile-driven order: **1. Lazy packument hydration** (`df6f70eb57`). `Package::versions` becomes a `PackageVersions` map whose entries hold the raw JSON fragment serde captured (`Arc<RawValue>`, shared not copied) and hydrate into `Arc<PackageVersion>` on first access, cached per slot. Key scans never hydrate; `pinned_version` hydrates only the winner; the publish-date filter moves slots without touching manifests; undecodable fragments degrade to "version absent" (matching the JS implementation's tolerance); serialization re-emits raw fragments verbatim. Picked manifests travel as `Arc<PackageVersion>`. Enables serde_json's `raw_value` feature (already a workspace dep). **2. Sharded in-memory packument cache** (`4c28c6679e`). Every resolve edge consults the shared meta cache, and its single `Mutex<HashMap>` was the top mutex-wait site in the profile; it is now the same `DashMap` shape the crate's other shared maps use. Honest note: wall time was unchanged by this alone — the post-hydration profile shows the warm resolve is critical-path-bound, not lock-bound — but the contention disappears and the cache no longer serializes workers under load. **3. Indexed on-disk metadata mirror** (`126a416ae8`, maintainer-approved cache-format break). The two-line NDJSON mirror (headers + verbatim body) becomes: ``` pacquet-meta-v1 <headers_len> <index_len>\n <headers JSON> etag, modified <index JSON> name, dist-tags, time, homepage, versions: [version, offset, len] <fragments> concatenated raw per-version JSON ``` The loader reads the file once and hands `PackageVersions` byte spans into that buffer — no structural re-scan, hydration parses a slice in place. The writer streams the registry's own bytes (fragments borrow from the lazily-parsed response body), so the **cold-install cost stays one temp-file + rename per package**. Old-format files read as cache misses and are rewritten on the next 200. A span-per-fragment `pread` variant was tried first and measured *worse* (sys 0.4 s → 4.5 s; the pick paths probe many candidate fragments per package), hence the single-buffer design. ## Measurements (warm babylon `--lockfile-only` resolve, 10-core M-series) | build | wall | notes | |---|---|---| | before this PR | ~9.1 s | malloc/free + `deserialize_any` dominate the profile | | + lazy hydration | ~7.7–8 s | parse microbench 3–3.8× (`typescript` 36.1→9.5 ms) | | + indexed mirror | **~5.5–6.7 s** | sys 0.4 s; no whole-body scan | Cold resolves keep the same hydration savings inside the parse tasks; cold write volume and syscall pattern are unchanged. ## Evaluated and deliberately not included Consolidating the install-phase thread pools (tokio + global rayon + the dedicated CAS-write pool + capped blocking threads show ~100k involuntary context switches on cold installs vs ~750 for the pnpm CLI). After the resolution fixes, repeated container A/Bs show no measurable wall-clock cost from the churn — it hides entirely behind network time — and history warns that speculative concurrency reshuffles here regress badly (see the #11903 prefetch revert). Deferred until a benchmark shows it on the critical path. ## Tests New `package_versions` unit tests (hydrate-on-demand + Arc-identity caching, undecodable-fragment-as-absent, verbatim raw round-trip incl. unknown keys, eager construction, hydration-free filtering) and rewritten `mirror` tests (headers/index/fragment round-trip, span hydration, truncation → cache miss, old-format → cache miss, atomic overwrite). Full suites green: `pacquet-registry` (23), `pacquet-resolving-npm-resolver` (235), `pacquet-package-manager` + `pacquet-cli` (768); workspace clippy `-D warnings` (pedantic set), dylint, fmt. |
||
|
Zoltan Kochan
|
3d50680eda |
fix(security): verify Node.js runtime SHASUMS OpenPGP signature (#12295)
Follow-up to #12292 (which verifies the **package-manager** binary). This closes the same class of gap for the **Node.js runtime**. When a repository requests a Node.js runtime — `devEngines.runtime: node@X` (with `onFail: download`, the default) or `useNodeVersion` — pnpm downloads and then executes a Node binary (it's used to run lifecycle / `run` / `exec` scripts). The download **mirror is repository-configurable** via `node-mirror:<channel>` (`nodeDownloadMirrors`) in project `.npmrc`, and the integrity comes from `SHASUMS256.txt` fetched **from that same mirror**. That's a circular check: a malicious mirror serves a tampered `node` tarball **and** a matching `SHASUMS256.txt`, the sha256 check passes, and pnpm runs the binary. Drive-by on a normal command in a cloned repo. ## Fix pnpm now fetches `SHASUMS256.txt.sig` and verifies its **detached OpenPGP signature** against the **Node.js release team's public keys, embedded in the pnpm CLI**, before trusting the hashes. A mirror that serves a tampered binary cannot also produce a valid signature, so verification fails. Any faithful mirror (one that proxies the real signed SHASUMS) keeps working. - `@pnpm/crypto.shasums-file`: new `fetchVerifiedNodeShasums` / `fetchVerifiedNodeShasumsFile` verify the signature via `openpgp` against the embedded keys. - The keys live in a generated file (`src/nodeReleaseKeys.ts`, 28 keys) mirrored from the canonical `nodejs/release-keys` list. `crypto/shasums-file/scripts/update-node-release-keys.mjs` keeps them current (`pnpm check:node-release-keys` / `--update`), and the **create-release-pr** workflow runs the check as a gate so a new release signer can't silently break verification. - `@pnpm/engine.runtime.node-resolver` verifies the **configurable-mirror** SHASUMS. The hardcoded `unofficial-builds.nodejs.org` musl mirror is **not** repo-configurable and is signed by a different key, so it stays trusted over TLS. ## Scope - **Pre-release channels (rc, nightly, …) are not verified** — Node only signs the `release` channel (no `SHASUMS256.txt.sig` exists for them, even on nodejs.org), so they remain unverifiable. Verification is gated on the `release` channel. - **Bun / Deno are unaffected** — their download/SHASUMS URLs are hardcoded to canonical GitHub (`github.com/oven-sh/bun`, `api.github.com/repos/denoland/deno`), not mirror-configurable, so a repo can't redirect them. - **Pacquet parity:** `pacquet/crates/engine-runtime-node-resolver` has the same mirror-configurable SHASUMS logic and needs the equivalent Rust port — tracked as a follow-up (per the repo's parity rule, opening the TS side first). |
||
|
Zoltan Kochan
|
ea204b07b7 |
feat(pacquet): configurational dependencies support (#12285)
Ports pnpm's `configDependencies` feature to pacquet end-to-end: config dependencies are resolved and installed ahead of regular deps into `node_modules/.pnpm-config/<name>` via the global virtual store, recorded in the **env lockfile** (the first YAML document of `pnpm-lock.yaml`), their `updateConfig` plugin hooks run before the main install (including `patchedDependencies`/`catalogs` injection), and `pnpm add --config` manages them.
The whole path runs via `pacquet install` / `pacquet add --config` and is covered by tests against the mocked registry.
## What's implemented
**Resolve + install** (`pacquet-env-installer`) — clean-specifier resolution + migration of the old inline (`version+integrity`) / object (`{ tarball?, integrity }`) formats; one level of optional subdeps with `os`/`cpu`/`libc` platform fields + host filtering (exact-version-only); env-lockfile pruning of stale entries; pnpm error codes (`ERR_PNPM_BAD_CONFIG_DEP`, `CONFIG_DEP_OPTIONAL_NOT_EXACT`, `ENV_LOCKFILE_CORRUPTED`, `FROZEN_LOCKFILE_WITH_OUTDATED_LOCKFILE`, …).
**`updateConfig` pnpmfile hook** (`pacquet-hooks` + `pacquet-config`) — new `updateConfig` trait method + Node-worker dispatch; `is_plugin_name` + plugin-pnpmfile resolution (`pnpm-plugin-*` / `@pnpm/plugin-*` / `@scope/pnpm-plugin-*`) from `.pnpm-config`; full `Config` round-trip via `WorkspaceSettings` (added `Serialize`), applying only hook-changed keys so `.npmrc`/CLI values aren't clobbered; `catalog`/`catalogs` are seeded into the hook input and captured back into `Config::catalogs` (the install prefers it over re-reading the manifest), so a hook can inject catalogs a `catalog:` dependency then resolves against.
**`pnpm add --config`** (`pacquet-cli` + `pacquet-workspace-manifest-writer`) — resolves + installs the dep and writes the clean specifier into `pnpm-workspace.yaml`'s `configDependencies` block with a format-preserving edit (extends the previously catalog-only writer).
**Wiring** (`pacquet-cli`) — config-dep install + `updateConfig` hooks run at config finalization, before the main install.
**Supporting** — `graph-hasher`: `calc_leaf_global_virtual_store_path` + `calc_global_virtual_store_path_with_subdeps`; `lockfile`: `EnvLockfile` multi-document read/write + env-doc preservation in the wanted-lockfile writer; `reporter`: `pnpm:installing-config-deps` event.
|
||
|
Zoltan Kochan
|
b73083537d |
feat(lockfile): emit pnpmfileChecksum in pacquet lockfiles (#12280)
* feat(lockfile): emit pnpmfileChecksum in pacquet lockfiles
Port pnpm's `calculatePnpmfileChecksum` so a pacquet install records the
`pnpmfileChecksum` field in `pnpm-lock.yaml` when the project's
`.pnpmfile.{cjs,mjs}` exports a `hooks` object — matching pnpm's value
byte-for-byte (sha256-base64 of the pnpmfile's CRLF-normalized contents)
and its position in the lockfile (right after `packageExtensionsChecksum`).
The checksum value is pure file hashing; only pnpm's
`entries.some(entry => entry.hooks != null)` gate consults the evaluated
module, answered here by a new `hasHooks` query on the existing
long-lived pnpmfile worker. A pnpmfile that exports no hooks records no
checksum, matching pnpm.
Addresses item 5 of pnpm/pnpm#12266.
* test(crypto-hash): isolate CRLF-normalization test with TempDir
`hash_from_file_normalizes_crlf` used a fixed directory under the OS temp
dir, so parallel test runs could contend on the same create/write/remove
path and flake. Use a per-test `tempfile::TempDir` instead and drop the
explicit cleanup.
|
||
|
Zoltan Kochan
|
0dc770aa7b |
perf(pacquet): share virtual-store slot linking pass (#12251)
Refs: pnpm/pnpm#12250 - share warm/cold virtual-store slot linking through one parallel helper - emit structured pacquet install phase metrics for virtual-store partition sizes and link-slot elapsed time - generate integrated-benchmark diagnostics artifacts and use them for the fresh pnpr cold-batch and pnpr-vs-direct guardrails - split client registry latency/bandwidth from pnpr server registry latency, while rewriting server-origin tarball URLs back to the measured client registry path - keep the benchmark PR comment focused on scenario tables and collapsed raw JSON; diagnostics stay available as artifacts instead of inline report noise |
||
|
Zoltan Kochan
|
c199198e94 |
perf(pnpr): stream /v1/resolve, and fix the integrated benchmark to actually exercise pnpr (#12237)
Closes #12234. This PR has two parts. The headline turned out to be the **benchmark**, not the feature. ## Part 1 — `/v1/resolve` streaming (the #12234 feature) `POST /v1/resolve` now streams **NDJSON** instead of buffering the whole lockfile: one `package` frame per resolved tarball as the server's tree walk yields it, then a terminal `done` (full lockfile + stats), `error`, or `violations` frame. The client fetches each tarball as its frame arrives, overlapping the server's resolution — matching the native `PrefetchingResolver` shape. - Server: new `ResolutionObserver`/`ObservingResolver` in `package-manager`, threaded through `Install`; `handle_resolve` runs the resolve in a detached task whose observer pushes frames into the response channel; `application/x-ndjson` is excluded from the gzip layer so frames flush incrementally. - Client: `resolve_streaming(opts, on_package)`; `install_via_pnpr` drives a new `TarballPrefetcher` that warms the shared mem cache as frames arrive. - Breaking change within protocol v1 (no version bump — experimental pnpr allows it). **Honest caveat:** streaming only helps when the *server's* resolve is slow (cold/distant server). Against a warm server it's inert — see the results below (`pnpr@HEAD` ≈ `pnpr@main` everywhere). Whether to keep this commit or defer it is an open question; the benchmark fixes below stand on their own. ## Part 2 — make the integrated benchmark actually measure pnpr While validating Part 1, the benchmark turned out not to be exercising pnpr **at all**, plus it was serving the registry far faster than reality. Fixes: - **`pnpr@<rev>` targets never routed through pnpr.** `.pnpr-env` exported a bare `PNPR_SERVER`, but pacquet reads config env vars only under the `PNPM_CONFIG_*` prefix, so `config.pnpr_server` was always `None` and every pnpr row was a silent duplicate of its direct row. Fixed the env var name; added a post-run guard that fails the benchmark if a pnpr target's `pnpr-storage` is empty (proof it never served a resolve). - **Emulate a real registry link for every client.** The latency proxy modeled RTT but not throughput, and fronted only direct targets. Generalized it to a `LinkProfile` (one-way delay + per-direction bandwidth cap), added `--registry-bandwidth-mbps`, and routed *all* registry traffic through it (direct installs, the pnpr server's resolve, the pnpr client's fetches) so the registry-mock is uniformly as remote as real npm. CI runs it at 50 ms + 200 Mbit/s (≈ the measured public-npm peak). - **Make "cold cache" cold for resolution.** Forced `cacheDir` bench-local and wipe it in cold-cache scenarios, so a direct install actually pays the packument-fetch waterfall (previously the global metadata mirror survived every wipe). - **New scenario** `fresh-install.cold-cache.hot-store` that isolates resolution (cold metadata, hot store → no download to mask it). ## Results (Linux CI, after the fixes) | Scenario | pacquet@HEAD | pnpr@HEAD | pnpr speedup | |---|---:|---:|---:| | fresh-install · cold cache · **hot store** | 5.06 s | 0.69 s | **7.4×** | | fresh-install · cold cache · cold store | 5.36 s | 2.03 s | **2.65×** | | fresh-install · hot cache · hot store | 1.46 s | 0.69 s | **2.1×** | | fresh-restore (frozen) · cold cache · cold store | 10.08 s | 5.13 s | **2.0×** | | fresh-restore (frozen) · hot cache · hot store | 0.71 s | 0.80 s | 0.89× (slower) | pnpr offloads the client's resolution (and, on the frozen path, lockfile verification) to its warm server: 2–7× faster wherever the client would otherwise pay that cost. The lone regression is the fully-warm frozen install, where there's nothing to offload and pnpr's one round trip is pure overhead. `pnpr@HEAD` vs `pnpr@main` is flat throughout — i.e. the streaming commit (Part 1) adds ~nothing against a warm server, while the base pnpr win (offload to a warm server) is large. |
||
|
dependabot[bot]
|
1f8b4a1ba4 |
chore(cargo): bump tabled from 0.17.0 to 0.20.0 (#12216)
Bumps [tabled](https://github.com/zhiburt/tabled) from 0.17.0 to 0.20.0. - [Changelog](https://github.com/zhiburt/tabled/blob/master/CHANGELOG.md) - [Commits](https://github.com/zhiburt/tabled/commits) --- updated-dependencies: - dependency-name: tabled dependency-version: 0.20.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
1f549ebbfa |
chore(cargo): bump serde-saphyr from 0.0.26 to 0.0.27 (#12219)
Bumps [serde-saphyr](https://github.com/bourumir-wyngs/serde-saphyr) from 0.0.26 to 0.0.27. - [Release notes](https://github.com/bourumir-wyngs/serde-saphyr/releases) - [Commits](https://github.com/bourumir-wyngs/serde-saphyr/compare/0.0.26...0.0.27) --- updated-dependencies: - dependency-name: serde-saphyr dependency-version: 0.0.27 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
66efa1d2d6 |
chore(cargo): bump dashmap from 6.1.0 to 6.2.1 (#12217)
Bumps [dashmap](https://github.com/xacrimon/dashmap) from 6.1.0 to 6.2.1. - [Release notes](https://github.com/xacrimon/dashmap/releases) - [Commits](https://github.com/xacrimon/dashmap/compare/v6.1.0...v6.2.1) --- updated-dependencies: - dependency-name: dashmap dependency-version: 6.2.1 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
dependabot[bot]
|
76587d3def |
chore(cargo): bump reqwest from 0.13.3 to 0.13.4 (#12215)
Bumps [reqwest](https://github.com/seanmonstar/reqwest) from 0.13.3 to 0.13.4. - [Release notes](https://github.com/seanmonstar/reqwest/releases) - [Changelog](https://github.com/seanmonstar/reqwest/blob/master/CHANGELOG.md) - [Commits](https://github.com/seanmonstar/reqwest/compare/v0.13.3...v0.13.4) --- updated-dependencies: - dependency-name: reqwest dependency-version: 0.13.4 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> |
||
|
Zoltan Kochan
|
089484aca8 |
perf(pnpr): resolve server-side and fetch tarballs directly (#12232)
## Summary Reworks pnpr from an install/file accelerator into a resolve-only accelerator: - `POST /v1/resolve` resolves against the client-supplied registries and returns a gzipped JSON lockfile response - pacquet/pnpm clients then fetch tarballs normally from registries with their own credentials and existing parallel fetch/integrity paths - pnpr no longer serves package file bytes or store-index rows, so the server-side file diff, file-frame response, grant table, and public-package byte-gating code are removed The follow-up resolution fast paths are included on the new measured path: - repeated public no-lockfile resolves use a bounded in-memory TTL cache - fresh frozen input lockfiles skip the server-side lockfile-only pacquet resolve after verification proves the lockfile is usable - input lockfile verification and the verdict cache are preserved ## Benchmark Integrated benchmark on Linux shows small improvements in all pnpr rows, with the clearest movement in hot restore. This should be treated as an incremental win rather than a large install-speed change. | Scenario | `pnpr@HEAD` | `pnpr@main` | Change | | --- | ---: | ---: | ---: | | fresh restore, cold cache + cold store | `1.677 s ± 0.090` | `1.686 s ± 0.070` | ~0.6% faster | | fresh restore, hot cache + hot store | `492.5 ms ± 18.1` | `521.9 ms ± 33.4` | ~5.6% faster | | fresh install, cold cache + cold store | `1.997 s ± 0.025` | `2.003 s ± 0.038` | ~0.3% faster | | fresh install, hot cache + hot store | `1.211 s ± 0.024` | `1.236 s ± 0.038` | ~2.0% faster | ## Trade-off Going registry-direct means pnpr no longer gates tarball bytes itself. Private package access is enforced by the upstream registry when the client fetches tarballs. Resolution policy still runs server-side: lockfile verification, release-age policy, trust policy, and resolved package selection continue to happen before the client fetches bytes. |
||
|
Zoltan Kochan
|
70554b8677 |
feat(pnpr): config-selectable networked-SQLite auth backend (#12199 phase 3) (#12206)
## What Implements the **auth half** of [#12199](https://github.com/pnpm/pnpm/issues/12199) (phase 3) — making pnpr's remaining per-instance state pluggable so the registry can run as stateless, horizontally-scaled replicas. ### Auth records behind config-selected backends Users + tokens now sit behind narrow async `UserBackend` / `TokenBackend` traits, built once at startup into `Arc<dyn …>` handles (the same build-once pattern #12198 used for the hosted store). Three implementations: - **Local** (default) — today's htpasswd file + SQLite token DB, or in-memory when no file is configured. Unchanged behavior. - **Networked SQLite (libsql / Turso)** — `LibsqlAuth` stores **both** records in one shared database, so several stateless replicas observe a consistent set of users and tokens. The `tokens` table DDL is shared verbatim with the local backend (a DB can migrate between them); users — which the local backend keeps in htpasswd — move into a `users` table. Selected via a new top-level YAML block: ```yaml backend: libsql: url: ${PNPR_LIBSQL_URL} authToken: ${PNPR_LIBSQL_TOKEN} # optional embedded replica for local-fast hot-path reads: replicaPath: ./auth-replica.db syncIntervalSecs: 60 ``` When the block is absent, auth stays on local disk exactly as before. ### Embedded-replica read acceleration Token lookups are on the request hot path, so against a remote primary every read would be a network round-trip. With `replicaPath` set, `LibsqlAuth` builds a libsql **embedded replica**: reads hit a local file that libsql keeps current in the background; writes go to the primary. `syncIntervalSecs` is the freshness knob that bounds token-revocation lag. ### Async access path `identify` / `enforce_access` are now async (a networked lookup is async). `enforce_access` is split into an async `resolve_identity` + a sync `authorize`, so the search endpoint resolves the caller once and authorizes each candidate synchronously (no async-in-`retain`). ### Concurrent-publish guard (cross-cutting follow-up from the issue) Closes the same-instance lost-update window in the three read-modify-write packument flows (publish, dist-tag change, partial-unpublish): a striped per-package lock serializes same-package writers on one instance while letting different packages proceed in parallel. The **cross-replica** half (S3 `If-Match` / ETag CAS) is documented in-code as the remaining piece — the issue files it under "fix when we get there," and it belongs with the multi-writer S3 publish work, not this auth branch. ## Tests All green — `cargo test -p pnpr`: - **176 lib unit tests** incl. new `LibsqlAuth` tests (run against an in-memory libsql DB — same driver + SQL, no server) and `backend.libsql` config-parsing tests (incl. the replica options). - New `concurrent_publishes_of_distinct_versions_all_survive` integration test for the publish guard. - Existing auth_persistence / auth_user_endpoints / auth_publish / server / s3_backend suites pass. - Clean under `cargo fmt`, `clippy`, `RUSTDOCFLAGS=-D warnings cargo doc`, **Dylint perfectionist**, and `taplo`. ## Docs `backend.libsql` (incl. embedded replica) documented in the bundled `config.yaml` and the `pnpr` npm README, mirroring how the S3 backend was documented in #12198. |
||
|
Zoltan Kochan
|
a6682244cd |
feat(pacquet): port the catalogMode auto-cataloging half (saveCatalogName / catalog writes) (#12202)
* feat(package-manager): port catalogMode auto-cataloging to pacquet Implements the auto-cataloging half of pnpm's `catalogMode` (the gate was ported in #11706): under `catalogMode: strict`/`prefer` (or `--save-catalog[-name]`), `add`/`update` write `catalog:`/`catalog:<name>` to `package.json`, insert the entry into `pnpm-workspace.yaml`, and record the resolved snapshot in `pnpm-lock.yaml`'s `catalogs:`. - config: add `saveCatalogName`; move `save-catalog-name` out of the parity NOT_PORTED list - new crate `pacquet-workspace-manifest-writer`: format-preserving (comment/blank-line/key-order/quote-style) catalog write-back, byte-for-byte with pnpm, on top of yamlpatch/yamlpath/yaml_serde - lockfile: add `catalogs:` snapshot (CatalogSnapshots/ResolvedCatalogEntry) - catalog_mode: `decide_catalog` per-dep decision (gate + auto-catalog) - add: parse `pkg@version`; `--save-catalog`/`--save-catalog-name` flags - update: route the catalog gate through the decision core, handling `--latest` on catalog deps Closes #12196 * fix(package-manager): address review feedback on catalog auto-cataloging - update: gate the `pnpm-workspace.yaml` write on `save` so `--no-save` persists nothing to disk, matching pnpm's `if (opts.save !== false)` - workspace-manifest-writer: detect each block's child indent dynamically instead of assuming two spaces - tests: assert the lockfile `catalogs:` snapshot in the named-catalog `update --latest` e2e test; add `--no-save` and four-space-indent cases |
||
|
Zoltan Kochan
|
8e5e764037 |
feat(pnpr): store hosted packages in an S3-compatible object store (#12198)
## What
Lets pnpr store its **hosted** packages (the ones published to it, plus static-served content) in an **S3-compatible object store** instead of a local directory. Because the same code targets any S3-compatible endpoint, this also covers **Cloudflare R2**, MinIO, Backblaze B2, Wasabi, etc.
The local `tokio::fs` path remains the default — nothing changes unless you add the new `s3:` config block.
## Why
The hosted store is pnpr's source of truth: durable, must be backed up, and can't be regenerated. That's exactly what belongs in object storage:
- The provider handles durability/replication, so there's no single-node volume to back up.
- Multiple **stateless pnpr replicas** can share one hosted store.
- R2 is the S3 API, so a configurable `endpoint` gets it (and the other S3-compatibles) for free.
The disposable proxy cache and the install-accelerator SQLite stores deliberately **stay on local disk** — they're ephemeral, latency-sensitive, and streamed/locked in filesystem-shaped ways.
## How
- New `s3.rs` module: `S3Settings` (the YAML `s3:` block), a client builder (`object_store` crate; AWS-env credentials with explicit override, plus R2/MinIO/path-style/HTTP knobs), and an `S3Store` adapter (packument get/put, streaming tarball get, staged upload, delete, prefix-scoped list).
- `storage.rs`: a `HostedStore { Fs | S3 }` backend enum routes the hosted ops; the `cached` store stays fs-only. Publish stages the decoded+verified tarball to local scratch, then finalize either renames (fs) or uploads (S3). `open_tarball` now returns a streaming response body so S3 reads stream straight through.
- `config.rs`: parses `s3:` and builds the client once at config-load time (the only fallible step), so `Storage` construction stays infallible.
- `search.rs`: local search now lists package names through the storage abstraction, so it works against a bucket too.
- Documented (commented) in the bundled `config.yaml`.
### Example: Cloudflare R2
```yaml
storage: ./storage # still backs the local proxy cache + upload staging
s3:
bucket: my-pnpr-packages
region: auto
endpoint: https://<account-id>.r2.cloudflarestorage.com
accessKeyId: ${PNPR_S3_ACCESS_KEY_ID}
secretAccessKey: ${PNPR_S3_SECRET_ACCESS_KEY}
```
|
||
|
shiminshen
|
e7e99f04e4 |
fix: do not crash when a catalog specifier is a range (#11706)
## Summary `pnpm update --recursive --lockfile-only <pkg>@<version>` crashed with `Invalid Version: <range>` when the catalog entry for `<pkg>` was a range (e.g. `^21.2.10`) and `catalogMode` was `strict` or `prefer`. This is the exact command Renovate's pnpm artifact updater runs; monorepos using `catalog:` with any range specifier were blocked from Renovate-driven lockfile updates. **Root cause:** in `installSome`, the catalog-match short-circuit guards `semver.eq(wantedDep.bareSpecifier, catalogDepSpecifier)` with `semver.validRange` on both sides. `validRange` returns truthy for ranges, but `semver.eq` constructs `new SemVer(...)` internally and throws on a range. **Fix:** use `semver.valid` instead of `semver.validRange` on both sides of the equality guard. Range specifiers now fall through to the existing mismatch handling (`CatalogVersionMismatchError` in `strict` mode, warn-and-use-direct in `prefer` mode) instead of crashing. Behavior for concrete-on-both-sides is unchanged. Closes #11570 ## Behavior after the fix This turns a crash into pnpm's normal catalog-mismatch handling; it does **not** make a strict-mode update succeed when the catalog is a range: - **`catalogMode: strict`** — rejects with `ERR_PNPM_CATALOG_VERSION_MISMATCH` (clean, actionable error instead of a stack trace). - **`catalogMode: prefer`** — warns and uses the direct version. - **concrete-vs-concrete** — unchanged (`semver.eq` still runs). ## pacquet parity The TypeScript fix patches a crash inside pnpm's `catalogMode` mismatch gate — a feature pacquet had not ported at all (`catalog-mode` was in the config parity test's `NOT_PORTED` list). Rather than just the one-liner, this PR ports that gate to pacquet so the two stacks match: - **config:** new `CatalogMode { Manual, Strict, Prefer }` enum (default `manual`), `Config.catalog_mode`, wired through `pnpm-workspace.yaml` (`catalogMode:`) and the env overlay; `catalog-mode` moved from `NOT_PORTED` to a mapped row in the `pnpm_default_parity` contract test. - **package-manager:** `check_catalog_mode` + `CatalogVersionMismatchError` (`ERR_PNPM_CATALOG_VERSION_MISMATCH`), invoked from `update` before the manifest is mutated. The comparison only treats both sides as equal when each parses as a concrete semver version, so a ranged catalog entry falls through to the mismatch path instead of reaching an exact-version comparison — the Rust analogue of the `semver.valid` guard above. The crash itself can't occur in pacquet (Rust's `node-semver` returns a `Result` rather than throwing); the port is the *feature* with the range-correct comparison built in, so pacquet behaves like fixed pnpm. **Not ported** (the surrounding pieces pacquet still lacks, so wiring them would diverge from pnpm rather than match it): the `add`-path cataloging that relies on `defaultCatalog` rewriting, and the `saveCatalogName` → `pnpm-workspace.yaml` auto-cataloging half. The gate is therefore wired into `update <pkg>@<version>` / `--latest` (the Renovate scenario), not `add`. |
||
|
Zoltan Kochan
|
3492bb8f4a |
feat(pnpr): gzip-compress package metadata over the wire (#12170)
Neither side used gzip for package metadata: pacquet fetched packuments uncompressed from every registry (unlike pnpm-TS, which gets gzip via undici), and pnpr served them uncompressed. Packuments are the largest payloads pulled during resolution and gzip ~5-10x, so this was a real resolution-time cost and a divergence from how a CDN-fronted registry behaves. Closes #12169. Both halves are needed and land together: - Client (pacquet): enable reqwest`s `gzip` feature and set `.gzip(true)` explicitly on the network client builder, so it sends `Accept-Encoding: gzip` and transparently decompresses. Tarballs are unaffected (served as `application/octet-stream` with no `Content-Encoding`, so reqwest leaves them alone and store-integrity verification is unchanged). It also transparently handles the install accelerator's already-gzipped `/v1/files` stream — the client's existing magic-byte check covers the now-auto-decompressed case. - Server (pnpr): add a `tower-http` `CompressionLayer`, scoped to JSON via `NotForContentType` so it compresses packuments / version manifests / dist-tags / search but never re-gzips an already-compressed payload: tarballs (`application/octet-stream`), the file stream (`application/x-pnpm-install`), or the resolve NDJSON (`application/x-ndjson`). pnpr is commonly hit directly with no CDN or nginx in front, so the application is the only layer that can compress; where a proxy/CDN is present, the `Content-Encoding: gzip` is passed through (no double compression). Tests assert a packument is gzipped when `Accept-Encoding: gzip` is sent, served plain otherwise, and a tarball is never re-gzipped. |
||
|
Zoltan Kochan
|
e1648a6ca0 |
perf(pnpr): coarsen packument time precision to shrink abbreviated responses (#12168)
pnpr serves abbreviated packuments uncompressed, so the verbatim `time` map is pure wire cost. Drop precision the resolvers don't need: seconds come off every entry, and entries older than a week lose the time-of-day entirely (down to a bare date). The reserved `unpublished` object and any non-RFC-3339 value pass through untouched. Timestamps are rounded *up* (next minute / next day, leaving values already on the boundary untouched), so a coarsened value is never earlier than the real publish time. `minimumReleaseAge`, the abbreviated-modified shortcut, and the trust checks can therefore only ever read a version as newer than it is — the fail-safe direction; a too-new version is never coarsened into looking mature. Both reduced forms are accepted by pnpm's lenient `new Date(...)`. pacquet parsed strict RFC 3339 only, so add a shared `parse_packument_timestamp` in resolving-resolver-base that also accepts minute precision (`2024-03-15T09:42Z`) and bare dates (`2024-03-15`, read as midnight UTC), and route every existing publish-timestamp parse site through it. |
||
|
suzunn
|
8bc25f3c26 |
fix: retry pacquet metadata fetches with a shared, config-driven retry policy (#12029)
## Problem `pacquet` metadata fetches made a single registry request, so a transient network failure or a retryable HTTP response (`408`/`429`/`5xx`) aborted resolution — even though the tarball path already followed pnpm's retry policy. pnpm wraps **every** registry request, metadata and tarball, in `@zkochan/retry` under one `fetch-retries` budget; pacquet only retried tarballs, and the metadata retry that was added in the first cut duplicated the tarball budget and ignored the user's config. Fixes #11841. ## Solution A single retry primitive now lives in `pacquet-network` and is driven by config on the install paths, matching pnpm exactly. - **One shared `RetryOpts`.** The retry budget (`fetch-retries` / `-factor` / `-mintimeout` / `-maxtimeout`) plus its exponential-backoff math moves into `pacquet-network`, next to `should_retry_status` and a generic `send_with_retry(client, url, opts, build_request) -> (guard, Response)` helper. `pacquet-tarball` re-exports `RetryOpts`; the metadata fetchers use `send_with_retry`. This removes the duplicate `MetadataRetryOpts`. - **No parked sockets/permits during backoff.** `send_with_retry` acquires the network permit *per attempt* and drops the response and its guard before each backoff sleep, so a flapping registry can't pin sockets or hold a concurrency permit while it waits. On the winning attempt the guard rides out with the `Response` so the caller keeps the permit through body streaming. - **Config-driven retries on every metadata path.** A config-sourced `RetryOpts` (via the existing `retry_opts_from_config`) is threaded through the resolution verifier, `NpmResolver`, `NamedRegistryResolver`, and `PickPackageContext`, so the metadata and verify paths honor the user's `fetch-retries*` settings instead of a hardcoded default — the same way the tarball path already did. ### Parity bug fixed Because the verifier and resolver hardcoded the default retry budget, a `5xx` flap on the main resolve/verify paths hung for **~70 s** (`10 s + 60 s` default backoff) regardless of the user's `fetch-retries` setting — a user who set `fetch-retries=0` to fail fast would still wait. Driving the budget from config fixes this; test harnesses use a zero-retry budget so failure-path cases don't wait out the backoff. ### Not pnpr `pnpr` is intentionally untouched. Retry is an install-time concern: pnpr's `/v1/install` accelerator already retries through pacquet's resolver and tarball fetcher, while the verdaccio-style registry-**proxy** path must fail fast (and serve stale on failure) rather than block a client request behind a 70 s backoff. The shared `RetryOpts` re-export keeps the accelerator's existing tarball-download retry compiling unchanged. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Khải
|
622c056bbc |
feat(pacquet/cli): initial implementations of run, exec, dlx (#11938)
> [!WARNING] > **Scope note.** Per [`pacquet/CONTRIBUTING.md`](https://github.com/pnpm/pnpm/blob/d4a2b0364c/pacquet/CONTRIBUTING.md), pacquet's current focus is Stage 1 (the headless installer); `exec` and `dlx` are new top-level commands, so this PR sits outside Stage 1 and is opened for review/discussion ([roadmap pnpm/pacquet#299](https://github.com/pnpm/pacquet/issues/299)). ## Summary Ports of `run`, `exec`, and `dlx` from the TypeScript pnpm CLI. - **`run`**: runs scripts through a new foreground `run_script` in `pacquet-executor` (sets up `node_modules/.bin` on `PATH` + the `npm_*` env). Handles `pre`/`post` scripts under `enablePrePostScripts`, arg shell-quoting (with the Windows `cmd /d /s /c` verbatim `raw_arg` path), script listing, hidden (`.`-prefixed) scripts, `--if-present`, the `start`→`server.js` fallback (with the NO_SCRIPT_OR_SERVER guard, including empty-string `start`), the `[ELIFECYCLE]` failure line (with the `test`-stage and signal-killed variants), and exit-code propagation. The recursive runner's scaffolding (`--resume-from` / `--report-summary`) landed separately on `main` via [#12093](https://github.com/pnpm/pnpm/pull/12093); this PR dispatches to it when `-r` is set and hardens it to match pnpm — per-project `pre`/`post`, the `PNPM_SCRIPT_SRC_DIR` recursion guard, pnpm's per-stage no-op guards, hidden-script handling, and `--no-bail`. - **`exec`**: runs a command with `node_modules/.bin` + `extraBinPaths` on `PATH` (resolved via `which`), stamps `npm_config_user_agent` / `PNPM_PACKAGE_NAME` / `NODE_OPTIONS`, supports `--shell-mode` (the joined command goes through the shared `push_script_arg` helper, so the Windows `cmd /d /s /c` verbatim path uses `raw_arg` and embedded quoting survives), rejects delimiter-containing dirs (`ERR_PNPM_BAD_PATH_DIR`). The **recursive variant** (`-r`) runs the command in every workspace project, topologically sorted and sequential, with `--resume-from` / `--report-summary` / `--no-bail` and pnpm's error codes (`ERR_PNPM_RECURSIVE_EXEC_NO_PACKAGE` / `ERR_PNPM_RECURSIVE_EXEC_FIRST_FAIL` / `ERR_PNPM_RECURSIVE_FAIL`). The workspace-graph / summary machinery is shared with recursive `run` through a new `cli_args::recursive` module. - **`dlx`**: installs the package(s) into a TTL cache dir (reusing the install pipeline, anchored at the cache dir, with a *fresh* per-install build-script allow-list — caller's `allow_builds`/`dangerouslyAllowAllBuilds` don't leak in), then runs the resolved bin in the process cwd. Supports `--package`, `--allow-build`, `--shell-mode` (same `push_script_arg` verbatim path as exec), `--cpu`/`--os`/`--libc` architecture overrides (folded into the per-axis `supportedArchitectures` of the dlx install **and** into the cache key, so different overrides don't share a cached install), `dlxCacheMaxAge`; same PATH guard as exec. New `Config` settings: `enablePrePostScripts`, `scriptShell`, `nodeOptions`, `dlxCacheMaxAge` (wired into `pnpm-workspace.yaml` + the `PNPM_CONFIG_*` overlay). Their defaults match pnpm and are asserted by the `pnpm_default_parity` contract test — this PR moves `enablePrePostScripts` (which pnpm defaults to `true`, a breaking change in [#7634](https://github.com/pnpm/pnpm/pull/7634) shipped in v9) and `dlxCacheMaxAge` into its mapped rows. `extraBinPaths` is kept as a computed field (empty until workspace support lands), matching pnpm — it is not a user-settable key. ## Deferred (documented in code) - **`--filter` and `--workspace-concurrency`.** Recursive `run` and `exec` run every workspace project sequentially; the `--filter` package-selector subsystem and `--workspace-concurrency` parallelism are not ported yet (the global `--filter` / `--recursive` flags are accepted via clap but `--filter` is not consumed). `dlx` stays single-package by design (matches pnpm). - `run`: the `/regexp/` script selector and the fuzzy "did you mean" hint are not ported (no regex/levenshtein dep); `scriptsPrependNodePath: always` can't prepend the node dir (pacquet resolves no node execpath anywhere yet). - `dlx`: the cache key uses raw specs (not resolved ids); no `approve-builds` prompt. |
||
|
Zoltan Kochan
|
32c07bfee0 |
feat(pnpr): offload lockfile verification to the server (#12139) (#12144)
Closes #12139. ## What When a `pnpr` server is configured, the client no longer runs `verifyLockfileResolutions` locally. It sends its on-disk lockfile plus its **full verification policy** to `/v1/install`; pnpr verifies that *input* lockfile under the **client's** policy *before* resolving, and streams back any violations so the client aborts with the identical `ERR_PNPM_*` diagnostic the local gate would have produced. This is faster (pnpr's packument cache is warm + shared) and removes the client's own registry-reachability requirement — it adds no new trust (the client already trusts pnpr to resolve and serve bytes). All three phases from the issue, delivered together. **Rust-only**: `pacquet` client + `pnpr` server. The TS agent server is deprecated and the TS client already skips local verification, so no TS changes were needed. ## How **Phase 1 — send lockfile + policy; pnpr verifies; client skips local verify** - Protocol (`install_accelerator/protocol.rs`, mirrored in `pnpr-client`): `/v1/install` now carries `lockfile`, `frozenLockfile`, and the full policy (`minimumReleaseAge[Exclude|IgnoreMissingTime]`, `trustPolicy[Exclude|IgnoreAfter]`). - `handle_install` verifies the input lockfile via `build_resolution_verifiers` + `collect_resolution_policy_violations` (under the client policy threaded into the server `config_for`) **before** resolving. On a violation it streams a `200` NDJSON `E` line of rendered violations; the client rebuilds the identical `VerifyError` (`PnprClientError::Verification`). - The pacquet CLI sends `state.lockfile` + policy, drops the `trustPolicy: no-downgrade` guard (pnpr enforces it now — input-lockfile verifier for reused entries + resolver pick-time check for new ones), and sets `trust_lockfile: true` on the local materialization so it never re-verifies or touches the local `lockfile-verified.jsonl`. **Phase 2 — `frozenLockfile` governs resolution reuse** - `resolve.rs` seeds resolution from the input lockfile (frozen → as-is; non-frozen → reuse pins + resolve new). **Phase 3 — SQLite whole-lockfile verdict cache on pnpr** - New `install_accelerator/verdict_cache.rs`: SQLite-backed (reuses the existing `rusqlite` dep), keyed by `(lockfile hash, merged policy snapshot)`, hit = all verifiers `can_trust_past_check`. Only *passes* are cached (monotonic age + hash pins versions → time-correct without a cutoff, same property as the local cache); LRU cap, no TTL. |
||
|
Zoltan Kochan
|
eec417b74f |
fix(lockfile): emit lockfile maps in canonical key order (#12120)
pacquet serialized the `importers`, `packages`, `snapshots`, and per-importer/per-snapshot dependency maps in `std::HashMap` iteration order — a per-instance random seed — so two installs of the same resolution could emit byte-different `pnpm-lock.yaml` files. This diverges from pnpm, whose lockfile is canonically ordered, and produces spurious git diffs on no-op re-installs (#12117). Sort every lockfile map by its rendered key string at emit time via two `serialize_with` helpers in `serialize_yaml`, matching pnpm's `sortLockfileKeys`/`lexCompare`. Sorting by the rendered key (not a field-wise `Ord`) is load-bearing: the `@` separating `name@version` and the leading `@` of a scoped name both order differently as struct fields than as the concatenated string pnpm compares. `overrides` is the one map pnpm leaves unsorted (declaration order), so it moves from `HashMap` to `IndexMap` to preserve insertion order rather than being sorted — deterministic and faithful to pnpm. The reuse-vs-fresh equivalence test now asserts byte-for-byte parity (was a parsed-struct comparison working around the old non-determinism), and a new test guards that a no-op re-install leaves the lockfile bytes unchanged. |
||
|
Alessio Attilio
|
5c669d7387 |
feat(pacquet): add pnpmfile hooks support (#12044)
Implements Tier 4 pnpmfile hooks for pacquet (#11633, point 4.1): pacquet now discovers and runs a project `.pnpmfile` during dependency management, matching pnpm. ## What it does - **Discovery** — looks for `.pnpmfile.mjs` then `.pnpmfile.cjs` (dotted names only, `.mjs` preferred), matching pnpm's `requireHooks`. Only actual files are accepted (`is_file()`). - **`readPackage`** — wired into resolution. Mirrors pnpm's `requirePnpmfile` contract: the four dependency fields are defaulted to `{}` before the hook runs, and the returned manifest is validated (must be a non-null object whose dependency fields, when present, are objects rather than arrays). A throwing/syntactically-invalid pnpmfile, a missing `require`, or a hook that returns nothing aborts the install (`PNPMFILE_FAIL`) instead of being silently ignored. - **`afterAllResolved`** — wired into the lockfile write. The resolved lockfile is passed to the hook and its return value is what gets written to `pnpm-lock.yaml`. The round-trip goes through `serde_json::Value` (the workspace already enables `preserve_order`) so hook-added keys the typed `Lockfile` cannot represent survive to disk; the round-trip only runs when a hook is present, so unmodified installs write byte-identical lockfiles. A throwing hook aborts the install. - **`preResolution`** — wired. Receives the resolution context (wanted/current lockfile, `existsCurrentLockfile`, `existsNonEmptyWantedLockfile`, lockfile dir, store dir, registries) over stdin. - **`filterLog`** — implemented in the bridge but not yet routed through the reporter (pacquet's reporter is a stateless synchronous emitter); deferred, see follow-ups. ## How hooks run Hooks are served by a long-lived Node.js worker, spawned lazily once per pnpmfile. Requests and responses are newline-delimited JSON over the worker's stdin/stdout, multiplexed by a monotonic request id so the concurrent `readPackage` calls the resolver makes (it resolves dependencies in parallel) share one process. This removes the per-package `node` startup cost on the resolution hot path and avoids interpolating payloads into a `node -e` argument (no `E2BIG` risk for large lockfiles). Each `context.log(...)` a hook emits is forwarded back to the call's log callback. `preResolution` keeps a one-shot `node` invocation since it runs once per install and needs an `info`/`warn` logger. ## Tests - Unit (hooks crate): readPackage validation (returns nothing / non-object / array dependency fields), manifest-field normalization, syntax-error and missing-module failures, worker request-id multiplexing under concurrency, and `context.log` forwarding. - Integration (package-manager): a `readPackage` hook pins a transitive dependency version; a hook that returns nothing aborts the install; a pnpmfile syntax error aborts the install; an `afterAllResolved` hook's mutation is written to `pnpm-lock.yaml`; a throwing `afterAllResolved` aborts the install. ## Scope The remaining pnpmfile-hook surface pnpm has but pacquet does not yet implement — wiring `filterLog` and the `pnpm:hook` log channel into the reporter, the `--pnpmfile` / `--global-pnpmfile` / `--ignore-pnpmfile` flags, pnpmfile checksum invalidation, `updateConfig`, and finders/resolvers/fetchers — is tracked in #12118. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
ae6e07705d |
feat(pacquet): implement the outdated command (#12112)
## What
Ports pnpm's `outdated` command to pacquet. It reports the direct dependencies whose newest published version (or highest in-range version under `--compatible`) is newer than the lockfile-pinned version, and exits with code `1` when any outdated dependency is found.
This widens pacquet's command surface beyond `install`/`add`/`update`/`remove`.
## How
- **Detection core** — `collect_outdated` / `OutdatedQuery` / `TargetVersion` in `pacquet/crates/cli/src/cli_args/outdated.rs`. This is shared with `update --interactive`, which already computed the same "what has a newer version" list inline; that code now calls the shared collector. The two callers differ only in the `TargetVersion` they compare against (`outdated` → `latest` tag / highest in-range under `--compatible`; `update` → the version a bump moves to). Packument fetches fan out concurrently via `futures::join_all`, mirroring pnpm's `Promise.all` (bounded by the HTTP client's per-registry concurrency limit).
- **Flags** — `--compatible`, `--long`, `--format table|list|json` (plus `--no-table` / `--json` shorthands), `-P/--prod` (`--production`), `-D/--dev`, `--no-optional` (following pnpm's `only`-normalization include-set), `--sort-by name` (default sort is by semver-change size then name), and positional name patterns (`@pnpm/config.matcher`).
- **Exit code** — `OutdatedArgs::run` returns an `OutdatedOutcome`; the single `process::exit(1)` lives in the top-level CLI dispatch, keeping the command composable.
- **`--global` and `--recursive` are rejected** with a "not supported yet" message, matching `pacquet update` (single-project scope).
- **Empty-manifest short-circuit** — a dependency-free manifest reports empty (exit 0) *before* the no-lockfile check, matching pnpm's `packageHasNoDeps` behavior so an empty, never-installed project doesn't error.
- **Rendering** — adds `tabled` (`Style::modern()` reproduces pnpm's `@zkochan/table` full-grid borders) and `owo-colors` (auto-disables on non-TTY, like chalk, so piped/JSON output is escape-free). Both are MIT (allowed by `deny.toml`). JSON output is keyed by package name with `current` / `latest` / `wanted` / `isDeprecated` / `dependencyType` (+ `latestManifest` under `--long`), matching pnpm's `renderOutdatedJSON`.
- Adds an optional `homepage` field to the registry `Package` for the `--long` details column.
## Parity notes / scope
pacquet loads a single (wanted) lockfile, so `current == wanted` and pnpm's "missing (wanted X)" state does not arise. `OUTDATED_NO_LOCKFILE` is raised when a manifest declares dependencies but no lockfile exists. Deprecated-but-current packages are still reported.
Deferred with their surrounding features (not yet ported to pacquet), each tracked by an upstream test that does not yet translate:
- `--long` **homepage** needs full registry metadata; the abbreviated fetch omits it, so it appears only when the registry serves it (deprecation details work fully).
- `minimumReleaseAge` / `minimumReleaseAgeExclude`, `pnpm.updateConfig.ignoreDependencies`, `catalog:` protocol replacement, `-g`/global packages, recursive workspace listing, and `runtime:` (node/deno/bun) dependencies.
## Tests
Ported the translatable pnpm `outdated` tests (`deps/inspection/outdated/test/*` and `deps/inspection/commands/test/outdated/*`):
- **10 unit tests** — semver-change classification, include-set normalization (default / `--prod` / `--dev` / `--no-optional`), default sort order, `renderLatest` (deprecated / not), and JSON shape (with/without `--long`).
- **14 integration tests** against the mocked registry — newer-version report, `--compatible` discrimination, JSON, JSON-empty `{}`, list (`--no-table`) format, up-to-date → exit 0, name-pattern filter, prod/dev filtering, npm-alias real-name reporting, deprecated package, `--long` deprecation details, no-deps/no-lockfile → empty exit 0, no-lockfile-with-deps error, and `--recursive` rejection.
|
||
|
Adam Lyrén
|
6f382f42ee |
fix: preserve integrity of remote tarball dependencies on re-resolution (#12096)
* fix: preserve integrity of remote tarball dependencies on re-resolution Re-resolving a remote tarball dependency without re-fetching it (e.g. `pnpm update`) produced a resolution with no integrity, so the previously recorded integrity was dropped from the lockfile, breaking later installs with ERR_PNPM_MISSING_TARBALL_INTEGRITY. Carry the integrity over from the previous lockfile entry when the rebuilt tarball resolution lost it and the URL is unchanged. This complements #12040, which fixes the same class of bug in the package-requester layer but does not cover this re-resolution path. Closes #12067. * test: cover integrity carryover on tarball re-resolution * refactor: check integrity before type in the tarball carryover guard * perf(pacquet): reuse the warm-store tarball on re-resolution instead of re-downloading --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |