mirror of
https://github.com/pnpm/pnpm.git
synced 2026-07-20 20:52:33 -04:00
node-runtime-fix
534 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Zoltan Kochan
|
a4f3c6d3b7 |
ci: enable manual releases for pacquet and pnpm (#11652)
Two related workflow changes: ### `pacquet-release-to-npm.yml`: switch to `workflow_dispatch` The trigger was "push to main touching `pacquet/npm/pacquet/package.json`" — the version came from a committed bump and the workflow auto-fired on every such commit. Switch to `workflow_dispatch` only, with a `version` input (validated as semver). The workflow patches `pacquet/npm/pacquet/package.json` before `generate-packages.mjs` runs, so the version is single-sourced from the manual trigger rather than needing a separate commit to bump the manifest first. The committed manifest now omits the `version` field entirely — it only exists at release time inside the runner. Dropped along the way: - The `check` job (EndBug/version-check against unpkg) — no longer needed when the operator types the version. - The `Create GitHub Release` step — no draft release, no `v*.*.*` git tag. The pacquet `v0.x.x` tag scheme collided with pnpm's `v11.x.x`; npm is the authoritative artifact store and provenance attestations stay attached via `--provenance` on `pnpm publish`. - `contents: write` on the publish job (no longer needs to create a tag). ### `release.yml`: add `workflow_dispatch` as a lib-only republish path Add a `workflow_dispatch:` trigger alongside the existing tag-push trigger. Tag-push behaves exactly as before. Manual dispatch becomes a fast **lib-only republish** path — useful after a version bump to one or more lib packages that doesn't warrant a full CLI release. On `workflow_dispatch` from any ref, the following are skipped (guarded with `if: startsWith(github.ref, 'refs/tags/')`): - `Publish @pnpm/exe` step — also contains the multi-minute `build-artifacts` call. - `Publish pnpm CLI` step. - `Copy Artifacts`, `Attest build provenance` (the `dist/*` attestation), `Generate release description`, `Release` (`softprops/action-gh-release`) — these are the GitHub-Release-side ceremony. Without an explicit `tag_name`, `softprops/action-gh-release@v2.5.0` defaults to `github.ref_name`, which on a manual dispatch from main would create a junk release tagged literally `main`. What still runs on `workflow_dispatch`: - `actions/checkout`, garnet scan, `pnpm/setup` - `Publish internal workspace packages (static token)` — i.e. `pn publish --filter=!pnpm --filter=!@pnpm/exe --access=public --provenance` Compilation is handled by each lib package's own `prepublishOnly: tsgo --build` hook (which `pnpm publish` runs automatically), same as the existing tag-push flow. The npm registry rejects any version already on it, so re-running on an already-released tree is a no-op — that's the safety net for accidental clicks. ## How to use **pacquet release**: Actions → Release Pacquet → Run workflow → fill in `version` (e.g. `0.2.3` or `0.2.3-rc.1`) → Run. No tag, no GitHub release. **pnpm full release**: still triggered by a `v*.*.*` tag push. Publishes @pnpm/exe + libs + CLI, attests, copies artifacts, creates a draft GitHub release. **pnpm lib-only republish**: Actions → Release → Run workflow → choose `main` → Run. Publishes just the internal workspace packages from whatever versions are currently in each `package.json`. Skips CLI, @pnpm/exe, build-artifacts, GitHub release. |
||
|
Zoltan Kochan
|
1575076d08 |
chore(pacquet): fold registry-mock into root workspace, fix npm metadata (#11643)
* chore(pacquet): fold registry-mock into root workspace, fix npm metadata
Two unrelated cleanups bundled because they touch the same publishing/
workspace plumbing:
1. **`pacquet/npm/pacquet/package.json` metadata** — the imported file
still pointed at the standalone repo: `repository.url` was
`pnpm/pacquet`, `repository.directory` was `npm/pacquet`, `homepage`
and `bugs` likewise. Repoint at `pnpm/pnpm`. Update
`generate-packages.mjs` so the per-platform packages it emits at
release time also point at `pnpm/pnpm`.
2. **Fold `pacquet/tasks/registry-mock` into the root pnpm workspace**.
Pacquet's standalone-repo nested-workspace setup pinned
`nodeLinker: hoisted` "for verdaccio CJS resolution," but pnpm's
own jest globalSetup (`__utils__/jest-config/with-registry/`) calls
the same `@pnpm/registry-mock.start()` API under the default
isolated linker without issue, and verified locally that
`node launch.mjs prepare` works after consolidation. The hoisted
constraint was scoped to standalone-pacquet's install pattern; in
the monorepo it's unnecessary.
Changes for (2):
- Add `pacquet/tasks/registry-mock` to `pnpm-workspace.yaml`.
- Rename the package `@pnpm-private/pacquet-registry-mock-launcher`
(private, matches the `@pnpm-private/*` convention used by other
internal workspace members) and switch `@pnpm/registry-mock` to
`catalog:` (the root catalog already pins it at 6.0.0).
- Delete `pacquet/tasks/registry-mock/pnpm-lock.yaml` and
`pnpm-workspace.yaml` — root install handles both now.
- Delete `pacquet/package.json` and `pacquet/pnpm-lock.yaml` — the
file only had a `cargo build` script + `devEngines: pnpm`, both
already covered by root, and nothing referenced it.
- `justfile install` is now just `pnpm install` (was
`cd pacquet/tasks/registry-mock && pnpm install --frozen-lockfile`).
- `pacquet-integrated-benchmark.yml` path filter and cache key
swap the deleted nested lockfile for the root `pnpm-lock.yaml` /
`pnpm-workspace.yaml`.
Verified: `pnpm install` resolves the workspace member, the lockfile
gains a `pacquet/tasks/registry-mock` importer entry, and
`pacquet/tasks/registry-mock/node_modules/@pnpm/registry-mock` is
linked correctly under the isolated layout.
* fix(pacquet): match meta-updater conventions in registry-mock launcher
The previous version of `pacquet/tasks/registry-mock/package.json`
omitted `version`, which crashed `pn lint:meta` (meta-updater hits
`manifest.version!.split('.')[0]` for every workspace package).
Backfill all the fields meta-updater would emit for an internal
`@pnpm-private/*` private package, matching `__typecheck__/package.json`
and friends:
- `version: 1100.0.0` (the pnpm 11.x convention for non-experimental
internal packages)
- self-devDep entry (`workspace:*`) that meta-updater would otherwise
inject
- `keywords: [pnpm, pnpm11]`
- `repository` pointing at this directory inside pnpm/pnpm
This is the same shape every other `@pnpm-private/*` private workspace
member uses; it lets `pn lint:meta --test` pass without modifying the
file.
* fix: update lockfile
* chore(pacquet): add build:pacquet script at root
Restore the `cargo build --release --bin pacquet` shortcut that lived in
the deleted `pacquet/package.json`. Naming it `build:pacquet` (rather
than `build`) matches the existing namespacing convention in this
file (`lint:ts`, `lint:meta`) and leaves room for a general `build`
script later. Invoke with `pnpm build:pacquet` or `pn build:pacquet`
from the repo root.
|
||
|
Zoltan Kochan
|
2e33bb1955 |
docs: split AGENTS.md into shared + pacquet-specific (#11640)
* docs: split AGENTS.md into shared + pacquet-specific
Before: root AGENTS.md and pacquet/AGENTS.md each maintained their own
copy of the GitHub PR workflow, agent-footer rule, "never ignore test
failures," Conventional Commits list, and code-reuse philosophy.
Drift waiting to happen.
After:
- Root AGENTS.md owns the shared conventions (PR workflow, agent
footer, conventional commits, code reuse, never-ignore-tests,
PR-conflict script) and marks TS-only sections explicitly
(setup/build, testing, linting, changesets, Standard Style, Jest
gotchas).
- pacquet/AGENTS.md opens with "Read ../AGENTS.md first" and keeps
only pacquet-specific rules (cardinal rule, branded types, just
recipes, insta snapshots, miette diagnostics, Rust style notes,
the `bench:` commit type, things-not-to-do that are Rust-flavored).
- Root adds a one-line entry for `pacquet/` in the repo structure
list so first-time readers find the cross-link.
CLAUDE.md and pacquet/{CLAUDE,GEMINI}.md are unchanged — they're
symlinks to AGENTS.md and follow automatically.
* docs(agents): require parity between pnpm and pacquet
Add a "Keep pnpm and pacquet in sync" section to root AGENTS.md spelling
out the bidirectional obligation: any user-visible change (CLI surface,
lockfile/manifest format, error codes, defaults, env-var handling, log
emissions, store layout) must land in both stacks in the same PR, or
the originating PR must spawn a tracking issue. Pure refactors / perf
wins / TS-only test cleanups don't need mirroring.
Cross-link from pacquet/AGENTS.md's "cardinal rule" so a pacquet-side
reader knows the obligation goes both ways and where the pnpm-side
version lives.
* docs(agents): restore Rust-specific dependency-level guidance
The root "Keep the dependency on the right level" bullet uses npm
vocabulary ("package," "shared package"). For a Rust reader that
required mentally translating "package" → "crate" and made the
workspace-vs-crate distinction less obvious. Restore the pacquet
phrasing alongside the existing pacquet-specific notes.
* docs(agents): hand off cross-stack porting via the same PR
Drop the "open a tracking issue" fallback — it lets one side drift
behind while the issue sits in the backlog. Instead, the PR author
opens the PR with their side and flags in the description what still
needs porting; someone else pushes the matching commits to the same
PR before it lands. Both sides land together or not at all.
* docs(agents): drop external-repo framing from the cardinal rule
pacquet now lives in the same repo as pnpm, so the cardinal rule no
longer needs the "fetch pnpm/pnpm main, compare ls-remote SHAs, watch
your local clone for drift" mechanics. The reference TypeScript code
is just a few directories over (`pnpm/`, `pkg-manager/`, `resolving/`,
`lockfile/`, `store/`, etc.), and pnpm is the source of truth by
position in the repo, not by branch tracking.
Updates:
- Root `AGENTS.md`: rephrase the cross-link line to drop the "follow
pnpm's main" framing.
- `pacquet/AGENTS.md` cardinal rule: redirect "find the equivalent
code" from `https://github.com/pnpm/pnpm` to the in-repo
TypeScript workspaces, drop the "confirm you're on the freshest
main" paragraph, and reword the source-of-truth wording.
- Permalink citation rule: generalize from "upstream pnpm" to "any
GitHub repository, including this one" — citation SHAs now usually
point at this repo's history.
* docs(agents): note pacquet's current scope is install-only
Without this caveat the parity rule reads as if every command needs
porting today. pacquet only implements `install` right now; resolution
and other commands (`update`, `add`, `remove`, `publish`, `exec`,
`run`, `dlx`, `audit`, etc.) live only in TypeScript, so changes there
don't need a pacquet-side port. The caveat also flags that the parity
rule's scope will widen as pacquet ports more commands.
|
||
|
Zoltan Kochan
|
763ddf1c99 |
chore(pacquet): wire pacquet workflows into monorepo (#11635)
* chore(pacquet): wire pacquet workflows into monorepo
Move Cargo workspace, Rust toolchain configs, justfile, composite actions,
and 7 workflow files out of `pacquet/` and up to the repo root so:
- cargo / just / taplo run from repo root, the way the rest of the
monorepo's tooling does
- GitHub Actions actually discovers the workflows (it only reads
`.github/workflows/` at the repo root)
Workflows are prefixed with `pacquet-` and renamed to "Pacquet ..." so
they don't collide with the existing pnpm CI. Path filters are scoped
to `pacquet/**` so they don't trigger on every commit. The cargo entry
from pacquet's standalone `dependabot.yml` is folded into the root one;
pacquet's `CODEOWNERS` and `pull_request_template.md` are dropped because
the root copies supersede them.
Path rewrites:
- `Cargo.toml` workspace members → `pacquet/crates/*`, `pacquet/tasks/*`
- all path-deps in `[workspace.dependencies]` → `pacquet/...`
- `justfile` recipes (`install`, `install-hooks`) point at `pacquet/...`
- `.taplo.toml` include globs → `pacquet/crates/*/*.toml`, `pacquet/tasks/*/*.toml`
- `pacquet/npm/pacquet/scripts/generate-packages.mjs` REPO_ROOT walks one
more level up
- workflow `paths:` filters, `hashFiles(...)`, and shell paths all updated
Verified: `cargo metadata` resolves the workspace, `cargo fmt --check`
clean, `taplo format --check` picks up all 26 Cargo.tomls, `actionlint`
reports no new issues (the `type:`-on-input warnings on the rustup action
predate this move).
* chore(pacquet): drop pnpm version pin from pacquet CI workflows
The monorepo's root `package.json` declares `pnpm@11.1.1` under
`packageManager`, which conflicts with the workflows' explicit
`version: 11.0.0-rc.5` and trips `pnpm/action-setup` ERR_PNPM_BAD_PM_VERSION.
The pin was a pacquet-era workaround for the v9 lockfile while pnpm 11
was still pre-release. Stable 11.x writes v9 too, so let action-setup
read the version from `packageManager` like every other workflow in
this repo does.
* chore(pacquet): use pnpm/setup matching the rest of the monorepo
Replaces `pnpm/action-setup@v6` with the same `pnpm/setup@b1cac3...`
SHA the rest of pnpm/pnpm uses (release.yml, test.yml, ci.yml,
benchmark.yml, audit.yml). Reads pnpm version from `packageManager`
in root package.json, and skips the implicit `pnpm install` since
pacquet does its own scoped install via `just install` (which only
touches `pacquet/tasks/registry-mock/`).
The release workflow now also installs Node via the same action
(`runtime: node@22`) instead of via `pnpm runtime -g set node 22`,
since pnpm/setup handles runtimes in one step.
* chore(pacquet): tighten permissions and Dependabot cooldown
Address zizmor warnings on the pacquet CI changes:
- `dependabot.yml`: the cargo entry I added in the previous commit
inherited from pacquet's standalone repo and is missing the
`cooldown: default-days: 7` the github-actions entry uses. Add it
so cargo and github-actions debounce updates consistently.
- `pacquet-ci.yml`, `pacquet-codecov.yml`, `pacquet-cargo-unused.yml`
lacked a top-level `permissions:` block, so GITHUB_TOKEN inherited
the repo default. Declare `contents: read` — every job in these
workflows only reads the repo and runs local checks.
The other four pacquet workflows already declare permissions
explicitly (integrated-benchmark/comment, micro-benchmark, release).
* chore(pacquet): add "reimagining" to cspell dictionary
cspell at the repo root scans all `**/README.md` and was rejecting
`pacquet/README.md` and `pacquet/npm/pacquet/README.md`, which describe
pacquet as "not a reimagining of pnpm." Add the word to the existing
allow-list rather than rewording two READMEs imported from a separate
repo.
* fix(pacquet): prefix workspace-relative paths with pacquet/
Two Rust source files looked up paths off the cargo workspace root
(\`cargo locate-project --workspace\`), which now resolves to the
monorepo root rather than the pacquet directory. Add the \`pacquet/\`
prefix:
- \`tasks/registry-mock/src/dirs.rs\` — \`registry_mock()\` was
pointing the node launcher at \`<repo>/tasks/registry-mock/launch.mjs\`
instead of \`<repo>/pacquet/tasks/registry-mock/launch.mjs\`, which
failed every Pacquet CI test job ("Cannot find module ...launch.mjs").
- \`tasks/micro-benchmark/src/main.rs\` — same idea for the
fixtures folder.
|
||
|
Zoltan Kochan
|
ceec335e85 |
fix(package-manager): link optionalDependencies siblings into slot (#526)
`create_symlink_layout` only iterated `snapshot.dependencies`, so a
package whose CPU/OS-specific siblings live entirely under
`optionalDependencies` (e.g. `@typescript/native-preview`,
`@reflink/reflink`, every `*-darwin-arm64` / `*-linux-x64` family)
ended up with a slot `node_modules/<scope>/` containing only the
parent package — no platform binary sibling. Consumers that do
`require.resolve('@typescript/native-preview-darwin-arm64')` from
inside `getExePath.js` walked parent directories and found nothing,
so `tsgo --version` (and every other tool that delegates to a
platform variant) crashed with `Unable to resolve … missing the
package on disk`.
Port upstream's `dependencies ∪ optionalDependencies` merge — the
graph builder at
https://github.com/pnpm/pnpm/blob/da65e6262/deps/graph-builder/src/lockfileToDepGraph.ts#L150-L156
unifies both maps into one `allDeps` for each node's children, and
`linkAllModules` then symlinks every child with two short-circuits:
`alias === depNode.name` (a snapshot referencing itself) and
`!pkg.installable && pkg.optional` (a non-materialized optional).
See https://github.com/pnpm/pnpm/blob/da65e6262/installing/deps-installer/src/install/link.ts#L521-L549.
`create_symlink_layout` now takes both maps and a `SkippedSnapshots`
reference, merges them, and applies both short-circuits. The skip
set is threaded through `InstallPackageBySnapshot` (cold batch) and
the warm-batch `CreateVirtualDirBySnapshot` call site in
`CreateVirtualStore::run`, so a target dropped by the installability
pass, by `--no-optional`, or by a swallowed optional fetch failure
gets no dangling symlink.
Five new unit tests in `create_symlink_layout/tests.rs` cover the
matching-optional happy path, the skipped-optional dangling-link
guard, the self-name guard for entries listed in either bucket, the
both-buckets-absent no-op, and the alias-resolve path (aliased deps
still link the alias filename while resolving the slot via the
target's name). End-to-end verification: `pacquet install
--frozen-lockfile` followed by `tsgo --version` in the pnpm v11
repo now succeeds; the matching `native-preview-darwin-arm64`
sibling shows up in the slot's `node_modules/@typescript/`.
---
Written by an agent (Claude Code, claude-opus-4-7).
|
||
|
Zoltan Kochan
|
da65e62625 |
fix: pacquet install --frozen-lockfile works against pnpm v11 repos (#525)
* fix(lockfile): skip env document in pnpm v11 combined lockfiles Pnpm v11 writes `pnpm-lock.yaml` as a stream of up to two YAML documents: an optional first document carries the package-manager bootstrap (`packageManagerDependencies` and the snapshots that back it) and the second document is the regular project lockfile. Pacquet hands the file straight to `serde_saphyr::from_str`, which rejects a multi-document stream with "multiple YAML documents detected" — so every install against a v11 repo that has a `packageManager` / `devEngines.packageManager` declaration fails before staleness checking even runs. Port upstream's `extractMainDocument` to a new `yaml_documents` module: if the file starts with `---\n`, return the slice after the next `\n---\n` separator; if there is no second separator, the file is env-only and the loader returns `Ok(None)`. `load_from_path` threads the lockfile content through the filter before deserializing, matching pnpm's `_read` call site at https://github.com/pnpm/pnpm/blob/31858c544b/lockfile/fs/src/read.ts#L103-L110. Three unit tests in `yaml_documents/tests` mirror upstream's `extractMainDocument` test cases, and two integration tests in `load_lockfile/tests.rs` cover the combined-document and env-only paths end-to-end. --- Written by an agent (Claude Code, claude-opus-4-7). * fix(package-manifest): reify devEngines.runtime into devDependencies With #511 / #512 pacquet recognises `runtime:` specifiers in the lockfile, so a frozen install against a v11 repo gets past the YAML parse but then trips the staleness check: the lockfile lists `node@runtime:24.6.0` under the root importer's `devDependencies`, while the on-disk `package.json` only declares the runtime through `devEngines.runtime`. The flat-record diff then surfaces a spurious "node@runtime:24.6.0 was removed" mismatch. Port upstream's `convertEnginesRuntimeToDependencies` to a new free function in `crates/package-manifest`. For each of `node`, `deno`, `bun`, if `devEngines.runtime` (or `engines.runtime`) declares the runtime with `onFail: "download"` and an explicit version, and the target dependencies bucket has no explicit entry yet, insert `<name>: "runtime:<version>"`. Array and single-object runtime shapes are both accepted. Skip when `onFail` is anything other than `"download"` or when the version is absent — upstream warns on the missing-version path; pacquet skips silently and the staleness check still surfaces the gap if it matters downstream. WebContainer's "no runtime download" branch is intentionally omitted since pacquet does not run there. `PackageManifest::read_from_file` calls the function for both `(devEngines, devDependencies)` and `(engines, dependencies)`, mirroring upstream's `convertManifestAfterRead` at https://github.com/pnpm/pnpm/blob/9cad8274fd/workspace/project-manifest-reader/src/index.ts#L227-L231. Six unit tests in `tests.rs` cover the happy path, the no-version skip, the non-`download` `onFail` skip, preservation of an explicit user-declared entry, the array-of-runtimes form, and the `engines` → `dependencies` variant. A seventh test exercises the hook through `PackageManifest::from_path` end-to-end. Upstream reference: https://github.com/pnpm/pnpm/blob/9cad8274fd/pkg-manifest/utils/src/convertEnginesRuntimeToDependencies.ts#L10-L45. --- Written by an agent (Claude Code, claude-opus-4-7). |
||
|
Zoltan Kochan
|
b07b152c19 |
feat(lockfile,package-manager,real-hoist): npm-alias importer dep versions (#524)
pnpm writes importer dependency `version:` fields in three shapes:
bare semver-with-peer (`4.0.0`), `link:<path>`, or — when a specifier
(typically `catalog:`) resolves to a different package name — the full
npm-alias `<name>@<version>`. The third shape is what
`refToRelative` recognises with the same leading-`@` / `@` before
`(`/`:` test that `SnapshotDepRef` already uses, and which pnpm v11
emits for entries like:
js-yaml:
specifier: 'catalog:'
version: '@zkochan/js-yaml@0.0.11'
Pacquet's `ImporterDepVersion` only modelled `Regular` and `Link`, so
deserialising a lockfile with an aliased catalog dep failed with
"Failed to parse importer dependency version".
Add an `Alias(PkgNameVerPeer)` variant and a `resolved_key` helper
that returns the correct snapshot-map key for each shape — the
importer-map key paired with the version for `Regular`, the alias's
own `(name, suffix)` for `Alias`, and `None` for `Link`. Every site
that previously built a key from `as_regular().map(|v| PkgNameVerPeer::new(name, v))`
now goes through `resolved_key`, so aliased deps reach the snapshot,
the skipped-set, the reachability BFS, the build-sequence root walk,
the runtime exclusion check, and both hoist passes correctly.
For symlink targets, an aliased dep links the importer-key name to
`<slot>/node_modules/<alias-real-name>` (the resolved package's true
name inside its slot), matching pnpm's `linkDirectDeps`. The
`pnpm:root added` event now reports `realName` as the resolved
package name for aliases, where before it always echoed the
importer-map key.
Upstream reference: `refToRelative` in
`pnpm/pnpm@8a80235c7b/deps/path/src/index.ts:96-110`.
|
||
|
Zoltan Kochan
|
fb221c626c |
feat(cmd-shim,package-manager): hoisted-bin precedence + post-build top-level bin re-link (#342) (#523)
Two related bin-linking behaviors deferred from #333. Behavior 1 — hoisted-bin precedence: - Add `BinOrigin { Direct, Hoisted }` discriminator to `PackageBinSource`. - New top tier in `pick_winner`: Direct wins outright over Hoisted regardless of ownership / lexical order. Mirrors upstream's `preferDirectCmds` partition at https://github.com/pnpm/pnpm/blob/4750fd370c/bins/linker/src/index.ts#L92. - New `link_top_level_bins(modules_dir, direct, hoisted)` helper in `pacquet-package-manager` mixes both candidate lists into one `link_bins_of_packages` call so the new tier resolves conflicts in a single pass — previously the two passes (SymlinkDirectDependencies for direct + hoist pass for publicly-hoisted) wrote shims independently and a hoisted bin could shadow a direct one when its package name was lexically smaller. Behavior 2 — lifecycle-script-created bins: - Add a post-`BuildModules` per-importer top-level bin link pass. Re-reading each direct dep's `package.json` after lifecycle scripts run picks up bins generated by `postinstall` (the `@pnpm.e2e/generated-bins` upstream fixture). Idempotent for unchanged shims via `is_shim_pointing_at`. - Mirrors upstream's `linkBinsOfImporter` pass that runs after `buildModules` at https://github.com/pnpm/pnpm/blob/4750fd370c/installing/deps-installer/src/install/index.ts#L1539. Supporting changes: - `PackageBinSource::new(location, manifest)` constructor + `with_origin` builder so existing call sites don't have to spell out the new field. - Public `direct_dep_names_for_importer` helper extracted from `SymlinkDirectDependencies` so the post-build pass uses the same filter (skipped / first-wins / link_only) as the symlink phase. - `InstallFrozenLockfileError::TopLevelBinLink` for the new failure surface. Tests: - `direct_origin_wins_over_hoisted_regardless_of_lexical` — pins the new tier overrides lexical ordering. - `hoisted_origin_loses_to_existing_direct` — pins both arms of the new tier (Direct incumbent vs Hoisted candidate). |
||
|
Zoltan Kochan
|
1ad6ffd152 |
feat(config,package-manager): hoistingLimits + externalDependencies knobs (#438 slice 10) (#522)
Plumbs the two programmatic-only hoister knobs from
`pnpm-workspace.yaml` through to the slice 4 walker and the slice 3
hoister. Both fields already existed on `HoistOpts`; this slice wires
them end-to-end.
- `Config::hoisting_limits: BTreeMap<String, BTreeSet<String>>` —
per-importer block-list, locator-keyed (`'.@'` for the root). Reads
`hoistingLimits: { ".@": [foo, bar] }` from yaml. Mirrors upstream's
https://github.com/pnpm/pnpm/blob/94240bc046/installing/linking/real-hoist/src/index.ts#L10
programmatic-only knob, exposed as yaml for parity since the
ergonomics of the locator-keyed map don't translate to a CLI flag.
- `Config::external_dependencies: BTreeSet<String>` — name slots
reserved at the root for an external linker (the Bit CLI is the
only known consumer upstream). Reads `externalDependencies: [...]`
from yaml.
- `LockfileToHoistedDepGraphOptions` gains both fields and forwards
them to `HoistOpts` in `build_dep_graph`.
- `InstallFrozenLockfile::run` clones the two `Config` fields into the
walker opts.
Both knobs default to empty (no limits, no externals), matching
upstream's default. Neither has any effect under `nodeLinker:
isolated` — the isolated linker keeps per-importer subtrees by
construction and doesn't consult the hoister.
Tests:
- `parses_hoisting_limits_from_yaml_and_applies` — yaml round-trip +
apply_to.
- `parses_external_dependencies_from_yaml_and_applies` — same.
- `omitting_hoisting_limits_and_external_dependencies_keeps_defaults`
— pins the apply_to skip-on-None branch so a yaml without these
keys doesn't accidentally overwrite Config defaults.
- `walker_forwards_external_dependencies_to_hoister` — end-to-end:
the walker observes an empty graph for an externalised alias
because the hoister stripped it. Pins the slice 10 plumbing.
|
||
|
Zoltan Kochan
|
6db0430e27 |
feat(real-hoist,package-manager): workspace-aware hoisting (#438 slice 9) (#521)
Enables `nodeLinker: hoisted` for multi-importer (workspace) lockfiles. Previously the hoister rejected any lockfile with importers beyond `.` with `UnsupportedWorkspace`; now the whole workspace shares one hoist plan so conflicting versions across projects dedupe, and each importer's project-tree node_modules is materialized per-project. real-hoist: - Drop the upfront UnsupportedWorkspace guard in `hoist()`. The wrapper already constructed non-root importers as Workspace-kind children of the virtual `.` root; only the guard needed to go. - Add `HoistOpts::hoist_workspace_packages: bool` (default true). When false, non-root importers stay out of the shared tree (matches upstream's `hoistWorkspacePackages: false` mode for the Bit CLI). Removed the corresponding error variant. hoisted_dep_graph walker: - Replace the `workspace:`-prefix skip with a recurse-into branch: for each Workspace-kind hoister child, walk its post-hoist children under `<lockfile_dir>/<importer_id>/node_modules`. The workspace node itself is NOT added to the graph or to the parent's hierarchy — it has no contents to import. - Add per-importer `hierarchy` and `direct_dependencies_by_importer_id` entries. Per-importer direct deps are computed from the lockfile (not from the workspace node's post-hoist children) because hoisted-up siblings don't show up in the workspace node's tree. First-recorded location wins, matching upstream's `pkgLocationsByDepPath[depPath][0]` pick. - Add `LockfileToHoistedDepGraphOptions::hoist_workspace_packages` (default true) and plumb to HoistOpts. Config: - Add `Config::hoist_workspace_packages: bool` (#[default = true]). Read from `pnpm-workspace.yaml` via `WorkspaceSettings`. SymlinkDirectDependencies: - Add `link_only: bool` flag. When true, skip every Regular dep and only materialize Link entries (workspace siblings). Used by the hoisted branch in InstallFrozenLockfile::run so workspace-sibling symlinks land under each importer's `node_modules/<alias>` even though the regular deps now live as real directories from the hoisted linker. InstallFrozenLockfile::run: - Plumb `config.hoist_workspace_packages` into the walker. - After link_hoisted_modules, run SymlinkDirectDependencies with `link_only: true` so workspace siblings get their per-project symlinks. Mirrors upstream's hoisted branch at https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-restorer/src/index.ts#L411-L440. Tests: - real-hoist: replace the now-removed `UnsupportedWorkspace` test with one that pins multi-importer hoister output (Workspace children with percent-encoded names + `workspace:<id>` references) and one that pins the `hoist_workspace_packages: false` opt-out. - walker: three new tests cover per-importer direct_deps emission, per-importer hierarchy entries, the `hoist_workspace_packages: false` root-only mode, and version-conflict-across-importers nesting. |
||
|
Zoltan Kochan
|
c2aa928d83 |
test(package-manager): variant-mismatch + --no-runtime install tests (#437 slice F) (#516)
* test(package-manager): variant-mismatch + --no-runtime install tests (#437 slice F) Two end-to-end install tests for the runtime-dependency pipeline, now that #512 (`PkgVerPeer::Prefix::Runtime`) unblocked fixture-lockfile parsing. - `frozen_lockfile_install_errors_when_no_variant_matches_host`: Lockfile with a `VariationsResolution` whose only variant targets `aix/ppc64` (a platform pacquet CI never runs on). Install fails with `NoMatchingPlatformVariant` from the cold-batch dispatcher. Variant selection runs before any network fetch, so the bogus archive URL is never read — the test stays hermetic. Closes the variant-mismatch checkbox. - `frozen_lockfile_install_skips_runtime_when_skip_runtimes_set`: Same lockfile, but with `skip_runtimes: true`. The `--no-runtime` filter iterates importer-direct deps, builds `node@runtime:22.0.0` from `(alias, version)`, sees the `@runtime:` substring, and adds the snapshot to the skip set — so variant selection never runs and the unmatchable variant doesn't fail the install. Asserts both the runtime slot and the direct-dep symlink are absent. Closes the `--no-runtime` checkbox. Both tests were verified to catch their respective regressions (disable the `skip_runtimes` filter → the second test fails with `NoMatchingPlatformVariant` instead of succeeding). Required a fix in `PkgNameVerPeer::without_peer`: the existing implementation dropped the `Prefix::Runtime` when stripping the peer-dependency suffix, so a runtime snapshot key like `node@runtime:22.0.0(some@peer)` would resolve to the non-existent `node@22.0.0` `packages:` entry. Now preserves the prefix through the strip. Slice F's remaining items (happy-path archive fetch, integrity mismatch, NODE_EXTRAS verified end-to-end, TEST_PORTING.md entries) need a mock HTTP server + recorded archive fixture, so defer to a follow-up — keeping this PR reviewable. * test(package-manager): use symlink_metadata for absent-entry checks Address CodeRabbit review on #516: `Path::exists()` follows symlinks, so a regression that created a *dangling* symlink at `<modules_dir>/node` would still pass the `!direct_dep.exists()` assertion — the install would have created the entry the skip set was supposed to suppress, but the test wouldn't notice. Switch both "must NOT exist" assertions to `std::fs::symlink_metadata(...).is_err()` so the entry-level check catches dangling symlinks too. |
||
|
Zoltan Kochan
|
6018359e4c |
feat(package-manager): BuildModules under hoisted node-linker (#438 slice 7) (#520)
Re-enables `BuildModules` for `nodeLinker: hoisted` installs. Slice 6 landed the hoisted install pipeline but skipped the build phase entirely because `BuildModules` walked virtual-store slot directories that don't exist under hoisted. Slice 7 routes the build phase through the slice 4 walker's per-node `dir` so postinstall scripts can run against the on-disk hoisted tree. Changes: - `BuildModules` gains two fields: `pkg_root_by_key: Option<&HashMap<PackageKey, PathBuf>>` overrides the per-snapshot pkg_root lookup with the slice 4 walker's `DependenciesGraphNode::dir` values; `gather_ancestor_bin_paths: bool` switches `extra_bin_paths` to the new `bin_dirs_in_all_parent_dirs` helper, a port of upstream's `binDirsInAllParentDirs` at https://github.com/pnpm/pnpm/blob/94240bc046/building/after-install/src/index.ts#L476-L487. - `pkg_root_for_key` helper: routes between the layout-based slot computation (isolated) and the override map (hoisted). Hoisted snapshots absent from the map (walker-dropped) take the same exit as the isolated `!pkg_dir.exists()` skip. - `InstallFrozenLockfile::run` now builds a snapshot-key → first-recorded-dir map from `walker_result.graph.values()` and threads it (plus `gather_ancestor_bin_paths: true`) into `BuildModules` instead of skipping the phase. Multiple graph nodes with the same dep_path collapse to the first entry, matching upstream's `pkgRoots[0]` pick at https://github.com/pnpm/pnpm/blob/94240bc046/building/after-install/src/index.ts#L348. - New private `HoistedLinkerOutput` struct bundles `hoisted_locations` and `pkg_root_by_key` so the hoisted-branch return doesn't trip `clippy::type_complexity`. Side-effects-cache key shape is unchanged — it's keyed by `pkg_id_with_patch_hash` + dep-graph hash, both layout-independent (`crates/graph-hasher/src/dep_state.rs`). `MISSING_HOISTED_LOCATIONS` is intentionally deferred — pacquet has no `rebuild` command, so the install path always re-runs the walker and never reads `.modules.yaml.hoisted_locations`. Tracked as a follow-up for when `pacquet rebuild` lands. Workspace-aware hoisting (slice 9) and `hoistingLimits` / `externalDependencies` (slice 10) remain. Tests: - Three new helper tests pin `bin_dirs_in_all_parent_dirs` against top-level, conflict-nested, and scoped-package shapes. - Three new helper tests pin `pkg_root_for_key` for the isolated pass-through, the hoisted override hit, and the hoisted-missing short-circuit. |
||
|
Zoltan Kochan
|
2c8f8ad60c |
feat(package-manager): wire nodeLinker=hoisted into install pipeline (#438 slice 6) (#518)
* feat(package-manager): wire node_linker hoisted into install pipeline (#438 slice 6) Branches `Install::run` / `InstallFrozenLockfile::run` on `config.node_linker == Hoisted`. Threads the slice 4 `lockfile_to_hoisted_dep_graph` walker output and the per-package CAS index produced by `CreateVirtualStore` (slot writes skipped under hoisted) into the slice 5 `link_hoisted_modules` linker. Persists the walker's `hoisted_locations` into `.modules.yaml` for rebuild and follow-up installs to consume. Pipeline changes under hoisted: - `CreateVirtualStore` skips `CreateVirtualDirBySnapshot` for both warm and cold batches, collects each snapshot's CAS file index keyed by `PkgIdWithPatchHash` into a new `cas_paths_by_pkg_id` output field. - `InstallPackageBySnapshot::run` returns the per-package CAS map unconditionally and skips the virtual-store-slot write when its new `node_linker` field is `Hoisted`. - `InstallFrozenLockfile::run` skips `SymlinkDirectDependencies`, `LinkVirtualStoreBins`, the isolated hoist pass, and `BuildModules` under hoisted; runs `lockfile_to_hoisted_dep_graph` + `link_hoisted_modules` in their place. Folds the walker's augmented skip set back into the install-time `SkippedSnapshots` so `.modules.yaml.skipped` reflects the union. - `Install::run`'s `build_modules_manifest` now takes the walker's `hoisted_locations` and writes it through `Modules.hoisted_locations` (only when non-empty so the isolated path doesn't produce a hoisted-only key). The build phase under hoisted (rebuild over `hoistedLocations` with ancestor-`.bin` lookup, `MISSING_HOISTED_LOCATIONS`) is slice 7's scope and is intentionally left as a no-op here. Workspace and `hoistingLimits` / `externalDependencies` knobs are slices 9-10. Mirrors upstream's hoisted branch in `headlessInstall` at https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-restorer/src/index.ts#L369-L425. --- Written by an agent (Claude Code, claude-opus-4-7). * fix(package-manager): docs intra-link disambiguation + skip-set merge fix (#438 slice 6) Docs CI was failing because `[`crate::link_hoisted_modules`]` is ambiguous between the function and the module of the same name. Disambiguate by switching to the function form `[`crate::link_hoisted_modules()`]` everywhere. Slice 6's hoisted-walker skip-set merge previously folded every entry in `walker_result.skipped` into `SkippedSnapshots::insert_installability`, which would promote pre-existing transient skips (`fetch_failed` / `optional_excluded`) into the persisted-on-disk `.modules.yaml.skipped` set. Diff against the input `walker_skipped` so only walker-discovered (genuinely-new) installability skips flow into the persisted subset. |
||
|
Zoltan Kochan
|
62900806fe |
feat(config,cli,tarball): support offline / preferOffline settings (#513)
* feat(config,cli,tarball): support `offline` / `preferOffline` settings Adds the two settings to `Config`, `pnpm-workspace.yaml` parsing, and `pacquet install` as `--offline` / `--prefer-offline` flags. CLI flags merge into Config (boolean OR) before leak, so any source (yaml, CLI) flips `true` and never the other way. Upstream pnpm gates the metadata-fetch path in [`pickPackage`](https://github.com/pnpm/pnpm/blob/94240bc046/resolving/npm-resolver/src/pickPackage.ts): when `offline: true` and no cached metadata exists, it fails with `ERR_PNPM_NO_OFFLINE_META`; `preferOffline: true` biases the resolver toward the cached copy. Pacquet's frozen-install path has **no metadata fetch** (the lockfile pins every resolution), so the upstream flag is semantically a no-op on pacquet's current flow. To make the flag actually useful for frozen installs, pacquet adds a **tarball-side gate** (no upstream equivalent — pnpm doesn't gate tarball downloads on `offline`): when both the warm prefetch and the SQLite `index.db` lookup miss, the fetcher refuses the network and errors with `ERR_PACQUET_NO_OFFLINE_TARBALL`. Same shape as `ERR_PNPM_NO_OFFLINE_META`, scoped to tarballs because that's what pacquet's frozen install needs network for. Documented as a pacquet- specific divergence in the field's rustdoc. `prefer_offline` is a no-op today — the warm prefetch + `load_cached_cas_paths` already prefer the local store. The field exists so `.npmrc` / yaml / CLI all parse cleanly; Stage 2's resolver will honour it on the metadata path. - `Config.offline` / `Config.prefer_offline`: bool, default `false`. - `WorkspaceYaml.offline` / `WorkspaceYaml.prefer_offline`: `Option<bool>`; applied via the existing `apply!` macro. - `InstallArgs.offline` / `InstallArgs.prefer_offline`: bool CLI flags; merged into Config in `cli_args.rs` between `config()` and `State::init` (the brief window when `Config::leak` returns `&'static mut Config`). - `DownloadTarballToStore.offline` / `DownloadZipArchiveToStore.offline`: bool fields. Wired through the three construction sites in `package-manager`. - `TarballError::NoOfflineTarball { package_id, url }`: new variant. `tarball_error_to_request_retry` updated for exhaustiveness; `code: ERR_PACQUET_NO_OFFLINE_TARBALL` in the retry-logger projection so a future surface that runs this error through the retry path renders the right code. - `offline_mode_skips_network_on_cache_miss` — `mockito` server with `.expect(0)`; the offline gate must fire before the network is touched. Asserts variant shape AND the diagnostic code (the code is part of the user-facing surface). - `offline_mode_still_uses_prefetched_cache` — same `.expect(0)` guard but with a prefetched cache hit; pins that the prefetch branch short-circuits *before* the offline check so warm installs under `--offline` succeed. Regression-catch verified: gating the gate behind `false &&` flipped `offline_mode_skips_network_on_cache_miss` red with the wrong-variant panic message. Reverted before commit. All bulk-edit fixups (15 `DownloadTarballToStore` / `DownloadZipArchiveToStore` literals in tests + micro-benchmark, 1 `Config` literal in package- manager tests) get `offline: false` to match the prior implicit default. * docs(cli): pin pickPackage link to commit SHA (#513 review) Per project guideline: upstream pnpm citations link to a specific commit SHA (first 10 hex chars), not a branch name. Same SHA the PR's rustdoc and commit body already use for the same file. Resolves CodeRabbit review comment on `crates/cli/README.md`: <https://github.com/pnpm/pacquet/pull/513#discussion_r3238182177>. |
||
|
Zoltan Kochan
|
888184226d |
feat(cli): --node-linker flag (#438 slice 8) (#514)
Add the `--node-linker [isolated|hoisted|pnp]` CLI flag to `pacquet install`, mirroring pnpm's flag. Overrides `Config.node_linker` for the invocation; absent flag → config's yaml/npmrc value wins. The CLI parses into a `NodeLinkerArg` mirror enum (kept in the CLI crate so `pacquet-config` stays free of `clap`), then `into_config()` converts to the canonical `pacquet_config::NodeLinker`. Threaded into the `Install` runner as an explicit `node_linker: NodeLinker` field rather than reading `config.node_linker` directly. Matches the existing pattern `supported_architectures` uses (`state.config` is `&'static`, so the override-merge happens at the CLI layer and lands here as a fully-resolved value). `build_modules_manifest` consumes it on the `.modules.yaml.nodeLinker` write so the persisted setting reflects the invocation, not just the config. `pacquet add` uses Config's value by default (`add` doesn't expose `--node-linker` per the umbrella scope; Slice 6's pipeline integration will revisit if needed). 5 new CLI tests: - `node_linker_default_is_none` — flag absent → field is None. - `node_linker_hoisted` / `node_linker_isolated` / `node_linker_pnp` — each value parses and round-trips through `into_config()`. - `node_linker_invalid_value_rejected` — clap rejects unknown values with the bad value in the error message. - `node_linker_arg_into_config_matches_every_variant` — every ValueEnum variant has a canonical mapping (compile-fails on future variants that forget the mapping). `NodeLinker` gains `Clone + Copy + Eq` so it can be threaded by value into `Install` and matched in tests. |
||
|
Zoltan Kochan
|
5483cc1661 |
feat(lockfile): support runtime: prefix on PkgVerPeer (#511) (#512)
Pnpm v11 writes runtime depPaths with a scheme prefix in front of
the semver — e.g. `node@runtime:22.0.0` in `packages:` /
`snapshots:` keys, and `runtime:22.0.0` in importer `dependencies:`
values. Pacquet's `PkgVerPeer` parser previously accepted only
bare semver and rejected the prefix, so it couldn't load any v11
lockfile containing a runtime dependency.
Extend `PkgVerPeer` to recognise a leading `runtime:` (closed enum
`Prefix::{None, Runtime}`) and preserve it through `Display` /
`serde`. The version + peer parts continue to parse the same way
— the prefix-strip happens once up front, after which the
parenthesis-based peer-suffix detection is unchanged.
Touch points:
- Added `Prefix` enum with `as_str()` and `Display`.
- `PkgVerPeer::prefix()` exposes the variant so downstream
consumers can discriminate runtime entries without
substring-searching the depPath.
- `PkgVerPeer::into_tuple()` keeps the `(Version, String)` shape
for backward compatibility — pre-runtime callers don't see the
prefix.
5 new unit tests pin: bare-semver no-prefix, runtime no-peer,
runtime with peer-suffix, runtime-substring-mid-version is NOT a
prefix (anchored at start), and serde round-trip on the runtime
form. All 100 existing `pacquet-lockfile` tests still pass.
Unblocks #437 slice F (end-to-end install fixtures) — pacquet's
lockfile loader now accepts `node@runtime:X.Y.Z`-shaped keys.
|
||
|
Zoltan Kochan
|
a603a52523 |
feat(network): switch reqwest TLS backend to rustls for PKCS#1 + EC keys (#509)
Replaces the `native-tls-vendored` reqwest feature with `rustls`, closing the PKCS#1 parity gap flagged in #499. Native-tls's `Identity::from_pkcs8_pem` accepted only `-----BEGIN PRIVATE KEY-----` (PKCS#8); rustls's `Identity::from_pem` accepts PKCS#1 (`-----BEGIN RSA PRIVATE KEY-----`), PKCS#8, and EC keys — the same surface Node's `tls.createSecureContext` exposes to pnpm via undici. Pacquet now matches pnpm bug-for-bug on the set of client-cert key formats accepted from `.npmrc`'s `key=` / `:key=` / `:keyfile=` entries. PKCS#12 (`.pfx`) stays out of scope — pnpm's `.npmrc` allow-list doesn't expose a `pfx=` option so pacquet doesn't either. ## What changed - `Cargo.toml`: drop `native-tls-vendored`, add `rustls`. Reqwest's `rustls` feature uses `aws-lc-rs` for crypto and `rustls-platform-verifier` for OS trust roots — closest behavioral match to native-tls's "consult the platform trust store" default. - `crates/network/src/lib.rs`: `apply_tls` swaps `Identity::from_pkcs8_pem(cert, key)` for `Identity::from_pem` applied to the concatenated `cert\nkey` PEM buffer. Adds a new `looks_like_pem_cert` syntactic check before `Certificate::from_pem` because rustls's `from_pem` stores the bytes verbatim and validates lazily at `Client::build()` time — a garbage CA entry would otherwise slip through silently and the install would proceed against an unknown trust root. - Updated doc comments on `apply_tls`, `TlsConfig::key`, `RegistryTls::key`, and `TlsError::InvalidClientIdentity` to describe the new surface and drop the PKCS#8-only caveat. - `deny.toml`: allow `CDLA-Permissive-2.0` for `webpki-root-certs` (pulled in by reqwest's `rustls` feature through `rustls-platform-verifier`'s fallback chain). - New fixtures at `crates/network/tests/fixtures/test-client-pkcs1.{crt,key}` loaded via `include_str!`. Regenerable with `openssl genrsa -traditional` + `openssl req -new -x509`. ## Tests `for_installs_with_pkcs1_client_key_builds` pins the contract — if a future change reverts the backend or otherwise narrows the accepted key formats, this build will fail with a clear `InvalidClientIdentity`. All 1175 workspace tests pass. ## Notes for review - **Cert store change.** `rustls-platform-verifier` reads the OS trust store on macOS / Windows / Linux. The lookup is a different syscall path from native-tls's; behavioral parity for "trust roots in the OS store" should hold, but corporate CAs that worked under native-tls and *don't* show up in `rustls-platform-verifier`'s enumeration would now silently fail. Users hitting that should add the CA explicitly via `cafile=` in `.npmrc`. - **Performance.** CI's integrated-benchmark will run on this PR; if it regresses materially on the warm-install path we'd consider falling back to the preprocessing approach (option 2 in #499). - **`hickory-dns` compatibility.** Verified by running the workspace test suite — DNS resolution is independent of the TLS backend. - **`cargo deny` posture.** One new license allowance (`CDLA-Permissive-2.0`) for `webpki-root-certs`. No new advisory surface area beyond what reqwest's `rustls` feature already pulls through `aws-lc-rs` / `rustls` / `rustls-platform-verifier`. |
||
|
Zoltan Kochan
|
38e2954f81 |
feat: ignoredOptionalDependencies config + lockfile field + outdated-setting check (#434 slice 7) (#507)
* feat: ignoredOptionalDependencies config + lockfile field + outdated-setting check (#434 slice 7) Last slice of the optional-dependencies umbrella (#434). Adds the user-facing `ignoredOptionalDependencies` setting: a list of dep-name patterns the user wants entirely excluded from resolution + install. Mirrors pnpm/pnpm@94240bc046's three surfaces: - **Hook**: `hooks/read-package-hook/src/createOptionalDependenciesRemover.ts` builds a matcher and drops matching keys from `optionalDependencies` AND `dependencies` (a package may list the same dep under both for "optional only when consumed"). - **Lockfile field**: `lockfile/types/src/index.ts:19` — `ignoredOptionalDependencies?: string[]` at the top level (sibling of `lockfileVersion`/`overrides`, NOT inside `settings`). - **Drift check**: `lockfile/settings-checker/src/getOutdatedLockfileSetting.ts:58-60` sorts both arrays and compares; mismatch triggers `needsFullResolution`. In pacquet's frozen-only flow this surfaces as `OutdatedLockfile`. ## Changes - **`pacquet-config`**: `Config::ignored_optional_dependencies: Option<Vec<String>>` + `WorkspaceSettings` field + `apply_to` wiring. - **`pacquet-lockfile`**: `Lockfile::ignored_optional_dependencies: Option<Vec<String>>` top-level field with serde round-trip; `check_lockfile_settings(lockfile, config_set)` sorts-and- compares; new `StalenessReason::IgnoredOptionalDependenciesChanged { lockfile, config }` variant. - **`pacquet-lockfile::satisfies_package_manifest`** extended with an `is_ignored_optional: &dyn Fn(&str) -> bool` parameter. Skips matching names in `flat_manifest_specs` and the per-field check so a manifest that still lists ignored entries doesn't falsely surface as drift against a lockfile the resolver correctly built without them. - **`pacquet-package-manager::install.rs`**: builds a matcher from `Config::ignored_optional_dependencies` (reuses `pacquet_config::matcher::create_matcher` — same glob engine as `hoistPattern`); calls `check_lockfile_settings` before `satisfies_package_manifest`; threads the matcher closure into the freshness check. - **`pacquet-package-manager::current_lockfile`**: preserves the lockfile's `ignored_optional_dependencies` through the slice 6 filter so the recorded set round-trips to the current lockfile. ## Tests - `pacquet_config`: yaml-parse + `apply_to` round-trip + omission baseline. - `pacquet_lockfile::freshness::check_settings_*`: both-sides- empty, sorted-match-regardless-of-order, drift in both directions. Test-the-test verified by removing the sort+compare guard. - `pacquet_lockfile::freshness::ignored_optional_filtered_*`: manifest-side filter passes when the matcher fires; polarity test confirms the unfiltered case surfaces as `SpecifiersDiffer`. - `pacquet_lockfile::freshness::ignored_optional_dependencies_round_trips_through_yaml`: serde wire-shape round-trip. Closes #503. Closes the #434 umbrella. * fix(lockfile): scope ignoredOptionalDependencies filter to prod+optional + set-based predicate CodeRabbit review on PR #507 (major): the filter wrongly applied to `devDependencies` too. Upstream's `hooks/read-package-hook/src/createOptionalDependenciesRemover.ts` iterates `manifest.optionalDependencies` and deletes matches from `optionalDependencies` AND `dependencies` only — `devDependencies` is untouched. The previous impl applied the filter to all three groups in both `flat_manifest_specs` and the per-field check, so a stale lockfile could incorrectly pass when the manifest added or removed a matching `devDependency`. Two fixes: 1. **Group gate** in `flat_manifest_specs` and the per-field check: apply the filter only when the group is `Prod` or `Optional`. `Dev` walks ignore the closure. 2. **Set-based predicate** at the call site (`Install::run`): build the "to drop" set from `manifest.optionalDependencies ∩ pattern`, not just from the pattern. A name listed only in `dependencies` (not `optionalDependencies`) that happens to match the pattern is NOT removed by upstream's hook (the hook never iterates that name). The set-based predicate captures that nuance. Both fixes together mirror the hook's exact semantics. Two new regression tests pin the dev-dependency behavior: `ignored_optional_does_not_apply_to_dev_dependencies` and `ignored_optional_dev_only_lockfile_entry_kept`. Test-the-test verified by dropping the group gate inside `flat_manifest_specs` — both tests fail. |
||
|
Zoltan Kochan
|
c7ab26a056 |
fix(package-manager): link_hoisted_modules PkgIdWithPatchHash types (#508)
The Slice 5 linker (#505) was written against `String` for `pkg_id_with_patch_hash` and merged onto a main that had already switched the underlying `DependenciesGraphNode` field to the `PkgIdWithPatchHash` newtype (#504). The merge auto-resolved without flagging the type incompatibility — `cargo check` on main now errors with: error[E0277]: the trait bound `String: Borrow<PkgIdWithPatchHash>` is not satisfied error[E0308]: mismatched types expected `String`, found `PkgIdWithPatchHash` Switch the two slice-5-introduced surfaces to match the newtype: - `CasPathsByPkgId = HashMap<PkgIdWithPatchHash, HashMap<String, PathBuf>>` — the per-package CAS index now keys on the brand the graph node carries, matching the post-#504 invariant across the workspace. - `LinkHoistedModulesError::MissingCasPaths.pkg_id_with_patch_hash: PkgIdWithPatchHash` — same type the graph node has, so the error round-trips the value without `.to_string()`. Tests updated to construct `PkgIdWithPatchHash::from(...)` keys/fields. All 8 linker tests pass on the fixed branch. This unblocks main building again. |
||
|
Zoltan Kochan
|
13a6970ae6 |
feat(package-manager): runtime bin-link + --no-runtime (#437 slice D3 + E) (#506)
* feat(package-manager): runtime bin-link + --no-runtime (#437 slice D3 + E) Closes the install-pipeline side of #437: **Slice D3 — runtime bin-link integration.** Runtime archives (`node@runtime:`, etc.) don't ship a `package.json`, so the existing bin-link step had nothing to consume off the slot. `fetch_binary_resolution_to_cas` now synthesizes one in the same shape upstream's `appendManifest` does: { "name": "<pkg.name>", "version": "<pkg.version>", "bin": <BinarySpec> } The bytes go through `StoreDir::write_cas_file` (same content- addressing as every other CAS-imported file), the resulting cas-path is inserted into the snapshot's `cas_paths` under `package.json`, and the existing `link_bins_of_packages` flow picks it up. `link_bins::build_has_bin_set` now includes `Binary` / `Variations` resolutions unconditionally — pnpm v11 doesn't emit `hasBin: true` on runtime metadata, but the synthesized manifest always carries bins, so the lookup must not be gated on the metadata flag. **Slice E — `--no-runtime` flag.** New `Config::skip_runtimes` (default false) and `InstallArgs::no_runtime` CLI flag. The CLI computes `config.skip_runtimes || --no-runtime` and threads it through `Install` → `InstallFrozenLockfile`. When set, every snapshot whose metadata resolution is `Binary` or `Variations` is added to the install-time skip set (re-using `add_optional_excluded` since the bucket count and `.modules.yaml.skipped` semantics line up with `--no-optional`). **Unit tests.** - `synthesize_runtime_manifest_emits_name_version_and_bin_single` — pins `BinarySpec::Single` → JSON string. - `synthesize_runtime_manifest_emits_name_version_and_bin_map` — pins `BinarySpec::Map` → JSON object with all bin entries. - `synthesize_runtime_manifest_preserves_scoped_name` — `@scope/name` round-trips through the `name` field. - `build_has_bin_set_includes_runtime_resolutions_even_when_has_bin_is_absent` — pins the runtime arm. Verified by removing the arm — test fails as expected. End-to-end install fixtures (slice F's remaining items — recorded-archive frozen-lockfile install, variant-mismatch error, `--no-runtime` integration, integrity-mismatch path) need a small recorded Node archive in the repo (or a mockable `pnpm-resolution-mirror`). Tracking them as a separate follow-up so this PR stays reviewable. * fix(package-manager): narrow --no-runtime to importer-direct deps Address CodeRabbit review on #506: my initial implementation iterated `packages` (every metadata entry) and added all `Binary` / `Variations` snapshots to the skip set, which widened the behavior beyond pnpm's `skipRuntimes`. Per the cardinal rule (CLAUDE.md: "Port behavior faithfully. ... Do not invent behavior that pnpm does not have."), match upstream exactly. Upstream's filter at [`installing/deps-installer/src/install/index.ts`](https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-installer/src/install/index.ts#L1374-L1387) iterates `dependenciesByProjectId` (per-importer direct deps) and adds `depPath` to `ctx.skipped` when `depPath.includes('@runtime:')`. Transitive runtime entries — unusual but possible — stay in the install. Pacquet now mirrors that exactly: walk each importer's `dependencies` / `dev_dependencies` / `optional_dependencies`, build the candidate snapshot key from `(alias, version)`, check for the `@runtime:` substring, and only add the metadata-confirmed `Binary` / `Variations` keys to `skipped`. Aliased runtime deps (unusual) fall through the same way upstream does — pnpm's lookup is by depPath, not by alias. |
||
|
Zoltan Kochan
|
be759bf13f |
feat(package-manager): linkHoistedModules linker (#438 slice 5) (#505)
* feat(package-manager): linkHoistedModules linker (#438 slice 5) New module `crates/package-manager/src/link_hoisted_modules.rs` produces the on-disk `node_modules/` tree from Slice 4's `LockfileToDepGraphResult`. Ports installing/deps-restorer/src/linkHoistedModules.ts. ## Three phases 1. **Orphan removal.** Every directory in `prev_graph` but not in `graph` is silently `rimraf`'d before any insert. `try_remove_dir` swallows all errors (NotFound + PermissionDenied + EBUSY), matching upstream's `tryRemoveDir` tolerance. 2. **Per-node import.** Hierarchy walked top-down, rayon-parallel at each level. Every node goes through `import_indexed_dir` with `force: true, keep_modules_dir: true` — the hoisted-linker call shape Slice 1 was designed for. 3. **Per-`node_modules` bin link.** After a level's children are all imported, `link_direct_dep_bins` populates `<parent>/node_modules/.bin` from the just-materialized direct children. Bin link runs only after every child's subtree is done so package.json reads always see the fully-populated package. ## API shape ```rust pub fn link_hoisted_modules<R: Reporter>( opts: &LinkHoistedModulesOpts<'_>, ) -> Result<(), LinkHoistedModulesError>; ``` `LinkHoistedModulesOpts` borrows `graph`, `prev_graph`, `hierarchy`, and a `cas_paths_by_pkg_id: HashMap<String, HashMap<String, PathBuf>>` keyed by `pkg_id_with_patch_hash`. ## Decoupling from the store Upstream's linker is async and calls `storeController.fetchPackage()` inline during the walk — pacquet's is sync and takes pre-fetched CAS paths. Slice 6 (the install pipeline) is responsible for fetching every package through pacquet's existing tarball / store-dir / git-fetcher machinery before invoking the linker. This keeps the linker focused on file-system layout and reuses the proven fetch chain that the isolated path already exercises. ## Optional-dep tolerance A graph node whose `pkg_id_with_patch_hash` is missing from `cas_paths_by_pkg_id`: - If `node.optional`: silently skipped, no directory created. Mirrors upstream's `if (depNode.optional) return` on fetch failure. - Otherwise: surfaces as `LinkHoistedModulesError::MissingCasPaths { pkg_id, dir }`. ## Tests 7 real-tempdir tests in `link_hoisted_modules/tests.rs`: - `import_pass_creates_package_directory` — single-package smoke. - `orphan_directory_is_removed` — `prev_graph` diff produces rimraf of stale directory; planted contents are gone after. - `nested_hierarchy_materializes_inner_node_modules` — version-conflict layout; loser ends up at `<outer>/node_modules/<inner>`. - `missing_cas_for_required_dep_errors` — required + missing CAS → typed `MissingCasPaths` error. - `missing_cas_for_optional_dep_skips_silently` — optional + missing CAS → no error, no directory. - `no_prev_graph_skips_orphan_pass` — fresh install (no prior lockfile) path. - `orphan_already_removed_is_tolerated` — phantom orphan in `prev_graph` not present on disk doesn't error. Each test plants synthetic CAS files in a tempdir and asserts the on-disk tree after the linker runs. * fix(package-manager): fail-fast on hierarchy/graph inconsistency (#505 review) The hierarchy walk silently skipped entries missing from `graph`, which would produce a partial install layout instead of surfacing the bug. Slice 4's walker keeps the two in sync today, but a future bug there shouldn't yield a partial tree. Add `LinkHoistedModulesError::MissingGraphNode { dir }` and return it when a hierarchy directory has no graph entry. Upstream effectively errors here too — its `graph[dir].fetching` read would throw `Cannot read properties of undefined` — pacquet just spells the failure out. Regression test `hierarchy_entry_missing_from_graph_errors` exercises the path with an empty graph and a hierarchy referencing a phantom dir. Caught by Coderabbit. |
||
|
Zoltan Kochan
|
3b9f5977c4 |
feat(lockfile): PkgIdWithPatchHash newtype, replace String in dep-graph nodes (#481) (#504)
* feat(lockfile): PkgIdWithPatchHash newtype, replace `String` in dep-graph nodes (#481) Port upstream's [`PkgIdWithPatchHash`](https://github.com/pnpm/pnpm/blob/94240bc046/core/types/src/misc.ts) branded string (`string & { __brand: 'PkgIdWithPatchHash' }`) as a non-validating newtype in `pacquet-lockfile`, matching `CLAUDE.md`'s rule 3 for non-validating brands: `From<String>` / `From<&str>` via `derive_more`, `#[serde(transparent)]` for wire-format identity with `String`, no validating constructor. Modeled on `pacquet_modules_yaml::DepPath` — the sibling brand from the same upstream `misc.ts` file under the same rules. Replaces the plain `String` field/parameter in the two pacquet consumers CodeRabbit flagged on #478: - `DependenciesGraphNode.pkg_id_with_patch_hash` in `package-manager/src/hoisted_dep_graph.rs`. - `create_full_pkg_id`'s first parameter and the `lockfile_to_dep_graph` local in `package-manager/src/virtual_store_layout.rs`. A workspace audit (`grep "pkg_id_with_patch_hash"`) turns up no other slots typed as plain `String`. Two new tests in `pkg_id_with_patch_hash.rs` pin the contract: a serde round-trip (`#[serde(transparent)]` keeps the on-disk shape a raw string) and the non-validating-construction property (empty string and `From<&str>`/`From<String>` both work). * fix(lockfile): replace intra-doc links to pacquet-modules-yaml with plain text CI \`Doc\` job runs with \`RUSTDOCFLAGS=-D warnings\`, so the two \`[\`pacquet_modules_yaml::DepPath\`]\` intra-doc links in \`pkg_id_with_patch_hash.rs\` failed \`rustdoc::broken-intra-doc-links\`: \`pacquet-lockfile\` doesn't depend on \`pacquet-modules-yaml\` (the dependency would go the wrong way), so the rustdoc resolver can't follow the link. Switched both references to bare \`pacquet_modules_yaml::DepPath\` prose, with a note that the bare reference is intentional. The docstring still tells a reader where to find the sibling brand; it just stops resolving as a clickable link, which costs nothing practical since the type name is searchable in any IDE. Adding \`pacquet-modules-yaml\` as a dev-dep purely for a doc link would invert the natural crate ordering (lockfile is upstream of modules-yaml in the build graph) — rejected as a cosmetic fix that introduces a real architectural smell. |
||
|
Zoltan Kochan
|
52364d4b5a |
feat(config,network,tarball,registry): per-registry TLS overrides (#502)
Closes #497.
## Summary
Adds per-registry TLS overrides keyed by nerf-darted `.npmrc` URI, the natural follow-up to #490's top-level TLS keys. Corporate environments running a private Verdaccio (or any registry with its own self-signed cert) can now pin scoped `:cafile=…` / `:cert=…` / `:key=…` per host without disabling strict-ssl globally.
Three commits, layered:
- **`feat(network)`** (eff1248e): adds `RegistryTls` + `PerRegistryTls` types in `pacquet-network` plus the lookup machinery — `pick_for_url` ports pnpm's [5-step `pickSettingByUrl`](https://github.com/pnpm/pnpm/blob/94240bc046/network/fetch/src/dispatcher.ts#L338-L375) exactly (exact > nerf-dart > no-port > shorter prefix > recursive no-port retry). `ThrottledClient::for_installs` gains a third `&PerRegistryTls` parameter and pre-builds one reqwest `Client` per non-empty override. New `acquire_for_url(url: &str)` routes per-request; `acquire()` keeps the default-client behavior for callers without a URL.
- **`feat(config)`** (4e69868a): `NpmrcAuth` parses the six per-registry TLS suffixes (`:ca`, `:cafile`, `:cert`, `:certfile`, `:key`, `:keyfile`) matching pnpm's `SSL_SUFFIX_RE` and applies onto `Config.tls_by_uri`. `*file` variants read from disk at parse time (silent on error); inline values get `\\n` → `\n` expansion. `:cert` and `:certfile` share the same `tls.cert` slot — last-write-wins inside one `.npmrc`.
- **`refactor(tarball,registry)`** (5f9cae93): three production call sites (registry metadata + version-tag fetches, plus two tarball download paths) move from `acquire()` to `acquire_for_url(url)` so the per-registry routing actually fires.
## Parity policy
Bug-for-bug with pnpm v11 ([SHA
|
||
|
Zoltan Kochan
|
13f401ab73 |
feat(package-manager): NODE_EXTRAS ignore filter for runtime archives (#437 slice D2) (#496)
Construct the per-archive ignore filter at the install dispatcher. For unscoped `node` the filter matches upstream's [`NODE_EXTRAS_IGNORE_PATTERN`](https://github.com/pnpm/pnpm/blob/94240bc046/engine/runtime/node-resolver/src/index.ts), which strips bundled `npm` / `corepack` from the Node.js runtime archive during the CAS write — pnpm (and pacquet) install pnpm itself as the package manager, so the bundled tooling is dead weight and would shadow the user's pnpm via `node_modules/.bin/`. Wiring matches upstream's [`archiveFilters: { node: NODE_EXTRAS_IGNORE_PATTERN }`](https://github.com/pnpm/pnpm/blob/94240bc046/installing/client/src/index.ts): the per-package-name table is keyed by `pkg.name`, so `@foo/node` and other packages keep `None` and the full archive lands unfiltered. Pacquet uses a hand-coded matcher rather than the upstream regex so `pacquet-tarball` doesn't have to pull in a regex engine. The three branches mirror the regex alternation exactly: 1. `^(?:lib/)?node_modules/(?:npm|corepack)(?:/|$)` 2. `^bin/(?:npm|npx|corepack)$` 3. `^(?:npm|npx|corepack)(?:\.(?:cmd|ps1))?$` A `OnceLock` caches the `Arc<IgnoreEntryFilter>` so per-snapshot clones share one trait object. Unit tests pin every branch of the alternation plus the negative cases (e.g. `lib/node_modules/yarn/...` is *not* matched, `bin/npm.cmd` is *not* matched — the regex's `$` and arm-specific extension rules are deliberately asymmetric). Verified by temporarily collapsing the `bin/` branch to a no-op — the test fails as expected. Bin-link cmd-shims for the runtime executables and `@runtime:` substring handling in skip lists / reporter prefixes are Slice D3. End-to-end runtime install fixtures land in Slice F. |
||
|
Zoltan Kochan
|
9c67bc82c7 |
feat(package-manager): prev_graph diff from current lockfile (#438 slice 4d) (#494)
* feat(package-manager): prev_graph diff from current lockfile (#438 slice 4d) `lockfile_to_hoisted_dep_graph` now takes an optional `current_lockfile: Option<&Lockfile>` and populates `result.prev_graph` from a second walk over that lockfile. Ports upstream's wrapper at installing/deps-restorer/src/lockfileToHoistedDepGraph.ts. When the current lockfile is `Some(lf)` with a non-empty `packages:` map, the second walk runs with `force: true` and `skipped: BTreeSet::new()`. Force matters: an orphan that landed under the previous install but would now fail installability (e.g., the host changed platforms) still surfaces in `prev_graph` so Slice 5's linker can find and rimraf the stale directory. Empty skipped matters: the previous install's own filter set is unrelated to "which directories still exist on disk." API change - The public `lockfile_to_hoisted_dep_graph(lockfile, opts)` signature gains a middle `current_lockfile: Option<&Lockfile>` argument. Single existing caller (the tests) updated to pass `None`. Splits the previous body into a private `build_dep_graph` helper that the wrapper calls once or twice depending on whether a current lockfile is present. prev_graph shape - `None` when no current lockfile is supplied, or when the supplied lockfile has no `packages:` map (a brand-new install in progress). Mirrors upstream's `prevGraph = {}` fallback — pacquet uses `None` rather than an empty map, but the linker treats both the same. - `Some(graph)` otherwise, keyed by directory just like the wanted-lockfile graph. Tests - `prev_graph_none_when_current_lockfile_absent` — no current lockfile → `prev_graph` is `None`, wanted graph still produced. - `prev_graph_none_when_current_lockfile_has_no_packages` — current lockfile with `packages: None` → `prev_graph` is `None`. - `prev_graph_contains_orphan_from_current_only_lockfile` — package in current but not wanted appears in `prev_graph`, not in `graph`. - `prev_graph_includes_orphan_even_when_now_incompatible` — darwin-only orphan in the current lockfile, host is linux, wanted lockfile is empty: `prev_graph` still contains it (proves `force: true` overrides the installability check), and the wanted-walk's `skipped` stays empty (proves the prev-walk's skipped set is independent). All 4 new tests pass alongside the 15 walker tests from 4a-4c. * fix(package-manager): collapse empty current packages to None for prev_graph (#494 review) `current.packages.is_some()` matched `Some(empty_map)` too, causing 4d to do an unnecessary second walk and return `Some(empty_graph)` for a case the doc-comment described as `None`. Tighten the guard to require a non-empty `packages` map. Pacquet uses `Option<DependenciesGraph>` for `prev_graph` (upstream uses an always-present `DependenciesGraph` with `{}` for the no-current case). The no-packages and empty-packages cases both produce the same observable behavior — "no orphans to consider" — so they should share one representation rather than be inconsistent. The doc-comment's stated contract was `None`; the code now matches. Added `prev_graph_none_when_current_lockfile_has_empty_packages` to pin the empty-map case. Caught by Coderabbit on #494. * style(package-manager): rustfmt the 4d follow-up |
||
|
Zoltan Kochan
|
3fe272f050 |
feat: filter the wanted lockfile when writing <virtual_store_dir>/lock.yaml (#434 slice 6) (#495)
Pacquet wrote the raw wanted lockfile as the current lockfile. Upstream pnpm writes a filtered version with optional + skipped subtrees pruned and `include` flags applied, so the next install diffs against what was actually materialized rather than the resolver's full ambition. Without the prune, dropped snapshots (slice 1 installability, slice 4 fetch failures, slice 5 `--no-optional`) were claimed present in the current lockfile and the follow-up install would skip work that should have run. Ports <https://github.com/pnpm/pnpm/blob/94240bc046/lockfile/filtering/src/filterLockfileByImportersAndEngine.ts#L46-L94> → <https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-restorer/src/index.ts#L687-L695>. Pacquet-specific simplification: instead of re-running the engine + supportedArchitectures + skipped checks at filter time, `filter_lockfile_for_current` reuses the `SkippedSnapshots` set already produced by the install pipeline. Its three subsets (`installability`, `fetch_failed`, `optional_excluded`) are the exact set upstream's filter would drop — same observable result, no duplicated walk. ## Changes - **`pacquet-lockfile`**: `Clone` derives on `Lockfile`, `LockfileSettings`, `ProjectSnapshot`, `SnapshotEntry`, `PackageMetadata`, `PeerDependencyMeta`, `ResolvedDependencySpec`. Needed to build a filtered lockfile by clone-and-mutate. - **`pacquet-package-manager/src/current_lockfile.rs`** (new): `filter_lockfile_for_current(lockfile, included, skipped) -> Lockfile`. Three-phase filter: 1. BFS the snapshot graph from filtered importer roots, skipping keys in `skipped`. Produces the reachable set. 2. Per-importer: clear dep maps whose `include` flag is false; trim `optional_dependencies` to entries whose target survived. 3. `snapshots:` / `packages:` filtered to the reachable set (packages key off `without_peer()` so peer-variant survivors keep their shared metadata row). - **`install.rs`**: replace the raw `save_current_to_virtual_store_dir` call with `filter_lockfile_for_current(...).save_current_to_virtual_store_dir(...)`. ## Tests 8 unit tests covering each filter behavior: - `skipped_snapshot_pruned_from_snapshots_and_importer_optional` - `include_optional_false_clears_importer_section` - `transitive_under_skipped_snapshot_is_pruned` - `snapshot_reachable_via_kept_path_survives` (mirrors upstream's `:712` shared-dep case at the filter level) - `packages_filtered_to_surviving_metadata_keys` (peer-variant metadata sharing) - `link_optional_entries_survive_post_filter` (workspace link: deps don't get post-filtered) - `empty_skipped_and_full_include_is_identity_for_reachables` (baseline) - `orphan_snapshots_are_pruned` (snapshots unreachable from any importer get dropped) Test-the-test verified by breaking the BFS walker — two tests fail. ## Out of scope - Hoisted-linker current-lockfile variant (`:633`) — pacquet's hoisted node-linker isn't fully wired through the lockfile-write path yet; tracked separately under #438. - `pnpm install --filter` slicing — pacquet has no `--filter` yet. Closes #493. |
||
|
Zoltan Kochan
|
d2cdb8de65 |
test(git-fetcher): end-to-end shallow-fetch argv assertion via PATH-shim (#436 §E follow-up) (#487)
Ports the last open §E test — upstream's [`fetching/git-fetcher/test/index.ts:183`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/git-fetcher/test/index.ts#L183) `still able to shallow fetch for allowed hosts`, which jest-mocks `execa` to spy on the `git` argv. Pacquet can't intercept `Command::new("git")` without touching production code, so the test uses a `/bin/sh` shim on `PATH`: 1. Writes `<tempdir>/shim/git`: a tiny shell script that tab-logs each invocation to a file and fakes `rev-parse HEAD` so the fetcher's commit-match check passes. 2. Prepends the shim dir to `PATH` for the test body via `unsafe { std::env::set_var(...) }`. Edition 2024 requires `unsafe`; the project's `cargo nextest` runner isolates each test in its own process so no sibling can race the modified env. PATH is restored before assertions. 3. Parses the log and asserts the shallow sequence: `git init` → `git remote add origin <url>` → `git fetch --depth 1 origin <commit>`, with `git clone` absent. `fetcher_clones_when_host_not_in_shallow_list` is the mirror — same shim, empty `git_shallow_hosts`, asserts `git clone <url> <dir>` appears and `init` / `fetch` don't. Together the two tests pin the gate's polarity end-to-end. The existing `should_use_shallow_matches_known_host` unit test only covers the predicate cross-platform; the new tests add Unix end-to-end argv coverage. Unix-only via `#[cfg(unix)]`. A Windows mirror would need a `.cmd` shim and a different launcher; the predicate-level test still covers Windows. |
||
|
Zoltan Kochan
|
6d9d0ad08c |
feat(config,network,cli): TLS keys + local-address from .npmrc (#490)
Closes #482. ## Summary Ports pnpm v11's TLS + `local-address` `.npmrc` keys onto pacquet (SHA [`94240bc046`](https://github.com/pnpm/pnpm/blob/94240bc046/network/fetch/src/dispatcher.ts)), the natural pair to the proxy support that landed in #476. Three layers: - **`feat(network)`** (e9ed56c9): adds `TlsConfig` next to `ProxyConfig` in `pacquet-network`, with `ca: Vec<String>`, `cert`/`key`, `strict_ssl: Option<bool>`, and `local_address: Option<IpAddr>`. `ThrottledClient::for_installs` gains a `&TlsConfig` parameter; the unified error surface is the new `ForInstallsError` enum carrying either `ProxyError` or `TlsError`. CAs route through `Certificate::from_pem`, client identities through `Identity::from_pkcs8_pem` (the only PKCS path reqwest exposes on the native-tls backend pacquet builds with). `strict_ssl` defaults to `true` at build site, matching pnpm's per-emit-site `strictSsl ?? true` rather than a config-layer default. - **`feat(config)`** (e8dcd87e): extends `NpmrcAuth` with the six new keys (`ca`, `cafile`, `cert`, `key`, `strict-ssl`, `local-address`) and adds `apply_tls_and_local_address` to populate `Config.tls`. `cafile` reads from disk and splits on `-----END CERTIFICATE-----` to mirror pnpm's [`loadCAFile`](https://github.com/pnpm/pnpm/blob/94240bc046/config/reader/src/loadNpmrcFiles.ts#L249-L265). Unreadable `cafile` is silently treated as unset. Invalid `strict-ssl` and `local-address` values drop silently. - **`feat(cli)`** (1759f9d2): one-line swap in `State::init` to pass `&config.tls` instead of `TlsConfig::default()`. Folds in two CI-parity fixups: the self-signed test cert moves to a shared `.pem` fixture under `crates/network/tests/fixtures/` (typos linter false positives on base64 DER), and four reqwest intra-doc links become plain backticks. ## Parity policy Faithful to pnpm — see the [research brief](https://github.com/pnpm/pacquet/issues/482) on the issue. Highlights: - **No new error codes.** pnpm doesn't define `ERR_PNPM_INVALID_CA` etc.; invalid PEMs surface as raw `tls.connect` errors at request time upstream. Pacquet validates eagerly via `Certificate::from_pem` / `Identity::from_pkcs8_pem` (pushing the failure to per-request time would silently degrade every install behind a broken `ca`) but deliberately omits a `code(...)` attribute on `TlsError` so reviewers can see at a glance it's a pacquet-only diagnostic, not a pnpm error code. - **Silent `cafile`-not-found.** Matches pnpm's `catch {}` swallow in `loadCAFile`. - **No env-var fallback.** pnpm reads only `.npmrc`; Node's implicit `NODE_EXTRA_CA_CERTS` / `NODE_TLS_REJECT_UNAUTHORIZED` honoring doesn't apply to pacquet's reqwest stack. - **`strict-ssl: false` disables both chain-of-trust and hostname verification**, matching Node's `rejectUnauthorized=false` short-circuit (pacquet uses reqwest's `danger_accept_invalid_certs(true)` which has the same combined semantics). ## Reviewer flags - **PKCS#8-only client keys.** Reqwest's native-tls backend exposes only `Identity::from_pkcs8_pem`; legacy PKCS#1 keys (`-----BEGIN RSA PRIVATE KEY-----`) and PKCS#12 bundles are not supported by this constructor. Documented at the `apply_tls` callsite with the `openssl pkcs8 -topk8 -nocrypt` conversion command. Switching to rustls-tls would broaden the supported formats but is out of scope here. - **Per-registry TLS overrides** (`//host:cafile=`, `//host:ca=`, `//host:cert=`, `//host:key=`) are **not** included. Same shape as the existing scoped-auth handling but a sizeable feature on its own; flagged in #482 as a follow-up. |
||
|
Zoltan Kochan
|
a55957d0c0 |
feat(package-manager): dispatch Binary/Variations to runtime fetchers (#437 slice D1) (#492)
Replace the cold-path `UnsupportedResolution` arms for
`LockfileResolution::{Binary, Variations}` with actual fetcher
dispatch. Slice C wired the per-archive download
(`DownloadTarballToStore` / `DownloadZipArchiveToStore`); this
slice routes lockfile resolutions through them.
- `Binary(b)` dispatches on `b.archive` (`Tarball` / `Zip`):
- `Tarball` → `DownloadTarballToStore` (same shape as
registry / tarball entries minus the `package_unpacked_size`
hint).
- `Zip` → `DownloadZipArchiveToStore` with
`archive_prefix: b.prefix.as_deref()` so the runtime
archive's top-level wrapper (e.g. `node-vX.Y.Z-darwin-arm64/`)
is stripped before the CAS keys are written.
- `Variations(v)` runs `select_platform_variant` against a
freshly-built `PlatformSelector` (`pacquet_graph_hasher`'s
`host_platform` / `host_arch` / `host_libc`, mapping
`"unknown"` → `None` per upstream's
`process.platform === 'linux' ? family : null` convention).
The picked variant's inner resolution must be `Binary`; any
other shape (corrupt lockfile or a future shape pacquet hasn't
learned about) raises the typed
`VariantHasNonBinaryResolution` error rather than silently
routing.
- `create_virtual_store::snapshot_cache_key` returns the proper
warm-cache key for both new arms: `Binary` uses
`store_index_key(integrity, pkg_id)`; `Variations` runs the
same selector and keys off the picked variant. A miss on
variant selection or a non-`Binary` inner shape returns
`Ok(None)` and lets the cold path raise the typed error.
- Two new error variants on `InstallPackageBySnapshotError`:
- `NoMatchingPlatformVariant { package_key, host_os, host_cpu,
host_libc, available_targets }` — fires when no variant
matches the host; carries the host triple + the rendered
available target list so the user can see at a glance why.
- `VariantHasNonBinaryResolution { package_key, inner_kind }`
— defensive guard for malformed lockfiles.
The Node-runtime `NODE_EXTRAS_IGNORE_PATTERN` filter (strips
bundled `npm` / `corepack` from the archive) and bin-link
cmd-shims for runtime executables will land in Slice D2; the
filter slot stays `None` for now and `BinaryResolution::bin` is
read but not yet acted on. Slice E adds `--no-runtime` and
Slice F adds the end-to-end install fixtures.
Two new unit tests cover the helpers:
- `host_platform_selector_omits_libc_on_non_linux_hosts` —
pins the `host_libc() == "unknown"` ⇔ `selector.libc.is_none()`
relationship without needing platform-gated tests.
- `render_variant_targets_formats_each_triple_with_optional_libc`
— locks in the `os/cpu[+libc]` rendering used by the
`NoMatchingPlatformVariant` error message.
|
||
|
Zoltan Kochan
|
11f9a27aa2 |
fix(config): default enableGlobalVirtualStore to false (#489)
Pacquet's `default_enable_global_virtual_store()` returned `true` and cited upstream's [`config/reader/src/index.ts:392-394`](https://github.com/pnpm/pnpm/blob/94240bc046/config/reader/src/index.ts#L392-L394) as the authority. But that range lives entirely inside the [`if (cliOptions['global']) { ... }`](https://github.com/pnpm/pnpm/blob/94240bc046/config/reader/src/index.ts#L348-L395) block — surrounded by `CONFIG_CONFLICT_*_WITH_GLOBAL` errors and closed by `} else if (!pnpmConfig.bin) { ... }`. So in pnpm v11: - `pnpm install --global foo` → `enableGlobalVirtualStore` defaults to `true` - `pnpm install` (regular) → stays `null`/unset → effectively `false` Pacquet doesn't have a `--global` CLI flag (only `install --frozen-lockfile`), so the applicable upstream default is the `false` one. Pre-fix, every pacquet install silently wrote slots to `<store>/links/...` and registered projects, even when the user never asked for GVS — and a project alternating between `pnpm install` and `pacquet install --frozen-lockfile` would see two different on-disk layouts. The default introduced by #444 cited the same `L392-L394` range but read it as an unconditional default. Corrected here. ## Test changes - `gvs_default_writes_links_into_global_virtual_store_dir` renamed to `gvs_default_is_off_and_paths_derive_cleanly` and inverted — now asserts the default-off behaviour. The path derivation still populates both `virtual_store_dir` and `global_virtual_store_dir` cleanly so downstream code can read either field without first checking the toggle. - `gvs_user_pinned_virtual_store_routes_into_global_virtual_store_dir` and `yaml_global_virtual_store_dir_wins_over_derivation`: yaml now opts into GVS explicitly (`enableGlobalVirtualStore: true`) so the GVS-on derivation path is still exercised. - `install/tests.rs` doc + inline comments updated to reflect that GVS-on tests need to pin the flag explicitly now. ## Benchmark The bench fixture's explicit `enableGlobalVirtualStore: false` pin is kept (it's now redundant but future-proofs the bench against a default flip), with an updated comment explaining the design intent. Same for `MinimalWorkspaceManifest.enable_global_virtual_store` doc. |
||
|
Zoltan Kochan
|
ff8f458bc8 |
feat(package-manager): hoisted dep-graph installability check (#438 slice 4c) (#491)
Sub-slice 4c of #438. Wires `package_is_installable` into the hoisted-graph walker shipped in 4b (#486), so optional packages on unsupported platforms get filtered into `result.skipped` and engine-strict mismatches surface as typed errors. Single-importer only; store I/O and `prev_graph` diff still to come (4d, then 5). ## Behavior Mirrors upstream's `if (!opts.force && packageIsInstallable(...) === false)` gate at [lockfileToHoistedDepGraph.ts:200-211](https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-restorer/src/lockfileToHoistedDepGraph.ts#L200-L211): | Upstream return | `package_is_installable` (Rust) | Walker action | |---|---|---| | `null` (no constraint) | `Ok(Installable)` | emit node | | `true` (warn but proceed) | `Ok(ProceedWithWarning)` | emit node (warning emit deferred) | | `false` (optional + incompatible) | `Ok(SkipOptional)` | add to `result.skipped`, skip node | | throws `UnsupportedEngineError` (strict) | `Err(InstallabilityError::Engine)` | `HoistedDepGraphError::Installability` | | throws `InvalidNodeVersionError` | `Err(InstallabilityError::InvalidNodeVersion)` | `HoistedDepGraphError::Installability` | ## New options on `LockfileToHoistedDepGraphOptions` - `force: bool` — bypass the check entirely. Used by Slice 4d's `prev_graph` walk, where the previous lockfile is replayed wholesale so the orphan diff catches packages that would now filter out. Mirrors upstream's `force` at [lockfileToHoistedDepGraph.ts:73](https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-restorer/src/lockfileToHoistedDepGraph.ts#L73-L76). - `engine_strict`, `current_node_version`, `current_os`, `current_cpu`, `current_libc`, `supported_architectures` — host-derived axes the check consumes. ## New output on `LockfileToDepGraphResult` - `skipped: BTreeSet<String>` — the input `opts.skipped` cloned and extended with depPaths added during the walk. Upstream mutates the input set in place; pacquet returns the augmented set so the caller can persist it into `.modules.yaml.skipped` without sharing mutable state. ## Tests 15 walker tests total — the 10 from 4b survive (one extended to assert the input depPath survives into output `skipped`), plus five new ones: - `walker_skips_optional_dep_on_unsupported_platform` — Linux host, package targets darwin only, `optional: true` → added to `result.skipped`, no graph entry, no `hoisted_locations`. - `walker_emits_required_dep_with_unsupported_platform_as_warning` — same shape but `optional: false` → walker proceeds (matches upstream `packageIsInstallable === true`); warning log emit is out of scope for 4c. - `walker_errors_on_engine_strict_mismatch` — `engine_strict: true` + `engines.node = ">=99.0.0"` → `HoistedDepGraphError::Installability`. - `walker_force_bypasses_installability_check` — `force: true` emits an incompatible required dep without erroring. - `walker_emits_compatible_dep` — sanity: compatible host + no constraint mismatch → graph entry, no skip. |
||
|
Zoltan Kochan
|
20455ebe9f |
feat: --no-optional filter for frozen lockfile install (#434 slice 5) (#485)
Slice 5 of the [#434 umbrella](https://github.com/pnpm/pacquet/issues/434) (`Proper optionalDependencies support`). Slices 1–4 (#439, #456, #467, #474) handle installability + the fetch-failure swallow. This slice closes the user-facing `--no-optional` flag: the CLI flag already exists and threads through `Install::dependency_groups`, but only `SymlinkDirectDependencies` and the on-disk `included` field actually honored it. The rest of the install pipeline walked `optional_dependencies` unconditionally, so the optional subtree was still extracted, linked, and built. Mirrors upstream's depNode filter at [`installing/deps-installer/src/install/link.ts:109-111`](https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-installer/src/install/link.ts#L109-L111): ```ts if (!opts.include.optionalDependencies) { depNodes = depNodes.filter(({ optional }) => !optional) } ``` ## Changes - **`SkippedSnapshots`**: gains a third disjoint subset `optional_excluded` alongside `installability` (slice 1) and `fetch_failed` (slice 4). The existing `contains` / `iter` already return the union, so every downstream consumer that filters by skip-set (`CreateVirtualStore`, `SymlinkDirectDependencies`, `BuildModules`, hoist, `link_bins`) now respects `--no-optional` through the same gate. `iter_installability` still returns only the persistent subset for `.modules.yaml.skipped` writes — `--no-optional` exclusions are transient (matching upstream's behavior of never updating `opts.skipped` at the filter site). - **`InstallFrozenLockfile::run`**: iterates the lockfile snapshots once and inserts every `snap.optional == true` entry into the new subset when the dispatch list doesn't contain `DependencyGroup::Optional`. The `SnapshotEntry::optional` flag is set by the resolver only when the snapshot is reachable **exclusively** through optional edges, so a snapshot reachable through any non-optional edge carries `optional: false` and survives the filter — covers [`optionalDependencies.ts:712`](https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-installer/test/install/optionalDependencies.ts#L712) (`dependency that is both optional and non-optional is installed`). |
||
|
Zoltan Kochan
|
dc834bdb8e |
feat(tarball): binary fetcher for zip archives (#437 slice C) (#472)
Slice C of [#437](https://github.com/pnpm/pacquet/issues/437) — the zip half of pnpm's binary fetcher. Slices A (lockfile types) and B (variant picker) have landed; this slice adds the per-archive download and CAS-extraction pipeline. The install-dispatch site (slice D) and the `--no-runtime` flag (slice E) will follow. Mirrors upstream's [`downloadAndUnpackZip` / `extractZipToTarget`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/binary-fetcher/src/index.ts): - `extract_zip_entries` walks every entry. Path validation runs *before* the `is_dir()` early-skip so directory entries like `../evil/` still surface `PATH_TRAVERSAL`. The canonical `cas_paths` / `pkg_files_idx` key is built from `enclosed_name()` rebuilt as `Normal` components joined with `/` — `.` segments collapse, separators are cross-platform consistent, and the ignore filter sees the same string the map is keyed by. - `DownloadZipArchiveToStore` is the public sibling of `DownloadTarballToStore` — same store-index lookup, prefetch cache reuse, store-index writer queue. The retry policy, network-permit shape, and post-download semaphore gating exactly match the tarball path. - `ignore_file_pattern` is `Option<Arc<IgnoreEntryFilter>>` on both `DownloadTarballToStore` and `DownloadZipArchiveToStore`, so the Slice D dispatcher can construct a filter per fetch from runtime config without pinning to `'static`. Cloned per retry attempt; the inner trait object is shared. - `run_with_mem_cache` doc-blocks the "stable filter per URL" invariant: the cache keys solely on `package_url` (matching pnpm's `tarballCache`), and callers keep the (URL, filter) relation functional. Upstream's `archiveFilters` is keyed by `pkg.name`, and tarball URLs encode `(name, version, integrity)`, so this naturally holds. - New `TarballError::ReadZipEntries(std::io::Error)` for zip per-entry I/O failures (`try_reserve` / `read_to_end`); maps to `ERR_PACQUET_ZIP` in the retry classifier, distinct from the tar-specific `ERR_PACQUET_TARBALL_TAR`. Zip `unix_mode()` is masked to `0o777` before write so `CafsFileInfo.mode` stays permission-only (matches `store_dir::add_files_from_dir::file_mode_from`). `TarballError::TarballTooLarge` display generalized from "Tarball at …" to "Archive at …" so the zip pipeline's OOM path doesn't surface a misleading message. - Adds `zip = "5", default-features = false, features = ["deflate"]` to the workspace; reused in `pacquet-tarball`. |
||
|
Zoltan Kochan
|
4cf66757d6 |
feat(package-manager): hoisted dep-graph walker (#438 slice 4b) (#486)
Sub-slice 4b of #438. Implements `lockfile_to_hoisted_dep_graph(lockfile, opts) -> LockfileToDepGraphResult` — the walker that consumes the Slice 3 hoister output and produces the directory-keyed graph + hierarchy + hoisted_locations + direct-deps-by-importer-id + injection-targets-by-dep-path that 4a (#478) pinned the type shape of. Single-importer only (multi-importer lockfiles surface as `HoistError::UnsupportedWorkspace` via the underlying hoister). ## What's deferred Three things are intentionally left to follow-ups so 4b stays focused: - **Store fetch wiring** — `fetching` / `files_index_file` on the graph node are still defaulted. 4c will wire `StoreController::fetchPackage` (and the skip-fetch optimization that consults `currentHoistedLocations`). - **Installability check** — the walker honors `opts.skipped` as input but doesn't run `packageIsInstallable` to add new entries. 4c will plumb `package_is_installable` from `pacquet-package-is-installable` (already used by the isolated linker path). - **`prev_graph` diff** — `prev_graph` is `None`. 4d will walk the *current* lockfile alongside the wanted one (using `force: true` + empty `skipped` per upstream) and produce the orphan-removal input Slice 5's linker consumes. ## Two-pass design Upstream's `fetchDeps` walks asynchronously via `await Promise.all(deps.map(async (dep) => { ... }))`. The key invariant: each sibling's async function runs its sync prologue (insert + push to `pkgLocationsByDepPath`) before any continuation fires, because `Promise.all` sync-invokes each function and the inner `await fetchDeps(...)` only yields after the prologue completes. So by the time any node's post-recursion `graph[dir].children = getChildren(...)` line runs, every node in the entire tree is already in `pkgLocationsByDepPath`. Pacquet runs synchronously. To preserve that invariant cleanly: 1. **Pass 1 (recursive walk).** Insert each node into the graph with `children: BTreeMap::new()` and push its dir to `pkg_locations_by_dep_path`. Recursion order matches upstream's outer body (insert → push → recurse → push to `hoistedLocations`). 2. **Pass 2 (`fill_children`).** Iterate every graph node and resolve `children: alias → dir` from the snapshot's `dependencies` + `optionalDependencies` via `SnapshotDepRef::resolve`, consulting the now-complete `pkg_locations_by_dep_path`. A single-pass walker that computed children inline produced incorrect `children` maps for hoisted transitive deps — the canonical case `root → a → b` with `b` flattened to root left `a.children["b"]` empty because `b`'s pkg_locations entry hadn't been recorded yet. Caught by `walker_transitive_dep_flattens_under_root` before the refactor. ## Tests - `walker_empty_lockfile_produces_empty_result` — empty importer → empty graph, with the `"."` root importer key still present. - `walker_single_root_dep_emits_one_node` — `root → a`, node at `<lockfile_dir>/node_modules/a`. - `walker_transitive_dep_flattens_under_root` — `root → a → b`: `b` hoists to root, `a.children["b"]` points at the root-level dir. - `walker_version_conflict_keeps_loser_nested` — `root → {a@1, c}` with `c → a@2`: `a@1` at root, `a@2` nested under `c`; `hoisted_locations` records both directories; `c.children["a"]` points at the nested copy. - `walker_honors_pre_skipped_dep_path` — depPaths in `opts.skipped` are dropped from the walk entirely (and not recorded in `hoisted_locations`). - `walker_records_directory_resolution_as_injection_target` — `directory:` resolutions populate `injection_targets_by_dep_path` for the post-install re-mirror pass. |
||
|
Zoltan Kochan
|
2c513ad89f |
feat(store-dir): pacquet store prune for the global virtual store (#458) (#475)
Closes #458. Ports upstream's [`pruneGlobalVirtualStore`](https://github.com/pnpm/pnpm/blob/94240bc046/store/controller/src/storeController/pruneGlobalVirtualStore.ts) and the read half of [`projectRegistry`](https://github.com/pnpm/pnpm/blob/94240bc046/store/controller/src/storeController/projectRegistry.ts#L37-L100) (pacquet shipped only the write half in #432 / #444). The CLI wiring already existed — `StoreCommand::Prune` → `StoreDir::prune()` — but `StoreDir::prune` was a `todo!()`. With this PR `pacquet store prune` actually runs. ## What it does **Mark phase**: walk every registered project (`<store>/projects/<short-hash>` → project root). For each project, find every `node_modules/`, follow symlinks that land under `<store>/links/...`, record reachable `<scope>/<name>/<version>/<hash>` slots, and recurse into the slot's own `node_modules/` for transitive deps. **Sweep phase**: walk `<store>/links/<scope>/<name>/<version>/<hash>` and remove every `<hash>` not in the reachable set. Empty `<version>/`, `<name>/`, and `<scope>/` parents get cleaned up. **Self-healing registry**: `get_registered_projects` unlinks entries whose project directory has been deleted (`ENOENT` on stat). Permission / unrelated errors surface as `PROJECT_INACCESSIBLE` / `PROJECT_REGISTRY_ENTRY_INACCESSIBLE` — matches upstream's strict stance, since silently dropping an inaccessible registry entry could remove slots the project still references. |
||
|
Zoltan Kochan
|
006f9fde41 |
perf(git-fetcher): skip CAS re-import when packlist matches input (#436 follow-up) (#479)
* perf(git-fetcher): skip CAS re-import when packlist matches input (#436 follow-up) Port upstream's [`gitHostedTarballFetcher.ts:88-100`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/tarball-fetcher/src/gitHostedTarballFetcher.ts#L88-L100) fast path: when `resolution.path` is `None` and `packlist.len() == cas_paths.len()`, the materialized tree is byte-identical to the input CAS files and re-importing every entry through hash-dedup is wasted work. Two branches mirror upstream: - `!should_be_built`: queue a `PackageFilesIndex` row at the final key with `requires_build: Some(false)`, then hand the input `cas_paths` straight to the dispatcher. Upstream copies the `\traw` row to the prepared key; pacquet's tarball download doesn't write a raw row at this key today, so the row is synthesized from `cas_paths` instead — the CAS files themselves are already in place. - `should_be_built && ignore_scripts`: return input as-is *without* queueing a row, so subsequent installs re-check the build gate. The synthesized row's per-file digest comes from the CAS path itself (pnpm v11's `files/XX/<rest>[-exec]` layout makes this exact); size comes from `fs::metadata`, mode from the `-exec` suffix. No per-file `fs::read + sha512` — the whole point of the optimization. * fix(git-fetcher): tighten cas_path_digest to full sha512 shape; pin sub-path slow-path CodeRabbit and Copilot review on #479: - `cas_path_digest` now requires the stem to be exactly 126 hex chars (the remainder of a sha512 hex digest after the 2-char shard). Previously accepted any non-empty hex stem, so paths like `.../files/ab/cd` would synthesize a 4-char "digest" that could poison `index.db`. Fail closed on anything that isn't a real v11 CAS path. - Replace the bogus "empty stem" malformed-path case (which never triggered — `Path::file_name()` of `/tmp/ab/` is `Some("ab")`) with cases that actually exercise the new length check: too short, too long, and right-length-with-a-non-hex-byte. - Rebuild `sub_path_never_takes_fast_path` so `packlist.len() == cas_paths.len()` (single sub-package tarball). Only the `path.is_none()` guard now keeps the fast path out — if a future refactor drops it, the test catches the misshaped `packages/sub/...` keys leaking into the dispatcher's output. |
||
|
Zoltan Kochan
|
9b21a726ab |
feat(package-manager): hoisted dep-graph type skeleton (#438 slice 4a) (#478)
* feat(package-manager): hoisted dep-graph type skeleton (#438 slice 4a) Defines the directory-keyed dependency graph types used by the hoisted-linker install path. Types only — the walker that fills them from a lockfile lands in a follow-up. Ported from upstream: - `DependenciesGraphNode` mirrors deps/graph-builder `DependenciesGraphNode` minus the `fetching` / `files_index_file` fields that only the store-controller-bound walker can populate. - `DependenciesGraph` = `BTreeMap<PathBuf, DependenciesGraphNode>`, keyed by absolute directory path (unlike pacquet's existing depPath-keyed `deps_graph` module — hoisted nodes can occupy several directories when a name conflict forces nesting). - `DepHierarchy` is the recursive directory tree the linker walks to decide population order and which `<dir>/node_modules/.bin` to wire up. Newtype-wrapped because Rust doesn't allow recursive type aliases. - `DirectDependenciesByImporterId` is the per-importer alias-to-directory table the linker hands to the bin pass. - `LockfileToDepGraphResult` bundles everything the walker returns to the install pipeline, including `prevGraph` for the orphan-diff pass and `injectionTargetsByDepPath` for the injected-workspace re-mirror step. - `LockfileToHoistedDepGraphOptions` carries the subset of upstream's options that the to-be-ported walker actually reads today; fields tied to the store controller, fetch concurrency, and workspace project list will be added when their consumers land. Smoke tests cover empty-result default construction, node insertion by `dir`, recursive hierarchy nesting, and default-options shape. Upstream: - <https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-restorer/src/lockfileToHoistedDepGraph.ts> - <https://github.com/pnpm/pnpm/blob/94240bc046/deps/graph-builder/src/lockfileToDepGraph.ts> * docs(package-manager): v9 depPath in 4a tests + map-key typing notes (#478 review) - Replace v5-era `/accepts/1.3.7` depPath strings in the smoke test with v9-shape `accepts@1.3.7` (the format pacquet's `PkgNameVerPeer` produces and the lockfile snapshots use). The legacy `/name/version` shape is only kept for read-side `hoistedAliases` compatibility. Caught by Copilot. - Tighten doc-comments on `hoisted_locations`, `injection_targets_by_dep_path`, and `skipped` to spell out *why* the keys are plain `String` (not `DepPath`): upstream types those fields with raw `string` in the hoisted-specific interface, even though the values are populated from depPaths. Mirrors upstream's literal typing. |
||
|
Zoltan Kochan
|
b33a481b17 |
feat: proxy support (.npmrc + env-var cascade, HTTP/HTTPS/SOCKS) (#476)
* feat(network): add ThrottledClient::for_installs with proxy support Ports pnpm v11's `getDispatcher` (`network/fetch/src/dispatcher.ts` at SHA |
||
|
Zoltan Kochan
|
279469632c |
feat: fetch-failure swallow for optional snapshots + ResolutionFailure payload (#434 slice 4) (#474)
Slice 4 of the [#434 umbrella](https://github.com/pnpm/pacquet/issues/434) (`Proper optionalDependencies support`). Slices 1–3 (#439, #456, #467) close the installability-driven skip path; this slice adds the second skip pathway upstream handles in the frozen install: an `optional: true` snapshot whose **fetch / extract** step fails at install time must not abort the install. Local-materialization errors (`CreateVirtualDir`) and config-shape errors still abort even for optional snapshots — matching upstream's `linkPkg` path which sits outside the catch. Mirrors the silent catch sites at [`deps/graph-builder/src/lockfileToDepGraph.ts:294-298`](https://github.com/pnpm/pnpm/blob/94240bc046/deps/graph-builder/src/lockfileToDepGraph.ts#L294-L298) and [`installing/deps-restorer/src/index.ts:912-921`](https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-restorer/src/index.ts#L912-L921): ```ts try { ... fetch ... } catch (err) { if (pkgSnapshot.optional) return; throw err } ``` ## Changes - **Reporter**: `SkippedOptionalPackage` becomes a `#[serde(untagged)]` enum with `Installed { id, name, version }` and `ResolutionFailure { name?, version?, bareSpecifier }` variants. Models upstream's discriminated union at [`core-loggers/src/skippedOptionalDependencyLogger.ts:10-31`](https://github.com/pnpm/pnpm/blob/94240bc046/core/core-loggers/src/skippedOptionalDependencyLogger.ts#L10-L31). The new variant is wire-shape-only in slice 4 — pacquet has no resolver yet, but a future resolver port at the upstream emit site ([`resolveDependencies.ts:1376-1383`](https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-resolver/src/resolveDependencies.ts#L1376-L1383)) lands without re-touching this type. - **`SkippedSnapshots`**: gains a `fetch_failed` subset disjoint from the existing `installability` subset. `contains` / `iter` return the union (downstream consumers treat both as absent); `iter_installability` returns only the persistent subset that `.modules.yaml.skipped` records, matching upstream's behavior of never updating `opts.skipped` at the fetch-failure catch sites. - **`CreateVirtualStore`**: cold-batch dispatch swallows per-snapshot errors when `snapshot.optional` is true **and** the error is fetch-side. A new `is_fetch_side_failure` helper restricts the swallow to `InstallPackageBySnapshotError::DownloadTarball` and `::GitFetch` — the same surface upstream wraps in `storeController.fetchPackage`. Local-materialization (`CreateVirtualDir`) and config-shape errors (`MissingTarballIntegrity` / `UnsupportedResolution`) propagate even for optional snapshots, matching upstream's `linkPkg` path which sits outside the catch. Side benefit: the warm-batch path (which only surfaces `CreateVirtualDir` errors) stays consistently abort-on-error without a parallel swallow site. The per-install `fetch_failed: HashSet<PackageKey>` rides out on `CreateVirtualStoreOutput`. - **`InstallFrozenLockfile::run`**: folds the returned `fetch_failed` into its mutable `SkippedSnapshots` so downstream phases (`SymlinkDirectDependencies`, `LinkVirtualStoreBins`, `BuildModules`, hoist) treat the dropped snapshots as absent through the same gate they already use for installability skips. ## Test plan - [x] `reporter::tests::skipped_optional_resolution_failure_event_matches_pnpm_wire_shape` — full payload (`bareSpecifier` camelCase, no `id`). - [x] `reporter::tests::skipped_optional_resolution_failure_omits_absent_name_and_version` — optional-field shape. - [x] `install::tests::frozen_install_silently_swallows_unreachable_optional_tarball` — ports [`installing/deps-restorer/test/index.ts:340-360`](https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-restorer/test/index.ts#L340-L360): unreachable optional tarball (`http://127.0.0.1:1/...`), install succeeds, slot not created, `.modules.yaml.skipped` left empty. **Test-the-test verified** by inverting the `optional` guard in the cold-batch match arm. Runs with `enable_global_virtual_store = false` so the slot-path assertion targets the legacy layout. - [x] `install::tests::frozen_install_propagates_non_optional_fetch_failure` — pins the polarity: same fixture, non-optional snapshot, install must error. - [x] `just ready` (typos + fmt + check + nextest + clippy). - [x] `taplo format --check`. - [x] `just dylint` (perfectionist). - [x] `RUSTDOCFLAGS="-D warnings" cargo doc --no-deps --workspace`. ## Out of scope - Resolver-side `resolution_failure` emit site — needs a resolver, tracked separately. - `--no-optional` / `--omit=optional` plumbing — umbrella slice 5. - Surfacing fetch failures to the reporter on the frozen path — upstream itself is silent here; only the resolver-side emit fires `pnpm:skipped-optional-dependency`. Closes #471. |
||
|
Zoltan Kochan
|
5d023a0476 |
test(git-fetcher): port real-npm §E tests with skip_if_no_npm helper (#436 §E follow-up) (#477)
Closes the three §E deferrals that needed a real package manager on the test host. Adds a `skip_if_no_npm()` helper parallel to the existing `skip_if_no_git()` so hosts without node tooling silently skip rather than fail — the rest of the suite stays portable for Rust-only sandboxes. ### Three new tests - **`fetcher_runs_prepare_script_when_allowed`** — ports [`fetching/git-fetcher/test/index.ts:129`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/git-fetcher/test/index.ts#L129). Manifest with `scripts.prepare = "node -e \"...writeFileSync('PREPARED.marker', 'ok')\""` and `allow_build` returning true. After the fetcher runs, asserts the marker file lands in `cas_paths` — proving the script actually executed, since the file didn't exist in the source tree. - **`fetcher_surfaces_prepare_failure`** — ports [`fetching/git-fetcher/test/index.ts:212`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/git-fetcher/test/index.ts#L212). Prepare script exits non-zero (`node -e "process.exit(1)"`). Expects `GitFetcherError::Prepare(PreparePackageError::LifecycleFailed)` carrying the `ERR_PNPM_PREPARE_PACKAGE` diagnostic code. - **`fetcher_runs_prepare_when_allow_build_returns_true`** — ports [`fetching/git-fetcher/test/index.ts:280`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/git-fetcher/test/index.ts#L280). Mirror of the existing block-test (`fetcher_blocks_build_when_not_allowed`) with a per-(name, version) `allow_build` closure: pins the gate's polarity so a regression that inverted "allow=true ⇒ run" would fail this test even though the block-test stays green. A shared `make_bare_repo_with_prepare_script` fixture builder takes a prepare-script string and returns the `(bare_repo, commit_sha)` pair. Manifest carries no dependencies so the synthesized `<pm>-install` step has nothing to fetch — keeps the test self-contained without verdaccio. ### `plans/TEST_PORTING.md` updates Three §E checkboxes (lines 583/587/591) flipped from `[ ]` to `[x]` with the covering pacquet test name. ## Out of scope (still on #436) - Two-slot CAS layout (`${filesIndexFile}\traw` + final). Pure perf optimization. - §E PATH-shim helper for the `still able to shallow fetch for allowed hosts` test (line 586). - §E Stage 2 install-level tests in `installing/deps-installer/test/install/fromRepo.ts` (resolver-dependent). |
||
|
Khải
|
9a0199c0a1 |
feat: auth (#337)
Resolves <https://github.com/pnpm/pacquet/issues/336>. ## Summary Ports pnpm v11's auth flow so `pacquet install` can talk to private registries on the same footing as `pnpm install`. Pacquet now parses credentials out of `.npmrc`, expands `${VAR}` references against the process environment, and attaches the right `Authorization` header on every metadata fetch and tarball download — including tarballs hosted on a CDN host that differs from the registry host. Upstream reference: [`pnpm/pnpm@601317e7a3`](https://github.com/pnpm/pnpm/tree/601317e7a3). ## What's wired up * `_auth`, `_authToken`, `username`, and `_password` keys, both default-registry (bare) and per-registry (`//host[:port]/path/:_authToken=…` family), parsed in `crates/config` (formerly `crates/npmrc`, renamed by #420). * `${VAR}` and `${VAR:-default}` substitution with backslash-escape semantics, ported from [`@pnpm/config.env-replace`](https://github.com/pnpm/components/blob/9c2bd17/config/env-replace/env-replace.ts). Unresolvable placeholders surface a `tracing::warn!` and leave the value verbatim — same best-effort behaviour as pnpm's [`substituteEnv`](https://github.com/pnpm/pnpm/blob/601317e7a3/config/reader/src/loadNpmrcFiles.ts#L156-L162). * URL-keyed lookup (`AuthHeaders`) in `crates/network`, mirroring [`createGetAuthHeaderByURI`](https://github.com/pnpm/pnpm/blob/601317e7a3/network/auth-header/src/index.ts): nerf-darts the request URL, walks parent path prefixes from longest to host-only, falls back to a port-stripped lookup matching upstream's [`removePort`](https://github.com/pnpm/pnpm/blob/601317e7a3/network/auth-header/src/helpers/removePort.ts) (any port, not just protocol defaults), and prefers inline `user:password@` basic auth when present. * Default-registry creds key against the *resolved* registry. The two-phase `apply_registry_and_warn` / `build_auth_headers` split inside `Config::current` ensures `pnpm-workspace.yaml`'s `registry` override propagates to the bearer-token nerf-dart key. * The `Authorization` header is attached on `Package::fetch_from_registry`, `PackageVersion::fetch_from_registry`, and inside `DownloadTarballToStore`'s retry loop. ## Test porting checklist Boxes ticked in `plans/TEST_PORTING.md` for the upstream auth tests this PR ports: * `network/auth-header/test/getAuthHeadersFromConfig.test.ts` — `should convert auth token to Bearer header`, `should convert basicAuth to Basic header`, `should handle default registry auth (empty key)`. * `network/auth-header/test/getAuthHeaderByURI.ts` — all 7 entries (`getAuthHeaderByURI()`, basic-auth-without-settings, basic-auth-with-settings, https-port-443, default-ports, registry-with-pathnames, default-registry-auth). * `config/reader/test/parseCreds.test.ts` — `authToken`, `authPairBase64`, `authUsername and authPassword`. The remaining auth checkboxes (`tokenHelper`, `pnpm auth file` precedence, redirect header-stripping, the `installing/deps-installer/test/install/auth.ts` integration tests) need features pacquet doesn't yet have — token helpers, the `auth.ini` layer, scoped registries, redirect handling — and stay open for follow-ups. ## Notes / scope * `always-auth` is intentionally not honoured. pnpm v11 doesn't either — the auth header is selected by URL match alone, and `always-auth` was deprecated upstream. * TLS / proxy / scoped `@scope:registry` keys remain unparsed; the parser silently ignores them, matching the pre-#336 behaviour. The `creds_by_uri` shape is ready to grow as those land. * The 401/403/404 fail-fast retry policy in `crates/tarball` was already in place from #259; this PR's auth header attaches before the retry loop, so the policy still applies unchanged. * Env-var injection follows the trait-per-capability DI pattern from [pnpm/pacquet#339](https://github.com/pnpm/pacquet/issues/339): `pub trait EnvVar` + `pub struct RealApi` in `crates/config/src/api.rs`, threaded through `env_replace`, `NpmrcAuth::from_ini`, and `Config::current` under one `Api: EnvVar` bound. Production callers turbofish `RealApi` explicitly: `Config::current::<RealApi, _, _, _, _>(...)`. |
||
|
Zoltan Kochan
|
3175a0f3b9 |
test(modules-yaml): pin hoistedLocations round-trip (#438 slice 2) (#473)
Slice 2 of #438 (the `nodeLinker: 'hoisted'` umbrella). Smaller than expected — when I looked, the schema field was already in place from #332's original `.modules.yaml` port, so the work reduced to pinning behavior rather than building plumbing. Two round-trip tests + a doc-comment on the field. ## What was actually missing The umbrella's §"Pacquet's current state" table claimed `hoistedLocations` was "Missing entirely" — that was wrong. The field is at `crates/modules-yaml/src/lib.rs:212` and serde-handles read/write correctly. What was missing: - **A test pinning the wire shape.** Nothing was stopping a future edit from breaking the camelCase mapping, the optional-with-skip-on-None behavior, or the `Record<string, string[]>` shape upstream relies on. - **A doc-comment** explaining what the field is for. Other Slice-related fields (`hoisted_dependencies`, `hoisted_aliases`) have full docstrings; this one was silent. ## Tests added Both live in `crates/modules-yaml/tests/real_fs.rs` (pacquet-side, no upstream counterpart — upstream's `installing/modules-yaml/test/index.ts` doesn't exercise `hoistedLocations` directly): - `hoisted_locations_round_trips` — a manifest with `hoistedLocations` populated survives write+read; the raw on-disk JSON keeps the per-depPath array shape (multi-entry arrays included, to pin the nested-conflict layout). - `absent_hoisted_locations_is_omitted_on_write` — `None` (the state every pacquet install writes today) serializes as the field *absent* from the file, matching upstream's `JSON.stringify(undefined)` behavior. Guards against accidental `Option::is_some` regressions on the `skip_serializing_if`. ## Regression catch verified Per `plans/TEST_PORTING.md`'s "break the subject to verify the test catches it" convention, I temporarily added `#[serde(rename = "wrongName")]` to the field. `hoisted_locations_round_trips` failed at the `expect("present")` on the deserialized value (the camelCase key no longer mapped). Reverted before commit. ## Type-shape decision Upstream's actual `ModulesRaw.hoistedLocations` is `Record<string, string[]> | undefined` — *not* `Record<DepPath, string[]>` despite the values being populated from depPaths internally. The umbrella's scope item ("`Option<BTreeMap<DepPath, Vec<String>>>`") was inconsistent with the upstream schema; I kept the existing `BTreeMap<String, Vec<String>>` because: 1. It already matches the upstream wire shape exactly. 2. Same pattern as the `hoisted_dependencies` field below, which deliberately keeps `String` keys (per its doc-comment at lines 148-153) because pnpm's `DepPath | ProjectId` union can't be statically disambiguated. |
||
|
Zoltan Kochan
|
b61c90d695 |
feat(git-fetcher): real npm-packlist semantics with .npmignore + bundleDependencies (#436 follow-up) (#468)
Replaces the MVP `files`-field-only packlist with a faithful port of [`npm-packlist`](https://github.com/npm/npm-packlist) / [`pnpm/fs/packlist`](https://github.com/pnpm/pnpm/blob/94240bc046/fs/packlist/src/index.ts). Four passes: 1. **`.npmignore` + `.gitignore` walk** via `ignore::WalkBuilder` with per-directory inheritance. A `.npmignore` in `lib/` applies to `lib/**` only; the root `.gitignore` applies to the whole tree. `parents(false)` keeps ignore-file lookup confined to `pkg_dir` so a `.gitignore` *above* the package can't leak into the published file set. 2. **`files`-field allowlist** compiled into a single `Gitignore` matcher. Uses `matched_path_or_any_parents` so a pattern like `cli` matches both `bin/cli` (file match) and `lib/cli/index.js` (directory match, where `lib/cli` is a dir). 3. **Always-included files**: `package.json`, `README*`, `LICEN[SC]E*`, `CHANGES*`, `CHANGELOG*`, `HISTORY*`, `NOTICE*` at the root, plus `main` / `bin` paths. Override `.npmignore` and the `files`-field filter, matching npm-packlist's `alwaysIncluded`. `main` and `bin` are vetted through the always-excluded check before insertion — a manifest with `"main": "package-lock.json"` or `"bin": ".git/hook"` does not get to re-add cruft. 4. **`bundleDependencies` / `bundledDependencies` recursion** with cycle detection: each bundled dep gets its own packlist pass under `node_modules/<name>/`. Both spellings accepted plus the `bundleDependencies: true` shorthand. Cycle protection uses a `HashSet<PathBuf>` of canonical visited paths + a `MAX_BUNDLE_DEPTH = 32` belt-and-braces cap so a symlink loop or self-referencing bundle can't stack-overflow the fetcher. `bundleDependencies` names are validated against the same path-traversal discipline `cas_io::join_checked` uses (`Component::Normal` / `CurDir` only; rejects `..`, root, drive prefix). ### Always-excluded set, split by type - `ALWAYS_EXCLUDED_DIR_SEGMENTS` — `.git`, `.svn`, `.hg`, `CVS`. Matched at any path segment, so VCS state is dropped at any depth. - `ALWAYS_EXCLUDED_BASENAMES` — `.npmrc`, `npm-debug.log`, `.DS_Store`, `package-lock.json`, `yarn.lock`, `pnpm-lock.yaml`. Basename-only match, so a regular file like `lib/cvs-tools.txt` ships even though it mentions `CVS`. - `ALWAYS_EXCLUDED_SUFFIXES` — `.orig` family. ### New runtime dep `ignore = "0.4.25"` (BurntSushi, used by ripgrep; `Unlicense OR MIT`, `cargo deny` clean). Replaces the hand-rolled `*` / `**` / `?` matcher in the old `packlist.rs` — the new matcher uses `ignore::gitignore::GitignoreBuilder::add_line`, which gives full gitignore semantics (negation, anchored vs unanchored, directory-suffix rules) for free. ### Intentional upstream divergences (documented in the module header) - When both `.npmignore` and `.gitignore` exist in the same directory, `ignore` combines their rules; npm-packlist would use only `.npmignore`. Same outcome for the common case (a `.npmignore` that's a strict superset); divergence only when `.npmignore` explicitly includes a path `.gitignore` excludes — rare in published packages. - `.git/info/exclude` and global `~/.gitignore` are NOT honored. The fetcher operates on a clean tarball / checkout, not the user's working tree. |
||
|
Zoltan Kochan
|
1c1f600816 |
feat(graph-hasher): engine-agnostic GVS hash gating (#459) (#469)
* feat(graph-hasher): engine-agnostic GVS hash gating (#459) Port pnpm's `transitivelyRequiresBuild` + `computeBuiltDepPaths` into `graph-hasher`, and extend `calc_graph_node_hash` with the `builtDepPaths`-driven engine-gating that mirrors upstream's [`calcGraphNodeHash`](https://github.com/pnpm/pnpm/blob/94240bc046/deps/graph-hasher/src/index.ts#L140-L142). When a snapshot — and its entire transitive subtree — has no entry in `allowBuilds` (and `dangerouslyAllowAllBuilds` is off), its GVS hash payload now sets `engine: null` instead of `engine: ENGINE_NAME`. Pure-JS packages share one slot directory across Node.js versions, OSes, and CPUs, matching upstream and unlocking the cross-host store sharing the GVS layout was designed for. `VirtualStoreLayout::new` grows an `Option<&AllowBuildPolicy>` parameter. `InstallFrozenLockfile::run` passes the install's `AllowBuildPolicy` through; callers that pass `None` retain the pre-#459 always-include-engine shape (matches upstream's `builtDepPaths === undefined` short-circuit). Tests: - six new `transitively_requires_build` unit tests covering self-hit, transitive ancestor, unrelated tree, missing node, cycle termination, and cycle-doesn't-mask-sibling-builder - four new `calc_graph_node_hash` tests covering the engine-agnostic vs engine-specific paths and the `built_dep_paths = None` shortcut - two end-to-end `VirtualStoreLayout::new` tests proving a pure-JS snapshot's slot directory is identical across two `engine` strings, and that a built snapshot's slot differs --- Written by an agent (Claude Code, claude-opus-4-7). * docs(graph-hasher): address Copilot feedback on calc_graph_node_hash docs - Drop the cross-module intra-doc link `[transitively_requires_build]: crate::dep_state::transitively_requires_build` that resolved to a pub(crate) item. The link rendered correctly under pacquet's CI (which runs `cargo doc --document-private-items`), but downstream doc builds without that flag would trip `private_intra_doc_links`. Switched to a plain-backticks reference + an inline "private to this crate" note so the doc remains informative either way. - Reword the `build_required_cache` hint to avoid the ambiguous `&mut HashMap::new()` phrasing — temporaries-in-call-args is a valid pattern but a confusing one to surface at the API doc. Pointed callers at the explicit `let mut cache = HashMap::new();` + `&mut cache` shape instead. |
||
|
Zoltan Kochan
|
91e537f656 |
feat: persist .modules.yaml.skipped + headless seed-and-recompute (#434 slice 3) (#467)
## Summary Slice 3 of the [#434 umbrella](https://github.com/pnpm/pacquet/issues/434) (`Proper optionalDependencies support`). Slice 1 (#439) computed a `SkippedSnapshots` set on every frozen-lockfile install but never serialized it; slice 2 (#456) closed the input side. This PR closes the loop by mirroring three upstream sites pinned at [pnpm/pnpm@94240bc046](https://github.com/pnpm/pnpm/tree/94240bc046): - **Write** — `Modules.skipped` was declared at [`crates/modules-yaml/src/lib.rs:197`](https://github.com/pnpm/pacquet/blob/97010013/crates/modules-yaml/src/lib.rs#L197) with sort-on-write but always serialized empty. `build_modules_manifest` now takes `&SkippedSnapshots` and produces the depPath string list pnpm writes at [`installing/deps-installer/src/install/index.ts:1625`](https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-installer/src/install/index.ts#L1625). - **Seed** — `install_frozen_lockfile::run` reads `.modules.yaml.skipped` before computing the in-memory set and passes the parsed `PackageKey`s as the seed to `compute_skipped_snapshots`. Mirrors upstream's load at [`installing/read-projects-context/src/index.ts:79`](https://github.com/pnpm/pnpm/blob/94240bc046/installing/read-projects-context/src/index.ts#L79) plus the early return at [`deps/graph-builder/src/lockfileToDepGraph.ts:194`](https://github.com/pnpm/pnpm/blob/94240bc046/deps/graph-builder/src/lockfileToDepGraph.ts#L194). - **Short-circuit** — `compute_skipped_snapshots` returns the seed verbatim on the fast path (no constraints in the lockfile) and, on the slow path, skips the per-snapshot re-check for keys already in the seed without emitting a duplicate `pnpm:skipped-optional-dependency` event. Non-seeded snapshots still re-run `package_is_installable`, covering the \"host arch changed since last install\" case upstream guards at [`:206-215`](https://github.com/pnpm/pnpm/blob/94240bc046/deps/graph-builder/src/lockfileToDepGraph.ts#L206-L215). `InstallFrozenLockfile::run` now returns a small `InstallFrozenLockfileOutput { hoisted_dependencies, skipped }` struct so `Install::run` can thread both into `.modules.yaml`. Read errors on `.modules.yaml` degrade to an empty seed — the file is a cache artifact, not authoritative. |
||
|
Zoltan Kochan
|
468a80d0a4 |
feat(real-hoist): multi-round convergence + hoistingLimits (#438 slice 3 close-out) (#465)
Closes out the Slice 3 hoist-algorithm port. Two additions on top of slice 3c's peer-aware hoist:
1. **Multi-round convergence.** The DFS runs in a fixed-point loop — a peer-shadow refusal in one round can be reconsidered in a later round once the blocking ident has moved out of the ancestor chain (or a previously-empty root slot has been filled with a compatible version). Bounded by O(N) rounds since every move is one-way.
2. **`hoistingLimits` enforcement.** The upfront `UnsupportedHoistingLimits` guard is gone; the algorithm now reads `opts.hoisting_limits[root_locator]` and refuses to hoist names listed there. New `AbsorbDecision::Border` variant alongside `Free` / `SameNode` / `Conflict` / `PeerShadow`.
Plus a backfill of behavior-pinning tests ported from `@yarnpkg/nm/tests/hoist.test.ts`.
## How multi-round works
- `hoist_subtree` returns `bool` for whether it moved any node in the current round.
- `hoist_into_root` wraps it in a `loop` that resets `visited` each iteration and exits when a round makes no moves.
- Mirrors upstream `hoistTo`'s `do { hoistGraph(); } while (anotherRoundNeeded)` without the per-candidate `dependsOn` bookkeeping; pacquet's coarse re-walk catches the same unlock cases at the cost of revisiting unchanged nodes.
Canonical case (`multi_round_unlocks_peer_friendly_hoist_after_blocker_moves`): `root → app → {widget(peer: x), x@1}` with root carrying no `x`.
- **Round 1.** Alphabetical iteration visits `widget` first. App supplies `x@1`, root has no `x` → mismatch → `PeerShadow`, widget stays at app. Then `x` is evaluated: `Free`, hoist to root.
- **Round 2.** Reconsider widget. App no longer has `x`; root has `x@1`. No ancestor disagreement → `Free`. Widget hoists.
- **Round 3.** No moves; loop exits.
## How hoistingLimits works
- Computed locator: `"{ident_name}@{reference}"` — for the wrapper's root that's `".@"`, matching pnpm's keying convention.
- Border-name set: `opts.hoisting_limits.get(root_locator)`, defaulting to empty when absent.
- New `AbsorbDecision::Border` is layered between the basic decision and the peer check. Names in the set never hoist; they stay where the lockfile placed them.
- Mirrors upstream's `isHoistBorder` flag set during `cloneTree`.
## Upstream test ports
From `yarnpkg/berry@4287909fa6` `packages/yarnpkg-nm/tests/hoist.test.ts`, expressed via lockfile fixtures so they exercise the wrapper too:
- **`hoisting_limits_keeps_blocked_name_at_parent`** — `should not hoist packages past hoist boundary`.
- **`hoisting_limits_blocks_multiple_names`** — `should not hoist multiple package past nohoist root`.
- **`hoisting_limits_keyed_on_unrelated_importer_is_inert`** — non-interference sanity check.
- **`self_dependency_does_not_loop`** — `should tolerate self-dependencies`.
- **`basic_cyclic_dependency_terminates`** — `should support basic cyclic dependencies`.
## Documented gaps
The `nm_hoist` doc-comment now lists what's deferred *intentionally* rather than "not done yet":
- **Popularity-based ident preference.** Upstream's `buildPreferenceMap` picks the ident with more incoming references when two distinct deps share an alias; pacquet picks the first-visited. Affects the handful of upstream test cases that exercise the tie-break (`should honor package popularity when hoisting`, etc.).
- **Multi-importer workspace hoist trees.** Still guarded upfront by `HoistError::UnsupportedWorkspace`. Workspace-aware hoisting needs per-importer roots and a different output shape.
- **`ExternalSoftLink` descendants.** The wrapper only creates soft-links as zero-children placeholders, so upstream's "only-hoist-when-all-descendants-hoist" rule has nothing to delay today. Tests for `should not hoist portal with unhoistable dependencies` and similar were skipped.
## Upstream reference
Aligned with `yarnpkg/berry@4287909fa6`:
- [`hoist.ts:109-119`](
|
||
|
Zoltan Kochan
|
cb6f9e58c3 |
feat(lockfile): select_platform_variant + PlatformSelector (#437 slice B) (#466)
* feat(lockfile): select_platform_variant + PlatformSelector (#437 slice B) Add the variant-picking logic for `VariationsResolution` ahead of Slice D's install-pipeline dispatch: - `PlatformSelector { os, cpu, libc: Option<String> }` mirrors upstream's [`PlatformSelector`](https://github.com/pnpm/pnpm/blob/94240bc046/resolving/resolver-base/src/index.ts#L78-L83). The `libc` tri-state encodes pnpm's `string | null | undefined` shape: `None` (host doesn't care about libc — macOS/Windows/BSD) and `Some("glibc")` collapse to the same matching arm (variant must have no `libc:` annotation); `Some("musl")` requires an exact `libc: "musl"` annotation so the glibc default doesn't silently win on a musl host. - `select_platform_variant(variants, selector)` ports [`selectPlatformVariant`](https://github.com/pnpm/pnpm/blob/94240bc046/resolving/resolver-base/src/index.ts#L92-L98). Iterates `variants` in declaration order and returns the first variant whose `targets[]` contains an `(os, cpu, libc)` triple matching the selector. Targets-array scan is linear because real archives ship 1–3 target entries per variant; quadratic cost is immaterial. - `libc_matches(variant_libc, requested_libc)` is the asymmetric helper from [`libcMatches`](https://github.com/pnpm/pnpm/blob/94240bc046/resolving/resolver-base/src/index.ts#L100-L107). Crate-private so the matching policy isn't part of the public API. Lives in `pacquet-lockfile` next to the variant types (mirrors upstream's `resolver-base` package boundary). The host-platform side (constructing the `PlatformSelector` from `pacquet-graph-hasher`'s existing `host_platform/arch/libc` helpers) lands at the install dispatcher in Slice D — keeps `pacquet-lockfile` free of the `graph-hasher` dep and lets tests drive the picker with synthetic hosts. Tests: first-match wins, multi-target variant matches any host triple, no-match returns `None`, musl-host rejects glibc-default variant, musl-host matches musl-annotated variant, and a six-row `libc_matches` truth table (None / "glibc" / "musl" / unknown-future-libc on both sides) pinning the asymmetric contract from upstream. Part of #437. --- Written by an agent (Claude Code, claude-opus-4-7). * docs: refresh stale references + add declaration-order tie-breaker test Two follow-ups from Copilot review on PR #466: - `resolution.rs:152-154` claimed "the variant picker checks at runtime that the resolved inner is atomic", but `select_platform_variant` does not. Reword to describe the actual contract: pacquet's `PlatformAssetResolution.resolution` is typed as the full `LockfileResolution` for serde uniformity, and the lockfile is trusted to honor upstream's atomic-inner invariant. No infinite-recursion risk because the install dispatcher doesn't call back into `select_platform_variant` for non-`Variations` inputs. - `resolution.rs:169` referenced `pick_variant` in `pacquet-package-manager`, but the picker actually lives in this module as `select_platform_variant`. Updated the pointer. - `pick_returns_first_when_multiple_variants_match` test: two variants both list the same `(darwin, arm64)` target; the test asserts the first one wins, pinning the `Array.prototype.find` semantics. Pnpm-written lockfiles can rely on declaration order (e.g., listing a preferred build before a fallback) — without this test, a future refactor that switched the iteration to a triple-keyed `BTreeMap` would silently break that. No behavior change; doc-comment text + test addition. --- Written by an agent (Claude Code, claude-opus-4-7). |
||
|
Zoltan Kochan
|
e2bebb1eb1 |
feat(lockfile): BinaryResolution + VariationsResolution (#437 slice A) (#457)
## Summary Slice A of #437. Adds the lockfile types pnpm v11 writes for runtime dependencies (`node@runtime:`, `deno@runtime:`, `bun@runtime:`); the install pipeline doesn't dispatch them yet (subsequent slices). No new workspace deps. - **`BinaryResolution { url, integrity, bin, archive, prefix? }`** — mirrors upstream's [`BinaryResolution`](https://github.com/pnpm/pnpm/blob/94240bc046/resolving/resolver-base/src/index.ts#L41-L49). `bin` is `BinarySpec::Single(String) | BinarySpec::Map(BTreeMap)` (untagged); `archive` is the lowercase `BinaryArchive::Tarball | BinaryArchive::Zip` discriminator; `prefix` (only set on the `.zip` branch upstream) skips serialization when `None`. - **`PlatformAssetTarget { os, cpu, libc? }`** + **`PlatformAssetResolution { resolution, targets }`** per [`resolver-base`](https://github.com/pnpm/pnpm/blob/94240bc046/resolving/resolver-base/src/index.ts#L60-L69). `libc` is `Option<String>` (not an enum) so an upstream addition beyond the current `musl` value lands without a serde migration. - **`VariationsResolution { variants }`** + the `LockfileResolution`/`TaggedResolution` arms routing both new shapes through the existing `from/into ResolutionSerde` round-trip. - **Lockfile major stays at 9** — confirmed against [`core/constants/src/index.ts`](https://github.com/pnpm/pnpm/blob/94240bc046/core/constants/src/index.ts) (`LOCKFILE_MAJOR_VERSION = '9'`), so the existing `lockfile_version.major == 9` assertion holds for v11 lockfiles carrying runtime entries. **Install-dispatch stubs.** `install_package_by_snapshot.rs` and `create_virtual_store.rs` raise `InstallPackageBySnapshotError::UnsupportedResolution { kind: "binary" | "variations" }` until Slice D wires the fetcher. Adding these arms keeps the workspace compiling without forcing the full install path to land in this slice. The warm-prefetch path returns `Ok(None)` for `Variations`, routing through the cold dispatcher where the unsupported-kind error fires; upstream unwraps `Variations` before this layer ever sees one, so the branch is unreachable on well-formed input. |
||
|
Zoltan Kochan
|
0a89e132fc |
feat: add hoisting support (hoistPattern + publicHoistPattern) (#435) (#445)
Closes #435. ## Summary - Ports `@pnpm/config.matcher` (`*` glob, `!` negation, first-match-wins) into a new `pacquet_config::matcher` module — no `regex` dependency. - Ports `getHoistedDependencies` and `symlinkHoistedDependencies` from upstream `installing/linking/hoist/src/index.ts` into a new `pacquet_package_manager::hoist` module. BFS by depth + lex on nodeId, public wins ties, first-seen wins per alias, root importer's direct-dep names seed the "already hoisted" set. - Wires the hoist pass into `InstallFrozenLockfile::run` between the per-slot bin link and the `importing_done` emit. Result is threaded through `Install::run` into `.modules.yaml`'s `hoistedDependencies`. - Promotes `Config.hoist_pattern` and `Config.public_hoist_pattern` to `Option<Vec<String>>` so the install-time guard mirrors upstream's `!= null` check (issue §D). Upstream pinned at pnpm v11 [`94240bc046`](https://github.com/pnpm/pnpm/tree/94240bc046). ## Tests - **Matcher (`crates/config/src/matcher.rs`)** — 6 tests, including direct ports of upstream's `matcher()` and `createMatcherWithIndex()` test cases plus regex-special-char-literal coverage. - **Hoist algorithm (`crates/package-manager/src/hoist/tests.rs`)** — 12 unit tests covering empty graph, default `*` pattern, public hoist, public-wins-ties, negation, first-seen-wins, root direct-dep blocking, skipped-snapshot exclusion, bin-set collection, and graph-builder + direct-deps-builder helpers. - **End-to-end (`crates/cli/tests/hoist.rs`)** — 9 integration tests using pnpm to generate the lockfile and pacquet to install with `--frozen-lockfile`. Cover default private hoist, `publicHoistPattern: '*'`, both-empty, `shamefullyHoist` legacy, scoped-pattern filter, `!`-negation, `.modules.yaml` round-trip, and bin linking on both private and public sides. - **Known failures (17 stubs)** — Ports the remainder of `installing/deps-installer/test/install/hoist.ts` into `known_failures` modules via `pacquet_testing_utils::allow_known_failure!`. Each stub names the upstream URL line + the feature blocking it (workspace install #431, partial install #433, skipped optional deps, direct-dep bin precedence, `extendNodePath`). `plans/TEST_PORTING.md` updated to mark every ported `hoist.ts` entry with the corresponding pacquet test name. |
||
|
Zoltan Kochan
|
918e79176b |
test(git-fetcher): port §E git-fetcher and tarball-fetcher tests from upstream (#436 §E) (#462)
Ports five new tests against the local-bare-repo and write-to-CAS fixtures already established by §B+D and §C, plus the `plans/TEST_PORTING.md` checkbox updates for the §E subset that ships today (15 items checked off; the rest carry concrete deferral reasons — Stage 2 resolver work, real-npm requirements, perf benchmarks, or follow-up tests that need a PATH-shim helper). ### New tests in `crates/git-fetcher/src/fetcher/tests.rs` - `fetcher_packs_subfolder_when_path_set` — ports [`fetching/git-fetcher/test/index.ts:69`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/git-fetcher/test/index.ts#L69). Builds a tiny monorepo bare repo, runs `GitFetcher` with `path: Some("packages/sub")`, asserts the returned `cas_paths` only contain files relative to the sub-dir (never sibling packages, never carrying the `packages/` prefix). - `fetcher_handles_repo_without_package_json` — ports [`fetching/git-fetcher/test/index.ts:150`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/git-fetcher/test/index.ts#L150). A repo with no `package.json` at root still imports cleanly: `prepare_package` returns `should_be_built: false`, and the fetcher hands the install dispatcher the files the packlist found. - `fetcher_skips_build_when_ignore_scripts` — ports [`fetching/git-fetcher/test/index.ts:247`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/git-fetcher/test/index.ts#L247). A repo whose `scripts.prepare` would fail if it ran (`exit 1`) still produces a successful fetcher run when `ignore_scripts: true`; observing success proves the lifecycle runner never spawned. Pins the `should_be_built: true` short-circuit so callers know the package wanted a build. ### New tests in `crates/git-fetcher/src/tarball_fetcher/tests.rs` - `tarball_path_traversal_attack_is_rejected` — ports [`fetching/tarball-fetcher/test/fetch.ts:610`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/tarball-fetcher/test/fetch.ts#L610). `path: Some("../escape")` must surface as `INVALID_PATH` from `prepare_package`'s `safe_join_path` before any extraction. - `tarball_path_to_missing_subdir_is_rejected` — ports [`fetching/tarball-fetcher/test/fetch.ts:637`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/tarball-fetcher/test/fetch.ts#L637). `path: Some("does/not/exist")` for a tarball that doesn't contain the sub-dir must surface as `INVALID_PATH` rather than silently packing the root. ### `plans/TEST_PORTING.md` updates 15 §E items moved from `[ ]` to `[x]` with the covering pacquet test name. Deferrals carry concrete reasons: - **Real-npm dependency** (3 items): `fetch a package from Git that has a prepare script`, `fail when preparing a git-hosted package`, `allow git package with prepare script`. Defer until a `skip-if-no-npm` test helper exists. - **PATH-shim helper required** (1 item): `still able to shallow fetch for allowed hosts`. The predicate is unit-tested; the end-to-end `fetch --depth 1` assertion needs a `git`-binary spy. - **Stage 2 resolver work** (most install-level fromRepo tests + the 4 git-resolver tests). - **Perf benchmarks** (2 items): `fetch a big repository`. ## Out of scope (still on #436) - Two-slot CAS layout (`${filesIndexFile}\traw` + final). - Real `npm-packlist` semantics (`.npmignore`, `.gitignore`, `bundleDependencies` walking). - The Stage 2 / real-npm / PATH-shim deferrals noted above. |