* feat(reporter): add NDJSON reporting engine with pnpm:stage smoke test
Introduces `pacquet-reporter` carrying the typed `LogEvent` enum, the
no-`self` `Reporter` capability trait per #339, and `NdjsonReporter` /
`SilentReporter` implementations. Adds the `--reporter=ndjson|silent`
flag and threads a `R: Reporter` generic through `Install::run`,
emitting `pnpm:stage` `importing_started` / `importing_done` to
bracket the install — one channel wired end-to-end as a smoke test
for the schema-mirror approach.
The wire format mirrors what `@pnpm/core-loggers` defines and
`@pnpm/cli.default-reporter` consumes under `pnpm --reporter=ndjson`.
Engine only. The JS-reporter spawn-and-pipe (`--reporter=default`),
the backfill of missing log events in already-ported code (#347), and
the porting convention in AGENTS.md (#346) land in follow-up PRs. The
default reporter is `silent` until the spawn-and-pipe exists, so user-
facing output is unchanged.
Refs #345. Design: #344.
* docs(package-manager): name pnpm's `lockfileDir` and link findWorkspaceDir
Upstream emits stage events with `prefix: lockfileDir` — the directory
holding `pnpm-lock.yaml`, which collapses to the workspace root when a
workspace is present. The previous comment described the *result* in
pacquet's no-workspace world but didn't name the upstream concept or
point at the helper to mirror once workspaces land. Replacing the
comment with a TODO that links `findWorkspaceDir` makes the migration
path explicit so the next person editing this site doesn't have to
re-derive it.
No behavior change.
* fix(cli): make --reporter accepted on either side of the subcommand
Adding `global = true` to the clap attribute lets users write either
`pacquet --reporter=ndjson install` or `pacquet install --reporter=ndjson`,
matching pnpm. Without it the flag is parsed by the top-level `CliArgs`
only, so a sub-command-side use ("pacquet install --reporter=...")
produced "unexpected argument '--reporter' found".
* refactor(reporter,package-manager): apply review fixes from #349
Address feedback on the reporter PR.
reporter:
- Drop the dependency-injection issue link (#339) from `Reporter`,
the recording-fake test, and the module-level docs; convention is
visible from the code.
- Wrap `bunyan` and `bole` references as markdown links.
- Demote `Envelope`'s `#[serde(flatten)]` rationale to a regular `//`
comment so private impl detail isn't part of `///` rustdoc.
- Introduce `pub trait EnvVar` + `pub struct RealEnvVar` (reporter-
local capability trait) and split `hostname` into a generic
`resolve_hostname<E>` plus a non-generic cached production wrapper.
Add four unit tests covering `HOSTNAME`, `COMPUTERNAME` fallback,
precedence, and the unset case.
- Switch the wire-shape test to `pipe-trait`; add `pipe-trait` to
dev-dependencies.
- Fix intra-link rule violation in the `SilentReporter` test doc.
package-manager:
- Tighten the install `prefix` extraction to fail loudly on
non-UTF-8 / parentless manifest paths instead of silently emitting
an empty string. Workspace-aware resolution tracked at #357.
- Replace `concat!` YAML literals in two tests with
`text_block_macros::text_block!`; add as dev-dependency.
- Convert bare-prose identifiers in
`install_emits_stage_events_bracketing_the_run` to intra-doc
links; drop the recording-fake DI prose.
Long inline `mod tests` extraction tracked at #358.
* fix(reporter): drop private-item link from `EnvVar` public doc
`EnvVar` is `pub` but `resolve_hostname` is private; rustdoc rejects
the intra-link with `-D rustdoc::private-intra-doc-links` (the rule
added to `CODE_STYLE_GUIDE.md` in #351). Rephrase the doc to talk
about "env-dependent helpers" generally without naming the private
function.
* refactor(reporter): switch hostname to gethostname syscall + DI capability
Address the second round of review feedback on #349.
reporter:
- Replace env-var-based hostname (HOSTNAME / COMPUTERNAME) with the
real gethostname(2) syscall via the `gethostname` crate. The
env-var approach was unsound: shells set `HOST` (zsh) or
`HOSTNAME` (bash) as private variables that don't propagate to
spawned binaries, so a packaged `pacquet` would have observed an
empty hostname in most production environments.
- Introduce `pub trait GetHostName` + `pub struct RealApi`. Real
resolution lives in `RealApi::get_host_name`. Tests can supply
their own impl.
- Cache the production value in `static HOSTNAME: LazyLock<String>
= LazyLock::new(RealApi::get_host_name)`. The wire emit path
reads `&HOSTNAME` directly; the previous `hostname()` wrapper is
no longer needed.
- Drop the now-redundant `EnvVar` trait, `RealEnvVar`,
`resolve_hostname`, and the four env-resolution unit tests (those
scenarios no longer exist after the syscall switch).
reporter test layout:
- Move the tests module into its own file at
`crates/reporter/src/tests.rs` per the unit-test-layout rule in
`CODE_STYLE_GUIDE.md`. Drop `use super::*` in favor of explicit
merged imports of just the items the tests touch.
- Two new tests: `get_host_name_capability_is_mockable` (proves the
trait dispatch works for fakes) and
`real_api_returns_a_non_empty_host_name` (smoke-checks the real
syscall path).
Documentation style:
- Restructure the workspace-prefix and stage-bracketing comments to
drop em dashes per `CONTRIBUTING.md`'s writing-style guidance and
to render `bunyan` as a markdown link.
- Fix the `@pnpm/core-loggers` link to point at the package
directory rather than its `src` subdirectory.
Imports:
- Replace seven occurrences of `pacquet_reporter::SilentReporter`
qualified at the call site with a single `use pacquet_reporter::
SilentReporter` in the test module.
* docs(style): warn against referencing private items from public docs
* docs(style): rewrite doc-comment rule to match writing style
* docs(style): correct doc-comment visibility framing
* docs(style): drop misleading comment on private helper in example
* docs(style): clarify that the rule targets more-private references
---------
Co-authored-by: Claude <noreply@anthropic.com>
* docs(plans/test-porting): allow tests beyond pnpm's coverage
Clarify that the porting plan is a floor, not a ceiling: pacquet-only
tests beyond what upstream pnpm has are welcome.
* docs(plans/test-porting): clarify extra tests are encouraged, not required
Reword the "more tests than pnpm" note so it's clear that pacquet-only
coverage is a plus but not an obligation.
* docs(plans/test-porting): align extra-tests note with writing style
Remove the em dash and contractions, and split the clauses into
standalone sentences to match the style guidance in CONTRIBUTING.md.
* docs(plans/test-porting): drop redundant clause about not inventing coverage
The earlier sentences already state that extra coverage is not required,
so the trailing clause repeats the point without adding new guidance.
* docs(plans/test-porting): use "just to" for the symmetry clause
"Just to keep" conveys that symmetry alone is the disallowed reason,
which "in order to" leaves ambiguous.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* docs(style): require intra-doc links for identifier mentions
Add a CODE_STYLE_GUIDE.md section telling contributors (and AI agents)
to turn identifier mentions inside `///` and `//!` doc comments into
rustdoc intra-doc links when the identifier is reachable from scope.
Intra-links give readers one-click navigation and let rustdoc surface
broken-link warnings when items are renamed or removed.
* docs(style): tighten intra-doc link guidance
Drop the redundant `//!` callout — saying "doc comment (`///` or `//!`)"
once at the top is enough; the second example only restated the rule.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* docs(plans): note pnpm/test integration coverage in test porting plan
The TEST_PORTING.md plan cited specific pnpm/test/... files inline but
never explained that pnpm/test/ is a parallel CLI-level integration test
tree covering nearly every feature in pnpm. Agents porting features kept
auditing only the per-package test/ directories and missing CLI-level
coverage. Add an explicit instruction in the intro to audit pnpm/test/
alongside the per-package tests for every feature.
https://claude.ai/code/session_01EXbEDL1UcPmvg1jZeT6XRK
* docs(plans): drop em dashes in TEST_PORTING per writing style
CONTRIBUTING.md flags em dashes as a symptom of loose phrasing and asks
that prose be restructured rather than have the em dash swapped for
another punctuation mark. Restructure the two prose paragraphs added in
the previous commit so each clause stands as its own complete sentence.
The em dash on line 303 is inside a literal upstream TypeScript test
description and is left alone.
https://claude.ai/code/session_01EXbEDL1UcPmvg1jZeT6XRK
---------
Co-authored-by: Claude <noreply@anthropic.com>
The test-porting plan in plans/TEST_PORTING.md tracks the upstream pnpm
tests scheduled for porting and the conventions expected of the ports,
but neither AGENTS.md nor CONTRIBUTING.md pointed at it. Add references
from both so contributors find the plan when working on ported tests.
https://claude.ai/code/session_01TESBtd59bbwcaNu6jsCaiU
Co-authored-by: Claude <noreply@anthropic.com>
* docs: warn agents against breaking method chains
* docs: allow justified method-chain rewrites
* docs: show the chain-preserving fix and contrast the diffs
* docs: trim chain-preservation section to two diffs
* docs: reframe chain rule around structural shape, not diff size
* docs: drop em dash and contraction from chain-preservation section
* docs: drop redundant meta-sentence from chain-preservation section
* docs: tighten closing paragraph of chain-preservation section
---------
Co-authored-by: Claude <noreply@anthropic.com>
* perf(package-manager): cache parent-dir creation in link_file
`link_file` was calling `fs::create_dir_all(parent_dir)` for every
file it materialized, even when the previous file in the same
package had just created the same parent. `create_dir_all` is
idempotent but not free: it still stats every component of the
path on every call. With ~50K files in a typical install and most
packages depositing into a small fixed set of dirs (`lib/`,
`dist/`, `src/`, ...), that's tens of thousands of redundant
stat syscalls during the link phase.
Mirrors pnpm v11's `dirsCreated` set in
`fs/indexed-pkg-importer/src/index.ts`. Add an `EnsuredDirsCache`
type alias (`DashSet<PathBuf>`, concurrent because rayon's
`par_iter` over `cas_paths` calls into `link_file` from multiple
worker threads at once) and thread it through `link_file`'s
signature. Allocate one cache per `create_cas_files` invocation —
that scope is one package's CAS import, where the parent-dir
overlap is densest, and the cache is dropped right after so a
benchmark-style `rm -rf node_modules` between iterations never
sees a stale entry.
Concurrency: the duplicate-mkdir race is benign. If two rayon
threads both find the parent missing in the cache, both call
`create_dir_all` (which is itself idempotent) and both insert.
Worst case: one extra `create_dir_all` per fresh dir on a
contended start, then steady-state cache-hit fast path. No
locks beyond what `DashSet` does internally.
Tests:
* `link_file_populates_ensured_dirs_cache_on_create` — assert the
parent goes into the cache after a successful link, so later
siblings actually skip the syscall.
* `link_file_skips_create_dir_all_on_cache_hit` — pre-populate
the cache with a path that doesn't exist on disk and ask
`link_file` to materialize a target under it. The cache
short-circuits `create_dir_all` and the underlying copy fails
with `NotFound`. Without the short-circuit, `create_dir_all`
would lazily create the missing dir and the call would
succeed — that's the failure mode the test pins down.
All other `link_file` callers (10 unit tests in this module) get
a fresh empty `EnsuredDirsCache::new()` per call, which
preserves the prior per-test isolation.
Expected impact on warm-cache installs: the per-file
`create_dir_all` was the second cited bottleneck on the macOS
install gap. Caching collapses N-files calls to N-unique-parents
per package — roughly an order of magnitude on a tree like
`lib/{a,b,c}.js + dist/{a,b,c}.js + src/...`. Wall-time delta
should be visible in the next CI integrated-benchmark.
* refactor(package-manager): port pnpm v11 dir-precompute into create_cas_files
The previous commit on this branch added an `EnsuredDirsCache`
(`DashSet<PathBuf>`) threaded through `link_file` so concurrent
rayon workers could skip redundant per-file `create_dir_all`
syscalls. The commit message claimed this mirrored pnpm v11's
`dirsCreated` set — it doesn't. No such set exists in pnpm v11.
What pnpm actually does in `tryImportIndexedDir` (see
https://github.com/pnpm/pnpm/blob/e30c22f0d5/fs/indexed-pkg-importer/src/importIndexedDir.ts#L185-L198):
const allDirs = new Set<string>()
for (const f of filenames.keys()) {
const dir = path.dirname(f)
if (dir === '.') continue
allDirs.add(dir)
}
Array.from(allDirs)
.sort((d1, d2) => d1.length - d2.length) // shortest to longest
.forEach((dir) => fs.mkdirSync(path.join(newDir, dir), { recursive: true }))
// ... then file imports happen with NO per-file mkdir
The pattern is: precompute the unique parent dirs upfront, sort
shortest-first so each `mkdir -p` finds its ancestor on disk
(one `mkdirat` per dir instead of walking up), then mkdir
sequentially before any file imports begin. The file-import
primitive (`importFile` in pnpm, `link_file` here) never mkdirs.
Port that structure faithfully:
* Drop `EnsuredDirsCache` and the `dashmap::DashSet` import from
`link_file`. Drop the `ensured_dirs_cache` parameter and the
`if let Some(parent_dir) = target_link.parent()` mkdir block.
`link_file` is now a leaf operation matching pnpm's `importFile`
contract: caller guarantees the parent exists.
* Drop `LinkFileError::CreateDir` — `link_file` no longer mkdirs,
so it can no longer fail with this kind of error.
* Add `CreateCasFilesError::CreateDir` since `create_cas_files`
now owns the mkdir responsibility.
* In `create_cas_files`: build a `HashSet<&str>` of unique relative
parents from `cas_paths` keys, skip empty (root) entries, sort
by string-byte-length, then sequentially `create_dir_all` each
joined absolute path. mkdir `dir_path` itself first to cover
files at the package root (matches pnpm's `importIndexedDir`
prelude that mkdirs `newDir` before calling `tryImportIndexedDir`).
* Drop the two cache-behavior tests
(`link_file_populates_ensured_dirs_cache_on_create`,
`link_file_skips_create_dir_all_on_cache_hit`) — they pinned
the cache mechanism, not behavior worth preserving.
* Update the four `link_file` tests that wrote to `nested/dst.txt`
to `fs::create_dir_all(dst.parent().unwrap())` first, since
`link_file` no longer auto-mkdirs.
Performance is essentially unchanged (kernel work is identical:
D mkdirs per package either way). The user-space hot loop is
slightly leaner — no per-file `DashSet::contains` shard-locked
lookup — but the difference is well within bench noise. The win
here is structural fidelity to upstream and one less concurrent
data structure on the link path.
All 248 workspace tests pass; clippy clean.
* perf(store-dir): globalize verifiedFilesCache across the install
Hoist `check_pkg_files_integrity`'s dedup `HashSet<PathBuf>` out
of the per-call scope and into an install-scoped
`Arc<DashSet<PathBuf>>` shared by every cached-tarball lookup.
Pnpm's [`store/cafs/src/checkPkgFilesIntegrity.ts`](https://github.com/pnpm/pnpm/blob/main/store/cafs/src/checkPkgFilesIntegrity.ts)
takes the cache as a parameter (`verifiedFilesCache: Set<string>`)
and threads one instance through every snapshot's verify pass for
the whole install — pacquet was allocating a fresh set per call,
so a CAFS blob shared across N packages paid the stat / re-hash N
times instead of once.
Plumbing:
* `pacquet_store_dir` now exports `VerifiedFilesCache`
(`DashSet<PathBuf>` — concurrent because the verifier fans out
across `tokio::task::spawn_blocking`) and
`SharedVerifiedFilesCache` (`Arc<…>`).
* `check_pkg_files_integrity` takes `&VerifiedFilesCache`. Per-file
dedup logic is unchanged: skip when the resolved CAFS path is
already in the cache, insert on successful verify, leave the
cache untouched on failure so the next snapshot re-stats.
* `DownloadTarballToStore` grows a `verified_files_cache` field
(cloned `Arc` per-package; the SQLite-lookup `spawn_blocking`
closure consumes its clone). `load_cached_cas_paths` forwards
it into the `check_pkg_files_integrity` call.
* `InstallPackageBySnapshot`, `InstallPackageFromRegistry`, and
`InstallWithoutLockfile::install_dependencies_from_registry` all
thread a `&SharedVerifiedFilesCache` through.
* `CreateVirtualStore::run` and `InstallWithoutLockfile::run`
allocate the cache once and bind same-Arc clones into each
per-snapshot future (the `async move` closures can't borrow
into the surrounding scope, so we clone-then-move the Arc).
Tests:
* `careful_path_dedups_across_calls_via_shared_cache` — plant a
file, verify it once, *delete the file*, verify again with the
same cache. Without the shared cache the second verify would
stat-and-fail; with it, the path is already cached so the
short-circuit returns `passed: true` despite the missing blob.
Real installs don't delete files mid-run; the deletion is a
test handle that proves the dedup actually skipped the stat.
* All other `check_pkg_files_integrity` tests now pass an empty
`VerifiedFilesCache::new()` (per-test isolation, matching what
the per-call `HashSet` used to give them).
Expected impact on warm-cache installs: the per-package
`fs::metadata` call inside `verify_file` is the cited bottleneck on
the macOS install gap (#280-style "12-13 s vs 7 s with pnpm"
report from a user). Across-package dedup lets the same shared
CAFS blob — common ones like LICENSE / README / empty
`.npmignore` files — pay one stat for the whole install instead
of one per consuming package. The N-files × N-packages worst
case collapses to N-files.
* docs(package-manager): reword Arc-clone comment in install_without_lockfile
Address Copilot review on #285. The previous wording — "an `async move`
can't borrow into the surrounding scope" — is misleading; an `async
move` block can capture references, the lifetime would just need to
outlive the resulting future. The actual reason for the clone is to
give each per-dependency future its own owned `Arc` handle.
* fix(package-manager): handle npm-alias dependency specs
`pacquet install` panicked when a manifest entry used the
`npm:<name>@<range>` alias form because the whole spec was fed straight
into `node_semver::Range::parse`. Decompose the spec at the install
layer so the registry sees the real package name and the resolver only
sees the version range, mirroring pnpm's `parseBareSpecifier`. The
manifest key stays as the directory name under `node_modules` while the
virtual store keeps using the registry-returned name, so plain and
aliased entries for the same target share one virtual store directory.
Closes#312
* fix(package-manager): handle latest tag in unversioned npm: alias resolution
- Remove generic `Tag` from `InstallPackageFromRegistry::run()` and parse
`version_range` as `PackageTag` directly. `PackageTag::from_str` handles
both `"latest"` (from unversioned `npm:foo` aliases) and pinned versions
like `"1.0.0"`, while semver ranges (`"^1.0.0"`) fall through to the
existing range-resolution branch. This prevents the panic where
`package.pinned_version("latest").unwrap()` was called with a tag string
that `node_semver::Range` cannot parse.
- Update call sites in `install_without_lockfile.rs` from `.run::<Version>()`
to `.run()`.
- Add e2e test `unversioned_npm_alias_defaults_to_latest` that installs
`npm:@pnpm.e2e/hello-world-js-bin` (no `@<version>`) and asserts the
alias symlink is created and the real-name virtual-store dir exists."
Agent-Logs-Url: https://github.com/pnpm/pacquet/sessions/930e25bb-53da-4614-92cd-d100ab9008fe
Co-authored-by: zkochan <1927579+zkochan@users.noreply.github.com>
* style(package-manager): apply rustfmt to has_real_name_dir check
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: zkochan <1927579+zkochan@users.noreply.github.com>
## Summary
- Add a [Stage 1](https://github.com/pnpm/pacquet/issues/299) test-porting map for TypeScript parity work.
- Add a `modules-yaml` crate.
- Add the first five 1:1 known-failure ports from pnpm's modules-yaml tests, each pinned to its upstream source permalink.
The inline mod tests block had grown to roughly 1290 lines, dwarfing
the production code in lib.rs and making the file hard to navigate.
Per CODE_STYLE_GUIDE.md ("Unit test file layout"), large unit-test
modules belong in an external file.
Extract the entire #[cfg(test)] mod tests { ... } block into a sibling
crates/tarball/src/tests.rs and reference it from lib.rs with the
standard:
#[cfg(test)]
mod tests;
This is pure motion — same 27 tests, same imports (use super::*; still
resolves to lib.rs), no behavior change. lib.rs shrinks from 2334 to
1045 lines.
Closes#307.
A bare `git fetch <repo>` writes only the source's `HEAD` into
`FETCH_HEAD`, so a SHA that is not reachable from the source's `HEAD`
never lands in the bench-repo. The subsequent `git checkout <sha>` then
fails with `unable to read tree`. This affects PR branches that are
behind `main`, where `git rev-parse main` on the runner returns a SHA
whose tree is missing in the bench-repo.
Resolve the revision against the source repository first and pass the
resolved SHA to `git fetch <repo> <sha>` so the bench-repo always has
the exact commit the next step is about to check out.
Refs: https://github.com/pnpm/pacquet/pull/321#issuecomment-4326141435https://claude.ai/code/session_01DS9GB92b1Bx9y8Tk9tH1w2
Co-authored-by: Claude <noreply@anthropic.com>
Hyperfine's --show-output piped each install's stdout/stderr (with
TRACE=pacquet=info enabled) into the workflow log, drowning the PR
summary in noise. Suppress per-run output and remove the now-pointless
TRACE env var; the markdown report still captures the timing summary.
Co-authored-by: Claude <noreply@anthropic.com>
* chore(workspace): switch to Rust edition 2024
Bump `edition` to 2024 in the workspace package metadata. The toolchain
is already pinned at 1.95.0 (latest stable as of 2026-04-14), which has
2024 edition support. Migration touches:
- collapse `if let { if let { … } }` chains into `let … && let …` to
satisfy the new `clippy::collapsible_if` shape under edition 2024
(let-chains are stable in 2024).
- wrap `env::set_var` / `env::remove_var` calls in `unsafe { … }`
blocks now that they require an `unsafe` context. The test-only
`EnvGuard` already serializes env mutation against the rest of the
test process via a process-wide `Mutex`, so the safety condition
("no other thread accesses the env concurrently") still holds; the
unsafe markers just make that contract explicit.
- bind a `Client` to a named local in `AutoMockInstance::load_or_init`
so it outlives the borrow inside `MockInstanceOptions`. Edition 2024
shortens temporary lifetimes in `let` initializers, so the previous
`&Client::new()` inside a struct literal would have dangled.
- replace `prefetched.get(&index_key).is_none()` with
`!prefetched.contains_key(&index_key)` per
`clippy::unnecessary_get_then_check`.
The remaining diff is `cargo fmt` reflowing imports under the 2024
style edition (alphabetical ordering inside grouped `use { … }`
blocks).
* fix(npmrc): collapse Windows get_drive_letter `if let` chain
The `clippy::collapsible_if` lint also fires on the Windows-only
`get_drive_letter` helper, which was missed by the Linux/macOS clippy
pass during the edition 2024 migration. Apply the same `if let … &&
let …` collapse the lint suggested.
* style(lockfile): re-sort imports under 2024 style edition
Files added by `refactor: replace serde_yaml with serde_saphyr` (#320)
on main use the 2021 import-name ordering (`{serialize_yaml, Lockfile}`,
`{ser, ..., SerializerOptions}`). The 2024 style edition sorts use names
alphabetically, so the PR's merge ref tripped `cargo fmt --all -- --check`
once main caught up.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* refactor(npmrc,tarball): replace hand-written Default impls with SmartDefault
Adds the `smart-default` crate and migrates the two hand-written
`Default` impls whose bodies were field-level constants:
* `RetryOpts` (`crates/tarball/src/lib.rs`): the four retry knobs now
carry their pnpm-matching defaults as `#[default = ...]` /
`#[default(Duration::from_millis(...))]` field attributes.
* `Npmrc` (`crates/npmrc/src/lib.rs`): every field now has a
`#[default(...)]` alongside its existing `#[serde(default = ...)]`.
`Npmrc::new()` finally calls `Self::default()` instead of round-
tripping through `serde_ini::from_str("")`, closing the in-line TODO.
`ThrottledClient`'s `Default` is left alone — it delegates to the
complex `new_for_installs()` builder, which is not expressible as
field-level defaults.
Resolves https://github.com/pnpm/pacquet/issues/306.
* refactor(store-dir): replace EncodeState::new with SmartDefault
`EncodeState`'s hand-written `fn new()` only overrode one of three
fields — `next_slot: FIRST_INNER_SLOT`. The other two (`cafs_slots`,
`side_effects_slots`) were initialized to `[None; N]`, which is the
type's `Default::default()`. With `SmartDefault`, the override
becomes a single `#[default(FIRST_INNER_SLOT)]` field attribute and
the explicit constructor goes away.
Use sites switch from `EncodeState::new()` to
`EncodeState::default()`.
* refactor(npmrc): point SmartDefault at the same helpers serde uses
The numeric `Npmrc` defaults (`modules_cache_max_age`, the
`fetch_retry*` family) were encoded twice — once as a `SmartDefault`
literal, once as a `default_*()` helper consumed by
`#[serde(default = "...")]`. Drop the literals and have
`SmartDefault` call the helpers via `_code` instead, so
`Npmrc::default()` and partial-input deserialization can't drift
apart.
Caught by Copilot in the #321 review.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* chore(workspace): replace serde_yaml with serde-saphyr
serde_yaml is deprecated and unmaintained. Switch to serde-saphyr,
which offers panic-free YAML parsing and is the replacement
recommended in pnpm/pacquet#311.
The new crate's API differs from serde_yaml in two places:
- It has no Value type. The single use site,
ProjectSnapshot::dependencies_meta, holds a passthrough YAML blob
that ports cleanly to serde_json::Value.
- Its serializer folds long single-line scalars into block style
(>-) by default, which would garble the integrity hashes in
pnpm-lock.yaml. A small serialize_yaml helper turns that off via
prefer_block_scalars: false so the output continues to match the
plain-scalar format js-yaml (and therefore pnpm) emits.
Resolves https://github.com/pnpm/pacquet/issues/311.
* test(lockfile): tighten serde-saphyr serialize tests
Address review feedback on PR #320:
- Switch resolution.rs callers from `crate::serialize_yaml::to_string`
to a `use crate::serialize_yaml;` import + `serialize_yaml::to_string`,
matching the rest of the crate's import style.
- Restore the original single-comparison shape (and `pipe_ref` chain) in
the `serialize` tests for `PkgName` and `PkgNameVer`. Compare the
serialized YAML string against the expected literal directly instead
of doing a roundtrip plus a `Display` check.
* test(lockfile): assert formatting invariants on saved lockfile
`round_trip_parse_save_parse_preserves_lockfile` only checks structural
equality after parse → save → parse, so it can't catch the formatting
regression that motivated `serialize_yaml` (e.g. `integrity: sha512-…`
being emitted as a folded block scalar `>-` instead of a plain scalar
the way pnpm/`js-yaml` does). Add two byte-level assertions:
- the saved bytes must not contain `>-`, and
- they must contain a plain `integrity: sha512-` prefix.
This protects the very invariant `serialize_yaml::to_string` is meant
to enforce.
---------
Co-authored-by: Claude <noreply@anthropic.com>
* perf: drop unnecessary PathBuf::from allocations
PathBuf::from(&str) heap-allocates a new buffer; Path::new is a free
borrow. Replace the call sites where Path::new compiles instead, and
collapse PathBuf::from_str(&s).map_err(...)? into PathBuf::from(s) (the
String is consumed without copying its buffer, and from_str returns
Infallible so the error mapping was dead code).
* style: use pipe-trait per CODE_STYLE_GUIDE.md
Address review feedback on PR #319: use the pipe chain in the npmrc
deserializer and restore the method chain in registry-mock dirs.
* fix(npmrc): gate Path import to non-Windows test target
The Path import is only used in a cfg-gated non-Windows test; on
Windows clippy --deny warnings would fail on the unused import.
---------
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
`run_with_mem_cache` deadlocks under enough concurrency: a DashMap `Ref` (which holds a synchronous shard read guard) is held across two `.await` points, so another task on the same worker can call `mem_cache.insert(...)` for a same-shard key, block on the write side, and starve the runtime. Fix: clone the inner `Arc` out of the `Ref` and drop the `Ref` before any `.await` so the shard guard's lifetime stays inside one synchronous step.
## Reproduce
`pacquet install` against the `alot` benchmark workspace (~1300 packages) hangs at 0% CPU. `sample` against the hung process shows every tokio worker parked in `RawRwLock::lock_exclusive_slow`, with the main thread stuck inside `pacquet_tarball::DownloadTarballToStore::run_with_mem_cache` → `dashmap::insert` → `lock_exclusive_slow`.
The pattern was pre-existing — likely intermittent before, reliably reproducible at higher concurrency.
## What changed
`crates/tarball/src/lib.rs` — `mem_cache.get(package_url)` is now consumed by `.map(|r| Arc::clone(r.value()))`, returning `Option<Arc<RwLock<CacheValue>>>` with no shard guard held. The rest of the if-let block is unchanged.
Per-platform packages were published with the binary at mode 0644, so
pacquet's JS shim's spawnSync failed with EACCES on install. The
chmodSync(0o755) in generate-packages.mjs ran on the publish runner,
but npm/pnpm pack normalizes file modes for deterministic tarballs:
files referenced in the manifest's bin field become 0755 and everything
else 0644.
publishConfig.executableFiles is the npm-native way to keep additional
files at 0755 in the published tarball without overloading bin's "this
is a CLI entry point" semantics. Both npm and pnpm honor it.
The workspace homepage and repository URLs still pointed at
anonrig/pacquet, the project's pre-donation home. The repo now lives
under pnpm/pacquet, so update the metadata to match. This only affects
crates.io display fields; the npm provenance fix was handled separately
in 9d0d3f2.
The publish job uses npm trusted publishing with --provenance. npm's
provenance verifier compares package.json's repository.url against the
GitHub repo recorded in the OIDC-signed provenance bundle and rejects
the publish when they disagree:
Error verifying sigstore provenance bundle: Failed to validate
repository information: package.json: "repository.url" is
"https://github.com/anonrig/pacquet", expected to match
"https://github.com/pnpm/pacquet" from provenance
The main npm/pacquet/package.json was already updated to pnpm/pacquet,
but the per-platform packages emitted by generate-packages.mjs still
hardcoded the old anonrig URL. Update the URL there, and fix the same
stale reference in the bin shim's "no prebuilt binary" error message
so users land on the right issue tracker.
actions/upload-artifact@v4 makes artifact names immutable within a run:
when multiple matrix jobs upload to the same name, only the first
succeeds and the others 409. With every platform uploading to
'binaries', only one file ended up downloadable, so the publish job
saw no .zip and the unzip step failed with "No zipfiles found".
Upload per-target as binaries-<code-target>, then download with
pattern + merge-multiple to reassemble them in one directory.
The linux-arm64 build job was pinned to ubuntu-20.04, which GitHub
retired from hosted runners. Jobs requesting that label hang in the
queue forever. Match the other ubuntu entries by using ubuntu-latest.
Cargo treats `_` and `-` as distinct in `-p` specs, so `cross build -p
pacquet_cli` failed with "package ID specification did not match any
packages". The crate is named `pacquet-cli` in crates/cli/Cargo.toml.
Switch the release workflow's publish step from `npm publish` (with a
long-lived `NPM_TOKEN` secret) to `pnpm publish` authenticated via
GitHub OIDC, matching pnpm/pnpm's release flow.
Setup uses `pnpm/action-setup@v6` with `standalone: true` and pins
pnpm to the same `11.0.0-rc.5` used in CI; Node is provisioned via
`pnpm runtime -g set node 22` so the package-generation script still
has `node` on PATH. With `id-token: write` already granted, the
registry verifies the workflow's OIDC token against each package's
trusted-publisher config, so no token is needed for auth — the same
token also backs the `--provenance` attestation. `--no-git-checks`
is added because the unzipped binary artifacts leave the working
tree dirty by the time publish runs. Drops the unused
`discussions: write` permission.
* docs: reframe project as the official pnpm rewrite
- rewrite the top-level README around the pnpm-rewrite framing and a
two-phase roadmap, dropping the stale TODO list
- move debugging, testing, and benchmarking sections into CONTRIBUTING.md
- add a README to the `npm/pacquet` package and update its description
and keywords
- update the `AGENTS.md` cross-reference for the relocated benchmark
instructions
* docs: surface "not production-ready" warning at the top of READMEs
Per review feedback, lift the in-development warning into the first
paragraph of `npm/pacquet/README.md` (with bold emphasis so it stays
visible on the npm registry page) and add a matching admonition near
the top of the workspace `README.md`. The goal is to make the
project's status unmissable for anyone landing on either page.
* docs(npm): mark the npm package description as a preview
* docs(npm): note the package is not production-ready in its description
## Summary
A single transient network blip during a `pacquet install` (connection
reset, TLS error, timeout, DNS hiccup, registry 5xx) used to abort the
entire install. pnpm has retried these for years; pacquet now does
too, with the same retry classification pnpm uses.
## What changes
- **`fetch_and_extract_with_retry`** wraps the *full* tarball pipeline
— request + body buffer + integrity check + gzip decode + tar
extract — in one retried closure. Mirrors pnpm's
[`remoteTarballFetcher.ts`](https://github.com/pnpm/pnpm/blob/main/fetching/tarball-fetcher/src/remoteTarballFetcher.ts):
the body fetch and `addFilesFromTarball` (integrity + extraction)
share a single retry boundary, so a flaky transfer that survives
TCP framing but fails the SHA-512 hash or trips gzip / tar parsing
recovers via re-fetch instead of aborting the install.
- **`is_transient_error`** matches pnpm's policy exactly: only HTTP
401, 403, 404 fail fast (per `remoteTarballFetcher.ts:77-85`).
Everything else — arbitrary 4xx (e.g. 410), 5xx, network reset,
timeout, integrity mismatch, gzip / tar parse error, allocator
refusal — retries.
- **`RetryOpts`** mirrors `@zkochan/retry`'s formula
(`min(min_timeout * factor.pow(attempt), max_timeout)`,
`randomize: false`) with pnpm's defaults: 2 retries, factor 10,
10s floor, 60s cap. Same values as
[`network/fetch/src/fetch.ts`](https://github.com/pnpm/pnpm/blob/main/network/fetch/src/fetch.ts#L29-L38)
and
[`config/config/src/index.ts`](https://github.com/pnpm/pnpm/blob/main/config/config/src/index.ts#L165-L168).
- **HTTP status check** added to the fetch path. Previously a 4xx
error body was downloaded and handed to the integrity check as if
it were a tarball; now non-2xx surfaces as
`TarballError::HttpStatus`.
- **Config plumbing**: `Npmrc` gains `fetch_retries`,
`fetch_retry_factor`, `fetch_retry_mintimeout`,
`fetch_retry_maxtimeout`. Read **only** from `pnpm-workspace.yaml`
as camelCase keys (`fetchRetries`, `fetchRetryFactor`, …) — pnpm 11s
[`isIniConfigKey`](https://github.com/pnpm/pnpm/blob/main/config/config/src/auth.ts#L93-L94)
excludes the `fetch-retry*` family from `NPM_AUTH_SETTINGS`, so a
`fetch-retries=…` line in `.npmrc` is ignored upstream and is
ignored here too. Conversion to `RetryOpts` is centralised in
`package-manager::retry_config::retry_opts_from_config`.
- **Permit lifecycle**: the post-download semaphore permit is
acquired *inside* each attempt so a backoff sleep doesnt park one
of the small pool of permits and block other downloads. The
store-index writers `queue` call moves *outside* the retry loop
so transient failures dont enqueue a half-built row that a
successful retry would duplicate.
Closes#259.
Branch-tip links such as `github.com/pnpm/pnpm/blob/main/...` drift as the
branch moves and 404 once a referenced file is renamed or deleted (two of
the existing pacquet links were already pointing at moved paths). Replace
every `blob/<branch>/` link in the repo with a `blob/<10-char-sha>/`
permalink so the references stay meaningful, and update the AI-facing
guidance in AGENTS.md to require permanent links going forward.
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
A `pacquet install` was failing on `pump-2.0.1.tgz` with the opaque `error sending request for url` while a parallel `pnpm` on the same network downloaded it cleanly. Walking that down surfaced a series of independent parity gaps where pacquet's reqwest client diverged from how pnpm fetches from the registry — each one capable of producing the same generic "request failed" surface, and none of them visible in the rendered error. This PR closes all of them and adds a flattened source-chain renderer plus regression test so the next regression is debuggable.
## What changes
### `crates/network` — align the install client with pnpm's wire behaviour
- **`User-Agent: pnpm`** is set as a default header on the throttled client, matching the literal string pnpm v11 sends in [`network/fetch/src/fetchFromRegistry.ts`](https://github.com/pnpm/pnpm/blob/main/network/fetch/src/fetchFromRegistry.ts#L9). A default `reqwest::Client` sends no UA, which some registry CDNs and corporate WAFs treat as bot traffic and either reject at the edge or RST mid-handshake. Sending the literal `pnpm` (not `pacquet/<version>`) is deliberate: pacquet is a port, so any UA-keyed allow / rate-limit rule that lets pnpm through has to let pacquet through.
- **No default `Accept` header.** Pnpm attaches `application/vnd.npm.install-v1+json; q=1.0, application/json; q=0.8, */*` to every fetch including tarballs, but that's an upstream quirk we have no reason to copy. `crates/registry`'s metadata calls already set the npm-specific `Accept` per-request; tarballs send no `Accept` and the registry serves them fine.
- **Pool idle timeout 30s → 4s**, matching [`agentkeepalive`'s](https://github.com/node-modules/agentkeepalive/blob/master/lib/agent.js#L39-L41) default `freeSocketTimeout` (the agent pnpm's connection pool sits on top of). CDN edges in front of `registry.npmjs.org` close idle sockets after 5–15s without sending a FIN that hyper notices, so a longer pool TTL lets pacquet reuse a half-dead socket and surface the next request as a generic "error sending request for url" — same symptom the user hit. 4s keeps the pool useful for back-to-back downloads (pacquet runs hundreds of fetches in seconds) while staying well below typical edge keepalive.
- **`hickory-dns` resolver** in place of reqwest's default. The default routes lookups through tokio's `lookup_host`, which calls the platform's blocking `getaddrinfo`. On macOS that goes through `mDNSResponder`, which spuriously returns `EAI_NONAME` ("nodename nor servname provided") for valid hostnames when many concurrent lookups pile up — e.g. the dozens of simultaneous tarball connections this client opens. pnpm doesn't hit it because Node's `dns.lookup` runs on libuv's 4-thread pool, throttling concurrent `getaddrinfo` naturally. `hickory-dns` queries DNS over UDP/TCP directly and bypasses `mDNSResponder` entirely. Enabled via the `hickory-dns` reqwest feature in `Cargo.toml`.
- **Concurrency now follows pnpm's `networkConcurrency` formula** instead of being hardcoded at 50. Mirrors [`installing/package-requester/src/packageRequester.ts:97`](1819226b51/installing/package-requester/src/packageRequester.ts (L97)):
```ts
networkConcurrency = Math.min(64, Math.max(calcMaxWorkers() * 3, 16))
// calcMaxWorkers() = Math.max(1, availableParallelism() - 1)
```
The previous hardcoded 50 came from pnpm's `DEFAULT_MAX_SOCKETS`, but that's the agentkeepalive per-host pool *ceiling*, not the actual fetch concurrency pnpm runs at. The real cap is the smaller `networkConcurrency` (16 on 4-core / 21 on 8-core / 27 on 10-core / capped at 64).
- **The semaphore now actually bounds concurrent sockets, and stops pinning permits behind decompression.** Two interacting bugs:
1. `run_with_permit(|c| c.get(url).send())` released the permit when `.send()` resolved — which is when the response **headers** arrive, not when the body has been read. The TCP socket FD stays open through body streaming, so under `try_join_all` fan-out the next batch of permits would `connect()` while previous bodies were still draining. Effective socket count was unbounded by the semaphore, and on real installs the FD count overran macOS's per-process limit, surfacing as `EMFILE` "too many open files" mid-fetch (visible only after `walk_reqwest_chain` started showing the leaf cause).
2. Acquiring `post_download_semaphore` (`num_cpus * 2`) between `.send()` and body streaming meant network permits stayed pinned waiting for the smaller cap. With `network_concurrency = 21` and `post_download = 16`, all 21 fetch slots could end up parked on the post-download queue, collapsing effective fetch concurrency to 16 and serializing the network pipeline behind decompression.
Replaced `run_with_permit` with `acquire`, returning a `ThrottledClientGuard` (RAII, derefs to `&Client`); the four callers now hold the guard across `.json().await` (registry metadata) or body buffering (tarball download). The tarball path additionally drops the guard once the body is buffered into RAM, then acquires `post_download_semaphore` only around the `spawn_blocking` decompression. The two semaphores no longer interact, so the network permit alone bounds both concurrent sockets and concurrent buffered tarballs to `default_network_concurrency()` — matching pnpm's pQueue, which holds one slot from `fetch` through body arrival per task.
### `crates/tarball` — make `NetworkError` carry the actual reason
Reqwest's own `Display` for a request-stage failure renders `error sending request for url (URL): <inner>` only when it can find an inner source; on some failure modes (e.g. the request was dropped before `connect()` was attempted) the inner is `None`, leaving the user with no diagnostic information at all. `NetworkError` now walks `error.source()` itself and joins every stage's `Display` with `: `, so the rendered message always carries the leaf reason (`Connection refused (os error 61)`, `tls handshake eof`, `dns error: failed to lookup address`, `Too many open files (os error 24)`, …) regardless of which intermediate `reqwest` / `hyper` / `io::Error` would otherwise elide it. The `reqwest::Error` is also marked `#[error(source)]` so miette can walk the chain structurally for renderers that prefer that form. `network_error_display_includes_reqwest_inner_chain` pins the leaf-cause shape against a deterministic ECONNREFUSED to `127.0.0.1:1`.
## Summary
Closes#294.
Replaces the per-key SELECT loop in `prefetch_cas_paths` with a single batched `SELECT key, data FROM package_index WHERE key IN (?, ?, …)`, chunked at 999 placeholders. SQLite walks the `package_index` PK B-tree once per chunk instead of once per key, collapsing N round-trips across the SQLite mutex into one query.
This drops the ~50 ms of all-miss query overhead the empty-store install paid after #292 introduced the install-wide prefetch — the cold-cache regression the issue reports.
## Changes
- `StoreIndex::get_many(&[String]) -> Result<HashMap<String, PackageFilesIndex>, StoreIndexError>` — batched lookup, chunked at `GET_MANY_CHUNK = 999` (safe vs. legacy `SQLITE_MAX_VARIABLE_NUMBER` builds; rusqlite's bundled SQLite caps at 32766).
- `prefetch_cas_paths` Phase 1 now calls `guard.get_many(&cache_keys)` instead of looping `guard.get(&cache_key)` per key.
## Behaviour notes
- Decode failures on a row are logged at `debug!` and skipped, matching `load_cached_cas_paths`'s `.ok()?` stance on the per-key path — a malformed row is a cache miss, not a batch failure.
- A SQLite error during the batched read bubbles up as `StoreIndexError::Read`; `prefetch_cas_paths` already turns that into a soft fallback (per-snapshot lookups).
