mirror of
https://github.com/pnpm/pnpm.git
synced 2026-07-22 21:53:02 -04:00
node-runtime-fix
65 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Zoltan Kochan
|
1575076d08 |
chore(pacquet): fold registry-mock into root workspace, fix npm metadata (#11643)
* chore(pacquet): fold registry-mock into root workspace, fix npm metadata
Two unrelated cleanups bundled because they touch the same publishing/
workspace plumbing:
1. **`pacquet/npm/pacquet/package.json` metadata** — the imported file
still pointed at the standalone repo: `repository.url` was
`pnpm/pacquet`, `repository.directory` was `npm/pacquet`, `homepage`
and `bugs` likewise. Repoint at `pnpm/pnpm`. Update
`generate-packages.mjs` so the per-platform packages it emits at
release time also point at `pnpm/pnpm`.
2. **Fold `pacquet/tasks/registry-mock` into the root pnpm workspace**.
Pacquet's standalone-repo nested-workspace setup pinned
`nodeLinker: hoisted` "for verdaccio CJS resolution," but pnpm's
own jest globalSetup (`__utils__/jest-config/with-registry/`) calls
the same `@pnpm/registry-mock.start()` API under the default
isolated linker without issue, and verified locally that
`node launch.mjs prepare` works after consolidation. The hoisted
constraint was scoped to standalone-pacquet's install pattern; in
the monorepo it's unnecessary.
Changes for (2):
- Add `pacquet/tasks/registry-mock` to `pnpm-workspace.yaml`.
- Rename the package `@pnpm-private/pacquet-registry-mock-launcher`
(private, matches the `@pnpm-private/*` convention used by other
internal workspace members) and switch `@pnpm/registry-mock` to
`catalog:` (the root catalog already pins it at 6.0.0).
- Delete `pacquet/tasks/registry-mock/pnpm-lock.yaml` and
`pnpm-workspace.yaml` — root install handles both now.
- Delete `pacquet/package.json` and `pacquet/pnpm-lock.yaml` — the
file only had a `cargo build` script + `devEngines: pnpm`, both
already covered by root, and nothing referenced it.
- `justfile install` is now just `pnpm install` (was
`cd pacquet/tasks/registry-mock && pnpm install --frozen-lockfile`).
- `pacquet-integrated-benchmark.yml` path filter and cache key
swap the deleted nested lockfile for the root `pnpm-lock.yaml` /
`pnpm-workspace.yaml`.
Verified: `pnpm install` resolves the workspace member, the lockfile
gains a `pacquet/tasks/registry-mock` importer entry, and
`pacquet/tasks/registry-mock/node_modules/@pnpm/registry-mock` is
linked correctly under the isolated layout.
* fix(pacquet): match meta-updater conventions in registry-mock launcher
The previous version of `pacquet/tasks/registry-mock/package.json`
omitted `version`, which crashed `pn lint:meta` (meta-updater hits
`manifest.version!.split('.')[0]` for every workspace package).
Backfill all the fields meta-updater would emit for an internal
`@pnpm-private/*` private package, matching `__typecheck__/package.json`
and friends:
- `version: 1100.0.0` (the pnpm 11.x convention for non-experimental
internal packages)
- self-devDep entry (`workspace:*`) that meta-updater would otherwise
inject
- `keywords: [pnpm, pnpm11]`
- `repository` pointing at this directory inside pnpm/pnpm
This is the same shape every other `@pnpm-private/*` private workspace
member uses; it lets `pn lint:meta --test` pass without modifying the
file.
* fix: update lockfile
* chore(pacquet): add build:pacquet script at root
Restore the `cargo build --release --bin pacquet` shortcut that lived in
the deleted `pacquet/package.json`. Naming it `build:pacquet` (rather
than `build`) matches the existing namespacing convention in this
file (`lint:ts`, `lint:meta`) and leaves room for a general `build`
script later. Invoke with `pnpm build:pacquet` or `pn build:pacquet`
from the repo root.
|
||
|
Zoltan Kochan
|
763ddf1c99 |
chore(pacquet): wire pacquet workflows into monorepo (#11635)
* chore(pacquet): wire pacquet workflows into monorepo
Move Cargo workspace, Rust toolchain configs, justfile, composite actions,
and 7 workflow files out of `pacquet/` and up to the repo root so:
- cargo / just / taplo run from repo root, the way the rest of the
monorepo's tooling does
- GitHub Actions actually discovers the workflows (it only reads
`.github/workflows/` at the repo root)
Workflows are prefixed with `pacquet-` and renamed to "Pacquet ..." so
they don't collide with the existing pnpm CI. Path filters are scoped
to `pacquet/**` so they don't trigger on every commit. The cargo entry
from pacquet's standalone `dependabot.yml` is folded into the root one;
pacquet's `CODEOWNERS` and `pull_request_template.md` are dropped because
the root copies supersede them.
Path rewrites:
- `Cargo.toml` workspace members → `pacquet/crates/*`, `pacquet/tasks/*`
- all path-deps in `[workspace.dependencies]` → `pacquet/...`
- `justfile` recipes (`install`, `install-hooks`) point at `pacquet/...`
- `.taplo.toml` include globs → `pacquet/crates/*/*.toml`, `pacquet/tasks/*/*.toml`
- `pacquet/npm/pacquet/scripts/generate-packages.mjs` REPO_ROOT walks one
more level up
- workflow `paths:` filters, `hashFiles(...)`, and shell paths all updated
Verified: `cargo metadata` resolves the workspace, `cargo fmt --check`
clean, `taplo format --check` picks up all 26 Cargo.tomls, `actionlint`
reports no new issues (the `type:`-on-input warnings on the rustup action
predate this move).
* chore(pacquet): drop pnpm version pin from pacquet CI workflows
The monorepo's root `package.json` declares `pnpm@11.1.1` under
`packageManager`, which conflicts with the workflows' explicit
`version: 11.0.0-rc.5` and trips `pnpm/action-setup` ERR_PNPM_BAD_PM_VERSION.
The pin was a pacquet-era workaround for the v9 lockfile while pnpm 11
was still pre-release. Stable 11.x writes v9 too, so let action-setup
read the version from `packageManager` like every other workflow in
this repo does.
* chore(pacquet): use pnpm/setup matching the rest of the monorepo
Replaces `pnpm/action-setup@v6` with the same `pnpm/setup@b1cac3...`
SHA the rest of pnpm/pnpm uses (release.yml, test.yml, ci.yml,
benchmark.yml, audit.yml). Reads pnpm version from `packageManager`
in root package.json, and skips the implicit `pnpm install` since
pacquet does its own scoped install via `just install` (which only
touches `pacquet/tasks/registry-mock/`).
The release workflow now also installs Node via the same action
(`runtime: node@22`) instead of via `pnpm runtime -g set node 22`,
since pnpm/setup handles runtimes in one step.
* chore(pacquet): tighten permissions and Dependabot cooldown
Address zizmor warnings on the pacquet CI changes:
- `dependabot.yml`: the cargo entry I added in the previous commit
inherited from pacquet's standalone repo and is missing the
`cooldown: default-days: 7` the github-actions entry uses. Add it
so cargo and github-actions debounce updates consistently.
- `pacquet-ci.yml`, `pacquet-codecov.yml`, `pacquet-cargo-unused.yml`
lacked a top-level `permissions:` block, so GITHUB_TOKEN inherited
the repo default. Declare `contents: read` — every job in these
workflows only reads the repo and runs local checks.
The other four pacquet workflows already declare permissions
explicitly (integrated-benchmark/comment, micro-benchmark, release).
* chore(pacquet): add "reimagining" to cspell dictionary
cspell at the repo root scans all `**/README.md` and was rejecting
`pacquet/README.md` and `pacquet/npm/pacquet/README.md`, which describe
pacquet as "not a reimagining of pnpm." Add the word to the existing
allow-list rather than rewording two READMEs imported from a separate
repo.
* fix(pacquet): prefix workspace-relative paths with pacquet/
Two Rust source files looked up paths off the cargo workspace root
(\`cargo locate-project --workspace\`), which now resolves to the
monorepo root rather than the pacquet directory. Add the \`pacquet/\`
prefix:
- \`tasks/registry-mock/src/dirs.rs\` — \`registry_mock()\` was
pointing the node launcher at \`<repo>/tasks/registry-mock/launch.mjs\`
instead of \`<repo>/pacquet/tasks/registry-mock/launch.mjs\`, which
failed every Pacquet CI test job ("Cannot find module ...launch.mjs").
- \`tasks/micro-benchmark/src/main.rs\` — same idea for the
fixtures folder.
|
||
|
Zoltan Kochan
|
62900806fe |
feat(config,cli,tarball): support offline / preferOffline settings (#513)
* feat(config,cli,tarball): support `offline` / `preferOffline` settings Adds the two settings to `Config`, `pnpm-workspace.yaml` parsing, and `pacquet install` as `--offline` / `--prefer-offline` flags. CLI flags merge into Config (boolean OR) before leak, so any source (yaml, CLI) flips `true` and never the other way. Upstream pnpm gates the metadata-fetch path in [`pickPackage`](https://github.com/pnpm/pnpm/blob/94240bc046/resolving/npm-resolver/src/pickPackage.ts): when `offline: true` and no cached metadata exists, it fails with `ERR_PNPM_NO_OFFLINE_META`; `preferOffline: true` biases the resolver toward the cached copy. Pacquet's frozen-install path has **no metadata fetch** (the lockfile pins every resolution), so the upstream flag is semantically a no-op on pacquet's current flow. To make the flag actually useful for frozen installs, pacquet adds a **tarball-side gate** (no upstream equivalent — pnpm doesn't gate tarball downloads on `offline`): when both the warm prefetch and the SQLite `index.db` lookup miss, the fetcher refuses the network and errors with `ERR_PACQUET_NO_OFFLINE_TARBALL`. Same shape as `ERR_PNPM_NO_OFFLINE_META`, scoped to tarballs because that's what pacquet's frozen install needs network for. Documented as a pacquet- specific divergence in the field's rustdoc. `prefer_offline` is a no-op today — the warm prefetch + `load_cached_cas_paths` already prefer the local store. The field exists so `.npmrc` / yaml / CLI all parse cleanly; Stage 2's resolver will honour it on the metadata path. - `Config.offline` / `Config.prefer_offline`: bool, default `false`. - `WorkspaceYaml.offline` / `WorkspaceYaml.prefer_offline`: `Option<bool>`; applied via the existing `apply!` macro. - `InstallArgs.offline` / `InstallArgs.prefer_offline`: bool CLI flags; merged into Config in `cli_args.rs` between `config()` and `State::init` (the brief window when `Config::leak` returns `&'static mut Config`). - `DownloadTarballToStore.offline` / `DownloadZipArchiveToStore.offline`: bool fields. Wired through the three construction sites in `package-manager`. - `TarballError::NoOfflineTarball { package_id, url }`: new variant. `tarball_error_to_request_retry` updated for exhaustiveness; `code: ERR_PACQUET_NO_OFFLINE_TARBALL` in the retry-logger projection so a future surface that runs this error through the retry path renders the right code. - `offline_mode_skips_network_on_cache_miss` — `mockito` server with `.expect(0)`; the offline gate must fire before the network is touched. Asserts variant shape AND the diagnostic code (the code is part of the user-facing surface). - `offline_mode_still_uses_prefetched_cache` — same `.expect(0)` guard but with a prefetched cache hit; pins that the prefetch branch short-circuits *before* the offline check so warm installs under `--offline` succeed. Regression-catch verified: gating the gate behind `false &&` flipped `offline_mode_skips_network_on_cache_miss` red with the wrong-variant panic message. Reverted before commit. All bulk-edit fixups (15 `DownloadTarballToStore` / `DownloadZipArchiveToStore` literals in tests + micro-benchmark, 1 `Config` literal in package- manager tests) get `offline: false` to match the prior implicit default. * docs(cli): pin pickPackage link to commit SHA (#513 review) Per project guideline: upstream pnpm citations link to a specific commit SHA (first 10 hex chars), not a branch name. Same SHA the PR's rustdoc and commit body already use for the same file. Resolves CodeRabbit review comment on `crates/cli/README.md`: <https://github.com/pnpm/pacquet/pull/513#discussion_r3238182177>. |
||
|
Zoltan Kochan
|
11f9a27aa2 |
fix(config): default enableGlobalVirtualStore to false (#489)
Pacquet's `default_enable_global_virtual_store()` returned `true` and cited upstream's [`config/reader/src/index.ts:392-394`](https://github.com/pnpm/pnpm/blob/94240bc046/config/reader/src/index.ts#L392-L394) as the authority. But that range lives entirely inside the [`if (cliOptions['global']) { ... }`](https://github.com/pnpm/pnpm/blob/94240bc046/config/reader/src/index.ts#L348-L395) block — surrounded by `CONFIG_CONFLICT_*_WITH_GLOBAL` errors and closed by `} else if (!pnpmConfig.bin) { ... }`. So in pnpm v11: - `pnpm install --global foo` → `enableGlobalVirtualStore` defaults to `true` - `pnpm install` (regular) → stays `null`/unset → effectively `false` Pacquet doesn't have a `--global` CLI flag (only `install --frozen-lockfile`), so the applicable upstream default is the `false` one. Pre-fix, every pacquet install silently wrote slots to `<store>/links/...` and registered projects, even when the user never asked for GVS — and a project alternating between `pnpm install` and `pacquet install --frozen-lockfile` would see two different on-disk layouts. The default introduced by #444 cited the same `L392-L394` range but read it as an unconditional default. Corrected here. ## Test changes - `gvs_default_writes_links_into_global_virtual_store_dir` renamed to `gvs_default_is_off_and_paths_derive_cleanly` and inverted — now asserts the default-off behaviour. The path derivation still populates both `virtual_store_dir` and `global_virtual_store_dir` cleanly so downstream code can read either field without first checking the toggle. - `gvs_user_pinned_virtual_store_routes_into_global_virtual_store_dir` and `yaml_global_virtual_store_dir_wins_over_derivation`: yaml now opts into GVS explicitly (`enableGlobalVirtualStore: true`) so the GVS-on derivation path is still exercised. - `install/tests.rs` doc + inline comments updated to reflect that GVS-on tests need to pin the flag explicitly now. ## Benchmark The bench fixture's explicit `enableGlobalVirtualStore: false` pin is kept (it's now redundant but future-proofs the bench against a default flip), with an updated comment explaining the design intent. Same for `MinimalWorkspaceManifest.enable_global_virtual_store` doc. |
||
|
Zoltan Kochan
|
dc834bdb8e |
feat(tarball): binary fetcher for zip archives (#437 slice C) (#472)
Slice C of [#437](https://github.com/pnpm/pacquet/issues/437) — the zip half of pnpm's binary fetcher. Slices A (lockfile types) and B (variant picker) have landed; this slice adds the per-archive download and CAS-extraction pipeline. The install-dispatch site (slice D) and the `--no-runtime` flag (slice E) will follow. Mirrors upstream's [`downloadAndUnpackZip` / `extractZipToTarget`](https://github.com/pnpm/pnpm/blob/94240bc046/fetching/binary-fetcher/src/index.ts): - `extract_zip_entries` walks every entry. Path validation runs *before* the `is_dir()` early-skip so directory entries like `../evil/` still surface `PATH_TRAVERSAL`. The canonical `cas_paths` / `pkg_files_idx` key is built from `enclosed_name()` rebuilt as `Normal` components joined with `/` — `.` segments collapse, separators are cross-platform consistent, and the ignore filter sees the same string the map is keyed by. - `DownloadZipArchiveToStore` is the public sibling of `DownloadTarballToStore` — same store-index lookup, prefetch cache reuse, store-index writer queue. The retry policy, network-permit shape, and post-download semaphore gating exactly match the tarball path. - `ignore_file_pattern` is `Option<Arc<IgnoreEntryFilter>>` on both `DownloadTarballToStore` and `DownloadZipArchiveToStore`, so the Slice D dispatcher can construct a filter per fetch from runtime config without pinning to `'static`. Cloned per retry attempt; the inner trait object is shared. - `run_with_mem_cache` doc-blocks the "stable filter per URL" invariant: the cache keys solely on `package_url` (matching pnpm's `tarballCache`), and callers keep the (URL, filter) relation functional. Upstream's `archiveFilters` is keyed by `pkg.name`, and tarball URLs encode `(name, version, integrity)`, so this naturally holds. - New `TarballError::ReadZipEntries(std::io::Error)` for zip per-entry I/O failures (`try_reserve` / `read_to_end`); maps to `ERR_PACQUET_ZIP` in the retry classifier, distinct from the tar-specific `ERR_PACQUET_TARBALL_TAR`. Zip `unix_mode()` is masked to `0o777` before write so `CafsFileInfo.mode` stays permission-only (matches `store_dir::add_files_from_dir::file_mode_from`). `TarballError::TarballTooLarge` display generalized from "Tarball at …" to "Archive at …" so the zip pipeline's OOM path doesn't surface a misleading message. - Adds `zip = "5", default-features = false, features = ["deflate"]` to the workspace; reused in `pacquet-tarball`. |
||
|
Khải
|
9a0199c0a1 |
feat: auth (#337)
Resolves <https://github.com/pnpm/pacquet/issues/336>. ## Summary Ports pnpm v11's auth flow so `pacquet install` can talk to private registries on the same footing as `pnpm install`. Pacquet now parses credentials out of `.npmrc`, expands `${VAR}` references against the process environment, and attaches the right `Authorization` header on every metadata fetch and tarball download — including tarballs hosted on a CDN host that differs from the registry host. Upstream reference: [`pnpm/pnpm@601317e7a3`](https://github.com/pnpm/pnpm/tree/601317e7a3). ## What's wired up * `_auth`, `_authToken`, `username`, and `_password` keys, both default-registry (bare) and per-registry (`//host[:port]/path/:_authToken=…` family), parsed in `crates/config` (formerly `crates/npmrc`, renamed by #420). * `${VAR}` and `${VAR:-default}` substitution with backslash-escape semantics, ported from [`@pnpm/config.env-replace`](https://github.com/pnpm/components/blob/9c2bd17/config/env-replace/env-replace.ts). Unresolvable placeholders surface a `tracing::warn!` and leave the value verbatim — same best-effort behaviour as pnpm's [`substituteEnv`](https://github.com/pnpm/pnpm/blob/601317e7a3/config/reader/src/loadNpmrcFiles.ts#L156-L162). * URL-keyed lookup (`AuthHeaders`) in `crates/network`, mirroring [`createGetAuthHeaderByURI`](https://github.com/pnpm/pnpm/blob/601317e7a3/network/auth-header/src/index.ts): nerf-darts the request URL, walks parent path prefixes from longest to host-only, falls back to a port-stripped lookup matching upstream's [`removePort`](https://github.com/pnpm/pnpm/blob/601317e7a3/network/auth-header/src/helpers/removePort.ts) (any port, not just protocol defaults), and prefers inline `user:password@` basic auth when present. * Default-registry creds key against the *resolved* registry. The two-phase `apply_registry_and_warn` / `build_auth_headers` split inside `Config::current` ensures `pnpm-workspace.yaml`'s `registry` override propagates to the bearer-token nerf-dart key. * The `Authorization` header is attached on `Package::fetch_from_registry`, `PackageVersion::fetch_from_registry`, and inside `DownloadTarballToStore`'s retry loop. ## Test porting checklist Boxes ticked in `plans/TEST_PORTING.md` for the upstream auth tests this PR ports: * `network/auth-header/test/getAuthHeadersFromConfig.test.ts` — `should convert auth token to Bearer header`, `should convert basicAuth to Basic header`, `should handle default registry auth (empty key)`. * `network/auth-header/test/getAuthHeaderByURI.ts` — all 7 entries (`getAuthHeaderByURI()`, basic-auth-without-settings, basic-auth-with-settings, https-port-443, default-ports, registry-with-pathnames, default-registry-auth). * `config/reader/test/parseCreds.test.ts` — `authToken`, `authPairBase64`, `authUsername and authPassword`. The remaining auth checkboxes (`tokenHelper`, `pnpm auth file` precedence, redirect header-stripping, the `installing/deps-installer/test/install/auth.ts` integration tests) need features pacquet doesn't yet have — token helpers, the `auth.ini` layer, scoped registries, redirect handling — and stay open for follow-ups. ## Notes / scope * `always-auth` is intentionally not honoured. pnpm v11 doesn't either — the auth header is selected by URL match alone, and `always-auth` was deprecated upstream. * TLS / proxy / scoped `@scope:registry` keys remain unparsed; the parser silently ignores them, matching the pre-#336 behaviour. The `creds_by_uri` shape is ready to grow as those land. * The 401/403/404 fail-fast retry policy in `crates/tarball` was already in place from #259; this PR's auth header attaches before the retry loop, so the policy still applies unchanged. * Env-var injection follows the trait-per-capability DI pattern from [pnpm/pacquet#339](https://github.com/pnpm/pacquet/issues/339): `pub trait EnvVar` + `pub struct RealApi` in `crates/config/src/api.rs`, threaded through `env_replace`, `NpmrcAuth::from_ini`, and `Config::current` under one `Api: EnvVar` bound. Production callers turbofish `RealApi` explicitly: `Config::current::<RealApi, _, _, _, _>(...)`. |
||
|
Zoltan Kochan
|
03499dd884 |
feat: activate global-virtual-store install path (#432) (#449)
## Summary Threads `VirtualStoreLayout` (from #444) through the frozen-lockfile install pipeline so packages installed with `enableGlobalVirtualStore: true` (pacquet's default since #444) land at the upstream-shaped `<store_dir>/links/<scope>/<name>/<version>/<hash>/node_modules/<pkg>` path. Closes the rest of #432 (Section B + C activation + D entry tests). ## Pipeline changes (frozen-lockfile path) - `InstallFrozenLockfile::run` constructs the install-scoped `VirtualStoreLayout` from the lockfile + engine name + config, then threads `&VirtualStoreLayout` through `CreateVirtualStore`, `CreateVirtualDirBySnapshot`, `create_symlink_layout`, `SymlinkDirectDependencies`, `InstallPackageBySnapshot`, `BuildModules`, and `LinkVirtualStoreBins`. Every site that used to compute `virtual_store_dir.join(key.to_virtual_store_name())` now goes through one `layout.slot_dir(key)` lookup. - `Install::run` calls `pacquet_store_dir::register_project` **once per importer** when `enable_global_virtual_store` is on, writing `<store_dir>/projects/<short-hash>` per workspace package so a future `pacquet store prune` can find every reachable consumer of `<store_dir>/links/...`. Best-effort: registry write failures degrade to `tracing::warn!` and the install continues. The walk runs *after* `InstallFrozenLockfile::run` succeeds so importer keys have already been validated. - `Config::current` now applies `apply_global_virtual_store_derivation` after yaml — pacquet's variant of upstream's [`extendInstallOptions.ts:343-355`](https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-installer/src/install/extendInstallOptions.ts#L343-L355). An explicit `globalVirtualStoreDir:` in yaml wins over the derivation; otherwise the field falls back to the user's pinned `virtualStoreDir` (under GVS-on) or to `<store_dir>/links`. ## Field-split deviation from upstream Upstream pnpm *mutates* `virtualStoreDir` in place under GVS, so every consumer that reads `virtualStoreDir` sees `<storeDir>/links`. Pacquet keeps `virtual_store_dir` at its project-local value and writes the GVS path into the separate `global_virtual_store_dir` field. The frozen-lockfile path consumes `global_virtual_store_dir` through `VirtualStoreLayout`; the legacy non-frozen `InstallWithoutLockfile` keeps reading `virtual_store_dir` (now via `VirtualStoreLayout::legacy`). The split is a pacquet-only concern: upstream has no without-lockfile install path and so doesn't need the second field. See the doc on `Config::apply_global_virtual_store_derivation` for the reasoning. ## Benchmarks The integrated-benchmark fixture pins `enableGlobalVirtualStore: false` so existing scenarios continue to compare apples-to-apples against the project-local `<modules>/.pnpm/<flat-name>` shape. Without the pin, every revision newer than this PR would silently switch to the GVS layout, while `pacquet@main` and the pnpm side of the comparison would still use the isolated layout — different shape, unfair comparison. GVS-specific benchmark runs are available by passing `--fixture-dir <dir>` at a copy of the fixture with the flag flipped. ## Tests - `install::tests::frozen_lockfile_under_gvs_registers_project_and_runs_clean` pins the registry write under default GVS-on (single-project case). - `install::tests::frozen_lockfile_under_gvs_registers_each_workspace_importer` pins per-importer registration: workspace root + `packages/web` each end up with their own symlink in `<store_dir>/projects/`. - `install::tests::frozen_lockfile_with_gvs_off_skips_project_registry` pins that GVS-off opts out of the registry write. - `config::tests::yaml_global_virtual_store_dir_wins_over_derivation` pins the yaml-override precedence. - All **829 workspace tests** pass under the new layout-threaded path. - Existing per-snapshot legacy tests construct `VirtualStoreLayout::legacy(...)` explicitly so they keep asserting the flat-name layout regardless of any global default. Partial-install tests from #442 pin `enable_global_virtual_store = false` so their seeded legacy-shape slots match the probe path. End-to-end port of upstream's [`installing/deps-installer/test/install/globalVirtualStore.ts`](https://github.com/pnpm/pnpm/blob/94240bc046/installing/deps-installer/test/install/globalVirtualStore.ts) is tracked as a follow-up — those tests need a small frozen-lockfile fixture against the registry-mock that doesn't exist yet. |
||
|
Zoltan Kochan
|
582ef43b6c |
refactor(config): rename pacquet-npmrc → pacquet-config and drop ini wiring (#420)
## Summary - pnpm 11 reads project-structural settings (storeDir, hoist, nodeLinker, fetchRetries, …) from `pnpm-workspace.yaml` and limits `.npmrc` to auth/registry/network keys. The crate that holds pacquet's resolved runtime config was still named `pacquet-npmrc` with a `Npmrc` type and carried `serde_ini` deserializer wiring on every field, which no longer reflected how the config is actually loaded on this branch. - Two-commit refactor: rename the crate / type / imports, then drop the dead serde_ini machinery and the seven duplicate ini-based tests. ## Commits 1. **`refactor(config): rename pacquet-npmrc crate to pacquet-config`** — mechanical rename. `crates/npmrc/` → `crates/config/`, `Npmrc` → `Config`, `pacquet-npmrc`/`pacquet_npmrc` references updated workspace-wide. `NpmrcAuth` (the `.npmrc` auth-subset parser) keeps its name. 2. **`refactor(config): drop ini deserializer wiring from Config`** — removes `Deserialize` + `#[serde(rename_all = "kebab-case")]` from `Config`, every per-field `deserialize_with`/`default = "..."` attribute, the helper fns that backed them (`bool_true`, `deserialize_bool`/`u32`/`u64`/`pathbuf`/`store_dir`/`registry`), and the `serde_ini` workspace dep. Drops seven ini-based `lib.rs` tests that already had equivalent yaml/npmrc-auth coverage. Renames `custom_deserializer` → `defaults` (only default factories left), and the `npmrc: &mut Config` local in `apply_to` / `Config::current` becomes `config`. After the cleanup `Config` is purely a merged runtime struct: yaml deserializes into `WorkspaceSettings` and applies onto a `Config`, `.npmrc` is hand-parsed by `NpmrcAuth::from_ini`, and `Config` itself is no longer deserialized from any file. |
||
|
Zoltan Kochan
|
2f64c727ea |
feat(executor): swallow optional-dep build failures and report via pnpm:skipped-optional-dependency (#397) (#419)
## Summary Closes item #6 of #397. Optional dependencies whose `postinstall` hooks fail no longer abort the install — pacquet now reports the failure through the `pnpm:skipped-optional-dependency` channel (reason `build_failure`) and continues, matching [pnpm v11's behavior at `building/during-install/src/index.ts:218-240`](https://github.com/pnpm/pnpm/blob/b4f8f47ac2/building/during-install/src/index.ts#L218-L240). Three commits: - **`feat(lockfile): surface snapshot-level optional flag`** — pacquet's lockfile reader previously dropped the `snapshots[<key>].optional` field. Adding it as `pub optional: bool` on `SnapshotEntry` with `#[serde(default, skip_serializing_if = "is_false")]` so absent ↔ false and `false` never serializes. The flag is pre-computed by pnpm's resolver at install time (ALL-paths- optional fold), so pacquet trusts the precomputed value rather than re-deriving it — matching [`lockfileToDepGraph` at deps/graph-builder/src/lockfileToDepGraph.ts:315](https://github.com/pnpm/pnpm/blob/b4f8f47ac2/deps/graph-builder/src/lockfileToDepGraph.ts#L315). - **`feat(reporter): add pnpm:skipped-optional-dependency event`** — new `LogEvent::SkippedOptionalDependency(SkippedOptionalDependencyLog)` variant + `SkippedOptionalPackage` + `SkippedOptionalReason` enum. Wire shape mirrors [pnpm's `SkippedOptionalDependencyMessage`](https://github.com/pnpm/pnpm/blob/b4f8f47ac2/core/core-loggers/src/skippedOptionalDependencyLogger.ts). All four upstream reasons (`BuildFailure`, `UnsupportedEngine`, `UnsupportedPlatform`, `ResolutionFailure`) are declared even though only `BuildFailure` is emitted today — keeps the enum closed so the other emit sites can land without widening it. - **`feat(package-manager): swallow optional-dep build failures`** — `BuildModules.run` reads `snapshot.optional`. When `run_postinstall_hooks` returns an `Err`, the dep's optional flag decides: optional ⇒ emit a `pnpm:skipped-optional-dependency` event (`reason: build_failure`, `package.id` = the dep's pkg dir to match upstream's `depNode.dir`) and continue; non-optional ⇒ propagate as `BuildModulesError::LifecycleScript` so the install aborts. Two `cfg(unix)`-gated tests cover both branches. |
||
|
Khải
|
a8e8dd3091 |
ci: setup dylint with perfectionist (#416)
Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Khải
|
ad467d7fda |
fix(integrated-benchmark): mirror settings in pnpm-workspace.yaml (#401)
Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Khải
|
aff5978e23 |
fix: optimize Claude Code Web session (#385)
Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Khải
|
e72dffc68d |
docs: make help message looks like help message (#392)
The AI treated this docs as a docs. It wrote overly detailed documentation. Unfit for a `--help` message. |
||
|
Khải
|
02865d8c85 |
ci(integrated-benchmark): extract compilation step (#390)
Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Zoltan Kochan
|
315e59b24a |
feat(reporter): emit pnpm:progress and pnpm:fetching-progress (#372)
Backfills two channels from #347 in one PR — both are "progress" shaped so they share the tarball-side reporter threading. ### `pnpm:progress` — per-package status transitions Mirrors pnpm's emit at [`installing/package-requester/src/packageRequester.ts:435`](https://github.com/pnpm/pnpm/blob/086c5e91e8/installing/package-requester/src/packageRequester.ts#L435). Four `status` values fire per package: - **`resolved`** — emitted before the tarball download in both lockfile paths. In the no-lockfile path, `install_package_from_registry::run` emits once the registry has picked a version. In the frozen-lockfile path, `create_virtual_store::run`'s warm batch and per-snapshot `install_package_by_snapshot::run`'s entry emit on each snapshot (the lockfile *is* the resolution). - **`fetched`** — emitted from `fetch_and_extract_with_retry` on the successful `Ok` branch. Failed attempts that retry don't fire it; cache-hit early returns don't fire it. - **`found_in_store`** — emitted from `run_without_mem_cache`'s cache-hit early returns (prefetched-cas + `load_cached_cas_paths`), from `create_virtual_store::run`'s warm batch where prefetch already settled the bytes, and from `run_with_mem_cache`'s in-process dedup hits (so the second requester of a shared URL also ticks the per-package counter). - **`imported`** — emitted after `create_cas_files` returns `Ok` in both `create_virtual_dir_by_snapshot::run` and `install_package_from_registry::install_package_version`. `method` is the optimistic wire-mapped configured method (`Auto`/`CloneOrCopy` → `clone`); pacquet doesn't surface the per-package resolved method past `link_file`'s install-scoped atomic. Tracked under #347 for refinement once that becomes per-package observable. ### `pnpm:fetching-progress` — per-tarball download progress Mirrors pnpm's emit at [`installing/package-requester/src/packageRequester.ts:560`](https://github.com/pnpm/pnpm/blob/086c5e91e8/installing/package-requester/src/packageRequester.ts#L560) and [`fetching/tarball-fetcher/src/remoteTarballFetcher.ts:143`](https://github.com/pnpm/pnpm/blob/086c5e91e8/fetching/tarball-fetcher/src/remoteTarballFetcher.ts#L143). Two `status` values: - **`started`** — fires exactly once per HTTP attempt, including attempts that fail before the response head arrives (DNS / connect / timeout) so retried attempts stay visible in the reporter. `size` is the response's `Content-Length` when a response head arrives, and JSON `null` (i.e. `None`) when there is no response, or when the response is chunked / unknown-length. pnpm's reporter checks `size != null` before rendering a percent gauge. - **`in_progress`** — gated and throttled to match pnpm exactly: - Skip entirely when `Content-Length` is unknown (chunked / streaming) or the tarball is smaller than `BIG_TARBALL_SIZE = 5 MB`. Most npm packages are well under this; per-byte progress for tiny tarballs floods the JS reporter with values that would render as 100% before any UI tick can show them. - Throttle to 500ms (matches `lodash.throttle(opts.onProgress, 500)`). - Trailing edge: a final emit after the body finishes if the last in-flight `downloaded` value differs from the last reported one. Matches `lodash.throttle`'s default `{leading: true, trailing: true}` so the consumer sees the actual end-of-download byte count. ### `run_with_mem_cache` deadlock fix Pre-existing bug exposed by review: when the *owning* requester's `run_without_mem_cache` errored, the function returned without flipping the cache slot to `Available` or notifying waiters. Concurrent siblings parked on `notify.notified().await` forever and the install hung — a single 404 could deadlock every later consumer of the same URL. Fixed by adding `CacheValue::Failed`. The owner uses `match` on the result; on `Err` it sets `Failed`, removes the entry from `mem_cache` (so a fresh request can retry without inheriting the failure), and notifies waiters. Waiters check for `Failed` on wake and surface a new `TarballError::SiblingFetchFailed { url }` instead of the `unreachable!` the previous code would have hit. ### Threading `pacquet-tarball` gains a `pacquet-reporter` dependency. `DownloadTarballToStore` carries a new `requester: &'a str` field (the install root, same value as the `prefix` in `pacquet_reporter::StageLog`). `<R: Reporter>` is threaded through `run_with_mem_cache`, `run_without_mem_cache`, `fetch_and_extract_with_retry`, and `fetch_and_extract_once`. The generic monomorphises away — zero runtime cost. `requester` and `<R>` cascade up through `InstallPackageBySnapshot`, `InstallPackageFromRegistry`, `CreateVirtualDirBySnapshot`, `CreateVirtualStore`, `InstallFrozenLockfile`, `InstallWithoutLockfile`, and `Install::run`. |
||
|
Khải
|
a5d6526867 |
refactor(registry-mock): drop portpicker, ask the OS for a port (#360)
Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Zoltan Kochan
|
0434c4e09f |
perf(store-dir): globalize verifiedFilesCache across the install (#285)
* perf(store-dir): globalize verifiedFilesCache across the install Hoist `check_pkg_files_integrity`'s dedup `HashSet<PathBuf>` out of the per-call scope and into an install-scoped `Arc<DashSet<PathBuf>>` shared by every cached-tarball lookup. Pnpm's [`store/cafs/src/checkPkgFilesIntegrity.ts`](https://github.com/pnpm/pnpm/blob/main/store/cafs/src/checkPkgFilesIntegrity.ts) takes the cache as a parameter (`verifiedFilesCache: Set<string>`) and threads one instance through every snapshot's verify pass for the whole install — pacquet was allocating a fresh set per call, so a CAFS blob shared across N packages paid the stat / re-hash N times instead of once. Plumbing: * `pacquet_store_dir` now exports `VerifiedFilesCache` (`DashSet<PathBuf>` — concurrent because the verifier fans out across `tokio::task::spawn_blocking`) and `SharedVerifiedFilesCache` (`Arc<…>`). * `check_pkg_files_integrity` takes `&VerifiedFilesCache`. Per-file dedup logic is unchanged: skip when the resolved CAFS path is already in the cache, insert on successful verify, leave the cache untouched on failure so the next snapshot re-stats. * `DownloadTarballToStore` grows a `verified_files_cache` field (cloned `Arc` per-package; the SQLite-lookup `spawn_blocking` closure consumes its clone). `load_cached_cas_paths` forwards it into the `check_pkg_files_integrity` call. * `InstallPackageBySnapshot`, `InstallPackageFromRegistry`, and `InstallWithoutLockfile::install_dependencies_from_registry` all thread a `&SharedVerifiedFilesCache` through. * `CreateVirtualStore::run` and `InstallWithoutLockfile::run` allocate the cache once and bind same-Arc clones into each per-snapshot future (the `async move` closures can't borrow into the surrounding scope, so we clone-then-move the Arc). Tests: * `careful_path_dedups_across_calls_via_shared_cache` — plant a file, verify it once, *delete the file*, verify again with the same cache. Without the shared cache the second verify would stat-and-fail; with it, the path is already cached so the short-circuit returns `passed: true` despite the missing blob. Real installs don't delete files mid-run; the deletion is a test handle that proves the dedup actually skipped the stat. * All other `check_pkg_files_integrity` tests now pass an empty `VerifiedFilesCache::new()` (per-test isolation, matching what the per-call `HashSet` used to give them). Expected impact on warm-cache installs: the per-package `fs::metadata` call inside `verify_file` is the cited bottleneck on the macOS install gap (#280-style "12-13 s vs 7 s with pnpm" report from a user). Across-package dedup lets the same shared CAFS blob — common ones like LICENSE / README / empty `.npmignore` files — pay one stat for the whole install instead of one per consuming package. The N-files × N-packages worst case collapses to N-files. * docs(package-manager): reword Arc-clone comment in install_without_lockfile Address Copilot review on #285. The previous wording — "an `async move` can't borrow into the surrounding scope" — is misleading; an `async move` block can capture references, the lifetime would just need to outlive the resulting future. The actual reason for the clone is to give each per-dependency future its own owned `Arc` handle. |
||
|
Khải
|
13b4762069 |
fix(benchmark): fetch resolved sha before checkout (#324)
A bare `git fetch <repo>` writes only the source's `HEAD` into `FETCH_HEAD`, so a SHA that is not reachable from the source's `HEAD` never lands in the bench-repo. The subsequent `git checkout <sha>` then fails with `unable to read tree`. This affects PR branches that are behind `main`, where `git rev-parse main` on the runner returns a SHA whose tree is missing in the bench-repo. Resolve the revision against the source repository first and pass the resolved SHA to `git fetch <repo> <sha>` so the bench-repo always has the exact commit the next step is about to check out. Refs: https://github.com/pnpm/pacquet/pull/321#issuecomment-4326141435 https://claude.ai/code/session_01DS9GB92b1Bx9y8Tk9tH1w2 Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Khải
|
51571b78c5 |
chore(rust): edition 2024 (#322)
* chore(workspace): switch to Rust edition 2024
Bump `edition` to 2024 in the workspace package metadata. The toolchain
is already pinned at 1.95.0 (latest stable as of 2026-04-14), which has
2024 edition support. Migration touches:
- collapse `if let { if let { … } }` chains into `let … && let …` to
satisfy the new `clippy::collapsible_if` shape under edition 2024
(let-chains are stable in 2024).
- wrap `env::set_var` / `env::remove_var` calls in `unsafe { … }`
blocks now that they require an `unsafe` context. The test-only
`EnvGuard` already serializes env mutation against the rest of the
test process via a process-wide `Mutex`, so the safety condition
("no other thread accesses the env concurrently") still holds; the
unsafe markers just make that contract explicit.
- bind a `Client` to a named local in `AutoMockInstance::load_or_init`
so it outlives the borrow inside `MockInstanceOptions`. Edition 2024
shortens temporary lifetimes in `let` initializers, so the previous
`&Client::new()` inside a struct literal would have dangled.
- replace `prefetched.get(&index_key).is_none()` with
`!prefetched.contains_key(&index_key)` per
`clippy::unnecessary_get_then_check`.
The remaining diff is `cargo fmt` reflowing imports under the 2024
style edition (alphabetical ordering inside grouped `use { … }`
blocks).
* fix(npmrc): collapse Windows get_drive_letter `if let` chain
The `clippy::collapsible_if` lint also fires on the Windows-only
`get_drive_letter` helper, which was missed by the Linux/macOS clippy
pass during the edition 2024 migration. Apply the same `if let … &&
let …` collapse the lint suggested.
* style(lockfile): re-sort imports under 2024 style edition
Files added by `refactor: replace serde_yaml with serde_saphyr` (#320)
on main use the 2021 import-name ordering (`{serialize_yaml, Lockfile}`,
`{ser, ..., SerializerOptions}`). The 2024 style edition sorts use names
alphabetically, so the PR's merge ref tripped `cargo fmt --all -- --check`
once main caught up.
---------
Co-authored-by: Claude <noreply@anthropic.com>
|
||
|
Khải
|
ec6b632151 |
refactor: drop unnecessary PathBuf::from allocations (#319)
* perf: drop unnecessary PathBuf::from allocations PathBuf::from(&str) heap-allocates a new buffer; Path::new is a free borrow. Replace the call sites where Path::new compiles instead, and collapse PathBuf::from_str(&s).map_err(...)? into PathBuf::from(s) (the String is consumed without copying its buffer, and from_str returns Infallible so the error mapping was dead code). * style: use pipe-trait per CODE_STYLE_GUIDE.md Address review feedback on PR #319: use the pipe chain in the npmrc deserializer and restore the method chain in registry-mock dirs. * fix(npmrc): gate Path import to non-Windows test target The Path import is only used in a cfg-gated non-Windows test; on Windows clippy --deny warnings would fail on the unused import. --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Zoltan Kochan
|
56fff6bc14 |
fix(tarball): retry transient fetch errors with exponential backoff (#301)
## Summary A single transient network blip during a `pacquet install` (connection reset, TLS error, timeout, DNS hiccup, registry 5xx) used to abort the entire install. pnpm has retried these for years; pacquet now does too, with the same retry classification pnpm uses. ## What changes - **`fetch_and_extract_with_retry`** wraps the *full* tarball pipeline — request + body buffer + integrity check + gzip decode + tar extract — in one retried closure. Mirrors pnpm's [`remoteTarballFetcher.ts`](https://github.com/pnpm/pnpm/blob/main/fetching/tarball-fetcher/src/remoteTarballFetcher.ts): the body fetch and `addFilesFromTarball` (integrity + extraction) share a single retry boundary, so a flaky transfer that survives TCP framing but fails the SHA-512 hash or trips gzip / tar parsing recovers via re-fetch instead of aborting the install. - **`is_transient_error`** matches pnpm's policy exactly: only HTTP 401, 403, 404 fail fast (per `remoteTarballFetcher.ts:77-85`). Everything else — arbitrary 4xx (e.g. 410), 5xx, network reset, timeout, integrity mismatch, gzip / tar parse error, allocator refusal — retries. - **`RetryOpts`** mirrors `@zkochan/retry`'s formula (`min(min_timeout * factor.pow(attempt), max_timeout)`, `randomize: false`) with pnpm's defaults: 2 retries, factor 10, 10s floor, 60s cap. Same values as [`network/fetch/src/fetch.ts`](https://github.com/pnpm/pnpm/blob/main/network/fetch/src/fetch.ts#L29-L38) and [`config/config/src/index.ts`](https://github.com/pnpm/pnpm/blob/main/config/config/src/index.ts#L165-L168). - **HTTP status check** added to the fetch path. Previously a 4xx error body was downloaded and handed to the integrity check as if it were a tarball; now non-2xx surfaces as `TarballError::HttpStatus`. - **Config plumbing**: `Npmrc` gains `fetch_retries`, `fetch_retry_factor`, `fetch_retry_mintimeout`, `fetch_retry_maxtimeout`. Read **only** from `pnpm-workspace.yaml` as camelCase keys (`fetchRetries`, `fetchRetryFactor`, …) — pnpm 11s [`isIniConfigKey`](https://github.com/pnpm/pnpm/blob/main/config/config/src/auth.ts#L93-L94) excludes the `fetch-retry*` family from `NPM_AUTH_SETTINGS`, so a `fetch-retries=…` line in `.npmrc` is ignored upstream and is ignored here too. Conversion to `RetryOpts` is centralised in `package-manager::retry_config::retry_opts_from_config`. - **Permit lifecycle**: the post-download semaphore permit is acquired *inside* each attempt so a backoff sleep doesnt park one of the small pool of permits and block other downloads. The store-index writers `queue` call moves *outside* the retry loop so transient failures dont enqueue a half-built row that a successful retry would duplicate. Closes #259. |
||
|
Khải
|
34a17056bf |
docs: pin github upstream links to commit shas (#304)
Branch-tip links such as `github.com/pnpm/pnpm/blob/main/...` drift as the branch moves and 404 once a referenced file is renamed or deleted (two of the existing pacquet links were already pointing at moved paths). Replace every `blob/<branch>/` link in the repo with a `blob/<10-char-sha>/` permalink so the references stay meaningful, and update the AI-facing guidance in AGENTS.md to require permanent links going forward. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Zoltan Kochan
|
5f01470381 |
fix(registry-mock): walk parent chain iteratively, with cycle guard (#297)
`is_descent_of` recursed up `Process::parent()` and overflowed the default 1 MiB stack on `windows-latest` during the post-test verdaccio cleanup (`pacquet-registry-mock end`). Test runs that otherwise passed all 200 tests still failed the job because the cleanup step crashed with `STATUS_STACK_OVERFLOW` (seen on #296 CI). Convert to a loop and add a `HashSet<Pid>` cycle guard. Two reasons: 1. **Stack safety.** Iterating uses constant stack regardless of process-tree depth, so deep ancestor chains can't take cleanup down again. 2. **Cycle safety.** `Process::parent()` on Windows returns `dwParentProcessID` as recorded at process-create time and is never updated when the parent dies, so the recorded parent PID can be reused by an unrelated later process and observably "loop back" through a snapshot. The `visited` set bounds the walk in that case too. |
||
|
Zoltan Kochan
|
0d9b140c31 |
test(integrated-benchmark): add headless-install-with-hot-cache scenario (#296)
* bench(integrated-benchmark): add headless-install-with-hot-cache scenario Mirrors pnpm v11's `Headless install (frozen lockfile, warm store+cache)` benchmark (`pnpm/v11/benchmarks/bench.sh`): the common "CI install" / "fresh clone in an existing developer checkout" path where the lockfile is present, `node_modules` is empty, but the store has been populated by a prior install and should be reused. The existing `frozen-lockfile` scenario stays as-is and remains cold-cache (each timed run wipes both `node_modules` and the per-revision `store-dir`). The new `frozen-lockfile-hot-cache` scenario only wipes `node_modules` between iterations; hyperfine's warmup run is what populates the store on the first iteration so every timed run sees a hot store. Per-revision cleanup paths now come from `BenchmarkScenario::cleanup_paths`. The CI workflow gains a second `Benchmark: …` step + summary section so PRs see both cold-cache and hot-cache numbers side by side. * refactor(integrated-benchmark): pre-wipe store-dir before hyperfine The hot-cache scenario's per-iteration `--prepare` intentionally preserves `store-dir` so subsequent iterations can reuse it. That means whatever a previous run / scenario / partial invocation left in `store-dir` would otherwise carry into the warmup, and the warmup wouldn't actually be the priming run. The CI workflow runs the cold-cache step before the hot-cache step in the same `bench-work-env`, which would have hit exactly this case — the cold-cache step ends with a populated `store-dir` from its last timed iteration, and the hot-cache step would inherit it. Wipe `node_modules` and `store-dir` for every benchmark target once at the top of `benchmark()`, regardless of scenario. For cold-cache scenarios this is redundant with the per-iteration wipe but harmless; for hot-cache it makes the warmup the priming run no matter what state the work-env was in (Copilot review on #296). * refactor(integrated-benchmark): drop dead `store-dir` line from generated .npmrc Pacquet's `.npmrc` parser (`crates/npmrc/src/npmrc_auth.rs`) explicitly ignores `store-dir`; pinned by the `ignores_non_auth_keys` test there. Project-structural settings like `storeDir` only come from `pnpm-workspace.yaml` now (true for pnpm too). The static fixture's `storeDir: ./store-dir` already resolves to `{bench_dir}/store-dir` under each per-revision CWD, so removing the `.npmrc` line doesn't change behaviour — it just deletes a dead config that was misleading anyone reading the generated `.npmrc`. * fix(integrated-benchmark): always ensure storeDir is set in pnpm-workspace.yaml Without a top-level `storeDir:` in the per-revision `pnpm-workspace.yaml`, both pnpm and pacquet route to their global default store (pacquet since `npmrc_auth.rs` explicitly ignores `store-dir` from `.npmrc`). The benchmark's cleanup machinery then wipes `{bench_dir}/store-dir`, which the install never wrote to, and state from previous runs leaks into the warmup. That silently invalidates the cold-cache scenario's per-iteration reset and the hot-cache scenario's "warmup is the priming run" guarantee. The default fixture has always shipped `storeDir: ./store-dir`, but custom `--fixture-dir` runs let users supply a workspace file that might omit it (or supply no workspace file at all, which the helper previously treated as fine). Now `create_pnpm_workspace` scans for a top-level `storeDir:` and prepends `storeDir: ./store-dir\n` if one isn't already there. If the user explicitly declares a different `storeDir`, we leave it alone — that's an intentional override (e.g. sharing a store across revisions to bench a specific scenario). Copilot review on #296. |
||
|
Zoltan Kochan
|
e7ce0b9155 |
perf: batch store-index prefetch + drive warm link phase on rayon (#292)
## Summary
Two structural changes that together cut warm-cache install wall time on a **1352-package lockfile** from ~10.85s → ~6.5s, beating pnpm v11.0.0-rc.5 (~8.5s on the same machine) by ~23%.
| | median wall |
|---|---|
| **this PR** | **6.52s** |
| pnpm v11.0.0-rc.5 | 8.51s |
| `main` baseline | 10.85s |
Measured on macOS (10-core M-class, APFS, store on same volume), warm cache, 8 runs each, `ulimit -n 65536`.
## Changes
### 1. Batched store-index prefetch (`crates/tarball/src/lib.rs::prefetch_cas_paths`)
Walks every `(integrity, pkg_id)` the lockfile mentions in one `spawn_blocking` task at install start, runs all integrity checks, and returns a `cache_key → cas_paths` HashMap. `DownloadTarballToStore` consults this map before falling through to the per-call `load_cached_cas_paths`.
This eliminates 1352 separate `spawn_blocking` round-trips a warm install was doing. Sample-profiling showed those each accumulating 20-60 ms of OS scheduling overhead while their actual work (≈40 µs SELECT + per-file integrity stats) is sub-ms — the queue across the default 512-thread blocking pool was dwarfing the work itself.
### 2. Warm batch via one global `rayon::par_iter` (`crates/package-manager/src/create_virtual_store.rs`)
Partition snapshots into "warm" (prefetched) and "cold" (cache miss). The warm batch bypasses tokio futures entirely:
```rust
warm.par_iter().try_for_each(|(snapshot_key, snapshot, cas_paths)| {
CreateVirtualDirBySnapshot { ... }.run().map_err(...)
})?;
```
The cold batch keeps the existing `try_join_all` + download path (rare on warm install, common on cold).
**Why this matters:** previously every per-snapshot tokio future called sync `rayon::join` from a tokio worker. With up to 10 such futures progressing concurrently and each one's inner `par_iter` saturating the rayon pool, the pool ended up processing **one snapshot at a time** despite tokio's worker count — `sum-of-link ≈ wall`, i.e. effectively 1× parallelism. Going straight to one big `par_iter` lets rayon schedule across all 1352 snapshots as one work-stealing graph, the shape pnpm's piscina pool gives implicitly.
### 3. Rayon pool sized at `availableParallelism * 2` (`crates/cli/src/lib.rs::configure_rayon_pool`)
The link phase is dominated by `clonefile` syscalls that block on the kernel's metadata journal, not CPU work. Oversubscribing CPUs gives more in-flight syscalls without losing wall time. Sweep on alot7:
| RAYON_NUM_THREADS | wall |
|---|---|
| 4 | 9.5s |
| 10 (default) | 9.4s |
| **20** | **9.25s** |
| 30 | 10.0s |
| 50 | 9.4s |
| 100 | 10.2s |
2x was the knee. Honours an explicit `RAYON_NUM_THREADS` env var.
|
||
|
Zoltan Kochan
|
00e1d3da2f |
perf(network,tarball): port pnpm v11 HTTP topology for tarball fetch (#281)
* perf(network,tarball): port pnpm v11 HTTP topology for tarball fetch Three HTTP/fetch-layer ports from pnpm v11, tracked in #280. All three are decisions pnpm landed upstream with explicit benchmark evidence, and all three are cases where a default `reqwest::Client` diverges from pnpm's chosen shape. 1. **HTTP/1.1 only** (`.http1_only()`). A default `reqwest::Client` negotiates HTTP/2 via ALPN whenever the registry advertises it, and registry.npmjs.org does. Pnpm v11's `network/fetch/src/dispatcher.ts:17-22` documents why they disabled H2: > With HTTP/2, undici multiplexes many streams over 1-2 TCP > connections sharing a single congestion window. In benchmarks > this was slower than opening ~50 independent HTTP/1.1 > connections that each get their own congestion window and can > saturate bandwidth in parallel. 2. **50 concurrent sockets**, matching pnpm's `DEFAULT_MAX_SOCKETS`. The previous `num_cpus.max(16)` semaphore under-subscribed on every machine benchmarked — on a 4-core GHA runner pacquet had 1/3 of pnpm's concurrent-fetch budget; even on a 10-core M3 the floor of 16 left bandwidth on the table. 3. **Pre-allocate the tarball buffer from `Content-Length`**. Ports `fetching/tarball-fetcher/src/remoteTarballFetcher.ts:148-164`. The old `response_head.bytes().await` let reqwest/hyper grow an internal `BytesMut` by doubling when CL wasn't used to pre-size — on a 1352-tarball cold install that's a lot of wasted realloc + copy per tarball. Switch to `bytes_stream()`, check `content_length()` on the response head, pre-allocate a `Vec<u8>::with_capacity(cl)` when known, and `extend_from_slice` each chunk in sequentially. Also catches size-mismatch via a new `TarballError::BadTarballSize` variant — pnpm surfaces the equivalent `BadTarballError` for the same case. An upstream bug or middleware truncation that produces fewer (or more) bytes than `Content-Length` now fails fast at the byte-accounting layer instead of silently succeeding until the ssri integrity check fails with a less-specific diagnostic. Requires adding `reqwest`'s `stream` feature (for `bytes_stream`) and pulling `futures-util` into the tarball crate (for `StreamExt::next`). Both are already in the workspace; this just scopes them to the tarball crate. Items 4 (post-download concurrency cap) and 5 (hardware SHA-512) from #280 are deliberately deferred: (4) wants Apple Silicon before/after numbers since the current cap was picked for a Linux CI failure, (5) wants a macOS profile to confirm SHA-512 is actually in the hot path before a crypto-crate swap. * style(cargo): reflow taplo-aligned columns Taplo recomputes column widths across the file when anything else in the block changes length. The 7ddb561 edit added `stream` to reqwest's feature list (longer line than the previous max in Cargo.toml) and `futures-util` to pacquet-tarball (a new longest name in that block), so taplo tightens the comment padding on `sha2` and widens the trailing padding on the surrounding deps. Ran `just fmt`; no logic changes. * fix(network,tarball): OOM-safe buffer alloc; rename constructor Three review fixes for #281: 1. **OOM protection on `Content-Length`.** Replaced the infallible `Vec::with_capacity(size as usize)` with a new helper, `allocate_tarball_buffer`, that guards the header as untrusted input. Two layers: * `usize::try_from(size)` — a `u64` CL may exceed `usize::MAX` on 32-bit targets. * `Vec::try_reserve_exact(capacity)` — refuses the allocation gracefully when memory pressure or an absurd claim would otherwise abort the process via the infallible path. Both failures surface as a new `TarballError::TarballTooLarge` variant so the install can reject the one offending package and continue instead of OOM-killing the whole process. 2. **Rename `ThrottledClient::new_from_cpu_count` → `new_for_installs`.** The previous commit dropped CPU-count sizing in favor of a fixed 50-socket cap matching pnpm's `DEFAULT_MAX_SOCKETS`, so the old name described a behavior the method no longer has. `new_for_installs` reads as "the default client tuned for pacquet install traffic", which is what it actually returns. Callers updated: `package-manager`'s `install_package_from_registry`, `cli::state`, and the `micro-benchmark` task. 3. **Dropped `BadTarballSize` variant and its checks.** Reviewer asked for a test exercising the variant; on closer look the checks can't actually fire behind reqwest/hyper, which enforces `Content-Length` framing on the receive side itself (a body shorter than CL errors the stream before our code sees a chunk sequence ending early; a body longer gets truncated or queued as the next request on a keep-alive connection). Pnpm's equivalent `BadTarballError` exists because undici in Node.js doesn't always enforce this. Keeping the variant as dead defense-in-depth on our side would just mean an unreachable branch and an untestable error path, so remove both. Adds three unit tests for `allocate_tarball_buffer` covering the chunked-transfer fallback, reasonable pre-sizing, and the `u64::MAX`-rejection case. The rejection test exercises both guards depending on target word size: 64-bit targets pass `try_from` and fail on `try_reserve_exact`; 32-bit targets fail on `try_from`. Either way the observable result is `TarballTooLarge`. |
||
|
Zoltan Kochan
|
99a790100c |
perf(store-dir): port pnpm's checkPkgFilesIntegrity model for warm-cache lookup (#271)
* perf(store-dir): port pnpm's checkPkgFilesIntegrity model for the warm-cache lookup Ports pnpm v11's [`store/cafs/src/checkPkgFilesIntegrity.ts`]( |
||
|
Zoltan Kochan
|
e19163c07d |
refactor(store-dir): batch store-index writes via a single writer task (#265)
* refactor(store-dir): batch store-index writes via a single writer task Port pnpm v11's `queueWrites` / `flush` / `setRawMany` pattern from `store/index/src/index.ts` so the per-tarball store-index INSERT stops opening a fresh SQLite connection and pulling its own blocking-pool thread. * Add `StoreIndex::set_many`, which wraps a batch in `BEGIN IMMEDIATE; INSERT OR REPLACE x N; COMMIT;`. One WAL fsync amortizes across the batch, with a rollback on per-batch error so a partial apply never leaves the index in a half-written state. Encoding happens on the caller's thread before the transaction begins, so the writer only pays SQLite work. * Introduce `StoreIndexWriter`, an `Arc`-cloneable handle around a tokio unbounded channel. Producers call `queue(key, value)` — no SQLite, no `spawn_blocking`, no per-row lock. A single blocking task drains the channel, collects each non-blocking burst into a batch (cap 256), and flushes via `set_many`. * `CreateVirtualStore` and `InstallWithoutLockfile` spawn the writer at the top of the run, thread the `Arc<StoreIndexWriter>` through `InstallPackageBySnapshot` / `InstallPackageFromRegistry` / `DownloadTarballToStore`, drop the orchestration's clone after `try_join_all`, then `await` the writer task so the final batch flushes before the install returns. * `DownloadTarballToStore::run_without_mem_cache` drops its per-tarball `spawn_blocking` for the index write entirely. Batch errors, `JoinError`s, and writer-open failures are all downgraded to `warn!` and "skip the row", matching the read side's graceful-degradation stance from #261. Tests pass against the existing `pacquet-registry-mock` harness. Context: opened while investigating #263. On macOS I haven't been able to demonstrate a wall-time win from this change yet — the measurable bottleneck in pacquet's cold-install sys time sits elsewhere (most likely tarball extraction / `write_cas_file` and sha512 per file), see the comment thread on #263 for the investigation notes and sample profile. The change still stands as structural cleanup and the pnpm-model alignment, and it will compound with whatever fix the real bottleneck gets. * refactor(store-dir): skip per-row encode failures in `set_many` `collect::<Result<_, _>>()?` turned a single `encode_package_files_index` failure into "drop the whole batch" — up to 256 otherwise-valid rows lost because of one malformed `PackageFilesIndex`. That's stricter than the "best-effort index" contract the writer task and the read-side already follow. Encode in a loop, log the failing key at `warn!`, and keep the rest of the batch alive for commit. SQLite errors inside the transaction still roll back the whole batch (that's how atomicity is supposed to work); only the per-row encoding step is skipped. Addresses Copilot review on #265. * refactor(store-dir,tarball): reword #263 ref; quiet `queue` warn on first failure; fix private-doc-link Three follow-up nits from the Copilot review on #265: * The comment in `tarball/src/lib.rs` described #263 as "fixed" — it isn't, this PR is one step toward fixing it. Reworded to "the per-write blocking thread churn described in #263" so the citation survives the landing of both. * `StoreIndexWriter::queue` logged a `warn!` on every `send` failure. When the writer task exits early (open failure, panic) every subsequent `queue` call fails — on a 1352-snapshot install that's a thousand identical warnings drowning everything else out. Add a `warn_on_send_failure: AtomicBool` one-shot guard: first failure logs (with "further failures silenced" hint), subsequent failures go silent. The observable signal the user actually cares about — "this install's rows aren't in the index, the next install will re-download" — surfaces on the next run regardless. * `StoreIndexWriter`'s doc block linked to the private constant `MAX_BATCH_SIZE`. `RUSTDOCFLAGS=-D warnings` in CI rejects the private-intra-doc-link lint, so replace the link with a literal "256 entries" plus a `see MAX_BATCH_SIZE` pointer in prose. Addresses Copilot review on #265. |
||
|
Zoltan Kochan
|
4fa08d5495 |
bench(integrated-benchmark): level cold-install comparison on macOS and across platforms (#264)
* bench(integrated-benchmark): pin storeDir in pnpm-workspace.yaml The fixture's `.npmrc` set `store-dir=<bench-dir>/store-dir`, but pacquet reads `.npmrc` for auth/registry only — project-structural settings like `storeDir` must come from `pnpm-workspace.yaml` (see `crates/npmrc/src/lib.rs:173-178`, matching pnpm 11). With no `storeDir` in the yaml the benchmark silently fell through to pacquet's default (`~/Library/pnpm/store` on macOS, `~/.local/share/pnpm/store` on Linux), so `hyperfine --prepare`'s `rm -rf <bench-dir>/store-dir` never actually wiped the store the install read from. On a developer machine whose default store already holds the fixture's 1352 snapshots — any prior `pnpm install` of a similar project — every bench iteration hit the warm-store cache-lookup path instead of the cold-install path. The measured wall time was dominated by the per-file `symlink_metadata` pass of #260, not by the fetch+extract+write pipeline the benchmark was designed to stress. Adding `storeDir: ./store-dir` makes the bench honour the prepared per-revision store. On CI this is a no-op (the default `~/.local/share/pnpm/store` is empty anyway on a fresh runner) — the fix prevents developer-local runs from producing meaningless numbers. * bench(integrated-benchmark): silence pnpm fsevents build warning on macOS `fsevents` is a macOS-only optional dependency of `chokidar` (via several fixture entries). On Linux CI it isn't installed and pnpm never complains; on a developer macOS machine it installs and pnpm emits `ERR_PNPM_IGNORED_BUILDS: fsevents@1.2.13` because the fixture's `.npmrc` sets `ignore-scripts=true`. That error exits pnpm with status 1, which hyperfine flags via the benchmark harness's exit-code check and takes the whole `just integrated-benchmark` run down on the first warmup. Adding `fsevents: false` to `allowBuilds` silences the warning specifically for this package without re-enabling script execution elsewhere. Matches what pnpm emits when you run `pnpm approve-builds` and decline fsevents. * bench(integrated-benchmark): force pnpm to install cross-platform optionals Pacquet doesn't filter optional dependencies by `os` / `cpu` / `libc` at the moment — `create_virtual_store` iterates every snapshot in the lockfile unconditionally, so `fsevents` (and every other platform-specific optional) ends up in the install payload regardless of runner. pnpm does filter, so on Linux CI it skips `fsevents` and friends entirely. That's a real payload difference: pnpm's install does strictly less tarball fetching / extraction / CAS writing than pacquet's on the same lockfile. Small in absolute terms for this fixture, but a hidden handicap in a benchmark whose whole point is to measure the fetch+extract+write pipeline. Set `supportedArchitectures` to cover every OS / CPU / libc combo we care about so pnpm pulls cross-platform optionals on every runner, matching pacquet's payload. The fix belongs on the pnpm side of the fixture because pacquet's filter-by-platform is a future code change with larger blast radius; bringing the two tools to the same install set in one-line config here is the lower-risk intermediate. |
||
|
Zoltan Kochan
|
0ba3ea0e4b |
bench(integrated-benchmark): replace 26-snapshot fixture with 1352-snapshot one (#262)
* bench(integrated-benchmark): replace 26-snapshot fixture with 1352-snapshot one
The previous `alotta-modules` fixture had 7 direct deps and 26
snapshots, small enough that a frozen-lockfile install completed in
~50 ms. At that scale the wall-time is dominated by per-process
overhead (binary startup, lockfile parse), so scaling wins in the
install pipeline — shared SQLite connections, import-method changes,
symlink-pass parallelism — don't show up.
Swap in a realistic 1352-snapshot lockfile of `alotta-modules` (389 KB
vs 13 KB) so the benchmark reflects a workload where the install
pipeline actually matters. First CI run will need to repopulate the
cached verdaccio storage; subsequent runs are served from the existing
cache step.
* bench(integrated-benchmark): add pnpm-workspace.yaml to the fixture
`allowBuilds: {core-js: false, es5-ext: false}` silences pnpm v11's
`ERR_PNPM_IGNORED_BUILDS` warnings for two packages whose postinstalls
would fire in this 1352-snapshot fixture. Pacquet's benchmark-side
`.npmrc` already sets `ignore-scripts=true` so nothing actually runs,
but pnpm still prints to stderr about the ignored builds and that
noise can reach hyperfine.
Stage the new file the same way `package.json` is staged — embedded
via `include_str!` for the default fixture, copied from `--fixture-dir`
when set. The `--fixture-dir` copy is gated on the file's existence
so existing fixture directories that predate this change don't start
erroring out.
|
||
|
Zoltan Kochan
|
6544b3ace5 |
perf(store-dir): share one read-only StoreIndex across cache lookups (#261)
* perf(store-dir): share one read-only StoreIndex across cache lookups `DownloadTarballToStore::load_cached_cas_paths` previously opened a fresh SQLite `Connection::open_with_flags` + `PRAGMA busy_timeout` on every snapshot, so a 1352-package frozen-lockfile install paid 1352 connection-opens just to read cache hints — all serialized on tokio's blocking pool because `spawn_blocking` can't parallelize opens against one file. Open a `StoreIndex` once per install, wrap it in `Arc<Mutex<…>>` (rusqlite `Connection` is `Send` but not `Sync`), and thread the handle through `CreateVirtualStore`, `InstallWithoutLockfile`, and `DownloadTarballToStore`. Each lookup now takes the mutex just for the sub-millisecond `SELECT`, releases it before the per-file validation pass, and frees the Arc when the install finishes. Measured on the #260 repro (1352 packages, release build, macOS APFS, warm CAS store): `rm -rf node_modules && pacquet install --frozen- lockfile` drops from ~17.4 s to ~13.9 s, with CPU utilization rising from ~135 % to ~160 % as the serialized-open bottleneck clears. * address review feedback * `create_virtual_store.rs`: bind `store_index.as_ref()` to a local and use `async move` so the closure's copy of the `Option<&…>` is moved into each future, rather than borrowing from the closure's per-iteration scope. * `load_cached_cas_paths`: treat a poisoned index mutex as a cache miss rather than propagating the panic. The `SELECT` is stateless, so the prior panic can't have left the index in an inconsistent shape, and cache lookups are a best-effort hint — falling through to a fresh download is strictly safer than crashing every subsequent snapshot. * park StoreIndex open on the blocking pool Copilot flagged that `StoreIndex::shared_readonly_in` does synchronous SQLite work (`Connection::open_with_flags` + `PRAGMA busy_timeout`) and calling it directly in an `async fn` stalls the tokio reactor thread for the duration of the open. It's once per install, not a hot spot, but matches the pattern the store-index writer already uses and costs us nothing to keep consistent. Wrap both call sites (frozen-lockfile and no-lockfile install paths) in `tokio::task::spawn_blocking` and await the result before kicking off concurrent package work. Also moves `tokio` from `dev-dependencies` into `dependencies` in package-manager's `Cargo.toml` since it's now a runtime dep for the library code, not just tests. * degrade JoinError on StoreIndex open to a cache miss `spawn_blocking(...).await` returns `Result<_, JoinError>` which fires on blocking-task panic or on cancellation during runtime shutdown. Panicking on that turns a transient runtime issue into a hard crash for an install that could otherwise succeed — cache lookups just miss. Drop the `.expect(...)` and degrade to `None` via `.ok().flatten()`. `shared_readonly_in` already yields `None` for a first-time install against an empty store, and every downstream caller already handles that shape, so this is the same failure mode the code already copes with — just reached from a different source. * use async move in install_dependencies_from_registry map closure Makes the recursive no-lockfile closure's captures explicit: bind `&node_modules_path` to a local `node_modules_path_ref` (Copy) and switch to `async move` so every captured variable is moved (copied, for all the `&T` and `Option<&T>` captures) into the future instead of borrowing the closure's per-iteration stack frame. Matches the pattern already used in `create_virtual_store.rs` and in the outer no-lockfile closure — keeps all three parallel-map sites consistent. * log JoinError before degrading to cache miss on StoreIndex open The silent `.ok().flatten()` hid two failure modes behind a cache miss: a panic inside the blocking task, and task cancellation during runtime shutdown. Neither is a crash-worthy error, but a blocking-task panic during install really should be visible somewhere. Promote the two call sites to a `match` and emit a `tracing::warn!` on the `Err(JoinError)` branch before falling back to `None`. Uses the `pacquet::install` target in common with the rest of this crate's tracing. Degradation path is unchanged. * chore: cargo fmt * log JoinError in tarball cache lookup, refresh stale comment Mirror the tracing change from the index-open call sites: when the blocking cache-lookup task returns `Err(JoinError)`, emit a `tracing::warn!` with the error and the `cache_key` before degrading to `None`. Panic / cancellation stays diagnosable; the caller still falls through to a fresh download. Also drop the stale "msgpackr-records decoding is a follow-up" wording from the cache-lookup rationale — `StoreIndex::get` has decoded pnpm-written msgpackr-records since |
||
|
Zoltan Kochan
|
f30b8cca11 |
ci(benchmark): compare against pnpm in Integrated-Benchmark (#253)
* ci(benchmark): compare against pnpm in Integrated-Benchmark Adds `--with-pnpm` to the `Integrated-Benchmark` workflow's hyperfine run, so every PR's results include a `pnpm` row alongside `pacquet@HEAD` / `pacquet@main`. Makes regressions vs. upstream obvious in the PR comment. While wiring that up, noticed that `WorkEnv::benchmark` built the `rm -rf` prepare-command from only `revision_ids()` — if the caller set `with_pnpm`, pnpm's bench dir would survive between iterations and the timed runs after the warmup would hit an already-installed `node_modules` instead of actually reinstalling. Fix that: include the `pnpm` bench dir in the cleanup set when `with_pnpm` is on, so pnpm and pacquet are measured under the same cold-`node_modules` conditions. Pnpm's global store stays warm across iterations, same as pacquet's — both tools get a fair repeat-install benchmark. Supersedes the 2023-era #177, which had the same intent but has bit-rotted beyond trivial rebasing (the store-dir schema, the benchmark task layout, the `just` harness, and the pnpm RC version all moved under it). * ci(benchmark): fix mislabelled BENCHMARK_REPORT copy filename In the commented-out Clean Install block, the JSON copy target was `BENCHMARK_REPORT_FROZEN_LOCKFILE.json` — if that block ever gets un-commented, it would overwrite the Frozen Lockfile report instead of producing a separate Clean Install one. Should be `BENCHMARK_REPORT_CLEAN_INSTALL.json` to match the markdown copy line and the summary section below it. |
||
|
Zoltan Kochan
|
0b0abf8189 |
feat!(store): migrate from v3 JSON index to v11 SQLite+msgpack (#244) (#247)
* feat!(store): migrate from v3 JSON index to v11 SQLite+msgpack layout Pacquet now writes the same on-disk store that pnpm v11 uses, so running pacquet and pnpm on the same machine can share the tarball cache instead of maintaining parallel stores. Closes half of #244. - **Layout:** \`<store>/v11/files/XX/<hex-remainder>[-exec]\` for content-addressed blobs + \`<store>/v11/index.db\` for the per-package file index. The legacy \`v3/files/XX/<hex>-index.json\` JSON layout is gone (we don't need backward compat — pacquet is pre-1.0). - **SQLite schema / PRAGMAs:** byte-identical to pnpm's \`store/index/src/index.ts\` — \`package_index (key TEXT PRIMARY KEY, data BLOB NOT NULL) WITHOUT ROWID\` with WAL journal, 5s busy_timeout, 512 MB mmap, 32 MB page cache, temp_store=MEMORY, wal_autocheckpoint=10000. Two concurrent pacquet tasks (or a pacquet + pnpm pair) contend correctly via SQLite's own locking. - **Value encoding:** msgpack via \`rmp-serde::to_vec_named\` so structs are serialized with named fields, matching pnpm's msgpackr record encoding. \`PackageFilesIndex\` and \`CafsFileInfo\` types mirror pnpm's \`@pnpm/store.cafs\` shape field-for-field (including the hex \`digest\` rather than pacquet's old SSRI \`integrity\` string). - **Key format:** \`"{integrity}\\t{pkg_id}"\` via \`store_index_key()\`, same as pnpm's \`storeIndexKey\`. \`pkg_id\` is \`"{name}@{version}"\` — pulled from \`PkgNameVerPeer::without_peer()\` on the frozen-lockfile path and assembled from \`PackageVersion\` on the registry path. Breaking API changes inside the workspace: - \`pacquet-store-dir\` loses the v3 JSON types (\`PackageFilesIndex\`, \`PackageFileInfo\`, \`WriteIndexFileError\`) and \`StoreDir::v3()\` / \`::write_index_file()\`. In their place: \`StoreIndex\`, \`StoreIndexError\`, the pnpm-shape \`PackageFilesIndex\` + \`CafsFileInfo\`, and \`store_index_key()\`. Unit-tested with 9 new round-trip / re-open / key-format tests. - \`DownloadTarballToStore\` gains a \`package_id\` field. All three callers (two in \`pacquet-package-manager\`, one in the micro-benchmark) pass \`"{name}@{version}"\`. - \`pacquet-testing-utils::add_mocked_registry\` now also writes \`pnpm-workspace.yaml\` with \`storeDir\` / \`cacheDir\` so pnpm 11 (which moved that config out of \`.npmrc\`) respects the test's store override. - \`crates/cli/tests/_utils.rs::index_file_contents\` now walks \`StoreIndex::keys()\` and returns \`BTreeMap<sqlite_key, BTreeMap<filename, CafsFileInfo>>\`. The insta snapshots will need re-accept once the test run against the new mock layout settles. - Dropped the \`#[ignore]\` on \`pnpm_compatibility::{same_file_structure, same_index_file_contents}\` — the v11 cutover is the prerequisite they were waiting on. The install-path snapshots in \`install.rs\` will likely need a re-accept through \`cargo insta test --accept\`; running them requires \`just registry-mock launch\` and is tracked at the top of \`crates/cli/tests/install.rs\`. Dependencies added at the workspace level: \`rusqlite = "0.38"\` (with the \`bundled\` feature so we don't need a system libsqlite3) and \`rmp-serde = "1.3\"\`. Dropped the unused \`base64\` dep from \`tarball\`. Verified end-to-end: \`pnpm install --lockfile-only\` against \`@pnpm/registry-mock\` followed by \`pacquet install --frozen-lockfile\` produces \`store-dir/v11/files/XX/…\` + \`store-dir/v11/index.db\` with 50 rows whose keys match pnpm's \`{integrity}\\t{pkgId}\` format (verified via sqlite3 CLI against the pacquet-written db). Follow-ups (tracked in #244 and this PR's description): - Teach pacquet's config loader to read \`pnpm-workspace.yaml\` in addition to \`.npmrc\` so \`storeDir\` / \`cacheDir\` / \`registry\` overrides resolve without requiring the .npmrc mirror. - Read the index at install-time to skip re-fetching already-cached tarballs. - Decode msgpackr \`useRecords\` entries so pacquet can read back records that pnpm itself wrote (pnpm reads pacquet's named-field writes fine today; the reverse direction is the missing piece). * chore(deny): allow Zlib license for foldhash (transitive via rusqlite) rusqlite pulls in foldhash 0.2, which is Zlib-licensed. Zlib is OSI-approved and widely accepted; add it to the allow list. * test: re-accept v11 store snapshots + scope pnpm_compatibility tests Aftershocks from the v11 store migration: - Three insta snapshots in crates/cli/tests/snapshots/install__*.snap referenced the old \`v3/files/.../*-index.json\` paths and had to be re-accepted via \`cargo insta test --accept\` to reflect the new \`v11/files/XX/<hex>[-exec]\` + \`v11/index.db\` layout. Inner file integrities are unchanged; only the store-dir path prefix differs. - \`pnpm_compatibility::same_file_structure\` now filters out three pnpm-11-specific artifacts that pacquet has no equivalent for, but which don't affect the shared-CAFS invariant the test is there to verify: * \`v11/projects/<hash>\` — pnpm-11 per-project metadata. * \`v11/links/<name>/<version>/<hash>/node_modules/...\` — pnpm 11's hoisted-symlinks layout; pacquet uses its own \`.pnpm/\`-based virtual store. * \`v11/index.db-{wal,shm}\` — SQLite WAL sidecars whose presence at compare-time depends on whether the checkpoint ran. With that filter the two tools' CAFS file listings match byte-for- byte. - \`pnpm_compatibility::same_index_file_contents\` goes back under \`#[ignore]\` with an updated reason. Pacquet-written \`index.db\` values round-trip fine (we serialize via \`rmp-serde::to_vec_named\`, string-keyed maps), but pnpm writes with msgpackr's \`useRecords: true\` extension-typed records that rmp-serde can't decode. Handling those is the last piece of #244 (msgpackr-record decoding for the read side) and will be addressed after this PR lands. * review: address Copilot feedback on the v11 store migration - \`TarballError::WriteStoreIndex\` display message was still "Failed to write tarball index" (a leftover from the per-package JSON era). Rephrased to "Failed to write store index (SQLite index)". - \`entry.header().size().unwrap_or(0)\` silently turned tar header parse errors into 0-byte entries in the index. Propagate the error via \`TarballError::ReadTarballEntries\` instead. - Open + PRAGMA + \`INSERT\` on the SQLite index are blocking (and can stall up to \`busy_timeout=5000\` ms under contention). Moved the write step into \`tokio::task::spawn_blocking\` so it runs on the blocking pool instead of the tokio reactor. The tarball-extraction loop (which holds a \`!Send\` \`tar::Archive\`) is now scoped into a sub-block so it's dropped before the new \`await\`. - Added \`StoreIndex::open_readonly\` / \`open_readonly_in\` that use \`SQLITE_OPEN_READ_ONLY\` and skip the schema-creation PRAGMAs. \`crates/cli/tests/_utils.rs::index_file_contents\` now uses it so the snapshot helper doesn't mutate the store or create WAL / SHM sidecar files during what is supposed to be a read. - Clarified the \`StoreIndex\` module-level doc comment: the DB lives at \`<store-root>/v11/index.db\` when opened via \`open_in\`, not at \`<store_dir>/index.db\` as the old wording suggested. |
||
|
Zoltan Kochan
|
e7005635bd |
chore(deps): bump all dependencies, clear the advisory ignore list (#245)
* chore(deps): bump all dependencies to latest and clear advisory ignores Bumps all workspace deps to the latest compatible versions including the major-version jumps. The `deny.toml` RUSTSEC ignore list that #243 had to carry is now empty — every advisory is resolved by the upstream version bumps rather than by ignore-listing. **Major version bumps (API-breaking):** - `base64` 0.21 → 0.22 - `dashmap` 5.5 → 6.1 - `derive_more` 1.0.0-beta.6 → 2.1.1 (stabilized) - `insta` 1.34 → 1.47 - `itertools` 0.11 → 0.14 - `miette` 5.9 → 7.6 - `reqwest` 0.11 → 0.13 - `split-first-char` 0.0.0 → 2.0.1 - `strum` 0.25 → 0.28 - `sysinfo` 0.29 → 0.38 - `text-block-macros` 0.1 → 0.2 - `which` 4.4 → 8.0 - `criterion` 0.5 → 0.8 **Not bumped:** `sha2` stays on 0.10.9 — 0.11 replaced `GenericArray` with a new `Array<_, _>` type that doesn't impl `LowerHex`, so `format!("{hash:x}")` in `store-dir/src/cas_file.rs` breaks. Revisit once the ecosystem catches up. **Code fallout from the bumps:** - `sysinfo` dropped its `*Ext` traits (methods are now inherent) and renamed `RefreshKind::new()` → `RefreshKind::nothing()` / `ProcessRefreshKind::new()` → `ProcessRefreshKind::nothing()`. Dropped `PidExt` / `ProcessExt` / `SystemExt` from imports in `tasks/registry-mock/src/{kill_verdaccio,mock_instance,registry_info,registry_anchor}.rs`. - `derive_more` 2 fully separates the derive macro from the `std::fmt::Display` trait; unqualified `Display` in trait bounds resolved to the derive. Switched to fully-qualified `std::fmt::Display` in `pkg_name_suffix.rs`'s bounds. - Newer ICU crates (pulled in transitively via `idna`) are licensed `Unicode-3.0` instead of `Unicode-DFS-2016`. Allow the new SPDX id in `deny.toml`. - Dropped two now-unused `std::env` imports that the bumps surfaced. **Verification (Rust 1.95 on macOS):** - `cargo build --workspace` — clean - `cargo clippy --locked --workspace --all-targets -- --deny warnings` — clean - `cargo fmt --all -- --check` — clean - `cargo deny check` — `advisories ok, bans ok, licenses ok, sources ok` - All unit tests on the rewritten crates green (the two pre-existing `pacquet-npmrc` env-pollution failures are identical to main). - End-to-end: `pnpm install --lockfile-only` against `@pnpm/registry-mock` then `pacquet install --frozen-lockfile` still produces the expected `node_modules` + `.pnpm/` layout. * ci: bump pnpm action-setup to v4 and pin pnpm to 11 The Integrated-Benchmark workflow was failing because it pinned pnpm 8 via `pnpm/action-setup@v2`. pnpm 8 writes and reads lockfile v6 only, so the v9 fixture bundled in `tasks/integrated-benchmark` was rejected with "Ignoring not compatible lockfile" followed by ERR_PNPM_NO_LOCKFILE during the proxy-cache population step. - `pnpm/action-setup@v2` → `@v4` across `ci.yml`, `codecov.yml`, `integrated-benchmark.yml`. - `version: 8` / `version: 8.9.2` → `version: 11`. - `actions/cache@v3` → `@v4` in the same three workflows (v3 is already EOL on GitHub-hosted runners). - Cache `path:` from `${PNPM_HOME}/store/v3` to `/store/v11` (pnpm 11 writes to v11 subdir, not v3). - Bust the three pnpm cache keys (`ci-pnpm-` → `ci-pnpm-v11-`, `codecov-pnpm-` → `codecov-pnpm-v11-`, `integrated-benchmark-pnpm` → `integrated-benchmark-pnpm-v11`) to avoid restoring v3 content into the new v11 path. * ci: pin pnpm to 11.0.0-rc.5 (stable 11 not released yet) * ci: bump pnpm/action-setup v4 -> v6 * ci: regenerate tasks/registry-mock/pnpm-lock.yaml as v9 for pnpm 11 * ci: approve esbuild + ignore core-js builds for pnpm 11 strict mode pnpm 11 turns unapproved build scripts into a hard `ERR_PNPM_IGNORED_BUILDS` failure. Mark esbuild (needs its postinstall to fetch the binary) as approved via `pnpm.onlyBuiltDependencies`, and opt-out core-js's ad postinstall via `pnpm.ignoredBuiltDependencies`. * ci: opt out core-js postinstall via pnpm-workspace.yaml allowBuilds Revert the package.json `pnpm.{onlyBuiltDependencies,ignoredBuiltDependencies}` I put in the previous commit — in pnpm 11 this configuration moved to `pnpm-workspace.yaml` as `allowBuilds: { <pkg>: false }`. * chore: apply taplo format + add pre-push hook for fmt checks - The long `sha2` inline comment in `Cargo.toml` bumped the alignment past its column; `taplo format` re-indents it. `cargo fmt` is already green. - Add `.githooks/pre-push` that runs `cargo fmt --all -- --check` and `taplo format --check` before push, so fmt violations don't sneak onto CI. `just install-hooks` wires the repo to it via `git config core.hooksPath .githooks`, and `just init` calls it so fresh clones pick the hook up without extra steps. * review: fix typo, codecov cache key, document pnpm RC pin - \`tasks/registry-mock/src/registry_anchor.rs\`: doc-comment typo "is spawn" → "is spawned". - \`.github/workflows/codecov.yml\`: the \`coverage\` job has no matrix, so \`matrix.os\` was empty in the cache key. Switch to \`runner.os\`. - All three workflows: add an inline comment next to \`version: 11.0.0-rc.5\` noting that stable pnpm 11 isn't released yet and that the pin should move to \`11\` once it ships. * test: update install snapshots + ignore pnpm-compat tests broken by pnpm 11 - \`@pnpm/registry-mock\` bumped 3.16 → 3.50 in my lockfile regen, which repackaged the \`@pnpm.e2e/*\` tarballs. The inner file integrities are unchanged, but the tarball-level sha512 (which drives the index-file location under \`v3/files/XX/…-index.json\`) shifted. Accept the three install.rs insta snapshots so they reflect the new mock tarballs. - \`pnpm_compatibility::{same_file_structure, same_index_file_contents}\` compared pacquet's store layout to pnpm's. Under pnpm 11 both assumptions behind the test are broken: 1. pnpm 11 writes to \`store/v11/…\` with a SQLite \`index.db\`, not the v3 \`files/XX/…-index.json\` layout pacquet still uses. 2. pnpm 11 moved \`store-dir\` config out of \`.npmrc\` into \`pnpm-workspace.yaml\`, so the test's current \`.npmrc\`-based override is silently ignored and pnpm uses the global store. Both are resolvable only by pacquet adopting the v11 store format — tracked in #244. Mark both as \`#[ignore]\` with a pointer to #244; they are expected to un-ignore once pacquet's store migrates. |
||
|
Zoltan Kochan
|
aef67087e9 |
feat!: pnpm lockfile v9 support + install error propagation + Rust 1.95 (#243)
* refactor(install): propagate errors through the Install stack
Replace the three `unimplemented!()` panics in the install dispatcher
with proper miette diagnostics, and thread `Result` propagation through
`Install`, `InstallWithoutLockfile`, `InstallFrozenLockfile`, and
`CreateVirtualStore`.
New `InstallError::NoLockfile` (code `pacquet_package_manager::no_lockfile`)
is returned when `--frozen-lockfile` is requested but no `pnpm-lock.yaml`
exists, mirroring pnpm's `ERR_PNPM_NO_LOCKFILE`. The two remaining
writable-lockfile paths surface as `InstallError::UnsupportedLockfileMode`
until lockfile writing lands.
Downstream `.unwrap()`s on `InstallPackageFromRegistryError` and
`InstallPackageBySnapshotError` now become `?`, and the `join_all` calls
switch to `try_join_all` to fail fast.
* feat(lockfile): add Lockfile::save_to_path and save_to_current_dir
Introduces the write side of the lockfile API paired with the existing
`load_lockfile` module. Emits YAML via `serde_yaml::to_string` and
propagates serde/io failures through a new `SaveLockfileError`
(codes `pacquet_lockfile::serialize_yaml`, `pacquet_lockfile::write_file`,
`pacquet_lockfile::current_dir`).
The output is semantically equivalent to the input (parse → save → reparse
yields an equal `Lockfile`), but is NOT yet byte-identical with pnpm's
`sortLockfileKeys`-based output — matching the exact key order and YAML
style pnpm produces is a separate piece and will come with the install
path that consumes this API.
* feat(package-manager): add build_snapshot helpers for PackageVersion -> PackageSnapshot
Introduces `registry_dependency_path` and `build_package_snapshot` in a
new `build_snapshot` module. Given a resolved `PackageVersion` plus an
injected map of resolved sub-dependencies and dev/optional flags, they
produce the `(DependencyPath, PackageSnapshot)` pair that can be dropped
into a `pnpm-lock.yaml`'s `packages` map.
The helpers are pure and unit-tested with synthetic fixtures — wiring
them into `InstallWithoutLockfile` to collect live resolution metadata
and construct the full `Lockfile` is the next step.
Errors propagate via `BuildSnapshotError` (`missing_integrity`,
`parse_name`) rather than panicking when input is malformed.
* feat!(lockfile): replace v6 types with v9 (packages + snapshots + importers)
pnpm v11 writes lockfile v9 exclusively. Pacquet's existing v6 types
cannot read these files. Per-session agreement, no backward
compatibility with v6 is kept.
Type rewrite in `pacquet-lockfile`:
- `Lockfile` now carries `importers: HashMap<String, ProjectSnapshot>`
(with `Lockfile::ROOT_IMPORTER_KEY = "."`) instead of a flattened
`RootProjectSnapshot`, plus separate `packages: HashMap<PackageKey,
PackageMetadata>` and `snapshots: HashMap<PackageKey, SnapshotEntry>`
maps keyed by the v9 `name@version[(peers)]` specifier (no leading
slash).
- `PackageMetadata` (new) holds per-version data (resolution, engines,
cpu/os/libc, deprecated, hasBin, peer metadata). `SnapshotEntry`
(new) holds per-instance data (dependencies, optional_dependencies,
transitive_peer_dependencies, id, patched).
- `PkgNameVerPeer::without_peer` strips the peer suffix so consumers
can look up a snapshot entry's metadata in the `packages:` map.
- Deleted: `dependency_path.rs`, `multi_project_snapshot.rs`,
`package_snapshot.rs`, `package_snapshot_dependency.rs`,
`root_project_snapshot.rs`.
- `LockfileVersion<9>` with the v6 compat test swapped for v9.
Consumer updates in `pacquet-package-manager`:
- `InstallFrozenLockfile` now takes `importers`, `packages`, and
`snapshots` separately.
- `CreateVirtualStore` iterates `snapshots` and resolves each
snapshot's metadata via `PackageKey::without_peer()`, returning
`MissingPackageMetadata` if the lockfile is inconsistent.
- `InstallPackageBySnapshot` takes `(package_key, metadata, snapshot)`
and reads resolution from `PackageMetadata`.
- `CreateVirtualDirBySnapshot` derives the virtual store dir from the
`PackageKey` and reads dependency wiring from the `SnapshotEntry`.
- `SymlinkDirectDependencies` looks up the root project via
`importers[Lockfile::ROOT_IMPORTER_KEY]` instead of pattern-matching
`RootProjectSnapshot::Single`.
- `create_symlink_layout` takes `HashMap<PkgName, PkgVerPeer>` (the v9
snapshot deps shape) instead of the legacy enum.
- `build_snapshot` returns `BuiltSnapshot { package_key, metadata,
snapshot }` matching the v9 decomposition.
Fixture updates:
- `BIG_LOCKFILE` regenerated as v9 via pnpm v11.
- Save-lockfile round-trip fixture rewritten in v9 shape.
- Save-lockfile error fixture uses `lockfileVersion: '9.0'`.
Verified end-to-end: `pnpm install --lockfile-only` generates a v9
lockfile against `@pnpm/registry-mock`, then `pacquet install
--frozen-lockfile` populates node_modules from that lockfile (50
packages, symlinks wired).
Benchmark on the pacquet fixture manifest against the registry mock
(hyperfine, 10 runs + 2 warmup):
- pacquet: 487.9 ms ± 106.6 ms (range 343-656 ms)
- pnpm@11: 739.6 ms ± 9.1 ms (range 725-751 ms)
- pacquet is 1.52x faster; worst-case pacquet still beats best-case pnpm.
* style(install): apply cargo fmt
* chore(deny): rewrite deny.toml for cargo-deny 0.19 schema
The old template predated cargo-deny 0.14. The schema has since moved
the graph/output keys under dedicated sections and changed advisory
level fields. Rewritten to follow the 0.19 shape.
Also adds a scoped ignore list for the RUSTSEC advisories that exist
in the already-pinned transitive deps (bytes, h2, idna, mio, openssl,
tar, tokio, tracing-subscriber) — resolving those requires bumping the
1.73.0 toolchain and a round of coordinated dep updates that is out of
scope for this PR.
* chore(toolchain): bump rust-toolchain.toml to 1.95.0 + fix CI fallout
Bumps the pinned Rust channel from 1.73.0 to 1.95.0 and addresses the
consequences:
- `tasks/registry-mock/src/registry_anchor.rs`: migrate off the
`advisory-lock` crate to Rust 1.89+'s stable `std::fs::File::{lock,
try_lock}` (the crate's trait methods now collide with std's). Drop
the workspace dep.
- New rustc/clippy lints under 1.95:
- `mismatched_lifetime_syntaxes` → add explicit `'_` to
`StoreDir::display`.
- `clippy::map_clone` → `.cloned()` in `PackageManifest::bundle_dependencies`.
- `clippy::manual_pattern_char_comparison` → array pattern in
`PkgVerPeer` parser.
- `clippy::suspicious_open_options` → add `.truncate(false)` to the
persistent lock file in `GuardFile::path`.
- `clippy::useless_into_iter_in_iterator_adapter` →
`InstallWithoutLockfile::run` dependencies call.
- `clippy::manual_unwrap_or` → collapse `if let Some` match in the
`pacquet start` fallback.
- `crates/cli/tests/install.rs`: move `BIG_LOCKFILE` / `BIG_MANIFEST` /
`OpenOptions` / `io::Write` imports behind the same
`cfg(not(macos|windows))` gate as their one consumer so they aren't
flagged as unused on macOS.
Also unblocks the Micro-Benchmark / Code Coverage / release-to-npm
workflows: bump `actions/upload-artifact@v3` → `@v4` (v3 is now
hard-rejected by GitHub Actions).
`cargo build --workspace`, `cargo clippy --locked --workspace
--all-targets -- --deny warnings`, `cargo fmt --all -- --check`, and
`cargo deny check` all pass locally on 1.95.
* chore(ci): bump actions/download-artifact to v4 (v3 is hard-rejected)
* test(cli): disable flaky big-lockfile test on all CI platforms
The test drives the mocked verdaccio with hundreds of concurrent tarball
fetches and reliably trips a connection error on every hosted-runner OS:
ConnectionAborted on Windows, ConnectionReset on macOS, and now
ConnectionClosed on Ubuntu (the regenerated v9 BIG_LOCKFILE has more
packages, so Ubuntu started hitting the same class of issue the other
platforms already gated out).
Align the cfg gate to exclude all three and document the manual run
recipe. The BIG_LOCKFILE / BIG_MANIFEST / OpenOptions / io::Write
imports follow the same gate.
* test(integrated-benchmark): regenerate lockfile fixture as v9
The fixture was v6 and consequently unparseable by HEAD after the v9
rewrite, which failed the Integrated-Benchmark CI run in Benchmark 1
(pacquet@HEAD) before it could even start hyperfine.
With the fixture at v9 the benchmark can at least evaluate pacquet@HEAD
standalone. The HEAD-vs-main hyperfine comparison that this workflow is
designed for is intrinsically broken by this PR (main still speaks v6
only, so it cannot read a v9 fixture); that failure is an expected
consequence of the breaking format change and will resolve once this
PR lands on main.
* review: replace remaining panics/silent-failures with proper diagnostics
Addresses Copilot review feedback on the install path. Every code path
that previously used `panic!` / `expect` / silent `return` now either
surfaces a structured error or — in the test-gating case — uses
`#[ignore]` so the test remains runnable locally.
- `InstallPackageBySnapshotError`: new `MissingTarballIntegrity` and
`UnsupportedResolution` variants; `install_package_by_snapshot` no
longer panics when a tarball resolution lacks an integrity field or
when the lockfile references Directory/Git resolutions that pacquet
does not yet handle.
- `CreateVirtualStoreError`: new `MissingPackagesSection` variant for
the case where a lockfile has `snapshots:` but no `packages:`. The
dispatcher no longer panics in that case.
- `SymlinkDirectDependenciesError`: new `MissingRootImporter` variant.
A lockfile with no `importers."."` entry now produces a clear
diagnostic instead of silently producing an incomplete install.
`InstallFrozenLockfile` gains a transparent variant that wraps it.
- `BuildSnapshotError::ParseVersion`: `registry_package_key` now
propagates `ParsePkgVerPeerError` instead of `.expect`-ing that
`node_semver::Version::to_string` always yields a valid `PkgVerPeer`.
The two grammars agree today but the assumption is no longer encoded
as a panic.
- `frozen_lockfile_should_be_able_to_handle_big_lockfile`: replace the
overly-broad `cfg(not(any(windows, macos, linux)))` gate with
`#[ignore = "flaky on CI…"]`. The test is now runnable locally via
`cargo test -- --ignored` per the updated comment.
- Drive-by: drop `.write(true)` next to `.append(true)` flagged by
`clippy::ineffective_open_options` after the import un-gating above.
Manual verification: `pnpm install --lockfile-only` against
`@pnpm/registry-mock` followed by `pacquet install --frozen-lockfile`
still produces the expected `node_modules` + `.pnpm/` layout.
|
||
|
Khải
|
8657d3aa65 | refactor: divide pacquet-fs into multiple files (#214) | ||
|
Khải
|
9d80419b31 |
feat(network): limit concurrent network requests (#210)
сlose #116 |
||
|
Khải
|
362607a51c |
refactor: reuse registry-mock in benchmark (#201)
* refactor: reuse registry-mock in benchmark * refactor: move executor to verify * feat(benchmark): install dependencies * ci(benchmark): fix * ci(benchmark): use just * refactor: reduce lines |
||
|
Khải
|
7ad33aa65b |
test: registry-mock (#198)
* test: fix snapshot * test: filter out index files temporarily * chore: install registry-mock * chore(just): add pnpm to test * feat: create registry-mock crate * feat(registry-mock): expose get_or_init instead * refactor: local static * feat(testing-utils): add_mocked_registry * refactor: rename new to init * fix: enable_all * refactor: use * feat: add ending / * fix: must be "localhost" * feat(registry-mock): use portpicker * feat(registry-mock): share a single instance * feat(testing-utils): stop skipping index files * test(cli/install): use registry-mock * fix: forgot to save * fix: forgot increment * fix: used to wrong syntax * feat(registry-mock): use advisory-lock * fix: save anchor before early exit * fix: drop before delete * tmp: stored * refactor: remove unused import * fix: truncate the file * feat: use try_lock * feat(registry-mock): separate lock from anchor * refactor: use RAII pattern * fix(registry-mock): user_count * refactor: rename user_count to ref_count * fix: kill groups * fix(registry-mock): ref_count * chore(git): revert broken fix This reverts commit 8c0ba7bafd7670dd6580a142ccd7dea5809ed782. * fix(registry-mock): use sysinfo to kill * fix(registry-mock): kill algorithm * feat(registry-mock): recurse regardless * feat(registry-mock): use reverse recursion instead * refactor: reduce complexity * refactor: remove unnecessary mod declarations * refactor: marks some struct as must_use * test(cli/add): use registry-mock * test(pnpm-compatibility): use registry-mock * refactor: remove unused items * docs: correction * chore(license): allow MPL-2.0 * chore(license): allow Unlicense * fix(registry-mock): failure on few threads * chore: move pnpm install to just * ci: cache pnpm * fix(test): kill leftovers * fix(test): allow delete to fail * fix(ci): Install just * fix(test): don't kill leftovers This reverts commit ef78d8c54cee691d93fa8e5c169141a0098047f2. * refactor: move some types to their own mods * refactor: move port_to_url * refactor: make listen lazy * refactor: rename `listen` to `url` * refactor: use qualified path * refactor: store port as u16 * feat(registry-mock): PreparedRegistryInfo * refactor: remove unnecessary async * refactor: move init code * refactor: define lib * feat(registry-mock): bin * refactor: call just test * ci: prepare a server * feat(registry-mock): unique prepared * docs(registry-mock): cli * feat(registry-mock/cli): locations of log files * docs(mocked-registry): important items * docs(registry-mock): ignore bin * style: taplo fmt * docs(readme): instruction to run test * refactor: move registry-mock to tasks * ci(codecov): fix * feat(registry-mock): bump max_retries |
||
|
Khải
|
b0d5549210 |
feat(benchmark): allow overriding lockfile (#196)
* feat(benchmark): allow overriding lockfile * test: fix snapshot * test: fix snapshot * test: filter out index files temporarily * test: fix snapshot |
||
|
Khải
|
672e4003e6 |
refactor: only use cache when required (#182)
* refactor: only use cache when required * refactor: remove unnecessary Arc * refactor: clarify "in-memory" part of the cache |
||
|
Khải
|
65d1a9ab95 |
refactor: parse integrity eagerly (#180)
* refactor: parse integrity eagerly The parsing of integrity has been moved from tarball to serde * docs: clarify * docs: remove the 3rd option * refactor: remove unnecessary `pipe_ref` |
||
|
Khải
|
222bc2ce1b |
feat: store dir structure compatible with pnpm (#166)
Resolves https://github.com/pnpm/pacquet/issues/165 **Other changes:** * A new crate named `pacquet-store-dir` has been created. * The field `store-dir` in `Npmrc` has been changed from `PathBuf` to `StoreDir`. * The function `write_sync` in `pacquet-cafs` has been replaced by 2 functions ( |
||
|
Khải
|
ca272026d5 | feat(fs): add executable bits to existing mode (#170) | ||
|
Khải
|
558040da5c |
feat(fs): create make_file_executable (#168)
* feat(fs): make_file_executable * refactor: re-use make_file_executable |
||
|
Khải
|
8d6bbf491b |
ci: integrated-benchmark (#159)
## Limitations * **Statistical outliers:** Caused by other jobs still running in parallel. The current workaround is to re-run the job. To completely fix this however, this benchmark must be set to be after all other jobs have been completed, which requires touching other workflows. * **No windows:** Executing this on Windows CI is just too hard: cannot use powershell because of execution policy, and cannot use bash because it points to WSL which causes error. * **No macOS:** The benchmark values are too chaotic and unreliable. * **No clean install yet:** It's too slow. |
||
|
Khải
|
0db25f85a2 |
refactor: re-organize code (#143)
## Summary * Move code that handles package managing to its own crate named `pacquet_package_manager`. * Convert functions with multiple arguments into structs. * Make `pacquet add` reuses the algorithm from `pacquet install`. * Achieved by having `pacquet add` creates an in-memory `package.json` and pass it to the `Install` subroutine. * Convert the error-prone tests in `crates/cli` into proper integration tests that invoke the `pacquet` binary. * Tests become shorter and more readable. * Solve race condition issues with `cargo test`. * All code with `set_current_dir` have been replaced by either integration tests or dependency injection. * Solve race condition issues with `cargo test`. * `thiserror` has been completely removed. `derive_more` is used in its stead. * `thiserror` cannot handle generic trait bound. * `derive_more` is already used to generate custom string types in the lockfile. * one less macro crate leads to slightly faster compile time. * `pacquet_package_json::PackageJson` has been renamed to `pacquet_package_manifest::PackageManifest`. * Other minor changes. |
||
|
Khải
|
eb5a9b1ce4 |
feat(benchmark): rename the benchmarks from {revision} and PNPM to pacquet@{revision} and pnpm (#156)
|
||
|
Khải
|
5580797911 |
refactor: reuse pid variable (#157)
|
||
|
Khải
|
cbb0dbaa9b | feat(benchmark): auto launching verdaccio (#155) |