For git-hosted tarballs (`codeload.github.com` / `gitlab.com` / `bitbucket.org`) the fetcher dropped the integrity it computed while downloading, so the lockfile only ever stored the URL. A compromised git host or man-in-the-middle could serve a substituted tarball on subsequent installs and pnpm would install it — the lockfile had no hash to compare against.
This pins the SHA-512 SRI of the raw tarball in the lockfile, in the same `sha512-<base64>` form npm-registry tarballs use. The only difference is the source: for npm we pass through `dist.integrity`, for git we compute it locally from the downloaded buffer. Subsequent installs validate the download against that integrity in the worker (`addTarballToStore` → `parseIntegrity` → hash compare), so a tampered tarball fails with `TarballIntegrityError`.
## Why git-hosted stays on `gitHostedStoreIndexKey`
The lockfile pins integrity for security, but the *store key* for git-hosted resolutions stays on `gitHostedStoreIndexKey(pkgId, { built })` rather than collapsing under the integrity-based key. Reason: git-hosted tarballs are post-processed (`preparePackage` / `packlist`), so the cached file set depends on whether build scripts ran during fetch. The integrity-only key would fold the built and not-built variants into a single slot, letting one overwrite the other and serving the wrong content if `ignoreScripts` was toggled between runs. Keeping git-hosted on the existing key shape preserves that dimension; the integrity is still validated on every fresh download.
## How the routing stays clean
The naive way to express "use gitHostedStoreIndexKey for git-hosted, integrity key for npm" is to call `isGitHostedPkgUrl(resolution.tarball)` everywhere a store key is computed — fragile, scattered, and easy to forget when adding new readers (Copilot caught two of those during review). Instead, a typed annotation: `TarballResolution` gets an optional `gitHosted: boolean` field. The git resolver sets it; the lockfile loader (`convertToLockfileObject`) backfills it for entries written by older pnpm versions; `toLockfileResolution` carries it through on serialize. Every consumer reads `resolution.gitHosted` directly. URL detection lives in exactly two places — the resolver and the loader — instead of seven.
## Changes
### Security fix
- `fetching/tarball-fetcher/src/gitHostedTarballFetcher.ts` — return the `integrity` that the inner remote-tarball fetch already computed (was being silently dropped by the destructure).
### Lockfile schema (additive)
- `@pnpm/lockfile.types` and `@pnpm/resolving.resolver-base` — `TarballResolution` gains optional `gitHosted: boolean`.
- `@pnpm/resolving.git-resolver` — sets `gitHosted: true` on every git-hosted tarball it produces.
- `@pnpm/lockfile.fs` (`convertToLockfileObject`) — backfills the field on load for older lockfiles via inlined URL detection.
- `@pnpm/lockfile.utils` (`toLockfileResolution`, `pkgSnapshotToResolution`) — preserve / read the field.
### Store-key consumers (now one-line typed reads, dropped the URL-sniffing dep)
- `installing/package-requester` (`getFilesIndexFilePath`)
- `store/pkg-finder` (`readPackageFileMap`)
- `modules-mounter/daemon` (`createFuseHandlers`)
- `building/after-install` (side-effects-cache lookup + write)
- `store/commands/storeStatus`
- `installing/deps-installer` (agent-mode store-controller wrapper)
### Fetcher routing
- `fetching/pick-fetcher` — `pickFetcher` prefers `resolution.gitHosted`; URL fallback retained for ad-hoc resolutions.
### Tests
- New integrity-validation test in `tarball-fetcher` (mismatched `integrity` on the resolution must throw `TarballIntegrityError`).
- New git-hosted lookup test in `pkg-finder` asserting routing through `gitHostedStoreIndexKey` even when integrity is present.
- New `toLockfileResolution` test asserting `gitHosted: true` flows through serialization.
- `fromRepo.ts` lockfile snapshot updated for the now-pinned integrity + `gitHosted: true`.
- `git-resolver` tests updated to assert `gitHosted: true` in produced resolutions.
* chore: upgrade @typescript/native-preview to 7.0.0-dev.20260421.2
- Add explicit `types: ["node"]` to the shared tsconfig because tsgo
20260421 no longer auto-acquires `@types/*` from `node_modules`.
- Refactor test files to explicitly import jest globals (`describe`,
`it`, `test`, `expect`, `beforeEach`, etc.) from `@jest/globals`
instead of relying on `@types/jest` ambient declarations. Under the
new tsgo build, `import { jest } from '@jest/globals'` shadows the
ambient `jest` namespace, breaking `@types/jest`'s `declare var
describe: jest.Describe;` globals.
- Add `@jest/globals` to each package's devDependencies where tests
now import from it, and add `@types/node` to packages that need it
but were relying on hoisted resolution.
- Replace `fail()` calls with `throw new Error(...)` since `fail` is
no longer globally available.
* chore: fix remaining tsgo type-strictness errors
- Strip `as <PnpmType>` casts on objects passed to toMatchObject /
toStrictEqual / toEqual; @jest/globals rejects the typed objects
(which include AsymmetricMatchers) vs. the repo-specific type.
- Type `jest.fn<...>()` explicitly where the mock's signature matters
for toHaveBeenCalledWith.
- Replace `beforeEach(() => X)` with `beforeEach(() => { X })` so the
return value is void, as the stricter jest typing requires.
- Use `expect.objectContaining({...})` in one place where the full
expected object triggered stricter type resolution.
- Cast `prompt.mock.calls` arg through `as unknown as Record<...>[]`
for patch.test.ts's nested-array matchers.
- Fix off-by-one `<reference path>` in pnpm/test/getConfig.test.ts
that only surfaced now.
- Move `@jest/globals` from devDependencies to dependencies in the
two `__utils__` packages that import it from `src/`.
- Clean up unused imports from the @jest/globals migration.
* chore: address Copilot review on #11332
- Move misplaced `@jest/globals` imports to the top import block in
checkEngine, run.ts, and workspace/root-finder tests where the
script dropped them below executable code.
- Replace `try { await x(); throw new Error('should have thrown') } catch`
in bins/linker, lockfile/fs, and resolving/local-resolver tests with
`await expect(x()).rejects.toMatchObject({...})`. The old pattern
swallowed an unrelated `throw` if the under-test call silently
succeeded, which would fail on the catch-block assertion with a
misleading message.
Library packages had `prepublishOnly: pn compile`, which expands to
`tsgo --build && pn lint --fix`. During `pn release` that runs eslint
against ~150 packages for no benefit — the code has already been linted
in CI and the release flow's upfront compile has already built dist/.
Switch lib prepublishOnly to a bare `tsgo --build` so the safety-net
compile stays but the per-package eslint cost is gone.
## Summary
Adds an opt-in **pnpm agent** server that resolves dependencies server-side and streams only the files missing from the client's content-addressable store.
- **`@pnpm/agent.server`** — multi-process HTTP server (Node.js `cluster`) with SQLite-backed metadata and file caches
- **`@pnpm/agent.client`** — streams an NDJSON response, dispatches worker threads to fetch files while the server is still resolving
- **New config**: `agent` in `pnpm-workspace.yaml` (opt-in)
## How it works
1. Client reads integrity hashes from its local store index
2. Sends `POST /v1/install` with dependencies + store integrities
3. Server resolves the dependency tree using pnpm's `install({ lockfileOnly: true })`, with a SQLite-backed `PackageMetaCache` for fast repeat resolution
4. As each package resolves, a wrapped `storeController.requestPackage` looks up its files and immediately streams digests the client is missing (NDJSON `D` lines)
5. Client reads the stream line by line; digest batches fill up and dispatch worker threads to `POST /v1/files` — file downloads overlap with server-side resolution
6. After resolution, server sends index entries (`I` lines) and lockfile (`L` line)
7. Client writes index entries to store, then runs headless install with a wrapped `fetchPackage` that calls `readPkgFromCafs` with `verifyStoreIntegrity: false` (files are trusted from the agent)
8. `/v1/files` response is gzip-streamed (274MB → ~80MB) — server pipes through `createGzip`, worker pipes through `createGunzip`, parsing and writing files to CAFS as data arrives
## Performance
1351-package project, cold local store, warm server (localhost):
| Scenario | Time |
|----------|------|
| Vanilla pnpm install (cold OS cache) | ~48s |
| Vanilla pnpm install (warm OS cache) | ~34s |
| With pnpm agent (consistent) | **~33s** |
### Key optimizations
1. **SQLite metadata cache** — server-side resolution drops from ~3.4s to ~0.9s
2. **SQLite file store** — consistent read performance regardless of OS file cache state
3. **Streaming `/v1/install`** — file digests stream during resolution, downloads start before resolution finishes
4. **Gzip-streamed `/v1/files`** — whole-stream gzip (274MB → ~80MB), significant savings on remote servers
5. **Worker-thread streaming HTTP** — workers pipe gzip → parse → write to CAFS as data arrives, no buffering
6. **No rehashing** — server-provided digests used directly, skipping 33K SHA-512 computations
7. **No re-verification** — wrapped `fetchPackage` calls `readPkgFromCafs` with `verifyStoreIntegrity: false`
8. **Direct `writeFileSync` with `wx`** — no stat + temp + rename
9. **Pre-packed msgpack** — server sends raw store index buffers, client writes directly to SQLite
10. **WAL checkpoint** — ensures store index entries written by agent are visible to headless install's worker threads
## Usage
Start the server:
```bash
node agent/server/lib/bin.js
```
Configure in `pnpm-workspace.yaml`:
```yaml
agent: http://localhost:4873
```
* fix: ensure PNPM_HOME/bin is in PATH during pnpm setup
When upgrading from old pnpm (global bin = PNPM_HOME) to new pnpm
(global bin = PNPM_HOME/bin), `pnpm setup` would fail because the
spawned `pnpm add -g` checks that the global bin dir is in PATH.
Prepend PNPM_HOME/bin to PATH in the spawned process env so the
check passes during the transition.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update pnpm to v11 beta 2
* chore: update pnpm to v11 beta 2
* chore: update pnpm to v11 beta 2
* chore: update pnpm to v11 beta 2
* fix: lint
* refactor: rename _-prefixed scripts to .-prefixed scripts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: update root package.json to use .test instead of _test
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* ci: update action-setup
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: reduce noisy warnings in test output
- Suppress ExperimentalWarning and DEP0169 via --disable-warning in NODE_OPTIONS
- Fix MaxListenersExceededWarning by raising limit in StoreIndex when adding exit listeners
- Update meta-updater to generate the new _test scripts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: stop streaming pnpm subprocess output during CLI tests
Buffer stdout/stderr from execPnpm instead of writing to the parent
process in real time. Output is still included in the error message on
failure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: pipe all subprocess output in CLI tests
Use stdio: 'pipe' for all pnpm/pnpx spawn helpers so subprocess output
is buffered instead of printed. Output is still included in error
messages on failure.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove duplicate @pnpm/installing.env-installer in pnpm/package.json
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: use pipe stdio in dlx and errorHandler tests
Replace stdio: 'inherit' and [null, 'pipe', 'inherit'] with 'pipe' to
prevent subprocess output from leaking into test output.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: skip maxListeners adjustment when set to unlimited (0)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add n/prefer-node-protocol rule and autofix all bare builtin imports
to use the node: prefix. Simplify the simple-import-sort builtins
pattern to just ^node: since all imports now use the prefix.
Add eslint-plugin-simple-import-sort to enforce consistent import ordering:
- Node.js builtins first
- External packages second
- Relative imports last
- Named imports sorted alphabetically within each statement
## Summary
Replace individual `.mpk` (MessagePack) files under `$STORE/index/` with a single SQLite database at `$STORE/index.db` using Node.js 22's built-in `node:sqlite` module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries.
Closes#10826
## Design
### New package: `@pnpm/store.index`
A new `StoreIndex` class wraps a SQLite database with a simple key-value API (`get`, `set`, `delete`, `has`, `entries`). Data is serialized with msgpackr and stored as BLOBs. The table uses `WITHOUT ROWID` for compact storage.
Key design decisions:
- **WAL mode** enables concurrent reads from workers while the main process writes.
- **`busy_timeout=5000`** plus a retry loop with `Atomics.wait`-based `sleepSync` handles `SQLITE_BUSY` errors from concurrent access.
- **Performance PRAGMAs**: `synchronous=NORMAL`, `mmap_size=512MB`, `cache_size=32MB`, `temp_store=MEMORY`, `wal_autocheckpoint=10000`.
- **Write batching**: `queueWrites()` batches pre-packed entries from tarball extraction and flushes them in a single transaction on `process.nextTick`. `setRawMany()` writes immediate batches (e.g. from `addFilesFromDir`).
- **Lifecycle**: `close()` auto-flushes pending writes, runs `PRAGMA optimize`, and closes the DB. A `process.on('exit')` handler ensures cleanup even on unexpected exits.
- **`VACUUM` after `deleteMany`** (used by `pnpm store prune`) to reclaim disk space.
### Key format
Keys are `integrity\tpkgId` (tab-separated). Git-hosted packages use `pkgId\tbuilt` or `pkgId\tnot-built`.
### Shared StoreIndex instance
A single `StoreIndex` instance is threaded through the entire install lifecycle — from `createNewStoreController` through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access.
### Worker architecture
Index writes are performed in the main process, not in worker threads. Workers send pre-packed `{ key, buffer }` pairs back to the main process via `postMessage`, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads.
### SQLite ExperimentalWarning suppression
`node:sqlite` emits an `ExperimentalWarning` on first load. This is suppressed via a `process.emitWarning` override injected through esbuild's `banner` option, which runs on line 1 of both `dist/pnpm.mjs` and `dist/worker.js` — before any module that loads `node:sqlite`.
### No migration from `.mpk` files
Old `.mpk` index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store).
## Changed packages
121 files changed across these areas:
- **`store/index/`** — New `@pnpm/store.index` package
- **`worker/`** — Write batching moved from worker module into `StoreIndex` class; workers send pre-packed buffers to main process
- **`store/package-store/`** — StoreIndex creation and lifecycle management
- **`store/cafs/`** — Removed `getFilePathInCafs` index-file utilities (no longer needed)
- **`store/pkg-finder/`** — Reads from StoreIndex instead of `.mpk` files
- **`store/plugin-commands-store/`** — `store status` uses StoreIndex
- **`store/plugin-commands-store-inspecting/`** — `cat-index` and `find-hash` use StoreIndex
- **`fetching/tarball-fetcher/`** — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading
- **`fetching/git-fetcher/`, `binary-fetcher/`, `pick-fetcher/`** — Accept StoreIndex parameter
- **`pkg-manager/`** — `client`, `core`, `headless`, `package-requester` thread StoreIndex
- **`reviewing/`** — `license-scanner`, `sbom`, `dependencies-hierarchy` accept StoreIndex
- **`cache/api/`** — Cache view uses StoreIndex
- **`pnpm/bundle.ts`** — esbuild banner for ExperimentalWarning suppression
## Test plan
- [x] `pnpm --filter @pnpm/store.index test` — Unit tests for StoreIndex CRUD and batching
- [x] `pnpm --filter @pnpm/package-store test` — Store controller lifecycle
- [x] `pnpm --filter @pnpm/package-requester test` — Package requester reads from SQLite index
- [x] `pnpm --filter @pnpm/tarball-fetcher test` — Tarball and git-hosted fetcher writes
- [x] `pnpm --filter @pnpm/headless test` — Headless install
- [x] `pnpm --filter @pnpm/core test` — Core install, side effects, patching
- [x] `pnpm --filter @pnpm/plugin-commands-rebuild test` — Rebuild reads from index
- [x] `pnpm --filter @pnpm/license-scanner test` — License scanning
- [x] e2e tests pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
