Replace the two-table design (metadata_index + metadata_blobs) with a
single metadata table storing etag, modified, cached_at, is_full flag,
and the raw JSON blob.
The separate index table added complexity without meaningful benefit —
we parse the full blob anyway to extract the resolved version's manifest
after picking. The single table keeps writes simple (1 INSERT with the
raw registry response) and reads simple (1 SELECT + JSON.parse).
The is_full flag ensures that abbreviated-only cache entries are not
served when full metadata is requested (e.g., for optional dependencies).
Replace metadata_manifests (per-version rows requiring JSON.stringify
per manifest) with metadata_blobs (single raw JSON blob per package).
Write path: store the raw registry response text as-is — zero
serialization on the hot path. Only the compact index fields
(dist-tags, version keys, deprecated flags) are extracted.
Read path: parse the lightweight index for version picking, then
parse the blob and extract just the resolved version's manifest.
This eliminates the cold install regression caused by hundreds of
JSON.stringify calls per install. The index table still provides
cheap header lookups for conditional requests.
Also tracks is_full flag on the index to avoid serving abbreviated
metadata when full is requested (e.g., for optional dependencies).
Split the single-blob metadata storage into two tables:
- metadata_index: dist-tags, version keys (with deprecated), time, and
cache headers — one row per package, ~10KB
- metadata_manifests: per-version manifest objects, keyed by (name,
version, type) — ~2KB each
During resolution, only the lightweight index is parsed to pick a
version. The full manifest for the resolved version is loaded separately.
For a package like typescript with 200+ versions, this avoids parsing
~400KB of unused manifest JSON.
The index is shared across abbreviated/full metadata types — only the
per-version manifests differ. This eliminates the type column from the
index and simplifies the abbreviated→full fallback to the manifest level.
encode-registry replaced : with + for filesystem compatibility. With
SQLite, colons are fine in DB keys. Use URL.host directly with a simple
Map cache for hot-path performance.
- Set meta.modified from DB row in loadMetaFromDb so If-Modified-Since
headers are sent even when modified comes from the DB column
- Use || instead of && for header lookup so both etag and modified are
populated even when one is already available from in-memory metadata
- Check pending writes in MetadataCache.get/getHeaders instead of
flushing in loadMetaFromDb, avoiding synchronous DB writes during
resolution
- Close MetadataCache when store controller is closed to prevent
resource leaks
- Add defensive slash-index guards in cache API functions
Rewrite cache list, delete, view, and list-registries commands to query
the MetadataCache SQLite DB instead of globbing JSON files on disk.
The cache.cmd handler no longer needs resolutionMode/registrySupportsTimeField
to determine which metadata subdirectory to use — all metadata types are
in the same DB.
* refactor: rename workspace.sort-packages and workspace.pkgs-graph
- workspace.sort-packages -> workspace.projects-sorter
- workspace.pkgs-graph -> workspace.projects-graph
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: rename packages/ to core/ and pkg-manifest.read-package-json to reader
- Rename packages/ directory to core/ for clarity
- Rename pkg-manifest/read-package-json to pkg-manifest/reader (@pnpm/pkg-manifest.reader)
- Update all tsconfig, package.json, and lockfile references
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: consolidate runtime resolvers under engine/runtime domain
- Remove unused @pnpm/engine.runtime.node.fetcher package
- Rename engine/runtime/node.resolver to node-resolver (dash convention)
- Move resolving/bun-resolver to engine/runtime/bun-resolver
- Move resolving/deno-resolver to engine/runtime/deno-resolver
- Update all package names, tsconfig paths, and lockfile references
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: update lockfile after removing node.fetcher
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: sort tsconfig references and package.json deps alphabetically
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: auto-fix import sorting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: update __typings__ paths in tsconfig.lint.json for moved resolvers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove deno-resolver from deps of bun-resolver
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add n/prefer-node-protocol rule and autofix all bare builtin imports
to use the node: prefix. Simplify the simple-import-sort builtins
pattern to just ^node: since all imports now use the prefix.
Add eslint-plugin-simple-import-sort to enforce consistent import ordering:
- Node.js builtins first
- External packages second
- Relative imports last
- Named imports sorted alphabetically within each statement
* feat: use JSON for npm registry metadata cache instead of msgpack
Switch the on-disk package metadata cache from msgpack (.mpk) to JSON (.json).
When metadata is not filtered, the raw JSON response from the registry is written
directly to disk with cachedAt injected, avoiding a parse-then-serialize round-trip.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: update lockfileOnly test to use .json metadata extension
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Update resolving/npm-resolver/src/pickPackage.ts
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
## Summary
Replace individual `.mpk` (MessagePack) files under `$STORE/index/` with a single SQLite database at `$STORE/index.db` using Node.js 22's built-in `node:sqlite` module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries.
Closes#10826
## Design
### New package: `@pnpm/store.index`
A new `StoreIndex` class wraps a SQLite database with a simple key-value API (`get`, `set`, `delete`, `has`, `entries`). Data is serialized with msgpackr and stored as BLOBs. The table uses `WITHOUT ROWID` for compact storage.
Key design decisions:
- **WAL mode** enables concurrent reads from workers while the main process writes.
- **`busy_timeout=5000`** plus a retry loop with `Atomics.wait`-based `sleepSync` handles `SQLITE_BUSY` errors from concurrent access.
- **Performance PRAGMAs**: `synchronous=NORMAL`, `mmap_size=512MB`, `cache_size=32MB`, `temp_store=MEMORY`, `wal_autocheckpoint=10000`.
- **Write batching**: `queueWrites()` batches pre-packed entries from tarball extraction and flushes them in a single transaction on `process.nextTick`. `setRawMany()` writes immediate batches (e.g. from `addFilesFromDir`).
- **Lifecycle**: `close()` auto-flushes pending writes, runs `PRAGMA optimize`, and closes the DB. A `process.on('exit')` handler ensures cleanup even on unexpected exits.
- **`VACUUM` after `deleteMany`** (used by `pnpm store prune`) to reclaim disk space.
### Key format
Keys are `integrity\tpkgId` (tab-separated). Git-hosted packages use `pkgId\tbuilt` or `pkgId\tnot-built`.
### Shared StoreIndex instance
A single `StoreIndex` instance is threaded through the entire install lifecycle — from `createNewStoreController` through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access.
### Worker architecture
Index writes are performed in the main process, not in worker threads. Workers send pre-packed `{ key, buffer }` pairs back to the main process via `postMessage`, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads.
### SQLite ExperimentalWarning suppression
`node:sqlite` emits an `ExperimentalWarning` on first load. This is suppressed via a `process.emitWarning` override injected through esbuild's `banner` option, which runs on line 1 of both `dist/pnpm.mjs` and `dist/worker.js` — before any module that loads `node:sqlite`.
### No migration from `.mpk` files
Old `.mpk` index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store).
## Changed packages
121 files changed across these areas:
- **`store/index/`** — New `@pnpm/store.index` package
- **`worker/`** — Write batching moved from worker module into `StoreIndex` class; workers send pre-packed buffers to main process
- **`store/package-store/`** — StoreIndex creation and lifecycle management
- **`store/cafs/`** — Removed `getFilePathInCafs` index-file utilities (no longer needed)
- **`store/pkg-finder/`** — Reads from StoreIndex instead of `.mpk` files
- **`store/plugin-commands-store/`** — `store status` uses StoreIndex
- **`store/plugin-commands-store-inspecting/`** — `cat-index` and `find-hash` use StoreIndex
- **`fetching/tarball-fetcher/`** — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading
- **`fetching/git-fetcher/`, `binary-fetcher/`, `pick-fetcher/`** — Accept StoreIndex parameter
- **`pkg-manager/`** — `client`, `core`, `headless`, `package-requester` thread StoreIndex
- **`reviewing/`** — `license-scanner`, `sbom`, `dependencies-hierarchy` accept StoreIndex
- **`cache/api/`** — Cache view uses StoreIndex
- **`pnpm/bundle.ts`** — esbuild banner for ExperimentalWarning suppression
## Test plan
- [x] `pnpm --filter @pnpm/store.index test` — Unit tests for StoreIndex CRUD and batching
- [x] `pnpm --filter @pnpm/package-store test` — Store controller lifecycle
- [x] `pnpm --filter @pnpm/package-requester test` — Package requester reads from SQLite index
- [x] `pnpm --filter @pnpm/tarball-fetcher test` — Tarball and git-hosted fetcher writes
- [x] `pnpm --filter @pnpm/headless test` — Headless install
- [x] `pnpm --filter @pnpm/core test` — Core install, side effects, patching
- [x] `pnpm --filter @pnpm/plugin-commands-rebuild test` — Rebuild reads from index
- [x] `pnpm --filter @pnpm/license-scanner test` — License scanning
- [x] e2e tests pass
🤖 Generated with [Claude Code](https://claude.com/claude-code)
* chore: `package.json` add type field
* chore: add type field to every package.json
* chore: add type field to every package.json
---------
Co-authored-by: Zoltan Kochan <z@kochan.io>
