mirror of
https://github.com/pnpm/pnpm.git
synced 2026-03-27 03:21:55 -04:00
b7f0f21582
## Summary Replace individual `.mpk` (MessagePack) files under `$STORE/index/` with a single SQLite database at `$STORE/index.db` using Node.js 22's built-in `node:sqlite` module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries. Closes #10826 ## Design ### New package: `@pnpm/store.index` A new `StoreIndex` class wraps a SQLite database with a simple key-value API (`get`, `set`, `delete`, `has`, `entries`). Data is serialized with msgpackr and stored as BLOBs. The table uses `WITHOUT ROWID` for compact storage. Key design decisions: - **WAL mode** enables concurrent reads from workers while the main process writes. - **`busy_timeout=5000`** plus a retry loop with `Atomics.wait`-based `sleepSync` handles `SQLITE_BUSY` errors from concurrent access. - **Performance PRAGMAs**: `synchronous=NORMAL`, `mmap_size=512MB`, `cache_size=32MB`, `temp_store=MEMORY`, `wal_autocheckpoint=10000`. - **Write batching**: `queueWrites()` batches pre-packed entries from tarball extraction and flushes them in a single transaction on `process.nextTick`. `setRawMany()` writes immediate batches (e.g. from `addFilesFromDir`). - **Lifecycle**: `close()` auto-flushes pending writes, runs `PRAGMA optimize`, and closes the DB. A `process.on('exit')` handler ensures cleanup even on unexpected exits. - **`VACUUM` after `deleteMany`** (used by `pnpm store prune`) to reclaim disk space. ### Key format Keys are `integrity\tpkgId` (tab-separated). Git-hosted packages use `pkgId\tbuilt` or `pkgId\tnot-built`. ### Shared StoreIndex instance A single `StoreIndex` instance is threaded through the entire install lifecycle — from `createNewStoreController` through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access. ### Worker architecture Index writes are performed in the main process, not in worker threads. Workers send pre-packed `{ key, buffer }` pairs back to the main process via `postMessage`, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads. ### SQLite ExperimentalWarning suppression `node:sqlite` emits an `ExperimentalWarning` on first load. This is suppressed via a `process.emitWarning` override injected through esbuild's `banner` option, which runs on line 1 of both `dist/pnpm.mjs` and `dist/worker.js` — before any module that loads `node:sqlite`. ### No migration from `.mpk` files Old `.mpk` index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store). ## Changed packages 121 files changed across these areas: - **`store/index/`** — New `@pnpm/store.index` package - **`worker/`** — Write batching moved from worker module into `StoreIndex` class; workers send pre-packed buffers to main process - **`store/package-store/`** — StoreIndex creation and lifecycle management - **`store/cafs/`** — Removed `getFilePathInCafs` index-file utilities (no longer needed) - **`store/pkg-finder/`** — Reads from StoreIndex instead of `.mpk` files - **`store/plugin-commands-store/`** — `store status` uses StoreIndex - **`store/plugin-commands-store-inspecting/`** — `cat-index` and `find-hash` use StoreIndex - **`fetching/tarball-fetcher/`** — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading - **`fetching/git-fetcher/`, `binary-fetcher/`, `pick-fetcher/`** — Accept StoreIndex parameter - **`pkg-manager/`** — `client`, `core`, `headless`, `package-requester` thread StoreIndex - **`reviewing/`** — `license-scanner`, `sbom`, `dependencies-hierarchy` accept StoreIndex - **`cache/api/`** — Cache view uses StoreIndex - **`pnpm/bundle.ts`** — esbuild banner for ExperimentalWarning suppression ## Test plan - [x] `pnpm --filter @pnpm/store.index test` — Unit tests for StoreIndex CRUD and batching - [x] `pnpm --filter @pnpm/package-store test` — Store controller lifecycle - [x] `pnpm --filter @pnpm/package-requester test` — Package requester reads from SQLite index - [x] `pnpm --filter @pnpm/tarball-fetcher test` — Tarball and git-hosted fetcher writes - [x] `pnpm --filter @pnpm/headless test` — Headless install - [x] `pnpm --filter @pnpm/core test` — Core install, side effects, patching - [x] `pnpm --filter @pnpm/plugin-commands-rebuild test` — Rebuild reads from index - [x] `pnpm --filter @pnpm/license-scanner test` — License scanning - [x] e2e tests pass 🤖 Generated with [Claude Code](https://claude.com/claude-code)
68 lines
1.8 KiB
JSON
68 lines
1.8 KiB
JSON
{
|
|
"name": "@pnpm/worker",
|
|
"version": "1000.3.0",
|
|
"description": "A worker for extracting package taralls to the store",
|
|
"keywords": [
|
|
"pnpm",
|
|
"pnpm11",
|
|
"tarball"
|
|
],
|
|
"license": "MIT",
|
|
"funding": "https://opencollective.com/pnpm",
|
|
"repository": "https://github.com/pnpm/pnpm/tree/main/worker",
|
|
"homepage": "https://github.com/pnpm/pnpm/tree/main/worker#readme",
|
|
"bugs": {
|
|
"url": "https://github.com/pnpm/pnpm/issues"
|
|
},
|
|
"type": "module",
|
|
"main": "lib/index.js",
|
|
"types": "lib/index.d.ts",
|
|
"exports": {
|
|
".": "./lib/index.js",
|
|
"./start": "./lib/start.js"
|
|
},
|
|
"files": [
|
|
"lib",
|
|
"!*.map"
|
|
],
|
|
"scripts": {
|
|
"lint": "eslint \"src/**/*.ts\" \"test/**/*.ts\"",
|
|
"prepublishOnly": "pnpm run compile",
|
|
"test": "pnpm run compile && pnpm run _test",
|
|
"compile": "tsgo --build && pnpm run lint --fix",
|
|
"_test": "cross-env NODE_OPTIONS=\"$NODE_OPTIONS --experimental-vm-modules\" jest"
|
|
},
|
|
"dependencies": {
|
|
"@pnpm/cafs-types": "workspace:*",
|
|
"@pnpm/create-cafs-store": "workspace:*",
|
|
"@pnpm/crypto.integrity": "workspace:*",
|
|
"@pnpm/error": "workspace:*",
|
|
"@pnpm/exec.pkg-requires-build": "workspace:*",
|
|
"@pnpm/fs.hard-link-dir": "workspace:*",
|
|
"@pnpm/graceful-fs": "workspace:*",
|
|
"@pnpm/store.cafs": "workspace:*",
|
|
"@pnpm/store.index": "workspace:*",
|
|
"@pnpm/symlink-dependency": "workspace:*",
|
|
"@rushstack/worker-pool": "catalog:",
|
|
"is-windows": "catalog:",
|
|
"p-limit": "catalog:",
|
|
"semver": "catalog:"
|
|
},
|
|
"peerDependencies": {
|
|
"@pnpm/logger": "catalog:"
|
|
},
|
|
"devDependencies": {
|
|
"@pnpm/logger": "workspace:*",
|
|
"@pnpm/types": "workspace:*",
|
|
"@pnpm/worker": "workspace:*",
|
|
"@types/is-windows": "catalog:",
|
|
"@types/semver": "catalog:"
|
|
},
|
|
"engines": {
|
|
"node": ">=22.13"
|
|
},
|
|
"jest": {
|
|
"preset": "@pnpm/jest-config"
|
|
}
|
|
}
|