mirror of
https://github.com/pnpm/pnpm.git
synced 2026-07-25 23:17:08 -04:00
b7f0f21582
## Summary Replace individual `.mpk` (MessagePack) files under `$STORE/index/` with a single SQLite database at `$STORE/index.db` using Node.js 22's built-in `node:sqlite` module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries. Closes #10826 ## Design ### New package: `@pnpm/store.index` A new `StoreIndex` class wraps a SQLite database with a simple key-value API (`get`, `set`, `delete`, `has`, `entries`). Data is serialized with msgpackr and stored as BLOBs. The table uses `WITHOUT ROWID` for compact storage. Key design decisions: - **WAL mode** enables concurrent reads from workers while the main process writes. - **`busy_timeout=5000`** plus a retry loop with `Atomics.wait`-based `sleepSync` handles `SQLITE_BUSY` errors from concurrent access. - **Performance PRAGMAs**: `synchronous=NORMAL`, `mmap_size=512MB`, `cache_size=32MB`, `temp_store=MEMORY`, `wal_autocheckpoint=10000`. - **Write batching**: `queueWrites()` batches pre-packed entries from tarball extraction and flushes them in a single transaction on `process.nextTick`. `setRawMany()` writes immediate batches (e.g. from `addFilesFromDir`). - **Lifecycle**: `close()` auto-flushes pending writes, runs `PRAGMA optimize`, and closes the DB. A `process.on('exit')` handler ensures cleanup even on unexpected exits. - **`VACUUM` after `deleteMany`** (used by `pnpm store prune`) to reclaim disk space. ### Key format Keys are `integrity\tpkgId` (tab-separated). Git-hosted packages use `pkgId\tbuilt` or `pkgId\tnot-built`. ### Shared StoreIndex instance A single `StoreIndex` instance is threaded through the entire install lifecycle — from `createNewStoreController` through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access. ### Worker architecture Index writes are performed in the main process, not in worker threads. Workers send pre-packed `{ key, buffer }` pairs back to the main process via `postMessage`, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads. ### SQLite ExperimentalWarning suppression `node:sqlite` emits an `ExperimentalWarning` on first load. This is suppressed via a `process.emitWarning` override injected through esbuild's `banner` option, which runs on line 1 of both `dist/pnpm.mjs` and `dist/worker.js` — before any module that loads `node:sqlite`. ### No migration from `.mpk` files Old `.mpk` index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store). ## Changed packages 121 files changed across these areas: - **`store/index/`** — New `@pnpm/store.index` package - **`worker/`** — Write batching moved from worker module into `StoreIndex` class; workers send pre-packed buffers to main process - **`store/package-store/`** — StoreIndex creation and lifecycle management - **`store/cafs/`** — Removed `getFilePathInCafs` index-file utilities (no longer needed) - **`store/pkg-finder/`** — Reads from StoreIndex instead of `.mpk` files - **`store/plugin-commands-store/`** — `store status` uses StoreIndex - **`store/plugin-commands-store-inspecting/`** — `cat-index` and `find-hash` use StoreIndex - **`fetching/tarball-fetcher/`** — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading - **`fetching/git-fetcher/`, `binary-fetcher/`, `pick-fetcher/`** — Accept StoreIndex parameter - **`pkg-manager/`** — `client`, `core`, `headless`, `package-requester` thread StoreIndex - **`reviewing/`** — `license-scanner`, `sbom`, `dependencies-hierarchy` accept StoreIndex - **`cache/api/`** — Cache view uses StoreIndex - **`pnpm/bundle.ts`** — esbuild banner for ExperimentalWarning suppression ## Test plan - [x] `pnpm --filter @pnpm/store.index test` — Unit tests for StoreIndex CRUD and batching - [x] `pnpm --filter @pnpm/package-store test` — Store controller lifecycle - [x] `pnpm --filter @pnpm/package-requester test` — Package requester reads from SQLite index - [x] `pnpm --filter @pnpm/tarball-fetcher test` — Tarball and git-hosted fetcher writes - [x] `pnpm --filter @pnpm/headless test` — Headless install - [x] `pnpm --filter @pnpm/core test` — Core install, side effects, patching - [x] `pnpm --filter @pnpm/plugin-commands-rebuild test` — Rebuild reads from index - [x] `pnpm --filter @pnpm/license-scanner test` — License scanning - [x] e2e tests pass 🤖 Generated with [Claude Code](https://claude.com/claude-code)
345 lines
5.3 KiB
JSON
345 lines
5.3 KiB
JSON
{
|
|
"words": [
|
|
"adduser",
|
|
"adipiscing",
|
|
"amet",
|
|
"andreineculau",
|
|
"appdata",
|
|
"applyq",
|
|
"archy",
|
|
"argumentless",
|
|
"armv",
|
|
"autocheckpoint",
|
|
"autocompleting",
|
|
"autofix",
|
|
"autofixed",
|
|
"autoinstalled",
|
|
"autozoom",
|
|
"babek",
|
|
"badheaders",
|
|
"behaviour",
|
|
"blabla",
|
|
"Bluesky",
|
|
"brasileiro",
|
|
"bryntum",
|
|
"cafile",
|
|
"cafs",
|
|
"camelcase",
|
|
"canonicalizer",
|
|
"canva",
|
|
"cerbos",
|
|
"certfile",
|
|
"clonedeep",
|
|
"cmds",
|
|
"codeload",
|
|
"codenames",
|
|
"codesign",
|
|
"colorterm",
|
|
"comver",
|
|
"copyfiles",
|
|
"corejs",
|
|
"corepack",
|
|
"corge",
|
|
"cowsay",
|
|
"cves",
|
|
"cwsay",
|
|
"cyclonedx",
|
|
"deburr",
|
|
"dedup",
|
|
"denoland",
|
|
"denolib",
|
|
"deptype",
|
|
"devextreme",
|
|
"devowl",
|
|
"dgimuvys",
|
|
"didyoumean",
|
|
"dirtyforms",
|
|
"diskusage",
|
|
"dislink",
|
|
"dpkg",
|
|
"drivelist",
|
|
"duplexify",
|
|
"eagain",
|
|
"ebadplatform",
|
|
"ebusy",
|
|
"eexist",
|
|
"ehrkoext",
|
|
"eintegrity",
|
|
"eisdir",
|
|
"elifecycle",
|
|
"elit",
|
|
"emfile",
|
|
"enametoolong",
|
|
"endregion",
|
|
"eneedauth",
|
|
"enoent",
|
|
"enotempty",
|
|
"enten",
|
|
"eperm",
|
|
"epipe",
|
|
"errcode",
|
|
"etamponi",
|
|
"exdev",
|
|
"execa",
|
|
"exploitability",
|
|
"fakehash",
|
|
"fellback",
|
|
"fetchings",
|
|
"filenamify",
|
|
"filesystem",
|
|
"filesystems",
|
|
"fnumber",
|
|
"foobarqar",
|
|
"foofoo",
|
|
"footgun",
|
|
"forgejo",
|
|
"fsevents",
|
|
"gabor",
|
|
"garply",
|
|
"gcttmf",
|
|
"getattr",
|
|
"ghsa",
|
|
"ghsas",
|
|
"gitea",
|
|
"globalconfig",
|
|
"globstar",
|
|
"grault",
|
|
"gruntfile",
|
|
"gwhitney",
|
|
"haptics",
|
|
"hardlink",
|
|
"hardlinked",
|
|
"hardlinking",
|
|
"hardlinks",
|
|
"hashbang",
|
|
"highmaps",
|
|
"hikljmi",
|
|
"hoistable",
|
|
"homepath",
|
|
"hosters",
|
|
"hyperdrive",
|
|
"idempotency",
|
|
"imurmurhash",
|
|
"ionicons",
|
|
"isexe",
|
|
"istvan",
|
|
"italiano",
|
|
"jega",
|
|
"jhcg",
|
|
"jnbpamcxayl",
|
|
"kebabcase",
|
|
"kevva",
|
|
"keyfile",
|
|
"killcb",
|
|
"kochan",
|
|
"koorchik",
|
|
"ldid",
|
|
"ldni",
|
|
"leniolabs",
|
|
"libc",
|
|
"libnpmpublish",
|
|
"libnpx",
|
|
"libzip",
|
|
"licence",
|
|
"licences",
|
|
"lifecycles",
|
|
"linuxstatic",
|
|
"localappdata",
|
|
"lockfiles",
|
|
"loglevel",
|
|
"logstream",
|
|
"longlink",
|
|
"longpaths",
|
|
"luca",
|
|
"martensson",
|
|
"maxtimeout",
|
|
"mdast",
|
|
"metafile",
|
|
"millis",
|
|
"mintimeout",
|
|
"mmap",
|
|
"monorepolint",
|
|
"moonrepo",
|
|
"mountpoint",
|
|
"msgpack",
|
|
"msgpackr",
|
|
"msvc",
|
|
"msys",
|
|
"mycomp",
|
|
"mycompany",
|
|
"myorg",
|
|
"mypackage",
|
|
"mytoken",
|
|
"ndjson",
|
|
"nerfed",
|
|
"NOASSERTION",
|
|
"nodetouch",
|
|
"noent",
|
|
"nonexec",
|
|
"noninjected",
|
|
"nopadding",
|
|
"noproxy",
|
|
"nosystem",
|
|
"nothrow",
|
|
"npmcli",
|
|
"npmignore",
|
|
"npmjs",
|
|
"ntfs",
|
|
"nushell",
|
|
"ofjergrg",
|
|
"onclickoutside",
|
|
"oomol",
|
|
"ossl",
|
|
"outfile",
|
|
"overrider",
|
|
"packlist",
|
|
"packr",
|
|
"packument",
|
|
"paralleljs",
|
|
"parallelly",
|
|
"parseable",
|
|
"partialmatch",
|
|
"pathext",
|
|
"pegjs",
|
|
"pidtree",
|
|
"pify",
|
|
"pkgname",
|
|
"pkgs",
|
|
"plotly",
|
|
"plugh",
|
|
"pnpmfile",
|
|
"pnpmfiles",
|
|
"pnpmjs",
|
|
"pnpmrc",
|
|
"pnpmtest",
|
|
"polyfilling",
|
|
"português",
|
|
"posix",
|
|
"postbuild",
|
|
"postfoo",
|
|
"postpack",
|
|
"postprepare",
|
|
"postpublish",
|
|
"postrestart",
|
|
"postshrinkwrap",
|
|
"poststart",
|
|
"poststop",
|
|
"posttest",
|
|
"postuninstall",
|
|
"postversion",
|
|
"preact",
|
|
"prefoo",
|
|
"prefs",
|
|
"preinstall",
|
|
"prepublish",
|
|
"prereleases",
|
|
"prerestart",
|
|
"preshrinkwrap",
|
|
"prestart",
|
|
"prestop",
|
|
"preuninstall",
|
|
"preversion",
|
|
"prioritizer",
|
|
"promisified",
|
|
"proxied",
|
|
"pwsh",
|
|
"quux",
|
|
"rcompare",
|
|
"redownload",
|
|
"refclone",
|
|
"reflattened",
|
|
"reflink",
|
|
"reflinked",
|
|
"reflinks",
|
|
"rehoist",
|
|
"reka",
|
|
"relinks",
|
|
"renderable",
|
|
"replit",
|
|
"reqheaders",
|
|
"rimrafed",
|
|
"rmgr",
|
|
"rpmdevtools",
|
|
"rpmlint",
|
|
"rstacruz",
|
|
"rushstack",
|
|
"safecrlf",
|
|
"scopeless",
|
|
"sdiff",
|
|
"searchexclude",
|
|
"searchlimit",
|
|
"searchopts",
|
|
"searchstaleness",
|
|
"sels",
|
|
"semistrict",
|
|
"serp",
|
|
"serverjs",
|
|
"shasums",
|
|
"sheetjs",
|
|
"shlex",
|
|
"sigstore",
|
|
"sindresorhus",
|
|
"sirv",
|
|
"soporan",
|
|
"sopts",
|
|
"spdxdocs",
|
|
"SPDXID",
|
|
"srcset",
|
|
"ssri",
|
|
"stackblitz",
|
|
"stacktracey",
|
|
"stdtype",
|
|
"subdep",
|
|
"subdependencies",
|
|
"subdependency",
|
|
"subdeps",
|
|
"subdir",
|
|
"subdirs",
|
|
"subpkg",
|
|
"subresource",
|
|
"supercede",
|
|
"syml",
|
|
"syncer",
|
|
"syscall",
|
|
"syscalls",
|
|
"szia",
|
|
"tabtab",
|
|
"taffydb",
|
|
"teambit",
|
|
"tempy",
|
|
"testcase",
|
|
"TLSV",
|
|
"todomvc",
|
|
"toplevel",
|
|
"tsgo",
|
|
"tsparticles",
|
|
"typecheck",
|
|
"unallowed",
|
|
"underperformance",
|
|
"undollar",
|
|
"uninstallation",
|
|
"unnest",
|
|
"unreviewed",
|
|
"unskip",
|
|
"unstar",
|
|
"usecase",
|
|
"userconfig",
|
|
"userprofile",
|
|
"ustar",
|
|
"uuidv",
|
|
"valign",
|
|
"vuln",
|
|
"webcontainer",
|
|
"winst",
|
|
"workleap",
|
|
"worktree",
|
|
"worktrees",
|
|
"wrappy",
|
|
"xmarw",
|
|
"yazl",
|
|
"zkochan",
|
|
"zoli",
|
|
"zoltan"
|
|
]
|
|
}
|