Files
pnpm/pkg-manager/plugin-commands-installation/test/fetch.ts
Zoltan Kochan b7f0f21582 feat: use SQLite for storing package index in the content-addressable store (#10827)
## Summary

Replace individual `.mpk` (MessagePack) files under `$STORE/index/` with a single SQLite database at `$STORE/index.db` using Node.js 22's built-in `node:sqlite` module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries.

Closes #10826

## Design

### New package: `@pnpm/store.index`

A new `StoreIndex` class wraps a SQLite database with a simple key-value API (`get`, `set`, `delete`, `has`, `entries`). Data is serialized with msgpackr and stored as BLOBs. The table uses `WITHOUT ROWID` for compact storage.

Key design decisions:

- **WAL mode** enables concurrent reads from workers while the main process writes.
- **`busy_timeout=5000`** plus a retry loop with `Atomics.wait`-based `sleepSync` handles `SQLITE_BUSY` errors from concurrent access.
- **Performance PRAGMAs**: `synchronous=NORMAL`, `mmap_size=512MB`, `cache_size=32MB`, `temp_store=MEMORY`, `wal_autocheckpoint=10000`.
- **Write batching**: `queueWrites()` batches pre-packed entries from tarball extraction and flushes them in a single transaction on `process.nextTick`. `setRawMany()` writes immediate batches (e.g. from `addFilesFromDir`).
- **Lifecycle**: `close()` auto-flushes pending writes, runs `PRAGMA optimize`, and closes the DB. A `process.on('exit')` handler ensures cleanup even on unexpected exits.
- **`VACUUM` after `deleteMany`** (used by `pnpm store prune`) to reclaim disk space.

### Key format

Keys are `integrity\tpkgId` (tab-separated). Git-hosted packages use `pkgId\tbuilt` or `pkgId\tnot-built`.

### Shared StoreIndex instance

A single `StoreIndex` instance is threaded through the entire install lifecycle — from `createNewStoreController` through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access.

### Worker architecture

Index writes are performed in the main process, not in worker threads. Workers send pre-packed `{ key, buffer }` pairs back to the main process via `postMessage`, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads.

### SQLite ExperimentalWarning suppression

`node:sqlite` emits an `ExperimentalWarning` on first load. This is suppressed via a `process.emitWarning` override injected through esbuild's `banner` option, which runs on line 1 of both `dist/pnpm.mjs` and `dist/worker.js` — before any module that loads `node:sqlite`.

### No migration from `.mpk` files

Old `.mpk` index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store).

## Changed packages

121 files changed across these areas:

- **`store/index/`** — New `@pnpm/store.index` package
- **`worker/`** — Write batching moved from worker module into `StoreIndex` class; workers send pre-packed buffers to main process
- **`store/package-store/`** — StoreIndex creation and lifecycle management
- **`store/cafs/`** — Removed `getFilePathInCafs` index-file utilities (no longer needed)
- **`store/pkg-finder/`** — Reads from StoreIndex instead of `.mpk` files
- **`store/plugin-commands-store/`** — `store status` uses StoreIndex
- **`store/plugin-commands-store-inspecting/`** — `cat-index` and `find-hash` use StoreIndex
- **`fetching/tarball-fetcher/`** — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading
- **`fetching/git-fetcher/`, `binary-fetcher/`, `pick-fetcher/`** — Accept StoreIndex parameter
- **`pkg-manager/`** — `client`, `core`, `headless`, `package-requester` thread StoreIndex
- **`reviewing/`** — `license-scanner`, `sbom`, `dependencies-hierarchy` accept StoreIndex
- **`cache/api/`** — Cache view uses StoreIndex
- **`pnpm/bundle.ts`** — esbuild banner for ExperimentalWarning suppression

## Test plan

- [x] `pnpm --filter @pnpm/store.index test` — Unit tests for StoreIndex CRUD and batching
- [x] `pnpm --filter @pnpm/package-store test` — Store controller lifecycle
- [x] `pnpm --filter @pnpm/package-requester test` — Package requester reads from SQLite index
- [x] `pnpm --filter @pnpm/tarball-fetcher test` — Tarball and git-hosted fetcher writes
- [x] `pnpm --filter @pnpm/headless test` — Headless install
- [x] `pnpm --filter @pnpm/core test` — Core install, side effects, patching
- [x] `pnpm --filter @pnpm/plugin-commands-rebuild test` — Rebuild reads from index
- [x] `pnpm --filter @pnpm/license-scanner test` — License scanning
- [x] e2e tests pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-03-06 12:59:04 +01:00

230 lines
6.1 KiB
TypeScript

import fs from 'fs'
import path from 'path'
import { STORE_VERSION } from '@pnpm/constants'
import { install, fetch } from '@pnpm/plugin-commands-installation'
import { prepare } from '@pnpm/prepare'
import { REGISTRY_MOCK_PORT } from '@pnpm/registry-mock'
import { closeAllStoreIndexes } from '@pnpm/store.index'
import { finishWorkers } from '@pnpm/worker'
import { sync as rimraf } from '@zkochan/rimraf'
const REGISTRY_URL = `http://localhost:${REGISTRY_MOCK_PORT}`
const DEFAULT_OPTIONS = {
argv: {
original: [],
},
bail: false,
bin: 'node_modules/.bin',
cliOptions: {},
deployAllFiles: false,
excludeLinksFromLockfile: false,
extraEnv: {},
include: {
dependencies: true,
devDependencies: true,
optionalDependencies: true,
},
lock: true,
preferWorkspacePackages: true,
pnpmfile: ['.pnpmfile.cjs'],
pnpmHomeDir: '',
rawConfig: { registry: REGISTRY_URL },
rawLocalConfig: { registry: REGISTRY_URL },
registries: {
default: REGISTRY_URL,
},
rootProjectManifestDir: '',
sort: true,
userConfig: {},
workspaceConcurrency: 1,
virtualStoreDirMaxLength: process.platform === 'win32' ? 60 : 120,
}
test('fetch dependencies', async () => {
const project = prepare({
dependencies: { 'is-positive': '1.0.0' },
devDependencies: { 'is-negative': '1.0.0' },
})
const storeDir = path.resolve('store')
await install.handler({
...DEFAULT_OPTIONS,
cacheDir: path.resolve('cache'),
dir: process.cwd(),
linkWorkspacePackages: true,
storeDir,
})
rimraf(path.resolve(project.dir(), 'node_modules'))
rimraf(path.resolve(project.dir(), './package.json'))
project.storeHasNot('is-negative')
project.storeHasNot('is-positive')
await fetch.handler({
...DEFAULT_OPTIONS,
cacheDir: path.resolve('cache'),
dir: process.cwd(),
storeDir,
})
project.storeHas('is-positive')
project.storeHas('is-negative')
})
test('fetch production dependencies', async () => {
const project = prepare({
dependencies: { 'is-positive': '1.0.0' },
devDependencies: { 'is-negative': '1.0.0' },
})
const storeDir = path.resolve('store')
await install.handler({
...DEFAULT_OPTIONS,
cacheDir: path.resolve('cache'),
dir: process.cwd(),
linkWorkspacePackages: true,
storeDir,
})
rimraf(path.resolve(project.dir(), 'node_modules'))
rimraf(path.resolve(project.dir(), './package.json'))
project.storeHasNot('is-negative')
project.storeHasNot('is-positive')
await fetch.handler({
...DEFAULT_OPTIONS,
cacheDir: path.resolve('cache'),
dev: true,
dir: process.cwd(),
storeDir,
})
project.storeHasNot('is-negative')
project.storeHas('is-positive')
})
test('fetch only dev dependencies', async () => {
const project = prepare({
dependencies: { 'is-positive': '1.0.0' },
devDependencies: { 'is-negative': '1.0.0' },
})
const storeDir = path.resolve('store')
await install.handler({
...DEFAULT_OPTIONS,
cacheDir: path.resolve('cache'),
dir: process.cwd(),
linkWorkspacePackages: true,
storeDir,
})
rimraf(path.resolve(project.dir(), 'node_modules'))
rimraf(path.resolve(project.dir(), './package.json'))
project.storeHasNot('is-negative')
project.storeHasNot('is-positive')
await fetch.handler({
...DEFAULT_OPTIONS,
cacheDir: path.resolve('cache'),
dev: true,
dir: process.cwd(),
storeDir,
})
project.storeHas('is-negative')
project.storeHasNot('is-positive')
})
// Regression test for https://github.com/pnpm/pnpm/issues/10460
// pnpm fetch should skip local file: protocol dependencies
// because they won't be available in Docker builds
test('fetch skips file: protocol dependencies that do not exist', async () => {
const project = prepare({
dependencies: {
'is-positive': '1.0.0',
'@local/pkg': 'file:./local-pkg',
},
})
const storeDir = path.resolve('store')
const localPkgDir = path.resolve(project.dir(), 'local-pkg')
// Create the local package for initial install to generate lockfile
fs.mkdirSync(localPkgDir, { recursive: true })
fs.writeFileSync(
path.join(localPkgDir, 'package.json'),
JSON.stringify({ name: '@local/pkg', version: '1.0.0' })
)
// Create a lockfile with the file: dependency
await install.handler({
...DEFAULT_OPTIONS,
cacheDir: path.resolve('cache'),
dir: process.cwd(),
linkWorkspacePackages: true,
storeDir,
})
rimraf(path.resolve(project.dir(), 'node_modules'))
rimraf(path.resolve(project.dir(), './package.json'))
// Remove the local package directory to simulate Docker build scenario
rimraf(localPkgDir)
project.storeHasNot('is-positive')
// This should not throw an error even though the file: dependency doesn't exist
await fetch.handler({
...DEFAULT_OPTIONS,
cacheDir: path.resolve('cache'),
dir: process.cwd(),
storeDir,
})
project.storeHas('is-positive')
})
test('fetch populates global virtual store links/', async () => {
prepare({
dependencies: {
'is-positive': '1.0.0',
},
devDependencies: {
'is-negative': '1.0.0',
},
})
const storeDir = path.resolve('store')
const globalVirtualStoreDir = path.join(storeDir, STORE_VERSION, 'links')
// Generate the lockfile only — no need for a full install
await install.handler({
...DEFAULT_OPTIONS,
cacheDir: path.resolve('cache'),
dir: process.cwd(),
linkWorkspacePackages: true,
lockfileOnly: true,
storeDir,
})
// Drain workers and close SQLite connections before removing the store (required on Windows)
await finishWorkers()
closeAllStoreIndexes()
// Remove the store — simulate a cold start with only the lockfile
rimraf(storeDir)
// Fetch with enableGlobalVirtualStore — should populate links/
await fetch.handler({
...DEFAULT_OPTIONS,
cacheDir: path.resolve('cache'),
dir: process.cwd(),
storeDir,
enableGlobalVirtualStore: true,
})
// The global virtual store links/ directory should exist and contain packages
expect(fs.existsSync(globalVirtualStoreDir)).toBeTruthy()
const entries = fs.readdirSync(globalVirtualStoreDir)
expect(entries.length).toBeGreaterThan(0)
})