4 Commits

Author	SHA1	Message	Date
Zoltan Kochan	d6b8e281b6	chore: use pn instead of pnpm (#11124)	2026-03-28 11:55:51 +01:00
Zoltan Kochan	4a36b9a110	refactor: rename internal packages to @pnpm/<domain>.<leaf> convention (#10997) ... ## Summary Rename all internal packages so their npm names follow the `@pnpm/<domain>.<leaf>` convention, matching their directory structure. Also rename directories to remove redundancy and improve clarity. ### Bulk rename (94 packages) All `@pnpm/` packages now derive their name from their directory path using dot-separated segments. Exceptions: `packages/`, `__utils__/`, and `pnpm/artifacts/` keep leaf names only. ### Directory renames (removing redundant prefixes) - `cli/cli-meta` → `cli/meta`, `cli/cli-utils` → `cli/utils` - `config/config` → `config/reader`, `config/config-writer` → `config/writer` - `fetching/fetching-types` → `fetching/types` - `lockfile/lockfile-to-pnp` → `lockfile/to-pnp` - `store/store-connection-manager` → `store/connection-manager` - `store/store-controller-types` → `store/controller-types` - `store/store-path` → `store/path` ### Targeted renames (clarity improvements) - `deps/dependency-path` → `deps/path` (`@pnpm/deps.path`) - `deps/calc-dep-state` → `deps/graph-hasher` (`@pnpm/deps.graph-hasher`) - `deps/inspection/dependencies-hierarchy` → `deps/inspection/tree-builder` (`@pnpm/deps.inspection.tree-builder`) - `bins/link-bins` → `bins/linker`, `bins/remove-bins` → `bins/remover`, `bins/package-bins` → `bins/resolver` - `installing/get-context` → `installing/context` - `store/package-store` → `store/controller` - `pkg-manifest/manifest-utils` → `pkg-manifest/utils` ### Manifest reader/writer renames - `workspace/read-project-manifest` → `workspace/project-manifest-reader` (`@pnpm/workspace.project-manifest-reader`) - `workspace/write-project-manifest` → `workspace/project-manifest-writer` (`@pnpm/workspace.project-manifest-writer`) - `workspace/read-manifest` → `workspace/workspace-manifest-reader` (`@pnpm/workspace.workspace-manifest-reader`) - `workspace/manifest-writer` → `workspace/workspace-manifest-writer` (`@pnpm/workspace.workspace-manifest-writer`) ### Workspace package renames - `workspace/find-packages` → `workspace/projects-reader` - `workspace/find-workspace-dir` → `workspace/root-finder` - `workspace/resolve-workspace-range` → `workspace/range-resolver` - `workspace/filter-packages-from-dir` merged into `workspace/filter-workspace-packages` → `workspace/projects-filter` ### Domain moves - `pkg-manifest/read-project-manifest` → `workspace/project-manifest-reader` - `pkg-manifest/write-project-manifest` → `workspace/project-manifest-writer` - `pkg-manifest/exportable-manifest` → `releasing/exportable-manifest` ### Scope - 1206 files changed - Updated: package.json names/deps, TypeScript imports, tsconfig references, changeset files, renovate.json, test fixtures, import ordering	2026-03-17 21:50:40 +01:00
Zoltan Kochan	b7f0f21582	feat: use SQLite for storing package index in the content-addressable store (#10827) ... ## Summary Replace individual `.mpk` (MessagePack) files under `$STORE/index/` with a single SQLite database at `$STORE/index.db` using Node.js 22's built-in `node:sqlite` module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries. Closes #10826 ## Design ### New package: `@pnpm/store.index` A new `StoreIndex` class wraps a SQLite database with a simple key-value API (`get`, `set`, `delete`, `has`, `entries`). Data is serialized with msgpackr and stored as BLOBs. The table uses `WITHOUT ROWID` for compact storage. Key design decisions: - **WAL mode** enables concurrent reads from workers while the main process writes. - **`busy_timeout=5000`** plus a retry loop with `Atomics.wait`-based `sleepSync` handles `SQLITE_BUSY` errors from concurrent access. - **Performance PRAGMAs**: `synchronous=NORMAL`, `mmap_size=512MB`, `cache_size=32MB`, `temp_store=MEMORY`, `wal_autocheckpoint=10000`. - **Write batching**: `queueWrites()` batches pre-packed entries from tarball extraction and flushes them in a single transaction on `process.nextTick`. `setRawMany()` writes immediate batches (e.g. from `addFilesFromDir`). - **Lifecycle**: `close()` auto-flushes pending writes, runs `PRAGMA optimize`, and closes the DB. A `process.on('exit')` handler ensures cleanup even on unexpected exits. - **`VACUUM` after `deleteMany`** (used by `pnpm store prune`) to reclaim disk space. ### Key format Keys are `integrity\tpkgId` (tab-separated). Git-hosted packages use `pkgId\tbuilt` or `pkgId\tnot-built`. ### Shared StoreIndex instance A single `StoreIndex` instance is threaded through the entire install lifecycle — from `createNewStoreController` through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access. ### Worker architecture Index writes are performed in the main process, not in worker threads. Workers send pre-packed `{ key, buffer }` pairs back to the main process via `postMessage`, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads. ### SQLite ExperimentalWarning suppression `node:sqlite` emits an `ExperimentalWarning` on first load. This is suppressed via a `process.emitWarning` override injected through esbuild's `banner` option, which runs on line 1 of both `dist/pnpm.mjs` and `dist/worker.js` — before any module that loads `node:sqlite`. ### No migration from `.mpk` files Old `.mpk` index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store). ## Changed packages 121 files changed across these areas: - **`store/index/`** — New `@pnpm/store.index` package - **`worker/`** — Write batching moved from worker module into `StoreIndex` class; workers send pre-packed buffers to main process - **`store/package-store/`** — StoreIndex creation and lifecycle management - **`store/cafs/`** — Removed `getFilePathInCafs` index-file utilities (no longer needed) - **`store/pkg-finder/`** — Reads from StoreIndex instead of `.mpk` files - **`store/plugin-commands-store/`** — `store status` uses StoreIndex - **`store/plugin-commands-store-inspecting/`** — `cat-index` and `find-hash` use StoreIndex - **`fetching/tarball-fetcher/`** — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading - **`fetching/git-fetcher/`, `binary-fetcher/`, `pick-fetcher/`** — Accept StoreIndex parameter - **`pkg-manager/`** — `client`, `core`, `headless`, `package-requester` thread StoreIndex - **`reviewing/`** — `license-scanner`, `sbom`, `dependencies-hierarchy` accept StoreIndex - **`cache/api/`** — Cache view uses StoreIndex - **`pnpm/bundle.ts`** — esbuild banner for ExperimentalWarning suppression ## Test plan - [x] `pnpm --filter @pnpm/store.index test` — Unit tests for StoreIndex CRUD and batching - [x] `pnpm --filter @pnpm/package-store test` — Store controller lifecycle - [x] `pnpm --filter @pnpm/package-requester test` — Package requester reads from SQLite index - [x] `pnpm --filter @pnpm/tarball-fetcher test` — Tarball and git-hosted fetcher writes - [x] `pnpm --filter @pnpm/headless test` — Headless install - [x] `pnpm --filter @pnpm/core test` — Core install, side effects, patching - [x] `pnpm --filter @pnpm/plugin-commands-rebuild test` — Rebuild reads from index - [x] `pnpm --filter @pnpm/license-scanner test` — License scanning - [x] e2e tests pass 🤖 Generated with [Claude Code](https://claude.com/claude-code)	2026-03-06 12:59:04 +01:00
Allan Kimmer Jensen	f92ac24c1b	feat(sbom): add pnpm sbom command (#10592) ... * feat(sbom): add `pnpm sbom` command (#9088) new command that generates SBOMs from the lockfile + store metadata. supports CycloneDX 1.6 JSON and SPDX 2.3 JSON via `--sbom-format`. two new packages following the existing `pnpm licenses` architecture: - `@pnpm/sbom` — core library (lockfile walking, store reading, serializers) - `@pnpm/plugin-commands-sbom` — CLI plugin wiring uses the lockfile walker for dependency traversal and reads package.json from the CAFS store for license/author/description metadata. `--lockfile-only` skips the store entirely for faster CI runs where metadata isn't needed. validated against official CycloneDX 1.6 and SPDX 2.3 JSON schemas. * chore: add sbom-related words to cspell dictionary * fix(sbom): address CycloneDX review feedback and bump to 1.7 Implements all 5 items from the CycloneDX maintainer review: split scoped names into group/name, move hashes to externalReferences distribution, use license.id for known SPDX identifiers, switch to modern tools.components structure with pnpm version, and bump specVersion to 1.7. Also adds spdx-license-ids for proper license classification and improves SPDX serializer test coverage. * fix(sbom): fix CI bundle failure for spdx-license-ids createRequire doesn't work in the esbuild bundle since it's a runtime resolve, switched back to regular import which esbuild can inline. * fix(sbom): use tarball URL for distribution externalReferences Use actual tarball download URL instead of PURL for CycloneDX distribution externalReferences, per review feedback. * feat(sbom): add CycloneDX metadata and improve SBOM quality scores adds $schema, timestamp, lifecycles (build/pre-build) to CycloneDX output to match what npm does. also enriches both CycloneDX and SPDX with metadata.authors, metadata.supplier, component supplier from author, vcs externalReferences from repository, and root component details (purl, license, description, author, vcs). SPDX now uses tarball URL for downloadLocation instead of NOASSERTION. renames CycloneDxToolInfo to CycloneDxOptions, passes lockfileOnly through to the serializer for lifecycle phase selection. adds store-dir to accepted CLI options. * fix(sbom): address CycloneDX review feedback round 2 switches license classification from spdx-license-ids to @cyclonedx/cyclonedx-library (SPDX.isSupportedSpdxId) for accurate CycloneDX license ID validation per jkowalleck's feedback. removes hardcoded metadata.authors and metadata.supplier — these are not appropriate for a tool to set. adds --sbom-authors and --sbom-supplier CLI flags so the SBOM consumer (e.g. ACME Corp) can declare who they are. removes supplier from components — supplier is the registry/distributor, not the package author. also fixes distribution externalReference to only emit when a real tarball URL exists, no PURL fallback. * fix(sbom): use sub-path import for CycloneDX library to fix bundle top-level import from @cyclonedx/cyclonedx-library drags in validation/serialize layers with optional deps (ajv-formats, libxmljs2, xmlbuilder2) that esbuild can't resolve during pnpm CLI bundling. switch to @cyclonedx/cyclonedx-library/SPDX which only pulls in the SPDX module we actually use — pure JS, no optional deps. * chore: update manifests * refactor: extract shared store-reading logic into @pnpm/store.pkg-finder Both @pnpm/license-scanner and @pnpm/sbom independently implemented nearly identical logic to read a package's file index from the content-addressable store. This extracts that into a new shared package that returns a uniform Map<string, string> (filename → absolute path), simplifying both consumers. Close #9088 --------- Co-authored-by: Zoltan Kochan <z@kochan.io>	2026-02-25 08:45:21 +01:00