Commit Graph

32 Commits

Author SHA1 Message Date
Zoltan Kochan
4a36b9a110 refactor: rename internal packages to @pnpm/<domain>.<leaf> convention (#10997)
## Summary

Rename all internal packages so their npm names follow the `@pnpm/<domain>.<leaf>` convention, matching their directory structure. Also rename directories to remove redundancy and improve clarity.

### Bulk rename (94 packages)

All `@pnpm/` packages now derive their name from their directory path using dot-separated segments. Exceptions: `packages/`, `__utils__/`, and `pnpm/artifacts/` keep leaf names only.

### Directory renames (removing redundant prefixes)

- `cli/cli-meta` → `cli/meta`, `cli/cli-utils` → `cli/utils`
- `config/config` → `config/reader`, `config/config-writer` → `config/writer`
- `fetching/fetching-types` → `fetching/types`
- `lockfile/lockfile-to-pnp` → `lockfile/to-pnp`
- `store/store-connection-manager` → `store/connection-manager`
- `store/store-controller-types` → `store/controller-types`
- `store/store-path` → `store/path`

### Targeted renames (clarity improvements)

- `deps/dependency-path` → `deps/path` (`@pnpm/deps.path`)
- `deps/calc-dep-state` → `deps/graph-hasher` (`@pnpm/deps.graph-hasher`)
- `deps/inspection/dependencies-hierarchy` → `deps/inspection/tree-builder` (`@pnpm/deps.inspection.tree-builder`)
- `bins/link-bins` → `bins/linker`, `bins/remove-bins` → `bins/remover`, `bins/package-bins` → `bins/resolver`
- `installing/get-context` → `installing/context`
- `store/package-store` → `store/controller`
- `pkg-manifest/manifest-utils` → `pkg-manifest/utils`

### Manifest reader/writer renames

- `workspace/read-project-manifest` → `workspace/project-manifest-reader` (`@pnpm/workspace.project-manifest-reader`)
- `workspace/write-project-manifest` → `workspace/project-manifest-writer` (`@pnpm/workspace.project-manifest-writer`)
- `workspace/read-manifest` → `workspace/workspace-manifest-reader` (`@pnpm/workspace.workspace-manifest-reader`)
- `workspace/manifest-writer` → `workspace/workspace-manifest-writer` (`@pnpm/workspace.workspace-manifest-writer`)

### Workspace package renames

- `workspace/find-packages` → `workspace/projects-reader`
- `workspace/find-workspace-dir` → `workspace/root-finder`
- `workspace/resolve-workspace-range` → `workspace/range-resolver`
- `workspace/filter-packages-from-dir` merged into `workspace/filter-workspace-packages` → `workspace/projects-filter`

### Domain moves

- `pkg-manifest/read-project-manifest` → `workspace/project-manifest-reader`
- `pkg-manifest/write-project-manifest` → `workspace/project-manifest-writer`
- `pkg-manifest/exportable-manifest` → `releasing/exportable-manifest`

### Scope

- 1206 files changed
- Updated: package.json names/deps, TypeScript imports, tsconfig references, changeset files, renovate.json, test fixtures, import ordering
2026-03-17 21:50:40 +01:00
Zoltan Kochan
5d5818e44f style: enforce node: protocol for builtin imports (#10951)
Add n/prefer-node-protocol rule and autofix all bare builtin imports
to use the node: prefix. Simplify the simple-import-sort builtins
pattern to just ^node: since all imports now use the prefix.
2026-03-13 07:59:51 +01:00
Zoltan Kochan
1c8c4e49f5 style: add eslint-plugin-simple-import-sort (#10947)
Add eslint-plugin-simple-import-sort to enforce consistent import ordering:
- Node.js builtins first
- External packages second
- Relative imports last
- Named imports sorted alphabetically within each statement
2026-03-13 02:02:38 +01:00
Zoltan Kochan
3a5bfaa94f chore: update zkochan packages to latest versions (#10930)
Update all packages from zkochan/packages to their latest major versions
and exclude them from minimumReleaseAge requirement. This includes
updating catalog entries, adapting to breaking API changes (default
exports replaced with named exports, sync functions renamed with Sync
suffix), and updating type declarations.
2026-03-11 13:47:46 +01:00
Zoltan Kochan
b7f0f21582 feat: use SQLite for storing package index in the content-addressable store (#10827)
## Summary

Replace individual `.mpk` (MessagePack) files under `$STORE/index/` with a single SQLite database at `$STORE/index.db` using Node.js 22's built-in `node:sqlite` module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries.

Closes #10826

## Design

### New package: `@pnpm/store.index`

A new `StoreIndex` class wraps a SQLite database with a simple key-value API (`get`, `set`, `delete`, `has`, `entries`). Data is serialized with msgpackr and stored as BLOBs. The table uses `WITHOUT ROWID` for compact storage.

Key design decisions:

- **WAL mode** enables concurrent reads from workers while the main process writes.
- **`busy_timeout=5000`** plus a retry loop with `Atomics.wait`-based `sleepSync` handles `SQLITE_BUSY` errors from concurrent access.
- **Performance PRAGMAs**: `synchronous=NORMAL`, `mmap_size=512MB`, `cache_size=32MB`, `temp_store=MEMORY`, `wal_autocheckpoint=10000`.
- **Write batching**: `queueWrites()` batches pre-packed entries from tarball extraction and flushes them in a single transaction on `process.nextTick`. `setRawMany()` writes immediate batches (e.g. from `addFilesFromDir`).
- **Lifecycle**: `close()` auto-flushes pending writes, runs `PRAGMA optimize`, and closes the DB. A `process.on('exit')` handler ensures cleanup even on unexpected exits.
- **`VACUUM` after `deleteMany`** (used by `pnpm store prune`) to reclaim disk space.

### Key format

Keys are `integrity\tpkgId` (tab-separated). Git-hosted packages use `pkgId\tbuilt` or `pkgId\tnot-built`.

### Shared StoreIndex instance

A single `StoreIndex` instance is threaded through the entire install lifecycle — from `createNewStoreController` through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access.

### Worker architecture

Index writes are performed in the main process, not in worker threads. Workers send pre-packed `{ key, buffer }` pairs back to the main process via `postMessage`, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads.

### SQLite ExperimentalWarning suppression

`node:sqlite` emits an `ExperimentalWarning` on first load. This is suppressed via a `process.emitWarning` override injected through esbuild's `banner` option, which runs on line 1 of both `dist/pnpm.mjs` and `dist/worker.js` — before any module that loads `node:sqlite`.

### No migration from `.mpk` files

Old `.mpk` index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store).

## Changed packages

121 files changed across these areas:

- **`store/index/`** — New `@pnpm/store.index` package
- **`worker/`** — Write batching moved from worker module into `StoreIndex` class; workers send pre-packed buffers to main process
- **`store/package-store/`** — StoreIndex creation and lifecycle management
- **`store/cafs/`** — Removed `getFilePathInCafs` index-file utilities (no longer needed)
- **`store/pkg-finder/`** — Reads from StoreIndex instead of `.mpk` files
- **`store/plugin-commands-store/`** — `store status` uses StoreIndex
- **`store/plugin-commands-store-inspecting/`** — `cat-index` and `find-hash` use StoreIndex
- **`fetching/tarball-fetcher/`** — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading
- **`fetching/git-fetcher/`, `binary-fetcher/`, `pick-fetcher/`** — Accept StoreIndex parameter
- **`pkg-manager/`** — `client`, `core`, `headless`, `package-requester` thread StoreIndex
- **`reviewing/`** — `license-scanner`, `sbom`, `dependencies-hierarchy` accept StoreIndex
- **`cache/api/`** — Cache view uses StoreIndex
- **`pnpm/bundle.ts`** — esbuild banner for ExperimentalWarning suppression

## Test plan

- [x] `pnpm --filter @pnpm/store.index test` — Unit tests for StoreIndex CRUD and batching
- [x] `pnpm --filter @pnpm/package-store test` — Store controller lifecycle
- [x] `pnpm --filter @pnpm/package-requester test` — Package requester reads from SQLite index
- [x] `pnpm --filter @pnpm/tarball-fetcher test` — Tarball and git-hosted fetcher writes
- [x] `pnpm --filter @pnpm/headless test` — Headless install
- [x] `pnpm --filter @pnpm/core test` — Core install, side effects, patching
- [x] `pnpm --filter @pnpm/plugin-commands-rebuild test` — Rebuild reads from index
- [x] `pnpm --filter @pnpm/license-scanner test` — License scanning
- [x] e2e tests pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-03-06 12:59:04 +01:00
Zoltan Kochan
c5d4d81f56 refactor: rename FilesIndex to FilesMap (#10427) 2026-01-08 18:17:57 +01:00
Zoltan Kochan
8b4bdf9a83 refactor: replace onlyBuiltDependencies and ignoredBuiltDependencies with allowBuilds (#10401) 2026-01-02 23:21:17 +01:00
klassiker
c5fbddee05 fix(git-fetcher): ensure the specified commit is used after checkout (#10310)
* fix(git-fetcher): ensure the specified commit is used after checkout

* fix(git-resolver): always resolve to a full commit

* chore: add changeset heavy-dragons-start

* test: fix related test case

* test: fix some other test that gets stuck

* Update heavy-dragons-start.md with PR reference

Add reference to pull request #10310 for clarity.
2025-12-17 03:26:18 +01:00
Zoltan Kochan
0048667db4 refactor: use Maps instead of Records (#10312) 2025-12-15 11:48:19 +01:00
Oren
ba065f6a8b fix(git-fetcher): block git dependencies from running prepare scripts unless allowed (#10288)
* fix(git-fetcher): block git dependencies from running prepare scripts unless allowed

* Update exec/prepare-package/src/index.ts

Co-authored-by: Zoltan Kochan <z@kochan.io>

* Also implement in gitHostedTarballFetcher

* refactor: move allowBuild function creation to the store manager

* refactor: pass allowBuild function to fetch function directly

* refactor: revert not needed changes and update changesets

* test: fix

* fix: implemented CR suggestions

* test: fix

* test: fix

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
2025-12-09 18:25:07 +01:00
Zoltan Kochan
cf9f14aa9e fix: update tempy 2025-08-28 14:00:51 +02:00
Zoltan Kochan
491a84fb26 feat: use ESM instead of commonjs (#9870) 2025-08-25 10:02:00 +02:00
Zoltan Kochan
27cbc09206 style: fix jest-related linting issues (#9894) 2025-08-22 21:56:49 +02:00
Zoltan Kochan
c89c93d59b test: use @jest/globals (#9877) 2025-08-19 00:16:25 +02:00
Zoltan Kochan
14c78e81d4 test: use jest.mocked (#9874) 2025-08-18 15:22:37 +02:00
Zoltan Kochan
298e5dcafd feat!: create a separate directory in the store for index files (#8700) 2024-10-27 21:00:28 +01:00
Khải
9b4f73caaf chore(scripts): typecheck-only (#8395)
* chore(scripts): typecheck-only

* feat: change all configuration

* feat: include pnpm/ and pnpm/test/

* chore(deps): remove unused dependency

* refactor(typescript-only): use find-packages

* refactor(typescript-only): refactor paths

* fix: typescript-only

* fix: update compile-only

* fix: compile pnpm

* fix: windows

* fix: windows

* chore: meta-updater

* refactor(tsconfig): remove explicit composite

* fix: path in windows

* feat: don't depend on cwd

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
2024-08-11 08:26:01 +02:00
Brandon Cheng
c7e1b6fae8 chore: configure TypeScript project references for tests (#8128)
* refactor: store link values before converting to references

* fix: use .sort() without localeCompare

https://github.com/pnpm/pnpm/pull/8128#discussion_r1614031566

> Nit, but you probably just want to call sort without a comparison
> function; these are already strings and locale compare is not a good
> comparison for anything but human readable strings since it will
> differ on different people's machines based on their language setting.
> I've hit this too many times before for code gen.

* feat: configure meta-updater to write test/tsconfig.json files

* fix: relative imports for __typings__

* chore: `pnpm run meta-updater`

* fix: explicitly use test/tsconfig.json for ts-jest
2024-05-31 12:48:13 +02:00
Zoltan Kochan
eaebfd53c7 test: fix 2024-03-18 00:12:15 +01:00
Zoltan Kochan
19ceaef706 test(git-fetcher): fix 2024-03-05 13:49:13 +01:00
Zoltan Kochan
75840d6570 fix: resolve symlink dirs when uploading side-effects (#7706)
close #7691
close #7253
2024-02-24 15:26:14 +01:00
Zoltan Kochan
36dcaa0163 fix!: only fetch the included files of a git-hosted dependency (#7638)
close #6512
close #7329
2024-02-12 13:23:24 +01:00
Rex Zeng
b13d2dc1ae feat: add sub folder support for git url (#7487)
close #4765

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
2024-01-24 02:15:02 +01:00
Zoltan Kochan
70b2830acc refactor: worker (#7033) 2023-09-03 02:30:08 +03:00
Zoltan Kochan
e6353f964c perf: fetch directories to CAFS in worker (#6984) 2023-08-26 15:43:32 +03:00
Zoltan Kochan
ee61ca4cb7 style: update eslint (#6236) 2023-03-18 14:57:29 +02:00
Zoltan Kochan
91b8027441 ci: use pnpm v8 (#6116) 2023-02-22 17:50:20 +02:00
Zoltan Kochan
c7b05cd9a4 fix: don't build git-hosted dependencies when scripts are ignored (#5897)
close #5876
2023-01-09 03:59:37 +02:00
Zoltan Kochan
ec97a31057 fix: print a better error message on prepare pkg failure (#5847)
ref #5845
2023-01-01 23:28:33 +02:00
Zoltan Kochan
339c0a7046 fix: run prepublish scripts of packages installed from Git (#5837)
close #5826
2022-12-25 21:18:18 +02:00
Zoltan Kochan
9cbe293876 refactor: put all non-public projects to directories that start with __ 2022-11-20 02:40:44 +02:00
Zoltan Kochan
4ca53b0b50 refactor: group projects in different subdirectories (#5659) 2022-11-20 01:35:22 +02:00