Commit Graph

24 Commits

Author SHA1 Message Date
Zoltan Kochan
4a36b9a110 refactor: rename internal packages to @pnpm/<domain>.<leaf> convention (#10997)
## Summary

Rename all internal packages so their npm names follow the `@pnpm/<domain>.<leaf>` convention, matching their directory structure. Also rename directories to remove redundancy and improve clarity.

### Bulk rename (94 packages)

All `@pnpm/` packages now derive their name from their directory path using dot-separated segments. Exceptions: `packages/`, `__utils__/`, and `pnpm/artifacts/` keep leaf names only.

### Directory renames (removing redundant prefixes)

- `cli/cli-meta` → `cli/meta`, `cli/cli-utils` → `cli/utils`
- `config/config` → `config/reader`, `config/config-writer` → `config/writer`
- `fetching/fetching-types` → `fetching/types`
- `lockfile/lockfile-to-pnp` → `lockfile/to-pnp`
- `store/store-connection-manager` → `store/connection-manager`
- `store/store-controller-types` → `store/controller-types`
- `store/store-path` → `store/path`

### Targeted renames (clarity improvements)

- `deps/dependency-path` → `deps/path` (`@pnpm/deps.path`)
- `deps/calc-dep-state` → `deps/graph-hasher` (`@pnpm/deps.graph-hasher`)
- `deps/inspection/dependencies-hierarchy` → `deps/inspection/tree-builder` (`@pnpm/deps.inspection.tree-builder`)
- `bins/link-bins` → `bins/linker`, `bins/remove-bins` → `bins/remover`, `bins/package-bins` → `bins/resolver`
- `installing/get-context` → `installing/context`
- `store/package-store` → `store/controller`
- `pkg-manifest/manifest-utils` → `pkg-manifest/utils`

### Manifest reader/writer renames

- `workspace/read-project-manifest` → `workspace/project-manifest-reader` (`@pnpm/workspace.project-manifest-reader`)
- `workspace/write-project-manifest` → `workspace/project-manifest-writer` (`@pnpm/workspace.project-manifest-writer`)
- `workspace/read-manifest` → `workspace/workspace-manifest-reader` (`@pnpm/workspace.workspace-manifest-reader`)
- `workspace/manifest-writer` → `workspace/workspace-manifest-writer` (`@pnpm/workspace.workspace-manifest-writer`)

### Workspace package renames

- `workspace/find-packages` → `workspace/projects-reader`
- `workspace/find-workspace-dir` → `workspace/root-finder`
- `workspace/resolve-workspace-range` → `workspace/range-resolver`
- `workspace/filter-packages-from-dir` merged into `workspace/filter-workspace-packages` → `workspace/projects-filter`

### Domain moves

- `pkg-manifest/read-project-manifest` → `workspace/project-manifest-reader`
- `pkg-manifest/write-project-manifest` → `workspace/project-manifest-writer`
- `pkg-manifest/exportable-manifest` → `releasing/exportable-manifest`

### Scope

- 1206 files changed
- Updated: package.json names/deps, TypeScript imports, tsconfig references, changeset files, renovate.json, test fixtures, import ordering
2026-03-17 21:50:40 +01:00
Zoltan Kochan
5d5818e44f style: enforce node: protocol for builtin imports (#10951)
Add n/prefer-node-protocol rule and autofix all bare builtin imports
to use the node: prefix. Simplify the simple-import-sort builtins
pattern to just ^node: since all imports now use the prefix.
2026-03-13 07:59:51 +01:00
Zoltan Kochan
1c8c4e49f5 style: add eslint-plugin-simple-import-sort (#10947)
Add eslint-plugin-simple-import-sort to enforce consistent import ordering:
- Node.js builtins first
- External packages second
- Relative imports last
- Named imports sorted alphabetically within each statement
2026-03-13 02:02:38 +01:00
Zoltan Kochan
3a5bfaa94f chore: update zkochan packages to latest versions (#10930)
Update all packages from zkochan/packages to their latest major versions
and exclude them from minimumReleaseAge requirement. This includes
updating catalog entries, adapting to breaking API changes (default
exports replaced with named exports, sync functions renamed with Sync
suffix), and updating type declarations.
2026-03-11 13:47:46 +01:00
Brandon Cheng
01914345d5 build: enable @typescript-eslint/no-import-type-side-effects (#10630)
* build: enable `@typescript-eslint/no-import-type-side-effects`

* build: disable `@typescript-eslint/consistent-type-imports`

* chore: apply fixes for `no-import-type-side-effects`

pnpm exec eslint "**/src/**/*.ts" "**/test/**/*.ts" --fix
2026-03-08 00:02:48 +01:00
Zoltan Kochan
b7f0f21582 feat: use SQLite for storing package index in the content-addressable store (#10827)
## Summary

Replace individual `.mpk` (MessagePack) files under `$STORE/index/` with a single SQLite database at `$STORE/index.db` using Node.js 22's built-in `node:sqlite` module. This reduces filesystem syscall overhead and improves space efficiency for small metadata entries.

Closes #10826

## Design

### New package: `@pnpm/store.index`

A new `StoreIndex` class wraps a SQLite database with a simple key-value API (`get`, `set`, `delete`, `has`, `entries`). Data is serialized with msgpackr and stored as BLOBs. The table uses `WITHOUT ROWID` for compact storage.

Key design decisions:

- **WAL mode** enables concurrent reads from workers while the main process writes.
- **`busy_timeout=5000`** plus a retry loop with `Atomics.wait`-based `sleepSync` handles `SQLITE_BUSY` errors from concurrent access.
- **Performance PRAGMAs**: `synchronous=NORMAL`, `mmap_size=512MB`, `cache_size=32MB`, `temp_store=MEMORY`, `wal_autocheckpoint=10000`.
- **Write batching**: `queueWrites()` batches pre-packed entries from tarball extraction and flushes them in a single transaction on `process.nextTick`. `setRawMany()` writes immediate batches (e.g. from `addFilesFromDir`).
- **Lifecycle**: `close()` auto-flushes pending writes, runs `PRAGMA optimize`, and closes the DB. A `process.on('exit')` handler ensures cleanup even on unexpected exits.
- **`VACUUM` after `deleteMany`** (used by `pnpm store prune`) to reclaim disk space.

### Key format

Keys are `integrity\tpkgId` (tab-separated). Git-hosted packages use `pkgId\tbuilt` or `pkgId\tnot-built`.

### Shared StoreIndex instance

A single `StoreIndex` instance is threaded through the entire install lifecycle — from `createNewStoreController` through the fetcher chain, package requester, license scanner, SBOM collector, and dependencies hierarchy. This replaces the previous pattern of each component creating its own file-based index access.

### Worker architecture

Index writes are performed in the main process, not in worker threads. Workers send pre-packed `{ key, buffer }` pairs back to the main process via `postMessage`, where they are batched and flushed to SQLite. This avoids SQLite write contention between threads.

### SQLite ExperimentalWarning suppression

`node:sqlite` emits an `ExperimentalWarning` on first load. This is suppressed via a `process.emitWarning` override injected through esbuild's `banner` option, which runs on line 1 of both `dist/pnpm.mjs` and `dist/worker.js` — before any module that loads `node:sqlite`.

### No migration from `.mpk` files

Old `.mpk` index files are not migrated. Packages missing from the new SQLite index are re-fetched on demand (the same behavior as a fresh store).

## Changed packages

121 files changed across these areas:

- **`store/index/`** — New `@pnpm/store.index` package
- **`worker/`** — Write batching moved from worker module into `StoreIndex` class; workers send pre-packed buffers to main process
- **`store/package-store/`** — StoreIndex creation and lifecycle management
- **`store/cafs/`** — Removed `getFilePathInCafs` index-file utilities (no longer needed)
- **`store/pkg-finder/`** — Reads from StoreIndex instead of `.mpk` files
- **`store/plugin-commands-store/`** — `store status` uses StoreIndex
- **`store/plugin-commands-store-inspecting/`** — `cat-index` and `find-hash` use StoreIndex
- **`fetching/tarball-fetcher/`** — Threads StoreIndex through fetchers; git-hosted fetcher flushes before reading
- **`fetching/git-fetcher/`, `binary-fetcher/`, `pick-fetcher/`** — Accept StoreIndex parameter
- **`pkg-manager/`** — `client`, `core`, `headless`, `package-requester` thread StoreIndex
- **`reviewing/`** — `license-scanner`, `sbom`, `dependencies-hierarchy` accept StoreIndex
- **`cache/api/`** — Cache view uses StoreIndex
- **`pnpm/bundle.ts`** — esbuild banner for ExperimentalWarning suppression

## Test plan

- [x] `pnpm --filter @pnpm/store.index test` — Unit tests for StoreIndex CRUD and batching
- [x] `pnpm --filter @pnpm/package-store test` — Store controller lifecycle
- [x] `pnpm --filter @pnpm/package-requester test` — Package requester reads from SQLite index
- [x] `pnpm --filter @pnpm/tarball-fetcher test` — Tarball and git-hosted fetcher writes
- [x] `pnpm --filter @pnpm/headless test` — Headless install
- [x] `pnpm --filter @pnpm/core test` — Core install, side effects, patching
- [x] `pnpm --filter @pnpm/plugin-commands-rebuild test` — Rebuild reads from index
- [x] `pnpm --filter @pnpm/license-scanner test` — License scanning
- [x] e2e tests pass

🤖 Generated with [Claude Code](https://claude.com/claude-code)
2026-03-06 12:59:04 +01:00
klassiker
c5fbddee05 fix(git-fetcher): ensure the specified commit is used after checkout (#10310)
* fix(git-fetcher): ensure the specified commit is used after checkout

* fix(git-resolver): always resolve to a full commit

* chore: add changeset heavy-dragons-start

* test: fix related test case

* test: fix some other test that gets stuck

* Update heavy-dragons-start.md with PR reference

Add reference to pull request #10310 for clarity.
2025-12-17 03:26:18 +01:00
Oren
ba065f6a8b fix(git-fetcher): block git dependencies from running prepare scripts unless allowed (#10288)
* fix(git-fetcher): block git dependencies from running prepare scripts unless allowed

* Update exec/prepare-package/src/index.ts

Co-authored-by: Zoltan Kochan <z@kochan.io>

* Also implement in gitHostedTarballFetcher

* refactor: move allowBuild function creation to the store manager

* refactor: pass allowBuild function to fetch function directly

* refactor: revert not needed changes and update changesets

* test: fix

* fix: implemented CR suggestions

* test: fix

* test: fix

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
2025-12-09 18:25:07 +01:00
Zoltan Kochan
298e5dcafd feat!: create a separate directory in the store for index files (#8700) 2024-10-27 21:00:28 +01:00
btea
9797a1e889 refactor: remove useless parameters from the catch statement (#8583) 2024-09-29 12:08:48 +02:00
Zoltan Kochan
c9db742824 fix: update typescript to v5.4 (#7905) 2024-04-12 14:03:45 +02:00
Khải
c17263c62a refactor(fetching): return type annotations (#7903) 2024-04-12 00:32:48 +02:00
Zoltan Kochan
0564745b1d style: use unknown in try/catch (#7709) 2024-03-06 01:47:04 +01:00
Zoltan Kochan
36dcaa0163 fix!: only fetch the included files of a git-hosted dependency (#7638)
close #6512
close #7329
2024-02-12 13:23:24 +01:00
Rex Zeng
b13d2dc1ae feat: add sub folder support for git url (#7487)
close #4765

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
2024-01-24 02:15:02 +01:00
Zoltan Kochan
70b2830acc refactor: worker (#7033) 2023-09-03 02:30:08 +03:00
Brandon Cheng
e9aa6f682a chore(deps): update TypeScript 5.1.6 -> 5.2.2 (#7016)
* test: use sha512 integrity in fixtures to fix ci break

* test: update snapshots for sha512 fixture change

* chore(deps): remove unneeded peer dependency exception

The `peerDependencyRules.allowedVersions` exception on
`@typescript-eslint/eslint-plugin` no longer seems to be necessary.
Removing it does not introduce any new peer dependency errors on
`pnpm install`.

I suspect this was needed for the
`eslint-config-standard-with-typescript` dependency in the past, but a
@typescript-eslint/eslint-plugin upgrade made it no longer necessary.

* chore(deps): update @typescript-eslint dependencies 5.62.0 -> 6.5.0

@typescript-eslint 6.5.0 is the first version to introduce support for
TypeScript 5.2.

https://github.com/typescript-eslint/typescript-eslint/releases/tag/v6.5.0

```
=============

WARNING: You are currently running a version of TypeScript which is not officially supported by @typescript-eslint/typescript-estree.

You may find that it works just fine, or you may not.

SUPPORTED TYPESCRIPT VERSIONS: >=3.3.1 <5.2.0

YOUR TYPESCRIPT VERSION: 5.2.2

Please only submit bug reports when using the officially supported version.
```

* chore(deps): update eslint-config-standard-with-typescript 37.0.0 -> 39.0.0

Version 38.0.0 is the first version to support @typescript-eslint v6.
https://github.com/standard/eslint-config-standard-with-typescript/releases/tag/v38.0.0

Otherwise the following error appears.

```
> eslint "src/**/*.ts" "test/**/*.ts" "--fix"

Oops! Something went wrong! :(

ESLint: 8.47.0

Error: ../../.eslintrc.json » @pnpm/eslint-config » eslint-config-standard-with-typescript:
        Configuration for rule "@typescript-eslint/restrict-plus-operands" is invalid:
        Value {"checkCompoundAssignments":true} should NOT have additional properties.

    at ConfigValidator.validateRuleOptions (/Users/gluxon/Developer/pnpm/node_modules/.pnpm/@eslint+eslintrc@2.1.2/node_modules/@eslint/eslintrc/dist/eslintrc.cjs:2039:23)
    at /Users/gluxon/Developer/pnpm/node_modules/.pnpm/@eslint+eslintrc@2.1.2/node_modules/@eslint/eslintrc/dist/eslintrc.cjs:2094:18
```

* chore: remove unnecessary disables for restrict-template-expressions

The `@typescript-eslint/restrict-template-expressions` rule relaxed
what types are allowed in template expressions.

c13ce0b4f7 (diff-b852e1e199d2976eee1183fc84ac12a5d42fc61f0ae4b1c290dd54d621546db0)

Many of these disables were for interpolated values that had an `any`
type.

* chore: remove unnecessary disables for restrict-plus-operands

The original error was:

```
Invalid operand for a '+' operation. Operands must each be a number or string. Got `any`. eslint@typescript-eslint/restrict-plus-operands
```

It look like the newer version now allows `any`.

* style: fix errors of prefer-optional-chain and prefer-nullish-coalescing

The `@typescript-eslint/prefer-optional-chain` and
`@typescript-eslint/prefer-nullish-coalescing` rules got a bit
smarter. This commit applies autofixes. I believe the changes should be
equivalent to what existed before.

Example of the new `@typescript-eslint/prefer-optional-chain` lints.

```
pnpm/pkg-manifest/exportable-manifest/src/index.ts
  71:10  error  Prefer using an optional chain expression instead, as it's more concise and easier to read  @typescript-eslint/prefer-optional-chain
  87:10  error  Prefer using an optional chain expression instead, as it's more concise and easier to read  @typescript-eslint/prefer-optional-chain
```

Example of the new `@typescript-eslint/prefer-nullish-coalescing` lints.

```
pnpm/fs/find-packages/src/index.ts
  32:38  error  Prefer using nullish coalescing operator (`??`) instead of a ternary expression, as it is simpler to read  @typescript-eslint/prefer-nullish-coalescing
```

* chore(deps): update TypeScript 5.1.6 -> 5.2.2

* chore(deps): update @yarnpkg/core->@types/lodash override to 4.14.197

This fixes a compilation error that appears on TypeScript 5.2.2. This
error was fixed in a later version of `@types/lodash`.

https://github.com/DefinitelyTyped/DefinitelyTyped/pull/66123

```
../node_modules/.pnpm/@types+lodash@4.14.181/node_modules/@types/lodash/index.d.ts:45:15 - error TS2428: All declarations of 'WeakMap' must have identical type parameters.

45     interface WeakMap<K extends object, V> { }
                 ~~~~~~~

Found 4 errors.
```
2023-08-31 16:27:01 +03:00
Zoltan Kochan
c8a6ea8ae9 refactor: rename tarball-worker to worker 2023-08-26 17:26:21 +03:00
Zoltan Kochan
e6353f964c perf: fetch directories to CAFS in worker (#6984) 2023-08-26 15:43:32 +03:00
Zoltan Kochan
083bbf5907 perf: extract packages in a worker pool (#6850) 2023-08-14 02:07:39 +03:00
Zoltan Kochan
c7b05cd9a4 fix: don't build git-hosted dependencies when scripts are ignored (#5897)
close #5876
2023-01-09 03:59:37 +02:00
Zoltan Kochan
ec97a31057 fix: print a better error message on prepare pkg failure (#5847)
ref #5845
2023-01-01 23:28:33 +02:00
Zoltan Kochan
339c0a7046 fix: run prepublish scripts of packages installed from Git (#5837)
close #5826
2022-12-25 21:18:18 +02:00
Zoltan Kochan
4ca53b0b50 refactor: group projects in different subdirectories (#5659) 2022-11-20 01:35:22 +02:00