Commit Graph

39 Commits

Author SHA1 Message Date
Victor Sumner
878a7734a0 perf(cafs): skip rename on cold CAS writes, use atomic rename for recovery (#11087)
## Problem

Every file extracted to the CAS goes through a temp-file-plus-rename cycle: `writeFile(temp, buffer)` then `renameOverwriteSync(temp, fileDest)`. For a typical cold install with ~30k files, this adds ~30k extra rename syscalls.

## Solution

Use `writeFileExclusive()` with `{ flag: 'wx' }` (O_CREAT|O_EXCL) to write directly to the final CAS path when the file doesn't exist — skipping the temp+rename overhead. For recovery paths (corrupt/partial files, EEXIST races), fall back to the existing atomic temp+rename via `optimisticRenameOverwrite`.

### Write paths

- **File doesn't exist (common cold-install path)** → `writeFileExclusive` writes directly, no rename
- **File exists with correct integrity** → return immediately, no write
- **File exists with wrong integrity (corruption/crash)** → atomic temp+rename recovery
- **EEXIST (concurrent write)** → verify integrity; if OK return, otherwise atomic temp+rename recovery

### Concurrent safety

- `writeFileExclusive` (`O_CREAT|O_EXCL`) ensures only one process creates a given CAS file
- Recovery overwrites use the battle-tested `optimisticRenameOverwrite` + `pathTemp` for atomic replacement
- `verifyFileIntegrity` is non-destructive (no `unlinkSync` on mismatch), safe when another process may be mid-write
- A crash mid-`writeFileExclusive` can leave a partial file, recovered on next access via atomic temp+rename

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
2026-03-25 15:20:34 +01:00
Victor Sumner
6656baaea3 fix(cafs): update locker cache when file exists with correct integrity (#11085)
* fix(cafs): update locker cache when file exists with correct integrity

The CAS locker cache was not updated when a file already existed on disk
with correct integrity. This caused repeated verifyFileIntegrity calls
on subsequent lookups within the same process, adding unnecessary I/O.

* fix(test): assert locker cache value not just key existence

Strengthen the test to verify locker.get() returns the correct
checkedAt timestamp, not just that the key exists.
2026-03-25 02:12:00 +01:00
Zoltan Kochan
0d88df854f chore: update all dependencies to latest versions (#11032)
* chore: update all dependencies to latest versions

Update all outdated dependencies across the monorepo catalog and fix
breaking changes from major version bumps.

Notable updates:
- ESLint 9 → 10 (fix custom rule API, disable new no-useless-assignment)
- @stylistic/eslint-plugin 4 → 5 (auto-fixed indent changes)
- @cyclonedx/cyclonedx-library 9 → 10 (adapt to removed SPDX API)
- esbuild 0.25 → 0.27
- TypeScript 5.9.2 → 5.9.3
- Various @types packages, test utilities, and build tools

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: update unified/remark/mdast imports for v11/v4 API changes

Update imports in get-release-text for the new ESM named exports:
- mdast-util-to-string: default → { toString }
- unified: default → { unified }

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: resolve typecheck errors from dependency updates

- isexe v4: use named import { sync } instead of default export
- remark-parse/remark-stringify v11: add vfile as packageExtension
  dependency so TypeScript can resolve type declarations
- get-release-text: remove unused @ts-expect-error directives

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert runtime dependency major version bumps

Revert major version bumps for runtime dependencies that are bundled
into pnpm to fix test failures where pnpm add silently fails:
- bin-links: keep ^5.0.0 (was ^6.0.0)
- cli-truncate: keep ^4.0.0 (was ^5.2.0)
- delay: keep ^6.0.0 (was ^7.0.0)
- filenamify: keep ^6.0.0 (was ^7.0.1)
- find-up: keep ^7.0.0 (was ^8.0.0)
- isexe: keep 2.0.0 (was 4.0.0)
- normalize-newline: keep 4.1.0 (was 5.0.0)
- p-queue: keep ^8.1.0 (was ^9.1.0)
- ps-list: keep ^8.1.1 (was ^9.0.0)
- string-length: keep ^6.0.0 (was ^7.0.1)
- symlink-dir: keep ^7.0.0 (was ^9.0.0)
- terminal-link: keep ^4.0.0 (was ^5.0.0)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: restore runtime dependency major version bumps

Re-apply all runtime dependency major version bumps that were
previously reverted. All packages maintain their default exports
except isexe v4 which needs named imports.

Updated runtime deps:
- bin-links: ^5.0.0 → ^6.0.0
- cli-truncate: ^4.0.0 → ^5.2.0
- delay: ^6.0.0 → ^7.0.0
- filenamify: ^6.0.0 → ^7.0.1
- find-up: ^7.0.0 → ^8.0.0
- isexe: 2.0.0 → 4.0.0 (fix: use named import { sync })
- normalize-newline: 4.1.0 → 5.0.0
- p-queue: ^8.1.0 → ^9.1.0
- ps-list: ^8.1.1 → ^9.0.0
- string-length: ^6.0.0 → ^7.0.1
- symlink-dir: ^7.0.0 → ^9.0.0
- terminal-link: ^4.0.0 → ^5.0.0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert tempy to 3.0.0 to fix bundle hang

tempy 3.2.0 pulls in temp-dir 3.0.0 which uses async fs.realpath()
inside its module init. When bundled by esbuild into the __esm lazy
init pattern, this causes a deadlock during module initialization,
making the pnpm binary hang silently on startup.

Keeping tempy at 3.0.0 which uses temp-dir 2.x (sync fs.realpathSync).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add comment explaining why tempy cannot be upgraded

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert nock to 13.3.4 for node-fetch compatibility

nock 14 changed its HTTP interception mechanism in a way that doesn't
properly intercept node-fetch requests, causing audit tests to hang
waiting for responses that are never intercepted.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add comment explaining why nock cannot be upgraded

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: update symlink-dir imports for v10 ESM named exports

symlink-dir v10 removed the default export and switched to named
exports: { symlinkDir, symlinkDirSync }.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: revert @typescript/native-preview to working version

Newer tsgo dev builds (>= 20260318) have a regression where
@types/node cannot be resolved, breaking all node built-in types.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: vulnerabilities

* fix: align comment indentation in runLifecycleHook

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: pin msgpackr to 1.11.8 for TypeScript 5.9 compatibility

msgpackr 1.11.9 has broken type definitions that use Iterable/Iterator
without required type arguments, causing compile errors with TS 5.9.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 23:28:53 +01:00
Zoltan Kochan
5d5818e44f style: enforce node: protocol for builtin imports (#10951)
Add n/prefer-node-protocol rule and autofix all bare builtin imports
to use the node: prefix. Simplify the simple-import-sort builtins
pattern to just ^node: since all imports now use the prefix.
2026-03-13 07:59:51 +01:00
Zoltan Kochan
1c8c4e49f5 style: add eslint-plugin-simple-import-sort (#10947)
Add eslint-plugin-simple-import-sort to enforce consistent import ordering:
- Node.js builtins first
- External packages second
- Relative imports last
- Named imports sorted alphabetically within each statement
2026-03-13 02:02:38 +01:00
Dhruv Bhanderi
ebe166c30c fix(store): include devDependencies in bundled manifest (#10711) 2026-03-01 15:50:01 +01:00
Zoltan Kochan
56a59df674 perf: persist bundled manifest in store index to avoid reading package.json from CAFS (#10473)
close #10461
2026-02-17 12:03:08 +01:00
Zoltan Kochan
e2e0a321b3 perf: optimize how the integrities of files in the CAFS are stored (#10504) 2026-01-24 21:41:11 +01:00
Zoltan Kochan
40b107efa7 perf: migrate internal cache and index files to MessagePack serialization (#10500) 2026-01-23 01:31:09 +01:00
Zoltan Kochan
2ea64631eb fix: skip symlinks pointing outside package root in git and file deps (#10493) 2026-01-21 15:45:56 +01:00
Zoltan Kochan
caabba44ff fix: normalize Windows backslash path traversal attempts in tarball entry filenames
to prevent security vulnerabilities.
2026-01-15 17:10:17 +01:00
Zoltan Kochan
da112f7cb2 revert: "perf: use v8 serialize/deserialize instead of JSON (#9971)" (#10420)
close #10409
2026-01-13 15:16:33 +01:00
Zoltan Kochan
0048667db4 refactor: use Maps instead of Records (#10312) 2025-12-15 11:48:19 +01:00
Zoltan Kochan
cf9f14aa9e fix: update tempy 2025-08-28 14:00:51 +02:00
Zoltan Kochan
491a84fb26 feat: use ESM instead of commonjs (#9870) 2025-08-25 10:02:00 +02:00
Zoltan Kochan
27cbc09206 style: fix jest-related linting issues (#9894) 2025-08-22 21:56:49 +02:00
Zoltan Kochan
facd7656e8 refactor: always use extensions in relative imports (#9878) 2025-08-19 15:25:11 +02:00
Zoltan Kochan
7ec017e13f chore: move all tarball files out from Git LFS to a package dependency (#8885) 2024-12-18 03:06:51 +01:00
Zoltan Kochan
298e5dcafd feat!: create a separate directory in the store for index files (#8700) 2024-10-27 21:00:28 +01:00
Brandon Cheng
5fea44486e chore: use Git LFS for pnpm development (#8509)
* chore: set up git-lfs hooks

* ci: checkout lfs files on CI

According to https://github.com/actions/checkout, checkout out LFS files
defaults to false.

* chore: track .tgz files in Git LFS
2024-10-14 08:49:30 +02:00
Zoltan Kochan
db420ab592 refactor(cafs): create a separate function for creating index file path (#8508) 2024-09-08 14:43:09 +02:00
Khải
9b4f73caaf chore(scripts): typecheck-only (#8395)
* chore(scripts): typecheck-only

* feat: change all configuration

* feat: include pnpm/ and pnpm/test/

* chore(deps): remove unused dependency

* refactor(typescript-only): use find-packages

* refactor(typescript-only): refactor paths

* fix: typescript-only

* fix: update compile-only

* fix: compile pnpm

* fix: windows

* fix: windows

* chore: meta-updater

* refactor(tsconfig): remove explicit composite

* fix: path in windows

* feat: don't depend on cwd

---------

Co-authored-by: Zoltan Kochan <z@kochan.io>
2024-08-11 08:26:01 +02:00
Brandon Cheng
c7e1b6fae8 chore: configure TypeScript project references for tests (#8128)
* refactor: store link values before converting to references

* fix: use .sort() without localeCompare

https://github.com/pnpm/pnpm/pull/8128#discussion_r1614031566

> Nit, but you probably just want to call sort without a comparison
> function; these are already strings and locale compare is not a good
> comparison for anything but human readable strings since it will
> differ on different people's machines based on their language setting.
> I've hit this too many times before for code gen.

* feat: configure meta-updater to write test/tsconfig.json files

* fix: relative imports for __typings__

* chore: `pnpm run meta-updater`

* fix: explicitly use test/tsconfig.json for ts-jest
2024-05-31 12:48:13 +02:00
Zoltan Kochan
75840d6570 fix: resolve symlink dirs when uploading side-effects (#7706)
close #7691
close #7253
2024-02-24 15:26:14 +01:00
Zoltan Kochan
6cdbf11dea fix: unpacking tarballs that appear to be not USTAR or GNU TAR (#7677)
close #7120
2024-02-22 02:49:49 +01:00
Zoltan Kochan
b3947185c6 fix: unpacking tarballs that contain hard links (#7062) 2023-09-06 21:48:06 +03:00
Zoltan Kochan
b548f2f438 fix: don't hang while reading package.json from the store
close #7051
2023-09-05 19:32:54 +03:00
Zoltan Kochan
69430e13ae perf: verify package integrity with a worker (#7022) 2023-09-02 19:34:19 +03:00
Zoltan Kochan
9eddf19692 fix: extracting tarballs that have prefixes with spaces (#7013)
close #6991
2023-08-31 16:35:24 +03:00
Zoltan Kochan
e6353f964c perf: fetch directories to CAFS in worker (#6984) 2023-08-26 15:43:32 +03:00
Zoltan Kochan
667891991d test(cafs): fix on Windows 2023-08-25 15:07:05 +03:00
Martin Jesper Low Madsen
0fd9e6a6c4 fix(store): don't prematurely bail out of adding source files if ENOENT is thrown (#6932)
Broken symbolic links will cause a `stat'-call to throw resulting in an
arbitrary amount of promises that won't get to settle before the index is
returned.

Processing subdirectories in the following iteration in the event loop makes
this consistently reproducible.

---------

Co-authored-by: Martin Madsen <mj@blackbird.online>
Co-authored-by: Zoltan Kochan <z@kochan.io>
2023-08-25 00:41:49 +03:00
Zoltan Kochan
083bbf5907 perf: extract packages in a worker pool (#6850) 2023-08-14 02:07:39 +03:00
Zoltan Kochan
73f2b68267 fix: don't fail when multiple containers use the same store (#6853)
ref #6817
2023-07-24 19:58:58 +03:00
Zoltan Kochan
fe1c5f48d3 fix: make shorter temp file names in the store (#6845)
close #6842
2023-07-22 02:54:57 +03:00
Lev Chelyadinov
e57e2d3400 fix: don't fail on malformed tar headers (#6674)
close #5362
2023-06-15 02:55:34 +03:00
Zoltan Kochan
ee61ca4cb7 style: update eslint (#6236) 2023-03-18 14:57:29 +02:00
Zoltan Kochan
98d6603f31 fix: corrupted side effects cache should be ignored (#5930)
close #4997
2023-01-18 23:33:50 +02:00
Zoltan Kochan
4ca53b0b50 refactor: group projects in different subdirectories (#5659) 2022-11-20 01:35:22 +02:00