mirror of
https://github.com/pnpm/pnpm.git
synced 2026-06-28 09:55:39 -04:00
## Motivation The [vlt.sh benchmarks](https://benchmarks.vlt.sh/) (2026-06-11 run, pacquet 0.11.3) show pacquet several times slower than the fastest package managers in the warm-metadata fresh-resolve cells (`cache`: 3.9–8.1x), the cold-cache frozen-install cells (`lockfile`: up to 10x on vue), and `clean`. Profiling the babylon and vue fixtures locally (macOS time profiles of the warm fresh resolve and the install tail) surfaced three independent causes, fixed here. ## Changes ### 1. Deprecation probing without manifest hydration (pacquet) With `minimumReleaseAge` active (the default), every range pick runs `filter_pkg_metadata_by_publish_date`, and any dist-tag pointing outside the maturity cutoff (`next`, `beta`, `canary`, a too-fresh `latest`) repopulates by scanning all candidates and reading each candidate's `deprecated` flag. Each read hydrated the full version manifest — a complete `serde_json` parse including the flattened catch-all map. On babylon's warm fresh resolve this was the single largest CPU consumer (~10 thread-seconds, all on the resolve task's critical path). `PackageVersions::is_deprecated` now answers from the raw fragment (substring pre-check, then a single-field deserialize with the same normalization as `PackageVersion::deprecated`), the tag-repopulation loop parses candidate versions once per filter call (mirroring the `parsedSemverCache` in pnpm's `filterPkgMetadataByPublishDate`), and the deprecated-pick fallback uses the probe instead of hydrating every version. **babylon warm fresh resolve: `resolve_workspace` 7.5s → 2.6s.** ### 2. Relative-symlink up-to-date check (pacquet) `force_symlink_dir` joined an existing link's relative contents onto the link parent and compared the result *verbatim* against the wanted target. Virtual-store links contain `..` segments (`../../<pkg>/node_modules/<name>`), so the joined path never compared equal and every up-to-date symlink was unlinked and recreated. Node's `path.relative` — which upstream `symlink-dir`'s `isExistingSymlinkUpToDate` builds on — resolves its arguments, so pnpm treats those links as current. Both sides now pass through `lexical_normalize`. The babylon install tail was dominated by exactly this unlink+symlink churn. **babylon warm install: 6.8s → 4.7s; warm frozen install: 4.2s → 2.3s.** ### 3. Default network concurrency floor 16 → 64 (pnpm + pacquet) The default was `min(64, max(workers * 3, 16))`. Downloads are I/O-bound, not CPU-bound: on a 4-vCPU CI runner the formula yields 16 concurrent requests, so a low-latency registry drains 600–1300-tarball installs 16 at a time while staying unsaturated — a large share of the cold-cell (`lockfile`/`clean`) gap on the benchmark runners. The default is now `min(96, max(workers * 3, 64))`; the `networkConcurrency` setting still overrides it. Applied to `@pnpm/installing.package-requester`, the lockfile-resolution verifier fan-out that mirrors its floor, and the same two spots in pacquet. Changeset included (minor). **This is a user-visible defaults change on both stacks — flagging it explicitly for review.** ## Local results (M-series macOS, vlt fixtures, isolated store/cache) | cell | before | after | |---|---|---| | vue `cache` | 1159 ms | **479 ms** | | vue `cache+lockfile` | 621 ms | **392 ms** | | vue no-op install | 48 ms | **41 ms** | | babylon `cache` | ~8.8 s | **4.75 s** | | babylon `cache+lockfile` | ~4.2 s | **2.27 s** | vue's warm cells are now ahead of every competitor measured locally; babylon's `cache` cell closed from ~2.5x behind the leader to ~1.35x (the remainder is the per-file store-integrity verify and per-file linking that the pnpm store contract requires). ## Validation - `cargo nextest`: registry, resolving-npm-resolver, resolving-deps-resolver, lockfile-verification, network, fs, tarball, package-manager, cli — 1300+ tests, all green; new unit tests cover the deprecation probe (string/bool/empty/corrupt shapes, nested-key false positives) and cross-parent relative-symlink reuse (fails without the fix). - Lockfile stability: `--lockfile-only` output is byte-identical before/after on vue; on babylon the resolved **package-version sets are identical across 6 runs (3 per binary)**. The babylon lockfile does flap between runs in the peer-suffix shape of `webpack-dev-server@5.2.2` (`(bufferutil@4.1.0)(utf-8-validate@5.0.10)` appearing/disappearing) — this is **pre-existing nondeterminism** reproducible with the unmodified binary against itself, in the optional-peer area; worth a separate issue. - Pre-push checks (fmt, taplo, `cargo doc -D warnings`, dylint) pass; eslint (root config) and `tsgo --build` pass for the two touched TS packages.
152 lines
5.8 KiB
YAML
152 lines
5.8 KiB
YAML
name: Test (reusable)
|
|
|
|
on:
|
|
workflow_call:
|
|
inputs:
|
|
node:
|
|
required: true
|
|
type: string
|
|
platform:
|
|
required: true
|
|
type: string
|
|
garnet:
|
|
required: false
|
|
type: boolean
|
|
default: false
|
|
secrets:
|
|
GARNET_API_TOKEN:
|
|
required: false
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
test:
|
|
name: Test
|
|
concurrency:
|
|
group: ${{ github.workflow }}-${{ github.ref }}-${{ inputs.platform }}-${{ inputs.node }}
|
|
cancel-in-progress: true
|
|
|
|
runs-on: ${{ inputs.platform }}
|
|
|
|
steps:
|
|
# A near-full "affected packages" sweep plus the pnpr Rust build
|
|
# can exhaust the hosted runner's free disk mid-test (the runner
|
|
# worker itself dies with "No space left on device"). Drop the
|
|
# preinstalled toolchains this job never uses (~25 GB) first.
|
|
- name: Free up runner disk space
|
|
if: ${{ runner.os == 'Linux' }}
|
|
run: |
|
|
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /opt/hostedtoolcache/CodeQL
|
|
df -h /
|
|
- name: Configure Git
|
|
run: |
|
|
git config --global core.autocrlf false
|
|
git config --global user.name "xyz"
|
|
git config --global user.email "x@y.z"
|
|
- name: Checkout Commit
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
with:
|
|
persist-credentials: false
|
|
- if: ${{ inputs.garnet }}
|
|
uses: garnet-org/action@2b7fc9d79b54f551b43358c27424a36064b3e078 # v2
|
|
with:
|
|
api_token: ${{ secrets.GARNET_API_TOKEN }}
|
|
- name: Install pnpm and Node
|
|
uses: pnpm/setup@b1cac37306e39c21283b9dd6cb0ac288fb35ba6b
|
|
with:
|
|
runtime: node@${{ inputs.node }}
|
|
- name: Verify Node version
|
|
shell: bash
|
|
env:
|
|
NODE_VERSION: ${{ inputs.node }}
|
|
# `pn node -v` falls back through `run`/`exec`, which would
|
|
# otherwise trigger a verifyDepsBeforeRun install just to print
|
|
# the version. Disable it so this step measures the runtime that
|
|
# pnpm/setup provisioned, not one a stray install pulled in.
|
|
pnpm_config_verify_deps_before_run: false
|
|
run: |
|
|
actual=$(pn node -v)
|
|
expected="v${NODE_VERSION}"
|
|
if [ "$actual" != "$expected" ]; then
|
|
echo "Expected Node version $expected but got $actual"
|
|
exit 1
|
|
fi
|
|
# npm is needed for preparing git-hosted dependencies (e.g. in dlx tests).
|
|
# `pnpm runtime set node` does not extract npm; the runner image's
|
|
# pre-installed Node toolchain provides it on PATH.
|
|
- name: Verify npm
|
|
run: npm --version
|
|
- name: Download compiled artifacts
|
|
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
|
|
with:
|
|
name: compiled-packages
|
|
- name: Extract compiled artifacts
|
|
run: tar -xzf compiled.tar.gz
|
|
# The test harness serves package fixtures through the in-repo
|
|
# `pnpr` server; `pnpr-prepare` turns the raw fixtures under
|
|
# `pnpr/.fixtures/packages` into the storage the server reads. Both
|
|
# are built from source so the tests exercise the current server
|
|
# (e.g. the install-accelerator endpoints) rather than a published
|
|
# `@pnpm/pnpr` binary that may predate it.
|
|
#
|
|
# The built binaries are cached and keyed on the Rust sources that
|
|
# produce them, so a run that only touches TypeScript restores them
|
|
# in seconds instead of recompiling. They are copied out of `target/`
|
|
# into a stable dir so `Swatinem/rust-cache`'s `target/` cleanup
|
|
# can't strip them before this cache saves.
|
|
- name: Restore prebuilt pnpr binaries
|
|
id: pnpr-bins
|
|
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
|
|
with:
|
|
path: .pnpr-bin
|
|
key: pnpr-bins-${{ runner.os }}-${{ hashFiles('rust-toolchain.toml', '**/Cargo.lock', '**/Cargo.toml', 'pnpr/**/*.rs', 'pacquet/**/*.rs') }}
|
|
- name: Install Rust
|
|
if: steps.pnpr-bins.outputs.cache-hit != 'true'
|
|
uses: ./.github/actions/rustup
|
|
with:
|
|
save-cache: ${{ github.ref_name == 'main' }}
|
|
shared-key: registry-prepare
|
|
- name: Build the pnpr server and registry fixture preparer
|
|
if: steps.pnpr-bins.outputs.cache-hit != 'true'
|
|
shell: bash
|
|
run: |
|
|
cargo build --locked --release -p pnpr --bin pnpr -p pnpr-fixtures --bin pnpr-prepare
|
|
mkdir -p .pnpr-bin
|
|
ext=""
|
|
[ -f target/release/pnpr.exe ] && ext=".exe"
|
|
cp "target/release/pnpr$ext" "target/release/pnpr-prepare$ext" .pnpr-bin/
|
|
- name: Export pnpr binary paths
|
|
shell: bash
|
|
run: |
|
|
ext=""
|
|
[ -f "$PWD/.pnpr-bin/pnpr.exe" ] && ext=".exe"
|
|
echo "PNPR_BIN=$PWD/.pnpr-bin/pnpr$ext" >> "$GITHUB_ENV"
|
|
echo "PNPR_PREPARE_BIN=$PWD/.pnpr-bin/pnpr-prepare$ext" >> "$GITHUB_ENV"
|
|
- name: Determine test scope
|
|
id: test-scope
|
|
shell: bash
|
|
env:
|
|
REF_NAME: ${{ github.ref_name }}
|
|
run: |
|
|
if [[ "$REF_NAME" == "main" || "$REF_NAME" == "chore/update-lockfile" || "$REF_NAME" == release/* ]]; then
|
|
echo "script=ci:test-all" >> "$GITHUB_OUTPUT"
|
|
echo "scope=all" >> "$GITHUB_OUTPUT"
|
|
else
|
|
git remote set-branches --add origin main && git fetch origin main --depth=1
|
|
if [ -n "$(git diff --name-only origin/main HEAD -- pnpm-workspace.yaml)" ]; then
|
|
echo "script=ci:test-all" >> "$GITHUB_OUTPUT"
|
|
echo "scope=all — pnpm-workspace.yaml modified" >> "$GITHUB_OUTPUT"
|
|
else
|
|
echo "script=ci:test-branch" >> "$GITHUB_OUTPUT"
|
|
echo "scope=affected packages" >> "$GITHUB_OUTPUT"
|
|
fi
|
|
fi
|
|
- name: Run tests (${{ steps.test-scope.outputs.scope }})
|
|
timeout-minutes: 70
|
|
shell: bash
|
|
env:
|
|
PNPM_WORKERS: 3
|
|
TEST_SCRIPT: ${{ steps.test-scope.outputs.script }}
|
|
run: pn run "$TEST_SCRIPT"
|