Files
pnpm/resolving/npm-resolver/test/ifModifiedSince.test.ts
Zoltan Kochan 1310ab53c4 perf: close repeat-install and warm-resolve gaps (lazy lockfile, pre-runtime fast path, 304 freshness renewal) (#12364)
* perf(cli): keep the repeat-install fast path off the lockfile parse and thread spawns

The "Already up to date" short-circuit decides from manifest mtimes
alone (mirroring upstream checkDepsStatus, which never reads the wanted
lockfile), yet pacquet parsed pnpm-lock.yaml eagerly in State::init
before the check ran — a multi-millisecond YAML parse on every no-op
install, scaling with lockfile size (babylon's 720 KB lockfile dominated
its repeat-install wall time).

pnpm-lock.yaml now loads through a LazyLockfile (OnceLock-backed) that
install forces only after the fast path has passed on the run; add /
update / remove / outdated / the pnpr path force it up front, keeping
their behavior unchanged. The repeat-install regenerate branch probes
for the file's existence instead of its parsed contents, so the
fast path stays mtime-cheap.

The rayon global pool is likewise no longer built eagerly at startup:
the worker count is published via RAYON_NUM_THREADS (set in fn main
while the process is still single-threaded) so the pool spawns lazily
on first parallel use — commands that never reach a parallel phase no
longer pay 2×CPUs thread spawns.

A corrupt lockfile now surfaces its parse error when the install
actually reads the file; an up-to-date project with an unreadable
lockfile reports "Already up to date" exactly as pnpm does.

* perf(cli): finish up-to-date installs before building the async runtime

A plain `pacquet install` that is already up to date now completes on
the main thread, before the tokio runtime, the rayon pool, the HTTP
client, or any install state exists. The new
`install_already_up_to_date` twin of the repeat-install short-circuit
reuses the exact same workspace discovery and
`check_optimistic_repeat_install` inputs as `Install::run`, and the
CLI invokes it from the (now synchronous) `main` after clap parsing.

Gates mirror everything that would make the full path behave
differently: `--frozen-lockfile` / `--lockfile-only`, a configured
pnpr server (that path never runs the optimistic check), `--recursive`
/ `--filter`, config dependencies, and pnpmfile updateConfig hooks
(both can mutate the config the check compares against). Any gate or
error falls through to the full install path, which re-runs the check
and reproduces failures with their established error shape; the
"Already up to date" + summary emissions are byte-identical.

Repeat-install instruction count on the vue fixture drops from ~203M
to ~41M retired instructions — within a rounding error of the
`pacquet --version` floor (~39M).

* perf(resolving): renew metadata-mirror freshness on 304 Not Modified

The minimumReleaseAge freshness shortcut treats a metadata mirror
younger than the cutoff as authoritative and resolves without touching
the network. But a 304 revalidation never rewrote the mirror file, so
its mtime froze at the last 200 response: once a cached packument grew
older than minimumReleaseAge (24h by default), every subsequent install
re-validated every package against the registry, forever.

A 304 proves the cached packument equals the registry's current
document, so the validation clock legitimately restarts at the
response: bump the mirror's mtime to now (fire-and-forget — a
read-only cache dir only costs the next install another conditional
request). Applied to both stacks: pnpm's pickPackage notModified
branch and pacquet's fetch_full_metadata_cached 304 path.

On a vue-fixture install with a stale cache, the second warm resolve
drops from ~2s (520 conditional requests) to ~250ms (zero requests).

* style(resolving): use clippy's preferred Duration units in the 304 mtime test

CI clippy denies duration_suboptimal_units; from_hours / from_mins
replace the hand-multiplied from_secs values.

* style(package-manager): use clippy's preferred Duration unit in the sync fast-path test

Same duration_suboptimal_units deny as the previous commit, one site
CI's clippy surfaced after it stopped at the first failing crate.

* fix(lockfile): address review — dir-addressed LazyLockfile, read-open 304 touch

LazyLockfile resolved pnpm-lock.yaml against the process cwd while the
CLI honours a canonicalized --dir without chdir, so the deferred load
and the existence probe could consult a different lockfile than the
rest of the install (which derives lockfile_path from the manifest's
directory). The lazy handle now carries the manifest's directory and
loads via load_wanted_from_dir; lockfile-disabled config gets an
explicit disabled() constructor. The pre-runtime fast path builds its
handle from the same directory, keeping verdict parity with
Install::run.

renew_mirror_freshness opened the mirror with append just to bump the
mtime; set_modified only needs a file handle plus ownership (futimens
semantics), so a plain read-open also covers mirrors whose mode
dropped write permission.

* test(integrated-benchmark): compare best-of-N samples in the slow-start proxy test

The test raced two single wall-clock samples, and a loaded CI runner
can inflate the ramped-vs-flat comparison in either direction (macOS
runner measured flat 318ms vs ramped 304ms against a ~66ms model).
Scheduler stalls only ever inflate a sample, while slow start's ramp
overhead is structural and survives in every sample — so the minimum
of several runs per side is the noise-resistant estimator.

* chore(deps): bump esbuild to 0.28.1 to clear GHSA-gv7w-rqvm-qjhr

The new advisory (install-module RCE via NPM_CONFIG_REGISTRY,
patched in 0.28.1) fails the audit gate. 0.28.1 was published within
the minimumReleaseAge window, so the patched version is excluded from
the age gate — the same mechanism pnpm audit --fix uses — including
its '@esbuild/*' platform packages, whose versions move in lockstep
with the root package.

* fix(lockfile): make the unloaded presence probe match the loader's absence rules

The loader treats an empty file and an env-only combined document as
an absent wanted lockfile (Ok(None)), but is_loaded_or_on_disk probed
bare file existence, so the repeat-install path could skip restoring a
semantically-missing pnpm-lock.yaml. The probe now reads the file and
checks the main document is non-empty (Lockfile::wanted_exists_in_dir)
— the loader's exact absence rules, still without the YAML parse.

* fix(cli): build the rayon pool after the fast-path gate instead of injecting env

Publishing the worker count through RAYON_NUM_THREADS leaked the
variable into every child process the install spawns — lifecycle
scripts, node probes, git — and pnpm exposes no such variable to
scripts. Build the global pool with ThreadPoolBuilder again, but only
once the repeat-install fast path has declined: real installs pay
exactly the cost they always did, the no-op path still spawns no
workers, and the process environment stays untouched (which also
drops the unsafe set_var and its single-threaded contract).

* fix(lockfile): treat only NotFound as absence in the presence probe

A permission or I/O failure reading pnpm-lock.yaml reported the file
as absent, which would send the repeat-install path into the
regenerate-on-missing branch — overwriting an existing lockfile it
merely could not read. Only NotFound counts as absent now; any other
read failure reports presence, and the real load surfaces the
underlying error when the contents are actually needed.

* fix(resolving): open the mirror write-capable for the 304 touch, read-only as fallback

Windows' set_modified requires write-attributes access on the handle,
so the read-only open silently failed there (caught by the Windows CI
run of a_304_renews_the_mirror_mtime). Append-mode open carries that
access; the read-only fallback still covers Unix mirrors whose mode
dropped write permission, where timestamp syscalls need ownership
rather than write access.

* fix(package-manager): never short-circuit partial installs as already up to date

add and remove mutate the manifest in memory and persist it only after
Install::run returns, so the on-disk mtimes the optimistic
repeat-install check reads still describe the pre-mutation project.
With a fresh workspace state, `pacquet add X` right after a clean
install reported "Already up to date", skipped the entire install,
and then saved a package.json declaring a dependency that was never
resolved, lockfiled, or materialized (self-healing on the next run,
which sees the newer manifest mtime).

Gate the short-circuit on is_full_install, mirroring upstream
installDeps calling checkDepsStatus only for the plain-install
mutation, never for installSome / uninstallSome. The new
partial_install_disables_optimistic_short_circuit test fails without
the gate.

The bug predates this PR (the KeepAll gate has carried add since the
optimistic path landed) — surfaced by CodeRabbit review on
pnpm/pnpm#12364.
2026-06-13 00:10:06 +02:00

172 lines
5.6 KiB
TypeScript

import fs from 'node:fs'
import path from 'node:path'
import { afterEach, beforeEach, expect, test } from '@jest/globals'
import { ABBREVIATED_META_DIR } from '@pnpm/constants'
import { createFetchFromRegistry } from '@pnpm/network.fetch'
import { createNpmResolver } from '@pnpm/resolving.npm-resolver'
import { fixtures } from '@pnpm/test-fixtures'
import type { Registries } from '@pnpm/types'
import { loadJsonFileSync } from 'load-json-file'
import { temporaryDirectory } from 'tempy'
import { getMockAgent, retryLoadJsonFile, setupMockAgent, teardownMockAgent } from './utils/index.js'
const f = fixtures(import.meta.dirname)
const registries: Registries = {
default: 'https://registry.npmjs.org/',
}
/* eslint-disable @typescript-eslint/no-explicit-any */
const isPositiveMeta = loadJsonFileSync<any>(f.find('is-positive.json'))
/* eslint-enable @typescript-eslint/no-explicit-any */
const fetch = createFetchFromRegistry({})
const getAuthHeader = () => undefined
const createResolveFromNpm = createNpmResolver.bind(null, fetch, getAuthHeader)
afterEach(async () => {
await teardownMockAgent()
})
beforeEach(async () => {
await setupMockAgent()
})
test('use local cache when registry returns 304 Not Modified', async () => {
const cacheDir = temporaryDirectory()
// Write cached metadata with etag to disk in NDJSON format:
// Line 1: cache headers, Line 2: registry metadata
const cacheDir2 = path.join(cacheDir, `${ABBREVIATED_META_DIR}/registry.npmjs.org`)
fs.mkdirSync(cacheDir2, { recursive: true })
const headers = JSON.stringify({ etag: '"abc123"', modified: isPositiveMeta.modified })
fs.writeFileSync(
path.join(cacheDir2, 'is-positive.jsonl'),
`${headers}\n${JSON.stringify(isPositiveMeta)}`,
'utf8'
)
// Registry returns 304 Not Modified — verify conditional headers are sent
getMockAgent().get(registries.default.replace(/\/$/, ''))
.intercept({
path: '/is-positive',
method: 'GET',
headers: {
'if-none-match': '"abc123"',
},
})
.reply(304, '')
const { resolveFromNpm } = createResolveFromNpm({
storeDir: temporaryDirectory(),
cacheDir,
registries,
})
const resolveResult = await resolveFromNpm(
{ alias: 'is-positive', bareSpecifier: '^3.0.0' },
{}
)
expect(resolveResult!.resolvedVia).toBe('npm-registry')
expect(resolveResult!.id).toBe('is-positive@3.1.0')
})
test('a 304 Not Modified renews the metadata file mtime so the publishedBy freshness shortcut can fire again', async () => {
const cacheDir = temporaryDirectory()
const metaDir = path.join(cacheDir, `${ABBREVIATED_META_DIR}/registry.npmjs.org`)
fs.mkdirSync(metaDir, { recursive: true })
const metaPath = path.join(metaDir, 'is-positive.jsonl')
const headers = JSON.stringify({ etag: '"abc123"', modified: isPositiveMeta.modified })
fs.writeFileSync(metaPath, `${headers}\n${JSON.stringify(isPositiveMeta)}`, 'utf8')
// Age the mirror far past any maturity cutoff.
const aged = new Date(Date.now() - 365 * 24 * 60 * 60 * 1000)
fs.utimesSync(metaPath, aged, aged)
getMockAgent().get(registries.default.replace(/\/$/, ''))
.intercept({
path: '/is-positive',
method: 'GET',
headers: {
'if-none-match': '"abc123"',
},
})
.reply(304, '')
const { resolveFromNpm } = createResolveFromNpm({
storeDir: temporaryDirectory(),
cacheDir,
registries,
})
const resolveResult = await resolveFromNpm(
{ alias: 'is-positive', bareSpecifier: '^3.0.0' },
{}
)
expect(resolveResult!.id).toBe('is-positive@3.1.0')
// The touch is fire-and-forget, so poll briefly instead of asserting
// immediately.
const renewed = () => fs.statSync(metaPath).mtime.getTime() > aged.getTime() + 1000
await new Promise<void>((resolve) => {
const start = Date.now()
const timer = setInterval(() => {
if (renewed() || Date.now() - start > 5000) {
clearInterval(timer)
resolve()
}
}, 50)
})
expect(renewed()).toBe(true)
})
test('store etag from 200 response in cache', async () => {
const cacheDir = temporaryDirectory()
const responseHeaders = {
etag: '"xyz789"',
}
getMockAgent().get(registries.default.replace(/\/$/, ''))
.intercept({ path: '/is-positive', method: 'GET' })
.reply(200, isPositiveMeta, { headers: responseHeaders })
const { resolveFromNpm } = createResolveFromNpm({
storeDir: temporaryDirectory(),
cacheDir,
registries,
})
const resolveResult = await resolveFromNpm(
{ alias: 'is-positive', bareSpecifier: '^3.0.0' },
{}
)
expect(resolveResult!.resolvedVia).toBe('npm-registry')
expect(resolveResult!.id).toBe('is-positive@3.1.0')
// Verify etag was saved to disk cache
const cachePath = path.join(cacheDir, `${ABBREVIATED_META_DIR}/registry.npmjs.org/is-positive.jsonl`)
// eslint-disable-next-line @typescript-eslint/no-explicit-any
const savedMeta = await retryLoadJsonFile<any>(cachePath)
expect(savedMeta.etag).toBe('"xyz789"')
})
test('fetch without conditional headers when no local cache exists', async () => {
// No cache file → no ETag/Last-Modified to send → normal 200 response
getMockAgent().get(registries.default.replace(/\/$/, ''))
.intercept({ path: '/is-positive', method: 'GET' })
.reply(200, isPositiveMeta)
const cacheDir = temporaryDirectory()
const { resolveFromNpm } = createResolveFromNpm({
storeDir: temporaryDirectory(),
cacheDir,
registries,
})
const resolveResult = await resolveFromNpm(
{ alias: 'is-positive', bareSpecifier: '^3.0.0' },
{}
)
expect(resolveResult!.resolvedVia).toBe('npm-registry')
expect(resolveResult!.id).toBe('is-positive@3.1.0')
})