mirror of
https://github.com/pnpm/pnpm.git
synced 2026-06-01 12:41:16 -04:00
Closes #11687. ## What Cache the result of the post-resolution lockfile verification gate (#11583) so repeat installs against an unchanged lockfile skip the per-package registry round trips entirely. Persisted as JSON Lines at `<cacheDir>/lockfile-verified.jsonl`. The cache layer is policy-neutral. Today there's one verifier (`minimumReleaseAge`); future resolver-side verifiers (jsr trust, attestation, …) plug in by declaring their own `policy` slot and `canTrustPastCheck` comparator — no install-side changes. ## Why #11583 re-hits the registry on every install for every locked (name, version) pair. On warm/repeat installs where the lockfile hasn't moved, that's a stack of per-package round trips with nothing to show for them. This change makes the steady-state case effectively free without weakening the protection — the gate still runs in full whenever the lockfile changes, any verifier's policy tightens, or no record exists. ## How ### Cache lookup, in order The cache is **indexed by content hash** so git worktrees with identical lockfile bytes share a cache entry. A secondary path-keyed index drives the same-machine stat shortcut. 1. **`stat()` shortcut** — when a previous record for this exact `lockfilePath` matches today's `size + mtime + inode`, trust the cached hash without reading anything. Zero I/O beyond the stat. Microseconds. 2. **Content lookup** — hash the in-memory lockfile (not the file bytes — we already have the parsed object) and look up by content hash. Catches worktrees (same content, different path) and CI checkouts (same content, reset stat). On hit, append a refreshed path/stat entry so the next install at this path takes the stat shortcut. 3. **Any active verifier rejects the cached `policy`** — run the full gate. 4. **No record** — run the full gate. The in-memory object is hashed with `hashObject` from `@pnpm/crypto.object-hasher` (streaming, key-order-stable). ### Record shape ```json { "lockfile": { "hash": "<sha256 base64>", "path": "/abs/path/to/pnpm-lock.yaml", "size": 154, "mtimeNs": "1736245123000000000", "inode": "12345" }, "verifiedAt": "2026-05-17T...", "policy": { "minimumReleaseAge": 1440 } } ``` `policy` is the union of every active verifier's `policy` contribution. Verifiers checking the same logical policy (e.g. `minimumReleaseAge` honored by multiple registries) name it the same and share the slot — no resolver namespacing. ### File semantics - **Sync fs throughout** — the cache is consulted once before verification fan-out and recorded once after. No concurrent install work to overlap with; keeping the call sites straight-line. - **JSONL appends are atomic** on POSIX/NTFS, so parallel pnpm processes (monorepo installs, CI matrices sharing a cache) write without coordination. Latest record per `(path, hash)` tuple wins on read. - **Bounded file** — capped at ~1000 entries; compaction is triggered by a single `stat()` of the cache file (1.5 MiB byte budget) so we never parse the file on the steady-state path. When triggered, the tail is rewritten via tempfile + rename. - **No record on rejection** — a failing verification deliberately doesn't write a record; the next install must rerun the gate. - **Single hash per install** — the in-memory hash is computed lazily and reused: `tryLockfileVerificationCache` returns the precomputed stat+hash to `recordVerification` on a miss, and the stat-shortcut hit forwards the cached record's hash unchanged. ## Plumbing The verifier contract changed alongside the cache to make this composable without install-side knowledge of each policy: - **`@pnpm/resolving.resolver-base`** — `ResolutionVerifier` is now `{ verify, policy, canTrustPastCheck }` (was a bare function in #11583). Each resolver-side verifier owns its policy snapshot and the comparator that decides whether a cached policy is still trustworthy. - **`@pnpm/resolving.npm-resolver`** — `createNpmResolutionVerifier` returns the new shape: `policy: { minimumReleaseAge }`, `canTrustPastCheck` reads `minimumReleaseAge` from the merged cached bag. - **`@pnpm/resolving.default-resolver`** — `createResolutionVerifier` (singular, returning a combined function) → `createResolutionVerifiers` (plural, returning a `ResolutionVerifier[]`). No combinator; each verifier handles its own protocol short-circuit inside `verify`, so dispatch happens naturally at the install side. - **`@pnpm/installing.client`** — `Client.verifyResolution?` → `Client.resolutionVerifiers: ResolutionVerifier[]`. Same rename propagates through `@pnpm/store.connection-manager`, `@pnpm/testing.temp-store`, and `StrictInstallOptions`. - **`@pnpm/installing.deps-installer`** — new `verifyLockfileResolutionsCache.ts` (`tryLockfileVerificationCache` + `recordVerification`). `verifyLockfileResolutions` takes the verifier list plus `cacheDir` + `lockfilePath` as flat options; the cache fires when both are present, otherwise the gate runs without memoization. The dedup key for in-flight candidates includes a serialization of `resolution` so two entries sharing a (name, version) but pinned via different protocols don't collapse. Breaking but safe — `@pnpm/resolving.npm-resolver` hasn't been released since #11583 introduced the verifier abstraction, so no downstream consumer is on the old shape. ## Tests - **17 unit tests** in `verifyLockfileResolutionsCache.ts`: cache miss/hit, stat shortcut, size mismatch falling through to hash lookup, hash-fallback on reset stat, content change with matching size, stricter/weaker policy, missing-field policy rejection, multi-verifier policy merge (shared field stored once), worktree case (same content, different path), JSONL append semantics, malformed-line tolerance. - **12 integration tests** in `verifyLockfileResolutions.ts`: dedup of peer/patch-suffix variants, distinct-resolution dedup at the same (name, version), stable violation ordering, the 20-entry cap, multi-verifier fan-out (first failure wins), cache short-circuit on a passing run, no cache write on a rejecting run, empty-verifier-list passthrough. - **1 e2e test** in `pnpm/test/install/minimumReleaseAge.ts`: bundled CLI plumbing — install once to seed the lockfile, enable `minimumReleaseAge` + `cacheDir`, install again, assert the cache file lands at `<cacheDir>/lockfile-verified.jsonl` with the documented record shape. - Existing `minimumReleaseAge` (13) and `frozenLockfile` (12) suites still pass.
88 lines
3.2 KiB
TypeScript
88 lines
3.2 KiB
TypeScript
import { UNIVERSAL_OPTIONS } from '@pnpm/cli.common-cli-options-help'
|
|
import { docsUrl } from '@pnpm/cli.utils'
|
|
import type { Config, ConfigContext } from '@pnpm/config.reader'
|
|
import { type InstallOptions, mutateModulesInSingleProject } from '@pnpm/installing.deps-installer'
|
|
import { createStoreController, type CreateStoreControllerOptions } from '@pnpm/store.connection-manager'
|
|
import type { ProjectRootDir } from '@pnpm/types'
|
|
import { renderHelp } from 'render-help'
|
|
|
|
import { cliOptionsTypes } from './install.js'
|
|
|
|
export const rcOptionsTypes = cliOptionsTypes
|
|
|
|
export { cliOptionsTypes }
|
|
|
|
export const shorthands: Record<string, string> = {
|
|
D: '--dev',
|
|
P: '--production',
|
|
}
|
|
|
|
export const commandNames = ['fetch']
|
|
|
|
export function help (): string {
|
|
return renderHelp({
|
|
description: 'Fetch packages from a lockfile into virtual store, package manifest is ignored. WARNING! This is an experimental command. Breaking changes may be introduced in non-major versions of the CLI',
|
|
descriptionLists: [
|
|
{
|
|
title: 'Options',
|
|
|
|
list: [
|
|
{
|
|
description: 'Only development packages will be fetched',
|
|
name: '--dev',
|
|
shortAlias: '-D',
|
|
},
|
|
{
|
|
description: 'Development packages will not be fetched',
|
|
name: '--prod',
|
|
shortAlias: '-P',
|
|
},
|
|
...UNIVERSAL_OPTIONS,
|
|
],
|
|
},
|
|
],
|
|
url: docsUrl('fetch'),
|
|
usages: ['pnpm fetch [--dev | --prod]'],
|
|
})
|
|
}
|
|
|
|
type FetchCommandOptions = Pick<Config, 'production' | 'dev' | 'enableGlobalVirtualStore' | 'patchedDependencies'> & Pick<ConfigContext, 'rootProjectManifest' | 'rootProjectManifestDir'> & CreateStoreControllerOptions
|
|
|
|
export async function handler (opts: FetchCommandOptions): Promise<void> {
|
|
const store = await createStoreController(opts)
|
|
const include = {
|
|
dependencies: opts.production !== false,
|
|
devDependencies: opts.dev !== false,
|
|
// when including optional deps, production is also required when perform headless install
|
|
optionalDependencies: opts.production !== false,
|
|
}
|
|
await mutateModulesInSingleProject({
|
|
manifest: {},
|
|
mutation: 'install',
|
|
pruneDirectDependencies: true,
|
|
rootDir: process.cwd() as ProjectRootDir,
|
|
}, {
|
|
...opts,
|
|
ignorePackageManifest: true,
|
|
ignoreLocalPackages: true,
|
|
include,
|
|
modulesCacheMaxAge: 0,
|
|
pruneStore: true,
|
|
storeController: store.ctrl,
|
|
storeDir: store.dir,
|
|
resolutionVerifiers: store.resolutionVerifiers,
|
|
// Hoisting is skipped anyway,
|
|
// so we store these empty patterns in node_modules/.modules.yaml
|
|
// to let the subsequent install know that hoisting should be performed.
|
|
hoistPattern: [],
|
|
publicHoistPattern: [],
|
|
// virtualStoreOnly skips post-import linking (symlinks, bins, hoisting)
|
|
// even if ignorePackageManifest handling changes in the future.
|
|
virtualStoreOnly: true,
|
|
// Ensure fetch can populate the virtual store even when the user has
|
|
// enable-modules-dir=false in their config — fetch always needs node_modules/.pnpm
|
|
// (unless GVS is active, in which case enableModulesDir doesn't matter).
|
|
enableModulesDir: true,
|
|
} as InstallOptions)
|
|
}
|