mirror of
https://github.com/pnpm/pnpm.git
synced 2026-10-10 16:22:02 -04:00
ea9f3bf4a0bb5ceff802f614c28fbb078b8b2f5f
152
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Zoltan Kochan
|
b1094872fa |
fix(cli): restore package script completion (#15086)
The completion server previously offered only candidates from clap's command and option definitions, leaving the run script position empty. Initialize clap metadata before scanning boolean options and register the run-script alias. Track positional arguments and directory options in the completion context. Reuse the npm project prefix lookup and manifest reader to return sorted script names. Share workspace-root lookup with command execution, preserve explicit directories, and respect equals-only options. Escape zsh candidate separators after prefix filtering. Read manifests only at the run script position, before configuration or hooks are initialized, and propagate malformed manifest errors. The missing-script regression affects v12 only. The TypeScript v11 implementation already reads scripts from the project manifest. A related Bash glob expansion bug affects both versions and is fixed in both generated scripts. Preserve candidate lines without splitting or pathname expansion, quote Bash candidates for insertion, escape PowerShell candidate syntax, and normalize its quoted prefixes. Reuse the short-option scanner so combined directory and workspace-root flags select the same project as command execution. Closes pnpm/pnpm#15034. |
||
|
Efe Baran DurmazandZoltan Kochan
|
9c1e317110 |
feat(cmd-shim): make project bin shims relocatable (#15041)
Project bin shims embedded absolute NODE_PATH entries and target markers, and .bin/node used an absolute symlink. Moving or copying the project could leave binaries resolving dependencies from the original location. Add relocatable_root to LinkBinsOptions. On Unix, write in-root NODE_PATH entries and target markers relative to the shim directory, and make in-root node runtime links relative. Apply the same option when injected dependencies are relinked. Resolve the shim directory physically with cd -P and pwd -P for both absolute and relative invocations. Node normalizes parent segments lexically, so an unresolved directory symlink can send NODE_PATH outside the project. This adds a subshell for project shims using a physical directory anchor. Escape the relative segments for double-quoted sh strings and stop if the directory cannot be resolved. Upgrade existing absolute in-root node links before the warm-install shortcut. Preserve already-relative links without recreating them. Keep Windows rendering and out-of-root shim behavior unchanged. Resolve bin directories, target parents and extra NODE_PATH entries before computing relative paths, including symlinked and not-yet-created directories. Broader moved-node_modules support remains outside this PR. Related to pnpm/pnpm#6937. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
|
871ca5f9b6 |
feat(cli/add): package urls (#14949)
`pnpm add` reads a `pkg:` selector as the Package URL specification defines it and routes it to the ecosystem its type names: npm to package.json, cargo to Cargo.toml, and pypi to pyproject.toml. Each purl is rewritten into the selector that ecosystem's own protocol already takes, so it inherits that path's flags, defaults, and errors instead of growing a second set. Every decoded component is validated before the pieces are joined back together. A name followed by a spec is how pnpm spells an npm alias, so a name or a version carrying a decoded at-sign or colon would otherwise redirect the install to another package. Qualifiers and subpaths are rejected rather than dropped: a repository_url qualifier or a subpath that pnpm ignores installs something other than what the selector names. AddArgs::package_names now holds the selectors the npm add path receives rather than the ones argv carried, which is what lets a purl reach the --global and --config forms as well. AddPipeline keeps only the selectors routed to another ecosystem, so the split has a single home, and dispatch_install::add delegates that split to a named step to stay within the local-binding limit perfectionist enforces. A purl always names a package to install. pnpm add node@22.0.0 records a runtime and pnpm add npm@11.0.0 records the project's package manager, both from a bare name a person typed, so the package-manager and runtime recognition skips a selector a purl was rewritten into. That mark belongs to the request rather than to its text. package_names holds AddRequest values, each carrying its own selector and whether a purl was rewritten into it, so two requests that normalize to one selector keep their own meanings and no second list has to be matched against or kept in sync. A rejected selector is quoted back by what it decodes to, because an encoded separator would otherwise hide a user:pass@ authority from a check made on the raw text. What decoding leaves unreadable to that check is stopped at instead: a colon followed by an at-sign is how a URL spells userinfo, and nothing pnpm installs pairs the two past its protocol, so no list of the characters that can sit between a scheme and its slashes has to be complete. --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
a9ee095ca7 |
feat(python): lock for several platforms and Python versions (#14970)
`pylock.toml` was resolved for the machine that ran the install: one marker environment, one wheel tag order, one wheel per distribution. A lockfile a repository commits could therefore not serve Linux CI and macOS or Windows contributors at once, which every repository surveyed in pnpm/pnpm#14945 needs. `python.platforms` and `python.pythonVersions` now name the environments `pylock.toml` is resolved for. Every platform is paired with every version, and a list left empty is the platform or the version of the interpreter running the install, so declaring neither keeps the previous behavior. Each environment is resolved on its own and the answers are merged into one lockfile: a package pins the wheel every environment takes for it and carries the PEP 751 `marker` saying which of them install it. `[tool.pnpm]` records the declared environments, so changing them resolves the project again. The interpreter reports what a declared environment resolves as, through the same host helper that reports its own tags and markers. A platform is named as a Rust target triple, or as an architecture and the libc baseline its wheels are built against; `linux`, `macos` and `windows` are short names for the most common three. An architecture the triple spells differently from the wheel tag is translated, and a baseline outside the glibc 2 or musl 1 series is refused. Two names for one environment are one environment, told apart by what the interpreter reports them as. `pnpm install` narrows the lockfile to the environment its interpreter matches: it installs the packages that environment's marker selects and, for each, the pinned wheel whose tags the interpreter prefers. An interpreter none of the declared environments stand for is refused, since nothing in the lockfile answers for it. A declared environment answers for a platform and a Python version and nothing else. A requirement whose marker it leaves undecided is refused rather than locked under a claim the resolution did not make. That covers a requirement gated on the kernel release, and one gated on a patch release of a minor version declared without one. Declared environments are resolved locally, since a pnpr server answers for one interpreter. An index page is fetched once per run and read back from its cache for every environment after the first. Separately, the rule that decides whether a lockfile's marker names the full interpreter version counted only release segments, so a locked package requiring `<3.12.0.post1` was claimed for every 3.12 patch release. A bound whose version falls between two releases now tells them apart. Related to pnpm/pnpm#14945. |
||
|
Zoltan Kochan
|
417601da4a |
fix(python): read an unparsable Requires-Python as none at all (#14920)
Every file an index lists is parsed when a distribution's candidates are read, so one release publishing a `Requires-Python` that is not a version specifier aborted the whole resolution. `openpyxl` 3.0.0 through 3.0.7 publish `>=3.6,`, which put `openpyxl` out of reach of every project, at any version: a release is immutable, so nothing the project does can correct the value. An unreadable `Requires-Python` is now read as if the release declared no interpreter range, which is what pip does. All three readers of the field agree on that: the index page a candidate comes from, the wheel metadata a resolution steps through, and the marker a solved lockfile writes. Closes pnpm/pnpm#14910 |
||
|
Zoltan Kochan
|
224aed744b |
refactor: move cmd-shim into the pnpm monorepo (#14893)
Import cmd-shim source, tests, and snapshots from pnpm/cmd-shim at 83f9b9449c2adeec26e1cc2d03ef0fe82d2e41ca. Maintaining the shim alongside its consumers removes the need for coordinated changes across repositories. Publish the package as `@pnpm/bins.cmd-shim` and replace external dependencies with workspace links. Preserve the original BSD-2-Clause license, including its copyright notice, and keep the manifest updater from replacing it with MIT. Reuse `@pnpm/fs.graceful-fs` and integrate the existing Node.js tests with the workspace test scripts. Retain the existing snapshots and align the version with the TypeScript workspace's required 1100.x band. |
||
|
Ayush SinghandZoltan Kochan
|
584b6c8388 |
fix(resolving): encode full registry path into metadata cache key (#14081)
Registry metadata mirrors were keyed on host[:port], so several registries served from one host under different path prefixes shared one directory and could answer with each other's versions, integrity hashes and tarball URLs. Both stacks now key the mirror on <scheme>%3A+<host>[+<port>][%2F<path>][%5F<sha256>]. The host and every path segment are percent-escaped down to [A-Za-z0-9._-], so a `%` or `+` in a key is always one the encoder wrote and the key can carry no path separator, no character Windows rejects in a filename, and no glob metacharacter — the cache commands feed the key to a glob whose matches `pnpm cache delete` removes. The scheme is included because http and https at one host and path are two different trust domains: metadata served over http can be rewritten in transit and must never reach a resolution configured for https. The separators are percent-escapes because every key pnpm wrote before this change was a URL host, which can never hold a `%`; no new key can therefore land on the stale directory of an unrelated one whose hostname held a separator, which would otherwise let `https://nexus/npm/` read what was cached for `https://nexus_npm/`. A path that is not all lowercase gets a sha256 suffix, the guard encodePkgName already applies to package names, so HFS+ and NTFS cannot merge two registries; a trailing `.` is escaped because Win32 strips one; and a key too long for a 255-byte filename is replaced by its own hash. Only the one trailing slash the resolver itself appends is normalized away; a repeated slash reaches the registry as a distinct request path and stays in the key. Every cache directory is renamed, so the first install after upgrading refetches registry metadata once. The package store is untouched. `pnpm cache view` decodes the key back to the registry rather than replacing `+` with `:`, the registry URL is redacted in both stacks' errors, and the Rust diagnostic codes now match pnpm's. Closes pnpm/pnpm#13558. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
3610b85e30 |
feat(pnpr): serve container images (#14629)
Adds OCI as a fourth ecosystem beside npm, Cargo, and Python: a hosted image registry that docker, podman, and skopeo push to and pull from. The distribution API cannot be moved under a path prefix. A client derives the API root from the image reference's host, so `pnpr.example.com/acme/app:1.0` always requests `/v2/acme/app/manifests/1.0`. `/v2/` therefore mounts at the host root whatever else is served, and the repository name alone selects the registry through the declared-provenance rules the other surfaces use. Artifactory has to burn the first path segment as a repository key for want of that invariant; pnpr does not, so the image name stays the image name. `/oci/v2/` and `/oci/~<name>/v2/` are served too, for podman and containerd, whose registry configuration does accept a path. Repository names are `/`-joined, which the pattern language had no shape for. `PackagePattern::parse` now takes the ecosystem and offers each one only the wildcard its names can carry: npm keeps `@scope/*` and `@*/*`, images get `<namespace>/*` over a single leading component, and Cargo and PyPI get neither, since a flat name could never match one. A `packages:` key is normalized through that language rather than through the name rules, so a wildcard key parses as itself. Blob uploads stream to a local file across requests instead of buffering in memory, and are verified against the promised digest before anything is stored. The manifest write is the commit point and rides the existing publish journal; blobs land outside it, content-addressed and invisible until a manifest names them, which is what leaves unreferenced blobs for a collector rather than half-publishing a release. A tag carries the time it last moved, so a transaction recovered after a crash cannot drag one back to an older manifest. `Basic` credentials now carry a token as the password under any username, which is how `docker login` sends them, and `GET /v2/` challenges an anonymous caller even where reads are open: a client settles its authentication scheme on that one response, so a 200 would leave it no way to authenticate a push. Verified end to end against docker 29.7.2 (login, push, pull, run), podman 5.8.4, and skopeo 1.22.2, plus an anonymous pull and a push refused after logout. The two client stacks take different upload paths and between them cover both: moby sends a blob monolithically, while containers/image chunks it. Not yet served: proxying an upstream image registry, blob collection, the referrers API, cross-repository blob mounts, and ranged blob downloads. |
||
|
Zoltan Kochan
|
6ba99fb386 |
feat(pnpr): share cargo compilation caches (#14620)
Expose named compiler caches through the WebDAV subset used by sccache. Apply pnpr account access and publication policies before buffering uploads, including existing token restrictions. Entries are immutable and verified against a digest bound to their cache scope, key, and bytes before serving. Reuse the shared artifact store's filesystem/S3 selection, quota accounting, publication lifecycle, and orphan reclamation. Bound compiler uploads to two concurrent bodies per server and reject excess requests before buffering. Store digest and payload as separate buffers in one conditional object write. Use metadata-only HEAD and duplicate checks; GET still verifies content before serving it. Stock sccache trusts the server and allowed publishers; it does not verify pnpm signed envelopes. Document trusted CI publication and HTTPS requirements. Also document sccache 0.17's absolute Rust build-path requirement and its read-only multilevel behavior: remote hits backfill disk, but new compilation misses are not cached locally when a tier is read-only. Add HTTP, policy, integrity, quota, and real Cargo integration coverage. Install sccache for Rust CI and local test setup. Extract the existing miette diagnostic normalization into a shared test helper to fix a shim assertion exposed by the full suite's longer temporary paths. |
||
|
Zoltan Kochan
|
09b546affe |
feat(pnpr): publish across ecosystems in one transaction (#14608)
`PUT /-/pnpr/v0/publish` takes a batch whose entries each name their ecosystem and carry what that surface's own publish endpoint takes, with the binary parts base64-encoded. An entry that names none is an npm publish document, so a body the npm batch endpoint accepts is already a valid one. The address is in pnpr's own namespace rather than beside the npm batch endpoint, which is part of the npm surface and answers under `/npm/` with it. `GET /-/pnpr` advertises the protocol as `publish: [0]`, and now answers for a registry-only tier, which has a pnpr protocol of its own for the first time. Every entry is parsed, authorized and verified before any of them is staged, every blob is staged before any of them is committed, and the commit is the single journal transaction the publish flow already had: a release that spans ecosystems becomes visible all at once. The per-ecosystem publish endpoints keep their own behavior; they and the batch now share one staging path, `StagedPublish`, which carries the ecosystem of the package it staged. Closes one checkbox of pnpm/pnpm#14599. |
||
|
Zoltan Kochan
|
1e83fc5f8b |
feat(pnpr): serve Cargo and Python registries beside npm (#14598)
Add Cargo and Python registry protocols to pnpr. Concrete registries declare an ecosystem, and routing filters mixed router sources by the requested protocol. Ecosystem prefixes provide default and named registry endpoints while preserving the original npm aliases. Reuse hosted document and blob storage, package access rules, upstream metadata caching, and verified artifact streaming across the new surfaces. Keep upstream artifact cache identities bound to metadata checksums. Check metadata before cache lookup so changed or removed entries take effect. Honor package-specific Cargo privacy rules when advertising auth-required. Require authentication before reading Python upload bodies. Bound multipart boundaries and use the existing regex crate for delimiter searches, checking boundary suffixes so binary lookalikes remain part of the uploaded file. Reject Cargo archive traversal, links, oversized decompressed streams, and manifests whose package identity differs from the publication metadata. Reject package configuration keys that collide after normalization. Apply route allowlists to Cargo and Python metadata and artifact fetches, including redirects. Require secure same-origin destinations for configured headers. Rebuild headers on every approved redirect so cross-origin metadata and artifact downloads succeed without forwarding upstream credentials. Share this redirect loop with the existing network metadata fetcher and retain its request-budget guard while response bodies are read. Transfer permits to streamed artifact responses so body consumption or cancellation releases them. Keep one pnpr timeout budget across redirects and the final response body. Stream through a bounded producer whose deadline runs independently of downstream polling. Reject hosted Python blobs absent from publication metadata. Abort publication on an immutable object-store conflict and remove the conflicting staged file. The filesystem backend remains single-writer; replicated deployments use the object-store backend. Journaled publication across all registry surfaces remains tracked in pnpm/pnpm#14599. Add protocol unit tests and HTTP regressions for publication, routing, authentication, content negotiation, caching, and integrity validation. |
||
|
Zoltan Kochan
|
427bece813 |
feat(cli): integrate python with shared install and artifact lifecycles (#14586)
Exercise the multi-ecosystem architecture through a usable Python integration, not test-only metadata writers. Keep Python requirement, marker, lockfile and environment semantics separate from npm and Cargo. Reuse the install-wide HTTP/auth budget, verified artifact ingestion, CAS and store index. Extract a shared install lifecycle without ecosystem-specific branches. Native tasks declare metadata footprints and return prepared projections. Settle all work before publishing Cargo or Python state, and reverse attempted publications before restoring metadata on failure. Retain resources when rollback fails so recovery remains possible. Enroll npm with its existing in-place materialization semantics, and keep npm-only dispatch on its early path. Do not unify native resolvers, lockfile formats, target identity or package layouts. Consolidate archive ingestion below the ecosystem boundary. Share cache validation, authenticated requests, extraction retries and publication across tarballs and ZIPs, retaining tar streaming and ZIP decoding. Test the shared contracts across both formats and preserve npm fast paths. Use standard pylock.toml, independently validated with uv. New features target pnpm v12 only. Shared archive URL-redaction fixes also cover pnpm v11. Related to pnpm/pnpm#14566 and pnpm/rfcs#34. |
||
|
Zoltan Kochan
|
19ba4d4bfd |
fix(network): use the system resolver on Linux (#14471)
On Linux the install client used reqwest's hickory-dns feature. Hickory parses /etc/resolv.conf with the resolv-conf crate, which rejects the whole file on any `options` token it does not recognise, including the `no_tld_query` spelling glibc accepts as an alias of `no-tld-query`. reqwest's Hickory glue swallows that parse error and silently builds a resolver against Google's 8.8.8.8/8.8.4.4, so pnpm never contacted the configured nameserver: fetches failed wherever public DNS is blocked and private registry names leaked to the public internet. Use the capped native getaddrinfo resolver on every platform, as macOS and Windows already do, and drop the hickory-dns feature. The system resolver is what pnpm 11 (Node's dns.lookup) and every other client on the host use, it honours nsswitch.conf sources Hickory bypasses, and it has no public-DNS fallback. The four-lookup cap that mirrors libuv's thread pool stays. Removing the feature drops hickory-resolver/-proto/-net, resolv-conf and moka from the dependency graph, which also retires the two hickory-proto advisory ignores in deny.toml. Closes pnpm/pnpm#14469. |
||
|
Zoltan Kochan
|
a83487c2f8 |
perf: reuse store content when restoring a remote build (#14189)
Reuse store content when restoring a remote build. A remote side-effects restore downloaded every file of the artifact, then handed the bytes to the store, which addresses content by the very digest the artifact manifest already carries. A built package's files are mostly its own, and artifacts share files with one another, so most of what was transferred was content the store already had. Ask the store first. `locateFileInStore` answers with the path it holds for a digest and mode, or nothing, and the restore only fetches what is missing. The store keeps executable and non-executable content apart, so the mode is part of the question. Nothing is trusted that was not trusted before: the store addresses content by its hash, so a file it holds under an artifact's digest is that artifact's bytes. The pacquet side already had the store helper this needs; its test pins the part that could silently drift — a digest or executable-bit rule diverging between the write side and the artifact side would not fail anything, it would just make the lookup always miss. Follow-up to pnpm/pnpm#14171. |
||
|
Zoltan Kochan
|
e7f3bccfff |
feat: workspace task orchestration (#14209)
Replace the chunked topological scheduler of recursive run/exec with
per-task scheduling in both stacks: a task — a (project, script) pair —
becomes runnable when every task it depends on has completed
successfully, and runnable tasks are dispatched under the
workspace-concurrency limit with no barrier between
dependency-independent tasks.
A new "tasks" section in pnpm-workspace.yaml declares task dependencies
with the caret convention ("^build" = the task in each workspace
dependency, "build" = the task in the same project). A task with no
entry behaves as depending on its own name in the workspace
dependencies, which is exactly what chunking implied. A project without
the script becomes a pass-through node that is reported skipped and
keeps the chain intact.
Also per the RFC: task-graph cycles are ERR_PNPM_TASK_CYCLE naming the
participating tasks, scoped to the invocation's selected graph, with
ignoreWorkspaceCycles: true downgrading the error to a warning;
--resume-from excludes exactly the anchor's transitive dependencies;
--reverse runs the reverse graph; under --no-bail dependents of a
failed task are reported skipped and do not add to the exit code; with
--bail, the first failure ends the run at once and nothing new is
dispatched; output is inherited only when at most one script can ever
be in flight; and pnpm -r run --dry-run [--json] prints the resolved
task graph without running anything (the verify-deps check included).
Implementation: the projects sorter exposes the tunneled dependency-edge
map (filteredProjectsDependencies / filtered_projects_dependencies)
instead of only its flattened chunks; the recursive summary is
task-keyed (dependsOn-pulled tasks get "<dir>#<task>" keys); pacquet's
inert --sequential now means concurrency 1 and its --no-sort
resume/reverse handling is aligned with the TypeScript CLI; the dead
chunk helpers are removed.
Related to https://github.com/pnpm/rfcs/pull/23
|
||
|
Zoltan Kochan
|
83350dea56 |
feat: integrate the remote side-effects cache with install (#14171)
Integrate the remote side-effects cache proof of concept with dependency installation in both TypeScript pnpm and pacquet. Plan eligible build candidates before contacting pnpr, batch lookup requests, verify the configured P-256 trust keys and artifact integrity, hydrate verified files into CAFS, and select the restored side-effects map before lifecycle scripts run. Fall back to the ordinary local build on all cache failures. Configure the feature through the remoteSideEffectsCache setting, assembled by the config reader from the workspace file, the global config file and the environment. A repository declares which organization and packages are eligible and nothing else: everything describing the act of signing stays with the machine holding the key, so a cloned repository cannot turn that key into a signing oracle. Capture and publish the actual post-build diff from trusted builders. Bound artifact hydration and blob downloads, retain CAFS paths instead of downloaded buffers, and withhold direct store writes from a store opened read-only. Add transactional owner and global storage accounting to pnpr alongside request, response, timeout, streaming-memory, manifest, variant, descriptor, lock-namespace, concurrency, and crash-residue bounds. Keep the proof of concept organization-scoped and Linux glibc-only. Defer lockfile pinning, persistent quarantine, publisher ownership, and key lifecycle policy to the RFC. Related to pnpm/rfcs#20. |
||
|
Zoltan Kochan
|
66398009c7 |
feat(config): honor the settings the Rust CLI only recognized (#14109)
Section 4 of the parity sweep in pnpm/pnpm#14101: settings present in pacquet's `types` table — so `pnpm config` handled them — that no code read. - `updateNotifier`: pacquet had no update notifier at all. `install` and `add` now ask the project's registry for pnpm's `latest` tag once a day, emit `pnpm:update-check`, and the default reporter prints the notice when that version is newer. The cadence lives in `<stateDir>/pnpm-state.json` under `lastUpdateCheck` — the same file, key, and `toUTCString` format pnpm writes, so the two CLIs share one throttle. The check is best-effort: an unreachable registry or an unwritable state dir cannot fail the install. Two deliberate gaps: an `install` finishing through the repeat-install fast path returns before the async runtime exists and is not covered, and the "to update, run" line never names `@pnpm/exe` — the native binary is published under both names with no marker telling them apart, and a global install of either replaces the other. - `legacyDirFiltering`: `use_glob_dir_filtering` was hardcoded on. - `initAuthorName` / `initAuthorEmail` / `initAuthorUrl` / `initLicense` / `initVersion`: `pnpm init` now writes them over the scaffold's placeholders, in place, so the key order is unchanged. The TypeScript reader dropped `init-version` from the `PNPM_CONFIG_*` env schema to keep the two `init` implementations aligned; that exclusion goes away with this commit. - `maxsockets` was inert in *both* stacks, not just in pacquet: the reader folded npm's lowercase spelling into `maxSockets` before any config file or env var had been applied, and `.npmrc` stopped carrying the key when the non-auth settings moved to yaml. The fold moves after the env loop, and pacquet gains the alias in both the settings file and the environment. The default stays `None` in pacquet (uncapped per origin, bounded by `networkConcurrency`) rather than npm's 50, which would cap it below the network concurrency it already resolves. `npmPath` is the fifth item on that list and needs no port: nothing reads it in the TypeScript CLI either, since publishing stopped shelling out to the npm CLI. Its one remaining mention — a `Pick` in `recursivePublish` that no code consumes — is dropped, so the deadness is visible. The pacquet command harness turns the notifier off for every suite: no test may reach the registry for pnpm's own `latest` tag or record the check in the developer's state directory, which the suites leave un-isolated. Related to pnpm/pnpm#14101. |
||
|
Zoltan Kochan
|
bf46078c19 |
fix(deploy): drop excluded dependency groups from the deployed project (#14069)
A shared-lockfile deploy prunes the package graph to the included dependency groups but kept every direct dependency in the deployed package.json and in the deployed lockfile's importer. The deployed lockfile therefore referenced packages it did not define, and the next install in the deploy directory tripped over that: pacquet linked the missing packages and left dangling symlinks, while the TypeScript CLI refused the install with ERR_PNPM_LOCKFILE_MISSING_DEPENDENCY. A --prod deploy of a project whose dev workspace dependency no longer exists pointed the importer at a missing directory the same way. Fill only the included groups into the deploy importer, in pacquet's create_deploy_files and in the TypeScript CLI's createDeployFiles, so the manifest, the importer, and the package graph agree. The graph walk no longer needs its own include gating for the importer maps; the snapshot-level optional handling from pnpm/pnpm#13641 stays as it is. Also drop an emptied group from pacquet's deploy importer instead of writing it as `{}` — the TypeScript lockfile writer omits an empty dependency map, so the two stacks were producing different deploy lockfiles for the same input. Resolve pacquet's --prod/--dev/--no-optional flags the way the TypeScript CLI does: --prod wins over --dev, and a dev-only install drops optional dependencies along with the production ones. Without it `pnpm install --dev` installed an optional dependency pnpm skips, and the deploy change above would have carried that divergence into `pnpm deploy --dev`'s manifest and lockfile. Closes pnpm/pnpm#13623 |
||
|
Pratik DulalandZoltan Kochan
|
03fd4caa00 |
fix(pack): write POSIX ustar headers for packed tarballs (#13925)
Fixes https://github.com/pnpm/pnpm/issues/13924 ## Summary - Write tar entries in the full POSIX ustar header form npm and pnpm 11 emit: `Header::new_ustar` for the `ustar\0` magic and `00` version, plus an explicit `0` regular-file typeflag (the typeflag byte stays NUL otherwise). - Add a regression test that pins the raw typeflag, magic, and version header bytes, since tar readers accept every header form and decoded values would not catch a format regression. - Add the required pacquet patch changeset, and `typeflag` to the cspell dictionary. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
daecf8765f |
perf(link): prefer hardlink over clone in Auto on Linux (#14012)
Bun materializes the same warm-store `node_modules` in roughly a third of pacquet's wall time on btrfs. Profiling the gap showed it was never syscall mechanics — it was the tier `Auto` picks: pacquet reflinks there, and a reflink is a new inode plus extent bookkeeping inside the filesystem's metadata trees, where a hardlink is one directory entry and an nlink bump. ## Measurements alotta-files fixture (39k files), warm store + lockfile, cold `node_modules`, btrfs, interleaved A/B: | | clone-first (before) | hardlink-first (after) | |---|---|---| | 32 threads | 0.91s wall / 5.4s sys | **0.53s / 2.9s** | | 4 threads | 1.14s / 1.5s | **0.66s / 0.9s** | Hardlink-first is also the default Bun ships (`--backend=hardlink`). ## Scope: pnpm 12 only This changes what the default materializes on disk, so it ships behind the v12 major. **pnpm 11's TypeScript importer deliberately keeps clone-first** — the two `Auto` implementations intentionally diverge on this until pnpm 11 is retired. The ladder's rustdoc and the changeset record the decision, and the changeset bumps only `pacquet`. ## What doesn't change - **pnpm 11**, entirely. - **ext4** (GitHub CI, the published benchmark): `FICLONE` is unsupported there, so `Auto` always ended up hardlinking after one failed reflink. Nothing moves. - **macOS** keeps clone-first — APFS `clonefile` is the platform's cheap primitive. - Explicit `packageImportMethod: clone` / `clone-or-copy` / `hardlink` / `copy` are untouched. ## The trade Clone-first bought store isolation on Linux CoW filesystems: a clone can't be corrupted by a package that mutates its own files at runtime. But every ext4 and Windows install already runs without that isolation, and the store's real guard is `verify-store-integrity`. v12 makes Linux stop paying extra for a protection the other platforms never had; users who want the isolation keep it with `packageImportMethod: clone`. ## What was tried and rejected Bun's other structural difference — `linkat` from open directory fds (a 256-entry store-prefix fd table plus a per-package dirfd) instead of absolute-path resolution — was implemented and benchmarked too. Every link was confirmed on the fd path (counted: 35k+ hits, 0 fallbacks), kernel time fell ~15%, and wall **regressed** (link phase 335ms → 531ms at 4 cores). Linux resolves hot cached paths through the lock-free RCU dcache walk; the fd anchoring saves nothing that was expensive. Making the per-package file loop sequential also regressed (straggler tail on thousand-file packages). Both reverted; noting it here so nobody re-walks that path without new evidence. ## Implementation The downgrade cache moves from `fetch_max` (which encoded the ladder in the constants' numeric order) to a compare-exchange step along a per-platform ladder (`next_auto_tier`), so racing rayon workers still converge without a lock. `pnpm:progress imported` telemetry reports the platform's ladder head instead of unconditionally claiming clone (a review catch). Tests pin the ladder order, the fresh-state hardlink, the telemetry mapping, and that a stale compare-exchange can neither skip nor regress a tier. |
||
|
cakeniandZoltan Kochan
|
e5007150fa |
fix: refetch metadata when publish times are incomplete (#13750)
Detect partial per-version publish-time maps before applying minimumReleaseAge. Some registries provide timestamps for only a subset of releases in abbreviated packuments. Re-fetch full metadata unless every version has a publish timestamp, so mature versions remain resolution candidates. Keep the TypeScript and pacquet implementations and regression coverage in sync. Closes pnpm/pnpm#13741. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
fe8ec8fbec |
refactor(pacquet): rename the Rust crates from pacquet- to pnpm- (#13948)
The Rust CLI ships as `pnpm`; `pacquet` survives only as the in-repo package identifier that keeps the npm wrapper from colliding with the TypeScript CLI's name. Carrying that identifier into all 75 Cargo crate names made every `use` statement and every `cargo nextest run -p ...` spell a name users never see. This renames the crates to the product they belong to. Mechanical, name-only: package/lib/bin names, the root `[workspace.dependencies]` keys, every dependency entry, and every Rust path reference. All the crates are `publish = false`, so nothing about the released artifacts changes and no changeset is needed. Also updated the tooling and docs that spell crate names: the `codecov` cargo alias, the `registry-mock` just recipe, the release workflow's `-p` targets and `libpnpm_napi.*` artifact paths, the `libpacquet` -> `libpnpm` cspell dictionary entry, and the crate names quoted across `AGENTS.md`, `CONTRIBUTING.md`, `CODE_STYLE_GUIDE.md`, and the plans. Two spots needed more than a token swap. The five insta snapshot files carry the crate name in their filename, since insta derives it from the module path, so they are renamed too — otherwise the tests fail with "snapshot not found" rather than a diff. And the private module in `pnpm-lockfile-verification` that mirrors the npm verifier's violation codes is named after the crate it shadows, so it moved with it. Two knock-on fixes: the shorter names let rustfmt fit three macro invocations onto one line, where `perfectionist::macro_trailing_comma` rejects the trailing comma that was legal while they were wrapped; and the `libpacquet` cspell entry became `libpnpm` for the NAPI addon's shared library. Left alone deliberately, none of which are crate names: the product name in prose and in the changeset package identifier, the `--pacquet` benchmark flag and testbed IDs, test helpers and temp-dir prefixes (`pacquet-test-*`, `../pacquet-store`), the released changesets that quote error codes shipped versions actually emitted, the `.github/workflows/pacquet-*.yml` filenames, and the `@pnpm-private/pacquet-registry-mock-launcher` npm package. |
||
|
Zoltan Kochan
|
7bee7dace4 |
fix(injected-deps-syncer): remove what the source dropped before replacing its parent (#13840)
When a workspace package replaced a directory with a file of the same name, that path landed in `modified` while the directory's contents landed in `removed`. The two ran concurrently, so a removal could be issued against a path whose parent had already been relinked as a file and come back ENOTDIR, which removeRecursive rethrows — aborting syncInjectedDepsAfterScripts partway through. Await the removals before applying any change. Nothing a change needs can be removed: a path absent from the source has all its children absent too, so no removal is ever an ancestor of an added or modified path. The Rust CLI reached the same ordering in pnpm/pnpm#13834, where a sequential patcher made this a deterministic failure rather than a race. |
||
|
Zoltan Kochan
|
4a5a10d2ca |
fix(setup): declare the module type in the manifest written for a standalone executable (#13732)
`pnpm setup` writes a package.json next to a standalone executable whose directory has none — the tarball that https://get.pnpm.io/install.sh downloads — so the global install has a package to install. That manifest omitted `"type": "module"`, so Node.js reparsed the ESM files shipped beside the executable (`dist/worker.js`) as CommonJS first and printed a MODULE_TYPELESS_PACKAGE_JSON warning on every spawn. The published `@pnpm/exe` manifest already declares the module type, so this only affected installs made through the standalone script. Fixed in both stacks. The manifest construction moves into `standaloneManifest` / `standalone_manifest`, so it can be asserted without running the global install the surrounding function performs. |
||
|
Zoltan Kochan
|
dbe2dad610 |
feat(pacquet): ship node-gyp beside the native binary (#13586)
pnpm v12 shipped no node-gyp, so any package that shells out to it during an install script died with `spawn node-gyp ENOENT`: `node-gyp-build` with no matching prebuild, `node-pre-gyp`, a plain `"install": "node-gyp rebuild"`, and the `node-gyp rebuild` script pnpm synthesizes for a package that ships a `binding.gyp` without one. Embedders worked around it by putting their own node-gyp on the lifecycle PATH. pnpm 11 solves this without a JS bundle: node-gyp is declared `external`, and `pnpm deploy` resolves its whole tree from this repo's lockfile at release time into a `dist/node_modules/node-gyp` directory that sits beside the entry point, with `dist/node-gyp-bin` holding the wrappers that go on a lifecycle script's PATH. Those are ordinary files next to the executable, so the native binary can ship them the same way — nothing about the mechanism depended on there being a bundle to hide node-gyp inside. Resolving the tree at release time rather than at install time is the point, not an implementation detail: the dependency tree is frozen and reviewed per pnpm release instead of being fetched from the registry at the moment an install script is about to execute arbitrary code. scripts/bundle-node-gyp.mjs builds the payload, reusing the TypeScript CLI's wrapper scripts so the two stacks cannot drift on how a script's node-gyp is dispatched, and refusing to produce a partial dist/ — a PATH entry that resolves nothing is worse than shipping none at all. Both wrappers ship it, since each places the binary next to itself. node-gyp moves to a catalog entry so both stacks pin one version; the Node.js-floor hold-back note moves with it. At runtime bundled_node_gyp_bin finds the wrapper dir relative to the executable and returns None when nothing was shipped, which is what a checkout build sees — lifecycle scripts then fall back to whatever node-gyp the environment provides, exactly as before. The wrapper itself honors npm_config_node_gyp and only falls back to the shipped copy when it is unset, so a user-supplied node-gyp, a workspace one, and a package's own dependency all keep winning. Refs https://github.com/pnpm/pnpm/issues/13580 |
||
|
Zoltan Kochan
|
c2b7cfb930 |
feat(napi): add returnListOfDepsRequiringBuild install option (#13532)
The napi install result populated depsRequiringBuild from the ignored- builds accumulation only, so an embedder that allows dependency builds to run (Bit) always received an empty list and wiped its recorded bit.depsRequiringBuild lockfile block on every install. Mirror the TypeScript engine's returnListOfDepsRequiringBuild option: when set, the fresh-resolve path reports every non-skipped package whose files carry install scripts, regardless of the allow-build policy, through a sink threaded into the install pipeline. Installs served from the frozen-lockfile path leave the field undefined so the embedder keeps its previously recorded list, matching the TS engine, which only computes the list from a fresh resolve's fetch results. |
||
|
Neil de CarteretandZoltan Kochan
|
3729d83d4c |
fix: approve git-hosted tarball builds by repository url (#12985)
Normalizes git-hosted tarball dep paths back to the canonical `name@git+https://host/org/repo.git` key that a clone of the same repository produces, so one hashless entry approves the package whether pnpm clones it or downloads a tarball. GitHub (codeload), GitLab archive, and Bitbucket download URLs are covered. The host is part of each derived key, and the GitHub and Bitbucket download hosts are matched against a literal value (GitLab's is captured generically to allow self-hosted instances), so a look-alike host cannot be rewritten into an unrelated repository key. Approving or denying a specific resolved commit by its full tarball dep path continues to work. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Abdullah Alaqeel
|
d47ab916ad |
fix(self-update): stop the project config from steering the pnpm download (#12813)
Fixes pnpm/pnpm#12803. `pnpm self-update` resolved the pnpm download through the project's own registry/auth config, loaded the repo's default `.pnpmfile.(c|m)js`, and took its `minimumReleaseAge` policy from the active workspace — so the outcome of a global tooling operation depended on the directory it ran in, and on config a checkout controls. Route the fetch through the trusted package-manager bootstrap config — the channel `switchCliVersion` already uses, which excludes the project `.npmrc` and workspace manifest — and stop auto-loading the repo pnpmfile, whose `updateConfig` hook and custom resolvers/fetchers reach the same requests. `getPackageManagerBootstrapConfig` moves into `@pnpm/config.reader` so the command package can reuse it. Stop reading the project's `minimumReleaseAge` and `trustPolicy` settings, and its `ci` flag, for self-update. Each is dangerous in both directions for a command that replaces the global binary: a cooldown lowered waives the protection the user configured, raised it pins the machine to the installed pnpm, including past a release that fixes a vulnerability in it; a trust policy turned off accepts a release whose evidence the user meant to reject, turned on blocks the update the same way; and `ci` decides whether an immature pick may be confirmed at the keyboard at all. Unlike a blocked dependency upgrade, those decisions follow the user into every other project. The policies come from the built-in defaults, the global config, the environment, and CLI flags instead. Other commands keep reading them from `pnpm-workspace.yaml`, and there is no new default. When an immature version is refused, an interactive run offers to update anyway, matching how a strict install prompts; non-interactive runs still fail closed. Mirrored in pacquet: `Config::current_for_self_update` skips the same settings, and `self-update` gained the matching prompt and error. |
||
|
Zoltan Kochan
|
29efb4ae49 |
fix: honor same-second edits on whole-second-mtime filesystems (#12975)
The optimistic repeat-install fast path and verify-deps-before-run decide whether a package.json, .pnpmfile.cjs, or patch file changed since the last install by comparing its mtime against the last-validated timestamp with a strict `>`. On a filesystem that records mtimes at whole-second resolution (ext4 with 128-byte inodes, HFS+, some CI runner disks) a file edited in the same second as the install gets an mtime that rounds down below the timestamp, so the edit looks unchanged, the fast path short-circuits, and re-resolution is skipped — the second install is a no-op and keeps the stale result. Detect a whole-second mtime (no sub-second component) and treat its whole second as possibly-after the reference, falling through to the authoritative content check instead of trusting the rounded-down mtime. Erring toward "modified" only ever runs the content check; it never skips a needed install. On sub-second filesystems the mtime carries a fractional part, so the comparison stays exact and the common case is completely unaffected — the existing deps-status unit tests pass unchanged. Landed in both stacks: pacquet's optimistic_repeat_install and pnpm's checkDepsStatus, with a regression test on each side that forces a whole-second mtime so the coarse-filesystem path is exercised on any filesystem. |
||
|
Zoltan Kochan
|
0dfce95b76 |
feat(release): compose changelogs at publish time (registry storage) (#12971)
Implement RFC 0006 `changelog.storage: registry` and make it the default: no CHANGELOG.md is committed. A release's section is parked under .changeset/changelogs/ at `pnpm version -r` time and composed into the published tarball at publish, on top of the previous published version's changelog (highest published version semver-lower than the one being published — never a dist-tag lookup, so concurrent lines chain correctly). Consumed intents are garbage-collected by a later `pnpm version -r` only once the registry confirms the ledgered version is published and its tarball's CHANGELOG.md carries the composed section; a foreign-published version keeps the intent instead of losing it. The pure `versioning` crate/package stays network-free: the publish check is injected (a callback in TS, a confirmed-key set in Rust) from the command layer, which already has registry access. The parked-section store captures dependency-update lines and exact formatting that can't be recomposed from intents at publish time. `versioning.changelog.storage: repository` keeps the previous committed CHANGELOG.md behavior. Landed in both stacks. |
||
|
armanandZoltan Kochan
|
b46f96165f |
fix: kill spawned process trees with taskkill on Windows error exits (#12925)
Main thread panicked: begin > end (105 > 28) when slicing `@pnpm/npm-lifecycle@1100.0.0(patch_hash=e3541c…)(supports-color@10.2.2)` at pacquet/crates/resolving-deps-resolver/src/resolve_peers.rs:2490:51 --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
826366b7bc |
feat(releasing): native workspace release management in both stacks — change intents, release plans, per-package release lanes (#12953)
Implements phases 1-2 of the native monorepo versioning plan (https://github.com/pnpm/pnpm/issues/12952, RFC https://github.com/pnpm/rfcs/pull/18) in both stacks. TypeScript: new @pnpm/releasing.versioning package (intent reader/writer for changesets-compatible .changeset/*.md files with the additive 'none' decline; the committed per-package consumed-intents ledger .changeset/ledger.yaml; the release-plan assembler with direct bumps, dependent propagation via materialized workspace: ranges under real semver semantics, fixed groups, ignore, maxBump enforced against the real version distance, --filter narrowing, per-package release lanes with cumulative stable-target escalation and graduation; the applier with format-preserving version-only manifest updates, changelog composition in repository storage mode, ledger append, and intent GC with the prerelease retention exemption). CLI: pnpm change / change status; bare pnpm version -r (--dry-run, the new root pnpm lane command manages versioning.lanes through the format-preserving workspace manifest writer. The versioning key is typed on PnpmSettings/Config and validated by the workspace manifest reader. Rust: new pacquet-versioning crate mirroring the engine (same file formats, error codes, messages, and output), versioning settings on WorkspaceSettings/Config, and new change/version/lane commands in pacquet-cli. The npm-style pnpm version <bump> forms remain unported in pacquet (they did not exist there before) and error with a clear message. The two deploy install futures are boxed because Config grew past clippy's large-future threshold. Both stacks' engine test suites mirror each other one for one; integration tests drive the real binaries through record -> plan -> release -> lane -> graduation. |
||
|
Alessio AttilioandZoltan Kochan
|
411bbe89ff |
feat(registry-access): implement team command in both TypeScript and Rust stacks (#12789)
The team command communicates with the registry through the standard npm team API endpoints. The scope:team format is parsed to separate the organization scope from the team name. For mutation subcommands (create, destroy, add, rm) the registry URL is resolved per scope from the registries map with an optional --registry override, the auth header is resolved from the configured credentials honoring scoped credentials, and the request is sent with retry support and bounded response reads. When an OTP is in play, the Rust side restricts redirects to the configured registry origins so the npm-otp header cannot leak to another host; the TypeScript fetch layer already strips it on cross-host redirects. The ls subcommand dispatches to listing teams within an org when given @scope and to listing members of a specific team when given @scope:team. Output supports three modes, the default human-readable listing, --parseable which emits newline-delimited names, and --json which emits structured arrays.
On the TypeScript side the command is registered in pnpm/src/cmd/index.ts and removed from the notImplemented list. On the Rust side it is added as a CliCommand variant, routed in dispatch, and dispatched in dispatch_query. Both sides include comprehensive tests covering all subcommands, error paths for 401 403 404 and 409 responses, empty results, and the three output formats.
pnpr serves the npm team API from each hosted registry's config-declared teams: GET /-/org/{scope}/team and GET /-/team/{scope}/{team}/user list teams and members, gated by the registry-level access with denials masked as not-found, while team mutations answer an explicit 403 since pnpr teams are config-managed.
@pnpm/cli.parse-cli-args no longer stops option parsing at an escape word (create, exec, test) that appears as another command's parameter, which previously made pnpm team create drop a trailing --registry option.
---------
Co-authored-by: Zoltan Kochan <z@kochan.io>
|
||
|
|
a8ad82d4dd |
fix: register pn alias in generated completions (#12861)
Register the pn short alias in generated shell completion scripts. pnpm exposes pn as a binary alias, but completion generation only registered pnpm with the supported shells. This meant zsh completions generated by pnpm completion zsh registered pnpm only, so pn did not receive the same completion function. Post-process the tabtab-generated scripts to register pn alongside pnpm for bash, fish, pwsh, and zsh, and cover each shell in the completion generator tests. Fixes pnpm/pnpm#11955. --------- Co-authored-by: ychampion <ychampion@users.noreply.github.com> Co-authored-by: Zoltan Kochan <zoltankochan@gmail.com> |
||
|
Zoltan Kochan
|
de1371daa9 |
feat(pacquet): add Node API bindings for the Rust engine (#12822)
Add `pacquet-napi`, a napi-rs cdylib crate, plus its `@pnpm/napi` npm wrapper, exposing pacquet's programmatic engine surface to Node.js so programmatic pnpm consumers can drive the Rust engine instead of the TypeScript pnpm packages. Bit is the reference consumer. Exports: install (in-memory importers, single and multi-importer workspaces, a synchronous readPackage hook per resolved dependency manifest, build-script approval, and depsRequiringBuild), rebuild, resolveDependency, pack, parseBareSpecifier, engineVersion, and auth via authHeaderByUri. The install runs on a dedicated 32 MiB-stack worker thread with its own tokio runtime; the napi async fn awaits the result over a oneshot channel, so pacquet's borrowed State never crosses the FFI boundary. A reporter bridge forwards the engine's wire-compatible log events to a JS callback, and errors carry pnpm's ERR_PNPM_* code and hint through a structured envelope. Engine-core changes are minimal and inert for existing callers: PackageManifest::from_value, and two Option fields on Install (pnpmfile_hook_override and workspace_projects_override) defaulted to None everywhere. A new napi-release profile sets panic = "unwind" since the workspace release profile uses panic = "abort". getPeerDependencyIssues is stubbed pending pacquet's own peer-issue renderer. Distribution follows the @pnpm/exe.* model via scripts/generate-packages.mjs. |
||
|
|
322f88f4f1 |
fix: avoid mutating unrelated deps on failed optional updates (#11373)
updateProjectManifest previously reconstructed the pairing between each resolved direct dependency and the wanted dependency it came from — first by alias, then by specifier shape, then by array position. Every one of those is fragile: a failed optional dependency drops out of directDependencies and shifts a positional pairing (#11267), and an aliasless selector that resolves to an alias already in the manifest matches the stale aliased entry instead of the request. Carry the originating wanted dependency on each resolved direct dependency (PkgAddressOrLinkBase.wantedDependency -> ResolvedDirectDependency.wantedDependency), set where the resolver builds the result and already has it in scope, and have updateProjectManifest read rdd.wantedDependency directly. This removes all the alias/position/specifier heuristics (and the earlier normalizeGitHubBareSpecifier band-aid) and fixes both correctness gaps structurally. Adds unit tests for the failed-optional (#11267), alias-collision re-add, and aliasless-optional-failure cases. Closes #11267 --------- Co-authored-by: cyphercodes <cyphercodes@users.noreply.github.com> Co-authored-by: Hermes Agent <hermes@example.invalid> Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
|
d3f68e2aa4 |
fix(audit): compute reachable vulnerabilities with Tarjan SCC (#12467)
`pnpm audit` enumerates the install paths to every vulnerable package. The reachability-based pruning added in 11.5.1 (pnpm/pnpm#12087) lets the walker skip subtrees that reach no unsaturated finding by precomputing, per node, the set of vulnerabilities reachable from it. That getter only memoised acyclic subtrees: a node whose subtree contained a cycle was `complete === false`, and so was every ancestor up to the importer roots. None of them were cached, so their reachable set was recomputed on every query. Real dependency graphs commonly contain cycles, and a single cycle high in the graph makes a large fraction of nodes non-memoisable, yielding an O(N^2) walk. This matched the report in pnpm/pnpm#12212 exactly (CPU-bound, identical audit output across versions). Reachability is now computed with Tarjan's strongly-connected-components algorithm. Every node is scanned once; all members of an SCC reach the same set of vulnerabilities and share one set, finalised in reverse-topological order. Cyclic graphs are handled in O(N + E). The reachable set is used only to prune, so it must never under-approximate (that would hide a real finding). Tarjan yields the exact set for every node, so no finding can be dropped, and the path-recording logic is unchanged. The getter returns ReadonlySet<string> so the shared sets cannot be mutated by callers, and a missing memo entry (an impossible-by-construction state) throws rather than silently returning an empty set. A regression test asserts the read-count growth ratio between two cycle sizes (L=200 and L=400) is sub-quadratic: the fix scales ~2x (linear), the previous code ~4x (quadratic). Asserting the ratio cancels the per-node constant, so the test is not brittle to constant-factor changes. Closes pnpm/pnpm#12212. --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
c86559c9bc | docs: add new silver sponsor (#12473) | ||
|
Hiten ShahandZoltan Kochan
|
1b02b47669 |
fix: remove macOS Gatekeeper quarantine xattr from native binaries (#11095)
Fixes #11056 ## Problem On macOS, pnpm imports files from its content-addressable store into `node_modules` via copy, reflink/clone, or hardlink. All three preserve extended attributes, including `com.apple.quarantine`. If a store blob carries that xattr — e.g. it was first written under a Gatekeeper-enabled app such as a Git client (`LSFileQuarantineEnabled=YES`) — the quarantine propagates into `node_modules`. Gatekeeper then blocks ad-hoc-signed native binaries (`.node`, `.dylib`, `.so`) from loading, even though pnpm has already verified each file's integrity against `pnpm-lock.yaml`. ## Solution After importing a package from the store, strip `com.apple.quarantine` from its native binaries — mirroring Homebrew's behaviour of dropping quarantine from downloads after checksum verification. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Abdullah AlaqeelandZoltan Kochan
|
61969fbddf |
fix(deps-status): detect lockfile-only changes (#12106)
## Summary Fixes `pnpm install` with `optimisticRepeatInstall` incorrectly returning `Already up to date` when `pnpm-lock.yaml` changed but project manifests did not. Fixes #12100. ## Root Cause `checkDepsStatus` used modified manifest mtimes as the only signal for whether it needed to validate dependency status. If no manifest was newer than `workspaceState.lastValidatedTimestamp`, it returned `upToDate: true` before checking whether the wanted lockfile had changed. That skipped lockfile validation for workflows like: - `git checkout HEAD~1 -- pnpm-lock.yaml` - restoring only `pnpm-lock.yaml` from a stash - external tools rewriting the lockfile without touching manifests ## Changes - Check wanted lockfile mtimes before taking the optimistic fast path. - If any wanted lockfile is missing or newer than the workspace state timestamp, validate all projects instead of only modified manifests. - Add a regression test proving a lockfile-only change does not skip wanted-lockfile validation. - Add a patch changeset for `@pnpm/deps.status` and `pnpm`. --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
TruffleandZoltan Kochan
|
6d35338691 |
fix: detect changes inside file: dependencies on repeat install (pacquet + pnpm) (#12317)
## Summary - `pnpm install` reports "Already up to date" after edits inside a `file:` dependency's directory or after repacking a `file:` tarball. This is a v11 regression from the `optimisticRepeatInstall` default flip in pnpm/pnpm#11158. Fixes pnpm/pnpm#11795. - `checkDepsStatus` gains a `treatLocalFileDepsAsOutdated` option: when set, any project manifest declaring a local file dependency makes the check report not up to date. `installDeps` sets it on the optimistic fast path, so projects with local file dependencies always run a real install, which refetches those dependencies (the v10 behavior). - The predicate covers `file:` specs, path-prefixed specs (`./`, `../`, `~/`, absolute POSIX paths, and Windows drive paths including drive-relative ones like `C:dir`, matching the local resolver's `isFilespec`), and bare tarball file names (`vendor/pkg.tgz`). It is deliberately narrower than the local resolver's bare-path matching: a bare `user/repo` is statically indistinguishable from a git shorthand at this layer, and matching it would kill the fast path for every project with git dependencies, so protocol-carrying and URL specs stay on the fast path. - `pnpm.overrides` entries are scanned with the same predicate: an override mapping to a local file spec redirects every matching dependency in the graph to that directory, so it has the same blind spot as a direct local file dependency. Registry and `link:` overrides keep the fast path. - The option is caller-scoped on purpose. `verifyDepsBeforeRun` also consumes `checkDepsStatus`, and treating `file:` deps as always stale there would force a reinstall before every `pnpm run`. Its behavior is unchanged, and a regression test pins that. - pacquet port in the same commit: `check_optimistic_repeat_install` bails unconditionally on `file:` specifiers, because its only caller is the install command, the one consumer that sets the flag upstream. `link:` specifiers are excluded on both sides: they are symlinked, so changes inside them flow through without a reinstall. ## Why Both branches of `checkDepsStatus` are blind to content changes inside a `file:` dependency. The workspace branch exits early with `upToDate: true` when no project manifest's mtime moved, without ever reaching `linkedPackagesAreUpToDate`. The non-workspace branch exits at the manifest-vs-lockfile mtime gate the same way. Editing a source file inside a `file:` dependency bumps neither, so the fast path can never see it; the fix has to bail before those gates rather than refine them. This is the fix shape (a) I proposed in my diagnosis on the issue thread ([comment](https://github.com/pnpm/pnpm/issues/11795#issuecomment-4504177744)): the cost is a full resolution on repeat installs only for projects that declare `file:` dependencies, which is exactly what v10 did. The manifest-only comparison in `@pnpm/lockfile.verification` (`allProjectsAreUpToDate`) is intentional for the install-proper path and asserted by its tests, so this PR leaves it untouched. ## Checks - `pnpm --filter @pnpm/deps.status test test/checkDepsStatus.test.ts` (31 passed, 13 new) - `pnpm --filter @pnpm/deps.status run compile` and `pnpm --filter @pnpm/installing.commands run compile` (tsgo + eslint clean) - `cargo test -p pacquet-package-manager optimistic_repeat_install` (51 passed, 7 new; run in a rust:1.95.0 container) - `cargo fmt --check -p pacquet-package-manager` - `RUSTDOCFLAGS="-D warnings" cargo doc -p pacquet-package-manager --no-deps` --- Written by an agent (Claude Code, claude-fable-5). --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Victor Sumner
|
61810aa684 |
feat: add --frozen-store for installs against a read-only store (#12190)
## What
Adds an opt-in `frozenStore` / `--frozen-store` setting (default `false`) that lets `pnpm install --offline --frozen-lockfile` run against a package store that lives on a **read-only filesystem** — a Nix store, a read-only bind mount, an OCI layer.
## Why
A normal install fails against such a store **not** because it writes package content, but because it unconditionally:
1. opens the SQLite `index.db` in WAL mode, which needs to create `-shm`/`-wal` sidecars in the store directory; and
2. writes a project-registry entry under the store.
Both fail with `attempt to write a readonly database` / `EROFS` on a read-only store directory, even when the store is complete and the lockfile is frozen. This blocks any deployment that wants an immutable, content-addressed store.
## How
When `frozenStore` is enabled, pnpm opens `index.db` through the SQLite **`immutable=1`** URI — which tells SQLite the file cannot change underneath it, so it bypasses the WAL/`-shm` sidecar machinery entirely and reads the raw file with zero sidecar creation — and suppresses every store-write path. Pair it with `--offline --frozen-lockfile` against a fully-populated store. It is incompatible with two settings that would write into the store, and each throws a clear config-conflict error before any network or store access: **`--force`** (which bypasses the no-write-on-hit skip → `ERR_PNPM_CONFIG_CONFLICT_FROZEN_STORE_WITH_FORCE`) and a configured **pnpr server** (which would fetch and write packages into the store → `ERR_PNPM_FROZEN_STORE_INCOMPATIBLE_WITH_PNPR`).
> Plain `SQLITE_OPEN_READ_ONLY` is **not** sufficient: opening a WAL-mode db read-only still tries to create the `-shm` sidecar, which fails on a read-only *directory*. `immutable=1` is the load-bearing piece.
> **Node.js requirement:** `node:sqlite` only passes `SQLITE_OPEN_URI` to SQLite (so the `immutable=1` query is honored rather than treated as part of a literal filename) starting in **v22.15.0** (22.x line), **v23.11.0**, and every **v24+**. pnpm's `engines` floor is `>=22.13`, so on a runtime older than that the frozen open is detected up front and fails with a clear `ERR_PNPM_FROZEN_STORE_UNSUPPORTED_NODE` instead of SQLite's cryptic "unable to open database file". (pacquet uses rusqlite with an explicit `SQLITE_OPEN_URI` flag, so it has no such floor.)
> The `immutable=1` URI path is also percent-encoded (`%`→`%25`, `?`→`%3f`, `#`→`%23`, in that order, leaving `/` literal) so a store path containing those characters doesn't truncate the path or inject a spurious query parameter — applied identically in both stacks.
### Build backstop under the global virtual store
Under the global virtual store (default), a package's directory lives **inside** the store (`{storeDir}/links/...`). Applying a patch or running an allowlisted lifecycle script writes into that directory — so on a frozen store it would crash mid-build with a raw `EROFS`. A fully-seeded store never reaches the build step (patched/built packages are imported from the side-effects cache and filtered out by the `isBuilt` gate), so any residual build candidate means the seed is **missing that package's build output**.
`buildModules` now refuses up front with `ERR_PNPM_FROZEN_STORE_NEEDS_BUILD` and actionable guidance ("rebuild the seed with their scripts enabled, or remove them from `onlyBuiltDependencies`") instead of failing cryptically once a script starts. The check is gated on the global virtual store — under the isolated linker, slot directories live in the writable project-local store, so builds there are fine. Non-allowlisted scripts never run, so they are not treated as a blocking write.
Bin-linking has its own read-only-store edge under the global virtual store. On a **warm** checkout (the project's `.bin/<name>` already points at the seed target) `linkBin` returns before touching the store, so it is write-free. But on a **fresh** checkout it (re)creates the bin and calls `fixBin`, whose `chmod` targets the bin's **source file inside the store** (`{storeDir}/links/...`) — which is refused with `EPERM`/`EACCES` on a read-only store, even though a complete seed already ships that bin executable (so the `chmod` is redundant). `@pnpm/bins.linker` now wraps that call in `ensureExecutable`: it swallows the refusal when the target is already executable and rethrows otherwise, so bin-linking is write-free against a frozen store on a cold checkout too, while a genuinely non-executable bin (a broken seed) still surfaces as an error.
The blocking predicate distinguishes the two write kinds: a **patch** is applied regardless of `ignoreScripts`, so a patched package is always blocked; a **lifecycle script** is suppressed under `ignoreScripts`, so an allowlisted build-requiring package is *not* blocked when scripts are off (it would write nothing). This avoids falsely rejecting a valid `--ignore-scripts` frozen install. **Optional dependencies are exempt**: a build or patch failure on an optional dependency is non-fatal at runtime, so a seed missing an optional package's build output skips that build (emitting the `skipped-optional-dependency` log) instead of blocking the install — in both stacks.
### Both stacks (parity rule)
**TypeScript pnpm CLI**
- Config plumbing (`@pnpm/config.reader`): `frozen-store` type, config-file key, default, `Config.frozenStore`.
- The read-only open branch (`@pnpm/store.index`): `immutable=1`, read-only statements, throwing mutators.
- Wiring through `@pnpm/store.controller` and `@pnpm/store.connection-manager` to the sole `StoreIndex` construction site.
- Gating the project-registry write (`@pnpm/installing.context`).
- The `--force` / pnpr-server conflict guards (`@pnpm/installing.deps-installer`) and CLI surface (`@pnpm/installing.commands`).
- **After-install rebuild** (`@pnpm/building.after-install`): the post-install rebuild opens its `StoreIndex` immutably under the flag, so re-reading the store for a rebuild never attempts a writable open against the frozen store.
- **Worker fix:** `@pnpm/worker` opens its *own* writable `StoreIndex` on every `readPkgFromCafs` cache hit, so a pure read crashed on a frozen store. `frozenStore` is threaded through to `getStoreIndex` and keyed into its connection cache.
- **Build backstop** (`@pnpm/building.during-install`): `buildModules` throws `ERR_PNPM_FROZEN_STORE_NEEDS_BUILD` for a GVS slot that would build/patch on a frozen store, honoring `ignoreScripts`; threaded from `@pnpm/installing.deps-installer`.
- **Bin-linking on a read-only store** (`@pnpm/bins.linker`): `linkBin` wraps the `fixBin` chmod in `ensureExecutable`, which tolerates `EPERM`/`EACCES` when the bin's store-resident source is already executable (a complete seed) and rethrows otherwise — so a fresh checkout against a frozen store links bins without crashing on the redundant chmod. Catch-on-failure keeps the writable hot path at zero added syscalls.
**Rust pacquet**
- A dedicated `open_immutable` / `shared_immutable_in` opens via `immutable=1`, selected only under the flag. Plain `open_readonly` keeps the ordinary `SQLITE_OPEN_READ_ONLY` open (WAL locking intact) because normal installs read the index while the same process's `StoreIndexWriter` writes it concurrently — an immutable connection skips all locking and change detection, so a concurrent writer would make those reads undefined.
- `--frozen-store` CLI flag + `frozenStore` workspace-yaml setting.
- The store-index writer is replaced with a drain-and-drop stub (`spawn_disabled`) and `init_store_dir_best_effort` is skipped under the flag.
- **Build backstop:** `build_modules` returns `BuildModulesError::FrozenStoreNeedsBuild` (`ERR_PNPM_FROZEN_STORE_NEEDS_BUILD`) under the same GVS + frozen-store condition, threaded from `config.frozen_store`. The gate keys off `should_run_scripts` (which already folds the allow-build policy), so it is correct without an explicit ignore-scripts branch — pacquet has no configurable ignore-scripts mode yet.
pacquet already separated read-only index access (`shared_readonly_in`, or `shared_immutable_in` under the flag) from writes, so it never had the worker-conflation bug; the flag makes the "no writes attempted" contract explicit and gates the remaining best-effort write attempts.
## Testing
- **TS:** `store.index` frozen-mode-on-`0555`-directory test (reads work, writes throw `ERR_PNPM_FROZEN_STORE_WRITE`) plus a path-with-`?` open test — both gated on the runtime's immutable-URI support, with a complementary test asserting `ERR_PNPM_FROZEN_STORE_UNSUPPORTED_NODE` fires where that support is absent (the CI Node 22.13.0 path); `config.reader` round-trip; `deps-installer` `--force` and pnpr-server conflict guards; `worker`/`package-requester` unit tests. End-to-end on a `chmod -R 0555` store: install succeeds, `node_modules` materializes, no `-shm`/`-wal`/`-journal` sidecars; negative control without the flag fails as expected; incomplete store → clean offline error.
- **pacquet:** `open_immutable_reads_wal_db_on_readonly_directory` unit test plus the `immutable_sqlite_uri` encoding test and a path-with-`?` open test; yaml + CLI fold tests; **integration test** `frozen_store_installs_against_a_read_only_store` — primes a store, `chmod 0555` the tree, runs `install --frozen-lockfile --frozen-store --offline`, asserts success + materialized `node_modules` + zero sidecars. Confirmed load-bearing by reverting the `immutable=1` fix (test then fails).
- **Build backstop (both stacks):** `building/during-install` unit tests — approved-build-not-cached and patched-not-cached refuse; cached, non-allowlisted, and non-GVS cases pass through; **approved-build-under-`ignoreScripts` passes through while patched-under-`ignoreScripts` still refuses** — and the matching pacquet `build_modules` tests (`frozen_store_gvs_patch_not_seeded_refuses` + GVS-off / frozen-off controls). Each confirmed load-bearing by disabling the relevant guard and watching the corresponding test fail.
- **Bin-linking (`bins/linker`):** `ensureExecutable` tests with `fixBin` mocked to reject with `EPERM` — an already-executable bin source resolves (and `fixBin` is asserted called, so it isn't the warm skip-guard passing), a non-executable one rethrows `EPERM`. Confirmed end-to-end by running the built `linkBins` against a real `chflags uchg`-immutable store: the executable-seed case resolves and links the bin, the non-executable-seed control throws `EPERM`.
A changeset is included with `"pnpm": minor` and `"@pnpm/bins.linker": patch` (the read-only-store bin-linking fix).
|
||
|
Zoltan Kochan
|
52be454d57 |
fix: infer missing platform fields of optional dependencies from the package name (#12312)
* fix: infer missing platform fields of optional deps from the package name Some registries strip the os/cpu/libc fields (or just libc) from the version objects of the packuments they serve. Resolution then saw every platform-specific optional dependency as platform-unrestricted, so pnpm downloaded and installed the binaries of every platform regardless of supportedArchitectures, and wrote lockfile entries without the platform fields, which broke installs from that lockfile on every machine. Platform-specific binary packages encode their platform in the package name (e.g. @nx/nx-win32-arm64-msvc), so packageIsInstallable now fills the missing platform fields of an optional dependency from the name's tokens. Since every install path decides installability through that check before fetching, foreign-platform binaries are skipped without even downloading them, in fresh resolution and in headless installs with both node linkers alike. A package that declares no platform fields at all is treated as platform-specific only when an operating system is recognized in its name, so a generic name segment (such as 'arm' on its own) never gets a package skipped. Fixes https://github.com/pnpm/pnpm/issues/11702 Fixes https://github.com/pnpm/pnpm/issues/9940 * chore: add platform name tokens to the cspell dictionary * fix(package-is-installable): infer missing platform fields of optional deps from the package name Port of pnpm commit https://github.com/pnpm/pnpm/commit/34875b2d7c (PR https://github.com/pnpm/pnpm/pull/12312). Some registries strip the os/cpu/libc fields (or just libc) from the version objects of the packuments they serve, and lockfile entries written from such metadata lack the fields too, so every platform's binaries were installed regardless of supportedArchitectures. Platform-specific binary packages encode their platform in the package name (e.g. @nx/nx-win32-arm64-msvc), so the installability check now fills the missing platform fields of an optional dependency from the name's tokens: infer_platform_from_package_name + inferred_platform in pacquet-package-is-installable, applied inside package_is_installable (hoisted linker) and in compute_skipped_snapshots (isolated linker, with the check cache keyed by the snapshot's optional flag since the verdicts can differ). The any_installability_constraint fast path now also considers optional snapshots whose names infer a platform their metadata row does not declare, so the inference is reachable on lockfiles without any declared constraint. Same guard rails as upstream: declared fields always win (each field is filled only when missing — a missing libc alone is inferred, disambiguating -gnu vs -musl), and a package declaring no platform fields at all engages the inference only when an operating-system token is recognized in its name, so a generic name segment such as 'arm' on its own never gets a package skipped. Fixes https://github.com/pnpm/pnpm/issues/11702 Fixes https://github.com/pnpm/pnpm/issues/9940 * test: shut the metadata-stripping proxy down cleanly and forward the request method |
||
|
Zoltan Kochan
|
5aed1200ea |
feat: add musl binaries for pacquet and pnpr (#12316)
Summary: - add Linux musl binary package selection to the pacquet and pnpr npm shims - generate linux-x64-musl and linux-arm64-musl native npm packages with libc metadata - build musl Rust release targets for both pacquet and pnpr - update package docs and cspell entries for the touched workflow files |
||
|
Zoltan Kochan
|
3d50680eda |
fix(security): verify Node.js runtime SHASUMS OpenPGP signature (#12295)
Follow-up to #12292 (which verifies the **package-manager** binary). This closes the same class of gap for the **Node.js runtime**. When a repository requests a Node.js runtime — `devEngines.runtime: node@X` (with `onFail: download`, the default) or `useNodeVersion` — pnpm downloads and then executes a Node binary (it's used to run lifecycle / `run` / `exec` scripts). The download **mirror is repository-configurable** via `node-mirror:<channel>` (`nodeDownloadMirrors`) in project `.npmrc`, and the integrity comes from `SHASUMS256.txt` fetched **from that same mirror**. That's a circular check: a malicious mirror serves a tampered `node` tarball **and** a matching `SHASUMS256.txt`, the sha256 check passes, and pnpm runs the binary. Drive-by on a normal command in a cloned repo. ## Fix pnpm now fetches `SHASUMS256.txt.sig` and verifies its **detached OpenPGP signature** against the **Node.js release team's public keys, embedded in the pnpm CLI**, before trusting the hashes. A mirror that serves a tampered binary cannot also produce a valid signature, so verification fails. Any faithful mirror (one that proxies the real signed SHASUMS) keeps working. - `@pnpm/crypto.shasums-file`: new `fetchVerifiedNodeShasums` / `fetchVerifiedNodeShasumsFile` verify the signature via `openpgp` against the embedded keys. - The keys live in a generated file (`src/nodeReleaseKeys.ts`, 28 keys) mirrored from the canonical `nodejs/release-keys` list. `crypto/shasums-file/scripts/update-node-release-keys.mjs` keeps them current (`pnpm check:node-release-keys` / `--update`), and the **create-release-pr** workflow runs the check as a gate so a new release signer can't silently break verification. - `@pnpm/engine.runtime.node-resolver` verifies the **configurable-mirror** SHASUMS. The hardcoded `unofficial-builds.nodejs.org` musl mirror is **not** repo-configurable and is signed by a different key, so it stays trusted over TLS. ## Scope - **Pre-release channels (rc, nightly, …) are not verified** — Node only signs the `release` channel (no `SHASUMS256.txt.sig` exists for them, even on nodejs.org), so they remain unverifiable. Verification is gated on the `release` channel. - **Bun / Deno are unaffected** — their download/SHASUMS URLs are hardcoded to canonical GitHub (`github.com/oven-sh/bun`, `api.github.com/repos/denoland/deno`), not mirror-configurable, so a repo can't redirect them. - **Pacquet parity:** `pacquet/crates/engine-runtime-node-resolver` has the same mirror-configurable SHASUMS logic and needs the equivalent Rust port — tracked as a follow-up (per the repo's parity rule, opening the TS side first). |
||
|
Zoltan Kochan
|
b4cc602b6d | docs: update the list of sponsors (#12265) | ||
|
Zoltan Kochan
|
70554b8677 |
feat(pnpr): config-selectable networked-SQLite auth backend (#12199 phase 3) (#12206)
## What Implements the **auth half** of [#12199](https://github.com/pnpm/pnpm/issues/12199) (phase 3) — making pnpr's remaining per-instance state pluggable so the registry can run as stateless, horizontally-scaled replicas. ### Auth records behind config-selected backends Users + tokens now sit behind narrow async `UserBackend` / `TokenBackend` traits, built once at startup into `Arc<dyn …>` handles (the same build-once pattern #12198 used for the hosted store). Three implementations: - **Local** (default) — today's htpasswd file + SQLite token DB, or in-memory when no file is configured. Unchanged behavior. - **Networked SQLite (libsql / Turso)** — `LibsqlAuth` stores **both** records in one shared database, so several stateless replicas observe a consistent set of users and tokens. The `tokens` table DDL is shared verbatim with the local backend (a DB can migrate between them); users — which the local backend keeps in htpasswd — move into a `users` table. Selected via a new top-level YAML block: ```yaml backend: libsql: url: ${PNPR_LIBSQL_URL} authToken: ${PNPR_LIBSQL_TOKEN} # optional embedded replica for local-fast hot-path reads: replicaPath: ./auth-replica.db syncIntervalSecs: 60 ``` When the block is absent, auth stays on local disk exactly as before. ### Embedded-replica read acceleration Token lookups are on the request hot path, so against a remote primary every read would be a network round-trip. With `replicaPath` set, `LibsqlAuth` builds a libsql **embedded replica**: reads hit a local file that libsql keeps current in the background; writes go to the primary. `syncIntervalSecs` is the freshness knob that bounds token-revocation lag. ### Async access path `identify` / `enforce_access` are now async (a networked lookup is async). `enforce_access` is split into an async `resolve_identity` + a sync `authorize`, so the search endpoint resolves the caller once and authorizes each candidate synchronously (no async-in-`retain`). ### Concurrent-publish guard (cross-cutting follow-up from the issue) Closes the same-instance lost-update window in the three read-modify-write packument flows (publish, dist-tag change, partial-unpublish): a striped per-package lock serializes same-package writers on one instance while letting different packages proceed in parallel. The **cross-replica** half (S3 `If-Match` / ETag CAS) is documented in-code as the remaining piece — the issue files it under "fix when we get there," and it belongs with the multi-writer S3 publish work, not this auth branch. ## Tests All green — `cargo test -p pnpr`: - **176 lib unit tests** incl. new `LibsqlAuth` tests (run against an in-memory libsql DB — same driver + SQL, no server) and `backend.libsql` config-parsing tests (incl. the replica options). - New `concurrent_publishes_of_distinct_versions_all_survive` integration test for the publish guard. - Existing auth_persistence / auth_user_endpoints / auth_publish / server / s3_backend suites pass. - Clean under `cargo fmt`, `clippy`, `RUSTDOCFLAGS=-D warnings cargo doc`, **Dylint perfectionist**, and `taplo`. ## Docs `backend.libsql` (incl. embedded replica) documented in the bundled `config.yaml` and the `pnpr` npm README, mirroring how the S3 backend was documented in #12198. |
||
|
Ruben NogueiraandZoltan Kochan
|
4e740d5562 |
fix(gvs): run dependency build scripts under the global virtual store (#11987)
* fix: unavailable dep * fix(gvs): re-link lifecycle-script bins into the GVS projection The post-lifecycle bin re-link pass used the classic virtualStoreDir path even under the global virtual store, so bins created by the build scripts this fix runs were never re-linked into the GVS projection. Use the same pkgModulesDir helper as the rest of the rebuild path. Also thread enableGlobalVirtualStore through the (recursive) rebuild command opts explicitly, and list pnpm in the changeset. * chore: add prebuild to cspell dictionary * test(gvs): cover rebuilding a shared dependency across workspace projects Adds a multi-project recursive rebuild test under the global virtual store with per-project lockfiles (sharedWorkspaceLockfile: false), which routes through recursiveRebuild's per-project concurrent branch. Both projects depend on the same package, deduped into one shared GVS projection, so the concurrent passes select the same projection directory and exercise the per-projection build lock. Asserts the projection is deduped to one directory and built exactly once. * test(gvs): reword comment to satisfy spellcheck * docs: trim verbose comments to 2-3 lines --------- Co-authored-by: Zoltan Kochan <z@kochan.io> |
||
|
Zoltan Kochan
|
8e5e764037 |
feat(pnpr): store hosted packages in an S3-compatible object store (#12198)
## What
Lets pnpr store its **hosted** packages (the ones published to it, plus static-served content) in an **S3-compatible object store** instead of a local directory. Because the same code targets any S3-compatible endpoint, this also covers **Cloudflare R2**, MinIO, Backblaze B2, Wasabi, etc.
The local `tokio::fs` path remains the default — nothing changes unless you add the new `s3:` config block.
## Why
The hosted store is pnpr's source of truth: durable, must be backed up, and can't be regenerated. That's exactly what belongs in object storage:
- The provider handles durability/replication, so there's no single-node volume to back up.
- Multiple **stateless pnpr replicas** can share one hosted store.
- R2 is the S3 API, so a configurable `endpoint` gets it (and the other S3-compatibles) for free.
The disposable proxy cache and the install-accelerator SQLite stores deliberately **stay on local disk** — they're ephemeral, latency-sensitive, and streamed/locked in filesystem-shaped ways.
## How
- New `s3.rs` module: `S3Settings` (the YAML `s3:` block), a client builder (`object_store` crate; AWS-env credentials with explicit override, plus R2/MinIO/path-style/HTTP knobs), and an `S3Store` adapter (packument get/put, streaming tarball get, staged upload, delete, prefix-scoped list).
- `storage.rs`: a `HostedStore { Fs | S3 }` backend enum routes the hosted ops; the `cached` store stays fs-only. Publish stages the decoded+verified tarball to local scratch, then finalize either renames (fs) or uploads (S3). `open_tarball` now returns a streaming response body so S3 reads stream straight through.
- `config.rs`: parses `s3:` and builds the client once at config-load time (the only fallible step), so `Storage` construction stays infallible.
- `search.rs`: local search now lists package names through the storage abstraction, so it works against a bucket too.
- Documented (commented) in the bundled `config.yaml`.
### Example: Cloudflare R2
```yaml
storage: ./storage # still backs the local proxy cache + upload staging
s3:
bucket: my-pnpr-packages
region: auto
endpoint: https://<account-id>.r2.cloudflarestorage.com
accessKeyId: ${PNPR_S3_ACCESS_KEY_ID}
secretAccessKey: ${PNPR_S3_SECRET_ACCESS_KEY}
```
|
||
|
Zoltan Kochan
|
33921c8019 |
fix(publish): normalize string repository field to object form (#12109)
* fix(publish): normalize string repository field to object form
Some registries (e.g. Gitea) reject a string `repository` field with a
500 Internal Server Error, since they decode it into an object-typed
struct. npm normalizes a string repository into { type, url } before
publishing; pnpm did not, so `pnpm publish` failed where npm succeeded.
Close #12099
* chore: add Codeberg to cspell dictionary
|