mirror of
https://github.com/pnpm/pnpm.git
synced 2026-04-26 18:09:06 -04:00
ccc606ed15
## Summary
Adds an opt-in **pnpm agent** server that resolves dependencies server-side and streams only the files missing from the client's content-addressable store.
- **`@pnpm/agent.server`** — multi-process HTTP server (Node.js `cluster`) with SQLite-backed metadata and file caches
- **`@pnpm/agent.client`** — streams an NDJSON response, dispatches worker threads to fetch files while the server is still resolving
- **New config**: `agent` in `pnpm-workspace.yaml` (opt-in)
## How it works
1. Client reads integrity hashes from its local store index
2. Sends `POST /v1/install` with dependencies + store integrities
3. Server resolves the dependency tree using pnpm's `install({ lockfileOnly: true })`, with a SQLite-backed `PackageMetaCache` for fast repeat resolution
4. As each package resolves, a wrapped `storeController.requestPackage` looks up its files and immediately streams digests the client is missing (NDJSON `D` lines)
5. Client reads the stream line by line; digest batches fill up and dispatch worker threads to `POST /v1/files` — file downloads overlap with server-side resolution
6. After resolution, server sends index entries (`I` lines) and lockfile (`L` line)
7. Client writes index entries to store, then runs headless install with a wrapped `fetchPackage` that calls `readPkgFromCafs` with `verifyStoreIntegrity: false` (files are trusted from the agent)
8. `/v1/files` response is gzip-streamed (274MB → ~80MB) — server pipes through `createGzip`, worker pipes through `createGunzip`, parsing and writing files to CAFS as data arrives
## Performance
1351-package project, cold local store, warm server (localhost):
| Scenario | Time |
|----------|------|
| Vanilla pnpm install (cold OS cache) | ~48s |
| Vanilla pnpm install (warm OS cache) | ~34s |
| With pnpm agent (consistent) | **~33s** |
### Key optimizations
1. **SQLite metadata cache** — server-side resolution drops from ~3.4s to ~0.9s
2. **SQLite file store** — consistent read performance regardless of OS file cache state
3. **Streaming `/v1/install`** — file digests stream during resolution, downloads start before resolution finishes
4. **Gzip-streamed `/v1/files`** — whole-stream gzip (274MB → ~80MB), significant savings on remote servers
5. **Worker-thread streaming HTTP** — workers pipe gzip → parse → write to CAFS as data arrives, no buffering
6. **No rehashing** — server-provided digests used directly, skipping 33K SHA-512 computations
7. **No re-verification** — wrapped `fetchPackage` calls `readPkgFromCafs` with `verifyStoreIntegrity: false`
8. **Direct `writeFileSync` with `wx`** — no stat + temp + rename
9. **Pre-packed msgpack** — server sends raw store index buffers, client writes directly to SQLite
10. **WAL checkpoint** — ensures store index entries written by agent are visible to headless install's worker threads
## Usage
Start the server:
```bash
node agent/server/lib/bin.js
```
Configure in `pnpm-workspace.yaml`:
```yaml
agent: http://localhost:4873
```
@pnpm/agent.client
Client library for the pnpm agent server. Reads the local store state, sends it to the server, and writes the received files into the content-addressable store.
How it works
- Reads integrity hashes from the local store index (
index.db). - Sends
POST /v1/installto the pnpm agent server with the project's dependencies and the store integrities. - Parses the NDJSON streaming response —
D-lines (missing file digests) are dispatched to worker downloads against/v1/files,I-lines are buffered as raw store-index entries, and the finalL-line yields the resolved lockfile and stats. - File download workers write each received file directly to the local CAFS (
files/{hash[:2]}/{hash[2:]}). - Writes store index entries for all new packages in a single SQLite transaction.
- Returns the resolved lockfile for use with pnpm's headless install (linking phase).
Usage
This package is used internally by pnpm when the agent config option is set. It is not intended to be called directly, but can be used programmatically:
import { fetchFromPnpmRegistry } from '@pnpm/agent.client'
import { StoreIndex } from '@pnpm/store.index'
const storeIndex = new StoreIndex('/path/to/store')
const { lockfile, stats } = await fetchFromPnpmRegistry({
registryUrl: 'http://localhost:4000',
storeDir: '/path/to/store',
storeIndex,
dependencies: { react: '^19.0.0' },
devDependencies: { typescript: '^5.0.0' },
})
console.log(`Resolved ${stats.totalPackages} packages`)
console.log(`${stats.alreadyInStore} cached, ${stats.filesToDownload} files downloaded`)
// lockfile is ready for headless install
Configuration
Add to pnpm-workspace.yaml to enable automatically during pnpm install:
agent: http://localhost:4000