mirror of
https://github.com/pnpm/pnpm.git
synced 2026-04-27 10:30:58 -04:00
ccc606ed15
## Summary
Adds an opt-in **pnpm agent** server that resolves dependencies server-side and streams only the files missing from the client's content-addressable store.
- **`@pnpm/agent.server`** — multi-process HTTP server (Node.js `cluster`) with SQLite-backed metadata and file caches
- **`@pnpm/agent.client`** — streams an NDJSON response, dispatches worker threads to fetch files while the server is still resolving
- **New config**: `agent` in `pnpm-workspace.yaml` (opt-in)
## How it works
1. Client reads integrity hashes from its local store index
2. Sends `POST /v1/install` with dependencies + store integrities
3. Server resolves the dependency tree using pnpm's `install({ lockfileOnly: true })`, with a SQLite-backed `PackageMetaCache` for fast repeat resolution
4. As each package resolves, a wrapped `storeController.requestPackage` looks up its files and immediately streams digests the client is missing (NDJSON `D` lines)
5. Client reads the stream line by line; digest batches fill up and dispatch worker threads to `POST /v1/files` — file downloads overlap with server-side resolution
6. After resolution, server sends index entries (`I` lines) and lockfile (`L` line)
7. Client writes index entries to store, then runs headless install with a wrapped `fetchPackage` that calls `readPkgFromCafs` with `verifyStoreIntegrity: false` (files are trusted from the agent)
8. `/v1/files` response is gzip-streamed (274MB → ~80MB) — server pipes through `createGzip`, worker pipes through `createGunzip`, parsing and writing files to CAFS as data arrives
## Performance
1351-package project, cold local store, warm server (localhost):
| Scenario | Time |
|----------|------|
| Vanilla pnpm install (cold OS cache) | ~48s |
| Vanilla pnpm install (warm OS cache) | ~34s |
| With pnpm agent (consistent) | **~33s** |
### Key optimizations
1. **SQLite metadata cache** — server-side resolution drops from ~3.4s to ~0.9s
2. **SQLite file store** — consistent read performance regardless of OS file cache state
3. **Streaming `/v1/install`** — file digests stream during resolution, downloads start before resolution finishes
4. **Gzip-streamed `/v1/files`** — whole-stream gzip (274MB → ~80MB), significant savings on remote servers
5. **Worker-thread streaming HTTP** — workers pipe gzip → parse → write to CAFS as data arrives, no buffering
6. **No rehashing** — server-provided digests used directly, skipping 33K SHA-512 computations
7. **No re-verification** — wrapped `fetchPackage` calls `readPkgFromCafs` with `verifyStoreIntegrity: false`
8. **Direct `writeFileSync` with `wx`** — no stat + temp + rename
9. **Pre-packed msgpack** — server sends raw store index buffers, client writes directly to SQLite
10. **WAL checkpoint** — ensures store index entries written by agent are visible to headless install's worker threads
## Usage
Start the server:
```bash
node agent/server/lib/bin.js
```
Configure in `pnpm-workspace.yaml`:
```yaml
agent: http://localhost:4873
```
66 lines
1.8 KiB
JSON
66 lines
1.8 KiB
JSON
{
|
|
"name": "pnpm-agent",
|
|
"version": "0.0.1-0",
|
|
"description": "pnpm agent server for server-side resolution and store-aware downloads",
|
|
"keywords": [
|
|
"pnpm",
|
|
"pnpm11"
|
|
],
|
|
"license": "MIT",
|
|
"funding": "https://opencollective.com/pnpm",
|
|
"repository": "https://github.com/pnpm/pnpm/tree/main/agent/server",
|
|
"homepage": "https://github.com/pnpm/pnpm/tree/main/agent/server#readme",
|
|
"bugs": {
|
|
"url": "https://github.com/pnpm/pnpm/issues"
|
|
},
|
|
"type": "module",
|
|
"main": "lib/index.js",
|
|
"types": "lib/index.d.ts",
|
|
"exports": {
|
|
".": "./lib/index.js"
|
|
},
|
|
"files": [
|
|
"lib",
|
|
"!*.map",
|
|
"bin"
|
|
],
|
|
"bin": {
|
|
"pnpm-agent": "./lib/bin.js"
|
|
},
|
|
"scripts": {
|
|
"start": "tsgo --watch",
|
|
"lint": "eslint \"src/**/*.ts\" \"test/**/*.ts\"",
|
|
"test": "pn compile && pn .test",
|
|
"prepublishOnly": "pn compile",
|
|
"compile": "tsgo --build && pn lint --fix",
|
|
".test": "cross-env NODE_OPTIONS=\"$NODE_OPTIONS --experimental-vm-modules --disable-warning=ExperimentalWarning --disable-warning=DEP0169\" jest"
|
|
},
|
|
"dependencies": {
|
|
"@pnpm/installing.client": "workspace:*",
|
|
"@pnpm/installing.deps-installer": "workspace:*",
|
|
"@pnpm/lockfile.fs": "workspace:*",
|
|
"@pnpm/lockfile.types": "workspace:*",
|
|
"@pnpm/store.cafs": "workspace:*",
|
|
"@pnpm/store.controller": "workspace:*",
|
|
"@pnpm/store.index": "workspace:*",
|
|
"@pnpm/types": "workspace:*"
|
|
},
|
|
"peerDependencies": {
|
|
"@pnpm/logger": "catalog:"
|
|
},
|
|
"devDependencies": {
|
|
"@jest/globals": "catalog:",
|
|
"@pnpm/agent.client": "workspace:*",
|
|
"@pnpm/logger": "workspace:*",
|
|
"@pnpm/registry-mock": "catalog:",
|
|
"cross-env": "catalog:",
|
|
"pnpm-agent": "workspace:*"
|
|
},
|
|
"engines": {
|
|
"node": ">=22.13"
|
|
},
|
|
"jest": {
|
|
"preset": "@pnpm/jest-config/with-registry"
|
|
}
|
|
}
|