mirror of
https://github.com/pnpm/pnpm.git
synced 2026-05-18 05:42:27 -04:00
## Summary
Adds an opt-in **pnpm agent** server that resolves dependencies server-side and streams only the files missing from the client's content-addressable store.
- **`@pnpm/agent.server`** — multi-process HTTP server (Node.js `cluster`) with SQLite-backed metadata and file caches
- **`@pnpm/agent.client`** — streams an NDJSON response, dispatches worker threads to fetch files while the server is still resolving
- **New config**: `agent` in `pnpm-workspace.yaml` (opt-in)
## How it works
1. Client reads integrity hashes from its local store index
2. Sends `POST /v1/install` with dependencies + store integrities
3. Server resolves the dependency tree using pnpm's `install({ lockfileOnly: true })`, with a SQLite-backed `PackageMetaCache` for fast repeat resolution
4. As each package resolves, a wrapped `storeController.requestPackage` looks up its files and immediately streams digests the client is missing (NDJSON `D` lines)
5. Client reads the stream line by line; digest batches fill up and dispatch worker threads to `POST /v1/files` — file downloads overlap with server-side resolution
6. After resolution, server sends index entries (`I` lines) and lockfile (`L` line)
7. Client writes index entries to store, then runs headless install with a wrapped `fetchPackage` that calls `readPkgFromCafs` with `verifyStoreIntegrity: false` (files are trusted from the agent)
8. `/v1/files` response is gzip-streamed (274MB → ~80MB) — server pipes through `createGzip`, worker pipes through `createGunzip`, parsing and writing files to CAFS as data arrives
## Performance
1351-package project, cold local store, warm server (localhost):
| Scenario | Time |
|----------|------|
| Vanilla pnpm install (cold OS cache) | ~48s |
| Vanilla pnpm install (warm OS cache) | ~34s |
| With pnpm agent (consistent) | **~33s** |
### Key optimizations
1. **SQLite metadata cache** — server-side resolution drops from ~3.4s to ~0.9s
2. **SQLite file store** — consistent read performance regardless of OS file cache state
3. **Streaming `/v1/install`** — file digests stream during resolution, downloads start before resolution finishes
4. **Gzip-streamed `/v1/files`** — whole-stream gzip (274MB → ~80MB), significant savings on remote servers
5. **Worker-thread streaming HTTP** — workers pipe gzip → parse → write to CAFS as data arrives, no buffering
6. **No rehashing** — server-provided digests used directly, skipping 33K SHA-512 computations
7. **No re-verification** — wrapped `fetchPackage` calls `readPkgFromCafs` with `verifyStoreIntegrity: false`
8. **Direct `writeFileSync` with `wx`** — no stat + temp + rename
9. **Pre-packed msgpack** — server sends raw store index buffers, client writes directly to SQLite
10. **WAL checkpoint** — ensures store index entries written by agent are visible to headless install's worker threads
## Usage
Start the server:
```bash
node agent/server/lib/bin.js
```
Configure in `pnpm-workspace.yaml`:
```yaml
agent: http://localhost:4873
```
57 lines
1.8 KiB
TypeScript
57 lines
1.8 KiB
TypeScript
import cluster from 'node:cluster'
|
|
import os from 'node:os'
|
|
|
|
import { createRegistryServer, warmupMetadataCache } from './createRegistryServer.js'
|
|
|
|
const port = parseInt(process.env['PORT'] ?? '4873', 10)
|
|
const storeDir = process.env['PNPM_AGENT_STORE_DIR'] ?? './store'
|
|
const cacheDir = process.env['PNPM_AGENT_CACHE_DIR'] ?? './cache'
|
|
const upstream = process.env['PNPM_AGENT_UPSTREAM'] ?? 'https://registry.npmjs.org/'
|
|
const numWorkers = parseInt(process.env['PNPM_AGENT_WORKERS'] ?? String(Math.max(2, os.availableParallelism() - 1)), 10)
|
|
|
|
void main().catch((err: unknown) => {
|
|
console.error('Failed to start pnpm agent server', err)
|
|
process.exit(1)
|
|
})
|
|
|
|
async function main (): Promise<void> {
|
|
if (cluster.isPrimary) {
|
|
console.log(`pnpm agent server starting on http://localhost:${port}`)
|
|
console.log(` store: ${storeDir}`)
|
|
console.log(` cache: ${cacheDir}`)
|
|
console.log(` upstream: ${upstream}`)
|
|
console.log(` workers: ${numWorkers}`)
|
|
|
|
// Warm the metadata cache once here so forked workers skip the scan —
|
|
// running it per worker duplicates expensive I/O and causes SQLite
|
|
// write contention.
|
|
const imported = await warmupMetadataCache({ storeDir, cacheDir })
|
|
if (imported > 0) {
|
|
console.log(` imported ${imported} metadata entries to SQLite`)
|
|
}
|
|
|
|
for (let i = 0; i < numWorkers; i++) {
|
|
cluster.fork()
|
|
}
|
|
|
|
cluster.on('exit', (worker, code) => {
|
|
if (code !== 0) {
|
|
console.log(`Worker ${worker.process.pid} died (code ${code}), restarting...`)
|
|
cluster.fork()
|
|
}
|
|
})
|
|
return
|
|
}
|
|
|
|
const server = await createRegistryServer({
|
|
storeDir,
|
|
cacheDir,
|
|
registries: { default: upstream },
|
|
skipCacheImport: true,
|
|
})
|
|
|
|
server.listen(port, () => {
|
|
console.log(` worker ${process.pid} listening`)
|
|
})
|
|
}
|