Files
pnpm/agent/server/src/bin.ts
Zoltan Kochan ccc606ed15 feat: pnpm agent — server-side resolution for faster installs (#11251)
## Summary

Adds an opt-in **pnpm agent** server that resolves dependencies server-side and streams only the files missing from the client's content-addressable store.

- **`@pnpm/agent.server`** — multi-process HTTP server (Node.js `cluster`) with SQLite-backed metadata and file caches
- **`@pnpm/agent.client`** — streams an NDJSON response, dispatches worker threads to fetch files while the server is still resolving
- **New config**: `agent` in `pnpm-workspace.yaml` (opt-in)

## How it works

1. Client reads integrity hashes from its local store index
2. Sends `POST /v1/install` with dependencies + store integrities
3. Server resolves the dependency tree using pnpm's `install({ lockfileOnly: true })`, with a SQLite-backed `PackageMetaCache` for fast repeat resolution
4. As each package resolves, a wrapped `storeController.requestPackage` looks up its files and immediately streams digests the client is missing (NDJSON `D` lines)
5. Client reads the stream line by line; digest batches fill up and dispatch worker threads to `POST /v1/files` — file downloads overlap with server-side resolution
6. After resolution, server sends index entries (`I` lines) and lockfile (`L` line)
7. Client writes index entries to store, then runs headless install with a wrapped `fetchPackage` that calls `readPkgFromCafs` with `verifyStoreIntegrity: false` (files are trusted from the agent)
8. `/v1/files` response is gzip-streamed (274MB → ~80MB) — server pipes through `createGzip`, worker pipes through `createGunzip`, parsing and writing files to CAFS as data arrives

## Performance

1351-package project, cold local store, warm server (localhost):

| Scenario | Time |
|----------|------|
| Vanilla pnpm install (cold OS cache) | ~48s |
| Vanilla pnpm install (warm OS cache) | ~34s |
| With pnpm agent (consistent) | **~33s** |

### Key optimizations

1. **SQLite metadata cache** — server-side resolution drops from ~3.4s to ~0.9s
2. **SQLite file store** — consistent read performance regardless of OS file cache state
3. **Streaming `/v1/install`** — file digests stream during resolution, downloads start before resolution finishes
4. **Gzip-streamed `/v1/files`** — whole-stream gzip (274MB → ~80MB), significant savings on remote servers
5. **Worker-thread streaming HTTP** — workers pipe gzip → parse → write to CAFS as data arrives, no buffering
6. **No rehashing** — server-provided digests used directly, skipping 33K SHA-512 computations
7. **No re-verification** — wrapped `fetchPackage` calls `readPkgFromCafs` with `verifyStoreIntegrity: false`
8. **Direct `writeFileSync` with `wx`** — no stat + temp + rename
9. **Pre-packed msgpack** — server sends raw store index buffers, client writes directly to SQLite
10. **WAL checkpoint** — ensures store index entries written by agent are visible to headless install's worker threads

## Usage

Start the server:
```bash
node agent/server/lib/bin.js
```

Configure in `pnpm-workspace.yaml`:
```yaml
agent: http://localhost:4873
```
2026-04-20 11:56:46 +02:00

57 lines
1.8 KiB
TypeScript

import cluster from 'node:cluster'
import os from 'node:os'
import { createRegistryServer, warmupMetadataCache } from './createRegistryServer.js'
const port = parseInt(process.env['PORT'] ?? '4873', 10)
const storeDir = process.env['PNPM_AGENT_STORE_DIR'] ?? './store'
const cacheDir = process.env['PNPM_AGENT_CACHE_DIR'] ?? './cache'
const upstream = process.env['PNPM_AGENT_UPSTREAM'] ?? 'https://registry.npmjs.org/'
const numWorkers = parseInt(process.env['PNPM_AGENT_WORKERS'] ?? String(Math.max(2, os.availableParallelism() - 1)), 10)
void main().catch((err: unknown) => {
console.error('Failed to start pnpm agent server', err)
process.exit(1)
})
async function main (): Promise<void> {
if (cluster.isPrimary) {
console.log(`pnpm agent server starting on http://localhost:${port}`)
console.log(` store: ${storeDir}`)
console.log(` cache: ${cacheDir}`)
console.log(` upstream: ${upstream}`)
console.log(` workers: ${numWorkers}`)
// Warm the metadata cache once here so forked workers skip the scan —
// running it per worker duplicates expensive I/O and causes SQLite
// write contention.
const imported = await warmupMetadataCache({ storeDir, cacheDir })
if (imported > 0) {
console.log(` imported ${imported} metadata entries to SQLite`)
}
for (let i = 0; i < numWorkers; i++) {
cluster.fork()
}
cluster.on('exit', (worker, code) => {
if (code !== 0) {
console.log(`Worker ${worker.process.pid} died (code ${code}), restarting...`)
cluster.fork()
}
})
return
}
const server = await createRegistryServer({
storeDir,
cacheDir,
registries: { default: upstream },
skipCacheImport: true,
})
server.listen(port, () => {
console.log(` worker ${process.pid} listening`)
})
}