## Summary
Adds an opt-in **pnpm agent** server that resolves dependencies server-side and streams only the files missing from the client's content-addressable store.
- **`@pnpm/agent.server`** — multi-process HTTP server (Node.js `cluster`) with SQLite-backed metadata and file caches
- **`@pnpm/agent.client`** — streams an NDJSON response, dispatches worker threads to fetch files while the server is still resolving
- **New config**: `agent` in `pnpm-workspace.yaml` (opt-in)
## How it works
1. Client reads integrity hashes from its local store index
2. Sends `POST /v1/install` with dependencies + store integrities
3. Server resolves the dependency tree using pnpm's `install({ lockfileOnly: true })`, with a SQLite-backed `PackageMetaCache` for fast repeat resolution
4. As each package resolves, a wrapped `storeController.requestPackage` looks up its files and immediately streams digests the client is missing (NDJSON `D` lines)
5. Client reads the stream line by line; digest batches fill up and dispatch worker threads to `POST /v1/files` — file downloads overlap with server-side resolution
6. After resolution, server sends index entries (`I` lines) and lockfile (`L` line)
7. Client writes index entries to store, then runs headless install with a wrapped `fetchPackage` that calls `readPkgFromCafs` with `verifyStoreIntegrity: false` (files are trusted from the agent)
8. `/v1/files` response is gzip-streamed (274MB → ~80MB) — server pipes through `createGzip`, worker pipes through `createGunzip`, parsing and writing files to CAFS as data arrives
## Performance
1351-package project, cold local store, warm server (localhost):
| Scenario | Time |
|----------|------|
| Vanilla pnpm install (cold OS cache) | ~48s |
| Vanilla pnpm install (warm OS cache) | ~34s |
| With pnpm agent (consistent) | **~33s** |
### Key optimizations
1. **SQLite metadata cache** — server-side resolution drops from ~3.4s to ~0.9s
2. **SQLite file store** — consistent read performance regardless of OS file cache state
3. **Streaming `/v1/install`** — file digests stream during resolution, downloads start before resolution finishes
4. **Gzip-streamed `/v1/files`** — whole-stream gzip (274MB → ~80MB), significant savings on remote servers
5. **Worker-thread streaming HTTP** — workers pipe gzip → parse → write to CAFS as data arrives, no buffering
6. **No rehashing** — server-provided digests used directly, skipping 33K SHA-512 computations
7. **No re-verification** — wrapped `fetchPackage` calls `readPkgFromCafs` with `verifyStoreIntegrity: false`
8. **Direct `writeFileSync` with `wx`** — no stat + temp + rename
9. **Pre-packed msgpack** — server sends raw store index buffers, client writes directly to SQLite
10. **WAL checkpoint** — ensures store index entries written by agent are visible to headless install's worker threads
## Usage
Start the server:
```bash
node agent/server/lib/bin.js
```
Configure in `pnpm-workspace.yaml`:
```yaml
agent: http://localhost:4873
```
pnpm-agent
A pnpm agent server that resolves dependencies server-side and streams only the files missing from the client's content-addressable store.
Status: experimental. Versions are pre-1.0; the wire protocol may change between releases.
How it works
- Client sends
POST /v1/installwith dependencies, an optional existing lockfile, and the integrity hashes of packages already in its store. - Server resolves the full dependency tree using pnpm's own resolution engine.
- Server computes which file digests the client is missing — at the individual file level, not just the package level.
- Server streams an NDJSON response on
/v1/install(D-lines for missing file digests,I-lines for pre-packed package index entries, a finalL-line with the lockfile + stats, or anE-line on error). - The client then requests the missing file contents from
POST /v1/files, which streams a gzip-compressed binary of packed file entries.
This eliminates sequential metadata round-trips (the server resolves in one shot) and avoids downloading files that already exist in the client's store from other packages.
Starting the server
Install from npm
pnpm add -g pnpm-agent
pnpm-agent
Docker
A Dockerfile is provided at agent/server/Dockerfile. It is layered on top of ghcr.io/pnpm/pnpm and installs Node.js and pnpm-agent inside the image.
# Build the image locally
docker build -t pnpm-agent agent/server
# Run it, persisting the store + cache in ./agent-data
docker run --rm \
-p 4873:4873 \
-v "$(pwd)/agent-data:/agent-data" \
pnpm-agent
Override the defaults with -e, same variables as described below:
docker run --rm \
-p 4000:4000 \
-e PORT=4000 \
-e PNPM_AGENT_UPSTREAM=https://my-proxy.example.com/ \
-v "$(pwd)/agent-data:/agent-data" \
pnpm-agent
The image exposes port 4873 and declares a /agent-data volume; mount a host directory there if you want the resolved metadata, store index, and file store to survive container restarts.
From source
# Build first
pnpm --filter pnpm-agent run compile
# Run with defaults (port 4873, upstream https://registry.npmjs.org/)
node lib/bin.js
# Or configure via environment variables
PORT=4000 \
PNPM_AGENT_STORE_DIR=./my-store \
PNPM_AGENT_CACHE_DIR=./my-cache \
PNPM_AGENT_UPSTREAM=https://registry.npmjs.org/ \
node lib/bin.js
Environment variables
| Variable | Default | Description |
|---|---|---|
PORT |
4873 |
Port to listen on |
PNPM_AGENT_STORE_DIR |
./store |
Directory for the server's content-addressable store |
PNPM_AGENT_CACHE_DIR |
./cache |
Directory for package metadata cache |
PNPM_AGENT_UPSTREAM |
https://registry.npmjs.org/ |
Upstream npm registry to resolve from |
Programmatic usage
import { createRegistryServer } from 'pnpm-agent'
const server = await createRegistryServer({
storeDir: '/var/lib/pnpm-agent/store',
cacheDir: '/var/lib/pnpm-agent/cache',
registries: { default: 'https://registry.npmjs.org/' },
})
server.listen(4000, () => {
console.log('pnpm agent listening on port 4000')
})
Quick start
Terminal 1 — start the server:
cd agent/server
pnpm run compile
node lib/bin.js
# pnpm agent server listening on http://localhost:4873
Terminal 2 — use it from any project:
cd my-project
Add to pnpm-workspace.yaml:
agent: http://localhost:4873
Or pass --config.agent=http://localhost:4873 on the command line.
Then run:
pnpm install
That's it. pnpm will resolve dependencies on the server, download only the files missing from your local store, and link node_modules as usual. Remove the agent setting to go back to normal behavior.
API
POST /v1/install
Request body (JSON):
{
"projects": [
{
"dir": ".",
"dependencies": { "react": "^19.0.0" },
"devDependencies": { "typescript": "^5.0.0" }
}
],
"overrides": {},
"lockfile": null,
"storeIntegrities": ["sha512-abc...", "sha512-def..."]
}
Response (NDJSON, Content-Type: application/x-ndjson). Each line is one message:
D\t{digest}\t{size}\t{executable}— file digest missing from the client's store.I\t{integrity}\t{pkgId}\t{base64-msgpack}— pre-packed package index entry.L\t{json}— final lockfile and stats. Emitted last on success.E\t{json}— error. Emitted if resolution fails.
POST /v1/files
Request body (JSON):
{ "digests": [{ "digest": "<hex>", "size": 123, "executable": false }] }
Response (gzip-compressed binary, Content-Type: application/x-pnpm-install):
[4 bytes: JSON metadata length]
[N bytes: JSON metadata]
[file entries: 64B digest + 4B size + 1B mode + content, repeated]
[64 zero bytes: end marker]