mirror of
https://github.com/pnpm/pnpm.git
synced 2026-04-27 10:30:58 -04:00
ccc606ed15
## Summary
Adds an opt-in **pnpm agent** server that resolves dependencies server-side and streams only the files missing from the client's content-addressable store.
- **`@pnpm/agent.server`** — multi-process HTTP server (Node.js `cluster`) with SQLite-backed metadata and file caches
- **`@pnpm/agent.client`** — streams an NDJSON response, dispatches worker threads to fetch files while the server is still resolving
- **New config**: `agent` in `pnpm-workspace.yaml` (opt-in)
## How it works
1. Client reads integrity hashes from its local store index
2. Sends `POST /v1/install` with dependencies + store integrities
3. Server resolves the dependency tree using pnpm's `install({ lockfileOnly: true })`, with a SQLite-backed `PackageMetaCache` for fast repeat resolution
4. As each package resolves, a wrapped `storeController.requestPackage` looks up its files and immediately streams digests the client is missing (NDJSON `D` lines)
5. Client reads the stream line by line; digest batches fill up and dispatch worker threads to `POST /v1/files` — file downloads overlap with server-side resolution
6. After resolution, server sends index entries (`I` lines) and lockfile (`L` line)
7. Client writes index entries to store, then runs headless install with a wrapped `fetchPackage` that calls `readPkgFromCafs` with `verifyStoreIntegrity: false` (files are trusted from the agent)
8. `/v1/files` response is gzip-streamed (274MB → ~80MB) — server pipes through `createGzip`, worker pipes through `createGunzip`, parsing and writing files to CAFS as data arrives
## Performance
1351-package project, cold local store, warm server (localhost):
| Scenario | Time |
|----------|------|
| Vanilla pnpm install (cold OS cache) | ~48s |
| Vanilla pnpm install (warm OS cache) | ~34s |
| With pnpm agent (consistent) | **~33s** |
### Key optimizations
1. **SQLite metadata cache** — server-side resolution drops from ~3.4s to ~0.9s
2. **SQLite file store** — consistent read performance regardless of OS file cache state
3. **Streaming `/v1/install`** — file digests stream during resolution, downloads start before resolution finishes
4. **Gzip-streamed `/v1/files`** — whole-stream gzip (274MB → ~80MB), significant savings on remote servers
5. **Worker-thread streaming HTTP** — workers pipe gzip → parse → write to CAFS as data arrives, no buffering
6. **No rehashing** — server-provided digests used directly, skipping 33K SHA-512 computations
7. **No re-verification** — wrapped `fetchPackage` calls `readPkgFromCafs` with `verifyStoreIntegrity: false`
8. **Direct `writeFileSync` with `wx`** — no stat + temp + rename
9. **Pre-packed msgpack** — server sends raw store index buffers, client writes directly to SQLite
10. **WAL checkpoint** — ensures store index entries written by agent are visible to headless install's worker threads
## Usage
Start the server:
```bash
node agent/server/lib/bin.js
```
Configure in `pnpm-workspace.yaml`:
```yaml
agent: http://localhost:4873
```
44 lines
1.8 KiB
Markdown
44 lines
1.8 KiB
Markdown
# @pnpm/agent.client
|
|
|
|
Client library for the pnpm agent server. Reads the local store state, sends it to the server, and writes the received files into the content-addressable store.
|
|
|
|
## How it works
|
|
|
|
1. Reads integrity hashes from the local store index (`index.db`).
|
|
2. Sends `POST /v1/install` to the pnpm agent server with the project's dependencies and the store integrities.
|
|
3. Parses the NDJSON streaming response — `D`-lines (missing file digests) are dispatched to worker downloads against `/v1/files`, `I`-lines are buffered as raw store-index entries, and the final `L`-line yields the resolved lockfile and stats.
|
|
4. File download workers write each received file directly to the local CAFS (`files/{hash[:2]}/{hash[2:]}`).
|
|
5. Writes store index entries for all new packages in a single SQLite transaction.
|
|
6. Returns the resolved lockfile for use with pnpm's headless install (linking phase).
|
|
|
|
## Usage
|
|
|
|
This package is used internally by pnpm when the `agent` config option is set. It is not intended to be called directly, but can be used programmatically:
|
|
|
|
```typescript
|
|
import { fetchFromPnpmRegistry } from '@pnpm/agent.client'
|
|
import { StoreIndex } from '@pnpm/store.index'
|
|
|
|
const storeIndex = new StoreIndex('/path/to/store')
|
|
|
|
const { lockfile, stats } = await fetchFromPnpmRegistry({
|
|
registryUrl: 'http://localhost:4000',
|
|
storeDir: '/path/to/store',
|
|
storeIndex,
|
|
dependencies: { react: '^19.0.0' },
|
|
devDependencies: { typescript: '^5.0.0' },
|
|
})
|
|
|
|
console.log(`Resolved ${stats.totalPackages} packages`)
|
|
console.log(`${stats.alreadyInStore} cached, ${stats.filesToDownload} files downloaded`)
|
|
// lockfile is ready for headless install
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Add to `pnpm-workspace.yaml` to enable automatically during `pnpm install`:
|
|
|
|
```yaml
|
|
agent: http://localhost:4000
|
|
```
|