mirror of
https://github.com/pnpm/pnpm.git
synced 2026-07-22 05:32:44 -04:00
* feat!(store): migrate from v3 JSON index to v11 SQLite+msgpack layout Pacquet now writes the same on-disk store that pnpm v11 uses, so running pacquet and pnpm on the same machine can share the tarball cache instead of maintaining parallel stores. Closes half of #244. - **Layout:** \`<store>/v11/files/XX/<hex-remainder>[-exec]\` for content-addressed blobs + \`<store>/v11/index.db\` for the per-package file index. The legacy \`v3/files/XX/<hex>-index.json\` JSON layout is gone (we don't need backward compat — pacquet is pre-1.0). - **SQLite schema / PRAGMAs:** byte-identical to pnpm's \`store/index/src/index.ts\` — \`package_index (key TEXT PRIMARY KEY, data BLOB NOT NULL) WITHOUT ROWID\` with WAL journal, 5s busy_timeout, 512 MB mmap, 32 MB page cache, temp_store=MEMORY, wal_autocheckpoint=10000. Two concurrent pacquet tasks (or a pacquet + pnpm pair) contend correctly via SQLite's own locking. - **Value encoding:** msgpack via \`rmp-serde::to_vec_named\` so structs are serialized with named fields, matching pnpm's msgpackr record encoding. \`PackageFilesIndex\` and \`CafsFileInfo\` types mirror pnpm's \`@pnpm/store.cafs\` shape field-for-field (including the hex \`digest\` rather than pacquet's old SSRI \`integrity\` string). - **Key format:** \`"{integrity}\\t{pkg_id}"\` via \`store_index_key()\`, same as pnpm's \`storeIndexKey\`. \`pkg_id\` is \`"{name}@{version}"\` — pulled from \`PkgNameVerPeer::without_peer()\` on the frozen-lockfile path and assembled from \`PackageVersion\` on the registry path. Breaking API changes inside the workspace: - \`pacquet-store-dir\` loses the v3 JSON types (\`PackageFilesIndex\`, \`PackageFileInfo\`, \`WriteIndexFileError\`) and \`StoreDir::v3()\` / \`::write_index_file()\`. In their place: \`StoreIndex\`, \`StoreIndexError\`, the pnpm-shape \`PackageFilesIndex\` + \`CafsFileInfo\`, and \`store_index_key()\`. Unit-tested with 9 new round-trip / re-open / key-format tests. - \`DownloadTarballToStore\` gains a \`package_id\` field. All three callers (two in \`pacquet-package-manager\`, one in the micro-benchmark) pass \`"{name}@{version}"\`. - \`pacquet-testing-utils::add_mocked_registry\` now also writes \`pnpm-workspace.yaml\` with \`storeDir\` / \`cacheDir\` so pnpm 11 (which moved that config out of \`.npmrc\`) respects the test's store override. - \`crates/cli/tests/_utils.rs::index_file_contents\` now walks \`StoreIndex::keys()\` and returns \`BTreeMap<sqlite_key, BTreeMap<filename, CafsFileInfo>>\`. The insta snapshots will need re-accept once the test run against the new mock layout settles. - Dropped the \`#[ignore]\` on \`pnpm_compatibility::{same_file_structure, same_index_file_contents}\` — the v11 cutover is the prerequisite they were waiting on. The install-path snapshots in \`install.rs\` will likely need a re-accept through \`cargo insta test --accept\`; running them requires \`just registry-mock launch\` and is tracked at the top of \`crates/cli/tests/install.rs\`. Dependencies added at the workspace level: \`rusqlite = "0.38"\` (with the \`bundled\` feature so we don't need a system libsqlite3) and \`rmp-serde = "1.3\"\`. Dropped the unused \`base64\` dep from \`tarball\`. Verified end-to-end: \`pnpm install --lockfile-only\` against \`@pnpm/registry-mock\` followed by \`pacquet install --frozen-lockfile\` produces \`store-dir/v11/files/XX/…\` + \`store-dir/v11/index.db\` with 50 rows whose keys match pnpm's \`{integrity}\\t{pkgId}\` format (verified via sqlite3 CLI against the pacquet-written db). Follow-ups (tracked in #244 and this PR's description): - Teach pacquet's config loader to read \`pnpm-workspace.yaml\` in addition to \`.npmrc\` so \`storeDir\` / \`cacheDir\` / \`registry\` overrides resolve without requiring the .npmrc mirror. - Read the index at install-time to skip re-fetching already-cached tarballs. - Decode msgpackr \`useRecords\` entries so pacquet can read back records that pnpm itself wrote (pnpm reads pacquet's named-field writes fine today; the reverse direction is the missing piece). * chore(deny): allow Zlib license for foldhash (transitive via rusqlite) rusqlite pulls in foldhash 0.2, which is Zlib-licensed. Zlib is OSI-approved and widely accepted; add it to the allow list. * test: re-accept v11 store snapshots + scope pnpm_compatibility tests Aftershocks from the v11 store migration: - Three insta snapshots in crates/cli/tests/snapshots/install__*.snap referenced the old \`v3/files/.../*-index.json\` paths and had to be re-accepted via \`cargo insta test --accept\` to reflect the new \`v11/files/XX/<hex>[-exec]\` + \`v11/index.db\` layout. Inner file integrities are unchanged; only the store-dir path prefix differs. - \`pnpm_compatibility::same_file_structure\` now filters out three pnpm-11-specific artifacts that pacquet has no equivalent for, but which don't affect the shared-CAFS invariant the test is there to verify: * \`v11/projects/<hash>\` — pnpm-11 per-project metadata. * \`v11/links/<name>/<version>/<hash>/node_modules/...\` — pnpm 11's hoisted-symlinks layout; pacquet uses its own \`.pnpm/\`-based virtual store. * \`v11/index.db-{wal,shm}\` — SQLite WAL sidecars whose presence at compare-time depends on whether the checkpoint ran. With that filter the two tools' CAFS file listings match byte-for- byte. - \`pnpm_compatibility::same_index_file_contents\` goes back under \`#[ignore]\` with an updated reason. Pacquet-written \`index.db\` values round-trip fine (we serialize via \`rmp-serde::to_vec_named\`, string-keyed maps), but pnpm writes with msgpackr's \`useRecords: true\` extension-typed records that rmp-serde can't decode. Handling those is the last piece of #244 (msgpackr-record decoding for the read side) and will be addressed after this PR lands. * review: address Copilot feedback on the v11 store migration - \`TarballError::WriteStoreIndex\` display message was still "Failed to write tarball index" (a leftover from the per-package JSON era). Rephrased to "Failed to write store index (SQLite index)". - \`entry.header().size().unwrap_or(0)\` silently turned tar header parse errors into 0-byte entries in the index. Propagate the error via \`TarballError::ReadTarballEntries\` instead. - Open + PRAGMA + \`INSERT\` on the SQLite index are blocking (and can stall up to \`busy_timeout=5000\` ms under contention). Moved the write step into \`tokio::task::spawn_blocking\` so it runs on the blocking pool instead of the tokio reactor. The tarball-extraction loop (which holds a \`!Send\` \`tar::Archive\`) is now scoped into a sub-block so it's dropped before the new \`await\`. - Added \`StoreIndex::open_readonly\` / \`open_readonly_in\` that use \`SQLITE_OPEN_READ_ONLY\` and skip the schema-creation PRAGMAs. \`crates/cli/tests/_utils.rs::index_file_contents\` now uses it so the snapshot helper doesn't mutate the store or create WAL / SHM sidecar files during what is supposed to be a read. - Clarified the \`StoreIndex\` module-level doc comment: the DB lives at \`<store-root>/v11/index.db\` when opened via \`open_in\`, not at \`<store_dir>/index.db\` as the old wording suggested.
30 lines
1.2 KiB
Rust
30 lines
1.2 KiB
Rust
use pacquet_store_dir::{CafsFileInfo, StoreDir, StoreIndex};
|
|
use std::{collections::BTreeMap, path::Path};
|
|
|
|
/// Snapshot-friendly view of every row in `<store>/v11/index.db`.
|
|
///
|
|
/// The outer key is the SQLite key (`"{integrity}\t{pkgId}"`). The inner
|
|
/// map is the package's files — one entry per path inside the tarball.
|
|
/// `checked_at` is scrubbed because its value depends on install time.
|
|
pub fn index_file_contents(store_dir: &Path) -> BTreeMap<String, BTreeMap<String, CafsFileInfo>> {
|
|
let store = StoreDir::new(store_dir);
|
|
// open_readonly: we're just reading for snapshot assertions, so don't
|
|
// create WAL sidecars or otherwise mutate the store.
|
|
let index = StoreIndex::open_readonly_in(&store).expect("open v11 index.db");
|
|
|
|
let mut out = BTreeMap::new();
|
|
for key in index.keys().expect("list index keys") {
|
|
let row = index.get(&key).expect("read index row").expect("row disappeared");
|
|
let files = row
|
|
.files
|
|
.into_iter()
|
|
.map(|(filename, mut info)| {
|
|
info.checked_at = None;
|
|
(filename, info)
|
|
})
|
|
.collect();
|
|
out.insert(key, files);
|
|
}
|
|
out
|
|
}
|