Files
pnpm/pacquet/crates/cli/tests/_utils.rs
Zoltan Kochan 0b0abf8189 feat!(store): migrate from v3 JSON index to v11 SQLite+msgpack (#244) (#247)
* feat!(store): migrate from v3 JSON index to v11 SQLite+msgpack layout

Pacquet now writes the same on-disk store that pnpm v11 uses, so running
pacquet and pnpm on the same machine can share the tarball cache instead
of maintaining parallel stores. Closes half of #244.

- **Layout:** \`<store>/v11/files/XX/<hex-remainder>[-exec]\` for
  content-addressed blobs + \`<store>/v11/index.db\` for the per-package
  file index. The legacy \`v3/files/XX/<hex>-index.json\` JSON layout is
  gone (we don't need backward compat — pacquet is pre-1.0).
- **SQLite schema / PRAGMAs:** byte-identical to pnpm's
  \`store/index/src/index.ts\` — \`package_index (key TEXT PRIMARY KEY,
  data BLOB NOT NULL) WITHOUT ROWID\` with WAL journal, 5s busy_timeout,
  512 MB mmap, 32 MB page cache, temp_store=MEMORY,
  wal_autocheckpoint=10000. Two concurrent pacquet tasks (or a pacquet
  + pnpm pair) contend correctly via SQLite's own locking.
- **Value encoding:** msgpack via \`rmp-serde::to_vec_named\` so structs
  are serialized with named fields, matching pnpm's msgpackr record
  encoding. \`PackageFilesIndex\` and \`CafsFileInfo\` types mirror pnpm's
  \`@pnpm/store.cafs\` shape field-for-field (including the hex \`digest\`
  rather than pacquet's old SSRI \`integrity\` string).
- **Key format:** \`"{integrity}\\t{pkg_id}"\` via \`store_index_key()\`,
  same as pnpm's \`storeIndexKey\`. \`pkg_id\` is \`"{name}@{version}"\` —
  pulled from \`PkgNameVerPeer::without_peer()\` on the frozen-lockfile
  path and assembled from \`PackageVersion\` on the registry path.

Breaking API changes inside the workspace:
- \`pacquet-store-dir\` loses the v3 JSON types (\`PackageFilesIndex\`,
  \`PackageFileInfo\`, \`WriteIndexFileError\`) and \`StoreDir::v3()\` /
  \`::write_index_file()\`. In their place: \`StoreIndex\`, \`StoreIndexError\`,
  the pnpm-shape \`PackageFilesIndex\` + \`CafsFileInfo\`, and
  \`store_index_key()\`. Unit-tested with 9 new round-trip / re-open /
  key-format tests.
- \`DownloadTarballToStore\` gains a \`package_id\` field. All three
  callers (two in \`pacquet-package-manager\`, one in the micro-benchmark)
  pass \`"{name}@{version}"\`.
- \`pacquet-testing-utils::add_mocked_registry\` now also writes
  \`pnpm-workspace.yaml\` with \`storeDir\` / \`cacheDir\` so pnpm 11 (which
  moved that config out of \`.npmrc\`) respects the test's store
  override.
- \`crates/cli/tests/_utils.rs::index_file_contents\` now walks
  \`StoreIndex::keys()\` and returns \`BTreeMap<sqlite_key,
  BTreeMap<filename, CafsFileInfo>>\`. The insta snapshots will need
  re-accept once the test run against the new mock layout settles.
- Dropped the \`#[ignore]\` on
  \`pnpm_compatibility::{same_file_structure, same_index_file_contents}\`
  — the v11 cutover is the prerequisite they were waiting on. The
  install-path snapshots in \`install.rs\` will likely need a re-accept
  through \`cargo insta test --accept\`; running them requires
  \`just registry-mock launch\` and is tracked at the top of
  \`crates/cli/tests/install.rs\`.

Dependencies added at the workspace level: \`rusqlite = "0.38"\` (with
the \`bundled\` feature so we don't need a system libsqlite3) and
\`rmp-serde = "1.3\"\`. Dropped the unused \`base64\` dep from \`tarball\`.

Verified end-to-end: \`pnpm install --lockfile-only\` against
\`@pnpm/registry-mock\` followed by \`pacquet install --frozen-lockfile\`
produces \`store-dir/v11/files/XX/…\` + \`store-dir/v11/index.db\` with
50 rows whose keys match pnpm's \`{integrity}\\t{pkgId}\` format
(verified via sqlite3 CLI against the pacquet-written db).

Follow-ups (tracked in #244 and this PR's description):
- Teach pacquet's config loader to read \`pnpm-workspace.yaml\` in
  addition to \`.npmrc\` so \`storeDir\` / \`cacheDir\` / \`registry\`
  overrides resolve without requiring the .npmrc mirror.
- Read the index at install-time to skip re-fetching already-cached
  tarballs.
- Decode msgpackr \`useRecords\` entries so pacquet can read back
  records that pnpm itself wrote (pnpm reads pacquet's named-field
  writes fine today; the reverse direction is the missing piece).

* chore(deny): allow Zlib license for foldhash (transitive via rusqlite)

rusqlite pulls in foldhash 0.2, which is Zlib-licensed. Zlib is
OSI-approved and widely accepted; add it to the allow list.

* test: re-accept v11 store snapshots + scope pnpm_compatibility tests

Aftershocks from the v11 store migration:

- Three insta snapshots in crates/cli/tests/snapshots/install__*.snap
  referenced the old \`v3/files/.../*-index.json\` paths and had to be
  re-accepted via \`cargo insta test --accept\` to reflect the new
  \`v11/files/XX/<hex>[-exec]\` + \`v11/index.db\` layout. Inner
  file integrities are unchanged; only the store-dir path prefix
  differs.

- \`pnpm_compatibility::same_file_structure\` now filters out three
  pnpm-11-specific artifacts that pacquet has no equivalent for, but
  which don't affect the shared-CAFS invariant the test is there to
  verify:
    * \`v11/projects/<hash>\` — pnpm-11 per-project metadata.
    * \`v11/links/<name>/<version>/<hash>/node_modules/...\` — pnpm
      11's hoisted-symlinks layout; pacquet uses its own
      \`.pnpm/\`-based virtual store.
    * \`v11/index.db-{wal,shm}\` — SQLite WAL sidecars whose presence
      at compare-time depends on whether the checkpoint ran.
  With that filter the two tools' CAFS file listings match byte-for-
  byte.

- \`pnpm_compatibility::same_index_file_contents\` goes back under
  \`#[ignore]\` with an updated reason. Pacquet-written \`index.db\`
  values round-trip fine (we serialize via \`rmp-serde::to_vec_named\`,
  string-keyed maps), but pnpm writes with msgpackr's \`useRecords:
  true\` extension-typed records that rmp-serde can't decode. Handling
  those is the last piece of #244 (msgpackr-record decoding for the
  read side) and will be addressed after this PR lands.

* review: address Copilot feedback on the v11 store migration

- \`TarballError::WriteStoreIndex\` display message was still "Failed to
  write tarball index" (a leftover from the per-package JSON era).
  Rephrased to "Failed to write store index (SQLite index)".
- \`entry.header().size().unwrap_or(0)\` silently turned tar header
  parse errors into 0-byte entries in the index. Propagate the error
  via \`TarballError::ReadTarballEntries\` instead.
- Open + PRAGMA + \`INSERT\` on the SQLite index are blocking (and can
  stall up to \`busy_timeout=5000\` ms under contention). Moved the
  write step into \`tokio::task::spawn_blocking\` so it runs on the
  blocking pool instead of the tokio reactor. The tarball-extraction
  loop (which holds a \`!Send\` \`tar::Archive\`) is now scoped into a
  sub-block so it's dropped before the new \`await\`.
- Added \`StoreIndex::open_readonly\` / \`open_readonly_in\` that use
  \`SQLITE_OPEN_READ_ONLY\` and skip the schema-creation PRAGMAs.
  \`crates/cli/tests/_utils.rs::index_file_contents\` now uses it so
  the snapshot helper doesn't mutate the store or create WAL / SHM
  sidecar files during what is supposed to be a read.
- Clarified the \`StoreIndex\` module-level doc comment: the DB lives
  at \`<store-root>/v11/index.db\` when opened via \`open_in\`, not at
  \`<store_dir>/index.db\` as the old wording suggested.
2026-04-23 19:06:14 +02:00

30 lines
1.2 KiB
Rust

use pacquet_store_dir::{CafsFileInfo, StoreDir, StoreIndex};
use std::{collections::BTreeMap, path::Path};
/// Snapshot-friendly view of every row in `<store>/v11/index.db`.
///
/// The outer key is the SQLite key (`"{integrity}\t{pkgId}"`). The inner
/// map is the package's files — one entry per path inside the tarball.
/// `checked_at` is scrubbed because its value depends on install time.
pub fn index_file_contents(store_dir: &Path) -> BTreeMap<String, BTreeMap<String, CafsFileInfo>> {
let store = StoreDir::new(store_dir);
// open_readonly: we're just reading for snapshot assertions, so don't
// create WAL sidecars or otherwise mutate the store.
let index = StoreIndex::open_readonly_in(&store).expect("open v11 index.db");
let mut out = BTreeMap::new();
for key in index.keys().expect("list index keys") {
let row = index.get(&key).expect("read index row").expect("row disappeared");
let files = row
.files
.into_iter()
.map(|(filename, mut info)| {
info.checked_at = None;
(filename, info)
})
.collect();
out.insert(key, files);
}
out
}