# Source runtime acceptance matrix

The R8 table in
[the source runtime reliability plan](../../plans/2026-09-15-source-runtime-reliability.md)
lists 28 scenarios. This file maps each one to the automated test that proves
it, states whether that test passes on the CI runner (Blacksmith Ubuntu
24.04, `core_tests.yml`), and records the observed behavior where it does not.
Nothing here depends on a personal machine or a NAS.

Status values:

- `passing`: the named test runs in CI and passes at the commit that landed
  this file.
- `failing`: the named test exists, is marked
  `#[ignore = "R8: <row> fails: <reason>"]`, and fails when run with
  `--ignored`. Each one is a fix brief.
- `fixed`: the row failed at the commit that landed this file and a later
  commit made its test pass; the row's section below records the root cause
  and the fix.
- `not automatable`: no Linux test can prove the row; the reason says why.

How the suites run:

- Colocated tests (`core/src/...`) run in the `--lib` suite of
  `cargo xtask test-core --unit`.
- The other three run as `cargo xtask test-core --acceptance`, the
  `acceptance` job of `core_tests.yml`:
  `crates/store/tests` ("Store crate tests");
  `core/tests/source_runtime_acceptance_test.rs`, one `Core` over temporary
  directories (the single-daemon rows);
  `core/tests/source_replication_test.rs`, two daemons in separate processes
  paired over loopback (the peer rows).

Run an ignored row by name, for example:

```sh
cargo test -p sd-core --lib repeated_clear_and_refill_keeps_allocation_bounded -- --ignored
```

## Matrix

| # | Scenario | Test | Status |
|---|---|---|---|
| 1 | Child before parent across batches | `core/src/ops/indexing/store.rs` `a_child_arriving_before_its_parent_batch_gets_real_ancestry`, `a_batch_builds_the_tree_whatever_order_it_arrives_in`; `crates/store/tests/files.rs` `a_child_ahead_of_its_parent_still_commits` | passing |
| 2 | Shallow browse inside an empty retained source | `core/src/ops/indexing/store.rs` `a_shallow_browse_in_an_empty_store_persists_its_ancestry` | passing |
| 3 | One failed row and successful siblings | `core/src/ops/indexing/store.rs` `one_failed_row_keeps_its_siblings_and_fails_the_flush` (siblings retained, flush refuses and counts the loss); `a_repeat_observation_repairs_a_failed_write` (the R1 proof that a repeat observation repairs the row) | passing / fixed (see F1) |
| 4 | Writer dies or pending query fails | `core/src/ops/indexing/store.rs` `a_dead_writer_is_a_typed_failure_not_an_empty_queue`, `a_failed_commit_surfaces_at_the_flush_barrier` | passing |
| 5 | Restart between identity assignment and commit | `core/src/ops/indexing/store.rs` `a_restart_after_assignment_keeps_the_identity_and_lands_the_write` | passing |
| 6 | Interrupted or partially denied scan | `crates/store/tests/files.rs` `an_interrupted_sweep_does_not_delete_the_source`, `a_subtree_the_walk_could_not_read_survives_the_sweep`, `a_sweep_that_saw_almost_nothing_is_refused`; `core/src/ops/indexing/store.rs` `a_locked_folder_costs_its_subtree_not_the_walk`; `core/src/ops/indexing/job.rs` `only_a_whole_unfiltered_walk_may_sweep`, `a_browse_may_not_sweep`, `rules_bar_a_sweep`, `a_depth_limit_bars_a_sweep` | passing |
| 7 | Old schema with origin offline | `crates/store/tests/files.rs` `an_index_that_predates_parent_addressing_is_refused_intact` (the compatibility-status branch: the old table and its assertions stay intact; no data-preserving migration exists yet) | passing |
| 8 | Same volume, nested roots, reversed registration order | `core/src/ops/indexing/sources.rs` `a_nested_source_never_redefines_its_volume_root`, `reversed_registration_order_keeps_one_volume_boundary`; `core/src/ops/indexing/volume_index.rs` `a_nested_source_shares_the_drive_it_sits_on`, `nested_sources_registered_inner_first_share_one_map_and_identity` | passing |
| 9 | Remount and APFS alias changes | Remount: `core/src/ops/indexing/sources.rs` `a_remount_keeps_the_source`, `folders_on_one_drive_are_distinct_sources`, `a_different_drive_at_the_same_mount_point_is_a_different_source`; `core/src/ops/indexing/volume_index.rs` `a_root_mismatched_snapshot_moves_aside_instead_of_deleting`. APFS alias: none | passing (remount) / not automatable (APFS: firmlink spellings such as `/System/Volumes/Data/Users/x` only exist on macOS; `VolumeManager::locate_path` resolves them through the live volume list, which a Linux runner cannot produce) |
| 10 | Missing or invalid restart snapshot | `core/tests/snapshot_rebuild_acceptance_test.rs` `a_missing_snapshot_rebuilds_the_map_from_the_store`, `a_corrupt_snapshot_is_quarantined_and_the_map_rebuilt_from_the_store` (real restart); `core/src/ops/indexing/volume_index.rs` `an_invalid_snapshot_leaves_the_source_visible_and_rebuilt_from_its_store`, `an_older_format_snapshot_is_quarantined_and_the_map_rebuilt_from_the_store`, `an_invalid_snapshot_is_retained_for_diagnosis`, `test_snapshot_roundtrip_and_detached_restore` | passing / fixed (see F2) |
| 11 | Failed watcher subscription | `core/tests/source_runtime_acceptance_test.rs` `a_refused_watch_is_not_reported_active` (refusal reported with its reason on `core.index_status`, then a retry flips the root to active) | fixed (see F3) |
| 12 | Offline client restart | `core/src/service/mounts/peer.rs` `a_cold_restore_rebuilds_the_inventory_without_the_owner`, `published_facts_outlive_the_owners_connection`, `a_manifest_without_facts_still_restores` | passing |
| 13 | One failing source among nine | `core/src/service/mounts/peer.rs` `one_failing_source_among_nine_stays_listed_as_unavailable` | passing |
| 14 | Listing/fetch generation race | `core/src/service/mounts/peer.rs` `the_generation_recorded_names_the_delivered_bytes`, `a_corrupt_delivery_never_replaces_a_good_artifact` | passing |
| 15 | Unchanged owner for ten intervals | `core/tests/source_replication_test.rs` `test_source_replication` (Bob's ten passes after convergence: same generation, same sync time, same loaded arena); `core/src/service/mounts/peer.rs` `dirtiness_alone_paces_while_a_moved_generation_transfers`, `an_unchanged_store_exports_identical_bytes`; `crates/store/tests/revision.rs` `reopening_an_unchanged_store_rewrites_no_triggers` | passing |
| 16 | Continuous real writes | `core/tests/source_replication_test.rs` `test_source_replication` (Alice writes and re-walks; Bob lands a newer generation holding every record); `crates/store/tests/revision.rs` `file_changes_move_the_revision_and_nothing_else_does` | passing |
| 17 | Repeated subtree clear and refill | `core/src/ops/indexing/arena.rs` `repeated_clear_and_refill_keeps_allocation_bounded` (slots bounded by the live tree across 100 cycles, snapshot carries live nodes), `a_shrunken_tree_compacts_once_vacancy_outgrows_it` (the compaction trigger), `a_snapshot_after_churn_carries_only_live_nodes` (round-trip after churn), `a_listing_cursor_taken_before_compaction_still_pages` (path cursors survive renumbering); `core/src/ops/indexing/nodes.rs` `a_vacated_slot_is_reused_and_unreadable_until_then` | fixed (see F4) |
| 18 | Cold search across 100 stores | `crates/store/tests/scale.rs` `a_hundred_cold_stores_answer_without_writers` (CI size: 100 stores x 200 records); `a_cold_fan_out_across_many_stores` (the measurement, ignored, sized by `SD_SCALE_*`) | passing |
| 19 | Same capture read through arena and SQLite | `core/src/ops/search/arena_search.rs` `the_store_backend_matches_the_arena_for_the_same_capture` (identities, matching, filters, scores, and the capture time and place both backends read from `facet_image`); `core/src/ops/search/pipeline.rs` `every_sort_field_orders_and_reverses`, `a_capture_order_keeps_undated_files_last_in_both_directions`, `equal_scores_tiebreak_deterministically`, `a_page_is_a_window_over_the_sorted_whole` (ordering and pagination shared by both backends) | passing |
| 20 | Suitable loaded arena returns no matches | `core/tests/source_runtime_acceptance_test.rs` `an_empty_answer_from_a_suitable_arena_is_final` | passing |
| 21 | Loaded arena has insufficient coverage or query support | `core/tests/source_runtime_acceptance_test.rs` `an_unwalked_source_answers_from_its_store` | passing |
| 22 | Five suitable arenas and 95 stores | `core/tests/source_runtime_acceptance_test.rs` `five_arenas_and_ninety_five_stores_page_the_same` | passing |
| 23 | Arena candidates need store-only filter or sort fields | `core/tests/source_runtime_acceptance_test.rs` `a_store_only_filter_narrows_before_pagination` (tags are the store-only field); row 29 for the image facet's capture time and place as sort and filter fields | passing |
| 24 | Search with requested limit five | `core/tests/source_runtime_acceptance_test.rs` `a_limit_of_five_returns_five_with_an_honest_total` | passing |
| 25 | Same record in multiple representations | `core/tests/source_runtime_acceptance_test.rs` `a_file_under_nested_sources_is_one_hit_from_the_arena` (loaded); `a_file_under_nested_sources_is_one_hit_from_the_stores` (store-backed) | passing / fixed (see F5) |
| 26 | Replica replacement with local assertions | none | not automatable today: replicas open read-only and carry no receiver-owned assertions (R6 results, "receiver-owned assertions on replica databases" is a registered follow-on gated on FD2), so there is no local assertion to preserve. `crates/store/tests/files.rs` `a_moved_file_carries_its_assertions_with_it` and `a_removal_takes_the_facet_and_leaves_the_assertion` cover the owner-side contract the replica path will have to reuse |
| 27 | Mapped volume with no source | `core/src/ops/indexing/volume_index.rs` `a_tracked_drive_maps_without_appearing_as_a_source` (browse and snapshot), `a_mapped_drive_without_a_source_is_watchable` (watcher); `core/src/ops/indexing/store.rs` `a_partition_with_no_store_still_browses` | passing |
| 28 | Status query and resource event | `core/tests/source_runtime_acceptance_test.rs` `listing_status_and_store_agree_on_a_sources_count` (`sources.list`, `core.index_status` and the store report one count and one observation time for one source) | passing |
| 29 | Capture time and place through arena and SQLite | `core/tests/source_runtime_acceptance_test.rs` `capture_facts_read_the_same_through_arena_and_store` (the EXIF fixture tracked and enriched; a `CapturedAt` order, a `CapturedAt` range and a `has_location` filter give one order, one total and the same capture facts per hit from the walked arena and, after the arenas drop, from the store; `crates/store/tests/read.rs` `an_entry_carries_the_image_facet_keyed_by_its_current_bytes` for the hash-guarded join underneath) | passing |

Totals: 27 rows fully passing (21 at the commit that landed this file, plus
rows 3, 10, 11, 17 and 25 fixed since, and row 29 added for the image
facet); row 9 passing except its APFS half, which no Linux runner can
produce; row 26 not automatable because the feature it names does not exist
yet. No ignored test remains.

Row 29 is not in the plan's table. It extends rows 19 and 23 to a facet:
the arena holds no capture time or place, so the arena path reads them from
the store before it sorts and filters (`core/src/ops/search/capture.rs`)
and the store path joins `facet_image` into every entry read, guarded by
the row's content hash. Neither backend needs the other to answer, and
both hand the same `File` to one predicate and one order.

## Failing and fixed rows

F1 to F5 record what failed at the commit that landed this file and what
fixed it.

### F1. Row 3: a repeat observation did not repair a failed write (fixed)

`core/src/ops/indexing/store.rs` `a_repeat_observation_repairs_a_failed_write`.
After a batch lost one row to a database failure, the ledger kept the
binding it made before the commit. The next walk observed the same file
unchanged, the ledger answered `Unchanged`, and nothing was written: the row
was missing for as long as the file's size and mtime held. Observed: the
store held `["docs"]` where `["docs", "docs/bad.txt"]` was expected.

Fix: `commit` (store.rs) now takes the ledger and calls
`Ledger::unsettle(uuid)` for every row the salvage could not land.
`unsettle` keeps the uuid at its path but clears the size and mtime
evidence, so the next observation resolves `Changed` and rewrites the whole
row under the identity the arena already holds. Dropping the binding
instead would have minted a second identity for the same file.

### F2. Row 10: an unreadable snapshot was deleted on load (fixed)

`core/src/ops/indexing/volume_index.rs` `an_invalid_snapshot_is_retained_for_diagnosis`.
`snapshot.rs` removed the file when it could not decode or parse it, so the
artifact was gone before anyone could inspect it. The source stayed visible
and its store still answered (the passing half of the row); only retention
failed.

Fix: `load_snapshot_impl` moves an artifact it cannot read, decode, or
accept the version of to `<name>.corrupt-<unix seconds>` beside the slot
and logs a WARN naming both paths. The slot is clear for the next save and
no launch parses the artifact again; the test asserts the bytes survive at
the aside path. One retained copy is enough: when a `.corrupt-*` sibling
already exists, a later unreadable artifact in the slot is removed instead,
so a recurring failure cannot fill the disk.

Follow-up: a partition whose snapshot is missing, quarantined, or from an
older format no longer costs a walk. `VolumeIndex::ensure_restored` starts
a background fill of every attached source on the drive from its store, the
same fill a library restore uses, and resolves at once so no reader waits
on it; each root is in progress until its fill lands, a listing serves the
store meanwhile, and the landed source announces its root, which routes it
to the arena and re-arms its watch. The coverage heal finds the source in
progress or covered and dispatches nothing. The rebuild
runs for a source whose registry row carries a record count, which a
snapshot save writes, so it is the evidence a map existed and was lost; a
source never walked to completion keeps answering from its store without
an arena (rows 4 and 5 of this matrix), and a source with no store records
is left for the heal to walk. A store rebuild covers
the registered sources and not the rest of the drive, so the discovery pass
(`map_attached_volumes` with defaults) still maps the drive around them in
the background, as it does for any drive without a snapshot. The real
restart is `core/tests/snapshot_rebuild_acceptance_test.rs`: the listing is
served from the store while the fill runs and from the arena once it has
landed, no job is dispatched, and `core.index_status` then reports the
source restored with nothing in progress. The 1M-record rebuild time is
recorded in `docs/core/design/ephemeral-snapshot-format.md`.

### F3. Row 11: a refused watch was reported active (fixed)

`core/tests/source_runtime_acceptance_test.rs` `a_refused_watch_is_not_reported_active`.
`FsWatcherService::watch_root` called `VolumeIndex::register_for_watching`
before `watcher.watch_path`. When the OS refused the subscription (the
directory was removed), the error propagated but the volume index kept the
root in `watched_paths`, so `is_watched` and `core.index_status` reported
an active watch that did not exist.

Fix: `watch_root` subscribes at the OS first and registers only after the
OS accepts; a refusal is recorded on the volume index with its reason
(`VolumeIndex::record_watch_refusal`) and surfaces as
`IndexStatus::refused_watches`, which `sd index status` prints beside the
watched roots. The service retries every refused root on
`FsWatcherServiceConfig::watch_retry_interval` (30 s) while it runs
(`retry_refused_watches`), and a retry the OS accepts moves the root from
the refused list to `watched_paths`. The test covers both halves: the
refusal with its reason, then the directory returning and one retry arming
the watch.

### F4. Row 17: arena allocation grows with history (fixed)

`core/src/ops/indexing/arena.rs` `repeated_clear_and_refill_keeps_allocation_bounded`.
One hundred clear-and-refill cycles over a 50-file branch left 54 live
paths and 5,154 allocated slots. `NodeArena::vacate` kept the slot, nothing
compacted, and the snapshot serialized every slot.

Fix (R4): `NodeArena` keeps a free list. `vacate` drops the node's heap,
marks the slot unoccupied (so a lookup through the old id answers nothing)
and queues it; the next `insert` takes a queued slot before growing. A
cleared and refilled branch therefore reuses exactly the slots it had: the
same 100 cycles end at 54 slots for 54 live paths. When a tree shrinks for
real the free list grows instead, and `Arena::compact_if_inflated`, called
after every removal, rebuilds the arena densely once vacant slots outnumber
live entries and number at least `COMPACT_MIN_VACANT` (4,096). The first
condition bounds allocation at twice the live tree and makes the pass
amortized (one walk of the live graph, paid for by at least as many
removals); the floor keeps a small arena from rebuilding over a few hundred
bytes of slack. `compact` renumbers parent links, child lists, both path
maps, uuid bindings, content kinds, collection flags, summary stubs and the
name registry together, and builds the new arena in full before swapping it
in, so a failed mapping leaves the old one answering.

Slot reuse and renumbering are safe because `EntryId` never leaves the arena
module: the public API speaks paths and uuids, the explorer's media cursor
is a source, a directory and a name, library search pages by offset over
sorted paths, and the watcher addresses entries by path. Record uuids,
rollups and query results are unchanged by either; a cursor taken before a
compaction names the same place after it, which
`a_listing_cursor_taken_before_compaction_still_pages` proves at the arena.

The restart snapshot (format version 4) is written through the same dense
projection whether or not the session's arena compacted, so it carries one
node per live entry and the reader restores a dense arena. A version 3
artifact is quarantined on load like any other mismatch and the source
reindexes once.

### F5. Row 25: store-backed search returned one hit per nested store (fixed)

`core/tests/source_runtime_acceptance_test.rs` `a_file_under_nested_sources_is_one_hit_from_the_stores`.
With an outer and an inner source both walked, the file under the inner root
is committed to both stores. After a restart with no snapshot, every source
answered from its store and `search_every_index` queried each store in turn,
so the file surfaced twice (`total_found` 2 where 1 was expected). The
loaded case was one hit because both registrations share one arena.

Fix: the store loop in `search_every_index` (arena_search.rs) reads sources
innermost first and keeps a set of local paths already answered, seeded
with every arena's full match set (collected before the page window
narrows it), so a file an inner store (its owner) returned is dropped from
the outer store's partition before it is counted. The total stays exact
because the de-duplication runs on each store's full match set ahead of the
page window.

## Not in the plan's table but covered on the way

- Unchanged refresh: `an_unchanged_generation_still_refreshes_the_owner_facts` (peer.rs).
- Resumable replica transfer: `an_interrupted_transfer_resumes_from_the_part_length`,
  `a_part_from_another_generation_starts_over` (peer.rs).
- Database replica restore: `a_database_artifact_restores_into_a_browsable_replica` (peer.rs).