The source runtime reliability plan graduates from draft to the
executing plan its register row already tracks. Library-sync research
records the split-library state between the Mac and titan and proposes
the small library-state contract; locations research records the
revised direction of configured source subtrees with continuous
processing rules over pins alone, cross-referenced from the
entries-teardown T4.4 row. The working register carries today's landed
work: search phases, fleet statistics, storage replicas, the hashing
loop fix, and the first reliability slices, with the code baseline
brought current.
From a parallel session. Inset content is the default: navigation sits
on the window background and the explorer renders as an inset sheet.
Settings > Appearance > Layout retains the floating-panels mode. Both
modes share mounted content and stable preview hosts through a shell
geometry context, the layout preference propagates to other windows on
the same origin, and the sidebar footer returns to circular Activity
and Settings controls with the activity count and busy indicator.
Preview and Size view follow the sheet bounds, with Size thumbnails in
container coordinates.
From a parallel session. thumbs.generate accepts a file or an indexed
folder with missing, stale, and force modes, runs in resumable batches,
and versions video tiles against host FFmpeg availability so a tile
regenerated with better tooling replaces its predecessor in place. File
and folder context menus and the new sd thumbs CLI domain call the same
operation. paths.context now reports whether the exact path has a
committed source record, and the path bar surfaces it with a refresh
affordance. Coverage and resume semantics are documented in
docs/core/thumbnail-generation.mdx.
Phase 1 of docs/plans/2026-09-15-search-refinement.md. Every parameter
search.files accepts is now either honored or fails closed; nothing is
validated, logged, and ignored anymore.
Ordering and pagination live in a new pipeline module with a
backend-neutral candidate type, per R6 of the source-runtime plan: one
deterministic global sort before final pagination, equal keys settling
on case-folded name and record id so results stop reshuffling between
keystrokes. Library search sums each partition's full filtered match
count into total_found and contributes only the page window of
candidates to the merge, so a partition can never lose a result that
belongs on the page. The 200-result cap and the synthetic pagination
block are gone; the response reports the caller's window over the true
total (a "spacedrive" search now reports 496 matches instead of a
complete-looking 200).
Filter fixes: extensions compare case-insensitively, hidden entries
are excluded by default and included with include_hidden (advertised
in available_filters), and a missing timestamp fails a date filter
instead of passing it, so the unanswerable AccessedAt and IndexedAt
fields fail closed rather than silently matching everything.
Deletes dead code the plan condemned: sorting.rs and the unused
facets.rs duplicates, plus the legacy entry-era imports in query.rs.
Verified live: --limit 5 returns five with the true total, size and
name sorts order and reverse, offset pages advance.
The plan owns everything above matching, which 4bc442e43 made correct:
scopes become folder/source/library with Location retired as vocabulary,
sort and pagination get implemented rather than deleted, the filter
panel ships the set the daemon can answer honestly, tags follow through
the library-db path-set join, and the redundancy views get a phase to
verify and fix their mechanism, since they drive search.files with
filters the ephemeral path ignores. Register rows record the plan and
the landed search fixes.
PROJECT_STATUS.md now sits on the b653676e8 baseline: the sidebar,
path-status, thumbnail, host-media, and startup rows cite their landed
commits with live proofs still open, the finished docs rows moved to
Recent landed, and the multi-device commits since 50fdbad59 are
recorded. The drive-catalog row flags that packages/drives is missing
from the checkout. Adds the entries final-drop plan and its references
in the teardown, convergence, and locations plans.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
README and CONTRIBUTING updates, the working register brought current,
locations and data-model pages reconciled, redundancy and overview
polish, media view tweaks, regenerated schemas, and the new Spacedrive
icon asset.
Library loading no longer dispatches a filesystem discovery pass. The
desktop requests one idempotent pass after the connected shell
renders, a bare daemon initiates nothing, and sd index start
--defaults gives CLI and server operators the same explicit path.
Restored volume snapshots prevent repeat full walks.
A machine-scoped registry discovers host FFmpeg and companion tools,
surfaced in Services settings. The thumbnail service uses them for
video posters and on-demand scrub sheets: thumbstrips bake per request
into the volume hot tier and serve over the local HTTP servers with
immutable content-versioned URLs, wired to the timeline scrubber.
The path bar reads a typed path-context query backed by the daemon's
source, volume map, watcher, persistence, and pin state, replacing the
location/index-mode indicator. Containment resolves on the daemon with
path components after volume alias normalization; a pin no longer
implies indexing or watching, and the docs say so.
One library scope, flat daily destinations, Places computed from
system folders without persisting them as locations, canonical rows,
and an Activity/Settings footer. Topology groups retire; their pins
survive seed convergence. Replica sources render under their owning
device and route into the explorer with that device's slug.
Replaces the Mintlify setup (mint.json, docs.json, snippets) with a
standalone Fumadocs app in docs/: Next.js app router, meta.json nav per
section, its own bun install. Run with `just dev-docs`.
Content pass alongside the migration: core architecture, data model,
CLI, and networking pages rewritten around the daemon/source-store/
library split, new product-direction.mdx defining the product boundary,
new drive-catalog plan, and the physical-drives plan updated to
reference it. Whitepaper page retired.
SourceConfig is the typed shape of the source row's config column,
parsed with defaults so old rows and future fields both survive.
sources.update renames a source or changes its capture policy, with
the one asymmetry named: widening dispatches a walk for what the rules
skipped, narrowing removes nothing because hiding is the lens's job.
Tracking only ever widens the policy, so a casual re-track cannot
demote an archival source. The locations-demoted doc gains the policy
taxonomy this settles: capture on the source, attention on the
location, display in the lens.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Groups carry the vdev tree and minimum member count, since "which boxed
drives make the pool, and how many suffice" is the question ZFS labels
cannot answer with the drives unplugged. zpool membership moves from D3
to D1 because the observation window closes at export. The runbook
gains the TrueNAS config backup, the encryption check, and the clean
export step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
What the store captures (verified against stat on titan), the deploy,
the per-dataset order and pass/fail gate, the pre-export steps, and the
known issues with workarounds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drives under volumes: an append-only observation log, an evidence
ladder for identity (WWN, GPT disk GUID, tiered serials, SMART
trajectory), and drive rows as conclusions over it. Groups for pools
and containers, human adjudication as the strongest evidence tier.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nothing produces a row in `entry` after this. The walk fills the volume
index and its source store, which is what the arena has been doing since
the storage map landed; the persistent half of the indexer was a second
writer to a second store that no source ever read.
Gone: IndexPersistence, and with it the branch in every phase. The state
machine is Discovery then Processing, since Aggregation existed to bubble
directory sizes the arena now maintains incrementally, and
ContentIdentification is ContentIdentityJob's job against the source store.
DatabaseAdapter, PersistentEventHandler and the location watching that fed
them go too, so ArenaWriter is the only change handler and the walk and the
watcher differ only in Notify.
DatabaseStorage was 1230 lines of entry writes wrapped around three things
the arena actually uses. Those are ops::indexing::metadata now: EntryMetadata,
extract_metadata and get_inode, plus is_hidden_path. delete_subtree moves onto
the entry entity, which is what it operates on and what it dies with.
locations.rescan, locations.enable_indexing and locations.triggerJob dispatched
the deleted indexer or its media jobs, so the ops and their UI go. Reindex and
Run Now are gone from the inspector; the job policy toggles stay, since they
outlive the ops that read them. Adding a location no longer dispatches a walk,
which is the point: the drive is already mapped.
The entry tables and their readers are still here. That is the next commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
P4 was titled "Delete entries" and quietly assumed locations went with them.
They don't. A location stops owning records and becomes a policy over a path:
the subtree kept at full fidelity and watched.
That's the Retention.covered parameter the storage map already has and is
currently filling from cache.sources() roots because there was nothing better
to ask. Locations are the better thing to ask, so this gives an existing
parameter its owner rather than adding a mechanism.
The row loses entry_id, device_id, volume_id, scan_state, last_scan_at,
error_message, the two counts, index_mode and job_policies. Each of those
lives somewhere that already has it, and the arena's rollups are the honest
version of the counts since a stub can say 400 GB was summarised and a
persisted integer can't.
Also covers the sweep across the ~64 files that reference entities::entry,
and flags triggerJob, import and export as the three ops that don't survive
mechanically and want deciding on their own merits.
The store's filesystem schema had one model with six fields, so a record
could say how big a file is and not what took the picture. The old library
had image_media_data and video_media_data and both were lost in the move.
Facets are schema-driven, so these are declarations rather than machinery:
generate_ddl emits facet_image, facet_video and facet_audio on create, and
diff_schemas adds them to a store that predates them. Audio came along with
the other two, being the same kind of fact and already shaped in the tree.
No writer yet, enrichment is still parked. What this buys is that the shape
can only be decided once.
The search contract named title = "name" on a model with no name field. A
filesystem record's display name is its filename, which the walk writes to
record.title, so the contract points at the record instead.
The archive engine kept its own registry in registry.db while filesystem sources
lived in library.db, so there were two lists of sources that could not see each
other. sources.list unioned them and said so in a comment.
Now the engine holds stores and adapters and is handed a SourceRef. Which
sources exist is a question for the library, with one answer: the sources table,
read back through ops/sources/registry. Creating a source mints the id and
writes the row here; the engine is told what to name the store. A sync answers
with what the run did and this side records it, so a status can't be written
without the row that shows it.
registry.db and its data_types table are deleted, unread by anything.
Two moves on disk. Adapter stores live under SourceDirs beside every other
source's and are named by uuid without dashes, the way filesystem stores already
were. Installed adapters moved to <data>/adapters, since an adapter is installed
once for a machine rather than once per library. <library>/archive/ is no longer
written.
SourceInfo::from_row replaces the two constructors that built the same shape
from different halves. Existing adapter sources do not carry over: their rows
were in registry.db and their stores are under the old path.
content had 0 rows because nothing ever called the two functions that fill it.
ContentIdentityJob does: it claims files a source holds and has not identified,
hashes them a few at a time, and hands the identities back through the same
writer as everything else, so a content row can never land before the record it
points at.
Dispatched at LOW behind the walk that produced the records and behind the drive
map, which is the order the storage map doc set out: what someone kept, then the
rest of the machine, then the bytes.
files_needing_content rebuilds each path from its parent and its name, the same
join Ledger::load uses. P2.6 means the path is not a column to read, and a file
whose parent has no path is skipped rather than guessed at.
Only the sampled tier runs. An integrity hash is a full read of every byte on
the drive and belongs behind a decision that needs one, which is the rule the
ladder in sd_store::content exists to keep.
set_content_identities commits a batch in one transaction; the single-file
version now shares its statement. A file that cannot be read keeps its empty
content_id and comes back on the next pass, and a pass that identifies nothing
has found the tail of the queue and stops rather than looping on it.
Nothing reads the rows yet: duplicate detection still queries content_identity
in library.db.
Three destinations had to agree about every change — the arena, the source
store, the clients — and each call site wrote all three by hand. They had
already drifted once: a move updated the arena and the store correctly and told
nobody, because creates emitted inline while moves went through a hook that was
handed the path the file had just left.
MemoryAdapter is now ArenaWriter with one entry point. apply(Seen) resolves
identity, mutates the arena, and answers with the ArenaChange that both the
store write and the client event are derived from, so "written durably but never
announced" stops being representable. The walk and the watcher differ only in
Notify: every change, one event per batch, or silence for a drive being mapped
in the background. run_ephemeral_processing_static lost its own copy of all
three writes and is a third of its former size.
Two things fell out of it:
- Mutation::Unchanged. A re-walk announced every entry it re-observed, which
meant serialising a million files nobody touched. The store still hears about
all of them, which is what keeps a sweep from condemning them.
- A recount now reaches clients. It was an arena write with no way to tell
anyone, so a summarised directory whose size changed was correct on the next
query and invisible to anything already on screen.
IndexPersistence had no production callers and goes, with PersistenceFactory and
DatabaseAdapterForJob. The one query the latter still served moved to the change
detector, which was its only caller. ChangeHandler stays for the database
adapter, minus emit_change_event, which the arena can no longer be asked to do
separately.
The Analyzer wants a live sunburst of a volume, and the thing every tool doing
this today cannot do is the one thing that is already free here: the arena
maintains subtree_bytes up the ancestor chain on every add and remove, so a
file landing anywhere corrects every ring outward with no rescan. The map being
live is a property of the arena that nothing surfaces yet.
What is missing is the map. Measured on one 990 GB drive: 11,131,151 entries
exist, 1,678,592 are mapped, 271 GB of 866 GB used. Two independent causes.
Nothing walks the volume at all, so /Applications and /Library have never been
enumerated. Rules prune the rest, which is where ~/Library's 231 GB went.
The reframing that makes it affordable is that traversal is unavoidable and
retention is not: a size cannot be known without enumerating, so the policy is
only ever about what to keep. Measured: 118,859 directories within seven levels
of the volume root, 54 MB, against 5.0 GB for all 11.1M entries. So home at
full fidelity plus a complete seven-ring structure is about what the daemon
holds today for two rings and a fifth of the disk.
Records the stub design, why delta tracking cannot keep one honest, the
dirty-and-recount that can, the queue order that puts a person's own files
first, and the per-volume setting. Phases at the end.
"Medium" named the abstraction after its own exception. Every mount the volume
manager can see gets a record, network shares included, so a filesystem with no
volume is a fallback rather than a co-equal case. The word for a drive here is
volume, and this is its index.
MediumSlot is VolumeIndex, MediumKey is VolumeKey with Id and Path variants
instead of Volume and Root, track_medium is track_volume, and the layout moves
from mediums/<id>/ to volumes/<id>/. The derived namespace uuid is unchanged,
so ids are the same; only the directory holding them is renamed.
Nothing else moves. A volume index is still the whole drive's map, still keyed
by the drive rather than by any registration, still what a nested source shares
instead of forking.
A machine needs full local search over every file and a size figure the
analyser can trust. Neither is the same question as what someone chose to keep,
and the sources migration collapsed the two.
A partition is keyed by source id today and routed by longest prefix, so a
source nested inside another forks rather than narrows: two partitions, two
snapshots, two ledgers, and two uuids for one file. A tag applied through one
root is invisible through the other, which is the second-namespace problem
between two filesystem sources.
P2.7 keys the arena by medium instead. One drive, one arena, one snapshot,
whatever is persisted from it. A source says persist this subtree and owns a
data.db, so nesting is ordinary because the nested things are no longer the
same thing. The arena mints identity once per medium and a source adopts what
is already there, which is both what stops the fork and what makes creating a
source over mapped files fast.
A path under no source still has a uuid and still cannot carry an assertion,
since assertions live in a source store. Choosing to keep something is what
gives it somewhere to keep things.
Also records the completeness cost. An internal volume walks with default
rules, so /Library, ~/Library, node_modules and gitignored files never enter
the map: 262 GB indexed against roughly 900 GB used on a 1 TB drive. Rules
become view-time lenses and the walk records what it can read, which takes a
2.1M entry map to somewhere between 5 and 8M, and 1.0 GB resident to between
2.5 and 4. collection_flags already has room for a filtered bit.
Settles the nesting question in the Open register. Both live documents were
right about their own half: one filesystem owns one record namespace, and that
namespace is the arena rather than the source.
external_id is the full relative path on every record and UNIQUE(type,
external_id) holds a second copy, which is most of a 2.1M-record store. The old
library stored a path once per directory and derived the rest from parent_id
and name. Dropping that was a mistake.
Measured by replaying the live store through the proposed schema, both
databases vacuumed, same 2,123,008 records. Record side 751 MB against 423 MB,
whole database 896 MB against 568 MB, 443 bytes per record against 281. The
index it removes serves path lookup; the one that replaces it answers list this
directory's children, which is the query the UI makes.
Lookup stays two probes because directories keep their paths: one into
directory_path for the parent, one into (parent_uuid, name). Renames stop
rewriting every descendant record and rewrite descendant directories only.
The replay found 0 unresolved parents, so the existing store can be backfilled
in place rather than re-walked. That matters because a re-walk is where the
surviving created_at values would go.
Ties P2.5 (2) to this: the tree is currently a fidelity bug and becomes the
addressing mechanism, so the two land together.
Compared the new per-source store against the last pre-sources library on the
same machine: 2.1M records against 23,157 entries, and the new one holds less
about each file. Three causes, tangled together.
The walk discards metadata it already stats. DirEntry carries five fields and
run_ephemeral_processing_static hardcodes created, accessed and permissions to
None, so birth time is gone for 2.1M files. The stat was already paid for.
The bulk walk does not build the tree. Parent linkage needs the ledger to have
resolved the directory first, which the incremental path satisfies and the walk
does not: 1,349 of 2.1M walked records are parented against 19,769 of 19,856
written by the watcher. 240,122 directories exist and 949 know their parent.
Media facets were never declared. The schema has one model with six fields
where the old library had EXIF, GPS, camera and codec tables.
Sidecars move into the source store. A sidecar is keyed by content uuid and
content now lives per source, so a row in library.db is a foreign key across
database files with nothing enforcing it. sidecar splits: content_sidecar in
the source store, sidecar_availability staying in the library because it is a
claim about a device. Answers the open question in the extensibility plan.
Also recorded under Open: external_id is the full path on every record and the
unique index holds a second copy, so 518 MB of a 1.01 GB database is paths,
while idx_record_parent costs 21 MB and is 99% empty.
JobStarted was emitted for every job while JobCompleted, JobFailed and
JobCancelled were all gated behind should_emit_events. So any job that
opted out of events still announced a start and never announced an end,
and the client had a row nothing could clear. That is every ephemeral
browse, since should_emit_events is should_persist unless the job is
indexing a whole volume.
Gating the start the same way makes an event-free job invisible end to
end, which is what the flag was for.
Also reorders the completion arm. Clients refetch the job list when they
see a completion, and the job was removed from running_jobs after the
event went out, so a refetch that landed first got the job back in the
answer with nothing further coming to clear it. Reading the output,
removing, then announcing closes that window.
Tracking a volume did not index it. There were three flows and none of
them was one gesture: the context menu offered Track and Index as
separate items, and AddStorageModal tracked the volume and then created a
Location, which is the model being deleted.
volumes.track now calls the same track_and_index that sources.track does.
The medium and the index stay separate underneath, since a drive can be
tracked with nothing on it indexed and a folder can be indexed on a
volume nobody tracks; what converges is the click.
The shared body matters more than the saving. Both entry points forget
the same two steps otherwise, seeding from the snapshot and clearing what
the new pass will not revisit, which is exactly what sources.track was
missing when it was written separately.
An external mount indexes unfiltered, since a drive being archived wants
completeness rather than rules that hide files on purpose. An unmounted
drive tracks without walking and gets its map when it returns.
Index Volume becomes Re-index Volume and only shows for tracked drives,
since the first index is no longer something to ask for.
sources.track skipped two steps volumes.index does. Without
ensure_restored a partition that never restored is barred from saving
over an existing snapshot, so tracking a root that already had one would
index and then fail to persist. Without clear_for_reindex, entries from a
previous pass that this one does not revisit linger in the arena as files
that no longer exist.
Also writes down how it differs from volumes.track, since the names
collide on the word rather than the meaning: volumes.track marks a medium
as belonging to a library and indexes nothing, this makes an index of
what is on one. volumes.index is the older overlap and retires once
callers move over, because it takes a fingerprint and can therefore only
ever mean a whole drive.
A filesystem source used to appear as a side effect: browsing a directory
dispatched a walk and the walk registered whatever partition it landed
in. sources.track is that intent made explicit. Whole drive and
folder-within-a-drive are the same call with a different path, since the
volume resolves from the path and the source anchors to it.
Indexing starts immediately rather than waiting to be asked, and the
action returns as soon as the row exists, because registration is instant
and a walk over a drive is not. The job then reports against a source
that already exists.
sources.list serves both kinds now, so the sidebar stopped needing a
group per registry: filesystem rows render as walked roots with their
attachment state, adapter rows as before. The union is transitional,
reading filesystem rows from the sources table and adapter rows from
registry.db, and folding the second into the first is the rest of P3.
Content ids stay out of the walk on purpose. The walk is fast and worth
keeping fast, and records_needing_content_identity is already the query a
separate pass would drive. Nothing generates them yet, which the plan now
says plainly rather than implying the sampled tier is wired.
The arena and the store were minting separate uuids for the same file.
MemoryAdapter generated one inline while Ledger::resolve minted its own
for the record, so every file had two identities and nothing joined them.
Tags hang off the arena's; record_overlay assertions, and the AI
annotations that follow, hang off the record's. Indexing at scale before
fixing this would have baked the split into every row.
The ledger assigns identity now and the arena writes what it is told.
SourceStore::identify resolves a batch and answers with record uuids, and
it is the same message that takes the batch in, so there is no second
call that could disagree. One round trip per batch rather than per file:
resolution is a hash lookup and the reply is sent before anything
commits, so the cost is the queue rather than a transaction.
Volume indexing used to defer uuids entirely and no longer needs to,
since the ledger has one whether or not anything asks. A partition with
no store still mints for the session, because there is nothing durable
for it to agree with.
entry_uuids stays in the snapshot and stops mattering: it now holds the
same uuid the ledger does, so a discard costs a re-identify rather than
an identity.
Native dependencies, job presets, plugins and sidecars are four things,
and separating them is most of the work. Checked against the tree first:
the codecs were already unbundled by the install-size plan, the SDK is
already a plugin SDK missing only a manifest and a lifecycle, and
SidecarKind::Thumb already exists content-addressed. Less is missing than
the dump assumes.
The preset is the primitive worth building, because it is the same shape
as the workflow row the filesystem intelligence plan already needs:
durable, parameterized, executable, logged, with its reasoning attached.
One table, two consumers.
Draws the line that keeps it safe: a model composes registered job kinds,
a plugin adds kinds. If a model can author the command rather than the
parameters, durable execution and monitoring stop being features and
start being a liability.
Answers the sidecar question as placement rather than design. The hot
tier and the thumb sidecar are both already specified and do not compete;
what is open is that the sidecar tree is library-scoped, which is wrong
for an archival drive whose previews should travel with it. And the
location-or-source line is source, since locations are being deleted.
Keeps the GPUI plan where it is: Photos is the right wedge, and a second
full interface should follow the teardown rather than race it.
P3 is partly done: sources are a table, T2.0b is ruled, and the two seams
it left visible are written down rather than left to be rediscovered.
P4 gets the structural answer the move bug pointed at. Each
ChangeHandler method updates arena, store and event bus by hand with
nothing forcing them to agree, and they drifted: a move wrote the first
two and emitted nothing. An arena mutation should produce a change record
that the store write and the client event are both derived from, which
makes that failure unrepresentable rather than fixed.
It also answers what such a record carries, since the same bug showed it:
both the destination and the vacated directory, because a client renders
directory listings and a move invalidates two of them.
Belongs in P4 because MemoryAdapter is being rewritten there anyway and
IndexPersistence dies with the entries world that needed it. Earlier
means building the seam twice.
An annotation is an assertion and assertions outlive the generation by
design, so a model's claim is treated like a person's. Costs no schema
and forecloses nothing, since overlay.updated_at and record.modified_at
already sit beside each other for anything that later wants to surface an
aged claim.
Qwen3.8-27B: Apache 2.0, 262k context, native multimodal, ~16-17GB at
4-bit. Reference rather than requirement, since the loop targets an
OpenAI-compatible endpoint and a tool schema.
Long context moves the loop's step from one directory to a whole branch,
which is a real change to the design. What it does not do is make
annotation cheaper: output tokens bind, not input, so uniform coverage is
still days of generation. The win is that fewer annotations become
sufficient, because seeing a whole subtree is what justifies one
assertion covering it instead of three thousand near-identical ones.
Tool calling drops down the risk list for this model class; the
structured-output path stays as a fallback for weaker providers. Adds the
hardware floor and what happens below it, which is that the map is still
complete and someone's own frontier agent still answers over MCP.
Also notes thumbs.pvcache as a latent second input for a multimodal pass,
scoped out of P5 day one.
Two agents, and only one is the interface. The interface is a person and
their agent in a harness over MCP. Filesystem intelligence is a
Spacedrive-native loop over a local model, headless, producing rows.
Reverses the earlier call that it should be a stateless batch pass. The
arithmetic says otherwise: a few hundred thousand directories at local
inference speed is days of continuous work, so uniform coverage is not
affordable and choosing where to spend attention is the feature rather
than an optimisation. Hierarchy pushes the same way, since a parent's
annotation wants its children's first.
The loop calls the same tools shipped over MCP. One definition, two
consumers, so it is a client of the public surface with no privileged
access and swapping in a frontier model is configuration. That also makes
it a completeness test: if the native agent needs something MCP does not
expose, the surface is wrong and it fails loudly. MCP therefore lands
before the loop rather than beside it.
LM Studio and Ollama first, both over OpenAI-compatible chat completions.
Spacebot is a variable and nothing here may assume it. Harnesses are
commoditising fast; the map is the asset because nobody else is building
it, and a harness is a depreciating one because everybody is.
Two rules with teeth fall out. MCP is the primary interface rather than
an export of one, so no capability may exist only in the app. And the
annotation pass is a batch job over directory metadata, not an agent
loop, which keeps the model swappable and makes the schema for what gets
written matter more than whatever writes it.
Also notes the honest risk: the annotation pass is the one phase an
outside harness could do better given the map, so it is a bet on scale
and it deletes cleanly if that bet stops paying.
Everyone is building agent tooling above the filesystem and assuming the
filesystem is one machine with everything mounted. That breaks for anyone
who owns more than one drive. The map is the product, and agents are the
interface to it.
The planning primitive was the open question and it falls out of the
catalog, which is already designed: the catalog holds actual placement, a
plan is desired placement, the workflow is the diff. Redundancy is a
predicate over the same rows, so at-risk needs no new concept.
Argues for pulling the catalog forward from teardown phase 5. It reads
from source stores, which now exist, so it does not wait on the entries
teardown. Deliberate sources plus catalog plus MCP answer the use case
with someone's agent as the planner, which puts the value ahead of the
product.
A sweep reads absence as deletion, which only holds when the walk would
have seen the file had it been there. enumerates_whole_source is that
test: volume indexing, recursive, no depth limit, no rules. Today one
path passes it, archiving a removable drive, which is also the one case
where the origin goes in a drawer and absence is all you will ever get.
A browse dispatches IndexScope::Current on every directory listing, and
a sweep after one of those would read the rest of the source as deleted.
Rules are a per-walk parameter rather than a source property, so a walk
that hides .git deletes what the walk before it recorded.
discovery treats an unreadable directory as a non-critical error and
finishes with that subtree missing, so finish_sweep takes the paths the
walk failed to open and spares everything under them. Their bindings
survive intact, and the next walk that can read them resolves them as
unchanged rather than rediscovering them.
Interrupt needed nothing: check_interrupt returns before the sweep
closes. sweep_open carries the rest across serialization, and a walk
resumed in a new process finds a ledger with no sweep and closes nothing.
The arena stays the read path and the store gets the same observation
through a queue, so nothing a browse waits on ever waits on SQLite. One
writer task per source owns the ledger and the db, which is what keeps
resolution lock-free: resolve needs &mut, and a single consumer means
nothing else can be halfway through a rebind while it runs.
MemoryAdapter turned out not to be where the walk and the watcher
converge. Only the watcher goes through it; the ephemeral walk batches
straight into the arena from the indexer job, and MemoryAdapter's
IndexPersistence half has no caller. So both sites fan out, and
collapsing them is P4 work rather than a precondition.
Ledger gains a sweep, so absence can become a removal, and rebind, so a
rename the watcher watched happen needs no evidence weighing. apply_files
orders each batch parents-first: parent_uuid is a foreign key and the
watcher reports what the kernel coalesced in the order it coalesced it.
The sweep is unwired and the filesystem passes no watermark. Both need
the walk's completeness semantics settled, which is its own decision.
The store half of P2. A walk becomes rows in the same record table the
adapters write, through its own ingest.
Ledger is the identity procedure the plan wanted as a table: every binding
loaded into two maps at attach, resolution a hash lookup. Path hit with the
same size and mtime writes nothing. Path hit with either changed keeps the
record and drops the content identity, since a hash that names the old bytes
is worse than none. Path miss with an inode hit rebinds only when size or
mtime corroborates, because inode numbers get reused and a wrong rebind moves
someone's assertions onto an unrelated file, while a missed one costs a
re-tag.
apply_files writes the batch, its removals and its watermark in one
transaction, so a kill partway through leaves neither a partial batch nor a
watermark claiming work that did not land.
Both writers share INSERT_RECORD, now conflicting on the uuid rather than on
(type, external_id): the walker resolves identity before writing, and a moved
file is one row updated instead of a delete and an insert that would strand
its assertions. UNIQUE (type, external_id) still stands, so two records
claiming one path fails loudly.
The filesystem is a declared data type, one model in Rust, so facet_file, the
_schema row and the migration diff all come from the machinery the adapters
use. FILE_SCHEMA goes: it was the same table declared twice. No index on
facet_file.inode, against the design doc, because the only reader is one
sequential scan at attach.
Convergence P1.6-P1.8, plus the facet the walker will write. All of it is
cheap now and a migration over user data once a filesystem source has put
millions of rows on these tables.
record.uuid is a BLOB, 16 bytes instead of 36, on the primary key, in
idx_record_parent, on both sides of every edge and on every facet row. The
crate speaks Uuid rather than String throughout. json_object refuses BLOBs,
so list_records_full selects the uuid beside the JSON and merges it in.
content carries a convergent uuid derived from its hash, sampled_hash is
UNIQUE, and set_content_identity upserts, so two copies of one file in a
source share a row. The uuid re-derives when the integrity tier lands, which
is why record.content_id stays the local rowid: the row has to be stable
across the ladder while the name of the bytes is not. A write carrying only
the cheap hash never walks a confirmed id back down to a guess.
record_overlay keys on record_uuid and carries (type, external_id,
content_uuid) as rebind evidence plus hlc and device_uuid. No foreign key to
record, deliberately: a cascade would delete the half of a store nothing can
rebuild. The old path key was quietly doing the rebinding, so rebind_overlays
now does it explicitly, content uuid first and the source key second, for a
rebuilt generation or a store copied off another device.
FILE_SCHEMA carries facet_file, declared in Rust rather than as a TOML data
type. Nothing applies it yet.
A store has two halves. The generation is rebuildable for as long as its
origin still answers, the assertion layer never is, and the origin stops
answering without an event. So the invariant is a rule about what code may
assume rather than a claim about stores, and the registry row tracks what is
actually known about the origin.
Falls out of that: reindex and evict are transactions over one half, and sync
replicates the generation while merging assertions. Which means the assertion
tables need their key and their merge columns settled before P2 fills them,
so P1 gains the re-key, convergent content ids and the uuid narrowing.
New doc is docs/core/design/source-durability.md. Also records two
contradictions found on the way: whether sources may nest (contract 1 and the
previs disagree), and that nothing reconciles record uuids when two devices
index the same origin.
The README still advertised a screening pipeline — Prompt Guard 2, quarantine,
trust metadata on results — that was a stub and is gone. Cut the section
rather than describe an intention as a feature.
The two archive docs went stale from later commits in the same session: they
still put durable assertions in registry.db, still said deleting a source
leaves them behind, and did not mention record_overlay in the store layout.
Both now describe one file per source with no shared durable layer.
Also: the crate layering is stated in both docs, AGENTS.md points at
crates/store for the record table, and the crate docs stop promising
cross-source edges — edge relates records within one source, and anything
spanning sources arrives with the catalog.
crates/store holds the store and nothing else: record, facets, content, edge,
record_overlay, schema parse/codegen/migration, SourceDb, SourceManager and
per-source FTS. It depends on no other first-party crate, which is the point —
the filesystem walker needs to write records in P2 without pulling in an
adapter engine to do it.
TrustTier moves with it. It is a property of a source, not of the ingest that
declared it, and both the registry and the search router need it.
sd-archive keeps what needs more than one source — Engine, Registry,
SearchRouter — plus the adapter runtime, and now sits on sd-store. Its
dependency list loses async-trait, futures, indexmap and dashmap; sd-store
takes indexmap and drops dashmap, which nothing used.
Errors split along the same line: sd-store carries the store failures,
sd-archive adds the adapter ones and passes the rest through with #[from].
Dead variants went with the split — SchemaCodegen, DataTypeNotFound, Search,
PathNotFound, SchemaMigrationRefused and Toml had no construction site.
tests/record.rs was a store test all along and moves to crates/store/tests.
The rest of the split waits for P3. Once library.db owns the source list,
Engine/Registry/SearchRouter own nothing that is not core's and move there,
and what is left of sd-archive is the adapter runtime.
grouping and lib_edge had no writer, no reader and no op — Engine and
SourceManager exposed link/unlink/neighbors/upsert_grouping as pass-throughs
that nothing called. They are also the wrong shape for where Photos
enrichment is going: a person appearing in photos on three drives is a
definition replicated into three source stores, not a row in a library-level
table that outlives all three.
record_overlay moves into source.db as SourceDb::{set_overlay, get_overlay,
overlays_for}, keyed (type, external_id) with source_id dropped — the source
is now implied by which file you opened. The search router reads overlays off
the same handle it just searched instead of joining a second database.
registry.db is down to sources and data_types. No durable layer, so there is
no shared file left to argue about.
Semantic change worth knowing: deleting a source now discards its assertions,
since they live in the file being deleted. Re-index is the operation that
keeps them — replace what an ingest produced, leave record_overlay alone —
and it does not exist yet. Nothing reaches delete_source with assertions
today, so nothing regresses, but re-index has to land before either is
reachable from an op.
Photo harvesting comes back as enrichment of a filesystem source — places,
faces, albums — not as an adapter minting its own records. That makes tags,
albums, faces and places one primitive: a definition with a stable identity
replicated into every source that uses it, plus membership edges local to the
source holding the member. It is the previs's tag rule applied to everything
shaped like it.
Follows from that:
- grouping and lib_edge are deleted. Neither has a writer, a reader or an op,
and a person appearing in photos on three drives is three replicated
definitions, not a row in a library-level table that outlives all three.
record_overlay stays for scalar assertions and moves into source.db.
- edge cannot carry enrichment yet: no payload column, and PRIMARY KEY
(src, dst, type) makes the same person twice in one photo unrepresentable.
Now a P1 item since P2 writes against it.
- P1 gains the crate split (sd-store + sd-adapter, engine and registry to core
at P3) and the Photos decision: retire the adapter, keep the reader as the
harvest path.
lib.rs, library.rs, source.rs and record.rs all opened by describing a
per-source index as a cache that gets rebuilt when a source is re-added.
The teardown register already ruled the other way — a source store is user
data. Rebuild is a recovery path, and plenty of origins cannot be re-scanned
on demand anyway: a detached drive, a revoked token, a closed account.
Where the two-file split is still load-bearing the docs now say so and point
at the convergence plan rather than defending it.
docs/core/design/archive.md and docs/archive/README.md described archive as
a second data plane beside the VDFS. The README also documented an
adapter.toml format and a JSONL protocol that never existed in this tree,
and claimed a subprocess sandbox that doesn't. Both rewritten from source.
"spine" is now "the record table" across the remaining docs, and
2026-08-17-record-spine-transplant.md renamed to match.
Adds docs/plans/2026-08-22-source-convergence.md: the register for getting
to one store shape and one source registry before the entries teardown
continues. AGENTS.md and the teardown execution plan point at it.
Jamie Pine