Nothing produces a row in `entry` after this. The walk fills the volume
index and its source store, which is what the arena has been doing since
the storage map landed; the persistent half of the indexer was a second
writer to a second store that no source ever read.
Gone: IndexPersistence, and with it the branch in every phase. The state
machine is Discovery then Processing, since Aggregation existed to bubble
directory sizes the arena now maintains incrementally, and
ContentIdentification is ContentIdentityJob's job against the source store.
DatabaseAdapter, PersistentEventHandler and the location watching that fed
them go too, so ArenaWriter is the only change handler and the walk and the
watcher differ only in Notify.
DatabaseStorage was 1230 lines of entry writes wrapped around three things
the arena actually uses. Those are ops::indexing::metadata now: EntryMetadata,
extract_metadata and get_inode, plus is_hidden_path. delete_subtree moves onto
the entry entity, which is what it operates on and what it dies with.
locations.rescan, locations.enable_indexing and locations.triggerJob dispatched
the deleted indexer or its media jobs, so the ops and their UI go. Reindex and
Run Now are gone from the inspector; the job policy toggles stay, since they
outlive the ops that read them. Adding a location no longer dispatches a walk,
which is the point: the drive is already mapped.
The entry tables and their readers are still here. That is the next commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
P4 was titled "Delete entries" and quietly assumed locations went with them.
They don't. A location stops owning records and becomes a policy over a path:
the subtree kept at full fidelity and watched.
That's the Retention.covered parameter the storage map already has and is
currently filling from cache.sources() roots because there was nothing better
to ask. Locations are the better thing to ask, so this gives an existing
parameter its owner rather than adding a mechanism.
The row loses entry_id, device_id, volume_id, scan_state, last_scan_at,
error_message, the two counts, index_mode and job_policies. Each of those
lives somewhere that already has it, and the arena's rollups are the honest
version of the counts since a stub can say 400 GB was summarised and a
persisted integer can't.
Also covers the sweep across the ~64 files that reference entities::entry,
and flags triggerJob, import and export as the three ops that don't survive
mechanically and want deciding on their own merits.
The store's filesystem schema had one model with six fields, so a record
could say how big a file is and not what took the picture. The old library
had image_media_data and video_media_data and both were lost in the move.
Facets are schema-driven, so these are declarations rather than machinery:
generate_ddl emits facet_image, facet_video and facet_audio on create, and
diff_schemas adds them to a store that predates them. Audio came along with
the other two, being the same kind of fact and already shaped in the tree.
No writer yet, enrichment is still parked. What this buys is that the shape
can only be decided once.
The search contract named title = "name" on a model with no name field. A
filesystem record's display name is its filename, which the walk writes to
record.title, so the contract points at the record instead.
The archive engine kept its own registry in registry.db while filesystem sources
lived in library.db, so there were two lists of sources that could not see each
other. sources.list unioned them and said so in a comment.
Now the engine holds stores and adapters and is handed a SourceRef. Which
sources exist is a question for the library, with one answer: the sources table,
read back through ops/sources/registry. Creating a source mints the id and
writes the row here; the engine is told what to name the store. A sync answers
with what the run did and this side records it, so a status can't be written
without the row that shows it.
registry.db and its data_types table are deleted, unread by anything.
Two moves on disk. Adapter stores live under SourceDirs beside every other
source's and are named by uuid without dashes, the way filesystem stores already
were. Installed adapters moved to <data>/adapters, since an adapter is installed
once for a machine rather than once per library. <library>/archive/ is no longer
written.
SourceInfo::from_row replaces the two constructors that built the same shape
from different halves. Existing adapter sources do not carry over: their rows
were in registry.db and their stores are under the old path.
content had 0 rows because nothing ever called the two functions that fill it.
ContentIdentityJob does: it claims files a source holds and has not identified,
hashes them a few at a time, and hands the identities back through the same
writer as everything else, so a content row can never land before the record it
points at.
Dispatched at LOW behind the walk that produced the records and behind the drive
map, which is the order the storage map doc set out: what someone kept, then the
rest of the machine, then the bytes.
files_needing_content rebuilds each path from its parent and its name, the same
join Ledger::load uses. P2.6 means the path is not a column to read, and a file
whose parent has no path is skipped rather than guessed at.
Only the sampled tier runs. An integrity hash is a full read of every byte on
the drive and belongs behind a decision that needs one, which is the rule the
ladder in sd_store::content exists to keep.
set_content_identities commits a batch in one transaction; the single-file
version now shares its statement. A file that cannot be read keeps its empty
content_id and comes back on the next pass, and a pass that identifies nothing
has found the tail of the queue and stops rather than looping on it.
Nothing reads the rows yet: duplicate detection still queries content_identity
in library.db.
Three destinations had to agree about every change — the arena, the source
store, the clients — and each call site wrote all three by hand. They had
already drifted once: a move updated the arena and the store correctly and told
nobody, because creates emitted inline while moves went through a hook that was
handed the path the file had just left.
MemoryAdapter is now ArenaWriter with one entry point. apply(Seen) resolves
identity, mutates the arena, and answers with the ArenaChange that both the
store write and the client event are derived from, so "written durably but never
announced" stops being representable. The walk and the watcher differ only in
Notify: every change, one event per batch, or silence for a drive being mapped
in the background. run_ephemeral_processing_static lost its own copy of all
three writes and is a third of its former size.
Two things fell out of it:
- Mutation::Unchanged. A re-walk announced every entry it re-observed, which
meant serialising a million files nobody touched. The store still hears about
all of them, which is what keeps a sweep from condemning them.
- A recount now reaches clients. It was an arena write with no way to tell
anyone, so a summarised directory whose size changed was correct on the next
query and invisible to anything already on screen.
IndexPersistence had no production callers and goes, with PersistenceFactory and
DatabaseAdapterForJob. The one query the latter still served moved to the change
detector, which was its only caller. ChangeHandler stays for the database
adapter, minus emit_change_event, which the arena can no longer be asked to do
separately.
The Analyzer wants a live sunburst of a volume, and the thing every tool doing
this today cannot do is the one thing that is already free here: the arena
maintains subtree_bytes up the ancestor chain on every add and remove, so a
file landing anywhere corrects every ring outward with no rescan. The map being
live is a property of the arena that nothing surfaces yet.
What is missing is the map. Measured on one 990 GB drive: 11,131,151 entries
exist, 1,678,592 are mapped, 271 GB of 866 GB used. Two independent causes.
Nothing walks the volume at all, so /Applications and /Library have never been
enumerated. Rules prune the rest, which is where ~/Library's 231 GB went.
The reframing that makes it affordable is that traversal is unavoidable and
retention is not: a size cannot be known without enumerating, so the policy is
only ever about what to keep. Measured: 118,859 directories within seven levels
of the volume root, 54 MB, against 5.0 GB for all 11.1M entries. So home at
full fidelity plus a complete seven-ring structure is about what the daemon
holds today for two rings and a fifth of the disk.
Records the stub design, why delta tracking cannot keep one honest, the
dirty-and-recount that can, the queue order that puts a person's own files
first, and the per-volume setting. Phases at the end.
"Medium" named the abstraction after its own exception. Every mount the volume
manager can see gets a record, network shares included, so a filesystem with no
volume is a fallback rather than a co-equal case. The word for a drive here is
volume, and this is its index.
MediumSlot is VolumeIndex, MediumKey is VolumeKey with Id and Path variants
instead of Volume and Root, track_medium is track_volume, and the layout moves
from mediums/<id>/ to volumes/<id>/. The derived namespace uuid is unchanged,
so ids are the same; only the directory holding them is renamed.
Nothing else moves. A volume index is still the whole drive's map, still keyed
by the drive rather than by any registration, still what a nested source shares
instead of forking.
A machine needs full local search over every file and a size figure the
analyser can trust. Neither is the same question as what someone chose to keep,
and the sources migration collapsed the two.
A partition is keyed by source id today and routed by longest prefix, so a
source nested inside another forks rather than narrows: two partitions, two
snapshots, two ledgers, and two uuids for one file. A tag applied through one
root is invisible through the other, which is the second-namespace problem
between two filesystem sources.
P2.7 keys the arena by medium instead. One drive, one arena, one snapshot,
whatever is persisted from it. A source says persist this subtree and owns a
data.db, so nesting is ordinary because the nested things are no longer the
same thing. The arena mints identity once per medium and a source adopts what
is already there, which is both what stops the fork and what makes creating a
source over mapped files fast.
A path under no source still has a uuid and still cannot carry an assertion,
since assertions live in a source store. Choosing to keep something is what
gives it somewhere to keep things.
Also records the completeness cost. An internal volume walks with default
rules, so /Library, ~/Library, node_modules and gitignored files never enter
the map: 262 GB indexed against roughly 900 GB used on a 1 TB drive. Rules
become view-time lenses and the walk records what it can read, which takes a
2.1M entry map to somewhere between 5 and 8M, and 1.0 GB resident to between
2.5 and 4. collection_flags already has room for a filtered bit.
Settles the nesting question in the Open register. Both live documents were
right about their own half: one filesystem owns one record namespace, and that
namespace is the arena rather than the source.
external_id is the full relative path on every record and UNIQUE(type,
external_id) holds a second copy, which is most of a 2.1M-record store. The old
library stored a path once per directory and derived the rest from parent_id
and name. Dropping that was a mistake.
Measured by replaying the live store through the proposed schema, both
databases vacuumed, same 2,123,008 records. Record side 751 MB against 423 MB,
whole database 896 MB against 568 MB, 443 bytes per record against 281. The
index it removes serves path lookup; the one that replaces it answers list this
directory's children, which is the query the UI makes.
Lookup stays two probes because directories keep their paths: one into
directory_path for the parent, one into (parent_uuid, name). Renames stop
rewriting every descendant record and rewrite descendant directories only.
The replay found 0 unresolved parents, so the existing store can be backfilled
in place rather than re-walked. That matters because a re-walk is where the
surviving created_at values would go.
Ties P2.5 (2) to this: the tree is currently a fidelity bug and becomes the
addressing mechanism, so the two land together.
Compared the new per-source store against the last pre-sources library on the
same machine: 2.1M records against 23,157 entries, and the new one holds less
about each file. Three causes, tangled together.
The walk discards metadata it already stats. DirEntry carries five fields and
run_ephemeral_processing_static hardcodes created, accessed and permissions to
None, so birth time is gone for 2.1M files. The stat was already paid for.
The bulk walk does not build the tree. Parent linkage needs the ledger to have
resolved the directory first, which the incremental path satisfies and the walk
does not: 1,349 of 2.1M walked records are parented against 19,769 of 19,856
written by the watcher. 240,122 directories exist and 949 know their parent.
Media facets were never declared. The schema has one model with six fields
where the old library had EXIF, GPS, camera and codec tables.
Sidecars move into the source store. A sidecar is keyed by content uuid and
content now lives per source, so a row in library.db is a foreign key across
database files with nothing enforcing it. sidecar splits: content_sidecar in
the source store, sidecar_availability staying in the library because it is a
claim about a device. Answers the open question in the extensibility plan.
Also recorded under Open: external_id is the full path on every record and the
unique index holds a second copy, so 518 MB of a 1.01 GB database is paths,
while idx_record_parent costs 21 MB and is 99% empty.
JobStarted was emitted for every job while JobCompleted, JobFailed and
JobCancelled were all gated behind should_emit_events. So any job that
opted out of events still announced a start and never announced an end,
and the client had a row nothing could clear. That is every ephemeral
browse, since should_emit_events is should_persist unless the job is
indexing a whole volume.
Gating the start the same way makes an event-free job invisible end to
end, which is what the flag was for.
Also reorders the completion arm. Clients refetch the job list when they
see a completion, and the job was removed from running_jobs after the
event went out, so a refetch that landed first got the job back in the
answer with nothing further coming to clear it. Reading the output,
removing, then announcing closes that window.
Tracking a volume did not index it. There were three flows and none of
them was one gesture: the context menu offered Track and Index as
separate items, and AddStorageModal tracked the volume and then created a
Location, which is the model being deleted.
volumes.track now calls the same track_and_index that sources.track does.
The medium and the index stay separate underneath, since a drive can be
tracked with nothing on it indexed and a folder can be indexed on a
volume nobody tracks; what converges is the click.
The shared body matters more than the saving. Both entry points forget
the same two steps otherwise, seeding from the snapshot and clearing what
the new pass will not revisit, which is exactly what sources.track was
missing when it was written separately.
An external mount indexes unfiltered, since a drive being archived wants
completeness rather than rules that hide files on purpose. An unmounted
drive tracks without walking and gets its map when it returns.
Index Volume becomes Re-index Volume and only shows for tracked drives,
since the first index is no longer something to ask for.
sources.track skipped two steps volumes.index does. Without
ensure_restored a partition that never restored is barred from saving
over an existing snapshot, so tracking a root that already had one would
index and then fail to persist. Without clear_for_reindex, entries from a
previous pass that this one does not revisit linger in the arena as files
that no longer exist.
Also writes down how it differs from volumes.track, since the names
collide on the word rather than the meaning: volumes.track marks a medium
as belonging to a library and indexes nothing, this makes an index of
what is on one. volumes.index is the older overlap and retires once
callers move over, because it takes a fingerprint and can therefore only
ever mean a whole drive.
A filesystem source used to appear as a side effect: browsing a directory
dispatched a walk and the walk registered whatever partition it landed
in. sources.track is that intent made explicit. Whole drive and
folder-within-a-drive are the same call with a different path, since the
volume resolves from the path and the source anchors to it.
Indexing starts immediately rather than waiting to be asked, and the
action returns as soon as the row exists, because registration is instant
and a walk over a drive is not. The job then reports against a source
that already exists.
sources.list serves both kinds now, so the sidebar stopped needing a
group per registry: filesystem rows render as walked roots with their
attachment state, adapter rows as before. The union is transitional,
reading filesystem rows from the sources table and adapter rows from
registry.db, and folding the second into the first is the rest of P3.
Content ids stay out of the walk on purpose. The walk is fast and worth
keeping fast, and records_needing_content_identity is already the query a
separate pass would drive. Nothing generates them yet, which the plan now
says plainly rather than implying the sampled tier is wired.
The arena and the store were minting separate uuids for the same file.
MemoryAdapter generated one inline while Ledger::resolve minted its own
for the record, so every file had two identities and nothing joined them.
Tags hang off the arena's; record_overlay assertions, and the AI
annotations that follow, hang off the record's. Indexing at scale before
fixing this would have baked the split into every row.
The ledger assigns identity now and the arena writes what it is told.
SourceStore::identify resolves a batch and answers with record uuids, and
it is the same message that takes the batch in, so there is no second
call that could disagree. One round trip per batch rather than per file:
resolution is a hash lookup and the reply is sent before anything
commits, so the cost is the queue rather than a transaction.
Volume indexing used to defer uuids entirely and no longer needs to,
since the ledger has one whether or not anything asks. A partition with
no store still mints for the session, because there is nothing durable
for it to agree with.
entry_uuids stays in the snapshot and stops mattering: it now holds the
same uuid the ledger does, so a discard costs a re-identify rather than
an identity.
Native dependencies, job presets, plugins and sidecars are four things,
and separating them is most of the work. Checked against the tree first:
the codecs were already unbundled by the install-size plan, the SDK is
already a plugin SDK missing only a manifest and a lifecycle, and
SidecarKind::Thumb already exists content-addressed. Less is missing than
the dump assumes.
The preset is the primitive worth building, because it is the same shape
as the workflow row the filesystem intelligence plan already needs:
durable, parameterized, executable, logged, with its reasoning attached.
One table, two consumers.
Draws the line that keeps it safe: a model composes registered job kinds,
a plugin adds kinds. If a model can author the command rather than the
parameters, durable execution and monitoring stop being features and
start being a liability.
Answers the sidecar question as placement rather than design. The hot
tier and the thumb sidecar are both already specified and do not compete;
what is open is that the sidecar tree is library-scoped, which is wrong
for an archival drive whose previews should travel with it. And the
location-or-source line is source, since locations are being deleted.
Keeps the GPUI plan where it is: Photos is the right wedge, and a second
full interface should follow the teardown rather than race it.
P3 is partly done: sources are a table, T2.0b is ruled, and the two seams
it left visible are written down rather than left to be rediscovered.
P4 gets the structural answer the move bug pointed at. Each
ChangeHandler method updates arena, store and event bus by hand with
nothing forcing them to agree, and they drifted: a move wrote the first
two and emitted nothing. An arena mutation should produce a change record
that the store write and the client event are both derived from, which
makes that failure unrepresentable rather than fixed.
It also answers what such a record carries, since the same bug showed it:
both the destination and the vacated directory, because a client renders
directory listings and a move invalidates two of them.
Belongs in P4 because MemoryAdapter is being rewritten there anyway and
IndexPersistence dies with the entries world that needed it. Earlier
means building the seam twice.
An annotation is an assertion and assertions outlive the generation by
design, so a model's claim is treated like a person's. Costs no schema
and forecloses nothing, since overlay.updated_at and record.modified_at
already sit beside each other for anything that later wants to surface an
aged claim.
Qwen3.8-27B: Apache 2.0, 262k context, native multimodal, ~16-17GB at
4-bit. Reference rather than requirement, since the loop targets an
OpenAI-compatible endpoint and a tool schema.
Long context moves the loop's step from one directory to a whole branch,
which is a real change to the design. What it does not do is make
annotation cheaper: output tokens bind, not input, so uniform coverage is
still days of generation. The win is that fewer annotations become
sufficient, because seeing a whole subtree is what justifies one
assertion covering it instead of three thousand near-identical ones.
Tool calling drops down the risk list for this model class; the
structured-output path stays as a fallback for weaker providers. Adds the
hardware floor and what happens below it, which is that the map is still
complete and someone's own frontier agent still answers over MCP.
Also notes thumbs.pvcache as a latent second input for a multimodal pass,
scoped out of P5 day one.
Two agents, and only one is the interface. The interface is a person and
their agent in a harness over MCP. Filesystem intelligence is a
Spacedrive-native loop over a local model, headless, producing rows.
Reverses the earlier call that it should be a stateless batch pass. The
arithmetic says otherwise: a few hundred thousand directories at local
inference speed is days of continuous work, so uniform coverage is not
affordable and choosing where to spend attention is the feature rather
than an optimisation. Hierarchy pushes the same way, since a parent's
annotation wants its children's first.
The loop calls the same tools shipped over MCP. One definition, two
consumers, so it is a client of the public surface with no privileged
access and swapping in a frontier model is configuration. That also makes
it a completeness test: if the native agent needs something MCP does not
expose, the surface is wrong and it fails loudly. MCP therefore lands
before the loop rather than beside it.
LM Studio and Ollama first, both over OpenAI-compatible chat completions.
Spacebot is a variable and nothing here may assume it. Harnesses are
commoditising fast; the map is the asset because nobody else is building
it, and a harness is a depreciating one because everybody is.
Two rules with teeth fall out. MCP is the primary interface rather than
an export of one, so no capability may exist only in the app. And the
annotation pass is a batch job over directory metadata, not an agent
loop, which keeps the model swappable and makes the schema for what gets
written matter more than whatever writes it.
Also notes the honest risk: the annotation pass is the one phase an
outside harness could do better given the map, so it is a bet on scale
and it deletes cleanly if that bet stops paying.
Everyone is building agent tooling above the filesystem and assuming the
filesystem is one machine with everything mounted. That breaks for anyone
who owns more than one drive. The map is the product, and agents are the
interface to it.
The planning primitive was the open question and it falls out of the
catalog, which is already designed: the catalog holds actual placement, a
plan is desired placement, the workflow is the diff. Redundancy is a
predicate over the same rows, so at-risk needs no new concept.
Argues for pulling the catalog forward from teardown phase 5. It reads
from source stores, which now exist, so it does not wait on the entries
teardown. Deliberate sources plus catalog plus MCP answer the use case
with someone's agent as the planner, which puts the value ahead of the
product.
A sweep reads absence as deletion, which only holds when the walk would
have seen the file had it been there. enumerates_whole_source is that
test: volume indexing, recursive, no depth limit, no rules. Today one
path passes it, archiving a removable drive, which is also the one case
where the origin goes in a drawer and absence is all you will ever get.
A browse dispatches IndexScope::Current on every directory listing, and
a sweep after one of those would read the rest of the source as deleted.
Rules are a per-walk parameter rather than a source property, so a walk
that hides .git deletes what the walk before it recorded.
discovery treats an unreadable directory as a non-critical error and
finishes with that subtree missing, so finish_sweep takes the paths the
walk failed to open and spares everything under them. Their bindings
survive intact, and the next walk that can read them resolves them as
unchanged rather than rediscovering them.
Interrupt needed nothing: check_interrupt returns before the sweep
closes. sweep_open carries the rest across serialization, and a walk
resumed in a new process finds a ledger with no sweep and closes nothing.
The arena stays the read path and the store gets the same observation
through a queue, so nothing a browse waits on ever waits on SQLite. One
writer task per source owns the ledger and the db, which is what keeps
resolution lock-free: resolve needs &mut, and a single consumer means
nothing else can be halfway through a rebind while it runs.
MemoryAdapter turned out not to be where the walk and the watcher
converge. Only the watcher goes through it; the ephemeral walk batches
straight into the arena from the indexer job, and MemoryAdapter's
IndexPersistence half has no caller. So both sites fan out, and
collapsing them is P4 work rather than a precondition.
Ledger gains a sweep, so absence can become a removal, and rebind, so a
rename the watcher watched happen needs no evidence weighing. apply_files
orders each batch parents-first: parent_uuid is a foreign key and the
watcher reports what the kernel coalesced in the order it coalesced it.
The sweep is unwired and the filesystem passes no watermark. Both need
the walk's completeness semantics settled, which is its own decision.
The store half of P2. A walk becomes rows in the same record table the
adapters write, through its own ingest.
Ledger is the identity procedure the plan wanted as a table: every binding
loaded into two maps at attach, resolution a hash lookup. Path hit with the
same size and mtime writes nothing. Path hit with either changed keeps the
record and drops the content identity, since a hash that names the old bytes
is worse than none. Path miss with an inode hit rebinds only when size or
mtime corroborates, because inode numbers get reused and a wrong rebind moves
someone's assertions onto an unrelated file, while a missed one costs a
re-tag.
apply_files writes the batch, its removals and its watermark in one
transaction, so a kill partway through leaves neither a partial batch nor a
watermark claiming work that did not land.
Both writers share INSERT_RECORD, now conflicting on the uuid rather than on
(type, external_id): the walker resolves identity before writing, and a moved
file is one row updated instead of a delete and an insert that would strand
its assertions. UNIQUE (type, external_id) still stands, so two records
claiming one path fails loudly.
The filesystem is a declared data type, one model in Rust, so facet_file, the
_schema row and the migration diff all come from the machinery the adapters
use. FILE_SCHEMA goes: it was the same table declared twice. No index on
facet_file.inode, against the design doc, because the only reader is one
sequential scan at attach.
Convergence P1.6-P1.8, plus the facet the walker will write. All of it is
cheap now and a migration over user data once a filesystem source has put
millions of rows on these tables.
record.uuid is a BLOB, 16 bytes instead of 36, on the primary key, in
idx_record_parent, on both sides of every edge and on every facet row. The
crate speaks Uuid rather than String throughout. json_object refuses BLOBs,
so list_records_full selects the uuid beside the JSON and merges it in.
content carries a convergent uuid derived from its hash, sampled_hash is
UNIQUE, and set_content_identity upserts, so two copies of one file in a
source share a row. The uuid re-derives when the integrity tier lands, which
is why record.content_id stays the local rowid: the row has to be stable
across the ladder while the name of the bytes is not. A write carrying only
the cheap hash never walks a confirmed id back down to a guess.
record_overlay keys on record_uuid and carries (type, external_id,
content_uuid) as rebind evidence plus hlc and device_uuid. No foreign key to
record, deliberately: a cascade would delete the half of a store nothing can
rebuild. The old path key was quietly doing the rebinding, so rebind_overlays
now does it explicitly, content uuid first and the source key second, for a
rebuilt generation or a store copied off another device.
FILE_SCHEMA carries facet_file, declared in Rust rather than as a TOML data
type. Nothing applies it yet.
A store has two halves. The generation is rebuildable for as long as its
origin still answers, the assertion layer never is, and the origin stops
answering without an event. So the invariant is a rule about what code may
assume rather than a claim about stores, and the registry row tracks what is
actually known about the origin.
Falls out of that: reindex and evict are transactions over one half, and sync
replicates the generation while merging assertions. Which means the assertion
tables need their key and their merge columns settled before P2 fills them,
so P1 gains the re-key, convergent content ids and the uuid narrowing.
New doc is docs/core/design/source-durability.md. Also records two
contradictions found on the way: whether sources may nest (contract 1 and the
previs disagree), and that nothing reconciles record uuids when two devices
index the same origin.
The README still advertised a screening pipeline — Prompt Guard 2, quarantine,
trust metadata on results — that was a stub and is gone. Cut the section
rather than describe an intention as a feature.
The two archive docs went stale from later commits in the same session: they
still put durable assertions in registry.db, still said deleting a source
leaves them behind, and did not mention record_overlay in the store layout.
Both now describe one file per source with no shared durable layer.
Also: the crate layering is stated in both docs, AGENTS.md points at
crates/store for the record table, and the crate docs stop promising
cross-source edges — edge relates records within one source, and anything
spanning sources arrives with the catalog.
crates/store holds the store and nothing else: record, facets, content, edge,
record_overlay, schema parse/codegen/migration, SourceDb, SourceManager and
per-source FTS. It depends on no other first-party crate, which is the point —
the filesystem walker needs to write records in P2 without pulling in an
adapter engine to do it.
TrustTier moves with it. It is a property of a source, not of the ingest that
declared it, and both the registry and the search router need it.
sd-archive keeps what needs more than one source — Engine, Registry,
SearchRouter — plus the adapter runtime, and now sits on sd-store. Its
dependency list loses async-trait, futures, indexmap and dashmap; sd-store
takes indexmap and drops dashmap, which nothing used.
Errors split along the same line: sd-store carries the store failures,
sd-archive adds the adapter ones and passes the rest through with #[from].
Dead variants went with the split — SchemaCodegen, DataTypeNotFound, Search,
PathNotFound, SchemaMigrationRefused and Toml had no construction site.
tests/record.rs was a store test all along and moves to crates/store/tests.
The rest of the split waits for P3. Once library.db owns the source list,
Engine/Registry/SearchRouter own nothing that is not core's and move there,
and what is left of sd-archive is the adapter runtime.
grouping and lib_edge had no writer, no reader and no op — Engine and
SourceManager exposed link/unlink/neighbors/upsert_grouping as pass-throughs
that nothing called. They are also the wrong shape for where Photos
enrichment is going: a person appearing in photos on three drives is a
definition replicated into three source stores, not a row in a library-level
table that outlives all three.
record_overlay moves into source.db as SourceDb::{set_overlay, get_overlay,
overlays_for}, keyed (type, external_id) with source_id dropped — the source
is now implied by which file you opened. The search router reads overlays off
the same handle it just searched instead of joining a second database.
registry.db is down to sources and data_types. No durable layer, so there is
no shared file left to argue about.
Semantic change worth knowing: deleting a source now discards its assertions,
since they live in the file being deleted. Re-index is the operation that
keeps them — replace what an ingest produced, leave record_overlay alone —
and it does not exist yet. Nothing reaches delete_source with assertions
today, so nothing regresses, but re-index has to land before either is
reachable from an op.
Photo harvesting comes back as enrichment of a filesystem source — places,
faces, albums — not as an adapter minting its own records. That makes tags,
albums, faces and places one primitive: a definition with a stable identity
replicated into every source that uses it, plus membership edges local to the
source holding the member. It is the previs's tag rule applied to everything
shaped like it.
Follows from that:
- grouping and lib_edge are deleted. Neither has a writer, a reader or an op,
and a person appearing in photos on three drives is three replicated
definitions, not a row in a library-level table that outlives all three.
record_overlay stays for scalar assertions and moves into source.db.
- edge cannot carry enrichment yet: no payload column, and PRIMARY KEY
(src, dst, type) makes the same person twice in one photo unrepresentable.
Now a P1 item since P2 writes against it.
- P1 gains the crate split (sd-store + sd-adapter, engine and registry to core
at P3) and the Photos decision: retire the adapter, keep the reader as the
harvest path.
lib.rs, library.rs, source.rs and record.rs all opened by describing a
per-source index as a cache that gets rebuilt when a source is re-added.
The teardown register already ruled the other way — a source store is user
data. Rebuild is a recovery path, and plenty of origins cannot be re-scanned
on demand anyway: a detached drive, a revoked token, a closed account.
Where the two-file split is still load-bearing the docs now say so and point
at the convergence plan rather than defending it.
docs/core/design/archive.md and docs/archive/README.md described archive as
a second data plane beside the VDFS. The README also documented an
adapter.toml format and a JSONL protocol that never existed in this tree,
and claimed a subprocess sandbox that doesn't. Both rewritten from source.
"spine" is now "the record table" across the remaining docs, and
2026-08-17-record-spine-transplant.md renamed to match.
Adds docs/plans/2026-08-22-source-convergence.md: the register for getting
to one store shape and one source registry before the entries teardown
continues. AGENTS.md and the teardown execution plan point at it.
embed.rs and search/vector.rs returned zero vectors and empty results;
safety.rs marked everything safe. engine.rs ran a post-sync pass over both
and logged counts for work that never happened. All three are gone, along
with _embedded_at and the three _safety_* columns on record.
TrustTier stays. Every adapter manifest declares one, and it is the input
screening policy keys on once screening is real.
Two things the verdict gate was hiding:
- upsert only refreshed the search index for records it had seen before,
because a new record was meant to enter the index later when screening
cleared it. Screening ran off the stub, and if the model failed to load a
separate fallback marked everything safe, so the index was populated by a
path nobody had read. upsert now indexes unconditionally.
- adapter stderr was piped and then dropped, which closes the pipe under an
adapter that writes to it. It drains into tracing at debug now.
tests/record.rs had a clear_for_indexing helper that existed only to walk
records past the gate, and tests/adapters.rs called mark_screened on its
probe for the same reason. Both gone; 40 tests green.
Photos no longer runs its own demo bake; it follows a Spacedrive explorer
window via a new navigation-focus resource and renders that window's
folder as a media grid, sourced from thumbnails baked by a new daemon-owned
thumbs service. Supporting changes: pvcache moves to a v2 format that
stores each tile's true aspect (envelope + frame) instead of forcing a
square crop, peer mounts now refresh on a generation check instead of only
at reconnect, and the archive crate's `spine` module is renamed to `record`
throughout.
Absolute paths from one developer's machine sat in an example, a shell script,
recorded benchmark metadata, task docs and design docs. The example and the
script were broken for anyone else as a result: the example now resolves against
CARGO_MANIFEST_DIR and the script derives the repo root from git. The duplicate
copy of that script under core/scripts is gone.
Also drops a pointer to a repository that is not public.
The entry schema is the last thing standing from the old model and nothing
needs it. These cover the cutover, the destination written as though built, and
the hazards for work landing next to it.
apps/native runs an atlas grid over real thumbnails read through pvcache, with
sd-bake producing cells on a priority pool. Pinned to our zed fork; not in
default-members, so it stays out of the normal check.
Image thumbs no longer sit behind ffmpeg gates, sidecar versions bust the cache,
the sidecar layout moves to a shared crate, and metadata backfill is separated
from generation. Video dimensions are read for real; the cas_id tier and dead
settings are removed.
Read-only share on loopback, a byte-range peer protocol for remote sources, and
a block cache in front of both so a second read of the same file does not
re-fetch it.
Sources resolve at startup without onboarding. Adds the Apple Photos native
adapter, collections and the analyzer, the bundle lens, and assertions with
read-time binding.
Each source owns its partition with detached restore, volume fingerprints are
machine-independent so an external drive keeps its identity across hosts, and
directory-size rollups update incrementally. Snapshots gain a restore gate, a
save invariant, unique tmp writes and an fsync before rename.
crates/archive carries record, content and edge tables with a durable overlay
beside them, so the rebuildable generation can be dropped and recreated without
losing anything written through the assertion surface.
Excludes source maps from the asset embed, sets default = [] on sd-core so
wasm, ffmpeg and heif are opt-in, and converts the icon and image sets to
webp. macOS thumbnails route through ImageIO and QuickLook instead of the
bundled codec stack, so the prebuilt bundle is a separate opt-in download.
- Add SHARE-000..011 task specs and docs/design/shares.md defining
the sd.app public sharing architecture and wire protocols
- Rework INDEX-010 to scope ephemeral UUID reconciliation per-library
instead of a single global map, since core can have multiple
libraries loaded at once
- Add sidecar serving and configurable bind host to apps/server, and
wire the web platform's daemon status to the server's own origin
- Add quick preview button to the file inspector, with a null-safe
useOptionalExplorer for use outside the explorer context
* feat: add redundancy awareness & cross-volume file comparison
Surface content redundancy data so users can answer "if this drive dies,
what do I lose?" — builds on existing content identity and volume systems.
Backend:
- New `redundancy.summary` library query with per-volume at-risk vs
redundant byte/file counts and a library-wide replication score
- Extend `SearchFilters` with `at_risk`, `on_volumes`, `not_on_volumes`,
`min_volume_count`, `max_volume_count` filters
- Add composite index migration on entries(content_id, volume_id)
Frontend:
- `/redundancy` dashboard with replication score, volume bars, at-risk callout
- `/redundancy/at-risk` paginated file list sorted by size
- `/redundancy/compare` two-volume comparison (unique/shared toggle)
- Sidebar ShieldCheck button linking to redundancy view
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* redundancy UI improvements + ZFS volume detection fix
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix ZFS pool capacity reporting and stats filtering
- ZFS: override total_capacity for pool-root volumes using zfs list
used+available. df under-reports pool-root Size because it only
counts the root dataset's own used bytes plus avail — on a 60 TB
raidz2 pool this shows as ~15 TB instead of ~62 TB. The pool root's
own used property includes descendants, so used+available is the
real usable capacity.
- Library stats: drop volumes where is_user_visible=false AND
re-apply should_hide_by_mount_path retroactively so stale DB rows
(detected before the Linux visibility filters existed) don't
inflate reported capacity.
- Extract should_hide_by_mount_path into volume/utils as a shared
helper used by both the list query and the stats calculation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* show capacity and visibility in sd volume list
Makes it possible to verify library-level capacity aggregation from
the CLI — previously the list only showed mount, fingerprint, and
tracked/mounted state, which meant debugging the ZFS pool capacity
issue required querying the library DB directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* document filesystem support matrix and detection
New docs/core/filesystems.mdx covering per-filesystem capabilities
(CoW, pool-awareness, visibility filtering, capacity correction),
platform detection strategies, the FilesystemHandler trait, Linux/
macOS/ZFS visibility rules, the ZFS pool-root capacity problem and
fix, copy strategy selection, and known limitations.
Registered under File Management in both mint.json and docs.json.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* WIP: redundancy filter wiring across search, CLI, and UI
- core/src/ops/search: wire redundancy filters (at_risk, on_volumes,
not_on_volumes, min/max volume_count) through the search query;
fix UUID-to-SQLite BLOB literal so volume UUID comparisons actually
match (volumes.uuid is stored as a 16-byte BLOB, quoted-string
comparison silently returned zero rows).
- apps/cli: new redundancy subcommand + populate the new
SearchFilters fields from search args.
- packages/interface: redundancy at-risk and compare pages reworked
to consume the new filter surface; explorer context/hook updates
to support redundancy-scoped views.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* add web context menu renderer and UI polish
- New WebContextMenuProvider + Radix DropdownMenu-based renderer anchored
at cursor via a 1x1 virtual trigger. Handles separators, submenus,
disabled, and the danger variant via text-status-error.
- useContextMenu now routes web clicks through the provider instead of
parking data in unused local state, and trims leading/trailing/adjacent
separators so condition-filtered menus don't render orphaned lines.
- Drop app-frame corner rounding on the web build.
- Add shrink-0 to the sidebar space switcher so the scrollable sibling
can't compress it vertically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* sd-server dev workflow: auto web build, shutdown watchdog, stable data dir
- build.rs runs `bun run build` in apps/web so `just dev-server` always
embeds the latest UI. rerun-if-changed covers apps/web/src,
packages/interface/src, and packages/ts-client/src so Rust-only edits
skip the rebuild. Skips gracefully when bun isn't on PATH or
SD_SKIP_WEB_BUILD is set; Dockerfile sets the latter since dist is
pre-built and bun isn't in the Rust stage.
- Graceful shutdown was hanging because the browser holds the /events
SSE stream open forever and axum waits for all connections to drain.
After the first signal, arm a background force-exit on second Ctrl+C
or 5s timeout so the process can't stick.
- Debug builds were starting from a fresh tempfile::tempdir() on every
run (the TempDir handle dropped at end of the closure, deleting the
dir we just took a path to). Default to ~/.spacedrive in debug so data
persists and `just dev-server` shares a data dir with the Tauri app.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* add Sources space item alongside Redundancy in default library layout
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* add TrueNAS native build script
Uses zig cc as C/C++ compiler on TrueNAS Scale where /usr is
read-only and no system gcc exists. Dev tools live at
/mnt/pool/dev-tools/ (zig, cmake, make, extracted deb headers).
Builds sd-server + sd-cli in ~4 min on a 12-core NAS. AI feature
disabled (whisper.cpp C11 atomics incompatible with zig clang-18).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* never block RPC on synchronous statistics calculation
On first load (fresh library, all stats zero), libraries.info used
to calculate statistics synchronously before responding. On large
libraries during active indexing this hangs indefinitely — the
closure-table walk in calculate_file_statistics loads every
descendant ID into a Vec then issues a WHERE IN(...) with millions
of entries, which SQLite can't finish while the indexer is writing.
Now always return cached (possibly zero) stats and let the
background recalculate_statistics task fill them in. The UI
refreshes via the ResourceChanged event when the calculation
completes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* self-heal protocol handler registration on re-init
Core::new() registers default protocol handlers after starting networking,
but swallows any failure (error is only logged). If the initial registration
fails — e.g. on a host where start_networking hasn't fully set up the event
loop command sender by the time register_default_protocol_handlers runs —
the registry is left empty. A subsequent call to Core::init_networking()
would see `services.networking().is_some()` and skip re-registration,
permanently leaving protocols unregistered for the life of the process.
sd-server calls init_networking() right after Core::new(), so it's the
client most exposed to this. Symptom: pairing over the web UI returns
"Pairing protocol not registered" while the same library works fine
from Tauri and mobile.
Fix: init_networking now queries the registry directly for the pairing
handler and re-registers the default set if it's missing, independent of
whether networking is already initialized.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fall back to pkarr+DNS discovery when mDNS port is unavailable
Iroh's endpoint.bind() fails wholesale if any configured discovery service
fails to initialize. MdnsDiscovery requires binding UDP :5353, which on
most Linux systems (including TrueNAS) is already owned by avahi-daemon.
Result: endpoint creation errors out with "Service 'mdns' error", the
event loop never starts, command_sender stays None, and protocol
registration fails — so sd-server has no working networking at all.
Make mDNS best-effort: on any error whose message mentions "mdns",
retry endpoint creation with only pkarr + DNS discovery. Local-network
auto-discovery is lost but remote pairing via node ID (which uses n0's
DNS infrastructure, not mDNS) continues to work normally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* succeed pairing if either mDNS or relay discovery wins
The dual-path discovery in start_pairing_as_joiner_with_code used
tokio::select! to race mDNS and relay. select! resolves on the first
branch to complete — including errors — so a host that can't bind
mDNS (e.g. a Linux box where avahi already owns UDP :5353) would fail
pairing wholesale: mDNS discovery errors out in <1ms with
"Failed to create mDNS discovery: Service 'mdns' error", that Err
wins the race, and relay discovery gets cancelled before it can even
begin.
Switch to futures::select_ok so we only return the error if EVERY
discovery path has failed. mDNS failing immediately now leaves relay
running to completion, which is the common case for remote pairing
into a NAS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
