A replica carries the index alone, so titan's 99,137 tiles are invisible from
the Mac. The cache is machine-local because the thumbnail request answers with
a path the client maps, which is also why the client needs no change: fill a
local cache file for the remote volume and the same answer works.
Adds the fetch design (where a remote tile lands, why the identities already
line up, why the owner's version has to travel with the tile, FetchTile and
WebP on the wire) as P6, and the durable sidecar tier as P7 behind L5. Records
the deployed rebuild and thumbnail results in the register.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The registry fix is on titan. Restoring the 1.76M-record replica from disk
at startup went from 56 minutes to 19.3 seconds, and the refetch that
follows a restart from 60 minutes to 8, nearly all of it now the download.
The restart still refetches: the restore publishes its share ten seconds
after the first sync pass asks for it. Speed alone will not close that.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rebuilding a replica's arena from a delivered database adds rows one at a
time through Arena::add_entry, and that method built a FileTypeRegistry per
call. Building one parses every built-in definition: 92 KB of TOML across
15 files and 349 types, about 3 ms. That was the whole cost of an insert.
Titan spent about 57 minutes rebuilding the Mac's 1.76M-entry home source
after each transfer. The database was never the problem: the full join scan
is 3 s, and one 2,000 row page is 7 ms against an optimal plan.
The built-in set never changes after startup, so one instance serves the
process. 20,000 inserts drop from 60 s to 143 ms, 3.00 ms to 7.2 µs each,
which puts that rebuild near 13 s in a debug build. The batch path built one
per batch and now shares the same instance.
core/tests/arena_insert_cost.rs fails if an insert starts rebuilding it.
Titan runs the store-revision build since 21:34. In the 19.5 hours before
the deploy it transferred nothing for the Mac's unchanged home source, where
the old file-metadata generation would have made about 18 full copies.
The NAS had FFmpeg all along, three static builds under the CCTV remux
tools, just not anywhere the daemon looks: its PATH is
/usr/local/bin:/usr/bin:/bin:/usr/games and no path variable was set.
bin/ffmpeg and bin/ffprobe are now n8.1.2 copies living beside the daemon,
so thumbnails do not depend on the cctv dataset, and the start command sets
SPACEDRIVE_FFMPEG_PATH and SPACEDRIVE_FFPROBE_PATH. A forced tile on a CCTV
clip spawned that ffmpeg with the 384px thumbnail filter and baked a frame.
Titan also has an RTX 3050 and a build with cuda, vaapi, qsv and vulkan,
unmeasured for single-frame seeks.
The restart exposed the next defect. A replica publishes its share only
after rebuilding its arena, about an hour for the home database, and the
first sync pass runs 30 seconds after the peer connects. It finds no share,
never reaches transfer_due, and refetches 736 MB it already had on disk with
the right generation in its manifest. Recorded as P1 in the replication
plan, where R3 already promised restarts would not reload.
A nested source's generation came from the size and mtime of data.db and
its WAL. SQLite deletes the WAL when a pool's last connection closes and
recreates it on the next read, and sqlx closes idle connections after 10
minutes, so the generation moved without any row changing. Titan re-copied
the Mac's unchanged home store in full four times overnight, each round a
736 MB transfer and a 57-minute arena rebuild.
Stores now keep a _revision row, a store id and a count of committed
changes, maintained by triggers on every tracked table. Updates that change
nothing, or only restamp scan_epoch or indexed_at, don't count. Listings and
database headers use the revision instead of file metadata.
The triggers put each single-row store write in a statement journal: the
300k-file scale test builds in 16-18s instead of 12-14s. temp_store=MEMORY
brings it to 9-10s but moves large sorts on the writable pool into memory,
so that's an open decision in the new incremental replication plan.
Verified live: after one switch-over copy, titan made no transfer for twelve
minutes, including across a Mac daemon restart.
Titan runs 4e58ddca5. Its library was backed up to
data/backup/pre-entries-drop before the FD4 migration converged it on the
14-table schema, with its nine sources, nine volumes, and pins intact. The
daemon it replaced had aborted on a failed allocation that morning, so the
reliability row carries what the restarted one looks like.
The device facts row moves to live on both machines and records two bugs
the deploy turned up: the periodic reconnect never dials a peer that was
down when the Mac started, and a rename over an existing file inside a
source fails to land in its store.
Paired devices answer a new DeviceFacts request with their hardware and
mounted volumes, under the ids their sources anchor to. The peer fetches
them on every sync beside the device summary and keeps them in the replica
manifest with its own receipt time, so an offline device still describes
itself as it last did.
volumes.list lists those volumes under their owner, mounted only while it's
connected, and devices.list fills in a paired device's hardware. The
overview folds replicas into their owner's volume bars the way local
sources fold into theirs. Another device's volumes show when they were last
seen and hide the local volume actions on desktop and mobile.
Volume classifications travel as names, so one a peer's build doesn't know
reads as unknown instead of failing the response.
AGENTS.md and CLAUDE.md flip their current direction from tearing down the
entries world to it being gone, and drop the ephemeral note T6.7 retired.
The locations page is deleted; the overview page that explained pins as
location rows becomes Add Sources and Pins, and the data model, library,
architecture, sync, and CLI pages describe pins as Space items. The React
explorer and normalized cache guides use sources and spaces for their
examples, the snapshot format doc points at the moved module, and
CONTRIBUTING tracks a source instead of adding a location.
The final-drop plan records FD5 with what still fails and why, the
convergence plan notes the old world is gone, and the registers move FD5 to
done with FDA ready.
FD4 of the final entries drop. One migration drops 29 retired tables inside
its own transaction, children before parents: with foreign keys on, dropping
a parent makes SQLite parse every child's constraints, and the first attempt
failed on sidecar, whose other parent entries was already gone. That attempt
also showed Migrator::up leaves SQLite migrations unwrapped, which is why
the migration opens its own transaction. Every library now converges on 14
tables. Proven on a copy of the live library, on a library the pre-drop
build left half-migrated, on a brand-new library, and on the live Mac
library after a backup; every dropped table held zero rows there apart from
the seeded kind lookup.
Locations leave with the schema: ops, domain type, entity, CLI domain, and
every consumer in the web and mobile clients. PathBar pins become
space-level Path items listed under Pinned, the sidebar's Places come from
a new paths.system_folders query, Add Storage tracks sources and validates
through sources.validate_path, device views list sources, and library
statistics count sources as source_count. Content kind statistics read the
source stores instead of a lookup table.
Fourteen entry-era models lose their sync registrations and Syncable impls:
entry, content_identity, sidecar, mime_type, the three media_data tables,
collection, collection_entry, location, tag, tag_relationship,
user_metadata, and user_metadata_tag. Entry's apply_state_change, tombstone
gating, and closure and directory-path rebuilds go with them, as do
backfill's entry hierarchy sort with its per-record self-referential FK
resolution and the sync helpers' directory-path payloads.
The engine survives for what still syncs: device, volume, space,
space_group, space_item, and audit_log, over the same transport, HLC,
watermarks, and dependency tracking the library-sync research plans to
evolve. A registry test now refuses any retired model that tries to
register again. Live capability audit passed: replica sources listed,
remote ops answered by titan, replica listings served, remote job
reporting responsive.
FD0 and FD1 of the final entries drop. The baseline is recorded in the
plan: the reference inventory, the 43 tables a fresh library still
creates, the registered sync models, and the pre-existing entry-era test
failures, with the live fleet standing as the fixture.
File::route_from_dependency is deleted rather than rewired: the only
emitters of file dependency events were legacy row-sync applies, and file
payloads have announced directly from the volume index since the tags
work. The ancestor entry-ID cache, the dead entries_for_content field, and
the SQL path resolver leave the indexer; pairing discovery totals read
source rows, with sources standing in for the retired location count; and
the indexing module docs stop describing a writer that no longer exists.
Library statistics counted the legacy tag table, which would have read zero
forever; the instance count now reads definitions across staging and the
stores, and the pre-open static count reads staging with the instance
recount behind it. tagging.mdx describes the built system instead of the
old design sketch, library.mdx stops claiming the library owns tags, and
the design doc carries its three reconciliations. The plans and register
record stretch 1 closed: delivery verified against titan, FD4's
preconditions banked, phase 4 landed.
The tags plan carries its execution record: P1 through P5 landed and
live-verified, the delivery ack waits on titan running this build, and P6
lists what remains. The final-drop plan drops FD2's migration machinery in
favor of the clean build, since every legacy tag table was verified empty,
and the register rows follow.
The R6 subsection carries the landed commits, the live dev-tools
database delivery proof, the scale measurement, and the follow-ons the
plan itself declares optional. Register row updated to the fleet's
actual state.
R7 ran against the live fleet on 4a690c84a: all ten stores validated
by direct SQL with zero dangling ancestry, the home store needed no
repair, offline cold-restart serves and searches all nine replicas,
byte reads and remote ops verified, and reconvergence transferred
nothing. Register row rewritten to current state. Also folds in the
Codex volume audit note and its research doc: Untitled is detected but
unmapped this session because hot-plug does not invoke the startup
mapping path.
Locations go in the breaking release without row migration or API
compatibility. Policies accept SdPath and persist a separate
{ source_id, relative_path } target resolved through the source layer;
SdPath gains no source variant; Space items own navigation. The
final-drop boundaries, teardown status, sync research, and core docs
pages now agree. Register brought current through 5a65d0998 and the
search refinement bar.
The source runtime reliability plan graduates from draft to the
executing plan its register row already tracks. Library-sync research
records the split-library state between the Mac and titan and proposes
the small library-state contract; locations research records the
revised direction of configured source subtrees with continuous
processing rules over pins alone, cross-referenced from the
entries-teardown T4.4 row. The working register carries today's landed
work: search phases, fleet statistics, storage replicas, the hashing
loop fix, and the first reliability slices, with the code baseline
brought current.
From a parallel session. Inset content is the default: navigation sits
on the window background and the explorer renders as an inset sheet.
Settings > Appearance > Layout retains the floating-panels mode. Both
modes share mounted content and stable preview hosts through a shell
geometry context, the layout preference propagates to other windows on
the same origin, and the sidebar footer returns to circular Activity
and Settings controls with the activity count and busy indicator.
Preview and Size view follow the sheet bounds, with Size thumbnails in
container coordinates.
From a parallel session. thumbs.generate accepts a file or an indexed
folder with missing, stale, and force modes, runs in resumable batches,
and versions video tiles against host FFmpeg availability so a tile
regenerated with better tooling replaces its predecessor in place. File
and folder context menus and the new sd thumbs CLI domain call the same
operation. paths.context now reports whether the exact path has a
committed source record, and the path bar surfaces it with a refresh
affordance. Coverage and resume semantics are documented in
docs/core/thumbnail-generation.mdx.
Phase 1 of docs/plans/2026-09-15-search-refinement.md. Every parameter
search.files accepts is now either honored or fails closed; nothing is
validated, logged, and ignored anymore.
Ordering and pagination live in a new pipeline module with a
backend-neutral candidate type, per R6 of the source-runtime plan: one
deterministic global sort before final pagination, equal keys settling
on case-folded name and record id so results stop reshuffling between
keystrokes. Library search sums each partition's full filtered match
count into total_found and contributes only the page window of
candidates to the merge, so a partition can never lose a result that
belongs on the page. The 200-result cap and the synthetic pagination
block are gone; the response reports the caller's window over the true
total (a "spacedrive" search now reports 496 matches instead of a
complete-looking 200).
Filter fixes: extensions compare case-insensitively, hidden entries
are excluded by default and included with include_hidden (advertised
in available_filters), and a missing timestamp fails a date filter
instead of passing it, so the unanswerable AccessedAt and IndexedAt
fields fail closed rather than silently matching everything.
Deletes dead code the plan condemned: sorting.rs and the unused
facets.rs duplicates, plus the legacy entry-era imports in query.rs.
Verified live: --limit 5 returns five with the true total, size and
name sorts order and reverse, offset pages advance.
The plan owns everything above matching, which 4bc442e43 made correct:
scopes become folder/source/library with Location retired as vocabulary,
sort and pagination get implemented rather than deleted, the filter
panel ships the set the daemon can answer honestly, tags follow through
the library-db path-set join, and the redundancy views get a phase to
verify and fix their mechanism, since they drive search.files with
filters the ephemeral path ignores. Register rows record the plan and
the landed search fixes.
PROJECT_STATUS.md now sits on the b653676e8 baseline: the sidebar,
path-status, thumbnail, host-media, and startup rows cite their landed
commits with live proofs still open, the finished docs rows moved to
Recent landed, and the multi-device commits since 50fdbad59 are
recorded. The drive-catalog row flags that packages/drives is missing
from the checkout. Adds the entries final-drop plan and its references
in the teardown, convergence, and locations plans.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
README and CONTRIBUTING updates, the working register brought current,
locations and data-model pages reconciled, redundancy and overview
polish, media view tweaks, regenerated schemas, and the new Spacedrive
icon asset.
Library loading no longer dispatches a filesystem discovery pass. The
desktop requests one idempotent pass after the connected shell
renders, a bare daemon initiates nothing, and sd index start
--defaults gives CLI and server operators the same explicit path.
Restored volume snapshots prevent repeat full walks.
A machine-scoped registry discovers host FFmpeg and companion tools,
surfaced in Services settings. The thumbnail service uses them for
video posters and on-demand scrub sheets: thumbstrips bake per request
into the volume hot tier and serve over the local HTTP servers with
immutable content-versioned URLs, wired to the timeline scrubber.
The path bar reads a typed path-context query backed by the daemon's
source, volume map, watcher, persistence, and pin state, replacing the
location/index-mode indicator. Containment resolves on the daemon with
path components after volume alias normalization; a pin no longer
implies indexing or watching, and the docs say so.
One library scope, flat daily destinations, Places computed from
system folders without persisting them as locations, canonical rows,
and an Activity/Settings footer. Topology groups retire; their pins
survive seed convergence. Replica sources render under their owning
device and route into the explorer with that device's slug.
Replaces the Mintlify setup (mint.json, docs.json, snippets) with a
standalone Fumadocs app in docs/: Next.js app router, meta.json nav per
section, its own bun install. Run with `just dev-docs`.
Content pass alongside the migration: core architecture, data model,
CLI, and networking pages rewritten around the daemon/source-store/
library split, new product-direction.mdx defining the product boundary,
new drive-catalog plan, and the physical-drives plan updated to
reference it. Whitepaper page retired.
SourceConfig is the typed shape of the source row's config column,
parsed with defaults so old rows and future fields both survive.
sources.update renames a source or changes its capture policy, with
the one asymmetry named: widening dispatches a walk for what the rules
skipped, narrowing removes nothing because hiding is the lens's job.
Tracking only ever widens the policy, so a casual re-track cannot
demote an archival source. The locations-demoted doc gains the policy
taxonomy this settles: capture on the source, attention on the
location, display in the lens.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Groups carry the vdev tree and minimum member count, since "which boxed
drives make the pool, and how many suffice" is the question ZFS labels
cannot answer with the drives unplugged. zpool membership moves from D3
to D1 because the observation window closes at export. The runbook
gains the TrueNAS config backup, the encryption check, and the clean
export step.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
What the store captures (verified against stat on titan), the deploy,
the per-dataset order and pass/fail gate, the pre-export steps, and the
known issues with workarounds.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Drives under volumes: an append-only observation log, an evidence
ladder for identity (WWN, GPT disk GUID, tiered serials, SMART
trajectory), and drive rows as conclusions over it. Groups for pools
and containers, human adjudication as the strongest evidence tier.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Nothing produces a row in `entry` after this. The walk fills the volume
index and its source store, which is what the arena has been doing since
the storage map landed; the persistent half of the indexer was a second
writer to a second store that no source ever read.
Gone: IndexPersistence, and with it the branch in every phase. The state
machine is Discovery then Processing, since Aggregation existed to bubble
directory sizes the arena now maintains incrementally, and
ContentIdentification is ContentIdentityJob's job against the source store.
DatabaseAdapter, PersistentEventHandler and the location watching that fed
them go too, so ArenaWriter is the only change handler and the walk and the
watcher differ only in Notify.
DatabaseStorage was 1230 lines of entry writes wrapped around three things
the arena actually uses. Those are ops::indexing::metadata now: EntryMetadata,
extract_metadata and get_inode, plus is_hidden_path. delete_subtree moves onto
the entry entity, which is what it operates on and what it dies with.
locations.rescan, locations.enable_indexing and locations.triggerJob dispatched
the deleted indexer or its media jobs, so the ops and their UI go. Reindex and
Run Now are gone from the inspector; the job policy toggles stay, since they
outlive the ops that read them. Adding a location no longer dispatches a walk,
which is the point: the drive is already mapped.
The entry tables and their readers are still here. That is the next commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
P4 was titled "Delete entries" and quietly assumed locations went with them.
They don't. A location stops owning records and becomes a policy over a path:
the subtree kept at full fidelity and watched.
That's the Retention.covered parameter the storage map already has and is
currently filling from cache.sources() roots because there was nothing better
to ask. Locations are the better thing to ask, so this gives an existing
parameter its owner rather than adding a mechanism.
The row loses entry_id, device_id, volume_id, scan_state, last_scan_at,
error_message, the two counts, index_mode and job_policies. Each of those
lives somewhere that already has it, and the arena's rollups are the honest
version of the counts since a stub can say 400 GB was summarised and a
persisted integer can't.
Also covers the sweep across the ~64 files that reference entities::entry,
and flags triggerJob, import and export as the three ops that don't survive
mechanically and want deciding on their own merits.
The store's filesystem schema had one model with six fields, so a record
could say how big a file is and not what took the picture. The old library
had image_media_data and video_media_data and both were lost in the move.
Facets are schema-driven, so these are declarations rather than machinery:
generate_ddl emits facet_image, facet_video and facet_audio on create, and
diff_schemas adds them to a store that predates them. Audio came along with
the other two, being the same kind of fact and already shaped in the tree.
No writer yet, enrichment is still parked. What this buys is that the shape
can only be decided once.
The search contract named title = "name" on a model with no name field. A
filesystem record's display name is its filename, which the walk writes to
record.title, so the contract points at the record instead.
The archive engine kept its own registry in registry.db while filesystem sources
lived in library.db, so there were two lists of sources that could not see each
other. sources.list unioned them and said so in a comment.
Now the engine holds stores and adapters and is handed a SourceRef. Which
sources exist is a question for the library, with one answer: the sources table,
read back through ops/sources/registry. Creating a source mints the id and
writes the row here; the engine is told what to name the store. A sync answers
with what the run did and this side records it, so a status can't be written
without the row that shows it.
registry.db and its data_types table are deleted, unread by anything.
Two moves on disk. Adapter stores live under SourceDirs beside every other
source's and are named by uuid without dashes, the way filesystem stores already
were. Installed adapters moved to <data>/adapters, since an adapter is installed
once for a machine rather than once per library. <library>/archive/ is no longer
written.
SourceInfo::from_row replaces the two constructors that built the same shape
from different halves. Existing adapter sources do not carry over: their rows
were in registry.db and their stores are under the old path.
content had 0 rows because nothing ever called the two functions that fill it.
ContentIdentityJob does: it claims files a source holds and has not identified,
hashes them a few at a time, and hands the identities back through the same
writer as everything else, so a content row can never land before the record it
points at.
Dispatched at LOW behind the walk that produced the records and behind the drive
map, which is the order the storage map doc set out: what someone kept, then the
rest of the machine, then the bytes.
files_needing_content rebuilds each path from its parent and its name, the same
join Ledger::load uses. P2.6 means the path is not a column to read, and a file
whose parent has no path is skipped rather than guessed at.
Only the sampled tier runs. An integrity hash is a full read of every byte on
the drive and belongs behind a decision that needs one, which is the rule the
ladder in sd_store::content exists to keep.
set_content_identities commits a batch in one transaction; the single-file
version now shares its statement. A file that cannot be read keeps its empty
content_id and comes back on the next pass, and a pass that identifies nothing
has found the tail of the queue and stops rather than looping on it.
Nothing reads the rows yet: duplicate detection still queries content_identity
in library.db.
Three destinations had to agree about every change — the arena, the source
store, the clients — and each call site wrote all three by hand. They had
already drifted once: a move updated the arena and the store correctly and told
nobody, because creates emitted inline while moves went through a hook that was
handed the path the file had just left.
MemoryAdapter is now ArenaWriter with one entry point. apply(Seen) resolves
identity, mutates the arena, and answers with the ArenaChange that both the
store write and the client event are derived from, so "written durably but never
announced" stops being representable. The walk and the watcher differ only in
Notify: every change, one event per batch, or silence for a drive being mapped
in the background. run_ephemeral_processing_static lost its own copy of all
three writes and is a third of its former size.
Two things fell out of it:
- Mutation::Unchanged. A re-walk announced every entry it re-observed, which
meant serialising a million files nobody touched. The store still hears about
all of them, which is what keeps a sweep from condemning them.
- A recount now reaches clients. It was an arena write with no way to tell
anyone, so a summarised directory whose size changed was correct on the next
query and invisible to anything already on screen.
IndexPersistence had no production callers and goes, with PersistenceFactory and
DatabaseAdapterForJob. The one query the latter still served moved to the change
detector, which was its only caller. ChangeHandler stays for the database
adapter, minus emit_change_event, which the arena can no longer be asked to do
separately.
The Analyzer wants a live sunburst of a volume, and the thing every tool doing
this today cannot do is the one thing that is already free here: the arena
maintains subtree_bytes up the ancestor chain on every add and remove, so a
file landing anywhere corrects every ring outward with no rescan. The map being
live is a property of the arena that nothing surfaces yet.
What is missing is the map. Measured on one 990 GB drive: 11,131,151 entries
exist, 1,678,592 are mapped, 271 GB of 866 GB used. Two independent causes.
Nothing walks the volume at all, so /Applications and /Library have never been
enumerated. Rules prune the rest, which is where ~/Library's 231 GB went.
The reframing that makes it affordable is that traversal is unavoidable and
retention is not: a size cannot be known without enumerating, so the policy is
only ever about what to keep. Measured: 118,859 directories within seven levels
of the volume root, 54 MB, against 5.0 GB for all 11.1M entries. So home at
full fidelity plus a complete seven-ring structure is about what the daemon
holds today for two rings and a fifth of the disk.
Records the stub design, why delta tracking cannot keep one honest, the
dirty-and-recount that can, the queue order that puts a person's own files
first, and the per-volume setting. Phases at the end.
"Medium" named the abstraction after its own exception. Every mount the volume
manager can see gets a record, network shares included, so a filesystem with no
volume is a fallback rather than a co-equal case. The word for a drive here is
volume, and this is its index.
MediumSlot is VolumeIndex, MediumKey is VolumeKey with Id and Path variants
instead of Volume and Root, track_medium is track_volume, and the layout moves
from mediums/<id>/ to volumes/<id>/. The derived namespace uuid is unchanged,
so ids are the same; only the directory holding them is renamed.
Nothing else moves. A volume index is still the whole drive's map, still keyed
by the drive rather than by any registration, still what a nested source shares
instead of forking.
A machine needs full local search over every file and a size figure the
analyser can trust. Neither is the same question as what someone chose to keep,
and the sources migration collapsed the two.
A partition is keyed by source id today and routed by longest prefix, so a
source nested inside another forks rather than narrows: two partitions, two
snapshots, two ledgers, and two uuids for one file. A tag applied through one
root is invisible through the other, which is the second-namespace problem
between two filesystem sources.
P2.7 keys the arena by medium instead. One drive, one arena, one snapshot,
whatever is persisted from it. A source says persist this subtree and owns a
data.db, so nesting is ordinary because the nested things are no longer the
same thing. The arena mints identity once per medium and a source adopts what
is already there, which is both what stops the fork and what makes creating a
source over mapped files fast.
A path under no source still has a uuid and still cannot carry an assertion,
since assertions live in a source store. Choosing to keep something is what
gives it somewhere to keep things.
Also records the completeness cost. An internal volume walks with default
rules, so /Library, ~/Library, node_modules and gitignored files never enter
the map: 262 GB indexed against roughly 900 GB used on a 1 TB drive. Rules
become view-time lenses and the walk records what it can read, which takes a
2.1M entry map to somewhere between 5 and 8M, and 1.0 GB resident to between
2.5 and 4. collection_flags already has room for a filtered bit.
Settles the nesting question in the Open register. Both live documents were
right about their own half: one filesystem owns one record namespace, and that
namespace is the arena rather than the source.
external_id is the full relative path on every record and UNIQUE(type,
external_id) holds a second copy, which is most of a 2.1M-record store. The old
library stored a path once per directory and derived the rest from parent_id
and name. Dropping that was a mistake.
Measured by replaying the live store through the proposed schema, both
databases vacuumed, same 2,123,008 records. Record side 751 MB against 423 MB,
whole database 896 MB against 568 MB, 443 bytes per record against 281. The
index it removes serves path lookup; the one that replaces it answers list this
directory's children, which is the query the UI makes.
Lookup stays two probes because directories keep their paths: one into
directory_path for the parent, one into (parent_uuid, name). Renames stop
rewriting every descendant record and rewrite descendant directories only.
The replay found 0 unresolved parents, so the existing store can be backfilled
in place rather than re-walked. That matters because a re-walk is where the
surviving created_at values would go.
Ties P2.5 (2) to this: the tree is currently a fidelity bug and becomes the
addressing mechanism, so the two land together.
Compared the new per-source store against the last pre-sources library on the
same machine: 2.1M records against 23,157 entries, and the new one holds less
about each file. Three causes, tangled together.
The walk discards metadata it already stats. DirEntry carries five fields and
run_ephemeral_processing_static hardcodes created, accessed and permissions to
None, so birth time is gone for 2.1M files. The stat was already paid for.
The bulk walk does not build the tree. Parent linkage needs the ledger to have
resolved the directory first, which the incremental path satisfies and the walk
does not: 1,349 of 2.1M walked records are parented against 19,769 of 19,856
written by the watcher. 240,122 directories exist and 949 know their parent.
Media facets were never declared. The schema has one model with six fields
where the old library had EXIF, GPS, camera and codec tables.
Sidecars move into the source store. A sidecar is keyed by content uuid and
content now lives per source, so a row in library.db is a foreign key across
database files with nothing enforcing it. sidecar splits: content_sidecar in
the source store, sidecar_availability staying in the library because it is a
claim about a device. Answers the open question in the extensibility plan.
Also recorded under Open: external_id is the full path on every record and the
unique index holds a second copy, so 518 MB of a 1.01 GB database is paths,
while idx_record_parent costs 21 MB and is 99% empty.
JobStarted was emitted for every job while JobCompleted, JobFailed and
JobCancelled were all gated behind should_emit_events. So any job that
opted out of events still announced a start and never announced an end,
and the client had a row nothing could clear. That is every ephemeral
browse, since should_emit_events is should_persist unless the job is
indexing a whole volume.
Gating the start the same way makes an event-free job invisible end to
end, which is what the flag was for.
Also reorders the completion arm. Clients refetch the job list when they
see a completion, and the job was removed from running_jobs after the
event went out, so a refetch that landed first got the job back in the
answer with nothing further coming to clear it. Reading the output,
removing, then announcing closes that window.
Tracking a volume did not index it. There were three flows and none of
them was one gesture: the context menu offered Track and Index as
separate items, and AddStorageModal tracked the volume and then created a
Location, which is the model being deleted.
volumes.track now calls the same track_and_index that sources.track does.
The medium and the index stay separate underneath, since a drive can be
tracked with nothing on it indexed and a folder can be indexed on a
volume nobody tracks; what converges is the click.
The shared body matters more than the saving. Both entry points forget
the same two steps otherwise, seeding from the snapshot and clearing what
the new pass will not revisit, which is exactly what sources.track was
missing when it was written separately.
An external mount indexes unfiltered, since a drive being archived wants
completeness rather than rules that hide files on purpose. An unmounted
drive tracks without walking and gets its map when it returns.
Index Volume becomes Re-index Volume and only shows for tracked drives,
since the first index is no longer something to ask for.
sources.track skipped two steps volumes.index does. Without
ensure_restored a partition that never restored is barred from saving
over an existing snapshot, so tracking a root that already had one would
index and then fail to persist. Without clear_for_reindex, entries from a
previous pass that this one does not revisit linger in the arena as files
that no longer exist.
Also writes down how it differs from volumes.track, since the names
collide on the word rather than the meaning: volumes.track marks a medium
as belonging to a library and indexes nothing, this makes an index of
what is on one. volumes.index is the older overlap and retires once
callers move over, because it takes a fingerprint and can therefore only
ever mean a whole drive.
A filesystem source used to appear as a side effect: browsing a directory
dispatched a walk and the walk registered whatever partition it landed
in. sources.track is that intent made explicit. Whole drive and
folder-within-a-drive are the same call with a different path, since the
volume resolves from the path and the source anchors to it.
Indexing starts immediately rather than waiting to be asked, and the
action returns as soon as the row exists, because registration is instant
and a walk over a drive is not. The job then reports against a source
that already exists.
sources.list serves both kinds now, so the sidebar stopped needing a
group per registry: filesystem rows render as walked roots with their
attachment state, adapter rows as before. The union is transitional,
reading filesystem rows from the sources table and adapter rows from
registry.db, and folding the second into the first is the rest of P3.
Content ids stay out of the walk on purpose. The walk is fast and worth
keeping fast, and records_needing_content_identity is already the query a
separate pass would drive. Nothing generates them yet, which the plan now
says plainly rather than implying the sampled tier is wired.
The arena and the store were minting separate uuids for the same file.
MemoryAdapter generated one inline while Ledger::resolve minted its own
for the record, so every file had two identities and nothing joined them.
Tags hang off the arena's; record_overlay assertions, and the AI
annotations that follow, hang off the record's. Indexing at scale before
fixing this would have baked the split into every row.
The ledger assigns identity now and the arena writes what it is told.
SourceStore::identify resolves a batch and answers with record uuids, and
it is the same message that takes the batch in, so there is no second
call that could disagree. One round trip per batch rather than per file:
resolution is a hash lookup and the reply is sent before anything
commits, so the cost is the queue rather than a transaction.
Volume indexing used to defer uuids entirely and no longer needs to,
since the ledger has one whether or not anything asks. A partition with
no store still mints for the session, because there is nothing durable
for it to agree with.
entry_uuids stays in the snapshot and stops mattering: it now holds the
same uuid the ledger does, so a discard costs a re-identify rather than
an identity.
Native dependencies, job presets, plugins and sidecars are four things,
and separating them is most of the work. Checked against the tree first:
the codecs were already unbundled by the install-size plan, the SDK is
already a plugin SDK missing only a manifest and a lifecycle, and
SidecarKind::Thumb already exists content-addressed. Less is missing than
the dump assumes.
The preset is the primitive worth building, because it is the same shape
as the workflow row the filesystem intelligence plan already needs:
durable, parameterized, executable, logged, with its reasoning attached.
One table, two consumers.
Draws the line that keeps it safe: a model composes registered job kinds,
a plugin adds kinds. If a model can author the command rather than the
parameters, durable execution and monitoring stop being features and
start being a liability.
Answers the sidecar question as placement rather than design. The hot
tier and the thumb sidecar are both already specified and do not compete;
what is open is that the sidecar tree is library-scoped, which is wrong
for an archival drive whose previews should travel with it. And the
location-or-source line is source, since locations are being deleted.
Keeps the GPUI plan where it is: Photos is the right wedge, and a second
full interface should follow the teardown rather than race it.
P3 is partly done: sources are a table, T2.0b is ruled, and the two seams
it left visible are written down rather than left to be rediscovered.
P4 gets the structural answer the move bug pointed at. Each
ChangeHandler method updates arena, store and event bus by hand with
nothing forcing them to agree, and they drifted: a move wrote the first
two and emitted nothing. An arena mutation should produce a change record
that the store write and the client event are both derived from, which
makes that failure unrepresentable rather than fixed.
It also answers what such a record carries, since the same bug showed it:
both the destination and the vacated directory, because a client renders
directory listings and a move invalidates two of them.
Belongs in P4 because MemoryAdapter is being rewritten there anyway and
IndexPersistence dies with the entries world that needed it. Earlier
means building the seam twice.
