* Resume interrupted replica fetches from a .part file
A replica transfer now lands in <source>.<generation>.<blake3 prefix>.part
and stays there when the stream breaks or the daemon stops. The next
attempt asks the owner for the artifact's tail from the part's length over
the new FetchSnapshotFrom and FetchDatabaseFrom requests, whose header
still names the whole artifact; a header that names different bytes
discards the part and starts over, and the finished file is validated
against the header checksum before it replaces the live replica as before.
Parts are removed only when the owner's listing no longer carries their
generation. A nonzero offset serves the on-disk snapshot without the
pre-fetch save, so a resumed copy finishes against the same bytes.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Cap replica transfer bandwidth with a persisted daemon setting
replication.max_bytes_per_sec in spacedrive.json bounds every replica
artifact served and received through one token bucket (zero is
unlimited); the cap applies to transfers already in flight. It is set
through config.app.update and sd config set replication.max_bytes_per_sec
(bytes, or K/M/G suffixes), read back through sd config get and show.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Pause and resume replica fetching
sd sources replication pause stops this device starting replica
transfers and halts any in flight at its next chunk, keeping the partial
file; resume continues from it and runs a replication pass against every
connected device at once. The switch is replication.paused in the daemon
config, so a paused daemon stays paused across a restart, and a pass that
runs while paused still refreshes listings, owner facts and fleet totals.
A pause does not count as a fetch failure, so no backoff accrues.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Show replica fetch progress in sources list, status and events
Every fetch in flight is tracked with bytes on disk, total and a recent
rate. sources.list carries it on the replica's row (or a replica_fetching
row for a first copy that has no share yet), core.status and the new
mounts.replication_status query summarize it with the pause switch and
cap, and the daemon emits a typed ReplicationProgress event about once a
second per transfer plus a finished one when it stops. sd sources list and
sd status print the figures; ts-client types regenerated.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Document resumable replica fetch, the cap, pause and progress
Source replication gets its own section in networking.mdx with the .part
contract, the wire change, the cap, the pause switch and the progress
fields; file-sync.mdx points there; the replication plan marks its cleanup
item superseded on the replica side and distinct from the restart refetch;
PROJECT_STATUS.md gains the register row.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Keep the part when a resumed request gets no header; cap sidecar pages
A resumed request that fails outright (dropped link, or an owner too old
to know the offset variants) says nothing about the bytes on disk, so the
part now stays and the whole-file retry lands on the same name. Sidecar
pages count against the bandwidth cap on both ends, as the setting's
contract says.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Add the replication field to the test harness config
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Check the pause switch after the cap's wait, not before
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Networking: serve streams on connections a protocol dials
The connection cache is shared between the dial side and the accept side,
so once a protocol dials a peer, the peer's own send path finds that
connection and opens its streams on it. Nothing read those streams: only
connections the event loop accepted, or those the reconnect loop handed
over, had a stream handler. The peer's request then sat unanswered until
its 30 s timeout while requests in the other direction answered in
milliseconds. This is the reverse RegisterDeviceRequest timeout SPAC-12
saw after join-remote.
get_or_create_connection now hands every connection it dials to the
event loop as TrackOutboundConnection, which spawns the same stream
handler an inbound connection gets. Messaging, sync, and job activity
pass their event loop handle; pairing keeps its own connection handling.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: the device that joins a library is the one that backfills
Every sync service started Uninitialized and backfilled from the first
connected partner, whatever state it already held. When the Mac Studio
joined the MacBook's library the MacBook was the one with a partner in
its device table, so it went Uninitialized -> Backfilling{Studio} and
applied the empty copy's snapshot over its own rows (LWW on the
deterministic default-space UUID). The Studio never backfilled because
the reverse registration had timed out and its device table held only
itself.
A copy of a library created from another device's library is the only
thing that may start Uninitialized. create_library_with_id now writes an
initial_backfill_pending marker into sync.db; PeerSync reads it on open
and starts Ready otherwise. A Ready device never requests a snapshot
(SharedChangeRequest.include_snapshot = false on catch-up) and repeated
catch-up failures no longer escalate to a full backfill. A copy still
waiting for its own first backfill refuses to serve as a join target.
The join carries the rule explicitly: RegisterDeviceRequest has
needs_initial_state, and RegisterDeviceResponse returns the responder's
own device record so the joiner's table is complete when the command
returns, replacing the spawned reverse request. sync_setup fails
instead of reporting devices_registered: true when that record is
missing.
Catch-up completion is recorded per peer in sync.db even when no rows
arrived; the Ready loop catches up once with a peer it has no record
for and no longer re-polls every peer whose watermark is 60 s old. A
missing watermark means never synced, not ancient, so the forced full
sync warning only fires for a real watermark past retention.
library_join_test pairs two daemons for real, joins Bob to Alice's
library, and asserts Alice's rows are unchanged, Bob holds Alice's
device row and spaces, and Alice runs no further catch-ups.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: test that catch-up with an empty peer runs once
Alice holds a volume, Bob holds nothing of his own, so Alice never
writes a per-resource watermark for Bob. Before the per-peer catch-up
record, that made Bob a new peer on every loop iteration and Alice
caught up with him forever (2 -> 5 sessions in 16 s on 5e285d3). The
test asserts the session count stops moving and the record exists.
Both new suites join the xtask integration list.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Library info: fleet totals count the same devices as device_count
add_device_summaries added every cached peer summary to a library's
statistics, whatever library the peer belonged to, and the summaries
carried no source count. A joined device therefore reported the member's
3 million files next to Sources 0 and Devices 1.
Summaries now carry the device's own source_count, and only summaries
of devices in the library's device table are added, so files, content,
sources, and capacity describe the members that device_count counts.
The CLI labels the block "Statistics (all library members)".
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Docs: describe the pairing and library join flow as it runs
pairing.mdx and devices.mdx described a confirmation prompt that does
not exist and pointed at `sd devices list` for pairing status, which
reads the library's device table and does not show a peer until it
joins the library. The CLI overview lacked the global `--device` flag,
and the sync setup walkthrough never named `library sync-setup
--action join-remote` or `--node-id`. All four now describe the flow
SPAC-12 ran, and library-sync.mdx gains sections on joining, catch-up,
and fleet totals. One PROJECT_STATUS register row.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: catch up once per connection session, never serve a snapshot while pending
Review of #3110 found two gaps. The per-peer catch-up record was never
cleared, so a peer that changed rows while disconnected was never caught
up with again after reconnecting (the live broadcast only lives in the
in-memory retry queue). The record is now cleared when the sync service
starts and when the peer's connection is lost; ConnectionEstablished is
not used because the event loop emits it per accepted stream.
The SharedChangeRequest handler attached a full snapshot without
checking the first-backfill marker, so two copies that both joined from
the same member could seed each other. It now refuses the snapshot while
pending, and the register handler reads the marker from sync.db directly
when the library's sync service is not up yet.
Also: a failed backfill or catch-up left active_backfill_sessions
incremented; the mock transport no longer lists a peer as a partner
before its sync service is registered; the catch-up test now starts
Alice Ready (marker cleared) and Bob pending, the shape of a real join,
and asserts exactly one catch-up and one backfill.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: move past a peer that refuses a snapshot, fail the guard closed, refresh stale catch-ups
A joiner whose chosen peer refused the snapshot kept choosing it: every
partner was scored has_complete_state: true and RTT is only recorded on
success, so the order never changed. A peer whose backfill failed now
scores without complete state until a backfill succeeds, so a connected
member holding the library is tried next.
The snapshot guard read an unreadable marker as "holds state"; it now
refuses, since a wrong refusal only delays a join.
A broadcast whose retries ran out while the QUIC session stayed up had
no recovery under the once-per-session rule. The Ready loop refreshes a
catch-up record older than ten minutes; with per-peer watermarks an
empty refresh is two small requests.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
sd file rename (--to or rules), undo, organize, flatten, archive, extract, attributes, link and trash list|restore|empty, plus sd job journal and --remove-extras on merge. Every one shows its findings and plan before the prompt through one run_with_preflight.
files.delete takes a Duplicates target: a scope with keep first (walk order) or a list of copies to keep with every other copy of their content going. The job reads each surplus copy and its keeper in full before removing, records the hashes it learns, and checkpoints the compare cursor so it resumes. Preflight validates the scope and keep list and previews keepers as skips and surplus copies as deletes.
Store gains duplicated_contents_beneath and copies_beneath. files.duplicates is registered as a core query and reports each copy's SdPath. CLI: sd file dedupe with --keep, --keep-under, --min-size, --dry-run.
Two opt-in methods per action, validate: and preview:, over the exact
input the action takes. Validation answers findings with stable codes and
the facts of the execution, runs again at dispatch and refuses on an
error. Preview answers an FsPlan read from the index, kept under a handle
so files.directory_listing can serve the directory as it would look after
the operation.
Copy, move, merge and delete answer both. Merge is new: a resumable job
that recurses into matching folders, reads both sides before skipping a
duplicate, applies a per-leaf conflict policy, leaves conflicts alone and
can consume its sources. Delete's validation counts, across every store,
which files are the last copy of their content.
The ValidationResult and ConfirmationRequest hook is gone, and so is the
CLI's local conflict detection. file copy, merge and delete render
preflight before the prompt and take --dry-run.
files.delete takes a comparison as its targets alongside a path list, so
the client sends five fields and the daemon derives the set from the
index as the job runs. The job drains the matcher the compare query
pages, checkpoints its cursor after each batch, and for `both` reads
whichever side lacks an integrity hash before removing a file, writing
what it learned back to the store. Pairs whose bytes differ, or whose
copy in B is gone by then, stay and are reported. The CLI gains
`sd file delete A --against B --show both` behind a y/N prompt, plus the
plain paths form.
Compare matched on content uuids, which re-derive from the integrity
hash once a copy is verified, so a copy verified on one side stopped
matching its unread twin. It now matches on the sampled hash every store
keys a content by, with integrity hashes deciding where both sides have
them; FsEntry carries both hashes for that.
Left and right only meant first and second argument, and nothing on
screen said which folder was which. "Same" meant different things per
mode, "changed" implied a direction compare can't know, and "location"
collided with the retired locations.
The output now names both folders as A and B up front, says how files
matched, then counts Only in A, Only in B, In both, and Different (by
path only). --by location becomes --by path and the --show values
follow, as do the API types, generated TS types, skill, and docs.
Single-side listings get Size and Modified columns under a title naming
the set, and --help explains each value.
files compare printed one page and handed back an --after cursor to
paste. The CLI now follows the cursor itself: totals first, then each
set's files streaming one per line as pages of 5000 arrive. --limit
caps the total and says N of M listed when it cuts a set short, a
closed pipe like head ends the walk quietly, and --format json emits
one document holding the totals and every entry. The skill doc
follows.
paths.compare answers what differs between two folders from the source
stores at query time. By location it merges both sides in index order
into only-left, only-right, changed and same. By content it streams one
side and asks the other side's stores which content ids they hold, so a
renamed or moved copy still matches. Pages resume from a cursor and the
first page carries every set's count. Unhashed pairs fall back to size
and mtime and get counted, so a page says how much rests on that.
sd-cli file compare wraps it.
On the store side, files_beneath takes a Start, including one that
resumes a nested store at its first directory, and contents_beneath
answers the content lookup a batch at a time. Working out which stores a
path reaches moved into ops/paths/reach.rs, shared with search.media and
paths.context.
No catalog.db: the P7 row, the filesystem-intelligence plan, the previs
and the design docs now treat cross-source questions as store queries.
The first client to connect starts automatic discovery: it tracks the
system volume, adds the home folder as a source and walks every attached
drive. For a library meant to hold only what you add, like one whose data
directory lives on the drive it catalogs, that indexes the Mac into it.
sd start --no-default-sources, and the same flag on restart and on
sd-daemon, now leaves those three out. The pass still restores snapshots,
heals coverage for sources already tracked and hashes them, so a source
you added keeps working across restarts. sd-server keeps the defaults.
The error variants, the permissions flag, the sync fixtures, and a long
tail of comments still named locations and entries after both left the
product. LocationNotFound goes from the query, action, and CLI errors;
can_manage_locations becomes can_manage_sources; ChangeConfig's
location_root becomes root; and the sync tests key their watermarks,
checkpoints, dependency parsing, and messages on models that still sync.
The transaction manager's stubbed pre-HLC methods and the bulk operation
types only they used are removed, and the spaces invalidation arm stops
matching a location resource nothing emits.
File transfers and deletes describe their allow-list as the source roots
it is. The CLI no longer points a new cloud volume at sd location add. In
the clients, mobile loses its location search scope and empty Locations
section, an unused overview card mock is deleted, and empty states and
copy say folder, place, or source instead of location.
The word marked a contrast with the persistent entries index, and that
index is gone. ops/indexing/ephemeral folds into ops/indexing, and the names
now say what each piece is, keeping the in-memory arena apart from the
durable registry and stores:
EphemeralIndexCache, ephemeral_cache() -> VolumeIndex, volume_index()
VolumeIndex (one drive's slot) -> Partition
EphemeralIndex -> Arena
EphemeralEventHandler -> FsEventHandler
watch_ephemeral / unwatch_ephemeral -> watch_root / unwatch_root
IndexerJobConfig::ephemeral_browse -> IndexerJobConfig::new
set_ephemeral_index, from_ephemeral -> set_arena, from_arena
ops/search/ephemeral_search -> arena_search
core.ephemeral_status / ephemeral_reset -> core.index_status / index_reset
The CLI's index subcommands become status and reset. Status output loses
its always-empty legacy fields, EntryRef loses an id that was always zero,
the indexer output loses a field nothing read, and module docs that
described browsing outside managed locations now describe the volume
index. On-disk snapshots keep the ephemeral.snapshot name: renaming the
file would orphan every snapshot and re-walk every drive. The job system's
ephemeral jobs, Spacedrop's ephemeral shares, and UI state that is never
persisted keep the word, since it means what it says there.
Search reports one index, so IndexType and the index_type field are gone,
along with the Locations filter kind, the locations filter and facet, the
constructors that only existed to stamp Persistent, and the Entry-based
enhanced output. Tags and hidden files are advertised filters with no
caveat, since both backends apply them.
Fifteen Event variants that nothing emits are removed: the Entry events,
FsRawChange, the location-keyed Indexing events, and the legacy location,
thumbnail, file-operation, and files-modified events, with their helper
types. events.list now derives its names from the enum instead of a list
that had drifted from it. The CLI printer, the Tauri and ts-client default
subscriptions, and the client-side filter types follow, and an uncompiled
stale copy of type_extraction.rs goes with them.
FD4 of the final entries drop. One migration drops 29 retired tables inside
its own transaction, children before parents: with foreign keys on, dropping
a parent makes SQLite parse every child's constraints, and the first attempt
failed on sidecar, whose other parent entries was already gone. That attempt
also showed Migrator::up leaves SQLite migrations unwrapped, which is why
the migration opens its own transaction. Every library now converges on 14
tables. Proven on a copy of the live library, on a library the pre-drop
build left half-migrated, on a brand-new library, and on the live Mac
library after a backup; every dropped table held zero rows there apart from
the seeded kind lookup.
Locations leave with the schema: ops, domain type, entity, CLI domain, and
every consumer in the web and mobile clients. PathBar pins become
space-level Path items listed under Pinned, the sidebar's Places come from
a new paths.system_folders query, Add Storage tracks sources and validates
through sources.validate_path, device views list sources, and library
statistics count sources as source_count. Content kind statistics read the
source stores instead of a lookup table.
tags.create stages a definition in library.db until first application adopts
it into a store; apply resolves targets through the arena with the stores as
the floor, so a cold daemon and a detached source both answer. Every claim is
stamped by a real HLC seeded from DeviceManager; the library-id-as-device
placeholder is gone. File targets carry the content key when the store knows
it, so tagging a copy reaches the bytes without the client choosing.
Writes against a remote-owned source author their rows durably in an
assertion outbox and deliver to the owner over remote ops when it answers:
sources.assertions.merge commits before replying, the reply is the ack, and
redelivery is absorbed by the assertion primary key. Pending claims overlay
replica listings immediately.
Reads decorate from the stores everywhere: directory listings on both the
arena and store paths, files.by_id, files.by_path, replica listings, and
files.by_tag, which no longer needs a loaded arena. File::announce decorates
its payload too, which is what makes tagging reactive: the event used to
carry empty tags and the client faithfully rendered that.
Deleted: TagManager, TagClosureService, TagConflictResolver,
TagUsageAnalyzer, TaggingFacade, UserMetadataManager, ops/metadata, the
validation module, and the semantic-tag field zoo. Tag is uuid, path, name,
color, icon. The XOR-era UI gates (the needs-indexing toast and the
inspector's isEphemeral message) are gone; the one honest refusal left is a
path outside every tracked source. The CLI gains tag unapply and delete. No
data migration: the legacy tables were verified empty, and their entities
wait for FD3 and FD4.
From a parallel session. thumbs.generate accepts a file or an indexed
folder with missing, stale, and force modes, runs in resumable batches,
and versions video tiles against host FFmpeg availability so a tile
regenerated with better tooling replaces its predecessor in place. File
and folder context menus and the new sd thumbs CLI domain call the same
operation. paths.context now reports whether the exact path has a
committed source record, and the path bar surfaces it with a refresh
affordance. Coverage and resume semantics are documented in
docs/core/thumbnail-generation.mdx.
Library loading no longer dispatches a filesystem discovery pass. The
desktop requests one idempotent pass after the connected shell
renders, a bare daemon initiates nothing, and sd index start
--defaults gives CLI and server operators the same explicit path.
Restored volume snapshots prevent repeat full walks.
Three gaps closed in the remote operation line:
Pairing completion now broadcasts ConnectionEstablished from both the
initiator and joiner, so peer mounts and job activity subscribe the
moment a device pairs instead of waiting for the next reconnect.
SubscribeLogs takes a device target. The local daemon opens a log
subscription on the paired device over remote-ops — now an enum
protocol with a streaming variant — and forwards its log bus frames
down the client connection, filtered on the serving side.
'sd --device titan logs follow' tails titan live. LogFilter matching
moved onto the type so both paths share it.
'sd job remote --watch' redraws every two seconds, which closes the
loop on remotely dispatched work: dispatch with --device, watch it run.
Remote job rows also carry updated_at now, and the sweep drops
non-terminal rows that stopped reporting, so a completion event lost to
a subscription gap can no longer show as running forever.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds a device field to the Action/Query envelope and a remote-ops iroh
protocol that carries one Wire method call to a paired device, executes
it through the same operation registries, and returns the JSON result.
The daemon resolves the target by name, slug, or id against its paired
devices; only paired devices are served, matching the byterange trust
boundary.
A forwarded request usually names no library — the target resolves its
own open library, since a caller can't know a peer's library ids
without asking first. The CLI macros send None whenever --device is
set, and library validation skips remote-targeted clients so a foreign
library id never lands in the local config.
execute_json_operation now delegates to a context-only variant so the
daemon socket, embedded hosts, and the network path share one dispatch.
Verified live from the Mac against titan:
sd --device titan sources list # nine sources over iroh
sd --device titan job list # titan's job history
sd --device titan sources verify <id> # dispatched and ran on titan
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The job activity plane had a subscriber with no caller and a cache that
could never fill: subscriptions were disabled in fc6a0dada, and the
cache only created a row on JobQueued, an event nothing emits. Every
jobs.remote.* query returned empty.
Subscriptions now follow connections: paired devices are subscribed as
they connect, with a boot sweep for connections that raced startup, and
a guard set so a device is never double-subscribed while its stream
lives. The cache builds rows from JobStarted and JobProgress, which
also covers subscribing to a device whose job is already mid-run.
'sd job remote' lists what paired devices are running.
Verified live: titan's content_identity jobs appear on a paired Mac as
they run and clear when they complete.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
SourceConfig is the typed shape of the source row's config column,
parsed with defaults so old rows and future fields both survive.
sources.update renames a source or changes its capture policy, with
the one asymmetry named: widening dispatches a walk for what the rules
skipped, narrowing removes nothing because hiding is the lens's job.
Tracking only ever widens the policy, so a casual re-track cannot
demote an archival source. The locations-demoted doc gains the policy
taxonomy this settles: capture on the source, attention on the
location, display in the lens.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The sampled tier groups candidates; deleting a copy needs certainty.
VerifyContentJob claims "shared content, integrity hash still null"
from the store the way the sampled job claims its queue, so it reads
exactly the files with duplicates, stays current as sampled results
keep landing, and resumes for free. The verdict carries the sampled
hash so it upgrades the existing content row, whose uuid moves to the
integrity-derived id per the ladder. sources.verify dispatches it,
sd sources verify from the CLI.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
sources.freeze writes a dated VACUUM INTO copy of the source's store
under sources/<id>/freezes/, verified by reading the record count back
out of the artifact. The live store keeps mirroring the disk; the freeze
is the record of what the disk was before a cleanup. New sd sources CLI
domain: track, list, freeze.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A location was fifteen fields rooted at an entry. It is six now: a name, the
source holding it, a path relative to that source's root, and whether someone
put it there. It owns nothing, so unpinning is a DELETE of one row and the
volume index does not notice.
Everything dropped lives somewhere that already had it, and mostly had it
better. scan_state and last_scan_at are the source's. total_size and file_count
come off the arena's rollups, which are current rather than as of the last scan
and can say a subtree was summarised where a stored integer cannot. index_mode
and job_policies configured an indexer and a job pipeline that are both gone.
entry_id is the ownership this phase exists to remove.
The path is relative so a pin survives its drive mounting somewhere else, which
is the whole reason a source carries a root and a location does not. Sources
nest, so the innermost one containing the path wins.
IndexMode goes with it. It was set on every IndexerJobConfig, threaded through
the walk, and never read: the phase that consulted it was deleted with the
persistent indexer, and content identity is its own job now. Four CLI flags and
two modals were choosing between values nothing looked at.
No rows carry over. Every one was anchored to an entry_id in a table this
series removes. The five folders a person recognises are written at library
creation instead, which is why there is no longer anything to add: Home,
Desktop, Documents, Downloads and Pictures are simply there.
Three more things fell out. File::from_entry_uuids and its siblings became
File::at_path, for_record and announce, which is the pattern five call sites
had each written by hand. SdPath::Content resolves through copies_of_content
rather than joining entries to locations to devices, so it finds a copy on any
drive rather than only where a location happened to be. And the peer file
protocols allow source roots instead of location paths, which reaches at least
as far and costs no query.
One thing removed rather than ported: user_metadata_tag checked that a
file-scoped tag came from the device owning the entry, via the location holding
it. A record belongs to whichever machine walked it and carries no device
column, so that check wants re-deriving when sync returns on top of sources
rather than being rewritten to always pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
user_metadata already had the right shape: entry_uuid or content_identity_uuid,
exactly one set. Nothing needed redesigning. The uuids just start meaning the
record the volume index minted and the content uuid the source store derives
from the bytes, and the joins that went through the entry table go through the
arena instead.
Tags stay in library.db, which is right rather than a compromise. A tag is a
claim by a person, so it syncs and travels with the library. That is the
opposite of a sidecar, which is a property of the bytes and belongs beside them.
TagTargets loses a variant it cannot have. Entry(Vec<i32>) addressed rows by
database id and there are no rows and no ids; the client never sent it. What is
left is Content and File, which is the real distinction: tag the bytes and reach
every copy on every drive, or tag this one. The client already chose content
when the file had been identified and fell back otherwise, so the policy was
already right and only the names were wrong.
ApplyToTargets was a second copy of the same enum with the same three variants.
tags.create takes TagTargets now, and both inputs share one validate.
Three helpers landed on the cache because four call sites wanted them:
content_of, copies_of_content and path_of_record. alternate_instances and
files.by_id had hand-rolled versions of the first and last; they use these now
and lost 100 lines between them.
Content-scoped changes emit events for every copy of the bytes, not just the
one that was named. Tagging a photo on one drive updates the row showing it on
another.
Two dead things fell out: find_entries_by_semantic_tags and find_files_by_tags,
both returning Vec<i32> entry ids, neither called.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four things with no callers, found by asking rather than assuming.
file_sync built a conduit between two locations and copied files along it;
`file_sync_service()` is constructed lazily and nothing ever asks for it.
locations.export and import dumped the entry tables to SQL and read them
back, unreachable from client or CLI, and a source store is already a single
SQLite file someone can copy. search/filters.rs built sea-orm conditions over
entry columns for the FTS search that went in 7d41e07ab; it looked live
because ops::search re-exports it. hierarchy.rs answered descendant queries
from the closure table, and its last caller left with the persistent indexer.
Library statistics were the real find. They summed entry.size over the
closure table, so with nothing writing entries they had quietly become zero,
and there were two copies of every calculation, static and instance, that had
to agree. Now each source reports its own totals and the two versions are one.
That needed the store to count more than it did. counts() answered records
and bytes; it answers directories and contents too, in the same query. Files
are records less directories, because a folder is a record and nobody asking
how many files they have means folders. content_count is a new column on
sources, since record_count counts rows and unique_bytes counts bytes and
neither says how many separate files a drive holds once copies fold together.
Unique content is per-source and will over-count across drives until the
catalog exists: two drives holding the same photo report one each. That is
visible on the overview, and it is the same gap files.duplicates documented.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every query that answered from `entry` now answers from the volume index,
which is the one that knows about the whole drive. Each of these had both
paths already and picked between them at runtime; the entries branch was
the one that only worked where someone had added a location.
search collapsed hardest. IndexType existed to choose between an FTS5 index
over entries and the in-memory one, and there is only the one now, so fast,
normal and full modes were three routes to the same table. A library-wide
search fans out over every partition, which is the search that was never
possible before: it used to mean "every location you added".
SearchScope loses Location. Nothing called it, and under the demotion a
location is a subtree rather than a thing you search inside.
files.media_listing walks the arena instead of joining four tables, and
sorts DateTaken by birth time, which is the closest thing the arena knows
and the one a photo library actually wants.
files.alternate_instances asks content identity rather than content_id, so
it spans drives: every source is asked about the same content uuid and they
agree because the id comes from the bytes. Two new store queries back it,
content_of and copies_of_content.
Copy estimates read subtree_bytes and file_count off the arena. Those
rollups are maintained as the arena is written, so the progress bar gets its
numbers from a lookup rather than from a walk.
files.unique_to_location had no callers and no meaning once a location stops
owning records.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing produces a row in `entry` after this. The walk fills the volume
index and its source store, which is what the arena has been doing since
the storage map landed; the persistent half of the indexer was a second
writer to a second store that no source ever read.
Gone: IndexPersistence, and with it the branch in every phase. The state
machine is Discovery then Processing, since Aggregation existed to bubble
directory sizes the arena now maintains incrementally, and
ContentIdentification is ContentIdentityJob's job against the source store.
DatabaseAdapter, PersistentEventHandler and the location watching that fed
them go too, so ArenaWriter is the only change handler and the walk and the
watcher differ only in Notify.
DatabaseStorage was 1230 lines of entry writes wrapped around three things
the arena actually uses. Those are ops::indexing::metadata now: EntryMetadata,
extract_metadata and get_inode, plus is_hidden_path. delete_subtree moves onto
the entry entity, which is what it operates on and what it dies with.
locations.rescan, locations.enable_indexing and locations.triggerJob dispatched
the deleted indexer or its media jobs, so the ops and their UI go. Reindex and
Run Now are gone from the inspector; the job policy toggles stay, since they
outlive the ops that read them. Adding a location no longer dispatches a walk,
which is the point: the drive is already mapped.
The entry tables and their readers are still here. That is the next commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An index becomes browsable two ways and only one of them armed a watch. A
walk finishes and the indexing job watches what it walked. A restart restores
the same index from a snapshot and watched nothing, so the drive browsed
perfectly and reported no changes until it was indexed again.
Measured on the running daemon: one source, /System/Volumes/Data, 2.1M
entries restored, no ephemeral index job this session, and a real file
created on the desktop produced no ResourceChanged on the event bus.
The cache now announces a root when its snapshot restore populates the index,
and the watcher service arms a watch for each. A channel rather than a handle
because the watcher service already holds the cache. Announced once per
source per session, and watch_ephemeral returns early for a root it already
watches, so nothing accumulates a reference count it never sheds.
core.ephemeral_status reports watched roots, and `sd-cli index
ephemeral-cache` prints them even when the list is empty. Empty is the whole
answer here, and reading it took a code walk twice.
Sidebar lists indexed drives including detached ones, volume rows and overview
cards carry persisted entry counts, and the explorer gains a list view over
daemon ops with event-driven refresh.
* feat: add redundancy awareness & cross-volume file comparison
Surface content redundancy data so users can answer "if this drive dies,
what do I lose?" — builds on existing content identity and volume systems.
Backend:
- New `redundancy.summary` library query with per-volume at-risk vs
redundant byte/file counts and a library-wide replication score
- Extend `SearchFilters` with `at_risk`, `on_volumes`, `not_on_volumes`,
`min_volume_count`, `max_volume_count` filters
- Add composite index migration on entries(content_id, volume_id)
Frontend:
- `/redundancy` dashboard with replication score, volume bars, at-risk callout
- `/redundancy/at-risk` paginated file list sorted by size
- `/redundancy/compare` two-volume comparison (unique/shared toggle)
- Sidebar ShieldCheck button linking to redundancy view
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* redundancy UI improvements + ZFS volume detection fix
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix ZFS pool capacity reporting and stats filtering
- ZFS: override total_capacity for pool-root volumes using zfs list
used+available. df under-reports pool-root Size because it only
counts the root dataset's own used bytes plus avail — on a 60 TB
raidz2 pool this shows as ~15 TB instead of ~62 TB. The pool root's
own used property includes descendants, so used+available is the
real usable capacity.
- Library stats: drop volumes where is_user_visible=false AND
re-apply should_hide_by_mount_path retroactively so stale DB rows
(detected before the Linux visibility filters existed) don't
inflate reported capacity.
- Extract should_hide_by_mount_path into volume/utils as a shared
helper used by both the list query and the stats calculation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* show capacity and visibility in sd volume list
Makes it possible to verify library-level capacity aggregation from
the CLI — previously the list only showed mount, fingerprint, and
tracked/mounted state, which meant debugging the ZFS pool capacity
issue required querying the library DB directly.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* document filesystem support matrix and detection
New docs/core/filesystems.mdx covering per-filesystem capabilities
(CoW, pool-awareness, visibility filtering, capacity correction),
platform detection strategies, the FilesystemHandler trait, Linux/
macOS/ZFS visibility rules, the ZFS pool-root capacity problem and
fix, copy strategy selection, and known limitations.
Registered under File Management in both mint.json and docs.json.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* WIP: redundancy filter wiring across search, CLI, and UI
- core/src/ops/search: wire redundancy filters (at_risk, on_volumes,
not_on_volumes, min/max volume_count) through the search query;
fix UUID-to-SQLite BLOB literal so volume UUID comparisons actually
match (volumes.uuid is stored as a 16-byte BLOB, quoted-string
comparison silently returned zero rows).
- apps/cli: new redundancy subcommand + populate the new
SearchFilters fields from search args.
- packages/interface: redundancy at-risk and compare pages reworked
to consume the new filter surface; explorer context/hook updates
to support redundancy-scoped views.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* add web context menu renderer and UI polish
- New WebContextMenuProvider + Radix DropdownMenu-based renderer anchored
at cursor via a 1x1 virtual trigger. Handles separators, submenus,
disabled, and the danger variant via text-status-error.
- useContextMenu now routes web clicks through the provider instead of
parking data in unused local state, and trims leading/trailing/adjacent
separators so condition-filtered menus don't render orphaned lines.
- Drop app-frame corner rounding on the web build.
- Add shrink-0 to the sidebar space switcher so the scrollable sibling
can't compress it vertically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* sd-server dev workflow: auto web build, shutdown watchdog, stable data dir
- build.rs runs `bun run build` in apps/web so `just dev-server` always
embeds the latest UI. rerun-if-changed covers apps/web/src,
packages/interface/src, and packages/ts-client/src so Rust-only edits
skip the rebuild. Skips gracefully when bun isn't on PATH or
SD_SKIP_WEB_BUILD is set; Dockerfile sets the latter since dist is
pre-built and bun isn't in the Rust stage.
- Graceful shutdown was hanging because the browser holds the /events
SSE stream open forever and axum waits for all connections to drain.
After the first signal, arm a background force-exit on second Ctrl+C
or 5s timeout so the process can't stick.
- Debug builds were starting from a fresh tempfile::tempdir() on every
run (the TempDir handle dropped at end of the closure, deleting the
dir we just took a path to). Default to ~/.spacedrive in debug so data
persists and `just dev-server` shares a data dir with the Tauri app.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* add Sources space item alongside Redundancy in default library layout
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* add TrueNAS native build script
Uses zig cc as C/C++ compiler on TrueNAS Scale where /usr is
read-only and no system gcc exists. Dev tools live at
/mnt/pool/dev-tools/ (zig, cmake, make, extracted deb headers).
Builds sd-server + sd-cli in ~4 min on a 12-core NAS. AI feature
disabled (whisper.cpp C11 atomics incompatible with zig clang-18).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* never block RPC on synchronous statistics calculation
On first load (fresh library, all stats zero), libraries.info used
to calculate statistics synchronously before responding. On large
libraries during active indexing this hangs indefinitely — the
closure-table walk in calculate_file_statistics loads every
descendant ID into a Vec then issues a WHERE IN(...) with millions
of entries, which SQLite can't finish while the indexer is writing.
Now always return cached (possibly zero) stats and let the
background recalculate_statistics task fill them in. The UI
refreshes via the ResourceChanged event when the calculation
completes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* self-heal protocol handler registration on re-init
Core::new() registers default protocol handlers after starting networking,
but swallows any failure (error is only logged). If the initial registration
fails — e.g. on a host where start_networking hasn't fully set up the event
loop command sender by the time register_default_protocol_handlers runs —
the registry is left empty. A subsequent call to Core::init_networking()
would see `services.networking().is_some()` and skip re-registration,
permanently leaving protocols unregistered for the life of the process.
sd-server calls init_networking() right after Core::new(), so it's the
client most exposed to this. Symptom: pairing over the web UI returns
"Pairing protocol not registered" while the same library works fine
from Tauri and mobile.
Fix: init_networking now queries the registry directly for the pairing
handler and re-registers the default set if it's missing, independent of
whether networking is already initialized.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fall back to pkarr+DNS discovery when mDNS port is unavailable
Iroh's endpoint.bind() fails wholesale if any configured discovery service
fails to initialize. MdnsDiscovery requires binding UDP :5353, which on
most Linux systems (including TrueNAS) is already owned by avahi-daemon.
Result: endpoint creation errors out with "Service 'mdns' error", the
event loop never starts, command_sender stays None, and protocol
registration fails — so sd-server has no working networking at all.
Make mDNS best-effort: on any error whose message mentions "mdns",
retry endpoint creation with only pkarr + DNS discovery. Local-network
auto-discovery is lost but remote pairing via node ID (which uses n0's
DNS infrastructure, not mDNS) continues to work normally.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* succeed pairing if either mDNS or relay discovery wins
The dual-path discovery in start_pairing_as_joiner_with_code used
tokio::select! to race mDNS and relay. select! resolves on the first
branch to complete — including errors — so a host that can't bind
mDNS (e.g. a Linux box where avahi already owns UDP :5353) would fail
pairing wholesale: mDNS discovery errors out in <1ms with
"Failed to create mDNS discovery: Service 'mdns' error", that Err
wins the race, and relay discovery gets cancelled before it can even
begin.
Switch to futures::select_ok so we only return the error if EVERY
discovery path has failed. mDNS failing immediately now leaves relay
running to completion, which is the common case for remote pairing
into a NAS.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Changed versioning from "2.0.0-pre.1" to "2.0.0-alpha.1" in various Cargo.toml files and package.json to reflect a new alpha release.
- Added support for ARM64 builds in the GitHub Actions workflow, including a new host configuration and installation of VA-API libraries for the ARM target.
- Introduced a new `bump` command in the xtask to facilitate version updates across all relevant files, improving version management.
Introduces a proper feature flag hierarchy for AI-related dependencies,
allowing lite builds without heavy speech-to-text dependencies.
Feature hierarchy:
- `ai` - umbrella feature for all AI capabilities
- `speech-to-text` - transcription capability (requires ffmpeg + whisper)
- `whisper` - Whisper speech recognition engine (internal)
- `ffmpeg` - audio/video processing (existing)
Changes:
- Make whisper-rs, hound, rubato optional deps behind `whisper` feature
- Add `speech-to-text` feature that requires both ffmpeg and whisper
- Add `ai` umbrella feature that enables all AI features
- Update all cfg guards to use semantic `#[cfg(feature = "speech-to-text")]`
- Expose new features in apps/server and apps/cli
- Update release workflow to include `ai` feature (maintains existing behavior)
This enables:
- Mobile builds without STT (smaller binary, already using default-features=false)
- Desktop "lite" builds for users who want minimal size
- Future AI features can be added under the `ai` umbrella
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Introduced new event types for proxy pairing confirmation and vouching readiness in the event summarization.
- Implemented a reload mechanism for protocol configurations to support dynamic updates during runtime.
- Enhanced the PairingProtocolHandler to manage joiner pairing handshakes and improve session handling.
- Added comprehensive unit tests for proxy pairing, covering session creation, state transitions, and vouching logic.
- Updated configuration structures to include default proxy pairing settings for better integration.
This commit introduces a new feature for safely ejecting removable volumes, enhancing the volume management capabilities. It adds the `VolumeEjectAction`, `VolumeEjectInput`, and `VolumeEjectOutput` structures to handle the eject operation. Additionally, the context menu in the UI is updated to include an option for ejecting volumes, improving user experience. The volume display logic is also refined to ensure accurate representation of volume properties. These changes collectively enhance the overall functionality and usability of the volume management system.
This commit refactors the volume management system by replacing the VolumeItem struct with the Volume struct in the VolumeListOutput, streamlining the data representation. It enhances the volume display logic to utilize display names and adds helper functions to skip serializing unknown disk and volume types. Additionally, the JobManager is updated to include a new field for event emission control, improving job tracking and management. These changes improve overall clarity and efficiency in volume handling and job management.
This commit introduces a detailed memory breakdown feature for the ephemeral indexing process, allowing users to view memory usage by component. It also enhances UUID management by implementing lazy generation and assignment of UUIDs during indexing and searching operations. These changes improve memory tracking and optimize UUID handling, contributing to better performance and user experience in the indexing system.
This commit introduces a new snapshot mechanism for ephemeral indexes, allowing for fast restoration between sessions. The snapshot functionality utilizes zero-copy binary serialization with zstd compression, significantly reducing load times from over 10 minutes to just 1-2 seconds. The implementation includes methods for saving and loading snapshots, as well as enhancements to the existing indexing structure to support this feature. Additionally, the codebase is updated to ensure compatibility with the new snapshot format, improving overall performance and user experience.
This commit improves the `CopyJobDetails` component by adding an auto-scroll feature that centers the currently copying file in the transfer queue. Additionally, it implements a mechanism to refetch copy metadata when the completed file count changes, ensuring that the displayed information is always up-to-date. The UI is further refined with updated icons and styles for better user experience during file operations.
- Fix 'access_token and refresh_token can not be set at the same time' error
- Update Dropbox to use only refresh_token (OpenDAL requirement)
- Add client_id/client_secret to OAuth credential storage
- Add validation for empty OAuth credentials
- Update CLI and volume manager for enhanced OAuth support
- Removed the static ASCII logo and implemented a new function to calculate brightness on a sphere, allowing for a dynamic rendering of the Spacedrive logo as a purple orb using ANSI colors and Unicode half-blocks.
- Enhanced the `print_logo_colored` function to utilize the new brightness calculation and color gradient, improving visual representation.
- Updated related documentation to reflect the changes in logo rendering.
- Added `remove_from_library` field to `DeviceRevokeInput` and `DeviceRevokeAction` to control whether a device should be removed from library databases during revocation.
- Updated the `from_input` method in `DeviceRevokeAction` to handle the new field.
- Modified the revocation logic to conditionally remove devices from libraries based on the `remove_from_library` flag, improving flexibility in device management.
- Adjusted the CLI argument parsing to default `remove_from_library` to false, ensuring devices remain in libraries unless explicitly specified for removal.
