* Networking: serve streams on connections a protocol dials
The connection cache is shared between the dial side and the accept side,
so once a protocol dials a peer, the peer's own send path finds that
connection and opens its streams on it. Nothing read those streams: only
connections the event loop accepted, or those the reconnect loop handed
over, had a stream handler. The peer's request then sat unanswered until
its 30 s timeout while requests in the other direction answered in
milliseconds. This is the reverse RegisterDeviceRequest timeout SPAC-12
saw after join-remote.
get_or_create_connection now hands every connection it dials to the
event loop as TrackOutboundConnection, which spawns the same stream
handler an inbound connection gets. Messaging, sync, and job activity
pass their event loop handle; pairing keeps its own connection handling.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: the device that joins a library is the one that backfills
Every sync service started Uninitialized and backfilled from the first
connected partner, whatever state it already held. When the Mac Studio
joined the MacBook's library the MacBook was the one with a partner in
its device table, so it went Uninitialized -> Backfilling{Studio} and
applied the empty copy's snapshot over its own rows (LWW on the
deterministic default-space UUID). The Studio never backfilled because
the reverse registration had timed out and its device table held only
itself.
A copy of a library created from another device's library is the only
thing that may start Uninitialized. create_library_with_id now writes an
initial_backfill_pending marker into sync.db; PeerSync reads it on open
and starts Ready otherwise. A Ready device never requests a snapshot
(SharedChangeRequest.include_snapshot = false on catch-up) and repeated
catch-up failures no longer escalate to a full backfill. A copy still
waiting for its own first backfill refuses to serve as a join target.
The join carries the rule explicitly: RegisterDeviceRequest has
needs_initial_state, and RegisterDeviceResponse returns the responder's
own device record so the joiner's table is complete when the command
returns, replacing the spawned reverse request. sync_setup fails
instead of reporting devices_registered: true when that record is
missing.
Catch-up completion is recorded per peer in sync.db even when no rows
arrived; the Ready loop catches up once with a peer it has no record
for and no longer re-polls every peer whose watermark is 60 s old. A
missing watermark means never synced, not ancient, so the forced full
sync warning only fires for a real watermark past retention.
library_join_test pairs two daemons for real, joins Bob to Alice's
library, and asserts Alice's rows are unchanged, Bob holds Alice's
device row and spaces, and Alice runs no further catch-ups.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: test that catch-up with an empty peer runs once
Alice holds a volume, Bob holds nothing of his own, so Alice never
writes a per-resource watermark for Bob. Before the per-peer catch-up
record, that made Bob a new peer on every loop iteration and Alice
caught up with him forever (2 -> 5 sessions in 16 s on 5e285d3). The
test asserts the session count stops moving and the record exists.
Both new suites join the xtask integration list.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Library info: fleet totals count the same devices as device_count
add_device_summaries added every cached peer summary to a library's
statistics, whatever library the peer belonged to, and the summaries
carried no source count. A joined device therefore reported the member's
3 million files next to Sources 0 and Devices 1.
Summaries now carry the device's own source_count, and only summaries
of devices in the library's device table are added, so files, content,
sources, and capacity describe the members that device_count counts.
The CLI labels the block "Statistics (all library members)".
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Docs: describe the pairing and library join flow as it runs
pairing.mdx and devices.mdx described a confirmation prompt that does
not exist and pointed at `sd devices list` for pairing status, which
reads the library's device table and does not show a peer until it
joins the library. The CLI overview lacked the global `--device` flag,
and the sync setup walkthrough never named `library sync-setup
--action join-remote` or `--node-id`. All four now describe the flow
SPAC-12 ran, and library-sync.mdx gains sections on joining, catch-up,
and fleet totals. One PROJECT_STATUS register row.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: catch up once per connection session, never serve a snapshot while pending
Review of #3110 found two gaps. The per-peer catch-up record was never
cleared, so a peer that changed rows while disconnected was never caught
up with again after reconnecting (the live broadcast only lives in the
in-memory retry queue). The record is now cleared when the sync service
starts and when the peer's connection is lost; ConnectionEstablished is
not used because the event loop emits it per accepted stream.
The SharedChangeRequest handler attached a full snapshot without
checking the first-backfill marker, so two copies that both joined from
the same member could seed each other. It now refuses the snapshot while
pending, and the register handler reads the marker from sync.db directly
when the library's sync service is not up yet.
Also: a failed backfill or catch-up left active_backfill_sessions
incremented; the mock transport no longer lists a peer as a partner
before its sync service is registered; the catch-up test now starts
Alice Ready (marker cleared) and Bob pending, the shape of a real join,
and asserts exactly one catch-up and one backfill.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: move past a peer that refuses a snapshot, fail the guard closed, refresh stale catch-ups
A joiner whose chosen peer refused the snapshot kept choosing it: every
partner was scored has_complete_state: true and RTT is only recorded on
success, so the order never changed. A peer whose backfill failed now
scores without complete state until a backfill succeeds, so a connected
member holding the library is tried next.
The snapshot guard read an unreadable marker as "holds state"; it now
refuses, since a wrong refusal only delays a join.
A broadcast whose retries ran out while the QUIC session stayed up had
no recovery under the once-per-session rule. The Ready loop refreshes a
catch-up record older than ten minutes; with per-peer watermarks an
empty refresh is two small requests.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
Every job that changes the filesystem records effects (created, moved, trashed with where it went, replaced with where the previous bytes went, removed, attributes before and after) in a job_journal table through ctx.record. The trash reports locations: NSFileManager on macOS, the trash crate's listing elsewhere, and a .spacedrive/trash/<job>/ directory on volumes with none. Overwrites in copy, merge and extract stash the previous file there. files.undo previews the reverse as an FsPlan, refuses what changed since, and its job writes its own journal.
Also: files.rename on preflight with per-filesystem name rules and case probing, files.rename_batch with ordered rules and a job that never overwrites; remove_extras on files.merge for a mirror; files.organize and files.flatten as in-place renames; files.archive and files.extract for zip and tar.zst, planned from the archive's directory; files.set_attributes and files.link; files.trash_list and files.trash_empty; jobs.journal and a journal summary on jobs.list. The copy job now honors the skip policy and merges a folder onto an existing folder instead of overwriting inside it, and the copy preview plans a keep-both copy at its numbered name.
30 tests over the fixture. Docs, the plan's status table and PROJECT_STATUS updated.
files.delete takes a Duplicates target: a scope with keep first (walk order) or a list of copies to keep with every other copy of their content going. The job reads each surplus copy and its keeper in full before removing, records the hashes it learns, and checkpoints the compare cursor so it resumes. Preflight validates the scope and keep list and previews keepers as skips and surplus copies as deletes.
Store gains duplicated_contents_beneath and copies_beneath. files.duplicates is registered as a core query and reports each copy's SdPath. CLI: sd file dedupe with --keep, --keep-under, --min-size, --dry-run.
Two opt-in methods per action, validate: and preview:, over the exact
input the action takes. Validation answers findings with stable codes and
the facts of the execution, runs again at dispatch and refuses on an
error. Preview answers an FsPlan read from the index, kept under a handle
so files.directory_listing can serve the directory as it would look after
the operation.
Copy, move, merge and delete answer both. Merge is new: a resumable job
that recurses into matching folders, reads both sides before skipping a
duplicate, applies a per-leaf conflict policy, leaves conflicts alone and
can consume its sources. Delete's validation counts, across every store,
which files are the last copy of their content.
The ValidationResult and ConfirmationRequest hook is gone, and so is the
CLI's local conflict detection. file copy, merge and delete render
preflight before the prompt and take --dry-run.
files.delete takes a comparison as its targets alongside a path list, so
the client sends five fields and the daemon derives the set from the
index as the job runs. The job drains the matcher the compare query
pages, checkpoints its cursor after each batch, and for `both` reads
whichever side lacks an integrity hash before removing a file, writing
what it learned back to the store. Pairs whose bytes differ, or whose
copy in B is gone by then, stay and are reported. The CLI gains
`sd file delete A --against B --show both` behind a y/N prompt, plus the
plain paths form.
Compare matched on content uuids, which re-derive from the integrity
hash once a copy is verified, so a copy verified on one side stopped
matching its unread twin. It now matches on the sampled hash every store
keys a content by, with integrity hashes deciding where both sides have
them; FsEntry carries both hashes for that.
Left and right only meant first and second argument, and nothing on
screen said which folder was which. "Same" meant different things per
mode, "changed" implied a direction compare can't know, and "location"
collided with the retired locations.
The output now names both folders as A and B up front, says how files
matched, then counts Only in A, Only in B, In both, and Different (by
path only). --by location becomes --by path and the --show values
follow, as do the API types, generated TS types, skill, and docs.
Single-side listings get Size and Modified columns under a title naming
the set, and --help explains each value.
paths.compare answers what differs between two folders from the source
stores at query time. By location it merges both sides in index order
into only-left, only-right, changed and same. By content it streams one
side and asks the other side's stores which content ids they hold, so a
renamed or moved copy still matches. Pages resume from a cursor and the
first page carries every set's count. Unhashed pairs fall back to size
and mtime and get counted, so a page says how much rests on that.
sd-cli file compare wraps it.
On the store side, files_beneath takes a Start, including one that
resumes a nested store at its first directory, and contents_beneath
answers the content lookup a batch at a time. Working out which stores a
path reaches moved into ops/paths/reach.rs, shared with search.media and
paths.context.
No catalog.db: the P7 row, the filesystem-intelligence plan, the previs
and the design docs now treat cross-source questions as store queries.
The first client to connect starts automatic discovery: it tracks the
system volume, adds the home folder as a source and walks every attached
drive. For a library meant to hold only what you add, like one whose data
directory lives on the drive it catalogs, that indexes the Mac into it.
sd start --no-default-sources, and the same flag on restart and on
sd-daemon, now leaves those three out. The pass still restores snapshots,
heals coverage for sources already tracked and hashes them, so a source
you added keeps working across restarts. sd-server keeps the defaults.
Replaces the Mintlify setup (mint.json, docs.json, snippets) with a
standalone Fumadocs app in docs/: Next.js app router, meta.json nav per
section, its own bun install. Run with `just dev-docs`.
Content pass alongside the migration: core architecture, data model,
CLI, and networking pages rewritten around the daemon/source-store/
library split, new product-direction.mdx defining the product boundary,
new drive-catalog plan, and the physical-drives plan updated to
reference it. Whitepaper page retired.
- Introduced `index verify` command to perform comprehensive integrity checks on the Spacedrive index.
- Implemented functionality to compare filesystem state with database entries, reporting discrepancies such as missing files, stale entries, and metadata mismatches.
- Added detailed output options for verification results, including a summary report of findings.
- Created associated input and output types for verification actions, enhancing the overall integrity management of the indexing system.
- Documented usage and examples in the CLI documentation for user guidance.
