3 Commits

Include renames

Author	SHA1	Message	Date
Jamie Pineandcapy-ai-staging[bot]	f9bdf724b6	Library join: fix inverted backfill, catch-up loop, reverse device registration, member totals (#3110) ... * Networking: serve streams on connections a protocol dials The connection cache is shared between the dial side and the accept side, so once a protocol dials a peer, the peer's own send path finds that connection and opens its streams on it. Nothing read those streams: only connections the event loop accepted, or those the reconnect loop handed over, had a stream handler. The peer's request then sat unanswered until its 30 s timeout while requests in the other direction answered in milliseconds. This is the reverse RegisterDeviceRequest timeout SPAC-12 saw after join-remote. get_or_create_connection now hands every connection it dials to the event loop as TrackOutboundConnection, which spawns the same stream handler an inbound connection gets. Messaging, sync, and job activity pass their event loop handle; pairing keeps its own connection handling. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Sync: the device that joins a library is the one that backfills Every sync service started Uninitialized and backfilled from the first connected partner, whatever state it already held. When the Mac Studio joined the MacBook's library the MacBook was the one with a partner in its device table, so it went Uninitialized -> Backfilling{Studio} and applied the empty copy's snapshot over its own rows (LWW on the deterministic default-space UUID). The Studio never backfilled because the reverse registration had timed out and its device table held only itself. A copy of a library created from another device's library is the only thing that may start Uninitialized. create_library_with_id now writes an initial_backfill_pending marker into sync.db; PeerSync reads it on open and starts Ready otherwise. A Ready device never requests a snapshot (SharedChangeRequest.include_snapshot = false on catch-up) and repeated catch-up failures no longer escalate to a full backfill. A copy still waiting for its own first backfill refuses to serve as a join target. The join carries the rule explicitly: RegisterDeviceRequest has needs_initial_state, and RegisterDeviceResponse returns the responder's own device record so the joiner's table is complete when the command returns, replacing the spawned reverse request. sync_setup fails instead of reporting devices_registered: true when that record is missing. Catch-up completion is recorded per peer in sync.db even when no rows arrived; the Ready loop catches up once with a peer it has no record for and no longer re-polls every peer whose watermark is 60 s old. A missing watermark means never synced, not ancient, so the forced full sync warning only fires for a real watermark past retention. library_join_test pairs two daemons for real, joins Bob to Alice's library, and asserts Alice's rows are unchanged, Bob holds Alice's device row and spaces, and Alice runs no further catch-ups. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Sync: test that catch-up with an empty peer runs once Alice holds a volume, Bob holds nothing of his own, so Alice never writes a per-resource watermark for Bob. Before the per-peer catch-up record, that made Bob a new peer on every loop iteration and Alice caught up with him forever (2 -> 5 sessions in 16 s on 5e285d3). The test asserts the session count stops moving and the record exists. Both new suites join the xtask integration list. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Library info: fleet totals count the same devices as device_count add_device_summaries added every cached peer summary to a library's statistics, whatever library the peer belonged to, and the summaries carried no source count. A joined device therefore reported the member's 3 million files next to Sources 0 and Devices 1. Summaries now carry the device's own source_count, and only summaries of devices in the library's device table are added, so files, content, sources, and capacity describe the members that device_count counts. The CLI labels the block "Statistics (all library members)". Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Docs: describe the pairing and library join flow as it runs pairing.mdx and devices.mdx described a confirmation prompt that does not exist and pointed at `sd devices list` for pairing status, which reads the library's device table and does not show a peer until it joins the library. The CLI overview lacked the global `--device` flag, and the sync setup walkthrough never named `library sync-setup --action join-remote` or `--node-id`. All four now describe the flow SPAC-12 ran, and library-sync.mdx gains sections on joining, catch-up, and fleet totals. One PROJECT_STATUS register row. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Sync: catch up once per connection session, never serve a snapshot while pending Review of #3110 found two gaps. The per-peer catch-up record was never cleared, so a peer that changed rows while disconnected was never caught up with again after reconnecting (the live broadcast only lives in the in-memory retry queue). The record is now cleared when the sync service starts and when the peer's connection is lost; ConnectionEstablished is not used because the event loop emits it per accepted stream. The SharedChangeRequest handler attached a full snapshot without checking the first-backfill marker, so two copies that both joined from the same member could seed each other. It now refuses the snapshot while pending, and the register handler reads the marker from sync.db directly when the library's sync service is not up yet. Also: a failed backfill or catch-up left active_backfill_sessions incremented; the mock transport no longer lists a peer as a partner before its sync service is registered; the catch-up test now starts Alice Ready (marker cleared) and Bob pending, the shape of a real join, and asserts exactly one catch-up and one backfill. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Sync: move past a peer that refuses a snapshot, fail the guard closed, refresh stale catch-ups A joiner whose chosen peer refused the snapshot kept choosing it: every partner was scored has_complete_state: true and RTT is only recorded on success, so the order never changed. A peer whose backfill failed now scores without complete state until a backfill succeeds, so a connected member holding the library is tried next. The snapshot guard read an unreadable marker as "holds state"; it now refuses, since a wrong refusal only delays a join. A broadcast whose retries ran out while the QUIC session stayed up had no recovery under the once-per-session rule. The Ready loop refreshes a catch-up record older than ten minutes; with per-peer watermarks an empty refresh is two small requests. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>	2026-10-05 10:35:00 +00:00
Jamie Pine	2d380fd264	Move docs to Fumadocs and rewrite the core docs ... Replaces the Mintlify setup (mint.json, docs.json, snippets) with a standalone Fumadocs app in docs/: Next.js app router, meta.json nav per section, its own bun install. Run with `just dev-docs`. Content pass alongside the migration: core architecture, data model, CLI, and networking pages rewritten around the daemon/source-store/ library split, new product-direction.mdx defining the product boundary, new drive-catalog plan, and the physical-drives plan updated to reference it. Whitepaper page retired.	2026-09-15 00:59:30 -07:00
James Pine	f7d7468bce	remove submodules	2025-11-14 21:31:21 -08:00