mirror of
https://github.com/spacedriveapp/spacedrive.git
synced 2026-10-30 21:09:14 -04:00
main
201
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jamie Pineandcapy-ai-staging[bot]
|
fc1798b05a |
Windows unit job gates main; daemon raises its fd limit; sd-version marker test (#3153)
* CI: the Windows unit job gates main Core Tests (windows unit) was green twice on main ( |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
e4b4cb71f5 |
Capture time and place in listings and search, read from the image facet on both backends (#3151)
* Capture time and place in listings and search, read from the image facet on both backends search.files sorts by CapturedAt and filters by a CapturedAt range and has_location; files.directory_listing sorts by datetaken; the media listing's datetaken reads the facet. The store joins facet_image into every entry read, guarded by the row's content hash, and the arena path reads the facet from the store ahead of its sort and filter, so both backends hand the same File to one predicate and one order (R8 row 19, new row 29). File.image_media_data is filled from the facet on every path; the inspector shows the capture time and the coordinates; Date Taken is in every explorer view's sort menu. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * search.media judges the capture filters on its rows and leaves replicas out under them; a directory listing reads facets ahead of the sort only for a capture order Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * search.media judges a replica's rows under a capture filter as search.files does instead of dropping the partition Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
15d7df1808 |
Windows: build and unit-test the daemon and CLI on a Windows runner, fix what it found, publish Windows nightlies (#3148)
* ci: build and unit-test sd-core and sd-cli on a Windows runner The job is the first Windows build since the rewrite and the evidence for the cross-platform release gate. It runs on GitHub's hosted windows-latest (Blacksmith has no Windows), builds sd-cli and sd-daemon in debug, then runs the sd-core lib tests and the sd-cli tests. continue-on-error keeps it from gating main until it has been green twice. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * cli: name the daemon binary and replace binaries the way Windows allows The CLI built the daemon's path as sd-daemon everywhere, which does not exist on Windows; sd_client::daemon_binary_name appends the platform's executable suffix. sd update renamed the staged file over the running binary, which Windows refuses for a loaded executable: it now moves the current binary to .bak first and renames the new one into place, removing the backup where the platform lets it and on the next update otherwise. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * indexing: keep the hidden bit in the arena instead of re-reading it on every read The arena rebuilt is_hidden from the path on every entry read. On Unix that is a name check; on Windows it is a GetFileAttributesW call per entry, and it answers false for anything not on disk at that moment, so search returned hidden files from a snapshot or a detached volume. PackedMetadata now carries a hidden bit (size drops to 59 bits), the arena records what the walk saw, and the writer judges a change by the metadata it carries. Snapshot restore recomputes the bit from names where names decide. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * volume: tell a Windows mount point from the directory a drive left behind is_mount_point answered by existence on Windows, so the empty folder an unplugged mapped drive leaves passed as mounted and the detached volume's source could be dispatched at it. GetVolumePathNameW names the mount point of the volume holding a path; a directory that is its own volume root is a mount point, anything else resolves to a parent volume. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * core: let three path-shaped unit tests read the same on Windows The compare and media tests compared displayed paths against slash-written expectations, and the track action test offered a Unix absolute path that Windows reads as relative. The tests now normalise separators and pick an absolute root the platform accepts; what they assert is unchanged. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * files: delete a moved source when the filesystem still has it, not when the volume index guesses After a streaming copy in move mode the source was deleted only when the volume manager said the two paths were on different volumes. On Windows it answered same-volume for a copy the router had already decided to stream, so the file stayed in both places and the move reported success. The job now checks whether the source still exists after the strategy ran and removes it then, which is the condition the comment described. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * ci: save the Windows rust cache on a red run too Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * ci: publish Windows x86_64 nightly binaries The nightly matrix gains x86_64-pc-windows-msvc on windows-latest, producing sd-windows-x86_64.exe and sd-daemon-windows-x86_64.exe with checksums. The build and prepare steps run under bash on every host, and the asset names carry .exe, which the updater already strips when matching. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * docs: Windows gap report and register row docs/plans/2026-10-09-windows-gap.md records the first Windows build since the rewrite: what compiled, what passed, each failure with its cause and fix, what the integration, FDA, R8 and locked volumes suites need to run on Windows, and the beta.1 recommendation with release notes text. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * review: judge mount points on the path as given, mark placeholder directories hidden, keep the rename-over update on Unix GetVolumePathNameW judges the string it is handed, so a subst or mapped drive letter must not be canonicalized to its target first or its own root reads as not a mount point. Arena::ensure_directory creates directory placeholders the walk has not statted; they now carry the hidden answer the path gives until the walk corrects them. The updater moves the running binary aside only on Windows, which needs it; elsewhere a .bak copy and a rename over the target keep a working file at the path at every instant. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
6dc6c2bef8 |
Housekeeping: drop sources triggers, remove libraries.export stub, prune restore-trash (#3143)
* ci: drop the sources branch from nightly and core tests triggers Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * core: remove the libraries.export stub left behind by backup/restore Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * core: prune restore-trash after a successful library restore A restore --replace parks the displaced library and store files under restore-trash/<library>-<time>/ and never removed them. Once the restored library is open, keep the newest backup.restore_trash_keep entries for that library (default 2, settable with sd config set) and remove the rest, never the one the restore just parked. Removed paths are logged and returned in the restore output. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * tests: add the backup field to the sync harness AppConfig literal Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
731744277e |
Add to Library step 4: keep an offline copy of an on-source catalog (#3142)
* Add to Library step 4: keep an offline copy of an on-source catalog as a library replica Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Say where a detached source's catalog is: offline copy, or on the drive Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Retire only a reader on the offline copy when publishing; drop the unused apply_setting Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Fill the new field in the CLI's source row test Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Fill offline_copy in sources.get; show the catalog hint on the card whether or not the drive has a badge Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
a19553f6f1 |
Locked volumes L5: volume state in the app, CLI and peer volume facts (#3141)
* Locked volumes L5: volume state in the app, CLI and peer volume facts RemoteVolumeInfo carries VolumeState and a device publishes its tracked unmounted and locked volumes, so a peer lists the owner's dataset as locked instead of a replica that stopped updating. RemoteSourceInfo carries each source's volume_state and sources.get reads the live index. VolumeMountChanged carries the state and fires on any state change, including a key unloaded from an already unmounted dataset. The CLI event printer names a lock; sd sources list has a State column and sd volumes list a State line. The sidebar, Storage cards, source detail and Overview show Locked, Unmounted or Offline and say the index stays browsable. The Add to Library modal refuses a locked or unmounted drive with the reason. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Refuse an add only for this device's away drives; printer falls back to is_mounted without a state Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Say why RemoteVolumeInfo.state stays last Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Hold the Add to Library refusal until the device list resolves Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
0e76551463 |
Nightly release: edit in place instead of delete and recreate (#3137)
* Nightly release: edit in place instead of delete and recreate Every gh release create notified everyone watching the repository for releases, about ten times a day. The publish job now moves the nightly tag with git, edits the release title and notes, uploads assets with --clobber and removes assets whose platform disappeared, creating the release only when it is missing. sd update reads the commit from the git ref API (annotated tags dereferenced) with the nightly.sha asset as the rate-limit fallback, because target_commitish is frozen at creation. A release without nightly.sha is recreated one last time so updaters built before the switch still see a newer build. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly publish: upload assets before moving the tag; treat only 'release not found' as missing Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly publish: upload nightly.sha after the binaries it names Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly publish: key the one-time recreate on the release notes, written before any upload Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly publish: one marker variable for the notes and the recreate probe Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
cbdc1db3fa |
sd mcp: MCP server over the operation registry (#3131)
* sd mcp: MCP server over the operation registry Every registered query, action, validate and preview is an MCP tool served over stdio with rmcp, with input schemas walked from the ops' specta types and a jobs.wait tool to follow jobs. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * docs: note that dotted MCP tool names are deliberate Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * sd mcp: repair a stale library selection; run the suite in the acceptance job Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * sd mcp: jobs.wait finds the job in the library that owns it Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * sd mcp: resolve jobs.wait candidates once and pin the owning library Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
f69c1fee89 |
Add to Library core: defaults and overrides, store placement, volume registration, portable identity, removal keeps the catalog (#3121)
* Add to Library defaults: library settings with per-add overrides Every entry point used to carry its own notion of what a source starts as: volumes.track defaulted external drives to unfiltered, folder tracking to filtered, and the Explorer sent unfiltered: false by hand. The library now holds the defaults under settings.adding (store placement in library, an offline copy kept, filtered capture, content identified), config.library.get reports them and config.library.update changes them for later adds only. sources.track and volumes.track take the same AddOverrides on top; an absent field is the default and an override never writes back. The effective settings are the source's SourceConfig, persisted with the registration and returned in the output together with the saved name and the store path. A name, settings or volume row that fails to persist now fails the add instead of reporting a tracked source. A source added without content identification gets its walk and nothing after. volumes.track keeps the archival behavior by passing an explicit unfiltered override for an external drive unless the caller set one. The CLI gains --on-source, --in-library, --keep-offline-copy and --no-identify on sd source track. Generated TypeScript types regenerated. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Resolve store placement per library in the source layer Placement is intent recorded in each source's config column; the serving device resolves it. VolumeIndex::store_dir is the one resolver: in library under <data>/sources/<id>, on source under <root>/.spacedrive/sources/<id>, following the source's current root so an on-source store tracks its volume across remounts and resolves to nothing while the drive is away. Writer opens, read-only opens, freezes, the status surface and sources.list all go through it, and SourceStore::open takes the resolved stores directory rather than the data-directory layout. On source is refused for network and cloud volumes, whose WAL database cannot live on the share. The volume index also gains register_source_with, which registers under an identity a store on disk already carries, and forget_source, which drops a registration and retires its writer and reader handles without touching the directory. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Adding a path tracks its volume ensure_volume_in_db wrote the row but skipped the sync publication and the VolumeTracked event that track_volume emits, and the add logged its failure and carried on. It now syncs and emits on insert, the add fails when the row cannot be written, and the volume index maps the drive at the same moment so the mount point check and the detached flag apply to the new source. A second folder on the same drive finds the row already there. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Portable source identity and managed-directory exclusions Every store now carries source.json beside data.db: a versioned descriptor with the source id, the library that wrote it, its volume anchor and path within the volume, and its settings. Adding a scope first looks for a store whose descriptor binds to this library, this volume and this path, and registers under that identity, so a removed folder re-added or a drive plugged in with its catalog on it reopens the catalog instead of minting a fresh source over an empty one. A descriptor from another library or another drive is ignored and a new source starts beside it. The own-data-directory check becomes crate::config::is_managed: the data directory under every spelling, any .spacedrive directory and what is under it, and the volume identity marker. It is a path-component check rather than a registry lookup, so it holds for an unfiltered source whose rules are off, for an enclosing source that was never told about the inner store, for a store on a drive nobody here tracks, and across remounts and aliases. The walk, the watcher and the writer all use it. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Removing a source keeps its catalog unless asked to delete it sources.delete used to delete the store directory through the adapter engine before unregistering, which would have turned removal of an on-source store into destruction of the portable catalog, assertions included. It now retires the source's handles, drops the registration and leaves the store where it is; delete_catalog is an explicit second choice. The volume row is never touched: removing the last folder on a drive does not forget the hardware. sd source untrack wraps the action. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Add to Library acceptance suite and status One Core over temporary directories tracks a folder with defaults and one with overrides, proves one volume row for both, finds the on-source store excluded from its own and an enclosing source, removes a source and sees its catalog and record identities survive the re-add, and sees deletion start over. The suite joins the acceptance group of xtask test-core; docs/core/acceptance/add-to-library.md maps the plan's cases to tests and names the rows the core half cannot prove. PROJECT_STATUS and the plan's status line record what landed and what the modal thread builds next. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Review fixes: placement-aware reads and backup, re-track keeps settings, adoption across placements The library's per-source read queries (sources.list_items, list_records, media_listing) opened the store through the engine's in-library layout, so an on-source catalog was unreachable from the explorer; they now resolve a registered filesystem source through the volume index and fall through to the engine only for adapter sources. libraries.backup reads each store from where placement put it. A re-track carries the previous value of every setting the add did not name instead of the library's current defaults, and never changes placement. portable_identity searches both placements so a drive arriving with its catalog is adopted whatever placement the add asked for, and the adopted catalog keeps the placement it was found under. SourceConfig::default agrees with its serde defaults. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Restore names an on-source catalog it cannot put back The archive now holds a store backed up from its drive, but a restore lays every store in the in-library layout, where an on-source registration does not read. The manifest records each source's placement; a restore leaves an on-source catalog where it landed, keeps it out of the restored count, and names it in on_source_catalogs with its path, so the caller knows what was not put back. Moving it onto the drive is relocation. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * catalog_reused means an identity was kept, not that a file exists The acceptance run on CI saw a fresh add report its catalog reused: the content identity job of an enclosing source opened the new source's store between registration and the file probe. The flag now answers from the registration (it existed, or a descriptor-bound catalog was adopted), which is what it claims. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Regenerate TypeScript types for the catalog_reused doc Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Volume tracking test follows the one-library-per-path rule volumes.track now surfaces a source setup failure instead of logging it, so tracking a volume another open library holds is refused. The multi-library test asserts the refusal, then the handover once the first library untracks the drive and removes its source. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
08eb622e29 |
Library backup and restore with manifest verification (#3115)
* Library backup and restore with manifest verification libraries.backup copies library.db, sync.db, every owned source store and its sidecars with VACUUM INTO while the daemon runs, hashes each file with blake3 and writes manifest.json to a directory or .tar.zst archive. libraries.backup.verify rehashes a backup without restoring it. libraries.restore verifies every hash, refuses a backup from a newer schema, stages under the data dir, swaps files into place with the open store and sidecar pools released first, parks the displaced state in restore-trash/, and reopens the library. Replacing a library with other member devices needs force, since their device rows and sync watermarks would diverge. CLI: sd library backup --to, sd library backup verify --from, sd library restore --from [--as-new | --replace] [--force]. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * CLI help: restore-trash is kept, not pruned Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Restore: keep stores closed across the swap, clear the arena, serialize library.json Review findings on #3115: the volume index kept the pre-restore arena and drive snapshots after a replace, a watcher event could reopen the old data.db between the release and the rename, and library.json was copied byte for byte while the daemon rewrites it in place. quiesce_stores takes every affected source's open gate and holds it until the renames are done, closes the cached handles, detaches the library's registrations and partitions, and removes the drive snapshots so the reopened library rebuilds from the restored stores. Backup serializes the open library's config instead of copying the file, and restore parses the staged config before touching the data directory. A failure after the first displacement now names the restore-trash directory. The test feeds source b's watcher during the replace and proves a later write reaches the restored file on disk. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Restore: load the staged config as LibraryConfig, hold sidecars across the swap Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
e58507a1ec |
R8 fixes F1, F2, F3, F5: salvage repair, snapshot quarantine, refused watch state, nested-store search duplicates (#3114)
* Store: unsettle a salvaged-away row so the next observation rewrites it R8 row 3 (F1). When a batch loses a row to a database failure the ledger kept the binding it made before the commit, so the next walk resolved the same file Unchanged and the row stayed missing for as long as its size and mtime held. commit now takes the ledger and unsettles each row it could not land: the uuid keeps its path, the size and mtime evidence is cleared, and the next observation resolves Changed and writes the whole row under the identity the arena already holds. Un-ignores a_repeat_observation_repairs_a_failed_write. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Snapshot: move an unreadable artifact aside instead of deleting it R8 row 10 (F2). load_snapshot_impl removed a snapshot it could not read, decode, or accept the version of, so the artifact was gone before anyone could inspect it. It now moves to <name>.corrupt-<unix seconds> beside the slot with a WARN naming both paths; the slot is clear for the next save and no launch parses it again. Un-ignores an_invalid_snapshot_is_retained_for_diagnosis, which now asserts the bytes survive at the aside path. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Watcher: register a root only after the OS accepts, report and retry refusals R8 row 11 (F3). watch_root registered the root with the volume index before subscribing at the OS, so a refused subscription left the root in watched_paths and is_watched and core.index_status reported a watch that did not exist. The OS subscription now comes first; a refusal is recorded on the volume index with its reason, surfaces as IndexStatus::refused_watches (and in sd index status), and is retried every watch_retry_interval (30 s) while the service runs. A retry the OS accepts moves the root to watched_paths. Un-ignores a_refused_watch_is_not_reported_active and extends it with the status report and the reconciling retry. Regenerates ts-client types. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Search: answer a file under nested sources once from the stores R8 row 25 (F5). With an outer and an inner source both walked, the file under the inner root sits in both stores, and after a restart with no snapshot search_every_index queried each store in turn and returned it twice. The store loop now reads sources innermost first and drops a hit whose local path an earlier partition (an inner store or a drive's arena) already answered, before it is counted, so total_found stays the number of distinct files. Un-ignores a_file_under_nested_sources_is_one_hit_from_the_stores. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Docs: R8 rows 3, 10, 11 and 25 fixed; row 17 remains the R4 brief Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Review: one retained corrupt snapshot, full-arena dedupe seed, reset clears refusals A recurring load failure no longer retains an artifact per launch: when a .corrupt-* sibling exists the slot is removed instead, keeping the oldest copy as the mismatched-root path does. The store de-duplication seed is collected from each arena's full match set before the page window narrows it, so total_found stays exact beyond the page. clear_all forgets refused watches along with the watched paths. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
5bc474e722 |
CLI and daemon housekeeping from the live Mac runs (#3111)
* Default the replication block in core.status so a newer CLI reads an older daemon A |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
10dbc18d0f |
Replica fetch: resumable .part transfers, bandwidth cap, pause, and progress (#3109)
* Resume interrupted replica fetches from a .part file A replica transfer now lands in <source>.<generation>.<blake3 prefix>.part and stays there when the stream breaks or the daemon stops. The next attempt asks the owner for the artifact's tail from the part's length over the new FetchSnapshotFrom and FetchDatabaseFrom requests, whose header still names the whole artifact; a header that names different bytes discards the part and starts over, and the finished file is validated against the header checksum before it replaces the live replica as before. Parts are removed only when the owner's listing no longer carries their generation. A nonzero offset serves the on-disk snapshot without the pre-fetch save, so a resumed copy finishes against the same bytes. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Cap replica transfer bandwidth with a persisted daemon setting replication.max_bytes_per_sec in spacedrive.json bounds every replica artifact served and received through one token bucket (zero is unlimited); the cap applies to transfers already in flight. It is set through config.app.update and sd config set replication.max_bytes_per_sec (bytes, or K/M/G suffixes), read back through sd config get and show. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Pause and resume replica fetching sd sources replication pause stops this device starting replica transfers and halts any in flight at its next chunk, keeping the partial file; resume continues from it and runs a replication pass against every connected device at once. The switch is replication.paused in the daemon config, so a paused daemon stays paused across a restart, and a pass that runs while paused still refreshes listings, owner facts and fleet totals. A pause does not count as a fetch failure, so no backoff accrues. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Show replica fetch progress in sources list, status and events Every fetch in flight is tracked with bytes on disk, total and a recent rate. sources.list carries it on the replica's row (or a replica_fetching row for a first copy that has no share yet), core.status and the new mounts.replication_status query summarize it with the pause switch and cap, and the daemon emits a typed ReplicationProgress event about once a second per transfer plus a finished one when it stops. sd sources list and sd status print the figures; ts-client types regenerated. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Document resumable replica fetch, the cap, pause and progress Source replication gets its own section in networking.mdx with the .part contract, the wire change, the cap, the pause switch and the progress fields; file-sync.mdx points there; the replication plan marks its cleanup item superseded on the replica side and distinct from the restart refetch; PROJECT_STATUS.md gains the register row. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Keep the part when a resumed request gets no header; cap sidecar pages A resumed request that fails outright (dropped link, or an owner too old to know the offset variants) says nothing about the bytes on disk, so the part now stays and the whole-file retry lands on the same name. Sidecar pages count against the bandwidth cap on both ends, as the setting's contract says. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Add the replication field to the test harness config Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Check the pause switch after the cap's wait, not before Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
f9bdf724b6 |
Library join: fix inverted backfill, catch-up loop, reverse device registration, member totals (#3110)
* Networking: serve streams on connections a protocol dials
The connection cache is shared between the dial side and the accept side,
so once a protocol dials a peer, the peer's own send path finds that
connection and opens its streams on it. Nothing read those streams: only
connections the event loop accepted, or those the reconnect loop handed
over, had a stream handler. The peer's request then sat unanswered until
its 30 s timeout while requests in the other direction answered in
milliseconds. This is the reverse RegisterDeviceRequest timeout SPAC-12
saw after join-remote.
get_or_create_connection now hands every connection it dials to the
event loop as TrackOutboundConnection, which spawns the same stream
handler an inbound connection gets. Messaging, sync, and job activity
pass their event loop handle; pairing keeps its own connection handling.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: the device that joins a library is the one that backfills
Every sync service started Uninitialized and backfilled from the first
connected partner, whatever state it already held. When the Mac Studio
joined the MacBook's library the MacBook was the one with a partner in
its device table, so it went Uninitialized -> Backfilling{Studio} and
applied the empty copy's snapshot over its own rows (LWW on the
deterministic default-space UUID). The Studio never backfilled because
the reverse registration had timed out and its device table held only
itself.
A copy of a library created from another device's library is the only
thing that may start Uninitialized. create_library_with_id now writes an
initial_backfill_pending marker into sync.db; PeerSync reads it on open
and starts Ready otherwise. A Ready device never requests a snapshot
(SharedChangeRequest.include_snapshot = false on catch-up) and repeated
catch-up failures no longer escalate to a full backfill. A copy still
waiting for its own first backfill refuses to serve as a join target.
The join carries the rule explicitly: RegisterDeviceRequest has
needs_initial_state, and RegisterDeviceResponse returns the responder's
own device record so the joiner's table is complete when the command
returns, replacing the spawned reverse request. sync_setup fails
instead of reporting devices_registered: true when that record is
missing.
Catch-up completion is recorded per peer in sync.db even when no rows
arrived; the Ready loop catches up once with a peer it has no record
for and no longer re-polls every peer whose watermark is 60 s old. A
missing watermark means never synced, not ancient, so the forced full
sync warning only fires for a real watermark past retention.
library_join_test pairs two daemons for real, joins Bob to Alice's
library, and asserts Alice's rows are unchanged, Bob holds Alice's
device row and spaces, and Alice runs no further catch-ups.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: test that catch-up with an empty peer runs once
Alice holds a volume, Bob holds nothing of his own, so Alice never
writes a per-resource watermark for Bob. Before the per-peer catch-up
record, that made Bob a new peer on every loop iteration and Alice
caught up with him forever (2 -> 5 sessions in 16 s on
|
||
|
Jamie Pineandcapy-ai-staging[bot]
|
5e285d3c97 |
Nightly CLI channel: rolling prerelease workflow, sha-aware self-update, update timer (#3107)
* Add nightly CLI channel: rolling prerelease workflow, sha-aware self-update, update timer Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * TEMP: trigger nightly on PR branch Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Raise the launchd daemon file-descriptor limit to 8192 Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Remove temporary nightly trigger for the PR branch Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly: save the rust cache, note why Linux builds on 22.04 Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Address review: semver-ordered stable updates, assert nightly target commit, quote systemd ExecStart Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly: skip the xtask daemon build, retry release creation Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * TEMP: trigger nightly on PR branch Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly: remove the runner's libunwind before setup.sh Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Remove temporary nightly trigger for the PR branch Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pine
|
a733bca637 |
CLI: rename, undo, mirror, organize, flatten, archive, extract, attributes, link, trash
sd file rename (--to or rules), undo, organize, flatten, archive, extract, attributes, link and trash list|restore|empty, plus sd job journal and --remove-extras on merge. Every one shows its findings and plan before the prompt through one run_with_preflight. |
||
|
Jamie Pine
|
e1e73f2b38 |
Dedupe as a delete target: first copy or chosen copies, read in full before removal
files.delete takes a Duplicates target: a scope with keep first (walk order) or a list of copies to keep with every other copy of their content going. The job reads each surplus copy and its keeper in full before removing, records the hashes it learns, and checkpoints the compare cursor so it resumes. Preflight validates the scope and keep list and previews keepers as skips and surplus copies as deletes. Store gains duplicated_contents_beneath and copies_beneath. files.duplicates is registered as a core query and reports each copy's SdPath. CLI: sd file dedupe with --keep, --keep-under, --min-size, --dry-run. |
||
|
Jamie Pine
|
3cd44f2daa |
Add preflight to actions and build folder merge on it
Two opt-in methods per action, validate: and preview:, over the exact input the action takes. Validation answers findings with stable codes and the facts of the execution, runs again at dispatch and refuses on an error. Preview answers an FsPlan read from the index, kept under a handle so files.directory_listing can serve the directory as it would look after the operation. Copy, move, merge and delete answer both. Merge is new: a resumable job that recurses into matching folders, reads both sides before skipping a duplicate, applies a per-leaf conflict policy, leaves conflicts alone and can consume its sources. Delete's validation counts, across every store, which files are the last copy of their content. The ValidationResult and ConfirmationRequest hook is gone, and so is the CLI's local conflict detection. file copy, merge and delete render preflight before the prompt and take --dry-run. |
||
|
Jamie Pine
|
903c8f2110 |
Delete from a folder what a comparison names
files.delete takes a comparison as its targets alongside a path list, so the client sends five fields and the daemon derives the set from the index as the job runs. The job drains the matcher the compare query pages, checkpoints its cursor after each batch, and for `both` reads whichever side lacks an integrity hash before removing a file, writing what it learned back to the store. Pairs whose bytes differ, or whose copy in B is gone by then, stay and are reported. The CLI gains `sd file delete A --against B --show both` behind a y/N prompt, plus the plain paths form. Compare matched on content uuids, which re-derive from the integrity hash once a copy is verified, so a copy verified on one side stopped matching its unread twin. It now matches on the sampled hash every store keys a content by, with integrity hashes deciding where both sides have them; FsEntry carries both hashes for that. |
||
|
Jamie Pine
|
16325fc5c1 |
Name compared folders A and B, and their sets after a Venn diagram
Left and right only meant first and second argument, and nothing on screen said which folder was which. "Same" meant different things per mode, "changed" implied a direction compare can't know, and "location" collided with the retired locations. The output now names both folders as A and B up front, says how files matched, then counts Only in A, Only in B, In both, and Different (by path only). --by location becomes --by path and the --show values follow, as do the API types, generated TS types, skill, and docs. Single-side listings get Size and Modified columns under a title naming the set, and --help explains each value. |
||
|
Jamie Pine
|
ac49fc1f09 |
Walk every page of a folder comparison in the CLI
files compare printed one page and handed back an --after cursor to paste. The CLI now follows the cursor itself: totals first, then each set's files streaming one per line as pages of 5000 arrive. --limit caps the total and says N of M listed when it cuts a set short, a closed pipe like head ends the walk quietly, and --format json emits one document holding the totals and every entry. The skill doc follows. |
||
|
Jamie Pine
|
e5e6749c26 |
Compare two folders over the source stores
paths.compare answers what differs between two folders from the source stores at query time. By location it merges both sides in index order into only-left, only-right, changed and same. By content it streams one side and asks the other side's stores which content ids they hold, so a renamed or moved copy still matches. Pages resume from a cursor and the first page carries every set's count. Unhashed pairs fall back to size and mtime and get counted, so a page says how much rests on that. sd-cli file compare wraps it. On the store side, files_beneath takes a Start, including one that resumes a nested store at its first directory, and contents_beneath answers the content lookup a batch at a time. Working out which stores a path reaches moved into ops/paths/reach.rs, shared with search.media and paths.context. No catalog.db: the P7 row, the filesystem-intelligence plan, the previs and the design docs now treat cross-source questions as store queries. |
||
|
Jamie Pine
|
8f12892c12 |
Let a daemon start without the default places
The first client to connect starts automatic discovery: it tracks the system volume, adds the home folder as a source and walks every attached drive. For a library meant to hold only what you add, like one whose data directory lives on the drive it catalogs, that indexes the Mac into it. sd start --no-default-sources, and the same flag on restart and on sd-daemon, now leaves those three out. The pass still restores snapshots, heals coverage for sources already tracked and hashes them, so a source you added keeps working across restarts. sd-server keeps the defaults. |
||
|
Jamie Pine
|
8a878a3c9b |
Clear location and entry residue from the code
The error variants, the permissions flag, the sync fixtures, and a long tail of comments still named locations and entries after both left the product. LocationNotFound goes from the query, action, and CLI errors; can_manage_locations becomes can_manage_sources; ChangeConfig's location_root becomes root; and the sync tests key their watermarks, checkpoints, dependency parsing, and messages on models that still sync. The transaction manager's stubbed pre-HLC methods and the bulk operation types only they used are removed, and the spaces invalidation arm stops matching a location resource nothing emits. File transfers and deletes describe their allow-list as the source roots it is. The CLI no longer points a new cloud volume at sd location add. In the clients, mobile loses its location search scope and empty Locations section, an unused overview card mock is deleted, and empty states and copy say folder, place, or source instead of location. |
||
|
Jamie Pine
|
cc5ffc15d3 |
Retire the ephemeral qualifier
The word marked a contrast with the persistent entries index, and that index is gone. ops/indexing/ephemeral folds into ops/indexing, and the names now say what each piece is, keeping the in-memory arena apart from the durable registry and stores: EphemeralIndexCache, ephemeral_cache() -> VolumeIndex, volume_index() VolumeIndex (one drive's slot) -> Partition EphemeralIndex -> Arena EphemeralEventHandler -> FsEventHandler watch_ephemeral / unwatch_ephemeral -> watch_root / unwatch_root IndexerJobConfig::ephemeral_browse -> IndexerJobConfig::new set_ephemeral_index, from_ephemeral -> set_arena, from_arena ops/search/ephemeral_search -> arena_search core.ephemeral_status / ephemeral_reset -> core.index_status / index_reset The CLI's index subcommands become status and reset. Status output loses its always-empty legacy fields, EntryRef loses an id that was always zero, the indexer output loses a field nothing read, and module docs that described browsing outside managed locations now describe the volume index. On-disk snapshots keep the ephemeral.snapshot name: renaming the file would orphan every snapshot and re-walk every drive. The job system's ephemeral jobs, Spacedrop's ephemeral shares, and UI state that is never persisted keep the word, since it means what it says there. |
||
|
Jamie Pine
|
9374855db2 |
Retire the persistent-era search and event surface
Search reports one index, so IndexType and the index_type field are gone, along with the Locations filter kind, the locations filter and facet, the constructors that only existed to stamp Persistent, and the Entry-based enhanced output. Tags and hidden files are advertised filters with no caveat, since both backends apply them. Fifteen Event variants that nothing emits are removed: the Entry events, FsRawChange, the location-keyed Indexing events, and the legacy location, thumbnail, file-operation, and files-modified events, with their helper types. events.list now derives its names from the enum instead of a list that had drifted from it. The CLI printer, the Tauri and ts-client default subscriptions, and the client-side filter types follow, and an uncompiled stale copy of type_extraction.rs goes with them. |
||
|
Jamie Pine
|
318ab3891a |
Replace the library schema and remove locations
FD4 of the final entries drop. One migration drops 29 retired tables inside its own transaction, children before parents: with foreign keys on, dropping a parent makes SQLite parse every child's constraints, and the first attempt failed on sidecar, whose other parent entries was already gone. That attempt also showed Migrator::up leaves SQLite migrations unwrapped, which is why the migration opens its own transaction. Every library now converges on 14 tables. Proven on a copy of the live library, on a library the pre-drop build left half-migrated, on a brand-new library, and on the live Mac library after a backup; every dropped table held zero rows there apart from the seeded kind lookup. Locations leave with the schema: ops, domain type, entity, CLI domain, and every consumer in the web and mobile clients. PathBar pins become space-level Path items listed under Pinned, the sidebar's Places come from a new paths.system_folders query, Add Storage tracks sources and validates through sources.validate_path, device views list sources, and library statistics count sources as source_count. Content kind statistics read the source stores instead of a lookup table. |
||
|
Jamie Pine
|
7d81966c34 |
Move tags onto source stores end to end
tags.create stages a definition in library.db until first application adopts it into a store; apply resolves targets through the arena with the stores as the floor, so a cold daemon and a detached source both answer. Every claim is stamped by a real HLC seeded from DeviceManager; the library-id-as-device placeholder is gone. File targets carry the content key when the store knows it, so tagging a copy reaches the bytes without the client choosing. Writes against a remote-owned source author their rows durably in an assertion outbox and deliver to the owner over remote ops when it answers: sources.assertions.merge commits before replying, the reply is the ack, and redelivery is absorbed by the assertion primary key. Pending claims overlay replica listings immediately. Reads decorate from the stores everywhere: directory listings on both the arena and store paths, files.by_id, files.by_path, replica listings, and files.by_tag, which no longer needs a loaded arena. File::announce decorates its payload too, which is what makes tagging reactive: the event used to carry empty tags and the client faithfully rendered that. Deleted: TagManager, TagClosureService, TagConflictResolver, TagUsageAnalyzer, TaggingFacade, UserMetadataManager, ops/metadata, the validation module, and the semantic-tag field zoo. Tag is uuid, path, name, color, icon. The XOR-era UI gates (the needs-indexing toast and the inspector's isEphemeral message) are gone; the one honest refusal left is a path outside every tracked source. The CLI gains tag unapply and delete. No data migration: the legacy tables were verified empty, and their entities wait for FD3 and FD4. |
||
|
Jamie Pine
|
25fd868a2f |
Add path-scoped thumbnail generation
From a parallel session. thumbs.generate accepts a file or an indexed folder with missing, stale, and force modes, runs in resumable batches, and versions video tiles against host FFmpeg availability so a tile regenerated with better tooling replaces its predecessor in place. File and folder context menus and the new sd thumbs CLI domain call the same operation. paths.context now reports whether the exact path has a committed source record, and the path bar surfaces it with a refresh affordance. Coverage and resume semantics are documented in docs/core/thumbnail-generation.mdx. |
||
|
Jamie Pine
|
c32768f2e4 |
Make startup discovery explicit and permission-aware
Library loading no longer dispatches a filesystem discovery pass. The desktop requests one idempotent pass after the connected shell renders, a bare daemon initiates nothing, and sd index start --defaults gives CLI and server operators the same explicit path. Restored volume snapshots prevent repeat full walks. |
||
|
Jamie PineandClaude Fable 5
|
29b9b8abc5 |
Stream remote logs, sync on pairing, watch remote jobs
Three gaps closed in the remote operation line: Pairing completion now broadcasts ConnectionEstablished from both the initiator and joiner, so peer mounts and job activity subscribe the moment a device pairs instead of waiting for the next reconnect. SubscribeLogs takes a device target. The local daemon opens a log subscription on the paired device over remote-ops — now an enum protocol with a streaming variant — and forwards its log bus frames down the client connection, filtered on the serving side. 'sd --device titan logs follow' tails titan live. LogFilter matching moved onto the type so both paths share it. 'sd job remote --watch' redraws every two seconds, which closes the loop on remotely dispatched work: dispatch with --device, watch it run. Remote job rows also carry updated_at now, and the sweep drops non-terminal rows that stopped reporting, so a completion event lost to a subscription gap can no longer show as running forever. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Jamie PineandClaude Fable 5
|
960febb2b4 |
Target any paired device from the CLI with --device
Adds a device field to the Action/Query envelope and a remote-ops iroh protocol that carries one Wire method call to a paired device, executes it through the same operation registries, and returns the JSON result. The daemon resolves the target by name, slug, or id against its paired devices; only paired devices are served, matching the byterange trust boundary. A forwarded request usually names no library — the target resolves its own open library, since a caller can't know a peer's library ids without asking first. The CLI macros send None whenever --device is set, and library validation skips remote-targeted clients so a foreign library id never lands in the local config. execute_json_operation now delegates to a context-only variant so the daemon socket, embedded hosts, and the network path share one dispatch. Verified live from the Mac against titan: sd --device titan sources list # nine sources over iroh sd --device titan job list # titan's job history sd --device titan sources verify <id> # dispatched and ran on titan Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Jamie PineandClaude Fable 5
|
cf90e61a02 |
cargo fmt
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Jamie PineandClaude Fable 5
|
1766bd175e |
Wire remote job activity end to end
The job activity plane had a subscriber with no caller and a cache that
could never fill: subscriptions were disabled in
|
||
|
Jamie PineandClaude Fable 5
|
7690309d8c |
Give capture policy a type and an op
SourceConfig is the typed shape of the source row's config column, parsed with defaults so old rows and future fields both survive. sources.update renames a source or changes its capture policy, with the one asymmetry named: widening dispatches a walk for what the rules skipped, narrowing removes nothing because hiding is the lens's job. Tracking only ever widens the policy, so a casual re-track cannot demote an archival source. The locations-demoted doc gains the policy taxonomy this settles: capture on the source, attention on the location, display in the lens. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Jamie PineandClaude Fable 5
|
ac9f266ed1 |
Verify duplicate content by reading every byte of it
The sampled tier groups candidates; deleting a copy needs certainty. VerifyContentJob claims "shared content, integrity hash still null" from the store the way the sampled job claims its queue, so it reads exactly the files with duplicates, stays current as sampled results keep landing, and resumes for free. The verdict carries the sampled hash so it upgrades the existing content row, whose uuid moves to the integrity-derived id per the ladder. sources.verify dispatches it, sd sources verify from the CLI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Jamie PineandClaude Fable 5
|
50ab6038e2 |
Freeze a source before reorganising it
sources.freeze writes a dated VACUUM INTO copy of the source's store under sources/<id>/freezes/, verified by reading the record count back out of the artifact. The live store keeps mirroring the disk; the freeze is the record of what the disk was before a cleanup. New sd sources CLI domain: track, list, freeze. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Jamie PineandClaude Opus 5
|
59520ee5e9 |
Demote a location to a pin
A location was fifteen fields rooted at an entry. It is six now: a name, the source holding it, a path relative to that source's root, and whether someone put it there. It owns nothing, so unpinning is a DELETE of one row and the volume index does not notice. Everything dropped lives somewhere that already had it, and mostly had it better. scan_state and last_scan_at are the source's. total_size and file_count come off the arena's rollups, which are current rather than as of the last scan and can say a subtree was summarised where a stored integer cannot. index_mode and job_policies configured an indexer and a job pipeline that are both gone. entry_id is the ownership this phase exists to remove. The path is relative so a pin survives its drive mounting somewhere else, which is the whole reason a source carries a root and a location does not. Sources nest, so the innermost one containing the path wins. IndexMode goes with it. It was set on every IndexerJobConfig, threaded through the walk, and never read: the phase that consulted it was deleted with the persistent indexer, and content identity is its own job now. Four CLI flags and two modals were choosing between values nothing looked at. No rows carry over. Every one was anchored to an entry_id in a table this series removes. The five folders a person recognises are written at library creation instead, which is why there is no longer anything to add: Home, Desktop, Documents, Downloads and Pictures are simply there. Three more things fell out. File::from_entry_uuids and its siblings became File::at_path, for_record and announce, which is the pattern five call sites had each written by hand. SdPath::Content resolves through copies_of_content rather than joining entries to locations to devices, so it finds a copy on any drive rather than only where a location happened to be. And the peer file protocols allow source roots instead of location paths, which reaches at least as far and costs no query. One thing removed rather than ported: user_metadata_tag checked that a file-scoped tag came from the device owning the entry, via the location holding it. A record belongs to whichever machine walked it and carries no device column, so that check wants re-deriving when sync returns on top of sources rather than being rewritten to always pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jamie PineandClaude Opus 5
|
f32da47771 |
Tag the bytes or tag the file, and mean it either way
user_metadata already had the right shape: entry_uuid or content_identity_uuid, exactly one set. Nothing needed redesigning. The uuids just start meaning the record the volume index minted and the content uuid the source store derives from the bytes, and the joins that went through the entry table go through the arena instead. Tags stay in library.db, which is right rather than a compromise. A tag is a claim by a person, so it syncs and travels with the library. That is the opposite of a sidecar, which is a property of the bytes and belongs beside them. TagTargets loses a variant it cannot have. Entry(Vec<i32>) addressed rows by database id and there are no rows and no ids; the client never sent it. What is left is Content and File, which is the real distinction: tag the bytes and reach every copy on every drive, or tag this one. The client already chose content when the file had been identified and fell back otherwise, so the policy was already right and only the names were wrong. ApplyToTargets was a second copy of the same enum with the same three variants. tags.create takes TagTargets now, and both inputs share one validate. Three helpers landed on the cache because four call sites wanted them: content_of, copies_of_content and path_of_record. alternate_instances and files.by_id had hand-rolled versions of the first and last; they use these now and lost 100 lines between them. Content-scoped changes emit events for every copy of the bytes, not just the one that was named. Tagging a photo on one drive updates the row showing it on another. Two dead things fell out: find_entries_by_semantic_tags and find_files_by_tags, both returning Vec<i32> entry ids, neither called. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jamie PineandClaude Opus 5
|
c78cf8f319 |
Delete what the entries world was still holding up
Four things with no callers, found by asking rather than assuming. file_sync built a conduit between two locations and copied files along it; `file_sync_service()` is constructed lazily and nothing ever asks for it. locations.export and import dumped the entry tables to SQL and read them back, unreachable from client or CLI, and a source store is already a single SQLite file someone can copy. search/filters.rs built sea-orm conditions over entry columns for the FTS search that went in 7d41e07ab; it looked live because ops::search re-exports it. hierarchy.rs answered descendant queries from the closure table, and its last caller left with the persistent indexer. Library statistics were the real find. They summed entry.size over the closure table, so with nothing writing entries they had quietly become zero, and there were two copies of every calculation, static and instance, that had to agree. Now each source reports its own totals and the two versions are one. That needed the store to count more than it did. counts() answered records and bytes; it answers directories and contents too, in the same query. Files are records less directories, because a folder is a record and nobody asking how many files they have means folders. content_count is a new column on sources, since record_count counts rows and unique_bytes counts bytes and neither says how many separate files a drive holds once copies fold together. Unique content is per-source and will over-count across drives until the catalog exists: two drives holding the same photo report one each. That is visible on the overview, and it is the same gap files.duplicates documented. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jamie PineandClaude Opus 5
|
7d41e07ab9 |
Read files from the arena, not from entries
Every query that answered from `entry` now answers from the volume index, which is the one that knows about the whole drive. Each of these had both paths already and picked between them at runtime; the entries branch was the one that only worked where someone had added a location. search collapsed hardest. IndexType existed to choose between an FTS5 index over entries and the in-memory one, and there is only the one now, so fast, normal and full modes were three routes to the same table. A library-wide search fans out over every partition, which is the search that was never possible before: it used to mean "every location you added". SearchScope loses Location. Nothing called it, and under the demotion a location is a subtree rather than a thing you search inside. files.media_listing walks the arena instead of joining four tables, and sorts DateTaken by birth time, which is the closest thing the arena knows and the one a photo library actually wants. files.alternate_instances asks content identity rather than content_id, so it spans drives: every source is asked about the same content uuid and they agree because the id comes from the bytes. Two new store queries back it, content_of and copies_of_content. Copy estimates read subtree_bytes and file_count off the arena. Those rollups are maintained as the arena is written, so the progress bar gets its numbers from a lookup rather than from a walk. files.unique_to_location had no callers and no meaning once a location stops owning records. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jamie PineandClaude Opus 5
|
9d5465832f |
Stop writing entries
Nothing produces a row in `entry` after this. The walk fills the volume index and its source store, which is what the arena has been doing since the storage map landed; the persistent half of the indexer was a second writer to a second store that no source ever read. Gone: IndexPersistence, and with it the branch in every phase. The state machine is Discovery then Processing, since Aggregation existed to bubble directory sizes the arena now maintains incrementally, and ContentIdentification is ContentIdentityJob's job against the source store. DatabaseAdapter, PersistentEventHandler and the location watching that fed them go too, so ArenaWriter is the only change handler and the walk and the watcher differ only in Notify. DatabaseStorage was 1230 lines of entry writes wrapped around three things the arena actually uses. Those are ops::indexing::metadata now: EntryMetadata, extract_metadata and get_inode, plus is_hidden_path. delete_subtree moves onto the entry entity, which is what it operates on and what it dies with. locations.rescan, locations.enable_indexing and locations.triggerJob dispatched the deleted indexer or its media jobs, so the ops and their UI go. Reindex and Run Now are gone from the inspector; the job policy toggles stay, since they outlive the ops that read them. Adding a location no longer dispatches a walk, which is the point: the drive is already mapped. The entry tables and their readers are still here. That is the next commit. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
Jamie Pine
|
8f2d1dbf3a |
Watch a source whose index came back from a snapshot
An index becomes browsable two ways and only one of them armed a watch. A walk finishes and the indexing job watches what it walked. A restart restores the same index from a snapshot and watched nothing, so the drive browsed perfectly and reported no changes until it was indexed again. Measured on the running daemon: one source, /System/Volumes/Data, 2.1M entries restored, no ephemeral index job this session, and a real file created on the desktop produced no ResourceChanged on the event bus. The cache now announces a root when its snapshot restore populates the index, and the watcher service arms a watch for each. A channel rather than a handle because the watcher service already holds the cache. Announced once per source per session, and watch_ephemeral returns early for a root it already watches, so nothing accumulates a reference count it never sheds. core.ephemeral_status reports watched roots, and `sd-cli index ephemeral-cache` prints them even when the list is empty. Empty is the whole answer here, and reading it took a code walk twice. |
||
|
Jamie Pine
|
2ab99b036c |
Surface indexed drives and persistent counts in the interface
Sidebar lists indexed drives including detached ones, volume rows and overview cards carry persisted entry counts, and the explorer gains a list view over daemon ops with event-driven refresh. |
||
|
Jamie PineandClaude Opus 4.7
|
60369e9f00 |
feat: redundancy awareness & cross-volume file comparison (#3053)
* feat: add redundancy awareness & cross-volume file comparison Surface content redundancy data so users can answer "if this drive dies, what do I lose?" — builds on existing content identity and volume systems. Backend: - New `redundancy.summary` library query with per-volume at-risk vs redundant byte/file counts and a library-wide replication score - Extend `SearchFilters` with `at_risk`, `on_volumes`, `not_on_volumes`, `min_volume_count`, `max_volume_count` filters - Add composite index migration on entries(content_id, volume_id) Frontend: - `/redundancy` dashboard with replication score, volume bars, at-risk callout - `/redundancy/at-risk` paginated file list sorted by size - `/redundancy/compare` two-volume comparison (unique/shared toggle) - Sidebar ShieldCheck button linking to redundancy view Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * redundancy UI improvements + ZFS volume detection fix Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix ZFS pool capacity reporting and stats filtering - ZFS: override total_capacity for pool-root volumes using zfs list used+available. df under-reports pool-root Size because it only counts the root dataset's own used bytes plus avail — on a 60 TB raidz2 pool this shows as ~15 TB instead of ~62 TB. The pool root's own used property includes descendants, so used+available is the real usable capacity. - Library stats: drop volumes where is_user_visible=false AND re-apply should_hide_by_mount_path retroactively so stale DB rows (detected before the Linux visibility filters existed) don't inflate reported capacity. - Extract should_hide_by_mount_path into volume/utils as a shared helper used by both the list query and the stats calculation. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * show capacity and visibility in sd volume list Makes it possible to verify library-level capacity aggregation from the CLI — previously the list only showed mount, fingerprint, and tracked/mounted state, which meant debugging the ZFS pool capacity issue required querying the library DB directly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * document filesystem support matrix and detection New docs/core/filesystems.mdx covering per-filesystem capabilities (CoW, pool-awareness, visibility filtering, capacity correction), platform detection strategies, the FilesystemHandler trait, Linux/ macOS/ZFS visibility rules, the ZFS pool-root capacity problem and fix, copy strategy selection, and known limitations. Registered under File Management in both mint.json and docs.json. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * WIP: redundancy filter wiring across search, CLI, and UI - core/src/ops/search: wire redundancy filters (at_risk, on_volumes, not_on_volumes, min/max volume_count) through the search query; fix UUID-to-SQLite BLOB literal so volume UUID comparisons actually match (volumes.uuid is stored as a 16-byte BLOB, quoted-string comparison silently returned zero rows). - apps/cli: new redundancy subcommand + populate the new SearchFilters fields from search args. - packages/interface: redundancy at-risk and compare pages reworked to consume the new filter surface; explorer context/hook updates to support redundancy-scoped views. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * add web context menu renderer and UI polish - New WebContextMenuProvider + Radix DropdownMenu-based renderer anchored at cursor via a 1x1 virtual trigger. Handles separators, submenus, disabled, and the danger variant via text-status-error. - useContextMenu now routes web clicks through the provider instead of parking data in unused local state, and trims leading/trailing/adjacent separators so condition-filtered menus don't render orphaned lines. - Drop app-frame corner rounding on the web build. - Add shrink-0 to the sidebar space switcher so the scrollable sibling can't compress it vertically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * sd-server dev workflow: auto web build, shutdown watchdog, stable data dir - build.rs runs `bun run build` in apps/web so `just dev-server` always embeds the latest UI. rerun-if-changed covers apps/web/src, packages/interface/src, and packages/ts-client/src so Rust-only edits skip the rebuild. Skips gracefully when bun isn't on PATH or SD_SKIP_WEB_BUILD is set; Dockerfile sets the latter since dist is pre-built and bun isn't in the Rust stage. - Graceful shutdown was hanging because the browser holds the /events SSE stream open forever and axum waits for all connections to drain. After the first signal, arm a background force-exit on second Ctrl+C or 5s timeout so the process can't stick. - Debug builds were starting from a fresh tempfile::tempdir() on every run (the TempDir handle dropped at end of the closure, deleting the dir we just took a path to). Default to ~/.spacedrive in debug so data persists and `just dev-server` shares a data dir with the Tauri app. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * add Sources space item alongside Redundancy in default library layout Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * add TrueNAS native build script Uses zig cc as C/C++ compiler on TrueNAS Scale where /usr is read-only and no system gcc exists. Dev tools live at /mnt/pool/dev-tools/ (zig, cmake, make, extracted deb headers). Builds sd-server + sd-cli in ~4 min on a 12-core NAS. AI feature disabled (whisper.cpp C11 atomics incompatible with zig clang-18). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * never block RPC on synchronous statistics calculation On first load (fresh library, all stats zero), libraries.info used to calculate statistics synchronously before responding. On large libraries during active indexing this hangs indefinitely — the closure-table walk in calculate_file_statistics loads every descendant ID into a Vec then issues a WHERE IN(...) with millions of entries, which SQLite can't finish while the indexer is writing. Now always return cached (possibly zero) stats and let the background recalculate_statistics task fill them in. The UI refreshes via the ResourceChanged event when the calculation completes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * self-heal protocol handler registration on re-init Core::new() registers default protocol handlers after starting networking, but swallows any failure (error is only logged). If the initial registration fails — e.g. on a host where start_networking hasn't fully set up the event loop command sender by the time register_default_protocol_handlers runs — the registry is left empty. A subsequent call to Core::init_networking() would see `services.networking().is_some()` and skip re-registration, permanently leaving protocols unregistered for the life of the process. sd-server calls init_networking() right after Core::new(), so it's the client most exposed to this. Symptom: pairing over the web UI returns "Pairing protocol not registered" while the same library works fine from Tauri and mobile. Fix: init_networking now queries the registry directly for the pairing handler and re-registers the default set if it's missing, independent of whether networking is already initialized. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fall back to pkarr+DNS discovery when mDNS port is unavailable Iroh's endpoint.bind() fails wholesale if any configured discovery service fails to initialize. MdnsDiscovery requires binding UDP :5353, which on most Linux systems (including TrueNAS) is already owned by avahi-daemon. Result: endpoint creation errors out with "Service 'mdns' error", the event loop never starts, command_sender stays None, and protocol registration fails — so sd-server has no working networking at all. Make mDNS best-effort: on any error whose message mentions "mdns", retry endpoint creation with only pkarr + DNS discovery. Local-network auto-discovery is lost but remote pairing via node ID (which uses n0's DNS infrastructure, not mDNS) continues to work normally. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * succeed pairing if either mDNS or relay discovery wins The dual-path discovery in start_pairing_as_joiner_with_code used tokio::select! to race mDNS and relay. select! resolves on the first branch to complete — including errors — so a host that can't bind mDNS (e.g. a Linux box where avahi already owns UDP :5353) would fail pairing wholesale: mDNS discovery errors out in <1ms with "Failed to create mDNS discovery: Service 'mdns' error", that Err wins the race, and relay discovery gets cancelled before it can even begin. Switch to futures::select_ok so we only return the error if EVERY discovery path has failed. mDNS failing immediately now leaves relay running to completion, which is the common case for remote pairing into a NAS. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Jamie Pine
|
b23ae40a01 | v2.0.0-alpha.2 | ||
|
Jamie Pine
|
9de7ee88ed |
Update versioning across multiple packages and enhance GitHub Actions workflow
- Changed versioning from "2.0.0-pre.1" to "2.0.0-alpha.1" in various Cargo.toml files and package.json to reflect a new alpha release. - Added support for ARM64 builds in the GitHub Actions workflow, including a new host configuration and installation of VA-API libraries for the ARM target. - Introduced a new `bump` command in the xtask to facilitate version updates across all relevant files, improving version management. |
||
|
StarbirdTechandClaude Opus 4.5
|
74d3b82a87 |
feat(core): add speech-to-text feature flag for optional AI dependencies
Introduces a proper feature flag hierarchy for AI-related dependencies, allowing lite builds without heavy speech-to-text dependencies. Feature hierarchy: - `ai` - umbrella feature for all AI capabilities - `speech-to-text` - transcription capability (requires ffmpeg + whisper) - `whisper` - Whisper speech recognition engine (internal) - `ffmpeg` - audio/video processing (existing) Changes: - Make whisper-rs, hound, rubato optional deps behind `whisper` feature - Add `speech-to-text` feature that requires both ffmpeg and whisper - Add `ai` umbrella feature that enables all AI features - Update all cfg guards to use semantic `#[cfg(feature = "speech-to-text")]` - Expose new features in apps/server and apps/cli - Update release workflow to include `ai` feature (maintains existing behavior) This enables: - Mobile builds without STT (smaller binary, already using default-features=false) - Desktop "lite" builds for users who want minimal size - Future AI features can be added under the `ai` umbrella Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> |
||
|
Jamie Pine
|
388b4dde28 | cargo fmt | ||
|
Jamie Pine
|
5d9a6918db | Merge branch 'main' into cursor/device-proxy-pairing-651c |