mirror of
https://github.com/spacedriveapp/spacedrive.git
synced 2026-11-03 13:55:30 -05:00
nightly
428
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Jamie Pineandcapy-ai-staging[bot]
|
26b57e1be3 |
Move 2.0 Beta date mentions to November 15, 2026 (#3164)
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
fc1798b05a |
Windows unit job gates main; daemon raises its fd limit; sd-version marker test (#3153)
* CI: the Windows unit job gates main Core Tests (windows unit) was green twice on main ( |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
a2831ac9b2 |
Run the pairing suites in CI: device persistence, proxy pairing, and relay pairing through a local relay (#3152)
* Run the pairing suites in CI with condition waits and a local relay Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Say what the integration group's wall-time budget is and which relay the mDNS suites still reach Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Name where the integration group's 15 minute target comes from Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Keep a newer same-ALPN connection alive when an older one to the same node closes The connection map is keyed by node and ALPN, so an incoming pairing connection and an outbound one opened for a single message share a key. When the older one closed, the handler and the per-node watcher removed the key and took the newer connection's last handle, so QUIC closed it under the peer before the message had been read. Carol's proxy acceptance to Alice was lost this way in 1 of 13 proxy_pairing_test runs in sequence. Removal now checks the stored connection is the closed one, and a one-shot send waits for the peer to acknowledge the bytes before returning. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Wait for the message acknowledgement off the event loop and say when a closed connection's key is kept Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
e4b4cb71f5 |
Capture time and place in listings and search, read from the image facet on both backends (#3151)
* Capture time and place in listings and search, read from the image facet on both backends search.files sorts by CapturedAt and filters by a CapturedAt range and has_location; files.directory_listing sorts by datetaken; the media listing's datetaken reads the facet. The store joins facet_image into every entry read, guarded by the row's content hash, and the arena path reads the facet from the store ahead of its sort and filter, so both backends hand the same File to one predicate and one order (R8 row 19, new row 29). File.image_media_data is filled from the facet on every path; the inspector shows the capture time and the coordinates; Date Taken is in every explorer view's sort menu. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * search.media judges the capture filters on its rows and leaves replicas out under them; a directory listing reads facets ahead of the sort only for a capture order Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * search.media judges a replica's rows under a capture filter as search.files does instead of dropping the partition Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
15d7df1808 |
Windows: build and unit-test the daemon and CLI on a Windows runner, fix what it found, publish Windows nightlies (#3148)
* ci: build and unit-test sd-core and sd-cli on a Windows runner The job is the first Windows build since the rewrite and the evidence for the cross-platform release gate. It runs on GitHub's hosted windows-latest (Blacksmith has no Windows), builds sd-cli and sd-daemon in debug, then runs the sd-core lib tests and the sd-cli tests. continue-on-error keeps it from gating main until it has been green twice. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * cli: name the daemon binary and replace binaries the way Windows allows The CLI built the daemon's path as sd-daemon everywhere, which does not exist on Windows; sd_client::daemon_binary_name appends the platform's executable suffix. sd update renamed the staged file over the running binary, which Windows refuses for a loaded executable: it now moves the current binary to .bak first and renames the new one into place, removing the backup where the platform lets it and on the next update otherwise. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * indexing: keep the hidden bit in the arena instead of re-reading it on every read The arena rebuilt is_hidden from the path on every entry read. On Unix that is a name check; on Windows it is a GetFileAttributesW call per entry, and it answers false for anything not on disk at that moment, so search returned hidden files from a snapshot or a detached volume. PackedMetadata now carries a hidden bit (size drops to 59 bits), the arena records what the walk saw, and the writer judges a change by the metadata it carries. Snapshot restore recomputes the bit from names where names decide. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * volume: tell a Windows mount point from the directory a drive left behind is_mount_point answered by existence on Windows, so the empty folder an unplugged mapped drive leaves passed as mounted and the detached volume's source could be dispatched at it. GetVolumePathNameW names the mount point of the volume holding a path; a directory that is its own volume root is a mount point, anything else resolves to a parent volume. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * core: let three path-shaped unit tests read the same on Windows The compare and media tests compared displayed paths against slash-written expectations, and the track action test offered a Unix absolute path that Windows reads as relative. The tests now normalise separators and pick an absolute root the platform accepts; what they assert is unchanged. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * files: delete a moved source when the filesystem still has it, not when the volume index guesses After a streaming copy in move mode the source was deleted only when the volume manager said the two paths were on different volumes. On Windows it answered same-volume for a copy the router had already decided to stream, so the file stayed in both places and the move reported success. The job now checks whether the source still exists after the strategy ran and removes it then, which is the condition the comment described. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * ci: save the Windows rust cache on a red run too Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * ci: publish Windows x86_64 nightly binaries The nightly matrix gains x86_64-pc-windows-msvc on windows-latest, producing sd-windows-x86_64.exe and sd-daemon-windows-x86_64.exe with checksums. The build and prepare steps run under bash on every host, and the asset names carry .exe, which the updater already strips when matching. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * docs: Windows gap report and register row docs/plans/2026-10-09-windows-gap.md records the first Windows build since the rewrite: what compiled, what passed, each failure with its cause and fix, what the integration, FDA, R8 and locked volumes suites need to run on Windows, and the beta.1 recommendation with release notes text. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * review: judge mount points on the path as given, mark placeholder directories hidden, keep the rename-over update on Unix GetVolumePathNameW judges the string it is handed, so a subst or mapped drive letter must not be canonicalized to its target first or its own root reads as not a mount point. Arena::ensure_directory creates directory placeholders the walk has not statted; they now carry the hidden answer the path gives until the walk corrects them. The updater moves the running binary aside only on Windows, which needs it; elsewhere a .bak copy and a rename over the target keep a working file at the path at every instant. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
7cab609f2c |
Read EXIF into the image facet behind content identification (#3146)
* Read EXIF into the image facet behind content identification An exif_enrich job follows every identification pass, claims image records whose facet_image row is missing or names other bytes, parses them on the blocking pool through sd-media-metadata, and writes one row per content hash to every record holding those bytes. The row carries the hash, so a changed file re-queues itself once it has a new identity and nothing is read twice. A source with content identification off is never enriched. records.exif answers from any hashed row and logs when it has to parse. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Index the facet fan-out, bound raw reads, and check the row's hash in records.exif The fan-out lookup spelled COALESCE(sampled, integrity) = ?, which no index serves; it is now one indexed arm per hash column. Four files parse at a time instead of 32, since the EXIF reader loads TIFF-shaped containers whole. The job sweeps once more from the start before it exits, so a photo re-identified behind its cursor is not left for the next pass. records.exif answers from a facet row only when the row's hash is the record's current content, and parses otherwise. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Skip rows the first sweep attempted when the metadata pass sweeps again A file that could not be opened stays pending, so the second sweep opened and counted it a second time. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Key the metadata pass's attempted set on row and hash A record whose bytes changed behind the cursor presents a new hash and is read again on the second sweep; an unreadable file presents the same one and is still skipped once. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
6dc6c2bef8 |
Housekeeping: drop sources triggers, remove libraries.export stub, prune restore-trash (#3143)
* ci: drop the sources branch from nightly and core tests triggers Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * core: remove the libraries.export stub left behind by backup/restore Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * core: prune restore-trash after a successful library restore A restore --replace parks the displaced library and store files under restore-trash/<library>-<time>/ and never removed them. Once the restored library is open, keep the newest backup.restore_trash_keep entries for that library (default 2, settable with sd config set) and remove the rest, never the one the restore just parked. Removed paths are logged and returned in the restore output. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * tests: add the backup field to the sync harness AppConfig literal Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
731744277e |
Add to Library step 4: keep an offline copy of an on-source catalog (#3142)
* Add to Library step 4: keep an offline copy of an on-source catalog as a library replica Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Say where a detached source's catalog is: offline copy, or on the drive Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Retire only a reader on the offline copy when publishing; drop the unused apply_setting Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Fill the new field in the CLI's source row test Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Fill offline_copy in sources.get; show the catalog hint on the card whether or not the drive has a badge Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
a19553f6f1 |
Locked volumes L5: volume state in the app, CLI and peer volume facts (#3141)
* Locked volumes L5: volume state in the app, CLI and peer volume facts RemoteVolumeInfo carries VolumeState and a device publishes its tracked unmounted and locked volumes, so a peer lists the owner's dataset as locked instead of a replica that stopped updating. RemoteSourceInfo carries each source's volume_state and sources.get reads the live index. VolumeMountChanged carries the state and fires on any state change, including a key unloaded from an already unmounted dataset. The CLI event printer names a lock; sd sources list has a State column and sd volumes list a State line. The sidebar, Storage cards, source detail and Overview show Locked, Unmounted or Offline and say the index stays browsable. The Add to Library modal refuses a locked or unmounted drive with the reason. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Refuse an add only for this device's away drives; printer falls back to is_mounted without a state Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Say why RemoteVolumeInfo.state stays last Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Hold the Add to Library refusal until the device list resolves Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
306f97fda5 |
Locked volumes L3 and L4: ZFS dataset state, volume index follows volume events (#3140)
* Locked volumes L3 and L4: ZFS dataset state in detection, volume index follows volume events A ZFS dataset that is unmounted or whose key is not loaded is now a volume detection returns, with locked set from keystatus, so sources.list and volumes.list report it locked rather than unmounted or empty. The volume index follows the volume manager's events instead of the monitor calling it: an unmount or lock drops the watches the OS dropped, a mount or key load re-arms them on the mounted filesystem and puts identifications that failed while the volume was away back in the pending set. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Guard the identity file write and auto-track on mounted volumes, resync the index after a lagged event window, fall back to the basic zfs list columns Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Run the ZFS locked volumes suite on a GitHub-hosted ubuntu-24.04 job, whose kernel carries the zfs module Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Point the acceptance matrix at the zfs job and run its suite single-threaded Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
2d781bf4b1 |
Rebuild a drive's arena from its source stores when the snapshot is missing or quarantined (#3139)
* Rebuild a drive's arena from its source stores when the snapshot is missing or quarantined Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Store rebuild runs off the request path and skips a partition a restore already filled Every reader on a drive funnels through the restore gate, so the fill now runs on its own task: the gate resolves as soon as it starts, roots stay in progress until they land and listings serve the store meanwhile. A partition rebuild_quiesced already filled resolves as restored without a second read. The fill reads a rebuild-only select without the content join: 18.9 s for a 1.01M-record store in release, from 33.1 s. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * A browse beneath a source being filled serves the store instead of clearing the subtree is_under_indexing is the ancestor-aware check: a store fill or recursive walk over a root covers everything beneath it, and a browse dispatched into that tree ran clear_for_reindex over what the fill had already placed. The acceptance test lists a subdirectory during the fill and the deepest one after it. REBUILD_PAGE moved above the fill's doc block. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Scope the mid-fill store answer to roots being filled from a store indexing_in_progress also holds walk roots, whose stores cannot promise an answer beneath them, so the prefix gate reads a separate filling_from_store set and falls through to the ordinary dispatch when the store has no row. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
f9d89c1d71 |
R4 arena compaction: reuse vacated slots, compact inflated arenas, snapshot live nodes only (#3138)
* R4 arena compaction: reuse vacated slots, compact inflated arenas, snapshot live nodes only Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * R4 arena compaction: reuse vacated slots, compact inflated arenas, snapshot live nodes only Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Arena: removing a directory takes its subtree so no parent link survives slot reuse Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Arena: rename moves a subtree under its new path instead of dropping it Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Arena::rename: keep the moved entry's own path spelled as given Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
26b67ff0d1 |
Extension file kinds K3 to K5: client renderer registry, viewer bundles, photos viewer (#3136)
* Extension file kinds K3 to K5: client renderer registry, viewer bundles, photos viewer Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Review: allow viewer bundle imports under the desktop CSP, clear the bundle host on unmount Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
0e76551463 |
Nightly release: edit in place instead of delete and recreate (#3137)
* Nightly release: edit in place instead of delete and recreate Every gh release create notified everyone watching the repository for releases, about ten times a day. The publish job now moves the nightly tag with git, edits the release title and notes, uploads assets with --clobber and removes assets whose platform disappeared, creating the release only when it is missing. sd update reads the commit from the git ref API (annotated tags dereferenced) with the nightly.sha asset as the rate-limit fallback, because target_commitish is frozen at creation. A release without nightly.sha is recreated one last time so updaters built before the switch still see a newer build. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly publish: upload assets before moving the tag; treat only 'release not found' as missing Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly publish: upload nightly.sha after the binaries it names Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly publish: key the one-time recreate on the release notes, written before any upload Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly publish: one marker variable for the notes and the recreate probe Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
0e5340394e |
Extension file kinds K1+K2: manifest kinds, layered registry, stored kind_name (#3135)
* Extension kinds K1: manifest kinds, layered FileTypeRegistry::current, conflicts in extensions.list A manifest may declare kinds (name, built-in parent, extensions, MIME types, magic bytes, preview). The registry becomes the built-in table plus every loaded extension's kinds in load order, swapped by the plugin manager on load and unload; the five builtin() callers and the context field read current(). A kind that would redefine a built-in extension is refused at load. load_all sorts directory names so a contested file extension resolves the same way on every machine; the dropped claim is reported by extensions.list and its magic patterns stay available. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Extension kinds K2: store schema v2 kind_name, identity phase writes kinds, reidentification on load and open content.kind_name (nullable, no backfill) and idx_facet_file_extension over lower(extension) arrive with schema version 2; a replica still at version 1 reads through a per-connection temp view that adds the column as NULL. The content identity phase stores the kind for every file and the kind name for extension kinds, reading the first 8 KB only when an extension kind's magic patterns are in play. Rows identified before an extension existed are named by extension when the extension loads and when a store opens. File gains content_kind_name and reads content_kind from the row when present. Photos declares raw; the test extension declares fake with a second fixture contesting it for the WASM suite. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Carry the stored kind across a verification write, filter store search on it, order unclaimed magic candidates by load bind_content takes the previous row's kind and kind_name onto the confirmed row when the write carries none, so a verification pass no longer drops what the identity phase stored. passes_store_filters answers a content_types filter from the stored kind before deriving from the name, matching the listing. magic_candidates for an extension nothing claims walks extension kinds in load order, so a header matching two kinds resolves the same way on every start. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Carry the stored kind only when the write describes the same bytes Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Add kind_name to the dedupe test's content identities Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
cbdc1db3fa |
sd mcp: MCP server over the operation registry (#3131)
* sd mcp: MCP server over the operation registry Every registered query, action, validate and preview is an MCP tool served over stdio with rmcp, with input schemas walked from the ops' specta types and a jobs.wait tool to follow jobs. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * docs: note that dotted MCP tool names are deliberate Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * sd mcp: repair a stale library selection; run the suite in the acceptance job Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * sd mcp: jobs.wait finds the job in the library that owns it Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * sd mcp: resolve jobs.wait candidates once and pin the owning library Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
746bebb2d5 |
Add to Library modal: one setup modal for folders and drives over sources.track (#3126)
* Add to Library modal: one setup modal for folders and drives over sources.track Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Add to Library modal: one setup modal for folders and drives over sources.track Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Add to Library modal: pass the volume spelling from suggested folders, save only the user's changes as defaults, rank the containing volume by the matching mount Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Merge main; wasm_extension_test uses TrackSourceInput.overrides (field renamed in #3121, broken by #3124) Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
f69c1fee89 |
Add to Library core: defaults and overrides, store placement, volume registration, portable identity, removal keeps the catalog (#3121)
* Add to Library defaults: library settings with per-add overrides Every entry point used to carry its own notion of what a source starts as: volumes.track defaulted external drives to unfiltered, folder tracking to filtered, and the Explorer sent unfiltered: false by hand. The library now holds the defaults under settings.adding (store placement in library, an offline copy kept, filtered capture, content identified), config.library.get reports them and config.library.update changes them for later adds only. sources.track and volumes.track take the same AddOverrides on top; an absent field is the default and an override never writes back. The effective settings are the source's SourceConfig, persisted with the registration and returned in the output together with the saved name and the store path. A name, settings or volume row that fails to persist now fails the add instead of reporting a tracked source. A source added without content identification gets its walk and nothing after. volumes.track keeps the archival behavior by passing an explicit unfiltered override for an external drive unless the caller set one. The CLI gains --on-source, --in-library, --keep-offline-copy and --no-identify on sd source track. Generated TypeScript types regenerated. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Resolve store placement per library in the source layer Placement is intent recorded in each source's config column; the serving device resolves it. VolumeIndex::store_dir is the one resolver: in library under <data>/sources/<id>, on source under <root>/.spacedrive/sources/<id>, following the source's current root so an on-source store tracks its volume across remounts and resolves to nothing while the drive is away. Writer opens, read-only opens, freezes, the status surface and sources.list all go through it, and SourceStore::open takes the resolved stores directory rather than the data-directory layout. On source is refused for network and cloud volumes, whose WAL database cannot live on the share. The volume index also gains register_source_with, which registers under an identity a store on disk already carries, and forget_source, which drops a registration and retires its writer and reader handles without touching the directory. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Adding a path tracks its volume ensure_volume_in_db wrote the row but skipped the sync publication and the VolumeTracked event that track_volume emits, and the add logged its failure and carried on. It now syncs and emits on insert, the add fails when the row cannot be written, and the volume index maps the drive at the same moment so the mount point check and the detached flag apply to the new source. A second folder on the same drive finds the row already there. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Portable source identity and managed-directory exclusions Every store now carries source.json beside data.db: a versioned descriptor with the source id, the library that wrote it, its volume anchor and path within the volume, and its settings. Adding a scope first looks for a store whose descriptor binds to this library, this volume and this path, and registers under that identity, so a removed folder re-added or a drive plugged in with its catalog on it reopens the catalog instead of minting a fresh source over an empty one. A descriptor from another library or another drive is ignored and a new source starts beside it. The own-data-directory check becomes crate::config::is_managed: the data directory under every spelling, any .spacedrive directory and what is under it, and the volume identity marker. It is a path-component check rather than a registry lookup, so it holds for an unfiltered source whose rules are off, for an enclosing source that was never told about the inner store, for a store on a drive nobody here tracks, and across remounts and aliases. The walk, the watcher and the writer all use it. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Removing a source keeps its catalog unless asked to delete it sources.delete used to delete the store directory through the adapter engine before unregistering, which would have turned removal of an on-source store into destruction of the portable catalog, assertions included. It now retires the source's handles, drops the registration and leaves the store where it is; delete_catalog is an explicit second choice. The volume row is never touched: removing the last folder on a drive does not forget the hardware. sd source untrack wraps the action. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Add to Library acceptance suite and status One Core over temporary directories tracks a folder with defaults and one with overrides, proves one volume row for both, finds the on-source store excluded from its own and an enclosing source, removes a source and sees its catalog and record identities survive the re-add, and sees deletion start over. The suite joins the acceptance group of xtask test-core; docs/core/acceptance/add-to-library.md maps the plan's cases to tests and names the rows the core half cannot prove. PROJECT_STATUS and the plan's status line record what landed and what the modal thread builds next. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Review fixes: placement-aware reads and backup, re-track keeps settings, adoption across placements The library's per-source read queries (sources.list_items, list_records, media_listing) opened the store through the engine's in-library layout, so an on-source catalog was unreachable from the explorer; they now resolve a registered filesystem source through the volume index and fall through to the engine only for adapter sources. libraries.backup reads each store from where placement put it. A re-track carries the previous value of every setting the add did not name instead of the library's current defaults, and never changes placement. portable_identity searches both placements so a drive arriving with its catalog is adopted whatever placement the add asked for, and the adopted catalog keeps the placement it was found under. SourceConfig::default agrees with its serde defaults. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Restore names an on-source catalog it cannot put back The archive now holds a store backed up from its drive, but a restore lays every store in the in-library layout, where an on-source registration does not read. The manifest records each source's placement; a restore leaves an on-source catalog where it landed, keeps it out of the restored count, and names it in on_source_catalogs with its path, so the caller knows what was not put back. Moving it onto the drive is relocation. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * catalog_reused means an identity was kept, not that a file exists The acceptance run on CI saw a fresh add report its catalog reused: the content identity job of an enclosing source opened the new source's store between registration and the file probe. The flag now answers from the registration (it existed, or a descriptor-bound catalog was adopted), which is what it claims. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Regenerate TypeScript types for the catalog_reused doc Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Volume tracking test follows the one-library-per-path rule volumes.track now surfaces a source setup failure instead of logging it, so tracking a volume another open library holds is refused. The multi-library test asserts the refusal, then the handover once the first library untracks the drive and removes its source. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
5bef71f8b3 |
Design: extension file kinds and previews (#3123)
* Add extension file kinds and previews design plan Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Plan: fix the registry surface, load order, reidentification selector, contested claims and viewer binding Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Plan: keep the doc under 300 lines Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Plan: trim phase summary Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Plan: trim open question Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Plan: one reidentification statement per kind; renames are not carried over in this slice Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Plan: trim Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
b9385262f0 |
Load WASM extensions in the daemon and run their jobs with checkpoint and resume (#3122)
* Load WASM extensions in the daemon and run their jobs with checkpoint and resume The plugin manager now keeps each extension's store and instance for its lifetime, discovers every manifest under <data dir>/extensions at startup before libraries open, and runs a registered job export on a blocking thread. The job host functions report into the running WasmJob through a bridge: progress, warnings, metrics, an interrupt flag the pause path sets, and checkpoints that land in the checkpoint table before the guest continues, so a pause or a kill resumes from the guest's last state. extensions.list and extensions.run_job expose the registry on the wire. The SDK's #[job] export calls the job function (async ones through a poll loop) and plugin_init installs a panic hook so a guest panic reaches the host log. wasmer moves to 6.1 because 4.4 no longer links on Rust 1.89+ (wasmer #5610). The test extension calls check_interrupt_sync, the name the SDK has for the synchronous check, and drops uuid's v4 feature it never used so it builds for wasm32-unknown-unknown again. The integration suite loads the committed fixture, runs the counter to completion, and pauses, restarts and resumes it; the whole integration group builds with the wasm feature so the suite shares one sd-core build. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Build and load the photos extension and stop at the first missing host function The photos extension builds for wasm32-unknown-unknown without wasm-bindgen imports: uuid and chrono drop their JS features, entropy is the unsupported getrandom backend until the SDK has a host function for it, and the FaceDetection glob ambiguity is resolved. Its four jobs get registered names and are listed in the extension macro, so plugin_init registers them. The committed photos.wasm fixture loads beside the test extension in the integration suite; its first job fails at JobContext::run, which has no host function, with the panic in the host log. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Keep guest progress messages, reload a trapped plugin, reload by directory Progress from a guest carried a message the host read and dropped; it now reaches the job as generic progress with the job name as its phase. A trap poisons the plugin runtime and the next job reloads the extension from disk, since an aborted guest may have died holding its allocator. reload_plugin reads the directory the plugin came from instead of assuming it equals the manifest id. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Probe a trapped plugin without its mutex and swap in the reload only on success The poison flag lives in an atomic beside the runtime, so the next job reads it under the manager's read lock instead of waiting on the mutex a running job holds, and the write lock is taken only for the reload. A reload reads, compiles and instantiates the new instance before the old one is unloaded, so a failed reload leaves the plugin as it was. A job that reached a poisoned instance fails instead of running on it. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Put the old plugin back when a reload's plugin_init fails; read mutex poison too A reload whose fresh instance fails plugin_init re-registers the old plugin's jobs and reinserts it, so one bad reload is one error rather than a missing extension. The poison probe also reads a poisoned runtime mutex, which a panic unwinding out of a guest call leaves behind without reaching the flag. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Run the WASM extension suite in the acceptance group with its own build wasmer in sd-core adds minutes to every test binary link, so carrying the wasm feature across the integration group took it from 51 to 78 minutes. The suite now builds alone with the feature in the acceptance job, which had 16 minutes to spare. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
df027c29c1 |
Volume index keeps one source registry per library (#3120)
* Volume index keeps one source registry per library attach_library_with replaced the whole registry, so a daemon with two libraries listed sources only for the library loaded last, in read_dir order. The registry is now keyed by library id while drive detection stays shared; attach adds or replaces one library's registrations, close and delete detach only that library's, and path lookups resolve across every attached library. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Review fixes: restore resolves its targets before the close, every_store is per library, tracking refuses cross-library overlap A replace restore closes the library before quiescing its stores, and the close now takes the registry away, so the snapshot and partition targets are resolved first and handed to quiesce_stores. detach keeps the shared drive partitions (the OS watches over them survive a close); quiesce replaces them with fresh ones that keep their watch registrations. every_store takes the asking library so dedupe, last-copy and library search stay inside it, and sources.track refuses a root another open library already covers. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Restore rebuilds each source's map from its store; overlap guard skips unlocatable sources A replace restore drops the drive partitions and their snapshots, so a change below a source root had nowhere to land until something walked the source again. The restore now reads each attached source's records back into its arena from the restored store (the peer replica loader's conversion, shared) and announces the root, which re-arms the watch, without a walk re-hashing what the store already knows. The cross-library overlap check ignores sources whose drive is away (empty root), and quiesce_targets resolves partitions whether or not the index persists. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Restore rebuilds every source on the quiesced drives, whichever library holds it A partition is the drive's, so quiescing one library's sources takes the arena away from another open library's sources on the same drive. rebuild_quiesced reads every attached source on those partitions back from its store. The multi-library acceptance test backs up and replace- restores the first library and checks a change below the second's root still reaches its store. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Restore rebuilds the quiesced maps whether or not the reopen succeeds The other libraries on the drive were never part of the restore and must not lose their maps to its failure. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
cedc8d6f7e |
Locked volumes L1 and L2: vanished volumes go offline, attached means mounted (#3119)
* Locked volumes L1: a volume that disappears goes offline The volume monitor marked a tracked volume offline only while detection still returned it, so a drive that vanished while the daemon was down, or a dataset whose key was not loaded, kept is_online = 1 and attach_library resolved its sources against the stale flag. The monitor now reconciles every tracked row against detection and marks a vanished volume offline, telling the volume index so sources on it detach or reattach without a restart. attach_library resolves against live detection when monitoring is on, corrects a disagreeing row in place, and falls back to the stored flag when it is off. Each mapped drive carries its mounted state, and a source on a drive that is away comes up detached with its map restored read-only, no watch armed and nothing dispatched at the directory left behind. sources.list reads attachment from the index instead of the root existing. KeyManager::close used a file named :memory: as its placeholder, which one process can only hold once; a second core closing in the same process left its secrets file locked for the next open. It now uses redb's in-memory backend. Acceptance runs on a loop-mounted ext4 image unmounted with its mount point left in place; the suite joins the acceptance group of xtask test-core and skips with a reason without sudo or a loop device. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Locked volumes L2: attached means mounted Attachment was Path::exists on the source root, and an unmounted drive leaves its mount point behind as an empty directory that passes it. The volume index now refuses to dispatch anything at a path whose drive is detached or whose mount point is not a mount point (on Unix, its device is the same as its parent directory's), whatever the stored state says: the source walk, the whole-drive map, the coverage heal, the content identification pass and the thumbnail job all check before running, and the indexer and identification jobs check again at run so a job resumed from a previous session cannot slip past. sources.track refuses the mount point of a known volume that is not mounted instead of registering a second source over the parent filesystem. The mount point check applies to drives learned from a volume row or from detection; a directory a test tracks as a drive keeps working. The acceptance scenario unmounts a loop volume under a running daemon with detection still saying mounted, drops the snapshot so a forced heal wants a walk, and proves no walk or hash job runs, the store keeps every record, tracking the directory refuses, and the listing reports the source unmounted with its counts intact. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Release the library lock on every shutdown path Library::shutdown returned early when the job manager could not pause a running job in time, leaving the lock file behind; the next process in the same hour then refused the library as in use. The acceptance run on CI hit this when the hashing pass was still running at the fixture's shutdown. The lock is now released after the inner shutdown whatever it returned, and the fixture waits for its queued jobs before stopping. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Review fixes: scope volume rows to this device, keep away drives resolvable, check detected drives' mount points attach_library_with and the monitor's reconcile now read and write only this device's volume rows; the table holds every device's, and a peer's row resolved against local detection was both corrected in place and entered into the volume index at the peer's mount point. The reconcile pass logs a row it cannot write and goes on to the next instead of aborting the library's pass. locate keeps a drive that is away in its fallback, so a mapped drive with no source over it still serves its map read-only from its own partition rather than scratch, with the mounted drive winning only at the same or a longer mount point. Drives detection returned are mapped through track_detected_volume, which carries their mount state and makes the mount point check apply to them; track_volume stays for cloud prefixes and the directories tests stand in for drives. The acceptance fixture waits for the hashing pass by its effect rather than by the running job list. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Cloud volume roots are never refused as unmounted A cloud volume mounts at a service prefix such as s3://bucket, which detection never lists and Path::exists never answers for, so the mount guards read it as away and volumes.index refused every cloud volume. A cloud root is reachable as long as its row says so: attach maps it mounted without correcting its row, root_attached and dispatch_refusal pass it, and it never carries the mount point check. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Stop the library watcher before closing libraries at shutdown The library directory watcher auto-opens a library whose lock file changes, and close_library removes the lock file, so a core shutting down reopened its own library from the watcher task and a core starting in the same process then found it in use. The acceptance job on CI hit this on every restart. The watcher now stops before the libraries close. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Acceptance tests: say what the library and index hold when a source is missing Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Acceptance diagnostics carry a failed query instead of reading as empty Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Acceptance diagnostics list the open libraries Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Locked volumes acceptance fixture keeps the daemon's one library The core's volume index serves the library opened last and library load order follows the data directory listing, so a fixture with two libraries came up with an empty index on some runners. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
1e1fc8340e |
FDA follow-up: Linux trash restore, rename over an existing file, KeyManager ':memory:' file (#3117)
* Linux trash: record the item under Trash/files and take its .trashinfo with it on restore and purge trash_os recorded the trash crate's item id as the location, which on the freedesktop backend is the .trashinfo path. restore saw that path exist, renamed it over the original, and the restored file held the trashinfo text. The location is now files/<name>, derived from the info path the way the crate derives it, and restore and purge remove the info/<name>.trashinfo companion so the trash lists no ghost. The two FDA trash tests run on Linux; they stay ignored on Windows, where the id is a shell parsing name. macOS is untouched. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Store: a rename over an existing file removes the overwritten row in the same batch When the watcher reports a rename whose destination name is already bound in the parent, the overwritten record is forgotten from the ledger and removed in the batch that lands the move, and apply_files now lands removals before writes so the mover takes the title without colliding on UNIQUE(parent_uuid, title). The overwritten record's content row stays: the bytes may sit behind another record and the hash is evidence either way. a_rename_over_an_existing_file_lands_as_one_row is no longer ignored and the known limit leaves PROJECT_STATUS. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * KeyManager::close swaps in redb's in-memory backend instead of a file named ':memory:' redb has no :memory: path convention, so close created a real file by that name in the process working directory, and every shut-down core in one process contended on its lock. The stand-in is now an InMemoryBackend database; closing_leaves_the_working_directory_alone proves close touches neither the working directory nor the data directory. The pin restart acceptance row runs in process again: its child-process detour existed only for this. non_utf8_child_walk keeps its child, since the tracing subscriber is process-global and the test reads what the walk logged. The FDA matrix rows, the plan status lines and the register row follow. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Trash: restore and purge accept a legacy .trashinfo location, and a stuck companion never fails a landed restore A journal row written before the location was the item names the .trashinfo path on Linux; item_behind maps it to files/<name> so restoring an old row brings the bytes back instead of the trashinfo text. Removing the companion after a rename or a removal is bookkeeping: a failure is logged, since the item is already back or already gone. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Store: a removal wins over a write staged in the same batch Landing removals first let a write staged earlier in the batch resurrect the row, so touch-then-rm left a ghost record and rm -rf of a just-written directory aborted the batch on the child's parent reference. apply_files skips any write whose uuid the batch removes; the rename arm no longer needs its own retain. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Undo: a journaled directory still holds after a child lands in it A job creates a folder, journals it, then fills it, and every child moves the folder's mtime, so undoing the merge was refused as 'changed since the job ran' whenever the child landed in a later millisecond. For a directory subject only the kind is checked. Surfaced by undoing_a_mirror_restores_the_extras_and_the_replaced_bytes once it ran on Linux. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Undo: move_back goes through trash::restore, and a directory subject is matched on its creation time move_back renamed the journaled location first, so an old row holding the .trashinfo path still put the trashinfo text over the original and a current row left its companion behind. It is now trash::restore, which maps a legacy location to its item and takes the companion with it. Checking only a directory's kind let an unrelated folder that took the job's folder's place pass as unchanged and be trashed by undo. Subject records a creation time where the filesystem keeps one, and a directory matches on kind and creation time, which survive its children being written; a row with no creation time matches on kind. ts-client types regenerated for the new field. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Store: removals land before subtree renames, the salvage path honours them, and a directory rebinds before its subtree is re-keyed A rename over an existing directory collided on directory_path.path because the subtree UPDATE ran before the removals; removals now land first in apply_files, the overwritten destination is forgotten as a tree so stale descendants go with it, and the row-by-row salvage skips writes the batch removes. Rebinding the mover before rename_tree fixes a pre-existing miss where a renamed directory resolved as unchanged under its old title. The acceptance test gains the directory form. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Undo: a journal row from before created_ms matches a file on kind, size and mtime Whole-struct equality refused every standing undo after the upgrade, since old rows decode with no creation time and the live metadata has one. A file compares kind, size and mtime; a directory kind and creation time where both sides have it. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
92dea58a46 |
ci: enforce the docs build, generated types, secrets scan, and license gates (#3118)
* docs: escape the <name> placeholder in library-sync-setup so MDX compiles Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * tauri: regenerate capability schemas for the updater plugin and pinned tauri-build Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * ci: enforce the docs build, generated types, secrets scan, and license gates on every PR Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * docs: name the ci.yml job behind each repository release gate Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * tauri: keep desktop-schema.json in its committed key order Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * ci: compare Tauri schemas as JSON since key order follows feature unification Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
ee76270f54 |
FDA: entries drop and file operations acceptance matrix as CI tests (#3116)
* R8: source runtime acceptance matrix as CI tests Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Drop the regenerated Tauri linux schema; unrelated build churn Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * FDA: entries drop and file operations acceptance matrix as CI tests Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Run the R8 suites as their own core_tests job The integration job took 63 min cold on Blacksmith with the three suites added; its original suites alone run close to the hour. An acceptance group in xtask keeps each job under the cap and lets the two run in parallel. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * PROJECT_STATUS: baseline follows the merged tree Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * FDA: follow records_for_tag's P1a signature Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * FDA matrix: correct totals, flush before revision reads, prove the boundary walk ran, pin the pre-drop fixture to the drop migration, build copy jobs through the action Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * FDA matrix: assert the boundary walk visited the tree; V6 states the 60 s ceiling Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * FDA matrix: the timing comment matches the measured figure Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
08eb622e29 |
Library backup and restore with manifest verification (#3115)
* Library backup and restore with manifest verification libraries.backup copies library.db, sync.db, every owned source store and its sidecars with VACUUM INTO while the daemon runs, hashes each file with blake3 and writes manifest.json to a directory or .tar.zst archive. libraries.backup.verify rehashes a backup without restoring it. libraries.restore verifies every hash, refuses a backup from a newer schema, stages under the data dir, swaps files into place with the open store and sidecar pools released first, parks the displaced state in restore-trash/, and reopens the library. Replacing a library with other member devices needs force, since their device rows and sync watermarks would diverge. CLI: sd library backup --to, sd library backup verify --from, sd library restore --from [--as-new | --replace] [--force]. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * CLI help: restore-trash is kept, not pruned Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Restore: keep stores closed across the swap, clear the arena, serialize library.json Review findings on #3115: the volume index kept the pre-restore arena and drive snapshots after a replace, a watcher event could reopen the old data.db between the release and the rename, and library.json was copied byte for byte while the daemon rewrites it in place. quiesce_stores takes every affected source's open gate and holds it until the renames are done, closes the cached handles, detaches the library's registrations and partitions, and removes the drive snapshots so the reopened library rebuilds from the restored stores. Backup serializes the open library's config instead of copying the file, and restore parses the staged config before touching the data directory. A failure after the first displacement now names the restore-trash directory. The test feeds source b's watcher during the replace and proves a later write reaches the restored file on disk. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Restore: load the staged config as LibraryConfig, hold sidecars across the swap Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
e58507a1ec |
R8 fixes F1, F2, F3, F5: salvage repair, snapshot quarantine, refused watch state, nested-store search duplicates (#3114)
* Store: unsettle a salvaged-away row so the next observation rewrites it R8 row 3 (F1). When a batch loses a row to a database failure the ledger kept the binding it made before the commit, so the next walk resolved the same file Unchanged and the row stayed missing for as long as its size and mtime held. commit now takes the ledger and unsettles each row it could not land: the uuid keeps its path, the size and mtime evidence is cleared, and the next observation resolves Changed and writes the whole row under the identity the arena already holds. Un-ignores a_repeat_observation_repairs_a_failed_write. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Snapshot: move an unreadable artifact aside instead of deleting it R8 row 10 (F2). load_snapshot_impl removed a snapshot it could not read, decode, or accept the version of, so the artifact was gone before anyone could inspect it. It now moves to <name>.corrupt-<unix seconds> beside the slot with a WARN naming both paths; the slot is clear for the next save and no launch parses it again. Un-ignores an_invalid_snapshot_is_retained_for_diagnosis, which now asserts the bytes survive at the aside path. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Watcher: register a root only after the OS accepts, report and retry refusals R8 row 11 (F3). watch_root registered the root with the volume index before subscribing at the OS, so a refused subscription left the root in watched_paths and is_watched and core.index_status reported a watch that did not exist. The OS subscription now comes first; a refusal is recorded on the volume index with its reason, surfaces as IndexStatus::refused_watches (and in sd index status), and is retried every watch_retry_interval (30 s) while the service runs. A retry the OS accepts moves the root to watched_paths. Un-ignores a_refused_watch_is_not_reported_active and extends it with the status report and the reconciling retry. Regenerates ts-client types. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Search: answer a file under nested sources once from the stores R8 row 25 (F5). With an outer and an inner source both walked, the file under the inner root sits in both stores, and after a restart with no snapshot search_every_index queried each store in turn and returned it twice. The store loop now reads sources innermost first and drops a hit whose local path an earlier partition (an inner store or a drive's arena) already answered, before it is counted, so total_found stays the number of distinct files. Un-ignores a_file_under_nested_sources_is_one_hit_from_the_stores. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Docs: R8 rows 3, 10, 11 and 25 fixed; row 17 remains the R4 brief Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Review: one retained corrupt snapshot, full-arena dedupe seed, reset clears refusals A recurring load failure no longer retains an artifact per launch: when a .corrupt-* sibling exists the slot is removed instead, keeping the oldest copy as the mismatched-root path does. The store de-duplication seed is collected from each arena's full match set before the page window narrows it, so total_found stays exact beyond the page. clear_all forgets refused watches along with the watched paths. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
179adcf6d5 |
R8: source runtime acceptance matrix as CI tests (#3113)
* R8: source runtime acceptance matrix as CI tests Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Drop the regenerated Tauri linux schema; unrelated build churn Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Run the R8 suites as their own core_tests job The integration job took 63 min cold on Blacksmith with the three suites added; its original suites alone run close to the hour. An acceptance group in xtask keeps each job under the cap and lets the two run in parallel. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * PROJECT_STATUS: baseline follows the merged tree Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
c81e3d6169 |
Addressing P0 and P1a: doc matches code, content rows record their own bytes (#3112)
* docs: describe addressing as the code does it, mark the target model planned (P0) Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * store: schema version in PRAGMA user_version, migrated at writer open (P1a) Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * store: content rows record each file's own bytes (P1a migration 1) Rebuild content so a row's integrity hash belongs to every record on it: candidate rows keyed by sampled hash, confirmed rows by integrity hash, candidate_uuid beside uuid. bind_content never binds a sampled-only write to a confirmed row, moves need size and mtime, verification claims by sampled hash, duplicate discovery groups by sampled hash, and content lookups match uuid or candidate_uuid. Compare-delete matches a file against any holder in B that was read in full. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * store: tests for schema version, migration 1 and content row ownership Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * status: addressing register row, P0 and P1a landed Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * store: duplicates group by sampled hash and split by integrity; version-0 tag lookup; content section in the doc Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * store: bound the duplicates query to the largest group_limit sampled hashes Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * store: batch migration 1 inserts; timing fixture; upgrade note in the register Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
5bc474e722 |
CLI and daemon housekeeping from the live Mac runs (#3111)
* Default the replication block in core.status so a newer CLI reads an older daemon A |
||
|
jamiepineandcapy-ai-staging[bot]
|
accc7235d9 |
README and docs: nightly downloads, beta November 1, main as branch of record
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
10dbc18d0f |
Replica fetch: resumable .part transfers, bandwidth cap, pause, and progress (#3109)
* Resume interrupted replica fetches from a .part file A replica transfer now lands in <source>.<generation>.<blake3 prefix>.part and stays there when the stream breaks or the daemon stops. The next attempt asks the owner for the artifact's tail from the part's length over the new FetchSnapshotFrom and FetchDatabaseFrom requests, whose header still names the whole artifact; a header that names different bytes discards the part and starts over, and the finished file is validated against the header checksum before it replaces the live replica as before. Parts are removed only when the owner's listing no longer carries their generation. A nonzero offset serves the on-disk snapshot without the pre-fetch save, so a resumed copy finishes against the same bytes. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Cap replica transfer bandwidth with a persisted daemon setting replication.max_bytes_per_sec in spacedrive.json bounds every replica artifact served and received through one token bucket (zero is unlimited); the cap applies to transfers already in flight. It is set through config.app.update and sd config set replication.max_bytes_per_sec (bytes, or K/M/G suffixes), read back through sd config get and show. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Pause and resume replica fetching sd sources replication pause stops this device starting replica transfers and halts any in flight at its next chunk, keeping the partial file; resume continues from it and runs a replication pass against every connected device at once. The switch is replication.paused in the daemon config, so a paused daemon stays paused across a restart, and a pass that runs while paused still refreshes listings, owner facts and fleet totals. A pause does not count as a fetch failure, so no backoff accrues. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Show replica fetch progress in sources list, status and events Every fetch in flight is tracked with bytes on disk, total and a recent rate. sources.list carries it on the replica's row (or a replica_fetching row for a first copy that has no share yet), core.status and the new mounts.replication_status query summarize it with the pause switch and cap, and the daemon emits a typed ReplicationProgress event about once a second per transfer plus a finished one when it stops. sd sources list and sd status print the figures; ts-client types regenerated. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Document resumable replica fetch, the cap, pause and progress Source replication gets its own section in networking.mdx with the .part contract, the wire change, the cap, the pause switch and the progress fields; file-sync.mdx points there; the replication plan marks its cleanup item superseded on the replica side and distinct from the restart refetch; PROJECT_STATUS.md gains the register row. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Keep the part when a resumed request gets no header; cap sidecar pages A resumed request that fails outright (dropped link, or an owner too old to know the offset variants) says nothing about the bytes on disk, so the part now stays and the whole-file retry lands on the same name. Sidecar pages count against the bandwidth cap on both ends, as the setting's contract says. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Add the replication field to the test harness config Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Check the pause switch after the cap's wait, not before Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
f9bdf724b6 |
Library join: fix inverted backfill, catch-up loop, reverse device registration, member totals (#3110)
* Networking: serve streams on connections a protocol dials
The connection cache is shared between the dial side and the accept side,
so once a protocol dials a peer, the peer's own send path finds that
connection and opens its streams on it. Nothing read those streams: only
connections the event loop accepted, or those the reconnect loop handed
over, had a stream handler. The peer's request then sat unanswered until
its 30 s timeout while requests in the other direction answered in
milliseconds. This is the reverse RegisterDeviceRequest timeout SPAC-12
saw after join-remote.
get_or_create_connection now hands every connection it dials to the
event loop as TrackOutboundConnection, which spawns the same stream
handler an inbound connection gets. Messaging, sync, and job activity
pass their event loop handle; pairing keeps its own connection handling.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: the device that joins a library is the one that backfills
Every sync service started Uninitialized and backfilled from the first
connected partner, whatever state it already held. When the Mac Studio
joined the MacBook's library the MacBook was the one with a partner in
its device table, so it went Uninitialized -> Backfilling{Studio} and
applied the empty copy's snapshot over its own rows (LWW on the
deterministic default-space UUID). The Studio never backfilled because
the reverse registration had timed out and its device table held only
itself.
A copy of a library created from another device's library is the only
thing that may start Uninitialized. create_library_with_id now writes an
initial_backfill_pending marker into sync.db; PeerSync reads it on open
and starts Ready otherwise. A Ready device never requests a snapshot
(SharedChangeRequest.include_snapshot = false on catch-up) and repeated
catch-up failures no longer escalate to a full backfill. A copy still
waiting for its own first backfill refuses to serve as a join target.
The join carries the rule explicitly: RegisterDeviceRequest has
needs_initial_state, and RegisterDeviceResponse returns the responder's
own device record so the joiner's table is complete when the command
returns, replacing the spawned reverse request. sync_setup fails
instead of reporting devices_registered: true when that record is
missing.
Catch-up completion is recorded per peer in sync.db even when no rows
arrived; the Ready loop catches up once with a peer it has no record
for and no longer re-polls every peer whose watermark is 60 s old. A
missing watermark means never synced, not ancient, so the forced full
sync warning only fires for a real watermark past retention.
library_join_test pairs two daemons for real, joins Bob to Alice's
library, and asserts Alice's rows are unchanged, Bob holds Alice's
device row and spaces, and Alice runs no further catch-ups.
Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com>
* Sync: test that catch-up with an empty peer runs once
Alice holds a volume, Bob holds nothing of his own, so Alice never
writes a per-resource watermark for Bob. Before the per-peer catch-up
record, that made Bob a new peer on every loop iteration and Alice
caught up with him forever (2 -> 5 sessions in 16 s on
|
||
|
Jamie Pineandcapy-ai-staging[bot]
|
5e285d3c97 |
Nightly CLI channel: rolling prerelease workflow, sha-aware self-update, update timer (#3107)
* Add nightly CLI channel: rolling prerelease workflow, sha-aware self-update, update timer Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * TEMP: trigger nightly on PR branch Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Raise the launchd daemon file-descriptor limit to 8192 Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Remove temporary nightly trigger for the PR branch Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly: save the rust cache, note why Linux builds on 22.04 Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Address review: semver-ordered stable updates, assert nightly target commit, quote systemd ExecStart Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly: skip the xtask daemon build, retry release creation Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * TEMP: trigger nightly on PR branch Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Nightly: remove the runner's libunwind before setup.sh Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> * Remove temporary nightly trigger for the PR branch Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pineandcapy-ai-staging[bot]
|
09d7ea2ca9 |
Reconnect paired peers after an in-process restart (#3106)
Close the iroh endpoint on networking shutdown, bound every reconnection dial, give the deferring side of the EndpointId rule a grace period after which it dials anyway, and re-arm the periodic sweep for any device still unconnected. Co-authored-by: capy-ai-staging[bot] <305915985+capy-ai-staging[bot]@users.noreply.github.com> |
||
|
Jamie Pine
|
9929d22c72 | Land MacBook work: sources.track fix, source hashing test, addressing and locked-volumes plans | ||
|
Jamie PineandClaude Opus 5.5
|
49eb621d8b |
Point doc commit references at the rewritten history
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Jamie PineandClaude Opus 5.5
|
fec139fbdf |
Remove Spacebot and the voice overlay
Spacebot ships separately. This drops its UI, the voice overlay and the Alt+Space shortcut that opened it, the spacebot app config and window, the @spacebot/api-client and @spacedrive/ai wiring, and the Spacebot and Console design docs. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
Jamie Pine
|
18021a11ab |
Record opening remote files and the titan deploy
The plan's status and what landed, with the SMB defects and the loopback decision: pairing covers the hop between devices, and the share guards against browsers with a Host check rather than a key, since RPC on 6969 is open to local processes anyway. The mounts design's 2026-09-24 status, the macOS volume detection note on the daemon's own mount, and the register row with titan on this build since 2026-09-24 19:52 PDT and its databases backed up first. |
||
|
Jamie Pine
|
bcc7337b0d |
Journal every file job, trash with recorded locations, undo, and the rest of the file operations
Every job that changes the filesystem records effects (created, moved, trashed with where it went, replaced with where the previous bytes went, removed, attributes before and after) in a job_journal table through ctx.record. The trash reports locations: NSFileManager on macOS, the trash crate's listing elsewhere, and a .spacedrive/trash/<job>/ directory on volumes with none. Overwrites in copy, merge and extract stash the previous file there. files.undo previews the reverse as an FsPlan, refuses what changed since, and its job writes its own journal. Also: files.rename on preflight with per-filesystem name rules and case probing, files.rename_batch with ordered rules and a job that never overwrites; remove_extras on files.merge for a mirror; files.organize and files.flatten as in-place renames; files.archive and files.extract for zip and tar.zst, planned from the archive's directory; files.set_attributes and files.link; files.trash_list and files.trash_empty; jobs.journal and a journal summary on jobs.list. The copy job now honors the skip policy and merges a folder onto an existing folder instead of overwriting inside it, and the copy preview plans a keep-both copy at its numbered name. 30 tests over the fixture. Docs, the plan's status table and PROJECT_STATUS updated. |
||
|
Jamie Pine
|
23abd01164 |
Plan the file operations still missing or off preflight
Delete and duplicate in the explorer, rename and batch rename on preflight, an operation journal with undo and trash restore, mirror as a merge option, organize and flatten, archive and extract, attributes, links and a trash view. Seven decisions are open. |
||
|
Jamie Pine
|
e1e73f2b38 |
Dedupe as a delete target: first copy or chosen copies, read in full before removal
files.delete takes a Duplicates target: a scope with keep first (walk order) or a list of copies to keep with every other copy of their content going. The job reads each surplus copy and its keeper in full before removing, records the hashes it learns, and checkpoints the compare cursor so it resumes. Preflight validates the scope and keep list and previews keepers as skips and surplus copies as deletes. Store gains duplicated_contents_beneath and copies_beneath. files.duplicates is registered as a core query and reports each copy's SdPath. CLI: sd file dedupe with --keep, --keep-under, --min-size, --dry-run. |
||
|
Jamie Pine
|
3cd44f2daa |
Add preflight to actions and build folder merge on it
Two opt-in methods per action, validate: and preview:, over the exact input the action takes. Validation answers findings with stable codes and the facts of the execution, runs again at dispatch and refuses on an error. Preview answers an FsPlan read from the index, kept under a handle so files.directory_listing can serve the directory as it would look after the operation. Copy, move, merge and delete answer both. Merge is new: a resumable job that recurses into matching folders, reads both sides before skipping a duplicate, applies a per-leaf conflict policy, leaves conflicts alone and can consume its sources. Delete's validation counts, across every store, which files are the last copy of their content. The ValidationResult and ConfirmationRequest hook is gone, and so is the CLI's local conflict detection. file copy, merge and delete render preflight before the prompt and take --dry-run. |
||
|
Jamie Pine
|
903c8f2110 |
Delete from a folder what a comparison names
files.delete takes a comparison as its targets alongside a path list, so the client sends five fields and the daemon derives the set from the index as the job runs. The job drains the matcher the compare query pages, checkpoints its cursor after each batch, and for `both` reads whichever side lacks an integrity hash before removing a file, writing what it learned back to the store. Pairs whose bytes differ, or whose copy in B is gone by then, stay and are reported. The CLI gains `sd file delete A --against B --show both` behind a y/N prompt, plus the plain paths form. Compare matched on content uuids, which re-derive from the integrity hash once a copy is verified, so a copy verified on one side stopped matching its unread twin. It now matches on the sampled hash every store keys a content by, with integrity hashes deciding where both sides have them; FsEntry carries both hashes for that. |
||
|
Jamie Pine
|
16325fc5c1 |
Name compared folders A and B, and their sets after a Venn diagram
Left and right only meant first and second argument, and nothing on screen said which folder was which. "Same" meant different things per mode, "changed" implied a direction compare can't know, and "location" collided with the retired locations. The output now names both folders as A and B up front, says how files matched, then counts Only in A, Only in B, In both, and Different (by path only). --by location becomes --by path and the --show values follow, as do the API types, generated TS types, skill, and docs. Single-side listings get Size and Modified columns under a title naming the set, and --help explains each value. |
||
|
Jamie Pine
|
05acd99e6f |
Plan action validation, preview, and folder merge
Two opt-in rails on actions, both taking the action's exact input: validate answers whether and how an operation runs and is re-run at dispatch, preview answers what will exist afterward as an FsPlan from the index, browsable through plan handles as an overlay on directory listings. Folder merge is the first tenant. The plan catalogs both rails across copy, move, merge, delete, batch rename, and dedupe, and registers the teardown of the unreachable ValidationResult rail and the CLI's local conflict logic. |
||
|
Jamie Pine
|
e5e6749c26 |
Compare two folders over the source stores
paths.compare answers what differs between two folders from the source stores at query time. By location it merges both sides in index order into only-left, only-right, changed and same. By content it streams one side and asks the other side's stores which content ids they hold, so a renamed or moved copy still matches. Pages resume from a cursor and the first page carries every set's count. Unhashed pairs fall back to size and mtime and get counted, so a page says how much rests on that. sd-cli file compare wraps it. On the store side, files_beneath takes a Start, including one that resumes a nested store at its first directory, and contents_beneath answers the content lookup a batch at a time. Working out which stores a path reaches moved into ops/paths/reach.rs, shared with search.media and paths.context. No catalog.db: the P7 row, the filesystem-intelligence plan, the previs and the design docs now treat cross-source questions as store queries. |
||
|
Jamie Pine
|
b16816b73b |
Page Photos from the source stores and follow searches
Photos listed only a folder's direct children and never saw the explorer's search, so searching from a folder of folders showed no photos. search.media pages images and videos straight from source SQLite. Files come out in the order the directory_path and sibling indexes already keep, so a page reads its own rows and resumes from a (source, directory, name) cursor. Extension, kind and hidden narrow in SQL; name, size, date and tags are judged per row the way the store backend of search.files judges them. A folder reaches its innermost source and any sources nested beneath it. The navigation focus now carries the running search, and Photos lists it in place of the folder. Folders list everything beneath them. Pages grow the grid as they land, and tile identities name their drive's cache, so a library-wide search across drives renders. |
||
|
Jamie Pine
|
c7d24c7ee8 |
Say which clients start discovery
Only the desktop app requests the launch pass. From a terminal it takes index start --defaults, and the web UI and Photos never ask. |
||
|
Jamie Pine
|
8f12892c12 |
Let a daemon start without the default places
The first client to connect starts automatic discovery: it tracks the system volume, adds the home folder as a source and walks every attached drive. For a library meant to hold only what you add, like one whose data directory lives on the drive it catalogs, that indexes the Mac into it. sd start --no-default-sources, and the same flag on restart and on sd-daemon, now leaves those three out. The pass still restores snapshots, heals coverage for sources already tracked and hashes them, so a source you added keeps working across restarts. sd-server keeps the defaults. |