Tests whose subject was the entry tree, locations, legacy tag tables, or
row sync of retired models are deleted. The rest track a source or browse
a directory the way production does. The indexing harness gains track(),
which waits on the walk tracking dispatched, and index_dir(), which walks
under the volume's spelling of a path, as browsing does; the hand-built
walks it replaces indexed /Users while search scoped to
/System/Volumes/Data/Users and found nothing. The harness also moves the
core's data directory beside the test files: the daemon never walks or
watches its own data, so every watcher test had been observing nothing.
The sync tests drive traffic with spaces now that indexing writes nothing
the library syncs. job_shutdown reads the pause off the bus, since the
job database closes with the core, and waits on the walk by id rather
than a sleep. The TypeScript bridge passes the daemon's device slug
instead of guessing it from the hostname, and the suites wrap
createSubscription with its current signature. Location-scoped search
cases become library-wide searches over a tracked source, and the typed
search bridge test is back in the core suite.
A machine needs full local search over every file and a size figure the
analyser can trust. Neither is the same question as what someone chose to keep,
and the sources migration collapsed the two.
A partition is keyed by source id today and routed by longest prefix, so a
source nested inside another forks rather than narrows: two partitions, two
snapshots, two ledgers, and two uuids for one file. A tag applied through one
root is invisible through the other, which is the second-namespace problem
between two filesystem sources.
P2.7 keys the arena by medium instead. One drive, one arena, one snapshot,
whatever is persisted from it. A source says persist this subtree and owns a
data.db, so nesting is ordinary because the nested things are no longer the
same thing. The arena mints identity once per medium and a source adopts what
is already there, which is both what stops the fork and what makes creating a
source over mapped files fast.
A path under no source still has a uuid and still cannot carry an assertion,
since assertions live in a source store. Choosing to keep something is what
gives it somewhere to keep things.
Also records the completeness cost. An internal volume walks with default
rules, so /Library, ~/Library, node_modules and gitignored files never enter
the map: 262 GB indexed against roughly 900 GB used on a 1 TB drive. Rules
become view-time lenses and the walk records what it can read, which takes a
2.1M entry map to somewhere between 5 and 8M, and 1.0 GB resident to between
2.5 and 4. collection_flags already has room for a filtered bit.
Settles the nesting question in the Open register. Both live documents were
right about their own half: one filesystem owns one record namespace, and that
namespace is the arena rather than the source.
grouping and lib_edge had no writer, no reader and no op — Engine and
SourceManager exposed link/unlink/neighbors/upsert_grouping as pass-throughs
that nothing called. They are also the wrong shape for where Photos
enrichment is going: a person appearing in photos on three drives is a
definition replicated into three source stores, not a row in a library-level
table that outlives all three.
record_overlay moves into source.db as SourceDb::{set_overlay, get_overlay,
overlays_for}, keyed (type, external_id) with source_id dropped — the source
is now implied by which file you opened. The search router reads overlays off
the same handle it just searched instead of joining a second database.
registry.db is down to sources and data_types. No durable layer, so there is
no shared file left to argue about.
Semantic change worth knowing: deleting a source now discards its assertions,
since they live in the file being deleted. Re-index is the operation that
keeps them — replace what an ingest produced, leave record_overlay alone —
and it does not exist yet. Nothing reaches delete_source with assertions
today, so nothing regresses, but re-index has to land before either is
reachable from an op.
Absolute paths from one developer's machine sat in an example, a shell script,
recorded benchmark metadata, task docs and design docs. The example and the
script were broken for anyone else as a result: the example now resolves against
CARGO_MANIFEST_DIR and the script derives the repo root from git. The duplicate
copy of that script under core/scripts is gone.
Also drops a pointer to a repository that is not public.
Sources resolve at startup without onboarding. Adds the Apple Photos native
adapter, collections and the analyzer, the bundle lens, and assertions with
read-time binding.
* fix: update tests and examples for upstream API changes
Align error types, struct fields, and constructor calls in 27 test/example
files to match current Core API:
- Box<dyn Error> -> Box<dyn Error + Send + Sync> to match Core::new()/shutdown()
- entry::ActiveModel device_id -> volume_id (field rename)
- location::ActiveModel add volume_id field (new nullable field)
- AppConfig add proxy_pairing and spacebot fields (config v5/v6)
- Re-export ProxyPairingConfig from config module
- VolumeFingerprint::new() -> from_primary/external/network_volume()
- DeviceInfo add device_type field, NetworkFingerprint public_key -> public_key_hash
- SecretKey::generate(thread_rng) -> from_bytes (rand_core version mismatch)
- SdPath::Physical.path String -> PathBuf, thumbnail_url &Uuid -> &str
- init_sync_service unwrap networking Option for Arc<dyn NetworkTransport>
* fix: prevent busy-loop in sync loop when real-time sync is active
The sync loop's Ready state used 'continue' when is_realtime_active()
returned true, which skipped the sleep at the bottom of the loop and
caused a CPU-saturating busy-loop logging 'Skipping catch-up' hundreds
of thousands of times per second. Replace with if/else so the
realtime-active branch falls through to the configurable sleep.
- Removed the last_sync_at field from the device model and related structures, as it is now superseded by per-peer watermark tracking in sync.db.
- Updated various components, including device registration, sync setup, and messaging protocols, to eliminate references to last_sync_at.
- Introduced a migration to drop the obsolete last_sync_at column from the devices table.
- Adjusted tests and helper functions to reflect the removal of last_sync_at, ensuring they align with the new sync architecture.
This commit adds a new example for debugging volume detection, showcasing how to detect and display various volume properties, including auto-track eligibility and primary volume identification. Additionally, it refactors the VolumeFingerprint struct to provide stable fingerprinting methods for primary, external, and network volumes, improving the reliability of volume identification across different scenarios. These changes enhance the overall volume management capabilities and provide a clearer debugging interface for developers.
- Modified test data paths in various integration tests to use the Spacedrive source code instead of user directories, ensuring consistent and deterministic test results across environments.
- Updated comments and documentation to reflect the new testing approach and clarify the purpose of using project source code for testing.
- Enhanced the GitHub Actions workflow to skip Rust toolchain setup on macOS self-hosted runners, assuming Rust is pre-installed.
- Added critical tasks for remote file access and updater functionality to the TODO list.
- Updated job logging configuration to include an option for logging ephemeral jobs, improving logging flexibility.
- Enhanced the job manager to conditionally log job events based on persistence settings, ensuring better tracking of job statuses.
- Refactored the inspector component to remove unnecessary console logs, streamlining the codebase.
- Improved the selection context to eliminate redundant logging during file synchronization, enhancing performance.
- Updated the CI workflow to run tests specifically for the package for clarity.
- Deleted outdated example test files for event serialization, location watcher, and simple metrics to reduce clutter.
- Enhanced error handling in for better robustness.
- Improved device data handling in by adding additional fields for comprehensive device information.
- Adjusted test assertions in to align with updated change detection logic.
This commit introduces an auto-switch feature for libraries created via sync. Users can now configure whether to automatically switch to a newly synced library. This involves updates to the preferences store, event handling in mobile and tauri apps, and modifications to the core library event system to include a `LibraryCreationSource` enum.
Co-authored-by: ijamespine <ijamespine@me.com>
- Introduced new modules and actions for generating media proxies and thumbstrips, improving media handling capabilities.
- Added configuration, error handling, and processing logic for both proxy and thumbstrip generation.
- Updated existing media operations to integrate new functionalities, ensuring seamless media processing and user experience.
- Enhanced tests to cover new features and ensure reliability in media operations.
- Added `sync_enabled` and `last_sync_at` fields to the device model to support synchronization capabilities.
- Created a migration script to update the devices table, consolidating sync-related data.
- Updated various components to ensure sync is enabled by default for new devices.
- Removed the obsolete sync partners table, streamlining the sync architecture.
- Introduced comprehensive sync integration tests to validate the new functionality and ensure robust performance.
- Implemented a complete WASM extension framework, enabling secure, sandboxed plugins.
- Added core components including `PluginManager`, `host_functions`, and `permissions` for managing the lifecycle and security of extensions.
- Integrated Wasmer runtime for executing WASM modules, enhancing the platform's extensibility.
- Developed a demo extension showcasing the new API, significantly reducing boilerplate code and improving developer experience.
- Updated documentation to reflect the new architecture and provide guidance for extension development.
- Prepared for testing and validation of the extension system, marking a significant step towards a robust plugin ecosystem.
- Introduced a comprehensive sync infrastructure, including a `Syncable` trait for models, enabling automatic logging of changes in a dedicated sync log database.
- Added `LeadershipManager` for managing leader election and lease tracking, ensuring reliable synchronization across devices.
- Implemented `TransactionManager` to handle atomic writes and sync logging, enhancing data integrity during state changes.
- Created a new `sync` module with necessary components such as `SyncLogDb`, `SyncLogEntry`, and sync protocol handlers for efficient communication between leader and follower devices.
- Developed integration tests to validate the sync functionality and ensure robust performance across various scenarios.
- Introduced three new markdown files: CLI_LIBRARY_SYNC_COMPLETE.md, IMPLEMENTATION_COMPLETE.md, and LIBRARY_SYNC_SETUP_IMPLEMENTATION.md for comprehensive documentation.
- Updated various CLI domain modules to improve argument handling and output formatting.
- Enhanced device, index, job, library, location, network, and search modules for better integration and user experience.
- Refactored code across multiple domains to improve maintainability and clarity.
- Enhanced the `Event` enum documentation for clarity on its purpose.
- Modified the Swift client generation script to create event samples for enum generation.
- Updated the Swift client to utilize the new `EventElement` type for event handling.
- Improved error handling and logging in the event subscription process.
- Refactored the types generated for Swift to align with the new event structure.
- Refactored `remove_emojis.sh` to enhance the emoji removal process by using a more efficient Python script for detecting and removing 3D/colorful emojis followed by spaces.
- Updated various Rust files to remove emoji characters from log messages and print statements, resulting in a cleaner output.
- Improved user experience with color-coded progress and results during the emoji removal process.
- Ensured that the script counts processed files and reports modifications accurately.
- Introduced `remove_emojis.sh`, a Bash script designed to identify and remove 3D/colorful emojis followed by a space in Rust files while preserving simple symbols.
- The script counts the number of Rust files processed and reports modifications made.
- Updated various Rust files to remove emoji characters from log messages and print statements for a cleaner output.
- Enhanced user experience by providing color-coded output for progress and results during the emoji removal process.
- Updated various print statements across the codebase to remove emoji indicators, ensuring a more uniform logging style.
- Adjusted output messages in CLI, job handling, and event processing to enhance clarity and maintainability.
- Refactored job-related logging to provide clearer status updates without emoji clutter.
Co-authored-by: ijamespine <ijamespine@me.com>
- Upgraded `bincode` to version `2.0.0-rc.3` with `serde` feature enabled in `Cargo.toml`.
- Refactored error handling to accommodate changes in `bincode` by updating the error conversion implementation in `error.rs`.
- Modified deserialization logic in macros to align with the new `bincode` API.
- Removed outdated metrics test file from the core examples.
Co-authored-by: ijamespine <ijamespine@me.com>
- Added a comprehensive testing guide for the FS Event Pipeline, detailing metrics collection, logging, and troubleshooting steps.
- Updated the CLI package name from "spacedrive-cli" to "sd-cli" in Cargo.toml for consistency.
- Modified various files to reflect the new package structure and improve logging and metrics handling.
Co-authored-by: ijamespine <ijamespine@me.com>
- Added a new end-to-end search test with real data to validate the complete search workflow, including file indexing and search result display.
- Removed the outdated search integration test to streamline testing efforts.
- Introduced the `chrono` dependency in the CLI for enhanced date handling in search operations.
- Updated the search output structure to include matched content and improved pagination handling.
These changes enhance the robustness of the search feature and ensure better integration with real-world data scenarios.
- Changed the visibility of the execute_fast_search method from private to public, allowing it to be accessed outside the FileSearchQuery implementation.
- This modification facilitates integration with other components that require fast search capabilities using FTS5.
Jamie Pine