This commit introduces a detailed memory breakdown feature for the ephemeral indexing process, allowing users to view memory usage by component. It also enhances UUID management by implementing lazy generation and assignment of UUIDs during indexing and searching operations. These changes improve memory tracking and optimize UUID handling, contributing to better performance and user experience in the indexing system.
This commit introduces a new snapshot mechanism for ephemeral indexes, allowing for fast restoration between sessions. The snapshot functionality utilizes zero-copy binary serialization with zstd compression, significantly reducing load times from over 10 minutes to just 1-2 seconds. The implementation includes methods for saving and loading snapshots, as well as enhancements to the existing indexing structure to support this feature. Additionally, the codebase is updated to ensure compatibility with the new snapshot format, improving overall performance and user experience.
This commit improves the `CopyJobDetails` component by adding an auto-scroll feature that centers the currently copying file in the transfer queue. Additionally, it implements a mechanism to refetch copy metadata when the completed file count changes, ensuring that the displayed information is always up-to-date. The UI is further refined with updated icons and styles for better user experience during file operations.
- Fix 'access_token and refresh_token can not be set at the same time' error
- Update Dropbox to use only refresh_token (OpenDAL requirement)
- Add client_id/client_secret to OAuth credential storage
- Add validation for empty OAuth credentials
- Update CLI and volume manager for enhanced OAuth support
- Removed the static ASCII logo and implemented a new function to calculate brightness on a sphere, allowing for a dynamic rendering of the Spacedrive logo as a purple orb using ANSI colors and Unicode half-blocks.
- Enhanced the `print_logo_colored` function to utilize the new brightness calculation and color gradient, improving visual representation.
- Updated related documentation to reflect the changes in logo rendering.
- Added `remove_from_library` field to `DeviceRevokeInput` and `DeviceRevokeAction` to control whether a device should be removed from library databases during revocation.
- Updated the `from_input` method in `DeviceRevokeAction` to handle the new field.
- Modified the revocation logic to conditionally remove devices from libraries based on the `remove_from_library` flag, improving flexibility in device management.
- Adjusted the CLI argument parsing to default `remove_from_library` to false, ensuring devices remain in libraries unless explicitly specified for removal.
- Introduced a new command `ResetCache` in the `IndexCmd` enum to allow users to reset the ephemeral index cache.
- Implemented `EphemeralCacheResetAction` to handle the cache reset logic, including confirmation checks and logging.
- Added input and output structures for the cache reset action, providing clear feedback on the number of cleared paths.
- Enhanced the `EphemeralIndexCache` with a `clear_all` method to facilitate the clearing of all cached paths and entries.
- Added a new `LibraryLoadFailed` event to capture and report errors when loading libraries, including error type categorization for frontend notifications.
- Introduced a method to count `.sdlibrary` directories without loading them, improving the library initialization process.
- Updated the library manager to emit the new event upon load failures, enhancing error handling and user feedback.
- Enhanced logging to provide better insights during library loading and initialization.
- Updated event handling in job management to include device IDs for better tracking of job activities across devices.
- Introduced a new JobActivityClient for subscribing to job activity from remote devices.
- Added hardware specifications to device configuration and database schema, improving device information tracking.
- Refactored event structures to accommodate new fields related to job activities and device management.
- Updated the user interface to display job activity for devices, enhancing user experience in monitoring job statuses.
- Added a step to set up native dependencies using `cargo xtask setup` in both CI and release workflows.
- Updated the build commands for CLI binaries to include `heif` and `ffmpeg` features, ensuring the binaries are built with the necessary capabilities.
- Modified `Cargo.toml` files for `sd-cli`, `sd-server`, and `tauri` to define `heif` and `ffmpeg` features separately, improving modularity and clarity in dependency management.
- Remove TTL-based ephemeral cache and switch to a permanent in-memory
cache.
- Reuse ephemeral UUIDs when creating persistent entries to preserve
continuity of user data.
- Populate ephemeral UUIDs during the processing phase and expose
get_ephemeral_uuid in the indexer state.
- Remove the location invalidation hook and related UI usage.
- Add a complete ephemeral indexing subsystem
- core/src/ops/core/ephemeral_status with input/output and query types
- core/src/ops/indexing/ephemeral with arena, cache, registry,
index_cache, types
- expose EphemeralIndexCache and EphemeralIndex through core modules
- EphemeralIndexCache supports
get/insert/create_for_indexing/mark_indexing_complete eviction and
stats
- Implement EphemeralIndex data structures for memory-efficient storage
- NodeArena, NameCache, NameRegistry, and related types
- Add EphemeralIndex status API
- EphemeralCacheStatusInput and EphemeralCacheStatusQuery
- EphemeralCacheStatus with per-index details
- Wire ephemeral indexing into the indexing flow
- Change default Ephemeral Indexer behavior to shallow mode
- Align code to EphemeralIndex usage across the codebase
- Enhance content kind detection in UI
- Add getContentKind(file) helper (prefers content_identity.kind, then
content_kind)
- Use getContentKind in Explorer utilities and UI components
- Invalidate directory listings when location index_mode changes
- Add useLocationChangeInvalidation to trigger refetches for ephemeral
vs persistent indexing transitions
- Misc refactors and formatting to accommodate the new modules and APIs
- Add SyncEventsArgs to export sync events from the CLI - Wire
SyncCmd::Events and implement export_events to fetch - format and write
results - Implement JSON, SQL, and Markdown exporters - with optional
device data in the output - Extend protocol with EventLogRequest and
EventLogResponse - Enable LogSyncHandler to handle event log requests
and return logs - Expose log_handler from BackfillManager for event
logging - Update docs with CLI examples and protocol overview
- Remove relay-URL based pairing flow and rely on pkarr/DNS for remote
discovery - Extend pairing code with optional node_id and adopt version
2 QR format - Generate pairing codes using node_id and drop relay_url in
non-QR flows - Enable combined mDNS, pkarr, and DNS discovery in the
endpoint - Cache relay_url in persisted device info for reconnection
optimization - Update protocol handlers to propagate node_id and
pkarr-based flow - Adjust tests and logging to reflect the new
cross-network pairing
- Add show_paired flag to device listing (CLI arg and input) - Extend
LibraryDeviceInfo with is_paired and is_connected; include Paired
devices in library listing when requested - Add Devices group to UI
(DevicesGroup) and hook into SpaceGroup - Extend device queries/types to
support show_paired and paired devices - Refactor Dockerfile to
multi-stage Bun + Rust builds; reuse web assets - Remove obsolete
core/ops/entries/mod.rs
- Added `include_descendants` option to event filters, allowing recursive path matching for resource events.
- Updated `affects_path` method to support descendant matching, improving event handling accuracy.
- Refactored subscription logic to utilize the new filtering capabilities, ensuring only relevant events are processed.
- Introduced tests for event filtering to validate exact vs. recursive matching, enhancing reliability of event-driven updates.
- Updated related components to leverage the new filtering options, improving overall performance and user experience.
- Introduced a new `unique_bytes` field in the volume model to track deduplicated content size.
- Implemented a volume refresh action to recalculate unique bytes for all volumes owned by the device, emitting a cache refresh event to invalidate frontend caches.
- Updated the storage overview component to display unique bytes and other volume statistics.
- Enhanced the event system to handle cache invalidation events, ensuring real-time updates across the application.
- Added necessary migrations and database updates to support the new volume tracking features.
- Refactored the event monitoring logic in the CLI to improve readability and maintainability.
- Updated the event summarization function to enhance formatting and clarity of event messages.
- Introduced a dedicated log streaming mechanism in the core, separating log handling from event processing.
- Enhanced the RPC server to support real-time log subscriptions, allowing for more efficient log message delivery.
- Cleaned up legacy log event handling, ensuring a clearer distinction between events and logs in the system.
- Introduced a new Events domain for monitoring events in real-time, allowing users to filter events by type, library ID, job ID, and device ID.
- Updated the CLI to include an Events command, enabling users to access the new monitoring features.
- Enhanced the Cargo.lock and related modules to support the new functionality, ensuring seamless integration with existing systems.
- Added necessary arguments and command handling for the Events command in the CLI.
- Introduced a new field `unique_content_count` in `LibraryStatistics` to track the number of unique content identities.
- Updated various components to calculate and display unique content count, enhancing library statistics output.
- Modified the CLI and interface to reflect the new unique content metric, improving user insights into library data.
- Adjusted serialization and API responses to include the unique content count for better data consistency across the system.
This commit introduces comprehensive metrics collection for the sync service. It includes tracking state transitions, operation counts, data volumes, performance indicators, and error events. The changes also add a new CLI command to view sync metrics and integrate metrics into the sync service's core functionality.
Co-authored-by: ijamespine <ijamespine@me.com>
Replace confusing RegisterOnly action with clearer ShareLocalLibrary and JoinRemoteLibrary actions. Add placeholder for future MergeLibraries.
- Remove RegisterOnly from LibrarySyncAction enum
- Add ShareLocalLibrary (share your library to remote device)
- Add JoinRemoteLibrary (join existing remote library)
- Add MergeLibraries stub for future implementation
- Update CLI with three-option menu instead of two
- Simplify CLI arg parsing for new actions
The new UX is clearer: users choose to either share their library or join a remote one, instead of the ambiguous "register only" concept.
