Extends the FileCopyJob to support PULL operations, enabling files to be
copied FROM remote devices TO the local device. Previously, only PUSH
operations worked (local -> remote).
Changes:
- Add PullRequest/PullResponse messages to FileTransferMessage enum
- Add TransferDirection enum to detect PUSH vs PULL operations
- Refactor RemoteTransferStrategy to detect direction automatically
- Implement execute_pull() for remote source -> local destination
- Add handle_pull_request() handler on the protocol side
- Add path validation for security (prevent directory traversal)
- Add integration test for PULL operations
The strategy automatically detects direction based on SdPath:
- Source is local + Destination is remote = PUSH (existing behavior)
- Source is remote + Destination is local = PULL (new behavior)
This enables users to drag files from remote device explorers to local
folders, which previously failed with 'Source must be local path' error.
Closes FILE-005
- Updated `stream_file_data` to bypass encryption, as Iroh provides end-to-end encryption for connections.
- Adjusted `FileTransferProtocolHandler` to skip decryption, directly using the received encrypted data.
- Enhanced logging to reflect changes in data handling during file transfers.
- Added tests to ensure session keys are distinct and validate key swapping functionality.
- Improved error handling in `DevicePersistence` for corrupted session keys during device loading.
- Updated documentation for copy strategies to clarify their use cases and selection logic.
- Improved logging in `DevicePersistence` to track loading of paired devices and handle expired session keys.
- Added detailed logging for session key retrieval in `FileTransferProtocolHandler`.
- Introduced a new test for file transfer with daemon restart to ensure session key persistence across restarts.
- Updated macOS subproject reference to indicate a dirty state.
- Refactored `FileCopyJob` to improve logging for final destination calculations.
- Enhanced `stream_file_data` logging to include destination path in transfer requests.
- Added `find_containing` method to `EphemeralIndex` for substring search functionality.
- Improved logging in `ephemeral_search` to track search results and filtering.
- Replaced print statements with tracing logs in `FileTransferProtocolHandler` for better debugging.
- Added `DirectoryPreview` component for rendering a grid view of subdirectories in the Quick Preview.
- Updated `ContentRenderer` to utilize `DirectoryPreview` for directory types.
- Introduced TypeScript integration test for verifying cache updates upon file deletions, ensuring accurate directory listing after deletions.
- Enhanced keyboard handling in Explorer to prevent shortcuts when input fields are focused.
- Added `run_in_background` field to `IndexerJobConfig` to allow jobs to run without persisting to the database or updating the UI.
- Updated `should_persist` method in `IndexerJob` to account for background jobs.
- Modified `handle_new_directory` in `DatabaseAdapter` to create background indexer jobs.
- Ensured default value for `run_in_background` is set to false in various configurations.
- Introduced a new integration test for bulk file moves to validate cache updates and file handling.
- Added a new test suite in `core/tests/typescript_bridge_test.rs` to facilitate end-to-end testing between Rust and TypeScript.
- Introduced `IndexingHarnessBuilder` enhancements to support a daemon RPC server for TypeScript tests.
- Created TypeScript test files for `useNormalizedQuery` to validate file move and folder rename operations.
- Configured Bun test environment with `bunfig.toml` and setup scripts for improved testing efficiency.
- Updated `TcpSocketTransport` to handle TCP connections for the TypeScript client, ensuring robust communication with the Rust backend.
- Enhanced logging and error handling for better visibility during test execution.
- Added a new function `validate_and_reset_library_if_needed` to check if the current library exists and reset the state if it doesn't.
- Integrated this validation process into the daemon connection initialization, ensuring the app state is consistent with the available libraries.
- Enhanced error handling and logging for better debugging during library validation.
- Introduced a new test suite for file move operations, covering various scenarios to ensure robust handling of file movements and state integrity.
- Introduced a new test suite for validating the tag workflow, including tag creation, application, and database persistence.
- Implemented event collection to verify resource events during tag operations.
- Added assertions to ensure correct tag associations with files and proper database entries.
- Enhanced logging for better traceability of test execution and outcomes.
- Refactored the `IndexPersistence` implementation to improve error handling and logging when querying directory paths.
- Updated the change detection logic to always load existing entries during reindexing, ensuring accurate detection of moves, modifications, and deletions.
- Introduced a new test suite for folder rename operations, covering both persistent and ephemeral indexing scenarios, validating UUID/inode preservation and parent-child relationship integrity.
- Enhanced the `IndexingHarnessBuilder` to allow disabling the filesystem watcher for specific tests, improving test flexibility.
- Added closure table integrity verification to ensure proper connections between renamed folders and their children, preventing data inconsistencies.
- Introduced a new `IndexMode::Stale` variant to enhance the indexing process by leveraging modified time (mtime) pruning.
- Updated `IndexerJobConfig` to utilize the new mode, allowing for more efficient directory exploration based on filesystem changes.
- Implemented logic in the discovery phase to skip directories with matching mtime, improving performance and reducing unnecessary indexing.
- Enhanced `IndexMode` with helper methods to check for mtime pruning and retrieve the inner indexing mode.
- Updated relevant functions and tests to support the new stale detection capabilities, ensuring robust handling of file indexing and statistics tracking.
- Refactored existing code to accommodate the new indexing strategy while maintaining backward compatibility with previous modes.
- Updated the TODO list to clarify the quick preview reporting issue.
- Enhanced the Tauri app's menu to include custom clipboard operations (cut, copy, paste) with appropriate state management.
- Introduced a new `statistics_listener` service to manage per-library statistics recalculation based on resource changes, improving performance and responsiveness.
- Added configuration options for enabling/disabling the statistics listener in the application settings.
- Refactored event handling in the job manager to improve logging and status monitoring.
- Updated tests to include the new statistics listener functionality and ensure proper event collection during file operations.
- Introduced a new `useClipboard` hook for managing clipboard operations, including copy and cut functionalities.
- Updated file context menu to utilize the clipboard for copy and cut actions, improving user experience.
- Enhanced file operation modal to handle paste operations using clipboard contents, streamlining file management.
- Refactored keyboard navigation in the Explorer component to support clipboard actions, ensuring intuitive user interactions.
- Improved job management by adding sound notifications specific to job types, enhancing feedback during file operations.
- Added tests to verify UUID preservation during reindexing and file modifications in `indexing_test.rs`.
- Updated `Inspector` and `ExplorerProvider` to handle virtual files and views, ensuring consistent navigation and state management.
- Introduced `VirtualPathBar` and `VolumeSizeBar` components for improved UI representation of virtual views and volume sizes.
- Refactored `PathBar` and `FileCard` components to integrate new virtual file handling logic, enhancing user experience in the Explorer interface.
- Added a new `file_count` column to the `content_kinds` table to track the number of content identities for each kind, improving statistics calculation efficiency.
- Implemented a migration script to add the `file_count` column and ensure backward compatibility.
- Updated the `Library` struct to include a method for updating content kind counts based on existing content identities.
- Introduced a new query for retrieving content kind statistics, including file counts, to facilitate better data analysis and reporting.
- Refactored related components to support the new statistics functionality, enhancing overall data integrity and performance.
- Updated the CI workflow to run all tests for the `sd-core` package, ensuring comprehensive coverage.
- Modified test execution commands to include specific tests for better granularity and clarity.
- Improved device management by regenerating slugs upon name changes and updating global device slugs.
- Enhanced session key generation for improved security during device pairing.
- Updated progress monitoring tests to ensure robust performance tracking for large file operations.
- Updated the CI workflow to run tests specifically for the package for clarity.
- Deleted outdated example test files for event serialization, location watcher, and simple metrics to reduce clutter.
- Enhanced error handling in for better robustness.
- Improved device data handling in by adding additional fields for comprehensive device information.
- Adjusted test assertions in to align with updated change detection logic.
- Adjusted the CI workflow to run tests with updated command syntax for improved clarity.
- Marked subproject commits in ios, landing, macos, and workbench as dirty to reflect uncommitted changes.
- Modified conflict resolution options in copy action tests to use `None` for better handling of file operations.
- Enhanced metadata handling in various tests to improve context and accuracy during execution.
- Removed the setup step for native dependencies in the CI workflow to streamline the build process.
- Updated the `copy_action_test` to include conflict resolution options for better handling of file conflicts.
- Enhanced metadata handling in `entry_move_integrity_test` to include job policies.
- Added metadata to event handling in `file_structure_test` for improved context during file operations.
- Updated service configuration in `file_sync_simple_test` and `file_sync_test` to use `fs_watcher_enabled` and included default logging settings.
- Deleted the outdated README.md from the tests directory to reduce clutter and improve documentation focus.
- Changed the CI workflow to run tests on self-hosted runners for better performance and flexibility.
- Updated the test execution condition to trigger on push events and specific pull requests.
- Consolidated test execution to run all tests in the workspace instead of a single test.
- Modified the default features in Cargo.toml for the server and core applications to include "heif" and "ffmpeg" for enhanced media support.
- Added metrics collection capabilities to the SyncProtocolHandler for improved monitoring of sync operations.
- Refactored sync tests to streamline setup and improve clarity in test scenarios.
- Refactored the upsert logic for Space, SpaceItem, and SpaceGroup entities to utilize atomic operations, preventing race conditions during synchronization.
- Enhanced database interactions with `on_conflict` handling based on UUID for improved data integrity.
- Introduced a comprehensive sync setup test to validate the functionality and ensure no UNIQUE constraint errors occur during device pairing and library sharing.
- Marked subproject commits in ios, landing, macos, and workbench as dirty to reflect uncommitted changes.
- Added detailed logging in the volume listing and sync processes to improve traceability and debugging capabilities, including information on fetched volumes and device pairing statuses.
- Introduced a new test for bidirectional volume sync to ensure both devices correctly receive each other's volumes, enhancing test coverage for sync functionality.
- Introduced AppResetContext to manage app reset operations across components.
- Updated App and RootLayout to utilize the reset context, allowing for a clean reset of the app state.
- Enhanced SettingsScreen with a reset data option, prompting users for confirmation before clearing all data and refreshing the app.
- Added core reset action in the backend to handle data deletion and cleanup operations.
- Improved user feedback during reset operations with alerts and status messages.
- Improved the `emit_change_event` method in `DatabaseAdapter` to handle resource events more effectively, including detailed logging for event emissions.
- Updated the `PersistentEventHandler` to ensure paths are registered with the filesystem watcher upon startup, enhancing real-time monitoring capabilities.
- Introduced new event collectors for better diagnostics and logging of filesystem and core events during tests.
- Enhanced integration tests to cover various scenarios, including batch file operations and delete/restore patterns, ensuring accurate event handling and preventing duplicates.
- Updated the `delete` method in `MemoryAdapter` to emit a `ResourceDeleted` event, allowing the frontend to update its cache accordingly.
- Modified `CollectedCoreEvent` structure to include an `event_type` field for better event categorization.
- Enhanced event collection logic in `CoreEventCollector` to handle both `ResourceChanged` and `ResourceDeleted` events, improving event summary reporting.
- Updated `remove_entry` and `remove_directory_tree` methods in `EphemeralIndex` to ensure proper removal of entries from parent nodes' children lists.
- Introduced `FsEventCollector` and `CoreEventCollector` for improved event collection from the filesystem watcher and core event bus, respectively.
- Enhanced test harness to utilize the new collectors, ensuring accurate event logging and summary reporting.
- Added scenarios to test delete and restore patterns, verifying correct entry counts and preventing duplicate entries.
- Implemented methods for batch creation and deletion of files and directories in the test harness.
- Added scenarios to verify the creation and deletion of multiple files and directories, ensuring proper event handling.
- Enhanced logging for better visibility during test execution.
- Updated the macOS event handler to improve directory deduplication and manage recent directory creations effectively.
- Updated `connect` methods in `EphemeralEventHandler` and `PersistentEventHandler` to be asynchronous, allowing for non-blocking connections to the `FsWatcherService`.
- Modified `init_handlers` in `FsWatcherService` to support async initialization of event handlers.
- Adjusted service startup methods to await handler initialization, ensuring proper setup before starting the watcher.
- Enhanced test harness to utilize the new async capabilities, improving event collection and verification processes.
- Introduced a new File Operation Modal for interactive file copy/move operations with conflict resolution options.
- Integrated sound effects for file operations, including copy and startup sounds.
- Updated the Explorer component to utilize the new modal for drag-and-drop file operations.
- Refactored job management hooks to unify job data handling and improve performance.
- Enhanced documentation for the File Operation Modal and sound effects integration.
- Add a complete ephemeral indexing subsystem
- core/src/ops/core/ephemeral_status with input/output and query types
- core/src/ops/indexing/ephemeral with arena, cache, registry,
index_cache, types
- expose EphemeralIndexCache and EphemeralIndex through core modules
- EphemeralIndexCache supports
get/insert/create_for_indexing/mark_indexing_complete eviction and
stats
- Implement EphemeralIndex data structures for memory-efficient storage
- NodeArena, NameCache, NameRegistry, and related types
- Add EphemeralIndex status API
- EphemeralCacheStatusInput and EphemeralCacheStatusQuery
- EphemeralCacheStatus with per-index details
- Wire ephemeral indexing into the indexing flow
- Change default Ephemeral Indexer behavior to shallow mode
- Align code to EphemeralIndex usage across the codebase
- Enhance content kind detection in UI
- Add getContentKind(file) helper (prefers content_identity.kind, then
content_kind)
- Use getContentKind in Explorer utilities and UI components
- Invalidate directory listings when location index_mode changes
- Add useLocationChangeInvalidation to trigger refetches for ephemeral
vs persistent indexing transitions
- Misc refactors and formatting to accommodate the new modules and APIs
- Revert to the November 16 baseline for realtime sync - Use a single
state watermark and drop per-resource watermarks, counts, and hashes
from watermark messages - Revert DataAvailableNotification handling and
the updated FK filtering approach in favor of NULL-based handling - Add
registry helper get_model_type_by_table to map tables to models for FK
resolution - Update mocks and tests to reflect the baseline (linkage
stats in summary) - Fix test infrastructure: remove block_in_place usage
and minor imports - Remove obsolete test fixture file
1. Watermark Messages: - Nov 16: Simple - single my_state_watermark,
single state_watermark in response - Main: Complex - per-resource
my_resource_watermarks HashMap, resource counts, content hashes 2.
DataAvailableNotification: - Nov 16: Doesn't exist - Main: New message
type that triggers watermark exchanges 3. FK Mapper: - Nov 16: Sets
missing FKs to NULL, applies all records - Main: Marks records with
missing FKs as failed, filters them out 4. Watermark Storage (internal,
not in messages): - Nov 16: Single last_watermark per resource - Main:
Dual watermarks - cursor_watermark and
- Scope the closure query to the indexing path by resolving its entry_id
from directory_paths and use that for descendants; fall back to the
location root if not found. - Seed all ancestors between the location
root and the indexing path into the in-memory cache during the
processing phase to ensure correct parent links. - Seed ancestor
directories into the create path cache to prevent ghost folders on
subpath indexing. - Add test core/tests/indexing_move_folder_bug_test.rs
reproducing the ghost-folder bug when moving a folder into a managed
location and asserting correct parent linkage.
- Added `include_descendants` option to event filters, allowing recursive path matching for resource events.
- Updated `affects_path` method to support descendant matching, improving event handling accuracy.
- Refactored subscription logic to utilize the new filtering capabilities, ensuring only relevant events are processed.
- Introduced tests for event filtering to validate exact vs. recursive matching, enhancing reliability of event-driven updates.
- Updated related components to leverage the new filtering options, improving overall performance and user experience.
- Introduced a new constructor for the HLC struct to allow explicit initialization with timestamp, counter, and device ID.
- Updated indexing tests to improve error handling for entry IDs and added a comprehensive test for reindexing after offline changes, ensuring all original and new files are detected correctly.
- Introduced new modules and actions for generating media proxies and thumbstrips, improving media handling capabilities.
- Added configuration, error handling, and processing logic for both proxy and thumbstrip generation.
- Updated existing media operations to integrate new functionalities, ensuring seamless media processing and user experience.
- Enhanced tests to cover new features and ensure reliability in media operations.
- Introduced new functionality to validate resource counts during watermark exchanges, improving synchronization accuracy.
- Added methods to retrieve device-owned resource counts for gap detection, ensuring that discrepancies are identified and addressed.
- Updated the `SyncMessage` structure to include actual resource counts, facilitating better tracking of synchronization state.
- Enhanced the `PeerSync` implementation to clear watermarks for mismatched resources, allowing for surgical recovery in case of count mismatches.
- Improved logging for better visibility into synchronization processes and potential issues.
- Introduced a new `DataAvailableNotification` message type to notify peers of new data availability, triggering immediate watermark exchanges for synchronization.
- Enhanced the `PeerSync` implementation to handle notifications and manage real-time activity tracking per peer, improving synchronization efficiency.
- Updated the `SyncProtocolHandler` to process the new notification type and initiate catch-up logic accordingly.
- Implemented periodic watermark checks to ensure timely synchronization and prevent missed events during idle periods.
- Added support for per-resource watermarks in the synchronization process, allowing for fine-grained comparison and catch-up logic.
- Implemented new configuration options for real-time batching, including maximum entries before flush and flush interval in milliseconds.
- Enhanced the `apply_state_change` function to check for tombstoned records, preventing the resurrection of deleted entries.
- Updated various components to utilize the new watermark and batching features, improving synchronization efficiency and robustness.
- Added clipboard permissions for reading and writing text in the default capabilities configuration.
- Enhanced the sync test harness to capture job lifecycle events, improving tracking of indexing completion and orphaned file management.
- Implemented checks for orphaned files during synchronization, ensuring robust handling of incomplete sync scenarios.
- Updated the submodule reference in `workbench`.
- Added debug logging in `ResourceManager` to track resource event emissions and virtual resource mappings.
- Improved the `PeerSync` implementation to emit resource events in a non-blocking manner, enhancing UI reactivity and ensuring proper resource formatting.
- Updated integration tests to include new resource types and refined synchronization checks for stability.
