- Simplified the processing of WatermarkExchangeResponse messages to ensure they are only handled via the bidirectional stream.
- Added a warning log for unexpected responses, improving visibility into potential misuse of the sync request mechanism.
- Removed unnecessary parameters from the response handling, enhancing code clarity and maintainability.
- Updated the watermark exchange process to utilize a request/response pattern, enhancing synchronization reliability.
- Enhanced logging to provide clearer insights into the request and response lifecycle during watermark exchanges.
- Added error handling for unexpected message types in the response, improving robustness.
- Clarified comments to better describe the asynchronous nature of the watermark exchange process.
- Revised the catch-up logic to make the sync loop passive, driven by watermark exchanges and reconnection events.
- Enhanced comments to clarify the new approach, focusing on edge cases where retry states indicate a stuck condition.
- Removed the previous time-based sync checks in favor of a more robust failure tracking mechanism.
- Updated the entry count query to utilize the entry_closure table for more accurate counting of entries associated with device locations.
- Enhanced comments to clarify the purpose of the query and the use of closure tables in the synchronization process.
- Changed logging from info to debug for various library operations to reduce log verbosity.
- Updated the periodic watermark check interval from 5 minutes to 1 minute for more frequent synchronization checks.
- Commented out the ContentBreakdown component in the Overview route for potential future use.
- Updated the watermark exchange methods to accept a database connection, enabling retrieval of peer resource counts during synchronization.
- Enhanced logging to provide visibility into the resource counts being exchanged, improving debugging and monitoring capabilities.
- Addressed potential issues with resource count availability by ensuring counts are fetched correctly during static exchanges.
- Introduced new functionality to validate resource counts during watermark exchanges, improving synchronization accuracy.
- Added methods to retrieve device-owned resource counts for gap detection, ensuring that discrepancies are identified and addressed.
- Updated the `SyncMessage` structure to include actual resource counts, facilitating better tracking of synchronization state.
- Enhanced the `PeerSync` implementation to clear watermarks for mismatched resources, allowing for surgical recovery in case of count mismatches.
- Improved logging for better visibility into synchronization processes and potential issues.
- Introduced a new `DataAvailableNotification` message type to notify peers of new data availability, triggering immediate watermark exchanges for synchronization.
- Enhanced the `PeerSync` implementation to handle notifications and manage real-time activity tracking per peer, improving synchronization efficiency.
- Updated the `SyncProtocolHandler` to process the new notification type and initiate catch-up logic accordingly.
- Implemented periodic watermark checks to ensure timely synchronization and prevent missed events during idle periods.
- Added support for per-resource watermarks in the synchronization process, allowing for fine-grained comparison and catch-up logic.
- Implemented new configuration options for real-time batching, including maximum entries before flush and flush interval in milliseconds.
- Enhanced the `apply_state_change` function to check for tombstoned records, preventing the resurrection of deleted entries.
- Updated various components to utilize the new watermark and batching features, improving synchronization efficiency and robustness.
- Updated the submodule reference in the landing app.
- Added a statistics recalculation listener in the LibraryManager to monitor resource changes.
- Included clipboard permissions in the capabilities configuration for enhanced functionality.
- Improved the LibraryInfoOutput struct to implement the Identifiable trait for better resource management.
- Updated the submodule reference in `workbench`.
- Added debug logging in `ResourceManager` to track resource event emissions and virtual resource mappings.
- Improved the `PeerSync` implementation to emit resource events in a non-blocking manner, enhancing UI reactivity and ensuring proper resource formatting.
- Updated integration tests to include new resource types and refined synchronization checks for stability.
- Updated the foreign key mapping logic in `fk_mapper.rs` to return an error when a referenced record is missing, instead of setting the foreign key to NULL. This change ensures that sync dependencies are properly managed and allows for retrying when dependencies are resolved.
- Enhanced the error handling in `peer.rs` to buffer state changes that depend on missing foreign keys, improving the robustness of the synchronization process and preventing data loss during updates.
- Updated the `Library` implementation to generate HLCs, append to the peer log, and emit real-time events for instant synchronization.
- Introduced a mechanism to collect entries for real-time broadcasts, ensuring immediate updates are sent to peers.
- Added a yield point in the content phase to prevent foreign key orphaning by ensuring content identity events are emitted before entry updates.
- Enhanced the `NetworkingService` to cache active connections, allowing for reuse of existing connections and reducing the overhead of establishing new ones.
- Implemented logic to check the status of cached connections before creating new ones, improving efficiency during synchronization and request handling.
- Added detailed logging for connection reuse and creation, aiding in monitoring and debugging network interactions.
- Introduced a batching mechanism in `PeerSync` to accumulate state change events, improving network efficiency by processing up to 100 entries at once or flushing every 50ms.
- Added a new method `flush_state_change_batch` to handle the processing of batched state changes, enhancing overall synchronization performance.
- Updated event handling logic to support batching, reducing the frequency of individual state change processing and improving throughput.
- Added a mechanism in `PeerSync` to check for active real-time broadcasts, allowing the system to skip catch-up if real-time sync is ongoing.
- Introduced a new field `last_realtime_activity` to track the timestamp of the last real-time broadcast, enhancing synchronization efficiency.
- Updated the catch-up logic in `SyncService` to incorporate the real-time sync status, improving the overall synchronization process and preventing data duplication.
- Updated the synchronization logic in `PeerSync` to avoid buffering state changes originating from the local device, addressing a critical data loss issue during backfill requests.
- Enhanced logging to clarify the nature of the broadcasted state change events, improving traceability of local changes.
- Updated `sync_depends_on` in the entry model to include `content_identity` and `user_metadata`, ensuring foreign key references are validated before entries are processed.
- Enhanced the synchronization process in `PeerSync` to broadcast buffered state and shared changes to peers after local application, addressing a critical data loss issue.
- Added logging for the number of processed state and shared changes to improve monitoring of synchronization activities.
- Introduced a new field `unique_content_count` in `LibraryStatistics` to track the number of unique content identities.
- Updated various components to calculate and display unique content count, enhancing library statistics output.
- Modified the CLI and interface to reflect the new unique content metric, improving user insights into library data.
- Adjusted serialization and API responses to include the unique content count for better data consistency across the system.
- Updated `get_min_acked_hlc` to exclude self-ACKs from the minimum acknowledged HLC calculation, ensuring stale self-ACKs do not block pruning.
- Enhanced `on_ack_received` to ignore ACKs from the local device, adding defensive logging to catch potential bugs.
- Modified backfill logic to skip sending ACKs for changes created by the local device, improving efficiency during synchronization.
- Updated submodule references for 'workbench' and 'landing' to their latest commits.
- Enhanced database connection pooling in `mod.rs` to support higher concurrency, adjusting max connections to a configurable pool size.
- Introduced batch processing functions in `fk_mapper.rs` for efficient UUID lookups and foreign key mappings, reducing database load during synchronization.
- Added new utility functions for batch processing in `sync/mod.rs` to streamline foreign key resolution and improve sync performance.
- Updated logging statements in volume detection and APFS parsing to use debug level instead of warn, improving log clarity and reducing noise.
- This change enhances the debugging process without affecting the functionality of volume detection.
- Introduced a dedicated sync event bus to improve coordination for sync events, preventing starvation from high-volume events.
- Updated TransactionManager to utilize both sync and general event buses for better event handling.
- Refactored PeerSync to listen for sync events, ensuring proper handling of state change and shared change events.
- Adjusted LibraryManager and related components to support the new sync event bus.
This change enhances the responsiveness and reliability of the synchronization process across the system.
Backend changes:
- Modify libraries.info query to return cached stats immediately if available
- Detect empty/stale stats and calculate synchronously on first load
- Trigger background recalculation on every query to keep cache fresh
- Emit ResourceChanged events (resource_type: library) when stats update
- Add job completion hooks to trigger stats recalculation after every job
Frontend changes:
- Add library-changed event emission when setCurrentLibrary() is called
- Listen for library-changed events in useNormalizedCache to auto-refetch
- Support instant cache updates via ResourceChanged event listeners
This enables:
✅ Immediate load of cached stats (0ms after first time)
✅ Real-time updates when jobs complete
✅ Background recalculation to keep stats fresh
✅ Event-driven cache updates via normalizedCache pattern
