Add a second full-sync-up mode to sliding sync. Previously - and still the default for backwards compatibility, but now named `PagingFullSync` - was to page through the list by the page-size, de-validating the previous page of items. The newly added `GrowingFullSync` instead grows the window by the given number `batch_size` per request, starting and keeping it from `0`. In the latter we might be pushing more data over the connection and are slightly slower, but the top always stays active and thus reactive to changes.
Furthermore the developer can now configure an optional maximum number to grow/paginate the full-sync up to (so stopping before actually having reached `count` whatever is smaller). This is already exposed via FFI, too.
- [x] add new full-sync-mode
- [x] add limited sync-up mode (similar to full sync but only to a limit `n`)
- [x] implement sync-up in jack-in
With these changes, the user of sliding sync can configure the SlidingSync-Builder to store and recover the state from storage. For that they should call cold_cache("my-sliding-sync-key") with their preferred storage location during setup time on SlidingSyncBuilder. If a cached version is found under that key, the internal state of the rooms-list as well as each view found (identified by their name) will be set from storage immediately - allowing the user to show the last cached state. The Builder then also uses the same key to store its latest state after every update received from remote.
👉 Note on room list:
As we store a disconnected state, we are saving all RoomList entries as either Empty or Invalidated. This allows for an easier updating when the server comes back with results, as we don't track the ranges in cache - in the view of the server, we haven't seen anything yet.
👉 Note on timeline events:
This does store all existing timeline events - initial and seen during the session (as we keep track of them right now) - and will recover that state. However, as we can't be sure wether we have gaps in the timeline, the timeline items are reset upon receiving updates for them from the server.
👉 Note on (full-sync-)view:
While we are caching the server returned results (view list, room count and room info) as is, we are not caching the internal state (whether we are catching up) nor the ranges (as they are likely to be out of date) - those will be acting the same way you configured them, just with preloaded results. So for a full-sync-view, it will still do requests in batches and replace the corresponding set of rooms. Which could mean you might see the same room appear twice, though the cached one would be marked as invalidated. This might be problematic if the list the UI shows is longer than the batches fetched, but would be resolved quickly when catching up.
👉 On recovery:
When the sliding sync receives a M_UNKNOWN_POS, indicating the server has expired our session, sliding sync now transparently retries with up to three times to restart the sliding sync with full set of extensions and the latest views at their existing windows, the current room state is held. For full sync this starts a new sync-up with the existing room list staying intact. This also works from the offline at start.
Add general extension framework for sliding sync, implementing the e2ee, to-device and account-data extensions as per existing proxy implementation. Add a new (ffi exposed) function to use activate the extensions.
Also extends jack-in to have permanent login and storage support now, rather than posting an access token and expose messages inside the sliding-sync layer to actually use the decrypted messages if given. Contains a lot of fixes around these aspects, too, like uploading any remaining messages from the olm-machine on every sliding-sync-request or processing even if no room data is present (which can happen now as processing only extensions might takes place).
* starting with jack-in
* starting by flying tui
* connecting to real server, showing info
* add .env to gitignore
* infrastructure for tests
* display loading time for syncv2
* minor design updates
* initial sync
* finalise first edition of sliding sync
* directly link to sliding sync and show rooms list
* nicer UI, toggle logs
* passing through sliding sync homeserver
* separate syncs and disable v2 autostart
* selecting rooms
* nicer view
* configurable batches and more default needed events
* selecting rooms
* calculate and show status info per room
* precalculated room stats
* restructure code to allow for cancellation of streams
* finish up merge updates
* fix calculation error in room list len
* cleaning up system flow
* fixing sync up
* new multi-view API
* move sliding sync in separate module
* fixing format
* adding and clearning views
* expose filters and timeline limits
* renamed
* adding room subscriptions to sliding sync
* update summary
* live fetching and subscriptions in jack-in
* subscribe to selected room
* starting to switch to tuireal - using example
* status bar and first linkup
* re-adding rooms
* implementing port for customised update event
* showing details and timeline
* fix formatting
* cleaner UI, updating details quickly
* make it green
* implement other Ops
* proper handling of invalidation
* saving sliding sync results to db, too
* saving new prev_batch field if given
* split events and timeline
* cleaning up
* live updates
* upgrading to latest ruma and matrix-sdk
* Update tui-logger to fix the broken build
* fixing latest ruma sliding-sync-branch updates
* feat: first set of ffi sliding sync
* expose sliding sync via FFI
* implement un-/subscribe
* implement view state updates
* updating to latest JSON format and ruma update
* implementing room selecting for new data model
* fixing room selection
* fixing feature flag requirements for sliding-sync
* fixing style, clippy and docs
* style(sliding-sync): fixing rust format styles
* fix(ffi): fixing sliding sync merge failure
* fix(jack-in): update jack-in to latest ruma
* fix(sliding-sync): need to have a version set before polling proxy
* expose sliding sync builder over ffi
* add SlidingSyncViewBuilder
* add forgotten changes on sdk itself
* new file logger feature on jack for deeper logging
* fix(http-client): log the raw request we are sending out
* feat(sliding-sync): better logging
* fix(sliding-sync): switch to full-listen after reaching live state in full-sync and make sure we replace the correct entries
* feat(ffi): expose sliding sync view ranges
* fix(ffi): fixing sliding sync start_sync loop to actually loop
* feat(sliding-sync): allow lookup of room data
* feat(sliding-sync-ffi): fetching name of room
* feat(ffi): expose unread_notifications of rooms
* feat(ffi): stoppable spawn for sliding sync
* fix(ffi): expose has_unread_notifications on room
* feat(sliding-sync): latest room message
* fix(sliding-sync): update to latest ruma layout
* doc(sliding-sync): adding docs to builder fns
* feat(sliding-sync): extra signal on the view to inform about general updates
* fix(sync-v4-ffi): expose new callbacks via ffi
* fix(sliding-sync): reduce default required states to make things faster
* fix(sliding-sync): fix build params
* feat(jack-in): extended instrumentation
* fix(sliding-sync): unbreak faulty feature-gate
* fix(sliding-sync-ffi): mut where mut is due
* fix(sdk): allow separate homeserver on every request to unbreak using send on client while in sliding sync on a proxy
* fix(jack-in): update to latest dependencies, that work
* feat(ffi): helper to patch sliding sync room into regular room
* style(jack-in): cargo fmt
* fix(sliding-sync): Update to latest ruma changes
* fix(sliding-sync): fix missing FFI updates to latest ruma
* feat(sliding-sync)!: simplify stream cancellation, cancel ffi sync if already existing
* fix: timeline backwards pagination must work without synctoken
* fix(sliding-sync): clarify order of messaes in alive TL; pick correct last item
* fix: update view delegate api for clarity
* style(jack-in): fix cargo warnings
* feat(sliding-sync): update room details
* fix(sliding-sync): only update room info selectively when given
* fix(sliding-sync-ffi): convert and store counts as u32, check against 0 for has notificaitons
* style: cargo fmt, file endings and a few other minor style fixes
* docs(jack-in): improving CLI and Readme
* feat(sliding-sync): allow setting of required event_states on viewbuilder
* style(sliding-sync): docs and minor fixes
* style(sliding-sync): various clippy fixes
* style(jack-in): clippy suggestions applied
* fix(sliding-sync): Delegate becomes observer
* test(sdk): adding test for request config
* docs: Fixing copyright header
* style(ffi): Nicer naming of params for observer
* fix(ffi): sliding sync is not optional for now
* fix(sdk): remove superflusous tracing instrumentation
* fix(sdk): use structured logging
* fix(jack-in): removed unneded log import
* fix(jack-in): use server_name rather than deprecated user_id on ClientBuilder
* style: typo and clippy
* style(sliding-sync): clippy and formatting
* fix(sliding-sync): cleaning up minor syntax issues
* fix: remove unneded feature-definition section
* fix(sliding-sync): minor fixes as per review
* fix(sliding-sync): Make Builders owned
* fix(sliding-sync): more minor style improvements
* fix(sliding-sync): minor style improvements
* fix(sliding-sync): remove homeserver from RequestConfig, use specific internal fn instead
Co-authored-by: Stefan Ceriu <stefanc@matrix.org>
