This patch makes the code more robust around event removals. Sorting
events by their position is no longer done in the `Deduplicator` but in
a new `RoomEventCacheState::remove_events` method, which removes events
in the store and in the `RoomEvents`. This method is responsible to sort
events, this stuff is less fragile like so.
This patch adds a test for `sort_events_by_position_descending`. It
also updates this function so that events are sorted by their chunk
identifier from newest to oldest, it makes no difference but it matches
the order of the position indices too. Everything “dimension” is
descending.
This patch uses `DeduplicationOutcome` to remove events either in
memory, or in the store, when required. The `remove_events_by_id` method
has been renamed `remove_events_by_position`.
This patch redesigns `Deduplicator::filter_duplicate_events`.
First off, `filter_duplicate_events` does remove events with no valid
ID. At the same time, it removes duplicate events within the new events
(`events`). This check was done in the `BloomFilterDeduplicator` but
not in the `StoreDeduplicator`. Now it's done at the front of these
implementations, directly inside `Deduplicator`.
Second, this patch introduces `DeduplicationOutcome` to replace the
return type `(Vec<Event>, Vec<OwnedEventId>)`, especially because
now it would have become `(Vec<Event>, Vec<(OwnedEventId, Position)>,
Vec<(OwnedEventId, Position)>)`. Why?
1. Because the positions of the duplicated events are returned,
2. We differentiate between in-memory vs. in-store duplicated events.
Third, now there are positions associated to duplicated events, events
must be sorted. It's the role of `sort_events_by_position_descending`.
This way, `DeduplicatorOutcome` brings guarantees and less checks are
required.
It could be that we have a mismatch between network and disk, after
running a back-pagination:
- network indicates start of the timeline, aka there's no previous-batch
token
- but in the persisted storage, we do have an initial empty events chunk
Because of this, we could have weird transitions from "I've reached the
start of the room" to "I haven't actually reached it", if calling the
`run_backwards()` method manually.
This patch rewrites the logic when returning `reached_start`, so that
it's more precise:
- when reloading an events chunk from disk, rely on the previous chunk
property to indicate whether we've reached the start of the timeline,
thus avoiding unnecessary calls to back-paginations.
- after resolving a gap via the network, override the result of
`reached_start` with a boolean that indicates 1. there are no more gaps
and 2. there's no previous chunk (actual previous or lazily-loaded).
In the future, we should consider NOT having empty events chunks, if we
can.
Token revocation was split out from MSC2964 to MSC4254, and RP-Initiated
logout is now mentioned only as an alternative.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
Will allow to share code when the backend is switched to the oauth2 crate too.
It will also allow to expose the device authorization grant directly in Oidc, if necessary.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
The `.from()`, `.with_delay()` and `.limit()` functions are not very
explicit about what they did, and the `with_delay` one in particular led
me to think that it would introduce a delay *in the response*, while it
indicated a delay was expected as part of the matched URL.
Instead, this patch proposes to prefix all matchers with `match_`, to
make it clearer that… they will match the incoming query: match_from,
match_delay, match_limit.
Thanks to this change, the `RoomMessagesResponseTemplate` can be renamed
from `delayed` to `with_delay`, which was my original intent, before I
noticed another `with_delay` with totally different semantics.
This patch adds the new `from_all_chunks` function in the
`linked_chunk::lazy_loader` module. It is only used for testing
purposes. It aims at replacing `LinkedChunkBuilderTest` (see next
patches). Why? Because `from_all_chunks` uses `from_last_chunk` and
`insert_new_first_chunk`: if `from_all_chunks` is able to find all
errors that `LinkedChunkBuilderTest` finds, it's a bingo. Transitively,
it proves that `from_last_chunk` and `insert_new_first_chunk` are
correct!
This patch updates `Update::RemoveChunk` to emit `VectorDiff::Remove`.
Until now, `RemoveChunk` was expecting the chunk to be
empty, because it is how it is used so far. However, with
https://github.com/matrix-org/matrix-rust-sdk/pull/4694, it can change
rapidly.
The code before this patch was doing this:
- look if there's any prev-batch token available right now, aka look if
there's a gap in the in-memory linked chunk
- look at the first chunk; if it's a gap, return to the caller so it
resolves it
The check is done twice at two different levels, which is confusing.
Instead, this patch rewrites it so that the chunk is done only in
`load_more_event_backwards()`.
Note this is also correct for the case storage is disabled; in this
case, we early return and always try to resolve the gap anyways.
This patch renames the `builder` module to `lazy_loader`.
The `LinkedChunkBuilder`'s methods are now functions.
The `LinkedChunkBuilder` struct is removed. Finally,
`LinkedChunkBuilderError` is renamed `LazyLoaderError`.
The `LinkedChunkBuilderTest` struct is kept for the moment. It's going
to be replaced soon.
This patch adds an index on `events.event_id` and on `events.room_id`
so that queries on this column are faster. It mostly happens for the
`Deduplicator`, which runs for every backwards pagination or sync.
This patch also updates the query in `filter_duplicated_events` to
sort event by their `chunk_id` and `position` so that the results are
constant, it helps when testing.
This is the method to get the server metadata in the latest draft of
[MSC2965](https://github.com/matrix-org/matrix-spec-proposals/pull/2965).
We still keep the old behavior with `GET /auth_issuer` as fallback for
now because it has wider server support.
There are some pre-main commit cleanups to simplify the main commit.
This can be reviewed commit by commit.
The changes were tested with the oidc_cli example on beta.matrix.org.
Closes#4550.
---------
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
