This patch fixes a bug in `xtask log sync` which can miss a `sync_once`
log when the `pos` field is absent. It happens when there is no `pos`!
Example where `pos` is absent before `timeout`. Note the double space
before `timeout`:
```
… > sync_once{conn_id="room-list" timeout=0} > send{request_id="REQ-15" …
```
While when the `pos` is present, it's:
```
… > sync_once{conn_id="room-list" pos="0/m67590980…" timeout=30000} > send{request_id="REQ-23" …
```
**Note:** _this pull request has a companion pull request in the
[`complement-crypto`](https://github.com/matrix-org/complement-crypto/pull/229)
repository, which must be merged in conjunction with this one._
_Before merging, this should be tested in conjunction with the Element X
iOS client to ensure that TLS v1.3 is working properly._ @stefanceriu
has agreed to work on this.
## Overview
The primary change in this pull request upgrades the `reqwest`
dependency to its latest version, which defaults to using `rustls` with
support for `rustls-platform-verifier` instead of `native-tls` (see
[`reqwest@0.13.0`](https://github.com/seanmonstar/reqwest/releases/tag/v0.13.0)).
The benefit here is that `rustls` supports TLS v1.3 on all platforms,
whereas [`native-tls` does
not](https://github.com/sfackler/rust-native-tls/pull/278).
Additionally, this pull request makes `rustls` the default TLS
implementation in all the crates in this repository.
This will be particularly helpful with element-hq/element-x-ios#786.
## Changes
- `reqwest` bumped to `0.13.1`
- The API for adding/replacing certificates has changed a bit, so this
required some updating in `HttpSettings::make_client`
- `oauth2-reqwest` added in favor of `oauth2/reqwest`
- This is required in order to be compatible with `reqwest^0.13`
- _**`oauth2-reqwest` is currently in alpha release, so it probably
makes sense to let this stabilize a bit.**_ For details, see
https://github.com/ramosbugs/oauth2-rs/issues/333#issuecomment-3906712203.
- `getrandom` bumped to `0.3.4`
- This is required in order to be compatible with `oauth2@5.1.0`
- `proptest` bumped to `1.9.0`
- This is required in order to be compatible with `getrandom@0.3.4`
- Make `rustls` the default TLS implementation
## Questions
### Mirror feature flag names?
A number of feature flags have been replaced in the dependencies above.
1. _**`reqwest/rustls-tls` => `reqwest/rustls`**_ - this is simply a
name change, but is semantically identical (see
[`reqwest@0.13.0`](https://github.com/seanmonstar/reqwest/releases/tag/v0.13.0)).
2. _**`getrandom/js` => `getrandom/wasm_js`**_ - the semantics here have
changed slightly, but it seems to just make it easier to enable the
`wasm_js` backend (see
[`getrandom@0.3.4`](https://github.com/rust-random/getrandom/blob/master/CHANGELOG.md#major-change-to-wasm_js-backend)).
At any rate, I have updated references to these flags in each of the
various `Cargo.toml` files, but have not changed the names of our
exposed features to mimic those in the dependencies.
Any thoughts or preferences on whether to mirror those names? That
would, of course, result in a breaking change.
### Default to using `rustls`? Deprecate `native-tls`?
Now that the dependencies have all been bumped, we can use `rustls` on
all platforms. Should this be the new default given that `native-tls`
will very likely never support TLS v1.3 on Apple devices? And should
`native-tls` be deprecated as a result?
**UPDATE:** _The consensus here seems to be that we should default to
using `rustls`, but that `native-tls` should still be available._
---
Fixes#5800.
- [ ] Public API changes documented in changelogs (optional)
Signed-off-by: Michael Goldenberg <m@mgoldenberg.net>
---------
Signed-off-by: Michael Goldenberg <m@mgoldenberg.net>
The log message can contain HTML data. It happens notably when the
homeserver is broken, and an HTML document is returned in some errors.
We don't want to parse the fields in this case, because HTML breaks
everything.
This patch adds the `open` attribute to the `<details>` for displaying
leaf nodes of the tree. My experience shows that it's nice to always
have them open. The “occurences” list is still folded, so it shouldn't
take to much space on the screen most of the time (one line per level of
logs in this node).
This patch adds the `log overview` task that generates a standalone HTML
report representing the logs as a tree where each node is a target, and
each leaf is a log location with occurrences, spans and fields.
Each node displays the sum of errors and warnings for this node.
It helps to quickly spot the problematic targets, and it guides to
exploration of the node without taking the reader's hand by trying to
draw conclusions. We don't want to guide the reader to a mistake: we
just want to guide the reader to draw its own conclusions.
Each line can be highlighted. When a node is folded/closed, it is also
highlighted if at least one of its child is highlighted.
The occurrences of logs are displayed on a timeline.
This patch updates the `cargo xtask log sync` command to extract the
`pos` and `timeout` fields so that we can display them. The `timeout`
is displayed in its own column, while the `pos` is displayed in the
line summary.
This patch introduces a new family of commands: `xtask log`. The goal
is to manipulate logs, to extract the right amount of data we need to
solve specific problems. The first member of this family is `sync` to
visualise logs about the sync process. It presents the sync requests and
responses in a table, with a "timeline" _à la_ network profiler graph.
The code is rather simple, on purpose. The generated HTML reports are
lightweight, and fully standalone: no JavaScript, pure HTML and CSS, no
external resources. These reports can be shared or archived super
easily.
Features:
- requests/responses are grouped by connection ID
- permalink to specific request ID
- status have colours
- time is displayed in a human form
- duration is calculated from the log timestamps
- view syncs in a "tree-like" flavor, a "time graph", super quick to
spot long requests
- each line can be "opened" to see details, so far only log line numbers
to get more context manually
Sometimes, running the wasm tests with a given runner will fail with
obscure, undecipherable reasons. As a result, it's convenient to be able
to locally run the wasm-pack tests with only a single runner, which this
commit allows.
This patch introduces the `sqlite` and `indexeddb` feature flag,
enabling the use of SQLite or IndexedDB for the stores. This patch also
introduces the ability to use non-persistent, in-memory stores.
The new `ClientBuilder::in_memory_store`, `ClientBuilder::sqlite_store`
and `ClientBuilder::indexeddb_store` methods are introduced to
configure the stores. This patch adds new `SqliteStoreBuilder` and
`IndexedDbStoreBuilder` structure.
**NOTE:** _this should not be merged until matrix-org/rust-indexed-db#1
is merged! The `[patch]` in this branch should point to the official
`matrix-org` fork of `rust-indexed_db`, but is currently pointed at my
personal fork._
## Background
This pull request makes updates
[`indexed_db_futures`](https://docs.rs/indexed_db_futures/latest/indexed_db_futures/index.html)
in the `matrix-sdk-indexeddb` crate. The reason we'd like to update this
dependency is because the version currently used does not fully support
the Chrome browser (see #5420).
The latest version of `indexed_db_futures` has significant changes. Many
of these changes can be integrated without issue. There is, however, a
single change which is incompatible with the `matrix-sdk-indexeddb`
crate. Namely, one cannot access the active transaction in the callback
to update the database (for details, see Alorel/rust-indexed-db#66).
### An Updated Proposal
Originally, new migrations were implemented in order to work around this
issue (see #5467). However, the proposal was ultimately rejected (see
@andybalaam's
[comment](https://github.com/matrix-org/matrix-rust-sdk/pull/5467#issuecomment-3149550617)).
For this reason, the dependency has instead been `[patch]`ed in the
top-level `Cargo.toml` with a modified version of `indexed_db_futures`
(see matrix-org/rust-indexed-db#1). Furthermore, these changes have been
proposed to the maintainer and are awaiting feedback (see
Alorel/rust-indexed-db#72).
### Why do we need the active transaction in our migrations?
The `crypto_store` module provides access to the active transaction to
its migrations (see
[here](ca89700dfe/crates/matrix-sdk-indexeddb/src/crypto_store/migrations/mod.rs (L211))).
Furthermore, there is a single migration (`v11_to_v12`) in the
`crypto_store` module which actually makes use of the active transaction
(see
[here](ca89700dfe/crates/matrix-sdk-indexeddb/src/crypto_store/migrations/v11_to_v12.rs (L23))).
For clarity, the reason `v11_to_v12` is problematic in the latest
versions of `indexed_db_futures` is because it is simply adding an index
to an object store which was created in a different migration and this
requires access to the active transaction. All the other migrations
create object stores and indices in the same migration, which does not
suffer from the same issue.
## Changes
- Move `indexed_db_futures` to the workspace `Cargo.toml` and add a
`[patch]` so that it points to a modified version.
- Add `GenericError` type and conversions in order to more easily map
`indexed_db_futures` errors into `matrix-sdk-*` errors.
- Update all IndexedDB interactions so that they use the upgraded
interface provided by `indexed_db_futures`
- Add functionality for running `wasm-pack` tests against Chrome
---
Closes#5420.
---
- [ ] Public API changes documented in changelogs (optional)
Signed-off-by: Michael Goldenberg <m@mgoldenberg.net>
---------
Signed-off-by: Michael Goldenberg <m@mgoldenberg.net>
The former has been merge in the latter, and it errors when generating
the docs in a recent version of nightly, like the one used on docs.rs.
This also requires to bump the version of nightly used in CI, otherwise
it would break the docs generation.
Signed-off-by: Kévin Commaille <zecakeh@tedomum.fr>
This PR makes some non-domain-specific changes across multiple crates
that are required for proper testing of features implemented for #5397.
* Adds a `experimental-encrypted-state-events` feature flag across the
SDK.
* Introduces a feature set into xtask to ensure feature-gated tests are
run during CI.
* Minor fix to a test that would otherwise fail with the newly
introduced CI.
We use `cargo ndk -p {package_name}`, where `-p` is short for
`--package`. However, `cargo ndk` has introduced the `-p` option (see
c6b93a89a2),
short for `--platform`. It creates a confusion and the command line
doesn't execute properly. Let's use the long option `--package` to
clarify everything.
The swift bindings aren't getting tested (they don't run) in CI anymore,
so building with the reldbg profile (that's a workaround to make it run
and not crash in production) doesn't provide more value than building in
debug mode, while taking much longer to build.
Let's use the default dev profile for this; we have to specify it
manually, because the default for the xtask command is to use the
`reldbg` profile otherwise.
This requires a fix for the dev profile, that consists in being able to
set the iOS deployment target, and set it to a high value in CI
settings. Production builds *don't* have to set it, though.
Process:
- set style_edition to 2024 in .rustfmt.toml
- run `cargo fmt`
- undo .rustfmt.toml change
- run `cargo fmt` again
- manually rewrap some strings
The system of platform targets was already quite messy, and becoming
even worse as we start preparing for Wasm support. Switch to features
instead to make this easier to work with.
Because yes, some weeks, we are very productive!
This patch adds `--limit 100` to `gh pr list` so that we are sure to not
miss pull requests if there are many.
