Fix https://github.com/FreshRSS/FreshRSS/issues/8268
To better support user management on FreshRSS instance with many users.
SQL speed improved. On a reduced test with 5 users, including some large accounts (PostgreSQL on a very tiny and slow server), improving from ~2.3s to ~1.8s, which gives ~20% speed improvement.
Then tested with 1000 users, with only the default feed (on my old desktop computer):
```sh
for i in {1..1000}; do ./cli/create-user.php --user=freshrss$i --password=freshrss; done
app/actualize_script.php
cli/access-permissions.sh
```
SQLite:
```console
$ time cli/user-info.php | wc -l
1001
real 0m1.366s
user 0m0.908s
sys 0m0.475s
```
PostgreSQL:
```console
$ time cli/user-info.php | wc -l
1001
real 0m28.498s
user 0m12.137s
sys 0m2.217s
```
MariaDB:
```console
# time ./cli/user-info.php | wc -l
1001
real 0m49.485s
user 0m1.276s
sys 0m2.258s
```
Yes, SQLite is much faster - not a surprise for such use-cases, where the TCP connection is not re-used.
I have added some CLI options to disable some statistics:
```sh
cli/user-info.php --no-db-size --no-db-counts
```
For the Web UI, I have disabled detailed user statistics if it takes too long, and retrieve missing user statistics asynchronously via JavaScript. Lazy loading of the user details based on IntersectionObserver, with maximum 10 requests in parallel.
Web UI tested on 1000 users as well. Checked with SeaMonkey.
* Change SQL update query for MariaDB / MySQL
fix https://github.com/FreshRSS/FreshRSS/issues/5707
* No change for SQLite
* Fix merge error
* Update MySQL version on the model of PostgreSQL
Performance to be tested
* Fix LEFT JOIN, also for PostgreSQL / SQLite
* Fix alias
* Reduce MySQL deadlock
* Fix compatibility with SQLite
* Back to identical SQL for all databases
I just realised that `commitNewEntries()` was not sorting articles properly before insertion in database when using MySQL: Articles were not sorted by publication date as expected from the temporary table before insertion in the final table. MySQL was not picking the correct field, so fixed with an explicit alias.
Discovered because I did some tests with MySQL in https://github.com/FreshRSS/FreshRSS/pull/6957
At the same time, I did the same change for PostgreSQL and SQLite although those were not affected.
* force correct index for MySQL when listing entries
* Make special case for MySQL / MariaDB
---------
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
* GReader API frss:priority
Experiment with a FreshRSS namespace in the GReader API to see whether there is any interest.
fix https://github.com/FreshRSS/FreshRSS/issues/1868
`'frss:priority'` can be: `'important'`, `'main'`, `'category'`, `'feed'` (there is also the value `hidden`, but which is filtered out and as such never sent through the API at the moment)
* Add visibility feed
https://github.com/FreshRSS/FreshRSS/pull/7972
Example: `userdate:PT1H` to select only articles modified by user during the last hour
Fix https://github.com/FreshRSS/FreshRSS/issues/4280#issuecomment-3393078024
Useful for instance to bulk mark as unread recently marked articles by error:
1. Click on the toggle button to show the read articles (making sure the toggle for the unread articles is off)
2. Sort by *User modified 9→1*
3. Filter by *user modified date*, for instance to the last 3 hours by typing `userdate:PT3H`
4. Click in the drop-down menu *Mark selection as unread*
P.S.: I have added at the same time a bunch of unit tests for date-related logic
* feat: Add user modified functionality
Closes https://github.com/FreshRSS/FreshRSS/issues/7862
Changes proposed in this pull request:
This is an implementation of the proposed feature. It allows entries to have a new field that will be updated whenever an item is marked as read/unread or bookmark/removed from bookmarks. And a new sort criteria to sort by it.
How to test the feature manually:
1. Mark items from a feed as read/unread
2. Mark items from a feed as bookmark / remove bookmark
3. Sort by the new criteria
* feat: Add sort functionality
* feat: Add sort nav button
* fix: Use correct migrations
* fix: Add internationalization
* fix: Linter errors
* chore: PR comments
* Update app/i18n/fr/index.php
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
* Update app/i18n/pl/index.php
Co-authored-by: Inverle <inverle@proton.me>
* Update app/i18n/nl/index.php
Co-authored-by: Frans de Jonge <fransdejonge@gmail.com>
* make fix-all
* Fixes
* More fixes sort
* Fix wrong index
* Fix unneeded column
* Fix auto-create indexes
* Some copilot suggestions
* One more fix
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
---------
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
Co-authored-by: Inverle <inverle@proton.me>
Co-authored-by: Frans de Jonge <fransdejonge@gmail.com>
Important feeds are not supposed to be marked as read from another view. This was already working for the main view, but this PR fixes the case of a category view.
Follow-up of:
* https://github.com/FreshRSS/FreshRSS/pull/5782
* Add new visibility priority *Show in its feed*
fix https://github.com/FreshRSS/FreshRSS/pull/7970#issuecomment-3293917428 (you can't directly filter a hidden feed, it just shows a 404 page)
And add a new visibility *Show in its feed* to show the feed in the list but not its articles.
Ensure that visibility *hidden* is not shown to API.
* TODO for later
* Update app/i18n/pl/sub.php
Co-authored-by: Inverle <inverle@proton.me>
fix https://github.com/FreshRSS/FreshRSS/issues/7887
We have two concepts: how much a feed is shown or not (controlled by priority), and how often a feed is refreshed (or not, in which case it is archived).
This PR removes the wording *Archived* from the *visibility* parameter, since this is not what it does.
* Sort by category name, feed name
fix https://github.com/FreshRSS/FreshRSS/issues/7698
Note that sorting is done with the default SQL collation for now, meaning that lower-case vs. upper-case and diacritics are influencing the sorting order. Improvements left for future work.
Watch out that those sorting criteria are slower due to additional joins, additional requests, and poorer indexes.
* i18n:pl
Co-authored-by: Inverle <inverle@proton.me>
* i18n: nl
Co-authored-by: Frans de Jonge <fransdejonge@gmail.com>
* Fix preserve sort
---------
Co-authored-by: Inverle <inverle@proton.me>
Co-authored-by: Frans de Jonge <fransdejonge@gmail.com>
Closes#3789, #6503
Icon setting when no custom icon is set yet:
- `Change...` button opens a file dialog, and after selecting a file shows the chosen icon in the preview on the left. `Submit` must be clicked after selecting the icon.
- `Reset to default` changes the preview icon to the default one, and also requires `Submit` to be clicked to apply the changes.
Full list of changes:
- CSP now includes `blob:` in `img-src` for
- `indexAction()` and `feedAction()` in `subscriptionController.php`
- all of the view actions in `indexController.php`
- Introduce new attribute `customFavicon (boolean)` for feeds that indicates if the feed has a custom favicon
- `hashFavicon()` in `Feed.php` is dependent on this attribute
- `hashFavicon()` has a new parameter called `skipCache (boolean)` that allows the reset of the favicon hash for the Feed object
- `resetFaviconHash()` just calls `hashFavicon(skipCache: true)`
- `f.php` URLs now have the format of `/f.php?h=XXXXX&t=cachebuster`, where the `t` parameter is only used for serving custom favicons
- if `t` parameter is set, `f.php` returns a `Cache-Control: immutable` header
- `stripos` and `strpos` were changed to `str_contains` in various places (refactor)
- JS for handling the custom favicon configuration logic is in `extra.js` inside `init_update_feed()` which is called when feed configuration is opened from the aside or when the subscription management page with the feed is loaded
- Server-side code for uploading the icon in `subscriptionController.php` under `feedAction()`
- Errors that may occur during the setting of a custom favicon:
- Unsupported image file type (handled only server-side with `isImgMime()`)
- When the file is bigger than 1 MiB (default), handled both client-side and server-side
- Standard feed error when `updateFeed()` fails
- JS vars `javascript_vars.phtml` are no longer escaped with `htmlspecialchars()`, instead with json encoding,
- CSS for disabled buttons was added
- Max favicon file size is configurable with the `max_favicon_upload_size` option in `config.php` (not exposed via UI)
- Custom favicons are currently deleted only when they are either reset to the default icon, or the feed gets deleted. They do not get deleted when the user deletes their account without removing their feeds first.
- ` faviconPrepare()` and `faviconRebuild()` are not allowed to be called when the `customFavicon` attribute is `true`
- New i18n strings:
- `'sub.feed.icon' => 'Icon'`
- `'sub.feed.change_favicon' => 'Change…'`
- `'sub.feed.reset_favicon' => 'Reset to default'`
- `'sub.feed.favicon_changed_by_ext' => 'The icon has been set by the <b>%s</b> extension.'`
- `'feedback.sub.feed.favicon.too_large' => 'Uploaded icon is too large. The maximum file size is <em>%s</em>.'`
- `'feedback.sub.feed.favicon.unsupported_format' => 'Unsupported image file format!'`
- Extension hook `custom_favicon_hash`
- `setCustomFavicon()` method
- `resetCustomFavicon()` method
- `customFaviconExt` and `customFaviconDisallowDel` attributes
- example of usage: https://github.com/FreshRSS/Extensions/pull/337
- Extension hook `custom_favicon_btn_url`
- Allows extensions to implement a button for setting a custom favicon for individual feeds by providing an URL. The URL will be sent a POST request with the `extAction` field set to either `query_icon_info` or `update_icon`, along with an `id` field which describes the feed's ID.
The list or articles with a user label with a custom sort was broken when using PostgreSQL
Example: `https://freshrss.example.net/i/?a=normal&get=T&sort=title&order=ASC`
```
SQL error FreshRSS_EntryDAO::listWhereRaw["42P10",7,"ERROR: for SELECT DISTINCT, ORDER BY expressions must appear in select list\nLINE 3: ...d_entry = e.id WHERE 1=1 AND e.id <= $1 ORDER BY e.title DE...\n ^"]
```
Add option to sort results by received date (existing, default), publication date, title, URL (link), random.
fix https://github.com/FreshRSS/FreshRSS/issues/1771
fix https://github.com/FreshRSS/FreshRSS/issues/2083
fix https://github.com/FreshRSS/FreshRSS/issues/2119
fix https://github.com/FreshRSS/FreshRSS/issues/2596
fix https://github.com/FreshRSS/FreshRSS/issues/3204
fix https://github.com/FreshRSS/FreshRSS/issues/4405
fix https://github.com/FreshRSS/FreshRSS/issues/5529
fix https://github.com/FreshRSS/FreshRSS/issues/5864
fix https://github.com/FreshRSS/Extensions/issues/161
URL parameters:
* `&sort=id` (current behaviour, sorting according to newest received articles)
* `&sort=date` (publication date, which is not indicative of how new an article is)
* `&sort=title`
* `&sort=link`
* `&sort=rand` (random order - which disables infinite scrolling, at least for now)
combined with `&order=ASC` or `&order=DESC`
## Implementation notes
The sorting criteria by *received date* (id), which is the default, and which was the only one before this PR, is the one that has the best sorting characteristics:
* *uniqueness*: no entries have the exact same received date
* *monotonicity*: new entries always have a higher received date
* *performance*: this field is efficiently indexed in database for fast usage, including for paging (indexing could also be done to other fields, but with lower effective performance)
In contrary, sorting criteria such as by *publication date*, by *title*, or by *link* are neither unique nor monotonic. In particular, multiple articles may share the same *publication date*, and we may receive articles with a *publication date* far in the future, and then later some new articles with a *publication date* far in the past.
To understand why sorting by *publication date* is problematic, it helps to think about sorting by *title* or by *link*, as sorting by *title* and by *publication date* share more or less the same characteristics.
### Problem 1: new articles
New articles may be received in the background after what is shown on screen, and before the next user action such as *mark all as read*. Due to the lack of *monotonicity* when sorting by e.g. *publication date* or *title*, users risk marking as read a batch of articles containing some fresh articles without seeing them.
Mitigation: A parameter `idMax` tracks the maximum ID related to a batch of actions such as *mark all as read* to exclude articles received after those that are displayed.
### Problem 2: paging / pagination
When navigating articles, only a few articles are displayed, and a new "page" of articles needs to be received from the database when scrolling down or when clicking the button to show more articles. When sorting by e.g. *publication date* or *title*, it is not trivial to show the next page without re-showing some of the same articles, and without skipping any. Indeed, views are often with additional criteria such as showing only unread articles, and users may mark some articles as read while viewing them, hereby removing some articles from the previous pages. And like for *Problem 1*, new articles may have been received in the background. Consequently, it is not possible to use `OFFSET` to implement pagination (so the patches suggested by a few users were wrong due to that, in particular).
Mitigation: `idMax` is also used (just like for *Problem 1*) and a *Keyset Pagination* approach is used, combining an unstable sorting criterion such as *publication date* or *title*, together with *id* to ensure stable sorting. (So, 2 sorting criteria + 1 filter criteria)
See e.g. https://www.alwaysdeveloping.net/dailydrop/2022/07/01-keyset-pagination/
### Problem 3: performance
Sorting by anything else than *received date* (id) is doomed to be slow(er) due to the combination of 3 criteria (see *Problem 2*). An `OFFSET` approach (which is not possible anyway as explained) would be even slower. Furthermore, we have no SQL index at the moment, but they would not necessarily help much due to the multiple sorting criteria needed and involving some `OR` logic which is difficult to optimise for databases.
The nicest syntax would be using tuples and corresponding indexes, but that is poorly supported by MySQL https://bugs.mysql.com/bug.php?id=104128
Mitigation: a compatibility SQL syntax is used to implement *Keyset Pagination*
### Problem 4: user confusion
Several users have shown that they do not fully understand the difference between *received date* and *publication date*, and particularly not the pitfalls of *publication date*.
Mitigation: the menus to mark-as-read *before 1 day* and *before 1 week* are disabled when sorting by anything else than *received date*. Likewise, the separation headers *Today* and *Yesterday* and *Before yesterday* are only shown when sorting by *received date*.
Again here, to better understand why, it helps to think about sorting by *title* or by *link*, as sorting by *title* and by *publication date* share more or less the same characteristics.
* [ ] We should write a Q&A and/or documentation about the problems associated to *sorting by publication date*: risks of not noticing new publication, of inadvertently marking them as read, of having some articles with a date in the future hanging at the top of the views (vice versa when sorting in ascending order), performance, etc.
### Problem 5: APIs
Sorting by anything else than *received date* breaks the guarantees needed for a successful synchronisation via API.
Mitigation: sorting by *received date* is ensured for all API calls.
* PHPStan 2.0
fix https://github.com/FreshRSS/FreshRSS/issues/6989https://github.com/phpstan/phpstan/releases/tag/2.0.0https://github.com/phpstan/phpstan/blob/2.0.x/UPGRADING.md
* More
* More
* Done
* fix i18n CLI
* Restore a PHPStan Next test
For work towards PHPStan Level 10
* 4 more on Level 10
* fix getTagsForEntry
* API at Level 10
* More Level 10
* Finish Minz at Level 10
* Finish CLI at Level 10
* Finish Controllers at Level 10
* More Level 10
* More
* Pass bleedingEdge
* Clean PHPStan options and add TODOs
* Level 10 for main config
* More
* Consitency array vs. list
* Sanitize themes get_infos
* Simplify TagDAO->getTagsForEntries()
* Finish reportAnyTypeWideningInVarTag
* Prepare checkBenevolentUnionTypes and checkImplicitMixed
* Fixes
* Refix
* Another fix
* Casing of __METHOD__ constant
* Regex search
fix https://github.com/FreshRSS/FreshRSS/issues/3549
* Fix PHPStan
* Fix escape
* Fix ungreedy
* Initial support for regex search in PostgreSQL and MySQL
* Improvements, support MySQL
* Fix multiline
* Add support for SQLite
* A few tests
* Added author: and inurl: support, documentation
* author example
* Remove \b for now
* Disable regex sanitization for now
* Fix getInurlRegex
* getNotInurlRegex
* Quotes for inurl:
* Fix test
* Fix quoted tags + regex for tags
https://github.com/FreshRSS/FreshRSS/issues/6761
* Fix wrong regex detection
* Add MariaDB
* Fix logic
* Increase requirements for MySQL and MariaDB
Check support for multiline mode in MySQL
* Remove sanitizeRegexes()
* Allow searching HTML code
Allow searching for instance `/<pre>/`
Fix https://github.com/FreshRSS/FreshRSS/issues/6775#issuecomment-2331769883
* Doc regex search HTML
* Fix Doctype
* Minor update whitespace PHPCS rules
To simplify our configuration, apply more rules, and be clearer about what is added or removed compared with PSR12.
Does not change our current conventions, but just a bit more consistent.
* Forgotten *.phtml
* Sort exclusion patterns + add a few for Extensions repo
* Relaxed some rules
* New feature: shareable user query
Share the output of a user query by RSS / HTML / OPML with other people through unique URLs.
Replaces the global admin token, which was the only option (but unsafe) to share RSS outputs with other people.
Also add a new HTML output for people without an RSS reader.
fix https://github.com/FreshRSS/FreshRSS/issues/3066#issuecomment-648977890
fix https://github.com/FreshRSS/FreshRSS/issues/3178#issuecomment-769435504
* Remove unused method
* Fix token saving
* Implement HTML view
* Update i18n for master token
* Revert i18n get_favorite
* Fix missing i18n for user queries from before this PR
* Remove irrelevant tests
* Add link to RSS version
* Fix getGet
* Fix getState
* Fix getSearch
* Alternative getSearch
* Default getOrder
* Explicit default state
* Fix test
* Add OPML sharing
* Remove many redundant SQL queries from original implementation of user queries
* Fix article tags
* Use default user settings
* Prepare public search
* Fixes
* Allow user search on article tags
* Implement user search
* Revert filter bug
* Revert wrong SQL left outer join change
* Implement checkboxes
* Safe check of OPML
* Fix label
* Remove RSS button to favour new sharing method
That sharing button was using a global admin token
* First version of HTTP 304
* Disallow some recusrivity
fix https://github.com/FreshRSS/FreshRSS/issues/6086
* Draft of nav
* Minor httpConditional
* Add support for offset for pagination
* Fix offset pagination
* Fix explicit order ASC
* Add documentation
* Help links i18n
* Note about deprecated master token
* Typo
* Doc about format
* Pass PHPStan level 8
And prepare for PHPStan level 9 https://phpstan.org/user-guide/rule-levels
* Revert wrong replace in comment
* Fix PHPStan level 8
* Update PHPStan and other dev dependencies
* Remove obsolete comment
* noVariableVariables and towards bleedingEdge
https://github.com/phpstan/phpstan-strict-ruleshttps://phpstan.org/blog/what-is-bleeding-edge
* More bleedingEdge
* A bit more PHPStan level 9
* More PHPStan level 9
* Prepare for booleansInConditions
Ignore int and null
* Revert wrong line
* More fixes
* Fix keep_max_n_unread
* Stricter attribute functions
* Stricter callHooks and more PHPStan level 9
* More typing
* A tiny more
