convertJsonToRss() HTML-escapes item fields before rendering the RSS,
but the is_string() guard skipped 'tags', which is always an array.
An unescaped & in a JSON category then produced invalid XML, so
SimplePie rejected the whole document and the feed showed as empty.
Minz_Helper::htmlspecialchars_utf8() already handles arrays,
so the guard is dropped. Add a test for array and string categories.
fix https://github.com/FreshRSS/FreshRSS/issues/9488
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* Preserve explicit sort overrides when searching
Closes#9466
Changes proposed in this pull request:
- Keep an explicit sort override when submitting the header search form, including `c.name`, `f.name` and `lastUserModified`.
- Continue using the already validated `FreshRSS_Context::$sort` value; omit the hidden field when no non-empty override was requested.
- Add header-rendering regression tests for all nine supported sorts, absent/empty overrides and invalid input.
* A bit more fixing
* Fix line mistake
* Forgotten plaintext
* Fix test
---------
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
* Sanitizer: block data: URI scheme, keep data:image for images
FreshRSS_SimplePieCustom only disallowed the `javascript` scheme, so
`data:text/html,...` URIs in feed content survived
sanitization on <a href>, <iframe src>, <audio|video|source|track src>.
The default CSP neutralises these on FreshRSS pages, but clients and
deployments without that CSP (API WebViews, extensions using amendCsp(),
proxies stripping headers) could execute them. Same class as
GHSA-3fw6-j8m4-82vj / #9272.
- Sanitize::is_allowed_uri(): element/attribute-aware check that still
permits inert `data:image/...` on img[src], image[src], video[poster]
while blocking `data:` elsewhere; replace_urls() now uses it.
is_allowed_scheme() is unchanged for backward compatibility.
- SimplePieCustom: disallow_uri_schemes(['javascript','data']).
- Tests: 4 blocked-input cases, a data:image keep-alive test and an
`unsafe:` prefix test. They fail without the patch and pass with it.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* Sync SimplePie
https://github.com/FreshRSS/simplepie/pull/92
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
Contributes to #6139 (export part; the import will follow in another PR)
Exporting favourites was collecting the whole file in memory. With many articles, it fails with the default PHP memory limit.
## Changes
- New `Minz_View::helperToFile()`, similar to `helperToString()`, writing the output of a helper into a file by chunks of 64 KiB instead of keeping it in memory.
- `FreshRSS_Export_Service`: the `generate*()` methods now return the path of a temporary file instead of the content, and `zip()` builds the archive from those files (`ZipArchive::addFile()` instead of `addFromString()`). Temporary files are created in `TMP_PATH` (already required to be writable) and deleted at the end of the script.
- The export is sent with `readfile()`: Web UI, `cli/export-zip-for-user.php`, `cli/export-opml-for-user.php` and the OPML export of the Google Reader API.
- Note: this changes what `FreshRSS_Export_Service` returns (a path instead of the content). All callers in the repository are updated, and I could not find any public extension using this service.
## Measurements
Peak PHP memory (`memory_get_peak_usage(true)`), SQLite database with synthetic starred articles:
| Scenario | Before | After |
|---|---|---|
| Web, export your favourites, 10,000 articles (69 MB JSON) | 136 MB (fails with the default `memory_limit` of 128 MB) | 6 MB |
| Web, ZIP (OPML + favourites + labelled articles + 4 feeds), 10,000 articles | 152 MB | 6 MB |
| CLI `export-zip-for-user.php`, 10,000 articles | 189 MB (fails with 128 MB) | 8 MB |
| Web, export your favourites, 30,000 articles (208 MB JSON) | 404 MB | 10 MB |
Co-authored-by: Alejandro Borrego <210245044+abp002@users.noreply.github.com>
* Fix NAT64 feed fetching on IPv6-only hosts
Closes#9371
Regression from https://github.com/FreshRSS/FreshRSS/pull/9195
Changes proposed in this pull request:
* Fix feed fetching on IPv6-only hosts using NAT64.
* Preserve SSRF protection by validating the IPv4 address embedded in RFC 6052 NAT64 addresses before allowing it in `CURLOPT_RESOLVE`.
* Add regression tests for both public and private/reserved NAT64 addresses.
* Minor formating
* Add test tearDown
* RFC8215 64:ff9b:1::/48
Credits: @onurcangnc
---------
Co-authored-by: Viktor Platov <TapuGitHub@users.noreply.github.com>
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
* Fix domain-wide Retry-After
- Fix the domain-wide HTTP `Retry-After` for public servers, which has not applied since #8195 (FreshRSS 1.28.0).
- `getRetryAfterFile()` passed the bare host name to `Minz_Request::serverIsPublic()`. That function expects a URL, so it always returned `false`.
- So a 429 or 503 paused only the URL that got it, despite the log saying "For that domain, will first retry after …", and FreshRSS went on requesting the other feeds of that server. With several Reddit feeds, for instance, each one got its own 429 in turn.
- `serverIsPublic()` now gets the URL. Its answer is kept per host for the rest of the request, because it resolves the host name (`dns_get_record()`), and a refresh may load many feeds of the same host.
- Fix the per-URL `Retry-After` of servers on a local network (#8195), which never applied to feeds whose URL, as stored, differs from the URL as requested.
- Stored URLs are HTML-encoded, e.g. `&` in an RSS-Bridge URL, and may end with `#force_feed`.
- `FreshRSS_Feed::load()` checked the stored URL, while the pause is stored for the URL as requested. It now checks the requested URL, as `loadJson()` and `loadHtmlXpath()` already do.
* Minor final
---------
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
* Fix muting of feeds gone with HTTP 410
A feed answering HTTP 410 Gone is meant to be muted (#3561), but:
* Since #6037, FreshRSS_Feed_Exception extends Minz_Exception, which
resets any code other than its own error levels to 0, so the check for
410 never matched. FreshRSS_Feed_Exception now keeps the HTTP status
code it is given.
* FreshRSS_FeedDAO::mute() mutes by negating the TTL, which leaves a feed
with the default TTL (0) unmuted. Mute the feed the way the feed
settings and dynamic OPML categories do instead.
* A bit of cleaning
---------
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
* Slightly more robust SessionTest and cleanup
Improvement of #9311
I noticed that my `/tmp` directory got filled up with random `sess_*` files and empty `frss_test_sessions_*` directories after repeatedly running PHPUnit, which means there was no proper cleanup being done in the tests.
See the first commit for changes, since I also moved the test to a more appropriate directory name (alongside `tests/lib/Minz`, `tests/app/Minz` was newly added in #9311).
Before the PR:
* Check output of `ls -lah /tmp/{frss_test_sessions_*,sess_*}; rm -r /tmp/{frss_test_sessions_*,sess_*}` to confirm there are initially no temporary files existing.
* Run `composer run phpunit`.
* Check output of the previous command again to confirm there is a new `sess_*` file and an empty `frss_test_sessions_*` directory.
After the PR:
* Check output of `ls -lah /tmp/{frss_test_sessions_*,sess_*}; rm -r /tmp/{frss_test_sessions_*,sess_*}` to confirm that no temporary files exist initially.
* Run `composer run phpunit`.
* Check output of the previous command again to confirm there are no temporary files, since they are now being cleaned up correctly.
* Move SessionTest from tests/app/Minz to tests/lib/Minz
TRUSTED_PROXY=192.168.1.1 is ignored and only 192.168.1.1/32 is honoured. checkCIDR() splits the range on the slash and rejects the value when the mask part is missing, so an address written without a subnet never matches.
An address without a subnet is now read as a single host: /32 for IPv4 and /128 for IPv6. That is the format docs/en/admins/09_AccessControl.md points at.
Host names such as gateway.localdomain are still not resolved.
tests/app/Utils/httpUtilTest.php gains 16 cases for checkCIDR, two of which fail without this change.
Refs #9210
Closes#9288
Changes proposed in this pull request:
The mark-as-read links contain the resolved state, so the automatic reading preference becomes an explicit `state=2` filter after the redirect. Once the last unread article is marked read, the page stays empty instead of showing all articles.
- Keep the requested state separately from the resolved state used by the action. A redirect state of `0` reapplies the reading preference; explicit filters still survive the redirect as intended by #9007.
- Carry that value through the menu, footer and “Load more” link. This matters with oldest-first pagination, where the last footer supplies the menu button's action.
- Add request-state and rendered-pagination tests, plus a changelog entry.
How to test the feature manually:
1. With both read and unread articles, choose “Show unreads if any, all articles otherwise”. Mark all as read from the footer, then repeat using the top button. The resulting page should show the read articles.
2. Set one article per page and oldest-first sorting. Load the remaining articles before marking all as read; the same fallback should work.
3. Select an explicit unread filter: marking everything read should still leave an empty unread view. Also check a favourite filter with a search term, including “Mark selection unread”; the search and favourite scope should remain intact.
Validation: `make NO_DOCKER=1 fix-all` and `make NO_DOCKER=1 test-all` pass with PHP 8.5.9 (719 PHPUnit tests, 1,378 assertions). Browser checks passed against SQLite in Chromium (desktop and mobile viewport) and Firefox. The original footer fails three of the six new rendered-pagination cases.
* Limit the length and parentheses nesting depth of a search query.
* Use exception
---------
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
* Only allow http/https URI schemes for enclosure and thumbnail URLs
Feed enclosure URLs were stored and rendered without URI scheme validation:
SimplePie's scheme check only applies to URLs inside content HTML
(Sanitize::replace_urls()), not to enclosure metadata, so a `javascript:`
URL survived the whole pipeline and was served as a clickable link in the
article view, including through the Google Reader compatible API.
Add FreshRSS_http_Util::isAllowedUrlScheme() and apply it:
- at store time, when building enclosure attributes in FreshRSS_Feed
- at render time, in FreshRSS_Entry::content(), so entries already stored
in existing databases are protected as well
https://github.com/FreshRSS/FreshRSS/security/advisories/GHSA-3fw6-j8m4-82vj
* Reuse existing SimplePie is_remote_uri() function
* Use more realistic test examples
* Add tests for protocol relative
And simplify tests further
---------
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
* Fix: infinite recursion in sanitizeHTML() when maxLength truncation doesn't converge
Sanitizing can grow a truncated HTML fragment (e.g. an unclosed tag gets
auto-closed), so the previous recursive shrink-and-retry could hit a fixed
point and recurse forever, crashing on a stack overflow. Bound the retries
and fall back to a hard truncation that always terminates.
Add unit tests for FreshRSS_SimplePieCustom::sanitizeHTML(), the XSS
sanitization applied to all untrusted feed content, which had no coverage.
* Update app/Models/SimplePieCustom.php
Co-authored-by: Frans de Jonge <fransdejonge@gmail.com>
* Some refactoring
Avoid two mb_strcut in a row. More cases for fallback. Better fallback. More edge cases.
* Address trailing incomplete tag or entity
* Whitespace reduction
---------
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
Co-authored-by: Frans de Jonge <fransdejonge@gmail.com>
The loop building the media credits assigned instead of appending, so an enclosure with several credits only kept the last one.
`app/views/index/rss.phtml` builds the `<media:credit>` elements for an enclosure in a loop, but assigns instead of appends:
```php
$mediaCredits = '';
foreach ($credits as $credit) {
$mediaCredits = '<media:credit>' . $credit . '</media:credit>';
}
```
So an enclosure with three credits emits only the last one. Every iteration throws away what the previous one built.
This came in with #6272 ("Allow multiple authors on enclosures", which closed#5066). Parsing and display got multi-credit support, but FreshRSS's own generated RSS still passes on a single credit, so a downstream reader subscribing to a FreshRSS feed sees the same symptom #5066 described.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Fix “mark as read older than…” widening the active search
Sibling searches of a `FreshRSS_BooleanSearch` are combined by OR, so
`prepend()` added the maximum publication date used by the “mark as read →
articles older than one day / one week” action as one more OR term instead of
an extra restriction.
With an active search and the stream sorted by publication date, that action
therefore marked as read every article older than the cut-off (ignoring the
search) plus every article matching the search at any date.
`prepend()` now wraps the existing expression first, reusing the logic that
`enforce()` already used for the same purpose.
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix SQL space
* Slight reduction of comments
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
* Complete two source TODOs: negative UTC offset date parsing and PostgreSQL tag insert
_noDelimit() stripped '-' and ':' from the whole ISO 8601 string, including a
trailing UTC offset, so a negative offset such as "-05:00" silently lost its
sign and corrupted the parsed date. Extract a trailing offset (or "Z") before
stripping delimiters and re-append it unchanged.
FreshRSS_TagDAOPGSQL::sqlIgnore() returned an empty string as a placeholder,
so re-tagging an already-tagged entry produced a real SQL error on PostgreSQL
instead of being a silent no-op like on MySQL/SQLite. Replace the sqlIgnore()
keyword-injection approach with the sqlIgnoreConflict(string $sql): string
pattern already used by EntryDAO, and implement it for PostgreSQL with
"ON CONFLICT DO NOTHING".
Fixes#7923.
Co-Authored-By: Claude <noreply@anthropic.com>
* Drop the negative-UTC-offset date fix, already covered by #9071#9071 already fixes this exact bug in lib_date.php, so keeping this
PR focused on the PostgreSQL tag-insert fix only.
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Gerard Alvear <gerard.alvear@logiqd.me>
Co-authored-by: Claude <noreply@anthropic.com>
On MySQL and MariaDB (the compressed content_bin code path,
isCompressed() === true), three of the four regex-search constructions
in FreshRSS_EntryDAO::sqlBooleanSearch() generated invalid SQL, so the
query failed and the search silently returned zero articles:
* intext:/regex/ appended a stray closing parenthesis after
sqlRegex(...) (MySQL error 1248);
* -intext:/regex/ had the same stray parenthesis;
* -/regex/ emitted ' ANT NOT ' instead of ' AND NOT '
(MySQL error 1064).
sqlRegex() already returns a self-contained, balanced expression
(REGEXP_LIKE(expr, ?, 'flags') or expr REGEXP ?), so no extra
parenthesis is needed, matching the uncompressed (SQLite / PostgreSQL)
branches.
The existing MySQL / MariaDB unit tests for intext:/regex/ had
captured the buggy unbalanced SQL as their expected strings; they are
corrected, and new cases are added for -intext:/regex/ and -/regex/
on both flavours.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* WebSub: ignore http/https scheme difference in Self URL comparison
The PubSubHubbub endpoint logged 'Self URL does not match registered
canonical URL' whenever a feed's self URL differed from the registered
canonical URL only by http vs https (common with http->https redirects),
spamming the log. Compare the URLs ignoring the http/https scheme while
still requiring host/path/query to match exactly.
Closes#3087
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Stylistic preferences
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
When the search query includes some feed IDs or category IDs, adjust feed visibility/priority filter to include at minimum feed or category visibility.
Fix: https://github.com/FreshRSS/FreshRSS/issues/8602
* Fix user query parsing
Fix https://github.com/FreshRSS/FreshRSS/issues/8531
We used to take some shortcuts in the code, but now that the logic has complexified, we need to parse those user queries more properly.
* More fixes
* Implement filter on last modified date by server
Especially relevant for API, to get the modified changes: the API will now return the articles that are new or which content has been modified since `ot`:
fix https://github.com/FreshRSS/FreshRSS/issues/7304
fix https://github.com/FreshRSS/FreshRSS/issues/2566https://github.com/jocmp/capyreader/discussions/533#discussioncomment-11341808
New corresponding search operator `mdate:` and new UI:
<img width="650" height="627" alt="image" src="https://github.com/user-attachments/assets/8ba02937-abc7-44bf-b718-cf269cc37caf" />
* Migration from existing id column
* Fix auto-update
* Index after update for performance
* Minor comment
* Minor whitespace
* Fix regex
* Minor .gitignore
* Changelog and warning
* Update app/i18n/pl/gen.php
Co-authored-by: Inverle <inverle@proton.me>
* make fix-all
* Optimise SQL auto-update
For speed and resilience
* Minor SQLite change of sequence
* Changelog
* Speed optimisation: No DEFAULT 0
* Better migration
* Revert small bug
* Prepare filtering on multiple dates for API
* make fix-all
* Update tests
* Remaining manual merge
* Update versions
* Remove warnings no longer relevant in changelog
* Implement in API, and COALESCE
* No lastModified when adding new article
* Rework logic
* Sort IS NOT NULL
* Remove forgotten lastModified
---------
Co-authored-by: Inverle <inverle@proton.me>
The string syntax of `strtr()` cannot handle mutibytes characters, so need to be rewritten using an array map approach.
Extend the fallback replacements to include the Windows/ISO charsets of the latin languages for which we have a translation
Dates should not be expanded/resolved in string serialisations of search expressions, as it otherwise break relative user queries such as `P30D`.
Fix of https://github.com/FreshRSS/FreshRSS/pull/8293
* Allows easier modifications of the search expression.
* Add proper `__toString()` instead of just returning the raw input string. Allows in particular showing the result of the actual parsing of the raw input string in the UI.
Needed for https://github.com/FreshRSS/FreshRSS/pull/8294
Allow the use of regex `\b` for word boundary (and `\B` for the opposite) even when using PostgreSQL.
Follow up of:
* https://github.com/FreshRSS/FreshRSS/pull/6706
For instance, `intitle:/\bnew\B/` will find *newest* but not *new* nor *renewal*.
Useful in particular to minimise the differences between PHP and database in:
* https://github.com/FreshRSS/FreshRSS/pull/7959
Example: `userdate:PT1H` to select only articles modified by user during the last hour
Fix https://github.com/FreshRSS/FreshRSS/issues/4280#issuecomment-3393078024
Useful for instance to bulk mark as unread recently marked articles by error:
1. Click on the toggle button to show the read articles (making sure the toggle for the unread articles is off)
2. Sort by *User modified 9→1*
3. Filter by *user modified date*, for instance to the last 3 hours by typing `userdate:PT3H`
4. Click in the drop-down menu *Mark selection as unread*
P.S.: I have added at the same time a bunch of unit tests for date-related logic
* Add UI for advanced search
To help users with the seach operators.
Obviously not as powerful as a manually-written search query.
Lack in particular negation and logical *and* for now, but I might try to do something about it.
<img width="939" height="1438" alt="image" src="https://github.com/user-attachments/assets/0bcad39b-eff3-4f44-876b-a2552af2af00" />
* Consistency: allow multiple user queries like S:1,2
* Fix user query and add tests
1. `include`, `include_once`, `require` and `require_once` are expressions not functions, parentheses are not necessary.
2. to move up the directory tree, it's better to use the `dirname` function instead of relying on `/..`.
