* Fix non-ASCII characters in scripts of fetched HTML pages
To add a <base> to a fetched HTML page, enforceHtmlBase() re-serialises it
with libxml2 saveHTML(). For a whole document, libxml2 2.9 only knows the
charset from <meta http-equiv="Content-Type">: with <meta charset="UTF-8">
or a byte order mark, it writes every non-ASCII character as an HTML entity.
In text that is harmless, but inside <script> entities are not decoded, so
e.g. the JSON read by HTML + XPath + JSON feeds gets "Wall Street’s"
instead of "Wall Street’s", which then shows in titles.
UTF-8 documents are now serialised node by node, which writes UTF-8.
Other charsets keep the previous output.
* Clarify comment about the byte order mark
* Remove comment about the byte order mark
* Minor test syntax preference
---------
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
* Fix NAT64 feed fetching on IPv6-only hosts
Closes#9371
Regression from https://github.com/FreshRSS/FreshRSS/pull/9195
Changes proposed in this pull request:
* Fix feed fetching on IPv6-only hosts using NAT64.
* Preserve SSRF protection by validating the IPv4 address embedded in RFC 6052 NAT64 addresses before allowing it in `CURLOPT_RESOLVE`.
* Add regression tests for both public and private/reserved NAT64 addresses.
* Minor formating
* Add test tearDown
* RFC8215 64:ff9b:1::/48
Credits: @onurcangnc
---------
Co-authored-by: Viktor Platov <TapuGitHub@users.noreply.github.com>
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
* Fix domain-wide Retry-After
- Fix the domain-wide HTTP `Retry-After` for public servers, which has not applied since #8195 (FreshRSS 1.28.0).
- `getRetryAfterFile()` passed the bare host name to `Minz_Request::serverIsPublic()`. That function expects a URL, so it always returned `false`.
- So a 429 or 503 paused only the URL that got it, despite the log saying "For that domain, will first retry after …", and FreshRSS went on requesting the other feeds of that server. With several Reddit feeds, for instance, each one got its own 429 in turn.
- `serverIsPublic()` now gets the URL. Its answer is kept per host for the rest of the request, because it resolves the host name (`dns_get_record()`), and a refresh may load many feeds of the same host.
- Fix the per-URL `Retry-After` of servers on a local network (#8195), which never applied to feeds whose URL, as stored, differs from the URL as requested.
- Stored URLs are HTML-encoded, e.g. `&` in an RSS-Bridge URL, and may end with `#force_feed`.
- `FreshRSS_Feed::load()` checked the stored URL, while the pause is stored for the URL as requested. It now checks the requested URL, as `loadJson()` and `loadHtmlXpath()` already do.
* Minor final
---------
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
TRUSTED_PROXY=192.168.1.1 is ignored and only 192.168.1.1/32 is honoured. checkCIDR() splits the range on the slash and rejects the value when the mask part is missing, so an address written without a subnet never matches.
An address without a subnet is now read as a single host: /32 for IPv4 and /128 for IPv6. That is the format docs/en/admins/09_AccessControl.md points at.
Host names such as gateway.localdomain are still not resolved.
tests/app/Utils/httpUtilTest.php gains 16 cases for checkCIDR, two of which fail without this change.
Refs #9210
* WebSub: ignore http/https scheme difference in Self URL comparison
The PubSubHubbub endpoint logged 'Self URL does not match registered
canonical URL' whenever a feed's self URL differed from the registered
canonical URL only by http vs https (common with http->https redirects),
spamming the log. Compare the URLs ignoring the http/https scheme while
still requiring host/path/query to match exactly.
Closes#3087
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Stylistic preferences
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
