Commit Graph

158 Commits

Author SHA1 Message Date
Alexandre Alapetite
412b60ca83 Fix transaction rollbacks during auto-update (#4622)
#fix https://github.com/FreshRSS/FreshRSS/issues/4600
We first need to commit the SQL transaction before being able to auto-update.
Avoid error such as:
> 25P02: 7 ERROR:  current transaction is aborted, commands ignored until end of transaction block
2022-09-19 11:51:44 +02:00
Alexandre Alapetite
03f5a42640 Fix MariaDB case sensitivity (#4537)
* Fix MariaDB case sensitivity
#fix https://github.com/FreshRSS/FreshRSS/issues/4535
Collation regression from https://github.com/FreshRSS/FreshRSS/pull/4505

* faster

* Minor syntax

* Minor parentheses
2022-08-25 23:16:24 +02:00
Alexandre Alapetite
4f111c5b30 Easier full-text search possibility (#4505)
* Easier full-text search possibility
Contributes to https://github.com/FreshRSS/FreshRSS/issues/1331
Avoid concats in searches to make text indexes easier to build

* Fix tests

* Documentation
2022-08-18 12:06:31 +02:00
Alexandre Alapetite
e27eb1ca91 Basic support for negative searches with parentheses (#4503)
* Basic support for negative searches with parentheses
* `!((author:Alice intitle:hello) OR (author:Bob intitle:world))`
* `(author:Alice intitle:hello) !(author:Bob intitle:world)`
* `!(S:1 OR S:2)`

* Minor documentation / comment

* Remove syslog debug line
2022-08-16 10:56:07 +02:00
Alexandre Alapetite
82ac1d1e67 Refactor entry-to-GReader API format (#4490)
* Refactor entry to GReader API format
Some code was copied in two locations and not completely uniform.
Cleaning of related variables and functions (e.g. better types for entries and categories as objects vs. as IDs).
Usecase: I need to call the same GReader-compatible serialization from an extension

* Fixed some edge cases

* Keep summary instead of content
`summary` and `content` seems to be used interchangeably in the Google Reader API. We have been using `summary` for our client API and  `content` in our export/import, so stick to that.
2022-08-08 12:04:02 +02:00
Alexandre Alapetite
0866fdaee8 Add database field attributes for entries (#4444)
* Add database field attributes for entries
Just like we already have for categories, feeds, etc.
No core use yet, but allows in particular extensions to save per-entry data

* Fix PHPStand

* Fix wrong variable
2022-07-19 15:17:08 +02:00
Alexandre Alapetite
07a52137a9 Fix last update & archive logic (#4422)
#fix https://github.com/FreshRSS/FreshRSS/issues/4401
2022-06-25 11:11:23 +02:00
Alexandre Alapetite
f85c510ed4 New search engine (#4378)
* New possibility to invoke user queries from a search expression
From the search field: `S:"My query"`.
Can be combined with other filters such as `S:"My query" date:P3d` as long as the user queries do not contain `OR`.
A use-case is to have an RSS filter with a stable address or an external API call with the ability to update the user query.

* Draft of parenthesis logic

* More draft

* Working parenthesis (a OR b) (c OR d)

* Working (A) OR (B)

* Support nested parentheses + unit tests + documentation

* search:MySearch and S:3
2022-06-02 08:41:08 +02:00
Alexandre Alapetite
1fe66ad020 Implement Web scraping "HTML + XPath" (#4220)
* More PHP type hints for Fever
Follow-up of https://github.com/FreshRSS/FreshRSS/pull/4201
Related to https://github.com/FreshRSS/FreshRSS/issues/4200

* Detail

* Draft

* Progress

* More draft

* Fix thumbnail PHP type hint
https://github.com/FreshRSS/FreshRSS/issues/4215

* More types

* A bit more

* Refactor FreshRSS_Entry::fromArray

* Progress

* Starts to work

* Categories

* Fonctional

* Layout update

* Fix relative URLs

* Cache system

* Forgotten files

* Remove a debug line

* Automatic form validation of XPath expressions

* data-leave-validation

* Fix reload action

* Simpler examples

* Fix column type for PostgreSQL

* Enforce HTTP encoding

* Readme

* Fix get full content

* target="_blank"

* gitignore

* htmlspecialchars_utf8

* Implement HTML <base>
And fix/revert `xml:base` support in SimplePie e49c578817

* SimplePie upstream PR merged
https://github.com/simplepie/simplepie/pull/723
2022-02-28 20:22:43 +01:00
Alexandre Alapetite
7c2da31418 More PHP type hints for Fever (#4202)
* More PHP type hints for Fever
Follow-up of https://github.com/FreshRSS/FreshRSS/pull/4201
Related to https://github.com/FreshRSS/FreshRSS/issues/4200
2022-02-07 00:56:45 +01:00
Alexandre Alapetite
1c5cf71859 Fix Fever 32 bit ID issue + more PHP type hints (#4201)
* Fix Fever 32 bit ID issue + more PHP type hints
#fix https://github.com/FreshRSS/FreshRSS/issues/4200
Follow up and fix regression from https://github.com/FreshRSS/FreshRSS/pull/4110

* More PHP type hints with PHPStan

* Fix pull problem

* Avoid more nulls
2022-02-06 14:31:36 +01:00
Thomas Renes
916df412f5 Fix various typos and spelling errors in documentation, comments and code. (#4134) 2022-01-08 16:25:17 +01:00
Frans de Jonge
d339b6dd45 [CI] PHPCS: check for opening brace on same line (#4122)
* [CI] PHPCS: check for opening brace on same line

* make fix-all

* Minor comments

Co-authored-by: Alexandre Alapetite <alexandre@alapetite.fr>
2022-01-05 00:52:24 +01:00
Alexandre Alapetite
1335a0e3cf PHPStan level 5 (#4110)
* Fix most PHPDocs errors
Contributes to https://github.com/FreshRSS/FreshRSS/issues/4103
https://phpstan.org/writing-php-code/phpdoc-types

* Avoid func_get_args
Use variadic syntax instead https://php.net/manual/functions.arguments#functions.variable-arg-list
And avoid dynamic functions names when possible to more easily identify calls and unused functions.
Contributes to https://github.com/FreshRSS/FreshRSS/issues/4103

* PHPStan level 3

* PHPStand level 4

* Update default to PHPStan level 4

* Towards level 5

* Fix level 4 regression

* Towards level 5

* Pass PHPStan level 5

* Towards level 6

* Remove erronenous regression from changelog
https://github.com/FreshRSS/FreshRSS/pull/4116
2022-01-04 13:59:09 +01:00
Alexandre Alapetite
3f6aa42b81 Fix most PHPDocs errors (#4107)
Contributes to https://github.com/FreshRSS/FreshRSS/issues/4103
https://phpstan.org/writing-php-code/phpdoc-types
2022-01-01 11:26:56 +01:00
Alexandre Alapetite
77e9877316 Add PHPStan (#4021)
* Add PHPStan
#fix https://github.com/FreshRSS/FreshRSS/issues/4016
https://phpstan.org/

```sh
composer run-script phpstan
```

* More fixes

* Fix global variables

* Add .phtml

* Fix merge
https://github.com/FreshRSS/FreshRSS/pull/4090

* Fix more warnings

* Fix view errors and enable in CI

* ReturnTypeWillChange

* Dynamic view type

* Fix Minz static/self bug
2021-12-31 17:00:51 +01:00
Alexandre Alapetite
fb15a2d804 Search on article IDs (#4058)
* Search on article IDs
Partial implementation of https://github.com/FreshRSS/FreshRSS/issues/4053
2021-12-31 13:10:41 +01:00
Alexandre Alapetite
705be9a6a1 Search labels (#3709)
* Search labels
#fix https://github.com/FreshRSS/FreshRSS/issues/3704

* Documentation

* Allow list without quotes

* Allow boolean AND searches

* Allow searching any label

* fix labels alias
2021-07-24 19:32:43 +02:00
Alexandre Alapetite
8d8b792007 Fix array_chunk key index bug (#3631)
* Fix array_chunk key index bug
#fix https://github.com/FreshRSS/FreshRSS/issues/3630
We still want array entries indexed sequentially from zero https://php.net/function.array-chunk

* One missing
2021-05-18 22:50:51 +02:00
Alexandre Alapetite
f2557bed9a More cases of max SQL variable number (#3586)
Follow up of https://github.com/FreshRSS/FreshRSS/pull/3553
2021-05-08 00:56:06 +02:00
Alexandre Alapetite
de40f3ad56 Fix TT-RSS import (#3553) 2021-03-24 19:04:51 +01:00
Alexandre Alapetite
0206fc1e5e fix updateEntryPrepared (#3500) 2021-03-07 19:41:17 +01:00
Alexandre Alapetite
b10cd770ee SQLite: fix updateEntry (#3461)
* Draft of fix for updateEntry

#fix https://github.com/FreshRSS/FreshRSS/issues/3130

* Fix for SQLite

* Update app/Models/EntryDAO.php

Fixed in https://github.com/FreshRSS/FreshRSS/pull/3500

* Unbuffered streaming only for MySQL
2021-03-07 17:34:47 +01:00
Alexandre Alapetite
947e918f05 Travis: Enforce phpcs line length + whitespace (#3488)
* Update Travis line length

* Also check whitespace in CSS files

* Fix line length ext.php

* More syntax, string templates

* Fix exclude-pattern

* Test JS files as well
2021-02-28 12:26:24 +01:00
Alexandre Alapetite
1ee1fcce91 INSERT ... ON CONFLIT DO NOTHING for addEntry (#3409)
#fix https://github.com/FreshRSS/FreshRSS/issues/3402
Explicit `INSERT OR IGNORE` / `ON CONFLICT DO NOTHING` for the
`addEntry()` method, which does expect some duplicates.
2021-02-02 00:03:59 +01:00
Craig Andrews
f9687ea18a Fix commitNewEntries query (#3357)
There was an erroneous '; at the end of the commitNewEntries DELETE query, resulting in an exception during actualization with this output:

```
FreshRSS starting feeds actualization at 2021-01-11T14:06:09+00:00
FreshRSS actualize candrews...
PHP Fatal error:  Uncaught PDOException: SQLSTATE[42000]: Syntax error or access violation: 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''' at line 1 in /var/www/localhost/htdocs/FreshRSS/lib/Minz/Pdo.php:45
Stack trace:
#0 /var/www/localhost/htdocs/FreshRSS/lib/Minz/Pdo.php(45): PDO->exec()
#1 /var/www/localhost/htdocs/FreshRSS/app/Models/EntryDAO.php(167): Minz_Pdo->exec()
#2 /var/www/localhost/htdocs/FreshRSS/app/Controllers/feedController.php(505): FreshRSS_EntryDAO->commitNewEntries()
#3 /var/www/localhost/htdocs/FreshRSS/app/Controllers/feedController.php(548): FreshRSS_feed_Controller::actualizeFeed()
#4 /var/www/localhost/htdocs/FreshRSS/lib/Minz/Dispatcher.php(119): FreshRSS_feed_Controller->actualizeAction()
#5 /var/www/localhost/htdocs/FreshRSS/lib/Minz/Dispatcher.php(47): Minz_Dispatcher->launchAction()
#6 /var/www/localhost/htdocs/FreshRSS/lib/Minz/FrontController.php(81): Minz_Dispatcher->run()
#7 /var/www/localhost/htdocs/FreshRSS/app/actualize_script.php(75): Minz_FrontController->run()
#8 {main}
  thrown in /var/www/localhost/htdocs/FreshRSS/lib/Minz/Pdo.php on line 45
Results:
```
2021-01-11 22:28:35 +01:00
Alexandre Alapetite
08a5cf5946 Ensure maximum date int (#3259)
#fix https://github.com/FreshRSS/FreshRSS/issues/3258
2020-11-23 23:04:06 +01:00
Alexandre Alapetite
6edbeaaf6a Add error log (#3055)
For https://github.com/FreshRSS/FreshRSS/issues/3054
2020-06-13 21:19:19 +02:00
Alexandre Alapetite
b1aac20839 Fever integer type for numbers (#2946)
* Fever integer type for feed ID

#Fix
https://github.com/FreshRSS/FreshRSS/issues/2940#issuecomment-623022435

* Fix feed_id

* Ensure string for entry ID

To be compatible with 32-bit platforms.
Before, the type was inconsistent depending on architecture + database

* Integer for entry count functions
2020-05-03 13:39:16 +02:00
Alexandre Alapetite
ae70374b03 Filter by multiple feed IDs (#2892)
Add the possibility to filter by feed ID like `f:123 more-search`
or multiple feed IDs, like `f:123,234,345 more-search` or an exclusion
like `!f:456,789 more-search`
2020-04-17 10:57:35 +02:00
Alexandre Alapetite
61c8026ac9 Implement negation for searching by date intervals (#2869)
* Implement negation for searching by date intervals

#fix https://github.com/FreshRSS/FreshRSS/issues/2866
Allow searching for e.g. `!date:P1W` to exlude all articles newer than
one week.
More generally, allows exclusion on some date intervals.

* Fix OR
2020-04-01 17:26:19 +02:00
Alexandre Alapetite
0f94402b7e Better performance with yield (#2588)
* Better performance with yield

Largely decrease the time to first byte, and reduced memory consumtion.
Before, we used to make several copies in memory of the whole list of
articles before sending them to the client. Now streamed as they are
processed.

* Travis
2020-02-29 18:19:09 +01:00
Alexandre Alapetite
cd5ebcee05 Fix SQLite CONCAT used for search (#2777)
Used e.g. for searches `inurl:xxx`

Co-authored-by: Marien Fressinaud <dev@marienfressinaud.fr>
2020-01-20 11:42:44 +01:00
Alexandre Alapetite
bec7566429 Sanitize UTF-8 before commiting entries (#2649)
Fix https://github.com/FreshRSS/FreshRSS/issues/2645
2019-11-11 19:48:45 +01:00
Alexandre Alapetite
fdfd8ce9be Workaround for MySQL limitation 1093 (#2586)
* Workaround for MySQL limitation 1093
https://github.com/FreshRSS/FreshRSS/pull/2335#issuecomment-545194606
including minor Travis fixes
2019-10-23 02:12:49 +02:00
Alexis Degrugillier
cc0db9af4f Feature/new archiving (#2335)
* Change archiving config page layout

I've changed some wording and moved actions into a
maintenance section.

* Update purge action

Now we have more control on the purge action. The configuration allows
us to choose what to keep and what to discard in a more precise way.
At the moment, the configuration applies for all feeds.

* Add purge configuration on feed level

Now the extend purge configuration is available on feed level.
It is stored as attributes and will be used in the purge action.

* Update purge action

Now the purge action uses the feed configuration if it exists and
defaults on user configuration if not.

* Add empty option in period list

* Fix configuration warnings

* Add archiving configuration on categories

See #2369

* Add user info back

* Add explanations in UI

* Fixes for SQLite + error + misc.

* Fix invalid feed reference

* Short array syntax

Only for new code, so far

* Fix prefix error

* Query performance, default values

Work in progress

* Fix default values and confirm before leaving

Form cancel and confirm changes before leaving were broken.
And start taking advantage of the short echo syntax `<?= ?>` as we have
moved to PHP 5.4+

* More work

* Tuning SQL

* Fix MariaDB + performance issue

* SQL performance

* Fix SQLite bug

* Fix some attributes JSON encoding bugs

Especially for SQLite export/import

* More uniform, fix bugs

More uniform between global, category, feed settings

* Drop special cases for old articles during refresh

Instead will use lastSeen date with the new archiving logic.
This was generating problems anyway
https://github.com/FreshRSS/FreshRSS/issues/2154

* Draft drop index keep_history

Not needed anymore

* MySQL typo

Now properly tested with MySQL, PostgreSQL, SQLite

* More work for legacy values

Important to avoid overriding user's preference and risking deleting
data erroneously

* Fix PHP 7.3 / 7.4 warnings

@aledeg "Trying to use values of type null, bool, int, float or resource
as an array (such as $null["key"]) will now generate a notice. "
https://php.net/migration74.incompatible

* Reintroduce min articles and take care of legacy parameters

* A few changes forgotten

* Draft of migration + DROP of feed.keep_history

* Fix several errors

And give up using const for SQL to allow multiple database types (and we
cannot redefine a const)

* Add keep_min to categories + factorise archiving logic

* Legacy fix

* Fix bug yield from

* Minor: Use JSON_UNESCAPED_SLASHE for attributes

And make more uniform

* Fix sign and missing variable

* Fine tune the logic
2019-10-23 00:52:15 +02:00
Alexandre Alapetite
3b65f2e586 Fix PostgreSQL and SQLite DB size estimation (#2562) 2019-10-06 22:17:28 +02:00
Alexandre Alapetite
be4c942cb3 SQL allow recreating existing user (#2555)
* SQL allow recreating existing user

Taking advantage of https://github.com/FreshRSS/FreshRSS/pull/2554
In a case when FreshRSS data is lost, but database data still intact (in
particular MySQL or PostgreSQL), this patch allows recreating previous
users without error

* Better error retrieval

Especially when error occur during the prepare statement
2019-10-06 00:14:19 +02:00
Alexandre Alapetite
38932ee3bb MySQL entry content MEDIUMBLOB (#2551)
Fix https://github.com/FreshRSS/FreshRSS/issues/2448
2019-10-05 10:37:15 +02:00
Alexandre Alapetite
7ae4e9f159 Quick fix: forgotten PDO values (#2546)
From https://github.com/FreshRSS/FreshRSS/pull/2522
2019-09-29 23:11:35 +02:00
Alexandre Alapetite
e3e5954394 PDO refactoring for code simplification (#2522)
* PDO refactor

* Automatic prefix when using the syntax `_tableName`
* Uniformity: MySQL is now PDO::ATTR_EMULATE_PREPARES = false just like SQLite and PostgreSQL, with consequences such as only one statement per query
* Use PDO methods exec(), query(), prepare() + execute() in a more efficient way
* Remove auto-update SQL code for versions older than FreshRSS 1.5 (3 years old)
* The name of the default category is set in PHP instead of in the DB (simplies SQL and allows changing the name according to the FreshRSS language)
* Rename `->bd` to `->pdo` (less of a frenshism, and more informative)
* Fix some requests, which were not compatible with MySQL prepared statements

* Whitespace

* Fix syntax for PostgreSQL sequences

+ MySQL install

* Minor formatting

* Fix lastInsertId for PostgreSQL

* Use PHP 5.6+ const

Take advantage of https://github.com/FreshRSS/FreshRSS/pull/2527
https://www.php.net/manual/en/migration56.new-features.php

* A bit of forgotten PHP 5.6 simplification for cURL

* Forgotten $s

* Mini fix custom user config

https://github.com/FreshRSS/FreshRSS/pull/2490/files#r326290346

* More work on install.php but not finished

* install.php working

* More cleaning of PDO in install

* Even more simplification

Take advantage of PDO->exec() to run multiple statements

* Disallow changing the name of the default category

https://github.com/FreshRSS/FreshRSS/pull/2522#discussion_r326967724
2019-09-29 16:22:50 +02:00
Alexandre Alapetite
c76a318193 CLI to export/import any database to/from SQLite (#2496)
* CLI to export/import any database to/from SQLite

Require PHP 5.5+ https://github.com/FreshRSS/FreshRSS/pull/2495

* Travis

* Execution rights

* Fix wrong static fields

* Fix MySQL bad default buffering

https://stackoverflow.com/questions/6895098/pdo-mysql-memory-consumption-with-large-result-set/6935271#6935271
https://php.net/manual/ref.pdo-mysql

* Fix count on progression

* Avoid static DB information

To ease working with two DBs at the same time

* Less static, simplify

Needs some testing

* Small corrections

* Special case for SQLite to SQLite

* Modify special case for SQLite

* Remove special case for SQLite

More uniform logic for the 3 databases.
Fix wrong DROP TABLE for SQLite.

* Drop indexes

* Revert "Drop indexes"

This reverts commit f28d2bae09.

* Fix deletion

* Fix classic export

* Update cli/README.md

Co-Authored-By: Marien Fressinaud <dev@marienfressinaud.fr>

* Addressing part of review

* Remove goto 😢

* Travis

* Comment for SQLite case

* Fix missing fields when inserting
2019-09-15 21:36:53 +02:00
Alexandre Alapetite
fd33d92d41 Require PHP 5.5+ (#2495)
* Require PHP 5.5+

https://github.com/FreshRSS/FreshRSS/issues/2469#issuecomment-522255093
I think it would be reasonable to require PHP 5.5+ for the core of
FreshRSS after all.

As Frenzie said, WordPress currently requires PHP 5.6.20+, and it is the
most popular PHP application.

We would loose about 20% of the PHP servers according to
https://w3techs.com/technologies/details/pl-php/5/all but I expect this
number to drop fast after the release of CentOS 8 (CentOS accounts for
17% of Linux servers
https://w3techs.com/technologies/details/os-linux/all/all ).

Distributions:
* no impact on Ubuntu, Fedora, Alpine, OpenWRT, FreeBSD, OpenSuze,
Mageia, as all active versions have PHP > 7
* no impact on OpenSuze, Synology, as all active versions have PHP > 5.5
* we drop Debian 8 Jessie (-2020) - we keep supporting Debian 9 Stretch
(2017-06) - current is Debian 10 Buster
* we drop Red Hat 7 (-2024) - we keep supporting RHEL 8 (2019-05)
* we drop CentOS 7 (-2024) - we will support CentOS 8 (to be released
soonish)

When dropping older versions, I can better like when it is for a good
reason, and there is actually one with PHP 5.5, namely generators
(yield) https://php.net/language.generators.overview which I consider
using.

* Version note for JSON.php

* hex2bin

* Update .travis.yml

Co-Authored-By: Frans de Jonge <fransdejonge@gmail.com>
2019-08-20 14:55:43 +02:00
Alexandre Alapetite
49013bf716 Cherry pick from 2335 (#2356)
Cherry pick bug fix from https://github.com/FreshRSS/FreshRSS/pull/2335
This code is currently not used.
2019-04-07 14:01:05 +02:00
Alexandre Alapetite
9cc72c0c48 Fix EntryDAO tags warning for Fever API
https://github.com/FreshRSS/FreshRSS/issues/2239
2019-02-03 21:24:06 +01:00
Alexandre Alapetite
34fe41ac78 Merge branch 'FreshRSS/dev' into FreshRSS/dev-1.14.0 2019-01-17 21:36:29 +01:00
Alexandre Alapetite
743c1b740b Export labels (#2217)
* Export labels

https://github.com/FreshRSS/FreshRSS/issues/2196

* Small fixes

* Backport code from 1.14.0

4888f919f1

* More fixes
2019-01-16 22:26:43 +01:00
Alexandre Alapetite
4888f919f1 Prepare for batch mark as read 2018-12-25 01:30:28 +01:00
Alexandre Alapetite
b672fc190d Tweaks for Vienna RSS (#2093)
* Tweaks for Vienna RSS

https://github.com/FreshRSS/FreshRSS/issues/2091
https://github.com/ViennaRSS/vienna-rss/issues/1197

* Fix get feed by URL

* Fix get item ids returning starred elements

* API add item ids by feed URL

* Add API filter `it`

https://feedhq.readthedocs.io/en/latest/api/reference.html#stream-items-ids

* API add `nt=` filter + refactoring

* No ; prefix for author

https://github.com/FreshRSS/FreshRSS/issues/2091#issuecomment-435562495

* Add id long form prefix and accept short id form

https://github.com/FreshRSS/FreshRSS/issues/2091#issuecomment-435631259

* Fix quote problem

https://github.com/FreshRSS/FreshRSS/issues/2091#issuecomment-435683930

* Isolate bug fix for News+

https://github.com/FreshRSS/FreshRSS/issues/2091#issuecomment-435687041

* Rework encoding conventions

https://github.com/FreshRSS/FreshRSS/issues/2091#issuecomment-437441834

* Unicode escaping alternative

Alternative approach to encode XML special characters and other
problematic characters into their Unicode fullwidth version when we
cannot use HTML-encoding because clients disagree wether they should
HTML-decode or not.
https://github.com/FreshRSS/FreshRSS/issues/2091#issuecomment-436059559
2018-11-11 17:31:50 +01:00
Alexandre Alapetite
cf899d8d25 TT-RSS import (#2099)
* TT-RSS import

Import of Tiny Tiny RSS favourites
https://github.com/FreshRSS/FreshRSS/issues/2018#issuecomment-432710462

* Fallback feed_url

* Simpler JSON

* TT-RSS import custom labels

* Fix syntax
2018-11-05 18:10:38 +01:00