Compare commits

...

336 Commits

Author SHA1 Message Date
Matthieu Gautier
9f19327023 Fix maybe initialized warning in httplib.
Patch has been send upstream :
https://github.com/yhirose/cpp-httplib/pull/1085
2021-11-05 17:12:33 +01:00
Matthieu Gautier
b2b907f813 Update httplib to last version (0.9.7)
Release url is https://github.com/yhirose/cpp-httplib/releases/tag/v0.9.7
2021-11-05 17:12:33 +01:00
Kelson
9479c0685d Merge pull request #623 from kiwix/update-ci
Re-introduce Ubuntu Impish in CI
2021-10-21 12:41:23 +02:00
Emmanuel Engelhart
09a55d71d6 Re-introduce Ubuntu Impish in CI 2021-10-21 12:36:05 +02:00
Matthieu Gautier
503eb5c4ce Merge pull request #621 from kiwix/fix_ci_docker_version 2021-10-19 11:49:20 +02:00
Matthieu Gautier
f714ff8d3e New docker image version is 31. 2021-10-18 18:09:36 +02:00
Matthieu Gautier
08e3d52957 Merge pull request #607 from kiwix/issue/571 2021-10-12 17:40:27 +02:00
Manan Jethwani
30e4c549e4 exposed fileExist, getMimeTypeForFile and getFileCoontent functions 2021-10-12 19:44:38 +05:30
Manan Jethwani
b7b385d87b added custom index template 2021-10-12 19:44:05 +05:30
Matthieu Gautier
e46b0c07b5 Merge pull request #617 from kiwix/adapt_new_libzim_api 2021-09-30 14:52:17 +02:00
Matthieu Gautier
cd9fb541fc Fix method call for new libzim API.
`add_archive` is now `addArchive`.
2021-09-29 11:55:22 +02:00
Matthieu Gautier
3b942bb745 Merge pull request #602 from kiwix/partial_opds_entries
OPDS feed with partial entries
2021-09-09 12:07:02 +02:00
Veloman Yunkan
c0bda426b4 Removed duplication across two mustache templates
Deduplicated the mustache templates static/templates/catalog_v2_entries.xml
and static/templates/catalog_v2_complete_entry.xml (the latter was
renamed to static/templates/catalog_v2_entry.xml).
2021-09-09 12:19:22 +04:00
Veloman Yunkan
b3f7556096 Added partial entries feed to the OPDS root feed 2021-09-09 12:19:22 +04:00
Veloman Yunkan
4c657c082e /catalog/v2/partial_entries OPDS API endpoint 2021-09-09 12:19:22 +04:00
Veloman Yunkan
e773a29f29 Rearranged elements in OPDS entry XML 2021-09-09 12:19:22 +04:00
Veloman Yunkan
e15a0f4338 /catalog/v2/entry/<entry_id> OPDS API endpoint 2021-09-09 12:19:22 +04:00
Veloman Yunkan
12d9b69806 OPDSDumper::dumpOPDSCompleteEntry() 2021-09-09 12:19:22 +04:00
Veloman Yunkan
027854e4f4 Extracted getSingleBookData() in opds_dumper.cpp 2021-09-09 12:19:22 +04:00
Matthieu Gautier
417e7471ac Merge pull request #614 from kiwix/disable-impish 2021-09-09 10:10:50 +02:00
Matthieu Gautier
51ac1240f8 PPA: Temporarily disable Impish builds 2021-09-09 10:05:18 +02:00
Kelson
ea6413ff88 Merge pull request #591 from kiwix/suggestion_range
Allow kiwix-serve to get suggestions of custom range
2021-08-20 08:09:35 +02:00
Maneesh P M
61209ea0d7 Allow kiwix-serve to get suggestions of custom range
This will allow handle_suggest API to accept two arguments `start` and
`suggestionLength` that will allow handle_suggest to retrieve
suggestions in the given range rather than the default 0-10 range.
2021-08-19 21:05:39 +05:30
Kelson
e9eaadde9e Merge pull request #567 from kiwix/suggestion_api_fix 2021-08-14 19:21:29 +02:00
Maneesh P M
8a4080baba Update libkiwix with new libzim api 2021-08-14 22:26:39 +05:30
Kelson
ba05999cba Merge pull request #604 from kiwix/issue/603 2021-08-11 06:50:30 +02:00
Manan Jethwani
a4c3cad018 fixed books availablity on larger screens and added zoom level support 2021-08-10 21:45:10 +05:30
Kelson
83e757a530 Merge pull request #600 from kiwix/dynamic_select_box_value
Use OPDS API to populate categories/languages select boxes on Kiwix Serve welcome page
2021-08-07 15:33:52 +02:00
Manan Jethwani
5e8f3a5505 added use of lang and category api for select boxes on welcome page 2021-08-07 02:39:50 +05:30
Manan Jethwani
fe93035a4c updated welcome page to support OPDS multiple Icon 2021-08-07 02:35:29 +05:30
Kelson
6e26c5aa75 Merge pull request #577 from kiwix/opds_multiple_icons
Support for multiple illustrations in OPDS entry
2021-08-05 23:22:07 +02:00
Veloman Yunkan
452283cfe6 Handling of /meta?name=Illustration_WxH@1 requests 2021-08-05 22:28:09 +04:00
Veloman Yunkan
e5168d8b3d Support for multiple illustrations in OPDS entry 2021-08-05 22:21:13 +04:00
Matthieu Gautier
b8aee8a42c Merge pull request #597 from kiwix/fix_get_results 2021-08-04 15:57:58 +02:00
Maneesh P M
9addd82d2d Fix usage of zim::Searcher::getResults() in libkiwix
The correct usage does not require the user to calculate an `end` using
the `pageLength`. We can directly use getResults(start, pageLength)
2021-08-04 19:20:50 +05:30
Maneesh P M
e74e7f5623 Add unit test for incremental searching
With this, we eventually want to see the usage of getResults giving
a FAILING TEST. This happens because the second argument to
getResults is NOT `end` of the range, but `maxResultCount` to retrieve.
This will be fixed in the next commit.
2021-08-04 19:20:05 +05:30
Matthieu Gautier
a032d65eb8 Merge pull request #576 from kiwix/extend_libkiwix_structures_to_use_libzim 2021-08-03 11:50:24 +02:00
Maneesh P M
19afe9442f Remove OriginId functions since they are not useful right now 2021-08-03 11:42:58 +02:00
Maneesh P M
a3ba7619df Update Manager to use Archive instead of Reader
kiwix::Manager uses Reader to import a zim file, it should be using
zim::Archive directly.
2021-08-03 11:42:58 +02:00
Maneesh P M
8b12434ff2 Update kiwix::book to use libzim structure
Some methods in kiwix::Book uses wrapper structure reader. This usage should
be extended from the native libzim structure zim::Archive
2021-08-03 11:42:58 +02:00
Matthieu Gautier
b4f7dfa5a2 Merge pull request #553 from kiwix/catalog_languages_endpoint 2021-08-03 11:41:31 +02:00
Veloman Yunkan
ab3095745e Languages OPDS feed includes book counts 2021-08-03 11:32:38 +02:00
Veloman Yunkan
45adda44b3 Got rid of <content> node in languages OPDS entry 2021-08-03 11:32:38 +02:00
Veloman Yunkan
96cf7e78a5 OPDSDumper::categoriesOPDSFeed() with no args 2021-08-03 11:32:38 +02:00
Veloman Yunkan
dd118df612 Got rid of langMap in opds_dumper.cpp
Language code to human friendly name translation is now done with the
help of the ICU library. It works if the line

```
-include $(LANGSRCDIR)/resfiles.mk
```

in the file `source/data/Makefile.in` of the icu4c dependency is not
commented out. Currently, the said line is commented out (along with
some other include's) by the `icu4c_custom_data.patch` patch of the
`kiwix-build` tool.
2021-08-03 11:32:38 +02:00
Veloman Yunkan
8a4248e48e Language code in /catalog/v2/languages entries 2021-08-03 11:32:38 +02:00
Veloman Yunkan
5f90f5ee2a Preliminary version of /catalog/v2/languages 2021-08-03 11:32:38 +02:00
Veloman Yunkan
64b55dbdc7 Made test library.xml a multi-language library 2021-08-03 11:32:38 +02:00
Veloman Yunkan
18871b4b15 Helper function Library::getBookPropValueSet()
Introduced a helper function `Library::getBookPropValueSet()` and
deduplicated Library::getBooks{Languages,Creators,Publishers}() methods.
2021-08-03 11:32:38 +02:00
Veloman Yunkan
b2027b397c List of languages entry in /catalog/v2/root.xml
Added a new entry in /catalog/v2/root.xml that points to a
not-yet-existing list of languages navigation feed.
2021-08-03 11:32:38 +02:00
Kelson
49322f5961 Merge pull request #596 from kiwix/better_filter
improved browser lang filter working
2021-07-31 23:24:14 +02:00
Manan Jethwani
0466b9759c improved browser lang filter working 2021-07-30 12:57:59 +05:30
Kelson
20cdefcdb8 Merge pull request #593 from kiwix/remove_groovy_package
Remove groovy deb package
2021-07-28 21:40:02 +02:00
Emmanuel Engelhart
6ea40f57da Remove groovy deb package 2021-07-28 21:34:24 +02:00
Kelson
a312d2218d Merge pull request #590 from kiwix/root_prefix_addition
corrected relative links in preview and icon url
2021-07-25 08:59:24 +02:00
Manan Jethwani
15839df594 corrected relative links in preview and icon url 2021-07-24 19:26:22 +05:30
Kelson
03a929e88e Merge pull request #583 from kiwix/download-modal
Modal download  box on Kiwix Serve welcome page
2021-07-13 16:47:58 +02:00
Manan Jethwani
646502f9cf changed font style for modal 2021-07-13 20:00:43 +05:30
Manan Jethwani
a8a96a99f4 corrected working of magnet link 2021-07-13 00:23:38 +05:30
Manan Jethwani
a517d3b529 added modal for downloading zim file on welcome page 2021-07-12 17:59:26 +05:30
Kelson
60f0f81286 Merge pull request #559 from kiwix/Css_revamp
Revamped Kiwix Serve Welcome page layout
2021-07-08 12:36:19 +02:00
Manan Jethwani
2ed9a50eca fixed button allignment 2021-07-08 12:33:28 +02:00
Manan Jethwani
bce922ab89 bug fix for loader 2021-07-08 12:33:28 +02:00
Manan Jethwani
ad7a63a471 minor change in UI 2021-07-08 12:33:28 +02:00
Manan Jethwani
6e8200637e corrected search button in mobile view 2021-07-08 12:33:28 +02:00
Manan Jethwani
cc45c840d1 fixed minor codefactor issue 2021-07-08 12:33:28 +02:00
Manan Jethwani
0590f27fa1 corrected select box and search bar design 2021-07-08 12:33:28 +02:00
Manan Jethwani
dd27c3a873 changed tile background color 2021-07-08 12:33:28 +02:00
Manan Jethwani
736841818d fixed font and other minor issues in title cards 2021-07-08 12:33:28 +02:00
Manan Jethwani
c1868e22f4 minor codefactor fix 2021-07-08 12:33:28 +02:00
Manan Jethwani
aabfc1d82e fixed card design 2021-07-08 12:33:28 +02:00
Manan Jethwani
2effb3490e minoor changes in responsive behaviour 2021-07-08 12:33:28 +02:00
Manan Jethwani
55672b0288 revamped basic layout and cards 2021-07-08 12:33:28 +02:00
Kelson
0abbeabfe2 Merge pull request #568 from kiwix/ppa-impish
PPA: Build for Ubuntu Impish
2021-07-08 10:24:52 +02:00
Kunal Mehta
1bf52e8ebe PPA: Build for Ubuntu Impish 2021-07-08 09:50:55 +02:00
Kelson
e2db1b3688 Merge pull request #574 from kiwix/remove_mustache_public_header
Fix public headers inclusion (+ small other fixes)
2021-07-07 18:00:17 +02:00
Matthieu Gautier
0b6b6716de Rename split argument from trimEmpty to dropEmpty. 2021-07-07 14:43:13 +02:00
Matthieu Gautier
18b6433322 Correct method declaration in SuggestionItem 2021-07-07 14:43:13 +02:00
Matthieu Gautier
b70c92cade Move back used helper functions to the public API.
- Add docstring
- Move the declaration in kiwix namespace.
- Adapt our include to include the right headers.
2021-07-07 14:43:13 +02:00
Matthieu Gautier
09d843da3a Add a (empty) include/tools.h header.
This header will contain our public tool functions.
2021-07-07 14:43:13 +02:00
Matthieu Gautier
fa83a61a54 Move all public *Tools.h in src.
This by definition remove all the tool functions from the public API.
2021-07-07 14:43:13 +02:00
Matthieu Gautier
967eb10cbf Merge pull request #578 from kiwix/fix_ci_deps
Use correct deps archive in the CI.
2021-07-07 10:59:53 +02:00
Matthieu Gautier
feeee25eac Use correct deps archive in the CI.
Now that project is named libkiwix, the dependencies archive is also
renamed.
2021-07-07 10:53:07 +02:00
Matthieu Gautier
1c0b4502cd Merge pull request #536 from kiwix/internally_drop_reader_searcher 2021-07-06 16:18:10 +02:00
Maneesh P M
6f639144ab Add unit tests for Searcher and Reader
Even though we will be removing the wrappers soon, the test coverage
should be complete and we could simply remove these files later.
2021-07-03 14:07:14 +05:30
Maneesh P M
a94a03cd22 Remove unwanted reader functions
Removing the functions in InternalServer that are no longer needed.
2021-07-03 14:07:14 +05:30
Maneesh P M
bc821638da Drop wrapper structures from handle_search
Since we now have SearcherRenderer that can work with native libzim
structure, we will drop the wrapper and use them instead.
2021-07-03 14:07:12 +05:30
Maneesh P M
bcece66960 Add SearchRenderer handles for libzim structures
Introduces a new member mp_search that houses the zim::Search object,
adds a new constructor for this purpose. This commit also add an
overload for getHtml that takes start and end integers as arguments
since they are not part of the search object we include.
2021-07-03 14:05:50 +05:30
Maneesh P M
c046f64d83 Drop Reader and Entry wrappers from handle_content 2021-07-03 14:05:50 +05:30
Maneesh P M
75b4d311d7 Drop Reader from InternalServer::handle_random 2021-07-03 14:04:04 +05:30
Maneesh P M
a236751c74 Drop usage of Reader from InternalServer::handle_suggest 2021-07-03 14:04:04 +05:30
Maneesh P M
7d68926539 Drop usage of Reader from InternalServer::handle_meta
This is essentially a code move of meta handlers from using Reader
functions to directly using Archive.
2021-07-03 14:04:02 +05:30
Maneesh P M
940368b8ac Add m_archives and getArchiveById to Library
These members will mirror the functionality offered by equivalent usage
of Reader class.
2021-07-03 14:02:31 +05:30
Kelson
0594e60df3 Merge pull request #527 from kiwix/catalog_search_url_generation
OpdsCatalog::getSearchUrl()
2021-06-30 21:35:43 +02:00
Veloman Yunkan
b5c1b26761 OpdsCatalog::getSearchUrl() 2021-06-30 18:27:00 +02:00
Kelson
4124ad30d5 Merge pull request #561 from kiwix/issue/557
Keep Kiwix Serve filter values over time
2021-06-25 08:05:21 +02:00
Manan Jethwani
3c5d73027d created separate variable for time delta 2021-06-24 15:41:28 +05:30
Manan Jethwani
d88bdd3ebf corrected filter on no results 2021-06-23 14:15:22 +05:30
Manan Jethwani
5cfe34a5c2 corrected filter working 2021-06-22 19:36:22 +05:30
Manan Jethwani
ad133bc9a3 added cookies for filter effect 2021-06-21 19:57:52 +05:30
Kelson
8eabae6286 Merge pull request #560 from kiwix/fix_compilation_illustration 2021-06-19 13:15:44 +02:00
Maneesh P M
f3c96b23fd Use getIllustrationItem instead of getFaviconEntry method
With openzim/libzim#540 we now have a new function to get
illustration(previously favicon in 48x48 size and unity scale) in
multiple sizes. We need to replace getFaviconEntry with this new
getIllustrationItem method.
2021-06-19 10:23:24 +05:30
Kelson
a92e9d8756 Merge pull request #490 from soumyankar/404ContentHomeButton
404 content home button
2021-06-17 09:59:45 +02:00
Vertigo
8d39b2c4c1 Added content ZIM home button on 404 2021-06-17 12:51:27 +05:30
Kelson
6d237ff1d5 Merge pull request #492 from kiwix/opds_categories_feed 2021-06-08 20:01:37 +02:00
Veloman Yunkan
78083f1f4a Moved OPDS templates under static/templates 2021-06-08 20:37:00 +04:00
Veloman Yunkan
dd60235010 Fixed the self link in the output of /catalog/v2/entries 2021-06-08 20:37:00 +04:00
Veloman Yunkan
e799f2ff1e OPDSDumper::dumpOPDSFeed() works via mustache
This changes the output of `/catalog/search` as follows:

- Entire search query (rather than only the value of the `q` parameter)
  is put in the <title> node.

- Search performed with an empty query presents itself as "All zims".

- The feed id remains stable for identical searches on the same
  library.
2021-06-08 20:37:00 +04:00
Veloman Yunkan
312f2cb560 Moved handle_catalog_v2*() methods into a new file 2021-06-08 20:37:00 +04:00
Veloman Yunkan
fa42cbc48f Pagination info in /catalog/v2/entries 2021-06-08 20:37:00 +04:00
Veloman Yunkan
f1797993af Reused InternalServer::search_catalog() 2021-06-08 20:37:00 +04:00
Veloman Yunkan
f886c8c07b Root url is normalized once in the constructor 2021-06-08 20:37:00 +04:00
Veloman Yunkan
9ca6bd006f /catalog/v2/categories goes through OPDSDumper too 2021-06-08 20:37:00 +04:00
Veloman Yunkan
cdacc0caf1 /catalog/v2/entries going through OPDSDumper
OPDSDumper sensed threats to its job security, so it lobbied to be
involved in handling the /catalog/v2 endpoints, too.
2021-06-08 20:37:00 +04:00
Veloman Yunkan
dfad1c3815 /catalog/v2/searchdescription.xml 2021-06-08 20:37:00 +04:00
Veloman Yunkan
07252a127a /catalog/v2/entries is also a search endpoint 2021-06-08 20:37:00 +04:00
Veloman Yunkan
b60e3ffb26 RequestContext::get_optional_param() 2021-06-08 20:37:00 +04:00
Veloman Yunkan
70d42aec98 A small simplification 2021-06-08 20:37:00 +04:00
Veloman Yunkan
4aa3c792aa Extracted get_search_filter() 2021-06-08 20:37:00 +04:00
Veloman Yunkan
208dece7e3 Reordered several statements
Reordered several statements so that the next couple of commits are a
little simpler.
2021-06-08 20:37:00 +04:00
Veloman Yunkan
19b59fd72f Serving /catalog/v2/entries
/catalog/v2/entries is intended to play the combined role of
/catalog/root.xml and /catalog/search of the old OPDS API. Currently,
the latter role is not yet implemented.

Implementation note: instead of tweaking and reusing
`OPDSDumper::dumpOPDSFeed()`, the generation of the OPDS feed is done via `mustache`
and a new template `static/catalog_v2_entries.xml`.
2021-06-08 20:37:00 +04:00
Veloman Yunkan
92c2de8d46 Enter InternalServer::m_library_id
The new field is intended to serve as a seed for generating semi-stable
OPDS feed ids that only need to change when the library is updated.
2021-06-08 20:37:00 +04:00
Veloman Yunkan
feeb9f206e /catalog/v2/* XMLs are OPDS 1.2 2021-06-08 20:37:00 +04:00
Veloman Yunkan
a1520ce7f1 Fixing the xenial build
Under Ubuntu 16.04/xenial, ccache seems to have issues with multiline
raw string literals used inside macros.
2021-06-08 20:37:00 +04:00
Veloman Yunkan
2e53b51696 Serving /catalog/v2/categories 2021-06-08 20:37:00 +04:00
Veloman Yunkan
b259afa408 Library::getBooksCategories()
Note: no unit test added
2021-06-08 20:37:00 +04:00
Veloman Yunkan
3c3cf08a1a Serving /catalog/v2/root.xml
Note: This commit somewhat relaxes validation of non variable
`<updated>` elements in the OPDS feed - the contents of any `<updated>`
element is replaced with the YYYY-MM-DDThh:mm:ssZ placeholder.
2021-06-08 16:03:43 +04:00
Veloman Yunkan
54b78eaf56 Moved gen_date_str() to tools/otherTools.cpp 2021-06-08 16:03:43 +04:00
Veloman Yunkan
1e0ff1fbb0 Fixed the double colon in OPDS date string 2021-06-08 16:03:43 +04:00
Veloman Yunkan
5b272ac49c Fixed handling of /catalogBLABLA/root.xml & alike
Also removed an unneeded namespace qualifier.
2021-06-08 16:03:43 +04:00
Veloman Yunkan
0a3d293ae0 Broke Server.404 with /catalogBLABLABLA/root.xml
The new negative test-point demonstrates that Kiwix server doesn't
distinguish /catalogBLABLABLA from /catalog.
2021-06-08 16:03:43 +04:00
Kelson
86ef2e2199 Merge pull request #550 from kiwix/remove-bintray
Remove Bintray badge
2021-06-07 16:01:37 +02:00
Emmanuel Engelhart
a0332e7599 Remove Bintray badge 2021-06-07 15:55:19 +02:00
Kelson
2ef488816c Merge pull request #534 from kiwix/filter_library
Add filters to kiwix-serve welcome page
2021-06-07 15:46:37 +02:00
Manan Jethwani
1ccafe2d97 minor changes in fadeout effect 2021-06-07 15:38:31 +02:00
Manan Jethwani
d6c62b3cd3 corrected spinner and fadeout effect 2021-06-07 15:37:20 +02:00
Manan Jethwani
f39c558d2a added fade out 2021-06-07 15:37:20 +02:00
Manan Jethwani
5b46ad5934 added spinned 2021-06-07 15:37:20 +02:00
Manan Jethwani
49dbd0aa52 fixed reset filters link 2021-06-07 15:37:20 +02:00
Manan Jethwani
179f0faeb1 added minor features 2021-06-07 15:37:20 +02:00
Manan Jethwani
bb92f26b60 added filter functionality 2021-06-07 15:37:20 +02:00
Matthieu Gautier
3a4e8303a0 Merge pull request #541 from kiwix/adding_dynamic_and_subset_loading
Dynamic and subset loading of catalogue in kiwix-serve
2021-06-07 15:35:13 +02:00
Manan Jethwani
063bb8cd65 added dynamic and subset loading of zim-files in kiwix-serve 2021-06-01 19:33:42 +05:30
Kelson
b54e5ab969 Merge pull request #543 from kiwix/add-libmicrohttpd-compilation-hint
Add libmicrohttpd compilation hint
2021-05-30 15:51:40 +02:00
Emmanuel Engelhart
2632a21d24 Move Repology to wiki 2021-05-30 15:46:45 +02:00
Emmanuel Engelhart
5c97b1fff9 gtest is need for testing 2021-05-30 15:43:21 +02:00
Emmanuel Engelhart
4f7175ad59 Libkiwix, not Kiwix library 2021-05-30 15:42:28 +02:00
Emmanuel Engelhart
f4b8d0c303 Add libmicrohttpd compilation hint 2021-05-30 15:35:40 +02:00
Matthieu Gautier
188694f2a1 Merge pull request #510 from kiwix/add_function_zimId 2021-05-26 15:15:13 +02:00
Maneesh P M
e2f6d91d51 Remove get_readerIndex in favor of get_zimId
The function get_readerIndex was used to get the zimId using an ordered
vector of readers. Now we can use get_zimId directly.
2021-05-26 14:45:25 +02:00
Maneesh P M
c35f6f9142 Add get_zimId method to Result
get_zimId method allows the user to get the uuid of the archive from
which a result is retrieved directly from the search result itself.
2021-05-26 14:45:25 +02:00
Matthieu Gautier
7f0d3004c9 Merge pull request #505 from kiwix/suggestion_snippets 2021-05-26 11:04:52 +02:00
Maneesh P M
5567d8ca49 Replace std::vector<std::string> with SuggestionItem
Each sugestions used to be stored as vector of strings to hold various values
such as title, path etc inside them. With this commit, we use the new
dedicated class `SuggestionItem` to do the same.
2021-05-26 10:53:39 +02:00
Maneesh P M
5315034afe Introduce SuggestionItem class
This is a helper class that allows to create and manage individual
suggestion item and their data.
2021-05-26 10:53:00 +02:00
Maneesh P M
3288cd80e5 Render suggestion snippet properly
To render the snippets properly, we need to use the _renderItem property
of the autocomple ui.
2021-05-26 10:53:00 +02:00
Maneesh P M
56434de79e Set label to title snippet if present
With openzim/libzim#545 we now support snippet generation of titles
which can be used as the display label on the ui for highlighted titles
via the "label" field.
The old version used plain title which is still available in the value
field.
2021-05-26 10:52:58 +02:00
Matthieu Gautier
e9ba151e6f Merge pull request #539 from kiwix/fix_windows_build
Avoid windows header to define min/max macros.
2021-05-26 09:52:43 +02:00
Matthieu Gautier
5f83944699 Avoid windows header to define min/max macros.
PR #507 use std::min.
But on windows, the header define min and max macros and so the
compilation is broken.
Add `-DNOMINMAX` define to avoid that.
2021-05-26 09:20:17 +02:00
Kelson
9c0ae835e2 Merge pull request #537 from kiwix/search_iterator_api_rename
Update libkiwix with search iterator rename in libzim
2021-05-26 08:49:26 +02:00
Maneesh P M
e5fac30cee Update libkiwix with search iterator rename in libzim
Search iterator API in libzim has been shifted to use camel case naming.
This has to be accomodated in libkiwix as well.
2021-05-26 08:39:13 +02:00
Matthieu Gautier
7ef08b670b Merge pull request #538 from kiwix/revert-530-adding_dynamic_and_subset_loading
Revert "Kiwix Serve welcome page dynamic and subset loading (OPDS based)"
2021-05-25 17:33:23 +02:00
Matthieu Gautier
2736a46cfe Revert "Kiwix Serve welcome page dynamic and subset loading (OPDS based)" 2021-05-25 17:30:05 +02:00
Kelson
672b4fc907 Merge pull request #530 from kiwix/adding_dynamic_and_subset_loading
Kiwix Serve welcome page dynamic and subset loading (OPDS based)
2021-05-25 16:22:54 +02:00
Manan Jethwani
012973d14a added dynamic and subset loading of zim-files in kiwix-serve 2021-05-25 02:41:12 +05:30
Kelson
67984cca5b Merge pull request #535 from kiwix/further-kiwix-lib-renaming
Rename kiwix-lib in libkiwix
2021-05-23 21:57:03 +02:00
Emmanuel Engelhart
d4e35c7067 Rename kiwix-lib in libkiwix 2021-05-23 21:46:52 +02:00
Matthieu Gautier
6e37cabaea Merge pull request #529 from kiwix/libkiwix-github-url-fix
Fix Libkiwix Github repository URLS
2021-05-21 16:09:15 +02:00
Emmanuel Engelhart
c8b7f8772a Fix Libkiwix Github repository URLS 2021-05-20 08:56:44 +02:00
Veloman Yunkan
fc7484ac86 Merge pull request #528 from kiwix/fix_suggestion_search_result_page
Check if bookName is available in url parameters
2021-05-19 00:36:15 +04:00
Maneesh P M
c236f3a32b Check if bookName is available in url parameters
In certain pages like the search result page, bookName is not of the
form `/bookName/endpoint?parameters`. Rather it is available as a query
parameter. From these pages bookName should be assigned from parameters.
2021-05-19 01:12:29 +05:30
Matthieu Gautier
3c7faddb6e Merge pull request #526 from kiwix/lizim_search_api_change
Fixed the libkiwix build broken by the changed libzim search API
2021-05-17 15:06:15 +02:00
Veloman Yunkan
cd02b4de3b Dummy application of new libzim search API
Didn't take any advantage of the new libzim search API. Just fixed the
libkiwix build in the most straightforward way.
2021-05-15 23:34:51 +04:00
Kelson
5188355878 Merge pull request #525 from kiwix/html-only-root-link
Insert root link only if html content
2021-05-14 16:51:33 +02:00
Emmanuel Engelhart
05cc3d015f Insert root link only if html content 2021-05-14 14:49:28 +02:00
Kelson
39b62c6108 Merge pull request #522 from kiwix/libkiwix_ci_fix
Fixed CI after repository rename
2021-05-13 11:09:57 +02:00
Veloman Yunkan
9c43353b72 Fixed CI after repository rename 2021-05-13 12:49:44 +04:00
Matthieu Gautier
b82fff9855 Merge pull request #507 from kiwix/fix_for_issue_504
/catalog/search handles out-of-bounds pagination
2021-05-10 11:47:49 +02:00
Veloman Yunkan
68189de162 /catalog/search handles out-of-bounds pagination 2021-05-10 11:25:06 +02:00
Matthieu Gautier
6aab9b6981 Merge pull request #506 from kiwix/library_filtering_by_empty_query
Empty query acts as a match-all query
2021-05-10 10:27:37 +02:00
Veloman Yunkan
41276341d0 Empty query acts as a match-all query
After switching to Xapian-based search in the library/catalog, an empty
query stopped acting as a match-all query. This commit restores the old
behaviour in that regard.
2021-05-09 15:14:43 +02:00
Kelson
02c3dff142 Merge pull request #508 from kiwix/remove_204_status_code
Revert "added 204 code for empty return of search"
2021-05-09 07:49:43 +02:00
Maneesh P M
be6b58c6ad Revert "added 204 code for empty return of search"
Returning status code 204 in case of an empty results doesn't show the
empty results page as described in #466. Reverting the changes in #396
fixes the issue.
2021-05-09 10:47:18 +05:30
Matthieu Gautier
ab0ffb55bc Merge pull request #502 from kiwix/no-meta4-file
No metalink file on fs
2021-05-04 13:52:54 +02:00
Emmanuel Engelhart
950e742116 No metalink file on fs 2021-05-04 13:15:43 +02:00
Matthieu Gautier
69257610e8 Merge pull request #495 from kiwix/replace_buggy_zim
Replace buggy example.zim with a correct one
2021-04-28 16:28:37 +02:00
maneeshpm
700d4becb9 Replace buggy example.zim with a correct one
The existing example.zim has an improper main page entry hence an
unusable index as reported in openzim/libzim#521
To replace the buggy zim, a new zim has been generated using the latest
zimwriterfs with latest libzim.

-------------------------------------------------------------------------
The directory used to create zim is given below and are two pages from
wikibooks site:

htmlContent
├── favicon.png
├── FreedomBox for Communities_Offline Wikipedia - Wikibooks, open books for an open world_files
│   ├── index.php
│   ├── load.php
│   ├── poweredby_mediawiki_88x31.png
│   └── wikimedia-button.png
├── FreedomBox for Communities_Offline Wikipedia - Wikibooks, open books for an open world.html
├── Wikibooks_files
│   ├── 234px-Megakaryocyte1.svg.png
│   ├── 287px-ChewyGingerCookies.jpg
│   ├── 36px-Commons-logo.svg.png
│   ├── 40px-Wikiquote-logo.svg.png
│   ├── 41px-Wikispecies-logo.svg.png
│   ├── 46px-Wikisource-logo.svg.png
│   ├── 48px-MediaWiki-2020-icon.svg.png
│   ├── 48px-Phacility_phabricator_logo.png
│   ├── 48px-Wikimedia_Cloud_Services_logo.svg.png
│   ├── 48px-Wikimedia_Community_Logo.svg.png
│   ├── 48px-Wikivoyage-Logo-v3-icon.svg.png
│   ├── 51px-Wiktionary-logo.svg.png
│   ├── 53px-Wikipedia-logo-v2.svg.png
│   ├── 59px-Wikiversity_logo_2017.svg.png
│   ├── 86px-Wikidata-logo.svg.png
│   ├── 88px-Wikinews-logo.svg.png
│   ├── Haskell-logo.png
│   ├── index.php
│   ├── load.php
│   ├── poweredby_mediawiki_88x31.png
│   └── wikimedia-button.png
└── Wikibooks.html

The command for writing the zim:
$ zimwriterfs --welcome=Wikibooks.html --favicon=favicon.png --language=en --title=Wikibooks --description=testZim --creator=test --publisher=test --verbose ./htmlContent  ./out/example.zim
2021-04-28 15:24:50 +02:00
Matthieu Gautier
dd795bd56d Merge pull request #498 from kiwix/issue/446
fixed suggestion system
2021-04-28 14:40:06 +02:00
Manan Jethwani
5fdc51b23e fixed suggestion system 2021-04-28 14:34:24 +02:00
Matthieu Gautier
32cc6b0dcb Merge pull request #499 from kiwix/const_correct_library
const-correct kiwix::Library
2021-04-28 14:31:28 +02:00
Veloman Yunkan
3879b82112 const-correct kiwix::Library
- Made most methods of kiwix::Library const.
- Also added const versions of getBookById() and getBookByPath()
  methods.
2021-04-28 11:42:55 +04:00
Matthieu Gautier
7336dcab1d Merge pull request #488 from kiwix/fully_xapian_powered_catalog_search 2021-04-27 15:10:40 +02:00
Veloman Yunkan
63e9a09259 Cleaned up/beautified Library::updateBookDB() 2021-04-27 16:59:21 +04:00
Veloman Yunkan
4178c169dd Xapian documents in book DB store only the book id 2021-04-27 16:59:21 +04:00
Veloman Yunkan
59e9a0cd77 Merged XmlLibraryTest with LibraryTest
The library set up by LibraryTest now contains two valid books
initialized via XML. Therefore XmlLibraryTest is not needed as a
separate test suite.
2021-04-27 16:59:21 +04:00
Veloman Yunkan
f751aff2fb Full case/diacritics insensitivity in catalog filtering
Catalog filtering should now be case/diacritics insensitive for all
fields. However it is not validated for language, name and category
fields, and is validated for tags, creator & publisher only for text
supplied in the filter (but not for values read from the book).
2021-04-27 16:59:21 +04:00
Veloman Yunkan
87dc9d2723 Made catalog filtering by query diacritics insensitive
Catalog filtering by titles/description was sensitive to diacritics
present in the query string. Fixed that.

Also enhanced the unit test to validate the insensitivity to diacritics
present in either the title/description or the query string.
2021-04-27 16:59:21 +04:00
Veloman Yunkan
9c7366890d Catalog filtering by tags works via Xapian 2021-04-27 16:59:21 +04:00
Veloman Yunkan
19e195cb7d Filter::Tags typedef 2021-04-27 16:59:21 +04:00
Veloman Yunkan
3d5fd8f585 Catalog filtering by creator works via Xapian 2021-04-27 16:59:21 +04:00
Veloman Yunkan
d3d5abe14d Handling of non-words in publisher query
This change fixes the failure of the LibraryTest.filterByPublisher
unit-test broken by the previous commit.

The previous approach used in `publisherQuery()` for building a phrase
query enforcing the specified prefix for all terms fails if

1. the input phrase contains a non-word term that Xapian's query parser
   doesn't like (e.g. a standalone ampersand character, 1/2, a#1, etc);
2. the input phrase contains at least three terms that Xapian's query
   parser has no issue with.

Using the `quest` tool (coming with xapian-tools under Ubuntu) the
issue can be demonstrated as follows:

```
$ quest -o phrase -d some_xapian_db "Energy & security"
Parsed Query: Query((energy@1 PHRASE 11 Zsecur@2))
Exactly 0 matches
MSet:

$ quest -o phrase -d some_xapian_db "Energy & security act"
UnimplementedError: OP_NEAR and OP_PHRASE only currently support leaf subqueries

$ quest -o phrase -d some_xapian_db 'Energy 1/2 security act'
UnimplementedError: OP_NEAR and OP_PHRASE only currently support leaf subqueries

$ quest -o phrase -d some_xapian_db "Energy a#1 security act"
UnimplementedError: OP_NEAR and OP_PHRASE only currently support leaf subqueries
```

The problem comes from parsing the query with the default operation set
to `OP_PHRASE` (exemplified by the `-o phrase` option in above
invocations of `quest`). A workaround is to parse the phrase with a
default operation of `OP_OR` and then combine all the terms with
`OP_PHRASE`.

Besides stemming should be disabled in order to target an exact phrase
match (save for the non-word terms, if any, that are ignored by the
query parser).
2021-04-27 16:59:21 +04:00
Veloman Yunkan
e805f68994 Enhanced & broke LibraryTest.filterByPublisher 2021-04-27 16:59:21 +04:00
Veloman Yunkan
a759ab989f Catalog filtering by publisher works via Xapian 2021-04-27 16:59:21 +04:00
Veloman Yunkan
7ccd9ffcce Catalog filtering by language works via Xapian 2021-04-27 16:59:21 +04:00
Veloman Yunkan
0c0a37073b Catalog filtering by category works via Xapian 2021-04-27 16:59:21 +04:00
Veloman Yunkan
415c65cf03 Catalog filtering by book name works via Xapian 2021-04-27 16:59:21 +04:00
Veloman Yunkan
8287f351e7 Final logic of Library::filterViaBookDB()
Moved the `filter.hasQuery()` check inside `buildXapianQuery()`.
`Library::filterViaBookDB()` only cares if the query that is going to be
run on the book DB would match all documents. The rest of changes
related to enhancing the usage of Xapian for the catalog search will
happen inside `buildXapianQuery()` and `updateBookDB()`.
2021-04-27 16:59:21 +04:00
Veloman Yunkan
ea779ac200 Extracted buildXapianQuery() 2021-04-27 16:59:21 +04:00
Veloman Yunkan
80cd1fc989 Renamed 2 functions in Filter and Library 2021-04-27 16:59:21 +04:00
Veloman Yunkan
2d76f8395e Dropped unused functions from Filter's private API
This should have been done back in PR #460
2021-04-27 16:59:21 +04:00
Veloman Yunkan
29a6a34ecf Delimited kiwix::Filter's public and private APIs 2021-04-27 16:59:21 +04:00
Veloman Yunkan
2f3f1a4859 Improved LibraryTest.filterByMultipleCriteria 2021-04-27 16:59:21 +04:00
Veloman Yunkan
b9be742085 LibraryTest.filterByMaxSize 2021-04-27 16:59:21 +04:00
Veloman Yunkan
95c354a5fa LibraryTest.filterByCategory 2021-04-27 16:59:21 +04:00
Veloman Yunkan
cdd272fc5a LibraryTest.filterByName 2021-04-27 16:59:21 +04:00
Veloman Yunkan
ef962a9174 LibraryTest.filterByPublisher 2021-04-27 16:59:21 +04:00
Veloman Yunkan
f063d350c6 LibraryTest.{filterLocal,filterRemote} 2021-04-27 16:59:21 +04:00
Veloman Yunkan
d8fe593f59 Extended the unit-test library with 2 XML entries 2021-04-27 16:59:21 +04:00
Veloman Yunkan
22b8625033 Enter EXPECT_FILTER_RESULTS()
This diff is easier to view if whitespace change is ignored.
2021-04-27 16:59:21 +04:00
Veloman Yunkan
0f277ffa34 Enhanced the LibraryTest.filterByTags unit-test 2021-04-27 16:59:21 +04:00
Veloman Yunkan
068f7e5e95 New unit-test LibraryTest.filterByCreator 2021-04-27 16:59:21 +04:00
Veloman Yunkan
8c810d2d2f Enhanced the LibraryTest.filterByQuery unit-test 2021-04-27 16:59:21 +04:00
Veloman Yunkan
8c18a37961 Split LibraryTest.filterCheck into several tests 2021-04-27 16:59:21 +04:00
Veloman Yunkan
db3e0d7f72 Enhanced the LibraryTest.filterCheck unit-test
Now the LibraryTest.filterCheck unit-test validates the actual entries
returned by `Library::filter` (previously only the count of the results
was checked).
2021-04-27 16:59:21 +04:00
Matthieu Gautier
d134ad417f Merge pull request #497 from MananJethwani/issue/481
removed redirect to articles in search
2021-04-20 17:13:01 +02:00
Manan Jethwani
965b9622c2 removed redirect to articles in search 2021-04-20 20:23:42 +05:30
Kelson
11db5dec4e Merge pull request #494 from kiwix/ripple_effect_of_libzim_pr524
get_url() was renamed in zim::search_iterator
2021-04-16 12:38:15 +02:00
Veloman Yunkan
9d4370403b get_url() was renamed in zim::search_iterator 2021-04-16 13:30:36 +04:00
Kelson
cb57178c23 Merge pull request #491 from kiwix/fix_macos_ci
Update brew before installing packages
2021-04-12 18:41:00 +02:00
Matthieu Gautier
9ba5ab4678 Update brew before installing packages
brew changed his backend repository, we must update brew itself first.
2021-04-12 18:31:29 +02:00
Matthieu Gautier
a597870025 Merge pull request #465 from soumyankar/master 2021-04-12 18:17:57 +02:00
Vertigo
611146aa37 Added Search Link for bad bookName/articleName on 404 2021-04-12 21:31:47 +05:30
Matthieu Gautier
6d2f227c42 Merge pull request #486 from kiwix/fix_for_issue462 2021-04-12 15:18:57 +02:00
Veloman Yunkan
0c7d19ab45 Testing of Manager.readXml() 2021-04-12 15:14:12 +02:00
Veloman Yunkan
b54215f146 Manager::readOpds() doesn't modify its input 2021-04-12 15:14:12 +02:00
Veloman Yunkan
9033f2f28e Manager::readXml() doesn't modify its input 2021-04-12 15:14:12 +02:00
Matthieu Gautier
5c289abd0e Merge pull request #485 from kiwix/fix_for_issue478 2021-04-12 15:05:13 +02:00
Veloman Yunkan
ec9186b174 Library::removeBookById() updates the search DB
This fix makes the `XmlLibraryTest.removeBookByIdUpdatesTheSearchDB`
unit-test pass.
2021-04-09 17:06:45 +04:00
Veloman Yunkan
aaaa5a637e Library::filter() doesn't create empty books
This changes how the `XmlLibraryTest.removeBookByIdUpdatesTheSearchDB`
unit-test fails.
2021-04-09 17:06:45 +04:00
Veloman Yunkan
49940a30d0 XmlLibraryTest.removeBookByIdUpdatesTheSearchDB
The new unit-test fails with a reason not expected before it was
written. The `Library::filter()` operation returns a correct result
after the call to `removeBookById()` (this was a surprise!) but it has
a side-effect of re-adding an empty book with the id still surviving
in the search DB (the emptiness of this re-created book doesn't allow
it to pass the other filtering criteria, which explains why the result
of `Library::filter()` is correct). Had to add a special check
to the new unit-test against that hidden side-effect of
`Library::removeBookById()` + `Library::filter()` combination.
2021-04-09 17:06:45 +04:00
Veloman Yunkan
24ed96a38c Library.removeBookById() drops the reader too
This fix makes the `XmlLibraryTest.removeBookByIdDropsTheReader`
unit-test pass.
2021-04-09 17:05:56 +04:00
Veloman Yunkan
ccdc316217 Two unit-tests for Library::removeBookById
The `XmlLibraryTest.removeBookByIdDropsTheReader` unit-test fails,
demonstrating a bug in `kiwix::Library::removeBookById()`.
2021-04-09 16:59:55 +04:00
Matthieu Gautier
ba44033273 Merge pull request #464 from MananJethwani/issue/kiwix-tools/205
adding kind and path attributes to suggest response object and using it in autocomplete
2021-04-07 18:08:29 +02:00
Manan Jethwani
5cb276a933 adding kind and path attributes to suggest response object and using it in autocomplete 2021-04-07 21:04:33 +05:30
Matthieu Gautier
e4be97a032 Merge pull request #470 from kiwix/xapian_should_not_be_exposed 2021-04-07 16:55:24 +02:00
Veloman Yunkan
aa2a031ba4 Xapian headers are not exposed through libkiwix 2021-04-07 18:24:33 +04:00
Kelson
803cb1c2c5 Merge pull request #476 from MananJethwani/correcting#474
changed method of injecting root link
2021-03-24 10:20:54 +01:00
Manan Jethwani
7872734f44 changed method of injecting root link 2021-03-24 14:17:58 +05:30
Kelson
d061697de7 Merge pull request #474 from MananJethwani/issue/473
injecting root link directly and renamed head_part to head_taskbar
2021-03-24 09:24:42 +01:00
Manan Jethwani
c557bb271b injecting root link directly and renamed head_part to head_taskbar 2021-03-24 02:10:16 +05:30
Kelson
1f45c42c32 Merge pull request #472 from MananJethwani/issue/407
added root functionality for block external link feature
2021-03-23 10:54:22 +01:00
Manan Jethwani
93264f7409 added root functionality for block external link feature 2021-03-23 03:17:14 +05:30
Matthieu Gautier
baed447dd3 Merge pull request #460 from kiwix/xapian_based_catalog_search 2021-03-17 14:45:56 +01:00
Veloman Yunkan
20b487da8d Added Xapian as direct dependency 2021-03-17 14:32:03 +01:00
Veloman Yunkan
e214efecd4 Language code conversion via ICU
Language code is converted from ISO 639-3 to ISO 639 (which is
understood by Xapian) via ICU. The previous approach via an explicit
map had its advantages since Xapian has more than one stemmer
implementations for some languages (selectable via Xapian-specific
identifiers). This commit relies on the defaults associated with the
ISO 639 language codes.
2021-03-17 14:32:03 +01:00
Veloman Yunkan
09233bf4f3 Support for partial queries in catalog search
The search text in the catalog query is interpreted as partial by
default, but partial query mode can be disabled in C++. The latter
possibility is not exposed via the /catalog/search kiwix-serve endpoint,
though.
2021-03-17 14:32:03 +01:00
Veloman Yunkan
47c67a4202 LibraryServerTest.catalog_search_with_word_exclusion 2021-03-17 14:32:03 +01:00
Veloman Yunkan
6b600a18eb LibraryServerTest.catalog_prefix_search 2021-03-17 14:32:03 +01:00
Veloman Yunkan
9e887cadf1 Added some diversity to test/data/library.xml 2021-03-17 14:32:03 +01:00
Veloman Yunkan
a599fb3892 Initial version of Xapian-based catalog search 2021-03-17 14:32:03 +01:00
Veloman Yunkan
a17fc0ef2d Library::getBooksByTitleOrDescription() 2021-03-17 14:32:03 +01:00
Veloman Yunkan
db06b2c7ca Library::BookIdCollection typedef 2021-03-17 14:32:03 +01:00
Veloman Yunkan
a20f9e2ce1 Library::filter() works in two stages
1. Get the subset of books matching the q (title/description) parameter
   of the search

2. Filter out books not matching the other parameters of the search.

Stage 1. currently works in the old way, but will be replaced by Xapian
based search in subsequent commits.
2021-03-17 14:32:03 +01:00
Kelson
f7c867f8a7 Merge pull request #459 from kiwix/opds_category_support
Support for book categories in OPDS feed
2021-03-17 12:37:16 +01:00
Veloman Yunkan
b7b0bdbdd8 Both Book::update() methods update the category 2021-03-17 14:10:57 +04:00
Veloman Yunkan
a870e05621 Slight enhancement of BookTest.updateTest 2021-03-17 14:10:57 +04:00
Veloman Yunkan
4abc4f8518 Support for book category attribute in library.xml 2021-03-17 14:10:57 +04:00
Veloman Yunkan
6b2067c236 Reading category element from OPDS stream 2021-03-17 14:10:57 +04:00
Veloman Yunkan
e55bf514e8 Dedicated 'category' parameter in catalog search 2021-03-17 14:10:57 +04:00
Veloman Yunkan
80d4f7e349 Extracted InternalServer::search_catalog() 2021-03-17 14:10:57 +04:00
Veloman Yunkan
f270724b1f Testing of a library entry without a category 2021-03-17 14:10:44 +04:00
Veloman Yunkan
58186ffb26 kiwix::Book::getCategory() 2021-03-17 14:09:48 +04:00
Veloman Yunkan
6d43fd065f Less boilerplate in LibraryServerTest unit-tests 2021-03-17 14:02:27 +04:00
Veloman Yunkan
071d2bedd3 LibraryServerTest.catalog_search_by_text 2021-03-17 14:02:27 +04:00
Veloman Yunkan
0b1740e6c5 LibraryServerTest.catalog_search_by_tag 2021-03-17 14:02:27 +04:00
Veloman Yunkan
9913f748e2 LibraryServerTest.catalog_searchdescription_xml 2021-03-17 14:02:27 +04:00
Veloman Yunkan
c5c40cb189 New unit-test LibraryServerTest.catalog_root_xml 2021-03-17 14:02:27 +04:00
Veloman Yunkan
ae32ff40c0 Dropped an extra colon from book <updated> dates 2021-03-17 14:02:27 +04:00
Veloman Yunkan
26331b401e Fixed the month in OPDS feed <updated> date
`tm::tm_mon` varies in the [0, 11] range.
2021-03-17 14:02:27 +04:00
Matthieu Gautier
0f368791a2 Merge pull request #461 from MananJethwani/issue/444 2021-03-15 16:57:10 +01:00
Manan Jethwani
fb26f6b9c5 moved autocomplete from head_part.html to taskbar.js 2021-03-15 18:10:10 +05:30
Matthieu Gautier
32643fbd94 Merge pull request #463 from soumyankar/master
Change Mustache-3.0 to Mustache-4.1
2021-03-15 11:38:57 +01:00
Vertigo
faa9e1f8b5 Change Mustache-3.0 to Mustache-4.1 2021-03-14 11:43:48 +05:30
Kelson
bd781f8e8b Merge pull request #458 from kiwix/html_decoded_suggestions
Correctly encoding/decoding HTML entities in search suggestions
2021-03-04 14:34:47 +01:00
Veloman Yunkan
c7d77395e7 label field of suggestions is also HTML escaped
Without this if the suggestion text contains a double quote, the
response stops being a valid json.
2021-03-04 14:18:58 +01:00
Veloman Yunkan
a7fea462b0 HTML decoding of suggestions in the frontend
Since the `value` field of the search suggestion results is HTML
escaped/encoded in the backend (see static/templates/suggestion.json) it
must be decoded in the frontend.
2021-03-04 14:18:58 +01:00
Matthieu Gautier
89d7e68a39 Merge pull request #446 from kiwix/libzim_random
Use the new libzim's getRandomEntry instead of implementing it ourselves.
2021-03-03 16:04:00 +01:00
Matthieu Gautier
67caae6c32 Use the new libzim's getRandomEntry instead of implementing it ourselves. 2021-03-02 14:16:09 +01:00
Kelson
d3f2e08b35 Merge pull request #429 from kiwix/open_zimfile_by_fd
JNI interface to opening ZIM archives (including embedded ones) by fd
2021-02-26 09:20:58 +01:00
Veloman Yunkan
839fc10a4f Fixed the Windows build
Opening ZIM archives by file descriptor (as well as embedded
ZIM archives) is not supported under Windows.
2021-02-10 14:19:47 +01:00
Veloman Yunkan
5a8b825c70 Testing of JNIKiwixReader.getDirectAccessInformation() 2021-02-10 14:19:47 +01:00
Veloman Yunkan
7a465e66d7 Renamed org.kiwix.kiwixlib.{Pair->DirectAccessInfo} 2021-02-10 14:19:47 +01:00
Veloman Yunkan
5a99634dfd Java wrapper test checks favicon.png too 2021-02-10 14:19:47 +01:00
Veloman Yunkan
e028bcbb04 Android's java.io.FileDescriptor is different 2021-02-10 14:19:47 +01:00
Veloman Yunkan
9cdf7a44c0 JNIKiwixReader can open an embedded ZIM archive 2021-02-10 14:19:47 +01:00
Veloman Yunkan
4d23e44de7 JNIKiwixReader ctor taking a file descriptor
... and a corresponding unit test
2021-02-10 14:19:47 +01:00
Veloman Yunkan
98d69ef59b Added testReader unit-test for the java wrapper 2021-02-10 14:19:47 +01:00
Veloman Yunkan
e40827fbac Renamed the java wrapper unit test runner script 2021-02-10 14:19:47 +01:00
Veloman Yunkan
a798e0c0a1 Made the java wrapper unit test run & pass
The kiwixlib java wrapper unit test can be run manually via the
src/wrapper/java/org/kiwix/testing/compile_test.sh script.

The test ZIM files in src/wrapper/java/org/kiwix/testing were created
using the create_test_zimfiles. They must be updated/re-generated and
committed in git whenever their source data or the create_test_zimfiles
script changes. Note: small.zim.embedded is not used at this point, it
was created for testing the enhancement coming in a few commits.
2021-02-10 14:19:47 +01:00
Matthieu Gautier
971374e049 Merge pull request #455 from kiwix/api-break-bumpup-major-version
Bump-up version to 10.0.0
2021-02-10 10:47:06 +01:00
Emmanuel Engelhart
f7608c378e Bump-up version to 10.0.0 2021-02-10 10:31:01 +01:00
Kelson
c13a43010a Merge pull request #453 from kiwix/fixed-typo
Typo fix in error message
2021-02-09 09:41:00 +01:00
Emmanuel Engelhart
eea10ec3f5 Typo fix in error message 2021-02-09 09:38:18 +01:00
Kelson
c9bc2b48b0 Merge pull request #452 from kiwix/version-update
Bump-up version (and libzim required version)
2021-02-08 21:30:11 +01:00
Veloman Yunkan
b1689e0d3c Bump-up version (and libzim required version) 2021-02-07 19:53:46 +04:00
Kelson
c70c370ae0 Merge pull request #435 from kiwix/ppa-soname-bump
PPA: Update for soname bump
2021-01-27 09:25:39 +01:00
Kunal Mehta
c0ec5eeffd PPA: Update for soname bump 2021-01-27 09:14:36 +01:00
Matthieu Gautier
c81c2a4630 Merge pull request #445 from kiwix/no_pthread 2021-01-26 18:06:16 +01:00
Matthieu Gautier
24b2e6e585 Remove unnecessary include. 2021-01-26 17:53:25 +01:00
Matthieu Gautier
3fd1310008 Use c++11 std::thread instead of pthread. 2021-01-26 17:53:25 +01:00
Matthieu Gautier
17bc1a3e1a [TEST] Do not try to use the bookId if it is wrong. 2021-01-26 17:53:25 +01:00
Matthieu Gautier
f5d9a3714d Merge pull request #449 from kiwix/handle_no_counter
Do not crash if zim file has no `Counter` metadata.
2021-01-26 17:52:33 +01:00
Matthieu Gautier
4749656828 Do not crash if zim file has no Counter metadata. 2021-01-26 15:15:27 +01:00
Kelson
e0bcafd89a Merge pull request #442 from kiwix/better-taskbar-insertion
Better taskbar insertion
2021-01-18 13:32:11 +01:00
Emmanuel Engelhart
84895c4036 Better </head> detection regex 2021-01-18 13:16:56 +01:00
Emmanuel Engelhart
a8bf9dd5b4 Better Kiwix Serve Taskbar insertion (after charset definition) 2021-01-18 11:18:53 +01:00
Emmanuel Engelhart
a61c94ef10 Add GPLv3 header 2021-01-18 10:54:33 +01:00
Kelson
a8a864f70e Merge pull request #434 from kiwix/mandatory_xapian
Build kiwix-lib only if meson is compiled with xapian.
2021-01-15 10:03:57 +01:00
Matthieu Gautier
a231dfd8e4 Build kiwix-lib only if meson is compiled with xapian. 2021-01-14 12:07:17 +01:00
Kelson
321d08e3d5 Merge pull request #440 from kiwix/better-taskbar-introduction
Fix taskbar insertion in case of '<head>' attributes
2021-01-12 11:48:57 +01:00
Emmanuel Engelhart
8c43fd8d36 Fix taskbar insertion in case of '<head>' attributes 2021-01-11 14:37:19 +01:00
Kelson
9e032b6eea Merge pull request #439 from kiwix/mediacount-video-audio
Support 'video/*' * 'audio/*' mimetypes in getMediaCount()
2021-01-08 13:01:42 +01:00
Emmanuel Engelhart
3e2810dff4 Support 'video/*' * 'audio/*' mimetypes in getMediaCount() 2021-01-07 12:32:32 +01:00
Kelson
18afa97674 Merge pull request #437 from kiwix/count-webp-media
More robust getMediaCount()
2021-01-03 21:15:55 +01:00
Emmanuel Engelhart
44c4aa931a Better use kiwix::startsWith() 2021-01-03 15:17:03 +01:00
Emmanuel Engelhart
95b32b168d More robust getMediaCount() 2021-01-01 17:05:32 +01:00
Kelson
2659f323cd Merge pull request #433 from kiwix/small_fixes
Fixes related to libzim next.
2020-12-11 06:46:40 +01:00
Matthieu Gautier
1002c15e0d Remove unnecessary checks.
`Reader` cannot be created with a null `zimArchive`.
We don't have to check for zimArchive being not null.
2020-12-09 14:25:02 +01:00
Matthieu Gautier
d51000c4a9 Use new libzim method hasFulltextIndex to check for fulltext index. 2020-12-09 14:25:02 +01:00
Matthieu Gautier
ba302bed33 Use new libzim method getFaviconEntry to get the favicon. 2020-12-09 14:25:02 +01:00
Kelson
9941d245e1 Merge pull request #432 from swills/fbsd-build-fix1
fix build on FreeBSD
2020-12-09 14:24:04 +01:00
Steve Wills
6900b4e506 fix build on FreeBSD
With this header, sockaddr_in and INADDR_ANY are not defined
2020-12-07 09:38:46 -05:00
Kelson
6fa20f6dcf Merge pull request #431 from kiwix/legoktm-patch-1
PPA: Build for Ubuntu Hirsute
2020-12-05 09:29:47 +01:00
Kunal Mehta
8eacb0d635 PPA: Build for Ubuntu Hirsute 2020-12-03 22:35:46 -08:00
Kelson
84c320d5d3 Merge pull request #428 from kiwix/libzim_next
Adapt kiwx-lib to the new api of libzim (libzim_next).
2020-12-03 09:11:28 +01:00
Matthieu Gautier
1a5a2e7a8e Adapt kiwix-lib to the new libzim api. 2020-12-02 12:16:48 +01:00
Matthieu Gautier
d87079ec13 Remove deprecated method in the reader. 2020-11-24 19:00:52 +01:00
123 changed files with 10689 additions and 3875 deletions

View File

@@ -13,14 +13,15 @@ jobs:
with:
python-version: '3.5'
- name: Install packages
run:
run: |
brew update
brew install gcovr pkg-config ninja
- name: Install python modules
run: pip3 install meson==0.49.2 pytest
- name: Install deps
shell: bash
run: |
ARCHIVE_NAME=deps2_osx_native_dyn_kiwix-lib.tar.xz
ARCHIVE_NAME=deps2_osx_native_dyn_libkiwix.tar.xz
wget -O- http://tmp.kiwix.org/ci/${ARCHIVE_NAME} | tar -xJ -C $HOME
- name: Compile
shell: bash
@@ -93,7 +94,7 @@ jobs:
HOME: /home/runner
runs-on: ubuntu-latest
container:
image: "kiwix/kiwix-build_ci:${{matrix.image_variant}}-26"
image: "kiwix/kiwix-build_ci:${{matrix.image_variant}}-31"
steps:
- name: Extract branch name
shell: bash
@@ -114,7 +115,7 @@ jobs:
- name: Install deps
shell: bash
run: |
ARCHIVE_NAME=deps2_${OS_NAME}_${{matrix.target}}_kiwix-lib.tar.xz
ARCHIVE_NAME=deps2_${OS_NAME}_${{matrix.target}}_libkiwix.tar.xz
wget -O- http://tmp.kiwix.org/ci/${ARCHIVE_NAME} | tar -xJ -C /home/runner
- name: Compile
shell: bash
@@ -133,7 +134,7 @@ jobs:
if [[ "${{matrix.target}}" =~ android_.* ]]; then
MESON_OPTION="$MESON_OPTION -Dandroid=true"
fi
cd $HOME/kiwix-lib
cd $HOME/libkiwix
meson . build ${MESON_OPTION}
cd build
ninja
@@ -144,7 +145,7 @@ jobs:
if: startsWith(matrix.target, 'native_')
shell: bash
run: |
cd $HOME/kiwix-lib/build
cd $HOME/libkiwix/build
meson test --verbose
ninja coverage
env:
@@ -153,7 +154,7 @@ jobs:
- name: Publish coverage
shell: bash
run: |
cd $HOME/kiwix-lib
cd $HOME/libkiwix
curl https://codecov.io/bash -o codecov.sh
bash codecov.sh -n "${OS_NAME}_${{matrix.target}}" -Z
rm codecov.sh

View File

@@ -7,7 +7,11 @@ jobs:
strategy:
fail-fast: false
matrix:
distro: [ubuntu-groovy, ubuntu-focal, ubuntu-bionic]
distro:
- ubuntu-impish
- ubuntu-hirsute
- ubuntu-focal
- ubuntu-bionic
steps:
- uses: actions/checkout@v2
@@ -30,10 +34,18 @@ jobs:
email: release+launchpad@kiwix.org
distro: ${{ matrix.distro }}
- uses: legoktm/gh-action-build-deb@ubuntu-groovy
if: matrix.distro == 'ubuntu-groovy'
name: Build package for ubuntu-groovy
id: build-ubuntu-groovy
- uses: legoktm/gh-action-build-deb@ubuntu-impish
if: matrix.distro == 'ubuntu-impish'
name: Build package for ubuntu-impish
id: build-ubuntu-impish
with:
args: --no-sign
ppa: ${{ steps.ppa.outputs.ppa }}
- uses: legoktm/gh-action-build-deb@ubuntu-hirsute
if: matrix.distro == 'ubuntu-hirsute'
name: Build package for ubuntu-hirsute
id: build-ubuntu-hirsute
with:
args: --no-sign
ppa: ${{ steps.ppa.outputs.ppa }}

1
.gitignore vendored
View File

@@ -4,3 +4,4 @@ subprojects/googletest-release*
*.class
build/
.vscode/
builddir/

View File

@@ -1,3 +1,8 @@
libkiwix 10.0.0
===============
* ...
kiwix-lib 9.4.1
===============

109
README.md
View File

@@ -1,50 +1,50 @@
Kiwix library
=============
Libkiwix
========
The Kiwix library provides the [Kiwix](https://kiwix.org) software
suite core. It contains the code shared by all Kiwix ports (Windows,
The Libkiwix provides the [Kiwix](https://kiwix.org) software suite
core. It contains the code shared by all Kiwix ports (Windows,
GNU/Linux, macOS, Android, iOS, ...).
[![Download](https://api.bintray.com/packages/kiwix/kiwix/kiwixlib/images/download.svg)](https://bintray.com/kiwix/kiwix/kiwixlib/_latestVersion)
[![Build Status](https://github.com/kiwix/kiwix-lib/workflows/CI/badge.svg?query=branch%3Amaster)](https://github.com/kiwix/kiwix-lib/actions?query=branch%3Amaster)
[![CodeFactor](https://www.codefactor.io/repository/github/kiwix/kiwix-lib/badge)](https://www.codefactor.io/repository/github/kiwix/kiwix-lib)
[![Codecov](https://codecov.io/gh/kiwix/kiwix-lib/branch/master/graph/badge.svg)](https://codecov.io/gh/kiwix/kiwix-lib)
[![Repositories](https://img.shields.io/repology/repositories/libkiwix?label=repositories)](https://github.com/kiwix/libkiwix/wiki/Repology)
[![Build Status](https://github.com/kiwix/libkiwix/workflows/CI/badge.svg?query=branch%3Amaster)](https://github.com/kiwix/libkiwix/actions?query=branch%3Amaster)
[![CodeFactor](https://www.codefactor.io/repository/github/kiwix/libkiwix/badge)](https://www.codefactor.io/repository/github/kiwix/libkiwix)
[![Codecov](https://codecov.io/gh/kiwix/libkiwix/branch/master/graph/badge.svg)](https://codecov.io/gh/kiwix/libkiwix)
[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
[![Packaging status](https://repology.org/badge/vertical-allrepos/kiwix-lib.svg)](https://repology.org/project/kiwix-lib/versions)
Disclaimer
----------
This document assumes you have a little knowledge about software
compilation. If you experience difficulties with the dependencies or
with the Kiwix libary compilation itself, we recommend to have a look
to [kiwix-build](https://github.com/kiwix/kiwix-build).
with the Libkiwix compilation itself, we recommend to have a look to
[kiwix-build](https://github.com/kiwix/kiwix-build).
Preamble
--------
Although the Kiwix library can be (cross-)compiled on/for many
sytems, the following documentation explains how to do it on POSIX
ones. It is primarly thought for GNU/Linux systems and has been tested
on recent releases of Ubuntu and Fedora.
Although the Libkiwix can be (cross-)compiled on/for many sytems, the
following documentation explains how to do it on POSIX ones. It is
primarly thought for GNU/Linux systems and has been tested on recent
releases of Ubuntu and Fedora.
Dependencies
------------
The Kiwix library relies on many third parts software libraries. They
are prerequisites to the Kiwix library compilation. Following
libraries need to be available:
The Libkiwix relies on many third party software libraries. They are
prerequisites to the Libkiwix compilation. Following libraries need to
be available:
* [ICU](https://site.icu-project.org/) (package `libicu-dev` on Ubuntu)
* [ZIM](https://openzim.org/) (package `libzim-dev` on Ubuntu)
* [Pugixml](https://pugixml.org/) (package `libpugixml-dev` on Ubuntu)
* [Mustache](https://github.com/kainjow/Mustache) (Just copy the
header `mustache.hpp` somewhere it can be found by the compiler and/or
set CPPFLAGS with correct `-I` option). Use Mustache version 3 only.
* [libcurl](https://curl.se/libcurl) (`libcurl4-gnutls-dev`, `libcurl4-nss-dev` or `libcurl4-openssl-dev` on Ubuntu)
* [microhttpd](https://www.gnu.org/software/libmicrohttpd) (package `libmicrohttpd-dev` on Ubuntu)
* [zlib](https://zlib.net/) (package `zlib1g-dev` on Ubuntu)
set CPPFLAGS with correct `-I` option). Use Mustache version 4.1 or above.
* [Libcurl](https://curl.se/libcurl) (`libcurl4-gnutls-dev`, `libcurl4-nss-dev` or `libcurl4-openssl-dev` on Ubuntu)
* [Microhttpd](https://www.gnu.org/software/libmicrohttpd) (package `libmicrohttpd-dev` on Ubuntu)
* [Zlib](https://zlib.net/) (package `zlib1g-dev` on Ubuntu)
To test the code:
* [Google Test](https://github.com/google/googletest) (package `googletest` on Ubuntu)
The following dependency needs to be available at runtime:
* [Aria2](https://aria2.github.io/) (package `aria2` on Ubuntu)
@@ -56,12 +56,12 @@ In the worse case, you will have to download and compile bleeding edge
version by hand.
If you want to install these dependencies locally, then use the
`kiwix-lib` directory as install prefix.
`libkiwix` directory as install prefix.
Environment
-------------
The Kiwix library builds using [Meson](https://mesonbuild.com/) version
The Libkiwix builds using [Meson](https://mesonbuild.com/) version
0.45 or higher. Meson relies itself on Ninja, pkg-config and few other
compilation tools.
@@ -77,7 +77,7 @@ section.
Compilation
-----------
Once all dependencies are installed, you can compile the Kiwix library
Once all dependencies are installed, you can compile the Libkiwix
with:
```bash
meson . build
@@ -85,7 +85,7 @@ ninja -C build
```
By default, it will compile dynamic linked libraries. All binary files
will be created in the "build" directory created automatically by
will be created in the `build` directory created automatically by
Meson. If you want statically linked libraries, you can add
`--default-library=static` option to the Meson command.
@@ -103,7 +103,7 @@ meson test
Installation
------------
If you want to install the Kiwix library and the headers you just have
If you want to install the Libkiwix and the headers you just have
compiled on your system, here we go:
```bash
ninja -C build install
@@ -146,6 +146,57 @@ cp ninja ../bin
cd ..
```
Custom Index Page
-----------------
to use custom welcome page mention `customIndexPage` argument in `kiwix::internalServer()` or use `kiwix::server->setCustomIndexTemplate()`.
(note - while using custom html file please mention all external links as absolute path.)
to create a HTML template with custom JS you need to have a look at various OPDS based endpoints as mentioned [here](https://wiki.kiwix.org/wiki/OPDS) to load books.
To use JS provided by kiwix-serve you can use the following template to start with ->
```
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<title><-- Custom Tittle --></title>
<script
type="text/javascript"
src="{{root}}/skin/jquery-ui/external/jquery/jquery.js"
></script>
<script
type="text/javascript"
src="{{root}}/skin/jquery-ui/jquery-ui.min.js"
></script>
<script src="{{root}}/skin/isotope.pkgd.min.js" defer></script>
<script src="{{root}}/skin/iso6391To3.js"></script>
<script type="text/javascript" src="{{root}}/skin/index.js" defer></script>
</head>
<body>
</body>
</html>
```
- To get books listed using `index.js` add - `<div class="book__list"></div>` under body tag.
- To get number of books listed add - `<h3 class="kiwixHomeBody__results"></h3>` under body tag.
- To add language select box add - `<select id="languageFilter"></select>` under body tag.
- To add language select box add - `<select id="categoryFilter"></select>` under body tag.
- To add search box for books use following form -
```
<form id='kiwixSearchForm'>
<input type="text" name="q" placeholder="Search" id="searchFilter" class='kiwixSearch filter'>
<input type="submit" class="searchButton" value="Search"/>
</form>
```
If you compile manually Libmicrohttpd, you might need to compile it
without GNU TLS, a bug here will empeach further compilation
otherwise.
If the compilation still fails, you might need to get a more recent
version of a dependency than the one packaged by your Linux
distribution. Try then with a source tarball distributed by the

View File

@@ -26,10 +26,10 @@ task writePom {
project {
groupId 'org.kiwix.kiwixlib'
artifactId 'kiwixlib'
version '9.4.1' + (System.env.KIWIXLIB_BUILDVERSION == null ? '' : '-'+System.env.KIWIXLIB_BUILDVERSION)
version '10.0.0' + (System.env.KIWIXLIB_BUILDVERSION == null ? '' : '-'+System.env.KIWIXLIB_BUILDVERSION)
packaging 'aar'
name 'kiwixlib'
url 'https://github.com/kiwix/kiwix-lib'
url 'https://github.com/kiwix/libkiwix'
licenses {
license {
name 'GPLv3'
@@ -44,9 +44,9 @@ task writePom {
}
}
scm {
connection 'https://github.com/kiwix/kiwix-lib.git'
developerConnection 'https://github.com/kiwix/kiwix-lib.git'
url 'https://github.com/kiwix/kiwix-lib'
connection 'https://github.com/kiwix/libkiwix.git'
developerConnection 'https://github.com/kiwix/libkiwix.git'
url 'https://github.com/kiwix/libkiwix'
}
}
}.withXml {

8
debian/control vendored
View File

@@ -15,14 +15,14 @@ Build-Depends: debhelper-compat (= 13),
zlib1g-dev
Standards-Version: 4.5.0
Section: libs
Homepage: https://github.com/kiwix/kiwix-lib
Homepage: https://github.com/kiwix/libkiwix
Rules-Requires-Root: no
Package: libkiwix-dev
Section: libdevel
Architecture: any
Multi-Arch: same
Depends: libkiwix9 (= ${binary:Version}), ${misc:Depends}, python3,
Depends: libkiwix10 (= ${binary:Version}), ${misc:Depends}, python3,
libzim-dev (>= 6.0.0),
libicu-dev,
libpugixml-dev,
@@ -35,11 +35,11 @@ Description: library of common code for Kiwix (development)
.
This package contains development files.
Package: libkiwix9
Package: libkiwix10
Architecture: any
Multi-Arch: same
Depends: ${shlibs:Depends}, ${misc:Depends}, aria2
Conflicts: libkiwix0, libkiwix3
Conflicts: libkiwix0, libkiwix3, libkiwix9
Description: library of common code for Kiwix
Kiwix is an offline Wikipedia reader. libkiwix provides the
software core for Kiwix, and contains the code shared by all

View File

View File

@@ -7,6 +7,7 @@ files=(
"include/common/otherTools.h"
"include/common/regexTools.h"
"include/common/networkTools.h"
"include/common/archiveTools.h"
"include/manager.h"
"include/reader.h"
"include/kiwix.h"
@@ -22,6 +23,7 @@ files=(
"src/common/pathTools.cpp"
"src/common/regexTools.cpp"
"src/common/otherTools.cpp"
"src/common/archiveTools.cpp"
"src/common/networkTools.cpp"
"src/common/stringTools.cpp"
"src/xapianSearcher.cpp"

View File

@@ -26,6 +26,10 @@ namespace pugi {
class xml_node;
}
namespace zim {
class Archive;
}
namespace kiwix
{
@@ -43,6 +47,7 @@ class Book
bool update(const Book& other);
void update(const Reader& reader);
void update(const zim::Archive& archive);
void updateFromXml(const pugi::xml_node& node, const std::string& baseDir);
void updateFromOpds(const pugi::xml_node& node, const std::string& urlHost);
std::string getHumanReadableIdFromPath() const;
@@ -59,6 +64,7 @@ class Book
const std::string& getDate() const { return m_date; }
const std::string& getUrl() const { return m_url; }
const std::string& getName() const { return m_name; }
std::string getCategory() const;
const std::string& getTags() const { return m_tags; }
std::string getTagStr(const std::string& tagName) const;
bool getTagBool(const std::string& tagName) const;
@@ -94,6 +100,9 @@ class Book
void setFaviconMimeType(const std::string& faviconMimeType) { m_faviconMimeType = faviconMimeType; }
void setDownloadId(const std::string& downloadId) { m_downloadId = downloadId; }
private:
std::string getCategoryFromTags() const;
protected:
std::string m_id;
std::string m_downloadId;
@@ -101,6 +110,7 @@ class Book
bool m_pathValid = false;
std::string m_title;
std::string m_description;
std::string m_category;
std::string m_language;
std::string m_creator;
std::string m_publisher;

View File

@@ -23,7 +23,6 @@
#include <string>
#include <vector>
#include <map>
#include <pthread.h>
#include <memory>
#include <stdexcept>

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2018 Matthieu Gautier <mgautier@kymeria.fr>
* Copyright 2018-2020 Matthieu Gautier <mgautier@kymeria.fr>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -21,7 +21,8 @@
#define KIWIX_ENTRY_H
#include <stdio.h>
#include <zim/article.h>
#include <zim/entry.h>
#include <zim/item.h>
#include <exception>
#include <string>
@@ -41,19 +42,12 @@ class NoEntry : public std::exception {};
class Entry
{
public:
/**
* Default constructor.
*
* Construct an invalid entry.
*/
Entry() = default;
/**
* Construct an entry making reference to an zim article.
*
* @param article a zim::Article object
*/
Entry(zim::Article article);
Entry(zim::Entry entry);
virtual ~Entry() = default;
/**
@@ -63,14 +57,14 @@ class Entry
*
* @return the path of the entry.
*/
std::string getPath() const;
std::string getPath() const { return entry.getPath(); }
/**
* Get the title of the entry.
*
* @return the title of the entry.
*/
std::string getTitle() const;
std::string getTitle() const { return entry.getTitle(); }
/**
* Get the content of the entry.
@@ -80,7 +74,7 @@ class Entry
*
* @return the content of the entry.
*/
std::string getContent() const;
std::string getContent() const { return entry.getItem().getData(); }
/**
* Get the blob of the entry.
@@ -90,7 +84,7 @@ class Entry
* @param offset The starting offset of the blob.
* @return the blob of the entry.
*/
zim::Blob getBlob(offset_type offset = 0) const;
zim::Blob getBlob(offset_type offset = 0) const { return entry.getItem().getData(offset); }
/**
* Get the blob of the entry.
@@ -101,7 +95,7 @@ class Entry
* @param size The size of the blob.
* @return the blob of the entry.
*/
zim::Blob getBlob(offset_type offset, size_type size) const;
zim::Blob getBlob(offset_type offset, size_type size) const { return entry.getItem().getData(offset, size); }
/**
* Get the info for direct access to the content of the entry.
@@ -109,7 +103,7 @@ class Entry
* Some entry (ie binary ones) have their content plain stored
* in the zim file. Knowing the offset where the content is stored
* an user can directly read the content in the zim file bypassing the
* kiwix-lib/libzim.
* libkiwix/libzim.
*
* @return A pair specifying where to read the content.
* The string is the real file to read (may be different that .zim
@@ -117,7 +111,7 @@ class Entry
* The offset is the offset to read in the file.
* Return <"",0> if is not possible to read directly.
*/
std::pair<std::string, offset_type> getDirectAccessInfo() const;
zim::Item::DirectAccessInfo getDirectAccessInfo() const { return entry.getItem().getDirectAccessInformation(); }
/**
* Get the size of the entry.
@@ -174,17 +168,14 @@ class Entry
Entry getFinalEntry() const;
/**
* Convert the entry to a boolean value.
* Get the zim entry wrapped by this (kiwix) entry
*
* @return True if the entry is valid.
* @return the zim entry
*/
explicit operator bool() const { return good(); }
const zim::Entry& getZimEntry() const { return entry; }
private:
zim::Article article;
mutable zim::Article final_article;
bool good() const { return article.good(); }
zim::Entry entry;
};
}

View File

@@ -24,6 +24,7 @@
#include <vector>
#include <map>
#include <memory>
#include <zim/archive.h>
#include "book.h"
#include "bookmark.h"
@@ -35,6 +36,7 @@ namespace kiwix
{
class OPDSDumper;
class Library;
enum supportedListSortBy { UNSORTED, TITLE, SIZE, DATE, CREATOR, PUBLISHER };
enum supportedListMode {
@@ -48,18 +50,23 @@ enum supportedListMode {
};
class Filter {
private:
public: // types
using Tags = std::vector<std::string>;
private: // data
uint64_t activeFilters;
std::vector<std::string> _acceptTags;
std::vector<std::string> _rejectTags;
Tags _acceptTags;
Tags _rejectTags;
std::string _category;
std::string _lang;
std::string _publisher;
std::string _creator;
size_t _maxSize;
std::string _query;
bool _queryIsPartial;
std::string _name;
public:
public: // functions
Filter();
~Filter() = default;
@@ -93,16 +100,42 @@ class Filter {
/**
* Set the filter to only accept book with corresponding tag.
*/
Filter& acceptTags(std::vector<std::string> tags);
Filter& rejectTags(std::vector<std::string> tags);
Filter& acceptTags(const Tags& tags);
Filter& rejectTags(const Tags& tags);
Filter& category(std::string category);
Filter& lang(std::string lang);
Filter& publisher(std::string publisher);
Filter& creator(std::string creator);
Filter& maxSize(size_t size);
Filter& query(std::string query);
Filter& query(std::string query, bool partial=true);
Filter& name(std::string name);
bool hasQuery() const;
const std::string& getQuery() const { return _query; }
bool queryIsPartial() const { return _queryIsPartial; }
bool hasName() const;
const std::string& getName() const { return _name; }
bool hasCategory() const;
const std::string& getCategory() const { return _category; }
bool hasLang() const;
const std::string& getLang() const { return _lang; }
bool hasPublisher() const;
const std::string& getPublisher() const { return _publisher; }
bool hasCreator() const;
const std::string& getCreator() const { return _creator; }
const Tags& getAcceptTags() const { return _acceptTags; }
const Tags& getRejectTags() const { return _rejectTags; }
private: // functions
friend class Library;
bool accept(const Book& book) const;
};
@@ -114,12 +147,27 @@ class Library
{
std::map<std::string, kiwix::Book> m_books;
std::map<std::string, std::shared_ptr<Reader>> m_readers;
std::map<std::string, std::shared_ptr<zim::Archive>> m_archives;
std::vector<kiwix::Bookmark> m_bookmarks;
class BookDB;
std::unique_ptr<BookDB> m_bookDB;
public:
typedef std::vector<std::string> BookIdCollection;
typedef std::map<std::string, int> AttributeCounts;
public:
Library();
~Library();
/**
* Library is not a copiable object. However it can be moved.
*/
Library(const Library& ) = delete;
Library(Library&& );
void operator=(const Library& ) = delete;
Library& operator=(Library&& );
/**
* Add a book to the library.
*
@@ -148,9 +196,12 @@ class Library
*/
bool removeBookmark(const std::string& zimId, const std::string& url);
const Book& getBookById(const std::string& id) const;
Book& getBookById(const std::string& id);
const Book& getBookByPath(const std::string& path) const;
Book& getBookByPath(const std::string& path);
std::shared_ptr<Reader> getReaderById(const std::string& id);
std::shared_ptr<zim::Archive> getArchiveById(const std::string& id);
/**
* Remove a book from the library.
@@ -166,7 +217,7 @@ class Library
* @param path the path of the file to write to.
* @return True if the library has been correctly saved.
*/
bool writeToFile(const std::string& path);
bool writeToFile(const std::string& path) const;
/**
* Write the library bookmarks to a file.
@@ -174,7 +225,7 @@ class Library
* @param path the path of the file to write to.
* @return True if the library has been correctly saved.
*/
bool writeBookmarksToFile(const std::string& path);
bool writeBookmarksToFile(const std::string& path) const;
/**
* Get the number of book in the library.
@@ -183,42 +234,56 @@ class Library
* @param remoteBooks If we must count remote books (books with an url)
* @return The number of books.
*/
unsigned int getBookCount(const bool localBooks, const bool remoteBooks);
unsigned int getBookCount(const bool localBooks, const bool remoteBooks) const;
/**
* Get all langagues of the books in the library.
* Get all languagues of the books in the library.
*
* @return A list of languages.
*/
std::vector<std::string> getBooksLanguages();
std::vector<std::string> getBooksLanguages() const;
/**
* Get all languagues of the books in the library with counts.
*
* @return A list of languages with the count of books in each language.
*/
AttributeCounts getBooksLanguagesWithCounts() const;
/**
* Get all categories of the books in the library.
*
* @return A list of categories.
*/
std::vector<std::string> getBooksCategories() const;
/**
* Get all book creators of the books in the library.
*
* @return A list of book creators.
*/
std::vector<std::string> getBooksCreators();
std::vector<std::string> getBooksCreators() const;
/**
* Get all book publishers of the books in the library.
*
* @return A list of book publishers.
*/
std::vector<std::string> getBooksPublishers();
std::vector<std::string> getBooksPublishers() const;
/**
* Get all bookmarks.
*
* @return A list of bookmarks
*/
const std::vector<kiwix::Bookmark> getBookmarks(bool onlyValidBookmarks = true);
const std::vector<kiwix::Bookmark> getBookmarks(bool onlyValidBookmarks = true) const;
/**
* Get all book ids of the books in the library.
*
* @return A list of book ids.
*/
std::vector<std::string> getBooksIds();
BookIdCollection getBooksIds() const;
/**
* Filter the library and generate a new one with the keep elements.
@@ -228,7 +293,7 @@ class Library
* @param search List only books with search in the title or description.
* @return The list of bookIds corresponding to the query.
*/
DEPRECATED std::vector<std::string> filter(const std::string& search);
DEPRECATED BookIdCollection filter(const std::string& search) const;
/**
@@ -237,7 +302,7 @@ class Library
* @param filter The filter to use.
* @return The list of bookIds corresponding to the filter.
*/
std::vector<std::string> filter(const Filter& filter);
BookIdCollection filter(const Filter& filter) const;
/**
@@ -247,7 +312,7 @@ class Library
* @param comparator how to sort the books
* @return The sorted list of books
*/
void sort(std::vector<std::string>& bookIds, supportedListSortBy sortBy, bool ascending);
void sort(BookIdCollection& bookIds, supportedListSortBy sortBy, bool ascending) const;
/**
* List books in the library.
@@ -271,7 +336,7 @@ class Library
* Set to 0 to cancel this filter.
* @return The list of bookIds corresponding to the query.
*/
DEPRECATED std::vector<std::string> listBooksIds(
DEPRECATED BookIdCollection listBooksIds(
int supportedListMode = ALL,
supportedListSortBy sortBy = UNSORTED,
const std::string& search = "",
@@ -279,11 +344,21 @@ class Library
const std::string& creator = "",
const std::string& publisher = "",
const std::vector<std::string>& tags = {},
size_t maxSize = 0);
size_t maxSize = 0) const;
friend class OPDSDumper;
friend class libXMLDumper;
private: // types
typedef const std::string& (Book::*BookStrPropMemFn)() const;
private: // functions
AttributeCounts getBookAttributeCounts(BookStrPropMemFn p) const;
std::vector<std::string> getBookPropValueSet(BookStrPropMemFn p) const;
BookIdCollection filterViaBookDB(const Filter& filter) const;
void updateBookDB(const Book& book);
};
}
#endif

View File

@@ -38,7 +38,7 @@ class LibXMLDumper
{
public:
LibXMLDumper() = default;
LibXMLDumper(Library* library);
LibXMLDumper(const Library* library);
~LibXMLDumper();
/**
@@ -69,10 +69,10 @@ class LibXMLDumper
*
* @param library The library to dump.
*/
void setLibrary(Library* library) { this->library = library; }
void setLibrary(const Library* library) { this->library = library; }
protected:
kiwix::Library* library;
const kiwix::Library* library;
std::string baseDir;
private:
void handleBook(Book book, pugi::xml_node root_node);

View File

@@ -13,19 +13,8 @@ headers = [
'search_renderer.h',
'server.h',
'kiwixserve.h',
'name_mapper.h'
'name_mapper.h',
'tools.h'
]
install_headers(headers, subdir:'kiwix')
install_headers(
'tools/base64.h',
'tools/networkTools.h',
'tools/otherTools.h',
'tools/pathTools.h',
'tools/regexTools.h',
'tools/stringTools.h',
'tools/lock.h',
subdir:'kiwix/tools'
)

33
include/opds_catalog.h Normal file
View File

@@ -0,0 +1,33 @@
/*
* Copyright 2021 Veloman Yunkan <veloman.yunkan@gmail.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
* MA 02110-1301, USA.
*/
#ifndef KIWIX_OPDS_CATALOG_H
#define KIWIX_OPDS_CATALOG_H
#include "library.h"
namespace kiwix
{
std::string getSearchUrl(const Filter& f);
} // namespace kiwix
#endif // KIWIX_OPDS_CATALOG_H

View File

@@ -26,9 +26,6 @@
#include <pugixml.hpp>
#include "tools/base64.h"
#include "tools/pathTools.h"
#include "tools/regexTools.h"
#include "library.h"
#include "reader.h"
@@ -51,24 +48,50 @@ class OPDSDumper
/**
* Dump the OPDS feed.
*
* @param id The id of the library.
* @param bookIds the ids of the books to include in the feed
* @param query the query used to obtain the list of book ids
* @return The OPDS feed.
*/
std::string dumpOPDSFeed(const std::vector<std::string>& bookIds);
std::string dumpOPDSFeed(const std::vector<std::string>& bookIds, const std::string& query) const;
/**
* Set the id of the opds stream.
* Dump the OPDS feed.
*
* @param bookIds the ids of the books to include in the feed
* @param query the query used to obtain the list of book ids
* @param partial whether the feed should include partial or complete entries
* @return The OPDS feed.
*/
std::string dumpOPDSFeedV2(const std::vector<std::string>& bookIds, const std::string& query, bool partial) const;
/**
* Dump the OPDS complete entry document.
*
* @param bookId the id of the book
* @return The OPDS complete entry document.
*/
std::string dumpOPDSCompleteEntry(const std::string& bookId) const;
/**
* Dump the categories OPDS feed.
*
* @return The OPDS feed.
*/
std::string categoriesOPDSFeed() const;
/**
* Dump the languages OPDS feed.
*
* @return The OPDS feed.
*/
std::string languagesOPDSFeed() const;
/**
* Set the id of the library.
*
* @param id the id to use.
*/
void setId(const std::string& id) { this->id = id;}
/**
* Set the title oft the opds stream.
*
* @param title the title to use.
*/
void setTitle(const std::string& title) { this->title = title; }
void setLibraryId(const std::string& id) { this->libraryId = id;}
/**
* Set the root location used when generating url.
@@ -77,13 +100,6 @@ class OPDSDumper
*/
void setRootLocation(const std::string& rootLocation) { this->rootLocation = rootLocation; }
/**
* Set the search url.
*
* @param searchUrl the search url to use.
*/
void setSearchDescriptionUrl(const std::string& searchDescriptionUrl) { this->searchDescriptionUrl = searchDescriptionUrl; }
/**
* Set some informations about the search results.
*
@@ -93,27 +109,13 @@ class OPDSDumper
*/
void setOpenSearchInfo(int totalResult, int startIndex, int count);
/**
* Set the library to dump.
*
* @param library The library to dump.
*/
void setLibrary(Library* library) { this->library = library; }
protected:
kiwix::Library* library;
std::string id;
std::string title;
std::string date;
std::string libraryId;
std::string rootLocation;
std::string searchDescriptionUrl;
int m_totalResults;
int m_startIndex;
int m_count;
bool m_isSearchResult = false;
private:
pugi::xml_node handleBook(Book book, pugi::xml_node root_node);
};
}

View File

@@ -21,30 +21,61 @@
#define KIWIX_READER_H
#include <stdio.h>
#include <zim/article.h>
#include <zim/file.h>
#include <zim/fileiterator.h>
#include <zim/zim.h>
#include <zim/archive.h>
#include <exception>
#include <map>
#include <sstream>
#include <string>
#include "common.h"
#include "entry.h"
#include "tools/pathTools.h"
#include "tools/stringTools.h"
using namespace std;
namespace kiwix
{
/**
* The SuggestionItem is a helper class that contains the info about a single
* suggestion item.
*/
class SuggestionItem
{
// Functions
// Temporarily making the constructor public until the code move is complete
public:
// Create a sugggestion item.
explicit SuggestionItem(const std::string& title, const std::string& normalizedTitle,
const std::string& path, const std::string& snippet = "") :
title(title),
normalizedTitle(normalizedTitle),
path(path),
snippet(snippet) {}
public:
const std::string& getTitle() const { return title;}
const std::string& getNormalizedTitle() const { return normalizedTitle;}
const std::string& getPath() const { return path;}
const std::string& getSnippet() const { return snippet;}
bool hasSnippet() const { return !snippet.empty();}
// Data
private:
std::string title;
std::string normalizedTitle;
std::string path;
std::string snippet;
friend class Reader;
};
/**
* The Reader class is the class who allow to get an entry content from a zim
* file.
*/
using SuggestionsList_t = std::vector<std::vector<std::string>>;
using SuggestionsList_t = std::vector<SuggestionItem>;
class Reader
{
public:
@@ -57,8 +88,19 @@ class Reader
* unsplitted path as if the file were not splitted
* (.zim extesion).
*/
Reader(const string zimFilePath);
~Reader();
explicit Reader(const string zimFilePath);
/**
* Create a Reader to read a zim file given by the Archive.
*
* @param archive The shared pointer to the Archive object.
*/
explicit Reader(const std::shared_ptr<zim::Archive> archive);
#ifndef _WIN32
explicit Reader(int fd);
Reader(int fd, zim::offset_type offset, zim::size_type size);
#endif
~Reader() = default;
/**
* Get the number of "displayable" entries in the zim file.
@@ -101,17 +143,6 @@ class Reader
*/
string getId() const;
/**
* Get the url of a random page.
*
* Deprecated : Use `getRandomPage` instead.
*
* @return Url of a random page. The page is picked from all entries in
* the 'A' namespace.
* The main page is excluded from the potential results.
*/
DEPRECATED string getRandomPageUrl() const;
/**
* Get a random page.
*
@@ -121,31 +152,6 @@ class Reader
*/
Entry getRandomPage() const;
/**
* Get the url of the first page.
*
* Deprecated : Use `getFirstPage` instead.
*
* @return Url of the first entry in the 'A' namespace.
*/
DEPRECATED string getFirstPageUrl() const;
/**
* Get the entry of the first page.
*
* @return The first entry in the 'A' namespace.
*/
Entry getFirstPage() const;
/**
* Get the url of the main page.
*
* Deprecated : Use `getMainPage` instead.
*
* @return Url of the main page as specified in the zim file.
*/
DEPRECATED string getMainPageUrl() const;
/**
* Get the entry of the main page.
*
@@ -286,16 +292,6 @@ class Reader
*/
string getScraper() const;
/**
* Get the origId of the zim file.
*
* The origId is only used in the case of patch zim file and is the Id
* of the original zim file.
*
* @return The origId of the zim file as specified in the zim metadata.
*/
string getOrigId() const;
/**
* Get the favicon of the zim file.
*
@@ -334,88 +330,6 @@ class Reader
*/
Entry getEntryFromTitle(const std::string& title) const;
/**
* Get the url of a page specified by a title.
*
* @param[in] title the title of the page.
* @param[out] url the url of the page.
* @return True if the page can be found.
*/
DEPRECATED bool getPageUrlFromTitle(const string& title, string& url) const;
/**
* Get the mimetype of a entry specified by a url.
*
* @param[in] url the url of the entry.
* @param[out] mimeType the mimeType of the entry.
* @return True if the mimeType has been found.
*/
DEPRECATED bool getMimeTypeByUrl(const string& url, string& mimeType) const;
/**
* Get the content of an entry specifed by a url.
*
* Alias to `getContentByEncodedUrl`
*/
DEPRECATED bool getContentByUrl(const string& url,
string& content,
string& title,
unsigned int& contentLength,
string& contentType) const;
/**
* Get the content of an entry specified by a url encoded url.
*
* Equivalent to getContentByDecodedUrl(urlDecode(url), ...).
*/
DEPRECATED bool getContentByEncodedUrl(const string& url,
string& content,
string& title,
unsigned int& contentLength,
string& contentType,
string& baseUrl) const;
/**
* Get the content of an entry specified by an url encoded url.
*
* Equivalent to getContentByEncodedUrl but without baseUrl.
*/
DEPRECATED bool getContentByEncodedUrl(const string& url,
string& content,
string& title,
unsigned int& contentLength,
string& contentType) const;
/**
* Get the content of an entry specified by a url.
*
* @param[in] url The url of the entry.
* @param[out] content The content of the entry.
* @param[out] title the title of the entry.
* @param[out] contentLength The size of the entry (size of content).
* @param[out] contentType The mimeType of the entry.
* @param[out] baseUrl Return the true url of the entry.
* If the specified entry is a redirection, contains
* the url of the targeted entry.
* @return True if the entry has been found.
*/
DEPRECATED bool getContentByDecodedUrl(const string& url,
string& content,
string& title,
unsigned int& contentLength,
string& contentType,
string& baseUrl) const;
/**
* Get the content of an entry specified by a url.
*
* Equivalent to getContentByDecodedUrl but withou the baseUrl.
*/
DEPRECATED bool getContentByDecodedUrl(const string& url,
string& content,
string& title,
unsigned int& contentLength,
string& contentType) const;
/**
* Search for entries with title starting with prefix (case sensitive).
*
@@ -497,16 +411,6 @@ class Reader
SuggestionsList_t& results);
/**
* Check if the url exists in the zim file.
*
* Deprecated : Use `pathExists` instead.
*
* @param url the url to check.
* @return True if the url exits in the zim file.
*/
DEPRECATED bool urlExists(const string& url) const;
/**
* Check if the path exists in the zim file.
*
@@ -562,16 +466,6 @@ class Reader
*/
bool isCorrupted() const;
/**
* Parse a full url into a namespace and url.
*
* @param[in] url The full url ("/N/url").
* @param[out] ns The namespace (N).
* @param[out] title The url (url).
* @return True
*/
DEPRECATED bool parseUrl(const string& url, char* ns, string& title) const;
/**
* Return the total size of the zim file.
*
@@ -586,24 +480,10 @@ class Reader
*
* @return The libzim file handler.
*/
zim::File* getZimFileHandler() const;
/**
* Get the zim article object associated to a url.
*
* @param[in] url The url of the article.
* @param[out] article The libzim article object.
* @return True if the url is good (article.good()).
*/
DEPRECATED bool getArticleObjectByDecodedUrl(const string& url,
zim::Article& article) const;
zim::Archive* getZimArchive() const;
protected:
zim::File* zimFileHandler;
zim::size_type firstArticleOffset;
zim::size_type lastArticleOffset;
zim::size_type nsACount;
zim::size_type nsICount;
std::shared_ptr<zim::Archive> zimArchive;
std::string zimFilePath;
SuggestionsList_t suggestions;

View File

@@ -21,6 +21,7 @@
#define KIWIX_SEARCH_RENDERER_H
#include <string>
#include <zim/search.h>
namespace kiwix
{
@@ -40,6 +41,8 @@ class SearchRenderer
* Used to generate pagination links.
*/
SearchRenderer(Searcher* searcher, NameMapper* mapper);
SearchRenderer(zim::SearchResultSet srs, NameMapper* mapper,
unsigned int start, unsigned int estimatedResultCount);
~SearchRenderer();
@@ -74,7 +77,7 @@ class SearchRenderer
protected:
std::string beautifyInteger(const unsigned int number);
Searcher* mp_searcher;
zim::SearchResultSet m_srs;
NameMapper* mp_nameMapper;
std::string searchContent;
std::string searchPattern;

View File

@@ -27,10 +27,10 @@
#include <cctype>
#include <locale>
#include <string>
#include <memory>
#include <vector>
#include <vector>
#include "tools/pathTools.h"
#include "tools/stringTools.h"
#include <zim/search.h>
using namespace std;
@@ -48,10 +48,11 @@ class Result
virtual std::string get_content() = 0;
virtual int get_wordCount() = 0;
virtual int get_size() = 0;
virtual int get_readerIndex() = 0;
virtual std::string get_zimId() = 0;
};
struct SearcherInternal;
struct SuggestionInternal;
/**
* The Searcher class is reponsible to do different kind of search using the
* fulltext index.
@@ -85,12 +86,12 @@ class Searcher
*
* @param search The search query.
* @param resultStart the start offset of the search results (used for pagination).
* @param resultEnd the end offset of the search results (used for pagination).
* @param maxResultCount Maximum results to get from start (used for pagination).
* @param verbose print some info on stdout if true.
*/
void search(const std::string& search,
unsigned int resultStart,
unsigned int resultEnd,
unsigned int maxResultCount,
const bool verbose = false);
/**
@@ -104,12 +105,12 @@ class Searcher
* @param longitude The longitude of the center point.
* @param distance The radius of the disc.
* @param resultStart the start offset of the search results (used for pagination).
* @param resultEnd the end offset of the search results (used for pagination).
* @param maxResultCount Maximum number of results to get from start (used for pagination).
* @param verbose print some info on stdout if true.
*/
void geo_search(float latitude, float longitude, float distance,
unsigned int resultStart,
unsigned int resultEnd,
unsigned int maxResultCount,
const bool verbose = false);
/**
@@ -142,23 +143,29 @@ class Searcher
*/
unsigned int getEstimatedResultCount();
/**
* Get a SearchResultSet object for current search
*/
zim::SearchResultSet getSearchResultSet();
unsigned int getResultStart() { return resultStart; }
unsigned int getResultEnd() { return resultEnd; }
unsigned int getMaxResultCount() { return maxResultCount; }
protected:
std::string beautifyInteger(const unsigned int number);
void closeIndex();
void searchInIndex(string& search,
const unsigned int resultStart,
const unsigned int resultEnd,
const unsigned int maxResultCount,
const bool verbose = false);
std::vector<Reader*> readers;
SearcherInternal* internal;
std::unique_ptr<SearcherInternal> internal;
std::unique_ptr<SuggestionInternal> suggestionInternal;
std::string searchPattern;
unsigned int estimatedResultCount;
unsigned int resultStart;
unsigned int resultEnd;
unsigned int maxResultCount;
private:
void reset();

View File

@@ -55,6 +55,7 @@ namespace kiwix
void setPort(int port) { m_port = port; }
void setNbThreads(int threads) { m_nbThreads = threads; }
void setVerbose(bool verbose) { m_verbose = verbose; }
void setIndexTemplateString(const std::string& indexTemplateString) { m_indexTemplateString = indexTemplateString; }
void setTaskbar(bool withTaskbar, bool withLibraryButton)
{ m_withTaskbar = withTaskbar; m_withLibraryButton = withLibraryButton; }
void setBlockExternalLinks(bool blockExternalLinks)
@@ -65,6 +66,7 @@ namespace kiwix
NameMapper* mp_nameMapper;
std::string m_root = "";
std::string m_addr = "";
std::string m_indexTemplateString = "";
int m_port = 80;
int m_nbThreads = 1;
bool m_verbose = false;

198
include/tools.h Normal file
View File

@@ -0,0 +1,198 @@
/*
* Copyright 2021 Matthieu Gautier <mgautier@kymeria.fr>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
* MA 02110-1301, USA.
*/
#ifndef KIWIX_TOOLS_H
#define KIWIX_TOOLS_H
#include <string>
#include <vector>
namespace kiwix {
/**
* Return the current directory.
*
* @return the current directory (utf8 encoded)
*/
std::string getCurrentDirectory();
/**
* Return the data directory.
*
* The data directory is a directory where to put data (zim files, ...)
* It depends of the platform and it may be changed by user using environment variable.
*
* The resolution order is :
* - `KIWIX_DATA_DIR` env variable (if set).
* - On Windows :
* . `$APPDATA/kiwix` if $APPDATA is set
* . `$USERPROFILE/kiwix` if $USERPROFILE is set
* - Else :
* . `$XDG_DATA_HOME/kiwix`if $XDG_DATA_HOME is set
* . `$HOME/.local/share/kiwx` if $HOWE is set
* - current directory
*
* @return the path of the data directory (utf8 encoded)
*/
std::string getDataDirectory();
/** Return the path of the executable
*
* Some application may be packaged in auto extractible archive (Appimage) and the
* real executable is different of the path of the archive.
* If `realPathOnly` is true, return the path of the real executable instead of the
* archive launched by the user.
*
* @param realPathOnly If we must return the real path of the executable.
* @return the path of the executable (utf8 encoded)
*/
std::string getExecutablePath(bool realPathOnly = false);
/** Tell if the path is a relative path.
*
* This function is provided as a small helper. It is probably better to use native tools
* to manipulate paths.
*
* @param path A utf8 encoded path.
* @return true if the path is relative.
*/
bool isRelativePath(const std::string& path);
/** Append a path to another one.
*
* This function is provided as a small helper. It is probably better to use native tools
* to manipulate paths.
*
* @param basePath the base path.
* @param relativePath a path to add to the base path, must be a relative path.
* @return The concatenation of the paths, using the right separator.
*/
std::string appendToDirectory(const std::string& basePath, const std::string& relativePath);
/** Remove the last element of a path.
*
* This function is provided as a small helper. It is probably better to use native tools
* to manipulate paths.
*
* @param path a path.
* @return The parent directory (or empty string if none).
*/
std::string removeLastPathElement(const std::string& path);
/** Get the last element of a path.
*
* This function is provided as a small helper. It is probably better to use native tools
* to manipulate paths.
*
* @param path a path.
* @return The base name of the path or empty string if none (ending with a separator).
*/
std::string getLastPathElement(const std::string& path);
/** Compute the absolute path of a relative path based on another one
*
* Equivalent to appendToDirectory followed by a normalization of the path.
*
* This function is provided as a small helper. It is probably better to use native tools
* to manipulate paths.
*
* @param path the base path (if empty, current directory is taken).
* @param relativePath the relative path.
* @return a absolute path.
*/
std::string computeAbsolutePath(const std::string& path, const std::string& relativePath);
/** Compute the relative path of a path relative to another one
*
* This function is provided as a small helper. It is probably better to use native tools
* to manipulate paths.
*
* @param path the base path.
* @param absolutePath the absolute path to find the relative path for.
* @return a relative path (pointing to absolutePath, relative to path).
*/
std::string computeRelativePath(const std::string& path, const std::string& absolutePath);
/** Sleep the current thread.
*
* This function is provided as a small helper. It is probably better to use native tools.
*
* @param milliseconds The number of milliseconds to wait for.
*/
void sleep(unsigned int milliseconds);
/** Split a string
*
* This function is provided as a small helper. It is probably better to use native tools.
*
* Assuming text = "foo:;bar;baz,oups;"
*
* split(text, ":;", true, true) => ["foo", ":", ";", "bar", ";", "baz,oups", ";"]
* split(text, ":;", true, false) => ["foo", "bar", "baz,oups"] (default)
* split(text, ":;", false, true) => ["foo", ":", "", ";", "bar", ";", "baz,oups", ";", ""]
* split(text, ":;", false, false) => ["foo", "", "bar", "baz,oups", ""]
*
* @param str The string to split.
* @param delims A string of potential delimiters.
* Each charater in the string can be a individual delimiters.
* @param dropEmpty true if empty part must be dropped from the result.
* @param keepDelim true if delimiter must be included from the result.
* @return a list of part (potentially containing delimiters)
*/
std::vector<std::string> split(const std::string& str, const std::string& delims, bool dropEmpty=true, bool keepDelim = false);
/** Convert language code from iso2 code to iso3
*
* This function is provided as a small helper. It is probably better to use native tools
* to manipulate locales.
*
* @param a2code a iso2 code string.
* @return the corresponding iso3 code.
* @throw std::out_of_range if iso2 code is not known.
*/
std::string converta2toa3(const std::string& a2code);
/** Extracts content from given file.
*
* This function provides content of a file provided it's path.
*
* @param path The absolute path provided in string format.
* @return Content of corresponding file in string format.
*/
std::string getFileContent(const std::string& path);
/** checks if file exists.
*
* This function returns boolean stating if file exists or not.
*
* @param path The absolute path provided in string format.
* @return Boolean representing if file exists or not.
*/
bool fileExists(const std::string& path);
/** provides mimetype from filename.
*
* This function provides mimetype from file-name.
*
* @param filename string containing filename.
* @return mimetype from filename in string format.
*/
std::string getMimeTypeForFile(const std::string& filename);
}
#endif // KIWIX_TOOLS_H

View File

@@ -1,46 +0,0 @@
#ifndef KIWIXLIB_TOOL_LOCK_H
#define KIWIXLIB_TOOL_LOCK_H
#include <pthread.h>
namespace kiwix {
class Lock
{
public:
explicit Lock(pthread_mutex_t* mutex) :
mp_mutex(mutex)
{
pthread_mutex_lock(mp_mutex);
}
~Lock() {
if (mp_mutex != nullptr) {
pthread_mutex_unlock(mp_mutex);
}
}
Lock(Lock && other) :
mp_mutex(other.mp_mutex)
{
other.mp_mutex = nullptr;
}
Lock & operator=(Lock && other)
{
mp_mutex = other.mp_mutex;
other.mp_mutex = nullptr;
return *this;
}
private:
pthread_mutex_t* mp_mutex;
Lock(Lock const &) = delete;
Lock & operator=(Lock const &) = delete;
};
}
#endif //KIWIXLIB_TOOL_LOCK_H

View File

@@ -1,5 +1,5 @@
project('kiwix-lib', 'cpp',
version : '9.4.1', # Also change this in android-kiwix-lib-publisher/kiwixLibAndroid/build.gradle
project('libkiwix', 'cpp',
version : '10.0.0', # Also change this in android-kiwix-lib-publisher/kiwixLibAndroid/build.gradle
license : 'GPLv3+',
default_options : ['c_std=c11', 'cpp_std=c++11', 'werror=true'])
@@ -18,18 +18,23 @@ if wrapper.contains('java')
add_languages('java')
endif
# See https://github.com/kiwix/kiwix-lib/issues/371
# See https://github.com/kiwix/libkiwix/issues/371
if ['arm', 'mips', 'm68k', 'ppc', 'sh4'].contains(target_machine.cpu_family())
extra_libs += '-latomic'
endif
thread_dep = dependency('threads')
if (compiler.get_id() == 'gcc' and build_machine.system() == 'linux') or target_machine.system() == 'freebsd'
# C++ std::thread is implemented using pthread on linux by gcc
thread_dep = dependency('threads')
else
thread_dep = dependency('', required:false)
endif
libicu_dep = dependency('icu-i18n', static:static_deps)
libzim_dep = dependency('libzim', version : '>=6.3.0', static:static_deps)
pugixml_dep = dependency('pugixml', static:static_deps)
libcurl_dep = dependency('libcurl', static:static_deps)
microhttpd_dep = dependency('libmicrohttpd', static:static_deps)
zlib_dep = dependency('zlib', static:static_deps)
xapian_dep = dependency('xapian-core', static:static_deps)
if compiler.has_header('mustache.hpp')
extra_include = []
@@ -39,13 +44,23 @@ else
error('Cannot found header mustache.hpp')
endif
libzim_dep = dependency('libzim', version : '>=7.0.0', static:static_deps)
if not compiler.has_header_symbol('zim/zim.h', 'LIBZIM_WITH_XAPIAN')
error('Libzim seems to be compiled without xapian. Xapian support is mandatory.')
endif
extra_cflags = ''
if target_machine.system() == 'windows' and static_deps
add_project_arguments('-DCURL_STATICLIB', language : 'cpp')
extra_cflags += '-DCURL_STATICLIB'
endif
all_deps = [thread_dep, libicu_dep, libzim_dep, pugixml_dep, libcurl_dep, microhttpd_dep, zlib_dep]
if target_machine.system() == 'windows'
add_project_arguments('-DNOMINMAX', language: 'cpp')
endif
all_deps = [thread_dep, libicu_dep, libzim_dep, pugixml_dep, libcurl_dep, microhttpd_dep, zlib_dep, xapian_dep]
inc = include_directories('include', extra_include)
@@ -64,7 +79,7 @@ subdir('static')
subdir('src')
subdir('test')
pkg_requires = ['libzim', 'icu-i18n', 'pugixml', 'libcurl', 'libmicrohttpd']
pkg_requires = ['libzim', 'icu-i18n', 'pugixml', 'libcurl', 'libmicrohttpd', 'xapian-core']
pkg_conf = configuration_data()
pkg_conf.set('prefix', get_option('prefix'))

View File

@@ -6,10 +6,10 @@
#include <sstream>
#include <thread>
#include <chrono>
#include <tools/otherTools.h>
#include <tools/pathTools.h>
#include <tools/stringTools.h>
#include <downloader.h> // For AriaError
#include "tools.h"
#include "tools/pathTools.h"
#include "tools/stringTools.h"
#include "downloader.h" // For AriaError
#ifdef _WIN32
# define ARIA2_CMD "aria2c.exe"
@@ -63,6 +63,7 @@ Aria2::Aria2():
// Try to use a potential installed aria2c.
callCmd.push_back(ARIA2_CMD);
}
callCmd.push_back("--follow-metalink=mem");
callCmd.push_back("--enable-rpc");
callCmd.push_back(rpc_secret.c_str());
callCmd.push_back(rpc_port.c_str());

View File

@@ -20,10 +20,16 @@
#include "book.h"
#include "reader.h"
#include "tools.h"
#include "tools/base64.h"
#include "tools/regexTools.h"
#include "tools/networkTools.h"
#include "tools/otherTools.h"
#include "tools/stringTools.h"
#include "tools/pathTools.h"
#include "tools/archiveTools.h"
#include <zim/archive.h>
#include <pugixml.hpp>
@@ -61,6 +67,7 @@ bool Book::update(const kiwix::Book& other)
m_name = other.m_name;
m_flavour = other.m_flavour;
m_tags = other.m_tags;
m_category = other.m_category;
m_origId = other.m_origId;
m_articleCount = other.m_articleCount;
m_mediaCount = other.m_mediaCount;
@@ -76,25 +83,28 @@ bool Book::update(const kiwix::Book& other)
void Book::update(const kiwix::Reader& reader)
{
m_path = reader.getZimFilePath();
m_pathValid = true;
m_id = reader.getId();
m_title = reader.getTitle();
m_description = reader.getDescription();
m_language = reader.getLanguage();
m_creator = reader.getCreator();
m_publisher = reader.getPublisher();
m_date = reader.getDate();
m_name = reader.getName();
m_flavour = reader.getFlavour();
m_tags = reader.getTags();
m_origId = reader.getOrigId();
m_articleCount = reader.getArticleCount();
m_mediaCount = reader.getMediaCount();
m_size = static_cast<uint64_t>(reader.getFileSize()) << 10;
m_pathValid = true;
update(*reader.getZimArchive());
}
reader.getFavicon(m_favicon, m_faviconMimeType);
void Book::update(const zim::Archive& archive) {
m_path = archive.getFilename();
m_pathValid = true;
m_id = getArchiveId(archive);
m_title = getArchiveTitle(archive);
m_description = getMetaDescription(archive);
m_language = getMetaLanguage(archive);
m_creator = getMetaCreator(archive);
m_publisher = getMetaPublisher(archive);
m_date = getMetaDate(archive);
m_name = getMetaName(archive);
m_flavour = getMetaFlavour(archive);
m_tags = getMetaTags(archive);
m_category = getCategoryFromTags();
m_articleCount = archive.getArticleCount();
m_mediaCount = getArchiveMediaCount(archive);
m_size = static_cast<uint64_t>(getArchiveFileSize(archive)) << 10;
getArchiveFavicon(archive, 48, m_favicon, m_faviconMimeType);
}
#define ATTR(name) node.attribute(name).value()
@@ -127,6 +137,8 @@ void Book::updateFromXml(const pugi::xml_node& node, const std::string& baseDir)
try {
m_downloadId = ATTR("downloadId");
} catch(...) {}
const auto catattr = node.attribute("category");
m_category = catattr.empty() ? getCategoryFromTags() : catattr.value();
}
#undef ATTR
@@ -156,6 +168,8 @@ void Book::updateFromOpds(const pugi::xml_node& node, const std::string& urlHost
m_name = VALUE("name");
m_flavour = VALUE("flavour");
m_tags = VALUE("tags");
const auto catnode = node.child("category");
m_category = catnode.empty() ? getCategoryFromTags() : catnode.child_value();
m_articleCount = strtoull(VALUE("articleCount"), 0, 0);
m_mediaCount = strtoull(VALUE("mediaCount"), 0, 0);
for(auto linkNode = node.child("link"); linkNode;
@@ -220,4 +234,21 @@ bool Book::getTagBool(const std::string& tagName) const {
return convertStrToBool(getTagStr(tagName));
}
std::string Book::getCategory() const
{
return m_category;
}
std::string Book::getCategoryFromTags() const
{
try
{
return getTagStr("category");
}
catch ( const std::out_of_range& )
{
return "";
}
}
}

View File

@@ -1,5 +1,5 @@
/*
* Copyright 2011 Emmanuel Engelhart <kelson@kiwix.org>
* Copyright 2018-2020 Matthieu Gautier <mgautier@kymeria.fr>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
@@ -20,121 +20,54 @@
#include "reader.h"
#include <time.h>
#include <zim/search.h>
namespace kiwix
{
Entry::Entry(zim::Article article)
: article(article)
Entry::Entry(zim::Entry entry)
: entry(entry)
{
}
#define RETURN_IF_INVALID(WHAT) if(!good()) { return (WHAT); }
std::string Entry::getPath() const
{
RETURN_IF_INVALID("");
return article.getLongUrl();
}
std::string Entry::getTitle() const
{
RETURN_IF_INVALID("");
return article.getTitle();
}
std::string Entry::getContent() const
{
RETURN_IF_INVALID("");
return article.getData();
}
zim::Blob Entry::getBlob(offset_type offset) const
{
RETURN_IF_INVALID(zim::Blob());
return article.getData(offset);
}
zim::Blob Entry::getBlob(offset_type offset, size_type size) const
{
RETURN_IF_INVALID(zim::Blob());
return article.getData(offset, size);
}
std::pair<std::string, offset_type> Entry::getDirectAccessInfo() const
{
RETURN_IF_INVALID(std::make_pair("", 0));
return article.getDirectAccessInformation();
}
size_type Entry::getSize() const
{
RETURN_IF_INVALID(0);
return article.getArticleSize();
if (entry.isRedirect()) {
return 0;
} else {
return entry.getItem().getSize();
}
}
std::string Entry::getMimetype() const
{
RETURN_IF_INVALID("");
try {
return article.getMimeType();
} catch (exception& e) {
return "application/octet-stream";
}
return entry.getItem(true).getMimetype();
}
bool Entry::isRedirect() const
{
RETURN_IF_INVALID(false);
return article.isRedirect();
}
bool Entry::isLinkTarget() const
{
RETURN_IF_INVALID(false);
return article.isLinktarget();
}
bool Entry::isDeleted() const
{
RETURN_IF_INVALID(false);
return article.isDeleted();
return entry.isRedirect();
}
Entry Entry::getRedirectEntry() const
{
RETURN_IF_INVALID(Entry());
if ( !article.isRedirect() ) {
if ( !entry.isRedirect() ) {
throw NoEntry();
}
auto targeted_article = article.getRedirectArticle();
if ( !targeted_article.good()) {
throw NoEntry();
}
return targeted_article;
return entry.getRedirectEntry();
}
Entry Entry::getFinalEntry() const
{
RETURN_IF_INVALID(Entry());
if (final_article.good()) {
return final_article;
}
int loopCounter = 42;
final_article = article;
while (final_article.isRedirect() && loopCounter--) {
final_article = final_article.getRedirectArticle();
if ( !final_article.good()) {
throw NoEntry();
}
auto final_entry = entry;
while (final_entry.isRedirect() && loopCounter--) {
final_entry = final_entry.getRedirectEntry();
}
// Prevent infinite loops.
if (final_article.isRedirect()) {
if (final_entry.isRedirect()) {
throw NoEntry();
}
return final_article;
return final_entry;
}
}

View File

@@ -9,6 +9,7 @@
# include <unistd.h>
#endif
#include "tools.h"
#include "tools/pathTools.h"
#include "tools/stringTools.h"

View File

@@ -22,6 +22,7 @@
#include "reader.h"
#include "libxml_dumper.h"
#include "tools.h"
#include "tools/base64.h"
#include "tools/regexTools.h"
#include "tools/pathTools.h"
@@ -30,14 +31,42 @@
#include <pugixml.hpp>
#include <algorithm>
#include <set>
#include <unicode/locid.h>
#include <xapian.h>
namespace kiwix
{
namespace
{
std::string iso639_3ToXapian(const std::string& lang) {
return icu::Locale(lang.c_str()).getLanguage();
};
std::string normalizeText(const std::string& text)
{
return removeAccents(text);
}
} // unnamed namespace
class Library::BookDB : public Xapian::WritableDatabase
{
public:
BookDB() : Xapian::WritableDatabase("", Xapian::DB_BACKEND_INMEMORY) {}
};
/* Constructor */
Library::Library()
: m_bookDB(new BookDB)
{
}
Library::Library(Library&& ) = default;
Library& Library::operator=(Library&& ) = default;
/* Destructor */
Library::~Library()
{
@@ -47,6 +76,7 @@ Library::~Library()
bool Library::addBook(const Book& book)
{
/* Try to find it */
updateBookDB(book);
try {
auto& oldbook = m_books.at(book.getId());
oldbook.update(book);
@@ -77,15 +107,24 @@ bool Library::removeBookmark(const std::string& zimId, const std::string& url)
bool Library::removeBookById(const std::string& id)
{
m_bookDB->delete_document("Q" + id);
m_readers.erase(id);
m_archives.erase(id);
return m_books.erase(id) == 1;
}
Book& Library::getBookById(const std::string& id)
const Book& Library::getBookById(const std::string& id) const
{
return m_books.at(id);
}
Book& Library::getBookByPath(const std::string& path)
Book& Library::getBookById(const std::string& id)
{
const Library& const_self = *this;
return const_cast<Book&>(const_self.getBookById(id));
}
const Book& Library::getBookByPath(const std::string& path) const
{
for(auto& it: m_books) {
auto& book = it.second;
@@ -97,22 +136,52 @@ Book& Library::getBookByPath(const std::string& path)
throw std::out_of_range(ss.str());
}
Book& Library::getBookByPath(const std::string& path)
{
const Library& const_self = *this;
return const_cast<Book&>(const_self.getBookByPath(path));
}
std::shared_ptr<Reader> Library::getReaderById(const std::string& id)
{
try {
return m_readers.at(id);
} catch (std::out_of_range& e) {}
try {
auto reader = make_shared<Reader>(m_archives.at(id));
m_readers[id] = reader;
return reader;
} catch (std::out_of_range& e) {}
auto book = getBookById(id);
if (!book.isPathValid())
return nullptr;
auto sptr = make_shared<Reader>(book.getPath());
m_readers[id] = sptr;
auto archive = make_shared<zim::Archive>(book.getPath());
m_archives[id] = archive;
auto reader = make_shared<Reader>(archive);
m_readers[id] = reader;
return reader;
}
std::shared_ptr<zim::Archive> Library::getArchiveById(const std::string& id)
{
try {
return m_archives.at(id);
} catch (std::out_of_range& e) {}
auto book = getBookById(id);
if (!book.isPathValid())
return nullptr;
auto sptr = make_shared<zim::Archive>(book.getPath());
m_archives[id] = sptr;
return sptr;
}
unsigned int Library::getBookCount(const bool localBooks,
const bool remoteBooks)
const bool remoteBooks) const
{
unsigned int result = 0;
for (auto& pair: m_books) {
@@ -125,7 +194,7 @@ unsigned int Library::getBookCount(const bool localBooks,
return result;
}
bool Library::writeToFile(const std::string& path)
bool Library::writeToFile(const std::string& path) const
{
auto baseDir = removeLastPathElement(path);
LibXMLDumper dumper(this);
@@ -133,70 +202,70 @@ bool Library::writeToFile(const std::string& path)
return writeTextFile(path, dumper.dumpLibXMLContent(getBooksIds()));
}
bool Library::writeBookmarksToFile(const std::string& path)
bool Library::writeBookmarksToFile(const std::string& path) const
{
LibXMLDumper dumper(this);
return writeTextFile(path, dumper.dumpLibXMLBookmark());
}
std::vector<std::string> Library::getBooksLanguages()
Library::AttributeCounts Library::getBookAttributeCounts(BookStrPropMemFn p) const
{
std::vector<std::string> booksLanguages;
std::map<std::string, bool> booksLanguagesMap;
AttributeCounts propValueCounts;
for (auto& pair: m_books) {
auto& book = pair.second;
auto& language = book.getLanguage();
if (booksLanguagesMap.find(language) == booksLanguagesMap.end()) {
if (book.getOrigId().empty()) {
booksLanguagesMap[language] = true;
booksLanguages.push_back(language);
}
for (const auto& pair: m_books) {
const auto& book = pair.second;
if (book.getOrigId().empty()) {
propValueCounts[(book.*p)()] += 1;
}
}
return propValueCounts;
}
std::vector<std::string> Library::getBookPropValueSet(BookStrPropMemFn p) const
{
std::vector<std::string> result;
for ( const auto& kv : getBookAttributeCounts(p) ) {
result.push_back(kv.first);
}
return result;
}
std::vector<std::string> Library::getBooksLanguages() const
{
return getBookPropValueSet(&Book::getLanguage);
}
Library::AttributeCounts Library::getBooksLanguagesWithCounts() const
{
return getBookAttributeCounts(&Book::getLanguage);
}
std::vector<std::string> Library::getBooksCategories() const
{
std::set<std::string> categories;
for (const auto& pair: m_books) {
const auto& book = pair.second;
const auto& c = book.getCategory();
if ( !c.empty() ) {
categories.insert(c);
}
}
return booksLanguages;
return std::vector<std::string>(categories.begin(), categories.end());
}
std::vector<std::string> Library::getBooksCreators()
std::vector<std::string> Library::getBooksCreators() const
{
std::vector<std::string> booksCreators;
std::map<std::string, bool> booksCreatorsMap;
for (auto& pair: m_books) {
auto& book = pair.second;
auto& creator = book.getCreator();
if (booksCreatorsMap.find(creator) == booksCreatorsMap.end()) {
if (book.getOrigId().empty()) {
booksCreatorsMap[creator] = true;
booksCreators.push_back(creator);
}
}
}
return booksCreators;
return getBookPropValueSet(&Book::getCreator);
}
std::vector<std::string> Library::getBooksPublishers()
std::vector<std::string> Library::getBooksPublishers() const
{
std::vector<std::string> booksPublishers;
std::map<std::string, bool> booksPublishersMap;
for (auto& pair:m_books) {
auto& book = pair.second;
auto& publisher = book.getPublisher();
if (booksPublishersMap.find(publisher) == booksPublishersMap.end()) {
if (book.getOrigId().empty()) {
booksPublishersMap[publisher] = true;
booksPublishers.push_back(publisher);
}
}
}
return booksPublishers;
return getBookPropValueSet(&Book::getPublisher);
}
const std::vector<kiwix::Bookmark> Library::getBookmarks(bool onlyValidBookmarks)
const std::vector<kiwix::Bookmark> Library::getBookmarks(bool onlyValidBookmarks) const
{
if (!onlyValidBookmarks) {
return m_bookmarks;
@@ -211,9 +280,9 @@ const std::vector<kiwix::Bookmark> Library::getBookmarks(bool onlyValidBookmarks
return validBookmarks;
}
std::vector<std::string> Library::getBooksIds()
Library::BookIdCollection Library::getBooksIds() const
{
std::vector<std::string> bookIds;
BookIdCollection bookIds;
for (auto& pair: m_books) {
bookIds.push_back(pair.first);
@@ -222,7 +291,7 @@ std::vector<std::string> Library::getBooksIds()
return bookIds;
}
std::vector<std::string> Library::filter(const std::string& search)
Library::BookIdCollection Library::filter(const std::string& search) const
{
if (search.empty()) {
return getBooksIds();
@@ -232,16 +301,195 @@ std::vector<std::string> Library::filter(const std::string& search)
}
std::vector<std::string> Library::filter(const Filter& filter)
void Library::updateBookDB(const Book& book)
{
std::vector<std::string> bookIds;
for(auto& pair:m_books) {
auto book = pair.second;
if(filter.accept(book)) {
bookIds.push_back(pair.first);
Xapian::Stem stemmer;
Xapian::TermGenerator indexer;
const std::string lang = book.getLanguage();
try {
stemmer = Xapian::Stem(iso639_3ToXapian(lang));
indexer.set_stemmer(stemmer);
indexer.set_stemming_strategy(Xapian::TermGenerator::STEM_SOME);
} catch (...) {}
Xapian::Document doc;
indexer.set_document(doc);
const std::string title = normalizeText(book.getTitle());
const std::string desc = normalizeText(book.getDescription());
// Index title and description without prefixes for general search
indexer.index_text(title);
indexer.increase_termpos();
indexer.index_text(desc);
// Index all fields for field-based search
indexer.index_text(title, 1, "S");
indexer.index_text(desc, 1, "XD");
indexer.index_text(lang, 1, "L");
indexer.index_text(normalizeText(book.getCreator()), 1, "A");
indexer.index_text(normalizeText(book.getPublisher()), 1, "XP");
indexer.index_text(normalizeText(book.getName()), 1, "XN");
indexer.index_text(normalizeText(book.getCategory()), 1, "XC");
for ( const auto& tag : split(normalizeText(book.getTags()), ";") )
doc.add_boolean_term("XT" + tag);
const std::string idterm = "Q" + book.getId();
doc.add_boolean_term(idterm);
doc.set_data(book.getId());
m_bookDB->replace_document(idterm, doc);
}
namespace
{
bool willSelectEverything(const Xapian::Query& query)
{
return query.get_type() == Xapian::Query::LEAF_MATCH_ALL;
}
Xapian::Query buildXapianQueryFromFilterQuery(const Filter& filter)
{
if ( !filter.hasQuery() || filter.getQuery().empty() ) {
// This is a thread-safe way to construct an equivalent of
// a Xapian::Query::MatchAll query
return Xapian::Query(std::string());
}
Xapian::QueryParser queryParser;
queryParser.set_default_op(Xapian::Query::OP_AND);
queryParser.add_prefix("title", "S");
queryParser.add_prefix("description", "XD");
queryParser.add_prefix("name", "XN");
queryParser.add_prefix("category", "XC");
queryParser.add_prefix("lang", "L");
queryParser.add_prefix("publisher", "XP");
queryParser.add_prefix("creator", "A");
queryParser.add_prefix("tag", "XT");
const auto partialQueryFlag = filter.queryIsPartial()
? Xapian::QueryParser::FLAG_PARTIAL
: 0;
// Language assumed for the query is not known for sure so stemming
// is not applied
//queryParser.set_stemmer(Xapian::Stem(iso639_3ToXapian(???)));
//queryParser.set_stemming_strategy(Xapian::QueryParser::STEM_SOME);
const auto flags = Xapian::QueryParser::FLAG_PHRASE
| Xapian::QueryParser::FLAG_BOOLEAN
| Xapian::QueryParser::FLAG_BOOLEAN_ANY_CASE
| Xapian::QueryParser::FLAG_LOVEHATE
| Xapian::QueryParser::FLAG_WILDCARD
| partialQueryFlag;
return queryParser.parse_query(normalizeText(filter.getQuery()), flags);
}
Xapian::Query nameQuery(const std::string& name)
{
return Xapian::Query("XN" + normalizeText(name));
}
Xapian::Query categoryQuery(const std::string& category)
{
return Xapian::Query("XC" + normalizeText(category));
}
Xapian::Query langQuery(const std::string& lang)
{
return Xapian::Query("L" + normalizeText(lang));
}
Xapian::Query publisherQuery(const std::string& publisher)
{
Xapian::QueryParser queryParser;
queryParser.set_default_op(Xapian::Query::OP_OR);
queryParser.set_stemming_strategy(Xapian::QueryParser::STEM_NONE);
const auto flags = 0;
const auto q = queryParser.parse_query(normalizeText(publisher), flags, "XP");
return Xapian::Query(Xapian::Query::OP_PHRASE, q.get_terms_begin(), q.get_terms_end(), q.get_length());
}
Xapian::Query creatorQuery(const std::string& creator)
{
Xapian::QueryParser queryParser;
queryParser.set_default_op(Xapian::Query::OP_OR);
queryParser.set_stemming_strategy(Xapian::QueryParser::STEM_NONE);
const auto flags = 0;
const auto q = queryParser.parse_query(normalizeText(creator), flags, "A");
return Xapian::Query(Xapian::Query::OP_PHRASE, q.get_terms_begin(), q.get_terms_end(), q.get_length());
}
Xapian::Query tagsQuery(const Filter::Tags& acceptTags, const Filter::Tags& rejectTags)
{
Xapian::Query q = Xapian::Query(std::string());
if (!acceptTags.empty()) {
for ( const auto& tag : acceptTags )
q &= Xapian::Query("XT" + normalizeText(tag));
}
if (!rejectTags.empty()) {
for ( const auto& tag : rejectTags )
q = Xapian::Query(Xapian::Query::OP_AND_NOT, q, "XT" + normalizeText(tag));
}
return q;
}
Xapian::Query buildXapianQuery(const Filter& filter)
{
auto q = buildXapianQueryFromFilterQuery(filter);
if ( filter.hasName() ) {
q = Xapian::Query(Xapian::Query::OP_AND, q, nameQuery(filter.getName()));
}
if ( filter.hasCategory() ) {
q = Xapian::Query(Xapian::Query::OP_AND, q, categoryQuery(filter.getCategory()));
}
if ( filter.hasLang() ) {
q = Xapian::Query(Xapian::Query::OP_AND, q, langQuery(filter.getLang()));
}
if ( filter.hasPublisher() ) {
q = Xapian::Query(Xapian::Query::OP_AND, q, publisherQuery(filter.getPublisher()));
}
if ( filter.hasCreator() ) {
q = Xapian::Query(Xapian::Query::OP_AND, q, creatorQuery(filter.getCreator()));
}
if ( !filter.getAcceptTags().empty() || !filter.getRejectTags().empty() ) {
const auto tq = tagsQuery(filter.getAcceptTags(), filter.getRejectTags());
q = Xapian::Query(Xapian::Query::OP_AND, q, tq);;
}
return q;
}
} // unnamed namespace
Library::BookIdCollection Library::filterViaBookDB(const Filter& filter) const
{
const auto query = buildXapianQuery(filter);
if ( willSelectEverything(query) )
return getBooksIds();
BookIdCollection bookIds;
Xapian::Enquire enquire(*m_bookDB);
enquire.set_query(query);
const auto results = enquire.get_mset(0, m_books.size());
for ( auto it = results.begin(); it != results.end(); ++it ) {
bookIds.push_back(it.get_document().get_data());
}
return bookIds;
}
Library::BookIdCollection Library::filter(const Filter& filter) const
{
BookIdCollection result;
for(auto id : filterViaBookDB(filter)) {
if(filter.accept(m_books.at(id))) {
result.push_back(id);
}
}
return bookIds;
return result;
}
template<supportedListSortBy SORT>
@@ -257,13 +505,13 @@ struct KEY_TYPE<SIZE> {
template<supportedListSortBy sort>
class Comparator {
private:
Library* lib;
bool ascending;
const Library* const lib;
const bool ascending;
inline typename KEY_TYPE<sort>::TYPE get_key(const std::string& id);
public:
Comparator(Library* lib, bool ascending) : lib(lib), ascending(ascending) {}
Comparator(const Library* lib, bool ascending) : lib(lib), ascending(ascending) {}
inline bool operator() (const std::string& id1, const std::string& id2) {
if (ascending) {
return get_key(id1) < get_key(id2);
@@ -303,7 +551,7 @@ std::string Comparator<PUBLISHER>::get_key(const std::string& id)
return lib->getBookById(id).getPublisher();
}
void Library::sort(std::vector<std::string>& bookIds, supportedListSortBy sort, bool ascending)
void Library::sort(BookIdCollection& bookIds, supportedListSortBy sort, bool ascending) const
{
switch(sort) {
case TITLE:
@@ -327,7 +575,7 @@ void Library::sort(std::vector<std::string>& bookIds, supportedListSortBy sort,
}
std::vector<std::string> Library::listBooksIds(
Library::BookIdCollection Library::listBooksIds(
int mode,
supportedListSortBy sortBy,
const std::string& search,
@@ -335,7 +583,7 @@ std::vector<std::string> Library::listBooksIds(
const std::string& creator,
const std::string& publisher,
const std::vector<std::string>& tags,
size_t maxSize) {
size_t maxSize) const {
Filter _filter;
if (mode & LOCAL)
@@ -391,6 +639,7 @@ enum filterTypes {
MAXSIZE = FLAG(11),
QUERY = FLAG(12),
NAME = FLAG(13),
CATEGORY = FLAG(14),
};
Filter& Filter::local(bool accept)
@@ -429,20 +678,27 @@ Filter& Filter::valid(bool accept)
return *this;
}
Filter& Filter::acceptTags(std::vector<std::string> tags)
Filter& Filter::acceptTags(const Tags& tags)
{
_acceptTags = tags;
activeFilters |= ACCEPTTAGS;
return *this;
}
Filter& Filter::rejectTags(std::vector<std::string> tags)
Filter& Filter::rejectTags(const Tags& tags)
{
_rejectTags = tags;
activeFilters |= REJECTTAGS;
return *this;
}
Filter& Filter::category(std::string category)
{
_category = category;
activeFilters |= CATEGORY;
return *this;
}
Filter& Filter::lang(std::string lang)
{
_lang = lang;
@@ -471,9 +727,10 @@ Filter& Filter::maxSize(size_t maxSize)
return *this;
}
Filter& Filter::query(std::string query)
Filter& Filter::query(std::string query, bool partial)
{
_query = query;
_queryIsPartial = partial;
activeFilters |= QUERY;
return *this;
}
@@ -487,6 +744,36 @@ Filter& Filter::name(std::string name)
#define ACTIVE(X) (activeFilters & (X))
#define FILTER(TAG, TEST) if (ACTIVE(TAG) && !(TEST)) { return false; }
bool Filter::hasQuery() const
{
return ACTIVE(QUERY);
}
bool Filter::hasName() const
{
return ACTIVE(NAME);
}
bool Filter::hasCategory() const
{
return ACTIVE(CATEGORY);
}
bool Filter::hasLang() const
{
return ACTIVE(LANG);
}
bool Filter::hasPublisher() const
{
return ACTIVE(_PUBLISHER);
}
bool Filter::hasCreator() const
{
return ACTIVE(_CREATOR);
}
bool Filter::accept(const Book& book) const
{
auto local = !book.getPath().empty();
@@ -502,40 +789,8 @@ bool Filter::accept(const Book& book) const
FILTER(_NOREMOTE, !remote)
FILTER(MAXSIZE, book.getSize() <= _maxSize)
FILTER(LANG, book.getLanguage() == _lang)
FILTER(_PUBLISHER, book.getPublisher() == _publisher)
FILTER(_CREATOR, book.getCreator() == _creator)
FILTER(NAME, book.getName() == _name)
if (ACTIVE(ACCEPTTAGS)) {
if (!_acceptTags.empty()) {
auto vBookTags = split(book.getTags(), ";");
std::set<std::string> sBookTags(vBookTags.begin(), vBookTags.end());
for (auto& t: _acceptTags) {
if (sBookTags.find(t) == sBookTags.end()) {
return false;
}
}
}
}
if (ACTIVE(REJECTTAGS)) {
if (!_rejectTags.empty()) {
auto vBookTags = split(book.getTags(), ";");
std::set<std::string> sBookTags(vBookTags.begin(), vBookTags.end());
for (auto& t: _rejectTags) {
if (sBookTags.find(t) != sBookTags.end()) {
return false;
}
}
}
}
if ( ACTIVE(QUERY)
&& !(matchRegex(book.getTitle(), "\\Q" + _query + "\\E")
|| matchRegex(book.getDescription(), "\\Q" + _query + "\\E")))
return false;
return true;
}
}

View File

@@ -20,15 +20,15 @@
#include "libxml_dumper.h"
#include "book.h"
#include "tools.h"
#include "tools/base64.h"
#include "tools/stringTools.h"
#include "tools/otherTools.h"
#include "tools/pathTools.h"
namespace kiwix
{
/* Constructor */
LibXMLDumper::LibXMLDumper(Library* library)
LibXMLDumper::LibXMLDumper(const Library* library)
: library(library)
{
}

View File

@@ -19,6 +19,7 @@
#include "manager.h"
#include "tools.h"
#include "tools/pathTools.h"
#include <pugixml.hpp>
@@ -80,7 +81,7 @@ bool Manager::readXml(const std::string& xml,
{
pugi::xml_document doc;
pugi::xml_parse_result result
= doc.load_buffer_inplace((void*)xml.data(), xml.size());
= doc.load_buffer((void*)xml.data(), xml.size());
if (result) {
this->parseXmlDom(doc, readOnly, libraryPath, trustLibrary);
@@ -124,7 +125,7 @@ bool Manager::readOpds(const std::string& content, const std::string& urlHost)
{
pugi::xml_document doc;
pugi::xml_parse_result result
= doc.load_buffer_inplace((void*)content.data(), content.size());
= doc.load_buffer((void*)content.data(), content.size());
if (result) {
this->parseOpdsDom(doc, urlHost);
@@ -214,8 +215,8 @@ bool Manager::readBookFromPath(const std::string& path, kiwix::Book* book)
tmp_path = computeAbsolutePath(getCurrentDirectory(), path);
}
try {
kiwix::Reader reader(tmp_path);
book->update(reader);
zim::Archive archive(tmp_path);
book->update(archive);
book->setPathValid(true);
} catch (const std::exception& e) {
book->setPathValid(false);

View File

@@ -19,13 +19,16 @@ kiwix_sources = [
'tools/stringTools.cpp',
'tools/networkTools.cpp',
'tools/otherTools.cpp',
'tools/archiveTools.cpp',
'kiwixserve.cpp',
'name_mapper.cpp',
'server/byte_range.cpp',
'server/etag.cpp',
'server/request_context.cpp',
'server/response.cpp',
'server/internalServer.cpp'
'server/internalServer.cpp',
'server/internalServer_catalog_v2.cpp',
'opds_catalog.cpp'
]
kiwix_sources += lib_resources

74
src/opds_catalog.cpp Normal file
View File

@@ -0,0 +1,74 @@
/*
* Copyright 2021 Veloman Yunkan <veloman.yunkan@gmail.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
* MA 02110-1301, USA.
*/
#include "opds_catalog.h"
#include "tools/stringTools.h"
#include <sstream>
namespace kiwix
{
namespace
{
const char opdsSearchEndpoint[] = "/catalog/v2/entries";
enum Separator { AMP };
std::ostringstream& operator<<(std::ostringstream& oss, Separator sep)
{
if ( oss.tellp() > 0 )
oss << "&";
return oss;
}
std::string buildSearchString(const Filter& f)
{
std::ostringstream oss;
if ( f.hasQuery() )
oss << AMP << "q=" << urlEncode(f.getQuery());
if ( f.hasCategory() )
oss << AMP << "category=" << urlEncode(f.getCategory());
if ( f.hasLang() )
oss << AMP << "lang=" << urlEncode(f.getLang());
if ( f.hasName() )
oss << AMP << "name=" << urlEncode(f.getName());
if ( !f.getAcceptTags().empty() )
oss << AMP << "tag=" << urlEncode(join(f.getAcceptTags(), ";"));
return oss.str();
}
} // unnamed namespace
std::string getSearchUrl(const Filter& f)
{
const std::string searchString = buildSearchString(f);
if ( searchString.empty() )
return opdsSearchEndpoint;
else
return opdsSearchEndpoint + ("?" + searchString);
}
} // namespace kiwix

View File

@@ -20,11 +20,16 @@
#include "opds_dumper.h"
#include "book.h"
#include "kiwixlib-resources.h"
#include <mustache.hpp>
#include <unicode/locid.h>
#include "tools/stringTools.h"
#include "tools/otherTools.h"
#include <iomanip>
namespace kiwix
{
/* Constructor */
OPDSDumper::OPDSDumper(Library* library)
: library(library)
@@ -35,120 +40,189 @@ OPDSDumper::~OPDSDumper()
{
}
std::string gen_date_str()
{
auto now = time(0);
auto tm = localtime(&now);
std::stringstream is;
is << std::setw(2) << std::setfill('0')
<< 1900+tm->tm_year << "-"
<< std::setw(2) << std::setfill('0') << tm->tm_mon << "-"
<< std::setw(2) << std::setfill('0') << tm->tm_mday << "T"
<< std::setw(2) << std::setfill('0') << tm->tm_hour << ":"
<< std::setw(2) << std::setfill('0') << tm->tm_min << ":"
<< std::setw(2) << std::setfill('0') << tm->tm_sec << "Z";
return is.str();
}
static std::string gen_date_from_yyyy_mm_dd(const std::string& date)
{
std::stringstream is;
is << date << "T00:00::00:Z";
return is.str();
}
void OPDSDumper::setOpenSearchInfo(int totalResults, int startIndex, int count)
{
m_totalResults = totalResults;
m_startIndex = startIndex,
m_count = count;
m_isSearchResult = true;
}
#define ADD_TEXT_ENTRY(node, child, value) (node).append_child((child)).append_child(pugi::node_pcdata).set_value((value).c_str())
pugi::xml_node OPDSDumper::handleBook(Book book, pugi::xml_node root_node) {
auto entry_node = root_node.append_child("entry");
ADD_TEXT_ENTRY(entry_node, "id", "urn:uuid:"+book.getId());
ADD_TEXT_ENTRY(entry_node, "title", book.getTitle());
ADD_TEXT_ENTRY(entry_node, "summary", book.getDescription());
ADD_TEXT_ENTRY(entry_node, "language", book.getLanguage());
ADD_TEXT_ENTRY(entry_node, "updated", gen_date_from_yyyy_mm_dd(book.getDate()));
ADD_TEXT_ENTRY(entry_node, "name", book.getName());
ADD_TEXT_ENTRY(entry_node, "flavour", book.getFlavour());
ADD_TEXT_ENTRY(entry_node, "tags", book.getTags());
ADD_TEXT_ENTRY(entry_node, "articleCount", to_string(book.getArticleCount()));
ADD_TEXT_ENTRY(entry_node, "mediaCount", to_string(book.getMediaCount()));
ADD_TEXT_ENTRY(entry_node, "icon", rootLocation + "/meta?name=favicon&content=" + book.getHumanReadableIdFromPath());
auto content_node = entry_node.append_child("link");
content_node.append_attribute("type") = "text/html";
content_node.append_attribute("href") = (rootLocation + "/" + book.getHumanReadableIdFromPath()).c_str();
auto author_node = entry_node.append_child("author");
ADD_TEXT_ENTRY(author_node, "name", book.getCreator());
auto publisher_node = entry_node.append_child("publisher");
ADD_TEXT_ENTRY(publisher_node, "name", book.getPublisher());
if (! book.getUrl().empty()) {
auto acquisition_link = entry_node.append_child("link");
acquisition_link.append_attribute("rel") = "http://opds-spec.org/acquisition/open-access";
acquisition_link.append_attribute("type") = "application/x-zim";
acquisition_link.append_attribute("href") = book.getUrl().c_str();
acquisition_link.append_attribute("length") = to_string(book.getSize()).c_str();
}
if (! book.getFaviconMimeType().empty() ) {
auto image_link = entry_node.append_child("link");
image_link.append_attribute("rel") = "http://opds-spec.org/image/thumbnail";
image_link.append_attribute("type") = book.getFaviconMimeType().c_str();
image_link.append_attribute("href") = (rootLocation + "/meta?name=favicon&content=" + book.getHumanReadableIdFromPath()).c_str();
}
return entry_node;
}
string OPDSDumper::dumpOPDSFeed(const std::vector<std::string>& bookIds)
namespace
{
date = gen_date_str();
pugi::xml_document doc;
auto root_node = doc.append_child("feed");
root_node.append_attribute("xmlns") = "http://www.w3.org/2005/Atom";
root_node.append_attribute("xmlns:opds") = "http://opds-spec.org/2010/catalog";
typedef kainjow::mustache::data MustacheData;
typedef kainjow::mustache::list BooksData;
typedef kainjow::mustache::list IllustrationInfo;
ADD_TEXT_ENTRY(root_node, "id", id);
ADD_TEXT_ENTRY(root_node, "title", title);
ADD_TEXT_ENTRY(root_node, "updated", date);
if (m_isSearchResult) {
ADD_TEXT_ENTRY(root_node, "totalResults", to_string(m_totalResults));
ADD_TEXT_ENTRY(root_node, "startIndex", to_string(m_startIndex));
ADD_TEXT_ENTRY(root_node, "itemsPerPage", to_string(m_count));
}
auto self_link_node = root_node.append_child("link");
self_link_node.append_attribute("rel") = "self";
self_link_node.append_attribute("href") = "";
self_link_node.append_attribute("type") = "application/atom+xml";
if (!searchDescriptionUrl.empty() ) {
auto search_link = root_node.append_child("link");
search_link.append_attribute("rel") = "search";
search_link.append_attribute("type") = "application/opensearchdescription+xml";
search_link.append_attribute("href") = searchDescriptionUrl.c_str();
}
if (library) {
for (auto& bookId: bookIds) {
handleBook(library->getBookById(bookId), root_node);
IllustrationInfo getBookIllustrationInfo(const Book& book)
{
kainjow::mustache::list illustrations;
if ( book.isPathValid() ) {
for ( auto illustration_size : zim::Archive(book.getPath()).getIllustrationSizes() ) {
illustrations.push_back(kainjow::mustache::object{
{"icon_width", to_string(illustration_size)},
{"icon_height", to_string(illustration_size)},
{"icon_scale", "1"},
});
}
}
return illustrations;
}
kainjow::mustache::object getSingleBookData(const Book& book)
{
const MustacheData bookUrl = book.getUrl().empty()
? MustacheData(false)
: MustacheData(book.getUrl());
return kainjow::mustache::object{
{"id", book.getId()},
{"name", book.getName()},
{"title", book.getTitle()},
{"description", book.getDescription()},
{"language", book.getLanguage()},
{"content_id", book.getHumanReadableIdFromPath()},
{"updated", book.getDate() + "T00:00:00Z"},
{"category", book.getCategory()},
{"flavour", book.getFlavour()},
{"tags", book.getTags()},
{"article_count", to_string(book.getArticleCount())},
{"media_count", to_string(book.getMediaCount())},
{"author_name", book.getCreator()},
{"publisher_name", book.getPublisher()},
{"url", bookUrl},
{"size", to_string(book.getSize())},
{"icons", getBookIllustrationInfo(book)},
};
}
std::string getSingleBookEntryXML(const Book& book, bool withXMLHeader, const std::string& endpointRoot, bool partial)
{
auto data = getSingleBookData(book);
data["with_xml_header"] = MustacheData(withXMLHeader);
data["dump_partial_entries"] = MustacheData(partial);
data["endpoint_root"] = endpointRoot;
return render_template(RESOURCE::templates::catalog_v2_entry_xml, data);
}
BooksData getBooksData(const Library* library, const std::vector<std::string>& bookIds, const std::string& endpointRoot, bool partial)
{
BooksData booksData;
for ( const auto& bookId : bookIds ) {
const Book& book = library->getBookById(bookId);
booksData.push_back(kainjow::mustache::object{
{"entry", getSingleBookEntryXML(book, false, endpointRoot, partial)}
});
}
return nodeToString(root_node);
return booksData;
}
std::string getLanguageSelfName(const std::string& lang) {
const icu::Locale locale(lang.c_str());
icu::UnicodeString ustring;
locale.getDisplayLanguage(locale, ustring);
std::string result;
ustring.toUTF8String(result);
return result;
};
} // unnamed namespace
string OPDSDumper::dumpOPDSFeed(const std::vector<std::string>& bookIds, const std::string& query) const
{
const auto booksData = getBooksData(library, bookIds, "", false);
const kainjow::mustache::object template_data{
{"date", gen_date_str()},
{"root", rootLocation},
{"feed_id", gen_uuid(libraryId + "/catalog/search?"+query)},
{"filter", query.empty() ? MustacheData(false) : MustacheData(query)},
{"totalResults", to_string(m_totalResults)},
{"startIndex", to_string(m_startIndex)},
{"itemsPerPage", to_string(m_count)},
{"books", booksData }
};
return render_template(RESOURCE::templates::catalog_entries_xml, template_data);
}
string OPDSDumper::dumpOPDSFeedV2(const std::vector<std::string>& bookIds, const std::string& query, bool partial) const
{
const auto endpointRoot = rootLocation + "/catalog/v2";
const auto booksData = getBooksData(library, bookIds, endpointRoot, partial);
const char* const endpoint = partial ? "/partial_entries" : "/entries";
const kainjow::mustache::object template_data{
{"date", gen_date_str()},
{"endpoint_root", endpointRoot},
{"feed_id", gen_uuid(libraryId + endpoint + "?" + query)},
{"filter", query.empty() ? MustacheData(false) : MustacheData(query)},
{"query", query.empty() ? "" : "?" + urlEncode(query)},
{"totalResults", to_string(m_totalResults)},
{"startIndex", to_string(m_startIndex)},
{"itemsPerPage", to_string(m_count)},
{"books", booksData },
{"dump_partial_entries", MustacheData(partial)}
};
return render_template(RESOURCE::templates::catalog_v2_entries_xml, template_data);
}
std::string OPDSDumper::dumpOPDSCompleteEntry(const std::string& bookId) const
{
return getSingleBookEntryXML(library->getBookById(bookId), true, "", false);
}
std::string OPDSDumper::categoriesOPDSFeed() const
{
const auto now = gen_date_str();
kainjow::mustache::list categoryData;
for ( const auto& category : library->getBooksCategories() ) {
const auto urlencodedCategoryName = urlEncode(category);
categoryData.push_back(kainjow::mustache::object{
{"name", category},
{"urlencoded_name", urlencodedCategoryName},
{"updated", now},
{"id", gen_uuid(libraryId + "/categories/" + urlencodedCategoryName)}
});
}
return render_template(
RESOURCE::templates::catalog_v2_categories_xml,
kainjow::mustache::object{
{"date", now},
{"endpoint_root", rootLocation + "/catalog/v2"},
{"feed_id", gen_uuid(libraryId + "/categories")},
{"categories", categoryData }
}
);
}
std::string OPDSDumper::languagesOPDSFeed() const
{
const auto now = gen_date_str();
kainjow::mustache::list languageData;
for ( const auto& langAndBookCount : library->getBooksLanguagesWithCounts() ) {
const std::string languageCode = langAndBookCount.first;
const int bookCount = langAndBookCount.second;
const auto languageSelfName = getLanguageSelfName(languageCode);
languageData.push_back(kainjow::mustache::object{
{"lang_code", languageCode},
{"lang_self_name", languageSelfName},
{"book_count", to_string(bookCount)},
{"updated", now},
{"id", gen_uuid(libraryId + "/languages/" + languageCode)}
});
}
return render_template(
RESOURCE::templates::catalog_v2_languages_xml,
kainjow::mustache::object{
{"date", now},
{"endpoint_root", rootLocation + "/catalog/v2"},
{"feed_id", gen_uuid(libraryId + "/languages")},
{"languages", languageData }
}
);
}
}

View File

@@ -21,51 +21,21 @@
#include <time.h>
#include <zim/search.h>
#include <zim/suggestion.h>
#include <zim/item.h>
#include <zim/error.h>
#include "tools.h"
#include "tools/stringTools.h"
#include "tools/otherTools.h"
inline char hi(char v)
{
char hex[] = "0123456789abcdef";
return hex[(v >> 4) & 0xf];
}
inline char lo(char v)
{
char hex[] = "0123456789abcdef";
return hex[v & 0xf];
}
std::string hexUUID(std::string in)
{
std::ostringstream out;
for (unsigned n = 0; n < 4; ++n) {
out << hi(in[n]) << lo(in[n]);
}
out << '-';
for (unsigned n = 4; n < 6; ++n) {
out << hi(in[n]) << lo(in[n]);
}
out << '-';
for (unsigned n = 6; n < 8; ++n) {
out << hi(in[n]) << lo(in[n]);
}
out << '-';
for (unsigned n = 8; n < 10; ++n) {
out << hi(in[n]) << lo(in[n]);
}
out << '-';
for (unsigned n = 10; n < 16; ++n) {
out << hi(in[n]) << lo(in[n]);
}
std::string op = out.str();
return op;
}
#include "tools/archiveTools.h"
namespace kiwix
{
/* Constructor */
Reader::Reader(const string zimFilePath) : zimFileHandler(NULL)
Reader::Reader(const string zimFilePath)
: zimArchive(nullptr),
zimFilePath(zimFilePath)
{
string tmpZimFilePath = zimFilePath;
@@ -76,43 +46,43 @@ Reader::Reader(const string zimFilePath) : zimFileHandler(NULL)
tmpZimFilePath.resize(tmpZimFilePath.size() - 2);
}
this->zimFileHandler = new zim::File(tmpZimFilePath);
if (this->zimFileHandler != NULL) {
this->firstArticleOffset
= this->zimFileHandler->getNamespaceBeginOffset('A');
this->lastArticleOffset = this->zimFileHandler->getNamespaceEndOffset('A');
this->nsACount = this->zimFileHandler->getNamespaceCount('A');
this->nsICount = this->zimFileHandler->getNamespaceCount('I');
this->zimFilePath = zimFilePath;
}
zimArchive.reset(new zim::Archive(tmpZimFilePath));
/* initialize random seed: */
srand(time(NULL));
srand(time(nullptr));
}
/* Destructor */
Reader::~Reader()
Reader::Reader(const std::shared_ptr<zim::Archive> archive)
: zimArchive(archive),
zimFilePath(archive->getFilename())
{}
#ifndef _WIN32
Reader::Reader(int fd)
: zimArchive(new zim::Archive(fd)),
zimFilePath("")
{
if (this->zimFileHandler != NULL) {
delete this->zimFileHandler;
}
/* initialize random seed: */
srand(time(nullptr));
}
zim::File* Reader::getZimFileHandler() const
Reader::Reader(int fd, zim::offset_type offset, zim::size_type size)
: zimArchive(new zim::Archive(fd, offset, size)),
zimFilePath("")
{
return this->zimFileHandler;
/* initialize random seed: */
srand(time(nullptr));
}
#endif // #ifndef _WIN32
zim::Archive* Reader::getZimArchive() const
{
return zimArchive.get();
}
MimeCounterType Reader::parseCounterMetadata() const
{
zim::Article article = this->zimFileHandler->getArticle('M', "Counter");
if (article.good()) {
return parseMimetypeCounter(article.getData());
}
return MimeCounterType();
return kiwix::parseArchiveCounter(*zimArchive);
}
/* Get the count of articles which can be indexed/displayed */
@@ -122,13 +92,9 @@ unsigned int Reader::getArticleCount() const
= this->parseCounterMetadata();
unsigned int counter = 0;
if (counterMap.empty()) {
counter = this->nsACount;
} else {
for(auto &pair:counterMap) {
if (startsWith(pair.first, "text/html")) {
counter += pair.second;
}
for(auto &pair:counterMap) {
if (startsWith(pair.first, "text/html")) {
counter += pair.second;
}
}
@@ -138,143 +104,51 @@ unsigned int Reader::getArticleCount() const
/* Get the count of medias content in the ZIM file */
unsigned int Reader::getMediaCount() const
{
std::map<const std::string, unsigned int> counterMap
= this->parseCounterMetadata();
unsigned int counter = 0;
if (counterMap.empty()) {
counter = this->nsICount;
} else {
auto it = counterMap.find("image/jpeg");
if (it != counterMap.end()) {
counter += it->second;
}
it = counterMap.find("image/gif");
if (it != counterMap.end()) {
counter += it->second;
}
it = counterMap.find("image/png");
if (it != counterMap.end()) {
counter += it->second;
}
}
return counter;
return kiwix::getArchiveMediaCount(*zimArchive);
}
/* Get the total of all items of a ZIM file, redirects included */
unsigned int Reader::getGlobalCount() const
{
return this->zimFileHandler->getCountArticles();
return zimArchive->getEntryCount();
}
/* Return the UID of the ZIM file */
string Reader::getId() const
{
std::ostringstream s;
s << this->zimFileHandler->getFileheader().getUuid();
return s.str();
}
/* Return a page url from a title */
bool Reader::getPageUrlFromTitle(const string& title, string& url) const
{
try {
auto entry = getEntryFromTitle(title);
entry = entry.getFinalEntry();
url = entry.getPath();
return true;
} catch (NoEntry& e) {
return false;
}
}
/* Return an URL from a title */
string Reader::getRandomPageUrl() const
{
return getRandomPage().getPath();
return kiwix::getArchiveId(*zimArchive);
}
Entry Reader::getRandomPage() const
{
if (!this->zimFileHandler) {
try {
return zimArchive->getRandomEntry();
} catch(...) {
throw NoEntry();
}
zim::Article article;
std::string mainPagePath = this->getMainPage().getPath();
int watchdog = 42;
do {
auto idx = this->firstArticleOffset
+ (zim::size_type)((double)rand() / ((double)RAND_MAX + 1)
* this->nsACount);
article = zimFileHandler->getArticle(idx);
if (!watchdog--) {
throw NoEntry();
}
} while (!article.good() && article.getLongUrl() == mainPagePath);
return article;
}
/* Return the welcome page URL */
string Reader::getMainPageUrl() const
{
return getMainPage().getPath();
}
Entry Reader::getMainPage() const
{
if (!this->zimFileHandler) {
throw NoEntry();
}
zim::Article article;
if (this->zimFileHandler->getFileheader().hasMainPage())
{
article = zimFileHandler->getArticle(
this->zimFileHandler->getFileheader().getMainPage());
}
if (!article.good())
{
return getFirstPage();
}
return article;
return zimArchive->getMainEntry();
}
bool Reader::getFavicon(string& content, string& mimeType) const
{
static const char* const paths[] = {"-/favicon", "-/favicon.png", "I/favicon.png", "I/favicon"};
for (auto &path: paths) {
try {
auto entry = getEntryFromPath(path);
entry = entry.getFinalEntry();
content = entry.getContent();
mimeType = entry.getMimetype();
return true;
} catch(NoEntry& e) {};
}
return false;
return kiwix::getArchiveFavicon(*zimArchive, 48, content, mimeType);
}
string Reader::getZimFilePath() const
{
return this->zimFilePath;
return zimFilePath;
}
/* Return a metatag value */
bool Reader::getMetadata(const string& name, string& value) const
{
try {
auto entry = getEntryFromPath("M/"+name);
value = entry.getContent();
value = zimArchive->getMetadata(name);
return true;
} catch(NoEntry& e) {
} catch(zim::EntryNotFound& e) {
return false;
}
}
@@ -283,48 +157,32 @@ bool Reader::getMetadata(const string& name, string& value) const
string Reader::getName() const
{
METADATA("Name")
return kiwix::getMetaName(*zimArchive);
}
string Reader::getTitle() const
{
string value;
this->getMetadata("Title", value);
if (value.empty()) {
value = getLastPathElement(zimFileHandler->getFilename());
std::replace(value.begin(), value.end(), '_', ' ');
size_t pos = value.find(".zim");
value = value.substr(0, pos);
}
return value;
return kiwix::getArchiveTitle(*zimArchive);
}
string Reader::getCreator() const
{
METADATA("Creator")
return kiwix::getMetaCreator(*zimArchive);
}
string Reader::getPublisher() const
{
METADATA("Publisher")
return kiwix::getMetaPublisher(*zimArchive);
}
string Reader::getDate() const
{
METADATA("Date")
return kiwix::getMetaDate(*zimArchive);
}
string Reader::getDescription() const
{
string value;
this->getMetadata("Description", value);
/* Mediawiki Collection tends to use the "Subtitle" name */
if (value.empty()) {
this->getMetadata("Subtitle", value);
}
return value;
return kiwix::getMetaDescription(*zimArchive);
}
string Reader::getLongDescription() const
@@ -334,7 +192,7 @@ string Reader::getLongDescription() const
string Reader::getLanguage() const
{
METADATA("Language")
return kiwix::getMetaLanguage(*zimArchive);
}
string Reader::getLicense() const
@@ -344,13 +202,7 @@ string Reader::getLicense() const
string Reader::getTags(bool original) const
{
string tags_str;
getMetadata("Tags", tags_str);
if (original) {
return tags_str;
}
auto tags = convertTags(tags_str);
return join(tags, ";");
return kiwix::getMetaTags(*zimArchive, original);
}
@@ -373,7 +225,7 @@ string Reader::getRelation() const
string Reader::getFlavour() const
{
METADATA("Flavour")
return kiwix::getMetaFlavour(*zimArchive);
}
string Reader::getSource() const
@@ -387,108 +239,13 @@ string Reader::getScraper() const
}
#undef METADATA
string Reader::getOrigId() const
{
string value;
this->getMetadata("startfileuid", value);
if (value.empty()) {
return "";
}
std::string id = value;
std::string origID;
std::string temp = "";
unsigned int k = 0;
char tempArray[16] = "";
for (unsigned int i = 0; i < id.size(); i++) {
if (id[i] == '\n') {
tempArray[k] = atoi(temp.c_str());
temp = "";
k++;
} else {
temp += id[i];
}
}
origID = hexUUID(tempArray);
return origID;
}
/* Return the first page URL */
string Reader::getFirstPageUrl() const
{
return getFirstPage().getPath();
}
Entry Reader::getFirstPage() const
{
if (!this->zimFileHandler) {
throw NoEntry();
}
auto firstPageOffset = zimFileHandler->getNamespaceBeginOffset('A');
auto article = zimFileHandler->getArticle(firstPageOffset);
if (! article.good()) {
throw NoEntry();
}
return article;
}
bool _parseUrl(const string& url, char* ns, string& title)
{
/* Offset to visit the url */
unsigned int urlLength = url.size();
unsigned int offset = 0;
/* Ignore the first '/' */
if (url[offset] == '/')
offset++;
if (url[offset] == '/' || offset >= urlLength)
return false;
/* Get namespace */
*ns = url[offset++];
if (url[offset] != '/' || offset >= urlLength)
return false;
offset++;
if ( offset >= urlLength)
return false;
/* Get content title */
title = url.substr(offset, urlLength - offset);
return true;
}
bool Reader::parseUrl(const string& url, char* ns, string& title) const
{
return _parseUrl(url, ns, title);
}
Entry Reader::getEntryFromPath(const std::string& path) const
{
char ns = 0;
std::string short_url;
if (!this->zimFileHandler) {
try {
return kiwix::getEntryFromPath(*zimArchive, path);
} catch (zim::EntryNotFound& e) {
throw NoEntry();
}
_parseUrl(path, &ns, short_url);
if (short_url.empty() && ns == 0) {
return getMainPage();
}
auto article = zimFileHandler->getArticle(ns, short_url);
if (!article.good()) {
throw NoEntry();
}
return article;
}
Entry Reader::getEntryFromEncodedPath(const std::string& path) const
@@ -498,202 +255,22 @@ Entry Reader::getEntryFromEncodedPath(const std::string& path) const
Entry Reader::getEntryFromTitle(const std::string& title) const
{
if (!this->zimFileHandler) {
try {
return zimArchive->getEntryByTitle(title);
} catch(zim::EntryNotFound& e) {
throw NoEntry();
}
auto article = this->zimFileHandler->getArticleByTitle('A', title);
if (!article.good()) {
throw NoEntry();
}
return article;
}
/* Return article by url */
bool Reader::getArticleObjectByDecodedUrl(const string& url,
zim::Article& article) const
{
if (this->zimFileHandler == NULL) {
return false;
}
/* Parse the url */
char ns = 0;
string urlStr;
_parseUrl(url, &ns, urlStr);
/* Main page */
if (urlStr.empty() && ns == 0) {
_parseUrl(this->getMainPage().getPath(), &ns, urlStr);
}
/* Extract the content from the zim file */
article = zimFileHandler->getArticle(ns, urlStr);
return article.good();
}
/* Return the mimeType without the content */
bool Reader::getMimeTypeByUrl(const string& url, string& mimeType) const
{
try {
auto entry = getEntryFromPath(url);
mimeType = entry.getMimetype();
return true;
} catch (NoEntry& e) {
mimeType = "";
return false;
}
}
bool get_content_by_decoded_url(const Reader& reader,
const string& url,
string& content,
string& title,
unsigned int& contentLength,
string& contentType,
string& baseUrl)
{
content = "";
contentType = "";
contentLength = 0;
try {
auto entry = reader.getEntryFromPath(url);
entry = entry.getFinalEntry();
baseUrl = entry.getPath();
contentType = entry.getMimetype();
content = entry.getContent();
contentLength = entry.getSize();
title = entry.getTitle();
/* Try to set a stub HTML header/footer if necesssary */
if (contentType.find("text/html") != string::npos
&& content.find("<body") == std::string::npos
&& content.find("<BODY") == std::string::npos) {
content = "<html><head><title>" + title +
"</title><meta http-equiv=\"Content-Type\" content=\"text/html; "
"charset=utf-8\" /></head><body>" +
content + "</body></html>";
}
return true;
} catch (NoEntry& e) {
return false;
}
}
/* Get a content from a zim file */
bool Reader::getContentByUrl(const string& url,
string& content,
string& title,
unsigned int& contentLength,
string& contentType) const
{
std::string stubRedirectUrl;
return get_content_by_decoded_url(*this,
kiwix::urlDecode(url),
content,
title,
contentLength,
contentType,
stubRedirectUrl);
}
bool Reader::getContentByEncodedUrl(const string& url,
string& content,
string& title,
unsigned int& contentLength,
string& contentType,
string& baseUrl) const
{
return get_content_by_decoded_url(*this,
kiwix::urlDecode(url),
content,
title,
contentLength,
contentType,
baseUrl);
}
bool Reader::getContentByEncodedUrl(const string& url,
string& content,
string& title,
unsigned int& contentLength,
string& contentType) const
{
std::string stubRedirectUrl;
return get_content_by_decoded_url(*this,
kiwix::urlDecode(url),
content,
title,
contentLength,
contentType,
stubRedirectUrl);
}
bool Reader::getContentByDecodedUrl(const string& url,
string& content,
string& title,
unsigned int& contentLength,
string& contentType) const
{
std::string stubRedirectUrl;
return get_content_by_decoded_url(*this,
url,
content,
title,
contentLength,
contentType,
stubRedirectUrl);
}
bool Reader::getContentByDecodedUrl(const string& url,
string& content,
string& title,
unsigned int& contentLength,
string& contentType,
string& baseUrl) const
{
return get_content_by_decoded_url(*this,
url,
content,
title,
contentLength,
contentType,
baseUrl);
}
/* Check if an article exists */
bool Reader::urlExists(const string& url) const
{
return pathExists(url);
}
bool Reader::pathExists(const string& path) const
{
if (!zimFileHandler)
{
return false;
}
char ns = 0;
string titleStr;
_parseUrl(path, &ns, titleStr);
zim::File::const_iterator findItr = zimFileHandler->find(ns, titleStr);
return findItr != zimFileHandler->end() && findItr->getUrl() == titleStr;
return zimArchive->hasEntryByPath(path);
}
/* Does the ZIM file has a fulltext index */
bool Reader::hasFulltextIndex() const
{
if (!zimFileHandler || zimFileHandler->is_multiPart() )
{
return false;
}
return ( pathExists("Z//fulltextIndex/xapian")
|| pathExists("X/fulltext/xapian"));
return zimArchive->hasFulltextIndex();
}
/* Search titles by prefix */
@@ -733,34 +310,28 @@ bool Reader::searchSuggestions(const string& prefix,
return false;
}
for (auto articleItr = zimFileHandler->findByTitle('A', prefix);
articleItr != zimFileHandler->end()
&& articleItr->getTitle().compare(0, prefix.size(), prefix) == 0
&& results.size() < suggestionsCount;
++articleItr) {
for (auto& entry: zimArchive->findByTitle(prefix)) {
if (results.size() >= suggestionsCount) {
break;
}
/* Extract the interesting part of article title & url */
std::string normalizedArticleTitle
= kiwix::normalize(articleItr->getTitle());
std::string articleFinalUrl = "/A/" + articleItr->getUrl();
if (articleItr->isRedirect()) {
zim::Article article = *articleItr;
unsigned int loopCounter = 0;
while (article.isRedirect() && loopCounter++ < 42) {
article = article.getRedirectArticle();
}
articleFinalUrl = "/A/" + article.getUrl();
}
= kiwix::normalize(entry.getTitle());
// Get the final path.
auto item = entry.getItem(true);
std::string articleFinalUrl = item.getPath();
/* Go through all already found suggestions and skip if this
article is already in the suggestions list (with an other
title) */
bool insert = true;
std::vector<std::vector<std::string>>::iterator suggestionItr;
std::vector<SuggestionItem>::iterator suggestionItr;
for (suggestionItr = results.begin();
suggestionItr != results.end();
suggestionItr++) {
int result = normalizedArticleTitle.compare((*suggestionItr)[2]);
if (result == 0 && articleFinalUrl.compare((*suggestionItr)[1]) == 0) {
int result = normalizedArticleTitle.compare((*suggestionItr).getNormalizedTitle());
if (result == 0 && articleFinalUrl.compare((*suggestionItr).getPath()) == 0) {
insert = false;
break;
} else if (result < 0) {
@@ -770,10 +341,7 @@ bool Reader::searchSuggestions(const string& prefix,
/* Insert if possible */
if (insert) {
std::vector<std::string> suggestion;
suggestion.push_back(articleItr->getTitle());
suggestion.push_back(articleFinalUrl);
suggestion.push_back(normalizedArticleTitle);
SuggestionItem suggestion(entry.getTitle(), normalizedArticleTitle, articleFinalUrl);
results.insert(suggestionItr, suggestion);
}
@@ -787,12 +355,7 @@ bool Reader::searchSuggestions(const string& prefix,
std::vector<std::string> Reader::getTitleVariants(
const std::string& title) const
{
std::vector<std::string> variants;
variants.push_back(title);
variants.push_back(kiwix::ucFirst(title));
variants.push_back(kiwix::lcFirst(title));
variants.push_back(kiwix::toTitle(title));
return variants;
return kiwix::getTitleVariants(title);
}
@@ -815,36 +378,36 @@ bool Reader::searchSuggestionsSmart(const string& prefix,
SuggestionsList_t& results)
{
std::vector<std::string> variants = this->getTitleVariants(prefix);
bool retVal = false;
/* Try to search in the title using fulltext search database */
const auto suggestionSearch
= this->getZimFileHandler()->suggestions(prefix, 0, suggestionsCount);
auto suggestionSearcher = zim::SuggestionSearcher(*zimArchive);
if (suggestionSearch->get_matches_estimated()) {
for (auto current = suggestionSearch->begin();
current != suggestionSearch->end();
current++) {
if (!current->good()) {
continue;
}
std::vector<std::string> suggestion;
suggestion.push_back(current->getTitle());
suggestion.push_back("/A/" + current->getUrl());
suggestion.push_back(kiwix::normalize(current->getTitle()));
if (zimArchive->hasTitleIndex()) {
auto suggestionSearch = suggestionSearcher.suggest(prefix);
const auto suggestions = suggestionSearch.getResults(0, suggestionsCount);
for (auto current : suggestions) {
SuggestionItem suggestion(current.getTitle(), kiwix::normalize(current.getTitle()),
current.getPath(), current.getSnippet());
results.push_back(suggestion);
}
retVal = true;
} else {
// Check some of the variants of the prefix
for (std::vector<std::string>::iterator variantsItr = variants.begin();
variantsItr != variants.end();
variantsItr++) {
retVal = this->searchSuggestions(*variantsItr, suggestionsCount, results)
|| retVal;
auto suggestionSearch = suggestionSearcher.suggest(*variantsItr);
for (auto current : suggestionSearch.getResults(0, suggestionsCount)) {
if (results.size() >= suggestionsCount) {
break;
}
SuggestionItem suggestion(current.getTitle(), kiwix::normalize(current.getTitle()),
current.getPath(), current.getSnippet());
results.push_back(suggestion);
}
}
}
return retVal;
return results.size() > 0;
}
/* Get next suggestion */
@@ -852,7 +415,7 @@ bool Reader::getNextSuggestion(string& title)
{
if (this->suggestionsOffset != this->suggestions.end()) {
/* title */
title = (*(this->suggestionsOffset))[0];
title = (*(this->suggestionsOffset)).getTitle();
/* increment the cursor for the next call */
this->suggestionsOffset++;
@@ -867,8 +430,8 @@ bool Reader::getNextSuggestion(string& title, string& url)
{
if (this->suggestionsOffset != this->suggestions.end()) {
/* title */
title = (*(this->suggestionsOffset))[0];
url = (*(this->suggestionsOffset))[1];
title = (*(this->suggestionsOffset)).getTitle();
url = (*(this->suggestionsOffset)).getPath();
/* increment the cursor for the next call */
this->suggestionsOffset++;
@@ -882,14 +445,14 @@ bool Reader::getNextSuggestion(string& title, string& url)
/* Check if the file has as checksum */
bool Reader::canCheckIntegrity() const
{
return this->zimFileHandler->getChecksum() != "";
return zimArchive->hasChecksum();
}
/* Return true if corrupted, false otherwise */
bool Reader::isCorrupted() const
{
try {
if (this->zimFileHandler->verify() == true) {
if (zimArchive->check() == true) {
return false;
}
} catch (exception& e) {
@@ -903,13 +466,7 @@ bool Reader::isCorrupted() const
/* Return the file size, works also for splitted files */
unsigned int Reader::getFileSize() const
{
zim::File* file = this->getZimFileHandler();
zim::size_type size = 0;
if (file != NULL) {
size = file->getFilesize();
}
return (size / 1024);
return kiwix::getArchiveFileSize(*zimArchive);
}
}

View File

@@ -30,17 +30,29 @@
#include <mustache.hpp>
#include "kiwixlib-resources.h"
#include "tools/stringTools.h"
namespace kiwix
{
/* Constructor */
SearchRenderer::SearchRenderer(Searcher* searcher, NameMapper* mapper)
: mp_searcher(searcher),
: m_srs(searcher->getSearchResultSet()),
mp_nameMapper(mapper),
protocolPrefix("zim://"),
searchProtocolPrefix("search://?")
searchProtocolPrefix("search://?"),
estimatedResultCount(searcher->getEstimatedResultCount()),
resultStart(searcher->getResultStart())
{}
SearchRenderer::SearchRenderer(zim::SearchResultSet srs, NameMapper* mapper,
unsigned int start, unsigned int estimatedResultCount)
: m_srs(srs),
mp_nameMapper(mapper),
protocolPrefix("zim://"),
searchProtocolPrefix("search://?"),
estimatedResultCount(estimatedResultCount),
resultStart(start)
{}
/* Destructor */
@@ -70,31 +82,26 @@ std::string SearchRenderer::getHtml()
{
kainjow::mustache::data results{kainjow::mustache::data::type::list};
mp_searcher->restart_search();
Result* p_result = NULL;
while ((p_result = mp_searcher->getNextResult())) {
for (auto it = m_srs.begin(); it != m_srs.end(); it++) {
kainjow::mustache::data result;
result.set("title", p_result->get_title());
result.set("url", p_result->get_url());
result.set("snippet", p_result->get_snippet());
auto readerIndex = p_result->get_readerIndex();
auto reader = mp_searcher->get_reader(readerIndex);
result.set("resultContentId", mp_nameMapper->getNameForId(reader->getId()));
result.set("title", it.getTitle());
result.set("url", it.getPath());
result.set("snippet", it.getSnippet());
std::ostringstream s;
s << it.getZimId();
result.set("resultContentId", mp_nameMapper->getNameForId(s.str()));
if (p_result->get_wordCount() >= 0) {
result.set("wordCount", kiwix::beautifyInteger(p_result->get_wordCount()));
if (it.getWordCount() >= 0) {
result.set("wordCount", kiwix::beautifyInteger(it.getWordCount()));
}
results.push_back(result);
delete p_result;
}
// pages
kainjow::mustache::data pages{kainjow::mustache::data::type::list};
auto resultStart = mp_searcher->getResultStart();
auto resultEnd = 0U;
auto estimatedResultCount = mp_searcher->getEstimatedResultCount();
auto currentPage = 0U;
auto pageStart = 0U;
auto pageEnd = 0U;
@@ -148,4 +155,4 @@ std::string SearchRenderer::getHtml()
return ss.str();
}
}
}

View File

@@ -18,14 +18,16 @@
*/
#include <cmath>
#include "searcher.h"
#include "reader.h"
#include <zim/search.h>
#include <zim/suggestion.h>
#include <mustache.hpp>
#include <cmath>
#include "tools/stringTools.h"
#include "kiwixlib-resources.h"
#define MAX_SEARCH_LEN 140
@@ -35,7 +37,8 @@ namespace kiwix
class _Result : public Result
{
public:
_Result(zim::Search::iterator& iterator);
_Result(zim::SearchResultSet::iterator iterator);
_Result(SuggestionItem suggestionItem);
virtual ~_Result(){};
virtual std::string get_url();
@@ -45,32 +48,38 @@ class _Result : public Result
virtual std::string get_content();
virtual int get_wordCount();
virtual int get_size();
virtual int get_readerIndex();
virtual std::string get_zimId();
private:
zim::Search::iterator iterator;
zim::SearchResultSet::iterator iterator;
SuggestionItem suggestionItem;
bool isSuggestion;
};
struct SearcherInternal {
const zim::Search* _search;
zim::Search::iterator current_iterator;
SearcherInternal() : _search(NULL) {}
~SearcherInternal()
struct SearcherInternal : zim::SearchResultSet {
explicit SearcherInternal(const zim::SearchResultSet& srs)
: zim::SearchResultSet(srs)
, current_iterator(srs.begin())
{
if (_search != NULL) {
delete _search;
}
}
zim::SearchResultSet::iterator current_iterator;
};
struct SuggestionInternal : zim::SuggestionResultSet {
explicit SuggestionInternal(const zim::SuggestionResultSet& srs)
: zim::SuggestionResultSet(srs),
currentIterator(srs.begin()) {}
zim::SuggestionResultSet::iterator currentIterator;
};
/* Constructor */
Searcher::Searcher()
: internal(new SearcherInternal()),
searchPattern(""),
: searchPattern(""),
estimatedResultCount(0),
resultStart(0),
resultEnd(0)
maxResultCount(0)
{
loadICUExternalTables();
}
@@ -78,7 +87,6 @@ Searcher::Searcher()
/* Destructor */
Searcher::~Searcher()
{
delete internal;
}
bool Searcher::add_reader(Reader* reader)
@@ -99,7 +107,7 @@ Reader* Searcher::get_reader(int readerIndex)
/* Search strings in the database */
void Searcher::search(const std::string& search,
unsigned int resultStart,
unsigned int resultEnd,
unsigned int maxResultCount,
const bool verbose)
{
this->reset();
@@ -110,25 +118,25 @@ void Searcher::search(const std::string& search,
this->searchPattern = search;
this->resultStart = resultStart;
this->resultEnd = resultEnd;
this->maxResultCount = maxResultCount;
/* Try to find results */
if (resultStart != resultEnd) {
if (maxResultCount != 0) {
/* Perform the search */
string unaccentedSearch = removeAccents(search);
std::vector<const zim::File*> zims;
std::vector<zim::Archive> archives;
for (auto current = this->readers.begin(); current != this->readers.end();
current++) {
if ( (*current)->hasFulltextIndex() ) {
zims.push_back((*current)->getZimFileHandler());
archives.push_back(*(*current)->getZimArchive());
}
}
zim::Search* search = new zim::Search(zims);
search->set_verbose(verbose);
search->set_query(unaccentedSearch);
search->set_range(resultStart, resultEnd);
internal->_search = search;
internal->current_iterator = internal->_search->begin();
this->estimatedResultCount = internal->_search->get_matches_estimated();
zim::Searcher searcher(archives);
searcher.setVerbose(verbose);
zim::Query query;
query.setQuery(unaccentedSearch);
zim::Search search = searcher.search(query);
internal.reset(new SearcherInternal(search.getResults(resultStart, maxResultCount)));
this->estimatedResultCount = search.getEstimatedMatches();
}
return;
@@ -137,7 +145,7 @@ void Searcher::search(const std::string& search,
void Searcher::geo_search(float latitude, float longitude, float distance,
unsigned int resultStart,
unsigned int resultEnd,
unsigned int maxResultCount,
const bool verbose)
{
this->reset();
@@ -151,43 +159,53 @@ void Searcher::geo_search(float latitude, float longitude, float distance,
oss << "Articles located less than " << distance << " meters of " << latitude << ";" << longitude;
this->searchPattern = oss.str();
this->resultStart = resultStart;
this->resultEnd = resultEnd;
this->maxResultCount = maxResultCount;
/* Try to find results */
if (resultStart == resultEnd) {
if (maxResultCount == 0) {
return;
}
std::vector<const zim::File*> zims;
std::vector<zim::Archive> archives;
for (auto current = this->readers.begin(); current != this->readers.end();
current++) {
zims.push_back((*current)->getZimFileHandler());
archives.push_back(*(*current)->getZimArchive());
}
zim::Search* search = new zim::Search(zims);
search->set_verbose(verbose);
search->set_query("");
search->set_georange(latitude, longitude, distance);
search->set_range(resultStart, resultEnd);
internal->_search = search;
internal->current_iterator = internal->_search->begin();
this->estimatedResultCount = internal->_search->get_matches_estimated();
zim::Searcher searcher(archives);
searcher.setVerbose(verbose);
zim::Query query;
query.setQuery("");
query.setGeorange(latitude, longitude, distance);
zim::Search search = searcher.search(query);
internal.reset(new SearcherInternal(search.getResults(resultStart, maxResultCount)));
this->estimatedResultCount = search.getEstimatedMatches();
}
void Searcher::restart_search()
{
if (internal->_search) {
internal->current_iterator = internal->_search->begin();
if (internal.get()) {
internal->current_iterator = internal->begin();
}
}
Result* Searcher::getNextResult()
{
if (internal->_search &&
internal->current_iterator != internal->_search->end()) {
if (internal.get() && internal->current_iterator != internal->end()) {
Result* result = new _Result(internal->current_iterator);
internal->current_iterator++;
return result;
} else if (suggestionInternal.get() &&
suggestionInternal->currentIterator != suggestionInternal->end()) {
SuggestionItem item(
suggestionInternal->currentIterator->getTitle(),
normalize(suggestionInternal->currentIterator->getTitle()),
suggestionInternal->currentIterator->getPath(),
suggestionInternal->currentIterator->getSnippet()
);
Result* result = new _Result(item);
suggestionInternal->currentIterator++;
return result;
}
return NULL;
}
@@ -210,22 +228,16 @@ void Searcher::suggestions(std::string& searchPattern, const bool verbose)
this->searchPattern = searchPattern;
this->resultStart = 0;
this->resultEnd = 10;
this->maxResultCount = 10;
string unaccentedSearch = removeAccents(searchPattern);
std::vector<const zim::File*> zims;
for (auto current = this->readers.begin(); current != this->readers.end();
current++) {
zims.push_back((*current)->getZimFileHandler());
}
zim::Search* search = new zim::Search(zims);
search->set_verbose(verbose);
search->set_query(unaccentedSearch);
search->set_range(resultStart, resultEnd);
search->set_suggestion_mode(true);
internal->_search = search;
internal->current_iterator = internal->_search->begin();
this->estimatedResultCount = internal->_search->get_matches_estimated();
// Multizim suggestion is not supported as of now! taking only one archive
zim::Archive archive = *(*this->readers.begin())->getZimArchive();
zim::SuggestionSearcher searcher(archive);
searcher.setVerbose(verbose);
zim::SuggestionSearch search = searcher.suggest(searchPattern);
suggestionInternal.reset(new SuggestionInternal(search.getResults(resultStart, maxResultCount)));
this->estimatedResultCount = search.getEstimatedMatches();
}
/* Return the result count estimation */
@@ -234,45 +246,78 @@ unsigned int Searcher::getEstimatedResultCount()
return this->estimatedResultCount;
}
_Result::_Result(zim::Search::iterator& iterator)
: iterator(iterator)
zim::SearchResultSet Searcher::getSearchResultSet()
{
return *(this->internal);
}
_Result::_Result(zim::SearchResultSet::iterator iterator)
: iterator(iterator),
suggestionItem("", "", ""),
isSuggestion(false)
{}
_Result::_Result(SuggestionItem item)
: iterator(),
suggestionItem(item.getTitle(), item.getNormalizedTitle(), item.getPath(), item.getSnippet()),
isSuggestion(true)
{}
std::string _Result::get_url()
{
return iterator.get_url();
if (isSuggestion) {
return suggestionItem.getPath();
}
return iterator.getPath();
}
std::string _Result::get_title()
{
return iterator.get_title();
if (isSuggestion) {
return suggestionItem.getTitle();
}
return iterator.getTitle();
}
int _Result::get_score()
{
return iterator.get_score();
if (isSuggestion) {
return 0;
}
return iterator.getScore();
}
std::string _Result::get_snippet()
{
return iterator.get_snippet();
if (isSuggestion) {
return suggestionItem.getSnippet();
}
return iterator.getSnippet();
}
std::string _Result::get_content()
{
if (iterator->good()) {
return iterator->getData();
}
return "";
if (isSuggestion) return "";
return iterator->getItem(true).getData();
}
int _Result::get_size()
{
return iterator.get_size();
if (isSuggestion) {
return 0;
}
return iterator.getSize();
}
int _Result::get_wordCount()
{
return iterator.get_wordCount();
if (isSuggestion) {
return 0;
}
return iterator.getWordCount();
}
int _Result::get_readerIndex()
std::string _Result::get_zimId()
{
return iterator.get_fileIndex();
if (isSuggestion) {
return "";
}
std::ostringstream s;
s << iterator.getZimId();
return s.str();
}

View File

@@ -48,7 +48,8 @@ bool Server::start() {
m_verbose,
m_withTaskbar,
m_withLibraryButton,
m_blockExternalLinks));
m_blockExternalLinks,
m_indexTemplateString));
return mp_server->start();
}

View File

@@ -19,6 +19,10 @@
#include "internalServer.h"
#ifdef __FreeBSD__
#include <netinet/in.h>
#endif
#ifdef _WIN32
# if !defined(__MINGW32__) && (_MSC_VER < 1600)
# include "stdint4win.h"
@@ -39,10 +43,11 @@ extern "C" {
#include "microhttpd_wrapper.h"
}
#include "tools/otherTools.h"
#include "tools.h"
#include "tools/pathTools.h"
#include "tools/regexTools.h"
#include "tools/stringTools.h"
#include "tools/archiveTools.h"
#include "library.h"
#include "name_mapper.h"
#include "entry.h"
@@ -51,10 +56,14 @@ extern "C" {
#include "opds_dumper.h"
#include <zim/uuid.h>
#include <zim/error.h>
#include <zim/search.h>
#include <zim/suggestion.h>
#include <zim/entry.h>
#include <zim/item.h>
#include <mustache.hpp>
#include <pthread.h>
#include <atomic>
#include <string>
#include <vector>
@@ -73,6 +82,31 @@ extern "C" {
namespace kiwix {
namespace
{
inline std::string normalizeRootUrl(std::string rootUrl)
{
while ( !rootUrl.empty() && rootUrl.back() == '/' )
rootUrl.pop_back();
while ( !rootUrl.empty() && rootUrl.front() == '/' )
rootUrl = rootUrl.substr(1);
return rootUrl.empty() ? rootUrl : "/" + rootUrl;
}
unsigned parseIllustration(const std::string& s)
{
int nw(0), nh(0), nEnd(0);
long int w(-1), h(-1);
if ( sscanf(s.c_str(), "Illustration_%n%ldx%n%ld@1%n)", &nw, &w, &nh, &h, &nEnd) == 2
&& nEnd == (int)s.size() && !isspace(s[nw]) && !isspace(s[nh]) && w == h && w >= 0) {
return w;
}
return 0;
}
} // unnamed namespace
static IdNameMapper defaultNameMapper;
static MHD_Result staticHandlerCallback(void* cls,
@@ -94,15 +128,17 @@ InternalServer::InternalServer(Library* library,
bool verbose,
bool withTaskbar,
bool withLibraryButton,
bool blockExternalLinks) :
bool blockExternalLinks,
std::string indexTemplateString) :
m_addr(addr),
m_port(port),
m_root(root),
m_root(normalizeRootUrl(root)),
m_nbThreads(nbThreads),
m_verbose(verbose),
m_withTaskbar(withTaskbar),
m_withLibraryButton(withLibraryButton),
m_blockExternalLinks(blockExternalLinks),
m_indexTemplateString(indexTemplateString.empty() ? RESOURCE::templates::index_html : indexTemplateString),
mp_daemon(nullptr),
mp_library(library),
mp_nameMapper(nameMapper ? nameMapper : &defaultNameMapper)
@@ -150,6 +186,7 @@ bool InternalServer::start() {
}
auto server_start_time = std::chrono::system_clock::now().time_since_epoch();
m_server_id = kiwix::to_string(server_start_time.count());
m_library_id = m_server_id;
return true;
}
@@ -234,16 +271,16 @@ std::unique_ptr<Response> InternalServer::handle_request(const RequestContext& r
{
try {
if (! request.is_valid_url())
return Response::build_404(*this, request, "");
return Response::build_404(*this, request, "", "");
const ETag etag = get_matching_if_none_match_etag(request);
if ( etag )
return Response::build_304(*this, etag);
if (kiwix::startsWith(request.get_url(), "/skin/"))
if (startsWith(request.get_url(), "/skin/"))
return handle_skin(request);
if (startsWith(request.get_url(), "/catalog"))
if (startsWith(request.get_url(), "/catalog/"))
return handle_catalog(request);
if (request.get_url() == "/meta")
@@ -278,27 +315,6 @@ MustacheData InternalServer::get_default_data() const
return data;
}
MustacheData InternalServer::homepage_data() const
{
auto data = get_default_data();
MustacheData books{MustacheData::type::list};
for (auto& bookId: mp_library->filter(kiwix::Filter().local(true).valid(true))) {
auto& currentBook = mp_library->getBookById(bookId);
MustacheData book;
book.set("name", mp_nameMapper->getNameForId(bookId));
book.set("title", currentBook.getTitle());
book.set("description", currentBook.getDescription());
book.set("articleCount", beautifyInteger(currentBook.getArticleCount()));
book.set("mediaCount", beautifyInteger(currentBook.getMediaCount()));
books.push_back(book);
}
data.set("books", books);
return data;
}
bool InternalServer::etag_not_needed(const RequestContext& request) const
{
const std::string url = request.get_url();
@@ -322,51 +338,94 @@ InternalServer::get_matching_if_none_match_etag(const RequestContext& r) const
std::unique_ptr<Response> InternalServer::build_homepage(const RequestContext& request)
{
return ContentResponse::build(*this, RESOURCE::templates::index_html, homepage_data(), "text/html; charset=utf-8");
return ContentResponse::build(*this, m_indexTemplateString, get_default_data(), "text/html; charset=utf-8", true);
}
/**
* Archive and Zim handlers begin
**/
// TODO: retrieve searcher from caching mechanism
SuggestionsList_t getSuggestions(const zim::Archive* const archive,
const std::string& queryString, int start, int suggestionCount)
{
SuggestionsList_t suggestions;
auto searcher = zim::SuggestionSearcher(*archive);
if (archive->hasTitleIndex()) {
auto search = searcher.suggest(queryString);
auto srs = search.getResults(start, suggestionCount);
for (auto it : srs) {
SuggestionItem suggestion(it.getTitle(), kiwix::normalize(it.getTitle()),
it.getPath(), it.getSnippet());
suggestions.push_back(suggestion);
}
} else {
// TODO: This case should be handled by libzim
std::vector<std::string> variants = getTitleVariants(queryString);
int currCount = 0;
for (auto it = variants.begin(); it != variants.end() && currCount < suggestionCount; it++) {
auto search = searcher.suggest(queryString);
auto srs = search.getResults(0, suggestionCount);
for (auto it : srs) {
SuggestionItem suggestion(it.getTitle(), kiwix::normalize(it.getTitle()),
it.getPath());
suggestions.push_back(suggestion);
currCount++;
}
}
}
return suggestions;
}
/**
* Archive and Zim handlers end
**/
std::unique_ptr<Response> InternalServer::handle_meta(const RequestContext& request)
{
std::string bookName;
std::string bookId;
std::string meta_name;
std::shared_ptr<Reader> reader;
std::shared_ptr<zim::Archive> archive;
try {
bookName = request.get_argument("content");
bookId = mp_nameMapper->getIdForName(bookName);
meta_name = request.get_argument("name");
reader = mp_library->getReaderById(bookId);
archive = mp_library->getArchiveById(bookId);
} catch (const std::out_of_range& e) {
return Response::build_404(*this, request, bookName);
return Response::build_404(*this, request, bookName, "");
}
if (reader == nullptr) {
return Response::build_404(*this, request, bookName);
if (archive == nullptr) {
return Response::build_404(*this, request, bookName, "");
}
std::string content;
std::string mimeType = "text";
if (meta_name == "title") {
content = reader->getTitle();
content = getArchiveTitle(*archive);
} else if (meta_name == "description") {
content = reader->getDescription();
content = getMetaDescription(*archive);
} else if (meta_name == "language") {
content = reader->getLanguage();
content = getMetaLanguage(*archive);
} else if (meta_name == "name") {
content = reader->getName();
content = getMetaName(*archive);
} else if (meta_name == "tags") {
content = reader->getTags();
content = getMetaTags(*archive);
} else if (meta_name == "date") {
content = reader->getDate();
content = getMetaDate(*archive);
} else if (meta_name == "creator") {
content = reader->getCreator();
content = getMetaCreator(*archive);
} else if (meta_name == "publisher") {
content = reader->getPublisher();
content = getMetaPublisher(*archive);
} else if (meta_name == "favicon") {
reader->getFavicon(content, mimeType);
getArchiveFavicon(*archive, 48, content, mimeType);
} else if (const unsigned illustrationSize = parseIllustration(meta_name)) {
getArchiveFavicon(*archive, illustrationSize, content, mimeType);
} else {
return Response::build_404(*this, request, bookName);
return Response::build_404(*this, request, bookName, "");
}
auto response = ContentResponse::build(*this, content, mimeType);
@@ -382,49 +441,71 @@ std::unique_ptr<Response> InternalServer::handle_suggest(const RequestContext& r
std::string content;
std::string mimeType;
unsigned int maxSuggestionCount = 10;
unsigned int suggestionCount = 0;
std::string bookName;
std::string bookId;
std::string term;
std::shared_ptr<Reader> reader;
std::string queryString;
std::shared_ptr<zim::Archive> archive;
try {
bookName = request.get_argument("content");
bookId = mp_nameMapper->getIdForName(bookName);
term = request.get_argument("term");
reader = mp_library->getReaderById(bookId);
queryString = request.get_argument("term");
archive = mp_library->getArchiveById(bookId);
} catch (const std::out_of_range&) {
return Response::build_404(*this, request, bookName);
return Response::build_404(*this, request, bookName, "");
}
auto start = 0;
try {
start = request.get_argument<unsigned int>("start");
} catch (const std::exception&) {}
unsigned int count = 10;
try {
count = request.get_argument<unsigned int>("count");
} catch (const std::exception&) {}
if (count == 0) {
count = 10;
}
if (archive == nullptr) {
return Response::build_404(*this, request, bookName, "");
}
if (m_verbose.load()) {
printf("Searching suggestions for: \"%s\"\n", term.c_str());
printf("Searching suggestions for: \"%s\"\n", queryString.c_str());
}
MustacheData results{MustacheData::type::list};
bool first = true;
if (reader != nullptr) {
/* Get the suggestions */
SuggestionsList_t suggestions;
reader->searchSuggestionsSmart(term, maxSuggestionCount, suggestions);
for(auto& suggestion:suggestions) {
MustacheData result;
result.set("label", suggestion[0]);
result.set("value", suggestion[0]);
result.set("first", first);
first = false;
results.push_back(result);
suggestionCount++;
/* Get the suggestions */
SuggestionsList_t suggestions = getSuggestions(archive.get(), queryString, start, count);
for(auto& suggestion:suggestions) {
MustacheData result;
result.set("label", suggestion.getTitle());
if (suggestion.hasSnippet()) {
result.set("label", suggestion.getSnippet());
}
result.set("value", suggestion.getTitle());
result.set("kind", "path");
result.set("path", suggestion.getPath());
result.set("first", first);
first = false;
results.push_back(result);
}
/* Propose the fulltext search if possible */
if (reader->hasFulltextIndex()) {
if (archive->hasFulltextIndex()) {
MustacheData result;
result.set("label", "containing '" + term + "'...");
result.set("value", term + " ");
result.set("label", "containing '" + queryString + "'...");
result.set("value", queryString + " ");
result.set("kind", "pattern");
result.set("first", first);
results.push_back(result);
}
@@ -451,7 +532,7 @@ std::unique_ptr<Response> InternalServer::handle_skin(const RequestContext& requ
response->set_cacheable();
return std::move(response);
} catch (const ResourceNotFound& e) {
return Response::build_404(*this, request, "");
return Response::build_404(*this, request, "", "");
}
}
@@ -486,54 +567,34 @@ std::unique_ptr<Response> InternalServer::handle_search(const RequestContext& re
} catch(const std::out_of_range&) {}
catch(const std::invalid_argument&) {}
std::shared_ptr<Reader> reader(nullptr);
std::shared_ptr<zim::Archive> archive;
try {
reader = mp_library->getReaderById(bookId);
archive = mp_library->getArchiveById(bookId);
} catch (const std::out_of_range&) {}
/* Try first to load directly the article */
if (reader != nullptr && !patternString.empty()) {
std::string patternCorrespondingUrl;
auto variants = reader->getTitleVariants(patternString);
auto variantsItr = variants.begin();
while (patternCorrespondingUrl.empty() && variantsItr != variants.end()) {
try {
auto entry = reader->getEntryFromTitle(*variantsItr);
entry = entry.getFinalEntry();
patternCorrespondingUrl = entry.getPath();
break;
} catch(kiwix::NoEntry& e) {
variantsItr++;
}
}
/* If article found then redirect directly to it */
if (!patternCorrespondingUrl.empty()) {
auto redirectUrl = m_root + "/" + bookName + "/" + patternCorrespondingUrl;
return Response::build_redirect(*this, redirectUrl);
}
}
/* Make the search */
if ( (!reader && !bookName.empty())
if ( (!archive && !bookName.empty())
|| (patternString.empty() && ! has_geo_query) ) {
auto data = get_default_data();
data.set("pattern", encodeDiples(patternString));
auto response = ContentResponse::build(*this, RESOURCE::templates::no_search_result_html, data, "text/html; charset=utf-8");
response->set_taskbar(bookName, reader ? reader->getTitle() : "");
response->set_taskbar(bookName, archive ? getArchiveTitle(*archive) : "");
response->set_code(MHD_HTTP_NOT_FOUND);
return std::move(response);
}
Searcher searcher;
if (reader) {
searcher.add_reader(reader.get());
std::shared_ptr<zim::Searcher> searcher;
if (archive) {
searcher = std::make_shared<zim::Searcher>(*archive);
} else {
for (auto& bookId: mp_library->filter(kiwix::Filter().local(true).valid(true))) {
auto currentReader = mp_library->getReaderById(bookId);
if (currentReader) {
searcher.add_reader(currentReader.get());
auto currentArchive = mp_library->getArchiveById(bookId);
if (currentArchive) {
if (! searcher) {
searcher = std::make_shared<zim::Searcher>(*currentArchive);
} else {
searcher->addArchive(*currentArchive);
}
}
}
}
@@ -554,30 +615,37 @@ std::unique_ptr<Response> InternalServer::handle_search(const RequestContext& re
pageLength = 25;
}
auto end = start + pageLength;
/* Get the results */
try {
zim::Query query;
if (patternString.empty()) {
searcher.geo_search(latitude, longitude, distance,
start, end, m_verbose.load());
// Execute geo-search
if (m_verbose.load()) {
cout << "Performing geo query `" << distance << "&(" << latitude << ";" << longitude << ")'" << endl;
}
query.setQuery("");
query.setGeorange(latitude, longitude, distance);
} else {
searcher.search(patternString,
start, end, m_verbose.load());
// Execute Ft search
if (m_verbose.load()) {
cout << "Performing query `" << patternString << "'" << endl;
}
std::string queryString = removeAccents(patternString);
query.setQuery(queryString);
}
SearchRenderer renderer(&searcher, mp_nameMapper);
zim::Search search = searcher->search(query);
SearchRenderer renderer(search.getResults(start, pageLength), mp_nameMapper, start,
search.getEstimatedMatches());
renderer.setSearchPattern(patternString);
renderer.setSearchContent(bookName);
renderer.setProtocolPrefix(m_root + "/");
renderer.setSearchProtocolPrefix(m_root + "/search?");
renderer.setPageLength(pageLength);
auto response = ContentResponse::build(*this, renderer.getHtml(), "text/html; charset=utf-8");
response->set_taskbar(bookName, reader ? reader->getTitle() : "");
//changing status code if no result obtained
if(searcher.getEstimatedResultCount() == 0)
{
response->set_code(MHD_HTTP_NO_CONTENT);
}
response->set_taskbar(bookName, archive ? getArchiveTitle(*archive) : "");
return std::move(response);
} catch (const std::exception& e) {
@@ -594,24 +662,24 @@ std::unique_ptr<Response> InternalServer::handle_random(const RequestContext& re
std::string bookName;
std::string bookId;
std::shared_ptr<Reader> reader;
std::shared_ptr<zim::Archive> archive;
try {
bookName = request.get_argument("content");
bookId = mp_nameMapper->getIdForName(bookName);
reader = mp_library->getReaderById(bookId);
archive = mp_library->getArchiveById(bookId);
} catch (const std::out_of_range&) {
return Response::build_404(*this, request, bookName);
return Response::build_404(*this, request, bookName, "");
}
if (reader == nullptr) {
return Response::build_404(*this, request, bookName);
if (archive == nullptr) {
return Response::build_404(*this, request, bookName, "");
}
try {
auto entry = reader->getRandomPage();
return build_redirect(bookName, entry.getFinalEntry());
} catch(kiwix::NoEntry& e) {
return Response::build_404(*this, request, bookName);
auto entry = archive->getRandomEntry();
return build_redirect(bookName, getFinalItem(*archive, entry));
} catch(zim::EntryNotFound& e) {
return Response::build_404(*this, request, bookName, "");
}
}
@@ -623,7 +691,7 @@ std::unique_ptr<Response> InternalServer::handle_captured_external(const Request
} catch (const std::out_of_range& e) {}
if (source.empty())
return Response::build_404(*this, request, "");
return Response::build_404(*this, request, "", "");
auto data = get_default_data();
data.set("source", source);
@@ -642,11 +710,15 @@ std::unique_ptr<Response> InternalServer::handle_catalog(const RequestContext& r
host = request.get_header("Host");
url = request.get_url_part(1);
} catch (const std::out_of_range&) {
return Response::build_404(*this, request, "");
return Response::build_404(*this, request, "", "");
}
if (url == "v2") {
return handle_catalog_v2(request);
}
if (url != "searchdescription.xml" && url != "root.xml" && url != "search") {
return Response::build_404(*this, request, "");
return Response::build_404(*this, request, "", "");
}
if (url == "searchdescription.xml") {
@@ -655,62 +727,79 @@ std::unique_ptr<Response> InternalServer::handle_catalog(const RequestContext& r
}
zim::Uuid uuid;
kiwix::OPDSDumper opdsDumper;
kiwix::OPDSDumper opdsDumper(mp_library);
opdsDumper.setRootLocation(m_root);
opdsDumper.setSearchDescriptionUrl("catalog/searchdescription.xml");
opdsDumper.setLibrary(mp_library);
opdsDumper.setLibraryId(m_library_id);
std::vector<std::string> bookIdsToDump;
if (url == "root.xml") {
opdsDumper.setTitle("All zims");
uuid = zim::Uuid::generate(host);
bookIdsToDump = mp_library->filter(kiwix::Filter().valid(true).local(true).remote(true));
} else if (url == "search") {
auto filter = kiwix::Filter().valid(true).local(true).remote(true);
string query("<Empty query>");
size_t count(10);
size_t startIndex(0);
bookIdsToDump = search_catalog(request, opdsDumper);
uuid = zim::Uuid::generate();
}
auto response = ContentResponse::build(
*this,
opdsDumper.dumpOPDSFeed(bookIdsToDump, request.get_query()),
"application/atom+xml; profile=opds-catalog; kind=acquisition; charset=utf-8");
return std::move(response);
}
namespace
{
Filter get_search_filter(const RequestContext& request)
{
auto filter = kiwix::Filter().valid(true).local(true);
try {
query = request.get_argument("q");
filter.query(query);
filter.query(request.get_argument("q"));
} catch (const std::out_of_range&) {}
try {
filter.maxSize(extractFromString<unsigned long>(request.get_argument("maxsize")));
filter.maxSize(request.get_argument<unsigned long>("maxsize"));
} catch (...) {}
try {
filter.name(request.get_argument("name"));
} catch (const std::out_of_range&) {}
try {
filter.lang(request.get_argument("lang"));
filter.category(request.get_argument("category"));
} catch (const std::out_of_range&) {}
try {
count = extractFromString<unsigned long>(request.get_argument("count"));
} catch (...) {}
try {
startIndex = extractFromString<unsigned long>(request.get_argument("start"));
} catch (...) {}
filter.lang(request.get_argument("lang"));
} catch (const std::out_of_range&) {}
try {
filter.acceptTags(kiwix::split(request.get_argument("tag"), ";"));
} catch (...) {}
try {
filter.rejectTags(kiwix::split(request.get_argument("notag"), ";"));
} catch (...) {}
opdsDumper.setTitle("Search result for " + query);
uuid = zim::Uuid::generate();
bookIdsToDump = mp_library->filter(filter);
auto totalResults = bookIdsToDump.size();
bookIdsToDump.erase(bookIdsToDump.begin(), bookIdsToDump.begin()+startIndex);
if (count>0 && bookIdsToDump.size() > count) {
bookIdsToDump.resize(count);
}
opdsDumper.setOpenSearchInfo(totalResults, startIndex, bookIdsToDump.size());
}
return filter;
}
opdsDumper.setId(kiwix::to_string(uuid));
auto response = ContentResponse::build(
*this,
opdsDumper.dumpOPDSFeed(bookIdsToDump),
"application/atom+xml; profile=opds-catalog; kind=acquisition; charset=utf-8");
return std::move(response);
template<class T>
std::vector<T> subrange(const std::vector<T>& v, size_t s, size_t n)
{
const size_t e = std::min(v.size(), s+n);
return std::vector<T>(v.begin()+std::min(v.size(), s), v.begin()+e);
}
} // unnamed namespace
std::vector<std::string>
InternalServer::search_catalog(const RequestContext& request,
kiwix::OPDSDumper& opdsDumper)
{
const auto filter = get_search_filter(request);
const std::string q = filter.hasQuery()
? filter.getQuery()
: "<Empty query>";
std::vector<std::string> bookIdsToDump = mp_library->filter(filter);
const auto totalResults = bookIdsToDump.size();
const size_t count = request.get_optional_param("count", 10UL);
const size_t startIndex = request.get_optional_param("start", 0UL);
bookIdsToDump = subrange(bookIdsToDump, startIndex, count);
opdsDumper.setOpenSearchInfo(totalResults, startIndex, bookIdsToDump.size());
return bookIdsToDump;
}
namespace
@@ -725,29 +814,28 @@ std::string get_book_name(const RequestContext& request)
}
}
} // unnamed namespace
std::shared_ptr<Reader>
InternalServer::get_reader(const std::string& bookName) const
std::string searchSuggestionHTML(const std::string& searchURL, const std::string& pattern)
{
std::shared_ptr<Reader> reader;
try {
const std::string bookId = mp_nameMapper->getIdForName(bookName);
reader = mp_library->getReaderById(bookId);
} catch (const std::out_of_range& e) {
}
return reader;
kainjow::mustache::mustache tmpl("Make a full text search for <a href=\"{{{searchURL}}}\">{{pattern}}</a>");
MustacheData data;
data.set("pattern", pattern);
data.set("searchURL", searchURL);
return (tmpl.render(data));
}
} // unnamed namespace
std::unique_ptr<Response>
InternalServer::build_redirect(const std::string& bookName, const kiwix::Entry& entry) const
InternalServer::build_redirect(const std::string& bookName, const zim::Item& item) const
{
auto redirectUrl = m_root + "/" + bookName + "/" + kiwix::urlEncode(entry.getPath());
auto redirectUrl = m_root + "/" + bookName + "/" + kiwix::urlEncode(item.getPath());
return Response::build_redirect(*this, redirectUrl);
}
std::unique_ptr<Response> InternalServer::handle_content(const RequestContext& request)
{
const std::string url = request.get_url();
const std::string pattern = url.substr((url.find_last_of('/'))+1);
if (m_verbose.load()) {
printf("** running handle_content\n");
}
@@ -756,9 +844,17 @@ std::unique_ptr<Response> InternalServer::handle_content(const RequestContext& r
if (bookName.empty())
return build_homepage(request);
const std::shared_ptr<Reader> reader = get_reader(bookName);
if (reader == nullptr) {
return Response::build_404(*this, request, bookName);
std::shared_ptr<zim::Archive> archive;
try {
const std::string bookId = mp_nameMapper->getIdForName(bookName);
archive = mp_library->getArchiveById(bookId);
} catch (const std::out_of_range& e) {}
if (archive == nullptr) {
std::string searchURL = m_root+"/search?pattern="+pattern; // Make a full search on the entire library.
const std::string details = searchSuggestionHTML(searchURL, kiwix::urlDecode(pattern));
return Response::build_404(*this, request, bookName, "", details);
}
auto urlStr = request.get_url().substr(bookName.size()+1);
@@ -766,33 +862,33 @@ std::unique_ptr<Response> InternalServer::handle_content(const RequestContext& r
urlStr = urlStr.substr(1);
}
kiwix::Entry entry;
try {
entry = reader->getEntryFromPath(urlStr);
auto entry = getEntryFromPath(*archive, urlStr);
if (entry.isRedirect() || urlStr.empty()) {
// If urlStr is empty, we want to mainPage.
// We must do a redirection to the real page.
return build_redirect(bookName, entry.getFinalEntry());
return build_redirect(bookName, getFinalItem(*archive, entry));
}
} catch(kiwix::NoEntry& e) {
auto response = ItemResponse::build(*this, request, entry.getItem());
try {
dynamic_cast<ContentResponse&>(*response).set_taskbar(bookName, getArchiveTitle(*archive));
} catch (std::bad_cast& e) {}
if (m_verbose.load()) {
printf("Found %s\n", entry.getPath().c_str());
printf("mimeType: %s\n", entry.getItem(true).getMimetype().c_str());
}
return response;
} catch(zim::EntryNotFound& e) {
if (m_verbose.load())
printf("Failed to find %s\n", urlStr.c_str());
return Response::build_404(*this, request, bookName);
std::string searchURL = m_root+"/search?content="+bookName+"&pattern="+pattern; // Make a search on this specific book only.
const std::string details = searchSuggestionHTML(searchURL, kiwix::urlDecode(pattern));
return Response::build_404(*this, request, bookName, getArchiveTitle(*archive), details);
}
auto response = EntryResponse::build(*this, request, entry);
try {
dynamic_cast<ContentResponse&>(*response).set_taskbar(bookName, reader->getTitle());
} catch (std::bad_cast& e) {}
if (m_verbose.load()) {
printf("Found %s\n", entry.getPath().c_str());
printf("mimeType: %s\n", entry.getMimetype().c_str());
}
return response;
}
}

View File

@@ -41,6 +41,7 @@ namespace kiwix {
typedef kainjow::mustache::data MustacheData;
class Entry;
class OPDSDumper;
class InternalServer {
public:
@@ -53,7 +54,8 @@ class InternalServer {
bool verbose,
bool withTaskbar,
bool withLibraryButton,
bool blockExternalLinks);
bool blockExternalLinks,
std::string indexTemplateString);
virtual ~InternalServer() = default;
MHD_Result handlerCallback(struct MHD_Connection* connection,
@@ -68,10 +70,16 @@ class InternalServer {
private: // functions
std::unique_ptr<Response> handle_request(const RequestContext& request);
std::unique_ptr<Response> build_redirect(const std::string& bookName, const kiwix::Entry& entry) const;
std::unique_ptr<Response> build_redirect(const std::string& bookName, const zim::Item& item) const;
std::unique_ptr<Response> build_homepage(const RequestContext& request);
std::unique_ptr<Response> handle_skin(const RequestContext& request);
std::unique_ptr<Response> handle_catalog(const RequestContext& request);
std::unique_ptr<Response> handle_catalog_v2(const RequestContext& request);
std::unique_ptr<Response> handle_catalog_v2_root(const RequestContext& request);
std::unique_ptr<Response> handle_catalog_v2_entries(const RequestContext& request, bool partial);
std::unique_ptr<Response> handle_catalog_v2_complete_entry(const RequestContext& request, const std::string& entryId);
std::unique_ptr<Response> handle_catalog_v2_categories(const RequestContext& request);
std::unique_ptr<Response> handle_catalog_v2_languages(const RequestContext& request);
std::unique_ptr<Response> handle_meta(const RequestContext& request);
std::unique_ptr<Response> handle_search(const RequestContext& request);
std::unique_ptr<Response> handle_suggest(const RequestContext& request);
@@ -79,10 +87,11 @@ class InternalServer {
std::unique_ptr<Response> handle_captured_external(const RequestContext& request);
std::unique_ptr<Response> handle_content(const RequestContext& request);
MustacheData get_default_data() const;
MustacheData homepage_data() const;
std::vector<std::string> search_catalog(const RequestContext& request,
kiwix::OPDSDumper& opdsDumper);
MustacheData get_default_data() const;
std::shared_ptr<Reader> get_reader(const std::string& bookName) const;
bool etag_not_needed(const RequestContext& r) const;
ETag get_matching_if_none_match_etag(const RequestContext& request) const;
@@ -95,16 +104,18 @@ class InternalServer {
bool m_withTaskbar;
bool m_withLibraryButton;
bool m_blockExternalLinks;
std::string m_indexTemplateString;
struct MHD_Daemon* mp_daemon;
Library* mp_library;
NameMapper* mp_nameMapper;
std::string m_server_id;
std::string m_library_id;
friend std::unique_ptr<Response> Response::build(const InternalServer& server);
friend std::unique_ptr<ContentResponse> ContentResponse::build(const InternalServer& server, const std::string& content, const std::string& mimetype);
friend std::unique_ptr<Response> EntryResponse::build(const InternalServer& server, const RequestContext& request, const Entry& entry);
friend std::unique_ptr<ContentResponse> ContentResponse::build(const InternalServer& server, const std::string& content, const std::string& mimetype, bool isHomePage);
friend std::unique_ptr<Response> ItemResponse::build(const InternalServer& server, const RequestContext& request, const zim::Item& item);
friend std::unique_ptr<Response> Response::build_500(const InternalServer& server, const std::string& msg);
};

View File

@@ -0,0 +1,149 @@
/*
* Copyright 2021 Veloman Yunkan <veloman.yunkan@gmail.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
* MA 02110-1301, USA.
*/
#include "internalServer.h"
#include "library.h"
#include "opds_dumper.h"
#include "request_context.h"
#include "response.h"
#include "tools/otherTools.h"
#include "kiwixlib-resources.h"
#include <mustache.hpp>
#include <string>
#include <vector>
namespace kiwix {
std::unique_ptr<Response> InternalServer::handle_catalog_v2(const RequestContext& request)
{
if (m_verbose.load()) {
printf("** running handle_catalog_v2");
}
std::string url;
try {
url = request.get_url_part(2);
} catch (const std::out_of_range&) {
return Response::build_404(*this, request, "", "");
}
if (url == "root.xml") {
return handle_catalog_v2_root(request);
} else if (url == "searchdescription.xml") {
const std::string endpoint_root = m_root + "/catalog/v2";
return ContentResponse::build(*this,
RESOURCE::catalog_v2_searchdescription_xml,
kainjow::mustache::object({{"endpoint_root", endpoint_root}}),
"application/opensearchdescription+xml"
);
} else if (url == "entry") {
const std::string entryId = request.get_url_part(3);
return handle_catalog_v2_complete_entry(request, entryId);
} else if (url == "entries") {
return handle_catalog_v2_entries(request, /*partial=*/false);
} else if (url == "partial_entries") {
return handle_catalog_v2_entries(request, /*partial=*/true);
} else if (url == "categories") {
return handle_catalog_v2_categories(request);
} else if (url == "languages") {
return handle_catalog_v2_languages(request);
} else {
return Response::build_404(*this, request, "", "");
}
}
std::unique_ptr<Response> InternalServer::handle_catalog_v2_root(const RequestContext& request)
{
return ContentResponse::build(
*this,
RESOURCE::templates::catalog_v2_root_xml,
kainjow::mustache::object{
{"date", gen_date_str()},
{"endpoint_root", m_root + "/catalog/v2"},
{"feed_id", gen_uuid(m_library_id)},
{"all_entries_feed_id", gen_uuid(m_library_id + "/entries")},
{"partial_entries_feed_id", gen_uuid(m_library_id + "/partial_entries")},
{"category_list_feed_id", gen_uuid(m_library_id + "/categories")},
{"language_list_feed_id", gen_uuid(m_library_id + "/languages")}
},
"application/atom+xml;profile=opds-catalog;kind=navigation"
);
}
std::unique_ptr<Response> InternalServer::handle_catalog_v2_entries(const RequestContext& request, bool partial)
{
OPDSDumper opdsDumper(mp_library);
opdsDumper.setRootLocation(m_root);
opdsDumper.setLibraryId(m_library_id);
const auto bookIds = search_catalog(request, opdsDumper);
const auto opdsFeed = opdsDumper.dumpOPDSFeedV2(bookIds, request.get_query(), partial);
return ContentResponse::build(
*this,
opdsFeed,
"application/atom+xml;profile=opds-catalog;kind=acquisition"
);
}
std::unique_ptr<Response> InternalServer::handle_catalog_v2_complete_entry(const RequestContext& request, const std::string& entryId)
{
try {
mp_library->getBookById(entryId);
} catch (const std::out_of_range&) {
return Response::build_404(*this, request, "", "");
}
OPDSDumper opdsDumper(mp_library);
opdsDumper.setRootLocation(m_root);
opdsDumper.setLibraryId(m_library_id);
const auto opdsFeed = opdsDumper.dumpOPDSCompleteEntry(entryId);
return ContentResponse::build(
*this,
opdsFeed,
"application/atom+xml;type=entry;profile=opds-catalog"
);
}
std::unique_ptr<Response> InternalServer::handle_catalog_v2_categories(const RequestContext& request)
{
OPDSDumper opdsDumper(mp_library);
opdsDumper.setRootLocation(m_root);
opdsDumper.setLibraryId(m_library_id);
return ContentResponse::build(
*this,
opdsDumper.categoriesOPDSFeed(),
"application/atom+xml;profile=opds-catalog;kind=navigation"
);
}
std::unique_ptr<Response> InternalServer::handle_catalog_v2_languages(const RequestContext& request)
{
OPDSDumper opdsDumper(mp_library);
opdsDumper.setRootLocation(m_root);
opdsDumper.setLibraryId(m_library_id);
return ContentResponse::build(
*this,
opdsDumper.languagesOPDSFeed(),
"application/atom+xml;profile=opds-catalog;kind=navigation"
);
}
} // namespace kiwix

View File

@@ -183,4 +183,14 @@ std::string RequestContext::get_header(const std::string& name) const {
return headers.at(lcAll(name));
}
std::string RequestContext::get_query() const {
std::string q;
const char* sep = "";
for ( const auto& a : arguments ) {
q += sep + a.first + '=' + a.second;
sep = "&";
}
return q;
}
}

View File

@@ -74,11 +74,21 @@ class RequestContext {
return v;
}
template<class T>
T get_optional_param(const std::string& name, T default_value) const
{
try {
return get_argument<T>(name);
} catch (...) {}
return default_value;
}
RequestMethod get_method() const;
std::string get_url() const;
std::string get_url_part(int part) const;
std::string get_full_url() const;
std::string get_query() const;
ByteRange get_range() const;

View File

@@ -1,5 +1,21 @@
/*
* Copyright 2019 Matthieu Gautier <mgautier@kymeria.fr>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
* MA 02110-1301, USA.
*/
#include "response.h"
#include "request_context.h"
@@ -8,6 +24,7 @@
#include "tools/regexTools.h"
#include "tools/stringTools.h"
#include "tools/otherTools.h"
#include "string.h"
#include <mustache.hpp>
@@ -22,21 +39,10 @@ namespace
{
// some utilities
std::string render_template(const std::string& template_str, kainjow::mustache::data data)
{
kainjow::mustache::mustache tmpl(template_str);
kainjow::mustache::data urlencode{kainjow::mustache::lambda2{
[](const std::string& str,const kainjow::mustache::renderer& r) { return urlEncode(r(str), true); }}};
data.set("urlencoded", urlencode);
std::stringstream ss;
tmpl.render(data, [&ss](const std::string& str) { ss << str; });
return ss.str();
}
std::string get_mime_type(const kiwix::Entry& entry)
std::string get_mime_type(const zim::Item& item)
{
try {
return entry.getMimetype();
return item.getMimetype();
} catch (exception& e) {
return "application/octet-stream";
}
@@ -77,14 +83,15 @@ std::unique_ptr<Response> Response::build_304(const InternalServer& server, cons
return response;
}
std::unique_ptr<Response> Response::build_404(const InternalServer& server, const RequestContext& request, const std::string& bookName)
std::unique_ptr<Response> Response::build_404(const InternalServer& server, const RequestContext& request, const std::string& bookName, const std::string& bookTitle, const std::string& details)
{
MustacheData results;
results.set("url", request.get_full_url());
results.set("details", details);
auto response = ContentResponse::build(server, RESOURCE::templates::_404_html, results, "text/html");
response->set_code(MHD_HTTP_NOT_FOUND);
response->set_taskbar(bookName, "");
response->set_taskbar(bookName, bookTitle);
return std::move(response);
}
@@ -131,17 +138,17 @@ static MHD_Result print_key_value (void *cls, enum MHD_ValueKind kind,
struct RunningResponse {
kiwix::Entry entry;
zim::Item item;
int range_start;
RunningResponse(kiwix::Entry entry,
RunningResponse(zim::Item item,
int range_start) :
entry(entry),
item(item),
range_start(range_start)
{}
};
static ssize_t callback_reader_from_entry(void* cls,
static ssize_t callback_reader_from_item(void* cls,
uint64_t pos,
char* buf,
size_t max)
@@ -150,13 +157,13 @@ static ssize_t callback_reader_from_entry(void* cls,
size_t max_size_to_set = min<size_t>(
max,
response->entry.getSize() - pos - response->range_start);
response->item.getSize() - pos - response->range_start);
if (max_size_to_set <= 0) {
return MHD_CONTENT_READER_END_WITH_ERROR;
}
zim::Blob blob = response->entry.getBlob(response->range_start+pos, max_size_to_set);
zim::Blob blob = response->item.getData(response->range_start+pos, max_size_to_set);
memcpy(buf, blob.data(), max_size_to_set);
return max_size_to_set;
}
@@ -178,20 +185,18 @@ void print_response_info(int retCode, MHD_Response* response)
}
void ContentResponse::introduce_taskbar()
{
kainjow::mustache::data data;
data.set("root", m_root);
data.set("content", m_bookName);
data.set("hascontent", !m_bookName.empty());
data.set("hascontent", (!m_bookName.empty() && !m_bookTitle.empty()));
data.set("title", m_bookTitle);
data.set("withlibrarybutton", m_withLibraryButton);
auto head_content = render_template(RESOURCE::templates::head_part_html, data);
m_content = appendToFirstOccurence(
auto head_content = render_template(RESOURCE::templates::head_taskbar_html, data);
m_content = prependToFirstOccurence(
m_content,
"<head>",
"</head[ \\t]*>",
head_content);
auto taskbar_part = render_template(RESOURCE::templates::taskbar_part_html, data);
@@ -207,12 +212,19 @@ void ContentResponse::inject_externallinks_blocker()
kainjow::mustache::data data;
data.set("root", m_root);
auto script_tag = render_template(RESOURCE::templates::external_blocker_part_html, data);
m_content = appendToFirstOccurence(
m_content = prependToFirstOccurence(
m_content,
"<head>",
"</head[ \\t]*>",
script_tag);
}
void ContentResponse::inject_root_link(){
m_content = prependToFirstOccurence(
m_content,
"</head[ \\t]*>",
"<link type=\"root\" href=\"" + m_root + "\">");
}
bool
ContentResponse::can_compress(const RequestContext& request) const
{
@@ -239,6 +251,8 @@ MHD_Response*
ContentResponse::create_mhd_response(const RequestContext& request)
{
if (contentDecorationAllowed()) {
inject_root_link();
if (m_withTaskbar) {
introduce_taskbar();
}
@@ -325,26 +339,26 @@ ContentResponse::ContentResponse(const std::string& root, bool verbose, bool wit
add_header(MHD_HTTP_HEADER_CONTENT_TYPE, m_mimeType);
}
std::unique_ptr<ContentResponse> ContentResponse::build(const InternalServer& server, const std::string& content, const std::string& mimetype)
std::unique_ptr<ContentResponse> ContentResponse::build(const InternalServer& server, const std::string& content, const std::string& mimetype, bool isHomePage)
{
return std::unique_ptr<ContentResponse>(new ContentResponse(
server.m_root,
server.m_verbose.load(),
server.m_withTaskbar,
server.m_withTaskbar && !isHomePage,
server.m_withLibraryButton,
server.m_blockExternalLinks,
content,
mimetype));
}
std::unique_ptr<ContentResponse> ContentResponse::build(const InternalServer& server, const std::string& template_str, kainjow::mustache::data data, const std::string& mimetype) {
std::unique_ptr<ContentResponse> ContentResponse::build(const InternalServer& server, const std::string& template_str, kainjow::mustache::data data, const std::string& mimetype, bool isHomePage) {
auto content = render_template(template_str, data);
return ContentResponse::build(server, content, mimetype);
return ContentResponse::build(server, content, mimetype, isHomePage);
}
EntryResponse::EntryResponse(bool verbose, const Entry& entry, const std::string& mimetype, const ByteRange& byterange) :
ItemResponse::ItemResponse(bool verbose, const zim::Item& item, const std::string& mimetype, const ByteRange& byterange) :
Response(verbose),
m_entry(entry),
m_item(item),
m_mimeType(mimetype)
{
m_byteRange = byterange;
@@ -352,48 +366,46 @@ EntryResponse::EntryResponse(bool verbose, const Entry& entry, const std::string
add_header(MHD_HTTP_HEADER_CONTENT_TYPE, m_mimeType);
}
std::unique_ptr<Response> EntryResponse::build(const InternalServer& server, const RequestContext& request, const Entry& entry)
std::unique_ptr<Response> ItemResponse::build(const InternalServer& server, const RequestContext& request, const zim::Item& item)
{
const std::string mimetype = get_mime_type(entry);
auto byteRange = request.get_range().resolve(entry.getSize());
const std::string mimetype = get_mime_type(item);
auto byteRange = request.get_range().resolve(item.getSize());
const bool noRange = byteRange.kind() == ByteRange::RESOLVED_FULL_CONTENT;
if (noRange && is_compressible_mime_type(mimetype)) {
// Return a contentResponse
zim::Blob raw_content = entry.getBlob();
const std::string content = string(raw_content.data(), raw_content.size());
auto response = ContentResponse::build(server, content, mimetype);
auto response = ContentResponse::build(server, item.getData(), mimetype);
response->set_cacheable();
response->m_byteRange = byteRange;
return std::move(response);
}
if (byteRange.kind() == ByteRange::RESOLVED_UNSATISFIABLE) {
auto response = Response::build_416(server, entry.getSize());
auto response = Response::build_416(server, item.getSize());
response->set_cacheable();
return response;
}
return std::unique_ptr<Response>(new EntryResponse(
return std::unique_ptr<Response>(new ItemResponse(
server.m_verbose.load(),
entry,
item,
mimetype,
byteRange));
}
MHD_Response*
EntryResponse::create_mhd_response(const RequestContext& request)
ItemResponse::create_mhd_response(const RequestContext& request)
{
const auto content_length = m_byteRange.length();
MHD_Response* response = MHD_create_response_from_callback(content_length,
16384,
callback_reader_from_entry,
new RunningResponse(m_entry, m_byteRange.first()),
callback_reader_from_item,
new RunningResponse(m_item, m_byteRange.first()),
callback_free_response);
MHD_add_response_header(response, MHD_HTTP_HEADER_ACCEPT_RANGES, "bytes");
if ( m_byteRange.kind() == ByteRange::RESOLVED_PARTIAL_CONTENT ) {
std::ostringstream oss;
oss << "bytes " << m_byteRange.first() << "-" << m_byteRange.last()
<< "/" << m_entry.getSize();
<< "/" << m_item.getSize();
MHD_add_response_header(response,
MHD_HTTP_HEADER_CONTENT_RANGE, oss.str().c_str());

View File

@@ -47,7 +47,7 @@ class Response {
static std::unique_ptr<Response> build(const InternalServer& server);
static std::unique_ptr<Response> build_304(const InternalServer& server, const ETag& etag);
static std::unique_ptr<Response> build_404(const InternalServer& server, const RequestContext& request, const std::string& bookName);
static std::unique_ptr<Response> build_404(const InternalServer& server, const RequestContext& request, const std::string& bookName, const std::string& bookTitle, const std::string& details="");
static std::unique_ptr<Response> build_416(const InternalServer& server, size_t resourceLength);
static std::unique_ptr<Response> build_500(const InternalServer& server, const std::string& msg);
static std::unique_ptr<Response> build_redirect(const InternalServer& server, const std::string& redirectUrl);
@@ -72,15 +72,15 @@ class Response {
ETag m_etag;
std::map<std::string, std::string> m_customHeaders;
friend class EntryResponse; // temporary to allow the builder to change m_mode
friend class ItemResponse;
};
class ContentResponse : public Response {
public:
ContentResponse(const std::string& root, bool verbose, bool withTaskbar, bool withLibraryButton, bool blockExternalLinks, const std::string& content, const std::string& mimetype);
static std::unique_ptr<ContentResponse> build(const InternalServer& server, const std::string& content, const std::string& mimetype);
static std::unique_ptr<ContentResponse> build(const InternalServer& server, const std::string& template_str, kainjow::mustache::data data, const std::string& mimetype);
static std::unique_ptr<ContentResponse> build(const InternalServer& server, const std::string& content, const std::string& mimetype, bool isHomePage = false);
static std::unique_ptr<ContentResponse> build(const InternalServer& server, const std::string& template_str, kainjow::mustache::data data, const std::string& mimetype, bool isHomePage = false);
void set_taskbar(const std::string& bookName, const std::string& bookTitle);
@@ -89,6 +89,7 @@ class ContentResponse : public Response {
void introduce_taskbar();
void inject_externallinks_blocker();
void inject_root_link();
bool can_compress(const RequestContext& request) const;
bool contentDecorationAllowed() const;
@@ -104,15 +105,15 @@ class ContentResponse : public Response {
std::string m_bookTitle;
};
class EntryResponse : public Response {
class ItemResponse : public Response {
public:
EntryResponse(bool verbose, const Entry& entry, const std::string& mimetype, const ByteRange& byterange);
static std::unique_ptr<Response> build(const InternalServer& server, const RequestContext& request, const Entry& entry);
ItemResponse(bool verbose, const zim::Item& item, const std::string& mimetype, const ByteRange& byterange);
static std::unique_ptr<Response> build(const InternalServer& server, const RequestContext& request, const zim::Item& item);
private:
MHD_Response* create_mhd_response(const RequestContext& request);
Entry m_entry;
zim::Item m_item;
std::string m_mimeType;
};

View File

@@ -12,46 +12,31 @@
UnixImpl::UnixImpl():
m_pid(0),
m_running(false),
m_mutex(PTHREAD_MUTEX_INITIALIZER),
m_waitingThread()
m_shouldQuit(false)
{
}
UnixImpl::~UnixImpl()
{
kill();
// Android has no pthread_cancel :(
#ifdef __ANDROID__
pthread_kill(m_waitingThread, SIGUSR1);
#else
pthread_cancel(m_waitingThread);
#endif
pthread_join(m_waitingThread, nullptr);
m_shouldQuit = true;
m_waitingThread.join();
}
#ifdef __ANDROID__
void thread_exit_handler(int sig) {
pthread_exit(0);
}
#endif
void* UnixImpl::waitForPID(void* _self)
{
#ifdef __ANDROID__
struct sigaction actions;
memset(&actions, 0, sizeof(actions));
sigemptyset(&actions.sa_mask);
actions.sa_flags = 0;
actions.sa_handler = thread_exit_handler;
sigaction(SIGUSR1, &actions, NULL);
#endif
UnixImpl* self = static_cast<UnixImpl*>(_self);
waitpid(self->m_pid, NULL, 0);
while (true) {
if (!waitpid(self->m_pid, NULL, WNOHANG)) {
break;
}
if (self->m_shouldQuit) {
return nullptr;
}
std::this_thread::sleep_for(std::chrono::milliseconds(100));
}
pthread_mutex_lock(&self->m_mutex);
self->m_running = false;
pthread_mutex_unlock(&self->m_mutex);
return self;
}
@@ -75,7 +60,7 @@ void UnixImpl::run(commandLine_t& commandLine)
default:
m_pid = pid;
m_running = true;
pthread_create(&m_waitingThread, NULL, waitForPID, this);
m_waitingThread = std::thread(waitForPID, this);
break;
}
}
@@ -87,8 +72,5 @@ bool UnixImpl::kill()
bool UnixImpl::isRunning()
{
pthread_mutex_lock(&m_mutex);
bool ret = m_running;
pthread_mutex_unlock(&m_mutex);
return ret;
return m_running;
}

View File

@@ -3,16 +3,16 @@
#include "subprocess.h"
#include <pthread.h>
#include <atomic>
#include <thread>
class UnixImpl : public SubprocessImpl
{
private:
int m_pid;
bool m_running;
pthread_mutex_t m_mutex;
pthread_t m_waitingThread;
std::atomic<bool> m_running;
std::atomic<bool> m_shouldQuit;
std::thread m_waitingThread;
public:
UnixImpl();

158
src/tools/archiveTools.cpp Normal file
View File

@@ -0,0 +1,158 @@
/*
* Copyright 2021 Maneesh P M <manu.pm55@gmail.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
* MA 02110-1301, USA.
*/
#include "archiveTools.h"
#include "tools.h"
#include "pathTools.h"
#include "otherTools.h"
#include "stringTools.h"
#include <zim/error.h>
#include <zim/item.h>
namespace kiwix
{
std::string getMetadata(const zim::Archive& archive, const std::string& name) {
try {
return archive.getMetadata(name);
} catch (zim::EntryNotFound& e) {
return "";
}
}
std::string getArchiveTitle(const zim::Archive& archive) {
std::string value = getMetadata(archive, "Title");
if (value.empty()) {
value = getLastPathElement(archive.getFilename());
std::replace(value.begin(), value.end(), '_', ' ');
size_t pos = value.find(".zim");
value = value.substr(0, pos);
}
return value;
}
std::string getMetaDescription(const zim::Archive& archive) {
std::string value;
value = getMetadata(archive, "Description");
/* Mediawiki Collection tends to use the "Subtitle" name */
if (value.empty()) {
value = getMetadata(archive, "Subtitle");
}
return value;
}
std::string getMetaTags(const zim::Archive& archive, bool original) {
std::string tags_str = getMetadata(archive, "Tags");
if (original) {
return tags_str;
}
auto tags = convertTags(tags_str);
return join(tags, ";");
}
std::string getMetaLanguage(const zim::Archive& archive) {
return getMetadata(archive, "Language");
}
std::string getMetaName(const zim::Archive& archive) {
return getMetadata(archive, "Name");
}
std::string getMetaDate(const zim::Archive& archive) {
return getMetadata(archive, "Date");
}
std::string getMetaCreator(const zim::Archive& archive) {
return getMetadata(archive, "Creator");
}
std::string getMetaPublisher(const zim::Archive& archive) {
return getMetadata(archive, "Publisher");
}
std::string getMetaFlavour(const zim::Archive& archive) {
return getMetadata(archive, "Flavour");
}
std::string getArchiveId(const zim::Archive& archive) {
return (std::string) archive.getUuid();
}
bool getArchiveFavicon(const zim::Archive& archive, unsigned size,
std::string& content, std::string& mimeType){
try {
auto item = archive.getIllustrationItem(size);
content = item.getData();
mimeType = item.getMimetype();
return true;
} catch(zim::EntryNotFound& e) {};
return false;
}
// should this be in libzim
unsigned int getArchiveMediaCount(const zim::Archive& archive) {
std::map<const std::string, unsigned int> counterMap = parseArchiveCounter(archive);
unsigned int counter = 0;
for (auto &pair:counterMap) {
if (startsWith(pair.first, "image/") ||
startsWith(pair.first, "video/") ||
startsWith(pair.first, "audio/")) {
counter += pair.second;
}
}
return counter;
}
unsigned int getArchiveFileSize(const zim::Archive& archive) {
return archive.getFilesize() / 1024;
}
zim::Item getFinalItem(const zim::Archive& archive, const zim::Entry& entry)
{
return entry.getItem(true);
}
zim::Entry getEntryFromPath(const zim::Archive& archive, const std::string& path)
{
try {
return archive.getEntryByPath(path);
} catch (zim::EntryNotFound& e) {
if (path.empty() || path == "/") {
return archive.getMainEntry();
}
}
throw zim::EntryNotFound("Cannot find entry for non empty path");
}
MimeCounterType parseArchiveCounter(const zim::Archive& archive) {
try {
auto counterContent = archive.getMetadata("Counter");
return parseMimetypeCounter(counterContent);
} catch (zim::EntryNotFound& e) {
return {};
}
}
} // kiwix

59
src/tools/archiveTools.h Normal file
View File

@@ -0,0 +1,59 @@
/*
* Copyright 2021 Maneesh P M <manu.pm55@gmail.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
* MA 02110-1301, USA.
*/
#ifndef KIWIX_ARCHIVETOOLS_H
#define KIWIX_ARCHIVETOOLS_H
#include <zim/archive.h>
#include <tools/otherTools.h>
/**
* This file contains all the functions that would make handling data related to
* an archive easier.
**/
namespace kiwix
{
std::string getMetadata(const zim::Archive& archive, const std::string& name);
std::string getArchiveTitle(const zim::Archive& archive);
std::string getMetaDescription(const zim::Archive& archive);
std::string getMetaTags(const zim::Archive& archive, bool original = false);
std::string getMetaLanguage(const zim::Archive& archive);
std::string getMetaName(const zim::Archive& archive);
std::string getMetaDate(const zim::Archive& archive);
std::string getMetaCreator(const zim::Archive& archive);
std::string getMetaPublisher(const zim::Archive& archive);
std::string getMetaFlavour(const zim::Archive& archive);
std::string getArchiveId(const zim::Archive& archive);
bool getArchiveFavicon(const zim::Archive& archive, unsigned size,
std::string& content, std::string& mimeType);
unsigned int getArchiveMediaCount(const zim::Archive& archive);
unsigned int getArchiveFileSize(const zim::Archive& archive);
zim::Item getFinalItem(const zim::Archive& archive, const zim::Entry& entry);
zim::Entry getEntryFromPath(const zim::Archive& archive, const std::string& path);
MimeCounterType parseArchiveCounter(const zim::Archive& archive);
}
#endif

View File

@@ -17,8 +17,13 @@
* MA 02110-1301, USA.
*/
// Implement function declared in tools.h and tools/otherTools.h
#include "tools.h"
#include "tools/otherTools.h"
#include <algorithm>
#include <iomanip>
#ifdef _WIN32
#include <windows.h>
@@ -32,6 +37,8 @@
#include <sstream>
#include <pugixml.hpp>
#include <zim/uuid.h>
static std::map<std::string, std::string> codeisomapping {
{ "aa", "aar" },
@@ -341,3 +348,35 @@ kiwix::MimeCounterType kiwix::parseMimetypeCounter(const std::string& counterDat
return counters;
}
std::string kiwix::gen_date_str()
{
auto now = std::time(0);
auto tm = std::localtime(&now);
std::stringstream is;
is << std::setw(2) << std::setfill('0')
<< 1900+tm->tm_year << "-"
<< std::setw(2) << std::setfill('0') << tm->tm_mon+1 << "-"
<< std::setw(2) << std::setfill('0') << tm->tm_mday << "T"
<< std::setw(2) << std::setfill('0') << tm->tm_hour << ":"
<< std::setw(2) << std::setfill('0') << tm->tm_min << ":"
<< std::setw(2) << std::setfill('0') << tm->tm_sec << "Z";
return is.str();
}
std::string kiwix::gen_uuid(const std::string& s)
{
return kiwix::to_string(zim::Uuid::generate(s));
}
std::string kiwix::render_template(const std::string& template_str, kainjow::mustache::data data)
{
kainjow::mustache::mustache tmpl(template_str);
kainjow::mustache::data urlencode{kainjow::mustache::lambda2{
[](const std::string& str,const kainjow::mustache::renderer& r) { return urlEncode(r(str), true); }}};
data.set("urlencoded", urlencode);
std::stringstream ss;
tmpl.render(data, [&ss](const std::string& str) { ss << str; });
return ss.str();
}

View File

@@ -24,6 +24,7 @@
#include <vector>
#include <map>
#include <zim/zim.h>
#include <mustache.hpp>
namespace pugi {
class xml_node;
@@ -31,9 +32,7 @@ namespace pugi {
namespace kiwix
{
void sleep(unsigned int milliseconds);
std::string nodeToString(const pugi::xml_node& node);
std::string converta2toa3(const std::string& a2code);
/*
* Convert all format tag string to new format
@@ -43,8 +42,13 @@ namespace kiwix
const std::string& tagName);
bool convertStrToBool(const std::string& value);
using MimeCounterType = std::map<const std::string, zim::article_index_type>;
using MimeCounterType = std::map<const std::string, zim::entry_index_type>;
MimeCounterType parseMimetypeCounter(const std::string& counterData);
std::string gen_date_str();
std::string gen_uuid(const std::string& s);
std::string render_template(const std::string& template_str, kainjow::mustache::data data);
}
#endif

View File

@@ -17,7 +17,10 @@
* MA 02110-1301, USA.
*/
// Implement method defined in <kiwix/tools.h> and "tools/pathTools.h"
#include "tools.h"
#include "tools/pathTools.h"
#include <stdexcept>
#ifdef __APPLE__
@@ -59,7 +62,6 @@
#define PATH_MAX 1024
#endif
#ifdef _WIN32
std::string WideToUtf8(const std::wstring& wstr)
{
@@ -78,7 +80,7 @@ std::wstring Utf8ToWide(const std::string& str)
}
#endif
bool isRelativePath(const std::string& path)
bool kiwix::isRelativePath(const std::string& path)
{
#ifdef _WIN32
if (path.size() < 3 ) {
@@ -173,7 +175,7 @@ std::vector<std::string> normalizeParts(std::vector<std::string>& parts, bool ab
return ret;
}
std::string computeRelativePath(const std::string& path, const std::string& absolutePath)
std::string kiwix::computeRelativePath(const std::string& path, const std::string& absolutePath)
{
auto parts = kiwix::split(path, SEPARATOR, false);
auto pathParts = normalizeParts(parts, false);
@@ -198,11 +200,11 @@ std::string computeRelativePath(const std::string& path, const std::string& abso
return ret;
}
std::string computeAbsolutePath(const std::string& path, const std::string& relativePath)
std::string kiwix::computeAbsolutePath(const std::string& path, const std::string& relativePath)
{
std::string absolutePath = path;
if (path.empty()) {
absolutePath = getCurrentDirectory();
absolutePath = kiwix::getCurrentDirectory();
}
auto parts = kiwix::split(absolutePath, SEPARATOR, false);
@@ -215,7 +217,7 @@ std::string computeAbsolutePath(const std::string& path, const std::string& rela
return ret;
}
std::string removeLastPathElement(const std::string& path)
std::string kiwix::removeLastPathElement(const std::string& path)
{
auto parts_ = kiwix::split(path, SEPARATOR, false);
auto parts = normalizeParts(parts_, false);
@@ -226,7 +228,7 @@ std::string removeLastPathElement(const std::string& path)
return ret;
}
std::string appendToDirectory(const std::string& directoryPath, const std::string& filename)
std::string kiwix::appendToDirectory(const std::string& directoryPath, const std::string& filename)
{
std::string newPath = directoryPath;
if (!directoryPath.empty() && directoryPath.back() != SEPARATOR[0]) {
@@ -236,7 +238,7 @@ std::string appendToDirectory(const std::string& directoryPath, const std::strin
return newPath;
}
std::string getLastPathElement(const std::string& path)
std::string kiwix::getLastPathElement(const std::string& path)
{
auto parts_ = kiwix::split(path, SEPARATOR);
auto parts = normalizeParts(parts_, false);
@@ -267,7 +269,7 @@ std::string getFileSizeAsString(const std::string& path)
return convert.str();
}
std::string getFileContent(const std::string& path)
std::string kiwix::getFileContent(const std::string& path)
{
#ifdef _WIN32
auto wpath = Utf8ToWide(path);
@@ -300,7 +302,7 @@ std::string getFileContent(const std::string& path)
return content;
}
bool fileExists(const std::string& path)
bool kiwix::fileExists(const std::string& path)
{
#ifdef _WIN32
return PathFileExistsW(Utf8ToWide(path).c_str());
@@ -339,7 +341,7 @@ std::string makeTmpDirectory()
_wmkdir(ctmp);
return WideToUtf8(ctmp);
#else
char _template_array[] = {"/tmp/kiwix-lib_XXXXXX"};
char _template_array[] = {"/tmp/libkiwix_XXXXXX"};
std::string dir = mkdtemp(_template_array);
return dir;
#endif
@@ -366,7 +368,7 @@ bool copyFile(const std::string& sourcePath, const std::string& destPath)
#endif
}
std::string getExecutablePath(bool realPathOnly)
std::string kiwix::getExecutablePath(bool realPathOnly)
{
if (!realPathOnly) {
char* cAppImage = ::getenv("APPIMAGE");
@@ -420,7 +422,7 @@ bool writeTextFile(const std::string& path, const std::string& content)
return true;
}
std::string getCurrentDirectory()
std::string kiwix::getCurrentDirectory()
{
#ifdef _WIN32
wchar_t* a_cwd = _wgetcwd(NULL, 0);
@@ -434,7 +436,7 @@ std::string getCurrentDirectory()
return ret;
}
std::string getDataDirectory()
std::string kiwix::getDataDirectory()
{
// Try to get the dataDir from the `KIWIX_DATA_DIR` env var
#ifdef _WIN32
@@ -503,7 +505,7 @@ static std::map<std::string, std::string> extMimeTypes = {
};
/* Try to get the mimeType from the file extension */
std::string getMimeTypeForFile(const std::string& filename)
std::string kiwix::getMimeTypeForFile(const std::string& filename)
{
std::string mimeType = "text/plain";
auto pos = filename.find_last_of(".");
@@ -524,4 +526,3 @@ std::string getMimeTypeForFile(const std::string& filename)
return mimeType;
}

View File

@@ -26,23 +26,13 @@
std::string WideToUtf8(const std::wstring& wstr);
std::wstring Utf8ToWide(const std::string& str);
#endif
bool isRelativePath(const std::string& path);
std::string computeAbsolutePath(const std::string& path, const std::string& relativePath);
std::string computeRelativePath(const std::string& path, const std::string& absolutePath);
std::string removeLastPathElement(const std::string& path);
std::string appendToDirectory(const std::string& directoryPath, const std::string& filename);
unsigned int getFileSize(const std::string& path);
std::string getFileSizeAsString(const std::string& path);
std::string getFileContent(const std::string& path);
bool fileExists(const std::string& path);
bool makeDirectory(const std::string& path);
std::string makeTmpDirectory();
bool copyFile(const std::string& sourcePath, const std::string& destPath);
std::string getLastPathElement(const std::string& path);
std::string getExecutablePath(bool realPathOnly = false);
std::string getCurrentDirectory();
std::string getDataDirectory();
bool writeTextFile(const std::string& path, const std::string& content);
std::string getMimeTypeForFile(const std::string& filename);
#endif

View File

@@ -18,7 +18,6 @@
*/
#include <tools/regexTools.h>
#include <tools/lock.h>
#include <unicode/regex.h>
#include <unicode/ucnv.h>
@@ -26,10 +25,10 @@
#include <memory>
#include <map>
#include <stdexcept>
#include <pthread.h>
#include <mutex>
std::map<std::string, std::shared_ptr<icu::RegexPattern>> regexCache;
static pthread_mutex_t regexLock = PTHREAD_MUTEX_INITIALIZER;
static std::mutex regexLock;
std::unique_ptr<icu::RegexMatcher> buildMatcher(const std::string& regex, icu::UnicodeString& content)
{
@@ -39,7 +38,7 @@ std::unique_ptr<icu::RegexMatcher> buildMatcher(const std::string& regex, icu::U
pattern = regexCache.at(regex);
} catch (std::out_of_range&) {
// Redo the search with a lock to avoid race condition.
kiwix::Lock l(&regexLock);
std::lock_guard<std::mutex> l(regexLock);
try {
pattern = regexCache.at(regex);
} catch (std::out_of_range&) {
@@ -95,3 +94,22 @@ std::string appendToFirstOccurence(const std::string& content,
return content;
}
std::string prependToFirstOccurence(const std::string& content,
const std::string& regex,
const std::string& replacement)
{
ucnv_setDefaultName("UTF-8");
icu::UnicodeString ucontent(content.c_str());
icu::UnicodeString ureplacement(replacement.c_str());
auto matcher = buildMatcher(regex, ucontent);
if (matcher->find()) {
UErrorCode status = U_ZERO_ERROR;
ucontent.insert(matcher->start(status), ureplacement);
std::string tmp;
ucontent.toUTF8String(tmp);
return tmp;
}
return content;
}

View File

@@ -29,5 +29,8 @@ std::string replaceRegex(const std::string& content,
std::string appendToFirstOccurence(const std::string& content,
const std::string& regex,
const std::string& replacement);
std::string prependToFirstOccurence(const std::string& content,
const std::string& regex,
const std::string& replacement);
#endif

View File

@@ -17,9 +17,11 @@
* MA 02110-1301, USA.
*/
#include <tools/stringTools.h>
// Implement function declared in tools.h and tools/stringTools.h
#include "tools.h"
#include "tools/stringTools.h"
#include <tools/pathTools.h>
#include "tools/pathTools.h"
#include <unicode/normlzr.h>
#include <unicode/rep.h>
#include <unicode/translit.h>
@@ -268,7 +270,7 @@ std::string kiwix::urlDecode(const std::string& value, bool component)
/* Split string in a token array */
std::vector<std::string> kiwix::split(const std::string& str,
const std::string& delims,
bool trimEmpty,
bool dropEmpty,
bool keepDelim)
{
std::string::size_type lastPos = 0;
@@ -277,7 +279,7 @@ std::vector<std::string> kiwix::split(const std::string& str,
while( (pos = str.find_first_of(delims, lastPos)) < str.length() )
{
auto token = str.substr(lastPos, pos - lastPos);
if (!trimEmpty || !token.empty()) {
if (!dropEmpty || !token.empty()) {
tokens.push_back(token);
}
if (keepDelim) {
@@ -287,7 +289,7 @@ std::vector<std::string> kiwix::split(const std::string& str,
}
auto token = str.substr(lastPos);
if (!trimEmpty || !token.empty()) {
if (!dropEmpty || !token.empty()) {
tokens.push_back(token);
}
return tokens;
@@ -395,3 +397,11 @@ bool kiwix::startsWith(const std::string& base, const std::string& start)
&& std::equal(start.begin(), start.end(), base.begin());
}
std::vector<std::string> kiwix::getTitleVariants(const std::string& title) {
std::vector<std::string> variants;
variants.push_back(title);
variants.push_back(kiwix::ucFirst(title));
variants.push_back(kiwix::lcFirst(title));
variants.push_back(kiwix::toTitle(title));
return variants;
}

View File

@@ -43,7 +43,6 @@ void loadICUExternalTables();
std::string urlEncode(const std::string& value, bool encodeReserved = false);
std::string urlDecode(const std::string& value, bool component = false);
std::vector<std::string> split(const std::string& str, const std::string& delims, bool trimEmpty = true, bool keepDelim = false);
std::string join(const std::vector<std::string>& list, const std::string& sep);
std::string ucAll(const std::string& word);
@@ -70,5 +69,7 @@ T extractFromString(const std::string& str) {
}
bool startsWith(const std::string& base, const std::string& start);
std::vector<std::string> getTitleVariants(const std::string& title);
} //namespace kiwix
#endif

View File

@@ -69,6 +69,7 @@ GETTER(jstring, getDate)
GETTER(jstring, getUrl)
GETTER(jstring, getName)
GETTER(jstring, getFlavour)
GETTER(jstring, getCategory)
GETTER(jstring, getTags)
GETTER(jlong, getArticleCount)
GETTER(jlong, getMediaCount)

View File

@@ -28,11 +28,7 @@
#include "utils.h"
#if __ANDROID__
pthread_mutex_t globalLock = PTHREAD_RECURSIVE_MUTEX_INITIALIZER;
#else
pthread_mutex_t globalLock = PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP;
#endif
std::mutex globalLock;
JNIEXPORT void JNICALL Java_org_kiwix_kiwixlib_JNIICU_setDataDirectory(
JNIEnv* env, jclass kclass, jstring dirStr)

View File

@@ -20,7 +20,6 @@
#include <jni.h>
#include <zim/file.h>
#include <exception>
#include "org_kiwix_kiwixlib_JNIKiwixReader.h"
@@ -46,6 +45,72 @@ JNIEXPORT jlong JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getNativeReader(
}
}
namespace
{
int jni2fd(const jobject& fdObj, JNIEnv* env)
{
jclass class_fdesc = env->FindClass("java/io/FileDescriptor");
jfieldID field_fd = env->GetFieldID(class_fdesc, "fd", "I");
if ( field_fd == NULL )
{
env->ExceptionClear();
// Under Android the (private) 'fd' field of java.io.FileDescriptor has been
// renamed to 'descriptor'. See, for example,
// https://android.googlesource.com/platform/libcore/+/refs/tags/android-8.1.0_r1/ojluni/src/main/java/java/io/FileDescriptor.java#55
field_fd = env->GetFieldID(class_fdesc, "descriptor", "I");
}
return env->GetIntField(fdObj, field_fd);
}
} // unnamed namespace
JNIEXPORT jlong JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getNativeReaderByFD(
JNIEnv* env, jobject obj, jobject fdObj)
{
#ifndef _WIN32
int fd = jni2fd(fdObj, env);
LOG("Attempting to create reader with fd: %d", fd);
Lock l;
try {
kiwix::Reader* reader = new kiwix::Reader(fd);
return reinterpret_cast<jlong>(new Handle<kiwix::Reader>(reader));
} catch (std::exception& e) {
LOG("Error opening ZIM file");
LOG(e.what());
return 0;
}
#else
jclass exception = env->FindClass("java/lang/UnsupportedOperationException");
env->ThrowNew(exception, "org.kiwix.kiwixlib.JNIKiwixReader.getNativeReaderByFD() is not supported under Windows");
return 0;
#endif
}
JNIEXPORT jlong JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getNativeReaderEmbedded(
JNIEnv* env, jobject obj, jobject fdObj, jlong offset, jlong size)
{
#ifndef _WIN32
int fd = jni2fd(fdObj, env);
LOG("Attempting to create reader with fd: %d", fd);
Lock l;
try {
kiwix::Reader* reader = new kiwix::Reader(fd, offset, size);
return reinterpret_cast<jlong>(new Handle<kiwix::Reader>(reader));
} catch (std::exception& e) {
LOG("Error opening ZIM file");
LOG(e.what());
return 0;
}
#else
jclass exception = env->FindClass("java/lang/UnsupportedOperationException");
env->ThrowNew(exception, "org.kiwix.kiwixlib.JNIKiwixReader.getNativeReaderEmbedded() is not supported under Windows");
return 0;
#endif
}
JNIEXPORT void JNICALL
Java_org_kiwix_kiwixlib_JNIKiwixReader_dispose(JNIEnv* env, jobject obj)
{
@@ -326,22 +391,22 @@ JNIEXPORT jobject JNICALL
Java_org_kiwix_kiwixlib_JNIKiwixReader_getDirectAccessInformation(
JNIEnv* env, jobject obj, jstring url)
{
jclass classPair = env->FindClass("org/kiwix/kiwixlib/Pair");
jmethodID midPairinit = env->GetMethodID(classPair, "<init>", "()V");
jobject pair = env->NewObject(classPair, midPairinit);
setPairObjValue("", 0, pair, env);
jclass daiClass = env->FindClass("org/kiwix/kiwixlib/DirectAccessInfo");
jmethodID daiInitMethod = env->GetMethodID(daiClass, "<init>", "()V");
jobject dai = env->NewObject(daiClass, daiInitMethod);
setDaiObjValue("", 0, dai, env);
std::string cUrl = jni2c(url, env);
try {
auto entry = READER->getEntryFromEncodedPath(cUrl);
entry = entry.getFinalEntry();
auto part_info = entry.getDirectAccessInfo();
setPairObjValue(part_info.first, part_info.second, pair, env);
setDaiObjValue(part_info.first, part_info.second, dai, env);
} catch (std::exception& e) {
LOG("Unable to get direct access info for url: %s", cUrl.c_str());
LOG(e.what());
}
return pair;
return dai;
}
JNIEXPORT jboolean JNICALL

View File

@@ -19,7 +19,6 @@
*/
#include <zim/file.h>
#include "org_kiwix_kiwixlib_JNIKiwixSearcher.h"
#include "org_kiwix_kiwixlib_JNIKiwixSearcher_Result.h"

View File

@@ -20,7 +20,6 @@
#include <jni.h>
#include <zim/file.h>
#include "org_kiwix_kiwixlib_JNIKiwixServer.h"
#include "tools/base64.h"

View File

@@ -19,7 +19,6 @@
#include <jni.h>
#include <zim/file.h>
#include "org_kiwix_kiwixlib_Manager.h"
#include "manager.h"

View File

@@ -12,7 +12,7 @@ java_sources = files([
'org/kiwix/kiwixlib/JNIKiwixString.java',
'org/kiwix/kiwixlib/JNIKiwixBool.java',
'org/kiwix/kiwixlib/JNIKiwixException.java',
'org/kiwix/kiwixlib/Pair.java'
'org/kiwix/kiwixlib/DirectAccessInfo.java'
])
kiwix_jni = custom_target('jni',

View File

@@ -24,6 +24,7 @@ public class Book
public native String getUrl();
public native String getName();
public native String getFlavour();
public native String getCategory();
public native String getTags();
/**
* Return the value associated to the tag tagName

View File

@@ -19,7 +19,7 @@
package org.kiwix.kiwixlib;
public class Pair
public class DirectAccessInfo
{
public String filename;
public long offset;

View File

@@ -24,7 +24,8 @@ import org.kiwix.kiwixlib.JNIKiwixException;
import org.kiwix.kiwixlib.JNIKiwixString;
import org.kiwix.kiwixlib.JNIKiwixInt;
import org.kiwix.kiwixlib.JNIKiwixSearcher;
import org.kiwix.kiwixlib.Pair;
import org.kiwix.kiwixlib.DirectAccessInfo;
import java.io.FileDescriptor;
public class JNIKiwixReader
{
@@ -102,13 +103,13 @@ public class JNIKiwixReader
* the zim file (or zim part) and directly read the content from it (and so
* bypassing the libzim).
*
* Return a `Pair` (filename, offset) where the content is located.
* Return a `DirectAccessInfo` (filename, offset) where the content is located.
*
* If the content cannot be directly accessed (content is compressed or zim
* file is cut in the middle of the content), the filename is an empty string
* and offset is zero.
*/
public native Pair getDirectAccessInformation(String url);
public native DirectAccessInfo getDirectAccessInformation(String url);
public native boolean searchSuggestions(String prefix, int count);
@@ -151,11 +152,31 @@ public class JNIKiwixReader
throw new JNIKiwixException("Cannot open zimfile "+filename);
}
}
public JNIKiwixReader(FileDescriptor fd) throws JNIKiwixException
{
nativeHandle = getNativeReaderByFD(fd);
if (nativeHandle == 0) {
throw new JNIKiwixException("Cannot open zimfile by fd "+fd.toString());
}
}
public JNIKiwixReader(FileDescriptor fd, long offset, long size)
throws JNIKiwixException
{
nativeHandle = getNativeReaderEmbedded(fd, offset, size);
if (nativeHandle == 0) {
throw new JNIKiwixException(String.format("Cannot open embedded zimfile (fd=%s, offset=%d, size=%d)", fd, offset, size));
}
}
public JNIKiwixReader() {
}
public native void dispose();
private native long getNativeReader(String filename);
private native long getNativeReaderByFD(FileDescriptor fd);
private native long getNativeReaderEmbedded(FileDescriptor fd, long offset, long size);
private long nativeHandle;
}

View File

@@ -0,0 +1,19 @@
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:opds="http://opds-spec.org/2010/catalog">
<id>00000000-0000-0000-0000-000000000000</id>
<entry>
<title>Test ZIM file</title>
<id>urn:uuid:86c91e51-55bf-8882-464e-072aca37a3e8</id>
<icon>/meta?name=favicon&amp;content=small</icon>
<updated>2020-11-27:00::00:Z</updated>
<language>en</language>
<summary>This is a ZIM file used in libzim unit-tests</summary>
<tags>unit;test</tags>
<link type="text/html" href="/small" />
<author>
<name>Kiwix</name>
</author>
<link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="http://localhost/small.zim" length="78982" />
<link rel="http://opds-spec.org/image/thumbnail" type="image/png" href="/meta?name=favicon&amp;content=small" />
</entry>
</feed>

View File

@@ -0,0 +1,37 @@
#!/usr/bin/bash
# This script compiles and runs the unit test to test the java wrapper.
# This is not integrated in meson because ... this is not so easy.
die()
{
echo >&2 "!!! ERROR: $*"
exit 1
}
KIWIX_LIB_JAR=$1
if [ -z $KIWIX_LIB_JAR ]
then
die "You must give the path to the kiwixlib.jar as first argument"
fi
KIWIX_LIB_DIR=$2
if [ -z $KIWIX_LIB_DIR ]
then
die "You must give the path to directory containing libkiwix.so as second argument"
fi
KIWIX_LIB_JAR=$(readlink -f "$KIWIX_LIB_JAR")
KIWIX_LIB_DIR=$(readlink -f "$KIWIX_LIB_DIR")
TEST_SOURCE_DIR=$(dirname "$(readlink -f $0)")
cd "$TEST_SOURCE_DIR"
javac -g -d . -s . -cp "junit-4.13.jar:$KIWIX_LIB_JAR" test.java \
|| die "Compilation failed"
java -Djava.library.path="$KIWIX_LIB_DIR" \
-cp "junit-4.13.jar:hamcrest-core-1.3.jar:$KIWIX_LIB_JAR:." \
org.junit.runner.JUnitCore test \
|| die "Unit test failed"

View File

@@ -1,26 +0,0 @@
#!/usr/bin/bash
# This script compile the unit test to test the java wrapper.
# This is not integrated in meson because ... this is not so easy.
KIWIX_LIB_JAR=$1
if [ -z $KIWIX_LIB_JAR ]
then
echo "You must give the path to the kiwixlib.jar as first argument"
exit 1
fi
KIWIX_LIB_DIR=$2
if [ -z $KIWIX_LIB_DIR ]
then
echo "You must give the path to directory containing libkiwix.so as second argument"
exit 1
fi
TEST_SOURCE_DIR=$(dirname $(readlink -f $0))
javac -g -d . -s . -cp $TEST_SOURCE_DIR/junit-4.13.jar:$KIWIX_LIB_JAR $TEST_SOURCE_DIR/test.java
java -Djava.library.path=$KIWIX_LIB_DIR -cp $TEST_SOURCE_DIR/junit-4.13.jar:$TEST_SOURCE_DIR/hamcrest-core-1.3.jar:$KIWIX_LIB_JAR:. org.junit.runner.JUnitCore test

View File

@@ -0,0 +1,28 @@
#!/usr/bin/env bash
die()
{
echo >&2 "!!! ERROR: $*"
exit 1
}
cd "$(dirname "$0")"
rm -f small.zim
zimwriterfs --withoutFTIndex \
-w main.html \
-f favicon.png \
-l en \
-t "Test ZIM file" \
-d "N/A" \
-c "N/A" \
-p "N/A" \
small_zimfile_data \
small.zim \
&& echo 'small.zim was successfully created' \
|| die 'Failed to create small.zim'
printf "BEGINZIM" > small.zim.embedded \
&& cat small.zim >> small.zim.embedded \
&& printf "ENDZIM" >> small.zim.embedded \
&& echo 'small.zim.embedded was successfully created' \
|| die 'Failed to create small.zim.embedded'

View File

Binary file not shown.

View File

Binary file not shown.

View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.1 KiB

View File

@@ -0,0 +1,11 @@
<html>
<head>
<meta charset="UTF-8">
<title>Test ZIM file</title>
<meta name="viewport" content="width=device-width, initial-scale=1.0">
</head>
<body>
Test ZIM file
</body>
</html>

View File

@@ -10,36 +10,143 @@ static {
System.loadLibrary("kiwix");
}
private static String getCatalogContent()
private static byte[] getFileContent(String path)
throws IOException
{
BufferedReader reader = new BufferedReader(new FileReader("catalog.xml"));
String line;
StringBuilder sb = new StringBuilder();
while ((line = reader.readLine()) != null)
{
sb.append(line + "\n");
}
reader.close();
return sb.toString();
File file = new File(path);
DataInputStream in = new DataInputStream(
new BufferedInputStream(
new FileInputStream(file)));
byte[] data = new byte[(int)file.length()];
in.read(data);
return data;
}
private static byte[] getFileContentPartial(String path, int offset, int size)
throws IOException
{
File file = new File(path);
DataInputStream in = new DataInputStream(
new BufferedInputStream(
new FileInputStream(file)));
byte[] data = new byte[size];
in.skipBytes(offset);
in.read(data, 0, size);
return data;
}
private static String getTextFileContent(String path)
throws IOException
{
return new String(getFileContent(path));
}
@Test
public void testSome()
public void testReader()
throws JNIKiwixException, IOException
{
JNIKiwixReader reader = new JNIKiwixReader("small.zim");
assertEquals("Test ZIM file", reader.getTitle());
assertEquals(45, reader.getFileSize()); // The file size is in KiB
assertEquals("A/main.html", reader.getMainPage());
String s = getTextFileContent("small_zimfile_data/main.html");
byte[] c = reader.getContent(new JNIKiwixString("A/main.html"),
new JNIKiwixString(),
new JNIKiwixString(),
new JNIKiwixInt());
assertEquals(s, new String(c));
byte[] faviconData = getFileContent("small_zimfile_data/favicon.png");
assertEquals(faviconData.length, reader.getArticleSize("I/favicon.png"));
c = reader.getContent(new JNIKiwixString("I/favicon.png"),
new JNIKiwixString(),
new JNIKiwixString(),
new JNIKiwixInt());
assertTrue(Arrays.equals(faviconData, c));
DirectAccessInfo dai = reader.getDirectAccessInformation("I/favicon.png");
assertNotEquals("", dai.filename);
c = getFileContentPartial(dai.filename, (int)dai.offset, faviconData.length);
assertTrue(Arrays.equals(faviconData, c));
}
@Test
public void testReaderByFd()
throws JNIKiwixException, IOException
{
FileInputStream fis = new FileInputStream("small.zim");
JNIKiwixReader reader = new JNIKiwixReader(fis.getFD());
assertEquals("Test ZIM file", reader.getTitle());
assertEquals(45, reader.getFileSize()); // The file size is in KiB
assertEquals("A/main.html", reader.getMainPage());
String s = getTextFileContent("small_zimfile_data/main.html");
byte[] c = reader.getContent(new JNIKiwixString("A/main.html"),
new JNIKiwixString(),
new JNIKiwixString(),
new JNIKiwixInt());
assertEquals(s, new String(c));
byte[] faviconData = getFileContent("small_zimfile_data/favicon.png");
assertEquals(faviconData.length, reader.getArticleSize("I/favicon.png"));
c = reader.getContent(new JNIKiwixString("I/favicon.png"),
new JNIKiwixString(),
new JNIKiwixString(),
new JNIKiwixInt());
assertTrue(Arrays.equals(faviconData, c));
DirectAccessInfo dai = reader.getDirectAccessInformation("I/favicon.png");
assertNotEquals("", dai.filename);
c = getFileContentPartial(dai.filename, (int)dai.offset, faviconData.length);
assertTrue(Arrays.equals(faviconData, c));
}
@Test
public void testReaderWithAnEmbeddedArchive()
throws JNIKiwixException, IOException
{
File plainArchive = new File("small.zim");
FileInputStream fis = new FileInputStream("small.zim.embedded");
JNIKiwixReader reader = new JNIKiwixReader(fis.getFD(), 8, plainArchive.length());
assertEquals("Test ZIM file", reader.getTitle());
assertEquals(45, reader.getFileSize()); // The file size is in KiB
assertEquals("A/main.html", reader.getMainPage());
String s = getTextFileContent("small_zimfile_data/main.html");
byte[] c = reader.getContent(new JNIKiwixString("A/main.html"),
new JNIKiwixString(),
new JNIKiwixString(),
new JNIKiwixInt());
assertEquals(s, new String(c));
byte[] faviconData = getFileContent("small_zimfile_data/favicon.png");
assertEquals(faviconData.length, reader.getArticleSize("I/favicon.png"));
c = reader.getContent(new JNIKiwixString("I/favicon.png"),
new JNIKiwixString(),
new JNIKiwixString(),
new JNIKiwixInt());
assertTrue(Arrays.equals(faviconData, c));
DirectAccessInfo dai = reader.getDirectAccessInformation("I/favicon.png");
assertNotEquals("", dai.filename);
c = getFileContentPartial(dai.filename, (int)dai.offset, faviconData.length);
assertTrue(Arrays.equals(faviconData, c));
}
@Test
public void testLibrary()
throws IOException
{
Library lib = new Library();
Manager manager = new Manager(lib);
String content = getCatalogContent();
manager.readOpds(content, "https://library.kiwix.org");
assertEquals(lib.getBookCount(true, true), 10);
String content = getTextFileContent("catalog.xml");
manager.readOpds(content, "http://localhost");
assertEquals(lib.getBookCount(true, true), 1);
String[] bookIds = lib.getBooksIds();
assertEquals(bookIds.length, 10);
assertEquals(bookIds.length, 1);
Book book = lib.getBookById(bookIds[0]);
assertEquals(book.getTitle(), "Wikisource");
assertEquals(book.getTags(), "wikisource;_category:wikisource;_pictures:no;_videos:no;_details:yes;_ftindex:yes");
assertEquals(book.getFaviconUrl(), "https://library.kiwix.org/meta?name=favicon&content=wikisource_fr_all_nopic_2020-01");
assertEquals(book.getUrl(), "http://download.kiwix.org/zim/wikisource/wikisource_fr_all_nopic_2020-01.zim.meta4");
assertEquals(book.getTitle(), "Test ZIM file");
assertEquals(book.getTags(), "unit;test");
assertEquals(book.getFaviconUrl(), "http://localhost/meta?name=favicon&content=small");
assertEquals(book.getUrl(), "http://localhost/small.zim");
}
static

View File

@@ -24,7 +24,7 @@
#include <jni.h>
#include <pthread.h>
#include <mutex>
#include <string>
#include <vector>
#include <iostream>
@@ -36,7 +36,7 @@
#define LOG(...)
#endif
extern pthread_mutex_t globalLock;
extern std::mutex globalLock;
template<typename T>
void setPtr(JNIEnv* env, jobject thisObj, T* ptr)
@@ -88,22 +88,10 @@ inline jobjectArray createArray(JNIEnv* env, size_t length, const std::string& t
return env->NewObjectArray(length, c, NULL);
}
class Lock
class Lock : public std::unique_lock<std::mutex>
{
protected:
pthread_mutex_t* lock;
public:
Lock() : lock(&globalLock) { pthread_mutex_lock(lock); }
Lock(const Lock&) = delete;
Lock& operator=(const Lock&) = delete;
Lock(Lock&& other) : lock(&globalLock) { other.lock = nullptr; }
virtual ~Lock()
{
if (lock) {
pthread_mutex_unlock(lock);
}
}
Lock() : std::unique_lock<std::mutex>(globalLock) { }
};
template <class T>
@@ -258,7 +246,7 @@ inline void setBoolObjValue(const bool value, const jobject obj, JNIEnv* env)
env->SetIntField(obj, objFid, c2jni(value, env));
}
inline void setPairObjValue(const std::string& filename, const long offset,
inline void setDaiObjValue(const std::string& filename, const long offset,
const jobject obj, JNIEnv* env)
{
jclass objClass = env->GetObjectClass(obj);

View File

@@ -0,0 +1,10 @@
<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
<ShortName>Zim catalog search</ShortName>
<Description>Search zim files in the catalog.</Description>
<Url type="application/atom+xml;profile=opds-catalog;kind=acquisition"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:k="http://kiwix.org/opensearchextension/1.0"
indexOffset="0"
template="{{endpoint_root}}/entries?q={searchTerms?}&lang={language?}&name={k:name?}&tag={k:tag?}&maxsize={k:maxsize?}&count={count?}&start={startIndex?}"/>
</OpenSearchDescription>

View File

@@ -6,5 +6,5 @@
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:k="http://kiwix.org/opensearchextension/1.0"
indexOffset="0"
template="/{{root}}/catalog/search?q={searchTerms?}&lang={language?}&name={k:name?}&tag={k:tag?}&notag={k:notag?}&maxsize={k:maxsize?}&count={count?}&start={startIndex?}"/>
template="{{root}}/catalog/search?q={searchTerms?}&lang={language?}&name={k:name?}&tag={k:tag?}&notag={k:notag?}&maxsize={k:maxsize?}&count={count?}&start={startIndex?}"/>
</OpenSearchDescription>

View File

@@ -18,8 +18,19 @@ skin/jquery-ui/images/ui-bg_glass_95_fef1ec_1x400.png
skin/jquery-ui/jquery-ui.theme.min.css
skin/jquery-ui/jquery-ui.min.css
skin/caret.png
skin/bittorrent.png
skin/magnet.png
skin/download.png
skin/hash.png
skin/search-icon.svg
skin/taskbar.js
skin/iso6391To3.js
skin/isotope.pkgd.min.js
skin/index.js
skin/taskbar.css
skin/home.css
skin/fonts/Poppins.ttf
skin/fonts/Roboto.ttf
skin/block_external.js
templates/search_result.html
templates/no_search_result.html
@@ -27,8 +38,15 @@ templates/404.html
templates/500.html
templates/index.html
templates/suggestion.json
templates/head_part.html
templates/head_taskbar.html
templates/taskbar_part.html
templates/external_blocker_part.html
templates/captured_external.html
templates/catalog_entries.xml
templates/catalog_v2_root.xml
templates/catalog_v2_entries.xml
templates/catalog_v2_entry.xml
templates/catalog_v2_categories.xml
templates/catalog_v2_languages.xml
opensearchdescription.xml
catalog_v2_searchdescription.xml

BIN
static/skin/bittorrent.png Normal file
View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

View File

@@ -1,5 +1,6 @@
const root = document.querySelector( `link[type='root']` ).getAttribute("href");
// `block_path` variable used by openzim/warc2zim to detect whether URL blocking is enabled or not
var block_path = "/catch/external";
var block_path = `${root}/catch/external`;
// called only on external links
function capture_event(e, target) { target.setAttribute("href", encodeURI(block_path + "?source=" + target.href)); }

BIN
static/skin/download.png Normal file
View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 722 B

View File

Binary file not shown.

View File

Binary file not shown.

BIN
static/skin/hash.png Normal file
View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 705 B

456
static/skin/home.css Normal file
View File

@@ -0,0 +1,456 @@
*,
*::after,
*::before {
margin: 0;
padding: 0;
box-sizing: inherit;
}
html {
font-size: 62.5%;
}
body {
position: relative;
box-sizing: border-box;
}
::selection {
background-color: #00b4e4;
color: white;
}
.kiwixNav {
background-color: #f4f6f8;
width: 100%;
padding: 20px 11vw 25px;
height: 140px;
}
.kiwixHomeBody__results {
font-size: 1.6rem;
color: #333333;
font-family: poppins;
margin: 10px 0 -5px;
position: relative;
top: -10px;
left: 13px;
}
.kiwixNav__filters {
display: grid;
grid-gap: 20px;
grid-template-columns: 231px 231px;
margin: 10px 0;
}
.kiwixNav__kiwixFilter {
-webkit-appearance: none;
-moz-appearance: none;
-ms-appearance: none;
appearance: none;
outline: 0;
box-shadow: none;
border: 0 !important;
background: white;
background-image: none;
border-radius: 1px;
width: 195px;
height: 35px;
flex: 1;
color: black;
padding: 7px 10px 10px;
cursor: pointer;
}
.kiwixNav__kiwixFilter:-ms-expand {
display: none;
}
.kiwixNav__select {
position: relative;
display: flex;
width: 231px;
height: 35px;
line-height: 3;
background: #909090;
overflow: hidden;
border-radius: 10px;
border: 1px solid #b5b2b2;
}
.kiwixNav__select::after {
content: '\25BC';
color: #fff;
background-color: #909090;
height: 100%;
position: absolute;
top: 0;
right: 0;
padding: 0 1em;
cursor: pointer;
pointer-events: none;
-webkit-transition: .25s all ease;
-o-transition: .25s all ease;
transition: .25s all ease;
}
.kiwixNav__kiwixFilter option {
background-color: #fff;
color: #000;
}
.kiwixSearch {
height: 35px;
width: 229px;
border-radius: 10px;
border: solid 1px #b5b2b2;
padding: 10px;
background-image: url('./search-icon.svg');
background-repeat: no-repeat;
background-position: right center;
background-origin: content-box;
background-size: 15px;
}
.searchButton {
margin: 0 17px;
height: 35px;
width: 100px;
border-radius: 10px;
color: white;
background-color: #909090;
border: solid 1px #b5b2b2;
}
.searchButton:hover {
background-color: #858585;
}
.kiwixHomeBody {
padding: 20px 15vw;
min-height: calc(100vh - 170px);
position: relative;
}
.book__list {
position: relative;
}
.book {
border-color: #acacac;
}
.book__wrapper {
color: #444343;
height: 280px;
width: 250px;
margin: 15px;
background-color: #f7f7f7;
border: 1px solid #ececec;
border-radius: 3px;
display: grid;
grid-template-columns: 65px 1fr;
grid-template-rows: 70px 120px 1fr 1fr;
grid-gap: 5px;
transition: transform 0.25s;
}
.book__wrapper:hover {
transform: scale(1.07);
}
.book__icon {
display: flex;
align-items: center;
align-content: center;
justify-content: center;
margin: 10px 0 0 10px;
}
.book__icon--image {
max-width: 100%;
max-height: 100%;
}
.book__title {
display: grid;
font-family: poppins;
color: black;
padding: 12px 10px 0 2px;
width: 100%;
height: 100%;
align-items: center;
align-content: center;
}
#bookSize {
font-size: 1.1rem;
margin: 3px 0;
}
#bookTitle {
overflow: hidden;
white-space: nowrap;
text-overflow: ellipsis;
font-size: 1.45rem;
}
.book__description {
grid-column: 1 / 3;
margin: 10px 10px 5px;
overflow: hidden;
display: -webkit-box;
-webkit-line-clamp: 7;
-webkit-box-orient: vertical;
font-family: roboto;
font-weight: 300;
font-size: 1.2rem;
color: #4d4d4d;
line-height: 1.25;
}
.book__languageTag {
display: flex;
align-items: center;
justify-content: center;
box-shadow: 0 0 2px #acacac;
background-color: #ffffff;
font-weight: bold;
font-family: roboto;
color: black;
height: 25px;
width: 25px;
margin: 10px auto 0 10px;
border-radius: 5px;
font-size: 0.85rem;
}
.book__tags {
display: flex;
text-align: end;
font-size: 1.1rem;
justify-content: flex-end;
color: #909090;
font-family: roboto;
margin-right: 10px;
margin-top: 10px;
overflow: hidden;
}
.book__tags--wrapper {
overflow: hidden;
display: -webkit-box;
-webkit-line-clamp: 2;
-webkit-box-orient: vertical;
}
.book__links {
display: flex;
text-align: end;
justify-content: flex-end;
grid-column: 2 / 3;
font-size: 1.1rem;
margin: 10px 4px;
color: #00b4e4;
}
.book__links > a, .book__links > span {
cursor: pointer;
text-decoration: none;
color: #00b4e4;
position: relative;
padding: 1px 6px 0;
font-family: roboto;
}
.book__links > a:hover, .book__links > span:hover {
background: #00b4e4;
color: white;
}
.modal-wrapper {
position: fixed;
z-index: 100;
top: 0;
left: 0;
width: 100vw;
height: 100vh;
display: flex;
justify-content: center;
align-items: center;
align-content: center;
background-color: rgba(0, 0, 0, 30%);
}
.modal {
color: #444343;
height: 280px;
width: 250px;
margin: 15px;
background-color: #f7f7f7;
border: 1px solid #ececec;
border-radius: 3px;
}
.modal-heading {
background-color: #f0f0f0;
height: 20%;
width: 100%;
border-bottom: 1px solid #ececec;
display: grid;
grid-template-columns: 3fr 1fr;
}
.modal-title {
display: flex;
font-size: 15px;
align-items: center;
padding-left: 20px;
font-family: poppins;
}
.modal-close-button {
cursor: pointer;
display: flex;
justify-content: center;
align-items: center;
}
.modal-content {
padding: 20px;
}
.modal-content div {
width: 100%;
height: 40px;
}
.modal-content a {
text-decoration: none;
}
.modal-content a>div {
position: relative;
top: -28px;
left: 50px;
text-decoration: underline;
font-size: 13px;
font-family: poppins;
color: #444343;
}
.modal-content img {
display: inline-block;
margin: 0 10px;
height: 30px;
}
.fadeOut {
position: fixed;
display: none;
bottom: 0;
left: 0;
z-index: 1;
background: linear-gradient(180deg, rgba(232, 232, 232, 0) 0%, #ffffff 100%);
height: 80px;
width: 100%;
}
.noResults {
font-size: 1.6rem;
text-align: center;
}
.noResults > a {
color: #3498db;
}
.loader-spinner {
position: absolute;
top: -50%;
left: 50%;
border: 5px solid #f3f3f3;
border-radius: 50%;
border-top: 5px solid #3498db;
width: 40px;
height: 40px;
margin: auto;
-webkit-animation: spin 1s linear infinite; /* Safari */
animation: spin 1s linear infinite;
margin-top: 35px;
margin-bottom: -35px;
z-index: 2;
}
/* Safari */
@-webkit-keyframes spin {
0% { -webkit-transform: rotate(0deg); }
100% { -webkit-transform: rotate(360deg); }
}
@keyframes spin {
0% { transform: rotate(0deg); }
100% { transform: rotate(360deg); }
}
.loader {
display: none;
position: relative;
height: 70px;
width: 100%;
}
.kiwixfooter {
display: flex;
width: 100%;
justify-content: center;
text-align: center;
font-size: 1.6rem;
height: 30px;
}
.kiwixfooter > a {
width: auto;
}
@media screen and (max-width: 1100px) {
.kiwixHomeBody {
padding: 20px 10vw;
min-height: calc(100vh - 170px);
}
.kiwixNav__filters {
grid-template-columns: repeat(4, 1fr);
}
}
@media screen and (max-width: 590px) {
.kiwixNav {
height: 257px;
}
.kiwixHomeBody {
min-height: calc(100vh - 287px);
}
.kiwixSearch {
margin-top: 11px;
}
.searchButton {
margin: 19px 0;
width: 229px;
}
.kiwixNav__filters {
grid-template-columns: 1fr;
}
}
@media screen and (max-width: 340px) {
.kiwixHomeBody {
padding: 20px 5vw;
}
}

415
static/skin/index.js Normal file
View File

@@ -0,0 +1,415 @@
(function() {
const root = $(`link[type='root']`).attr('href');
const incrementalLoadingParams = {
start: 0,
count: viewPortToCount()
};
const bookOrderMap = new Map();
const filterCookieName = 'filters';
const oneDayDelta = 86400000;
let loader;
let footer;
let fadeOutDiv;
let iso;
let isFetching = false;
let noResultInjected = false;
let filters = getCookie(filterCookieName);
let params = new URLSearchParams(window.location.search || filters || '');
let timer;
function queryUrlBuilder() {
let url = `${root}/catalog/search?`;
url += Object.keys(incrementalLoadingParams).map(key => `${key}=${incrementalLoadingParams[key]}`).join("&");
params.forEach((value, key) => {url+= value ? `&${key}=${value}` : ''});
return (url);
}
function setCookie(cookieName, cookieValue) {
const date = new Date();
date.setTime(date.getTime() + oneDayDelta);
document.cookie = `${cookieName}=${cookieValue};expires=${date.toUTCString()};sameSite=Strict`;
}
function getCookie(cookieName) {
const name = cookieName + "=";
let result;
decodeURIComponent(document.cookie).split('; ').forEach(val => {
if (val.indexOf(name) === 0) {
result = val.substring(name.length);
}
});
return result;
}
const humanFriendlySize = (fileSize) => {
if (fileSize === 0) {
return '';
}
const units = ['bytes', 'kB', 'MB', 'GB', 'TB'];
let quotient = Math.floor(Math.log10(fileSize) / 3);
quotient = quotient < units.length ? quotient : units.length - 1;
fileSize /= (1000 ** quotient);
return `${+fileSize.toFixed(2)} ${units[quotient]}`;
};
const humanFriendlyTitle = (title) => {
title = title.replace(/_/g, ' ');
return htmlEncode(title[0].toUpperCase() + title.slice(1));
}
function htmlEncode(str) {
return str.replace(/[\u00A0-\u9999<>\&]/gim, (i) => `&#${i.charCodeAt(0)};`);
}
function viewPortToCount(){
const zoom = Math.floor((( window.outerWidth - 10 ) / window.innerWidth) * 100);
return Math.floor(window.innerHeight/(3*zoom) + 1)*(window.innerWidth/(2.5*zoom) + 1);
}
function getInnerHtml(node, query) {
return node.querySelector(query).innerHTML;
}
function generateBookHtml(book, sort = false) {
const link = book.querySelector('link[type="text/html"]').getAttribute('href');
let iconUrl;
book.querySelectorAll('link[rel="http://opds-spec.org/image/thumbnail"]').forEach(link => {
if (link.getAttribute('type').split(';')[1] == 'width=48' && !iconUrl) {
iconUrl = link.getAttribute('href');
}
});
const title = getInnerHtml(book, 'title');
const description = getInnerHtml(book, 'summary');
const id = getInnerHtml(book, 'id');
const language = getInnerHtml(book, 'language');
const tags = getInnerHtml(book, 'tags');
let tagHtml = tags.split(';').filter(tag => {return !(tag.split(':')[0].startsWith('_'))})
.map((tag) => {return tag.charAt(0).toUpperCase() + tag.slice(1)})
.join(' | ').replace(/_/g, ' ');
let downloadLink;
let zimSize = 0;
try {
const downloadBookLink = book.querySelector('link[type="application/x-zim"]')
zimSize = parseInt(downloadBookLink.getAttribute('length'));
downloadLink = downloadBookLink.getAttribute('href').split('.meta4')[0];
} catch {
downloadLink = '';
}
const humanFriendlyZimSize = humanFriendlySize(zimSize);
const divTag = document.createElement('div');
divTag.setAttribute('class', 'book');
divTag.setAttribute('data-id', id);
if (sort) {
divTag.setAttribute('data-idx', bookOrderMap.get(id));
}
divTag.innerHTML = `<div class="book__wrapper"><div class='book__icon' ><img class="book__icon--image" src='${root}${iconUrl}'></div>
<div class='book__title' title='${title}'>
<div id="bookTitle">${title}</div>
${humanFriendlyZimSize ? `<div id='bookSize'>${humanFriendlyZimSize}</div>`: ''}
</div>
<div class='book__description' title='${description}'>${description}</div>
<div class='book__languageTag'>${language.substr(0, 2).toUpperCase()}</div>
<div class='book__tags'><div class="book__tags--wrapper">${tagHtml}</div></div>
<div class='book__links'> <a href="${root}${link}" data-hover="Preview">Preview</a>${downloadLink ? `&nbsp;|&nbsp;<span class="download" data-link=${downloadLink} class="modal-button">Download</span>` : ''} </div></div>`;
return divTag;
}
function toggleFooter(show=false) {
if (show) {
footer.style.display = 'block';
} else {
footer.style.display = 'none';
fadeOutDiv.style.display = 'block';
}
}
function insertModal(button) {
const downloadLink = button.getAttribute('data-link');
button.addEventListener('click', () => {
document.body.insertAdjacentHTML('beforeend', `<div class="modal-wrapper">
<div class="modal">
<div class="modal-heading">
<div class="modal-title">
<div>
Download
</div>
</div>
<div onclick="closeModal()" class="modal-close-button">
<div>
<svg xmlns="http://www.w3.org/2000/svg" width="14" height="14" viewBox="0 0 14 14" fill="none">
<path fill-rule="evenodd" clip-rule="evenodd" d="M13.7071 1.70711C14.0976 1.31658 14.0976
0.683417 13.7071 0.292893C13.3166 -0.0976311 12.6834 -0.0976311 12.2929 0.292893L7 5.58579L1.70711
0.292893C1.31658 -0.0976311 0.683417 -0.0976311 0.292893 0.292893C-0.0976311 0.683417
-0.0976311 1.31658 0.292893 1.70711L5.58579 7L0.292893 12.2929C-0.0976311 12.6834
-0.0976311 13.3166 0.292893 13.7071C0.683417 14.0976 1.31658 14.0976 1.70711 13.7071L7
8.41421L12.2929 13.7071C12.6834 14.0976 13.3166 14.0976 13.7071 13.7071C14.0976 13.3166
14.0976 12.6834 13.7071 12.2929L8.41421 7L13.7071 1.70711Z" fill="black" />
</svg>
</div>
</div>
</div>
<div class="modal-content">
<div class="modal-regular-download">
<a href="${downloadLink}" download>
<img src="../skin/download.png" alt="direct download" />
<div>Direct</div>
</a>
</div>
<div class="modal-regular-download">
<a href="${downloadLink}.sha256" download>
<img src="../skin/hash.png" alt="download hash" />
<div>Sha256 hash</div>
</a>
</div>
<div class="modal-regular-download">
<a href="${downloadLink}.magnet" target="_blank">
<img src="../skin/magnet.png" alt="download magnet" />
<div>Magnet link</div>
</a>
</div>
<div class="modal-regular-download">
<a href="${downloadLink}.torrent" download>
<img src="../skin/bittorrent.png" alt="download torrent" />
<div>Torrent file</div>
</a>
</div>
</div>
</div>
</div>`);
})
}
async function getBookCount(query) {
const url = `${root}/catalog/search?${query}`;
return await fetch(url).then(async (resp) => {
const data = new window.DOMParser().parseFromString(await resp.text(), 'application/xml');
return parseInt(data.querySelector('totalResults').innerHTML);
});
}
async function loadBooks() {
loader.style.display = 'block';
return await fetch(queryUrlBuilder()).then(async (resp) => {
const data = new window.DOMParser().parseFromString(await resp.text(), 'application/xml');
const books = data.querySelectorAll('entry');
books.forEach((book, idx) => {
bookOrderMap.set(getInnerHtml(book, 'id'), idx);
});
incrementalLoadingParams.start += books.length;
const results = parseInt(data.querySelector('totalResults').innerHTML)
if (results === bookOrderMap.size) {
incrementalLoadingParams.count = 0;
toggleFooter(true);
} else {
toggleFooter();
}
const kiwixResultText = document.querySelector('.kiwixHomeBody__results')
if (results) {
let resultText = `${results} books`;
if (results === 1) {
resultText = `${results} book`;
}
kiwixResultText.innerHTML = resultText;
} else {
kiwixResultText.innerHTML = ``;
}
loader.style.display = 'none';
return books;
});
}
async function loadAndDisplayOptions(nodeQuery, query, valueEntryNode) {
await fetch(query).then(async (resp) => {
const data = new window.DOMParser().parseFromString(await resp.text(), 'application/xml');
let optionStr = '';
data.querySelectorAll('entry').forEach(entry => {
const title = getInnerHtml(entry, 'title');
const value = getInnerHtml(entry, valueEntryNode);
optionStr += `<option value='${value}'>${humanFriendlyTitle(title)}</option>`;
});
document.querySelector(nodeQuery).innerHTML += optionStr;
});
}
function checkAndInjectEmptyMessage() {
const kiwixHomeBody = document.querySelector('.kiwixHomeBody');
if (!bookOrderMap.size) {
if (!noResultInjected) {
noResultInjected = true;
iso.remove(document.getElementsByClassName('book__list')[0].getElementsByTagName('div'));
iso.layout();
setTimeout(() => {
const divTag = document.createElement('div');
divTag.setAttribute('class', 'noResults');
divTag.innerHTML = `No result. Would you like to <a href="/?lang=">reset filter</a>?`;
kiwixHomeBody.append(divTag);
kiwixHomeBody.setAttribute('style', 'display: flex; justify-content: center; align-items: center');
divTag.getElementsByTagName('a')[0].onclick = (event) => {
event.preventDefault();
window.history.pushState({}, null, `${window.location.href.split('?')[0]}?lang=`);
setCookie(filterCookieName, 'lang=');
resetAndFilter();
document.querySelectorAll('.filter').forEach(filter => {
filter.value = params.get(filter.name) || '';
if (filter.value) {
filter.style = 'background-color: #858585; color: #fff';
} else {
filter.style = 'background-color: #ffffff; color: black';
}
})
};
loader.setAttribute('style', 'position: absolute; top: 50%');
}, 300);
}
return true;
} else if (noResultInjected) {
noResultInjected = false;
document.getElementsByClassName('noResults')[0].remove();
kiwixHomeBody.removeAttribute('style');
}
loader.removeAttribute('style');
return false;
}
async function loadAndDisplayBooks(sort = false) {
if (isFetching) return;
isFetching = true;
await loadAndDisplayBooksUnguarded(sort);
isFetching = false;
}
async function loadAndDisplayBooksUnguarded(sort) {
let books = await loadBooks();
if (checkAndInjectEmptyMessage()) {return}
const booksToFilter = new Set();
const booksToDelete = new Set();
iso.arrange({
filter: function (idx, elem) {
const id = elem.getAttribute('data-id');
const retVal = bookOrderMap.has(id);
if (retVal) {
booksToFilter.add(id);
if (sort) {
elem.setAttribute('data-idx', bookOrderMap.get(id));
iso.updateSortData(elem);
}
} else {
booksToDelete.add(elem);
}
return retVal;
}
});
books = [...books].filter((book) => {return !booksToFilter.has(getInnerHtml(book, 'id'))});
booksToDelete.forEach(book => {iso.remove(book);});
books.forEach((book) => {
iso.insert(generateBookHtml(book, sort))
const downloadButton = document.querySelector(`[data-id="${getInnerHtml(book, 'id')}"] .download`);
if (downloadButton) {
insertModal(downloadButton);
}
});
}
async function resetAndFilter(filterType = '', filterValue = '') {
isFetching = false;
incrementalLoadingParams.start = 0;
incrementalLoadingParams.count = viewPortToCount();
fadeOutDiv.style.display = 'none';
bookOrderMap.clear();
params = new URLSearchParams(window.location.search);
if (filterType) {
params.set(filterType, filterValue);
window.history.pushState({}, null, `${window.location.href.split('?')[0]}?${params.toString()}`);
setCookie(filterCookieName, params.toString());
}
document.querySelectorAll('.filter').forEach(filter => {
if (filter.value) {
filter.style = 'background-color: #858585; color: #fff';
} else {
filter.style = 'background-color: #ffffff; color: black';
}
});
await loadAndDisplayBooks(true);
}
window.addEventListener('popstate', async () => {
await resetAndFilter();
document.querySelectorAll('.filter').forEach(filter => {filter.value = params.get(filter.name) || ''});
});
async function loadSubset() {
if (window.innerHeight + window.scrollY >= document.body.offsetHeight) {
if (incrementalLoadingParams.count) {
loadAndDisplayBooks();
}
else {
fadeOutDiv.style.display = 'none';
}
}
}
window.addEventListener('resize', (event) => {
if (timer) {clearTimeout(timer)}
timer = setTimeout(() => {
incrementalLoadingParams.count = incrementalLoadingParams.count && viewPortToCount();
loadSubset();
}, 100, event);
});
window.addEventListener('scroll', loadSubset);
window.onload = async () => {
iso = new Isotope( '.book__list', {
itemSelector: '.book',
getSortData:{
weight: function( itemElem ) {
const index = itemElem.getAttribute('data-idx');
return index ? parseInt(index) : Infinity;
}
},
sortBy: 'weight',
layoutMode: 'cellsByRow',
cellsByRow: {
columnWidth: '.book',
rowHeight: '.book'
}
});
footer = document.getElementById('kiwixfooter');
fadeOutDiv = document.getElementById('fadeOut');
loader = document.querySelector('.loader');
await loadAndDisplayBooks();
await loadAndDisplayOptions('#languageFilter', `${root}/catalog/v2/languages`, 'language');
await loadAndDisplayOptions('#categoryFilter', `${root}/catalog/v2/categories`, 'title');
document.querySelectorAll('.filter').forEach(filter => {
filter.addEventListener('change', () => {resetAndFilter(filter.name, filter.value)});
});
if (filters) {
window.history.pushState({}, null, `${window.location.href.split('?')[0]}?${params.toString()}`);
}
params.forEach((value, key) => {
const selectBox = document.getElementsByName(key)[0];
if (selectBox) {
selectBox.value = value
}
});
document.getElementById('kiwixSearchForm').onsubmit = (event) => {event.preventDefault()};
if (!window.location.search) {
const browserLang = navigator.language.split('-')[0];
const langFilter = document.getElementById('languageFilter');
const lang = browserLang.length === 3 ? browserLang : iso6391To3[browserLang];
if (await getBookCount(`lang=${lang}`)) {
langFilter.value = lang;
langFilter.dispatchEvent(new Event('change'));
}
}
document.querySelectorAll('.filter').forEach(filter => {
if (filter.value) {
filter.style = 'background-color: #858585; color: #fff';
} else {
filter.style = 'background-color: #ffffff; color: black';
}
});
setCookie(filterCookieName, params.toString());
}
})();

124
static/skin/iso6391To3.js Normal file
View File

@@ -0,0 +1,124 @@
// eslint-disable-next-line no-unused-vars
const iso6391To3 = {
"aa": "aar",
"af": "afr",
"ak": "aka",
"am": "amh",
"ar": "ara",
"as": "asm",
"az": "aze",
"ba": "bak",
"be": "bel",
"bg": "bul",
"bm": "bam",
"bn": "ben",
"bo": "bod",
"br": "bre",
"bs": "bos",
"ca": "cat",
"ce": "che",
"co": "cos",
"cs": "ces",
"cv": "chv",
"cy": "cym",
"da": "dan",
"de": "deu",
"dz": "dzo",
"ee": "ewe",
"en": "eng",
"es": "spa",
"et": "est",
"eu": "eus",
"fa": "fas",
"ff": "ful",
"fi": "fin",
"fo": "fao",
"fr": "fra",
"ga": "gle",
"gl": "glg",
"gn": "grn",
"gu": "guj",
"gv": "glv",
"ha": "hau",
"he": "heb",
"hi": "hin",
"hr": "hrv",
"hu": "hun",
"hy": "hye",
"id": "ind",
"ig": "ibo",
"is": "isl",
"it": "ita",
"iu": "iku",
"ja": "jpn",
"jv": "jav",
"ka": "kat",
"ki": "kik",
"kk": "kaz",
"km": "khm",
"kn": "kan",
"ko": "kor",
"ks": "kas",
"ku": "kur",
"kw": "cor",
"ky": "kir",
"lb": "ltz",
"lg": "lug",
"ln": "lin",
"lo": "lao",
"lt": "lit",
"lv": "lav",
"mg": "mlg",
"mi": "mri",
"mk": "mkd",
"ml": "mal",
"mn": "mon",
"mr": "mar",
"mt": "mlt",
"my": "mya",
"nl": "nld",
"ny": "nya",
"om": "orm",
"pl": "pol",
"pt": "por",
"qu": "que",
"rm": "roh",
"rn": "run",
"ro": "ron",
"ru": "rus",
"rw": "kin",
"sa": "san",
"sd": "snd",
"sg": "sag",
"si": "sin",
"sk": "slk",
"sl": "slv",
"sn": "sna",
"so": "som",
"sq": "sqi",
"sr": "srp",
"ss": "ssw",
"sv": "swe",
"ta": "tam",
"te": "tel",
"tg": "tgk",
"th": "tha",
"ti": "tir",
"tk": "tuk",
"tn": "tsn",
"tr": "tur",
"ts": "tso",
"tt": "tat",
"ug": "uig",
"uk": "ukr",
"ur": "urd",
"uz": "uzb",
"ve": "ven",
"vi": "vie",
"wa": "wln",
"wo": "wol",
"xh": "xho",
"yo": "yor",
"zh": "zho",
"zu": "zul"
}

78
static/skin/isotope.pkgd.min.js vendored Normal file
View File

File diff suppressed because one or more lines are too long

BIN
static/skin/magnet.png Normal file
View File

Binary file not shown.

After

Width:  |  Height:  |  Size: 818 B

View File

@@ -0,0 +1,6 @@
<svg xmlns="http://www.w3.org/2000/svg" version="1.0" width="512.000000" height="512.000000" preserveAspectRatio="xMidYMid meet" style=""><rect id="backgroundrect" width="100%" height="100%" x="0" y="0" fill="none" stroke="none"/>
<g class="currentLayer" style=""><title>Layer 1</title><g transform="translate(0,512) scale(0.10000000149011612,-0.10000000149011612) " fill="#484848" stroke="none" id="svg_1" class="selected" fill-opacity="1">
<path d="M1858 5104 c-341 -50 -678 -192 -963 -408 -104 -78 -304 -278 -384 -384 -124 -164 -256 -409 -315 -583 -311 -929 47 -1926 877 -2442 662 -410 1535 -405 2188 14 l76 49 125 -128 c69 -70 352 -364 630 -653 574 -597 548 -575 688 -567 85 5 123 21 176 75 52 52 77 114 77 193 0 98 -24 142 -151 273 -420 434 -1132 1177 -1132 1182 0 3 22 33 49 68 164 209 291 471 362 746 159 614 28 1255 -363 1774 -78 104 -279 305 -386 386 -288 218 -618 355 -974 406 -130 19 -448 18 -580 -1z m502 -545 c308 -38 607 -184 844 -409 476 -455 606 -1146 327 -1745 -128 -273 -356 -519 -626 -674 -387 -222 -857 -259 -1282 -102 -626 231 -1040 879 -983 1540 13 146 33 241 77 371 214 634 830 1060 1493 1032 36 -1 103 -7 150 -13z" id="svg_2" fill="#484848" fill-opacity="1"/>
</g></g></svg>

After

Width:  |  Height:  |  Size: 1.2 KiB

View File

@@ -1,49 +1,96 @@
function htmlDecode(input) {
var doc = new DOMParser().parseFromString(input, "text/html");
return doc.documentElement.textContent;
}
(function ($) {
if ($(window).width() < 520) {
var didScroll;
var lastScrollTop = 0;
var delta = 5;
// on scroll, let the interval function know the user has scrolled
$(window).scroll(function (event) {
didScroll = true;
const jq = jQuery.noConflict(true);
jq(document).ready(() => {
(function ($) {
const root = $( `link[type='root']` ).attr("href");
const bookName = (window.location.pathname == `${root}/search`)
? (new URLSearchParams(window.location.search)).get('content')
: window.location.pathname.split(`${root}/`)[1].split('/')[0];
$( "#kiwixsearchbox" ).autocomplete({
source: `${root}/suggest?content=${bookName}`,
dataType: "json",
cache: false,
response: function( event, ui ) {
for(const item of ui.content) {
item.label = htmlDecode(item.label);
item.value = htmlDecode(item.value);
if (item.path) item.path = htmlDecode(item.path);
}
},
select: function(event, ui) {
if (ui.item.kind === 'path') {
window.location.href = `${root}/${bookName}/${encodeURI(ui.item.path)}`;
} else {
$( "#kiwixsearchbox" ).val(ui.item.value);
$( "#kiwixsearchform" ).submit();
}
},
}).data( "ui-autocomplete" )._renderItem = function( ul, item ) {
return $( "<li>" )
.data( "ui-autocomplete-item", item )
.append( item.label )
.appendTo( ul );
};
/* cybook hack */
if (navigator.userAgent.indexOf("bookeen/cybook") != -1) {
$("html").addClass("cybook");
}
if ($(window).width() < 520) {
var didScroll;
var lastScrollTop = 0;
var delta = 5;
// on scroll, let the interval function know the user has scrolled
$(window).scroll(function (event) {
didScroll = true;
});
// run hasScrolled() and reset didScroll status
setInterval(function () {
if (didScroll) {
hasScrolled();
didScroll = false;
}
}, 250);
function hasScrolled() {
var st = $(this).scrollTop();
// Make sure they scroll more than delta
if (Math.abs(lastScrollTop - st) <= delta)
return;
// If they scrolled down and are past the navbar, add class .nav-up.
// This is necessary so you never see what is "behind" the navbar.
if (st > lastScrollTop) {
// Scroll Down
$('#kiwixtoolbar').css({ top: '-100%' });
} else {
// Scroll Up
$('#kiwixtoolbar').css({ top: '0' });
}
lastScrollTop = st;
}
}
$('#kiwixsearchbox').on({
focus: function () {
$('.kiwix_searchform').addClass('full_width');
$('label[for="kiwix_button_show_toggle"], .kiwix_button_cont').addClass('searching');
},
blur: function () {
$('.kiwix_searchform').removeClass('full_width');
$('label[for="kiwix_button_show_toggle"], .kiwix_button_cont').removeClass('searching');
}
});
// run hasScrolled() and reset didScroll status
setInterval(function () {
if (didScroll) {
hasScrolled();
didScroll = false;
}
}, 250);
function hasScrolled() {
var st = $(this).scrollTop();
// Make sure they scroll more than delta
if (Math.abs(lastScrollTop - st) <= delta)
return;
// If they scrolled down and are past the navbar, add class .nav-up.
// This is necessary so you never see what is "behind" the navbar.
if (st > lastScrollTop) {
// Scroll Down
$('#kiwixtoolbar').css({ top: '-100%' });
} else {
// Scroll Up
$('#kiwixtoolbar').css({ top: '0' });
}
lastScrollTop = st;
}
}
$('#kiwixsearchbox').on({
focus: function () {
$('.kiwix_searchform').addClass('full_width');
$('label[for="kiwix_button_show_toggle"], .kiwix_button_cont').addClass('searching');
},
blur: function () {
$('.kiwix_searchform').removeClass('full_width');
$('label[for="kiwix_button_show_toggle"], .kiwix_button_cont').removeClass('searching');
}
});
})(jQuery);
})(jq);
})

View File

@@ -9,5 +9,10 @@
<p>
The requested URL "{{url}}" was not found on this server.
</p>
{{#details}}
<p>
{{{details}}}
</p>
{{/details}}
</body>
</html>

View File

@@ -6,7 +6,7 @@
</head>
<body class="kiwix">
<h1>External link blocked</h1>
<p>This instance of Kiwix protects you from accidentaly going to external (out-of ZIM) links.</p>
<p>This instance of Kiwix protects you from accidentally going to external (out-of ZIM) links.</p>
<p>If you intend to go to such locations, please click the link below.</p>
<p><a href="{{ source }}">Go to {{ source }}</a></p>
<div id="kiwixfooter">Powered by <a href="https://kiwix.org">Kiwix</a></div>

View File

@@ -0,0 +1,12 @@
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:opds="http://opds-spec.org/2010/catalog">
<id>{{feed_id}}</id>
<title>{{^filter}}All zims{{/filter}}{{#filter}}Filtered zims ({{filter}}){{/filter}}</title>
<updated>{{date}}</updated>
{{#filter}}
<totalResults>{{totalResults}}</totalResults>
<startIndex>{{startIndex}}</startIndex>
<itemsPerPage>{{itemsPerPage}}</itemsPerPage>
{{/filter}}
<link rel="self" href="" type="application/atom+xml" />
<link rel="search" type="application/opensearchdescription+xml" href="{{root}}/catalog/searchdescription.xml" />
{{#books}}{{{entry}}}{{/books}}</feed>

View File

@@ -0,0 +1,25 @@
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:opds="https://specs.opds.io/opds-1.2">
<id>{{feed_id}}</id>
<link rel="self"
href="{{endpoint_root}}/categories"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<link rel="start"
href="{{endpoint_root}}/root.xml"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<title>List of categories</title>
<updated>{{date}}</updated>
{{#categories}}
<entry>
<title>{{name}}</title>
<link rel="subsection"
href="{{endpoint_root}}/entries?category={{{urlencoded_name}}}"
type="application/atom+xml;profile=opds-catalog;kind=acquisition"/>
<updated>{{updated}}</updated>
<id>{{id}}</id>
<content type="text">All entries with category of '{{name}}'.</content>
</entry>
{{/categories}}
</feed>

View File

@@ -0,0 +1,24 @@
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:opds="https://specs.opds.io/opds-1.2"
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
<id>{{feed_id}}</id>
<link rel="self"
href="{{endpoint_root}}/{{#dump_partial_entries}}partial_{{/dump_partial_entries}}entries{{{query}}}"
type="application/atom+xml;profile=opds-catalog;kind=acquisition"/>
<link rel="start"
href="{{endpoint_root}}/root.xml"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<link rel="up"
href="{{endpoint_root}}/root.xml"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<title>{{^filter}}All Entries{{/filter}}{{#filter}}Filtered Entries ({{filter}}){{/filter}}</title>
<updated>{{date}}</updated>
{{#filter}}
<totalResults>{{totalResults}}</totalResults>
<startIndex>{{startIndex}}</startIndex>
<itemsPerPage>{{itemsPerPage}}</itemsPerPage>
{{/filter}}
{{#books}}{{{entry}}}{{/books}}</feed>

Some files were not shown because too many files have changed in this diff Show More