Compare commits

..

5 Commits

Author SHA1 Message Date
Manan Jethwani
9d5abd7e35 replaced topbar with custom topbar 2021-05-11 20:25:05 +05:30
Manan Jethwani
619dbf4e85 fixed resizing issue in subset loading 2021-05-10 20:46:35 +05:30
Manan Jethwani
766f12aaf7 added subset loading(solves kiwix-tools/issue/457) 2021-05-10 17:38:12 +05:30
Manan Jethwani
5dce6d0e40 added html encoding 2021-05-10 15:33:54 +05:30
Manan Jethwani
1dbed28499 added dynamic loading of data using OPDS 2021-05-10 15:33:54 +05:30
47 changed files with 528 additions and 1716 deletions

View File

@@ -134,7 +134,7 @@ jobs:
if [[ "${{matrix.target}}" =~ android_.* ]]; then
MESON_OPTION="$MESON_OPTION -Dandroid=true"
fi
cd $HOME/libkiwix
cd $HOME/kiwix-lib
meson . build ${MESON_OPTION}
cd build
ninja
@@ -145,7 +145,7 @@ jobs:
if: startsWith(matrix.target, 'native_')
shell: bash
run: |
cd $HOME/libkiwix/build
cd $HOME/kiwix-lib/build
meson test --verbose
ninja coverage
env:
@@ -154,7 +154,7 @@ jobs:
- name: Publish coverage
shell: bash
run: |
cd $HOME/libkiwix
cd $HOME/kiwix-lib
curl https://codecov.io/bash -o codecov.sh
bash codecov.sh -n "${OS_NAME}_${{matrix.target}}" -Z
rm codecov.sh

1
.gitignore vendored
View File

@@ -4,4 +4,3 @@ subprojects/googletest-release*
*.class
build/
.vscode/
builddir/

View File

@@ -1,8 +1,3 @@
libkiwix 10.0.0
===============
* ...
kiwix-lib 9.4.1
===============

View File

@@ -1,50 +1,50 @@
Libkiwix
========
Kiwix library
=============
The Libkiwix provides the [Kiwix](https://kiwix.org) software suite
core. It contains the code shared by all Kiwix ports (Windows,
The Kiwix library provides the [Kiwix](https://kiwix.org) software
suite core. It contains the code shared by all Kiwix ports (Windows,
GNU/Linux, macOS, Android, iOS, ...).
[![Repositories](https://img.shields.io/repology/repositories/libkiwix?label=repositories)](https://github.com/kiwix/libkiwix/wiki/Repology)
[![Build Status](https://github.com/kiwix/libkiwix/workflows/CI/badge.svg?query=branch%3Amaster)](https://github.com/kiwix/libkiwix/actions?query=branch%3Amaster)
[![CodeFactor](https://www.codefactor.io/repository/github/kiwix/libkiwix/badge)](https://www.codefactor.io/repository/github/kiwix/libkiwix)
[![Codecov](https://codecov.io/gh/kiwix/libkiwix/branch/master/graph/badge.svg)](https://codecov.io/gh/kiwix/libkiwix)
[![Download](https://api.bintray.com/packages/kiwix/kiwix/kiwixlib/images/download.svg)](https://bintray.com/kiwix/kiwix/kiwixlib/_latestVersion)
[![Build Status](https://github.com/kiwix/kiwix-lib/workflows/CI/badge.svg?query=branch%3Amaster)](https://github.com/kiwix/kiwix-lib/actions?query=branch%3Amaster)
[![CodeFactor](https://www.codefactor.io/repository/github/kiwix/kiwix-lib/badge)](https://www.codefactor.io/repository/github/kiwix/kiwix-lib)
[![Codecov](https://codecov.io/gh/kiwix/kiwix-lib/branch/master/graph/badge.svg)](https://codecov.io/gh/kiwix/kiwix-lib)
[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
[![Packaging status](https://repology.org/badge/vertical-allrepos/kiwix-lib.svg)](https://repology.org/project/kiwix-lib/versions)
Disclaimer
----------
This document assumes you have a little knowledge about software
compilation. If you experience difficulties with the dependencies or
with the Libkiwix compilation itself, we recommend to have a look to
[kiwix-build](https://github.com/kiwix/kiwix-build).
with the Kiwix libary compilation itself, we recommend to have a look
to [kiwix-build](https://github.com/kiwix/kiwix-build).
Preamble
--------
Although the Libkiwix can be (cross-)compiled on/for many sytems, the
following documentation explains how to do it on POSIX ones. It is
primarly thought for GNU/Linux systems and has been tested on recent
releases of Ubuntu and Fedora.
Although the Kiwix library can be (cross-)compiled on/for many
sytems, the following documentation explains how to do it on POSIX
ones. It is primarly thought for GNU/Linux systems and has been tested
on recent releases of Ubuntu and Fedora.
Dependencies
------------
The Libkiwix relies on many third party software libraries. They are
prerequisites to the Libkiwix compilation. Following libraries need to
be available:
The Kiwix library relies on many third parts software libraries. They
are prerequisites to the Kiwix library compilation. Following
libraries need to be available:
* [ICU](https://site.icu-project.org/) (package `libicu-dev` on Ubuntu)
* [ZIM](https://openzim.org/) (package `libzim-dev` on Ubuntu)
* [Pugixml](https://pugixml.org/) (package `libpugixml-dev` on Ubuntu)
* [Mustache](https://github.com/kainjow/Mustache) (Just copy the
header `mustache.hpp` somewhere it can be found by the compiler and/or
set CPPFLAGS with correct `-I` option). Use Mustache version 4.1 or above.
* [Libcurl](https://curl.se/libcurl) (`libcurl4-gnutls-dev`, `libcurl4-nss-dev` or `libcurl4-openssl-dev` on Ubuntu)
* [Microhttpd](https://www.gnu.org/software/libmicrohttpd) (package `libmicrohttpd-dev` on Ubuntu)
* [Zlib](https://zlib.net/) (package `zlib1g-dev` on Ubuntu)
To test the code:
* [Google Test](https://github.com/google/googletest) (package `googletest` on Ubuntu)
* [libcurl](https://curl.se/libcurl) (`libcurl4-gnutls-dev`, `libcurl4-nss-dev` or `libcurl4-openssl-dev` on Ubuntu)
* [microhttpd](https://www.gnu.org/software/libmicrohttpd) (package `libmicrohttpd-dev` on Ubuntu)
* [zlib](https://zlib.net/) (package `zlib1g-dev` on Ubuntu)
The following dependency needs to be available at runtime:
* [Aria2](https://aria2.github.io/) (package `aria2` on Ubuntu)
@@ -56,12 +56,12 @@ In the worse case, you will have to download and compile bleeding edge
version by hand.
If you want to install these dependencies locally, then use the
`libkiwix` directory as install prefix.
`kiwix-lib` directory as install prefix.
Environment
-------------
The Libkiwix builds using [Meson](https://mesonbuild.com/) version
The Kiwix library builds using [Meson](https://mesonbuild.com/) version
0.45 or higher. Meson relies itself on Ninja, pkg-config and few other
compilation tools.
@@ -77,7 +77,7 @@ section.
Compilation
-----------
Once all dependencies are installed, you can compile the Libkiwix
Once all dependencies are installed, you can compile the Kiwix library
with:
```bash
meson . build
@@ -85,7 +85,7 @@ ninja -C build
```
By default, it will compile dynamic linked libraries. All binary files
will be created in the `build` directory created automatically by
will be created in the "build" directory created automatically by
Meson. If you want statically linked libraries, you can add
`--default-library=static` option to the Meson command.
@@ -103,7 +103,7 @@ meson test
Installation
------------
If you want to install the Libkiwix and the headers you just have
If you want to install the Kiwix library and the headers you just have
compiled on your system, here we go:
```bash
ninja -C build install
@@ -146,10 +146,6 @@ cp ninja ../bin
cd ..
```
If you compile manually Libmicrohttpd, you might need to compile it
without GNU TLS, a bug here will empeach further compilation
otherwise.
If the compilation still fails, you might need to get a more recent
version of a dependency than the one packaged by your Linux
distribution. Try then with a source tarball distributed by the

View File

@@ -29,7 +29,7 @@ task writePom {
version '10.0.0' + (System.env.KIWIXLIB_BUILDVERSION == null ? '' : '-'+System.env.KIWIXLIB_BUILDVERSION)
packaging 'aar'
name 'kiwixlib'
url 'https://github.com/kiwix/libkiwix'
url 'https://github.com/kiwix/kiwix-lib'
licenses {
license {
name 'GPLv3'
@@ -44,9 +44,9 @@ task writePom {
}
}
scm {
connection 'https://github.com/kiwix/libkiwix.git'
developerConnection 'https://github.com/kiwix/libkiwix.git'
url 'https://github.com/kiwix/libkiwix'
connection 'https://github.com/kiwix/kiwix-lib.git'
developerConnection 'https://github.com/kiwix/kiwix-lib.git'
url 'https://github.com/kiwix/kiwix-lib'
}
}
}.withXml {

2
debian/control vendored
View File

@@ -15,7 +15,7 @@ Build-Depends: debhelper-compat (= 13),
zlib1g-dev
Standards-Version: 4.5.0
Section: libs
Homepage: https://github.com/kiwix/libkiwix
Homepage: https://github.com/kiwix/kiwix-lib
Rules-Requires-Root: no
Package: libkiwix-dev

View File

@@ -103,7 +103,7 @@ class Entry
* Some entry (ie binary ones) have their content plain stored
* in the zim file. Knowing the offset where the content is stored
* an user can directly read the content in the zim file bypassing the
* libkiwix/libzim.
* kiwix-lib/libzim.
*
* @return A pair specifying where to read the content.
* The string is the real file to read (may be different that .zim

View File

@@ -233,19 +233,12 @@ class Library
unsigned int getBookCount(const bool localBooks, const bool remoteBooks) const;
/**
* Get all languagues of the books in the library.
* Get all langagues of the books in the library.
*
* @return A list of languages.
*/
std::vector<std::string> getBooksLanguages() const;
/**
* Get all categories of the books in the library.
*
* @return A list of categories.
*/
std::vector<std::string> getBooksCategories() const;
/**
* Get all book creators of the books in the library.
*

View File

@@ -51,35 +51,24 @@ class OPDSDumper
/**
* Dump the OPDS feed.
*
* @param bookIds the ids of the books to include in the feed
* @param query the query used to obtain the list of book ids
* @param id The id of the library.
* @return The OPDS feed.
*/
std::string dumpOPDSFeed(const std::vector<std::string>& bookIds, const std::string& query) const;
std::string dumpOPDSFeed(const std::vector<std::string>& bookIds);
/**
* Dump the OPDS feed.
*
* @param bookIds the ids of the books to include in the feed
* @param query the query used to obtain the list of book ids
* @return The OPDS feed.
*/
std::string dumpOPDSFeedV2(const std::vector<std::string>& bookIds, const std::string& query) const;
/**
* Dump the categories OPDS feed.
*
* @param categories list of category names
* @return The OPDS feed.
*/
std::string categoriesOPDSFeed(const std::vector<std::string>& categories) const;
/**
* Set the id of the library.
* Set the id of the opds stream.
*
* @param id the id to use.
*/
void setLibraryId(const std::string& id) { this->libraryId = id;}
void setId(const std::string& id) { this->id = id;}
/**
* Set the title oft the opds stream.
*
* @param title the title to use.
*/
void setTitle(const std::string& title) { this->title = title; }
/**
* Set the root location used when generating url.
@@ -88,6 +77,13 @@ class OPDSDumper
*/
void setRootLocation(const std::string& rootLocation) { this->rootLocation = rootLocation; }
/**
* Set the search url.
*
* @param searchUrl the search url to use.
*/
void setSearchDescriptionUrl(const std::string& searchDescriptionUrl) { this->searchDescriptionUrl = searchDescriptionUrl; }
/**
* Set some informations about the search results.
*
@@ -97,13 +93,27 @@ class OPDSDumper
*/
void setOpenSearchInfo(int totalResult, int startIndex, int count);
/**
* Set the library to dump.
*
* @param library The library to dump.
*/
void setLibrary(Library* library) { this->library = library; }
protected:
kiwix::Library* library;
std::string libraryId;
std::string id;
std::string title;
std::string date;
std::string rootLocation;
std::string searchDescriptionUrl;
int m_totalResults;
int m_startIndex;
int m_count;
bool m_isSearchResult = false;
private:
pugi::xml_node handleBook(Book book, pugi::xml_node root_node);
};
}

View File

@@ -37,47 +37,12 @@ using namespace std;
namespace kiwix
{
/**
* The SuggestionItem is a helper class that contains the info about a single
* suggestion item.
*/
class SuggestionItem
{
// Functions
private:
// Create a sugggestion item.
explicit SuggestionItem(std::string title, std::string normalizedTitle,
std::string path, std::string snippet = "") :
title(title),
normalizedTitle(normalizedTitle),
path(path),
snippet(snippet) {}
public:
const std::string getTitle() {return title;}
const std::string getNormalizedTitle() {return normalizedTitle;}
const std::string getPath() {return path;}
const std::string getSnippet() {return snippet;}
const bool hasSnippet() {return !snippet.empty();}
// Data
private:
std::string title;
std::string normalizedTitle;
std::string path;
std::string snippet;
friend class Reader;
};
/**
* The Reader class is the class who allow to get an entry content from a zim
* file.
*/
using SuggestionsList_t = std::vector<SuggestionItem>;
using SuggestionsList_t = std::vector<std::vector<std::string>>;
class Reader
{
public:

View File

@@ -27,7 +27,7 @@
#include <cctype>
#include <locale>
#include <string>
#include <memory>
#include <vector>
#include <vector>
#include "tools/pathTools.h"
#include "tools/stringTools.h"
@@ -48,7 +48,7 @@ class Result
virtual std::string get_content() = 0;
virtual int get_wordCount() = 0;
virtual int get_size() = 0;
virtual std::string get_zimId() = 0;
virtual int get_readerIndex() = 0;
};
struct SearcherInternal;
@@ -154,7 +154,7 @@ class Searcher
const bool verbose = false);
std::vector<Reader*> readers;
std::unique_ptr<SearcherInternal> internal;
SearcherInternal* internal;
std::string searchPattern;
unsigned int estimatedResultCount;
unsigned int resultStart;

View File

@@ -24,7 +24,6 @@
#include <vector>
#include <map>
#include <zim/zim.h>
#include <mustache.hpp>
namespace pugi {
class xml_node;
@@ -46,11 +45,6 @@ namespace kiwix
using MimeCounterType = std::map<const std::string, zim::entry_index_type>;
MimeCounterType parseMimetypeCounter(const std::string& counterData);
std::string gen_date_str();
std::string gen_uuid(const std::string& s);
std::string render_template(const std::string& template_str, kainjow::mustache::data data);
}
#endif

View File

@@ -1,4 +1,4 @@
project('libkiwix', 'cpp',
project('kiwix-lib', 'cpp',
version : '10.0.0', # Also change this in android-kiwix-lib-publisher/kiwixLibAndroid/build.gradle
license : 'GPLv3+',
default_options : ['c_std=c11', 'cpp_std=c++11', 'werror=true'])
@@ -18,7 +18,7 @@ if wrapper.contains('java')
add_languages('java')
endif
# See https://github.com/kiwix/libkiwix/issues/371
# See https://github.com/kiwix/kiwix-lib/issues/371
if ['arm', 'mips', 'm68k', 'ppc', 'sh4'].contains(target_machine.cpu_family())
extra_libs += '-latomic'
endif
@@ -56,10 +56,6 @@ if target_machine.system() == 'windows' and static_deps
extra_cflags += '-DCURL_STATICLIB'
endif
if target_machine.system() == 'windows'
add_project_arguments('-DNOMINMAX', language: 'cpp')
endif
all_deps = [thread_dep, libicu_dep, libzim_dep, pugixml_dep, libcurl_dep, microhttpd_dep, zlib_dep, xapian_dep]
inc = include_directories('include', extra_include)

View File

@@ -201,21 +201,6 @@ std::vector<std::string> Library::getBooksLanguages() const
return booksLanguages;
}
std::vector<std::string> Library::getBooksCategories() const
{
std::set<std::string> categories;
for (const auto& pair: m_books) {
const auto& book = pair.second;
const auto& c = book.getCategory();
if ( !c.empty() ) {
categories.insert(c);
}
}
return std::vector<std::string>(categories.begin(), categories.end());
}
std::vector<std::string> Library::getBooksCreators() const
{
std::vector<std::string> booksCreators;

View File

@@ -25,8 +25,7 @@ kiwix_sources = [
'server/etag.cpp',
'server/request_context.cpp',
'server/response.cpp',
'server/internalServer.cpp',
'server/internalServer_catalog_v2.cpp'
'server/internalServer.cpp'
]
kiwix_sources += lib_resources

View File

@@ -21,13 +21,10 @@
#include "book.h"
#include "tools/otherTools.h"
#include "kiwixlib-resources.h"
#include <mustache.hpp>
#include <iomanip>
namespace kiwix
{
/* Constructor */
OPDSDumper::OPDSDumper(Library* library)
: library(library)
@@ -38,111 +35,121 @@ OPDSDumper::~OPDSDumper()
{
}
std::string gen_date_str()
{
auto now = time(0);
auto tm = localtime(&now);
std::stringstream is;
is << std::setw(2) << std::setfill('0')
<< 1900+tm->tm_year << "-"
<< std::setw(2) << std::setfill('0') << tm->tm_mon+1 << "-"
<< std::setw(2) << std::setfill('0') << tm->tm_mday << "T"
<< std::setw(2) << std::setfill('0') << tm->tm_hour << ":"
<< std::setw(2) << std::setfill('0') << tm->tm_min << ":"
<< std::setw(2) << std::setfill('0') << tm->tm_sec << "Z";
return is.str();
}
static std::string gen_date_from_yyyy_mm_dd(const std::string& date)
{
std::stringstream is;
is << date << "T00:00::00Z";
return is.str();
}
void OPDSDumper::setOpenSearchInfo(int totalResults, int startIndex, int count)
{
m_totalResults = totalResults;
m_startIndex = startIndex,
m_count = count;
m_isSearchResult = true;
}
namespace
{
#define ADD_TEXT_ENTRY(node, child, value) (node).append_child((child)).append_child(pugi::node_pcdata).set_value((value).c_str())
typedef kainjow::mustache::data MustacheData;
typedef kainjow::mustache::list BookData;
pugi::xml_node OPDSDumper::handleBook(Book book, pugi::xml_node root_node) {
auto entry_node = root_node.append_child("entry");
ADD_TEXT_ENTRY(entry_node, "id", "urn:uuid:"+book.getId());
ADD_TEXT_ENTRY(entry_node, "title", book.getTitle());
ADD_TEXT_ENTRY(entry_node, "summary", book.getDescription());
ADD_TEXT_ENTRY(entry_node, "language", book.getLanguage());
ADD_TEXT_ENTRY(entry_node, "updated", gen_date_from_yyyy_mm_dd(book.getDate()));
ADD_TEXT_ENTRY(entry_node, "name", book.getName());
ADD_TEXT_ENTRY(entry_node, "flavour", book.getFlavour());
ADD_TEXT_ENTRY(entry_node, "category", book.getCategory());
ADD_TEXT_ENTRY(entry_node, "tags", book.getTags());
ADD_TEXT_ENTRY(entry_node, "articleCount", to_string(book.getArticleCount()));
ADD_TEXT_ENTRY(entry_node, "mediaCount", to_string(book.getMediaCount()));
ADD_TEXT_ENTRY(entry_node, "icon", rootLocation + "/meta?name=favicon&content=" + book.getHumanReadableIdFromPath());
BookData getBookData(const Library* library, const std::vector<std::string>& bookIds)
{
BookData bookData;
for ( const auto& bookId : bookIds ) {
const Book& book = library->getBookById(bookId);
const MustacheData bookUrl = book.getUrl().empty()
? MustacheData(false)
: MustacheData(book.getUrl());
bookData.push_back(kainjow::mustache::object{
{"id", "urn:uuid:"+book.getId()},
{"name", book.getName()},
{"title", book.getTitle()},
{"description", book.getDescription()},
{"language", book.getLanguage()},
{"content_id", book.getHumanReadableIdFromPath()},
{"updated", book.getDate() + "T00:00:00Z"},
{"category", book.getCategory()},
{"flavour", book.getFlavour()},
{"tags", book.getTags()},
{"article_count", to_string(book.getArticleCount())},
{"media_count", to_string(book.getMediaCount())},
{"author_name", book.getCreator()},
{"publisher_name", book.getPublisher()},
{"url", bookUrl},
{"size", to_string(book.getSize())},
});
auto content_node = entry_node.append_child("link");
content_node.append_attribute("type") = "text/html";
content_node.append_attribute("href") = (rootLocation + "/" + book.getHumanReadableIdFromPath()).c_str();
auto author_node = entry_node.append_child("author");
ADD_TEXT_ENTRY(author_node, "name", book.getCreator());
auto publisher_node = entry_node.append_child("publisher");
ADD_TEXT_ENTRY(publisher_node, "name", book.getPublisher());
if (! book.getUrl().empty()) {
auto acquisition_link = entry_node.append_child("link");
acquisition_link.append_attribute("rel") = "http://opds-spec.org/acquisition/open-access";
acquisition_link.append_attribute("type") = "application/x-zim";
acquisition_link.append_attribute("href") = book.getUrl().c_str();
acquisition_link.append_attribute("length") = to_string(book.getSize()).c_str();
}
return bookData;
if (! book.getFaviconMimeType().empty() ) {
auto image_link = entry_node.append_child("link");
image_link.append_attribute("rel") = "http://opds-spec.org/image/thumbnail";
image_link.append_attribute("type") = book.getFaviconMimeType().c_str();
image_link.append_attribute("href") = (rootLocation + "/meta?name=favicon&content=" + book.getHumanReadableIdFromPath()).c_str();
}
return entry_node;
}
} // unnamed namespace
string OPDSDumper::dumpOPDSFeed(const std::vector<std::string>& bookIds, const std::string& query) const
string OPDSDumper::dumpOPDSFeed(const std::vector<std::string>& bookIds)
{
const auto bookData = getBookData(library, bookIds);
const kainjow::mustache::object template_data{
{"date", gen_date_str()},
{"root", rootLocation},
{"feed_id", gen_uuid(libraryId + "/catalog/search?"+query)},
{"filter", query.empty() ? MustacheData(false) : MustacheData(query)},
{"totalResults", to_string(m_totalResults)},
{"startIndex", to_string(m_startIndex)},
{"itemsPerPage", to_string(m_count)},
{"books", bookData }
};
date = gen_date_str();
pugi::xml_document doc;
return render_template(RESOURCE::templates::catalog_entries_xml, template_data);
}
auto root_node = doc.append_child("feed");
root_node.append_attribute("xmlns") = "http://www.w3.org/2005/Atom";
root_node.append_attribute("xmlns:opds") = "http://opds-spec.org/2010/catalog";
string OPDSDumper::dumpOPDSFeedV2(const std::vector<std::string>& bookIds, const std::string& query) const
{
const auto bookData = getBookData(library, bookIds);
ADD_TEXT_ENTRY(root_node, "id", id);
const kainjow::mustache::object template_data{
{"date", gen_date_str()},
{"endpoint_root", rootLocation + "/catalog/v2"},
{"feed_id", gen_uuid(libraryId + "/entries?"+query)},
{"filter", query.empty() ? MustacheData(false) : MustacheData(query)},
{"query", query.empty() ? "" : "?" + urlEncode(query)},
{"totalResults", to_string(m_totalResults)},
{"startIndex", to_string(m_startIndex)},
{"itemsPerPage", to_string(m_count)},
{"books", bookData }
};
ADD_TEXT_ENTRY(root_node, "title", title);
ADD_TEXT_ENTRY(root_node, "updated", date);
return render_template(RESOURCE::templates::catalog_v2_entries_xml, template_data);
}
std::string OPDSDumper::categoriesOPDSFeed(const std::vector<std::string>& categories) const
{
const auto now = gen_date_str();
kainjow::mustache::list categoryData;
for ( const auto& category : categories ) {
const auto urlencodedCategoryName = urlEncode(category);
categoryData.push_back(kainjow::mustache::object{
{"name", category},
{"urlencoded_name", urlencodedCategoryName},
{"updated", now},
{"id", gen_uuid(libraryId + "/categories/" + urlencodedCategoryName)}
});
if (m_isSearchResult) {
ADD_TEXT_ENTRY(root_node, "totalResults", to_string(m_totalResults));
ADD_TEXT_ENTRY(root_node, "startIndex", to_string(m_startIndex));
ADD_TEXT_ENTRY(root_node, "itemsPerPage", to_string(m_count));
}
return render_template(
RESOURCE::templates::catalog_v2_categories_xml,
kainjow::mustache::object{
{"date", now},
{"endpoint_root", rootLocation + "/catalog/v2"},
{"feed_id", gen_uuid(libraryId + "/categories")},
{"categories", categoryData }
}
);
auto self_link_node = root_node.append_child("link");
self_link_node.append_attribute("rel") = "self";
self_link_node.append_attribute("href") = "";
self_link_node.append_attribute("type") = "application/atom+xml";
if (!searchDescriptionUrl.empty() ) {
auto search_link = root_node.append_child("link");
search_link.append_attribute("rel") = "search";
search_link.append_attribute("type") = "application/opensearchdescription+xml";
search_link.append_attribute("href") = searchDescriptionUrl.c_str();
}
if (library) {
for (auto& bookId: bookIds) {
handleBook(library->getBookById(bookId), root_node);
}
}
return nodeToString(root_node);
}
}

View File

@@ -184,7 +184,8 @@ Entry Reader::getMainPage() const
bool Reader::getFavicon(string& content, string& mimeType) const
{
try {
auto item = zimArchive->getIllustrationItem();
auto entry = zimArchive->getFaviconEntry();
auto item = entry.getItem(true);
content = item.getData();
mimeType = item.getMimetype();
return true;
@@ -431,12 +432,12 @@ bool Reader::searchSuggestions(const string& prefix,
article is already in the suggestions list (with an other
title) */
bool insert = true;
std::vector<SuggestionItem>::iterator suggestionItr;
std::vector<std::vector<std::string>>::iterator suggestionItr;
for (suggestionItr = results.begin();
suggestionItr != results.end();
suggestionItr++) {
int result = normalizedArticleTitle.compare((*suggestionItr).getNormalizedTitle());
if (result == 0 && articleFinalUrl.compare((*suggestionItr).getPath()) == 0) {
int result = normalizedArticleTitle.compare((*suggestionItr)[2]);
if (result == 0 && articleFinalUrl.compare((*suggestionItr)[1]) == 0) {
insert = false;
break;
} else if (result < 0) {
@@ -446,7 +447,10 @@ bool Reader::searchSuggestions(const string& prefix,
/* Insert if possible */
if (insert) {
SuggestionItem suggestion(entry.getTitle(), normalizedArticleTitle, articleFinalUrl);
std::vector<std::string> suggestion;
suggestion.push_back(entry.getTitle());
suggestion.push_back(articleFinalUrl);
suggestion.push_back(normalizedArticleTitle);
results.insert(suggestionItr, suggestion);
}
@@ -491,19 +495,19 @@ bool Reader::searchSuggestionsSmart(const string& prefix,
bool retVal = false;
/* Try to search in the title using fulltext search database */
auto suggestionSearch = zim::Search(*zimArchive);
suggestionSearch.set_query(prefix);
suggestionSearch.set_range(0, suggestionsCount);
suggestionSearch.set_suggestion_mode(true);
auto suggestionSearcher = zim::Searcher(*zimArchive);
zim::Query suggestionQuery;
suggestionQuery.setQuery(prefix, true);
auto suggestionSearch = suggestionSearcher.search(suggestionQuery);
if (suggestionSearch.getEstimatedMatches()) {
const auto suggestions = suggestionSearch.getResults(0, suggestionsCount);
for (auto current = suggestions.begin();
current != suggestions.end();
if (suggestionSearch.get_matches_estimated()) {
for (auto current = suggestionSearch.begin();
current != suggestionSearch.end();
current++) {
SuggestionItem suggestion(current.getTitle(), kiwix::normalize(current.getTitle()),
current.getPath(), current.getSnippet());
std::vector<std::string> suggestion;
suggestion.push_back(current->getTitle());
suggestion.push_back(current->getPath());
suggestion.push_back(kiwix::normalize(current->getTitle()));
results.push_back(suggestion);
}
retVal = true;
@@ -524,7 +528,7 @@ bool Reader::getNextSuggestion(string& title)
{
if (this->suggestionsOffset != this->suggestions.end()) {
/* title */
title = (*(this->suggestionsOffset)).getTitle();
title = (*(this->suggestionsOffset))[0];
/* increment the cursor for the next call */
this->suggestionsOffset++;
@@ -539,8 +543,8 @@ bool Reader::getNextSuggestion(string& title, string& url)
{
if (this->suggestionsOffset != this->suggestions.end()) {
/* title */
title = (*(this->suggestionsOffset)).getTitle();
url = (*(this->suggestionsOffset)).getPath();
title = (*(this->suggestionsOffset))[0];
url = (*(this->suggestionsOffset))[1];
/* increment the cursor for the next call */
this->suggestionsOffset++;

View File

@@ -77,7 +77,9 @@ std::string SearchRenderer::getHtml()
result.set("title", p_result->get_title());
result.set("url", p_result->get_url());
result.set("snippet", p_result->get_snippet());
result.set("resultContentId", mp_nameMapper->getNameForId(p_result->get_zimId()));
auto readerIndex = p_result->get_readerIndex();
auto reader = mp_searcher->get_reader(readerIndex);
result.set("resultContentId", mp_nameMapper->getNameForId(reader->getId()));
if (p_result->get_wordCount() >= 0) {
result.set("wordCount", kiwix::beautifyInteger(p_result->get_wordCount()));

View File

@@ -35,7 +35,7 @@ namespace kiwix
class _Result : public Result
{
public:
_Result(zim::SearchResultSet::iterator iterator);
_Result(zim::Search::iterator& iterator);
virtual ~_Result(){};
virtual std::string get_url();
@@ -45,25 +45,29 @@ class _Result : public Result
virtual std::string get_content();
virtual int get_wordCount();
virtual int get_size();
virtual std::string get_zimId();
virtual int get_readerIndex();
private:
zim::SearchResultSet::iterator iterator;
zim::Search::iterator iterator;
};
struct SearcherInternal : zim::SearchResultSet {
explicit SearcherInternal(const zim::SearchResultSet& srs)
: zim::SearchResultSet(srs)
, current_iterator(srs.begin())
{
}
struct SearcherInternal {
const zim::Search* _search;
zim::Search::iterator current_iterator;
zim::SearchResultSet::iterator current_iterator;
SearcherInternal() : _search(NULL) {}
~SearcherInternal()
{
if (_search != NULL) {
delete _search;
}
}
};
/* Constructor */
Searcher::Searcher()
: searchPattern(""),
: internal(new SearcherInternal()),
searchPattern(""),
estimatedResultCount(0),
resultStart(0),
resultEnd(0)
@@ -74,6 +78,7 @@ Searcher::Searcher()
/* Destructor */
Searcher::~Searcher()
{
delete internal;
}
bool Searcher::add_reader(Reader* reader)
@@ -117,13 +122,13 @@ void Searcher::search(const std::string& search,
archives.push_back(*(*current)->getZimArchive());
}
}
zim::Searcher searcher(archives);
zim::Query query;
query.setQuery(unaccentedSearch, false);
query.setVerbose(verbose);
zim::Search search = searcher.search(query);
internal.reset(new SearcherInternal(search.getResults(resultStart, resultEnd)));
this->estimatedResultCount = search.getEstimatedMatches();
zim::Search* search = new zim::Search(archives);
search->set_verbose(verbose);
search->set_query(unaccentedSearch);
search->set_range(resultStart, resultEnd);
internal->_search = search;
internal->current_iterator = internal->_search->begin();
this->estimatedResultCount = internal->_search->get_matches_estimated();
}
return;
@@ -158,28 +163,28 @@ void Searcher::geo_search(float latitude, float longitude, float distance,
current++) {
archives.push_back(*(*current)->getZimArchive());
}
zim::Searcher searcher(archives);
zim::Query query;
query.setVerbose(verbose);
query.setQuery("", false);
query.setGeorange(latitude, longitude, distance);
zim::Search search = searcher.search(query);
internal.reset(new SearcherInternal(search.getResults(resultStart, resultEnd)));
this->estimatedResultCount = search.getEstimatedMatches();
zim::Search* search = new zim::Search(archives);
search->set_verbose(verbose);
search->set_query("");
search->set_georange(latitude, longitude, distance);
search->set_range(resultStart, resultEnd);
internal->_search = search;
internal->current_iterator = internal->_search->begin();
this->estimatedResultCount = internal->_search->get_matches_estimated();
}
void Searcher::restart_search()
{
if (internal.get()) {
internal->current_iterator = internal->begin();
if (internal->_search) {
internal->current_iterator = internal->_search->begin();
}
}
Result* Searcher::getNextResult()
{
if (internal.get() &&
internal->current_iterator != internal->end()) {
if (internal->_search &&
internal->current_iterator != internal->_search->end()) {
Result* result = new _Result(internal->current_iterator);
internal->current_iterator++;
return result;
@@ -213,13 +218,14 @@ void Searcher::suggestions(std::string& searchPattern, const bool verbose)
current++) {
archives.push_back(*(*current)->getZimArchive());
}
zim::Searcher searcher(archives);
zim::Query query;
query.setVerbose(verbose);
query.setQuery(unaccentedSearch, true);
zim::Search search = searcher.search(query);
internal.reset(new SearcherInternal(search.getResults(resultStart, resultEnd)));
this->estimatedResultCount = search.getEstimatedMatches();
zim::Search* search = new zim::Search(archives);
search->set_verbose(verbose);
search->set_query(unaccentedSearch);
search->set_range(resultStart, resultEnd);
search->set_suggestion_mode(true);
internal->_search = search;
internal->current_iterator = internal->_search->begin();
this->estimatedResultCount = internal->_search->get_matches_estimated();
}
/* Return the result count estimation */
@@ -228,26 +234,26 @@ unsigned int Searcher::getEstimatedResultCount()
return this->estimatedResultCount;
}
_Result::_Result(zim::SearchResultSet::iterator iterator)
_Result::_Result(zim::Search::iterator& iterator)
: iterator(iterator)
{
}
std::string _Result::get_url()
{
return iterator.getPath();
return iterator.get_path();
}
std::string _Result::get_title()
{
return iterator.getTitle();
return iterator.get_title();
}
int _Result::get_score()
{
return iterator.getScore();
return iterator.get_score();
}
std::string _Result::get_snippet()
{
return iterator.getSnippet();
return iterator.get_snippet();
}
std::string _Result::get_content()
{
@@ -255,17 +261,15 @@ std::string _Result::get_content()
}
int _Result::get_size()
{
return iterator.getSize();
return iterator.get_size();
}
int _Result::get_wordCount()
{
return iterator.getWordCount();
return iterator.get_wordCount();
}
std::string _Result::get_zimId()
int _Result::get_readerIndex()
{
std::ostringstream s;
s << iterator.getZimId();
return s.str();
return iterator.get_fileIndex();
}

View File

@@ -76,21 +76,6 @@ extern "C" {
namespace kiwix {
namespace
{
inline std::string normalizeRootUrl(std::string rootUrl)
{
while ( !rootUrl.empty() && rootUrl.back() == '/' )
rootUrl.pop_back();
while ( !rootUrl.empty() && rootUrl.front() == '/' )
rootUrl = rootUrl.substr(1);
return rootUrl.empty() ? rootUrl : "/" + rootUrl;
}
} // unnamed namespace
static IdNameMapper defaultNameMapper;
static MHD_Result staticHandlerCallback(void* cls,
@@ -115,7 +100,7 @@ InternalServer::InternalServer(Library* library,
bool blockExternalLinks) :
m_addr(addr),
m_port(port),
m_root(normalizeRootUrl(root)),
m_root(root),
m_nbThreads(nbThreads),
m_verbose(verbose),
m_withTaskbar(withTaskbar),
@@ -168,7 +153,6 @@ bool InternalServer::start() {
}
auto server_start_time = std::chrono::system_clock::now().time_since_epoch();
m_server_id = kiwix::to_string(server_start_time.count());
m_library_id = m_server_id;
return true;
}
@@ -253,16 +237,16 @@ std::unique_ptr<Response> InternalServer::handle_request(const RequestContext& r
{
try {
if (! request.is_valid_url())
return Response::build_404(*this, request, "", "");
return Response::build_404(*this, request, "");
const ETag etag = get_matching_if_none_match_etag(request);
if ( etag )
return Response::build_304(*this, etag);
if (startsWith(request.get_url(), "/skin/"))
if (kiwix::startsWith(request.get_url(), "/skin/"))
return handle_skin(request);
if (startsWith(request.get_url(), "/catalog/"))
if (startsWith(request.get_url(), "/catalog"))
return handle_catalog(request);
if (request.get_url() == "/meta")
@@ -297,6 +281,27 @@ MustacheData InternalServer::get_default_data() const
return data;
}
MustacheData InternalServer::homepage_data() const
{
auto data = get_default_data();
MustacheData books{MustacheData::type::list};
for (auto& bookId: mp_library->filter(kiwix::Filter().local(true).valid(true))) {
auto& currentBook = mp_library->getBookById(bookId);
MustacheData book;
book.set("name", mp_nameMapper->getNameForId(bookId));
book.set("title", currentBook.getTitle());
book.set("description", currentBook.getDescription());
book.set("articleCount", beautifyInteger(currentBook.getArticleCount()));
book.set("mediaCount", beautifyInteger(currentBook.getMediaCount()));
books.push_back(book);
}
data.set("books", books);
return data;
}
bool InternalServer::etag_not_needed(const RequestContext& request) const
{
const std::string url = request.get_url();
@@ -320,7 +325,7 @@ InternalServer::get_matching_if_none_match_etag(const RequestContext& r) const
std::unique_ptr<Response> InternalServer::build_homepage(const RequestContext& request)
{
return ContentResponse::build(*this, RESOURCE::templates::index_html, get_default_data(), "text/html; charset=utf-8", true);
return ContentResponse::build(*this, RESOURCE::templates::index_html, homepage_data(), "text/html; charset=utf-8", true);
}
std::unique_ptr<Response> InternalServer::handle_meta(const RequestContext& request)
@@ -335,11 +340,11 @@ std::unique_ptr<Response> InternalServer::handle_meta(const RequestContext& requ
meta_name = request.get_argument("name");
reader = mp_library->getReaderById(bookId);
} catch (const std::out_of_range& e) {
return Response::build_404(*this, request, bookName, "");
return Response::build_404(*this, request, bookName);
}
if (reader == nullptr) {
return Response::build_404(*this, request, bookName, "");
return Response::build_404(*this, request, bookName);
}
std::string content;
@@ -364,7 +369,7 @@ std::unique_ptr<Response> InternalServer::handle_meta(const RequestContext& requ
} else if (meta_name == "favicon") {
reader->getFavicon(content, mimeType);
} else {
return Response::build_404(*this, request, bookName, "");
return Response::build_404(*this, request, bookName);
}
auto response = ContentResponse::build(*this, content, mimeType);
@@ -393,7 +398,7 @@ std::unique_ptr<Response> InternalServer::handle_suggest(const RequestContext& r
term = request.get_argument("term");
reader = mp_library->getReaderById(bookId);
} catch (const std::out_of_range&) {
return Response::build_404(*this, request, bookName, "");
return Response::build_404(*this, request, bookName);
}
if (m_verbose.load()) {
@@ -409,15 +414,10 @@ std::unique_ptr<Response> InternalServer::handle_suggest(const RequestContext& r
reader->searchSuggestionsSmart(term, maxSuggestionCount, suggestions);
for(auto& suggestion:suggestions) {
MustacheData result;
result.set("label", suggestion.getTitle());
if (suggestion.hasSnippet()) {
result.set("label", suggestion.getSnippet());
}
result.set("value", suggestion.getTitle());
result.set("label", suggestion[0]);
result.set("value", suggestion[0]);
result.set("kind", "path");
result.set("path", suggestion.getPath());
result.set("path", suggestion[1]);
result.set("first", first);
first = false;
results.push_back(result);
@@ -457,7 +457,7 @@ std::unique_ptr<Response> InternalServer::handle_skin(const RequestContext& requ
response->set_cacheable();
return std::move(response);
} catch (const ResourceNotFound& e) {
return Response::build_404(*this, request, "", "");
return Response::build_404(*this, request, "");
}
}
@@ -577,18 +577,18 @@ std::unique_ptr<Response> InternalServer::handle_random(const RequestContext& re
bookId = mp_nameMapper->getIdForName(bookName);
reader = mp_library->getReaderById(bookId);
} catch (const std::out_of_range&) {
return Response::build_404(*this, request, bookName, "");
return Response::build_404(*this, request, bookName);
}
if (reader == nullptr) {
return Response::build_404(*this, request, bookName, "");
return Response::build_404(*this, request, bookName);
}
try {
auto entry = reader->getRandomPage();
return build_redirect(bookName, entry.getFinalEntry());
} catch(kiwix::NoEntry& e) {
return Response::build_404(*this, request, bookName, "");
return Response::build_404(*this, request, bookName);
}
}
@@ -600,7 +600,7 @@ std::unique_ptr<Response> InternalServer::handle_captured_external(const Request
} catch (const std::out_of_range& e) {}
if (source.empty())
return Response::build_404(*this, request, "", "");
return Response::build_404(*this, request, "");
auto data = get_default_data();
data.set("source", source);
@@ -619,15 +619,11 @@ std::unique_ptr<Response> InternalServer::handle_catalog(const RequestContext& r
host = request.get_header("Host");
url = request.get_url_part(1);
} catch (const std::out_of_range&) {
return Response::build_404(*this, request, "", "");
}
if (url == "v2") {
return handle_catalog_v2(request);
return Response::build_404(*this, request, "");
}
if (url != "searchdescription.xml" && url != "root.xml" && url != "search") {
return Response::build_404(*this, request, "", "");
return Response::build_404(*this, request, "");
}
if (url == "searchdescription.xml") {
@@ -636,11 +632,13 @@ std::unique_ptr<Response> InternalServer::handle_catalog(const RequestContext& r
}
zim::Uuid uuid;
kiwix::OPDSDumper opdsDumper(mp_library);
kiwix::OPDSDumper opdsDumper;
opdsDumper.setRootLocation(m_root);
opdsDumper.setLibraryId(m_library_id);
opdsDumper.setSearchDescriptionUrl("catalog/searchdescription.xml");
opdsDumper.setLibrary(mp_library);
std::vector<std::string> bookIdsToDump;
if (url == "root.xml") {
opdsDumper.setTitle("All zims");
uuid = zim::Uuid::generate(host);
bookIdsToDump = mp_library->filter(kiwix::Filter().valid(true).local(true).remote(true));
} else if (url == "search") {
@@ -648,24 +646,28 @@ std::unique_ptr<Response> InternalServer::handle_catalog(const RequestContext& r
uuid = zim::Uuid::generate();
}
opdsDumper.setId(kiwix::to_string(uuid));
auto response = ContentResponse::build(
*this,
opdsDumper.dumpOPDSFeed(bookIdsToDump, request.get_query()),
opdsDumper.dumpOPDSFeed(bookIdsToDump),
"application/atom+xml; profile=opds-catalog; kind=acquisition; charset=utf-8");
return std::move(response);
}
namespace
std::vector<std::string>
InternalServer::search_catalog(const RequestContext& request,
kiwix::OPDSDumper& opdsDumper)
{
Filter get_search_filter(const RequestContext& request)
{
auto filter = kiwix::Filter().valid(true).local(true);
auto filter = kiwix::Filter().valid(true).local(true).remote(true);
string query("<Empty query>");
size_t count(10);
size_t startIndex(0);
try {
filter.query(request.get_argument("q"));
query = request.get_argument("q");
filter.query(query);
} catch (const std::out_of_range&) {}
try {
filter.maxSize(request.get_argument<unsigned long>("maxsize"));
filter.maxSize(extractFromString<unsigned long>(request.get_argument("maxsize")));
} catch (...) {}
try {
filter.name(request.get_argument("name"));
@@ -676,37 +678,26 @@ Filter get_search_filter(const RequestContext& request)
try {
filter.lang(request.get_argument("lang"));
} catch (const std::out_of_range&) {}
try {
count = extractFromString<unsigned long>(request.get_argument("count"));
} catch (...) {}
try {
startIndex = extractFromString<unsigned long>(request.get_argument("start"));
} catch (...) {}
try {
filter.acceptTags(kiwix::split(request.get_argument("tag"), ";"));
} catch (...) {}
try {
filter.rejectTags(kiwix::split(request.get_argument("notag"), ";"));
} catch (...) {}
return filter;
}
template<class T>
std::vector<T> subrange(const std::vector<T>& v, size_t s, size_t n)
{
const size_t e = std::min(v.size(), s+n);
return std::vector<T>(v.begin()+std::min(v.size(), s), v.begin()+e);
}
} // unnamed namespace
std::vector<std::string>
InternalServer::search_catalog(const RequestContext& request,
kiwix::OPDSDumper& opdsDumper)
{
const auto filter = get_search_filter(request);
const std::string q = filter.hasQuery()
? filter.getQuery()
: "<Empty query>";
opdsDumper.setTitle("Search result for " + query);
std::vector<std::string> bookIdsToDump = mp_library->filter(filter);
const auto totalResults = bookIdsToDump.size();
const size_t count = request.get_optional_param("count", 10UL);
const size_t startIndex = request.get_optional_param("start", 0UL);
bookIdsToDump = subrange(bookIdsToDump, startIndex, count);
const auto s = std::min(startIndex, totalResults);
bookIdsToDump.erase(bookIdsToDump.begin(), bookIdsToDump.begin()+s);
if (count>0 && bookIdsToDump.size() > count) {
bookIdsToDump.resize(count);
}
opdsDumper.setOpenSearchInfo(totalResults, startIndex, bookIdsToDump.size());
return bookIdsToDump;
}
@@ -770,7 +761,7 @@ std::unique_ptr<Response> InternalServer::handle_content(const RequestContext& r
std::string searchURL = m_root+"/search?pattern="+pattern; // Make a full search on the entire library.
const std::string details = searchSuggestionHTML(searchURL, kiwix::urlDecode(pattern));
return Response::build_404(*this, request, bookName, "", details);
return Response::build_404(*this, request, bookName, details);
}
auto urlStr = request.get_url().substr(bookName.size()+1);
@@ -803,7 +794,7 @@ std::unique_ptr<Response> InternalServer::handle_content(const RequestContext& r
std::string searchURL = m_root+"/search?content="+bookName+"&pattern="+pattern; // Make a search on this specific book only.
const std::string details = searchSuggestionHTML(searchURL, kiwix::urlDecode(pattern));
return Response::build_404(*this, request, bookName, reader->getTitle(), details);
return Response::build_404(*this, request, bookName, details);
}
}

View File

@@ -73,10 +73,6 @@ class InternalServer {
std::unique_ptr<Response> build_homepage(const RequestContext& request);
std::unique_ptr<Response> handle_skin(const RequestContext& request);
std::unique_ptr<Response> handle_catalog(const RequestContext& request);
std::unique_ptr<Response> handle_catalog_v2(const RequestContext& request);
std::unique_ptr<Response> handle_catalog_v2_root(const RequestContext& request);
std::unique_ptr<Response> handle_catalog_v2_entries(const RequestContext& request);
std::unique_ptr<Response> handle_catalog_v2_categories(const RequestContext& request);
std::unique_ptr<Response> handle_meta(const RequestContext& request);
std::unique_ptr<Response> handle_search(const RequestContext& request);
std::unique_ptr<Response> handle_suggest(const RequestContext& request);
@@ -88,6 +84,7 @@ class InternalServer {
kiwix::OPDSDumper& opdsDumper);
MustacheData get_default_data() const;
MustacheData homepage_data() const;
std::shared_ptr<Reader> get_reader(const std::string& bookName) const;
bool etag_not_needed(const RequestContext& r) const;
@@ -108,7 +105,6 @@ class InternalServer {
NameMapper* mp_nameMapper;
std::string m_server_id;
std::string m_library_id;
friend std::unique_ptr<Response> Response::build(const InternalServer& server);
friend std::unique_ptr<ContentResponse> ContentResponse::build(const InternalServer& server, const std::string& content, const std::string& mimetype, bool isHomePage);

View File

@@ -1,109 +0,0 @@
/*
* Copyright 2021 Veloman Yunkan <veloman.yunkan@gmail.com>
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of the GNU General Public License as published by
* the Free Software Foundation; either version 3 of the License, or
* any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
* MA 02110-1301, USA.
*/
#include "internalServer.h"
#include "library.h"
#include "opds_dumper.h"
#include "request_context.h"
#include "response.h"
#include "tools/otherTools.h"
#include "kiwixlib-resources.h"
#include <mustache.hpp>
#include <string>
#include <vector>
namespace kiwix {
std::unique_ptr<Response> InternalServer::handle_catalog_v2(const RequestContext& request)
{
if (m_verbose.load()) {
printf("** running handle_catalog_v2");
}
std::string url;
try {
url = request.get_url_part(2);
} catch (const std::out_of_range&) {
return Response::build_404(*this, request, "", "");
}
if (url == "root.xml") {
return handle_catalog_v2_root(request);
} else if (url == "searchdescription.xml") {
const std::string endpoint_root = m_root + "/catalog/v2";
return ContentResponse::build(*this,
RESOURCE::catalog_v2_searchdescription_xml,
kainjow::mustache::object({{"endpoint_root", endpoint_root}}),
"application/opensearchdescription+xml"
);
} else if (url == "entries") {
return handle_catalog_v2_entries(request);
} else if (url == "categories") {
return handle_catalog_v2_categories(request);
} else {
return Response::build_404(*this, request, "", "");
}
}
std::unique_ptr<Response> InternalServer::handle_catalog_v2_root(const RequestContext& request)
{
return ContentResponse::build(
*this,
RESOURCE::templates::catalog_v2_root_xml,
kainjow::mustache::object{
{"date", gen_date_str()},
{"endpoint_root", m_root + "/catalog/v2"},
{"feed_id", gen_uuid(m_library_id)},
{"all_entries_feed_id", gen_uuid(m_library_id + "/entries")},
{"category_list_feed_id", gen_uuid(m_library_id + "/categories")}
},
"application/atom+xml;profile=opds-catalog;kind=navigation"
);
}
std::unique_ptr<Response> InternalServer::handle_catalog_v2_entries(const RequestContext& request)
{
OPDSDumper opdsDumper(mp_library);
opdsDumper.setRootLocation(m_root);
opdsDumper.setLibraryId(m_library_id);
const auto bookIds = search_catalog(request, opdsDumper);
const auto opdsFeed = opdsDumper.dumpOPDSFeedV2(bookIds, request.get_query());
return ContentResponse::build(
*this,
opdsFeed,
"application/atom+xml;profile=opds-catalog;kind=acquisition"
);
}
std::unique_ptr<Response> InternalServer::handle_catalog_v2_categories(const RequestContext& request)
{
OPDSDumper opdsDumper(mp_library);
opdsDumper.setRootLocation(m_root);
opdsDumper.setLibraryId(m_library_id);
return ContentResponse::build(
*this,
opdsDumper.categoriesOPDSFeed(mp_library->getBooksCategories()),
"application/atom+xml;profile=opds-catalog;kind=navigation"
);
}
} // namespace kiwix

View File

@@ -183,14 +183,4 @@ std::string RequestContext::get_header(const std::string& name) const {
return headers.at(lcAll(name));
}
std::string RequestContext::get_query() const {
std::string q;
const char* sep = "";
for ( const auto& a : arguments ) {
q += sep + a.first + '=' + a.second;
sep = "&";
}
return q;
}
}

View File

@@ -74,21 +74,11 @@ class RequestContext {
return v;
}
template<class T>
T get_optional_param(const std::string& name, T default_value) const
{
try {
return get_argument<T>(name);
} catch (...) {}
return default_value;
}
RequestMethod get_method() const;
std::string get_url() const;
std::string get_url_part(int part) const;
std::string get_full_url() const;
std::string get_query() const;
ByteRange get_range() const;

View File

@@ -24,7 +24,6 @@
#include "tools/regexTools.h"
#include "tools/stringTools.h"
#include "tools/otherTools.h"
#include "string.h"
#include <mustache.hpp>
@@ -39,6 +38,17 @@ namespace
{
// some utilities
std::string render_template(const std::string& template_str, kainjow::mustache::data data)
{
kainjow::mustache::mustache tmpl(template_str);
kainjow::mustache::data urlencode{kainjow::mustache::lambda2{
[](const std::string& str,const kainjow::mustache::renderer& r) { return urlEncode(r(str), true); }}};
data.set("urlencoded", urlencode);
std::stringstream ss;
tmpl.render(data, [&ss](const std::string& str) { ss << str; });
return ss.str();
}
std::string get_mime_type(const zim::Item& item)
{
try {
@@ -83,7 +93,7 @@ std::unique_ptr<Response> Response::build_304(const InternalServer& server, cons
return response;
}
std::unique_ptr<Response> Response::build_404(const InternalServer& server, const RequestContext& request, const std::string& bookName, const std::string& bookTitle, const std::string& details)
std::unique_ptr<Response> Response::build_404(const InternalServer& server, const RequestContext& request, const std::string& bookName, const std::string& details)
{
MustacheData results;
results.set("url", request.get_full_url());
@@ -91,7 +101,7 @@ std::unique_ptr<Response> Response::build_404(const InternalServer& server, cons
auto response = ContentResponse::build(server, RESOURCE::templates::_404_html, results, "text/html");
response->set_code(MHD_HTTP_NOT_FOUND);
response->set_taskbar(bookName, bookTitle);
response->set_taskbar(bookName, "");
return std::move(response);
}
@@ -190,7 +200,7 @@ void ContentResponse::introduce_taskbar()
kainjow::mustache::data data;
data.set("root", m_root);
data.set("content", m_bookName);
data.set("hascontent", (!m_bookName.empty() && !m_bookTitle.empty()));
data.set("hascontent", !m_bookName.empty());
data.set("title", m_bookTitle);
data.set("withlibrarybutton", m_withLibraryButton);
auto head_content = render_template(RESOURCE::templates::head_taskbar_html, data);
@@ -250,9 +260,9 @@ Response::create_mhd_response(const RequestContext& request)
MHD_Response*
ContentResponse::create_mhd_response(const RequestContext& request)
{
if (contentDecorationAllowed()) {
inject_root_link();
inject_root_link();
if (contentDecorationAllowed()) {
if (m_withTaskbar) {
introduce_taskbar();
}

View File

@@ -47,7 +47,7 @@ class Response {
static std::unique_ptr<Response> build(const InternalServer& server);
static std::unique_ptr<Response> build_304(const InternalServer& server, const ETag& etag);
static std::unique_ptr<Response> build_404(const InternalServer& server, const RequestContext& request, const std::string& bookName, const std::string& bookTitle, const std::string& details="");
static std::unique_ptr<Response> build_404(const InternalServer& server, const RequestContext& request, const std::string& bookName, const std::string& details="");
static std::unique_ptr<Response> build_416(const InternalServer& server, size_t resourceLength);
static std::unique_ptr<Response> build_500(const InternalServer& server, const std::string& msg);
static std::unique_ptr<Response> build_redirect(const InternalServer& server, const std::string& redirectUrl);

View File

@@ -19,7 +19,6 @@
#include "tools/otherTools.h"
#include <algorithm>
#include <iomanip>
#ifdef _WIN32
#include <windows.h>
@@ -33,8 +32,6 @@
#include <sstream>
#include <pugixml.hpp>
#include <zim/uuid.h>
static std::map<std::string, std::string> codeisomapping {
{ "aa", "aar" },
@@ -344,35 +341,3 @@ kiwix::MimeCounterType kiwix::parseMimetypeCounter(const std::string& counterDat
return counters;
}
std::string kiwix::gen_date_str()
{
auto now = std::time(0);
auto tm = std::localtime(&now);
std::stringstream is;
is << std::setw(2) << std::setfill('0')
<< 1900+tm->tm_year << "-"
<< std::setw(2) << std::setfill('0') << tm->tm_mon+1 << "-"
<< std::setw(2) << std::setfill('0') << tm->tm_mday << "T"
<< std::setw(2) << std::setfill('0') << tm->tm_hour << ":"
<< std::setw(2) << std::setfill('0') << tm->tm_min << ":"
<< std::setw(2) << std::setfill('0') << tm->tm_sec << "Z";
return is.str();
}
std::string kiwix::gen_uuid(const std::string& s)
{
return kiwix::to_string(zim::Uuid::generate(s));
}
std::string kiwix::render_template(const std::string& template_str, kainjow::mustache::data data)
{
kainjow::mustache::mustache tmpl(template_str);
kainjow::mustache::data urlencode{kainjow::mustache::lambda2{
[](const std::string& str,const kainjow::mustache::renderer& r) { return urlEncode(r(str), true); }}};
data.set("urlencoded", urlencode);
std::stringstream ss;
tmpl.render(data, [&ss](const std::string& str) { ss << str; });
return ss.str();
}

View File

@@ -339,7 +339,7 @@ std::string makeTmpDirectory()
_wmkdir(ctmp);
return WideToUtf8(ctmp);
#else
char _template_array[] = {"/tmp/libkiwix_XXXXXX"};
char _template_array[] = {"/tmp/kiwix-lib_XXXXXX"};
std::string dir = mkdtemp(_template_array);
return dir;
#endif

View File

@@ -1,10 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
<ShortName>Zim catalog search</ShortName>
<Description>Search zim files in the catalog.</Description>
<Url type="application/atom+xml;profile=opds-catalog;kind=acquisition"
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:k="http://kiwix.org/opensearchextension/1.0"
indexOffset="0"
template="{{endpoint_root}}/entries?q={searchTerms?}&lang={language?}&name={k:name?}&tag={k:tag?}&maxsize={k:maxsize?}&count={count?}&start={startIndex?}"/>
</OpenSearchDescription>

View File

@@ -6,5 +6,5 @@
xmlns:atom="http://www.w3.org/2005/Atom"
xmlns:k="http://kiwix.org/opensearchextension/1.0"
indexOffset="0"
template="{{root}}/catalog/search?q={searchTerms?}&lang={language?}&name={k:name?}&tag={k:tag?}&notag={k:notag?}&maxsize={k:maxsize?}&count={count?}&start={startIndex?}"/>
template="/{{root}}/catalog/search?q={searchTerms?}&lang={language?}&name={k:name?}&tag={k:tag?}&notag={k:notag?}&maxsize={k:maxsize?}&count={count?}&start={startIndex?}"/>
</OpenSearchDescription>

View File

@@ -19,10 +19,6 @@ skin/jquery-ui/jquery-ui.theme.min.css
skin/jquery-ui/jquery-ui.min.css
skin/caret.png
skin/taskbar.js
skin/langList.js
skin/categoryList.js
skin/iso6391To3.js
skin/isotope.pkgd.min.js
skin/index.js
skin/taskbar.css
skin/block_external.js
@@ -36,9 +32,4 @@ templates/head_taskbar.html
templates/taskbar_part.html
templates/external_blocker_part.html
templates/captured_external.html
templates/catalog_entries.xml
templates/catalog_v2_root.xml
templates/catalog_v2_entries.xml
templates/catalog_v2_categories.xml
opensearchdescription.xml
catalog_v2_searchdescription.xml

View File

@@ -1,19 +0,0 @@
// eslint-disable-next-line no-unused-vars
const categoryList = {
"other": "Other",
"gutenberg": "Gutenberg",
"mooc": "Mooc",
"phet": "Phet",
"psiram": "Psiram",
"stack_exchange": "Stack Exchange",
"ted": "Ted",
"vikidia": "Vikidia",
"wikibooks": "Wikibooks",
"wikinews": "Wikinews",
"wikipedia": "Wikipedia",
"wikiquote": "Wikiquote",
"wikisource": "Wikisource",
"wikiversity": "Wikiversity",
"wikivoyage": "Wikivoyage",
"wiktionary": "Wiktionary"
}

View File

@@ -1,250 +1,95 @@
(function() {
const root = $(`link[type='root']`).attr('href');
const incrementalLoadingParams = {
start: 0,
count: viewPortToCount()
};
const filterTypes = ['lang', 'category', 'q'];
const bookOrderMap = new Map();
const filterCookieName = 'filters';
const oneDayDelta = 86400000;
let footer;
let fadeOutDiv;
let iso;
let isFetching = false;
let noResultInjected = false;
let filters = getCookie(filterCookieName);
let params = new URLSearchParams(window.location.search || filters || '');
let timer;
const root = $( `link[type='root']` ).attr("href");
let viewPortHeight = window.innerHeight;
let isFetching = false;
let isEnd = false;
const params = new URLSearchParams(window.location.search);
const basicConfig = {
'start': 0,
'count': Math.floor(viewPortHeight/100 + 1) * 3
};
function queryUrlBuilder() {
let url = `${root}/catalog/search?`;
url += Object.keys(incrementalLoadingParams).map(key => `${key}=${incrementalLoadingParams[key]}`).join("&");
params.forEach((value, key) => {url+= value ? `&${key}=${value}` : ''});
return (url);
}
function setCookie(cookieName, cookieValue) {
const date = new Date();
date.setTime(date.getTime() + oneDayDelta);
document.cookie = `${cookieName}=${cookieValue};expires=${date.toUTCString()};sameSite=Strict`;
}
function getCookie(cookieName) {
const name = cookieName + "=";
let result;
decodeURIComponent(document.cookie).split('; ').forEach(val => {
if (val.indexOf(name) === 0) {
result = val.substring(name.length);
}
});
return result;
}
function htmlEncode(str) {
return str.replace(/[\u00A0-\u9999<>\&]/gim, (i) => `&#${i.charCodeAt(0)};`);
}
function viewPortToCount(){
return Math.floor(window.innerHeight/100 + 1)*(window.innerWidth>1000 ? 3 : 2);
}
function getInnerHtml(node, query) {
return node.querySelector(query).innerHTML;
}
function generateBookHtml(book, sort = false) {
const link = book.querySelector('link').getAttribute('href');
const title = getInnerHtml(book, 'title');
const description = getInnerHtml(book, 'summary');
const id = getInnerHtml(book, 'id');
const iconUrl = getInnerHtml(book, 'icon');
const articleCount = getInnerHtml(book, 'articleCount');
const mediaCount = getInnerHtml(book, 'mediaCount');
const linkTag = document.createElement('a');
linkTag.setAttribute('class', 'book');
linkTag.setAttribute('data-id', id);
linkTag.setAttribute('href', link);
if (sort) {
linkTag.setAttribute('data-idx', bookOrderMap.get(id));
}
linkTag.innerHTML = `<div class='book__background' style="background-image: url('${iconUrl}');">
<div class='book__title' title='${title}'>${title}</div>
<div class='book__description' title='${description}'>${description}</div>
<div class='book__info'>${articleCount} articles, ${mediaCount} medias</div>
</div>`;
return linkTag;
}
function toggleFooter(show=false) {
if (show) {
footer.style.display = 'block';
} else {
footer.style.display = 'none';
fadeOutDiv.style.display = 'block';
}
}
async function loadBooks() {
const loader = document.querySelector('.loader');
loader.style.display = 'block';
return await fetch(queryUrlBuilder()).then(async (resp) => {
const data = new window.DOMParser().parseFromString(await resp.text(), 'application/xml');
const books = data.querySelectorAll('entry');
books.forEach((book, idx) => {
bookOrderMap.set(getInnerHtml(book, 'id'), idx);
});
incrementalLoadingParams.start += books.length;
if (parseInt(data.querySelector('totalResults').innerHTML) === bookOrderMap.size) {
incrementalLoadingParams.count = 0;
toggleFooter(true);
} else {
toggleFooter();
}
loader.style.display = 'none';
return books;
});
}
async function loadAndDisplayOptions(nodeQuery, query) {
// currently taking an object in place of query, will replace it with query while fetching data from backend later on.
document.querySelector(nodeQuery).innerHTML += Object.keys(query)
.map((option) => {return `<option value='${option}'>${htmlEncode(query[option])}</option>`})
.join('');
}
function checkAndInjectEmptyMessage() {
if (!bookOrderMap.size) {
if (!noResultInjected) {
noResultInjected = true;
iso.remove(document.getElementsByClassName('book__list')[0].getElementsByTagName('a'));
iso.layout();
const spanTag = document.createElement('span');
spanTag.setAttribute('class', 'noResults');
spanTag.innerHTML = `No result. Would you like to <a href="/?lang=">reset filter?</a>`;
document.querySelector('body').append(spanTag);
spanTag.getElementsByTagName('a')[0].onclick = (event) => {
event.preventDefault();
window.history.pushState({}, null, `${window.location.href.split('?')[0]}?lang=`);
setCookie(filterCookieName, 'lang=');
resetAndFilter();
filterTypes.forEach(key => {document.getElementsByName(key)[0].value = params.get(key) || ''});
};
}
return true;
} else if (noResultInjected) {
noResultInjected = false;
document.getElementsByClassName('noResults')[0].remove();
}
return false;
}
async function loadAndDisplayBooks(sort = false) {
if (isFetching) return;
isFetching = true;
await loadAndDisplayBooksUnguarded(sort);
isFetching = false;
}
async function loadAndDisplayBooksUnguarded(sort) {
let books = await loadBooks();
if (checkAndInjectEmptyMessage()) {return}
const booksToFilter = new Set();
const booksToDelete = new Set();
iso.arrange({
filter: function (idx, elem) {
const id = elem.getAttribute('data-id');
const retVal = bookOrderMap.has(id);
if (retVal) {
booksToFilter.add(id);
if (sort) {
elem.setAttribute('data-idx', bookOrderMap.get(id));
iso.updateSortData(elem);
}
} else {
booksToDelete.add(elem);
}
return retVal;
}
});
books = [...books].filter((book) => {return !booksToFilter.has(getInnerHtml(book, 'id'))});
booksToDelete.forEach(book => {iso.remove(book);});
books.forEach((book) => {iso.insert(generateBookHtml(book, sort))});
}
async function resetAndFilter(filterType = '', filterValue = '') {
isFetching = false;
incrementalLoadingParams.start = 0;
incrementalLoadingParams.count = viewPortToCount();
fadeOutDiv.style.display = 'none';
bookOrderMap.clear();
params = new URLSearchParams(window.location.search);
if (filterType) {
params.set(filterType, filterValue);
window.history.pushState({}, null, `${window.location.href.split('?')[0]}?${params.toString()}`);
setCookie(filterCookieName, params.toString());
}
await loadAndDisplayBooks(true);
}
window.addEventListener('popstate', async () => {
await resetAndFilter();
filterTypes.forEach(key => {document.getElementsByName(key)[0].value = params.get(key) || ''});
function queryUrlBuilder() {
let url = `${root}/catalog/search?`
Object.keys(basicConfig).forEach((key, idx) => {
url += `${key}=${basicConfig[key]}${idx !== Object.keys(basicConfig).length - 1 ? '&' : ''}`;
});
return url + (params.toString() ? `&${params.toString()}` : '');
}
async function loadSubset() {
if (window.innerHeight + window.scrollY >= document.body.offsetHeight) {
if (incrementalLoadingParams.count) {
loadAndDisplayBooks();
}
else {
fadeOutDiv.style.display = 'none';
}
}
}
window.addEventListener('resize', (event) => {
if (timer) {clearTimeout(timer)}
timer = setTimeout(() => {
incrementalLoadingParams.count = incrementalLoadingParams.count && viewPortToCount();
loadSubset();
}, 100, event);
async function loadAndDisplayBooks(append = false) {
isFetching = true;
await fetch(queryUrlBuilder())
.then(async (resp) => {
const data = new window.DOMParser().parseFromString(await resp.text(), 'application/xml');
const books = data.querySelectorAll("entry");
let bookHtml = '';
books.forEach((book) => {
const link = book.querySelector('link').getAttribute('href');
const title = getInnerHtml(book, 'title');
const description = getInnerHtml(book, 'summary');
bookHtml += `<a href='${link}'><div class='book'>
<div class='book__background' style="background-image: url('${getInnerHtml(book, 'icon')}');">
<div class='book__title' title='${title}'>${title}</div>
<div class='book__description' title='${description}'>${description}</div>
<div class='book__info'>${getInnerHtml(book, 'articleCount')} articles, ${getInnerHtml(book, 'mediaCount')} medias</div>
</div>
</div></a>`;
});
document.querySelector('.book__list').innerHTML = (append ? document.querySelector('.book__list').innerHTML : '') + bookHtml;
isFetching = false;
isEnd = !books.length;
});
}
window.addEventListener('scroll', loadSubset);
async function loadAndDisplayOptions(nodeQuery, query) {
// currently taking an array in place of query, will replace it with query while fetching data from backend later on.
query.forEach((option) => {
let value = Object.keys(option)[0];
let label = option[value];
document.querySelector(nodeQuery).innerHTML += `<option value='${value}'>${label}</option>`;
})
}
window.onload = async () => {
iso = new Isotope( '.book__list', {
itemSelector: '.book',
getSortData:{
weight: function( itemElem ) {
const index = itemElem.getAttribute('data-idx');
return index ? parseInt(index) : Infinity;
}
},
sortBy: 'weight'
});
footer = document.getElementById('kiwixfooter');
fadeOutDiv = document.getElementById('fadeOut');
await loadAndDisplayBooks();
await loadAndDisplayOptions('#languageFilter', langList);
await loadAndDisplayOptions('#categoryFilter', categoryList);
filterTypes.forEach((filter) => {
const filterTag = document.getElementsByName(filter)[0];
filterTag.addEventListener('change', () => {resetAndFilter(filterTag.name, filterTag.value)});
});
if (filters) {
window.history.pushState({}, null, `${window.location.href.split('?')[0]}?${params.toString()}`);
}
params.forEach((value, key) => {document.getElementsByName(key)[0].value = value});
document.getElementById('kiwixSearchForm').onsubmit = (event) => {event.preventDefault()};
if (!window.location.search) {
const browserLang = navigator.language.split('-')[0];
const langFilter = document.getElementById('languageFilter');
langFilter.value = browserLang.length === 3 ? browserLang : iso6391To3[browserLang];
langFilter.dispatchEvent(new Event('change'));
}
setCookie(filterCookieName, params.toString());
async function filterBooks(filterType, filterValue) {
isEnd = false;
isFetching = false;
basicConfig['start'] = 0;
const params = new URLSearchParams(window.location.search);
if (!filterValue) {
params.delete(filterType);
} else {
params.set(filterType, filterValue);
}
})();
window.location.search = params.toString();
loadAndDisplayBooks();
}
function getInnerHtml(node, query) {
return node.querySelector(query).innerHTML;
}
window.addEventListener('resize', async () => {
if (isFetching || isEnd) return;
viewPortHeight = window.innerHeight;
basicConfig['count'] = Math.floor(viewPortHeight/100 + 1) * 3;
basicConfig['start'] = basicConfig['start'] + basicConfig['count'];
loadAndDisplayBooks(true);
});
window.addEventListener('scroll', async () => {
if (isFetching || isEnd) return;
if (viewPortHeight + window.scrollY >= document.body.offsetHeight) {
basicConfig['start'] = basicConfig['start'] + basicConfig['count'];
loadAndDisplayBooks(true);
}
});
window.onload = async (event) => {
loadAndDisplayBooks();
loadAndDisplayOptions('#languageFilter', [{'eng': 'English'}, {'fra': 'french'}, {'ara': 'arab'}, {'hin': 'Hindi'}]);
loadAndDisplayOptions('#categoryFilter', [{'stack_exchange': 'stack exchange'}, {'wikipedia': 'wikipedia'}, {'phet': 'phet'}, {'youtube': 'youtube'}]);
for (const key of params.keys()) {
document.getElementsByName(key)[0].value = params.get(key);
}
}

View File

@@ -1,124 +0,0 @@
// eslint-disable-next-line no-unused-vars
const iso6391To3 = {
"aa": "aar",
"af": "afr",
"ak": "aka",
"am": "amh",
"ar": "ara",
"as": "asm",
"az": "aze",
"ba": "bak",
"be": "bel",
"bg": "bul",
"bm": "bam",
"bn": "ben",
"bo": "bod",
"br": "bre",
"bs": "bos",
"ca": "cat",
"ce": "che",
"co": "cos",
"cs": "ces",
"cv": "chv",
"cy": "cym",
"da": "dan",
"de": "deu",
"dz": "dzo",
"ee": "ewe",
"en": "eng",
"es": "spa",
"et": "est",
"eu": "eus",
"fa": "fas",
"ff": "ful",
"fi": "fin",
"fo": "fao",
"fr": "fra",
"ga": "gle",
"gl": "glg",
"gn": "grn",
"gu": "guj",
"gv": "glv",
"ha": "hau",
"he": "heb",
"hi": "hin",
"hr": "hrv",
"hu": "hun",
"hy": "hye",
"id": "ind",
"ig": "ibo",
"is": "isl",
"it": "ita",
"iu": "iku",
"ja": "jpn",
"jv": "jav",
"ka": "kat",
"ki": "kik",
"kk": "kaz",
"km": "khm",
"kn": "kan",
"ko": "kor",
"ks": "kas",
"ku": "kur",
"kw": "cor",
"ky": "kir",
"lb": "ltz",
"lg": "lug",
"ln": "lin",
"lo": "lao",
"lt": "lit",
"lv": "lav",
"mg": "mlg",
"mi": "mri",
"mk": "mkd",
"ml": "mal",
"mn": "mon",
"mr": "mar",
"mt": "mlt",
"my": "mya",
"nl": "nld",
"ny": "nya",
"om": "orm",
"pl": "pol",
"pt": "por",
"qu": "que",
"rm": "roh",
"rn": "run",
"ro": "ron",
"ru": "rus",
"rw": "kin",
"sa": "san",
"sd": "snd",
"sg": "sag",
"si": "sin",
"sk": "slk",
"sl": "slv",
"sn": "sna",
"so": "som",
"sq": "sqi",
"sr": "srp",
"ss": "ssw",
"sv": "swe",
"ta": "tam",
"te": "tel",
"tg": "tgk",
"th": "tha",
"ti": "tir",
"tk": "tuk",
"tn": "tsn",
"tr": "tur",
"ts": "tso",
"tt": "tat",
"ug": "uig",
"uk": "ukr",
"ur": "urd",
"uz": "uzb",
"ve": "ven",
"vi": "vie",
"wa": "wln",
"wo": "wol",
"xh": "xho",
"yo": "yor",
"zh": "zho",
"zu": "zul"
}

View File

File diff suppressed because one or more lines are too long

View File

@@ -1,124 +0,0 @@
// eslint-disable-next-line no-unused-vars
const langList = {
"aar": "Afaraf",
"afr": "Afrikaans",
"aka": "Akan",
"amh": "አማርኛ",
"ara": "اللغة العربية",
"asm": "অসমীয়া",
"aze": "azərbaycan dili",
"bak": "башҡорт теле",
"bel": "беларуская мова",
"bul": "български език",
"bam": "bamanankan",
"ben": "বাংলা",
"bod": "བོད་ཡིག",
"bre": "brezhoneg",
"bos": "bosanski jezik",
"cat": "Català",
"che": "нохчийн мотт",
"cos": "corsu",
"ces": "čeština",
"chv": "чӑваш чӗлхи",
"cym": "Cymraeg",
"dan": "dansk",
"deu": "Deutsch",
"dzo": "རྫོང་ཁ",
"ewe": "Eʋegbe",
"eng": "English",
"spa": "Español",
"est": "eesti",
"eus": "euskara",
"fas": "فارسی",
"ful": "Fulfulde",
"fin": "suomi",
"fao": "føroyskt",
"fra": "Français",
"gle": "Gaeilge",
"glg": "galego",
"grn": "Avañe'ẽ",
"guj": "ગુજરાતી",
"glv": "Gaelg",
"hau": "هَوُسَ",
"heb": "עברית",
"hin": "हिन्दी",
"hrv": "hrvatski jezik",
"hun": "magyar",
"hye": "Հայերեն",
"ind": "Bahasa Indonesia",
"ibo": "Asụsụ Igbo",
"isl": "Íslenska",
"ita": "Italiano",
"iku": "ᐃᓄᒃᑎᑐᑦ",
"jpn": "日本語",
"jav": "basa Jawa",
"kat": "ქართული",
"kik": "Gĩkũyũ",
"kaz": "қазақ тілі",
"khm": "ខេមរភាសា",
"kan": "ಕನ್ನಡ",
"kor": "한국어",
"kas": "कश्मीरी",
"kur": "Kurdî",
"cor": "Kernewek",
"kir": "Кыргызча",
"ltz": "Lëtzebuergesch",
"lug": "Luganda",
"lin": "Lingála",
"lao": "ພາສາ",
"lit": "lietuvių kalba",
"lav": "latviešu valoda",
"mlg": "fiteny malagasy",
"mri": "te reo Māori",
"mkd": "македонски јазик",
"mal": "മലയാളം",
"mon": "Монгол хэл",
"mar": "मराठी",
"mlt": "Malti",
"mya": "ဗမာစာ",
"nld": "Nederlands",
"nya": "chiCheŵa",
"orm": "Afaan Oromoo",
"pol": "język polski",
"por": "Português",
"que": "Runa Simi",
"roh": "rumantsch grischun",
"run": "Ikirundi",
"ron": "Română",
"rus": "Русский",
"kin": "Ikinyarwanda",
"san": "संस्कृतम्",
"snd": "सिन्धी",
"sag": "yângâ tî sängö",
"sin": "සිංහල",
"slk": "slovenčina",
"slv": "slovenski jezik",
"sna": "chiShona",
"som": "Soomaaliga",
"sqi": "Shqip",
"srp": "српски језик",
"ssw": "SiSwati",
"swe": "svenska",
"tam": "தமிழ்",
"tel": "తెలుగు",
"tgk": "тоҷикӣ",
"tha": "ไทย",
"tir": "ትግርኛ",
"tuk": "Türkmen",
"tsn": "Setswana",
"tur": "Türkçe",
"tso": "Xitsonga",
"tat": "татар теле",
"uig": "ئۇيغۇرچە‎",
"ukr": "Українська",
"urd": "اردو",
"uzb": "Ўзбек",
"ven": "Tshivenḓa",
"vie": "Tiếng Việt",
"wln": "walon",
"wol": "Wollof",
"xho": "isiXhosa",
"yor": "Yorùbá",
"zho": "中文",
"zul": "isiZulu"
}

View File

@@ -7,10 +7,7 @@ const jq = jQuery.noConflict(true);
jq(document).ready(() => {
(function ($) {
const root = $( `link[type='root']` ).attr("href");
const bookName = (window.location.pathname == `${root}/search`)
? (new URLSearchParams(window.location.search)).get('content')
: window.location.pathname.split(`${root}/`)[1].split('/')[0];
const bookName = window.location.pathname.split(`${root}/`)[1].split('/')[0];
$( "#kiwixsearchbox" ).autocomplete({
@@ -34,12 +31,7 @@ jq(document).ready(() => {
$( "#kiwixsearchform" ).submit();
}
},
}).data( "ui-autocomplete" )._renderItem = function( ul, item ) {
return $( "<li>" )
.data( "ui-autocomplete-item", item )
.append( item.label )
.appendTo( ul );
};
});
/* cybook hack */
if (navigator.userAgent.indexOf("bookeen/cybook") != -1) {
@@ -93,4 +85,4 @@ jq(document).ready(() => {
}
});
})(jq);
})
})

View File

@@ -1,38 +0,0 @@
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:opds="http://opds-spec.org/2010/catalog">
<id>{{feed_id}}</id>
<title>{{^filter}}All zims{{/filter}}{{#filter}}Filtered zims ({{filter}}){{/filter}}</title>
<updated>{{date}}</updated>
{{#filter}}
<totalResults>{{totalResults}}</totalResults>
<startIndex>{{startIndex}}</startIndex>
<itemsPerPage>{{itemsPerPage}}</itemsPerPage>
{{/filter}}
<link rel="self" href="" type="application/atom+xml" />
<link rel="search" type="application/opensearchdescription+xml" href="{{root}}/catalog/searchdescription.xml" />
{{#books}}
<entry>
<id>{{id}}</id>
<title>{{title}}</title>
<summary>{{description}}</summary>
<language>{{language}}</language>
<updated>{{updated}}</updated>
<name>{{name}}</name>
<flavour>{{flavour}}</flavour>
<category>{{category}}</category>
<tags>{{tags}}</tags>
<articleCount>{{article_count}}</articleCount>
<mediaCount>{{media_count}}</mediaCount>
<icon>/meta?name=favicon&amp;content={{{content_id}}}</icon>
<link type="text/html" href="/{{{content_id}}}" />
<author>
<name>{{author_name}}</name>
</author>
<publisher>
<name>{{publisher_name}}</name>
</publisher>
{{#url}}
<link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="{{{url}}}" length="{{{size}}}" />
{{/url}}
</entry>
{{/books}}
</feed>

View File

@@ -1,25 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:opds="https://specs.opds.io/opds-1.2">
<id>{{feed_id}}</id>
<link rel="self"
href="{{endpoint_root}}/categories"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<link rel="start"
href="{{endpoint_root}}/root.xml"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<title>List of categories</title>
<updated>{{date}}</updated>
{{#categories}}
<entry>
<title>{{name}}</title>
<link rel="subsection"
href="{{endpoint_root}}/entries?category={{{urlencoded_name}}}"
type="application/atom+xml;profile=opds-catalog;kind=acquisition"/>
<updated>{{updated}}</updated>
<id>{{id}}</id>
<content type="text">All entries with category of '{{name}}'.</content>
</entry>
{{/categories}}
</feed>

View File

@@ -1,50 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:opds="https://specs.opds.io/opds-1.2"
xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">
<id>{{feed_id}}</id>
<link rel="self"
href="{{endpoint_root}}/entries{{{query}}}"
type="application/atom+xml;profile=opds-catalog;kind=acquisition"/>
<link rel="start"
href="{{endpoint_root}}/root.xml"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<link rel="up"
href="{{endpoint_root}}/root.xml"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<title>{{^filter}}All Entries{{/filter}}{{#filter}}Filtered Entries ({{filter}}){{/filter}}</title>
<updated>{{date}}</updated>
{{#filter}}
<totalResults>{{totalResults}}</totalResults>
<startIndex>{{startIndex}}</startIndex>
<itemsPerPage>{{itemsPerPage}}</itemsPerPage>
{{/filter}}
{{#books}}
<entry>
<id>{{id}}</id>
<title>{{title}}</title>
<summary>{{description}}</summary>
<language>{{language}}</language>
<updated>{{updated}}</updated>
<name>{{name}}</name>
<flavour>{{flavour}}</flavour>
<category>{{category}}</category>
<tags>{{tags}}</tags>
<articleCount>{{article_count}}</articleCount>
<mediaCount>{{media_count}}</mediaCount>
<icon>/meta?name=favicon&amp;content={{{content_id}}}</icon>
<link type="text/html" href="/{{{content_id}}}" />
<author>
<name>{{author_name}}</name>
</author>
<publisher>
<name>{{publisher_name}}</name>
</publisher>
{{#url}}
<link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="{{{url}}}" length="{{{size}}}" />
{{/url}}
</entry>
{{/books}}
</feed>

View File

@@ -1,35 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:opds="https://specs.opds.io/opds-1.2">
<id>{{feed_id}}</id>
<link rel="self"
href="{{endpoint_root}}/root.xml"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<link rel="start"
href="{{endpoint_root}}/root.xml"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<link rel="search"
href="{{endpoint_root}}/searchdescription.xml"
type="application/opensearchdescription+xml"/>
<title>OPDS Catalog Root</title>
<updated>{{date}}</updated>
<entry>
<title>All entries</title>
<link rel="subsection"
href="{{endpoint_root}}/entries"
type="application/atom+xml;profile=opds-catalog;kind=acquisition"/>
<updated>{{date}}</updated>
<id>{{all_entries_feed_id}}</id>
<content type="text">All entries from this catalog.</content>
</entry>
<entry>
<title>List of categories</title>
<link rel="subsection"
href="{{endpoint_root}}/categories"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<updated>{{date}}</updated>
<id>{{category_list_feed_id}}</id>
<content type="text">List of all categories in this catalog.</content>
</entry>
</feed>

View File

@@ -1,206 +1,63 @@
<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta charset="UTF-8" />
<title>Welcome to Kiwix Server</title>
<script
type="text/javascript"
src="{{root}}/skin/jquery-ui/external/jquery/jquery.js"
></script>
<script
type="text/javascript"
src="{{root}}/skin/jquery-ui/jquery-ui.min.js"
></script>
<link
type="text/css"
href="{{root}}/skin/jquery-ui/jquery-ui.min.css"
rel="Stylesheet"
/>
<link
type="text/css"
href="{{root}}/skin/jquery-ui/jquery-ui.theme.min.css"
rel="Stylesheet"
/>
<style>
html {
min-height: 100%;
position: relative;
}
body {
background: radial-gradient(#eeeeee 15%, transparent 16%) 0 0,
radial-gradient(#eeeeee 15%, transparent 16%) 8px 8px,
radial-gradient(rgba(255, 255, 255, 0.1) 15%, transparent 20%) 0 1px,
radial-gradient(rgba(255, 255, 255, 0.1) 15%, transparent 20%) 8px 9px;
background-color: #e8e8e8;
background-size: 16px 16px;
margin-left: auto;
margin-right: auto;
max-width: 1100px;
min-height: 100%;
}
.book__list {
text-align: center;
}
.kiwixHomeBody {
position: relative;
text-align: center;
min-height: 100%;
margin: 0 0 15px;
}
.book {
display: inline-block;
vertical-align: bottom;
margin: 8px;
padding: 12px 15px;
width: 300px;
border: 1px solid #ccc;
border-radius: 8px;
text-align: left;
color: #000;
font-family: sans-serif;
font-size: 13px;
background-color: #f1f1f1;
box-shadow: 2px 2px 5px 0 #ccc;
}
#kiwixfooter {
text-align: center;
margin: 0.5em;
position: absolute;
bottom: 0;
left: 46%;
}
.kiwixHomeNavbar {
display: flex;
justify-content: center;
}
.kiwixFilter {
margin: 8px 10px;
}
.kiwixSearch, .searchButton {
margin: 0 13px 0 0;
}
.kiwixSearchForm {
margin: 8px 10px;
float: right;
}
@media (max-width: 1100px) {
.kiwixHomeBody {
padding: 0 125px;
}
}
.book:hover {
background-color: #f9f9f9;
box-shadow: none;
}
.book__background {
background-repeat: no-repeat;
background-size: 48px 48px;
background-position: top right;
}
.book__title {
padding: 0 55px 0 0;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
font-size: 18px;
color: #0645ad;
line-height: 1em;
}
.book__description {
padding: 5px 55px 5px 0;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
font-size: 15px;
line-height: 1em;
}
.book__info {
color: #777;
font-weight: bold;
font-size: 13px;
line-height: 1em;
}
a:link {
text-decoration: none;
}
a:visited {
text-decoration: none;
}
.noResults {
position: absolute;
top: 48%;
left: 42%;
}
.loader-spinner {
position: absolute;
top: -50%;
left: 50%;
border: 5px solid #f3f3f3;
border-radius: 50%;
border-top: 5px solid #3498db;
width: 40px;
height: 40px;
margin: auto;
-webkit-animation: spin 1s linear infinite; /* Safari */
animation: spin 1s linear infinite;
margin-top: 35px;
margin-bottom: -35px;
z-index: 2;
}
/* Safari */
@-webkit-keyframes spin {
0% { -webkit-transform: rotate(0deg); }
100% { -webkit-transform: rotate(360deg); }
}
@keyframes spin {
0% { transform: rotate(0deg); }
100% { transform: rotate(360deg); }
}
.loader {
position: relative;
height: 70px;
width: 100%;
}
.fadeOut {
position: fixed;
display: none;
bottom: 0;
left: 0;
z-index: 1;
background: linear-gradient(180deg, rgba(232, 232, 232, 0) 0%, rgb(232, 232, 232) 100%);
height: 80px;
width: 100%;
}
.spacer {
height: 20px;
background: transparent;
}
</style>
<script src="{{root}}/skin/isotope.pkgd.min.js" defer></script>
<script src="{{root}}/skin/categoryList.js"></script>
<script src="{{root}}/skin/langList.js"></script>
<script src="{{root}}/skin/iso6391To3.js"></script>
<script type="text/javascript" src="{{root}}/skin/index.js" defer></script>
</head>
<body class="kiwix">
<div class='kiwixHomeNavbar'>
<select name="lang" id="languageFilter" class='kiwixFilter'>
<option value="" selected>All languages</option>
</select>
<select name="category" id="categoryFilter" class='kiwixFilter'>
<option value="" selected>All categories</option>
</select>
<form id='kiwixSearchForm' class='kiwixSearchForm'>
<input type="text" name="q" id="searchFilter" class='kiwixSearch'>
<input type="submit" class="searchButton" value="Search"/>
</form>
</div>
<div class="kiwixHomeBody">
<div class="book__list"></div>
<div id="fadeOut" class="fadeOut"></div>
</div>
<div class="loader"><div class="loader-spinner"></div></div>
<div class="spacer"></div>
<div id="kiwixfooter">Powered by <a href="https://kiwix.org">Kiwix</a></div>
</body>
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width">
<title>Welcome to Kiwix Server</title>
<script type="text/javascript" src="{{root}}/skin/jquery-ui/external/jquery/jquery.js"></script>
<script type="text/javascript" src="{{root}}/skin/jquery-ui/jquery-ui.min.js"></script>
<link type="text/css" href="{{root}}/skin/jquery-ui/jquery-ui.min.css" rel="Stylesheet" />
<link type="text/css" href="{{root}}/skin/jquery-ui/jquery-ui.theme.min.css" rel="Stylesheet" />
<style>
body {
background:
radial-gradient(#EEEEEE 15%, transparent 16%) 0 0,
radial-gradient(#EEEEEE 15%, transparent 16%) 8px 8px,
radial-gradient(rgba(255,255,255,.1) 15%, transparent 20%) 0 1px,
radial-gradient(rgba(255,255,255,.1) 15%, transparent 20%) 8px 9px;
background-color:#E8E8E8;
background-size:16px 16px;
margin-left: auto;
margin-right: auto;
max-width: 1100px;
}
.book__list { text-align: center; }
.book {
display: inline-block; vertical-align: bottom; margin: 8px; padding: 12px 15px; width: 300px;
border: 1px solid #ccc; border-radius: 8px;
text-align: left; color: #000; font-family: sans-serif; font-size: 13px;
background-color:#F1F1F1;
box-shadow: 2px 2px 5px 0px #ccc;
}
.book:hover { background-color: #F9F9F9; box-shadow: none;}
.book__background { background-repeat: no-repeat; background-size: 48px 48px; background-position: top right; }
.book__title {
padding: 0 55px 0 0;overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
font-size: 18px; color: #0645ad; line-height: 1em;
}
.book__description {
padding: 5px 55px 5px 0px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
font-size: 15px; line-height: 1em;
}
.book__info { color: #777; font-weight: bold; font-size: 13px; line-height: 1em; }
</style>
<script type="text/javascript" src="{{root}}/skin/index.js" defer></script>
</head>
<body class="kiwix">
<div class='kiwixHomeNavbar'>
<select name="lang" id="languageFilter" onchange="filterBooks('lang', this.value)">
<option value="" disabled selected>Languages</option>
</select>
<select name="category" id="categoryFilter" onchange="filterBooks('category', this.value)">
<option value="" disabled selected>Categories</option>
</select>
<input type="text" name="q" id="searchFilter" onchange="filterBooks('q', this.value)">
</div>
<div class="kiwix">
<div class='book__list'></div>
</div>
<div id="kiwixfooter">
Powered by <a href="https://kiwix.org">Kiwix</a>
</div>
</body>
</html>

View File

@@ -8,18 +8,18 @@
<input autocomplete="off" class="ui-autocomplete-input" id="kiwixsearchbox" name="pattern" type="text">
</form>
</div>
{{#hascontent}}
<input type="checkbox" id="kiwix_button_show_toggle">
<label for="kiwix_button_show_toggle"><img src="{{root}}/skin/caret.png" alt=""></label>
<div class="kiwix_button_cont">
{{#withlibrarybutton}}
<a id="kiwix_serve_taskbar_library_button" href="{{root}}/"><button>&#x1f3e0;</button></a>
{{/withlibrarybutton}}
{{#hascontent}}
<a id="kiwix_serve_taskbar_home_button" href="{{root}}/{{content}}/"><button>{{title}}</button></a>
<a id="kiwix_serve_taskbar_random_button"
href="{{root}}/random?content={{#urlencoded}}{{{content}}}{{/urlencoded}}"><button>&#x1F3B2;</button></a>
{{/hascontent}}
</div>
{{/hascontent}}
</div>
</span>
</span>

View File

@@ -2,7 +2,7 @@
<book
id="raycharles"
path="./zimfile.zim"
url="https://github.com/kiwix/libkiwix/raw/master/test/data/zimfile.zim"
url="https://github.com/kiwix/kiwix-lib/raw/master/test/data/zimfile.zim"
title="Ray Charles"
description="Wikipedia articles about Ray Charles"
language="eng"
@@ -18,7 +18,7 @@
<book
id="raycharles_uncategorized"
path="./zimfile.zim"
url="https://github.com/kiwix/libkiwix/raw/master/test/data/zimfile.zim"
url="https://github.com/kiwix/kiwix-lib/raw/master/test/data/zimfile.zim"
title="Ray (uncategorized) Charles"
description="No category is assigned to this library entry."
language="eng"
@@ -34,7 +34,7 @@
<book
id="charlesray"
path="./zimfile.zim"
url="https://github.com/kiwix/libkiwix/raw/master/test/data/zimfile.zim"
url="https://github.com/kiwix/kiwix-lib/raw/master/test/data/zimfile.zim"
title="Charles, Ray"
description="Wikipedia articles about Ray Charles"
language="eng"

View File

@@ -2890,15 +2890,12 @@ inline ssize_t Stream::write(const std::string &s) {
template <typename... Args>
inline ssize_t Stream::write_format(const char *fmt, const Args &... args) {
std::array<char, 2048> buf;
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunknown-warning-option"
#pragma GCC diagnostic ignored "-Wmaybe-uninitialized"
#if defined(_MSC_VER) && _MSC_VER < 1900
auto sn = _snprintf_s(buf, bufsiz, buf.size() - 1, fmt, args...);
#else
auto sn = snprintf(buf.data(), buf.size() - 1, fmt, args...);
#endif
#pragma GCC diagnostic pop
if (sn <= 0) { return sn; }
auto n = static_cast<size_t>(sn);
@@ -3654,11 +3651,7 @@ Server::process_request(Stream &strm, bool last_connection,
const std::function<void(Request &)> &setup_request) {
std::array<char, 2048> buf{};
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunknown-warning-option"
#pragma GCC diagnostic ignored "-Wmaybe-uninitialized"
detail::stream_line_reader line_reader(strm, buf.data(), buf.size());
#pragma GCC diagnostic pop
detail::stream_line_reader line_reader(strm, buf.data(), buf.size());
// Connection has been closed on client
if (!line_reader.getline()) { return false; }
@@ -3764,11 +3757,9 @@ inline socket_t Client::create_client_socket() const {
inline bool Client::read_response_line(Stream &strm, Response &res) {
std::array<char, 2048> buf;
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wunknown-warning-option"
#pragma GCC diagnostic ignored "-Wmaybe-uninitialized"
detail::stream_line_reader line_reader(strm, buf.data(), buf.size());
#pragma GCC diagnostic pop
if (!line_reader.getline()) { return false; }
const static std::regex re("(HTTP/1\\.[01]) (\\d+?) .*\r\n");

View File

@@ -186,7 +186,7 @@ const char sampleLibraryXML[] = R"(
<book
id="raycharles"
path="./zimfile.zim"
url="https://github.com/kiwix/libkiwix/raw/master/test/data/zimfile.zim"
url="https://github.com/kiwix/kiwix-lib/raw/master/test/data/zimfile.zim"
title="Ray Charles"
description="Wikipedia articles about Ray Charles"
language="eng"

View File

@@ -240,7 +240,6 @@ const char* urls404[] = {
"/skin/non-existent-skin-resource",
"/catalog",
"/catalog/non-existent-item",
"/catalogBLABLABLA/root.xml",
"/meta",
"/meta?content=zimfile",
"/meta?content=zimfile&name=non-existent-item",
@@ -571,21 +570,18 @@ protected:
}
};
// Returns a copy of 'text' where every line that fully matches 'pattern'
// preceded by optional whitespace is replaced with the fixed string
// 'replacement' preserving the leading whitespace
// Returns a copy of 'text' with every line that fully matches 'pattern'
// replaced with the fixed string 'replacement'
std::string replaceLines(const std::string& text,
const std::string& pattern,
const std::string& replacement)
{
std::regex regex("^ *" + pattern + "$");
std::regex regex("^" + pattern + "$");
std::ostringstream oss;
std::istringstream iss(text);
std::string line;
while ( std::getline(iss, line) ) {
if ( std::regex_match(line, regex) ) {
for ( size_t i = 0; i < line.size() && line[i] == ' '; ++i )
oss << ' ';
oss << replacement << "\n";
} else {
oss << line << "\n";
@@ -596,10 +592,10 @@ std::string replaceLines(const std::string& text,
std::string maskVariableOPDSFeedData(std::string s)
{
s = replaceLines(s, R"(<updated>\d\d\d\d-\d\d-\d\dT\d\d:\d\d:\d\dZ</updated>)",
"<updated>YYYY-MM-DDThh:mm:ssZ</updated>");
s = replaceLines(s, "<id>[[:xdigit:]]{8}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{4}-[[:xdigit:]]{12}</id>",
"<id>12345678-90ab-cdef-1234-567890abcdef</id>");
s = replaceLines(s, " <updated>.+</updated>",
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>");
s = replaceLines(s, " <id>.+</id>",
" <id>12345678-90ab-cdef-1234-567890abcdef</id>");
return s;
}
@@ -611,7 +607,7 @@ std::string maskVariableOPDSFeedData(std::string s)
" <link rel=\"self\" href=\"\" type=\"application/atom+xml\" />\n" \
" <link rel=\"search\"" \
" type=\"application/opensearchdescription+xml\"" \
" href=\"/catalog/searchdescription.xml\" />\n"
" href=\"catalog/searchdescription.xml\" />\n"
#define CHARLES_RAY_CATALOG_ENTRY \
" <entry>\n" \
@@ -619,7 +615,7 @@ std::string maskVariableOPDSFeedData(std::string s)
" <title>Charles, Ray</title>\n" \
" <summary>Wikipedia articles about Ray Charles</summary>\n" \
" <language>eng</language>\n" \
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n" \
" <updated>2020-03-31T00:00::00Z</updated>\n" \
" <name>wikipedia_en_ray_charles</name>\n" \
" <flavour></flavour>\n" \
" <category>jazz</category>\n" \
@@ -634,7 +630,7 @@ std::string maskVariableOPDSFeedData(std::string s)
" <publisher>\n" \
" <name>Kiwix</name>\n" \
" </publisher>\n" \
" <link rel=\"http://opds-spec.org/acquisition/open-access\" type=\"application/x-zim\" href=\"https://github.com/kiwix/libkiwix/raw/master/test/data/zimfile.zim\" length=\"569344\" />\n" \
" <link rel=\"http://opds-spec.org/acquisition/open-access\" type=\"application/x-zim\" href=\"https://github.com/kiwix/kiwix-lib/raw/master/test/data/zimfile.zim\" length=\"569344\" />\n" \
" </entry>\n"
#define RAY_CHARLES_CATALOG_ENTRY \
@@ -643,7 +639,7 @@ std::string maskVariableOPDSFeedData(std::string s)
" <title>Ray Charles</title>\n" \
" <summary>Wikipedia articles about Ray Charles</summary>\n" \
" <language>eng</language>\n" \
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n" \
" <updated>2020-03-31T00:00::00Z</updated>\n" \
" <name>wikipedia_en_ray_charles</name>\n" \
" <flavour></flavour>\n" \
" <category>wikipedia</category>\n" \
@@ -658,7 +654,7 @@ std::string maskVariableOPDSFeedData(std::string s)
" <publisher>\n" \
" <name>Kiwix</name>\n" \
" </publisher>\n" \
" <link rel=\"http://opds-spec.org/acquisition/open-access\" type=\"application/x-zim\" href=\"https://github.com/kiwix/libkiwix/raw/master/test/data/zimfile.zim\" length=\"569344\" />\n" \
" <link rel=\"http://opds-spec.org/acquisition/open-access\" type=\"application/x-zim\" href=\"https://github.com/kiwix/kiwix-lib/raw/master/test/data/zimfile.zim\" length=\"569344\" />\n" \
" </entry>\n"
#define UNCATEGORIZED_RAY_CHARLES_CATALOG_ENTRY \
@@ -667,7 +663,7 @@ std::string maskVariableOPDSFeedData(std::string s)
" <title>Ray (uncategorized) Charles</title>\n" \
" <summary>No category is assigned to this library entry.</summary>\n" \
" <language>eng</language>\n" \
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n" \
" <updated>2020-03-31T00:00::00Z</updated>\n" \
" <name>wikipedia_en_ray_charles</name>\n" \
" <flavour></flavour>\n" \
" <category></category>\n" \
@@ -682,7 +678,7 @@ std::string maskVariableOPDSFeedData(std::string s)
" <publisher>\n" \
" <name>Kiwix</name>\n" \
" </publisher>\n" \
" <link rel=\"http://opds-spec.org/acquisition/open-access\" type=\"application/x-zim\" href=\"https://github.com/kiwix/libkiwix/raw/master/test/data/zimfile.zim\" length=\"125952\" />\n" \
" <link rel=\"http://opds-spec.org/acquisition/open-access\" type=\"application/x-zim\" href=\"https://github.com/kiwix/kiwix-lib/raw/master/test/data/zimfile.zim\" length=\"125952\" />\n" \
" </entry>\n"
TEST_F(LibraryServerTest, catalog_root_xml)
@@ -694,7 +690,6 @@ TEST_F(LibraryServerTest, catalog_root_xml)
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n"
" <title>All zims</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
"\n"
CATALOG_LINK_TAGS
CHARLES_RAY_CATALOG_ENTRY
RAY_CHARLES_CATALOG_ENTRY
@@ -716,7 +711,7 @@ TEST_F(LibraryServerTest, catalog_searchdescription_xml)
" xmlns:atom=\"http://www.w3.org/2005/Atom\"\n"
" xmlns:k=\"http://kiwix.org/opensearchextension/1.0\"\n"
" indexOffset=\"0\"\n"
" template=\"/catalog/search?q={searchTerms?}&lang={language?}&name={k:name?}&tag={k:tag?}&notag={k:notag?}&maxsize={k:maxsize?}&count={count?}&start={startIndex?}\"/>\n"
" template=\"//catalog/search?q={searchTerms?}&lang={language?}&name={k:name?}&tag={k:tag?}&notag={k:notag?}&maxsize={k:maxsize?}&count={count?}&start={startIndex?}\"/>\n"
"</OpenSearchDescription>\n"
);
}
@@ -728,7 +723,7 @@ TEST_F(LibraryServerTest, catalog_search_by_phrase)
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
OPDS_FEED_TAG
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n"
" <title>Filtered zims (q=&quot;ray charles&quot;)</title>\n"
" <title>Search result for \"ray charles\"</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>2</totalResults>\n"
" <startIndex>0</startIndex>\n"
@@ -747,7 +742,7 @@ TEST_F(LibraryServerTest, catalog_search_by_words)
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
OPDS_FEED_TAG
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n"
" <title>Filtered zims (q=ray charles)</title>\n"
" <title>Search result for ray charles</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>3</totalResults>\n"
" <startIndex>0</startIndex>\n"
@@ -768,7 +763,7 @@ TEST_F(LibraryServerTest, catalog_prefix_search)
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
OPDS_FEED_TAG
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n"
" <title>Filtered zims (q=description:ray description:charles)</title>\n"
" <title>Search result for description:ray description:charles</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>2</totalResults>\n"
" <startIndex>0</startIndex>\n"
@@ -785,7 +780,7 @@ TEST_F(LibraryServerTest, catalog_prefix_search)
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
OPDS_FEED_TAG
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n"
" <title>Filtered zims (q=title:&quot;ray charles&quot;)</title>\n"
" <title>Search result for title:\"ray charles\"</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>1</totalResults>\n"
" <startIndex>0</startIndex>\n"
@@ -804,7 +799,7 @@ TEST_F(LibraryServerTest, catalog_search_with_word_exclusion)
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
OPDS_FEED_TAG
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n"
" <title>Filtered zims (q=ray -uncategorized)</title>\n"
" <title>Search result for ray -uncategorized</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>2</totalResults>\n"
" <startIndex>0</startIndex>\n"
@@ -823,7 +818,7 @@ TEST_F(LibraryServerTest, catalog_search_by_tag)
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
OPDS_FEED_TAG
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n"
" <title>Filtered zims (tag=_category:jazz)</title>\n"
" <title>Search result for &lt;Empty query&gt;</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>1</totalResults>\n"
" <startIndex>0</startIndex>\n"
@@ -841,7 +836,7 @@ TEST_F(LibraryServerTest, catalog_search_by_category)
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
OPDS_FEED_TAG
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n"
" <title>Filtered zims (category=jazz)</title>\n"
" <title>Search result for &lt;Empty query&gt;</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>1</totalResults>\n"
" <startIndex>0</startIndex>\n"
@@ -860,7 +855,7 @@ TEST_F(LibraryServerTest, catalog_search_results_pagination)
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
OPDS_FEED_TAG
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n"
" <title>Filtered zims (count=1)</title>\n"
" <title>Search result for &lt;Empty query&gt;</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>3</totalResults>\n"
" <startIndex>0</startIndex>\n"
@@ -876,7 +871,7 @@ TEST_F(LibraryServerTest, catalog_search_results_pagination)
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
OPDS_FEED_TAG
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n"
" <title>Filtered zims (count=1&amp;start=1)</title>\n"
" <title>Search result for &lt;Empty query&gt;</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>3</totalResults>\n"
" <startIndex>1</startIndex>\n"
@@ -892,217 +887,13 @@ TEST_F(LibraryServerTest, catalog_search_results_pagination)
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
OPDS_FEED_TAG
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n"
" <title>Filtered zims (count=10&amp;start=100)</title>\n"
" <title>Search result for &lt;Empty query&gt;</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>3</totalResults>\n"
" <startIndex>100</startIndex>\n"
" <itemsPerPage>0</itemsPerPage>\n"
CATALOG_LINK_TAGS
" \n"
"</feed>\n"
);
}
}
TEST_F(LibraryServerTest, catalog_v2_root)
{
const auto r = zfs1_->GET("/catalog/v2/root.xml");
EXPECT_EQ(r->status, 200);
const char expected_output[] = R"(<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:opds="https://specs.opds.io/opds-1.2">
<id>12345678-90ab-cdef-1234-567890abcdef</id>
<link rel="self"
href="/catalog/v2/root.xml"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<link rel="start"
href="/catalog/v2/root.xml"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<link rel="search"
href="/catalog/v2/searchdescription.xml"
type="application/opensearchdescription+xml"/>
<title>OPDS Catalog Root</title>
<updated>YYYY-MM-DDThh:mm:ssZ</updated>
<entry>
<title>All entries</title>
<link rel="subsection"
href="/catalog/v2/entries"
type="application/atom+xml;profile=opds-catalog;kind=acquisition"/>
<updated>YYYY-MM-DDThh:mm:ssZ</updated>
<id>12345678-90ab-cdef-1234-567890abcdef</id>
<content type="text">All entries from this catalog.</content>
</entry>
<entry>
<title>List of categories</title>
<link rel="subsection"
href="/catalog/v2/categories"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<updated>YYYY-MM-DDThh:mm:ssZ</updated>
<id>12345678-90ab-cdef-1234-567890abcdef</id>
<content type="text">List of all categories in this catalog.</content>
</entry>
</feed>
)";
EXPECT_EQ(maskVariableOPDSFeedData(r->body), expected_output);
}
TEST_F(LibraryServerTest, catalog_v2_searchdescription_xml)
{
const auto r = zfs1_->GET("/catalog/v2/searchdescription.xml");
EXPECT_EQ(r->status, 200);
EXPECT_EQ(r->body,
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"
"<OpenSearchDescription xmlns=\"http://a9.com/-/spec/opensearch/1.1/\">\n"
" <ShortName>Zim catalog search</ShortName>\n"
" <Description>Search zim files in the catalog.</Description>\n"
" <Url type=\"application/atom+xml;profile=opds-catalog;kind=acquisition\"\n"
" xmlns:atom=\"http://www.w3.org/2005/Atom\"\n"
" xmlns:k=\"http://kiwix.org/opensearchextension/1.0\"\n"
" indexOffset=\"0\"\n"
" template=\"/catalog/v2/entries?q={searchTerms?}&lang={language?}&name={k:name?}&tag={k:tag?}&maxsize={k:maxsize?}&count={count?}&start={startIndex?}\"/>\n"
"</OpenSearchDescription>\n"
);
}
TEST_F(LibraryServerTest, catalog_v2_categories)
{
const auto r = zfs1_->GET("/catalog/v2/categories");
EXPECT_EQ(r->status, 200);
const char expected_output[] = R"(<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
xmlns:opds="https://specs.opds.io/opds-1.2">
<id>12345678-90ab-cdef-1234-567890abcdef</id>
<link rel="self"
href="/catalog/v2/categories"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<link rel="start"
href="/catalog/v2/root.xml"
type="application/atom+xml;profile=opds-catalog;kind=navigation"/>
<title>List of categories</title>
<updated>YYYY-MM-DDThh:mm:ssZ</updated>
<entry>
<title>jazz</title>
<link rel="subsection"
href="/catalog/v2/entries?category=jazz"
type="application/atom+xml;profile=opds-catalog;kind=acquisition"/>
<updated>YYYY-MM-DDThh:mm:ssZ</updated>
<id>12345678-90ab-cdef-1234-567890abcdef</id>
<content type="text">All entries with category of 'jazz'.</content>
</entry>
<entry>
<title>wikipedia</title>
<link rel="subsection"
href="/catalog/v2/entries?category=wikipedia"
type="application/atom+xml;profile=opds-catalog;kind=acquisition"/>
<updated>YYYY-MM-DDThh:mm:ssZ</updated>
<id>12345678-90ab-cdef-1234-567890abcdef</id>
<content type="text">All entries with category of 'wikipedia'.</content>
</entry>
</feed>
)";
EXPECT_EQ(maskVariableOPDSFeedData(r->body), expected_output);
}
#define CATALOG_V2_ENTRIES_PREAMBLE(q) \
"<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n" \
"<feed xmlns=\"http://www.w3.org/2005/Atom\"\n" \
" xmlns:opds=\"https://specs.opds.io/opds-1.2\"\n" \
" xmlns:opensearch=\"http://a9.com/-/spec/opensearch/1.1/\">\n" \
" <id>12345678-90ab-cdef-1234-567890abcdef</id>\n" \
"\n" \
" <link rel=\"self\"\n" \
" href=\"/catalog/v2/entries" q "\"\n" \
" type=\"application/atom+xml;profile=opds-catalog;kind=acquisition\"/>\n" \
" <link rel=\"start\"\n" \
" href=\"/catalog/v2/root.xml\"\n" \
" type=\"application/atom+xml;profile=opds-catalog;kind=navigation\"/>\n" \
" <link rel=\"up\"\n" \
" href=\"/catalog/v2/root.xml\"\n" \
" type=\"application/atom+xml;profile=opds-catalog;kind=navigation\"/>\n" \
"\n" \
TEST_F(LibraryServerTest, catalog_v2_entries)
{
const auto r = zfs1_->GET("/catalog/v2/entries");
EXPECT_EQ(r->status, 200);
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
CATALOG_V2_ENTRIES_PREAMBLE("")
" <title>All Entries</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
"\n"
CHARLES_RAY_CATALOG_ENTRY
RAY_CHARLES_CATALOG_ENTRY
UNCATEGORIZED_RAY_CHARLES_CATALOG_ENTRY
"</feed>\n"
);
}
TEST_F(LibraryServerTest, catalog_v2_entries_filtered_by_range)
{
{
const auto r = zfs1_->GET("/catalog/v2/entries?start=1");
EXPECT_EQ(r->status, 200);
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
CATALOG_V2_ENTRIES_PREAMBLE("?start=1")
" <title>Filtered Entries (start=1)</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>3</totalResults>\n"
" <startIndex>1</startIndex>\n"
" <itemsPerPage>2</itemsPerPage>\n"
RAY_CHARLES_CATALOG_ENTRY
UNCATEGORIZED_RAY_CHARLES_CATALOG_ENTRY
"</feed>\n"
);
}
{
const auto r = zfs1_->GET("/catalog/v2/entries?count=2");
EXPECT_EQ(r->status, 200);
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
CATALOG_V2_ENTRIES_PREAMBLE("?count=2")
" <title>Filtered Entries (count=2)</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>3</totalResults>\n"
" <startIndex>0</startIndex>\n"
" <itemsPerPage>2</itemsPerPage>\n"
CHARLES_RAY_CATALOG_ENTRY
RAY_CHARLES_CATALOG_ENTRY
"</feed>\n"
);
}
{
const auto r = zfs1_->GET("/catalog/v2/entries?start=1&count=1");
EXPECT_EQ(r->status, 200);
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
CATALOG_V2_ENTRIES_PREAMBLE("?count=1&start=1")
" <title>Filtered Entries (count=1&amp;start=1)</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>3</totalResults>\n"
" <startIndex>1</startIndex>\n"
" <itemsPerPage>1</itemsPerPage>\n"
RAY_CHARLES_CATALOG_ENTRY
"</feed>\n"
);
}
}
TEST_F(LibraryServerTest, catalog_v2_entries_filtered_by_search_terms)
{
const auto r = zfs1_->GET("/catalog/v2/entries?q=\"ray%20charles\"");
EXPECT_EQ(r->status, 200);
EXPECT_EQ(maskVariableOPDSFeedData(r->body),
CATALOG_V2_ENTRIES_PREAMBLE("?q=%22ray%20charles%22")
" <title>Filtered Entries (q=&quot;ray charles&quot;)</title>\n"
" <updated>YYYY-MM-DDThh:mm:ssZ</updated>\n"
" <totalResults>2</totalResults>\n"
" <startIndex>0</startIndex>\n"
" <itemsPerPage>2</itemsPerPage>\n"
RAY_CHARLES_CATALOG_ENTRY
CHARLES_RAY_CATALOG_ENTRY
"</feed>\n"
);
}