Merge pull request #234 from kiwix/new_version

New version 0.7.0

.travis.yml

@@ -2,22 +2,11 @@ language: cpp
 dist: trusty
 sudo: required
 cache: ccache
-before_install:
-- eval "${MATRIX_EVAL}"
-- ${CXX} --version
 install: travis/install_deps.sh
 script: travis/compile.sh
 env:
-  global:
-    - MATRIX_EVAL="CC=gcc-5 && CXX=g++-5"
   matrix:
     - PLATFORM="native_static"
     - PLATFORM="native_dyn"
     - PLATFORM="win32_static"
     - PLATFORM="win32_dyn"
-addons:
-  apt:
-    sources:
-      - ubuntu-toolchain-r-test
-    packages:
-      - g++-5

Changelog

@@ -1,3 +1,67 @@
+kiwix-tools 0.7.0
+=================
+
+ * Adapt to kiwix-lib new API
+
+kiwix-serve
+-----------
+
+ * Dumps only valid books in the opdsfeed.
+ * Allow the opds feed to be filtered by lang and paginated.
+
+kiwix-manage
+------------
+
+ * Add a download command to download a remote book locally
+ * Book are referenced by bookId not index.
+ * No more indexType option as it is always XAPIAN.
+
+kiwix-tools 0.6.1
+=================
+
+kiwix-serve
+-----------
+
+ * Update README.
+ * Fix crash when `--library` flag is provided without value.
+ * Correctly handle mimetype of file without extension on 64bits.
+ * Minor fixes
+
+kiwix-tools 0.6.0
+=================
+
+ * remove kiwix-install tool.
+
+kiwix-serve
+-----------
+
+ * Improved taskbar #160
+ * Fix global page when using the option `--nosearchbar`
+ * Return 404 for missing resources
+ * Fix compilation for gcc 4.8.
+
+kiwix-manage
+------------
+
+ * Returns proper exit code (not always 0)
+
+
+kiwix-tools 0.5.0
+=================
+
+ * Build kiwix-tools setting the RPATH
+ * Compile without warning.
+
+
+kiwix-serve
+------------
+
+ * Serve metadata information using the "/meta" url.
+ * Serve an OPDS stream of all zim handled by kiwix-serve
+   All informations cannot be infer from the zim file itself,
+   you should use a library.xml to provide needed information (url, ...)
+ * Update kiwix-serve to use the new API of kiwix-lib
+
 kiwix-tools 0.4.0
 =================
 

README.md

@@ -1,14 +1,14 @@
 Kiwix tools
 ===========
 
-The Kiwix tools gathers kiwix command line tools.
+The Kiwix tools is a collection of Kiwix related command line tools.
 
 Disclaimer
 ----------
 
 This document assumes you have a little knowledge about software
 compilation. If you experience difficulties with the dependencies or
-with the Kiwix libary compilation itself, we recommend to have a look
+with the Kiwix tools compilation itself, we recommend to have a look
 to [kiwix-build](https://github.com/kiwix/kiwix-build).
 
 Preamble
@@ -22,12 +22,12 @@ on recent releases of Ubuntu and Fedora.
 Dependencies
 ------------
 
-The Kiwix tools rely on a few third parts software libraries. They
-are prerequisites to the Kiwix library compilation. Following
+The Kiwix tools rely on a few third party software libraries. They are
+prerequisites to the Kiwix tools compilation. Therefore, following
 libraries need to be available:
 
 * Kiwix lib ....................... https://github.com/kiwix/kiwix-lib
-(no package for now)
+(no package so far)
 * Libmicrohttpd .......... https://www.gnu.org/software/libmicrohttpd/
 (package libmicrohttpd-dev on Ubuntu)
 * CTPP2 ..................................... http://ctpp.havoc.ru/en/
@@ -53,47 +53,26 @@ If you compile manually Libmicrohttpd, you might need to compile it
 without GNU TLS, a bug here will empeach further compilation of Kiwix
 tools otherwise.
 
-Environnement
+Environment
 -------------
 
-The Kiwix library builds using [Meson](http://mesonbuild.com/) version
-0.34 or higher. Meson relies itself on Ninja, pkg-config and few other
-compilation tools.
-
-Install first the few common compilation tools:
-* Automake
-* Libtool
-* Virtualenv
+The Kiwix tools build using [Meson](http://mesonbuild.com/) version
+0.39 or higher. Meson relies itself on Ninja, pkg-config and few other
+compilation tools. Install them first:
+* Meson
+* Ninja
 * Pkg-config
 
-Then install Meson itself:
-```
-virtualenv -p python3 ./ # Create virtualenv
-source bin/activate      # Activate the virtualenv
-pip install meson        # Install Meson
-hash -r                  # Refresh bash paths
-```
-
-Finally download and build Ninja locally:
-```
-git clone git://github.com/ninja-build/ninja.git
-cd ninja
-git checkout release
-./configure.py --bootstrap
-mkdir ../bin                      
-cp ninja ../bin
-cd ..
-```
+These tools should be packaged if you use a cutting edge operating
+system. If not, have a look to the "Troubleshooting" section.
 
 Compilation
 -----------
 
-Once all dependencies are installed, you can compile kiwix-lib with:
+Once all dependencies are installed, you can compile Kiwix tools with:
 ```
-mkdir build
 meson . build
-cd build
-ninja
+ninja -C build
 ```
 
 By default, it will compile dynamic linked libraries. If you want
@@ -105,16 +84,55 @@ Depending of you system, `ninja` may be called `ninja-build`.
 Installation
 ------------
 
-If you want to install the Kiwix tools you just have compiled on your
-system, here we go:
+If you want to install the Kiwix tools, here we go:
 
 ```
-ninja install
+ninja -C build install
+```
+
+You might need to run the command as root (or using 'sudo'), depending
+where you want to install the Kiwix tools. After the installation
+succeeded, you may need to run ldconfig (as root).
+
+Uninstallation
+------------
+
+If you want to uninstall the Kiwix tools:
+
+```
+ninja -C build uninstall
+```
+
+Like for the installation, you might need to run the command as root
+(or using 'sudo').
+
+Troubleshooting
+---------------
+
+If you need to install Meson "manually":
+```
+virtualenv -p python3 ./ # Create virtualenv
+source bin/activate      # Activate the virtualenv
+pip3 install meson       # Install Meson
+hash -r                  # Refresh bash paths
+```
+
+If you need to install Ninja "manually":
+```
+git clone git://github.com/ninja-build/ninja.git
+cd ninja
+git checkout release
+./configure.py --bootstrap
+mkdir ../bin
+cp ninja ../bin
 cd ..
 ```
 
-You might need to run the command as root, depending where you want to
-install the libraries.
+If the compilation still fails, you might need to get a more recent
+version of a dependency than the one packaged by your Linux
+distribution. Try then with a source tarball distributed by the
+problematic upstream project or even directly from the source code
+repository.
 
 License
 -------

meson.build

@@ -1,7 +1,7 @@
 project('kiwix-tools', 'cpp',
-  version : '0.4.0',
+  version : '0.7.0',
   license : 'GPL',
-  default_options: ['c_std=c11', 'cpp_std=c++11'])
+  default_options: ['c_std=c11', 'cpp_std=c++11', 'werror=true'])
 
 compiler = meson.get_compiler('cpp')
 
@@ -11,7 +11,7 @@ if static_linkage
 endif
 
 thread_dep = dependency('threads')
-kiwixlib_dep = dependency('kiwix', version:'>=1.0.0', static:static_linkage)
+kiwixlib_dep = dependency('kiwix', version:'>=3.0.0', static:static_linkage)
 microhttpd_dep = dependency('libmicrohttpd', static:static_linkage)
 z_dep = dependency('zlib', static:static_linkage)
 

src/installer/kiwix-install.cpp

@@ -1,173 +0,0 @@
-/*
- * Copyright 2011-2014 Emmanuel Engelhart <kelson@kiwix.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU  General Public License as published by
- * the Free Software Foundation; either version 3 of the License, or
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
-
-#include <getopt.h>
-#include <kiwix/common/pathTools.h>
-#include <kiwix/manager.h>
-#include <kiwix/reader.h>
-
-enum supportedAction { NONE, ADDCONTENT };
-
-void usage()
-{
-  cout << "Usage: kiwix-install [--verbose] addcontent ZIM_PATH KIWIX_PATH"
-       << endl;
-  exit(1);
-}
-
-int main(int argc, char** argv)
-{
-  /* Init the variables */
-  const char* contentPath = NULL;
-  const char* kiwixPath = NULL;
-  supportedAction action = NONE;
-  bool verboseFlag = false;
-  int option_index = 0;
-  int c = 0;
-
-  /* Argument parsing */
-  while (42) {
-    static struct option long_options[]
-        = {{"verbose", no_argument, 0, 'v'}, {0, 0, 0, 0}};
-
-    if (c != -1) {
-      c = getopt_long(argc, argv, "vi", long_options, &option_index);
-
-      switch (c) {
-        case 'v':
-          verboseFlag = true;
-          break;
-      }
-    } else {
-      if (optind < argc) {
-        if (action == NONE) {
-          string actionString = argv[optind++];
-          if (actionString == "addcontent" || actionString == "ADDCONTENT") {
-            action = ADDCONTENT;
-          }
-        } else if (contentPath == NULL) {
-          contentPath = argv[optind++];
-        } else if (kiwixPath == NULL) {
-          kiwixPath = argv[optind++];
-        } else {
-          usage();
-        }
-      } else {
-        break;
-      }
-    }
-  }
-
-  /* Check if we have enough arguments */
-  if (contentPath == NULL || kiwixPath == NULL) {
-    usage();
-  }
-
-  /* Make the action */
-  if (action == ADDCONTENT) {
-    /* Check if the content path exists and is readable */
-    if (verboseFlag) {
-      std::cout << "Check if the ZIM file exists..." << std::endl;
-    }
-    if (!fileExists(contentPath)) {
-      cerr << "The content path '" << contentPath
-           << "' does not exist or is not readable." << endl;
-      exit(1);
-    }
-
-    /* Check if this is a ZIM file */
-    try {
-      if (verboseFlag) {
-        std::cout << "Check if the ZIM file is valid..." << std::endl;
-      }
-      kiwix::Reader* reader = new kiwix::Reader(contentPath);
-      delete reader;
-    } catch (exception& e) {
-      cerr << "The content available at '" << contentPath
-           << "' is not a ZIM file." << endl;
-      exit(1);
-    }
-    string contentFilename = getLastPathElement(contentPath);
-
-    /* Check if kiwixPath/kiwix/kiwix.exe exists */
-    if (verboseFlag) {
-      std::cout << "Check if the Kiwix path is valid..." << std::endl;
-    }
-    string kiwixBinaryPath = computeAbsolutePath(kiwixPath, "kiwix/kiwix.exe");
-    if (!fileExists(kiwixBinaryPath)) {
-      kiwixBinaryPath = computeAbsolutePath(kiwixPath, "kiwix/kiwix");
-      if (!fileExists(kiwixBinaryPath)) {
-        cerr << "Unable to find the Kiwix binary at '" << kiwixBinaryPath
-             << "[.exe]'." << endl;
-        exit(1);
-      }
-    }
-
-    /* Check if the directory "data" structure exists */
-    if (verboseFlag) {
-      std::cout << "Check the target data directory structure..." << std::endl;
-    }
-    string dataPath = computeAbsolutePath(kiwixPath, "data/");
-    if (!fileExists(dataPath)) {
-      makeDirectory(dataPath);
-    }
-
-    /* Check if the directory "data/content" structure exists */
-    string dataContentPath = computeAbsolutePath(kiwixPath, "data/content/");
-    if (!fileExists(dataContentPath)) {
-      makeDirectory(dataContentPath);
-    }
-
-    /* Check if the directory "data/library" structure exists */
-    string dataLibraryPath = computeAbsolutePath(kiwixPath, "data/library/");
-    if (!fileExists(dataLibraryPath)) {
-      makeDirectory(dataLibraryPath);
-    }
-
-    /* Copy the file to the data/content directory */
-    if (verboseFlag) {
-      std::cout << "Copy ZIM file to the target directory..." << std::endl;
-    }
-    string newContentPath
-        = computeAbsolutePath(dataContentPath, contentFilename);
-    if (!fileExists(newContentPath)
-        || getFileSize(contentPath) != getFileSize(newContentPath)) {
-      copyFile(contentPath, newContentPath);
-    }
-
-    /* Add the file to the library.xml */
-    if (verboseFlag) {
-      std::cout << "Create the library XML file..." << std::endl;
-    }
-    kiwix::Manager libraryManager;
-    string libraryPath
-        = computeAbsolutePath(dataLibraryPath, contentFilename + ".xml");
-    string bookId = libraryManager.addBookFromPathAndGetId(
-        newContentPath, "../content/" + contentFilename, "", false);
-    if (!bookId.empty()) {
-      libraryManager.setCurrentBookId(bookId);
-      libraryManager.writeFile(libraryPath);
-    } else {
-      cerr << "Unable to build or save library file '" << libraryPath << "'"
-           << endl;
-    }
-  }
-
-  exit(0);
-}

src/installer/meson.build

@@ -1,3 +0,0 @@
-executable('kiwix-install', ['kiwix-install.cpp'],
-  dependencies:all_deps,
-  install:true)

src/man/fr/kiwix-install.1

@@ -1,40 +0,0 @@
-.TH KIWIX 1 "21 May 2012"
-.SH NAME
-kiwix\-install \- Installeur de fichiers ZIM
-.SH SYNOPSIS
-.IX Header SYNOPSIS
-kiwix\-install [\-\-verbose|-v] [\-\-backend|\-b=xapian|clucene] [\-\-buildIndex|\-i] addcontent ZIM_PATH KIWIX_PATH
-.SH DESCRIPTION
-.PP
-Créé une arborescence contenant Kiwix et des données pour redistribution.
-.br
-Si demandé, créé aussi un index plein texte.
-
-.TP
-\fB\-\-verbose\fR
-Active le mode verbeux de la sortie.
-
-.TP
-\fB\-\-backend=xapian|clucene\fR
-Séléctionne un moteur d'indexation.
-
-.TP
-\fB\-\-buildIndex\fR
-Créer un index plein texte pour le fichier ZIM.
-
-.TP
-\fBZIM_PATH\fR
-Chemin du fichier de contenu ZIM.
-
-.TP
-\fBKIWIX_PATH\fR
-Chemin d'accès dossier kiwix/.
-.br
-Le chemin doit contenit le binaire statique de Kiwix.
-
-.SH SEE ALSO
-kiwix(1) kiwix\-serve(1) kiwix\-manage(1)
-.SH AUTHOR
-Emmanuel Engelhart <kelson@kiwix.org>
-.br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/fr/kiwix.1

@@ -1,52 +0,0 @@
-.TH KIWIX 1 "12 juin 2012"
-.SH NAME
-Kiwix \- Lecteur hors-ligne de fichiers ZIM
-.SH SYNOPSIS
-.B kiwix [-jsconsole] [-articleByUrl] [-articleByTitle] [FILE]
-.SH DESCRIPTION
-.PP
-Kiwix est un lecteur de content au format ZIM.
-.br
-Les fichiers ZIM sont des archives de contenus compressés (généralement HTML)
-.br
-Les fichiers ZIM les plus populaires sont ceux de Wikipédia et Wikileaks.
-
-.TP
-\fB\-jsconsole\fR
-Active la console de debogage JavaScript de Xulrunner.
-\fB\-articleByUrl url\fR
-Ouvre un article en particulier en renseignant son url. FILE doit être donné.
-.TP
-\fB\-articleByTitle title\fR
-Ouvre un article en particulier en renseignant son titre. FILE doit être donné.
-
-.PP
-Fonctionnalités:
- * Lecteur de ZIM natif
- * Moteur de recherche plein texte
- * Signets et Notes
- * Serveur Web pour diffuser les fichiers ZIM sur le réseau
- * Export HTML et PDF
- * Traduits dans de nombreuses langues
- * Suggestions de recherche
- * Indexation des fichiers ZIM
- * Onglets de navigation
- * Bibliothèque de contenus intégrée
-
-.SH SEE ALSO
-kiwix\-install(1) kiwix\-serve(1)
-.br
-kiwix\-manage(1)
-
-
-.SH TROUBLESHOOTING
-Aller à http://reportabug.kiwix.org pour plus de détails sur comment rapporter un bogue.
-
-.SH AUTHORS
- Emmanuel Engelhart <kelson@kiwix.org>
- Guillaume Duhamel <gduhamel@linterweb.com>
- Fabien Coullon <fcoulon@linterweb.com>
- Renaud Gaudin <rgaudin@gmail.com>
- Wilfredo Rodriguez <wilfredor@kiwix.org>
-.br
- Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/fr/meson.build

@@ -1,4 +1,3 @@
-install_man('kiwix-install.1',
-            'kiwix-manage.1',
+install_man('kiwix-manage.1',
             'kiwix-serve.1',
-            install_dir:get_option('mandir')+'/fr/man1')
+            install_dir:get_option('mandir')+'/fr/man1')

src/man/kiwix-install.1

@@ -1,40 +0,0 @@
-.TH KIWIX 1 "21 May 2012"
-.SH NAME
-kiwix\-install \- Kiwix ZIM Installer
-.SH SYNOPSIS
-.IX Header SYNOPSIS
-kiwix\-install [\-\-verbose|-v] [\-\-backend|\-b=xapian|clucene] [\-\-buildIndex|\-i] addcontent ZIM_PATH KIWIX_PATH
-.SH DESCRIPTION
-.PP
-Creates a standalone Kiwix + data hierarchy.
-.br
-If specified, also creates a full\-text index.
-
-.TP
-\fB\-\-verbose\fR
-Enable verbose output.
-
-.TP
-\fB\-\-backend=xapian|clucene\fR
-Select an indexing backend.
-
-.TP
-\fB\-\-buildIndex\fR
-Build an index for the ZIM file.
-
-.TP
-\fBZIM_PATH\fR
-ZIM content file path.
-
-.TP
-\fBKIWIX_PATH\fR
-Path to the kiwix/ folder.
-.br
-The path must contain the kiwix standalone binary.
-
-.SH SEE ALSO
-kiwix(1) kiwix\-serve(1) kiwix\-manage(1)
-.SH AUTHOR
-Emmanuel Engelhart <kelson@kiwix.org>
-.br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/meson.build

@@ -1,4 +1,3 @@
-install_man('kiwix-install.1',
-            'kiwix-manage.1',
+install_man('kiwix-manage.1',
             'kiwix-serve.1')
 subdir('fr')

src/manager/kiwix-manage.cpp

@@ -22,33 +22,39 @@
 #endif
 #include <getopt.h>
 #include <kiwix/common/pathTools.h>
+#include <kiwix/common/stringTools.h>
 #include <kiwix/manager.h>
+#include <kiwix/downloader.h>
 #include <cstdlib>
 #include <iostream>
 
+#include <thread>
+#include <chrono>
+
 using namespace std;
 
-enum supportedAction { NONE, ADD, SHOW, REMOVE };
+enum supportedAction { NONE, ADD, SHOW, REMOVE, DOWNLOAD };
 
-void show(kiwix::Library library)
+void show(kiwix::Library* library)
 {
-  std::vector<kiwix::Book>::iterator itr;
+  auto booksIds = library->getBooksIds();
   unsigned int inc = 1;
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
+  for(auto& id: booksIds) {
+    auto& book = library->getBookById(id);
     std::cout << "#" << inc++ << std::endl
-              << "id:\t\t" << itr->id << std::endl
-              << "path:\t\t" << itr->path << std::endl
-              << "indexpath:\t" << itr->indexPath << std::endl
-              << "url:\t\t" << itr->url << std::endl
-              << "title:\t\t" << itr->title << std::endl
-              << "name:\t\t" << itr->name << std::endl
-              << "tags:\t\t" << itr->tags << std::endl
-              << "description:\t" << itr->description << std::endl
-              << "creator:\t" << itr->creator << std::endl
-              << "date:\t\t" << itr->date << std::endl
-              << "articleCount:\t" << itr->articleCount << std::endl
-              << "mediaCount:\t" << itr->mediaCount << std::endl
-              << "size:\t\t" << itr->size << " KB" << std::endl
+              << "id:\t\t" << book.getId() << std::endl
+              << "path:\t\t" << book.getPath() << std::endl
+              << "indexpath:\t" << book.getIndexPath() << std::endl
+              << "url:\t\t" << book.getUrl() << std::endl
+              << "title:\t\t" << book.getTitle() << std::endl
+              << "name:\t\t" << book.getName() << std::endl
+              << "tags:\t\t" << book.getTags() << std::endl
+              << "description:\t" << book.getDescription() << std::endl
+              << "creator:\t" << book.getCreator() << std::endl
+              << "date:\t\t" << book.getDate() << std::endl
+              << "articleCount:\t" << book.getArticleCount() << std::endl
+              << "mediaCount:\t" << book.getMediaCount() << std::endl
+              << "size:\t\t" << book.getSize() << " KB" << std::endl
               << std::endl;
   }
 }
@@ -67,14 +73,179 @@ void usage()
   cerr << "\tkiwix-manage LIBRARY_PATH remove CONTENTID1 [CONTENTID2]" << endl;
 }
 
+
+bool handle_show(kiwix::Library* library, const std::string& libraryPath,
+                 int argc, char* argv[])
+{
+  show(library);
+  return(0);
+}
+
+bool handle_add(kiwix::Library* library, const std::string& libraryPath,
+                int argc, char* argv[])
+{
+  string zimPath;
+  string zimPathToSave = ".";
+  string indexPath;
+  string url;
+  string origID = "";
+  int option_index = 0;
+  int c = 0;
+  bool resultCode = 0;
+
+  if (argc > 3) {
+    zimPath = argv[3];
+  }
+
+  /* Options parsing */
+  optind = 2;
+  while (42) {
+    static struct option long_options[]
+        = {{"url", required_argument, 0, 'u'},
+           {"origId", required_argument, 0, 'o'},
+           {"indexPath", required_argument, 0, 'i'},
+           {"zimPathToSave", required_argument, 0, 'z'},
+           {0, 0, 0, 0}};
+
+    c = getopt_long(argc, argv, "cz:u:i:", long_options, &option_index);
+
+    if (c != -1) {
+      switch (c) {
+        case 'u':
+          url = optarg;
+          break;
+
+        case 'o':
+          origID = optarg;
+          break;
+
+        case 'i':
+          indexPath = optarg;
+          break;
+
+        case 'z':
+          zimPathToSave = optarg;
+          break;
+      }
+    } else {
+      break;
+    }
+  }
+
+  if (!zimPath.empty()) {
+    kiwix::Manager manager(library);
+    zimPathToSave = zimPathToSave == "." ? zimPath : zimPathToSave;
+    string bookId = manager.addBookFromPathAndGetId(
+        zimPath, zimPathToSave, url, false);
+     if (!bookId.empty()) {
+      /* Save the index infos if necessary */
+      if (!indexPath.empty()) {
+        if (isRelativePath(indexPath)) {
+          indexPath = computeAbsolutePath(indexPath, getCurrentDirectory());
+        }
+        library->getBookById(bookId).setIndexPath(indexPath);
+      }
+     } else {
+      cerr << "Unable to build or save library file '" << libraryPath << "'"
+           << endl;
+      resultCode = 1;
+    }
+  } else {
+    std::cerr << "Invalid zim file path" << std::endl;
+    resultCode = 1;
+  }
+
+  return(resultCode);
+}
+
+bool handle_remove(kiwix::Library* library, const std::string& libraryPath,
+                   int argc, char* argv[])
+{
+  std::string bookId;
+  const unsigned int totalBookCount = library->getBookCount(true, true);
+  bool exitCode = 0;
+
+  if (argc > 3) {
+    bookId = argv[3];
+  }
+
+  if (!library->removeBookById(bookId)) {
+    if (totalBookCount > 0) {
+      std::cerr
+          << "Invalid book id." << std::endl;
+      exitCode = 1;
+    } else {
+      std::cerr
+          << "Invalid book id. Library is empty, no book to delete."
+          << std::endl;
+      exitCode = 1;
+    }
+  }
+
+  return(exitCode);
+}
+
+bool handle_download(kiwix::Library* library, const std::string& libraryPath,
+                     int argc, char* argv[])
+{
+  std::string bookId;
+  bool exitCode = false;
+
+  if (argc > 3) {
+     bookId = argv[3];
+  }
+
+  auto& book = library->getBookById(bookId);
+  auto did = book.getDownloadId();
+  kiwix::Downloader downloader;
+  kiwix::Download* download = nullptr;
+  if (!did.empty()) {
+    std::cout << "try resume " << did << std::endl;
+    try {
+      download = downloader.getDownload(did);
+    } catch(...) {}
+  }
+  if (nullptr == download || download->getStatus() == kiwix::Download::K_UNKNOWN) {
+    download = downloader.startDownload(book.getUrl());
+    book.setDownloadId(download->getDid());
+  }
+  int step = 60*5;
+  while (step--) {
+    download->updateStatus();
+    if (download->getStatus() == kiwix::Download::K_COMPLETE) {
+      auto followingId = download->getFollowedBy();
+      if (followingId.empty()) {
+        book.setPath(download->getPath());
+        book.setDownloadId("");
+        std::cout << "File downloaded to " << book.getPath() << std::endl;
+        break;
+      } else {
+        download = downloader.getDownload(followingId);
+      }
+    } else if (download->getStatus() == kiwix::Download::K_ACTIVE) {
+      std::cout << download->getDid() << " : "
+                << kiwix::beautifyFileSize(download->getCompletedLength()) << "/"
+                << kiwix::beautifyFileSize(download->getTotalLength())
+                << " (" << kiwix::beautifyFileSize(download->getDownloadSpeed()) << "/s) "
+                << " [" << kiwix::beautifyFileSize(download->getVerifiedLength()) << "]"
+                << "[" << step << "]    \n" << std::flush;
+    } else if (download->getStatus() == kiwix::Download::K_ERROR) {
+      std::cout << "File Error" << std::endl;
+      exitCode = true;
+      break;
+    }
+    std::this_thread::sleep_for(std::chrono::seconds(1));
+  }
+
+  downloader.close();
+  return exitCode;
+}
+
 int main(int argc, char** argv)
 {
   string libraryPath = "";
   supportedAction action = NONE;
-  string zimPath = "";
-  kiwix::Manager libraryManager;
-  int option_index = 0;
-  int c = 0;
+  kiwix::Library library;
 
   /* Argument parsing */
   if (argc > 2) {
@@ -87,6 +258,8 @@ int main(int argc, char** argv)
       action = SHOW;
     else if (actionString == "remove" || actionString == "delete")
       action = REMOVE;
+    else if (actionString == "download")
+      action = DOWNLOAD;
   }
 
   /* Print usage)) if necessary */
@@ -99,121 +272,32 @@ int main(int argc, char** argv)
   libraryPath = isRelativePath(libraryPath)
                     ? computeAbsolutePath(getCurrentDirectory(), libraryPath)
                     : libraryPath;
-  libraryManager.readFile(libraryPath, false);
+  kiwix::Manager manager(&library);
+  manager.readFile(libraryPath, false);
 
   /* SHOW */
-  if (action == SHOW) {
-    show(libraryManager.cloneLibrary());
-  } else if (action == ADD) {
-    string zimPath;
-    string zimPathToSave = ".";
-    string indexPath;
-    kiwix::supportedIndexType indexBackend = kiwix::XAPIAN;
-    string url;
-    string origID = "";
-    bool setCurrent = false;
-
-    if (argc > 3) {
-      zimPath = argv[3];
-    }
-
-    /* Options parsing */
-    optind = 2;
-    while (42) {
-      static struct option long_options[]
-          = {{"url", required_argument, 0, 'u'},
-             {"origId", required_argument, 0, 'o'},
-             {"indexPath", required_argument, 0, 'i'},
-             {"indexBackend", required_argument, 0, 'b'},
-             {"zimPathToSave", required_argument, 0, 'z'},
-             {"current", no_argument, 0, 'c'},
-             {0, 0, 0, 0}};
-
-      c = getopt_long(argc, argv, "cz:u:i:b:", long_options, &option_index);
-
-      if (c != -1) {
-        switch (c) {
-          case 'u':
-            url = optarg;
-            break;
-
-          case 'o':
-            origID = optarg;
-            break;
-
-          case 'c':
-            setCurrent = true;
-            break;
-
-          case 'i':
-            indexPath = optarg;
-            break;
-
-          case 'b':
-            if (!strcmp(optarg, "xapian")) {
-              indexBackend = kiwix::XAPIAN;
-            } else {
-              usage();
-            }
-            break;
-
-          case 'z':
-            zimPathToSave = optarg;
-            break;
-        }
-      } else {
-        break;
-      }
-    }
-
-    if (!zimPath.empty()) {
-      zimPathToSave = zimPathToSave == "." ? zimPath : zimPathToSave;
-      string bookId = libraryManager.addBookFromPathAndGetId(
-          zimPath, zimPathToSave, url, false);
-
-      if (!bookId.empty()) {
-        if (setCurrent)
-          libraryManager.setCurrentBookId(bookId);
-
-        /* Save the index infos if necessary */
-        if (!indexPath.empty())
-          libraryManager.setBookIndex(bookId, indexPath, indexBackend);
-
-      } else {
-        cerr << "Unable to build or save library file '" << libraryPath << "'"
-             << endl;
-      }
-    } else {
-      std::cerr << "Invalid zim file path" << std::endl;
-    }
-
-  } else if (action == REMOVE) {
-    unsigned int bookIndex = 0;
-    const unsigned int totalBookCount = libraryManager.getBookCount(true, true);
-
-    if (argc > 3) {
-      bookIndex = atoi(argv[3]);
-    }
-
-    if (bookIndex > 0 && bookIndex <= totalBookCount) {
-      libraryManager.removeBookByIndex(bookIndex - 1);
-    } else {
-      if (totalBookCount > 0) {
-        std::cerr
-            << "Invalid book index number. Please give a number between 1 and "
-            << totalBookCount << std::endl;
-      } else {
-        std::cerr
-            << "Invalid book index number. Library is empty, no book to delete."
-            << std::endl;
-      }
-    }
+  bool exitCode = 0;
+  switch (action) {
+    case SHOW:
+      exitCode = handle_show(&library, libraryPath, argc, argv);
+      break;
+    case ADD:
+      exitCode = handle_add(&library, libraryPath, argc, argv);
+      break;
+    case REMOVE:
+      exitCode = handle_remove(&library, libraryPath, argc, argv);
+      break;
+    case DOWNLOAD:
+      exitCode = handle_download(&library, libraryPath, argc, argv);
+      break;
+    case NONE:
+      break;
   }
 
   /* Rewrite the library file */
-  if (action == REMOVE || action == ADD) {
-    libraryManager.writeFile(libraryPath);
+  if (action == REMOVE || action == ADD || action == DOWNLOAD) {
+    library.writeToFile(libraryPath);
   }
 
-  exit(0);
+  exit(exitCode);
 }

src/manager/meson.build

@@ -1,3 +1,4 @@
 executable('kiwix-manage', ['kiwix-manage.cpp'],
   dependencies:all_deps,
-  install:true)
+  install:true,
+  install_rpath: join_paths(get_option('prefix'), get_option('libdir')))

src/meson.build

@@ -1,5 +1,4 @@
 
-subdir('installer')
 subdir('manager')
 subdir('reader')
 subdir('searcher')

src/reader/kiwix-read.cpp

@@ -79,10 +79,8 @@ int main(int argc, char** argv)
 
   /* Start to read an article */
   if (reader != NULL) {
-    string mainPageUrl = reader->getMainPageUrl();
     string content;
     string contentType;
-    unsigned int contentLength = 0;
     string suggestion;
 
     if (pattern != NULL) {
@@ -94,13 +92,6 @@ int main(int argc, char** argv)
       }
     }
 
-    /*
-    if (reader->getContentByUrl(mainPageUrl, content, contentLength,
-    contentType)) {
-      cout << content << endl;
-    }
-    */
-
     delete reader;
   } else {
     cerr << "Unable instanciate the Kiwix reader." << endl;

src/reader/meson.build

@@ -1,3 +1,4 @@
 executable('kiwix-read', ['kiwix-read.cpp'],
   dependencies:all_deps,
-  install:true)
+  install:true,
+  install_rpath: join_paths(get_option('prefix'), get_option('libdir')))

src/searcher/meson.build

@@ -1,3 +1,4 @@
 executable('kiwix-search', ['kiwix-search.cpp'],
   dependencies:all_deps,
-  install:true)
+  install:true,
+  install_rpath: join_paths(get_option('prefix'), get_option('libdir')))

src/server/kiwix-serve.cpp

@@ -51,6 +51,7 @@ extern "C" {
 #include <kiwix/manager.h>
 #include <kiwix/reader.h>
 #include <kiwix/searcher.h>
+#include <kiwix/opds_dumper.h>
 #include <pthread.h>
 #include <stdio.h>
 #include <stdlib.h>
@@ -78,6 +79,7 @@ extern "C" {
 #include <netdb.h>
 #include <stdint.h>
 #include <sys/socket.h>
+#include <netinet/in.h>
 #include <unistd.h>
 #endif
 
@@ -92,21 +94,35 @@ using namespace std;
 static bool noLibraryButtonFlag = false;
 static bool noSearchBarFlag = false;
 static string welcomeHTML;
+static string catalogOpenSearchDescription;
 static std::atomic_bool isVerbose(false);
 static std::string rootLocation = "";
 static std::map<std::string, std::string> extMimeTypes;
 static std::map<std::string, kiwix::Reader*> readers;
 static std::map<std::string, kiwix::Searcher*> searchers;
 static kiwix::Searcher* globalSearcher = nullptr;
+static kiwix::Library library;
 static pthread_mutex_t searchLock = PTHREAD_MUTEX_INITIALIZER;
 static pthread_mutex_t compressorLock = PTHREAD_MUTEX_INITIALIZER;
 static pthread_mutex_t regexLock = PTHREAD_MUTEX_INITIALIZER;
 
+std::pair<kiwix::Reader*, kiwix::Searcher*>
+get_from_humanReadableBookId(const std::string& humanReadableBookId) {
+  kiwix::Searcher* searcher
+        = searchers.find(humanReadableBookId) != searchers.end()
+              ? searchers.find(humanReadableBookId)->second
+              : globalSearcher;
+  kiwix::Reader* reader = readers.find(humanReadableBookId) != readers.end()
+                                ? readers.find(humanReadableBookId)->second
+                                : NULL;
+  return std::pair<kiwix::Reader*, kiwix::Searcher*>(reader, searcher);
+}
+
 /* Try to get the mimeType from the file extension */
 static std::string getMimeTypeForFile(const std::string& filename)
 {
   std::string mimeType = "text/plain";
-  unsigned int pos = filename.find_last_of(".");
+  auto pos = filename.find_last_of(".");
 
   if (pos != std::string::npos) {
     std::string extension = filename.substr(pos + 1);
@@ -135,7 +151,10 @@ static bool startswith(const std::string& base, const std::string& start)
 
 void introduceTaskbar(string& content, const string& humanReadableBookId)
 {
+  string zimTitle;
+
   pthread_mutex_lock(&regexLock);
+
   if (!noSearchBarFlag) {
     content = appendToFirstOccurence(
         content,
@@ -157,10 +176,18 @@ void introduceTaskbar(string& content, const string& humanReadableBookId)
              RESOURCE::taskbar_html_part,
              humanReadableBookId,
              "__CONTENT__"));
+
+      auto reader = get_from_humanReadableBookId(humanReadableBookId).first;
+      if (reader != nullptr) {
+        zimTitle = reader->getTitle();
+      }
     }
-    content = replaceRegex(content, rootLocation, "__ROOT_LOCATION__");
-    content = replaceRegex(content, replaceRegex(humanReadableBookId, "%26", "&"), "__CONTENT_ESCAPED__");
   }
+
+  content = replaceRegex(content, rootLocation, "__ROOT_LOCATION__");
+  content = replaceRegex(content, replaceRegex(zimTitle, "%26", "&"), "__ZIM_TITLE__");
+  content = replaceRegex(content, replaceRegex(humanReadableBookId, "%26", "&"), "__CONTENT_ESCAPED__");
+
   pthread_mutex_unlock(&regexLock);
 }
 
@@ -218,8 +245,8 @@ static struct MHD_Response* build_response(const void* data,
                                            bool cacheEnabled)
 {
   /* Create the response */
-  struct MHD_Response* response = MHD_create_response_from_data(
-      length, const_cast<void*>(data), MHD_NO, MHD_YES);
+  struct MHD_Response* response = MHD_create_response_from_buffer(
+      length, const_cast<void*>(data), MHD_RESPMEM_MUST_COPY);
 
   /* Make a redirection if necessary otherwise send the content */
   if (!httpRedirection.empty()) {
@@ -292,22 +319,18 @@ static struct MHD_Response* build_homepage(RequestContext* request)
 }
 
 struct RunningResponse {
-   zim::Article* article;
+   kiwix::Entry entry;
    int range_start;
 
-   RunningResponse(zim::Article* article,
+   RunningResponse(kiwix::Entry entry,
                    int range_start) :
-     article(article),
+     entry(entry),
      range_start(range_start)
    {}
-
-   ~RunningResponse() {
-     delete article;
-   }
 };
 
 
-ssize_t callback_reader_from_article(void* cls,
+ssize_t callback_reader_from_entry(void* cls,
                                   uint64_t pos,
                                   char* buf,
                                   size_t max)
@@ -316,44 +339,44 @@ ssize_t callback_reader_from_article(void* cls,
 
   size_t max_size_to_set = min<size_t>(
     max,
-    response->article->getArticleSize() - pos - response->range_start);
+    response->entry.getSize() - pos - response->range_start);
 
   if (max_size_to_set <= 0) {
     return MHD_CONTENT_READER_END_WITH_ERROR;
   }
 
-  zim::Blob blob = response->article->getData(response->range_start+pos, max_size_to_set);
+  zim::Blob blob = response->entry.getBlob(response->range_start+pos, max_size_to_set);
   memcpy(buf, blob.data(), max_size_to_set);
   return max_size_to_set;
 }
 
-void callback_free_article(void* cls)
+void callback_free_response(void* cls)
 {
   RunningResponse* response = static_cast<RunningResponse*>(cls);
   delete response;
 }
 
-static struct MHD_Response* build_callback_response_from_article(
-    zim::Article& article, int range_start, int range_len, const std::string& mimeType)
+static struct MHD_Response* build_callback_response_from_entry(
+   kiwix::Entry entry, int range_start, int range_len, const std::string& mimeType)
 {
   RunningResponse* run_response =
-      new RunningResponse(new zim::Article(article), range_start);
+      new RunningResponse(entry, range_start);
 
   struct MHD_Response* response
-      = MHD_create_response_from_callback(article.getArticleSize(),
+      = MHD_create_response_from_callback(entry.getSize(),
                                           16384,
-                                          callback_reader_from_article,
+                                          callback_reader_from_entry,
                                           run_response,
-                                          callback_free_article);
+                                          callback_free_response);
   /* Tell the client that byte ranges are accepted */
   MHD_add_response_header(response, MHD_HTTP_HEADER_ACCEPT_RANGES, "bytes");
   std::ostringstream oss;
-  oss << "bytes " << range_start << "-" << range_start + range_len - 1 << "/" << article.getArticleSize();
+  oss << "bytes " << range_start << "-" << range_start + range_len - 1 << "/" << entry.getSize();
 
   MHD_add_response_header(response, MHD_HTTP_HEADER_CONTENT_RANGE, oss.str().c_str());
 
   MHD_add_response_header(response, MHD_HTTP_HEADER_CONTENT_LENGTH,
-    std::to_string(range_len).c_str());
+    kiwix::to_string(range_len).c_str());
 
   /* Specify the mime type */
   MHD_add_response_header(
@@ -370,16 +393,48 @@ static struct MHD_Response* build_callback_response_from_article(
   return response;
 }
 
-std::pair<kiwix::Reader*, kiwix::Searcher*>
-get_from_humanReadableBookId(const std::string& humanReadableBookId) {
-  kiwix::Searcher* searcher
-        = searchers.find(humanReadableBookId) != searchers.end()
-              ? searchers.find(humanReadableBookId)->second
-              : globalSearcher;
-  kiwix::Reader* reader = readers.find(humanReadableBookId) != readers.end()
-                                ? readers.find(humanReadableBookId)->second
-                                : NULL;
-  return std::pair<kiwix::Reader*, kiwix::Searcher*>(reader, searcher);
+static struct MHD_Response* handle_meta(RequestContext* request)
+{
+  std::string humanReadableBookId;
+  std::string meta_name;
+  try {
+    humanReadableBookId = request->get_argument("content");
+    meta_name = request->get_argument("name");
+  } catch (const std::out_of_range& e) {
+    return build_404(request, humanReadableBookId);
+  }
+
+  auto reader = get_from_humanReadableBookId(humanReadableBookId).first;
+  if (reader == nullptr) {
+    return build_404(request, humanReadableBookId);
+  }
+
+  std::string content;
+  std::string mimeType = "text";
+
+  if (meta_name == "title") {
+    content = reader->getTitle();
+  } else if (meta_name == "description") {
+    content = reader->getDescription();
+  } else if (meta_name == "language") {
+    content = reader->getLanguage();
+  } else if (meta_name == "name") {
+    content = reader->getName();
+  } else if (meta_name == "tags") {
+    content = reader->getTags();
+  } else if (meta_name == "date") {
+    content = reader->getDate();
+  } else if (meta_name == "creator") {
+    content = reader->getCreator();
+  } else if (meta_name == "publisher") {
+    content = reader->getPublisher();
+  } else if (meta_name == "favicon") {
+    reader->getFavicon(content, mimeType);
+  } else {
+    return build_404(request, humanReadableBookId);
+  }
+
+  return build_response(content.data(), content.size(), "", mimeType, false, true);
 }
 
 static struct MHD_Response* handle_suggest(RequestContext* request)
@@ -400,7 +455,7 @@ static struct MHD_Response* handle_suggest(RequestContext* request)
     humanReadableBookId = request->get_argument("content");
     term = request->get_argument("term");
   } catch (const std::out_of_range&) {
-    return build_homepage(request);
+    return build_404(request, "");
   }
 
   if (isVerbose.load()) {
@@ -509,8 +564,14 @@ static struct MHD_Response* handle_search(RequestContext* request)
     auto variantsItr = variants.begin();
 
     while (patternCorrespondingUrl.empty() && variantsItr != variants.end()) {
-      reader->getPageUrlFromTitle(*variantsItr, patternCorrespondingUrl);
-      variantsItr++;
+      try {
+        auto entry = reader->getEntryFromTitle(*variantsItr);
+        entry = entry.getFinalEntry();
+        patternCorrespondingUrl = entry.getPath();
+        break;
+      } catch(kiwix::NoEntry& e) {
+        variantsItr++;
+      }
     }
 
     /* If article found then redirect directly to it */
@@ -579,18 +640,101 @@ static struct MHD_Response* handle_random(RequestContext* request)
   try {
     humanReadableBookId = request->get_argument("content");
   } catch (const std::out_of_range&) {
-    return build_homepage(request);
+    return build_404(request, "");
   }
 
   auto reader = get_from_humanReadableBookId(humanReadableBookId).first;
   if (reader == nullptr) {
-    return build_homepage(request);
+    return build_404(request, "");
   }
 
-  std::string randomUrl = reader->getRandomPageUrl();
-  httpRedirection
-      = rootLocation + "/" + humanReadableBookId + "/" + kiwix::urlEncode(randomUrl);
-  return build_response("", 0, httpRedirection, "", false, false);
+  try {
+    auto entry = reader->getRandomPage();
+    entry = entry.getFinalEntry();
+    httpRedirection
+      = rootLocation + "/" + humanReadableBookId + "/" + kiwix::urlEncode(entry.getPath());
+    return build_response("", 0, httpRedirection, "", false, false);
+  } catch(kiwix::NoEntry& e) {
+    return build_404(request, humanReadableBookId);
+  }
+}
+
+static struct MHD_Response* handle_catalog(RequestContext* request)
+{
+  if (isVerbose.load()) {
+    printf("** running handle_catalog");
+  }
+
+  std::string host;
+  std::string url;
+  try {
+    host = request->get_header("Host");
+    url  = request->get_url_part(1);
+  } catch (const std::out_of_range&) {
+    return build_404(request, "");
+  }
+
+  std::string content;
+  std::string mimeType;
+
+  if (url == "searchdescription.xml") {
+    content = catalogOpenSearchDescription;
+    mimeType = "application/opensearchdescription+xml";
+  } else {
+    zim::Uuid uuid;
+    kiwix::OPDSDumper opdsDumper;
+    opdsDumper.setRootLocation(rootLocation);
+    opdsDumper.setSearchDescriptionUrl("catalog/searchdescription.xml");
+    opdsDumper.setId(kiwix::to_string(uuid));
+    opdsDumper.setLibrary(&library);
+    mimeType = "application/atom+xml;profile=opds-catalog;kind=acquisition; charset=utf-8";
+    std::vector<std::string> bookIdsToDump;
+    if (url == "root.xml") {
+      opdsDumper.setTitle("All zims");
+      uuid = zim::Uuid::generate(host);
+
+      bookIdsToDump = library.listBooksIds(
+        kiwix::VALID|kiwix::LOCAL|kiwix::REMOTE);
+    } else if (url == "search") {
+      std::string query;
+      std::string language;
+      size_t count(10);
+      size_t startIndex(0);
+      try {
+        query = request->get_argument("q");
+      } catch (const std::out_of_range&) {}
+      try {
+        language = request->get_argument("lang");
+      } catch (const std::out_of_range&) {}
+      try {
+        count = stoul(request->get_argument("count"));
+      } catch (...) {}
+      try {
+        startIndex = stoul(request->get_argument("start"));
+      } catch (...) {}
+      opdsDumper.setTitle("Search result for " + query);
+      uuid = zim::Uuid::generate();
+      bookIdsToDump = library.listBooksIds(
+        kiwix::VALID|kiwix::LOCAL|kiwix::REMOTE,
+        kiwix::UNSORTED,
+        query,
+        language);
+      auto totalResults = bookIdsToDump.size();
+      bookIdsToDump.erase(bookIdsToDump.begin(), bookIdsToDump.begin()+startIndex);
+      if (count>0 && bookIdsToDump.size() > count) {
+        bookIdsToDump.resize(count);
+      }
+      opdsDumper.setOpenSearchInfo(totalResults, startIndex, bookIdsToDump.size());
+    } else {
+      return build_404(request, "");
+    }
+
+    content = opdsDumper.dumpOPDSFeed(bookIdsToDump);
+  }
+
+  bool deflated = request->can_compress() && compress_content(content, mimeType);
+  return build_response(
+      content.data(), content.size(), "", mimeType, deflated, false);
 }
 
 static struct MHD_Response* handle_content(RequestContext* request)
@@ -603,8 +747,7 @@ static struct MHD_Response* handle_content(RequestContext* request)
   std::string content;
   std::string mimeType;
 
-  bool found = false;
-  zim::Article article;
+  kiwix::Entry entry;
 
   std::string humanReadableBookId;
   try {
@@ -615,31 +758,19 @@ static struct MHD_Response* handle_content(RequestContext* request)
 
   auto reader = get_from_humanReadableBookId(humanReadableBookId).first;
   if (reader == nullptr) {
-    return build_homepage(request);
+    if (humanReadableBookId.size() == 0){
+        return build_homepage(request);
+    } else {
+        return build_404(request, "");
+    }
   }
 
   auto urlStr = request->get_url().substr(humanReadableBookId.size()+1);
 
   try {
-    found = reader->getArticleObjectByDecodedUrl(urlStr, article);
-
-    if (found) {
-      /* If redirect */
-      unsigned int loopCounter = 0;
-      while (article.isRedirect() && ++loopCounter < 42) {
-        article = article.getRedirectArticle();
-      }
-
-      /* To many loop */
-      if (loopCounter == 42)
-        found = false;
-    }
-  } catch (const std::exception& e) {
-    std::cerr << e.what() << std::endl;
-    found = false;
-  }
-
-  if (!found) {
+    entry = reader->getEntryFromPath(urlStr);
+    entry = entry.getFinalEntry();
+  } catch(kiwix::NoEntry& e) {
     if (isVerbose.load())
       printf("Failed to find %s\n", urlStr.c_str());
 
@@ -647,7 +778,7 @@ static struct MHD_Response* handle_content(RequestContext* request)
   }
 
   try {
-    mimeType = article.getMimeType();
+    mimeType = entry.getMimetype();
   } catch (exception& e) {
     mimeType = "application/octet-stream";
   }
@@ -660,14 +791,13 @@ static struct MHD_Response* handle_content(RequestContext* request)
   if (mimeType.find("text/") != string::npos
       || mimeType.find("application/javascript") != string::npos
       || mimeType.find("application/json") != string::npos) {
-    zim::Blob raw_content = article.getData();
+    zim::Blob raw_content = entry.getBlob();
     content = string(raw_content.data(), raw_content.size());
 
     /* Special rewrite URL in case of ZIM file use intern *asbolute* url like
      * /A/Kiwix */
     if (mimeType.find("text/html") != string::npos) {
-      baseUrl = "/" + std::string(1, article.getNamespace()) + "/"
-                + article.getUrl();
+      baseUrl = "/" + entry.getPath();
       pthread_mutex_lock(&regexLock);
       content = replaceRegex(content,
                              "$1$2" + rootLocation + "/" + humanReadableBookId + "/$3/",
@@ -696,12 +826,12 @@ static struct MHD_Response* handle_content(RequestContext* request)
   } else {
     int range_len;
     if (request->get_range().second == -1) {
-       range_len = article.getArticleSize() - request->get_range().first;
+       range_len = entry.getSize() - request->get_range().first;
     } else {
        range_len = request->get_range().second - request->get_range().first;
     }
-    return build_callback_response_from_article(
-      article,
+    return build_callback_response_from_entry(
+      entry,
       request->get_range().first,
       range_len,
       mimeType);
@@ -746,10 +876,14 @@ static int accessHandlerCallback(void* cls,
   request.httpResponseCode = request.has_range() ? MHD_HTTP_PARTIAL_CONTENT : MHD_HTTP_OK;
 
   if (! request.is_valid_url()) {
-    response = build_homepage(&request);
+    response = build_404(&request, "");
   } else {
     if (startswith(request.get_url(), "/skin/")) {
       response = handle_skin(&request);
+    } else if (startswith(request.get_url(), "/catalog")) {
+      response = handle_catalog(&request);
+    } else if (request.get_url() == "/meta") {
+      response = handle_meta(&request);
     } else if (request.get_url() == "/search") {
       response = handle_search(&request);
     } else if (request.get_url() == "/suggest") {
@@ -784,12 +918,11 @@ int main(int argc, char** argv)
   string rootPath;
   string interface;
   int serverPort = 80;
-  int daemonFlag = false;
+  int daemonFlag [[gnu::unused]] = false;
   int libraryFlag = false;
   string PPIDString;
   unsigned int PPID = 0;
   unsigned int nb_threads = std::thread::hardware_concurrency();
-  kiwix::Manager libraryManager;
 
   static struct option long_options[]
       = {{"daemon", no_argument, 0, 'd'},
@@ -860,15 +993,15 @@ int main(int argc, char** argv)
           break;
       }
     } else {
-      if (optind <= argc) {
+      if (optind < argc) {
         if (libraryFlag) {
           libraryPath = argv[optind++];
         } else {
           while (optind < argc)
             zimPathes.push_back(std::string(argv[optind++]));
         }
-        break;
       }
+      break;
     }
   }
 
@@ -897,6 +1030,7 @@ int main(int argc, char** argv)
   }
 
   /* Setup the library manager and get the list of books */
+  kiwix::Manager manager(&library);
   if (libraryFlag) {
     vector<string> libraryPaths = kiwix::split(libraryPath, ";");
     vector<string>::iterator itr;
@@ -910,7 +1044,7 @@ int main(int argc, char** argv)
               = isRelativePath(*itr)
                     ? computeAbsolutePath(getCurrentDirectory(), *itr)
                     : *itr;
-          retVal = libraryManager.readFile(libraryPath, true);
+          retVal = manager.readFile(libraryPath, true);
         } catch (...) {
           retVal = false;
         }
@@ -924,74 +1058,69 @@ int main(int argc, char** argv)
     }
 
     /* Check if the library is not empty (or only remote books)*/
-    if (libraryManager.getBookCount(true, false) == 0) {
+    if (library.getBookCount(true, false) == 0) {
       cerr << "The XML library file '" << libraryPath
            << "' is empty (or has only remote books)." << endl;
     }
   } else {
     std::vector<std::string>::iterator it;
     for (it = zimPathes.begin(); it != zimPathes.end(); it++) {
-      if (!libraryManager.addBookFromPath(*it, *it, "", false)) {
+      if (!manager.addBookFromPath(*it, *it, "", false)) {
         cerr << "Unable to add the ZIM file '" << *it
              << "' to the internal library." << endl;
         exit(1);
       }
     }
     if (!indexPath.empty()) {
-      libraryManager.setBookIndex(libraryManager.getBooksIds()[0], indexPath);
+      if (isRelativePath(indexPath)) {
+        indexPath = computeAbsolutePath(indexPath, getCurrentDirectory());
+      }
+      library.getBookById(library.getBooksIds()[0]).setIndexPath(indexPath);
     }
   }
 
   /* Instance the readers and searcher and build the corresponding maps */
-  vector<string> booksIds = libraryManager.getBooksIds();
-  vector<string>::iterator itr;
-  kiwix::Book currentBook;
+  vector<string> booksIds = library.listBooksIds(kiwix::LOCAL);
   globalSearcher = new kiwix::Searcher();
   globalSearcher->setProtocolPrefix(rootLocation + "/");
   globalSearcher->setSearchProtocolPrefix(rootLocation + "/" + "search?");
-  for (itr = booksIds.begin(); itr != booksIds.end(); ++itr) {
-    bool zimFileOk = false;
-    libraryManager.getBookById(*itr, currentBook);
-    std::string zimPath = currentBook.pathAbsolute;
+  for (auto& bookId: booksIds) {
+    auto& currentBook = library.getBookById(bookId);
+    auto zimPath = currentBook.getPath();
+    auto indexPath = currentBook.getIndexPath();
 
-    if (!zimPath.empty()) {
-      indexPath = currentBook.indexPathAbsolute;
+    /* Instanciate the ZIM file handler */
+    kiwix::Reader* reader = NULL;
+    try {
+      reader = new kiwix::Reader(zimPath);
+    } catch (...) {
+      cerr << "Unable to open the ZIM file '" << zimPath << "'." << endl;
+      continue;
+    }
 
-      /* Instanciate the ZIM file handler */
-      kiwix::Reader* reader = NULL;
+    auto humanReadableId = currentBook.getHumanReadableIdFromPath();
+    readers[humanReadableId] = reader;
+
+    if (reader->hasFulltextIndex()) {
+      kiwix::Searcher* searcher = new kiwix::Searcher(humanReadableId);
+      searcher->setProtocolPrefix(rootLocation + "/");
+      searcher->setSearchProtocolPrefix(rootLocation + "/" + "search?");
+      searcher->add_reader(reader, humanReadableId);
+      globalSearcher->add_reader(reader, humanReadableId);
+      searchers[humanReadableId] = searcher;
+    } else if ( !indexPath.empty() ) {
       try {
-        reader = new kiwix::Reader(zimPath);
-        zimFileOk = true;
+        kiwix::Searcher* searcher = new kiwix::Searcher(indexPath, reader, humanReadableId);
+        searcher->setProtocolPrefix(rootLocation + "/");
+        searcher->setSearchProtocolPrefix(rootLocation + "/" + "search?");
+        searchers[humanReadableId] = searcher;
       } catch (...) {
-        cerr << "Unable to open the ZIM file '" << zimPath << "'." << endl;
-      }
-
-      if (zimFileOk) {
-        string humanReadableId = currentBook.getHumanReadableIdFromPath();
-        readers[humanReadableId] = reader;
-
-        if ( reader->hasFulltextIndex()) {
-          kiwix::Searcher* searcher = new kiwix::Searcher(humanReadableId);
-          searcher->setProtocolPrefix(rootLocation + "/");
-          searcher->setSearchProtocolPrefix(rootLocation + "/" + "search?");
-          searcher->add_reader(reader, humanReadableId);
-          globalSearcher->add_reader(reader, humanReadableId);
-          searchers[humanReadableId] = searcher;
-        } else if ( !indexPath.empty() ) {
-          try {
-            kiwix::Searcher* searcher = new kiwix::Searcher(indexPath, reader, humanReadableId);
-            searcher->setProtocolPrefix(rootLocation + "/");
-            searcher->setSearchProtocolPrefix(rootLocation + "/" + "search?");
-            searchers[humanReadableId] = searcher;
-          } catch (...) {
-            cerr << "Unable to open the search index '" << indexPath << "'."
-                 << endl;
-            searchers[humanReadableId] = nullptr;
-          }
-        } else {
-            searchers[humanReadableId] = nullptr;
-        }
+        cerr << "Unable to open the search index '" << indexPath << "'."
+             << endl;
+        searchers[humanReadableId] = nullptr;
       }
+    } else {
+        searchers[humanReadableId] = nullptr;
     }
   }
 
@@ -999,24 +1128,25 @@ int main(int argc, char** argv)
   string welcomeBooksHtml
       = ""
         "<div class='book__list'>";
-  for (itr = booksIds.begin(); itr != booksIds.end(); ++itr) {
-    libraryManager.getBookById(*itr, currentBook);
+  for (auto& bookId: booksIds) {
+    auto& currentBook = library.getBookById(bookId);
 
-    if (!currentBook.path.empty()
+    if (!currentBook.getPath().empty()
         && readers.find(currentBook.getHumanReadableIdFromPath())
                != readers.end()) {
       welcomeBooksHtml += ""
 "<a href='" + rootLocation + "/" + currentBook.getHumanReadableIdFromPath() + "/'>"
     "<div class='book'>"
-        "<div class='book__background' style='background-image: url(data:" + currentBook.faviconMimeType+ ";base64," + currentBook.favicon + ");'>"
-            "<div class='book__title' title='" + currentBook.title + "'>" + currentBook.title + "</div>"
-            "<div class='book__description' title='" + currentBook.description + "'>" + currentBook.description + "</div>"
+        "<div class='book__background' style=\"background-image: url('/meta?content="
+             + currentBook.getHumanReadableIdFromPath() + "&name=favicon');\">"
+            "<div class='book__title' title='" + currentBook.getTitle() + "'>" + currentBook.getTitle() + "</div>"
+            "<div class='book__description' title='" + currentBook.getDescription() + "'>" + currentBook.getDescription() + "</div>"
             "<div class='book__info'>"
-                "" + kiwix::beautifyInteger(atoi(currentBook.articleCount.c_str())) + " articles, " + kiwix::beautifyInteger(atoi(currentBook.mediaCount.c_str())) + " medias"
+                "" + kiwix::beautifyInteger(currentBook.getArticleCount()) + " articles, " + kiwix::beautifyInteger(currentBook.getMediaCount()) + " medias"
             "</div>"
         "</div>"
     "</div>"
-"</a>";
+"</a>\n";
     }
   }
   welcomeBooksHtml += ""
@@ -1029,6 +1159,11 @@ int main(int argc, char** argv)
 
   introduceTaskbar(welcomeHTML, "");
 
+  /* Compute the OpenSearch description */
+  catalogOpenSearchDescription = RESOURCE::opensearchdescription_xml;
+  catalogOpenSearchDescription = replaceRegex(catalogOpenSearchDescription, rootLocation, "__ROOT_LOCATION__");
+
+
 #ifndef _WIN32
   /* Fork if necessary */
   if (daemonFlag) {

src/server/meson.build

@@ -4,4 +4,5 @@ sources += server_resources
 
 executable('kiwix-serve', sources,
   dependencies:all_deps,
-  install:true)
+  install:true,
+  install_rpath: join_paths(get_option('prefix'), get_option('libdir')))

src/server/request_context.cpp

@@ -23,13 +23,13 @@
 #include <string.h>
 #include <stdexcept>
 #include <sstream>
+#include <cstdio>
 
 RequestContext::RequestContext(struct MHD_Connection* connection,
                                std::string rootLocation,
                                const std::string& _url,
                                const std::string& method,
                                const std::string& version) :
-  connection(connection),
   full_url(_url),
   url(_url),
   valid_url(true),
@@ -196,16 +196,6 @@ std::string RequestContext::get_argument(const std::string& name) {
   return arguments.at(name);
 }
 
-template<>
-unsigned int RequestContext::get_argument(const std::string& name) {
-  return std::stoi(arguments.at(name).c_str());
-}
-
-template<>
-float RequestContext::get_argument(const std::string& name) {
-  return std::stof(arguments.at(name).c_str());
-}
-
 std::string RequestContext::get_header(const std::string& name) {
   return headers.at(name);
 }

src/server/request_context.h

@@ -23,6 +23,7 @@
 #define REQUEST_CONTEXT_H
 
 #include <string>
+#include <sstream>
 #include <map>
 #include <stdexcept>
 
@@ -62,7 +63,13 @@ class RequestContext {
 
     std::string get_header(const std::string& name);
     template<typename T=std::string>
-    T get_argument(const std::string& name);
+    T get_argument(const std::string& name) {
+        std::istringstream stream(arguments.at(name));
+        T v;
+        stream >> v;
+        return v;
+    }
+
 
     RequestMethod get_method();
     std::string get_url();
@@ -78,7 +85,6 @@ class RequestContext {
     int httpResponseCode;
 
   private:
-    struct MHD_Connection* connection;
     std::string full_url;
     std::string url;
     bool valid_url;
@@ -96,5 +102,7 @@ class RequestContext {
     static int fill_argument(void *, enum MHD_ValueKind, const char*, const char*);
 };
 
+template<> std::string RequestContext::get_argument(const std::string& name);
+
 
 #endif //REQUEST_CONTEXT_H

static/server/global_taskbar.html.part

@@ -4,7 +4,7 @@
       <div class="kiwix_searchform">
         <form class="kiwixsearch" method="GET" action="__ROOT_LOCATION__/search" id="kiwixsearchform">
           <input autocomplete="off" class="ui-autocomplete-input" id="kiwixsearchbox" name="pattern" type="text">
-          <input type="submit" value="Search">
+          <input type="submit" value="&#x1f50d;">
         </form>
       </div>
     </div>

static/server/opensearchdescription.xml

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
+  <ShortName>Zim catalog search</ShortName>
+  <Description>Search zim files in the catalog.</Description>
+  <Url type="application/atom+xml;profile=opds-catalog" 
+       xmlns:atom="http://www.w3.org/2005/Atom"
+       indexOffset="0"
+       template="/__ROOT_LOCATION__/catalog/search?q={searchTerms}&lang={language}&count={count}&start={startIndex}"/>
+</OpenSearchDescription>

static/server/resources_list.txt

@@ -22,3 +22,4 @@ include.html.part
 taskbar.css
 taskbar.html.part
 global_taskbar.html.part
+opensearchdescription.xml

static/server/taskbar.html.part

@@ -2,15 +2,15 @@
   <span id="kiwixtoolbar" class="ui-widget-header">
     <div class="kiwix_centered">
       <div class="kiwix_button_wrapper">
-        <a id="kiwix_serve_taskbar_library_button" href="__ROOT_LOCATION__/"><button>Library</button></a>
-        <a id="kiwix_serve_taskbar_home_button" href="__ROOT_LOCATION__/__CONTENT__/"><button>Home</button></a>
-        <a id="kiwix_serve_taskbar_random_button" href="__ROOT_LOCATION__/random?content=__CONTENT_ESCAPED__"><button>Random</button></a>
+        <a id="kiwix_serve_taskbar_library_button" href="__ROOT_LOCATION__/"><button>&#x1f3e0;</button></a>
+        <a id="kiwix_serve_taskbar_home_button" href="__ROOT_LOCATION__/__CONTENT__/"><button>__ZIM_TITLE__</button></a>
+        <a id="kiwix_serve_taskbar_random_button" href="__ROOT_LOCATION__/random?content=__CONTENT_ESCAPED__"><button>&#x1F3B2;</button></a>
       </div>
       <div class="kiwix_searchform">
         <form class="kiwixsearch" method="GET" action="__ROOT_LOCATION__/search" id="kiwixsearchform">
           <input type="hidden" name="content" value="__CONTENT__" />
           <input autocomplete="off" class="ui-autocomplete-input" id="kiwixsearchbox" name="pattern" type="text">
-          <input type="submit" value="Search">
+          <input type="submit" value="&#x1f50d;">
         </form>
       </div>
     </div>

travis/install_deps.sh

@@ -3,7 +3,7 @@
 set -e
 
 REPO_NAME=${TRAVIS_REPO_SLUG#*/}
-ARCHIVE_NAME=deps_${PLATFORM}_${REPO_NAME}.tar.gz
+ARCHIVE_NAME=deps_${TRAVIS_OS_NAME}_${PLATFORM}_${REPO_NAME}.tar.gz
 
 # Packages.
 case ${PLATFORM} in
@@ -42,6 +42,4 @@ sudo cp ninja /bin
 # Dependencies comming from kiwix-build.
 cd ${HOME}
 wget http://tmp.kiwix.org/ci/${ARCHIVE_NAME}
-mkdir -p BUILD_${PLATFORM}
-cd BUILD_${PLATFORM}
 tar xf ${HOME}/${ARCHIVE_NAME}