Merge pull request #234 from kiwix/new_version

New version 0.7.0

.clang-format

@@ -0,0 +1,12 @@
+BasedOnStyle:  Google
+BinPackArguments: false
+BinPackParameters: false
+BreakBeforeBinaryOperators: All
+BreakBeforeBraces: Linux
+DerivePointerAlignment: false
+SpacesInContainerLiterals: false
+Standard: Cpp11
+
+AllowShortFunctionsOnASingleLine: Inline
+AllowShortIfStatementsOnASingleLine: false
+AllowShortLoopsOnASingleLine: false

.travis.yml

@@ -5,7 +5,8 @@ cache: ccache
 install: travis/install_deps.sh
 script: travis/compile.sh
 env:
-- PLATFORM="native_static"
-- PLATFORM="native_dyn"
-- PLATFORM="win32_static"
-- PLATFORM="win32_dyn"
+  matrix:
+    - PLATFORM="native_static"
+    - PLATFORM="native_dyn"
+    - PLATFORM="win32_static"
+    - PLATFORM="win32_dyn"

Changelog

@@ -1,10 +1,121 @@
+kiwix-tools 0.7.0
+=================
+
+ * Adapt to kiwix-lib new API
+
+kiwix-serve
+-----------
+
+ * Dumps only valid books in the opdsfeed.
+ * Allow the opds feed to be filtered by lang and paginated.
+
+kiwix-manage
+------------
+
+ * Add a download command to download a remote book locally
+ * Book are referenced by bookId not index.
+ * No more indexType option as it is always XAPIAN.
+
+kiwix-tools 0.6.1
+=================
+
+kiwix-serve
+-----------
+
+ * Update README.
+ * Fix crash when `--library` flag is provided without value.
+ * Correctly handle mimetype of file without extension on 64bits.
+ * Minor fixes
+
+kiwix-tools 0.6.0
+=================
+
+ * remove kiwix-install tool.
+
+kiwix-serve
+-----------
+
+ * Improved taskbar #160
+ * Fix global page when using the option `--nosearchbar`
+ * Return 404 for missing resources
+ * Fix compilation for gcc 4.8.
+
+kiwix-manage
+------------
+
+ * Returns proper exit code (not always 0)
+
+
+kiwix-tools 0.5.0
+=================
+
+ * Build kiwix-tools setting the RPATH
+ * Compile without warning.
+
+
+kiwix-serve
+------------
+
+ * Serve metadata information using the "/meta" url.
+ * Serve an OPDS stream of all zim handled by kiwix-serve
+   All informations cannot be infer from the zim file itself,
+   you should use a library.xml to provide needed information (url, ...)
+ * Update kiwix-serve to use the new API of kiwix-lib
+
+kiwix-tools 0.4.0
+=================
+
+ * Use gcc-5 on travis.
+
+kiwix-serve
+-----------
+
+ * Accept zim file with `&` in the name
+ * Do not cache (on client side) global search (as it can depends on the zim
+   files handled)
+ * Fix HTTP byte range handling. (#91)
+ * Fix opening of relative path (#70)
+ * Add a small (and hidden) API to do geo search.
+ * Better request parsing. (#91)
+ * Better handling of invalid request (#116)
+ * Various bug fixes (#146, #150, #153, #165, #168, #165)
+
+kiwix-search
+------------
+
+ * Add an option `--suggestion` to do suggestion search with
+   kiwix-search.(#132)
+
+kiwix-tools 0.3.0
+=================
+
+ * Move to C++11
+
+kiwix-serve
+-----------
+
+ * Add a global taskbar in the welcome page to search in all zims (#49)
+ * Serve the taskbar as css file instead of including it in the html (#68):
+   * Better client caching
+   * The html encoding is now in the first 1024 bytes and firefox correctly
+     detect the encoding
+ * Make kiwix-server multi-threaded (#82)
+ * Correctly return 404 instead of crashing when request inexistant skin file
+   (#83)
+ * Correctly respond to bytes-range requests.(#84)
+ * Directly respond to first request for a url instead of refusing the first
+   connexion
+ * Add support to relative url location. (#86)
+ * Remove caching (on client side) for the welcome page. (#86)
+
+
 kiwix-tools 0.2.0
 =================
 
  * Remove indexer tools
 
 kiwix-serve
-----------
+-----------
 
  * Correctly fix the deflate data we send over http. (#15)
  * Update in the taskbar (or topbar):

README.md

@@ -1,14 +1,14 @@
 Kiwix tools
 ===========
 
-The Kiwix tools gathers kiwix command line tools.
+The Kiwix tools is a collection of Kiwix related command line tools.
 
 Disclaimer
 ----------
 
 This document assumes you have a little knowledge about software
 compilation. If you experience difficulties with the dependencies or
-with the Kiwix libary compilation itself, we recommend to have a look
+with the Kiwix tools compilation itself, we recommend to have a look
 to [kiwix-build](https://github.com/kiwix/kiwix-build).
 
 Preamble
@@ -22,12 +22,12 @@ on recent releases of Ubuntu and Fedora.
 Dependencies
 ------------
 
-The Kiwix tools rely on a few third parts software libraries. They
-are prerequisites to the Kiwix library compilation. Following
+The Kiwix tools rely on a few third party software libraries. They are
+prerequisites to the Kiwix tools compilation. Therefore, following
 libraries need to be available:
 
 * Kiwix lib ....................... https://github.com/kiwix/kiwix-lib
-(no package for now)
+(no package so far)
 * Libmicrohttpd .......... https://www.gnu.org/software/libmicrohttpd/
 (package libmicrohttpd-dev on Ubuntu)
 * CTPP2 ..................................... http://ctpp.havoc.ru/en/
@@ -53,47 +53,26 @@ If you compile manually Libmicrohttpd, you might need to compile it
 without GNU TLS, a bug here will empeach further compilation of Kiwix
 tools otherwise.
 
-Environnement
+Environment
 -------------
 
-The Kiwix library builds using [Meson](http://mesonbuild.com/) version
-0.34 or higher. Meson relies itself on Ninja, pkg-config and few other
-compilation tools.
-
-Install first the few common compilation tools:
-* Automake
-* Libtool
-* Virtualenv
+The Kiwix tools build using [Meson](http://mesonbuild.com/) version
+0.39 or higher. Meson relies itself on Ninja, pkg-config and few other
+compilation tools. Install them first:
+* Meson
+* Ninja
 * Pkg-config
 
-Then install Meson itself:
-```
-virtualenv -p python3 ./ # Create virtualenv
-source bin/activate      # Activate the virtualenv
-pip install meson        # Install Meson
-hash -r                  # Refresh bash paths
-```
-
-Finally download and build Ninja locally:
-```
-git clone git://github.com/ninja-build/ninja.git
-cd ninja
-git checkout release
-./configure.py --bootstrap
-mkdir ../bin                      
-cp ninja ../bin
-cd ..
-```
+These tools should be packaged if you use a cutting edge operating
+system. If not, have a look to the "Troubleshooting" section.
 
 Compilation
 -----------
 
-Once all dependencies are installed, you can compile kiwix-lib with:
+Once all dependencies are installed, you can compile Kiwix tools with:
 ```
-mkdir build
 meson . build
-cd build
-ninja
+ninja -C build
 ```
 
 By default, it will compile dynamic linked libraries. If you want
@@ -105,16 +84,55 @@ Depending of you system, `ninja` may be called `ninja-build`.
 Installation
 ------------
 
-If you want to install the Kiwix tools you just have compiled on your
-system, here we go:
+If you want to install the Kiwix tools, here we go:
 
 ```
-ninja install
+ninja -C build install
+```
+
+You might need to run the command as root (or using 'sudo'), depending
+where you want to install the Kiwix tools. After the installation
+succeeded, you may need to run ldconfig (as root).
+
+Uninstallation
+------------
+
+If you want to uninstall the Kiwix tools:
+
+```
+ninja -C build uninstall
+```
+
+Like for the installation, you might need to run the command as root
+(or using 'sudo').
+
+Troubleshooting
+---------------
+
+If you need to install Meson "manually":
+```
+virtualenv -p python3 ./ # Create virtualenv
+source bin/activate      # Activate the virtualenv
+pip3 install meson       # Install Meson
+hash -r                  # Refresh bash paths
+```
+
+If you need to install Ninja "manually":
+```
+git clone git://github.com/ninja-build/ninja.git
+cd ninja
+git checkout release
+./configure.py --bootstrap
+mkdir ../bin
+cp ninja ../bin
 cd ..
 ```
 
-You might need to run the command as root, depending where you want to
-install the libraries.
+If the compilation still fails, you might need to get a more recent
+version of a dependency than the one packaged by your Linux
+distribution. Try then with a source tarball distributed by the
+problematic upstream project or even directly from the source code
+repository.
 
 License
 -------

format_code.sh

@@ -0,0 +1,15 @@
+#!/usr/bin/bash
+
+files=(
+"src/installer/kiwix-install.cpp"
+"src/searcher/kiwix-search.cpp"
+"src/reader/kiwix-read.cpp"
+"src/manager/kiwix-manage.cpp"
+"src/server/kiwix-serve.cpp"
+)
+
+for i in "${files[@]}"
+do
+  echo $i
+  clang-format -i -style=file $i
+done

meson.build

@@ -1,6 +1,7 @@
 project('kiwix-tools', 'cpp',
-  version : '0.2.0',
-  license : 'GPL')
+  version : '0.7.0',
+  license : 'GPL',
+  default_options: ['c_std=c11', 'cpp_std=c++11', 'werror=true'])
 
 compiler = meson.get_compiler('cpp')
 
@@ -10,26 +11,18 @@ if static_linkage
 endif
 
 thread_dep = dependency('threads')
-kiwixlib_dep = dependency('kiwix', static:static_linkage)
+kiwixlib_dep = dependency('kiwix', version:'>=3.0.0', static:static_linkage)
 microhttpd_dep = dependency('libmicrohttpd', static:static_linkage)
 z_dep = dependency('zlib', static:static_linkage)
 
-# This is a temporary workaround until xapian fix it's xapian-core.pc
-# For a correct dependency checking we should test if uuid is really installed
-# but for a workaround, we assume that in xapian is installed, uuid also.
-if meson.is_cross_build() and host_machine.system() == 'windows'
-    # xapian doesn't use uuid on windows.
-    uuid_dep = declare_dependency()
-else
-    xapian_dep = dependency('xapian-core', required : false)
-    if xapian_dep.found()
-        uuid_dep = declare_dependency(link_args:['-luuid', '-lrt'])
-    else
-        uuid_dep = declare_dependency()
-    endif
-endif
+all_deps = [thread_dep, kiwixlib_dep, microhttpd_dep, z_dep]
 
-all_deps = [thread_dep, kiwixlib_dep, microhttpd_dep, z_dep, uuid_dep]
+if static_linkage
+  librt = compiler.find_library('rt', required:false)
+  if librt.found()
+    all_deps += librt
+  endif
+endif
 
 #subdir('include')
 subdir('static')

src/installer/kiwix-install.cpp

@@ -1,155 +0,0 @@
-/*
- * Copyright 2011-2014 Emmanuel Engelhart <kelson@kiwix.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU  General Public License as published by
- * the Free Software Foundation; either version 3 of the License, or
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
-
-#include <getopt.h>
-#include <kiwix/common/pathTools.h>
-#include <kiwix/reader.h>
-#include <kiwix/manager.h>
-
-enum supportedAction { NONE, ADDCONTENT };
-
-void usage() {
-    cout << "Usage: kiwix-install [--verbose] addcontent ZIM_PATH KIWIX_PATH" << endl;
-    exit(1);
-}
-
-int main(int argc, char **argv) {
-
-  /* Init the variables */
-  const char *contentPath = NULL;
-  const char *kiwixPath = NULL;
-  supportedAction action = NONE;
-  bool verboseFlag = false;
-  int option_index = 0;
-  int c = 0;
-
-  /* Argument parsing */
-  while (42) {
-
-    static struct option long_options[] = {
-      {"verbose", no_argument, 0, 'v'},
-      {0, 0, 0, 0}
-    };
-
-    if (c != -1) {
-      c = getopt_long(argc, argv, "vi", long_options, &option_index);
-
-      switch (c) {
-	case 'v':
-	  verboseFlag = true;
-	  break;
-      }
-    } else {
-      if (optind < argc) {
-	if (action == NONE) {
-	  string actionString = argv[optind++];
-	  if (actionString == "addcontent" || actionString == "ADDCONTENT") {
-	    action = ADDCONTENT;
-	  }
-	} else if (contentPath == NULL) {
-	  contentPath = argv[optind++];
-	} else if (kiwixPath == NULL) {
-	  kiwixPath = argv[optind++];
-	} else {
-	  usage();
-	}
-      } else {
-	break;
-      }
-    }
-  }
-
-  /* Check if we have enough arguments */
-  if (contentPath == NULL || kiwixPath == NULL) {
-    usage();
-  }
-
-  /* Make the action */
-  if (action == ADDCONTENT) {
-
-    /* Check if the content path exists and is readable */
-    if (verboseFlag) { std::cout << "Check if the ZIM file exists..." << std::endl; }
-    if (!fileExists(contentPath)) {
-      cerr << "The content path '" << contentPath << "' does not exist or is not readable." << endl;
-      exit(1);
-    }
-
-    /* Check if this is a ZIM file */
-    try {
-      if (verboseFlag) { std::cout << "Check if the ZIM file is valid..." << std::endl; }
-      kiwix::Reader *reader = new kiwix::Reader(contentPath);
-      delete reader;
-    } catch (exception &e) {
-      cerr << "The content available at '" << contentPath << "' is not a ZIM file." << endl;
-      exit(1);
-    }
-    string contentFilename = getLastPathElement(contentPath);
-
-    /* Check if kiwixPath/kiwix/kiwix.exe exists */
-    if (verboseFlag) { std::cout << "Check if the Kiwix path is valid..." << std::endl; }
-    string kiwixBinaryPath = computeAbsolutePath(kiwixPath, "kiwix/kiwix.exe");
-    if (!fileExists(kiwixBinaryPath)) {
-      kiwixBinaryPath = computeAbsolutePath(kiwixPath, "kiwix/kiwix");
-      if (!fileExists(kiwixBinaryPath)) {
-	cerr << "Unable to find the Kiwix binary at '" << kiwixBinaryPath << "[.exe]'." << endl;
-	exit(1);
-      }
-    }
-
-    /* Check if the directory "data" structure exists */
-    if (verboseFlag) { std::cout << "Check the target data directory structure..." << std::endl; }
-    string dataPath = computeAbsolutePath(kiwixPath, "data/");
-    if (!fileExists(dataPath)) {
-      makeDirectory(dataPath);
-    }
-
-    /* Check if the directory "data/content" structure exists */
-    string dataContentPath = computeAbsolutePath(kiwixPath, "data/content/");
-    if (!fileExists(dataContentPath)) {
-      makeDirectory(dataContentPath);
-    }
-
-    /* Check if the directory "data/library" structure exists */
-    string dataLibraryPath = computeAbsolutePath(kiwixPath, "data/library/");
-    if (!fileExists(dataLibraryPath)) {
-      makeDirectory(dataLibraryPath);
-    }
-
-    /* Copy the file to the data/content directory */
-    if (verboseFlag) { std::cout << "Copy ZIM file to the target directory..." << std::endl; }
-    string newContentPath = computeAbsolutePath(dataContentPath, contentFilename);
-    if (!fileExists(newContentPath) || getFileSize(contentPath) != getFileSize(newContentPath)) {
-      copyFile(contentPath, newContentPath);
-    }
-
-    /* Add the file to the library.xml */
-    if (verboseFlag) { std::cout << "Create the library XML file..." << std::endl; }
-    kiwix::Manager libraryManager;
-    string libraryPath = computeAbsolutePath(dataLibraryPath, contentFilename + ".xml");
-    string bookId = libraryManager.addBookFromPathAndGetId(newContentPath, "../content/" + contentFilename, "", false);
-    if (!bookId.empty()) {
-      libraryManager.setCurrentBookId(bookId);
-      libraryManager.writeFile(libraryPath);
-    } else {
-      cerr << "Unable to build or save library file '" << libraryPath << "'" << endl;
-    }
-  }
-
-  exit(0);
-}

src/installer/meson.build

@@ -1,3 +0,0 @@
-executable('kiwix-install', ['kiwix-install.cpp'],
-  dependencies:all_deps,
-  install:true)

src/man/fr/kiwix-compact.1

@@ -1,23 +0,0 @@
-.TH KIWIX 1 "21 May 2012"
-.SH NAME
-kiwix-compact \- Compacteur d'index pour fichier ZIM
-.SH SYNOPSIS
-.IX Header SYNOPSIS
-kiwix-compact INDEX_PATH
-.SH DESCRIPTION
-.PP
-Compacte les fichiers d'index des ZIM à l'aide de xapian-compact.
-
-.TP
-\fBINDEX_PATH\fR
-Chemin de l'index plein texte à compacter.
-
-.SH SEE ALSO
-kiwix(1) kiwix-index(1) kiwix-manage(1) xapian-compact(1)
-.br
-kiwix-install(1) kiwix-serve(1)
-
-.SH AUTHOR
-Emmanuel Engelhart <kelson@kiwix.org>
-.br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/fr/kiwix-index.1

@@ -1,31 +0,0 @@
-.TH KIWIX 1 "21 May 2012"
-.SH NAME
-kiwix\-index \- Indexeur de fichiers ZIM
-.SH SYNOPSIS
-.IX Header SYNOPSIS
-kiwix\-index [\-\-backend|\-b=xapian|clucene] ZIM_PATH INDEX_PATH
-.SH DESCRIPTION
-.PP
-Génère un index plein texte xapian ou cLucene pour un fichier ZIM.
-
-.TP
-\fB\-\-backend=xapian|clucene\fR
-Séléctionne un moteur d'indexation.
-
-.TP
-\fBZIM_PATH\fR
-Chemin du fichier ZIM à indexer.
-
-.TP
-\fBINDEX_PATH\fR
-Chemin où enregistrer l'index plein texte.
-
-.SH SEE ALSO
-kiwix\-compact(1) kiwix\-manage(1) kiwix\-serve(1)
-.br
-kiwix(1) kiwix\-install(1)
-
-.SH AUTHOR
-Emmanuel Engelhart <kelson@kiwix.org>
-.br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/fr/kiwix-install.1

@@ -1,40 +0,0 @@
-.TH KIWIX 1 "21 May 2012"
-.SH NAME
-kiwix\-install \- Installeur de fichiers ZIM
-.SH SYNOPSIS
-.IX Header SYNOPSIS
-kiwix\-install [\-\-verbose|-v] [\-\-backend|\-b=xapian|clucene] [\-\-buildIndex|\-i] addcontent ZIM_PATH KIWIX_PATH
-.SH DESCRIPTION
-.PP
-Créé une arborescence contenant Kiwix et des données pour redistribution.
-.br
-Si demandé, créé aussi un index plein texte.
-
-.TP
-\fB\-\-verbose\fR
-Active le mode verbeux de la sortie.
-
-.TP
-\fB\-\-backend=xapian|clucene\fR
-Séléctionne un moteur d'indexation.
-
-.TP
-\fB\-\-buildIndex\fR
-Créer un index plein texte pour le fichier ZIM.
-
-.TP
-\fBZIM_PATH\fR
-Chemin du fichier de contenu ZIM.
-
-.TP
-\fBKIWIX_PATH\fR
-Chemin d'accès dossier kiwix/.
-.br
-Le chemin doit contenit le binaire statique de Kiwix.
-
-.SH SEE ALSO
-kiwix(1) kiwix\-index(1) kiwix\-serve(1) kiwix\-compact(1) kiwix\-manage(1)
-.SH AUTHOR
-Emmanuel Engelhart <kelson@kiwix.org>
-.br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/fr/kiwix-manage.1

@@ -57,8 +57,8 @@ Chemin vers l'index plein texte correspondant au fichier ZIM.
 Définit l'adresse URL corresponsant au fichier ZIM pour pouvoir être téléchargé depuis Kiwix.
 
 .SH SEE ALSO
-kiwix(1) kiwix\-index(1) kiwix\-install(1) kiwix\-serve(1) kiwix\-compact(1)
+kiwix(1) kiwix\-install(1) kiwix\-serve(1)
 .SH AUTHOR
 Emmanuel Engelhart <kelson@kiwix.org>
 .br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)
+Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/fr/kiwix-serve.1

@@ -52,10 +52,10 @@ Chemin vers le fichier bibliothèque de Kiwix.
 Le fichier bibliothèque est un fichier XML créé avec \fBkiwix-manage\fB.
 
 .SH SEE ALSO
-kiwix(1) kiwix\-index(1) kiwix\-manage(1)
+kiwix(1) kiwix\-manage(1)
 .br
-kiwix\-install(1) kiwix\-compact(1)
+kiwix\-install(1)
 .SH AUTHOR
 Emmanuel Engelhart <kelson@kiwix.org>
 .br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)
+Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/fr/kiwix.1

@@ -1,52 +0,0 @@
-.TH KIWIX 1 "12 juin 2012"
-.SH NAME
-Kiwix \- Lecteur hors-ligne de fichiers ZIM
-.SH SYNOPSIS
-.B kiwix [-jsconsole] [-articleByUrl] [-articleByTitle] [FILE]
-.SH DESCRIPTION
-.PP
-Kiwix est un lecteur de content au format ZIM.
-.br
-Les fichiers ZIM sont des archives de contenus compressés (généralement HTML)
-.br
-Les fichiers ZIM les plus populaires sont ceux de Wikipédia et Wikileaks.
-
-.TP
-\fB\-jsconsole\fR
-Active la console de debogage JavaScript de Xulrunner.
-\fB\-articleByUrl url\fR
-Ouvre un article en particulier en renseignant son url. FILE doit être donné.
-.TP
-\fB\-articleByTitle title\fR
-Ouvre un article en particulier en renseignant son titre. FILE doit être donné.
-
-.PP
-Fonctionnalités:
- * Lecteur de ZIM natif
- * Moteur de recherche plein texte
- * Signets et Notes
- * Serveur Web pour diffuser les fichiers ZIM sur le réseau
- * Export HTML et PDF
- * Traduits dans de nombreuses langues
- * Suggestions de recherche
- * Indexation des fichiers ZIM
- * Onglets de navigation
- * Bibliothèque de contenus intégrée
-
-.SH SEE ALSO
-kiwix-index(1) kiwix-install(1) kiwix-serve(1)
-.br
-kiwix-compact(1) kiwix-manage(1)
-
-
-.SH TROUBLESHOOTING
-Aller à http://reportabug.kiwix.org pour plus de détails sur comment rapporter un bogue.
-
-.SH AUTHORS
- Emmanuel Engelhart <kelson@kiwix.org>
- Guillaume Duhamel <gduhamel@linterweb.com>
- Fabien Coullon <fcoulon@linterweb.com>
- Renaud Gaudin <rgaudin@gmail.com>
- Wilfredo Rodriguez <wilfredor@kiwix.org>
-.br
- Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/fr/meson.build

@@ -1,5 +1,3 @@
-install_man('kiwix-index.1',
-            'kiwix-install.1',
-            'kiwix-manage.1',
+install_man('kiwix-manage.1',
             'kiwix-serve.1',
-            install_dir:get_option('mandir')+'/fr/man1')
+            install_dir:get_option('mandir')+'/fr/man1')

src/man/kiwix-compact.1

@@ -1,23 +0,0 @@
-.TH KIWIX 1 "21 May 2012"
-.SH NAME
-kiwix-compact \- ZIM index compacter
-.SH SYNOPSIS
-.IX Header SYNOPSIS
-kiwix-compact INDEX_PATH
-.SH DESCRIPTION
-.PP
-Compacts ZIM index files used by Kiwix using xapian-compact
-
-.TP
-\fBINDEX_PATH\fR
-Full-text index path to compact.
-
-.SH SEE ALSO
-kiwix(1) kiwix-index(1) kiwix-manage(1) xapian-compact(1)
-.br
-kiwix-install(1) kiwix-serve(1)
-
-.SH AUTHOR
-Emmanuel Engelhart <kelson@kiwix.org>
-.br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/kiwix-index.1

@@ -1,31 +0,0 @@
-.TH KIWIX 1 "21 May 2012"
-.SH NAME
-kiwix\-index \- Kiwix ZIM Indexer
-.SH SYNOPSIS
-.IX Header SYNOPSIS
-kiwix\-index [\-\-backend|\-b=xapian|clucene] ZIM_PATH INDEX_PATH
-.SH DESCRIPTION
-.PP
-Generates a xapian or clucene full-text index for a ZIM file.
-
-.TP
-\fB\-\-backend=xapian|clucene\fR
-Select an indexing backend.
-
-.TP
-\fBZIM_PATH\fR
-ZIM file path to index.
-
-.TP
-\fBINDEX_PATH\fR
-Path to save the full\-text index to.
-
-.SH SEE ALSO
-kiwix\-compact(1) kiwix\-manage(1) kiwix\-serve(1)
-.br
-kiwix(1) kiwix\-install(1)
-
-.SH AUTHOR
-Emmanuel Engelhart <kelson@kiwix.org>
-.br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/kiwix-install.1

@@ -1,40 +0,0 @@
-.TH KIWIX 1 "21 May 2012"
-.SH NAME
-kiwix\-install \- Kiwix ZIM Installer
-.SH SYNOPSIS
-.IX Header SYNOPSIS
-kiwix\-install [\-\-verbose|-v] [\-\-backend|\-b=xapian|clucene] [\-\-buildIndex|\-i] addcontent ZIM_PATH KIWIX_PATH
-.SH DESCRIPTION
-.PP
-Creates a standalone Kiwix + data hierarchy.
-.br
-If specified, also creates a full\-text index.
-
-.TP
-\fB\-\-verbose\fR
-Enable verbose output.
-
-.TP
-\fB\-\-backend=xapian|clucene\fR
-Select an indexing backend.
-
-.TP
-\fB\-\-buildIndex\fR
-Build an index for the ZIM file.
-
-.TP
-\fBZIM_PATH\fR
-ZIM content file path.
-
-.TP
-\fBKIWIX_PATH\fR
-Path to the kiwix/ folder.
-.br
-The path must contain the kiwix standalone binary.
-
-.SH SEE ALSO
-kiwix(1) kiwix\-index(1) kiwix\-serve(1) kiwix\-compact(1) kiwix\-manage(1)
-.SH AUTHOR
-Emmanuel Engelhart <kelson@kiwix.org>
-.br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/kiwix-manage.1

@@ -55,8 +55,8 @@ Path to full-text index for that ZIM file.
 Set the content location of the ZIM file over the network for in\-kiwix download.
 
 .SH SEE ALSO
-kiwix(1) kiwix\-index(1) kiwix\-install(1) kiwix\-serve(1) kiwix\-compact(1)
+kiwix(1) kiwix\-install(1) kiwix\-serve(1)
 .SH AUTHOR
 Emmanuel Engelhart <kelson@kiwix.org>
 .br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)
+Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/kiwix-serve.1

@@ -13,7 +13,7 @@ Stand\-alone HTTP server for serving ZIM content over the network.
 
 .TP
 \fB\-\-index=INDEX_PATH\fR
-Path to index folder created using \fBkiwix-index\fB.
+Path to index folder created using \fBkiwix-install\fB.
 
 .TP
 \fB\-\-port=PORT\fR
@@ -52,10 +52,10 @@ Kiwix library file path.
 Library is XML file created using \fBkiwix-manage\fB.
 
 .SH SEE ALSO
-kiwix(1) kiwix\-index(1) kiwix\-manage(1)
+kiwix(1) kiwix\-manage(1)
 .br
-kiwix\-install(1) kiwix\-compact(1)
+kiwix\-install(1)
 .SH AUTHOR
 Emmanuel Engelhart <kelson@kiwix.org>
 .br
-Vasudev Kamath <kamathvasudev@gmail.com> (Manual)
+Vasudev Kamath <kamathvasudev@gmail.com> (Manual)

src/man/meson.build

@@ -1,5 +1,3 @@
-install_man('kiwix-index.1',
-            'kiwix-install.1',
-            'kiwix-manage.1',
+install_man('kiwix-manage.1',
             'kiwix-serve.1')
 subdir('fr')

src/manager/kiwix-manage.cpp

@@ -21,53 +21,231 @@
 #include <unistd.h>
 #endif
 #include <getopt.h>
-#include <iostream>
-#include <cstdlib>
 #include <kiwix/common/pathTools.h>
+#include <kiwix/common/stringTools.h>
 #include <kiwix/manager.h>
+#include <kiwix/downloader.h>
+#include <cstdlib>
+#include <iostream>
+
+#include <thread>
+#include <chrono>
 
 using namespace std;
 
-enum supportedAction { NONE, ADD, SHOW, REMOVE };
+enum supportedAction { NONE, ADD, SHOW, REMOVE, DOWNLOAD };
 
-
-void show(kiwix::Library library) {
-    std::vector<kiwix::Book>::iterator itr;
-    unsigned int inc = 1;
-    for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-      std::cout << "#" << inc++
-		<< std::endl << "id:\t\t" << itr->id
-		<< std::endl << "path:\t\t" << itr->path
-		<< std::endl << "indexpath:\t" << itr->indexPath
-		<< std::endl << "url:\t\t" << itr->url
-		<< std::endl << "title:\t\t" << itr->title
-		<< std::endl << "name:\t\t" << itr->name
-		<< std::endl << "tags:\t\t" << itr->tags
-		<< std::endl << "description:\t" << itr->description
-		<< std::endl << "creator:\t" << itr->creator
-		<< std::endl << "date:\t\t" << itr->date
-		<< std::endl << "articleCount:\t" << itr->articleCount
-		<< std::endl << "mediaCount:\t" << itr->mediaCount
-		<< std::endl << "size:\t\t" << itr->size << " KB"
-		<< std::endl << std::endl;
-    }
+void show(kiwix::Library* library)
+{
+  auto booksIds = library->getBooksIds();
+  unsigned int inc = 1;
+  for(auto& id: booksIds) {
+    auto& book = library->getBookById(id);
+    std::cout << "#" << inc++ << std::endl
+              << "id:\t\t" << book.getId() << std::endl
+              << "path:\t\t" << book.getPath() << std::endl
+              << "indexpath:\t" << book.getIndexPath() << std::endl
+              << "url:\t\t" << book.getUrl() << std::endl
+              << "title:\t\t" << book.getTitle() << std::endl
+              << "name:\t\t" << book.getName() << std::endl
+              << "tags:\t\t" << book.getTags() << std::endl
+              << "description:\t" << book.getDescription() << std::endl
+              << "creator:\t" << book.getCreator() << std::endl
+              << "date:\t\t" << book.getDate() << std::endl
+              << "articleCount:\t" << book.getArticleCount() << std::endl
+              << "mediaCount:\t" << book.getMediaCount() << std::endl
+              << "size:\t\t" << book.getSize() << " KB" << std::endl
+              << std::endl;
+  }
 }
 
-void usage() {
-    cerr << "Usage:" << endl;
-    cerr << "\tkiwix-manage LIBRARY_PATH add ZIM_PATH [--zimPathToSave=../content/foobar.zim] [--current] [--indexBackend=xapian] [--indexPath=FULLTEXT_IDX_PATH] [--url=http://...metalink]" << endl;
-    cerr << "\tkiwix-manage LIBRARY_PATH show [CONTENTID1] [CONTENTID2] ... (show everything if no param.)" << endl;
-    cerr << "\tkiwix-manage LIBRARY_PATH remove CONTENTID1 [CONTENTID2]" << endl;
+void usage()
+{
+  cerr << "Usage:" << endl;
+  cerr << "\tkiwix-manage LIBRARY_PATH add ZIM_PATH "
+          "[--zimPathToSave=../content/foobar.zim] [--current] "
+          "[--indexBackend=xapian] [--indexPath=FULLTEXT_IDX_PATH] "
+          "[--url=http://...metalink]"
+       << endl;
+  cerr << "\tkiwix-manage LIBRARY_PATH show [CONTENTID1] [CONTENTID2] ... "
+          "(show everything if no param.)"
+       << endl;
+  cerr << "\tkiwix-manage LIBRARY_PATH remove CONTENTID1 [CONTENTID2]" << endl;
 }
 
-int main(int argc, char **argv) {
 
-  string libraryPath = "";
-  supportedAction action = NONE;
-  string zimPath = "";
-  kiwix::Manager libraryManager;
+bool handle_show(kiwix::Library* library, const std::string& libraryPath,
+                 int argc, char* argv[])
+{
+  show(library);
+  return(0);
+}
+
+bool handle_add(kiwix::Library* library, const std::string& libraryPath,
+                int argc, char* argv[])
+{
+  string zimPath;
+  string zimPathToSave = ".";
+  string indexPath;
+  string url;
+  string origID = "";
   int option_index = 0;
   int c = 0;
+  bool resultCode = 0;
+
+  if (argc > 3) {
+    zimPath = argv[3];
+  }
+
+  /* Options parsing */
+  optind = 2;
+  while (42) {
+    static struct option long_options[]
+        = {{"url", required_argument, 0, 'u'},
+           {"origId", required_argument, 0, 'o'},
+           {"indexPath", required_argument, 0, 'i'},
+           {"zimPathToSave", required_argument, 0, 'z'},
+           {0, 0, 0, 0}};
+
+    c = getopt_long(argc, argv, "cz:u:i:", long_options, &option_index);
+
+    if (c != -1) {
+      switch (c) {
+        case 'u':
+          url = optarg;
+          break;
+
+        case 'o':
+          origID = optarg;
+          break;
+
+        case 'i':
+          indexPath = optarg;
+          break;
+
+        case 'z':
+          zimPathToSave = optarg;
+          break;
+      }
+    } else {
+      break;
+    }
+  }
+
+  if (!zimPath.empty()) {
+    kiwix::Manager manager(library);
+    zimPathToSave = zimPathToSave == "." ? zimPath : zimPathToSave;
+    string bookId = manager.addBookFromPathAndGetId(
+        zimPath, zimPathToSave, url, false);
+     if (!bookId.empty()) {
+      /* Save the index infos if necessary */
+      if (!indexPath.empty()) {
+        if (isRelativePath(indexPath)) {
+          indexPath = computeAbsolutePath(indexPath, getCurrentDirectory());
+        }
+        library->getBookById(bookId).setIndexPath(indexPath);
+      }
+     } else {
+      cerr << "Unable to build or save library file '" << libraryPath << "'"
+           << endl;
+      resultCode = 1;
+    }
+  } else {
+    std::cerr << "Invalid zim file path" << std::endl;
+    resultCode = 1;
+  }
+
+  return(resultCode);
+}
+
+bool handle_remove(kiwix::Library* library, const std::string& libraryPath,
+                   int argc, char* argv[])
+{
+  std::string bookId;
+  const unsigned int totalBookCount = library->getBookCount(true, true);
+  bool exitCode = 0;
+
+  if (argc > 3) {
+    bookId = argv[3];
+  }
+
+  if (!library->removeBookById(bookId)) {
+    if (totalBookCount > 0) {
+      std::cerr
+          << "Invalid book id." << std::endl;
+      exitCode = 1;
+    } else {
+      std::cerr
+          << "Invalid book id. Library is empty, no book to delete."
+          << std::endl;
+      exitCode = 1;
+    }
+  }
+
+  return(exitCode);
+}
+
+bool handle_download(kiwix::Library* library, const std::string& libraryPath,
+                     int argc, char* argv[])
+{
+  std::string bookId;
+  bool exitCode = false;
+
+  if (argc > 3) {
+     bookId = argv[3];
+  }
+
+  auto& book = library->getBookById(bookId);
+  auto did = book.getDownloadId();
+  kiwix::Downloader downloader;
+  kiwix::Download* download = nullptr;
+  if (!did.empty()) {
+    std::cout << "try resume " << did << std::endl;
+    try {
+      download = downloader.getDownload(did);
+    } catch(...) {}
+  }
+  if (nullptr == download || download->getStatus() == kiwix::Download::K_UNKNOWN) {
+    download = downloader.startDownload(book.getUrl());
+    book.setDownloadId(download->getDid());
+  }
+  int step = 60*5;
+  while (step--) {
+    download->updateStatus();
+    if (download->getStatus() == kiwix::Download::K_COMPLETE) {
+      auto followingId = download->getFollowedBy();
+      if (followingId.empty()) {
+        book.setPath(download->getPath());
+        book.setDownloadId("");
+        std::cout << "File downloaded to " << book.getPath() << std::endl;
+        break;
+      } else {
+        download = downloader.getDownload(followingId);
+      }
+    } else if (download->getStatus() == kiwix::Download::K_ACTIVE) {
+      std::cout << download->getDid() << " : "
+                << kiwix::beautifyFileSize(download->getCompletedLength()) << "/"
+                << kiwix::beautifyFileSize(download->getTotalLength())
+                << " (" << kiwix::beautifyFileSize(download->getDownloadSpeed()) << "/s) "
+                << " [" << kiwix::beautifyFileSize(download->getVerifiedLength()) << "]"
+                << "[" << step << "]    \n" << std::flush;
+    } else if (download->getStatus() == kiwix::Download::K_ERROR) {
+      std::cout << "File Error" << std::endl;
+      exitCode = true;
+      break;
+    }
+    std::this_thread::sleep_for(std::chrono::seconds(1));
+  }
+
+  downloader.close();
+  return exitCode;
+}
+
+int main(int argc, char** argv)
+{
+  string libraryPath = "";
+  supportedAction action = NONE;
+  kiwix::Library library;
 
   /* Argument parsing */
   if (argc > 2) {
@@ -80,6 +258,8 @@ int main(int argc, char **argv) {
       action = SHOW;
     else if (actionString == "remove" || actionString == "delete")
       action = REMOVE;
+    else if (actionString == "download")
+      action = DOWNLOAD;
   }
 
   /* Print usage)) if necessary */
@@ -89,123 +269,35 @@ int main(int argc, char **argv) {
   }
 
   /* Try to read the file */
-  libraryPath = isRelativePath(libraryPath) ?
-    computeAbsolutePath(getCurrentDirectory(), libraryPath) :
-    libraryPath;
-  libraryManager.readFile(libraryPath, false);
+  libraryPath = isRelativePath(libraryPath)
+                    ? computeAbsolutePath(getCurrentDirectory(), libraryPath)
+                    : libraryPath;
+  kiwix::Manager manager(&library);
+  manager.readFile(libraryPath, false);
 
   /* SHOW */
-  if (action == SHOW) {
-    show(libraryManager.cloneLibrary());
-  } else if (action == ADD) {
-    string zimPath;
-    string zimPathToSave = ".";
-    string indexPath;
-    kiwix::supportedIndexType indexBackend = kiwix::XAPIAN;
-    string url;
-    string origID="";
-    bool setCurrent = false;
-
-    if (argc>3) {
-      zimPath = argv[3];
-    }
-
-    /* Options parsing */
-    optind = 2;
-    while (42) {
-
-      static struct option long_options[] = {
-	{"url", required_argument, 0, 'u'},
-	{"origId", required_argument, 0, 'o'},
-	{"indexPath", required_argument, 0, 'i'},
-	{"indexBackend", required_argument, 0, 'b'},
-	{"zimPathToSave", required_argument, 0, 'z'},
-	{"current", no_argument, 0, 'c'},
-	{0, 0, 0, 0}
-      };
-      
-      c = getopt_long(argc, argv, "cz:u:i:b:", long_options, &option_index);
-      
-      if (c != -1) {
-
-	switch (c) {
-        case 'u':
-	  url = optarg;
-	  break;
-
-        case 'o':
-	  origID = optarg;
-	  break;
-
-        case 'c':
-	  setCurrent = true;
-	  break;
-
-	case 'i':
-	  indexPath = optarg;
-	  break;
-
-	case 'b':
-	  if (!strcmp(optarg, "xapian")) {
-	    indexBackend = kiwix::XAPIAN;
-	  } else {
-	    usage();
-	  }
-	  break;
-
-	case 'z':
-	  zimPathToSave = optarg;
-	  break;
-
-	}
-      } else {
-	break;
-      }
-    }
-
-    if (!zimPath.empty()) {
-      zimPathToSave = zimPathToSave == "." ? zimPath : zimPathToSave;
-      string bookId = libraryManager.addBookFromPathAndGetId(zimPath, zimPathToSave, url, false);
-
-      if (!bookId.empty()) {
-
-	if (setCurrent)
-	  libraryManager.setCurrentBookId(bookId);
-
-	/* Save the index infos if necessary */
-	if (!indexPath.empty())
-	  libraryManager.setBookIndex(bookId, indexPath, indexBackend);
-
-      } else {
-	cerr << "Unable to build or save library file '" << libraryPath << "'" << endl;
-      }
-    } else {
-      std::cerr << "Invalid zim file path" << std::endl;
-    }
-
-  } else if (action == REMOVE) {
-    unsigned int bookIndex = 0;
-    const unsigned int totalBookCount = libraryManager.getBookCount(true, true);
-
-    if (argc>3) {
-      bookIndex = atoi(argv[3]);
-    }
-
-    if (bookIndex > 0 && bookIndex <= totalBookCount) {
-      libraryManager.removeBookByIndex(bookIndex - 1);
-    } else {
-      if (totalBookCount > 0) {
-	std::cerr << "Invalid book index number. Please give a number between 1 and " << totalBookCount << std::endl;
-      } else {
-	std::cerr << "Invalid book index number. Library is empty, no book to delete." << std::endl;
-      }
-    }
+  bool exitCode = 0;
+  switch (action) {
+    case SHOW:
+      exitCode = handle_show(&library, libraryPath, argc, argv);
+      break;
+    case ADD:
+      exitCode = handle_add(&library, libraryPath, argc, argv);
+      break;
+    case REMOVE:
+      exitCode = handle_remove(&library, libraryPath, argc, argv);
+      break;
+    case DOWNLOAD:
+      exitCode = handle_download(&library, libraryPath, argc, argv);
+      break;
+    case NONE:
+      break;
   }
 
   /* Rewrite the library file */
-  if (action == REMOVE || action == ADD) {
-    libraryManager.writeFile(libraryPath);
+  if (action == REMOVE || action == ADD || action == DOWNLOAD) {
+    library.writeToFile(libraryPath);
   }
 
-  exit(0);
+  exit(exitCode);
 }

src/manager/meson.build

@@ -1,3 +1,4 @@
 executable('kiwix-manage', ['kiwix-manage.cpp'],
   dependencies:all_deps,
-  install:true)
+  install:true,
+  install_rpath: join_paths(get_option('prefix'), get_option('libdir')))

src/meson.build

@@ -1,5 +1,4 @@
 
-subdir('installer')
 subdir('manager')
 subdir('reader')
 subdir('searcher')

src/reader/kiwix-read.cpp

@@ -18,59 +18,53 @@
  */
 
 #include <getopt.h>
-#include <unistd.h>
-#include <string>
-#include <map>
-#include <kiwix/common/tree.h>
 #include <kiwix/reader.h>
+#include <unistd.h>
+#include <map>
+#include <string>
 
-void usage() {
-    cout << "Usage: kiwix-read --suggest=<PATTERN> ZIM_FILE_PATH" << endl;
-    exit(1);
+void usage()
+{
+  cout << "Usage: kiwix-read --suggest=<PATTERN> ZIM_FILE_PATH" << endl;
+  exit(1);
 }
 
-void updateSuggestionTree(tree< pair<string, unsigned> > &suggestionTree, string suggestion) {
-  return;
-}
-
-int main(int argc, char **argv) {
-
+int main(int argc, char** argv)
+{
   /* Init the variables */
-  const char *filePath = NULL;
-  const char *pattern = NULL;
+  const char* filePath = NULL;
+  const char* pattern = NULL;
   int option_index = 0;
   int c = 0;
 
-  kiwix::Reader *reader = NULL;
+  kiwix::Reader* reader = NULL;
 
   /* Argument parsing */
   while (42) {
+    static struct option long_options[]
+        = {{"verbose", no_argument, 0, 'v'},
+           {"suggest", required_argument, 0, 's'},
+           {0, 0, 0, 0}};
 
-    static struct option long_options[] = {
-      {"verbose", no_argument, 0, 'v'},
-      {"suggest", required_argument, 0, 's'},
-      {0, 0, 0, 0}
-    };
-    
     if (c != -1) {
       c = getopt_long(argc, argv, "vs:", long_options, &option_index);
 
       switch (c) {
-	case 'v':
-	  break;
-      case 's':
-	pattern = optarg;
-	break;
+        case 'v':
+          break;
+        case 's':
+          pattern = optarg;
+          break;
       }
     } else {
       if (optind < argc) {
-	if (filePath == NULL) {
-	  filePath = argv[optind++];
-	} else {
-	  usage();
-	}
+        if (filePath == NULL) {
+          filePath = argv[optind++];
+        } else {
+          usage();
+        }
       } else {
-	break;
+        break;
       }
     }
   }
@@ -85,30 +79,19 @@ int main(int argc, char **argv) {
 
   /* Start to read an article */
   if (reader != NULL) {
-    string mainPageUrl = reader->getMainPageUrl();
     string content;
     string contentType;
-    unsigned int contentLength = 0;
     string suggestion;
-    
+
     if (pattern != NULL) {
       std::cout << "Searching suggestions for: " << pattern << std::endl;
 
       reader->searchSuggestionsSmart(pattern, 10);
       while (reader->getNextSuggestion(suggestion)) {
-	std::cout << suggestion << std::endl;
+        std::cout << suggestion << std::endl;
       }
     }
 
-    /*
-    if (reader->getContentByUrl(mainPageUrl, content, contentLength, contentType)) {
-      cout << content << endl;
-    }
-    */
-
-    //    tree< pair<string, unsigned> > tree;
-    //updateSuggestionTree(tree, string(suggestion));
-
     delete reader;
   } else {
     cerr << "Unable instanciate the Kiwix reader." << endl;

src/reader/meson.build

@@ -1,3 +1,4 @@
 executable('kiwix-read', ['kiwix-read.cpp'],
   dependencies:all_deps,
-  install:true)
+  install:true,
+  install_rpath: join_paths(get_option('prefix'), get_option('libdir')))

src/searcher/kiwix-search.cpp

@@ -18,57 +18,69 @@
  */
 
 #include <getopt.h>
-#include <unistd.h>
 #include <kiwix/reader.h>
 #include <kiwix/searcher.h>
+#include <unistd.h>
 
-void usage() {
-    cout << "Usage: kiwix-search [--verbose|-v] ZIM_PATH SEARCH" << endl;
-    exit(1);
+
+void usage()
+{
+  cout << "Usage: kiwix-search [OPTIONS] ZIM PATTERN" << endl << endl
+       << "  kiwix-search allows to find articles based on the a fulltext search pattern." << endl << endl
+       << "  ZIM is the full path of the ZIM file. This can also be a disctinct fulltext" << endl
+       << "  index directory (usually distributed with the *.idx extension)." << endl
+       << "  PATTERN is/are word(s) - or part of - to search in the ZIM." << endl << endl
+       << "  -s, --suggestion\tSuggest article titles based on the few letters of the PATTERN instead of making a fulltext search. Work a bit like a completion solution." << endl
+       << "  -v, --verbose\t\tGive details about the search process" << endl;
+  exit(1);
 }
 
-int main(int argc, char **argv) {
-
+int main(int argc, char** argv)
+{
   /* Init the variables */
-  //const char *indexPath = "/home/itamar/.www.kiwix.org/kiwix/43k0i1j4.default/6d2e587b-d586-dc6a-dc6a-e4ef035a1495d15c.index";
-  //const char *indexPath = "/home/itamar/testindex";
-  const char *zimPath = NULL;
-  const char *search = NULL;
+  // const char *indexPath =
+  // "/home/itamar/.www.kiwix.org/kiwix/43k0i1j4.default/6d2e587b-d586-dc6a-dc6a-e4ef035a1495d15c.index";
+  // const char *indexPath = "/home/itamar/testindex";
+  const char* zimPath = NULL;
+  const char* search = NULL;
   bool verboseFlag = false;
+  bool suggestionFlag = false;
   int option_index = 0;
   int c = 0;
 
-  kiwix::Searcher *searcher = NULL;
-  kiwix::Reader *reader = NULL;
+  kiwix::Searcher* searcher = NULL;
+  kiwix::Reader* reader = NULL;
 
   /* Argument parsing */
   while (42) {
-
-    static struct option long_options[] = {
-      {"verbose", no_argument, 0, 'v'},
-      {0, 0, 0, 0}
-    };
+    static struct option long_options[]
+        = {{"verbose", no_argument, 0, 'v'},
+           {"suggestion", no_argument, 0, 's'},
+           {0, 0, 0, 0}};
 
     if (c != -1) {
-      c = getopt_long(argc, argv, "vb:", long_options, &option_index);
+      c = getopt_long(argc, argv, "vsb:", long_options, &option_index);
 
       switch (c) {
-	case 'v':
-	  verboseFlag = true;
-	  break;
+        case 'v':
+          verboseFlag = true;
+          break;
+        case 's':
+          suggestionFlag = true;
+          break;
       }
     } else {
       if (optind < argc) {
-	if (zimPath == NULL) {
-	  zimPath = argv[optind++];
-	} else if (search == NULL) {
-	  search = argv[optind++];
-	} else {
-	  cout << zimPath << endl;
-	  usage();
-	}
+        if (zimPath == NULL) {
+          zimPath = argv[optind++];
+        } else if (search == NULL) {
+          search = argv[optind++];
+        } else {
+          cout << zimPath << endl;
+          usage();
+        }
       } else {
-	break;
+        break;
       }
     }
   }
@@ -80,28 +92,34 @@ int main(int argc, char **argv) {
 
   /* Try to prepare the indexing */
   try {
-      reader = new kiwix::Reader(zimPath);
+    reader = new kiwix::Reader(zimPath);
   } catch (...) {
-      /* Cannot open the zimPath, maybe it is a plain old xapian database directory */
+    /* Cannot open the zimPath, maybe it is a plain old xapian database
+     * directory */
   }
 
-  if ( reader ) {
-      searcher = new kiwix::Searcher("", reader);
+  if (reader) {
+    searcher = new kiwix::Searcher();
+    searcher->add_reader(reader, "");
   } else {
-      try {
-          searcher = new kiwix::Searcher(zimPath, NULL);
-      } catch (...) {
-         cerr << "Unable to search through zim '" << zimPath << "'." << endl;
-         exit(1);
-      }
+    try {
+      searcher = new kiwix::Searcher(zimPath, NULL, "");
+    } catch (...) {
+      cerr << "Unable to search through zim '" << zimPath << "'." << endl;
+      exit(1);
+    }
   }
 
   /* Start the indexing */
   if (searcher != NULL) {
     string searchString(search);
-    searcher->search(searchString, 0, 10);
+    if (suggestionFlag) {
+        searcher->suggestions(searchString, verboseFlag);
+    } else  {
+        searcher->search(searchString, 0, 10, verboseFlag);
+    }
     kiwix::Result* p_result;
-    while ( (p_result = searcher->getNextResult()) ) {
+    while ((p_result = searcher->getNextResult())) {
       cout << p_result->get_title() << endl;
       delete p_result;
     }

src/searcher/meson.build

@@ -1,3 +1,4 @@
 executable('kiwix-search', ['kiwix-search.cpp'],
   dependencies:all_deps,
-  install:true)
+  install:true,
+  install_rpath: join_paths(get_option('prefix'), get_option('libdir')))

src/server/meson.build

@@ -1,7 +1,8 @@
 
-sources = ['kiwix-serve.cpp']
+sources = ['kiwix-serve.cpp', 'request_context.cpp']
 sources += server_resources
 
 executable('kiwix-serve', sources,
   dependencies:all_deps,
-  install:true)
+  install:true,
+  install_rpath: join_paths(get_option('prefix'), get_option('libdir')))

src/server/request_context.cpp

@@ -0,0 +1,201 @@
+/*
+ * Copyright 2009-2016 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright 2017 Matthieu Gautier<mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+
+#include "request_context.h"
+#include <string.h>
+#include <stdexcept>
+#include <sstream>
+#include <cstdio>
+
+RequestContext::RequestContext(struct MHD_Connection* connection,
+                               std::string rootLocation,
+                               const std::string& _url,
+                               const std::string& method,
+                               const std::string& version) :
+  full_url(_url),
+  url(_url),
+  valid_url(true),
+  version(version),
+  acceptEncodingDeflate(false),
+  accept_range(false),
+  range_pair(0, -1)
+{
+  if (method == "GET") {
+    this->method = RequestMethod::GET;
+  } else if (method == "HEAD") {
+    this->method = RequestMethod::HEAD;
+  } else if (method == "POST") {
+    this->method = RequestMethod::POST;
+  } else if (method == "PUT") {
+    this->method = RequestMethod::PUT;
+  } else if (method == "DELETE") {
+    this->method = RequestMethod::DELETE_;
+  } else if (method == "CONNECT") {
+    this->method = RequestMethod::CONNECT;
+  } else if (method == "OPTIONS") {
+    this->method = RequestMethod::OPTIONS;
+  } else if (method == "TRACE") {
+    this->method = RequestMethod::TRACE;
+  } else if (method == "PATCH") {
+    this->method = RequestMethod::PATCH;
+  } else {
+    this->method = RequestMethod::OTHER;
+  }
+
+  MHD_get_connection_values(connection, MHD_HEADER_KIND, &RequestContext::fill_header, this);
+  MHD_get_connection_values(connection, MHD_GET_ARGUMENT_KIND, &RequestContext::fill_argument, this);
+
+  valid_url = true;
+  if (rootLocation.empty()) {
+    // nothing special to handle.
+    url = full_url;
+  } else {
+    if (full_url.size() > rootLocation.size() &&
+        full_url.substr(0, rootLocation.size()+1) == rootLocation + "/") {
+      url = full_url.substr(rootLocation.size());
+    } else {
+      valid_url = false;
+    }
+  }
+
+  try {
+    acceptEncodingDeflate =
+        (get_header(MHD_HTTP_HEADER_ACCEPT_ENCODING).find("deflate") != std::string::npos);
+  } catch (const std::out_of_range&) {}
+
+  /*Check if range is requested. */
+  try {
+    auto range = get_header(MHD_HTTP_HEADER_RANGE);
+    int start = 0;
+    int end = -1;
+    std::istringstream iss(range);
+    char c;
+
+    iss >> start >> c;
+    if (iss.good() && c=='-') {
+      iss >> end;
+      if (iss.fail()) {
+        // Something went wrong will extracting.
+        end = -1;
+      }
+      if (iss.eof()) {
+        accept_range = true;
+        range_pair = std::pair<int, int>(start, end);
+      }
+    }
+  } catch (const std::out_of_range&) {}
+
+
+}
+
+RequestContext::~RequestContext()
+{}
+
+
+int RequestContext::fill_header(void *__this, enum MHD_ValueKind kind,
+                                 const char *key, const char *value)
+{
+  RequestContext *_this = static_cast<RequestContext*>(__this);
+  _this->headers[key] = value;
+  return MHD_YES;
+}
+
+int RequestContext::fill_argument(void *__this, enum MHD_ValueKind kind,
+                                   const char *key, const char* value)
+{
+  RequestContext *_this = static_cast<RequestContext*>(__this);
+  _this->arguments[key] = value == nullptr ? "" : value;
+  return MHD_YES;
+}
+
+void RequestContext::print_debug_info() {
+  printf("method    : %s (%d)\n", method==RequestMethod::GET ? "GET" :
+                                  method==RequestMethod::POST ? "POST" :
+                                  "OTHER", (int)method);
+  printf("version   : %s\n", version.c_str());
+  printf("headers   :\n");
+  for (auto it=headers.begin(); it!=headers.end(); it++) {
+    printf(" - %s : '%s'\n", it->first.c_str(), it->second.c_str());
+  }
+  printf("arguments :\n");
+  for (auto it=arguments.begin(); it!=arguments.end(); it++) {
+    printf(" - %s : '%s'\n", it->first.c_str(), it->second.c_str());
+  }
+  printf("Parsed : \n");
+  printf("url   : %s\n", url.c_str());
+  printf("acceptEncodingDeflate : %d\n", acceptEncodingDeflate);
+  printf("has_range : %d\n", accept_range);
+  printf(".............\n");
+}
+
+
+RequestMethod RequestContext::get_method() {
+  return method;
+}
+
+std::string RequestContext::get_url() {
+  return url;
+}
+
+std::string RequestContext::get_url_part(int number) {
+  size_t start = 1;
+  while(true) {
+    auto found = url.find('/', start);
+    if (number == 0) {
+      if (found == std::string::npos) {
+        return url.substr(start);
+      } else {
+        return url.substr(start, found-start);
+      }
+    } else {
+      if (found == std::string::npos) {
+        throw std::out_of_range("No parts");
+      }
+      start = found + 1;
+      number -= 1;
+    }
+  }
+}
+
+std::string RequestContext::get_full_url() {
+  return full_url;
+}
+
+bool RequestContext::is_valid_url() {
+  return valid_url;
+}
+
+bool RequestContext::has_range() {
+  return accept_range;
+}
+
+std::pair<int, int> RequestContext::get_range() {
+  return range_pair;
+}
+
+template<>
+std::string RequestContext::get_argument(const std::string& name) {
+  return arguments.at(name);
+}
+
+std::string RequestContext::get_header(const std::string& name) {
+  return headers.at(name);
+}

src/server/request_context.h

@@ -0,0 +1,108 @@
+/*
+ * Copyright 2009-2016 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright 2017 Matthieu Gautier<mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+
+#ifndef REQUEST_CONTEXT_H
+#define REQUEST_CONTEXT_H
+
+#include <string>
+#include <sstream>
+#include <map>
+#include <stdexcept>
+
+extern "C" {
+#include <microhttpd.h>
+}
+
+enum class RequestMethod {
+    GET,
+    HEAD,
+    POST,
+    PUT,
+    DELETE_,
+    CONNECT,
+    OPTIONS,
+    TRACE,
+    PATCH,
+    OTHER
+};
+
+class KeyError : public std::runtime_error {};
+class IndexError: public std::runtime_error {};
+
+
+class RequestContext {
+  public:
+    RequestContext(struct MHD_Connection* connection,
+                   std::string rootLocation,
+                   const std::string& url,
+                   const std::string& method,
+                   const std::string& version);
+    ~RequestContext();
+
+    void print_debug_info();
+
+    bool is_valid_url();
+
+    std::string get_header(const std::string& name);
+    template<typename T=std::string>
+    T get_argument(const std::string& name) {
+        std::istringstream stream(arguments.at(name));
+        T v;
+        stream >> v;
+        return v;
+    }
+
+
+    RequestMethod get_method();
+    std::string get_url();
+    std::string get_url_part(int part);
+    std::string get_full_url();
+
+    bool has_range();
+    std::pair<int, int> get_range();
+
+    bool can_compress() { return acceptEncodingDeflate; }
+
+    // [TODO] Move this to the response builder
+    int httpResponseCode;
+
+  private:
+    std::string full_url;
+    std::string url;
+    bool valid_url;
+    RequestMethod method;
+    std::string version;
+
+    bool acceptEncodingDeflate;
+
+    bool accept_range;
+    std::pair<int, int> range_pair;
+    std::map<std::string, std::string> headers;
+    std::map<std::string, std::string> arguments;
+
+    static int fill_header(void *, enum MHD_ValueKind, const char*, const char*);
+    static int fill_argument(void *, enum MHD_ValueKind, const char*, const char*);
+};
+
+template<> std::string RequestContext::get_argument(const std::string& name);
+
+
+#endif //REQUEST_CONTEXT_H

static/meson.build

@@ -1,4 +1,4 @@
 
-res_compiler = find_program('compile_resources.py')
+res_compiler = find_program('kiwix-compile-resources', 'compile_resources.py')
 
 subdir('server')

static/server/global_taskbar.html.part

@@ -0,0 +1,13 @@
+<span class="kiwix">
+  <span id="kiwixtoolbar" class="ui-widget-header">
+    <div class="kiwix_centered">
+      <div class="kiwix_searchform">
+        <form class="kiwixsearch" method="GET" action="__ROOT_LOCATION__/search" id="kiwixsearchform">
+          <input autocomplete="off" class="ui-autocomplete-input" id="kiwixsearchbox" name="pattern" type="text">
+          <input type="submit" value="&#x1f50d;">
+        </form>
+      </div>
+    </div>
+  </span>
+</span>
+<div style="display: block; height: 5em;"></div>

static/server/home.html.tmpl

@@ -3,10 +3,10 @@
 <head>
   <meta charset="UTF-8" />
   <title>Welcome to Kiwix Server</title>
-  <script type="text/javascript" src="/skin/jquery-ui/external/jquery/jquery.js"></script>
-  <script type="text/javascript" src="/skin/jquery-ui/jquery-ui.min.js"></script>
-  <link type="text/css" href="/skin/jquery-ui/jquery-ui.min.css" rel="Stylesheet" />
-  <link type="text/css" href="/skin/jquery-ui/jquery-ui.theme.min.css" rel="Stylesheet" />
+  <script type="text/javascript" src="__ROOT_LOCATION__/skin/jquery-ui/external/jquery/jquery.js"></script>
+  <script type="text/javascript" src="__ROOT_LOCATION__/skin/jquery-ui/jquery-ui.min.js"></script>
+  <link type="text/css" href="__ROOT_LOCATION__/skin/jquery-ui/jquery-ui.min.css" rel="Stylesheet" />
+  <link type="text/css" href="__ROOT_LOCATION__/skin/jquery-ui/jquery-ui.theme.min.css" rel="Stylesheet" />
   <style>
     	body {
 	        background:

static/server/include.html.part

@@ -1,11 +1,14 @@
-<script type="text/javascript" src="/skin/jquery-ui/external/jquery/jquery.js"></script>
-<script type="text/javascript" src="/skin/jquery-ui/jquery-ui.min.js"></script>
+<link type="text/css" href="__ROOT_LOCATION__/skin/jquery-ui/jquery-ui.min.css" rel="Stylesheet" />
+<link type="text/css" href="__ROOT_LOCATION__/skin/jquery-ui/jquery-ui.theme.min.css" rel="Stylesheet" />
+<link type="text/css" href="__ROOT_LOCATION__/skin/taskbar.css" rel="Stylesheet" />
+<script type="text/javascript" src="__ROOT_LOCATION__/skin/jquery-ui/external/jquery/jquery.js"></script>
+<script type="text/javascript" src="__ROOT_LOCATION__/skin/jquery-ui/jquery-ui.min.js"></script>
 <script>
   var jk = jQuery.noConflict();
   jk(function() {
   jk( "#kiwixsearchbox" ).autocomplete({
 
-  source: "/suggest?content=__CONTENT__",
+  source: "__ROOT_LOCATION__/suggest?content=__CONTENT_ESCAPED__",
   dataType: "json",
   cache: false,
 

static/server/opensearchdescription.xml

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<OpenSearchDescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
+  <ShortName>Zim catalog search</ShortName>
+  <Description>Search zim files in the catalog.</Description>
+  <Url type="application/atom+xml;profile=opds-catalog" 
+       xmlns:atom="http://www.w3.org/2005/Atom"
+       indexOffset="0"
+       template="/__ROOT_LOCATION__/catalog/search?q={searchTerms}&lang={language}&count={count}&start={startIndex}"/>
+</OpenSearchDescription>

static/server/resources_list.txt

@@ -21,3 +21,5 @@ home.html.tmpl
 include.html.part
 taskbar.css
 taskbar.html.part
+global_taskbar.html.part
+opensearchdescription.xml

static/server/taskbar.html.part

@@ -1,18 +1,16 @@
-<link type="text/css" href="/skin/jquery-ui/jquery-ui.min.css" rel="Stylesheet" />
-<link type="text/css" href="/skin/jquery-ui/jquery-ui.theme.min.css" rel="Stylesheet" />
 <span class="kiwix">
   <span id="kiwixtoolbar" class="ui-widget-header">
     <div class="kiwix_centered">
       <div class="kiwix_button_wrapper">
-        <a id="kiwix_serve_taskbar_library_button" href="/"><button>Library</button></a>
-        <a id="kiwix_serve_taskbar_home_button" href="/__CONTENT__/"><button>Home</button></a>
-        <a id="kiwix_serve_taskbar_random_button" href="/random?content=__CONTENT__"><button>Random</button></a>
+        <a id="kiwix_serve_taskbar_library_button" href="__ROOT_LOCATION__/"><button>&#x1f3e0;</button></a>
+        <a id="kiwix_serve_taskbar_home_button" href="__ROOT_LOCATION__/__CONTENT__/"><button>__ZIM_TITLE__</button></a>
+        <a id="kiwix_serve_taskbar_random_button" href="__ROOT_LOCATION__/random?content=__CONTENT_ESCAPED__"><button>&#x1F3B2;</button></a>
       </div>
       <div class="kiwix_searchform">
-        <form class="kiwixsearch" method="GET" action="/search" id="kiwixsearchform">
+        <form class="kiwixsearch" method="GET" action="__ROOT_LOCATION__/search" id="kiwixsearchform">
           <input type="hidden" name="content" value="__CONTENT__" />
           <input autocomplete="off" class="ui-autocomplete-input" id="kiwixsearchbox" name="pattern" type="text">
-          <input type="submit" value="Search">
+          <input type="submit" value="&#x1f50d;">
         </form>
       </div>
     </div>

travis/compile.sh

@@ -8,24 +8,17 @@ INSTALL_DIR=${BUILD_DIR}/INSTALL
 
 case ${PLATFORM} in
     "native_static")
-         MESON_OPTION="--default-library=static"
+         MESON_OPTION="-Dstatic-linkage=true"
          ;;
     "native_dyn")
-         MESON_OPTION="--default-library=shared"
+         MESON_OPTION=""
          ;;
     "win32_static")
-         MESON_OPTION="--default-library=static --cross-file ${BUILD_DIR}/meson_cross_file.txt"
+         MESON_OPTION="-Dstatic-linkage=true --cross-file ${BUILD_DIR}/meson_cross_file.txt"
          ;;
     "win32_dyn")
-         MESON_OPTION="--default-library=shared --cross-file ${BUILD_DIR}/meson_cross_file.txt"
+         MESON_OPTION="--cross-file ${BUILD_DIR}/meson_cross_file.txt"
          ;;
-    "android_arm")
-         MESON_OPTION="-Dandroid=true --default-library=shared --cross-file ${BUILD_DIR}/meson_cross_file.txt"
-         ;;
-    "android_arm64")
-         MESON_OPTION="-Dandroid=true --default-library=shared --cross-file ${BUILD_DIR}/meson_cross_file.txt"
-         ;;
-
 esac
 
 cd ${TRAVIS_BUILD_DIR}

travis/install_deps.sh

@@ -3,7 +3,7 @@
 set -e
 
 REPO_NAME=${TRAVIS_REPO_SLUG#*/}
-ARCHIVE_NAME=deps_${PLATFORM}_${REPO_NAME}.tar.gz
+ARCHIVE_NAME=deps_${TRAVIS_OS_NAME}_${PLATFORM}_${REPO_NAME}.tar.gz
 
 # Packages.
 case ${PLATFORM} in
@@ -29,7 +29,7 @@ esac
 
 sudo apt-get update -qq
 sudo apt-get install -qq python3-pip ${PACKAGES}
-pip3 install meson
+sudo pip3 install meson==0.43.0
 
 # Ninja
 cd $HOME
@@ -42,6 +42,4 @@ sudo cp ninja /bin
 # Dependencies comming from kiwix-build.
 cd ${HOME}
 wget http://tmp.kiwix.org/ci/${ARCHIVE_NAME}
-mkdir -p BUILD_${PLATFORM}
-cd BUILD_${PLATFORM}
 tar xf ${HOME}/${ARCHIVE_NAME}