New version 3.0.3

Fix library

.clang-format

@@ -0,0 +1,12 @@
+BasedOnStyle:  Google
+BinPackArguments: false
+BinPackParameters: false
+BreakBeforeBinaryOperators: All
+BreakBeforeBraces: Linux
+DerivePointerAlignment: false
+SpacesInContainerLiterals: false
+Standard: Cpp11
+
+AllowShortFunctionsOnASingleLine: Inline
+AllowShortIfStatementsOnASingleLine: false
+AllowShortLoopsOnASingleLine: false

.travis.yml

@@ -0,0 +1,36 @@
+language: cpp
+dist: trusty
+sudo: false
+cache: ccache
+before_install:
+- PATH=$PATH:$HOME/bin
+install: travis/install_deps.sh
+script: travis/compile.sh
+env:
+  matrix:
+    - PLATFORM="native_static"
+    - PLATFORM="native_dyn"
+    - PLATFORM="win32_static"
+    - PLATFORM="win32_dyn"
+    - PLATFORM="android_arm"
+    - PLATFORM="android_arm64"
+addons:
+  apt:
+    packages:
+      - cmake
+      - python3-pip
+      - libbz2-dev
+      - ccache
+      - zlib1g-dev
+      - uuid-dev
+      - libctpp2-dev
+      - ctpp2-utils
+      - libmicrohttpd-dev
+      - g++-mingw-w64-i686
+      - gcc-mingw-w64-i686
+      - gcc-mingw-w64-base
+      - mingw-w64-tools
+matrix:
+  include:
+    - env: PLATFORM="native_dyn"
+      os: osx

AUTHORS

@@ -0,0 +1,17 @@
+Automactic <christopherliqd@gmail.com>
+Ayoub DARDORY <ayoubuto@gmail.com>
+Cristian Patrasciuc <cristip@google.com>
+Dattaz <taz@dattaz.fr>
+Elad Keyshawn <elad.keyshawn@gmail.com>
+Emmanuel Engelhart <kelson@kiwix.org>
+Isaac <mhutti1@gmail.com>
+jleow00 <leow.yonghan.jerome@gmail.com>
+Julian Harty <julianharty@gmail.com>
+Kiran Mathew Koshy <kiranmathewkoshy@gmail.com>
+Kunal Mehta <legoktm@member.fsf.org>
+Matthieu Gautier <mgautier@kymeria.fr>
+Rashiq Ahmad <rashiq.z@gmail.com>
+Renaud Gaudin <reg@kiwix.org>
+Shivam <ssarodia@gmail.com>
+Steve Wills <steve@mouf.net>
+Synhershko <synhershko@users.sourceforge.net>

ChangeLog

@@ -0,0 +1,157 @@
+kiwix-lib 3.0.3
+===============
+
+ * Add the 'en' language to the mapping alpha2-code ('en') to alpha3-code
+ ('eng').
+ * Correctly write the 'ArticleCount' and 'MediaCount' in the library.xml.
+ * Correctly fill the book size for the zim file size.
+
+kiwix-lib 3.0.2
+===============
+
+ * Use the correct path separator when computing relativePath on Windows.
+
+kiwix-lib 3.0.1
+===============
+
+ * Small fix about parsing the opdsStream.
+
+kiwix-lib 3.0.0
+===============
+
+ * Change the downloader to use aria2 using a separated process (with rpc)
+   instead of using the libaria2. This simplify a lot the link process to
+   libaria2 on Windows.
+    - kiwix-lib doesn't depend on libaria2 anymore.
+    - kiwix-lib now depends on libcurl.
+ * [API break] Library class API has been updated :
+    - Books are referenced by id, not index. A lot of methods have been
+      updated this way.
+    - Books "list" is now private.
+    - There is no more "current" book.
+    - listBooksIds's filters have been updated.
+ * [API break] Book class API has been updated :
+    - Move the definition of Book in `book.h`.
+    - Use getter/setter methods instead public members.
+    - Size (getSize/setSize) is now returned in bytes, not kB.
+    - Dependending of how the book has been initialized (opdsfeed), the
+      faviconUrl may be stored in the book, the favicon being downloaded when
+      using `getFavicon`.
+    - The path (and indexPath) are always absolute path.
+    - Book has now a downloadId, corresponding to the aria2 download id (if
+      exists)
+  * [API break] Manager class API has been updated :
+    - The manager is mainly use to fill a Libray from a "library.xml" file or
+      opds feed. Other operations (has removeBookById, setBookPath, filter, ...)
+      have been removed.
+    - The manager use a intermediate class (LibraryManipulator) to add book to
+      the library. This dependency injection allow caller code to hook the add
+      of a book to the library.
+    - The manager work on a existing Library. It doesn't how a internal
+      Library.
+  * [API break] OpdsDumper class API has been updated :
+    - dumpOPDSFeed method now take the list of bookIds to dump instead of
+      dumping all books in the library.
+    - OpdsDumper can now dump openSearch result information (total result
+      count, start index, ...).
+  * [API break] Common tools API has been updated :
+    - `base64_encode` and `base64_decode` take std::string as arguments.
+    - New `download` function in networkTools.h using libcurl.
+    - New `getDataDirectory` function in pathTools.
+    - Better `beautifyInteger` and `beautifyFileSize` functions.
+    - New `nodeToString` function serializing a pugi::xml_node to a string.
+    - New `converta2toa3` function to convert alpha2 language code to aplha3
+      language code.
+
+
+kiwix-lib 2.0.2
+===============
+
+ * [Android] Forward c++ errors message de Java world.
+ * Follow redirection of favicon.
+ * Make aria2 dependency optional.
+ * Inculde unistd.h only on unix platform.
+
+kiwix-lib 2.0.1
+===============
+
+ * Fix parsing of url.
+ * Remove unused static resources.
+ * Correctly decode reserved characters in URLs.
+ * Explicitly use icu namespace to allow use of packaged icu lib.
+
+kiwix-lib 2.0.0
+===============
+
+ * Introduce a new API to retrive content from a reader.
+   * Introduce the `Entry` class.
+   * Reader's methods return an `Entry`.
+   * Content and other information can be retrieved from the `Entry`.
+   * Older Reader's methods are depreciated.
+ * Add an `OPDSDumper` class to dump a whole `Library` as an OPDS feed.
+ * Add a tool function to get the content of a file.
+ * Add a tool function to create a tempory directory.
+ * Add a `Downloader` class to download a file.
+ * Allow the manager to populate a `Library` from an OPDS feed.
+ * Try to locate libctpp2 in default system libdir and then fallback in 'lib'
+   directory.
+ * Build kiwix-lib setting RPATH.
+ * Build kiwix-lib without warning (werror=true)
+ * Build kiwix-lib on macos.
+
+kiwix-lib 1.1.1
+===============
+
+ * Correct the name of kiwix-lib (from `kiwixlib`) in meson.build to generate
+   dist archive with the correct name.
+ * Libzim version need to be at least 3.2.0
+
+kiwix-lib 1.1.0
+===============
+
+ * Allow for more than 70 search result per page in html results rendering
+   (kiwix/kiwix-tools#92)
+ * Add a small api to do geo queries.
+ * Add multi-search support in the JNI (#67)
+ * Add an API to get only one part of an article.
+ * Add an API to get direct location of an article content in the zim file.
+ * Improve urlencoding
+ * Fix pagination in html results rendering.
+ * Compile using gcc-5 on Travis.
+ * Allow JNI to access search snippets.
+ * JNI throw an exception instead of returning an invalid object if something
+   goes wrong.
+ * Add doctext documentation. (#116)
+ * Various bug fixes.
+
+kiwix-lib 1.0.0
+===============
+
+ * Correctly regenerate template resource using cttp2c at compilation time.
+ * Suggestion use xapian database when available
+ * Support multi-zim search in kiwix-lib (a search can now search on several
+   embedded database in zims in the same time)
+ * Fix some wording
+ * Fix license issues
+ * Add out argument to jni getContent* method to get the title of article in
+   the same time we get the content
+ * Rename `compile_resources.py` script to `kiwix-compile-resources`
+ * Use static lib when building for android or in "static mode"
+ * Make the ResourceNotFound exception public
+
+kiwix-lib 0.2.0
+===============
+
+ * Generate the snippet from the article content if the snippet is not
+ directly in the database.
+   This provide better snippets as they now depending of the query.
+ * Use the stopwords and the language stored in the fulltext index database to
+ parse the user query.
+ * Remove the indexer functionnality.
+ * Move to C++11 standard.
+ * Use the fulltext search of the zimlib.
+   We still have the fulltext search code in kiwix-lib to be able to search in
+   fulltext index by side of a zim file. (To be remove in the future)
+ * Few API hanges
+   * Change a lot of `Reader` methods to const methods.
+ * Fix some crashes.

README.md

@@ -15,9 +15,9 @@ to [kiwix-build](https://github.com/kiwix/kiwix-build).
 Preamble
 --------
 
-Although the Kiwix library can be compiled/cross-compiled on/for many
+Although the Kiwix library can be (cross-)compiled on/for many
 sytems, the following documentation explains how to do it on POSIX
-ones. It is primarly though for GNU/Linux systems and has been tested
+ones. It is primarly thought for GNU/Linux systems and has been tested
 on recent releases of Ubuntu and Fedora.
 
 Dependencies
@@ -37,6 +37,8 @@ libraries need to be available:
 (package libctpp2-dev on Ubuntu)
 * Xapian ......................................... https://xapian.org/
 (package libxapian-dev on Ubuntu)
+* libaria2 .................................. https://aria2.github.io/
+(no package on Ubuntu)
 
 These dependencies may or may not be packaged by your operating
 system. They may also be packaged but only in an older version. The
@@ -51,68 +53,91 @@ If you compile ctpp2 from source and want to compile the Kiwix library
 statically then you will probably need to rename ctpp2 static library
 from ctpp2-st.a to ctpp2.a.
 
-Environnement
+Environment
 -------------
 
 The Kiwix library builds using [Meson](http://mesonbuild.com/) version
-0.34 or higher. Meson relies itself on Ninja, pkg-config and few other
+0.39 or higher. Meson relies itself on Ninja, pkg-config and few other
 compilation tools.
 
 Install first the few common compilation tools:
-* Automake
-* Libtool
-* Virtualenv
+* Meson
+* Ninja
 * Pkg-config
 
-Then install Meson itself:
-```
-virtualenv -p python3 ./ # Create virtualenv
-source bin/activate      # Activate the virtualenv
-pip install meson        # Install Meson
-hash -r                  # Refresh bash paths
-```
-
-Finally download and build Ninja locally:
-```
-git clone git://github.com/ninja-build/ninja.git
-cd ninja
-git checkout release
-./configure.py --bootstrap
-mkdir ../bin                      
-cp ninja ../bin
-cd ..
-```
+These tools should be packaged if you use a cutting edge operating
+system. If not, have a look to the "Troubleshooting" section.
 
 Compilation
 -----------
 
-Once all dependencies are installed, you can compile kiwix-lib with:
+Once all dependencies are installed, you can compile the Kiwix library
+with:
 ```
-mkdir build
 meson . build
-cd build
-ninja
+ninja -C build
 ```
 
-By default, it will compile dynamic linked libraries. If you want
-statically linked libraries, you can add `--default-library=static`
-option to the Meson command.
+By default, it will compile dynamic linked libraries. All binary files
+will be created in the "build" directory created automatically by
+Meson. If you want statically linked libraries, you can add
+`--default-library=static` option to the Meson command.
 
 Depending of you system, `ninja` may be called `ninja-build`.
 
 Installation
 ------------
 
-If you want to install the libraries you just have compiled on your
-system, here we go:
+If you want to install the Kiwix library and the headers you just have
+compiled on your system, here we go:
 
 ```
-ninja install
+ninja -C build install
+```
+
+You might need to run the command as root (or using 'sudo'), depending
+where you want to install the libraries. After the installation
+succeeded, you may need to run ldconfig (as root).
+
+Uninstallation
+------------
+
+If you want to uninstall the Kiwix library:
+
+```
+ninja -C build uninstall
+```
+
+Like for the installation, you might need to run the command as root
+(or using 'sudo').
+
+Troubleshooting
+---------------
+
+If you need to install Meson "manually":
+```
+virtualenv -p python3 ./ # Create virtualenv
+source bin/activate      # Activate the virtualenv
+pip3 install meson       # Install Meson
+hash -r                  # Refresh bash paths
+```
+
+If you need to install Ninja "manually":
+```
+git clone git://github.com/ninja-build/ninja.git
+cd ninja
+git checkout release
+./configure.py --bootstrap
+mkdir ../bin
+cp ninja ../bin
 cd ..
 ```
 
-You might need to run the command as root, depending where you want to
-install the libraries.
+If the compilation still fails, you might need to get a more recent
+version of a dependency than the one packaged by your Linux
+distribution. Try then with a source tarball distributed by the
+problematic upstream project or even directly from the source code
+repository.
 
 License
 -------

format_code.sh

@@ -0,0 +1,36 @@
+#!/usr/bin/bash
+
+files=(
+"include/library.h"
+"include/common/stringTools.h"
+"include/common/pathTools.h"
+"include/common/otherTools.h"
+"include/common/regexTools.h"
+"include/common/networkTools.h"
+"include/manager.h"
+"include/reader.h"
+"include/kiwix.h"
+"include/xapianSearcher.h"
+"include/searcher.h"
+"src/library.cpp"
+"src/android/kiwix.cpp"
+"src/android/org/kiwix/kiwixlib/JNIKiwixBool.java"
+"src/android/org/kiwix/kiwixlib/JNIKiwix.java"
+"src/android/org/kiwix/kiwixlib/JNIKiwixString.java"
+"src/android/org/kiwix/kiwixlib/JNIKiwixInt.java"
+"src/searcher.cpp"
+"src/common/pathTools.cpp"
+"src/common/regexTools.cpp"
+"src/common/otherTools.cpp"
+"src/common/networkTools.cpp"
+"src/common/stringTools.cpp"
+"src/xapianSearcher.cpp"
+"src/manager.cpp"
+"src/reader.cpp"
+)
+
+for i in "${files[@]}"
+do
+  echo $i
+  clang-format -i -style=file $i
+done

include/book.h

@@ -0,0 +1,125 @@
+/*
+ * Copyright 2011 Emmanuel Engelhart <kelson@kiwix.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#ifndef KIWIX_BOOK_H
+#define KIWIX_BOOK_H
+
+#include <string>
+
+namespace pugi {
+class xml_node;
+}
+
+namespace kiwix
+{
+enum supportedIndexType { UNKNOWN, XAPIAN };
+
+class OPDSDumper;
+class Reader;
+
+/**
+ * A class to store information about a book (a zim file)
+ */
+class Book
+{
+ public:
+  Book();
+  ~Book();
+
+  bool update(const Book& other);
+  void update(const Reader& reader);
+  void updateFromXml(const pugi::xml_node& node, const std::string& baseDir);
+  void updateFromOpds(const pugi::xml_node& node, const std::string& urlHost);
+  std::string getHumanReadableIdFromPath();
+
+  bool readOnly() const { return m_readOnly; }
+  const std::string& getId() const { return m_id; }
+  const std::string& getPath() const { return m_path; }
+  bool isPathValid() const { return m_pathValid; }
+  const std::string& getIndexPath() const { return m_indexPath; }
+  const supportedIndexType& getIndexType() const { return m_indexType; }
+  const std::string& getTitle() const { return m_title; }
+  const std::string& getDescription() const { return m_description; }
+  const std::string& getLanguage() const { return m_language; }
+  const std::string& getCreator() const { return m_creator; }
+  const std::string& getPublisher() const { return m_publisher; }
+  const std::string& getDate() const { return m_date; }
+  const std::string& getUrl() const { return m_url; }
+  const std::string& getName() const { return m_name; }
+  const std::string& getTags() const { return m_tags; }
+  const std::string& getOrigId() const { return m_origId; }
+  const uint64_t& getArticleCount() const { return m_articleCount; }
+  const uint64_t& getMediaCount() const { return m_mediaCount; }
+  const uint64_t& getSize() const { return m_size; }
+  const std::string& getFavicon() const;
+  const std::string& getFaviconMimeType() const { return m_faviconMimeType; }
+  const std::string& getDownloadId() const { return m_downloadId; }
+
+  void setReadOnly(bool readOnly) { m_readOnly = readOnly; }
+  void setId(const std::string& id) { m_id = id; }
+  void setPath(const std::string& path);
+  void setPathValid(bool valid) { m_pathValid = valid; }
+  void setIndexPath(const std::string& indexPath);
+  void setIndexType(supportedIndexType indexType) { m_indexType = indexType;}
+  void setTitle(const std::string& title) { m_title = title; }
+  void setDescription(const std::string& description) { m_description = description; }
+  void setLanguage(const std::string& language) { m_language = language; }
+  void setCreator(const std::string& creator) { m_creator = creator; }
+  void setPublisher(const std::string& publisher) { m_publisher = publisher; }
+  void setDate(const std::string& date) { m_date = date; }
+  void setUrl(const std::string& url) { m_url = url; }
+  void setName(const std::string& name) { m_name = name; }
+  void setTags(const std::string& tags) { m_tags = tags; }
+  void setOrigId(const std::string& origId) { m_origId = origId; }
+  void setArticleCount(uint64_t articleCount) { m_articleCount = articleCount; }
+  void setMediaCount(uint64_t mediaCount) { m_mediaCount = mediaCount; }
+  void setSize(uint64_t size) { m_size = size; }
+  void setFavicon(const std::string& favicon) { m_favicon = favicon; }
+  void setFaviconMimeType(const std::string& faviconMimeType) { m_faviconMimeType = faviconMimeType; }
+  void setDownloadId(const std::string& downloadId) { m_downloadId = downloadId; }
+
+ protected:
+  std::string m_id;
+  std::string m_downloadId;
+  std::string m_path;
+  bool m_pathValid;
+  std::string m_indexPath;
+  supportedIndexType m_indexType;
+  std::string m_title;
+  std::string m_description;
+  std::string m_language;
+  std::string m_creator;
+  std::string m_publisher;
+  std::string m_date;
+  std::string m_url;
+  std::string m_name;
+  std::string m_tags;
+  std::string m_origId;
+  uint64_t m_articleCount;
+  uint64_t m_mediaCount;
+  bool m_readOnly;
+  uint64_t m_size;
+  mutable std::string m_favicon;
+  std::string m_faviconUrl;
+  std::string m_faviconMimeType;
+};
+
+}
+
+#endif

include/common.h

@@ -0,0 +1,24 @@
+
+#ifndef _KIWIX_COMMON_H_
+#define _KIWIX_COMMON_H_
+
+#include <zim/zim.h>
+
+#ifdef __GNUC__
+#define DEPRECATED __attribute__((deprecated))
+#elif defined(_MSC_VER)
+#define DEPRECATED __declspec(deprecated)
+#else
+#praga message("WARNING: You need to implement DEPRECATED for this compiler")
+#define DEPRECATED
+#endif
+
+
+namespace kiwix {
+
+typedef zim::size_type size_type;
+typedef zim::offset_type offset_type;
+
+}
+
+#endif //_KIWIX_COMMON_H_

include/common/base64.h

@@ -1,4 +1,4 @@
 #include <string>
 
-std::string base64_encode(unsigned char const* , unsigned int len);
-std::string base64_decode(std::string const& s);
+std::string base64_encode(const std::string& inString);
+std::string base64_decode(const std::string& s);

include/common/networkTools.h

@@ -20,29 +20,14 @@
 #ifndef KIWIX_NETWORKTOOLS_H
 #define KIWIX_NETWORKTOOLS_H
 
-#ifdef _WIN32
-#include <winsock2.h>
-#include <ws2tcpip.h>
-#else
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <unistd.h>
-#include <sys/ioctl.h>
-#include <sys/socket.h>
-#include <sys/types.h>
-#include <net/if.h>
-#include <netdb.h>
-#endif
-
-#include <iostream>
-#include <vector>
-#include <string>
 #include <map>
+#include <string>
 
-namespace kiwix {
-  std::map<std::string, std::string> getNetworkInterfaces();
-  std::string getBestPublicIp();
+namespace kiwix
+{
+std::map<std::string, std::string> getNetworkInterfaces();
+std::string getBestPublicIp();
+std::string download(const std::string& url);
 }
 
 #endif

include/common/otherTools.h

@@ -26,8 +26,13 @@
 #include <unistd.h>
 #endif
 
-namespace kiwix {
-  void sleep(unsigned int milliseconds);
+#include <pugixml.hpp>
+
+namespace kiwix
+{
+void sleep(unsigned int milliseconds);
+std::string nodeToString(pugi::xml_node node);
+std::string converta2toa3(const std::string& a2code);
 }
 
 #endif

include/common/pathTools.h

@@ -20,18 +20,18 @@
 #ifndef KIWIX_PATHTOOLS_H
 #define KIWIX_PATHTOOLS_H
 
+#include <fcntl.h>
+#include <limits.h>
+#include <stdio.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <fstream>
+#include <ios>
+#include <iostream>
+#include <sstream>
 #include <string>
 #include <vector>
-#include <sstream>
-#include <iostream>
-#include <fstream>
-#include <string.h>
-#include <stdio.h>
-#include <sys/types.h>
-#include <fcntl.h>
-#include <sys/stat.h>
-#include <ios>
-#include <limits.h>
 
 #ifdef _WIN32
 #include <direct.h>
@@ -41,20 +41,24 @@
 
 using namespace std;
 
-bool isRelativePath(const string &path);
+bool isRelativePath(const string& path);
 string computeAbsolutePath(const string path, const string relativePath);
 string computeRelativePath(const string path, const string absolutePath);
-string removeLastPathElement(const string path, const bool removePreSeparator = false, 
-			     const bool removePostSeparator = false);
-string appendToDirectory(const string &directoryPath, const string &filename);
+string removeLastPathElement(const string path,
+                             const bool removePreSeparator = false,
+                             const bool removePostSeparator = false);
+string appendToDirectory(const string& directoryPath, const string& filename);
 
-unsigned int getFileSize(const string &path);
-string getFileSizeAsString(const string &path);
-bool fileExists(const string &path);
-bool makeDirectory(const string &path);
-bool copyFile(const string &sourcePath, const string &destPath);
-string getLastPathElement(const string &path);
+unsigned int getFileSize(const string& path);
+string getFileSizeAsString(const string& path);
+string getFileContent(const string& path);
+bool fileExists(const string& path);
+bool makeDirectory(const string& path);
+string makeTmpDirectory();
+bool copyFile(const string& sourcePath, const string& destPath);
+string getLastPathElement(const string& path);
 string getExecutablePath();
 string getCurrentDirectory();
-bool writeTextFile(const string &path, const string &content);
+string getDataDirectory();
+bool writeTextFile(const string& path, const string& content);
 #endif

include/common/regexTools.h

@@ -22,11 +22,15 @@
 
 #include <unicode/regex.h>
 #include <unicode/ucnv.h>
-#include <string>
 #include <map>
+#include <string>
 
-bool matchRegex(const std::string &content, const std::string &regex);
-std::string replaceRegex(const std::string &content, const std::string &replacement, const std::string &regex);
-std::string appendToFirstOccurence(const std::string &content, const std::string regex, const std::string &replacement);
+bool matchRegex(const std::string& content, const std::string& regex);
+std::string replaceRegex(const std::string& content,
+                         const std::string& replacement,
+                         const std::string& regex);
+std::string appendToFirstOccurence(const std::string& content,
+                                   const std::string regex,
+                                   const std::string& replacement);
 
 #endif

include/common/stringTools.h

@@ -20,52 +20,64 @@
 #ifndef KIWIX_STRINGTOOLS_H
 #define KIWIX_STRINGTOOLS_H
 
-#include <unicode/translit.h>
-#include <unicode/normlzr.h>
 #include <unicode/unistr.h>
-#include <unicode/rep.h>
-#include <unicode/uniset.h>
-#include <unicode/ustring.h>
-#include <unicode/ucnv.h>
 
-#include <iostream>
-#include <vector>
-#include <string>
 #include <fstream>
+#include <iomanip>
+#include <iostream>
 #include <sstream>
+#include <string>
+#include <vector>
 
 #include "pathTools.h"
 
-namespace kiwix {
-
+namespace kiwix
+{
 #ifndef __ANDROID__
 
-  std::string beautifyInteger(const unsigned int number);
-  std::string beautifyFileSize(const unsigned int number);
-  std::string urlEncode(const std::string &c);
-  void printStringInHexadecimal(const char *s);
-  void printStringInHexadecimal(UnicodeString s);
-  void stringReplacement(std::string& str, const std::string& oldStr, const std::string& newStr);
-  std::string encodeDiples(const std::string& str);
+std::string beautifyInteger(uint64_t number);
+std::string beautifyFileSize(uint64_t number);
+void printStringInHexadecimal(const char* s);
+void printStringInHexadecimal(icu::UnicodeString s);
+void stringReplacement(std::string& str,
+                       const std::string& oldStr,
+                       const std::string& newStr);
+std::string encodeDiples(const std::string& str);
 
 #endif
 
-  std::string removeAccents(const std::string &text);
-  void loadICUExternalTables();
-  std::string urlDecode(const std::string &c);
+std::string removeAccents(const std::string& text);
+void loadICUExternalTables();
 
-  std::vector<std::string> split(const std::string&, const std::string&);
-  std::vector<std::string> split(const char*, const char*);
-  std::vector<std::string> split(const std::string&, const char*);
-  std::vector<std::string> split(const char*, const std::string&);
+std::string urlEncode(const std::string& value, bool encodeReserved = false);
+std::string urlDecode(const std::string& value, bool component = false);
 
-  std::string ucAll(const std::string &word);
-  std::string lcAll(const std::string &word);
-  std::string ucFirst(const std::string &word);
-  std::string lcFirst(const std::string &word);
-  std::string toTitle(const std::string &word);
+std::vector<std::string> split(const std::string&, const std::string&);
+std::vector<std::string> split(const char*, const char*);
+std::vector<std::string> split(const std::string&, const char*);
+std::vector<std::string> split(const char*, const std::string&);
 
-  std::string normalize(const std::string &word);
+std::string ucAll(const std::string& word);
+std::string lcAll(const std::string& word);
+std::string ucFirst(const std::string& word);
+std::string lcFirst(const std::string& word);
+std::string toTitle(const std::string& word);
+
+std::string normalize(const std::string& word);
+template<typename T>
+std::string to_string(T value)
+{
+    std::ostringstream oss;
+    oss << value;
+    return oss.str();
 }
 
+template<typename T>
+T extractFromString(const std::string& str) {
+    std::istringstream iss(str);
+    T ret;
+    iss >> ret;
+    return ret;
+}
+} //namespace kiwix
 #endif

include/downloader.h

@@ -0,0 +1,104 @@
+/*
+ * Copyright 2018 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#ifndef KIWIX_DOWNLOADER_H
+#define KIWIX_DOWNLOADER_H
+
+#include <string>
+#include <vector>
+#include <map>
+#include <pthread.h>
+#include <memory>
+
+namespace kiwix
+{
+
+class Aria2;
+struct DownloadedFile {
+  DownloadedFile()
+   : success(false) {}
+  bool success;
+  std::string path;
+};
+
+class AriaError : public std::runtime_error {
+ public:
+  AriaError(const std::string& message) : std::runtime_error(message) {}
+};
+
+
+class Download {
+ public:
+  typedef enum { K_ACTIVE, K_WAITING, K_PAUSED, K_ERROR, K_COMPLETE, K_REMOVED, K_UNKNOWN } StatusResult;
+
+  Download() :
+    m_status(K_UNKNOWN) {}
+  Download(std::shared_ptr<Aria2> p_aria, std::string did)
+    : mp_aria(p_aria),
+      m_status(K_UNKNOWN),
+      m_did(did) {};
+  void updateStatus(bool follow=false);
+  StatusResult getStatus()          { return m_status; }
+  std::string  getDid()             { return m_did; }
+  std::string  getFollowedBy()      { return m_followedBy; }
+  uint64_t     getTotalLength()     { return m_totalLength; }
+  uint64_t     getCompletedLength() { return m_completedLength; }
+  uint64_t     getDownloadSpeed()   { return m_downloadSpeed; }
+  uint64_t     getVerifiedLength()  { return m_verifiedLength; }
+  std::string  getPath()            { return m_path; }
+  std::vector<std::string>&  getUris()             { return m_uris; }
+
+ protected:
+  std::shared_ptr<Aria2> mp_aria;
+  StatusResult m_status;
+  std::string m_did = "";
+  std::string m_followedBy = "";
+  uint64_t m_totalLength;
+  uint64_t m_completedLength;
+  uint64_t m_downloadSpeed;
+  uint64_t m_verifiedLength;
+  std::vector<std::string> m_uris;
+  std::string m_path;
+};
+
+/**
+ * A tool to download things.
+ *
+ */
+class Downloader
+{
+ public:
+  Downloader();
+  virtual ~Downloader();
+
+  void close();
+
+  Download* startDownload(const std::string& uri);
+  Download* getDownload(const std::string& did);
+
+  size_t getNbDownload() { return m_knownDownloads.size(); }
+  std::vector<std::string> getDownloadIds();
+
+ private:
+  std::map<std::string, std::unique_ptr<Download>> m_knownDownloads;
+  std::shared_ptr<Aria2> mp_aria;
+};
+}
+
+#endif

include/entry.h

@@ -0,0 +1,191 @@
+/*
+ * Copyright 2018 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#ifndef KIWIX_ENTRY_H
+#define KIWIX_ENTRY_H
+
+#include <stdio.h>
+#include <zim/article.h>
+#include <exception>
+#include <string>
+#include "common.h"
+
+using namespace std;
+
+namespace kiwix
+{
+
+
+class NoEntry : public std::exception {};
+
+/**
+ * A entry represent an.. entry in a zim file.
+ */
+class Entry
+{
+  public:
+    /**
+     * Default constructor.
+     *
+     * Construct an invalid entry.
+     */
+    Entry() = default;
+
+    /**
+     * Construct an entry making reference to an zim article.
+     *
+     * @param article a zim::Article object
+     */
+    Entry(zim::Article article);
+    virtual ~Entry() = default;
+
+    /**
+     * Get the path of the entry.
+     *
+     * The path is the "key" of an entry.
+     *
+     * @return the path of the entry.
+     */
+    std::string getPath() const;
+    
+    /**
+     * Get the title of the entry.
+     *
+     * @return the title of the entry.
+     */
+    std::string getTitle() const;
+        
+    /**
+     * Get the content of the entry.
+     *
+     * The string is a copy of the content.
+     * If you don't want to do a copy, use get_blob.
+     *
+     * @return the content of the entry.
+     */
+    std::string getContent() const;
+        
+    /**
+     * Get the blob of the entry.
+     *
+     * A blob make reference to the content without copying it.
+     *
+     * @param offset The starting offset of the blob.
+     * @return the blob of the entry.
+     */
+    zim::Blob   getBlob(offset_type offset = 0) const;
+        
+    /**
+     * Get the blob of the entry.
+     *
+     * A blob make reference to the content without copying it.
+     *
+     * @param offset The starting offset of the blob.
+     * @param size The size of the blob.
+     * @return the blob of the entry.
+     */
+    zim::Blob   getBlob(offset_type offset, size_type size) const;
+        
+    /**
+     * Get the info for direct access to the content of the entry.
+     *
+     * Some entry (ie binary ones) have their content plain stored
+     * in the zim file. Knowing the offset where the content is stored
+     * an user can directly read the content in the zim file bypassing the
+     * kiwix-lib/libzim.
+     *
+     * @return A pair specifying where to read the content.
+     *         The string is the real file to read (may be different that .zim
+     *         file if zim is cut).
+     *         The offset is the offset to read in the file.
+     *         Return <"",0> if is not possible to read directly.
+     */
+    std::pair<std::string, offset_type> getDirectAccessInfo() const;
+        
+    /**
+     * Get the size of the entry.
+     *
+     * @return the size of the entry.
+     */
+    size_type   getSize() const;
+
+    /**
+     * Get the mime_type of the entry.
+     *
+     * @return the mime_type of the entry.
+     */
+    std::string getMimetype() const;
+    
+    
+    /**
+     * Get if the entry is a redirect entry.
+     *
+     * @return True if the entry is a redirect.
+     */
+    bool isRedirect() const;
+
+    /**
+     * Get if the entry is a link target entry.
+     *
+     * @return True if the entry is a link target.
+     */
+    bool isLinkTarget() const;
+
+    /**
+     * Get if the entry is a deleted entry.
+     *
+     * @return True if the entry is a deleted entry.
+     */
+    bool isDeleted() const;
+
+    /**
+     * Get the entry pointed by this entry.
+     *
+     * @return the entry pointed.
+     * @throw NoEntry if the entry is not a redirected entry.
+     */
+    Entry getRedirectEntry() const;
+
+    /**
+     * Get the final entry pointed by this entry.
+     *
+     * Follow the redirection until a "not redirecting" entry is found.
+     * If the entry is not a redirected entry, return the entry itself.
+     *
+     * @return the final entry.
+     */
+    Entry getFinalEntry() const;
+
+    /**
+     * Convert the entry to a boolean value.
+     *
+     * @return True if the entry is valid.
+     */
+    explicit operator bool() const { return good(); }
+
+  private:
+    zim::Article article;
+    mutable zim::Article final_article;
+
+    bool good() const { return article.good(); }
+};
+
+}
+
+#endif // KIWIX_ENTRY_H

include/indexer.h

@@ -1,173 +0,0 @@
-/*
- * Copyright 2014 Emmanuel Engelhart <kelson@kiwix.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU  General Public License as published by
- * the Free Software Foundation; either version 3 of the License, or
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
-
-#ifndef KIWIX_INDEXER_H
-#define KIWIX_INDEXER_H
-
-#include <string>
-#include <vector>
-#include <stack>
-#include <queue>
-#include <fstream>
-#include <iostream>
-#include <sstream>
-
-#include <pthread.h>
-#include "common/stringTools.h"
-#include "common/otherTools.h"
-#include <zim/file.h>
-#include <zim/article.h>
-#include <zim/fileiterator.h>
-#include "reader.h"
-
-using namespace std;
-
-namespace kiwix {
-
-  struct indexerToken {
-    string url;
-    string accentedTitle;
-    string title;
-    string keywords;
-    string content;
-    string snippet;
-    string size;
-    string wordCount;
-  };
-  
-  class Indexer {
-      
-  typedef void (* ProgressCallback)(const unsigned int processedArticleCount, const unsigned int totalArticleCount);
-      
-  public:
-    Indexer();
-    virtual ~Indexer();
-
-    bool start(const string zimPath, const string indexPath, ProgressCallback callback = NULL);
-    bool stop();
-    bool isRunning();
-    unsigned int getProgression();
-    void setVerboseFlag(const bool value);
-
-  protected:
-    virtual void indexingPrelude(const string indexPath) = 0;
-    virtual void index(const string &url, 
-		       const string &title, 
-		       const string &unaccentedTitle,
-		       const string &keywords, 
-		       const string &content,
-		       const string &snippet,
-		       const string &size,
-		       const string &wordCount) = 0;
-    virtual void flush() = 0;
-    virtual void indexingPostlude(const string indexPath) = 0;
-
-    /* Stop words */
-    std::vector<std::string> stopWords;
-    void readStopWords(const string languageCode);
-
-    /* Others */
-    unsigned int countWords(const string &text);
-
-    /* Boost factor */
-    unsigned int keywordsBoostFactor;
-    inline unsigned int getTitleBoostFactor(const unsigned int contentLength) {
-      return contentLength / 500 + 1;
-    }
-
-    /* Verbose */
-    pthread_mutex_t verboseMutex;
-    bool getVerboseFlag();
-    bool verboseFlag;
-
-  private:
-    ProgressCallback progressCallback;
-    pthread_mutex_t threadIdsMutex;
-
-    /* Article extraction */
-    pthread_t articleExtractor;
-    pthread_mutex_t articleExtractorRunningMutex;
-    static void *extractArticles(void *ptr);
-    bool articleExtractorRunningFlag;
-    bool isArticleExtractorRunning();
-    void articleExtractorRunning(bool value);
-
-    /* Article parsing */
-    pthread_t articleParser;
-    pthread_mutex_t articleParserRunningMutex;
-    static void *parseArticles(void *ptr);
-    bool articleParserRunningFlag;
-    bool isArticleParserRunning();
-    void articleParserRunning(bool value);
-
-    /* Index writting */
-    pthread_t articleIndexer;
-    pthread_mutex_t articleIndexerRunningMutex;
-    static void *indexArticles(void *ptr);
-    bool articleIndexerRunningFlag;
-    bool isArticleIndexerRunning();
-    void articleIndexerRunning(bool value);
-
-    /* To parse queue */
-    std::queue<indexerToken> toParseQueue;
-    pthread_mutex_t toParseQueueMutex;
-    void pushToParseQueue(indexerToken &token);
-    bool popFromToParseQueue(indexerToken &token);
-    bool isToParseQueueEmpty();
-
-    /* To index queue */
-    std::queue<indexerToken> toIndexQueue;
-    pthread_mutex_t toIndexQueueMutex;
-    void pushToIndexQueue(indexerToken &token);
-    bool popFromToIndexQueue(indexerToken &token);
-    bool isToIndexQueueEmpty();
-
-    /* Article Count & Progression */
-    unsigned int articleCount;
-    pthread_mutex_t articleCountMutex;
-    void setArticleCount(const unsigned int articleCount);
-    unsigned int getArticleCount();
-
-    /* Progression */
-    unsigned int progression;
-    pthread_mutex_t progressionMutex;
-    void setProgression(const unsigned int progression);
-    /* getProgression() is public */
-
-    /* ZIM path */
-    pthread_mutex_t zimPathMutex;
-    string zimPath;
-    void setZimPath(const string path);
-    string getZimPath();
-
-    /* Index path */
-    pthread_mutex_t indexPathMutex;
-    string indexPath;
-    void setIndexPath(const string path);
-    string getIndexPath();
-
-    /* ZIM id */
-    pthread_mutex_t zimIdMutex;
-    string zimId;
-    void setZimId(const string id);
-    string getZimId();
-  };
-}
-
-#endif

include/kiwix.h

@@ -22,5 +22,4 @@
 
 #include "library.h"
 
-
 #endif

include/library.h

@@ -20,88 +20,150 @@
 #ifndef KIWIX_LIBRARY_H
 #define KIWIX_LIBRARY_H
 
-#include <stdio.h>
-#include <stdlib.h>
 #include <string>
-#include <string.h>
 #include <vector>
-#include <stack>
-
-#include "common/stringTools.h"
-#include "common/regexTools.h"
+#include <map>
 
 #define KIWIX_LIBRARY_VERSION "20110515"
 
-using namespace std;
+namespace kiwix
+{
 
-namespace kiwix {
+class Book;
+class OPDSDumper;
 
-  enum supportedIndexType { UNKNOWN, XAPIAN };
+enum supportedListSortBy { UNSORTED, TITLE, SIZE, DATE, CREATOR, PUBLISHER };
+enum supportedListMode {
+  ALL = 0,
+  LOCAL = 1,
+  REMOTE = 1 << 1,
+  NOLOCAL = 1 << 2,
+  NOREMOTE = 1 << 3,
+  VALID = 1 << 4,
+  NOVALID = 1 << 5
+};
+/**
+ * A Library store several books.
+ */
+class Library
+{
+  std::map<std::string, kiwix::Book> books;
+ public:
+  Library();
+  ~Library();
 
-  class Book {
+  std::string version;
+  /**
+   * Add a book to the library.
+   *
+   * If a book already exist in the library with the same id, update
+   * the existing book instead of adding a new one.
+   *
+   * @param book The book to add.
+   * @return True if the book has been added.
+   *         False if a book has been updated.
+   */
+  bool addBook(const Book& book);
 
-  public:
-    Book();
-    ~Book();
+  Book& getBookById(const std::string& id);
 
-    static bool sortByLastOpen(const Book &a, const Book &b);
-    static bool sortByTitle(const Book &a, const Book &b);
-    static bool sortBySize(const Book &a, const Book &b);
-    static bool sortByDate(const Book &a, const Book &b);
-    static bool sortByCreator(const Book &a, const Book &b);
-    static bool sortByPublisher(const Book &a, const Book &b);
-    static bool sortByLanguage(const Book &a, const Book &b);
-    string getHumanReadableIdFromPath();
+  /**
+   * Remove a book from the library.
+   *
+   * @param id the id of the book to remove.
+   * @return True if the book were in the lirbrary and has been removed.
+   */
+  bool removeBookById(const std::string& id);
 
-    string id;
-    string path;
-    string pathAbsolute;
-    string last;
-    string indexPath;
-    string indexPathAbsolute;
-    supportedIndexType indexType;
-    string title;
-    string description;
-    string language;
-    string creator;
-    string publisher;
-    string date;
-    string url;
-    string name;
-    string tags;
-    string origId;
-    string articleCount;
-    string mediaCount;
-    bool readOnly;
-    string size;
-    string favicon;
-    string faviconMimeType;
-  };
+  /**
+   * Write the library to a file.
+   *
+   * @param path the path of the file to write to.
+   * @return True if the library has been correctly save.
+   */
+  bool writeToFile(const std::string& path);
 
-  class Library {
+  /**
+   * Get the number of book in the library.
+   *
+   * @param localBooks If we must count local books (books with a path).
+   * @param remoteBooks If we must count remote books (books with an url)
+   * @return The number of books.
+   */
+  unsigned int getBookCount(const bool localBooks, const bool remoteBooks);
 
-  public:
-    Library();
-    ~Library();
+  /**
+   * Get all langagues of the books in the library.
+   *
+   * @return A list of languages.
+   */
+  std::vector<std::string> getBooksLanguages();
 
-    string version;
-    bool addBook(const Book &book);
-    bool removeBookByIndex(const unsigned int bookIndex);
-    vector <kiwix::Book> books;
+  /**
+   * Get all book creators of the books in the library.
+   *
+   * @return A list of book creators.
+   */
+  std::vector<std::string> getBooksCreators();
 
-    /*
-     * 'current' is the variable storing the current content/book id
-     * in the library. This is used to be able to load per default a
-     * content. As Kiwix may work with many library XML files, you may
-     * have "current" defined many time with different values. The
-     * last XML file read has the priority, Although we do not have an
-     * library object for each file, we want to be able to fallback to
-     * an 'old' current book if the one which should be load
-     * failed. That is the reason why we need a stack here
-     */
-    stack<string> current;
-  };
+  /**
+   * Get all book publishers of the books in the library.
+   *
+   * @return A list of book publishers.
+   */
+  std::vector<std::string> getBooksPublishers();
 
+  /**
+   * Get all book ids of the books in the library.
+   *
+   * @return A list of book ids.
+   */
+  std::vector<std::string> getBooksIds();
+
+  /**
+   * Filter the library and generate a new one with the keep elements.
+   *
+   * This is equivalent to `listBookIds(ALL, UNSORTED, search)`.
+   *
+   * @param search List only books with search in the title or description.
+   * @return The list of bookIds corresponding to the query.
+   */
+  std::vector<std::string> filter(const std::string& search);
+
+
+  /**
+   * List books in the library.
+   *
+   * @param mode The mode of listing :
+   *             - LOCAL    : list only local books (with a path).
+   *             - REMOTE   : list only remote books (with an url).
+   *             - VALID    : list only valid books (without a path or with a
+   *                          path pointing to a valid zim file).
+   *             - NOLOCAL  : list only books without valid path.
+   *             - NOREMOTE : list only books without url.
+   *             - NOVALID  : list only books not valid.
+   *             - ALL : Do not do any filter (LOCAL or REMOTE)
+   *             - Flags can be combined.
+   * @param sortBy Attribute to sort by the book list.
+   * @param search List only books with search in the title, description.
+   * @param language List only books in this language.
+   * @param creator List only books of this creator.
+   * @param publisher List only books of this publisher.
+   * @param maxSize Do not list book bigger than maxSize.
+   *                Set to 0 to cancel this filter.
+   * @return The list of bookIds corresponding to the query.
+   */
+  std::vector<std::string> listBooksIds(
+    int supportedListMode = ALL,
+    supportedListSortBy sortBy = UNSORTED,
+    const std::string& search = "",
+    const std::string& language = "",
+    const std::string& creator = "",
+    const std::string& publisher = "",
+    size_t maxSize = 0);
+
+  friend class OPDSDumper;
+};
 }
 
 #endif

include/manager.h

@@ -20,73 +20,201 @@
 #ifndef KIWIX_MANAGER_H
 #define KIWIX_MANAGER_H
 
-#include <string>
-#include <sstream>
-#include <time.h>
-
-#include <pugixml.hpp>
-
-#include "common/base64.h"
-#include "common/regexTools.h"
-#include "common/pathTools.h"
+#include "book.h"
 #include "library.h"
 #include "reader.h"
 
-using namespace std;
+#include <string>
+#include <vector>
 
-namespace kiwix {
+namespace pugi {
+class xml_document;
+}
 
-  enum supportedListMode { LASTOPEN, REMOTE, LOCAL };
-  enum supportedListSortBy { TITLE, SIZE, DATE, CREATOR, PUBLISHER };
+namespace kiwix
+{
 
-  class Manager {
+class LibraryManipulator {
+ public:
+  virtual ~LibraryManipulator() {}
+  virtual bool addBookToLibrary(Book book) = 0;
+};
 
-  public:
-    Manager();
-    ~Manager();
+class DefaultLibraryManipulator : public LibraryManipulator {
+ public:
+  DefaultLibraryManipulator(Library* library) :
+    library(library) {}
+  virtual ~DefaultLibraryManipulator() {}
+  bool addBookToLibrary(Book book) {
+    return library->addBook(book);
+  }
+ private:
+   kiwix::Library* library;
+};
 
-    bool readFile(const string path, const bool readOnly = true);
-    bool readFile(const string nativePath, const string UTF8Path, const bool readOnly = true);
-    bool readXml(const string xml, const bool readOnly = true, const string libraryPath = "");
-    bool writeFile(const string path);
-    bool removeBookByIndex(const unsigned int bookIndex);
-    bool removeBookById(const string id);
-    bool setCurrentBookId(const string id);
-    string getCurrentBookId();
-    bool setBookIndex(const string id, const string path, const supportedIndexType type);
-    bool setBookIndex(const string id, const string path);
-    bool setBookPath(const string id, const string path);
-    string addBookFromPathAndGetId(const string pathToOpen, const string pathToSave = "", const string url = "",
-				   const bool checkMetaData = false);
-    bool addBookFromPath(const string pathToOpen, const string pathToSave = "", const string url = "",
-			 const bool checkMetaData = false);
-    Library cloneLibrary();
-    bool getBookById(const string id, Book &book);
-    bool getCurrentBook(Book &book);
-    unsigned int getBookCount(const bool localBooks, const bool remoteBooks);
-    bool updateBookLastOpenDateById(const string id);
-    void removeBookPaths();
-    bool listBooks(const supportedListMode mode, const supportedListSortBy sortBy, const unsigned int maxSize,
-		   const string language, const string creator, const string publisher, const string search);
-    vector<string> getBooksLanguages();
-    vector<string> getBooksCreators();
-    vector<string> getBooksPublishers();
-    vector<string> getBooksIds();
+/**
+ * A tool to manage a `Library`.
+ *
+ * A `Manager` handle a internal `Library`.
+ * This `Library` can be retrived with `cloneLibrary` method.
+ */
+class Manager
+{
+ public:
+  Manager(LibraryManipulator* manipulator);
+  Manager(Library* library);
+  ~Manager();
 
-    string writableLibraryPath;
+  /**
+   * Read a `library.xml` and add book in the file to the library.
+   *
+   * @param path The path to the `library.xml`.
+   * @param readOnly Set if the libray path could be overwritten latter with
+   *                 updated content.
+   * @return True if file has been properly parsed.
+   */
+  bool readFile(const std::string& path, const bool readOnly = true);
 
-    vector<std::string> bookIdList;
+  /**
+   * Read a `library.xml` and add book in the file to the library.
+   *
+   * @param nativePath The path of the `library.xml`
+   * @param UTF8Path The utf8 version (?) of the path. Also the path where the
+   *                 library will be writen i readOnly is False.
+   * @param readOnly Set if the libray path could be overwritten latter with
+   *                 updated content.
+   * @return True if file has been properly parsed.
+   */
+  bool readFile(const std::string& nativePath,
+                const std::string& UTF8Path,
+                const bool readOnly = true);
 
-  protected:
-    kiwix::Library library;
+  /**
+   * Load a library content store in the string.
+   *
+   * @param xml The content corresponding of the library xml
+   * @param readOnly Set if the libray path could be overwritten latter with
+   *                 updated content.
+   * @param libraryPath The library path (used to resolve relative path)
+   * @return True if the content has been properly parsed.
+   */
+  bool readXml(const std::string& xml,
+               const bool readOnly = true,
+               const std::string& libraryPath = "");
 
-    bool readBookFromPath(const string path, Book *book = NULL);
-    bool parseXmlDom(const pugi::xml_document &doc, const bool readOnly, const string libraryPath);
+  /**
+   * Load a library content stored in a OPDS stream.
+   *
+   * @param content The content of the OPDS stream.
+   * @param readOnly Set if the library path could be overwritten later with
+   *                 updated content.
+   * @param libraryPath The library path (used to resolve relative path)
+   * @return True if the content has been properly parsed.
+   */
+  bool readOpds(const std::string& content, const std::string& urlHost);
 
-  private:
-    void checkAndCleanBookPaths(Book &book, const string &libraryPath);
-  };
+  /**
+   * Add a book to the library.
+   *
+   * @param pathToOpen The path to the zim file to add.
+   * @param pathToSave The path to store in the library in place of pathToOpen.
+   * @param url        The url of the book to store in the library.
+   * @param checMetaData Tell if we check metadata before adding book to the
+   *                     library.
+   * @return The id of the book if the book has been added to the library.
+   *         Else, an empty string.
+   */
+  std::string addBookFromPathAndGetId(const std::string& pathToOpen,
+                                 const std::string& pathToSave = "",
+                                 const std::string& url = "",
+                                 const bool checkMetaData = false);
 
+  /**
+   * Add a book to the library.
+   *
+   * @param pathToOpen The path to the zim file to add.
+   * @param pathToSave The path to store in the library in place of pathToOpen.
+   * @param url        The url of the book to store in the library.
+   * @param checMetaData Tell if we check metadata before adding book to the
+   *                     library.
+   * @return True if the book has been added to the library.
+   */
+
+  bool addBookFromPath(const std::string& pathToOpen,
+                       const std::string& pathToSave = "",
+                       const std::string& url = "",
+                       const bool checkMetaData = false);
+
+  /**
+   * Get the book corresponding to an id.
+   *
+   * @param[in] id The id of the book
+   * @param[out] book The book corresponding to the id.
+   * @return True if the book has been found.
+   */
+  bool getBookById(const std::string& id, Book& book);
+
+  /**
+   * Update the "last open date" of a book
+   *
+   * @param id the id of the book.
+   * @return True if the book is in the library.
+   */
+  bool updateBookLastOpenDateById(const std::string& id);
+
+  /**
+   * Remove (set to empty) paths of all books in the library.
+   */
+  void removeBookPaths();
+
+  /**
+   * List books in the library.
+   *
+   * The books list will be available in public vector member `bookIdList`.
+   *
+   * @param mode The mode of listing :
+   *             - LASTOPEN sort by last opened book.
+   *             - LOCAL list only local file.
+   *             - REMOTE list only remote file.
+   * @param sortBy Attribute to sort by the book list.
+   * @param maxSize Do not list book bigger than maxSize MiB.
+   *                Set to 0 to cancel this filter.
+   * @param language List only books in this language.
+   * @param creator List only books of this creator.
+   * @param publisher List only books of this publisher.
+   * @param search List only books with search in the title, description or
+   *               language.
+   * @return True
+   */
+  bool listBooks(const supportedListMode mode,
+                 const supportedListSortBy sortBy,
+                 const unsigned int maxSize,
+                 const std::string& language,
+                 const std::string& creator,
+                 const std::string& publisher,
+                 const std::string& search);
+
+  std::string writableLibraryPath;
+
+  bool m_hasSearchResult = false;
+  uint64_t m_totalBooks = 0;
+  uint64_t m_startIndex = 0;
+  uint64_t m_itemsPerPage = 0;
+
+ protected:
+  kiwix::LibraryManipulator* manipulator;
+  bool mustDeleteManipulator;
+
+  bool readBookFromPath(const std::string& path, Book* book);
+  bool parseXmlDom(const pugi::xml_document& doc,
+                   const bool readOnly,
+                   const std::string& libraryPath);
+  bool parseOpdsDom(const pugi::xml_document& doc,
+                    const std::string& urlHost);
+
+ private:
+  void checkAndCleanBookPaths(Book& book, const std::string& libraryPath);
+};
 }
 
 #endif

include/meson.build

@@ -1,16 +1,17 @@
 headers = [
+  'book.h',
+  'common.h',
   'library.h',
   'manager.h',
+  'opds_dumper.h',
+  'downloader.h',
   'reader.h',
+  'entry.h',
   'searcher.h'
 ]
 
-if not get_option('android')
-  headers += ['indexer.h']
-endif
-
 if xapian_dep.found()
-  headers += ['xapianIndexer.h', 'xapianSearcher.h']
+  headers += ['xapianSearcher.h']
 endif
 
 install_headers(headers, subdir:'kiwix')
@@ -22,7 +23,6 @@ install_headers(
   'common/pathTools.h',
   'common/regexTools.h',
   'common/stringTools.h',
-  'common/tree.h',
   subdir:'kiwix/common'
 )
 

include/opds_dumper.h

@@ -0,0 +1,120 @@
+/*
+ * Copyright 2017 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#ifndef KIWIX_OPDS_DUMPER_H
+#define KIWIX_OPDS_DUMPER_H
+
+#include <time.h>
+#include <sstream>
+#include <string>
+
+#include <pugixml.hpp>
+
+#include "common/base64.h"
+#include "common/pathTools.h"
+#include "common/regexTools.h"
+#include "library.h"
+#include "reader.h"
+
+using namespace std;
+
+namespace kiwix
+{
+
+/**
+ * A tool to dump a `Library` into a opds stream.
+ *
+ */
+class OPDSDumper
+{
+ public:
+  OPDSDumper() = default;
+  OPDSDumper(Library* library);
+  ~OPDSDumper();
+
+  /**
+   * Dump the OPDS feed.
+   *
+   * @param id The id of the library.
+   * @return The OPDS feed.
+   */
+  std::string dumpOPDSFeed(const std::vector<std::string>& bookIds);
+
+  /**
+   * Set the id of the opds stream.
+   *
+   * @param id the id to use.
+   */
+  void setId(const std::string& id) { this->id = id;}
+
+  /**
+   * Set the title oft the opds stream.
+   *
+   * @param title the title to use.
+   */
+  void setTitle(const std::string& title) { this->title = title; }
+
+  /**
+   * Set the root location used when generating url.
+   *
+   * @param rootLocation the root location to use.
+   */
+  void setRootLocation(const std::string& rootLocation) { this->rootLocation = rootLocation; }
+
+  /**
+   * Set the search url.
+   *
+   * @param searchUrl the search url to use.
+   */
+  void setSearchDescriptionUrl(const std::string& searchDescriptionUrl) { this->searchDescriptionUrl = searchDescriptionUrl; }
+
+  /**
+   * Set some informations about the search results.
+   *
+   * @param totalResult the total number of results of the search.
+   * @param startIndex the start index of the result.
+   * @param count the number of result of the current set (or page).
+   */
+  void setOpenSearchInfo(int totalResult, int startIndex, int count);
+
+  /**
+   * Set the library to dump.
+   *
+   * @param library The library to dump.
+   */
+  void setLibrary(Library* library) { this->library = library; }
+
+ protected:
+  kiwix::Library* library;
+  std::string id;
+  std::string title;
+  std::string date;
+  std::string rootLocation;
+  std::string searchDescriptionUrl;
+  int m_totalResults;
+  int m_startIndex;
+  int m_count;
+  bool m_isSearchResult = false;
+
+ private:
+  pugi::xml_node handleBook(Book book, pugi::xml_node root_node);
+};
+}
+
+#endif // KIWIX_OPDS_DUMPER_H

include/reader.h

@@ -20,85 +20,486 @@
 #ifndef KIWIX_READER_H
 #define KIWIX_READER_H
 
-#include <zim/zim.h>
-#include <zim/file.h>
-#include <zim/article.h>
-#include <zim/fileiterator.h>
 #include <stdio.h>
-#include <string>
+#include <zim/article.h>
+#include <zim/file.h>
+#include <zim/fileiterator.h>
+#include <zim/zim.h>
 #include <exception>
-#include <sstream>
 #include <map>
+#include <sstream>
+#include <string>
+#include "common.h"
+#include "entry.h"
 #include "common/pathTools.h"
 #include "common/stringTools.h"
 
 using namespace std;
 
-namespace kiwix {
+namespace kiwix
+{
 
-  class Reader {
+/**
+ * The Reader class is the class who allow to get an entry content from a zim
+ * file.
+ */
+class Reader
+{
+ public:
+  /**
+   * Create a Reader to read a zim file specified by zimFilePath.
+   *
+   * @param zimFilePath The path to the zim file to read.
+   *                    The zim file can be splitted (.zimaa, .zimab, ...).
+   *                    In this case, the file path must still point to the
+   *                    unsplitted path as if the file were not splitted
+   *                    (.zim extesion).
+   */
+  Reader(const string zimFilePath);
+  ~Reader();
 
-  public:
-    Reader(const string zimFilePath);
-    ~Reader();
+  /**
+   * Get the number of "displayable" entries in the zim file.
+   *
+   * @return If the zim file has a /M/Counter metadata, return the number of
+   *         entries with the 'text/html' MIMEtype specified in the metadata.
+   *         Else return the number of entries in the 'A' namespace.
+   */
+  unsigned int getArticleCount() const;
 
-    void reset();
-    unsigned int getArticleCount();
-    unsigned int getMediaCount();
-    unsigned int getGlobalCount();
-    string getZimFilePath();
-    string getId();
-    string getRandomPageUrl();
-    string getFirstPageUrl();
-    string getMainPageUrl();
-    bool getMetatag(const string &url, string &content);
-    string getTitle();
-    string getDescription();
-    string getLanguage();
-    string getName();
-    string getTags();
-    string getDate();
-    string getCreator();
-    string getPublisher();
-    string getOrigId();
-    bool getFavicon(string &content, string &mimeType);
-    bool getPageUrlFromTitle(const string &title, string &url);
-    bool getMimeTypeByUrl(const string &url, string &mimeType);
-    bool getContentByUrl(const string &url, string &content, unsigned int &contentLength, string &contentType);
-    bool getContentByEncodedUrl(const string &url, string &content, unsigned int &contentLength, string &contentType, string &baseUrl);
-    bool getContentByEncodedUrl(const string &url, string &content, unsigned int &contentLength, string &contentType);
-    bool getContentByDecodedUrl(const string &url, string &content, unsigned int &contentLength, string &contentType, string &baseUrl);
-    bool getContentByDecodedUrl(const string &url, string &content, unsigned int &contentLength, string &contentType);
-    bool searchSuggestions(const string &prefix, unsigned int suggestionsCount, const bool reset = true);
-    bool searchSuggestionsSmart(const string &prefix, unsigned int suggestionsCount);
-    bool urlExists(const string &url);
-    bool hasFulltextIndex();
-    std::vector<std::string> getTitleVariants(const std::string &title);
-    bool getNextSuggestion(string &title);
-    bool getNextSuggestion(string &title, string &url);
-    bool canCheckIntegrity();
-    bool isCorrupted();
-    bool parseUrl(const string &url, char *ns, string &title);
-    unsigned int getFileSize();
-    zim::File* getZimFileHandler();
-    bool getArticleObjectByDecodedUrl(const string &url, zim::Article &article);
+  /**
+   * Get the number of media in the zim file.
+   *
+   * @return If the zim file has a /M/Counter metadata, return the number of
+   *         entries with the 'image/jpeg', 'image/gif' and 'image/png' in
+   *         the metadata.
+   *         Else return the number of entries in the 'I' namespace.
+   */
+  unsigned int getMediaCount() const;
 
-  protected:
-    zim::File* zimFileHandler;
-    zim::size_type firstArticleOffset;
-    zim::size_type lastArticleOffset;
-    zim::size_type currentArticleOffset;
-    zim::size_type nsACount;
-    zim::size_type nsICount;
-    std::string zimFilePath;
-    
-    std::vector< std::vector<std::string> > suggestions;
-    std::vector< std::vector<std::string> >::iterator suggestionsOffset;
+  /**
+   * Get the number of all entries in the zim file.
+   *
+   * @return Return the number of all the entries, whatever their MIMEtype or
+   *         their namespace.
+   */
+  unsigned int getGlobalCount() const;
 
-  private:
-    std::map<std::string, unsigned int> parseCounterMetadata();
-  };
+  /**
+   * Get the path of the zim file.
+   *
+   * @return the path of the zim file as given in the constructor.
+   */
+  string getZimFilePath() const;
 
+  /**
+   * Get the Id of the zim file.
+   *
+   * @return The uuid stored in the zim file.
+   */
+  string getId() const;
+
+  /**
+   * Get the url of a random page.
+   *
+   * Deprecated : Use `getRandomPage` instead.
+   *
+   * @return Url of a random page. The page is picked from all entries in
+   *         the 'A' namespace.
+   *         The main page is excluded from the potential results.
+   */
+  DEPRECATED string getRandomPageUrl() const;
+
+  /**
+   * Get a random page.
+   *
+   * @return A random Entry. The entry is picked from all entries in
+   *         the 'A' namespace.
+   *         The main entry is excluded from the potential results.
+   */
+  Entry getRandomPage() const;
+
+  /**
+   * Get the url of the first page.
+   *
+   * Deprecated : Use `getFirstPage` instead.
+   *
+   * @return Url of the first entry in the 'A' namespace.
+   */
+  DEPRECATED string getFirstPageUrl() const;
+
+  /**
+   * Get the entry of the first page.
+   *
+   * @return The first entry in the 'A' namespace.
+   */
+  Entry getFirstPage() const;
+
+  /**
+   * Get the url of the main page.
+   *
+   * Deprecated : Use `getMainPage` instead.
+   *
+   * @return Url of the main page as specified in the zim file.
+   */
+  DEPRECATED string getMainPageUrl() const;
+
+  /**
+   * Get the entry of the main page.
+   *
+   * @return Entry of the main page as specified in the zim file.
+   */
+  Entry getMainPage() const;
+
+  /**
+   * Get the content of a metadata.
+   *
+   * @param[in]  name The name of the metadata.
+   * @param[out] value The value will be set to the content of the metadata.
+   * @return True if it was possible to get the content of the metadata.
+   */
+  bool getMetatag(const string& name, string& value) const;
+
+  /**
+   * Get the title of the zim file.
+   *
+   * @return The title of zim file as specified in the zim metadata.
+   *         If no title has been set, return a title computed from the
+   *         file path.
+   */
+  string getTitle() const;
+
+  /**
+   * Get the description of the zim file.
+   *
+   * @return The description of the zim file as specified in the zim metadata.
+   *         If no description has been set, return the subtitle.
+   */
+  string getDescription() const;
+
+  /**
+   * Get the language of the zim file.
+   *
+   * @return The language of the zim file as specified in the zim metadata.
+   */
+  string getLanguage() const;
+
+  /**
+   * Get the name of the zim file.
+   *
+   * @return The name of the zim file as specified in the zim metadata.
+   */
+  string getName() const;
+
+  /**
+   * Get the tags of the zim file.
+   *
+   * @return The tags of the zim file as specified in the zim metadata.
+   */
+  string getTags() const;
+
+  /**
+   * Get the date of the zim file.
+   *
+   * @return The date of the zim file as specified in the zim metadata.
+   */
+  string getDate() const;
+
+  /**
+   * Get the creator of the zim file.
+   *
+   * @return The creator of the zim file as specified in the zim metadata.
+   */
+  string getCreator() const;
+
+  /**
+   * Get the publisher of the zim file.
+   *
+   * @return The publisher of the zim file as specified in the zim metadata.
+   */
+  string getPublisher() const;
+
+  /**
+   * Get the origId of the zim file.
+   *
+   * The origId is only used in the case of patch zim file and is the Id
+   * of the original zim file.
+   *
+   * @return The origId of the zim file as specified in the zim metadata.
+   */
+  string getOrigId() const;
+
+  /**
+   * Get the favicon of the zim file.
+   *
+   * @param[out] content The content of the favicon.
+   * @param[out] mimeType The mimeType of the favicon.
+   * @return True if a favicon has been found.
+   */
+  bool getFavicon(string& content, string& mimeType) const;
+
+  /**
+   * Get an entry associated to an path.
+   *
+   * @param path The path of the entry.
+   * @return The entry.
+   * @throw NoEntry If no entry correspond to the path.
+   */
+  Entry getEntryFromPath(const std::string& path) const;
+
+  /**
+   * Get an entry associated to an url encoded path.
+   *
+   * Equivalent to `getEntryFromPath(urlDecode(path));`
+   *
+   * @param path The url encoded path.
+   * @return The entry.
+   * @throw NoEntry If no entry correspond to the path.
+   */
+  Entry getEntryFromEncodedPath(const std::string& path) const;
+
+  /**
+   * Get un entry associated to a title.
+   *
+   * @param title The title.
+   * @return The entry
+   * throw NoEntry If no entry correspond to the url.
+   */
+  Entry getEntryFromTitle(const std::string& title) const;
+
+  /**
+   * Get the url of a page specified by a title.
+   *
+   * @param[in] title the title of the page.
+   * @param[out] url the url of the page.
+   * @return True if the page can be found.
+   */
+  DEPRECATED bool getPageUrlFromTitle(const string& title, string& url) const;
+
+  /**
+   * Get the mimetype of a entry specified by a url.
+   *
+   * @param[in] url the url of the entry.
+   * @param[out] mimeType the mimeType of the entry.
+   * @return True if the mimeType has been found.
+   */
+  DEPRECATED bool getMimeTypeByUrl(const string& url, string& mimeType) const;
+
+  /**
+   * Get the content of an entry specifed by a url.
+   *
+   * Alias to `getContentByEncodedUrl`
+   */
+  DEPRECATED bool getContentByUrl(const string& url,
+                       string& content,
+                       string& title,
+                       unsigned int& contentLength,
+                       string& contentType) const;
+
+  /**
+   * Get the content of an entry specified by a url encoded url.
+   *
+   * Equivalent to getContentByDecodedUrl(urlDecode(url), ...).
+   */
+  DEPRECATED bool getContentByEncodedUrl(const string& url,
+                              string& content,
+                              string& title,
+                              unsigned int& contentLength,
+                              string& contentType,
+                              string& baseUrl) const;
+
+  /**
+   * Get the content of an entry specified by an url encoded url.
+   *
+   * Equivalent to getContentByEncodedUrl but without baseUrl.
+   */
+  DEPRECATED bool getContentByEncodedUrl(const string& url,
+                              string& content,
+                              string& title,
+                              unsigned int& contentLength,
+                              string& contentType) const;
+
+  /**
+   * Get the content of an entry specified by a url.
+   *
+   * @param[in] url The url of the entry.
+   * @param[out] content The content of the entry.
+   * @param[out] title the title of the entry.
+   * @param[out] contentLength The size of the entry (size of content).
+   * @param[out] contentType The mimeType of the entry.
+   * @param[out] baseUrl Return the true url of the entry.
+   *                     If the specified entry is a redirection, contains
+   *                     the url of the targeted entry.
+   * @return True if the entry has been found.
+   */
+  DEPRECATED bool getContentByDecodedUrl(const string& url,
+                              string& content,
+                              string& title,
+                              unsigned int& contentLength,
+                              string& contentType,
+                              string& baseUrl) const;
+  /**
+   * Get the content of an entry specified by a url.
+   *
+   * Equivalent to getContentByDecodedUrl but withou the baseUrl.
+   */
+  DEPRECATED bool getContentByDecodedUrl(const string& url,
+                              string& content,
+                              string& title,
+                              unsigned int& contentLength,
+                              string& contentType) const;
+
+  /**
+   * Search for entries with title starting with prefix (case sensitive).
+   *
+   * Suggestions are stored in an internal vector and can be retrieved using
+   * `getNextSuggestion` method.
+   *
+   * @param prefix The prefix to search.
+   * @param suggestionsCount How many suggestions to search for.
+   * @param reset If true, remove previous suggestions in the internal vector.
+   *              If false, add suggestions to the internal vector
+   *              (until internal vector size is suggestionCount (or no more
+   *               suggestion))
+   * @return True if some suggestions where added to the internal vector.
+   */
+  bool searchSuggestions(const string& prefix,
+                         unsigned int suggestionsCount,
+                         const bool reset = true);
+
+  /**
+   * Search for entries for the given prefix.
+   *
+   * If the zim file has a internal fulltext index, the suggestions will be
+   * searched using it.
+   * Else the suggestions will be search using `searchSuggestions` while trying
+   * to be smart about case sensitivity (using `getTitleVariants`).
+   *
+   * In any case, suggestions are stored in an internal vector and can be
+   * retrieved using `getNextSuggestion` method.
+   * The internal vector will be reset.
+   *
+   * @param prefix The prefix to search for.
+   * @param suggestionsCount How many suggestions to search for.
+   */
+  bool searchSuggestionsSmart(const string& prefix,
+                              unsigned int suggestionsCount);
+
+  /**
+   * Check if the url exists in the zim file.
+   *
+   * Deprecated : Use `pathExists` instead.
+   *
+   * @param url the url to check.
+   * @return True if the url exits in the zim file.
+   */
+  DEPRECATED bool urlExists(const string& url) const;
+
+  /**
+   * Check if the path exists in the zim file.
+   *
+   * @param path the path to check.
+   * @return True if the path exists in the zim file.
+   */
+  bool pathExists(const string& path) const;
+
+  /**
+   * Check if the zim file has a embedded fulltext index.
+   *
+   * @return True if the zim file has a embedded fulltext index
+   *         and is not split (else the fulltext is not accessible).
+   */
+  bool hasFulltextIndex() const;
+
+  /**
+   * Get potential case title variations for a title.
+   *
+   * @param title a title.
+   * @return the list of variantions.
+   */
+  std::vector<std::string> getTitleVariants(const std::string& title) const;
+
+  /**
+   * Get the next suggestion title.
+   *
+   * @param[out] title the title of the suggestion.
+   * @return True if title has been set.
+   */
+  bool getNextSuggestion(string& title);
+
+  /**
+   * Get the next suggestion title and url.
+   *
+   * @param[out] title the title of the suggestion.
+   * @param[out] url the url of the suggestion.
+   * @return True if title and url have been set.
+   */
+  bool getNextSuggestion(string& title, string& url);
+
+  /**
+   * Get if we can check zim file integrity (has a checksum).
+   *
+   * @return True if zim file have a checksum.
+   */
+  bool canCheckIntegrity() const;
+
+  /**
+   * Check is zim file is corrupted.
+   *
+   * @return True if zim file is corrupted.
+   */
+  bool isCorrupted() const;
+
+  /**
+   * Parse a full url into a namespace and url.
+   *
+   * @param[in] url The full url ("/N/url").
+   * @param[out] ns The namespace (N).
+   * @param[out] title The url (url).
+   * @return True
+   */
+  DEPRECATED bool parseUrl(const string& url, char* ns, string& title) const;
+
+  /**
+   * Return the total size of the zim file.
+   *
+   * If zim file is split, return the sum of all parts' size.
+   *
+   * @return Size of the size file is KiB.
+   */
+  unsigned int getFileSize() const;
+
+  /**
+   * Get the zim file handler.
+   *
+   * @return The libzim file handler.
+   */
+  zim::File* getZimFileHandler() const;
+
+  /**
+   * Get the zim article object associated to a url.
+   *
+   * @param[in] url The url of the article.
+   * @param[out] article The libzim article object.
+   * @return True if the url is good (article.good()).
+   */
+  DEPRECATED bool getArticleObjectByDecodedUrl(const string& url,
+                                    zim::Article& article) const;
+
+ protected:
+  zim::File* zimFileHandler;
+  zim::size_type firstArticleOffset;
+  zim::size_type lastArticleOffset;
+  zim::size_type nsACount;
+  zim::size_type nsICount;
+  std::string zimFilePath;
+
+  std::vector<std::vector<std::string>> suggestions;
+  std::vector<std::vector<std::string>>::iterator suggestionsOffset;
+
+ private:
+  std::map<const std::string, unsigned int> parseCounterMetadata() const;
+};
 }
 
 #endif

include/searcher.h

@@ -22,68 +22,208 @@
 
 #include <stdio.h>
 #include <stdlib.h>
-#include <string>
+#include <unicode/putil.h>
 #include <algorithm>
-#include <vector>
-#include <locale>
 #include <cctype>
+#include <locale>
+#include <string>
+#include <vector>
 #include <vector>
 #include "common/pathTools.h"
 #include "common/stringTools.h"
-#include <unicode/putil.h>
 #include "kiwix_config.h"
 
 using namespace std;
 
-struct Result
+namespace kiwix
 {
-  string url;
-  string title;
-  int score;
-  string snippet;
-  int wordCount;
-  int size;
+class Reader;
+class Result
+{
+ public:
+  virtual ~Result(){};
+  virtual std::string get_url() = 0;
+  virtual std::string get_title() = 0;
+  virtual int get_score() = 0;
+  virtual std::string get_snippet() = 0;
+  virtual std::string get_content() = 0;
+  virtual int get_wordCount() = 0;
+  virtual int get_size() = 0;
+  virtual int get_readerIndex() = 0;
 };
 
-namespace kiwix {
+struct SearcherInternal;
+/**
+ * The Searcher class is reponsible to do different kind of search using the
+ * fulltext index.
+ *
+ * Historically, there are two kind of fulltext index :
+ *  - The legacy one, is the external fulltext index. A directory stored outside
+ *    of the zim file.
+ *  - The new one, a embedded fulltext index in the zim file.
+ *
+ * Legacy external fulltext index has to be considered as obsolet format with
+ * less functionnalities:
+ *  - No multi zim search ;
+ *  - No geo_search ;
+ *  - No suggestions search ;
+ *
+ * To reflect this, there is two Search creation "API":
+ *  - One for the external fulltext index, using the constructor taking a
+ *    xapianDirectoryPath) ;
+ *  - One for the embedded fulltext index, using a "empty" constructor and the
+ *  `add_reader` method".
+ *
+ *  On top of that, the Searcher may (if compiled with ctpp2) be used to
+ *  generate a html page for the search result. This use a template that need a
+ *  humanReaderName. This feature is only used by kiwix-serve and this should be
+ *  move outside of Searcher (and with a better API). If you don't use the html
+ *  rendering (getHtml method), you better should simply ignore the different
+ *  humanReadeableName attributes (or give an empty string).
+ */
+class Searcher
+{
+ public:
+  /**
+   * The default constructor.
+   *
+   * @param humanReadableName The global zim's humanReadableName.
+   *                          Used to generate pagination links.
+   */
+  Searcher(const string& humanReadableName = "");
 
-  class Searcher {
+  /**
+   * The constructor for legacy external fulltext index.
+   *
+   * @param xapianDirectoryPath The path to the external index directory.
+   * @param reader The reader associated to the external index.
+   *               It will be used retrive the article content or generate
+   *               the snippet.
+   * @param humanReadableName The humanReadableName for the zim.
+   */
+  Searcher(const string& xapianDirectoryPath,
+           Reader* reader,
+           const string& humanReadableName);
+  ~Searcher();
 
-  public:
-    Searcher();
-    virtual ~Searcher();
+  /**
+   * Add a reader (containing embedded fulltext index) to the search.
+   *
+   * @param reader The Reader for the zim containing the fulltext index.
+   * @param humanReaderName The human readable name of the reader.
+   * @return true if the reader has been added.
+   *         false if the reader cannot be added (no embedded fulltext index present)
+   */
+  bool add_reader(Reader* reader, const std::string& humanReaderName);
 
-    void search(std::string &search, unsigned int resultStart,
-		unsigned int resultEnd, const bool verbose=false);
-    bool getNextResult(string &url, string &title, unsigned int &score);
-    unsigned int getEstimatedResultCount();
-    bool setProtocolPrefix(const std::string prefix);
-    bool setSearchProtocolPrefix(const std::string prefix);
-    void reset();
-    void setContentHumanReadableId(const string &contentHumanReadableId);
+  /**
+   * Start a search on the zim associated to the Searcher.
+   *
+   * Search results should be retrived using the getNextResult method.
+   *
+   * @param search The search query.
+   * @param resultStart the start offset of the search results (used for pagination).
+   * @param resultEnd the end offset of the search results (used for pagination).
+   * @param verbose print some info on stdout if true.
+   */
+  void search(std::string& search,
+              unsigned int resultStart,
+              unsigned int resultEnd,
+              const bool verbose = false);
+
+  /**
+   * Start a geographique search.
+   * The search return result for entry in a disc of center latitude/longitude
+   * and radius distance.
+   *
+   * Search results should be retrived using the getNextResult method.
+   *
+   * @param latitude The latitude of the center point.
+   * @param longitude The longitude of the center point.
+   * @param distance The radius of the disc.
+   * @param resultStart the start offset of the search results (used for pagination).
+   * @param resultEnd the end offset of the search results (used for pagination).
+   * @param verbose print some info on stdout if true.
+   */
+  void geo_search(float latitude, float longitude, float distance,
+                  unsigned int resultStart,
+                  unsigned int resultEnd,
+                  const bool verbose = false);
+
+  /**
+   * Start a suggestion search.
+   * The search made depend of the "version" of the embedded index.
+   *  - If the index is newer enough and have a title namespace, the search is
+   *    made in the titles only.
+   *  - Else the search is made on the whole article content.
+   * In any case, the search is made "partial" (as adding '*' at the end of the query)
+   *
+   * @param search The search query.
+   * @param verbose print some info on stdout if true.
+   */
+  void suggestions(std::string& search, const bool verbose = false);
+
+  /**
+   * Get the next result of a started search.
+   * This is the method to use to loop hover the search results.
+   */
+  Result* getNextResult();
+
+  /**
+   * Restart the previous search.
+   * Next call to getNextResult will return the first result.
+   */
+  void restart_search();
+
+  /**
+   * Get a estimation of the result count.
+   */
+  unsigned int getEstimatedResultCount();
+
+  /**
+   * Set protocol prefix.
+   * Only used by getHtml.
+   */
+  bool setProtocolPrefix(const std::string prefix);
+
+  /**
+   * Set search protocol prefix.
+   * Only used by getHtml.
+   */
+  bool setSearchProtocolPrefix(const std::string prefix);
 
 #ifdef ENABLE_CTPP2
-    string getHtml();
+  /**
+   * Generate the html page with the resutls of the search.
+   */
+  string getHtml();
 #endif
-    
-  protected:
-    std::string beautifyInteger(const unsigned int number);
-    virtual void closeIndex() = 0;
-    virtual void searchInIndex(string &search, const unsigned int resultStart,
-			       const unsigned int resultEnd, const bool verbose=false) = 0;
 
-    std::vector<Result> results;
-    std::vector<Result>::iterator resultOffset;
-    std::string searchPattern;
-    std::string protocolPrefix;
-    std::string searchProtocolPrefix;
-    std::string template_ct2;
-    unsigned int resultCountPerPage;
-    unsigned int estimatedResultCount;
-    unsigned int resultStart;
-    unsigned int resultEnd;
-    std::string contentHumanReadableId;
-  };
+ protected:
+  std::string beautifyInteger(const unsigned int number);
+  void closeIndex();
+  void searchInIndex(string& search,
+                     const unsigned int resultStart,
+                     const unsigned int resultEnd,
+                     const bool verbose = false);
+
+  std::vector<Reader*> readers;
+  std::vector<std::string> humanReaderNames;
+  SearcherInternal* internal;
+  std::string searchPattern;
+  std::string protocolPrefix;
+  std::string searchProtocolPrefix;
+  unsigned int resultCountPerPage;
+  unsigned int estimatedResultCount;
+  unsigned int resultStart;
+  unsigned int resultEnd;
+  std::string contentHumanReadableId;
+
+ private:
+  void reset();
+
+};
+
 
 }
 

include/xapianIndexer.h

@@ -1,56 +0,0 @@
-/*
- * Copyright 2011 Emmanuel Engelhart <kelson@kiwix.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU  General Public License as published by
- * the Free Software Foundation; either version 3 of the License, or
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
-
-#ifndef KIWIX_XAPIAN_INDEXER_H
-#define KIWIX_XAPIAN_INDEXER_H
-
-#include <xapian.h>
-#include "indexer.h"
-
-using namespace std;
-
-namespace kiwix {
-  
-  class XapianIndexer : public Indexer {
-    
-  public:
-    XapianIndexer();
-    
-  protected:
-    void indexingPrelude(const string indexPath);
-    void index(const string &url, 
-	       const string &title, 
-	       const string &unaccentedTitle,
-	       const string &keywords, 
-	       const string &content,
-	       const string &snippet,
-	       const string &size,
-	       const string &wordCount);
-    void flush();
-    void indexingPostlude(const string indexPath);
-    
-    Xapian::WritableDatabase writableDatabase;
-    Xapian::Stem stemmer;
-    Xapian::SimpleStopper stopper;
-    Xapian::TermGenerator indexer;
-  };
-
-}
-
-#endif

include/xapianSearcher.h

@@ -21,34 +21,78 @@
 #define KIWIX_XAPIAN_SEARCHER_H
 
 #include <xapian.h>
+#include "reader.h"
 #include "searcher.h"
 
+#include <map>
+#include <string>
+
 using namespace std;
 
-namespace kiwix {
+namespace kiwix
+{
+class XapianSearcher;
 
-  class NoXapianIndexInZim: public exception {
-    virtual const char* what() const throw() {
-      return "There is no fulltext index in the zim file";
-    }
-  };
+class XapianResult : public Result
+{
+ public:
+  XapianResult(XapianSearcher* searcher, Xapian::MSetIterator& iterator);
+  virtual ~XapianResult(){};
 
-  class XapianSearcher : public Searcher {
-    
-  public:
-    XapianSearcher(const string &xapianDirectoryPath);
-    virtual ~XapianSearcher() {};
-    void searchInIndex(string &search, const unsigned int resultStart, const unsigned int resultEnd, 
-		       const bool verbose=false);
+  virtual std::string get_url();
+  virtual std::string get_title();
+  virtual int get_score();
+  virtual std::string get_snippet();
+  virtual std::string get_content();
+  virtual int get_wordCount();
+  virtual int get_size();
+  virtual int get_readerIndex() { return 0; };
 
-  protected:
-    void closeIndex();
-    void openIndex(const string &xapianDirectoryPath);
+ private:
+  XapianSearcher* searcher;
+  Xapian::MSetIterator iterator;
+  Xapian::Document document;
+};
 
-    Xapian::Database readableDatabase;
-    Xapian::Stem stemmer;
-  };
+class NoXapianIndexInZim : public exception
+{
+  virtual const char* what() const throw()
+  {
+    return "There is no fulltext index in the zim file";
+  }
+};
 
+class XapianSearcher
+{
+  friend class XapianResult;
+
+ public:
+  XapianSearcher(const string& xapianDirectoryPath, Reader* reader);
+  virtual ~XapianSearcher(){};
+  void searchInIndex(string& search,
+                     const unsigned int resultStart,
+                     const unsigned int resultEnd,
+                     const bool verbose = false);
+  virtual Result* getNextResult();
+  void restart_search();
+
+  Xapian::MSet results;
+
+ protected:
+  void closeIndex();
+  void openIndex(const string& xapianDirectoryPath);
+  void setup_queryParser();
+
+  Reader* reader;
+  Xapian::Database readableDatabase;
+  std::string language;
+  std::string stopwords;
+  Xapian::QueryParser queryParser;
+  Xapian::Stem stemmer;
+  Xapian::SimpleStopper stopper;
+  Xapian::MSetIterator current_result;
+  std::map<std::string, int> valuesmap;
+};
 }
 
 #endif

kiwix.pc.in

@@ -4,7 +4,7 @@ includedir=${prefix}/include
 
 Name: libkiwix
 Description: A library that contains a lot of things used by used by other kiwix programs
-Version: 1.0
+Version: @version@
 Requires: @requires@
 Libs: -L${libdir} -lkiwix @extra_libs@
 Cflags: -I${includedir}/ @extra_cflags@

meson.build

@@ -1,16 +1,33 @@
-project('kiwixlib', 'cpp',
-  version : '0.1.0',
-  license : 'GPL')
+project('kiwix-lib', 'cpp',
+  version : '3.0.3',
+  license : 'GPL',
+  default_options : ['c_std=c11', 'cpp_std=c++11', 'werror=true'])
 
 compiler = meson.get_compiler('cpp')
 find_library_in_compiler = meson.version().version_compare('>=0.31.0')
 
+static_deps = get_option('android') or get_option('default_library') == 'static'
+if get_option('android')
+  extra_libs = ['-llog']
+else
+  extra_libs = []
+endif
+
 thread_dep = dependency('threads')
-libicu_dep = dependency('icu-i18n')
-libzim_dep = dependency('libzim')
-pugixml_dep = dependency('pugixml')
+libicu_dep = dependency('icu-i18n', static:static_deps)
+libzim_dep = dependency('libzim', version : '>=4.0.0', static:static_deps)
+pugixml_dep = dependency('pugixml', static:static_deps)
+libcurl_dep = dependency('libcurl', static:static_deps)
+
+extra_cflags = ''
+if target_machine.system() == 'windows' and static_deps
+  add_project_arguments('-DCURL_STATICLIB', language : 'cpp')
+  extra_cflags += '-DCURL_STATICLIB'
+endif
 
 
+
+ctpp2_include_path = ''
 has_ctpp2_dep = false
 ctpp2_prefix_install = get_option('ctpp2-install-prefix')
 ctpp2_link_args = []
@@ -45,8 +62,14 @@ else
   ctpp2_include_args = ['-I'+ctpp2_include_path]
   if compiler.has_header('ctpp2/CTPP2Logger.hpp', args:ctpp2_include_args)
     ctpp2_include_dir = include_directories(ctpp2_include_path, is_system:true)
-    ctpp2_lib_path = ctpp2_prefix_install+'/lib'
-    ctpp2_lib = compiler.find_library('ctpp2', dirs:ctpp2_lib_path)
+    ctpp2_lib_path = join_paths(ctpp2_prefix_install, get_option('libdir'))
+    message(ctpp2_lib_path)
+    ctpp2_lib = compiler.find_library('ctpp2', dirs:ctpp2_lib_path, required:false)
+    if not ctpp2_lib.found()
+      ctpp2_lib_path = join_paths(ctpp2_prefix_install, 'lib')
+      message(ctpp2_lib_path)
+      ctpp2_lib = compiler.find_library('ctpp2', dirs:ctpp2_lib_path)
+    endif
     ctpp2_link_args = ['-L'+ctpp2_lib_path, '-lctpp2']
     if meson.is_cross_build() and host_machine.system() == 'windows'
       iconv_lib = compiler.find_library('iconv', required:false)
@@ -61,9 +84,9 @@ else
   endif
 endif
 
-xapian_dep = dependency('xapian-core', required:false)
+xapian_dep = dependency('xapian-core', required:false, static:static_deps)
 
-all_deps = [thread_dep, libicu_dep, libzim_dep, xapian_dep, pugixml_dep]
+all_deps = [thread_dep, libicu_dep, libzim_dep, xapian_dep, pugixml_dep, libcurl_dep]
 if has_ctpp2_dep
   all_deps += [ctpp2_dep]
 endif
@@ -74,22 +97,27 @@ conf = configuration_data()
 conf.set('VERSION', '"@0@"'.format(meson.project_version()))
 conf.set('ENABLE_CTPP2', has_ctpp2_dep)
 
+if build_machine.system() == 'windows'
+     extra_link_args = ['-lshlwapi', '-lwinmm']
+else
+     extra_link_args = []
+endif
+
 subdir('include')
 subdir('scripts')
 subdir('static')
 subdir('src')
+subdir('test')
 
-pkg_requires = ['libzim', 'icu-i18n', 'pugixml']
+pkg_requires = ['libzim', 'icu-i18n', 'pugixml', 'libcurl']
 if xapian_dep.found()
     pkg_requires += ['xapian-core']
 endif
 
-extra_libs = []
-extra_cflags = ''
 if has_ctpp2_dep
   extra_libs += ctpp2_link_args
   if ctpp2_include_path != ''
-    extra_cflags = '-I'+ctpp2_include_path
+    extra_cflags = ' -I'+ctpp2_include_path
   endif
 endif
 
@@ -98,6 +126,7 @@ pkg_conf.set('prefix', get_option('prefix'))
 pkg_conf.set('requires', ' '.join(pkg_requires))
 pkg_conf.set('extra_libs', ' '.join(extra_libs))
 pkg_conf.set('extra_cflags', extra_cflags)
+pkg_conf.set('version', meson.project_version())
 configure_file(output : 'kiwix.pc',
                configuration : pkg_conf,
                input : 'kiwix.pc.in',

scripts/ctpp2c.sh

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+
+
+ctpp2c=$1
+SOURCE=$(pwd)/$2
+DEST=$3
+
+$ctpp2c $SOURCE $DEST

scripts/kiwix-compile-resources

@@ -1,5 +1,24 @@
 #!/usr/bin/env python3
 
+'''
+Copyright 2016 Matthieu Gautier <mgautier@kymeria.fr>
+
+This program is free software; you can redistribute it and/or modify
+it under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 3 of the License, or any
+later version.
+
+This program is distributed in the hope that it will be useful, but
+WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+General Public License for more details.
+
+You should have received a copy of the GNU General Public License
+along with this program; if not, write to the Free Software
+Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+02110-1301, USA.
+'''
+
 import argparse
 import os.path
 import re
@@ -38,12 +57,21 @@ extern const std::string {identifier};
 {namespaces_close}"""
 
 class Resource:
-    def __init__(self, base_dir, filename):
+    def __init__(self, base_dirs, filename):
         filename = filename.strip()
         self.filename = filename
         self.identifier = full_identifier(filename)
-        with open(os.path.join(base_dir, filename), 'rb') as f:
-            self.data = f.read()
+        found = False
+        for base_dir in base_dirs:
+            try:
+                with open(os.path.join(base_dir, filename), 'rb') as f:
+                    self.data = f.read()
+                found = True
+                break
+            except FileNotFoundError:
+                continue
+        if not found:
+            raise Exception("Impossible to found {}".format(filename))
 
     def dump_impl(self):
         nb_row = len(self.data)//16 + (1 if len(self.data) % 16 else 0)
@@ -78,16 +106,8 @@ master_c_template = """//This file is automaically generated. Do not modify it.
 
 #include <stdlib.h>
 #include <fstream>
-#include <stdexcept>
 #include "{include_file}"
 
-class ResourceNotFound : public std::runtime_error {{
-  public:
-    ResourceNotFound(const std::string& what_arg):
-      std::runtime_error(what_arg)
-    {{ }};
-}};
-
 static std::string init_resource(const char* name, const unsigned char* content, int len)
 {{
     char * resPath = getenv(name);
@@ -125,11 +145,19 @@ master_h_template = """//This file is automaically generated. Do not modify it.
 #define KIWIX_{BASENAME}
 
 #include <string>
+#include <stdexcept>
 
 namespace RESOURCE {{
     {RESOURCES}
 }};
 
+class ResourceNotFound : public std::runtime_error {{
+  public:
+    ResourceNotFound(const std::string& what_arg):
+      std::runtime_error(what_arg)
+    {{ }};
+}};
+
 const std::string& getResource_{basename}(const std::string& name);
 
 #define getResource(a) (getResource_{basename}(a))
@@ -151,13 +179,18 @@ if __name__ == "__main__":
                         help='The Cpp file name to generate')
     parser.add_argument('--hfile',
                         help='The h file name to generate')
+    parser.add_argument('--source_dir',
+                        help="Additional directory where to look for resources.",
+                        action='append')
     parser.add_argument('resource_file',
                         help='The list of resources to compile.')
     args = parser.parse_args()
 
     base_dir = os.path.dirname(os.path.realpath(args.resource_file))
+    source_dir = args.source_dir or []
     with open(args.resource_file, 'r') as f:
-        resources = [Resource(base_dir, filename) for filename in f.readlines()]
+        resources = [Resource([base_dir]+source_dir, filename)
+                        for filename in f.readlines()]
 
     h_identifier = to_identifier(os.path.basename(args.hfile))
     with open(args.hfile, 'w') as f:

scripts/meson.build

@@ -1,4 +1,5 @@
 
-res_compiler = find_program('compile_resources.py')
+res_compiler = find_program('kiwix-compile-resources')
+intermediate_ctpp2c = find_program('ctpp2c.sh')
 
 install_data(res_compiler.path(), install_dir:get_option('bindir'))

src/android/AndroidManifest.xml

@@ -0,0 +1,13 @@
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+
+  package="kiwix.org.kiwixlib"
+>
+
+  <application android:allowBackup="true"
+    android:label="@string/app_name"
+    android:supportsRtl="true"
+  >
+
+  </application>
+
+</manifest>

src/android/gen_kiwix.sh

@@ -1,11 +0,0 @@
-#!/usr/bin/env bash
-
-set -e
-
-BUILD_PATH=$(pwd)
-
-javac -d $BUILD_PATH/src/android $1 $2 $3 $4
-
-cd $BUILD_PATH/src/android
-javah -jni org.kiwix.kiwixlib.JNIKiwix
-cd $BUILD_PATH

src/android/kiwix.cpp

@@ -1,486 +1,44 @@
+/*
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright (C) 2017 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
 #include <jni.h>
 #include "org_kiwix_kiwixlib_JNIKiwix.h"
 
-#include <stdio.h>
-#include <string.h>
-
 #include <iostream>
 #include <string>
 
 #include "unicode/putil.h"
-#include "reader.h"
-#include "xapianSearcher.h"
-#include "common/base64.h"
 
-#include <android/log.h>
-#define  LOGI(...)  __android_log_print(ANDROID_LOG_INFO, "kiwix", __VA_ARGS__)
+#include "utils.h"
 
-#include <xapian.h>
-#include <zim/zim.h>
-#include <zim/file.h>
-#include <zim/article.h>
-#include <zim/error.h>
+pthread_mutex_t globalLock = PTHREAD_RECURSIVE_MUTEX_INITIALIZER;
 
-/* global variables */
-kiwix::Reader *reader = NULL;
-kiwix::XapianSearcher *searcher = NULL;
-
-static pthread_mutex_t readerLock = PTHREAD_MUTEX_INITIALIZER;
-static pthread_mutex_t searcherLock = PTHREAD_MUTEX_INITIALIZER;
-
-/* c2jni type conversion functions */
-jboolean c2jni(const bool &val) {
-  return val ? JNI_TRUE : JNI_FALSE;
-}
-
-jstring c2jni(const std::string &val, JNIEnv *env) {
-  return env->NewStringUTF(val.c_str());
-}
-
-jint c2jni(const int val) {
-  return (jint)val;
-}
-
-jint c2jni(const unsigned val) {
-  return (unsigned)val;
-}
-
-/* jni2c type conversion functions */
-bool jni2c(const jboolean &val) {
-  return val == JNI_TRUE;
-}
-
-std::string jni2c(const jstring &val, JNIEnv *env) {
-  return std::string(env->GetStringUTFChars(val, 0));
-}
-
-int jni2c(const jint val) {
-  return (int)val;
-}
-
-/* Method to deal with variable passed by reference */
-void setStringObjValue(const std::string &value, const jobject obj, JNIEnv *env) {
-  jclass objClass = env->GetObjectClass(obj);
-  jfieldID objFid = env->GetFieldID(objClass, "value", "Ljava/lang/String;");
-  env->SetObjectField(obj, objFid, c2jni(value, env));
-}
-
-void setIntObjValue(const int value, const jobject obj, JNIEnv *env) {
-  jclass objClass = env->GetObjectClass(obj);
-  jfieldID objFid = env->GetFieldID(objClass, "value", "I");
-  env->SetIntField(obj, objFid, value);
-}
-
-void setBoolObjValue(const bool value, const jobject obj, JNIEnv *env) {
-  jclass objClass = env->GetObjectClass(obj);
-  jfieldID objFid = env->GetFieldID(objClass, "value", "Z");
-  env->SetIntField(obj, objFid, c2jni(value));
-}
-
-/* Kiwix library functions */
-JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getMainPage(JNIEnv *env, jobject obj) {
-  jstring url;
-  
-  pthread_mutex_lock(&readerLock);
-  if (reader != NULL) {
-    try {
-      std::string cUrl = reader->getMainPageUrl();
-      url = c2jni(cUrl, env);
-    } catch (...) {
-      std::cerr << "Unable to get ZIM main page" << std::endl;
-    }
-  }
-  pthread_mutex_unlock(&readerLock);
-  
-  return url;
-}
-
-JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getId(JNIEnv *env, jobject obj) {
-  jstring id;
-  
-  pthread_mutex_lock(&readerLock);
-  if (reader != NULL) {
-    try {
-      std::string cId = reader->getId();
-      id = c2jni(cId, env);
-    } catch (...) {
-      std::cerr << "Unable to get ZIM id" << std::endl;
-    }
-  }
-  pthread_mutex_unlock(&readerLock);
-  
-  return id;
-}
-
-JNIEXPORT jint JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getFileSize(JNIEnv *env, jobject obj) {
-  jint size;
-  
-  pthread_mutex_lock(&readerLock);
-  if (reader != NULL) {
-    try {
-      int cSize = reader->getFileSize();
-      size = c2jni(cSize);
-    } catch (...) {
-      std::cerr << "Unable to get ZIM file size" << std::endl;
-    }
-  }
-  pthread_mutex_unlock(&readerLock);
-  
-  return size;
-}
-
-JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getCreator(JNIEnv *env, jobject obj) {
-  jstring creator;
-  
-  pthread_mutex_lock(&readerLock);
-  if (reader != NULL) {
-    try {
-      std::string cCreator = reader->getCreator();
-      creator = c2jni(cCreator, env);
-    } catch (...) {
-      std::cerr << "Unable to get ZIM creator" << std::endl;
-    }
-  }
-  pthread_mutex_unlock(&readerLock);
-  
-  return creator;
-}
-
-JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getPublisher(JNIEnv *env, jobject obj) {
-  jstring publisher;
-  
-  pthread_mutex_lock(&readerLock);
-  if (reader != NULL) {
-    try {
-      std::string cPublisher = reader->getPublisher();
-      publisher = c2jni(cPublisher, env);
-    } catch (...) {
-      std::cerr << "Unable to get ZIM creator" << std::endl;
-    }
-  }
-  pthread_mutex_unlock(&readerLock);
-  
-  return publisher;
-}
-
-JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getName(JNIEnv *env, jobject obj) {
-  jstring name;
-  
-  pthread_mutex_lock(&readerLock);
-  if (reader != NULL) {
-    try {
-      std::string cName = reader->getName();
-      name = c2jni(cName, env);
-    } catch (...) {
-      std::cerr << "Unable to get ZIM name" << std::endl;
-    }
-  }
-  pthread_mutex_unlock(&readerLock);
-  
-  return name;
-}
-
-
-JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getFavicon(JNIEnv *env, jobject obj) {
-  jstring favicon;
-  
-  pthread_mutex_lock(&readerLock);
-  if (reader != NULL) {
-    try {
-      std::string cContent;
-      std::string cMime;
-      reader->getFavicon(cContent, cMime);
-      favicon = c2jni(base64_encode(reinterpret_cast<const unsigned char*>(cContent.c_str()), cContent.length()), env);
-    } catch (...) {
-      std::cerr << "Unable to get ZIM favicon" << std::endl;
-    }
-  }
-  pthread_mutex_unlock(&readerLock);
-  
-  return favicon;
-}
-
-JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getDate(JNIEnv *env, jobject obj) {
-  jstring date;
-  
-  pthread_mutex_lock(&readerLock);
-  if (reader != NULL) {
-    try {
-      std::string cDate = reader->getDate();
-      date = c2jni(cDate, env);
-    } catch (...) {
-      std::cerr << "Unable to get ZIM date" << std::endl;
-    }
-  }
-  pthread_mutex_unlock(&readerLock);
-  
-  return date;
-}
-
-JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getLanguage(JNIEnv *env, jobject obj) {
-  jstring language;
-  
-  pthread_mutex_lock(&readerLock);
-  if (reader != NULL) {
-    try {
-      std::string cLanguage = reader->getLanguage();
-      language = c2jni(cLanguage, env);
-    } catch (...) {
-      std::cerr << "Unable to get ZIM language" << std::endl;
-    }
-  }
-  pthread_mutex_unlock(&readerLock);
-  
-  return language;
-}
-
-JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getMimeType(JNIEnv *env, jobject obj, jstring url) {
-  jstring mimeType;
-  
-  pthread_mutex_lock(&readerLock);
-  if (reader != NULL) {
-    std::string cUrl = jni2c(url, env);
-    try {
-      std::string cMimeType;
-      reader->getMimeTypeByUrl(cUrl, cMimeType);
-      mimeType = c2jni(cMimeType, env);
-    } catch (...) {
-      std::cerr << "Unable to get mime-type for url " << cUrl << std::endl;
-    }
-  }
-  pthread_mutex_unlock(&readerLock);
-  
-  return mimeType;
-}
-
-JNIEXPORT jboolean JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_loadZIM(JNIEnv *env, jobject obj, jstring path) {
-  jboolean retVal = JNI_TRUE;
-  std::string cPath = jni2c(path, env);
-
-  pthread_mutex_lock(&readerLock);
-  try {
-    if (reader != NULL) delete reader;
-    reader = new kiwix::Reader(cPath);
-  } catch (...) {
-    std::cerr << "Unable to load ZIM " << cPath << std::endl;
-    reader = NULL;
-    retVal = JNI_FALSE;
-  }
-  pthread_mutex_unlock(&readerLock);
-
-  return retVal;
-}
-
-JNIEXPORT jbyteArray JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getContent(JNIEnv *env, jobject obj, jstring url, jobject mimeTypeObj, jobject sizeObj) {
-
-  /* Default values */
-  setStringObjValue("", mimeTypeObj, env);
-  setIntObjValue(0, sizeObj, env);
-  jbyteArray data = env->NewByteArray(0);
-
-  /* Retrieve the content */
-  if (reader != NULL) {
-    std::string cUrl = jni2c(url, env);
-    std::string cData;
-    std::string cMimeType;
-    unsigned int cSize = 0;
-
-    pthread_mutex_lock(&readerLock);
-    try {
-      if (reader->getContentByUrl(cUrl, cData, cSize, cMimeType)) {
-        data = env->NewByteArray(cSize);
-        env->SetByteArrayRegion(data, 0, cSize, reinterpret_cast<const jbyte*>(cData.c_str()));
-        setStringObjValue(cMimeType, mimeTypeObj, env);
-        setIntObjValue(cSize, sizeObj, env);
-      }
-    } catch (...) {
-      std::cerr << "Unable to get content for url " << cUrl << std::endl;
-    }
-    pthread_mutex_unlock(&readerLock);
-  }
-  
-  return data;
-}
-
-JNIEXPORT jboolean JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_searchSuggestions
-(JNIEnv *env, jobject obj, jstring prefix, jint count) {
-  jboolean retVal = JNI_FALSE;
-  std::string cPrefix = jni2c(prefix, env);
-  unsigned int cCount = jni2c(count);
-
-  pthread_mutex_lock(&readerLock);
-  try {
-    if (reader != NULL) {
-      if (reader->searchSuggestionsSmart(cPrefix, cCount)) {
-        retVal = JNI_TRUE;
-      }
-    }
-  } catch (...) {
-    std::cerr << "Unable to search suggestions for pattern " << cPrefix << std::endl;
-  }
-  pthread_mutex_unlock(&readerLock);
-
-  return retVal;
-}
-
-JNIEXPORT jboolean JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getNextSuggestion
-(JNIEnv *env, jobject obj, jobject titleObj) {
-  jboolean retVal = JNI_FALSE;
-  std::string cTitle;
-
-  pthread_mutex_lock(&readerLock);
-  try {
-    if (reader != NULL) {
-      if (reader->getNextSuggestion(cTitle)) {
-        setStringObjValue(cTitle, titleObj, env);
-        retVal = JNI_TRUE;
-      }
-    }
-  } catch (...) {
-    std::cerr << "Unable to get next suggestion" << std::endl;
-  }
-  pthread_mutex_unlock(&readerLock);
-
-  return retVal;
-}
-
-JNIEXPORT jboolean JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getPageUrlFromTitle
-(JNIEnv *env, jobject obj, jstring title, jobject urlObj) {
-  jboolean retVal = JNI_FALSE;
-  std::string cTitle = jni2c(title, env);
-  std::string cUrl;
-
-  pthread_mutex_lock(&readerLock);
-  try {
-    if (reader != NULL) {
-      if (reader->getPageUrlFromTitle(cTitle, cUrl)) {
-        setStringObjValue(cUrl, urlObj, env);
-        retVal = JNI_TRUE;
-      }
-    }
-  } catch (...) {
-    std::cerr << "Unable to get URL for title " << cTitle << std::endl;
-  }
-  pthread_mutex_unlock(&readerLock);
-    
-  return retVal;
-}
-
-JNIEXPORT jboolean JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getTitle
-(JNIEnv *env , jobject obj, jobject titleObj) {
-  jboolean retVal = JNI_FALSE;
-  std::string cTitle;
-
-  pthread_mutex_lock(&readerLock);
-  try {
-    if (reader != NULL) {
-      std::string cTitle = reader->getTitle();
-      setStringObjValue(cTitle, titleObj, env);
-      retVal = JNI_TRUE;
-    }
-  } catch (...) {
-    std::cerr << "Unable to get ZIM title" << std::endl;
-  }
-  pthread_mutex_unlock(&readerLock);
-
-  return retVal;
-
-}
-
-JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getDescription(JNIEnv *env, jobject obj) {
-  jstring description;
-  
-  pthread_mutex_lock(&readerLock);
-  if (reader != NULL) {
-    try {
-      std::string cDescription = reader->getDescription();
-      description = c2jni(cDescription, env);
-    } catch (...) {
-      std::cerr << "Unable to get ZIM description" << std::endl;
-    }
-  }
-  pthread_mutex_unlock(&readerLock);
-  
-  return description;
-}
-
-JNIEXPORT jboolean JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_getRandomPage
-(JNIEnv *env, jobject obj, jobject urlObj) {
-  jboolean retVal = JNI_FALSE;
-  std::string cUrl;
-
-  pthread_mutex_lock(&readerLock);
-  try {
-    if (reader != NULL) {
-      std::string cUrl = reader->getRandomPageUrl();
-      setStringObjValue(cUrl, urlObj, env);
-      retVal = JNI_TRUE;
-    }
-  } catch (...) {
-    std::cerr << "Unable to get random page" << std::endl;
-  }
-  pthread_mutex_unlock(&readerLock);
-
-  return retVal;
-}
-
-JNIEXPORT void JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_setDataDirectory
-  (JNIEnv *env, jobject obj, jstring dirStr) {
+JNIEXPORT void JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_setDataDirectory(
+    JNIEnv* env, jobject obj, jstring dirStr)
+{
   std::string cPath = jni2c(dirStr, env);
 
-  pthread_mutex_lock(&readerLock); 
+  Lock l;
   try {
     u_setDataDirectory(cPath.c_str());
   } catch (...) {
     std::cerr << "Unable to set data directory " << cPath << std::endl;
   }
-  pthread_mutex_unlock(&readerLock);
 }
-
-JNIEXPORT jboolean JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_loadFulltextIndex(JNIEnv *env, jobject obj, jstring path) {
-  jboolean retVal = JNI_TRUE;
-  std::string cPath = jni2c(path, env);
-
-  pthread_mutex_lock(&searcherLock);
-  searcher = NULL;
-  try {
-    if (searcher != NULL) delete searcher;
-    searcher = new kiwix::XapianSearcher(cPath);
-  } catch (...) {
-    searcher = NULL;
-    retVal = JNI_FALSE;
-    std::cerr << "Unable to load full text index " << cPath << std::endl;
-  }
-  pthread_mutex_unlock(&searcherLock);
-
-  return retVal;
-}
-
-JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_indexedQuery
-  (JNIEnv *env, jclass obj, jstring query, jint count) {
-  std::string cQuery = jni2c(query, env);
-  unsigned int cCount = jni2c(count);
-  std::string url;
-  std::string title;
-  std::string result;
-  unsigned int score;
-      
-  pthread_mutex_lock(&searcherLock);
-  try {
-    if (searcher != NULL) {
-      searcher->search(cQuery, 0, count);
-      while (searcher->getNextResult(url, title, score) &&
-	     !title.empty() &&
-	     !url.empty()) {
-	result += title + "\n";
-      }
-    }
-  } catch (...) {
-    std::cerr << "Unable to make indexed query " << cQuery << std::endl;
-  }
-  pthread_mutex_unlock(&searcherLock);
-
-  return env->NewStringUTF(result.c_str());
-}
-
-

src/android/kiwixreader.cpp

@@ -0,0 +1,424 @@
+/*
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright (C) 2017 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+
+#include <jni.h>
+#include <zim/file.h>
+#include <android/log.h>
+#include "org_kiwix_kiwixlib_JNIKiwixReader.h"
+
+#include "common/base64.h"
+#include "reader.h"
+#include "utils.h"
+
+/* Kiwix Reader JNI functions */
+JNIEXPORT jlong JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getNativeReader(
+    JNIEnv* env, jobject obj, jstring filename)
+{
+  std::string cPath = jni2c(filename, env);
+
+  __android_log_print(ANDROID_LOG_INFO, "kiwix", "Attempting to create reader with: %s", cPath.c_str());
+  Lock l;
+  try {
+    kiwix::Reader* reader = new kiwix::Reader(cPath);
+    return reinterpret_cast<jlong>(new Handle<kiwix::Reader>(reader));
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", "Error opening ZIM file");
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", e.what());
+    return 0;
+  }
+}
+
+JNIEXPORT void JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_dispose(JNIEnv* env, jobject obj)
+{
+  Handle<kiwix::Reader>::dispose(env, obj);
+}
+
+#define READER (Handle<kiwix::Reader>::getHandle(env, obj))
+
+/* Kiwix library functions */
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getMainPage(JNIEnv* env, jobject obj)
+{
+  jstring url;
+
+  try {
+    std::string cUrl = READER->getMainPage().getPath();
+    url = c2jni(cUrl, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM main page");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    url = NULL;
+  }
+  return url;
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getId(JNIEnv* env, jobject obj)
+{
+  jstring id;
+
+  try {
+    std::string cId = READER->getId();
+    id = c2jni(cId, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM id");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    id = NULL;
+  }
+
+  return id;
+}
+
+JNIEXPORT jint JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getFileSize(JNIEnv* env, jobject obj)
+{
+  jint size;
+
+  try {
+    int cSize = READER->getFileSize();
+    size = c2jni(cSize);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM file size");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+  }
+
+  return size;
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getCreator(JNIEnv* env, jobject obj)
+{
+  jstring creator;
+
+  try {
+    std::string cCreator = READER->getCreator();
+    creator = c2jni(cCreator, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM creator");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    creator = NULL;
+  }
+
+  return creator;
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getPublisher(JNIEnv* env, jobject obj)
+{
+  jstring publisher;
+
+  try {
+    std::string cPublisher = READER->getPublisher();
+    publisher = c2jni(cPublisher, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM publish");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    publisher = NULL;
+  }
+  return publisher;
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getName(JNIEnv* env, jobject obj)
+{
+  jstring name;
+
+  try {
+    std::string cName = READER->getName();
+    name = c2jni(cName, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM name");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    name = NULL;
+  }
+  return name;
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getFavicon(JNIEnv* env, jobject obj)
+{
+  jstring favicon;
+
+  try {
+    std::string cContent;
+    std::string cMime;
+    READER->getFavicon(cContent, cMime);
+    favicon = c2jni(
+        base64_encode(cContent),
+        env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM favicon");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    favicon = NULL;
+  }
+  return favicon;
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getDate(JNIEnv* env, jobject obj)
+{
+  jstring date;
+
+  try {
+    std::string cDate = READER->getDate();
+    date = c2jni(cDate, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM date");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    date = NULL;
+  }
+  return date;
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getLanguage(JNIEnv* env, jobject obj)
+{
+  jstring language;
+
+  try {
+    std::string cLanguage = READER->getLanguage();
+    language = c2jni(cLanguage, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM language");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    language = NULL;
+  }
+
+  return language;
+}
+
+JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getMimeType(
+    JNIEnv* env, jobject obj, jstring url)
+{
+  jstring mimeType;
+
+  std::string cUrl = jni2c(url, env);
+  try {
+    auto entry = READER->getEntryFromEncodedPath(cUrl);
+    auto cMimeType = entry.getMimetype();
+    mimeType = c2jni(cMimeType, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get mime-type for url: %s", cUrl.c_str());
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    mimeType = NULL;
+  }
+  return mimeType;
+}
+
+JNIEXPORT jbyteArray JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getContent(
+    JNIEnv* env, jobject obj, jstring url, jobject titleObj, jobject mimeTypeObj, jobject sizeObj)
+{
+  /* Default values */
+  setStringObjValue("", titleObj, env);
+  setStringObjValue("", mimeTypeObj, env);
+  setIntObjValue(0, sizeObj, env);
+  jbyteArray data = env->NewByteArray(0);
+
+  /* Retrieve the content */
+  std::string cUrl = jni2c(url, env);
+  unsigned int cSize = 0;
+
+  try {
+    auto entry = READER->getEntryFromEncodedPath(cUrl);
+    entry = entry.getFinalEntry();
+    cSize = entry.getSize();
+    setIntObjValue(cSize, sizeObj, env);
+
+    data = env->NewByteArray(cSize);
+    env->SetByteArrayRegion(
+        data, 0, cSize, reinterpret_cast<const jbyte*>(entry.getBlob().data()));
+
+    setStringObjValue(entry.getMimetype(), mimeTypeObj, env);
+    setStringObjValue(entry.getTitle(), titleObj, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get content for url: %s", cUrl.c_str());
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+  }
+
+  return data;
+}
+
+JNIEXPORT jbyteArray JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getContentPart(
+    JNIEnv* env, jobject obj, jstring url, jint offset, jint len, jobject sizeObj)
+{
+  jbyteArray data = env->NewByteArray(0);
+  setIntObjValue(0, sizeObj, env);
+
+  /* Default values */
+  /* Retrieve the content */
+  std::string cUrl = jni2c(url, env);
+  unsigned int cOffset = jni2c(offset);
+  unsigned int cLen = jni2c(len);
+  try {
+    auto entry = READER->getEntryFromEncodedPath(cUrl);
+    entry = entry.getFinalEntry();
+
+    if (cLen == 0) {
+      setIntObjValue(entry.getSize(), sizeObj, env);
+    } else if (cOffset+cLen < entry.getSize()) {
+      auto blob = entry.getBlob(cOffset, cLen);
+      data = env->NewByteArray(cLen);
+      env->SetByteArrayRegion(
+          data, 0, cLen, reinterpret_cast<const jbyte*>(blob.data()));
+      setIntObjValue(cLen, sizeObj, env);
+    }
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get partial content for url: %s (%u : %u)", cUrl.c_str(), cOffset, cLen);
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+  }
+  return data;
+}
+
+JNIEXPORT jobject JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getDirectAccessInformation(
+    JNIEnv* env, jobject obj, jstring url)
+{
+   jclass classPair = env->FindClass("org/kiwix/kiwixlib/Pair");
+   jmethodID midPairinit = env->GetMethodID(classPair, "<init>", "()V");
+   jobject pair = env->NewObject(classPair, midPairinit);
+   setPairObjValue("", 0, pair, env);
+
+   std::string cUrl = jni2c(url, env);
+   try {
+    auto entry = READER->getEntryFromEncodedPath(cUrl);
+    entry = entry.getFinalEntry();
+    auto part_info = entry.getDirectAccessInfo();
+    setPairObjValue(part_info.first, part_info.second, pair, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get direct access info for url: %s", cUrl.c_str());
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+  }
+  return pair;
+}
+
+JNIEXPORT jboolean JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_searchSuggestions(JNIEnv* env,
+                                                         jobject obj,
+                                                         jstring prefix,
+                                                         jint count)
+{
+  jboolean retVal = JNI_FALSE;
+  std::string cPrefix = jni2c(prefix, env);
+  unsigned int cCount = jni2c(count);
+
+  try {
+    if (READER->searchSuggestionsSmart(cPrefix, cCount)) {
+      retVal = JNI_TRUE;
+    }
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", "Unable to get search results for pattern: %s", cPrefix.c_str());
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", e.what());
+  }
+
+  return retVal;
+}
+
+JNIEXPORT jboolean JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getNextSuggestion(JNIEnv* env,
+                                                         jobject obj,
+                                                         jobject titleObj)
+{
+  jboolean retVal = JNI_FALSE;
+  std::string cTitle;
+
+  try {
+    if (READER->getNextSuggestion(cTitle)) {
+      setStringObjValue(cTitle, titleObj, env);
+      retVal = JNI_TRUE;
+    }
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", "Unable to get next suggestion");
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", e.what());
+  }
+
+  return retVal;
+}
+
+JNIEXPORT jboolean JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getPageUrlFromTitle(JNIEnv* env,
+                                                           jobject obj,
+                                                           jstring title,
+                                                           jobject urlObj)
+{
+  std::string cTitle = jni2c(title, env);
+
+  try {
+    auto entry = READER->getEntryFromTitle(cTitle);
+    entry = entry.getFinalEntry();
+    setStringObjValue(entry.getPath(), urlObj, env);
+    return JNI_TRUE;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", "Unable to get url for title %s: ", cTitle.c_str());
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", e.what());
+  }
+
+  return JNI_FALSE;
+}
+
+JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getTitle(
+    JNIEnv* env, jobject obj)
+{
+  jstring title;
+
+  try {
+    std::string cTitle = READER->getTitle();
+    title = c2jni(cTitle, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get zim title");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    title = NULL;
+  }
+  return title;
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixReader_getDescription(JNIEnv* env, jobject obj)
+{
+  jstring description;
+
+  try {
+    std::string cDescription = READER->getDescription();
+    description = c2jni(cDescription, env);
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get zim description");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    description = NULL;
+  }
+  return description;
+}
+
+JNIEXPORT jboolean JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getRandomPage(
+    JNIEnv* env, jobject obj, jobject urlObj)
+{
+  jboolean retVal = JNI_FALSE;
+  std::string cUrl;
+
+  try {
+    std::string cUrl = READER->getRandomPage().getPath();
+    setStringObjValue(cUrl, urlObj, env);
+    retVal = JNI_TRUE;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get random page");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+  }
+  return retVal;
+}

src/android/kiwixsearcher.cpp

@@ -0,0 +1,124 @@
+/*
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright (C) 2017 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+
+#include <zim/file.h>
+#include "org_kiwix_kiwixlib_JNIKiwixSearcher.h"
+#include "org_kiwix_kiwixlib_JNIKiwixSearcher_Result.h"
+
+#include "reader.h"
+#include "searcher.h"
+#include "utils.h"
+
+#define SEARCHER (Handle<kiwix::Searcher>::getHandle(env, obj))
+#define RESULT (Handle<kiwix::Result>::getHandle(env, obj))
+
+
+JNIEXPORT void JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixSearcher_dispose(JNIEnv* env, jobject obj)
+{
+  Handle<kiwix::Searcher>::dispose(env, obj);
+}
+
+/* Kiwix Reader JNI functions */
+JNIEXPORT jlong JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixSearcher_getNativeHandle(JNIEnv* env,
+                                                          jobject obj)
+{
+  kiwix::Searcher* searcher = new kiwix::Searcher();
+  return reinterpret_cast<jlong>(new Handle<kiwix::Searcher>(searcher));
+}
+
+/* Kiwix library functions */
+JNIEXPORT void JNICALL Java_org_kiwix_kiwixlib_JNIKiwixSearcher_addReader(
+    JNIEnv* env, jobject obj, jobject reader)
+{
+  auto searcher = SEARCHER;
+
+  searcher->add_reader(*(Handle<kiwix::Reader>::getHandle(env, reader)), "");
+}
+
+JNIEXPORT void JNICALL Java_org_kiwix_kiwixlib_JNIKiwixSearcher_search(
+    JNIEnv* env, jobject obj, jstring query, jint count)
+{
+  std::string cquery = jni2c(query, env);
+  unsigned int ccount = jni2c(count);
+
+  SEARCHER->search(cquery, 0, ccount);
+}
+
+JNIEXPORT jobject JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixSearcher_getNextResult(JNIEnv* env,
+                                                         jobject obj)
+{
+  jobject result = nullptr;
+
+  kiwix::Result* cresult = SEARCHER->getNextResult();
+  if (cresult != nullptr) {
+    jclass resultclass
+        = env->FindClass("org/kiwix/kiwixlib/JNIKiwixSearcher$Result");
+    jmethodID ctor = env->GetMethodID(
+        resultclass, "<init>", "(Lorg/kiwix/kiwixlib/JNIKiwixSearcher;JLorg/kiwix/kiwixlib/JNIKiwixSearcher;)V");
+    result = env->NewObject(resultclass, ctor, obj, reinterpret_cast<jlong>(new Handle<kiwix::Result>(cresult)), obj);
+  }
+  return result;
+}
+
+JNIEXPORT void JNICALL Java_org_kiwix_kiwixlib_JNIKiwixSearcher_00024Result_dispose(
+    JNIEnv* env, jobject obj)
+{
+  Handle<kiwix::Result>::dispose(env, obj);
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixSearcher_00024Result_getUrl(JNIEnv* env,
+                                                        jobject obj)
+{
+  try {
+    return c2jni(RESULT->get_url(), env);
+  } catch (...) {
+    return nullptr;
+  }
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixSearcher_00024Result_getTitle(JNIEnv* env,
+                                                          jobject obj)
+{
+  try {
+    return c2jni(RESULT->get_title(), env);
+  } catch (...) {
+    return nullptr;
+  }
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixSearcher_00024Result_getSnippet(JNIEnv* env,
+                                                            jobject obj)
+{
+  return c2jni(RESULT->get_snippet(), env);
+}
+
+JNIEXPORT jstring JNICALL
+Java_org_kiwix_kiwixlib_JNIKiwixSearcher_00024Result_getContent(JNIEnv* env,
+                                                            jobject obj)
+{
+  return c2jni(RESULT->get_content(), env);
+}

src/android/meson.build

@@ -1,13 +1,26 @@
 
-jni_generator = find_program('gen_kiwix.sh')
-
 kiwix_jni = custom_target('jni',
   input: ['org/kiwix/kiwixlib/JNIKiwix.java',
+          'org/kiwix/kiwixlib/JNIKiwixReader.java',
+          'org/kiwix/kiwixlib/JNIKiwixSearcher.java',
           'org/kiwix/kiwixlib/JNIKiwixInt.java',
           'org/kiwix/kiwixlib/JNIKiwixString.java',
-          'org/kiwix/kiwixlib/JNIKiwixBool.java'],
-  output: ['org_kiwix_kiwixlib_JNIKiwix.h'],
-  command:[jni_generator, '@INPUT@']
+          'org/kiwix/kiwixlib/JNIKiwixBool.java',
+          'org/kiwix/kiwixlib/JNIKiwixException.java',
+          'org/kiwix/kiwixlib/Pair.java'],
+  output: ['org_kiwix_kiwixlib_JNIKiwix.h',
+           'org_kiwix_kiwixlib_JNIKiwixReader.h',
+           'org_kiwix_kiwixlib_JNIKiwixSearcher.h',
+           'org_kiwix_kiwixlib_JNIKiwixSearcher_Result.h'],
+  command:['javac', '-d', '@OUTDIR@', '-h', '@OUTDIR@', '@INPUT@']
 )
 
-kiwix_sources += ['android/kiwix.cpp', kiwix_jni]
+kiwix_sources += [
+    'android/kiwix.cpp',
+    'android/kiwixreader.cpp',
+    'android/kiwixsearcher.cpp',
+    kiwix_jni]
+
+install_subdir('org', install_dir: 'kiwix-lib/java')
+install_subdir('res', install_dir: 'kiwix-lib')
+install_data('AndroidManifest.xml', install_dir: 'kiwix-lib')

src/android/org/kiwix/kiwixlib/JNIKiwix.java

@@ -1,5 +1,6 @@
 /*
- * Copyright 2013
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright (C) 2017 Matthieu Gautier <mgautier@kymeria.fr>
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU  General Public License as published by
@@ -19,59 +20,12 @@
 
 package org.kiwix.kiwixlib;
 
+import org.kiwix.kiwixlib.JNIKiwixReader;
 import org.kiwix.kiwixlib.JNIKiwixString;
-import org.kiwix.kiwixlib.JNIKiwixBool;
-import org.kiwix.kiwixlib.JNIKiwixInt;
 
-public class JNIKiwix {
-
-  static {
-    System.loadLibrary("kiwix");
-  }
-
-  public native String getMainPage();
-
-  public native String getId();
-
-  public native String getLanguage();
-
-  public native String getMimeType(String url);
-
-  public native boolean loadZIM(String path);
-
-    public native boolean loadFulltextIndex(String path);
-    
-  public native byte[] getContent(String url, JNIKiwixString mimeType, JNIKiwixInt size);
-
-  public native boolean searchSuggestions(String prefix, int count);
-
-  public native boolean getNextSuggestion(JNIKiwixString title);
-
-  public native boolean getPageUrlFromTitle(String title, JNIKiwixString url);
-
-  public native boolean getTitle(JNIKiwixString title);
-
-  public native String getDescription();
-
-  public native String getDate();
-
-  public native String getFavicon();
-
-  public native String getCreator();
-
-  public native String getPublisher();
-
-  public native String getName();
-
-  public native int getFileSize();
-
-  public native int getArticleCount();
-
-  public native int getMediaCount();
-
-  public native boolean getRandomPage(JNIKiwixString url);
+public class JNIKiwix
+{
+  static { System.loadLibrary("kiwix"); }
 
   public native void setDataDirectory(String icuDataDir);
-
-  public static native String indexedQuery(String db, int count);
 }

src/android/org/kiwix/kiwixlib/JNIKiwixBool.java

@@ -1,5 +1,5 @@
 /*
- * Copyright 2013
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU  General Public License as published by
@@ -19,7 +19,7 @@
 
 package org.kiwix.kiwixlib;
 
-public class JNIKiwixBool {
-
+public class JNIKiwixBool
+{
   public boolean value;
 }

src/android/org/kiwix/kiwixlib/JNIKiwixException.java

@@ -0,0 +1,27 @@
+/*
+ * Copyright (C) 2017 Matthieu Gautier <mgautier@kymeria.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+package org.kiwix.kiwixlib;
+
+public class JNIKiwixException extends Exception
+{
+  public JNIKiwixException(String message) {
+    super(message);
+  }
+}

src/android/org/kiwix/kiwixlib/JNIKiwixInt.java

@@ -1,5 +1,5 @@
 /*
- * Copyright 2013
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU  General Public License as published by
@@ -19,8 +19,7 @@
 
 package org.kiwix.kiwixlib;
 
-public class JNIKiwixInt {
-
+public class JNIKiwixInt
+{
   public int value;
 }
-

src/android/org/kiwix/kiwixlib/JNIKiwixReader.java

@@ -0,0 +1,127 @@
+/*
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright (C) 2017 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+package org.kiwix.kiwixlib;
+
+import org.kiwix.kiwixlib.JNIKiwixException;
+import org.kiwix.kiwixlib.JNIKiwixString;
+import org.kiwix.kiwixlib.JNIKiwixInt;
+import org.kiwix.kiwixlib.JNIKiwixSearcher;
+import org.kiwix.kiwixlib.Pair;
+
+public class JNIKiwixReader
+{
+  public native String getMainPage();
+
+  public native String getTitle();
+
+  public native String getId();
+
+  public native String getLanguage();
+
+  public native String getMimeType(String url);
+
+  public native byte[] getContent(String url,
+                                  JNIKiwixString title,
+                                  JNIKiwixString mimeType,
+                                  JNIKiwixInt size);
+
+  /**
+   * getContentPart.
+   *
+   * Get only a part of the content of the article.
+   * Return a byte array of `len` size starting from offset `offset`.
+   * Set `size` to the number of bytes read
+   * (`len` if everything is ok, 0 in case of error).
+   * If `len` == 0, no bytes are read but `size` is set to the total size of the
+   * article.
+   */
+  public native byte[] getContentPart(String url,
+                                      int offest,
+                                      int len,
+                                      JNIKiwixInt size);
+
+  /**
+   * getDirectAccessInformation.
+   *
+   * Return information giving where the content is located in the zim file.
+   *
+   * Some contents (binary content) are stored uncompressed in the zim file.
+   * Knowing this information, it could be interesting to directly open
+   * the zim file (or zim part) and directly read the content from it (and so
+   * bypassing the libzim).
+   *
+   * Return a `Pair` (filename, offset) where the content is located.
+   *
+   * If the content cannot be directly accessed (content is compressed or zim
+   * file is cut in the middle of the content), the filename is an empty string
+   * and offset is zero.
+   */
+  public native Pair getDirectAccessInformation(String url);
+
+  public native boolean searchSuggestions(String prefix, int count);
+
+  public native boolean getNextSuggestion(JNIKiwixString title);
+
+  public native boolean getPageUrlFromTitle(String title, JNIKiwixString url);
+
+  public native String getDescription();
+
+  public native String getDate();
+
+  public native String getFavicon();
+
+  public native String getCreator();
+
+  public native String getPublisher();
+
+  public native String getName();
+
+  public native int getFileSize();
+
+  public native int getArticleCount();
+
+  public native int getMediaCount();
+
+  public native boolean getRandomPage(JNIKiwixString url);
+
+  public JNIKiwixSearcher search(String query, int count)
+  {
+    JNIKiwixSearcher searcher = new JNIKiwixSearcher();
+    searcher.addKiwixReader(this);
+    searcher.search(query, count);
+    return searcher;
+  }
+
+  public JNIKiwixReader(String filename) throws JNIKiwixException
+  {
+    nativeHandle = getNativeReader(filename);
+    if (nativeHandle == 0) {
+        throw new JNIKiwixException("Cannot open zimfile "+filename);
+    }
+  }
+  public JNIKiwixReader() {
+
+  }
+  public native void dispose();
+
+  private native long getNativeReader(String filename);
+  private long nativeHandle;
+}

src/android/org/kiwix/kiwixlib/JNIKiwixSearcher.java

@@ -0,0 +1,67 @@
+/*
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright (C) 2017 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+
+package org.kiwix.kiwixlib;
+
+import org.kiwix.kiwixlib.JNIKiwixReader;
+import java.util.Vector;
+
+public class JNIKiwixSearcher
+{
+  public class Result
+  {
+    private long nativeHandle;
+    private JNIKiwixSearcher searcher;
+    public Result(long handle, JNIKiwixSearcher _searcher)
+    {
+      nativeHandle = handle;
+      searcher = _searcher;
+    }
+    public native String getUrl();
+    public native String getTitle();
+    public native String getContent();
+    public native String getSnippet();
+    public native void dispose();
+  }
+
+  public JNIKiwixSearcher()
+  {
+    nativeHandle = getNativeHandle();
+    usedReaders = new Vector();
+  }
+  public native void dispose();
+
+  private native long getNativeHandle();
+  private long nativeHandle;
+  private Vector usedReaders;
+
+  public native void addReader(JNIKiwixReader reader);
+  public void addKiwixReader(JNIKiwixReader reader)
+  {
+    addReader(reader);
+    usedReaders.addElement(reader);
+  };
+
+  public native void search(String query, int count);
+
+  public native Result getNextResult();
+  public native boolean hasMoreResult();
+}

src/android/org/kiwix/kiwixlib/JNIKiwixString.java

@@ -1,5 +1,5 @@
 /*
- * Copyright 2013
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU  General Public License as published by
@@ -19,7 +19,7 @@
 
 package org.kiwix.kiwixlib;
 
-public class JNIKiwixString {
-
+public class JNIKiwixString
+{
   public String value;
 }

src/android/org/kiwix/kiwixlib/Pair.java

@@ -0,0 +1,26 @@
+/*
+ * Copyright (C) 2017 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+package org.kiwix.kiwixlib;
+
+public class Pair
+{
+  public String filename;
+  public int offset;
+}

src/android/res/values/strings.xml

@@ -0,0 +1,3 @@
+<resources>
+  <string name="app_name">Kiwix Lib</string>
+</resources>

src/android/utils.h

@@ -0,0 +1,150 @@
+/*
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright (C) 2017 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+
+#ifndef _ANDROID_JNI_UTILS_H
+#define _ANDROID_JNI_UTILS_H
+
+#include <jni.h>
+
+#include <pthread.h>
+#include <string>
+
+extern pthread_mutex_t globalLock;
+
+inline jfieldID getHandleField(JNIEnv* env, jobject obj)
+{
+  jclass c = env->GetObjectClass(obj);
+  // J is the type signature for long:
+  return env->GetFieldID(c, "nativeHandle", "J");
+}
+
+class Lock
+{
+ protected:
+  pthread_mutex_t* lock;
+
+ public:
+  Lock() : lock(&globalLock) { pthread_mutex_lock(lock); }
+  Lock(const Lock&) = delete;
+  Lock& operator=(const Lock&) = delete;
+  Lock(Lock&& other) : lock(&globalLock) { other.lock = nullptr; }
+  virtual ~Lock()
+  {
+    if (lock) {
+      pthread_mutex_unlock(lock);
+    }
+  }
+};
+
+template <class T>
+class LockedHandle;
+
+template <class T>
+class Handle
+{
+ protected:
+  T* h;
+
+ public:
+  Handle(T* h) : h(h){};
+
+  // No destructor. This must and will be handled by dispose method.
+
+  static LockedHandle<T> getHandle(JNIEnv* env, jobject obj)
+  {
+    jlong handle = env->GetLongField(obj, getHandleField(env, obj));
+    return LockedHandle<T>(reinterpret_cast<Handle<T>*>(handle));
+  }
+
+  static void dispose(JNIEnv* env, jobject obj)
+  {
+    auto lHandle = getHandle(env, obj);
+    auto handle = lHandle.h;
+    delete handle->h;
+    delete handle;
+  }
+  friend class LockedHandle<T>;
+};
+
+template <class T>
+struct LockedHandle : public Lock {
+  Handle<T>* h;
+  LockedHandle(Handle<T>* h) : h(h) {}
+  T* operator->() { return h->h; }
+  T* operator*() { return h->h; }
+  operator bool() const { return (h->h != nullptr); }
+};
+
+/* c2jni type conversion functions */
+inline jboolean c2jni(const bool& val) { return val ? JNI_TRUE : JNI_FALSE; }
+inline jstring c2jni(const std::string& val, JNIEnv* env)
+{
+  return env->NewStringUTF(val.c_str());
+}
+
+inline jint c2jni(const int val) { return (jint)val; }
+inline jint c2jni(const unsigned val) { return (unsigned)val; }
+/* jni2c type conversion functions */
+inline bool jni2c(const jboolean& val) { return val == JNI_TRUE; }
+inline std::string jni2c(const jstring& val, JNIEnv* env)
+{
+  const char* chars = env->GetStringUTFChars(val, 0);
+  std::string ret(chars);
+  env->ReleaseStringUTFChars(val, chars);
+  return ret;
+}
+
+inline int jni2c(const jint val) { return (int)val; }
+/* Method to deal with variable passed by reference */
+inline void setStringObjValue(const std::string& value,
+                              const jobject obj,
+                              JNIEnv* env)
+{
+  jclass objClass = env->GetObjectClass(obj);
+  jfieldID objFid = env->GetFieldID(objClass, "value", "Ljava/lang/String;");
+  env->SetObjectField(obj, objFid, c2jni(value, env));
+}
+
+inline void setIntObjValue(const int value, const jobject obj, JNIEnv* env)
+{
+  jclass objClass = env->GetObjectClass(obj);
+  jfieldID objFid = env->GetFieldID(objClass, "value", "I");
+  env->SetIntField(obj, objFid, value);
+}
+
+inline void setBoolObjValue(const bool value, const jobject obj, JNIEnv* env)
+{
+  jclass objClass = env->GetObjectClass(obj);
+  jfieldID objFid = env->GetFieldID(objClass, "value", "Z");
+  env->SetIntField(obj, objFid, c2jni(value));
+}
+
+inline void setPairObjValue(const std::string& filename, const int offset,
+                            const jobject obj, JNIEnv* env)
+{
+  jclass objClass = env->GetObjectClass(obj);
+  jfieldID filenameFid = env->GetFieldID(objClass, "filename", "Ljava/lang/String;");
+  env->SetObjectField(obj, filenameFid, c2jni(filename, env));
+  jfieldID offsetFid = env->GetFieldID(objClass, "offset", "I");
+  env->SetIntField(obj, offsetFid, offset);
+}
+
+#endif // _ANDROID_JNI_UTILS_H

src/aria2.cpp

@@ -0,0 +1,190 @@
+
+
+#include "aria2.h"
+#include "xmlrpc.h"
+#include <sstream>
+#include <thread>
+#include <chrono>
+#include <common/otherTools.h>
+#include <common/pathTools.h>
+#include <downloader.h> // For AriaError
+
+namespace kiwix {
+
+Aria2::Aria2():
+  mp_aria(nullptr),
+  m_port(42042),
+  m_secret("kiwixariarpc"),
+  mp_curl(nullptr),
+  m_lock(PTHREAD_MUTEX_INITIALIZER)
+{
+  m_downloadDir = getDataDirectory();
+  std::vector<const char*> callCmd;
+
+  std::string rpc_port = "--rpc-listen-port=" + to_string(m_port);
+  std::string download_dir = "--dir=" + getDataDirectory();
+  std::string session_file = appendToDirectory(getDataDirectory(), "kiwix.session");
+  std::string session = "--save-session=" + session_file;
+  std::string inputFile = "--input-file=" + session_file;
+//  std::string log_dir = "--log=\"" + logDir + "\"";
+#ifdef _WIN32
+  int pid = GetCurrentProcessId();
+#else
+  pid_t pid = getpid();
+#endif
+  std::string stop_with_pid = "--stop-with-process=" + to_string(pid);
+  std::string rpc_secret = "--rpc-secret=" + m_secret;
+  m_secret = "token:"+m_secret;
+
+#ifdef _WIN32
+  callCmd.push_back("aria2c.exe");
+#else
+  callCmd.push_back("aria2c");
+#endif
+  callCmd.push_back("--enable-rpc");
+  callCmd.push_back(rpc_secret.c_str());
+  callCmd.push_back(rpc_port.c_str());
+  callCmd.push_back(download_dir.c_str());
+  if (fileExists(session_file)) {
+    callCmd.push_back(inputFile.c_str());
+  }
+  callCmd.push_back(session.c_str());
+//  callCmd.push_back(log_dir.c_str());
+  callCmd.push_back("--auto-save-interval=10");
+  callCmd.push_back(stop_with_pid.c_str());
+  callCmd.push_back("--allow-overwrite=true");
+  callCmd.push_back("--dht-entry-point=router.bittorrent.com:6881");
+  callCmd.push_back("--dht-entry-point6=router.bittorrent.com:6881");
+  callCmd.push_back("--quiet=true");
+  callCmd.push_back("--bt-enable-lpd=true");
+  callCmd.push_back("--always-resume=true");
+  callCmd.push_back("--max-concurrent-downloads=42");
+  callCmd.push_back("--rpc-max-request-size=6M");
+  callCmd.push_back("--file-allocation=none");
+  mp_aria = Subprocess::run(callCmd);
+  mp_curl = curl_easy_init();
+  curl_easy_setopt(mp_curl, CURLOPT_URL, "http://localhost/rpc");
+  curl_easy_setopt(mp_curl, CURLOPT_PORT, m_port);
+  curl_easy_setopt(mp_curl, CURLOPT_POST, 1L);
+
+  int watchdog = 50;
+  while(--watchdog) {
+    std::this_thread::sleep_for(std::chrono::microseconds(100));
+    auto res = curl_easy_perform(mp_curl);
+    if (res == CURLE_OK) {
+      break;
+    }
+  }
+  if (!watchdog) {
+    curl_easy_cleanup(mp_curl);
+    throw std::runtime_error("Cannot connect to aria2c rpc");
+  }
+}
+
+Aria2::~Aria2()
+{
+  curl_easy_cleanup(mp_curl);
+}
+
+void Aria2::close()
+{
+  saveSession();
+  shutdown();
+}
+
+size_t write_callback_to_iss(char* ptr, size_t size, size_t nmemb, void* userdata)
+{
+  auto str = static_cast<std::stringstream*>(userdata);
+  str->write(ptr, nmemb);
+  return nmemb;
+}
+
+std::string Aria2::doRequest(const MethodCall& methodCall)
+{
+  pthread_mutex_lock(&m_lock);
+  auto requestContent = methodCall.toString();
+  std::stringstream stringstream;
+  CURLcode res;
+  curl_easy_setopt(mp_curl, CURLOPT_POSTFIELDSIZE, requestContent.size());
+  curl_easy_setopt(mp_curl, CURLOPT_POSTFIELDS, requestContent.c_str());
+  curl_easy_setopt(mp_curl, CURLOPT_WRITEFUNCTION, &write_callback_to_iss);
+  curl_easy_setopt(mp_curl, CURLOPT_WRITEDATA, &stringstream);
+  res = curl_easy_perform(mp_curl);
+  if (res == CURLE_OK) {
+    long response_code;
+    curl_easy_getinfo(mp_curl, CURLINFO_RESPONSE_CODE, &response_code);
+    pthread_mutex_unlock(&m_lock);
+    if (response_code != 200) {
+      throw std::runtime_error("Invalid return code from aria");
+    }
+    auto responseContent = stringstream.str();
+    MethodResponse response(responseContent);
+    if (response.isFault()) {
+      throw AriaError(response.getFault().getFaultString());
+    }
+    return responseContent;
+  }
+  pthread_mutex_unlock(&m_lock);
+  throw std::runtime_error("Cannot perform request");
+}
+
+std::string Aria2::addUri(const std::vector<std::string>& uris)
+{
+  MethodCall methodCall("aria2.addUri", m_secret);
+  auto uriParams = methodCall.newParamValue().getArray();
+  for (auto& uri : uris) {
+    uriParams.addValue().set(uri);
+  }
+  auto ret = doRequest(methodCall);
+  MethodResponse response(ret);
+  return response.getParamValue(0).getAsS();
+}
+
+std::string Aria2::tellStatus(const std::string& gid, const std::vector<std::string>& statusKey)
+{
+  MethodCall methodCall("aria2.tellStatus", m_secret);
+  methodCall.newParamValue().set(gid);
+  if (!statusKey.empty()) {
+    auto statusArray = methodCall.newParamValue().getArray();
+    for (auto& key : statusKey) {
+      statusArray.addValue().set(key);
+    }
+  }
+  return doRequest(methodCall);
+}
+
+std::vector<std::string> Aria2::tellActive()
+{
+  MethodCall methodCall("aria2.tellActive", m_secret);
+  auto statusArray = methodCall.newParamValue().getArray();
+  statusArray.addValue().set(std::string("gid"));
+  statusArray.addValue().set(std::string("following"));
+  auto responseContent = doRequest(methodCall);
+  MethodResponse response(responseContent);
+  std::vector<std::string> activeGID;
+  int index = 0;
+  while(true) {
+    try {
+      auto structNode = response.getParamValue(0).getArray().getValue(index++).getStruct();
+      auto gidNode = structNode.getMember("gid");
+      activeGID.push_back(gidNode.getValue().getAsS());
+    } catch (InvalidRPCNode& e) { break; }
+  }
+  return activeGID;
+}
+
+void Aria2::saveSession()
+{
+  MethodCall methodCall("aria2.saveSession", m_secret);
+  doRequest(methodCall);
+  std::cout << "session saved" << std::endl;
+}
+
+void Aria2::shutdown()
+{
+  MethodCall methodCall("aria2.shutdown", m_secret);
+  doRequest(methodCall);
+}
+
+
+} // end namespace kiwix

src/aria2.h

@@ -0,0 +1,46 @@
+
+
+#ifndef KIWIXLIB_ARIA2_H_
+#define KIWIXLIB_ARIA2_H_
+
+#ifdef _WIN32
+// winsock2.h need to be included before windows.h (included by curl.h)
+# include <winsock2.h>
+#endif
+
+#include "subprocess.h"
+#include "xmlrpc.h"
+
+#include <memory>
+#include <curl/curl.h>
+#include <pthread.h>
+
+namespace kiwix {
+
+class Aria2
+{
+  private:
+    std::unique_ptr<Subprocess> mp_aria;
+    int m_port;
+    std::string m_secret;
+    std::string m_downloadDir;
+    CURL* mp_curl;
+    pthread_mutex_t m_lock;
+
+    std::string doRequest(const MethodCall& methodCall);
+
+  public:
+    Aria2();
+    virtual ~Aria2();
+    void close();
+
+    std::string addUri(const std::vector<std::string>& uri);
+    std::string tellStatus(const std::string& gid, const std::vector<std::string>& statusKey);
+    std::vector<std::string> tellActive();
+    void saveSession();
+    void shutdown();
+};
+
+}; //end namespace kiwix
+
+#endif // KIWIXLIB_ARIA2_H_

src/book.cpp

@@ -0,0 +1,207 @@
+/*
+ * Copyright 2011 Emmanuel Engelhart <kelson@kiwix.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#include "book.h"
+#include "reader.h"
+
+#include "common/base64.h"
+#include "common/regexTools.h"
+#include "common/networkTools.h"
+
+#include <pugixml.hpp>
+
+namespace kiwix
+{
+/* Constructor */
+Book::Book() : m_readOnly(false)
+{
+}
+/* Destructor */
+Book::~Book()
+{
+}
+
+bool Book::update(const kiwix::Book& other)
+{
+  if (m_readOnly)
+    return false;
+
+  m_readOnly = other.m_readOnly;
+
+  if (m_path.empty()) {
+    m_path = other.m_path;
+  }
+
+  if (m_url.empty()) {
+    m_url = other.m_url;
+  }
+
+  if (m_tags.empty()) {
+    m_tags = other.m_tags;
+  }
+
+  if (m_name.empty()) {
+    m_name = other.m_name;
+  }
+
+  if (m_indexPath.empty()) {
+    m_indexPath = other.m_indexPath;
+    m_indexType = other.m_indexType;
+  }
+
+  if (m_faviconMimeType.empty()) {
+    m_favicon = other.m_favicon;
+    m_faviconMimeType = other.m_faviconMimeType;
+  }
+  return true;
+}
+
+void Book::update(const kiwix::Reader& reader)
+{
+  m_path = reader.getZimFilePath();
+  m_id = reader.getId();
+  m_description = reader.getDescription();
+  m_language = reader.getLanguage();
+  m_date = reader.getDate();
+  m_creator = reader.getCreator();
+  m_publisher = reader.getPublisher();
+  m_title = reader.getTitle();
+  m_name = reader.getName();
+  m_tags = reader.getTags();
+  m_origId = reader.getOrigId();
+  m_articleCount = reader.getArticleCount();
+  m_mediaCount = reader.getMediaCount();
+  m_size = static_cast<uint64_t>(reader.getFileSize()) << 10;
+
+  reader.getFavicon(m_favicon, m_faviconMimeType);
+}
+
+#define ATTR(name) node.attribute(name).value()
+void Book::updateFromXml(const pugi::xml_node& node, const std::string& baseDir)
+{
+  m_id = ATTR("id");
+  std::string path = ATTR("path");
+  if (isRelativePath(path)) {
+    path = computeAbsolutePath(baseDir, path);
+  }
+  m_path = path;
+  path = ATTR("indexPath");
+  if (!path.empty()) {
+    if (isRelativePath(path)) {
+      path = computeAbsolutePath(baseDir, path);
+    }
+    m_indexPath = path;
+    m_indexType = XAPIAN;
+  }
+  m_title = ATTR("title");
+  m_name = ATTR("name");
+  m_tags = ATTR("tags");
+  m_description = ATTR("description");
+  m_language = ATTR("language");
+  m_date = ATTR("date");
+  m_creator = ATTR("creator");
+  m_publisher = ATTR("publisher");
+  m_url = ATTR("url");
+  m_origId = ATTR("origId");
+  m_articleCount = strtoull(ATTR("articleCount"), 0, 0);
+  m_mediaCount = strtoull(ATTR("mediaCount"), 0, 0);
+  m_size = strtoull(ATTR("size"), 0, 0) << 10;
+  m_favicon = base64_decode(ATTR("favicon"));
+  m_faviconMimeType = ATTR("faviconMimeType");
+  try {
+    m_downloadId = ATTR("downloadId");
+  } catch(...) {}
+}
+#undef ATTR
+
+
+#define VALUE(name) node.child(name).child_value()
+void Book::updateFromOpds(const pugi::xml_node& node, const std::string& urlHost)
+{
+  m_id = VALUE("id");
+  if (!m_id.compare(0, 9, "urn:uuid:")) {
+    m_id.erase(0, 9);
+  }
+  m_title = VALUE("title");
+  m_description = VALUE("description");
+  m_language = VALUE("language");
+  m_date = VALUE("updated");
+  m_creator = node.child("author").child("name").child_value();
+  for(auto linkNode = node.child("link"); linkNode;
+           linkNode = linkNode.next_sibling("link")) {
+    std::string rel = linkNode.attribute("rel").value();
+
+    if (rel == "http://opds-spec.org/acquisition/open-access") {
+      m_url = linkNode.attribute("href").value();
+      m_size = strtoull(linkNode.attribute("length").value(), 0, 0);
+    }
+    if (rel == "http://opds-spec.org/image/thumbnail") {
+      m_faviconUrl = urlHost + linkNode.attribute("href").value();
+      m_faviconMimeType = linkNode.attribute("type").value();
+    }
+ }
+
+}
+#undef VALUE
+
+std::string Book::getHumanReadableIdFromPath()
+{
+  std::string id = m_path;
+  if (!id.empty()) {
+    kiwix::removeAccents(id);
+
+#ifdef _WIN32
+    id = replaceRegex(id, "", "^.*\\\\");
+#else
+    id = replaceRegex(id, "", "^.*/");
+#endif
+
+    id = replaceRegex(id, "", "\\.zim[a-z]*$");
+    id = replaceRegex(id, "_", " ");
+    id = replaceRegex(id, "plus", "\\+");
+  }
+  return id;
+}
+
+void Book::setPath(const std::string& path)
+{
+ m_path = isRelativePath(path)
+   ? computeAbsolutePath(getCurrentDirectory(), path)
+   : path;
+}
+
+void Book::setIndexPath(const std::string& indexPath)
+{
+  m_indexPath = isRelativePath(indexPath)
+    ? computeAbsolutePath(getCurrentDirectory(), indexPath)
+    : indexPath;
+}
+
+const std::string& Book::getFavicon() const {
+  if (m_favicon.empty() && !m_faviconUrl.empty()) {
+    try {
+      m_favicon = download(m_faviconUrl);
+    } catch(...) {
+      std::cerr << "Cannot download favicon from " << m_faviconUrl;
+    }
+  }
+  return m_favicon;
+}
+
+}

src/common/base64.cpp

@@ -37,8 +37,10 @@ static inline bool is_base64(unsigned char c) {
   return (isalnum(c) || (c == '+') || (c == '/'));
 }
 
-std::string base64_encode(unsigned char const* bytes_to_encode, unsigned int in_len) {
+std::string base64_encode(const std::string& inString) {
   std::string ret;
+  auto in_len = inString.size();
+  const unsigned char* bytes_to_encode = reinterpret_cast<const unsigned char*>(inString.data());
   int i = 0;
   int j = 0;
   unsigned char char_array_3[3];

src/common/networkTools.cpp

@@ -19,44 +19,75 @@
 
 #include <common/networkTools.h>
 
-std::map<std::string, std::string> kiwix::getNetworkInterfaces() {
+#ifdef _WIN32
+#include <winsock2.h>
+#include <ws2tcpip.h>
+#else
+#include <net/if.h>
+#include <netdb.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+#include <sys/socket.h>
+#include <sys/types.h>
+#include <unistd.h>
+#endif
+
+#include <curl/curl.h>
+
+#include <sstream>
+#include <iostream>
+
+
+std::map<std::string, std::string> kiwix::getNetworkInterfaces()
+{
   std::map<std::string, std::string> interfaces;
 
 #ifdef _WIN32
   SOCKET sd = WSASocket(AF_INET, SOCK_DGRAM, 0, 0, 0, 0);
-  if (sd == SOCKET_ERROR) {
-    std::cerr << "Failed to get a socket. Error " << WSAGetLastError() <<
-      std::endl;
+  if (sd == (SOCKET)SOCKET_ERROR) {
+    std::cerr << "Failed to get a socket. Error " << WSAGetLastError()
+              << std::endl;
     return interfaces;
   }
 
   INTERFACE_INFO InterfaceList[20];
   unsigned long nBytesReturned;
-  if (WSAIoctl(sd, SIO_GET_INTERFACE_LIST, 0, 0, &InterfaceList,
-	       sizeof(InterfaceList), &nBytesReturned, 0, 0) == SOCKET_ERROR) {
-    std::cerr << "Failed calling WSAIoctl: error " << WSAGetLastError() <<
-      std::endl;
+  if (WSAIoctl(sd,
+               SIO_GET_INTERFACE_LIST,
+               0,
+               0,
+               &InterfaceList,
+               sizeof(InterfaceList),
+               &nBytesReturned,
+               0,
+               0)
+      == SOCKET_ERROR) {
+    std::cerr << "Failed calling WSAIoctl: error " << WSAGetLastError()
+              << std::endl;
     return interfaces;
   }
 
   int nNumInterfaces = nBytesReturned / sizeof(INTERFACE_INFO);
   for (int i = 0; i < nNumInterfaces; ++i) {
-    sockaddr_in *pAddress;
-    pAddress = (sockaddr_in *) & (InterfaceList[i].iiAddress);
+    sockaddr_in* pAddress;
+    pAddress = (sockaddr_in*)&(InterfaceList[i].iiAddress);
 
     /* Add to the map */
     std::string interfaceName = std::string(inet_ntoa(pAddress->sin_addr));
     std::string interfaceIp = std::string(inet_ntoa(pAddress->sin_addr));
-    interfaces.insert(std::pair<std::string, std::string>(interfaceName, interfaceIp));
+    interfaces.insert(
+        std::pair<std::string, std::string>(interfaceName, interfaceIp));
   }
 #else
   /* Get Network interfaces information */
   char buf[16384];
   struct ifconf ifconf;
   int fd = socket(PF_INET, SOCK_DGRAM, 0); /* Only IPV4 */
-  ifconf.ifc_len=sizeof buf;
-  ifconf.ifc_buf=buf;
-  if(ioctl(fd, SIOCGIFCONF, &ifconf)!=0) {
+  ifconf.ifc_len = sizeof buf;
+  ifconf.ifc_buf = buf;
+  if (ioctl(fd, SIOCGIFCONF, &ifconf) != 0) {
     perror("ioctl(SIOCGIFCONF)");
     exit(EXIT_FAILURE);
   }
@@ -64,74 +95,115 @@ std::map<std::string, std::string> kiwix::getNetworkInterfaces() {
   /* Go through each interface */
   int i;
   size_t len;
-  struct ifreq *ifreq;
+  struct ifreq* ifreq;
   ifreq = ifconf.ifc_req;
-  for (i = 0; i < ifconf.ifc_len; ) {
+  for (i = 0; i < ifconf.ifc_len;) {
     if (ifreq->ifr_addr.sa_family == AF_INET) {
       /* Get the network interface ip */
-      char host[128] = { 0 };
-      const int error = getnameinfo(&(ifreq->ifr_addr), sizeof ifreq->ifr_addr,
-                                    host, sizeof host,
-                                    0, 0, NI_NUMERICHOST);
+      char host[128] = {0};
+      const int error = getnameinfo(&(ifreq->ifr_addr),
+                                    sizeof ifreq->ifr_addr,
+                                    host,
+                                    sizeof host,
+                                    0,
+                                    0,
+                                    NI_NUMERICHOST);
       if (!error) {
         std::string interfaceName = std::string(ifreq->ifr_name);
         std::string interfaceIp = std::string(host);
         /* Add to the map */
-        interfaces.insert(std::pair<std::string, std::string>(interfaceName, interfaceIp));
+        interfaces.insert(
+            std::pair<std::string, std::string>(interfaceName, interfaceIp));
       } else {
         perror("getnameinfo()");
       }
     }
 
-    /* some systems have ifr_addr.sa_len and adjust the length that
-     * way, but not mine. weird */
-#ifndef linux
-    len=IFNAMSIZ + ifreq->ifr_addr.sa_len;
+/* some systems have ifr_addr.sa_len and adjust the length that
+ * way, but not mine. weird */
+#ifndef __linux__
+    len = IFNAMSIZ + ifreq->ifr_addr.sa_len;
 #else
-    len=sizeof *ifreq;
+    len = sizeof *ifreq;
 #endif
-    ifreq=(struct ifreq*)((char*)ifreq+len);
-    i+=len;
+    ifreq = (struct ifreq*)((char*)ifreq + len);
+    i += len;
   }
 #endif
   return interfaces;
 }
 
-std::string kiwix::getBestPublicIp() {
+std::string kiwix::getBestPublicIp()
+{
   std::map<std::string, std::string> interfaces = kiwix::getNetworkInterfaces();
 
 #ifndef _WIN32
-  const char* const prioritizedNames[] =
-      { "eth0", "eth1", "wlan0", "wlan1", "en0", "en1" };
+  const char* const prioritizedNames[]
+      = {"eth0", "eth1", "wlan0", "wlan1", "en0", "en1"};
   const int count = (sizeof prioritizedNames) / (sizeof prioritizedNames[0]);
   for (int i = 0; i < count; ++i) {
-    std::map<std::string, std::string>::const_iterator it =
-        interfaces.find(prioritizedNames[i]);
-    if (it != interfaces.end())
+    std::map<std::string, std::string>::const_iterator it
+        = interfaces.find(prioritizedNames[i]);
+    if (it != interfaces.end()) {
       return it->second;
+    }
   }
 #endif
 
   for (std::map<std::string, std::string>::iterator iter = interfaces.begin();
-      iter != interfaces.end(); ++iter) {
+       iter != interfaces.end();
+       ++iter) {
     std::string interfaceIp = iter->second;
-    if (interfaceIp.length() >= 7 && interfaceIp.substr(0, 7) == "192.168")
+    if (interfaceIp.length() >= 7 && interfaceIp.substr(0, 7) == "192.168") {
       return interfaceIp;
+    }
   }
 
   for (std::map<std::string, std::string>::iterator iter = interfaces.begin();
-      iter != interfaces.end(); ++iter) {
+       iter != interfaces.end();
+       ++iter) {
     std::string interfaceIp = iter->second;
-    if (interfaceIp.length() >= 7 && interfaceIp.substr(0, 7) == "172.16.")
+    if (interfaceIp.length() >= 7 && interfaceIp.substr(0, 7) == "172.16.") {
       return interfaceIp;
+    }
   }
 
   for (std::map<std::string, std::string>::iterator iter = interfaces.begin();
-      iter != interfaces.end(); ++iter) {
+       iter != interfaces.end();
+       ++iter) {
     std::string interfaceIp = iter->second;
-    if (interfaceIp.length() >= 3 && interfaceIp.substr(0, 3) == "10.")
+    if (interfaceIp.length() >= 3 && interfaceIp.substr(0, 3) == "10.") {
       return interfaceIp;
+    }
   }
 
   return "127.0.0.1";
 }
+
+size_t write_callback_to_iss(char* ptr, size_t size, size_t nmemb, void* userdata)
+{
+  auto str = static_cast<std::stringstream*>(userdata);
+  str->write(ptr, nmemb);
+  return nmemb;
+}
+
+std::string kiwix::download(const std::string& url) {
+  auto curl = curl_easy_init();
+  std::stringstream ss;
+  curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
+  curl_easy_setopt(curl, CURLOPT_HTTPGET, 1L);
+  curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &write_callback_to_iss);
+  curl_easy_setopt(curl, CURLOPT_WRITEDATA, &ss);
+  auto res = curl_easy_perform(curl);
+  if (res != CURLE_OK) {
+    curl_easy_cleanup(curl);
+    throw std::runtime_error("Cannot perform request");
+  }
+  long response_code;
+  curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &response_code);
+  curl_easy_cleanup(curl);
+  if (response_code != 200) {
+    throw std::runtime_error("Invalid return code from server");
+  }
+  return ss.str();
+}

src/common/otherTools.cpp

@@ -18,11 +18,309 @@
  */
 
 #include <common/otherTools.h>
+#include <map>
 
-void kiwix::sleep(unsigned int milliseconds) {
+static std::map<std::string, std::string> codeisomapping {
+//a
+  { "ad", "and" },
+  { "ae", "are" },
+  { "af", "afg" },
+  { "ag", "atg" },
+  { "ai", "aia" },
+  { "al", "alb" },
+  { "am", "arm" },
+  { "an", "ant" },
+  { "ao", "ago" },
+  { "aq", "ata" },
+  { "ar", "arg" },
+  { "as", "asm" },
+  { "at", "aut" },
+  { "au", "aus" },
+  { "aw", "abw" },
+  { "ax", "ala" },
+  { "az", "aze" },
+//b
+  { "ba", "bih" },
+  { "bb", "brb" },
+  { "bd", "bgd" },
+  { "be", "bel" },
+  { "bf", "bfa" },
+  { "bg", "bgr" },
+  { "bh", "bhr" },
+  { "bi", "bdi" },
+  { "bj", "ben" },
+  { "bl", "blm" },
+  { "bn", "brn" },
+  { "bm", "bmu" },
+  { "bo", "bol" },
+  { "br", "bra" },
+  { "bs", "bhs" },
+  { "bt", "btn" },
+  { "bv", "bvt" },
+  { "bw", "bwa" },
+  { "by", "blr" },
+  { "bz", "blz" },
+//c
+  { "ca", "can" },
+  { "cc", "cck" },
+  { "cd", "cod" },
+  { "cf", "caf" },
+  { "cg", "cog" },
+  { "ch", "che" },
+  { "ci", "civ" },
+  { "ck", "cok" },
+  { "cl", "chl" },
+  { "cm", "cmr" },
+  { "cn", "chn" },
+  { "co", "col" },
+  { "cr", "cri" },
+  { "cu", "cub" },
+  { "cv", "cpv" },
+  { "cx", "cxr" },
+  { "cy", "cyp" },
+  { "cz", "cze" },
+//d
+  { "de", "deu" },
+  { "dj", "dji" },
+  { "dk", "dnk" },
+  { "dm", "dma" },
+  { "do", "dom" },
+  { "dz", "dza" },
+//e
+  { "ec", "ecu" },
+  { "ee", "est" },
+  { "eg", "egy" },
+  { "eh", "esh" },
+  { "en", "eng" },
+  { "er", "eri" },
+  { "es", "esp" },
+  { "et", "eth" },
+//f
+  { "fi", "fin" },
+  { "fj", "fji" },
+  { "fk", "flk" },
+  { "fm", "fsm" },
+  { "fo", "fro" },
+  { "fr", "fra" },
+//g
+  { "ga", "gab" },
+  { "gb", "gbr" },
+  { "gd", "grd" },
+  { "ge", "geo" },
+  { "gf", "guf" },
+  { "gg", "ggy" },
+  { "gh", "gha" },
+  { "gi", "gib" },
+  { "gl", "grl" },
+  { "gm", "gmb" },
+  { "gn", "gin" },
+  { "gp", "glp" },
+  { "gq", "gnq" },
+  { "gr", "grc" },
+  { "gs", "sgs" },
+  { "gt", "gtm" },
+  { "gu", "gum" },
+  { "gw", "gnb" },
+  { "gy", "guy" },
+//h
+  { "hk", "hkg" },
+  { "hm", "hmd" },
+  { "hn", "hnd" },
+  { "hr", "hrv" },
+  { "ht", "hti" },
+  { "hu", "hun" },
+//i
+  { "id", "idn" },
+  { "ie", "irl" },
+  { "il", "isr" },
+  { "im", "imn" },
+  { "in", "ind" },
+  { "io", "iot" },
+  { "iq", "irq" },
+  { "ir", "irn" },
+  { "is", "isl" },
+  { "it", "ita" },
+//j
+  { "je", "jey" },
+  { "jm", "jam" },
+  { "jo", "jor" },
+  { "jp", "jpn" },
+//k
+  { "ke", "ken" },
+  { "kg", "kgz" },
+  { "kh", "khm" },
+  { "ki", "kir" },
+  { "km", "com" },
+  { "kn", "kna" },
+  { "kp", "prk" },
+  { "kr", "kor" },
+  { "kw", "kwt" },
+  { "ky", "cym" },
+  { "kz", "kaz" },
+//l
+  { "la", "lao" },
+  { "lb", "lbn" },
+  { "lc", "lca" },
+  { "li", "lie" },
+  { "lk", "lka" },
+  { "lr", "lbr" },
+  { "ls", "lso" },
+  { "lt", "ltu" },
+  { "lu", "lux" },
+  { "lv", "lva" },
+  { "ly", "lby" },
+//m
+  { "ma", "mar" },
+  { "mc", "mco" },
+  { "md", "mda" },
+  { "me", "mne" },
+  { "mf", "maf" },
+  { "mg", "mdg" },
+  { "mh", "mhl" },
+  { "mk", "mkd" },
+  { "ml", "mli" },
+  { "mm", "mmr" },
+  { "mn", "mng" },
+  { "mo", "mac" },
+  { "mp", "mnp" },
+  { "mq", "mtq" },
+  { "mr", "mrt" },
+  { "ms", "msr" },
+  { "mt", "mlt" },
+  { "mu", "mus" },
+  { "mv", "mdv" },
+  { "mw", "mwi" },
+  { "mx", "mex" },
+  { "my", "mys" },
+  { "mz", "moz" },
+//n
+  { "na", "nam" },
+  { "nc", "ncl" },
+  { "ne", "ner" },
+  { "nf", "nfk" },
+  { "ng", "nga" },
+  { "ni", "nic" },
+  { "nl", "nld" },
+  { "no", "nor" },
+  { "np", "npl" },
+  { "nr", "nru" },
+  { "nu", "niu" },
+  { "nz", "nzl" },
+//o
+  { "om", "omn" },
+//p
+  { "pa", "pan" },
+  { "pe", "per" },
+  { "pf", "pyf" },
+  { "pg", "png" },
+  { "ph", "phl" },
+  { "pk", "pak" },
+  { "pl", "pol" },
+  { "pm", "spm" },
+  { "pn", "pcn" },
+  { "pr", "pri" },
+  { "ps", "pse" },
+  { "pt", "prt" },
+  { "pw", "plw" },
+  { "py", "pry" },
+//q
+  { "qa", "qat" },
+//r
+  { "re", "reu" },
+  { "ro", "rou" },
+  { "rs", "srb" },
+  { "ru", "rus" },
+  { "rw", "rwa" },
+//s
+  { "sa", "sau" },
+  { "sb", "slb" },
+  { "sc", "syc" },
+  { "sd", "sdn" },
+  { "se", "swe" },
+  { "sg", "sgp" },
+  { "sh", "shn" },
+  { "si", "svn" },
+  { "sj", "sjm" },
+  { "sk", "svk" },
+  { "sl", "sle" },
+  { "sm", "smr" },
+  { "sn", "sen" },
+  { "so", "som" },
+  { "sr", "sur" },
+  { "ss", "ssd" },
+  { "st", "stp" },
+  { "sv", "slv" },
+  { "sy", "syr" },
+  { "sz", "swz" },
+//t
+  { "tc", "tca" },
+  { "td", "tcd" },
+  { "tf", "atf" },
+  { "tg", "tgo" },
+  { "th", "tha" },
+  { "tj", "tjk" },
+  { "tk", "tkl" },
+  { "tl", "tls" },
+  { "tm", "tkm" },
+  { "tn", "tun" },
+  { "to", "ton" },
+  { "tr", "tur" },
+  { "tt", "tto" },
+  { "tv", "tuv" },
+  { "tw", "twn" },
+  { "tz", "tza" },
+//u
+  { "ua", "ukr" },
+  { "ug", "uga" },
+  { "um", "umi" },
+  { "us", "usa" },
+  { "uy", "ury" },
+  { "uz", "uzb" },
+//v
+  { "va", "vat" },
+  { "vc", "vct" },
+  { "ve", "ven" },
+  { "vg", "vgb" },
+  { "vi", "vir" },
+  { "vn", "vnm" },
+  { "vu", "vut" },
+//w
+  { "wf", "wlf" },
+  { "ws", "wsm" },
+//y
+  { "ye", "yem" },
+  { "yt", "myt" },
+// z
+  { "za", "zaf" },
+  { "zm", "zmb" },
+  { "zw", "zwe" }
+};
+
+void kiwix::sleep(unsigned int milliseconds)
+{
 #ifdef _WIN32
-      Sleep(milliseconds);
+  Sleep(milliseconds);
 #else
-      usleep(1000 * milliseconds);
+  usleep(1000 * milliseconds);
 #endif
 }
+
+
+struct XmlStringWriter: pugi::xml_writer
+{
+  std::string result;
+  virtual void write(const void* data, size_t size){
+    result.append(static_cast<const char*>(data), size);
+  }
+};
+
+std::string kiwix::nodeToString(pugi::xml_node node)
+{
+  XmlStringWriter writer;
+  node.print(writer, "  ");
+  return writer.result;
+}
+
+std::string kiwix::converta2toa3(const std::string& a2code){
+  return codeisomapping.at(a2code);
+}

src/common/pathTools.cpp

@@ -20,13 +20,13 @@
 #include <common/pathTools.h>
 
 #ifdef __APPLE__
-#include <mach-o/dyld.h>
 #include <limits.h>
+#include <mach-o/dyld.h>
 #elif _WIN32
+#include <direct.h>
 #include <windows.h>
 #include "shlwapi.h"
-#include <direct.h>
-#define getcwd _getcwd // stupid MSFT "deprecation" warning
+#define getcwd _getcwd  // stupid MSFT "deprecation" warning
 #endif
 
 #ifdef _WIN32
@@ -35,9 +35,9 @@
 #endif
 
 #ifdef _WIN32
-#define SEPARATOR "\\"
+const std::string SEPARATOR("\\");
 #else
-#define SEPARATOR "/"
+const std::string SEPARATOR("/");
 #include <unistd.h>
 #endif
 
@@ -47,7 +47,8 @@
 #define PATH_MAX 1024
 #endif
 
-bool isRelativePath(const string &path) {
+bool isRelativePath(const string& path)
+{
 #ifdef _WIN32
   return path.empty() || path.substr(1, 2) == ":\\" ? false : true;
 #else
@@ -55,45 +56,45 @@ bool isRelativePath(const string &path) {
 #endif
 }
 
-string computeRelativePath(const string path, const string absolutePath) {
+string computeRelativePath(const string path, const string absolutePath)
+{
   std::vector<std::string> pathParts = kiwix::split(path, SEPARATOR);
-  std::vector<std::string> absolutePathParts = kiwix::split(absolutePath, SEPARATOR);
+  std::vector<std::string> absolutePathParts
+      = kiwix::split(absolutePath, SEPARATOR);
 
   unsigned int commonCount = 0;
-  while (commonCount < pathParts.size() && 
-	 commonCount < absolutePathParts.size() && 
-	 pathParts[commonCount] == absolutePathParts[commonCount]) {
-    if (!pathParts[commonCount].empty()) {
+  while (commonCount < pathParts.size()
+         && commonCount < absolutePathParts.size()
+         && pathParts[commonCount] == absolutePathParts[commonCount]) {
       commonCount++;
-    }
   }
-  
+
   string relativePath;
 #ifdef _WIN32
   /* On Windows you have a token more because the root is represented
      by a letter */
   if (commonCount == 0) {
-    relativePath = "../";
+    relativePath = ".." + SEPARATOR;
   }
 #endif
 
-  for (unsigned int i = commonCount ; i < pathParts.size() ; i++) {
-    relativePath += "../";
+  for (unsigned int i = commonCount; i < pathParts.size(); i++) {
+    relativePath += ".." + SEPARATOR;
   }
-  for (unsigned int i = commonCount ; i < absolutePathParts.size() ; i++) {
+  for (unsigned int i = commonCount; i < absolutePathParts.size(); i++) {
     relativePath += absolutePathParts[i];
-    relativePath += i + 1 < absolutePathParts.size() ? "/" : "";
+    relativePath += i + 1 < absolutePathParts.size() ? SEPARATOR : "";
   }
-
   return relativePath;
 }
 
 /* Warning: the relative path must be with slashes */
-string computeAbsolutePath(const string path, const string relativePath) {
+string computeAbsolutePath(const string path, const string relativePath)
+{
   string absolutePath;
 
   if (path.empty()) {
-    char *path=NULL;
+    char* path = NULL;
     size_t size = 0;
 
 #ifdef _WIN32
@@ -104,15 +105,17 @@ string computeAbsolutePath(const string path, const string relativePath) {
 
     absolutePath = string(path) + SEPARATOR;
   } else {
-    absolutePath = path.substr(path.length() - 1, 1) == SEPARATOR ? path : path + SEPARATOR;
+    absolutePath = path.substr(path.length() - 1, 1) == SEPARATOR
+                       ? path
+                       : path + SEPARATOR;
   }
 
 #if _WIN32
-  char *cRelativePath = _strdup(relativePath.c_str());
+  char* cRelativePath = _strdup(relativePath.c_str());
 #else
-  char *cRelativePath = strdup(relativePath.c_str());
+  char* cRelativePath = strdup(relativePath.c_str());
 #endif
-  char *token = strtok(cRelativePath, "/");
+  char* token = strtok(cRelativePath, "/");
 
   while (token != NULL) {
     if (string(token) == "..") {
@@ -121,8 +124,9 @@ string computeAbsolutePath(const string path, const string relativePath) {
     } else if (strcmp(token, ".") && strcmp(token, "")) {
       absolutePath += string(token);
       token = strtok(NULL, "/");
-      if (token != NULL)
-	absolutePath += SEPARATOR;
+      if (token != NULL) {
+        absolutePath += SEPARATOR;
+      }
     } else {
       token = strtok(NULL, "/");
     }
@@ -131,31 +135,38 @@ string computeAbsolutePath(const string path, const string relativePath) {
   return absolutePath;
 }
 
-string removeLastPathElement(const string path, const bool removePreSeparator, const bool removePostSeparator) {
+string removeLastPathElement(const string path,
+                             const bool removePreSeparator,
+                             const bool removePostSeparator)
+{
   string newPath = path;
   size_t offset = newPath.find_last_of(SEPARATOR);
-  if (removePreSeparator && 
+  if (removePreSeparator &&
 #ifndef _WIN32
-      offset != newPath.find_first_of(SEPARATOR) && 
+      offset != newPath.find_first_of(SEPARATOR) &&
 #endif
-      offset == newPath.length()-1) {
+      offset == newPath.length() - 1) {
     newPath = newPath.substr(0, offset);
     offset = newPath.find_last_of(SEPARATOR);
   }
-  newPath = removePostSeparator ? newPath.substr(0, offset) : newPath.substr(0, offset+1);
+  newPath = removePostSeparator ? newPath.substr(0, offset)
+                                : newPath.substr(0, offset + 1);
   return newPath;
 }
 
-string appendToDirectory(const string &directoryPath, const string &filename) {
+string appendToDirectory(const string& directoryPath, const string& filename)
+{
   string newPath = directoryPath + SEPARATOR + filename;
   return newPath;
 }
 
-string getLastPathElement(const string &path) {
+string getLastPathElement(const string& path)
+{
   return path.substr(path.find_last_of(SEPARATOR) + 1);
 }
 
-unsigned int getFileSize(const string &path) {
+unsigned int getFileSize(const string& path)
+{
 #ifdef _WIN32
   struct _stat filestatus;
   _stat(path.c_str(), &filestatus);
@@ -167,12 +178,29 @@ unsigned int getFileSize(const string &path) {
   return filestatus.st_size / 1024;
 }
 
-string getFileSizeAsString(const string &path) {
-  ostringstream convert; convert << getFileSize(path);
+string getFileSizeAsString(const string& path)
+{
+  ostringstream convert;
+  convert << getFileSize(path);
   return convert.str();
 }
 
-bool fileExists(const string &path) {
+string getFileContent(const string& path)
+{
+  std::ifstream f(path, std::ios::in|std::ios::ate);
+  std::string content;
+  if (f.is_open()) {
+    auto size = f.tellg();
+    content.reserve(size);
+    f.seekg(0, std::ios::beg);
+    content.assign((std::istreambuf_iterator<char>(f)),
+                    std::istreambuf_iterator<char>());
+  }
+  return content;
+}
+
+bool fileExists(const string& path)
+{
 #ifdef _WIN32
   return PathFileExists(path.c_str());
 #else
@@ -187,7 +215,8 @@ bool fileExists(const string &path) {
 #endif
 }
 
-bool makeDirectory(const string &path) {
+bool makeDirectory(const string& path)
+{
 #ifdef _WIN32
   int status = _mkdir(path.c_str());
 #else
@@ -196,19 +225,44 @@ bool makeDirectory(const string &path) {
   return status == 0;
 }
 
+string makeTmpDirectory()
+{
+#ifdef _WIN32
+  char cbase[MAX_PATH+1];
+  int base_len = GetTempPath(MAX_PATH+1, cbase);
+  UUID uuid;
+  UuidCreate(&uuid);
+  char* dir_name;
+  UuidToString(&uuid, reinterpret_cast<unsigned char**>(&dir_name));
+  string dir(cbase, base_len);
+  dir += dir_name;
+  _mkdir(dir.c_str());
+  RpcStringFree(reinterpret_cast<unsigned char**>(&dir_name));
+#else
+  string base = "/tmp";
+  auto _template = base + "/kiwix-lib_XXXXXX";
+  char* _template_array = new char[_template.size()+1];
+  memcpy(_template_array, _template.c_str(), _template.size());
+  string dir = mkdtemp(_template_array);
+  delete[] _template_array;
+#endif
+  return dir;
+}
+
 /* Try to create a link and if does not work then make a copy */
-bool copyFile(const string &sourcePath, const string &destPath) {
+bool copyFile(const string& sourcePath, const string& destPath)
+{
   try {
 #ifndef _WIN32
     if (link(sourcePath.c_str(), destPath.c_str()) != 0) {
 #endif
-	std::ifstream infile(sourcePath.c_str(), std::ios_base::binary);
-	std::ofstream outfile(destPath.c_str(), std::ios_base::binary);
-	outfile << infile.rdbuf();
+      std::ifstream infile(sourcePath.c_str(), std::ios_base::binary);
+      std::ofstream outfile(destPath.c_str(), std::ios_base::binary);
+      outfile << infile.rdbuf();
 #ifndef _WIN32
     }
 #endif
-  } catch (exception &e) {
+  } catch (exception& e) {
     cerr << e.what() << endl;
     return false;
   }
@@ -216,18 +270,19 @@ bool copyFile(const string &sourcePath, const string &destPath) {
   return true;
 }
 
-string getExecutablePath() {
+string getExecutablePath()
+{
   char binRootPath[PATH_MAX];
-  
+
 #ifdef _WIN32
-  GetModuleFileName( NULL, binRootPath, PATH_MAX);
+  GetModuleFileName(NULL, binRootPath, PATH_MAX);
   return std::string(binRootPath);
 #elif __APPLE__
   uint32_t max = (uint32_t)PATH_MAX;
   _NSGetExecutablePath(binRootPath, &max);
   return std::string(binRootPath);
 #else
-  ssize_t size =  readlink("/proc/self/exe", binRootPath, PATH_MAX);
+  ssize_t size = readlink("/proc/self/exe", binRootPath, PATH_MAX);
   if (size != -1) {
     return std::string(binRootPath, size);
   }
@@ -236,7 +291,8 @@ string getExecutablePath() {
   return "";
 }
 
-bool writeTextFile(const string &path, const string &content) {
+bool writeTextFile(const string& path, const string& content)
+{
   std::ofstream file;
   file.open(path.c_str());
   file << content;
@@ -244,9 +300,36 @@ bool writeTextFile(const string &path, const string &content) {
   return true;
 }
 
-string getCurrentDirectory() {
-  char* a_cwd = getcwd(NULL,0);
+string getCurrentDirectory()
+{
+  char* a_cwd = getcwd(NULL, 0);
   string s_cwd(a_cwd);
   free(a_cwd);
   return s_cwd;
 }
+
+string getDataDirectory()
+{
+#ifdef _WIN32
+  char* cDataDir = ::getenv("APPDATA");
+#else
+  char* cDataDir = ::getenv("KIWIX_DATA_DIR");
+#endif
+  std::string dataDir = cDataDir==nullptr ? "" : cDataDir;
+  if (!dataDir.empty())
+    return dataDir;
+#ifdef _WIN32
+  cDataDir = ::getenv("USERPROFILE");
+  dataDir = cDataDir==nullptr ? getCurrentDirectory() : cDataDir;
+#else
+  cDataDir = ::getenv("XDG_DATA_HOME");
+  dataDir = cDataDir==nullptr ? "" : cDataDir;
+  if (dataDir.empty()) {
+    cDataDir = ::getenv("HOME");
+    dataDir = cDataDir==nullptr ? getCurrentDirectory() : cDataDir;
+    dataDir = appendToDirectory(dataDir, ".local");
+    dataDir = appendToDirectory(dataDir, "share");
+  }
+#endif
+  return appendToDirectory(dataDir, "kiwix");
+}

src/common/regexTools.cpp

@@ -19,12 +19,13 @@
 
 #include <common/regexTools.h>
 
-std::map<std::string, RegexMatcher*> regexCache;
+std::map<std::string, icu::RegexMatcher*> regexCache;
+
+icu::RegexMatcher* buildRegex(const std::string& regex)
+{
+  icu::RegexMatcher* matcher;
+  auto itr = regexCache.find(regex);
 
-RegexMatcher *buildRegex(const std::string &regex) {
-  RegexMatcher *matcher;
-  std::map<std::string, RegexMatcher*>::iterator itr = regexCache.find(regex);
-  
   /* Regex is in cache */
   if (itr != regexCache.end()) {
     matcher = itr->second;
@@ -33,8 +34,8 @@ RegexMatcher *buildRegex(const std::string &regex) {
   /* Regex needs to be parsed (and cached) */
   else {
     UErrorCode status = U_ZERO_ERROR;
-    UnicodeString uregex = UnicodeString(regex.c_str());
-    matcher = new RegexMatcher(uregex, UREGEX_CASE_INSENSITIVE, status);
+    icu::UnicodeString uregex(regex.c_str());
+    matcher = new icu::RegexMatcher(uregex, UREGEX_CASE_INSENSITIVE, status);
     regexCache[regex] = matcher;
   }
 
@@ -42,40 +43,47 @@ RegexMatcher *buildRegex(const std::string &regex) {
 }
 
 /* todo */
-void freeRegexCache() {
+void freeRegexCache()
+{
 }
-
-bool matchRegex(const std::string &content, const std::string &regex) {
+bool matchRegex(const std::string& content, const std::string& regex)
+{
   ucnv_setDefaultName("UTF-8");
-  UnicodeString ucontent = UnicodeString(content.c_str());
-  RegexMatcher *matcher = buildRegex(regex);
+  icu::UnicodeString ucontent(content.c_str());
+  auto matcher = buildRegex(regex);
   matcher->reset(ucontent);
   return matcher->find();
 }
 
-std::string replaceRegex(const std::string &content, const std::string &replacement, const std::string &regex) {
+std::string replaceRegex(const std::string& content,
+                         const std::string& replacement,
+                         const std::string& regex)
+{
   ucnv_setDefaultName("UTF-8");
-  UnicodeString ucontent = UnicodeString(content.c_str());
-  UnicodeString ureplacement = UnicodeString(replacement.c_str());
-  RegexMatcher *matcher = buildRegex(regex);
+  icu::UnicodeString ucontent(content.c_str());
+  icu::UnicodeString ureplacement(replacement.c_str());
+  auto matcher = buildRegex(regex);
   matcher->reset(ucontent);
   UErrorCode status = U_ZERO_ERROR;
-  UnicodeString uresult = matcher->replaceAll(ureplacement, status);
+  auto uresult = matcher->replaceAll(ureplacement, status);
   std::string tmp;
   uresult.toUTF8String(tmp);
   return tmp;
 }
 
-std::string appendToFirstOccurence(const std::string &content, const  std::string regex, const std::string &replacement) {
+std::string appendToFirstOccurence(const std::string& content,
+                                   const std::string regex,
+                                   const std::string& replacement)
+{
   ucnv_setDefaultName("UTF-8");
-  UnicodeString ucontent = UnicodeString(content.c_str());
-  UnicodeString ureplacement = UnicodeString(replacement.c_str());
-  RegexMatcher *matcher = buildRegex(regex);
+  icu::UnicodeString ucontent(content.c_str());
+  icu::UnicodeString ureplacement(replacement.c_str());
+  auto matcher = buildRegex(regex);
   matcher->reset(ucontent);
 
   if (matcher->find()) {
     UErrorCode status = U_ZERO_ERROR;
-    ucontent.insert(matcher->end(status), ureplacement);   
+    ucontent.insert(matcher->end(status), ureplacement);
     std::string tmp;
     ucontent.toUTF8String(tmp);
     return tmp;
@@ -83,4 +91,3 @@ std::string appendToFirstOccurence(const std::string &content, const  std::strin
 
   return content;
 }
-

src/common/stringTools.cpp

@@ -19,26 +19,37 @@
 
 #include <common/stringTools.h>
 
+#include <unicode/normlzr.h>
+#include <unicode/rep.h>
+#include <unicode/translit.h>
+#include <unicode/ucnv.h>
+#include <unicode/uniset.h>
+#include <unicode/ustring.h>
+
 /* tell ICU where to find its dat file (tables) */
-void kiwix::loadICUExternalTables() {
+void kiwix::loadICUExternalTables()
+{
 #ifdef __APPLE__
-    std::string executablePath = getExecutablePath();
-    std::string executableDirectory = removeLastPathElement(executablePath);
-    std::string datPath = computeAbsolutePath(executableDirectory, "icudt49l.dat");
-    try {
-        u_setDataDirectory(datPath.c_str());
-    } catch (exception &e) {
-        std::cerr << e.what() << std::endl;
-    }
+  std::string executablePath = getExecutablePath();
+  std::string executableDirectory = removeLastPathElement(executablePath);
+  std::string datPath
+      = computeAbsolutePath(executableDirectory, "icudt58l.dat");
+  try {
+    u_setDataDirectory(datPath.c_str());
+  } catch (exception& e) {
+    std::cerr << e.what() << std::endl;
+  }
 #endif
 }
 
-std::string kiwix::removeAccents(const std::string &text) {
+std::string kiwix::removeAccents(const std::string& text)
+{
   loadICUExternalTables();
   ucnv_setDefaultName("UTF-8");
   UErrorCode status = U_ZERO_ERROR;
-  Transliterator *removeAccentsTrans = Transliterator::createInstance("Lower; NFD; [:M:] remove; NFC", UTRANS_FORWARD, status);
-  UnicodeString ustring = UnicodeString(text.c_str());
+  auto removeAccentsTrans = icu::Transliterator::createInstance(
+      "Lower; NFD; [:M:] remove; NFC", UTRANS_FORWARD, status);
+  icu::UnicodeString ustring(text.c_str());
   removeAccentsTrans->transliterate(ustring);
   delete removeAccentsTrans;
   std::string unaccentedText;
@@ -49,7 +60,8 @@ std::string kiwix::removeAccents(const std::string &text) {
 #ifndef __ANDROID__
 
 /* Prepare integer for display */
-std::string kiwix::beautifyInteger(const unsigned int number) {
+std::string kiwix::beautifyInteger(uint64_t number)
+{
   std::stringstream numberStream;
   numberStream << number;
   std::string numberString = numberStream.str();
@@ -63,210 +75,304 @@ std::string kiwix::beautifyInteger(const unsigned int number) {
   return numberString;
 }
 
-std::string kiwix::beautifyFileSize(const unsigned int number) {
-  if (number > 1024*1024) {
-    return kiwix::beautifyInteger(number/(1024*1024)) + " GB";
-  } else {
-    return kiwix::beautifyInteger(number/1024 !=
-				  0 ? number/1024 : 1) + " MB";
-  }
+std::string kiwix::beautifyFileSize(uint64_t number)
+{
+  std::stringstream ss;
+  ss << std::fixed << std::setprecision(2);
+  if (number>>30)
+    ss << (number/(1024.0*1024*1024)) << " GB";
+  else if (number>>20)
+    ss << (number/(1024.0*1024)) << " MB";
+  else if (number>>10)
+    ss << (number/1024.0) << " KB";
+  else
+    ss << number << " B";
+  return ss.str();
 }
 
-void kiwix::printStringInHexadecimal(UnicodeString s) {
+void kiwix::printStringInHexadecimal(icu::UnicodeString s)
+{
   std::cout << std::showbase << std::hex;
-  for (int i=0; i<s.length(); i++) {
+  for (int i = 0; i < s.length(); i++) {
     char c = (char)((s.getTerminatedBuffer())[i]);
-    if (c & 0x80)
+    if (c & 0x80) {
       std::cout << (c & 0xffff) << " ";
-    else
+    } else {
       std::cout << c << " ";
+    }
   }
   std::cout << std::endl;
 }
 
-void kiwix::printStringInHexadecimal(const char *s) {
+void kiwix::printStringInHexadecimal(const char* s)
+{
   std::cout << std::showbase << std::hex;
   for (char const* pc = s; *pc; ++pc) {
-    if (*pc & 0x80)
+    if (*pc & 0x80) {
       std::cout << (*pc & 0xffff);
-    else
+    } else {
       std::cout << *pc;
+    }
     std::cout << ' ';
   }
   std::cout << std::endl;
 }
 
-void kiwix::stringReplacement(std::string& str, const std::string& oldStr, const std::string& newStr) {
+void kiwix::stringReplacement(std::string& str,
+                              const std::string& oldStr,
+                              const std::string& newStr)
+{
   size_t pos = 0;
-  while((pos = str.find(oldStr, pos)) != std::string::npos) {
+  while ((pos = str.find(oldStr, pos)) != std::string::npos) {
     str.replace(pos, oldStr.length(), newStr);
     pos += newStr.length();
   }
 }
 
 /* Encode string to avoid XSS attacks */
-std::string kiwix::encodeDiples(const std::string& str) {
+std::string kiwix::encodeDiples(const std::string& str)
+{
   std::string result = str;
   kiwix::stringReplacement(result, "<", "&lt;");
   kiwix::stringReplacement(result, ">", "&gt;");
   return result;
 }
 
-// Urlencode
-//based on javascript encodeURIComponent()
-
-std::string char2hex(char dec) {
-  char dig1 = (dec&0xF0)>>4;
-  char dig2 = (dec&0x0F);
-  if ( 0<= dig1 && dig1<= 9) dig1+=48;    //0,48inascii
-  if (10<= dig1 && dig1<=15) dig1+=97-10; //a,97inascii
-  if ( 0<= dig2 && dig2<= 9) dig2+=48;
-  if (10<= dig2 && dig2<=15) dig2+=97-10;
-
-  std::string r;
-  r.append( &dig1, 1);
-  r.append( &dig2, 1);
-  return r;
-}
-
-std::string kiwix::urlEncode(const std::string &c) {
-  std::string escaped="";
-  int max = c.length();
-  for(int i=0; i<max; i++)
-    {
-      if ( (48 <= c[i] && c[i] <= 57) ||//0-9
-	   (65 <= c[i] && c[i] <= 90) ||//abc...xyz
-	   (97 <= c[i] && c[i] <= 122) || //ABC...XYZ
-	   (c[i]=='~' || c[i]=='!' || c[i]=='*' || c[i]=='(' || c[i]==')' || c[i]=='\'')
-	   )
-        {
-	  escaped.append( &c[i], 1);
-        }
-      else
-        {
-	  escaped.append("%");
-	  escaped.append( char2hex(c[i]) );//converts char 255 to string "ff"
-        }
-    }
-  return escaped;
-}
-
 #endif
 
-static char charFromHex(std::string a) {
-  std::istringstream Blat(a);
-  int Z;
-  Blat >> std::hex >> Z;
-  return char (Z);
+/* urlEncode() based on javascript encodeURI() &
+   encodeURIComponent(). Mostly code from rstudio/httpuv (GPLv3) */
+
+bool isReservedUrlChar(char c)
+{
+  switch (c) {
+  case ';':
+  case ',':
+  case '/':
+  case '?':
+  case ':':
+  case '@':
+  case '&':
+  case '=':
+  case '+':
+  case '$':
+    return true;
+  default:
+    return false;
+  }
 }
 
-std::string kiwix::urlDecode(const std::string &originalUrl) {
-  std::string url = originalUrl;
-  std::string::size_type pos = 0;
-  while ((pos = url.find('%', pos)) != std::string::npos &&
-	 pos + 2 < url.length()) {
-    url.replace(pos, 3, 1, charFromHex(url.substr(pos + 1, 2)));
-    ++pos;
+bool needsEscape(char c, bool encodeReserved)
+{
+  if (c >= 'a' && c <= 'z')
+    return false;
+  if (c >= 'A' && c <= 'Z')
+    return false;
+  if (c >= '0' && c <= '9')
+    return false;
+  if (isReservedUrlChar(c))
+    return encodeReserved;
+  switch (c) {
+  case '-':
+  case '_':
+  case '.':
+  case '!':
+  case '~':
+  case '*':
+  case '\'':
+  case '(':
+  case ')':
+    return false;
   }
-  return url;
+  return true;
+}
+
+int hexToInt(char c) {
+  switch (c) {
+  case '0': return 0;
+  case '1': return 1;
+  case '2': return 2;
+  case '3': return 3;
+  case '4': return 4;
+  case '5': return 5;
+  case '6': return 6;
+  case '7': return 7;
+  case '8': return 8;
+  case '9': return 9;
+  case 'A': case 'a': return 10;
+  case 'B': case 'b': return 11;
+  case 'C': case 'c': return 12;
+  case 'D': case 'd': return 13;
+  case 'E': case 'e': return 14;
+  case 'F': case 'f': return 15;
+  default: return -1;
+  }
+}
+
+std::string kiwix::urlEncode(const std::string& value, bool encodeReserved)
+{
+  std::ostringstream os;
+  os << std::hex << std::uppercase;
+  for (std::string::const_iterator it = value.begin();
+       it != value.end();
+       it++) {
+
+    if (!needsEscape(*it, encodeReserved)) {
+      os << *it;
+    } else {
+      os << '%' << std::setw(2) << static_cast<unsigned int>(static_cast<unsigned char>(*it));
+    }
+  }
+  return os.str();
+}
+
+std::string kiwix::urlDecode(const std::string& value, bool component)
+{
+  std::ostringstream os;
+  for (std::string::const_iterator it = value.begin();
+       it != value.end();
+       it++) {
+
+    // If there aren't enough characters left for this to be a
+    // valid escape code, just use the character and move on
+    if (it > value.end() - 3) {
+      os << *it;
+      continue;
+    }
+
+    if (*it == '%') {
+      char hi = *(++it);
+      char lo = *(++it);
+      int iHi = hexToInt(hi);
+      int iLo = hexToInt(lo);
+      if (iHi < 0 || iLo < 0) {
+	// Invalid escape sequence
+	os << '%' << hi << lo;
+	continue;
+      }
+      char c = (char)(iHi << 4 | iLo);
+      if (!component && isReservedUrlChar(c)) {
+	os << '%' << hi << lo;
+      } else {
+	os << c;
+      }
+    } else {
+      os << *it;
+    }
+  }
+
+  return os.str();
 }
 
 /* Split string in a token array */
-std::vector<std::string> kiwix::split(const std::string & str,
-                                      const std::string & delims=" *-")
+std::vector<std::string> kiwix::split(const std::string& str,
+                                      const std::string& delims = " *-")
 {
   std::string::size_type lastPos = str.find_first_not_of(delims, 0);
   std::string::size_type pos = str.find_first_of(delims, lastPos);
   std::vector<std::string> tokens;
 
-  while (std::string::npos != pos || std::string::npos != lastPos)
-    {
-      tokens.push_back(str.substr(lastPos, pos - lastPos));
-      lastPos = str.find_first_not_of(delims, pos);
-      pos     = str.find_first_of(delims, lastPos);
-    }
+  while (std::string::npos != pos || std::string::npos != lastPos) {
+    tokens.push_back(str.substr(lastPos, pos - lastPos));
+    lastPos = str.find_first_not_of(delims, pos);
+    pos = str.find_first_of(delims, lastPos);
+  }
 
   return tokens;
 }
 
-std::vector<std::string> kiwix::split(const char* lhs, const char* rhs){
-  const std::string m1 (lhs), m2 (rhs);
+std::vector<std::string> kiwix::split(const char* lhs, const char* rhs)
+{
+  const std::string m1(lhs), m2(rhs);
   return split(m1, m2);
 }
 
-std::vector<std::string> kiwix::split(const char* lhs, const std::string& rhs){
+std::vector<std::string> kiwix::split(const char* lhs, const std::string& rhs)
+{
   return split(lhs, rhs.c_str());
 }
 
-std::vector<std::string> kiwix::split(const std::string& lhs, const char* rhs){
+std::vector<std::string> kiwix::split(const std::string& lhs, const char* rhs)
+{
   return split(lhs.c_str(), rhs);
 }
 
-std::string kiwix::ucFirst (const std::string &word) {
-  if (word.empty())
+std::string kiwix::ucFirst(const std::string& word)
+{
+  if (word.empty()) {
     return "";
+  }
 
   std::string result;
 
-  UnicodeString unicodeWord(word.c_str());
-  UnicodeString unicodeFirstLetter = UnicodeString(unicodeWord, 0, 1).toUpper();
+  icu::UnicodeString unicodeWord(word.c_str());
+  auto unicodeFirstLetter = icu::UnicodeString(unicodeWord, 0, 1).toUpper();
   unicodeWord.replace(0, 1, unicodeFirstLetter);
   unicodeWord.toUTF8String(result);
 
   return result;
 }
 
-std::string kiwix::ucAll (const std::string &word) {
-  if (word.empty())
+std::string kiwix::ucAll(const std::string& word)
+{
+  if (word.empty()) {
     return "";
+  }
 
   std::string result;
 
-  UnicodeString unicodeWord(word.c_str());
+  icu::UnicodeString unicodeWord(word.c_str());
   unicodeWord.toUpper().toUTF8String(result);
 
   return result;
 }
 
-std::string kiwix::lcFirst (const std::string &word) {
-  if (word.empty())
+std::string kiwix::lcFirst(const std::string& word)
+{
+  if (word.empty()) {
     return "";
+  }
 
   std::string result;
 
-  UnicodeString unicodeWord(word.c_str());
-  UnicodeString unicodeFirstLetter = UnicodeString(unicodeWord, 0, 1).toLower();
+  icu::UnicodeString unicodeWord(word.c_str());
+  auto unicodeFirstLetter = icu::UnicodeString(unicodeWord, 0, 1).toLower();
   unicodeWord.replace(0, 1, unicodeFirstLetter);
   unicodeWord.toUTF8String(result);
 
   return result;
 }
 
-std::string kiwix::lcAll (const std::string &word) {
-  if (word.empty())
+std::string kiwix::lcAll(const std::string& word)
+{
+  if (word.empty()) {
     return "";
+  }
 
   std::string result;
 
-  UnicodeString unicodeWord(word.c_str());
+  icu::UnicodeString unicodeWord(word.c_str());
   unicodeWord.toLower().toUTF8String(result);
 
   return result;
 }
 
-std::string kiwix::toTitle (const std::string &word) {
-  if (word.empty())
+std::string kiwix::toTitle(const std::string& word)
+{
+  if (word.empty()) {
     return "";
+  }
 
   std::string result;
 
-  UnicodeString unicodeWord(word.c_str());
+  icu::UnicodeString unicodeWord(word.c_str());
   unicodeWord = unicodeWord.toTitle(0);
   unicodeWord.toUTF8String(result);
 
   return result;
 }
 
-std::string kiwix::normalize (const std::string &word) {
+std::string kiwix::normalize(const std::string& word)
+{
   return kiwix::lcAll(word);
 }

src/downloader.cpp

@@ -0,0 +1,154 @@
+/*
+ * Copyright 2018 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#include "downloader.h"
+#include "common/pathTools.h"
+
+#include <algorithm>
+#include <thread>
+#include <chrono>
+
+#include <iostream>
+
+#include "aria2.h"
+#include "xmlrpc.h"
+#include "common/otherTools.h"
+#include <pugixml.hpp>
+
+namespace kiwix
+{
+
+void Download::updateStatus(bool follow)
+{
+  static std::vector<std::string> statusKey = {"status", "files", "totalLength",
+                                               "completedLength", "followedBy",
+                                               "downloadSpeed", "verifiedLength"};
+  std::string strStatus;
+  if(follow && !m_followedBy.empty()) {
+    strStatus = mp_aria->tellStatus(m_followedBy, statusKey);
+  } else {
+    strStatus = mp_aria->tellStatus(m_did, statusKey);
+  }
+//  std::cout << strStatus << std::endl;
+  MethodResponse response(strStatus);
+  if (response.isFault()) {
+    m_status = Download::K_UNKNOWN;
+    return;
+  }
+  auto structNode = response.getParams().getParam(0).getValue().getStruct();
+  auto _status = structNode.getMember("status").getValue().getAsS();
+  auto status = _status == "active" ? Download::K_ACTIVE
+              : _status == "waiting" ? Download::K_WAITING
+              : _status == "paused" ? Download::K_PAUSED
+              : _status == "error" ? Download::K_ERROR
+              : _status == "complete" ? Download::K_COMPLETE
+              : _status == "removed" ? Download::K_REMOVED
+              : Download::K_UNKNOWN;
+  if (status == K_COMPLETE) {
+    try {
+      auto followedByMember = structNode.getMember("followedBy");
+      m_followedBy = followedByMember.getValue().getArray().getValue(0).getAsS();
+      if (follow) {
+        status = K_ACTIVE;
+        updateStatus(true);
+        return;
+      }
+    } catch (InvalidRPCNode& e) { }
+  }
+  m_status = status;
+  m_totalLength = extractFromString<uint64_t>(structNode.getMember("totalLength").getValue().getAsS());
+  m_completedLength = extractFromString<uint64_t>(structNode.getMember("completedLength").getValue().getAsS());
+  m_downloadSpeed = extractFromString<uint64_t>(structNode.getMember("downloadSpeed").getValue().getAsS());
+  try {
+    auto verifiedLengthValue = structNode.getMember("verifiedLength").getValue();
+    m_verifiedLength = extractFromString<uint64_t>(verifiedLengthValue.getAsS());
+  } catch (InvalidRPCNode& e) { m_verifiedLength = 0; }
+  auto filesMember = structNode.getMember("files");
+  auto fileStruct = filesMember.getValue().getArray().getValue(0).getStruct();
+  m_path = fileStruct.getMember("path").getValue().getAsS();
+  auto urisArray = fileStruct.getMember("uris").getValue().getArray();
+  int index = 0;
+  m_uris.clear();
+  while(true) {
+    try {
+      auto uriNode = urisArray.getValue(index++).getStruct().getMember("uri");
+      m_uris.push_back(uriNode.getValue().getAsS());
+    } catch(InvalidRPCNode& e) { break; }
+  }
+}
+
+/* Constructor */
+Downloader::Downloader() :
+  mp_aria(new Aria2())
+{
+  for (auto gid : mp_aria->tellActive()) {
+    m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
+    m_knownDownloads[gid]->updateStatus();
+  }
+}
+
+
+/* Destructor */
+Downloader::~Downloader()
+{
+}
+
+void Downloader::close()
+{
+  mp_aria->close();
+}
+
+std::vector<std::string> Downloader::getDownloadIds() {
+  std::vector<std::string> ret;
+  for(auto& p:m_knownDownloads) {
+    ret.push_back(p.first);
+  }
+  return ret;
+}
+
+Download* Downloader::startDownload(const std::string& uri)
+{
+  for (auto& p: m_knownDownloads) {
+    auto& d = p.second;
+    auto& uris = d->getUris();
+    if (std::find(uris.begin(), uris.end(), uri) != uris.end())
+      return d.get();
+  }
+  std::vector<std::string> uris = {uri};
+  auto gid = mp_aria->addUri(uris);
+  m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
+  return m_knownDownloads[gid].get();
+}
+
+Download* Downloader::getDownload(const std::string& did)
+{
+  try {
+    return m_knownDownloads.at(did).get();
+  } catch(exception& e) {
+    for (auto gid : mp_aria->tellActive()) {
+      if (gid == did) {
+        m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
+        return m_knownDownloads[gid].get();
+      }
+    }
+    throw e;
+  }
+}
+
+}

src/entry.cpp

@@ -0,0 +1,138 @@
+/*
+ * Copyright 2011 Emmanuel Engelhart <kelson@kiwix.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#include "reader.h"
+#include <time.h>
+
+#include <zim/search.h>
+
+namespace kiwix
+{
+
+Entry::Entry(zim::Article article)
+  : article(article)
+{
+}
+
+#define RETURN_IF_INVALID(WHAT) if(!good()) { return (WHAT); }
+
+std::string Entry::getPath() const
+{
+  RETURN_IF_INVALID("");
+  return article.getLongUrl();
+}
+
+std::string Entry::getTitle() const
+{
+  RETURN_IF_INVALID("");
+  return article.getTitle();
+}
+
+std::string Entry::getContent() const
+{
+  RETURN_IF_INVALID("");
+  return article.getData();
+}
+
+zim::Blob Entry::getBlob(offset_type offset) const
+{
+  RETURN_IF_INVALID(zim::Blob());
+  return article.getData(offset);
+}
+
+zim::Blob Entry::getBlob(offset_type offset, size_type size) const
+{
+  RETURN_IF_INVALID(zim::Blob());
+  return article.getData(offset, size);
+}
+
+std::pair<std::string, offset_type> Entry::getDirectAccessInfo() const
+{
+  RETURN_IF_INVALID(std::make_pair("", 0));
+  return article.getDirectAccessInformation();
+}
+
+size_type Entry::getSize() const
+{
+  RETURN_IF_INVALID(0);
+  return article.getArticleSize();
+}
+
+std::string Entry::getMimetype() const
+{
+  RETURN_IF_INVALID("");
+  try {
+    return article.getMimeType();
+  } catch (exception& e) {
+    return "application/octet-stream";
+  }
+}
+
+bool Entry::isRedirect() const
+{
+  RETURN_IF_INVALID(false);
+  return article.isRedirect();
+}
+
+bool Entry::isLinkTarget() const
+{
+  RETURN_IF_INVALID(false);
+  return article.isLinktarget();
+}
+
+bool Entry::isDeleted() const
+{
+  RETURN_IF_INVALID(false);
+  return article.isDeleted();
+}
+
+Entry Entry::getRedirectEntry() const
+{
+  RETURN_IF_INVALID(Entry());
+  if ( !article.isRedirect() ) {
+    throw NoEntry();
+  }
+
+  auto targeted_article = article.getRedirectArticle();
+  if ( !targeted_article.good()) {
+    throw NoEntry();
+  }
+  return targeted_article;
+}
+
+Entry Entry::getFinalEntry() const
+{
+  RETURN_IF_INVALID(Entry());
+  if (final_article.good()) {
+    return final_article;
+  }
+
+  int loopCounter = 42;
+  final_article = article;
+  while (final_article.isRedirect() && loopCounter--) {
+    final_article = final_article.getRedirectArticle();
+    if ( !final_article.good()) {
+      throw NoEntry();
+    }
+  }
+
+  return final_article;
+}
+
+}

src/indexer.cpp

@@ -1,528 +0,0 @@
-/*
- * Copyright 2011-2014 Emmanuel Engelhart <kelson@kiwix.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU  General Public License as published by
- * the Free Software Foundation; either version 3 of the License, or
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
-
-#include "indexer.h"
-#include "xapian/myhtmlparse.h"
-#include "kiwixlib-resources.h"
-
-namespace kiwix {
-
-  /* Count word */
-  unsigned int Indexer::countWords(const string &text) {
-    unsigned int numWords = 1;
-    unsigned int length = text.size();
-
-    for(unsigned int i=0; i<length;) {
-      while(i<length && text[i] != ' ') {
-	i++;
-      }
-      numWords++;
-      i++;
-    }
-
-    return numWords;
-  }
-  
-  /* Constructor */
-  Indexer::Indexer() :
-    keywordsBoostFactor(3),
-    verboseFlag(false) {
-
-    /* Initialize mutex */
-    pthread_mutex_init(&threadIdsMutex, NULL);
-    pthread_mutex_init(&toParseQueueMutex, NULL);
-    pthread_mutex_init(&toIndexQueueMutex, NULL);
-    pthread_mutex_init(&articleExtractorRunningMutex, NULL);
-    pthread_mutex_init(&articleParserRunningMutex, NULL);
-    pthread_mutex_init(&articleIndexerRunningMutex, NULL);
-    pthread_mutex_init(&articleCountMutex, NULL);
-    pthread_mutex_init(&zimPathMutex, NULL);
-    pthread_mutex_init(&zimIdMutex, NULL);
-    pthread_mutex_init(&indexPathMutex, NULL);
-    pthread_mutex_init(&progressionMutex, NULL);
-    pthread_mutex_init(&verboseMutex, NULL);
-  }
-
-  /* Destructor */
-  Indexer::~Indexer() {
-  }
-
-  /* Read the stopwords */
-  void Indexer::readStopWords(const string languageCode) {
-    std::string stopWord;
-    std::istringstream file(getResource("stopwords/" + languageCode));
-
-    this->stopWords.clear();
-
-    while (getline(file, stopWord, '\n')) {
-      this->stopWords.push_back(stopWord);
-    }
-    
-    if (this->verboseFlag) {
-      std::cout << "Read stop words, lang code:" << languageCode << ", count:" << this->stopWords.size() << std::endl;
-    }
-  }
-  
-#pragma mark - Extractor
-  
-  /* Article extractor methods */
-  void *Indexer::extractArticles(void *ptr) {
-    pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL);
-    kiwix::Indexer *self = (kiwix::Indexer *)ptr;
-
-    /* Get the number of article to index and the ZIM id */
-    kiwix::Reader reader(self->getZimPath());
-    unsigned int articleCount = reader.getArticleCount();
-    self->setArticleCount(articleCount);
-    string zimId = reader.getId();
-    self->setZimId(zimId);
-
-    /* Progression */
-    unsigned int readArticleCount = 0;
-    unsigned int currentProgression = 0;
-    self->setProgression(currentProgression);
-    unsigned int newProgress;
-
-    /* StopWords */
-    self->readStopWords(reader.getLanguage());
-
-    /* Goes trough all articles */
-    zim::File *zimHandler = reader.getZimFileHandler();
-    unsigned int currentOffset = zimHandler->getNamespaceBeginOffset('A');
-    unsigned int lastOffset = zimHandler->getNamespaceEndOffset('A');
-    zim::Article currentArticle;
-
-    while (currentOffset < lastOffset) {
-      currentArticle = zimHandler->getArticle(currentOffset);
-      
-      if (!currentArticle.isRedirect()) {
-        /* Add articles to the queue */
-        indexerToken token;
-        token.title = currentArticle.getTitle();
-        token.url = currentArticle.getLongUrl();
-        token.content = string(currentArticle.getData().data(), currentArticle.getData().size());
-        self->pushToParseQueue(token);
-        readArticleCount += 1;
-        
-        /* Update progress */
-        if (self->progressCallback) {
-            self->progressCallback(readArticleCount, articleCount);
-        }
-        newProgress = (unsigned int)((float)readArticleCount / (float)articleCount * 100);
-        if (newProgress != currentProgression) {
-          self->setProgression(newProgress);
-        }
-      }
-        
-      currentOffset += 1;
-
-      /* Test if the thread should be cancelled */
-      pthread_testcancel();
-    } 
-
-    self->articleExtractorRunning(false);
-    pthread_exit(NULL);
-    return NULL;
-  }
-  
-  void Indexer::articleExtractorRunning(bool value) {
-    pthread_mutex_lock(&articleExtractorRunningMutex);
-    this->articleExtractorRunningFlag = value;
-    pthread_mutex_unlock(&articleExtractorRunningMutex); 
-  }
-  
-  bool Indexer::isArticleExtractorRunning() {
-    pthread_mutex_lock(&articleExtractorRunningMutex);
-    bool retVal = this->articleExtractorRunningFlag;
-    pthread_mutex_unlock(&articleExtractorRunningMutex); 
-    return retVal;
-  }
-  
-#pragma mark - Parser
-  
-  /* Article parser methods */
-  void *Indexer::parseArticles(void *ptr) {
-    pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL);
-    kiwix::Indexer *self = (kiwix::Indexer *)ptr;
-    size_t found;
-    indexerToken token;
-
-    while (self->popFromToParseQueue(token)) {
-      MyHtmlParser htmlParser;
-
-      /* The parser generate a lot of exceptions which should be avoided */
-      try {
-	htmlParser.parse_html(token.content, "UTF-8", true);
-      } catch (...) {
-      }
-
-      /* If content does not have the noindex meta tag */
-      /* Seems that the parser generates an exception in such case */
-      found = htmlParser.dump.find("NOINDEX");
-      
-      if (found == string::npos) {
-	/* Get the accented title */
-	token.accentedTitle = (htmlParser.title.empty() ? token.title : htmlParser.title);
-
-	/* count words */
-	stringstream countWordStringStream;
-	countWordStringStream << self->countWords(htmlParser.dump);
-	token.wordCount = countWordStringStream.str();
-	
-	/* snippet */
-	std::string snippet = std::string(htmlParser.dump, 0, 300);
-	std::string::size_type last = snippet.find_last_of('.');
-	if (last == snippet.npos)
-	  last = snippet.find_last_of(' ');
-	if (last != snippet.npos)
-	  snippet = snippet.substr(0, last);
-	token.snippet = snippet;
-
-	/* size */
-	stringstream sizeStringStream;
-	sizeStringStream << token.content.size() / 1024;
-	token.size = sizeStringStream.str();
-
-	/* Remove accent */
-	token.title = kiwix::removeAccents(token.accentedTitle);
-	token.keywords = kiwix::removeAccents(htmlParser.keywords);
-	token.content = kiwix::removeAccents(htmlParser.dump);
-	self->pushToIndexQueue(token);
-      }
-
-      /* Test if the thread should be cancelled */
-      pthread_testcancel(); 
-    }
-    
-    self->articleParserRunning(false);
-    pthread_exit(NULL);
-    return NULL;
-  }
-
-  void Indexer::articleParserRunning(bool value) {
-    pthread_mutex_lock(&articleParserRunningMutex);
-    this->articleParserRunningFlag = value;
-    pthread_mutex_unlock(&articleParserRunningMutex); 
-  }
-  
-  bool Indexer::isArticleParserRunning() {
-    pthread_mutex_lock(&articleParserRunningMutex);
-    bool retVal = this->articleParserRunningFlag;
-    pthread_mutex_unlock(&articleParserRunningMutex); 
-    return retVal;
-  }
-  
-#pragma mark - Indexer
-
-  /* Article indexer methods */
-  void *Indexer::indexArticles(void *ptr) {
-    pthread_setcanceltype(PTHREAD_CANCEL_DEFERRED, NULL);
-    kiwix::Indexer *self = (kiwix::Indexer *)ptr;
-    unsigned int indexedArticleCount = 0;
-    indexerToken token;
-
-    self->indexingPrelude(self->getIndexPath()); 
-
-    while (self->popFromToIndexQueue(token)) {
-      self->index(token.url, 
-		  token.accentedTitle,
-		  token.title, 
-		  token.keywords,
-		  token.content,
-		  token.snippet,
-		  token.size,
-		  token.wordCount
-		  );
-
-      indexedArticleCount += 1;
-
-      /* Make a hard-disk flush every 10.000 articles */
-      if (indexedArticleCount % 5000 == 0) {
-	self->flush();
-      }
-
-      /* Test if the thread should be cancelled */
-      pthread_testcancel();
-    }
-    self->indexingPostlude(self->getIndexPath());
-
-    /* Write content id file */
-    string path = appendToDirectory(self->getIndexPath(), "content.id");
-    writeTextFile(path, self->getZimId());
-
-    self->setProgression(100);
-    kiwix::sleep(100);
-
-    self->articleIndexerRunning(false);
-    pthread_exit(NULL);
-    return NULL;
-  }
-
-  void Indexer::articleIndexerRunning(bool value) {
-    pthread_mutex_lock(&articleIndexerRunningMutex);
-    this->articleIndexerRunningFlag = value;
-    pthread_mutex_unlock(&articleIndexerRunningMutex); 
-  }
-  
-  bool Indexer::isArticleIndexerRunning() {
-    pthread_mutex_lock(&articleIndexerRunningMutex);
-    bool retVal = this->articleIndexerRunningFlag;
-    pthread_mutex_unlock(&articleIndexerRunningMutex); 
-    return retVal;
-  }
-  
-#pragma mark - Parse Queue
-
-  /* ToParseQueue methods */
-  bool Indexer::isToParseQueueEmpty() {
-    pthread_mutex_lock(&toParseQueueMutex);
-    bool retVal = this->toParseQueue.empty();
-    pthread_mutex_unlock(&toParseQueueMutex);
-    return retVal;
-  }
-
-  void Indexer::pushToParseQueue(indexerToken &token) {
-    pthread_mutex_lock(&toParseQueueMutex); 
-    this->toParseQueue.push(token);
-    pthread_mutex_unlock(&toParseQueueMutex); 
-    kiwix::sleep(int(this->toParseQueue.size() / 200) / 10 * 1000);
-  }
-
-  bool Indexer::popFromToParseQueue(indexerToken &token) {
-    while (this->isToParseQueueEmpty() && this->isArticleExtractorRunning()) {
-      kiwix::sleep(500);
-      if (this->getVerboseFlag()) {
-	std::cout << "Waiting... ToParseQueue is empty for now..." << std::endl;
-      }
-
-      pthread_testcancel();
-    }
-
-    if (!this->isToParseQueueEmpty()) {
-      pthread_mutex_lock(&toParseQueueMutex); 
-      token = this->toParseQueue.front();
-      this->toParseQueue.pop();
-      pthread_mutex_unlock(&toParseQueueMutex); 
-    } else {
-      return false;
-    }
-
-    return true;
-  }
-  
-#pragma mark - Index Queue
-
-  /* ToIndexQueue methods */
-  bool Indexer::isToIndexQueueEmpty() {
-    pthread_mutex_lock(&toIndexQueueMutex);
-    bool retVal = this->toIndexQueue.empty();
-    pthread_mutex_unlock(&toIndexQueueMutex);
-    return retVal;
-  }
-
-  void Indexer::pushToIndexQueue(indexerToken &token) {
-    pthread_mutex_lock(&toIndexQueueMutex); 
-    this->toIndexQueue.push(token);
-    pthread_mutex_unlock(&toIndexQueueMutex);
-    kiwix::sleep(int(this->toIndexQueue.size() / 200) / 10 * 1000);
-  }
-
-  bool Indexer::popFromToIndexQueue(indexerToken &token) {
-    while (this->isToIndexQueueEmpty() && this->isArticleParserRunning()) {
-      kiwix::sleep(500);
-      if (this->getVerboseFlag()) {
-	std::cout << "Waiting... ToIndexQueue is empty for now..." << std::endl;
-      }
-
-      pthread_testcancel();
-    }
-
-    if (!this->isToIndexQueueEmpty()) {
-      pthread_mutex_lock(&toIndexQueueMutex); 
-      token = this->toIndexQueue.front();
-      this->toIndexQueue.pop();
-      pthread_mutex_unlock(&toIndexQueueMutex); 
-    } else {
-      return false;
-    }
-
-    return true;
-  }
-  
-#pragma mark - Properties Getter & Setter
-
-  /* ZIM & Index methods */
-  void Indexer::setZimPath(const string path) {
-    pthread_mutex_lock(&zimPathMutex);
-    this->zimPath = path;
-    pthread_mutex_unlock(&zimPathMutex); 
-  }
-
-  string Indexer::getZimPath() {
-    pthread_mutex_lock(&zimPathMutex); 
-    string retVal = this->zimPath;
-    pthread_mutex_unlock(&zimPathMutex);
-    return retVal;
-  }
-
-  void Indexer::setIndexPath(const string path) {
-    pthread_mutex_lock(&indexPathMutex); 
-    this->indexPath = path;
-    pthread_mutex_unlock(&indexPathMutex); 
-  }
-
-  string Indexer::getIndexPath() {
-    pthread_mutex_lock(&indexPathMutex); 
-    string retVal = this->indexPath;
-    pthread_mutex_unlock(&indexPathMutex);
-    return retVal;
-  }
-
-  void Indexer::setArticleCount(const unsigned int articleCount) {
-    pthread_mutex_lock(&articleCountMutex);
-    this->articleCount = articleCount;
-    pthread_mutex_unlock(&articleCountMutex); 
-  }
-
-  unsigned int Indexer::getArticleCount() {
-    pthread_mutex_lock(&articleCountMutex);
-    unsigned int retVal = this->articleCount;
-    pthread_mutex_unlock(&articleCountMutex);
-    return retVal;
-  }
-
-  void Indexer::setProgression(const unsigned int progression) {
-    pthread_mutex_lock(&progressionMutex); 
-    this->progression = progression;
-    pthread_mutex_unlock(&progressionMutex); 
-  }
-
-  unsigned int Indexer::getProgression() {
-    pthread_mutex_lock(&progressionMutex); 
-    unsigned int retVal = this->progression;
-    pthread_mutex_unlock(&progressionMutex); 
-    return retVal;
-  }
-
-  void Indexer::setZimId(const string id) {
-    pthread_mutex_lock(&zimIdMutex);
-    this->zimId = id;
-    pthread_mutex_unlock(&zimIdMutex); 
-  }
-
-  string Indexer::getZimId() {
-    pthread_mutex_lock(&zimIdMutex); 
-    string retVal = this->zimId;
-    pthread_mutex_unlock(&zimIdMutex);
-    return retVal;
-  }
-  
-#pragma mark - Status Management
-
-  /* Manage */
-  bool Indexer::start(const string zimPath, const string indexPath, ProgressCallback callback) {
-    if (this->getVerboseFlag()) {
-      std::cout << "Indexing of '" << zimPath << "' starting..." <<std::endl;
-    }
-    
-    if (callback) {
-        this->progressCallback = callback;
-    }
-
-    this->setArticleCount(0);
-    this->setProgression(0);
-    this->setZimPath(zimPath);
-    this->setIndexPath(indexPath);
-
-    pthread_mutex_lock(&threadIdsMutex); 
-    this->articleExtractorRunning(true);
-    pthread_create(&(this->articleExtractor), NULL, Indexer::extractArticles, (void*)this);
-    pthread_detach(this->articleExtractor);
-
-    while(this->isArticleExtractorRunning() && this->getArticleCount() == 0) {
-      kiwix::sleep(100);
-    }
-    
-    this->articleParserRunning(true);
-    pthread_create(&(this->articleParser), NULL, Indexer::parseArticles, (void*)this);
-    pthread_detach(this->articleParser);
-
-    this->articleIndexerRunning(true);
-    pthread_create(&(this->articleIndexer), NULL, Indexer::indexArticles, (void*)this);
-    pthread_detach(this->articleIndexer);
-    pthread_mutex_unlock(&threadIdsMutex);
-
-    return true;
-  }
-
-  bool Indexer::isRunning() {
-      if (this->getVerboseFlag()) {
-	std::cout << "isArticleExtractor running: " << (this->isArticleExtractorRunning() ? "yes" : "no") << std::endl;
-	std::cout << "isArticleParser running: " << (this->isArticleParserRunning() ? "yes" : "no") << std::endl;
-	std::cout << "isArticleIndexer running: " << (this->isArticleIndexerRunning() ? "yes" : "no") << std::endl;
-      }
-
-    return this->isArticleExtractorRunning() || this->isArticleIndexerRunning() || this->isArticleParserRunning();
-  }
-
-  bool Indexer::stop() {
-    if (this->isRunning()) {
-      bool isArticleExtractorRunning = this->isArticleExtractorRunning();
-      bool isArticleIndexerRunning = this->isArticleIndexerRunning();
-      bool isArticleParserRunning = this->isArticleParserRunning();
-      
-      pthread_mutex_lock(&threadIdsMutex); 
-      
-      if (isArticleIndexerRunning) {
-	pthread_cancel(this->articleIndexer);
-	this->articleIndexerRunning(false);
-      }
-      if (isArticleParserRunning) {
-	pthread_cancel(this->articleParser);
-	this->articleParserRunning(false);
-      }
-      if (isArticleExtractorRunning) {
-	pthread_cancel(this->articleExtractor);
-	this->articleExtractorRunning(false);
-      }
-
-      pthread_mutex_unlock(&threadIdsMutex); 
-    }
-
-    return true;
-  }
-  
-#pragma mark - verbose
-
-  /* Manage the verboseFlag */
-  void Indexer::setVerboseFlag(const bool value) {
-    pthread_mutex_lock(&verboseMutex);
-    this->verboseFlag = value;
-    pthread_mutex_unlock(&verboseMutex);
-  }
-
-  bool Indexer::getVerboseFlag() {
-    bool value;
-    pthread_mutex_lock(&verboseMutex);
-    value = this->verboseFlag;
-    pthread_mutex_unlock(&verboseMutex);
-    return value;
-  }
-
-}

src/library.cpp

@@ -18,126 +18,349 @@
  */
 
 #include "library.h"
+#include "book.h"
 
-namespace kiwix {
+#include "common/base64.h"
+#include "common/regexTools.h"
+#include "common/pathTools.h"
 
-  /* Constructor */
-  Book::Book():
-    readOnly(false) {
-  }
-  
-  /* Destructor */
-  Book::~Book() {
+#include <pugixml.hpp>
+#include <algorithm>
+
+namespace kiwix
+{
+/* Constructor */
+Library::Library() : version(KIWIX_LIBRARY_VERSION)
+{
+}
+/* Destructor */
+Library::~Library()
+{
+}
+
+
+bool Library::addBook(const Book& book)
+{
+  /* Try to find it */
+  try {
+    auto& oldbook = books.at(book.getId());
+    oldbook.update(book);
+    return false;
+  } catch (std::out_of_range&) {
+    books[book.getId()] = book;
+    return true;
   }
+}
 
-  /* Sort functions */
-  bool Book::sortByLastOpen(const kiwix::Book &a, const kiwix::Book &b) {
-    return atoi(a.last.c_str()) > atoi(b.last.c_str());
-  }
 
-  bool Book::sortByTitle(const kiwix::Book &a, const kiwix::Book &b) {
-    return strcmp(a.title.c_str(), b.title.c_str()) < 0;
-  }
+bool Library::removeBookById(const std::string& id)
+{
+  return books.erase(id) == 1;
+}
 
-  bool Book::sortByDate(const kiwix::Book &a, const kiwix::Book &b) {
-    return strcmp(a.date.c_str(), b.date.c_str()) > 0;
-  }
+Book& Library::getBookById(const std::string& id)
+{
+  return books.at(id);
+}
 
-  bool Book::sortBySize(const kiwix::Book &a, const kiwix::Book &b) {
-    return atoi(a.size.c_str()) < atoi(b.size.c_str());
-  }
-
-  bool Book::sortByPublisher(const kiwix::Book &a, const kiwix::Book &b) {
-    return strcmp(a.publisher.c_str(), b.publisher.c_str()) < 0;
-  }
-
-  bool Book::sortByCreator(const kiwix::Book &a, const kiwix::Book &b) {
-    return strcmp(a.creator.c_str(), b.creator.c_str()) < 0;
-  }
-
-  bool Book::sortByLanguage(const kiwix::Book &a, const kiwix::Book &b) {
-    return strcmp(a.language.c_str(), b.language.c_str()) < 0;
-  }
-
-  std::string Book::getHumanReadableIdFromPath() {
-    std::string id = pathAbsolute;
-    if (!id.empty()) {
-      kiwix::removeAccents(id);
-
-#ifdef _WIN32
-      id = replaceRegex(id, "", "^.*\\\\");
-#else
-      id = replaceRegex(id, "", "^.*/");
-#endif
-
-      id = replaceRegex(id, "", "\\.zim[a-z]*$");
-      id = replaceRegex(id, "_", " ");
-      id = replaceRegex(id, "plus", "\\+");
+unsigned int Library::getBookCount(const bool localBooks,
+                                   const bool remoteBooks)
+{
+  unsigned int result = 0;
+  for (auto& pair: books) {
+    auto& book = pair.second;
+    if ((!book.getPath().empty() && localBooks)
+        || (book.getPath().empty() && remoteBooks)) {
+      result++;
     }
-    return id;
   }
+  return result;
+}
 
-  /* Constructor */
-  Library::Library():
-    version(KIWIX_LIBRARY_VERSION) {
-  }
-  
-  /* Destructor */
-  Library::~Library() {
-  }
+bool Library::writeToFile(const std::string& path) {
+  pugi::xml_document doc;
+  auto baseDir = removeLastPathElement(path, true, false);
 
-  bool Library::addBook(const Book &book) {
+  /* Add the library node */
+  pugi::xml_node libraryNode = doc.append_child("library");
 
-    /* Try to find it */
-    std::vector<kiwix::Book>::iterator itr;
-    for ( itr = this->books.begin(); itr != this->books.end(); ++itr ) {
-      if (itr->id == book.id) {
-	if (!itr->readOnly) {
-	  itr->readOnly = book.readOnly;
-	  
-	  if (itr->path.empty())
-	    itr->path = book.path;
-	  
-	  if (itr->pathAbsolute.empty()) 
-	    itr->pathAbsolute = book.pathAbsolute;
-	  
-	  if (itr->url.empty())
-	    itr->url = book.url;
+  if (!version.empty())
+    libraryNode.append_attribute("version") = version.c_str();
 
-	  if (itr->tags.empty())
-	    itr->tags = book.tags;
+  /* Add each book */
+  for (auto& pair: books) {
+    auto& book = pair.second;
+    if (!book.readOnly()) {
+      pugi::xml_node bookNode = libraryNode.append_child("book");
+      bookNode.append_attribute("id") = book.getId().c_str();
 
-	  if (itr->name.empty())
-	    itr->name = book.name;
-	  
-	  if (itr->indexPath.empty()) {
-	    itr->indexPath = book.indexPath;
-	    itr->indexType = book.indexType;
-	  }
-	  
-	  if (itr->indexPathAbsolute.empty()) {
-	    itr->indexPathAbsolute = book.indexPathAbsolute;
-	    itr->indexType = book.indexType;
-	  }
-	  
-	  if (itr->faviconMimeType.empty()) {
-	    itr->favicon = book.favicon;
-	    itr->faviconMimeType = book.faviconMimeType;
-	  }
-	}
+      if (!book.getPath().empty()) {
+        bookNode.append_attribute("path") = computeRelativePath(
+            baseDir, book.getPath()).c_str();
+      }
 
-	return false;
+      if (!book.getIndexPath().empty()) {
+        bookNode.append_attribute("indexPath") = computeRelativePath(
+            baseDir, book.getIndexPath()).c_str();
+        bookNode.append_attribute("indexType") = "xapian";
+      }
+
+      if (book.getOrigId().empty()) {
+        if (!book.getTitle().empty())
+          bookNode.append_attribute("title") = book.getTitle().c_str();
+
+        if (!book.getName().empty())
+          bookNode.append_attribute("name") = book.getName().c_str();
+
+        if (!book.getTags().empty())
+          bookNode.append_attribute("tags") = book.getTags().c_str();
+
+        if (!book.getDescription().empty())
+          bookNode.append_attribute("description") = book.getDescription().c_str();
+
+        if (!book.getLanguage().empty())
+          bookNode.append_attribute("language") = book.getLanguage().c_str();
+
+        if (!book.getCreator().empty())
+          bookNode.append_attribute("creator") = book.getCreator().c_str();
+
+        if (!book.getPublisher().empty())
+          bookNode.append_attribute("publisher") = book.getPublisher().c_str();
+
+        if (!book.getFavicon().empty())
+          bookNode.append_attribute("favicon") = base64_encode(book.getFavicon()).c_str();
+
+        if (!book.getFaviconMimeType().empty())
+          bookNode.append_attribute("faviconMimeType")
+              = book.getFaviconMimeType().c_str();
+      } else {
+        bookNode.append_attribute("origId") = book.getOrigId().c_str();
+      }
+
+      if (!book.getDate().empty()) {
+        bookNode.append_attribute("date") = book.getDate().c_str();
+      }
+
+      if (!book.getUrl().empty()) {
+        bookNode.append_attribute("url") = book.getUrl().c_str();
+      }
+
+      if (book.getArticleCount())
+        bookNode.append_attribute("articleCount") = to_string(book.getArticleCount()).c_str();
+
+      if (book.getMediaCount())
+        bookNode.append_attribute("mediaCount") = to_string(book.getMediaCount()).c_str();
+
+      if (book.getSize()) {
+        bookNode.append_attribute("size") = to_string(book.getSize()>>10).c_str();
+      }
+
+      if (!book.getDownloadId().empty()) {
+        bookNode.append_attribute("downloadId") = book.getDownloadId().c_str();
       }
     }
-
-    /* otherwise */
-    this->books.push_back(book);
-    return true;
-  }
-
-  bool Library::removeBookByIndex(const unsigned int bookIndex) {
-    books.erase(books.begin()+bookIndex);
-    return true;
   }
 
+  /* saving file */
+  return doc.save_file(path.c_str());
+}
+
+std::vector<std::string> Library::getBooksLanguages()
+{
+  std::vector<std::string> booksLanguages;
+  std::map<std::string, bool> booksLanguagesMap;
+
+  for (auto& pair: books) {
+    auto& book = pair.second;
+    auto& language = book.getLanguage();
+    if (booksLanguagesMap.find(language) == booksLanguagesMap.end()) {
+      if (book.getOrigId().empty()) {
+        booksLanguagesMap[language] = true;
+        booksLanguages.push_back(language);
+      }
+    }
+  }
+
+  return booksLanguages;
+}
+
+std::vector<std::string> Library::getBooksCreators()
+{
+  std::vector<std::string> booksCreators;
+  std::map<std::string, bool> booksCreatorsMap;
+
+  for (auto& pair: books) {
+    auto& book = pair.second;
+    auto& creator = book.getCreator();
+    if (booksCreatorsMap.find(creator) == booksCreatorsMap.end()) {
+      if (book.getOrigId().empty()) {
+        booksCreatorsMap[creator] = true;
+        booksCreators.push_back(creator);
+      }
+    }
+  }
+
+  return booksCreators;
+}
+
+std::vector<std::string> Library::getBooksPublishers()
+{
+  std::vector<std::string> booksPublishers;
+  std::map<std::string, bool> booksPublishersMap;
+
+  for (auto& pair:books) {
+    auto& book = pair.second;
+    auto& publisher = book.getPublisher();
+    if (booksPublishersMap.find(publisher) == booksPublishersMap.end()) {
+      if (book.getOrigId().empty()) {
+        booksPublishersMap[publisher] = true;
+        booksPublishers.push_back(publisher);
+      }
+    }
+  }
+
+  return booksPublishers;
+}
+
+std::vector<std::string> Library::getBooksIds()
+{
+  std::vector<std::string> bookIds;
+
+  for (auto& pair: books) {
+    bookIds.push_back(pair.first);
+  }
+
+  return bookIds;
+}
+
+std::vector<std::string> Library::filter(const std::string& search)
+{
+  if (search.empty()) {
+    return getBooksIds();
+  }
+
+  std::vector<std::string> bookIds;
+  for(auto& pair:books) {
+     auto& book = pair.second;
+     if (matchRegex(book.getTitle(), "\\Q" + search + "\\E")
+         || matchRegex(book.getDescription(), "\\Q" + search + "\\E")) {
+       bookIds.push_back(pair.first);
+     }
+  }
+
+  return bookIds;
+}
+
+template<supportedListSortBy sort>
+struct Comparator {
+  Library* lib;
+  Comparator(Library* lib) : lib(lib) {}
+
+  bool operator() (const std::string& id1, const std::string& id2) {
+    return get_keys(id1) < get_keys(id2);
+  }
+
+  std::string get_keys(const std::string& id);
+  unsigned int get_keyi(const std::string& id);
+};
+
+template<>
+std::string Comparator<TITLE>::get_keys(const std::string& id)
+{
+  return lib->getBookById(id).getTitle();
+}
+
+template<>
+unsigned int Comparator<SIZE>::get_keyi(const std::string& id)
+{
+  return lib->getBookById(id).getSize();
+}
+
+template<>
+bool Comparator<SIZE>::operator() (const std::string& id1, const std::string& id2)
+{
+  return get_keyi(id1) < get_keyi(id2);
+}
+
+template<>
+std::string Comparator<DATE>::get_keys(const std::string& id)
+{
+  return lib->getBookById(id).getDate();
+}
+
+template<>
+std::string Comparator<CREATOR>::get_keys(const std::string& id)
+{
+  return lib->getBookById(id).getCreator();
+}
+
+template<>
+std::string Comparator<PUBLISHER>::get_keys(const std::string& id)
+{
+  return lib->getBookById(id).getPublisher();
+}
+
+
+std::vector<std::string> Library::listBooksIds(
+    int mode,
+    supportedListSortBy sortBy,
+    const std::string& search,
+    const std::string& language,
+    const std::string& creator,
+    const std::string& publisher,
+    size_t maxSize) {
+
+  std::vector<std::string> bookIds;
+  for(auto& pair:books) {
+    auto& book = pair.second;
+    auto local = !book.getPath().empty();
+    if (mode & LOCAL && !local)
+      continue;
+    if (mode & NOLOCAL && local)
+      continue;
+    auto valid = book.isPathValid();
+    if (mode & VALID && !valid)
+      continue;
+    if (mode & NOVALID && valid)
+      continue;
+    auto remote = !book.getUrl().empty();
+    if (mode & REMOTE && !remote)
+      continue;
+    if (mode & NOREMOTE && remote)
+      continue;
+    if (maxSize != 0 && book.getSize() > maxSize)
+      continue;
+    if (!language.empty() && book.getLanguage() != language)
+      continue;
+    if (!publisher.empty() && book.getPublisher() != publisher)
+      continue;
+    if (!creator.empty() && book.getCreator() != creator)
+      continue;
+    if (!search.empty() && !(matchRegex(book.getTitle(), "\\Q" + search + "\\E")
+                          || matchRegex(book.getDescription(), "\\Q" + search + "\\E")))
+      continue;
+    bookIds.push_back(pair.first);
+  }
+
+  switch(sortBy) {
+    case TITLE:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<TITLE>(this));
+      break;
+    case SIZE:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<SIZE>(this));
+      break;
+    case DATE:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<DATE>(this));
+      break;
+    case CREATOR:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<CREATOR>(this));
+      break;
+    case PUBLISHER:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<PUBLISHER>(this));
+      break;
+    default:
+      break;
+  }
+  return bookIds;
+}
 }

src/manager.cpp

@@ -19,544 +19,206 @@
 
 #include "manager.h"
 
-namespace kiwix {
+#include <pugixml.hpp>
 
-  /* Constructor */
-  Manager::Manager() :
-    writableLibraryPath("") {
+namespace kiwix
+{
+/* Constructor */
+Manager::Manager(LibraryManipulator* manipulator):
+  writableLibraryPath(""),
+  manipulator(manipulator),
+  mustDeleteManipulator(false)
+{
+}
+
+Manager::Manager(Library* library) :
+  writableLibraryPath(""),
+  manipulator(new DefaultLibraryManipulator(library)),
+  mustDeleteManipulator(true)
+{
+}
+
+/* Destructor */
+Manager::~Manager()
+{
+  if (mustDeleteManipulator) {
+    delete manipulator;
   }
+}
+bool Manager::parseXmlDom(const pugi::xml_document& doc,
+                          const bool readOnly,
+                          const std::string& libraryPath)
+{
+  pugi::xml_node libraryNode = doc.child("library");
 
-  /* Destructor */
-  Manager::~Manager() {
-  }
+  std::string libraryVersion = libraryNode.attribute("version").value();
 
-  bool Manager::parseXmlDom(const pugi::xml_document &doc, const bool readOnly, const string libraryPath) {
-    pugi::xml_node libraryNode = doc.child("library");
-
-    if (strlen(libraryNode.attribute("current").value()))
-      this->setCurrentBookId(libraryNode.attribute("current").value());
-
-    string libraryVersion = libraryNode.attribute("version").value();
-
-    for (pugi::xml_node bookNode = libraryNode.child("book"); bookNode; bookNode = bookNode.next_sibling("book")) {
-      bool ok = true;
-      kiwix::Book book;
-
-      book.readOnly = readOnly;
-      book.id = bookNode.attribute("id").value();
-      book.path = bookNode.attribute("path").value();
-      book.last = (std::string(bookNode.attribute("last").value()) != "undefined" ?
-		   bookNode.attribute("last").value() : "");
-      book.indexPath = bookNode.attribute("indexPath").value();
-      book.indexType = XAPIAN;
-      book.title = bookNode.attribute("title").value();
-      book.name = bookNode.attribute("name").value();
-      book.tags = bookNode.attribute("tags").value();
-      book.description = bookNode.attribute("description").value();
-      book.language = bookNode.attribute("language").value();
-      book.date = bookNode.attribute("date").value();
-      book.creator = bookNode.attribute("creator").value();
-      book.publisher = bookNode.attribute("publisher").value();
-      book.url = bookNode.attribute("url").value();
-      book.origId = bookNode.attribute("origId").value();
-      book.articleCount = bookNode.attribute("articleCount").value();
-      book.mediaCount = bookNode.attribute("mediaCount").value();
-      book.size = bookNode.attribute("size").value();
-      book.favicon = bookNode.attribute("favicon").value();
-      book.faviconMimeType = bookNode.attribute("faviconMimeType").value();
-
-      /* Check absolute and relative paths */
-      this->checkAndCleanBookPaths(book, libraryPath);
-
-      /* Update the book properties with the new importer */
-      if (libraryVersion.empty() || atoi(libraryVersion.c_str()) <= atoi(KIWIX_LIBRARY_VERSION)) {
-	if (!book.path.empty()) {
-	  ok = this->readBookFromPath(book.pathAbsolute);
-	}
-      }
-
-      if (ok) {
-	library.addBook(book);
-      }
-    }
-
-    return true;
-  }
-
-  bool Manager::readXml(const string xml, const bool readOnly, const string libraryPath) {
-    pugi::xml_document doc;
-    pugi::xml_parse_result result = doc.load_buffer_inplace((void*)xml.data(), xml.size());
-
-    if (result) {
-      this->parseXmlDom(doc, readOnly, libraryPath);
-    }
-
-    return true;
-  }
-
-  bool Manager::readFile(const string path, const bool readOnly) {
-    return this->readFile(path, path, readOnly);
-  }
-
-  bool Manager::readFile(const string nativePath, const string UTF8Path, const bool readOnly) {
-    bool retVal = true;
-    pugi::xml_document doc;
-    pugi::xml_parse_result result = doc.load_file(nativePath.c_str());
-
-    if (result) {
-      this->parseXmlDom(doc, readOnly, UTF8Path);
-    } else {
-      retVal = false;
-    }
-
-    /* This has to be set (although if the file does not exists) to be
-     * able to know where to save the library if new content are
-     * available */
-    if (!readOnly) {
-      this->writableLibraryPath = UTF8Path;
-    }
-
-    return retVal;
-  }
-
-  bool Manager::writeFile(const string path) {
-    pugi::xml_document doc;
-
-    /* Add the library node */
-    pugi::xml_node libraryNode = doc.append_child("library");
-
-    if (!getCurrentBookId().empty()) {
-      libraryNode.append_attribute("current") = getCurrentBookId().c_str();
-    }
-
-    if (!library.version.empty())
-      libraryNode.append_attribute("version") = library.version.c_str();
-
-    /* Add each book */
-    std::vector<kiwix::Book>::iterator itr;
-    for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-
-      if (!itr->readOnly) {
-	this->checkAndCleanBookPaths(*itr, path);
-
-	pugi::xml_node bookNode = libraryNode.append_child("book");
-	bookNode.append_attribute("id") = itr->id.c_str();
-
-	if (!itr->path.empty())
-	  bookNode.append_attribute("path") = itr->path.c_str();
-
-	if (!itr->last.empty() && itr->last != "undefined") {
-	  bookNode.append_attribute("last") = itr->last.c_str();
-	}
-
-	if (!itr->indexPath.empty())
-	  bookNode.append_attribute("indexPath") = itr->indexPath.c_str();
-
-	if (!itr->indexPath.empty() || !itr->indexPathAbsolute.empty()) {
-	  if (itr->indexType == XAPIAN)
-	    bookNode.append_attribute("indexType") = "xapian";
-	}
-
-	if (itr->origId.empty()) {
-	  if (!itr->title.empty())
-	    bookNode.append_attribute("title") = itr->title.c_str();
-
-	  if (!itr->name.empty())
-	    bookNode.append_attribute("name") = itr->name.c_str();
-
-	  if (!itr->tags.empty())
-	    bookNode.append_attribute("tags") = itr->tags.c_str();
-	  
-	  if (!itr->description.empty())
-	    bookNode.append_attribute("description") = itr->description.c_str();
-	  
-	  if (!itr->language.empty())
-	    bookNode.append_attribute("language") = itr->language.c_str();
-	  
-	  if (!itr->creator.empty())
-	    bookNode.append_attribute("creator") = itr->creator.c_str();
-	  
-	  if (!itr->publisher.empty())
-	    bookNode.append_attribute("publisher") = itr->publisher.c_str();
-	  
-	  if (!itr->favicon.empty())
-	    bookNode.append_attribute("favicon") = itr->favicon.c_str();
-	  
-	  if (!itr->faviconMimeType.empty())
-	    bookNode.append_attribute("faviconMimeType") = itr->faviconMimeType.c_str();
-	}
-
-	if (!itr->date.empty())
-	  bookNode.append_attribute("date") = itr->date.c_str();
-
-	if (!itr->url.empty())
-	  bookNode.append_attribute("url") = itr->url.c_str();
-	
-	if (!itr->origId.empty())
-	  bookNode.append_attribute("origId") = itr->origId.c_str();
-	
-	if (!itr->articleCount.empty())
-	  bookNode.append_attribute("articleCount") = itr->articleCount.c_str();
-
-	if (!itr->mediaCount.empty())
-	  bookNode.append_attribute("mediaCount") = itr->mediaCount.c_str();
-
-	if (!itr->size.empty())
-	  bookNode.append_attribute("size") = itr->size.c_str();
-      }
-    }
-
-    /* saving file */
-    doc.save_file(path.c_str());
-
-    return true;
-  }
-
-  bool Manager::setCurrentBookId(const string id) {
-    if (library.current.empty() || library.current.top() != id) {
-      if (id.empty() && !library.current.empty())
-	library.current.pop();
-      else
-	library.current.push(id);
-    }
-    return true;
-  }
-
-  string Manager::getCurrentBookId() {
-    return library.current.empty() ?
-      "" : library.current.top();
-  }
-
-  /* Add a book to the library. Return empty string if failed, book id otherwise */
-  string Manager::addBookFromPathAndGetId(const string pathToOpen, const string pathToSave,
-					  const string url, const bool checkMetaData) {
+  for (pugi::xml_node bookNode = libraryNode.child("book"); bookNode;
+       bookNode = bookNode.next_sibling("book")) {
     kiwix::Book book;
 
-    if (this->readBookFromPath(pathToOpen, &book)) {
+    book.setReadOnly(readOnly);
+    book.updateFromXml(bookNode,
+                       removeLastPathElement(libraryPath, true, false));
 
-      if (pathToSave != pathToOpen) {
-	book.path = pathToSave;
-	book.pathAbsolute = isRelativePath(pathToSave) ?
-	  computeAbsolutePath(removeLastPathElement(writableLibraryPath, true, false), pathToSave) : pathToSave;
-      }
-
-      if (!checkMetaData ||
-	  (checkMetaData && !book.title.empty() && !book.language.empty() && !book.date.empty())) {
-	book.url = url;
-	library.addBook(book);
-	return book.id;
+    /* Update the book properties with the new importer */
+    if (libraryVersion.empty()
+        || atoi(libraryVersion.c_str()) <= atoi(KIWIX_LIBRARY_VERSION)) {
+      if (!book.getPath().empty()) {
+        this->readBookFromPath(book.getPath(), &book);
       }
     }
-
-    return "";
+    manipulator->addBookToLibrary(book);
   }
 
-  /* Wrapper over Manager::addBookFromPath which return a bool instead of a string */
-  bool Manager::addBookFromPath(const string pathToOpen, const string pathToSave, const string url, const bool checkMetaData) {
-    return !(this->addBookFromPathAndGetId(pathToOpen, pathToSave, url, checkMetaData).empty());
+  return true;
+}
+
+bool Manager::readXml(const std::string& xml,
+                      const bool readOnly,
+                      const std::string& libraryPath)
+{
+  pugi::xml_document doc;
+  pugi::xml_parse_result result
+      = doc.load_buffer_inplace((void*)xml.data(), xml.size());
+
+  if (result) {
+    this->parseXmlDom(doc, readOnly, libraryPath);
   }
 
-  bool Manager::readBookFromPath(const string path, kiwix::Book *book) {
-    try {
-      kiwix::Reader *reader = new kiwix::Reader(path);
+  return true;
+}
 
-      if (book != NULL) {
-	book->path = path;
-	book->pathAbsolute = path;
-	book->id = reader->getId();
-	book->description = reader->getDescription();
-	book->language = reader->getLanguage();
-	book->date = reader->getDate();
-	book->creator = reader->getCreator();
-	book->publisher = reader->getPublisher();
-	book->title = reader->getTitle();
-	book->name = reader->getName();
-	book->tags = reader->getTags();
-	book->origId = reader->getOrigId();
-	std::ostringstream articleCountStream;
-	articleCountStream << reader->getArticleCount();
-	book->articleCount = articleCountStream.str();
 
-	std::ostringstream mediaCountStream;
-	mediaCountStream << reader->getMediaCount();
-	book->mediaCount = mediaCountStream.str();
 
-	ostringstream convert; convert << reader->getFileSize();
-	book->size = convert.str();
+bool Manager::parseOpdsDom(const pugi::xml_document& doc, const std::string& urlHost)
+{
+  pugi::xml_node libraryNode = doc.child("feed");
 
-	string favicon;
-	string faviconMimeType;
-	if (reader->getFavicon(favicon, faviconMimeType)) {
-	  book->favicon = base64_encode(reinterpret_cast<const unsigned char*>(favicon.c_str()), favicon.length());
-	  book->faviconMimeType = faviconMimeType;
-	}
-      }
+  try {
+    m_totalBooks = strtoull(libraryNode.child("totalResults").child_value(), 0, 0);
+    m_startIndex = strtoull(libraryNode.child("startIndex").child_value(), 0, 0);
+    m_itemsPerPage = strtoull(libraryNode.child("itemsPerPage").child_value(), 0, 0);
+    m_hasSearchResult = true;
+  } catch(...) {
+    m_hasSearchResult = false;
+  }
 
-      delete reader;
-    } catch (const std::exception& e) {
-      std::cerr << e.what() << std::endl;
-      return false;
-    }
+  for (pugi::xml_node entryNode = libraryNode.child("entry"); entryNode;
+       entryNode = entryNode.next_sibling("entry")) {
+    kiwix::Book book;
 
+    book.setReadOnly(false);
+    book.updateFromOpds(entryNode, urlHost);
+
+    /* Update the book properties with the new importer */
+    manipulator->addBookToLibrary(book);
+  }
+
+  return true;
+}
+
+
+
+bool Manager::readOpds(const std::string& content, const std::string& urlHost)
+{
+  pugi::xml_document doc;
+  pugi::xml_parse_result result
+      = doc.load_buffer_inplace((void*)content.data(), content.size());
+
+  if (result) {
+    this->parseOpdsDom(doc, urlHost);
     return true;
   }
 
-  bool Manager::removeBookByIndex(const unsigned int bookIndex) {
-    return this->library.removeBookByIndex(bookIndex);
+  return false;
+}
+
+bool Manager::readFile(const std::string& path, const bool readOnly)
+{
+  return this->readFile(path, path, readOnly);
+}
+
+bool Manager::readFile(const std::string& nativePath,
+                       const std::string& UTF8Path,
+                       const bool readOnly)
+{
+  bool retVal = true;
+  pugi::xml_document doc;
+  pugi::xml_parse_result result = doc.load_file(nativePath.c_str());
+
+  if (result) {
+    this->parseXmlDom(doc, readOnly, UTF8Path);
+  } else {
+    retVal = false;
   }
 
-  bool Manager::removeBookById(const string id) {
-    unsigned int bookIndex = 0;
-    std::vector<kiwix::Book>::iterator itr;
-    for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-      if ( itr->id == id) {
-	return this->library.removeBookByIndex(bookIndex);
-      }
-      bookIndex++;
+  /* This has to be set (although if the file does not exists) to be
+   * able to know where to save the library if new content are
+   * available */
+  if (!readOnly) {
+    this->writableLibraryPath = UTF8Path;
+  }
+
+  return retVal;
+}
+
+
+/* Add a book to the library. Return empty string if failed, book id otherwise
+ */
+std::string Manager::addBookFromPathAndGetId(const std::string& pathToOpen,
+                                             const std::string& pathToSave,
+                                             const std::string& url,
+                                             const bool checkMetaData)
+{
+  kiwix::Book book;
+
+  if (this->readBookFromPath(pathToOpen, &book)) {
+    if (pathToSave != pathToOpen) {
+      book.setPath(isRelativePath(pathToSave)
+                ? computeAbsolutePath(
+                      removeLastPathElement(writableLibraryPath, true, false),
+                      pathToSave)
+                : pathToSave);
     }
+
+    if (!checkMetaData
+        || (checkMetaData && !book.getTitle().empty() && !book.getLanguage().empty()
+            && !book.getDate().empty())) {
+      book.setUrl(url);
+      manipulator->addBookToLibrary(book);
+      return book.getId();
+    }
+  }
+
+  return "";
+}
+
+/* Wrapper over Manager::addBookFromPath which return a bool instead of a string
+ */
+bool Manager::addBookFromPath(const std::string& pathToOpen,
+                              const std::string& pathToSave,
+                              const std::string& url,
+                              const bool checkMetaData)
+{
+  return !(
+      this->addBookFromPathAndGetId(pathToOpen, pathToSave, url, checkMetaData)
+          .empty());
+}
+
+bool Manager::readBookFromPath(const std::string& path, kiwix::Book* book)
+{
+  try {
+    kiwix::Reader reader(path);
+    book->update(reader);
+    book->setPathValid(true);
+  } catch (const std::exception& e) {
+    std::cerr << "Invalid " << path << " : " << e.what() << std::endl;
+    book->setPathValid(false);
     return false;
   }
 
-  vector<string> Manager::getBooksLanguages() {
-    std::vector<string> booksLanguages;
-    std::vector<kiwix::Book>::iterator itr;
-    std::map<string, bool> booksLanguagesMap;
-
-    std::sort(library.books.begin(), library.books.end(), kiwix::Book::sortByLanguage);
-    for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-      if (booksLanguagesMap.find(itr->language) == booksLanguagesMap.end()) {
-        if (itr->origId.empty()) {
-	  booksLanguagesMap[itr->language] = true;
-	  booksLanguages.push_back(itr->language);
-        }
-      }
-    }
-
-    return booksLanguages;
-  }
-
-  vector<string> Manager::getBooksCreators() {
-    std::vector<string> booksCreators;
-    std::vector<kiwix::Book>::iterator itr;
-    std::map<string, bool> booksCreatorsMap;
-
-    std::sort(library.books.begin(), library.books.end(), kiwix::Book::sortByCreator);
-    for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-      if (booksCreatorsMap.find(itr->creator) == booksCreatorsMap.end()) {
-        if (itr->origId.empty()) {
-	  booksCreatorsMap[itr->creator] = true;
-	  booksCreators.push_back(itr->creator);
-        }
-      }
-    }
-
-    return booksCreators;
-  }
-
-
-  vector<string> Manager::getBooksIds() {
-    std::vector<string> booksIds;
-    std::vector<kiwix::Book>::iterator itr;
-
-    for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-      booksIds.push_back(itr->id);
-    }
-
-    return booksIds;
-  }
-
-  vector<string> Manager::getBooksPublishers() {
-    std::vector<string> booksPublishers;
-    std::vector<kiwix::Book>::iterator itr;
-    std::map<string, bool> booksPublishersMap;
-
-    std::sort(library.books.begin(), library.books.end(), kiwix::Book::sortByPublisher);
-    for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-      if (booksPublishersMap.find(itr->publisher) == booksPublishersMap.end()) {
-        if (itr->origId.empty()) {
-	  booksPublishersMap[itr->publisher] = true;
-	  booksPublishers.push_back(itr->publisher);
-        }
-      }
-    }
-
-    return booksPublishers;
-  }
-
-  kiwix::Library Manager::cloneLibrary() {
-    return this->library;
-  }
-
-  bool Manager::getCurrentBook(Book &book) {
-    string currentBookId = getCurrentBookId();
-    if (currentBookId.empty()) {
-      return false;
-    } else {
-      getBookById(currentBookId, book);
-      return true;
-    }
-  }
-
-  bool Manager::getBookById(const string id, Book &book) {
-    std::vector<kiwix::Book>::iterator itr;
-    for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-      if ( itr->id == id) {
-	book = *itr;
-	return true;
-      }
-    }
-    return false;
-  }
-
-  bool Manager::updateBookLastOpenDateById(const string id) {
-    std::vector<kiwix::Book>::iterator itr;
-    for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-      if ( itr->id == id) {
-	char unixdate[12];
-	sprintf (unixdate, "%d", (int)time(NULL));
-	itr->last = unixdate;
-	return true;
-      }
-    }
-
-    return false;
-  }
-
-  bool Manager::setBookIndex(const string id, const string path, const supportedIndexType type) {
-    std::vector<kiwix::Book>::iterator itr;
-    for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-      if ( itr->id == id) {
-	itr->indexPath = path;
-	itr->indexPathAbsolute = isRelativePath(path) ?
-	  computeAbsolutePath(removeLastPathElement(writableLibraryPath, true, false), path) : path;
-	itr->indexType = type;
-	return true;
-      }
-    }
-
-    return false;
-  }
-
-  bool Manager::setBookIndex(const string id, const string path) {
-    return this->setBookIndex(id, path, XAPIAN);
-  }
-
-  bool Manager::setBookPath(const string id, const string path) {
-    std::vector<kiwix::Book>::iterator itr;
-    for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-      if ( itr->id == id) {
-	itr->path = path;
-	itr->pathAbsolute = isRelativePath(path) ?
-	  computeAbsolutePath(removeLastPathElement(writableLibraryPath, true, false), path) : path;
-	return true;
-      }
-    }
-
-    return false;
-  }
-
-  void Manager::removeBookPaths() {
-    std::vector<kiwix::Book>::iterator itr;
-    for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-      itr->path = "";
-      itr->pathAbsolute = "";
-    }
-  }
-
-  unsigned int Manager::getBookCount(const bool localBooks, const bool remoteBooks) {
-    unsigned int result = 0;
-    std::vector<kiwix::Book>::iterator itr;
-    for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-      if ((!itr->path.empty() && localBooks) || (itr->path.empty() && remoteBooks))
-	result++;
-    }
-    return result;
-  }
-
-  bool Manager::listBooks(const supportedListMode mode, const supportedListSortBy sortBy,
-			  const unsigned int maxSize, const string language, const string creator,
-			  const string publisher, const string search) {
-    this->bookIdList.clear();
-    std::vector<kiwix::Book>::iterator itr;
-
-    /* Sort */
-    if (sortBy == TITLE) {
-      std::sort(library.books.begin(), library.books.end(), kiwix::Book::sortByTitle);
-    } else if (sortBy == SIZE) {
-      std::sort(library.books.begin(), library.books.end(), kiwix::Book::sortBySize);
-    } else if (sortBy == DATE) {
-      std::sort(library.books.begin(), library.books.end(), kiwix::Book::sortByDate);
-    } else if (sortBy == CREATOR) {
-      std::sort(library.books.begin(), library.books.end(), kiwix::Book::sortByCreator);
-    } else if (sortBy == PUBLISHER) {
-      std::sort(library.books.begin(), library.books.end(), kiwix::Book::sortByPublisher);
-    }
-
-    /* Special sort for LASTOPEN */
-    if (mode == LASTOPEN) {
-      std::sort(library.books.begin(), library.books.end(), kiwix::Book::sortByLastOpen);
-      for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-	if (!itr->last.empty())
-	  this->bookIdList.push_back(itr->id);
-      }
-    } else {
-      /* Generate the list of book id */
-      for ( itr = library.books.begin(); itr != library.books.end(); ++itr ) {
-	bool ok = true;
-
-	if (mode == LOCAL && itr->path.empty())
-	  ok = false;
-
-	if (ok == true && mode == REMOTE && (!itr->path.empty() || itr->url.empty()))
-	  ok = false;
-
-	if (ok == true && maxSize != 0 && (unsigned int)atoi(itr->size.c_str()) > maxSize * 1024 * 1024)
-	  ok = false;
-
-	if (ok == true && !language.empty() && !matchRegex(itr->language, language))
-	  ok = false;
-
-	if (ok == true && !creator.empty() && itr->creator != creator)
-	  ok = false;
-
-	if (ok == true && !publisher.empty() && itr->publisher != publisher)
-	  ok = false;
-
-	if ((ok == true && !search.empty()) && !(matchRegex(itr->title, "\\Q" + search + "\\E") || 
-						 matchRegex(itr->description,  "\\Q" + search + "\\E") ||
-						 matchRegex(itr->language,  "\\Q" + search + "\\E")
-						 ))
-	  ok = false;
-
-	if (ok == true) {
-	  this->bookIdList.push_back(itr->id);
-	}
-      }
-    }
-
-    return true;
-  }
-
-  void Manager::checkAndCleanBookPaths(Book &book, const string &libraryPath) {
-    if (!book.path.empty()) {
-      if (isRelativePath(book.path)) {
-	book.pathAbsolute = computeAbsolutePath(removeLastPathElement(libraryPath, true, false), book.path);
-      } else {
-	book.pathAbsolute = book.path;
-	book.path = computeRelativePath(removeLastPathElement(libraryPath, true, false), book.pathAbsolute);
-      }
-    }
-
-    if (!book.indexPath.empty()) {
-      if (isRelativePath(book.indexPath)) {
-	book.indexPathAbsolute =
-	  computeAbsolutePath(removeLastPathElement(libraryPath, true, false), book.indexPath);
-      } else {
-	book.indexPathAbsolute = book.indexPath;
-	book.indexPath =
-	  computeRelativePath(removeLastPathElement(libraryPath, true, false), book.indexPathAbsolute);
-      }
-    }
-  }
+  return true;
+}
 
 }

src/meson.build

@@ -1,8 +1,14 @@
 kiwix_sources = [
+  'book.cpp',
   'library.cpp',
   'manager.cpp',
+  'opds_dumper.cpp',
+  'downloader.cpp',
   'reader.cpp',
+  'entry.cpp',
   'searcher.cpp',
+  'subprocess.cpp',
+  'aria2.cpp',
   'common/base64.cpp',
   'common/pathTools.cpp',
   'common/regexTools.cpp',
@@ -14,19 +20,24 @@ kiwix_sources = [
 ]
 kiwix_sources += lib_resources
 
-if xapian_dep.found()
-  kiwix_sources += ['xapianSearcher.cpp']
-  if not get_option('android')
-    kiwix_sources += ['xapianIndexer.cpp']
-  endif
+if host_machine.system() == 'windows'
+  kiwix_sources += 'subprocess_windows.cpp'
+else
+  kiwix_sources += 'subprocess_unix.cpp'
 endif
 
-if not get_option('android')
-  kiwix_sources += ['indexer.cpp']
-else
-  subdir('android')
+if xapian_dep.found()
+  kiwix_sources += ['xapianSearcher.cpp']
 endif
 
+if get_option('android')
+  subdir('android')
+  install_dir = 'kiwix-lib/jniLibs/' + meson.get_cross_property('android_abi')
+else
+  install_dir = get_option('libdir')
+endif
+
+
 if has_ctpp2_dep
   kiwix_sources += ['ctpp2/CTPP2VMStringLoader.cpp']
 endif
@@ -40,5 +51,7 @@ kiwixlib = library('kiwix',
                    kiwix_sources,
                    include_directories : inc,
                    dependencies : all_deps,
-                   version: '1.0.0',
-                   install : true)
+                   version: meson.project_version(),
+                   install: true,
+                   install_dir: install_dir,
+                   install_rpath: '$ORIGIN')

src/opds_dumper.cpp

@@ -0,0 +1,137 @@
+/*
+ * Copyright 2017 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#include "opds_dumper.h"
+#include "book.h"
+
+#include <common/otherTools.h>
+
+namespace kiwix
+{
+/* Constructor */
+OPDSDumper::OPDSDumper(Library* library)
+  : library(library)
+{
+}
+/* Destructor */
+OPDSDumper::~OPDSDumper()
+{
+}
+
+std::string gen_date_str()
+{
+  auto now = time(0);
+  auto tm = localtime(&now);
+
+  std::stringstream is;
+  is << std::setw(2) << std::setfill('0')
+     << 1900+tm->tm_year << "-"
+     << std::setw(2) << std::setfill('0') << tm->tm_mon << "-"
+     << std::setw(2) << std::setfill('0') << tm->tm_mday << "T"
+     << std::setw(2) << std::setfill('0') << tm->tm_hour << ":"
+     << std::setw(2) << std::setfill('0') << tm->tm_min << ":"
+     << std::setw(2) << std::setfill('0') << tm->tm_sec << "Z";
+  return is.str();
+}
+
+void OPDSDumper::setOpenSearchInfo(int totalResults, int startIndex, int count)
+{
+  m_totalResults = totalResults;
+  m_startIndex = startIndex,
+  m_count = count;
+  m_isSearchResult = true;
+}
+
+#define ADD_TEXT_ENTRY(node, child, value) (node).append_child((child)).append_child(pugi::node_pcdata).set_value((value).c_str())
+
+pugi::xml_node OPDSDumper::handleBook(Book book, pugi::xml_node root_node) {
+  auto entry_node = root_node.append_child("entry");
+  ADD_TEXT_ENTRY(entry_node, "title", book.getTitle());
+  ADD_TEXT_ENTRY(entry_node, "id", "urn:uuid:"+book.getId());
+  ADD_TEXT_ENTRY(entry_node, "icon", rootLocation + "/meta?name=favicon&content=" + book.getHumanReadableIdFromPath());
+  ADD_TEXT_ENTRY(entry_node, "updated", date);
+  ADD_TEXT_ENTRY(entry_node, "summary", book.getDescription());
+
+  auto content_node = entry_node.append_child("link");
+  content_node.append_attribute("type") = "text/html";
+  content_node.append_attribute("href") = (rootLocation + "/" + book.getHumanReadableIdFromPath()).c_str();
+
+  auto author_node = entry_node.append_child("author");
+  ADD_TEXT_ENTRY(author_node, "name", book.getCreator());
+
+  if (! book.getUrl().empty()) {
+    auto acquisition_link = entry_node.append_child("link");
+    acquisition_link.append_attribute("rel") = "http://opds-spec.org/acquisition/open-access";
+    acquisition_link.append_attribute("type") = "application/x-zim";
+    acquisition_link.append_attribute("href") = book.getUrl().c_str();
+    acquisition_link.append_attribute("length") = to_string(book.getSize()).c_str();
+  }
+
+  if (! book.getFaviconMimeType().empty() ) {
+    auto image_link = entry_node.append_child("link");
+    image_link.append_attribute("rel") = "http://opds-spec.org/image/thumbnail";
+    image_link.append_attribute("type") = book.getFaviconMimeType().c_str();
+    image_link.append_attribute("href") = (rootLocation + "/meta?name=favicon&content=" + book.getHumanReadableIdFromPath()).c_str();
+  }
+  return entry_node;
+}
+
+string OPDSDumper::dumpOPDSFeed(const std::vector<std::string>& bookIds)
+{
+  date = gen_date_str();
+  pugi::xml_document doc;
+
+  auto root_node = doc.append_child("feed");
+  root_node.append_attribute("xmlns") = "http://www.w3.org/2005/Atom";
+  root_node.append_attribute("xmlns:opds") = "http://opds-spec.org/2010/catalog";
+
+  ADD_TEXT_ENTRY(root_node, "id", id);
+
+  ADD_TEXT_ENTRY(root_node, "title", title);
+  ADD_TEXT_ENTRY(root_node, "updated", date);
+
+  if (m_isSearchResult) {
+    ADD_TEXT_ENTRY(root_node, "totalResults", to_string(m_totalResults));
+    ADD_TEXT_ENTRY(root_node, "startIndex", to_string(m_startIndex));
+    ADD_TEXT_ENTRY(root_node, "itemsPerPage", to_string(m_count));
+  }
+
+  auto self_link_node = root_node.append_child("link");
+  self_link_node.append_attribute("rel") = "self";
+  self_link_node.append_attribute("href") = "";
+  self_link_node.append_attribute("type") = "application/atom+xml";
+
+
+  if (!searchDescriptionUrl.empty() ) {
+    auto search_link = root_node.append_child("link");
+    search_link.append_attribute("rel") = "search";
+    search_link.append_attribute("type") = "application/opensearchdescription+xml";
+    search_link.append_attribute("href") = searchDescriptionUrl.c_str();
+  }
+
+  if (library) {
+    for (auto& bookId: bookIds) {
+      handleBook(library->getBookById(bookId), root_node);
+    }
+  }
+
+  return nodeToString(root_node);
+}
+
+}

src/searcher.cpp

@@ -17,206 +17,449 @@
  * MA 02110-1301, USA.
  */
 
+
+#include <cmath>
+
 #include "searcher.h"
-#include "kiwixlib-resources.h"
+#include "reader.h"
+#include "xapianSearcher.h"
+
+#include <zim/search.h>
 
 #ifdef ENABLE_CTPP2
 #include <ctpp2/CDT.hpp>
 #include <ctpp2/CTPP2FileLogger.hpp>
 #include <ctpp2/CTPP2SimpleVM.hpp>
 #include "ctpp2/CTPP2VMStringLoader.hpp"
+#include "kiwixlib-resources.h"
 
 using namespace CTPP;
 #endif
 
+#define MAX_SEARCH_LEN 140
 
-namespace kiwix {
+namespace kiwix
+{
+class _Result : public Result
+{
+ public:
+  _Result(zim::Search::iterator& iterator);
+  virtual ~_Result(){};
 
-  /* Constructor */
-  Searcher::Searcher() :
-    searchPattern(""),
-    protocolPrefix("zim://"),
-    searchProtocolPrefix("search://?"),
-    resultCountPerPage(0),
-    estimatedResultCount(0),
-    resultStart(0),
-    resultEnd(0)
+  virtual std::string get_url();
+  virtual std::string get_title();
+  virtual int get_score();
+  virtual std::string get_snippet();
+  virtual std::string get_content();
+  virtual int get_wordCount();
+  virtual int get_size();
+  virtual int get_readerIndex();
+
+ private:
+  zim::Search::iterator iterator;
+};
+
+struct SearcherInternal {
+  const zim::Search* _search;
+  XapianSearcher* _xapianSearcher;
+  zim::Search::iterator current_iterator;
+
+  SearcherInternal() : _search(NULL), _xapianSearcher(NULL) {}
+  ~SearcherInternal()
   {
-    template_ct2 = RESOURCE::results_ct2;
-    loadICUExternalTables();
+    if (_search != NULL) {
+      delete _search;
+    }
+    if (_xapianSearcher != NULL) {
+      delete _xapianSearcher;
+    }
   }
-  
-  /* Destructor */
-  Searcher::~Searcher() {}
-  
-  /* Search strings in the database */
-  void Searcher::search(std::string &search, unsigned int resultStart,
-			unsigned int resultEnd, const bool verbose) {
-    this->reset();
+};
 
-    if (verbose == true) {
-      cout << "Performing query `" << search << "'" << endl;
+/* Constructor */
+Searcher::Searcher(const string& xapianDirectoryPath,
+                   Reader* reader,
+                   const string& humanReadableName)
+    : internal(new SearcherInternal()),
+      searchPattern(""),
+      protocolPrefix("zim://"),
+      searchProtocolPrefix("search://?"),
+      resultCountPerPage(0),
+      estimatedResultCount(0),
+      resultStart(0),
+      resultEnd(0),
+      contentHumanReadableId(humanReadableName)
+{
+  loadICUExternalTables();
+  if (!reader || !reader->hasFulltextIndex()) {
+    internal->_xapianSearcher = new XapianSearcher(xapianDirectoryPath, reader);
+  }
+  this->humanReaderNames.push_back(humanReadableName);
+}
+
+Searcher::Searcher(const std::string& humanReadableName)
+    : internal(new SearcherInternal()),
+      searchPattern(""),
+      protocolPrefix("zim://"),
+      searchProtocolPrefix("search://?"),
+      resultCountPerPage(0),
+      estimatedResultCount(0),
+      resultStart(0),
+      resultEnd(0),
+      contentHumanReadableId(humanReadableName)
+{
+  loadICUExternalTables();
+}
+
+/* Destructor */
+Searcher::~Searcher()
+{
+  delete internal;
+}
+
+bool Searcher::add_reader(Reader* reader, const std::string& humanReadableName)
+{
+  if (!reader->hasFulltextIndex()) {
+      return false;
+  }
+  this->readers.push_back(reader);
+  this->humanReaderNames.push_back(humanReadableName);
+  return true;
+}
+
+/* Search strings in the database */
+void Searcher::search(std::string& search,
+                      unsigned int resultStart,
+                      unsigned int resultEnd,
+                      const bool verbose)
+{
+  this->reset();
+
+  if (verbose == true) {
+    cout << "Performing query `" << search << "'" << endl;
+  }
+
+  /* If resultEnd & resultStart inverted */
+  if (resultStart > resultEnd) {
+    resultEnd += resultStart;
+    resultStart = resultEnd - resultStart;
+    resultEnd -= resultStart;
+  }
+
+  /* Try to find results */
+  if (resultStart != resultEnd) {
+    /* Avoid big researches */
+    this->resultCountPerPage = resultEnd - resultStart;
+    if (this->resultCountPerPage > MAX_SEARCH_LEN) {
+      resultEnd = resultStart + MAX_SEARCH_LEN;
+      this->resultCountPerPage = MAX_SEARCH_LEN;
     }
 
-    /* If resultEnd & resultStart inverted */
-    if (resultStart > resultEnd) {
-      resultEnd += resultStart;
-      resultStart = resultEnd - resultStart;
-      resultEnd -= resultStart; 
-    }
-
-    /* Try to find results */
-    if (resultStart != resultEnd) {
-
-      /* Avoid big researches */
-      this->resultCountPerPage = resultEnd - resultStart;
-      if (this->resultCountPerPage > 70) {
-	resultEnd = resultStart + 70;
-	this->resultCountPerPage = 70;
+    /* Perform the search */
+    this->searchPattern = search;
+    this->resultStart = resultStart;
+    this->resultEnd = resultEnd;
+    string unaccentedSearch = removeAccents(search);
+    if (internal->_xapianSearcher) {
+      internal->_xapianSearcher->searchInIndex(
+          unaccentedSearch, resultStart, resultEnd, verbose);
+      this->estimatedResultCount
+          = internal->_xapianSearcher->results.get_matches_estimated();
+    } else {
+      std::vector<const zim::File*> zims;
+      for (auto current = this->readers.begin(); current != this->readers.end();
+           current++) {
+        if ( (*current)->hasFulltextIndex() ) {
+            zims.push_back((*current)->getZimFileHandler());
+        }
       }
-
-      /* Perform the search */
-      this->searchPattern = search;
-      this->resultStart = resultStart;
-      this->resultEnd = resultEnd;
-      string unaccentedSearch = removeAccents(search);
-      searchInIndex(unaccentedSearch, resultStart, resultEnd, verbose);
-      this->resultOffset = this->results.begin();
+      zim::Search* search = new zim::Search(zims);
+      search->set_query(unaccentedSearch);
+      search->set_range(resultStart, resultEnd);
+      internal->_search = search;
+      internal->current_iterator = internal->_search->begin();
+      this->estimatedResultCount = internal->_search->get_matches_estimated();
     }
+  }
 
+  return;
+}
+
+
+void Searcher::geo_search(float latitude, float longitude, float distance,
+                          unsigned int resultStart,
+                          unsigned int resultEnd,
+                          const bool verbose)
+{
+  this->reset();
+
+  if (verbose == true) {
+    cout << "Performing geo query `" << distance << "&(" << latitude << ";" << longitude << ")'" << endl;
+  }
+
+  /* If resultEnd & resultStart inverted */
+  if (resultStart > resultEnd) {
+    resultEnd += resultStart;
+    resultStart = resultEnd - resultStart;
+    resultEnd -= resultStart;
+  }
+
+  /* Try to find results */
+  if (resultStart == resultEnd) {
     return;
   }
 
-  /* Reset the results */
-  void Searcher::reset() {
-    this->results.clear();
-    this->resultOffset = this->results.begin();
+  if (internal->_xapianSearcher) {
+    return;
+  }
+
+  /* Avoid big researches */
+  this->resultCountPerPage = resultEnd - resultStart;
+  if (this->resultCountPerPage > MAX_SEARCH_LEN) {
+    resultEnd = resultStart + MAX_SEARCH_LEN;
+    this->resultCountPerPage = MAX_SEARCH_LEN;
+  }
+
+  /* Perform the search */
+  std::ostringstream oss;
+  oss << "Articles located less than " << distance << " meters of " << latitude << ";" << longitude;
+  this->searchPattern = oss.str();
+  this->resultStart = resultStart;
+  this->resultEnd = resultEnd;
+
+  std::vector<const zim::File*> zims;
+  for (auto current = this->readers.begin(); current != this->readers.end();
+       current++) {
+    zims.push_back((*current)->getZimFileHandler());
+  }
+  zim::Search* search = new zim::Search(zims);
+  search->set_query("");
+  search->set_georange(latitude, longitude, distance);
+  search->set_range(resultStart, resultEnd);
+  internal->_search = search;
+  internal->current_iterator = internal->_search->begin();
+  this->estimatedResultCount = internal->_search->get_matches_estimated();
+}
+
+
+void Searcher::restart_search()
+{
+  if (internal->_xapianSearcher) {
+    internal->_xapianSearcher->restart_search();
+  } else if (internal->_search) {
+    internal->current_iterator = internal->_search->begin();
+  }
+}
+
+Result* Searcher::getNextResult()
+{
+  if (internal->_xapianSearcher) {
+    return internal->_xapianSearcher->getNextResult();
+  } else if (internal->_search &&
+             internal->current_iterator != internal->_search->end()) {
+    Result* result = new _Result(internal->current_iterator);
+    internal->current_iterator++;
+    return result;
+  }
+  return NULL;
+}
+
+/* Reset the results */
+void Searcher::reset()
+{
+  this->estimatedResultCount = 0;
+  this->searchPattern = "";
+  return;
+}
+
+void Searcher::suggestions(std::string& search, const bool verbose)
+{
+  this->reset();
+
+  if (verbose == true) {
+    cout << "Performing suggestion query `" << search << "`" << endl;
+  }
+
+  this->searchPattern = search;
+  this->resultStart = 0;
+  this->resultEnd = 10;
+  string unaccentedSearch = removeAccents(search);
+
+  if (internal->_xapianSearcher) {
+    /* [TODO] Suggestion on a external database ?
+     * We do not support that. */
     this->estimatedResultCount = 0;
-    this->searchPattern = "";
-    return;
-  }
-
-  /* Return the result count estimation */
-  unsigned int Searcher::getEstimatedResultCount() {
-    return this->estimatedResultCount;
-  }
-
-  /* Get next result */
-  bool Searcher::getNextResult(string &url, string &title, unsigned int &score) {
-    bool retVal = false;
-
-    if (this->resultOffset != this->results.end()) {
-
-      /* url */
-      url = this->resultOffset->url;
-
-      /* title */
-      title = this->resultOffset->title;
-
-      /* score */
-      score =  this->resultOffset->score;
-
-      /* increment the cursor for the next call */
-      this->resultOffset++;
-
-      retVal = true;
+  } else {
+    std::vector<const zim::File*> zims;
+    for (auto current = this->readers.begin(); current != this->readers.end();
+         current++) {
+      zims.push_back((*current)->getZimFileHandler());
     }
-
-    return retVal;
+    zim::Search* search = new zim::Search(zims);
+    search->set_query(unaccentedSearch);
+    search->set_range(resultStart, resultEnd);
+    search->set_suggestion_mode(true);
+    internal->_search = search;
+    internal->current_iterator = internal->_search->begin();
+    this->estimatedResultCount = internal->_search->get_matches_estimated();
   }
+}
 
-  bool Searcher::setProtocolPrefix(const std::string prefix) {
-    this->protocolPrefix = prefix;
-    return true;
+/* Return the result count estimation */
+unsigned int Searcher::getEstimatedResultCount()
+{
+  return this->estimatedResultCount;
+}
+
+bool Searcher::setProtocolPrefix(const std::string prefix)
+{
+  this->protocolPrefix = prefix;
+  return true;
+}
+
+bool Searcher::setSearchProtocolPrefix(const std::string prefix)
+{
+  this->searchProtocolPrefix = prefix;
+  return true;
+}
+
+_Result::_Result(zim::Search::iterator& iterator)
+    : iterator(iterator)
+{
+}
+
+std::string _Result::get_url()
+{
+  return iterator.get_url();
+}
+std::string _Result::get_title()
+{
+  return iterator.get_title();
+}
+int _Result::get_score()
+{
+  return iterator.get_score();
+}
+std::string _Result::get_snippet()
+{
+  return iterator.get_snippet();
+}
+std::string _Result::get_content()
+{
+  if (iterator->good()) {
+    return iterator->getData();
   }
-
-  bool Searcher::setSearchProtocolPrefix(const std::string prefix) {
-    this->searchProtocolPrefix = prefix;
-    return true;
-  }
-
-  void Searcher::setContentHumanReadableId(const string &contentHumanReadableId) {
-    this->contentHumanReadableId = contentHumanReadableId;
-  }
-
+  return "";
+}
+int _Result::get_size()
+{
+  return iterator.get_size();
+}
+int _Result::get_wordCount()
+{
+  return iterator.get_wordCount();
+}
+int _Result::get_readerIndex()
+{
+  return iterator.get_fileIndex();
+}
 #ifdef ENABLE_CTPP2
-  
-  string Searcher::getHtml() {
 
-    SimpleVM oSimpleVM;
+string Searcher::getHtml()
+{
+  SimpleVM oSimpleVM(
+      1024, //iIMaxFunctions (default value)
+      4096, //iIMaxArgStackSize (default value)
+      4096, //iIMaxCodeStackSize (default value)
+      10240 * 2 //iIMaxSteps (default*2)
+  );
 
-    // Fill data
-    CDT oData;
-    CDT resultsCDT(CDT::ARRAY_VAL);
+  // Fill data
+  CDT oData;
+  CDT resultsCDT(CDT::ARRAY_VAL);
 
-    this->resultOffset = this->results.begin();
-    while (this->resultOffset != this->results.end()) {
-      CDT result;
-      result["title"] = this->resultOffset->title;
-      result["url"] = this->resultOffset->url;
-      result["snippet"] = this->resultOffset->snippet;
+  this->restart_search();
+  Result* p_result = NULL;
+  while ((p_result = this->getNextResult())) {
+    CDT result;
+    result["title"] = p_result->get_title();
+    result["url"] = p_result->get_url();
+    result["snippet"] = p_result->get_snippet();
+    result["contentId"] = humanReaderNames[p_result->get_readerIndex()];
 
-      if (this->resultOffset->size >= 0)
-    result["size"] = kiwix::beautifyInteger(this->resultOffset->size);
-
-      if (this->resultOffset->wordCount >= 0)
-    result["wordCount"] = kiwix::beautifyInteger(this->resultOffset->wordCount);
-
-      resultsCDT.PushBack(result);
-      this->resultOffset++;
+    if (p_result->get_size() >= 0) {
+      result["size"] = kiwix::beautifyInteger(p_result->get_size());
     }
-    this->resultOffset = this->results.begin();
-    oData["results"] = resultsCDT;
 
-    // pages
-    CDT pagesCDT(CDT::ARRAY_VAL);
-
-    unsigned int pageStart = this->resultStart / this->resultCountPerPage >= 5 ? this->resultStart / this->resultCountPerPage - 4 : 0;
-    unsigned int pageCount = this->estimatedResultCount / this->resultCountPerPage + 1 - pageStart;
-
-    if (pageCount > 10)
-      pageCount = 10;
-    else if (pageCount == 1)
-      pageCount = 0;
-
-    for (unsigned int i=pageStart; i<pageStart+pageCount; i++) {
-      CDT page;
-      page["label"] = i + 1;
-      page["start"] = i * this->resultCountPerPage;
-      page["end"] = (i+1) * this->resultCountPerPage;
-
-      if (i * this->resultCountPerPage == this->resultStart)
-    page["selected"] = true;
-
-      pagesCDT.PushBack(page);
+    if (p_result->get_wordCount() >= 0) {
+      result["wordCount"] = kiwix::beautifyInteger(p_result->get_wordCount());
     }
-    oData["pages"] = pagesCDT;
-
-    oData["count"] = kiwix::beautifyInteger(this->estimatedResultCount);
-    oData["searchPattern"] = kiwix::encodeDiples(this->searchPattern);
-    oData["searchPatternEncoded"] = urlEncode(this->searchPattern);
-    oData["resultStart"] = this->resultStart + 1;
-    oData["resultEnd"] = (this->resultEnd > this->estimatedResultCount ? this->estimatedResultCount : this->resultEnd);
-    oData["resultRange"] = this->resultCountPerPage;
-    oData["resultLastPageStart"] = this->estimatedResultCount > this->resultCountPerPage ? this->estimatedResultCount - this->resultCountPerPage : 0;
-    oData["protocolPrefix"] = this->protocolPrefix;
-    oData["searchProtocolPrefix"] = this->searchProtocolPrefix;
-    oData["contentId"] = this->contentHumanReadableId;
-
-    VMStringLoader oLoader(template_ct2.c_str(), template_ct2.size());
-
-    FileLogger oLogger(stderr);
-
-    // DEBUG only (write output to stdout)
-    // oSimpleVM.Run(oData, oLoader, stdout, oLogger);
-
-    std::string sResult;
-    oSimpleVM.Run(oData, oLoader, sResult, oLogger);
-
-    return sResult;
 
+    resultsCDT.PushBack(result);
+    delete p_result;
   }
+  this->restart_search();
+  oData["results"] = resultsCDT;
+
+  // pages
+  CDT pagesCDT(CDT::ARRAY_VAL);
+
+  unsigned int pageStart
+      = this->resultStart / this->resultCountPerPage >= 5
+            ? this->resultStart / this->resultCountPerPage - 4
+            : 0;
+  unsigned int pageCount
+      = this->estimatedResultCount / this->resultCountPerPage + 1 - pageStart;
+
+  if (pageCount > 10) {
+    pageCount = 10;
+  } else if (pageCount == 1) {
+    pageCount = 0;
+  }
+
+  for (unsigned int i = pageStart; i < pageStart + pageCount; i++) {
+    CDT page;
+    page["label"] = i + 1;
+    page["start"] = i * this->resultCountPerPage;
+    page["end"] = (i + 1) * this->resultCountPerPage;
+
+    if (i * this->resultCountPerPage == this->resultStart) {
+      page["selected"] = true;
+    }
+
+    pagesCDT.PushBack(page);
+  }
+  oData["pages"] = pagesCDT;
+
+  oData["count"] = kiwix::beautifyInteger(this->estimatedResultCount);
+  oData["searchPattern"] = kiwix::encodeDiples(this->searchPattern);
+  oData["searchPatternEncoded"] = urlEncode(this->searchPattern);
+  oData["resultStart"] = this->resultStart + 1;
+  oData["resultEnd"] = (this->resultEnd > this->estimatedResultCount
+                            ? this->estimatedResultCount
+                            : this->resultEnd);
+  oData["resultRange"] = this->resultCountPerPage;
+  oData["resultLastPageStart"]
+      = this->estimatedResultCount > this->resultCountPerPage
+            ? std::round(this->estimatedResultCount / this->resultCountPerPage) * this->resultCountPerPage
+            : 0;
+  oData["protocolPrefix"] = this->protocolPrefix;
+  oData["searchProtocolPrefix"] = this->searchProtocolPrefix;
+  oData["contentId"] = this->contentHumanReadableId;
+
+  std::string template_ct2 = RESOURCE::results_ct2;
+  VMStringLoader oLoader(template_ct2.c_str(), template_ct2.size());
+
+  FileLogger oLogger(stderr);
+
+  // DEBUG only (write output to stdout)
+  // oSimpleVM.Run(oData, oLoader, stdout, oLogger);
+
+  std::string sResult;
+  oSimpleVM.Run(oData, oLoader, sResult, oLogger);
+
+  return sResult;
+}
 
 #endif
-
 }

src/subprocess.cpp

@@ -0,0 +1,40 @@
+
+
+#include "subprocess.h"
+
+#ifdef _WIN32
+# include "subprocess_windows.h"
+#else
+# include "subprocess_unix.h"
+#endif
+
+Subprocess::Subprocess(std::unique_ptr<SubprocessImpl> impl, commandLine_t& commandLine) :
+  mp_impl(std::move(impl))
+{
+  mp_impl->run(commandLine);
+}
+
+Subprocess::~Subprocess()
+{
+  mp_impl->kill();
+}
+
+std::unique_ptr<Subprocess> Subprocess::run(commandLine_t& commandLine)
+{
+#ifdef _WIN32
+  auto impl = std::unique_ptr<SubprocessImpl>(new WinImpl);
+#else
+  auto impl = std::unique_ptr<UnixImpl>(new UnixImpl);
+#endif
+  return std::unique_ptr<Subprocess>(new Subprocess(std::move(impl), commandLine));
+}
+
+bool Subprocess::isRunning()
+{
+  return mp_impl->isRunning();
+}
+
+bool Subprocess::kill()
+{
+  return mp_impl->kill();
+}

src/subprocess.h

@@ -0,0 +1,36 @@
+
+#ifndef KIWIX_SUBPROCESS_H_
+#define KIWIX_SUBPROCESS_H_
+
+#include <string>
+#include <memory>
+#include <vector>
+
+typedef std::vector<const char *> commandLine_t;
+
+class SubprocessImpl
+{
+  public:
+    virtual void run(commandLine_t& commandLine) = 0;
+    virtual bool kill() = 0;
+    virtual bool isRunning() = 0;
+    virtual ~SubprocessImpl() = default;
+};
+
+class Subprocess
+{
+  private:
+    // Impl depends of the system (window, unix, ...)
+    std::unique_ptr<SubprocessImpl> mp_impl;
+    Subprocess(std::unique_ptr<SubprocessImpl> impl, commandLine_t& commandLine);
+
+  public:
+    static std::unique_ptr<Subprocess> run(commandLine_t& commandLine);
+    ~Subprocess();
+
+    bool isRunning();
+    bool kill();
+};
+
+
+#endif // KIWIX_SUBPROCESS_H_

src/subprocess_unix.cpp

@@ -0,0 +1,93 @@
+
+
+#include "subprocess_unix.h"
+
+#include <sys/types.h>
+#include <unistd.h>
+#include <iostream>
+#include <signal.h>
+#include <sys/wait.h>
+#include <stdlib.h>
+
+UnixImpl::UnixImpl():
+  m_pid(0),
+  m_running(false),
+  m_mutex(PTHREAD_MUTEX_INITIALIZER),
+  m_waitingThread()
+{
+}
+
+UnixImpl::~UnixImpl()
+{
+  kill();
+// Android has no pthread_cancel :(
+#ifdef __ANDROID__
+  pthread_kill(m_waitingThread, SIGUSR1);
+#else
+  pthread_cancel(m_waitingThread);
+#endif
+}
+
+#ifdef __ANDROID__
+void thread_exit_handler(int sig) {
+  pthread_exit(0);
+}
+#endif
+
+void* UnixImpl::waitForPID(void* _self)
+{
+#ifdef __ANDROID__
+  struct sigaction actions;
+  memset(&actions, 0, sizeof(actions));
+  sigemptyset(&actions.sa_mask);
+  actions.sa_flags = 0;
+  actions.sa_handler = thread_exit_handler;
+  sigaction(SIGUSR1, &actions, NULL);
+#endif
+
+  UnixImpl* self = static_cast<UnixImpl*>(_self);
+  waitpid(self->m_pid, NULL, WEXITED);
+
+  pthread_mutex_lock(&self->m_mutex);
+  self->m_running = false;
+  pthread_mutex_unlock(&self->m_mutex);
+
+  return self;
+}
+
+void UnixImpl::run(commandLine_t& commandLine)
+{
+  const char* binary = commandLine[0];
+  int pid = fork();
+  switch(pid) {
+    case -1:
+      std::cerr << "cannot fork" << std::endl;
+      break;
+    case 0:
+      commandLine.push_back(NULL);
+      if (execvp(binary, const_cast<char* const*>(commandLine.data()))) {
+        perror("Cannot launch\n");
+        exit(-1);
+      }
+
+      break;
+    default:
+      m_pid = pid;
+      m_running = true;
+      pthread_create(&m_waitingThread, NULL, waitForPID, this);
+      break;
+  }
+}
+
+bool UnixImpl::kill()
+{
+  return (::kill(m_pid, SIGKILL) == 0);
+}
+
+bool UnixImpl::isRunning()
+{
+  pthread_mutex_lock(&m_mutex);
+  bool ret = m_running;
+  pthread_mutex_unlock(&m_mutex);
+  return ret;
+}

src/subprocess_unix.h

@@ -0,0 +1,28 @@
+#ifndef KIWIX_SUBPROCESS_UNIX_H_
+#define KIWIX_SUBPROCESS_UNIX_H_
+
+#include "subprocess.h"
+
+#include <pthread.h>
+
+
+class UnixImpl : public SubprocessImpl
+{
+  private:
+    int m_pid;
+    bool m_running;
+    pthread_mutex_t m_mutex;
+    pthread_t m_waitingThread;
+
+  public:
+    UnixImpl();
+    virtual ~UnixImpl();
+
+    void run(commandLine_t& commandLine);
+    bool kill();
+    bool isRunning();
+
+    static void* waitForPID(void* self);
+};
+
+#endif //KIWIX_SUBPROCESS_UNIX_H_

src/subprocess_windows.cpp

@@ -0,0 +1,93 @@
+
+
+#include "subprocess_windows.h"
+
+#include <windows.h>
+#include <winbase.h>
+#include <iostream>
+#include <sstream>
+
+WinImpl::WinImpl():
+  m_pid(0),
+  m_running(false),
+  m_handle(INVALID_HANDLE_VALUE)
+{
+  InitializeCriticalSection(&m_criticalSection);
+}
+
+WinImpl::~WinImpl()
+{
+  kill();
+  CloseHandle(m_handle);
+  DeleteCriticalSection(&m_criticalSection);
+}
+
+DWORD WINAPI WinImpl::waitForPID(void* _self)
+{
+  WinImpl* self = static_cast<WinImpl*>(_self);
+  WaitForSingleObject(self->m_handle, INFINITE);
+
+  EnterCriticalSection(&self->m_criticalSection);
+  self->m_running = false;
+  LeaveCriticalSection(&self->m_criticalSection);
+
+  return 0;
+}
+
+std::unique_ptr<wchar_t[]> toWideChar(const std::string& value)
+{
+  auto size = MultiByteToWideChar(CP_UTF8, 0,
+                value.c_str(), -1, nullptr, 0);
+  auto wdata = std::unique_ptr<wchar_t[]>(new wchar_t[size]);
+  auto ret = MultiByteToWideChar(CP_UTF8, 0,
+                value.c_str(), -1, wdata.get(), size);
+  if (0 == ret) {
+    std::ostringstream oss;
+    oss << "Cannot convert to wchar : " << GetLastError();
+    throw std::runtime_error(oss.str());
+  }
+  return wdata;
+}
+
+
+void WinImpl::run(commandLine_t& commandLine)
+{
+  STARTUPINFOW startInfo = {0};
+  PROCESS_INFORMATION procInfo;
+  startInfo.cb = sizeof(startInfo);
+  std::ostringstream oss;
+  for(auto& item: commandLine) {
+    oss << item << " ";
+  }
+  auto wCommandLine = toWideChar(oss.str());
+  if (CreateProcessW(
+    NULL,
+    wCommandLine.get(),
+    NULL,
+    NULL,
+    false,
+    CREATE_NO_WINDOW,
+    NULL,
+    NULL,
+    &startInfo,
+    &procInfo)) {
+    m_pid = procInfo.dwProcessId;
+    m_handle = procInfo.hProcess;
+    CloseHandle(procInfo.hThread);
+    m_running = true;
+    CreateThread(NULL, 0, &waitForPID, this, 0, NULL	);
+  }
+}
+
+bool WinImpl::kill()
+{
+  return TerminateProcess(m_handle, 0);
+}
+
+bool WinImpl::isRunning()
+{
+  EnterCriticalSection(&m_criticalSection);
+  bool ret = m_running;
+  LeaveCriticalSection(&m_criticalSection);
+  return ret;
+}

src/subprocess_windows.h

@@ -0,0 +1,28 @@
+#ifndef KIWIX_SUBPROCESS_WINDOWS_H_
+#define KIWIX_SUBPROCESS_WINDOWS_H_
+
+#include "subprocess.h"
+
+#include <windows.h>
+#include <synchapi.h>
+
+class WinImpl : public SubprocessImpl
+{
+  private:
+    int m_pid;
+    bool m_running;
+    HANDLE m_handle;
+    CRITICAL_SECTION m_criticalSection;
+
+  public:
+    WinImpl();
+    virtual ~WinImpl();
+
+    void run(commandLine_t& commandLine);
+    bool kill();
+    bool isRunning();
+
+    static DWORD WINAPI waitForPID(void* self);
+};
+
+#endif //KIWIX_SUBPROCESS_WINDOWS_H_

src/xapian/myhtmlparse.h

@@ -40,6 +40,7 @@ class MyHtmlParser : public HtmlParser {
 	void process_text(const string &text);
 	void opening_tag(const string &tag);
 	void closing_tag(const string &tag);
+        using HtmlParser::parse_html;
 	void parse_html(const string &text, const string &charset_,
 			bool charset_from_meta_);
 	MyHtmlParser() :

src/xapianIndexer.cpp

@@ -1,111 +0,0 @@
-/*
- * Copyright 2011 Emmanuel Engelhart <kelson@kiwix.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU  General Public License as published by
- * the Free Software Foundation; either version 3 of the License, or
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
-
-#include "xapianIndexer.h"
-
-namespace kiwix {
-
-  /* Constructor */
-  XapianIndexer::XapianIndexer() {
-    /*
-    stemmer(Xapian::Stem("french")) {
-    this->indexer.set_stemmer(this->stemmer);
-    */ 
-  }
-
-  void XapianIndexer::indexingPrelude(const string indexPath) {
-    this->writableDatabase = Xapian::WritableDatabase(indexPath+".tmp", Xapian::DB_CREATE_OR_OVERWRITE | Xapian::DB_BACKEND_GLASS);
-    this->writableDatabase.begin_transaction(true);
-
-    /* Insert the stopwords */
-    if (!this->stopWords.empty()) {
-      std::vector<std::string>::iterator it = this->stopWords.begin();
-      for( ; it != this->stopWords.end(); ++it) {
-	this->stopper.add(*it);
-      }
-
-      this->indexer.set_stopper(&(this->stopper));
-    }
-  }
-  
-  void XapianIndexer::index(const string &url, 
-			    const string &title, 
-			    const string &unaccentedTitle,
-			    const string &keywords, 
-			    const string &content,
-			    const string &snippet,
-			    const string &size,
-			    const string &wordCount) {
-    
-    /* Put the data in the document */
-    Xapian::Document currentDocument; 
-    currentDocument.clear_values();
-    currentDocument.add_value(0, title);
-    currentDocument.add_value(1, snippet);
-    currentDocument.add_value(2, size);
-    currentDocument.add_value(3, wordCount);
-    currentDocument.set_data(url);
-    indexer.set_document(currentDocument);
-
-    /* Index the title */
-    if (!unaccentedTitle.empty()) {
-      this->indexer.index_text_without_positions(unaccentedTitle, this->getTitleBoostFactor(content.size()));
-    }
-    
-    /* Index the keywords */
-    if (!keywords.empty()) {
-      this->indexer.index_text_without_positions(keywords, keywordsBoostFactor);
-    }
-    
-    /* Index the content */
-    if (!content.empty()) {
-      this->indexer.index_text_without_positions(content);
-    }
-    
-    /* add to the database */
-    this->writableDatabase.add_document(currentDocument);
-  }
-  
-  void XapianIndexer::flush() {
-    this->writableDatabase.commit_transaction();
-    this->writableDatabase.begin_transaction(true);
-  }
-
-  void XapianIndexer::indexingPostlude(const string indexPath) {
-    this->flush();
-    this->writableDatabase.commit_transaction();
-#ifdef _WIN32
-    this->writableDatabase.close();
-#endif
-    
-    /* Compacting the index */
-    Xapian::Compactor compactor;
-    try {
-      Xapian::Database src;
-      src.add_database(Xapian::Database(indexPath+".tmp"));
-      src.compact(indexPath, Xapian::Compactor::FULL | Xapian::DBCOMPACT_SINGLE_FILE, 0, compactor);
-    } catch (const Xapian::Error &error) {
-      cerr << indexPath << ": " << error.get_description() << endl;
-      exit(1);
-    } catch (const char * msg) {
-      cerr << indexPath << ": " << msg << endl;
-      exit(1);
-    }
-  }
-}

src/xapianSearcher.cpp

@@ -18,82 +18,214 @@
  */
 
 #include "xapianSearcher.h"
-#include <zim/zim.h>
-#include <zim/file.h>
+#include <sys/types.h>
+#include <unicode/locid.h>
+#ifndef _WIN32
+#  include <unistd.h>
+#endif
 #include <zim/article.h>
 #include <zim/error.h>
-#include <sys/types.h>
-#include <unistd.h>
+#include <zim/file.h>
+#include <zim/zim.h>
+#include "xapian/myhtmlparse.h"
 
-namespace kiwix {
+#include <vector>
 
-  /* Constructor */
-  XapianSearcher::XapianSearcher(const string &xapianDirectoryPath) 
-    : Searcher(),
-      stemmer(Xapian::Stem("english")) {
-    this->openIndex(xapianDirectoryPath);
+namespace kiwix
+{
+std::map<std::string, int> read_valuesmap(const std::string& s)
+{
+  std::map<std::string, int> result;
+  std::vector<std::string> elems = split(s, ";");
+  for (std::vector<std::string>::iterator elem = elems.begin();
+       elem != elems.end();
+       elem++) {
+    std::vector<std::string> tmp_elems = split(*elem, ":");
+    result.insert(
+        std::pair<std::string, int>(tmp_elems[0], atoi(tmp_elems[1].c_str())));
   }
+  return result;
+}
 
-  /* Open Xapian readable database */
-  void XapianSearcher::openIndex(const string &directoryPath) {
-    try
-    {
-      zim::File zimFile = zim::File(directoryPath);
-      zim::Article xapianArticle = zimFile.getArticle('Z', "/fulltextIndex/xapian");
-      if (!xapianArticle.good())
-	throw NoXapianIndexInZim();
-      zim::offset_type dbOffset = xapianArticle.getOffset();
-      int databasefd = open(directoryPath.c_str(), O_RDONLY);
-      lseek(databasefd, dbOffset, SEEK_SET);
-      this->readableDatabase = Xapian::Database(databasefd);
+/* Constructor */
+XapianSearcher::XapianSearcher(const string& xapianDirectoryPath,
+                               Reader* reader)
+    : reader(reader)
+{
+  this->openIndex(xapianDirectoryPath);
+}
+
+/* Open Xapian readable database */
+void XapianSearcher::openIndex(const string& directoryPath)
+{
+  this->readableDatabase = Xapian::Database(directoryPath);
+  this->valuesmap
+      = read_valuesmap(this->readableDatabase.get_metadata("valuesmap"));
+  this->language = this->readableDatabase.get_metadata("language");
+  this->stopwords = this->readableDatabase.get_metadata("stopwords");
+  setup_queryParser();
+}
+
+/* Close Xapian writable database */
+void XapianSearcher::closeIndex()
+{
+  return;
+}
+void XapianSearcher::setup_queryParser()
+{
+  queryParser.set_database(readableDatabase);
+  if (!language.empty()) {
+    /* Build ICU Local object to retrieve ISO-639 language code (from
+       ISO-639-3) */
+    icu::Locale languageLocale(language.c_str());
+
+    /* Configuring language base steemming */
+    try {
+      stemmer = Xapian::Stem(languageLocale.getLanguage());
+      queryParser.set_stemmer(stemmer);
+      queryParser.set_stemming_strategy(Xapian::QueryParser::STEM_ALL);
     } catch (...) {
-      this->readableDatabase = Xapian::Database(directoryPath);
+      std::cout << "No steemming for language '" << languageLocale.getLanguage()
+                << "'" << std::endl;
     }
   }
-  
-  /* Close Xapian writable database */
-  void XapianSearcher::closeIndex() {
-    return;
-  }
-  
-  /* Search strings in the database */
-  void XapianSearcher::searchInIndex(string &search, const unsigned int resultStart, 
-				     const unsigned int resultEnd, const bool verbose) {
-    /* Create the query */
-    Xapian::QueryParser queryParser;
-    Xapian::Query query = queryParser.parse_query(search);    
 
-    /* Create the enquire object */
-    Xapian::Enquire enquire(this->readableDatabase);
-    enquire.set_query(query);
-
-    /* Get the results */
-    Xapian::MSet matches = enquire.get_mset(resultStart, resultEnd - resultStart);
-    
-    Xapian::MSetIterator i;
-    for (i = matches.begin(); i != matches.end(); ++i) {
-      Xapian::Document doc = i.get_document();
-      
-      Result result;
-      result.url = doc.get_data();
-      result.title = doc.get_value(0);
-      result.snippet = doc.get_value(1);
-      result.size = (doc.get_value(2).empty() == true ? -1 : atoi(doc.get_value(2).c_str()));
-      result.wordCount = (doc.get_value(3).empty() == true ? -1 : atoi(doc.get_value(3).c_str()));
-      result.score = i.get_percent();
-      
-      this->results.push_back(result);
-
-      if (verbose) {
-	std::cout << "Document ID " << *i << "   \t";
-	std::cout << i.get_percent() << "% ";
-	std::cout << "\t[" << doc.get_data() << "] - " << doc.get_value(0) << std::endl;
-      }
+  if (!stopwords.empty()) {
+    std::string stopWord;
+    std::istringstream file(this->stopwords);
+    while (std::getline(file, stopWord, '\n')) {
+      this->stopper.add(stopWord);
     }
-
-    /* Update the global resultCount value*/
-    this->estimatedResultCount = matches.get_matches_estimated();
-
-    return;
+    queryParser.set_stopper(&(this->stopper));
   }
 }
+
+/* Search strings in the database */
+void XapianSearcher::searchInIndex(string& search,
+                                   const unsigned int resultStart,
+                                   const unsigned int resultEnd,
+                                   const bool verbose)
+{
+  /* Create the query */
+  Xapian::Query query = queryParser.parse_query(search);
+
+  /* Create the enquire object */
+  Xapian::Enquire enquire(this->readableDatabase);
+  enquire.set_query(query);
+
+  /* Get the results */
+  this->results = enquire.get_mset(resultStart, resultEnd - resultStart);
+  this->current_result = this->results.begin();
+}
+
+/* Get next result */
+Result* XapianSearcher::getNextResult()
+{
+  if (this->current_result != this->results.end()) {
+    XapianResult* result = new XapianResult(this, this->current_result);
+    this->current_result++;
+    return result;
+  }
+  return NULL;
+}
+
+void XapianSearcher::restart_search()
+{
+  this->current_result = this->results.begin();
+}
+
+XapianResult::XapianResult(XapianSearcher* searcher,
+                           Xapian::MSetIterator& iterator)
+    : searcher(searcher), iterator(iterator), document(iterator.get_document())
+{
+}
+
+std::string XapianResult::get_url()
+{
+  return document.get_data();
+}
+std::string XapianResult::get_title()
+{
+  if (searcher->valuesmap.empty()) {
+    /* This is the old legacy version. Guess and try */
+    return document.get_value(0);
+  } else if (searcher->valuesmap.find("title") != searcher->valuesmap.end()) {
+    return document.get_value(searcher->valuesmap["title"]);
+  }
+  return "";
+}
+
+int XapianResult::get_score()
+{
+  return iterator.get_percent();
+}
+std::string XapianResult::get_snippet()
+{
+  if (searcher->valuesmap.empty()) {
+    /* This is the old legacy version. Guess and try */
+    std::string stored_snippet = document.get_value(1);
+    if (!stored_snippet.empty()) {
+      return stored_snippet;
+    }
+    /* Let's continue here, and see if we can genenate one */
+  } else if (searcher->valuesmap.find("snippet") != searcher->valuesmap.end()) {
+    return document.get_value(searcher->valuesmap["snippet"]);
+  }
+  /* No reader, no snippet */
+  if (!searcher->reader) {
+    return "";
+  }
+  /* Get the content of the article to generate a snippet.
+     We parse it and use the html dump to avoid remove html tags in the
+     content and be able to nicely cut the text at random place. */
+  MyHtmlParser htmlParser;
+  std::string content = get_content();
+  if (content.empty()) {
+    return content;
+  }
+  try {
+    htmlParser.parse_html(content, "UTF-8", true);
+  } catch (...) {
+  }
+  return searcher->results.snippet(htmlParser.dump, 500);
+}
+
+std::string XapianResult::get_content()
+{
+  if (!searcher->reader) {
+    return "";
+  }
+  auto entry = searcher->reader->getEntryFromEncodedPath(get_url());
+  return entry.getContent();
+}
+
+int XapianResult::get_size()
+{
+  if (searcher->valuesmap.empty()) {
+    /* This is the old legacy version. Guess and try */
+    return document.get_value(2).empty() == true
+               ? -1
+               : atoi(document.get_value(2).c_str());
+  } else if (searcher->valuesmap.find("size") != searcher->valuesmap.end()) {
+    return atoi(document.get_value(searcher->valuesmap["size"]).c_str());
+  }
+  /* The size is never used. Do we really want to get the content and
+     calculate the size ? */
+  return -1;
+}
+
+int XapianResult::get_wordCount()
+{
+  if (searcher->valuesmap.empty()) {
+    /* This is the old legacy version. Guess and try */
+    return document.get_value(3).empty() == true
+               ? -1
+               : atoi(document.get_value(3).c_str());
+  } else if (searcher->valuesmap.find("wordcount")
+             != searcher->valuesmap.end()) {
+    return atoi(document.get_value(searcher->valuesmap["wordcount"]).c_str());
+  }
+  return -1;
+}
+
+}  // Kiwix namespace

src/xmlrpc.h

@@ -0,0 +1,264 @@
+
+
+#ifndef KIWIX_XMLRPC_H_
+#define KIWIX_XMLRPC_H_
+
+#include <common/otherTools.h>
+
+namespace kiwix {
+
+class InvalidRPCNode : public std::runtime_error {
+  public:
+    InvalidRPCNode(const std::string& msg) : std::runtime_error(msg) {};
+};
+
+class Struct;
+class Array;
+
+class Value {
+    pugi::xml_node m_value;
+
+  public:
+    Value(pugi::xml_node value) : m_value(value) { }
+
+    void set(int value) {
+      if (!m_value.child("int"))
+        m_value.append_child("int");
+      m_value.child("int").text().set(value);
+    };
+
+    int getAsI() const {
+      if (!m_value.child("int"))
+        throw InvalidRPCNode("Type Error");
+      return m_value.child("int").text().as_int();
+    }
+
+    void set(bool value) {
+      if (!m_value.child("boolean"))
+        m_value.append_child("boolean");
+      m_value.child("boolean").text().set(value);
+    };
+
+    int getAsB() const {
+      if (!m_value.child("boolean"))
+        throw InvalidRPCNode("Type Error");
+      return m_value.child("boolean").text().as_bool();
+    }
+
+    void set(const std::string& value) {
+      if (!m_value.child("string"))
+        m_value.append_child("string");
+      m_value.child("string").text().set(value.c_str());
+    };
+
+    std::string getAsS() const {
+      if (!m_value.child("string"))
+        throw InvalidRPCNode("Type Error");
+      return m_value.child("string").text().as_string();
+    }
+
+    void set(double value) {
+      if (!m_value.child("double"))
+        m_value.append_child("double");
+      m_value.child("double").text().set(value);
+    };
+
+    double getAsD() const {
+      if (!m_value.child("double"))
+        throw InvalidRPCNode("Type Error");
+      return m_value.child("double").text().as_double();
+    }
+
+    inline Struct getStruct();
+    inline Array getArray();
+};
+
+class Array {
+    pugi::xml_node m_array;
+  public:
+    Array(pugi::xml_node array) : m_array(array) {
+      if (!m_array.child("data"))
+        m_array.append_child("data");
+    }
+
+    Value addValue() {
+      auto value = m_array.child("data").append_child("value");
+      return Value(value);
+    }
+
+    Value getValue(int index) const {
+      auto value = m_array.child("data").child("value");
+      while(index && value) {
+        value = value.next_sibling();
+        index--;
+      }
+      if (0==index) {
+        return Value(value);
+      } else {
+        throw InvalidRPCNode("Index error");
+      }
+    }
+
+};
+
+class Member {
+    pugi::xml_node m_member;
+
+  public:
+    Member(pugi::xml_node member) : m_member(member) { }
+
+    Value getValue() const {
+      return Value(m_member.child("value"));
+    };
+};
+
+class Struct {
+    pugi::xml_node m_struct;
+
+  public:
+    Struct(pugi::xml_node _struct) : m_struct(_struct) { }
+
+    Member getMember(const std::string& name) const {
+      for(auto member=m_struct.first_child(); member; member=member.next_sibling()) {
+        std::string member_name = member.child("name").text().get();
+        if (member_name == name) {
+          return Member(member);
+        }
+      }
+      throw InvalidRPCNode("Key Error");
+   }
+
+   Member addMember(const std::string& name) {
+     auto member = m_struct.append_child("member");
+     member.append_child("name").text().set(name.c_str());
+     member.append_child("value");
+     return Member(member);
+   }
+};
+
+class Fault : public Struct {
+  public:
+    Fault(pugi::xml_node fault) : Struct(fault) {};
+
+    int getFaultCode() const {
+      return getMember("faultCode").getValue().getAsI();
+    }
+
+    std::string getFaultString() const {
+      return getMember("faultString").getValue().getAsS();
+    }
+};
+
+Struct Value::getStruct() {
+  if (!m_value.child("struct"))
+    m_value.append_child("struct");
+  return Struct(m_value.child("struct"));
+}
+
+Array Value::getArray() {
+  if (!m_value.child("array"))
+    m_value.append_child("array");
+  return Array(m_value.child("array"));
+}
+
+class Param {
+    pugi::xml_node m_param;
+  public:
+    Param(pugi::xml_node param) : m_param(param) {
+      if (!m_param.child("value"))
+        m_param.append_child("value");
+    };
+
+    Value getValue() const {
+      return Value(m_param.child("value"));
+    };
+};
+
+class Params {
+    pugi::xml_node m_params;
+
+  public:
+    Params(pugi::xml_node params) : m_params(params) {};
+
+    Param addParam() {
+      auto param = m_params.append_child("param");
+      return Param(param);
+    }
+    Param getParam(int index) const {
+      auto param = m_params.child("param");
+      while(index && param) {
+        param = param.next_sibling();
+        index--;
+      }
+      if (0==index) {
+        return Param(param);
+      } else {
+        throw InvalidRPCNode("Index Error");
+      }
+    }
+};
+
+class MethodCall {
+    pugi::xml_document m_doc;
+
+  public:
+    MethodCall(const std::string& methodName, const std::string& secret) {
+      auto mCall = m_doc.append_child("methodCall");
+      mCall.append_child("methodName").text().set(methodName.c_str());
+      mCall.append_child("params");
+      if (!secret.empty()) {
+        getParams().addParam().getValue().set(secret);
+      }
+    }
+
+    Params getParams() const {
+      return Params(m_doc.child("methodCall").child("params"));
+    }
+
+    Value newParamValue() {
+      return getParams().addParam().getValue();
+    }
+
+    std::string toString() const {
+      return nodeToString(m_doc);
+    }
+};
+
+
+class MethodResponse {
+    pugi::xml_document m_doc;
+
+  public:
+    MethodResponse(const std::string& content) {
+      m_doc.load_buffer(content.c_str(), content.size());
+    }
+
+    Params getParams() const {
+      auto params = m_doc.child("methodResponse").child("params");
+      if (!params)
+        throw InvalidRPCNode("No params");
+      return Params(params);
+    }
+
+    Value getParamValue(int index) const {
+      return getParams().getParam(index).getValue();
+    }
+
+    bool isFault() const {
+      return (!!m_doc.child("methodResponse").child("fault"));
+    }
+
+    Fault getFault() const {
+      auto fault = m_doc.child("methodResponse").child("fault");
+      if (!fault)
+        throw InvalidRPCNode("No fault");
+      return Fault(fault.child("value").child("struct"));
+    }
+};
+
+
+
+};
+
+
+#endif // KIWIX_XMLRPC_H_

static/meson.build

@@ -1,7 +1,24 @@
 
 
-lib_resources = custom_target('resources',
-  input: 'resources_list.txt',
-  output: ['kiwixlib-resources.cpp', 'kiwixlib-resources.h'],
-  command:[res_compiler, '--cxxfile', '@OUTPUT0@', '--hfile', '@OUTPUT1@', '@INPUT@']
-)
+ctpp2c = find_program('ctpp2c', required:false)
+
+if ctpp2c.found()
+  search_result_template = custom_target('result_template',
+    input: 'results.tmpl',
+    output: 'results.ct2',
+    command: [intermediate_ctpp2c, ctpp2c, '@INPUT@', '@OUTPUT@']
+  )
+  resources_list = 'resources_list_ctpp2.txt'
+  lib_resources = custom_target('resources',
+    input: 'resources_list.txt',
+    output: ['kiwixlib-resources.cpp', 'kiwixlib-resources.h'],
+    command:[res_compiler,
+             '--cxxfile', '@OUTPUT0@',
+             '--hfile', '@OUTPUT1@',
+             '--source_dir', '@OUTDIR@',
+             '@INPUT@'],
+    depends: [search_result_template]
+  )
+else
+ lib_resources = []
+endif

static/resources_list.txt

@@ -1,4 +1 @@
 results.ct2
-stopwords/en
-stopwords/he
-stopwords/fra

static/results.tmpl

@@ -0,0 +1,159 @@
+<!DOCTYPE html>
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <meta content="text/html; charset=utf-8" http-equiv="content-type" />
+    <style type="text/css">
+      body{
+      color: #00000;
+      font: small/normal Arial,Helvetica,Sans-Serif;
+      margin-top: 0.5em;
+      font-size: 90%;
+      }
+
+      a{
+      color: #04c;
+      }
+
+      a:visited {
+      color: #639
+      }
+
+      a:hover {
+      text-decoration: underline
+      }
+
+      .header {
+      font-size: 120%;
+      }
+
+      ul {
+      margin:0;
+      padding:0
+      }
+
+      .results {
+      font-size: 110%;
+      }
+
+      .results li {
+      list-style-type:none;
+      margin-top: 0.5em;
+      }
+
+      .results a {
+      font-size: 110%;
+      text-decoration: underline
+      }
+
+      cite {
+      font-style:normal;
+      word-wrap:break-word;
+      display: block;
+      font-size: 100%;
+      }
+
+      .informations {
+      color: #388222;
+      font-size: 100%;
+      }
+
+      .footer {
+      padding: 0;
+      margin-top: 1em;
+      width: 100%;
+      float: left
+      }
+
+      .footer a, .footer span {
+      display: block;
+      padding: .3em .7em;
+      margin: 0 .38em 0 0;
+      text-align:center;
+      text-decoration: none;
+      }
+
+      .footer a:hover {
+      background: #ededed;
+      }
+
+      .footer ul, .footer li {
+      list-style:none;
+      margin: 0;
+      padding: 0;
+      }
+
+      .footer li {
+      float: left;
+      }
+
+      .selected {
+      background: #ededed;
+      }
+
+    </style>
+    <title>Search: <TMPL_var searchPattern></title>
+  </head>
+  <body bgcolor="white">
+    <div class="header">
+      <TMPL_if results>
+        Results
+        <b>
+          <TMPL_var resultStart>-<TMPL_var resultEnd>
+        </b> of <b>
+          <TMPL_var count>
+        </b> for <b>
+          <TMPL_var searchPattern>
+        </b>
+      <TMPL_else>
+        No results were found for <b><TMPL_var searchPattern></b>
+      </TMPL_if>
+    </div>
+
+    <div class="results">
+      <ul>
+        <TMPL_foreach results as result>
+          <li>
+            <a href="<TMPL_var protocolPrefix><TMPL_var result.contentId>/<TMPL_var result.url>">
+              <TMPL_var result.title>
+            </a>
+            <cite>
+              <TMPL_if result.snippet>
+                <TMPL_var result.snippet>...
+              </TMPL_if>
+            </cite>
+            <TMPL_if wordCount>
+              <div class="informations"><TMPL_var wordCount> words</div>
+            </TMPL_if>
+          </li>
+        </TMPL_foreach>
+      </ul>
+    </div>
+
+    <div class="footer">
+      <ul>
+        <TMPL_if (resultLastPageStart>0)>
+          <li>
+            <a href="<TMPL_var searchProtocolPrefix>pattern=<TMPL_var searchPatternEncoded><TMPL_if contentId>&content=<TMPL_var contentId></TMPL_if>&start=0&end=<TMPL_var resultRange>">
+              ◀
+            </a>
+          </li>
+        </TMPL_if>
+        <TMPL_foreach pages as page>
+          <li>
+            <a <TMPL_if page.selected>class="selected"</TMPL_if>
+               href="<TMPL_var searchProtocolPrefix>pattern=<TMPL_var searchPatternEncoded><TMPL_if contentId>&content=<TMPL_var contentId></TMPL_if>&start=<TMPL_var page.start>&end=<TMPL_var page.end>">
+              <TMPL_var page.label>
+            </a>
+          </li>
+        </TMPL_foreach>
+        <TMPL_if (resultLastPageStart>0)>
+          <li>
+            <a href="<TMPL_var searchProtocolPrefix>pattern=<TMPL_var searchPatternEncoded><TMPL_if contentId>&content=<TMPL_var contentId></TMPL_if>&start=<TMPL_var resultLastPageStart>&end=<TMPL_var (resultLastPageStart+resultRange)>">
+              ▶
+            </a>
+          </li>
+        </TMPL_if>
+      </ul>
+    </div>
+  </body>
+</html>

static/stopwords/en

@@ -1,671 +0,0 @@
-a
-able
-about
-above
-abst
-accordance
-according
-accordingly
-across
-act
-actually
-added
-adj
-adopted
-affected
-affecting
-affects
-after
-afterwards
-again
-against
-ah
-all
-almost
-alone
-along
-already
-also
-although
-always
-am
-among
-amongst
-an
-and
-announce
-another
-any
-anybody
-anyhow
-anymore
-anyone
-anything
-anyway
-anyways
-anywhere
-apparently
-approximately
-are
-aren
-arent
-arise
-around
-as
-aside
-ask
-asking
-at
-auth
-available
-away
-awfully
-b
-back
-be
-became
-because
-become
-becomes
-becoming
-been
-before
-beforehand
-begin
-beginning
-beginnings
-begins
-behind
-being
-believe
-below
-beside
-besides
-between
-beyond
-biol
-both
-brief
-briefly
-but
-by
-c
-ca
-came
-can
-cannot
-can't
-cause
-causes
-certain
-certainly
-co
-com
-come
-comes
-contain
-containing
-contains
-could
-couldnt
-d
-date
-did
-didn't
-different
-do
-does
-doesn't
-doing
-done
-don't
-down
-downwards
-due
-during
-e
-each
-ed
-edu
-effect
-eg
-eight
-eighty
-either
-else
-elsewhere
-end
-ending
-enough
-especially
-et
-et-al
-etc
-even
-ever
-every
-everybody
-everyone
-everything
-everywhere
-ex
-except
-f
-far
-few
-ff
-fifth
-first
-five
-fix
-followed
-following
-follows
-for
-former
-formerly
-forth
-found
-four
-from
-further
-furthermore
-g
-gave
-get
-gets
-getting
-give
-given
-gives
-giving
-go
-goes
-gone
-got
-gotten
-h
-had
-happens
-hardly
-has
-hasn't
-have
-haven't
-having
-he
-hed
-hence
-her
-here
-hereafter
-hereby
-herein
-heres
-hereupon
-hers
-herself
-hes
-hi
-hid
-him
-himself
-his
-hither
-home
-how
-howbeit
-however
-hundred
-i
-id
-ie
-if
-i'll
-im
-immediate
-immediately
-importance
-important
-in
-inc
-indeed
-index
-information
-instead
-into
-invention
-inward
-is
-isn't
-it
-itd
-it'll
-its
-itself
-i've
-j
-just
-k
-keep
-keeps
-kept
-keys
-kg
-km
-know
-known
-knows
-l
-largely
-last
-lately
-later
-latter
-latterly
-least
-less
-lest
-let
-lets
-like
-liked
-likely
-line
-little
-'ll
-look
-looking
-looks
-ltd
-m
-made
-mainly
-make
-makes
-many
-may
-maybe
-me
-mean
-means
-meantime
-meanwhile
-merely
-mg
-might
-million
-miss
-ml
-more
-moreover
-most
-mostly
-mr
-mrs
-much
-mug
-must
-my
-myself
-n
-na
-name
-namely
-nay
-nd
-near
-nearly
-necessarily
-necessary
-need
-needs
-neither
-never
-nevertheless
-new
-next
-nine
-ninety
-no
-nobody
-non
-none
-nonetheless
-noone
-nor
-normally
-nos
-not
-noted
-nothing
-now
-nowhere
-o
-obtain
-obtained
-obviously
-of
-off
-often
-oh
-ok
-okay
-old
-omitted
-on
-once
-one
-ones
-only
-onto
-or
-ord
-other
-others
-otherwise
-ought
-our
-ours
-ourselves
-out
-outside
-over
-overall
-owing
-own
-p
-page
-pages
-part
-particular
-particularly
-past
-per
-perhaps
-placed
-please
-plus
-poorly
-possible
-possibly
-potentially
-pp
-predominantly
-present
-previously
-primarily
-probably
-promptly
-proud
-provides
-put
-q
-que
-quickly
-quite
-qv
-r
-ran
-rather
-rd
-re
-readily
-really
-recent
-recently
-ref
-refs
-regarding
-regardless
-regards
-related
-relatively
-research
-respectively
-resulted
-resulting
-results
-right
-run
-s
-said
-same
-saw
-say
-saying
-says
-sec
-section
-see
-seeing
-seem
-seemed
-seeming
-seems
-seen
-self
-selves
-sent
-seven
-several
-shall
-she
-shed
-she'll
-shes
-should
-shouldn't
-show
-showed
-shown
-showns
-shows
-significant
-significantly
-similar
-similarly
-since
-six
-slightly
-so
-some
-somebody
-somehow
-someone
-somethan
-something
-sometime
-sometimes
-somewhat
-somewhere
-soon
-sorry
-specifically
-specified
-specify
-specifying
-state
-states
-still
-stop
-strongly
-sub
-substantially
-successfully
-such
-sufficiently
-suggest
-sup
-sure
-	t
-take
-taken
-taking
-tell
-tends
-th
-than
-thank
-thanks
-thanx
-that
-that'll
-thats
-that've
-the
-their
-theirs
-them
-themselves
-then
-thence
-there
-thereafter
-thereby
-thered
-therefore
-therein
-there'll
-thereof
-therere
-theres
-thereto
-thereupon
-there've
-these
-they
-theyd
-they'll
-theyre
-they've
-think
-this
-those
-thou
-though
-thoughh
-thousand
-throug
-through
-throughout
-thru
-thus
-til
-tip
-to
-together
-too
-took
-toward
-towards
-tried
-tries
-truly
-try
-trying
-ts
-twice
-two
-u
-un
-under
-unfortunately
-unless
-unlike
-unlikely
-until
-unto
-up
-upon
-ups
-us
-use
-used
-useful
-usefully
-usefulness
-uses
-using
-usually
-v
-value
-various
-'ve
-very
-via
-viz
-vol
-vols
-vs
-w
-want
-wants
-was
-wasn't
-way
-we
-wed
-welcome
-we'll
-went
-were
-weren't
-we've
-what
-whatever
-what'll
-whats
-when
-whence
-whenever
-where
-whereafter
-whereas
-whereby
-wherein
-wheres
-whereupon
-wherever
-whether
-which
-while
-whim
-whither
-who
-whod
-whoever
-whole
-who'll
-whom
-whomever
-whos
-whose
-why
-widely
-willing
-wish
-with
-within
-without
-won't
-words
-world
-would
-wouldn't
-www
-x
-y
-yes
-yet
-you
-youd
-you'll
-your
-youre
-yours
-yourself
-yourselves
-you've
-z
-zero

static/stopwords/fra

@@ -1,124 +0,0 @@
-alors
-au
-aucuns
-aussi
-autre
-avant
-avec
-avoir
-bon
-car
-ce
-cela
-ces
-ceux
-chaque
-ci
-comme
-comment
-dans
-des
-du
-dedans
-dehors
-depuis
-deux
-devrait
-doit
-donc
-dos
-droite
-début
-elle
-elles
-en
-encore
-essai
-est
-et
-eu
-fait
-faites
-fois
-font
-force
-haut
-hors
-ici
-il
-ils
-je
-la
-le
-les
-leur
-là
-ma
-maintenant
-mais
-mes
-mine
-moins
-mon
-mot
-même
-ni
-nommés
-notre
-nous
-nouveaux
-ou
-où
-par
-parce
-parole
-pas
-personnes
-peut
-peu
-pièce
-plupart
-pour
-pourquoi
-quand
-que
-quel
-quelle
-quelles
-quels
-qui
-sa
-sans
-ses
-seulement
-si
-sien
-son
-sont
-sous
-soyez
-sur
-ta
-tandis
-tellement
-tels
-tes
-ton
-tous
-tout
-trop
-très
-tu
-valeur
-voie
-voient
-vont
-votre
-vous
-vu
-ça
-étaient
-état
-étions
-été
-être

static/stopwords/he

@@ -1,87 +0,0 @@
-של
-את
-על
-לא
-כי
-עם
-הוא
-גם
-ב
-זה
-היא
-כל
-יותר
-או
-אבל
-בין
-היה
-אם
-מיליון
-יש
-כך
-אני
-הם
-דולר
-אמר
-עד
-לאחר
-ישראל
-רק
-שקל
-כדי
-מה
-לפני
-אחד
-החברה
-כמו
-זאת
-היום
-אך
-ל
-ה
-כ
-אין
-אתמול
-שלא
-כבר
-עוד
-לו
-זו
-אל
-בן
-אותו
-שני
-בית
-ידי
-כמה
-ביותר
-ולא
-הממשלה
-אחרי
-חברת
-היתה
-שלו
-היו
-נגד
-בכל
-אביב
-ראש
-בישראל
-לי
-שנים
-פי
-בו
-מ
-מאוד
-להיות
-שהוא
-מי
-אלף
-אלא
-אף
-אחר
-הזה
-אחת
-בבית
-אלה
-אנחנו

subprojects/gtest/.gitignore

@@ -0,0 +1,2 @@
+# Ignore CI build directory
+build/

subprojects/gtest/.travis.yml

@@ -0,0 +1,46 @@
+# Build matrix / environment variable are explained on:
+# http://about.travis-ci.org/docs/user/build-configuration/
+# This file can be validated on:
+# http://lint.travis-ci.org/
+
+install:
+# /usr/bin/gcc is 4.6 always, but gcc-X.Y is available.
+- if [ "$CXX" = "g++" ]; then export CXX="g++-4.9" CC="gcc-4.9"; fi
+# /usr/bin/clang is 3.4, lets override with modern one.
+- if [ "$CXX" = "clang++" ] && [ "$TRAVIS_OS_NAME" = "linux" ]; then export CXX="clang++-3.7" CC="clang-3.7"; fi
+- echo ${PATH}
+- echo ${CXX}
+- ${CXX} --version
+- ${CXX} -v
+addons:
+  apt:
+    # List of whitelisted in travis packages for ubuntu-precise can be found here:
+    #   https://github.com/travis-ci/apt-package-whitelist/blob/master/ubuntu-precise
+    # List of whitelisted in travis apt-sources:
+    #   https://github.com/travis-ci/apt-source-whitelist/blob/master/ubuntu.json
+    sources:
+    - ubuntu-toolchain-r-test
+    - llvm-toolchain-precise-3.7
+    packages:
+    - gcc-4.9
+    - g++-4.9
+    - clang-3.7
+    - valgrind
+os:
+  - linux
+  - osx
+language: cpp
+compiler:
+  - gcc
+  - clang
+script: ./travis.sh
+env:
+  matrix:
+    - GTEST_TARGET=googletest SHARED_LIB=OFF STATIC_LIB=ON CMAKE_PKG=OFF BUILD_TYPE=debug   VERBOSE_MAKE=true VERBOSE
+    - GTEST_TARGET=googlemock SHARED_LIB=OFF STATIC_LIB=ON CMAKE_PKG=OFF BUILD_TYPE=debug   VERBOSE_MAKE=true VERBOSE
+    - GTEST_TARGET=googlemock SHARED_LIB=OFF STATIC_LIB=ON CMAKE_PKG=OFF BUILD_TYPE=debug CXX_FLAGS=-std=c++11  VERBOSE_MAKE=true VERBOSE
+#    - GTEST_TARGET=googletest SHARED_LIB=ON  STATIC_LIB=ON CMAKE_PKG=ON  BUILD_TYPE=release VERBOSE_MAKE=false
+#    - GTEST_TARGET=googlemock SHARED_LIB=ON  STATIC_LIB=ON CMAKE_PKG=ON  BUILD_TYPE=release VERBOSE_MAKE=false
+notifications:
+  email: false
+sudo: false

subprojects/gtest/CMakeLists.txt

@@ -0,0 +1,16 @@
+cmake_minimum_required(VERSION 2.6.2)
+
+project( googletest-distribution )
+
+enable_testing()
+
+option(BUILD_GTEST "Builds the googletest subproject" OFF)
+
+#Note that googlemock target already builds googletest
+option(BUILD_GMOCK "Builds the googlemock subproject" ON)
+
+if(BUILD_GMOCK)
+  add_subdirectory( googlemock )
+elseif(BUILD_GTEST)
+  add_subdirectory( googletest )
+endif()

subprojects/gtest/README.md

@@ -0,0 +1,142 @@
+
+# Google Test #
+
+[![Build Status](https://travis-ci.org/google/googletest.svg?branch=master)](https://travis-ci.org/google/googletest)
+[![Build status](https://ci.appveyor.com/api/projects/status/4o38plt0xbo1ubc8/branch/master?svg=true)](https://ci.appveyor.com/project/BillyDonahue/googletest/branch/master)
+
+Welcome to **Google Test**, Google's C++ test framework!
+
+This repository is a merger of the formerly separate GoogleTest and
+GoogleMock projects. These were so closely related that it makes sense to
+maintain and release them together.
+
+Please see the project page above for more information as well as the
+mailing list for questions, discussions, and development.  There is
+also an IRC channel on OFTC (irc.oftc.net) #gtest available.  Please
+join us!
+
+Getting started information for **Google Test** is available in the 
+[Google Test Primer](googletest/docs/Primer.md) documentation.
+
+**Google Mock** is an extension to Google Test for writing and using C++ mock
+classes.  See the separate [Google Mock documentation](googlemock/README.md).
+
+More detailed documentation for googletest (including build instructions) are
+in its interior [googletest/README.md](googletest/README.md) file.
+
+## Features ##
+
+  * An [XUnit](https://en.wikipedia.org/wiki/XUnit) test framework.
+  * Test discovery.
+  * A rich set of assertions.
+  * User-defined assertions.
+  * Death tests.
+  * Fatal and non-fatal failures.
+  * Value-parameterized tests.
+  * Type-parameterized tests.
+  * Various options for running the tests.
+  * XML test report generation.
+
+## Platforms ##
+
+Google test has been used on a variety of platforms:
+
+  * Linux
+  * Mac OS X
+  * Windows
+  * Cygwin
+  * MinGW
+  * Windows Mobile
+  * Symbian
+
+## Who Is Using Google Test? ##
+
+In addition to many internal projects at Google, Google Test is also used by
+the following notable projects:
+
+  * The [Chromium projects](http://www.chromium.org/) (behind the Chrome
+    browser and Chrome OS).
+  * The [LLVM](http://llvm.org/) compiler.
+  * [Protocol Buffers](https://github.com/google/protobuf), Google's data
+    interchange format.
+  * The [OpenCV](http://opencv.org/) computer vision library.
+
+## Related Open Source Projects ##
+
+[Google Test UI](https://github.com/ospector/gtest-gbar) is test runner that runs
+your test binary, allows you to track its progress via a progress bar, and
+displays a list of test failures. Clicking on one shows failure text. Google
+Test UI is written in C#.
+
+[GTest TAP Listener](https://github.com/kinow/gtest-tap-listener) is an event
+listener for Google Test that implements the
+[TAP protocol](https://en.wikipedia.org/wiki/Test_Anything_Protocol) for test
+result output. If your test runner understands TAP, you may find it useful.
+
+## Requirements ##
+
+Google Test is designed to have fairly minimal requirements to build
+and use with your projects, but there are some.  Currently, we support
+Linux, Windows, Mac OS X, and Cygwin.  We will also make our best
+effort to support other platforms (e.g. Solaris, AIX, and z/OS).
+However, since core members of the Google Test project have no access
+to these platforms, Google Test may have outstanding issues there.  If
+you notice any problems on your platform, please notify
+<googletestframework@googlegroups.com>. Patches for fixing them are
+even more welcome!
+
+### Linux Requirements ###
+
+These are the base requirements to build and use Google Test from a source
+package (as described below):
+
+  * GNU-compatible Make or gmake
+  * POSIX-standard shell
+  * POSIX(-2) Regular Expressions (regex.h)
+  * A C++98-standard-compliant compiler
+
+### Windows Requirements ###
+
+  * Microsoft Visual C++ v7.1 or newer
+
+### Cygwin Requirements ###
+
+  * Cygwin v1.5.25-14 or newer
+
+### Mac OS X Requirements ###
+
+  * Mac OS X v10.4 Tiger or newer
+  * Xcode Developer Tools
+
+### Requirements for Contributors ###
+
+We welcome patches.  If you plan to contribute a patch, you need to
+build Google Test and its own tests from a git checkout (described
+below), which has further requirements:
+
+  * [Python](https://www.python.org/) v2.3 or newer (for running some of
+    the tests and re-generating certain source files from templates)
+  * [CMake](https://cmake.org/) v2.6.4 or newer
+
+## Regenerating Source Files ##
+
+Some of Google Test's source files are generated from templates (not
+in the C++ sense) using a script.
+For example, the
+file include/gtest/internal/gtest-type-util.h.pump is used to generate
+gtest-type-util.h in the same directory.
+
+You don't need to worry about regenerating the source files
+unless you need to modify them.  You would then modify the
+corresponding `.pump` files and run the '[pump.py](googletest/scripts/pump.py)'
+generator script.  See the [Pump Manual](googletest/docs/PumpManual.md).
+
+### Contributing Code ###
+
+We welcome patches.  Please read the
+[Developer's Guide](googletest/docs/DevGuide.md)
+for how you can contribute. In particular, make sure you have signed
+the Contributor License Agreement, or we won't be able to accept the
+patch.
+
+Happy testing!

subprojects/gtest/appveyor.yml

@@ -0,0 +1,71 @@
+version: '{build}'
+
+os: Visual Studio 2015
+
+environment:
+  matrix:
+  - Toolset: v140
+  - Toolset: v120
+  - Toolset: v110
+  - Toolset: v100
+
+platform:
+  - Win32
+  - x64
+
+configuration:
+#  - Release
+  - Debug
+
+build:
+  verbosity: minimal
+
+artifacts:
+  - path: '_build/Testing/Temporary/*'
+    name: test_results
+
+before_build:
+- ps: |
+    Write-Output "Configuration: $env:CONFIGURATION"
+    Write-Output "Platform: $env:PLATFORM"
+    $generator = switch ($env:TOOLSET)
+    {
+        "v140" {"Visual Studio 14 2015"}
+        "v120" {"Visual Studio 12 2013"}
+        "v110" {"Visual Studio 11 2012"}
+        "v100" {"Visual Studio 10 2010"}
+    }
+    if ($env:PLATFORM -eq "x64")
+    {
+        $generator = "$generator Win64"
+    }
+
+build_script:
+- ps: |
+    if (($env:TOOLSET -eq "v100") -and ($env:PLATFORM -eq "x64"))
+    {
+        return
+    }
+    md _build -Force | Out-Null
+    cd _build
+
+    & cmake -G "$generator" -DCMAKE_CONFIGURATION_TYPES="Debug;Release" -Dgtest_build_tests=ON -Dgtest_build_samples=ON -Dgmock_build_tests=ON ..
+    if ($LastExitCode -ne 0) {
+        throw "Exec: $ErrorMessage"
+    }
+    & cmake --build . --config $env:CONFIGURATION
+    if ($LastExitCode -ne 0) {
+        throw "Exec: $ErrorMessage"
+    }
+
+test_script:
+- ps: |
+    if (($env:Toolset -eq "v100") -and ($env:PLATFORM -eq "x64"))
+    {
+        return
+    }
+
+    & ctest -C $env:CONFIGURATION --output-on-failure
+    if ($LastExitCode -ne 0) {
+        throw "Exec: $ErrorMessage"
+    }

subprojects/gtest/googlemock/CHANGES

@@ -0,0 +1,126 @@
+Changes for 1.7.0:
+
+* All new improvements in Google Test 1.7.0.
+* New feature: matchers DoubleNear(), FloatNear(),
+  NanSensitiveDoubleNear(), NanSensitiveFloatNear(),
+  UnorderedElementsAre(), UnorderedElementsAreArray(), WhenSorted(),
+  WhenSortedBy(), IsEmpty(), and SizeIs().
+* Improvement: Google Mock can now be built as a DLL.
+* Improvement: when compiled by a C++11 compiler, matchers AllOf()
+  and AnyOf() can accept an arbitrary number of matchers.
+* Improvement: when compiled by a C++11 compiler, matchers
+  ElementsAreArray() can accept an initializer list.
+* Improvement: when exceptions are enabled, a mock method with no
+  default action now throws instead crashing the test.
+* Improvement: added class testing::StringMatchResultListener to aid
+  definition of composite matchers.
+* Improvement: function return types used in MOCK_METHOD*() macros can
+  now contain unprotected commas.
+* Improvement (potentially breaking): EXPECT_THAT() and ASSERT_THAT()
+  are now more strict in ensuring that the value type and the matcher
+  type are compatible, catching potential bugs in tests.
+* Improvement: Pointee() now works on an optional<T>.
+* Improvement: the ElementsAreArray() matcher can now take a vector or
+  iterator range as input, and makes a copy of its input elements
+  before the conversion to a Matcher.
+* Improvement: the Google Mock Generator can now generate mocks for
+  some class templates.
+* Bug fix: mock object destruction triggerred by another mock object's
+  destruction no longer hangs.
+* Improvement: Google Mock Doctor works better with newer Clang and
+  GCC now.
+* Compatibility fixes.
+* Bug/warning fixes.
+
+Changes for 1.6.0:
+
+* Compilation is much faster and uses much less memory, especially
+  when the constructor and destructor of a mock class are moved out of
+  the class body.
+* New matchers: Pointwise(), Each().
+* New actions: ReturnPointee() and ReturnRefOfCopy().
+* CMake support.
+* Project files for Visual Studio 2010.
+* AllOf() and AnyOf() can handle up-to 10 arguments now.
+* Google Mock doctor understands Clang error messages now.
+* SetArgPointee<> now accepts string literals.
+* gmock_gen.py handles storage specifier macros and template return
+  types now.
+* Compatibility fixes.
+* Bug fixes and implementation clean-ups.
+* Potentially incompatible changes: disables the harmful 'make install'
+  command in autotools.
+
+Potentially breaking changes:
+
+* The description string for MATCHER*() changes from Python-style
+  interpolation to an ordinary C++ string expression.
+* SetArgumentPointee is deprecated in favor of SetArgPointee.
+* Some non-essential project files for Visual Studio 2005 are removed.
+
+Changes for 1.5.0:
+
+ * New feature: Google Mock can be safely used in multi-threaded tests
+   on platforms having pthreads.
+ * New feature: function for printing a value of arbitrary type.
+ * New feature: function ExplainMatchResult() for easy definition of
+   composite matchers.
+ * The new matcher API lets user-defined matchers generate custom
+   explanations more directly and efficiently.
+ * Better failure messages all around.
+ * NotNull() and IsNull() now work with smart pointers.
+ * Field() and Property() now work when the matcher argument is a pointer
+   passed by reference.
+ * Regular expression matchers on all platforms.
+ * Added GCC 4.0 support for Google Mock Doctor.
+ * Added gmock_all_test.cc for compiling most Google Mock tests
+   in a single file.
+ * Significantly cleaned up compiler warnings.
+ * Bug fixes, better test coverage, and implementation clean-ups.
+
+ Potentially breaking changes:
+
+ * Custom matchers defined using MatcherInterface or MakePolymorphicMatcher()
+   need to be updated after upgrading to Google Mock 1.5.0; matchers defined
+   using MATCHER or MATCHER_P* aren't affected.
+ * Dropped support for 'make install'.
+
+Changes for 1.4.0 (we skipped 1.2.* and 1.3.* to match the version of
+Google Test):
+
+ * Works in more environments: Symbian and minGW, Visual C++ 7.1.
+ * Lighter weight: comes with our own implementation of TR1 tuple (no
+   more dependency on Boost!).
+ * New feature: --gmock_catch_leaked_mocks for detecting leaked mocks.
+ * New feature: ACTION_TEMPLATE for defining templatized actions.
+ * New feature: the .After() clause for specifying expectation order.
+ * New feature: the .With() clause for for specifying inter-argument
+   constraints.
+ * New feature: actions ReturnArg<k>(), ReturnNew<T>(...), and
+   DeleteArg<k>().
+ * New feature: matchers Key(), Pair(), Args<...>(), AllArgs(), IsNull(),
+   and Contains().
+ * New feature: utility class MockFunction<F>, useful for checkpoints, etc.
+ * New feature: functions Value(x, m) and SafeMatcherCast<T>(m).
+ * New feature: copying a mock object is rejected at compile time.
+ * New feature: a script for fusing all Google Mock and Google Test
+   source files for easy deployment.
+ * Improved the Google Mock doctor to diagnose more diseases.
+ * Improved the Google Mock generator script.
+ * Compatibility fixes for Mac OS X and gcc.
+ * Bug fixes and implementation clean-ups.
+
+Changes for 1.1.0:
+
+ * New feature: ability to use Google Mock with any testing framework.
+ * New feature: macros for easily defining new matchers
+ * New feature: macros for easily defining new actions.
+ * New feature: more container matchers.
+ * New feature: actions for accessing function arguments and throwing
+   exceptions.
+ * Improved the Google Mock doctor script for diagnosing compiler errors.
+ * Bug fixes and implementation clean-ups.
+
+Changes for 1.0.0:
+
+ * Initial Open Source release of Google Mock

subprojects/gtest/googlemock/CMakeLists.txt

@@ -0,0 +1,202 @@
+########################################################################
+# CMake build script for Google Mock.
+#
+# To run the tests for Google Mock itself on Linux, use 'make test' or
+# ctest.  You can select which tests to run using 'ctest -R regex'.
+# For more options, run 'ctest --help'.
+
+# BUILD_SHARED_LIBS is a standard CMake variable, but we declare it here to
+# make it prominent in the GUI.
+option(BUILD_SHARED_LIBS "Build shared libraries (DLLs)." OFF)
+
+option(gmock_build_tests "Build all of Google Mock's own tests." OFF)
+
+# A directory to find Google Test sources.
+if (EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/gtest/CMakeLists.txt")
+  set(gtest_dir gtest)
+else()
+  set(gtest_dir ../googletest)
+endif()
+
+# Defines pre_project_set_up_hermetic_build() and set_up_hermetic_build().
+include("${gtest_dir}/cmake/hermetic_build.cmake" OPTIONAL)
+
+if (COMMAND pre_project_set_up_hermetic_build)
+  # Google Test also calls hermetic setup functions from add_subdirectory,
+  # although its changes will not affect things at the current scope.
+  pre_project_set_up_hermetic_build()
+endif()
+
+########################################################################
+#
+# Project-wide settings
+
+# Name of the project.
+#
+# CMake files in this project can refer to the root source directory
+# as ${gmock_SOURCE_DIR} and to the root binary directory as
+# ${gmock_BINARY_DIR}.
+# Language "C" is required for find_package(Threads).
+project(gmock CXX C)
+cmake_minimum_required(VERSION 2.6.2)
+
+if (COMMAND set_up_hermetic_build)
+  set_up_hermetic_build()
+endif()
+
+# Instructs CMake to process Google Test's CMakeLists.txt and add its
+# targets to the current scope.  We are placing Google Test's binary
+# directory in a subdirectory of our own as VC compilation may break
+# if they are the same (the default).
+add_subdirectory("${gtest_dir}" "${gmock_BINARY_DIR}/gtest")
+
+# Although Google Test's CMakeLists.txt calls this function, the
+# changes there don't affect the current scope.  Therefore we have to
+# call it again here.
+config_compiler_and_linker()  # from ${gtest_dir}/cmake/internal_utils.cmake
+
+# Adds Google Mock's and Google Test's header directories to the search path.
+include_directories("${gmock_SOURCE_DIR}/include"
+                    "${gmock_SOURCE_DIR}"
+                    "${gtest_SOURCE_DIR}/include"
+                    # This directory is needed to build directly from Google
+                    # Test sources.
+                    "${gtest_SOURCE_DIR}")
+
+# Summary of tuple support for Microsoft Visual Studio:
+# Compiler    version(MS)  version(cmake)  Support
+# ----------  -----------  --------------  -----------------------------
+# <= VS 2010  <= 10        <= 1600         Use Google Tests's own tuple.
+# VS 2012     11           1700            std::tr1::tuple + _VARIADIC_MAX=10
+# VS 2013     12           1800            std::tr1::tuple
+if (MSVC AND MSVC_VERSION EQUAL 1700)
+  add_definitions(/D _VARIADIC_MAX=10)
+endif()
+
+########################################################################
+#
+# Defines the gmock & gmock_main libraries.  User tests should link
+# with one of them.
+
+# Google Mock libraries.  We build them using more strict warnings than what
+# are used for other targets, to ensure that Google Mock can be compiled by
+# a user aggressive about warnings.
+cxx_library(gmock
+            "${cxx_strict}"
+            "${gtest_dir}/src/gtest-all.cc"
+            src/gmock-all.cc)
+
+cxx_library(gmock_main
+            "${cxx_strict}"
+            "${gtest_dir}/src/gtest-all.cc"
+            src/gmock-all.cc
+            src/gmock_main.cc)
+
+# If the CMake version supports it, attach header directory information
+# to the targets for when we are part of a parent build (ie being pulled
+# in via add_subdirectory() rather than being a standalone build).
+if (DEFINED CMAKE_VERSION AND NOT "${CMAKE_VERSION}" VERSION_LESS "2.8.11")
+  target_include_directories(gmock      INTERFACE "${gmock_SOURCE_DIR}/include")
+  target_include_directories(gmock_main INTERFACE "${gmock_SOURCE_DIR}/include")
+endif()
+
+########################################################################
+#
+# Install rules
+install(TARGETS gmock gmock_main
+  DESTINATION lib)
+install(DIRECTORY ${gmock_SOURCE_DIR}/include/gmock
+  DESTINATION include)
+
+########################################################################
+#
+# Google Mock's own tests.
+#
+# You can skip this section if you aren't interested in testing
+# Google Mock itself.
+#
+# The tests are not built by default.  To build them, set the
+# gmock_build_tests option to ON.  You can do it by running ccmake
+# or specifying the -Dgmock_build_tests=ON flag when running cmake.
+
+if (gmock_build_tests)
+  # This must be set in the root directory for the tests to be run by
+  # 'make test' or ctest.
+  enable_testing()
+
+  ############################################################
+  # C++ tests built with standard compiler flags.
+
+  cxx_test(gmock-actions_test gmock_main)
+  cxx_test(gmock-cardinalities_test gmock_main)
+  cxx_test(gmock_ex_test gmock_main)
+  cxx_test(gmock-generated-actions_test gmock_main)
+  cxx_test(gmock-generated-function-mockers_test gmock_main)
+  cxx_test(gmock-generated-internal-utils_test gmock_main)
+  cxx_test(gmock-generated-matchers_test gmock_main)
+  cxx_test(gmock-internal-utils_test gmock_main)
+  cxx_test(gmock-matchers_test gmock_main)
+  cxx_test(gmock-more-actions_test gmock_main)
+  cxx_test(gmock-nice-strict_test gmock_main)
+  cxx_test(gmock-port_test gmock_main)
+  cxx_test(gmock-spec-builders_test gmock_main)
+  cxx_test(gmock_link_test gmock_main test/gmock_link2_test.cc)
+  cxx_test(gmock_test gmock_main)
+
+  if (CMAKE_USE_PTHREADS_INIT)
+    cxx_test(gmock_stress_test gmock)
+  endif()
+
+  # gmock_all_test is commented to save time building and running tests.
+  # Uncomment if necessary.
+  # cxx_test(gmock_all_test gmock_main)
+
+  ############################################################
+  # C++ tests built with non-standard compiler flags.
+
+  cxx_library(gmock_main_no_exception "${cxx_no_exception}"
+    "${gtest_dir}/src/gtest-all.cc" src/gmock-all.cc src/gmock_main.cc)
+
+  cxx_library(gmock_main_no_rtti "${cxx_no_rtti}"
+    "${gtest_dir}/src/gtest-all.cc" src/gmock-all.cc src/gmock_main.cc)
+
+  if (NOT MSVC OR MSVC_VERSION LESS 1600)  # 1600 is Visual Studio 2010.
+    # Visual Studio 2010, 2012, and 2013 define symbols in std::tr1 that
+    # conflict with our own definitions. Therefore using our own tuple does not
+    # work on those compilers.
+    cxx_library(gmock_main_use_own_tuple "${cxx_use_own_tuple}"
+      "${gtest_dir}/src/gtest-all.cc" src/gmock-all.cc src/gmock_main.cc)
+
+    cxx_test_with_flags(gmock_use_own_tuple_test "${cxx_use_own_tuple}"
+      gmock_main_use_own_tuple test/gmock-spec-builders_test.cc)
+  endif()
+
+  cxx_test_with_flags(gmock-more-actions_no_exception_test "${cxx_no_exception}"
+    gmock_main_no_exception test/gmock-more-actions_test.cc)
+
+  cxx_test_with_flags(gmock_no_rtti_test "${cxx_no_rtti}"
+    gmock_main_no_rtti test/gmock-spec-builders_test.cc)
+
+  cxx_shared_library(shared_gmock_main "${cxx_default}"
+    "${gtest_dir}/src/gtest-all.cc" src/gmock-all.cc src/gmock_main.cc)
+
+  # Tests that a binary can be built with Google Mock as a shared library.  On
+  # some system configurations, it may not possible to run the binary without
+  # knowing more details about the system configurations. We do not try to run
+  # this binary. To get a more robust shared library coverage, configure with
+  # -DBUILD_SHARED_LIBS=ON.
+  cxx_executable_with_flags(shared_gmock_test_ "${cxx_default}"
+    shared_gmock_main test/gmock-spec-builders_test.cc)
+  set_target_properties(shared_gmock_test_
+    PROPERTIES
+    COMPILE_DEFINITIONS "GTEST_LINKED_AS_SHARED_LIBRARY=1")
+
+  ############################################################
+  # Python tests.
+
+  cxx_executable(gmock_leak_test_ test gmock_main)
+  py_test(gmock_leak_test)
+
+  cxx_executable(gmock_output_test_ test gmock)
+  py_test(gmock_output_test)
+endif()

subprojects/gtest/googlemock/CONTRIBUTORS

@@ -0,0 +1,40 @@
+# This file contains a list of people who've made non-trivial
+# contribution to the Google C++ Mocking Framework project.  People
+# who commit code to the project are encouraged to add their names
+# here.  Please keep the list sorted by first names.
+
+Benoit Sigoure <tsuna@google.com>
+Bogdan Piloca <boo@google.com>
+Chandler Carruth <chandlerc@google.com>
+Dave MacLachlan <dmaclach@gmail.com>
+David Anderson <danderson@google.com>
+Dean Sturtevant
+Gene Volovich <gv@cite.com>
+Hal Burch <gmock@hburch.com>
+Jeffrey Yasskin <jyasskin@google.com>
+Jim Keller <jimkeller@google.com>
+Joe Walnes <joe@truemesh.com>
+Jon Wray <jwray@google.com>
+Keir Mierle <mierle@gmail.com>
+Keith Ray <keith.ray@gmail.com>
+Kostya Serebryany <kcc@google.com>
+Lev Makhlis
+Manuel Klimek <klimek@google.com>
+Mario Tanev <radix@google.com>
+Mark Paskin
+Markus Heule <markus.heule@gmail.com>
+Matthew Simmons <simmonmt@acm.org>
+Mike Bland <mbland@google.com>
+Neal Norwitz <nnorwitz@gmail.com>
+Nermin Ozkiranartli <nermin@google.com>
+Owen Carlsen <ocarlsen@google.com>
+Paneendra Ba <paneendra@google.com>
+Paul Menage <menage@google.com>
+Piotr Kaminski <piotrk@google.com>
+Russ Rufer <russ@pentad.com>
+Sverre Sundsdal <sundsdal@gmail.com>
+Takeshi Yoshino <tyoshino@google.com>
+Vadim Berman <vadimb@google.com>
+Vlad Losev <vladl@google.com>
+Wolfgang Klier <wklier@google.com>
+Zhanyong Wan <wan@google.com>

subprojects/gtest/googlemock/LICENSE

@@ -0,0 +1,28 @@
+Copyright 2008, Google Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+    * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+    * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

subprojects/gtest/googlemock/Makefile.am

@@ -0,0 +1,224 @@
+# Automake file
+
+# Nonstandard package files for distribution.
+EXTRA_DIST = LICENSE
+
+# We may need to build our internally packaged gtest. If so, it will be
+# included in the 'subdirs' variable.
+SUBDIRS = $(subdirs)
+
+# This is generated by the configure script, so clean it for distribution.
+DISTCLEANFILES = scripts/gmock-config
+
+# We define the global AM_CPPFLAGS as everything we compile includes from these
+# directories.
+AM_CPPFLAGS = $(GTEST_CPPFLAGS) -I$(srcdir)/include
+
+# Modifies compiler and linker flags for pthreads compatibility.
+if HAVE_PTHREADS
+  AM_CXXFLAGS = @PTHREAD_CFLAGS@ -DGTEST_HAS_PTHREAD=1
+  AM_LIBS = @PTHREAD_LIBS@
+endif
+
+# Build rules for libraries.
+lib_LTLIBRARIES = lib/libgmock.la lib/libgmock_main.la
+
+lib_libgmock_la_SOURCES = src/gmock-all.cc
+
+pkginclude_HEADERS = \
+  include/gmock/gmock-actions.h \
+  include/gmock/gmock-cardinalities.h \
+  include/gmock/gmock-generated-actions.h \
+  include/gmock/gmock-generated-function-mockers.h \
+  include/gmock/gmock-generated-matchers.h \
+  include/gmock/gmock-generated-nice-strict.h \
+  include/gmock/gmock-matchers.h \
+  include/gmock/gmock-more-actions.h \
+  include/gmock/gmock-more-matchers.h \
+  include/gmock/gmock-spec-builders.h \
+  include/gmock/gmock.h
+
+pkginclude_internaldir = $(pkgincludedir)/internal
+pkginclude_internal_HEADERS = \
+  include/gmock/internal/gmock-generated-internal-utils.h \
+  include/gmock/internal/gmock-internal-utils.h \
+  include/gmock/internal/gmock-port.h \
+  include/gmock/internal/custom/gmock-generated-actions.h \
+  include/gmock/internal/custom/gmock-matchers.h \
+  include/gmock/internal/custom/gmock-port.h
+
+lib_libgmock_main_la_SOURCES = src/gmock_main.cc
+lib_libgmock_main_la_LIBADD = lib/libgmock.la
+
+# Build rules for tests. Automake's naming for some of these variables isn't
+# terribly obvious, so this is a brief reference:
+#
+# TESTS -- Programs run automatically by "make check"
+# check_PROGRAMS -- Programs built by "make check" but not necessarily run
+
+TESTS=
+check_PROGRAMS=
+AM_LDFLAGS = $(GTEST_LDFLAGS)
+
+# This exercises all major components of Google Mock.  It also
+# verifies that libgmock works.
+TESTS += test/gmock-spec-builders_test
+check_PROGRAMS += test/gmock-spec-builders_test
+test_gmock_spec_builders_test_SOURCES = test/gmock-spec-builders_test.cc
+test_gmock_spec_builders_test_LDADD = $(GTEST_LIBS) lib/libgmock.la
+
+# This tests using Google Mock in multiple translation units.  It also
+# verifies that libgmock_main and libgmock work.
+TESTS += test/gmock_link_test
+check_PROGRAMS += test/gmock_link_test
+test_gmock_link_test_SOURCES = \
+  test/gmock_link2_test.cc \
+  test/gmock_link_test.cc \
+  test/gmock_link_test.h
+test_gmock_link_test_LDADD = $(GTEST_LIBS) lib/libgmock_main.la  lib/libgmock.la
+
+if HAVE_PYTHON
+  # Tests that fused gmock files compile and work.
+  TESTS += test/gmock_fused_test
+  check_PROGRAMS += test/gmock_fused_test
+  test_gmock_fused_test_SOURCES = \
+    fused-src/gmock-gtest-all.cc \
+    fused-src/gmock/gmock.h \
+    fused-src/gmock_main.cc \
+    fused-src/gtest/gtest.h \
+    test/gmock_test.cc
+  test_gmock_fused_test_CPPFLAGS = -I"$(srcdir)/fused-src"
+endif
+
+# Google Mock source files that we don't compile directly.
+GMOCK_SOURCE_INGLUDES = \
+  src/gmock-cardinalities.cc \
+  src/gmock-internal-utils.cc \
+  src/gmock-matchers.cc \
+  src/gmock-spec-builders.cc \
+  src/gmock.cc
+
+EXTRA_DIST += $(GMOCK_SOURCE_INGLUDES)
+
+# C++ tests that we don't compile using autotools.
+EXTRA_DIST += \
+  test/gmock-actions_test.cc \
+  test/gmock_all_test.cc \
+  test/gmock-cardinalities_test.cc \
+  test/gmock_ex_test.cc \
+  test/gmock-generated-actions_test.cc \
+  test/gmock-generated-function-mockers_test.cc \
+  test/gmock-generated-internal-utils_test.cc \
+  test/gmock-generated-matchers_test.cc \
+  test/gmock-internal-utils_test.cc \
+  test/gmock-matchers_test.cc \
+  test/gmock-more-actions_test.cc \
+  test/gmock-nice-strict_test.cc \
+  test/gmock-port_test.cc \
+  test/gmock_stress_test.cc
+
+# Python tests, which we don't run using autotools.
+EXTRA_DIST += \
+  test/gmock_leak_test.py \
+  test/gmock_leak_test_.cc \
+  test/gmock_output_test.py \
+  test/gmock_output_test_.cc \
+  test/gmock_output_test_golden.txt \
+  test/gmock_test_utils.py
+
+# Nonstandard package files for distribution.
+EXTRA_DIST += \
+  CHANGES \
+  CONTRIBUTORS \
+  make/Makefile
+
+# Pump scripts for generating Google Mock headers.
+# TODO(chandlerc@google.com): automate the generation of *.h from *.h.pump.
+EXTRA_DIST += \
+  include/gmock/gmock-generated-actions.h.pump \
+  include/gmock/gmock-generated-function-mockers.h.pump \
+  include/gmock/gmock-generated-matchers.h.pump \
+  include/gmock/gmock-generated-nice-strict.h.pump \
+  include/gmock/internal/gmock-generated-internal-utils.h.pump \
+  include/gmock/internal/custom/gmock-generated-actions.h.pump
+
+# Script for fusing Google Mock and Google Test source files.
+EXTRA_DIST += scripts/fuse_gmock_files.py
+
+# The Google Mock Generator tool from the cppclean project.
+EXTRA_DIST += \
+  scripts/generator/LICENSE \
+  scripts/generator/README \
+  scripts/generator/README.cppclean \
+  scripts/generator/cpp/__init__.py \
+  scripts/generator/cpp/ast.py \
+  scripts/generator/cpp/gmock_class.py \
+  scripts/generator/cpp/keywords.py \
+  scripts/generator/cpp/tokenize.py \
+  scripts/generator/cpp/utils.py \
+  scripts/generator/gmock_gen.py
+
+# Script for diagnosing compiler errors in programs that use Google
+# Mock.
+EXTRA_DIST += scripts/gmock_doctor.py
+
+# CMake scripts.
+EXTRA_DIST += \
+  CMakeLists.txt
+
+# Microsoft Visual Studio 2005 projects.
+EXTRA_DIST += \
+  msvc/2005/gmock.sln \
+  msvc/2005/gmock.vcproj \
+  msvc/2005/gmock_config.vsprops \
+  msvc/2005/gmock_main.vcproj \
+  msvc/2005/gmock_test.vcproj
+
+# Microsoft Visual Studio 2010 projects.
+EXTRA_DIST += \
+  msvc/2010/gmock.sln \
+  msvc/2010/gmock.vcxproj \
+  msvc/2010/gmock_config.props \
+  msvc/2010/gmock_main.vcxproj \
+  msvc/2010/gmock_test.vcxproj
+
+if HAVE_PYTHON
+# gmock_test.cc does not really depend on files generated by the
+# fused-gmock-internal rule.  However, gmock_test.o does, and it is
+# important to include test/gmock_test.cc as part of this rule in order to
+# prevent compiling gmock_test.o until all dependent files have been
+# generated.
+$(test_gmock_fused_test_SOURCES): fused-gmock-internal
+
+# TODO(vladl@google.com): Find a way to add Google Tests's sources here.
+fused-gmock-internal: $(pkginclude_HEADERS) $(pkginclude_internal_HEADERS) \
+                      $(lib_libgmock_la_SOURCES) $(GMOCK_SOURCE_INGLUDES) \
+                      $(lib_libgmock_main_la_SOURCES) \
+                      scripts/fuse_gmock_files.py
+	mkdir -p "$(srcdir)/fused-src"
+	chmod -R u+w "$(srcdir)/fused-src"
+	rm -f "$(srcdir)/fused-src/gtest/gtest.h"
+	rm -f "$(srcdir)/fused-src/gmock/gmock.h"
+	rm -f "$(srcdir)/fused-src/gmock-gtest-all.cc"
+	"$(srcdir)/scripts/fuse_gmock_files.py" "$(srcdir)/fused-src"
+	cp -f "$(srcdir)/src/gmock_main.cc" "$(srcdir)/fused-src"
+
+maintainer-clean-local:
+	rm -rf "$(srcdir)/fused-src"
+endif
+
+# Death tests may produce core dumps in the build directory. In case
+# this happens, clean them to keep distcleancheck happy.
+CLEANFILES = core
+
+# Disables 'make install' as installing a compiled version of Google
+# Mock can lead to undefined behavior due to violation of the
+# One-Definition Rule.
+
+install-exec-local:
+	echo "'make install' is dangerous and not supported. Instead, see README for how to integrate Google Mock into your build system."
+	false
+
+install-data-local:
+	echo "'make install' is dangerous and not supported. Instead, see README for how to integrate Google Mock into your build system."
+	false

subprojects/gtest/googlemock/README.md

@@ -0,0 +1,333 @@
+## Google Mock ##
+
+The Google C++ mocking framework.
+
+### Overview ###
+
+Google's framework for writing and using C++ mock classes.
+It can help you derive better designs of your system and write better tests.
+
+It is inspired by:
+
+  * [jMock](http://www.jmock.org/),
+  * [EasyMock](http://www.easymock.org/), and
+  * [Hamcrest](http://code.google.com/p/hamcrest/),
+
+and designed with C++'s specifics in mind.
+
+Google mock:
+
+  * lets you create mock classes trivially using simple macros.
+  * supports a rich set of matchers and actions.
+  * handles unordered, partially ordered, or completely ordered expectations.
+  * is extensible by users.
+
+We hope you find it useful!
+
+### Features ###
+
+  * Provides a declarative syntax for defining mocks.
+  * Can easily define partial (hybrid) mocks, which are a cross of real
+    and mock objects.
+  * Handles functions of arbitrary types and overloaded functions.
+  * Comes with a rich set of matchers for validating function arguments.
+  * Uses an intuitive syntax for controlling the behavior of a mock.
+  * Does automatic verification of expectations (no record-and-replay needed).
+  * Allows arbitrary (partial) ordering constraints on
+    function calls to be expressed,.
+  * Lets a user extend it by defining new matchers and actions.
+  * Does not use exceptions.
+  * Is easy to learn and use.
+
+Please see the project page above for more information as well as the
+mailing list for questions, discussions, and development.  There is
+also an IRC channel on OFTC (irc.oftc.net) #gtest available.  Please
+join us!
+
+Please note that code under [scripts/generator](scripts/generator/) is
+from [cppclean](http://code.google.com/p/cppclean/) and released under
+the Apache License, which is different from Google Mock's license.
+
+## Getting Started ##
+
+If you are new to the project, we suggest that you read the user
+documentation in the following order:
+
+  * Learn the [basics](../googletest/docs/Primer.md) of
+    Google Test, if you choose to use Google Mock with it (recommended).
+  * Read [Google Mock for Dummies](docs/ForDummies.md).
+  * Read the instructions below on how to build Google Mock.
+
+You can also watch Zhanyong's [talk](http://www.youtube.com/watch?v=sYpCyLI47rM) on Google Mock's usage and implementation.
+
+Once you understand the basics, check out the rest of the docs:
+
+  * [CheatSheet](docs/CheatSheet.md) - all the commonly used stuff
+    at a glance.
+  * [CookBook](docs/CookBook.md) - recipes for getting things done,
+    including advanced techniques.
+
+If you need help, please check the
+[KnownIssues](docs/KnownIssues.md) and
+[FrequentlyAskedQuestions](docs/FrequentlyAskedQuestions.md) before
+posting a question on the
+[discussion group](http://groups.google.com/group/googlemock).
+
+
+### Using Google Mock Without Google Test ###
+
+Google Mock is not a testing framework itself.  Instead, it needs a
+testing framework for writing tests.  Google Mock works seamlessly
+with [Google Test](http://code.google.com/p/googletest/), but
+you can also use it with [any C++ testing framework](googlemock/ForDummies.md#Using_Google_Mock_with_Any_Testing_Framework).
+
+### Requirements for End Users ###
+
+Google Mock is implemented on top of [Google Test](
+http://github.com/google/googletest/), and depends on it.
+You must use the bundled version of Google Test when using Google Mock.
+
+You can also easily configure Google Mock to work with another testing
+framework, although it will still need Google Test.  Please read
+["Using_Google_Mock_with_Any_Testing_Framework"](
+    docs/ForDummies.md#Using_Google_Mock_with_Any_Testing_Framework)
+for instructions.
+
+Google Mock depends on advanced C++ features and thus requires a more
+modern compiler. The following are needed to use Google Mock:
+
+#### Linux Requirements ####
+
+  * GNU-compatible Make or "gmake"
+  * POSIX-standard shell
+  * POSIX(-2) Regular Expressions (regex.h)
+  * C++98-standard-compliant compiler (e.g. GCC 3.4 or newer)
+
+#### Windows Requirements ####
+
+  * Microsoft Visual C++ 8.0 SP1 or newer
+
+#### Mac OS X Requirements ####
+
+  * Mac OS X 10.4 Tiger or newer
+  * Developer Tools Installed
+
+### Requirements for Contributors ###
+
+We welcome patches. If you plan to contribute a patch, you need to
+build Google Mock and its tests, which has further requirements:
+
+  * Automake version 1.9 or newer
+  * Autoconf version 2.59 or newer
+  * Libtool / Libtoolize
+  * Python version 2.3 or newer (for running some of the tests and
+    re-generating certain source files from templates)
+
+### Building Google Mock ###
+
+#### Preparing to Build (Unix only) ####
+
+If you are using a Unix system and plan to use the GNU Autotools build
+system to build Google Mock (described below), you'll need to
+configure it now.
+
+To prepare the Autotools build system:
+
+    cd googlemock
+    autoreconf -fvi
+
+To build Google Mock and your tests that use it, you need to tell your
+build system where to find its headers and source files.  The exact
+way to do it depends on which build system you use, and is usually
+straightforward.
+
+This section shows how you can integrate Google Mock into your
+existing build system.
+
+Suppose you put Google Mock in directory `${GMOCK_DIR}` and Google Test
+in `${GTEST_DIR}` (the latter is `${GMOCK_DIR}/gtest` by default).  To
+build Google Mock, create a library build target (or a project as
+called by Visual Studio and Xcode) to compile
+
+    ${GTEST_DIR}/src/gtest-all.cc and ${GMOCK_DIR}/src/gmock-all.cc
+
+with
+
+    ${GTEST_DIR}/include and ${GMOCK_DIR}/include
+
+in the system header search path, and
+
+    ${GTEST_DIR} and ${GMOCK_DIR}
+
+in the normal header search path.  Assuming a Linux-like system and gcc,
+something like the following will do:
+
+    g++ -isystem ${GTEST_DIR}/include -I${GTEST_DIR} \
+        -isystem ${GMOCK_DIR}/include -I${GMOCK_DIR} \
+        -pthread -c ${GTEST_DIR}/src/gtest-all.cc
+    g++ -isystem ${GTEST_DIR}/include -I${GTEST_DIR} \
+        -isystem ${GMOCK_DIR}/include -I${GMOCK_DIR} \
+        -pthread -c ${GMOCK_DIR}/src/gmock-all.cc
+    ar -rv libgmock.a gtest-all.o gmock-all.o
+
+(We need -pthread as Google Test and Google Mock use threads.)
+
+Next, you should compile your test source file with
+${GTEST\_DIR}/include and ${GMOCK\_DIR}/include in the header search
+path, and link it with gmock and any other necessary libraries:
+
+    g++ -isystem ${GTEST_DIR}/include -isystem ${GMOCK_DIR}/include \
+        -pthread path/to/your_test.cc libgmock.a -o your_test
+
+As an example, the make/ directory contains a Makefile that you can
+use to build Google Mock on systems where GNU make is available
+(e.g. Linux, Mac OS X, and Cygwin).  It doesn't try to build Google
+Mock's own tests.  Instead, it just builds the Google Mock library and
+a sample test.  You can use it as a starting point for your own build
+script.
+
+If the default settings are correct for your environment, the
+following commands should succeed:
+
+    cd ${GMOCK_DIR}/make
+    make
+    ./gmock_test
+
+If you see errors, try to tweak the contents of
+[make/Makefile](make/Makefile) to make them go away.
+
+### Windows ###
+
+The msvc/2005 directory contains VC++ 2005 projects and the msvc/2010
+directory contains VC++ 2010 projects for building Google Mock and
+selected tests.
+
+Change to the appropriate directory and run "msbuild gmock.sln" to
+build the library and tests (or open the gmock.sln in the MSVC IDE).
+If you want to create your own project to use with Google Mock, you'll
+have to configure it to use the `gmock_config` propety sheet.  For that:
+
+ * Open the Property Manager window (View | Other Windows | Property Manager)
+ * Right-click on your project and select "Add Existing Property Sheet..."
+ * Navigate to `gmock_config.vsprops` or `gmock_config.props` and select it.
+ * In Project Properties | Configuration Properties | General | Additional
+   Include Directories, type <path to Google Mock>/include.
+
+### Tweaking Google Mock ###
+
+Google Mock can be used in diverse environments.  The default
+configuration may not work (or may not work well) out of the box in
+some environments.  However, you can easily tweak Google Mock by
+defining control macros on the compiler command line.  Generally,
+these macros are named like `GTEST_XYZ` and you define them to either 1
+or 0 to enable or disable a certain feature.
+
+We list the most frequently used macros below.  For a complete list,
+see file [${GTEST\_DIR}/include/gtest/internal/gtest-port.h](
+../googletest/include/gtest/internal/gtest-port.h).
+
+### Choosing a TR1 Tuple Library ###
+
+Google Mock uses the C++ Technical Report 1 (TR1) tuple library
+heavily.  Unfortunately TR1 tuple is not yet widely available with all
+compilers.  The good news is that Google Test 1.4.0+ implements a
+subset of TR1 tuple that's enough for Google Mock's need.  Google Mock
+will automatically use that implementation when the compiler doesn't
+provide TR1 tuple.
+
+Usually you don't need to care about which tuple library Google Test
+and Google Mock use.  However, if your project already uses TR1 tuple,
+you need to tell Google Test and Google Mock to use the same TR1 tuple
+library the rest of your project uses, or the two tuple
+implementations will clash.  To do that, add
+
+    -DGTEST_USE_OWN_TR1_TUPLE=0
+
+to the compiler flags while compiling Google Test, Google Mock, and
+your tests.  If you want to force Google Test and Google Mock to use
+their own tuple library, just add
+
+    -DGTEST_USE_OWN_TR1_TUPLE=1
+
+to the compiler flags instead.
+
+If you want to use Boost's TR1 tuple library with Google Mock, please
+refer to the Boost website (http://www.boost.org/) for how to obtain
+it and set it up.
+
+### As a Shared Library (DLL) ###
+
+Google Mock is compact, so most users can build and link it as a static
+library for the simplicity.  Google Mock can be used as a DLL, but the
+same DLL must contain Google Test as well.  See
+[Google Test's README][gtest_readme]
+for instructions on how to set up necessary compiler settings.
+
+### Tweaking Google Mock ###
+
+Most of Google Test's control macros apply to Google Mock as well.
+Please see [Google Test's README][gtest_readme] for how to tweak them.
+
+### Upgrading from an Earlier Version ###
+
+We strive to keep Google Mock releases backward compatible.
+Sometimes, though, we have to make some breaking changes for the
+users' long-term benefits.  This section describes what you'll need to
+do if you are upgrading from an earlier version of Google Mock.
+
+#### Upgrading from 1.1.0 or Earlier ####
+
+You may need to explicitly enable or disable Google Test's own TR1
+tuple library.  See the instructions in section "[Choosing a TR1 Tuple
+Library](../googletest/#choosing-a-tr1-tuple-library)".
+
+#### Upgrading from 1.4.0 or Earlier ####
+
+On platforms where the pthread library is available, Google Test and
+Google Mock use it in order to be thread-safe.  For this to work, you
+may need to tweak your compiler and/or linker flags.  Please see the
+"[Multi-threaded Tests](../googletest#multi-threaded-tests
+)" section in file Google Test's README for what you may need to do.
+
+If you have custom matchers defined using `MatcherInterface` or
+`MakePolymorphicMatcher()`, you'll need to update their definitions to
+use the new matcher API (
+[monomorphic](http://code.google.com/p/googlemock/wiki/CookBook#Writing_New_Monomorphic_Matchers),
+[polymorphic](http://code.google.com/p/googlemock/wiki/CookBook#Writing_New_Polymorphic_Matchers)).
+Matchers defined using `MATCHER()` or `MATCHER_P*()` aren't affected.
+
+### Developing Google Mock ###
+
+This section discusses how to make your own changes to Google Mock.
+
+#### Testing Google Mock Itself ####
+
+To make sure your changes work as intended and don't break existing
+functionality, you'll want to compile and run Google Test's own tests.
+For that you'll need Autotools.  First, make sure you have followed
+the instructions above to configure Google Mock.
+Then, create a build output directory and enter it.  Next,
+
+    ${GMOCK_DIR}/configure  # try --help for more info
+
+Once you have successfully configured Google Mock, the build steps are
+standard for GNU-style OSS packages.
+
+    make        # Standard makefile following GNU conventions
+    make check  # Builds and runs all tests - all should pass.
+
+Note that when building your project against Google Mock, you are building
+against Google Test as well.  There is no need to configure Google Test
+separately.
+
+#### Contributing a Patch ####
+
+We welcome patches.
+Please read the [Developer's Guide](docs/DevGuide.md)
+for how you can contribute. In particular, make sure you have signed
+the Contributor License Agreement, or we won't be able to accept the
+patch.
+
+Happy testing!
+
+[gtest_readme]: ../googletest/README.md "googletest"

subprojects/gtest/googlemock/configure.ac

@@ -0,0 +1,146 @@
+m4_include(../googletest/m4/acx_pthread.m4)
+
+AC_INIT([Google C++ Mocking Framework],
+        [1.7.0],
+        [googlemock@googlegroups.com],
+        [gmock])
+
+# Provide various options to initialize the Autoconf and configure processes.
+AC_PREREQ([2.59])
+AC_CONFIG_SRCDIR([./LICENSE])
+AC_CONFIG_AUX_DIR([build-aux])
+AC_CONFIG_HEADERS([build-aux/config.h])
+AC_CONFIG_FILES([Makefile])
+AC_CONFIG_FILES([scripts/gmock-config], [chmod +x scripts/gmock-config])
+
+# Initialize Automake with various options. We require at least v1.9, prevent
+# pedantic complaints about package files, and enable various distribution
+# targets.
+AM_INIT_AUTOMAKE([1.9 dist-bzip2 dist-zip foreign subdir-objects])
+
+# Check for programs used in building Google Test.
+AC_PROG_CC
+AC_PROG_CXX
+AC_LANG([C++])
+AC_PROG_LIBTOOL
+
+# TODO(chandlerc@google.com): Currently we aren't running the Python tests
+# against the interpreter detected by AM_PATH_PYTHON, and so we condition
+# HAVE_PYTHON by requiring "python" to be in the PATH, and that interpreter's
+# version to be >= 2.3. This will allow the scripts to use a "/usr/bin/env"
+# hashbang.
+PYTHON=  # We *do not* allow the user to specify a python interpreter
+AC_PATH_PROG([PYTHON],[python],[:])
+AS_IF([test "$PYTHON" != ":"],
+      [AM_PYTHON_CHECK_VERSION([$PYTHON],[2.3],[:],[PYTHON=":"])])
+AM_CONDITIONAL([HAVE_PYTHON],[test "$PYTHON" != ":"])
+
+# TODO(chandlerc@google.com) Check for the necessary system headers.
+
+# Configure pthreads.
+AC_ARG_WITH([pthreads],
+            [AS_HELP_STRING([--with-pthreads],
+               [use pthreads (default is yes)])],
+            [with_pthreads=$withval],
+            [with_pthreads=check])
+
+have_pthreads=no
+AS_IF([test "x$with_pthreads" != "xno"],
+      [ACX_PTHREAD(
+        [],
+        [AS_IF([test "x$with_pthreads" != "xcheck"],
+               [AC_MSG_FAILURE(
+                 [--with-pthreads was specified, but unable to be used])])])
+       have_pthreads="$acx_pthread_ok"])
+AM_CONDITIONAL([HAVE_PTHREADS],[test "x$have_pthreads" == "xyes"])
+AC_SUBST(PTHREAD_CFLAGS)
+AC_SUBST(PTHREAD_LIBS)
+
+# GoogleMock currently has hard dependencies upon GoogleTest above and beyond
+# running its own test suite, so we both provide our own version in
+# a subdirectory and provide some logic to use a custom version or a system
+# installed version.
+AC_ARG_WITH([gtest],
+            [AS_HELP_STRING([--with-gtest],
+                            [Specifies how to find the gtest package. If no
+                            arguments are given, the default behavior, a
+                            system installed gtest will be used if present,
+                            and an internal version built otherwise. If a
+                            path is provided, the gtest built or installed at
+                            that prefix will be used.])],
+            [],
+            [with_gtest=yes])
+AC_ARG_ENABLE([external-gtest],
+              [AS_HELP_STRING([--disable-external-gtest],
+                              [Disables any detection or use of a system
+                              installed or user provided gtest. Any option to
+                              '--with-gtest' is ignored. (Default is enabled.)])
+              ], [], [enable_external_gtest=yes])
+AS_IF([test "x$with_gtest" == "xno"],
+      [AC_MSG_ERROR([dnl
+Support for GoogleTest was explicitly disabled. Currently GoogleMock has a hard
+dependency upon GoogleTest to build, please provide a version, or allow
+GoogleMock to use any installed version and fall back upon its internal
+version.])])
+
+# Setup various GTEST variables. TODO(chandlerc@google.com): When these are
+# used below, they should be used such that any pre-existing values always
+# trump values we set them to, so that they can be used to selectively override
+# details of the detection process.
+AC_ARG_VAR([GTEST_CONFIG],
+           [The exact path of Google Test's 'gtest-config' script.])
+AC_ARG_VAR([GTEST_CPPFLAGS],
+           [C-like preprocessor flags for Google Test.])
+AC_ARG_VAR([GTEST_CXXFLAGS],
+           [C++ compile flags for Google Test.])
+AC_ARG_VAR([GTEST_LDFLAGS],
+           [Linker path and option flags for Google Test.])
+AC_ARG_VAR([GTEST_LIBS],
+           [Library linking flags for Google Test.])
+AC_ARG_VAR([GTEST_VERSION],
+           [The version of Google Test available.])
+HAVE_BUILT_GTEST="no"
+
+GTEST_MIN_VERSION="1.7.0"
+
+AS_IF([test "x${enable_external_gtest}" = "xyes"],
+      [# Begin filling in variables as we are able.
+      AS_IF([test "x${with_gtest}" != "xyes"],
+            [AS_IF([test -x "${with_gtest}/scripts/gtest-config"],
+                   [GTEST_CONFIG="${with_gtest}/scripts/gtest-config"],
+                   [GTEST_CONFIG="${with_gtest}/bin/gtest-config"])
+            AS_IF([test -x "${GTEST_CONFIG}"], [],
+                  [AC_MSG_ERROR([dnl
+Unable to locate either a built or installed Google Test at '${with_gtest}'.])
+                  ])])
+
+      AS_IF([test -x "${GTEST_CONFIG}"], [],
+            [AC_PATH_PROG([GTEST_CONFIG], [gtest-config])])
+      AS_IF([test -x "${GTEST_CONFIG}"],
+            [AC_MSG_CHECKING([for Google Test version >= ${GTEST_MIN_VERSION}])
+            AS_IF([${GTEST_CONFIG} --min-version=${GTEST_MIN_VERSION}],
+                  [AC_MSG_RESULT([yes])
+                  HAVE_BUILT_GTEST="yes"],
+                  [AC_MSG_RESULT([no])])])])
+
+AS_IF([test "x${HAVE_BUILT_GTEST}" = "xyes"],
+      [GTEST_CPPFLAGS=`${GTEST_CONFIG} --cppflags`
+      GTEST_CXXFLAGS=`${GTEST_CONFIG} --cxxflags`
+      GTEST_LDFLAGS=`${GTEST_CONFIG} --ldflags`
+      GTEST_LIBS=`${GTEST_CONFIG} --libs`
+      GTEST_VERSION=`${GTEST_CONFIG} --version`],
+      [AC_CONFIG_SUBDIRS([../googletest])
+      # GTEST_CONFIG needs to be executable both in a Makefile environmont and
+      # in a shell script environment, so resolve an absolute path for it here.
+      GTEST_CONFIG="`pwd -P`/../googletest/scripts/gtest-config"
+      GTEST_CPPFLAGS='-I$(top_srcdir)/../googletest/include'
+      GTEST_CXXFLAGS='-g'
+      GTEST_LDFLAGS=''
+      GTEST_LIBS='$(top_builddir)/../googletest/lib/libgtest.la'
+      GTEST_VERSION="${GTEST_MIN_VERSION}"])
+
+# TODO(chandlerc@google.com) Check the types, structures, and other compiler
+# and architecture characteristics.
+
+# Output the generated files. No further autoconf macros may be used.
+AC_OUTPUT

subprojects/gtest/googlemock/docs/CheatSheet.md

@@ -0,0 +1,562 @@
+
+
+# Defining a Mock Class #
+
+## Mocking a Normal Class ##
+
+Given
+```
+class Foo {
+  ...
+  virtual ~Foo();
+  virtual int GetSize() const = 0;
+  virtual string Describe(const char* name) = 0;
+  virtual string Describe(int type) = 0;
+  virtual bool Process(Bar elem, int count) = 0;
+};
+```
+(note that `~Foo()` **must** be virtual) we can define its mock as
+```
+#include "gmock/gmock.h"
+
+class MockFoo : public Foo {
+  MOCK_CONST_METHOD0(GetSize, int());
+  MOCK_METHOD1(Describe, string(const char* name));
+  MOCK_METHOD1(Describe, string(int type));
+  MOCK_METHOD2(Process, bool(Bar elem, int count));
+};
+```
+
+To create a "nice" mock object which ignores all uninteresting calls,
+or a "strict" mock object, which treats them as failures:
+```
+NiceMock<MockFoo> nice_foo;     // The type is a subclass of MockFoo.
+StrictMock<MockFoo> strict_foo; // The type is a subclass of MockFoo.
+```
+
+## Mocking a Class Template ##
+
+To mock
+```
+template <typename Elem>
+class StackInterface {
+ public:
+  ...
+  virtual ~StackInterface();
+  virtual int GetSize() const = 0;
+  virtual void Push(const Elem& x) = 0;
+};
+```
+(note that `~StackInterface()` **must** be virtual) just append `_T` to the `MOCK_*` macros:
+```
+template <typename Elem>
+class MockStack : public StackInterface<Elem> {
+ public:
+  ...
+  MOCK_CONST_METHOD0_T(GetSize, int());
+  MOCK_METHOD1_T(Push, void(const Elem& x));
+};
+```
+
+## Specifying Calling Conventions for Mock Functions ##
+
+If your mock function doesn't use the default calling convention, you
+can specify it by appending `_WITH_CALLTYPE` to any of the macros
+described in the previous two sections and supplying the calling
+convention as the first argument to the macro. For example,
+```
+  MOCK_METHOD_1_WITH_CALLTYPE(STDMETHODCALLTYPE, Foo, bool(int n));
+  MOCK_CONST_METHOD2_WITH_CALLTYPE(STDMETHODCALLTYPE, Bar, int(double x, double y));
+```
+where `STDMETHODCALLTYPE` is defined by `<objbase.h>` on Windows.
+
+# Using Mocks in Tests #
+
+The typical flow is:
+  1. Import the Google Mock names you need to use. All Google Mock names are in the `testing` namespace unless they are macros or otherwise noted.
+  1. Create the mock objects.
+  1. Optionally, set the default actions of the mock objects.
+  1. Set your expectations on the mock objects (How will they be called? What wil they do?).
+  1. Exercise code that uses the mock objects; if necessary, check the result using [Google Test](../../googletest/) assertions.
+  1. When a mock objects is destructed, Google Mock automatically verifies that all expectations on it have been satisfied.
+
+Here is an example:
+```
+using ::testing::Return;                            // #1
+
+TEST(BarTest, DoesThis) {
+  MockFoo foo;                                    // #2
+
+  ON_CALL(foo, GetSize())                         // #3
+      .WillByDefault(Return(1));
+  // ... other default actions ...
+
+  EXPECT_CALL(foo, Describe(5))                   // #4
+      .Times(3)
+      .WillRepeatedly(Return("Category 5"));
+  // ... other expectations ...
+
+  EXPECT_EQ("good", MyProductionFunction(&foo));  // #5
+}                                                 // #6
+```
+
+# Setting Default Actions #
+
+Google Mock has a **built-in default action** for any function that
+returns `void`, `bool`, a numeric value, or a pointer.
+
+To customize the default action for functions with return type `T` globally:
+```
+using ::testing::DefaultValue;
+
+// Sets the default value to be returned. T must be CopyConstructible.
+DefaultValue<T>::Set(value);
+// Sets a factory. Will be invoked on demand. T must be MoveConstructible.
+//   T MakeT();
+DefaultValue<T>::SetFactory(&MakeT);
+// ... use the mocks ...
+// Resets the default value.
+DefaultValue<T>::Clear();
+```
+
+To customize the default action for a particular method, use `ON_CALL()`:
+```
+ON_CALL(mock_object, method(matchers))
+    .With(multi_argument_matcher)  ?
+    .WillByDefault(action);
+```
+
+# Setting Expectations #
+
+`EXPECT_CALL()` sets **expectations** on a mock method (How will it be
+called? What will it do?):
+```
+EXPECT_CALL(mock_object, method(matchers))
+    .With(multi_argument_matcher)  ?
+    .Times(cardinality)            ?
+    .InSequence(sequences)         *
+    .After(expectations)           *
+    .WillOnce(action)              *
+    .WillRepeatedly(action)        ?
+    .RetiresOnSaturation();        ?
+```
+
+If `Times()` is omitted, the cardinality is assumed to be:
+
+  * `Times(1)` when there is neither `WillOnce()` nor `WillRepeatedly()`;
+  * `Times(n)` when there are `n WillOnce()`s but no `WillRepeatedly()`, where `n` >= 1; or
+  * `Times(AtLeast(n))` when there are `n WillOnce()`s and a `WillRepeatedly()`, where `n` >= 0.
+
+A method with no `EXPECT_CALL()` is free to be invoked _any number of times_, and the default action will be taken each time.
+
+# Matchers #
+
+A **matcher** matches a _single_ argument.  You can use it inside
+`ON_CALL()` or `EXPECT_CALL()`, or use it to validate a value
+directly:
+
+| `EXPECT_THAT(value, matcher)` | Asserts that `value` matches `matcher`. |
+|:------------------------------|:----------------------------------------|
+| `ASSERT_THAT(value, matcher)` | The same as `EXPECT_THAT(value, matcher)`, except that it generates a **fatal** failure. |
+
+Built-in matchers (where `argument` is the function argument) are
+divided into several categories:
+
+## Wildcard ##
+|`_`|`argument` can be any value of the correct type.|
+|:--|:-----------------------------------------------|
+|`A<type>()` or `An<type>()`|`argument` can be any value of type `type`.     |
+
+## Generic Comparison ##
+
+|`Eq(value)` or `value`|`argument == value`|
+|:---------------------|:------------------|
+|`Ge(value)`           |`argument >= value`|
+|`Gt(value)`           |`argument > value` |
+|`Le(value)`           |`argument <= value`|
+|`Lt(value)`           |`argument < value` |
+|`Ne(value)`           |`argument != value`|
+|`IsNull()`            |`argument` is a `NULL` pointer (raw or smart).|
+|`NotNull()`           |`argument` is a non-null pointer (raw or smart).|
+|`Ref(variable)`       |`argument` is a reference to `variable`.|
+|`TypedEq<type>(value)`|`argument` has type `type` and is equal to `value`. You may need to use this instead of `Eq(value)` when the mock function is overloaded.|
+
+Except `Ref()`, these matchers make a _copy_ of `value` in case it's
+modified or destructed later. If the compiler complains that `value`
+doesn't have a public copy constructor, try wrap it in `ByRef()`,
+e.g. `Eq(ByRef(non_copyable_value))`. If you do that, make sure
+`non_copyable_value` is not changed afterwards, or the meaning of your
+matcher will be changed.
+
+## Floating-Point Matchers ##
+
+|`DoubleEq(a_double)`|`argument` is a `double` value approximately equal to `a_double`, treating two NaNs as unequal.|
+|:-------------------|:----------------------------------------------------------------------------------------------|
+|`FloatEq(a_float)`  |`argument` is a `float` value approximately equal to `a_float`, treating two NaNs as unequal.  |
+|`NanSensitiveDoubleEq(a_double)`|`argument` is a `double` value approximately equal to `a_double`, treating two NaNs as equal.  |
+|`NanSensitiveFloatEq(a_float)`|`argument` is a `float` value approximately equal to `a_float`, treating two NaNs as equal.    |
+
+The above matchers use ULP-based comparison (the same as used in
+[Google Test](../../googletest/)). They
+automatically pick a reasonable error bound based on the absolute
+value of the expected value.  `DoubleEq()` and `FloatEq()` conform to
+the IEEE standard, which requires comparing two NaNs for equality to
+return false. The `NanSensitive*` version instead treats two NaNs as
+equal, which is often what a user wants.
+
+|`DoubleNear(a_double, max_abs_error)`|`argument` is a `double` value close to `a_double` (absolute error <= `max_abs_error`), treating two NaNs as unequal.|
+|:------------------------------------|:--------------------------------------------------------------------------------------------------------------------|
+|`FloatNear(a_float, max_abs_error)`  |`argument` is a `float` value close to `a_float` (absolute error <= `max_abs_error`), treating two NaNs as unequal.  |
+|`NanSensitiveDoubleNear(a_double, max_abs_error)`|`argument` is a `double` value close to `a_double` (absolute error <= `max_abs_error`), treating two NaNs as equal.  |
+|`NanSensitiveFloatNear(a_float, max_abs_error)`|`argument` is a `float` value close to `a_float` (absolute error <= `max_abs_error`), treating two NaNs as equal.    |
+
+## String Matchers ##
+
+The `argument` can be either a C string or a C++ string object:
+
+|`ContainsRegex(string)`|`argument` matches the given regular expression.|
+|:----------------------|:-----------------------------------------------|
+|`EndsWith(suffix)`     |`argument` ends with string `suffix`.           |
+|`HasSubstr(string)`    |`argument` contains `string` as a sub-string.   |
+|`MatchesRegex(string)` |`argument` matches the given regular expression with the match starting at the first character and ending at the last character.|
+|`StartsWith(prefix)`   |`argument` starts with string `prefix`.         |
+|`StrCaseEq(string)`    |`argument` is equal to `string`, ignoring case. |
+|`StrCaseNe(string)`    |`argument` is not equal to `string`, ignoring case.|
+|`StrEq(string)`        |`argument` is equal to `string`.                |
+|`StrNe(string)`        |`argument` is not equal to `string`.            |
+
+`ContainsRegex()` and `MatchesRegex()` use the regular expression
+syntax defined
+[here](../../googletest/docs/AdvancedGuide.md#regular-expression-syntax).
+`StrCaseEq()`, `StrCaseNe()`, `StrEq()`, and `StrNe()` work for wide
+strings as well.
+
+## Container Matchers ##
+
+Most STL-style containers support `==`, so you can use
+`Eq(expected_container)` or simply `expected_container` to match a
+container exactly.   If you want to write the elements in-line,
+match them more flexibly, or get more informative messages, you can use:
+
+| `ContainerEq(container)` | The same as `Eq(container)` except that the failure message also includes which elements are in one container but not the other. |
+|:-------------------------|:---------------------------------------------------------------------------------------------------------------------------------|
+| `Contains(e)`            | `argument` contains an element that matches `e`, which can be either a value or a matcher.                                       |
+| `Each(e)`                | `argument` is a container where _every_ element matches `e`, which can be either a value or a matcher.                           |
+| `ElementsAre(e0, e1, ..., en)` | `argument` has `n + 1` elements, where the i-th element matches `ei`, which can be a value or a matcher. 0 to 10 arguments are allowed. |
+| `ElementsAreArray({ e0, e1, ..., en })`, `ElementsAreArray(array)`, or `ElementsAreArray(array, count)` | The same as `ElementsAre()` except that the expected element values/matchers come from an initializer list, STL-style container, or C-style array. |
+| `IsEmpty()`              | `argument` is an empty container (`container.empty()`).                                                                          |
+| `Pointwise(m, container)` | `argument` contains the same number of elements as in `container`, and for all i, (the i-th element in `argument`, the i-th element in `container`) match `m`, which is a matcher on 2-tuples. E.g. `Pointwise(Le(), upper_bounds)` verifies that each element in `argument` doesn't exceed the corresponding element in `upper_bounds`. See more detail below. |
+| `SizeIs(m)`              | `argument` is a container whose size matches `m`. E.g. `SizeIs(2)` or `SizeIs(Lt(2))`.                                           |
+| `UnorderedElementsAre(e0, e1, ..., en)` | `argument` has `n + 1` elements, and under some permutation each element matches an `ei` (for a different `i`), which can be a value or a matcher. 0 to 10 arguments are allowed. |
+| `UnorderedElementsAreArray({ e0, e1, ..., en })`, `UnorderedElementsAreArray(array)`, or `UnorderedElementsAreArray(array, count)` | The same as `UnorderedElementsAre()` except that the expected element values/matchers come from an initializer list, STL-style container, or C-style array. |
+| `WhenSorted(m)`          | When `argument` is sorted using the `<` operator, it matches container matcher `m`. E.g. `WhenSorted(UnorderedElementsAre(1, 2, 3))` verifies that `argument` contains elements `1`, `2`, and `3`, ignoring order. |
+| `WhenSortedBy(comparator, m)` | The same as `WhenSorted(m)`, except that the given comparator instead of `<` is used to sort `argument`. E.g. `WhenSortedBy(std::greater<int>(), ElementsAre(3, 2, 1))`. |
+
+Notes:
+
+  * These matchers can also match:
+    1. a native array passed by reference (e.g. in `Foo(const int (&a)[5])`), and
+    1. an array passed as a pointer and a count (e.g. in `Bar(const T* buffer, int len)` -- see [Multi-argument Matchers](#Multiargument_Matchers.md)).
+  * The array being matched may be multi-dimensional (i.e. its elements can be arrays).
+  * `m` in `Pointwise(m, ...)` should be a matcher for `::testing::tuple<T, U>` where `T` and `U` are the element type of the actual container and the expected container, respectively. For example, to compare two `Foo` containers where `Foo` doesn't support `operator==` but has an `Equals()` method, one might write:
+
+```
+using ::testing::get;
+MATCHER(FooEq, "") {
+  return get<0>(arg).Equals(get<1>(arg));
+}
+...
+EXPECT_THAT(actual_foos, Pointwise(FooEq(), expected_foos));
+```
+
+## Member Matchers ##
+
+|`Field(&class::field, m)`|`argument.field` (or `argument->field` when `argument` is a plain pointer) matches matcher `m`, where `argument` is an object of type _class_.|
+|:------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------|
+|`Key(e)`                 |`argument.first` matches `e`, which can be either a value or a matcher. E.g. `Contains(Key(Le(5)))` can verify that a `map` contains a key `<= 5`.|
+|`Pair(m1, m2)`           |`argument` is an `std::pair` whose `first` field matches `m1` and `second` field matches `m2`.                                                |
+|`Property(&class::property, m)`|`argument.property()` (or `argument->property()` when `argument` is a plain pointer) matches matcher `m`, where `argument` is an object of type _class_.|
+
+## Matching the Result of a Function or Functor ##
+
+|`ResultOf(f, m)`|`f(argument)` matches matcher `m`, where `f` is a function or functor.|
+|:---------------|:---------------------------------------------------------------------|
+
+## Pointer Matchers ##
+
+|`Pointee(m)`|`argument` (either a smart pointer or a raw pointer) points to a value that matches matcher `m`.|
+|:-----------|:-----------------------------------------------------------------------------------------------|
+|`WhenDynamicCastTo<T>(m)`| when `argument` is passed through `dynamic_cast<T>()`, it matches matcher `m`.                 |
+
+## Multiargument Matchers ##
+
+Technically, all matchers match a _single_ value. A "multi-argument"
+matcher is just one that matches a _tuple_. The following matchers can
+be used to match a tuple `(x, y)`:
+
+|`Eq()`|`x == y`|
+|:-----|:-------|
+|`Ge()`|`x >= y`|
+|`Gt()`|`x > y` |
+|`Le()`|`x <= y`|
+|`Lt()`|`x < y` |
+|`Ne()`|`x != y`|
+
+You can use the following selectors to pick a subset of the arguments
+(or reorder them) to participate in the matching:
+
+|`AllArgs(m)`|Equivalent to `m`. Useful as syntactic sugar in `.With(AllArgs(m))`.|
+|:-----------|:-------------------------------------------------------------------|
+|`Args<N1, N2, ..., Nk>(m)`|The tuple of the `k` selected (using 0-based indices) arguments matches `m`, e.g. `Args<1, 2>(Eq())`.|
+
+## Composite Matchers ##
+
+You can make a matcher from one or more other matchers:
+
+|`AllOf(m1, m2, ..., mn)`|`argument` matches all of the matchers `m1` to `mn`.|
+|:-----------------------|:---------------------------------------------------|
+|`AnyOf(m1, m2, ..., mn)`|`argument` matches at least one of the matchers `m1` to `mn`.|
+|`Not(m)`                |`argument` doesn't match matcher `m`.               |
+
+## Adapters for Matchers ##
+
+|`MatcherCast<T>(m)`|casts matcher `m` to type `Matcher<T>`.|
+|:------------------|:--------------------------------------|
+|`SafeMatcherCast<T>(m)`| [safely casts](CookBook.md#casting-matchers) matcher `m` to type `Matcher<T>`. |
+|`Truly(predicate)` |`predicate(argument)` returns something considered by C++ to be true, where `predicate` is a function or functor.|
+
+## Matchers as Predicates ##
+
+|`Matches(m)(value)`|evaluates to `true` if `value` matches `m`. You can use `Matches(m)` alone as a unary functor.|
+|:------------------|:---------------------------------------------------------------------------------------------|
+|`ExplainMatchResult(m, value, result_listener)`|evaluates to `true` if `value` matches `m`, explaining the result to `result_listener`.       |
+|`Value(value, m)`  |evaluates to `true` if `value` matches `m`.                                                   |
+
+## Defining Matchers ##
+
+| `MATCHER(IsEven, "") { return (arg % 2) == 0; }` | Defines a matcher `IsEven()` to match an even number. |
+|:-------------------------------------------------|:------------------------------------------------------|
+| `MATCHER_P(IsDivisibleBy, n, "") { *result_listener << "where the remainder is " << (arg % n); return (arg % n) == 0; }` | Defines a macher `IsDivisibleBy(n)` to match a number divisible by `n`. |
+| `MATCHER_P2(IsBetween, a, b, std::string(negation ? "isn't" : "is") + " between " + PrintToString(a) + " and " + PrintToString(b)) { return a <= arg && arg <= b; }` | Defines a matcher `IsBetween(a, b)` to match a value in the range [`a`, `b`]. |
+
+**Notes:**
+
+  1. The `MATCHER*` macros cannot be used inside a function or class.
+  1. The matcher body must be _purely functional_ (i.e. it cannot have any side effect, and the result must not depend on anything other than the value being matched and the matcher parameters).
+  1. You can use `PrintToString(x)` to convert a value `x` of any type to a string.
+
+## Matchers as Test Assertions ##
+
+|`ASSERT_THAT(expression, m)`|Generates a [fatal failure](../../googletest/docs/Primer.md#assertions) if the value of `expression` doesn't match matcher `m`.|
+|:---------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------|
+|`EXPECT_THAT(expression, m)`|Generates a non-fatal failure if the value of `expression` doesn't match matcher `m`.                                                          |
+
+# Actions #
+
+**Actions** specify what a mock function should do when invoked.
+
+## Returning a Value ##
+
+|`Return()`|Return from a `void` mock function.|
+|:---------|:----------------------------------|
+|`Return(value)`|Return `value`. If the type of `value` is different to the mock function's return type, `value` is converted to the latter type <i>at the time the expectation is set</i>, not when the action is executed.|
+|`ReturnArg<N>()`|Return the `N`-th (0-based) argument.|
+|`ReturnNew<T>(a1, ..., ak)`|Return `new T(a1, ..., ak)`; a different object is created each time.|
+|`ReturnNull()`|Return a null pointer.             |
+|`ReturnPointee(ptr)`|Return the value pointed to by `ptr`.|
+|`ReturnRef(variable)`|Return a reference to `variable`.  |
+|`ReturnRefOfCopy(value)`|Return a reference to a copy of `value`; the copy lives as long as the action.|
+
+## Side Effects ##
+
+|`Assign(&variable, value)`|Assign `value` to variable.|
+|:-------------------------|:--------------------------|
+| `DeleteArg<N>()`         | Delete the `N`-th (0-based) argument, which must be a pointer. |
+| `SaveArg<N>(pointer)`    | Save the `N`-th (0-based) argument to `*pointer`. |
+| `SaveArgPointee<N>(pointer)` | Save the value pointed to by the `N`-th (0-based) argument to `*pointer`. |
+| `SetArgReferee<N>(value)` |	Assign value to the variable referenced by the `N`-th (0-based) argument. |
+|`SetArgPointee<N>(value)` |Assign `value` to the variable pointed by the `N`-th (0-based) argument.|
+|`SetArgumentPointee<N>(value)`|Same as `SetArgPointee<N>(value)`. Deprecated. Will be removed in v1.7.0.|
+|`SetArrayArgument<N>(first, last)`|Copies the elements in source range [`first`, `last`) to the array pointed to by the `N`-th (0-based) argument, which can be either a pointer or an iterator. The action does not take ownership of the elements in the source range.|
+|`SetErrnoAndReturn(error, value)`|Set `errno` to `error` and return `value`.|
+|`Throw(exception)`        |Throws the given exception, which can be any copyable value. Available since v1.1.0.|
+
+## Using a Function or a Functor as an Action ##
+
+|`Invoke(f)`|Invoke `f` with the arguments passed to the mock function, where `f` can be a global/static function or a functor.|
+|:----------|:-----------------------------------------------------------------------------------------------------------------|
+|`Invoke(object_pointer, &class::method)`|Invoke the {method on the object with the arguments passed to the mock function.                                  |
+|`InvokeWithoutArgs(f)`|Invoke `f`, which can be a global/static function or a functor. `f` must take no arguments.                       |
+|`InvokeWithoutArgs(object_pointer, &class::method)`|Invoke the method on the object, which takes no arguments.                                                        |
+|`InvokeArgument<N>(arg1, arg2, ..., argk)`|Invoke the mock function's `N`-th (0-based) argument, which must be a function or a functor, with the `k` arguments.|
+
+The return value of the invoked function is used as the return value
+of the action.
+
+When defining a function or functor to be used with `Invoke*()`, you can declare any unused parameters as `Unused`:
+```
+  double Distance(Unused, double x, double y) { return sqrt(x*x + y*y); }
+  ...
+  EXPECT_CALL(mock, Foo("Hi", _, _)).WillOnce(Invoke(Distance));
+```
+
+In `InvokeArgument<N>(...)`, if an argument needs to be passed by reference, wrap it inside `ByRef()`. For example,
+```
+  InvokeArgument<2>(5, string("Hi"), ByRef(foo))
+```
+calls the mock function's #2 argument, passing to it `5` and `string("Hi")` by value, and `foo` by reference.
+
+## Default Action ##
+
+|`DoDefault()`|Do the default action (specified by `ON_CALL()` or the built-in one).|
+|:------------|:--------------------------------------------------------------------|
+
+**Note:** due to technical reasons, `DoDefault()` cannot be used inside  a composite action - trying to do so will result in a run-time error.
+
+## Composite Actions ##
+
+|`DoAll(a1, a2, ..., an)`|Do all actions `a1` to `an` and return the result of `an` in each invocation. The first `n - 1` sub-actions must return void. |
+|:-----------------------|:-----------------------------------------------------------------------------------------------------------------------------|
+|`IgnoreResult(a)`       |Perform action `a` and ignore its result. `a` must not return void.                                                           |
+|`WithArg<N>(a)`         |Pass the `N`-th (0-based) argument of the mock function to action `a` and perform it.                                         |
+|`WithArgs<N1, N2, ..., Nk>(a)`|Pass the selected (0-based) arguments of the mock function to action `a` and perform it.                                      |
+|`WithoutArgs(a)`        |Perform action `a` without any arguments.                                                                                     |
+
+## Defining Actions ##
+
+| `ACTION(Sum) { return arg0 + arg1; }` | Defines an action `Sum()` to return the sum of the mock function's argument #0 and #1. |
+|:--------------------------------------|:---------------------------------------------------------------------------------------|
+| `ACTION_P(Plus, n) { return arg0 + n; }` | Defines an action `Plus(n)` to return the sum of the mock function's argument #0 and `n`. |
+| `ACTION_Pk(Foo, p1, ..., pk) { statements; }` | Defines a parameterized action `Foo(p1, ..., pk)` to execute the given `statements`.   |
+
+The `ACTION*` macros cannot be used inside a function or class.
+
+# Cardinalities #
+
+These are used in `Times()` to specify how many times a mock function will be called:
+
+|`AnyNumber()`|The function can be called any number of times.|
+|:------------|:----------------------------------------------|
+|`AtLeast(n)` |The call is expected at least `n` times.       |
+|`AtMost(n)`  |The call is expected at most `n` times.        |
+|`Between(m, n)`|The call is expected between `m` and `n` (inclusive) times.|
+|`Exactly(n) or n`|The call is expected exactly `n` times. In particular, the call should never happen when `n` is 0.|
+
+# Expectation Order #
+
+By default, the expectations can be matched in _any_ order.  If some
+or all expectations must be matched in a given order, there are two
+ways to specify it.  They can be used either independently or
+together.
+
+## The After Clause ##
+
+```
+using ::testing::Expectation;
+...
+Expectation init_x = EXPECT_CALL(foo, InitX());
+Expectation init_y = EXPECT_CALL(foo, InitY());
+EXPECT_CALL(foo, Bar())
+    .After(init_x, init_y);
+```
+says that `Bar()` can be called only after both `InitX()` and
+`InitY()` have been called.
+
+If you don't know how many pre-requisites an expectation has when you
+write it, you can use an `ExpectationSet` to collect them:
+
+```
+using ::testing::ExpectationSet;
+...
+ExpectationSet all_inits;
+for (int i = 0; i < element_count; i++) {
+  all_inits += EXPECT_CALL(foo, InitElement(i));
+}
+EXPECT_CALL(foo, Bar())
+    .After(all_inits);
+```
+says that `Bar()` can be called only after all elements have been
+initialized (but we don't care about which elements get initialized
+before the others).
+
+Modifying an `ExpectationSet` after using it in an `.After()` doesn't
+affect the meaning of the `.After()`.
+
+## Sequences ##
+
+When you have a long chain of sequential expectations, it's easier to
+specify the order using **sequences**, which don't require you to given
+each expectation in the chain a different name.  <i>All expected<br>
+calls</i> in the same sequence must occur in the order they are
+specified.
+
+```
+using ::testing::Sequence;
+Sequence s1, s2;
+...
+EXPECT_CALL(foo, Reset())
+    .InSequence(s1, s2)
+    .WillOnce(Return(true));
+EXPECT_CALL(foo, GetSize())
+    .InSequence(s1)
+    .WillOnce(Return(1));
+EXPECT_CALL(foo, Describe(A<const char*>()))
+    .InSequence(s2)
+    .WillOnce(Return("dummy"));
+```
+says that `Reset()` must be called before _both_ `GetSize()` _and_
+`Describe()`, and the latter two can occur in any order.
+
+To put many expectations in a sequence conveniently:
+```
+using ::testing::InSequence;
+{
+  InSequence dummy;
+
+  EXPECT_CALL(...)...;
+  EXPECT_CALL(...)...;
+  ...
+  EXPECT_CALL(...)...;
+}
+```
+says that all expected calls in the scope of `dummy` must occur in
+strict order. The name `dummy` is irrelevant.)
+
+# Verifying and Resetting a Mock #
+
+Google Mock will verify the expectations on a mock object when it is destructed, or you can do it earlier:
+```
+using ::testing::Mock;
+...
+// Verifies and removes the expectations on mock_obj;
+// returns true iff successful.
+Mock::VerifyAndClearExpectations(&mock_obj);
+...
+// Verifies and removes the expectations on mock_obj;
+// also removes the default actions set by ON_CALL();
+// returns true iff successful.
+Mock::VerifyAndClear(&mock_obj);
+```
+
+You can also tell Google Mock that a mock object can be leaked and doesn't
+need to be verified:
+```
+Mock::AllowLeak(&mock_obj);
+```
+
+# Mock Classes #
+
+Google Mock defines a convenient mock class template
+```
+class MockFunction<R(A1, ..., An)> {
+ public:
+  MOCK_METHODn(Call, R(A1, ..., An));
+};
+```
+See this [recipe](CookBook.md#using-check-points) for one application of it.
+
+# Flags #
+
+| `--gmock_catch_leaked_mocks=0` | Don't report leaked mock objects as failures. |
+|:-------------------------------|:----------------------------------------------|
+| `--gmock_verbose=LEVEL`        | Sets the default verbosity level (`info`, `warning`, or `error`) of Google Mock messages. |

subprojects/gtest/googlemock/docs/DesignDoc.md

@@ -0,0 +1,280 @@
+This page discusses the design of new Google Mock features.
+
+
+
+# Macros for Defining Actions #
+
+## Problem ##
+
+Due to the lack of closures in C++, it currently requires some
+non-trivial effort to define a custom action in Google Mock.  For
+example, suppose you want to "increment the value pointed to by the
+second argument of the mock function and return it", you could write:
+
+```
+int IncrementArg1(Unused, int* p, Unused) {
+  return ++(*p);
+}
+
+... WillOnce(Invoke(IncrementArg1));
+```
+
+There are several things unsatisfactory about this approach:
+
+  * Even though the action only cares about the second argument of the mock function, its definition needs to list other arguments as dummies.  This is tedious.
+  * The defined action is usable only in mock functions that takes exactly 3 arguments - an unnecessary restriction.
+  * To use the action, one has to say `Invoke(IncrementArg1)`, which isn't as nice as `IncrementArg1()`.
+
+The latter two problems can be overcome using `MakePolymorphicAction()`,
+but it requires much more boilerplate code:
+
+```
+class IncrementArg1Action {
+ public:
+  template <typename Result, typename ArgumentTuple>
+  Result Perform(const ArgumentTuple& args) const {
+    return ++(*tr1::get<1>(args));
+  }
+};
+
+PolymorphicAction<IncrementArg1Action> IncrementArg1() {
+  return MakePolymorphicAction(IncrementArg1Action());
+}
+
+... WillOnce(IncrementArg1());
+```
+
+Our goal is to allow defining custom actions with the least amount of
+boiler-plate C++ requires.
+
+## Solution ##
+
+We propose to introduce a new macro:
+```
+ACTION(name) { statements; }
+```
+
+Using this in a namespace scope will define an action with the given
+name that executes the statements.  Inside the statements, you can
+refer to the K-th (0-based) argument of the mock function as `argK`.
+For example:
+```
+ACTION(IncrementArg1) { return ++(*arg1); }
+```
+allows you to write
+```
+... WillOnce(IncrementArg1());
+```
+
+Note that you don't need to specify the types of the mock function
+arguments, as brevity is a top design goal here.  Rest assured that
+your code is still type-safe though: you'll get a compiler error if
+`*arg1` doesn't support the `++` operator, or if the type of
+`++(*arg1)` isn't compatible with the mock function's return type.
+
+Another example:
+```
+ACTION(Foo) {
+  (*arg2)(5);
+  Blah();
+  *arg1 = 0;
+  return arg0;
+}
+```
+defines an action `Foo()` that invokes argument #2 (a function pointer)
+with 5, calls function `Blah()`, sets the value pointed to by argument
+#1 to 0, and returns argument #0.
+
+For more convenience and flexibility, you can also use the following
+pre-defined symbols in the body of `ACTION`:
+
+| `argK_type` | The type of the K-th (0-based) argument of the mock function |
+|:------------|:-------------------------------------------------------------|
+| `args`      | All arguments of the mock function as a tuple                |
+| `args_type` | The type of all arguments of the mock function as a tuple    |
+| `return_type` | The return type of the mock function                         |
+| `function_type` | The type of the mock function                                |
+
+For example, when using an `ACTION` as a stub action for mock function:
+```
+int DoSomething(bool flag, int* ptr);
+```
+we have:
+| **Pre-defined Symbol** | **Is Bound To** |
+|:-----------------------|:----------------|
+| `arg0`                 | the value of `flag` |
+| `arg0_type`            | the type `bool` |
+| `arg1`                 | the value of `ptr` |
+| `arg1_type`            | the type `int*` |
+| `args`                 | the tuple `(flag, ptr)` |
+| `args_type`            | the type `std::tr1::tuple<bool, int*>` |
+| `return_type`          | the type `int`  |
+| `function_type`        | the type `int(bool, int*)` |
+
+## Parameterized actions ##
+
+Sometimes you'll want to parameterize the action.   For that we propose
+another macro
+```
+ACTION_P(name, param) { statements; }
+```
+
+For example,
+```
+ACTION_P(Add, n) { return arg0 + n; }
+```
+will allow you to write
+```
+// Returns argument #0 + 5.
+... WillOnce(Add(5));
+```
+
+For convenience, we use the term _arguments_ for the values used to
+invoke the mock function, and the term _parameters_ for the values
+used to instantiate an action.
+
+Note that you don't need to provide the type of the parameter either.
+Suppose the parameter is named `param`, you can also use the
+Google-Mock-defined symbol `param_type` to refer to the type of the
+parameter as inferred by the compiler.
+
+We will also provide `ACTION_P2`, `ACTION_P3`, and etc to support
+multi-parameter actions.  For example,
+```
+ACTION_P2(ReturnDistanceTo, x, y) {
+  double dx = arg0 - x;
+  double dy = arg1 - y;
+  return sqrt(dx*dx + dy*dy);
+}
+```
+lets you write
+```
+... WillOnce(ReturnDistanceTo(5.0, 26.5));
+```
+
+You can view `ACTION` as a degenerated parameterized action where the
+number of parameters is 0.
+
+## Advanced Usages ##
+
+### Overloading Actions ###
+
+You can easily define actions overloaded on the number of parameters:
+```
+ACTION_P(Plus, a) { ... }
+ACTION_P2(Plus, a, b) { ... }
+```
+
+### Restricting the Type of an Argument or Parameter ###
+
+For maximum brevity and reusability, the `ACTION*` macros don't let
+you specify the types of the mock function arguments and the action
+parameters.  Instead, we let the compiler infer the types for us.
+
+Sometimes, however, we may want to be more explicit about the types.
+There are several tricks to do that.  For example:
+```
+ACTION(Foo) {
+  // Makes sure arg0 can be converted to int.
+  int n = arg0;
+  ... use n instead of arg0 here ...
+}
+
+ACTION_P(Bar, param) {
+  // Makes sure the type of arg1 is const char*.
+  ::testing::StaticAssertTypeEq<const char*, arg1_type>();
+
+  // Makes sure param can be converted to bool.
+  bool flag = param;
+}
+```
+where `StaticAssertTypeEq` is a compile-time assertion we plan to add to
+Google Test (the name is chosen to match `static_assert` in C++0x).
+
+### Using the ACTION Object's Type ###
+
+If you are writing a function that returns an `ACTION` object, you'll
+need to know its type.  The type depends on the macro used to define
+the action and the parameter types.  The rule is relatively simple:
+| **Given Definition** | **Expression** | **Has Type** |
+|:---------------------|:---------------|:-------------|
+| `ACTION(Foo)`        | `Foo()`        | `FooAction`  |
+| `ACTION_P(Bar, param)` | `Bar(int_value)` | `BarActionP<int>` |
+| `ACTION_P2(Baz, p1, p2)` | `Baz(bool_value, int_value)` | `BazActionP2<bool, int>` |
+| ...                  | ...            | ...          |
+
+Note that we have to pick different suffixes (`Action`, `ActionP`,
+`ActionP2`, and etc) for actions with different numbers of parameters,
+or the action definitions cannot be overloaded on the number of
+parameters.
+
+## When to Use ##
+
+While the new macros are very convenient, please also consider other
+means of implementing actions (e.g. via `ActionInterface` or
+`MakePolymorphicAction()`), especially if you need to use the defined
+action a lot.  While the other approaches require more work, they give
+you more control on the types of the mock function arguments and the
+action parameters, which in general leads to better compiler error
+messages that pay off in the long run.  They also allow overloading
+actions based on parameter types, as opposed to just the number of
+parameters.
+
+## Related Work ##
+
+As you may have realized, the `ACTION*` macros resemble closures (also
+known as lambda expressions or anonymous functions).  Indeed, both of
+them seek to lower the syntactic overhead for defining a function.
+
+C++0x will support lambdas, but they are not part of C++ right now.
+Some non-standard libraries (most notably BLL or Boost Lambda Library)
+try to alleviate this problem.  However, they are not a good choice
+for defining actions as:
+
+  * They are non-standard and not widely installed.  Google Mock only depends on standard libraries and `tr1::tuple`, which is part of the new C++ standard and comes with gcc 4+.  We want to keep it that way.
+  * They are not trivial to learn.
+  * They will become obsolete when C++0x's lambda feature is widely supported.  We don't want to make our users use a dying library.
+  * Since they are based on operators, they are rather ad hoc: you cannot use statements, and you cannot pass the lambda arguments to a function, for example.
+  * They have subtle semantics that easily confuses new users.  For example, in expression `_1++ + foo++`, `foo` will be incremented only once where the expression is evaluated, while `_1` will be incremented every time the unnamed function is invoked.  This is far from intuitive.
+
+`ACTION*` avoid all these problems.
+
+## Future Improvements ##
+
+There may be a need for composing `ACTION*` definitions (i.e. invoking
+another `ACTION` inside the definition of one `ACTION*`).  We are not
+sure we want it yet, as one can get a similar effect by putting
+`ACTION` definitions in function templates and composing the function
+templates.  We'll revisit this based on user feedback.
+
+The reason we don't allow `ACTION*()` inside a function body is that
+the current C++ standard doesn't allow function-local types to be used
+to instantiate templates.  The upcoming C++0x standard will lift this
+restriction.  Once this feature is widely supported by compilers, we
+can revisit the implementation and add support for using `ACTION*()`
+inside a function.
+
+C++0x will also support lambda expressions.  When they become
+available, we may want to support using lambdas as actions.
+
+# Macros for Defining Matchers #
+
+Once the macros for defining actions are implemented, we plan to do
+the same for matchers:
+
+```
+MATCHER(name) { statements; }
+```
+
+where you can refer to the value being matched as `arg`.  For example,
+given:
+
+```
+MATCHER(IsPositive) { return arg > 0; }
+```
+
+you can use `IsPositive()` as a matcher that matches a value iff it is
+greater than 0.
+
+We will also add `MATCHER_P`, `MATCHER_P2`, and etc for parameterized
+matchers.

subprojects/gtest/googlemock/docs/DevGuide.md

@@ -0,0 +1,132 @@
+
+
+If you are interested in understanding the internals of Google Mock,
+building from source, or contributing ideas or modifications to the
+project, then this document is for you.
+
+# Introduction #
+
+First, let's give you some background of the project.
+
+## Licensing ##
+
+All Google Mock source and pre-built packages are provided under the [New BSD License](http://www.opensource.org/licenses/bsd-license.php).
+
+## The Google Mock Community ##
+
+The Google Mock community exists primarily through the [discussion group](http://groups.google.com/group/googlemock), the
+[issue tracker](https://github.com/google/googletest/issues) and, to a lesser extent, the [source control repository](../). You are definitely encouraged to contribute to the
+discussion and you can also help us to keep the effectiveness of the
+group high by following and promoting the guidelines listed here.
+
+### Please Be Friendly ###
+
+Showing courtesy and respect to others is a vital part of the Google
+culture, and we strongly encourage everyone participating in Google
+Mock development to join us in accepting nothing less. Of course,
+being courteous is not the same as failing to constructively disagree
+with each other, but it does mean that we should be respectful of each
+other when enumerating the 42 technical reasons that a particular
+proposal may not be the best choice. There's never a reason to be
+antagonistic or dismissive toward anyone who is sincerely trying to
+contribute to a discussion.
+
+Sure, C++ testing is serious business and all that, but it's also
+a lot of fun. Let's keep it that way. Let's strive to be one of the
+friendliest communities in all of open source.
+
+### Where to Discuss Google Mock ###
+
+As always, discuss Google Mock in the official [Google C++ Mocking Framework discussion group](http://groups.google.com/group/googlemock).  You don't have to actually submit
+code in order to sign up. Your participation itself is a valuable
+contribution.
+
+# Working with the Code #
+
+If you want to get your hands dirty with the code inside Google Mock,
+this is the section for you.
+
+## Checking Out the Source from Subversion ##
+
+Checking out the Google Mock source is most useful if you plan to
+tweak it yourself.  You check out the source for Google Mock using a
+[Subversion](http://subversion.tigris.org/) client as you would for any
+other project hosted on Google Code.  Please see the instruction on
+the [source code access page](../) for how to do it.
+
+## Compiling from Source ##
+
+Once you check out the code, you can find instructions on how to
+compile it in the [README](../README.md) file.
+
+## Testing ##
+
+A mocking framework is of no good if itself is not thoroughly tested.
+Tests should be written for any new code, and changes should be
+verified to not break existing tests before they are submitted for
+review. To perform the tests, follow the instructions in [README](http://code.google.com/p/googlemock/source/browse/trunk/README) and
+verify that there are no failures.
+
+# Contributing Code #
+
+We are excited that Google Mock is now open source, and hope to get
+great patches from the community. Before you fire up your favorite IDE
+and begin hammering away at that new feature, though, please take the
+time to read this section and understand the process. While it seems
+rigorous, we want to keep a high standard of quality in the code
+base.
+
+## Contributor License Agreements ##
+
+You must sign a Contributor License Agreement (CLA) before we can
+accept any code.  The CLA protects you and us.
+
+  * If you are an individual writing original source code and you're sure you own the intellectual property, then you'll need to sign an [individual CLA](http://code.google.com/legal/individual-cla-v1.0.html).
+  * If you work for a company that wants to allow you to contribute your work to Google Mock, then you'll need to sign a [corporate CLA](http://code.google.com/legal/corporate-cla-v1.0.html).
+
+Follow either of the two links above to access the appropriate CLA and
+instructions for how to sign and return it.
+
+## Coding Style ##
+
+To keep the source consistent, readable, diffable and easy to merge,
+we use a fairly rigid coding style, as defined by the [google-styleguide](https://github.com/google/styleguide) project.  All patches will be expected
+to conform to the style outlined [here](https://github.com/google/styleguide/blob/gh-pages/cppguide.xml).
+
+## Submitting Patches ##
+
+Please do submit code. Here's what you need to do:
+
+  1. Normally you should make your change against the SVN trunk instead of a branch or a tag, as the latter two are for release control and should be treated mostly as read-only.
+  1. Decide which code you want to submit. A submission should be a set of changes that addresses one issue in the [Google Mock issue tracker](http://code.google.com/p/googlemock/issues/list). Please don't mix more than one logical change per submittal, because it makes the history hard to follow. If you want to make a change that doesn't have a corresponding issue in the issue tracker, please create one.
+  1. Also, coordinate with team members that are listed on the issue in question. This ensures that work isn't being duplicated and communicating your plan early also generally leads to better patches.
+  1. Ensure that your code adheres to the [Google Mock source code style](#Coding_Style.md).
+  1. Ensure that there are unit tests for your code.
+  1. Sign a Contributor License Agreement.
+  1. Create a patch file using `svn diff`.
+  1. We use [Rietveld](http://codereview.appspot.com/) to do web-based code reviews.  You can read about the tool [here](https://github.com/rietveld-codereview/rietveld/wiki).  When you are ready, upload your patch via Rietveld and notify `googlemock@googlegroups.com` to review it.  There are several ways to upload the patch.  We recommend using the [upload\_gmock.py](../scripts/upload_gmock.py) script, which you can find in the `scripts/` folder in the SVN trunk.
+
+## Google Mock Committers ##
+
+The current members of the Google Mock engineering team are the only
+committers at present. In the great tradition of eating one's own
+dogfood, we will be requiring each new Google Mock engineering team
+member to earn the right to become a committer by following the
+procedures in this document, writing consistently great code, and
+demonstrating repeatedly that he or she truly gets the zen of Google
+Mock.
+
+# Release Process #
+
+We follow the typical release process for Subversion-based projects:
+
+  1. A release branch named `release-X.Y` is created.
+  1. Bugs are fixed and features are added in trunk; those individual patches are merged into the release branch until it's stable.
+  1. An individual point release (the `Z` in `X.Y.Z`) is made by creating a tag from the branch.
+  1. Repeat steps 2 and 3 throughout one release cycle (as determined by features or time).
+  1. Go back to step 1 to create another release branch and so on.
+
+
+---
+
+This page is based on the [Making GWT Better](http://code.google.com/webtoolkit/makinggwtbetter.html) guide from the [Google Web Toolkit](http://code.google.com/webtoolkit/) project.  Except as otherwise [noted](http://code.google.com/policies.html#restrictions), the content of this page is licensed under the [Creative Commons Attribution 2.5 License](http://creativecommons.org/licenses/by/2.5/).