Merge pull request #243 from kiwix/new_version

New version 5.2.0

.gitignore

@@ -0,0 +1 @@
+.idea/

.travis.yml

@@ -1,16 +1,12 @@
 language: cpp
-dist: trusty
-sudo: false
+dist: xenial
+sudo: true
 cache: ccache
 before_install:
-- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then eval "${MATRIX_EVAL}"; fi
 - PATH=$PATH:$HOME/bin
-- ${CXX} --version
 install: travis/install_deps.sh
 script: travis/compile.sh
 env:
-  global:
-    - MATRIX_EVAL="CC=gcc-5 && CXX=g++-5"
   matrix:
     - PLATFORM="native_static"
     - PLATFORM="native_dyn"
@@ -20,11 +16,9 @@ env:
     - PLATFORM="android_arm64"
 addons:
   apt:
-    sources:
-      - ubuntu-toolchain-r-test
     packages:
-      - g++-5
       - cmake
+      - python3.5
       - python3-pip
       - libbz2-dev
       - ccache

ChangeLog

@@ -1,3 +1,151 @@
+kiwix-lib 5.2.0
+===============
+
+ * kiwix-serve integration (as a seperated process).
+ * Fix crash in the suggestion search.
+ * Better API to filter the library books.
+ * New kiwix-lib application for android. (.aar)
+ * Use ReLinker to link with libkiwix.so in android.
+ * Correctly set the verbosity of zim search.
+
+kiwix-lib 5.1.0
+===============
+
+ * Add function to pause, resume and stop downloads.
+ * Add zim's tags in the opds stream.
+ * Addapt to new libzim 5.0.0 API.
+
+kiwix-lib 5.0.0
+===============
+
+ * Remove error message when trying to open an wrong zim file.
+ * Rewrite `makeTmpDirectory` to not use uuid functions on windows.
+ * [API break] Remove `getNetworkInterfaces` and `getBestPublicIp`.
+ * Remove rpath
+ * Detect infinite (and too long) redirect loops.
+
+kiwix-lib 4.1.0
+===============
+
+ * Allow the library to be filtered by tags.
+ * Fix language mapping.
+ * Update README about mustache dependency.
+
+kiwix-lib 4.0.1
+===============
+
+ * Fix "maybe uninitialize variable" issue.
+ * Ensure path are stored correctly (absolute path) in the library.
+ * [CI] Use the new deps archive xz
+
+kiwix-lib 4.0.0
+===============
+
+ * [API break] Remove support for external index.
+ * Move to the mustache templating system instead of ctpp2.
+ * Make meson.build works for meson>=0.43.0
+ * [API break] Move the basic tools from the `common` directory to `tools`.
+
+kiwix-lib 3.1.1
+===============
+
+ * The OPDS feed book's date must be the date of the book, not the date of the
+ feed generation.
+ * Convert the standard opds date to our format (YYYY-MM-DD)
+ * Remove duplicate language attribute in the libxml dumper.
+ * Create the datadirectory to not fail to write a file in a non-existent
+   directory
+
+kiwix-lib 3.1.0
+===============
+
+ * Add a method to get the favicon url of book (if available).
+ * Move dump code of library.xml in a specific class.
+ * Add a first support to bookmarks
+
+kiwix-lib 3.0.3
+===============
+
+ * Add the 'en' language to the mapping alpha2-code ('en') to alpha3-code
+ ('eng').
+ * Correctly write the 'ArticleCount' and 'MediaCount' in the library.xml.
+ * Correctly fill the book size for the zim file size.
+ * Fix launch of aria2c.
+
+kiwix-lib 3.0.2
+===============
+
+ * Use the correct path separator when computing relativePath on Windows.
+
+kiwix-lib 3.0.1
+===============
+
+ * Small fix about parsing the opdsStream.
+
+kiwix-lib 3.0.0
+===============
+
+ * Change the downloader to use aria2 using a separated process (with rpc)
+   instead of using the libaria2. This simplify a lot the link process to
+   libaria2 on Windows.
+    - kiwix-lib doesn't depend on libaria2 anymore.
+    - kiwix-lib now depends on libcurl.
+ * [API break] Library class API has been updated :
+    - Books are referenced by id, not index. A lot of methods have been
+      updated this way.
+    - Books "list" is now private.
+    - There is no more "current" book.
+    - listBooksIds's filters have been updated.
+ * [API break] Book class API has been updated :
+    - Move the definition of Book in `book.h`.
+    - Use getter/setter methods instead public members.
+    - Size (getSize/setSize) is now returned in bytes, not kB.
+    - Dependending of how the book has been initialized (opdsfeed), the
+      faviconUrl may be stored in the book, the favicon being downloaded when
+      using `getFavicon`.
+    - The path (and indexPath) are always absolute path.
+    - Book has now a downloadId, corresponding to the aria2 download id (if
+      exists)
+  * [API break] Manager class API has been updated :
+    - The manager is mainly use to fill a Libray from a "library.xml" file or
+      opds feed. Other operations (has removeBookById, setBookPath, filter, ...)
+      have been removed.
+    - The manager use a intermediate class (LibraryManipulator) to add book to
+      the library. This dependency injection allow caller code to hook the add
+      of a book to the library.
+    - The manager work on a existing Library. It doesn't how a internal
+      Library.
+  * [API break] OpdsDumper class API has been updated :
+    - dumpOPDSFeed method now take the list of bookIds to dump instead of
+      dumping all books in the library.
+    - OpdsDumper can now dump openSearch result information (total result
+      count, start index, ...).
+  * [API break] Common tools API has been updated :
+    - `base64_encode` and `base64_decode` take std::string as arguments.
+    - New `download` function in networkTools.h using libcurl.
+    - New `getDataDirectory` function in pathTools.
+    - Better `beautifyInteger` and `beautifyFileSize` functions.
+    - New `nodeToString` function serializing a pugi::xml_node to a string.
+    - New `converta2toa3` function to convert alpha2 language code to aplha3
+      language code.
+
+
+kiwix-lib 2.0.2
+===============
+
+ * [Android] Forward c++ errors message de Java world.
+ * Follow redirection of favicon.
+ * Make aria2 dependency optional.
+ * Inculde unistd.h only on unix platform.
+
+kiwix-lib 2.0.1
+===============
+
+ * Fix parsing of url.
+ * Remove unused static resources.
+ * Correctly decode reserved characters in URLs.
+ * Explicitly use icu namespace to allow use of packaged icu lib.
+
 kiwix-lib 2.0.0
 ===============
 

README.md

@@ -15,9 +15,9 @@ to [kiwix-build](https://github.com/kiwix/kiwix-build).
 Preamble
 --------
 
-Although the Kiwix library can be compiled/cross-compiled on/for many
+Although the Kiwix library can be (cross-)compiled on/for many
 sytems, the following documentation explains how to do it on POSIX
-ones. It is primarly though for GNU/Linux systems and has been tested
+ones. It is primarly thought for GNU/Linux systems and has been tested
 on recent releases of Ubuntu and Fedora.
 
 Dependencies
@@ -27,17 +27,13 @@ The Kiwix library relies on many third parts software libraries. They
 are prerequisites to the Kiwix library compilation. Following
 libraries need to be available:
 
-* ICU ................................... http://site.icu-project.org/
-(package libicu-dev on Ubuntu)
-* ZIM ........................................ http://www.openzim.org/
-(package libzim-dev on Ubuntu)
-* Pugixml ........................................ http://pugixml.org/
-(package libpugixml-dev on Ubuntu)
-* ctpp2 ........................................ http://ctpp.havoc.ru/
-(package libctpp2-dev on Ubuntu)
-* Xapian ......................................... https://xapian.org/
-(package libxapian-dev on Ubuntu)
-* libaria2 .................................. https://aria2.github.io/
+* [ICU](https://site.icu-project.org/) (package `libicu-dev` on Ubuntu)
+* [ZIM](https://openzim.org/) (package `libzim-dev` on Ubuntu)
+* [Pugixml](https://pugixml.org/) (package `libpugixml-dev` on Ubuntu)
+* [Aria2](https://aria2.github.io/) (package `aria2` on Ubuntu)
+* [Mustache](https://github.com/kainjow/Mustache) (Just copy the
+header `mustache.hpp` somewhere it can be found by the compiler and/or
+set CPPFLAGS with correct `-I` option)
 
 These dependencies may or may not be packaged by your operating
 system. They may also be packaged but only in an older version. The
@@ -46,74 +42,97 @@ In the worse case, you will have to download and compile bleeding edge
 version by hand.
 
 If you want to install these dependencies locally, then use the
-kiwix-lib directory as install prefix.
+`kiwix-lib` directory as install prefix.
 
-If you compile ctpp2 from source and want to compile the Kiwix library
-statically then you will probably need to rename ctpp2 static library
-from ctpp2-st.a to ctpp2.a.
-
-Environnement
+Environment
 -------------
 
-The Kiwix library builds using [Meson](http://mesonbuild.com/) version
-0.34 or higher. Meson relies itself on Ninja, pkg-config and few other
+The Kiwix library builds using [Meson](https://mesonbuild.com/) version
+0.43 or higher. Meson relies itself on Ninja, pkg-config and few other
 compilation tools.
 
 Install first the few common compilation tools:
-* Automake
-* Libtool
-* Virtualenv
+* Meson
+* Ninja
 * Pkg-config
 
-Then install Meson itself:
-```
-virtualenv -p python3 ./ # Create virtualenv
-source bin/activate      # Activate the virtualenv
-pip3 install meson        # Install Meson
-hash -r                  # Refresh bash paths
-```
-
-Finally download and build Ninja locally:
-```
-git clone git://github.com/ninja-build/ninja.git
-cd ninja
-git checkout release
-./configure.py --bootstrap
-mkdir ../bin                      
-cp ninja ../bin
-cd ..
-```
+These tools should be packaged if you use a cutting edge operating
+system. If not, have a look to the [Troubleshooting](#Troubleshooting)
+section.
 
 Compilation
 -----------
 
-Once all dependencies are installed, you can compile kiwix-lib with:
-```
-mkdir build
+Once all dependencies are installed, you can compile the Kiwix library
+with:
+
+```bash
 meson . build
-cd build
-ninja
+ninja -C build
 ```
 
-By default, it will compile dynamic linked libraries. If you want
-statically linked libraries, you can add `--default-library=static`
-option to the Meson command.
+By default, it will compile dynamic linked libraries. All binary files
+will be created in the "build" directory created automatically by
+Meson. If you want statically linked libraries, you can add
+`--default-library=static` option to the Meson command.
 
 Depending of you system, `ninja` may be called `ninja-build`.
 
 Installation
 ------------
 
-If you want to install the libraries you just have compiled on your
-system, here we go:
+If you want to install the Kiwix library and the headers you just have
+compiled on your system, here we go:
 
+```bash
+ninja -C build install
 ```
-ninja install
+
+You might need to run the command as root (or using `sudo`), depending
+where you want to install the libraries. After the installation
+succeeded, you may need to run `ldconfig` (as root).
+
+Uninstallation
+------------
+
+If you want to uninstall the Kiwix library:
+
+```bash
+ninja -C build uninstall
+```
+
+Like for the installation, you might need to run the command as root
+(or using `sudo`).
+
+Troubleshooting
+---------------
+
+If you need to install Meson "manually":
+
+```bash
+virtualenv -p python3 ./ # Create virtualenv
+source bin/activate      # Activate the virtualenv
+pip3 install meson       # Install Meson
+hash -r                  # Refresh bash paths
+```
+
+If you need to install Ninja "manually":
+
+```bash
+git clone git://github.com/ninja-build/ninja.git
+cd ninja
+git checkout release
+./configure.py --bootstrap
+mkdir ../bin
+cp ninja ../bin
 cd ..
 ```
 
-You might need to run the command as root, depending where you want to
-install the libraries.
+If the compilation still fails, you might need to get a more recent
+version of a dependency than the one packaged by your Linux
+distribution. Try then with a source tarball distributed by the
+problematic upstream project or even directly from the source code
+repository.
 
 License
 -------

android-kiwix-lib-publisher/.gitignore

@@ -0,0 +1,13 @@
+*.iml
+.gradle
+/local.properties
+/.idea/caches
+/.idea/libraries
+/.idea/modules.xml
+/.idea/workspace.xml
+/.idea/navEditor.xml
+/.idea/assetWizardSettings.xml
+.DS_Store
+/build
+/captures
+.externalNativeBuild

android-kiwix-lib-publisher/build.gradle

@@ -0,0 +1,25 @@
+// Top-level build file where you can add configuration options common to all sub-projects/modules.
+
+buildscript {
+    repositories {
+        google()
+        jcenter()
+
+    }
+    dependencies {
+        classpath 'com.android.tools.build:gradle:3.4.1'
+        // NOTE: Do not place your application dependencies here; they belong
+        // in the individual module build.gradle files
+    }
+}
+
+allprojects {
+    repositories {
+        google()
+        jcenter()
+    }
+}
+
+task clean(type: Delete) {
+    delete rootProject.buildDir
+}

android-kiwix-lib-publisher/gradle.properties

@@ -0,0 +1,15 @@
+# Project-wide Gradle settings.
+# IDE (e.g. Android Studio) users:
+# Gradle settings configured through the IDE *will override*
+# any settings specified in this file.
+# For more details on how to configure your build environment visit
+# http://www.gradle.org/docs/current/userguide/build_environment.html
+# Specifies the JVM arguments used for the daemon process.
+# The setting is particularly useful for tweaking memory settings.
+org.gradle.jvmargs=-Xmx1536m
+# When configured, Gradle will run in incubating parallel mode.
+# This option should only be used with decoupled projects. More details, visit
+# http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:decoupled_projects
+# org.gradle.parallel=true
+# Kotlin code style for this project: "official" or "obsolete":
+kotlin.code.style=official

android-kiwix-lib-publisher/gradle/wrapper/gradle-wrapper.properties

@@ -0,0 +1,6 @@
+#Wed Jun 19 15:28:39 BST 2019
+distributionBase=GRADLE_USER_HOME
+distributionPath=wrapper/dists
+zipStoreBase=GRADLE_USER_HOME
+zipStorePath=wrapper/dists
+distributionUrl=https\://services.gradle.org/distributions/gradle-5.1.1-all.zip

android-kiwix-lib-publisher/gradlew

@@ -0,0 +1,172 @@
+#!/usr/bin/env sh
+
+##############################################################################
+##
+##  Gradle start up script for UN*X
+##
+##############################################################################
+
+# Attempt to set APP_HOME
+# Resolve links: $0 may be a link
+PRG="$0"
+# Need this for relative symlinks.
+while [ -h "$PRG" ] ; do
+    ls=`ls -ld "$PRG"`
+    link=`expr "$ls" : '.*-> \(.*\)$'`
+    if expr "$link" : '/.*' > /dev/null; then
+        PRG="$link"
+    else
+        PRG=`dirname "$PRG"`"/$link"
+    fi
+done
+SAVED="`pwd`"
+cd "`dirname \"$PRG\"`/" >/dev/null
+APP_HOME="`pwd -P`"
+cd "$SAVED" >/dev/null
+
+APP_NAME="Gradle"
+APP_BASE_NAME=`basename "$0"`
+
+# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
+DEFAULT_JVM_OPTS=""
+
+# Use the maximum available, or set MAX_FD != -1 to use that value.
+MAX_FD="maximum"
+
+warn () {
+    echo "$*"
+}
+
+die () {
+    echo
+    echo "$*"
+    echo
+    exit 1
+}
+
+# OS specific support (must be 'true' or 'false').
+cygwin=false
+msys=false
+darwin=false
+nonstop=false
+case "`uname`" in
+  CYGWIN* )
+    cygwin=true
+    ;;
+  Darwin* )
+    darwin=true
+    ;;
+  MINGW* )
+    msys=true
+    ;;
+  NONSTOP* )
+    nonstop=true
+    ;;
+esac
+
+CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
+
+# Determine the Java command to use to start the JVM.
+if [ -n "$JAVA_HOME" ] ; then
+    if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
+        # IBM's JDK on AIX uses strange locations for the executables
+        JAVACMD="$JAVA_HOME/jre/sh/java"
+    else
+        JAVACMD="$JAVA_HOME/bin/java"
+    fi
+    if [ ! -x "$JAVACMD" ] ; then
+        die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
+
+Please set the JAVA_HOME variable in your environment to match the
+location of your Java installation."
+    fi
+else
+    JAVACMD="java"
+    which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
+
+Please set the JAVA_HOME variable in your environment to match the
+location of your Java installation."
+fi
+
+# Increase the maximum file descriptors if we can.
+if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then
+    MAX_FD_LIMIT=`ulimit -H -n`
+    if [ $? -eq 0 ] ; then
+        if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then
+            MAX_FD="$MAX_FD_LIMIT"
+        fi
+        ulimit -n $MAX_FD
+        if [ $? -ne 0 ] ; then
+            warn "Could not set maximum file descriptor limit: $MAX_FD"
+        fi
+    else
+        warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT"
+    fi
+fi
+
+# For Darwin, add options to specify how the application appears in the dock
+if $darwin; then
+    GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\""
+fi
+
+# For Cygwin, switch paths to Windows format before running java
+if $cygwin ; then
+    APP_HOME=`cygpath --path --mixed "$APP_HOME"`
+    CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
+    JAVACMD=`cygpath --unix "$JAVACMD"`
+
+    # We build the pattern for arguments to be converted via cygpath
+    ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`
+    SEP=""
+    for dir in $ROOTDIRSRAW ; do
+        ROOTDIRS="$ROOTDIRS$SEP$dir"
+        SEP="|"
+    done
+    OURCYGPATTERN="(^($ROOTDIRS))"
+    # Add a user-defined pattern to the cygpath arguments
+    if [ "$GRADLE_CYGPATTERN" != "" ] ; then
+        OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)"
+    fi
+    # Now convert the arguments - kludge to limit ourselves to /bin/sh
+    i=0
+    for arg in "$@" ; do
+        CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -`
+        CHECK2=`echo "$arg"|egrep -c "^-"`                                 ### Determine if an option
+
+        if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then                    ### Added a condition
+            eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"`
+        else
+            eval `echo args$i`="\"$arg\""
+        fi
+        i=$((i+1))
+    done
+    case $i in
+        (0) set -- ;;
+        (1) set -- "$args0" ;;
+        (2) set -- "$args0" "$args1" ;;
+        (3) set -- "$args0" "$args1" "$args2" ;;
+        (4) set -- "$args0" "$args1" "$args2" "$args3" ;;
+        (5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
+        (6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
+        (7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
+        (8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
+        (9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
+    esac
+fi
+
+# Escape application args
+save () {
+    for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
+    echo " "
+}
+APP_ARGS=$(save "$@")
+
+# Collect all arguments for the java command, following the shell quoting and substitution rules
+eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"
+
+# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong
+if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then
+  cd "$(dirname "$0")"
+fi
+
+exec "$JAVACMD" "$@"

android-kiwix-lib-publisher/gradlew.bat

@@ -0,0 +1,84 @@
+@if "%DEBUG%" == "" @echo off
+@rem ##########################################################################
+@rem
+@rem  Gradle startup script for Windows
+@rem
+@rem ##########################################################################
+
+@rem Set local scope for the variables with windows NT shell
+if "%OS%"=="Windows_NT" setlocal
+
+set DIRNAME=%~dp0
+if "%DIRNAME%" == "" set DIRNAME=.
+set APP_BASE_NAME=%~n0
+set APP_HOME=%DIRNAME%
+
+@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
+set DEFAULT_JVM_OPTS=
+
+@rem Find java.exe
+if defined JAVA_HOME goto findJavaFromJavaHome
+
+set JAVA_EXE=java.exe
+%JAVA_EXE% -version >NUL 2>&1
+if "%ERRORLEVEL%" == "0" goto init
+
+echo.
+echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
+echo.
+echo Please set the JAVA_HOME variable in your environment to match the
+echo location of your Java installation.
+
+goto fail
+
+:findJavaFromJavaHome
+set JAVA_HOME=%JAVA_HOME:"=%
+set JAVA_EXE=%JAVA_HOME%/bin/java.exe
+
+if exist "%JAVA_EXE%" goto init
+
+echo.
+echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
+echo.
+echo Please set the JAVA_HOME variable in your environment to match the
+echo location of your Java installation.
+
+goto fail
+
+:init
+@rem Get command-line arguments, handling Windows variants
+
+if not "%OS%" == "Windows_NT" goto win9xME_args
+
+:win9xME_args
+@rem Slurp the command line arguments.
+set CMD_LINE_ARGS=
+set _SKIP=2
+
+:win9xME_args_slurp
+if "x%~1" == "x" goto execute
+
+set CMD_LINE_ARGS=%*
+
+:execute
+@rem Setup the command line
+
+set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
+
+@rem Execute Gradle
+"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS%
+
+:end
+@rem End local scope for the variables with windows NT shell
+if "%ERRORLEVEL%"=="0" goto mainEnd
+
+:fail
+rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
+rem the _cmd.exe /c_ return code!
+if  not "" == "%GRADLE_EXIT_CONSOLE%" exit 1
+exit /b 1
+
+:mainEnd
+if "%OS%"=="Windows_NT" endlocal
+
+:omega

android-kiwix-lib-publisher/kiwixLibAndroid/.gitignore

@@ -0,0 +1 @@
+/build

android-kiwix-lib-publisher/kiwixLibAndroid/build.gradle

@@ -0,0 +1,21 @@
+apply plugin: 'com.android.library'
+
+android {
+    compileSdkVersion 28
+    defaultConfig {
+        minSdkVersion 15
+        targetSdkVersion 28
+        versionCode 1
+        versionName "1.0"
+    }
+    buildTypes {
+        release {
+            minifyEnabled false
+            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
+        }
+    }
+}
+
+dependencies {
+    implementation 'com.getkeepsafe.relinker:relinker:1.3.1'
+}

android-kiwix-lib-publisher/kiwixLibAndroid/proguard-rules.pro

@@ -0,0 +1,21 @@
+# Add project specific ProGuard rules here.
+# You can control the set of applied configuration files using the
+# proguardFiles setting in build.gradle.
+#
+# For more details, see
+#   http://developer.android.com/guide/developing/tools/proguard.html
+
+# If your project uses WebView with JS, uncomment the following
+# and specify the fully qualified class name to the JavaScript interface
+# class:
+#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
+#   public *;
+#}
+
+# Uncomment this to preserve the line number information for
+# debugging stack traces.
+#-keepattributes SourceFile,LineNumberTable
+
+# If you keep the line number information, uncomment this to
+# hide the original source file name.
+#-renamesourcefileattribute SourceFile

android-kiwix-lib-publisher/kiwixLibAndroid/src/main/AndroidManifest.xml

@@ -0,0 +1,10 @@
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+    package="kiwix.org.kiwixlib">
+
+    <application
+        android:allowBackup="true"
+        android:supportsRtl="true">
+
+    </application>
+
+</manifest>

android-kiwix-lib-publisher/settings.gradle

@@ -0,0 +1 @@
+include ':kiwixLibAndroid'

include/book.h

@@ -0,0 +1,119 @@
+/*
+ * Copyright 2011 Emmanuel Engelhart <kelson@kiwix.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#ifndef KIWIX_BOOK_H
+#define KIWIX_BOOK_H
+
+#include <string>
+
+namespace pugi {
+class xml_node;
+}
+
+namespace kiwix
+{
+
+class OPDSDumper;
+class Reader;
+
+/**
+ * A class to store information about a book (a zim file)
+ */
+class Book
+{
+ public:
+  Book();
+  ~Book();
+
+  bool update(const Book& other);
+  void update(const Reader& reader);
+  void updateFromXml(const pugi::xml_node& node, const std::string& baseDir);
+  void updateFromOpds(const pugi::xml_node& node, const std::string& urlHost);
+  std::string getHumanReadableIdFromPath();
+
+  bool readOnly() const { return m_readOnly; }
+  const std::string& getId() const { return m_id; }
+  const std::string& getPath() const { return m_path; }
+  bool isPathValid() const { return m_pathValid; }
+  const std::string& getTitle() const { return m_title; }
+  const std::string& getDescription() const { return m_description; }
+  const std::string& getLanguage() const { return m_language; }
+  const std::string& getCreator() const { return m_creator; }
+  const std::string& getPublisher() const { return m_publisher; }
+  const std::string& getDate() const { return m_date; }
+  const std::string& getUrl() const { return m_url; }
+  const std::string& getName() const { return m_name; }
+  const std::string& getTags() const { return m_tags; }
+  const std::string& getOrigId() const { return m_origId; }
+  const uint64_t& getArticleCount() const { return m_articleCount; }
+  const uint64_t& getMediaCount() const { return m_mediaCount; }
+  const uint64_t& getSize() const { return m_size; }
+  const std::string& getFavicon() const;
+  const std::string& getFaviconUrl() const { return m_faviconUrl; }
+  const std::string& getFaviconMimeType() const { return m_faviconMimeType; }
+  const std::string& getDownloadId() const { return m_downloadId; }
+
+  void setReadOnly(bool readOnly) { m_readOnly = readOnly; }
+  void setId(const std::string& id) { m_id = id; }
+  void setPath(const std::string& path);
+  void setPathValid(bool valid) { m_pathValid = valid; }
+  void setTitle(const std::string& title) { m_title = title; }
+  void setDescription(const std::string& description) { m_description = description; }
+  void setLanguage(const std::string& language) { m_language = language; }
+  void setCreator(const std::string& creator) { m_creator = creator; }
+  void setPublisher(const std::string& publisher) { m_publisher = publisher; }
+  void setDate(const std::string& date) { m_date = date; }
+  void setUrl(const std::string& url) { m_url = url; }
+  void setName(const std::string& name) { m_name = name; }
+  void setTags(const std::string& tags) { m_tags = tags; }
+  void setOrigId(const std::string& origId) { m_origId = origId; }
+  void setArticleCount(uint64_t articleCount) { m_articleCount = articleCount; }
+  void setMediaCount(uint64_t mediaCount) { m_mediaCount = mediaCount; }
+  void setSize(uint64_t size) { m_size = size; }
+  void setFavicon(const std::string& favicon) { m_favicon = favicon; }
+  void setFaviconMimeType(const std::string& faviconMimeType) { m_faviconMimeType = faviconMimeType; }
+  void setDownloadId(const std::string& downloadId) { m_downloadId = downloadId; }
+
+ protected:
+  std::string m_id;
+  std::string m_downloadId;
+  std::string m_path;
+  bool m_pathValid;
+  std::string m_title;
+  std::string m_description;
+  std::string m_language;
+  std::string m_creator;
+  std::string m_publisher;
+  std::string m_date;
+  std::string m_url;
+  std::string m_name;
+  std::string m_tags;
+  std::string m_origId;
+  uint64_t m_articleCount;
+  uint64_t m_mediaCount;
+  bool m_readOnly;
+  uint64_t m_size;
+  mutable std::string m_favicon;
+  std::string m_faviconUrl;
+  std::string m_faviconMimeType;
+};
+
+}
+
+#endif

include/bookmark.h

@@ -0,0 +1,68 @@
+/*
+ * Copyright 2018 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#ifndef KIWIX_BOOKMARK_H
+#define KIWIX_BOOKMARK_H
+
+#include <string>
+
+namespace pugi {
+class xml_node;
+}
+
+namespace kiwix
+{
+
+/**
+ * A class to store information about a bookmark (an article in a book)
+ */
+class Bookmark
+{
+ public:
+  Bookmark();
+  ~Bookmark();
+
+  void updateFromXml(const pugi::xml_node& node);
+
+  const std::string& getBookId() const { return m_bookId; }
+  const std::string& getBookTitle()  const { return m_bookTitle; }
+  const std::string& getUrl() const { return m_url; }
+  const std::string& getTitle() const { return m_title; }
+  const std::string& getLanguage() const { return m_language; }
+  const std::string& getDate() const { return m_date; }
+
+  void setBookId(const std::string& bookId) { m_bookId = bookId; }
+  void setBookTitle(const std::string& bookTitle) { m_bookTitle = bookTitle; }
+  void setUrl(const std::string& url) { m_url = url; }
+  void setTitle(const std::string& title) { m_title = title; }
+  void setLanguage(const std::string& language) { m_language = language; }
+  void setDate(const std::string& date) { m_date = date; }
+
+ protected:
+  std::string m_bookId;
+  std::string m_bookTitle;
+  std::string m_url;
+  std::string m_title;
+  std::string m_language;
+  std::string m_date;
+};
+
+}
+
+#endif

include/common/base64.h

@@ -1,4 +0,0 @@
-#include <string>
-
-std::string base64_encode(unsigned char const* , unsigned int len);
-std::string base64_decode(std::string const& s);

include/ctpp2/CTPP2VMStringLoader.hpp

@@ -1,79 +0,0 @@
-/*
- * Copyright 2013 Renaud Gaudin <reg@kiwix.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU  General Public License as published by
- * the Free Software Foundation; either version 3 of the License, or
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
-
-#ifndef _CTPP2_VM_STRING_LOADER_HPP__
-#define _CTPP2_VM_STRING_LOADER_HPP__ 1
-
-#include <ctpp2/CTPP2VMLoader.hpp>
-#include <ctpp2/CTPP2Util.hpp>
-#include <ctpp2/CTPP2Exception.hpp>
-#include <ctpp2/CTPP2VMExecutable.hpp>
-#include <ctpp2/CTPP2VMInstruction.hpp>
-#include <ctpp2/CTPP2VMMemoryCore.hpp>
-
-#include <sys/types.h>
-#include <sys/stat.h>
-
-#include <errno.h>
-#include <stdio.h>
-#include <stdlib.h>
-
-#include <iostream>
-#include <string>
-
-/**
-  @file VMStringLoader.hpp
-  @brief Load program core from file
-*/
-
-namespace CTPP // C++ Template Engine
-{
-// FWD
-struct VMExecutable;
-
-/**
-  @class VMStringLoader CTPP2VMStringLoader.hpp <CTPP2VMStringLoader.hpp>
-  @brief Load program core from file
-*/
-class CTPP2DECL VMStringLoader:
-  public VMLoader
-{
-public:
-	/**
-	*/
-	VMStringLoader(CCHAR_P rawContent, size_t rawContentSize);
-	/**
-	  @brief Get ready-to-run program
-	*/
-	const VMMemoryCore * GetCore() const;
-
-	/**
-	  @brief A destructor
-	*/
-	~VMStringLoader() throw();
-private:
-	/** Program core             */
-	VMExecutable  * oCore;
-	/** Ready-to-run program     */
-	VMMemoryCore  * pVMMemoryCore;
-};
-
-} // namespace CTPP
-#endif // _CTPP2_VM_STRING_LOADER_HPP__
-// End.

include/downloader.h

@@ -21,13 +21,15 @@
 #define KIWIX_DOWNLOADER_H
 
 #include <string>
-#include <aria2/aria2.h>
+#include <vector>
+#include <map>
 #include <pthread.h>
+#include <memory>
 
 namespace kiwix
 {
 
-
+class Aria2;
 struct DownloadedFile {
   DownloadedFile()
    : success(false) {}
@@ -35,6 +37,49 @@ struct DownloadedFile {
   std::string path;
 };
 
+class AriaError : public std::runtime_error {
+ public:
+  AriaError(const std::string& message) : std::runtime_error(message) {}
+};
+
+
+class Download {
+ public:
+  typedef enum { K_ACTIVE, K_WAITING, K_PAUSED, K_ERROR, K_COMPLETE, K_REMOVED, K_UNKNOWN } StatusResult;
+
+  Download() :
+    m_status(K_UNKNOWN) {}
+  Download(std::shared_ptr<Aria2> p_aria, std::string did)
+    : mp_aria(p_aria),
+      m_status(K_UNKNOWN),
+      m_did(did) {};
+  void updateStatus(bool follow=false);
+  void pauseDownload();
+  void resumeDownload();
+  void cancelDownload();
+  StatusResult getStatus()          { return m_status; }
+  std::string  getDid()             { return m_did; }
+  std::string  getFollowedBy()      { return m_followedBy; }
+  uint64_t     getTotalLength()     { return m_totalLength; }
+  uint64_t     getCompletedLength() { return m_completedLength; }
+  uint64_t     getDownloadSpeed()   { return m_downloadSpeed; }
+  uint64_t     getVerifiedLength()  { return m_verifiedLength; }
+  std::string  getPath()            { return m_path; }
+  std::vector<std::string>&  getUris()             { return m_uris; }
+
+ protected:
+  std::shared_ptr<Aria2> mp_aria;
+  StatusResult m_status;
+  std::string m_did = "";
+  std::string m_followedBy = "";
+  uint64_t m_totalLength;
+  uint64_t m_completedLength;
+  uint64_t m_downloadSpeed;
+  uint64_t m_verifiedLength;
+  std::vector<std::string> m_uris;
+  std::string m_path;
+};
+
 /**
  * A tool to download things.
  *
@@ -43,27 +88,19 @@ class Downloader
 {
  public:
   Downloader();
-  ~Downloader();
+  virtual ~Downloader();
 
-  /**
-   * Download a content.
-   *
-   * @param url the url to download
-   * @return the content downloaded.
-   */
-  DownloadedFile download(const std::string& url);
+  void close();
+
+  Download* startDownload(const std::string& uri);
+  Download* getDownload(const std::string& did);
+
+  size_t getNbDownload() { return m_knownDownloads.size(); }
+  std::vector<std::string> getDownloadIds();
 
  private:
-  static pthread_mutex_t globalLock;
-
-  aria2::Session* session;
-  DownloadedFile* fileHandle;
-  std::string tmpDir;
-
-  static int downloadEventCallback(aria2::Session* session,
-                                   aria2::DownloadEvent event,
-                                   aria2::A2Gid gid,
-                                   void* userData);
+  std::map<std::string, std::unique_ptr<Download>> m_knownDownloads;
+  std::shared_ptr<Aria2> mp_aria;
 };
 }
 

include/entry.h

@@ -24,7 +24,6 @@
 #include <zim/article.h>
 #include <exception>
 #include <string>
-#include "common.h"
 
 using namespace std;
 

include/kiwixserve.h

@@ -0,0 +1,27 @@
+#ifndef KIWIXLIB_KIWIX_SERVE_H_
+#define KIWIXLIB_KIWIX_SERVE_H_
+
+#include <memory>
+
+class Subprocess;
+namespace kiwix {
+
+class KiwixServe
+{
+  public:
+    KiwixServe(int port = 8181);
+    ~KiwixServe();
+
+    void run();
+    void shutDown();
+    bool isRunning();
+    int getPort() { return m_port; }
+
+  private:
+  std::unique_ptr<Subprocess> mp_kiwixServe;
+  int m_port;
+};
+
+}; //end namespace kiwix
+
+#endif // KIWIXLIB_KIWIX_SERVE_H_

include/library.h

@@ -20,78 +20,102 @@
 #ifndef KIWIX_LIBRARY_H
 #define KIWIX_LIBRARY_H
 
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <stack>
 #include <string>
 #include <vector>
+#include <map>
 
-#include "common/regexTools.h"
-#include "common/stringTools.h"
+#include "book.h"
+#include "bookmark.h"
+#include "common.h"
 
 #define KIWIX_LIBRARY_VERSION "20110515"
 
-using namespace std;
-
 namespace kiwix
 {
-enum supportedIndexType { UNKNOWN, XAPIAN };
 
+class OPDSDumper;
 
-/**
- * A class to store information about a book (a zim file)
- */
-class Book
-{
- public:
-  Book();
-  ~Book();
-
-  static bool sortByLastOpen(const Book& a, const Book& b);
-  static bool sortByTitle(const Book& a, const Book& b);
-  static bool sortBySize(const Book& a, const Book& b);
-  static bool sortByDate(const Book& a, const Book& b);
-  static bool sortByCreator(const Book& a, const Book& b);
-  static bool sortByPublisher(const Book& a, const Book& b);
-  static bool sortByLanguage(const Book& a, const Book& b);
-  string getHumanReadableIdFromPath();
-
-  string id;
-  string path;
-  string pathAbsolute;
-  string last;
-  string indexPath;
-  string indexPathAbsolute;
-  supportedIndexType indexType;
-  string title;
-  string description;
-  string language;
-  string creator;
-  string publisher;
-  string date;
-  string url;
-  string name;
-  string tags;
-  string origId;
-  string articleCount;
-  string mediaCount;
-  bool readOnly;
-  string size;
-  string favicon;
-  string faviconMimeType;
+enum supportedListSortBy { UNSORTED, TITLE, SIZE, DATE, CREATOR, PUBLISHER };
+enum supportedListMode {
+  ALL = 0,
+  LOCAL = 1,
+  REMOTE = 1 << 1,
+  NOLOCAL = 1 << 2,
+  NOREMOTE = 1 << 3,
+  VALID = 1 << 4,
+  NOVALID = 1 << 5
 };
 
+class Filter {
+  private:
+    uint64_t activeFilters;
+    std::vector<std::string> _acceptTags;
+    std::vector<std::string> _rejectTags;
+    std::string _lang;
+    std::string _publisher;
+    std::string _creator;
+    size_t _maxSize;
+    std::string _query;
+
+  public:
+    Filter();
+    ~Filter() = default;
+
+    /**
+     *  Set the filter to check local.
+     *
+     *  A local book is a book with a path.
+     *  If accept is true, only local book are accepted.
+     *  If accept is false, only non local book are accepted.
+     */
+    Filter& local(bool accept);
+
+    /**
+     *  Set the filter to check remote.
+     *
+     *  A remote book is a book with a url.
+     *  If accept is true, only remote book are accepted.
+     *  If accept is false, only non remote book are accepted.
+     */
+    Filter& remote(bool accept);
+
+    /**
+     *  Set the filter to check validity.
+     *
+     *  A valid book is a book with a path pointing to a existing zim file.
+     *  If accept is true, only valid book are accepted.
+     *  If accept is false, only non valid book are accepted.
+     */
+    Filter& valid(bool accept);
+
+    /**
+     * Set the filter to only accept book with corresponding tag.
+     */
+    Filter& acceptTags(std::vector<std::string> tags);
+    Filter& rejectTags(std::vector<std::string> tags);
+
+    Filter& lang(std::string lang);
+    Filter& publisher(std::string publisher);
+    Filter& creator(std::string creator);
+    Filter& maxSize(size_t size);
+    Filter& query(std::string query);
+
+    bool accept(const Book& book) const;
+};
+
+
 /**
  * A Library store several books.
  */
 class Library
 {
+  std::map<std::string, kiwix::Book> m_books;
+  std::vector<kiwix::Bookmark> m_bookmarks;
+
  public:
   Library();
   ~Library();
 
-  string version;
   /**
    * Add a book to the library.
    *
@@ -104,26 +128,155 @@ class Library
    */
   bool addBook(const Book& book);
 
+  /**
+   * Add a bookmark to the library.
+   *
+   * @param bookmark the book to add.
+   */
+  void addBookmark(const Bookmark& bookmark);
+
+  /**
+   * Remove a bookmarkk
+   *
+   * @param zimId The zimId of the bookmark.
+   * @param url The url of the bookmark.
+   * @return True if the bookmark has been removed.
+   */
+  bool removeBookmark(const std::string& zimId, const std::string& url);
+
+  Book& getBookById(const std::string& id);
+
   /**
    * Remove a book from the library.
    *
-   * @param bookIndex the index of the book to remove.
-   * @return True
+   * @param id the id of the book to remove.
+   * @return True if the book were in the lirbrary and has been removed.
    */
-  bool removeBookByIndex(const unsigned int bookIndex);
-  vector<kiwix::Book> books;
+  bool removeBookById(const std::string& id);
 
-  /*
-   * 'current' is the variable storing the current content/book id
-   * in the library. This is used to be able to load per default a
-   * content. As Kiwix may work with many library XML files, you may
-   * have "current" defined many time with different values. The
-   * last XML file read has the priority, Although we do not have an
-   * library object for each file, we want to be able to fallback to
-   * an 'old' current book if the one which should be load
-   * failed. That is the reason why we need a stack here
+  /**
+   * Write the library to a file.
+   *
+   * @param path the path of the file to write to.
+   * @return True if the library has been correctly saved.
    */
-  stack<string> current;
+  bool writeToFile(const std::string& path);
+
+  /**
+   * Write the library bookmarks to a file.
+   *
+   * @param path the path of the file to write to.
+   * @return True if the library has been correctly saved.
+   */
+  bool writeBookmarksToFile(const std::string& path);
+
+  /**
+   * Get the number of book in the library.
+   *
+   * @param localBooks If we must count local books (books with a path).
+   * @param remoteBooks If we must count remote books (books with an url)
+   * @return The number of books.
+   */
+  unsigned int getBookCount(const bool localBooks, const bool remoteBooks);
+
+  /**
+   * Get all langagues of the books in the library.
+   *
+   * @return A list of languages.
+   */
+  std::vector<std::string> getBooksLanguages();
+
+  /**
+   * Get all book creators of the books in the library.
+   *
+   * @return A list of book creators.
+   */
+  std::vector<std::string> getBooksCreators();
+
+  /**
+   * Get all book publishers of the books in the library.
+   *
+   * @return A list of book publishers.
+   */
+  std::vector<std::string> getBooksPublishers();
+
+  /**
+   * Get all bookmarks.
+   *
+   * @return A list of bookmarks
+   */
+  const std::vector<kiwix::Bookmark>& getBookmarks() { return m_bookmarks; }
+
+  /**
+   * Get all book ids of the books in the library.
+   *
+   * @return A list of book ids.
+   */
+  std::vector<std::string> getBooksIds();
+
+  /**
+   * Filter the library and generate a new one with the keep elements.
+   *
+   * This is equivalent to `listBookIds(ALL, UNSORTED, search)`.
+   *
+   * @param search List only books with search in the title or description.
+   * @return The list of bookIds corresponding to the query.
+   */
+  DEPRECATED std::vector<std::string> filter(const std::string& search);
+
+
+  /**
+   * Filter the library and return the id of the keep elements.
+   *
+   * @param filter The filter to use.
+   * @return The list of bookIds corresponding to the filter.
+   */
+  std::vector<std::string> filter(const Filter& filter);
+
+
+  /**
+   * Sort (in place) bookIds using the given comparator.
+   *
+   * @param bookIds the list of book Ids to sort
+   * @param comparator how to sort the books
+   * @return The sorted list of books
+   */
+  void sort(std::vector<std::string>& bookIds, supportedListSortBy sortBy, bool ascending);
+
+  /**
+   * List books in the library.
+   *
+   * @param mode The mode of listing :
+   *             - LOCAL    : list only local books (with a path).
+   *             - REMOTE   : list only remote books (with an url).
+   *             - VALID    : list only valid books (without a path or with a
+   *                          path pointing to a valid zim file).
+   *             - NOLOCAL  : list only books without valid path.
+   *             - NOREMOTE : list only books without url.
+   *             - NOVALID  : list only books not valid.
+   *             - ALL : Do not do any filter (LOCAL or REMOTE)
+   *             - Flags can be combined.
+   * @param sortBy Attribute to sort by the book list.
+   * @param search List only books with search in the title, description.
+   * @param language List only books in this language.
+   * @param creator List only books of this creator.
+   * @param publisher List only books of this publisher.
+   * @param maxSize Do not list book bigger than maxSize.
+   *                Set to 0 to cancel this filter.
+   * @return The list of bookIds corresponding to the query.
+   */
+  DEPRECATED std::vector<std::string> listBooksIds(
+    int supportedListMode = ALL,
+    supportedListSortBy sortBy = UNSORTED,
+    const std::string& search = "",
+    const std::string& language = "",
+    const std::string& creator = "",
+    const std::string& publisher = "",
+    const std::vector<std::string>& tags = {},
+    size_t maxSize = 0);
+
+  friend class OPDSDumper;
+  friend class libXMLDumper;
 };
 }
 

include/libxml_dumper.h

@@ -0,0 +1,83 @@
+/*
+ * Copyright 2018 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#ifndef KIWIX_LIBXML_DUMPER_H
+#define KIWIX_LIBXML_DUMPER_H
+
+#include <string>
+#include <vector>
+
+#include <pugixml.hpp>
+
+#include "library.h"
+
+namespace kiwix
+{
+
+/**
+ * A tool to dump a `Library` into a basic library.xml
+ *
+ */
+class LibXMLDumper
+{
+ public:
+  LibXMLDumper() = default;
+  LibXMLDumper(Library* library);
+  ~LibXMLDumper();
+
+  /**
+   * Dump the library.xml
+   *
+   * @param id The id of the library.
+   * @return The library.xml content.
+   */
+  std::string dumpLibXMLContent(const std::vector<std::string>& bookIds);
+
+
+  /**
+   * Dump the bookmark of the library.
+   *
+   * @return The bookmark.xml content.
+   */
+  std::string dumpLibXMLBookmark();
+
+  /**
+   * Set the base directory used.
+   *
+   * @param baseDir the base directory to use.
+   */
+  void setBaseDir(const std::string& baseDir) { this->baseDir = baseDir; }
+
+  /**
+   * Set the library to dump.
+   *
+   * @param library The library to dump.
+   */
+  void setLibrary(Library* library) { this->library = library; }
+
+ protected:
+  kiwix::Library* library;
+  std::string baseDir;
+ private:
+  void handleBook(Book book, pugi::xml_node root_node);
+  void handleBookmark(Bookmark bookmark, pugi::xml_node root_node);
+};
+}
+
+#endif // KIWIX_OPDS_DUMPER_H

include/manager.h

@@ -20,35 +20,50 @@
 #ifndef KIWIX_MANAGER_H
 #define KIWIX_MANAGER_H
 
-#include <time.h>
-#include <sstream>
-#include <string>
-
-#include <pugixml.hpp>
-
-#include "common/base64.h"
-#include "common/pathTools.h"
-#include "common/regexTools.h"
+#include "book.h"
 #include "library.h"
 #include "reader.h"
 
-using namespace std;
+#include <string>
+#include <vector>
+
+namespace pugi {
+class xml_document;
+}
 
 namespace kiwix
 {
-enum supportedListMode { LASTOPEN, REMOTE, LOCAL };
-enum supportedListSortBy { TITLE, SIZE, DATE, CREATOR, PUBLISHER };
+
+class LibraryManipulator {
+ public:
+  virtual ~LibraryManipulator() {}
+  virtual bool addBookToLibrary(Book book) = 0;
+  virtual void addBookmarkToLibrary(Bookmark bookmark) = 0;
+};
+
+class DefaultLibraryManipulator : public LibraryManipulator {
+ public:
+  DefaultLibraryManipulator(Library* library) :
+    library(library) {}
+  virtual ~DefaultLibraryManipulator() {}
+  bool addBookToLibrary(Book book) {
+    return library->addBook(book);
+  }
+  void addBookmarkToLibrary(Bookmark bookmark) {
+    library->addBookmark(bookmark);
+  }
+ private:
+   kiwix::Library* library;
+};
 
 /**
  * A tool to manage a `Library`.
- *
- * A `Manager` handle a internal `Library`.
- * This `Library` can be retrived with `cloneLibrary` method.
  */
 class Manager
 {
  public:
-  Manager();
+  Manager(LibraryManipulator* manipulator);
+  Manager(Library* library);
   ~Manager();
 
   /**
@@ -59,7 +74,7 @@ class Manager
    *                 updated content.
    * @return True if file has been properly parsed.
    */
-  bool readFile(const string path, const bool readOnly = true);
+  bool readFile(const std::string& path, const bool readOnly = true);
 
   /**
    * Read a `library.xml` and add book in the file to the library.
@@ -71,8 +86,8 @@ class Manager
    *                 updated content.
    * @return True if file has been properly parsed.
    */
-  bool readFile(const string nativePath,
-                const string UTF8Path,
+  bool readFile(const std::string& nativePath,
+                const std::string& UTF8Path,
                 const bool readOnly = true);
 
   /**
@@ -84,9 +99,9 @@ class Manager
    * @param libraryPath The library path (used to resolve relative path)
    * @return True if the content has been properly parsed.
    */
-  bool readXml(const string& xml,
+  bool readXml(const std::string& xml,
                const bool readOnly = true,
-               const string libraryPath = "");
+               const std::string& libraryPath = "");
 
   /**
    * Load a library content stored in a OPDS stream.
@@ -97,69 +112,16 @@ class Manager
    * @param libraryPath The library path (used to resolve relative path)
    * @return True if the content has been properly parsed.
    */
-  bool readOpds(const string& content, const std::string& urlHost);
-
-  /**
-   * Write the library to a file.
-   *
-   * @param path the path of the file to write.
-   * @return True.
-   */
-  bool writeFile(const string path);
+  bool readOpds(const std::string& content, const std::string& urlHost);
 
 
   /**
-   * Remove a book from the library.
+   * Load a bookmark file.
    *
-   * @param bookIndex the index of the book to remove
-   * @return True
+   * @param path The path of the file to read.
+   * @return True if the content has been properly parsed.
    */
-  bool removeBookByIndex(const unsigned int bookIndex);
-
-  /**
-   * Remove a book from the library.
-   *
-   * @param id the id of the book to remove.
-   * @return True if the book were in the library.
-   */
-  bool removeBookById(const string id);
-
-  /**
-   * Set the current book.
-   *
-   * @param id The id to add to the stack of current books.
-   *           If id is empty, remove the current book from the stack.
-   * @return True
-   */
-  bool setCurrentBookId(const string id);
-
-  /**
-   * Get the current book id.
-   *
-   * @return The id of the current book (or empty string if no current book).
-   */
-  string getCurrentBookId() const;
-
-  /**
-   * Set the path of the external fulltext index associated to a book.
-   *
-   * @param id The id of the book to set.
-   * @param path The path of the external fullext index.
-   * @param supportedIndexType The type of the fulltext index.
-   * @return True if the book is in the library.
-   */
-  bool setBookIndex(const string id,
-                    const string path,
-                    const supportedIndexType type = XAPIAN);
-
-  /**
-   * Set the path of the zim file associated to a book.
-   *
-   * @param id The id of the book to set.
-   * @param path The path of the zim file.
-   * @return True if the book is in the library.
-   */
-  bool setBookPath(const string id, const string path);
+  bool readBookmarkFile(const std::string& path);
 
   /**
    * Add a book to the library.
@@ -172,9 +134,9 @@ class Manager
    * @return The id of the book if the book has been added to the library.
    *         Else, an empty string.
    */
-  string addBookFromPathAndGetId(const string pathToOpen,
-                                 const string pathToSave = "",
-                                 const string url = "",
+  std::string addBookFromPathAndGetId(const std::string& pathToOpen,
+                                 const std::string& pathToSave = "",
+                                 const std::string& url = "",
                                  const bool checkMetaData = false);
 
   /**
@@ -188,137 +150,29 @@ class Manager
    * @return True if the book has been added to the library.
    */
 
-  bool addBookFromPath(const string pathToOpen,
-                       const string pathToSave = "",
-                       const string url = "",
+  bool addBookFromPath(const std::string& pathToOpen,
+                       const std::string& pathToSave = "",
+                       const std::string& url = "",
                        const bool checkMetaData = false);
 
-  /**
-   * Clone and return the internal library.
-   *
-   * @return A clone of the library.
-   */
-  Library cloneLibrary();
+  std::string writableLibraryPath;
 
-  /**
-   * Get the book corresponding to an id.
-   *
-   * @param[in] id The id of the book
-   * @param[out] book The book corresponding to the id.
-   * @return True if the book has been found.
-   */
-  bool getBookById(const string id, Book& book);
-
-  /**
-   * Get the current book.
-   *
-   * @param[out] The current book.
-   * @return True if there is a current book.
-   */
-  bool getCurrentBook(Book& book);
-
-  /**
-   * Get the number of book in the library.
-   *
-   * @param localBooks If we must count local books (books with a path).
-   * @param remoteBooks If we must count remote books (books with an url)
-   * @return The number of books.
-   */
-  unsigned int getBookCount(const bool localBooks, const bool remoteBooks);
-
-  /**
-   * Update the "last open date" of a book
-   *
-   * @param id the id of the book.
-   * @return True if the book is in the library.
-   */
-  bool updateBookLastOpenDateById(const string id);
-
-  /**
-   * Remove (set to empty) paths of all books in the library.
-   */
-  void removeBookPaths();
-
-  /**
-   * List books in the library.
-   *
-   * The books list will be available in public vector member `bookIdList`.
-   *
-   * @param mode The mode of listing :
-   *             - LASTOPEN sort by last opened book.
-   *             - LOCAL list only local file.
-   *             - REMOTE list only remote file.
-   * @param sortBy Attribute to sort by the book list.
-   * @param maxSize Do not list book bigger than maxSize MiB.
-   *                Set to 0 to cancel this filter.
-   * @param language List only books in this language.
-   * @param creator List only books of this creator.
-   * @param publisher List only books of this publisher.
-   * @param search List only books with search in the title, description or
-   *               language.
-   * @return True
-   */
-  bool listBooks(const supportedListMode mode,
-                 const supportedListSortBy sortBy,
-                 const unsigned int maxSize,
-                 const string language,
-                 const string creator,
-                 const string publisher,
-                 const string search);
-
-  /**
-   * Filter the library and generate a new one with the keep elements.
-   *
-   * @param search List only books with search in the title or description.
-   * @return A `Library`.
-   */
-  Library filter(const string& search);
-
-
-  /**
-   * Get all langagues of the books in the library.
-   *
-   * @return A list of languages.
-   */
-  vector<string> getBooksLanguages();
-
-  /**
-   * Get all book creators of the books in the library.
-   *
-   * @return A list of book creators.
-   */
-  vector<string> getBooksCreators();
-
-  /**
-   * Get all book publishers of the books in the library.
-   *
-   * @return A list of book publishers.
-   */
-  vector<string> getBooksPublishers();
-
-  /**
-   * Get all book ids of the books in the library.
-   *
-   * @return A list of book ids.
-   */
-  vector<string> getBooksIds();
-
-  string writableLibraryPath;
-
-  vector<std::string> bookIdList;
+  bool m_hasSearchResult = false;
+  uint64_t m_totalBooks = 0;
+  uint64_t m_startIndex = 0;
+  uint64_t m_itemsPerPage = 0;
 
  protected:
-  kiwix::Library library;
+  kiwix::LibraryManipulator* manipulator;
+  bool mustDeleteManipulator;
 
-  bool readBookFromPath(const string path, Book* book = NULL);
+  bool readBookFromPath(const std::string& path, Book* book);
   bool parseXmlDom(const pugi::xml_document& doc,
                    const bool readOnly,
-                   const string libraryPath);
+                   const std::string& libraryPath);
   bool parseOpdsDom(const pugi::xml_document& doc,
                     const std::string& urlHost);
 
- private:
-  void checkAndCleanBookPaths(Book& book, const string& libraryPath);
 };
 }
 

include/meson.build

@@ -1,34 +1,27 @@
 headers = [
+  'book.h',
+  'bookmark.h',
   'common.h',
   'library.h',
   'manager.h',
+  'libxml_dumper.h',
   'opds_dumper.h',
   'downloader.h',
   'reader.h',
   'entry.h',
-  'searcher.h'
+  'searcher.h',
+  'kiwixserve.h'
 ]
 
-if xapian_dep.found()
-  headers += ['xapianSearcher.h']
-endif
-
 install_headers(headers, subdir:'kiwix')
 
 install_headers(
-  'common/base64.h',
-  'common/networkTools.h',
-  'common/otherTools.h',
-  'common/pathTools.h',
-  'common/regexTools.h',
-  'common/stringTools.h',
-  subdir:'kiwix/common'
+  'tools/base64.h',
+  'tools/networkTools.h',
+  'tools/otherTools.h',
+  'tools/pathTools.h',
+  'tools/regexTools.h',
+  'tools/stringTools.h',
+  subdir:'kiwix/tools'
 )
 
-if has_ctpp2_dep
-  install_headers(
-    'ctpp2/CTPP2VMStringLoader.hpp',
-    subdir:'kiwix/ctpp2'
-  )
-endif
-

include/opds_dumper.h

@@ -26,9 +26,9 @@
 
 #include <pugixml.hpp>
 
-#include "common/base64.h"
-#include "common/pathTools.h"
-#include "common/regexTools.h"
+#include "tools/base64.h"
+#include "tools/pathTools.h"
+#include "tools/regexTools.h"
 #include "library.h"
 #include "reader.h"
 
@@ -45,7 +45,7 @@ class OPDSDumper
 {
  public:
   OPDSDumper() = default;
-  OPDSDumper(Library library);
+  OPDSDumper(Library* library);
   ~OPDSDumper();
 
   /**
@@ -54,7 +54,7 @@ class OPDSDumper
    * @param id The id of the library.
    * @return The OPDS feed.
    */
-  std::string dumpOPDSFeed();
+  std::string dumpOPDSFeed(const std::vector<std::string>& bookIds);
 
   /**
    * Set the id of the opds stream.
@@ -84,20 +84,33 @@ class OPDSDumper
    */
   void setSearchDescriptionUrl(const std::string& searchDescriptionUrl) { this->searchDescriptionUrl = searchDescriptionUrl; }
 
+  /**
+   * Set some informations about the search results.
+   *
+   * @param totalResult the total number of results of the search.
+   * @param startIndex the start index of the result.
+   * @param count the number of result of the current set (or page).
+   */
+  void setOpenSearchInfo(int totalResult, int startIndex, int count);
+
   /**
    * Set the library to dump.
    *
    * @param library The library to dump.
    */
-  void setLibrary(Library library) { this->library = library; }
+  void setLibrary(Library* library) { this->library = library; }
 
  protected:
-  kiwix::Library library;
+  kiwix::Library* library;
   std::string id;
   std::string title;
   std::string date;
   std::string rootLocation;
   std::string searchDescriptionUrl;
+  int m_totalResults;
+  int m_startIndex;
+  int m_count;
+  bool m_isSearchResult = false;
 
  private:
   pugi::xml_node handleBook(Book book, pugi::xml_node root_node);

include/reader.h

@@ -31,8 +31,8 @@
 #include <string>
 #include "common.h"
 #include "entry.h"
-#include "common/pathTools.h"
-#include "common/stringTools.h"
+#include "tools/pathTools.h"
+#include "tools/stringTools.h"
 
 using namespace std;
 

include/searcher.h

@@ -29,8 +29,8 @@
 #include <string>
 #include <vector>
 #include <vector>
-#include "common/pathTools.h"
-#include "common/stringTools.h"
+#include "tools/pathTools.h"
+#include "tools/stringTools.h"
 #include "kiwix_config.h"
 
 using namespace std;
@@ -57,24 +57,7 @@ struct SearcherInternal;
  * The Searcher class is reponsible to do different kind of search using the
  * fulltext index.
  *
- * Historically, there are two kind of fulltext index :
- *  - The legacy one, is the external fulltext index. A directory stored outside
- *    of the zim file.
- *  - The new one, a embedded fulltext index in the zim file.
- *
- * Legacy external fulltext index has to be considered as obsolet format with
- * less functionnalities:
- *  - No multi zim search ;
- *  - No geo_search ;
- *  - No suggestions search ;
- *
- * To reflect this, there is two Search creation "API":
- *  - One for the external fulltext index, using the constructor taking a
- *    xapianDirectoryPath) ;
- *  - One for the embedded fulltext index, using a "empty" constructor and the
- *  `add_reader` method".
- *
- *  On top of that, the Searcher may (if compiled with ctpp2) be used to
+ *  Searcher may (if compiled with ctpp2) be used to
  *  generate a html page for the search result. This use a template that need a
  *  humanReaderName. This feature is only used by kiwix-serve and this should be
  *  move outside of Searcher (and with a better API). If you don't use the html
@@ -92,18 +75,6 @@ class Searcher
    */
   Searcher(const string& humanReadableName = "");
 
-  /**
-   * The constructor for legacy external fulltext index.
-   *
-   * @param xapianDirectoryPath The path to the external index directory.
-   * @param reader The reader associated to the external index.
-   *               It will be used retrive the article content or generate
-   *               the snippet.
-   * @param humanReadableName The humanReadableName for the zim.
-   */
-  Searcher(const string& xapianDirectoryPath,
-           Reader* reader,
-           const string& humanReadableName);
   ~Searcher();
 
   /**
@@ -192,12 +163,10 @@ class Searcher
    */
   bool setSearchProtocolPrefix(const std::string prefix);
 
-#ifdef ENABLE_CTPP2
   /**
    * Generate the html page with the resutls of the search.
    */
   string getHtml();
-#endif
 
  protected:
   std::string beautifyInteger(const unsigned int number);

include/tools/base64.h

@@ -0,0 +1,4 @@
+#include <string>
+
+std::string base64_encode(const std::string& inString);
+std::string base64_decode(const std::string& s);

include/tools/networkTools.h

@@ -1,5 +1,5 @@
 /*
- * Copyright 2014 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright 2012 Emmanuel Engelhart <kelson@kiwix.org>
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU  General Public License as published by
@@ -17,13 +17,14 @@
  * MA 02110-1301, USA.
  */
 
-#include <common/otherTools.h>
+#ifndef KIWIX_NETWORKTOOLS_H
+#define KIWIX_NETWORKTOOLS_H
 
-void kiwix::sleep(unsigned int milliseconds)
+#include <string>
+
+namespace kiwix
 {
-#ifdef _WIN32
-  Sleep(milliseconds);
-#else
-  usleep(1000 * milliseconds);
-#endif
+std::string download(const std::string& url);
 }
+
+#endif

include/tools/otherTools.h

@@ -26,9 +26,13 @@
 #include <unistd.h>
 #endif
 
+#include <pugixml.hpp>
+
 namespace kiwix
 {
 void sleep(unsigned int milliseconds);
+std::string nodeToString(pugi::xml_node node);
+std::string converta2toa3(const std::string& a2code);
 }
 
 #endif

include/tools/pathTools.h

@@ -59,5 +59,6 @@ bool copyFile(const string& sourcePath, const string& destPath);
 string getLastPathElement(const string& path);
 string getExecutablePath();
 string getCurrentDirectory();
+string getDataDirectory();
 bool writeTextFile(const string& path, const string& content);
 #endif

include/tools/stringTools.h

@@ -33,19 +33,15 @@
 
 namespace kiwix
 {
-#ifndef __ANDROID__
-
-std::string beautifyInteger(const unsigned int number);
-std::string beautifyFileSize(const unsigned int number);
+std::string beautifyInteger(uint64_t number);
+std::string beautifyFileSize(uint64_t number);
 void printStringInHexadecimal(const char* s);
-void printStringInHexadecimal(UnicodeString s);
+void printStringInHexadecimal(icu::UnicodeString s);
 void stringReplacement(std::string& str,
                        const std::string& oldStr,
                        const std::string& newStr);
 std::string encodeDiples(const std::string& str);
 
-#endif
-
 std::string removeAccents(const std::string& text);
 void loadICUExternalTables();
 
@@ -64,6 +60,20 @@ std::string lcFirst(const std::string& word);
 std::string toTitle(const std::string& word);
 
 std::string normalize(const std::string& word);
+template<typename T>
+std::string to_string(T value)
+{
+    std::ostringstream oss;
+    oss << value;
+    return oss.str();
 }
 
+template<typename T>
+T extractFromString(const std::string& str) {
+    std::istringstream iss(str);
+    T ret;
+    iss >> ret;
+    return ret;
+}
+} //namespace kiwix
 #endif

include/xapianSearcher.h

@@ -1,98 +0,0 @@
-/*
- * Copyright 2011 Emmanuel Engelhart <kelson@kiwix.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU  General Public License as published by
- * the Free Software Foundation; either version 3 of the License, or
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
-
-#ifndef KIWIX_XAPIAN_SEARCHER_H
-#define KIWIX_XAPIAN_SEARCHER_H
-
-#include <xapian.h>
-#include "reader.h"
-#include "searcher.h"
-
-#include <map>
-#include <string>
-
-using namespace std;
-
-namespace kiwix
-{
-class XapianSearcher;
-
-class XapianResult : public Result
-{
- public:
-  XapianResult(XapianSearcher* searcher, Xapian::MSetIterator& iterator);
-  virtual ~XapianResult(){};
-
-  virtual std::string get_url();
-  virtual std::string get_title();
-  virtual int get_score();
-  virtual std::string get_snippet();
-  virtual std::string get_content();
-  virtual int get_wordCount();
-  virtual int get_size();
-  virtual int get_readerIndex() { return 0; };
-
- private:
-  XapianSearcher* searcher;
-  Xapian::MSetIterator iterator;
-  Xapian::Document document;
-};
-
-class NoXapianIndexInZim : public exception
-{
-  virtual const char* what() const throw()
-  {
-    return "There is no fulltext index in the zim file";
-  }
-};
-
-class XapianSearcher
-{
-  friend class XapianResult;
-
- public:
-  XapianSearcher(const string& xapianDirectoryPath, Reader* reader);
-  virtual ~XapianSearcher(){};
-  void searchInIndex(string& search,
-                     const unsigned int resultStart,
-                     const unsigned int resultEnd,
-                     const bool verbose = false);
-  virtual Result* getNextResult();
-  void restart_search();
-
-  Xapian::MSet results;
-
- protected:
-  void closeIndex();
-  void openIndex(const string& xapianDirectoryPath);
-  void setup_queryParser();
-
-  Reader* reader;
-  Xapian::Database readableDatabase;
-  std::string language;
-  std::string stopwords;
-  Xapian::QueryParser queryParser;
-  Xapian::Stem stemmer;
-  Xapian::SimpleStopper stopper;
-  Xapian::MSetIterator current_result;
-  std::map<std::string, int> valuesmap;
-};
-}
-
-#endif

meson.build

@@ -1,107 +1,53 @@
 project('kiwix-lib', 'cpp',
-  version : '2.0.0',
+  version : '5.2.0',
   license : 'GPL',
   default_options : ['c_std=c11', 'cpp_std=c++11', 'werror=true'])
 
 compiler = meson.get_compiler('cpp')
-find_library_in_compiler = meson.version().version_compare('>=0.31.0')
 
 static_deps = get_option('android') or get_option('default_library') == 'static'
+if get_option('android')
+  extra_libs = ['-llog']
+else
+  extra_libs = []
+endif
 
 thread_dep = dependency('threads')
 libicu_dep = dependency('icu-i18n', static:static_deps)
-libzim_dep = dependency('libzim', version : '>=3.3.0', static:static_deps)
+libzim_dep = dependency('libzim', version : '>=5.0.0', static:static_deps)
 pugixml_dep = dependency('pugixml', static:static_deps)
-libaria2_dep = dependency('libaria2', static:static_deps)
+libcurl_dep = dependency('libcurl', static:static_deps)
 
-ctpp2_include_path = ''
-has_ctpp2_dep = false
-ctpp2_prefix_install = get_option('ctpp2-install-prefix')
-ctpp2_link_args = []
-if ctpp2_prefix_install == ''
-  if compiler.has_header('ctpp2/CTPP2Logger.hpp')
-    if find_library_in_compiler
-      ctpp2_lib = compiler.find_library('ctpp2')
-    else
-      ctpp2_lib = find_library('ctpp2')
-    endif
-    ctpp2_link_args = ['-lctpp2']
-    if meson.is_cross_build() and host_machine.system() == 'windows'
-      if find_library_in_compiler
-        iconv_lib = compiler.find_library('iconv', required:false)
-      else
-        iconv_lib = find_library('iconv', required:false)
-      endif
-      if iconv_lib.found()
-        ctpp2_link_args += ['-liconv']
-      endif
-    endif
-    has_ctpp2_dep = true
-    ctpp2_dep = declare_dependency(link_args:ctpp2_link_args)
-  else
-    message('ctpp2/CTPP2Logger.hpp not found. Compiling without CTPP2 support')
-  endif
-else
-  if not find_library_in_compiler
-    error('For custom ctpp2_prefix_install you need a meson version >=0.31.0')
-  endif
-  ctpp2_include_path = ctpp2_prefix_install + '/include'
-  ctpp2_include_args = ['-I'+ctpp2_include_path]
-  if compiler.has_header('ctpp2/CTPP2Logger.hpp', args:ctpp2_include_args)
-    ctpp2_include_dir = include_directories(ctpp2_include_path, is_system:true)
-    ctpp2_lib_path = join_paths(ctpp2_prefix_install, get_option('libdir'))
-    message(ctpp2_lib_path)
-    ctpp2_lib = compiler.find_library('ctpp2', dirs:ctpp2_lib_path, required:false)
-    if not ctpp2_lib.found()
-      ctpp2_lib_path = join_paths(ctpp2_prefix_install, 'lib')
-      message(ctpp2_lib_path)
-      ctpp2_lib = compiler.find_library('ctpp2', dirs:ctpp2_lib_path)
-    endif
-    ctpp2_link_args = ['-L'+ctpp2_lib_path, '-lctpp2']
-    if meson.is_cross_build() and host_machine.system() == 'windows'
-      iconv_lib = compiler.find_library('iconv', required:false)
-      if iconv_lib.found()
-        ctpp2_link_args += ['-liconv']
-      endif
-    endif
-    has_ctpp2_dep = true
-    ctpp2_dep = declare_dependency(include_directories:ctpp2_include_dir, link_args:ctpp2_link_args)
-  else
-    message('ctpp2/CTPP2Logger.hpp not found. Compiling without CTPP2 support')
-  endif
+if not compiler.has_header('mustache.hpp')
+  error('Cannot found header mustache.hpp')
 endif
 
-xapian_dep = dependency('xapian-core', required:false, static:static_deps)
-
-all_deps = [thread_dep, libicu_dep, libzim_dep, xapian_dep, pugixml_dep, libaria2_dep]
-if has_ctpp2_dep
-  all_deps += [ctpp2_dep]
+extra_cflags = ''
+if target_machine.system() == 'windows' and static_deps
+  add_project_arguments('-DCURL_STATICLIB', language : 'cpp')
+  extra_cflags += '-DCURL_STATICLIB'
 endif
 
+all_deps = [thread_dep, libicu_dep, libzim_dep, pugixml_dep, libcurl_dep]
+
 inc = include_directories('include')
 
 conf = configuration_data()
 conf.set('VERSION', '"@0@"'.format(meson.project_version()))
-conf.set('ENABLE_CTPP2', has_ctpp2_dep)
+
+if build_machine.system() == 'windows'
+     extra_link_args = ['-lshlwapi', '-lwinmm']
+else
+     extra_link_args = []
+endif
 
 subdir('include')
 subdir('scripts')
 subdir('static')
 subdir('src')
+subdir('test')
 
-pkg_requires = ['libzim', 'icu-i18n', 'pugixml', 'libaria2']
-if xapian_dep.found()
-    pkg_requires += ['xapian-core']
-endif
-
-extra_libs = []
-extra_cflags = ''
-if has_ctpp2_dep
-  extra_libs += ctpp2_link_args
-  if ctpp2_include_path != ''
-    extra_cflags = '-I'+ctpp2_include_path
-  endif
-endif
+pkg_requires = ['libzim', 'icu-i18n', 'pugixml', 'libcurl']
 
 pkg_conf = configuration_data()
 pkg_conf.set('prefix', get_option('prefix'))

meson_options.txt

@@ -1,4 +1,2 @@
-option('ctpp2-install-prefix', type : 'string', value : '',
-  description : 'Prefix where ctpp libs has been installed')
 option('android', type : 'boolean', value : false,
   description : 'Do we make a kiwix-lib for android')

scripts/ctpp2c.sh

@@ -1,8 +0,0 @@
-#!/usr/bin/env bash
-
-
-ctpp2c=$1
-SOURCE=$(pwd)/$2
-DEST=$3
-
-$ctpp2c $SOURCE $DEST

scripts/meson.build

@@ -1,5 +1,4 @@
 
 res_compiler = find_program('kiwix-compile-resources')
-intermediate_ctpp2c = find_program('ctpp2c.sh')
 
 install_data(res_compiler.path(), install_dir:get_option('bindir'))

src/android/gen_kiwix.sh

@@ -1,16 +0,0 @@
-#!/usr/bin/env bash
-
-set -e
-
-BUILD_PATH=$(pwd)
-
-echo "javac -d $BUILD_PATH/src/android $@"
-javac -d $BUILD_PATH/src/android/ "$@"
-
-
-cd $BUILD_PATH/src/android
-echo "javah -jni org.kiwix.kiwixlib"
-javah -jni org.kiwix.kiwixlib.JNIKiwix
-javah -jni org.kiwix.kiwixlib.JNIKiwixReader
-javah -jni org.kiwix.kiwixlib.JNIKiwixSearcher
-cd $BUILD_PATH

src/android/kiwixicu.cpp

@@ -19,7 +19,7 @@
  */
 
 #include <jni.h>
-#include "org_kiwix_kiwixlib_JNIKiwix.h"
+#include "org_kiwix_kiwixlib_JNIICU.h"
 
 #include <iostream>
 #include <string>
@@ -30,7 +30,7 @@
 
 pthread_mutex_t globalLock = PTHREAD_RECURSIVE_MUTEX_INITIALIZER;
 
-JNIEXPORT void JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_setDataDirectory(
+JNIEXPORT void JNICALL Java_org_kiwix_kiwixlib_JNIICU_setDataDirectory(
     JNIEnv* env, jobject obj, jstring dirStr)
 {
   std::string cPath = jni2c(dirStr, env);

src/android/kiwixreader.cpp

@@ -21,9 +21,10 @@
 
 #include <jni.h>
 #include <zim/file.h>
+#include <android/log.h>
 #include "org_kiwix_kiwixlib_JNIKiwixReader.h"
 
-#include "common/base64.h"
+#include "tools/base64.h"
 #include "reader.h"
 #include "utils.h"
 
@@ -33,16 +34,16 @@ JNIEXPORT jlong JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getNativeReader(
 {
   std::string cPath = jni2c(filename, env);
 
+  __android_log_print(ANDROID_LOG_INFO, "kiwix", "Attempting to create reader with: %s", cPath.c_str());
   Lock l;
-  kiwix::Reader* reader = nullptr;
   try {
-    reader = new kiwix::Reader(cPath);
-  } catch (...) {
-    std::cerr << "Unable to load ZIM " << cPath << std::endl;
-    reader = NULL;
+    kiwix::Reader* reader = new kiwix::Reader(cPath);
+    return reinterpret_cast<jlong>(new Handle<kiwix::Reader>(reader));
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", "Error opening ZIM file");
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", e.what());
+    return 0;
   }
-
-  return reinterpret_cast<jlong>(new Handle<kiwix::Reader>(reader));
 }
 
 JNIEXPORT void JNICALL
@@ -62,8 +63,10 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getMainPage(JNIEnv* env, jobject obj)
   try {
     std::string cUrl = READER->getMainPage().getPath();
     url = c2jni(cUrl, env);
-  } catch (...) {
-    std::cerr << "Unable to get ZIM main page" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM main page");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    url = NULL;
   }
   return url;
 }
@@ -76,8 +79,10 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getId(JNIEnv* env, jobject obj)
   try {
     std::string cId = READER->getId();
     id = c2jni(cId, env);
-  } catch (...) {
-    std::cerr << "Unable to get ZIM id" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM id");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    id = NULL;
   }
 
   return id;
@@ -86,13 +91,14 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getId(JNIEnv* env, jobject obj)
 JNIEXPORT jint JNICALL
 Java_org_kiwix_kiwixlib_JNIKiwixReader_getFileSize(JNIEnv* env, jobject obj)
 {
-  jint size;
+  jint size = 0;
 
   try {
     int cSize = READER->getFileSize();
     size = c2jni(cSize);
-  } catch (...) {
-    std::cerr << "Unable to get ZIM file size" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM file size");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
   }
 
   return size;
@@ -106,8 +112,10 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getCreator(JNIEnv* env, jobject obj)
   try {
     std::string cCreator = READER->getCreator();
     creator = c2jni(cCreator, env);
-  } catch (...) {
-    std::cerr << "Unable to get ZIM creator" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM creator");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    creator = NULL;
   }
 
   return creator;
@@ -121,8 +129,10 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getPublisher(JNIEnv* env, jobject obj)
   try {
     std::string cPublisher = READER->getPublisher();
     publisher = c2jni(cPublisher, env);
-  } catch (...) {
-    std::cerr << "Unable to get ZIM creator" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM publish");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    publisher = NULL;
   }
   return publisher;
 }
@@ -135,8 +145,10 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getName(JNIEnv* env, jobject obj)
   try {
     std::string cName = READER->getName();
     name = c2jni(cName, env);
-  } catch (...) {
-    std::cerr << "Unable to get ZIM name" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM name");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    name = NULL;
   }
   return name;
 }
@@ -151,11 +163,12 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getFavicon(JNIEnv* env, jobject obj)
     std::string cMime;
     READER->getFavicon(cContent, cMime);
     favicon = c2jni(
-        base64_encode(reinterpret_cast<const unsigned char*>(cContent.c_str()),
-                      cContent.length()),
+        base64_encode(cContent),
         env);
-  } catch (...) {
-    std::cerr << "Unable to get ZIM favicon" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM favicon");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    favicon = NULL;
   }
   return favicon;
 }
@@ -168,8 +181,10 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getDate(JNIEnv* env, jobject obj)
   try {
     std::string cDate = READER->getDate();
     date = c2jni(cDate, env);
-  } catch (...) {
-    std::cerr << "Unable to get ZIM date" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM date");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    date = NULL;
   }
   return date;
 }
@@ -182,8 +197,10 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getLanguage(JNIEnv* env, jobject obj)
   try {
     std::string cLanguage = READER->getLanguage();
     language = c2jni(cLanguage, env);
-  } catch (...) {
-    std::cerr << "Unable to get ZIM language" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get ZIM language");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    language = NULL;
   }
 
   return language;
@@ -199,8 +216,10 @@ JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getMimeType(
     auto entry = READER->getEntryFromEncodedPath(cUrl);
     auto cMimeType = entry.getMimetype();
     mimeType = c2jni(cMimeType, env);
-  } catch (...) {
-    std::cerr << "Unable to get mime-type for url " << cUrl << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get mime-type for url: %s", cUrl.c_str());
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    mimeType = NULL;
   }
   return mimeType;
 }
@@ -230,8 +249,9 @@ JNIEXPORT jbyteArray JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getContent(
 
     setStringObjValue(entry.getMimetype(), mimeTypeObj, env);
     setStringObjValue(entry.getTitle(), titleObj, env);
-  } catch (...) {
-    std::cerr << "Unable to get content for url " << cUrl << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get content for url: %s", cUrl.c_str());
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
   }
 
   return data;
@@ -261,9 +281,9 @@ JNIEXPORT jbyteArray JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getContentPa
           data, 0, cLen, reinterpret_cast<const jbyte*>(blob.data()));
       setIntObjValue(cLen, sizeObj, env);
     }
-  } catch (...) {
-     std::cerr << "Unable to get partial content for url " << cUrl
-               << "(" << cOffset << ":" << cLen << ")" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get partial content for url: %s (%u : %u)", cUrl.c_str(), cOffset, cLen);
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
   }
   return data;
 }
@@ -283,10 +303,9 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getDirectAccessInformation(
     entry = entry.getFinalEntry();
     auto part_info = entry.getDirectAccessInfo();
     setPairObjValue(part_info.first, part_info.second, pair, env);
-  } catch (...) {
-    std::cerr << "Unable to locate direct access information for url " << cUrl
-              << std::endl;
-
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get direct access info for url: %s", cUrl.c_str());
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
   }
   return pair;
 }
@@ -305,9 +324,9 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_searchSuggestions(JNIEnv* env,
     if (READER->searchSuggestionsSmart(cPrefix, cCount)) {
       retVal = JNI_TRUE;
     }
-  } catch (...) {
-    std::cerr << "Unable to search suggestions for pattern " << cPrefix
-              << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", "Unable to get search results for pattern: %s", cPrefix.c_str());
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", e.what());
   }
 
   return retVal;
@@ -326,8 +345,9 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getNextSuggestion(JNIEnv* env,
       setStringObjValue(cTitle, titleObj, env);
       retVal = JNI_TRUE;
     }
-  } catch (...) {
-    std::cerr << "Unable to get next suggestion" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", "Unable to get next suggestion");
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", e.what());
   }
 
   return retVal;
@@ -346,8 +366,9 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getPageUrlFromTitle(JNIEnv* env,
     entry = entry.getFinalEntry();
     setStringObjValue(entry.getPath(), urlObj, env);
     return JNI_TRUE;
-  } catch (...) {
-    std::cerr << "Unable to get URL for title " << cTitle << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", "Unable to get url for title %s: ", cTitle.c_str());
+    __android_log_print(ANDROID_LOG_WARN, "kiwix", e.what());
   }
 
   return JNI_FALSE;
@@ -361,8 +382,10 @@ JNIEXPORT jstring JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getTitle(
   try {
     std::string cTitle = READER->getTitle();
     title = c2jni(cTitle, env);
-  } catch (...) {
-    std::cerr << "Unable to get ZIM title" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get zim title");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    title = NULL;
   }
   return title;
 }
@@ -375,8 +398,10 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getDescription(JNIEnv* env, jobject obj)
   try {
     std::string cDescription = READER->getDescription();
     description = c2jni(cDescription, env);
-  } catch (...) {
-    std::cerr << "Unable to get ZIM description" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get zim description");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
+    description = NULL;
   }
   return description;
 }
@@ -391,9 +416,9 @@ JNIEXPORT jboolean JNICALL Java_org_kiwix_kiwixlib_JNIKiwixReader_getRandomPage(
     std::string cUrl = READER->getRandomPage().getPath();
     setStringObjValue(cUrl, urlObj, env);
     retVal = JNI_TRUE;
-  } catch (...) {
-    std::cerr << "Unable to get random page" << std::endl;
+  } catch (std::exception& e) {
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", "Unable to get random page");
+    __android_log_print(ANDROID_LOG_ERROR, "kiwix", e.what());
   }
-
   return retVal;
 }

src/android/meson.build

@@ -1,8 +1,6 @@
 
-jni_generator = find_program('gen_kiwix.sh')
-
 kiwix_jni = custom_target('jni',
-  input: ['org/kiwix/kiwixlib/JNIKiwix.java',
+  input: ['org/kiwix/kiwixlib/JNIICU.java',
           'org/kiwix/kiwixlib/JNIKiwixReader.java',
           'org/kiwix/kiwixlib/JNIKiwixSearcher.java',
           'org/kiwix/kiwixlib/JNIKiwixInt.java',
@@ -14,11 +12,11 @@ kiwix_jni = custom_target('jni',
            'org_kiwix_kiwixlib_JNIKiwixReader.h',
            'org_kiwix_kiwixlib_JNIKiwixSearcher.h',
            'org_kiwix_kiwixlib_JNIKiwixSearcher_Result.h'],
-  command:[jni_generator, '@INPUT@']
+  command:['javac', '-d', '@OUTDIR@', '-h', '@OUTDIR@', '@INPUT@']
 )
 
 kiwix_sources += [
-    'android/kiwix.cpp',
+    'android/kiwixicu.cpp',
     'android/kiwixreader.cpp',
     'android/kiwixsearcher.cpp',
     kiwix_jni]

src/android/org/kiwix/kiwixlib/JNIICU.java

@@ -1,5 +1,6 @@
 /*
- * Copyright 2012 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright (C) 2017 Matthieu Gautier <mgautier@kymeria.fr>
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU  General Public License as published by
@@ -17,33 +18,9 @@
  * MA 02110-1301, USA.
  */
 
-#ifndef KIWIX_NETWORKTOOLS_H
-#define KIWIX_NETWORKTOOLS_H
+package org.kiwix.kiwixlib;
 
-#ifdef _WIN32
-#include <winsock2.h>
-#include <ws2tcpip.h>
-#else
-#include <net/if.h>
-#include <netdb.h>
-#include <stdio.h>
-#include <stdlib.h>
-#include <string.h>
-#include <sys/ioctl.h>
-#include <sys/socket.h>
-#include <sys/types.h>
-#include <unistd.h>
-#endif
-
-#include <iostream>
-#include <map>
-#include <string>
-#include <vector>
-
-namespace kiwix
+public class JNIICU
 {
-std::map<std::string, std::string> getNetworkInterfaces();
-std::string getBestPublicIp();
+  static public native void setDataDirectory(String icuDataDir);
 }
-
-#endif

src/android/org/kiwix/kiwixlib/JNIKiwix.java

@@ -20,12 +20,17 @@
 
 package org.kiwix.kiwixlib;
 
-import org.kiwix.kiwixlib.JNIKiwixReader;
-import org.kiwix.kiwixlib.JNIKiwixString;
+import android.content.Context;
+import com.getkeepsafe.relinker.ReLinker;
+import org.kiwix.kiwixlib.JNIICU;
 
 public class JNIKiwix
 {
-  static { System.loadLibrary("kiwix"); }
+  public JNIKiwix(final Context context){
+    ReLinker.loadLibrary(context, "kiwix");
+  }
 
-  public native void setDataDirectory(String icuDataDir);
+  public void setDataDirectory(String icuDataDir) {
+    JNIICU.setDataDirectory(icuDataDir);
+  }
 }

src/aria2.cpp

@@ -0,0 +1,243 @@
+
+
+#include "aria2.h"
+#include "xmlrpc.h"
+#include <sstream>
+#include <thread>
+#include <chrono>
+#include <tools/otherTools.h>
+#include <tools/pathTools.h>
+#include <downloader.h> // For AriaError
+
+#ifdef _WIN32
+# define ARIA2_CMD "aria2c.exe"
+#else
+# define ARIA2_CMD "aria2c"
+#endif
+
+
+namespace kiwix {
+
+Aria2::Aria2():
+  mp_aria(nullptr),
+  m_port(42042),
+  m_secret("kiwixariarpc"),
+  mp_curl(nullptr),
+  m_lock(PTHREAD_MUTEX_INITIALIZER)
+{
+  m_downloadDir = getDataDirectory();
+  makeDirectory(m_downloadDir);
+  std::vector<const char*> callCmd;
+
+  std::string rpc_port = "--rpc-listen-port=" + to_string(m_port);
+  std::string download_dir = "--dir=" + getDataDirectory();
+  std::string session_file = appendToDirectory(getDataDirectory(), "kiwix.session");
+  std::string session = "--save-session=" + session_file;
+  std::string inputFile = "--input-file=" + session_file;
+//  std::string log_dir = "--log=\"" + logDir + "\"";
+#ifdef _WIN32
+  int pid = GetCurrentProcessId();
+#else
+  pid_t pid = getpid();
+#endif
+  std::string stop_with_pid = "--stop-with-process=" + to_string(pid);
+  std::string rpc_secret = "--rpc-secret=" + m_secret;
+  m_secret = "token:"+m_secret;
+
+  std::string aria2cmd = appendToDirectory(
+    removeLastPathElement(getExecutablePath(), true, true),
+    ARIA2_CMD);
+  if (fileExists(aria2cmd)) {
+    // A local aria2c exe exists (packaged with kiwix-desktop), use it.
+    callCmd.push_back(aria2cmd.c_str());
+  } else {
+    // Try to use a potential installed aria2c.
+    callCmd.push_back(ARIA2_CMD);
+  }
+  callCmd.push_back("--enable-rpc");
+  callCmd.push_back(rpc_secret.c_str());
+  callCmd.push_back(rpc_port.c_str());
+  callCmd.push_back(download_dir.c_str());
+  if (fileExists(session_file)) {
+    callCmd.push_back(inputFile.c_str());
+  }
+  callCmd.push_back(session.c_str());
+//  callCmd.push_back(log_dir.c_str());
+  callCmd.push_back("--auto-save-interval=10");
+  callCmd.push_back(stop_with_pid.c_str());
+  callCmd.push_back("--allow-overwrite=true");
+  callCmd.push_back("--dht-entry-point=router.bittorrent.com:6881");
+  callCmd.push_back("--dht-entry-point6=router.bittorrent.com:6881");
+  callCmd.push_back("--quiet=true");
+  callCmd.push_back("--bt-enable-lpd=true");
+  callCmd.push_back("--always-resume=true");
+  callCmd.push_back("--max-concurrent-downloads=42");
+  callCmd.push_back("--rpc-max-request-size=6M");
+  callCmd.push_back("--file-allocation=none");
+  mp_aria = Subprocess::run(callCmd);
+  mp_curl = curl_easy_init();
+  curl_easy_setopt(mp_curl, CURLOPT_URL, "http://localhost/rpc");
+  curl_easy_setopt(mp_curl, CURLOPT_PORT, m_port);
+  curl_easy_setopt(mp_curl, CURLOPT_POST, 1L);
+
+  int watchdog = 50;
+  while(--watchdog) {
+    sleep(10);
+    auto res = curl_easy_perform(mp_curl);
+    if (res == CURLE_OK) {
+      break;
+    }
+  }
+  if (!watchdog) {
+    curl_easy_cleanup(mp_curl);
+    throw std::runtime_error("Cannot connect to aria2c rpc");
+  }
+}
+
+Aria2::~Aria2()
+{
+  curl_easy_cleanup(mp_curl);
+}
+
+void Aria2::close()
+{
+  saveSession();
+  shutdown();
+}
+
+size_t write_callback_to_iss(char* ptr, size_t size, size_t nmemb, void* userdata)
+{
+  auto str = static_cast<std::stringstream*>(userdata);
+  str->write(ptr, nmemb);
+  return nmemb;
+}
+
+std::string Aria2::doRequest(const MethodCall& methodCall)
+{
+  pthread_mutex_lock(&m_lock);
+  auto requestContent = methodCall.toString();
+  std::stringstream stringstream;
+  CURLcode res;
+  curl_easy_setopt(mp_curl, CURLOPT_POSTFIELDSIZE, requestContent.size());
+  curl_easy_setopt(mp_curl, CURLOPT_POSTFIELDS, requestContent.c_str());
+  curl_easy_setopt(mp_curl, CURLOPT_WRITEFUNCTION, &write_callback_to_iss);
+  curl_easy_setopt(mp_curl, CURLOPT_WRITEDATA, &stringstream);
+  res = curl_easy_perform(mp_curl);
+  if (res == CURLE_OK) {
+    long response_code;
+    curl_easy_getinfo(mp_curl, CURLINFO_RESPONSE_CODE, &response_code);
+    pthread_mutex_unlock(&m_lock);
+    if (response_code != 200) {
+      throw std::runtime_error("Invalid return code from aria");
+    }
+    auto responseContent = stringstream.str();
+    MethodResponse response(responseContent);
+    if (response.isFault()) {
+      throw AriaError(response.getFault().getFaultString());
+    }
+    return responseContent;
+  }
+  pthread_mutex_unlock(&m_lock);
+  throw std::runtime_error("Cannot perform request");
+}
+
+std::string Aria2::addUri(const std::vector<std::string>& uris)
+{
+  MethodCall methodCall("aria2.addUri", m_secret);
+  auto uriParams = methodCall.newParamValue().getArray();
+  for (auto& uri : uris) {
+    uriParams.addValue().set(uri);
+  }
+  auto ret = doRequest(methodCall);
+  MethodResponse response(ret);
+  return response.getParamValue(0).getAsS();
+}
+
+std::string Aria2::tellStatus(const std::string& gid, const std::vector<std::string>& statusKey)
+{
+  MethodCall methodCall("aria2.tellStatus", m_secret);
+  methodCall.newParamValue().set(gid);
+  if (!statusKey.empty()) {
+    auto statusArray = methodCall.newParamValue().getArray();
+    for (auto& key : statusKey) {
+      statusArray.addValue().set(key);
+    }
+  }
+  return doRequest(methodCall);
+}
+
+std::vector<std::string> Aria2::tellActive()
+{
+  MethodCall methodCall("aria2.tellActive", m_secret);
+  auto statusArray = methodCall.newParamValue().getArray();
+  statusArray.addValue().set(std::string("gid"));
+  auto responseContent = doRequest(methodCall);
+  MethodResponse response(responseContent);
+  std::vector<std::string> activeGID;
+  int index = 0;
+  while(true) {
+    try {
+      auto structNode = response.getParamValue(0).getArray().getValue(index++).getStruct();
+      auto gidNode = structNode.getMember("gid");
+      activeGID.push_back(gidNode.getValue().getAsS());
+    } catch (InvalidRPCNode& e) { break; }
+  }
+  return activeGID;
+}
+
+std::vector<std::string> Aria2::tellWaiting()
+{
+  MethodCall methodCall("aria2.tellWaiting", m_secret);
+  methodCall.newParamValue().set(0);
+  methodCall.newParamValue().set(99); // max number of downloads to be returned, don't know how to set this properly assumed that there will not be more than 99 paused downloads.
+  auto statusArray = methodCall.newParamValue().getArray();
+  statusArray.addValue().set(std::string("gid"));
+  auto responseContent = doRequest(methodCall);
+  MethodResponse response(responseContent);
+  std::vector<std::string> waitingGID;
+  int index = 0;
+  while(true) {
+    try {
+      auto structNode = response.getParamValue(0).getArray().getValue(index++).getStruct();
+      auto gidNode = structNode.getMember("gid");
+      waitingGID.push_back(gidNode.getValue().getAsS());
+    } catch (InvalidRPCNode& e) { break; }
+  }
+  return waitingGID;
+}
+
+void Aria2::saveSession()
+{
+  MethodCall methodCall("aria2.saveSession", m_secret);
+  doRequest(methodCall);
+  std::cout << "session saved" << std::endl;
+}
+
+void Aria2::shutdown()
+{
+  MethodCall methodCall("aria2.shutdown", m_secret);
+  doRequest(methodCall);
+}
+
+void Aria2::pause(const std::string& gid)
+{
+    MethodCall methodCall("aria2.pause", m_secret);
+    methodCall.newParamValue().set(gid);
+    doRequest(methodCall);
+}
+
+void Aria2::unpause(const std::string& gid)
+{
+    MethodCall methodCall("aria2.unpause", m_secret);
+    methodCall.newParamValue().set(gid);
+    doRequest(methodCall);
+}
+
+void Aria2::remove(const std::string& gid)
+{
+    MethodCall methodCall("aria2.remove", m_secret);
+    methodCall.newParamValue().set(gid);
+    doRequest(methodCall);
+}
+
+} // end namespace kiwix

src/aria2.h

@@ -0,0 +1,50 @@
+
+
+#ifndef KIWIXLIB_ARIA2_H_
+#define KIWIXLIB_ARIA2_H_
+
+#ifdef _WIN32
+// winsock2.h need to be included before windows.h (included by curl.h)
+# include <winsock2.h>
+#endif
+
+#include "subprocess.h"
+#include "xmlrpc.h"
+
+#include <memory>
+#include <curl/curl.h>
+#include <pthread.h>
+
+namespace kiwix {
+
+class Aria2
+{
+  private:
+    std::unique_ptr<Subprocess> mp_aria;
+    int m_port;
+    std::string m_secret;
+    std::string m_downloadDir;
+    CURL* mp_curl;
+    pthread_mutex_t m_lock;
+
+    std::string doRequest(const MethodCall& methodCall);
+
+  public:
+    Aria2();
+    virtual ~Aria2();
+    void close();
+
+    std::string addUri(const std::vector<std::string>& uri);
+    std::string tellStatus(const std::string& gid, const std::vector<std::string>& statusKey);
+    std::vector<std::string> tellActive();
+    std::vector<std::string> tellWaiting();
+    void saveSession();
+    void shutdown();
+    void pause(const std::string& gid);
+    void unpause(const std::string& gid);
+    void remove(const std::string& gid);
+};
+
+}; //end namespace kiwix
+
+#endif // KIWIXLIB_ARIA2_H_

src/book.cpp

@@ -0,0 +1,196 @@
+/*
+ * Copyright 2011 Emmanuel Engelhart <kelson@kiwix.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#include "book.h"
+#include "reader.h"
+
+#include "tools/base64.h"
+#include "tools/regexTools.h"
+#include "tools/networkTools.h"
+
+#include <pugixml.hpp>
+
+namespace kiwix
+{
+/* Constructor */
+Book::Book() : m_readOnly(false)
+{
+}
+/* Destructor */
+Book::~Book()
+{
+}
+
+bool Book::update(const kiwix::Book& other)
+{
+  if (m_readOnly)
+    return false;
+
+  m_readOnly = other.m_readOnly;
+
+  if (m_path.empty()) {
+    m_path = other.m_path;
+  }
+
+  if (m_url.empty()) {
+    m_url = other.m_url;
+  }
+
+  if (m_tags.empty()) {
+    m_tags = other.m_tags;
+  }
+
+  if (m_name.empty()) {
+    m_name = other.m_name;
+  }
+
+  if (m_faviconMimeType.empty()) {
+    m_favicon = other.m_favicon;
+    m_faviconMimeType = other.m_faviconMimeType;
+  }
+  return true;
+}
+
+void Book::update(const kiwix::Reader& reader)
+{
+  m_path = reader.getZimFilePath();
+  m_id = reader.getId();
+  m_description = reader.getDescription();
+  m_language = reader.getLanguage();
+  m_date = reader.getDate();
+  m_creator = reader.getCreator();
+  m_publisher = reader.getPublisher();
+  m_title = reader.getTitle();
+  m_name = reader.getName();
+  m_tags = reader.getTags();
+  m_origId = reader.getOrigId();
+  m_articleCount = reader.getArticleCount();
+  m_mediaCount = reader.getMediaCount();
+  m_size = static_cast<uint64_t>(reader.getFileSize()) << 10;
+
+  reader.getFavicon(m_favicon, m_faviconMimeType);
+}
+
+#define ATTR(name) node.attribute(name).value()
+void Book::updateFromXml(const pugi::xml_node& node, const std::string& baseDir)
+{
+  m_id = ATTR("id");
+  std::string path = ATTR("path");
+  if (isRelativePath(path)) {
+    path = computeAbsolutePath(baseDir, path);
+  }
+  m_path = path;
+  m_title = ATTR("title");
+  m_name = ATTR("name");
+  m_tags = ATTR("tags");
+  m_description = ATTR("description");
+  m_language = ATTR("language");
+  m_date = ATTR("date");
+  m_creator = ATTR("creator");
+  m_publisher = ATTR("publisher");
+  m_url = ATTR("url");
+  m_origId = ATTR("origId");
+  m_articleCount = strtoull(ATTR("articleCount"), 0, 0);
+  m_mediaCount = strtoull(ATTR("mediaCount"), 0, 0);
+  m_size = strtoull(ATTR("size"), 0, 0) << 10;
+  m_favicon = base64_decode(ATTR("favicon"));
+  m_faviconMimeType = ATTR("faviconMimeType");
+  try {
+    m_downloadId = ATTR("downloadId");
+  } catch(...) {}
+}
+#undef ATTR
+
+
+static std::string fromOpdsDate(const std::string& date)
+{
+  //The opds date use the standard <YYYY>-<MM>-<DD>T<HH>:<mm>:<SS>Z
+  //and we want <YYYY>-<MM>-<DD>. That's easy, let's take the first 10 char
+  return date.substr(0, 10);
+}
+
+
+#define VALUE(name) node.child(name).child_value()
+void Book::updateFromOpds(const pugi::xml_node& node, const std::string& urlHost)
+{
+  m_id = VALUE("id");
+  if (!m_id.compare(0, 9, "urn:uuid:")) {
+    m_id.erase(0, 9);
+  }
+  m_title = VALUE("title");
+  m_description = VALUE("description");
+  m_language = VALUE("language");
+  m_date = fromOpdsDate(VALUE("updated"));
+  m_creator = node.child("author").child("name").child_value();
+  m_tags = VALUE("tags");
+  for(auto linkNode = node.child("link"); linkNode;
+           linkNode = linkNode.next_sibling("link")) {
+    std::string rel = linkNode.attribute("rel").value();
+
+    if (rel == "http://opds-spec.org/acquisition/open-access") {
+      m_url = linkNode.attribute("href").value();
+      m_size = strtoull(linkNode.attribute("length").value(), 0, 0);
+    }
+    if (rel == "http://opds-spec.org/image/thumbnail") {
+      m_faviconUrl = urlHost + linkNode.attribute("href").value();
+      m_faviconMimeType = linkNode.attribute("type").value();
+    }
+ }
+
+}
+#undef VALUE
+
+std::string Book::getHumanReadableIdFromPath()
+{
+  std::string id = m_path;
+  if (!id.empty()) {
+    kiwix::removeAccents(id);
+
+#ifdef _WIN32
+    id = replaceRegex(id, "", "^.*\\\\");
+#else
+    id = replaceRegex(id, "", "^.*/");
+#endif
+
+    id = replaceRegex(id, "", "\\.zim[a-z]*$");
+    id = replaceRegex(id, "_", " ");
+    id = replaceRegex(id, "plus", "\\+");
+  }
+  return id;
+}
+
+void Book::setPath(const std::string& path)
+{
+ m_path = isRelativePath(path)
+   ? computeAbsolutePath(getCurrentDirectory(), path)
+   : path;
+}
+
+const std::string& Book::getFavicon() const {
+  if (m_favicon.empty() && !m_faviconUrl.empty()) {
+    try {
+      m_favicon = download(m_faviconUrl);
+    } catch(...) {
+      std::cerr << "Cannot download favicon from " << m_faviconUrl;
+    }
+  }
+  return m_favicon;
+}
+
+}

src/bookmark.cpp

@@ -0,0 +1,47 @@
+/*
+ * Copyright 2018 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#include "bookmark.h"
+
+#include <pugixml.hpp>
+
+namespace kiwix
+{
+/* Constructor */
+Bookmark::Bookmark()
+{
+}
+
+/* Destructor */
+Bookmark::~Bookmark()
+{
+}
+
+void Bookmark::updateFromXml(const pugi::xml_node& node)
+{
+  auto bookNode = node.child("book");
+  m_bookId = bookNode.child("id").child_value();
+  m_bookTitle = bookNode.child("title").child_value();
+  m_language = bookNode.child("language").child_value();
+  m_date = bookNode.child("date").child_value();
+  m_title = node.child("title").child_value();
+  m_url = node.child("url").child_value();
+}
+
+}

src/common/networkTools.cpp

@@ -1,162 +0,0 @@
-/*
- * Copyright 2012 Emmanuel Engelhart <kelson@kiwix.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU  General Public License as published by
- * the Free Software Foundation; either version 3 of the License, or
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
-
-#include <common/networkTools.h>
-
-
-
-std::map<std::string, std::string> kiwix::getNetworkInterfaces()
-{
-  std::map<std::string, std::string> interfaces;
-
-#ifdef _WIN32
-  SOCKET sd = WSASocket(AF_INET, SOCK_DGRAM, 0, 0, 0, 0);
-  if (sd == (SOCKET)SOCKET_ERROR) {
-    std::cerr << "Failed to get a socket. Error " << WSAGetLastError()
-              << std::endl;
-    return interfaces;
-  }
-
-  INTERFACE_INFO InterfaceList[20];
-  unsigned long nBytesReturned;
-  if (WSAIoctl(sd,
-               SIO_GET_INTERFACE_LIST,
-               0,
-               0,
-               &InterfaceList,
-               sizeof(InterfaceList),
-               &nBytesReturned,
-               0,
-               0)
-      == SOCKET_ERROR) {
-    std::cerr << "Failed calling WSAIoctl: error " << WSAGetLastError()
-              << std::endl;
-    return interfaces;
-  }
-
-  int nNumInterfaces = nBytesReturned / sizeof(INTERFACE_INFO);
-  for (int i = 0; i < nNumInterfaces; ++i) {
-    sockaddr_in* pAddress;
-    pAddress = (sockaddr_in*)&(InterfaceList[i].iiAddress);
-
-    /* Add to the map */
-    std::string interfaceName = std::string(inet_ntoa(pAddress->sin_addr));
-    std::string interfaceIp = std::string(inet_ntoa(pAddress->sin_addr));
-    interfaces.insert(
-        std::pair<std::string, std::string>(interfaceName, interfaceIp));
-  }
-#else
-  /* Get Network interfaces information */
-  char buf[16384];
-  struct ifconf ifconf;
-  int fd = socket(PF_INET, SOCK_DGRAM, 0); /* Only IPV4 */
-  ifconf.ifc_len = sizeof buf;
-  ifconf.ifc_buf = buf;
-  if (ioctl(fd, SIOCGIFCONF, &ifconf) != 0) {
-    perror("ioctl(SIOCGIFCONF)");
-    exit(EXIT_FAILURE);
-  }
-
-  /* Go through each interface */
-  int i;
-  size_t len;
-  struct ifreq* ifreq;
-  ifreq = ifconf.ifc_req;
-  for (i = 0; i < ifconf.ifc_len;) {
-    if (ifreq->ifr_addr.sa_family == AF_INET) {
-      /* Get the network interface ip */
-      char host[128] = {0};
-      const int error = getnameinfo(&(ifreq->ifr_addr),
-                                    sizeof ifreq->ifr_addr,
-                                    host,
-                                    sizeof host,
-                                    0,
-                                    0,
-                                    NI_NUMERICHOST);
-      if (!error) {
-        std::string interfaceName = std::string(ifreq->ifr_name);
-        std::string interfaceIp = std::string(host);
-        /* Add to the map */
-        interfaces.insert(
-            std::pair<std::string, std::string>(interfaceName, interfaceIp));
-      } else {
-        perror("getnameinfo()");
-      }
-    }
-
-/* some systems have ifr_addr.sa_len and adjust the length that
- * way, but not mine. weird */
-#ifndef __linux__
-    len = IFNAMSIZ + ifreq->ifr_addr.sa_len;
-#else
-    len = sizeof *ifreq;
-#endif
-    ifreq = (struct ifreq*)((char*)ifreq + len);
-    i += len;
-  }
-#endif
-  return interfaces;
-}
-
-std::string kiwix::getBestPublicIp()
-{
-  std::map<std::string, std::string> interfaces = kiwix::getNetworkInterfaces();
-
-#ifndef _WIN32
-  const char* const prioritizedNames[]
-      = {"eth0", "eth1", "wlan0", "wlan1", "en0", "en1"};
-  const int count = (sizeof prioritizedNames) / (sizeof prioritizedNames[0]);
-  for (int i = 0; i < count; ++i) {
-    std::map<std::string, std::string>::const_iterator it
-        = interfaces.find(prioritizedNames[i]);
-    if (it != interfaces.end()) {
-      return it->second;
-    }
-  }
-#endif
-
-  for (std::map<std::string, std::string>::iterator iter = interfaces.begin();
-       iter != interfaces.end();
-       ++iter) {
-    std::string interfaceIp = iter->second;
-    if (interfaceIp.length() >= 7 && interfaceIp.substr(0, 7) == "192.168") {
-      return interfaceIp;
-    }
-  }
-
-  for (std::map<std::string, std::string>::iterator iter = interfaces.begin();
-       iter != interfaces.end();
-       ++iter) {
-    std::string interfaceIp = iter->second;
-    if (interfaceIp.length() >= 7 && interfaceIp.substr(0, 7) == "172.16.") {
-      return interfaceIp;
-    }
-  }
-
-  for (std::map<std::string, std::string>::iterator iter = interfaces.begin();
-       iter != interfaces.end();
-       ++iter) {
-    std::string interfaceIp = iter->second;
-    if (interfaceIp.length() >= 3 && interfaceIp.substr(0, 3) == "10.") {
-      return interfaceIp;
-    }
-  }
-
-  return "127.0.0.1";
-}

src/config.h.in

@@ -1,4 +1,3 @@
 
 #mesondefine VERSION
 
-#mesondefine ENABLE_CTPP2

src/ctpp2/CTPP2VMStringLoader.cpp

@@ -1,210 +0,0 @@
-/*
- * Copyright 2013 Renaud Gaudin <reg@kiwix.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU  General Public License as published by
- * the Free Software Foundation; either version 3 of the License, or
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
-
-#include <ctpp2/CTPP2VMStringLoader.hpp>
-
-namespace CTPP // C++ Template Engine
-{
-
-//
-// Convert byte order
-//
-static void ConvertExecutable(VMExecutable * oCore)
-{
-	// Code entry point
-	oCore -> entry_point = Swap32(oCore -> entry_point);
-	// Offset of code segment
-	oCore -> code_offset = Swap32(oCore -> code_offset);
-
-	// Code segment size
-	oCore -> code_size   = Swap32(oCore -> code_size);
-
-	// Offset of static text segment
-	oCore -> syscalls_offset    = Swap32(oCore -> syscalls_offset);
-	// Static text segment size
-	oCore -> syscalls_data_size = Swap32(oCore -> syscalls_data_size);
-
-	// Offset of static text index segment
-	oCore -> syscalls_index_offset = Swap32(oCore -> syscalls_index_offset);
-	// Static text index segment size
-	oCore -> syscalls_index_size   = Swap32(oCore -> syscalls_index_size);
-
-	// Offset of static data segment
-	oCore -> static_data_offset    = Swap32(oCore -> static_data_offset);
-
-	// Static data segment size
-	oCore -> static_data_data_size = Swap32(oCore -> static_data_data_size);
-
-	// Offset of static text segment
-	oCore -> static_text_offset    = Swap32(oCore -> static_text_offset);
-	// Static text segment size
-	oCore -> static_text_data_size = Swap32(oCore -> static_text_data_size);
-
-	// Offset of static text index segment
-	oCore -> static_text_index_offset = Swap32(oCore -> static_text_index_offset);
-	// Static text index segment size
-	oCore -> static_text_index_size   = Swap32(oCore -> static_text_index_size);
-
-	// Version 2.2+
-	// Offset of static data bit index
-	oCore -> static_data_bit_index_offset = Swap32(oCore -> static_data_bit_index_offset);
-	/// Offset of static data bit index
-	oCore -> static_data_bit_index_size = Swap32(oCore -> static_data_bit_index_size);
-
-	// Platform
-	oCore -> platform      = Swap64(oCore -> platform);
-
-	// Ugly-jolly hack!
-	// ... dereferencing type-punned pointer will break strict-aliasing rules ...
-	UINT_64 iTMP;
-	memcpy(&iTMP, &(oCore -> ieee754double), sizeof(UINT_64));
-	iTMP = Swap64(iTMP);
-	memcpy(&(oCore -> ieee754double), &iTMP, sizeof(UINT_64));
-
-	// Cyclic Redundancy Check
-	oCore -> crc = 0;
-
-	// Convert data structures
-
-	// Convert code segment
-	VMInstruction * pInstructions = const_cast<VMInstruction *>(VMExecutable::GetCodeSeg(oCore));
-	UINT_32 iI = 0;
-	UINT_32 iSteps = oCore -> code_size / sizeof(VMInstruction);
-	for(iI = 0; iI < iSteps; ++iI)
-	{
-		pInstructions -> instruction = Swap32(pInstructions -> instruction);
-		pInstructions -> argument    = Swap32(pInstructions -> argument);
-		pInstructions -> reserved    = Swap64(pInstructions -> reserved);
-		++pInstructions;
-	}
-
-	// Convert syscalls index
-	TextDataIndex * pTextIndex = const_cast<TextDataIndex *>(VMExecutable::GetSyscallsIndexSeg(oCore));
-	iSteps = oCore -> syscalls_index_size / sizeof(TextDataIndex);
-	for(iI = 0; iI < iSteps; ++iI)
-	{
-		pTextIndex -> offset = Swap32(pTextIndex -> offset);
-		pTextIndex -> length = Swap32(pTextIndex -> length);
-		++pTextIndex;
-	}
-
-	// Convert static text index
-	pTextIndex = const_cast<TextDataIndex *>(VMExecutable::GetStaticTextIndexSeg(oCore));
-	iSteps = oCore -> static_text_index_size / sizeof(TextDataIndex);
-	for(iI = 0; iI < iSteps; ++iI)
-	{
-		pTextIndex -> offset = Swap32(pTextIndex -> offset);
-		pTextIndex -> length = Swap32(pTextIndex -> length);
-		++pTextIndex;
-	}
-
-	// Convert static data
-	StaticDataVar * pStaticDataVar = const_cast<StaticDataVar *>(VMExecutable::GetStaticDataSeg(oCore));
-	iSteps = oCore -> static_data_data_size / sizeof(StaticDataVar);
-	for(iI = 0; iI < iSteps; ++iI)
-	{
-		(*pStaticDataVar).i_data = Swap64((*pStaticDataVar).i_data);
-		++pStaticDataVar;
-	}
-}
-
-//
-// Constructor
-//
-VMStringLoader::VMStringLoader(CCHAR_P rawContent, size_t rawContentSize)
-{
-    oCore = (VMExecutable *)malloc(rawContentSize + 1);
-    memcpy(oCore, rawContent, rawContentSize);
-
-	if (oCore -> magic[0] == 'C' &&
-	    oCore -> magic[1] == 'T' &&
-	    oCore -> magic[2] == 'P' &&
-	    oCore -> magic[3] == 'P')
-	{
-		// Check version
-		if (oCore -> version[0] >= 1)
-		{
-			// Platform-dependent data (byte order)
-			if (oCore -> platform == 0x4142434445464748ull)
-			{
-#ifdef _DEBUG
-				fprintf(stderr, "Big/Little Endian conversion: Nothing to do\n");
-#endif
-
-				// Nothing to do, only check crc
-				UINT_32 iCRC = oCore -> crc;
-				oCore -> crc = 0;
-
-				// Calculate CRC of file
-                // KELSON: next line used to refer to oStat.st_size
-                // changed it to rawContentSize
-				if (iCRC != crc32((UCCHAR_P)oCore, rawContentSize))
-				{
-					free(oCore);
-					throw CTPPLogicError("CRC checksum invalid");
-				}
-			}
-			// Platform-dependent data (byte order)
-			else if (oCore -> platform == 0x4847464544434241ull)
-			{
-				// Need to reconvert data
-#ifdef _DEBUG
-				fprintf(stderr, "Big/Little Endian conversion: Need to reconvert core\n");
-#endif
-				ConvertExecutable(oCore);
-			}
-			else
-			{
-				free(oCore);
-				throw CTPPLogicError("Conversion of middle-end architecture does not supported.");
-			}
-
-			// Check IEEE 754 format
-			if (oCore -> ieee754double != 15839800103804824402926068484019465486336.0)
-			{
-				free(oCore);
-				throw CTPPLogicError("IEEE 754 format is broken, cannot convert file");
-			}
-		}
-
-		pVMMemoryCore = new VMMemoryCore(oCore);
-	}
-	else
-	{
-		free(oCore);
-		throw CTPPLogicError("Not an CTPP bytecode file.");
-	}
-}
-
-//
-// Get ready-to-run program
-//
-const VMMemoryCore * VMStringLoader::GetCore() const { return pVMMemoryCore; }
-
-//
-// A destructor
-//
-VMStringLoader::~VMStringLoader() throw()
-{
-	delete pVMMemoryCore;
-	free(oCore);
-}
-
-} // namespace CTPP
-// End.

src/downloader.cpp

@@ -18,95 +18,179 @@
  */
 
 #include "downloader.h"
-#include "common/pathTools.h"
+#include "tools/pathTools.h"
+
+#include <algorithm>
+#include <thread>
+#include <chrono>
 
-#include <unistd.h>
 #include <iostream>
 
+#include "aria2.h"
+#include "xmlrpc.h"
+#include "tools/otherTools.h"
+#include <pugixml.hpp>
+
 namespace kiwix
 {
 
-pthread_mutex_t Downloader::globalLock = PTHREAD_MUTEX_INITIALIZER;
+void Download::updateStatus(bool follow)
+{
+  if (m_status == Download::K_REMOVED)
+    return;
+  static std::vector<std::string> statusKey = {"status", "files", "totalLength",
+                                               "completedLength", "followedBy",
+                                               "downloadSpeed", "verifiedLength"};
+  std::string strStatus;
+  if(follow && !m_followedBy.empty()) {
+    strStatus = mp_aria->tellStatus(m_followedBy, statusKey);
+  } else {
+    strStatus = mp_aria->tellStatus(m_did, statusKey);
+  }
+//  std::cout << strStatus << std::endl;
+  MethodResponse response(strStatus);
+  if (response.isFault()) {
+    m_status = Download::K_UNKNOWN;
+    return;
+  }
+  auto structNode = response.getParams().getParam(0).getValue().getStruct();
+  auto _status = structNode.getMember("status").getValue().getAsS();
+  auto status = _status == "active" ? Download::K_ACTIVE
+              : _status == "waiting" ? Download::K_WAITING
+              : _status == "paused" ? Download::K_PAUSED
+              : _status == "error" ? Download::K_ERROR
+              : _status == "complete" ? Download::K_COMPLETE
+              : _status == "removed" ? Download::K_REMOVED
+              : Download::K_UNKNOWN;
+  if (status == K_COMPLETE) {
+    try {
+      auto followedByMember = structNode.getMember("followedBy");
+      m_followedBy = followedByMember.getValue().getArray().getValue(0).getAsS();
+      if (follow) {
+        status = K_ACTIVE;
+        updateStatus(true);
+        return;
+      }
+    } catch (InvalidRPCNode& e) { }
+  }
+  m_status = status;
+  m_totalLength = extractFromString<uint64_t>(structNode.getMember("totalLength").getValue().getAsS());
+  m_completedLength = extractFromString<uint64_t>(structNode.getMember("completedLength").getValue().getAsS());
+  m_downloadSpeed = extractFromString<uint64_t>(structNode.getMember("downloadSpeed").getValue().getAsS());
+  try {
+    auto verifiedLengthValue = structNode.getMember("verifiedLength").getValue();
+    m_verifiedLength = extractFromString<uint64_t>(verifiedLengthValue.getAsS());
+  } catch (InvalidRPCNode& e) { m_verifiedLength = 0; }
+  auto filesMember = structNode.getMember("files");
+  auto fileStruct = filesMember.getValue().getArray().getValue(0).getStruct();
+  m_path = fileStruct.getMember("path").getValue().getAsS();
+  auto urisArray = fileStruct.getMember("uris").getValue().getArray();
+  int index = 0;
+  m_uris.clear();
+  while(true) {
+    try {
+      auto uriNode = urisArray.getValue(index++).getStruct().getMember("uri");
+      m_uris.push_back(uriNode.getValue().getAsS());
+    } catch(InvalidRPCNode& e) { break; }
+  }
+}
 
+void Download::resumeDownload()
+{
+    if (!m_followedBy.empty())
+        mp_aria->unpause(m_followedBy);
+    else
+        mp_aria->unpause(m_did);
+    updateStatus(true);
+}
+
+void Download::pauseDownload()
+{
+    if (!m_followedBy.empty())
+        mp_aria->pause(m_followedBy);
+    else
+        mp_aria->pause(m_did);
+    updateStatus(true);
+}
+
+void Download::cancelDownload()
+{
+    if (!m_followedBy.empty())
+        mp_aria->remove(m_followedBy);
+    else
+        mp_aria->remove(m_did);
+    m_status = Download::K_REMOVED;
+}
 
 /* Constructor */
-Downloader::Downloader()
+Downloader::Downloader() :
+  mp_aria(new Aria2())
 {
-  aria2::SessionConfig config;
-  config.downloadEventCallback = Downloader::downloadEventCallback;
-  config.userData = this;
-  tmpDir = makeTmpDirectory();
-  aria2::KeyVals options;
-  options.push_back(std::pair<std::string, std::string>("dir", tmpDir));
-  session = aria2::sessionNew(options, config);
+  for (auto gid : mp_aria->tellActive()) {
+    m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
+    m_knownDownloads[gid]->updateStatus();
+  }
+  for (auto gid : mp_aria->tellWaiting()) {
+    m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
+    m_knownDownloads[gid]->updateStatus();
+  }
 }
 
 
 /* Destructor */
 Downloader::~Downloader()
 {
-  aria2::sessionFinal(session);
-  rmdir(tmpDir.c_str());
 }
 
-
-int Downloader::downloadEventCallback(aria2::Session* session,
-                                      aria2::DownloadEvent event,
-                                      aria2::A2Gid gid,
-                                      void* userData)
+void Downloader::close()
 {
-  Downloader* downloader = static_cast<Downloader*>(userData);
-
-  auto fileHandle = downloader->fileHandle;
-  auto dh = aria2::getDownloadHandle(session, gid);
-
-  if (!dh) {
-    return 0;
-  }
-
-  switch (event) {
-    case aria2::EVENT_ON_DOWNLOAD_COMPLETE:
-      {
-        if (dh->getNumFiles() > 0) {
-          auto f = dh->getFile(1);
-          fileHandle->path = f.path;
-          fileHandle->success = true;
-        }
-      }
-      break;
-    case aria2::EVENT_ON_DOWNLOAD_ERROR:
-      {
-        fileHandle->success = false;
-      }
-      break;
-    default:
-      break;
-  }
-  aria2::deleteDownloadHandle(dh);
-  return 0;
+  mp_aria->close();
 }
 
-DownloadedFile Downloader::download(const std::string& url) {
-  pthread_mutex_lock(&globalLock);
-  DownloadedFile fileHandle;
+std::vector<std::string> Downloader::getDownloadIds() {
+  std::vector<std::string> ret;
+  for(auto& p:m_knownDownloads) {
+    ret.push_back(p.first);
+  }
+  return ret;
+}
+
+Download* Downloader::startDownload(const std::string& uri)
+{
+  for (auto& p: m_knownDownloads) {
+    auto& d = p.second;
+    auto& uris = d->getUris();
+    if (std::find(uris.begin(), uris.end(), uri) != uris.end())
+      return d.get();
+  }
+  std::vector<std::string> uris = {uri};
+  auto gid = mp_aria->addUri(uris);
+  m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
+  return m_knownDownloads[gid].get();
+}
+
+Download* Downloader::getDownload(const std::string& did)
+{
   try {
-    std::vector<std::string> uris = {url};
-    aria2::KeyVals options;
-    aria2::A2Gid gid;
-    int ret;
-    DownloadedFile fileHandle;
-    
-    ret = aria2::addUri(session, &gid, uris, options);
-    if (ret < 0) {
-      std::cerr << "Failed to download" << std::endl;
-    } else {
-      this->fileHandle = &fileHandle;
-      aria2::run(session, aria2::RUN_DEFAULT);
+    m_knownDownloads.at(did).get()->updateStatus(true);
+    return m_knownDownloads.at(did).get();
+  } catch(exception& e) {
+    for (auto gid : mp_aria->tellActive()) {
+      if (gid == did) {
+        m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
+        m_knownDownloads.at(gid).get()->updateStatus(true);
+        return m_knownDownloads[gid].get();
+      }
     }
-  } catch (...) {};
-  this->fileHandle = nullptr;
-  pthread_mutex_unlock(&globalLock);
-  return fileHandle;
+    for (auto gid : mp_aria->tellWaiting()) {
+      if (gid == did) {
+        m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
+        m_knownDownloads.at(gid).get()->updateStatus(true);
+        return m_knownDownloads[gid].get();
+      }
+  }
+    throw e;
+  }
 }
 
 }

src/entry.cpp

@@ -122,7 +122,6 @@ Entry Entry::getFinalEntry() const
   if (final_article.good()) {
     return final_article;
   }
-
   int loopCounter = 42;
   final_article = article;
   while (final_article.isRedirect() && loopCounter--) {
@@ -131,7 +130,10 @@ Entry Entry::getFinalEntry() const
       throw NoEntry();
     }
   }
-
+  // Prevent infinite loops.
+  if (final_article.isRedirect()) {
+    throw NoEntry();
+  }
   return final_article;
 }
 

src/kiwixserve.cpp

@@ -0,0 +1,69 @@
+#include "kiwixserve.h"
+#include "subprocess.h"
+
+#ifdef _WIN32
+    # define KIWIXSERVE_CMD "kiwix-serve.exe"
+    # include <windows.h>
+#else
+    # define KIWIXSERVE_CMD "kiwix-serve"
+    # include <unistd.h>
+#endif
+
+#include "tools/pathTools.h"
+
+namespace kiwix {
+
+KiwixServe::KiwixServe(int port) : m_port(port)
+{
+}
+
+KiwixServe::~KiwixServe()
+{
+    shutDown();
+}
+
+void KiwixServe::run()
+{
+    #ifdef _WIN32
+        int pid = GetCurrentProcessId();
+    #else
+        pid_t pid = getpid();
+
+    #endif
+
+    std::vector<const char*> callCmd;
+    std::string kiwixServeCmd = appendToDirectory(
+        removeLastPathElement(getExecutablePath(), true, true),
+        KIWIXSERVE_CMD);
+    if (fileExists(kiwixServeCmd)) {
+        // A local kiwix-serve exe exists (packaged with kiwix-desktop), use it.
+        callCmd.push_back(kiwixServeCmd.c_str());
+    } else {
+        // Try to use a potential installed kiwix-serve.
+        callCmd.push_back(KIWIXSERVE_CMD);
+    }
+    std::string libraryPath = getDataDirectory() + "/library.xml";
+    std::string attachProcessOpt = "-a" + to_string(pid);
+    std::string portOpt = "-p" + to_string(m_port);
+    callCmd.push_back(attachProcessOpt.c_str());
+    callCmd.push_back(portOpt.c_str());
+    callCmd.push_back("-l");
+    callCmd.push_back(libraryPath.c_str());
+    mp_kiwixServe = Subprocess::run(callCmd);
+}
+
+void KiwixServe::shutDown()
+{
+    if (mp_kiwixServe)
+        mp_kiwixServe->kill();
+}
+
+bool KiwixServe::isRunning()
+{
+    if (mp_kiwixServe) {
+        return (mp_kiwixServe->isRunning());
+    }
+    return false;
+}
+
+}

src/library.cpp

@@ -18,137 +18,480 @@
  */
 
 #include "library.h"
+#include "book.h"
+#include "libxml_dumper.h"
+
+#include "tools/base64.h"
+#include "tools/regexTools.h"
+#include "tools/pathTools.h"
+
+#include <pugixml.hpp>
+#include <algorithm>
+#include <set>
 
 namespace kiwix
 {
 /* Constructor */
-Book::Book() : readOnly(false)
-{
-}
-/* Destructor */
-Book::~Book()
-{
-}
-/* Sort functions */
-bool Book::sortByLastOpen(const kiwix::Book& a, const kiwix::Book& b)
-{
-  return atoi(a.last.c_str()) > atoi(b.last.c_str());
-}
-
-bool Book::sortByTitle(const kiwix::Book& a, const kiwix::Book& b)
-{
-  return strcmp(a.title.c_str(), b.title.c_str()) < 0;
-}
-
-bool Book::sortByDate(const kiwix::Book& a, const kiwix::Book& b)
-{
-  return strcmp(a.date.c_str(), b.date.c_str()) > 0;
-}
-
-bool Book::sortBySize(const kiwix::Book& a, const kiwix::Book& b)
-{
-  return atoi(a.size.c_str()) < atoi(b.size.c_str());
-}
-
-bool Book::sortByPublisher(const kiwix::Book& a, const kiwix::Book& b)
-{
-  return strcmp(a.publisher.c_str(), b.publisher.c_str()) < 0;
-}
-
-bool Book::sortByCreator(const kiwix::Book& a, const kiwix::Book& b)
-{
-  return strcmp(a.creator.c_str(), b.creator.c_str()) < 0;
-}
-
-bool Book::sortByLanguage(const kiwix::Book& a, const kiwix::Book& b)
-{
-  return strcmp(a.language.c_str(), b.language.c_str()) < 0;
-}
-
-std::string Book::getHumanReadableIdFromPath()
-{
-  std::string id = pathAbsolute;
-  if (!id.empty()) {
-    kiwix::removeAccents(id);
-
-#ifdef _WIN32
-    id = replaceRegex(id, "", "^.*\\\\");
-#else
-    id = replaceRegex(id, "", "^.*/");
-#endif
-
-    id = replaceRegex(id, "", "\\.zim[a-z]*$");
-    id = replaceRegex(id, "_", " ");
-    id = replaceRegex(id, "plus", "\\+");
-  }
-  return id;
-}
-
-/* Constructor */
-Library::Library() : version(KIWIX_LIBRARY_VERSION)
+Library::Library()
 {
 }
 /* Destructor */
 Library::~Library()
 {
 }
+
+
 bool Library::addBook(const Book& book)
 {
   /* Try to find it */
-  std::vector<kiwix::Book>::iterator itr;
-  for (itr = this->books.begin(); itr != this->books.end(); ++itr) {
-    if (itr->id == book.id) {
-      if (!itr->readOnly) {
-        itr->readOnly = book.readOnly;
+  try {
+    auto& oldbook = m_books.at(book.getId());
+    oldbook.update(book);
+    return false;
+  } catch (std::out_of_range&) {
+    m_books[book.getId()] = book;
+    return true;
+  }
+}
 
-        if (itr->path.empty()) {
-          itr->path = book.path;
-        }
+void Library::addBookmark(const Bookmark& bookmark)
+{
+  m_bookmarks.push_back(bookmark);
+}
 
-        if (itr->pathAbsolute.empty()) {
-          itr->pathAbsolute = book.pathAbsolute;
-        }
+bool Library::removeBookmark(const std::string& zimId, const std::string& url)
+{
+  for(auto it=m_bookmarks.begin(); it!=m_bookmarks.end(); it++) {
+    if (it->getBookId() == zimId && it->getUrl() == url) {
+      m_bookmarks.erase(it);
+      return true;
+    }
+  }
+  return false;
+}
 
-        if (itr->url.empty()) {
-          itr->url = book.url;
-        }
 
-        if (itr->tags.empty()) {
-          itr->tags = book.tags;
-        }
 
-        if (itr->name.empty()) {
-          itr->name = book.name;
-        }
+bool Library::removeBookById(const std::string& id)
+{
+  return m_books.erase(id) == 1;
+}
 
-        if (itr->indexPath.empty()) {
-          itr->indexPath = book.indexPath;
-          itr->indexType = book.indexType;
-        }
+Book& Library::getBookById(const std::string& id)
+{
+  return m_books.at(id);
+}
 
-        if (itr->indexPathAbsolute.empty()) {
-          itr->indexPathAbsolute = book.indexPathAbsolute;
-          itr->indexType = book.indexType;
-        }
+unsigned int Library::getBookCount(const bool localBooks,
+                                   const bool remoteBooks)
+{
+  unsigned int result = 0;
+  for (auto& pair: m_books) {
+    auto& book = pair.second;
+    if ((!book.getPath().empty() && localBooks)
+        || (book.getPath().empty() && remoteBooks)) {
+      result++;
+    }
+  }
+  return result;
+}
 
-        if (itr->faviconMimeType.empty()) {
-          itr->favicon = book.favicon;
-          itr->faviconMimeType = book.faviconMimeType;
-        }
+bool Library::writeToFile(const std::string& path)
+{
+  auto baseDir = removeLastPathElement(path, true, false);
+  LibXMLDumper dumper(this);
+  dumper.setBaseDir(baseDir);
+  return writeTextFile(path, dumper.dumpLibXMLContent(getBooksIds()));
+}
+
+bool Library::writeBookmarksToFile(const std::string& path)
+{
+  LibXMLDumper dumper(this);
+  return writeTextFile(path, dumper.dumpLibXMLBookmark());
+}
+
+std::vector<std::string> Library::getBooksLanguages()
+{
+  std::vector<std::string> booksLanguages;
+  std::map<std::string, bool> booksLanguagesMap;
+
+  for (auto& pair: m_books) {
+    auto& book = pair.second;
+    auto& language = book.getLanguage();
+    if (booksLanguagesMap.find(language) == booksLanguagesMap.end()) {
+      if (book.getOrigId().empty()) {
+        booksLanguagesMap[language] = true;
+        booksLanguages.push_back(language);
       }
-
-      return false;
     }
   }
 
-  /* otherwise */
-  this->books.push_back(book);
-  return true;
+  return booksLanguages;
 }
 
-bool Library::removeBookByIndex(const unsigned int bookIndex)
+std::vector<std::string> Library::getBooksCreators()
 {
-  books.erase(books.begin() + bookIndex);
+  std::vector<std::string> booksCreators;
+  std::map<std::string, bool> booksCreatorsMap;
+
+  for (auto& pair: m_books) {
+    auto& book = pair.second;
+    auto& creator = book.getCreator();
+    if (booksCreatorsMap.find(creator) == booksCreatorsMap.end()) {
+      if (book.getOrigId().empty()) {
+        booksCreatorsMap[creator] = true;
+        booksCreators.push_back(creator);
+      }
+    }
+  }
+
+  return booksCreators;
+}
+
+std::vector<std::string> Library::getBooksPublishers()
+{
+  std::vector<std::string> booksPublishers;
+  std::map<std::string, bool> booksPublishersMap;
+
+  for (auto& pair:m_books) {
+    auto& book = pair.second;
+    auto& publisher = book.getPublisher();
+    if (booksPublishersMap.find(publisher) == booksPublishersMap.end()) {
+      if (book.getOrigId().empty()) {
+        booksPublishersMap[publisher] = true;
+        booksPublishers.push_back(publisher);
+      }
+    }
+  }
+
+  return booksPublishers;
+}
+
+std::vector<std::string> Library::getBooksIds()
+{
+  std::vector<std::string> bookIds;
+
+  for (auto& pair: m_books) {
+    bookIds.push_back(pair.first);
+  }
+
+  return bookIds;
+}
+
+std::vector<std::string> Library::filter(const std::string& search)
+{
+  if (search.empty()) {
+    return getBooksIds();
+  }
+
+  return filter(Filter().query(search));
+}
+
+
+std::vector<std::string> Library::filter(const Filter& filter)
+{
+  std::vector<std::string> bookIds;
+  for(auto& pair:m_books) {
+    auto book = pair.second;
+    if(filter.accept(book)) {
+      bookIds.push_back(pair.first);
+    }
+  }
+  return bookIds;
+}
+
+template<supportedListSortBy SORT>
+struct KEY_TYPE {
+  typedef std::string TYPE;
+};
+
+template<>
+struct KEY_TYPE<SIZE> {
+  typedef size_t TYPE;
+};
+
+template<supportedListSortBy sort>
+class Comparator {
+  private:
+    Library* lib;
+    bool     ascending;
+
+    inline typename KEY_TYPE<sort>::TYPE get_key(const std::string& id);
+
+  public:
+    Comparator(Library* lib, bool ascending) : lib(lib), ascending(ascending) {}
+    inline bool operator() (const std::string& id1, const std::string& id2) {
+      if (ascending) {
+        return get_key(id1) < get_key(id2);
+      } else {
+        return get_key(id2) < get_key(id1);
+      }
+    }
+};
+
+template<>
+std::string Comparator<TITLE>::get_key(const std::string& id)
+{
+  return lib->getBookById(id).getTitle();
+}
+
+template<>
+size_t Comparator<SIZE>::get_key(const std::string& id)
+{
+  return lib->getBookById(id).getSize();
+}
+
+template<>
+std::string Comparator<DATE>::get_key(const std::string& id)
+{
+  return lib->getBookById(id).getDate();
+}
+
+template<>
+std::string Comparator<CREATOR>::get_key(const std::string& id)
+{
+  return lib->getBookById(id).getCreator();
+}
+
+template<>
+std::string Comparator<PUBLISHER>::get_key(const std::string& id)
+{
+  return lib->getBookById(id).getPublisher();
+}
+
+void Library::sort(std::vector<std::string>& bookIds, supportedListSortBy sort, bool ascending)
+{
+  switch(sort) {
+    case TITLE:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<TITLE>(this, ascending));
+      break;
+    case SIZE:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<SIZE>(this, ascending));
+      break;
+    case DATE:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<DATE>(this, ascending));
+      break;
+    case CREATOR:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<CREATOR>(this, ascending));
+      break;
+    case PUBLISHER:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<PUBLISHER>(this, ascending));
+      break;
+    default:
+      break;
+  }
+}
+
+
+std::vector<std::string> Library::listBooksIds(
+    int mode,
+    supportedListSortBy sortBy,
+    const std::string& search,
+    const std::string& language,
+    const std::string& creator,
+    const std::string& publisher,
+    const std::vector<std::string>& tags,
+    size_t maxSize) {
+
+  Filter _filter;
+  if (mode & LOCAL)
+    _filter.local(true);
+  if (mode & NOLOCAL)
+    _filter.local(false);
+  if (mode & VALID)
+    _filter.valid(true);
+  if (mode & NOVALID)
+    _filter.valid(false);
+  if (mode & REMOTE)
+    _filter.remote(true);
+  if (mode & NOREMOTE)
+    _filter.remote(false);
+  if (!tags.empty())
+    _filter.acceptTags(tags);
+  if (maxSize != 0)
+    _filter.maxSize(maxSize);
+  if (!language.empty())
+    _filter.lang(language);
+  if (!publisher.empty())
+    _filter.publisher(publisher);
+  if (!creator.empty())
+    _filter.creator(creator);
+  if (!search.empty())
+    _filter.query(search);
+
+  auto bookIds = filter(_filter);
+
+  sort(bookIds, sortBy, true);
+  return bookIds;
+}
+
+Filter::Filter()
+  : activeFilters(0),
+    _maxSize(0)
+{};
+
+#define FLAG(x) (1 << x)
+enum filterTypes {
+  NONE = 0,
+  _LOCAL = FLAG(0),
+  _REMOTE = FLAG(1),
+  _NOLOCAL = FLAG(2),
+  _NOREMOTE = FLAG(3),
+  _VALID = FLAG(4),
+  _NOVALID = FLAG(5),
+  ACCEPTTAGS = FLAG(6),
+  REJECTTAGS = FLAG(7),
+  LANG = FLAG(8),
+  _PUBLISHER = FLAG(9),
+  _CREATOR = FLAG(10),
+  MAXSIZE = FLAG(11),
+  QUERY = FLAG(12),
+};
+
+Filter& Filter::local(bool accept)
+{
+  if (accept) {
+    activeFilters |= _LOCAL;
+    activeFilters &= ~_NOLOCAL;
+  } else {
+    activeFilters |= _NOLOCAL;
+    activeFilters &= ~_LOCAL;
+  }
+  return *this;
+}
+
+Filter& Filter::remote(bool accept)
+{
+  if (accept) {
+    activeFilters |= _REMOTE;
+    activeFilters &= ~_NOREMOTE;
+  } else {
+    activeFilters |= _NOREMOTE;
+    activeFilters &= ~_REMOTE;
+  }
+  return *this;
+}
+
+Filter& Filter::valid(bool accept)
+{
+  if (accept) {
+    activeFilters |= _VALID;
+    activeFilters &= ~_NOVALID;
+  } else {
+    activeFilters |= _NOVALID;
+    activeFilters &= ~_VALID;
+  }
+  return *this;
+}
+
+Filter& Filter::acceptTags(std::vector<std::string> tags)
+{
+  _acceptTags = tags;
+  activeFilters |= ACCEPTTAGS;
+  return *this;
+}
+
+Filter& Filter::rejectTags(std::vector<std::string> tags)
+{
+  _rejectTags = tags;
+  activeFilters |= REJECTTAGS;
+  return *this;
+}
+
+Filter& Filter::lang(std::string lang)
+{
+  _lang = lang;
+  activeFilters |= LANG;
+  return *this;
+}
+
+Filter& Filter::publisher(std::string publisher)
+{
+  _publisher = publisher;
+  activeFilters |= _PUBLISHER;
+  return *this;
+}
+
+Filter& Filter::creator(std::string creator)
+{
+  _creator = creator;
+  activeFilters |= _CREATOR;
+  return *this;
+}
+
+Filter& Filter::maxSize(size_t maxSize)
+{
+  _maxSize = maxSize;
+  activeFilters |= MAXSIZE;
+  return *this;
+}
+
+Filter& Filter::query(std::string query)
+{
+  _query = query;
+  activeFilters |= QUERY;
+  return *this;
+}
+
+#define ACTIVE(X) (activeFilters & (X))
+bool Filter::accept(const Book& book) const
+{
+  auto local = !book.getPath().empty();
+  if (ACTIVE(_LOCAL) && !local)
+    return false;
+  if (ACTIVE(_NOLOCAL) && local)
+    return false;
+  auto valid = book.isPathValid();
+  if (ACTIVE(_VALID) && !valid)
+    return false;
+  if (ACTIVE(_NOVALID) && valid)
+    return false;
+  auto remote = !book.getUrl().empty();
+  if (ACTIVE(_REMOTE) && !remote)
+    return false;
+  if (ACTIVE(_NOREMOTE) && remote)
+    return false;
+  if (ACTIVE(ACCEPTTAGS)) {
+    if (!_acceptTags.empty()) {
+      auto vBookTags = split(book.getTags(), ";");
+      std::set<std::string> sBookTags(vBookTags.begin(), vBookTags.end());
+      for (auto& t: _acceptTags) {
+        if (sBookTags.find(t) == sBookTags.end()) {
+          return false;
+        }
+      }
+    }
+  }
+  if (ACTIVE(REJECTTAGS)) {
+    if (!_rejectTags.empty()) {
+      auto vBookTags = split(book.getTags(), ";");
+      std::set<std::string> sBookTags(vBookTags.begin(), vBookTags.end());
+      for (auto& t: _rejectTags) {
+        if (sBookTags.find(t) != sBookTags.end()) {
+          return false;
+        }
+      }
+    }
+  }
+  if (ACTIVE(MAXSIZE) && book.getSize() > _maxSize)
+    return false;
+
+  if (ACTIVE(LANG) && book.getLanguage() != _lang)
+    return false;
+
+  if (ACTIVE(_PUBLISHER) && book.getPublisher() != _publisher)
+    return false;
+
+  if (ACTIVE(_CREATOR) && book.getCreator() != _creator)
+    return false;
+
+  if ( ACTIVE(QUERY)
+    && !(matchRegex(book.getTitle(), "\\Q" + _query + "\\E")
+        || matchRegex(book.getDescription(), "\\Q" + _query + "\\E")))
+    return false;
+
   return true;
+
 }
+
 }

src/libxml_dumper.cpp

@@ -0,0 +1,139 @@
+/*
+ * Copyright 2017 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#include "libxml_dumper.h"
+#include "book.h"
+
+#include <tools/base64.h>
+#include <tools/stringTools.h>
+#include <tools/otherTools.h>
+
+namespace kiwix
+{
+/* Constructor */
+LibXMLDumper::LibXMLDumper(Library* library)
+  : library(library)
+{
+}
+/* Destructor */
+LibXMLDumper::~LibXMLDumper()
+{
+}
+
+#define ADD_ATTRIBUTE(node, name, value) { (node).append_attribute((name)) = (value).c_str(); }
+#define ADD_ATTR_NOT_EMPTY(node, name, value) { if (!(value).empty()) ADD_ATTRIBUTE(node, name, value); }
+
+void LibXMLDumper::handleBook(Book book, pugi::xml_node root_node) {
+  if (book.readOnly())
+    return;
+
+  auto entry_node = root_node.append_child("book");
+  ADD_ATTRIBUTE(entry_node, "id", book.getId());
+
+  if (!book.getPath().empty()) {
+    ADD_ATTRIBUTE(entry_node, "path", computeRelativePath(baseDir, book.getPath()));
+  }
+
+  if (book.getOrigId().empty()) {
+    ADD_ATTR_NOT_EMPTY(entry_node, "title", book.getTitle());
+    ADD_ATTR_NOT_EMPTY(entry_node, "name", book.getName());
+    ADD_ATTR_NOT_EMPTY(entry_node, "tags", book.getTags());
+    ADD_ATTR_NOT_EMPTY(entry_node, "description", book.getDescription());
+    ADD_ATTR_NOT_EMPTY(entry_node, "language", book.getLanguage());
+    ADD_ATTR_NOT_EMPTY(entry_node, "creator", book.getCreator());
+    ADD_ATTR_NOT_EMPTY(entry_node, "publisher", book.getPublisher());
+    ADD_ATTR_NOT_EMPTY(entry_node, "faviconMimeType", book.getFaviconMimeType());
+    if (!book.getFavicon().empty())
+      ADD_ATTRIBUTE(entry_node, "favicon", base64_encode(book.getFavicon()));
+  } else {
+    ADD_ATTRIBUTE(entry_node, "origId", book.getOrigId());
+  }
+
+  ADD_ATTR_NOT_EMPTY(entry_node, "date", book.getDate());
+  ADD_ATTR_NOT_EMPTY(entry_node, "url", book.getUrl());
+
+  if (book.getArticleCount())
+    ADD_ATTRIBUTE(entry_node, "articleCount", to_string(book.getArticleCount()));
+
+  if (book.getMediaCount())
+    ADD_ATTRIBUTE(entry_node, "mediaCount", to_string(book.getMediaCount()));
+
+  if (book.getSize())
+    ADD_ATTRIBUTE(entry_node, "size", to_string(book.getSize()>>10));
+
+  ADD_ATTR_NOT_EMPTY(entry_node, "downloadId", book.getDownloadId());
+}
+
+#define ADD_TEXT_ENTRY(node, child, value) (node).append_child((child)).append_child(pugi::node_pcdata).set_value((value).c_str())
+
+void LibXMLDumper::handleBookmark(Bookmark bookmark, pugi::xml_node root_node) {
+
+  auto entry_node = root_node.append_child("bookmark");
+  auto book_node = entry_node.append_child("book");
+
+  try {
+    auto book = library->getBookById(bookmark.getBookId());
+    ADD_TEXT_ENTRY(book_node, "id", book.getId());
+    ADD_TEXT_ENTRY(book_node, "title", book.getTitle());
+    ADD_TEXT_ENTRY(book_node, "language", book.getLanguage());
+    ADD_TEXT_ENTRY(book_node, "date", book.getDate());
+  } catch (...) {
+    ADD_TEXT_ENTRY(book_node, "id", bookmark.getBookId());
+    ADD_TEXT_ENTRY(book_node, "title", bookmark.getBookTitle());
+    ADD_TEXT_ENTRY(book_node, "language", bookmark.getLanguage());
+    ADD_TEXT_ENTRY(book_node, "date", bookmark.getDate());
+  }
+  ADD_TEXT_ENTRY(entry_node, "title", bookmark.getTitle());
+  ADD_TEXT_ENTRY(entry_node, "url", bookmark.getUrl());
+}
+
+
+std::string LibXMLDumper::dumpLibXMLContent(const std::vector<std::string>& bookIds)
+{
+  pugi::xml_document doc;
+
+  /* Add the library node */
+  pugi::xml_node libraryNode = doc.append_child("library");
+
+  libraryNode.append_attribute("version") = KIWIX_LIBRARY_VERSION;
+
+  if (library) {
+    for (auto& bookId: bookIds) {
+      handleBook(library->getBookById(bookId), libraryNode);
+    }
+  }
+  return nodeToString(libraryNode);
+}
+
+std::string LibXMLDumper::dumpLibXMLBookmark()
+{
+  pugi::xml_document doc;
+
+  /* Add the library node */
+  pugi::xml_node bookmarksNode = doc.append_child("bookmarks");
+
+  if (library) {
+    for (auto& bookmark: library->getBookmarks()) {
+      handleBookmark(bookmark, bookmarksNode);
+    }
+  }
+  return nodeToString(bookmarksNode);
+}
+
+}

src/manager.cpp

@@ -18,80 +18,65 @@
  */
 
 #include "manager.h"
-#include "downloader.h"
+
+#include <pugixml.hpp>
 
 namespace kiwix
 {
 /* Constructor */
-Manager::Manager() : writableLibraryPath("")
+Manager::Manager(LibraryManipulator* manipulator):
+  writableLibraryPath(""),
+  manipulator(manipulator),
+  mustDeleteManipulator(false)
 {
 }
+
+Manager::Manager(Library* library) :
+  writableLibraryPath(""),
+  manipulator(new DefaultLibraryManipulator(library)),
+  mustDeleteManipulator(true)
+{
+}
+
 /* Destructor */
 Manager::~Manager()
 {
+  if (mustDeleteManipulator) {
+    delete manipulator;
+  }
 }
 bool Manager::parseXmlDom(const pugi::xml_document& doc,
                           const bool readOnly,
-                          const string libraryPath)
+                          const std::string& libraryPath)
 {
   pugi::xml_node libraryNode = doc.child("library");
 
-  if (strlen(libraryNode.attribute("current").value()))
-    this->setCurrentBookId(libraryNode.attribute("current").value());
-
-  string libraryVersion = libraryNode.attribute("version").value();
+  std::string libraryVersion = libraryNode.attribute("version").value();
 
   for (pugi::xml_node bookNode = libraryNode.child("book"); bookNode;
        bookNode = bookNode.next_sibling("book")) {
-    bool ok = true;
     kiwix::Book book;
 
-    book.readOnly = readOnly;
-    book.id = bookNode.attribute("id").value();
-    book.path = bookNode.attribute("path").value();
-    book.last = (std::string(bookNode.attribute("last").value()) != "undefined"
-                     ? bookNode.attribute("last").value()
-                     : "");
-    book.indexPath = bookNode.attribute("indexPath").value();
-    book.indexType = XAPIAN;
-    book.title = bookNode.attribute("title").value();
-    book.name = bookNode.attribute("name").value();
-    book.tags = bookNode.attribute("tags").value();
-    book.description = bookNode.attribute("description").value();
-    book.language = bookNode.attribute("language").value();
-    book.date = bookNode.attribute("date").value();
-    book.creator = bookNode.attribute("creator").value();
-    book.publisher = bookNode.attribute("publisher").value();
-    book.url = bookNode.attribute("url").value();
-    book.origId = bookNode.attribute("origId").value();
-    book.articleCount = bookNode.attribute("articleCount").value();
-    book.mediaCount = bookNode.attribute("mediaCount").value();
-    book.size = bookNode.attribute("size").value();
-    book.favicon = bookNode.attribute("favicon").value();
-    book.faviconMimeType = bookNode.attribute("faviconMimeType").value();
-
-    /* Check absolute and relative paths */
-    this->checkAndCleanBookPaths(book, libraryPath);
+    book.setReadOnly(readOnly);
+    book.updateFromXml(bookNode,
+                       removeLastPathElement(libraryPath, true, false));
 
     /* Update the book properties with the new importer */
     if (libraryVersion.empty()
         || atoi(libraryVersion.c_str()) <= atoi(KIWIX_LIBRARY_VERSION)) {
-      if (!book.path.empty()) {
-        ok = this->readBookFromPath(book.pathAbsolute);
+      if (!book.getPath().empty()) {
+        this->readBookFromPath(book.getPath(), &book);
       }
     }
-
-    if (ok) {
-      library.addBook(book);
-    }
+    manipulator->addBookToLibrary(book);
   }
 
   return true;
 }
 
-bool Manager::readXml(const string& xml,
+bool Manager::readXml(const std::string& xml,
                       const bool readOnly,
-                      const string libraryPath)
+                      const std::string& libraryPath)
 {
   pugi::xml_document doc;
   pugi::xml_parse_result result
@@ -110,40 +95,24 @@ bool Manager::parseOpdsDom(const pugi::xml_document& doc, const std::string& url
 {
   pugi::xml_node libraryNode = doc.child("feed");
 
+  try {
+    m_totalBooks = strtoull(libraryNode.child("totalResults").child_value(), 0, 0);
+    m_startIndex = strtoull(libraryNode.child("startIndex").child_value(), 0, 0);
+    m_itemsPerPage = strtoull(libraryNode.child("itemsPerPage").child_value(), 0, 0);
+    m_hasSearchResult = true;
+  } catch(...) {
+    m_hasSearchResult = false;
+  }
+
   for (pugi::xml_node entryNode = libraryNode.child("entry"); entryNode;
        entryNode = entryNode.next_sibling("entry")) {
     kiwix::Book book;
 
-    book.readOnly = false;
-    book.id = entryNode.child("id").child_value();
-    book.title = entryNode.child("title").child_value();
-    book.description = entryNode.child("summary").child_value();
-    book.language = entryNode.child("language").child_value();
-    book.date = entryNode.child("updated").child_value();
-    book.creator = entryNode.child("author").child("name").child_value();
-    for(pugi::xml_node linkNode = entryNode.child("link"); linkNode;
-        linkNode = linkNode.next_sibling("link")) {
-       std::string rel = linkNode.attribute("rel").value();
-
-       if (rel == "http://opds-spec.org/image/thumbnail") {
-         auto faviconUrl = urlHost + linkNode.attribute("href").value();
-         auto downloader = Downloader();
-         auto fileHandle = downloader.download(faviconUrl);
-         if (fileHandle.success) {
-           auto content = getFileContent(fileHandle.path);
-           book.favicon = base64_encode((const unsigned char*)content.data(), content.size());
-           book.faviconMimeType = linkNode.attribute("type").value();
-         } else {
-           std::cerr << "Cannot get favicon content from " << faviconUrl << std::endl;
-         }
-
-       } else if (rel == "http://opds-spec.org/acquisition/open-access") {
-         book.url = linkNode.attribute("href").value();
-       }
-    }
+    book.setReadOnly(false);
+    book.updateFromOpds(entryNode, urlHost);
 
     /* Update the book properties with the new importer */
-    library.addBook(book);
+    manipulator->addBookToLibrary(book);
   }
 
   return true;
@@ -151,7 +120,7 @@ bool Manager::parseOpdsDom(const pugi::xml_document& doc, const std::string& url
 
 
 
-bool Manager::readOpds(const string& content, const std::string& urlHost)
+bool Manager::readOpds(const std::string& content, const std::string& urlHost)
 {
   pugi::xml_document doc;
   pugi::xml_parse_result result
@@ -165,13 +134,13 @@ bool Manager::readOpds(const string& content, const std::string& urlHost)
   return false;
 }
 
-bool Manager::readFile(const string path, const bool readOnly)
+bool Manager::readFile(const std::string& path, const bool readOnly)
 {
   return this->readFile(path, path, readOnly);
 }
 
-bool Manager::readFile(const string nativePath,
-                       const string UTF8Path,
+bool Manager::readFile(const std::string& nativePath,
+                       const std::string& UTF8Path,
                        const bool readOnly)
 {
   bool retVal = true;
@@ -194,149 +163,31 @@ bool Manager::readFile(const string nativePath,
   return retVal;
 }
 
-bool Manager::writeFile(const string path)
-{
-  pugi::xml_document doc;
-
-  /* Add the library node */
-  pugi::xml_node libraryNode = doc.append_child("library");
-
-  if (!getCurrentBookId().empty()) {
-    libraryNode.append_attribute("current") = getCurrentBookId().c_str();
-  }
-
-  if (!library.version.empty())
-    libraryNode.append_attribute("version") = library.version.c_str();
-
-  /* Add each book */
-  std::vector<kiwix::Book>::iterator itr;
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    if (!itr->readOnly) {
-      this->checkAndCleanBookPaths(*itr, path);
-
-      pugi::xml_node bookNode = libraryNode.append_child("book");
-      bookNode.append_attribute("id") = itr->id.c_str();
-
-      if (!itr->path.empty()) {
-        bookNode.append_attribute("path") = itr->path.c_str();
-      }
-
-      if (!itr->last.empty() && itr->last != "undefined") {
-        bookNode.append_attribute("last") = itr->last.c_str();
-      }
-
-      if (!itr->indexPath.empty())
-        bookNode.append_attribute("indexPath") = itr->indexPath.c_str();
-
-      if (!itr->indexPath.empty() || !itr->indexPathAbsolute.empty()) {
-        if (itr->indexType == XAPIAN) {
-          bookNode.append_attribute("indexType") = "xapian";
-        }
-      }
-
-      if (itr->origId.empty()) {
-        if (!itr->title.empty())
-          bookNode.append_attribute("title") = itr->title.c_str();
-
-        if (!itr->name.empty())
-          bookNode.append_attribute("name") = itr->name.c_str();
-
-        if (!itr->tags.empty())
-          bookNode.append_attribute("tags") = itr->tags.c_str();
-
-        if (!itr->description.empty())
-          bookNode.append_attribute("description") = itr->description.c_str();
-
-        if (!itr->language.empty())
-          bookNode.append_attribute("language") = itr->language.c_str();
-
-        if (!itr->creator.empty())
-          bookNode.append_attribute("creator") = itr->creator.c_str();
-
-        if (!itr->publisher.empty())
-          bookNode.append_attribute("publisher") = itr->publisher.c_str();
-
-        if (!itr->favicon.empty())
-          bookNode.append_attribute("favicon") = itr->favicon.c_str();
-
-        if (!itr->faviconMimeType.empty())
-          bookNode.append_attribute("faviconMimeType")
-              = itr->faviconMimeType.c_str();
-      }
-
-      if (!itr->date.empty()) {
-        bookNode.append_attribute("date") = itr->date.c_str();
-      }
-
-      if (!itr->url.empty()) {
-        bookNode.append_attribute("url") = itr->url.c_str();
-      }
-
-      if (!itr->origId.empty())
-        bookNode.append_attribute("origId") = itr->origId.c_str();
-
-      if (!itr->articleCount.empty())
-        bookNode.append_attribute("articleCount") = itr->articleCount.c_str();
-
-      if (!itr->mediaCount.empty())
-        bookNode.append_attribute("mediaCount") = itr->mediaCount.c_str();
-
-      if (!itr->size.empty()) {
-        bookNode.append_attribute("size") = itr->size.c_str();
-      }
-    }
-  }
-
-  /* saving file */
-  doc.save_file(path.c_str());
-
-  return true;
-}
-
-
-bool Manager::setCurrentBookId(const string id)
-{
-  if (library.current.empty() || library.current.top() != id) {
-    if (id.empty() && !library.current.empty()) {
-      library.current.pop();
-    } else {
-      library.current.push(id);
-    }
-  }
-  return true;
-}
-
-string Manager::getCurrentBookId() const
-{
-  return library.current.empty() ? "" : library.current.top();
-}
 
 /* Add a book to the library. Return empty string if failed, book id otherwise
  */
-string Manager::addBookFromPathAndGetId(const string pathToOpen,
-                                        const string pathToSave,
-                                        const string url,
-                                        const bool checkMetaData)
+std::string Manager::addBookFromPathAndGetId(const std::string& pathToOpen,
+                                             const std::string& pathToSave,
+                                             const std::string& url,
+                                             const bool checkMetaData)
 {
   kiwix::Book book;
 
   if (this->readBookFromPath(pathToOpen, &book)) {
     if (pathToSave != pathToOpen) {
-      book.path = pathToSave;
-      book.pathAbsolute
-          = isRelativePath(pathToSave)
+      book.setPath(isRelativePath(pathToSave)
                 ? computeAbsolutePath(
                       removeLastPathElement(writableLibraryPath, true, false),
                       pathToSave)
-                : pathToSave;
+                : pathToSave);
     }
 
     if (!checkMetaData
-        || (checkMetaData && !book.title.empty() && !book.language.empty()
-            && !book.date.empty())) {
-      book.url = url;
-      library.addBook(book);
-      return book.id;
+        || (checkMetaData && !book.getTitle().empty() && !book.getLanguage().empty()
+            && !book.getDate().empty())) {
+      book.setUrl(url);
+      manipulator->addBookToLibrary(book);
+      return book.getId();
     }
   }
 
@@ -345,9 +196,9 @@ string Manager::addBookFromPathAndGetId(const string pathToOpen,
 
 /* Wrapper over Manager::addBookFromPath which return a bool instead of a string
  */
-bool Manager::addBookFromPath(const string pathToOpen,
-                              const string pathToSave,
-                              const string url,
+bool Manager::addBookFromPath(const std::string& pathToOpen,
+                              const std::string& pathToSave,
+                              const std::string& url,
                               const bool checkMetaData)
 {
   return !(
@@ -355,380 +206,45 @@ bool Manager::addBookFromPath(const string pathToOpen,
           .empty());
 }
 
-bool Manager::readBookFromPath(const string path, kiwix::Book* book)
+bool Manager::readBookFromPath(const std::string& path, kiwix::Book* book)
 {
+  std::string tmp_path = path;
+  if (isRelativePath(path)) {
+    tmp_path = computeAbsolutePath(getCurrentDirectory(), path);
+  }
   try {
-    kiwix::Reader* reader = new kiwix::Reader(path);
-
-    if (book != NULL) {
-      book->path = path;
-      book->pathAbsolute = path;
-      book->id = reader->getId();
-      book->description = reader->getDescription();
-      book->language = reader->getLanguage();
-      book->date = reader->getDate();
-      book->creator = reader->getCreator();
-      book->publisher = reader->getPublisher();
-      book->title = reader->getTitle();
-      book->name = reader->getName();
-      book->tags = reader->getTags();
-      book->origId = reader->getOrigId();
-      std::ostringstream articleCountStream;
-      articleCountStream << reader->getArticleCount();
-      book->articleCount = articleCountStream.str();
-
-      std::ostringstream mediaCountStream;
-      mediaCountStream << reader->getMediaCount();
-      book->mediaCount = mediaCountStream.str();
-
-      ostringstream convert;
-      convert << reader->getFileSize();
-      book->size = convert.str();
-
-      string favicon;
-      string faviconMimeType;
-      if (reader->getFavicon(favicon, faviconMimeType)) {
-        book->favicon = base64_encode(
-            reinterpret_cast<const unsigned char*>(favicon.c_str()),
-            favicon.length());
-        book->faviconMimeType = faviconMimeType;
-      }
-    }
-
-    delete reader;
+    kiwix::Reader reader(tmp_path);
+    book->update(reader);
+    book->setPathValid(true);
   } catch (const std::exception& e) {
-    std::cerr << e.what() << std::endl;
+    book->setPathValid(false);
     return false;
   }
 
   return true;
 }
 
-bool Manager::removeBookByIndex(const unsigned int bookIndex)
+bool Manager::readBookmarkFile(const std::string& path)
 {
-  return this->library.removeBookByIndex(bookIndex);
-}
+  pugi::xml_document doc;
+  pugi::xml_parse_result result = doc.load_file(path.c_str());
 
-bool Manager::removeBookById(const string id)
-{
-  unsigned int bookIndex = 0;
-  std::vector<kiwix::Book>::iterator itr;
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    if (itr->id == id) {
-      return this->library.removeBookByIndex(bookIndex);
-    }
-    bookIndex++;
-  }
-  return false;
-}
-
-vector<string> Manager::getBooksLanguages()
-{
-  std::vector<string> booksLanguages;
-  std::vector<kiwix::Book>::iterator itr;
-  std::map<string, bool> booksLanguagesMap;
-
-  std::sort(
-      library.books.begin(), library.books.end(), kiwix::Book::sortByLanguage);
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    if (booksLanguagesMap.find(itr->language) == booksLanguagesMap.end()) {
-      if (itr->origId.empty()) {
-        booksLanguagesMap[itr->language] = true;
-        booksLanguages.push_back(itr->language);
-      }
-    }
-  }
-
-  return booksLanguages;
-}
-
-vector<string> Manager::getBooksCreators()
-{
-  std::vector<string> booksCreators;
-  std::vector<kiwix::Book>::iterator itr;
-  std::map<string, bool> booksCreatorsMap;
-
-  std::sort(
-      library.books.begin(), library.books.end(), kiwix::Book::sortByCreator);
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    if (booksCreatorsMap.find(itr->creator) == booksCreatorsMap.end()) {
-      if (itr->origId.empty()) {
-        booksCreatorsMap[itr->creator] = true;
-        booksCreators.push_back(itr->creator);
-      }
-    }
-  }
-
-  return booksCreators;
-}
-
-vector<string> Manager::getBooksIds()
-{
-  std::vector<string> booksIds;
-  std::vector<kiwix::Book>::iterator itr;
-
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    booksIds.push_back(itr->id);
-  }
-
-  return booksIds;
-}
-
-vector<string> Manager::getBooksPublishers()
-{
-  std::vector<string> booksPublishers;
-  std::vector<kiwix::Book>::iterator itr;
-  std::map<string, bool> booksPublishersMap;
-
-  std::sort(
-      library.books.begin(), library.books.end(), kiwix::Book::sortByPublisher);
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    if (booksPublishersMap.find(itr->publisher) == booksPublishersMap.end()) {
-      if (itr->origId.empty()) {
-        booksPublishersMap[itr->publisher] = true;
-        booksPublishers.push_back(itr->publisher);
-      }
-    }
-  }
-
-  return booksPublishers;
-}
-
-kiwix::Library Manager::cloneLibrary()
-{
-  return this->library;
-}
-bool Manager::getCurrentBook(Book& book)
-{
-  string currentBookId = getCurrentBookId();
-  if (currentBookId.empty()) {
+  if (!result) {
     return false;
-  } else {
-    getBookById(currentBookId, book);
-    return true;
-  }
-}
-
-bool Manager::getBookById(const string id, Book& book)
-{
-  std::vector<kiwix::Book>::iterator itr;
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    if (itr->id == id) {
-      book = *itr;
-      return true;
-    }
-  }
-  return false;
-}
-
-bool Manager::updateBookLastOpenDateById(const string id)
-{
-  std::vector<kiwix::Book>::iterator itr;
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    if (itr->id == id) {
-      char unixdate[12];
-      sprintf(unixdate, "%d", (int)time(NULL));
-      itr->last = unixdate;
-      return true;
-    }
   }
 
-  return false;
-}
+  pugi::xml_node libraryNode = doc.child("bookmarks");
 
-bool Manager::setBookIndex(const string id,
-                           const string path,
-                           const supportedIndexType type)
-{
-  std::vector<kiwix::Book>::iterator itr;
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    if (itr->id == id) {
-      itr->indexPath = path;
-      itr->indexPathAbsolute
-          = isRelativePath(path)
-                ? computeAbsolutePath(
-                      removeLastPathElement(writableLibraryPath, true, false),
-                      path)
-                : path;
-      itr->indexType = type;
-      return true;
-    }
-  }
+  for (pugi::xml_node node = libraryNode.child("bookmark"); node;
+       node = node.next_sibling("bookmark")) {
+    kiwix::Bookmark bookmark;
 
-  return false;
-}
+    bookmark.updateFromXml(node);
 
-bool Manager::setBookPath(const string id, const string path)
-{
-  std::vector<kiwix::Book>::iterator itr;
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    if (itr->id == id) {
-      itr->path = path;
-      itr->pathAbsolute
-          = isRelativePath(path)
-                ? computeAbsolutePath(
-                      removeLastPathElement(writableLibraryPath, true, false),
-                      path)
-                : path;
-      return true;
-    }
-  }
-
-  return false;
-}
-
-void Manager::removeBookPaths()
-{
-  std::vector<kiwix::Book>::iterator itr;
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    itr->path = "";
-    itr->pathAbsolute = "";
-  }
-}
-
-unsigned int Manager::getBookCount(const bool localBooks,
-                                   const bool remoteBooks)
-{
-  unsigned int result = 0;
-  std::vector<kiwix::Book>::iterator itr;
-  for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-    if ((!itr->path.empty() && localBooks)
-        || (itr->path.empty() && remoteBooks)) {
-      result++;
-    }
-  }
-  return result;
-}
-
-bool Manager::listBooks(const supportedListMode mode,
-                        const supportedListSortBy sortBy,
-                        const unsigned int maxSize,
-                        const string language,
-                        const string creator,
-                        const string publisher,
-                        const string search)
-{
-  this->bookIdList.clear();
-  std::vector<kiwix::Book>::iterator itr;
-
-  /* Sort */
-  if (sortBy == TITLE) {
-    std::sort(
-        library.books.begin(), library.books.end(), kiwix::Book::sortByTitle);
-  } else if (sortBy == SIZE) {
-    std::sort(
-        library.books.begin(), library.books.end(), kiwix::Book::sortBySize);
-  } else if (sortBy == DATE) {
-    std::sort(
-        library.books.begin(), library.books.end(), kiwix::Book::sortByDate);
-  } else if (sortBy == CREATOR) {
-    std::sort(
-        library.books.begin(), library.books.end(), kiwix::Book::sortByCreator);
-  } else if (sortBy == PUBLISHER) {
-    std::sort(library.books.begin(),
-              library.books.end(),
-              kiwix::Book::sortByPublisher);
-  }
-
-  /* Special sort for LASTOPEN */
-  if (mode == LASTOPEN) {
-    std::sort(library.books.begin(),
-              library.books.end(),
-              kiwix::Book::sortByLastOpen);
-    for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-      if (!itr->last.empty()) {
-        this->bookIdList.push_back(itr->id);
-      }
-    }
-  } else {
-    /* Generate the list of book id */
-    for (itr = library.books.begin(); itr != library.books.end(); ++itr) {
-      bool ok = true;
-
-      if (mode == LOCAL && itr->path.empty()) {
-        ok = false;
-      }
-
-      if (ok == true && mode == REMOTE
-          && (!itr->path.empty() || itr->url.empty())) {
-        ok = false;
-      }
-
-      if (ok == true && maxSize != 0
-          && (unsigned int)atoi(itr->size.c_str()) > maxSize * 1024 * 1024) {
-        ok = false;
-      }
-
-      if (ok == true && !language.empty()
-          && !matchRegex(itr->language, language)) {
-        ok = false;
-      }
-
-      if (ok == true && !creator.empty() && itr->creator != creator) {
-        ok = false;
-      }
-
-      if (ok == true && !publisher.empty() && itr->publisher != publisher) {
-        ok = false;
-      }
-
-      if ((ok == true && !search.empty())
-          && !(matchRegex(itr->title, "\\Q" + search + "\\E")
-               || matchRegex(itr->description, "\\Q" + search + "\\E")
-               || matchRegex(itr->language, "\\Q" + search + "\\E"))) {
-        ok = false;
-      }
-
-      if (ok == true) {
-        this->bookIdList.push_back(itr->id);
-      }
-    }
+    manipulator->addBookmarkToLibrary(bookmark);
   }
 
   return true;
 }
 
-
-Library Manager::filter(const std::string& search) {
-  Library library;
-
-  if (search.empty()) {
-    return library;
-  }
-
-  for(auto book:this->library.books) {
-     if (matchRegex(book.title, "\\Q" + search + "\\E")
-         || matchRegex(book.description, "\\Q" + search + "\\E")) {
-       library.addBook(book);
-     }
-  }
-
-  return library;
-}
-
-void Manager::checkAndCleanBookPaths(Book& book, const string& libraryPath)
-{
-  if (!book.path.empty()) {
-    if (isRelativePath(book.path)) {
-      book.pathAbsolute = computeAbsolutePath(
-          removeLastPathElement(libraryPath, true, false), book.path);
-    } else {
-      book.pathAbsolute = book.path;
-      book.path = computeRelativePath(
-          removeLastPathElement(libraryPath, true, false), book.pathAbsolute);
-    }
-  }
-
-  if (!book.indexPath.empty()) {
-    if (isRelativePath(book.indexPath)) {
-      book.indexPathAbsolute = computeAbsolutePath(
-          removeLastPathElement(libraryPath, true, false), book.indexPath);
-    } else {
-      book.indexPathAbsolute = book.indexPath;
-      book.indexPath
-          = computeRelativePath(removeLastPathElement(libraryPath, true, false),
-                                book.indexPathAbsolute);
-    }
-  }
-}
 }

src/meson.build

@@ -1,24 +1,30 @@
 kiwix_sources = [
+  'book.cpp',
+  'bookmark.cpp',
   'library.cpp',
   'manager.cpp',
+  'libxml_dumper.cpp',
   'opds_dumper.cpp',
   'downloader.cpp',
   'reader.cpp',
   'entry.cpp',
   'searcher.cpp',
-  'common/base64.cpp',
-  'common/pathTools.cpp',
-  'common/regexTools.cpp',
-  'common/stringTools.cpp',
-  'common/networkTools.cpp',
-  'common/otherTools.cpp',
-  'xapian/htmlparse.cc',
-  'xapian/myhtmlparse.cc'
+  'subprocess.cpp',
+  'aria2.cpp',
+  'tools/base64.cpp',
+  'tools/pathTools.cpp',
+  'tools/regexTools.cpp',
+  'tools/stringTools.cpp',
+  'tools/networkTools.cpp',
+  'tools/otherTools.cpp',
+  'kiwixserve.cpp',
 ]
 kiwix_sources += lib_resources
 
-if xapian_dep.found()
-  kiwix_sources += ['xapianSearcher.cpp']
+if host_machine.system() == 'windows'
+  kiwix_sources += 'subprocess_windows.cpp'
+else
+  kiwix_sources += 'subprocess_unix.cpp'
 endif
 
 if get_option('android')
@@ -28,11 +34,6 @@ else
   install_dir = get_option('libdir')
 endif
 
-
-if has_ctpp2_dep
-  kiwix_sources += ['ctpp2/CTPP2VMStringLoader.cpp']
-endif
-
 config_h = configure_file(output : 'kiwix_config.h',
                configuration : conf,
                input : 'config.h.in')
@@ -44,5 +45,4 @@ kiwixlib = library('kiwix',
                    dependencies : all_deps,
                    version: meson.project_version(),
                    install: true,
-                   install_dir: install_dir,
-                   install_rpath: '$ORIGIN')
+                   install_dir: install_dir)

src/opds_dumper.cpp

@@ -18,11 +18,14 @@
  */
 
 #include "opds_dumper.h"
+#include "book.h"
+
+#include <tools/otherTools.h>
 
 namespace kiwix
 {
 /* Constructor */
-OPDSDumper::OPDSDumper(Library library)
+OPDSDumper::OPDSDumper(Library* library)
   : library(library)
 {
 }
@@ -31,24 +34,6 @@ OPDSDumper::~OPDSDumper()
 {
 }
 
-struct xml_string_writer: pugi::xml_writer
-{
-    std::string result;
-
-    virtual void write(const void* data, size_t size)
-    {
-        result.append(static_cast<const char*>(data), size);
-    }
-};
-
-std::string node_to_string(pugi::xml_node node)
-{
-    xml_string_writer writer;
-    node.print(writer, "  ");
-
-    return writer.result;
-}
-
 std::string gen_date_str()
 {
   auto now = time(0);
@@ -65,40 +50,57 @@ std::string gen_date_str()
   return is.str();
 }
 
+static std::string gen_date_from_yyyy_mm_dd(const std::string& date)
+{
+  std::stringstream is;
+  is << date << "T00:00::00:Z";
+  return is.str();
+}
+
+void OPDSDumper::setOpenSearchInfo(int totalResults, int startIndex, int count)
+{
+  m_totalResults = totalResults;
+  m_startIndex = startIndex,
+  m_count = count;
+  m_isSearchResult = true;
+}
+
 #define ADD_TEXT_ENTRY(node, child, value) (node).append_child((child)).append_child(pugi::node_pcdata).set_value((value).c_str())
 
 pugi::xml_node OPDSDumper::handleBook(Book book, pugi::xml_node root_node) {
   auto entry_node = root_node.append_child("entry");
-  ADD_TEXT_ENTRY(entry_node, "title", book.title);
-  ADD_TEXT_ENTRY(entry_node, "id", "urn:uuid:"+book.id);
+  ADD_TEXT_ENTRY(entry_node, "title", book.getTitle());
+  ADD_TEXT_ENTRY(entry_node, "id", "urn:uuid:"+book.getId());
   ADD_TEXT_ENTRY(entry_node, "icon", rootLocation + "/meta?name=favicon&content=" + book.getHumanReadableIdFromPath());
-  ADD_TEXT_ENTRY(entry_node, "updated", date);
-  ADD_TEXT_ENTRY(entry_node, "summary", book.description);
+  ADD_TEXT_ENTRY(entry_node, "updated", gen_date_from_yyyy_mm_dd(book.getDate()));
+  ADD_TEXT_ENTRY(entry_node, "summary", book.getDescription());
+  ADD_TEXT_ENTRY(entry_node, "tags", book.getTags());
 
   auto content_node = entry_node.append_child("link");
   content_node.append_attribute("type") = "text/html";
   content_node.append_attribute("href") = (rootLocation + "/" + book.getHumanReadableIdFromPath()).c_str();
 
   auto author_node = entry_node.append_child("author");
-  ADD_TEXT_ENTRY(author_node, "name", book.creator);
+  ADD_TEXT_ENTRY(author_node, "name", book.getCreator());
 
-  if (! book.url.empty()) {
+  if (! book.getUrl().empty()) {
     auto acquisition_link = entry_node.append_child("link");
     acquisition_link.append_attribute("rel") = "http://opds-spec.org/acquisition/open-access";
     acquisition_link.append_attribute("type") = "application/x-zim";
-    acquisition_link.append_attribute("href") = book.url.c_str();
+    acquisition_link.append_attribute("href") = book.getUrl().c_str();
+    acquisition_link.append_attribute("length") = to_string(book.getSize()).c_str();
   }
 
-  if (! book.faviconMimeType.empty() ) {
+  if (! book.getFaviconMimeType().empty() ) {
     auto image_link = entry_node.append_child("link");
     image_link.append_attribute("rel") = "http://opds-spec.org/image/thumbnail";
-    image_link.append_attribute("type") = book.faviconMimeType.c_str();
+    image_link.append_attribute("type") = book.getFaviconMimeType().c_str();
     image_link.append_attribute("href") = (rootLocation + "/meta?name=favicon&content=" + book.getHumanReadableIdFromPath()).c_str();
   }
   return entry_node;
 }
 
-string OPDSDumper::dumpOPDSFeed()
+string OPDSDumper::dumpOPDSFeed(const std::vector<std::string>& bookIds)
 {
   date = gen_date_str();
   pugi::xml_document doc;
@@ -112,6 +114,12 @@ string OPDSDumper::dumpOPDSFeed()
   ADD_TEXT_ENTRY(root_node, "title", title);
   ADD_TEXT_ENTRY(root_node, "updated", date);
 
+  if (m_isSearchResult) {
+    ADD_TEXT_ENTRY(root_node, "totalResults", to_string(m_totalResults));
+    ADD_TEXT_ENTRY(root_node, "startIndex", to_string(m_startIndex));
+    ADD_TEXT_ENTRY(root_node, "itemsPerPage", to_string(m_count));
+  }
+
   auto self_link_node = root_node.append_child("link");
   self_link_node.append_attribute("rel") = "self";
   self_link_node.append_attribute("href") = "";
@@ -125,11 +133,13 @@ string OPDSDumper::dumpOPDSFeed()
     search_link.append_attribute("href") = searchDescriptionUrl.c_str();
   }
 
-  for (auto book: library.books) {
-    handleBook(book, root_node);
+  if (library) {
+    for (auto& bookId: bookIds) {
+      handleBook(library->getBookById(bookId), root_node);
+    }
   }
 
-  return node_to_string(root_node);
+  return nodeToString(root_node);
 }
 
 }

src/reader.cpp

@@ -241,8 +241,6 @@ Entry Reader::getMainPage() const
     throw NoEntry();
   }
 
-  string url = "";
-
   zim::Article article;
   if (this->zimFileHandler->getFileheader().hasMainPage())
   {
@@ -265,6 +263,7 @@ bool Reader::getFavicon(string& content, string& mimeType) const
   for (auto &path: paths) {
     try {
       auto entry = getEntryFromPath(path);
+      entry = entry.getFinalEntry();
       content = entry.getContent();
       mimeType = entry.getMimetype();
       return true;
@@ -411,30 +410,26 @@ bool _parseUrl(const string& url, char* ns, string& title)
   unsigned int urlLength = url.size();
   unsigned int offset = 0;
 
-  /* Ignore the '/' */
-  while ((offset < urlLength) && (url[offset] == '/')) {
+  /* Ignore the first '/' */
+  if (url[offset] == '/')
     offset++;
-  }
+
+  if (url[offset] == '/' || offset >= urlLength)
+    return false;
 
   /* Get namespace */
-  while ((offset < urlLength) && (url[offset] != '/')) {
-    *ns = url[offset];
-    offset++;
-  }
+  *ns = url[offset++];
 
-  /* Ignore the '/' */
-  while ((offset < urlLength) && (url[offset] == '/')) {
-    offset++;
-  }
+  if (url[offset] != '/' || offset >= urlLength)
+    return false;
+
+  offset++;
+
+  if ( offset >= urlLength)
+    return false;
 
   /* Get content title */
-  unsigned int titleOffset = offset;
-  while (offset < urlLength) {
-    offset++;
-  }
-
-  /* unescape title */
-  title = url.substr(titleOffset, offset - titleOffset);
+  title = url.substr(offset, urlLength - offset);
 
   return true;
 }
@@ -468,7 +463,7 @@ Entry Reader::getEntryFromPath(const std::string& path) const
 
 Entry Reader::getEntryFromEncodedPath(const std::string& path) const
 {
-  return getEntryFromPath(urlDecode(path));
+  return getEntryFromPath(urlDecode(path, true));
 }
 
 Entry Reader::getEntryFromTitle(const std::string& title) const
@@ -765,18 +760,21 @@ bool Reader::searchSuggestionsSmart(const string& prefix,
                                     unsigned int suggestionsCount)
 {
   std::vector<std::string> variants = this->getTitleVariants(prefix);
-  bool retVal;
+  bool retVal = false;
 
   this->suggestions.clear();
   this->suggestionsOffset = this->suggestions.begin();
   /* Try to search in the title using fulltext search database */
-  const zim::Search* suggestionSearch
+  const auto suggestionSearch
       = this->getZimFileHandler()->suggestions(prefix, 0, suggestionsCount);
 
   if (suggestionSearch->get_matches_estimated()) {
     for (auto current = suggestionSearch->begin();
          current != suggestionSearch->end();
          current++) {
+      if (!current->good()) {
+          continue;
+      }
       std::vector<std::string> suggestion;
       suggestion.push_back(current->getTitle());
       suggestion.push_back("/A/" + current->getUrl());
@@ -854,7 +852,7 @@ bool Reader::isCorrupted() const
 unsigned int Reader::getFileSize() const
 {
   zim::File* file = this->getZimFileHandler();
-  zim::offset_type size = 0;
+  zim::size_type size = 0;
 
   if (file != NULL) {
     size = file->getFilesize();

src/searcher.cpp

@@ -21,20 +21,12 @@
 #include <cmath>
 
 #include "searcher.h"
-#include "kiwixlib-resources.h"
 #include "reader.h"
-#include "xapianSearcher.h"
 
 #include <zim/search.h>
 
-#ifdef ENABLE_CTPP2
-#include <ctpp2/CDT.hpp>
-#include <ctpp2/CTPP2FileLogger.hpp>
-#include <ctpp2/CTPP2SimpleVM.hpp>
-#include "ctpp2/CTPP2VMStringLoader.hpp"
-
-using namespace CTPP;
-#endif
+#include <mustache.hpp>
+#include "kiwixlib-resources.h"
 
 #define MAX_SEARCH_LEN 140
 
@@ -61,42 +53,18 @@ class _Result : public Result
 
 struct SearcherInternal {
   const zim::Search* _search;
-  XapianSearcher* _xapianSearcher;
   zim::Search::iterator current_iterator;
 
-  SearcherInternal() : _search(NULL), _xapianSearcher(NULL) {}
+  SearcherInternal() : _search(NULL) {}
   ~SearcherInternal()
   {
     if (_search != NULL) {
       delete _search;
     }
-    if (_xapianSearcher != NULL) {
-      delete _xapianSearcher;
-    }
   }
 };
 
 /* Constructor */
-Searcher::Searcher(const string& xapianDirectoryPath,
-                   Reader* reader,
-                   const string& humanReadableName)
-    : internal(new SearcherInternal()),
-      searchPattern(""),
-      protocolPrefix("zim://"),
-      searchProtocolPrefix("search://?"),
-      resultCountPerPage(0),
-      estimatedResultCount(0),
-      resultStart(0),
-      resultEnd(0),
-      contentHumanReadableId(humanReadableName)
-{
-  loadICUExternalTables();
-  if (!reader || !reader->hasFulltextIndex()) {
-    internal->_xapianSearcher = new XapianSearcher(xapianDirectoryPath, reader);
-  }
-  this->humanReaderNames.push_back(humanReadableName);
-}
-
 Searcher::Searcher(const std::string& humanReadableName)
     : internal(new SearcherInternal()),
       searchPattern(""),
@@ -160,26 +128,20 @@ void Searcher::search(std::string& search,
     this->resultStart = resultStart;
     this->resultEnd = resultEnd;
     string unaccentedSearch = removeAccents(search);
-    if (internal->_xapianSearcher) {
-      internal->_xapianSearcher->searchInIndex(
-          unaccentedSearch, resultStart, resultEnd, verbose);
-      this->estimatedResultCount
-          = internal->_xapianSearcher->results.get_matches_estimated();
-    } else {
-      std::vector<const zim::File*> zims;
-      for (auto current = this->readers.begin(); current != this->readers.end();
-           current++) {
-        if ( (*current)->hasFulltextIndex() ) {
-            zims.push_back((*current)->getZimFileHandler());
-        }
+    std::vector<const zim::File*> zims;
+    for (auto current = this->readers.begin(); current != this->readers.end();
+         current++) {
+      if ( (*current)->hasFulltextIndex() ) {
+          zims.push_back((*current)->getZimFileHandler());
       }
-      zim::Search* search = new zim::Search(zims);
-      search->set_query(unaccentedSearch);
-      search->set_range(resultStart, resultEnd);
-      internal->_search = search;
-      internal->current_iterator = internal->_search->begin();
-      this->estimatedResultCount = internal->_search->get_matches_estimated();
     }
+    zim::Search* search = new zim::Search(zims);
+    search->set_verbose(verbose);
+    search->set_query(unaccentedSearch);
+    search->set_range(resultStart, resultEnd);
+    internal->_search = search;
+    internal->current_iterator = internal->_search->begin();
+    this->estimatedResultCount = internal->_search->get_matches_estimated();
   }
 
   return;
@@ -209,10 +171,6 @@ void Searcher::geo_search(float latitude, float longitude, float distance,
     return;
   }
 
-  if (internal->_xapianSearcher) {
-    return;
-  }
-
   /* Avoid big researches */
   this->resultCountPerPage = resultEnd - resultStart;
   if (this->resultCountPerPage > MAX_SEARCH_LEN) {
@@ -233,6 +191,7 @@ void Searcher::geo_search(float latitude, float longitude, float distance,
     zims.push_back((*current)->getZimFileHandler());
   }
   zim::Search* search = new zim::Search(zims);
+  search->set_verbose(verbose);
   search->set_query("");
   search->set_georange(latitude, longitude, distance);
   search->set_range(resultStart, resultEnd);
@@ -244,18 +203,14 @@ void Searcher::geo_search(float latitude, float longitude, float distance,
 
 void Searcher::restart_search()
 {
-  if (internal->_xapianSearcher) {
-    internal->_xapianSearcher->restart_search();
-  } else if (internal->_search) {
+  if (internal->_search) {
     internal->current_iterator = internal->_search->begin();
   }
 }
 
 Result* Searcher::getNextResult()
 {
-  if (internal->_xapianSearcher) {
-    return internal->_xapianSearcher->getNextResult();
-  } else if (internal->_search &&
+  if (internal->_search &&
              internal->current_iterator != internal->_search->end()) {
     Result* result = new _Result(internal->current_iterator);
     internal->current_iterator++;
@@ -272,37 +227,32 @@ void Searcher::reset()
   return;
 }
 
-void Searcher::suggestions(std::string& search, const bool verbose)
+void Searcher::suggestions(std::string& searchPattern, const bool verbose)
 {
   this->reset();
 
   if (verbose == true) {
-    cout << "Performing suggestion query `" << search << "`" << endl;
+    cout << "Performing suggestion query `" << searchPattern << "`" << endl;
   }
 
-  this->searchPattern = search;
+  this->searchPattern = searchPattern;
   this->resultStart = 0;
   this->resultEnd = 10;
-  string unaccentedSearch = removeAccents(search);
+  string unaccentedSearch = removeAccents(searchPattern);
 
-  if (internal->_xapianSearcher) {
-    /* [TODO] Suggestion on a external database ?
-     * We do not support that. */
-    this->estimatedResultCount = 0;
-  } else {
-    std::vector<const zim::File*> zims;
-    for (auto current = this->readers.begin(); current != this->readers.end();
-         current++) {
-      zims.push_back((*current)->getZimFileHandler());
-    }
-    zim::Search* search = new zim::Search(zims);
-    search->set_query(unaccentedSearch);
-    search->set_range(resultStart, resultEnd);
-    search->set_suggestion_mode(true);
-    internal->_search = search;
-    internal->current_iterator = internal->_search->begin();
-    this->estimatedResultCount = internal->_search->get_matches_estimated();
+  std::vector<const zim::File*> zims;
+  for (auto current = this->readers.begin(); current != this->readers.end();
+       current++) {
+    zims.push_back((*current)->getZimFileHandler());
   }
+  zim::Search* search = new zim::Search(zims);
+  search->set_verbose(verbose);
+  search->set_query(unaccentedSearch);
+  search->set_range(resultStart, resultEnd);
+  search->set_suggestion_mode(true);
+  internal->_search = search;
+  internal->current_iterator = internal->_search->begin();
+  this->estimatedResultCount = internal->_search->get_matches_estimated();
 }
 
 /* Return the result count estimation */
@@ -363,46 +313,30 @@ int _Result::get_readerIndex()
 {
   return iterator.get_fileIndex();
 }
-#ifdef ENABLE_CTPP2
 
 string Searcher::getHtml()
 {
-  SimpleVM oSimpleVM(
-      1024, //iIMaxFunctions (default value)
-      4096, //iIMaxArgStackSize (default value)
-      4096, //iIMaxCodeStackSize (default value)
-      10240 * 2 //iIMaxSteps (default*2)
-  );
-
-  // Fill data
-  CDT oData;
-  CDT resultsCDT(CDT::ARRAY_VAL);
+  kainjow::mustache::data results{kainjow::mustache::data::type::list};
 
   this->restart_search();
   Result* p_result = NULL;
   while ((p_result = this->getNextResult())) {
-    CDT result;
-    result["title"] = p_result->get_title();
-    result["url"] = p_result->get_url();
-    result["snippet"] = p_result->get_snippet();
-    result["contentId"] = humanReaderNames[p_result->get_readerIndex()];
-
-    if (p_result->get_size() >= 0) {
-      result["size"] = kiwix::beautifyInteger(p_result->get_size());
-    }
+    kainjow::mustache::data result;
+    result.set("title", p_result->get_title());
+    result.set("url", p_result->get_url());
+    result.set("snippet", p_result->get_snippet());
+    result.set("resultContentId", humanReaderNames[p_result->get_readerIndex()]);
 
     if (p_result->get_wordCount() >= 0) {
-      result["wordCount"] = kiwix::beautifyInteger(p_result->get_wordCount());
+      result.set("wordCount", kiwix::beautifyInteger(p_result->get_wordCount()));
     }
 
-    resultsCDT.PushBack(result);
+    results.push_back(result);
     delete p_result;
   }
-  this->restart_search();
-  oData["results"] = resultsCDT;
 
   // pages
-  CDT pagesCDT(CDT::ARRAY_VAL);
+  kainjow::mustache::data pages{kainjow::mustache::data::type::list};
 
   unsigned int pageStart
       = this->resultStart / this->resultCountPerPage >= 5
@@ -418,48 +352,41 @@ string Searcher::getHtml()
   }
 
   for (unsigned int i = pageStart; i < pageStart + pageCount; i++) {
-    CDT page;
-    page["label"] = i + 1;
-    page["start"] = i * this->resultCountPerPage;
-    page["end"] = (i + 1) * this->resultCountPerPage;
+    kainjow::mustache::data page;
+    page.set("label", to_string(i + 1));
+    page.set("start", to_string(i * this->resultCountPerPage));
+    page.set("end", to_string((i + 1) * this->resultCountPerPage));
 
     if (i * this->resultCountPerPage == this->resultStart) {
-      page["selected"] = true;
+      page.set("selected", true);
     }
-
-    pagesCDT.PushBack(page);
+    pages.push_back(page);
   }
-  oData["pages"] = pagesCDT;
 
-  oData["count"] = kiwix::beautifyInteger(this->estimatedResultCount);
-  oData["searchPattern"] = kiwix::encodeDiples(this->searchPattern);
-  oData["searchPatternEncoded"] = urlEncode(this->searchPattern);
-  oData["resultStart"] = this->resultStart + 1;
-  oData["resultEnd"] = (this->resultEnd > this->estimatedResultCount
-                            ? this->estimatedResultCount
-                            : this->resultEnd);
-  oData["resultRange"] = this->resultCountPerPage;
-  oData["resultLastPageStart"]
-      = this->estimatedResultCount > this->resultCountPerPage
-            ? std::round(this->estimatedResultCount / this->resultCountPerPage) * this->resultCountPerPage
-            : 0;
-  oData["protocolPrefix"] = this->protocolPrefix;
-  oData["searchProtocolPrefix"] = this->searchProtocolPrefix;
-  oData["contentId"] = this->contentHumanReadableId;
+  std::string template_str = RESOURCE::search_result_tmpl;
+  kainjow::mustache::mustache tmpl(template_str);
 
-  std::string template_ct2 = RESOURCE::results_ct2;
-  VMStringLoader oLoader(template_ct2.c_str(), template_ct2.size());
+  kainjow::mustache::data allData;
+  allData.set("results", results);
+  allData.set("pages", pages);
+  allData.set("hasResult", this->estimatedResultCount != 0);
+  allData.set("count", kiwix::beautifyInteger(this->estimatedResultCount));
+  allData.set("searchPattern", kiwix::encodeDiples(this->searchPattern));
+  allData.set("searchPatternEncoded", urlEncode(this->searchPattern));
+  allData.set("resultStart", to_string(this->resultStart + 1));
+  allData.set("resultEnd", to_string(min(this->resultEnd, this->estimatedResultCount)));
+  allData.set("resultRange", to_string(this->resultCountPerPage));
+  allData.set("resultLastPageStart", to_string(this->estimatedResultCount > this->resultCountPerPage
+             ? round(this->estimatedResultCount / this->resultCountPerPage) * this->resultCountPerPage
+             : 0));
+  allData.set("lastResult", to_string(this->estimatedResultCount));
+  allData.set("protocolPrefix", this->protocolPrefix);
+  allData.set("searchProtocolPrefix", this->searchProtocolPrefix);
+  allData.set("contentId", this->contentHumanReadableId);
 
-  FileLogger oLogger(stderr);
-
-  // DEBUG only (write output to stdout)
-  // oSimpleVM.Run(oData, oLoader, stdout, oLogger);
-
-  std::string sResult;
-  oSimpleVM.Run(oData, oLoader, sResult, oLogger);
-
-  return sResult;
+  std::stringstream ss;
+  tmpl.render(allData, [&ss](const std::string& str) { ss << str; });
+  return ss.str();
 }
 
-#endif
 }

src/subprocess.cpp

@@ -0,0 +1,40 @@
+
+
+#include "subprocess.h"
+
+#ifdef _WIN32
+# include "subprocess_windows.h"
+#else
+# include "subprocess_unix.h"
+#endif
+
+Subprocess::Subprocess(std::unique_ptr<SubprocessImpl> impl, commandLine_t& commandLine) :
+  mp_impl(std::move(impl))
+{
+  mp_impl->run(commandLine);
+}
+
+Subprocess::~Subprocess()
+{
+  mp_impl->kill();
+}
+
+std::unique_ptr<Subprocess> Subprocess::run(commandLine_t& commandLine)
+{
+#ifdef _WIN32
+  auto impl = std::unique_ptr<SubprocessImpl>(new WinImpl);
+#else
+  auto impl = std::unique_ptr<UnixImpl>(new UnixImpl);
+#endif
+  return std::unique_ptr<Subprocess>(new Subprocess(std::move(impl), commandLine));
+}
+
+bool Subprocess::isRunning()
+{
+  return mp_impl->isRunning();
+}
+
+bool Subprocess::kill()
+{
+  return mp_impl->kill();
+}

src/subprocess.h

@@ -0,0 +1,36 @@
+
+#ifndef KIWIX_SUBPROCESS_H_
+#define KIWIX_SUBPROCESS_H_
+
+#include <string>
+#include <memory>
+#include <vector>
+
+typedef std::vector<const char *> commandLine_t;
+
+class SubprocessImpl
+{
+  public:
+    virtual void run(commandLine_t& commandLine) = 0;
+    virtual bool kill() = 0;
+    virtual bool isRunning() = 0;
+    virtual ~SubprocessImpl() = default;
+};
+
+class Subprocess
+{
+  private:
+    // Impl depends of the system (window, unix, ...)
+    std::unique_ptr<SubprocessImpl> mp_impl;
+    Subprocess(std::unique_ptr<SubprocessImpl> impl, commandLine_t& commandLine);
+
+  public:
+    static std::unique_ptr<Subprocess> run(commandLine_t& commandLine);
+    ~Subprocess();
+
+    bool isRunning();
+    bool kill();
+};
+
+
+#endif // KIWIX_SUBPROCESS_H_

src/subprocess_unix.cpp

@@ -0,0 +1,93 @@
+
+
+#include "subprocess_unix.h"
+
+#include <sys/types.h>
+#include <unistd.h>
+#include <iostream>
+#include <signal.h>
+#include <sys/wait.h>
+#include <stdlib.h>
+
+UnixImpl::UnixImpl():
+  m_pid(0),
+  m_running(false),
+  m_mutex(PTHREAD_MUTEX_INITIALIZER),
+  m_waitingThread()
+{
+}
+
+UnixImpl::~UnixImpl()
+{
+  kill();
+// Android has no pthread_cancel :(
+#ifdef __ANDROID__
+  pthread_kill(m_waitingThread, SIGUSR1);
+#else
+  pthread_cancel(m_waitingThread);
+#endif
+}
+
+#ifdef __ANDROID__
+void thread_exit_handler(int sig) {
+  pthread_exit(0);
+}
+#endif
+
+void* UnixImpl::waitForPID(void* _self)
+{
+#ifdef __ANDROID__
+  struct sigaction actions;
+  memset(&actions, 0, sizeof(actions));
+  sigemptyset(&actions.sa_mask);
+  actions.sa_flags = 0;
+  actions.sa_handler = thread_exit_handler;
+  sigaction(SIGUSR1, &actions, NULL);
+#endif
+
+  UnixImpl* self = static_cast<UnixImpl*>(_self);
+  waitpid(self->m_pid, NULL, 0);
+
+  pthread_mutex_lock(&self->m_mutex);
+  self->m_running = false;
+  pthread_mutex_unlock(&self->m_mutex);
+
+  return self;
+}
+
+void UnixImpl::run(commandLine_t& commandLine)
+{
+  const char* binary = commandLine[0];
+  int pid = fork();
+  switch(pid) {
+    case -1:
+      std::cerr << "cannot fork" << std::endl;
+      break;
+    case 0:
+      commandLine.push_back(NULL);
+      if (execvp(binary, const_cast<char* const*>(commandLine.data()))) {
+        perror("Cannot launch\n");
+        _exit(-1);
+      }
+
+      break;
+    default:
+      m_pid = pid;
+      m_running = true;
+      pthread_create(&m_waitingThread, NULL, waitForPID, this);
+      break;
+  }
+}
+
+bool UnixImpl::kill()
+{
+  return (::kill(m_pid, SIGKILL) == 0);
+}
+
+bool UnixImpl::isRunning()
+{
+  pthread_mutex_lock(&m_mutex);
+  bool ret = m_running;
+  pthread_mutex_unlock(&m_mutex);
+  return ret;
+}

src/subprocess_unix.h

@@ -0,0 +1,28 @@
+#ifndef KIWIX_SUBPROCESS_UNIX_H_
+#define KIWIX_SUBPROCESS_UNIX_H_
+
+#include "subprocess.h"
+
+#include <pthread.h>
+
+
+class UnixImpl : public SubprocessImpl
+{
+  private:
+    int m_pid;
+    bool m_running;
+    pthread_mutex_t m_mutex;
+    pthread_t m_waitingThread;
+
+  public:
+    UnixImpl();
+    virtual ~UnixImpl();
+
+    void run(commandLine_t& commandLine);
+    bool kill();
+    bool isRunning();
+
+    static void* waitForPID(void* self);
+};
+
+#endif //KIWIX_SUBPROCESS_UNIX_H_

src/subprocess_windows.cpp

@@ -0,0 +1,93 @@
+
+
+#include "subprocess_windows.h"
+
+#include <windows.h>
+#include <winbase.h>
+#include <iostream>
+#include <sstream>
+
+WinImpl::WinImpl():
+  m_pid(0),
+  m_running(false),
+  m_handle(INVALID_HANDLE_VALUE)
+{
+  InitializeCriticalSection(&m_criticalSection);
+}
+
+WinImpl::~WinImpl()
+{
+  kill();
+  CloseHandle(m_handle);
+  DeleteCriticalSection(&m_criticalSection);
+}
+
+DWORD WINAPI WinImpl::waitForPID(void* _self)
+{
+  WinImpl* self = static_cast<WinImpl*>(_self);
+  WaitForSingleObject(self->m_handle, INFINITE);
+
+  EnterCriticalSection(&self->m_criticalSection);
+  self->m_running = false;
+  LeaveCriticalSection(&self->m_criticalSection);
+
+  return 0;
+}
+
+std::unique_ptr<wchar_t[]> toWideChar(const std::string& value)
+{
+  auto size = MultiByteToWideChar(CP_UTF8, 0,
+                value.c_str(), -1, nullptr, 0);
+  auto wdata = std::unique_ptr<wchar_t[]>(new wchar_t[size]);
+  auto ret = MultiByteToWideChar(CP_UTF8, 0,
+                value.c_str(), -1, wdata.get(), size);
+  if (0 == ret) {
+    std::ostringstream oss;
+    oss << "Cannot convert to wchar : " << GetLastError();
+    throw std::runtime_error(oss.str());
+  }
+  return wdata;
+}
+
+
+void WinImpl::run(commandLine_t& commandLine)
+{
+  STARTUPINFOW startInfo = {0};
+  PROCESS_INFORMATION procInfo;
+  startInfo.cb = sizeof(startInfo);
+  std::ostringstream oss;
+  for(auto& item: commandLine) {
+    oss << item << " ";
+  }
+  auto wCommandLine = toWideChar(oss.str());
+  if (CreateProcessW(
+    NULL,
+    wCommandLine.get(),
+    NULL,
+    NULL,
+    false,
+    CREATE_NO_WINDOW,
+    NULL,
+    NULL,
+    &startInfo,
+    &procInfo)) {
+    m_pid = procInfo.dwProcessId;
+    m_handle = procInfo.hProcess;
+    CloseHandle(procInfo.hThread);
+    m_running = true;
+    CreateThread(NULL, 0, &waitForPID, this, 0, NULL	);
+  }
+}
+
+bool WinImpl::kill()
+{
+  return TerminateProcess(m_handle, 0);
+}
+
+bool WinImpl::isRunning()
+{
+  EnterCriticalSection(&m_criticalSection);
+  bool ret = m_running;
+  LeaveCriticalSection(&m_criticalSection);
+  return ret;
+}

src/subprocess_windows.h

@@ -0,0 +1,28 @@
+#ifndef KIWIX_SUBPROCESS_WINDOWS_H_
+#define KIWIX_SUBPROCESS_WINDOWS_H_
+
+#include "subprocess.h"
+
+#include <windows.h>
+#include <synchapi.h>
+
+class WinImpl : public SubprocessImpl
+{
+  private:
+    int m_pid;
+    bool m_running;
+    HANDLE m_handle;
+    CRITICAL_SECTION m_criticalSection;
+
+  public:
+    WinImpl();
+    virtual ~WinImpl();
+
+    void run(commandLine_t& commandLine);
+    bool kill();
+    bool isRunning();
+
+    static DWORD WINAPI waitForPID(void* self);
+};
+
+#endif //KIWIX_SUBPROCESS_WINDOWS_H_

src/tools/base64.cpp

@@ -24,7 +24,7 @@
    René Nyffenegger rene.nyffenegger@adp-gmbh.ch
 */
 
-#include <common/base64.h>
+#include <tools/base64.h>
 #include <iostream>
 
 static const std::string base64_chars = 
@@ -37,8 +37,10 @@ static inline bool is_base64(unsigned char c) {
   return (isalnum(c) || (c == '+') || (c == '/'));
 }
 
-std::string base64_encode(unsigned char const* bytes_to_encode, unsigned int in_len) {
+std::string base64_encode(const std::string& inString) {
   std::string ret;
+  auto in_len = inString.size();
+  const unsigned char* bytes_to_encode = reinterpret_cast<const unsigned char*>(inString.data());
   int i = 0;
   int j = 0;
   unsigned char char_array_3[3];

src/tools/networkTools.cpp

@@ -0,0 +1,59 @@
+/*
+ * Copyright 2012 Emmanuel Engelhart <kelson@kiwix.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#include <tools/networkTools.h>
+
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+
+#include <curl/curl.h>
+
+#include <sstream>
+#include <iostream>
+#include <stdexcept>
+
+
+size_t write_callback_to_iss(char* ptr, size_t size, size_t nmemb, void* userdata)
+{
+  auto str = static_cast<std::stringstream*>(userdata);
+  str->write(ptr, nmemb);
+  return nmemb;
+}
+
+std::string kiwix::download(const std::string& url) {
+  auto curl = curl_easy_init();
+  std::stringstream ss;
+  curl_easy_setopt(curl, CURLOPT_URL, url.c_str());
+  curl_easy_setopt(curl, CURLOPT_HTTPGET, 1L);
+  curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, &write_callback_to_iss);
+  curl_easy_setopt(curl, CURLOPT_WRITEDATA, &ss);
+  auto res = curl_easy_perform(curl);
+  if (res != CURLE_OK) {
+    curl_easy_cleanup(curl);
+    throw std::runtime_error("Cannot perform request");
+  }
+  long response_code;
+  curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, &response_code);
+  curl_easy_cleanup(curl);
+  if (response_code != 200) {
+    throw std::runtime_error("Invalid return code from server");
+  }
+  return ss.str();
+}

src/tools/otherTools.cpp

@@ -0,0 +1,197 @@
+/*
+ * Copyright 2014 Emmanuel Engelhart <kelson@kiwix.org>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+#include <tools/otherTools.h>
+#include <map>
+
+static std::map<std::string, std::string> codeisomapping {
+{ "aa", "aar" },
+{ "af", "afr" },
+{ "ak", "aka" },
+{ "am", "amh" },
+{ "ar", "ara" },
+{ "as", "asm" },
+{ "az", "aze" },
+{ "ba", "bak" },
+{ "be", "bel" },
+{ "bg", "bul" },
+{ "bm", "bam" },
+{ "bn", "ben" },
+{ "bo", "bod" },
+{ "br", "bre" },
+{ "bs", "bos" },
+{ "ca", "cat" },
+{ "ce", "che" },
+{ "co", "cos" },
+{ "cs", "ces" },
+{ "cu", "chu" },
+{ "cv", "chv" },
+{ "cy", "cym" },
+{ "da", "dan" },
+{ "de", "deu" },
+{ "dv", "div" },
+{ "dz", "dzo" },
+{ "ee", "ewe" },
+{ "el", "ell" },
+{ "en", "eng" },
+{ "es", "spa" },
+{ "et", "est" },
+{ "eu", "eus" },
+{ "fa", "fas" },
+{ "ff", "ful" },
+{ "fi", "fin" },
+{ "fo", "fao" },
+{ "fr", "fra" },
+{ "fy", "fry" },
+{ "ga", "gle" },
+{ "gd", "gla" },
+{ "gl", "glg" },
+{ "gn", "grn" },
+{ "gu", "guj" },
+{ "gv", "glv" },
+{ "ha", "hau" },
+{ "he", "heb" },
+{ "hi", "hin" },
+{ "hr", "hrv" },
+{ "hu", "hun" },
+{ "hy", "hye" },
+{ "ia", "ina" },
+{ "id", "ind" },
+{ "ig", "ibo" },
+{ "is", "isl" },
+{ "it", "ita" },
+{ "iu", "iku" },
+{ "ja", "jpn" },
+{ "jv", "jav" },
+{ "ka", "kat" },
+{ "ki", "kik" },
+{ "kk", "kaz" },
+{ "kl", "kal" },
+{ "km", "khm" },
+{ "kn", "kan" },
+{ "ko", "kor" },
+{ "ks", "kas" },
+{ "ku", "kur" },
+{ "kw", "cor" },
+{ "ky", "kir" },
+{ "lb", "ltz" },
+{ "lg", "lug" },
+{ "ln", "lin" },
+{ "lo", "lao" },
+{ "lt", "lit" },
+{ "lv", "lav" },
+{ "mg", "mlg" },
+{ "mi", "mri" },
+{ "mi", "mri" },
+{ "mk", "mkd" },
+{ "ml", "mal" },
+{ "mn", "mon" },
+{ "mr", "mar" },
+{ "ms", "msa" },
+{ "mt", "mlt" },
+{ "my", "mya" },
+{ "nb", "nob" },
+{ "ne", "nep" },
+{ "nl", "nld" },
+{ "nn", "nno" },
+{ "no", "nor" },
+{ "ny", "nya" },
+{ "oc", "oci" },
+{ "om", "orm" },
+{ "or", "ori" },
+{ "os", "oss" },
+{ "pa", "pan" },
+{ "pl", "pol" },
+{ "ps", "pus" },
+{ "pt", "por" },
+{ "qu", "que" },
+{ "rm", "roh" },
+{ "rn", "run" },
+{ "ro", "ron" },
+{ "ru", "rus" },
+{ "rw", "kin" },
+{ "sa", "san" },
+{ "sd", "snd" },
+{ "se", "sme" },
+{ "sg", "sag" },
+{ "si", "sin" },
+{ "sk", "slk" },
+{ "sl", "slv" },
+{ "sn", "sna" },
+{ "so", "som" },
+{ "sq", "sqi" },
+{ "sr", "srp" },
+{ "ss", "ssw" },
+{ "st", "sot" },
+{ "sv", "swe" },
+{ "sw", "swa" },
+{ "ta", "tam" },
+{ "te", "tel" },
+{ "tg", "tgk" },
+{ "th", "tha" },
+{ "ti", "tir" },
+{ "tk", "tuk" },
+{ "tl", "tgl" },
+{ "tn", "tsn" },
+{ "to", "ton" },
+{ "tr", "tur" },
+{ "ts", "tso" },
+{ "tt", "tat" },
+{ "ug", "uig" },
+{ "uk", "ukr" },
+{ "ur", "urd" },
+{ "uz", "uzb" },
+{ "ve", "ven" },
+{ "vi", "vie" },
+{ "wa", "wln" },
+{ "wo", "wol" },
+{ "xh", "xho" },
+{ "yo", "yor" },
+{ "zh", "zho" },
+{ "zu", "zul" }
+};
+
+void kiwix::sleep(unsigned int milliseconds)
+{
+#ifdef _WIN32
+  Sleep(milliseconds);
+#else
+  usleep(1000 * milliseconds);
+#endif
+}
+
+
+struct XmlStringWriter: pugi::xml_writer
+{
+  std::string result;
+  virtual void write(const void* data, size_t size){
+    result.append(static_cast<const char*>(data), size);
+  }
+};
+
+std::string kiwix::nodeToString(pugi::xml_node node)
+{
+  XmlStringWriter writer;
+  node.print(writer, "  ");
+  return writer.result;
+}
+
+std::string kiwix::converta2toa3(const std::string& a2code){
+  return codeisomapping.at(a2code);
+}

src/tools/pathTools.cpp

@@ -17,7 +17,7 @@
  * MA 02110-1301, USA.
  */
 
-#include <common/pathTools.h>
+#include <tools/pathTools.h>
 
 #ifdef __APPLE__
 #include <limits.h>
@@ -30,14 +30,9 @@
 #endif
 
 #ifdef _WIN32
+const std::string SEPARATOR("\\");
 #else
-#include <unistd.h>
-#endif
-
-#ifdef _WIN32
-#define SEPARATOR "\\"
-#else
-#define SEPARATOR "/"
+const std::string SEPARATOR("/");
 #include <unistd.h>
 #endif
 
@@ -66,9 +61,7 @@ string computeRelativePath(const string path, const string absolutePath)
   while (commonCount < pathParts.size()
          && commonCount < absolutePathParts.size()
          && pathParts[commonCount] == absolutePathParts[commonCount]) {
-    if (!pathParts[commonCount].empty()) {
       commonCount++;
-    }
   }
 
   string relativePath;
@@ -76,18 +69,17 @@ string computeRelativePath(const string path, const string absolutePath)
   /* On Windows you have a token more because the root is represented
      by a letter */
   if (commonCount == 0) {
-    relativePath = "../";
+    relativePath = ".." + SEPARATOR;
   }
 #endif
 
   for (unsigned int i = commonCount; i < pathParts.size(); i++) {
-    relativePath += "../";
+    relativePath += ".." + SEPARATOR;
   }
   for (unsigned int i = commonCount; i < absolutePathParts.size(); i++) {
     relativePath += absolutePathParts[i];
-    relativePath += i + 1 < absolutePathParts.size() ? "/" : "";
+    relativePath += i + 1 < absolutePathParts.size() ? SEPARATOR : "";
   }
-
   return relativePath;
 }
 
@@ -231,25 +223,20 @@ bool makeDirectory(const string& path)
 string makeTmpDirectory()
 {
 #ifdef _WIN32
-  char cbase[MAX_PATH+1];
-  int base_len = GetTempPath(MAX_PATH+1, cbase);
-  UUID uuid;
-  UuidCreate(&uuid);
-  char* dir_name;
-  UuidToString(&uuid, reinterpret_cast<unsigned char**>(&dir_name));
-  string dir(cbase, base_len);
-  dir += dir_name;
-  _mkdir(dir.c_str());
-  RpcStringFree(reinterpret_cast<unsigned char**>(&dir_name));
+  char cbase[MAX_PATH];
+  char ctmp[MAX_PATH];
+  GetTempPath(MAX_PATH-14, cbase);
+  // This create a file for us, ensure it is unique.
+  // So we need to delete it and create the directory using the same name.
+  GetTempFileName(cbase, "kiwix", 0, ctmp);
+  DeleteFile(ctmp);
+  _mkdir(ctmp);
+  return string(ctmp);
 #else
-  string base = "/tmp";
-  auto _template = base + "/kiwix-lib_XXXXXX";
-  char* _template_array = new char[_template.size()+1];
-  memcpy(_template_array, _template.c_str(), _template.size());
+  char _template_array[] = {"/tmp/kiwix-lib_XXXXXX"};
   string dir = mkdtemp(_template_array);
-  delete[] _template_array;
-#endif
   return dir;
+#endif
 }
 
 /* Try to create a link and if does not work then make a copy */
@@ -310,3 +297,29 @@ string getCurrentDirectory()
   free(a_cwd);
   return s_cwd;
 }
+
+string getDataDirectory()
+{
+#ifdef _WIN32
+  char* cDataDir = ::getenv("APPDATA");
+#else
+  char* cDataDir = ::getenv("KIWIX_DATA_DIR");
+#endif
+  std::string dataDir = cDataDir==nullptr ? "" : cDataDir;
+  if (!dataDir.empty())
+    return dataDir;
+#ifdef _WIN32
+  cDataDir = ::getenv("USERPROFILE");
+  dataDir = cDataDir==nullptr ? getCurrentDirectory() : cDataDir;
+#else
+  cDataDir = ::getenv("XDG_DATA_HOME");
+  dataDir = cDataDir==nullptr ? "" : cDataDir;
+  if (dataDir.empty()) {
+    cDataDir = ::getenv("HOME");
+    dataDir = cDataDir==nullptr ? getCurrentDirectory() : cDataDir;
+    dataDir = appendToDirectory(dataDir, ".local");
+    dataDir = appendToDirectory(dataDir, "share");
+  }
+#endif
+  return appendToDirectory(dataDir, "kiwix");
+}

src/tools/regexTools.cpp

@@ -17,14 +17,14 @@
  * MA 02110-1301, USA.
  */
 
-#include <common/regexTools.h>
+#include <tools/regexTools.h>
 
-std::map<std::string, RegexMatcher*> regexCache;
+std::map<std::string, icu::RegexMatcher*> regexCache;
 
-RegexMatcher* buildRegex(const std::string& regex)
+icu::RegexMatcher* buildRegex(const std::string& regex)
 {
-  RegexMatcher* matcher;
-  std::map<std::string, RegexMatcher*>::iterator itr = regexCache.find(regex);
+  icu::RegexMatcher* matcher;
+  auto itr = regexCache.find(regex);
 
   /* Regex is in cache */
   if (itr != regexCache.end()) {
@@ -34,8 +34,8 @@ RegexMatcher* buildRegex(const std::string& regex)
   /* Regex needs to be parsed (and cached) */
   else {
     UErrorCode status = U_ZERO_ERROR;
-    UnicodeString uregex = UnicodeString(regex.c_str());
-    matcher = new RegexMatcher(uregex, UREGEX_CASE_INSENSITIVE, status);
+    icu::UnicodeString uregex(regex.c_str());
+    matcher = new icu::RegexMatcher(uregex, UREGEX_CASE_INSENSITIVE, status);
     regexCache[regex] = matcher;
   }
 
@@ -49,8 +49,8 @@ void freeRegexCache()
 bool matchRegex(const std::string& content, const std::string& regex)
 {
   ucnv_setDefaultName("UTF-8");
-  UnicodeString ucontent = UnicodeString(content.c_str());
-  RegexMatcher* matcher = buildRegex(regex);
+  icu::UnicodeString ucontent(content.c_str());
+  auto matcher = buildRegex(regex);
   matcher->reset(ucontent);
   return matcher->find();
 }
@@ -60,12 +60,12 @@ std::string replaceRegex(const std::string& content,
                          const std::string& regex)
 {
   ucnv_setDefaultName("UTF-8");
-  UnicodeString ucontent = UnicodeString(content.c_str());
-  UnicodeString ureplacement = UnicodeString(replacement.c_str());
-  RegexMatcher* matcher = buildRegex(regex);
+  icu::UnicodeString ucontent(content.c_str());
+  icu::UnicodeString ureplacement(replacement.c_str());
+  auto matcher = buildRegex(regex);
   matcher->reset(ucontent);
   UErrorCode status = U_ZERO_ERROR;
-  UnicodeString uresult = matcher->replaceAll(ureplacement, status);
+  auto uresult = matcher->replaceAll(ureplacement, status);
   std::string tmp;
   uresult.toUTF8String(tmp);
   return tmp;
@@ -76,9 +76,9 @@ std::string appendToFirstOccurence(const std::string& content,
                                    const std::string& replacement)
 {
   ucnv_setDefaultName("UTF-8");
-  UnicodeString ucontent = UnicodeString(content.c_str());
-  UnicodeString ureplacement = UnicodeString(replacement.c_str());
-  RegexMatcher* matcher = buildRegex(regex);
+  icu::UnicodeString ucontent(content.c_str());
+  icu::UnicodeString ureplacement(replacement.c_str());
+  auto matcher = buildRegex(regex);
   matcher->reset(ucontent);
 
   if (matcher->find()) {

src/tools/stringTools.cpp

@@ -17,7 +17,7 @@
  * MA 02110-1301, USA.
  */
 
-#include <common/stringTools.h>
+#include <tools/stringTools.h>
 
 #include <unicode/normlzr.h>
 #include <unicode/rep.h>
@@ -33,7 +33,7 @@ void kiwix::loadICUExternalTables()
   std::string executablePath = getExecutablePath();
   std::string executableDirectory = removeLastPathElement(executablePath);
   std::string datPath
-      = computeAbsolutePath(executableDirectory, "icudt49l.dat");
+      = computeAbsolutePath(executableDirectory, "icudt58l.dat");
   try {
     u_setDataDirectory(datPath.c_str());
   } catch (exception& e) {
@@ -47,9 +47,9 @@ std::string kiwix::removeAccents(const std::string& text)
   loadICUExternalTables();
   ucnv_setDefaultName("UTF-8");
   UErrorCode status = U_ZERO_ERROR;
-  Transliterator* removeAccentsTrans = Transliterator::createInstance(
+  auto removeAccentsTrans = icu::Transliterator::createInstance(
       "Lower; NFD; [:M:] remove; NFC", UTRANS_FORWARD, status);
-  UnicodeString ustring = UnicodeString(text.c_str());
+  icu::UnicodeString ustring(text.c_str());
   removeAccentsTrans->transliterate(ustring);
   delete removeAccentsTrans;
   std::string unaccentedText;
@@ -57,10 +57,8 @@ std::string kiwix::removeAccents(const std::string& text)
   return unaccentedText;
 }
 
-#ifndef __ANDROID__
-
 /* Prepare integer for display */
-std::string kiwix::beautifyInteger(const unsigned int number)
+std::string kiwix::beautifyInteger(uint64_t number)
 {
   std::stringstream numberStream;
   numberStream << number;
@@ -75,17 +73,22 @@ std::string kiwix::beautifyInteger(const unsigned int number)
   return numberString;
 }
 
-std::string kiwix::beautifyFileSize(const unsigned int number)
+std::string kiwix::beautifyFileSize(uint64_t number)
 {
-  if (number > 1024 * 1024) {
-    return kiwix::beautifyInteger(number / (1024 * 1024)) + " GB";
-  } else {
-    return kiwix::beautifyInteger(number / 1024 != 0 ? number / 1024 : 1)
-           + " MB";
-  }
+  std::stringstream ss;
+  ss << std::fixed << std::setprecision(2);
+  if (number>>30)
+    ss << (number/(1024.0*1024*1024)) << " GB";
+  else if (number>>20)
+    ss << (number/(1024.0*1024)) << " MB";
+  else if (number>>10)
+    ss << (number/1024.0) << " KB";
+  else
+    ss << number << " B";
+  return ss.str();
 }
 
-void kiwix::printStringInHexadecimal(UnicodeString s)
+void kiwix::printStringInHexadecimal(icu::UnicodeString s)
 {
   std::cout << std::showbase << std::hex;
   for (int i = 0; i < s.length(); i++) {
@@ -133,8 +136,6 @@ std::string kiwix::encodeDiples(const std::string& str)
   return result;
 }
 
-#endif
-
 /* urlEncode() based on javascript encodeURI() &
    encodeURIComponent(). Mostly code from rstudio/httpuv (GPLv3) */
 
@@ -300,8 +301,8 @@ std::string kiwix::ucFirst(const std::string& word)
 
   std::string result;
 
-  UnicodeString unicodeWord(word.c_str());
-  UnicodeString unicodeFirstLetter = UnicodeString(unicodeWord, 0, 1).toUpper();
+  icu::UnicodeString unicodeWord(word.c_str());
+  auto unicodeFirstLetter = icu::UnicodeString(unicodeWord, 0, 1).toUpper();
   unicodeWord.replace(0, 1, unicodeFirstLetter);
   unicodeWord.toUTF8String(result);
 
@@ -316,7 +317,7 @@ std::string kiwix::ucAll(const std::string& word)
 
   std::string result;
 
-  UnicodeString unicodeWord(word.c_str());
+  icu::UnicodeString unicodeWord(word.c_str());
   unicodeWord.toUpper().toUTF8String(result);
 
   return result;
@@ -330,8 +331,8 @@ std::string kiwix::lcFirst(const std::string& word)
 
   std::string result;
 
-  UnicodeString unicodeWord(word.c_str());
-  UnicodeString unicodeFirstLetter = UnicodeString(unicodeWord, 0, 1).toLower();
+  icu::UnicodeString unicodeWord(word.c_str());
+  auto unicodeFirstLetter = icu::UnicodeString(unicodeWord, 0, 1).toLower();
   unicodeWord.replace(0, 1, unicodeFirstLetter);
   unicodeWord.toUTF8String(result);
 
@@ -346,7 +347,7 @@ std::string kiwix::lcAll(const std::string& word)
 
   std::string result;
 
-  UnicodeString unicodeWord(word.c_str());
+  icu::UnicodeString unicodeWord(word.c_str());
   unicodeWord.toLower().toUTF8String(result);
 
   return result;
@@ -360,7 +361,7 @@ std::string kiwix::toTitle(const std::string& word)
 
   std::string result;
 
-  UnicodeString unicodeWord(word.c_str());
+  icu::UnicodeString unicodeWord(word.c_str());
   unicodeWord = unicodeWord.toTitle(0);
   unicodeWord.toUTF8String(result);
 

src/xapian/htmlparse.cc

@@ -1,373 +0,0 @@
-/* htmlparse.cc: simple HTML parser for omega indexer
- *
- * Copyright 1999,2000,2001 BrightStation PLC
- * Copyright 2001 Ananova Ltd
- * Copyright 2002,2006,2007,2008 Olly Betts
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation; either version 2 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301
- * USA
- */
-
-// #include <config.h>
-
-#include "htmlparse.h"
-
-#include <xapian.h>
-
-// #include "utf8convert.h"
-
-#include <algorithm>
-
-#include <ctype.h>
-#include <cstring>
-#include <stdio.h>
-#include <stdlib.h>
-
-using namespace std;
-
-inline void
-lowercase_string(string &str)
-{
-    for (string::iterator i = str.begin(); i != str.end(); ++i) {
-	*i = tolower(static_cast<unsigned char>(*i));
-    }
-}
-
-map<string, unsigned int> HtmlParser::named_ents;
-
-inline static bool
-p_notdigit(char c)
-{
-    return !isdigit(static_cast<unsigned char>(c));
-}
-
-inline static bool
-p_notxdigit(char c)
-{
-    return !isxdigit(static_cast<unsigned char>(c));
-}
-
-inline static bool
-p_notalnum(char c)
-{
-    return !isalnum(static_cast<unsigned char>(c));
-}
-
-inline static bool
-p_notwhitespace(char c)
-{
-    return !isspace(static_cast<unsigned char>(c));
-}
-
-inline static bool
-p_nottag(char c)
-{
-    return !isalnum(static_cast<unsigned char>(c)) &&
-	c != '.' && c != '-' && c != ':'; // ':' for XML namespaces.
-}
-
-inline static bool
-p_whitespacegt(char c)
-{
-    return isspace(static_cast<unsigned char>(c)) || c == '>';
-}
-
-inline static bool
-p_whitespaceeqgt(char c)
-{
-    return isspace(static_cast<unsigned char>(c)) || c == '=' || c == '>';
-}
-
-bool
-HtmlParser::get_parameter(const string & param, string & value)
-{
-    map<string, string>::const_iterator i = parameters.find(param);
-    if (i == parameters.end()) return false;
-    value = i->second;
-    return true;
-}
-
-HtmlParser::HtmlParser()
-{
-    static const struct ent { const char *n; unsigned int v; } ents[] = {
-#include "namedentities.h"
-	{ NULL, 0 }
-    };
-    if (named_ents.empty()) {
-	const struct ent *i = ents;
-	while (i->n) {
-	    named_ents[string(i->n)] = i->v;
-	    ++i;
-	}
-    }
-}
-
-void
-HtmlParser::decode_entities(string &s)
-{
-    // We need a const_iterator version of s.end() - otherwise the
-    // find() and find_if() templates don't work...
-    string::const_iterator amp = s.begin(), s_end = s.end();
-    while ((amp = find(amp, s_end, '&')) != s_end) {
-	unsigned int val = 0;
-	string::const_iterator end, p = amp + 1;
-	if (p != s_end && *p == '#') {
-	    p++;
-	    if (p != s_end && (*p == 'x' || *p == 'X')) {
-		// hex
-		p++;
-		end = find_if(p, s_end, p_notxdigit);
-		sscanf(s.substr(p - s.begin(), end - p).c_str(), "%x", &val);
-	    } else {
-		// number
-		end = find_if(p, s_end, p_notdigit);
-		val = atoi(s.substr(p - s.begin(), end - p).c_str());
-	    }
-	} else {
-	    end = find_if(p, s_end, p_notalnum);
-	    string code = s.substr(p - s.begin(), end - p);
-	    map<string, unsigned int>::const_iterator i;
-	    i = named_ents.find(code);
-	    if (i != named_ents.end()) val = i->second;
-	}
-	if (end < s_end && *end == ';') end++;
-	if (val) {
-	    string::size_type amp_pos = amp - s.begin();
-	    if (val < 0x80) {
-		s.replace(amp_pos, end - amp, 1u, char(val));
-	    } else {
-		// Convert unicode value val to UTF-8.
-		char seq[4];
-		unsigned len = Xapian::Unicode::nonascii_to_utf8(val, seq);
-		s.replace(amp_pos, end - amp, seq, len);
-	    }
-	    s_end = s.end();
-	    // We've modified the string, so the iterators are no longer
-	    // valid...
-	    amp = s.begin() + amp_pos + 1;
-	} else {
-	    amp = end;
-	}
-    }
-}
-
-void
-HtmlParser::parse_html(const string &body)
-{
-    in_script = false;
-
-    parameters.clear();
-    string::const_iterator start = body.begin();
-
-    while (true) {
-	// Skip through until we find an HTML tag, a comment, or the end of
-	// document.  Ignore isolated occurrences of `<' which don't start
-	// a tag or comment.
-	string::const_iterator p = start;
-	while (true) {
-	    p = find(p, body.end(), '<');
-	    if (p == body.end()) break;
-	    unsigned char ch = *(p + 1);
-
-	    // Tag, closing tag, or comment (or SGML declaration).
-	    if ((!in_script && isalpha(ch)) || ch == '/' || ch == '!') break;
-
-	    if (ch == '?') {
-		// PHP code or XML declaration.
-		// XML declaration is only valid at the start of the first line.
-		// FIXME: need to deal with BOMs...
-		if (p != body.begin() || body.size() < 20) break;
-
-		// XML declaration looks something like this:
-		// <?xml version="1.0" encoding="UTF-8"?>
-		if (p[2] != 'x' || p[3] != 'm' || p[4] != 'l') break;
-		if (strchr(" \t\r\n", p[5]) == NULL) break;
-
-		string::const_iterator decl_end = find(p + 6, body.end(), '?');
-		if (decl_end == body.end()) break;
-
-		// Default charset for XML is UTF-8.
-		charset = "UTF-8";
-
-		string decl(p + 6, decl_end);
-		size_t enc = decl.find("encoding");
-		if (enc == string::npos) break;
-
-		enc = decl.find_first_not_of(" \t\r\n", enc + 8);
-		if (enc == string::npos || enc == decl.size()) break;
-
-		if (decl[enc] != '=') break;
-		
-		enc = decl.find_first_not_of(" \t\r\n", enc + 1);
-		if (enc == string::npos || enc == decl.size()) break;
-
-		if (decl[enc] != '"' && decl[enc] != '\'') break;
-
-		char quote = decl[enc++];
-		size_t enc_end = decl.find(quote, enc);
-
-		if (enc != string::npos)
-		    charset = decl.substr(enc, enc_end - enc);
-
-		break;
-	    }
-	    p++;
-	}
-
-	// Process text up to start of tag.
-	if (p > start) {
-	    string text = body.substr(start - body.begin(), p - start);
-	    // convert_to_utf8(text, charset);
-	    decode_entities(text);
-	    process_text(text);
-	}
-
-	if (p == body.end()) break;
-
-	start = p + 1;
-
-	if (start == body.end()) break;
-
-	if (*start == '!') {
-	    if (++start == body.end()) break;
-	    if (++start == body.end()) break;
-	    // comment or SGML declaration
-	    if (*(start - 1) == '-' && *start == '-') {
-		++start;
-		string::const_iterator close = find(start, body.end(), '>');
-		// An unterminated comment swallows rest of document
-		// (like Netscape, but unlike MSIE IIRC)
-		if (close == body.end()) break;
-
-		p = close;
-		// look for -->
-		while (p != body.end() && (*(p - 1) != '-' || *(p - 2) != '-'))
-		    p = find(p + 1, body.end(), '>');
-
-		if (p != body.end()) {
-		    // Check for htdig's "ignore this bit" comments.
-		    if (p - start == 15 && string(start, p - 2) == "htdig_noindex") {
-			string::size_type i;
-			i = body.find("<!--/htdig_noindex-->", p + 1 - body.begin());
-			if (i == string::npos) break;
-			start = body.begin() + i + 21;
-			continue;
-		    }
-		    // If we found --> skip to there.
-		    start = p;
-		} else {
-		    // Otherwise skip to the first > we found (as Netscape does).
-		    start = close;
-		}
-	    } else {
-		// just an SGML declaration, perhaps giving the DTD - ignore it
-		start = find(start - 1, body.end(), '>');
-		if (start == body.end()) break;
-	    }
-	    ++start;
-	} else if (*start == '?') {
-	    if (++start == body.end()) break;
-	    // PHP - swallow until ?> or EOF
-	    start = find(start + 1, body.end(), '>');
-
-	    // look for ?>
-	    while (start != body.end() && *(start - 1) != '?')
-		start = find(start + 1, body.end(), '>');
-
-	    // unterminated PHP swallows rest of document (rather arbitrarily
-	    // but it avoids polluting the database when things go wrong)
-	    if (start != body.end()) ++start;
-	} else {
-	    // opening or closing tag
-	    int closing = 0;
-
-	    if (*start == '/') {
-		closing = 1;
-		start = find_if(start + 1, body.end(), p_notwhitespace);
-	    }
-
-	    p = start;
-	    start = find_if(start, body.end(), p_nottag);
-	    string tag = body.substr(p - body.begin(), start - p);
-	    // convert tagname to lowercase
-	    lowercase_string(tag);
-
-	    if (closing) {
-		closing_tag(tag);
-		if (in_script && tag == "script") in_script = false;
-
-		/* ignore any bogus parameters on closing tags */
-		p = find(start, body.end(), '>');
-		if (p == body.end()) break;
-		start = p + 1;
-	    } else {
-		// FIXME: parse parameters lazily.
-		while (start < body.end() && *start != '>') {
-		    string name, value;
-
-		    p = find_if(start, body.end(), p_whitespaceeqgt);
-
-		    name.assign(body, start - body.begin(), p - start);
-
-		    p = find_if(p, body.end(), p_notwhitespace);
-
-		    start = p;
-		    if (start != body.end() && *start == '=') {
-			start = find_if(start + 1, body.end(), p_notwhitespace);
-
-			p = body.end();
-
-			int quote = *start;
-			if (quote == '"' || quote == '\'') {
-			    start++;
-			    p = find(start, body.end(), quote);
-			}
-
-			if (p == body.end()) {
-			    // unquoted or no closing quote
-			    p = find_if(start, body.end(), p_whitespacegt);
-			}
-			value.assign(body, start - body.begin(), p - start);
-			start = find_if(p, body.end(), p_notwhitespace);
-
-			if (!name.empty()) {
-			    // convert parameter name to lowercase
-			    lowercase_string(name);
-			    // in case of multiple entries, use the first
-			    // (as Netscape does)
-			    parameters.insert(make_pair(name, value));
-			}
-		    }
-		}
-#if 0
-		cout << "<" << tag;
-		map<string, string>::const_iterator x;
-		for (x = parameters.begin(); x != parameters.end(); x++) {
-		    cout << " " << x->first << "=\"" << x->second << "\"";
-		}
-		cout << ">\n";
-#endif
-		opening_tag(tag);
-		parameters.clear();
-
-		// In <script> tags we ignore opening tags to avoid problems
-		// with "a<b".
-		if (tag == "script") in_script = true;
-
-		if (start != body.end() && *start == '>') ++start;
-	    }
-	}
-    }
-}

src/xapian/htmlparse.h

@@ -1,49 +0,0 @@
-/* htmlparse.h: simple HTML parser for omega indexer
- *
- * Copyright 1999,2000,2001 BrightStation PLC
- * Copyright 2002,2006,2008 Olly Betts
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation; either version 2 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301
- * USA
- */
-
-#ifndef OMEGA_INCLUDED_HTMLPARSE_H
-#define OMEGA_INCLUDED_HTMLPARSE_H
-
-#include <string>
-#include <map>
-
-using std::string;
-using std::map;
-
-class HtmlParser {
-	map<string, string> parameters;
-    protected:
-	void decode_entities(string &s);
-	bool in_script;
-	string charset;
-	static map<string, unsigned int> named_ents;
-
-	bool get_parameter(const string & param, string & value);
-    public:
-	virtual void process_text(const string &/*text*/) { }
-	virtual void opening_tag(const string &/*tag*/) { }
-	virtual void closing_tag(const string &/*tag*/) { }
-	virtual void parse_html(const string &text);
-	HtmlParser();
-	virtual ~HtmlParser() { }
-};
-
-#endif // OMEGA_INCLUDED_HTMLPARSE_H

src/xapian/myhtmlparse.cc

@@ -1,302 +0,0 @@
-/* myhtmlparse.cc: subclass of HtmlParser for extracting text.
- *
- * Copyright 1999,2000,2001 BrightStation PLC
- * Copyright 2002,2003,2004,2006,2007,2008 Olly Betts
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation; either version 2 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301
- * USA
- */
-
-// #include <config.h>
-
-#include "myhtmlparse.h"
-
-// #include "utf8convert.h"
-
-#include <ctype.h>
-#include <string.h>
-
-inline void
-lowercase_string(string &str)
-{
-    for (string::iterator i = str.begin(); i != str.end(); ++i) {
-	*i = tolower(static_cast<unsigned char>(*i));
-    }
-}
-
-void
-MyHtmlParser::parse_html(const string &text, const string &charset_,
-			 bool charset_from_meta_)
-{
-    charset = charset_;
-    charset_from_meta = charset_from_meta_;
-    HtmlParser::parse_html(text);
-}
-
-void
-MyHtmlParser::process_text(const string &text)
-{
-    if (!text.empty() && !in_script_tag && !in_style_tag) {
-	string::size_type b = text.find_first_not_of(WHITESPACE);
-	if (b) pending_space = true;
-	while (b != string::npos) {
-	    if (pending_space && !dump.empty()) dump += ' ';
-	    string::size_type e = text.find_first_of(WHITESPACE, b);
-	    pending_space = (e != string::npos);
-	    if (!pending_space) {
-		dump.append(text.data() + b, text.size() - b);
-		return;
-	    }
-	    dump.append(text.data() + b, e - b);
-	    b = text.find_first_not_of(WHITESPACE, e + 1);
-	}
-    }
-}
-
-void
-MyHtmlParser::opening_tag(const string &tag)
-{
-    if (tag.empty()) return;
-    switch (tag[0]) {
-	case 'a':
-	    if (tag == "address") pending_space = true;
-	    break;
-	case 'b':
-	    if (tag == "body") {
-		dump.resize(0);
-		break;
-	    }
-	    if (tag == "blockquote" || tag == "br") pending_space = true;
-	    break;
-	case 'c':
-	    if (tag == "center") pending_space = true;
-	    break;
-	case 'd':
-	    if (tag == "dd" || tag == "dir" || tag == "div" || tag == "dl" ||
-		tag == "dt") pending_space = true;
-	    break;
-	case 'e':
-	    if (tag == "embed") pending_space = true;
-	    break;
-	case 'f':
-	    if (tag == "fieldset" || tag == "form") pending_space = true;
-	    break;
-	case 'h':
-	    // hr, and h1, ..., h6
-	    if (tag.length() == 2 && strchr("r123456", tag[1]))
-		pending_space = true;
-	    break;
-	case 'i':
-	    if (tag == "iframe" || tag == "img" || tag == "isindex" ||
-		tag == "input") pending_space = true;
-	    break;
-	case 'k':
-	    if (tag == "keygen") pending_space = true;
-	    break;
-	case 'l':
-	    if (tag == "legend" || tag == "li" || tag == "listing")
-		pending_space = true;
-	    break;
-	case 'm':
-	    if (tag == "meta") {
-		string content;
-		if (get_parameter("content", content)) {
-		    string name;
-		    if (get_parameter("name", name)) {
-			lowercase_string(name);
-			if (name == "description") {
-			    if (sample.empty()) {
-				swap(sample, content);
-				// convert_to_utf8(sample, charset);
-				decode_entities(sample);
-			    }
-			} else if (name == "keywords") {
-			    if (!keywords.empty()) keywords += ' ';
-			    // convert_to_utf8(content, charset);
-			    decode_entities(content);
-			    keywords += content;
-			} else if (name == "robots") {
-			    decode_entities(content);
-			    lowercase_string(content);
-			    if (content.find("none") != string::npos ||
-				content.find("noindex") != string::npos) {
-				indexing_allowed = false;
-				throw true;
-			    }
-			}
-			break;
-		    }
-		    // If the current charset came from a meta tag, don't
-		    // force reparsing again!
-		    if (charset_from_meta) break;
-		    string hdr;
-		    if (get_parameter("http-equiv", hdr)) {
-			lowercase_string(hdr);
-			if (hdr == "content-type") {
-			    lowercase_string(content);
-			    size_t start = content.find("charset=");
-			    if (start == string::npos) break;
-			    start += 8;
-			    if (start == content.size()) break;
-			    size_t end = start;
-			    if (content[start] != '"') {
-				while (end < content.size()) {
-				    unsigned char ch = content[end];
-				    if (ch <= 32 || ch >= 127 ||
-					strchr(";()<>@,:\\\"/[]?={}", ch))
-					break;
-				    ++end;
-				}
-			    } else {
-				++start;
-				++end;
-				while (end < content.size()) {
-				    unsigned char ch = content[end];
-				    if (ch == '"') break;
-				    if (ch == '\\') content.erase(end, 1);
-				    ++end;
-				}
-			    }
-			    string newcharset(content, start, end - start);
-			    if (charset != newcharset) {
-				throw newcharset;
-			    }
-			}
-		    }
-		    break;
-		}
-		if (charset_from_meta) break;
-		string newcharset;
-		if (get_parameter("charset", newcharset)) {
-		    // HTML5 added: <meta charset="...">
-		    lowercase_string(newcharset);
-		    if (charset != newcharset) {
-			throw newcharset;
-		    }
-		}
-		break;
-	    }
-	    if (tag == "marquee" || tag == "menu" || tag == "multicol")
-		pending_space = true;
-	    break;
-	case 'o':
-	    if (tag == "ol" || tag == "option") pending_space = true;
-	    break;
-	case 'p':
-	    if (tag == "p" || tag == "pre" || tag == "plaintext")
-		pending_space = true;
-	    break;
-	case 'q':
-	    if (tag == "q") pending_space = true;
-	    break;
-	case 's':
-	    if (tag == "style") {
-		in_style_tag = true;
-		break;
-	    }
-	    if (tag == "script") {
-		in_script_tag = true;
-		break;
-	    }
-	    if (tag == "select") pending_space = true;
-	    break;
-	case 't':
-	    if (tag == "table" || tag == "td" || tag == "textarea" ||
-		tag == "th") pending_space = true;
-	    break;
-	case 'u':
-	    if (tag == "ul") pending_space = true;
-	    break;
-	case 'x':
-	    if (tag == "xmp") pending_space = true;
-	    break;
-    }
-}
-
-void
-MyHtmlParser::closing_tag(const string &tag)
-{
-    if (tag.empty()) return;
-    switch (tag[0]) {
-	case 'a':
-	    if (tag == "address") pending_space = true;
-	    break;
-	case 'b':
-	    if (tag == "body") {
-		throw true;
-	    }
-	    if (tag == "blockquote" || tag == "br") pending_space = true;
-	    break;
-	case 'c':
-	    if (tag == "center") pending_space = true;
-	    break;
-	case 'd':
-	    if (tag == "dd" || tag == "dir" || tag == "div" || tag == "dl" ||
-		tag == "dt") pending_space = true;
-	    break;
-	case 'f':
-	    if (tag == "fieldset" || tag == "form") pending_space = true;
-	    break;
-	case 'h':
-	    // hr, and h1, ..., h6
-	    if (tag.length() == 2 && strchr("r123456", tag[1]))
-		pending_space = true;
-	    break;
-	case 'i':
-	    if (tag == "iframe") pending_space = true;
-	    break;
-	case 'l':
-	    if (tag == "legend" || tag == "li" || tag == "listing")
-		pending_space = true;
-	    break;
-	case 'm':
-	    if (tag == "marquee" || tag == "menu") pending_space = true;
-	    break;
-	case 'o':
-	    if (tag == "ol" || tag == "option") pending_space = true;
-	    break;
-	case 'p':
-	    if (tag == "p" || tag == "pre") pending_space = true;
-	    break;
-	case 'q':
-	    if (tag == "q") pending_space = true;
-	    break;
-	case 's':
-	    if (tag == "style") {
-		in_style_tag = false;
-		break;
-	    }
-	    if (tag == "script") {
-		in_script_tag = false;
-		break;
-	    }
-	    if (tag == "select") pending_space = true;
-	    break;
-	case 't':
-	    if (tag == "title") {
-		if (title.empty()) swap(title, dump);
-		break;
-	    }
-	    if (tag == "table" || tag == "td" || tag == "textarea" ||
-		tag == "th") pending_space = true;
-	    break;
-	case 'u':
-	    if (tag == "ul") pending_space = true;
-	    break;
-	case 'x':
-	    if (tag == "xmp") pending_space = true;
-	    break;
-    }
-}

src/xapian/myhtmlparse.h

@@ -1,66 +0,0 @@
-/* myhtmlparse.h: subclass of HtmlParser for extracting text
- *
- * Copyright 1999,2000,2001 BrightStation PLC
- * Copyright 2002,2003,2004,2006,2008 Olly Betts
- *
- * This program is free software; you can redistribute it and/or
- * modify it under the terms of the GNU General Public License as
- * published by the Free Software Foundation; either version 2 of the
- * License, or (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301
- * USA
- */
-
-#ifndef OMEGA_INCLUDED_MYHTMLPARSE_H
-#define OMEGA_INCLUDED_MYHTMLPARSE_H
-
-#include "htmlparse.h"
-
-// FIXME: Should we include \xa0 which is non-breaking space in iso-8859-1, but
-// not in all charsets and perhaps spans of all \xa0 should become a single
-// \xa0?
-#define WHITESPACE " \t\n\r"
-
-class MyHtmlParser : public HtmlParser {
-    public:
-	bool in_script_tag;
-	bool in_style_tag;
-	bool pending_space;
-	bool indexing_allowed;
-	bool charset_from_meta;
-	string title, sample, keywords, dump;
-	void process_text(const string &text);
-	void opening_tag(const string &tag);
-	void closing_tag(const string &tag);
-        using HtmlParser::parse_html;
-	void parse_html(const string &text, const string &charset_,
-			bool charset_from_meta_);
-	MyHtmlParser() :
-		in_script_tag(false),
-		in_style_tag(false),
-		pending_space(false),
-		indexing_allowed(true),
-		charset_from_meta(false) { }
-
-	void reset() {
-	    in_script_tag = false;
-	    in_style_tag = false;
-	    pending_space = false;
-	    indexing_allowed = true;
-	    charset_from_meta = false;
-	    title.resize(0);
-	    sample.resize(0);
-	    keywords.resize(0);
-	    dump.resize(0);
-	}
-};
-
-#endif // OMEGA_INCLUDED_MYHTMLPARSE_H

src/xapian/namedentities.h

@@ -1,279 +0,0 @@
-/* namedentities.h: named HTML entities.
- *
- * Copyright (C) 2006,2007 Olly Betts
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU General Public License as published by
- * the Free Software Foundation; either version 2 of the License, or
- * (at your option) any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
- */
-
-#ifndef OMEGA_INCLUDED_NAMEDENTITIES_H
-#define OMEGA_INCLUDED_NAMEDENTITIES_H
-
-// Names and values from: "Character entity references in HTML 4"
-// http://www.w3.org/TR/html4/sgml/entities.html
-{ "quot", 34 },
-{ "amp", 38 },
-{ "apos", 39 }, // Not in HTML 4 list but used in OpenOffice XML.
-{ "lt", 60 },
-{ "gt", 62 },
-{ "nbsp", 160 },
-{ "iexcl", 161 },
-{ "cent", 162 },
-{ "pound", 163 },
-{ "curren", 164 },
-{ "yen", 165 },
-{ "brvbar", 166 },
-{ "sect", 167 },
-{ "uml", 168 },
-{ "copy", 169 },
-{ "ordf", 170 },
-{ "laquo", 171 },
-{ "not", 172 },
-{ "shy", 173 },
-{ "reg", 174 },
-{ "macr", 175 },
-{ "deg", 176 },
-{ "plusmn", 177 },
-{ "sup2", 178 },
-{ "sup3", 179 },
-{ "acute", 180 },
-{ "micro", 181 },
-{ "para", 182 },
-{ "middot", 183 },
-{ "cedil", 184 },
-{ "sup1", 185 },
-{ "ordm", 186 },
-{ "raquo", 187 },
-{ "frac14", 188 },
-{ "frac12", 189 },
-{ "frac34", 190 },
-{ "iquest", 191 },
-{ "Agrave", 192 },
-{ "Aacute", 193 },
-{ "Acirc", 194 },
-{ "Atilde", 195 },
-{ "Auml", 196 },
-{ "Aring", 197 },
-{ "AElig", 198 },
-{ "Ccedil", 199 },
-{ "Egrave", 200 },
-{ "Eacute", 201 },
-{ "Ecirc", 202 },
-{ "Euml", 203 },
-{ "Igrave", 204 },
-{ "Iacute", 205 },
-{ "Icirc", 206 },
-{ "Iuml", 207 },
-{ "ETH", 208 },
-{ "Ntilde", 209 },
-{ "Ograve", 210 },
-{ "Oacute", 211 },
-{ "Ocirc", 212 },
-{ "Otilde", 213 },
-{ "Ouml", 214 },
-{ "times", 215 },
-{ "Oslash", 216 },
-{ "Ugrave", 217 },
-{ "Uacute", 218 },
-{ "Ucirc", 219 },
-{ "Uuml", 220 },
-{ "Yacute", 221 },
-{ "THORN", 222 },
-{ "szlig", 223 },
-{ "agrave", 224 },
-{ "aacute", 225 },
-{ "acirc", 226 },
-{ "atilde", 227 },
-{ "auml", 228 },
-{ "aring", 229 },
-{ "aelig", 230 },
-{ "ccedil", 231 },
-{ "egrave", 232 },
-{ "eacute", 233 },
-{ "ecirc", 234 },
-{ "euml", 235 },
-{ "igrave", 236 },
-{ "iacute", 237 },
-{ "icirc", 238 },
-{ "iuml", 239 },
-{ "eth", 240 },
-{ "ntilde", 241 },
-{ "ograve", 242 },
-{ "oacute", 243 },
-{ "ocirc", 244 },
-{ "otilde", 245 },
-{ "ouml", 246 },
-{ "divide", 247 },
-{ "oslash", 248 },
-{ "ugrave", 249 },
-{ "uacute", 250 },
-{ "ucirc", 251 },
-{ "uuml", 252 },
-{ "yacute", 253 },
-{ "thorn", 254 },
-{ "yuml", 255 },
-{ "OElig", 338 },
-{ "oelig", 339 },
-{ "Scaron", 352 },
-{ "scaron", 353 },
-{ "Yuml", 376 },
-{ "fnof", 402 },
-{ "circ", 710 },
-{ "tilde", 732 },
-{ "Alpha", 913 },
-{ "Beta", 914 },
-{ "Gamma", 915 },
-{ "Delta", 916 },
-{ "Epsilon", 917 },
-{ "Zeta", 918 },
-{ "Eta", 919 },
-{ "Theta", 920 },
-{ "Iota", 921 },
-{ "Kappa", 922 },
-{ "Lambda", 923 },
-{ "Mu", 924 },
-{ "Nu", 925 },
-{ "Xi", 926 },
-{ "Omicron", 927 },
-{ "Pi", 928 },
-{ "Rho", 929 },
-{ "Sigma", 931 },
-{ "Tau", 932 },
-{ "Upsilon", 933 },
-{ "Phi", 934 },
-{ "Chi", 935 },
-{ "Psi", 936 },
-{ "Omega", 937 },
-{ "alpha", 945 },
-{ "beta", 946 },
-{ "gamma", 947 },
-{ "delta", 948 },
-{ "epsilon", 949 },
-{ "zeta", 950 },
-{ "eta", 951 },
-{ "theta", 952 },
-{ "iota", 953 },
-{ "kappa", 954 },
-{ "lambda", 955 },
-{ "mu", 956 },
-{ "nu", 957 },
-{ "xi", 958 },
-{ "omicron", 959 },
-{ "pi", 960 },
-{ "rho", 961 },
-{ "sigmaf", 962 },
-{ "sigma", 963 },
-{ "tau", 964 },
-{ "upsilon", 965 },
-{ "phi", 966 },
-{ "chi", 967 },
-{ "psi", 968 },
-{ "omega", 969 },
-{ "thetasym", 977 },
-{ "upsih", 978 },
-{ "piv", 982 },
-{ "ensp", 8194 },
-{ "emsp", 8195 },
-{ "thinsp", 8201 },
-{ "zwnj", 8204 },
-{ "zwj", 8205 },
-{ "lrm", 8206 },
-{ "rlm", 8207 },
-{ "ndash", 8211 },
-{ "mdash", 8212 },
-{ "lsquo", 8216 },
-{ "rsquo", 8217 },
-{ "sbquo", 8218 },
-{ "ldquo", 8220 },
-{ "rdquo", 8221 },
-{ "bdquo", 8222 },
-{ "dagger", 8224 },
-{ "Dagger", 8225 },
-{ "bull", 8226 },
-{ "hellip", 8230 },
-{ "permil", 8240 },
-{ "prime", 8242 },
-{ "Prime", 8243 },
-{ "lsaquo", 8249 },
-{ "rsaquo", 8250 },
-{ "oline", 8254 },
-{ "frasl", 8260 },
-{ "euro", 8364 },
-{ "image", 8465 },
-{ "weierp", 8472 },
-{ "real", 8476 },
-{ "trade", 8482 },
-{ "alefsym", 8501 },
-{ "larr", 8592 },
-{ "uarr", 8593 },
-{ "rarr", 8594 },
-{ "darr", 8595 },
-{ "harr", 8596 },
-{ "crarr", 8629 },
-{ "lArr", 8656 },
-{ "uArr", 8657 },
-{ "rArr", 8658 },
-{ "dArr", 8659 },
-{ "hArr", 8660 },
-{ "forall", 8704 },
-{ "part", 8706 },
-{ "exist", 8707 },
-{ "empty", 8709 },
-{ "nabla", 8711 },
-{ "isin", 8712 },
-{ "notin", 8713 },
-{ "ni", 8715 },
-{ "prod", 8719 },
-{ "sum", 8721 },
-{ "minus", 8722 },
-{ "lowast", 8727 },
-{ "radic", 8730 },
-{ "prop", 8733 },
-{ "infin", 8734 },
-{ "ang", 8736 },
-{ "and", 8743 },
-{ "or", 8744 },
-{ "cap", 8745 },
-{ "cup", 8746 },
-{ "int", 8747 },
-{ "there4", 8756 },
-{ "sim", 8764 },
-{ "cong", 8773 },
-{ "asymp", 8776 },
-{ "ne", 8800 },
-{ "equiv", 8801 },
-{ "le", 8804 },
-{ "ge", 8805 },
-{ "sub", 8834 },
-{ "sup", 8835 },
-{ "nsub", 8836 },
-{ "sube", 8838 },
-{ "supe", 8839 },
-{ "oplus", 8853 },
-{ "otimes", 8855 },
-{ "perp", 8869 },
-{ "sdot", 8901 },
-{ "lceil", 8968 },
-{ "rceil", 8969 },
-{ "lfloor", 8970 },
-{ "rfloor", 8971 },
-{ "lang", 9001 },
-{ "rang", 9002 },
-{ "loz", 9674 },
-{ "spades", 9824 },
-{ "clubs", 9827 },
-{ "hearts", 9829 },
-{ "diams", 9830 },
-
-#endif // OMEGA_INCLUDED_NAMEDENTITIES_H

src/xapianSearcher.cpp

@@ -1,229 +0,0 @@
-/*
- * Copyright 2011 Emmanuel Engelhart <kelson@kiwix.org>
- *
- * This program is free software; you can redistribute it and/or modify
- * it under the terms of the GNU  General Public License as published by
- * the Free Software Foundation; either version 3 of the License, or
- * any later version.
- *
- * This program is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
- * GNU General Public License for more details.
- *
- * You should have received a copy of the GNU General Public License
- * along with this program; if not, write to the Free Software
- * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
- * MA 02110-1301, USA.
- */
-
-#include "xapianSearcher.h"
-#include <sys/types.h>
-#include <unicode/locid.h>
-#include <unistd.h>
-#include <zim/article.h>
-#include <zim/error.h>
-#include <zim/file.h>
-#include <zim/zim.h>
-#include "xapian/myhtmlparse.h"
-
-#include <vector>
-
-namespace kiwix
-{
-std::map<std::string, int> read_valuesmap(const std::string& s)
-{
-  std::map<std::string, int> result;
-  std::vector<std::string> elems = split(s, ";");
-  for (std::vector<std::string>::iterator elem = elems.begin();
-       elem != elems.end();
-       elem++) {
-    std::vector<std::string> tmp_elems = split(*elem, ":");
-    result.insert(
-        std::pair<std::string, int>(tmp_elems[0], atoi(tmp_elems[1].c_str())));
-  }
-  return result;
-}
-
-/* Constructor */
-XapianSearcher::XapianSearcher(const string& xapianDirectoryPath,
-                               Reader* reader)
-    : reader(reader)
-{
-  this->openIndex(xapianDirectoryPath);
-}
-
-/* Open Xapian readable database */
-void XapianSearcher::openIndex(const string& directoryPath)
-{
-  this->readableDatabase = Xapian::Database(directoryPath);
-  this->valuesmap
-      = read_valuesmap(this->readableDatabase.get_metadata("valuesmap"));
-  this->language = this->readableDatabase.get_metadata("language");
-  this->stopwords = this->readableDatabase.get_metadata("stopwords");
-  setup_queryParser();
-}
-
-/* Close Xapian writable database */
-void XapianSearcher::closeIndex()
-{
-  return;
-}
-void XapianSearcher::setup_queryParser()
-{
-  queryParser.set_database(readableDatabase);
-  if (!language.empty()) {
-    /* Build ICU Local object to retrieve ISO-639 language code (from
-       ISO-639-3) */
-    icu::Locale languageLocale(language.c_str());
-
-    /* Configuring language base steemming */
-    try {
-      stemmer = Xapian::Stem(languageLocale.getLanguage());
-      queryParser.set_stemmer(stemmer);
-      queryParser.set_stemming_strategy(Xapian::QueryParser::STEM_ALL);
-    } catch (...) {
-      std::cout << "No steemming for language '" << languageLocale.getLanguage()
-                << "'" << std::endl;
-    }
-  }
-
-  if (!stopwords.empty()) {
-    std::string stopWord;
-    std::istringstream file(this->stopwords);
-    while (std::getline(file, stopWord, '\n')) {
-      this->stopper.add(stopWord);
-    }
-    queryParser.set_stopper(&(this->stopper));
-  }
-}
-
-/* Search strings in the database */
-void XapianSearcher::searchInIndex(string& search,
-                                   const unsigned int resultStart,
-                                   const unsigned int resultEnd,
-                                   const bool verbose)
-{
-  /* Create the query */
-  Xapian::Query query = queryParser.parse_query(search);
-
-  /* Create the enquire object */
-  Xapian::Enquire enquire(this->readableDatabase);
-  enquire.set_query(query);
-
-  /* Get the results */
-  this->results = enquire.get_mset(resultStart, resultEnd - resultStart);
-  this->current_result = this->results.begin();
-}
-
-/* Get next result */
-Result* XapianSearcher::getNextResult()
-{
-  if (this->current_result != this->results.end()) {
-    XapianResult* result = new XapianResult(this, this->current_result);
-    this->current_result++;
-    return result;
-  }
-  return NULL;
-}
-
-void XapianSearcher::restart_search()
-{
-  this->current_result = this->results.begin();
-}
-
-XapianResult::XapianResult(XapianSearcher* searcher,
-                           Xapian::MSetIterator& iterator)
-    : searcher(searcher), iterator(iterator), document(iterator.get_document())
-{
-}
-
-std::string XapianResult::get_url()
-{
-  return document.get_data();
-}
-std::string XapianResult::get_title()
-{
-  if (searcher->valuesmap.empty()) {
-    /* This is the old legacy version. Guess and try */
-    return document.get_value(0);
-  } else if (searcher->valuesmap.find("title") != searcher->valuesmap.end()) {
-    return document.get_value(searcher->valuesmap["title"]);
-  }
-  return "";
-}
-
-int XapianResult::get_score()
-{
-  return iterator.get_percent();
-}
-std::string XapianResult::get_snippet()
-{
-  if (searcher->valuesmap.empty()) {
-    /* This is the old legacy version. Guess and try */
-    std::string stored_snippet = document.get_value(1);
-    if (!stored_snippet.empty()) {
-      return stored_snippet;
-    }
-    /* Let's continue here, and see if we can genenate one */
-  } else if (searcher->valuesmap.find("snippet") != searcher->valuesmap.end()) {
-    return document.get_value(searcher->valuesmap["snippet"]);
-  }
-  /* No reader, no snippet */
-  if (!searcher->reader) {
-    return "";
-  }
-  /* Get the content of the article to generate a snippet.
-     We parse it and use the html dump to avoid remove html tags in the
-     content and be able to nicely cut the text at random place. */
-  MyHtmlParser htmlParser;
-  std::string content = get_content();
-  if (content.empty()) {
-    return content;
-  }
-  try {
-    htmlParser.parse_html(content, "UTF-8", true);
-  } catch (...) {
-  }
-  return searcher->results.snippet(htmlParser.dump, 500);
-}
-
-std::string XapianResult::get_content()
-{
-  if (!searcher->reader) {
-    return "";
-  }
-  auto entry = searcher->reader->getEntryFromEncodedPath(get_url());
-  return entry.getContent();
-}
-
-int XapianResult::get_size()
-{
-  if (searcher->valuesmap.empty()) {
-    /* This is the old legacy version. Guess and try */
-    return document.get_value(2).empty() == true
-               ? -1
-               : atoi(document.get_value(2).c_str());
-  } else if (searcher->valuesmap.find("size") != searcher->valuesmap.end()) {
-    return atoi(document.get_value(searcher->valuesmap["size"]).c_str());
-  }
-  /* The size is never used. Do we really want to get the content and
-     calculate the size ? */
-  return -1;
-}
-
-int XapianResult::get_wordCount()
-{
-  if (searcher->valuesmap.empty()) {
-    /* This is the old legacy version. Guess and try */
-    return document.get_value(3).empty() == true
-               ? -1
-               : atoi(document.get_value(3).c_str());
-  } else if (searcher->valuesmap.find("wordcount")
-             != searcher->valuesmap.end()) {
-    return atoi(document.get_value(searcher->valuesmap["wordcount"]).c_str());
-  }
-  return -1;
-}
-
-}  // Kiwix namespace

src/xmlrpc.h

@@ -0,0 +1,264 @@
+
+
+#ifndef KIWIX_XMLRPC_H_
+#define KIWIX_XMLRPC_H_
+
+#include <tools/otherTools.h>
+
+namespace kiwix {
+
+class InvalidRPCNode : public std::runtime_error {
+  public:
+    InvalidRPCNode(const std::string& msg) : std::runtime_error(msg) {};
+};
+
+class Struct;
+class Array;
+
+class Value {
+    pugi::xml_node m_value;
+
+  public:
+    Value(pugi::xml_node value) : m_value(value) { }
+
+    void set(int value) {
+      if (!m_value.child("int"))
+        m_value.append_child("int");
+      m_value.child("int").text().set(value);
+    };
+
+    int getAsI() const {
+      if (!m_value.child("int"))
+        throw InvalidRPCNode("Type Error");
+      return m_value.child("int").text().as_int();
+    }
+
+    void set(bool value) {
+      if (!m_value.child("boolean"))
+        m_value.append_child("boolean");
+      m_value.child("boolean").text().set(value);
+    };
+
+    int getAsB() const {
+      if (!m_value.child("boolean"))
+        throw InvalidRPCNode("Type Error");
+      return m_value.child("boolean").text().as_bool();
+    }
+
+    void set(const std::string& value) {
+      if (!m_value.child("string"))
+        m_value.append_child("string");
+      m_value.child("string").text().set(value.c_str());
+    };
+
+    std::string getAsS() const {
+      if (!m_value.child("string"))
+        throw InvalidRPCNode("Type Error");
+      return m_value.child("string").text().as_string();
+    }
+
+    void set(double value) {
+      if (!m_value.child("double"))
+        m_value.append_child("double");
+      m_value.child("double").text().set(value);
+    };
+
+    double getAsD() const {
+      if (!m_value.child("double"))
+        throw InvalidRPCNode("Type Error");
+      return m_value.child("double").text().as_double();
+    }
+
+    inline Struct getStruct();
+    inline Array getArray();
+};
+
+class Array {
+    pugi::xml_node m_array;
+  public:
+    Array(pugi::xml_node array) : m_array(array) {
+      if (!m_array.child("data"))
+        m_array.append_child("data");
+    }
+
+    Value addValue() {
+      auto value = m_array.child("data").append_child("value");
+      return Value(value);
+    }
+
+    Value getValue(int index) const {
+      auto value = m_array.child("data").child("value");
+      while(index && value) {
+        value = value.next_sibling();
+        index--;
+      }
+      if (0==index) {
+        return Value(value);
+      } else {
+        throw InvalidRPCNode("Index error");
+      }
+    }
+
+};
+
+class Member {
+    pugi::xml_node m_member;
+
+  public:
+    Member(pugi::xml_node member) : m_member(member) { }
+
+    Value getValue() const {
+      return Value(m_member.child("value"));
+    };
+};
+
+class Struct {
+    pugi::xml_node m_struct;
+
+  public:
+    Struct(pugi::xml_node _struct) : m_struct(_struct) { }
+
+    Member getMember(const std::string& name) const {
+      for(auto member=m_struct.first_child(); member; member=member.next_sibling()) {
+        std::string member_name = member.child("name").text().get();
+        if (member_name == name) {
+          return Member(member);
+        }
+      }
+      throw InvalidRPCNode("Key Error");
+   }
+
+   Member addMember(const std::string& name) {
+     auto member = m_struct.append_child("member");
+     member.append_child("name").text().set(name.c_str());
+     member.append_child("value");
+     return Member(member);
+   }
+};
+
+class Fault : public Struct {
+  public:
+    Fault(pugi::xml_node fault) : Struct(fault) {};
+
+    int getFaultCode() const {
+      return getMember("faultCode").getValue().getAsI();
+    }
+
+    std::string getFaultString() const {
+      return getMember("faultString").getValue().getAsS();
+    }
+};
+
+Struct Value::getStruct() {
+  if (!m_value.child("struct"))
+    m_value.append_child("struct");
+  return Struct(m_value.child("struct"));
+}
+
+Array Value::getArray() {
+  if (!m_value.child("array"))
+    m_value.append_child("array");
+  return Array(m_value.child("array"));
+}
+
+class Param {
+    pugi::xml_node m_param;
+  public:
+    Param(pugi::xml_node param) : m_param(param) {
+      if (!m_param.child("value"))
+        m_param.append_child("value");
+    };
+
+    Value getValue() const {
+      return Value(m_param.child("value"));
+    };
+};
+
+class Params {
+    pugi::xml_node m_params;
+
+  public:
+    Params(pugi::xml_node params) : m_params(params) {};
+
+    Param addParam() {
+      auto param = m_params.append_child("param");
+      return Param(param);
+    }
+    Param getParam(int index) const {
+      auto param = m_params.child("param");
+      while(index && param) {
+        param = param.next_sibling();
+        index--;
+      }
+      if (0==index) {
+        return Param(param);
+      } else {
+        throw InvalidRPCNode("Index Error");
+      }
+    }
+};
+
+class MethodCall {
+    pugi::xml_document m_doc;
+
+  public:
+    MethodCall(const std::string& methodName, const std::string& secret) {
+      auto mCall = m_doc.append_child("methodCall");
+      mCall.append_child("methodName").text().set(methodName.c_str());
+      mCall.append_child("params");
+      if (!secret.empty()) {
+        getParams().addParam().getValue().set(secret);
+      }
+    }
+
+    Params getParams() const {
+      return Params(m_doc.child("methodCall").child("params"));
+    }
+
+    Value newParamValue() {
+      return getParams().addParam().getValue();
+    }
+
+    std::string toString() const {
+      return nodeToString(m_doc);
+    }
+};
+
+
+class MethodResponse {
+    pugi::xml_document m_doc;
+
+  public:
+    MethodResponse(const std::string& content) {
+      m_doc.load_buffer(content.c_str(), content.size());
+    }
+
+    Params getParams() const {
+      auto params = m_doc.child("methodResponse").child("params");
+      if (!params)
+        throw InvalidRPCNode("No params");
+      return Params(params);
+    }
+
+    Value getParamValue(int index) const {
+      return getParams().getParam(index).getValue();
+    }
+
+    bool isFault() const {
+      return (!!m_doc.child("methodResponse").child("fault"));
+    }
+
+    Fault getFault() const {
+      auto fault = m_doc.child("methodResponse").child("fault");
+      if (!fault)
+        throw InvalidRPCNode("No fault");
+      return Fault(fault.child("value").child("struct"));
+    }
+};
+
+
+
+};
+
+
+#endif // KIWIX_XMLRPC_H_

static/meson.build

@@ -1,27 +1,11 @@
 
-
-ctpp2c = find_program('ctpp2c', required:false)
-
-if ctpp2c.found()
-  search_result_template = custom_target('result_template',
-    input: 'results.tmpl',
-    output: 'results.ct2',
-    command: [intermediate_ctpp2c, ctpp2c, '@INPUT@', '@OUTPUT@']
- )
- resources_list = 'resources_list_ctpp2.txt'
- resources_depends = [search_result_template]
-else
- resources_list = 'resources_list_noctpp2.txt'
- resources_depends = []
-endif
-
 lib_resources = custom_target('resources',
-  input: resources_list,
-  output: ['kiwixlib-resources.cpp', 'kiwixlib-resources.h'],
-  command:[res_compiler,
-           '--cxxfile', '@OUTPUT0@',
-           '--hfile', '@OUTPUT1@',
-           '--source_dir', '@OUTDIR@',
-           '@INPUT@'],
-  depends: resources_depends
+    input: 'resources_list.txt',
+    output: ['kiwixlib-resources.cpp', 'kiwixlib-resources.h'],
+    command:[res_compiler,
+             '--cxxfile', '@OUTPUT0@',
+             '--hfile', '@OUTPUT1@',
+             '--source_dir', '@OUTDIR@',
+             '@INPUT@'],
+    depend_files: files('search_result.tmpl')
 )

static/resources_list.txt

@@ -0,0 +1 @@
+search_result.tmpl

static/resources_list_ctpp2.txt

@@ -1,4 +0,0 @@
-results.ct2
-stopwords/en
-stopwords/he
-stopwords/fra

static/resources_list_noctpp2.txt

@@ -1,3 +0,0 @@
-stopwords/en
-stopwords/he
-stopwords/fra

static/search_result.tmpl

@@ -91,68 +91,67 @@
       }
 
     </style>
-    <title>Search: <TMPL_var searchPattern></title>
+    <title>Search: {{searchPattern}}</title>
   </head>
   <body bgcolor="white">
     <div class="header">
-      <TMPL_if results>
+      {{#hasResult}}
         Results
         <b>
-          <TMPL_var resultStart>-<TMPL_var resultEnd>
+          {{resultStart}}-{{resultEnd}}
         </b> of <b>
-          <TMPL_var count>
+          {{count}}
         </b> for <b>
-          <TMPL_var searchPattern>
+          {{searchPattern}}
         </b>
-      <TMPL_else>
-        No results were found for <b><TMPL_var searchPattern></b>
-      </TMPL_if>
+      {{/hasResult}}
+      {{^hasResult}}
+        No results were found for <b>{{searchPattern}}</b>
+      {{/hasResult}}
     </div>
 
     <div class="results">
       <ul>
-        <TMPL_foreach results as result>
+        {{#results}}
           <li>
-            <a href="<TMPL_var protocolPrefix><TMPL_var result.contentId>/<TMPL_var result.url>">
-              <TMPL_var result.title>
+            <a href="{{protocolPrefix}}{{resultContentId}}/{{url}}">
+              {{title}}
             </a>
-            <cite>
-              <TMPL_if result.snippet>
-                <TMPL_var result.snippet>...
-              </TMPL_if>
-            </cite>
-            <TMPL_if wordCount>
-              <div class="informations"><TMPL_var wordCount> words</div>
-            </TMPL_if>
+            {{#snippet}}
+              <cite>{{>snippet}}...</cite>
+            {{/snippet}}
+            {{#wordCount}}
+              <div class="informations">{{wordCount}} words</div>
+            {{/wordCount}}
           </li>
-        </TMPL_foreach>
+        {{/results}}
       </ul>
     </div>
 
     <div class="footer">
       <ul>
-        <TMPL_if (resultLastPageStart>0)>
+        {{#resultLastPageStart}}
           <li>
-            <a href="<TMPL_var searchProtocolPrefix>pattern=<TMPL_var searchPatternEncoded><TMPL_if contentId>&content=<TMPL_var contentId></TMPL_if>&start=0&end=<TMPL_var resultRange>">
+            <a href="{{searchProtocolPrefix}}pattern={{searchPatternEncoded}}{{#contentId}}&content={{.}}{{/contentId}}&start=0&end={{resultRange}}">
               ◀
             </a>
           </li>
-        </TMPL_if>
-        <TMPL_foreach pages as page>
+        {{/resultLastPageStart}}
+        {{#pages}}
           <li>
-            <a <TMPL_if page.selected>class="selected"</TMPL_if>
-               href="<TMPL_var searchProtocolPrefix>pattern=<TMPL_var searchPatternEncoded><TMPL_if contentId>&content=<TMPL_var contentId></TMPL_if>&start=<TMPL_var page.start>&end=<TMPL_var page.end>">
-              <TMPL_var page.label>
+            <a {{#selected}}class="selected"{{/selected}}
+               href="{{searchProtocolPrefix}}pattern={{searchPatternEncoded}}{{#contentId}}&content={{.}}{{/contentId}}&start={{start}}&end={{end}}">
+              {{label}}
             </a>
           </li>
-        </TMPL_foreach>
-        <TMPL_if (resultLastPageStart>0)>
+        {{/pages}}
+        {{#resultLastPageStart}}
           <li>
-            <a href="<TMPL_var searchProtocolPrefix>pattern=<TMPL_var searchPatternEncoded><TMPL_if contentId>&content=<TMPL_var contentId></TMPL_if>&start=<TMPL_var resultLastPageStart>&end=<TMPL_var (resultLastPageStart+resultRange)>">
+            <a href="{{searchProtocolPrefix}}pattern={{searchPatternEncoded}}{{#contentId}}&content={{.}}{{/contentId}}&start={{resultLastPageStart}}&end={{lastResult}}">
               ▶
             </a>
           </li>
-        </TMPL_if>
+        {{/resultLastPageStart}}
       </ul>
     </div>
   </body>

static/stopwords/en

@@ -1,671 +0,0 @@
-a
-able
-about
-above
-abst
-accordance
-according
-accordingly
-across
-act
-actually
-added
-adj
-adopted
-affected
-affecting
-affects
-after
-afterwards
-again
-against
-ah
-all
-almost
-alone
-along
-already
-also
-although
-always
-am
-among
-amongst
-an
-and
-announce
-another
-any
-anybody
-anyhow
-anymore
-anyone
-anything
-anyway
-anyways
-anywhere
-apparently
-approximately
-are
-aren
-arent
-arise
-around
-as
-aside
-ask
-asking
-at
-auth
-available
-away
-awfully
-b
-back
-be
-became
-because
-become
-becomes
-becoming
-been
-before
-beforehand
-begin
-beginning
-beginnings
-begins
-behind
-being
-believe
-below
-beside
-besides
-between
-beyond
-biol
-both
-brief
-briefly
-but
-by
-c
-ca
-came
-can
-cannot
-can't
-cause
-causes
-certain
-certainly
-co
-com
-come
-comes
-contain
-containing
-contains
-could
-couldnt
-d
-date
-did
-didn't
-different
-do
-does
-doesn't
-doing
-done
-don't
-down
-downwards
-due
-during
-e
-each
-ed
-edu
-effect
-eg
-eight
-eighty
-either
-else
-elsewhere
-end
-ending
-enough
-especially
-et
-et-al
-etc
-even
-ever
-every
-everybody
-everyone
-everything
-everywhere
-ex
-except
-f
-far
-few
-ff
-fifth
-first
-five
-fix
-followed
-following
-follows
-for
-former
-formerly
-forth
-found
-four
-from
-further
-furthermore
-g
-gave
-get
-gets
-getting
-give
-given
-gives
-giving
-go
-goes
-gone
-got
-gotten
-h
-had
-happens
-hardly
-has
-hasn't
-have
-haven't
-having
-he
-hed
-hence
-her
-here
-hereafter
-hereby
-herein
-heres
-hereupon
-hers
-herself
-hes
-hi
-hid
-him
-himself
-his
-hither
-home
-how
-howbeit
-however
-hundred
-i
-id
-ie
-if
-i'll
-im
-immediate
-immediately
-importance
-important
-in
-inc
-indeed
-index
-information
-instead
-into
-invention
-inward
-is
-isn't
-it
-itd
-it'll
-its
-itself
-i've
-j
-just
-k
-keep
-keeps
-kept
-keys
-kg
-km
-know
-known
-knows
-l
-largely
-last
-lately
-later
-latter
-latterly
-least
-less
-lest
-let
-lets
-like
-liked
-likely
-line
-little
-'ll
-look
-looking
-looks
-ltd
-m
-made
-mainly
-make
-makes
-many
-may
-maybe
-me
-mean
-means
-meantime
-meanwhile
-merely
-mg
-might
-million
-miss
-ml
-more
-moreover
-most
-mostly
-mr
-mrs
-much
-mug
-must
-my
-myself
-n
-na
-name
-namely
-nay
-nd
-near
-nearly
-necessarily
-necessary
-need
-needs
-neither
-never
-nevertheless
-new
-next
-nine
-ninety
-no
-nobody
-non
-none
-nonetheless
-noone
-nor
-normally
-nos
-not
-noted
-nothing
-now
-nowhere
-o
-obtain
-obtained
-obviously
-of
-off
-often
-oh
-ok
-okay
-old
-omitted
-on
-once
-one
-ones
-only
-onto
-or
-ord
-other
-others
-otherwise
-ought
-our
-ours
-ourselves
-out
-outside
-over
-overall
-owing
-own
-p
-page
-pages
-part
-particular
-particularly
-past
-per
-perhaps
-placed
-please
-plus
-poorly
-possible
-possibly
-potentially
-pp
-predominantly
-present
-previously
-primarily
-probably
-promptly
-proud
-provides
-put
-q
-que
-quickly
-quite
-qv
-r
-ran
-rather
-rd
-re
-readily
-really
-recent
-recently
-ref
-refs
-regarding
-regardless
-regards
-related
-relatively
-research
-respectively
-resulted
-resulting
-results
-right
-run
-s
-said
-same
-saw
-say
-saying
-says
-sec
-section
-see
-seeing
-seem
-seemed
-seeming
-seems
-seen
-self
-selves
-sent
-seven
-several
-shall
-she
-shed
-she'll
-shes
-should
-shouldn't
-show
-showed
-shown
-showns
-shows
-significant
-significantly
-similar
-similarly
-since
-six
-slightly
-so
-some
-somebody
-somehow
-someone
-somethan
-something
-sometime
-sometimes
-somewhat
-somewhere
-soon
-sorry
-specifically
-specified
-specify
-specifying
-state
-states
-still
-stop
-strongly
-sub
-substantially
-successfully
-such
-sufficiently
-suggest
-sup
-sure
-	t
-take
-taken
-taking
-tell
-tends
-th
-than
-thank
-thanks
-thanx
-that
-that'll
-thats
-that've
-the
-their
-theirs
-them
-themselves
-then
-thence
-there
-thereafter
-thereby
-thered
-therefore
-therein
-there'll
-thereof
-therere
-theres
-thereto
-thereupon
-there've
-these
-they
-theyd
-they'll
-theyre
-they've
-think
-this
-those
-thou
-though
-thoughh
-thousand
-throug
-through
-throughout
-thru
-thus
-til
-tip
-to
-together
-too
-took
-toward
-towards
-tried
-tries
-truly
-try
-trying
-ts
-twice
-two
-u
-un
-under
-unfortunately
-unless
-unlike
-unlikely
-until
-unto
-up
-upon
-ups
-us
-use
-used
-useful
-usefully
-usefulness
-uses
-using
-usually
-v
-value
-various
-'ve
-very
-via
-viz
-vol
-vols
-vs
-w
-want
-wants
-was
-wasn't
-way
-we
-wed
-welcome
-we'll
-went
-were
-weren't
-we've
-what
-whatever
-what'll
-whats
-when
-whence
-whenever
-where
-whereafter
-whereas
-whereby
-wherein
-wheres
-whereupon
-wherever
-whether
-which
-while
-whim
-whither
-who
-whod
-whoever
-whole
-who'll
-whom
-whomever
-whos
-whose
-why
-widely
-willing
-wish
-with
-within
-without
-won't
-words
-world
-would
-wouldn't
-www
-x
-y
-yes
-yet
-you
-youd
-you'll
-your
-youre
-yours
-yourself
-yourselves
-you've
-z
-zero

static/stopwords/fra

@@ -1,124 +0,0 @@
-alors
-au
-aucuns
-aussi
-autre
-avant
-avec
-avoir
-bon
-car
-ce
-cela
-ces
-ceux
-chaque
-ci
-comme
-comment
-dans
-des
-du
-dedans
-dehors
-depuis
-deux
-devrait
-doit
-donc
-dos
-droite
-début
-elle
-elles
-en
-encore
-essai
-est
-et
-eu
-fait
-faites
-fois
-font
-force
-haut
-hors
-ici
-il
-ils
-je
-la
-le
-les
-leur
-là
-ma
-maintenant
-mais
-mes
-mine
-moins
-mon
-mot
-même
-ni
-nommés
-notre
-nous
-nouveaux
-ou
-où
-par
-parce
-parole
-pas
-personnes
-peut
-peu
-pièce
-plupart
-pour
-pourquoi
-quand
-que
-quel
-quelle
-quelles
-quels
-qui
-sa
-sans
-ses
-seulement
-si
-sien
-son
-sont
-sous
-soyez
-sur
-ta
-tandis
-tellement
-tels
-tes
-ton
-tous
-tout
-trop
-très
-tu
-valeur
-voie
-voient
-vont
-votre
-vous
-vu
-ça
-étaient
-état
-étions
-été
-être

static/stopwords/he

@@ -1,87 +0,0 @@
-של
-את
-על
-לא
-כי
-עם
-הוא
-גם
-ב
-זה
-היא
-כל
-יותר
-או
-אבל
-בין
-היה
-אם
-מיליון
-יש
-כך
-אני
-הם
-דולר
-אמר
-עד
-לאחר
-ישראל
-רק
-שקל
-כדי
-מה
-לפני
-אחד
-החברה
-כמו
-זאת
-היום
-אך
-ל
-ה
-כ
-אין
-אתמול
-שלא
-כבר
-עוד
-לו
-זו
-אל
-בן
-אותו
-שני
-בית
-ידי
-כמה
-ביותר
-ולא
-הממשלה
-אחרי
-חברת
-היתה
-שלו
-היו
-נגד
-בכל
-אביב
-ראש
-בישראל
-לי
-שנים
-פי
-בו
-מ
-מאוד
-להיות
-שהוא
-מי
-אלף
-אלא
-אף
-אחר
-הזה
-אחת
-בבית
-אלה
-אנחנו

subprojects/gtest/.gitignore

@@ -0,0 +1,2 @@
+# Ignore CI build directory
+build/

subprojects/gtest/.travis.yml

@@ -0,0 +1,46 @@
+# Build matrix / environment variable are explained on:
+# http://about.travis-ci.org/docs/user/build-configuration/
+# This file can be validated on:
+# http://lint.travis-ci.org/
+
+install:
+# /usr/bin/gcc is 4.6 always, but gcc-X.Y is available.
+- if [ "$CXX" = "g++" ]; then export CXX="g++-4.9" CC="gcc-4.9"; fi
+# /usr/bin/clang is 3.4, lets override with modern one.
+- if [ "$CXX" = "clang++" ] && [ "$TRAVIS_OS_NAME" = "linux" ]; then export CXX="clang++-3.7" CC="clang-3.7"; fi
+- echo ${PATH}
+- echo ${CXX}
+- ${CXX} --version
+- ${CXX} -v
+addons:
+  apt:
+    # List of whitelisted in travis packages for ubuntu-precise can be found here:
+    #   https://github.com/travis-ci/apt-package-whitelist/blob/master/ubuntu-precise
+    # List of whitelisted in travis apt-sources:
+    #   https://github.com/travis-ci/apt-source-whitelist/blob/master/ubuntu.json
+    sources:
+    - ubuntu-toolchain-r-test
+    - llvm-toolchain-precise-3.7
+    packages:
+    - gcc-4.9
+    - g++-4.9
+    - clang-3.7
+    - valgrind
+os:
+  - linux
+  - osx
+language: cpp
+compiler:
+  - gcc
+  - clang
+script: ./travis.sh
+env:
+  matrix:
+    - GTEST_TARGET=googletest SHARED_LIB=OFF STATIC_LIB=ON CMAKE_PKG=OFF BUILD_TYPE=debug   VERBOSE_MAKE=true VERBOSE
+    - GTEST_TARGET=googlemock SHARED_LIB=OFF STATIC_LIB=ON CMAKE_PKG=OFF BUILD_TYPE=debug   VERBOSE_MAKE=true VERBOSE
+    - GTEST_TARGET=googlemock SHARED_LIB=OFF STATIC_LIB=ON CMAKE_PKG=OFF BUILD_TYPE=debug CXX_FLAGS=-std=c++11  VERBOSE_MAKE=true VERBOSE
+#    - GTEST_TARGET=googletest SHARED_LIB=ON  STATIC_LIB=ON CMAKE_PKG=ON  BUILD_TYPE=release VERBOSE_MAKE=false
+#    - GTEST_TARGET=googlemock SHARED_LIB=ON  STATIC_LIB=ON CMAKE_PKG=ON  BUILD_TYPE=release VERBOSE_MAKE=false
+notifications:
+  email: false
+sudo: false

subprojects/gtest/CMakeLists.txt

@@ -0,0 +1,16 @@
+cmake_minimum_required(VERSION 2.6.2)
+
+project( googletest-distribution )
+
+enable_testing()
+
+option(BUILD_GTEST "Builds the googletest subproject" OFF)
+
+#Note that googlemock target already builds googletest
+option(BUILD_GMOCK "Builds the googlemock subproject" ON)
+
+if(BUILD_GMOCK)
+  add_subdirectory( googlemock )
+elseif(BUILD_GTEST)
+  add_subdirectory( googletest )
+endif()

subprojects/gtest/README.md

@@ -0,0 +1,142 @@
+
+# Google Test #
+
+[![Build Status](https://travis-ci.org/google/googletest.svg?branch=master)](https://travis-ci.org/google/googletest)
+[![Build status](https://ci.appveyor.com/api/projects/status/4o38plt0xbo1ubc8/branch/master?svg=true)](https://ci.appveyor.com/project/BillyDonahue/googletest/branch/master)
+
+Welcome to **Google Test**, Google's C++ test framework!
+
+This repository is a merger of the formerly separate GoogleTest and
+GoogleMock projects. These were so closely related that it makes sense to
+maintain and release them together.
+
+Please see the project page above for more information as well as the
+mailing list for questions, discussions, and development.  There is
+also an IRC channel on OFTC (irc.oftc.net) #gtest available.  Please
+join us!
+
+Getting started information for **Google Test** is available in the 
+[Google Test Primer](googletest/docs/Primer.md) documentation.
+
+**Google Mock** is an extension to Google Test for writing and using C++ mock
+classes.  See the separate [Google Mock documentation](googlemock/README.md).
+
+More detailed documentation for googletest (including build instructions) are
+in its interior [googletest/README.md](googletest/README.md) file.
+
+## Features ##
+
+  * An [XUnit](https://en.wikipedia.org/wiki/XUnit) test framework.
+  * Test discovery.
+  * A rich set of assertions.
+  * User-defined assertions.
+  * Death tests.
+  * Fatal and non-fatal failures.
+  * Value-parameterized tests.
+  * Type-parameterized tests.
+  * Various options for running the tests.
+  * XML test report generation.
+
+## Platforms ##
+
+Google test has been used on a variety of platforms:
+
+  * Linux
+  * Mac OS X
+  * Windows
+  * Cygwin
+  * MinGW
+  * Windows Mobile
+  * Symbian
+
+## Who Is Using Google Test? ##
+
+In addition to many internal projects at Google, Google Test is also used by
+the following notable projects:
+
+  * The [Chromium projects](http://www.chromium.org/) (behind the Chrome
+    browser and Chrome OS).
+  * The [LLVM](http://llvm.org/) compiler.
+  * [Protocol Buffers](https://github.com/google/protobuf), Google's data
+    interchange format.
+  * The [OpenCV](http://opencv.org/) computer vision library.
+
+## Related Open Source Projects ##
+
+[Google Test UI](https://github.com/ospector/gtest-gbar) is test runner that runs
+your test binary, allows you to track its progress via a progress bar, and
+displays a list of test failures. Clicking on one shows failure text. Google
+Test UI is written in C#.
+
+[GTest TAP Listener](https://github.com/kinow/gtest-tap-listener) is an event
+listener for Google Test that implements the
+[TAP protocol](https://en.wikipedia.org/wiki/Test_Anything_Protocol) for test
+result output. If your test runner understands TAP, you may find it useful.
+
+## Requirements ##
+
+Google Test is designed to have fairly minimal requirements to build
+and use with your projects, but there are some.  Currently, we support
+Linux, Windows, Mac OS X, and Cygwin.  We will also make our best
+effort to support other platforms (e.g. Solaris, AIX, and z/OS).
+However, since core members of the Google Test project have no access
+to these platforms, Google Test may have outstanding issues there.  If
+you notice any problems on your platform, please notify
+<googletestframework@googlegroups.com>. Patches for fixing them are
+even more welcome!
+
+### Linux Requirements ###
+
+These are the base requirements to build and use Google Test from a source
+package (as described below):
+
+  * GNU-compatible Make or gmake
+  * POSIX-standard shell
+  * POSIX(-2) Regular Expressions (regex.h)
+  * A C++98-standard-compliant compiler
+
+### Windows Requirements ###
+
+  * Microsoft Visual C++ v7.1 or newer
+
+### Cygwin Requirements ###
+
+  * Cygwin v1.5.25-14 or newer
+
+### Mac OS X Requirements ###
+
+  * Mac OS X v10.4 Tiger or newer
+  * Xcode Developer Tools
+
+### Requirements for Contributors ###
+
+We welcome patches.  If you plan to contribute a patch, you need to
+build Google Test and its own tests from a git checkout (described
+below), which has further requirements:
+
+  * [Python](https://www.python.org/) v2.3 or newer (for running some of
+    the tests and re-generating certain source files from templates)
+  * [CMake](https://cmake.org/) v2.6.4 or newer
+
+## Regenerating Source Files ##
+
+Some of Google Test's source files are generated from templates (not
+in the C++ sense) using a script.
+For example, the
+file include/gtest/internal/gtest-type-util.h.pump is used to generate
+gtest-type-util.h in the same directory.
+
+You don't need to worry about regenerating the source files
+unless you need to modify them.  You would then modify the
+corresponding `.pump` files and run the '[pump.py](googletest/scripts/pump.py)'
+generator script.  See the [Pump Manual](googletest/docs/PumpManual.md).
+
+### Contributing Code ###
+
+We welcome patches.  Please read the
+[Developer's Guide](googletest/docs/DevGuide.md)
+for how you can contribute. In particular, make sure you have signed
+the Contributor License Agreement, or we won't be able to accept the
+patch.
+
+Happy testing!

subprojects/gtest/appveyor.yml

@@ -0,0 +1,71 @@
+version: '{build}'
+
+os: Visual Studio 2015
+
+environment:
+  matrix:
+  - Toolset: v140
+  - Toolset: v120
+  - Toolset: v110
+  - Toolset: v100
+
+platform:
+  - Win32
+  - x64
+
+configuration:
+#  - Release
+  - Debug
+
+build:
+  verbosity: minimal
+
+artifacts:
+  - path: '_build/Testing/Temporary/*'
+    name: test_results
+
+before_build:
+- ps: |
+    Write-Output "Configuration: $env:CONFIGURATION"
+    Write-Output "Platform: $env:PLATFORM"
+    $generator = switch ($env:TOOLSET)
+    {
+        "v140" {"Visual Studio 14 2015"}
+        "v120" {"Visual Studio 12 2013"}
+        "v110" {"Visual Studio 11 2012"}
+        "v100" {"Visual Studio 10 2010"}
+    }
+    if ($env:PLATFORM -eq "x64")
+    {
+        $generator = "$generator Win64"
+    }
+
+build_script:
+- ps: |
+    if (($env:TOOLSET -eq "v100") -and ($env:PLATFORM -eq "x64"))
+    {
+        return
+    }
+    md _build -Force | Out-Null
+    cd _build
+
+    & cmake -G "$generator" -DCMAKE_CONFIGURATION_TYPES="Debug;Release" -Dgtest_build_tests=ON -Dgtest_build_samples=ON -Dgmock_build_tests=ON ..
+    if ($LastExitCode -ne 0) {
+        throw "Exec: $ErrorMessage"
+    }
+    & cmake --build . --config $env:CONFIGURATION
+    if ($LastExitCode -ne 0) {
+        throw "Exec: $ErrorMessage"
+    }
+
+test_script:
+- ps: |
+    if (($env:Toolset -eq "v100") -and ($env:PLATFORM -eq "x64"))
+    {
+        return
+    }
+
+    & ctest -C $env:CONFIGURATION --output-on-failure
+    if ($LastExitCode -ne 0) {
+        throw "Exec: $ErrorMessage"
+    }

subprojects/gtest/googlemock/CHANGES

@@ -0,0 +1,126 @@
+Changes for 1.7.0:
+
+* All new improvements in Google Test 1.7.0.
+* New feature: matchers DoubleNear(), FloatNear(),
+  NanSensitiveDoubleNear(), NanSensitiveFloatNear(),
+  UnorderedElementsAre(), UnorderedElementsAreArray(), WhenSorted(),
+  WhenSortedBy(), IsEmpty(), and SizeIs().
+* Improvement: Google Mock can now be built as a DLL.
+* Improvement: when compiled by a C++11 compiler, matchers AllOf()
+  and AnyOf() can accept an arbitrary number of matchers.
+* Improvement: when compiled by a C++11 compiler, matchers
+  ElementsAreArray() can accept an initializer list.
+* Improvement: when exceptions are enabled, a mock method with no
+  default action now throws instead crashing the test.
+* Improvement: added class testing::StringMatchResultListener to aid
+  definition of composite matchers.
+* Improvement: function return types used in MOCK_METHOD*() macros can
+  now contain unprotected commas.
+* Improvement (potentially breaking): EXPECT_THAT() and ASSERT_THAT()
+  are now more strict in ensuring that the value type and the matcher
+  type are compatible, catching potential bugs in tests.
+* Improvement: Pointee() now works on an optional<T>.
+* Improvement: the ElementsAreArray() matcher can now take a vector or
+  iterator range as input, and makes a copy of its input elements
+  before the conversion to a Matcher.
+* Improvement: the Google Mock Generator can now generate mocks for
+  some class templates.
+* Bug fix: mock object destruction triggerred by another mock object's
+  destruction no longer hangs.
+* Improvement: Google Mock Doctor works better with newer Clang and
+  GCC now.
+* Compatibility fixes.
+* Bug/warning fixes.
+
+Changes for 1.6.0:
+
+* Compilation is much faster and uses much less memory, especially
+  when the constructor and destructor of a mock class are moved out of
+  the class body.
+* New matchers: Pointwise(), Each().
+* New actions: ReturnPointee() and ReturnRefOfCopy().
+* CMake support.
+* Project files for Visual Studio 2010.
+* AllOf() and AnyOf() can handle up-to 10 arguments now.
+* Google Mock doctor understands Clang error messages now.
+* SetArgPointee<> now accepts string literals.
+* gmock_gen.py handles storage specifier macros and template return
+  types now.
+* Compatibility fixes.
+* Bug fixes and implementation clean-ups.
+* Potentially incompatible changes: disables the harmful 'make install'
+  command in autotools.
+
+Potentially breaking changes:
+
+* The description string for MATCHER*() changes from Python-style
+  interpolation to an ordinary C++ string expression.
+* SetArgumentPointee is deprecated in favor of SetArgPointee.
+* Some non-essential project files for Visual Studio 2005 are removed.
+
+Changes for 1.5.0:
+
+ * New feature: Google Mock can be safely used in multi-threaded tests
+   on platforms having pthreads.
+ * New feature: function for printing a value of arbitrary type.
+ * New feature: function ExplainMatchResult() for easy definition of
+   composite matchers.
+ * The new matcher API lets user-defined matchers generate custom
+   explanations more directly and efficiently.
+ * Better failure messages all around.
+ * NotNull() and IsNull() now work with smart pointers.
+ * Field() and Property() now work when the matcher argument is a pointer
+   passed by reference.
+ * Regular expression matchers on all platforms.
+ * Added GCC 4.0 support for Google Mock Doctor.
+ * Added gmock_all_test.cc for compiling most Google Mock tests
+   in a single file.
+ * Significantly cleaned up compiler warnings.
+ * Bug fixes, better test coverage, and implementation clean-ups.
+
+ Potentially breaking changes:
+
+ * Custom matchers defined using MatcherInterface or MakePolymorphicMatcher()
+   need to be updated after upgrading to Google Mock 1.5.0; matchers defined
+   using MATCHER or MATCHER_P* aren't affected.
+ * Dropped support for 'make install'.
+
+Changes for 1.4.0 (we skipped 1.2.* and 1.3.* to match the version of
+Google Test):
+
+ * Works in more environments: Symbian and minGW, Visual C++ 7.1.
+ * Lighter weight: comes with our own implementation of TR1 tuple (no
+   more dependency on Boost!).
+ * New feature: --gmock_catch_leaked_mocks for detecting leaked mocks.
+ * New feature: ACTION_TEMPLATE for defining templatized actions.
+ * New feature: the .After() clause for specifying expectation order.
+ * New feature: the .With() clause for for specifying inter-argument
+   constraints.
+ * New feature: actions ReturnArg<k>(), ReturnNew<T>(...), and
+   DeleteArg<k>().
+ * New feature: matchers Key(), Pair(), Args<...>(), AllArgs(), IsNull(),
+   and Contains().
+ * New feature: utility class MockFunction<F>, useful for checkpoints, etc.
+ * New feature: functions Value(x, m) and SafeMatcherCast<T>(m).
+ * New feature: copying a mock object is rejected at compile time.
+ * New feature: a script for fusing all Google Mock and Google Test
+   source files for easy deployment.
+ * Improved the Google Mock doctor to diagnose more diseases.
+ * Improved the Google Mock generator script.
+ * Compatibility fixes for Mac OS X and gcc.
+ * Bug fixes and implementation clean-ups.
+
+Changes for 1.1.0:
+
+ * New feature: ability to use Google Mock with any testing framework.
+ * New feature: macros for easily defining new matchers
+ * New feature: macros for easily defining new actions.
+ * New feature: more container matchers.
+ * New feature: actions for accessing function arguments and throwing
+   exceptions.
+ * Improved the Google Mock doctor script for diagnosing compiler errors.
+ * Bug fixes and implementation clean-ups.
+
+Changes for 1.0.0:
+
+ * Initial Open Source release of Google Mock

subprojects/gtest/googlemock/CMakeLists.txt

@@ -0,0 +1,202 @@
+########################################################################
+# CMake build script for Google Mock.
+#
+# To run the tests for Google Mock itself on Linux, use 'make test' or
+# ctest.  You can select which tests to run using 'ctest -R regex'.
+# For more options, run 'ctest --help'.
+
+# BUILD_SHARED_LIBS is a standard CMake variable, but we declare it here to
+# make it prominent in the GUI.
+option(BUILD_SHARED_LIBS "Build shared libraries (DLLs)." OFF)
+
+option(gmock_build_tests "Build all of Google Mock's own tests." OFF)
+
+# A directory to find Google Test sources.
+if (EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/gtest/CMakeLists.txt")
+  set(gtest_dir gtest)
+else()
+  set(gtest_dir ../googletest)
+endif()
+
+# Defines pre_project_set_up_hermetic_build() and set_up_hermetic_build().
+include("${gtest_dir}/cmake/hermetic_build.cmake" OPTIONAL)
+
+if (COMMAND pre_project_set_up_hermetic_build)
+  # Google Test also calls hermetic setup functions from add_subdirectory,
+  # although its changes will not affect things at the current scope.
+  pre_project_set_up_hermetic_build()
+endif()
+
+########################################################################
+#
+# Project-wide settings
+
+# Name of the project.
+#
+# CMake files in this project can refer to the root source directory
+# as ${gmock_SOURCE_DIR} and to the root binary directory as
+# ${gmock_BINARY_DIR}.
+# Language "C" is required for find_package(Threads).
+project(gmock CXX C)
+cmake_minimum_required(VERSION 2.6.2)
+
+if (COMMAND set_up_hermetic_build)
+  set_up_hermetic_build()
+endif()
+
+# Instructs CMake to process Google Test's CMakeLists.txt and add its
+# targets to the current scope.  We are placing Google Test's binary
+# directory in a subdirectory of our own as VC compilation may break
+# if they are the same (the default).
+add_subdirectory("${gtest_dir}" "${gmock_BINARY_DIR}/gtest")
+
+# Although Google Test's CMakeLists.txt calls this function, the
+# changes there don't affect the current scope.  Therefore we have to
+# call it again here.
+config_compiler_and_linker()  # from ${gtest_dir}/cmake/internal_utils.cmake
+
+# Adds Google Mock's and Google Test's header directories to the search path.
+include_directories("${gmock_SOURCE_DIR}/include"
+                    "${gmock_SOURCE_DIR}"
+                    "${gtest_SOURCE_DIR}/include"
+                    # This directory is needed to build directly from Google
+                    # Test sources.
+                    "${gtest_SOURCE_DIR}")
+
+# Summary of tuple support for Microsoft Visual Studio:
+# Compiler    version(MS)  version(cmake)  Support
+# ----------  -----------  --------------  -----------------------------
+# <= VS 2010  <= 10        <= 1600         Use Google Tests's own tuple.
+# VS 2012     11           1700            std::tr1::tuple + _VARIADIC_MAX=10
+# VS 2013     12           1800            std::tr1::tuple
+if (MSVC AND MSVC_VERSION EQUAL 1700)
+  add_definitions(/D _VARIADIC_MAX=10)
+endif()
+
+########################################################################
+#
+# Defines the gmock & gmock_main libraries.  User tests should link
+# with one of them.
+
+# Google Mock libraries.  We build them using more strict warnings than what
+# are used for other targets, to ensure that Google Mock can be compiled by
+# a user aggressive about warnings.
+cxx_library(gmock
+            "${cxx_strict}"
+            "${gtest_dir}/src/gtest-all.cc"
+            src/gmock-all.cc)
+
+cxx_library(gmock_main
+            "${cxx_strict}"
+            "${gtest_dir}/src/gtest-all.cc"
+            src/gmock-all.cc
+            src/gmock_main.cc)
+
+# If the CMake version supports it, attach header directory information
+# to the targets for when we are part of a parent build (ie being pulled
+# in via add_subdirectory() rather than being a standalone build).
+if (DEFINED CMAKE_VERSION AND NOT "${CMAKE_VERSION}" VERSION_LESS "2.8.11")
+  target_include_directories(gmock      INTERFACE "${gmock_SOURCE_DIR}/include")
+  target_include_directories(gmock_main INTERFACE "${gmock_SOURCE_DIR}/include")
+endif()
+
+########################################################################
+#
+# Install rules
+install(TARGETS gmock gmock_main
+  DESTINATION lib)
+install(DIRECTORY ${gmock_SOURCE_DIR}/include/gmock
+  DESTINATION include)
+
+########################################################################
+#
+# Google Mock's own tests.
+#
+# You can skip this section if you aren't interested in testing
+# Google Mock itself.
+#
+# The tests are not built by default.  To build them, set the
+# gmock_build_tests option to ON.  You can do it by running ccmake
+# or specifying the -Dgmock_build_tests=ON flag when running cmake.
+
+if (gmock_build_tests)
+  # This must be set in the root directory for the tests to be run by
+  # 'make test' or ctest.
+  enable_testing()
+
+  ############################################################
+  # C++ tests built with standard compiler flags.
+
+  cxx_test(gmock-actions_test gmock_main)
+  cxx_test(gmock-cardinalities_test gmock_main)
+  cxx_test(gmock_ex_test gmock_main)
+  cxx_test(gmock-generated-actions_test gmock_main)
+  cxx_test(gmock-generated-function-mockers_test gmock_main)
+  cxx_test(gmock-generated-internal-utils_test gmock_main)
+  cxx_test(gmock-generated-matchers_test gmock_main)
+  cxx_test(gmock-internal-utils_test gmock_main)
+  cxx_test(gmock-matchers_test gmock_main)
+  cxx_test(gmock-more-actions_test gmock_main)
+  cxx_test(gmock-nice-strict_test gmock_main)
+  cxx_test(gmock-port_test gmock_main)
+  cxx_test(gmock-spec-builders_test gmock_main)
+  cxx_test(gmock_link_test gmock_main test/gmock_link2_test.cc)
+  cxx_test(gmock_test gmock_main)
+
+  if (CMAKE_USE_PTHREADS_INIT)
+    cxx_test(gmock_stress_test gmock)
+  endif()
+
+  # gmock_all_test is commented to save time building and running tests.
+  # Uncomment if necessary.
+  # cxx_test(gmock_all_test gmock_main)
+
+  ############################################################
+  # C++ tests built with non-standard compiler flags.
+
+  cxx_library(gmock_main_no_exception "${cxx_no_exception}"
+    "${gtest_dir}/src/gtest-all.cc" src/gmock-all.cc src/gmock_main.cc)
+
+  cxx_library(gmock_main_no_rtti "${cxx_no_rtti}"
+    "${gtest_dir}/src/gtest-all.cc" src/gmock-all.cc src/gmock_main.cc)
+
+  if (NOT MSVC OR MSVC_VERSION LESS 1600)  # 1600 is Visual Studio 2010.
+    # Visual Studio 2010, 2012, and 2013 define symbols in std::tr1 that
+    # conflict with our own definitions. Therefore using our own tuple does not
+    # work on those compilers.
+    cxx_library(gmock_main_use_own_tuple "${cxx_use_own_tuple}"
+      "${gtest_dir}/src/gtest-all.cc" src/gmock-all.cc src/gmock_main.cc)
+
+    cxx_test_with_flags(gmock_use_own_tuple_test "${cxx_use_own_tuple}"
+      gmock_main_use_own_tuple test/gmock-spec-builders_test.cc)
+  endif()
+
+  cxx_test_with_flags(gmock-more-actions_no_exception_test "${cxx_no_exception}"
+    gmock_main_no_exception test/gmock-more-actions_test.cc)
+
+  cxx_test_with_flags(gmock_no_rtti_test "${cxx_no_rtti}"
+    gmock_main_no_rtti test/gmock-spec-builders_test.cc)
+
+  cxx_shared_library(shared_gmock_main "${cxx_default}"
+    "${gtest_dir}/src/gtest-all.cc" src/gmock-all.cc src/gmock_main.cc)
+
+  # Tests that a binary can be built with Google Mock as a shared library.  On
+  # some system configurations, it may not possible to run the binary without
+  # knowing more details about the system configurations. We do not try to run
+  # this binary. To get a more robust shared library coverage, configure with
+  # -DBUILD_SHARED_LIBS=ON.
+  cxx_executable_with_flags(shared_gmock_test_ "${cxx_default}"
+    shared_gmock_main test/gmock-spec-builders_test.cc)
+  set_target_properties(shared_gmock_test_
+    PROPERTIES
+    COMPILE_DEFINITIONS "GTEST_LINKED_AS_SHARED_LIBRARY=1")
+
+  ############################################################
+  # Python tests.
+
+  cxx_executable(gmock_leak_test_ test gmock_main)
+  py_test(gmock_leak_test)
+
+  cxx_executable(gmock_output_test_ test gmock)
+  py_test(gmock_output_test)
+endif()

subprojects/gtest/googlemock/CONTRIBUTORS

@@ -0,0 +1,40 @@
+# This file contains a list of people who've made non-trivial
+# contribution to the Google C++ Mocking Framework project.  People
+# who commit code to the project are encouraged to add their names
+# here.  Please keep the list sorted by first names.
+
+Benoit Sigoure <tsuna@google.com>
+Bogdan Piloca <boo@google.com>
+Chandler Carruth <chandlerc@google.com>
+Dave MacLachlan <dmaclach@gmail.com>
+David Anderson <danderson@google.com>
+Dean Sturtevant
+Gene Volovich <gv@cite.com>
+Hal Burch <gmock@hburch.com>
+Jeffrey Yasskin <jyasskin@google.com>
+Jim Keller <jimkeller@google.com>
+Joe Walnes <joe@truemesh.com>
+Jon Wray <jwray@google.com>
+Keir Mierle <mierle@gmail.com>
+Keith Ray <keith.ray@gmail.com>
+Kostya Serebryany <kcc@google.com>
+Lev Makhlis
+Manuel Klimek <klimek@google.com>
+Mario Tanev <radix@google.com>
+Mark Paskin
+Markus Heule <markus.heule@gmail.com>
+Matthew Simmons <simmonmt@acm.org>
+Mike Bland <mbland@google.com>
+Neal Norwitz <nnorwitz@gmail.com>
+Nermin Ozkiranartli <nermin@google.com>
+Owen Carlsen <ocarlsen@google.com>
+Paneendra Ba <paneendra@google.com>
+Paul Menage <menage@google.com>
+Piotr Kaminski <piotrk@google.com>
+Russ Rufer <russ@pentad.com>
+Sverre Sundsdal <sundsdal@gmail.com>
+Takeshi Yoshino <tyoshino@google.com>
+Vadim Berman <vadimb@google.com>
+Vlad Losev <vladl@google.com>
+Wolfgang Klier <wklier@google.com>
+Zhanyong Wan <wan@google.com>

subprojects/gtest/googlemock/LICENSE

@@ -0,0 +1,28 @@
+Copyright 2008, Google Inc.
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are
+met:
+
+    * Redistributions of source code must retain the above copyright
+notice, this list of conditions and the following disclaimer.
+    * Redistributions in binary form must reproduce the above
+copyright notice, this list of conditions and the following disclaimer
+in the documentation and/or other materials provided with the
+distribution.
+    * Neither the name of Google Inc. nor the names of its
+contributors may be used to endorse or promote products derived from
+this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
+"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
+LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
+A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
+OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
+SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
+LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
+DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
+THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
+(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.