Merge pull request #243 from kiwix/new_version

New version 5.2.0

.gitignore

@@ -0,0 +1 @@
+.idea/

.travis.yml

@@ -1,6 +1,6 @@
 language: cpp
-dist: trusty
-sudo: false
+dist: xenial
+sudo: true
 cache: ccache
 before_install:
 - PATH=$PATH:$HOME/bin

ChangeLog

@@ -1,3 +1,43 @@
+kiwix-lib 5.2.0
+===============
+
+ * kiwix-serve integration (as a seperated process).
+ * Fix crash in the suggestion search.
+ * Better API to filter the library books.
+ * New kiwix-lib application for android. (.aar)
+ * Use ReLinker to link with libkiwix.so in android.
+ * Correctly set the verbosity of zim search.
+
+kiwix-lib 5.1.0
+===============
+
+ * Add function to pause, resume and stop downloads.
+ * Add zim's tags in the opds stream.
+ * Addapt to new libzim 5.0.0 API.
+
+kiwix-lib 5.0.0
+===============
+
+ * Remove error message when trying to open an wrong zim file.
+ * Rewrite `makeTmpDirectory` to not use uuid functions on windows.
+ * [API break] Remove `getNetworkInterfaces` and `getBestPublicIp`.
+ * Remove rpath
+ * Detect infinite (and too long) redirect loops.
+
+kiwix-lib 4.1.0
+===============
+
+ * Allow the library to be filtered by tags.
+ * Fix language mapping.
+ * Update README about mustache dependency.
+
+kiwix-lib 4.0.1
+===============
+
+ * Fix "maybe uninitialize variable" issue.
+ * Ensure path are stored correctly (absolute path) in the library.
+ * [CI] Use the new deps archive xz
+
 kiwix-lib 4.0.0
 ===============
 

README.md

@@ -27,14 +27,13 @@ The Kiwix library relies on many third parts software libraries. They
 are prerequisites to the Kiwix library compilation. Following
 libraries need to be available:
 
-* ICU ................................... http://site.icu-project.org/
-(package libicu-dev on Ubuntu)
-* ZIM ........................................ http://www.openzim.org/
-(package libzim-dev on Ubuntu)
-* Pugixml ........................................ http://pugixml.org/
-(package libpugixml-dev on Ubuntu)
-* libaria2 .................................. https://aria2.github.io/
-(no package on Ubuntu)
+* [ICU](https://site.icu-project.org/) (package `libicu-dev` on Ubuntu)
+* [ZIM](https://openzim.org/) (package `libzim-dev` on Ubuntu)
+* [Pugixml](https://pugixml.org/) (package `libpugixml-dev` on Ubuntu)
+* [Aria2](https://aria2.github.io/) (package `aria2` on Ubuntu)
+* [Mustache](https://github.com/kainjow/Mustache) (Just copy the
+header `mustache.hpp` somewhere it can be found by the compiler and/or
+set CPPFLAGS with correct `-I` option)
 
 These dependencies may or may not be packaged by your operating
 system. They may also be packaged but only in an older version. The
@@ -43,13 +42,13 @@ In the worse case, you will have to download and compile bleeding edge
 version by hand.
 
 If you want to install these dependencies locally, then use the
-kiwix-lib directory as install prefix.
+`kiwix-lib` directory as install prefix.
 
 Environment
 -------------
 
-The Kiwix library builds using [Meson](http://mesonbuild.com/) version
-0.39 or higher. Meson relies itself on Ninja, pkg-config and few other
+The Kiwix library builds using [Meson](https://mesonbuild.com/) version
+0.43 or higher. Meson relies itself on Ninja, pkg-config and few other
 compilation tools.
 
 Install first the few common compilation tools:
@@ -58,14 +57,16 @@ Install first the few common compilation tools:
 * Pkg-config
 
 These tools should be packaged if you use a cutting edge operating
-system. If not, have a look to the "Troubleshooting" section.
+system. If not, have a look to the [Troubleshooting](#Troubleshooting)
+section.
 
 Compilation
 -----------
 
 Once all dependencies are installed, you can compile the Kiwix library
 with:
-```
+
+```bash
 meson . build
 ninja -C build
 ```
@@ -83,31 +84,32 @@ Installation
 If you want to install the Kiwix library and the headers you just have
 compiled on your system, here we go:
 
-```
+```bash
 ninja -C build install
 ```
 
-You might need to run the command as root (or using 'sudo'), depending
+You might need to run the command as root (or using `sudo`), depending
 where you want to install the libraries. After the installation
-succeeded, you may need to run ldconfig (as root).
+succeeded, you may need to run `ldconfig` (as root).
 
 Uninstallation
 ------------
 
 If you want to uninstall the Kiwix library:
 
-```
+```bash
 ninja -C build uninstall
 ```
 
 Like for the installation, you might need to run the command as root
-(or using 'sudo').
+(or using `sudo`).
 
 Troubleshooting
 ---------------
 
 If you need to install Meson "manually":
-```
+
+```bash
 virtualenv -p python3 ./ # Create virtualenv
 source bin/activate      # Activate the virtualenv
 pip3 install meson       # Install Meson
@@ -115,7 +117,8 @@ hash -r                  # Refresh bash paths
 ```
 
 If you need to install Ninja "manually":
-```
+
+```bash
 git clone git://github.com/ninja-build/ninja.git
 cd ninja
 git checkout release

android-kiwix-lib-publisher/.gitignore

@@ -0,0 +1,13 @@
+*.iml
+.gradle
+/local.properties
+/.idea/caches
+/.idea/libraries
+/.idea/modules.xml
+/.idea/workspace.xml
+/.idea/navEditor.xml
+/.idea/assetWizardSettings.xml
+.DS_Store
+/build
+/captures
+.externalNativeBuild

android-kiwix-lib-publisher/build.gradle

@@ -0,0 +1,25 @@
+// Top-level build file where you can add configuration options common to all sub-projects/modules.
+
+buildscript {
+    repositories {
+        google()
+        jcenter()
+
+    }
+    dependencies {
+        classpath 'com.android.tools.build:gradle:3.4.1'
+        // NOTE: Do not place your application dependencies here; they belong
+        // in the individual module build.gradle files
+    }
+}
+
+allprojects {
+    repositories {
+        google()
+        jcenter()
+    }
+}
+
+task clean(type: Delete) {
+    delete rootProject.buildDir
+}

android-kiwix-lib-publisher/gradle.properties

@@ -0,0 +1,15 @@
+# Project-wide Gradle settings.
+# IDE (e.g. Android Studio) users:
+# Gradle settings configured through the IDE *will override*
+# any settings specified in this file.
+# For more details on how to configure your build environment visit
+# http://www.gradle.org/docs/current/userguide/build_environment.html
+# Specifies the JVM arguments used for the daemon process.
+# The setting is particularly useful for tweaking memory settings.
+org.gradle.jvmargs=-Xmx1536m
+# When configured, Gradle will run in incubating parallel mode.
+# This option should only be used with decoupled projects. More details, visit
+# http://www.gradle.org/docs/current/userguide/multi_project_builds.html#sec:decoupled_projects
+# org.gradle.parallel=true
+# Kotlin code style for this project: "official" or "obsolete":
+kotlin.code.style=official

android-kiwix-lib-publisher/gradle/wrapper/gradle-wrapper.properties

@@ -0,0 +1,6 @@
+#Wed Jun 19 15:28:39 BST 2019
+distributionBase=GRADLE_USER_HOME
+distributionPath=wrapper/dists
+zipStoreBase=GRADLE_USER_HOME
+zipStorePath=wrapper/dists
+distributionUrl=https\://services.gradle.org/distributions/gradle-5.1.1-all.zip

android-kiwix-lib-publisher/gradlew

@@ -0,0 +1,172 @@
+#!/usr/bin/env sh
+
+##############################################################################
+##
+##  Gradle start up script for UN*X
+##
+##############################################################################
+
+# Attempt to set APP_HOME
+# Resolve links: $0 may be a link
+PRG="$0"
+# Need this for relative symlinks.
+while [ -h "$PRG" ] ; do
+    ls=`ls -ld "$PRG"`
+    link=`expr "$ls" : '.*-> \(.*\)$'`
+    if expr "$link" : '/.*' > /dev/null; then
+        PRG="$link"
+    else
+        PRG=`dirname "$PRG"`"/$link"
+    fi
+done
+SAVED="`pwd`"
+cd "`dirname \"$PRG\"`/" >/dev/null
+APP_HOME="`pwd -P`"
+cd "$SAVED" >/dev/null
+
+APP_NAME="Gradle"
+APP_BASE_NAME=`basename "$0"`
+
+# Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
+DEFAULT_JVM_OPTS=""
+
+# Use the maximum available, or set MAX_FD != -1 to use that value.
+MAX_FD="maximum"
+
+warn () {
+    echo "$*"
+}
+
+die () {
+    echo
+    echo "$*"
+    echo
+    exit 1
+}
+
+# OS specific support (must be 'true' or 'false').
+cygwin=false
+msys=false
+darwin=false
+nonstop=false
+case "`uname`" in
+  CYGWIN* )
+    cygwin=true
+    ;;
+  Darwin* )
+    darwin=true
+    ;;
+  MINGW* )
+    msys=true
+    ;;
+  NONSTOP* )
+    nonstop=true
+    ;;
+esac
+
+CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar
+
+# Determine the Java command to use to start the JVM.
+if [ -n "$JAVA_HOME" ] ; then
+    if [ -x "$JAVA_HOME/jre/sh/java" ] ; then
+        # IBM's JDK on AIX uses strange locations for the executables
+        JAVACMD="$JAVA_HOME/jre/sh/java"
+    else
+        JAVACMD="$JAVA_HOME/bin/java"
+    fi
+    if [ ! -x "$JAVACMD" ] ; then
+        die "ERROR: JAVA_HOME is set to an invalid directory: $JAVA_HOME
+
+Please set the JAVA_HOME variable in your environment to match the
+location of your Java installation."
+    fi
+else
+    JAVACMD="java"
+    which java >/dev/null 2>&1 || die "ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
+
+Please set the JAVA_HOME variable in your environment to match the
+location of your Java installation."
+fi
+
+# Increase the maximum file descriptors if we can.
+if [ "$cygwin" = "false" -a "$darwin" = "false" -a "$nonstop" = "false" ] ; then
+    MAX_FD_LIMIT=`ulimit -H -n`
+    if [ $? -eq 0 ] ; then
+        if [ "$MAX_FD" = "maximum" -o "$MAX_FD" = "max" ] ; then
+            MAX_FD="$MAX_FD_LIMIT"
+        fi
+        ulimit -n $MAX_FD
+        if [ $? -ne 0 ] ; then
+            warn "Could not set maximum file descriptor limit: $MAX_FD"
+        fi
+    else
+        warn "Could not query maximum file descriptor limit: $MAX_FD_LIMIT"
+    fi
+fi
+
+# For Darwin, add options to specify how the application appears in the dock
+if $darwin; then
+    GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\""
+fi
+
+# For Cygwin, switch paths to Windows format before running java
+if $cygwin ; then
+    APP_HOME=`cygpath --path --mixed "$APP_HOME"`
+    CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
+    JAVACMD=`cygpath --unix "$JAVACMD"`
+
+    # We build the pattern for arguments to be converted via cygpath
+    ROOTDIRSRAW=`find -L / -maxdepth 1 -mindepth 1 -type d 2>/dev/null`
+    SEP=""
+    for dir in $ROOTDIRSRAW ; do
+        ROOTDIRS="$ROOTDIRS$SEP$dir"
+        SEP="|"
+    done
+    OURCYGPATTERN="(^($ROOTDIRS))"
+    # Add a user-defined pattern to the cygpath arguments
+    if [ "$GRADLE_CYGPATTERN" != "" ] ; then
+        OURCYGPATTERN="$OURCYGPATTERN|($GRADLE_CYGPATTERN)"
+    fi
+    # Now convert the arguments - kludge to limit ourselves to /bin/sh
+    i=0
+    for arg in "$@" ; do
+        CHECK=`echo "$arg"|egrep -c "$OURCYGPATTERN" -`
+        CHECK2=`echo "$arg"|egrep -c "^-"`                                 ### Determine if an option
+
+        if [ $CHECK -ne 0 ] && [ $CHECK2 -eq 0 ] ; then                    ### Added a condition
+            eval `echo args$i`=`cygpath --path --ignore --mixed "$arg"`
+        else
+            eval `echo args$i`="\"$arg\""
+        fi
+        i=$((i+1))
+    done
+    case $i in
+        (0) set -- ;;
+        (1) set -- "$args0" ;;
+        (2) set -- "$args0" "$args1" ;;
+        (3) set -- "$args0" "$args1" "$args2" ;;
+        (4) set -- "$args0" "$args1" "$args2" "$args3" ;;
+        (5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
+        (6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
+        (7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
+        (8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
+        (9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
+    esac
+fi
+
+# Escape application args
+save () {
+    for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
+    echo " "
+}
+APP_ARGS=$(save "$@")
+
+# Collect all arguments for the java command, following the shell quoting and substitution rules
+eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"
+
+# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong
+if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then
+  cd "$(dirname "$0")"
+fi
+
+exec "$JAVACMD" "$@"

android-kiwix-lib-publisher/gradlew.bat

@@ -0,0 +1,84 @@
+@if "%DEBUG%" == "" @echo off
+@rem ##########################################################################
+@rem
+@rem  Gradle startup script for Windows
+@rem
+@rem ##########################################################################
+
+@rem Set local scope for the variables with windows NT shell
+if "%OS%"=="Windows_NT" setlocal
+
+set DIRNAME=%~dp0
+if "%DIRNAME%" == "" set DIRNAME=.
+set APP_BASE_NAME=%~n0
+set APP_HOME=%DIRNAME%
+
+@rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
+set DEFAULT_JVM_OPTS=
+
+@rem Find java.exe
+if defined JAVA_HOME goto findJavaFromJavaHome
+
+set JAVA_EXE=java.exe
+%JAVA_EXE% -version >NUL 2>&1
+if "%ERRORLEVEL%" == "0" goto init
+
+echo.
+echo ERROR: JAVA_HOME is not set and no 'java' command could be found in your PATH.
+echo.
+echo Please set the JAVA_HOME variable in your environment to match the
+echo location of your Java installation.
+
+goto fail
+
+:findJavaFromJavaHome
+set JAVA_HOME=%JAVA_HOME:"=%
+set JAVA_EXE=%JAVA_HOME%/bin/java.exe
+
+if exist "%JAVA_EXE%" goto init
+
+echo.
+echo ERROR: JAVA_HOME is set to an invalid directory: %JAVA_HOME%
+echo.
+echo Please set the JAVA_HOME variable in your environment to match the
+echo location of your Java installation.
+
+goto fail
+
+:init
+@rem Get command-line arguments, handling Windows variants
+
+if not "%OS%" == "Windows_NT" goto win9xME_args
+
+:win9xME_args
+@rem Slurp the command line arguments.
+set CMD_LINE_ARGS=
+set _SKIP=2
+
+:win9xME_args_slurp
+if "x%~1" == "x" goto execute
+
+set CMD_LINE_ARGS=%*
+
+:execute
+@rem Setup the command line
+
+set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar
+
+@rem Execute Gradle
+"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %CMD_LINE_ARGS%
+
+:end
+@rem End local scope for the variables with windows NT shell
+if "%ERRORLEVEL%"=="0" goto mainEnd
+
+:fail
+rem Set variable GRADLE_EXIT_CONSOLE if you need the _script_ return code instead of
+rem the _cmd.exe /c_ return code!
+if  not "" == "%GRADLE_EXIT_CONSOLE%" exit 1
+exit /b 1
+
+:mainEnd
+if "%OS%"=="Windows_NT" endlocal
+
+:omega

android-kiwix-lib-publisher/kiwixLibAndroid/.gitignore

@@ -0,0 +1 @@
+/build

android-kiwix-lib-publisher/kiwixLibAndroid/build.gradle

@@ -0,0 +1,21 @@
+apply plugin: 'com.android.library'
+
+android {
+    compileSdkVersion 28
+    defaultConfig {
+        minSdkVersion 15
+        targetSdkVersion 28
+        versionCode 1
+        versionName "1.0"
+    }
+    buildTypes {
+        release {
+            minifyEnabled false
+            proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
+        }
+    }
+}
+
+dependencies {
+    implementation 'com.getkeepsafe.relinker:relinker:1.3.1'
+}

android-kiwix-lib-publisher/kiwixLibAndroid/proguard-rules.pro

@@ -0,0 +1,21 @@
+# Add project specific ProGuard rules here.
+# You can control the set of applied configuration files using the
+# proguardFiles setting in build.gradle.
+#
+# For more details, see
+#   http://developer.android.com/guide/developing/tools/proguard.html
+
+# If your project uses WebView with JS, uncomment the following
+# and specify the fully qualified class name to the JavaScript interface
+# class:
+#-keepclassmembers class fqcn.of.javascript.interface.for.webview {
+#   public *;
+#}
+
+# Uncomment this to preserve the line number information for
+# debugging stack traces.
+#-keepattributes SourceFile,LineNumberTable
+
+# If you keep the line number information, uncomment this to
+# hide the original source file name.
+#-renamesourcefileattribute SourceFile

android-kiwix-lib-publisher/kiwixLibAndroid/src/main/AndroidManifest.xml

@@ -0,0 +1,10 @@
+<manifest xmlns:android="http://schemas.android.com/apk/res/android"
+    package="kiwix.org.kiwixlib">
+
+    <application
+        android:allowBackup="true"
+        android:supportsRtl="true">
+
+    </application>
+
+</manifest>

android-kiwix-lib-publisher/settings.gradle

@@ -0,0 +1 @@
+include ':kiwixLibAndroid'

include/downloader.h

@@ -54,6 +54,9 @@ class Download {
       m_status(K_UNKNOWN),
       m_did(did) {};
   void updateStatus(bool follow=false);
+  void pauseDownload();
+  void resumeDownload();
+  void cancelDownload();
   StatusResult getStatus()          { return m_status; }
   std::string  getDid()             { return m_did; }
   std::string  getFollowedBy()      { return m_followedBy; }

include/kiwixserve.h

@@ -0,0 +1,27 @@
+#ifndef KIWIXLIB_KIWIX_SERVE_H_
+#define KIWIXLIB_KIWIX_SERVE_H_
+
+#include <memory>
+
+class Subprocess;
+namespace kiwix {
+
+class KiwixServe
+{
+  public:
+    KiwixServe(int port = 8181);
+    ~KiwixServe();
+
+    void run();
+    void shutDown();
+    bool isRunning();
+    int getPort() { return m_port; }
+
+  private:
+  std::unique_ptr<Subprocess> mp_kiwixServe;
+  int m_port;
+};
+
+}; //end namespace kiwix
+
+#endif // KIWIXLIB_KIWIX_SERVE_H_

include/library.h

@@ -26,6 +26,7 @@
 
 #include "book.h"
 #include "bookmark.h"
+#include "common.h"
 
 #define KIWIX_LIBRARY_VERSION "20110515"
 
@@ -44,6 +45,65 @@ enum supportedListMode {
   VALID = 1 << 4,
   NOVALID = 1 << 5
 };
+
+class Filter {
+  private:
+    uint64_t activeFilters;
+    std::vector<std::string> _acceptTags;
+    std::vector<std::string> _rejectTags;
+    std::string _lang;
+    std::string _publisher;
+    std::string _creator;
+    size_t _maxSize;
+    std::string _query;
+
+  public:
+    Filter();
+    ~Filter() = default;
+
+    /**
+     *  Set the filter to check local.
+     *
+     *  A local book is a book with a path.
+     *  If accept is true, only local book are accepted.
+     *  If accept is false, only non local book are accepted.
+     */
+    Filter& local(bool accept);
+
+    /**
+     *  Set the filter to check remote.
+     *
+     *  A remote book is a book with a url.
+     *  If accept is true, only remote book are accepted.
+     *  If accept is false, only non remote book are accepted.
+     */
+    Filter& remote(bool accept);
+
+    /**
+     *  Set the filter to check validity.
+     *
+     *  A valid book is a book with a path pointing to a existing zim file.
+     *  If accept is true, only valid book are accepted.
+     *  If accept is false, only non valid book are accepted.
+     */
+    Filter& valid(bool accept);
+
+    /**
+     * Set the filter to only accept book with corresponding tag.
+     */
+    Filter& acceptTags(std::vector<std::string> tags);
+    Filter& rejectTags(std::vector<std::string> tags);
+
+    Filter& lang(std::string lang);
+    Filter& publisher(std::string publisher);
+    Filter& creator(std::string creator);
+    Filter& maxSize(size_t size);
+    Filter& query(std::string query);
+
+    bool accept(const Book& book) const;
+};
+
+
 /**
  * A Library store several books.
  */
@@ -162,9 +222,27 @@ class Library
    * @param search List only books with search in the title or description.
    * @return The list of bookIds corresponding to the query.
    */
-  std::vector<std::string> filter(const std::string& search);
+  DEPRECATED std::vector<std::string> filter(const std::string& search);
 
 
+  /**
+   * Filter the library and return the id of the keep elements.
+   *
+   * @param filter The filter to use.
+   * @return The list of bookIds corresponding to the filter.
+   */
+  std::vector<std::string> filter(const Filter& filter);
+
+
+  /**
+   * Sort (in place) bookIds using the given comparator.
+   *
+   * @param bookIds the list of book Ids to sort
+   * @param comparator how to sort the books
+   * @return The sorted list of books
+   */
+  void sort(std::vector<std::string>& bookIds, supportedListSortBy sortBy, bool ascending);
+
   /**
    * List books in the library.
    *
@@ -187,13 +265,14 @@ class Library
    *                Set to 0 to cancel this filter.
    * @return The list of bookIds corresponding to the query.
    */
-  std::vector<std::string> listBooksIds(
+  DEPRECATED std::vector<std::string> listBooksIds(
     int supportedListMode = ALL,
     supportedListSortBy sortBy = UNSORTED,
     const std::string& search = "",
     const std::string& language = "",
     const std::string& creator = "",
     const std::string& publisher = "",
+    const std::vector<std::string>& tags = {},
     size_t maxSize = 0);
 
   friend class OPDSDumper;

include/manager.h

@@ -58,9 +58,6 @@ class DefaultLibraryManipulator : public LibraryManipulator {
 
 /**
  * A tool to manage a `Library`.
- *
- * A `Manager` handle a internal `Library`.
- * This `Library` can be retrived with `cloneLibrary` method.
  */
 class Manager
 {
@@ -158,55 +155,6 @@ class Manager
                        const std::string& url = "",
                        const bool checkMetaData = false);
 
-  /**
-   * Get the book corresponding to an id.
-   *
-   * @param[in] id The id of the book
-   * @param[out] book The book corresponding to the id.
-   * @return True if the book has been found.
-   */
-  bool getBookById(const std::string& id, Book& book);
-
-  /**
-   * Update the "last open date" of a book
-   *
-   * @param id the id of the book.
-   * @return True if the book is in the library.
-   */
-  bool updateBookLastOpenDateById(const std::string& id);
-
-  /**
-   * Remove (set to empty) paths of all books in the library.
-   */
-  void removeBookPaths();
-
-  /**
-   * List books in the library.
-   *
-   * The books list will be available in public vector member `bookIdList`.
-   *
-   * @param mode The mode of listing :
-   *             - LASTOPEN sort by last opened book.
-   *             - LOCAL list only local file.
-   *             - REMOTE list only remote file.
-   * @param sortBy Attribute to sort by the book list.
-   * @param maxSize Do not list book bigger than maxSize MiB.
-   *                Set to 0 to cancel this filter.
-   * @param language List only books in this language.
-   * @param creator List only books of this creator.
-   * @param publisher List only books of this publisher.
-   * @param search List only books with search in the title, description or
-   *               language.
-   * @return True
-   */
-  bool listBooks(const supportedListMode mode,
-                 const supportedListSortBy sortBy,
-                 const unsigned int maxSize,
-                 const std::string& language,
-                 const std::string& creator,
-                 const std::string& publisher,
-                 const std::string& search);
-
   std::string writableLibraryPath;
 
   bool m_hasSearchResult = false;
@@ -225,8 +173,6 @@ class Manager
   bool parseOpdsDom(const pugi::xml_document& doc,
                     const std::string& urlHost);
 
- private:
-  void checkAndCleanBookPaths(Book& book, const std::string& libraryPath);
 };
 }
 

include/meson.build

@@ -9,7 +9,8 @@ headers = [
   'downloader.h',
   'reader.h',
   'entry.h',
-  'searcher.h'
+  'searcher.h',
+  'kiwixserve.h'
 ]
 
 install_headers(headers, subdir:'kiwix')

include/tools/networkTools.h

@@ -20,13 +20,10 @@
 #ifndef KIWIX_NETWORKTOOLS_H
 #define KIWIX_NETWORKTOOLS_H
 
-#include <map>
 #include <string>
 
 namespace kiwix
 {
-std::map<std::string, std::string> getNetworkInterfaces();
-std::string getBestPublicIp();
 std::string download(const std::string& url);
 }
 

meson.build

@@ -1,5 +1,5 @@
 project('kiwix-lib', 'cpp',
-  version : '4.0.0',
+  version : '5.2.0',
   license : 'GPL',
   default_options : ['c_std=c11', 'cpp_std=c++11', 'werror=true'])
 
@@ -14,7 +14,7 @@ endif
 
 thread_dep = dependency('threads')
 libicu_dep = dependency('icu-i18n', static:static_deps)
-libzim_dep = dependency('libzim', version : '>=4.0.0', static:static_deps)
+libzim_dep = dependency('libzim', version : '>=5.0.0', static:static_deps)
 pugixml_dep = dependency('pugixml', static:static_deps)
 libcurl_dep = dependency('libcurl', static:static_deps)
 

src/android/kiwixicu.cpp

@@ -19,7 +19,7 @@
  */
 
 #include <jni.h>
-#include "org_kiwix_kiwixlib_JNIKiwix.h"
+#include "org_kiwix_kiwixlib_JNIICU.h"
 
 #include <iostream>
 #include <string>
@@ -30,7 +30,7 @@
 
 pthread_mutex_t globalLock = PTHREAD_RECURSIVE_MUTEX_INITIALIZER;
 
-JNIEXPORT void JNICALL Java_org_kiwix_kiwixlib_JNIKiwix_setDataDirectory(
+JNIEXPORT void JNICALL Java_org_kiwix_kiwixlib_JNIICU_setDataDirectory(
     JNIEnv* env, jobject obj, jstring dirStr)
 {
   std::string cPath = jni2c(dirStr, env);

src/android/kiwixreader.cpp

@@ -91,7 +91,7 @@ Java_org_kiwix_kiwixlib_JNIKiwixReader_getId(JNIEnv* env, jobject obj)
 JNIEXPORT jint JNICALL
 Java_org_kiwix_kiwixlib_JNIKiwixReader_getFileSize(JNIEnv* env, jobject obj)
 {
-  jint size;
+  jint size = 0;
 
   try {
     int cSize = READER->getFileSize();

src/android/meson.build

@@ -1,6 +1,6 @@
 
 kiwix_jni = custom_target('jni',
-  input: ['org/kiwix/kiwixlib/JNIKiwix.java',
+  input: ['org/kiwix/kiwixlib/JNIICU.java',
           'org/kiwix/kiwixlib/JNIKiwixReader.java',
           'org/kiwix/kiwixlib/JNIKiwixSearcher.java',
           'org/kiwix/kiwixlib/JNIKiwixInt.java',
@@ -16,7 +16,7 @@ kiwix_jni = custom_target('jni',
 )
 
 kiwix_sources += [
-    'android/kiwix.cpp',
+    'android/kiwixicu.cpp',
     'android/kiwixreader.cpp',
     'android/kiwixsearcher.cpp',
     kiwix_jni]

src/android/org/kiwix/kiwixlib/JNIICU.java

@@ -0,0 +1,26 @@
+/*
+ * Copyright (C) 2013 Emmanuel Engelhart <kelson@kiwix.org>
+ * Copyright (C) 2017 Matthieu Gautier <mgautier@kymeria.fr>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU  General Public License as published by
+ * the Free Software Foundation; either version 3 of the License, or
+ * any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,
+ * MA 02110-1301, USA.
+ */
+
+package org.kiwix.kiwixlib;
+
+public class JNIICU
+{
+  static public native void setDataDirectory(String icuDataDir);
+}

src/android/org/kiwix/kiwixlib/JNIKiwix.java

@@ -20,12 +20,17 @@
 
 package org.kiwix.kiwixlib;
 
-import org.kiwix.kiwixlib.JNIKiwixReader;
-import org.kiwix.kiwixlib.JNIKiwixString;
+import android.content.Context;
+import com.getkeepsafe.relinker.ReLinker;
+import org.kiwix.kiwixlib.JNIICU;
 
 public class JNIKiwix
 {
-  static { System.loadLibrary("kiwix"); }
+  public JNIKiwix(final Context context){
+    ReLinker.loadLibrary(context, "kiwix");
+  }
 
-  public native void setDataDirectory(String icuDataDir);
+  public void setDataDirectory(String icuDataDir) {
+    JNIICU.setDataDirectory(icuDataDir);
+  }
 }

src/aria2.cpp

@@ -82,7 +82,7 @@ Aria2::Aria2():
 
   int watchdog = 50;
   while(--watchdog) {
-    std::this_thread::sleep_for(std::chrono::microseconds(10000));
+    sleep(10);
     auto res = curl_easy_perform(mp_curl);
     if (res == CURLE_OK) {
       break;
@@ -171,7 +171,6 @@ std::vector<std::string> Aria2::tellActive()
   MethodCall methodCall("aria2.tellActive", m_secret);
   auto statusArray = methodCall.newParamValue().getArray();
   statusArray.addValue().set(std::string("gid"));
-  statusArray.addValue().set(std::string("following"));
   auto responseContent = doRequest(methodCall);
   MethodResponse response(responseContent);
   std::vector<std::string> activeGID;
@@ -186,6 +185,27 @@ std::vector<std::string> Aria2::tellActive()
   return activeGID;
 }
 
+std::vector<std::string> Aria2::tellWaiting()
+{
+  MethodCall methodCall("aria2.tellWaiting", m_secret);
+  methodCall.newParamValue().set(0);
+  methodCall.newParamValue().set(99); // max number of downloads to be returned, don't know how to set this properly assumed that there will not be more than 99 paused downloads.
+  auto statusArray = methodCall.newParamValue().getArray();
+  statusArray.addValue().set(std::string("gid"));
+  auto responseContent = doRequest(methodCall);
+  MethodResponse response(responseContent);
+  std::vector<std::string> waitingGID;
+  int index = 0;
+  while(true) {
+    try {
+      auto structNode = response.getParamValue(0).getArray().getValue(index++).getStruct();
+      auto gidNode = structNode.getMember("gid");
+      waitingGID.push_back(gidNode.getValue().getAsS());
+    } catch (InvalidRPCNode& e) { break; }
+  }
+  return waitingGID;
+}
+
 void Aria2::saveSession()
 {
   MethodCall methodCall("aria2.saveSession", m_secret);
@@ -199,5 +219,25 @@ void Aria2::shutdown()
   doRequest(methodCall);
 }
 
+void Aria2::pause(const std::string& gid)
+{
+    MethodCall methodCall("aria2.pause", m_secret);
+    methodCall.newParamValue().set(gid);
+    doRequest(methodCall);
+}
+
+void Aria2::unpause(const std::string& gid)
+{
+    MethodCall methodCall("aria2.unpause", m_secret);
+    methodCall.newParamValue().set(gid);
+    doRequest(methodCall);
+}
+
+void Aria2::remove(const std::string& gid)
+{
+    MethodCall methodCall("aria2.remove", m_secret);
+    methodCall.newParamValue().set(gid);
+    doRequest(methodCall);
+}
 
 } // end namespace kiwix

src/aria2.h

@@ -37,8 +37,12 @@ class Aria2
     std::string addUri(const std::vector<std::string>& uri);
     std::string tellStatus(const std::string& gid, const std::vector<std::string>& statusKey);
     std::vector<std::string> tellActive();
+    std::vector<std::string> tellWaiting();
     void saveSession();
     void shutdown();
+    void pause(const std::string& gid);
+    void unpause(const std::string& gid);
+    void remove(const std::string& gid);
 };
 
 }; //end namespace kiwix

src/book.cpp

@@ -138,6 +138,7 @@ void Book::updateFromOpds(const pugi::xml_node& node, const std::string& urlHost
   m_language = VALUE("language");
   m_date = fromOpdsDate(VALUE("updated"));
   m_creator = node.child("author").child("name").child_value();
+  m_tags = VALUE("tags");
   for(auto linkNode = node.child("link"); linkNode;
            linkNode = linkNode.next_sibling("link")) {
     std::string rel = linkNode.attribute("rel").value();

src/downloader.cpp

@@ -36,6 +36,8 @@ namespace kiwix
 
 void Download::updateStatus(bool follow)
 {
+  if (m_status == Download::K_REMOVED)
+    return;
   static std::vector<std::string> statusKey = {"status", "files", "totalLength",
                                                "completedLength", "followedBy",
                                                "downloadSpeed", "verifiedLength"};
@@ -93,6 +95,33 @@ void Download::updateStatus(bool follow)
   }
 }
 
+void Download::resumeDownload()
+{
+    if (!m_followedBy.empty())
+        mp_aria->unpause(m_followedBy);
+    else
+        mp_aria->unpause(m_did);
+    updateStatus(true);
+}
+
+void Download::pauseDownload()
+{
+    if (!m_followedBy.empty())
+        mp_aria->pause(m_followedBy);
+    else
+        mp_aria->pause(m_did);
+    updateStatus(true);
+}
+
+void Download::cancelDownload()
+{
+    if (!m_followedBy.empty())
+        mp_aria->remove(m_followedBy);
+    else
+        mp_aria->remove(m_did);
+    m_status = Download::K_REMOVED;
+}
+
 /* Constructor */
 Downloader::Downloader() :
   mp_aria(new Aria2())
@@ -101,6 +130,10 @@ Downloader::Downloader() :
     m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
     m_knownDownloads[gid]->updateStatus();
   }
+  for (auto gid : mp_aria->tellWaiting()) {
+    m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
+    m_knownDownloads[gid]->updateStatus();
+  }
 }
 
 
@@ -139,14 +172,23 @@ Download* Downloader::startDownload(const std::string& uri)
 Download* Downloader::getDownload(const std::string& did)
 {
   try {
+    m_knownDownloads.at(did).get()->updateStatus(true);
     return m_knownDownloads.at(did).get();
   } catch(exception& e) {
     for (auto gid : mp_aria->tellActive()) {
       if (gid == did) {
         m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
+        m_knownDownloads.at(gid).get()->updateStatus(true);
         return m_knownDownloads[gid].get();
       }
     }
+    for (auto gid : mp_aria->tellWaiting()) {
+      if (gid == did) {
+        m_knownDownloads[gid] = std::unique_ptr<Download>(new Download(mp_aria, gid));
+        m_knownDownloads.at(gid).get()->updateStatus(true);
+        return m_knownDownloads[gid].get();
+      }
+  }
     throw e;
   }
 }

src/entry.cpp

@@ -122,7 +122,6 @@ Entry Entry::getFinalEntry() const
   if (final_article.good()) {
     return final_article;
   }
-
   int loopCounter = 42;
   final_article = article;
   while (final_article.isRedirect() && loopCounter--) {
@@ -131,7 +130,10 @@ Entry Entry::getFinalEntry() const
       throw NoEntry();
     }
   }
-
+  // Prevent infinite loops.
+  if (final_article.isRedirect()) {
+    throw NoEntry();
+  }
   return final_article;
 }
 

src/kiwixserve.cpp

@@ -0,0 +1,69 @@
+#include "kiwixserve.h"
+#include "subprocess.h"
+
+#ifdef _WIN32
+    # define KIWIXSERVE_CMD "kiwix-serve.exe"
+    # include <windows.h>
+#else
+    # define KIWIXSERVE_CMD "kiwix-serve"
+    # include <unistd.h>
+#endif
+
+#include "tools/pathTools.h"
+
+namespace kiwix {
+
+KiwixServe::KiwixServe(int port) : m_port(port)
+{
+}
+
+KiwixServe::~KiwixServe()
+{
+    shutDown();
+}
+
+void KiwixServe::run()
+{
+    #ifdef _WIN32
+        int pid = GetCurrentProcessId();
+    #else
+        pid_t pid = getpid();
+
+    #endif
+
+    std::vector<const char*> callCmd;
+    std::string kiwixServeCmd = appendToDirectory(
+        removeLastPathElement(getExecutablePath(), true, true),
+        KIWIXSERVE_CMD);
+    if (fileExists(kiwixServeCmd)) {
+        // A local kiwix-serve exe exists (packaged with kiwix-desktop), use it.
+        callCmd.push_back(kiwixServeCmd.c_str());
+    } else {
+        // Try to use a potential installed kiwix-serve.
+        callCmd.push_back(KIWIXSERVE_CMD);
+    }
+    std::string libraryPath = getDataDirectory() + "/library.xml";
+    std::string attachProcessOpt = "-a" + to_string(pid);
+    std::string portOpt = "-p" + to_string(m_port);
+    callCmd.push_back(attachProcessOpt.c_str());
+    callCmd.push_back(portOpt.c_str());
+    callCmd.push_back("-l");
+    callCmd.push_back(libraryPath.c_str());
+    mp_kiwixServe = Subprocess::run(callCmd);
+}
+
+void KiwixServe::shutDown()
+{
+    if (mp_kiwixServe)
+        mp_kiwixServe->kill();
+}
+
+bool KiwixServe::isRunning()
+{
+    if (mp_kiwixServe) {
+        return (mp_kiwixServe->isRunning());
+    }
+    return false;
+}
+
+}

src/library.cpp

@@ -27,6 +27,7 @@
 
 #include <pugixml.hpp>
 #include <algorithm>
+#include <set>
 
 namespace kiwix
 {
@@ -95,14 +96,16 @@ unsigned int Library::getBookCount(const bool localBooks,
   return result;
 }
 
-bool Library::writeToFile(const std::string& path) {
+bool Library::writeToFile(const std::string& path)
+{
   auto baseDir = removeLastPathElement(path, true, false);
   LibXMLDumper dumper(this);
   dumper.setBaseDir(baseDir);
   return writeTextFile(path, dumper.dumpLibXMLContent(getBooksIds()));
 }
 
-bool Library::writeBookmarksToFile(const std::string& path) {
+bool Library::writeBookmarksToFile(const std::string& path)
+{
   LibXMLDumper dumper(this);
   return writeTextFile(path, dumper.dumpLibXMLBookmark());
 }
@@ -181,67 +184,104 @@ std::vector<std::string> Library::filter(const std::string& search)
     return getBooksIds();
   }
 
+  return filter(Filter().query(search));
+}
+
+
+std::vector<std::string> Library::filter(const Filter& filter)
+{
   std::vector<std::string> bookIds;
   for(auto& pair:m_books) {
-     auto& book = pair.second;
-     if (matchRegex(book.getTitle(), "\\Q" + search + "\\E")
-         || matchRegex(book.getDescription(), "\\Q" + search + "\\E")) {
-       bookIds.push_back(pair.first);
-     }
+    auto book = pair.second;
+    if(filter.accept(book)) {
+      bookIds.push_back(pair.first);
+    }
   }
-
   return bookIds;
 }
 
-template<supportedListSortBy sort>
-struct Comparator {
-  Library* lib;
-  Comparator(Library* lib) : lib(lib) {}
-
-  bool operator() (const std::string& id1, const std::string& id2) {
-    return get_keys(id1) < get_keys(id2);
-  }
-
-  std::string get_keys(const std::string& id);
-  unsigned int get_keyi(const std::string& id);
+template<supportedListSortBy SORT>
+struct KEY_TYPE {
+  typedef std::string TYPE;
 };
 
 template<>
-std::string Comparator<TITLE>::get_keys(const std::string& id)
+struct KEY_TYPE<SIZE> {
+  typedef size_t TYPE;
+};
+
+template<supportedListSortBy sort>
+class Comparator {
+  private:
+    Library* lib;
+    bool     ascending;
+
+    inline typename KEY_TYPE<sort>::TYPE get_key(const std::string& id);
+
+  public:
+    Comparator(Library* lib, bool ascending) : lib(lib), ascending(ascending) {}
+    inline bool operator() (const std::string& id1, const std::string& id2) {
+      if (ascending) {
+        return get_key(id1) < get_key(id2);
+      } else {
+        return get_key(id2) < get_key(id1);
+      }
+    }
+};
+
+template<>
+std::string Comparator<TITLE>::get_key(const std::string& id)
 {
   return lib->getBookById(id).getTitle();
 }
 
 template<>
-unsigned int Comparator<SIZE>::get_keyi(const std::string& id)
+size_t Comparator<SIZE>::get_key(const std::string& id)
 {
   return lib->getBookById(id).getSize();
 }
 
 template<>
-bool Comparator<SIZE>::operator() (const std::string& id1, const std::string& id2)
-{
-  return get_keyi(id1) < get_keyi(id2);
-}
-
-template<>
-std::string Comparator<DATE>::get_keys(const std::string& id)
+std::string Comparator<DATE>::get_key(const std::string& id)
 {
   return lib->getBookById(id).getDate();
 }
 
 template<>
-std::string Comparator<CREATOR>::get_keys(const std::string& id)
+std::string Comparator<CREATOR>::get_key(const std::string& id)
 {
   return lib->getBookById(id).getCreator();
 }
 
 template<>
-std::string Comparator<PUBLISHER>::get_keys(const std::string& id)
+std::string Comparator<PUBLISHER>::get_key(const std::string& id)
 {
   return lib->getBookById(id).getPublisher();
 }
 
+void Library::sort(std::vector<std::string>& bookIds, supportedListSortBy sort, bool ascending)
+{
+  switch(sort) {
+    case TITLE:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<TITLE>(this, ascending));
+      break;
+    case SIZE:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<SIZE>(this, ascending));
+      break;
+    case DATE:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<DATE>(this, ascending));
+      break;
+    case CREATOR:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<CREATOR>(this, ascending));
+      break;
+    case PUBLISHER:
+      std::sort(bookIds.begin(), bookIds.end(), Comparator<PUBLISHER>(this, ascending));
+      break;
+    default:
+      break;
+  }
+}
+
 
 std::vector<std::string> Library::listBooksIds(
     int mode,
@@ -250,59 +290,208 @@ std::vector<std::string> Library::listBooksIds(
     const std::string& language,
     const std::string& creator,
     const std::string& publisher,
+    const std::vector<std::string>& tags,
     size_t maxSize) {
 
-  std::vector<std::string> bookIds;
-  for(auto& pair:m_books) {
-    auto& book = pair.second;
-    auto local = !book.getPath().empty();
-    if (mode & LOCAL && !local)
-      continue;
-    if (mode & NOLOCAL && local)
-      continue;
-    auto valid = book.isPathValid();
-    if (mode & VALID && !valid)
-      continue;
-    if (mode & NOVALID && valid)
-      continue;
-    auto remote = !book.getUrl().empty();
-    if (mode & REMOTE && !remote)
-      continue;
-    if (mode & NOREMOTE && remote)
-      continue;
-    if (maxSize != 0 && book.getSize() > maxSize)
-      continue;
-    if (!language.empty() && book.getLanguage() != language)
-      continue;
-    if (!publisher.empty() && book.getPublisher() != publisher)
-      continue;
-    if (!creator.empty() && book.getCreator() != creator)
-      continue;
-    if (!search.empty() && !(matchRegex(book.getTitle(), "\\Q" + search + "\\E")
-                          || matchRegex(book.getDescription(), "\\Q" + search + "\\E")))
-      continue;
-    bookIds.push_back(pair.first);
-  }
+  Filter _filter;
+  if (mode & LOCAL)
+    _filter.local(true);
+  if (mode & NOLOCAL)
+    _filter.local(false);
+  if (mode & VALID)
+    _filter.valid(true);
+  if (mode & NOVALID)
+    _filter.valid(false);
+  if (mode & REMOTE)
+    _filter.remote(true);
+  if (mode & NOREMOTE)
+    _filter.remote(false);
+  if (!tags.empty())
+    _filter.acceptTags(tags);
+  if (maxSize != 0)
+    _filter.maxSize(maxSize);
+  if (!language.empty())
+    _filter.lang(language);
+  if (!publisher.empty())
+    _filter.publisher(publisher);
+  if (!creator.empty())
+    _filter.creator(creator);
+  if (!search.empty())
+    _filter.query(search);
 
-  switch(sortBy) {
-    case TITLE:
-      std::sort(bookIds.begin(), bookIds.end(), Comparator<TITLE>(this));
-      break;
-    case SIZE:
-      std::sort(bookIds.begin(), bookIds.end(), Comparator<SIZE>(this));
-      break;
-    case DATE:
-      std::sort(bookIds.begin(), bookIds.end(), Comparator<DATE>(this));
-      break;
-    case CREATOR:
-      std::sort(bookIds.begin(), bookIds.end(), Comparator<CREATOR>(this));
-      break;
-    case PUBLISHER:
-      std::sort(bookIds.begin(), bookIds.end(), Comparator<PUBLISHER>(this));
-      break;
-    default:
-      break;
-  }
+  auto bookIds = filter(_filter);
+
+  sort(bookIds, sortBy, true);
   return bookIds;
 }
+
+Filter::Filter()
+  : activeFilters(0),
+    _maxSize(0)
+{};
+
+#define FLAG(x) (1 << x)
+enum filterTypes {
+  NONE = 0,
+  _LOCAL = FLAG(0),
+  _REMOTE = FLAG(1),
+  _NOLOCAL = FLAG(2),
+  _NOREMOTE = FLAG(3),
+  _VALID = FLAG(4),
+  _NOVALID = FLAG(5),
+  ACCEPTTAGS = FLAG(6),
+  REJECTTAGS = FLAG(7),
+  LANG = FLAG(8),
+  _PUBLISHER = FLAG(9),
+  _CREATOR = FLAG(10),
+  MAXSIZE = FLAG(11),
+  QUERY = FLAG(12),
+};
+
+Filter& Filter::local(bool accept)
+{
+  if (accept) {
+    activeFilters |= _LOCAL;
+    activeFilters &= ~_NOLOCAL;
+  } else {
+    activeFilters |= _NOLOCAL;
+    activeFilters &= ~_LOCAL;
+  }
+  return *this;
+}
+
+Filter& Filter::remote(bool accept)
+{
+  if (accept) {
+    activeFilters |= _REMOTE;
+    activeFilters &= ~_NOREMOTE;
+  } else {
+    activeFilters |= _NOREMOTE;
+    activeFilters &= ~_REMOTE;
+  }
+  return *this;
+}
+
+Filter& Filter::valid(bool accept)
+{
+  if (accept) {
+    activeFilters |= _VALID;
+    activeFilters &= ~_NOVALID;
+  } else {
+    activeFilters |= _NOVALID;
+    activeFilters &= ~_VALID;
+  }
+  return *this;
+}
+
+Filter& Filter::acceptTags(std::vector<std::string> tags)
+{
+  _acceptTags = tags;
+  activeFilters |= ACCEPTTAGS;
+  return *this;
+}
+
+Filter& Filter::rejectTags(std::vector<std::string> tags)
+{
+  _rejectTags = tags;
+  activeFilters |= REJECTTAGS;
+  return *this;
+}
+
+Filter& Filter::lang(std::string lang)
+{
+  _lang = lang;
+  activeFilters |= LANG;
+  return *this;
+}
+
+Filter& Filter::publisher(std::string publisher)
+{
+  _publisher = publisher;
+  activeFilters |= _PUBLISHER;
+  return *this;
+}
+
+Filter& Filter::creator(std::string creator)
+{
+  _creator = creator;
+  activeFilters |= _CREATOR;
+  return *this;
+}
+
+Filter& Filter::maxSize(size_t maxSize)
+{
+  _maxSize = maxSize;
+  activeFilters |= MAXSIZE;
+  return *this;
+}
+
+Filter& Filter::query(std::string query)
+{
+  _query = query;
+  activeFilters |= QUERY;
+  return *this;
+}
+
+#define ACTIVE(X) (activeFilters & (X))
+bool Filter::accept(const Book& book) const
+{
+  auto local = !book.getPath().empty();
+  if (ACTIVE(_LOCAL) && !local)
+    return false;
+  if (ACTIVE(_NOLOCAL) && local)
+    return false;
+  auto valid = book.isPathValid();
+  if (ACTIVE(_VALID) && !valid)
+    return false;
+  if (ACTIVE(_NOVALID) && valid)
+    return false;
+  auto remote = !book.getUrl().empty();
+  if (ACTIVE(_REMOTE) && !remote)
+    return false;
+  if (ACTIVE(_NOREMOTE) && remote)
+    return false;
+  if (ACTIVE(ACCEPTTAGS)) {
+    if (!_acceptTags.empty()) {
+      auto vBookTags = split(book.getTags(), ";");
+      std::set<std::string> sBookTags(vBookTags.begin(), vBookTags.end());
+      for (auto& t: _acceptTags) {
+        if (sBookTags.find(t) == sBookTags.end()) {
+          return false;
+        }
+      }
+    }
+  }
+  if (ACTIVE(REJECTTAGS)) {
+    if (!_rejectTags.empty()) {
+      auto vBookTags = split(book.getTags(), ";");
+      std::set<std::string> sBookTags(vBookTags.begin(), vBookTags.end());
+      for (auto& t: _rejectTags) {
+        if (sBookTags.find(t) != sBookTags.end()) {
+          return false;
+        }
+      }
+    }
+  }
+  if (ACTIVE(MAXSIZE) && book.getSize() > _maxSize)
+    return false;
+
+  if (ACTIVE(LANG) && book.getLanguage() != _lang)
+    return false;
+
+  if (ACTIVE(_PUBLISHER) && book.getPublisher() != _publisher)
+    return false;
+
+  if (ACTIVE(_CREATOR) && book.getCreator() != _creator)
+    return false;
+
+  if ( ACTIVE(QUERY)
+    && !(matchRegex(book.getTitle(), "\\Q" + _query + "\\E")
+        || matchRegex(book.getDescription(), "\\Q" + _query + "\\E")))
+    return false;
+
+  return true;
+
+}
+
 }

src/manager.cpp

@@ -208,12 +208,15 @@ bool Manager::addBookFromPath(const std::string& pathToOpen,
 
 bool Manager::readBookFromPath(const std::string& path, kiwix::Book* book)
 {
+  std::string tmp_path = path;
+  if (isRelativePath(path)) {
+    tmp_path = computeAbsolutePath(getCurrentDirectory(), path);
+  }
   try {
-    kiwix::Reader reader(path);
+    kiwix::Reader reader(tmp_path);
     book->update(reader);
     book->setPathValid(true);
   } catch (const std::exception& e) {
-    std::cerr << "Invalid " << path << " : " << e.what() << std::endl;
     book->setPathValid(false);
     return false;
   }

src/meson.build

@@ -17,6 +17,7 @@ kiwix_sources = [
   'tools/stringTools.cpp',
   'tools/networkTools.cpp',
   'tools/otherTools.cpp',
+  'kiwixserve.cpp',
 ]
 kiwix_sources += lib_resources
 
@@ -44,5 +45,4 @@ kiwixlib = library('kiwix',
                    dependencies : all_deps,
                    version: meson.project_version(),
                    install: true,
-                   install_dir: install_dir,
-                   install_rpath: '$ORIGIN')
+                   install_dir: install_dir)

src/opds_dumper.cpp

@@ -74,6 +74,7 @@ pugi::xml_node OPDSDumper::handleBook(Book book, pugi::xml_node root_node) {
   ADD_TEXT_ENTRY(entry_node, "icon", rootLocation + "/meta?name=favicon&content=" + book.getHumanReadableIdFromPath());
   ADD_TEXT_ENTRY(entry_node, "updated", gen_date_from_yyyy_mm_dd(book.getDate()));
   ADD_TEXT_ENTRY(entry_node, "summary", book.getDescription());
+  ADD_TEXT_ENTRY(entry_node, "tags", book.getTags());
 
   auto content_node = entry_node.append_child("link");
   content_node.append_attribute("type") = "text/html";

src/reader.cpp

@@ -241,8 +241,6 @@ Entry Reader::getMainPage() const
     throw NoEntry();
   }
 
-  string url = "";
-
   zim::Article article;
   if (this->zimFileHandler->getFileheader().hasMainPage())
   {
@@ -767,13 +765,16 @@ bool Reader::searchSuggestionsSmart(const string& prefix,
   this->suggestions.clear();
   this->suggestionsOffset = this->suggestions.begin();
   /* Try to search in the title using fulltext search database */
-  const zim::Search* suggestionSearch
+  const auto suggestionSearch
       = this->getZimFileHandler()->suggestions(prefix, 0, suggestionsCount);
 
   if (suggestionSearch->get_matches_estimated()) {
     for (auto current = suggestionSearch->begin();
          current != suggestionSearch->end();
          current++) {
+      if (!current->good()) {
+          continue;
+      }
       std::vector<std::string> suggestion;
       suggestion.push_back(current->getTitle());
       suggestion.push_back("/A/" + current->getUrl());

src/searcher.cpp

@@ -136,6 +136,7 @@ void Searcher::search(std::string& search,
       }
     }
     zim::Search* search = new zim::Search(zims);
+    search->set_verbose(verbose);
     search->set_query(unaccentedSearch);
     search->set_range(resultStart, resultEnd);
     internal->_search = search;
@@ -190,6 +191,7 @@ void Searcher::geo_search(float latitude, float longitude, float distance,
     zims.push_back((*current)->getZimFileHandler());
   }
   zim::Search* search = new zim::Search(zims);
+  search->set_verbose(verbose);
   search->set_query("");
   search->set_georange(latitude, longitude, distance);
   search->set_range(resultStart, resultEnd);
@@ -244,6 +246,7 @@ void Searcher::suggestions(std::string& searchPattern, const bool verbose)
     zims.push_back((*current)->getZimFileHandler());
   }
   zim::Search* search = new zim::Search(zims);
+  search->set_verbose(verbose);
   search->set_query(unaccentedSearch);
   search->set_range(resultStart, resultEnd);
   search->set_suggestion_mode(true);

src/subprocess_unix.cpp

@@ -46,7 +46,7 @@ void* UnixImpl::waitForPID(void* _self)
 #endif
 
   UnixImpl* self = static_cast<UnixImpl*>(_self);
-  waitpid(self->m_pid, NULL, WEXITED);
+  waitpid(self->m_pid, NULL, 0);
 
   pthread_mutex_lock(&self->m_mutex);
   self->m_running = false;
@@ -67,7 +67,7 @@ void UnixImpl::run(commandLine_t& commandLine)
       commandLine.push_back(NULL);
       if (execvp(binary, const_cast<char* const*>(commandLine.data()))) {
         perror("Cannot launch\n");
-        exit(-1);
+        _exit(-1);
       }
 
       break;

src/tools/networkTools.cpp

@@ -19,167 +19,17 @@
 
 #include <tools/networkTools.h>
 
-#ifdef _WIN32
-#include <winsock2.h>
-#include <ws2tcpip.h>
-#else
-#include <net/if.h>
-#include <netdb.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
-#include <sys/ioctl.h>
-#include <sys/socket.h>
-#include <sys/types.h>
-#include <unistd.h>
-#endif
 
 #include <curl/curl.h>
 
 #include <sstream>
 #include <iostream>
+#include <stdexcept>
 
 
-std::map<std::string, std::string> kiwix::getNetworkInterfaces()
-{
-  std::map<std::string, std::string> interfaces;
-
-#ifdef _WIN32
-  SOCKET sd = WSASocket(AF_INET, SOCK_DGRAM, 0, 0, 0, 0);
-  if (sd == (SOCKET)SOCKET_ERROR) {
-    std::cerr << "Failed to get a socket. Error " << WSAGetLastError()
-              << std::endl;
-    return interfaces;
-  }
-
-  INTERFACE_INFO InterfaceList[20];
-  unsigned long nBytesReturned;
-  if (WSAIoctl(sd,
-               SIO_GET_INTERFACE_LIST,
-               0,
-               0,
-               &InterfaceList,
-               sizeof(InterfaceList),
-               &nBytesReturned,
-               0,
-               0)
-      == SOCKET_ERROR) {
-    std::cerr << "Failed calling WSAIoctl: error " << WSAGetLastError()
-              << std::endl;
-    return interfaces;
-  }
-
-  int nNumInterfaces = nBytesReturned / sizeof(INTERFACE_INFO);
-  for (int i = 0; i < nNumInterfaces; ++i) {
-    sockaddr_in* pAddress;
-    pAddress = (sockaddr_in*)&(InterfaceList[i].iiAddress);
-
-    /* Add to the map */
-    std::string interfaceName = std::string(inet_ntoa(pAddress->sin_addr));
-    std::string interfaceIp = std::string(inet_ntoa(pAddress->sin_addr));
-    interfaces.insert(
-        std::pair<std::string, std::string>(interfaceName, interfaceIp));
-  }
-#else
-  /* Get Network interfaces information */
-  char buf[16384];
-  struct ifconf ifconf;
-  int fd = socket(PF_INET, SOCK_DGRAM, 0); /* Only IPV4 */
-  ifconf.ifc_len = sizeof buf;
-  ifconf.ifc_buf = buf;
-  if (ioctl(fd, SIOCGIFCONF, &ifconf) != 0) {
-    perror("ioctl(SIOCGIFCONF)");
-    exit(EXIT_FAILURE);
-  }
-
-  /* Go through each interface */
-  int i;
-  size_t len;
-  struct ifreq* ifreq;
-  ifreq = ifconf.ifc_req;
-  for (i = 0; i < ifconf.ifc_len;) {
-    if (ifreq->ifr_addr.sa_family == AF_INET) {
-      /* Get the network interface ip */
-      char host[128] = {0};
-      const int error = getnameinfo(&(ifreq->ifr_addr),
-                                    sizeof ifreq->ifr_addr,
-                                    host,
-                                    sizeof host,
-                                    0,
-                                    0,
-                                    NI_NUMERICHOST);
-      if (!error) {
-        std::string interfaceName = std::string(ifreq->ifr_name);
-        std::string interfaceIp = std::string(host);
-        /* Add to the map */
-        interfaces.insert(
-            std::pair<std::string, std::string>(interfaceName, interfaceIp));
-      } else {
-        perror("getnameinfo()");
-      }
-    }
-
-/* some systems have ifr_addr.sa_len and adjust the length that
- * way, but not mine. weird */
-#ifndef __linux__
-    len = IFNAMSIZ + ifreq->ifr_addr.sa_len;
-#else
-    len = sizeof *ifreq;
-#endif
-    ifreq = (struct ifreq*)((char*)ifreq + len);
-    i += len;
-  }
-#endif
-  return interfaces;
-}
-
-std::string kiwix::getBestPublicIp()
-{
-  std::map<std::string, std::string> interfaces = kiwix::getNetworkInterfaces();
-
-#ifndef _WIN32
-  const char* const prioritizedNames[]
-      = {"eth0", "eth1", "wlan0", "wlan1", "en0", "en1"};
-  const int count = (sizeof prioritizedNames) / (sizeof prioritizedNames[0]);
-  for (int i = 0; i < count; ++i) {
-    std::map<std::string, std::string>::const_iterator it
-        = interfaces.find(prioritizedNames[i]);
-    if (it != interfaces.end()) {
-      return it->second;
-    }
-  }
-#endif
-
-  for (std::map<std::string, std::string>::iterator iter = interfaces.begin();
-       iter != interfaces.end();
-       ++iter) {
-    std::string interfaceIp = iter->second;
-    if (interfaceIp.length() >= 7 && interfaceIp.substr(0, 7) == "192.168") {
-      return interfaceIp;
-    }
-  }
-
-  for (std::map<std::string, std::string>::iterator iter = interfaces.begin();
-       iter != interfaces.end();
-       ++iter) {
-    std::string interfaceIp = iter->second;
-    if (interfaceIp.length() >= 7 && interfaceIp.substr(0, 7) == "172.16.") {
-      return interfaceIp;
-    }
-  }
-
-  for (std::map<std::string, std::string>::iterator iter = interfaces.begin();
-       iter != interfaces.end();
-       ++iter) {
-    std::string interfaceIp = iter->second;
-    if (interfaceIp.length() >= 3 && interfaceIp.substr(0, 3) == "10.") {
-      return interfaceIp;
-    }
-  }
-
-  return "127.0.0.1";
-}
-
 size_t write_callback_to_iss(char* ptr, size_t size, size_t nmemb, void* userdata)
 {
   auto str = static_cast<std::stringstream*>(userdata);

src/tools/otherTools.cpp

@@ -21,279 +21,150 @@
 #include <map>
 
 static std::map<std::string, std::string> codeisomapping {
-//a
-  { "ad", "and" },
-  { "ae", "are" },
-  { "af", "afg" },
-  { "ag", "atg" },
-  { "ai", "aia" },
-  { "al", "alb" },
-  { "am", "arm" },
-  { "an", "ant" },
-  { "ao", "ago" },
-  { "aq", "ata" },
-  { "ar", "arg" },
-  { "as", "asm" },
-  { "at", "aut" },
-  { "au", "aus" },
-  { "aw", "abw" },
-  { "ax", "ala" },
-  { "az", "aze" },
-//b
-  { "ba", "bih" },
-  { "bb", "brb" },
-  { "bd", "bgd" },
-  { "be", "bel" },
-  { "bf", "bfa" },
-  { "bg", "bgr" },
-  { "bh", "bhr" },
-  { "bi", "bdi" },
-  { "bj", "ben" },
-  { "bl", "blm" },
-  { "bn", "brn" },
-  { "bm", "bmu" },
-  { "bo", "bol" },
-  { "br", "bra" },
-  { "bs", "bhs" },
-  { "bt", "btn" },
-  { "bv", "bvt" },
-  { "bw", "bwa" },
-  { "by", "blr" },
-  { "bz", "blz" },
-//c
-  { "ca", "can" },
-  { "cc", "cck" },
-  { "cd", "cod" },
-  { "cf", "caf" },
-  { "cg", "cog" },
-  { "ch", "che" },
-  { "ci", "civ" },
-  { "ck", "cok" },
-  { "cl", "chl" },
-  { "cm", "cmr" },
-  { "cn", "chn" },
-  { "co", "col" },
-  { "cr", "cri" },
-  { "cu", "cub" },
-  { "cv", "cpv" },
-  { "cx", "cxr" },
-  { "cy", "cyp" },
-  { "cz", "cze" },
-//d
-  { "de", "deu" },
-  { "dj", "dji" },
-  { "dk", "dnk" },
-  { "dm", "dma" },
-  { "do", "dom" },
-  { "dz", "dza" },
-//e
-  { "ec", "ecu" },
-  { "ee", "est" },
-  { "eg", "egy" },
-  { "eh", "esh" },
-  { "en", "eng" },
-  { "er", "eri" },
-  { "es", "esp" },
-  { "et", "eth" },
-//f
-  { "fi", "fin" },
-  { "fj", "fji" },
-  { "fk", "flk" },
-  { "fm", "fsm" },
-  { "fo", "fro" },
-  { "fr", "fra" },
-//g
-  { "ga", "gab" },
-  { "gb", "gbr" },
-  { "gd", "grd" },
-  { "ge", "geo" },
-  { "gf", "guf" },
-  { "gg", "ggy" },
-  { "gh", "gha" },
-  { "gi", "gib" },
-  { "gl", "grl" },
-  { "gm", "gmb" },
-  { "gn", "gin" },
-  { "gp", "glp" },
-  { "gq", "gnq" },
-  { "gr", "grc" },
-  { "gs", "sgs" },
-  { "gt", "gtm" },
-  { "gu", "gum" },
-  { "gw", "gnb" },
-  { "gy", "guy" },
-//h
-  { "hk", "hkg" },
-  { "hm", "hmd" },
-  { "hn", "hnd" },
-  { "hr", "hrv" },
-  { "ht", "hti" },
-  { "hu", "hun" },
-//i
-  { "id", "idn" },
-  { "ie", "irl" },
-  { "il", "isr" },
-  { "im", "imn" },
-  { "in", "ind" },
-  { "io", "iot" },
-  { "iq", "irq" },
-  { "ir", "irn" },
-  { "is", "isl" },
-  { "it", "ita" },
-//j
-  { "je", "jey" },
-  { "jm", "jam" },
-  { "jo", "jor" },
-  { "jp", "jpn" },
-//k
-  { "ke", "ken" },
-  { "kg", "kgz" },
-  { "kh", "khm" },
-  { "ki", "kir" },
-  { "km", "com" },
-  { "kn", "kna" },
-  { "kp", "prk" },
-  { "kr", "kor" },
-  { "kw", "kwt" },
-  { "ky", "cym" },
-  { "kz", "kaz" },
-//l
-  { "la", "lao" },
-  { "lb", "lbn" },
-  { "lc", "lca" },
-  { "li", "lie" },
-  { "lk", "lka" },
-  { "lr", "lbr" },
-  { "ls", "lso" },
-  { "lt", "ltu" },
-  { "lu", "lux" },
-  { "lv", "lva" },
-  { "ly", "lby" },
-//m
-  { "ma", "mar" },
-  { "mc", "mco" },
-  { "md", "mda" },
-  { "me", "mne" },
-  { "mf", "maf" },
-  { "mg", "mdg" },
-  { "mh", "mhl" },
-  { "mk", "mkd" },
-  { "ml", "mli" },
-  { "mm", "mmr" },
-  { "mn", "mng" },
-  { "mo", "mac" },
-  { "mp", "mnp" },
-  { "mq", "mtq" },
-  { "mr", "mrt" },
-  { "ms", "msr" },
-  { "mt", "mlt" },
-  { "mu", "mus" },
-  { "mv", "mdv" },
-  { "mw", "mwi" },
-  { "mx", "mex" },
-  { "my", "mys" },
-  { "mz", "moz" },
-//n
-  { "na", "nam" },
-  { "nc", "ncl" },
-  { "ne", "ner" },
-  { "nf", "nfk" },
-  { "ng", "nga" },
-  { "ni", "nic" },
-  { "nl", "nld" },
-  { "no", "nor" },
-  { "np", "npl" },
-  { "nr", "nru" },
-  { "nu", "niu" },
-  { "nz", "nzl" },
-//o
-  { "om", "omn" },
-//p
-  { "pa", "pan" },
-  { "pe", "per" },
-  { "pf", "pyf" },
-  { "pg", "png" },
-  { "ph", "phl" },
-  { "pk", "pak" },
-  { "pl", "pol" },
-  { "pm", "spm" },
-  { "pn", "pcn" },
-  { "pr", "pri" },
-  { "ps", "pse" },
-  { "pt", "prt" },
-  { "pw", "plw" },
-  { "py", "pry" },
-//q
-  { "qa", "qat" },
-//r
-  { "re", "reu" },
-  { "ro", "rou" },
-  { "rs", "srb" },
-  { "ru", "rus" },
-  { "rw", "rwa" },
-//s
-  { "sa", "sau" },
-  { "sb", "slb" },
-  { "sc", "syc" },
-  { "sd", "sdn" },
-  { "se", "swe" },
-  { "sg", "sgp" },
-  { "sh", "shn" },
-  { "si", "svn" },
-  { "sj", "sjm" },
-  { "sk", "svk" },
-  { "sl", "sle" },
-  { "sm", "smr" },
-  { "sn", "sen" },
-  { "so", "som" },
-  { "sr", "sur" },
-  { "ss", "ssd" },
-  { "st", "stp" },
-  { "sv", "slv" },
-  { "sy", "syr" },
-  { "sz", "swz" },
-//t
-  { "tc", "tca" },
-  { "td", "tcd" },
-  { "tf", "atf" },
-  { "tg", "tgo" },
-  { "th", "tha" },
-  { "tj", "tjk" },
-  { "tk", "tkl" },
-  { "tl", "tls" },
-  { "tm", "tkm" },
-  { "tn", "tun" },
-  { "to", "ton" },
-  { "tr", "tur" },
-  { "tt", "tto" },
-  { "tv", "tuv" },
-  { "tw", "twn" },
-  { "tz", "tza" },
-//u
-  { "ua", "ukr" },
-  { "ug", "uga" },
-  { "um", "umi" },
-  { "us", "usa" },
-  { "uy", "ury" },
-  { "uz", "uzb" },
-//v
-  { "va", "vat" },
-  { "vc", "vct" },
-  { "ve", "ven" },
-  { "vg", "vgb" },
-  { "vi", "vir" },
-  { "vn", "vnm" },
-  { "vu", "vut" },
-//w
-  { "wf", "wlf" },
-  { "ws", "wsm" },
-//y
-  { "ye", "yem" },
-  { "yt", "myt" },
-// z
-  { "za", "zaf" },
-  { "zm", "zmb" },
-  { "zw", "zwe" }
+{ "aa", "aar" },
+{ "af", "afr" },
+{ "ak", "aka" },
+{ "am", "amh" },
+{ "ar", "ara" },
+{ "as", "asm" },
+{ "az", "aze" },
+{ "ba", "bak" },
+{ "be", "bel" },
+{ "bg", "bul" },
+{ "bm", "bam" },
+{ "bn", "ben" },
+{ "bo", "bod" },
+{ "br", "bre" },
+{ "bs", "bos" },
+{ "ca", "cat" },
+{ "ce", "che" },
+{ "co", "cos" },
+{ "cs", "ces" },
+{ "cu", "chu" },
+{ "cv", "chv" },
+{ "cy", "cym" },
+{ "da", "dan" },
+{ "de", "deu" },
+{ "dv", "div" },
+{ "dz", "dzo" },
+{ "ee", "ewe" },
+{ "el", "ell" },
+{ "en", "eng" },
+{ "es", "spa" },
+{ "et", "est" },
+{ "eu", "eus" },
+{ "fa", "fas" },
+{ "ff", "ful" },
+{ "fi", "fin" },
+{ "fo", "fao" },
+{ "fr", "fra" },
+{ "fy", "fry" },
+{ "ga", "gle" },
+{ "gd", "gla" },
+{ "gl", "glg" },
+{ "gn", "grn" },
+{ "gu", "guj" },
+{ "gv", "glv" },
+{ "ha", "hau" },
+{ "he", "heb" },
+{ "hi", "hin" },
+{ "hr", "hrv" },
+{ "hu", "hun" },
+{ "hy", "hye" },
+{ "ia", "ina" },
+{ "id", "ind" },
+{ "ig", "ibo" },
+{ "is", "isl" },
+{ "it", "ita" },
+{ "iu", "iku" },
+{ "ja", "jpn" },
+{ "jv", "jav" },
+{ "ka", "kat" },
+{ "ki", "kik" },
+{ "kk", "kaz" },
+{ "kl", "kal" },
+{ "km", "khm" },
+{ "kn", "kan" },
+{ "ko", "kor" },
+{ "ks", "kas" },
+{ "ku", "kur" },
+{ "kw", "cor" },
+{ "ky", "kir" },
+{ "lb", "ltz" },
+{ "lg", "lug" },
+{ "ln", "lin" },
+{ "lo", "lao" },
+{ "lt", "lit" },
+{ "lv", "lav" },
+{ "mg", "mlg" },
+{ "mi", "mri" },
+{ "mi", "mri" },
+{ "mk", "mkd" },
+{ "ml", "mal" },
+{ "mn", "mon" },
+{ "mr", "mar" },
+{ "ms", "msa" },
+{ "mt", "mlt" },
+{ "my", "mya" },
+{ "nb", "nob" },
+{ "ne", "nep" },
+{ "nl", "nld" },
+{ "nn", "nno" },
+{ "no", "nor" },
+{ "ny", "nya" },
+{ "oc", "oci" },
+{ "om", "orm" },
+{ "or", "ori" },
+{ "os", "oss" },
+{ "pa", "pan" },
+{ "pl", "pol" },
+{ "ps", "pus" },
+{ "pt", "por" },
+{ "qu", "que" },
+{ "rm", "roh" },
+{ "rn", "run" },
+{ "ro", "ron" },
+{ "ru", "rus" },
+{ "rw", "kin" },
+{ "sa", "san" },
+{ "sd", "snd" },
+{ "se", "sme" },
+{ "sg", "sag" },
+{ "si", "sin" },
+{ "sk", "slk" },
+{ "sl", "slv" },
+{ "sn", "sna" },
+{ "so", "som" },
+{ "sq", "sqi" },
+{ "sr", "srp" },
+{ "ss", "ssw" },
+{ "st", "sot" },
+{ "sv", "swe" },
+{ "sw", "swa" },
+{ "ta", "tam" },
+{ "te", "tel" },
+{ "tg", "tgk" },
+{ "th", "tha" },
+{ "ti", "tir" },
+{ "tk", "tuk" },
+{ "tl", "tgl" },
+{ "tn", "tsn" },
+{ "to", "ton" },
+{ "tr", "tur" },
+{ "ts", "tso" },
+{ "tt", "tat" },
+{ "ug", "uig" },
+{ "uk", "ukr" },
+{ "ur", "urd" },
+{ "uz", "uzb" },
+{ "ve", "ven" },
+{ "vi", "vie" },
+{ "wa", "wln" },
+{ "wo", "wol" },
+{ "xh", "xho" },
+{ "yo", "yor" },
+{ "zh", "zho" },
+{ "zu", "zul" }
 };
 
 void kiwix::sleep(unsigned int milliseconds)

src/tools/pathTools.cpp

@@ -29,11 +29,6 @@
 #define getcwd _getcwd  // stupid MSFT "deprecation" warning
 #endif
 
-#ifdef _WIN32
-#else
-#include <unistd.h>
-#endif
-
 #ifdef _WIN32
 const std::string SEPARATOR("\\");
 #else
@@ -228,25 +223,20 @@ bool makeDirectory(const string& path)
 string makeTmpDirectory()
 {
 #ifdef _WIN32
-  char cbase[MAX_PATH+1];
-  int base_len = GetTempPath(MAX_PATH+1, cbase);
-  UUID uuid;
-  UuidCreate(&uuid);
-  char* dir_name;
-  UuidToString(&uuid, reinterpret_cast<unsigned char**>(&dir_name));
-  string dir(cbase, base_len);
-  dir += dir_name;
-  _mkdir(dir.c_str());
-  RpcStringFree(reinterpret_cast<unsigned char**>(&dir_name));
+  char cbase[MAX_PATH];
+  char ctmp[MAX_PATH];
+  GetTempPath(MAX_PATH-14, cbase);
+  // This create a file for us, ensure it is unique.
+  // So we need to delete it and create the directory using the same name.
+  GetTempFileName(cbase, "kiwix", 0, ctmp);
+  DeleteFile(ctmp);
+  _mkdir(ctmp);
+  return string(ctmp);
 #else
-  string base = "/tmp";
-  auto _template = base + "/kiwix-lib_XXXXXX";
-  char* _template_array = new char[_template.size()+1];
-  memcpy(_template_array, _template.c_str(), _template.size());
+  char _template_array[] = {"/tmp/kiwix-lib_XXXXXX"};
   string dir = mkdtemp(_template_array);
-  delete[] _template_array;
-#endif
   return dir;
+#endif
 }
 
 /* Try to create a link and if does not work then make a copy */

test/library.cpp

@@ -0,0 +1,241 @@
+/*
+ * Copyright (C) 2013 Tommi Maekitalo
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of the
+ * License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * is provided AS IS, WITHOUT ANY WARRANTY; without even the implied
+ * warranty of MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, and
+ * NON-INFRINGEMENT.  See the GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA
+ *
+ */
+
+#include "gtest/gtest.h"
+#include <string>
+
+
+const char * sampleOpdsStream = R"(
+<feed xmlns="http://www.w3.org/2005/Atom" xmlns:opds="http://opds-spec.org/2010/catalog">
+  <id>00000000-0000-0000-0000-000000000000</id>
+  <entry>
+    <title>Encyclopédie de la Tunisie</title>
+    <id>urn:uuid:0c45160e-f917-760a-9159-dfe3c53cdcdd</id>
+    <icon>/meta?name=favicon&amp;content=wikipedia_fr_tunisie_novid_2018-10</icon>
+    <updated>2018-10-08T00:00::00:Z</updated>
+    <language>fra</language>
+    <summary>Le meilleur de Wikipédia sur la Tunisie</summary>
+    <tags>wikipedia;novid;_ftindex</tags>
+    <link type="text/html" href="/wikipedia_fr_tunisie_novid_2018-10" />
+    <author>
+      <name>Wikipedia</name>
+    </author>
+    <link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="http://download.kiwix.org/zim/wikipedia/wikipedia_fr_tunisie_novid_2018-10.zim.meta4" length="90030080" />
+    <link rel="http://opds-spec.org/image/thumbnail" type="image/png" href="/meta?name=favicon&amp;content=wikipedia_fr_tunisie_novid_2018-10" />
+  </entry>
+  <entry>
+    <title>Tania Louis</title>
+    <id>urn:uuid:0d0bcd57-d3f6-cb22-44cc-a723ccb4e1b2</id>
+    <icon>/meta?name=favicon&amp;content=biologie-tout-compris_fr_all_2018-06</icon>
+    <updated>2018-06-23T00:00::00:Z</updated>
+    <language>fra</language>
+    <summary>Tania Louis videos</summary>
+    <tags>youtube</tags>
+    <link type="text/html" href="/biologie-tout-compris_fr_all_2018-06" />
+    <author>
+      <name>Tania Louis</name>
+    </author>
+    <link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="http://download.kiwix.org/zim/other/biologie-tout-compris_fr_all_2018-06.zim.meta4" length="2172639232" />
+    <link rel="http://opds-spec.org/image/thumbnail" type="image/png" href="/meta?name=favicon&amp;content=biologie-tout-compris_fr_all_2018-06" />
+  </entry>
+  <entry>
+    <title>Wikiquote</title>
+    <id>urn:uuid:0ea1cde6-441d-6c58-f2c7-21c2838e659f</id>
+    <icon>/meta?name=favicon&amp;content=wikiquote_fr_all_nopic_2019-06</icon>
+    <updated>2019-06-05T00:00::00:Z</updated>
+    <language>fra</language>
+    <summary>Une page de Wikiquote, le recueil des citations libres.</summary>
+    <tags>wikiquote;nopic</tags>
+    <link type="text/html" href="/wikiquote_fr_all_nopic_2019-06" />
+    <author>
+      <name>Wikiquote</name>
+    </author>
+    <link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="http://download.kiwix.org/zim/wikiquote/wikiquote_fr_all_nopic_2019-06.zim.meta4" length="21368832" />
+    <link rel="http://opds-spec.org/image/thumbnail" type="image/png" href="/meta?name=favicon&amp;content=wikiquote_fr_all_nopic_2019-06" />
+  </entry>
+  <entry>
+    <title>Géographie par Wikipédia</title>
+    <id>urn:uuid:1123e574-6eef-6d54-28fc-13e4caeae474</id>
+    <icon>/meta?name=favicon&amp;content=wikipedia_fr_geography_nopic_2019-06</icon>
+    <updated>2019-06-02T00:00::00:Z</updated>
+    <summary>Une sélection d'articles de Wikipédia sur la géographie</summary>
+    <language>fra</language>
+    <tags>wikipedia;nopic</tags>
+    <link type="text/html" href="/wikipedia_fr_geography_nopic_2019-06" />
+    <author>
+      <name>Wikipedia</name>
+    </author>
+    <link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="http://download.kiwix.org/zim/wikipedia/wikipedia_fr_geography_nopic_2019-06.zim.meta4" length="157586432" />
+    <link rel="http://opds-spec.org/image/thumbnail" type="image/png" href="/meta?name=favicon&amp;content=wikipedia_fr_geography_nopic_2019-06" />
+  </entry>
+  <entry>
+    <title>Mathématiques</title>
+    <id>urn:uuid:14829621-c490-c376-0792-9de558b57efa</id>
+    <icon>/meta?name=favicon&amp;content=wikipedia_fr_mathematics_nopic_2019-05</icon>
+    <updated>2019-05-13T00:00::00:Z</updated>
+    <language>fra</language>
+    <summary>Une</summary>
+    <tags>wikipedia;nopic</tags>
+    <link type="text/html" href="/wikipedia_fr_mathematics_nopic_2019-05" />
+    <author>
+      <name>Wikipedia</name>
+    </author>
+    <link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="http://download.kiwix.org/zim/wikipedia/wikipedia_fr_mathematics_nopic_2019-05.zim.meta4" length="223368192" />
+    <link rel="http://opds-spec.org/image/thumbnail" type="image/png" href="/meta?name=favicon&amp;content=wikipedia_fr_mathematics_nopic_2019-05" />
+  </entry>
+<entry>
+    <title>Granblue Fantasy Wiki</title>
+    <id>urn:uuid:006cbd1b-16d8-b00d-a584-c1ae110a94ed</id>
+    <icon>/meta?name=favicon&amp;content=granbluefantasy_en_all_all_nopic_2018-10</icon>
+    <updated>2018-10-14T00:00::00:Z</updated>
+    <language>eng</language>
+    <summary>Granblue Fantasy Wiki</summary>
+    <tags>gbf;nopic;_ftindex</tags>
+    <link type="text/html" href="/granbluefantasy_en_all_all_nopic_2018-10" />
+    <author>
+      <name>Wiki</name>
+    </author>
+    <link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="http://download.kiwix.org/zim/other/granbluefantasy_en_all_all_nopic_2018-10.zim.meta4" length="23197696" />
+    <link rel="http://opds-spec.org/image/thumbnail" type="image/png" href="/meta?name=favicon&amp;content=granbluefantasy_en_all_all_nopic_2018-10" />
+  </entry>
+  <entry>
+    <title>Movies &amp; TV Stack Exchange</title>
+    <id>urn:uuid:00f37b00-f4da-0675-995a-770f9c72903e</id>
+    <icon>/meta?name=favicon&amp;content=movies.stackexchange.com_en_all_2019-02</icon>
+    <updated>2019-02-03T00:00::00:Z</updated>
+    <language>eng</language>
+    <summary>Q&amp;A for movie and tv enthusiasts</summary>
+    <tags>stackexchange;_ftindex</tags>
+    <link type="text/html" href="/movies.stackexchange.com_en_all_2019-02" />
+    <author>
+      <name>Movies &amp; TV Stack Exchange</name>
+    </author>
+    <link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="http://download.kiwix.org/zim/stack_exchange/movies.stackexchange.com_en_all_2019-02.zim.meta4" length="859463680" />
+    <link rel="http://opds-spec.org/image/thumbnail" type="image/png" href="/meta?name=favicon&amp;content=movies.stackexchange.com_en_all_2019-02" />
+  </entry>
+  <entry>
+    <title>TED talks - Business</title>
+    <id>urn:uuid:0189d9be-2fd0-b4b6-7300-20fab0b5cdc8</id>
+    <icon>/meta?name=favicon&amp;content=ted_en_business_2018-07</icon>
+    <updated>2018-07-23T00:00::00:Z</updated>
+    <language>eng</language>
+    <summary>Ideas worth spreading</summary>
+    <tags></tags>
+    <link type="text/html" href="/ted_en_business_2018-07" />
+    <author>
+      <name>TED</name>
+    </author>
+    <link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="http://download.kiwix.org/zim/ted/ted_en_business_2018-07.zim.meta4" length="8855827456" />
+    <link rel="http://opds-spec.org/image/thumbnail" type="image/png" href="/meta?name=favicon&amp;content=ted_en_business_2018-07" />
+  </entry>
+  <entry>
+    <title>Mythology &amp; Folklore Stack Exchange</title>
+    <id>urn:uuid:028055ac-4acc-1d54-65e0-a96de45e1b22</id>
+    <icon>/meta?name=favicon&amp;content=mythology.stackexchange.com_en_all_2019-02</icon>
+    <updated>2019-02-03T00:00::00:Z</updated>
+    <language>eng</language>
+    <summary>Q&amp;A for enthusiasts and scholars of mythology and folklore</summary>
+    <tags>stackexchange;_ftindex</tags>
+    <link type="text/html" href="/mythology.stackexchange.com_en_all_2019-02" />
+    <author>
+      <name>Mythology &amp; Folklore Stack Exchange</name>
+    </author>
+    <link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="http://download.kiwix.org/zim/stack_exchange/mythology.stackexchange.com_en_all_2019-02.zim.meta4" length="47005696" />
+    <link rel="http://opds-spec.org/image/thumbnail" type="image/png" href="/meta?name=favicon&amp;content=mythology.stackexchange.com_en_all_2019-02" />
+  </entry>
+  <entry>
+    <title>Islam Stack Exchange</title>
+    <id>urn:uuid:02e9c7ff-36fc-9c6e-6ac7-cd7085989029</id>
+    <icon>/meta?name=favicon&amp;content=islam.stackexchange.com_en_all_2019-01</icon>
+    <updated>2019-01-31T00:00::00:Z</updated>
+    <language>eng</language>
+    <summary>Q&amp;A for Muslims, experts in Islam, and those interested in learning more about Islam</summary>
+    <tags>stackexchange;_ftindex</tags>
+    <link type="text/html" href="/islam.stackexchange.com_en_all_2019-01" />
+    <author>
+      <name>Islam Stack Exchange</name>
+    </author>
+    <link rel="http://opds-spec.org/acquisition/open-access" type="application/x-zim" href="http://download.kiwix.org/zim/stack_exchange/islam.stackexchange.com_en_all_2019-01.zim.meta4" length="135346176" />
+    <link rel="http://opds-spec.org/image/thumbnail" type="image/png" href="/meta?name=favicon&amp;content=islam.stackexchange.com_en_all_2019-01" />
+  </entry>
+</feed>
+
+)";
+
+#include "../include/library.h"
+#include "../include/manager.h"
+
+namespace
+{
+
+class LibraryTest : public ::testing::Test {
+ protected:
+  void SetUp() override {
+     kiwix::Manager manager(&lib);
+     manager.readOpds(sampleOpdsStream, "foo.urlHost");
+  }
+
+  kiwix::Library lib;
+};
+
+TEST_F(LibraryTest, sanityCheck)
+{
+  EXPECT_EQ(lib.getBookCount(true, true), 10U);
+  EXPECT_EQ(lib.getBooksLanguages().size(), 2U);
+  EXPECT_EQ(lib.getBooksCreators().size(), 8U);
+  EXPECT_EQ(lib.getBooksPublishers().size(), 1U);
+}
+
+TEST_F(LibraryTest, filterCheck)
+{
+  auto bookIds = lib.filter(kiwix::Filter());
+  EXPECT_EQ(bookIds, lib.getBooksIds());
+
+  bookIds = lib.filter(kiwix::Filter().lang("eng"));
+  EXPECT_EQ(bookIds.size(), 5U);
+
+  bookIds = lib.filter(kiwix::Filter().acceptTags({"stackexchange"}));
+  EXPECT_EQ(bookIds.size(), 3U);
+
+  bookIds = lib.filter(kiwix::Filter().acceptTags({"wikipedia"}));
+  EXPECT_EQ(bookIds.size(), 3U);
+
+  bookIds = lib.filter(kiwix::Filter().acceptTags({"wikipedia", "nopic"}));
+  EXPECT_EQ(bookIds.size(), 2U);
+
+  bookIds = lib.filter(kiwix::Filter().acceptTags({"wikipedia"}).rejectTags({"nopic"}));
+  EXPECT_EQ(bookIds.size(), 1U);
+
+  bookIds = lib.filter(kiwix::Filter().query("folklore"));
+  EXPECT_EQ(bookIds.size(), 1U);
+
+  bookIds = lib.filter(kiwix::Filter().query("Wiki"));
+  EXPECT_EQ(bookIds.size(), 3U);
+
+  bookIds = lib.filter(kiwix::Filter().query("Wiki").creator("Wiki"));
+  EXPECT_EQ(bookIds.size(), 1U);
+
+}
+};
+
+int main(int argc, char** argv)
+{
+  ::testing::InitGoogleTest(&argc, argv);
+  return RUN_ALL_TESTS();
+}

test/meson.build

@@ -1,7 +1,8 @@
 
 
 tests = [
-    'parseUrl'
+    'parseUrl',
+    'library'
 ]
 
 

travis/install_deps.sh

@@ -3,25 +3,26 @@
 set -e
 
 REPO_NAME=${TRAVIS_REPO_SLUG#*/}
-ARCHIVE_NAME=deps_${TRAVIS_OS_NAME}_${PLATFORM}_${REPO_NAME}.tar.gz
 
 # Ninja
 cd $HOME
 if [[ "$TRAVIS_OS_NAME" == "osx" ]]
 then
-  pip3 install meson
+  pip3 install meson==0.49.2
 
   wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-mac.zip
   unzip ninja-mac.zip ninja
+  ARCHIVE_NAME=deps_osx_${PLATFORM}_${REPO_NAME}.tar.xz
 else
   wget https://bootstrap.pypa.io/get-pip.py
   python3.5 get-pip.py --user
 
   python3.5 -m pip install --user --upgrade pip
-  python3.5 -m pip install --user meson
+  python3.5 -m pip install --user meson==0.49.2
 
   wget https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip
   unzip ninja-linux.zip ninja
+  ARCHIVE_NAME=deps_linux_xenial_${PLATFORM}_${REPO_NAME}.tar.xz
 fi
 
 mkdir -p $HOME/bin
@@ -31,3 +32,4 @@ cp ninja $HOME/bin
 cd ${HOME}
 wget http://tmp.kiwix.org/ci/${ARCHIVE_NAME}
 tar xf ${HOME}/${ARCHIVE_NAME}
+sudo ln -s travis ../ci_builder