Adapt kiwix-tools to new libkiwix API.

.github/workflows/ci.yml

@@ -14,10 +14,10 @@ jobs:
           - win32_dyn
         include:
           - target: native_static
-            image_variant: focal
+            image_variant: bionic
             lib_postfix: '/x86_64-linux-gnu'
           - target: native_dyn
-            image_variant: focal
+            image_variant: bionic
             lib_postfix: '/x86_64-linux-gnu'
           - target: win32_static
             image_variant: f35
@@ -37,10 +37,11 @@ jobs:
       id: extract_branch
     - name: Checkout code
       uses: actions/checkout@v3
-    - name: Install dependencies
-      uses: kiwix/kiwix-build/actions/dl_deps_archive@main
-      with:
-        target_platform: ${{ matrix.target }}
+    - name: Install deps
+      shell: bash
+      run: |
+        ARCHIVE_NAME=deps2_${OS_NAME}_${{matrix.target}}_kiwix-tools.tar.xz
+        wget -O- http://tmp.kiwix.org/ci/${ARCHIVE_NAME} | tar -xJ -C ${{env.HOME}}
     - name: Compile
       shell: bash
       run: |

.github/workflows/package.yml

@@ -8,6 +8,7 @@ jobs:
       fail-fast: false
       matrix:
         distro:
+          - ubuntu-kinetic
           - ubuntu-jammy
           - ubuntu-focal
     steps:
@@ -40,6 +41,14 @@ jobs:
           args: --no-sign
           ppa: ${{ steps.ppa.outputs.ppa }}
 
+      - uses: legoktm/gh-action-build-deb@ubuntu-kinetic
+        if: matrix.distro == 'ubuntu-kinetic'
+        name: Build package for ubuntu-kinetic
+        id: build-ubuntu-kinetic
+        with:
+          args: --no-sign
+          ppa: ${{ steps.ppa.outputs.ppa }}
+
       - uses: legoktm/gh-action-build-deb@ubuntu-focal
         if: matrix.distro == 'ubuntu-focal'
         name: Build package for ubuntu-focal

.readthedocs.yaml

@@ -1,21 +0,0 @@
-# Read the Docs configuration file
-# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
-
-# Required
-version: 2
-
-# Set the version of Python and other tools you might need
-build:
-  os: ubuntu-22.04
-  tools:
-    python: "3.11"
-
-# Build documentation in the docs/ directory with Sphinx
-sphinx:
-  configuration: docs/conf.py
-
-# We recommend specifying your dependencies to enable reproducible builds:
-# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
-python:
-  install:
-  - requirements: docs/requirements.txt

Changelog

@@ -1,17 +1,7 @@
-kiwix-tools 3.7.0
-=================
+Unreleased
+==========
 
- * Fixed ZIM name vs Book name confusion in documentation (@veloman-yunkan #663)
- * Fixes compilation dependencies to rely on appropriate version (@kelson42 #667)
- * New --skipInvalid Kiwix Server command line option (@schuellerf @kelson42 #666)
-
-kiwix-tools 3.6.0
-=================
-
- * Improved kiwix-serve man page (@iArchitSharma #626)
- * C++17 compliant code base (@mgautierfr #636)
- * Support of libkiwix13 (@mgautierfr #633)
- * Additional docker images archs for armv6 and i386 (@rgaudin #622)
+ * Additional docker images archs for armv6 and i386.
 
 kiwix-tools 3.5.0
 =================

README.md

@@ -4,7 +4,7 @@ Kiwix tools
 The Kiwix tools is a collection of [Kiwix](https://kiwix.org) related
 command line tools:
 * kiwix-manage: Manage XML based library of ZIM files
-* kiwix-search: Full text search in ZIM files
+* kiwix-search: Fulltext search in ZIM files
 * kiwix-serve: HTTP daemon serving ZIM files
 
 [![latest release](https://img.shields.io/github/v/tag/kiwix/kiwix-tools?label=latest%20release&sort=semver)](https://download.kiwix.org/release/kiwix-tools/)
@@ -29,8 +29,8 @@ Preamble
 --------
 
 Although the Kiwix tools can be compiled/cross-compiled on/for many
-systems, the following documentation explains how to do it on POSIX
-ones. It is primarily thought for GNU/Linux systems and has been tested
+sytems, the following documentation explains how to do it on POSIX
+ones. It is primarly thought for GNU/Linux systems and has been tested
 on recent releases of
 [Debian](https://debian.org)/[Ubuntu](https://ubuntu.com) and
 [Fedora](https://getfedora.org).
@@ -62,7 +62,7 @@ Environment
 -------------
 
 The Kiwix tools build using [Meson](http://mesonbuild.com/) version
-0.43 or higher. Meson relies itself on Ninja, pkg-config and a few other
+0.43 or higher. Meson relies itself on Ninja, pkg-config and few other
 compilation tools. Install them first:
 * [Meson](http://mesonbuild.com/)
 * [Ninja](https://ninja-build.org/)
@@ -96,7 +96,7 @@ ninja -C build install
 ```
 
 You might need to run the command as `root` (or using `sudo`),
-depending on where you want to install the Kiwix tools. After the
+depending where you want to install the Kiwix tools. After the
 installation succeeded, you may need to run ldconfig (as `root`).
 
 Uninstallation

debian/control

@@ -3,8 +3,8 @@ Section: utils
 Priority: optional
 Maintainer: Kiwix team <kiwix@kiwix.org>
 Build-Depends: debhelper-compat (= 13),
-               libzim-dev (>= 9.0.0), libzim-dev (<< 10.0.0),
-               libkiwix-dev (>= 13.0.0), libkiwix-dev (<< 14.0.0),
+               libkiwix-dev (>= 10.0.0~),
+               libzim-dev (>= 7.2.0~),
                meson,
                pkg-config,
 Standards-Version: 4.5.0

docker/server/README.md

@@ -42,12 +42,7 @@ Build an image for an ARM based GNU/Linux:
 docker build . -t ghcr.io/kiwix/kiwix-serve:latest --build-arg ARCH="arm32v7/"
 ```
 
-Docker Compose
---------------
-
-You can also deploy kiwix with
-[`docker-compose`](https://docs.docker.com/compose/). Check out a
-sample at [docker-compose.yml.example](docker-compose.yml.example).
+You can also deploy kiwix with [`docker-compose`](https://docs.docker.com/compose/). Check out a sample at [docker-compose.yml.example](docker-compose.yml.example)
 
 Screenshots
 -----------

docs/conf.py

@@ -18,7 +18,7 @@ import os
 # -- Project information -----------------------------------------------------
 
 project = 'kiwix-tools'
-copyright = '2024, kiwix-team'
+copyright = '2022, kiwix-team'
 author = 'kiwix-team'
 
 
@@ -30,7 +30,6 @@ on_rtd = os.environ.get('READTHEDOCS', None) == 'True'
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-        "sphinx_rtd_theme"
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -42,7 +41,8 @@ templates_path = ['_templates']
 exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
 
 
-html_theme = 'sphinx_rtd_theme'
+if not on_rtd:
+    html_theme = 'sphinx_rtd_theme'
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,

docs/kiwix-serve.rst

@@ -339,8 +339,8 @@ filtering criteria must be specified via the following URL parameters:
 * ``q`` - include in the results only entries that contain the specified text
   in the title or description.
 
-* ``name`` - include in the results only entries with a matching
-  :term:`book name <Book name>`.
+* ``name`` - include in the results only the entry with the specified
+  :term:`name <ZIM name>`.
 
 
 Examples:
@@ -532,7 +532,7 @@ specified ZIM file.
 
 **Parameters:**
 
-  ``content``: :term:`name of the ZIM file <ZIM name>`.
+  ``content``: :term:`name <ZIM name>` of the ZIM file.
 
 
 .. _raw:
@@ -609,7 +609,7 @@ A multi-ZIM search request must comply with the following constraints:
     below list take precedence over subsequent ones (the later ones, even if
     present in the request, are simply ignored).
 
-    ``content``: :term:`name of the ZIM file <ZIM name>` (for a single-ZIM
+    ``content``: :term:`name <ZIM name>` of the ZIM file (for a single-ZIM
     search). This is a :ref:`legacy parameter <deprecation>`. ``books.name``
     should be used instead.
 
@@ -617,23 +617,9 @@ A multi-ZIM search request must comply with the following constraints:
     a multi-ZIM search, however must respect the :ref:`multi-ZIM search
     constraints <multi-zim-search-constraints>`.
 
-      .. note::
-
-        If any of the provided ``books.id`` values refers to a book missing
-        from the library then an error is returned instead of running the
-        search on the remaining (valid) entries.
-
-    ``books.name``: :term:`name of the ZIM file <ZIM name>` (not to be confused
-    with ``books.filter.name`` which selects/filters based on the :term:`book
-    name <Book name>`). Can be repeated for a multi-ZIM search, however must
-    respect the :ref:`multi-ZIM search constraints
-    <multi-zim-search-constraints>`.
-
-      .. note::
-
-        If any of the provided ``books.name`` values refers to a book missing
-        from the library then an error is returned instead of running the
-        search on the remaining (valid) entries.
+    ``books.name``: :term:`name <ZIM name>` of the ZIM file. Can be repeated
+    for a multi-ZIM search, however must respect the :ref:`multi-ZIM search
+    constraints <multi-zim-search-constraints>`.
 
     ``books.filter.{criteria}``: allows to take full advantage of :ref:`library
     filtering <library-filtering>` functionality of the `/catalog/v2/entries`_
@@ -753,7 +739,7 @@ added as an option to perform a full text search in the said ZIM file.
 
 **Parameters:**
 
-  ``content`` (mandatory): :term:`name of the ZIM file <ZIM name>`.
+  ``content`` (mandatory): :term:`name <ZIM name>` of the ZIM file.
 
   ``term`` (optional; defaults to an empty string): query text.
 
@@ -813,21 +799,6 @@ Glossary
 
 .. glossary::
 
-  Book name
-
-    Name of the book as specified in the ZIM file metadata (for a
-    ``kiwix-serve`` started *WITHOUT* the :option:`--library` option) or the
-    library XML file (for a ``kiwix-serve`` started with the
-    :option:`--library` option).
-
-    .. note::
-
-      Two or more books may have the same name in the library. That's not
-      considered a conflict, because there may be multiple versions of the
-      "same" book (differing by the settings of the scraper, date, etc).
-      :ref:`Library filtering <library-filtering>` by name will return all
-      matching books.
-
   ZIM filename
 
     Name of a ZIM file on the server filesystem.
@@ -837,22 +808,20 @@ Glossary
     Identifier of a ZIM file in the server's library (used for referring to a
     particular ZIM file in requests).
 
-    ZIM names are derived from the filenames as follows:
+    For a ``kiwix-serve`` started with a list of ZIM files, ZIM names are
+    derived from the filename by dropping the extension and replacing certain
+    characters (spaces are replaced with underscores, and ``+`` symbols are
+    replaced with the text ``plus``). Presence of the
+    :option:`-z`/:option:`--nodatealiases` option will create additional names
+    (aliases) for filenames with dates.
 
-    - file extension is removed,
-    - all characters are converted to lowercase,
-    - diacritics are removed,
-    - spaces are replaced with underscores,
-    - ``+`` symbols are replaced with the text ``plus``.
-
-    Presence of the :option:`-z`/:option:`--nodatealiases` option will create
-    additional names (aliases) for filenames with dates.
+    For a ``kiwix-serve`` started with the :option:`--library` option, ZIM
+    names come from the library XML file.
 
     ZIM names are expected to be unique across the library. Any name conflicts
     (including those caused by the usage of the
     :option:`-z`/:option:`--nodatealiases` option) are reported on STDERR but,
-    otherwise, are ignored (i.e. only one of the entries can be accessed via
-    the conflicting name).
+    otherwise, are ignored.
 
   ZIM title
 

meson.build

@@ -1,7 +1,7 @@
 project('kiwix-tools', 'cpp',
-  version : '3.7.0',
+  version : '3.5.0',
   license : 'GPL',
-  default_options: ['c_std=c11', 'cpp_std=c++17', 'werror=true'])
+  default_options: ['c_std=c11', 'cpp_std=c++11', 'werror=true'])
 
 compiler = meson.get_compiler('cpp')
 
@@ -16,10 +16,8 @@ if static_linkage
 endif
 
 thread_dep = dependency('threads')
-libzim_dep = dependency('libzim', version:'>=9.0.0', static:static_linkage)
-libzim_dep = dependency('libzim', version:'<10.0.0', static:static_linkage)
-kiwixlib_dep = dependency('kiwix', version:'>=13.0.0', static:static_linkage)
-kiwixlib_dep = dependency('kiwix', version:'<14.0.0', static:static_linkage)
+kiwixlib_dep = dependency('kiwix', version:'>=12.0.0', static:static_linkage)
+libzim_dep = dependency('libzim', version:'>=8.1.0', static:static_linkage)
 
 all_deps = [thread_dep, kiwixlib_dep, libzim_dep]
 

src/man/kiwix-serve.1

@@ -88,10 +88,6 @@ Override the welcome page with a custom HTML file.
 \fB-L N, --ipConnectionLimit=N\fR
 Max number of (concurrent) connections per IP (default: infinite, recommended: >= 6).
 
-.TP
-\fB-k, --skipInvalid\fR
-Startup even when ZIM files are invalid (those will be skipped)
-
 .TP
 \fB-v, --verbose\fR
 Print debug log to STDOUT.

src/manager/kiwix-manage.cpp

@@ -113,7 +113,7 @@ int handle_show(const kiwix::Library& library, const std::string& libraryPath,
   return(0);
 }
 
-int handle_add(kiwix::LibraryPtr library, const std::string& libraryPath,
+int handle_add(std::shared_ptr<kiwix::Library> library, const std::string& libraryPath,
                 int argc, char* argv[])
 {
   string zimPath;
@@ -182,11 +182,11 @@ int handle_add(kiwix::LibraryPtr library, const std::string& libraryPath,
   return(resultCode);
 }
 
-int handle_remove(kiwix::Library& library, const std::string& libraryPath,
+int handle_remove(std::shared_ptr<kiwix::Library> library, const std::string& libraryPath,
                    int argc, char* argv[])
 {
   std::string bookId;
-  const unsigned int totalBookCount = library.getBookCount(true, true);
+  const unsigned int totalBookCount = library->getBookCount(true, true);
   int exitCode = 0;
 
   if (argc <= 3) {
@@ -203,7 +203,7 @@ int handle_remove(kiwix::Library& library, const std::string& libraryPath,
   for (int i = 3; i<argc; i++) {
     bookId = argv[i];
 
-    if (!library.removeBookById(bookId)) {
+    if (!library->removeBookById(bookId)) {
       std::cerr << "Invalid book id '" << bookId << "'." << std::endl;
       exitCode = 1;
     }
@@ -216,7 +216,7 @@ int main(int argc, char** argv)
 {
   string libraryPath = "";
   supportedAction action = NONE;
-  auto library = kiwix::Library::create();
+  auto library = std::make_shared<kiwix::Library>();
 
   /* General argument parsing */
   static struct option long_options[] = {
@@ -279,7 +279,7 @@ int main(int argc, char** argv)
       exitCode = handle_add(library, libraryPath, argc, argv);
       break;
     case REMOVE:
-      exitCode = handle_remove(*library, libraryPath, argc, argv);
+      exitCode = handle_remove(library, libraryPath, argc, argv);
       break;
     case NONE:
       break;

src/server/kiwix-serve.cpp

@@ -77,7 +77,6 @@ void usage()
             << "\t-z, --nodatealiases\tCreate URL aliases for each content by removing the date" << std::endl
             << "\t-c, --customIndex\tAdd path to custom index.html for welcome page" << std::endl
             << "\t-L, --ipConnectionLimit\tMax number of (concurrent) connections per IP (default: infinite, recommended: >= 6)" << std::endl
-            << "\t-k, --skipInvalid\tStartup even when ZIM files are invalid (those will be skipped)" << std::endl
             << std::endl
 
             << "Documentation:" << std::endl
@@ -198,7 +197,7 @@ int main(int argc, char** argv)
 #endif
 
   std::string rootLocation = "/";
-  auto library = kiwix::Library::create();
+  auto library = std::make_shared<kiwix::Library>();
   unsigned int nb_threads = DEFAULT_THREADS;
   std::vector<std::string> zimPathes;
   std::string libraryPath;
@@ -218,7 +217,6 @@ int main(int argc, char** argv)
   unsigned int PPID = 0;
   int ipConnectionLimit = 0;
   int searchLimit = 0;
-  bool skipInvalid = false;
 
   static struct option long_options[]
       = {{"daemon", no_argument, 0, 'd'},
@@ -239,7 +237,6 @@ int main(int argc, char** argv)
          {"monitorLibrary", no_argument, 0, 'M'},
          {"ipConnectionLimit", required_argument, 0, 'L'},
          {"searchLimit", required_argument, 0, 's'},
-         {"skipInvalid", no_argument, 0, 'k'},
          {0, 0, 0, 0}};
 
   std::set<int> usedOptions;
@@ -310,9 +307,6 @@ int main(int argc, char** argv)
         case 's':
           searchLimit = atoi(optarg);
           break;
-        case 'k':
-          skipInvalid = true;
-          break;
         case '?':
           usage();
           return 2;
@@ -354,13 +348,9 @@ int main(int argc, char** argv)
     std::vector<std::string>::iterator it;
     for (it = zimPathes.begin(); it != zimPathes.end(); it++) {
       if (!manager.addBookFromPath(*it, *it, "", false)) {
-        if (skipInvalid) {
-          std::cerr << "Skipping invalid '" << *it << "' ...continuing" << std::endl;
-        } else {
-          std::cerr << "Unable to add the ZIM file '" << *it
-               << "' to the internal library." << std::endl;
-          exit(1);
-        }
+        std::cerr << "Unable to add the ZIM file '" << *it
+             << "' to the internal library." << std::endl;
+        exit(1);
       }
     }
   }
@@ -387,7 +377,7 @@ int main(int argc, char** argv)
 #endif
 
   auto nameMapper = std::make_shared<kiwix::UpdatableNameMapper>(library, noDateAliasesFlag);
-  kiwix::Server server(library, nameMapper);
+  kiwix::Server::Configuration configuration(library, nameMapper);
 
   if (!customIndexPath.empty()) {
     try {
@@ -398,17 +388,18 @@ int main(int argc, char** argv)
     }
   }
 
-  server.setAddress(address);
-  server.setRoot(rootLocation);
-  server.setPort(serverPort);
-  server.setNbThreads(nb_threads);
-  server.setVerbose(isVerboseFlag);
-  server.setTaskbar(!noSearchBarFlag, !noLibraryButtonFlag);
-  server.setBlockExternalLinks(blockExternalLinks);
-  server.setIndexTemplateString(indexTemplateString);
-  server.setIpConnectionLimit(ipConnectionLimit);
-  server.setMultiZimSearchLimit(searchLimit);
+  configuration.setAddress(address);
+  configuration.setRoot(rootLocation);
+  configuration.setPort(serverPort);
+  configuration.setNbThreads(nb_threads);
+  configuration.setVerbose(isVerboseFlag);
+  configuration.setTaskbar(!noSearchBarFlag, !noLibraryButtonFlag);
+  configuration.setBlockExternalLinks(blockExternalLinks);
+  configuration.setIndexTemplateString(indexTemplateString);
+  configuration.setIpConnectionLimit(ipConnectionLimit);
+  configuration.setMultiZimSearchLimit(searchLimit);
 
+  kiwix::Server server(configuration);
   if (! server.start()) {
     exit(1);
   }