Update text files for 4.0.0Alpha1

* Translate ascii control chars below value 32

* Try to make code and tests consistent

* More test fixing

* Delete too much

* Different approach

* Finally got it?

* Start from 0

* Convert \0 to _ for all systems

* Check if CH_ILLEGAL_WIN is translated to CH_LEGAL_WIN

* Test specific chars

.github/workflows/build_release.yml

@@ -87,7 +87,7 @@ jobs:
       # We need the official Python, because the GA ones only support newer macOS versions
       # The deployment target is picked up by the Python build tools automatically
       # If updated, make sure to also set LSMinimumSystemVersion in SABnzbd.spec
-      PYTHON_VERSION: "3.11.1"
+      PYTHON_VERSION: "3.11.2"
       MACOSX_DEPLOYMENT_TARGET: "10.9"
       # We need to force compile for universal2 support
       CFLAGS: -arch x86_64 -arch arm64

.github/workflows/integration_testing.yml

@@ -20,7 +20,7 @@ jobs:
             builder/SABnzbd.spec
             tests
             --line-length=120
-            --target-version=py37
+            --target-version=py38
             --check
             --diff
 
@@ -31,7 +31,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
         os: [ubuntu-20.04]
         include:
           - name: macOS

INSTALL.txt

@@ -52,7 +52,7 @@ Specific guides to install from source are available for Windows and macOS:
     https://sabnzbd.org/wiki/installation/install-macos
     https://sabnzbd.org/wiki/installation/install-from-source-windows
 
-Only Python 3.7 and above is supported.
+Only Python 3.8 and above is supported.
 
 On Linux systems you need to install:
     par2 unrar unzip python3-setuptools python3-pip

PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 1.0
 Name: SABnzbd
-Version: 3.7.2
-Summary: SABnzbd-3.7.2
+Version: 4.0.0Alpha1
+Summary: SABnzbd-4.0.0Alpha1
 Home-page: https://sabnzbd.org
 Author: The SABnzbd Team
 Author-email: team@sabnzbd.org

README.md

@@ -16,7 +16,7 @@ If you want to know more you can head over to our website: https://sabnzbd.org.
 
 SABnzbd has a few dependencies you'll need before you can get running. If you've previously run SABnzbd from one of the various Linux packages, then you likely already have all the needed dependencies. If not, here's what you're looking for:
 
-- `python` (Python 3.7 and above, often called `python3`)
+- `python` (Python 3.8 and above, often called `python3`)
 - Python modules listed in `requirements.txt`. Install with `python3 -m pip install -r requirements.txt -U`
 - `par2` (Multi-threaded par2 installation guide can be found [here](https://sabnzbd.org/wiki/installation/multicore-par2))
 - `unrar` (make sure you get the "official" non-free version of unrar)

README.mkd

@@ -1,60 +1,24 @@
-Release Notes - SABnzbd 3.7.2
+Release Notes - SABnzbd 4.0.0 Alpha 1
 =========================================================
 
-## Bugfixes and changes since 3.7.1
-- Ignore permissions inside archives during unpacking by UnRar.
-- Improvements to connection error messages.
-- Apply other changes only after updating the `Category` in multi-edit.
-- Categories were not sorted correctly in dropdowns.
-- Prevent crash when `Automatically sort queue` was enabled.
-- Apply `History Retention` setting during startup.
-- Tweaks to download performance.
-- Linux: Update appstream metadata.
+## Changes since 3.7.2
+- In this major update we replaced a core part of Python's SSL handling
+  with our own improved version. This results in large performance increases
+  when downloading from news servers with SSL enabled.
+  In addition, the general connection handling was overhauled, resulting in
+  performance improvements for all news servers.
+  Special thanks to: mnightingale, puzzledsab and animetosho!
+- When adding a new news server, SSL is enabled by default.
+- File assembly performance significantly improved by relying on the
+  CRC32 instead of the MD5 to perform QuickCheck of files.
+- Slowdown more gracefully when the cache fills up.
+- HTTPS files are included in the `Backup`.
+- Improved `Watched Folder` scanning and processing.
+- Dropped support for Python 3.7.
 
-## Bugfixes and changes since 3.7.0
-- Minor improvements in download performance.
-- Scripts set `On queue finish` are no longer persistent by default.
-- Improved `Test Server` to handle more failure cases.
-- Priority list in `Add NZB`-window was missing `Paused` priority.
-- Keyboard shortcuts did not work if not in Tabbed-mode.
-- Keyboard shortcut `S` did not reload status information.
-- In `history` API-call the `stage_log` could be empty.
-- Using the `-` character broke the queue/history search.
-- Improved detection and handling of stuck jobs.
-
-## Changes since 3.6.1
-- The queue and history can be filtered using keywords:
-  `cat` and `priority`. For example: `show name cat:tv`.
-- Use shortcut `shift + arrow-key` to navigate the queue/history pages.
-- The backup is now created in a local folder for security.
-- Recurring backups can be configured using the scheduler.
-- Improvements to Deobfuscate Final Filenames.
-- RSS overview shows the rule that accepted the job.
-- Added option to sort the queue by `% downloaded`.
-- Added option to replace underscores with dots in folder names.
-- SABnzbd Host input will be validated before being applied.
-- Moved system load information from the main page to the Status window.
-- Console logging is now written to `stdout` instead of `stderr`.
-- Removed Special settings `enable_meta`, `disable_key`,
-  `replace_illegal`, `osx_speed` and `show_sysload`.
-- Merged Special settings `win_menu` and `osx_menu` into `tray_icon`.
-- macOS/Windows: Use Python 3.11, slightly boosting overall performance.
-- macOS/Windows: Updated UnRar to 6.12.
-- Windows: Updated MultiPar to 1.3.2.5.
-
-# API changes since 3.6.1
-- Minor improvements in API performance.
-- Removed fields `scripts` and `categories` from `queue` API call.
-- Moved `loadavg` from `queue` to `status` API call.
-
-# Bugfixes since 3.6.1
-- Free Space Detection was too strict when using Direct Unpack.
-- File uploads with special characters would be parsed incorrectly.
-- Passwords from NZB meta-data were tried multiple times.
-- Passwords were not always supplied to the pre-queue script.
-- RSS-feed names were not sanitized when renamed.
-- Make sure short-dates are detected as `YY-MM-DD` in Sorting.
-- Show the custom job name in History when the NZB could not be fetched.
+# Bugfixes since 3.7.2
+- Restore applying `History Retention` setting at startup.
+- Windows: Not all invalid characters were removed from filenames.
 
 ## Upgrade notices
 - The download statistics file `totals10.sab` is updated in 3.2.x

SABnzbd.py

@@ -17,8 +17,8 @@
 
 import sys
 
-if sys.hexversion < 0x03070000:
-    print("Sorry, requires Python 3.7 or above")
+if sys.hexversion < 0x03080000:
+    print("Sorry, requires Python 3.8 or above")
     print("You can read more at: https://sabnzbd.org/wiki/installation/install-off-modules")
     sys.exit(1)
 
@@ -40,6 +40,7 @@ import gc
 from typing import List, Dict, Any
 
 try:
+    import sabctools
     import Cheetah
     import feedparser
     import configobj
@@ -76,6 +77,7 @@ from sabnzbd.constants import (
     DEF_LOG_FILE,
     DEF_STD_CONFIG,
     DEF_LOG_CHERRY,
+    CONFIG_BACKUP_HTTPS,
 )
 import sabnzbd.newsunpack
 from sabnzbd.misc import (
@@ -398,15 +400,13 @@ def get_user_profile_paths():
         return
 
     elif sabnzbd.MACOS:
-        home = os.environ.get("HOME")
-        if home:
+        if home := os.environ.get("HOME"):
             sabnzbd.DIR_LCLDATA = "%s/Library/Application Support/SABnzbd" % home
             sabnzbd.DIR_HOME = home
             return
     else:
         # Unix/Linux
-        home = os.environ.get("HOME")
-        if home:
+        if home := os.environ.get("HOME"):
             sabnzbd.DIR_LCLDATA = "%s/.%s" % (home, DEF_WORKDIR)
             sabnzbd.DIR_HOME = home
             return
@@ -418,25 +418,26 @@ def get_user_profile_paths():
 
 def print_modules():
     """Log all detected optional or external modules"""
-    if sabnzbd.decoder.SABYENC_ENABLED:
-        # Yes, we have SABYenc, and it's the correct version, so it's enabled
-        logging.info("SABYenc module (v%s)... found!", sabnzbd.decoder.SABYENC_VERSION)
-        logging.info("SABYenc module is using SIMD set: %s", sabnzbd.decoder.SABYENC_SIMD)
+    if sabnzbd.decoder.SABCTOOLS_ENABLED:
+        # Yes, we have SABCTools, and it's the correct version, so it's enabled
+        logging.info("SABCTools module (v%s)... found!", sabnzbd.decoder.SABCTOOLS_VERSION)
+        logging.info("SABCTools module is using SIMD set: %s", sabnzbd.decoder.SABCTOOLS_SIMD)
+        logging.info("SABCTools module is linked to OpenSSL: %s", sabnzbd.decoder.SABCTOOLS_OPENSSL_LINKED)
+
+        # Check if we managed to link, warning for now
+        if not sabnzbd.decoder.SABCTOOLS_OPENSSL_LINKED:
+            logging.warning(
+                "Could not link to OpenSSL library, please report here: "
+                "https://github.com/sabnzbd/sabnzbd/issues/2421"
+            )
     else:
-        # Something wrong with SABYenc, so let's determine and print what:
-        if sabnzbd.decoder.SABYENC_VERSION:
-            # We have a VERSION, thus a SABYenc module, but it's not the correct version
-            logging.error(
-                T("SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"),
-                sabnzbd.decoder.SABYENC_VERSION,
-                sabnzbd.constants.SABYENC_VERSION_REQUIRED,
-            )
-        else:
-            # No SABYenc module at all
-            logging.error(
-                T("SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"),
-                sabnzbd.constants.SABYENC_VERSION_REQUIRED,
-            )
+        # Wrong SABCTools version, if it was fully missing it would fail to start due to check at the very top
+        logging.error(
+            T("SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"),
+            sabnzbd.decoder.SABCTOOLS_VERSION,
+            sabnzbd.constants.SABCTOOLS_VERSION_REQUIRED,
+        )
+
         # Do not allow downloading
         sabnzbd.NO_DOWNLOADING = True
 
@@ -1444,6 +1445,12 @@ def main():
         logging.error(T("Failed to start web-interface: "), exc_info=True)
         abort_and_show_error(browserhost, cherryport)
 
+    # Create a record of the active cert/key/chain files, for use with config.create_config_backup()
+    if enable_https:
+        for setting in CONFIG_BACKUP_HTTPS.values():
+            if full_path := getattr(sabnzbd.cfg, setting).get_path():
+                sabnzbd.CONFIG_BACKUP_HTTPS_OK.append(full_path)
+
     if sabnzbd.WIN32:
         if enable_https:
             mode = "s"

builder/osx/requirements.txt

@@ -1,3 +1,3 @@
 # Special requirements for macOS universal2 binary release
 # This way dependabot can auto-update them
-cryptography==39.0.0
+cryptography==39.0.1

builder/requirements.txt

@@ -1,19 +1,18 @@
 # Basic build requirements
 # Note that not all sub-dependencies are listed, but only ones we know could cause trouble
-pyinstaller==5.7.0
-pyinstaller-hooks-contrib==2022.14
+pyinstaller==5.8.0
+pyinstaller-hooks-contrib==2022.15
 altgraph==0.17.3
 wrapt==1.14.1
-setuptools==65.6.3
+setuptools==67.2.0
 pkginfo==1.9.6
 PyGithub==1.57
 charset-normalizer==3.0.1
 certifi
 
 # orjson does not support 32bit Windows, exclude it based on Python-version
-# This way we also test ujson on Python 3.7 and 3.8 in the CI-tests
-# Fixed to 3.8.3 due to issue in 3.8.4: https://github.com/ijl/orjson/issues/331
-orjson==3.8.3; python_version > '3.8'
+# This way we also test ujson on Python 3.8 in the CI-tests
+orjson==3.8.6; python_version > '3.8'
 
 # For the macOS build
 dmgbuild==1.6.0; sys_platform == 'darwin'

interfaces/Config/templates/config.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Config"#-->
-<!--#set global $help_uri="configuration/3.7/configure"#-->
+<!--#set global $help_uri="configuration/4.0/configure"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <!--#from sabnzbd.encoding import CODEPAGE#-->
@@ -54,12 +54,12 @@
                     </td>
                 </tr>
                 <!--#end if#-->
-                <!--#if not $have_sabyenc#-->
+                <!--#if not $have_sabctools#-->
                 <tr>
-                    <th scope="row">SABYenc:</th>
+                    <th scope="row">SABCTools:</th>
                     <td>
                         <span class="label label-danger">$T('notAvailable')</span>
-                        <a href="$helpuri$help_uri#no_sabyenc" target="_blank"><span class="glyphicon glyphicon-question-sign"></span></a>
+                        <a href="$helpuri$help_uri#no_sabctools" target="_blank"><span class="glyphicon glyphicon-question-sign"></span></a>
                     </td>
                 </tr>
                 <!--#end if#-->

interfaces/Config/templates/config_cat.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Categories"#-->
-<!--#set global $help_uri="configuration/3.7/categories"#-->
+<!--#set global $help_uri="configuration/4.0/categories"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 <div class="colmask">
     <div class="section">

interfaces/Config/templates/config_folders.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Folders"#-->
-<!--#set global $help_uri="configuration/3.7/folders"#-->
+<!--#set global $help_uri="configuration/4.0/folders"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <div class="colmask">

interfaces/Config/templates/config_general.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="General"#-->
-<!--#set global $help_uri="configuration/3.7/general"#-->
+<!--#set global $help_uri="configuration/4.0/general"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <div class="colmask">
@@ -375,7 +375,7 @@
     })
 
     // Only allow re-generate if default certs
-    if(\$('#https_cert').val() != 'server.cert') {
+    if(\$('#https_cert').val() != '$def_https_cert_file') {
         \$('.generate_cert').attr('disabled', 'disabled')
     }
 

interfaces/Config/templates/config_notify.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Email"#-->
-<!--#set global $help_uri="configuration/3.7/notifications"#-->
+<!--#set global $help_uri="configuration/4.0/notifications"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <!--#def show_notify_checkboxes($section_label)#-->

interfaces/Config/templates/config_rss.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="RSS"#-->
-<!--#set global $help_uri="configuration/3.7/rss"#-->
+<!--#set global $help_uri="configuration/4.0/rss"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 <!--#import html#-->
 <div class="colmask">

interfaces/Config/templates/config_scheduling.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Scheduling"#-->
-<!--#set global $help_uri="configuration/3.7/scheduling"#-->
+<!--#set global $help_uri="configuration/4.0/scheduling"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <%

interfaces/Config/templates/config_server.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Servers"#-->
-<!--#set global $help_uri="configuration/3.7/servers"#-->
+<!--#set global $help_uri="configuration/4.0/servers"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <!--#import json#-->
@@ -51,11 +51,11 @@
                     </div>
                     <div class="field-pair advanced-settings">
                         <label class="config" for="port">$T('srv-port')</label>
-                        <input type="number" name="port" id="port" size="8" value="119" min="0" />
+                        <input type="number" name="port" id="port" size="8" value="563" min="0" />
                     </div>
                     <div class="field-pair">
                         <label class="config" for="ssl">$T('srv-ssl')</label>
-                        <input type="checkbox" name="ssl" id="ssl" value="1" />
+                        <input type="checkbox" name="ssl" id="ssl" value="1" checked />
                         <span class="desc">$T('explain-ssl')</span>
                     </div>
                     <!-- Tricks to avoid browser auto-fill, fixed on-submit with javascript -->

interfaces/Config/templates/config_sorting.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Sorting"#-->
-<!--#set global $help_uri="configuration/3.7/sorting"#-->
+<!--#set global $help_uri="configuration/4.0/sorting"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <div class="colmask">

interfaces/Config/templates/config_special.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Special"#-->
-<!--#set global $help_uri="configuration/3.7/special"#-->
+<!--#set global $help_uri="configuration/4.0/special"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <div class="colmask">

interfaces/Config/templates/config_switches.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Switches"#-->
-<!--#set global $help_uri="configuration/3.7/switches"#-->
+<!--#set global $help_uri="configuration/4.0/switches"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <div class="colmask">

interfaces/wizard/one.html

@@ -57,7 +57,7 @@
                             <div class="form-group">
                                 <label for="port" class="col-sm-4 control-label">$T('srv-port')</label>
                                 <div class="col-sm-8">
-                                    <input type="number" class="form-control" name="port" id="port" value="<!--#if $port then $port else '119' #-->" />
+                                    <input type="number" class="form-control" name="port" id="port" value="<!--#if $port then $port else '563' #-->" />
                                 </div>
                             </div>
                             <div class="form-group">

po/email/SABemail.pot

@@ -5,7 +5,7 @@
 #
 msgid ""
 msgstr ""
-"Project-Id-Version: SABnzbd-3.8.0-develop\n"
+"Project-Id-Version: SABnzbd-4.0.0-develop\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: team@sabnzbd.org\n"
 "Language-Team: SABnzbd <team@sabnzbd.org>\n"

po/email/cs.po

@@ -1,7 +1,7 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/da.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/de.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/es.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/fi.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/fr.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/he.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # ION, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/nb.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/nl.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/pl.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/pt_BR.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/ro.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/ru.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/sr.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/sv.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/email/zh_CN.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file EMAIL
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/main/SABnzbd.pot

@@ -5,7 +5,7 @@
 #
 msgid ""
 msgstr ""
-"Project-Id-Version: SABnzbd-3.8.0-develop\n"
+"Project-Id-Version: SABnzbd-4.0.0-develop\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: team@sabnzbd.org\n"
 "Language-Team: SABnzbd <team@sabnzbd.org>\n"
@@ -35,12 +35,7 @@ msgstr ""
 
 #. Error message
 #: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+msgid "SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -598,7 +593,7 @@ msgstr ""
 msgid "API Key incorrect, Use the api key from Config->General in your 3rd party program:"
 msgstr ""
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr ""
 

po/main/cs.po

@@ -1,16 +1,16 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Pavel C <quoing_transifex@mess.cz>, 2021
 # Safihre <safihre@sabnzbd.org>, 2023
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Czech (https://www.transifex.com/sabnzbd/teams/111101/cs/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -38,18 +38,12 @@ msgstr "Nezdařilo se spustit webové rozhraní"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Šablona pro web nebyla nalezena: %s, zkouším standardní šablonu"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc vypnut: Nenalezena správná verze! (Nalezena v%s, očekávána v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"Modul SABYenc... nebyl nalezen! Očekávána v%s - https://sabnzbd.org/sabyenc"
+"SABCTools vypnut: Nenalezena správná verze! (Nalezena v%s, očekávána v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -645,7 +639,7 @@ msgstr ""
 "Nesprávný API klíč, použijte api klíč z Nastavení->Obecné ve vašem programu "
 "třetí strany:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Přihlášené selhalo, zkontrolujte jméno a heslo."
 

po/main/da.po

@@ -1,15 +1,15 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2023
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Danish (https://www.transifex.com/sabnzbd/teams/111101/da/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -37,19 +37,13 @@ msgstr "Kunne ikke starte web-interface"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Kan ikke finde webskabeloner: %s, forsøger med standardskabelon"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc deaktiveret: Der blev ikke fundet nogen korrekt version (Fandt v%s, "
-"forventede v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"SABYenc modul... IKKE fundet! Forventede v%s - https://sabnzbd.org/sabyenc"
+"SABCTools deaktiveret: Der blev ikke fundet nogen korrekt version (Fandt "
+"v%s, forventede v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -653,7 +647,7 @@ msgstr ""
 "Forkert API-nøgle, anvend api-nøglen fra Konfiguration->Generelt i dit "
 "tredjepartsprogram:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Godkendelse mislykkedes, kontrollere brugernavn/adgangskode."
 

po/main/de.po

@@ -1,7 +1,7 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # C E <githubce@eiselt.ch>, 2020
 # Nikolai Bohl <n.kay01@gmail.com>, 2020
@@ -12,12 +12,12 @@
 # Nils Briggen, 2022
 # reloxx13 <reloxx@interia.pl>, 2022
 # Safihre <safihre@sabnzbd.org>, 2023
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: German (https://www.transifex.com/sabnzbd/teams/111101/de/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -47,19 +47,13 @@ msgstr ""
 "Konnte Web-Vorlage nicht finden: %s Versuche die Standard-Vorlage zu "
 "verwenden."
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc deaktiviert: Keine korrekte Version gefunden! (Gefunden v%s, "
-"Erwartet v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"SABYenc Modul... Nicht gefunden! Erwarte v%s - https://sabnzbd.org/sabyenc"
+"SABCTools deaktiviert: Keine korrekte Version gefunden! (Gefunden v%s, "
+"Erwartet v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -686,7 +680,7 @@ msgstr ""
 "API-Schlüssel ungültig. Bitte API-Schlüssel aus Einstellungen->Allgemein in "
 "die externe Anwendung eingeben:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr ""
 "Authentifizierung fehlgeschlagen. Überprüfen Sie Benutzername und Passwort."

po/main/es.po

@@ -1,12 +1,12 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Ester Molla Aragones <moarages@gmail.com>, 2020
 # 1024mb <angelb2203@gmail.com>, 2023
 # Safihre <safihre@sabnzbd.org>, 2023
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
@@ -41,20 +41,13 @@ msgstr ""
 "No se puede encontrar la plantilla web: %s, intentando con la plantilla "
 "estandar"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc deshabilitado: ¡no se ha encontrado la versión correcta! (Se ha "
-"encontrado la v%s, se esperaba la v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"Módulo SABYenc... ¡NO encontrado! Se esperaba la v%s - "
-"https://sabnzbd.org/sabyenc"
+"SABCTools deshabilitado: ¡no se ha encontrado la versión correcta! (Se ha "
+"encontrado la v%s, se esperaba la v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -676,7 +669,7 @@ msgstr ""
 "Clave de API erróneo, favor ingresar la clave correcta desde Config->General"
 " en tu aplicacion externa:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Autenticación fallida, compruebe el usuario o la contraseña."
 

po/main/fi.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2022
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
@@ -37,15 +37,10 @@ msgstr "Web-käyttöliittymän käynnistys epäonnistui"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Web-mallia %s ei löydy, yritetään käyttää oletusmallia"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -649,7 +644,7 @@ msgstr ""
 "API avain virheellinen, käytä Asetukset->Yleiset löytyvää api avainta "
 "käyttämääsi kolmannen osapuolen ohjelmaan:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Varmennus epäonnistui, tarkista käyttäjänimi/salasana."
 

po/main/fr.po

@@ -1,16 +1,16 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Fred L <88com88@gmail.com>, 2023
 # Safihre <safihre@sabnzbd.org>, 2023
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Fred L <88com88@gmail.com>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: French (https://www.transifex.com/sabnzbd/teams/111101/fr/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -40,19 +40,13 @@ msgstr ""
 "Impossible de trouver le template de l'interface web : %s, nouvelle "
 "tentative avec le template standard"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc désactivé: aucune version correcte n'a été trouvée ! (v%s trouvée, "
-"v%s attendue)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"Module SABYenc... NON trouvé ! v%s attendue - https://sabnzbd.org/sabyenc"
+"SABCTools désactivé: aucune version correcte n'a été trouvée ! (v%s trouvée,"
+" v%s attendue)"
 
 #. Error message
 #: SABnzbd.py
@@ -684,7 +678,7 @@ msgstr ""
 "Clé API incorrecte, utilisez la clé API de la configuration générale dans "
 "votre application tierce :"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Echec d'authentification, vérifiez les identifiant/mot de passe."
 
@@ -3122,7 +3116,7 @@ msgstr "Dossiers système"
 
 #: sabnzbd/skintext.py
 msgid "Hidden Folders"
-msgstr ""
+msgstr "Dossiers cachés"
 
 #: sabnzbd/skintext.py
 msgid "Administrative Folder"

po/main/he.po

@@ -1,16 +1,16 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # ION, 2022
 # Safihre <safihre@sabnzbd.org>, 2023
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Hebrew (https://www.transifex.com/sabnzbd/teams/111101/he/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -38,16 +38,11 @@ msgstr "נכשל בהתחלת ממשק רשת"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "לא ניתן למצוא תבניות רשת: %s, מנסה תבנית תקנית"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr "SABYenc מושבת: גרסה נכונה לא נמצאה! (%s נמצאה, מצפה אל %s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
-msgstr "מודול SABYenc… לא נמצא! מצפה אל %s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
+msgstr "SABCTools מושבת: גרסה נכונה לא נמצאה! (%s נמצאה, מצפה אל %s)"
 
 #. Error message
 #: SABnzbd.py
@@ -647,7 +642,7 @@ msgid ""
 "program:"
 msgstr "מפתח API שגוי, השתמש במפתח ה־API מתצורה->כללי בתוכנית הצד השלישי שלך:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "אימות נכשל, בדוק שם משתמש/סיסמה."
 

po/main/nb.po

@@ -1,15 +1,15 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2023
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Norwegian Bokmål (https://www.transifex.com/sabnzbd/teams/111101/nb/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -37,18 +37,12 @@ msgstr "Kunne ikke starte webgrensesnittet"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Kan ikke finne webmal: %s, prøver standardmal"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc deaktivert: Fant ikke korrekt versjon! (Fant v%s, forventet v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"SABYenc modul... IKKE funnet! Forventet v%s - https://sabnzbd.org/sabyenc"
+"SABCTools deaktivert: Fant ikke korrekt versjon! (Fant v%s, forventet v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -645,7 +639,7 @@ msgstr ""
 "API-nøkkel er feil, bruk API-nøkkel fra Konfigurasjon->Generelt i ditt "
 "tredjepartsprogram:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Godkjenning mislyktes, kontroller brukernavn og passord."
 

po/main/nl.po

@@ -1,16 +1,16 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Rik Brouwer, 2022
 # Safihre <safihre@sabnzbd.org>, 2023
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Dutch (https://www.transifex.com/sabnzbd/teams/111101/nl/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -38,19 +38,13 @@ msgstr "Webinterface kan niet gestart worden"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Websjabloon %s niet te vinden; het standaardsjabloon wordt gebruikt."
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc uitgeschakeld, geen bruikbare versie gevonden! (V%s gevonden, V%s "
-"verwacht)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"SABYenc module... NIET gevonden! Verwacht V%s - https://sabnzbd.org/sabyenc"
+"SABCTools uitgeschakeld, geen bruikbare versie gevonden! (V%s gevonden, V%s "
+"verwacht)"
 
 #. Error message
 #: SABnzbd.py
@@ -678,7 +672,7 @@ msgstr ""
 "API-sleutel incorrect; vul de API-sleutel van 'Configuratie' => 'Algemeen' "
 "in bij het externe programma:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Inloggen mislukt, controleer gebruikersnaam en wachtwoord."
 

po/main/pl.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2022
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
@@ -37,15 +37,10 @@ msgstr "Nie udało się uruchomić interfejsu WWW"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Nie znaleziono szablonu: %s, próbuję użyć standardowego szablonu"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -644,7 +639,7 @@ msgstr ""
 "Klucz API jest nieprawidłowy, użyj klucza API z sekcji Konfiguracja->Ogólne "
 "w zewnętrznym programie:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Błąd połączenia, sprawdź nazwę użytkownika i hasło."
 

po/main/pt_BR.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2022
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
@@ -38,15 +38,10 @@ msgid "Cannot find web template: %s, trying standard template"
 msgstr ""
 "Não foi possível encontrar o template web: %s. Tentando o template padrão"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -648,7 +643,7 @@ msgstr ""
 "Chave de API incorreta. Use a chave de API de Configuração->Geral em seu "
 "programa de terceiros:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Falha de autenticação, verifique usuário / senha."
 

po/main/ro.po

@@ -1,16 +1,16 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Eduard Baniceru <war4peace@gmail.com>, 2021
 # Safihre <safihre@sabnzbd.org>, 2023
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Romanian (https://www.transifex.com/sabnzbd/teams/111101/ro/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -38,19 +38,13 @@ msgstr "Pornirea interfeţei-web nereuşită"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Nu se poate găsi şablon web:%s, se încearcă şablon standard"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc dezactivat: nu s-a găsit o versiune corectă! (Găsită v%s, se "
-"așteaptă v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"Modul SABYenc... NEgăsit! Se așteaptă v%s - https://sabnzbd.org/sabyenc"
+"SABCTools dezactivat: nu s-a găsit o versiune corectă! (Găsită v%s, se "
+"așteaptă v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -662,7 +656,7 @@ msgstr ""
 "Cheie API incorectă, Folosiţi cheia api din Configurare->General în "
 "programul dumneavoastră terţ:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Autentificare nereuşită, verifică nume utilizator/parolă."
 

po/main/ru.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2022
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
@@ -39,15 +39,10 @@ msgstr ""
 "Не удаётся найти шаблон веб-интерфейса: %s. Выполняется попытка использовать"
 " стандартный шаблон"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -644,7 +639,7 @@ msgstr ""
 "Неправильный ключ API. Используйте в сторонней программе ключ API из раздела"
 " «Настройка -> Общие»:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Ошибка проверки подлинности. Проверьте имя и пароль."
 

po/main/sr.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2022
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
@@ -37,15 +37,10 @@ msgstr "Neuspešno pokretanje web interfejsa"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Немогуће наћи веб модел: %s, програм покушава са стандардним моделом"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -641,7 +636,7 @@ msgid ""
 msgstr ""
 "API кључ је погрешан, унети у спољни програм API кључ из Подешавања->Опште:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Аутентификација погрешна, проверити име/лозинку."
 

po/main/sv.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2022
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
@@ -37,15 +37,10 @@ msgstr "Det gick inte att starta webbgränssnittet"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Hittar inte webbmall: %s, försöker med standardmall"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -643,7 +638,7 @@ msgstr ""
 "API-nyckel felaktig, använd api-nyckeln från Konfiguration-> Allmänt i ditt "
 "tredjepartsprogram:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Autentisering misslyckades, kontrollera användarnamn och lösenord."
 

po/main/zh_CN.po

@@ -1,15 +1,15 @@
 # SABnzbd Translation Template file MAIN
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2023
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Chinese (China) (https://www.transifex.com/sabnzbd/teams/111101/zh_CN/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -37,16 +37,11 @@ msgstr "web 界面启动失败"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "无法找到 web 模板: %s，正在尝试标准模板"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr "SABYenc 已禁用：未找到正确的版本！（找到 v%s，要求 v%s）"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
-msgstr "SABYenc 模块... 未找到！要求  v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
+msgstr "SABCTools 已禁用：未找到正确的版本！（找到 v%s，要求 v%s）"
 
 #. Error message
 #: SABnzbd.py
@@ -636,7 +631,7 @@ msgid ""
 "program:"
 msgstr "API Key 不正确，请在第三方程序中使用“配置”->“常规”中的 api key:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "身份认证失败，请检查用户名/密码。"
 

po/nsis/SABnsis.pot

@@ -5,7 +5,7 @@
 #
 msgid ""
 msgstr ""
-"Project-Id-Version: SABnzbd-3.8.0-develop\n"
+"Project-Id-Version: SABnzbd-4.0.0-develop\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: team@sabnzbd.org\n"
 "Language-Team: SABnzbd <team@sabnzbd.org>\n"

po/nsis/cs.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Pavel C <quoing_transifex@mess.cz>, 2022
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/da.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/de.po

@@ -1,14 +1,14 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
 # reloxx13 <reloxx@interia.pl>, 2022
-#
+# 
 msgid ""
 msgstr ""
-"Project-Id-Version: SABnzbd-3.8.0-develop\n"
+"Project-Id-Version: SABnzbd-4.0.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:56+0000\n"
 "Last-Translator: reloxx13 <reloxx@interia.pl>, 2022\n"
 "Language-Team: German (https://www.transifex.com/sabnzbd/teams/111101/de/)\n"

po/nsis/es.po

@@ -1,11 +1,11 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
 # Ester Molla Aragones <moarages@gmail.com>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/fi.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/fr.po

@@ -1,11 +1,11 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
 # Fred L <88com88@gmail.com>, 2021
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/he.po

@@ -1,11 +1,11 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
 # ION, 2021
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/nb.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/nl.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2021
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/pl.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/pt_BR.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/ro.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/ru.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/sr.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/sv.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

po/nsis/zh_CN.po

@@ -1,10 +1,10 @@
 # SABnzbd Translation Template file NSIS
 # Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
-#
+# 
 # Translators:
 # Safihre <safihre@sabnzbd.org>, 2020
-#
+# 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"

requirements.txt

@@ -1,11 +1,11 @@
 # Main requirements
 # Note that not all sub-dependencies are listed, but only ones we know could cause trouble
-sabyenc3==5.4.4
+sabctools==6.1.0
 cheetah3==3.2.6.post1
 cffi==1.15.1
 pycparser==2.21
 feedparser==6.0.10
-configobj==5.0.6
+configobj==5.0.8
 cheroot==9.0.0
 six==1.16.0
 cherrypy==18.8.0
@@ -13,12 +13,12 @@ jaraco.functools==3.5.2
 jaraco.collections==3.8.0
 jaraco.text==3.8.1 # Newer version introduces irrelevant extra dependencies
 jaraco.classes==3.2.3
-jaraco.context==4.2.0
+jaraco.context==4.3.0
 more-itertools==9.0.0
 zc.lockfile==2.0
 python-dateutil==2.8.2
-tempora==5.2.0
-pytz==2022.7
+tempora==5.2.1
+pytz==2022.7.1
 sgmllib3k==1.0.0
 portend==3.1.0
 chardet==5.1.0
@@ -30,7 +30,7 @@ rebulk==3.1.0
 
 # Recent cryptography versions require Rust. If you run into issues compiling this
 # SABnzbd will also work with older pre-Rust versions such as cryptography==3.3.2
-cryptography==39.0.0
+cryptography==39.0.1
 
 # We recommend using "orjson" as it is 2x as fast as "ujson". However, it requires
 # Rust so SABnzbd works just as well with "ujson" or the Python built in "json" module

sabnzbd/__init__.py

@@ -162,7 +162,7 @@ WIN_SERVICE = None  # Instance of our Win32 Service Class
 BROWSER_URL = None
 
 CERTIFICATE_VALIDATION = True
-NO_DOWNLOADING = False  # When essentials are missing (SABYenc/par2/unrar)
+NO_DOWNLOADING = False  # When essentials are missing (SABCTools/par2/unrar)
 
 WEB_DIR = None
 WEB_DIR_CONFIG = None
@@ -190,6 +190,9 @@ DOWNLOAD_DIR_SPEED = 0
 COMPLETE_DIR_SPEED = 0
 INTERNET_BANDWIDTH = 0
 
+# Record of HTTPS config files at startup
+CONFIG_BACKUP_HTTPS_OK = []
+
 # Rendering of original command line arguments in Config
 CMDLINE = " ".join(['"%s"' % arg for arg in sys.argv])
 

sabnzbd/articlecache.py

@@ -26,7 +26,7 @@ from typing import Dict, List
 
 import sabnzbd
 from sabnzbd.decorators import synchronized
-from sabnzbd.constants import GIGI, ANFO, MEBI, LIMIT_DECODE_QUEUE, MIN_DECODE_QUEUE
+from sabnzbd.constants import GIGI, ANFO, MEBI, LIMIT_DECODE_QUEUE, MIN_DECODE_QUEUE, ASSEMBLER_WRITE_THRESHOLD
 from sabnzbd.nzbstuff import Article
 
 # Operations on the article table are handled via try/except.
@@ -45,6 +45,8 @@ class ArticleCache:
         # so it can be larger on memory-rich systems
         self.decoder_cache_article_limit = 0
 
+        self.assembler_write_trigger: int = 1
+
         # On 32 bit we only allow the user to set 1GB
         # For 64 bit we allow up to 4GB, in case somebody wants that
         self.__cache_upper_limit = GIGI
@@ -68,6 +70,16 @@ class ArticleCache:
         # The cache should also not be too small
         self.decoder_cache_article_limit = max(decoder_cache_limit, MIN_DECODE_QUEUE)
 
+        # Set assembler_write_trigger to be the equivalent of ASSEMBLER_WRITE_THRESHOLD %
+        # of the total cache, assuming an article size of 750 000 bytes
+        self.assembler_write_trigger = int(self.__cache_limit * ASSEMBLER_WRITE_THRESHOLD / 100 / 750_000) + 1
+
+        logging.debug(
+            "Decoder cache limit = %d - Assembler trigger = %d",
+            self.decoder_cache_article_limit,
+            self.assembler_write_trigger,
+        )
+
     @synchronized(ARTICLE_COUNTER_LOCK)
     def reserve_space(self, data_size: int):
         """Reserve space in the cache"""
@@ -92,9 +104,10 @@ class ArticleCache:
         # Register article for bookkeeping in case the job is deleted
         nzo.add_saved_article(article)
 
-        if article.lowest_partnum and not article.nzf.import_finished:
-            # Write the first-fetched articles to disk
-            # Otherwise the cache could overflow
+        if article.lowest_partnum and not (article.nzf.import_finished or article.nzf.filename_checked):
+            # Write the first-fetched articles to temporary file unless downloading
+            # of the rest of the parts has started or filename is verified.
+            # Otherwise the cache could overflow.
             self.__flush_article_to_disk(article, data)
             return
 

sabnzbd/assembler.py

@@ -24,7 +24,6 @@ import queue
 import logging
 import re
 from threading import Thread
-import hashlib
 import ctypes
 from typing import Tuple, Optional, List
 
@@ -56,11 +55,8 @@ class Assembler(Thread):
     def process(self, nzo: NzbObject, nzf: Optional[NzbFile] = None, file_done: Optional[bool] = None):
         self.queue.put((nzo, nzf, file_done))
 
-    def queue_full(self):
-        return self.queue.qsize() >= MAX_ASSEMBLER_QUEUE
-
-    def partial_nzf_in_queue(self, nzf: NzbFile):
-        return (nzf.nzo, nzf, False) in self.queue.queue
+    def queue_level(self) -> float:
+        return self.queue.qsize() / MAX_ASSEMBLER_QUEUE
 
     def run(self):
         while 1:
@@ -78,9 +74,7 @@ class Assembler(Thread):
                     self.diskspace_check(nzo, nzf)
 
                 # Prepare filepath
-                filepath = nzf.prepare_filepath()
-
-                if filepath:
+                if filepath := nzf.prepare_filepath():
                     try:
                         logging.debug("Decoding part of %s", filepath)
                         self.assemble(nzo, nzf, file_done)
@@ -170,9 +164,6 @@ class Assembler(Thread):
         1) Partial write: write what we have
         2) Nothing written before: write all
         """
-        # New hash-object needed?
-        if not nzf.md5:
-            nzf.md5 = hashlib.md5()
 
         # We write large article-sized chunks, so we can safely skip the buffering of Python
         with open(nzf.filepath, "ab", buffering=0) as fout:
@@ -191,7 +182,7 @@ class Assembler(Thread):
                     # Could be empty in case nzo was deleted
                     if data:
                         fout.write(data)
-                        nzf.md5.update(data)
+                        nzf.update_crc32(article.crc32, len(data))
                         article.on_disk = True
                     else:
                         logging.info("No data found when trying to write %s", article)
@@ -207,7 +198,7 @@ class Assembler(Thread):
         # Final steps
         if file_done:
             set_permissions(nzf.filepath)
-            nzf.md5sum = nzf.md5.digest()
+            nzf.assembled = True
 
     @staticmethod
     def check_encrypted_and_unwanted(nzo: NzbObject, nzf: NzbFile):

sabnzbd/bpsmeter.py

@@ -434,40 +434,6 @@ class BPSMeter:
         # We record every second, but display at the user's refresh-rate
         return self.bps_list[::refresh_rate]
 
-    def get_stable_speed(self, timespan: int = 10) -> Optional[int]:
-        """See if there is a stable speed the last <timespan> seconds
-        None: indicates it can't determine yet
-        0: the speed was not stable during <timespan>
-        Positive float: the speed was stable
-        """
-        if len(self.bps_list) < timespan:
-            return None
-
-        # Check if speed fell by more than 15%
-        try:
-            if self.bps_list[-1] / self.bps_list[-timespan] < 0.85:
-                return 0
-        except:
-            pass
-
-        # Calculate the variance in the speed
-        avg = sum(self.bps_list[-timespan:]) / timespan
-        vari = 0
-        for bps in self.bps_list[-timespan:]:
-            vari += abs(bps - avg)
-        vari = vari / timespan
-
-        try:
-            # See if the variance is less than 5%
-            if (vari / (self.bps / KIBI)) < 0.05:
-                return avg
-            else:
-                return 0
-        except:
-            # Probably one of the values was 0
-            pass
-        return None
-
     def reset_quota(self, force: bool = False):
         """Check if it's time to reset the quota, optionally resuming
         Return True, when still paused or should be paused

sabnzbd/cfg.py

@@ -47,6 +47,8 @@ from sabnzbd.constants import (
     DEF_COMPLETE_DIR,
     DEF_FOLDER_MAX,
     DEF_STD_WEB_COLOR,
+    DEF_HTTPS_CERT_FILE,
+    DEF_HTTPS_KEY_FILE,
 )
 
 
@@ -279,8 +281,8 @@ bandwidth_max = OptionStr("misc", "bandwidth_max")
 cache_limit = OptionStr("misc", "cache_limit")
 web_dir = OptionStr("misc", "web_dir", DEF_STD_WEB_DIR)
 web_color = OptionStr("misc", "web_color", DEF_STD_WEB_COLOR)
-https_cert = OptionDir("misc", "https_cert", "server.cert", create=False)
-https_key = OptionDir("misc", "https_key", "server.key", create=False)
+https_cert = OptionDir("misc", "https_cert", DEF_HTTPS_CERT_FILE, create=False)
+https_key = OptionDir("misc", "https_key", DEF_HTTPS_KEY_FILE, create=False)
 https_chain = OptionDir("misc", "https_chain", create=False)
 enable_https = OptionBool("misc", "enable_https", False)
 # 0=local-only, 1=nzb, 2=api, 3=full_api, 4=webui, 5=webui with login for external
@@ -434,6 +436,7 @@ host_whitelist = OptionList("misc", "host_whitelist", validation=all_lowercase)
 local_ranges = OptionList("misc", "local_ranges", protect=True)
 max_url_retries = OptionNumber("misc", "max_url_retries", 10, minval=1)
 downloader_sleep_time = OptionNumber("misc", "downloader_sleep_time", 10, minval=0)
+receive_threads = OptionNumber("misc", "receive_threads", 2, minval=1)
 num_simd_decoders = OptionNumber("misc", "num_simd_decoders", 2, minval=1)
 ssdp_broadcast_interval = OptionNumber("misc", "ssdp_broadcast_interval", 15, minval=1, maxval=600)
 ext_rename_ignore = OptionList("misc", "ext_rename_ignore", validation=lower_case_ext)

sabnzbd/config.py

@@ -34,7 +34,14 @@ from urllib.parse import urlparse
 import configobj
 
 import sabnzbd
-from sabnzbd.constants import CONFIG_VERSION, NORMAL_PRIORITY, DEFAULT_PRIORITY, CONFIG_BACKUP_FILES, DEF_INI_FILE
+from sabnzbd.constants import (
+    CONFIG_VERSION,
+    NORMAL_PRIORITY,
+    DEFAULT_PRIORITY,
+    CONFIG_BACKUP_FILES,
+    CONFIG_BACKUP_HTTPS,
+    DEF_INI_FILE,
+)
 from sabnzbd.decorators import synchronized
 from sabnzbd.filesystem import clip_path, real_path, create_real_path, renamer, remove_file, is_writable
 
@@ -943,6 +950,17 @@ def create_config_backup() -> Union[str, bool]:
                     if os.path.isfile(full_path):
                         with open(full_path, "rb") as data:
                             zip_ref.writestr(filename, data.read())
+                for filename, setting in CONFIG_BACKUP_HTTPS.items():
+                    full_path = getattr(sabnzbd.cfg, setting).get_path()
+                    # Only accept HTTPS config files that were successfully loaded by cherrypy on
+                    # startup to protect against last-minute breaking config changes as well as
+                    # inclusion of unrelated files in the backup through manipulated settings.
+                    if full_path and os.path.isfile(full_path) and full_path in sabnzbd.CONFIG_BACKUP_HTTPS_OK:
+                        logging.debug("Adding %s file %s to backup", setting, full_path)
+                        with open(full_path, "rb") as data:
+                            # Add the https cert/key/chain files with a fixed relative filename,
+                            # regardless of where they are actually stored on the filesystem
+                            zip_ref.writestr(filename, data.read())
                 with open(CFG_OBJ.filename, "rb") as data:
                     zip_ref.writestr(DEF_INI_FILE, data.read())
         return clip_path(complete_path)
@@ -965,6 +983,7 @@ def validate_config_backup(config_backup_data: bytes) -> bool:
 
 def restore_config_backup(config_backup_data: bytes):
     """Restore configuration files from zip file"""
+    global CFG_MODIFIED
     try:
         with io.BytesIO(config_backup_data) as backup_ref:
             with zipfile.ZipFile(backup_ref, "r") as zip_ref:
@@ -977,16 +996,22 @@ def restore_config_backup(config_backup_data: bytes):
 
                 # Write the rest of the admin files that we want to recover
                 adminpath = sabnzbd.cfg.admin_dir.get_path()
-                for filename in CONFIG_BACKUP_FILES:
+                for filename in CONFIG_BACKUP_FILES + list(CONFIG_BACKUP_HTTPS.keys()):
                     try:
                         zip_ref.getinfo(filename)
                         destination_file = os.path.join(adminpath, filename)
                         logging.debug("Writing backup of %s to %s", filename, destination_file)
                         with open(destination_file, "wb") as destination_ref:
                             destination_ref.write(zip_ref.read(filename))
+                        # For HTTPS config files, point the associated setting to the restored file
+                        if setting := CONFIG_BACKUP_HTTPS.get(filename):
+                            logging.debug("Setting value of %s to restored file %s", setting, filename)
+                            getattr(sabnzbd.cfg, setting).set(filename)
+                            CFG_MODIFIED = True
                     except KeyError:
                         # File not in archive
                         pass
+                save_config()
     except:
         logging.warning(T("Could not restore backup"))
         logging.info("Traceback: ", exc_info=True)

sabnzbd/constants.py

@@ -49,17 +49,11 @@ RENAMES_FILE = "__renames__"
 ATTRIB_FILE = "SABnzbd_attrib"
 REPAIR_REQUEST = "repair-all.sab"
 
-SABYENC_VERSION_REQUIRED = "5.4.4"
+SABCTOOLS_VERSION_REQUIRED = "6.1.0"
 
 DB_HISTORY_VERSION = 1
 DB_HISTORY_NAME = "history%s.db" % DB_HISTORY_VERSION
 
-CONFIG_BACKUP_FILES = [
-    BYTES_FILE_NAME,
-    RSS_FILE_NAME,
-    DB_HISTORY_NAME,
-]
-
 DEF_DOWNLOAD_DIR = os.path.normpath("Downloads/incomplete")
 DEF_COMPLETE_DIR = os.path.normpath("Downloads/complete")
 DEF_ADMIN_DIR = "admin"
@@ -82,14 +76,30 @@ DEF_ARTICLE_CACHE_DEFAULT = "500M"
 DEF_ARTICLE_CACHE_MAX = "1G"
 DEF_TIMEOUT = 60
 DEF_SCANRATE = 5
+DEF_HTTPS_CERT_FILE = "server.cert"
+DEF_HTTPS_KEY_FILE = "server.key"
 MAX_WARNINGS = 20
 MAX_BAD_ARTICLES = 5
 
+CONFIG_BACKUP_FILES = [
+    BYTES_FILE_NAME,
+    RSS_FILE_NAME,
+    DB_HISTORY_NAME,
+]
+CONFIG_BACKUP_HTTPS = {  # "basename": "associated setting"
+    DEF_HTTPS_CERT_FILE: "https_cert",
+    DEF_HTTPS_KEY_FILE: "https_key",
+    "server.chain": "https_chain",
+}
+
 # Constants affecting download performance
 MIN_DECODE_QUEUE = 10
 LIMIT_DECODE_QUEUE = 100
-DIRECT_WRITE_TRIGGER = 35
-MAX_ASSEMBLER_QUEUE = 5
+MAX_ASSEMBLER_QUEUE = 10
+SOFT_QUEUE_LIMIT = 0.6
+# Percentage of cache to use before adding file to assembler
+ASSEMBLER_WRITE_THRESHOLD = 5
+NNTP_BUFFER_SIZE = int(800 * KIBI)
 
 REPAIR_PRIORITY = 3
 FORCE_PRIORITY = 2

sabnzbd/decoder.py

@@ -26,28 +26,31 @@ import binascii
 from io import BytesIO
 from threading import Thread
 from typing import Tuple, List, Optional
+from zlib import crc32
 
 import sabnzbd
 import sabnzbd.cfg as cfg
-from sabnzbd.constants import SABYENC_VERSION_REQUIRED
+from sabnzbd.constants import SABCTOOLS_VERSION_REQUIRED
 from sabnzbd.encoding import ubtou
 from sabnzbd.nzbstuff import Article
 from sabnzbd.misc import match_str
 
-# Check for correct SABYenc version
-SABYENC_VERSION = None
-SABYENC_SIMD = None
+# Check for correct SABCTools version
+SABCTOOLS_VERSION = None
+SABCTOOLS_SIMD = None
+SABCTOOLS_OPENSSL_LINKED = None
 try:
-    import sabyenc3
+    import sabctools
 
-    SABYENC_ENABLED = True
-    SABYENC_VERSION = sabyenc3.__version__
-    SABYENC_SIMD = sabyenc3.simd
+    SABCTOOLS_ENABLED = True
+    SABCTOOLS_VERSION = sabctools.__version__
+    SABCTOOLS_SIMD = sabctools.simd
+    SABCTOOLS_OPENSSL_LINKED = sabctools.openssl_linked
     # Verify version to at least match minor version
-    if SABYENC_VERSION[:3] != SABYENC_VERSION_REQUIRED[:3]:
+    if SABCTOOLS_VERSION[:3] != SABCTOOLS_VERSION_REQUIRED[:3]:
         raise ImportError
 except:
-    SABYENC_ENABLED = False
+    SABCTOOLS_ENABLED = False
 
 
 class BadData(Exception):
@@ -102,13 +105,13 @@ class Decoder:
             except:
                 pass
 
-    def process(self, article: Article, raw_data: List[bytes], data_size: int):
-        sabnzbd.ArticleCache.reserve_space(data_size)
-        self.decoder_queue.put((article, raw_data, data_size))
+    def process(self, article: Article, raw_data: bytearray, raw_data_size: int):
+        sabnzbd.ArticleCache.reserve_space(raw_data_size)
+        self.decoder_queue.put((article, raw_data, raw_data_size))
 
-    def queue_full(self) -> bool:
-        # Check if the queue size exceeds the limits
-        return self.decoder_queue.qsize() >= sabnzbd.ArticleCache.decoder_cache_article_limit
+    def queue_level(self) -> float:
+        # Return level of decoder queue. 0 = empty, >=1 = full.
+        return self.decoder_queue.qsize() / sabnzbd.ArticleCache.decoder_cache_article_limit
 
 
 class DecoderWorker(Thread):
@@ -117,14 +120,14 @@ class DecoderWorker(Thread):
     def __init__(self, decoder_queue):
         super().__init__()
         logging.debug("Initializing decoder %s", self.name)
-        self.decoder_queue: queue.Queue[Tuple[Optional[Article], Optional[List[bytes]], Optional[int]]] = decoder_queue
+        self.decoder_queue: queue.Queue[Tuple[Optional[Article], Optional[bytearray], Optional[int]]] = decoder_queue
 
     def run(self):
         while 1:
             # Set Article and NzbObject objects to None so references from this
             # thread do not keep the parent objects alive (see #1628)
             decoded_data = raw_data = article = nzo = None
-            article, raw_data, data_size = self.decoder_queue.get()
+            article, raw_data, raw_data_size = self.decoder_queue.get()
             if not article:
                 logging.debug("Shutting down decoder %s", self.name)
                 break
@@ -133,7 +136,7 @@ class DecoderWorker(Thread):
             art_id = article.article
 
             # Free space in the decoder-queue
-            sabnzbd.ArticleCache.free_reserved_space(data_size)
+            sabnzbd.ArticleCache.free_reserved_space(raw_data_size)
 
             # Keeping track
             article_success = False
@@ -146,9 +149,12 @@ class DecoderWorker(Thread):
                     logging.debug("Decoding %s", art_id)
 
                 if article.nzf.type == "uu":
+                    # TODO: UU needs to be fixed
+                    n = 250000
+                    raw_data = [bytes(raw_data[i : min(raw_data_size, i + n)]) for i in range(0, len(raw_data), n)]
                     decoded_data = decode_uu(article, raw_data)
                 else:
-                    decoded_data = decode_yenc(article, raw_data)
+                    decoded_data = decode_yenc(article, raw_data, raw_data_size)
 
                 article_success = True
 
@@ -180,7 +186,7 @@ class DecoderWorker(Thread):
 
             except (BadYenc, ValueError):
                 # Handles precheck and badly formed articles
-                if nzo.precheck and raw_data and raw_data[0].startswith(b"223 "):
+                if nzo.precheck and raw_data and raw_data.startswith(b"223 "):
                     # STAT was used, so we only get a status code
                     article_success = True
                 else:
@@ -194,9 +200,9 @@ class DecoderWorker(Thread):
                             pass
                     # Only bother with further checks if uu-decoding didn't work out
                     if not article_success:
-                        # Convert the initial chunks of raw socket data to article lines,
+                        # Convert the first 2000 bytes of raw socket data to article lines,
                         # and examine the headers (for precheck) or body (for download).
-                        for line in b"".join(raw_data[:2]).split(b"\r\n"):
+                        for line in raw_data[:2000].split(b"\r\n"):
                             lline = line.lower()
                             if lline.startswith(b"message-id:"):
                                 article_success = True
@@ -240,29 +246,32 @@ class DecoderWorker(Thread):
             sabnzbd.NzbQueue.register_article(article, article_success)
 
 
-def decode_yenc(article: Article, raw_data: List[bytes]) -> bytes:
-    # Let SABYenc do all the heavy lifting
-    decoded_data, yenc_filename, crc_correct = sabyenc3.decode_usenet_chunks(raw_data)
+def decode_yenc(article: Article, data: bytearray, raw_data_size: int) -> bytearray:
+    # Let SABCTools do all the heavy lifting
+    yenc_filename, crc_correct = sabctools.yenc_decode(data)
 
+    nzf = article.nzf
     # Assume it is yenc
-    article.nzf.type = "yenc"
+    nzf.type = "yenc"
 
     # Only set the name if it was found and not obfuscated
-    if not article.nzf.filename_checked and yenc_filename:
+    if not nzf.filename_checked and yenc_filename:
         # Set the md5-of-16k if this is the first article
         if article.lowest_partnum:
-            article.nzf.md5of16k = hashlib.md5(decoded_data[:16384]).digest()
+            nzf.md5of16k = hashlib.md5(data[:16384]).digest()
 
         # Try the rename, even if it's not the first article
         # For example when the first article was missing
-        article.nzf.nzo.verify_nzf_filename(article.nzf, yenc_filename)
+        nzf.nzo.verify_nzf_filename(nzf, yenc_filename)
 
     # CRC check
-    if not crc_correct:
+    if crc_correct is None:
         logging.info("CRC Error in %s", article.article)
-        raise BadData(decoded_data)
+        raise BadData(data)
 
-    return decoded_data
+    article.crc32 = crc_correct
+
+    return data
 
 
 def decode_uu(article: Article, raw_data: List[bytes]) -> bytes:
@@ -384,7 +393,9 @@ def decode_uu(article: Article, raw_data: List[bytes]) -> bytes:
             if not article.nzf.filename_checked and uu_filename:
                 article.nzf.nzo.verify_nzf_filename(article.nzf, uu_filename)
 
-        return decoded_data.getvalue()
+        data = decoded_data.getvalue()
+        article.crc32 = crc32(data)
+        return data
 
 
 def search_new_server(article: Article) -> bool:

sabnzbd/directunpacker.py

@@ -344,10 +344,14 @@ class DirectUnpacker(threading.Thread):
     def have_next_volume(self):
         """Check if next volume of set is available, start
         from the end of the list where latest completed files are
-        Make sure that files are 100% written to disk by checking md5sum
+        Make sure that files are 100% written to disk by checking nzf.assembled
         """
         for nzf_search in reversed(self.nzo.finished_files):
-            if nzf_search.setname == self.cur_setname and nzf_search.vol == (self.cur_volume + 1) and nzf_search.md5sum:
+            if (
+                nzf_search.setname == self.cur_setname
+                and nzf_search.vol == (self.cur_volume + 1)
+                and nzf_search.assembled
+            ):
                 return nzf_search
         return False
 

sabnzbd/dirscanner.py

@@ -19,10 +19,11 @@
 sabnzbd.dirscanner - Scanner for Watched Folder
 """
 
+import asyncio
 import os
-import time
 import logging
 import threading
+from typing import Generator, Set, Optional, Tuple
 
 import sabnzbd
 from sabnzbd.constants import SCAN_FILE_NAME, VALID_ARCHIVES, VALID_NZB_FILES
@@ -30,6 +31,9 @@ import sabnzbd.filesystem as filesystem
 import sabnzbd.config as config
 import sabnzbd.cfg as cfg
 
+DIR_SCANNER_LOCK = threading.RLock()
+VALID_EXTENSIONS = set(VALID_NZB_FILES + VALID_ARCHIVES)
+
 
 def compare_stat_tuple(tup1, tup2):
     """Test equality of two stat-tuples, content-related parts only"""
@@ -44,18 +48,10 @@ def compare_stat_tuple(tup1, tup2):
     return True
 
 
-def clean_file_list(inp_list, folder, files):
+async def clean_file_list(inp_list, files):
     """Remove elements of "inp_list" not found in "files" """
-    for path in sorted(inp_list):
-        fld, name = os.path.split(path)
-        if fld == folder:
-            present = False
-            for name in files:
-                if os.path.join(folder, name) == path:
-                    present = True
-                    break
-            if not present:
-                del inp_list[path]
+    for path in set(inp_list.keys()).difference(files):
+        del inp_list[path]
 
 
 class DirScanner(threading.Thread):
@@ -68,7 +64,15 @@ class DirScanner(threading.Thread):
     def __init__(self):
         super().__init__()
 
-        self.newdir()
+        self.loop: Optional[asyncio.AbstractEventLoop] = None
+        self.scanner_task: Optional[asyncio.Task] = None
+        self.lock = asyncio.Lock()  # Prevents concurrent scans
+        self.error_reported = False  # Prevents multiple reporting of missing watched folder
+        self.dirscan_dir = cfg.dirscan_dir.get_path()
+        self.dirscan_speed = cfg.dirscan_speed()
+        cfg.dirscan_dir.callback(self.newdir)
+        cfg.dirscan_speed.callback(self.newspeed)
+
         try:
             dirscan_dir, self.ignored, self.suspected = sabnzbd.filesystem.load_admin(SCAN_FILE_NAME)
             if dirscan_dir != self.dirscan_dir:
@@ -79,34 +83,24 @@ class DirScanner(threading.Thread):
             # successfully processed ones that cannot be deleted
             self.suspected = {}  # Will hold name/attributes of suspected candidates
 
-        self.loop_condition = threading.Condition(threading.Lock())
-        self.shutdown = False
-        self.error_reported = False  # Prevents multiple reporting of missing watched folder
-        self.dirscan_dir = cfg.dirscan_dir.get_path()
-        self.dirscan_speed = cfg.dirscan_speed() or None  # If set to 0, use None so the wait() is forever
-        self.busy = False
-        cfg.dirscan_dir.callback(self.newdir)
-        cfg.dirscan_speed.callback(self.newspeed)
-
     def newdir(self):
         """We're notified of a dir change"""
         self.ignored = {}
         self.suspected = {}
         self.dirscan_dir = cfg.dirscan_dir.get_path()
-        self.dirscan_speed = cfg.dirscan_speed()
+
+        self.start_scanner()
 
     def newspeed(self):
         """We're notified of a scan speed change"""
-        # If set to 0, use None so the wait() is forever
-        self.dirscan_speed = cfg.dirscan_speed() or None
-        with self.loop_condition:
-            self.loop_condition.notify()
+        self.dirscan_speed = cfg.dirscan_speed()
+
+        self.start_scanner()
 
     def stop(self):
         """Stop the dir scanner"""
-        self.shutdown = True
-        with self.loop_condition:
-            self.loop_condition.notify()
+        if self.loop:
+            asyncio.run_coroutine_threadsafe(self.shutdown(), self.loop)
 
     def save(self):
         """Save dir scanner bookkeeping"""
@@ -115,99 +109,168 @@ class DirScanner(threading.Thread):
     def run(self):
         """Start the scanner"""
         logging.info("Dirscanner starting up")
-        self.shutdown = False
 
-        while not self.shutdown:
-            # Wait to be woken up or triggered
-            with self.loop_condition:
-                self.loop_condition.wait(self.dirscan_speed)
-            if self.dirscan_speed and not self.shutdown:
-                self.scan()
+        self.loop = asyncio.new_event_loop()
 
-    def scan(self):
-        """Do one scan of the watched folder"""
+        try:
+            self.start_scanner()
+            self.loop.run_forever()
+        finally:
+            self.loop.close()
 
-        def run_dir(folder, catdir):
-            try:
-                files = os.listdir(folder)
-            except OSError:
-                if not self.error_reported and not catdir:
-                    logging.error(T("Cannot read Watched Folder %s"), filesystem.clip_path(folder))
-                    self.error_reported = True
-                files = []
+    def start_scanner(self):
+        """Start the scanner if it is not already running"""
+        with DIR_SCANNER_LOCK:
+            if not self.loop:
+                logging.debug("Can not start scanner because loop not found")
+                return
 
-            for filename in files:
-                if self.shutdown:
-                    break
-                path = os.path.join(folder, filename)
-                if os.path.isdir(path) or path in self.ignored or filename[0] == ".":
-                    continue
+            if not self.scanner_task or self.scanner_task.done():
+                self.scanner_task = asyncio.run_coroutine_threadsafe(self.scanner(), self.loop)
 
-                if filesystem.get_ext(path) in VALID_NZB_FILES + VALID_ARCHIVES:
-                    try:
-                        stat_tuple = os.stat(path)
-                    except OSError:
+    def get_suspected_files(
+        self, folder: str, catdir: Optional[str] = None
+    ) -> Generator[Tuple[str, Optional[str], Optional[os.stat_result]], None, None]:
+        """Generator listing possible paths to NZB files"""
+
+        if catdir is None:
+            cats = config.get_categories()
+        else:
+            cats = {}
+
+        try:
+            with os.scandir(os.path.join(folder, catdir or "")) as it:
+                for entry in it:
+                    path = entry.path
+
+                    if path in self.ignored:
+                        # We still need to know that an ignored file is still present when we clean up
+                        yield path, catdir, None
                         continue
-                else:
-                    self.ignored[path] = 1
-                    continue
 
-                if path in self.suspected:
-                    if compare_stat_tuple(self.suspected[path], stat_tuple):
-                        # Suspected file still has the same attributes
+                    # If the entry is a catdir then recursion
+                    if entry.is_dir():
+                        if not catdir and entry.name.lower() in cats:
+                            yield from self.get_suspected_files(folder, entry.name)
                         continue
-                    else:
-                        del self.suspected[path]
 
-                if stat_tuple.st_size > 0:
-                    logging.info("Trying to import %s", path)
-
-                    # Wait until the attributes are stable for 1 second, but give up after 3 sec
-                    # This indicates that the file is fully written to disk
-                    for n in range(3):
-                        time.sleep(1.0)
+                    if filesystem.get_ext(path) in VALID_EXTENSIONS:
                         try:
-                            stat_tuple_tmp = os.stat(path)
+                            # https://docs.python.org/3/library/os.html#os.DirEntry.stat
+                            # On Windows, the st_ino, st_dev and st_nlink attributes of the stat_result are always set
+                            # to zero. Call os.stat() to get these attributes.
+                            if sabnzbd.WIN32:
+                                stat_tuple = os.stat(path)
+                            else:
+                                stat_tuple = entry.stat()
                         except OSError:
                             continue
-                        if compare_stat_tuple(stat_tuple, stat_tuple_tmp):
-                            break
-                        stat_tuple = stat_tuple_tmp
-                    else:
-                        # Not stable
-                        continue
-
-                    # Add the NZB's
-                    res, _ = sabnzbd.nzbparser.add_nzbfile(path, catdir=catdir, keep=False)
-                    if res < 0:
-                        # Retry later, for example when we can't read the file
-                        self.suspected[path] = stat_tuple
-                    elif res == 0:
-                        self.error_reported = False
                     else:
                         self.ignored[path] = 1
+                        yield path, catdir, None
+                        continue
+
+                    if path in self.suspected:
+                        if not compare_stat_tuple(self.suspected[path], stat_tuple):
+                            # Suspected file attributes have changed
+                            del self.suspected[path]
+
+                    yield path, catdir, stat_tuple
+        except:
+            if not self.error_reported and not catdir:
+                logging.error(T("Cannot read Watched Folder %s"), filesystem.clip_path(folder))
+                logging.info("Traceback: ", exc_info=True)
+                self.error_reported = True
+
+    async def when_stable_add_nzbfile(self, path: str, catdir: Optional[str], stat_tuple: os.stat_result):
+        """Try and import the NZB but wait until the attributes are stable for 1 second, but give up after 3 sec"""
+
+        logging.info("Trying to import %s", path)
+
+        # Wait until the attributes are stable for 1 second, but give up after 3 sec
+        # This indicates that the file is fully written to disk
+        for n in range(3):
+            await asyncio.sleep(1.0)
+
+            try:
+                stat_tuple_tmp = os.stat(path)
+            except OSError:
+                continue
+            if compare_stat_tuple(stat_tuple, stat_tuple_tmp):
+                break
+            stat_tuple = stat_tuple_tmp
+        else:
+            # Not stable
+            return
+
+        # Add the NZB's
+        res, _ = sabnzbd.nzbparser.add_nzbfile(path, catdir=catdir, keep=False)
+        if res < 0:
+            # Retry later, for example when we can't read the file
+            self.suspected[path] = stat_tuple
+        elif res == 0:
+            self.error_reported = False
+        else:
+            self.ignored[path] = 1
+
+    def scan(self):
+        """Schedule a scan of the watched folder"""
+        if not self.loop:
+            return
+
+        if not (dirscan_dir := self.dirscan_dir):
+            return
+
+        asyncio.run_coroutine_threadsafe(self.scan_async(dirscan_dir), self.loop)
+
+    async def scan_async(self, dirscan_dir: str):
+        """Do one scan of the watched folder"""
+        async with self.lock:
+            if sabnzbd.PAUSED_ALL:
+                return
+
+            files: Set[str] = set()
+            futures: Set[asyncio.Task] = set()
+
+            for path, catdir, stat_tuple in self.get_suspected_files(dirscan_dir):
+                files.add(path)
+
+                if path in self.ignored or path in self.suspected:
+                    continue
+
+                if stat_tuple.st_size > 0:
+                    futures.add(asyncio.create_task(self.when_stable_add_nzbfile(path, catdir, stat_tuple)))
+                    await asyncio.sleep(0)
 
             # Remove files from the bookkeeping that are no longer on the disk
-            clean_file_list(self.ignored, folder, files)
-            clean_file_list(self.suspected, folder, files)
+            # Wait for the paths found in this scan to finish
+            await asyncio.gather(clean_file_list(self.ignored, files), clean_file_list(self.suspected, files), *futures)
 
-        if not self.busy:
-            self.busy = True
-            dirscan_dir = self.dirscan_dir
-            if dirscan_dir and not sabnzbd.PAUSED_ALL:
-                run_dir(dirscan_dir, None)
+    async def scanner(self):
+        """Periodically scan the directory and add NZB files to the queue"""
+        while True:
+            if not (dirscan_speed := self.dirscan_speed):
+                break
 
-                try:
-                    dirscan_list = os.listdir(dirscan_dir)
-                except OSError:
-                    if not self.error_reported:
-                        logging.error(T("Cannot read Watched Folder %s"), filesystem.clip_path(dirscan_dir))
-                        self.error_reported = True
-                    dirscan_list = []
+            if not (dirscan_dir := self.dirscan_dir):
+                break
 
-                cats = config.get_categories()
-                for dd in dirscan_list:
-                    dpath = os.path.join(dirscan_dir, dd)
-                    if os.path.isdir(dpath) and dd.lower() in cats:
-                        run_dir(dpath, dd.lower())
-            self.busy = False
+            await self.scan_async(dirscan_dir)
+
+            await asyncio.sleep(dirscan_speed)
+
+    async def shutdown(self):
+        """Cancel all tasks and stop the loop"""
+        loop = asyncio.get_event_loop()
+
+        # Get all tasks except for this one
+        tasks = filter(lambda task: task is not asyncio.current_task(), asyncio.all_tasks())
+
+        # Cancel them all
+        for task in tasks:
+            task.cancel()
+
+        # Wait for the tasks to be done
+        await asyncio.gather(*tasks, return_exceptions=True)
+
+        loop.stop()

sabnzbd/downloader.py

@@ -29,14 +29,16 @@ import random
 import sys
 import ssl
 from typing import List, Dict, Optional, Union
+from concurrent.futures import ThreadPoolExecutor
 
 import sabnzbd
 from sabnzbd.decorators import synchronized, NzbQueueLocker, DOWNLOADER_CV
 from sabnzbd.newswrapper import NewsWrapper, NNTPPermanentError
 import sabnzbd.config as config
 import sabnzbd.cfg as cfg
-from sabnzbd.misc import from_units, nntp_to_msg, get_server_addrinfo, helpful_warning, int_conv
+from sabnzbd.misc import from_units, get_server_addrinfo, helpful_warning, int_conv
 from sabnzbd.utils.happyeyeballs import happyeyeballs
+from sabnzbd.constants import SOFT_QUEUE_LIMIT
 
 
 # Timeout penalty in minutes for each cause
@@ -55,6 +57,8 @@ _SERVER_CHECK_DELAY = 0.5
 _BPSMETER_UPDATE_DELAY = 0.05
 # How many articles should be prefetched when checking the next articles?
 _ARTICLE_PREFETCH = 20
+# Minimum expected size of TCP receive buffer
+_DEFAULT_CHUNK_SIZE = 65536
 
 TIMER_LOCK = RLock()
 
@@ -217,6 +221,28 @@ class Server:
             self.request = True
             Thread(target=self._request_info_internal).start()
 
+    def get_article(self):
+        """Get article from pre-fetched and pre-fetch new ones if necessary.
+        Articles that are too old for this server are immediately marked as tried"""
+        if self.article_queue:
+            return self.article_queue.pop(0)
+        elif self.next_article_search < time.time():
+            # Pre-fetch new articles
+            self.article_queue = sabnzbd.NzbQueue.get_articles(self, sabnzbd.Downloader.servers, _ARTICLE_PREFETCH)
+            if self.article_queue:
+                article = self.article_queue.pop(0)
+                # Mark expired articles as tried on this server
+                if self.retention and article.nzf.nzo.avg_stamp < time.time() - self.retention:
+                    sabnzbd.Downloader.decode(article)
+                    while self.article_queue:
+                        sabnzbd.Downloader.decode(self.article_queue.pop())
+                else:
+                    return article
+            else:
+                # No available articles, skip this server for a short time
+                self.next_article_search = time.time() + _SERVER_CHECK_DELAY
+        return None
+
     def reset_article_queue(self):
         logging.debug("Resetting article queue for %s", self)
         for article in self.article_queue:
@@ -249,6 +275,8 @@ class Downloader(Thread):
         "bandwidth_limit",
         "bandwidth_perc",
         "sleep_time",
+        "recv_pool",
+        "recv_threads",
         "paused_for_postproc",
         "shutdown",
         "server_restarts",
@@ -278,6 +306,10 @@ class Downloader(Thread):
         self.sleep_time_set()
         cfg.downloader_sleep_time.callback(self.sleep_time_set)
 
+        self.recv_threads: int = cfg.receive_threads()
+        self.recv_pool: Optional[ThreadPoolExecutor] = ThreadPoolExecutor(self.recv_threads)
+        logging.debug("Receive threads: %s", self.recv_threads)
+
         self.paused_for_postproc: bool = False
         self.shutdown: bool = False
 
@@ -502,7 +534,7 @@ class Downloader(Thread):
             # Make sure server address resolution is refreshed
             server.info = None
 
-    def decode(self, article, raw_data: Optional[List[bytes]] = None, data_size: Optional[int] = None):
+    def decode(self, article, raw_data: Optional[bytearray] = None, raw_data_size: Optional[int] = None):
         """Decode article and check the status of
         the decoder and the assembler
         """
@@ -517,33 +549,37 @@ class Downloader(Thread):
             return
 
         # Send to decoder-queue
-        sabnzbd.Decoder.process(article, raw_data, data_size)
+        sabnzbd.Decoder.process(article, raw_data, raw_data_size)
 
         # See if we need to delay because the queues are full
         logged_counter = 0
-        decoder_full = sabnzbd.Decoder.queue_full()
-        assembler_full = sabnzbd.Assembler.queue_full()
-        while not self.shutdown and (decoder_full or assembler_full):
-            # Only log/update once every second, to not waste any CPU-cycles
-            if not logged_counter % 10:
-                # Make sure the BPS-meter is updated
-                sabnzbd.BPSMeter.update()
 
-                # Update who is delaying us
-                sabnzbd.BPSMeter.delayed_decoder += int(decoder_full)
-                sabnzbd.BPSMeter.delayed_assembler += int(assembler_full)
-                logging.debug(
-                    "Delayed - %d seconds - Decoder queue: %d - Assembler queue: %d",
-                    logged_counter / 10,
-                    sabnzbd.Decoder.decoder_queue.qsize(),
-                    sabnzbd.Assembler.queue.qsize(),
-                )
+        decoder_level = sabnzbd.Decoder.queue_level()
+        assembler_level = sabnzbd.Assembler.queue_level()
 
-            # Wait and update the queue sizes
-            time.sleep(0.1)
-            logged_counter += 1
-            decoder_full = sabnzbd.Decoder.queue_full()
-            assembler_full = sabnzbd.Assembler.queue_full()
+        # Sleep for an increasing amount of time, depending on queue sizes.
+        if decoder_level > SOFT_QUEUE_LIMIT or assembler_level > SOFT_QUEUE_LIMIT:
+            time.sleep((decoder_level + assembler_level - SOFT_QUEUE_LIMIT) / 2)
+            sabnzbd.BPSMeter.delayed_decoder += int(decoder_level > SOFT_QUEUE_LIMIT)
+            sabnzbd.BPSMeter.delayed_assembler += int(assembler_level > SOFT_QUEUE_LIMIT)
+
+            while not self.shutdown and (sabnzbd.Decoder.queue_level() >= 1 or sabnzbd.Assembler.queue_level() >= 1):
+                # Only log/update once every second, to not waste any CPU-cycles
+                if not logged_counter % 10:
+                    # Make sure the BPS-meter is updated
+                    sabnzbd.BPSMeter.update()
+
+                    # Update who is delaying us
+                    logging.debug(
+                        "Delayed - %d seconds - Decoder queue: %d - Assembler queue: %d",
+                        logged_counter / 10,
+                        sabnzbd.Decoder.decoder_queue.qsize(),
+                        sabnzbd.Assembler.queue.qsize(),
+                    )
+
+                # Wait and update the queue sizes
+                time.sleep(0.1)
+                logged_counter += 1
 
     def run(self):
         # First check IPv6 connectivity
@@ -563,10 +599,13 @@ class Downloader(Thread):
         BPSMeter.update()
         next_bpsmeter_update = 0
 
-        # can_be_slowed variables
-        can_be_slowed: Optional[float] = None
-        can_be_slowed_timer: float = 0.0
-        next_stable_speed_check: float = 0.0
+        # Sleep check variables
+        last_max_chunk_size: int = 0
+        max_chunk_size: int = _DEFAULT_CHUNK_SIZE
+        # Debugging code for v4 test release
+        sleep_count_start: float = time.time()
+        sleep_count: int = 0
+        time_slept: float = 0
 
         # Check server expiration dates
         check_server_expiration()
@@ -632,32 +671,13 @@ class Downloader(Thread):
                             server.request_info()
                         break
 
-                    # Get article from pre-fetched ones or fetch new ones
-                    if server.article_queue:
-                        article = server.article_queue.pop(0)
-                    else:
-                        # Pre-fetch new articles
-                        server.article_queue = sabnzbd.NzbQueue.get_articles(server, self.servers, _ARTICLE_PREFETCH)
-                        if server.article_queue:
-                            article = server.article_queue.pop(0)
-                            # Mark expired articles as tried on this server
-                            if server.retention and article.nzf.nzo.avg_stamp < now - server.retention:
-                                self.decode(article)
-                                while server.article_queue:
-                                    self.decode(server.article_queue.pop())
-                                # Move to the next server, allowing the next server to already start
-                                # fetching the articles that were too old for this server
-                                break
-                        else:
-                            # Skip this server for a short time
-                            server.next_article_search = now + _SERVER_CHECK_DELAY
-                            break
+                    nw.article = server.get_article()
+                    if not nw.article:
+                        break
 
                     server.idle_threads.remove(nw)
                     server.busy_threads.append(nw)
 
-                    nw.article = article
-
                     if nw.connected:
                         self.__request_article(nw)
                     else:
@@ -694,43 +714,44 @@ class Downloader(Thread):
                     logging.info("Shutting down")
                     break
 
+            # If less data than possible was received then it should be ok to sleep a bit
+            if self.sleep_time:
+                if last_max_chunk_size > max_chunk_size:
+                    logging.debug("New max_chunk_size %d -> %d", max_chunk_size, last_max_chunk_size)
+                    max_chunk_size = last_max_chunk_size
+                elif last_max_chunk_size < max_chunk_size / 3:
+                    time_before = time.time()
+                    time.sleep(self.sleep_time)
+                    now = time.time()
+                    # Debugging code for v4 test release
+                    if now - time_before > self.sleep_time + 0.02:
+                        logging.debug("Slept %.5f seconds, sleep_time = %s", now - time_before, self.sleep_time)
+                    time_slept += now - time_before
+                    sleep_count += 1
+                    if sleep_count_start + 20 < now:
+                        if sleep_count > 21:
+                            logging.debug(
+                                "Slept %d times for an average of %.5f seconds the last %.2f seconds. sleep_time = %s",
+                                sleep_count,
+                                time_slept / sleep_count,
+                                now - sleep_count_start,
+                                self.sleep_time,
+                            )
+                        sleep_count_start = now
+                        sleep_count = 0
+                        time_slept = 0
+
+                last_max_chunk_size = 0
+
             # Use select to find sockets ready for reading/writing
             readkeys = self.read_fds.keys()
             if readkeys:
                 read, _, _ = select.select(readkeys, (), (), 1.0)
-
-                # Add a sleep if there are too few results compared to the number of active connections
-                if self.sleep_time:
-                    if can_be_slowed and len(read) < 1 + len(readkeys) / 10:
-                        time.sleep(self.sleep_time)
-
-                    # Initialize by waiting for stable speed and then enable sleep
-                    if can_be_slowed is None or can_be_slowed_timer:
-                        # Wait for stable speed to start testing
-
-                        if not can_be_slowed_timer and now > next_stable_speed_check:
-                            if BPSMeter.get_stable_speed(timespan=10):
-                                can_be_slowed_timer = now + 8
-                                can_be_slowed = 1
-                            else:
-                                next_stable_speed_check = now + _BPSMETER_UPDATE_DELAY
-
-                        # Check 10 seconds after enabling slowdown
-                        if can_be_slowed_timer and now > can_be_slowed_timer:
-                            # Now let's check if it was stable in the last 10 seconds
-                            can_be_slowed = BPSMeter.get_stable_speed(timespan=10)
-                            can_be_slowed_timer = 0
-                            if not can_be_slowed:
-                                self.sleep_time = 0
-                            logging.debug("Downloader-slowdown: %r", can_be_slowed)
-
             else:
                 read = []
-
                 BPSMeter.reset()
-
                 time.sleep(1.0)
-
+                max_chunk_size = _DEFAULT_CHUNK_SIZE
                 with DOWNLOADER_CV:
                     while (
                         (sabnzbd.NzbQueue.is_empty() or self.no_active_jobs() or self.paused_for_postproc)
@@ -747,92 +768,119 @@ class Downloader(Thread):
             if not read:
                 continue
 
-            for selected in read:
-                nw = self.read_fds[selected]
-                article = nw.article
-                server = nw.server
+            if self.recv_threads > 1:
+                for nw, bytes_received, done in self.recv_pool.map(self.__recv, read):
+                    self.__handle_recv_result(nw, bytes_received, done)
+                if self.bandwidth_limit:
+                    self.__check_speed()
+            else:
+                for selected in read:
+                    nw, bytes_received, done = self.__recv(selected)
+                    self.__handle_recv_result(nw, bytes_received, done)
+                    if self.bandwidth_limit and bytes_received:
+                        self.__check_speed()
 
-                try:
-                    bytes_received, done = nw.recv_chunk()
-                except ssl.SSLWantReadError:
-                    continue
-                except:
-                    self.__reset_nw(nw, "server closed connection", wait=False)
-                    continue
+    def __recv(self, selected):
+        nw = None
+        try:
+            nw = self.read_fds[selected]
+            bytes_received, done = nw.recv_chunk()
+            return nw, bytes_received, done
+        except ssl.SSLWantReadError:
+            return nw, 0, False
+        except:
+            return nw, 0, True
 
-                BPSMeter.update(server.id, bytes_received)
-                if self.bandwidth_limit and BPSMeter.bps + BPSMeter.sum_cached_amount > self.bandwidth_limit:
-                    BPSMeter.update()
-                    while BPSMeter.bps > self.bandwidth_limit:
-                        time.sleep(0.01)
-                        BPSMeter.update()
+    def __check_speed(self):
+        BPSMeter = sabnzbd.BPSMeter
+        if BPSMeter.bps + BPSMeter.sum_cached_amount > self.bandwidth_limit:
+            BPSMeter.update()
+            while BPSMeter.bps > self.bandwidth_limit:
+                time.sleep(0.01)
+                BPSMeter.update()
 
-                if nw.status_code != 222 and not done:
-                    if not nw.connected or nw.status_code == 480:
-                        if not self.__finish_connect_nw(nw):
-                            continue
-                        if nw.connected:
-                            logging.info("Connecting %s@%s finished", nw.thrdnum, nw.server.host)
-                            self.__request_article(nw)
+    def __handle_recv_result(self, nw: NewsWrapper, bytes_received: int = 0, done: bool = False):
+        if not bytes_received:
+            if done:
+                self.__reset_nw(nw, "server closed connection", wait=False)
+            return
 
-                    elif nw.status_code == 223:
-                        done = True
-                        logging.debug("Article <%s> is present", article.article)
+        article = nw.article
+        server = nw.server
+        sabnzbd.BPSMeter.update(server.id, bytes_received)
 
-                    elif nw.status_code == 211:
-                        logging.debug("group command ok -> %s", nntp_to_msg(nw.data))
-                        nw.group = nw.article.nzf.nzo.group
-                        nw.clear_data()
-                        self.__request_article(nw)
+        if nw.status_code != 222 and not done:
+            if not nw.connected or nw.status_code == 480:
+                if not self.__finish_connect_nw(nw):
+                    return
+                if nw.connected:
+                    logging.info("Connecting %s@%s finished", nw.thrdnum, nw.server.host)
+                    self.__request_article(nw)
 
-                    elif nw.status_code in (411, 423, 430):
-                        done = True
-                        logging.debug(
-                            "Thread %s@%s: Article %s missing (error=%s)",
-                            nw.thrdnum,
-                            nw.server.host,
-                            article.article,
-                            nw.status_code,
-                        )
-                        nw.clear_data()
+            elif nw.status_code == 223:
+                done = True
+                logging.debug("Article <%s> is present", article.article)
 
-                    elif nw.status_code == 500:
-                        if article.nzf.nzo.precheck:
-                            # Assume "STAT" command is not supported
-                            server.have_stat = False
-                            logging.debug("Server %s does not support STAT", server.host)
-                        else:
-                            # Assume "BODY" command is not supported
-                            server.have_body = False
-                            logging.debug("Server %s does not support BODY", server.host)
-                        nw.clear_data()
-                        self.__request_article(nw)
+            elif nw.status_code == 211:
+                logging.debug("group command ok -> %s", nw.nntp_msg)
+                nw.group = nw.article.nzf.nzo.group
+                nw.reset_data_buffer()
+                self.__request_article(nw)
 
-                if done:
-                    # Successful data, clear "bad" counter
-                    server.bad_cons = 0
-                    server.errormsg = server.warning = ""
+            elif nw.status_code in (411, 423, 430):
+                done = True
+                logging.debug(
+                    "Thread %s@%s: Article %s missing (error=%s)",
+                    nw.thrdnum,
+                    nw.server.host,
+                    article.article,
+                    nw.status_code,
+                )
+                nw.reset_data_buffer()
 
-                    # Update statistics and decode
-                    article.nzf.nzo.update_download_stats(BPSMeter.bps, server.id, nw.data_size)
-                    self.decode(article, nw.data, nw.data_size)
+            elif nw.status_code == 500:
+                if article.nzf.nzo.precheck:
+                    # Assume "STAT" command is not supported
+                    server.have_stat = False
+                    logging.debug("Server %s does not support STAT", server.host)
+                else:
+                    # Assume "BODY" command is not supported
+                    server.have_body = False
+                    logging.debug("Server %s does not support BODY", server.host)
+                nw.reset_data_buffer()
+                self.__request_article(nw)
 
-                    if sabnzbd.LOG_ALL:
-                        logging.debug("Thread %s@%s: %s done", nw.thrdnum, server.host, article.article)
+        if done:
+            # Successful data, clear "bad" counter
+            server.bad_cons = 0
+            server.errormsg = server.warning = ""
 
-                    # Reset connection for new activity
-                    nw.soft_reset()
-                    server.busy_threads.remove(nw)
-                    server.idle_threads.append(nw)
-                    self.remove_socket(nw)
+            # Update statistics and decode
+            article.nzf.nzo.update_download_stats(sabnzbd.BPSMeter.bps, server.id, nw.data_position)
+            self.decode(article, nw.get_data_buffer(), nw.data_position)
+
+            if sabnzbd.LOG_ALL:
+                logging.debug("Thread %s@%s: %s done", nw.thrdnum, server.host, article.article)
+
+            # Reset connection for new activity
+            nw.soft_reset()
+            # Request a new article immediately if possible
+            if nw.connected and server.active and not (self.paused or self.shutdown or self.paused_for_postproc):
+                nw.article = server.get_article()
+                if nw.article:
+                    self.__request_article(nw)
+                    return
+            server.busy_threads.remove(nw)
+            server.idle_threads.append(nw)
+            self.remove_socket(nw)
 
     def __finish_connect_nw(self, nw: NewsWrapper) -> bool:
         server = nw.server
         try:
             nw.finish_connect(nw.status_code)
             if sabnzbd.LOG_ALL:
-                logging.debug("%s@%s last message -> %s", nw.thrdnum, server.host, nntp_to_msg(nw.data))
-            nw.clear_data()
+                logging.debug("%s@%s last message -> %s", nw.thrdnum, server.host, nw.nntp_msg)
+            nw.reset_data_buffer()
         except NNTPPermanentError as error:
             # Handle login problems
             block = False
@@ -901,7 +949,7 @@ class Downloader(Thread):
                 T("Connecting %s@%s failed, message=%s"),
                 nw.thrdnum,
                 nw.server.host,
-                nntp_to_msg(nw.data),
+                nw.nntp_msg,
             )
             # No reset-warning needed, above logging is sufficient
             self.__reset_nw(nw, retry_article=False)

sabnzbd/encoding.py

@@ -36,7 +36,7 @@ def utob(str_in: AnyStr) -> bytes:
 
 def ubtou(str_in: AnyStr) -> str:
     """Shorthand for converting unicode bytes to UTF-8 string"""
-    if not isinstance(str_in, bytes):
+    if isinstance(str_in, str):
         return str_in
     return str_in.decode("utf-8")
 

sabnzbd/filesystem.py

@@ -67,8 +67,7 @@ def is_listed_ext(ext: str, ext_list: list) -> bool:
     thus return false for extensions such as 'r007' despite the substring match on 'r00').
     """
     for item in ext_list:
-        RE_EXT = sabnzbd.misc.convert_filter(item)
-        if RE_EXT:
+        if RE_EXT := sabnzbd.misc.convert_filter(item):
             try:
                 if len(RE_EXT.match(ext).group()) == len(ext):
                     return True
@@ -121,6 +120,14 @@ def is_writable(path: str) -> bool:
         return True
 
 
+def is_size(filepath: str, size: int) -> bool:
+    """Return True if filepath exists and is specified size"""
+    try:
+        return os.path.getsize(filepath) == size
+    except:
+        return False
+
+
 _DEVICES = (
     "con",
     "prn",
@@ -177,10 +184,13 @@ def has_win_device(filename: str) -> bool:
     return False
 
 
-CH_ILLEGAL = "/"
-CH_LEGAL = "+"
-CH_ILLEGAL_WIN = '\\/<>?*|"\t:'
-CH_LEGAL_WIN = "++{}!@#'+-"
+CH_ILLEGAL = "\0/"
+CH_LEGAL = "_+"
+CH_ILLEGAL_WIN = '\\/<>?*|":'
+CH_LEGAL_WIN = "++{}!@#'-"
+for i in range(1, 32):
+    CH_ILLEGAL_WIN += chr(i)
+    CH_LEGAL_WIN += "_"
 
 
 def sanitize_filename(name: str) -> str:
@@ -614,8 +624,7 @@ def set_chmod(path: str, permissions: int, allow_failures: bool = False):
 def set_permissions(path: str, recursive: bool = True):
     """Give folder tree and its files their proper permissions"""
     if not sabnzbd.WIN32:
-        custom_permissions = sabnzbd.cfg.permissions()
-        if custom_permissions:
+        if custom_permissions := sabnzbd.cfg.permissions():
             # If user set permissions, parse them
             custom_permissions = int(custom_permissions, 8)
 
@@ -1212,8 +1221,7 @@ def backup_exists(filename: str) -> bool:
 
 def backup_nzb(nzb_path: str):
     """Backup NZB file, return path to nzb if it was saved"""
-    nzb_backup_dir = sabnzbd.cfg.nzb_backup_dir.get_path()
-    if nzb_backup_dir:
+    if nzb_backup_dir := sabnzbd.cfg.nzb_backup_dir.get_path():
         logging.debug("Saving copy of %s in %s", get_filename(nzb_path), nzb_backup_dir)
         shutil.copy(nzb_path, nzb_backup_dir)
 

sabnzbd/interface.py

@@ -70,7 +70,13 @@ import sabnzbd.newsunpack
 from sabnzbd.utils.servertests import test_nntp_server_dict
 from sabnzbd.utils.getperformance import getcpu
 import sabnzbd.utils.ssdp
-from sabnzbd.constants import DEF_STD_CONFIG, DEFAULT_PRIORITY, CHEETAH_DIRECTIVES, EXCLUDED_GUESSIT_PROPERTIES
+from sabnzbd.constants import (
+    DEF_STD_CONFIG,
+    DEFAULT_PRIORITY,
+    CHEETAH_DIRECTIVES,
+    EXCLUDED_GUESSIT_PROPERTIES,
+    DEF_HTTPS_CERT_FILE,
+)
 from sabnzbd.lang import list_languages
 from sabnzbd.api import (
     list_scripts,
@@ -420,7 +426,7 @@ class MainPage:
             info["have_watched_dir"] = bool(cfg.dirscan_dir())
 
             info["cpumodel"] = getcpu()
-            info["cpusimd"] = sabnzbd.decoder.SABYENC_SIMD
+            info["cpusimd"] = sabnzbd.decoder.SABCTOOLS_SIMD
 
             # Have logout only with HTML and if inet=5, only when we are external
             info["have_logout"] = (
@@ -519,7 +525,7 @@ class Wizard:
             info["username"] = ""
             info["password"] = ""
             info["connections"] = ""
-            info["ssl"] = 0
+            info["ssl"] = 1
             info["ssl_verify"] = 2
         else:
             # Sort servers to get the first enabled one
@@ -676,7 +682,7 @@ class ConfigPage:
 
         conf["have_unzip"] = bool(sabnzbd.newsunpack.ZIP_COMMAND)
         conf["have_7zip"] = bool(sabnzbd.newsunpack.SEVENZIP_COMMAND)
-        conf["have_sabyenc"] = sabnzbd.decoder.SABYENC_ENABLED
+        conf["have_sabctools"] = sabnzbd.decoder.SABCTOOLS_ENABLED
         conf["have_mt_par2"] = sabnzbd.newsunpack.PAR2_MT
 
         conf["certificate_validation"] = sabnzbd.CERTIFICATE_VALIDATION
@@ -880,6 +886,7 @@ SPECIAL_VALUE_LIST = (
     "wait_ext_drive",
     "max_foldername_length",
     "url_base",
+    "receive_threads",
     "num_simd_decoders",
     "direct_unpack_threads",
     "ipv6_servers",
@@ -979,6 +986,7 @@ class ConfigGeneral:
 
         conf["language"] = cfg.language()
         conf["lang_list"] = list_languages()
+        conf["def_https_cert_file"] = DEF_HTTPS_CERT_FILE
 
         for kw in GENERAL_LIST:
             conf[kw] = config.get_config("misc", kw)()

sabnzbd/misc.py

@@ -1058,20 +1058,6 @@ def match_str(text: AnyStr, matches: Tuple[AnyStr, ...]) -> Optional[AnyStr]:
     return None
 
 
-def nntp_to_msg(text: Union[List[AnyStr], str]) -> str:
-    """Format raw NNTP bytes data for display"""
-    if isinstance(text, list):
-        text = text[0]
-
-    # Only need to split if it was raw data
-    # Sometimes (failed login) we put our own texts
-    if not isinstance(text, bytes):
-        return text
-    else:
-        lines = text.split(b"\r\n")
-        return ubtou(lines[0])
-
-
 def recursive_html_escape(input_dict_or_list: Union[Dict[str, Any], List], exclude_items: Tuple[str, ...] = ()):
     """Recursively update the input_dict in-place with html-safe values"""
     if isinstance(input_dict_or_list, (dict, list)):

sabnzbd/newsunpack.py

@@ -61,6 +61,7 @@ from sabnzbd.filesystem import (
     build_filelists,
     get_filename,
     SEVENMULTI_RE,
+    is_size,
 )
 from sabnzbd.nzbstuff import NzbObject
 from sabnzbd.sorting import SeriesSorter
@@ -1975,7 +1976,7 @@ def rar_sort(a: str, b: str) -> int:
 
 
 def quick_check_set(setname: str, nzo: NzbObject) -> bool:
-    """Check all on-the-fly md5sums of a set"""
+    """Check all on-the-fly crc32s of a set"""
     par2pack = nzo.par2packs.get(setname)
     if par2pack is None:
         return False
@@ -1997,7 +1998,11 @@ def quick_check_set(setname: str, nzo: NzbObject) -> bool:
             # Do a simple filename based check
             if file == nzf.filename:
                 found = True
-                if (nzf.md5sum is not None) and nzf.md5sum == par2info.filehash:
+                if (
+                    nzf.crc32 is not None
+                    and nzf.crc32 == par2info.filehash
+                    and is_size(nzf.filepath, par2info.filesize)
+                ):
                     logging.debug("Quick-check of file %s OK", file)
                     result &= True
                 elif file_to_ignore:
@@ -2010,7 +2015,7 @@ def quick_check_set(setname: str, nzo: NzbObject) -> bool:
                 break
 
             # Now let's do obfuscation check
-            if nzf.md5sum == par2info.filehash:
+            if nzf.crc32 is not None and nzf.crc32 == par2info.filehash and is_size(nzf.filepath, par2info.filesize):
                 try:
                     logging.debug("Quick-check will rename %s to %s", nzf.filename, file)
 
@@ -2155,9 +2160,10 @@ def sfv_check(sfvs: List[str], nzo: NzbObject) -> bool:
     verifytotal = len(nzo.finished_files)
     verifynum = 0
     for nzf in nzf_list:
-        verifynum += 1
-        nzo.set_action_line(T("Verifying"), "%02d/%02d" % (verifynum, verifytotal))
-        calculated_crc32[nzf.filename] = crc_calculate(os.path.join(nzo.download_path, nzf.filename))
+        if nzf.crc32 is not None:
+            verifynum += 1
+            nzo.set_action_line(T("Verifying"), "%02d/%02d" % (verifynum, verifytotal))
+            calculated_crc32[nzf.filename] = b"%08x" % (nzf.crc32 & 0xFFFFFFFF)
 
     sfv_parse_results = {}
     nzo.set_action_line(T("Trying SFV verification"), "...")
@@ -2176,7 +2182,7 @@ def sfv_check(sfvs: List[str], nzo: NzbObject) -> bool:
             # Do a simple filename based check
             if file == nzf.filename:
                 found = True
-                if nzf.filename in calculated_crc32 and calculated_crc32[nzf.filename] == sfv_parse_results[file]:
+                if calculated_crc32.get(nzf.filename, "") == sfv_parse_results[file]:
                     logging.debug("SFV-check of file %s OK", file)
                     result &= True
                 elif file_to_ignore:
@@ -2189,7 +2195,7 @@ def sfv_check(sfvs: List[str], nzo: NzbObject) -> bool:
                 break
 
             # Now lets do obfuscation check
-            if nzf.filename in calculated_crc32 and calculated_crc32[nzf.filename] == sfv_parse_results[file]:
+            if calculated_crc32.get(nzf.filename, "") == sfv_parse_results[file]:
                 try:
                     logging.debug("SFV-check will rename %s to %s", nzf.filename, file)
                     renamer(os.path.join(nzo.download_path, nzf.filename), os.path.join(nzo.download_path, file))
@@ -2205,7 +2211,7 @@ def sfv_check(sfvs: List[str], nzo: NzbObject) -> bool:
         if not found:
             if file_to_ignore:
                 # We don't care about these files
-                logging.debug("SVF-check ignoring missing file %s", file)
+                logging.debug("SFV-check ignoring missing file %s", file)
                 continue
 
             logging.info("Cannot SFV-check missing file %s!", file)
@@ -2234,18 +2240,6 @@ def parse_sfv(sfv_filename):
     return results
 
 
-def crc_calculate(path):
-    """Calculate crc32 of the given file"""
-    crc = 0
-    with open(path, "rb") as fp:
-        while 1:
-            data = fp.read(4096)
-            if not data:
-                break
-            crc = zlib.crc32(data, crc)
-    return b"%08x" % (crc & 0xFFFFFFFF)
-
-
 def add_time_left(perc: float, start_time: Optional[float] = None, time_used: Optional[float] = None) -> str:
     """Calculate time left based on current progress, if it is taking more than 10 seconds"""
     if not time_used:

sabnzbd/newswrapper.py

@@ -25,13 +25,14 @@ from threading import Thread
 import time
 import logging
 import ssl
-from typing import List, Optional, Tuple, AnyStr
+import sabctools
+from typing import Optional, Tuple
 
 import sabnzbd
 import sabnzbd.cfg
-from sabnzbd.constants import DEF_TIMEOUT
-from sabnzbd.encoding import utob
-from sabnzbd.misc import nntp_to_msg, is_ipv4_addr, is_ipv6_addr, get_server_addrinfo
+from sabnzbd.constants import DEF_TIMEOUT, NNTP_BUFFER_SIZE
+from sabnzbd.encoding import utob, ubtou
+from sabnzbd.misc import is_ipv4_addr, is_ipv6_addr, get_server_addrinfo
 
 # Set pre-defined socket timeout
 socket.setdefaulttimeout(DEF_TIMEOUT)
@@ -56,7 +57,8 @@ class NewsWrapper:
         "timeout",
         "article",
         "data",
-        "data_size",
+        "data_view",
+        "data_position",
         "nntp",
         "connected",
         "user_sent",
@@ -65,7 +67,6 @@ class NewsWrapper:
         "user_ok",
         "pass_ok",
         "force_login",
-        "status_code",
     )
 
     def __init__(self, server, thrdnum, block=False):
@@ -75,8 +76,10 @@ class NewsWrapper:
 
         self.timeout: Optional[float] = None
         self.article: Optional[sabnzbd.nzbstuff.Article] = None
-        self.data: List[AnyStr] = []
-        self.data_size: int = 0
+
+        self.data: Optional[bytearray] = None
+        self.data_view: Optional[memoryview] = None
+        self.data_position: int = 0
 
         self.nntp: Optional[NNTP] = None
 
@@ -87,7 +90,15 @@ class NewsWrapper:
         self.pass_ok: bool = False
         self.force_login: bool = False
         self.group: Optional[str] = None
-        self.status_code: Optional[int] = None
+
+    @property
+    def status_code(self) -> Optional[int]:
+        if self.data_position >= 3:
+            return int(self.data[:3])
+
+    @property
+    def nntp_msg(self) -> str:
+        return ubtou(self.data[: self.data_position]).strip()
 
     def init_connect(self):
         """Setup the connection in NNTP object"""
@@ -96,7 +107,10 @@ class NewsWrapper:
         if self.blocking and not self.server.info:
             self.server.info = get_server_addrinfo(self.server.host, self.server.port)
 
-        # Construct NNTP object
+        # Construct buffer and NNTP object
+        self.data = bytearray(NNTP_BUFFER_SIZE)
+        self.data_view = memoryview(self.data)
+        self.reset_data_buffer()
         self.nntp = NNTP(self, self.server.hostip)
         self.timeout = time.time() + self.server.timeout
 
@@ -109,14 +123,6 @@ class NewsWrapper:
             self.pass_sent = True
             self.pass_ok = True
 
-        if code == 501 and self.user_sent:
-            # Change to a sensible text
-            code = 481
-            self.data[0] = "%d %s" % (code, T("Authentication failed, check username/password."))
-            self.status_code = code
-            self.user_ok = True
-            self.pass_sent = True
-
         if code == 480:
             self.force_login = True
             self.connected = False
@@ -126,11 +132,11 @@ class NewsWrapper:
             self.pass_ok = False
 
         if code in (400, 500, 502):
-            raise NNTPPermanentError(nntp_to_msg(self.data), code)
+            raise NNTPPermanentError(self.nntp_msg, code)
         elif not self.user_sent:
             command = utob("authinfo user %s\r\n" % self.server.username)
             self.nntp.sock.sendall(command)
-            self.clear_data()
+            self.reset_data_buffer()
             self.user_sent = True
         elif not self.user_ok:
             if code == 381:
@@ -145,12 +151,12 @@ class NewsWrapper:
         if self.user_ok and not self.pass_sent:
             command = utob("authinfo pass %s\r\n" % self.server.password)
             self.nntp.sock.sendall(command)
-            self.clear_data()
+            self.reset_data_buffer()
             self.pass_sent = True
         elif self.user_ok and not self.pass_ok:
             if code != 281:
                 # Assume that login failed (code 481 or other)
-                raise NNTPPermanentError(nntp_to_msg(self.data), code)
+                raise NNTPPermanentError(self.nntp_msg, code)
             else:
                 self.connected = True
 
@@ -169,62 +175,65 @@ class NewsWrapper:
         else:
             command = utob("ARTICLE <%s>\r\n" % self.article.article)
         self.nntp.sock.sendall(command)
-        self.clear_data()
+        self.reset_data_buffer()
 
     def send_group(self, group: str):
         """Send the NNTP GROUP command"""
         self.timeout = time.time() + self.server.timeout
         command = utob("GROUP %s\r\n" % group)
         self.nntp.sock.sendall(command)
-        self.clear_data()
+        self.reset_data_buffer()
 
     def recv_chunk(self) -> Tuple[int, bool]:
-        """Receive data, return #bytes, done"""
-        if self.nntp.nw.server.ssl:
-            # SSL chunks come in 16K frames
-            # Setting higher limits results in slowdown
-            chunk = self.nntp.sock.recv(16384)
-        else:
-            chunk = self.nntp.sock.recv(262144)
+        """Receive data, return #bytes, done, skip"""
+        # Resize the buffer in the extremely unlikely case that it got full
+        if len(self.data) - self.data_position == 0:
+            self.nntp.nw.increase_data_buffer()
 
-        chunk_len = len(chunk)
-        if chunk_len == 0:
+        # Receive data into the pre-allocated buffer
+        if self.nntp.nw.server.ssl and not self.nntp.nw.blocking and sabctools.openssl_linked:
+            # Use patched version when downloading
+            bytes_recv = sabctools.unlocked_ssl_recv_into(self.nntp.sock, self.data_view[self.data_position :])
+        else:
+            bytes_recv = self.nntp.sock.recv_into(self.data_view[self.data_position :])
+
+        # No data received
+        if bytes_recv == 0:
             raise ConnectionError("server closed connection")
 
-        if not self.data:
-            try:
-                self.status_code = int(chunk[:3])
-            except:
-                self.status_code = None
-
-        # Append so we can do 1 join(), much faster than multiple!
-        self.data.append(chunk)
-        self.data_size += chunk_len
+        # Success, move timeout and internal data position
         self.timeout = time.time() + self.server.timeout
+        self.data_position += bytes_recv
 
-        # Official end-of-article is ".\r\n" but sometimes it can get lost between 2 chunks
-        if chunk[-5:] == b"\r\n.\r\n":
-            return chunk_len, True
-        elif chunk_len < 5 and len(self.data) > 1:
-            # We need to make sure the end is not split over 2 chunks
-            # This is faster than join()
-            if self.data[-2][-5 + chunk_len :] + chunk == b"\r\n.\r\n":
-                return chunk_len, True
+        # Official end-of-article is "\r\n.\r\n",
+        # Using the data directly seems faster than the memoryview
+        if self.data[self.data_position - 5 : self.data_position] == b"\r\n.\r\n":
+            return bytes_recv, True
 
         # Still in middle of data, so continue!
-        return chunk_len, False
+        return bytes_recv, False
 
     def soft_reset(self):
         """Reset for the next article"""
         self.timeout = None
         self.article = None
-        self.clear_data()
+        self.reset_data_buffer()
 
-    def clear_data(self):
-        """Clear the stored raw data"""
-        self.data = []
-        self.data_size = 0
-        self.status_code = None
+    def reset_data_buffer(self):
+        """Reset the data position"""
+        self.data_position = 0
+
+    def increase_data_buffer(self):
+        """Resize the buffer in the extremely unlikely case that it overflows"""
+        new_buffer = bytearray(len(self.data) + NNTP_BUFFER_SIZE)
+        new_buffer[: len(self.data)] = self.data
+        logging.info("Increasing buffer from %d to %d for %s", len(self.data), len(new_buffer), str(self))
+        self.data = new_buffer
+        self.data_view = memoryview(self.data)
+
+    def get_data_buffer(self) -> bytearray:
+        """Get a copy of the data buffer in a new bytes object"""
+        return bytearray(self.data_view[: self.data_position])
 
     def hard_reset(self, wait: bool = True, send_quit: bool = True):
         """Destroy and restart"""

sabnzbd/nzbparser.py

@@ -449,7 +449,6 @@ def nzbfile_parser(full_nzb_path: str, nzo):
                                     logging.info("Skipping duplicate article (%s)", article_id)
                             elif segment_size <= 0 or segment_size >= 2**23:
                                 # Perform sanity check (not negative, 0 or larger than 8MB) on article size
-                                # We use this value later to allocate memory in cache and sabyenc
                                 logging.info("Skipping article %s due to strange size (%s)", article_id, segment_size)
                                 nzo.increase_bad_articles_counter("bad_articles")
                                 bad_articles = True

sabnzbd/nzbqueue.py

@@ -47,7 +47,6 @@ from sabnzbd.constants import (
     VERIFIED_FILE,
     Status,
     IGNORED_FILES_AND_FOLDERS,
-    DIRECT_WRITE_TRIGGER,
 )
 
 import sabnzbd.cfg as cfg
@@ -584,7 +583,8 @@ class NzbQueue:
             logging.info("Sorting by average date... (reversed: %s)", reverse)
             sort_function = lambda nzo: nzo.avg_date
         elif field == "remaining":
-            logging.debug("Sorting by percentage downloaded...")
+            if self.__nzo_list:
+                logging.debug("Sorting by percentage downloaded...")
             sort_function = lambda nzo: nzo.remaining / nzo.bytes if nzo.bytes else 1
         else:
             logging.debug("Sort: %s not recognized", field)
@@ -755,14 +755,14 @@ class NzbQueue:
 
         # Write data if file is done or at trigger time
         # Skip if the file is already queued, since all available articles will then be written
-        if file_done or (
-            articles_left
-            and (articles_left % DIRECT_WRITE_TRIGGER) == 0
-            and not sabnzbd.Assembler.partial_nzf_in_queue(nzf)
+        if (
+            file_done
+            or (article.lowest_partnum and nzf.filename_checked and not nzf.import_finished)
+            or (articles_left and (articles_left % sabnzbd.ArticleCache.assembler_write_trigger) == 0)
         ):
             if not nzo.precheck:
                 # Only start decoding if we have a filename and type
-                # The type is only set if sabyenc could decode the article
+                # The type is only set if sabctools could decode the article
                 if nzf.filename and nzf.type:
                     sabnzbd.Assembler.process(nzo, nzf, file_done)
                 elif nzf.filename.lower().endswith(".par2"):

sabnzbd/nzbstuff.py

@@ -30,6 +30,7 @@ from typing import List, Dict, Any, Tuple, Optional, Union, BinaryIO
 
 # SABnzbd modules
 import sabnzbd
+import sabctools
 from sabnzbd.constants import (
     GIGI,
     ATTRIB_FILE,
@@ -156,7 +157,7 @@ class TryList:
 ##############################################################################
 # Article
 ##############################################################################
-ArticleSaver = ("article", "art_id", "bytes", "lowest_partnum", "decoded", "on_disk", "nzf")
+ArticleSaver = ("article", "art_id", "bytes", "lowest_partnum", "decoded", "on_disk", "nzf", "crc32")
 
 
 class Article(TryList):
@@ -176,6 +177,7 @@ class Article(TryList):
         self.tries: int = 0  # Try count
         self.decoded: bool = False
         self.on_disk: bool = False
+        self.crc32: Optional[int] = None
         self.nzf: NzbFile = nzf
 
     def reset_try_list(self):
@@ -288,7 +290,8 @@ NzbFileSaver = (
     "deleted",
     "valid",
     "import_finished",
-    "md5sum",
+    "crc32",
+    "assembled",
     "md5of16k",
 )
 
@@ -297,7 +300,7 @@ class NzbFile(TryList):
     """Representation of one file consisting of multiple articles"""
 
     # Pre-define attributes to save memory
-    __slots__ = NzbFileSaver + ("md5",)
+    __slots__ = NzbFileSaver
 
     def __init__(self, date, subject, raw_article_db, file_bytes, nzo):
         """Setup object"""
@@ -327,8 +330,8 @@ class NzbFile(TryList):
         self.deleted = False
         self.import_finished = False
 
-        self.md5 = None
-        self.md5sum: Optional[bytes] = None
+        self.crc32: Optional[int] = 0
+        self.assembled: bool = False
         self.md5of16k: Optional[bytes] = None
 
         self.valid: bool = bool(raw_article_db)
@@ -394,6 +397,12 @@ class NzbFile(TryList):
         self.vol = vol
         self.blocks = int_conv(blocks)
 
+    def update_crc32(self, crc32: Optional[int], length: int) -> None:
+        if self.crc32 is None or crc32 is None:
+            self.crc32 = None
+        else:
+            self.crc32 = sabctools.crc32_combine(self.crc32, crc32, length)
+
     def get_articles(self, server: Server, servers: List[Server], fetch_limit: int) -> List[Article]:
         """Get next articles to be downloaded"""
         articles = []
@@ -456,9 +465,6 @@ class NzbFile(TryList):
         if isinstance(self.decodetable, dict):
             self.decodetable = [self.decodetable[partnum] for partnum in sorted(self.decodetable)]
 
-        # Set non-transferable values
-        self.md5 = None
-
     def __eq__(self, other: "NzbFile"):
         """Assume it's the same file if the number bytes and first article
         are the same or if there are no articles left, use the filenames.

sabnzbd/par2file.py

@@ -23,6 +23,7 @@ import logging
 import os
 import re
 import struct
+import sabctools
 from dataclasses import dataclass
 from typing import Dict, Optional, Tuple, BinaryIO
 
@@ -35,6 +36,7 @@ PAR_PKT_ID = b"PAR2\x00PKT"
 PAR_MAIN_ID = b"PAR 2.0\x00Main\x00\x00\x00\x00"
 PAR_FILE_ID = b"PAR 2.0\x00FileDesc"
 PAR_CREATOR_ID = b"PAR 2.0\x00Creator\x00"
+PAR_SLICE_ID = b"PAR 2.0\x00IFSC\x00\x00\x00\x00"
 PAR_RECOVERY_ID = b"RecvSlic"
 
 
@@ -106,40 +108,71 @@ def parse_par2_file(fname: str, md5of16k: Dict[bytes, str]) -> Tuple[str, Dict[s
     http://parchive.sourceforge.net/docs/specifications/parity-volume-spec/article-spec.html
     """
     total_size = os.path.getsize(fname)
+    metadata = {}
+    metadata["nr_files"] = -1
+    filedata = {}
     table = {}
     duplicates16k = []
-    total_nr_files = None
 
     try:
         with open(fname, "rb") as f:
             header = f.read(8)
             while header:
                 if header == PAR_PKT_ID:
-                    name, filehash, hash16k, filesize, set_id, nr_files = parse_par2_packet(f)
-                    if name:
-                        table[name] = FilePar2Info(hash16k, filehash, filesize)
-                        if hash16k not in md5of16k:
-                            md5of16k[hash16k] = name
-                        elif md5of16k[hash16k] != name:
-                            # Not unique and not already linked to this file
-                            # Mark and remove to avoid false-renames
-                            duplicates16k.append(hash16k)
-                            table[name].has_duplicate = True
-
-                    # Store the number of files for later
-                    if nr_files:
-                        total_nr_files = nr_files
+                    parse_par2_packet(f, metadata, filedata)
 
                     # On large files, we stop after seeing all the listings
                     # On smaller files, we scan them fully to get the par2-creator
-                    if total_size > SCAN_LIMIT and len(table) == total_nr_files:
+                    if total_size > SCAN_LIMIT and len(filedata) == metadata["nr_files"]:
                         break
 
                 header = f.read(8)
+
+        set_id = metadata["set_id"]
+        slice_size = metadata["slice_size"]
+        coeff = sabctools.crc32_xpow8n(slice_size)
+
+        for fileid in filedata:
+            name = filedata[fileid][0]
+            hash16k = filedata[fileid][1]
+            filesize = filedata[fileid][3]
+
+            crclist = filedata[fileid][4]
+            if not crclist:
+                logging.debug("Missing CRC32 data in %s. Unfinished download?", fname)
+                table = {}
+                break
+
+            slices = filesize // slice_size
+            tail_size = filesize % slice_size
+            crc32 = 0
+            slice_nr = 0
+            # logging.debug("File %s size %d slices %d tail %d, list %d", name, filesize, slices, tail_size, len(crclist))
+            while slice_nr < slices:
+                crc32 = sabctools.crc32_multiply(crc32, coeff) ^ crclist[slice_nr]
+                slice_nr += 1
+
+            if tail_size:
+                crc32 = sabctools.crc32_combine(
+                    crc32, sabctools.crc32_zero_unpad(crclist[slice_nr], slice_size - tail_size), tail_size
+                )
+            # logging.debug("File %s crc32 %s, int %d", name, hex(crc32), crc32)
+
+            table[name] = FilePar2Info(hash16k, crc32, filesize)
+
+            if hash16k not in md5of16k:
+                md5of16k[hash16k] = name
+            elif md5of16k[hash16k] != name:
+                # Not unique and not already linked to this file
+                # Mark and remove to avoid false-renames
+                duplicates16k.append(hash16k)
+                table[name].has_duplicate = True
+
     except:
         logging.info("Par2 parser crashed in file %s", fname)
         logging.debug("Traceback: ", exc_info=True)
         table = {}
+        set_id = None
 
     # Have to remove duplicates at the end to make sure
     # no trace is left in case of multi-duplicates
@@ -151,13 +184,9 @@ def parse_par2_file(fname: str, md5of16k: Dict[bytes, str]) -> Tuple[str, Dict[s
     return set_id, table
 
 
-def parse_par2_packet(
-    f: BinaryIO,
-) -> Tuple[Optional[str], Optional[bytes], Optional[bytes], Optional[int], Optional[str], Optional[int]]:
+def parse_par2_packet(f: BinaryIO, metadata: Dict, filedata: Dict):
     """Look up and analyze a PAR2 packet"""
 
-    filename, filehash, hash16k, filesize, set_id, nr_files = nothing = None, None, None, None, None, None
-
     # All packages start with a header before the body
     # 8	  : PAR2\x00PKT
     # 8	  : Length of the entire packet. Must be multiple of 4. (NB: Includes length of header.)
@@ -169,7 +198,7 @@ def parse_par2_packet(
     # Length must be multiple of 4 and at least 20
     pack_len = struct.unpack("<Q", f.read(8))[0]
     if int(pack_len / 4) * 4 != pack_len or pack_len < 20:
-        return nothing
+        return
 
     # Next 16 bytes is md5sum of this packet
     md5sum = f.read(16)
@@ -180,14 +209,14 @@ def parse_par2_packet(
     md5 = hashlib.md5()
     md5.update(data)
     if md5sum != md5.digest():
-        return nothing
-
-    # Get the Recovery Set ID
-    set_id = data[:16].hex()
+        return
 
     # See if it's any of the packages we care about
     par2_packet_type = data[16:32]
 
+    # Get the Recovery Set ID
+    metadata["set_id"] = data[:16].hex()
+
     if par2_packet_type == PAR_FILE_ID:
         # The FileDesc packet looks like:
         # 16 : "PAR 2.0\0FileDesc"
@@ -196,20 +225,41 @@ def parse_par2_packet(
         # 16 : Hash for first 16K
         #  8 : File length
         # xx : Name (multiple of 4, padded with \0 if needed)
+
+        fileid = data[32:48].hex()
+        if filedata.get(fileid):
+            # Already have data
+            return
         filehash = data[48:64]
         hash16k = data[64:80]
-        filesize = int.from_bytes(data[80:88], byteorder="little", signed=False)
+        filesize = struct.unpack("<Q", data[80:88])[0]
         filename = correct_unknown_encoding(data[88:].strip(b"\0"))
+        filedata[fileid] = [filename, hash16k, filehash, filesize, []]
     elif par2_packet_type == PAR_CREATOR_ID:
         # From here until the end is the creator-text
         # Useful in case of bugs in the par2-creating software
+        # "PAR 2.0\x00Creator\x00"
         par2creator = data[32:].strip(b"\0")  # Remove any trailing \0
+        metadata["creator"] = par2creator
         logging.debug("Par2-creator of %s is: %s", os.path.basename(f.name), correct_unknown_encoding(par2creator))
     elif par2_packet_type == PAR_MAIN_ID:
         # The Main packet looks like:
         # 16 : "PAR 2.0\0Main"
         # 8  : Slice size
         # 4  : Number of files in the recovery set
-        nr_files = struct.unpack("<I", data[40:44])[0]
-
-    return filename, filehash, hash16k, filesize, set_id, nr_files
+        metadata["slice_size"] = struct.unpack("<Q", data[32:40])[0]
+        metadata["nr_files"] = struct.unpack("<I", data[40:44])[0]
+    elif par2_packet_type == PAR_SLICE_ID:
+        # "PAR 2.0\0IFSC\0\0\0\0"
+        fileid = data[32:48].hex()
+        try:
+            if filedata[fileid][4]:
+                # Already have data
+                return
+        except KeyError:
+            logging.debug("Unknown fileid %s for par2 slice, skipping", fileid)
+            return
+        i = 48
+        for i in range(48, pack_len - 32, 20):
+            filedata[fileid][4].append(struct.unpack("<I", data[i + 16 : i + 20])[0])
+    return

sabnzbd/postproc.py

@@ -534,8 +534,7 @@ def process_job(nzo: NzbObject):
                     deobfuscate.deobfuscate(nzo, newfiles, nzo.final_name)
 
                 # Run the user script
-                script_path = make_script_path(script)
-                if script_path:
+                if script_path := make_script_path(script):
                     # Set the current nzo status to "Ext Script...". Used in History
                     nzo.status = Status.RUNNING
                     nzo.set_action_line(T("Running script"), script)
@@ -543,10 +542,9 @@ def process_job(nzo: NzbObject):
                     script_log, script_ret = external_processing(
                         script_path, nzo, clip_path(workdir_complete), nzo.final_name, job_result
                     )
-                    script_line = get_last_line(script_log)
                     if script_log:
                         script_output = nzo.nzo_id
-                    if script_line:
+                    if script_line := get_last_line(script_log):
                         nzo.set_unpack_info("Script", script_line, unique=True)
                     else:
                         nzo.set_unpack_info("Script", T("Ran %s") % script, unique=True)
@@ -911,7 +909,7 @@ def rar_renamer(nzo: NzbObject) -> int:
         file_to_check = os.path.join(nzo.download_path, file_to_check)
 
         # We only want files:
-        if not (os.path.isfile(file_to_check)):
+        if not os.path.isfile(file_to_check):
             continue
 
         if rarfile.is_rarfile(file_to_check):
@@ -1211,8 +1209,7 @@ def rename_and_collapse_folder(oldpath, newpath, files):
 
 def set_marker(folder: str) -> Optional[str]:
     """Set marker file and return name"""
-    name = cfg.marker_file()
-    if name:
+    if name := cfg.marker_file():
         path = os.path.join(folder, name)
         logging.debug("Create marker file %s", path)
         try:

sabnzbd/utils/diskspeed.py

@@ -9,7 +9,6 @@ import logging
 import time
 
 _DUMP_DATA_SIZE = 10 * 1024 * 1024
-_DUMP_DATA = os.urandom(_DUMP_DATA_SIZE)
 
 
 def diskspeedmeasure(dirname: str) -> float:
@@ -18,6 +17,7 @@ def diskspeedmeasure(dirname: str) -> float:
     Then divide bytes written by time passed
     In case of problems (ie non-writable dir or file), return 0.0
     """
+    dump_data = os.urandom(_DUMP_DATA_SIZE)
     start = time.time()
     maxtime = 0.5  # sec
     total_written = 0
@@ -34,7 +34,7 @@ def diskspeedmeasure(dirname: str) -> float:
         total_time = 0.0
         while total_time < maxtime:
             start = time.time()
-            os.write(fp_testfile, _DUMP_DATA)
+            os.write(fp_testfile, dump_data)
             os.fsync(fp_testfile)
             total_time += time.time() - start
             total_written += _DUMP_DATA_SIZE

sabnzbd/utils/servertests.py

@@ -24,7 +24,7 @@ import sys
 
 from sabnzbd.constants import DEF_TIMEOUT
 from sabnzbd.newswrapper import NewsWrapper, NNTPPermanentError
-from sabnzbd.downloader import Server, clues_login, clues_too_many, nntp_to_msg
+from sabnzbd.downloader import Server, clues_login, clues_too_many
 from sabnzbd.config import get_servers
 from sabnzbd.misc import int_conv, match_str
 
@@ -89,7 +89,6 @@ def test_nntp_server_dict(kwargs):
         nw = NewsWrapper(server=s, thrdnum=-1, block=True)
         nw.init_connect()
         while not nw.connected:
-            nw.clear_data()
             nw.recv_chunk()
             nw.finish_connect(nw.status_code)
 
@@ -123,7 +122,7 @@ def test_nntp_server_dict(kwargs):
     if not username or not password:
         nw.nntp.sock.sendall(b"ARTICLE <test@home>\r\n")
         try:
-            nw.clear_data()
+            nw.reset_data_buffer()
             nw.recv_chunk()
         except:
             # Some internal error, not always safe to close connection
@@ -137,14 +136,14 @@ def test_nntp_server_dict(kwargs):
         elif nw.status_code < 300 or nw.status_code in (411, 423, 430):
             # If no username/password set and we requested fake-article, it will return 430 Not Found
             return_status = (True, T("Connection Successful!"))
-        elif nw.status_code == 502 or clues_login(nntp_to_msg(nw.data)):
+        elif nw.status_code == 502 or clues_login(nw.nntp_msg):
             return_status = (False, T("Authentication failed, check username/password."))
-        elif clues_too_many(nntp_to_msg(nw.data)):
+        elif clues_too_many(nw.nntp_msg):
             return_status = (False, T("Too many connections, please pause downloading or try again later"))
 
     # Fallback in case no data was received or unknown status
     if not return_status:
-        return_status = (False, T("Could not determine connection result (%s)") % nntp_to_msg(nw.data))
+        return_status = (False, T("Could not determine connection result (%s)") % nw.nntp_msg)
 
     # Close the connection and return result
     nw.hard_reset(send_quit=True)