Update text files for 4.0.0 Alpha 2

* Update last_max_chunk_size for each call to recv

* Reduce _DEFAULT_CHUNK_SIZE

.github/workflows/build_release.yml

@@ -87,7 +87,7 @@ jobs:
       # We need the official Python, because the GA ones only support newer macOS versions
       # The deployment target is picked up by the Python build tools automatically
       # If updated, make sure to also set LSMinimumSystemVersion in SABnzbd.spec
-      PYTHON_VERSION: "3.11.1"
+      PYTHON_VERSION: "3.11.2"
       MACOSX_DEPLOYMENT_TARGET: "10.9"
       # We need to force compile for universal2 support
       CFLAGS: -arch x86_64 -arch arm64

.github/workflows/integration_testing.yml

@@ -20,7 +20,7 @@ jobs:
             builder/SABnzbd.spec
             tests
             --line-length=120
-            --target-version=py37
+            --target-version=py38
             --check
             --diff
 
@@ -31,7 +31,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11"]
+        python-version: ["3.8", "3.9", "3.10", "3.11"]
         os: [ubuntu-20.04]
         include:
           - name: macOS

COPYRIGHT.txt

@@ -1,5 +1,5 @@
 
-(c) Copyright 2007-2022 by "The SABnzbd-team" <team@sabnzbd.org>
+(c) Copyright 2007-2023 by "The SABnzbd-team" <team@sabnzbd.org>
 
 The SABnzbd-team is:
 

INSTALL.txt

@@ -4,7 +4,7 @@
 0) LICENSE
 -------------------------------------------------------------------------------
 
-(c) Copyright 2007-2022 by "The SABnzbd-team" <team@sabnzbd.org>
+(c) Copyright 2007-2023 by "The SABnzbd-team" <team@sabnzbd.org>
 
 This program is free software; you can redistribute it and/or
 modify it under the terms of the GNU General Public License
@@ -52,7 +52,7 @@ Specific guides to install from source are available for Windows and macOS:
     https://sabnzbd.org/wiki/installation/install-macos
     https://sabnzbd.org/wiki/installation/install-from-source-windows
 
-Only Python 3.7 and above is supported.
+Only Python 3.8 and above is supported.
 
 On Linux systems you need to install:
     par2 unrar unzip python3-setuptools python3-pip

LICENSE.txt

@@ -1,4 +1,4 @@
-(c) Copyright 2007-2022 by "The SABnzbd-team" <team@sabnzbd.org>
+(c) Copyright 2007-2023 by "The SABnzbd-team" <team@sabnzbd.org>
 
 This program is free software; you can redistribute it and/or
 modify it under the terms of the GNU General Public License

PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 1.0
 Name: SABnzbd
-Version: 3.8.0-develop
-Summary: SABnzbd-3.8.0-develop
+Version: 4.0.0Alpha2
+Summary: SABnzbd-4.0.0Alpha2
 Home-page: https://sabnzbd.org
 Author: The SABnzbd Team
 Author-email: team@sabnzbd.org

README.md

@@ -16,7 +16,7 @@ If you want to know more you can head over to our website: https://sabnzbd.org.
 
 SABnzbd has a few dependencies you'll need before you can get running. If you've previously run SABnzbd from one of the various Linux packages, then you likely already have all the needed dependencies. If not, here's what you're looking for:
 
-- `python` (Python 3.7 and above, often called `python3`)
+- `python` (Python 3.8 and above, often called `python3`)
 - Python modules listed in `requirements.txt`. Install with `python3 -m pip install -r requirements.txt -U`
 - `par2` (Multi-threaded par2 installation guide can be found [here](https://sabnzbd.org/wiki/installation/multicore-par2))
 - `unrar` (make sure you get the "official" non-free version of unrar)

README.mkd

@@ -1,39 +1,30 @@
-Release Notes - SABnzbd 3.7.0 Beta 1
+Release Notes - SABnzbd 4.0.0 Alpha 2
 =========================================================
 
-## Changes since 3.6.1
-- The queue and history can be filtered using keywords:
-  `cat` and `priority`. For example: `show name cat:tv`.
-- Use shortcut `shift + arrow-key` to navigate the queue/history pages.
-- The backup is now created in a local folder for security.
-- Recurring backups can be configured using the scheduler.
-- Improvements to Deobfuscate Final Filenames.
-- RSS overview shows the rule that accepted the job.
-- Added option to sort the queue by `% downloaded`.
-- Added option to replace underscores with dots in folder names.
-- SABnzbd Host input will be validated before being applied.
-- Moved system load information from the main page to the Status window.
-- Console logging is now written to `stdout` instead of `stderr`.
-- Removed Special settings `enable_meta`, `disable_key`,
-  `replace_illegal`, `osx_speed` and `show_sysload`.
-- Merged Special settings `win_menu` and `osx_menu` into `tray_icon`.
-- macOS/Windows: Use Python 3.11rc2, boosting overall performance.
-- macOS/Windows: Updated UnRar to 6.12.
-- Windows: Updated MultiPar to 1.3.2.5.
+## Changes since 4.0.0 Alpha 1
+- There are now multiple settings that can tweak performance, see:
+  https://github.com/sabnzbd/sabnzbd/discussions/2474
+  We are trying to find the most optimal default settings, so you
+  can help us by letting us know the results on your system!
 
-# API changes since 3.6.1
-- Minor improvements in API call performance.
-- Removed fields `scripts` and `categories` from `queue` API call.
-- Moved `loadavg` from `queue` to `status` API call.
+## Changes since 3.7.2
+- In this major update we replaced a core part of Python's SSL handling
+  with our own improved version. This results in large performance increases
+  when downloading from news servers with SSL enabled.
+  In addition, the general connection handling was overhauled, resulting in
+  performance improvements for all news servers.
+  Special thanks to: mnightingale, puzzledsab and animetosho!
+- When adding a new news server, SSL is enabled by default.
+- File assembly performance significantly improved by relying on the
+  CRC32 instead of the MD5 to perform QuickCheck of files.
+- Slowdown more gracefully when the cache fills up.
+- HTTPS files are included in the `Backup`.
+- Improved `Watched Folder` scanning and processing.
+- Dropped support for Python 3.7.
 
-# Bugfixes since 3.6.1
-- Free Space Detection was too strict when using Direct Unpack.
-- File uploads with special characters would be parsed incorrectly.
-- Passwords from NZB meta-data were tried multiple times.
-- Passwords were not always supplied to the pre-queue script.
-- RSS-feed names were not sanitized when renamed.
-- Make sure short-dates are detected as `YY-MM-DD` in Sorting.
-- Show the custom job name in History when the NZB could not be fetched.
+# Bugfixes since 3.7.2
+- Restore applying `History Retention` setting at startup.
+- Windows: Not all invalid characters were removed from filenames.
 
 ## Upgrade notices
 - The download statistics file `totals10.sab` is updated in 3.2.x
@@ -50,4 +41,4 @@ Release Notes - SABnzbd 3.7.0 Beta 1
   that automatically verify, repair, extract and clean up posts downloaded
   from Usenet.
 
-  (c) Copyright 2007-2022 by "The SABnzbd-team" \<team@sabnzbd.org\>
+  (c) Copyright 2007-2023 by "The SABnzbd-team" \<team@sabnzbd.org\>

SABnzbd.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -17,8 +17,8 @@
 
 import sys
 
-if sys.hexversion < 0x03070000:
-    print("Sorry, requires Python 3.7 or above")
+if sys.hexversion < 0x03080000:
+    print("Sorry, requires Python 3.8 or above")
     print("You can read more at: https://sabnzbd.org/wiki/installation/install-off-modules")
     sys.exit(1)
 
@@ -40,6 +40,7 @@ import gc
 from typing import List, Dict, Any
 
 try:
+    import sabctools
     import Cheetah
     import feedparser
     import configobj
@@ -76,6 +77,7 @@ from sabnzbd.constants import (
     DEF_LOG_FILE,
     DEF_STD_CONFIG,
     DEF_LOG_CHERRY,
+    CONFIG_BACKUP_HTTPS,
 )
 import sabnzbd.newsunpack
 from sabnzbd.misc import (
@@ -240,7 +242,7 @@ def print_version():
             """
 %s-%s
 
-Copyright (C) 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+Copyright (C) 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 SABnzbd comes with ABSOLUTELY NO WARRANTY.
 This is free software, and you are welcome to redistribute it
 under certain conditions. It is licensed under the
@@ -398,15 +400,13 @@ def get_user_profile_paths():
         return
 
     elif sabnzbd.MACOS:
-        home = os.environ.get("HOME")
-        if home:
+        if home := os.environ.get("HOME"):
             sabnzbd.DIR_LCLDATA = "%s/Library/Application Support/SABnzbd" % home
             sabnzbd.DIR_HOME = home
             return
     else:
         # Unix/Linux
-        home = os.environ.get("HOME")
-        if home:
+        if home := os.environ.get("HOME"):
             sabnzbd.DIR_LCLDATA = "%s/.%s" % (home, DEF_WORKDIR)
             sabnzbd.DIR_HOME = home
             return
@@ -418,25 +418,26 @@ def get_user_profile_paths():
 
 def print_modules():
     """Log all detected optional or external modules"""
-    if sabnzbd.decoder.SABYENC_ENABLED:
-        # Yes, we have SABYenc, and it's the correct version, so it's enabled
-        logging.info("SABYenc module (v%s)... found!", sabnzbd.decoder.SABYENC_VERSION)
-        logging.info("SABYenc module is using SIMD set: %s", sabnzbd.decoder.SABYENC_SIMD)
+    if sabnzbd.decoder.SABCTOOLS_ENABLED:
+        # Yes, we have SABCTools, and it's the correct version, so it's enabled
+        logging.info("SABCTools module (v%s)... found!", sabnzbd.decoder.SABCTOOLS_VERSION)
+        logging.info("SABCTools module is using SIMD set: %s", sabnzbd.decoder.SABCTOOLS_SIMD)
+        logging.info("SABCTools module is linked to OpenSSL: %s", sabnzbd.decoder.SABCTOOLS_OPENSSL_LINKED)
+
+        # Check if we managed to link, warning for now
+        if not sabnzbd.decoder.SABCTOOLS_OPENSSL_LINKED:
+            logging.warning(
+                "Could not link to OpenSSL library, please report here: "
+                "https://github.com/sabnzbd/sabnzbd/issues/2421"
+            )
     else:
-        # Something wrong with SABYenc, so let's determine and print what:
-        if sabnzbd.decoder.SABYENC_VERSION:
-            # We have a VERSION, thus a SABYenc module, but it's not the correct version
-            logging.error(
-                T("SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"),
-                sabnzbd.decoder.SABYENC_VERSION,
-                sabnzbd.constants.SABYENC_VERSION_REQUIRED,
-            )
-        else:
-            # No SABYenc module at all
-            logging.error(
-                T("SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"),
-                sabnzbd.constants.SABYENC_VERSION_REQUIRED,
-            )
+        # Wrong SABCTools version, if it was fully missing it would fail to start due to check at the very top
+        logging.error(
+            T("SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"),
+            sabnzbd.decoder.SABCTOOLS_VERSION,
+            sabnzbd.constants.SABCTOOLS_VERSION_REQUIRED,
+        )
+
         # Do not allow downloading
         sabnzbd.NO_DOWNLOADING = True
 
@@ -1444,6 +1445,12 @@ def main():
         logging.error(T("Failed to start web-interface: "), exc_info=True)
         abort_and_show_error(browserhost, cherryport)
 
+    # Create a record of the active cert/key/chain files, for use with config.create_config_backup()
+    if enable_https:
+        for setting in CONFIG_BACKUP_HTTPS.values():
+            if full_path := getattr(sabnzbd.cfg, setting).get_path():
+                sabnzbd.CONFIG_BACKUP_HTTPS_OK.append(full_path)
+
     if sabnzbd.WIN32:
         if enable_https:
             mode = "s"

builder/osx/requirements.txt

@@ -1,3 +1,3 @@
 # Special requirements for macOS universal2 binary release
 # This way dependabot can auto-update them
-cryptography==39.0.0
+cryptography==39.0.1

builder/requirements.txt

@@ -1,18 +1,18 @@
 # Basic build requirements
 # Note that not all sub-dependencies are listed, but only ones we know could cause trouble
-pyinstaller==5.7.0
-pyinstaller-hooks-contrib==2022.14
+pyinstaller==5.8.0
+pyinstaller-hooks-contrib==2022.15
 altgraph==0.17.3
 wrapt==1.14.1
-setuptools==65.6.3
+setuptools==67.2.0
 pkginfo==1.9.6
 PyGithub==1.57
 charset-normalizer==3.0.1
 certifi
 
 # orjson does not support 32bit Windows, exclude it based on Python-version
-# This way we also test ujson on Python 3.7 and 3.8 in the CI-tests
-orjson==3.8.4; python_version > '3.8'
+# This way we also test ujson on Python 3.8 in the CI-tests
+orjson==3.8.6; python_version > '3.8'
 
 # For the macOS build
 dmgbuild==1.6.0; sys_platform == 'darwin'

interfaces/Config/templates/config.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Config"#-->
-<!--#set global $help_uri="configuration/3.7/configure"#-->
+<!--#set global $help_uri="configuration/4.0/configure"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <!--#from sabnzbd.encoding import CODEPAGE#-->
@@ -54,12 +54,12 @@
                     </td>
                 </tr>
                 <!--#end if#-->
-                <!--#if not $have_sabyenc#-->
+                <!--#if not $have_sabctools#-->
                 <tr>
-                    <th scope="row">SABYenc:</th>
+                    <th scope="row">SABCTools:</th>
                     <td>
                         <span class="label label-danger">$T('notAvailable')</span>
-                        <a href="$helpuri$help_uri#no_sabyenc" target="_blank"><span class="glyphicon glyphicon-question-sign"></span></a>
+                        <a href="$helpuri$help_uri#no_sabctools" target="_blank"><span class="glyphicon glyphicon-question-sign"></span></a>
                     </td>
                 </tr>
                 <!--#end if#-->
@@ -124,7 +124,7 @@
 
 <div class="colmask">
     <div class="padding alt">
-        <h5 class="copyright">Copyright &copy; 2007-2022 The SABnzbd Team &lt;<a href="mailto:team@sabnzbd.org">team@sabnzbd.org</a>&gt;</h5>
+        <h5 class="copyright">Copyright &copy; 2007-2023 The SABnzbd Team &lt;<a href="mailto:team@sabnzbd.org">team@sabnzbd.org</a>&gt;</h5>
         <p class="copyright"><small>$T('yourRights')</small></p>
     </div>
 

interfaces/Config/templates/config_cat.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Categories"#-->
-<!--#set global $help_uri="configuration/3.7/categories"#-->
+<!--#set global $help_uri="configuration/4.0/categories"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 <div class="colmask">
     <div class="section">

interfaces/Config/templates/config_folders.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Folders"#-->
-<!--#set global $help_uri="configuration/3.7/folders"#-->
+<!--#set global $help_uri="configuration/4.0/folders"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <div class="colmask">

interfaces/Config/templates/config_general.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="General"#-->
-<!--#set global $help_uri="configuration/3.7/general"#-->
+<!--#set global $help_uri="configuration/4.0/general"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <div class="colmask">
@@ -375,7 +375,7 @@
     })
 
     // Only allow re-generate if default certs
-    if(\$('#https_cert').val() != 'server.cert') {
+    if(\$('#https_cert').val() != '$def_https_cert_file') {
         \$('.generate_cert').attr('disabled', 'disabled')
     }
 

interfaces/Config/templates/config_notify.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Email"#-->
-<!--#set global $help_uri="configuration/3.7/notifications"#-->
+<!--#set global $help_uri="configuration/4.0/notifications"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <!--#def show_notify_checkboxes($section_label)#-->

interfaces/Config/templates/config_rss.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="RSS"#-->
-<!--#set global $help_uri="configuration/3.7/rss"#-->
+<!--#set global $help_uri="configuration/4.0/rss"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 <!--#import html#-->
 <div class="colmask">

interfaces/Config/templates/config_scheduling.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Scheduling"#-->
-<!--#set global $help_uri="configuration/3.7/scheduling"#-->
+<!--#set global $help_uri="configuration/4.0/scheduling"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <%

interfaces/Config/templates/config_server.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Servers"#-->
-<!--#set global $help_uri="configuration/3.7/servers"#-->
+<!--#set global $help_uri="configuration/4.0/servers"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <!--#import json#-->
@@ -51,11 +51,11 @@
                     </div>
                     <div class="field-pair advanced-settings">
                         <label class="config" for="port">$T('srv-port')</label>
-                        <input type="number" name="port" id="port" size="8" value="119" min="0" />
+                        <input type="number" name="port" id="port" size="8" value="563" min="0" />
                     </div>
                     <div class="field-pair">
                         <label class="config" for="ssl">$T('srv-ssl')</label>
-                        <input type="checkbox" name="ssl" id="ssl" value="1" />
+                        <input type="checkbox" name="ssl" id="ssl" value="1" checked />
                         <span class="desc">$T('explain-ssl')</span>
                     </div>
                     <!-- Tricks to avoid browser auto-fill, fixed on-submit with javascript -->

interfaces/Config/templates/config_sorting.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Sorting"#-->
-<!--#set global $help_uri="configuration/3.7/sorting"#-->
+<!--#set global $help_uri="configuration/4.0/sorting"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <div class="colmask">

interfaces/Config/templates/config_special.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Special"#-->
-<!--#set global $help_uri="configuration/3.7/special"#-->
+<!--#set global $help_uri="configuration/4.0/special"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <div class="colmask">

interfaces/Config/templates/config_switches.tmpl

@@ -1,5 +1,5 @@
 <!--#set global $pane="Switches"#-->
-<!--#set global $help_uri="configuration/3.7/switches"#-->
+<!--#set global $help_uri="configuration/4.0/switches"#-->
 <!--#include $webdir + "/_inc_header_uc.tmpl"#-->
 
 <div class="colmask">

interfaces/wizard/one.html

@@ -57,7 +57,7 @@
                             <div class="form-group">
                                 <label for="port" class="col-sm-4 control-label">$T('srv-port')</label>
                                 <div class="col-sm-8">
-                                    <input type="number" class="form-control" name="port" id="port" value="<!--#if $port then $port else '119' #-->" />
+                                    <input type="number" class="form-control" name="port" id="port" value="<!--#if $port then $port else '563' #-->" />
                                 </div>
                             </div>
                             <div class="form-group">

linux/org.sabnzbd.sabnzbd.appdata.xml

@@ -1,5 +1,5 @@
 <?xml version="1.0" encoding="UTF-8"?>
-<!-- Copyright 2022 The SABnzbd-Team <team@sabnzbd.org> -->
+<!-- Copyright 2022-2023 The SABnzbd-Team <team@sabnzbd.org> -->
 <component type="desktop-application">
   <id>org.sabnzbd.sabnzbd</id>
   <metadata_license>MIT</metadata_license>

po/email/SABemail.pot

@@ -1,11 +1,11 @@
 #
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 #
 msgid ""
 msgstr ""
-"Project-Id-Version: SABnzbd-3.8.0-develop\n"
+"Project-Id-Version: SABnzbd-4.0.0Alpha1\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: team@sabnzbd.org\n"
 "Language-Team: SABnzbd <team@sabnzbd.org>\n"

po/email/cs.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 msgid ""

po/email/da.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/de.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/es.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/fi.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/fr.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/he.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/nb.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/nl.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/pl.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/pt_BR.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/ro.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/ru.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/sr.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/sv.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/email/zh_CN.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file EMAIL
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/main/SABnzbd.pot

@@ -1,11 +1,11 @@
 #
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 #
 msgid ""
 msgstr ""
-"Project-Id-Version: SABnzbd-3.8.0-develop\n"
+"Project-Id-Version: SABnzbd-4.0.0Alpha1\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: team@sabnzbd.org\n"
 "Language-Team: SABnzbd <team@sabnzbd.org>\n"
@@ -35,12 +35,7 @@ msgstr ""
 
 #. Error message
 #: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+msgid "SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -598,7 +593,7 @@ msgstr ""
 msgid "API Key incorrect, Use the api key from Config->General in your 3rd party program:"
 msgstr ""
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr ""
 

po/main/cs.po

@@ -1,16 +1,16 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
 # Pavel C <quoing_transifex@mess.cz>, 2021
-# Safihre <safihre@sabnzbd.org>, 2022
+# Safihre <safihre@sabnzbd.org>, 2023
 # 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Czech (https://www.transifex.com/sabnzbd/teams/111101/cs/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -38,18 +38,12 @@ msgstr "Nezdařilo se spustit webové rozhraní"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Šablona pro web nebyla nalezena: %s, zkouším standardní šablonu"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc vypnut: Nenalezena správná verze! (Nalezena v%s, očekávána v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"Modul SABYenc... nebyl nalezen! Očekávána v%s - https://sabnzbd.org/sabyenc"
+"SABCTools vypnut: Nenalezena správná verze! (Nalezena v%s, očekávána v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -645,7 +639,7 @@ msgstr ""
 "Nesprávný API klíč, použijte api klíč z Nastavení->Obecné ve vašem programu "
 "třetí strany:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Přihlášené selhalo, zkontrolujte jméno a heslo."
 

po/main/da.po

@@ -1,15 +1,15 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
-# Safihre <safihre@sabnzbd.org>, 2022
+# Safihre <safihre@sabnzbd.org>, 2023
 # 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Danish (https://www.transifex.com/sabnzbd/teams/111101/da/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -37,19 +37,13 @@ msgstr "Kunne ikke starte web-interface"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Kan ikke finde webskabeloner: %s, forsøger med standardskabelon"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc deaktiveret: Der blev ikke fundet nogen korrekt version (Fandt v%s, "
-"forventede v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"SABYenc modul... IKKE fundet! Forventede v%s - https://sabnzbd.org/sabyenc"
+"SABCTools deaktiveret: Der blev ikke fundet nogen korrekt version (Fandt "
+"v%s, forventede v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -653,7 +647,7 @@ msgstr ""
 "Forkert API-nøgle, anvend api-nøglen fra Konfiguration->Generelt i dit "
 "tredjepartsprogram:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Godkendelse mislykkedes, kontrollere brugernavn/adgangskode."
 

po/main/de.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
@@ -11,13 +11,13 @@
 # Simon W., 2021
 # Nils Briggen, 2022
 # reloxx13 <reloxx@interia.pl>, 2022
-# Safihre <safihre@sabnzbd.org>, 2022
+# Safihre <safihre@sabnzbd.org>, 2023
 # 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: German (https://www.transifex.com/sabnzbd/teams/111101/de/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -47,19 +47,13 @@ msgstr ""
 "Konnte Web-Vorlage nicht finden: %s Versuche die Standard-Vorlage zu "
 "verwenden."
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc deaktiviert: Keine korrekte Version gefunden! (Gefunden v%s, "
-"Erwartet v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"SABYenc Modul... Nicht gefunden! Erwarte v%s - https://sabnzbd.org/sabyenc"
+"SABCTools deaktiviert: Keine korrekte Version gefunden! (Gefunden v%s, "
+"Erwartet v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -686,7 +680,7 @@ msgstr ""
 "API-Schlüssel ungültig. Bitte API-Schlüssel aus Einstellungen->Allgemein in "
 "die externe Anwendung eingeben:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr ""
 "Authentifizierung fehlgeschlagen. Überprüfen Sie Benutzername und Passwort."

po/main/es.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
@@ -41,20 +41,13 @@ msgstr ""
 "No se puede encontrar la plantilla web: %s, intentando con la plantilla "
 "estandar"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc deshabilitado: ¡no se ha encontrado la versión correcta! (Se ha "
-"encontrado la v%s, se esperaba la v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"Módulo SABYenc... ¡NO encontrado! Se esperaba la v%s - "
-"https://sabnzbd.org/sabyenc"
+"SABCTools deshabilitado: ¡no se ha encontrado la versión correcta! (Se ha "
+"encontrado la v%s, se esperaba la v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -676,7 +669,7 @@ msgstr ""
 "Clave de API erróneo, favor ingresar la clave correcta desde Config->General"
 " en tu aplicacion externa:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Autenticación fallida, compruebe el usuario o la contraseña."
 

po/main/fi.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
@@ -37,15 +37,10 @@ msgstr "Web-käyttöliittymän käynnistys epäonnistui"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Web-mallia %s ei löydy, yritetään käyttää oletusmallia"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -649,7 +644,7 @@ msgstr ""
 "API avain virheellinen, käytä Asetukset->Yleiset löytyvää api avainta "
 "käyttämääsi kolmannen osapuolen ohjelmaan:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Varmennus epäonnistui, tarkista käyttäjänimi/salasana."
 

po/main/fr.po

@@ -1,16 +1,16 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
-# Safihre <safihre@sabnzbd.org>, 2022
-# Fred L <88com88@gmail.com>, 2022
+# Fred L <88com88@gmail.com>, 2023
+# Safihre <safihre@sabnzbd.org>, 2023
 # 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Fred L <88com88@gmail.com>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: French (https://www.transifex.com/sabnzbd/teams/111101/fr/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -40,19 +40,13 @@ msgstr ""
 "Impossible de trouver le template de l'interface web : %s, nouvelle "
 "tentative avec le template standard"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc désactivé: aucune version correcte n'a été trouvée ! (v%s trouvée, "
-"v%s attendue)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"Module SABYenc... NON trouvé ! v%s attendue - https://sabnzbd.org/sabyenc"
+"SABCTools désactivé: aucune version correcte n'a été trouvée ! (v%s trouvée,"
+" v%s attendue)"
 
 #. Error message
 #: SABnzbd.py
@@ -684,7 +678,7 @@ msgstr ""
 "Clé API incorrecte, utilisez la clé API de la configuration générale dans "
 "votre application tierce :"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Echec d'authentification, vérifiez les identifiant/mot de passe."
 
@@ -3122,7 +3116,7 @@ msgstr "Dossiers système"
 
 #: sabnzbd/skintext.py
 msgid "Hidden Folders"
-msgstr ""
+msgstr "Dossiers cachés"
 
 #: sabnzbd/skintext.py
 msgid "Administrative Folder"

po/main/he.po

@@ -1,16 +1,16 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
 # ION, 2022
-# Safihre <safihre@sabnzbd.org>, 2022
+# Safihre <safihre@sabnzbd.org>, 2023
 # 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Hebrew (https://www.transifex.com/sabnzbd/teams/111101/he/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -38,16 +38,11 @@ msgstr "נכשל בהתחלת ממשק רשת"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "לא ניתן למצוא תבניות רשת: %s, מנסה תבנית תקנית"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr "SABYenc מושבת: גרסה נכונה לא נמצאה! (%s נמצאה, מצפה אל %s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
-msgstr "מודול SABYenc… לא נמצא! מצפה אל %s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
+msgstr "SABCTools מושבת: גרסה נכונה לא נמצאה! (%s נמצאה, מצפה אל %s)"
 
 #. Error message
 #: SABnzbd.py
@@ -647,7 +642,7 @@ msgid ""
 "program:"
 msgstr "מפתח API שגוי, השתמש במפתח ה־API מתצורה->כללי בתוכנית הצד השלישי שלך:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "אימות נכשל, בדוק שם משתמש/סיסמה."
 

po/main/nb.po

@@ -1,15 +1,15 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
-# Safihre <safihre@sabnzbd.org>, 2022
+# Safihre <safihre@sabnzbd.org>, 2023
 # 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Norwegian Bokmål (https://www.transifex.com/sabnzbd/teams/111101/nb/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -37,18 +37,12 @@ msgstr "Kunne ikke starte webgrensesnittet"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Kan ikke finne webmal: %s, prøver standardmal"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc deaktivert: Fant ikke korrekt versjon! (Fant v%s, forventet v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"SABYenc modul... IKKE funnet! Forventet v%s - https://sabnzbd.org/sabyenc"
+"SABCTools deaktivert: Fant ikke korrekt versjon! (Fant v%s, forventet v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -645,7 +639,7 @@ msgstr ""
 "API-nøkkel er feil, bruk API-nøkkel fra Konfigurasjon->Generelt i ditt "
 "tredjepartsprogram:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Godkjenning mislyktes, kontroller brukernavn og passord."
 

po/main/nl.po

@@ -1,16 +1,16 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
 # Rik Brouwer, 2022
-# Safihre <safihre@sabnzbd.org>, 2022
+# Safihre <safihre@sabnzbd.org>, 2023
 # 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Dutch (https://www.transifex.com/sabnzbd/teams/111101/nl/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -38,19 +38,13 @@ msgstr "Webinterface kan niet gestart worden"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Websjabloon %s niet te vinden; het standaardsjabloon wordt gebruikt."
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc uitgeschakeld, geen bruikbare versie gevonden! (V%s gevonden, V%s "
-"verwacht)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"SABYenc module... NIET gevonden! Verwacht V%s - https://sabnzbd.org/sabyenc"
+"SABCTools uitgeschakeld, geen bruikbare versie gevonden! (V%s gevonden, V%s "
+"verwacht)"
 
 #. Error message
 #: SABnzbd.py
@@ -678,7 +672,7 @@ msgstr ""
 "API-sleutel incorrect; vul de API-sleutel van 'Configuratie' => 'Algemeen' "
 "in bij het externe programma:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Inloggen mislukt, controleer gebruikersnaam en wachtwoord."
 

po/main/pl.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
@@ -37,15 +37,10 @@ msgstr "Nie udało się uruchomić interfejsu WWW"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Nie znaleziono szablonu: %s, próbuję użyć standardowego szablonu"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -644,7 +639,7 @@ msgstr ""
 "Klucz API jest nieprawidłowy, użyj klucza API z sekcji Konfiguracja->Ogólne "
 "w zewnętrznym programie:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Błąd połączenia, sprawdź nazwę użytkownika i hasło."
 

po/main/pt_BR.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
@@ -38,15 +38,10 @@ msgid "Cannot find web template: %s, trying standard template"
 msgstr ""
 "Não foi possível encontrar o template web: %s. Tentando o template padrão"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -648,7 +643,7 @@ msgstr ""
 "Chave de API incorreta. Use a chave de API de Configuração->Geral em seu "
 "programa de terceiros:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Falha de autenticação, verifique usuário / senha."
 

po/main/ro.po

@@ -1,16 +1,16 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
 # Eduard Baniceru <war4peace@gmail.com>, 2021
-# Safihre <safihre@sabnzbd.org>, 2022
+# Safihre <safihre@sabnzbd.org>, 2023
 # 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Romanian (https://www.transifex.com/sabnzbd/teams/111101/ro/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -38,19 +38,13 @@ msgstr "Pornirea interfeţei-web nereuşită"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Nu se poate găsi şablon web:%s, se încearcă şablon standard"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-"SABYenc dezactivat: nu s-a găsit o versiune corectă! (Găsită v%s, se "
-"așteaptă v%s)"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
-"Modul SABYenc... NEgăsit! Se așteaptă v%s - https://sabnzbd.org/sabyenc"
+"SABCTools dezactivat: nu s-a găsit o versiune corectă! (Găsită v%s, se "
+"așteaptă v%s)"
 
 #. Error message
 #: SABnzbd.py
@@ -662,7 +656,7 @@ msgstr ""
 "Cheie API incorectă, Folosiţi cheia api din Configurare->General în "
 "programul dumneavoastră terţ:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Autentificare nereuşită, verifică nume utilizator/parolă."
 

po/main/ru.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
@@ -39,15 +39,10 @@ msgstr ""
 "Не удаётся найти шаблон веб-интерфейса: %s. Выполняется попытка использовать"
 " стандартный шаблон"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -644,7 +639,7 @@ msgstr ""
 "Неправильный ключ API. Используйте в сторонней программе ключ API из раздела"
 " «Настройка -> Общие»:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Ошибка проверки подлинности. Проверьте имя и пароль."
 

po/main/sr.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
@@ -37,15 +37,10 @@ msgstr "Neuspešno pokretanje web interfejsa"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Немогуће наћи веб модел: %s, програм покушава са стандардним моделом"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -641,7 +636,7 @@ msgid ""
 msgstr ""
 "API кључ је погрешан, унети у спољни програм API кључ из Подешавања->Опште:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Аутентификација погрешна, проверити име/лозинку."
 

po/main/sv.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
@@ -37,15 +37,10 @@ msgstr "Det gick inte att starta webbgränssnittet"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "Hittar inte webbmall: %s, försöker med standardmall"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr ""
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
 msgstr ""
 
 #. Error message
@@ -643,7 +638,7 @@ msgstr ""
 "API-nyckel felaktig, använd api-nyckeln från Konfiguration-> Allmänt i ditt "
 "tredjepartsprogram:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "Autentisering misslyckades, kontrollera användarnamn och lösenord."
 

po/main/zh_CN.po

@@ -1,15 +1,15 @@
 # SABnzbd Translation Template file MAIN
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
-# Safihre <safihre@sabnzbd.org>, 2022
+# Safihre <safihre@sabnzbd.org>, 2023
 # 
 msgid ""
 msgstr ""
 "Project-Id-Version: SABnzbd-3.8.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:49+0000\n"
-"Last-Translator: Safihre <safihre@sabnzbd.org>, 2022\n"
+"Last-Translator: Safihre <safihre@sabnzbd.org>, 2023\n"
 "Language-Team: Chinese (China) (https://www.transifex.com/sabnzbd/teams/111101/zh_CN/)\n"
 "MIME-Version: 1.0\n"
 "Content-Type: text/plain; charset=UTF-8\n"
@@ -37,16 +37,11 @@ msgstr "web 界面启动失败"
 msgid "Cannot find web template: %s, trying standard template"
 msgstr "无法找到 web 模板: %s，正在尝试标准模板"
 
-#. Error message
-#: SABnzbd.py
-msgid "SABYenc disabled: no correct version found! (Found v%s, expecting v%s)"
-msgstr "SABYenc 已禁用：未找到正确的版本！（找到 v%s，要求 v%s）"
-
 #. Error message
 #: SABnzbd.py
 msgid ""
-"SABYenc module... NOT found! Expecting v%s - https://sabnzbd.org/sabyenc"
-msgstr "SABYenc 模块... 未找到！要求  v%s - https://sabnzbd.org/sabyenc"
+"SABCTools disabled: no correct version found! (Found v%s, expecting v%s)"
+msgstr "SABCTools 已禁用：未找到正确的版本！（找到 v%s，要求 v%s）"
 
 #. Error message
 #: SABnzbd.py
@@ -636,7 +631,7 @@ msgid ""
 "program:"
 msgstr "API Key 不正确，请在第三方程序中使用“配置”->“常规”中的 api key:"
 
-#: sabnzbd/interface.py, sabnzbd/newswrapper.py, sabnzbd/utils/servertests.py
+#: sabnzbd/interface.py, sabnzbd/utils/servertests.py
 msgid "Authentication failed, check username/password."
 msgstr "身份认证失败，请检查用户名/密码。"
 

po/nsis/SABnsis.pot

@@ -1,11 +1,11 @@
 #
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 #
 msgid ""
 msgstr ""
-"Project-Id-Version: SABnzbd-3.8.0-develop\n"
+"Project-Id-Version: SABnzbd-4.0.0Alpha1\n"
 "PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
 "Last-Translator: team@sabnzbd.org\n"
 "Language-Team: SABnzbd <team@sabnzbd.org>\n"

po/nsis/cs.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/da.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/de.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:
@@ -8,7 +8,7 @@
 # 
 msgid ""
 msgstr ""
-"Project-Id-Version: SABnzbd-3.8.0-develop\n"
+"Project-Id-Version: SABnzbd-4.0.0-develop\n"
 "PO-Revision-Date: 2020-06-27 15:56+0000\n"
 "Last-Translator: reloxx13 <reloxx@interia.pl>, 2022\n"
 "Language-Team: German (https://www.transifex.com/sabnzbd/teams/111101/de/)\n"

po/nsis/es.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/fi.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/fr.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/he.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/nb.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/nl.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/pl.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/pt_BR.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/ro.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/ru.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/sr.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/sv.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

po/nsis/zh_CN.po

@@ -1,5 +1,5 @@
 # SABnzbd Translation Template file NSIS
-# Copyright 2007-2022 The SABnzbd-Team
+# Copyright 2007-2023 The SABnzbd-Team
 #   team@sabnzbd.org
 # 
 # Translators:

requirements.txt

@@ -1,11 +1,11 @@
 # Main requirements
 # Note that not all sub-dependencies are listed, but only ones we know could cause trouble
-sabyenc3==5.4.4
+sabctools==6.1.0
 cheetah3==3.2.6.post1
 cffi==1.15.1
 pycparser==2.21
 feedparser==6.0.10
-configobj==5.0.6
+configobj==5.0.8
 cheroot==9.0.0
 six==1.16.0
 cherrypy==18.8.0
@@ -13,12 +13,12 @@ jaraco.functools==3.5.2
 jaraco.collections==3.8.0
 jaraco.text==3.8.1 # Newer version introduces irrelevant extra dependencies
 jaraco.classes==3.2.3
-jaraco.context==4.2.0
+jaraco.context==4.3.0
 more-itertools==9.0.0
 zc.lockfile==2.0
 python-dateutil==2.8.2
-tempora==5.2.0
-pytz==2022.7
+tempora==5.2.1
+pytz==2022.7.1
 sgmllib3k==1.0.0
 portend==3.1.0
 chardet==5.1.0
@@ -30,7 +30,7 @@ rebulk==3.1.0
 
 # Recent cryptography versions require Rust. If you run into issues compiling this
 # SABnzbd will also work with older pre-Rust versions such as cryptography==3.3.2
-cryptography==39.0.0
+cryptography==39.0.1
 
 # We recommend using "orjson" as it is 2x as fast as "ujson". However, it requires
 # Rust so SABnzbd works just as well with "ujson" or the Python built in "json" module

sabnzbd/__init__.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -162,7 +162,7 @@ WIN_SERVICE = None  # Instance of our Win32 Service Class
 BROWSER_URL = None
 
 CERTIFICATE_VALIDATION = True
-NO_DOWNLOADING = False  # When essentials are missing (SABYenc/par2/unrar)
+NO_DOWNLOADING = False  # When essentials are missing (SABCTools/par2/unrar)
 
 WEB_DIR = None
 WEB_DIR_CONFIG = None
@@ -190,12 +190,16 @@ DOWNLOAD_DIR_SPEED = 0
 COMPLETE_DIR_SPEED = 0
 INTERNET_BANDWIDTH = 0
 
+# Record of HTTPS config files at startup
+CONFIG_BACKUP_HTTPS_OK = []
+
 # Rendering of original command line arguments in Config
 CMDLINE = " ".join(['"%s"' % arg for arg in sys.argv])
 
 __INITIALIZED__ = False
 __SHUTTING_DOWN__ = False
 
+
 ##############################################################################
 # Signal Handler
 ##############################################################################
@@ -228,6 +232,8 @@ def initialize(pause_downloader=False, clean_up=False, repair=0):
 
     sabnzbd.__SHUTTING_DOWN__ = False
 
+    sys.setswitchinterval(cfg.switchinterval())
+
     # Set global database connection for Web-UI threads
     cherrypy.engine.subscribe("start_thread", get_db_connection)
 

sabnzbd/api.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License

sabnzbd/articlecache.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -26,7 +26,7 @@ from typing import Dict, List
 
 import sabnzbd
 from sabnzbd.decorators import synchronized
-from sabnzbd.constants import GIGI, ANFO, MEBI, LIMIT_DECODE_QUEUE, MIN_DECODE_QUEUE
+from sabnzbd.constants import GIGI, ANFO, MEBI, LIMIT_DECODE_QUEUE, MIN_DECODE_QUEUE, ASSEMBLER_WRITE_THRESHOLD
 from sabnzbd.nzbstuff import Article
 
 # Operations on the article table are handled via try/except.
@@ -45,6 +45,8 @@ class ArticleCache:
         # so it can be larger on memory-rich systems
         self.decoder_cache_article_limit = 0
 
+        self.assembler_write_trigger: int = 1
+
         # On 32 bit we only allow the user to set 1GB
         # For 64 bit we allow up to 4GB, in case somebody wants that
         self.__cache_upper_limit = GIGI
@@ -68,6 +70,16 @@ class ArticleCache:
         # The cache should also not be too small
         self.decoder_cache_article_limit = max(decoder_cache_limit, MIN_DECODE_QUEUE)
 
+        # Set assembler_write_trigger to be the equivalent of ASSEMBLER_WRITE_THRESHOLD %
+        # of the total cache, assuming an article size of 750 000 bytes
+        self.assembler_write_trigger = int(self.__cache_limit * ASSEMBLER_WRITE_THRESHOLD / 100 / 750_000) + 1
+
+        logging.debug(
+            "Decoder cache limit = %d - Assembler trigger = %d",
+            self.decoder_cache_article_limit,
+            self.assembler_write_trigger,
+        )
+
     @synchronized(ARTICLE_COUNTER_LOCK)
     def reserve_space(self, data_size: int):
         """Reserve space in the cache"""
@@ -92,9 +104,10 @@ class ArticleCache:
         # Register article for bookkeeping in case the job is deleted
         nzo.add_saved_article(article)
 
-        if article.lowest_partnum and not article.nzf.import_finished:
-            # Write the first-fetched articles to disk
-            # Otherwise the cache could overflow
+        if article.lowest_partnum and not (article.nzf.import_finished or article.nzf.filename_checked):
+            # Write the first-fetched articles to temporary file unless downloading
+            # of the rest of the parts has started or filename is verified.
+            # Otherwise the cache could overflow.
             self.__flush_article_to_disk(article, data)
             return
 

sabnzbd/assembler.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -24,7 +24,6 @@ import queue
 import logging
 import re
 from threading import Thread
-import hashlib
 import ctypes
 from typing import Tuple, Optional, List
 
@@ -56,11 +55,8 @@ class Assembler(Thread):
     def process(self, nzo: NzbObject, nzf: Optional[NzbFile] = None, file_done: Optional[bool] = None):
         self.queue.put((nzo, nzf, file_done))
 
-    def queue_full(self):
-        return self.queue.qsize() >= MAX_ASSEMBLER_QUEUE
-
-    def partial_nzf_in_queue(self, nzf: NzbFile):
-        return (nzf.nzo, nzf, False) in self.queue.queue
+    def queue_level(self) -> float:
+        return self.queue.qsize() / MAX_ASSEMBLER_QUEUE
 
     def run(self):
         while 1:
@@ -78,9 +74,7 @@ class Assembler(Thread):
                     self.diskspace_check(nzo, nzf)
 
                 # Prepare filepath
-                filepath = nzf.prepare_filepath()
-
-                if filepath:
+                if filepath := nzf.prepare_filepath():
                     try:
                         logging.debug("Decoding part of %s", filepath)
                         self.assemble(nzo, nzf, file_done)
@@ -170,9 +164,6 @@ class Assembler(Thread):
         1) Partial write: write what we have
         2) Nothing written before: write all
         """
-        # New hash-object needed?
-        if not nzf.md5:
-            nzf.md5 = hashlib.md5()
 
         # We write large article-sized chunks, so we can safely skip the buffering of Python
         with open(nzf.filepath, "ab", buffering=0) as fout:
@@ -191,7 +182,7 @@ class Assembler(Thread):
                     # Could be empty in case nzo was deleted
                     if data:
                         fout.write(data)
-                        nzf.md5.update(data)
+                        nzf.update_crc32(article.crc32, len(data))
                         article.on_disk = True
                     else:
                         logging.info("No data found when trying to write %s", article)
@@ -207,7 +198,7 @@ class Assembler(Thread):
         # Final steps
         if file_done:
             set_permissions(nzf.filepath)
-            nzf.md5sum = nzf.md5.digest()
+            nzf.assembled = True
 
     @staticmethod
     def check_encrypted_and_unwanted(nzo: NzbObject, nzf: NzbFile):

sabnzbd/bpsmeter.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -434,40 +434,6 @@ class BPSMeter:
         # We record every second, but display at the user's refresh-rate
         return self.bps_list[::refresh_rate]
 
-    def get_stable_speed(self, timespan: int = 10) -> Optional[int]:
-        """See if there is a stable speed the last <timespan> seconds
-        None: indicates it can't determine yet
-        0: the speed was not stable during <timespan>
-        Positive float: the speed was stable
-        """
-        if len(self.bps_list) < timespan:
-            return None
-
-        # Check if speed fell by more than 15%
-        try:
-            if self.bps_list[-1] / self.bps_list[-timespan] < 0.85:
-                return 0
-        except:
-            pass
-
-        # Calculate the variance in the speed
-        avg = sum(self.bps_list[-timespan:]) / timespan
-        vari = 0
-        for bps in self.bps_list[-timespan:]:
-            vari += abs(bps - avg)
-        vari = vari / timespan
-
-        try:
-            # See if the variance is less than 5%
-            if (vari / (self.bps / KIBI)) < 0.05:
-                return avg
-            else:
-                return 0
-        except:
-            # Probably one of the values was 0
-            pass
-        return None
-
     def reset_quota(self, force: bool = False):
         """Check if it's time to reset the quota, optionally resuming
         Return True, when still paused or should be paused

sabnzbd/cfg.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -47,6 +47,8 @@ from sabnzbd.constants import (
     DEF_COMPLETE_DIR,
     DEF_FOLDER_MAX,
     DEF_STD_WEB_COLOR,
+    DEF_HTTPS_CERT_FILE,
+    DEF_HTTPS_KEY_FILE,
 )
 
 
@@ -258,7 +260,6 @@ configlock = OptionBool("misc", "config_lock", False)
 # One time trackers
 ##############################################################################
 fixed_ports = OptionBool("misc", "fixed_ports", False)
-sched_converted = OptionBool("misc", "sched_converted", False)
 notified_new_skin = OptionNumber("misc", "notified_new_skin", 0)
 direct_unpack_tested = OptionBool("misc", "direct_unpack_tested", False)
 
@@ -279,8 +280,8 @@ bandwidth_max = OptionStr("misc", "bandwidth_max")
 cache_limit = OptionStr("misc", "cache_limit")
 web_dir = OptionStr("misc", "web_dir", DEF_STD_WEB_DIR)
 web_color = OptionStr("misc", "web_color", DEF_STD_WEB_COLOR)
-https_cert = OptionDir("misc", "https_cert", "server.cert", create=False)
-https_key = OptionDir("misc", "https_key", "server.key", create=False)
+https_cert = OptionDir("misc", "https_cert", DEF_HTTPS_CERT_FILE, create=False)
+https_key = OptionDir("misc", "https_key", DEF_HTTPS_KEY_FILE, create=False)
 https_chain = OptionDir("misc", "https_chain", create=False)
 enable_https = OptionBool("misc", "enable_https", False)
 # 0=local-only, 1=nzb, 2=api, 3=full_api, 4=webui, 5=webui with login for external
@@ -434,7 +435,9 @@ host_whitelist = OptionList("misc", "host_whitelist", validation=all_lowercase)
 local_ranges = OptionList("misc", "local_ranges", protect=True)
 max_url_retries = OptionNumber("misc", "max_url_retries", 10, minval=1)
 downloader_sleep_time = OptionNumber("misc", "downloader_sleep_time", 10, minval=0)
+receive_threads = OptionNumber("misc", "receive_threads", 2, minval=1)
 num_simd_decoders = OptionNumber("misc", "num_simd_decoders", 2, minval=1)
+switchinterval = OptionNumber("misc", "switchinterval", 0.005)
 ssdp_broadcast_interval = OptionNumber("misc", "ssdp_broadcast_interval", 15, minval=1, maxval=600)
 ext_rename_ignore = OptionList("misc", "ext_rename_ignore", validation=lower_case_ext)
 

sabnzbd/config.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -34,7 +34,14 @@ from urllib.parse import urlparse
 import configobj
 
 import sabnzbd
-from sabnzbd.constants import CONFIG_VERSION, NORMAL_PRIORITY, DEFAULT_PRIORITY, CONFIG_BACKUP_FILES, DEF_INI_FILE
+from sabnzbd.constants import (
+    CONFIG_VERSION,
+    NORMAL_PRIORITY,
+    DEFAULT_PRIORITY,
+    CONFIG_BACKUP_FILES,
+    CONFIG_BACKUP_HTTPS,
+    DEF_INI_FILE,
+)
 from sabnzbd.decorators import synchronized
 from sabnzbd.filesystem import clip_path, real_path, create_real_path, renamer, remove_file, is_writable
 
@@ -394,7 +401,6 @@ class ConfigServer:
     """Class defining a single server"""
 
     def __init__(self, name, values):
-
         self.__name = clean_section_name(name)
         name = "servers," + self.__name
 
@@ -944,6 +950,17 @@ def create_config_backup() -> Union[str, bool]:
                     if os.path.isfile(full_path):
                         with open(full_path, "rb") as data:
                             zip_ref.writestr(filename, data.read())
+                for filename, setting in CONFIG_BACKUP_HTTPS.items():
+                    full_path = getattr(sabnzbd.cfg, setting).get_path()
+                    # Only accept HTTPS config files that were successfully loaded by cherrypy on
+                    # startup to protect against last-minute breaking config changes as well as
+                    # inclusion of unrelated files in the backup through manipulated settings.
+                    if full_path and os.path.isfile(full_path) and full_path in sabnzbd.CONFIG_BACKUP_HTTPS_OK:
+                        logging.debug("Adding %s file %s to backup", setting, full_path)
+                        with open(full_path, "rb") as data:
+                            # Add the https cert/key/chain files with a fixed relative filename,
+                            # regardless of where they are actually stored on the filesystem
+                            zip_ref.writestr(filename, data.read())
                 with open(CFG_OBJ.filename, "rb") as data:
                     zip_ref.writestr(DEF_INI_FILE, data.read())
         return clip_path(complete_path)
@@ -966,6 +983,7 @@ def validate_config_backup(config_backup_data: bytes) -> bool:
 
 def restore_config_backup(config_backup_data: bytes):
     """Restore configuration files from zip file"""
+    global CFG_MODIFIED
     try:
         with io.BytesIO(config_backup_data) as backup_ref:
             with zipfile.ZipFile(backup_ref, "r") as zip_ref:
@@ -978,16 +996,22 @@ def restore_config_backup(config_backup_data: bytes):
 
                 # Write the rest of the admin files that we want to recover
                 adminpath = sabnzbd.cfg.admin_dir.get_path()
-                for filename in CONFIG_BACKUP_FILES:
+                for filename in CONFIG_BACKUP_FILES + list(CONFIG_BACKUP_HTTPS.keys()):
                     try:
                         zip_ref.getinfo(filename)
                         destination_file = os.path.join(adminpath, filename)
                         logging.debug("Writing backup of %s to %s", filename, destination_file)
                         with open(destination_file, "wb") as destination_ref:
                             destination_ref.write(zip_ref.read(filename))
+                        # For HTTPS config files, point the associated setting to the restored file
+                        if setting := CONFIG_BACKUP_HTTPS.get(filename):
+                            logging.debug("Setting value of %s to restored file %s", setting, filename)
+                            getattr(sabnzbd.cfg, setting).set(filename)
+                            CFG_MODIFIED = True
                     except KeyError:
                         # File not in archive
                         pass
+                save_config()
     except:
         logging.warning(T("Could not restore backup"))
         logging.info("Traceback: ", exc_info=True)

sabnzbd/constants.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -49,17 +49,11 @@ RENAMES_FILE = "__renames__"
 ATTRIB_FILE = "SABnzbd_attrib"
 REPAIR_REQUEST = "repair-all.sab"
 
-SABYENC_VERSION_REQUIRED = "5.4.4"
+SABCTOOLS_VERSION_REQUIRED = "6.1.0"
 
 DB_HISTORY_VERSION = 1
 DB_HISTORY_NAME = "history%s.db" % DB_HISTORY_VERSION
 
-CONFIG_BACKUP_FILES = [
-    BYTES_FILE_NAME,
-    RSS_FILE_NAME,
-    DB_HISTORY_NAME,
-]
-
 DEF_DOWNLOAD_DIR = os.path.normpath("Downloads/incomplete")
 DEF_COMPLETE_DIR = os.path.normpath("Downloads/complete")
 DEF_ADMIN_DIR = "admin"
@@ -82,14 +76,30 @@ DEF_ARTICLE_CACHE_DEFAULT = "500M"
 DEF_ARTICLE_CACHE_MAX = "1G"
 DEF_TIMEOUT = 60
 DEF_SCANRATE = 5
+DEF_HTTPS_CERT_FILE = "server.cert"
+DEF_HTTPS_KEY_FILE = "server.key"
 MAX_WARNINGS = 20
 MAX_BAD_ARTICLES = 5
 
+CONFIG_BACKUP_FILES = [
+    BYTES_FILE_NAME,
+    RSS_FILE_NAME,
+    DB_HISTORY_NAME,
+]
+CONFIG_BACKUP_HTTPS = {  # "basename": "associated setting"
+    DEF_HTTPS_CERT_FILE: "https_cert",
+    DEF_HTTPS_KEY_FILE: "https_key",
+    "server.chain": "https_chain",
+}
+
 # Constants affecting download performance
 MIN_DECODE_QUEUE = 10
 LIMIT_DECODE_QUEUE = 100
-DIRECT_WRITE_TRIGGER = 35
-MAX_ASSEMBLER_QUEUE = 5
+MAX_ASSEMBLER_QUEUE = 10
+SOFT_QUEUE_LIMIT = 0.6
+# Percentage of cache to use before adding file to assembler
+ASSEMBLER_WRITE_THRESHOLD = 5
+NNTP_BUFFER_SIZE = int(800 * KIBI)
 
 REPAIR_PRIORITY = 3
 FORCE_PRIORITY = 2

sabnzbd/database.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -554,7 +554,7 @@ def unpack_history_info(item: Union[Dict, sqlite3.Row]):
     return item
 
 
-def midnight_history_purge():
+def scheduled_history_purge():
     logging.info("Scheduled history purge")
     with HistoryDB() as history_db:
         history_db.auto_history_purge()

sabnzbd/decoder.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -26,28 +26,31 @@ import binascii
 from io import BytesIO
 from threading import Thread
 from typing import Tuple, List, Optional
+from zlib import crc32
 
 import sabnzbd
 import sabnzbd.cfg as cfg
-from sabnzbd.constants import SABYENC_VERSION_REQUIRED
+from sabnzbd.constants import SABCTOOLS_VERSION_REQUIRED
 from sabnzbd.encoding import ubtou
 from sabnzbd.nzbstuff import Article
 from sabnzbd.misc import match_str
 
-# Check for correct SABYenc version
-SABYENC_VERSION = None
-SABYENC_SIMD = None
+# Check for correct SABCTools version
+SABCTOOLS_VERSION = None
+SABCTOOLS_SIMD = None
+SABCTOOLS_OPENSSL_LINKED = None
 try:
-    import sabyenc3
+    import sabctools
 
-    SABYENC_ENABLED = True
-    SABYENC_VERSION = sabyenc3.__version__
-    SABYENC_SIMD = sabyenc3.simd
+    SABCTOOLS_ENABLED = True
+    SABCTOOLS_VERSION = sabctools.__version__
+    SABCTOOLS_SIMD = sabctools.simd
+    SABCTOOLS_OPENSSL_LINKED = sabctools.openssl_linked
     # Verify version to at least match minor version
-    if SABYENC_VERSION[:3] != SABYENC_VERSION_REQUIRED[:3]:
+    if SABCTOOLS_VERSION[:3] != SABCTOOLS_VERSION_REQUIRED[:3]:
         raise ImportError
 except:
-    SABYENC_ENABLED = False
+    SABCTOOLS_ENABLED = False
 
 
 class BadData(Exception):
@@ -68,7 +71,6 @@ class Decoder:
     """Implement thread-like coordinator for the decoders"""
 
     def __init__(self):
-
         # Initialize queue and servers
         self.decoder_queue = queue.Queue()
 
@@ -103,13 +105,13 @@ class Decoder:
             except:
                 pass
 
-    def process(self, article: Article, raw_data: List[bytes], data_size: int):
-        sabnzbd.ArticleCache.reserve_space(data_size)
-        self.decoder_queue.put((article, raw_data, data_size))
+    def process(self, article: Article, raw_data: bytearray, raw_data_size: int):
+        sabnzbd.ArticleCache.reserve_space(raw_data_size)
+        self.decoder_queue.put((article, raw_data, raw_data_size))
 
-    def queue_full(self) -> bool:
-        # Check if the queue size exceeds the limits
-        return self.decoder_queue.qsize() >= sabnzbd.ArticleCache.decoder_cache_article_limit
+    def queue_level(self) -> float:
+        # Return level of decoder queue. 0 = empty, >=1 = full.
+        return self.decoder_queue.qsize() / sabnzbd.ArticleCache.decoder_cache_article_limit
 
 
 class DecoderWorker(Thread):
@@ -118,14 +120,14 @@ class DecoderWorker(Thread):
     def __init__(self, decoder_queue):
         super().__init__()
         logging.debug("Initializing decoder %s", self.name)
-        self.decoder_queue: queue.Queue[Tuple[Optional[Article], Optional[List[bytes]], Optional[int]]] = decoder_queue
+        self.decoder_queue: queue.Queue[Tuple[Optional[Article], Optional[bytearray], Optional[int]]] = decoder_queue
 
     def run(self):
         while 1:
             # Set Article and NzbObject objects to None so references from this
             # thread do not keep the parent objects alive (see #1628)
             decoded_data = raw_data = article = nzo = None
-            article, raw_data, data_size = self.decoder_queue.get()
+            article, raw_data, raw_data_size = self.decoder_queue.get()
             if not article:
                 logging.debug("Shutting down decoder %s", self.name)
                 break
@@ -134,7 +136,7 @@ class DecoderWorker(Thread):
             art_id = article.article
 
             # Free space in the decoder-queue
-            sabnzbd.ArticleCache.free_reserved_space(data_size)
+            sabnzbd.ArticleCache.free_reserved_space(raw_data_size)
 
             # Keeping track
             article_success = False
@@ -149,7 +151,7 @@ class DecoderWorker(Thread):
                 if article.nzf.type == "uu":
                     decoded_data = decode_uu(article, raw_data)
                 else:
-                    decoded_data = decode_yenc(article, raw_data)
+                    decoded_data = decode_yenc(article, raw_data, raw_data_size)
 
                 article_success = True
 
@@ -181,7 +183,7 @@ class DecoderWorker(Thread):
 
             except (BadYenc, ValueError):
                 # Handles precheck and badly formed articles
-                if nzo.precheck and raw_data and raw_data[0].startswith(b"223 "):
+                if nzo.precheck and raw_data and raw_data.startswith(b"223 "):
                     # STAT was used, so we only get a status code
                     article_success = True
                 else:
@@ -195,9 +197,9 @@ class DecoderWorker(Thread):
                             pass
                     # Only bother with further checks if uu-decoding didn't work out
                     if not article_success:
-                        # Convert the initial chunks of raw socket data to article lines,
+                        # Convert the first 2000 bytes of raw socket data to article lines,
                         # and examine the headers (for precheck) or body (for download).
-                        for line in b"".join(raw_data[:2]).split(b"\r\n"):
+                        for line in raw_data[:2000].split(b"\r\n"):
                             lline = line.lower()
                             if lline.startswith(b"message-id:"):
                                 article_success = True
@@ -241,32 +243,35 @@ class DecoderWorker(Thread):
             sabnzbd.NzbQueue.register_article(article, article_success)
 
 
-def decode_yenc(article: Article, raw_data: List[bytes]) -> bytes:
-    # Let SABYenc do all the heavy lifting
-    decoded_data, yenc_filename, crc_correct = sabyenc3.decode_usenet_chunks(raw_data)
+def decode_yenc(article: Article, data: bytearray, raw_data_size: int) -> bytearray:
+    # Let SABCTools do all the heavy lifting
+    yenc_filename, crc_correct = sabctools.yenc_decode(data)
 
+    nzf = article.nzf
     # Assume it is yenc
-    article.nzf.type = "yenc"
+    nzf.type = "yenc"
 
     # Only set the name if it was found and not obfuscated
-    if not article.nzf.filename_checked and yenc_filename:
+    if not nzf.filename_checked and yenc_filename:
         # Set the md5-of-16k if this is the first article
         if article.lowest_partnum:
-            article.nzf.md5of16k = hashlib.md5(decoded_data[:16384]).digest()
+            nzf.md5of16k = hashlib.md5(data[:16384]).digest()
 
         # Try the rename, even if it's not the first article
         # For example when the first article was missing
-        article.nzf.nzo.verify_nzf_filename(article.nzf, yenc_filename)
+        nzf.nzo.verify_nzf_filename(nzf, yenc_filename)
 
     # CRC check
-    if not crc_correct:
+    if crc_correct is None:
         logging.info("CRC Error in %s", article.article)
-        raise BadData(decoded_data)
+        raise BadData(data)
 
-    return decoded_data
+    article.crc32 = crc_correct
+
+    return data
 
 
-def decode_uu(article: Article, raw_data: List[bytes]) -> bytes:
+def decode_uu(article: Article, raw_data: bytearray) -> bytes:
     """Try to uu-decode an article. The raw_data may or may not contain headers.
     If there are headers, they will be separated from the body by at least one
     empty line. In case of no headers, the first line seems to always be the nntp
@@ -276,10 +281,7 @@ def decode_uu(article: Article, raw_data: List[bytes]) -> bytes:
         raise BadUu
 
     # Line up the raw_data
-    with BytesIO() as encoded_data:
-        for data in raw_data:
-            encoded_data.write(data)
-        raw_data = encoded_data.getvalue().split(b"\r\n")
+    raw_data = raw_data.split(b"\r\n")
 
     # Index of the uu payload start in raw_data
     uu_start = 0
@@ -293,7 +295,7 @@ def decode_uu(article: Article, raw_data: List[bytes]) -> bytes:
     # Try to find an empty line separating the body from headers or response
     # code and set the expected payload start to the next line.
     try:
-        uu_start = raw_data[:limit].index(b"") + 1
+        uu_start = raw_data[:limit].index(bytearray(b"")) + 1
     except ValueError:
         # No empty line, look for a response code instead
         if raw_data[0].startswith(b"222 "):
@@ -385,7 +387,9 @@ def decode_uu(article: Article, raw_data: List[bytes]) -> bytes:
             if not article.nzf.filename_checked and uu_filename:
                 article.nzf.nzo.verify_nzf_filename(article.nzf, uu_filename)
 
-        return decoded_data.getvalue()
+        data = decoded_data.getvalue()
+        article.crc32 = crc32(data)
+        return data
 
 
 def search_new_server(article: Article) -> bool:

sabnzbd/decorators.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License

sabnzbd/deobfuscate_filenames.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License

sabnzbd/directunpacker.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -344,10 +344,14 @@ class DirectUnpacker(threading.Thread):
     def have_next_volume(self):
         """Check if next volume of set is available, start
         from the end of the list where latest completed files are
-        Make sure that files are 100% written to disk by checking md5sum
+        Make sure that files are 100% written to disk by checking nzf.assembled
         """
         for nzf_search in reversed(self.nzo.finished_files):
-            if nzf_search.setname == self.cur_setname and nzf_search.vol == (self.cur_volume + 1) and nzf_search.md5sum:
+            if (
+                nzf_search.setname == self.cur_setname
+                and nzf_search.vol == (self.cur_volume + 1)
+                and nzf_search.assembled
+            ):
                 return nzf_search
         return False
 

sabnzbd/dirscanner.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -19,10 +19,11 @@
 sabnzbd.dirscanner - Scanner for Watched Folder
 """
 
+import asyncio
 import os
-import time
 import logging
 import threading
+from typing import Generator, Set, Optional, Tuple
 
 import sabnzbd
 from sabnzbd.constants import SCAN_FILE_NAME, VALID_ARCHIVES, VALID_NZB_FILES
@@ -30,6 +31,9 @@ import sabnzbd.filesystem as filesystem
 import sabnzbd.config as config
 import sabnzbd.cfg as cfg
 
+DIR_SCANNER_LOCK = threading.RLock()
+VALID_EXTENSIONS = set(VALID_NZB_FILES + VALID_ARCHIVES)
+
 
 def compare_stat_tuple(tup1, tup2):
     """Test equality of two stat-tuples, content-related parts only"""
@@ -44,18 +48,10 @@ def compare_stat_tuple(tup1, tup2):
     return True
 
 
-def clean_file_list(inp_list, folder, files):
+async def clean_file_list(inp_list, files):
     """Remove elements of "inp_list" not found in "files" """
-    for path in sorted(inp_list):
-        fld, name = os.path.split(path)
-        if fld == folder:
-            present = False
-            for name in files:
-                if os.path.join(folder, name) == path:
-                    present = True
-                    break
-            if not present:
-                del inp_list[path]
+    for path in set(inp_list.keys()).difference(files):
+        del inp_list[path]
 
 
 class DirScanner(threading.Thread):
@@ -68,7 +64,15 @@ class DirScanner(threading.Thread):
     def __init__(self):
         super().__init__()
 
-        self.newdir()
+        self.loop: Optional[asyncio.AbstractEventLoop] = None
+        self.scanner_task: Optional[asyncio.Task] = None
+        self.lock = asyncio.Lock()  # Prevents concurrent scans
+        self.error_reported = False  # Prevents multiple reporting of missing watched folder
+        self.dirscan_dir = cfg.dirscan_dir.get_path()
+        self.dirscan_speed = cfg.dirscan_speed()
+        cfg.dirscan_dir.callback(self.newdir)
+        cfg.dirscan_speed.callback(self.newspeed)
+
         try:
             dirscan_dir, self.ignored, self.suspected = sabnzbd.filesystem.load_admin(SCAN_FILE_NAME)
             if dirscan_dir != self.dirscan_dir:
@@ -79,34 +83,24 @@ class DirScanner(threading.Thread):
             # successfully processed ones that cannot be deleted
             self.suspected = {}  # Will hold name/attributes of suspected candidates
 
-        self.loop_condition = threading.Condition(threading.Lock())
-        self.shutdown = False
-        self.error_reported = False  # Prevents multiple reporting of missing watched folder
-        self.dirscan_dir = cfg.dirscan_dir.get_path()
-        self.dirscan_speed = cfg.dirscan_speed() or None  # If set to 0, use None so the wait() is forever
-        self.busy = False
-        cfg.dirscan_dir.callback(self.newdir)
-        cfg.dirscan_speed.callback(self.newspeed)
-
     def newdir(self):
         """We're notified of a dir change"""
         self.ignored = {}
         self.suspected = {}
         self.dirscan_dir = cfg.dirscan_dir.get_path()
-        self.dirscan_speed = cfg.dirscan_speed()
+
+        self.start_scanner()
 
     def newspeed(self):
         """We're notified of a scan speed change"""
-        # If set to 0, use None so the wait() is forever
-        self.dirscan_speed = cfg.dirscan_speed() or None
-        with self.loop_condition:
-            self.loop_condition.notify()
+        self.dirscan_speed = cfg.dirscan_speed()
+
+        self.start_scanner()
 
     def stop(self):
         """Stop the dir scanner"""
-        self.shutdown = True
-        with self.loop_condition:
-            self.loop_condition.notify()
+        if self.loop:
+            asyncio.run_coroutine_threadsafe(self.shutdown(), self.loop)
 
     def save(self):
         """Save dir scanner bookkeeping"""
@@ -115,99 +109,168 @@ class DirScanner(threading.Thread):
     def run(self):
         """Start the scanner"""
         logging.info("Dirscanner starting up")
-        self.shutdown = False
 
-        while not self.shutdown:
-            # Wait to be woken up or triggered
-            with self.loop_condition:
-                self.loop_condition.wait(self.dirscan_speed)
-            if self.dirscan_speed and not self.shutdown:
-                self.scan()
+        self.loop = asyncio.new_event_loop()
 
-    def scan(self):
-        """Do one scan of the watched folder"""
+        try:
+            self.start_scanner()
+            self.loop.run_forever()
+        finally:
+            self.loop.close()
 
-        def run_dir(folder, catdir):
-            try:
-                files = os.listdir(folder)
-            except OSError:
-                if not self.error_reported and not catdir:
-                    logging.error(T("Cannot read Watched Folder %s"), filesystem.clip_path(folder))
-                    self.error_reported = True
-                files = []
+    def start_scanner(self):
+        """Start the scanner if it is not already running"""
+        with DIR_SCANNER_LOCK:
+            if not self.loop:
+                logging.debug("Can not start scanner because loop not found")
+                return
 
-            for filename in files:
-                if self.shutdown:
-                    break
-                path = os.path.join(folder, filename)
-                if os.path.isdir(path) or path in self.ignored or filename[0] == ".":
-                    continue
+            if not self.scanner_task or self.scanner_task.done():
+                self.scanner_task = asyncio.run_coroutine_threadsafe(self.scanner(), self.loop)
 
-                if filesystem.get_ext(path) in VALID_NZB_FILES + VALID_ARCHIVES:
-                    try:
-                        stat_tuple = os.stat(path)
-                    except OSError:
+    def get_suspected_files(
+        self, folder: str, catdir: Optional[str] = None
+    ) -> Generator[Tuple[str, Optional[str], Optional[os.stat_result]], None, None]:
+        """Generator listing possible paths to NZB files"""
+
+        if catdir is None:
+            cats = config.get_categories()
+        else:
+            cats = {}
+
+        try:
+            with os.scandir(os.path.join(folder, catdir or "")) as it:
+                for entry in it:
+                    path = entry.path
+
+                    if path in self.ignored:
+                        # We still need to know that an ignored file is still present when we clean up
+                        yield path, catdir, None
                         continue
-                else:
-                    self.ignored[path] = 1
-                    continue
 
-                if path in self.suspected:
-                    if compare_stat_tuple(self.suspected[path], stat_tuple):
-                        # Suspected file still has the same attributes
+                    # If the entry is a catdir then recursion
+                    if entry.is_dir():
+                        if not catdir and entry.name.lower() in cats:
+                            yield from self.get_suspected_files(folder, entry.name)
                         continue
-                    else:
-                        del self.suspected[path]
 
-                if stat_tuple.st_size > 0:
-                    logging.info("Trying to import %s", path)
-
-                    # Wait until the attributes are stable for 1 second, but give up after 3 sec
-                    # This indicates that the file is fully written to disk
-                    for n in range(3):
-                        time.sleep(1.0)
+                    if filesystem.get_ext(path) in VALID_EXTENSIONS:
                         try:
-                            stat_tuple_tmp = os.stat(path)
+                            # https://docs.python.org/3/library/os.html#os.DirEntry.stat
+                            # On Windows, the st_ino, st_dev and st_nlink attributes of the stat_result are always set
+                            # to zero. Call os.stat() to get these attributes.
+                            if sabnzbd.WIN32:
+                                stat_tuple = os.stat(path)
+                            else:
+                                stat_tuple = entry.stat()
                         except OSError:
                             continue
-                        if compare_stat_tuple(stat_tuple, stat_tuple_tmp):
-                            break
-                        stat_tuple = stat_tuple_tmp
-                    else:
-                        # Not stable
-                        continue
-
-                    # Add the NZB's
-                    res, _ = sabnzbd.nzbparser.add_nzbfile(path, catdir=catdir, keep=False)
-                    if res < 0:
-                        # Retry later, for example when we can't read the file
-                        self.suspected[path] = stat_tuple
-                    elif res == 0:
-                        self.error_reported = False
                     else:
                         self.ignored[path] = 1
+                        yield path, catdir, None
+                        continue
+
+                    if path in self.suspected:
+                        if not compare_stat_tuple(self.suspected[path], stat_tuple):
+                            # Suspected file attributes have changed
+                            del self.suspected[path]
+
+                    yield path, catdir, stat_tuple
+        except:
+            if not self.error_reported and not catdir:
+                logging.error(T("Cannot read Watched Folder %s"), filesystem.clip_path(folder))
+                logging.info("Traceback: ", exc_info=True)
+                self.error_reported = True
+
+    async def when_stable_add_nzbfile(self, path: str, catdir: Optional[str], stat_tuple: os.stat_result):
+        """Try and import the NZB but wait until the attributes are stable for 1 second, but give up after 3 sec"""
+
+        logging.info("Trying to import %s", path)
+
+        # Wait until the attributes are stable for 1 second, but give up after 3 sec
+        # This indicates that the file is fully written to disk
+        for n in range(3):
+            await asyncio.sleep(1.0)
+
+            try:
+                stat_tuple_tmp = os.stat(path)
+            except OSError:
+                continue
+            if compare_stat_tuple(stat_tuple, stat_tuple_tmp):
+                break
+            stat_tuple = stat_tuple_tmp
+        else:
+            # Not stable
+            return
+
+        # Add the NZB's
+        res, _ = sabnzbd.nzbparser.add_nzbfile(path, catdir=catdir, keep=False)
+        if res < 0:
+            # Retry later, for example when we can't read the file
+            self.suspected[path] = stat_tuple
+        elif res == 0:
+            self.error_reported = False
+        else:
+            self.ignored[path] = 1
+
+    def scan(self):
+        """Schedule a scan of the watched folder"""
+        if not self.loop:
+            return
+
+        if not (dirscan_dir := self.dirscan_dir):
+            return
+
+        asyncio.run_coroutine_threadsafe(self.scan_async(dirscan_dir), self.loop)
+
+    async def scan_async(self, dirscan_dir: str):
+        """Do one scan of the watched folder"""
+        async with self.lock:
+            if sabnzbd.PAUSED_ALL:
+                return
+
+            files: Set[str] = set()
+            futures: Set[asyncio.Task] = set()
+
+            for path, catdir, stat_tuple in self.get_suspected_files(dirscan_dir):
+                files.add(path)
+
+                if path in self.ignored or path in self.suspected:
+                    continue
+
+                if stat_tuple.st_size > 0:
+                    futures.add(asyncio.create_task(self.when_stable_add_nzbfile(path, catdir, stat_tuple)))
+                    await asyncio.sleep(0)
 
             # Remove files from the bookkeeping that are no longer on the disk
-            clean_file_list(self.ignored, folder, files)
-            clean_file_list(self.suspected, folder, files)
+            # Wait for the paths found in this scan to finish
+            await asyncio.gather(clean_file_list(self.ignored, files), clean_file_list(self.suspected, files), *futures)
 
-        if not self.busy:
-            self.busy = True
-            dirscan_dir = self.dirscan_dir
-            if dirscan_dir and not sabnzbd.PAUSED_ALL:
-                run_dir(dirscan_dir, None)
+    async def scanner(self):
+        """Periodically scan the directory and add NZB files to the queue"""
+        while True:
+            if not (dirscan_speed := self.dirscan_speed):
+                break
 
-                try:
-                    dirscan_list = os.listdir(dirscan_dir)
-                except OSError:
-                    if not self.error_reported:
-                        logging.error(T("Cannot read Watched Folder %s"), filesystem.clip_path(dirscan_dir))
-                        self.error_reported = True
-                    dirscan_list = []
+            if not (dirscan_dir := self.dirscan_dir):
+                break
 
-                cats = config.get_categories()
-                for dd in dirscan_list:
-                    dpath = os.path.join(dirscan_dir, dd)
-                    if os.path.isdir(dpath) and dd.lower() in cats:
-                        run_dir(dpath, dd.lower())
-            self.busy = False
+            await self.scan_async(dirscan_dir)
+
+            await asyncio.sleep(dirscan_speed)
+
+    async def shutdown(self):
+        """Cancel all tasks and stop the loop"""
+        loop = asyncio.get_event_loop()
+
+        # Get all tasks except for this one
+        tasks = filter(lambda task: task is not asyncio.current_task(), asyncio.all_tasks())
+
+        # Cancel them all
+        for task in tasks:
+            task.cancel()
+
+        # Wait for the tasks to be done
+        await asyncio.gather(*tasks, return_exceptions=True)
+
+        loop.stop()

sabnzbd/downloader.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -29,14 +29,16 @@ import random
 import sys
 import ssl
 from typing import List, Dict, Optional, Union
+from concurrent.futures import ThreadPoolExecutor
 
 import sabnzbd
 from sabnzbd.decorators import synchronized, NzbQueueLocker, DOWNLOADER_CV
 from sabnzbd.newswrapper import NewsWrapper, NNTPPermanentError
 import sabnzbd.config as config
 import sabnzbd.cfg as cfg
-from sabnzbd.misc import from_units, nntp_to_msg, get_server_addrinfo, helpful_warning, int_conv
+from sabnzbd.misc import from_units, get_server_addrinfo, helpful_warning, int_conv
 from sabnzbd.utils.happyeyeballs import happyeyeballs
+from sabnzbd.constants import SOFT_QUEUE_LIMIT
 
 
 # Timeout penalty in minutes for each cause
@@ -55,6 +57,8 @@ _SERVER_CHECK_DELAY = 0.5
 _BPSMETER_UPDATE_DELAY = 0.05
 # How many articles should be prefetched when checking the next articles?
 _ARTICLE_PREFETCH = 20
+# Minimum expected size of TCP receive buffer
+_DEFAULT_CHUNK_SIZE = 32768
 
 TIMER_LOCK = RLock()
 
@@ -116,7 +120,6 @@ class Server:
         optional=False,
         retention=0,
     ):
-
         self.id: str = server_id
         self.newid: Optional[str] = None
         self.restart: bool = False
@@ -218,6 +221,28 @@ class Server:
             self.request = True
             Thread(target=self._request_info_internal).start()
 
+    def get_article(self):
+        """Get article from pre-fetched and pre-fetch new ones if necessary.
+        Articles that are too old for this server are immediately marked as tried"""
+        if self.article_queue:
+            return self.article_queue.pop(0)
+        elif self.next_article_search < time.time():
+            # Pre-fetch new articles
+            self.article_queue = sabnzbd.NzbQueue.get_articles(self, sabnzbd.Downloader.servers, _ARTICLE_PREFETCH)
+            if self.article_queue:
+                article = self.article_queue.pop(0)
+                # Mark expired articles as tried on this server
+                if self.retention and article.nzf.nzo.avg_stamp < time.time() - self.retention:
+                    sabnzbd.Downloader.decode(article)
+                    while self.article_queue:
+                        sabnzbd.Downloader.decode(self.article_queue.pop())
+                else:
+                    return article
+            else:
+                # No available articles, skip this server for a short time
+                self.next_article_search = time.time() + _SERVER_CHECK_DELAY
+        return None
+
     def reset_article_queue(self):
         logging.debug("Resetting article queue for %s", self)
         for article in self.article_queue:
@@ -250,6 +275,8 @@ class Downloader(Thread):
         "bandwidth_limit",
         "bandwidth_perc",
         "sleep_time",
+        "recv_pool",
+        "recv_threads",
         "paused_for_postproc",
         "shutdown",
         "server_restarts",
@@ -279,6 +306,10 @@ class Downloader(Thread):
         self.sleep_time_set()
         cfg.downloader_sleep_time.callback(self.sleep_time_set)
 
+        self.recv_threads: int = cfg.receive_threads()
+        self.recv_pool: Optional[ThreadPoolExecutor] = ThreadPoolExecutor(self.recv_threads)
+        logging.debug("Receive threads: %s", self.recv_threads)
+
         self.paused_for_postproc: bool = False
         self.shutdown: bool = False
 
@@ -503,7 +534,7 @@ class Downloader(Thread):
             # Make sure server address resolution is refreshed
             server.info = None
 
-    def decode(self, article, raw_data: Optional[List[bytes]] = None, data_size: Optional[int] = None):
+    def decode(self, article, raw_data: Optional[bytearray] = None, raw_data_size: Optional[int] = None):
         """Decode article and check the status of
         the decoder and the assembler
         """
@@ -518,39 +549,45 @@ class Downloader(Thread):
             return
 
         # Send to decoder-queue
-        sabnzbd.Decoder.process(article, raw_data, data_size)
+        sabnzbd.Decoder.process(article, raw_data, raw_data_size)
 
         # See if we need to delay because the queues are full
         logged_counter = 0
-        decoder_full = sabnzbd.Decoder.queue_full()
-        assembler_full = sabnzbd.Assembler.queue_full()
-        while not self.shutdown and (decoder_full or assembler_full):
-            # Only log/update once every second, to not waste any CPU-cycles
-            if not logged_counter % 10:
-                # Make sure the BPS-meter is updated
-                sabnzbd.BPSMeter.update()
 
-                # Update who is delaying us
-                sabnzbd.BPSMeter.delayed_decoder += int(decoder_full)
-                sabnzbd.BPSMeter.delayed_assembler += int(assembler_full)
-                logging.debug(
-                    "Delayed - %d seconds - Decoder queue: %d - Assembler queue: %d",
-                    logged_counter / 10,
-                    sabnzbd.Decoder.decoder_queue.qsize(),
-                    sabnzbd.Assembler.queue.qsize(),
-                )
+        decoder_level = sabnzbd.Decoder.queue_level()
+        assembler_level = sabnzbd.Assembler.queue_level()
 
-            # Wait and update the queue sizes
-            time.sleep(0.1)
-            logged_counter += 1
-            decoder_full = sabnzbd.Decoder.queue_full()
-            assembler_full = sabnzbd.Assembler.queue_full()
+        # Sleep for an increasing amount of time, depending on queue sizes.
+        if decoder_level > SOFT_QUEUE_LIMIT or assembler_level > SOFT_QUEUE_LIMIT:
+            time.sleep((decoder_level + assembler_level - SOFT_QUEUE_LIMIT) / 2)
+            sabnzbd.BPSMeter.delayed_decoder += int(decoder_level > SOFT_QUEUE_LIMIT)
+            sabnzbd.BPSMeter.delayed_assembler += int(assembler_level > SOFT_QUEUE_LIMIT)
+
+            while not self.shutdown and (sabnzbd.Decoder.queue_level() >= 1 or sabnzbd.Assembler.queue_level() >= 1):
+                # Only log/update once every second, to not waste any CPU-cycles
+                if not logged_counter % 10:
+                    # Make sure the BPS-meter is updated
+                    sabnzbd.BPSMeter.update()
+
+                    # Update who is delaying us
+                    logging.debug(
+                        "Delayed - %d seconds - Decoder queue: %d - Assembler queue: %d",
+                        logged_counter / 10,
+                        sabnzbd.Decoder.decoder_queue.qsize(),
+                        sabnzbd.Assembler.queue.qsize(),
+                    )
+
+                # Wait and update the queue sizes
+                time.sleep(0.1)
+                logged_counter += 1
 
     def run(self):
         # First check IPv6 connectivity
         sabnzbd.EXTERNAL_IPV6 = sabnzbd.misc.test_ipv6()
         logging.debug("External IPv6 test result: %s", sabnzbd.EXTERNAL_IPV6)
 
+        logging.debug("switchinterval = %s", sys.getswitchinterval())
+
         # Then we check SSL certificate checking
         sabnzbd.CERTIFICATE_VALIDATION = sabnzbd.misc.test_cert_checking()
         logging.debug("SSL verification test: %s", sabnzbd.CERTIFICATE_VALIDATION)
@@ -564,10 +601,13 @@ class Downloader(Thread):
         BPSMeter.update()
         next_bpsmeter_update = 0
 
-        # can_be_slowed variables
-        can_be_slowed: Optional[float] = None
-        can_be_slowed_timer: float = 0.0
-        next_stable_speed_check: float = 0.0
+        # Sleep check variables
+        last_max_chunk_size: int = 0
+        max_chunk_size: int = _DEFAULT_CHUNK_SIZE
+        # Debugging code for v4 test release
+        sleep_count_start: float = time.time()
+        sleep_count: int = 0
+        time_slept: float = 0
 
         # Check server expiration dates
         check_server_expiration()
@@ -633,32 +673,13 @@ class Downloader(Thread):
                             server.request_info()
                         break
 
-                    # Get article from pre-fetched ones or fetch new ones
-                    if server.article_queue:
-                        article = server.article_queue.pop(0)
-                    else:
-                        # Pre-fetch new articles
-                        server.article_queue = sabnzbd.NzbQueue.get_articles(server, self.servers, _ARTICLE_PREFETCH)
-                        if server.article_queue:
-                            article = server.article_queue.pop(0)
-                            # Mark expired articles as tried on this server
-                            if server.retention and article.nzf.nzo.avg_stamp < now - server.retention:
-                                self.decode(article)
-                                while server.article_queue:
-                                    self.decode(server.article_queue.pop())
-                                # Move to the next server, allowing the next server to already start
-                                # fetching the articles that were too old for this server
-                                break
-                        else:
-                            # Skip this server for a short time
-                            server.next_article_search = now + _SERVER_CHECK_DELAY
-                            break
+                    nw.article = server.get_article()
+                    if not nw.article:
+                        break
 
                     server.idle_threads.remove(nw)
                     server.busy_threads.append(nw)
 
-                    nw.article = article
-
                     if nw.connected:
                         self.__request_article(nw)
                     else:
@@ -695,43 +716,44 @@ class Downloader(Thread):
                     logging.info("Shutting down")
                     break
 
+            # If less data than possible was received then it should be ok to sleep a bit
+            if self.sleep_time:
+                if last_max_chunk_size > max_chunk_size:
+                    logging.debug("New max_chunk_size %d -> %d", max_chunk_size, last_max_chunk_size)
+                    max_chunk_size = last_max_chunk_size
+                elif last_max_chunk_size < max_chunk_size / 3:
+                    time_before = time.time()
+                    time.sleep(self.sleep_time)
+                    now = time.time()
+                    # Debugging code for v4 test release
+                    if now - time_before > self.sleep_time + 0.02:
+                        logging.debug("Slept %.5f seconds, sleep_time = %s", now - time_before, self.sleep_time)
+                    time_slept += now - time_before
+                    sleep_count += 1
+                    if sleep_count_start + 20 < now:
+                        if sleep_count > 21:
+                            logging.debug(
+                                "Slept %d times for an average of %.5f seconds the last %.2f seconds. sleep_time = %s",
+                                sleep_count,
+                                time_slept / sleep_count,
+                                now - sleep_count_start,
+                                self.sleep_time,
+                            )
+                        sleep_count_start = now
+                        sleep_count = 0
+                        time_slept = 0
+
+                last_max_chunk_size = 0
+
             # Use select to find sockets ready for reading/writing
             readkeys = self.read_fds.keys()
             if readkeys:
                 read, _, _ = select.select(readkeys, (), (), 1.0)
-
-                # Add a sleep if there are too few results compared to the number of active connections
-                if self.sleep_time:
-                    if can_be_slowed and len(read) < 1 + len(readkeys) / 10:
-                        time.sleep(self.sleep_time)
-
-                    # Initialize by waiting for stable speed and then enable sleep
-                    if can_be_slowed is None or can_be_slowed_timer:
-                        # Wait for stable speed to start testing
-
-                        if not can_be_slowed_timer and now > next_stable_speed_check:
-                            if BPSMeter.get_stable_speed(timespan=10):
-                                can_be_slowed_timer = now + 8
-                                can_be_slowed = 1
-                            else:
-                                next_stable_speed_check = now + _BPSMETER_UPDATE_DELAY
-
-                        # Check 10 seconds after enabling slowdown
-                        if can_be_slowed_timer and now > can_be_slowed_timer:
-                            # Now let's check if it was stable in the last 10 seconds
-                            can_be_slowed = BPSMeter.get_stable_speed(timespan=10)
-                            can_be_slowed_timer = 0
-                            if not can_be_slowed:
-                                self.sleep_time = 0
-                            logging.debug("Downloader-slowdown: %r", can_be_slowed)
-
             else:
                 read = []
-
                 BPSMeter.reset()
-
                 time.sleep(1.0)
-
+                max_chunk_size = _DEFAULT_CHUNK_SIZE
                 with DOWNLOADER_CV:
                     while (
                         (sabnzbd.NzbQueue.is_empty() or self.no_active_jobs() or self.paused_for_postproc)
@@ -748,92 +770,123 @@ class Downloader(Thread):
             if not read:
                 continue
 
-            for selected in read:
-                nw = self.read_fds[selected]
-                article = nw.article
-                server = nw.server
+            if self.recv_threads > 1:
+                for nw, bytes_received, done in self.recv_pool.map(self.__recv, read):
+                    if bytes_received > last_max_chunk_size:
+                        last_max_chunk_size = bytes_received
+                    self.__handle_recv_result(nw, bytes_received, done)
+                if self.bandwidth_limit:
+                    self.__check_speed()
+            else:
+                for selected in read:
+                    nw, bytes_received, done = self.__recv(selected)
+                    if bytes_received > last_max_chunk_size:
+                        last_max_chunk_size = bytes_received
+                    self.__handle_recv_result(nw, bytes_received, done)
+                    if self.bandwidth_limit and bytes_received:
+                        self.__check_speed()
 
-                try:
-                    bytes_received, done = nw.recv_chunk()
-                except ssl.SSLWantReadError:
-                    continue
-                except:
-                    self.__reset_nw(nw, "server closed connection", wait=False)
-                    continue
+    def __recv(self, selected):
+        nw = None
+        try:
+            nw = self.read_fds[selected]
+            bytes_received, done = nw.recv_chunk()
+            return nw, bytes_received, done
+        except ssl.SSLWantReadError:
+            return nw, 0, False
+        except:
+            return nw, 0, True
 
-                BPSMeter.update(server.id, bytes_received)
-                if self.bandwidth_limit and BPSMeter.bps + BPSMeter.sum_cached_amount > self.bandwidth_limit:
-                    BPSMeter.update()
-                    while BPSMeter.bps > self.bandwidth_limit:
-                        time.sleep(0.01)
-                        BPSMeter.update()
+    def __check_speed(self):
+        BPSMeter = sabnzbd.BPSMeter
+        if BPSMeter.bps + BPSMeter.sum_cached_amount > self.bandwidth_limit:
+            BPSMeter.update()
+            while BPSMeter.bps > self.bandwidth_limit:
+                time.sleep(0.01)
+                BPSMeter.update()
 
-                if nw.status_code != 222 and not done:
-                    if not nw.connected or nw.status_code == 480:
-                        if not self.__finish_connect_nw(nw):
-                            continue
-                        if nw.connected:
-                            logging.info("Connecting %s@%s finished", nw.thrdnum, nw.server.host)
-                            self.__request_article(nw)
+    def __handle_recv_result(self, nw: NewsWrapper, bytes_received: int = 0, done: bool = False):
+        if not bytes_received:
+            if done:
+                self.__reset_nw(nw, "server closed connection", wait=False)
+            return
 
-                    elif nw.status_code == 223:
-                        done = True
-                        logging.debug("Article <%s> is present", article.article)
+        article = nw.article
+        server = nw.server
+        sabnzbd.BPSMeter.update(server.id, bytes_received)
 
-                    elif nw.status_code == 211:
-                        logging.debug("group command ok -> %s", nntp_to_msg(nw.data))
-                        nw.group = nw.article.nzf.nzo.group
-                        nw.clear_data()
-                        self.__request_article(nw)
+        if nw.status_code != 222 and not done:
+            if not nw.connected or nw.status_code == 480:
+                if not self.__finish_connect_nw(nw):
+                    return
+                if nw.connected:
+                    logging.info("Connecting %s@%s finished", nw.thrdnum, nw.server.host)
+                    self.__request_article(nw)
 
-                    elif nw.status_code in (411, 423, 430):
-                        done = True
-                        logging.debug(
-                            "Thread %s@%s: Article %s missing (error=%s)",
-                            nw.thrdnum,
-                            nw.server.host,
-                            article.article,
-                            nw.status_code,
-                        )
-                        nw.clear_data()
+            elif nw.status_code == 223:
+                done = True
+                logging.debug("Article <%s> is present", article.article)
 
-                    elif nw.status_code == 500:
-                        if article.nzf.nzo.precheck:
-                            # Assume "STAT" command is not supported
-                            server.have_stat = False
-                            logging.debug("Server %s does not support STAT", server.host)
-                        else:
-                            # Assume "BODY" command is not supported
-                            server.have_body = False
-                            logging.debug("Server %s does not support BODY", server.host)
-                        nw.clear_data()
-                        self.__request_article(nw)
+            elif nw.status_code == 211:
+                logging.debug("group command ok -> %s", nw.nntp_msg)
+                nw.group = nw.article.nzf.nzo.group
+                nw.reset_data_buffer()
+                self.__request_article(nw)
 
-                if done:
-                    # Successful data, clear "bad" counter
-                    server.bad_cons = 0
-                    server.errormsg = server.warning = ""
+            elif nw.status_code in (411, 423, 430):
+                done = True
+                logging.debug(
+                    "Thread %s@%s: Article %s missing (error=%s)",
+                    nw.thrdnum,
+                    nw.server.host,
+                    article.article,
+                    nw.status_code,
+                )
+                nw.reset_data_buffer()
 
-                    # Update statistics and decode
-                    article.nzf.nzo.update_download_stats(BPSMeter.bps, server.id, nw.data_size)
-                    self.decode(article, nw.data, nw.data_size)
+            elif nw.status_code == 500:
+                if article.nzf.nzo.precheck:
+                    # Assume "STAT" command is not supported
+                    server.have_stat = False
+                    logging.debug("Server %s does not support STAT", server.host)
+                else:
+                    # Assume "BODY" command is not supported
+                    server.have_body = False
+                    logging.debug("Server %s does not support BODY", server.host)
+                nw.reset_data_buffer()
+                self.__request_article(nw)
 
-                    if sabnzbd.LOG_ALL:
-                        logging.debug("Thread %s@%s: %s done", nw.thrdnum, server.host, article.article)
+        if done:
+            # Successful data, clear "bad" counter
+            server.bad_cons = 0
+            server.errormsg = server.warning = ""
 
-                    # Reset connection for new activity
-                    nw.soft_reset()
-                    server.busy_threads.remove(nw)
-                    server.idle_threads.append(nw)
-                    self.remove_socket(nw)
+            # Update statistics and decode
+            article.nzf.nzo.update_download_stats(sabnzbd.BPSMeter.bps, server.id, nw.data_position)
+            self.decode(article, nw.get_data_buffer(), nw.data_position)
+
+            if sabnzbd.LOG_ALL:
+                logging.debug("Thread %s@%s: %s done", nw.thrdnum, server.host, article.article)
+
+            # Reset connection for new activity
+            nw.soft_reset()
+            # Request a new article immediately if possible
+            if nw.connected and server.active and not (self.paused or self.shutdown or self.paused_for_postproc):
+                nw.article = server.get_article()
+                if nw.article:
+                    self.__request_article(nw)
+                    return
+            server.busy_threads.remove(nw)
+            server.idle_threads.append(nw)
+            self.remove_socket(nw)
 
     def __finish_connect_nw(self, nw: NewsWrapper) -> bool:
         server = nw.server
         try:
             nw.finish_connect(nw.status_code)
             if sabnzbd.LOG_ALL:
-                logging.debug("%s@%s last message -> %s", nw.thrdnum, server.host, nntp_to_msg(nw.data))
-            nw.clear_data()
+                logging.debug("%s@%s last message -> %s", nw.thrdnum, server.host, nw.nntp_msg)
+            nw.reset_data_buffer()
         except NNTPPermanentError as error:
             # Handle login problems
             block = False
@@ -902,7 +955,7 @@ class Downloader(Thread):
                 T("Connecting %s@%s failed, message=%s"),
                 nw.thrdnum,
                 nw.server.host,
-                nntp_to_msg(nw.data),
+                nw.nntp_msg,
             )
             # No reset-warning needed, above logging is sufficient
             self.__reset_nw(nw, retry_article=False)

sabnzbd/emailer.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License

sabnzbd/encoding.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -36,7 +36,7 @@ def utob(str_in: AnyStr) -> bytes:
 
 def ubtou(str_in: AnyStr) -> str:
     """Shorthand for converting unicode bytes to UTF-8 string"""
-    if not isinstance(str_in, bytes):
+    if isinstance(str_in, str):
         return str_in
     return str_in.decode("utf-8")
 

sabnzbd/filesystem.py

@@ -67,8 +67,7 @@ def is_listed_ext(ext: str, ext_list: list) -> bool:
     thus return false for extensions such as 'r007' despite the substring match on 'r00').
     """
     for item in ext_list:
-        RE_EXT = sabnzbd.misc.convert_filter(item)
-        if RE_EXT:
+        if RE_EXT := sabnzbd.misc.convert_filter(item):
             try:
                 if len(RE_EXT.match(ext).group()) == len(ext):
                     return True
@@ -121,6 +120,14 @@ def is_writable(path: str) -> bool:
         return True
 
 
+def is_size(filepath: str, size: int) -> bool:
+    """Return True if filepath exists and is specified size"""
+    try:
+        return os.path.getsize(filepath) == size
+    except:
+        return False
+
+
 _DEVICES = (
     "con",
     "prn",
@@ -177,10 +184,13 @@ def has_win_device(filename: str) -> bool:
     return False
 
 
-CH_ILLEGAL = "/"
-CH_LEGAL = "+"
-CH_ILLEGAL_WIN = '\\/<>?*|"\t:'
-CH_LEGAL_WIN = "++{}!@#'+-"
+CH_ILLEGAL = "\0/"
+CH_LEGAL = "_+"
+CH_ILLEGAL_WIN = '\\/<>?*|":'
+CH_LEGAL_WIN = "++{}!@#'-"
+for i in range(1, 32):
+    CH_ILLEGAL_WIN += chr(i)
+    CH_LEGAL_WIN += "_"
 
 
 def sanitize_filename(name: str) -> str:
@@ -614,8 +624,7 @@ def set_chmod(path: str, permissions: int, allow_failures: bool = False):
 def set_permissions(path: str, recursive: bool = True):
     """Give folder tree and its files their proper permissions"""
     if not sabnzbd.WIN32:
-        custom_permissions = sabnzbd.cfg.permissions()
-        if custom_permissions:
+        if custom_permissions := sabnzbd.cfg.permissions():
             # If user set permissions, parse them
             custom_permissions = int(custom_permissions, 8)
 
@@ -1212,8 +1221,7 @@ def backup_exists(filename: str) -> bool:
 
 def backup_nzb(nzb_path: str):
     """Backup NZB file, return path to nzb if it was saved"""
-    nzb_backup_dir = sabnzbd.cfg.nzb_backup_dir.get_path()
-    if nzb_backup_dir:
+    if nzb_backup_dir := sabnzbd.cfg.nzb_backup_dir.get_path():
         logging.debug("Saving copy of %s in %s", get_filename(nzb_path), nzb_backup_dir)
         shutil.copy(nzb_path, nzb_backup_dir)
 

sabnzbd/getipaddress.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License

sabnzbd/interface.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -70,7 +70,13 @@ import sabnzbd.newsunpack
 from sabnzbd.utils.servertests import test_nntp_server_dict
 from sabnzbd.utils.getperformance import getcpu
 import sabnzbd.utils.ssdp
-from sabnzbd.constants import DEF_STD_CONFIG, DEFAULT_PRIORITY, CHEETAH_DIRECTIVES, EXCLUDED_GUESSIT_PROPERTIES
+from sabnzbd.constants import (
+    DEF_STD_CONFIG,
+    DEFAULT_PRIORITY,
+    CHEETAH_DIRECTIVES,
+    EXCLUDED_GUESSIT_PROPERTIES,
+    DEF_HTTPS_CERT_FILE,
+)
 from sabnzbd.lang import list_languages
 from sabnzbd.api import (
     list_scripts,
@@ -420,7 +426,7 @@ class MainPage:
             info["have_watched_dir"] = bool(cfg.dirscan_dir())
 
             info["cpumodel"] = getcpu()
-            info["cpusimd"] = sabnzbd.decoder.SABYENC_SIMD
+            info["cpusimd"] = sabnzbd.decoder.SABCTOOLS_SIMD
 
             # Have logout only with HTML and if inet=5, only when we are external
             info["have_logout"] = (
@@ -519,7 +525,7 @@ class Wizard:
             info["username"] = ""
             info["password"] = ""
             info["connections"] = ""
-            info["ssl"] = 0
+            info["ssl"] = 1
             info["ssl_verify"] = 2
         else:
             # Sort servers to get the first enabled one
@@ -676,7 +682,7 @@ class ConfigPage:
 
         conf["have_unzip"] = bool(sabnzbd.newsunpack.ZIP_COMMAND)
         conf["have_7zip"] = bool(sabnzbd.newsunpack.SEVENZIP_COMMAND)
-        conf["have_sabyenc"] = sabnzbd.decoder.SABYENC_ENABLED
+        conf["have_sabctools"] = sabnzbd.decoder.SABCTOOLS_ENABLED
         conf["have_mt_par2"] = sabnzbd.newsunpack.PAR2_MT
 
         conf["certificate_validation"] = sabnzbd.CERTIFICATE_VALIDATION
@@ -880,7 +886,9 @@ SPECIAL_VALUE_LIST = (
     "wait_ext_drive",
     "max_foldername_length",
     "url_base",
+    "receive_threads",
     "num_simd_decoders",
+    "switchinterval",
     "direct_unpack_threads",
     "ipv6_servers",
     "selftest_host",
@@ -959,7 +967,6 @@ class ConfigGeneral:
 
     @secured_expose(check_configlock=True)
     def index(self, **kwargs):
-
         conf = build_header(sabnzbd.WEB_DIR_CONFIG)
 
         conf["certificate_validation"] = sabnzbd.CERTIFICATE_VALIDATION
@@ -980,6 +987,7 @@ class ConfigGeneral:
 
         conf["language"] = cfg.language()
         conf["lang_list"] = list_languages()
+        conf["def_https_cert_file"] = DEF_HTTPS_CERT_FILE
 
         for kw in GENERAL_LIST:
             conf[kw] = config.get_config("misc", kw)()

sabnzbd/lang.py

@@ -1,6 +1,6 @@
 #!/usr/bin/python3 -OO
 # -*- coding: utf-8 -*-
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License

sabnzbd/misc.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python3 -OO
-# Copyright 2007-2022 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2007-2023 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -1058,20 +1058,6 @@ def match_str(text: AnyStr, matches: Tuple[AnyStr, ...]) -> Optional[AnyStr]:
     return None
 
 
-def nntp_to_msg(text: Union[List[AnyStr], str]) -> str:
-    """Format raw NNTP bytes data for display"""
-    if isinstance(text, list):
-        text = text[0]
-
-    # Only need to split if it was raw data
-    # Sometimes (failed login) we put our own texts
-    if not isinstance(text, bytes):
-        return text
-    else:
-        lines = text.split(b"\r\n")
-        return ubtou(lines[0])
-
-
 def recursive_html_escape(input_dict_or_list: Union[Dict[str, Any], List], exclude_items: Tuple[str, ...] = ()):
     """Recursively update the input_dict in-place with html-safe values"""
     if isinstance(input_dict_or_list, (dict, list)):