Update text files for 1.2.2

We converted the password to "-p<password>" and when we had to retry it due to a Windows long-path fail, it would become "-p-p<password" and would incorrectly report that the password was wrong.

INSTALL.txt

@@ -1,4 +1,4 @@
-                                SABnzbd 1.2.0
+                                SABnzbd 1.2.1
 
 -------------------------------------------------------------------------------
 0) LICENSE
@@ -78,7 +78,7 @@ Optional modules
                             https://sabnzbd.org/wiki/installation/yenc-0.4_py2.7.rar (Win32-only)
     openssl => 1.0.0        http://www.openssl.org/
                             v0.9.8 will work, but limits certificate validation
-    cryptography >= 1.2     use "pip install cryptography"
+    cryptography >= 1.0     use "pip install cryptography"
                             Enables certificate generation and detection of encrypted RAR-files
 
 Optional modules Unix/Linux/OSX

PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 1.0
 Name: SABnzbd
-Version: 1.2.0
-Summary: SABnzbd-1.2.0
+Version: 1.2.2
+Summary: SABnzbd-1.2.2
 Home-page: http://sabnzbd.org
 Author: The SABnzbd Team
 Author-email: team@sabnzbd.org

README.md

@@ -70,7 +70,7 @@ Basically:
 - `feature/my_feature` is a temporary feature branch based on `develop`.
 - `hotfix/my_hotfix` is an optional temporary branch for bugfix(es) based on `develop`.
 
-Condtions:
+Conditions:
 - Merging of a stable release into `master` will be simple: the release branch is always right.
 - `master` is not merged back to `develop`.
 - `develop` is not re-based on `master`.

README.mkd

@@ -1,50 +1,37 @@
-Release Notes  -  SABnzbd 1.2.0
-====================================
+Release Notes  -  SABnzbd 1.2.2
+==============================================
 
-## What's new in 1.2.0
-- New SSL engine:
-  * SSL connections will always negotiate strongest encryption available
-  * Strict verification of certificates (optional)
-  * Set custom SSL-Ciphers to increase performance (lowers encryption strength)
-  * Detail information in Status window on type of encryption used
-  * Webserver can use stronger encryption for HTTPS connections
-- New RAR-file processing with full support for RAR5:
-  * All passwords are now tried on the first complete RAR-file during download
-  * RAR-based verification when no par2-files or SFV-files are available
-- Improvements to Categories processing:
-  * Matching of start of category inside NZB with user-categories
-    For example 'Movies > HD' will match a 'movies' category
-  * Categories can now be ordered to customize matching to categories in NZB's
-- Improvements to RSS feed management:
-  * Multiple feeds can now share the same set of filters
-  * Added 'From Show SxxEyy' filter
-  * Feeds will only re-evaluate after clicking 'Apply Filters' or 'Read Feed'
-  * Information from newznab, nZEDb and nntmux tags used (age, size and show info)
-  * Feeds now show the category, age and when the NZB was added to the queue
-- Detection of par2-files in completely obfuscated NZB's
-- Verification (par2) and UnRAR can now be aborted from Glitter
-- Faster startup and restart of SABnzbd
-- Duplicates can now be marked as Failed (to notify external tools)
-- Config Search function
+## Bug fix in 1.2.2
+- Windows: job-directory incorrectly passed to PostProcessing-script
 
-## Changes:
-- Python post/pre/notification-scripts now require execute (+x) permissions
-- Dropped dependency on PyOpenSSL, now only requires cryptography package
-- Update 7zip to 16.04 and UnRar to 5.40 for Windows/macOS
-- Update Python to 2.7.13 on Windows and macOS binaries
-- CherryPy logging only enabled when forced via command line
-- Not having C yEnc module will trigger warning on startup
+## What's new in 1.2.1
+- QuickCheck will perform fast rename of obfuscated posts
+- RSS Downloaded page now shows icon to indicate source
+- HTML tags are filtered from single-line script output
+- New self-signed certificates now list local IP in SAN-list
+- Handle jobs on Windows with forbidden names (Con.*, Aux.*,..)
 
-## Bug fixes
-- Re-use IP-address for new connections when a server has open connections
-- Auto-disables CPU-usage optimizations if they result in slowdown
-- no_penalties now applies to all types of penalties
-- par2cmdline would fail jobs in folders ending on ".par2"
-- Fix problem where memory usage wasn't correctly limited, causing MemoryError's
-- Removing files from active download was not working correctly
-- Accept & fail for pre-queue script was not working correctly
-- Recursive unpack failed on Windows with very long paths
-- Problems loading Notifications page with Scripts Folder set
+## Bug fixes in 1.2.1
+- Fix crashing Assembler
+- 'Only Download Top of Queue' was broken for a long time
+- Cloaked files (RAR within RAR) were not detected anymore
+- Incorrectly labeled some downloads as Encrypted
+- Passwords were not parsed correctly from filenames
+- RSS reading could fail on missing attributes
+- Multi-feed RSS will not stop if only 1 feed is not functioning
+- Duplicate detection set to Fail would not work for RSS feeds
+- Incorrectly marking jobs with folders inside as failed
+- Categories were not matched properly if a list of tags was set
+- PostProcessing-script was not called on Accept&Fail or Dupe detect
+- Support for newer par2cmdline(-mt) versions that need -B parameter
+- Some newsservers would timeout when connecting
+- More robust detection of execute permissions for scripts
+- CPU type reporting on Windows and macOS
+- Failed to start with some localhost configs
+- Removed some more stalling issues
+- Retry rename 3x before falling back to copy during "Moving"
+- Catch several SSL errors of the webserver
+- Disk-space information is now only checked every 10 seconds
 
 ## Translations
 - Many translations updated, thanks to our translators!

SABnzbd.py

@@ -434,7 +434,7 @@ def print_modules():
         logging.warning(T('_yenc module... NOT found!'))
 
     if sabnzbd.HAVE_CRYPTOGRAPHY:
-        logging.info('Cryptography module... found!')
+        logging.info('Cryptography module (v%s)... found!', sabnzbd.HAVE_CRYPTOGRAPHY)
     else:
         logging.info('Cryptography module... NOT found!')
 
@@ -492,6 +492,9 @@ def all_localhosts():
     ips = []
     for item in info:
         item = item[4][0]
+        # Avoid problems on strange Linux settings
+        if not isinstance(item, basestring):
+            continue
         # Only return IPv6 when enabled
         if item not in ips and ('::1' not in item or sabnzbd.cfg.ipv6_hosting()):
             ips.append(item)
@@ -673,7 +676,7 @@ def find_free_port(host, currentport):
     n = 0
     while n < 10 and currentport <= 49151:
         try:
-            cherrypy.process.servers.check_port(host, currentport, timeout=0.1)
+            cherrypy.process.servers.check_port(host, currentport, timeout=0.025)
             return currentport
         except:
             currentport += 5
@@ -1032,13 +1035,13 @@ def main():
     if sabnzbd.DAEMON:
         if enable_https and https_port:
             try:
-                cherrypy.process.servers.check_port(cherryhost, https_port, timeout=0.1)
+                cherrypy.process.servers.check_port(cherryhost, https_port, timeout=0.025)
             except IOError, error:
                 Bail_Out(browserhost, cherryport)
             except:
                 Bail_Out(browserhost, cherryport, '49')
         try:
-            cherrypy.process.servers.check_port(cherryhost, cherryport, timeout=0.1)
+            cherrypy.process.servers.check_port(cherryhost, cherryport, timeout=0.025)
         except IOError, error:
             Bail_Out(browserhost, cherryport)
         except:
@@ -1060,7 +1063,7 @@ def main():
     if enable_https:
         port = https_port or cherryport
         try:
-            cherrypy.process.servers.check_port(browserhost, port, timeout=0.1)
+            cherrypy.process.servers.check_port(browserhost, port, timeout=0.025)
         except IOError, error:
             if str(error) == 'Port not bound.':
                 pass
@@ -1071,19 +1074,22 @@ def main():
                     if new_instance or not check_for_sabnzbd(url, upload_nzbs, autobrowser):
                         newport = find_free_port(browserhost, port)
                         if newport > 0:
-                            sabnzbd.cfg.https_port.set(newport)
                             notify_port_change = True
+                            # Save the new port
                             if https_port:
                                 https_port = newport
+                                sabnzbd.cfg.https_port.set(newport)
                             else:
+                                # In case HTTPS == HTTP port
                                 http_port = newport
+                                sabnzbd.cfg.port.set(newport)
         except:
             Bail_Out(browserhost, cherryport, '49')
 
     # NonSSL check if there's no HTTPS or we only use 1 port
     if not (enable_https and not https_port):
         try:
-            cherrypy.process.servers.check_port(browserhost, cherryport, timeout=0.1)
+            cherrypy.process.servers.check_port(browserhost, cherryport, timeout=0.025)
         except IOError, error:
             if str(error) == 'Port not bound.':
                 pass
@@ -1236,7 +1242,7 @@ def main():
         try:
             from ctypes import cdll
             libc = cdll.LoadLibrary('/usr/lib/libc.dylib')
-            boolSetResult = libc.setiopolicy_np(0, 1, 3)  # @UnusedVariable
+            boolSetResult = libc.setiopolicy_np(0, 1, 3)
             logging.info('[osx] IO priority set to throttle for process scope')
         except:
             logging.info('[osx] IO priority setting not supported')
@@ -1261,7 +1267,6 @@ def main():
     web_dirc = Web_Template(None, DEF_STDCONFIG, '')
 
     wizard_dir = os.path.join(sabnzbd.DIR_INTERFACES, 'wizard')
-    # sabnzbd.lang.install_language(os.path.join(wizard_dir, DEF_INT_LANGUAGE), sabnzbd.cfg.language(), 'wizard')
 
     sabnzbd.WEB_DIR = web_dir
     sabnzbd.WEB_DIR2 = web_dir2

cherrypy/wsgiserver/ssl_builtin.py

@@ -82,15 +82,16 @@ class BuiltinSSLAdapter(wsgiserver.SSLAdapter):
                 if 'http request' in e.args[1]:
                     # The client is speaking HTTP to an HTTPS server.
                     raise wsgiserver.NoSSLError
-                elif 'unknown protocol' in e.args[1]:
-                    # The client is speaking some non-HTTP protocol.
-                    # Drop the conn.
-                    return None, {}
-                elif 'unknown ca' in e.args[1]:
-                    # This error is thrown by builtin SSL if Safari connects
-                    # when self-signed certificates are used. The connection
-                    # can be dropped until the users adds the exception
-                    return None, {}
+
+                # Check if it's one of the known errors
+                # Errors that are caught by PyOpenSSL, but thrown by built-in ssl
+                _block_errors = ('unknown protocol', 'unknown ca', 'unknown_ca',
+                                 'inappropriate fallback', 'wrong version number',
+                                 'no shared cipher', 'certificate unknown', 'ccs received early')
+                for error_text in _block_errors:
+                    if error_text in e.args[1].lower():
+                        # Accepted error, let's pass
+                        return None, {}
             elif 'handshake operation timed out' in e.args[0]:
                 # This error is thrown by builtin SSL after a timeout
                 # when client is speaking HTTP to an HTTPS server.

interfaces/Config/templates/_inc_header_uc.tmpl

@@ -31,9 +31,9 @@
     <link rel="apple-touch-icon" sizes="180x180" href="${root}staticcfg/ico/apple-touch-icon-180x180-precomposed.png" />
     <link rel="apple-touch-icon" sizes="192x192" href="${root}staticcfg/ico/android-192x192.png" />
 
-    <link rel="stylesheet" type="text/css" href="${root}staticcfg/bootstrap/css/bootstrap.min.css" />
+    <link rel="stylesheet" type="text/css" href="${root}staticcfg/bootstrap/css/bootstrap.min.css?v=$version" />
     <link rel="stylesheet" type="text/css" href="${root}staticcfg/css/style.css?p=$pid" />
-    <link rel="shortcut icon" href="${root}staticcfg/ico/favicon.ico?v=1.1.0" />
+    <link rel="shortcut icon" href="${root}staticcfg/ico/favicon.ico?v=$version" />
 
     <script type="text/javascript">
         // Keeping track of the form state
@@ -59,9 +59,9 @@
         configTranslate.confirmLeave = "$T('confirmWithoutSavingPrompt')";
         configTranslate.searchPages = ['$T('cmenu-general')', '$T('cmenu-folders')', '$T('cmenu-switches')', '$T('cmenu-sorting')', '$T('cmenu-notif')', '$T('cmenu-special')']
     </script>
-    <script type="text/javascript" src="${root}staticcfg/js/jquery-3.1.1.min.js"></script>
-    <script type="text/javascript" src="${root}staticcfg/bootstrap/js/bootstrap.min.js"></script>
-    <script type="text/javascript" src="${root}staticcfg/js/script.js?p=$pid"></script>
+    <script type="text/javascript" src="${root}staticcfg/js/jquery-3.1.1.min.js?v=$version"></script>
+    <script type="text/javascript" src="${root}staticcfg/bootstrap/js/bootstrap.min.js?v=$version"></script>
+    <script type="text/javascript" src="${root}staticcfg/js/script.js?v=$version"></script>
     <script type="text/javascript">
         // Set default functions for the autocomplete everywhere
         \$.extend(\$.fn.typeahead.defaults, {

interfaces/Config/templates/config_general.tmpl

@@ -23,6 +23,11 @@
                     <input type="number" name="port" id="port" value="$port" size="8" data-original="$port" />
                     <span class="desc">$T('explain-port')</span>
                 </div>
+                <div class="field-pair <!--#if int($have_ssl) == 0 then "disabled" else ""#-->">
+                    <label class="config" for="enable_https">$T('opt-enable_https')</label>
+                    <input type="checkbox" name="enable_https" id="enable_https" value="1" <!--#if int($enable_https) > 0 then 'checked="checked"' else ""#--> <!--#if int($have_ssl) == 0 then "disabled" else ""#--> />
+                    <span class="desc">$T('explain-enable_https')</span>
+                </div>
                 <div class="field-pair">
                     <label class="config" for="web_dir">$T('opt-web_dir')</label>
                     <select name="web_dir" id="web_dir">
@@ -66,46 +71,39 @@
                        $T('explain-ask-language') <a href="https://sabnzbd.org/wiki/translate" target="_blank" class="alert-link">https://sabnzbd.org/wiki/translate</a>
                     </div>
                 </div>
-                <div class="field-pair <!--#if int($have_ssl) == 0 then "disabled" else ""#-->">
-                    <label class="config" for="enable_https">$T('opt-enable_https')</label>
-                    <input type="checkbox" name="enable_https" id="enable_https" value="1" <!--#if int($enable_https) > 0 then 'checked="checked"' else ""#--> <!--#if int($have_ssl) == 0 then "disabled" else ""#--> />
-                    <span class="desc">$T('explain-enable_https')</span>
+                <div class="field-pair advanced-settings">
+                    <h5 class="darkred nomargin">$T('base-folder'): <span class="path">$my_lcldata</span></h5>
                 </div>
-                <div id="enable_https_options" <!--#if int($enable_https) < 1 then 'style="display:none"' else ""#-->>
-                    <div class="field-pair">
-                        <h5 class="darkred nomargin">$T('base-folder'): <span class="path">$my_lcldata</span></h5>
-                    </div>
-                    <div class="field-pair">
-                        <label class="config" for="https_port">$T('opt-https_port')</label>
-                        <input type="number" name="https_port" id="https_port" value="$https_port" size="8" data-original="$https_port" />
-                        <span class="desc">$T('explain-https_port')</span>
-                    </div>
-
-                    <div class="field-pair">
-                        <label class="config" for="https_cert">$T('opt-https_cert')</label>
-                        <input type="text" name="https_cert" id="https_cert" value="$https_cert" />
-                        <button class="btn btn-default generate_cert" title="$T('explain-new-cert')" <!--#if int($have_cryptography) == 0 then "disabled" else ""#-->>
-                            <span class="glyphicon glyphicon-repeat"></span>
-                        </button>
-                        <span class="desc">$T('explain-https_cert')</span>
-                    </div>
-                    <div class="field-pair">
-                        <label class="config" for="https_key">$T('opt-https_key')</label>
-                        <input type="text" name="https_key" id="https_key" value="$https_key" />
-                        <button class="btn btn-default generate_cert" title="$T('explain-new-cert')" <!--#if int($have_cryptography) == 0 then "disabled" else ""#-->>
-                            <span class="glyphicon glyphicon-repeat"></span>
-                        </button>
-                        <span class="desc">$T('explain-https_key')</span>
-                    </div>
-                    <div class="field-pair">
-                        <label class="config" for="https_chain">$T('opt-https_chain')</label>
-                        <input type="text" name="https_chain" id="https_chain" value="$https_chain" />
-                        <span class="desc">$T('explain-https_chain')</span>
-                    </div>
+                <div class="field-pair advanced-settings">
+                    <label class="config" for="https_port">$T('opt-https_port')</label>
+                    <input type="number" name="https_port" id="https_port" value="$https_port" size="8" data-original="$https_port" />
+                    <span class="desc">$T('explain-https_port')</span>
+                </div>
+                <div class="field-pair advanced-settings">
+                    <label class="config" for="https_cert">$T('opt-https_cert')</label>
+                    <input type="text" name="https_cert" id="https_cert" value="$https_cert" />
+                    <button class="btn btn-default generate_cert" title="$T('explain-new-cert')" <!--#if int($have_cryptography) == 0 then "disabled" else ""#-->>
+                        <span class="glyphicon glyphicon-repeat"></span>
+                    </button>
+                    <span class="desc">$T('explain-https_cert')</span>
+                </div>
+                <div class="field-pair advanced-settings">
+                    <label class="config" for="https_key">$T('opt-https_key')</label>
+                    <input type="text" name="https_key" id="https_key" value="$https_key" />
+                    <button class="btn btn-default generate_cert" title="$T('explain-new-cert')" <!--#if int($have_cryptography) == 0 then "disabled" else ""#-->>
+                        <span class="glyphicon glyphicon-repeat"></span>
+                    </button>
+                    <span class="desc">$T('explain-https_key')</span>
+                </div>
+                <div class="field-pair advanced-settings">
+                    <label class="config" for="https_chain">$T('opt-https_chain')</label>
+                    <input type="text" name="https_chain" id="https_chain" value="$https_chain" />
+                    <span class="desc">$T('explain-https_chain')</span>
                 </div>
                 <div class="field-pair">
                     <button class="btn btn-default saveButton"><span class="glyphicon glyphicon-ok"></span> $T('button-saveChanges')</button>
                     <button class="btn btn-default sabnzbd_restart"><span class="glyphicon glyphicon-refresh"></span> $T('button-restart') SABnzbd</button>
+                    <button class="btn btn-default advancedButton enable_https_options"><span class="glyphicon glyphicon-cog"></span> $T('button-advanced')</button>
                 </div>
             </fieldset>
         </div>
@@ -142,7 +140,7 @@
                                 <option value="5" <!--#if $inet_exposure == 5 then 'selected="selected"' else ""#-->>$T('inet-ui') - $T('inet-external_login')</option>
                             </optgroup>
                         </select>
-                    <span class="desc">$T('explain-inet_exposure').replace('.','.<br>')</span>
+                    <span class="desc">$T('explain-inet_exposure').replace('. ','.<br><span class="label label-warning">'+$T('warning').upper()+'</span> ')</span>
                 </div>
                 <div class="field-pair">
                     <label class="config" for="local_ranges">$T('opt-local_ranges')</label>

interfaces/Config/templates/config_rss.tmpl

@@ -401,16 +401,16 @@
                             <form action="download" method="get">
                                 <input type="hidden" value="$feed" name="feed" />
                                 <input type="hidden" name="session" value="$session" />
-                                <input type="hidden" name="url" value="$job[0]" />
-                                <input type="hidden" name="nzbname" value="$job[4]" />
+                                <input type="hidden" name="url" value="$job['url']" />
+                                <input type="hidden" name="nzbname" value="$job['nzbname']" />
                                 <button type="submit" class="btn btn-default"><span class="glyphicon glyphicon-plus-sign"></span> $T('link-download')</button>
                             </form>
                         </td>
-                        <td>$job[3] $job[2]</td>
-                        <td data-sort-value="$job[6]">$job[5]</td>
-                        <td>$job[1]</td>
-                        <td>$job[11]</td>
-                        <td data-sort-value="$job[8]">$job[7]</td>
+                        <td>$job['rule'] $job['skip']</td>
+                        <td data-sort-value="$job['size']">$job['size_units']</td>
+                        <td>$job['title']</td>
+                        <td>$job['cat']</td>
+                        <td data-sort-value="$job['age_ms']">$job['age']</td>
                     </tr>
                 <!--#end for#-->
             </table>
@@ -437,16 +437,16 @@
                             <form action="download" method="get">
                                 <input type="hidden" value="$feed" name="feed" />
                                 <input type="hidden" name="session" value="$session" />
-                                <input type="hidden" name="url" value="$job[0]" />
-                                <input type="hidden" name="nzbname" value="$job[4]" />
+                                <input type="hidden" name="url" value="$job['url']" />
+                                <input type="hidden" name="nzbname" value="$job['nzbname']" />
                                 <button type="submit" class="btn btn-default"><span class="glyphicon glyphicon-plus-sign"></span> $T('link-download')</button>
                             </form>
                         </td>
-                        <td>$job[3] $job[2]</td>
-                        <td data-sort-value="$job[6]">$job[5]</td>
-                        <td>$job[1]</td>
-                        <td>$job[11]</td>
-                        <td data-sort-value="$job[8]">$job[7]</td>
+                        <td>$job['rule'] $job['skip']</td>
+                        <td data-sort-value="$job['size']">$job['size_units']</td>
+                        <td>$job['title']</td>
+                        <td>$job['cat']</td>
+                        <td data-sort-value="$job['age_ms']">$job['age']</td>
                     </tr>
                 <!--#end for#-->
             </table>
@@ -466,14 +466,20 @@
                             <th>$T('size')</th>
                             <th width="60%">$T('sort-title')</th>
                             <th>$T('category')</th>
+                            <th>$T('source')</th>
                         </tr>
                     </thead>
                     <!--#for $job in $downloaded#-->
                         <tr class="infoTableSeperator">
-                            <td data-sort-value="$job[10]">$job[9]</td>
-                            <td data-sort-value="$job[6]">$job[5]</td>
-                            <td>$job[1]</td>
-                            <td>$job[11]</td>
+                            <td data-sort-value="$job['time_downloaded_ms']">$job['time_downloaded']</td>
+                            <td data-sort-value="$job['size']">$job['size_units']</td>
+                            <td>$job['title']</td>
+                            <td>$job['cat']</td>
+                            <td data-sort-value="$job['baselink']" title="$job['baselink']">
+                                <!--#if $job['baselink']#-->
+                                <div class="favicon source-icon" style="background-image: url(//$job['baselink']/favicon.ico);" data-domain="$job['baselink']"></div>
+                                <!--#end if#-->
+                            </td>
                         </tr>
                     <!--#end for#-->
                 </table>

interfaces/Config/templates/config_server.tmpl

@@ -325,11 +325,6 @@
         setTimeout(function() { portBox.removeClass('port-highlight') }, 2000)
     })
 
-    \$('.advancedButton').click(function(event){
-        \$('.advanced-settings').toggle()
-        return false;
-    })
-
     \$('.testServer').click(function(event){
         removeObfuscation()
         var theButton = \$(this)
@@ -341,9 +336,11 @@
             url: "../../tapi",
             data: "mode=config&output=json&name=test_server&" + \$(this).parents('form:first').serialize()
         }).then(function(data) {
+            // Let's replace the link
+            msg = data.value.message.replace('https://sabnzbd.org/certificate-errors', '<a href="https://sabnzbd.org/certificate-errors" class="alert-link" target="_blank">https://sabnzbd.org/certificate-errors</a>')
             // Fill the box and enable the button
             resultBox.removeClass('alert-success alert-danger').show()
-            resultBox.text(data.value.message)
+            resultBox.html(msg)
             theButton.removeAttr("disabled")
             theButton.find('span').toggleClass('glyphicon-sort glyphicon-refresh spin-glyphicon')
 

interfaces/Config/templates/config_switches.tmpl

@@ -201,12 +201,6 @@
                         <input type="checkbox" name="overwrite_files" id="overwrite_files" value="1" <!--#if int($overwrite_files) > 0 then 'checked="checked"' else ""#--> />
                         <span class="desc">$T('explain-overwrite_files')</span>
                     </div>
-                    <div class="field-pair">
-                        <label class="config" for="unpack_check">$T('opt-unpack_check')</label>
-                        <input type="checkbox" name="unpack_check" id="unpack_check" value="1" <!--#if int($unpack_check) > 0 then 'checked="checked"' else ""#--> />
-                        <span class="desc">$T('explain-unpack_check')</span>
-                    </div>
-
                     <div class="field-pair">
                         <label class="config" for="script_can_fail">$T('opt-script_can_fail')</label>
                         <input type="checkbox" name="script_can_fail" id="script_can_fail" value="1" <!--#if int($script_can_fail) > 0 then 'checked="checked"' else ""#--> />

interfaces/Config/templates/login/main.tmpl

@@ -11,13 +11,13 @@
     <link rel="apple-touch-icon" sizes="152x152" href="../staticcfg/ico/apple-touch-icon-152x152-precomposed.png" />
     <link rel="apple-touch-icon" sizes="180x180" href="../staticcfg/ico/apple-touch-icon-180x180-precomposed.png" />
     <link rel="apple-touch-icon" sizes="192x192" href="../staticcfg/ico/android-192x192.png" />
-    <link rel="shortcut icon" href="../staticcfg/ico/favicon.ico?v=1.1.0" />
+    <link rel="shortcut icon" href="../staticcfg/ico/favicon.ico?v=$version" />
 
-    <link rel="stylesheet" type="text/css" href="../staticcfg/bootstrap/css/bootstrap.min.css" />
-    <link rel="stylesheet" type="text/css" href="../staticcfg/css/login.css?p=$pid" />
+    <link rel="stylesheet" type="text/css" href="../staticcfg/bootstrap/css/bootstrap.min.css?v=$version" />
+    <link rel="stylesheet" type="text/css" href="../staticcfg/css/login.css?v=$version" />
 
-    <script type="text/javascript" src="../staticcfg/js/jquery-3.1.1.min.js"></script>
-    <script type="text/javascript" src="../staticcfg/bootstrap/js/bootstrap.min.js"></script>
+    <script type="text/javascript" src="../staticcfg/js/jquery-3.1.1.min.js?v=$version"></script>
+    <script type="text/javascript" src="../staticcfg/bootstrap/js/bootstrap.min.js?v=$version"></script>
 </head>
 <html>
     <body>

interfaces/Config/templates/staticcfg/css/style.css

@@ -272,6 +272,9 @@ textarea:hover, input[type="date"]:hover, input[type="datetime"]:hover, input[ty
     overflow: auto;
     clear: both;
 }
+.label {
+    font-style: normal;
+}
 .padTable h3 {
     margin-top: 0;
 }
@@ -557,6 +560,9 @@ h2.activeRSS {
     margin: 0 6px 0 2px;
     text-align: center;
 }
+.source-icon span {
+    top: -3px;
+}
 .feed {
     text-decoration: none;
 }
@@ -981,7 +987,7 @@ input[type="checkbox"] {
     opacity: 0.7;
 }
 
-.Servers .advanced-settings {
+.advanced-settings {
     display: none;
 }
 

interfaces/Config/templates/staticcfg/js/script.js

@@ -402,7 +402,12 @@ $(document).ready(function () {
 
     // Hide or show HTTPS
     $('#enable_https').on('change', function() {
-        $('#enable_https_options').toggle()
+        $('.enable_https_options').toggle()
+    })
+
+    $('.advancedButton').click(function(event){
+        $('.advanced-settings').toggle()
+        return false;
     })
 });
 

interfaces/Glitter/templates/include_overlays.tmpl

@@ -122,7 +122,7 @@
                             <div class="col-sm-6" data-bind="visible: hasStatusInfo">
                                 <span data-bind="text: statusInfo.pystone"></span>
                                 <a href="#" data-bind="click: loadStatusInfo" data-tooltip="true" data-placement="right" title="$T('dashboard-repeatTest')"><span class="glyphicon glyphicon-repeat"></span></a>
-                                <small data-bind="truncatedText: statusInfo.cpumodel, length: 25"></small>
+                                <small data-bind="truncatedText: statusInfo.cpumodel, length: 25, attr: { 'data-original-title': statusInfo.cpumodel }" data-tooltip="true"></small>
                             </div>
                             <div class="col-sm-6 col-loading" data-bind="visible: !hasStatusInfo()">$T('Glitter-loading')<span class="loader-dot-one">.</span><span class="loader-dot-two">.</span><span class="loader-dot-three">.</span></div>
                         </div>
@@ -131,7 +131,7 @@
                             <div class="col-sm-6" data-bind="visible: hasDiskStatusInfo">
                                 <span data-bind="text: statusInfo.downloaddirspeed()"></span> MB/s
                                 <a href="#" class="diskspeed-button" data-bind="click: testDiskSpeed" data-tooltip="true" data-placement="right" title="$T('dashboard-repeatTest')"><span class="glyphicon glyphicon-repeat"></span></a>
-                                <small>(<span data-bind="truncatedText: statusInfo.downloaddir, length: 24"></span>)</small>
+                                <small>(<span data-bind="truncatedText: statusInfo.downloaddir, length: 24, attr: { 'data-original-title': statusInfo.downloaddir }" data-tooltip="true"></span>)</small>
                             </div>
                             <div class="col-sm-6 col-loading" data-bind="visible: !hasDiskStatusInfo()">$T('Glitter-loading')<span class="loader-dot-one">.</span><span class="loader-dot-two">.</span><span class="loader-dot-three">.</span></div>
                         </div>
@@ -140,7 +140,7 @@
                             <div class="col-sm-6" data-bind="visible: hasDiskStatusInfo">
                                 <span data-bind="text: statusInfo.completedirspeed()"></span> MB/s
                                 <a href="#" class="diskspeed-button" data-bind="click: testDiskSpeed" data-tooltip="true" data-placement="right" title="$T('dashboard-repeatTest')"><span class="glyphicon glyphicon-repeat"></span></a>
-                                <small>(<span data-bind="truncatedText: statusInfo.completedir, length: 24"></span>)</small>
+                                <small>(<span data-bind="truncatedText: statusInfo.completedir, length: 24, attr: { 'data-original-title': statusInfo.completedir }" data-tooltip="true"></span>)</small>
                             </div>
                             <div class="col-sm-6 col-loading" data-bind="visible: !hasDiskStatusInfo()">$T('Glitter-loading')<span class="loader-dot-one">.</span><span class="loader-dot-two">.</span><span class="loader-dot-three">.</span></div>
                         </div>

interfaces/wizard/inc_top.tmpl

@@ -3,11 +3,11 @@
     <head>
         <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
         <title>$T('wizard-quickstart')</title>
-        <link rel="stylesheet" type="text/css" href="../staticcfg/bootstrap/css/bootstrap.min.css?pid=$version"/>
-        <link rel="stylesheet" type="text/css" href="static/style.css?pid=$version"/>
-        <link rel="shortcut icon" href="../staticcfg/ico/favicon.ico?v=1.1.0" />
-        <script type="text/javascript" src="../staticcfg/js/jquery-3.1.1.min.js"></script>
-        <script type="text/javascript" src="../staticcfg/bootstrap/js/bootstrap.min.js"></script>
+        <link rel="stylesheet" type="text/css" href="../staticcfg/bootstrap/css/bootstrap.min.css?v=$version"/>
+        <link rel="stylesheet" type="text/css" href="static/style.css?v=$version"/>
+        <link rel="shortcut icon" href="../staticcfg/ico/favicon.ico?v=$version" />
+        <script type="text/javascript" src="../staticcfg/js/jquery-3.1.1.min.js?v=$version"></script>
+        <script type="text/javascript" src="../staticcfg/bootstrap/js/bootstrap.min.js?v=$version"></script>
     </head>
     <body>
         <div id="logo">

po/main/en.po

@@ -60,10 +60,6 @@ msgstr "Disable server:"
 msgid "enable server"
 msgstr "Enable server:"
 
-#: sabnzbd/skintext.py:661
-msgid "Generic Sorting"
-msgstr "Movie Sorting"
-
 #: sabnzbd/emailer.py:117
 msgid "The server didn't reply properly to the helo greeting"
 msgstr "The server didn't reply properly to the hello greeting"
@@ -135,6 +131,6 @@ msgstr ""
 
 #: sabnzbd/skintext.py:333
 msgid "If empty, the standard port will only listen to HTTPS."
-msgstr "If empty, the SABnzbd Port set above will only listen to HTTPS."
+msgstr "If empty, the SABnzbd Port set above will listen to HTTPS."
 
 

sabnzbd/__init__.py

@@ -96,7 +96,7 @@ except:
 
 try:
     import cryptography
-    HAVE_CRYPTOGRAPHY = True
+    HAVE_CRYPTOGRAPHY = cryptography.__version__
 except:
     HAVE_CRYPTOGRAPHY = False
 
@@ -292,9 +292,8 @@ def initialize(pause_downloader=False, clean_up=False, evalSched=False, repair=0
     sabnzbd.encoding.change_fsys(cfg.fsys_type())
 
     # Set cache limit
-    if sabnzbd.WIN32 or sabnzbd.DARWIN:
-        if cfg.cache_limit() == '' or cfg.cache_limit() == '200M':
-            cfg.cache_limit.set('450M')
+    if not cfg.cache_limit() or (cfg.cache_limit() == '200M' and (sabnzbd.WIN32 or sabnzbd.DARWIN)):
+        cfg.cache_limit.set(misc.get_cache_limit())
     ArticleCache.do.new_limit(cfg.cache_limit.get_int())
 
     check_incomplete_vs_complete()
@@ -609,7 +608,7 @@ def save_compressed(folder, filename, data):
     # Need to go to the save folder to
     # prevent the pathname being embedded in the GZ file
     here = os.getcwd()
-    os.chdir(misc.short_path(folder))
+    os.chdir(folder)
 
     if filename.endswith('.nzb'):
         filename += '.gz'
@@ -858,7 +857,7 @@ def keep_awake():
 def CheckFreeSpace():
     """ Check if enough disk space is free, if not pause downloader and send email """
     if cfg.download_free() and not sabnzbd.downloader.Downloader.do.paused:
-        if misc.diskfree(cfg.download_dir.get_path()) < cfg.download_free.get_float() / GIGI:
+        if misc.diskspace(cfg.download_dir.get_path(), force=True)[1] < cfg.download_free.get_float() / GIGI:
             logging.warning(T('Too little diskspace forcing PAUSE'))
             # Pause downloader, but don't save, since the disk is almost full!
             Downloader.do.pause(save=False)

sabnzbd/api.py

@@ -54,7 +54,7 @@ from sabnzbd.utils.json import JsonWriter
 
 from sabnzbd.utils.rsslib import RSS, Item
 from sabnzbd.utils.pathbrowser import folders_at_path
-from sabnzbd.misc import loadavg, to_units, diskfree, disktotal, get_ext, \
+from sabnzbd.misc import loadavg, to_units, diskspace, get_ext, \
     get_filename, int_conv, globber, globber_full, time_format, remove_all, \
     starts_with_path, cat_convert, clip_path, create_https_certificates, calc_age
 from sabnzbd.encoding import xml_name, unicoder, special_fixer, platform_encode, html_escape
@@ -64,6 +64,7 @@ from sabnzbd.utils.servertests import test_nntp_server_dict
 from sabnzbd.bpsmeter import BPSMeter
 from sabnzbd.rating import Rating
 from sabnzbd.getipaddress import localipv4, publicipv4, ipv6
+from sabnzbd.newsunpack import userxbit
 from sabnzbd.database import build_history_info, unpack_history_info, HistoryDB
 import sabnzbd.notifier
 import sabnzbd.rss
@@ -1472,8 +1473,8 @@ def qstatus_data():
         "noofslots": len(pnfo_list),
         "noofslots_total": qnfo.q_fullsize,
         "have_warnings": str(sabnzbd.GUIHANDLER.count()),
-        "diskspace1": diskfree(cfg.download_dir.get_path()),
-        "diskspace2": diskfree(cfg.complete_dir.get_path()),
+        "diskspace1": diskspace(cfg.download_dir.get_path())[1],
+        "diskspace2": diskspace(cfg.complete_dir.get_path())[1],
         "timeleft": calc_timeleft(qnfo.bytes_left, bpsnow),
         "loadavg": loadavg(),
         "speedlimit": "{1:0.{0}f}".format(int(speed_limit % 1 > 0), speed_limit),
@@ -1708,16 +1709,16 @@ def build_header(prim, webdir=''):
     if speed_limit_abs <= 0:
         speed_limit_abs = ''
 
-    free1 = diskfree(cfg.download_dir.get_path())
-    free2 = diskfree(cfg.complete_dir.get_path())
+    disk_total1, disk_free1 = diskspace(cfg.download_dir.get_path())
+    disk_total2, disk_free2 = diskspace(cfg.complete_dir.get_path())
 
     header['helpuri'] = 'https://sabnzbd.org/wiki/'
-    header['diskspace1'] = "%.2f" % free1
-    header['diskspace2'] = "%.2f" % free2
-    header['diskspace1_norm'] = to_units(free1 * GIGI)
-    header['diskspace2_norm'] = to_units(free2 * GIGI)
-    header['diskspacetotal1'] = "%.2f" % disktotal(cfg.download_dir.get_path())
-    header['diskspacetotal2'] = "%.2f" % disktotal(cfg.complete_dir.get_path())
+    header['diskspace1'] = "%.2f" % disk_free1
+    header['diskspace2'] = "%.2f" % disk_free2
+    header['diskspace1_norm'] = to_units(disk_free1 * GIGI)
+    header['diskspace2_norm'] = to_units(disk_free2 * GIGI)
+    header['diskspacetotal1'] = "%.2f" % disk_total1
+    header['diskspacetotal2'] = "%.2f" % disk_total2
     header['loadavg'] = loadavg()
     # Special formatting so only decimal points when needed
     header['speedlimit'] = "{1:0.{0}f}".format(int(speed_limit % 1 > 0), speed_limit)
@@ -2054,7 +2055,7 @@ def list_scripts(default=False, none=True):
                 if (sabnzbd.WIN32 and os.path.splitext(script)[1].lower() in PATHEXT and
                                       not win32api.GetFileAttributes(script) & win32file.FILE_ATTRIBUTE_HIDDEN) or \
                    script.endswith('.py') or \
-                   (not sabnzbd.WIN32 and os.access(script, os.X_OK) and not os.path.basename(script).startswith('.')):
+                   (not sabnzbd.WIN32 and userxbit(script) and not os.path.basename(script).startswith('.')):
                     lst.append(os.path.basename(script))
         if none:
             lst.insert(0, 'None')

sabnzbd/assembler.py

@@ -35,7 +35,7 @@ except:
 
 import sabnzbd
 from sabnzbd.misc import get_filepath, sanitize_filename, get_unique_filename, renamer, \
-    set_permissions, flag_file, long_path, clip_path, get_all_passwords, short_path
+    set_permissions, flag_file, long_path, clip_path, has_win_device, get_all_passwords
 from sabnzbd.constants import QCHECK_FILE, Status
 import sabnzbd.cfg as cfg
 from sabnzbd.articlecache import ArticleCache
@@ -76,7 +76,8 @@ class Assembler(Thread):
 
             if nzf:
                 sabnzbd.CheckFreeSpace()
-                filename = sanitize_filename(nzf.filename)
+                # We allow win_devices because otherwise par2cmdline fails to repair
+                filename = sanitize_filename(nzf.filename, allow_win_devices=True)
                 nzf.filename = filename
 
                 dupe = nzo.check_for_dupe(nzf)
@@ -119,7 +120,6 @@ class Assembler(Thread):
                         else:
                             logging.warning(T('WARNING: Aborted job "%s" because of encrypted RAR file (if supplied, all passwords were tried)'), nzo.final_name)
                             nzo.fail_msg = T('Aborted, encryption detected')
-                            import sabnzbd.nzbqueue
                             sabnzbd.nzbqueue.NzbQueue.do.end_job(nzo)
 
                     if unwanted_file:
@@ -132,7 +132,6 @@ class Assembler(Thread):
                         if cfg.action_on_unwanted_extensions() == 2:
                             logging.debug('Unwanted extension ... aborting')
                             nzo.fail_msg = T('Aborted, unwanted extension detected')
-                            import sabnzbd.nzbqueue
                             sabnzbd.nzbqueue.NzbQueue.do.end_job(nzo)
 
                     filter, reason = nzo_filtered_by_rating(nzo)
@@ -142,7 +141,6 @@ class Assembler(Thread):
                     elif filter == 2:
                         logging.warning(Ta('WARNING: Aborted job "%s" because of rating (%s)'), nzo.final_name, reason)
                         nzo.fail_msg = T('Aborted, rating filter matched (%s)') % reason
-                        import sabnzbd.nzbqueue
                         sabnzbd.nzbqueue.NzbQueue.do.end_job(nzo)
 
                     nzf.completed = True
@@ -312,24 +310,28 @@ def check_encrypted_and_unwanted_files(nzo, filepath):
     encrypted = False
     unwanted = None
 
-    if cfg.unwanted_extensions() or (nzo.encrypted == 0 and cfg.pause_on_pwrar()):
-        # Safe-format for Windows
-        # RarFile requires de-unicoded filenames for zf.testrar()
-        filepath_split = os.path.split(filepath)
-        workdir_short = short_path(filepath_split[0])
-        filepath = deunicode(os.path.join(workdir_short, filepath_split[1]))
+    if (cfg.unwanted_extensions() and cfg.action_on_unwanted_extensions()) or (nzo.encrypted == 0 and cfg.pause_on_pwrar()):
+        # These checks should not break the assembler
+        try:
+            # Rarfile freezes on Windows special names, so don't try those!
+            if sabnzbd.WIN32 and has_win_device(filepath):
+                return encrypted, unwanted
 
-        # Is it even a rarfile?
-        if rarfile.is_rarfile(filepath):
-            try:
+            # Is it even a rarfile?
+            if rarfile.is_rarfile(filepath):
                 zf = rarfile.RarFile(filepath, all_names=True)
+
                 # Check for encryption
                 if nzo.encrypted == 0 and cfg.pause_on_pwrar() and (zf.needs_password() or is_cloaked(filepath, zf.namelist())):
                     # Load all passwords
                     passwords = get_all_passwords(nzo)
 
-                    # if no cryptography installed, only error when no password was set
-                    if not sabnzbd.HAVE_CRYPTOGRAPHY and not passwords:
+                    # Cloaked job?
+                    if is_cloaked(filepath, zf.namelist()):
+                        nzo.encrypted = 1
+                        encrypted = True
+                    elif not sabnzbd.HAVE_CRYPTOGRAPHY and not passwords:
+                        # if no cryptography installed, only error when no password was set
                         logging.info(T('%s missing'), 'Python Cryptography')
                         nzo.encrypted = 1
                         encrypted = True
@@ -344,6 +346,11 @@ def check_encrypted_and_unwanted_files(nzo, filepath):
                                 logging.info('Trying password "%s" on job "%s"', password, nzo.final_name)
                                 try:
                                     zf.setpassword(password)
+                                except:
+                                    # On weird passwords the setpassword() will fail
+                                    # but the actual rartest() will work
+                                    pass
+                                try:
                                     zf.testrar()
                                     password_hit = password
                                     break
@@ -351,7 +358,11 @@ def check_encrypted_and_unwanted_files(nzo, filepath):
                                     # On CRC error we can continue!
                                     password_hit = password
                                     break
-                                except:
+                                except Exception as e:
+                                    # Did we start from the right volume?
+                                    if 'need to start extraction from a previous volume' in e[0]:
+                                        return encrypted, unwanted
+                                    # This one failed
                                     pass
 
                         # Did any work?
@@ -370,17 +381,16 @@ def check_encrypted_and_unwanted_files(nzo, filepath):
                         encrypted = False
 
                 # Check for unwanted extensions
-                if cfg.unwanted_extensions():
+                if cfg.unwanted_extensions() and cfg.action_on_unwanted_extensions():
                     for somefile in zf.namelist():
                         logging.debug('File contains: %s', somefile)
                         if os.path.splitext(somefile)[1].replace('.', '').lower() in cfg.unwanted_extensions():
                             logging.debug('Unwanted file %s', somefile)
                             unwanted = somefile
-                            zf.close()
                 zf.close()
                 del zf
-            except:
-                logging.debug('RAR file %s cannot be inspected', filepath)
+        except:
+            logging.info('Error during inspection of RAR-file %s', filepath, exc_info=True)
 
     return encrypted, unwanted
 

sabnzbd/bpsmeter.py

@@ -156,7 +156,6 @@ class BPSMeter(object):
     def defaults(self):
         """ Get the latest data from the database and assign to a fake server """
         logging.debug('Setting default BPS meter values')
-        import sabnzbd.database
         history_db = sabnzbd.database.HistoryDB()
         grand, month, week = history_db.get_history_size()
         history_db.close()
@@ -245,9 +244,8 @@ class BPSMeter(object):
             if self.have_quota and self.quota_enabled:
                 self.left -= amount
                 if self.left <= 0.0:
-                    from sabnzbd.downloader import Downloader
-                    if Downloader.do and not Downloader.do.paused:
-                        Downloader.do.pause()
+                    if sabnzbd.downloader.Downloader.do and not Downloader.do.paused:
+                        sabnzbd.downloader.Downloader.do.pause()
                         logging.warning(T('Quota spent, pausing downloading'))
 
         # Speedometer
@@ -465,9 +463,8 @@ class BPSMeter(object):
 
     def resume(self):
         """ Resume downloading """
-        from sabnzbd.downloader import Downloader
-        if cfg.quota_resume() and Downloader.do and Downloader.do.paused:
-            Downloader.do.resume()
+        if cfg.quota_resume() and sabnzbd.downloader.Downloader.do and sabnzbd.downloader.Downloader.do.paused:
+            sabnzbd.downloader.Downloader.do.resume()
 
     def midnight(self):
         """ Midnight action: dummy update for all servers """

sabnzbd/cfg.py

@@ -21,10 +21,9 @@ sabnzbd.cfg - Configuration Parameters
 import re
 
 import sabnzbd
-from sabnzbd.constants import DEF_HOST, DEF_PORT_WIN_SSL, DEF_PORT_WIN, DEF_STDINTF, \
-    DEF_DOWNLOAD_DIR, DEF_NZBBACK_DIR, DEF_PORT_UNIX_SSL, \
-    DEF_SCANRATE, DEF_PORT_UNIX, DEF_COMPLETE_DIR, \
-    DEF_ADMIN_DIR
+from sabnzbd.constants import DEF_HOST, DEF_PORT, DEF_STDINTF, DEF_ADMIN_DIR, \
+    DEF_DOWNLOAD_DIR, DEF_NZBBACK_DIR, DEF_SCANRATE, DEF_COMPLETE_DIR
+
 from sabnzbd.config import OptionBool, OptionNumber, OptionPassword, \
     OptionDir, OptionStr, OptionList, no_nonsense, \
     validate_octal, validate_safedir, \
@@ -189,7 +188,6 @@ complete_dir = OptionDir('misc', 'complete_dir', DEF_COMPLETE_DIR, create=False,
 script_dir = OptionDir('misc', 'script_dir', create=True, writable=False)
 nzb_backup_dir = OptionDir('misc', 'nzb_backup_dir', DEF_NZBBACK_DIR)
 admin_dir = OptionDir('misc', 'admin_dir', DEF_ADMIN_DIR, validation=validate_safedir)
-# log_dir = OptionDir('misc', 'log_dir', 'logs')
 dirscan_dir = OptionDir('misc', 'dirscan_dir', create=False)
 dirscan_speed = OptionNumber('misc', 'dirscan_speed', DEF_SCANRATE, 0, 3600)
 size_limit = OptionStr('misc', 'size_limit', '0')
@@ -201,14 +199,8 @@ sanitize_safe = OptionBool('misc', 'sanitize_safe', False)
 api_logging = OptionBool('misc', 'api_logging', True)
 
 cherryhost = OptionStr('misc', 'host', DEF_HOST)
-if sabnzbd.WIN32:
-    cherryport = OptionStr('misc', 'port', DEF_PORT_WIN)
-else:
-    cherryport = OptionStr('misc', 'port', DEF_PORT_UNIX)
-if sabnzbd.WIN32:
-    https_port = OptionStr('misc', 'https_port', DEF_PORT_WIN_SSL)
-else:
-    https_port = OptionStr('misc', 'https_port', DEF_PORT_UNIX_SSL)
+cherryport = OptionStr('misc', 'port', DEF_PORT)
+https_port = OptionStr('misc', 'https_port')
 
 username = OptionStr('misc', 'username')
 password = OptionPassword('misc', 'password')
@@ -242,14 +234,10 @@ https_chain = OptionDir('misc', 'https_chain', create=False)
 enable_https = OptionBool('misc', 'enable_https', False)
 
 language = OptionStr('misc', 'language', 'en')
-unpack_check = OptionBool('misc', 'unpack_check', True)
 no_penalties = OptionBool('misc', 'no_penalties', False)
 load_balancing = OptionNumber('misc', 'load_balancing', 2)
 ipv6_servers = OptionNumber('misc', 'ipv6_servers', 1, 0, 2)
 
-# Internal options, not saved in INI file
-debug_delay = OptionNumber('misc', 'debug_delay', 0, add=False)
-
 api_key = OptionStr('misc', 'api_key', create_api_key())
 nzb_key = OptionStr('misc', 'nzb_key', create_api_key())
 disable_key = OptionBool('misc', 'disable_api_key', False, protect=True)

sabnzbd/config.py

@@ -79,7 +79,7 @@ class Option(object):
 
     def get(self):
         """ Retrieve value field """
-        if self.__value != None:
+        if self.__value is not None:
             return self.__value
         else:
             return self.__default_val
@@ -340,7 +340,7 @@ class OptionPassword(Option):
 
 @synchronized(CONFIG_LOCK)
 def add_to_database(section, keyword, obj):
-    """ add object as secion/keyword to INI database """
+    """ add object as section/keyword to INI database """
     global database
     if section not in database:
         database[section] = {}
@@ -607,7 +607,7 @@ class ConfigRSS(object):
 
 
 def get_dconfig(section, keyword, nested=False):
-    """ Return a config values dictonary,
+    """ Return a config values dictionary,
         Single item or slices based on 'section', 'keyword'
     """
     data = {}
@@ -917,7 +917,7 @@ def get_categories(cat=0):
     if '*' not in cats:
         ConfigCat('*', {'pp': old_def('dirscan_opts', '3'), 'script': old_def('dirscan_script', 'None'),
                         'priority': old_def('dirscan_priority', NORMAL_PRIORITY)})
-        # Add some categorie suggestions
+        # Add some category suggestions
         ConfigCat('movies', {})
         ConfigCat('tv', {})
         ConfigCat('audio', {})
@@ -951,6 +951,7 @@ def get_ordered_categories():
 
     return categories
 
+
 def define_rss():
     """ Define rss-feeds listed in the Setup file
         return a list of ConfigRSS instances

sabnzbd/constants.py

@@ -73,14 +73,12 @@ DEF_SKIN_COLORS = {'smpl': 'white', 'Glitter': 'Default', 'plush': 'gold'}
 DEF_MAIN_TMPL = 'templates/main.tmpl'
 DEF_INI_FILE = 'sabnzbd.ini'
 DEF_HOST = '127.0.0.1'
-DEF_PORT_WIN = 8080
-DEF_PORT_UNIX = 8080
-DEF_PORT_WIN_SSL = 9090
-DEF_PORT_UNIX_SSL = 9090
+DEF_PORT = 8080
 DEF_WORKDIR = 'sabnzbd'
 DEF_LOG_FILE = 'sabnzbd.log'
 DEF_LOG_ERRFILE = 'sabnzbd.error.log'
 DEF_LOG_CHERRY = 'cherrypy.log'
+DEF_CACHE_LIMIT = '450M'
 DEF_TIMEOUT = 60
 MIN_TIMEOUT = 10
 MAX_TIMEOUT = 200

sabnzbd/downloader.py

@@ -433,7 +433,7 @@ class Downloader(Thread):
                 if not server.idle_threads or server.restart or self.is_paused() or self.shutdown or self.delayed or self.postproc:
                     continue
 
-                if not (server.active and sabnzbd.nzbqueue.NzbQueue.do.has_articles_for(server)):
+                if not server.active:
                     continue
 
                 for nw in server.idle_threads[:]:
@@ -444,9 +444,6 @@ class Downloader(Thread):
                         else:
                             nw.timeout = None
 
-                    if not server.active:
-                        break
-
                     if server.info is None:
                         self.maybe_block_server(server)
                         request_server_info(server)

sabnzbd/getipaddress.py

@@ -1,5 +1,5 @@
 #!/usr/bin/python -OO
-# Copyright 2008-2016 The SABnzbd-Team <team@sabnzbd.org>
+# Copyright 2008-2017 The SABnzbd-Team <team@sabnzbd.org>
 #
 # This program is free software; you can redistribute it and/or
 # modify it under the terms of the GNU General Public License
@@ -24,7 +24,7 @@ import sabnzbd
 import sabnzbd.cfg
 import multiprocessing.pool
 import functools
-import socket
+
 
 # decorator stuff:
 def timeout(max_timeout):
@@ -41,19 +41,20 @@ def timeout(max_timeout):
         return func_wrapper
     return timeout_decorator
 
+
 @timeout(3.0)
 def addresslookup(myhost):
-    return socket.getaddrinfo(myhost,80)
+    return socket.getaddrinfo(myhost, 80)
+
 
 @timeout(3.0)
 def addresslookup4(myhost):
-    return socket.getaddrinfo(myhost,80, socket.AF_INET)
+    return socket.getaddrinfo(myhost, 80, socket.AF_INET)
+
 
 @timeout(3.0)
 def addresslookup6(myhost):
-    return socket.getaddrinfo(myhost,80, socket.AF_INET6)
-
-
+    return socket.getaddrinfo(myhost, 80, socket.AF_INET6)
 
 
 def localipv4():
@@ -66,6 +67,7 @@ def localipv4():
         ipv4 = None
     return ipv4
 
+
 def publicipv4():
     # Because of dual IPv4/IPv6 clients, finding the public ipv4 needs special attention,
     # meaning forcing IPv4 connections, and not allowing IPv6 connections
@@ -75,7 +77,7 @@ def publicipv4():
         # we only want IPv4 resolving, so socket.AF_INET:
         result = addresslookup4(sabnzbd.cfg.selftest_host())
     except:
-	# something very bad: no urllib2, no resolving of selftest_host, no network at all
+    # something very bad: no urllib2, no resolving of selftest_host, no network at all
         public_ipv4 = None
         return public_ipv4
     # we got one or more IPv4 address(es), so let's connect to them
@@ -89,7 +91,7 @@ def publicipv4():
             # specify the Host, because we only provide the IPv4 address in the URL:
             req.add_header('Host', sabnzbd.cfg.selftest_host())
             # get the response
-            public_ipv4 = urllib2.urlopen(req, timeout=2).read() # timeout 2 seconds, in case website is not accessible
+            public_ipv4 = urllib2.urlopen(req, timeout=2).read()  # timeout 2 seconds, in case the website is not accessible
             # ... check the response is indeed an IPv4 address:
             socket.inet_aton(public_ipv4)  # if we got anything else than a plain IPv4 address, this will raise an exception
             # if we get here without exception, we're done:
@@ -99,10 +101,11 @@ def publicipv4():
             # the connect OR the inet_aton raised an exception, so:
             # continue the for loop to try next server IPv4 address
             pass
-    if not ipv4_found :
+    if not ipv4_found:
         public_ipv4 = None
     return public_ipv4
 
+
 def ipv6():
     try:
         s_ipv6 = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)

sabnzbd/interface.py

@@ -1264,7 +1264,7 @@ class ConfigPage(object):
 
         conf['have_unzip'] = bool(sabnzbd.newsunpack.ZIP_COMMAND)
         conf['have_7zip'] = bool(sabnzbd.newsunpack.SEVEN_COMMAND)
-        conf['have_cryptography'] = sabnzbd.HAVE_CRYPTOGRAPHY
+        conf['have_cryptography'] = bool(sabnzbd.HAVE_CRYPTOGRAPHY)
         conf['have_yenc'] = HAVE_YENC
 
         if sabnzbd.HAVE_SSL:
@@ -1419,7 +1419,7 @@ SWITCH_LIST = \
              'safe_postproc', 'no_dupes', 'replace_spaces', 'replace_dots', 'replace_illegal',
              'ignore_samples', 'pause_on_post_processing', 'quick_check', 'nice', 'ionice',
              'pre_script', 'pause_on_pwrar', 'sfv_check', 'folder_rename', 'load_balancing',
-             'unpack_check', 'quota_size', 'quota_day', 'quota_resume', 'quota_period',
+             'quota_size', 'quota_day', 'quota_resume', 'quota_period',
              'pre_check', 'max_art_tries', 'max_art_opt', 'fail_hopeless_jobs', 'enable_all_par',
              'enable_recursive', 'no_series_dupes', 'script_can_fail', 'new_nzb_on_failure',
              'unwanted_extensions', 'action_on_unwanted_extensions', 'enable_meta', 'sanitize_safe',
@@ -1609,7 +1609,7 @@ class ConfigGeneral(object):
 
         conf['have_ssl'] = sabnzbd.HAVE_SSL
         conf['have_ssl_context'] = sabnzbd.HAVE_SSL_CONTEXT
-        conf['have_cryptography'] = sabnzbd.HAVE_CRYPTOGRAPHY
+        conf['have_cryptography'] = bool(sabnzbd.HAVE_CRYPTOGRAPHY)
 
         wlist = []
         interfaces = globber_full(sabnzbd.DIR_INTERFACES)
@@ -2255,7 +2255,6 @@ class ConfigRss(object):
                 sabnzbd.add_url(url, pp, script, cat, prio, nzbname)
             # Need to pass the title instead
             sabnzbd.rss.flag_downloaded(feed, url)
-            self.__evaluate = True
         raise rssRaiser(self.__root, kwargs)
 
     @cherrypy.expose
@@ -2795,55 +2794,67 @@ def ShowString(name, string):
 
 def GetRssLog(feed):
     def make_item(job):
-        url = job.get('url', '')
-        title = xml_name(job.get('title', ''))
-        size = job.get('size')
-        age = job.get('age', 0)
-        age_ms = job.get('age', 0)
-        time_downloaded = job.get('time_downloaded')
-        time_downloaded_ms = job.get('time_downloaded')
+        # Make a copy
+        job = job.copy()
 
-        if sabnzbd.rss.special_rss_site(url):
-            nzbname = ""
+        # Now we apply some formatting
+        job['title'] = xml_name(job['title'])
+        job['skip'] = '*' * int(job.get('status', '').endswith('*'))
+        # These fields could be empty
+        job['cat'] = job.get('cat', '')
+        job['size'] = job.get('size', '')
+
+        # Auto-fetched jobs didn't have these fields set
+        if job.get('url'):
+            job['baselink'] = get_base_url(job.get('url'))
+            if sabnzbd.rss.special_rss_site(job.get('url')):
+                job['nzbname'] = ''
+            else:
+                job['nzbname'] = xml_name(job['title'])
         else:
-            nzbname = xml_name(job.get('title', ''))
+            job['baselink'] = ''
+            job['nzbname'] = xml_name(job['title'])
 
-        if size:
-            size = to_units(size)
+        if job.get('size', 0):
+            job['size_units'] = to_units(job['size'])
+        else:
+            job['size_units'] = '-'
 
-        if age:
-            age = calc_age(age, True)
-            age_ms = time.mktime(age_ms.timetuple())
+        # And we add extra fields for sorting
+        if job.get('age', 0):
+            job['age_ms'] = time.mktime(job['age'].timetuple())
+            job['age'] = calc_age(job['age'], True)
+        else:
+            job['age_ms'] = ''
+            job['age'] = ''
 
-        if time_downloaded:
-            time_downloaded = time.strftime(time_format('%H:%M %a %d %b'), time_downloaded).decode(codepage)
-            time_downloaded_ms = time.mktime(time_downloaded_ms)
+        if job.get('time_downloaded'):
+            job['time_downloaded_ms'] = time.mktime(job['time_downloaded'])
+            job['time_downloaded'] = time.strftime(time_format('%H:%M %a %d %b'), job['time_downloaded']).decode(codepage)
+        else:
+            job['time_downloaded_ms'] = ''
+            job['time_downloaded'] = ''
 
-        # Also return extra fields for sorting
-        return url, \
-               title, \
-               '*' * int(job.get('status', '').endswith('*')), \
-               job.get('rule', 0), \
-               nzbname, \
-               size, \
-               job.get('size'), \
-               age, \
-               age_ms, \
-               time_downloaded, \
-               time_downloaded_ms,\
-               job.get('cat')
+        return job
 
-    jobs = sabnzbd.rss.show_result(feed)
-    names = jobs.keys()
-    # Sort in the order the jobs came from the feed
-    names.sort(lambda x, y: jobs[x].get('order', 0) - jobs[y].get('order', 0))
+    jobs = sabnzbd.rss.show_result(feed).values()
+    good, bad, done = ([], [], [])
+    for job in jobs:
+        if job['status'][0] == 'G':
+            good.append(make_item(job))
+        elif job['status'][0] == 'B':
+            bad.append(make_item(job))
+        elif job['status'] == 'D':
+            done.append(make_item(job))
 
-    good = [make_item(jobs[job]) for job in names if jobs[job]['status'][0] == 'G']
-    bad = [make_item(jobs[job]) for job in names if jobs[job]['status'][0] == 'B']
-
-    # Sort in reverse order of time stamp for 'Done'
-    names.sort(lambda x, y: int(jobs[y].get('time', 0) - jobs[x].get('time', 0)))
-    done = [make_item(jobs[job]) for job in names if jobs[job]['status'] == 'D']
+    try:
+        # Sort based on actual age, in try-catch just to be sure
+        good.sort(key=lambda job: job['age_ms'], reverse=True)
+        bad.sort(key=lambda job: job['age_ms'], reverse=True)
+        done.sort(key=lambda job: job['time_downloaded_ms'], reverse=True)
+    except:
+        # Let the javascript do it then..
+        pass
 
     return done, good, bad
 

sabnzbd/misc.py

@@ -36,7 +36,8 @@ from urlparse import urlparse
 
 import sabnzbd
 from sabnzbd.decorators import synchronized
-from sabnzbd.constants import DEFAULT_PRIORITY, FUTURE_Q_FOLDER, JOB_ADMIN, GIGI, MEBI
+from sabnzbd.constants import DEFAULT_PRIORITY, FUTURE_Q_FOLDER, JOB_ADMIN, \
+     GIGI, MEBI, DEF_CACHE_LIMIT
 import sabnzbd.config as config
 import sabnzbd.cfg as cfg
 from sabnzbd.encoding import unicoder, special_fixer, gUTF
@@ -186,9 +187,11 @@ def cat_convert(cat):
     """
     if cat and cat.lower() != 'none':
         cats = config.get_ordered_categories()
+        raw_cats = config.get_categories()
         for ucat in cats:
             try:
-                indexer = ucat['newzbin']
+                # Ordered cat-list has tags only as string
+                indexer = raw_cats[ucat['name']].newzbin()
                 if not isinstance(indexer, list):
                     indexer = [indexer]
             except:
@@ -236,6 +239,17 @@ def replace_win_devices(name):
                 break
     return name
 
+
+def has_win_device(p):
+    """ Return True if filename part contains forbidden name
+    """
+    p = os.path.split(p)[1].lower()
+    for dev in _DEVICES:
+        if p == dev or p.startswith(dev + '.'):
+            return True
+    return False
+
+
 if sabnzbd.WIN32:
     # the colon should be here too, but we'll handle that separately
     CH_ILLEGAL = r'\/<>?*|"'
@@ -245,7 +259,7 @@ else:
     CH_LEGAL = r'+'
 
 
-def sanitize_filename(name):
+def sanitize_filename(name, allow_win_devices=False):
     """ Return filename with illegal chars converted to legal ones
         and with the par2 extension always in lowercase
     """
@@ -262,6 +276,9 @@ def sanitize_filename(name):
             # Compensate for the foolish way par2 on OSX handles a colon character
             name = name[name.rfind(':') + 1:]
 
+    if sabnzbd.WIN32 and not allow_win_devices:
+        name = replace_win_devices(name)
+
     lst = []
     for ch in name.strip():
         if ch in illegal:
@@ -356,6 +373,24 @@ def sanitize_and_trim_path(path):
     return os.path.abspath(os.path.normpath(new_path))
 
 
+def sanitize_files_in_folder(folder):
+    """ Sanitize each file in the folder, return list of new names
+    """
+    lst = []
+    for root, _, files in os.walk(folder):
+        for file_ in files:
+            path = os.path.join(root, file_)
+            new_path = os.path.join(root, sanitize_filename(file_))
+            if path != new_path:
+                try:
+                    os.rename(path, new_path)
+                    path = new_path
+                except:
+                    logging.debug('Cannot rename %s to %s', path, new_path)
+            lst.append(path)
+    return lst
+
+
 def flag_file(path, flag, create=False):
     """ Create verify flag file or return True if it already exists """
     path = os.path.join(path, JOB_ADMIN)
@@ -787,6 +822,28 @@ def check_mount(path):
     return not m
 
 
+def get_cache_limit():
+    """ Depending on OS, calculate cache limit """
+    # OSX/Windows use Default value
+    if sabnzbd.WIN32 or sabnzbd.DARWIN:
+        return DEF_CACHE_LIMIT
+
+    # Calculate, if possible
+    try:
+        # Use 1/4th of available memory
+        mem_bytes = (os.sysconf('SC_PAGE_SIZE') * os.sysconf('SC_PHYS_PAGES'))/4
+        # Not more than the maximum we think is reasonable
+        if mem_bytes > from_units(DEF_CACHE_LIMIT):
+            return DEF_CACHE_LIMIT
+        elif mem_bytes > from_units('32M'):
+            # We make sure it's at least a valid value
+            return to_units(mem_bytes)
+    except:
+        pass
+    # If failed, leave empty so user needs to decide
+    return ''
+
+
 ##############################################################################
 # Locked directory operations
 ##############################################################################
@@ -854,19 +911,24 @@ def move_to_path(path, new_path):
         new_path = get_unique_filename(new_path)
 
     if new_path:
-        logging.debug("Moving. Old path:%s new path:%s overwrite?:%s",
+        logging.debug("Moving. Old path: %s New path: %s Overwrite: %s",
                                                   path, new_path, overwrite)
         try:
             # First try cheap rename
             renamer(path, new_path)
         except:
             # Cannot rename, try copying
+            logging.debug("File could not be renamed, trying copying: %s", path)
             try:
                 if not os.path.exists(os.path.dirname(new_path)):
                     create_dirs(os.path.dirname(new_path))
                 shutil.copyfile(path, new_path)
                 os.remove(path)
             except:
+                # Check if the old-file actually exists (possible delete-delays)
+                if not os.path.exists(path):
+                    logging.debug("File not moved, original path gone: %s", path)
+                    return True, None
                 if not (cfg.marker_file() and cfg.marker_file() in path):
                     logging.error(T('Failed moving %s to %s'), clip_path(path), clip_path(new_path))
                     logging.info("Traceback: ", exc_info=True)
@@ -1101,59 +1163,63 @@ if sabnzbd.WIN32:
     except:
         pass
 
-    def diskfree(_dir):
-        """ Return amount of free diskspace in GBytes """
+    def diskspace_base(_dir):
+        """ Return amount of free and used diskspace in GBytes """
         _dir = find_dir(_dir)
         try:
             available, disk_size, total_free = win32api.GetDiskFreeSpaceEx(_dir)
-            return available / GIGI
+            return disk_size / GIGI, available / GIGI
         except:
-            return 0.0
+            return 0.0, 0.0
 
-    def disktotal(_dir):
-        """ Return amount of free diskspace in GBytes """
-        _dir = find_dir(_dir)
-        try:
-            available, disk_size, total_free = win32api.GetDiskFreeSpaceEx(_dir)
-            return disk_size / GIGI
-        except:
-            return 0.0
 else:
     try:
         os.statvfs
         # posix diskfree
-
-        def diskfree(_dir):
-            """ Return amount of free diskspace in GBytes """
-            _dir = find_dir(_dir)
-            try:
-                s = os.statvfs(_dir)
-                if s.f_bavail < 0:
-                    return float(sys.maxint) * float(s.f_frsize) / GIGI
-                else:
-                    return float(s.f_bavail) * float(s.f_frsize) / GIGI
-            except OSError:
-                return 0.0
-
-        def disktotal(_dir):
-            """ Return amount of total diskspace in GBytes """
+        def diskspace_base(_dir):
+            """ Return amount of free and used diskspace in GBytes """
             _dir = find_dir(_dir)
             try:
                 s = os.statvfs(_dir)
                 if s.f_blocks < 0:
-                    return float(sys.maxint) * float(s.f_frsize) / GIGI
+                    disk_size = float(sys.maxint) * float(s.f_frsize)
                 else:
-                    return float(s.f_blocks) * float(s.f_frsize) / GIGI
-            except OSError:
-                return 0.0
+                    disk_size = float(s.f_blocks) * float(s.f_frsize)
+                if s.f_bavail < 0:
+                    available = float(sys.maxint) * float(s.f_frsize)
+                else:
+                    available = float(s.f_bavail) * float(s.f_frsize)
+                return disk_size / GIGI, available / GIGI
+            except:
+                return 0.0, 0.0
     except ImportError:
-        def diskfree(_dir):
-            return 10.0
-
-        def disktotal(_dir):
-            return 20.0
+        def diskspace_base(_dir):
+            return 20.0, 10.0
 
 
+__LAST_DISK_RESULT = {}
+__LAST_DISK_CALL = {}
+def diskspace(_dir, force=False):
+    """ Wrapper to cache results """
+    if _dir not in __LAST_DISK_RESULT:
+        __LAST_DISK_RESULT[_dir] = [0.0, 0.0]
+        __LAST_DISK_CALL[_dir] = 0.0
+
+    # When forced, ignore any cache to avoid problems in UI
+    if force:
+        return diskspace_base(_dir)
+
+    # Check against cache
+    if time.time() > __LAST_DISK_CALL[_dir] + 10.0:
+        __LAST_DISK_RESULT[_dir] = diskspace_base(_dir)
+        __LAST_DISK_CALL[_dir] = time.time()
+
+    return __LAST_DISK_RESULT[_dir]
+
+
+##############################################################################
+# Other support functions
+##############################################################################
 def create_https_certificates(ssl_cert, ssl_key):
     """ Create self-signed HTTPS certificates and store in paths 'ssl_cert' and 'ssl_key' """
     if not sabnzbd.HAVE_CRYPTOGRAPHY:
@@ -1164,7 +1230,7 @@ def create_https_certificates(ssl_cert, ssl_key):
     try:
         from sabnzbd.utils.certgen import generate_key, generate_local_cert
         private_key = generate_key(key_size=2048, output_file=ssl_key)
-        cert = generate_local_cert(private_key, days_valid=356*10, output_file=ssl_cert, LN=u'SABnzbd', ON=u'SABnzbd', CN=u'SABnzbd')
+        cert = generate_local_cert(private_key, days_valid=3560, output_file=ssl_cert, LN=u'SABnzbd', ON=u'SABnzbd', CN=u'localhost')
         logging.info('Self-signed certificates generated successfully')
     except:
         logging.error(T('Error creating SSL key and certificate'))
@@ -1283,6 +1349,18 @@ def renamer(old, new):
     if sabnzbd.WIN32:
         retries = 15
         while retries > 0:
+            # First we try 3 times with os.rename
+            if retries > 12:
+                try:
+                    os.rename(old, new)
+                    return
+                except:
+                    retries -= 1
+                    time.sleep(3)
+                    continue
+
+            # Now we try the back-up method
+            logging.debug('Could not rename, trying move for %s to %s', old, new)
             try:
                 shutil.move(old, new)
                 return
@@ -1435,31 +1513,10 @@ def set_permissions(path, recursive=True):
             set_chmod(path, umask_file, report)
 
 
-def short_path(path, always=True):
-    """ For Windows, return shortened ASCII path, for others same path
-        When `always` is off, only return a short path when size is above 259
-    """
-    if sabnzbd.WIN32:
-        import win32api
-        path = os.path.normpath(path)
-        if always or len(path) > 259:
-            # First make the path "long"
-            path = long_path(path)
-            if os.path.exists(path):
-                # Existing path can always be shortened
-                path = win32api.GetShortPathName(path)
-            else:
-                # For new path, shorten only existing part (recursive)
-                path1, name = os.path.split(path)
-                path = os.path.join(short_path(path1, always), name)
-        path = clip_path(path)
-    return path
-
-
 def clip_path(path):
     r""" Remove \\?\ or \\?\UNC\ prefix from Windows path """
     if sabnzbd.WIN32 and path and '?' in path:
-        path = path.replace(u'\\\\?\\UNC\\', u'\\\\').replace(u'\\\\?\\', u'')
+        path = path.replace(u'\\\\?\\UNC\\', u'\\\\', 1).replace(u'\\\\?\\', u'', 1)
     return path
 
 

sabnzbd/newsunpack.py

@@ -33,7 +33,8 @@ from sabnzbd.encoding import TRANS, UNTRANS, unicode2local, \
     reliable_unpack_names, unicoder, platform_encode, deunicode
 import sabnzbd.utils.rarfile as rarfile
 from sabnzbd.misc import format_time_string, find_on_path, make_script_path, int_conv, \
-    flag_file, real_path, globber, globber_full, short_path, get_all_passwords
+    flag_file, real_path, globber, globber_full, get_all_passwords, renamer, clip_path, \
+    has_win_device
 from sabnzbd.tvsort import SeriesSorter
 import sabnzbd.cfg as cfg
 from sabnzbd.constants import Status, QCHECK_FILE, RENAMES_FILE
@@ -204,7 +205,7 @@ def unpack_magic(nzo, workdir, workdir_complete, dele, one_folder, joinables, zi
 
     if depth == 1:
         # First time, ignore anything in workdir_complete
-        xjoinables, xzips, xrars, xsevens, xts = build_filelists(workdir, None)
+        xjoinables, xzips, xrars, xsevens, xts = build_filelists(workdir)
     else:
         xjoinables, xzips, xrars, xsevens, xts = build_filelists(workdir, workdir_complete)
 
@@ -445,8 +446,7 @@ def rar_unpack(nzo, workdir, workdir_complete, delete, one_folder, rars):
         if workdir_complete and rarpath.startswith(workdir):
             extraction_path = workdir_complete
         else:
-            # Make sure that path is not too long
-            extraction_path = short_path(os.path.split(rarpath)[0])
+            extraction_path = os.path.split(rarpath)[0]
 
         logging.info("Extracting rarfile %s (belonging to %s) to %s",
                      rarpath, rar_set, extraction_path)
@@ -532,18 +532,10 @@ def rar_extract_core(rarfile_path, numrars, one_folder, nzo, setname, extraction
 
     logging.debug("rar_extract(): Extractionpath: %s", extraction_path)
 
-    try:
-        zf = rarfile.RarFile(rarfile_path)
-        expected_files = zf.namelist()
-        zf.close()
-    except:
-        logging.info('Archive %s probably has full encryption', rarfile_path)
-        expected_files = []
-
     if password:
-        password = '-p%s' % password
+        password_command = '-p%s' % password
     else:
-        password = '-p-'
+        password_command = '-p-'
 
     ############################################################################
 
@@ -559,15 +551,16 @@ def rar_extract_core(rarfile_path, numrars, one_folder, nzo, setname, extraction
         rename = '-or'    # Auto renaming
     if sabnzbd.WIN32:
         # Use all flags
-        command = ['%s' % RAR_COMMAND, action, '-idp', overwrite, rename, '-ai', password,
-                   '%s' % rarfile_path, '%s/' % extraction_path]
+        # See: https://github.com/sabnzbd/sabnzbd/pull/771
+        command = ['%s' % RAR_COMMAND, action, '-idp', overwrite, rename, '-ai', password_command,
+                   '%s' % clip_path(rarfile_path), '%s\\' % extraction_path]
     elif RAR_PROBLEM:
         # Use only oldest options (specifically no "-or")
-        command = ['%s' % RAR_COMMAND, action, '-idp', overwrite, password,
+        command = ['%s' % RAR_COMMAND, action, '-idp', overwrite, password_command,
                    '%s' % rarfile_path, '%s/' % extraction_path]
     else:
         # Don't use "-ai" (not needed for non-Windows)
-        command = ['%s' % RAR_COMMAND, action, '-idp', overwrite, rename, password,
+        command = ['%s' % RAR_COMMAND, action, '-idp', overwrite, rename, password_command,
                    '%s' % rarfile_path, '%s/' % extraction_path]
 
     if cfg.ignore_unrar_dates():
@@ -650,6 +643,10 @@ def rar_extract_core(rarfile_path, numrars, one_folder, nzo, setname, extraction
             logging.error(T('ERROR: write error (%s)'), line[11:])
             fail = 1
 
+        elif line.startswith('Cannot create') and sabnzbd.WIN32 and extraction_path.startswith('\\\\?\\'):
+            # Can be due to Unicode problems on Windows, let's retry
+            fail = 4
+
         elif line.startswith('Cannot create'):
             line2 = proc.readline()
             if 'must not exceed 260' in line2:
@@ -723,43 +720,19 @@ def rar_extract_core(rarfile_path, numrars, one_folder, nzo, setname, extraction
             if proc:
                 proc.close()
             p.wait()
+            logging.debug('UNRAR output %s', '\n'.join(lines))
 
-            return fail, (), ()
+            # Unicode problems, lets start again but now we try without \\?\
+            # This will only fail if the download contains forbidden-Windows-names
+            if fail == 4:
+                return rar_extract_core(rarfile_path, numrars, one_folder, nzo, setname, clip_path(extraction_path), password)
+            else:
+                return fail, (), ()
 
     if proc:
         proc.close()
     p.wait()
 
-    if cfg.unpack_check():
-        if reliable_unpack_names() and not RAR_PROBLEM:
-            missing = []
-            # Loop through and check for the presence of all the files the archive contained
-            for path in expected_files:
-                if one_folder or cfg.flat_unpack():
-                    path = os.path.split(path)[1]
-                path = unicode2local(path)
-                if '?' in path:
-                    logging.info('Skipping check of file %s', path)
-                    continue
-                fullpath = os.path.join(extraction_path, path)
-                logging.debug("Checking existence of %s", fullpath)
-                if path.endswith('/'):
-                    # Folder
-                    continue
-                if not os.path.exists(fullpath):
-                    # There was a missing file, show a warning
-                    missing.append(path)
-                    logging.info(T('Missing expected file: %s => unrar error?'), path)
-
-            if missing:
-                nzo.fail_msg = T('Unpacking failed, an expected file was not unpacked')
-                logging.debug("Expecting files: %s" % str(expected_files))
-                msg = T('Unpacking failed, these file(s) are missing:') + ';' + u';'.join([unicoder(item) for item in missing])
-                nzo.set_unpack_info('Unpack', msg, set=setname)
-                return (1, (), ())
-        else:
-            logging.info('Skipping unrar file check due to unreliable file names or old unrar')
-
     logging.debug('UNRAR output %s', '\n'.join(lines))
     nzo.fail_msg = ''
     msg = T('Unpacked %s files/folders in %s') % (str(len(extracted)), format_time_string(time() - start))
@@ -1002,7 +975,6 @@ def par2_repair(parfile_nzf, nzo, workdir, setname, single):
                 break
 
     # Shorten just the workdir on Windows
-    workdir = short_path(workdir)
     parfile = os.path.join(workdir, parfile_nzf.filename)
 
     old_dir_content = os.listdir(workdir)
@@ -1042,7 +1014,7 @@ def par2_repair(parfile_nzf, nzo, workdir, setname, single):
             nzo.set_action_line(T('Repair'), T('Starting Repair'))
             logging.info('Scanning "%s"', parfile)
 
-            joinables, zips, rars, sevens, ts = build_filelists(workdir, None, check_rar=False)
+            joinables, zips, rars, sevens, ts = build_filelists(workdir, check_rar=False)
 
             finished, readd, pars, datafiles, used_joinables, used_par2 = PAR_Verify(parfile, parfile_nzf, nzo,
                                                                                      setname, joinables, single=single)
@@ -1177,14 +1149,6 @@ def PAR_Verify(parfile, parfile_nzf, nzo, setname, joinables, classic=False, sin
 
     logging.debug('Par2-classic/cmdline = %s', classic)
 
-    # We need to check for the bad par2cmdline that skips blocks
-    # Only if we're not doing multicore and user hasn't set options
-    if not tbb and not options:
-        par2text = run_simple([command[0], '-h'])
-        if 'No data skipping' in par2text:
-            logging.info('Detected par2cmdline version that skips blocks, adding -N parameter')
-            command.insert(2, '-N')
-
     # Append the wildcard for this set
     parfolder = os.path.split(parfile)[0]
     if single or len(globber(parfolder, setname + '*')) < 2:
@@ -1201,9 +1165,32 @@ def PAR_Verify(parfile, parfile_nzf, nzo, setname, joinables, classic=False, sin
         flist = [item for item in globber_full(parfolder, wildcard) if os.path.isfile(item)]
         command.extend(flist)
 
-    stup, need_shell, command, creationflags = build_command(command)
-    logging.debug('Starting par2: %s', command)
+    # We need to check for the bad par2cmdline that skips blocks
+    # Or the one that complains about basepath
+    # Only if we're not doing multicore
+    if not tbb:
+        par2text = run_simple([command[0], '-h'])
+        if 'No data skipping' in par2text:
+            logging.info('Detected par2cmdline version that skips blocks, adding -N parameter')
+            command.insert(2, '-N')
+        if 'Set the basepath' in par2text:
+            logging.info('Detected par2cmdline version that needs basepath, adding -B<path> parameter')
+            command.insert(2, '-B')
+            command.insert(3, parfolder)
 
+    stup, need_shell, command, creationflags = build_command(command)
+
+    # par2multicore wants to see \\.\ paths on Windows
+    # But par2cmdline doesn't support that notation, or \\?\ notation
+    # See: https://github.com/sabnzbd/sabnzbd/pull/771
+    if sabnzbd.WIN32 and (tbb or has_win_device(parfile)):
+        command = [x.replace('\\\\?\\', '\\\\.\\', 1) if x.startswith('\\\\?\\') else x for x in command]
+    elif sabnzbd.WIN32:
+        # For par2cmdline on Windows we need clipped paths
+        command = [clip_path(x) if x.startswith('\\\\?\\') else x for x in command]
+
+    # Run the external command
+    logging.debug('Starting par2: %s', command)
     lines = []
     try:
         p = subprocess.Popen(command, shell=need_shell, stdin=subprocess.PIPE,
@@ -1465,6 +1452,15 @@ def PAR_Verify(parfile, parfile_nzf, nzo, setname, joinables, classic=False, sin
                     logging.debug('PAR2 will rename "%s" to "%s"', old_name, new_name)
                     renames[new_name] = old_name
 
+                    # Show progress
+                    if verifytotal == 0 or verifynum < verifytotal:
+                        verifynum += 1
+                        nzo.set_action_line(T('Verifying'), '%02d/%02d' % (verifynum, verifytotal))
+
+            elif 'Scanning extra files' in line:
+                # Obfuscated post most likely, so reset counter to show progress
+                verifynum = 1
+
             elif 'No details available for recoverable file' in line:
                 msg = unicoder(line.strip())
                 nzo.fail_msg = msg
@@ -1545,6 +1541,18 @@ def fix_env():
     else:
         return None
 
+def userxbit(filename):
+    # Returns boolean if the x-bit for user is set on the given file
+    # This is a workaround: os.access(filename, os.X_OK) does not work on certain mounted file systems
+    # Does not work on Windows, but it is not called on Windows
+
+    # rwx rwx rwx
+    # 876 543 210      # we want bit 6 from the right, counting from 0
+    userxbit = 1<<6 # bit 6
+    rwxbits = os.stat(filename)[0] # the first element of os.stat() is "mode"
+    # do logical AND, check if it is not 0:
+    xbitset = (rwxbits & userxbit) > 0
+    return xbitset
 
 def build_command(command):
     """ Prepare list from running an external program """
@@ -1555,7 +1563,7 @@ def build_command(command):
     if not sabnzbd.WIN32:
         if command[0].endswith('.py'):
             with open(command[0], 'r') as script_file:
-                if not os.access(command[0], os.X_OK):
+                if not userxbit(command[0]):
                     # Inform user that Python scripts need x-bit and then stop
                     logging.error(T('Python script "%s" does not have execute (+x) permission set'), command[0])
                     raise IOError
@@ -1594,6 +1602,7 @@ def build_command(command):
         # scripts with spaces in the path don't work.
         if need_shell and ' ' in command[0]:
             command[0] = win32api.GetShortPathName(command[0])
+
         if need_shell:
             command = list2cmdline(command)
 
@@ -1630,7 +1639,7 @@ def par_sort(a, b):
         return 1
 
 
-def build_filelists(workdir, workdir_complete, check_rar=True):
+def build_filelists(workdir, workdir_complete=None, check_rar=True):
     """ Build filelists, if workdir_complete has files, ignore workdir.
         Optionally test content to establish RAR-ness
     """
@@ -1703,11 +1712,14 @@ def QuickCheck(set, nzo):
 
     result = False
     nzf_list = nzo.finished_files
+    renames = {}
 
     for file in md5pack:
         found = False
+        file_platform = platform_encode(file)
         for nzf in nzf_list:
-            if platform_encode(file) == nzf.filename:
+            # Do a simple filename based check
+            if file_platform == nzf.filename:
                 found = True
                 if (nzf.md5sum is not None) and nzf.md5sum == md5pack[file]:
                     logging.debug('Quick-check of file %s OK', file)
@@ -1716,9 +1728,27 @@ def QuickCheck(set, nzo):
                     logging.info('Quick-check of file %s failed!', file)
                     return False  # When any file fails, just stop
                 break
+
+            # Now lets do obfuscation check
+            if nzf.md5sum == md5pack[file]:
+                renames[file_platform] = nzf.filename
+                logging.debug('Quick-check renamed %s to %s', nzf.filename, file_platform)
+                renamer(os.path.join(nzo.downpath, nzf.filename), os.path.join(nzo.downpath, file_platform))
+                nzf.filename = file_platform
+                result = True
+                found = True
+                break
+
         if not found:
             logging.info('Cannot Quick-check missing file %s!', file)
             return False  # Missing file is failure
+
+    # Save renames
+    if renames:
+        previous = load_data(RENAMES_FILE, nzo.workpath, remove=False)
+        for name in previous or {}:
+            renames[name] = previous[name]
+        save_data(renames, RENAMES_FILE, nzo.workpath)
     return result
 
 

sabnzbd/newswrapper.py

@@ -241,10 +241,9 @@ class NNTP(object):
             if not block:
                 Thread(target=con, args=(self.sock, self.host, self.port, sslenabled, write_fds, self)).start()
             else:
-                # if blocking (server test) only wait for 4 seconds during connect until timeout
-                self.sock.settimeout(4)
+                # if blocking (server test) only wait for 15 seconds during connect until timeout
+                self.sock.settimeout(15)
                 self.sock.connect((self.host, self.port))
-
                 if sslenabled and sabnzbd.HAVE_SSL:
                     # Log SSL/TLS info
                     logging.info("%s@%s: Connected using %s (%s)",
@@ -277,16 +276,18 @@ class NNTP(object):
         # Catch certificate errors
         if type(error) == CertificateError or 'CERTIFICATE_VERIFY_FAILED' in str(error):
             error = T('Server %s uses an untrusted certificate [%s]') % (self.nw.server.host, str(error))
+            error += ' - https://sabnzbd.org/certificate-errors'
             # Prevent throwing a lot of errors or when testing server
             if error not in self.nw.server.warning and self.nw.server.id != -1:
                 logging.error(error)
 
-        msg = "Failed to connect: %s" % (str(error))
-        msg = "%s %s@%s:%s" % (msg, self.nw.thrdnum, self.host, self.port)
-        self.error_msg = msg
+        # Blocking = server-test, pass directly to display code
         if self.blocking:
-            raise socket.error(errno.ECONNREFUSED, msg)
+            raise socket.error(errno.ECONNREFUSED, str(error))
         else:
+            msg = "Failed to connect: %s" % (str(error))
+            msg = "%s %s@%s:%s" % (msg, self.nw.thrdnum, self.host, self.port)
+            self.error_msg = msg
             logging.info(msg)
             self.nw.server.warning = msg
 

sabnzbd/nzbqueue.py

@@ -47,16 +47,12 @@ from sabnzbd.encoding import platform_encode
 from sabnzbd.bpsmeter import BPSMeter
 
 
-class NzbQueue(TryList):
+class NzbQueue:
     """ Singleton NzbQueue """
     do = None
 
     def __init__(self):
-        TryList.__init__(self)
-
-        self.__top_only = False  # cfg.top_only()
-        self.__top_nzo = None
-
+        self.__top_only = cfg.top_only()
         self.__nzo_list = []
         self.__nzo_table = {}
 
@@ -323,8 +319,6 @@ class NzbQueue(TryList):
 
                 if cfg.auto_sort():
                     self.sort_by_avg_age()
-
-                self.reset_try_list()
             except:
                 logging.error(T('Error while adding %s, removing'), nzo_id)
                 logging.info("Traceback: ", exc_info=True)
@@ -397,7 +391,6 @@ class NzbQueue(TryList):
 
         # Reset try_lists
         nzo.reset_try_list()
-        self.reset_try_list()
 
         if nzo.nzo_id:
             nzo.deleted = False
@@ -560,7 +553,6 @@ class NzbQueue(TryList):
             nzo.reset_all_try_lists()
             logging.debug("Resumed nzo: %s", nzo_id)
             handled.append(nzo_id)
-        self.reset_try_list()
         return handled
 
     @synchronized(NZBQUEUE_LOCK)
@@ -759,13 +751,11 @@ class NzbQueue(TryList):
             nzf.reset_try_list()
         if nzo:
             nzo.reset_try_list()
-        self.reset_try_list()
 
     @synchronized(NZBQUEUE_LOCK)
     def reset_all_try_lists(self):
         for nzo in self.__nzo_list:
             nzo.reset_all_try_lists()
-        self.reset_try_list()
 
     @synchronized(NZBQUEUE_LOCK)
     def has_articles_for(self, server):
@@ -780,7 +770,7 @@ class NzbQueue(TryList):
                 if not cfg.propagation_delay() or nzo.priority == TOP_PRIORITY or (nzo.avg_stamp + float(cfg.propagation_delay() * 60)) < time.time():
                     # Check if category allowed
                     if nzo.server_allowed(server) or self.__top_only:
-                        return not self.server_in_try_list(server)
+                        return True
         return False
 
     @synchronized(NZBQUEUE_LOCK)
@@ -801,14 +791,14 @@ class NzbQueue(TryList):
                 # Check if past propagation delay, or forced
                 if not cfg.propagation_delay() or nzo.priority == TOP_PRIORITY or (nzo.avg_stamp + float(cfg.propagation_delay() * 60)) < time.time():
                     # Don't try to get an article if server is in try_list of nzo and category allowed by server
-                    if nzo.server_allowed(server) and (not nzo.server_in_try_list(server) or self.__top_only):
+                    if nzo.server_allowed(server) and not nzo.server_in_try_list(server):
                         article = nzo.get_article(server, servers)
                         if article:
                             return article
-
-        # No articles for this server, block server (until reset issued)
-        if not self.__top_only:
-            self.add_to_try_list(server)
+                    # Stop after first job that wasn't paused/propagating/etc
+                    # But make sure not to get stuck behind bad category
+                    if self.__top_only and not nzo.server_allowed(server):
+                        return
 
     @synchronized(NZBQUEUE_LOCK)
     def register_article(self, article, found=True):
@@ -823,9 +813,6 @@ class NzbQueue(TryList):
 
         filename = nzf.filename
 
-        if reset:
-            self.reset_try_list()
-
         if nzo.is_gone():
             logging.debug('Discarding file completion %s for deleted job', filename)
         else:

sabnzbd/nzbstuff.py

@@ -913,8 +913,7 @@ class NzbObject(TryList):
             self.deleted = True
             self.status = Status.FAILED
             nzo_id = sabnzbd.NzbQueue.do.add(self, quiet=True)
-            sabnzbd.NzbQueue.do.remove(nzo_id)
-            self.purge_data(keep_basic=True)
+            sabnzbd.NzbQueue.do.end_job(self)
             # Raise error, so it's not added
             raise TypeError
 
@@ -1261,7 +1260,6 @@ class NzbObject(TryList):
                         logging.info('Prospectively added %s repair blocks to %s', new_nzf.blocks, self.final_name)
                     # Reset all try lists
                     self.reset_all_try_lists()
-                    sabnzbd.NzbQueue.do.reset_try_list()
 
 
     def check_quality(self, req_ratio=0):
@@ -1819,6 +1817,10 @@ def scan_password(name):
 
     # Look for name/password, but make sure that '/' comes before any {{
     if slash >= 0 and slash < braces and 'password=' not in name:
+        # Is it maybe in 'name / password' notation?
+        if slash == name.find(' / ') + 1:
+            # Remove the extra space after name and before password
+            return name[:slash-1].strip('. '), name[slash + 2:]
         return name[:slash].strip('. '), name[slash + 1:]
 
     # Look for "name password=password"

sabnzbd/osxmenu.py

@@ -37,7 +37,7 @@ import sabnzbd
 import sabnzbd.cfg
 
 from sabnzbd.constants import VALID_ARCHIVES, MEBI, Status
-from sabnzbd.misc import get_filename, get_ext, diskfree, to_units
+from sabnzbd.misc import get_filename, get_ext, diskspace, to_units
 from sabnzbd.panic import launch_a_browser
 import sabnzbd.notifier as notifier
 
@@ -618,8 +618,8 @@ class SABnzbdDelegate(NSObject):
 
     def diskspaceUpdate(self):
         try:
-            self.completefolder_menu_item.setTitle_("%s%.2f GB" % (T('Complete Folder') + '\t\t\t', diskfree(sabnzbd.cfg.complete_dir.get_path())))
-            self.incompletefolder_menu_item.setTitle_("%s%.2f GB" % (T('Incomplete Folder') + '\t\t', diskfree(sabnzbd.cfg.download_dir.get_path())))
+            self.completefolder_menu_item.setTitle_("%s%.2f GB" % (T('Complete Folder') + '\t\t\t', diskspace(sabnzbd.cfg.complete_dir.get_path())[1]))
+            self.incompletefolder_menu_item.setTitle_("%s%.2f GB" % (T('Incomplete Folder') + '\t\t', diskspace(sabnzbd.cfg.download_dir.get_path())[1]))
         except:
             logging.info("[osx] diskspaceUpdate Exception %s" % (sys.exc_info()[0]))
 
@@ -727,7 +727,6 @@ class SABnzbdDelegate(NSObject):
     def restartSafeHost_(self, sender):
         sabnzbd.cfg.cherryhost.set('127.0.0.1')
         sabnzbd.cfg.cherryport.set('8080')
-        sabnzbd.cfg.https_port.set('8090')
         sabnzbd.cfg.enable_https.set(False)
         sabnzbd.config.save_config()
         self.setMenuTitle_("\n\n%s\n" % (T('Stopping...')))

sabnzbd/postproc.py

@@ -24,6 +24,7 @@ import Queue
 import logging
 import sabnzbd
 import xml.sax.saxutils
+import xml.etree.ElementTree
 import time
 import re
 
@@ -31,10 +32,10 @@ from sabnzbd.newsunpack import unpack_magic, par2_repair, external_processing, \
     sfv_check, build_filelists, rar_sort
 from threading import Thread
 from sabnzbd.misc import real_path, get_unique_path, create_dirs, move_to_path, \
-    make_script_path, short_path, long_path, clip_path, \
+    make_script_path, long_path, clip_path, \
     on_cleanup_list, renamer, remove_dir, remove_all, globber, globber_full, \
     set_permissions, cleanup_empty_directories, check_win_maxpath, fix_unix_encoding, \
-    sanitize_and_trim_path
+    sanitize_and_trim_path, sanitize_files_in_folder
 from sabnzbd.tvsort import Sorter
 from sabnzbd.constants import REPAIR_PRIORITY, TOP_PRIORITY, POSTPROC_QUEUE_FILE_NAME, \
     POSTPROC_QUEUE_VERSION, sample_match, JOB_ADMIN, Status, VERIFIED_FILE
@@ -233,7 +234,7 @@ def process_job(nzo):
     nzb_list = []
     # These need to be initialized in case of a crash
     workdir_complete = ''
-    postproc_time = 0  # @UnusedVariable -- pep8 bug?
+    postproc_time = 0
     script_log = ''
     script_line = ''
     crash_msg = ''
@@ -310,14 +311,15 @@ def process_job(nzo):
 
         # Par processing, if enabled
         if all_ok and flag_repair:
-            if not check_win_maxpath(workdir):
-                crash_msg = T('Path exceeds 260, repair by "par2" is not possible')
-                raise WindowsError
             par_error, re_add = parring(nzo, workdir)
             if re_add:
                 # Try to get more par files
                 return False
 
+        # Sanitize the resulting files
+        if sabnzbd.WIN32:
+            sanitize_files_in_folder(workdir)
+
         # Check if user allows unsafe post-processing
         if flag_repair and cfg.safe_postproc():
             all_ok = all_ok and not par_error
@@ -374,10 +376,10 @@ def process_job(nzo):
                     # set the current nzo status to "Extracting...". Used in History
                     nzo.status = Status.EXTRACTING
                     logging.info("Running unpack_magic on %s", filename)
-                    short_complete = short_path(tmp_workdir_complete)
-                    unpack_error, newfiles = unpack_magic(nzo, short_path(workdir), short_complete, flag_delete, one_folder, (), (), (), (), ())
-                    if short_complete != tmp_workdir_complete:
-                        newfiles = [f.replace(short_complete, tmp_workdir_complete) for f in newfiles]
+                    unpack_error, newfiles = unpack_magic(nzo, workdir, tmp_workdir_complete, flag_delete, one_folder, (), (), (), (), ())
+                    if sabnzbd.WIN32:
+                        # Sanitize the resulting files
+                        newfiles = sanitize_files_in_folder(tmp_workdir_complete)
                     logging.info("unpack_magic finished on %s", filename)
                 else:
                     nzo.set_unpack_info('Unpack', T('No post-processing because of failed verification'))
@@ -395,7 +397,8 @@ def process_job(nzo):
                             path = os.path.join(root, file_)
                             new_path = path.replace(workdir, tmp_workdir_complete)
                             ok, new_path = move_to_path(path, new_path)
-                            newfiles.append(new_path)
+                            if new_path:
+                                newfiles.append(new_path)
                             if not ok:
                                 nzo.set_unpack_info('Unpack', T('Failed moving %s to %s') % (unicoder(path), unicoder(new_path)))
                                 all_ok = False
@@ -464,11 +467,18 @@ def process_job(nzo):
             # Run the user script
             script_path = make_script_path(script)
             if (all_ok or not cfg.safe_postproc()) and (not nzb_list) and script_path:
+                # For windows, we use Short-Paths until 2.0.0 for compatibility
+                if sabnzbd.WIN32:
+                    import win32api
+                    workdir_complete = clip_path(workdir_complete)
+                    if len(workdir_complete) > 259:
+                        workdir_complete = win32api.GetShortPathName(workdir_complete)
+
                 # set the current nzo status to "Ext Script...". Used in History
                 nzo.status = Status.RUNNING
                 nzo.set_action_line(T('Running script'), unicoder(script))
                 nzo.set_unpack_info('Script', T('Running user script %s') % unicoder(script), unique=True)
-                script_log, script_ret = external_processing(short_path(script_path, False), short_path(workdir_complete, False), nzo.filename,
+                script_log, script_ret = external_processing(script_path, workdir_complete, nzo.filename,
                                                              dirname, cat, nzo.group, job_result,
                                                              nzo.nzo_info.get('failure', ''))
                 script_line = get_last_line(script_log)
@@ -505,11 +515,11 @@ def process_job(nzo):
                 script_ret = ''
             if len(script_log.rstrip().split('\n')) > 1:
                 nzo.set_unpack_info('Script',
-                                    u'%s%s <a href="./scriptlog?name=%s">(%s)</a>' % (script_ret, xml.sax.saxutils.escape(script_line),
+                                    u'%s%s <a href="./scriptlog?name=%s">(%s)</a>' % (script_ret, script_line,
                                     xml.sax.saxutils.escape(script_output), T('More')), unique=True)
             else:
                 # No '(more)' button needed
-                nzo.set_unpack_info('Script', u'%s%s ' % (script_ret, xml.sax.saxutils.escape(script_line)), unique=True)
+                nzo.set_unpack_info('Script', u'%s%s ' % (script_ret, script_line), unique=True)
 
 
         # Cleanup again, including NZB files
@@ -755,7 +765,7 @@ def try_rar_check(nzo, workdir, setname):
         When setname is '', all RAR files will be used, otherwise only the matching one
         If no RAR's are found, returns True
     """
-    _, _, rars, _, _ = build_filelists(short_path(workdir), None)
+    _, _, rars, _, _ = build_filelists(workdir)
 
     if setname:
         # Filter based on set
@@ -773,7 +783,14 @@ def try_rar_check(nzo, workdir, setname):
             # Set path to unrar and open the file
             # Requires de-unicode for RarFile to work!
             rarfile.UNRAR_TOOL = sabnzbd.newsunpack.RAR_COMMAND
-            zf = rarfile.RarFile(deunicode(rars[0]))
+            zf = rarfile.RarFile(rars[0])
+
+            # Skip if it's encrypted
+            if zf.needs_password():
+                msg = T('[%s] RAR-based verification failed: %s') % (unicoder(os.path.basename(rars[0])), T('Passworded'))
+                nzo.set_unpack_info('Repair', msg, set=setname)
+                return True
+
             # Will throw exception if something is wrong
             zf.testrar()
             # Success!
@@ -892,8 +909,13 @@ def one_file_or_folder(folder):
     return folder
 
 
+TAG_RE = re.compile(r'<[^>]+>')
 def get_last_line(txt):
     """ Return last non-empty line of a text, trim to 150 max """
+    # First we remove HTML code in a basic way
+    txt = TAG_RE.sub(' ', txt)
+
+    # Then we get the last line
     lines = txt.split('\n')
     n = len(lines) - 1
     while n >= 0 and not lines[n].strip('\r\t '):

sabnzbd/rss.py

@@ -307,31 +307,31 @@ class RSSQueue(object):
                 logging.debug("Running feedparser on %s", uri)
                 feed_parsed = feedparser.parse(uri.replace('feed://', 'http://'))
                 logging.debug("Done parsing %s", uri)
+
                 if not feed_parsed:
                     msg = T('Failed to retrieve RSS from %s: %s') % (uri, '?')
                     logging.info(msg)
-                    return unicoder(msg)
 
                 status = feed_parsed.get('status', 999)
                 if status in (401, 402, 403):
                     msg = T('Do not have valid authentication for feed %s') % feed
                     logging.info(msg)
-                    return unicoder(msg)
+
                 if status >= 500 and status <= 599:
                     msg = T('Server side error (server code %s); could not get %s on %s') % (status, feed, uri)
                     logging.info(msg)
-                    return unicoder(msg)
 
                 entries = feed_parsed.get('entries')
                 if 'bozo_exception' in feed_parsed and not entries:
                     msg = str(feed_parsed['bozo_exception'])
                     if 'CERTIFICATE_VERIFY_FAILED' in msg:
                         msg = T('Server %s uses an untrusted HTTPS certificate') % get_urlbase(uri)
+                        msg += ' - https://sabnzbd.org/certificate-errors'
                         logging.error(msg)
                     else:
                         msg = T('Failed to retrieve RSS from %s: %s') % (uri, xml_name(msg))
                     logging.info(msg)
-                    return unicoder(msg)
+
                 if not entries:
                     msg = T('RSS Feed %s was empty') % uri
                     logging.info(msg)
@@ -492,9 +492,15 @@ class RSSQueue(object):
 
                     if cfg.no_dupes() and self.check_duplicate(title):
                         if cfg.no_dupes() == 1:
+                            # Dupe-detection: Discard
                             logging.info("Ignoring duplicate job %s", title)
                             continue
+                        elif cfg.no_dupes() == 3:
+                            # Dupe-detection: Fail
+                            # We accept it so the Queue can send it to the History
+                            logging.info("Found duplicate job %s", title)
                         else:
+                            # Dupe-detection: Pause
                             myPrio = DUP_PRIORITY
 
                     act = download and not first
@@ -519,7 +525,7 @@ class RSSQueue(object):
             emailer.rss_mail(feed, new_downloads)
 
         remove_obsolete(jobs, newlinks)
-        return ''
+        return msg
 
     def run(self):
         """ Run all the URI's and filters """
@@ -621,12 +627,21 @@ def _HandleLink(jobs, link, title, size, age, season, episode, flag, orgcat, cat
         pp = None
 
     jobs[link] = {}
+    jobs[link]['title'] = title
+    jobs[link]['url'] = link
+    jobs[link]['cat'] = cat
+    jobs[link]['pp'] = pp
+    jobs[link]['script'] = script
+    jobs[link]['prio'] = str(priority)
     jobs[link]['order'] = order
     jobs[link]['orgcat'] = orgcat
     jobs[link]['size'] = size
     jobs[link]['age'] = age
+    jobs[link]['time'] = time.time()
+    jobs[link]['rule'] = rule
     jobs[link]['season'] = season
     jobs[link]['episode'] = episode
+
     if special_rss_site(link):
         nzbname = None
     else:
@@ -634,7 +649,6 @@ def _HandleLink(jobs, link, title, size, age, season, episode, flag, orgcat, cat
 
     if download:
         jobs[link]['status'] = 'D'
-        jobs[link]['title'] = title
         jobs[link]['time_downloaded'] = time.localtime()
         logging.info("Adding %s (%s) to queue", link, title)
         sabnzbd.add_url(link, pp=pp, script=script, cat=cat, priority=priority, nzbname=nzbname)
@@ -643,16 +657,6 @@ def _HandleLink(jobs, link, title, size, age, season, episode, flag, orgcat, cat
             jobs[link]['status'] = flag + '*'
         else:
             jobs[link]['status'] = flag
-        jobs[link]['title'] = title
-        jobs[link]['url'] = link
-        jobs[link]['cat'] = cat
-        jobs[link]['pp'] = pp
-        jobs[link]['script'] = script
-        jobs[link]['prio'] = str(priority)
-
-    jobs[link]['time'] = time.time()
-    jobs[link]['rule'] = rule
-
 
 def _get_link(uri, entry):
     """ Retrieve the post link from this entry

sabnzbd/skintext.py

@@ -445,8 +445,6 @@ SKIN_TEXT = {
     'explain-unwanted_extensions' : TT('List all unwanted extensions. For example: <b>exe</b> or <b>exe, com</b>'),
     'opt-sfv_check' : TT('Enable SFV-based checks'),
     'explain-sfv_check' : TT('Do an extra verification based on SFV files.'),
-    'opt-unpack_check' : TT('Check result of unpacking'),
-    'explain-unpack_check' : TT('Check result of unpacking (needs to be off for some file systems).'),
     'opt-script_can_fail' : TT('User script can flag job as failed'),
     'explain-script_can_fail' : TT('When the user script returns a non-zero exit code, the job will be flagged as failed.'),
     'opt-new_nzb_on_failure' : TT('On failure, try alternative NZB'),
@@ -694,7 +692,7 @@ SKIN_TEXT = {
     'button-evalFeed' : TT('Apply filters'),
     'presetSort' : TT('Presets'),
     'example' : TT('Example'),
-    'movieSort' : TT('Generic Sorting'),
+    'movieSort' : TT('Movie Sorting'),
     'opt-movieSort' : TT('Enable Movie Sorting'),
     'opt-movieExtra' : TT('Keep loose downloads in extra folders'),
     'affectedCat' : TT('Affected Categories'),

sabnzbd/urlgrabber.py

@@ -117,6 +117,7 @@ class URLGrabber(Thread):
                     logging.debug('Error "%s" trying to get the url %s', error1, url)
                     if 'certificate_verify_failed' in error1 or 'certificateerror' in error0:
                         msg = T('Server %s uses an untrusted HTTPS certificate') % ''
+                        msg += ' - https://sabnzbd.org/certificate-errors'
                         retry = False
                     elif 'nodename nor servname provided' in error1:
                         msg = T('Server name does not resolve')
@@ -219,7 +220,7 @@ class URLGrabber(Thread):
                 if '<nzb' in data and misc.get_ext(filename) != '.nzb':
                     filename += '.nzb'
 
-                # Sanatize filename first
+                # Sanitize filename first (also removing forbidden Windows-names)
                 filename = misc.sanitize_filename(filename)
 
                 # Write data to temp file

sabnzbd/utils/certgen.py

@@ -12,7 +12,7 @@ from cryptography import x509
 from cryptography.x509.oid import NameOID
 import datetime
 import os
-import struct
+from sabnzbd.getipaddress import localipv4
 
 
 # Ported from cryptography/utils.py
@@ -20,25 +20,16 @@ def int_from_bytes(data, byteorder, signed=False):
     assert byteorder == 'big'
     assert not signed
 
-    if len(data) % 4 != 0:
-        data = (b'\x00' * (4 - (len(data) % 4))) + data
-
-    result = 0
-
-    while len(data) > 0:
-        digit, = struct.unpack('>I', data[:4])
-        result = (result << 32) + digit
-        # TODO: this is quadratic in the length of data
-        data = data[4:]
-
-    return result
+    # call bytes() on data to allow the use of bytearrays
+    return int(bytes(data).encode('hex'), 16)
 
 
-# Ported from cryptography/utils.py
+# Ported from cryptography/x509/base.py
 def random_serial_number():
     return int_from_bytes(os.urandom(20), "big") >> 1
 
 
+# Ported from cryptography docs/x509/tutorial.rst (set with no encryption)
 def generate_key(key_size=2048, output_file='key.pem'):
     # Generate our key
     private_key = rsa.generate_private_key(
@@ -53,20 +44,32 @@ def generate_key(key_size=2048, output_file='key.pem'):
             encoding=serialization.Encoding.PEM,
             format=serialization.PrivateFormat.TraditionalOpenSSL,
             encryption_algorithm=serialization.NoEncryption()
+            # encryption_algorithm=serialization.BestAvailableEncryption(b"passphrase")
         ))
 
     return private_key
 
 
-def generate_local_cert(private_key, days_valid=356, output_file='cert.cert', LN='', ON='', CN=''):
+# Ported from cryptography docs/x509/tutorial.rst
+def generate_local_cert(private_key, days_valid=3560, output_file='cert.cert', LN=u'SABnzbd', ON=u'SABnzbd', CN=u'localhost'):
     # Various details about who we are. For a self-signed certificate the
     # subject and issuer are always the same.
     subject = issuer = x509.Name([
         x509.NameAttribute(NameOID.LOCALITY_NAME, LN),
         x509.NameAttribute(NameOID.ORGANIZATION_NAME, ON),
-        x509.NameAttribute(NameOID.COMMON_NAME, CN),
+        # x509.NameAttribute(NameOID.COMMON_NAME, CN),
     ])
 
+    # build SubjectAltName list since we are not using a common name
+    san_list = [
+        x509.DNSName(u"localhost"),
+        x509.DNSName(u"127.0.0.1"),
+        ]
+    # append local v4 ip (functions already has try/catch logic)
+    mylocalipv4 = localipv4()
+    if mylocalipv4:
+        san_list.append(x509.DNSName(u"" + mylocalipv4))
+
     cert = x509.CertificateBuilder().subject_name(
         subject
     ).issuer_name(
@@ -76,11 +79,12 @@ def generate_local_cert(private_key, days_valid=356, output_file='cert.cert', LN
     ).not_valid_before(
         datetime.datetime.utcnow()
     ).not_valid_after(
-        # Our certificate will be valid for 10 days
         datetime.datetime.utcnow() + datetime.timedelta(days=days_valid)
     ).serial_number(
         random_serial_number()
-    # Sign our certificate with our private key
+    ).add_extension(
+        x509.SubjectAlternativeName(san_list),
+        critical=True,
     ).sign(private_key, hashes.SHA256(), default_backend())
 
     # Write our certificate out to disk.
@@ -91,7 +95,6 @@ def generate_local_cert(private_key, days_valid=356, output_file='cert.cert', LN
 
 if __name__ == '__main__':
     print 'Making key'
-    private_key = generate_key(key_size=2048, output_file='key.pem')
+    private_key = generate_key()
     print 'Making cert'
-    cert = generate_local_cert(private_key, 356*10, 'cert.cert', u'SABnzbd', u'SABnzbd', u'SABnzbd')
-
+    cert = generate_local_cert(private_key)

sabnzbd/utils/feedparser.py

@@ -1775,11 +1775,15 @@ class _FeedParserMixin:
         # Add the dict
         if 'newznab' not in context:
             context['newznab'] = {}
-        # Add keys
-        context['newznab'][attrsD['name']] = attrsD['value']
-        # Try to get date-object
-        if attrsD['name'] == 'usenetdate':
-            context['newznab'][attrsD['name'] + '_parsed'] = _parse_date(attrsD['value'])
+        # Don't crash when it fails
+        try:
+            # Add keys
+            context['newznab'][attrsD['name']] = attrsD['value']
+            # Try to get date-object
+            if attrsD['name'] == 'usenetdate':
+                context['newznab'][attrsD['name'] + '_parsed'] = _parse_date(attrsD['value'])
+        except KeyError:
+            pass
     _start_nZEDb_attr = _start_newznab_attr
     _start_nzedb_attr = _start_newznab_attr
     _start_nntmux_attr = _start_newznab_attr

sabnzbd/utils/getperformance.py

@@ -1,14 +1,39 @@
+import platform, subprocess
+
+
 def getcpu():
-    # On Linux, let's get the CPU model name:
+    # find the CPU name (which needs a different method per OS), and return it
+    # return None if none found
+    # works on Linux, MacOS (aka OSX), Windows
+
     cputype = None
+
     try:
-        for myline in open("/proc/cpuinfo"):
-            if myline.startswith(('model name')):
-                cputype = myline[13:].rstrip()
-                break
+        if platform.system() == "Windows":
+            import _winreg as winreg	# needed on Python 2
+            key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, r"Hardware\Description\System\CentralProcessor\0")
+            cputype = winreg.QueryValueEx(key, "ProcessorNameString")[0]
+            winreg.CloseKey(key)
+
+        elif platform.system() == "Darwin":
+            cputype = subprocess.check_output(['sysctl', "-n", "machdep.cpu.brand_string"]).strip()
+
+        elif platform.system() == "Linux":
+            for myline in open("/proc/cpuinfo"):
+                if myline.startswith(('model name')):
+                    # Typical line:
+                    # model name      : Intel(R) Xeon(R) CPU           E5335  @ 2.00GHz
+                    cputype = myline.split(":", 1)[1]	# get everything after the first ":"
+                    break # we're done
+
     except:
-        # probably not on Linux
+        # An exception, maybe due to a subprocess call gone wrong
+        cputype = "Unknown"
         pass
+
+    if cputype:
+        # remove unnneeded space:
+        cputype = " ".join(cputype.split())
     return cputype
 
 

sabnzbd/utils/rarfile.py

@@ -688,9 +688,13 @@ class RarFile(object):
         return self._file_parser.needs_password()
 
     def namelist(self):
-        '''Return list of filenames in archive.'''
+        '''Return list of file and foldernames in archive.'''
         return [f.filename for f in self.infolist()]
 
+    def filelist(self):
+        '''Return list of file and foldernames in archive.'''
+        return [f.filename for f in self.infolist() if not f.isdir()]
+
     def infolist(self):
         '''Return RarInfo objects for all files/directories in archive.'''
         return self._file_parser.infolist()
@@ -818,10 +822,16 @@ class RarFile(object):
     def testrar(self):
         """Let 'unrar' test the archive.
         """
+        # Modified for SABnzbd by clipping paths
+        # and de-unicoding only here
+        from sabnzbd.misc import clip_path
+        from sabnzbd.encoding import deunicode
+        rarpath = deunicode(clip_path(self._rarfile))
+
         cmd = [UNRAR_TOOL] + list(TEST_ARGS)
         add_password_arg(cmd, self._password)
         cmd.append('--')
-        with XTempFile(self._rarfile) as rarfile:
+        with XTempFile(rarpath) as rarfile:
             cmd.append(rarfile)
             p = custom_popen(cmd)
             output = p.communicate()[0]

sabnzbd/wizard.py

@@ -209,15 +209,19 @@ class Wizard(object):
         urls = []
         for sock in socks:
             if sock:
-                if cfg.enable_https():
+                if cfg.enable_https() and cfg.https_port():
                     url = 'https://%s:%s/sabnzbd/' % (sock, cfg.https_port())
+                elif cfg.enable_https():
+                    url = 'https://%s:%s/sabnzbd/' % (sock, cfg.cherryport())
                 else:
                     url = 'http://%s:%s/sabnzbd/' % (sock, cfg.cherryport())
 
                 urls.append(url)
 
-        if cfg.enable_https():
-            access_url = 'https://%s:%s/sabnzbd/' % (access_uri, cfg.https_port())
+        if cfg.enable_https() and cfg.https_port():
+            access_url = 'https://%s:%s/sabnzbd/' % (sock, cfg.https_port())
+        elif cfg.enable_https():
+            access_url = 'https://%s:%s/sabnzbd/' % (access_uri, cfg.cherryport())
         else:
             access_url = 'http://%s:%s/sabnzbd/' % (access_uri, cfg.cherryport())