updated version string to "20.0"

Fix for GCC 7 needing option “-fpermissive” to compile it’s own file
“arm_acle.h”.

.gitignore

@@ -26,11 +26,14 @@
 # GNU Autotools
 .deps/
 config.h
+config.h.in~
 config.log
 config.status
 Makefile
 stamp-h1
 autom4te.cache/
+.dirstamp
+*.o-*
 
 # Visual Studio User-specific files
 *.suo
@@ -57,6 +60,10 @@ ipch/
 *.svclog
 *.scc
 *.sln
+.vscode/
+
+# macOS
+.DS_Store
 
 # NZBGet specific
 nzbget
@@ -64,3 +71,4 @@ code_revision.cpp
 *.temp
 *.pyc
 pytest.ini
+.cache

.travis.yml

@@ -13,6 +13,8 @@ matrix:
           packages:
             - g++-5
             - unrar
+            - p7zip-full
+            - par2
       env:
         - COMPILER=g++-5
 
@@ -24,6 +26,8 @@ matrix:
           packages:
             - g++-4.9
             - unrar
+            - p7zip-full
+            - par2
       env:
         - COMPILER=g++-4.9
 
@@ -32,6 +36,8 @@ matrix:
         apt:
           packages:
             - unrar
+            - p7zip-full
+            - par2
       env:
         - COMPILER=g++-4.8
         - CXXFLAGS="-std=c++11 -O2 -s"
@@ -46,10 +52,16 @@ matrix:
           packages:
             - clang-3.6
             - unrar
+            - p7zip-full
+            - par2
       env:
         - COMPILER=clang++-3.6
 
+install:
+  - sudo pip install -U pytest
+
 script:
   - $COMPILER --version
   - CXX=$COMPILER ./configure $CONFIGUREOPTS --enable-tests && make
   - ./nzbget --tests
+  - cd tests/functional && pytest -v

ChangeLog

@@ -1,3 +1,119 @@
+nzbget-20.0:
+  - massive performance optimisations in downloader:
+        - improved yEnc decoder;
+        - improved CRC32 calculation;
+        - processing data in one pass;
+        - SIMD implementation of decoder and CRC functions on x86 and ARM CPUs;
+          SIMD code relies on node-yencode library by Anime Tosho
+          (https://github.com/animetosho/node-yencode);
+        - overall performance improvement up to +500% on x86 and +250% on ARM
+          (better speed or less CPU usage);
+  - using glibc instead of uClibc in universal installer builds for Linux:
+        - this significantly improves performance;
+        - compatibility with Android and other systems is hopefully also improved;
+        - in universal installer glibc is used on x86 and ARM;
+        - uClibc is still used on MIPS and PPC;
+  - performance optimisations in web-interface:
+        - reduced number of requests when loading webui by combining all
+          javascript-files into one and all css-files into one;
+        - reduced load time by calling multiple API-methods simultaneously;
+        - extensive use of browser caching for static files significantly
+          reduces the amount of data transferred on webui initialisation;
+        - extensive use of browser caching for API requests reduces the amount
+          of data transferred during webui status updates, especially when
+          nzbget is in idle state and there are no changes in download queue or
+          history;
+        - avoid work in browser on status updates if there are no changes in
+          download queue or history;
+        - support for keep alive connections significantly reduces overhead for
+          establishing connections on webui status updates, especially when
+          connecting to nzbget via TLS/SSL (avoiding TLS handshakes);
+  - a number of performance optimisations for large download queue with
+    thousands of items:
+        - much faster loading of queue from disk greatly improves program start
+          up time;
+        - improved queue management for faster download speed;
+  - now offering 64 bit binaries for Windows:
+        - installer includes 32 and 64 bit nzbget binaries;
+        - when updating from older versions the 64 bit binary is installed
+          automatically, although into the old location to keep all your
+          shortcuts intact;
+        - using word "windows" instead of "win32" in the setup file name;
+  - automatic update check:
+        - new option "UpdateCheck" to check for stable or testing versions (or
+          disable);
+        - when a new version is found a notification is shown;
+        - the update check is enabled by default for stable versions;
+  - significantly improved logging performance, especially in debug builds;
+  - par-check prints par2 creator application to help identify creator app
+    issues;
+  - added support for Unix domain sockets (POSIX only);
+  - better error handling when fetching rss feeds;
+  - updated POSIX build files to newer autotools version:
+        - compatibility with newer autotools;
+        - compatibility with newer platforms such as aarch64;
+  - better username/password validation when testing connection on settings
+    page;
+  - improved rar-renamer to better handle certain cases;
+  - new option "SkipWrite" for easier speed tests;
+  - support for redirect codes 303, 307 and 308 in web-client for fetching of
+    rss feeds and nzb-files;
+  - installer for FreeBSD is now built using Clang instead of GCC; this fixes
+    incompatibility with FreeBSD 11;
+  - universal Linux installer can now be used on FreeBSD (via Linux
+    compatibility mode);
+  - updated unrar to v5.50;
+  - more robust news server connection test;
+  - enhancements in NServ:
+        - memory cache to reduce disk load during speed tests - new command line
+          switch "-m";
+        - speed control - new command line switches "-w" and "-r";
+        - show IP address of incoming connection;
+  - changed default location of log-file;
+  - better handling of broken connections;
+  - removed obsolete or less useful options "SaveQueue", "ReloadQueue",
+    "TerminateTimeout", "AccurateRate", "BrokenLog";
+  - renamed option "LogBufferSize" to "LogBuffer";
+  - passwords of failed login attempts are no longer printed to log to improve
+    security;
+  - cookies in web interface are now saved with "httpOnly" attribute to improve
+    security;
+  - titles and duplicate keys in duplicate check are now case insensitive;
+  - added LibreSSL support;
+  - web interface now has a better icon for favorites or home screen of mobile
+    devices;
+  - improved duplicate detection for obfuscated downloads having files with
+    same subjects;
+  - direct rename and direct unpack are now active by default on new
+    installations, except for slow Linux systems;
+  - added advice for letsencrypt in option descriptions;
+  - fixed incorrect renaming in rar-renamer which could cause some downloads to
+    fail;
+  - fixed race condition in queue script coordinator which could cause crashes;
+  - fixed: post-processing parameters were sometimes case sensitive causing
+    issues;
+  - fixed DNS resolving issues on Android;
+  - fixed: backup servers not used on certain article decoding errors;
+  - fixed: when direct rename was active certain downloads with damaged
+    par2-files become paused at near completion and required manual resuming;
+  - fixed: crash when flushing article cache after direct rename;
+  - fixed: deleting active par-job may crash the program;
+  - fixed: functional tests may fail on Windows due to locked files;
+  - fixed: unpack using password file doesn't work on Windows;
+  - fixed: compiler error when building using GnuTLS;
+  - fixed: Linux installer failure on android emulator;
+  - fixed: options formatted as password fields when they shouldn't;
+  - fixed: slightly off article statistics after direct rename;
+  - fixed: NServ terminated if client interrupted connection;
+  - fixed: example pp-scripts may not work properly if nzbget password or
+    username contained special characters;
+  - fix in functional tests to not rely on sizes of externally generated files;
+  - fixed: option AuthorizedIP did not work with IPv6 addresses;
+  - fixed crash on certain malformed articles;
+  - fixed crash which could happen on Windows when reloading or terminating the
+    program;
+  - fixed logging of IPv6 addresses.
+
 nzbget-19.1:
   - proper handling of changing category (and destination path) during direct
     unpack; direct unpack now gracefully aborts with cleanup; the files will
@@ -10,66 +126,67 @@ nzbget-19.1:
 
 nzbget-19.0:
   - unpack during downloading:
-    - downloaded files can now be unpacked as soon as every archive part is
-      downloaded;
-    - new option "DirectUnpack" to activate direct unpacking;
-    - direct unpack works even with obfuscated downloads; option "DirectRename"
-      (see below) must be active for that;
-    - option "ReorderFiles" (see below) should be also active for optimal file
-      download order;
-    - direct unpack works for rar-archives; 7-zip archives and simply splitted
-      files are processed by default unpack module;
-    - direct unpack obviously works only for healthy download; if download is
-      damaged the direct unpack cancels and the download is unpacked during
-        post-processing stage after files are repaired;
-    - direct unpack reduces the time needed to complete download and
+        - downloaded files can now be unpacked as soon as every archive part is
+          downloaded;
+        - new option "DirectUnpack" to activate direct unpacking;
+        - direct unpack works even with obfuscated downloads; option
+         "DirectRename" (see below) must be active for that;
+        - option "ReorderFiles" (see below) should be also active for optimal
+          file download order;
+        - direct unpack works for rar-archives; 7-zip archives and simply
+          splitted files are processed by default unpack module;
+        - direct unpack obviously works only for healthy download; if download
+          is damaged the direct unpack cancels and the download is unpacked
+          during post-processing stage after files are repaired;
+        - direct unpack reduces the time needed to complete download and
       post-processing;
-    - it also allows to start watching of video files during download (requires
-      compatible video player software);
+        - it also allows to start watching of video files during download
+          (requires compatible video player software);
   - renaming of obfuscated file names during downloading:
-    - correct file names for obfuscated downloads are now determined during
-      download stage (instead of post-processing stage);
-    - downloaded files are saved into disk directly with correct names;
-    - direct renaming uses par2-files to restore correct file names;
-    - new option "DirectRename" to activate direct renaming;
-    - new queue-event NZB_NAMED, sent after the inner files are renamed;
+        - correct file names for obfuscated downloads are now determined during
+          download stage (instead of post-processing stage);
+        - downloaded files are saved into disk directly with correct names;
+        - direct renaming uses par2-files to restore correct file names;
+        - new option "DirectRename" to activate direct renaming;
+        - new queue-event NZB_NAMED, sent after the inner files are renamed;
   - automatic reordering of files:
-    - inner files within nzb reordered to ensure download of files in archive
-      parts order;
-    - the files are reordered when nzb is added to queue;
-    - if direct renaming is active (option "DirectRename") the files are
+        - inner files within nzb reordered to ensure download of files in
+          archive parts order;
+        - the files are reordered when nzb is added to queue;
+        - if direct renaming is active (option "DirectRename") the files are
       reordered again after the correct names becomes known;
-    - new option "ReorderFiles";
-    - new command "GroupSortFiles" in api-method "editqueue";
-    - new subcommand "SF" of remote command "-E/--edit";
+        - new option "ReorderFiles";
+        - new command "GroupSortFiles" in api-method "editqueue";
+        - new subcommand "SF" of remote command "-E/--edit";
   - new option "FileNaming" to control how to name obfuscated files (before they
     get renamed by par-rename, rar-rename or direct-rename);
   - TLS certificate verification:
-    - when connecting to a news server (for downloading) or a web server (for
-      fetching of rss feeds and nzb-files) the authenticity of the server is
-      validated using server security certificate. If the check fails that means
-      the connection cannot be trusted and must be closed with an error message
-      explaining the security issue;
-    - new options "CertCheck" and "CertStore";
-    - official NZBGet packages come with activated certificate check;
-    - when updating from an older NZBGet version the option CertCheck will be
-      automatically activated when the settings is saved (switch to Settings
-      page in web-interface and click "Save all changed");
+        - when connecting to a news server (for downloading) or a web server
+         (for fetching of rss feeds and nzb-files) the authenticity of the
+          server is validated using server security certificate. If the check
+          fails that means the connection cannot be trusted and must be closed
+          with an error message explaining the security issue;
+        - new options "CertCheck" and "CertStore";
+        - official NZBGet packages come with activated certificate check;
+        - when updating from an older NZBGet version the option CertCheck will
+          be automatically activated when the settings is saved (switch to
+          Settings page in web-interface and click "Save all changed");
   - authentication via form in web-interface as alternative to HTTP
     authentication:
-    - that must help with password tools having issues with HTTP authentication
-      dialog;
-    - new option "FormAuth";
+        - that must help with password tools having issues with HTTP
+          authentication dialog;
+        - new option "FormAuth";
   - drop-downs (context menus) for priority, category and status columns:
-    - quicker changing of priority and category;
-    - easier access to actions via drop-down (context menu) in status column;
+        - quicker changing of priority and category;
+        - easier access to actions via drop-down (context menu) in status
+          column;
   - extensions scripts can now be executed from settings page:
-    - script authors define custom buttons;
-    - when clicked the script is executed in a special mode and obtain extra
-      parameters;
-    - example script "Email.py" extended with button "Send test e-mail";
+        - script authors define custom buttons;
+        - when clicked the script is executed in a special mode and obtain extra
+          parameters;
+        - example script "Email.py" extended with button "Send test e-mail";
   - on Windows NZBGet can now associate itself with nzb-files:
-    - use option in Windows installer to register NZBGet for nzb-files;
+        - use option in Windows installer to register NZBGet for nzb-files;
   - unrar shipped within Linux package is now compiled with "fallocate" option
     to improve compatibility with media players when watching videos during
     downloading and unpacking;
@@ -83,9 +200,6 @@ nzbget-19.0:
   - save changes before performing actions in history dialog;
   - proper exit code on client command success or failure.
   - added host name to all error messages regarding connection issues;
-  - improved continuos integration with Travis CI:
-    - added gcc 4.8 to test matrix;
-    - installing unrar into test system to allow unit tests requiring unrar;
   - new button "Volume Statistics" in section "News Servers" of settings page;
     shows the same volume data as in global statistics dialog;
   - new option "ServerX.Notes" for user comments on news servers;
@@ -96,9 +210,9 @@ nzbget-19.0:
   - updated unrar to v5.40;
   - clear script execution log before executing script;
   - added support for crash dumps on Windows:
-    - renamed option "DumpCore" to "CrashDump";
-    - new option "CrashTrace" to make it possible to disable default printing off
-      call stack in order to produce more relevant crash dumps;
+        - renamed option "DumpCore" to "CrashDump";
+        - new option "CrashTrace" to make it possible to disable default
+          printing off call stack in order to produce more relevant crash dumps;
   - fixed: startup scheduler tasks can be executed again;
   - fixed: "fatal" messages when compiling from sources.
   - fixed: per-nzb download statistics could be wrong if the program was

Makefile.am

@@ -58,8 +58,8 @@ nzbget_SOURCES = \
 	daemon/frontend/LoggableFrontend.h \
 	daemon/frontend/NCursesFrontend.cpp \
 	daemon/frontend/NCursesFrontend.h \
-    daemon/main/CommandLineParser.cpp \
-    daemon/main/CommandLineParser.h \
+	daemon/main/CommandLineParser.cpp \
+	daemon/main/CommandLineParser.h \
 	daemon/main/DiskService.cpp \
 	daemon/main/DiskService.h \
 	daemon/main/Maintenance.cpp \
@@ -214,6 +214,25 @@ nzbget_SOURCES += \
 	lib/par2/verificationpacket.h
 endif
 
+# Simd decoder and Crc32
+nzbget_SOURCES += \
+	lib/yencode/YEncode.h \
+	lib/yencode/SimdInit.cpp \
+	lib/yencode/SimdDecoder.cpp \
+	lib/yencode/ScalarDecoder.cpp \
+	lib/yencode/Sse2Decoder.cpp \
+	lib/yencode/Ssse3Decoder.cpp \
+	lib/yencode/PclmulCrc.cpp \
+	lib/yencode/NeonDecoder.cpp \
+	lib/yencode/AcleCrc.cpp \
+	lib/yencode/SliceCrc.cpp
+
+lib/yencode/Sse2Decoder.$(OBJEXT) : CXXFLAGS+=$(SSE2_CXXFLAGS)
+lib/yencode/Ssse3Decoder.$(OBJEXT) : CXXFLAGS+=$(SSSE3_CXXFLAGS)
+lib/yencode/PclmulCrc.$(OBJEXT) : CXXFLAGS+=$(PCLMUL_CXXFLAGS)
+lib/yencode/NeonDecoder.$(OBJEXT) : CXXFLAGS+=$(NEON_CXXFLAGS)
+lib/yencode/AcleCrc.$(OBJEXT) : CXXFLAGS+=$(ACLECRC_CXXFLAGS)
+
 AM_CPPFLAGS = \
 	-I$(srcdir)/daemon/connect \
 	-I$(srcdir)/daemon/extension \
@@ -226,7 +245,8 @@ AM_CPPFLAGS = \
 	-I$(srcdir)/daemon/remote \
 	-I$(srcdir)/daemon/util \
 	-I$(srcdir)/daemon/nserv \
-	-I$(srcdir)/lib/par2
+	-I$(srcdir)/lib/par2 \
+	-I$(srcdir)/lib/yencode
 
 if WITH_TESTS
 nzbget_SOURCES += \
@@ -283,9 +303,9 @@ windows_FILES = \
 	windows/resources/trayicon_idle.ico \
 	windows/resources/trayicon_paused.ico \
 	windows/resources/trayicon_working.ico \
-	windows/setup/nzbget-setup.nsi \
-	windows/setup/install.bmp \
-	windows/setup/uninstall.bmp
+	windows/resources/install.bmp \
+	windows/resources/uninstall.bmp \
+	windows/nzbget-setup.nsi
 
 osx_FILES = \
 	osx/App_Prefix.pch \
@@ -369,7 +389,9 @@ webui_FILES = \
 	webui/img/favicon.ico \
 	webui/img/download-anim-green-2x.png \
 	webui/img/download-anim-orange-2x.png \
-	webui/img/transmit-reload-2x.gif
+	webui/img/transmit-reload-2x.gif \
+	webui/img/favicon-256x256-opaque.png \
+	webui/img/favicon-256x256.png
 
 scripts_FILES = \
 	scripts/EMail.py \

config.h.in

@@ -13,6 +13,9 @@
 /* Define to 1 to disable gzip-support */
 #undef DISABLE_GZIP
 
+/* Define to 1 to not use libxml2, only for development purposes */
+#undef DISABLE_LIBXML2
+
 /* Define to 1 to disable par-verification and repair */
 #undef DISABLE_PARCHECK
 
@@ -155,6 +158,9 @@
 /* Define to the one symbol short name of this package. */
 #undef PACKAGE_TARNAME
 
+/* Define to the home page for this package. */
+#undef PACKAGE_URL
+
 /* Define to the version of this package. */
 #undef PACKAGE_VERSION
 
@@ -170,9 +176,10 @@
 /* Version number of package */
 #undef VERSION
 
-/* Define to 1 if your processor stores words with the most significant byte
-   first (like Motorola and SPARC, unlike Intel and VAX). */
-#undef WORDS_BIGENDIAN
+/* Enable large inode numbers on Mac OS X 10.5.  */
+#ifndef _DARWIN_USE_64_BIT_INODE
+# define _DARWIN_USE_64_BIT_INODE 1
+#endif
 
 /* Number of bits in a file offset, on hosts where this is settable. */
 #undef _FILE_OFFSET_BITS

configure.ac

@@ -1,7 +1,7 @@
 #
 #  This file is part of nzbget. See <http://nzbget.net>.
 #
-# Copyright (C) 2008-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
+# Copyright (C) 2008-2018 Andrey Prygunkov <hugbug@users.sourceforge.net>
 #
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -20,11 +20,11 @@
 #                                               -*- Autoconf -*-
 # Process this file with autoconf to produce a configure script.
 
-AC_PREREQ(2.59)
-AC_INIT(nzbget, 19.1, hugbug@users.sourceforge.net)
+AC_PREREQ(2.65)
+AC_INIT(nzbget, 20.0, hugbug@users.sourceforge.net)
 AC_CONFIG_AUX_DIR(posix)
 AC_CANONICAL_TARGET
-AM_INIT_AUTOMAKE([foreign])
+AM_INIT_AUTOMAKE([foreign subdir-objects])
 AC_CONFIG_SRCDIR([daemon/main/nzbget.cpp])
 AC_CONFIG_HEADERS([config.h])
 AM_MAINTAINER_MODE
@@ -65,8 +65,7 @@ fi
 dnl
 dnl Checks for header files.
 dnl
-AC_CHECK_HEADERS(sys/prctl.h)
-AC_CHECK_HEADERS(regex.h)
+AC_CHECK_HEADERS(sys/prctl.h regex.h endian.h getopt.h)
 
 
 dnl
@@ -146,7 +145,7 @@ if test "$FOUND" = "no"; then
 		[	char* szHost; struct hostent hinfobuf; char* strbuf; int h_errnop;
 			struct hostent* hinfo = gethostbyname_r(szHost, &hinfobuf, strbuf, 1024, &h_errnop); ],
 		AC_MSG_RESULT([[yes, and it takes 5 arguments]])
-		FOUND="yes"                      
+		FOUND="yes"
 		AC_DEFINE([HAVE_GETHOSTBYNAME_R_5], 1, [Define to 1 if gethostbyname_r takes 5 arguments]),
 		FOUND="no")
 	
@@ -196,17 +195,17 @@ AC_TRY_COMPILE([
 #include <sys/types.h>
 #include <sys/socket.h>],[
 (void)getsockopt (1, 1, 1, NULL, (size_t*)NULL)],[
-    AC_MSG_RESULT(size_t)
-    SOCKLEN_T=size_t],[
-    AC_TRY_COMPILE([
+	AC_MSG_RESULT(size_t)
+	SOCKLEN_T=size_t],[
+	AC_TRY_COMPILE([
 #include <stddef.h>
 #include <sys/types.h>
 #include <sys/socket.h>],[
 (void)getsockopt (1, 1, 1, NULL, (int*)NULL)],[
-      AC_MSG_RESULT(int)
-      SOCKLEN_T=int],[
-      AC_MSG_WARN(could not determine)
-      SOCKLEN_T=int])])])
+	AC_MSG_RESULT(int)
+	SOCKLEN_T=int],[
+	AC_MSG_WARN(could not determine)
+	SOCKLEN_T=int])])])
 AC_DEFINE_UNQUOTED(SOCKLEN_T, $SOCKLEN_T, [Determine what socket length (socklen_t) data type is])
 
 
@@ -238,26 +237,36 @@ AC_TRY_COMPILE(
 dnl 
 dnl checks for libxml2 includes and libraries.
 dnl
-AC_ARG_WITH(libxml2_includes,
-	[AS_HELP_STRING([--with-libxml2-includes=DIR], [libxml2 include directory])],
-	[CPPFLAGS="${CPPFLAGS} -I${withval}"]
-	[INCVAL="yes"],
-	[INCVAL="no"])
-AC_ARG_WITH(libxml2_libraries,
-	[AS_HELP_STRING([--with-libxml2-libraries=DIR], [libxml2 library directory])],
-	[LDFLAGS="${LDFLAGS} -L${withval}"]
-	[LIBVAL="yes"],
-	[LIBVAL="no"])
-if test "$INCVAL" = "no" -o "$LIBVAL" = "no"; then
-	PKG_CHECK_MODULES(libxml2, libxml-2.0,
-		[LIBS="${LIBS} $libxml2_LIBS"]
-		[CPPFLAGS="${CPPFLAGS} $libxml2_CFLAGS"],
+AC_MSG_CHECKING(whether to use libxml2)
+AC_ARG_ENABLE(libxml2,
+	[AS_HELP_STRING([--disable-libxml2], [do not use libxml2 (removes dependency from libxml2-library, only for development purposes)])],
+	[USELIBXML2=$enableval],
+	[USELIBXML2=yes] )
+AC_MSG_RESULT($USELIBXML2)
+if test "$USELIBXML2" = "yes"; then
+	AC_ARG_WITH(libxml2_includes,
+		[AS_HELP_STRING([--with-libxml2-includes=DIR], [libxml2 include directory])],
+		[CPPFLAGS="${CPPFLAGS} -I${withval}"]
+		[INCVAL="yes"],
+		[INCVAL="no"])
+	AC_ARG_WITH(libxml2_libraries,
+		[AS_HELP_STRING([--with-libxml2-libraries=DIR], [libxml2 library directory])],
+		[LDFLAGS="${LDFLAGS} -L${withval}"]
+		[LIBVAL="yes"],
+		[LIBVAL="no"])
+	if test "$INCVAL" = "no" -o "$LIBVAL" = "no"; then
+		PKG_CHECK_MODULES(libxml2, libxml-2.0,
+			[LIBS="${LIBS} $libxml2_LIBS"]
+			[CPPFLAGS="${CPPFLAGS} $libxml2_CFLAGS"],
+			AC_MSG_ERROR("libxml2 library not found"))
+	fi
+	AC_CHECK_HEADER(libxml/tree.h,,
+		AC_MSG_ERROR("libxml2 header files not found"))
+	AC_SEARCH_LIBS([xmlNewNode], [xml2], ,
 		AC_MSG_ERROR("libxml2 library not found"))
+else
+	AC_DEFINE([DISABLE_LIBXML2],1,[Define to 1 to not use libxml2, only for development purposes])
 fi
-AC_CHECK_HEADER(libxml/tree.h,,
-	AC_MSG_ERROR("libxml2 header files not found"))
-AC_SEARCH_LIBS([xmlNewNode], [xml2], ,
-	AC_MSG_ERROR("libxml2 library not found"))
 
 
 dnl
@@ -327,11 +336,8 @@ AC_MSG_RESULT($ENABLEPARCHECK)
 if test "$ENABLEPARCHECK" = "yes"; then
 	dnl PAR2 checks. 
 	dnl 
-	dnl Checks for header files.
-	AC_CHECK_HEADERS([endian.h] [getopt.h])
 	dnl Checks for typedefs, structures, and compiler characteristics.
 	AC_TYPE_SIZE_T
-	AC_C_BIGENDIAN
 	AC_FUNC_FSEEKO
 	dnl Checks for library functions.
 	AC_CHECK_FUNCS([stricmp])
@@ -541,6 +547,36 @@ else
 fi
 
 
+dnl
+dnl Determine if CPU supports SIMD instructions
+dnl
+AC_MSG_CHECKING(whether to use SIMD-optimized routines)
+USE_SIMD=no
+case $host_cpu in
+	i?86|x86_64)
+		SSE2_CXXFLAGS="-msse2"
+		SSSE3_CXXFLAGS="-mssse3"
+		PCLMUL_CXXFLAGS="-msse4.1 -mpclmul"
+		USE_SIMD=yes
+		;;
+	arm*)
+		NEON_CXXFLAGS="-mfpu=neon"
+		ACLECRC_CXXFLAGS="-march=armv8-a+crc -fpermissive"
+		USE_SIMD=yes
+		;;
+	aarch64)
+		ACLECRC_CXXFLAGS="-march=armv8-a+crc -fpermissive"
+		USE_SIMD=yes
+		;;
+esac
+AC_MSG_RESULT($USE_SIMD)
+AC_SUBST([SSE2_CXXFLAGS])
+AC_SUBST([SSSE3_CXXFLAGS])
+AC_SUBST([PCLMUL_CXXFLAGS])
+AC_SUBST([NEON_CXXFLAGS])
+AC_SUBST([ACLECRC_CXXFLAGS])
+
+
 dnl 
 dnl Some Linux systems require an empty signal handler for SIGCHLD
 dnl in order for exit codes to be correctly delivered to parent process.
@@ -599,11 +635,10 @@ dnl
 dnl variadic macros
 dnl 
 AC_MSG_CHECKING(for variadic macros)
-AC_COMPILE_IFELSE([
-	#define macro(...)   macrofunc(__VA_ARGS__)
-	int macrofunc(int a, int b) { return a + b; }
-	int test() { return macro(1, 2); }
-	],
+AC_TRY_COMPILE(
+	[ #define macro(...)   macrofunc(__VA_ARGS__) ]
+	[ int macrofunc(int a, int b) { return a + b; } ],
+	[ int a=macro(1, 2); ],
 	AC_MSG_RESULT([yes])
 	AC_DEFINE([HAVE_VARIADIC_MACROS], 1, Define to 1 if variadic macros are supported),
 	AC_MSG_RESULT([no]))

daemon/connect/Connection.cpp

@@ -2,7 +2,7 @@
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
  *  Copyright (C) 2004 Sven Henkel <sidddy@users.sourceforge.net>
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -22,6 +22,7 @@
 #include "nzbget.h"
 #include "Connection.h"
 #include "Log.h"
+#include "FileSystem.h"
 
 static const int CONNECTION_READBUFFER_SIZE = 1024;
 #ifndef HAVE_GETADDRINFO
@@ -79,6 +80,10 @@ void closesocket_gracefully(SOCKET socket)
 	closesocket(socket);
 }
 
+#ifdef __linux__
+CString ResolveAndroidHost(const char* host);
+#endif
+
 void Connection::Init()
 {
 	debug("Initializing global connection data");
@@ -131,6 +136,7 @@ Connection::Connection(SOCKET socket, bool tls)
 	m_tls = tls;
 	m_status = csConnected;
 	m_socket = socket;
+	m_host = GetRemoteAddr();
 	m_bufAvail = 0;
 	m_timeout = 60;
 	m_suppressErrors = true;
@@ -209,91 +215,122 @@ bool Connection::Bind()
 		return true;
 	}
 
-#ifdef HAVE_GETADDRINFO
-	struct addrinfo addr_hints, *addr_list, *addr;
-
-	memset(&addr_hints, 0, sizeof(addr_hints));
-	addr_hints.ai_family = m_ipVersion == ipV4 ? AF_INET : m_ipVersion == ipV6 ? AF_INET6 : AF_UNSPEC;
-	addr_hints.ai_socktype = SOCK_STREAM,
-	addr_hints.ai_flags = AI_PASSIVE;    // For wildcard IP address
-
-	BString<100> portStr("%d", m_port);
-
-	int res = getaddrinfo(m_host, portStr, &addr_hints, &addr_list);
-	if (res != 0)
-	{
-		ReportError("Could not resolve hostname %s", m_host, true
 #ifndef WIN32
-			, res != EAI_SYSTEM ? res : 0
-			, res != EAI_SYSTEM ? gai_strerror(res) : nullptr
-#endif
-			);
-		return false;
-	}
-
-	m_broken = false;
-	m_socket = INVALID_SOCKET;
-	for (addr = addr_list; addr != nullptr; addr = addr->ai_next)
+	if (m_host && m_host[0] == '/')
 	{
-		m_socket = socket(addr->ai_family, addr->ai_socktype, addr->ai_protocol);
-#ifdef WIN32
-		SetHandleInformation((HANDLE)m_socket, HANDLE_FLAG_INHERIT, 0);
-#endif
-		if (m_socket != INVALID_SOCKET)
+		struct sockaddr_un addr;
+		addr.sun_family = AF_UNIX;
+		if (strlen(m_host) >= sizeof(addr.sun_path))
+		{
+			ReportError("Binding socket failed for %s: name too long\n", m_host, false);
+			return false;
+		}
+		strcpy(addr.sun_path, m_host);
+
+		m_socket = socket(PF_UNIX, SOCK_STREAM, 0);
+		if (m_socket == INVALID_SOCKET)
+		{
+			ReportError("Socket creation failed for %s", m_host, true);
+			return false;
+		}
+
+		unlink(m_host);
+
+		if (bind(m_socket, (struct sockaddr *)&addr, sizeof(addr)) == -1)
 		{
-			int opt = 1;
-			setsockopt(m_socket, SOL_SOCKET, SO_REUSEADDR, (char*)&opt, sizeof(opt));
-			res = bind(m_socket, addr->ai_addr, addr->ai_addrlen);
-			if (res != -1)
-			{
-				// Connection established
-				break;
-			}
 			// Connection failed
 			closesocket(m_socket);
 			m_socket = INVALID_SOCKET;
 		}
 	}
+	else
+#endif
+	{
+#ifdef HAVE_GETADDRINFO
+		struct addrinfo addr_hints, *addr_list, *addr;
 
-	freeaddrinfo(addr_list);
+		memset(&addr_hints, 0, sizeof(addr_hints));
+		addr_hints.ai_family = m_ipVersion == ipV4 ? AF_INET : m_ipVersion == ipV6 ? AF_INET6 : AF_UNSPEC;
+		addr_hints.ai_socktype = SOCK_STREAM;
+		addr_hints.ai_flags = AI_PASSIVE;    // For wildcard IP address
+
+		BString<100> portStr("%d", m_port);
+
+		int res = getaddrinfo(m_host, portStr, &addr_hints, &addr_list);
+		if (res != 0)
+		{
+			ReportError("Could not resolve hostname %s", m_host, true
+#ifndef WIN32
+				, res != EAI_SYSTEM ? res : 0
+				, res != EAI_SYSTEM ? gai_strerror(res) : nullptr
+#endif
+				);
+			return false;
+		}
+
+		m_socket = INVALID_SOCKET;
+		for (addr = addr_list; addr != nullptr; addr = addr->ai_next)
+		{
+			m_socket = socket(addr->ai_family, addr->ai_socktype, addr->ai_protocol);
+#ifdef WIN32
+			SetHandleInformation((HANDLE)m_socket, HANDLE_FLAG_INHERIT, 0);
+#endif
+			if (m_socket != INVALID_SOCKET)
+			{
+				int opt = 1;
+				setsockopt(m_socket, SOL_SOCKET, SO_REUSEADDR, (char*)&opt, sizeof(opt));
+				res = bind(m_socket, addr->ai_addr, addr->ai_addrlen);
+				if (res != -1)
+				{
+					// Connection established
+					break;
+				}
+				// Connection failed
+				closesocket(m_socket);
+				m_socket = INVALID_SOCKET;
+			}
+		}
+
+		freeaddrinfo(addr_list);
 
 #else
 
-	struct sockaddr_in	sSocketAddress;
-	memset(&sSocketAddress, 0, sizeof(sSocketAddress));
-	sSocketAddress.sin_family = AF_INET;
-	if (!m_host || strlen(m_host) == 0)
-	{
-		sSocketAddress.sin_addr.s_addr = htonl(INADDR_ANY);
-	}
-	else
-	{
-		sSocketAddress.sin_addr.s_addr = ResolveHostAddr(m_host);
-		if (sSocketAddress.sin_addr.s_addr == INADDR_NONE)
+		struct sockaddr_in	sSocketAddress;
+		memset(&sSocketAddress, 0, sizeof(sSocketAddress));
+		sSocketAddress.sin_family = AF_INET;
+		if (!m_host || strlen(m_host) == 0)
 		{
+			sSocketAddress.sin_addr.s_addr = htonl(INADDR_ANY);
+		}
+		else
+		{
+			sSocketAddress.sin_addr.s_addr = ResolveHostAddr(m_host);
+			if (sSocketAddress.sin_addr.s_addr == INADDR_NONE)
+			{
+				return false;
+			}
+		}
+		sSocketAddress.sin_port = htons(m_port);
+
+		m_socket = socket(PF_INET, SOCK_STREAM, 0);
+		if (m_socket == INVALID_SOCKET)
+		{
+			ReportError("Socket creation failed for %s", m_host, true);
 			return false;
 		}
-	}
-	sSocketAddress.sin_port = htons(m_port);
 
-	m_socket = socket(PF_INET, SOCK_STREAM, 0);
-	if (m_socket == INVALID_SOCKET)
-	{
-		ReportError("Socket creation failed for %s", m_host, true);
-		return false;
-	}
+		int opt = 1;
+		setsockopt(m_socket, SOL_SOCKET, SO_REUSEADDR, (char*)&opt, sizeof(opt));
 
-	int opt = 1;
-	setsockopt(m_socket, SOL_SOCKET, SO_REUSEADDR, (char*)&opt, sizeof(opt));
-
-	int res = bind(m_socket, (struct sockaddr *) &sSocketAddress, sizeof(sSocketAddress));
-	if (res == -1)
-	{
-		// Connection failed
-		closesocket(m_socket);
-		m_socket = INVALID_SOCKET;
-	}
+		int res = bind(m_socket, (struct sockaddr *) &sSocketAddress, sizeof(sSocketAddress));
+		if (res == -1)
+		{
+			// Connection failed
+			closesocket(m_socket);
+			m_socket = INVALID_SOCKET;
+		}
 #endif
+	}
 
 	if (m_socket == INVALID_SOCKET)
 	{
@@ -324,7 +361,7 @@ int Connection::WriteLine(const char* buffer)
 	int res = send(m_socket, buffer, strlen(buffer), 0);
 	if (res <= 0)
 	{
-		m_broken = true;
+		m_status = csBroken;
 	}
 
 	return res;
@@ -345,7 +382,7 @@ bool Connection::Send(const char* buffer, int size)
 		int res = send(m_socket, buffer + bytesSent, size-bytesSent, 0);
 		if (res <= 0)
 		{
-			m_broken = true;
+			m_status = csBroken;
 			return false;
 		}
 		bytesSent += res;
@@ -374,13 +411,17 @@ char* Connection::ReadLine(char* buffer, int size, int* bytesReadOut)
 			if (bufAvail < 0)
 			{
 				ReportError("Could not receive data on socket from %s", m_host, true);
-				m_broken = true;
+				m_status = csBroken;
 				break;
 			}
 			else if (bufAvail == 0)
 			{
 				break;
 			}
+			else
+			{
+				m_totalBytesRead += bufAvail;
+			}
 			bufPtr = m_readBuf;
 			m_readBuf[bufAvail] = '\0';
 		}
@@ -423,8 +464,6 @@ char* Connection::ReadLine(char* buffer, int size, int* bytesReadOut)
 		*bytesReadOut = bytesRead;
 	}
 
-	m_totalBytesRead += bytesRead;
-
 	if (inpBuffer == buffer)
 	{
 		return nullptr;
@@ -452,12 +491,14 @@ std::unique_ptr<Connection> Connection::Accept()
 		return nullptr;
 	}
 
+	InitSocketOpts(socket);
+
 	return std::make_unique<Connection>(socket, m_tls);
 }
 
 int Connection::TryRecv(char* buffer, int size)
 {
-	debug("Receiving data");
+	//debug("Receiving data");
 
 	memset(buffer, 0, size);
 
@@ -467,13 +508,17 @@ int Connection::TryRecv(char* buffer, int size)
 	{
 		ReportError("Could not receive data on socket from %s", m_host, true);
 	}
+	else
+	{
+		m_totalBytesRead += received;
+	}
 
 	return received;
 }
 
 bool Connection::Recv(char * buffer, int size)
 {
-	debug("Receiving data (full buffer)");
+	//debug("Receiving data (full buffer)");
 
 	memset(buffer, 0, size);
 
@@ -502,6 +547,7 @@ bool Connection::Recv(char * buffer, int size)
 		}
 		bufPtr += received;
 		NeedBytes -= received;
+		m_totalBytesRead += received;
 	}
 	return true;
 }
@@ -511,113 +557,156 @@ bool Connection::DoConnect()
 	debug("Do connecting");
 
 	m_socket = INVALID_SOCKET;
-	m_broken = false;
 
-#ifdef HAVE_GETADDRINFO
-	struct addrinfo addr_hints, *addr_list, *addr;
-
-	memset(&addr_hints, 0, sizeof(addr_hints));
-	addr_hints.ai_family = m_ipVersion == ipV4 ? AF_INET : m_ipVersion == ipV6 ? AF_INET6 : AF_UNSPEC;
-	addr_hints.ai_socktype = SOCK_STREAM;
-
-	BString<100> portStr("%d", m_port);
-
-	int res = getaddrinfo(m_host, portStr, &addr_hints, &addr_list);
-	if (res != 0)
-	{
-		ReportError("Could not resolve hostname %s", m_host, true
 #ifndef WIN32
-					, res != EAI_SYSTEM ? res : 0
-					, res != EAI_SYSTEM ? gai_strerror(res) : nullptr
-#endif
-					);
-		return false;
-	}
-
-	std::vector<SockAddr> triedAddr;
-	bool connected = false;
-
-	for (addr = addr_list; addr != nullptr; addr = addr->ai_next)
+	if (m_host && m_host[0] == '/')
 	{
-		// don't try the same combinations of ai_family, ai_socktype, ai_protocol multiple times
-		SockAddr sa = { addr->ai_family, addr->ai_socktype, addr->ai_protocol };
-		if (std::find(triedAddr.begin(), triedAddr.end(), sa) != triedAddr.end())
+		struct sockaddr_un addr;
+		addr.sun_family = AF_UNIX;
+		if (strlen(m_host) >= sizeof(addr.sun_path))
 		{
-			continue;
+			ReportError("Connection to %s failed: name too long\n", m_host, false);
+			return false;
 		}
-		triedAddr.push_back(sa);
+		strcpy(addr.sun_path, m_host);
 
-		if (m_socket != INVALID_SOCKET)
-		{
-			closesocket(m_socket);
-		}
-
-		m_socket = socket(addr->ai_family, addr->ai_socktype, addr->ai_protocol);
-#ifdef WIN32
-		SetHandleInformation((HANDLE)m_socket, HANDLE_FLAG_INHERIT, 0);
-#endif
+		m_socket = socket(PF_UNIX, SOCK_STREAM, 0);
 		if (m_socket == INVALID_SOCKET)
 		{
-			// try another addr/family/protocol
-			continue;
+			ReportError("Socket creation failed for %s", m_host, true);
+			return false;
 		}
 
-		if (ConnectWithTimeout(addr->ai_addr, addr->ai_addrlen))
+		if (!ConnectWithTimeout(&addr, sizeof(addr)))
 		{
-			// Connection established
-			connected = true;
-			break;
+			ReportError("Connection to %s failed", m_host, true);
+			closesocket(m_socket);
+			m_socket = INVALID_SOCKET;
+			return false;
 		}
 	}
-
-	if (m_socket == INVALID_SOCKET && addr_list)
+	else
+#endif
 	{
-		ReportError("Socket creation failed for %s", m_host, true);
-	}
+#ifdef HAVE_GETADDRINFO
+		struct addrinfo addr_hints, *addr_list, *addr;
 
-	if (!connected && m_socket != INVALID_SOCKET)
-	{
-		ReportError("Connection to %s failed", m_host, true);
-		closesocket(m_socket);
-		m_socket = INVALID_SOCKET;
-	}
+		memset(&addr_hints, 0, sizeof(addr_hints));
+		addr_hints.ai_family = m_ipVersion == ipV4 ? AF_INET : m_ipVersion == ipV6 ? AF_INET6 : AF_UNSPEC;
+		addr_hints.ai_socktype = SOCK_STREAM;
 
-	freeaddrinfo(addr_list);
+		BString<100> portStr("%d", m_port);
 
-	if (m_socket == INVALID_SOCKET)
-	{
-		return false;
-	}
+		int res = getaddrinfo(m_host, portStr, &addr_hints, &addr_list);
+		debug("getaddrinfo for %s: %i", *m_host, res);
+
+#ifdef __linux__
+		if (res != 0)
+		{
+			CString resolvedHost = ResolveAndroidHost(m_host);
+			if (!resolvedHost.Empty())
+			{
+				res = getaddrinfo(resolvedHost, portStr, &addr_hints, &addr_list);
+			}
+		}
+#endif
+
+		if (res != 0)
+		{
+			ReportError("Could not resolve hostname %s", m_host, true
+#ifndef WIN32
+						, res != EAI_SYSTEM ? res : 0
+						, res != EAI_SYSTEM ? gai_strerror(res) : nullptr
+#endif
+						);
+			return false;
+		}
+
+		std::vector<SockAddr> triedAddr;
+		bool connected = false;
+
+		for (addr = addr_list; addr != nullptr; addr = addr->ai_next)
+		{
+			// don't try the same combinations of ai_family, ai_socktype, ai_protocol multiple times
+			SockAddr sa = { addr->ai_family, addr->ai_socktype, addr->ai_protocol };
+			if (std::find(triedAddr.begin(), triedAddr.end(), sa) != triedAddr.end())
+			{
+				continue;
+			}
+			triedAddr.push_back(sa);
+
+			if (m_socket != INVALID_SOCKET)
+			{
+				closesocket(m_socket);
+			}
+
+			m_socket = socket(addr->ai_family, addr->ai_socktype, addr->ai_protocol);
+#ifdef WIN32
+			SetHandleInformation((HANDLE)m_socket, HANDLE_FLAG_INHERIT, 0);
+#endif
+			if (m_socket == INVALID_SOCKET)
+			{
+				// try another addr/family/protocol
+				continue;
+			}
+
+			if (ConnectWithTimeout(addr->ai_addr, addr->ai_addrlen))
+			{
+				// Connection established
+				connected = true;
+				break;
+			}
+		}
+
+		if (m_socket == INVALID_SOCKET && addr_list)
+		{
+			ReportError("Socket creation failed for %s", m_host, true);
+		}
+
+		if (!connected && m_socket != INVALID_SOCKET)
+		{
+			ReportError("Connection to %s failed", m_host, true);
+			closesocket(m_socket);
+			m_socket = INVALID_SOCKET;
+		}
+
+		freeaddrinfo(addr_list);
+
+		if (m_socket == INVALID_SOCKET)
+		{
+			return false;
+		}
 
 #else
 
-	struct sockaddr_in	sSocketAddress;
-	memset(&sSocketAddress, 0, sizeof(sSocketAddress));
-	sSocketAddress.sin_family = AF_INET;
-	sSocketAddress.sin_port = htons(m_port);
-	sSocketAddress.sin_addr.s_addr = ResolveHostAddr(m_host);
-	if (sSocketAddress.sin_addr.s_addr == INADDR_NONE)
-	{
-		return false;
-	}
+		struct sockaddr_in	sSocketAddress;
+		memset(&sSocketAddress, 0, sizeof(sSocketAddress));
+		sSocketAddress.sin_family = AF_INET;
+		sSocketAddress.sin_port = htons(m_port);
+		sSocketAddress.sin_addr.s_addr = ResolveHostAddr(m_host);
+		if (sSocketAddress.sin_addr.s_addr == INADDR_NONE)
+		{
+			return false;
+		}
 
-	m_socket = socket(PF_INET, SOCK_STREAM, 0);
-	if (m_socket == INVALID_SOCKET)
-	{
-		ReportError("Socket creation failed for %s", m_host, true);
-		return false;
-	}
+		m_socket = socket(PF_INET, SOCK_STREAM, 0);
+		if (m_socket == INVALID_SOCKET)
+		{
+			ReportError("Socket creation failed for %s", m_host, true);
+			return false;
+		}
 
-	if (!ConnectWithTimeout(&sSocketAddress, sizeof(sSocketAddress)))
-	{
-		ReportError("Connection to %s failed", m_host, true);
-		closesocket(m_socket);
-		m_socket = INVALID_SOCKET;
-		return false;
-	}
+		if (!ConnectWithTimeout(&sSocketAddress, sizeof(sSocketAddress)))
+		{
+			ReportError("Connection to %s failed", m_host, true);
+			closesocket(m_socket);
+			m_socket = INVALID_SOCKET;
+			return false;
+		}
 #endif
+	}
 
-	if (!InitSocketOpts())
+	if (!InitSocketOpts(m_socket))
 	{
 		return false;
 	}
@@ -632,7 +721,7 @@ bool Connection::DoConnect()
 	return true;
 }
 
-bool Connection::InitSocketOpts()
+bool Connection::InitSocketOpts(SOCKET socket)
 {
 	char* optbuf = nullptr;
 	int optsize = 0;
@@ -647,13 +736,13 @@ bool Connection::InitSocketOpts()
 	optbuf = (char*)&TimeVal;
 	optsize = sizeof(TimeVal);
 #endif
-	int err = setsockopt(m_socket, SOL_SOCKET, SO_RCVTIMEO, optbuf, optsize);
+	int err = setsockopt(socket, SOL_SOCKET, SO_RCVTIMEO, optbuf, optsize);
 	if (err != 0)
 	{
 		ReportError("Socket initialization failed for %s", m_host, true);
 		return false;
 	}
-	err = setsockopt(m_socket, SOL_SOCKET, SO_SNDTIMEO, optbuf, optsize);
+	err = setsockopt(socket, SOL_SOCKET, SO_SNDTIMEO, optbuf, optsize);
 	if (err != 0)
 	{
 		ReportError("Socket initialization failed for %s", m_host, true);
@@ -719,7 +808,7 @@ bool Connection::ConnectWithTimeout(void* address, int address_len)
 	//connect succeeded right away?
 	if (ret != 0)
 	{
-		ret = select(m_socket + 1, &rset, &wset, nullptr, m_timeout ? &ts : nullptr);
+		ret = select((int)m_socket + 1, &rset, &wset, nullptr, m_timeout ? &ts : nullptr);
 		//we are waiting for connect to complete now
 		if (ret < 0)
 		{
@@ -780,6 +869,15 @@ bool Connection::DoDisconnect()
 	{
 #ifndef DISABLE_TLS
 		CloseTls();
+#endif
+#ifndef WIN32
+		int is_listening;
+		socklen_t len = sizeof(is_listening);
+		if (m_host && m_host[0] == '/'
+			&& getsockopt(m_socket, SOL_SOCKET, SO_ACCEPTCONN, &is_listening, &len) == 0 && is_listening)
+		{
+			unlink(m_host);
+		}
 #endif
 		if (m_gracefull)
 		{
@@ -809,10 +907,17 @@ void Connection::Cancel()
 	if (m_socket != INVALID_SOCKET)
 	{
 		m_status = csCancelled;
-		int r = shutdown(m_socket, SHUT_RDWR);
-		if (r == -1)
+
+		if (m_forceClose)
 		{
-			ReportError("Could not shutdown connection for %s", m_host, true);
+			SOCKET socket = m_socket;
+			m_socket = INVALID_SOCKET;
+			shutdown(socket, SHUT_RDWR);
+			closesocket(socket);
+		}
+		else
+		{
+			shutdown(m_socket, SHUT_RDWR);
 		}
 	}
 }
@@ -987,16 +1092,47 @@ in_addr_t Connection::ResolveHostAddr(const char* host)
 
 const char* Connection::GetRemoteAddr()
 {
-	struct sockaddr_in PeerName;
-	int peerNameLength = sizeof(PeerName);
-	if (getpeername(m_socket, (struct sockaddr*)&PeerName, (SOCKLEN_T*) &peerNameLength) >= 0)
+#ifndef WIN32
+	if (m_host && m_host[0] == '/')
+	{
+		return "-";
+	}
+#endif
+
+	m_remoteAddr.Clear();
+
+	char peerName[1024];
+	int peerNameLength = sizeof(peerName);
+	if (getpeername(m_socket, (sockaddr*)&peerName, (SOCKLEN_T*)&peerNameLength) >= 0)
 	{
 #ifdef WIN32
-		m_remoteAddr = inet_ntoa(PeerName.sin_addr);
+		HMODULE module = LoadLibrary("ws2_32.dll");
+		if (module)
+		{
+			using inet_ntop_t = PCTSTR WSAAPI (INT Family, PVOID pAddr, PTSTR pStringBuf, size_t StringBufSize);
+			inet_ntop_t* inet_ntop = (inet_ntop_t*)GetProcAddress(module, "inet_ntop");
+			if (inet_ntop)
+			{
+				inet_ntop(((sockaddr_in*)&peerName)->sin_family,
+					((sockaddr_in*)&peerName)->sin_family == AF_INET6 ?
+					(void*)&((sockaddr_in6*)&peerName)->sin6_addr :
+					(void*)&((sockaddr_in*)&peerName)->sin_addr,
+					m_remoteAddr, m_remoteAddr.Capacity());
+			}
+			FreeLibrary(module);
+		}
+		if (m_remoteAddr.Empty())
+		{
+			m_remoteAddr = inet_ntoa(((sockaddr_in*)&peerName)->sin_addr);
+		}
 #else
-		inet_ntop(AF_INET, &PeerName.sin_addr, m_remoteAddr, m_remoteAddr.Capacity());
-		m_remoteAddr[m_remoteAddr.Capacity() - 1] = '\0';
+		inet_ntop(((sockaddr_in*)&peerName)->sin_family,
+			((sockaddr_in*)&peerName)->sin_family == AF_INET6 ?
+			(void*)&((sockaddr_in6*)&peerName)->sin6_addr :
+			(void*)&((sockaddr_in*)&peerName)->sin_addr,
+			m_remoteAddr, m_remoteAddr.Capacity());
 #endif
+		m_remoteAddr[m_remoteAddr.Capacity() - 1] = '\0';
 	}
 
 	return m_remoteAddr;
@@ -1008,3 +1144,210 @@ int Connection::FetchTotalBytesRead()
 	m_totalBytesRead = 0;
 	return total;
 }
+
+
+#ifdef __linux__
+
+//******************************************************************************
+// Android resolver proxy from AOSP (reworked):
+// https://github.com/aosp-mirror/platform_bionic/blob/6c1d23f059986e4ee6f52c44a4944089aae9181d/libc/dns/net/gethnamaddr.c
+
+#define	MAXALIASES	35
+#define	MAXADDRS	35
+#define ALIGNBYTES (sizeof(uintptr_t) - 1)
+#define ALIGN(p) (((uintptr_t)(p) + ALIGNBYTES) &~ ALIGNBYTES)
+
+// This should be synchronized to ResponseCode.h
+static const int DnsProxyQueryResult = 222;
+
+FILE* android_open_proxy()
+{
+	const char* cache_mode = getenv("ANDROID_DNS_MODE");
+	bool use_proxy = (cache_mode == nullptr || strcmp(cache_mode, "local") != 0);
+	if (!use_proxy) {
+		return nullptr;
+	}
+
+	int s = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
+	if (s == -1) {
+		return nullptr;
+	}
+
+	const int one = 1;
+	setsockopt(s, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one));
+
+	struct sockaddr_un proxy_addr;
+	memset(&proxy_addr, 0, sizeof(proxy_addr));
+	proxy_addr.sun_family = AF_UNIX;
+	strncpy(proxy_addr.sun_path, "/dev/socket/dnsproxyd", sizeof(proxy_addr.sun_path));
+
+	if (connect(s, (const struct sockaddr*) &proxy_addr, sizeof(proxy_addr)) != 0) {
+		close(s);
+		return nullptr;
+	}
+
+	return fdopen(s, "r+");
+}
+
+static struct hostent * android_read_hostent(FILE* proxy, struct hostent* hp, char* hbuf, size_t hbuflen)
+{
+	uint32_t size;
+	char buf[4];
+	if (fread(buf, 1, sizeof(buf), proxy) != sizeof(buf)) return nullptr;
+
+	// This is reading serialized data from system/netd/server/DnsProxyListener.cpp
+	// and changes here need to be matched there.
+	int result_code = strtol(buf, nullptr, 10);
+	if (result_code != DnsProxyQueryResult) {
+		fread(&size, 1, sizeof(size), proxy);
+		return nullptr;
+	}
+
+	if (fread(&size, 1, sizeof(size), proxy) != sizeof(size)) return nullptr;
+	size = ntohl(size);
+
+	memset(hp, 0, sizeof(*hp));
+	char *ptr = hbuf;
+	char *hbuf_end = hbuf + hbuflen;
+
+	if (ptr + size > hbuf_end) {
+		return nullptr;
+	}
+	if (fread(ptr, 1, size, proxy) != size) return nullptr;
+	hp->h_name = ptr;
+	ptr += size;
+
+	char *aliases_ptrs[MAXALIASES];
+	char **aliases = &aliases_ptrs[0];
+
+	while (1) {
+		if (fread(&size, 1, sizeof(size), proxy) != sizeof(size)) return nullptr;
+		size = ntohl(size);
+
+		if (size == 0) {
+			*aliases = nullptr;
+			break;
+		}
+		if (ptr + size > hbuf_end) {
+			return nullptr;
+		}
+		if (fread(ptr, 1, size, proxy) != size) return nullptr;
+		if (aliases < &aliases_ptrs[MAXALIASES - 1]) {
+		  *aliases++ = ptr;
+		}
+		ptr += size;
+	}
+
+	// Fix alignment after variable-length data.
+	ptr = (char*)ALIGN(ptr);
+
+	int aliases_len = ((int)(aliases - aliases_ptrs) + 1) * sizeof(*hp->h_aliases);
+	if (ptr + aliases_len > hbuf_end) {
+		return nullptr;
+	}
+	hp->h_aliases = (char**)ptr;
+	memcpy(ptr, aliases_ptrs, aliases_len);
+	ptr += aliases_len;
+
+	if (fread(&size, 1, sizeof(size), proxy) != sizeof(size)) return nullptr;
+	hp->h_addrtype = ntohl(size);
+
+	if (fread(&size, 1, sizeof(size), proxy) != sizeof(size)) return nullptr;
+	hp->h_length = ntohl(size);
+
+	char *addr_ptrs[MAXADDRS];
+	char **addr_p = &addr_ptrs[0];
+
+	while (1) {
+		if (fread(&size, 1, sizeof(size), proxy) != sizeof(size)) return nullptr;
+		size = ntohl(size);
+		if (size == 0) {
+			*addr_p = nullptr;
+			break;
+		}
+		if (ptr + size > hbuf_end) {
+			return nullptr;
+		}
+		if (fread(ptr, 1, size, proxy) != size) return nullptr;
+		if (addr_p < &addr_ptrs[MAXADDRS - 1]) {
+		  *addr_p++ = ptr;
+		}
+		ptr += size;
+	}
+
+	// Fix alignment after variable-length data.
+	ptr = (char*)ALIGN(ptr);
+
+	int addrs_len = ((int)(addr_p - addr_ptrs) + 1) * sizeof(*hp->h_addr_list);
+	if (ptr + addrs_len > hbuf_end) {
+		return nullptr;
+	}
+	hp->h_addr_list = (char**)ptr;
+	memcpy(ptr, addr_ptrs, addrs_len);
+	return hp;
+}
+
+// very similar in proxy-ness to android_getaddrinfo_proxy
+static struct hostent * android_gethostbyname_internal(const char *name, int af,
+	struct hostent *hp, char *hbuf, size_t hbuflen)
+{
+	FILE* proxy = android_open_proxy();
+	if (proxy == nullptr) {
+		return nullptr;
+	}
+
+	// This is writing to system/netd/server/DnsProxyListener.cpp and changes
+	// here need to be matched there.
+	if (fprintf(proxy, "gethostbyname %s %s %d",
+			"^",
+			name == nullptr ? "^" : name,
+			af) < 0) {
+		fclose(proxy);
+		return nullptr;
+	}
+
+	if (fputc(0, proxy) == EOF || fflush(proxy) != 0) {
+		fclose(proxy);
+		return nullptr;
+	}
+
+	struct hostent* result = android_read_hostent(proxy, hp, hbuf, hbuflen);
+	fclose(proxy);
+	return result;
+}
+
+CString ResolveAndroidHost(const char* host)
+{
+	debug("ResolveAndroidHost");
+
+	bool android = FileSystem::DirectoryExists("/data/dalvik-cache");
+	if (!android)
+	{
+		debug("Not android");
+		return nullptr;
+	}
+
+	struct hostent hinfobuf;
+	char strbuf[1024];
+
+	struct hostent* hinfo = android_gethostbyname_internal(host, AF_INET, &hinfobuf, strbuf, sizeof(strbuf));
+	if (hinfo == nullptr)
+	{
+		// trying IPv6
+		hinfo = android_gethostbyname_internal(host, AF_INET6, &hinfobuf, strbuf, sizeof(strbuf));
+	}
+
+	if (hinfo == nullptr)
+	{
+		debug("android_gethostbyname_r failed");
+		return nullptr;
+	}
+
+	BString<1024> result;
+	inet_ntop(hinfo->h_addrtype, hinfo->h_addr_list[0], result, result.Capacity());
+
+	debug("android_gethostbyname_r returned %s", *result);
+	return *result;
+}
+
+#endif

daemon/connect/Connection.h

@@ -2,7 +2,7 @@
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
  *  Copyright (C) 2004 Sven Henkel <sidddy@users.sourceforge.net>
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -41,7 +41,8 @@ public:
 		csConnected,
 		csDisconnected,
 		csListening,
-		csCancelled
+		csCancelled,
+		csBroken
 	};
 
 	enum EIPVersion
@@ -79,6 +80,7 @@ public:
 	const char* GetRemoteAddr();
 	bool GetGracefull() { return m_gracefull; }
 	void SetGracefull(bool gracefull) { m_gracefull = gracefull; }
+	void SetForceClose(bool forceClose) { m_forceClose = forceClose; }
 #ifndef DISABLE_TLS
 	bool StartTls(bool isClient, const char* certFile, const char* keyFile);
 #endif
@@ -99,8 +101,8 @@ protected:
 	bool m_suppressErrors = true;
 	BString<100> m_remoteAddr;
 	int m_totalBytesRead = 0;
-	bool m_broken = false;
 	bool m_gracefull = false;
+	bool m_forceClose = false;
 
 	struct SockAddr
 	{
@@ -138,7 +140,7 @@ protected:
 	virtual void PrintError(const char* errMsg);
 	bool DoConnect();
 	bool DoDisconnect();
-	bool InitSocketOpts();
+	bool InitSocketOpts(SOCKET socket);
 	bool ConnectWithTimeout(void* address, int address_len);
 #ifndef HAVE_GETADDRINFO
 	in_addr_t ResolveHostAddr(const char* host);

daemon/connect/TlsSocket.cpp

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2008-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2008-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -198,11 +198,36 @@ void TlsSocket::Final()
 #ifdef HAVE_LIBGNUTLS
 	gnutls_global_deinit();
 #endif /* HAVE_LIBGNUTLS */
+
+#ifdef HAVE_OPENSSL
+#ifndef LIBRESSL_VERSION_NUMBER
+	FIPS_mode_set(0);
+#endif
+#ifdef NEED_CRYPTO_LOCKING
+	CRYPTO_set_locking_callback(nullptr);
+	CRYPTO_set_id_callback(nullptr);
+#endif
+	ERR_remove_state(0);
+#if	OPENSSL_VERSION_NUMBER >= 0x10002000L && ! defined (LIBRESSL_VERSION_NUMBER)
+	SSL_COMP_free_compression_methods();
+#endif
+	//ENGINE_cleanup();
+	CONF_modules_free();
+	CONF_modules_unload(1);
+	COMP_zlib_cleanup();
+	ERR_free_strings();
+	EVP_cleanup();
+	CRYPTO_cleanup_all_ex_data();
+#endif /* HAVE_OPENSSL */
 }
 
 TlsSocket::~TlsSocket()
 {
 	Close();
+
+#ifdef HAVE_OPENSSL
+	ERR_remove_state(0);
+#endif
 }
 
 void TlsSocket::ReportError(const char* errMsg, bool suppressable)
@@ -415,14 +440,14 @@ bool TlsSocket::Start()
 		return false;
 	}
 
-	if (m_host && !SSL_set_tlsext_host_name((SSL*)m_session, m_host))
+	if (m_isClient && m_host && !SSL_set_tlsext_host_name((SSL*)m_session, m_host))
 	{
 		ReportError("Could not set host name for TLS");
 		Close();
 		return false;
 	}
 
-	if (!SSL_set_fd((SSL*)m_session, m_socket))
+	if (!SSL_set_fd((SSL*)m_session, (int)m_socket))
 	{
 		ReportError("Could not set the file descriptor for TLS");
 		Close();
@@ -515,7 +540,7 @@ bool TlsSocket::ValidateCert()
 		}
 
 		gnutls_datum_t msgdata;
-		if (gnutls_certificate-verification_status_print(status, GNUTLS_CRT_X509, &msgdata, 0) == 0)
+		if (gnutls_certificate_verification_status_print(status, GNUTLS_CRT_X509, &msgdata, 0) == 0)
 		{
 			PrintError(BString<1024>("TLS certificate verification failed for %s: %s."
 				" For more info visit http://nzbget.net/certificate-verification", *m_host, msgdata.data));

daemon/connect/WebDownloader.cpp

@@ -441,7 +441,9 @@ WebDownloader::EStatus WebDownloader::CheckResponse(const char* response)
 		warn("URL %s failed: %s", *m_infoName, hTTPResponse);
 		return adNotFound;
 	}
-	else if (!strncmp(hTTPResponse, "301", 3) || !strncmp(hTTPResponse, "302", 3))
+	else if (!strncmp(hTTPResponse, "301", 3) || !strncmp(hTTPResponse, "302", 3) ||
+		 !strncmp(hTTPResponse, "303", 3) || !strncmp(hTTPResponse, "307", 3) ||
+		 !strncmp(hTTPResponse, "308", 3))
 	{
 		m_redirecting = true;
 		return adRunning;
@@ -648,21 +650,6 @@ void WebDownloader::Stop()
 	debug("WebDownloader stopped successfully");
 }
 
-bool WebDownloader::Terminate()
-{
-	std::unique_ptr<Connection> connection = std::move(m_connection);
-	bool terminated = Kill();
-	if (terminated && connection)
-	{
-		debug("Terminating connection");
-		connection->SetSuppressErrors(true);
-		connection->Cancel();
-		connection->Disconnect();
-		connection.reset();
-	}
-	return terminated;
-}
-
 void WebDownloader::FreeConnection()
 {
 	if (m_connection)

daemon/connect/WebDownloader.h

@@ -50,7 +50,6 @@ public:
 	virtual void Stop();
 	EStatus Download();
 	EStatus DownloadWithRedirects(int maxRedirects);
-	bool Terminate();
 	void SetInfoName(const char* infoName) { m_infoName = infoName; }
 	const char* GetInfoName() { return m_infoName; }
 	void SetUrl(const char* url);

daemon/extension/QueueScript.cpp

@@ -409,11 +409,10 @@ void QueueScriptCoordinator::CheckQueue()
 		return;
 	}
 
-	m_curItem.reset();
-
 	GuardedDownloadQueue downloadQueue = DownloadQueue::Guard();
 	Guard guard(m_queueMutex);
 
+	m_curItem.reset();
 	NzbInfo* curNzbInfo = nullptr;
 	Queue::iterator itCurItem;
 

daemon/feed/FeedCoordinator.cpp

@@ -87,7 +87,7 @@ void FeedCoordinator::Run()
 		usleep(20 * 1000);
 	}
 
-	if (g_Options->GetServerMode() && g_Options->GetSaveQueue() && g_Options->GetReloadQueue())
+	if (g_Options->GetServerMode())
 	{
 		Guard guard(m_downloadsMutex);
 		g_DiskState->LoadFeeds(&m_feeds, &m_feedHistory);
@@ -118,8 +118,9 @@ void FeedCoordinator::Run()
 					for (FeedInfo* feedInfo : &m_feeds)
 					{
 						if (((feedInfo->GetInterval() > 0 &&
-							(current - feedInfo->GetLastUpdate() >= feedInfo->GetInterval() * 60 ||
-							 current < feedInfo->GetLastUpdate())) ||
+							 (feedInfo->GetNextUpdate() == 0 ||
+							  current >= feedInfo->GetNextUpdate() ||
+							  current < feedInfo->GetNextUpdate() - feedInfo->GetInterval() * 60)) ||
 							feedInfo->GetFetch()) &&
 							feedInfo->GetStatus() != FeedInfo::fsRunning)
 						{
@@ -182,8 +183,7 @@ void FeedCoordinator::Stop()
 
 void FeedCoordinator::ResetHangingDownloads()
 {
-	const int timeout = g_Options->GetTerminateTimeout();
-	if (timeout == 0)
+	if (g_Options->GetUrlTimeout() == 0)
 	{
 		return;
 	}
@@ -191,31 +191,15 @@ void FeedCoordinator::ResetHangingDownloads()
 	Guard guard(m_downloadsMutex);
 	time_t tm = Util::CurrentTime();
 
-	m_activeDownloads.erase(std::remove_if(m_activeDownloads.begin(), m_activeDownloads.end(),
-		[timeout, tm](FeedDownloader* feedDownloader)
+	for (FeedDownloader* feedDownloader: m_activeDownloads)
+	{
+		if (tm - feedDownloader->GetLastUpdateTime() > g_Options->GetUrlTimeout() + 10 &&
+			feedDownloader->GetStatus() == FeedDownloader::adRunning)
 		{
-			if (tm - feedDownloader->GetLastUpdateTime() > timeout &&
-				feedDownloader->GetStatus() == FeedDownloader::adRunning)
-			{
-				debug("Terminating hanging download %s", feedDownloader->GetInfoName());
-				if (feedDownloader->Terminate())
-				{
-					error("Terminated hanging download %s", feedDownloader->GetInfoName());
-					feedDownloader->GetFeedInfo()->SetStatus(FeedInfo::fsUndefined);
-				}
-				else
-				{
-					error("Could not terminate hanging download %s", feedDownloader->GetInfoName());
-				}
-
-				// it's not safe to destroy feedDownloader, because the state of object is unknown
-				delete feedDownloader;
-
-				return true;
-			}
-			return false;
-		}),
-		m_activeDownloads.end());
+			error("Cancelling hanging feed download %s", feedDownloader->GetInfoName());
+			feedDownloader->Stop();
+		}
+	}
 }
 
 void FeedCoordinator::LogDebugInfo()
@@ -291,6 +275,8 @@ void FeedCoordinator::FeedCompleted(FeedDownloader* feedDownloader)
 		m_activeDownloads.erase(std::find(m_activeDownloads.begin(), m_activeDownloads.end(), feedDownloader));
 	}
 
+	SchedulerNextUpdate(feedInfo, statusOK);
+
 	if (statusOK)
 	{
 		if (!feedInfo->GetPreview())
@@ -337,6 +323,30 @@ void FeedCoordinator::FeedCompleted(FeedDownloader* feedDownloader)
 	}
 }
 
+void FeedCoordinator::SchedulerNextUpdate(FeedInfo* feedInfo, bool success)
+{
+	time_t current = Util::CurrentTime();
+	int interval;
+
+	if (success)
+	{
+		interval = feedInfo->GetInterval() * 60;
+		feedInfo->SetLastInterval(0);
+	}
+	else
+	{
+		// On failure schedule next update sooner:
+		// starting with 1 minute and increasing, but not greater than FeedX.Interval
+		interval = feedInfo->GetLastInterval() * 2;
+		interval = std::max(interval, 60);
+		interval = std::min(interval, feedInfo->GetInterval() * 60);
+		feedInfo->SetLastInterval(interval);
+	}
+
+	detail("Scheduling update for feed %s in %i minute(s)", feedInfo->GetName(), interval / 60);
+	feedInfo->SetNextUpdate(current + interval);
+}
+
 void FeedCoordinator::FilterFeed(FeedInfo* feedInfo, FeedItemList* feedItems)
 {
 	debug("Filtering feed %s", feedInfo->GetName());
@@ -588,7 +598,7 @@ std::unique_ptr<FeedFile> FeedCoordinator::parseFeed(FeedInfo* feedInfo)
 		error("Feed file %s kept for troubleshooting (will be deleted on next successful feed fetch)", feedInfo->GetOutputFilename());
 		feedFile.reset();
 	}
-	return std::move(feedFile);
+	return feedFile;
 }
 
 void FeedCoordinator::DownloadQueueUpdate(Subject* caller, void* aspect)
@@ -620,11 +630,11 @@ bool FeedCoordinator::HasActiveDownloads()
 
 void FeedCoordinator::CheckSaveFeeds()
 {
-	debug("CheckSaveFeeds");
 	Guard guard(m_downloadsMutex);
 	if (m_save)
 	{
-		if (g_Options->GetSaveQueue() && g_Options->GetServerMode())
+		debug("CheckSaveFeeds: save");
+		if (g_Options->GetServerMode())
 		{
 			g_DiskState->SaveFeeds(&m_feeds, &m_feedHistory);
 		}

daemon/feed/FeedCoordinator.h

@@ -121,6 +121,7 @@ private:
 	void CleanupCache();
 	void CheckSaveFeeds();
 	std::unique_ptr<FeedFile> parseFeed(FeedInfo* feedInfo);
+	void SchedulerNextUpdate(FeedInfo* feedInfo, bool success);
 };
 
 extern FeedCoordinator* g_FeedCoordinator;

daemon/feed/FeedFile.cpp

@@ -360,6 +360,10 @@ bool FeedFile::ParseFeed(IUnknown* nzb)
 
 bool FeedFile::Parse()
 {
+#ifdef DISABLE_LIBXML2
+	error("Could not parse rss feed, program was compiled without libxml2 support");
+	return false;
+#else
 	xmlSAXHandler SAX_handler = {0};
 	SAX_handler.startElement = reinterpret_cast<startElementSAXFunc>(SAX_StartElement);
 	SAX_handler.endElement = reinterpret_cast<endElementSAXFunc>(SAX_EndElement);
@@ -378,6 +382,7 @@ bool FeedFile::Parse()
 	}
 
 	return true;
+#endif
 }
 
 void FeedFile::Parse_StartElement(const char *name, const char **atts)
@@ -566,7 +571,11 @@ void FeedFile::SAX_characters(FeedFile* file, const char * xmlstr, int len)
 
 void* FeedFile::SAX_getEntity(FeedFile* file, const char * name)
 {
+#ifdef DISABLE_LIBXML2
+	void* e = nullptr;
+#else
 	xmlEntityPtr e = xmlGetPredefinedEntity((xmlChar* )name);
+#endif
 	if (!e)
 	{
 		warn("entity not found");

daemon/feed/FeedFilter.cpp

@@ -897,7 +897,7 @@ void FeedFilter::Rule::ExpandRefValues(FeedItemInfo& feedItemInfo, CString* dest
 			break; // error
 		}
 
-		curvalue.Replace(dollar - curvalue, 2 + varlen + 1, varvalue);
+		curvalue.Replace((int)(dollar - curvalue), 2 + varlen + 1, varvalue);
 	}
 
 	*destStr = std::move(curvalue);

daemon/feed/FeedInfo.h

@@ -51,6 +51,10 @@ public:
 	const char* GetExtensions() { return m_extensions; }
 	time_t GetLastUpdate() { return m_lastUpdate; }
 	void SetLastUpdate(time_t lastUpdate) { m_lastUpdate = lastUpdate; }
+	time_t GetNextUpdate() { return m_nextUpdate; }
+	void SetNextUpdate(time_t nextUpdate) { m_nextUpdate = nextUpdate; }
+	int GetLastInterval() { return m_lastInterval; }
+	void SetLastInterval(int lastInterval) { m_lastInterval = lastInterval; }
 	bool GetPreview() { return m_preview; }
 	void SetPreview(bool preview) { m_preview = preview; }
 	EStatus GetStatus() { return m_status; }
@@ -76,6 +80,8 @@ private:
 	CString m_extensions;
 	int m_priority;
 	time_t m_lastUpdate = 0;
+	time_t m_nextUpdate = 0;
+	int m_lastInterval = 0;
 	bool m_preview = false;
 	EStatus m_status = fsUndefined;
 	CString m_outputFilename;

daemon/main/CommandLineParser.cpp

@@ -894,7 +894,7 @@ void CommandLineParser::ParseFileIdList(int argc, const char* argv[], int optind
 			if (p)
 			{
 				BString<100> buf;
-				buf.Set(optarg, p - optarg);
+				buf.Set(optarg, (int)(p - optarg));
 				editQueueIdFrom = atoi(buf);
 				editQueueIdTo = atoi(p + 1);
 				if (editQueueIdFrom <= 0 || editQueueIdTo <= 0)

daemon/main/Maintenance.cpp

@@ -179,7 +179,7 @@ bool Maintenance::ReadPackageInfoStr(const char* key, CString& value)
 		return false;
 	}
 
-	int len = pend - p;
+	int len = (int)(pend - p);
 	if (len >= sizeof(fileName))
 	{
 		error("Could not parse file %s", *fileName);

daemon/main/Options.cpp

@@ -69,20 +69,18 @@ static const char* OPTION_CERTCHECK				= "CertCheck";
 static const char* OPTION_AUTHORIZEDIP			= "AuthorizedIP";
 static const char* OPTION_ARTICLETIMEOUT		= "ArticleTimeout";
 static const char* OPTION_URLTIMEOUT			= "UrlTimeout";
-static const char* OPTION_SAVEQUEUE				= "SaveQueue";
+static const char* OPTION_REMOTETIMEOUT			= "RemoteTimeout";
 static const char* OPTION_FLUSHQUEUE			= "FlushQueue";
-static const char* OPTION_RELOADQUEUE			= "ReloadQueue";
-static const char* OPTION_BROKENLOG				= "BrokenLog";
 static const char* OPTION_NZBLOG				= "NzbLog";
-static const char* OPTION_DECODE				= "Decode";
+static const char* OPTION_RAWARTICLE			= "RawArticle";
+static const char* OPTION_SKIPWRITE				= "SkipWrite";
 static const char* OPTION_ARTICLERETRIES		= "ArticleRetries";
 static const char* OPTION_ARTICLEINTERVAL		= "ArticleInterval";
 static const char* OPTION_URLRETRIES			= "UrlRetries";
 static const char* OPTION_URLINTERVAL			= "UrlInterval";
-static const char* OPTION_TERMINATETIMEOUT		= "TerminateTimeout";
 static const char* OPTION_CONTINUEPARTIAL		= "ContinuePartial";
 static const char* OPTION_URLCONNECTIONS		= "UrlConnections";
-static const char* OPTION_LOGBUFFERSIZE			= "LogBufferSize";
+static const char* OPTION_LOGBUFFER				= "LogBuffer";
 static const char* OPTION_INFOTARGET			= "InfoTarget";
 static const char* OPTION_WARNINGTARGET			= "WarningTarget";
 static const char* OPTION_ERRORTARGET			= "ErrorTarget";
@@ -118,7 +116,6 @@ static const char* OPTION_SCRIPTPAUSEQUEUE		= "ScriptPauseQueue";
 static const char* OPTION_NZBCLEANUPDISK		= "NzbCleanupDisk";
 static const char* OPTION_PARTIMELIMIT			= "ParTimeLimit";
 static const char* OPTION_KEEPHISTORY			= "KeepHistory";
-static const char* OPTION_ACCURATERATE			= "AccurateRate";
 static const char* OPTION_UNPACK				= "Unpack";
 static const char* OPTION_DIRECTUNPACK			= "DirectUnpack";
 static const char* OPTION_UNPACKCLEANUPDISK		= "UnpackCleanupDisk";
@@ -142,6 +139,7 @@ static const char* OPTION_MONTHLYQUOTA			= "MonthlyQuota";
 static const char* OPTION_QUOTASTARTDAY			= "QuotaStartDay";
 static const char* OPTION_DAILYQUOTA			= "DailyQuota";
 static const char* OPTION_REORDERFILES			= "ReorderFiles";
+static const char* OPTION_UPDATECHECK			= "UpdateCheck";
 
 // obsolete options
 static const char* OPTION_POSTLOGKIND			= "PostLogKind";
@@ -168,6 +166,13 @@ static const char* OPTION_HISTORYCLEANUPDISK	= "HistoryCleanupDisk";
 static const char* OPTION_SCANSCRIPT			= "ScanScript";
 static const char* OPTION_QUEUESCRIPT			= "QueueScript";
 static const char* OPTION_FEEDSCRIPT			= "FeedScript";
+static const char* OPTION_DECODE				= "Decode";
+static const char* OPTION_SAVEQUEUE				= "SaveQueue";
+static const char* OPTION_RELOADQUEUE			= "ReloadQueue";
+static const char* OPTION_TERMINATETIMEOUT		= "TerminateTimeout";
+static const char* OPTION_ACCURATERATE			= "AccurateRate";
+static const char* OPTION_CREATEBROKENLOG		= "CreateBrokenLog";
+static const char* OPTION_BROKENLOG				= "BrokenLog";
 
 const char* BoolNames[] = { "yes", "no", "true", "false", "1", "0", "on", "off", "enable", "disable", "enabled", "disabled" };
 const int BoolValues[] = { 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0 };
@@ -416,7 +421,7 @@ void Options::InitDefaults()
 	SetOption(OPTION_QUEUEDIR, "${MainDir}/queue");
 	SetOption(OPTION_NZBDIR, "${MainDir}/nzb");
 	SetOption(OPTION_LOCKFILE, "${MainDir}/nzbget.lock");
-	SetOption(OPTION_LOGFILE, "${DestDir}/nzbget.log");
+	SetOption(OPTION_LOGFILE, "${MainDir}/nzbget.log");
 	SetOption(OPTION_SCRIPTDIR, "${MainDir}/scripts");
 	SetOption(OPTION_REQUIREDDIR, "");
 	SetOption(OPTION_WRITELOG, "append");
@@ -443,20 +448,18 @@ void Options::InitDefaults()
 	SetOption(OPTION_AUTHORIZEDIP, "");
 	SetOption(OPTION_ARTICLETIMEOUT, "60");
 	SetOption(OPTION_URLTIMEOUT, "60");
-	SetOption(OPTION_SAVEQUEUE, "yes");
+	SetOption(OPTION_REMOTETIMEOUT, "90");
 	SetOption(OPTION_FLUSHQUEUE, "yes");
-	SetOption(OPTION_RELOADQUEUE, "yes");
-	SetOption(OPTION_BROKENLOG, "yes");
 	SetOption(OPTION_NZBLOG, "yes");
-	SetOption(OPTION_DECODE, "yes");
+	SetOption(OPTION_RAWARTICLE, "no");
+	SetOption(OPTION_SKIPWRITE, "no");
 	SetOption(OPTION_ARTICLERETRIES, "3");
 	SetOption(OPTION_ARTICLEINTERVAL, "10");
 	SetOption(OPTION_URLRETRIES, "3");
 	SetOption(OPTION_URLINTERVAL, "10");
-	SetOption(OPTION_TERMINATETIMEOUT, "600");
 	SetOption(OPTION_CONTINUEPARTIAL, "no");
 	SetOption(OPTION_URLCONNECTIONS, "4");
-	SetOption(OPTION_LOGBUFFERSIZE, "1000");
+	SetOption(OPTION_LOGBUFFER, "1000");
 	SetOption(OPTION_INFOTARGET, "both");
 	SetOption(OPTION_WARNINGTARGET, "both");
 	SetOption(OPTION_ERRORTARGET, "both");
@@ -495,7 +498,6 @@ void Options::InitDefaults()
 	SetOption(OPTION_NZBCLEANUPDISK, "no");
 	SetOption(OPTION_PARTIMELIMIT, "0");
 	SetOption(OPTION_KEEPHISTORY, "7");
-	SetOption(OPTION_ACCURATERATE, "no");
 	SetOption(OPTION_UNPACK, "no");
 	SetOption(OPTION_DIRECTUNPACK, "no");
 	SetOption(OPTION_UNPACKCLEANUPDISK, "no");
@@ -522,6 +524,7 @@ void Options::InitDefaults()
 	SetOption(OPTION_QUOTASTARTDAY, "1");
 	SetOption(OPTION_DAILYQUOTA, "0");
 	SetOption(OPTION_REORDERFILES, "no");
+	SetOption(OPTION_UPDATECHECK, "none");
 }
 
 void Options::InitOptFile()
@@ -694,7 +697,7 @@ void Options::InitOptions()
 	m_downloadRate			= ParseIntValue(OPTION_DOWNLOADRATE, 10) * 1024;
 	m_articleTimeout		= ParseIntValue(OPTION_ARTICLETIMEOUT, 10);
 	m_urlTimeout			= ParseIntValue(OPTION_URLTIMEOUT, 10);
-	m_terminateTimeout		= ParseIntValue(OPTION_TERMINATETIMEOUT, 10);
+	m_remoteTimeout			= ParseIntValue(OPTION_REMOTETIMEOUT, 10);
 	m_articleRetries		= ParseIntValue(OPTION_ARTICLERETRIES, 10);
 	m_articleInterval		= ParseIntValue(OPTION_ARTICLEINTERVAL, 10);
 	m_urlRetries			= ParseIntValue(OPTION_URLRETRIES, 10);
@@ -702,7 +705,7 @@ void Options::InitOptions()
 	m_controlPort			= ParseIntValue(OPTION_CONTROLPORT, 10);
 	m_securePort			= ParseIntValue(OPTION_SECUREPORT, 10);
 	m_urlConnections		= ParseIntValue(OPTION_URLCONNECTIONS, 10);
-	m_logBufferSize			= ParseIntValue(OPTION_LOGBUFFERSIZE, 10);
+	m_logBuffer				= ParseIntValue(OPTION_LOGBUFFER, 10);
 	m_rotateLog				= ParseIntValue(OPTION_ROTATELOG, 10);
 	m_umask					= ParseIntValue(OPTION_UMASK, 8);
 	m_updateInterval		= ParseIntValue(OPTION_UPDATEINTERVAL, 10);
@@ -728,11 +731,9 @@ void Options::InitOptions()
 	m_quotaStartDay			= ParseIntValue(OPTION_QUOTASTARTDAY, 10);
 	m_dailyQuota			= ParseIntValue(OPTION_DAILYQUOTA, 10);
 
-	m_brokenLog				= (bool)ParseEnumValue(OPTION_BROKENLOG, BoolCount, BoolNames, BoolValues);
 	m_nzbLog				= (bool)ParseEnumValue(OPTION_NZBLOG, BoolCount, BoolNames, BoolValues);
 	m_appendCategoryDir		= (bool)ParseEnumValue(OPTION_APPENDCATEGORYDIR, BoolCount, BoolNames, BoolValues);
 	m_continuePartial		= (bool)ParseEnumValue(OPTION_CONTINUEPARTIAL, BoolCount, BoolNames, BoolValues);
-	m_saveQueue				= (bool)ParseEnumValue(OPTION_SAVEQUEUE, BoolCount, BoolNames, BoolValues);
 	m_flushQueue			= (bool)ParseEnumValue(OPTION_FLUSHQUEUE, BoolCount, BoolNames, BoolValues);
 	m_dupeCheck				= (bool)ParseEnumValue(OPTION_DUPECHECK, BoolCount, BoolNames, BoolValues);
 	m_parRepair				= (bool)ParseEnumValue(OPTION_PARREPAIR, BoolCount, BoolNames, BoolValues);
@@ -740,19 +741,18 @@ void Options::InitOptions()
 	m_parRename				= (bool)ParseEnumValue(OPTION_PARRENAME, BoolCount, BoolNames, BoolValues);
 	m_rarRename				= (bool)ParseEnumValue(OPTION_RARRENAME, BoolCount, BoolNames, BoolValues);
 	m_directRename			= (bool)ParseEnumValue(OPTION_DIRECTRENAME, BoolCount, BoolNames, BoolValues);
-	m_reloadQueue			= (bool)ParseEnumValue(OPTION_RELOADQUEUE, BoolCount, BoolNames, BoolValues);
 	m_cursesNzbName			= (bool)ParseEnumValue(OPTION_CURSESNZBNAME, BoolCount, BoolNames, BoolValues);
 	m_cursesTime			= (bool)ParseEnumValue(OPTION_CURSESTIME, BoolCount, BoolNames, BoolValues);
 	m_cursesGroup			= (bool)ParseEnumValue(OPTION_CURSESGROUP, BoolCount, BoolNames, BoolValues);
 	m_crcCheck				= (bool)ParseEnumValue(OPTION_CRCCHECK, BoolCount, BoolNames, BoolValues);
 	m_directWrite			= (bool)ParseEnumValue(OPTION_DIRECTWRITE, BoolCount, BoolNames, BoolValues);
-	m_decode				= (bool)ParseEnumValue(OPTION_DECODE, BoolCount, BoolNames, BoolValues);
+	m_rawArticle			= (bool)ParseEnumValue(OPTION_RAWARTICLE, BoolCount, BoolNames, BoolValues);
+	m_skipWrite				= (bool)ParseEnumValue(OPTION_SKIPWRITE, BoolCount, BoolNames, BoolValues);
 	m_crashTrace			= (bool)ParseEnumValue(OPTION_CRASHTRACE, BoolCount, BoolNames, BoolValues);
 	m_crashDump				= (bool)ParseEnumValue(OPTION_CRASHDUMP, BoolCount, BoolNames, BoolValues);
 	m_parPauseQueue			= (bool)ParseEnumValue(OPTION_PARPAUSEQUEUE, BoolCount, BoolNames, BoolValues);
 	m_scriptPauseQueue		= (bool)ParseEnumValue(OPTION_SCRIPTPAUSEQUEUE, BoolCount, BoolNames, BoolValues);
 	m_nzbCleanupDisk		= (bool)ParseEnumValue(OPTION_NZBCLEANUPDISK, BoolCount, BoolNames, BoolValues);
-	m_accurateRate			= (bool)ParseEnumValue(OPTION_ACCURATERATE, BoolCount, BoolNames, BoolValues);
 	m_formAuth				= (bool)ParseEnumValue(OPTION_FORMAUTH, BoolCount, BoolNames, BoolValues);
 	m_secureControl			= (bool)ParseEnumValue(OPTION_SECURECONTROL, BoolCount, BoolNames, BoolValues);
 	m_unpack				= (bool)ParseEnumValue(OPTION_UNPACK, BoolCount, BoolNames, BoolValues);
@@ -917,7 +917,7 @@ void Options::SetOption(const char* optname, const char* value)
 			const char* varvalue = GetOption(variable);
 			if (varvalue)
 			{
-				curvalue.Replace(dollar - curvalue, 2 + varlen + 1, varvalue);
+				curvalue.Replace((int)(dollar - curvalue), 2 + varlen + 1, varvalue);
 			}
 			else
 			{
@@ -1502,7 +1502,7 @@ bool Options::SplitOptionString(const char* option, CString& optName, CString& o
 		return false;
 	}
 
-	optName.Set(option, eq - option);
+	optName.Set(option, (int)(eq - option));
 	optValue.Set(eq + 1);
 
 	ConvertOldOption(optName, optValue);
@@ -1601,7 +1601,13 @@ bool Options::ValidateOptionName(const char* optname, const char* optvalue)
 		!strcasecmp(optname, OPTION_RELOADPOSTQUEUE) ||
 		!strcasecmp(optname, OPTION_PARCLEANUPQUEUE) ||
 		!strcasecmp(optname, OPTION_DELETECLEANUPDISK) ||
-		!strcasecmp(optname, OPTION_HISTORYCLEANUPDISK))
+		!strcasecmp(optname, OPTION_HISTORYCLEANUPDISK) ||
+		!strcasecmp(optname, OPTION_SAVEQUEUE) ||
+		!strcasecmp(optname, OPTION_RELOADQUEUE) ||
+		!strcasecmp(optname, OPTION_TERMINATETIMEOUT) ||
+		!strcasecmp(optname, OPTION_ACCURATERATE) ||
+		!strcasecmp(optname, OPTION_CREATEBROKENLOG) ||
+		!strcasecmp(optname, OPTION_BROKENLOG))
 	{
 		ConfigWarn("Option \"%s\" is obsolete, ignored", optname);
 		return true;
@@ -1721,15 +1727,21 @@ void Options::ConvertOldOption(CString& option, CString& value)
 		option = "ArticleInterval";
 	}
 
-	if (!strcasecmp(option, "CreateBrokenLog"))
-	{
-		option = "BrokenLog";
-	}
-
 	if (!strcasecmp(option, "DumpCore"))
 	{
 		option = OPTION_CRASHDUMP;
 	}
+
+	if (!strcasecmp(option, OPTION_DECODE))
+	{
+		option = OPTION_RAWARTICLE;
+		value = !strcasecmp(value, "no") ? "yes" : "no";
+	}
+
+	if (!strcasecmp(option, "LogBufferSize"))
+	{
+		option = OPTION_LOGBUFFER;
+	}
 }
 
 void Options::CheckOptions()
@@ -1803,11 +1815,16 @@ void Options::CheckOptions()
 		m_certCheck = false;
 	}
 
-	if (!m_decode)
+	if (m_rawArticle)
 	{
 		m_directWrite = false;
 	}
 
+	if (m_skipWrite)
+	{
+		m_directRename = false;
+	}
+
 	// if option "ConfigTemplate" is not set, use "WebDir" as default location for template
 	// (for compatibility with versions 9 and 10).
 	if (m_configTemplate.Empty() && !m_noDiskAccess)

daemon/main/Options.h

@@ -203,7 +203,6 @@ public:
 	const char* GetConfigTemplate() { return m_configTemplate; }
 	const char* GetScriptDir() { return m_scriptDir; }
 	const char* GetRequiredDir() { return m_requiredDir; }
-	bool GetBrokenLog() const { return m_brokenLog; }
 	bool GetNzbLog() const { return m_nzbLog; }
 	EMessageTarget GetInfoTarget() const { return m_infoTarget; }
 	EMessageTarget GetWarningTarget() const { return m_warningTarget; }
@@ -212,15 +211,15 @@ public:
 	EMessageTarget GetDetailTarget() const { return m_detailTarget; }
 	int GetArticleTimeout() { return m_articleTimeout; }
 	int GetUrlTimeout() { return m_urlTimeout; }
-	int GetTerminateTimeout() { return m_terminateTimeout; }
-	bool GetDecode() { return m_decode; };
+	int GetRemoteTimeout() { return m_remoteTimeout; }
+	bool GetRawArticle() { return m_rawArticle; };
+	bool GetSkipWrite() { return m_skipWrite; };
 	bool GetAppendCategoryDir() { return m_appendCategoryDir; }
 	bool GetContinuePartial() { return m_continuePartial; }
 	int GetArticleRetries() { return m_articleRetries; }
 	int GetArticleInterval() { return m_articleInterval; }
 	int GetUrlRetries() { return m_urlRetries; }
 	int GetUrlInterval() { return m_urlInterval; }
-	bool GetSaveQueue() { return m_saveQueue; }
 	bool GetFlushQueue() { return m_flushQueue; }
 	bool GetDupeCheck() { return m_dupeCheck; }
 	const char* GetControlIp() { return m_controlIp; }
@@ -242,9 +241,8 @@ public:
 	const char* GetLockFile() { return m_lockFile; }
 	const char* GetDaemonUsername() { return m_daemonUsername; }
 	EOutputMode GetOutputMode() { return m_outputMode; }
-	bool GetReloadQueue() { return m_reloadQueue; }
 	int GetUrlConnections() { return m_urlConnections; }
-	int GetLogBufferSize() { return m_logBufferSize; }
+	int GetLogBuffer() { return m_logBuffer; }
 	EWriteLog GetWriteLog() { return m_writeLog; }
 	const char* GetLogFile() { return m_logFile; }
 	int GetRotateLog() { return m_rotateLog; }
@@ -279,7 +277,6 @@ public:
 	bool GetNzbCleanupDisk() { return m_nzbCleanupDisk; }
 	int GetParTimeLimit() { return m_parTimeLimit; }
 	int GetKeepHistory() { return m_keepHistory; }
-	bool GetAccurateRate() { return m_accurateRate; }
 	bool GetUnpack() { return m_unpack; }
 	bool GetDirectUnpack() { return m_directUnpack; }
 	bool GetUnpackCleanupDisk() { return m_unpackCleanupDisk; }
@@ -361,19 +358,18 @@ private:
 	EMessageTarget m_errorTarget = mtScreen;
 	EMessageTarget m_debugTarget = mtNone;
 	EMessageTarget m_detailTarget = mtScreen;
-	bool m_decode = true;
-	bool m_brokenLog = false;
+	bool m_skipWrite = false;
+	bool m_rawArticle = false;
 	bool m_nzbLog = false;
 	int m_articleTimeout = 0;
 	int m_urlTimeout = 0;
-	int m_terminateTimeout = 0;
+	int m_remoteTimeout = 0;
 	bool m_appendCategoryDir = false;
 	bool m_continuePartial = false;
 	int m_articleRetries = 0;
 	int m_articleInterval = 0;
 	int m_urlRetries = 0;
 	int m_urlInterval = 0;
-	bool m_saveQueue = false;
 	bool m_flushQueue = false;
 	bool m_dupeCheck = false;
 	CString m_controlIp;
@@ -395,9 +391,8 @@ private:
 	CString m_lockFile;
 	CString m_daemonUsername;
 	EOutputMode m_outputMode = omLoggable;
-	bool m_reloadQueue = false;
 	int m_urlConnections = 0;
-	int m_logBufferSize = 0;
+	int m_logBuffer = 0;
 	EWriteLog m_writeLog = wlAppend;
 	int m_rotateLog = 0;
 	CString m_logFile;
@@ -433,7 +428,6 @@ private:
 	bool m_nzbCleanupDisk = false;
 	int m_parTimeLimit = 0;
 	int m_keepHistory = 0;
-	bool m_accurateRate = false;
 	bool m_unpack = false;
 	bool m_directUnpack = false;
 	bool m_unpackCleanupDisk = false;

daemon/main/nzbget.cpp

@@ -52,6 +52,7 @@
 #include "FileSystem.h"
 #include "StackTrace.h"
 #include "CommandScript.h"
+#include "YEncode.h"
 #ifdef WIN32
 #include "WinService.h"
 #include "WinConsole.h"
@@ -112,6 +113,7 @@ int main(int argc, char *argv[], char *argp[])
 #endif
 
 	Util::Init();
+	YEncode::init();
 
 	g_ArgumentCount = argc;
 	g_Arguments = (char*(*)[])argv;
@@ -145,7 +147,7 @@ int main(int argc, char *argv[], char *argp[])
 	InstallUninstallServiceCheck(argc, argv);
 #endif
 
-	srand(Util::CurrentTime());
+	srand((unsigned int)Util::CurrentTime());
 
 #ifdef WIN32
 	for (int i=0; i < argc; i++)
@@ -499,7 +501,11 @@ void NZBGet::StartRemoteServer()
 	m_remoteServer = std::make_unique<RemoteServer>(false);
 	m_remoteServer->Start();
 
-	if (m_options->GetSecureControl())
+	if (m_options->GetSecureControl()
+#ifndef WIN32
+		&& !(m_options->GetControlIp() && m_options->GetControlIp()[0] == '/')
+#endif
+)
 	{
 		m_remoteSecureServer = std::make_unique<RemoteServer>(true);
 		m_remoteSecureServer->Start();
@@ -512,37 +518,55 @@ void NZBGet::StopRemoteServer()
 	{
 		debug("stopping RemoteServer");
 		m_remoteServer->Stop();
-		int maxWaitMSec = 1000;
-		while (m_remoteServer->IsRunning() && maxWaitMSec > 0)
-		{
-			usleep(100 * 1000);
-			maxWaitMSec -= 100;
-		}
-		if (m_remoteServer->IsRunning())
-		{
-			debug("Killing RemoteServer");
-			m_remoteServer->Kill();
-		}
-		debug("RemoteServer stopped");
 	}
 
 	if (m_remoteSecureServer)
 	{
 		debug("stopping RemoteSecureServer");
 		m_remoteSecureServer->Stop();
-		int maxWaitMSec = 1000;
-		while (m_remoteSecureServer->IsRunning() && maxWaitMSec > 0)
-		{
-			usleep(100 * 1000);
-			maxWaitMSec -= 100;
-		}
-		if (m_remoteSecureServer->IsRunning())
-		{
-			debug("Killing RemoteSecureServer");
-			m_remoteSecureServer->Kill();
-		}
-		debug("RemoteSecureServer stopped");
 	}
+
+	int maxWaitMSec = 5000;
+	while (((m_remoteServer && m_remoteServer->IsRunning()) ||
+		(m_remoteSecureServer && m_remoteSecureServer->IsRunning())) &&
+		maxWaitMSec > 0)
+	{
+		usleep(100 * 1000);
+		maxWaitMSec -= 100;
+	}
+
+	if (m_remoteServer && m_remoteServer->IsRunning())
+	{
+		m_remoteServer->ForceStop();
+	}
+
+	if (m_remoteSecureServer && m_remoteSecureServer->IsRunning())
+	{
+		m_remoteSecureServer->ForceStop();
+	}
+
+	maxWaitMSec = 5000;
+	while (((m_remoteServer && m_remoteServer->IsRunning()) ||
+		(m_remoteSecureServer && m_remoteSecureServer->IsRunning())) &&
+		maxWaitMSec > 0)
+	{
+		usleep(100 * 1000);
+		maxWaitMSec -= 100;
+	}
+
+	if (m_remoteServer && m_remoteServer->IsRunning())
+	{
+		debug("Killing RemoteServer");
+		m_remoteServer->Kill();
+	}
+
+	if (m_remoteSecureServer && m_remoteSecureServer->IsRunning())
+	{
+		debug("Killing RemoteSecureServer");
+		m_remoteSecureServer->Kill();
+	}
+
+	debug("RemoteServer stopped");
 }
 
 void NZBGet::StartFrontend()

daemon/main/nzbget.h

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -59,9 +59,6 @@ compiled */
 /* Define to 1 if variadic macros are supported */
 #define HAVE_VARIADIC_MACROS
 
-/* Define to 1 if libpar2 supports cancelling (needs a special patch) */
-#define HAVE_PAR2_CANCEL
-
 /* Define to 1 if function GetAddrInfo is supported */
 #define HAVE_GETADDRINFO
 
@@ -69,7 +66,16 @@ compiled */
 #define SOCKLEN_T socklen_t
 
 /* Define to 1 if you have the <regex.h> header file. */
+#ifndef DISABLE_REGEX
 #define HAVE_REGEX_H 1
+// Static linking to regex library
+#define REGEX_STATIC
+#endif
+
+#ifndef DISABLE_GZIP
+// Static linking to zlib library
+#define ZLIB_WINAPI
+#endif
 
 /* Suppress warnings */
 #define _CRT_SECURE_NO_DEPRECATE
@@ -77,13 +83,21 @@ compiled */
 /* Suppress warnings */
 #define _CRT_NONSTDC_NO_WARNINGS
 
+#ifndef _WIN64
 #define _USE_32BIT_TIME_T
+#endif
 
 #if _WIN32_WINNT < 0x0501
 #undef _WIN32_WINNT
 #define _WIN32_WINNT 0x0501
 #endif
 
+#ifdef _WIN64
+#define __amd64__
+#else
+#define __i686__
+#endif
+
 #ifdef _DEBUG
 // detection of memory leaks
 #define _CRTDBG_MAP_ALLOC
@@ -154,21 +168,28 @@ using namespace MSXML;
 #include <sys/resource.h>
 #include <sys/statvfs.h>
 #include <sys/wait.h>
+#include <sys/un.h>
 #include <arpa/inet.h>
 #include <netinet/in.h>
 #include <stdint.h>
 #include <pwd.h>
 #include <dirent.h>
 
+#ifndef DISABLE_LIBXML2
 #include <libxml/parser.h>
 #include <libxml/xmlreader.h>
 #include <libxml/xmlerror.h>
 #include <libxml/entities.h>
+#endif
 
 #ifdef HAVE_SYS_PRCTL_H
 #include <sys/prctl.h>
 #endif
 
+#ifdef HAVE_ENDIAN_H
+#include <endian.h>
+#endif
+
 #ifdef HAVE_BACKTRACE
 #include <execinfo.h>
 #endif
@@ -194,13 +215,19 @@ using namespace MSXML;
 #include <list>
 #include <set>
 #include <map>
+#include <unordered_map>
 #include <iterator>
 #include <algorithm>
-#include <iostream>
 #include <fstream>
 #include <memory>
 #include <functional>
 
+// NOTE: do not include <iostream> in "nzbget.h". <iostream> contains objects requiring
+// intialization, causing every unit in nzbget to have initialization routine. This in particular
+// is causing fatal problems in SIMD units which must not have static initialization because
+// they contain code with runtime CPU dispatching.
+//#include <iostream>
+
 #ifdef HAVE_LIBGNUTLS
 #ifdef WIN32
 #include <BaseTsd.h>
@@ -227,6 +254,7 @@ typedef int pid_t;
 #include <openssl/sha.h>
 #include <openssl/pem.h>
 #include <openssl/x509v3.h>
+#include <openssl/comp.h>
 #endif /* HAVE_OPENSSL */
 
 #ifdef HAVE_REGEX_H
@@ -295,11 +323,22 @@ typedef int pid_t;
 #define FOPEN_WB "wbN"
 #define FOPEN_AB "abN"
 
+#define __SSE2__
+#define __SSSE3__
+#define __PCLMUL__
+
 #ifdef DEBUG
 // redefine "exit" to avoid printing memory leaks report when terminated because of wrong command line switches
 #define exit(code) ExitProcess(code)
 #endif
 
+#ifdef HAVE_OPENSSL
+FILE _iob[] = {*stdin, *stdout, *stderr};
+extern "C" FILE * __cdecl __iob_func(void) { return _iob; }
+// For static linking of OpenSSL libraries:
+#pragma comment (lib, "legacy_stdio_definitions.lib")
+#endif /* HAVE_OPENSSL */
+
 #else
 
 // POSIX
@@ -345,6 +384,13 @@ typedef unsigned long long uint64;
 
 typedef unsigned char uchar;
 
+// Assume little endian if byte order is not defined
+#ifndef __BYTE_ORDER
+#define __LITTLE_ENDIAN 1234
+#define __BIG_ENDIAN    4321
+#define __BYTE_ORDER __LITTLE_ENDIAN
+#endif
+
 #ifdef __GNUC__
 #define PRINTF_SYNTAX(strindex) __attribute__ ((format (printf, strindex, strindex+1)))
 #define SCANF_SYNTAX(strindex) __attribute__ ((format (scanf, strindex, strindex+1)))

daemon/nntp/ArticleDownloader.cpp

@@ -2,7 +2,7 @@
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
  *  Copyright (C) 2004 Sven Henkel <sidddy@users.sourceforge.net>
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -33,7 +33,6 @@ ArticleDownloader::ArticleDownloader()
 {
 	debug("Creating ArticleDownloader");
 
-	m_articleWriter.SetOwner(this);
 	SetLastUpdateTimeNow();
 }
 
@@ -321,7 +320,8 @@ ArticleDownloader::EStatus ArticleDownloader::Download()
 	}
 
 	// retrieve article
-	response = m_connection->Request(BString<1024>("ARTICLE %s\r\n", m_articleInfo->GetMessageId()));
+	response = m_connection->Request(BString<1024>("%s %s\r\n",
+		g_Options->GetRawArticle() ? "ARTICLE" : "BODY", m_articleInfo->GetMessageId()));
 
 	status = CheckResponse(response, "could not fetch article");
 	if (status != adFinished)
@@ -329,28 +329,16 @@ ArticleDownloader::EStatus ArticleDownloader::Download()
 		return status;
 	}
 
-	if (g_Options->GetDecode())
-	{
-		m_yDecoder.Clear();
-		m_yDecoder.SetCrcCheck(g_Options->GetCrcCheck());
-		m_uDecoder.Clear();
-	}
+	m_decoder.Clear();
+	m_decoder.SetCrcCheck(g_Options->GetCrcCheck());
+	m_decoder.SetRawMode(g_Options->GetRawArticle());
 
-	bool body = false;
-	bool end = false;
-	CharBuffer lineBuf(1024*10);
 	status = adRunning;
+	CharBuffer lineBuf(1024*4);
 
-	while (!IsStopped())
+	while (!IsStopped() && !m_decoder.GetEof())
 	{
-		time_t oldTime = m_lastUpdateTime;
-		SetLastUpdateTimeNow();
-		if (oldTime != m_lastUpdateTime)
-		{
-			AddServerData();
-		}
-
-		// Throttle the bandwidth
+		// throttle the bandwidth
 		while (!IsStopped() && (g_Options->GetDownloadRate() > 0.0f) &&
 			(g_StatMeter->CalcCurrentDownloadSpeed() > g_Options->GetDownloadRate() ||
 			g_StatMeter->CalcMomentaryDownloadSpeed() > g_Options->GetDownloadRate()))
@@ -359,17 +347,17 @@ ArticleDownloader::EStatus ArticleDownloader::Download()
 			usleep(10 * 1000);
 		}
 
-		int len = 0;
-		char* line = m_connection->ReadLine(lineBuf, lineBuf.Size(), &len);
-
-		g_StatMeter->AddSpeedReading(len);
-		if (g_Options->GetAccurateRate())
+		char* buffer;
+		int len;
+		m_connection->ReadBuffer(&buffer, &len);
+		if (len == 0)
 		{
-			AddServerData();
+			len = m_connection->TryRecv(lineBuf, lineBuf.Size());
+			buffer = lineBuf;
 		}
 
-		// Have we encountered a timeout?
-		if (!line)
+		// have we encountered a timeout?
+		if (len <= 0)
 		{
 			if (!IsStopped())
 			{
@@ -379,67 +367,25 @@ ArticleDownloader::EStatus ArticleDownloader::Download()
 			break;
 		}
 
-		//detect end of article
-		if (!strcmp(line, ".\r\n") || !strcmp(line, ".\n"))
+		g_StatMeter->AddSpeedReading(len);
+		time_t oldTime = m_lastUpdateTime;
+		SetLastUpdateTimeNow();
+		if (oldTime != m_lastUpdateTime)
 		{
-			end = true;
-			break;
+			AddServerData();
 		}
 
-		//detect lines starting with "." (marked as "..")
-		if (!strncmp(line, "..", 2))
-		{
-			line++;
-			len--;
-		}
-
-		if (!body)
-		{
-			// detect body of article
-			if (*line == '\r' || *line == '\n')
-			{
-				body = true;
-			}
-			// check id of returned article
-			else if (!strncmp(line, "Message-ID: ", 12))
-			{
-				char* p = line + 12;
-				if (strncmp(p, m_articleInfo->GetMessageId(), strlen(m_articleInfo->GetMessageId())))
-				{
-					if (char* e = strrchr(p, '\r')) *e = '\0'; // remove trailing CR-character
-					detail("Article %s @ %s failed: Wrong message-id, expected %s, returned %s", *m_infoName,
-						*m_connectionName, m_articleInfo->GetMessageId(), p);
-					status = adFailed;
-					break;
-				}
-			}
-		}
-
-		if (m_format == Decoder::efUnknown && g_Options->GetDecode())
-		{
-			m_format = Decoder::DetectFormat(line, len, body);
-			if (m_format != Decoder::efUnknown)
-			{
-				// sometimes news servers misbehave and send article body without new line separator between headers and body
-				// if we found decoder signature we know the body is already arrived
-				body = true;
-			}
-		}
+		// decode article data
+		len = m_decoder.DecodeBuffer(buffer, len);
 
 		// write to output file
-		if (((body && m_format != Decoder::efUnknown) || !g_Options->GetDecode()) && !Write(line, len))
+		if (len > 0 && !Write(buffer, len))
 		{
 			status = adFatalError;
 			break;
 		}
 	}
 
-	if (!end && status == adRunning && !IsStopped())
-	{
-		detail("Article %s @ %s failed: article incomplete", *m_infoName, *m_connectionName);
-		status = adFailed;
-	}
-
 	if (IsStopped())
 	{
 		status = adFailed;
@@ -498,57 +444,47 @@ ArticleDownloader::EStatus ArticleDownloader::CheckResponse(const char* response
 	}
 }
 
-bool ArticleDownloader::Write(char* line, int len)
+bool ArticleDownloader::Write(char* buffer, int len)
 {
 	const char* articleFilename = nullptr;
 	int64 articleFileSize = 0;
 	int64 articleOffset = 0;
 	int articleSize = 0;
 
-	if (g_Options->GetDecode())
+	if (!m_writingStarted)
 	{
-		if (m_format == Decoder::efYenc)
+		if (!g_Options->GetRawArticle())
 		{
-			len = m_yDecoder.DecodeBuffer(line, len);
-			articleFilename = m_yDecoder.GetArticleFilename();
-			articleFileSize = m_yDecoder.GetSize();
-		}
-		else if (m_format == Decoder::efUx)
-		{
-			len = m_uDecoder.DecodeBuffer(line, len);
-			articleFilename = m_uDecoder.GetArticleFilename();
-		}
-		else
-		{
-			detail("Decoding %s failed: unsupported encoding", *m_infoName);
-			return false;
-		}
-
-		if (len > 0 && m_format == Decoder::efYenc)
-		{
-			if (m_yDecoder.GetBegin() == 0 || m_yDecoder.GetEnd() == 0)
+			articleFilename = m_decoder.GetArticleFilename();
+			if (m_decoder.GetFormat() == Decoder::efYenc)
 			{
-				return false;
+				if (m_decoder.GetBeginPos() == 0 || m_decoder.GetEndPos() == 0)
+				{
+					return false;
+				}
+				articleFileSize = m_decoder.GetSize();
+				articleOffset = m_decoder.GetBeginPos() - 1;
+				articleSize = (int)(m_decoder.GetEndPos() - m_decoder.GetBeginPos() + 1);
+				if (articleSize <= 0 || articleSize > 1024*1024*1024)
+				{
+					warn("Malformed article %s: size %i out of range", *m_infoName, articleSize);
+					return false;
+				}
 			}
-			articleOffset = m_yDecoder.GetBegin() - 1;
-			articleSize = (int)(m_yDecoder.GetEnd() - m_yDecoder.GetBegin() + 1);
 		}
-	}
 
-	if (!m_writingStarted && len > 0)
-	{
-		if (!m_articleWriter.Start(m_format, articleFilename, articleFileSize, articleOffset, articleSize))
+		if (!m_articleWriter.Start(m_decoder.GetFormat(), articleFilename, articleFileSize, articleOffset, articleSize))
 		{
 			return false;
 		}
 		m_writingStarted = true;
 	}
 
-	bool ok = len == 0 || m_articleWriter.Write(line, len);
+	bool ok = m_articleWriter.Write(buffer, len);
 
 	if (m_contentAnalyzer)
 	{
-		m_contentAnalyzer->Append(line, len);
+		m_contentAnalyzer->Append(buffer, len);
 	}
 
 	return ok;
@@ -556,36 +492,21 @@ bool ArticleDownloader::Write(char* line, int len)
 
 ArticleDownloader::EStatus ArticleDownloader::DecodeCheck()
 {
-	if (g_Options->GetDecode())
+	if (!g_Options->GetRawArticle())
 	{
-		Decoder* decoder = nullptr;
-		if (m_format == Decoder::efYenc)
-		{
-			decoder = &m_yDecoder;
-		}
-		else if (m_format == Decoder::efUx)
-		{
-			decoder = &m_uDecoder;
-		}
-		else
-		{
-			detail("Decoding %s failed: no binary data or unsupported encoding format", *m_infoName);
-			return adFailed;
-		}
-
-		Decoder::EStatus status = decoder->Check();
+		Decoder::EStatus status = m_decoder.Check();
 
 		if (status == Decoder::dsFinished)
 		{
-			if (decoder->GetArticleFilename())
+			if (m_decoder.GetArticleFilename())
 			{
-				m_articleFilename = decoder->GetArticleFilename();
+				m_articleFilename = m_decoder.GetArticleFilename();
 			}
 
-			if (m_format == Decoder::efYenc)
+			if (m_decoder.GetFormat() == Decoder::efYenc)
 			{
 				m_articleInfo->SetCrc(g_Options->GetCrcCheck() ?
-					m_yDecoder.GetCalculatedCrc() : m_yDecoder.GetExpectedCrc());
+					m_decoder.GetCalculatedCrc() : m_decoder.GetExpectedCrc());
 			}
 
 			return adFinished;
@@ -646,22 +567,6 @@ void ArticleDownloader::Stop()
 	debug("ArticleDownloader stopped successfully");
 }
 
-bool ArticleDownloader::Terminate()
-{
-	NntpConnection* connection = m_connection;
-	bool terminated = Kill();
-	if (terminated && connection)
-	{
-		debug("Terminating connection");
-		connection->SetSuppressErrors(true);
-		connection->Cancel();
-		connection->Disconnect();
-		g_StatMeter->AddServerData(connection->FetchTotalBytesRead(), connection->GetNewsServer()->GetId());
-		g_ServerPool->FreeConnection(connection, true);
-	}
-	return terminated;
-}
-
 void ArticleDownloader::FreeConnection(bool keepConnected)
 {
 	if (m_connection)

daemon/nntp/ArticleDownloader.h

@@ -2,7 +2,7 @@
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
  *  Copyright (C) 2004 Sven Henkel <sidddy@users.sourceforge.net>
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -56,16 +56,6 @@ public:
 		adFatalError
 	};
 
-	class ArticleWriterImpl : public ArticleWriter
-	{
-	public:
-		void SetOwner(ArticleDownloader* owner) { m_owner = owner; }
-	protected:
-		virtual void SetLastUpdateTimeNow() { m_owner->SetLastUpdateTimeNow(); }
-	private:
-		ArticleDownloader* m_owner;
-	};
-
 	ArticleDownloader();
 	virtual ~ArticleDownloader();
 	void SetFileInfo(FileInfo* fileInfo) { m_fileInfo = fileInfo; }
@@ -76,7 +66,6 @@ public:
 	ServerStatList* GetServerStats() { return &m_serverStats; }
 	virtual void Run();
 	virtual void Stop();
-	bool Terminate();
 	time_t GetLastUpdateTime() { return m_lastUpdateTime; }
 	void SetLastUpdateTimeNow();
 	const char* GetArticleFilename() { return m_articleFilename; }
@@ -101,10 +90,8 @@ private:
 	CString m_connectionName;
 	CString m_articleFilename;
 	time_t m_lastUpdateTime;
-	Decoder::EFormat m_format = Decoder::efUnknown;
-	YDecoder m_yDecoder;
-	UDecoder m_uDecoder;
-	ArticleWriterImpl m_articleWriter;
+	Decoder m_decoder;
+	ArticleWriter m_articleWriter;
 	ServerStatList m_serverStats;
 	bool m_writingStarted;
 	int m_downloadedSize = 0;
@@ -115,7 +102,7 @@ private:
 	void FreeConnection(bool keepConnected);
 	EStatus CheckResponse(const char* response, const char* comment);
 	void SetStatus(EStatus status) { m_status = status; }
-	bool Write(char* line, int len);
+	bool Write(char* buffer, int len);
 	void AddServerData();
 };
 

daemon/nntp/ArticleWriter.cpp

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2014-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2014-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -106,7 +106,7 @@ bool ArticleWriter::Start(Decoder::EFormat format, const char* filename, int64 f
 	}
 
 	// allocate cache buffer
-	if (g_Options->GetArticleCache() > 0 && g_Options->GetDecode() &&
+	if (g_Options->GetArticleCache() > 0 && !g_Options->GetRawArticle() &&
 		(!g_Options->GetDirectWrite() || m_format == Decoder::efYenc))
 	{
 		m_articleData = g_ArticleCache->Alloc(m_articleSize);
@@ -147,22 +147,31 @@ bool ArticleWriter::Start(Decoder::EFormat format, const char* filename, int64 f
 
 bool ArticleWriter::Write(char* buffer, int len)
 {
-	if (g_Options->GetDecode())
+	if (!g_Options->GetRawArticle())
 	{
 		m_articlePtr += len;
 	}
 
-	if (g_Options->GetDecode() && m_articleData.GetData())
+	if (m_articlePtr > m_articleSize)
+	{
+		// An attempt to write beyond article border is detected.
+		// That's an error condition (damaged article).
+		// We return 'false' since this isn't a fatal disk error and
+		// article size mismatch will be detected in decoder check anyway.
+		return true;
+	}
+
+	if (!g_Options->GetRawArticle() && m_articleData.GetData())
 	{
-		if (m_articlePtr > m_articleSize)
-		{
-			detail("Decoding %s failed: article size mismatch", *m_infoName);
-			return false;
-		}
 		memcpy(m_articleData.GetData() + m_articlePtr - len, buffer, len);
 		return true;
 	}
 
+	if (g_Options->GetSkipWrite())
+	{
+		return true;
+	}
+
 	return m_outFile.Write(buffer, len) > 0;
 }
 
@@ -179,7 +188,7 @@ void ArticleWriter::Finish(bool success)
 
 	bool directWrite = (g_Options->GetDirectWrite() || m_fileInfo->GetForceDirectWrite()) && m_format == Decoder::efYenc;
 
-	if (g_Options->GetDecode())
+	if (!g_Options->GetRawArticle())
 	{
 		if (!directWrite && !m_articleData.GetData())
 		{
@@ -189,10 +198,9 @@ void ArticleWriter::Finish(bool success)
 					"Could not rename file %s to %s: %s", *m_tempFilename, m_resultFilename,
 					*FileSystem::GetLastErrorMessage());
 			}
+			FileSystem::DeleteFile(m_tempFilename);
 		}
 
-		FileSystem::DeleteFile(m_tempFilename);
-
 		if (m_articleData.GetData())
 		{
 			if (m_articleSize != m_articlePtr)
@@ -224,19 +232,20 @@ void ArticleWriter::Finish(bool success)
 /* creates output file and subdirectores */
 bool ArticleWriter::CreateOutputFile(int64 size)
 {
-	if (g_Options->GetDirectWrite() && FileSystem::FileExists(m_outputFilename) &&
-		FileSystem::FileSize(m_outputFilename) == size)
+	if (FileSystem::FileExists(m_outputFilename))
 	{
-		// keep existing old file from previous program session
-		return true;
+		if (FileSystem::FileSize(m_outputFilename) == size)
+		{
+			// keep existing old file from previous program session
+			return true;
+		}
+		// delete existing old file from previous program session
+		FileSystem::DeleteFile(m_outputFilename);
 	}
 
-	// delete eventually existing old file from previous program session
-	FileSystem::DeleteFile(m_outputFilename);
-
 	// ensure the directory exist
 	BString<1024> destDir;
-	destDir.Set(m_outputFilename, FileSystem::BaseFileName(m_outputFilename) - m_outputFilename);
+	destDir.Set(m_outputFilename, (int)(FileSystem::BaseFileName(m_outputFilename) - m_outputFilename));
 	CString errmsg;
 
 	if (!FileSystem::ForceDirectories(destDir, errmsg))
@@ -305,7 +314,7 @@ void ArticleWriter::CompleteFileParts()
 
 	bool cached = m_fileInfo->GetCachedArticles() > 0;
 
-	if (!g_Options->GetDecode())
+	if (g_Options->GetRawArticle())
 	{
 		detail("Moving articles for %s", *infoFilename);
 	}
@@ -344,7 +353,7 @@ void ArticleWriter::CompleteFileParts()
 	DiskFile outfile;
 	BString<1024> tmpdestfile("%s.tmp", *ofn);
 
-	if (g_Options->GetDecode() && !directWrite)
+	if (!g_Options->GetRawArticle() && !directWrite)
 	{
 		FileSystem::DeleteFile(tmpdestfile);
 		if (!outfile.Open(tmpdestfile, DiskFile::omWrite))
@@ -364,7 +373,7 @@ void ArticleWriter::CompleteFileParts()
 		}
 		tmpdestfile = *m_outputFilename;
 	}
-	else if (!g_Options->GetDecode())
+	else if (g_Options->GetRawArticle())
 	{
 		FileSystem::DeleteFile(tmpdestfile);
 		if (!FileSystem::CreateDirectory(ofn))
@@ -392,7 +401,7 @@ void ArticleWriter::CompleteFileParts()
 		CharBuffer buffer;
 		bool firstArticle = true;
 
-		if (g_Options->GetDecode() && !directWrite)
+		if (!g_Options->GetRawArticle() && !directWrite)
 		{
 			buffer.Reserve(1024 * 64);
 		}
@@ -404,22 +413,27 @@ void ArticleWriter::CompleteFileParts()
 				continue;
 			}
 
-			if (g_Options->GetDecode() && !directWrite && pa->GetSegmentOffset() > -1 &&
+			if (!g_Options->GetRawArticle() && !directWrite && pa->GetSegmentOffset() > -1 &&
 				pa->GetSegmentOffset() > outfile.Position() && outfile.Position() > -1)
 			{
 				memset(buffer, 0, buffer.Size());
-				while (pa->GetSegmentOffset() > outfile.Position() && outfile.Position() > -1 &&
-					outfile.Write(buffer, std::min((int)(pa->GetSegmentOffset() - outfile.Position()), buffer.Size())));
+				if (!g_Options->GetSkipWrite())
+				{
+					while (pa->GetSegmentOffset() > outfile.Position() && outfile.Position() > -1 &&
+						outfile.Write(buffer, std::min((int)(pa->GetSegmentOffset() - outfile.Position()), buffer.Size())));
+				}
 			}
 
 			if (pa->GetSegmentContent())
 			{
-				outfile.Seek(pa->GetSegmentOffset());
-				outfile.Write(pa->GetSegmentContent(), pa->GetSegmentSize());
+				if (!g_Options->GetSkipWrite())
+				{
+					outfile.Seek(pa->GetSegmentOffset());
+					outfile.Write(pa->GetSegmentContent(), pa->GetSegmentSize());
+				}
 				pa->DiscardSegment();
-				SetLastUpdateTimeNow();
 			}
-			else if (g_Options->GetDecode() && !directWrite)
+			else if (!g_Options->GetRawArticle() && !directWrite && !g_Options->GetSkipWrite())
 			{
 				DiskFile infile;
 				if (pa->GetResultFilename() && infile.Open(pa->GetResultFilename(), DiskFile::omRead))
@@ -429,7 +443,6 @@ void ArticleWriter::CompleteFileParts()
 					{
 						cnt = (int)infile.Read(buffer, buffer.Size());
 						outfile.Write(buffer, cnt);
-						SetLastUpdateTimeNow();
 					}
 					infile.Close();
 				}
@@ -443,7 +456,7 @@ void ArticleWriter::CompleteFileParts()
 						(int)m_fileInfo->GetArticles()->size());
 				}
 			}
-			else if (!g_Options->GetDecode())
+			else if (g_Options->GetRawArticle())
 			{
 				BString<1024> dstFileName("%s%c%03i", *ofn, PATH_SEPARATOR, pa->GetPartNumber());
 				if (!FileSystem::MoveFile(pa->GetResultFilename(), dstFileName))
@@ -456,7 +469,7 @@ void ArticleWriter::CompleteFileParts()
 
 			if (m_format == Decoder::efYenc)
 			{
-				crc = firstArticle ? pa->GetCrc() : Util::Crc32Combine(crc, pa->GetCrc(), pa->GetSegmentSize());
+				crc = firstArticle ? pa->GetCrc() : Crc32::Combine(crc, pa->GetCrc(), pa->GetSegmentSize());
 				firstArticle = false;
 			}
 		}
@@ -492,7 +505,7 @@ void ArticleWriter::CompleteFileParts()
 		{
 			debug("Checking old dir for: %s", *m_outputFilename);
 			BString<1024> oldDestDir;
-			oldDestDir.Set(m_outputFilename, FileSystem::BaseFileName(m_outputFilename) - m_outputFilename);
+			oldDestDir.Set(m_outputFilename, (int)(FileSystem::BaseFileName(m_outputFilename) - m_outputFilename));
 			if (FileSystem::DirEmpty(oldDestDir))
 			{
 				debug("Deleting old dir: %s", *oldDestDir);
@@ -522,18 +535,6 @@ void ArticleWriter::CompleteFileParts()
 			"%i of %i article downloads failed for \"%s\"",
 			m_fileInfo->GetMissedArticles() + m_fileInfo->GetFailedArticles(),
 			m_fileInfo->GetTotalArticles(), *infoFilename);
-
-		if (g_Options->GetBrokenLog())
-		{
-			BString<1024> brokenLogName("%s%c_brokenlog.txt", *nzbDestDir, PATH_SEPARATOR);
-			DiskFile file;
-			if (file.Open(brokenLogName, DiskFile::omAppend))
-			{
-				file.Print("%s (%i/%i)%s", *filename, m_fileInfo->GetSuccessArticles(),
-					m_fileInfo->GetTotalArticles(), LINE_ENDING);
-				file.Close();
-			}
-		}
 	}
 	else
 	{
@@ -581,10 +582,18 @@ void ArticleWriter::FlushCache()
 		ArticleCache::FlushGuard flushGuard = g_ArticleCache->GuardFlush();
 
 		std::vector<ArticleInfo*> cachedArticles;
-		cachedArticles.reserve(m_fileInfo->GetArticles()->size());
 
 		{
 			Guard contentGuard = g_ArticleCache->GuardContent();
+
+			if (m_fileInfo->GetFlushLocked())
+			{
+				return;
+			}
+
+			m_fileInfo->SetFlushLocked(true);
+
+			cachedArticles.reserve(m_fileInfo->GetArticles()->size());
 			for (ArticleInfo* pa : m_fileInfo->GetArticles())
 			{
 				if (pa->GetSegmentContent())
@@ -646,7 +655,10 @@ void ArticleWriter::FlushCache()
 				outfile.Seek(pa->GetSegmentOffset());
 			}
 
-			outfile.Write(pa->GetSegmentContent(), pa->GetSegmentSize());
+			if (!g_Options->GetSkipWrite())
+			{
+				outfile.Write(pa->GetSegmentContent(), pa->GetSegmentSize());
+			}
 
 			flushedSize += pa->GetSegmentSize();
 			flushedArticles++;
@@ -671,6 +683,7 @@ void ArticleWriter::FlushCache()
 		{
 			Guard contentGuard = g_ArticleCache->GuardContent();
 			m_fileInfo->SetCachedArticles(m_fileInfo->GetCachedArticles() - flushedArticles);
+			m_fileInfo->SetFlushLocked(false);
 		}
 	}
 
@@ -714,57 +727,6 @@ bool ArticleWriter::MoveCompletedFiles(NzbInfo* nzbInfo, const char* oldDestDir)
 		}
 	}
 
-	// move brokenlog.txt
-	if (g_Options->GetBrokenLog())
-	{
-		BString<1024> oldBrokenLogName("%s%c_brokenlog.txt", oldDestDir, PATH_SEPARATOR);
-		if (FileSystem::FileExists(oldBrokenLogName))
-		{
-			BString<1024> brokenLogName("%s%c_brokenlog.txt", nzbInfo->GetDestDir(), PATH_SEPARATOR);
-
-			detail("Moving file %s to %s", *oldBrokenLogName, *brokenLogName);
-			if (FileSystem::FileExists(brokenLogName))
-			{
-				// copy content to existing new file, then delete old file
-				DiskFile outfile;
-				if (outfile.Open(brokenLogName, DiskFile::omAppend))
-				{
-					DiskFile infile;
-					if (infile.Open(oldBrokenLogName, DiskFile::omRead))
-					{
-						CharBuffer buffer(1024 * 50);
-						int cnt = buffer.Size();
-						while (cnt == buffer.Size())
-						{
-							cnt = (int)infile.Read(buffer, buffer.Size());
-							outfile.Write(buffer, cnt);
-						}
-						infile.Close();
-						FileSystem::DeleteFile(oldBrokenLogName);
-					}
-					else
-					{
-						nzbInfo->PrintMessage(Message::mkError, "Could not open file %s", *oldBrokenLogName);
-					}
-					outfile.Close();
-				}
-				else
-				{
-					nzbInfo->PrintMessage(Message::mkError, "Could not open file %s", *brokenLogName);
-				}
-			}
-			else
-			{
-				// move to new destination
-				if (!FileSystem::MoveFile(oldBrokenLogName, brokenLogName))
-				{
-					nzbInfo->PrintMessage(Message::mkError, "Could not move file %s to %s: %s",
-						*oldBrokenLogName, *brokenLogName, *FileSystem::GetLastErrorMessage());
-				}
-			}
-		}
-	}
-
 	// delete old directory (if empty)
 	if (FileSystem::DirEmpty(oldDestDir))
 	{
@@ -809,7 +771,7 @@ CachedSegmentData ArticleCache::Alloc(int size)
 		p = malloc(size);
 		if (p)
 		{
-			if (!m_allocated && g_Options->GetSaveQueue() && g_Options->GetServerMode() && g_Options->GetContinuePartial())
+			if (!m_allocated && g_Options->GetServerMode() && g_Options->GetContinuePartial())
 			{
 				g_DiskState->WriteCacheFlag();
 			}
@@ -829,6 +791,7 @@ bool ArticleCache::Realloc(CachedSegmentData* segment, int newSize)
 	{
 		m_allocated += newSize - segment->m_size;
 		segment->m_size = newSize;
+		segment->m_data = (char*)p;
 	}
 
 	return p;
@@ -842,7 +805,7 @@ void ArticleCache::Free(CachedSegmentData* segment)
 
 		Guard guard(m_allocMutex);
 		m_allocated -= segment->m_size;
-		if (!m_allocated && g_Options->GetSaveQueue() && g_Options->GetServerMode() && g_Options->GetContinuePartial())
+		if (!m_allocated && g_Options->GetServerMode() && g_Options->GetContinuePartial())
 		{
 			g_DiskState->DeleteCacheFlag();
 		}

daemon/nntp/ArticleWriter.h

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2014-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2014-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -60,9 +60,6 @@ public:
 	static bool MoveCompletedFiles(NzbInfo* nzbInfo, const char* oldDestDir);
 	void FlushCache();
 
-protected:
-	virtual void SetLastUpdateTimeNow() {}
-
 private:
 	FileInfo* m_fileInfo;
 	ArticleInfo* m_articleInfo;

daemon/nntp/Decoder.cpp

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -22,22 +22,129 @@
 #include "Decoder.h"
 #include "Log.h"
 #include "Util.h"
+#include "YEncode.h"
 
-const char* Decoder::FormatNames[] = { "Unknown", "yEnc", "UU" };
+Decoder::Decoder()
+{
+	debug("%s", YEncode::decode_simd ? "SIMD yEnc decoder can be used" : "SIMD yEnc decoder isn't available for this CPU");
+	debug("%s", YEncode::crc_simd ? "SIMD Crc routine can be used" : "SIMD Crc routine isn't available for this CPU");
+
+	Clear();
+}
 
 void Decoder::Clear()
 {
 	m_articleFilename.Clear();
+	m_body = false;
+	m_begin = false;
+	m_part = false;
+	m_end = false;
+	m_crc = false;
+	m_eof = false;
+	m_expectedCRC = 0;
+	m_crc32.Reset();
+	m_beginPos = 0;
+	m_endPos = 0;
+	m_size = 0;
+	m_endSize = 0;
+	m_outSize = 0;
+	m_state = 0;
+	m_crcCheck = false;
+	m_lineBuf.Reserve(1024*8);
+	m_lineBuf.SetLength(0);
 }
 
-Decoder::EFormat Decoder::DetectFormat(const char* buffer, int len, bool inBody)
+/* At the beginning of article the processing goes line by line to find '=ybegin'-marker.
+ * Once the yEnc-data is started switches to blockwise processing.
+ * At the end of yEnc-data switches back to line by line mode to
+ * process '=yend'-marker and EOF-marker.
+ * UU-encoded articles are processed completely in line by line mode.
+ */
+int Decoder::DecodeBuffer(char* buffer, int len)
+{
+	if (m_rawMode)
+	{
+		ProcessRaw(buffer, len);
+		return len;
+	}
+
+	int outlen = 0;
+
+	if (m_body && m_format == efYenc)
+	{
+		outlen = DecodeYenc(buffer, buffer, len);
+		if (m_body)
+		{
+			return outlen;
+		}
+	}
+	else
+	{
+		m_lineBuf.Append(buffer, len);
+	}
+
+	char* line = (char*)m_lineBuf;
+	while (char* end = strchr(line, '\n'))
+	{
+		int llen = (int)(end - line + 1);
+
+		if (line[0] == '.' && line[1] == '\r')
+		{
+			m_eof = true;
+			m_lineBuf.SetLength(0);
+			return outlen;
+		}
+
+		if (m_format == efUnknown)
+		{
+			m_format = DetectFormat(line, llen);
+		}
+
+		if (m_format == efYenc)
+		{
+			ProcessYenc(line, llen);
+			if (m_body)
+			{
+				outlen = DecodeYenc(end + 1, buffer, m_lineBuf.Length() - (int)(end + 1 - m_lineBuf));
+				if (m_body)
+				{
+					m_lineBuf.SetLength(0);
+					return outlen;
+				}
+				line = (char*)m_lineBuf;
+				continue;
+			}
+		}
+		else if (m_format == efUx)
+		{
+			outlen += DecodeUx(line, llen);
+		}
+
+		line = end + 1;
+	}
+
+	if (*line)
+	{
+		len = m_lineBuf.Length() - (int)(line - m_lineBuf);
+		memmove((char*)m_lineBuf, line, len);
+		m_lineBuf.SetLength(len);
+	}
+	else
+	{
+		m_lineBuf.SetLength(0);
+	}
+
+	return outlen;
+}
+
+Decoder::EFormat Decoder::DetectFormat(const char* buffer, int len)
 {
 	if (!strncmp(buffer, "=ybegin ", 8))
 	{
 		return efYenc;
 	}
 
-	if (inBody && (len == 62 || len == 63) && (buffer[62] == '\n' || buffer[62] == '\r') && *buffer == 'M')
+	if ((len == 62 || len == 63) && (buffer[62] == '\n' || buffer[62] == '\r') && *buffer == 'M')
 	{
 		return efUx;
 	}
@@ -64,139 +171,122 @@ Decoder::EFormat Decoder::DetectFormat(const char* buffer, int len, bool inBody)
 	return efUnknown;
 }
 
-/**
-  * YDecoder: fast implementation of yEnc-Decoder
-  */
-
-YDecoder::YDecoder()
+void Decoder::ProcessYenc(char* buffer, int len)
 {
-	Clear();
-}
-
-void YDecoder::Clear()
-{
-	Decoder::Clear();
-
-	m_body = false;
-	m_begin = false;
-	m_part = false;
-	m_end = false;
-	m_crc = false;
-	m_expectedCRC = 0;
-	m_calculatedCRC = 0xFFFFFFFF;
-	m_beginPos = 0;
-	m_endPos = 0;
-	m_size = 0;
-	m_endSize = 0;
-	m_crcCheck = false;
-}
-
-int YDecoder::DecodeBuffer(char* buffer, int len)
-{
-	if (m_body && !m_end)
+	if (!strncmp(buffer, "=ybegin ", 8))
 	{
-		if (!strncmp(buffer, "=yend ", 6))
+		m_begin = true;
+		char* pb = strstr(buffer, " name=");
+		if (pb)
 		{
-			m_end = true;
-			char* pb = strstr(buffer, m_part ? " pcrc32=" : " crc32=");
-			if (pb)
-			{
-				m_crc = true;
-				pb += 7 + (int)m_part; //=strlen(" crc32=") or strlen(" pcrc32=")
-				m_expectedCRC = strtoul(pb, nullptr, 16);
-			}
-			pb = strstr(buffer, " size=");
-			if (pb)
-			{
-				pb += 6; //=strlen(" size=")
-				m_endSize = (int64)atoll(pb);
-			}
-			return 0;
+			pb += 6; //=strlen(" name=")
+			char* pe;
+			for (pe = pb; *pe != '\0' && *pe != '\n' && *pe != '\r'; pe++);
+			m_articleFilename = WebUtil::Latin1ToUtf8(CString(pb, (int)(pe - pb)));
 		}
-
-		char* iptr = buffer;
-		char* optr = buffer;
-		while (true)
+		pb = strstr(buffer, " size=");
+		if (pb)
 		{
-			switch (*iptr)
-			{
-				case '=':	//escape-sequence
-					iptr++;
-					*optr = *iptr - 64 - 42;
-					optr++;
-					break;
-				case '\n':	// ignored char
-				case '\r':	// ignored char
-					break;
-				case '\0':
-					goto BreakLoop;
-				default:	// normal char
-					*optr = *iptr - 42;
-					optr++;
-					break;
-			}
-			iptr++;
+			pb += 6; //=strlen(" size=")
+			m_size = (int64)atoll(pb);
 		}
-BreakLoop:
-
-		if (m_crcCheck)
+		m_part = strstr(buffer, " part=");
+		if (!m_part)
 		{
-			m_calculatedCRC = Util::Crc32m(m_calculatedCRC, (uchar *)buffer, (uint32)(optr - buffer));
-		}
-		return optr - buffer;
-	}
-	else
-	{
-		if (!m_part && !strncmp(buffer, "=ybegin ", 8))
-		{
-			m_begin = true;
-			char* pb = strstr(buffer, " name=");
-			if (pb)
-			{
-				pb += 6; //=strlen(" name=")
-				char* pe;
-				for (pe = pb; *pe != '\0' && *pe != '\n' && *pe != '\r'; pe++) ;
-				m_articleFilename = WebUtil::Latin1ToUtf8(CString(pb, pe - pb));
-			}
-			pb = strstr(buffer, " size=");
-			if (pb)
-			{
-				pb += 6; //=strlen(" size=")
-				m_size = (int64)atoll(pb);
-			}
-			m_part = strstr(buffer, " part=");
-			if (!m_part)
-			{
-				m_body = true;
-				m_beginPos = 1;
-				m_endPos = m_size;
-			}
-		}
-		else if (m_part && !strncmp(buffer, "=ypart ", 7))
-		{
-			m_part = true;
 			m_body = true;
-			char* pb = strstr(buffer, " begin=");
-			if (pb)
-			{
-				pb += 7; //=strlen(" begin=")
-				m_beginPos = (int64)atoll(pb);
-			}
-			pb = strstr(buffer, " end=");
-			if (pb)
-			{
-				pb += 5; //=strlen(" end=")
-				m_endPos = (int64)atoll(pb);
-			}
+			m_beginPos = 1;
+			m_endPos = m_size;
+		}
+	}
+	else if (!strncmp(buffer, "=ypart ", 7))
+	{
+		m_part = true;
+		m_body = true;
+		char* pb = strstr(buffer, " begin=");
+		if (pb)
+		{
+			pb += 7; //=strlen(" begin=")
+			m_beginPos = (int64)atoll(pb);
+		}
+		pb = strstr(buffer, " end=");
+		if (pb)
+		{
+			pb += 5; //=strlen(" end=")
+			m_endPos = (int64)atoll(pb);
+		}
+	}
+	else if (!strncmp(buffer, "=yend ", 6))
+	{
+		m_end = true;
+		char* pb = strstr(buffer, m_part ? " pcrc32=" : " crc32=");
+		if (pb)
+		{
+			m_crc = true;
+			pb += 7 + (int)m_part; //=strlen(" crc32=") or strlen(" pcrc32=")
+			m_expectedCRC = strtoul(pb, nullptr, 16);
+		}
+		pb = strstr(buffer, " size=");
+		if (pb)
+		{
+			pb += 6; //=strlen(" size=")
+			m_endSize = (int64)atoll(pb);
 		}
 	}
-
-	return 0;
 }
 
-Decoder::EStatus YDecoder::Check()
+int Decoder::DecodeYenc(char* buffer, char* outbuf, int len)
 {
-	m_calculatedCRC ^= 0xFFFFFFFF;
+	const unsigned char* src = (unsigned char*)buffer;
+	unsigned char* dst = (unsigned char*)outbuf;
+
+	int endseq = YEncode::decode(&src, &dst, len, (YEncode::YencDecoderState*)&m_state);
+	int outlen = (int)((char*)dst - outbuf);
+
+	// endseq:
+	//   0: no end sequence found
+	//   1: \r\n=y sequence found, src points to byte after 'y'
+	//   2: \r\n.\r\n sequence found, src points to byte after last '\n'
+	if (endseq != 0)
+	{
+		// switch back to line mode to process '=yend'- or eof- marker
+		m_lineBuf.SetLength(0);
+		m_lineBuf.Append(endseq == 1 ? "=y" : ".\r\n");
+		int rem = len - (int)((const char*)src - buffer);
+		if (rem > 0)
+		{
+			m_lineBuf.Append((const char*)src, rem);
+		}
+		m_body = false;
+	}
+
+	if (m_crcCheck)
+	{
+		m_crc32.Append((uchar*)outbuf, (uint32)outlen);
+	}
+
+	m_outSize += outlen;
+
+	return outlen;
+}
+
+Decoder::EStatus Decoder::Check()
+{
+	switch (m_format)
+	{
+		case efYenc:
+			return CheckYenc();
+			 
+		case efUx:
+			return CheckUx();
+
+		default:
+			return dsUnknownError;
+	}
+}
+
+Decoder::EStatus Decoder::CheckYenc()
+{
+	m_calculatedCRC = m_crc32.Finish();
 
 	debug("Expected crc32=%x", m_expectedCRC);
 	debug("Calculated crc32=%x", m_calculatedCRC);
@@ -209,7 +299,7 @@ Decoder::EStatus YDecoder::Check()
 	{
 		return dsArticleIncomplete;
 	}
-	else if (!m_part && m_size != m_endSize)
+	else if ((!m_part && m_size != m_endSize) || (m_endSize != m_outSize))
 	{
 		return dsInvalidSize;
 	}
@@ -222,24 +312,7 @@ Decoder::EStatus YDecoder::Check()
 }
 
 
-/**
-  * UDecoder: supports UU encoding formats
-  */
-
-UDecoder::UDecoder()
-{
-	Clear();
-}
-
-void UDecoder::Clear()
-{
-	Decoder::Clear();
-
-	m_body = false;
-	m_end = false;
-}
-
-/* DecodeBuffer-function uses portions of code from tool UUDECODE by Clem Dye
+/* DecodeUx-function uses portions of code from tool UUDECODE by Clem Dye
  * UUDECODE.c (http://www.bastet.com/uue.zip)
  * Copyright (C) 1998 Clem Dye
  *
@@ -248,7 +321,7 @@ void UDecoder::Clear()
 
 #define UU_DECODE_CHAR(c) (c == '`' ? 0 : (((c) - ' ') & 077))
 
-int UDecoder::DecodeBuffer(char* buffer, int len)
+int Decoder::DecodeUx(char* buffer, int len)
 {
 	if (!m_body)
 	{
@@ -264,7 +337,7 @@ int UDecoder::DecodeBuffer(char* buffer, int len)
 			// extracting filename
 			char* pe;
 			for (pe = pb; *pe != '\0' && *pe != '\n' && *pe != '\r'; pe++) ;
-			m_articleFilename = WebUtil::Latin1ToUtf8(CString(pb, pe - pb));
+			m_articleFilename = WebUtil::Latin1ToUtf8(CString(pb, (int)(pe - pb)));
 
 			m_body = true;
 			return 0;
@@ -312,13 +385,13 @@ int UDecoder::DecodeBuffer(char* buffer, int len)
 			}
 		}
 
-		return optr - buffer;
+		return (int)(optr - buffer);
 	}
 
 	return 0;
 }
 
-Decoder::EStatus UDecoder::Check()
+Decoder::EStatus Decoder::CheckUx()
 {
 	if (!m_body)
 	{
@@ -327,3 +400,50 @@ Decoder::EStatus UDecoder::Check()
 
 	return dsFinished;
 }
+
+void Decoder::ProcessRaw(char* buffer, int len)
+{
+	switch (m_state)
+	{
+		case 1:
+			m_eof = len >= 4 && buffer[0] == '\n' &&
+				buffer[1] == '.' && buffer[2] == '\r' && buffer[3] == '\n';
+			break;
+
+		case 2:
+			m_eof = len >= 3 && buffer[0] == '.' && buffer[1] == '\r' && buffer[2] == '\n';
+			break;
+
+		case 3:
+			m_eof = len >= 2 && buffer[0] == '\r' && buffer[1] == '\n';
+			break;
+
+		case 4:
+			m_eof = len >= 1 && buffer[0] == '\n';
+			break;
+	}
+
+	m_eof |= len >= 5 && strstr(buffer, "\r\n.\r\n");
+
+	if (len >= 4 && buffer[len-4] == '\r' && buffer[len-3] == '\n' &&
+		buffer[len-2] == '.' && buffer[len-1] == '\r')
+	{
+		m_state = 4;
+	}
+	else if (len >= 3 && buffer[len-3] == '\r' && buffer[len-2] == '\n' && buffer[len-1] == '.')
+	{
+		m_state = 3;
+	}
+	else if (len >= 2 && buffer[len-2] == '\r' && buffer[len-1] == '\n')
+	{
+		m_state = 2;
+	}
+	else if (len >= 1 && buffer[len-1] == '\r')
+	{
+		m_state = 1;
+	}
+	else
+	{
+		m_state = 0;
+	}
+}

daemon/nntp/Decoder.h

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -22,6 +22,7 @@
 #define DECODER_H
 
 #include "NString.h"
+#include "Util.h"
 
 class Decoder
 {
@@ -43,37 +44,26 @@ public:
 		efUx,
 	};
 
-	static const char* FormatNames[];
-
-	virtual ~Decoder() {}
-	virtual EStatus Check() = 0;
-	virtual void Clear();
-	virtual int DecodeBuffer(char* buffer, int len) = 0;
-	const char* GetArticleFilename() { return m_articleFilename; }
-	static EFormat DetectFormat(const char* buffer, int len, bool inBody);
-
-protected:
-	CString m_articleFilename;
-};
-
-class YDecoder: public Decoder
-{
-public:
-	YDecoder();
-	virtual EStatus Check();
-	virtual void Clear();
-	virtual int DecodeBuffer(char* buffer, int len);
+	Decoder();
+	EStatus Check();
+	void Clear();
+	int DecodeBuffer(char* buffer, int len);
 	void SetCrcCheck(bool crcCheck) { m_crcCheck = crcCheck; }
-	int64 GetBegin() { return m_beginPos; }
-	int64 GetEnd() { return m_endPos; }
+	void SetRawMode(bool rawMode) { m_rawMode = rawMode; }
+	EFormat GetFormat() { return m_format; }
+	int64 GetBeginPos() { return m_beginPos; }
+	int64 GetEndPos() { return m_endPos; }
 	int64 GetSize() { return m_size; }
 	uint32 GetExpectedCrc() { return m_expectedCRC; }
 	uint32 GetCalculatedCrc() { return m_calculatedCRC; }
+	bool GetEof() { return m_eof; }
+	const char* GetArticleFilename() { return m_articleFilename; }
 
-private:
+private: 
+	EFormat m_format = efUnknown;
 	bool m_begin;
 	bool m_part;
-	bool m_body;
+	bool m_body; 
 	bool m_end;
 	bool m_crc;
 	uint32 m_expectedCRC;
@@ -82,20 +72,22 @@ private:
 	int64 m_endPos;
 	int64 m_size;
 	int64 m_endSize;
+	int64 m_outSize;
+	bool m_eof;
 	bool m_crcCheck;
-};
+	char m_state;
+	bool m_rawMode = false;
+	CString m_articleFilename;
+	StringBuilder m_lineBuf;
+	Crc32 m_crc32;
 
-class UDecoder: public Decoder
-{
-public:
-	UDecoder();
-	virtual EStatus Check();
-	virtual void Clear();
-	virtual int DecodeBuffer(char* buffer, int len);
-
-private:
-	bool m_body;
-	bool m_end;
+	EFormat DetectFormat(const char* buffer, int len);
+	void ProcessYenc(char* buffer, int len);
+	int DecodeYenc(char* buffer, char* outbuf, int len);
+	EStatus CheckYenc();
+	int DecodeUx(char* buffer, int len);
+	EStatus CheckUx();
+	void ProcessRaw(char* buffer, int len);
 };
 
 #endif

daemon/nntp/NntpConnection.cpp

@@ -227,10 +227,7 @@ bool NntpConnection::Disconnect()
 {
 	if (m_status == csConnected)
 	{
-		if (!m_broken)
-		{
-			Request("quit\r\n");
-		}
+		Request("quit\r\n");
 		m_activeGroup = nullptr;
 	}
 	return Connection::Disconnect();

daemon/nntp/StatMeter.cpp

@@ -308,8 +308,6 @@ void StatMeter::AddSpeedReading(int bytes)
 	time_t curTime = Util::CurrentTime();
 	int nowSlot = (int)curTime / SPEEDMETER_SLOTSIZE;
 
-	Guard guard(g_Options->GetAccurateRate() ? &m_speedMutex : nullptr);
-
 	if (curTime != m_curSecTime)
 	{
 		m_curSecTime =	curTime;
@@ -490,7 +488,7 @@ void StatMeter::CalcQuotaUsage(int64& monthBytes, int64& dayBytes)
 	ServerVolume totalVolume = m_serverVolumes[0];
 
 	time_t locTime = Util::CurrentTime() + g_Options->GetLocalTimeOffset();
-	int daySlot = locTime / 86400 - totalVolume.GetFirstDay();
+	int daySlot = (int)(locTime / 86400) - totalVolume.GetFirstDay();
 
 	dayBytes = 0;
 	if (daySlot < (int)totalVolume.BytesPerDays()->size())
@@ -534,7 +532,7 @@ int StatMeter::CalcMonthSlots(ServerVolume& volume)
 			dayparts.tm_mon++;
 			prevMonth = Util::Timegm(&dayparts);
 		}
-		elapsedDays = (locCurTime - prevMonth) / 60 / 60 / 24 + 1;
+		elapsedDays = (int)(locCurTime - prevMonth) / 60 / 60 / 24 + 1;
 	}
 	else
 	{

daemon/nntp/StatMeter.h

@@ -106,7 +106,6 @@ private:
 	int m_speedBytesIndex;
 	int m_curSecBytes;
 	time_t m_curSecTime;
-	Mutex m_speedMutex;
 
 	// time
 	int64 m_allBytes = 0;

daemon/nserv/NServMain.cpp

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2016-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -44,6 +44,10 @@ struct NServOpts
 	bool generateNzb;
 	int segmentSize;
 	bool quit;
+	int latency;
+	int speed;
+	bool memCache;
+	bool paramError;
 
 	NServOpts(int argc, char* argv[], Options::CmdOptList& cmdOpts);
 };
@@ -59,7 +63,7 @@ int NServMain(int argc, char* argv[])
 	Options::CmdOptList cmdOpts;
 	NServOpts opts(argc, argv, cmdOpts);
 
-	if (opts.dataDir.Empty())
+	if (opts.dataDir.Empty() || opts.paramError)
 	{
 		NServPrintUsage(argv[0]);
 		return 1;
@@ -85,6 +89,10 @@ int NServMain(int argc, char* argv[])
 	TlsSocket::Init();
 #endif
 
+#ifndef WIN32
+	signal(SIGPIPE, SIG_IGN);
+#endif
+
 	NServFrontend frontend;
 	frontend.Start();
 
@@ -105,11 +113,13 @@ int NServMain(int argc, char* argv[])
 	}
 
 	std::vector<std::unique_ptr<NntpServer>> instances;
+	NntpCache cache;
 
 	for (int i = 0; i < opts.instances; i++)
 	{
 		instances.emplace_back(std::make_unique<NntpServer>(i + 1, opts.bindAddress,
-			opts.firstPort + i, opts.secureCert, opts.secureKey, opts.dataDir, opts.cacheDir));
+			opts.firstPort + i, opts.secureCert, opts.secureKey, opts.dataDir, opts.cacheDir,
+			opts.latency, opts.speed, opts.memCache ? &cache : nullptr));
 		instances.back()->Start();
 	}
 
@@ -143,12 +153,15 @@ void NServPrintUsage(const char* com)
 		"    -d <data-dir>   - directory whose files will be served\n"
 		"  Optional switches:\n"
 		"    -c <cache-dir>  - directory to store encoded articles\n"
+		"    -m              - in-memory cache (unlimited, use with care)\n"
 		"    -l <log-file>   - write into log-file (disabled by default)\n"
 		"    -i <instances>  - number of server instances (default is 1)\n"
 		"    -b <address>    - ip address to bind to (default is 0.0.0.0)\n"
 		"    -p <port>       - port number for the first instance (default is 6791)\n"
 		"    -s <cert> <key> - paths to SSL certificate and key files\n"
 		"    -v <verbose>    - verbosity level 0..3 (default is 2)\n"
+		"    -w <msec>       - response latency (in milliseconds)\n"
+		"    -r <KB/s>       - speed throttling (in kilobytes per second)\n"
 		"    -z <seg-size>   - generate nzbs for all files in data-dir (size in bytes)\n"
 		"    -q              - quit after generating nzbs (in combination with -z)\n"
 		, FileSystem::BaseFileName(com));
@@ -162,9 +175,13 @@ NServOpts::NServOpts(int argc, char* argv[], Options::CmdOptList& cmdOpts)
 	generateNzb = false;
 	segmentSize = 500000;
 	quit = false;
+	latency = 0;
+	memCache = false;
+	speed = 0;
+	paramError = false;
 	int verbosity = 2;
 
-	char short_options[] = "b:c:d:l:p:i:s:v:z:q";
+	char short_options[] = "b:c:d:l:p:i:ms:v:w:r:z:q";
 
 	optind = 2;
 	while (true)
@@ -181,6 +198,10 @@ NServOpts::NServOpts(int argc, char* argv[], Options::CmdOptList& cmdOpts)
 				cacheDir = optind > argc ? nullptr : argv[optind - 1];
 				break;
 
+			case 'm':
+				memCache = true;
+				break;
+
 			case 'l':
 				logFile = optind > argc ? nullptr : argv[optind - 1];
 				break;
@@ -207,6 +228,14 @@ NServOpts::NServOpts(int argc, char* argv[], Options::CmdOptList& cmdOpts)
 				verbosity = atoi(optind > argc ? "1" : argv[optind - 1]);
 				break;
 
+			case 'w':
+				latency = atoi(optind > argc ? "0" : argv[optind - 1]);
+				break;
+
+			case 'r':
+				speed = atoi(optind > argc ? "0" : argv[optind - 1]);
+				break;
+
 			case 'z':
 				generateNzb = true;
 				segmentSize = atoi(optind > argc ? "500000" : argv[optind - 1]);
@@ -218,6 +247,11 @@ NServOpts::NServOpts(int argc, char* argv[], Options::CmdOptList& cmdOpts)
 		}
 	}
 
+	if (optind < argc)
+	{
+		paramError = true;
+	}
+
 	if (logFile.Empty())
 	{
 		cmdOpts.push_back("WriteLog=none");

daemon/nserv/NntpServer.cpp

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2016-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -28,9 +28,10 @@ class NntpProcessor : public Thread
 {
 public:
 	NntpProcessor(int id, int serverId, const char* dataDir, const char* cacheDir,
-		const char* secureCert, const char* secureKey) :
+		const char* secureCert, const char* secureKey, int latency, int speed, NntpCache* cache) :
 		m_id(id), m_serverId(serverId), m_dataDir(dataDir), m_cacheDir(cacheDir),
-		m_secureCert(secureCert), m_secureKey(secureKey) {}
+		m_secureCert(secureCert), m_secureKey(secureKey), m_latency(latency),
+		m_speed(speed), m_cache(cache) {}
 	~NntpProcessor() { m_connection->Disconnect(); }
 	virtual void Run();
 	void SetConnection(std::unique_ptr<Connection>&& connection) { m_connection = std::move(connection); }
@@ -39,6 +40,8 @@ private:
 	int m_id;
 	int m_serverId;
 	std::unique_ptr<Connection> m_connection;
+	int m_latency;
+	int m_speed;
 	const char* m_dataDir;
 	const char* m_cacheDir;
 	const char* m_secureCert;
@@ -49,18 +52,29 @@ private:
 	int64 m_offset;
 	int m_size;
 	bool m_sendHeaders;
+	int64 m_start;
+	NntpCache* m_cache;
 
 	void ServArticle();
 	void SendSegment();
 	bool ServerInList(const char* servList);
+	void SendData(const char* buffer, int size);
 };
 
+
 void NntpServer::Run()
 {
 	debug("Entering NntpServer-loop");
 
 	info("Listening on port %i", m_port);
 
+#ifdef WIN32
+	if (m_speed > 0)
+	{
+		timeBeginPeriod(1);
+	}
+#endif
+
 	int num = 1;
 
 	while (!IsStopped())
@@ -93,8 +107,8 @@ void NntpServer::Run()
 			continue;
 		}
 		
-		NntpProcessor* commandThread = new NntpProcessor(num++, m_id,
-			m_dataDir, m_cacheDir, m_secureCert, m_secureKey);
+		NntpProcessor* commandThread = new NntpProcessor(num++, m_id, m_dataDir,
+			m_cacheDir, m_secureCert, m_secureKey, m_latency, m_speed, m_cache);
 		commandThread->SetAutoDestroy(true);
 		commandThread->SetConnection(std::move(acceptedConnection));
 		commandThread->Start();
@@ -134,6 +148,7 @@ void NntpProcessor::Run()
 	}
 #endif
 
+	info("[%i] Incoming connection from: %s", m_id, m_connection->GetHost() );
 	m_connection->WriteLine("200 Welcome (NServ)\r\n");
 
 	CharBuffer buf(1024);
@@ -199,6 +214,11 @@ void NntpProcessor::ServArticle()
 {
 	detail("[%i] Serving: %s", m_id, m_messageid);
 
+	if (m_latency)
+	{
+		usleep(1000 * m_latency);
+	}
+
 	bool ok = false;
 
 	const char* from = strchr(m_messageid, '?');
@@ -209,7 +229,7 @@ void NntpProcessor::ServArticle()
 
 	if (from && off && to && end)
 	{
-		m_filename.Set(m_messageid + 1, from - m_messageid - 1);
+		m_filename.Set(m_messageid + 1, (int)(from - m_messageid - 1));
 		m_part = atoi(from + 1);
 		m_offset = atoll(off + 1);
 		m_size = atoi(to + 1);
@@ -250,14 +270,32 @@ void NntpProcessor::SendSegment()
 {
 	detail("[%i] Sending segment %s (%i=%lli:%i)", m_id, *m_filename, m_part, (long long)m_offset, m_size);
 
+	if (m_speed > 0)
+	{
+		m_start = Util::GetCurrentTicks();
+	}
+
 	BString<1024> fullFilename("%s/%s", m_dataDir, *m_filename);
 	BString<1024> cacheFileDir("%s/%s", m_cacheDir, *m_filename);
 	BString<1024> cacheFileName("%i=%lli-%i", m_part, (long long)m_offset, m_size);
 	BString<1024> cacheFullFilename("%s/%s", *cacheFileDir, *cacheFileName);
+	BString<1024> cacheKey("%s/%s", *m_filename, *cacheFileName);
+
+	const char* cachedData = nullptr;
+	int cachedSize;
+	if (m_cache)
+	{
+		m_cache->Find(cacheKey, cachedData, cachedSize);
+	}
 
 	DiskFile cacheFile;
-	bool readCache = m_cacheDir && cacheFile.Open(cacheFullFilename, DiskFile::omRead);
-	bool writeCache = m_cacheDir && !readCache;
+	bool readCache = !cachedData && m_cacheDir && cacheFile.Open(cacheFullFilename, DiskFile::omRead);
+	bool writeCache = !cachedData && m_cacheDir && !readCache;
+	StringBuilder cacheMem;
+	if (m_cache && !cachedData)
+	{
+		cacheMem.Reserve((int)(m_size * 1.1));
+	}
 
 	CString errmsg;
 	if (writeCache && !FileSystem::ForceDirectories(cacheFileDir, errmsg))
@@ -270,23 +308,27 @@ void NntpProcessor::SendSegment()
 		error("Could not create file %s: %s", *cacheFullFilename, *FileSystem::GetLastErrorMessage());
 	}
 
-	if (!readCache && !FileSystem::FileExists(fullFilename))
+	if (!cachedData && !readCache && !FileSystem::FileExists(fullFilename))
 	{
 		m_connection->WriteLine(CString::FormatStr("430 Article not found\r\n"));
 		return;
 	}
 
 	YEncoder encoder(fullFilename, m_part, m_offset, m_size, 
-		[con = m_connection.get(), writeCache, &cacheFile](const char* buf, int size)
+		[proc = this, writeCache, &cacheFile, &cacheMem](const char* buf, int size)
 		{
+			if (proc->m_cache)
+			{
+				cacheMem.Append(buf);
+			}
 			if (writeCache)
 			{
 				cacheFile.Write(buf, size);
 			}
-			con->Send(buf, size);
+			proc->SendData(buf, size);
 		});
 
-	if (!readCache && !encoder.OpenFile(errmsg))
+	if (!cachedData && !readCache && !encoder.OpenFile(errmsg))
 	{
 		m_connection->WriteLine(CString::FormatStr("403 %s\r\n", *errmsg));
 		return;
@@ -300,7 +342,11 @@ void NntpProcessor::SendSegment()
 		m_connection->WriteLine("\r\n");
 	}
 
-	if (readCache)
+	if (cachedData)
+	{
+		SendData(cachedData, cachedSize);
+	}
+	else if (readCache)
 	{
 		cacheFile.Seek(0, DiskFile::soEnd);
 		int size = (int)cacheFile.Position();
@@ -310,12 +356,88 @@ void NntpProcessor::SendSegment()
 		{
 			error("Could not read file %s: %s", *cacheFullFilename, *FileSystem::GetLastErrorMessage());
 		}
-		m_connection->Send(buf, size);
+		if (m_cache)
+		{
+			cacheMem.Append(buf, size);
+		}
+		SendData(buf, size);
 	}
 	else
 	{
 		encoder.WriteSegment();
 	}
 
+	if (!cachedData && cacheMem.Length() > 0)
+	{
+		m_cache->Append(cacheKey, cacheMem, cacheMem.Length());
+	}
+
 	m_connection->WriteLine(".\r\n");
 }
+
+void NntpProcessor::SendData(const char* buffer, int size)
+{
+	if (m_speed == 0)
+	{
+		m_connection->Send(buffer, size);
+		return;
+	}
+
+	int64 expectedTime = (int64)1000 * size / (m_speed * 1024) - (Util::GetCurrentTicks() - m_start) / 1000;
+
+	int chunkNum = 21;
+	int chunkSize = size;
+	int pause = 0;
+
+	while (pause < 1 && chunkNum > 1)
+	{
+		chunkNum--;
+		chunkSize = size / chunkNum;
+		pause = (int)(expectedTime / chunkNum);
+	}
+
+	int sent = 0;
+	for (int i = 0; i < chunkNum; i++)
+	{
+		int len = sent + chunkSize < size ? chunkSize : size - sent;
+
+		while (sent + len < size && *(buffer + sent + len) != '\r')
+		{
+			len++;
+		}
+		
+		m_connection->Send(buffer + sent, len);
+		int64 now = Util::GetCurrentTicks();
+		if (now + pause * 1000 < m_start + expectedTime * 1000)
+		{
+			usleep(pause * 1000);
+		}
+		sent += len;
+	}
+}
+
+
+void NntpCache::Append(const char* key, const char* data, int len)
+{
+	Guard guard(m_lock);
+	if (!len)
+	{
+		len = strlen(data);
+	}
+	m_items.emplace(key, std::make_unique<CacheItem>(key, data, len));
+}
+
+bool NntpCache::Find(const char* key, const char*& data, int& size)
+{
+	Guard guard(m_lock);
+
+	CacheMap::iterator pos = m_items.find(key);
+	if (pos != m_items.end())
+	{
+		data = (*pos).second->m_data;
+		size = (*pos).second->m_size;
+		return true;
+	}
+
+	return false;
+}

daemon/nserv/NntpServer.h

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2016-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -23,14 +23,41 @@
 
 #include "Thread.h"
 #include "Connection.h"
+#include "Util.h"
+
+class NntpCache
+{
+public:
+	void Append(const char* key, const char* data, int len = 0);
+	bool Find(const char* key, const char*& data, int& size);
+
+private:
+	class CacheItem
+	{
+	public:
+		CacheItem(const char* key, const char* data, int size) :
+			m_key(key), m_data(data), m_size(size) {}
+
+		CString m_key;
+		CString m_data;
+		int m_size = 0;
+	};
+
+	typedef std::unordered_map<std::string, std::unique_ptr<CacheItem>> CacheMap;
+
+	CacheMap m_items;
+	Mutex m_lock;
+};
 
 class NntpServer : public Thread
 {
 public:
 	NntpServer(int id, const char* host, int port, const char* secureCert,
-		const char* secureKey, const char* dataDir, const char* cacheDir) :
+		const char* secureKey, const char* dataDir, const char* cacheDir,
+		int latency, int speed, NntpCache* cache) :
 		m_id(id), m_host(host), m_port(port), m_secureCert(secureCert),
-		m_secureKey(secureKey), m_dataDir(dataDir), m_cacheDir(cacheDir) {}
+		m_secureKey(secureKey), m_dataDir(dataDir), m_cacheDir(cacheDir),
+		m_latency(latency), m_speed(speed), m_cache(cache) {}
 	virtual void Run();
 	virtual void Stop();
 
@@ -38,11 +65,14 @@ private:
 	int m_id;
 	CString m_host;
 	int m_port;
+	int m_latency;
+	int m_speed;
 	CString m_dataDir;
 	CString m_cacheDir;
 	CString m_secureCert;
 	CString m_secureKey;
 	std::unique_ptr<Connection> m_connection;
+	NntpCache* m_cache;
 };
 
 #endif

daemon/nserv/YEncoder.cpp

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2016-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -67,7 +67,7 @@ void YEncoder::WriteSegment()
 	outbuf.Append(CString::FormatStr("=ybegin part=%i line=128 size=%lli name=%s\r\n", m_part, (long long)m_fileSize, FileSystem::BaseFileName(m_filename)));
 	outbuf.Append(CString::FormatStr("=ypart begin=%lli end=%lli\r\n", (long long)(m_offset + 1), (long long)(m_offset + m_size)));
 
-	uint32 crc = 0xFFFFFFFF;
+	Crc32 crc;
 	CharBuffer inbuf(std::min(m_size, 16 * 1024 * 1024));
 	int lnsz = 0;
 	char* out = (char*)outbuf + outbuf.Length();
@@ -82,7 +82,7 @@ void YEncoder::WriteSegment()
 			return; // error;
 		}
 
-		crc = Util::Crc32m(crc, (uchar*)(const char*)inbuf, (int)readBytes);
+		crc.Append((uchar*)(const char*)inbuf, (int)readBytes);
 
 		char* in = inbuf;
 		while (readBytes > 0)
@@ -122,10 +122,8 @@ void YEncoder::WriteSegment()
 			}
 		}
 	}
-	crc ^= 0xFFFFFFFF;
-
 	m_diskfile.Close();
 
-	outbuf.Append(CString::FormatStr("=yend size=%i part=0 pcrc32=%08x\r\n", m_size, (unsigned int)crc));
+	outbuf.Append(CString::FormatStr("=yend size=%i part=0 pcrc32=%08x\r\n", m_size, (unsigned int)crc.Finish()));
 	m_writeFunc(outbuf, outbuf.Length());
 }

daemon/postprocess/DirectUnpack.cpp

@@ -55,7 +55,7 @@ void DirectUnpack::Run()
 		m_destDir = nzbInfo->GetDestDir();
 		m_finalDir = nzbInfo->BuildFinalDirName();
 
-		NzbParameter* parameter = nzbInfo->GetParameters()->Find("*Unpack:Password", false);
+		NzbParameter* parameter = nzbInfo->GetParameters()->Find("*Unpack:Password");
 		if (parameter)
 		{
 			m_password = parameter->GetValue();
@@ -346,7 +346,7 @@ void DirectUnpack::AddMessage(Message::EKind kind, const char* text)
 	if (!strncmp(text, "Unrar: Insert disk with", 23) && strstr(text, " [C]ontinue, [Q]uit"))
 	{
 		BString<1024> filename;
-		filename.Set(text + 24, strstr(text, " [C]ontinue, [Q]uit") - text - 24);
+		filename.Set(text + 24, (int)(strstr(text, " [C]ontinue, [Q]uit") - text - 24));
 		WaitNextVolume(filename);
 		return;
 	}
@@ -539,7 +539,7 @@ void DirectUnpack::AddExtraTime(NzbInfo* nzbInfo)
 {
 	if (m_extraStartTime)
 	{
-		time_t extraTime = Util::CurrentTime() - m_extraStartTime;
+		int extraTime = (int)(Util::CurrentTime() - m_extraStartTime);
 		nzbInfo->SetUnpackSec(nzbInfo->GetUnpackSec() + extraTime);
 		nzbInfo->SetPostTotalSec(nzbInfo->GetPostTotalSec() + extraTime);
 		m_extraStartTime = 0;

daemon/postprocess/ParChecker.cpp

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -106,6 +106,11 @@ private:
 	volatile bool m_working = false;
 };
 
+class RepairCreatorPacket : public Par2::CreatorPacket
+{
+	friend class ParChecker;
+};
+
 Par2::Result Repairer::PreProcess(const char *parFilename)
 {
 	BString<100> memParam("-m%i", g_Options->GetParBuffer());
@@ -368,6 +373,7 @@ int ParChecker::StreamBuf::overflow(int ch)
 
 void ParChecker::Cleanup()
 {
+	Guard guard(m_repairerMutex);
 	m_repairer.reset();
 	m_queuedParFiles.clear();
 	m_processedFiles.clear();
@@ -425,11 +431,6 @@ ParChecker::EStatus ParChecker::RunParCheckAll()
 			{
 				allStatus = status;
 			}
-
-			if (g_Options->GetBrokenLog())
-			{
-				WriteBrokenLog(status);
-			}
 		}
 	}
 
@@ -464,6 +465,9 @@ ParChecker::EStatus ParChecker::RunParCheck(const char* parFilename)
 		return psFailed;
 	}
 
+	CString creator = GetPacketCreator();
+	info("Recovery files created by: %s", creator.Empty() ? "<unknown program>" : *creator);
+
 	m_stage = ptVerifyingSources;
 	res = GetRepairer()->Process(false);
 
@@ -580,7 +584,8 @@ ParChecker::EStatus ParChecker::RunParCheck(const char* parFilename)
 		{
 			m_errMsg = Par2CmdLineErrStr[res];
 		}
-		PrintMessage(Message::mkError, "Repair failed for %s: %s", *m_infoName, *m_errMsg);
+		PrintMessage(Message::mkError, "Repair failed for %s: %s. Recovery files created by: %s",
+			*m_infoName, *m_errMsg, creator.Empty() ? "<unknown program>" : *creator);
 	}
 
 	Cleanup();
@@ -594,7 +599,10 @@ int ParChecker::PreProcessPar()
 	{
 		Cleanup();
 
-		m_repairer = std::make_unique<Repairer>(this);
+		{
+			Guard guard(m_repairerMutex);
+			m_repairer = std::make_unique<Repairer>(this);
+		}
 
 		res = GetRepairer()->PreProcess(m_parFilename);
 		debug("ParChecker: PreProcess-result=%i", res);
@@ -814,7 +822,7 @@ bool ParChecker::AddSplittedFragments()
 	DirBrowser dir(m_destDir);
 	while (const char* filename = dir.Next())
 	{
-		if (strcmp(filename, "_brokenlog.txt") && !IsParredFile(filename) && !IsProcessedFile(filename))
+		if (IsParredFile(filename) && !IsProcessedFile(filename))
 		{
 			for (Par2::Par2RepairerSourceFile *sourcefile : GetRepairer()->sourcefiles)
 			{
@@ -942,8 +950,7 @@ bool ParChecker::AddExtraFiles(bool onlyMissing, bool externalDir, const char* d
 	DirBrowser dir(directory);
 	while (const char* filename = dir.Next())
 	{
-		if (strcmp(filename, "_brokenlog.txt") &&
-			(externalDir || (!IsParredFile(filename) && !IsProcessedFile(filename))))
+		if (externalDir || (!IsParredFile(filename) && !IsProcessedFile(filename)))
 		{
 			BString<1024> fullfilename("%s%c%s", directory, PATH_SEPARATOR, filename);
 			extrafiles.emplace_back(*fullfilename, FileSystem::FileSize(fullfilename));
@@ -1186,49 +1193,14 @@ void ParChecker::CheckEmptyFiles()
 
 void ParChecker::Cancel()
 {
-	GetRepairer()->cancelled = true;
-	QueueChanged();
-}
-
-void ParChecker::WriteBrokenLog(EStatus status)
-{
-	BString<1024> brokenLogName("%s%c_brokenlog.txt", *m_destDir, PATH_SEPARATOR);
-
-	if (status != psRepairNotNeeded || FileSystem::FileExists(brokenLogName))
 	{
-		DiskFile file;
-		if (file.Open(brokenLogName, DiskFile::omAppend))
+		Guard guard(m_repairerMutex);
+		if (m_repairer)
 		{
-			if (status == psFailed)
-			{
-				if (IsStopped())
-				{
-					file.Print("Repair cancelled for %s\n", *m_infoName);
-				}
-				else
-				{
-					file.Print("Repair failed for %s: %s\n", *m_infoName, *m_errMsg);
-				}
-			}
-			else if (status == psRepairPossible)
-			{
-				file.Print("Repair possible for %s\n", *m_infoName);
-			}
-			else if (status == psRepaired)
-			{
-				file.Print("Successfully repaired %s\n", *m_infoName);
-			}
-			else if (status == psRepairNotNeeded)
-			{
-				file.Print("Repair not needed for %s\n", *m_infoName);
-			}
-			file.Close();
-		}
-		else
-		{
-			PrintMessage(Message::mkError, "Could not open file %s", *brokenLogName);
+			m_repairer->GetRepairer()->cancelled = true;
 		}
 	}
+	QueueChanged();
 }
 
 void ParChecker::SaveSourceList()
@@ -1399,7 +1371,7 @@ bool ParChecker::VerifySuccessDataFile(void* diskfile, void* sourcefile, uint32
 	{
 		const Par2::FILEVERIFICATIONENTRY* entry = packet->VerificationEntry(i);
 		Par2::u32 blockCrc = entry->crc;
-		parCrc = i == 0 ? blockCrc : Util::Crc32Combine(parCrc, blockCrc, (uint32)blocksize);
+		parCrc = i == 0 ? blockCrc : Crc32::Combine(parCrc, blockCrc, (uint32)blocksize);
 	}
 	debug("Block-CRC: %x, filename: %s", parCrc, FileSystem::BaseFileName(sourceFile->GetTargetFile()->FileName().c_str()));
 
@@ -1477,7 +1449,7 @@ bool ParChecker::VerifyPartialDataFile(void* diskfile, void* sourcefile, Segment
 			}
 			const Par2::FILEVERIFICATIONENTRY* entry = packet->VerificationEntry(i);
 			Par2::u32 blockCrc = entry->crc;
-			parCrc = blockStart == i ? blockCrc : Util::Crc32Combine(parCrc, blockCrc, (uint32)blocksize);
+			parCrc = blockStart == i ? blockCrc : Crc32::Combine(parCrc, blockCrc, (uint32)blocksize);
 		}
 		else
 		{
@@ -1537,7 +1509,7 @@ bool ParChecker::SmartCalcFileRangeCrc(DiskFile& file, int64 start, int64 end, S
 
 		if (segment.GetOffset() >= start && segment.GetOffset() + segment.GetSize() <= end)
 		{
-			downloadCrc = !started ? segment.GetCrc() : Util::Crc32Combine(downloadCrc, segment.GetCrc(), (uint32)segment.GetSize());
+			downloadCrc = !started ? segment.GetCrc() : Crc32::Combine(downloadCrc, segment.GetCrc(), (uint32)segment.GetSize());
 			started = true;
 		}
 
@@ -1555,7 +1527,7 @@ bool ParChecker::SmartCalcFileRangeCrc(DiskFile& file, int64 start, int64 end, S
 				return false;
 			}
 
-			downloadCrc = Util::Crc32Combine(downloadCrc, (uint32)partialCrc, (uint32)(end - segment.GetOffset() + 1));
+			downloadCrc = Crc32::Combine(downloadCrc, (uint32)partialCrc, (uint32)(end - segment.GetOffset() + 1));
 
 			break;
 		}
@@ -1576,21 +1548,37 @@ bool ParChecker::DumbCalcFileRangeCrc(DiskFile& file, int64 start, int64 end, ui
 	}
 
 	CharBuffer buffer(1024 * 64);
-	uint32 downloadCrc = 0xFFFFFFFF;
+	Crc32 downloadCrc;
 
 	int cnt = buffer.Size();
 	while (cnt == buffer.Size() && start < end)
 	{
 		int needBytes = end - start + 1 > buffer.Size() ? buffer.Size() : (int)(end - start + 1);
 		cnt = (int)file.Read(buffer, needBytes);
-		downloadCrc = Util::Crc32m(downloadCrc, (uchar*)(char*)buffer, cnt);
+		downloadCrc.Append((uchar*)(char*)buffer, cnt);
 		start += cnt;
 	}
 
-	downloadCrc ^= 0xFFFFFFFF;
-
-	*downloadCrcOut = downloadCrc;
+	*downloadCrcOut = downloadCrc.Finish();
 	return true;
 }
 
+CString ParChecker::GetPacketCreator()
+{
+	Par2::CREATORPACKET* creatorpacket;
+	if (GetRepairer()->creatorpacket &&
+		(creatorpacket = (Par2::CREATORPACKET*)(((RepairCreatorPacket*)GetRepairer()->creatorpacket)->packetdata)))
+	{
+		int len = (int)(creatorpacket->header.length - sizeof(Par2::PACKET_HEADER));
+		BString<1024> creator;
+		if (len > 0)
+		{
+			creator.Set((const char*)creatorpacket->client, len);
+		}
+		return *creator;
+	}
+
+	return nullptr;
+}
+
 #endif

daemon/postprocess/ParChecker.h

@@ -186,6 +186,7 @@ private:
 	StreamBuf m_parErrStream{this, Message::mkError};
 	std::ostream m_parCout{&m_parOutStream};
 	std::ostream m_parCerr{&m_parErrStream};
+	Mutex m_repairerMutex;
 
 	// "m_repairer" should be of type "Par2::Par2Repairer", however to prevent the
 	// including of libpar2-headers into this header-file we use an empty abstract class.
@@ -204,7 +205,6 @@ private:
 	bool AddDupeFiles();
 	bool AddExtraFiles(bool onlyMissing, bool externalDir, const char* directory);
 	bool IsProcessedFile(const char* filename);
-	void WriteBrokenLog(EStatus status);
 	void SaveSourceList();
 	void DeleteLeftovers();
 	void signal_filename(std::string str);
@@ -220,6 +220,7 @@ private:
 		uint32* downloadCrc);
 	bool DumbCalcFileRangeCrc(DiskFile& file, int64 start, int64 end, uint32* downloadCrc);
 	void CheckEmptyFiles();
+	CString GetPacketCreator();
 
 	friend class Repairer;
 };

daemon/postprocess/PrePostProcessor.cpp

@@ -52,7 +52,7 @@ void PrePostProcessor::Run()
 		usleep(20 * 1000);
 	}
 
-	if (g_Options->GetServerMode() && g_Options->GetSaveQueue() && g_Options->GetReloadQueue())
+	if (g_Options->GetServerMode())
 	{
 		SanitisePostQueue();
 	}
@@ -309,7 +309,7 @@ void PrePostProcessor::NzbDownloaded(DownloadQueue* downloadQueue, NzbInfo* nzbI
 		g_QueueScriptCoordinator->EnqueueScript(nzbInfo, QueueScriptCoordinator::qeNzbDeleted);
 	}
 
-	if (!nzbInfo->GetPostInfo() && g_Options->GetDecode())
+	if (!nzbInfo->GetPostInfo() && !g_Options->GetRawArticle() && !g_Options->GetSkipWrite())
 	{
 		nzbInfo->PrintMessage(Message::mkInfo, "Queueing %s for post-processing", nzbInfo->GetName());
 
@@ -426,12 +426,12 @@ void PrePostProcessor::DeleteCleanup(NzbInfo* nzbInfo)
 			}
 		}
 
-		// delete .out.tmp-files and _brokenlog.txt
+		// delete .out.tmp-files
 		DirBrowser dir(nzbInfo->GetDestDir());
 		while (const char* filename = dir.Next())
 		{
 			int len = strlen(filename);
-			if ((len > 8 && !strcmp(filename + len - 8, ".out.tmp")) || !strcmp(filename, "_brokenlog.txt"))
+			if (len > 8 && !strcmp(filename + len - 8, ".out.tmp"))
 			{
 				BString<1024> fullFilename("%s%c%s", nzbInfo->GetDestDir(), PATH_SEPARATOR, filename);
 				detail("Deleting file %s", filename);
@@ -706,7 +706,7 @@ void PrePostProcessor::StartJob(DownloadQueue* downloadQueue, PostInfo* postInfo
 	}
 #endif
 
-	NzbParameter* unpackParameter = postInfo->GetNzbInfo()->GetParameters()->Find("*Unpack:", false);
+	NzbParameter* unpackParameter = postInfo->GetNzbInfo()->GetParameters()->Find("*Unpack:");
 	bool wantUnpack = !(unpackParameter && !strcasecmp(unpackParameter->GetValue(), "no"));
 	bool unpack = wantUnpack && postInfo->GetNzbInfo()->GetUnpackStatus() == NzbInfo::usNone &&
 		postInfo->GetNzbInfo()->GetDeleteStatus() == NzbInfo::dsNone;
@@ -927,7 +927,7 @@ void PrePostProcessor::FileDownloaded(DownloadQueue* downloadQueue, NzbInfo* nzb
 		g_QueueScriptCoordinator->EnqueueScript(nzbInfo, QueueScriptCoordinator::qeFileDownloaded);
 	}
 
-	if (g_Options->GetDirectUnpack() && g_Options->GetDecode())
+	if (g_Options->GetDirectUnpack() && !g_Options->GetRawArticle() && !g_Options->GetSkipWrite())
 	{
 		bool allowPar;
 		if (nzbInfo->GetDirectUnpackStatus() == NzbInfo::nsNone &&
@@ -935,7 +935,7 @@ void PrePostProcessor::FileDownloaded(DownloadQueue* downloadQueue, NzbInfo* nzb
 			DirectUnpack::IsArchiveFilename(fileInfo->GetFilename()) &&
 			CanRunMoreJobs(&allowPar))
 		{
-			NzbParameter* unpackParameter = nzbInfo->GetParameters()->Find("*Unpack:", false);
+			NzbParameter* unpackParameter = nzbInfo->GetParameters()->Find("*Unpack:");
 			bool wantUnpack = !(unpackParameter && !strcasecmp(unpackParameter->GetValue(), "no"));
 			if (wantUnpack && nzbInfo->GetFailedArticles() == 0)
 			{

daemon/postprocess/RarRenamer.cpp

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2016-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -202,7 +202,8 @@ void RarRenamer::MakeSets()
 	// find first volumes and create initial incomplete sets
 	for (RarVolume& volume : m_volumes)
 	{
-		if (!volume.GetFiles()->empty() && volume.GetVolumeNo() == 0)
+		if (!volume.GetFiles()->empty() && volume.GetVolumeNo() == 0 &&
+			!volume.GetFiles()->front().GetSplitBefore())
 		{
 			m_sets.push_back({&volume});
 		}
@@ -217,6 +218,8 @@ void RarRenamer::MakeSets()
 			while (found)
 			{
 				found = false;
+				std::vector<RarVolume*> candidates;
+
 				RarVolume* lastVolume = set.back();
 				for (RarVolume& volume : *volumes)
 				{
@@ -229,12 +232,37 @@ void RarRenamer::MakeSets()
 						 !strcmp(volume.GetFiles()->front().GetFilename(), lastVolume->GetFiles()->back().GetFilename())) ||
 						 (!volume.GetFiles()->front().GetSplitBefore() && !lastVolume->GetFiles()->back().GetSplitAfter())))
 					{
-						debug("   adding %s", FileSystem::BaseFileName(volume.GetFilename()));
-						set.push_back(&volume);
-						found = true;
-						break;
+						debug("   found candidate %s", FileSystem::BaseFileName(volume.GetFilename()));
+						candidates.push_back(&volume);
 					}
 				}
+
+				RarVolume* nextVolume = nullptr;
+
+				if (candidates.size() > 1)
+				{
+					for (RarVolume* volume : candidates)
+					{
+						if (SameArchiveName(FileSystem::BaseFileName(set[0]->GetFilename()),
+							FileSystem::BaseFileName(volume->GetFilename()), set[0]->GetNewNaming()))
+						{
+							nextVolume = volume;
+							break;
+						}
+					}
+				}
+
+				if (!nextVolume && !candidates.empty())
+				{
+					nextVolume = candidates.front();
+				}
+
+				if (nextVolume)
+				{
+					debug("   adding %s", FileSystem::BaseFileName(nextVolume->GetFilename()));
+					set.push_back(nextVolume);
+					found = true;
+				}
 			}
 
 			RarVolume* lastVolume = set.back();
@@ -256,6 +284,42 @@ void RarRenamer::MakeSets()
 	}
 }
 
+bool RarRenamer::SameArchiveName(const char* filename1, const char* filename2, bool newNaming)
+{
+	if (strlen(filename1) != strlen(filename2))
+	{
+		return false;
+	}
+
+	const char* ext1 = strrchr(filename1, '.');
+	const char* ext2 = strrchr(filename2, '.');
+
+	if (!(ext1 && ext2 && strlen(ext1) == strlen(ext2)))
+	{
+		return false;
+	}
+
+	if (newNaming)
+	{
+		if (ext1 == filename1 || ext2 == filename2)
+		{
+			return false;
+		}
+		BString<1024> name1, name2;
+		name1.Set(filename1, (int)(ext1 - filename1));
+		name2.Set(filename2, (int)(ext2 - filename2));
+		ext1 = strrchr(name1, '.');
+		ext2 = strrchr(name2, '.');
+		return ext1 && ext2 && strlen(ext1) == strlen(ext2) &&
+			!strncmp(ext1, ".part", 5) && !strncmp(ext2, ".part", 5) &&
+			!strncmp(name1, name2, ext1 - name1);
+	}
+	else
+	{
+		return !strncmp(filename1, filename2, ext1 - filename1);
+	}
+}
+
 bool RarRenamer::IsSetProperlyNamed(RarVolumeSet& set)
 {
 	RegEx regExPart(".*.part([0-9]+)\\.rar$");

daemon/postprocess/RarRenamer.h

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2016-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -76,6 +76,7 @@ private:
 	void MakeSets();
 	bool IsSetProperlyNamed(RarVolumeSet& set);
 	RarFile* FindMainFile(RarVolumeSet& set);
+	static bool SameArchiveName(const char* filename1, const char* filename2, bool newNaming);
 };
 
 #endif

daemon/postprocess/Rename.cpp

@@ -137,7 +137,7 @@ void RenameController::ExecRename(const char* destDir, const char* finalDir, con
 		m_rarRenamer.SetInfoName(nzbName);
 		m_rarRenamer.SetIgnoreExt(g_Options->GetUnpackIgnoreExt());
 
-		NzbParameter* parameter = m_postInfo->GetNzbInfo()->GetParameters()->Find("*Unpack:Password", false);
+		NzbParameter* parameter = m_postInfo->GetNzbInfo()->GetParameters()->Find("*Unpack:Password");
 		if (parameter)
 		{
 			m_rarRenamer.SetPassword(parameter->GetValue());

daemon/postprocess/Unpack.cpp

@@ -60,10 +60,10 @@ void UnpackController::Run()
 		m_finalDir = m_postInfo->GetNzbInfo()->GetFinalDir();
 		m_name = m_postInfo->GetNzbInfo()->GetName();
 
-		NzbParameter* parameter = m_postInfo->GetNzbInfo()->GetParameters()->Find("*Unpack:", false);
+		NzbParameter* parameter = m_postInfo->GetNzbInfo()->GetParameters()->Find("*Unpack:");
 		unpack = !(parameter && !strcasecmp(parameter->GetValue(), "no"));
 
-		parameter = m_postInfo->GetNzbInfo()->GetParameters()->Find("*Unpack:Password", false);
+		parameter = m_postInfo->GetNzbInfo()->GetParameters()->Find("*Unpack:Password");
 		if (parameter)
 		{
 			m_password = parameter->GetValue();
@@ -191,7 +191,7 @@ void UnpackController::UnpackArchives(EUnpacker unpacker, bool multiVolumes)
 		if (!m_unpackOk && m_hasParFiles && !m_unpackPasswordError &&
 			m_postInfo->GetNzbInfo()->GetParStatus() <= NzbInfo::psSkipped)
 		{
-			// for rar4- or 7z-archives try par-check first, before trying password file
+			debug("For rar4- or 7z-archives try par-check first, before trying password file");
 			return;
 		}
 	}
@@ -218,6 +218,7 @@ void UnpackController::UnpackArchives(EUnpacker unpacker, bool multiVolumes)
 			(m_unpackDecryptError || m_unpackPasswordError) &&
 			infile.ReadLine(password, sizeof(password) - 1))
 		{
+			debug("Password line: %s", password);
 			// trim trailing <CR> and <LF>
 			char* end = password + strlen(password) - 1;
 			while (end >= password && (*end == '\n' || *end == '\r')) *end-- = '\0';
@@ -304,7 +305,7 @@ void UnpackController::ExecuteUnrar(const char* password)
 	SetProgressLabel("");
 
 	m_unpackOk = exitCode == 0 && m_allOkMessageReceived && !GetTerminated();
-	m_unpackStartError = exitCode == -1;
+	m_unpackStartError = exitCode == -1 && !m_autoTerminated;
 	m_unpackSpaceError = exitCode == 5;
 	m_unpackPasswordError |= exitCode == 11; // only for rar5-archives
 
@@ -359,7 +360,7 @@ void UnpackController::ExecuteSevenZip(const char* password, bool multiVolumes)
 	SetProgressLabel("");
 
 	m_unpackOk = exitCode == 0 && m_allOkMessageReceived && !GetTerminated();
-	m_unpackStartError = exitCode == -1;
+	m_unpackStartError = exitCode == -1 && !m_autoTerminated;
 
 	if (!m_unpackOk && exitCode > 0)
 	{

daemon/queue/DirectRenamer.cpp

@@ -186,7 +186,7 @@ void DirectRenamer::ArticleDownloaded(DownloadQueue* downloadQueue, FileInfo* fi
 		debug("file: %s; article-hash16k: %s", fileInfo->GetFilename(), fileInfo->GetHash16k());
 	}
 
-	detail("Detected %s %s", (contentAnalyzer->GetParFile() ? "par2-file" : "non-par2-file"), fileInfo->GetFilename());
+	fileInfo->GetNzbInfo()->PrintMessage(Message::mkDetail, "Detected %s %s", (contentAnalyzer->GetParFile() ? "par2-file" : "non-par2-file"), fileInfo->GetFilename());
 
 	if (fileInfo->GetParFile() != contentAnalyzer->GetParFile())
 	{
@@ -199,8 +199,15 @@ void DirectRenamer::ArticleDownloaded(DownloadQueue* downloadQueue, FileInfo* fi
 		nzbInfo->SetParCurrentSuccessSize(nzbInfo->GetParCurrentSuccessSize() + fileInfo->GetSuccessSize() * delta);
 		nzbInfo->SetParCurrentFailedSize(nzbInfo->GetParCurrentFailedSize() +
 			fileInfo->GetFailedSize() * delta + fileInfo->GetMissedSize() * delta);
+		nzbInfo->SetParFailedSize(nzbInfo->GetParFailedSize() + fileInfo->GetMissedSize() * delta);
 		nzbInfo->SetRemainingParCount(nzbInfo->GetRemainingParCount() + 1 * delta);
 
+		if (!fileInfo->GetParFile() && fileInfo->GetPaused())
+		{
+			fileInfo->GetNzbInfo()->PrintMessage(Message::mkInfo, "Resuming non-par2-file %s", fileInfo->GetFilename());
+			fileInfo->SetPaused(false);
+		}
+
 		downloadQueue->Save();
 	}
 
@@ -314,7 +321,7 @@ void DirectRenamer::UnpausePars(NzbInfo* nzbInfo)
 			FileInfo* fileInfo = nzbInfo->GetFileList()->Find(parFile.GetId());
 			if (fileInfo)
 			{
-				nzbInfo->PrintMessage(Message::mkDetail, "Increasing priority for par2-file %s", fileInfo->GetFilename());
+				nzbInfo->PrintMessage(Message::mkInfo, "Increasing priority for par2-file %s", fileInfo->GetFilename());
 				fileInfo->SetPaused(false);
 				fileInfo->SetExtraPriority(true);
 			}

daemon/queue/DiskState.cpp

@@ -27,6 +27,10 @@
 #include "FileSystem.h"
 
 static const char* FORMATVERSION_SIGNATURE = "nzbget diskstate file version ";
+const int DISKSTATE_QUEUE_VERSION = 60;
+const int DISKSTATE_FILE_VERSION = 5;
+const int DISKSTATE_STATS_VERSION = 3;
+const int DISKSTATE_FEEDS_VERSION = 3;
 
 class StateDiskFile : public DiskFile
 {
@@ -265,7 +269,7 @@ bool DiskState::SaveDownloadQueue(DownloadQueue* downloadQueue, bool saveHistory
 	bool ok = true;
 
 	{
-		StateFile stateFile("queue", 60, true);
+		StateFile stateFile("queue", DISKSTATE_QUEUE_VERSION, true);
 		if (!downloadQueue->GetQueue()->empty())
 		{
 			StateDiskFile* outfile = stateFile.BeginWrite();
@@ -288,7 +292,7 @@ bool DiskState::SaveDownloadQueue(DownloadQueue* downloadQueue, bool saveHistory
 
 	if (saveHistory)
 	{
-		StateFile stateFile("history", 60, true);
+		StateFile stateFile("history", DISKSTATE_QUEUE_VERSION, true);
 		if (!downloadQueue->GetHistory()->empty())
 		{
 			StateDiskFile* outfile = stateFile.BeginWrite();
@@ -320,7 +324,7 @@ bool DiskState::LoadDownloadQueue(DownloadQueue* downloadQueue, Servers* servers
 	int formatVersion = 0;
 
 	{
-		StateFile stateFile("queue", 60, true);
+		StateFile stateFile("queue", DISKSTATE_QUEUE_VERSION, true);
 		if (stateFile.FileExists())
 		{
 			StateDiskFile* infile = stateFile.BeginRead();
@@ -349,7 +353,7 @@ bool DiskState::LoadDownloadQueue(DownloadQueue* downloadQueue, Servers* servers
 
 	if (formatVersion == 0 || formatVersion >= 57)
 	{
-		StateFile stateFile("history", 60, true);
+		StateFile stateFile("history", DISKSTATE_QUEUE_VERSION, true);
 		if (stateFile.FileExists())
 		{
 			StateDiskFile* infile = stateFile.BeginRead();
@@ -361,6 +365,8 @@ bool DiskState::LoadDownloadQueue(DownloadQueue* downloadQueue, Servers* servers
 		}
 	}
 
+	LoadAllFileInfos(downloadQueue);
+
 	CleanupQueueDir(downloadQueue);
 
 	if (!LoadAllFileStates(downloadQueue, servers)) goto error;
@@ -819,19 +825,14 @@ bool DiskState::LoadNzbInfo(NzbInfo* nzbInfo, Servers* servers, StateDiskFile& i
 
 		std::unique_ptr<FileInfo> fileInfo = std::make_unique<FileInfo>();
 		fileInfo->SetId(id);
-
-		bool res = LoadFile(fileInfo.get(), true, false);
-		if (res)
+		fileInfo->SetPaused(paused);
+		if (formatVersion < 56)
 		{
-			fileInfo->SetPaused(paused);
-			if (formatVersion < 56)
-			{
-				fileInfo->SetTime(time);
-			}
-			fileInfo->SetExtraPriority((bool)extraPriority);
-			fileInfo->SetNzbInfo(nzbInfo);
-			nzbInfo->GetFileList()->Add(std::move(fileInfo));
+			fileInfo->SetTime(time);
 		}
+		fileInfo->SetExtraPriority((bool)extraPriority);
+		fileInfo->SetNzbInfo(nzbInfo);
+		nzbInfo->GetFileList()->Add(std::move(fileInfo));
 	}
 
 	return true;
@@ -884,7 +885,7 @@ bool DiskState::SaveFile(FileInfo* fileInfo)
 	debug("Saving FileInfo %i to disk", fileInfo->GetId());
 
 	BString<100> filename("%i", fileInfo->GetId());
-	StateFile stateFile(filename, 5, false);
+	StateFile stateFile(filename, DISKSTATE_FILE_VERSION, false);
 
 	StateDiskFile* outfile = stateFile.BeginWrite();
 	if (!outfile)
@@ -892,10 +893,10 @@ bool DiskState::SaveFile(FileInfo* fileInfo)
 		return false;
 	}
 
-	return SaveFileInfo(fileInfo, *outfile) && stateFile.FinishWrite();
+	return SaveFileInfo(fileInfo, *outfile, true) && stateFile.FinishWrite();
 }
 
-bool DiskState::SaveFileInfo(FileInfo* fileInfo, StateDiskFile& outfile)
+bool DiskState::SaveFileInfo(FileInfo* fileInfo, StateDiskFile& outfile, bool articles)
 {
 	outfile.PrintLine("%s", fileInfo->GetSubject());
 	outfile.PrintLine("%s", fileInfo->GetFilename());
@@ -918,11 +919,14 @@ bool DiskState::SaveFileInfo(FileInfo* fileInfo, StateDiskFile& outfile)
 		outfile.PrintLine("%s", *group);
 	}
 
-	outfile.PrintLine("%i", (int)fileInfo->GetArticles()->size());
-	for (ArticleInfo* articleInfo : fileInfo->GetArticles())
+	if (articles)
 	{
-		outfile.PrintLine("%i,%i", articleInfo->GetPartNumber(), articleInfo->GetSize());
-		outfile.PrintLine("%s", articleInfo->GetMessageId());
+		outfile.PrintLine("%i", (int)fileInfo->GetArticles()->size());
+		for (ArticleInfo* articleInfo : fileInfo->GetArticles())
+		{
+			outfile.PrintLine("%i,%i", articleInfo->GetPartNumber(), articleInfo->GetSize());
+			outfile.PrintLine("%s", articleInfo->GetMessageId());
+		}
 	}
 
 	return true;
@@ -938,7 +942,7 @@ bool DiskState::LoadFile(FileInfo* fileInfo, bool fileSummary, bool articles)
 	debug("Loading FileInfo %i from disk", fileInfo->GetId());
 
 	BString<100> filename("%i", fileInfo->GetId());
-	StateFile stateFile(filename, 5, false);
+	StateFile stateFile(filename, DISKSTATE_FILE_VERSION, false);
 
 	StateDiskFile* infile = stateFile.BeginRead();
 	if (!infile)
@@ -999,10 +1003,9 @@ bool DiskState::LoadFileInfo(FileInfo* fileInfo, StateDiskFile& infile, int form
 		if (fileSummary) fileInfo->GetGroups()->push_back(buf);
 	}
 
-	if (infile.ScanLine("%i", &size) != 1) goto error;
-
 	if (articles)
 	{
+		if (infile.ScanLine("%i", &size) != 1) goto error;
 		for (int i = 0; i < size; i++)
 		{
 			int PartNumber, PartSize;
@@ -1030,7 +1033,7 @@ bool DiskState::SaveFileState(FileInfo* fileInfo, bool completed)
 	debug("Saving FileState %i to disk", fileInfo->GetId());
 
 	BString<100> filename("%i%s", fileInfo->GetId(), completed ? "c" : "s");
-	StateFile stateFile(filename, 5, false);
+	StateFile stateFile(filename, DISKSTATE_FILE_VERSION, false);
 
 	StateDiskFile* outfile = stateFile.BeginWrite();
 	if (!outfile)
@@ -1073,7 +1076,7 @@ bool DiskState::LoadFileState(FileInfo* fileInfo, Servers* servers, bool complet
 	debug("Loading FileInfo %i from disk", fileInfo->GetId());
 
 	BString<100> filename("%i%s", fileInfo->GetId(), completed ? "c" : "s");
-	StateFile stateFile(filename, 5, false);
+	StateFile stateFile(filename, DISKSTATE_FILE_VERSION, false);
 
 	StateDiskFile* infile = stateFile.BeginRead();
 	if (!infile)
@@ -1426,6 +1429,50 @@ void DiskState::CleanupTempDir(DownloadQueue* downloadQueue)
 
 void DiskState::CleanupQueueDir(DownloadQueue* downloadQueue)
 {
+	// Prepare sorted id lists for faster search
+
+	std::vector<int> nzbIdList;
+	std::vector<int> fileIdList;
+
+	for (NzbInfo* nzbInfo : downloadQueue->GetQueue())
+	{
+		nzbIdList.push_back(nzbInfo->GetId());
+
+		for (FileInfo* fileInfo : nzbInfo->GetFileList())
+		{
+			fileIdList.push_back(fileInfo->GetId());
+		}
+
+		for (CompletedFile& completedFile : nzbInfo->GetCompletedFiles())
+		{
+			fileIdList.push_back(completedFile.GetId());
+		}
+	}
+
+	for (HistoryInfo* historyInfo : downloadQueue->GetHistory())
+	{
+		if (historyInfo->GetKind() == HistoryInfo::hkNzb)
+		{
+			NzbInfo* nzbInfo = historyInfo->GetNzbInfo();
+			nzbIdList.push_back(nzbInfo->GetId());
+
+			for (FileInfo* fileInfo : nzbInfo->GetFileList())
+			{
+				fileIdList.push_back(fileInfo->GetId());
+			}
+
+			for (CompletedFile& completedFile : nzbInfo->GetCompletedFiles())
+			{
+				fileIdList.push_back(completedFile.GetId());
+			}
+		}
+	}
+
+	std::sort(nzbIdList.begin(), nzbIdList.end());
+	std::sort(fileIdList.begin(), fileIdList.end());
+
+	// Do cleanup
+
 	int deletedFiles = 0;
 
 	DirBrowser dir(g_Options->GetQueueDir());
@@ -1438,74 +1485,12 @@ void DiskState::CleanupQueueDir(DownloadQueue* downloadQueue)
 		if ((sscanf(filename, "%i%c", &id, &suffix) == 2 && (suffix == 's' || suffix == 'c')) ||
 			(sscanf(filename, "%i", &id) == 1 && !strchr(filename, '.')))
 		{
-			for (NzbInfo* nzbInfo : downloadQueue->GetQueue())
-			{
-				for (FileInfo* fileInfo : nzbInfo->GetFileList())
-				{
-					if (fileInfo->GetId() == id)
-					{
-						goto next;
-					}
-				}
-
-				for (CompletedFile& completedFile : nzbInfo->GetCompletedFiles())
-				{
-					if (completedFile.GetId() == id)
-					{
-						goto next;
-					}
-				}
-			}
-
-			for (HistoryInfo* historyInfo : downloadQueue->GetHistory())
-			{
-				if (historyInfo->GetKind() == HistoryInfo::hkNzb)
-				{
-					NzbInfo* nzbInfo = historyInfo->GetNzbInfo();
-
-					for (FileInfo* fileInfo : nzbInfo->GetFileList())
-					{
-						if (fileInfo->GetId() == id)
-						{
-							goto next;
-						}
-					}
-
-					for (CompletedFile& completedFile : nzbInfo->GetCompletedFiles())
-					{
-						if (completedFile.GetId() == id)
-						{
-							goto next;
-						}
-					}
-				}
-			}
-
-			del = true;
+			del = !std::binary_search(fileIdList.begin(), fileIdList.end(), id);
 		}
 
 		if (!del && sscanf(filename, "n%i.log", &id) == 1)
 		{
-			for (NzbInfo* nzbInfo : downloadQueue->GetQueue())
-			{
-				if (nzbInfo->GetId() == id)
-				{
-					goto next;
-				}
-			}
-
-			for (HistoryInfo* historyInfo : downloadQueue->GetHistory())
-			{
-				if (historyInfo->GetKind() == HistoryInfo::hkNzb)
-				{
-					if (historyInfo->GetNzbInfo()->GetId() == id)
-					{
-						goto next;
-					}
-				}
-			}
-
-			del = true;
+			del = !std::binary_search(nzbIdList.begin(), nzbIdList.end(), id);
 		}
 
 		if (del)
@@ -1515,8 +1500,6 @@ void DiskState::CleanupQueueDir(DownloadQueue* downloadQueue)
 			FileSystem::DeleteFile(fullFilename);
 			deletedFiles++;
 		}
-
-		next:;
 	}
 
 	if (deletedFiles > 0)
@@ -1534,7 +1517,7 @@ bool DiskState::SaveFeeds(Feeds* feeds, FeedHistory* feedHistory)
 {
 	debug("Saving feeds state to disk");
 
-	StateFile stateFile("feeds", 3, true);
+	StateFile stateFile("feeds", DISKSTATE_FEEDS_VERSION, true);
 
 	if (feeds->empty() && feedHistory->empty())
 	{
@@ -1562,7 +1545,7 @@ bool DiskState::LoadFeeds(Feeds* feeds, FeedHistory* feedHistory)
 {
 	debug("Loading feeds state from disk");
 
-	StateFile stateFile("feeds", 3, true);
+	StateFile stateFile("feeds", DISKSTATE_FEEDS_VERSION, true);
 
 	if (!stateFile.FileExists())
 	{
@@ -1704,6 +1687,119 @@ void DiskState::CalcFileStats(DownloadQueue* downloadQueue, int formatVersion)
 	}
 }
 
+bool DiskState::SaveAllFileInfos(DownloadQueue* downloadQueue)
+{
+	bool ok = true;
+	StateFile stateFile("files", DISKSTATE_FILE_VERSION, true);
+	if (!downloadQueue->GetQueue()->empty())
+	{
+		StateDiskFile* outfile = stateFile.BeginWrite();
+		if (!outfile)
+		{
+			return false;
+		}
+
+		// save file-infos
+
+		int fileCount = 0;
+		for (NzbInfo* nzbInfo : downloadQueue->GetQueue())
+		{
+			fileCount += nzbInfo->GetFileList()->size();
+		}
+		outfile->PrintLine("%i", fileCount);
+
+		for (NzbInfo* nzbInfo : downloadQueue->GetQueue())
+		{
+			for (FileInfo* fileInfo : nzbInfo->GetFileList())
+			{
+				outfile->PrintLine("%i", fileInfo->GetId());
+				SaveFileInfo(fileInfo, *outfile, false);
+			}
+		}
+
+		// now rename to dest file name
+		ok = stateFile.FinishWrite();
+	}
+	else
+	{
+		stateFile.Discard();
+	}
+
+	return ok;
+}
+
+bool DiskState::LoadAllFileInfos(DownloadQueue* downloadQueue)
+{
+	if (downloadQueue->GetQueue()->empty())
+	{
+		return true;
+	}
+
+	StateFile stateFile("files", DISKSTATE_FILE_VERSION, false);
+	StateDiskFile* infile = nullptr;
+	bool useHibernate = false;
+
+	if (stateFile.FileExists())
+	{
+		infile = stateFile.BeginRead();
+		useHibernate = infile != nullptr;
+		if (useHibernate)
+		{
+			int fileCount = 0;
+			for (NzbInfo* nzbInfo : downloadQueue->GetQueue())
+			{
+				fileCount += nzbInfo->GetFileList()->size();
+			}
+			int size = 0;
+			useHibernate = infile->ScanLine("%i", &size) == 1 && size == fileCount;
+		}
+		if (!useHibernate)
+		{
+			stateFile.Discard();
+		}
+	}
+
+	for (NzbInfo* nzbInfo : downloadQueue->GetQueue())
+	{
+		RawFileList brokenFileInfos;
+
+		for (FileInfo* fileInfo : nzbInfo->GetFileList())
+		{
+			bool res = false;
+			if (useHibernate)
+			{
+				int id = 0;
+				infile->ScanLine("%i", &id);
+				if (id == fileInfo->GetId())
+				{
+					res = LoadFileInfo(fileInfo, *infile, stateFile.GetFileVersion(), true, false);
+				}
+			}
+			if (!res)
+			{
+				res = LoadFile(fileInfo, true, false);
+			}
+			if (!res)
+			{
+				brokenFileInfos.push_back(fileInfo);
+			}
+		}
+
+		for (FileInfo* fileInfo : brokenFileInfos)
+		{
+			nzbInfo->GetFileList()->Remove(fileInfo);
+		}
+	}
+
+	return true;
+}
+
+void DiskState::DiscardQuickFileInfos()
+{
+	StateFile stateFile("files", DISKSTATE_FILE_VERSION, false);
+	stateFile.Discard();
+}
+
 bool DiskState::LoadAllFileStates(DownloadQueue* downloadQueue, Servers* servers)
 {
 	BString<1024> cacheFlagFilename("%s%c%s", g_Options->GetQueueDir(), PATH_SEPARATOR, "acache");
@@ -1751,7 +1847,7 @@ bool DiskState::SaveStats(Servers* servers, ServerVolumes* serverVolumes)
 {
 	debug("Saving stats to disk");
 
-	StateFile stateFile("stats", 3, true);
+	StateFile stateFile("stats", DISKSTATE_STATS_VERSION, true);
 
 	if (servers->empty())
 	{
@@ -1779,7 +1875,7 @@ bool DiskState::LoadStats(Servers* servers, ServerVolumes* serverVolumes, bool*
 {
 	debug("Loading stats from disk");
 
-	StateFile stateFile("stats", 3, true);
+	StateFile stateFile("stats", DISKSTATE_STATS_VERSION, true);
 
 	if (!stateFile.FileExists())
 	{

daemon/queue/DiskState.h

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -38,6 +38,8 @@ public:
 	bool LoadDownloadQueue(DownloadQueue* downloadQueue, Servers* servers);
 	bool SaveFile(FileInfo* fileInfo);
 	bool LoadFile(FileInfo* fileInfo, bool fileSummary, bool articles);
+	bool SaveAllFileInfos(DownloadQueue* downloadQueue);
+	void DiscardQuickFileInfos();
 	bool SaveFileState(FileInfo* fileInfo, bool completed);
 	bool LoadFileState(FileInfo* fileInfo, Servers* servers, bool completed);
 	bool LoadArticles(FileInfo* fileInfo);
@@ -55,7 +57,7 @@ public:
 	void LoadNzbMessages(int nzbId, MessageList* messages);
 
 private:
-	bool SaveFileInfo(FileInfo* fileInfo, StateDiskFile& outfile);
+	bool SaveFileInfo(FileInfo* fileInfo, StateDiskFile& outfile, bool articles);
 	bool LoadFileInfo(FileInfo* fileInfo, StateDiskFile& outfile, int formatVersion, bool fileSummary, bool articles);
 	bool SaveFileState(FileInfo* fileInfo, StateDiskFile& outfile, bool completed);
 	bool LoadFileState(FileInfo* fileInfo, Servers* servers, StateDiskFile& infile, int formatVersion, bool completed);
@@ -76,6 +78,7 @@ private:
 	bool SaveVolumeStat(ServerVolumes* serverVolumes, StateDiskFile& outfile);
 	bool LoadVolumeStat(Servers* servers, ServerVolumes* serverVolumes, StateDiskFile& infile, int formatVersion);
 	void CalcFileStats(DownloadQueue* downloadQueue, int formatVersion);
+	bool LoadAllFileInfos(DownloadQueue* downloadQueue);
 	bool LoadAllFileStates(DownloadQueue* downloadQueue, Servers* servers);
 	void SaveServerStats(ServerStatList* serverStatList, StateDiskFile& outfile);
 	bool LoadServerStats(ServerStatList* serverStatList, Servers* servers, StateDiskFile& infile);

daemon/queue/DownloadInfo.cpp

@@ -40,7 +40,7 @@ void NzbParameterList::SetParameter(const char* name, const char* value)
 	iterator pos = std::find_if(begin(), end(),
 		[name](NzbParameter& parameter)
 		{
-			return !strcmp(parameter.GetName(), name);
+			return !strcasecmp(parameter.GetName(), name);
 		});
 
 	if (emptyVal && pos != end())
@@ -57,12 +57,11 @@ void NzbParameterList::SetParameter(const char* name, const char* value)
 	}
 }
 
-NzbParameter* NzbParameterList::Find(const char* name, bool caseSensitive)
+NzbParameter* NzbParameterList::Find(const char* name)
 {
 	for (NzbParameter& parameter : this)
 	{
-		if ((caseSensitive && !strcmp(parameter.GetName(), name)) ||
-			(!caseSensitive && !strcasecmp(parameter.GetName(), name)))
+		if (!strcasecmp(parameter.GetName(), name))
 		{
 			return &parameter;
 		}
@@ -376,13 +375,13 @@ void NzbInfo::AddMessage(Message::EKind kind, const char * text)
 
 	m_messages.emplace_back(++m_idMessageGen, kind, Util::CurrentTime(), text);
 
-	if (g_Options->GetSaveQueue() && g_Options->GetServerMode() && g_Options->GetNzbLog())
+	if (g_Options->GetServerMode() && g_Options->GetNzbLog())
 	{
 		g_DiskState->AppendNzbMessage(m_id, kind, text);
 		m_messageCount++;
 	}
 
-	while (m_messages.size() > (uint32)g_Options->GetLogBufferSize())
+	while (m_messages.size() > (uint32)g_Options->GetLogBuffer())
 	{
 		m_messages.pop_front();
 	}
@@ -474,14 +473,14 @@ void NzbInfo::SetActiveDownloads(int activeDownloads)
 		}
 		else
 		{
-			m_downloadSec = m_downloadStartSec + (Util::CurrentTime() - m_downloadStartTime);
+			m_downloadSec = m_downloadStartSec + (int)(Util::CurrentTime() - m_downloadStartTime);
 			m_downloadStartTime = 0;
 			m_changed = true;
 		}
 	}
 	else if (activeDownloads > 0)
 	{
-		m_downloadSec = m_downloadStartSec + (Util::CurrentTime() - m_downloadStartTime);
+		m_downloadSec = m_downloadStartSec + (int)(Util::CurrentTime() - m_downloadStartTime);
 		m_changed = true;
 	}
 	m_activeDownloads = activeDownloads;
@@ -652,6 +651,7 @@ void NzbInfo::UpdateCurrentStats()
 	m_currentFailedSize = m_failedSize;
 	m_parCurrentSuccessSize = m_parSuccessSize;
 	m_parCurrentFailedSize = m_parFailedSize;
+	m_extraPriority = 0;
 
 	m_currentServerStats.ListOp(&m_serverStats, ServerStatList::soSet);
 
@@ -662,6 +662,7 @@ void NzbInfo::UpdateCurrentStats()
 		m_currentFailedArticles += fileInfo->GetFailedArticles();
 		m_currentSuccessSize += fileInfo->GetSuccessSize();
 		m_currentFailedSize += fileInfo->GetFailedSize();
+		m_extraPriority += fileInfo->GetExtraPriority() ? 1 : 0;
 
 		if (fileInfo->GetPaused())
 		{
@@ -685,6 +686,7 @@ void NzbInfo::UpdateCompletedStats(FileInfo* fileInfo)
 	m_failedSize += fileInfo->GetFailedSize();
 	m_failedArticles += fileInfo->GetFailedArticles();
 	m_successArticles += fileInfo->GetSuccessArticles();
+	m_extraPriority -= fileInfo->GetExtraPriority() ? 1 : 0;
 
 	if (fileInfo->GetParFile())
 	{
@@ -713,6 +715,7 @@ void NzbInfo::UpdateDeletedStats(FileInfo* fileInfo)
 	m_currentSuccessArticles -= fileInfo->GetSuccessArticles();
 	m_currentFailedArticles -= fileInfo->GetFailedArticles() + fileInfo->GetMissedArticles();
 	m_remainingSize -= fileInfo->GetRemainingSize();
+	m_extraPriority -= fileInfo->GetExtraPriority() ? 1 : 0;
 
 	if (fileInfo->GetParFile())
 	{
@@ -796,6 +799,15 @@ void FileInfo::SetPaused(bool paused)
 	m_paused = paused;
 }
 
+void FileInfo::SetExtraPriority(bool extraPriority)
+{
+	if (m_extraPriority != extraPriority && m_nzbInfo)
+	{
+		m_nzbInfo->SetExtraPriority(m_nzbInfo->GetExtraPriority() + (extraPriority ? 1 : -1));
+	}
+	m_extraPriority = extraPriority;
+}
+
 void FileInfo::MakeValidFilename()
 {
 	m_filename = FileSystem::MakeValidFilename(m_filename);

daemon/queue/DownloadInfo.h

@@ -177,7 +177,7 @@ public:
 	bool GetOutputInitialized() { return m_outputInitialized; }
 	void SetOutputInitialized(bool outputInitialized) { m_outputInitialized = outputInitialized; }
 	bool GetExtraPriority() { return m_extraPriority; }
-	void SetExtraPriority(bool extraPriority) { m_extraPriority = extraPriority; }
+	void SetExtraPriority(bool extraPriority);
 	int GetActiveDownloads() { return m_activeDownloads; }
 	void SetActiveDownloads(int activeDownloads);
 	bool GetDupeDeleted() { return m_dupeDeleted; }
@@ -196,6 +196,8 @@ public:
 	void SetHash16k(const char* hash16k) { m_hash16k = hash16k; }
 	const char* GetParSetId() { return m_parSetId; }
 	void SetParSetId(const char* parSetId) { m_parSetId = parSetId; }
+	bool GetFlushLocked() { return m_flushLocked; }
+	void SetFlushLocked(bool flushLocked) { m_flushLocked = flushLocked; }
 
 	ServerStatList* GetServerStats() { return &m_serverStats; }
 
@@ -235,6 +237,7 @@ private:
 	uint32 m_crc = 0;
 	CString m_hash16k;
 	CString m_parSetId;
+	bool m_flushLocked = false;
 
 	static int m_idGen;
 	static int m_idMax;
@@ -304,7 +307,7 @@ class NzbParameterList : public NzbParameterListBase
 {
 public:
 	void SetParameter(const char* name, const char* value);
-	NzbParameter* Find(const char* name, bool caseSensitive);
+	NzbParameter* Find(const char* name);
 	void CopyFrom(NzbParameterList* sourceParameters);
 };
 
@@ -509,6 +512,9 @@ public:
 	void SetCurrentFailedArticles(int currentFailedArticles) { m_currentFailedArticles = currentFailedArticles; }
 	int GetPriority() { return m_priority; }
 	void SetPriority(int priority) { m_priority = priority; }
+	int GetExtraPriority() { return m_extraPriority; }
+	void SetExtraPriority(int extraPriority) { m_extraPriority = extraPriority; }
+	bool HasExtraPriority() { return m_extraPriority > 0; }
 	bool GetForcePriority() { return m_priority >= FORCE_PRIORITY; }
 	time_t GetMinTime() { return m_minTime; }
 	void SetMinTime(time_t minTime) { m_minTime = minTime; }
@@ -664,6 +670,7 @@ private:
 	time_t m_minTime = 0;
 	time_t m_maxTime = 0;
 	int m_priority = 0;
+	int m_extraPriority = 0;
 	CompletedFileList m_completedFiles;
 	EDirectRenameStatus m_directRenameStatus = tsNone;
 	EPostRenameStatus m_parRenameStatus = rsNone;

daemon/queue/DupeCoordinator.cpp

@@ -31,8 +31,8 @@ bool DupeCoordinator::SameNameOrKey(const char* name1, const char* dupeKey1,
 	const char* name2, const char* dupeKey2)
 {
 	bool hasDupeKeys = !Util::EmptyStr(dupeKey1) && !Util::EmptyStr(dupeKey2);
-	return (hasDupeKeys && !strcmp(dupeKey1, dupeKey2)) ||
-		(!hasDupeKeys && !strcmp(name1, name2));
+	return (hasDupeKeys && !strcasecmp(dupeKey1, dupeKey2)) ||
+		(!hasDupeKeys && !strcasecmp(name1, name2));
 }
 
 /**

daemon/queue/HistoryCoordinator.cpp

@@ -87,7 +87,7 @@ void HistoryCoordinator::ServiceWork()
 
 void HistoryCoordinator::DeleteDiskFiles(NzbInfo* nzbInfo)
 {
-	if (g_Options->GetSaveQueue() && g_Options->GetServerMode())
+	if (g_Options->GetServerMode())
 	{
 		// delete parked files
 		g_DiskState->DiscardFiles(nzbInfo);

daemon/queue/NzbFile.cpp

@@ -374,7 +374,7 @@ void NzbFile::ProcessFiles()
 
 	CalcHashes();
 
-	if (g_Options->GetSaveQueue() && g_Options->GetServerMode())
+	if (g_Options->GetServerMode())
 	{
 		for (FileInfo* fileInfo : m_nzbInfo->GetFileList())
 		{
@@ -595,6 +595,10 @@ bool NzbFile::ParseNzb(IUnknown* nzb)
 
 bool NzbFile::Parse()
 {
+#ifdef DISABLE_LIBXML2
+	error("Could not parse rss feed, program was compiled without libxml2 support");
+	return false;
+#else
 	xmlSAXHandler SAX_handler = {0};
 	SAX_handler.startElement = reinterpret_cast<startElementSAXFunc>(SAX_StartElement);
 	SAX_handler.endElement = reinterpret_cast<endElementSAXFunc>(SAX_EndElement);
@@ -623,6 +627,7 @@ bool NzbFile::Parse()
 	ProcessFiles();
 
 	return true;
+#endif
 }
 
 void NzbFile::Parse_StartElement(const char *name, const char **atts)
@@ -805,7 +810,11 @@ void NzbFile::SAX_characters(NzbFile* file, const char * xmlstr, int len)
 
 void* NzbFile::SAX_getEntity(NzbFile* file, const char * name)
 {
+#ifdef DISABLE_LIBXML2
+	void* e = nullptr;
+#else
 	xmlEntityPtr e = xmlGetPredefinedEntity((xmlChar* )name);
+#endif
 	if (!e)
 	{
 		file->m_nzbInfo->AddMessage(Message::mkWarning, "entity not found");

daemon/queue/QueueCoordinator.cpp

@@ -58,9 +58,10 @@ void QueueCoordinator::CoordinatorDownloadQueue::Save()
 		return;
 	}
 
-	if (g_Options->GetSaveQueue() && g_Options->GetServerMode())
+	if (g_Options->GetServerMode())
 	{
 		g_DiskState->SaveDownloadQueue(this, m_historyChanged);
+		m_stateChanged = true;
 	}
 
 	for (NzbInfo* nzbInfo : GetQueue())
@@ -102,11 +103,11 @@ void QueueCoordinator::Load()
 	bool perfectServerMatch = true;
 	bool queueLoaded = false;
 
-	if (g_Options->GetServerMode() && g_Options->GetSaveQueue())
+	if (g_Options->GetServerMode())
 	{
 		statLoaded = g_StatMeter->Load(&perfectServerMatch);
 
-		if (g_Options->GetReloadQueue() && g_DiskState->DownloadQueueExists())
+		if (g_DiskState->DownloadQueueExists())
 		{
 			queueLoaded = g_DiskState->LoadDownloadQueue(downloadQueue, g_ServerPool->GetServers());
 		}
@@ -133,7 +134,7 @@ void QueueCoordinator::Load()
 		downloadQueue->Save();
 
 		// re-save file states into diskstate to update server ids
-		if (g_Options->GetServerMode() && g_Options->GetSaveQueue())
+		if (g_Options->GetServerMode())
 		{
 			for (NzbInfo* nzbInfo : downloadQueue->GetQueue())
 			{
@@ -256,12 +257,14 @@ void QueueCoordinator::Run()
 			}
 			resetCounter = 0;
 			g_StatMeter->IntervalCheck();
+			g_Log->IntervalCheck();
 			AdjustDownloadsLimit();
 		}
 	}
 
 	WaitJobs();
 	SaveAllPartialState();
+	SaveAllFileState();
 
 	debug("Exiting QueueCoordinator-loop");
 }
@@ -331,7 +334,7 @@ NzbInfo* QueueCoordinator::AddNzbFileToQueue(std::unique_ptr<NzbInfo> nzbInfo, N
 		for (FileInfo* fileInfo: nzbInfo->GetFileList())
 		{
 			allPaused &= fileInfo->GetPaused();
-			if (g_Options->GetSaveQueue() && g_Options->GetServerMode())
+			if (g_Options->GetServerMode())
 			{
 				g_DiskState->DiscardFile(fileInfo->GetId(), true, false, false);
 			}
@@ -398,11 +401,6 @@ void QueueCoordinator::CheckDupeFileInfos(NzbInfo* nzbInfo)
 {
 	debug("CheckDupeFileInfos");
 
-	if (!g_Options->GetDupeCheck() || nzbInfo->GetDupeMode() == dmForce)
-	{
-		return;
-	}
-
 	RawFileList dupeList;
 
 	int index1 = 0;
@@ -419,9 +417,19 @@ void QueueCoordinator::CheckDupeFileInfos(NzbInfo* nzbInfo)
 				(fileInfo->GetSize() < fileInfo2->GetSize() ||
 				 (fileInfo->GetSize() == fileInfo2->GetSize() && index2 < index1)))
 			{
-				warn("File \"%s\" appears twice in collection, adding only the biggest file", fileInfo->GetFilename());
-				dupe = true;
-				break;
+				// If more than two files have same filename we don't filter them out since that
+				// naming might be intentional and correct filenames must be read from article bodies.
+				int dupeCount = std::count_if(nzbInfo->GetFileList()->begin(), nzbInfo->GetFileList()->end(),
+					[fileInfo2](std::unique_ptr<FileInfo>& fileInfo3)
+					{
+						return !strcmp(fileInfo3->GetFilename(), fileInfo2->GetFilename());
+					});
+				if (dupeCount == 2)
+				{
+					warn("File \"%s\" appears twice in collection, adding only the biggest file", fileInfo->GetFilename());
+					dupe = true;
+					break;
+				}
 			}
 		}
 		if (dupe)
@@ -435,7 +443,7 @@ void QueueCoordinator::CheckDupeFileInfos(NzbInfo* nzbInfo)
 	{
 		nzbInfo->UpdateDeletedStats(fileInfo);
 		nzbInfo->GetFileList()->Remove(fileInfo);
-		if (g_Options->GetSaveQueue() && g_Options->GetServerMode())
+		if (g_Options->GetServerMode())
 		{
 			g_DiskState->DiscardFile(fileInfo->GetId(), true, false, false);
 		}
@@ -464,8 +472,7 @@ bool QueueCoordinator::GetNextArticle(DownloadQueue* downloadQueue, FileInfo* &f
 	// if the file doesn't have any articles left for download, we store that fact and search again,
 	// ignoring all files which were previously marked as not having any articles.
 
-	// special case: if the file has ExtraPriority-flag set, it has the highest priority and the
-	// Paused-flag is ignored.
+	// special case: if the file has ExtraPriority-flag set, it has the highest priority.
 
 	//debug("QueueCoordinator::GetNextArticle()");
 
@@ -480,20 +487,34 @@ bool QueueCoordinator::GetNextArticle(DownloadQueue* downloadQueue, FileInfo* &f
 
 		for (NzbInfo* nzbInfo : downloadQueue->GetQueue())
 		{
-			for (FileInfo* fileInfo1 : nzbInfo->GetFileList())
+			bool nzbHigherPriority = fileInfo &&
+				((nzbInfo->HasExtraPriority() == fileInfo->GetNzbInfo()->HasExtraPriority() &&
+					nzbInfo->GetPriority() > fileInfo->GetNzbInfo()->GetPriority()) ||
+					(nzbInfo->HasExtraPriority() > fileInfo->GetNzbInfo()->HasExtraPriority()));
+
+			bool nzbPaused = nzbInfo->GetFileList()->size() - nzbInfo->GetPausedFileCount() <= 0;
+
+			if ((!fileInfo || nzbHigherPriority) && !nzbPaused &&
+				(!(g_Options->GetPauseDownload() || g_Options->GetQuotaReached()) || nzbInfo->GetForcePriority()))
 			{
-				if ((checkedFiles.empty() ||
-					 std::find(checkedFiles.begin(), checkedFiles.end(), fileInfo1) == checkedFiles.end()) &&
-					!fileInfo1->GetPaused() && !fileInfo1->GetDeleted() &&
-					(g_Options->GetPropagationDelay() == 0 ||
-					 (int)fileInfo1->GetTime() < (int)curDate - g_Options->GetPropagationDelay()) &&
-					(!(g_Options->GetPauseDownload() || g_Options->GetQuotaReached()) || nzbInfo->GetForcePriority()) &&
-					(!fileInfo ||
-					 (fileInfo1->GetExtraPriority() == fileInfo->GetExtraPriority() &&
-					  fileInfo1->GetNzbInfo()->GetPriority() > fileInfo->GetNzbInfo()->GetPriority()) ||
-					 (fileInfo1->GetExtraPriority() > fileInfo->GetExtraPriority())))
+				for (FileInfo* fileInfo1 : nzbInfo->GetFileList())
 				{
-					fileInfo = fileInfo1;
+					bool alreadyChecked = !checkedFiles.empty() &&
+						std::find(checkedFiles.begin(), checkedFiles.end(), fileInfo1) != checkedFiles.end();
+
+					bool propagationWait = g_Options->GetPropagationDelay() > 0 &&
+						(int)fileInfo1->GetTime() + g_Options->GetPropagationDelay() >= (int)curDate;
+
+					bool higherPriority = fileInfo &&
+						((fileInfo1->GetExtraPriority() == fileInfo->GetExtraPriority() &&
+							fileInfo1->GetNzbInfo()->GetPriority() > fileInfo->GetNzbInfo()->GetPriority()) ||
+							(fileInfo1->GetExtraPriority() > fileInfo->GetExtraPriority()));
+
+					if (!alreadyChecked && !propagationWait && !fileInfo1->GetPaused() && 
+						!fileInfo1->GetDeleted() && (!fileInfo || higherPriority))
+					{
+						fileInfo = fileInfo1;
+					}
 				}
 			}
 		}
@@ -512,7 +533,7 @@ bool QueueCoordinator::GetNextArticle(DownloadQueue* downloadQueue, FileInfo* &f
 			return true;
 		}
 
-		if (fileInfo->GetArticles()->empty() && g_Options->GetSaveQueue() && g_Options->GetServerMode())
+		if (fileInfo->GetArticles()->empty() && g_Options->GetServerMode())
 		{
 			g_DiskState->LoadArticles(fileInfo);
 			LoadPartialState(fileInfo);
@@ -546,7 +567,7 @@ bool QueueCoordinator::GetNextFirstArticle(NzbInfo* nzbInfo, FileInfo* &fileInfo
 	{
 		if (!fileInfo1->GetFilenameConfirmed())
 		{
-			if (fileInfo1->GetArticles()->empty() && g_Options->GetSaveQueue() && g_Options->GetServerMode())
+			if (fileInfo1->GetArticles()->empty() && g_Options->GetServerMode())
 			{
 				g_DiskState->LoadArticles(fileInfo1);
 				LoadPartialState(fileInfo1);
@@ -582,7 +603,7 @@ void QueueCoordinator::StartArticleDownload(FileInfo* fileInfo, ArticleInfo* art
 	articleDownloader->SetArticleInfo(articleInfo);
 	articleDownloader->SetConnection(connection);
 
-	if (articleInfo->GetPartNumber() == 1 && g_Options->GetDirectRename() && g_Options->GetDecode())
+	if (articleInfo->GetPartNumber() == 1 && g_Options->GetDirectRename() && !g_Options->GetRawArticle())
 	{
 		articleDownloader->SetContentAnalyzer(m_directRenamer.MakeArticleContentAnalyzer());
 	}
@@ -779,7 +800,7 @@ void QueueCoordinator::DeleteFileInfo(DownloadQueue* downloadQueue, FileInfo* fi
 		fileInfo->GetSuccessArticles() > 0 || fileInfo->GetFailedArticles() > 0 ? CompletedFile::cfPartial :
 		CompletedFile::cfNone;
 
-	if (g_Options->GetSaveQueue() && g_Options->GetServerMode())
+	if (g_Options->GetServerMode())
 	{
 		g_DiskState->DiscardFile(fileInfo->GetId(), fileStatus == CompletedFile::cfSuccess || (fileDeleted && !parking), true, false);
 		if (fileStatus == CompletedFile::cfPartial && (completed || parking))
@@ -847,7 +868,7 @@ void QueueCoordinator::DiscardTempFiles(FileInfo* fileInfo)
 
 void QueueCoordinator::SaveAllPartialState()
 {
-	if (!(g_Options->GetServerMode() && g_Options->GetSaveQueue()))
+	if (!g_Options->GetServerMode())
 	{
 		return;
 	}
@@ -906,6 +927,15 @@ void QueueCoordinator::LoadPartialState(FileInfo* fileInfo)
 	}
 }
 
+void QueueCoordinator::SaveAllFileState()
+{
+	if (g_Options->GetServerMode() && m_downloadQueue.m_stateChanged)
+	{
+		GuardedDownloadQueue downloadQueue = DownloadQueue::Guard();
+		g_DiskState->SaveAllFileInfos(downloadQueue);
+	}
+}
+
 void QueueCoordinator::CheckHealth(DownloadQueue* downloadQueue, FileInfo* fileInfo)
 {
 	if (g_Options->GetHealthCheck() == Options::hcNone ||
@@ -962,7 +992,7 @@ void QueueCoordinator::LogDebugInfo()
 
 void QueueCoordinator::ResetHangingDownloads()
 {
-	if (g_Options->GetTerminateTimeout() == 0 && g_Options->GetArticleTimeout() == 0)
+	if (g_Options->GetArticleTimeout() == 0)
 	{
 		return;
 	}
@@ -970,46 +1000,16 @@ void QueueCoordinator::ResetHangingDownloads()
 	GuardedDownloadQueue guard = DownloadQueue::Guard();
 	time_t tm = Util::CurrentTime();
 
-	m_activeDownloads.erase(std::remove_if(m_activeDownloads.begin(), m_activeDownloads.end(),
-		[tm](ArticleDownloader* articleDownloader)
+	for (ArticleDownloader* articleDownloader : m_activeDownloads)
+	{
+		if (tm - articleDownloader->GetLastUpdateTime() > g_Options->GetArticleTimeout() + 1 &&
+			articleDownloader->GetStatus() == ArticleDownloader::adRunning)
 		{
-			if (tm - articleDownloader->GetLastUpdateTime() > g_Options->GetArticleTimeout() + 1 &&
-				articleDownloader->GetStatus() == ArticleDownloader::adRunning)
-			{
-				error("Cancelling hanging download %s @ %s", articleDownloader->GetInfoName(),
-					articleDownloader->GetConnectionName());
-				articleDownloader->Stop();
-			}
-
-			if (tm - articleDownloader->GetLastUpdateTime() > g_Options->GetTerminateTimeout() &&
-				articleDownloader->GetStatus() == ArticleDownloader::adRunning)
-			{
-				ArticleInfo* articleInfo = articleDownloader->GetArticleInfo();
-				debug("Terminating hanging download %s", articleDownloader->GetInfoName());
-				if (articleDownloader->Terminate())
-				{
-					error("Terminated hanging download %s @ %s", articleDownloader->GetInfoName(),
-						articleDownloader->GetConnectionName());
-					articleInfo->SetStatus(ArticleInfo::aiUndefined);
-				}
-				else
-				{
-					error("Could not terminate hanging download %s @ %s", articleDownloader->GetInfoName(),
-						articleDownloader->GetConnectionName());
-				}
-
-				articleDownloader->GetFileInfo()->SetActiveDownloads(articleDownloader->GetFileInfo()->GetActiveDownloads() - 1);
-				articleDownloader->GetFileInfo()->GetNzbInfo()->SetActiveDownloads(articleDownloader->GetFileInfo()->GetNzbInfo()->GetActiveDownloads() - 1);
-				articleDownloader->GetFileInfo()->GetNzbInfo()->SetDownloadedSize(articleDownloader->GetFileInfo()->GetNzbInfo()->GetDownloadedSize() + articleDownloader->GetDownloadedSize());
-
-				// it's not safe to destroy pArticleDownloader, because the state of object is unknown
-				delete articleDownloader;
-
-				return true;
-			}
-			return false;
-		}),
-		m_activeDownloads.end());
+			error("Cancelling hanging download %s @ %s", articleDownloader->GetInfoName(),
+				articleDownloader->GetConnectionName());
+			articleDownloader->Stop();
+		}
+	}
 }
 
 /*
@@ -1292,7 +1292,7 @@ void QueueCoordinator::DirectRenameCompleted(DownloadQueue* downloadQueue, NzbIn
 {
 	for (FileInfo* fileInfo : nzbInfo->GetFileList())
 	{
-		if (g_Options->GetSaveQueue() && g_Options->GetServerMode() && !fileInfo->GetArticles()->empty())
+		if (g_Options->GetServerMode() && !fileInfo->GetArticles()->empty())
 		{
 			// save new file name into disk state file
 			g_DiskState->SaveFile(fileInfo);
@@ -1328,55 +1328,36 @@ void QueueCoordinator::DiscardDirectRename(DownloadQueue* downloadQueue, NzbInfo
 
 	for (FileInfo* fileInfo : nzbInfo->GetFileList())
 	{
-		if (fileInfo->GetParFile() && fileInfo->GetCompletedArticles() == 1 && fileInfo->GetActiveDownloads() == 0)
+		if (fileInfo->GetParFile() && fileInfo->GetCompletedArticles() == 1 &&
+			fileInfo->GetActiveDownloads() == 0)
 		{
-			// discard downloaded articles from partially downloaded par-files
-			discardedSize += fileInfo->GetSuccessSize();
-			discardedCount++;
-
-			nzbInfo->SetCurrentSuccessArticles(nzbInfo->GetCurrentSuccessArticles() - fileInfo->GetSuccessArticles());
-			nzbInfo->SetCurrentSuccessSize(nzbInfo->GetCurrentSuccessSize() - fileInfo->GetSuccessSize());
-			nzbInfo->SetParCurrentSuccessSize(nzbInfo->GetParCurrentSuccessSize() - fileInfo->GetSuccessSize());
-			fileInfo->SetSuccessSize(0);
-			fileInfo->SetSuccessArticles(0);
-
-			nzbInfo->SetCurrentFailedArticles(nzbInfo->GetCurrentFailedArticles() - fileInfo->GetFailedArticles());
-			nzbInfo->SetCurrentFailedSize(nzbInfo->GetCurrentFailedSize() - fileInfo->GetFailedSize());
-			nzbInfo->SetParCurrentFailedSize(nzbInfo->GetParCurrentFailedSize() - fileInfo->GetFailedSize());
-			fileInfo->SetFailedSize(0);
-			fileInfo->SetFailedArticles(0);
-
-			fileInfo->SetCompletedArticles(0);
-			fileInfo->SetRemainingSize(fileInfo->GetSize() - fileInfo->GetMissedSize());
-
-			// discard temporary files
-			DiscardTempFiles(fileInfo);
-			g_DiskState->DiscardFile(fileInfo->GetId(), false, true, false);
-
-			fileInfo->SetOutputFilename(nullptr);
-			fileInfo->SetOutputInitialized(false);
-			fileInfo->SetCachedArticles(0);
-			fileInfo->SetPartialChanged(false);
-			fileInfo->SetPartialState(FileInfo::psNone);
-
-			if (g_Options->GetSaveQueue() && g_Options->GetServerMode())
+			bool locked = false;
 			{
-				// free up memory used by articles if possible
-				fileInfo->GetArticles()->clear();
-			}
-			else
-			{
-				// reset article states if discarding isn't possible
-				for (ArticleInfo* articleInfo : fileInfo->GetArticles())
+				Guard contentGuard = g_ArticleCache->GuardContent();
+				locked = fileInfo->GetFlushLocked();
+				if (!locked)
 				{
-					articleInfo->SetStatus(ArticleInfo::aiUndefined);
-					articleInfo->SetResultFilename(nullptr);
-					articleInfo->DiscardSegment();
+					fileInfo->SetFlushLocked(true);
 				}
 			}
+
+			if (!locked)
+			{
+				// discard downloaded articles from partially downloaded par-files
+				discardedSize += fileInfo->GetSuccessSize();
+				discardedCount++;
+
+				DiscardDownloadedArticles(nzbInfo, fileInfo);
+			}
+
+			if (!locked)
+			{
+				Guard contentGuard = g_ArticleCache->GuardContent();
+				fileInfo->SetFlushLocked(false);
+			}
 		}
 
-		if (g_Options->GetSaveQueue() && g_Options->GetServerMode() &&
+		if (g_Options->GetServerMode() &&
 			!fileInfo->GetArticles()->empty() && g_Options->GetContinuePartial() &&
 			fileInfo->GetActiveDownloads() == 0 && fileInfo->GetCachedArticles() == 0)
 		{
@@ -1394,3 +1375,55 @@ void QueueCoordinator::DiscardDirectRename(DownloadQueue* downloadQueue, NzbInfo
 			*Util::FormatSize(discardedSize), discardedCount);
 	}
 }
+
+void QueueCoordinator::DiscardDownloadedArticles(NzbInfo* nzbInfo, FileInfo* fileInfo)
+{
+	nzbInfo->SetRemainingSize(nzbInfo->GetRemainingSize() + fileInfo->GetSuccessSize() + fileInfo->GetFailedSize());
+	if (fileInfo->GetPaused())
+	{
+		nzbInfo->SetPausedSize(nzbInfo->GetPausedSize() + fileInfo->GetSuccessSize() + fileInfo->GetFailedSize());
+	}
+	nzbInfo->GetCurrentServerStats()->ListOp(fileInfo->GetServerStats(), ServerStatList::soSubtract);
+	fileInfo->GetServerStats()->clear();
+
+	nzbInfo->SetCurrentSuccessArticles(nzbInfo->GetCurrentSuccessArticles() - fileInfo->GetSuccessArticles());
+	nzbInfo->SetCurrentSuccessSize(nzbInfo->GetCurrentSuccessSize() - fileInfo->GetSuccessSize());
+	nzbInfo->SetParCurrentSuccessSize(nzbInfo->GetParCurrentSuccessSize() - fileInfo->GetSuccessSize());
+	fileInfo->SetSuccessSize(0);
+	fileInfo->SetSuccessArticles(0);
+
+	nzbInfo->SetCurrentFailedArticles(nzbInfo->GetCurrentFailedArticles() - fileInfo->GetFailedArticles());
+	nzbInfo->SetCurrentFailedSize(nzbInfo->GetCurrentFailedSize() - fileInfo->GetFailedSize());
+	nzbInfo->SetParCurrentFailedSize(nzbInfo->GetParCurrentFailedSize() - fileInfo->GetFailedSize());
+	fileInfo->SetFailedSize(0);
+	fileInfo->SetFailedArticles(0);
+
+	fileInfo->SetCompletedArticles(0);
+	fileInfo->SetRemainingSize(fileInfo->GetSize() - fileInfo->GetMissedSize());
+
+	// discard temporary files
+	DiscardTempFiles(fileInfo);
+	g_DiskState->DiscardFile(fileInfo->GetId(), false, true, false);
+
+	fileInfo->SetOutputFilename(nullptr);
+	fileInfo->SetOutputInitialized(false);
+	fileInfo->SetCachedArticles(0);
+	fileInfo->SetPartialChanged(false);
+	fileInfo->SetPartialState(FileInfo::psNone);
+
+	if (g_Options->GetServerMode())
+	{
+		// free up memory used by articles if possible
+		fileInfo->GetArticles()->clear();
+	}
+	else
+	{
+		// reset article states if discarding isn't possible
+		for (ArticleInfo* articleInfo : fileInfo->GetArticles())
+		{
+			articleInfo->SetStatus(ArticleInfo::aiUndefined);
+			articleInfo->SetResultFilename(nullptr);
+			articleInfo->DiscardSegment();
+		}
+	}
+}

daemon/queue/QueueCoordinator.h

@@ -72,6 +72,7 @@ private:
 		bool m_massEdit = false;
 		bool m_wantSave = false;
 		bool m_historyChanged = false;
+		bool m_stateChanged = false;
 		friend class QueueCoordinator;
 	};
 
@@ -102,6 +103,7 @@ private:
 	void DeleteFileInfo(DownloadQueue* downloadQueue, FileInfo* fileInfo, bool completed);
 	void DirectRenameCompleted(DownloadQueue* downloadQueue, NzbInfo* nzbInfo);
 	void DiscardDirectRename(DownloadQueue* downloadQueue, NzbInfo* nzbInfo);
+	void DiscardDownloadedArticles(NzbInfo* nzbInfo, FileInfo* fileInfo);
 	void CheckHealth(DownloadQueue* downloadQueue, FileInfo* fileInfo);
 	void ResetHangingDownloads();
 	void AdjustDownloadsLimit();
@@ -109,6 +111,7 @@ private:
 	void SaveAllPartialState();
 	void SavePartialState(FileInfo* fileInfo);
 	void LoadPartialState(FileInfo* fileInfo);
+	void SaveAllFileState();
 	void WaitJobs();
 };
 

daemon/queue/QueueEditor.cpp

@@ -650,7 +650,7 @@ void QueueEditor::PrepareList(ItemList* itemList, IdList* idList,
 bool QueueEditor::BuildIdListFromNameList(IdList* idList, NameList* nameList, DownloadQueue::EMatchMode matchMode, DownloadQueue::EEditAction action)
 {
 #ifndef HAVE_REGEX_H
-	if (matchMode == mmRegEx)
+	if (matchMode == DownloadQueue::mmRegEx)
 	{
 		return false;
 	}
@@ -840,10 +840,7 @@ void QueueEditor::PausePars(RawFileList* fileList, bool extraParsOnly)
 
 	for (FileInfo* fileInfo : fileList)
 	{
-		BString<1024> loFileName = fileInfo->GetFilename();
-		for (char* p = loFileName; *p; p++) *p = tolower(*p); // convert string to lowercase
-
-		if (strstr(loFileName, ".par2"))
+		if (fileInfo->GetParFile())
 		{
 			if (!extraParsOnly)
 			{
@@ -851,6 +848,8 @@ void QueueEditor::PausePars(RawFileList* fileList, bool extraParsOnly)
 			}
 			else
 			{
+				BString<1024> loFileName = fileInfo->GetFilename();
+				for (char* p = loFileName; *p; p++) *p = tolower(*p); // convert string to lowercase
 				if (strstr(loFileName, ".vol"))
 				{
 					Vols.push_back(fileInfo);

daemon/queue/Scanner.cpp

@@ -371,7 +371,10 @@ void Scanner::InitPPParameters(const char* category, NzbParameterList* parameter
 		}
 	}
 
-	parameters->SetParameter("*Unpack:", unpack ? "yes" : "no");
+	if (!parameters->Find("*Unpack:"))
+	{
+		parameters->SetParameter("*Unpack:", unpack ? "yes" : "no");
+	}
 
 	if (!Util::EmptyStr(extensions))
 	{
@@ -381,10 +384,12 @@ void Scanner::InitPPParameters(const char* category, NzbParameterList* parameter
 		{
 			for (ScriptConfig::Script& script : g_ScriptConfig->GetScripts())
 			{
+				BString<1024> paramName("%s:", scriptName);
 				if ((script.GetPostScript() || script.GetQueueScript()) &&
+					!parameters->Find(paramName) &&
 					FileSystem::SameFilename(scriptName, script.GetName()))
 				{
-					parameters->SetParameter(BString<1024>("%s:", scriptName), "yes");
+					parameters->SetParameter(paramName, "yes");
 				}
 			}
 		}

daemon/queue/UrlCoordinator.cpp

@@ -158,8 +158,7 @@ void UrlCoordinator::Stop()
 
 void UrlCoordinator::ResetHangingDownloads()
 {
-	const int timeout = g_Options->GetTerminateTimeout();
-	if (timeout == 0)
+	if (g_Options->GetUrlTimeout() == 0)
 	{
 		return;
 	}
@@ -167,32 +166,15 @@ void UrlCoordinator::ResetHangingDownloads()
 	GuardedDownloadQueue guard = DownloadQueue::Guard();
 	time_t tm = Util::CurrentTime();
 
-	m_activeDownloads.erase(std::remove_if(m_activeDownloads.begin(), m_activeDownloads.end(),
-		[timeout, tm](UrlDownloader* urlDownloader)
+	for (UrlDownloader* urlDownloader: m_activeDownloads)
+	{
+		if (tm - urlDownloader->GetLastUpdateTime() > g_Options->GetUrlTimeout() + 10 &&
+			urlDownloader->GetStatus() == UrlDownloader::adRunning)
 		{
-			if (tm - urlDownloader->GetLastUpdateTime() > timeout &&
-				urlDownloader->GetStatus() == UrlDownloader::adRunning)
-			{
-				NzbInfo* nzbInfo = urlDownloader->GetNzbInfo();
-				debug("Terminating hanging download %s", urlDownloader->GetInfoName());
-				if (urlDownloader->Terminate())
-				{
-					error("Terminated hanging download %s", urlDownloader->GetInfoName());
-					nzbInfo->SetUrlStatus(NzbInfo::lsNone);
-				}
-				else
-				{
-					error("Could not terminate hanging download %s", urlDownloader->GetInfoName());
-				}
-
-				// it's not safe to destroy urlDownloader, because the state of object is unknown
-				delete urlDownloader;
-
-				return true;
-			}
-			return false;
-		}),
-		m_activeDownloads.end());
+			error("Cancelling hanging url download %s", urlDownloader->GetInfoName());
+			urlDownloader->Stop();
+		}
+	}
 }
 
 void UrlCoordinator::LogDebugInfo()

daemon/remote/BinRpc.cpp

@@ -871,7 +871,7 @@ void EditQueueBinCommand::Execute()
 	else
 	{
 #ifndef HAVE_REGEX_H
-		if ((QueueEditor::EMatchMode)matchMode == QueueEditor::mmRegEx)
+		if ((DownloadQueue::EMatchMode)matchMode == DownloadQueue::mmRegEx)
 		{
 			SendBoolResponse(false, "Edit-Command failed: the program was compiled without RegEx-support");
 			return;

daemon/remote/RemoteServer.cpp

@@ -2,7 +2,7 @@
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
  *  Copyright (C) 2005 Bo Cordes Petersen <placebodk@sourceforge.net>
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -60,7 +60,7 @@ void RemoteServer::Run()
 			m_connection = std::make_unique<Connection>(g_Options->GetControlIp(),
 				m_tls ? g_Options->GetSecurePort() : g_Options->GetControlPort(),
 				m_tls);
-			m_connection->SetTimeout(g_Options->GetUrlTimeout());
+			m_connection->SetTimeout(g_Options->GetRemoteTimeout());
 			m_connection->SetSuppressErrors(false);
 			bind = m_connection->Bind();
 		}
@@ -83,13 +83,19 @@ void RemoteServer::Run()
 			continue;
 		}
 
-		RequestProcessor* commandThread = new RequestProcessor();
-		commandThread->SetAutoDestroy(true);
-		commandThread->SetConnection(std::move(acceptedConnection));
+		if (!IsStopped())
+		{
+			RequestProcessor* commandThread = new RequestProcessor();
+			commandThread->SetAutoDestroy(true);
+			commandThread->SetConnection(std::move(acceptedConnection));
 #ifndef DISABLE_TLS
-		commandThread->SetTls(m_tls);
+			commandThread->SetTls(m_tls);
 #endif
-		commandThread->Start();
+			Guard guard(m_processorsMutex);
+			m_activeProcessors.push_back(commandThread);
+			commandThread->Attach(this);
+			commandThread->Start();
+		}
 	}
 
 	if (m_connection)
@@ -97,6 +103,19 @@ void RemoteServer::Run()
 		m_connection->Disconnect();
 	}
 
+	// waiting for request processors
+	debug("RemoteServer: waiting for request processor to complete");
+	bool completed = false;
+	while (!completed)
+	{
+		{
+			Guard guard(m_processorsMutex);
+			completed = m_activeProcessors.size() == 0;
+		}
+		usleep(100 * 1000);
+	}
+	debug("RemoteServer: request processor are completed");
+
 	debug("Exiting RemoteServer-loop");
 }
 
@@ -106,11 +125,39 @@ void RemoteServer::Stop()
 	if (m_connection)
 	{
 		m_connection->SetSuppressErrors(true);
+		m_connection->SetForceClose(true);
 		m_connection->Cancel();
-#ifdef WIN32
-		m_connection->Disconnect();
-#endif
+
+		debug("Stopping RequestProcessors");
+		Guard guard(m_processorsMutex);
+		for (RequestProcessor* requestProcessor : m_activeProcessors)
+		{
+			requestProcessor->Stop();
+		}
+		debug("RequestProcessors are notified");
 	}
+	debug("RemoteServer stop end");
+}
+
+void RemoteServer::ForceStop()
+{
+	debug("Killing RequestProcessors");
+	Guard guard(m_processorsMutex);
+	for (RequestProcessor* requestProcessor : m_activeProcessors)
+	{
+		requestProcessor->Kill();
+	}
+	m_activeProcessors.clear();
+	debug("RequestProcessors are killed");
+}
+
+void RemoteServer::Update(Subject* caller, void* aspect)
+{
+	debug("Notification from RequestProcessor received");
+
+	RequestProcessor* requestProcessor = (RequestProcessor*)caller;
+	Guard guard(m_processorsMutex);
+	m_activeProcessors.erase(std::find(m_activeProcessors.begin(), m_activeProcessors.end(), requestProcessor));
 }
 
 //*****************************************************************
@@ -123,8 +170,22 @@ RequestProcessor::~RequestProcessor()
 
 void RequestProcessor::Run()
 {
-	bool ok = false;
+	Execute();
+	Notify(nullptr);
+}
 
+void RequestProcessor::Stop()
+{
+	Thread::Stop();
+#ifdef WIN32
+	m_connection->SetForceClose(true);
+#endif
+	m_connection->Cancel();
+}
+
+void RequestProcessor::Execute()
+{
+	bool ok = false;
 	m_connection->SetSuppressErrors(true);
 
 #ifndef DISABLE_TLS
@@ -136,7 +197,7 @@ void RequestProcessor::Run()
 #endif
 
 	// Read the first 4 bytes to determine request type
-	int signature = 0;
+	uint32 signature = 0;
 	if (!m_connection->Recv((char*)&signature, 4))
 	{
 		debug("Could not read request signature");
@@ -156,39 +217,18 @@ void RequestProcessor::Run()
 		!strncmp((char*)&signature, "OPTI", 4))
 	{
 		// HTTP request received
-		char buffer[1024];
-		if (m_connection->ReadLine(buffer, sizeof(buffer), nullptr))
+		ok = true;
+		while (ServWebRequest((char*)&signature))
 		{
-			WebProcessor::EHttpMethod httpMethod = WebProcessor::hmGet;
-			char* url = buffer;
-			if (!strncmp((char*)&signature, "POST", 4))
+			if (!m_connection->Recv((char*)&signature, 4))
 			{
-				httpMethod = WebProcessor::hmPost;
-				url++;
+				debug("Could not read request signature");
+				break;
 			}
-			if (!strncmp((char*)&signature, "OPTI", 4) && strlen(url) > 4)
-			{
-				httpMethod = WebProcessor::hmOptions;
-				url += 4;
-			}
-			if (char* p = strchr(url, ' '))
-			{
-				*p = '\0';
-			}
-
-			debug("url: %s", url);
-
-			WebProcessor processor;
-			processor.SetConnection(m_connection.get());
-			processor.SetUrl(url);
-			processor.SetHttpMethod(httpMethod);
-			processor.Execute();
-
-			m_connection->SetGracefull(true);
-			m_connection->Disconnect();
-
-			ok = true;
 		}
+
+		m_connection->SetGracefull(true);
+		m_connection->Disconnect();
 	}
 
 	if (!ok)
@@ -196,3 +236,45 @@ void RequestProcessor::Run()
 		warn("Non-nzbget request received on port %i from %s", m_tls ? g_Options->GetSecurePort() : g_Options->GetControlPort(), m_connection->GetRemoteAddr());
 	}
 }
+
+bool RequestProcessor::ServWebRequest(const char* signature)
+{
+	// HTTP request received
+	char buffer[1024];
+	if (!m_connection->ReadLine(buffer, sizeof(buffer), nullptr))
+	{
+		return false;
+	}
+
+	WebProcessor::EHttpMethod httpMethod = WebProcessor::hmGet;
+	char* url = buffer;
+	if (!strncmp(signature, "POST", 4))
+	{
+		httpMethod = WebProcessor::hmPost;
+		url++;
+	}
+	else if (!strncmp(signature, "OPTI", 4) && strlen(url) > 4)
+	{
+		httpMethod = WebProcessor::hmOptions;
+		url += 4;
+	}
+	else if (!(!strncmp(signature, "GET ", 4)))
+	{
+		return false;
+	}
+
+	if (char* p = strchr(url, ' '))
+	{
+		*p = '\0';
+	}
+
+	debug("url: %s", url);
+
+	WebProcessor processor;
+	processor.SetConnection(m_connection.get());
+	processor.SetUrl(url);
+	processor.SetHttpMethod(httpMethod);
+	processor.Execute();
+
+	return processor.GetKeepAlive();
+}

daemon/remote/RemoteServer.h

@@ -2,7 +2,7 @@
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
  *  Copyright (C) 2005 Bo Cordes Petersen <placebodk@users.sourceforge.net>
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -24,30 +24,43 @@
 
 #include "Thread.h"
 #include "Connection.h"
+#include "Observer.h"
 
-class RemoteServer : public Thread
+class RequestProcessor;
+
+class RemoteServer : public Thread, public Observer
 {
 public:
 	RemoteServer(bool tls) : m_tls(tls) {}
 	virtual void Run();
 	virtual void Stop();
+	void ForceStop();
+	void Update(Subject* caller, void* aspect);
 
 private:
+	typedef std::deque<RequestProcessor*> RequestProcessors;
+
 	bool m_tls;
 	std::unique_ptr<Connection> m_connection;
+	RequestProcessors m_activeProcessors;
+	Mutex m_processorsMutex;
 };
 
-class RequestProcessor : public Thread
+class RequestProcessor : public Thread, public Subject
 {
 public:
 	~RequestProcessor();
 	virtual void Run();
+	virtual void Stop();
 	void SetTls(bool tls) { m_tls = tls; }
 	void SetConnection(std::unique_ptr<Connection>&& connection) { m_connection = std::move(connection); }
 
 private:
 	bool m_tls;
 	std::unique_ptr<Connection> m_connection;
+
+	bool ServWebRequest(const char* signature);
+	void Execute();
 };
 
 #endif

daemon/remote/WebServer.cpp

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2012-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2012-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -26,6 +26,13 @@
 #include "Util.h"
 #include "FileSystem.h"
 
+#ifndef DISABLE_PARCHECK
+#include "par2cmdline.h"
+#include "md5.h"
+#endif
+
+static const char* ERR_HTTP_OK = "200 OK";
+static const char* ERR_HTTP_NOT_MODIFIED = "304 Not Modified";
 static const char* ERR_HTTP_BAD_REQUEST = "400 Bad Request";
 static const char* ERR_HTTP_NOT_FOUND = "404 Not Found";
 static const char* ERR_HTTP_SERVICE_UNAVAILABLE = "503 Service Unavailable";
@@ -174,6 +181,14 @@ void WebProcessor::ParseHeaders()
 		{
 			m_forwardedFor = p + 17;
 		}
+		else if (!strncasecmp(p, "If-None-Match: ", 15))
+		{
+			m_oldETag = p + 15;
+		}
+		else if (!strncasecmp(p, "Connection: keep-alive", 22))
+		{
+			m_keepAlive = true;
+		}
 		else if (*p == '\0')
 		{
 			break;
@@ -268,10 +283,10 @@ bool WebProcessor::CheckCredentials()
 		}
 		else
 		{
-			warn("Request received on port %i from %s%s, but username or password invalid (%s:%s)",
+			warn("Request received on port %i from %s%s, but username (%s) or password invalid",
 				g_Options->GetControlPort(), m_connection->GetRemoteAddr(),
 				!m_forwardedFor.Empty() ? (char*)BString<1024>(" (forwarded for: %s)", *m_forwardedFor) : "",
-				m_authInfo, pw);
+				m_authInfo);
 			return false;
 		}
 	}
@@ -314,7 +329,7 @@ void WebProcessor::Dispatch()
 		processor.SetUserAccess((XmlRpcProcessor::EUserAccess)m_userAccess);
 		processor.SetUrl(m_url);
 		processor.Execute();
-		SendBodyResponse(processor.GetResponse(), strlen(processor.GetResponse()), processor.GetContentType());
+		SendBodyResponse(processor.GetResponse(), strlen(processor.GetResponse()), processor.GetContentType(), processor.IsSafeMethod());
 		return;
 	}
 
@@ -330,43 +345,42 @@ void WebProcessor::Dispatch()
 		return;
 	}
 
-	// for security reasons we allow only characters "0..9 A..Z a..z . - _ /" in the URLs
+	// for security reasons we allow only characters "0..9 A..Z a..z . - + _ / ?" in the URLs
 	// we also don't allow ".." in the URLs
 	for (char *p = m_url; *p; p++)
 	{
 		if (!((*p >= '0' && *p <= '9') || (*p >= 'A' && *p <= 'Z') || (*p >= 'a' && *p <= 'z') ||
-			*p == '.' || *p == '-' || *p == '_' || *p == '/') || (*p == '.' && p[1] == '.'))
+			*p == '.' || *p == '-' || *p == '+' || *p == '?' || *p == '_' || *p == '/') || (*p == '.' && p[1] == '.'))
 		{
 			SendErrorResponse(ERR_HTTP_NOT_FOUND, true);
 			return;
 		}
 	}
 
-	const char *defRes = "";
-	if (m_url[strlen(m_url)-1] == '/')
+	if (!strncmp(m_url, "/combined.", 10) && strchr(m_url, '?'))
 	{
-		// default file in directory (if not specified) is "index.html"
-		defRes = "index.html";
+		SendMultiFileResponse();
+	}
+	else
+	{
+		SendSingleFileResponse();
 	}
-
-	BString<1024> disk_filename("%s%s%s", g_Options->GetWebDir(), *m_url, defRes);
-
-	SendFileResponse(disk_filename);
 }
 
 void WebProcessor::SendAuthResponse()
 {
 	const char* AUTH_RESPONSE_HEADER =
-		"HTTP/1.0 401 Unauthorized\r\n"
+		"HTTP/1.1 401 Unauthorized\r\n"
 		"%s"
-		"Connection: close\r\n"
+		"Connection: %s\r\n"
 		"Content-Type: text/plain\r\n"
+		"Content-Length: 0\r\n"
 		"Server: nzbget-%s\r\n"
 		"\r\n";
 
 	BString<1024> responseHeader(AUTH_RESPONSE_HEADER,
 		g_Options->GetFormAuth() ? "" : "WWW-Authenticate: Basic realm=\"NZBGet\"\r\n",
-		Util::VersionRevision());
+		m_keepAlive ? "keep-alive" : "close", Util::VersionRevision());
 
 	// Send the response answer
 	debug("ResponseHeader=%s", *responseHeader);
@@ -377,8 +391,9 @@ void WebProcessor::SendOptionsResponse()
 {
 	const char* OPTIONS_RESPONSE_HEADER =
 		"HTTP/1.1 200 OK\r\n"
-		"Connection: close\r\n"
+		"Connection: %s\r\n"
 		//"Content-Type: plain/text\r\n"
+		"Content-Length: 0\r\n"
 		"Access-Control-Allow-Methods: GET, POST, OPTIONS\r\n"
 		"Access-Control-Allow-Origin: %s\r\n"
 		"Access-Control-Allow-Credentials: true\r\n"
@@ -387,6 +402,7 @@ void WebProcessor::SendOptionsResponse()
 		"Server: nzbget-%s\r\n"
 		"\r\n";
 	BString<1024> responseHeader(OPTIONS_RESPONSE_HEADER,
+		m_keepAlive ? "keep-alive" : "close", 
 		m_origin.Str(), Util::VersionRevision());
 
 	// Send the response answer
@@ -397,8 +413,8 @@ void WebProcessor::SendOptionsResponse()
 void WebProcessor::SendErrorResponse(const char* errCode, bool printWarning)
 {
 	const char* RESPONSE_HEADER =
-		"HTTP/1.0 %s\r\n"
-		"Connection: close\r\n"
+		"HTTP/1.1 %s\r\n"
+		"Connection: %s\r\n"
 		"Content-Length: %i\r\n"
 		"Content-Type: text/html\r\n"
 		"Server: nzbget-%s\r\n"
@@ -413,7 +429,9 @@ void WebProcessor::SendErrorResponse(const char* errCode, bool printWarning)
 		errCode, errCode);
 	int pageContentLen = responseBody.Length();
 
-	BString<1024> responseHeader(RESPONSE_HEADER, errCode, pageContentLen, Util::VersionRevision());
+	BString<1024> responseHeader(RESPONSE_HEADER, errCode,
+		m_keepAlive ? "keep-alive" : "close",
+		pageContentLen, Util::VersionRevision());
 
 	// Send the response answer
 	m_connection->Send(responseHeader, responseHeader.Length());
@@ -423,36 +441,66 @@ void WebProcessor::SendErrorResponse(const char* errCode, bool printWarning)
 void WebProcessor::SendRedirectResponse(const char* url)
 {
 	const char* REDIRECT_RESPONSE_HEADER =
-		"HTTP/1.0 301 Moved Permanently\r\n"
+		"HTTP/1.1 301 Moved Permanently\r\n"
 		"Location: %s\r\n"
-		"Connection: close\r\n"
+		"Connection: %s\r\n"
+		"Content-Length: 0\r\n"
 		"Server: nzbget-%s\r\n"
 		"\r\n";
-	BString<1024> responseHeader(REDIRECT_RESPONSE_HEADER, url, Util::VersionRevision());
+	BString<1024> responseHeader(REDIRECT_RESPONSE_HEADER, url,
+		m_keepAlive ? "keep-alive" : "close", Util::VersionRevision());
 
 	// Send the response answer
 	debug("ResponseHeader=%s", *responseHeader);
 	m_connection->Send(responseHeader, responseHeader.Length());
 }
 
-void WebProcessor::SendBodyResponse(const char* body, int bodyLen, const char* contentType)
+void WebProcessor::SendBodyResponse(const char* body, int bodyLen, const char* contentType, bool cachable)
 {
 	const char* RESPONSE_HEADER =
-		"HTTP/1.1 200 OK\r\n"
-		"Connection: close\r\n"
+		"HTTP/1.1 %s\r\n"
+		"Connection: %s\r\n"
 		"Access-Control-Allow-Methods: GET, POST, OPTIONS\r\n"
 		"Access-Control-Allow-Origin: %s\r\n"
 		"Access-Control-Allow-Credentials: true\r\n"
 		"Access-Control-Max-Age: 86400\r\n"
 		"Access-Control-Allow-Headers: Content-Type, Authorization\r\n"
 		"Set-Cookie: Auth-Type=%s\r\n"
-		"Set-Cookie: Auth-Token=%s\r\n"
+		"Set-Cookie: Auth-Token=%s; HttpOnly\r\n"
 		"Content-Length: %i\r\n"
 		"%s"					// Content-Type: xxx
 		"%s"					// Content-Encoding: gzip
+		"%s"					// ETag
 		"Server: nzbget-%s\r\n"
 		"\r\n";
 
+	BString<1024> eTagHeader;
+	bool unchanged = false;
+
+	if (cachable)
+	{
+		BString<1024> newETag;
+
+#ifndef DISABLE_PARCHECK
+		Par2::MD5Hash hash;
+		Par2::MD5Context md5;
+		md5.Update(body, bodyLen);
+		md5.Final(hash);
+		newETag.Format("\"%s\"", hash.print().c_str());
+#else
+		uint32 hash = Util::HashBJ96(body, bodyLen, 0);
+		newETag.Format("\"%x\"", hash);
+#endif
+
+		unchanged = m_oldETag && !strcmp(newETag, m_oldETag);
+		if (unchanged)
+		{
+			body = "";
+			bodyLen = 0;
+		}
+		eTagHeader.Format("ETag: %s\r\n", *newETag);
+	}
+
 #ifndef DISABLE_GZIP
 	CharBuffer gbuf;
 	bool gzip = m_gzip && bodyLen > MAX_UNCOMPRESSED_SIZE;
@@ -482,24 +530,39 @@ void WebProcessor::SendBodyResponse(const char* body, int bodyLen, const char* c
 	}
 
 	BString<1024> responseHeader(RESPONSE_HEADER,
+		unchanged ? ERR_HTTP_NOT_MODIFIED : ERR_HTTP_OK,
+		m_keepAlive ? "keep-alive" : "close",
 		m_origin.Str(),
 		g_Options->GetFormAuth() ? "form" : "http",
 		m_authorized ? m_serverAuthToken[m_userAccess] : "",
-		bodyLen, *contentTypeHeader,
+		bodyLen,
+		*contentTypeHeader,
 		gzip ? "Content-Encoding: gzip\r\n" : "",
+		cachable ? *eTagHeader : "",
 		Util::VersionRevision());
 
+	debug("[%s] (%s) %s", *m_url, *m_oldETag, *responseHeader);
+
 	// Send the request answer
 	m_connection->Send(responseHeader, responseHeader.Length());
 	m_connection->Send(body, bodyLen);
 }
 
-void WebProcessor::SendFileResponse(const char* filename)
+void WebProcessor::SendSingleFileResponse()
 {
-	debug("serving file: %s", filename);
+	const char *defRes = "";
+	if (m_url[strlen(m_url)-1] == '/')
+	{
+		// default file in directory (if not specified) is "index.html"
+		defRes = "index.html";
+	}
+
+	BString<1024> filename("%s%s%s", g_Options->GetWebDir(), *m_url, defRes);
+
+	debug("serving file: %s", *filename);
 
 	CharBuffer body;
-	if (!FileSystem::LoadFileIntoBuffer(filename, body, false))
+	if (!FileSystem::LoadFileIntoBuffer(filename, body, true))
 	{
 		// do not print warnings "404 not found" for certain files
 		bool ignorable = !strcmp(filename, "package-info.json") ||
@@ -510,7 +573,48 @@ void WebProcessor::SendFileResponse(const char* filename)
 		return;
 	}
 
-	SendBodyResponse(body, body.Size(), DetectContentType(filename));
+	const char* contentType = DetectContentType(filename);
+	int len = body.Size() - 1;
+
+#ifdef DEBUG
+	if (contentType && !strcmp(contentType, "text/html"))
+	{
+		Util::ReduceStr(body, "<!-- %if-debug%", "");
+		Util::ReduceStr(body, "<!-- %if-not-debug% -->", "<!--");
+		Util::ReduceStr(body, "<!-- %end% -->", "-->");
+		Util::ReduceStr(body, "%end% -->", "");
+		len = strlen(body);
+	}
+#endif
+
+	SendBodyResponse(body, len, contentType, true);
+}
+
+void WebProcessor::SendMultiFileResponse()
+{
+	debug("serving multiple files: %s", *m_url);
+
+	StringBuilder response;
+	char* filelist = strchr(m_url, '?');
+	*filelist++ = '\0';
+
+	Tokenizer tok(filelist, "+");
+	while (const char* filename = tok.Next())
+	{
+		BString<1024> diskFilename("%s%c%s", g_Options->GetWebDir(), PATH_SEPARATOR, filename);
+
+		CharBuffer body;
+		if (!FileSystem::LoadFileIntoBuffer(diskFilename, body, true))
+		{
+			warn("Web-Server: %s, Resource: /%s", ERR_HTTP_NOT_FOUND, filename);
+			SendErrorResponse(ERR_HTTP_NOT_FOUND, false);
+			return;
+		}
+
+		response.Append(body);
+	}
+
+	SendBodyResponse(response, response.Length(), DetectContentType(m_url), true);
 }
 
 const char* WebProcessor::DetectContentType(const char* filename)

daemon/remote/WebServer.h

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2012-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2012-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -39,6 +39,7 @@ public:
 	void SetConnection(Connection* connection) { m_connection = connection; }
 	void SetUrl(const char* url) { m_url = url; }
 	void SetHttpMethod(EHttpMethod httpMethod) { m_httpMethod = httpMethod; }
+	bool GetKeepAlive() { return m_keepAlive; }
 
 private:
 	enum EUserAccess
@@ -62,13 +63,16 @@ private:
 	char m_authToken[48+1];
 	static char m_serverAuthToken[3][48+1];
 	CString m_forwardedFor;
+	CString m_oldETag;
+	bool m_keepAlive = false;
 
 	void Dispatch();
 	void SendAuthResponse();
 	void SendOptionsResponse();
 	void SendErrorResponse(const char* errCode, bool printWarning);
-	void SendFileResponse(const char* filename);
-	void SendBodyResponse(const char* body, int bodyLen, const char* contentType);
+	void SendSingleFileResponse();
+	void SendMultiFileResponse();
+	void SendBodyResponse(const char* body, int bodyLen, const char* contentType, bool cachable);
 	void SendRedirectResponse(const char* url);
 	const char* DetectContentType(const char* filename);
 	bool IsAuthorizedIp(const char* remoteAddr);

daemon/remote/XmlRpc.cpp

@@ -38,12 +38,19 @@
 extern void ExitProc();
 extern void Reload();
 
+class SafeXmlCommand: public XmlCommand
+{
+public:
+	virtual bool IsSafeMethod() { return true; };
+};
+
 class ErrorXmlCommand: public XmlCommand
 {
 public:
 	ErrorXmlCommand(int errCode, const char* errText) :
 		m_errCode(errCode), m_errText(errText) {}
 	virtual void Execute();
+	virtual bool IsError() { return true; };
 
 private:
 	int m_errCode;
@@ -87,13 +94,13 @@ public:
 	virtual void Execute();
 };
 
-class VersionXmlCommand: public XmlCommand
+class VersionXmlCommand: public SafeXmlCommand
 {
 public:
 	virtual void Execute();
 };
 
-class DumpDebugXmlCommand: public XmlCommand
+class DumpDebugXmlCommand: public SafeXmlCommand
 {
 public:
 	virtual void Execute();
@@ -105,13 +112,13 @@ public:
 	virtual void Execute();
 };
 
-class StatusXmlCommand: public XmlCommand
+class StatusXmlCommand: public SafeXmlCommand
 {
 public:
 	virtual void Execute();
 };
 
-class LogXmlCommand: public XmlCommand
+class LogXmlCommand: public SafeXmlCommand
 {
 public:
 	virtual void Execute();
@@ -122,14 +129,14 @@ protected:
 	virtual GuardedMessageList GuardMessages();
 };
 
-class NzbInfoXmlCommand: public XmlCommand
+class NzbInfoXmlCommand: public SafeXmlCommand
 {
 protected:
 	void AppendNzbInfoFields(NzbInfo* nzbInfo);
 	void AppendPostInfoFields(PostInfo* postInfo, int logEntries, bool postQueue);
 };
 
-class ListFilesXmlCommand: public XmlCommand
+class ListFilesXmlCommand: public SafeXmlCommand
 {
 public:
 	virtual void Execute();
@@ -187,19 +194,19 @@ private:
 	const char* DetectStatus(HistoryInfo* historyInfo);
 };
 
-class UrlQueueXmlCommand: public XmlCommand
+class UrlQueueXmlCommand: public SafeXmlCommand
 {
 public:
 	virtual void Execute();
 };
 
-class ConfigXmlCommand: public XmlCommand
+class ConfigXmlCommand: public SafeXmlCommand
 {
 public:
 	virtual void Execute();
 };
 
-class LoadConfigXmlCommand: public XmlCommand
+class LoadConfigXmlCommand: public SafeXmlCommand
 {
 public:
 	virtual void Execute();
@@ -211,7 +218,7 @@ public:
 	virtual void Execute();
 };
 
-class ConfigTemplatesXmlCommand: public XmlCommand
+class ConfigTemplatesXmlCommand: public SafeXmlCommand
 {
 public:
 	virtual void Execute();
@@ -238,7 +245,7 @@ public:
 	virtual void Execute();
 };
 
-class ReadUrlXmlCommand: public XmlCommand
+class ReadUrlXmlCommand: public SafeXmlCommand
 {
 public:
 	virtual void Execute();
@@ -262,7 +269,7 @@ protected:
 	virtual GuardedMessageList GuardMessages();
 };
 
-class ServerVolumesXmlCommand: public XmlCommand
+class ServerVolumesXmlCommand: public SafeXmlCommand
 {
 public:
 	virtual void Execute();
@@ -420,8 +427,17 @@ void XmlRpcProcessor::Dispatch()
 		command->SetHttpMethod(m_httpMethod);
 		command->SetUserAccess(m_userAccess);
 		command->PrepareParams();
-		command->Execute();
-		BuildResponse(command->GetResponse(), command->GetCallbackFunc(), command->GetFault(), requestId);
+		m_safeMethod = command->IsSafeMethod();
+		bool safeToExecute = m_safeMethod || m_httpMethod == XmlRpcProcessor::hmPost || m_protocol == XmlRpcProcessor::rpJsonPRpc;
+		if (safeToExecute || command->IsError())
+		{
+			command->Execute();
+			BuildResponse(command->GetResponse(), command->GetCallbackFunc(), command->GetFault(), requestId);
+		}
+		else
+		{
+			BuildErrorResponse(4, "Not safe procedure for HTTP-Method GET. Use Method POST instead");
+		}
 	}
 }
 
@@ -452,6 +468,7 @@ void XmlRpcProcessor::MutliCall()
 
 		std::unique_ptr<XmlCommand> command = CreateCommand(methodName);
 		command->SetRequest(requestPtr);
+		m_safeMethod |= command->IsSafeMethod();
 		command->Execute();
 
 		debug("MutliCall, Response=%s", command->GetResponse());
@@ -476,12 +493,7 @@ void XmlRpcProcessor::MutliCall()
 
 	if (error)
 	{
-		ErrorXmlCommand command(4, "Parse error");
-		command.SetRequest(m_request);
-		command.SetProtocol(rpXmlRpc);
-		command.PrepareParams();
-		command.Execute();
-		BuildResponse(command.GetResponse(), "", command.GetFault(), nullptr);
+		BuildErrorResponse(4, "Parse error");
 	}
 	else
 	{
@@ -544,6 +556,16 @@ void XmlRpcProcessor::BuildResponse(const char* response, const char* callbackFu
 	m_contentType = xmlRpc ? "text/xml" : "application/json";
 }
 
+void XmlRpcProcessor::BuildErrorResponse(int errCode, const char* errText)
+{
+	ErrorXmlCommand command(errCode, errText);
+	command.SetRequest(m_request);
+	command.SetProtocol(m_protocol);
+	command.PrepareParams();
+	command.Execute();
+	BuildResponse(command.GetResponse(), "", command.GetFault(), nullptr);
+}
+
 std::unique_ptr<XmlCommand> XmlRpcProcessor::CreateCommand(const char* methodName)
 {
 	std::unique_ptr<XmlCommand> command;
@@ -1067,16 +1089,6 @@ void XmlCommand::DecodeStr(char* str)
 	}
 }
 
-bool XmlCommand::CheckSafeMethod()
-{
-	bool safe = m_httpMethod == XmlRpcProcessor::hmPost || m_protocol == XmlRpcProcessor::rpJsonPRpc;
-	if (!safe)
-	{
-		BuildErrorResponse(4, "Not safe procedure for HTTP-Method GET. Use Method POST instead");
-	}
-	return safe;
-}
-
 //*****************************************************************
 // Commands
 
@@ -1087,11 +1099,6 @@ void ErrorXmlCommand::Execute()
 
 void PauseUnpauseXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	bool ok = true;
 
 	g_Options->SetResumeTime(0);
@@ -1120,11 +1127,6 @@ void PauseUnpauseXmlCommand::Execute()
 // bool scheduleresume(int Seconds)
 void ScheduleResumeXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	int seconds = 0;
 	if (!NextParamAsInt(&seconds) || seconds < 0)
 	{
@@ -1141,22 +1143,12 @@ void ScheduleResumeXmlCommand::Execute()
 
 void ShutdownXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	BuildBoolResponse(true);
 	ExitProc();
 }
 
 void ReloadXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	BuildBoolResponse(true);
 	Reload();
 }
@@ -1179,11 +1171,6 @@ void DumpDebugXmlCommand::Execute()
 
 void SetDownloadRateXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	int rate = 0;
 	if (!NextParamAsInt(&rate) || rate < 0)
 	{
@@ -1365,8 +1352,8 @@ void StatusXmlCommand::Execute()
 	Util::SplitInt64(freeDiskSpace, &freeDiskSpaceHi, &freeDiskSpaceLo);
 	int freeDiskSpaceMB = (int)(freeDiskSpace / 1024 / 1024);
 
-	int serverTime = Util::CurrentTime();
-	int resumeTime = g_Options->GetResumeTime();
+	int serverTime = (int)Util::CurrentTime();
+	int resumeTime = (int)g_Options->GetResumeTime();
 	bool feedActive = g_FeedCoordinator->HasActiveDownloads();
 	int queuedScripts = g_QueueScriptCoordinator->GetQueueSize();
 
@@ -2091,11 +2078,6 @@ EditCommandEntry EditCommandNameMap[] = {
 //    bool editqueue(string Command, int Offset, string Args, int[] IDs)
 void EditQueueXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	char* editCommand;
 	if (!NextParamAsStr(&editCommand))
 	{
@@ -2161,11 +2143,6 @@ void EditQueueXmlCommand::Execute()
 //   bool append(string NZBFilename, string Category, int Priority, bool AddToTop, string Content, bool AddPaused, string DupeKey, int DupeScore, string DupeMode)
 void DownloadXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	bool v13 = true;
 
 	char* nzbFilename;
@@ -2378,11 +2355,6 @@ void PostQueueXmlCommand::Execute()
 
 void WriteLogXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	char* kind;
 	char* text;
 	if (!NextParamAsStr(&kind) || !NextParamAsStr(&text))
@@ -2426,11 +2398,6 @@ void WriteLogXmlCommand::Execute()
 
 void ClearLogXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	g_Log->Clear();
 
 	BuildBoolResponse(true);
@@ -2438,11 +2405,6 @@ void ClearLogXmlCommand::Execute()
 
 void ScanXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	bool syncMode = false;
 	// optional parameter "SyncMode"
 	NextParamAsBool(&syncMode);
@@ -2948,11 +2910,6 @@ void ViewFeedXmlCommand::Execute()
 // bool fetchfeed(int ID)
 void FetchFeedXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	int id;
 	if (!NextParamAsInt(&id))
 	{
@@ -2968,11 +2925,6 @@ void FetchFeedXmlCommand::Execute()
 // bool editserver(int ID, bool Active)
 void EditServerXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	bool ok = false;
 	int first = true;
 
@@ -3092,11 +3044,6 @@ void CheckUpdatesXmlCommand::Execute()
 // bool startupdate(string branch)
 void StartUpdateXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	char* branchName;
 	if (!NextParamAsStr(&branchName))
 	{
@@ -3268,11 +3215,6 @@ void ServerVolumesXmlCommand::Execute()
 // bool resetservervolume(int serverid, string counter);
 void ResetServerVolumeXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	int serverId;
 	char* counter;
 	if (!NextParamAsInt(&serverId) || !NextParamAsStr(&counter))
@@ -3351,11 +3293,6 @@ void TestServerXmlCommand::Execute()
 	const char* XML_RESPONSE_STR_BODY = "<string>%s</string>";
 	const char* JSON_RESPONSE_STR_BODY = "\"%s\"";
 
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	char* host;
 	int port;
 	char* username;
@@ -3378,6 +3315,17 @@ void TestServerXmlCommand::Execute()
 	connection.SetSuppressErrors(false);
 
 	bool ok = connection.Connect();
+	if (ok)
+	{
+		// generate a unique non-existent message-id since we don't want a real article to be returned
+		BString<1024> id;
+		while (id.Length() < 30)
+		{
+			id.AppendFmt("%i", rand());
+		}
+		const char* response = connection.Request(BString<1024>("ARTICLE <%s@nzbget.net>\r\n", *id));
+		ok = response && (*response == '4' || *response == '2');
+	}
 
 	BString<1024> content(IsJson() ? JSON_RESPONSE_STR_BODY : XML_RESPONSE_STR_BODY,
 		ok ? "" : m_errText.Empty() ? "Unknown error" : *m_errText);
@@ -3396,11 +3344,6 @@ void TestServerXmlCommand::PrintError(const char* errMsg)
 // bool startscript(string script, string command, string context, struct[] options);
 void StartScriptXmlCommand::Execute()
 {
-	if (!CheckSafeMethod())
-	{
-		return;
-	}
-
 	char* script;
 	char* command;
 	char* context;

daemon/remote/XmlRpc.h

@@ -59,6 +59,7 @@ public:
 	const char* GetResponse() { return m_response; }
 	const char* GetContentType() { return m_contentType; }
 	static bool IsRpcRequest(const char* url);
+	bool IsSafeMethod() { return m_safeMethod; };
 
 private:
 	char* m_request = nullptr;
@@ -68,11 +69,13 @@ private:
 	EUserAccess m_userAccess;
 	CString m_url;
 	StringBuilder m_response;
+	bool m_safeMethod = false;
 
 	void Dispatch();
 	std::unique_ptr<XmlCommand> CreateCommand(const char* methodName);
 	void MutliCall();
 	void BuildResponse(const char* response, const char* callbackFunc, bool fault, const char* requestId);
+	void BuildErrorResponse(int errCode, const char* errText);
 };
 
 class XmlCommand
@@ -89,6 +92,8 @@ public:
 	const char* GetResponse() { return m_response; }
 	const char* GetCallbackFunc() { return m_callbackFunc; }
 	bool GetFault() { return m_fault; }
+	virtual bool IsSafeMethod() { return false; };
+	virtual bool IsError() { return false; };
 
 protected:
 	char* m_request = nullptr;
@@ -107,7 +112,6 @@ protected:
 	void AppendFmtResponse(const char* format, ...);
 	void AppendCondResponse(const char* part, bool cond);
 	bool IsJson();
-	bool CheckSafeMethod();
 	bool NextParamAsInt(int* value);
 	bool NextParamAsBool(bool* value);
 	bool NextParamAsStr(char** valueBuf);

daemon/util/FileSystem.cpp

@@ -428,7 +428,7 @@ CString FileSystem::MakeUniqueFilename(const char* destDir, const char* basename
 		if (extension && extension != basename)
 		{
 			BString<1024> filenameWithoutExt = basename;
-			int end = extension - basename;
+			int end = (int)(extension - basename);
 			filenameWithoutExt[end < 1024 ? end : 1024-1] = '\0';
 
 			if (!strcasecmp(extension, ".par2"))

daemon/util/Log.cpp

@@ -75,31 +75,55 @@ void Log::Filelog(const char* msg, ...)
 
 	if ((int)rawtime/86400 != (int)m_lastWritten/86400 && g_Options->GetWriteLog() == Options::wlRotate)
 	{
+		if (m_logFile)
+		{
+			m_logFile.reset();
+		}
 		RotateLog();
 	}
 
 	m_lastWritten = rawtime;
 
-	DiskFile file;
-	if (file.Open(m_logFilename, DiskFile::omAppend))
+	if (!m_logFile)
 	{
+		m_logFile = std::make_unique<DiskFile>();
+		if (!m_logFile->Open(m_logFilename, DiskFile::omAppend))
+		{
+			perror(m_logFilename);
+			m_logFile.reset();
+			return;
+		}
+	}
+
+	m_logFile->Seek(0, DiskFile::soEnd);
+
 #ifdef WIN32
-		uint64 processId = GetCurrentProcessId();
-		uint64 threadId = GetCurrentThreadId();
+	uint64 processId = GetCurrentProcessId();
+	uint64 threadId = GetCurrentThreadId();
 #else
-		uint64 processId = (uint64)getpid();
-		uint64 threadId = (uint64)pthread_self();
+	uint64 processId = (uint64)getpid();
+	uint64 threadId = (uint64)pthread_self();
 #endif
 #ifdef DEBUG
-		file.Print("%s\t%llu\t%llu\t%s%s", time, processId, threadId, tmp2, LINE_ENDING);
+	m_logFile->Print("%s\t%llu\t%llu\t%s%s", time, processId, threadId, tmp2, LINE_ENDING);
 #else
-		file.Print("%s\t%s%s", time, tmp2, LINE_ENDING);
+	m_logFile->Print("%s\t%s%s", time, tmp2, LINE_ENDING);
 #endif
-		file.Close();
-	}
-	else
+
+	m_logFile->Flush();
+}
+
+void Log::IntervalCheck()
+{
+	// Close log-file on idle (if last write into log was more than a second ago)
+	if (m_logFile)
 	{
-		perror(m_logFilename);
+		time_t curTime = Util::CurrentTime() + g_Options->GetTimeCorrection();
+		if (std::abs(curTime - m_lastWritten) > 1)
+		{
+			Guard guard(m_logMutex);
+			m_logFile.reset();
+		}
 	}
 }
 
@@ -111,6 +135,11 @@ void debug(const char* filename, const char* funcname, int lineNr, const char* m
 void debug(const char* msg, ...)
 #endif
 {
+	if (!g_Log)
+	{
+		return;
+	}
+
 	char tmp1[1024];
 
 	va_list ap;
@@ -257,7 +286,7 @@ void Log::AddMessage(Message::EKind kind, const char * text)
 
 	if (m_optInit && g_Options)
 	{
-		while (m_messages.size() > (uint32)g_Options->GetLogBufferSize())
+		while (m_messages.size() > (uint32)g_Options->GetLogBuffer())
 		{
 			m_messages.pop_front();
 		}

daemon/util/Log.h

@@ -70,6 +70,7 @@ typedef std::deque<Message> MessageList;
 typedef GuardedPtr<MessageList> GuardedMessageList;
 
 class Debuggable;
+class DiskFile;
 
 class Log
 {
@@ -83,6 +84,7 @@ public:
 	void RegisterDebuggable(Debuggable* debuggable);
 	void UnregisterDebuggable(Debuggable* debuggable);
 	void LogDebugInfo();
+	void IntervalCheck();
 
 private:
 	typedef std::list<Debuggable*> Debuggables;
@@ -92,6 +94,7 @@ private:
 	Debuggables m_debuggables;
 	Mutex m_debugMutex;
 	CString m_logFilename;
+	std::unique_ptr<DiskFile> m_logFile;
 	uint32 m_idGen = 0;
 	time_t m_lastWritten = 0;
 	bool m_optInit = false;

daemon/util/Script.cpp

@@ -473,6 +473,8 @@ void ScriptController::StartProcess(int* pipein, int* pipeout)
 		cmdLine = cmdLineBuf;
 	}
 
+	debug("Starting process: %s", cmdLine);
+
 	WString wideWorkingDir = FileSystem::UtfPathToWidePath(workingDir);
 	if (strlen(workingDir) > 260 - 14)
 	{
@@ -582,6 +584,12 @@ void ScriptController::StartProcess(int* pipein, int* pipeout)
 	args.emplace_back(nullptr);
 	char* const* argdata = (char* const*)args.data();
 
+	debug("Starting  process: %s", script);
+	for (const char* arg : m_args)
+	{
+		debug("arg: %s", arg);
+	}
+
 	debug("forking");
 	pid_t pid = fork();
 

daemon/util/Util.cpp

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2007-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2007-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -20,6 +20,7 @@
 
 #include "nzbget.h"
 #include "Util.h"			
+#include "YEncode.h"
 
 #ifndef WIN32
 // function "code_revision" is automatically generated in file "code_revision.cpp" on each build
@@ -650,158 +651,6 @@ void Util::SetStandByMode(bool standBy)
 #endif
 }
 
-static uint32 crc32_tab[] = {
-	0x00000000, 0x77073096, 0xee0e612c, 0x990951ba, 0x076dc419, 0x706af48f,
-	0xe963a535, 0x9e6495a3,	0x0edb8832, 0x79dcb8a4, 0xe0d5e91e, 0x97d2d988,
-	0x09b64c2b, 0x7eb17cbd, 0xe7b82d07, 0x90bf1d91, 0x1db71064, 0x6ab020f2,
-	0xf3b97148, 0x84be41de,	0x1adad47d, 0x6ddde4eb, 0xf4d4b551, 0x83d385c7,
-	0x136c9856, 0x646ba8c0, 0xfd62f97a, 0x8a65c9ec,	0x14015c4f, 0x63066cd9,
-	0xfa0f3d63, 0x8d080df5,	0x3b6e20c8, 0x4c69105e, 0xd56041e4, 0xa2677172,
-	0x3c03e4d1, 0x4b04d447, 0xd20d85fd, 0xa50ab56b,	0x35b5a8fa, 0x42b2986c,
-	0xdbbbc9d6, 0xacbcf940,	0x32d86ce3, 0x45df5c75, 0xdcd60dcf, 0xabd13d59,
-	0x26d930ac, 0x51de003a, 0xc8d75180, 0xbfd06116, 0x21b4f4b5, 0x56b3c423,
-	0xcfba9599, 0xb8bda50f, 0x2802b89e, 0x5f058808, 0xc60cd9b2, 0xb10be924,
-	0x2f6f7c87, 0x58684c11, 0xc1611dab, 0xb6662d3d,	0x76dc4190, 0x01db7106,
-	0x98d220bc, 0xefd5102a, 0x71b18589, 0x06b6b51f, 0x9fbfe4a5, 0xe8b8d433,
-	0x7807c9a2, 0x0f00f934, 0x9609a88e, 0xe10e9818, 0x7f6a0dbb, 0x086d3d2d,
-	0x91646c97, 0xe6635c01, 0x6b6b51f4, 0x1c6c6162, 0x856530d8, 0xf262004e,
-	0x6c0695ed, 0x1b01a57b, 0x8208f4c1, 0xf50fc457, 0x65b0d9c6, 0x12b7e950,
-	0x8bbeb8ea, 0xfcb9887c, 0x62dd1ddf, 0x15da2d49, 0x8cd37cf3, 0xfbd44c65,
-	0x4db26158, 0x3ab551ce, 0xa3bc0074, 0xd4bb30e2, 0x4adfa541, 0x3dd895d7,
-	0xa4d1c46d, 0xd3d6f4fb, 0x4369e96a, 0x346ed9fc, 0xad678846, 0xda60b8d0,
-	0x44042d73, 0x33031de5, 0xaa0a4c5f, 0xdd0d7cc9, 0x5005713c, 0x270241aa,
-	0xbe0b1010, 0xc90c2086, 0x5768b525, 0x206f85b3, 0xb966d409, 0xce61e49f,
-	0x5edef90e, 0x29d9c998, 0xb0d09822, 0xc7d7a8b4, 0x59b33d17, 0x2eb40d81,
-	0xb7bd5c3b, 0xc0ba6cad, 0xedb88320, 0x9abfb3b6, 0x03b6e20c, 0x74b1d29a,
-	0xead54739, 0x9dd277af, 0x04db2615, 0x73dc1683, 0xe3630b12, 0x94643b84,
-	0x0d6d6a3e, 0x7a6a5aa8, 0xe40ecf0b, 0x9309ff9d, 0x0a00ae27, 0x7d079eb1,
-	0xf00f9344, 0x8708a3d2, 0x1e01f268, 0x6906c2fe, 0xf762575d, 0x806567cb,
-	0x196c3671, 0x6e6b06e7, 0xfed41b76, 0x89d32be0, 0x10da7a5a, 0x67dd4acc,
-	0xf9b9df6f, 0x8ebeeff9, 0x17b7be43, 0x60b08ed5, 0xd6d6a3e8, 0xa1d1937e,
-	0x38d8c2c4, 0x4fdff252, 0xd1bb67f1, 0xa6bc5767, 0x3fb506dd, 0x48b2364b,
-	0xd80d2bda, 0xaf0a1b4c, 0x36034af6, 0x41047a60, 0xdf60efc3, 0xa867df55,
-	0x316e8eef, 0x4669be79, 0xcb61b38c, 0xbc66831a, 0x256fd2a0, 0x5268e236,
-	0xcc0c7795, 0xbb0b4703, 0x220216b9, 0x5505262f, 0xc5ba3bbe, 0xb2bd0b28,
-	0x2bb45a92, 0x5cb36a04, 0xc2d7ffa7, 0xb5d0cf31, 0x2cd99e8b, 0x5bdeae1d,
-	0x9b64c2b0, 0xec63f226, 0x756aa39c, 0x026d930a, 0x9c0906a9, 0xeb0e363f,
-	0x72076785, 0x05005713, 0x95bf4a82, 0xe2b87a14, 0x7bb12bae, 0x0cb61b38,
-	0x92d28e9b, 0xe5d5be0d, 0x7cdcefb7, 0x0bdbdf21, 0x86d3d2d4, 0xf1d4e242,
-	0x68ddb3f8, 0x1fda836e, 0x81be16cd, 0xf6b9265b, 0x6fb077e1, 0x18b74777,
-	0x88085ae6, 0xff0f6a70, 0x66063bca, 0x11010b5c, 0x8f659eff, 0xf862ae69,
-	0x616bffd3, 0x166ccf45, 0xa00ae278, 0xd70dd2ee, 0x4e048354, 0x3903b3c2,
-	0xa7672661, 0xd06016f7, 0x4969474d, 0x3e6e77db, 0xaed16a4a, 0xd9d65adc,
-	0x40df0b66, 0x37d83bf0, 0xa9bcae53, 0xdebb9ec5, 0x47b2cf7f, 0x30b5ffe9,
-	0xbdbdf21c, 0xcabac28a, 0x53b39330, 0x24b4a3a6, 0xbad03605, 0xcdd70693,
-	0x54de5729, 0x23d967bf, 0xb3667a2e, 0xc4614ab8, 0x5d681b02, 0x2a6f2b94,
-	0xb40bbe37, 0xc30c8ea1, 0x5a05df1b, 0x2d02ef8d
-};
-
-/* This is a modified version of chksum_crc() from
- * crc32.c (http://www.koders.com/c/fid699AFE0A656F0022C9D6B9D1743E697B69CE5815.aspx)
- * (c) 1999,2000 Krzysztof Dabrowski
- * (c) 1999,2000 ElysiuM deeZine
- *
- * chksum_crc() -- to a given block, this one calculates the
- *				crc32-checksum until the length is
- *				reached. the crc32-checksum will be
- *				the result.
- */
-uint32 Util::Crc32m(uint32 startCrc, uchar *block, uint32 length)
-{
-	uint32 crc = startCrc;
-	for (uint32 i = 0; i < length; i++)
-	{
-		crc = ((crc >> 8) & 0x00FFFFFF) ^ crc32_tab[(crc ^ *block++) & 0xFF];
-	}
-	return crc;
-}
-
-uint32 Util::Crc32(uchar *block, uint32 length)
-{
-	return Util::Crc32m(0xFFFFFFFF, block, length) ^ 0xFFFFFFFF;
-}
-
-/* From zlib/crc32.c (http://www.zlib.net/)
- * Copyright (C) 1995-2006, 2010, 2011, 2012 Mark Adler
- */
-
-#define GF2_DIM 32      /* dimension of GF(2) vectors (length of CRC) */
-
-uint32 gf2_matrix_times(uint32 *mat, uint32 vec)
-{
-	uint32 sum;
-
-	sum = 0;
-	while (vec) {
-		if (vec & 1)
-			sum ^= *mat;
-		vec >>= 1;
-		mat++;
-	}
-	return sum;
-}
-
-void gf2_matrix_square(uint32 *square, uint32 *mat)
-{
-	int n;
-
-	for (n = 0; n < GF2_DIM; n++)
-		square[n] = gf2_matrix_times(mat, mat[n]);
-}
-
-uint32 Util::Crc32Combine(uint32 crc1, uint32 crc2, uint32 len2)
-{
-	int n;
-	uint32 row;
-	uint32 even[GF2_DIM];    /* even-power-of-two zeros operator */
-	uint32 odd[GF2_DIM];     /* odd-power-of-two zeros operator */
-
-	/* degenerate case (also disallow negative lengths) */
-	if (len2 <= 0)
-		return crc1;
-
-	/* put operator for one zero bit in odd */
-	odd[0] = 0xedb88320UL;          /* CRC-32 polynomial */
-	row = 1;
-	for (n = 1; n < GF2_DIM; n++) {
-		odd[n] = row;
-		row <<= 1;
-	}
-
-	/* put operator for two zero bits in even */
-	gf2_matrix_square(even, odd);
-
-	/* put operator for four zero bits in odd */
-	gf2_matrix_square(odd, even);
-
-	/* apply len2 zeros to crc1 (first square will put the operator for one
-	   zero byte, eight zero bits, in even) */
-	do {
-		/* apply zeros operator for this bit of len2 */
-		gf2_matrix_square(even, odd);
-		if (len2 & 1)
-			crc1 = gf2_matrix_times(even, crc1);
-		len2 >>= 1;
-
-		/* if no more bits set, then done */
-		if (len2 == 0)
-			break;
-
-		/* another iteration of the loop with odd and even swapped */
-		gf2_matrix_square(odd, even);
-		if (len2 & 1)
-			crc1 = gf2_matrix_times(odd, crc1);
-		len2 >>= 1;
-
-		/* if no more bits set, then done */
-	} while (len2 != 0);
-
-	/* return combined crc */
-	crc1 ^= crc2;
-
-	return crc1;
-}
-
 int Util::NumberOfCpuCores()
 {
 #ifdef WIN32
@@ -1573,7 +1422,7 @@ void URL::ParseUrl()
 		return;
 	}
 
-	m_protocol.Set(m_address, protEnd - m_address);
+	m_protocol.Set(m_address, (int)(protEnd - m_address));
 
 	char* hostStart = protEnd + 3;
 	char* slash = strchr(hostStart, '/');
@@ -1607,7 +1456,7 @@ void URL::ParseUrl()
 	if (slash)
 	{
 		char* resEnd = m_address + strlen(m_address);
-		m_resource.Set(slash, resEnd - slash + 1);
+		m_resource.Set(slash, (int)(resEnd - slash + 1));
 
 		hostEnd = slash - 1;
 	}
@@ -1625,7 +1474,7 @@ void URL::ParseUrl()
 		m_port = atoi(colon + 1);
 	}
 
-	m_host.Set(hostStart, hostEnd - hostStart + 1);
+	m_host.Set(hostStart, (int)(hostEnd - hostStart + 1));
 
 	m_valid = true;
 }
@@ -1687,7 +1536,7 @@ int RegEx::GetMatchStart(int index)
 #ifdef HAVE_REGEX_H
 	return m_matches[index].rm_so;
 #else
-	return nullptr;
+	return 0;
 #endif
 }
 
@@ -1732,14 +1581,14 @@ bool WildMask::Match(const char* text)
 			if (!qmark)
 			{
 				ExpandArray();
-				m_wildStart[m_wildCount-1] = str - text;
+				m_wildStart[m_wildCount-1] = (int)(str - text);
 				m_wildLen[m_wildCount-1] = 0;
 				qmark = true;
 			}
 		}
 		else if (m_wantsPositions && qmark)
 		{
-			m_wildLen[m_wildCount-1] = str - (text + m_wildStart[m_wildCount-1]);
+			m_wildLen[m_wildCount-1] = (int)(str - (text + m_wildStart[m_wildCount-1]));
 			qmark = false;
 		}
 
@@ -1754,7 +1603,7 @@ bool WildMask::Match(const char* text)
 
 	if (m_wantsPositions && qmark)
 	{
-		m_wildLen[m_wildCount-1] = str - (text + m_wildStart[m_wildCount-1]);
+		m_wildLen[m_wildCount-1] = (int)(str - (text + m_wildStart[m_wildCount-1]));
 		qmark = false;
 	}
 
@@ -1764,13 +1613,13 @@ bool WildMask::Match(const char* text)
 		{
 			if (m_wantsPositions && qmark)
 			{
-				m_wildLen[m_wildCount-1] = str - (text + m_wildStart[m_wildCount-1]);
+				m_wildLen[m_wildCount-1] = (int)(str - (text + m_wildStart[m_wildCount-1]));
 				qmark = false;
 			}
 			if (m_wantsPositions && !star)
 			{
 				ExpandArray();
-				m_wildStart[m_wildCount-1] = str - text;
+				m_wildStart[m_wildCount-1] = (int)(str - text);
 				m_wildLen[m_wildCount-1] = 0;
 				star = true;
 			}
@@ -1792,7 +1641,7 @@ bool WildMask::Match(const char* text)
 			if (m_wantsPositions && !qmark)
 			{
 				ExpandArray();
-				m_wildStart[m_wildCount-1] = str - text;
+				m_wildStart[m_wildCount-1] = (int)(str - text);
 				m_wildLen[m_wildCount-1] = 0;
 				qmark = true;
 			}
@@ -1804,12 +1653,12 @@ bool WildMask::Match(const char* text)
 		{
 			if (m_wantsPositions && qmark)
 			{
-				m_wildLen[m_wildCount-1] = str - (text + m_wildStart[m_wildCount-1]);
+				m_wildLen[m_wildCount-1] = (int)(str - (text + m_wildStart[m_wildCount-1]));
 				qmark = false;
 			}
 			else if (m_wantsPositions && star)
 			{
-				m_wildLen[m_wildCount-1] = str - (text + m_wildStart[m_wildCount-1]);
+				m_wildLen[m_wildCount-1] = (int)(str - (text + m_wildStart[m_wildCount-1]));
 				star = false;
 			}
 
@@ -1832,13 +1681,13 @@ bool WildMask::Match(const char* text)
 
 	if (m_wantsPositions && qmark)
 	{
-		m_wildLen[m_wildCount-1] = str - (text + m_wildStart[m_wildCount-1]);
+		m_wildLen[m_wildCount-1] = (int)(str - (text + m_wildStart[m_wildCount-1]));
 	}
 
 	if (*pat == '*' && m_wantsPositions && !star)
 	{
 		ExpandArray();
-		m_wildStart[m_wildCount-1] = str - text;
+		m_wildStart[m_wildCount-1] = (int)(str - text);
 		m_wildLen[m_wildCount-1] = strlen(str);
 	}
 
@@ -1987,3 +1836,101 @@ char* Tokenizer::Next()
 	}
 	return token;
 }
+
+void Crc32::Reset()
+{
+	static_assert(sizeof(m_state) >= sizeof(YEncode::crc_state), "m_state has invalid size");
+
+	YEncode::crc_init((YEncode::crc_state*)State());
+}
+
+void Crc32::Append(uchar* block, uint32 length)
+{
+	YEncode::crc_incr((YEncode::crc_state*)State(), block, length);
+}
+
+uint32 Crc32::Finish()
+{
+	return YEncode::crc_finish((YEncode::crc_state*)State());
+}
+
+/* From zlib/crc32.c (http://www.zlib.net/)
+ * Copyright (C) 1995-2006, 2010, 2011, 2012 Mark Adler
+ */
+
+#define GF2_DIM 32      /* dimension of GF(2) vectors (length of CRC) */
+
+uint32 gf2_matrix_times(uint32 *mat, uint32 vec)
+{
+	uint32 sum;
+
+	sum = 0;
+	while (vec) {
+		if (vec & 1)
+			sum ^= *mat;
+		vec >>= 1;
+		mat++;
+	}
+	return sum;
+}
+
+void gf2_matrix_square(uint32 *square, uint32 *mat)
+{
+	int n;
+
+	for (n = 0; n < GF2_DIM; n++)
+		square[n] = gf2_matrix_times(mat, mat[n]);
+}
+
+uint32 Crc32::Combine(uint32 crc1, uint32 crc2, uint32 len2)
+{
+	int n;
+	uint32 row;
+	uint32 even[GF2_DIM];    /* even-power-of-two zeros operator */
+	uint32 odd[GF2_DIM];     /* odd-power-of-two zeros operator */
+
+	/* degenerate case (also disallow negative lengths) */
+	if (len2 <= 0)
+		return crc1;
+
+	/* put operator for one zero bit in odd */
+	odd[0] = 0xedb88320UL;          /* CRC-32 polynomial */
+	row = 1;
+	for (n = 1; n < GF2_DIM; n++) {
+		odd[n] = row;
+		row <<= 1;
+	}
+
+	/* put operator for two zero bits in even */
+	gf2_matrix_square(even, odd);
+
+	/* put operator for four zero bits in odd */
+	gf2_matrix_square(odd, even);
+
+	/* apply len2 zeros to crc1 (first square will put the operator for one
+	   zero byte, eight zero bits, in even) */
+	do {
+		/* apply zeros operator for this bit of len2 */
+		gf2_matrix_square(even, odd);
+		if (len2 & 1)
+			crc1 = gf2_matrix_times(even, crc1);
+		len2 >>= 1;
+
+		/* if no more bits set, then done */
+		if (len2 == 0)
+			break;
+
+		/* another iteration of the loop with odd and even swapped */
+		gf2_matrix_square(odd, even);
+		if (len2 & 1)
+			crc1 = gf2_matrix_times(odd, crc1);
+		len2 >>= 1;
+
+		/* if no more bits set, then done */
+	} while (len2 != 0);
+
+	/* return combined crc */
+	crc1 ^= crc2;
+
+	return crc1;
+}

daemon/util/Util.h

@@ -86,10 +86,6 @@ public:
 
 	static void Init();
 
-	static uint32 Crc32(uchar *block, uint32 length);
-	static uint32 Crc32m(uint32 startCrc, uchar *block, uint32 length);
-	static uint32 Crc32Combine(uint32 crc1, uint32 crc2, uint32 len2);
-
 	/*
 	* Returns number of available CPU cores or -1 if it could not be determined
 	*/
@@ -317,4 +313,24 @@ private:
 	bool m_working = false;
 };
 
+class Crc32
+{
+public:
+	Crc32() { Reset(); }
+	void Reset();
+	void Append(uchar* block, uint32 length);
+	uint32 Finish();
+	static uint32 Combine(uint32 crc1, uint32 crc2, uint32 len2);
+
+private:
+#if defined(WIN32) && !defined(_WIN64)
+	// VC++ in 32 bit mode can not "alignas(16)" dynamically allocated objects
+	alignas(8) uint32_t m_state[4 * 5 + 8]; // = YEncode::crc_state
+	void* State() { void* p = &m_state; size_t s = sizeof(m_state); return std::align(16, 4 * 5, p, s); }
+#else
+	alignas(16) uint32_t m_state[4 * 5]; // = YEncode::crc_state
+	void* State() { return &m_state; }
+#endif
+};
+
 #endif

daemon/windows/WinConsole.cpp

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2014-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2014-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -307,6 +307,7 @@ LRESULT WinConsole::TrayWndProc(HWND hwndWin, UINT uMsg, WPARAM wParam, LPARAM l
 			return 0;
 
 		case UM_QUIT:
+		case WM_ENDSESSION:
 			ExitProc();
 			return 0;
 
@@ -456,12 +457,12 @@ void WinConsole::ShowAboutBox()
 	m_modal = false;
 }
 
-BOOL CALLBACK WinConsole::AboutDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
+INT_PTR CALLBACK WinConsole::AboutDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
 {
 	return g_WinConsole->AboutDialogProc(hwndDlg, uMsg, wParam, lParam);
 }
 
-BOOL WinConsole::AboutDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
+INT_PTR WinConsole::AboutDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
 {
 	switch(uMsg)
 	{
@@ -507,7 +508,7 @@ BOOL WinConsole::AboutDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM
 				(HWND)wParam == GetDlgItem(hwndDlg, IDC_ABOUT_GPL))
 			{
 				SetCursor(m_handCursor);
-				SetWindowLong(hwndDlg, DWL_MSGRESULT, TRUE);
+				SetWindowLongPtr(hwndDlg, DWLP_MSGRESULT, TRUE);
 				return TRUE;
 			}
 			return FALSE;
@@ -529,12 +530,12 @@ void WinConsole::ShowPrefsDialog()
 	m_modal = false;
 }
 
-BOOL CALLBACK WinConsole::PrefsDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
+INT_PTR CALLBACK WinConsole::PrefsDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
 {
 	return g_WinConsole->PrefsDialogProc(hwndDlg, uMsg, wParam, lParam);
 }
 
-BOOL WinConsole::PrefsDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
+INT_PTR WinConsole::PrefsDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
 {
 	switch(uMsg)
 	{
@@ -691,7 +692,7 @@ void WinConsole::ShowRunningDialog()
 	HWND hTrayWindow = FindWindow("NZBGet tray window", nullptr);
 	m_running = true;
 
-	int result = DialogBox(m_instance, MAKEINTRESOURCE(IDD_RUNNINGDIALOG), m_trayWindow, RunningDialogProcStat);
+	INT_PTR result = DialogBox(m_instance, MAKEINTRESOURCE(IDD_RUNNINGDIALOG), m_trayWindow, RunningDialogProcStat);
 
 	switch (result)
 	{
@@ -710,12 +711,12 @@ void WinConsole::ShowRunningDialog()
 	}
 }
 
-BOOL CALLBACK WinConsole::RunningDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
+INT_PTR CALLBACK WinConsole::RunningDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
 {
 	return g_WinConsole->RunningDialogProc(hwndDlg, uMsg, wParam, lParam);
 }
 
-BOOL WinConsole::RunningDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
+INT_PTR WinConsole::RunningDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
 {
 	switch(uMsg)
 	{
@@ -913,7 +914,7 @@ void WinConsole::ShowFactoryResetDialog()
 	HWND hTrayWindow = FindWindow("NZBGet tray window", nullptr);
 	m_running = true;
 
-	int result = DialogBox(m_instance, MAKEINTRESOURCE(IDD_FACTORYRESETDIALOG), m_trayWindow, FactoryResetDialogProcStat);
+	INT_PTR result = DialogBox(m_instance, MAKEINTRESOURCE(IDD_FACTORYRESETDIALOG), m_trayWindow, FactoryResetDialogProcStat);
 
 	switch (result)
 	{
@@ -923,12 +924,12 @@ void WinConsole::ShowFactoryResetDialog()
 	}
 }
 
-BOOL CALLBACK WinConsole::FactoryResetDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
+INT_PTR CALLBACK WinConsole::FactoryResetDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
 {
 	return g_WinConsole->FactoryResetDialogProc(hwndDlg, uMsg, wParam, lParam);
 }
 
-BOOL WinConsole::FactoryResetDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
+INT_PTR WinConsole::FactoryResetDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam)
 {
 	switch(uMsg)
 	{

daemon/windows/WinConsole.h

@@ -1,7 +1,7 @@
 /*
  *  This file is part of nzbget. See <http://nzbget.net>.
  *
- *  Copyright (C) 2014-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *  Copyright (C) 2014-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
  *
  *  This program is free software; you can redistribute it and/or modify
  *  it under the terms of the GNU General Public License as published by
@@ -89,14 +89,14 @@ private:
 	static BOOL WINAPI ConsoleCtrlHandler(DWORD dwCtrlType);
 	static LRESULT CALLBACK TrayWndProcStat(HWND hwndWin, UINT uMsg, WPARAM wParam, LPARAM lParam);
 	LRESULT TrayWndProc(HWND hwndWin, UINT uMsg, WPARAM wParam, LPARAM lParam);
-	static BOOL CALLBACK AboutDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
-	BOOL AboutDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
-	static BOOL CALLBACK PrefsDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
-	BOOL PrefsDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
-	static BOOL CALLBACK RunningDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
-	BOOL RunningDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
-	static BOOL CALLBACK FactoryResetDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
-	BOOL FactoryResetDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
+	static INT_PTR CALLBACK AboutDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
+	INT_PTR AboutDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
+	static INT_PTR CALLBACK PrefsDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
+	INT_PTR PrefsDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
+	static INT_PTR CALLBACK RunningDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
+	INT_PTR RunningDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
+	static INT_PTR CALLBACK FactoryResetDialogProcStat(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
+	INT_PTR FactoryResetDialogProc(HWND hwndDlg, UINT uMsg, WPARAM wParam, LPARAM lParam);
 };
 
 #endif

lib/par2/commandline.cpp

@@ -18,6 +18,7 @@
 //  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA
 
 #include "nzbget.h"
+#include <iostream>
 #include "par2cmdline.h"
 
 #ifdef _MSC_VER

lib/par2/par2cmdline.h

@@ -25,12 +25,6 @@
 //#define snprintf _snprintf
 #define stat _stat
 
-#define __LITTLE_ENDIAN 1234
-#define __BIG_ENDIAN    4321
-#define __PDP_ENDIAN    3412
-
-#define __BYTE_ORDER __LITTLE_ENDIAN
-
 namespace Par2
 {
 
@@ -75,29 +69,6 @@ typedef unsigned long long u64;
 
 #define _MAX_PATH 255
 
-#if HAVE_ENDIAN_H
-#  include <endian.h>
-#  ifndef __LITTLE_ENDIAN
-#    ifdef _LITTLE_ENDIAN
-#      define __LITTLE_ENDIAN _LITTLE_ENDIAN
-#      define __LITTLE_ENDIAN _LITTLE_ENDIAN
-#      define __BIG_ENDIAN _BIG_ENDIAN
-#      define __PDP_ENDIAN _PDP_ENDIAN
-#    else
-#      error <endian.h> does not define __LITTLE_ENDIAN etc.
-#    endif
-#  endif
-#else
-#  define __LITTLE_ENDIAN 1234
-#  define __BIG_ENDIAN    4321
-#  define __PDP_ENDIAN    3412
-#  if WORDS_BIGENDIAN
-#    define __BYTE_ORDER __BIG_ENDIAN
-#  else
-#    define __BYTE_ORDER __LITTLE_ENDIAN
-#  endif
-#endif
-
 #endif
 
 #ifdef WIN32

lib/par2/parheaders.h

@@ -17,9 +17,6 @@
 //  along with this program; if not, write to the Free Software
 //  Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA
 
-#include <stdlib.h>
-#include <iostream>
-
 namespace Par2
 {
 

lib/yencode/AcleCrc.cpp

@@ -0,0 +1,102 @@
+/*
+ *  Based on node-yencode library by Anime Tosho:
+ *  https://github.com/animetosho/node-yencode
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+// inspired off https://github.com/jocover/crc32_armv8/blob/master/crc32_armv8.c
+
+#include "nzbget.h"
+
+#include "YEncode.h"
+
+#ifdef __ARM_FEATURE_CRC32
+#include <arm_acle.h>
+#endif
+
+namespace YEncode
+{
+#ifdef __ARM_FEATURE_CRC32
+
+void crc_arm_init(crc_state *const s)
+{
+	s->crc0[0] = ~0;
+}
+
+void crc_arm(crc_state *const s, const unsigned char *src, long len) {
+	uint32_t crc = s->crc0[0];
+
+	// initial alignment
+	if (len >= 16) { // 16 is an arbitrary number; it just needs to be >=8
+		if ((uintptr_t)src & sizeof(uint8_t)) {
+			crc = __crc32b(crc, *src);
+			src++;
+			len--;
+		}
+		if ((uintptr_t)src & sizeof(uint16_t)) {
+			crc = __crc32h(crc, *((uint16_t *)src));
+			src += sizeof(uint16_t);
+			len -= sizeof(uint16_t);
+		}
+		
+#ifdef __aarch64__
+		if ((uintptr_t)src & sizeof(uint32_t)) {
+			crc = __crc32w(crc, *((uint32_t *)src));
+			src += sizeof(uint32_t);
+			len -= sizeof(uint32_t);
+		}
+	}
+	while ((len -= sizeof(uint64_t)) >= 0) {
+		crc = __crc32d(crc, *((uint64_t *)src));
+		src += sizeof(uint64_t);
+	}
+	if (len & sizeof(uint32_t)) {
+		crc = __crc32w(crc, *((uint32_t *)src));
+		src += sizeof(uint32_t);
+	}
+#else
+	}
+	while ((len -= sizeof(uint32_t)) >= 0) {
+		crc = __crc32w(crc, *((uint32_t *)src));
+		src += sizeof(uint32_t);
+	}
+#endif
+	if (len & sizeof(uint16_t)) {
+		crc = __crc32h(crc, *((uint16_t *)src));
+		src += sizeof(uint16_t);
+	}
+	if (len & sizeof(uint8_t))
+		crc = __crc32b(crc, *src);
+	
+	s->crc0[0] = crc;
+}
+
+uint32_t crc_arm_finish(crc_state *const s)
+{
+	return ~s->crc0[0];
+}
+#endif
+
+void init_crc_acle()
+{
+#ifdef __ARM_FEATURE_CRC32
+	crc_init = &crc_arm_init;
+	crc_incr = &crc_arm;
+	crc_finish = &crc_arm_finish;
+	crc_simd = true;
+#endif
+}
+
+}

lib/yencode/NeonDecoder.cpp

@@ -0,0 +1,50 @@
+/*
+ *  Based on node-yencode library by Anime Tosho:
+ *  https://github.com/animetosho/node-yencode
+ *
+ *  Copyright (C) 2017 Anime Tosho (animetosho)
+ *  Copyright (C) 2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+
+#include "nzbget.h"
+
+#include "YEncode.h"
+
+#ifdef __ARM_NEON
+#include <arm_neon.h>
+#endif
+
+namespace YEncode
+{
+
+namespace Neon
+{
+#ifdef __ARM_NEON
+#define SIMD_DECODER
+#include "SimdDecoder.cpp"
+#endif
+}
+
+void init_decode_neon() {
+#ifdef __ARM_NEON
+	decode = &YEncode::Neon::do_decode_simd<sizeof(uint8x16_t), YEncode::Neon::do_decode_neon>;
+	YEncode::Neon::decoder_init();
+	decode_simd = true;
+#endif
+}
+
+}

lib/yencode/PclmulCrc.cpp

@@ -0,0 +1,471 @@
+/*
+ *  Based on node-yencode library by Anime Tosho:
+ *  https://github.com/animetosho/node-yencode
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+// taken from zlib-ng / Intel's zlib patch, modified to remove zlib dependencies
+/*
+ * Compute the CRC32 using a parallelized folding approach with the PCLMULQDQ 
+ * instruction.
+ *
+ * A white paper describing this algorithm can be found at:
+ * http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/fast-crc-computation-generic-polynomials-pclmulqdq-paper.pdf
+ *
+ * Copyright (C) 2013 Intel Corporation. All rights reserved.
+ * Authors:
+ *  Wajdi Feghali   <wajdi.k.feghali@intel.com>
+ *  Jim Guilford	<james.guilford@intel.com>
+ *  Vinodh Gopal	<vinodh.gopal@intel.com>
+ *  Erdinc Ozturk   <erdinc.ozturk@intel.com>
+ *  Jim Kukunas	 <james.t.kukunas@linux.intel.com>
+ *
+ * For conditions of distribution and use, see copyright notice in zlib.h
+ */
+
+#include "nzbget.h"
+
+#include "YEncode.h"
+
+#ifdef __PCLMUL__
+#include <immintrin.h>
+#endif
+
+namespace YEncode
+{
+#ifdef __PCLMUL__
+
+void fold_1(__m128i *xmm_crc0, __m128i *xmm_crc1, __m128i *xmm_crc2, __m128i *xmm_crc3) {
+	const __m128i xmm_fold4 = _mm_set_epi32(
+			0x00000001, 0x54442bd4,
+			0x00000001, 0xc6e41596);
+
+	__m128i x_tmp3;
+	__m128 ps_crc0, ps_crc3, ps_res;
+
+	x_tmp3 = *xmm_crc3;
+
+	*xmm_crc3 = *xmm_crc0;
+	*xmm_crc0 = _mm_clmulepi64_si128(*xmm_crc0, xmm_fold4, 0x01);
+	*xmm_crc3 = _mm_clmulepi64_si128(*xmm_crc3, xmm_fold4, 0x10);
+	ps_crc0 = _mm_castsi128_ps(*xmm_crc0);
+	ps_crc3 = _mm_castsi128_ps(*xmm_crc3);
+	ps_res = _mm_xor_ps(ps_crc0, ps_crc3);
+
+	*xmm_crc0 = *xmm_crc1;
+	*xmm_crc1 = *xmm_crc2;
+	*xmm_crc2 = x_tmp3;
+	*xmm_crc3 = _mm_castps_si128(ps_res);
+}
+
+void fold_2(__m128i *xmm_crc0, __m128i *xmm_crc1, __m128i *xmm_crc2, __m128i *xmm_crc3) {
+	const __m128i xmm_fold4 = _mm_set_epi32(
+			0x00000001, 0x54442bd4,
+			0x00000001, 0xc6e41596);
+
+	__m128i x_tmp3, x_tmp2;
+	__m128 ps_crc0, ps_crc1, ps_crc2, ps_crc3, ps_res31, ps_res20;
+
+	x_tmp3 = *xmm_crc3;
+	x_tmp2 = *xmm_crc2;
+
+	*xmm_crc3 = *xmm_crc1;
+	*xmm_crc1 = _mm_clmulepi64_si128(*xmm_crc1, xmm_fold4, 0x01);
+	*xmm_crc3 = _mm_clmulepi64_si128(*xmm_crc3, xmm_fold4, 0x10);
+	ps_crc3 = _mm_castsi128_ps(*xmm_crc3);
+	ps_crc1 = _mm_castsi128_ps(*xmm_crc1);
+	ps_res31 = _mm_xor_ps(ps_crc3, ps_crc1);
+
+	*xmm_crc2 = *xmm_crc0;
+	*xmm_crc0 = _mm_clmulepi64_si128(*xmm_crc0, xmm_fold4, 0x01);
+	*xmm_crc2 = _mm_clmulepi64_si128(*xmm_crc2, xmm_fold4, 0x10);
+	ps_crc0 = _mm_castsi128_ps(*xmm_crc0);
+	ps_crc2 = _mm_castsi128_ps(*xmm_crc2);
+	ps_res20 = _mm_xor_ps(ps_crc0, ps_crc2);
+
+	*xmm_crc0 = x_tmp2;
+	*xmm_crc1 = x_tmp3;
+	*xmm_crc2 = _mm_castps_si128(ps_res20);
+	*xmm_crc3 = _mm_castps_si128(ps_res31);
+}
+
+void fold_3(__m128i *xmm_crc0, __m128i *xmm_crc1, __m128i *xmm_crc2, __m128i *xmm_crc3) {
+	const __m128i xmm_fold4 = _mm_set_epi32(
+			0x00000001, 0x54442bd4,
+			0x00000001, 0xc6e41596);
+
+	__m128i x_tmp3;
+	__m128 ps_crc0, ps_crc1, ps_crc2, ps_crc3, ps_res32, ps_res21, ps_res10;
+
+	x_tmp3 = *xmm_crc3;
+
+	*xmm_crc3 = *xmm_crc2;
+	*xmm_crc2 = _mm_clmulepi64_si128(*xmm_crc2, xmm_fold4, 0x01);
+	*xmm_crc3 = _mm_clmulepi64_si128(*xmm_crc3, xmm_fold4, 0x10);
+	ps_crc2 = _mm_castsi128_ps(*xmm_crc2);
+	ps_crc3 = _mm_castsi128_ps(*xmm_crc3);
+	ps_res32 = _mm_xor_ps(ps_crc2, ps_crc3);
+
+	*xmm_crc2 = *xmm_crc1;
+	*xmm_crc1 = _mm_clmulepi64_si128(*xmm_crc1, xmm_fold4, 0x01);
+	*xmm_crc2 = _mm_clmulepi64_si128(*xmm_crc2, xmm_fold4, 0x10);
+	ps_crc1 = _mm_castsi128_ps(*xmm_crc1);
+	ps_crc2 = _mm_castsi128_ps(*xmm_crc2);
+	ps_res21 = _mm_xor_ps(ps_crc1, ps_crc2);
+
+	*xmm_crc1 = *xmm_crc0;
+	*xmm_crc0 = _mm_clmulepi64_si128(*xmm_crc0, xmm_fold4, 0x01);
+	*xmm_crc1 = _mm_clmulepi64_si128(*xmm_crc1, xmm_fold4, 0x10);
+	ps_crc0 = _mm_castsi128_ps(*xmm_crc0);
+	ps_crc1 = _mm_castsi128_ps(*xmm_crc1);
+	ps_res10 = _mm_xor_ps(ps_crc0, ps_crc1);
+
+	*xmm_crc0 = x_tmp3;
+	*xmm_crc1 = _mm_castps_si128(ps_res10);
+	*xmm_crc2 = _mm_castps_si128(ps_res21);
+	*xmm_crc3 = _mm_castps_si128(ps_res32);
+}
+
+void fold_4(__m128i *xmm_crc0, __m128i *xmm_crc1, __m128i *xmm_crc2, __m128i *xmm_crc3) {
+	const __m128i xmm_fold4 = _mm_set_epi32(
+			0x00000001, 0x54442bd4,
+			0x00000001, 0xc6e41596);
+
+	__m128i x_tmp0, x_tmp1, x_tmp2, x_tmp3;
+	__m128 ps_crc0, ps_crc1, ps_crc2, ps_crc3;
+	__m128 ps_t0, ps_t1, ps_t2, ps_t3;
+	__m128 ps_res0, ps_res1, ps_res2, ps_res3;
+
+	x_tmp0 = *xmm_crc0;
+	x_tmp1 = *xmm_crc1;
+	x_tmp2 = *xmm_crc2;
+	x_tmp3 = *xmm_crc3;
+
+	*xmm_crc0 = _mm_clmulepi64_si128(*xmm_crc0, xmm_fold4, 0x01);
+	x_tmp0 = _mm_clmulepi64_si128(x_tmp0, xmm_fold4, 0x10);
+	ps_crc0 = _mm_castsi128_ps(*xmm_crc0);
+	ps_t0 = _mm_castsi128_ps(x_tmp0);
+	ps_res0 = _mm_xor_ps(ps_crc0, ps_t0);
+
+	*xmm_crc1 = _mm_clmulepi64_si128(*xmm_crc1, xmm_fold4, 0x01);
+	x_tmp1 = _mm_clmulepi64_si128(x_tmp1, xmm_fold4, 0x10);
+	ps_crc1 = _mm_castsi128_ps(*xmm_crc1);
+	ps_t1 = _mm_castsi128_ps(x_tmp1);
+	ps_res1 = _mm_xor_ps(ps_crc1, ps_t1);
+
+	*xmm_crc2 = _mm_clmulepi64_si128(*xmm_crc2, xmm_fold4, 0x01);
+	x_tmp2 = _mm_clmulepi64_si128(x_tmp2, xmm_fold4, 0x10);
+	ps_crc2 = _mm_castsi128_ps(*xmm_crc2);
+	ps_t2 = _mm_castsi128_ps(x_tmp2);
+	ps_res2 = _mm_xor_ps(ps_crc2, ps_t2);
+
+	*xmm_crc3 = _mm_clmulepi64_si128(*xmm_crc3, xmm_fold4, 0x01);
+	x_tmp3 = _mm_clmulepi64_si128(x_tmp3, xmm_fold4, 0x10);
+	ps_crc3 = _mm_castsi128_ps(*xmm_crc3);
+	ps_t3 = _mm_castsi128_ps(x_tmp3);
+	ps_res3 = _mm_xor_ps(ps_crc3, ps_t3);
+
+	*xmm_crc0 = _mm_castps_si128(ps_res0);
+	*xmm_crc1 = _mm_castps_si128(ps_res1);
+	*xmm_crc2 = _mm_castps_si128(ps_res2);
+	*xmm_crc3 = _mm_castps_si128(ps_res3);
+}
+
+alignas(32) const unsigned  pshufb_shf_table[60] = {
+	0x84838281, 0x88878685, 0x8c8b8a89, 0x008f8e8d, /* shl 15 (16 - 1)/shr1 */
+	0x85848382, 0x89888786, 0x8d8c8b8a, 0x01008f8e, /* shl 14 (16 - 3)/shr2 */
+	0x86858483, 0x8a898887, 0x8e8d8c8b, 0x0201008f, /* shl 13 (16 - 4)/shr3 */
+	0x87868584, 0x8b8a8988, 0x8f8e8d8c, 0x03020100, /* shl 12 (16 - 4)/shr4 */
+	0x88878685, 0x8c8b8a89, 0x008f8e8d, 0x04030201, /* shl 11 (16 - 5)/shr5 */
+	0x89888786, 0x8d8c8b8a, 0x01008f8e, 0x05040302, /* shl 10 (16 - 6)/shr6 */
+	0x8a898887, 0x8e8d8c8b, 0x0201008f, 0x06050403, /* shl  9 (16 - 7)/shr7 */
+	0x8b8a8988, 0x8f8e8d8c, 0x03020100, 0x07060504, /* shl  8 (16 - 8)/shr8 */
+	0x8c8b8a89, 0x008f8e8d, 0x04030201, 0x08070605, /* shl  7 (16 - 9)/shr9 */
+	0x8d8c8b8a, 0x01008f8e, 0x05040302, 0x09080706, /* shl  6 (16 -10)/shr10*/
+	0x8e8d8c8b, 0x0201008f, 0x06050403, 0x0a090807, /* shl  5 (16 -11)/shr11*/
+	0x8f8e8d8c, 0x03020100, 0x07060504, 0x0b0a0908, /* shl  4 (16 -12)/shr12*/
+	0x008f8e8d, 0x04030201, 0x08070605, 0x0c0b0a09, /* shl  3 (16 -13)/shr13*/
+	0x01008f8e, 0x05040302, 0x09080706, 0x0d0c0b0a, /* shl  2 (16 -14)/shr14*/
+	0x0201008f, 0x06050403, 0x0a090807, 0x0e0d0c0b  /* shl  1 (16 -15)/shr15*/
+};
+
+void partial_fold(const size_t len, __m128i *xmm_crc0, __m128i *xmm_crc1,
+		__m128i *xmm_crc2, __m128i *xmm_crc3, __m128i *xmm_crc_part) {
+
+	const __m128i xmm_fold4 = _mm_set_epi32(
+			0x00000001, 0x54442bd4,
+			0x00000001, 0xc6e41596);
+	const __m128i xmm_mask3 = _mm_set1_epi32(0x80808080);
+
+	__m128i xmm_shl, xmm_shr, xmm_tmp1, xmm_tmp2, xmm_tmp3;
+	__m128i xmm_a0_0, xmm_a0_1;
+	__m128 ps_crc3, psa0_0, psa0_1, ps_res;
+
+	xmm_shl = _mm_load_si128((__m128i *)pshufb_shf_table + (len - 1));
+	xmm_shr = xmm_shl;
+	xmm_shr = _mm_xor_si128(xmm_shr, xmm_mask3);
+
+	xmm_a0_0 = _mm_shuffle_epi8(*xmm_crc0, xmm_shl);
+
+	*xmm_crc0 = _mm_shuffle_epi8(*xmm_crc0, xmm_shr);
+	xmm_tmp1 = _mm_shuffle_epi8(*xmm_crc1, xmm_shl);
+	*xmm_crc0 = _mm_or_si128(*xmm_crc0, xmm_tmp1);
+
+	*xmm_crc1 = _mm_shuffle_epi8(*xmm_crc1, xmm_shr);
+	xmm_tmp2 = _mm_shuffle_epi8(*xmm_crc2, xmm_shl);
+	*xmm_crc1 = _mm_or_si128(*xmm_crc1, xmm_tmp2);
+
+	*xmm_crc2 = _mm_shuffle_epi8(*xmm_crc2, xmm_shr);
+	xmm_tmp3 = _mm_shuffle_epi8(*xmm_crc3, xmm_shl);
+	*xmm_crc2 = _mm_or_si128(*xmm_crc2, xmm_tmp3);
+
+	*xmm_crc3 = _mm_shuffle_epi8(*xmm_crc3, xmm_shr);
+	*xmm_crc_part = _mm_shuffle_epi8(*xmm_crc_part, xmm_shl);
+	*xmm_crc3 = _mm_or_si128(*xmm_crc3, *xmm_crc_part);
+
+	xmm_a0_1 = _mm_clmulepi64_si128(xmm_a0_0, xmm_fold4, 0x10);
+	xmm_a0_0 = _mm_clmulepi64_si128(xmm_a0_0, xmm_fold4, 0x01);
+
+	ps_crc3 = _mm_castsi128_ps(*xmm_crc3);
+	psa0_0 = _mm_castsi128_ps(xmm_a0_0);
+	psa0_1 = _mm_castsi128_ps(xmm_a0_1);
+
+	ps_res = _mm_xor_ps(ps_crc3, psa0_0);
+	ps_res = _mm_xor_ps(ps_res, psa0_1);
+
+	*xmm_crc3 = _mm_castps_si128(ps_res);
+}
+
+alignas(16) const unsigned crc_k[] = {
+	0xccaa009e, 0x00000000, /* rk1 */
+	0x751997d0, 0x00000001, /* rk2 */
+	0xccaa009e, 0x00000000, /* rk5 */
+	0x63cd6124, 0x00000001, /* rk6 */
+	0xf7011640, 0x00000001, /* rk7 */
+	0xdb710640, 0x00000001  /* rk8 */
+};
+
+alignas(16) const unsigned crc_mask[4] = {
+	0xFFFFFFFF, 0xFFFFFFFF, 0x00000000, 0x00000000
+};
+
+alignas(16) const unsigned crc_mask2[4] = {
+	0x00000000, 0xFFFFFFFF, 0xFFFFFFFF, 0xFFFFFFFF
+};
+
+#define CRC_LOAD(s) \
+    do { \
+        __m128i xmm_crc0 = _mm_loadu_si128((__m128i *)s->crc0 + 0);\
+        __m128i xmm_crc1 = _mm_loadu_si128((__m128i *)s->crc0 + 1);\
+        __m128i xmm_crc2 = _mm_loadu_si128((__m128i *)s->crc0 + 2);\
+        __m128i xmm_crc3 = _mm_loadu_si128((__m128i *)s->crc0 + 3);\
+        __m128i xmm_crc_part = _mm_loadu_si128((__m128i *)s->crc0 + 4);
+
+#define CRC_SAVE(s) \
+        _mm_storeu_si128((__m128i *)s->crc0 + 0, xmm_crc0);\
+        _mm_storeu_si128((__m128i *)s->crc0 + 1, xmm_crc1);\
+        _mm_storeu_si128((__m128i *)s->crc0 + 2, xmm_crc2);\
+        _mm_storeu_si128((__m128i *)s->crc0 + 3, xmm_crc3);\
+        _mm_storeu_si128((__m128i *)s->crc0 + 4, xmm_crc_part);\
+    } while (0);
+
+void crc_fold_init(crc_state *const s) {
+    CRC_LOAD(s)
+
+    xmm_crc0 = _mm_cvtsi32_si128(0x9db42487);
+    xmm_crc1 = _mm_setzero_si128();
+    xmm_crc2 = _mm_setzero_si128();
+    xmm_crc3 = _mm_setzero_si128();
+
+    CRC_SAVE(s)
+}
+
+void crc_fold(crc_state *const s, const unsigned char *src, long len) {
+    unsigned long algn_diff;
+    __m128i xmm_t0, xmm_t1, xmm_t2, xmm_t3;
+
+    CRC_LOAD(s)
+
+    if (len < 16) {
+        if (len == 0)
+            return;
+        xmm_crc_part = _mm_loadu_si128((__m128i *)src);
+        goto partial;
+    }
+
+    algn_diff = (0 - (uintptr_t)src) & 0xF;
+    if (algn_diff) {
+        xmm_crc_part = _mm_loadu_si128((__m128i *)src);
+        src += algn_diff;
+        len -= algn_diff;
+
+        partial_fold(algn_diff, &xmm_crc0, &xmm_crc1, &xmm_crc2, &xmm_crc3, &xmm_crc_part);
+    }
+
+    while ((len -= 64) >= 0) {
+        xmm_t0 = _mm_load_si128((__m128i *)src);
+        xmm_t1 = _mm_load_si128((__m128i *)src + 1);
+        xmm_t2 = _mm_load_si128((__m128i *)src + 2);
+        xmm_t3 = _mm_load_si128((__m128i *)src + 3);
+
+        fold_4(&xmm_crc0, &xmm_crc1, &xmm_crc2, &xmm_crc3);
+
+        xmm_crc0 = _mm_xor_si128(xmm_crc0, xmm_t0);
+        xmm_crc1 = _mm_xor_si128(xmm_crc1, xmm_t1);
+        xmm_crc2 = _mm_xor_si128(xmm_crc2, xmm_t2);
+        xmm_crc3 = _mm_xor_si128(xmm_crc3, xmm_t3);
+
+        src += 64;
+    }
+
+    /*
+     * len = num bytes left - 64
+     */
+    if (len + 16 >= 0) {
+        len += 16;
+
+        xmm_t0 = _mm_load_si128((__m128i *)src);
+        xmm_t1 = _mm_load_si128((__m128i *)src + 1);
+        xmm_t2 = _mm_load_si128((__m128i *)src + 2);
+
+        fold_3(&xmm_crc0, &xmm_crc1, &xmm_crc2, &xmm_crc3);
+
+        xmm_crc1 = _mm_xor_si128(xmm_crc1, xmm_t0);
+        xmm_crc2 = _mm_xor_si128(xmm_crc2, xmm_t1);
+        xmm_crc3 = _mm_xor_si128(xmm_crc3, xmm_t2);
+
+        if (len == 0)
+            goto done;
+
+        xmm_crc_part = _mm_load_si128((__m128i *)src + 3);
+    } else if (len + 32 >= 0) {
+        len += 32;
+
+        xmm_t0 = _mm_load_si128((__m128i *)src);
+        xmm_t1 = _mm_load_si128((__m128i *)src + 1);
+
+        fold_2(&xmm_crc0, &xmm_crc1, &xmm_crc2, &xmm_crc3);
+
+        xmm_crc2 = _mm_xor_si128(xmm_crc2, xmm_t0);
+        xmm_crc3 = _mm_xor_si128(xmm_crc3, xmm_t1);
+
+        if (len == 0)
+            goto done;
+
+        xmm_crc_part = _mm_load_si128((__m128i *)src + 2);
+    } else if (len + 48 >= 0) {
+        len += 48;
+
+        xmm_t0 = _mm_load_si128((__m128i *)src);
+
+        fold_1(&xmm_crc0, &xmm_crc1, &xmm_crc2, &xmm_crc3);
+
+        xmm_crc3 = _mm_xor_si128(xmm_crc3, xmm_t0);
+
+        if (len == 0)
+            goto done;
+
+        xmm_crc_part = _mm_load_si128((__m128i *)src + 1);
+    } else {
+        len += 64;
+        if (len == 0)
+            goto done;
+        xmm_crc_part = _mm_load_si128((__m128i *)src);
+    }
+
+partial:
+    partial_fold(len, &xmm_crc0, &xmm_crc1, &xmm_crc2, &xmm_crc3, &xmm_crc_part);
+done:
+    CRC_SAVE(s)
+}
+
+uint32_t crc_fold_512to32(crc_state *const s) {
+    const __m128i xmm_mask  = _mm_load_si128((__m128i *)crc_mask);
+    const __m128i xmm_mask2 = _mm_load_si128((__m128i *)crc_mask2);
+
+    uint32_t crc;
+    __m128i x_tmp0, x_tmp1, x_tmp2, crc_fold;
+
+    CRC_LOAD(s)
+
+    /*
+     * k1
+     */
+    crc_fold = _mm_load_si128((__m128i *)crc_k);
+
+    x_tmp0 = _mm_clmulepi64_si128(xmm_crc0, crc_fold, 0x10);
+    xmm_crc0 = _mm_clmulepi64_si128(xmm_crc0, crc_fold, 0x01);
+    xmm_crc1 = _mm_xor_si128(xmm_crc1, x_tmp0);
+    xmm_crc1 = _mm_xor_si128(xmm_crc1, xmm_crc0);
+
+    x_tmp1 = _mm_clmulepi64_si128(xmm_crc1, crc_fold, 0x10);
+    xmm_crc1 = _mm_clmulepi64_si128(xmm_crc1, crc_fold, 0x01);
+    xmm_crc2 = _mm_xor_si128(xmm_crc2, x_tmp1);
+    xmm_crc2 = _mm_xor_si128(xmm_crc2, xmm_crc1);
+
+    x_tmp2 = _mm_clmulepi64_si128(xmm_crc2, crc_fold, 0x10);
+    xmm_crc2 = _mm_clmulepi64_si128(xmm_crc2, crc_fold, 0x01);
+    xmm_crc3 = _mm_xor_si128(xmm_crc3, x_tmp2);
+    xmm_crc3 = _mm_xor_si128(xmm_crc3, xmm_crc2);
+
+    /*
+     * k5
+     */
+    crc_fold = _mm_load_si128((__m128i *)crc_k + 1);
+
+    xmm_crc0 = xmm_crc3;
+    xmm_crc3 = _mm_clmulepi64_si128(xmm_crc3, crc_fold, 0);
+    xmm_crc0 = _mm_srli_si128(xmm_crc0, 8);
+    xmm_crc3 = _mm_xor_si128(xmm_crc3, xmm_crc0);
+
+    xmm_crc0 = xmm_crc3;
+    xmm_crc3 = _mm_slli_si128(xmm_crc3, 4);
+    xmm_crc3 = _mm_clmulepi64_si128(xmm_crc3, crc_fold, 0x10);
+    xmm_crc3 = _mm_xor_si128(xmm_crc3, xmm_crc0);
+    xmm_crc3 = _mm_and_si128(xmm_crc3, xmm_mask2);
+
+    /*
+     * k7
+     */
+    xmm_crc1 = xmm_crc3;
+    xmm_crc2 = xmm_crc3;
+    crc_fold = _mm_load_si128((__m128i *)crc_k + 2);
+
+    xmm_crc3 = _mm_clmulepi64_si128(xmm_crc3, crc_fold, 0);
+    xmm_crc3 = _mm_xor_si128(xmm_crc3, xmm_crc2);
+    xmm_crc3 = _mm_and_si128(xmm_crc3, xmm_mask);
+
+    xmm_crc2 = xmm_crc3;
+    xmm_crc3 = _mm_clmulepi64_si128(xmm_crc3, crc_fold, 0x10);
+    xmm_crc3 = _mm_xor_si128(xmm_crc3, xmm_crc2);
+    xmm_crc3 = _mm_xor_si128(xmm_crc3, xmm_crc1);
+
+    crc = _mm_extract_epi32(xmm_crc3, 2);
+    return ~crc;
+    CRC_SAVE(s)
+}
+#endif
+
+void init_crc_pclmul()
+{
+#ifdef __PCLMUL__
+	crc_init = &crc_fold_init;
+	crc_incr = &crc_fold;
+	crc_finish = &crc_fold_512to32;
+	crc_simd = true;
+#endif
+}
+
+}

lib/yencode/ScalarDecoder.cpp

@@ -0,0 +1,213 @@
+/*
+ *  Based on node-yencode library by Anime Tosho:
+ *  https://github.com/animetosho/node-yencode
+ *
+ *  Copyright (C) 2017 Anime Tosho (animetosho)
+ *  Copyright (C) 2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+
+#include "nzbget.h"
+
+#include "YEncode.h"
+
+
+namespace YEncode
+{
+
+// return values:
+// - 0: no end sequence found
+// - 1: \r\n=y sequence found, src points to byte after 'y'
+// - 2: \r\n.\r\n sequence found, src points to byte after last '\n'
+int decode_scalar(const unsigned char** src, unsigned char** dest, size_t len, YencDecoderState* state) {
+	const unsigned char *es = (*src) + len; // end source pointer
+	unsigned char *p = *dest; // destination pointer
+	long i = -(long)len; // input position
+	unsigned char c; // input character
+	
+	if(len < 1) return 0;
+	
+#define YDEC_CHECK_END(s) if(i == 0) { \
+	*state = s; \
+	*src = es; \
+	*dest = p; \
+	return 0; \
+}
+	if(state) switch(*state) {
+		case YDEC_STATE_CRLFEQ: do_decode_endable_scalar_ceq:
+			if(es[i] == 'y') {
+				*state = YDEC_STATE_NONE;
+				*src = es+i+1;
+				*dest = p;
+				return 1;
+			} // else fall thru and escape
+		case YDEC_STATE_EQ:
+			c = es[i];
+			*p++ = c - 42 - 64;
+			i++;
+			if(c != '\r') break;
+			YDEC_CHECK_END(YDEC_STATE_CR)
+			// fall through
+		case YDEC_STATE_CR:
+			if(es[i] != '\n') break;
+			i++;
+			YDEC_CHECK_END(YDEC_STATE_CRLF)
+		case YDEC_STATE_CRLF: do_decode_endable_scalar_c0:
+			if(es[i] == '.') {
+				i++;
+				YDEC_CHECK_END(YDEC_STATE_CRLFDT)
+				// fallthru
+			} else if(es[i] == '=') {
+				i++;
+				YDEC_CHECK_END(YDEC_STATE_CRLFEQ)
+				goto do_decode_endable_scalar_ceq;
+			} else
+				break;
+		case YDEC_STATE_CRLFDT:
+			if(es[i] == '\r') {
+				i++;
+				YDEC_CHECK_END(YDEC_STATE_CRLFDTCR)
+				// fallthru
+			} else if(es[i] == '=') { // check for dot-stuffed ending: \r\n.=y
+				i++;
+				YDEC_CHECK_END(YDEC_STATE_CRLFEQ)
+				goto do_decode_endable_scalar_ceq;
+			} else
+				break;
+		case YDEC_STATE_CRLFDTCR:
+			if(es[i] == '\n') {
+				*state = YDEC_STATE_CRLF;
+				*src = es + i + 1;
+				*dest = p;
+				return 2;
+			} else
+				break;
+		case YDEC_STATE_NONE: break; // silence compiler warning
+	} else // treat as YDEC_STATE_CRLF
+		goto do_decode_endable_scalar_c0;
+	
+	for(; i < -2; i++) {
+		c = es[i];
+		switch(c) {
+			case '\r': {
+				if(es[i+1] == '\n') {
+					c = es[i+2];
+					if(c == '.') {
+						// skip past \r\n. sequences
+						i += 3;
+						YDEC_CHECK_END(YDEC_STATE_CRLFDT)
+						// check for end
+						if(es[i] == '\r') {
+							i++;
+							YDEC_CHECK_END(YDEC_STATE_CRLFDTCR)
+							if(es[i] == '\n') {
+								*src = es + i + 1;
+								*dest = p;
+								*state = YDEC_STATE_CRLF;
+								return 2;
+							} else i--;
+						} else if(es[i] == '=') {
+							i++;
+							YDEC_CHECK_END(YDEC_STATE_CRLFEQ)
+							if(es[i] == 'y') {
+								*src = es + i + 1;
+								*dest = p;
+								*state = YDEC_STATE_NONE;
+								return 1;
+							} else {
+								// escape char & continue
+								c = es[i];
+								*p++ = c - 42 - 64;
+								i -= (c == '\r');
+							}
+						} else i--;
+					}
+					else if(c == '=') {
+						i += 3;
+						YDEC_CHECK_END(YDEC_STATE_CRLFEQ)
+						if(es[i] == 'y') {
+							// ended
+							*src = es + i + 1;
+							*dest = p;
+							*state = YDEC_STATE_NONE;
+							return 1;
+						} else {
+							// escape char & continue
+							c = es[i];
+							*p++ = c - 42 - 64;
+							i -= (c == '\r');
+						}
+					}
+				}
+			} case '\n':
+				continue;
+			case '=':
+				c = es[i+1];
+				*p++ = c - 42 - 64;
+				i += (c != '\r'); // if we have a \r, reprocess character to deal with \r\n. case
+				continue;
+			default:
+				*p++ = c - 42;
+		}
+	}
+	if(state) *state = YDEC_STATE_NONE;
+	
+	if(i == -2) { // 2nd last char
+		c = es[i];
+		switch(c) {
+			case '\r':
+				if(state && es[i+1] == '\n') {
+					*state = YDEC_STATE_CRLF;
+					*src = es;
+					*dest = p;
+					return 0;
+				}
+			case '\n':
+				break;
+			case '=':
+				c = es[i+1];
+				*p++ = c - 42 - 64;
+				i += (c != '\r');
+				break;
+			default:
+				*p++ = c - 42;
+		}
+		i++;
+	}
+	
+	// do final char; we process this separately to prevent an overflow if the final char is '='
+	if(i == -1) {
+		c = es[i];
+		if(c != '\n' && c != '\r' && c != '=') {
+			*p++ = c - 42;
+		} else if(state) {
+			if(c == '=') *state = YDEC_STATE_EQ;
+			else if(c == '\r') *state = YDEC_STATE_CR;
+			else *state = YDEC_STATE_NONE;
+		}
+	}
+#undef YDEC_CHECK_END
+	
+	*src = es;
+	*dest = p;
+	return 0;
+}
+
+void init_decode_scalar() {
+	decode = decode_scalar;
+}
+
+}

lib/yencode/SimdDecoder.cpp

@@ -0,0 +1,674 @@
+/*
+ *  Based on node-yencode library by Anime Tosho:
+ *  https://github.com/animetosho/node-yencode
+ *
+ *  Copyright (C) 2017 Anime Tosho (animetosho)
+ *  Copyright (C) 2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+
+#ifdef SIMD_DECODER
+
+// combine two 8-bit ints into a 16-bit one
+#if __BYTE_ORDER == __LITTLE_ENDIAN
+#define UINT16_PACK(a, b) ((a) | ((b) << 8))
+#define UINT32_PACK(a, b, c, d) ((a) | ((b) << 8) | ((c) << 16) | ((d) << 24))
+#else
+#define UINT16_PACK(a, b) (((a) << 8) | (b))
+#define UINT32_PACK(a, b, c, d) (((a) << 24) | ((b) << 16) | ((c) << 8) | (d))
+#endif
+
+// table from http://graphics.stanford.edu/~seander/bithacks.html#CountBitsSetTable
+static const unsigned char BitsSetTable256[256] = 
+{
+#   define B2(n) n,     n+1,     n+1,     n+2
+#   define B4(n) B2(n), B2(n+1), B2(n+1), B2(n+2)
+#   define B6(n) B4(n), B4(n+1), B4(n+1), B4(n+2)
+    B6(0), B6(1), B6(1), B6(2)
+#undef B2
+#undef B4
+#undef B6
+};
+
+template<int width, void kernel(size_t&, const uint8_t*, unsigned char*&, unsigned char&, uint16_t&)>
+int do_decode_simd(const unsigned char** src, unsigned char** dest, size_t len, YencDecoderState* state) {
+	if(len <= width*2) return decode_scalar(src, dest, len, state);
+	
+	YencDecoderState tState = YDEC_STATE_CRLF;
+	YencDecoderState* pState = state ? state : &tState;
+	if((uintptr_t)(*src) & ((width-1))) {
+		// find source memory alignment
+		unsigned char* aSrc = (unsigned char*)(((uintptr_t)(*src) + (width-1)) & ~(width-1));
+		int amount = (int)(aSrc - *src);
+		len -= amount;
+		int ended = decode_scalar(src, dest, amount, pState);
+		if(ended) return ended;
+	}
+	
+	size_t lenBuffer = width -1;
+	lenBuffer += 3 + 1;
+
+	if(len > lenBuffer) {
+		unsigned char *p = *dest; // destination pointer
+		unsigned char escFirst = 0; // input character; first char needs escaping
+		uint16_t nextMask = 0;
+		// handle finicky case of special sequences straddled across initial boundary
+		switch(*pState) {
+			case YDEC_STATE_CRLF:
+				if(**src == '.') {
+					nextMask = 1;
+					if(*(uint16_t*)(*src +1) == UINT16_PACK('\r','\n')) {
+						(*src) += 3;
+						*pState = YDEC_STATE_CRLF;
+						return 2;
+					}
+					if(*(uint16_t*)(*src +1) == UINT16_PACK('=','y')) {
+						(*src) += 3;
+						*pState = YDEC_STATE_NONE;
+						return 1;
+					}
+				}
+				else if(*(uint16_t*)(*src) == UINT16_PACK('=','y')) {
+					(*src) += 2;
+					*pState = YDEC_STATE_NONE;
+					return 1;
+				}
+				break;
+			case YDEC_STATE_CR:
+				if(*(uint16_t*)(*src) == UINT16_PACK('\n','.')) {
+					nextMask = 2;
+					if(*(uint16_t*)(*src +2) == UINT16_PACK('\r','\n')) {
+						(*src) += 4;
+						*pState = YDEC_STATE_CRLF;
+						return 2;
+					}
+					if(*(uint16_t*)(*src +2) == UINT16_PACK('=','y')) {
+						(*src) += 4;
+						*pState = YDEC_STATE_NONE;
+						return 1;
+					}
+				}
+				else if((*(uint32_t*)(*src) & 0xffffff) == UINT32_PACK('\n','=','y',0)) {
+					(*src) += 3;
+					*pState = YDEC_STATE_NONE;
+					return 1;
+				}
+				break;
+			case YDEC_STATE_CRLFDT:
+				if(*(uint16_t*)(*src) == UINT16_PACK('\r','\n')) {
+					(*src) += 2;
+					*pState = YDEC_STATE_CRLF;
+					return 2;
+				}
+				if(*(uint16_t*)(*src) == UINT16_PACK('=','y')) {
+					(*src) += 2;
+					*pState = YDEC_STATE_NONE;
+					return 1;
+				}
+				break;
+			case YDEC_STATE_CRLFDTCR:
+				if(**src == '\n') {
+					(*src) += 1;
+					*pState = YDEC_STATE_CRLF;
+					return 2;
+				}
+				break;
+			case YDEC_STATE_CRLFEQ:
+				if(**src == 'y') {
+					(*src) += 1;
+					*pState = YDEC_STATE_NONE;
+					return 1;
+				}
+				break;
+			default: break; // silence compiler warning
+		}
+		escFirst = (*pState == YDEC_STATE_EQ || *pState == YDEC_STATE_CRLFEQ);
+		
+		// our algorithm may perform an aligned load on the next part, of which we consider 2 bytes (for \r\n. sequence checking)
+		size_t dLen = len - lenBuffer;
+		dLen = (dLen + (width-1)) & ~(width-1);
+		const uint8_t* dSrc = (const uint8_t*)(*src) + dLen;
+
+		kernel(dLen, dSrc, p, escFirst, nextMask);
+
+		if(escFirst) *pState = YDEC_STATE_EQ; // escape next character
+		else if(nextMask == 1) *pState = YDEC_STATE_CRLF; // next character is '.', where previous two were \r\n
+		else if(nextMask == 2) *pState = YDEC_STATE_CR; // next characters are '\n.', previous is \r
+		else *pState = YDEC_STATE_NONE;
+		
+		*src += dLen;
+		len -= dLen;
+		*dest = p;
+	}
+	
+	// end alignment
+	if(len)
+		return decode_scalar(src, dest, len, pState);
+
+	return 0;
+}
+
+alignas(32) static const uint8_t pshufb_combine_table[272] = {
+	0x00,0x01,0x02,0x03,0x04,0x05,0x06,0x07,0x08,0x09,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f,
+	0x00,0x01,0x02,0x03,0x04,0x05,0x06,0x08,0x09,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f,0x80,
+	0x00,0x01,0x02,0x03,0x04,0x05,0x08,0x09,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f,0x80,0x80,
+	0x00,0x01,0x02,0x03,0x04,0x08,0x09,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f,0x80,0x80,0x80,
+	0x00,0x01,0x02,0x03,0x08,0x09,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f,0x80,0x80,0x80,0x80,
+	0x00,0x01,0x02,0x08,0x09,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f,0x80,0x80,0x80,0x80,0x80,
+	0x00,0x01,0x08,0x09,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f,0x80,0x80,0x80,0x80,0x80,0x80,
+	0x00,0x08,0x09,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f,0x80,0x80,0x80,0x80,0x80,0x80,0x80,
+	0x08,0x09,0x0a,0x0b,0x0c,0x0d,0x0e,0x0f,0x80,0x80,0x80,0x80,0x80,0x80,0x80,0x80,
+};
+
+#ifdef __SSE2__
+
+#define XMM_SIZE 16 /*== (signed int)sizeof(__m128i)*/
+
+#if defined(__tune_core2__) || defined(__tune_atom__)
+/* on older Intel CPUs, plus first gen Atom, it is faster to store XMM registers in half */
+# define STOREU_XMM(dest, xmm) \
+  _mm_storel_epi64((__m128i*)(dest), xmm); \
+  _mm_storeh_pi(((__m64*)(dest) +1), _mm_castsi128_ps(xmm))
+#else
+# define STOREU_XMM(dest, xmm) \
+  _mm_storeu_si128((__m128i*)(dest), xmm)
+#endif
+
+#define LOAD_HALVES(a, b) _mm_castps_si128(_mm_loadh_pi( \
+	_mm_castsi128_ps(_mm_loadl_epi64((__m128i*)(a))), \
+	(b) \
+))
+
+uint8_t eqFixLUT[256];
+alignas(32) __m64 eqAddLUT[256];
+#ifdef __SSSE3__
+alignas(32) __m64 unshufLUT[256];
+#endif
+
+template<bool use_ssse3>
+static inline void do_decode_sse(size_t& dLen, const uint8_t* dSrc, unsigned char*& p, unsigned char& escFirst, uint16_t& nextMask) {
+	long dI = -(long)dLen;
+
+	for(; dI; dI += sizeof(__m128i)) {
+		const uint8_t* src = dSrc + dI;
+
+		__m128i data = _mm_load_si128((__m128i *)src);
+		
+		// search for special chars
+		__m128i cmpEq = _mm_cmpeq_epi8(data, _mm_set1_epi8('=')),
+#ifdef __AVX512VL__
+		cmp = _mm_ternarylogic_epi32(
+			_mm_cmpeq_epi8(data, _mm_set1_epi16(0x0a0d)),
+			_mm_cmpeq_epi8(data, _mm_set1_epi16(0x0d0a)),
+			cmpEq,
+			0xFE
+		);
+#else
+		cmp = _mm_or_si128(
+			_mm_or_si128(
+				_mm_cmpeq_epi8(data, _mm_set1_epi16(0x0a0d)), // \r\n
+				_mm_cmpeq_epi8(data, _mm_set1_epi16(0x0d0a))  // \n\r
+			),
+			cmpEq
+		);
+#endif
+		uint16_t mask = _mm_movemask_epi8(cmp); // not the most accurate mask if we have invalid sequences; we fix this up later
+		
+		__m128i oData;
+		if(escFirst) { // rarely hit branch: seems to be faster to use 'if' than a lookup table, possibly due to values being able to be held in registers?
+			// first byte needs escaping due to preceeding = in last loop iteration
+			oData = _mm_sub_epi8(data, _mm_set_epi8(42,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42+64));
+			mask &= ~1;
+		} else {
+			oData = _mm_sub_epi8(data, _mm_set1_epi8(42));
+		}
+		mask |= nextMask;
+		
+		if (mask != 0) {
+			// a spec compliant encoder should never generate sequences: ==, =\n and =\r, but we'll handle them to be spec compliant
+			// the yEnc specification requires any character following = to be unescaped, not skipped over, so we'll deal with that
+			
+			// firstly, resolve invalid sequences of = to deal with cases like '===='
+			uint16_t maskEq = _mm_movemask_epi8(cmpEq);
+			uint16_t tmp = eqFixLUT[(maskEq&0xff) & ~escFirst];
+			maskEq = (eqFixLUT[(maskEq>>8) & ~(tmp>>7)] << 8) | tmp;
+			
+			unsigned char oldEscFirst = escFirst;
+			escFirst = (maskEq >> (sizeof(__m128i)-1));
+			// next, eliminate anything following a `=` from the special char mask; this eliminates cases of `=\r` so that they aren't removed
+			maskEq <<= 1;
+			mask &= ~maskEq;
+			
+			// unescape chars following `=`
+#if defined(__AVX512VL__) && defined(__AVX512BW__)
+			// GCC < 7 seems to generate rubbish assembly for this
+			oData = _mm_mask_add_epi8(
+				oData,
+				maskEq,
+				oData,
+				_mm_set1_epi8(-64)
+			);
+#else
+			oData = _mm_add_epi8(
+				oData,
+				LOAD_HALVES(
+					eqAddLUT + (maskEq&0xff),
+					eqAddLUT + ((maskEq>>8)&0xff)
+				)
+			);
+#endif
+			
+			// handle \r\n. sequences
+			// RFC3977 requires the first dot on a line to be stripped, due to dot-stuffing
+			// find instances of \r\n
+			__m128i tmpData1, tmpData2, tmpData3, tmpData4;
+#if defined(__SSSE3__) && !defined(__tune_btver1__)
+			if(use_ssse3) {
+				__m128i nextData = _mm_load_si128((__m128i *)src + 1);
+				tmpData1 = _mm_alignr_epi8(nextData, data, 1);
+				tmpData2 = _mm_alignr_epi8(nextData, data, 2);
+				tmpData3 = _mm_alignr_epi8(nextData, data, 3);
+				tmpData4 = _mm_alignr_epi8(nextData, data, 4);
+			} else {
+#endif
+				tmpData1 = _mm_insert_epi16(_mm_srli_si128(data, 1), *(uint16_t*)(src + sizeof(__m128i)-1), 7);
+				tmpData2 = _mm_insert_epi16(_mm_srli_si128(data, 2), *(uint16_t*)(src + sizeof(__m128i)), 7);
+				tmpData3 = _mm_insert_epi16(_mm_srli_si128(tmpData1, 2), *(uint16_t*)(src + sizeof(__m128i)+1), 7);
+				tmpData4 = _mm_insert_epi16(_mm_srli_si128(tmpData2, 2), *(uint16_t*)(src + sizeof(__m128i)+2), 7);
+#ifdef __SSSE3__
+			}
+#endif
+			__m128i matchNl1 = _mm_cmpeq_epi16(data, _mm_set1_epi16(0x0a0d));
+			__m128i matchNl2 = _mm_cmpeq_epi16(tmpData1, _mm_set1_epi16(0x0a0d));
+			
+			__m128i matchDots, matchNlDots;
+			uint16_t killDots;
+			matchDots = _mm_cmpeq_epi8(tmpData2, _mm_set1_epi8('.'));
+			// merge preparation (for non-raw, it doesn't matter if this is shifted or not)
+			matchNl1 = _mm_srli_si128(matchNl1, 1);
+			
+			// merge matches of \r\n with those for .
+#ifdef __AVX512VL__
+			matchNlDots = _mm_ternarylogic_epi32(matchDots, matchNl1, matchNl2, 0xE0);
+#else
+			matchNlDots = _mm_and_si128(matchDots, _mm_or_si128(matchNl1, matchNl2));
+#endif
+			killDots = _mm_movemask_epi8(matchNlDots);
+
+			__m128i cmpB1 = _mm_cmpeq_epi16(tmpData2, _mm_set1_epi16(0x793d)); // "=y"
+			__m128i cmpB2 = _mm_cmpeq_epi16(tmpData3, _mm_set1_epi16(0x793d));
+			if(killDots) {
+				// match instances of \r\n.\r\n and \r\n.=y
+				__m128i cmpC1 = _mm_cmpeq_epi16(tmpData3, _mm_set1_epi16(0x0a0d)); // "\r\n"
+				__m128i cmpC2 = _mm_cmpeq_epi16(tmpData4, _mm_set1_epi16(0x0a0d));
+				cmpC1 = _mm_or_si128(cmpC1, cmpB2);
+				cmpC2 = _mm_or_si128(cmpC2, _mm_cmpeq_epi16(tmpData4, _mm_set1_epi16(0x793d)));
+				cmpC2 = _mm_slli_si128(cmpC2, 1);
+				
+				// prepare cmpB
+				cmpB1 = _mm_and_si128(cmpB1, matchNl1);
+				cmpB2 = _mm_and_si128(cmpB2, matchNl2);
+				
+				// and w/ dots
+#ifdef __AVX512VL__
+				cmpC1 = _mm_ternarylogic_epi32(cmpC1, cmpC2, matchNlDots, 0xA8);
+				cmpB1 = _mm_ternarylogic_epi32(cmpB1, cmpB2, cmpC1, 0xFE);
+#else
+				cmpC1 = _mm_and_si128(_mm_or_si128(cmpC1, cmpC2), matchNlDots);
+				cmpB1 = _mm_or_si128(cmpC1, _mm_or_si128(
+					cmpB1, cmpB2
+				));
+#endif
+			} else {
+#ifdef __AVX512VL__
+				cmpB1 = _mm_ternarylogic_epi32(cmpB1, matchNl1, _mm_and_si128(cmpB2, matchNl2), 0xEA);
+#else
+				cmpB1 = _mm_or_si128(
+					_mm_and_si128(cmpB1, matchNl1),
+					_mm_and_si128(cmpB2, matchNl2)
+				);
+#endif
+			}
+			if(_mm_movemask_epi8(cmpB1)) {
+				// terminator found
+				// there's probably faster ways to do this, but reverting to scalar code should be good enough
+				escFirst = oldEscFirst;
+				dLen += dI;
+				return;
+			}
+			mask |= (killDots << 2) & 0xffff;
+			nextMask = killDots >> (sizeof(__m128i)-2);
+
+			// all that's left is to 'compress' the data (skip over masked chars)
+#ifdef __SSSE3__
+			if(use_ssse3) {
+# if defined(__POPCNT__) && (defined(__tune_znver1__) || defined(__tune_btver2__))
+				unsigned char skipped = _mm_popcnt_u32(mask & 0xff);
+# else
+				unsigned char skipped = BitsSetTable256[mask & 0xff];
+# endif
+				// lookup compress masks and shuffle
+				// load up two halves
+				__m128i shuf = LOAD_HALVES(unshufLUT + (mask&0xff), unshufLUT + (mask>>8));
+				
+				// offset upper half by 8
+				shuf = _mm_add_epi8(shuf, _mm_set_epi32(0x08080808, 0x08080808, 0, 0));
+				// shift down upper half into lower
+				// TODO: consider using `mask & 0xff` in table instead of counting bits
+				shuf = _mm_shuffle_epi8(shuf, _mm_load_si128((const __m128i*)pshufb_combine_table + skipped));
+				
+				// shuffle data
+				oData = _mm_shuffle_epi8(oData, shuf);
+				STOREU_XMM(p, oData);
+				
+				// increment output position
+# if defined(__POPCNT__) && !defined(__tune_btver1__)
+				p += XMM_SIZE - _mm_popcnt_u32(mask);
+# else
+				p += XMM_SIZE - skipped - BitsSetTable256[mask >> 8];
+# endif
+				
+			} else {
+#endif
+				alignas(32) uint32_t mmTmp[4];
+				_mm_store_si128((__m128i*)mmTmp, oData);
+				
+				for(int j=0; j<4; j++) {
+					if(mask & 0xf) {
+						unsigned char* pMmTmp = (unsigned char*)(mmTmp + j);
+						unsigned int maskn = ~mask;
+						*p = pMmTmp[0];
+						p += (maskn & 1);
+						*p = pMmTmp[1];
+						p += (maskn & 2) >> 1;
+						*p = pMmTmp[2];
+						p += (maskn & 4) >> 2;
+						*p = pMmTmp[3];
+						p += (maskn & 8) >> 3;
+					} else {
+						*(uint32_t*)p = mmTmp[j];
+						p += 4;
+					}
+					mask >>= 4;
+				}
+#ifdef __SSSE3__
+			}
+#endif
+		} else {
+			STOREU_XMM(p, oData);
+			p += XMM_SIZE;
+			escFirst = 0;
+			nextMask = 0;
+		}
+	}
+}
+#endif
+
+
+#ifdef __ARM_NEON
+inline uint16_t neon_movemask(uint8x16_t in) {
+	uint8x16_t mask = vandq_u8(in, (uint8x16_t){1,2,4,8,16,32,64,128, 1,2,4,8,16,32,64,128});
+# if defined(__aarch64__)
+	return (vaddv_u8(vget_high_u8(mask)) << 8) | vaddv_u8(vget_low_u8(mask));
+# else
+	uint8x8_t res = vpadd_u8(vget_low_u8(mask), vget_high_u8(mask));
+	res = vpadd_u8(res, res);
+	res = vpadd_u8(res, res);
+	return vget_lane_u16(vreinterpret_u16_u8(res), 0);
+# endif
+}
+
+uint8_t eqFixLUT[256];
+alignas(32) uint8x8_t eqAddLUT[256];
+alignas(32) uint8x8_t unshufLUT[256];
+
+static inline void do_decode_neon(size_t& dLen, const uint8_t* dSrc, unsigned char*& p, unsigned char& escFirst, uint16_t& nextMask) {
+	long dI = -(long)dLen;
+
+	for(; dI; dI += sizeof(uint8x16_t)) {
+		const uint8_t* src = dSrc + dI;
+
+		uint8x16_t data = vld1q_u8(src);
+		
+		// search for special chars
+		uint8x16_t cmpEq = vceqq_u8(data, vdupq_n_u8('=')),
+		cmp = vorrq_u8(
+			vorrq_u8(
+				vceqq_u8(data, vreinterpretq_u8_u16(vdupq_n_u16(0x0a0d))), // \r\n
+				vceqq_u8(data, vreinterpretq_u8_u16(vdupq_n_u16(0x0d0a)))  // \n\r
+			),
+			cmpEq
+		);
+		uint16_t mask = neon_movemask(cmp); // not the most accurate mask if we have invalid sequences; we fix this up later
+		
+		uint8x16_t oData;
+		if(escFirst) { // rarely hit branch: seems to be faster to use 'if' than a lookup table, possibly due to values being able to be held in registers?
+			// first byte needs escaping due to preceeding = in last loop iteration
+			oData = vsubq_u8(data, (uint8x16_t){42+64,42,42,42,42,42,42,42,42,42,42,42,42,42,42,42});
+			mask &= ~1;
+		} else {
+			oData = vsubq_u8(data, vdupq_n_u8(42));
+		}
+		mask |= nextMask;
+		
+		if (mask != 0) {
+			// a spec compliant encoder should never generate sequences: ==, =\n and =\r, but we'll handle them to be spec compliant
+			// the yEnc specification requires any character following = to be unescaped, not skipped over, so we'll deal with that
+			
+			// firstly, resolve invalid sequences of = to deal with cases like '===='
+			uint16_t maskEq = neon_movemask(cmpEq);
+			uint16_t tmp = eqFixLUT[(maskEq&0xff) & ~escFirst];
+			maskEq = (eqFixLUT[(maskEq>>8) & ~(tmp>>7)] << 8) | tmp;
+			
+			unsigned char oldEscFirst = escFirst;
+			escFirst = (maskEq >> (sizeof(uint8x16_t)-1));
+			// next, eliminate anything following a `=` from the special char mask; this eliminates cases of `=\r` so that they aren't removed
+			maskEq <<= 1;
+			mask &= ~maskEq;
+			
+			// unescape chars following `=`
+			oData = vaddq_u8(
+				oData,
+				vcombine_u8(
+					vld1_u8((uint8_t*)(eqAddLUT + (maskEq&0xff))),
+					vld1_u8((uint8_t*)(eqAddLUT + ((maskEq>>8)&0xff)))
+				)
+			);
+			
+			// handle \r\n. sequences
+			// RFC3977 requires the first dot on a line to be stripped, due to dot-stuffing
+			// find instances of \r\n
+			uint8x16_t tmpData1, tmpData2, tmpData3, tmpData4;
+			uint8x16_t nextData = vld1q_u8(src + sizeof(uint8x16_t));
+			tmpData1 = vextq_u8(data, nextData, 1);
+			tmpData2 = vextq_u8(data, nextData, 2);
+			tmpData3 = vextq_u8(data, nextData, 3);
+			tmpData4 = vextq_u8(data, nextData, 4);
+			uint8x16_t matchNl1 = vreinterpretq_u8_u16(vceqq_u16(vreinterpretq_u16_u8(data), vdupq_n_u16(0x0a0d)));
+			uint8x16_t matchNl2 = vreinterpretq_u8_u16(vceqq_u16(vreinterpretq_u16_u8(tmpData1), vdupq_n_u16(0x0a0d)));
+			
+			uint8x16_t matchDots, matchNlDots;
+			uint16_t killDots;
+			matchDots = vceqq_u8(tmpData2, vdupq_n_u8('.'));
+			// merge preparation (for non-raw, it doesn't matter if this is shifted or not)
+			matchNl1 = vextq_u8(matchNl1, vdupq_n_u8(0), 1);
+			
+			// merge matches of \r\n with those for .
+			matchNlDots = vandq_u8(matchDots, vorrq_u8(matchNl1, matchNl2));
+			killDots = neon_movemask(matchNlDots);
+
+			uint8x16_t cmpB1 = vreinterpretq_u8_u16(vceqq_u16(vreinterpretq_u16_u8(tmpData2), vdupq_n_u16(0x793d))); // "=y"
+			uint8x16_t cmpB2 = vreinterpretq_u8_u16(vceqq_u16(vreinterpretq_u16_u8(tmpData3), vdupq_n_u16(0x793d)));
+			if(killDots) {
+				// match instances of \r\n.\r\n and \r\n.=y
+				uint8x16_t cmpC1 = vreinterpretq_u8_u16(vceqq_u16(vreinterpretq_u16_u8(tmpData3), vdupq_n_u16(0x0a0d)));
+				uint8x16_t cmpC2 = vreinterpretq_u8_u16(vceqq_u16(vreinterpretq_u16_u8(tmpData4), vdupq_n_u16(0x0a0d)));
+				cmpC1 = vorrq_u8(cmpC1, cmpB2);
+				cmpC2 = vorrq_u8(cmpC2, vreinterpretq_u8_u16(vceqq_u16(vreinterpretq_u16_u8(tmpData4), vdupq_n_u16(0x793d))));
+				cmpC2 = vextq_u8(vdupq_n_u8(0), cmpC2, 15);
+				cmpC1 = vorrq_u8(cmpC1, cmpC2);
+				
+				// and w/ dots
+				cmpC1 = vandq_u8(cmpC1, matchNlDots);
+				// then merge w/ cmpB
+				cmpB1 = vandq_u8(cmpB1, matchNl1);
+				cmpB2 = vandq_u8(cmpB2, matchNl2);
+				
+				cmpB1 = vorrq_u8(cmpC1, vorrq_u8(
+					cmpB1, cmpB2
+				));
+			} else {
+				cmpB1 = vorrq_u8(
+					vandq_u8(cmpB1, matchNl1),
+					vandq_u8(cmpB2, matchNl2)
+				);
+			}
+#ifdef __aarch64__
+			if(vget_lane_u64(vreinterpret_u64_u32(vqmovn_u64(vreinterpretq_u64_u8(cmpB1))), 0))
+#else
+			uint32x4_t tmp1 = vreinterpretq_u32_u8(cmpB1);
+			uint32x2_t tmp2 = vorr_u32(vget_low_u32(tmp1), vget_high_u32(tmp1));
+			if(vget_lane_u32(vpmax_u32(tmp2, tmp2), 0))
+#endif
+			{
+				// terminator found
+				// there's probably faster ways to do this, but reverting to scalar code should be good enough
+				escFirst = oldEscFirst;
+				dLen += dI;
+				return;
+			}
+			mask |= (killDots << 2) & 0xffff;
+			nextMask = killDots >> (sizeof(uint8x16_t)-2);
+
+			// all that's left is to 'compress' the data (skip over masked chars)
+			unsigned char skipped = BitsSetTable256[mask & 0xff];
+			// lookup compress masks and shuffle
+			oData = vcombine_u8(
+				vtbl1_u8(vget_low_u8(oData),  vld1_u8((uint8_t*)(unshufLUT + (mask&0xff)))),
+				vtbl1_u8(vget_high_u8(oData), vld1_u8((uint8_t*)(unshufLUT + (mask>>8))))
+			);
+			// compact down
+			uint8x16_t compact = vld1q_u8(pshufb_combine_table + skipped*sizeof(uint8x16_t));
+#ifdef __aarch64__
+			oData = vqtbl1q_u8(oData, compact);
+#else
+			uint8x8x2_t dataH = {vget_low_u8(oData), vget_high_u8(oData)};
+			oData = vcombine_u8(vtbl2_u8(dataH, vget_low_u8(compact)),
+								vtbl2_u8(dataH, vget_high_u8(compact)));
+#endif
+			vst1q_u8(p, oData);
+			
+			// increment output position
+			p += sizeof(uint8x16_t) - skipped - BitsSetTable256[mask >> 8];
+			
+		} else {
+			vst1q_u8(p, oData);
+			p += sizeof(uint8x16_t);
+			escFirst = 0;
+			nextMask = 0;
+		}
+	}
+}
+#endif
+
+void decoder_init() {
+#ifdef __SSE2__
+	for(int i=0; i<256; i++) {
+		int k = i;
+		uint8_t res[8];
+		int p = 0;
+		
+		// fix LUT
+		k = i;
+		p = 0;
+		for(int j=0; j<8; j++) {
+			k = i >> j;
+			if(k & 1) {
+				p |= 1 << j;
+				j++;
+			}
+		}
+		eqFixLUT[i] = p;
+		
+		// sub LUT
+		k = i;
+		for(int j=0; j<8; j++) {
+			res[j] = (k & 1) ? 192 /* == -64 */ : 0;
+			k >>= 1;
+		}
+		_mm_storel_epi64((__m128i*)(eqAddLUT + i), _mm_loadl_epi64((__m128i*)res));
+	}
+#endif
+
+#ifdef __SSSE3__
+	// generate unshuf LUT
+	for(int i=0; i<256; i++) {
+		int k = i;
+		uint8_t res[8];
+		int p = 0;
+		for(int j=0; j<8; j++) {
+			if(!(k & 1)) {
+				res[p++] = j;
+			}
+			k >>= 1;
+		}
+		for(; p<8; p++)
+			res[p] = 0;
+		_mm_storel_epi64((__m128i*)(unshufLUT + i), _mm_loadl_epi64((__m128i*)res));
+	}
+#endif
+
+#ifdef __ARM_NEON
+	for(int i=0; i<256; i++) {
+		int k = i;
+		uint8_t res[8];
+		int p = 0;
+		
+		// fix LUT
+		k = i;
+		p = 0;
+		for(int j=0; j<8; j++) {
+			k = i >> j;
+			if(k & 1) {
+				p |= 1 << j;
+				j++;
+			}
+		}
+		eqFixLUT[i] = p;
+		
+		// sub LUT
+		k = i;
+		for(int j=0; j<8; j++) {
+			res[j] = (k & 1) ? 192 /* == -64 */ : 0;
+			k >>= 1;
+		}
+		vst1_u8((uint8_t*)(eqAddLUT + i), vld1_u8(res));
+		
+		k = i;
+		p = 0;
+		for(int j=0; j<8; j++) {
+			if(!(k & 1)) {
+				res[p++] = j;
+			}
+			k >>= 1;
+		}
+		for(; p<8; p++)
+			res[p] = 0;
+		vst1_u8((uint8_t*)(unshufLUT + i), vld1_u8(res));
+	}
+#endif
+}
+#endif

lib/yencode/SimdInit.cpp

@@ -0,0 +1,129 @@
+/*
+ *  Based on node-yencode library by Anime Tosho:
+ *  https://github.com/animetosho/node-yencode
+ *
+ *  Copyright (C) 2017 Anime Tosho (animetosho)
+ *  Copyright (C) 2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+
+#include "nzbget.h"
+
+#if (defined(__i686__) || defined(__amd64__)) && !defined(WIN32)
+#include <cpuid.h>
+#endif
+
+#include "YEncode.h"
+
+namespace YEncode
+{
+
+int (*decode)(const unsigned char**, unsigned char**, size_t, YencDecoderState*) = nullptr;
+extern void init_decode_scalar();
+bool decode_simd = false;
+
+void (*crc_init)(crc_state *const s) = nullptr;
+void (*crc_incr)(crc_state *const s, const unsigned char *src, long len) = nullptr;
+uint32_t (*crc_finish)(crc_state *const s) = nullptr;
+extern void init_crc_slice();
+bool crc_simd = false;
+
+#if defined(__i686__) || defined(__amd64__)
+extern void init_decode_sse2();
+extern void init_decode_ssse3();
+extern void init_crc_pclmul();
+
+class CpuId
+{
+	uint32_t regs[4];
+public:
+	CpuId(unsigned level)
+	{
+#ifdef WIN32
+		__cpuid((int *)regs, (int)level);
+#else
+		__cpuid(level, regs[0], regs[1], regs[2], regs[3]);
+#endif
+	}
+	const uint32_t &EAX() const {return regs[0];}
+	const uint32_t &EBX() const {return regs[1];}
+	const uint32_t &ECX() const {return regs[2];}
+	const uint32_t &EDX() const {return regs[3];}
+};
+#endif
+
+#if defined(__arm__) || defined(__aarch64__)
+extern void init_decode_neon();
+extern void init_crc_acle();
+#endif
+
+void init()
+{
+	init_decode_scalar();
+	init_crc_slice();
+
+#if defined(__i686__) || defined(__amd64__)
+	CpuId cpuid(1);
+
+	bool cpu_supports_sse2 = cpuid.EDX() & 0x04000000;
+	bool cpu_supports_ssse3 = cpuid.ECX() & 0x00000200;
+	bool cpu_supports_sse41 = cpuid.ECX() & 0x00080000;
+	bool cpu_supports_pclmul = cpuid.ECX() & 0x00000002;
+
+	if (cpu_supports_sse2)
+	{
+		init_decode_sse2();
+	}
+	if (cpu_supports_ssse3)
+	{
+		init_decode_ssse3();
+	}
+	if (cpu_supports_sse41 && cpu_supports_pclmul)
+	{
+		init_crc_pclmul();
+	}
+#endif
+
+#if defined(__arm__) || defined(__aarch64__)
+	bool cpu_supports_neon = false;
+	bool cpu_supports_crc = false;
+
+#ifdef __linux__
+	if (FILE* file = fopen("/proc/cpuinfo", "r"))
+	{
+		char buf[200];
+		while (fgets(buf, sizeof(buf), file))
+		{
+			cpu_supports_neon |= !strncasecmp(buf, "Features", 8) &&
+				(strstr(buf, " neon ") || strstr(buf, " asimd "));
+			cpu_supports_crc |= !strncasecmp(buf, "Features", 8) && strstr(buf, " crc32 ");
+		}
+		fclose(file);
+	}
+#endif
+
+	if (cpu_supports_neon)
+	{
+		init_decode_neon();
+	}
+	if (cpu_supports_crc)
+	{
+		init_crc_acle();
+	}
+#endif
+}
+
+}

lib/yencode/SliceCrc.cpp

@@ -0,0 +1,227 @@
+/*
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "nzbget.h"
+
+#include "YEncode.h"
+
+namespace YEncode
+{
+
+// Crc32 implementation:
+// http://www.intel.com/technology/comms/perfnet/download/CRC_generators.pdf
+// http://sourceforge.net/projects/slicing-by-8/
+
+static uint32_t crc32_tab[4][256] =
+{
+	{
+		0x00000000, 0x77073096, 0xEE0E612C, 0x990951BA, 0x076DC419, 0x706AF48F, 0xE963A535, 0x9E6495A3,
+		0x0EDB8832, 0x79DCB8A4, 0xE0D5E91E, 0x97D2D988, 0x09B64C2B, 0x7EB17CBD, 0xE7B82D07, 0x90BF1D91,
+		0x1DB71064, 0x6AB020F2, 0xF3B97148, 0x84BE41DE, 0x1ADAD47D, 0x6DDDE4EB, 0xF4D4B551, 0x83D385C7,
+		0x136C9856, 0x646BA8C0, 0xFD62F97A, 0x8A65C9EC, 0x14015C4F, 0x63066CD9, 0xFA0F3D63, 0x8D080DF5,
+		0x3B6E20C8, 0x4C69105E, 0xD56041E4, 0xA2677172, 0x3C03E4D1, 0x4B04D447, 0xD20D85FD, 0xA50AB56B,
+		0x35B5A8FA, 0x42B2986C, 0xDBBBC9D6, 0xACBCF940, 0x32D86CE3, 0x45DF5C75, 0xDCD60DCF, 0xABD13D59,
+		0x26D930AC, 0x51DE003A, 0xC8D75180, 0xBFD06116, 0x21B4F4B5, 0x56B3C423, 0xCFBA9599, 0xB8BDA50F,
+		0x2802B89E, 0x5F058808, 0xC60CD9B2, 0xB10BE924, 0x2F6F7C87, 0x58684C11, 0xC1611DAB, 0xB6662D3D,
+		0x76DC4190, 0x01DB7106, 0x98D220BC, 0xEFD5102A, 0x71B18589, 0x06B6B51F, 0x9FBFE4A5, 0xE8B8D433,
+		0x7807C9A2, 0x0F00F934, 0x9609A88E, 0xE10E9818, 0x7F6A0DBB, 0x086D3D2D, 0x91646C97, 0xE6635C01,
+		0x6B6B51F4, 0x1C6C6162, 0x856530D8, 0xF262004E, 0x6C0695ED, 0x1B01A57B, 0x8208F4C1, 0xF50FC457,
+		0x65B0D9C6, 0x12B7E950, 0x8BBEB8EA, 0xFCB9887C, 0x62DD1DDF, 0x15DA2D49, 0x8CD37CF3, 0xFBD44C65,
+		0x4DB26158, 0x3AB551CE, 0xA3BC0074, 0xD4BB30E2, 0x4ADFA541, 0x3DD895D7, 0xA4D1C46D, 0xD3D6F4FB,
+		0x4369E96A, 0x346ED9FC, 0xAD678846, 0xDA60B8D0, 0x44042D73, 0x33031DE5, 0xAA0A4C5F, 0xDD0D7CC9,
+		0x5005713C, 0x270241AA, 0xBE0B1010, 0xC90C2086, 0x5768B525, 0x206F85B3, 0xB966D409, 0xCE61E49F,
+		0x5EDEF90E, 0x29D9C998, 0xB0D09822, 0xC7D7A8B4, 0x59B33D17, 0x2EB40D81, 0xB7BD5C3B, 0xC0BA6CAD,
+		0xEDB88320, 0x9ABFB3B6, 0x03B6E20C, 0x74B1D29A, 0xEAD54739, 0x9DD277AF, 0x04DB2615, 0x73DC1683,
+		0xE3630B12, 0x94643B84, 0x0D6D6A3E, 0x7A6A5AA8, 0xE40ECF0B, 0x9309FF9D, 0x0A00AE27, 0x7D079EB1,
+		0xF00F9344, 0x8708A3D2, 0x1E01F268, 0x6906C2FE, 0xF762575D, 0x806567CB, 0x196C3671, 0x6E6B06E7,
+		0xFED41B76, 0x89D32BE0, 0x10DA7A5A, 0x67DD4ACC, 0xF9B9DF6F, 0x8EBEEFF9, 0x17B7BE43, 0x60B08ED5,
+		0xD6D6A3E8, 0xA1D1937E, 0x38D8C2C4, 0x4FDFF252, 0xD1BB67F1, 0xA6BC5767, 0x3FB506DD, 0x48B2364B,
+		0xD80D2BDA, 0xAF0A1B4C, 0x36034AF6, 0x41047A60, 0xDF60EFC3, 0xA867DF55, 0x316E8EEF, 0x4669BE79,
+		0xCB61B38C, 0xBC66831A, 0x256FD2A0, 0x5268E236, 0xCC0C7795, 0xBB0B4703, 0x220216B9, 0x5505262F,
+		0xC5BA3BBE, 0xB2BD0B28, 0x2BB45A92, 0x5CB36A04, 0xC2D7FFA7, 0xB5D0CF31, 0x2CD99E8B, 0x5BDEAE1D,
+		0x9B64C2B0, 0xEC63F226, 0x756AA39C, 0x026D930A, 0x9C0906A9, 0xEB0E363F, 0x72076785, 0x05005713,
+		0x95BF4A82, 0xE2B87A14, 0x7BB12BAE, 0x0CB61B38, 0x92D28E9B, 0xE5D5BE0D, 0x7CDCEFB7, 0x0BDBDF21,
+		0x86D3D2D4, 0xF1D4E242, 0x68DDB3F8, 0x1FDA836E, 0x81BE16CD, 0xF6B9265B, 0x6FB077E1, 0x18B74777,
+		0x88085AE6, 0xFF0F6A70, 0x66063BCA, 0x11010B5C, 0x8F659EFF, 0xF862AE69, 0x616BFFD3, 0x166CCF45,
+		0xA00AE278, 0xD70DD2EE, 0x4E048354, 0x3903B3C2, 0xA7672661, 0xD06016F7, 0x4969474D, 0x3E6E77DB,
+		0xAED16A4A, 0xD9D65ADC, 0x40DF0B66, 0x37D83BF0, 0xA9BCAE53, 0xDEBB9EC5, 0x47B2CF7F, 0x30B5FFE9,
+		0xBDBDF21C, 0xCABAC28A, 0x53B39330, 0x24B4A3A6, 0xBAD03605, 0xCDD70693, 0x54DE5729, 0x23D967BF,
+		0xB3667A2E, 0xC4614AB8, 0x5D681B02, 0x2A6F2B94, 0xB40BBE37, 0xC30C8EA1, 0x5A05DF1B, 0x2D02EF8D,
+	},
+	{
+		0x00000000, 0x191B3141, 0x32366282, 0x2B2D53C3, 0x646CC504, 0x7D77F445, 0x565AA786, 0x4F4196C7,
+		0xC8D98A08, 0xD1C2BB49, 0xFAEFE88A, 0xE3F4D9CB, 0xACB54F0C, 0xB5AE7E4D, 0x9E832D8E, 0x87981CCF,
+		0x4AC21251, 0x53D92310, 0x78F470D3, 0x61EF4192, 0x2EAED755, 0x37B5E614, 0x1C98B5D7, 0x05838496,
+		0x821B9859, 0x9B00A918, 0xB02DFADB, 0xA936CB9A, 0xE6775D5D, 0xFF6C6C1C, 0xD4413FDF, 0xCD5A0E9E,
+		0x958424A2, 0x8C9F15E3, 0xA7B24620, 0xBEA97761, 0xF1E8E1A6, 0xE8F3D0E7, 0xC3DE8324, 0xDAC5B265,
+		0x5D5DAEAA, 0x44469FEB, 0x6F6BCC28, 0x7670FD69, 0x39316BAE, 0x202A5AEF, 0x0B07092C, 0x121C386D,
+		0xDF4636F3, 0xC65D07B2, 0xED705471, 0xF46B6530, 0xBB2AF3F7, 0xA231C2B6, 0x891C9175, 0x9007A034,
+		0x179FBCFB, 0x0E848DBA, 0x25A9DE79, 0x3CB2EF38, 0x73F379FF, 0x6AE848BE, 0x41C51B7D, 0x58DE2A3C,
+		0xF0794F05, 0xE9627E44, 0xC24F2D87, 0xDB541CC6, 0x94158A01, 0x8D0EBB40, 0xA623E883, 0xBF38D9C2,
+		0x38A0C50D, 0x21BBF44C, 0x0A96A78F, 0x138D96CE, 0x5CCC0009, 0x45D73148, 0x6EFA628B, 0x77E153CA,
+		0xBABB5D54, 0xA3A06C15, 0x888D3FD6, 0x91960E97, 0xDED79850, 0xC7CCA911, 0xECE1FAD2, 0xF5FACB93,
+		0x7262D75C, 0x6B79E61D, 0x4054B5DE, 0x594F849F, 0x160E1258, 0x0F152319, 0x243870DA, 0x3D23419B,
+		0x65FD6BA7, 0x7CE65AE6, 0x57CB0925, 0x4ED03864, 0x0191AEA3, 0x188A9FE2, 0x33A7CC21, 0x2ABCFD60,
+		0xAD24E1AF, 0xB43FD0EE, 0x9F12832D, 0x8609B26C, 0xC94824AB, 0xD05315EA, 0xFB7E4629, 0xE2657768,
+		0x2F3F79F6, 0x362448B7, 0x1D091B74, 0x04122A35, 0x4B53BCF2, 0x52488DB3, 0x7965DE70, 0x607EEF31,
+		0xE7E6F3FE, 0xFEFDC2BF, 0xD5D0917C, 0xCCCBA03D, 0x838A36FA, 0x9A9107BB, 0xB1BC5478, 0xA8A76539,
+		0x3B83984B, 0x2298A90A, 0x09B5FAC9, 0x10AECB88, 0x5FEF5D4F, 0x46F46C0E, 0x6DD93FCD, 0x74C20E8C,
+		0xF35A1243, 0xEA412302, 0xC16C70C1, 0xD8774180, 0x9736D747, 0x8E2DE606, 0xA500B5C5, 0xBC1B8484,
+		0x71418A1A, 0x685ABB5B, 0x4377E898, 0x5A6CD9D9, 0x152D4F1E, 0x0C367E5F, 0x271B2D9C, 0x3E001CDD,
+		0xB9980012, 0xA0833153, 0x8BAE6290, 0x92B553D1, 0xDDF4C516, 0xC4EFF457, 0xEFC2A794, 0xF6D996D5,
+		0xAE07BCE9, 0xB71C8DA8, 0x9C31DE6B, 0x852AEF2A, 0xCA6B79ED, 0xD37048AC, 0xF85D1B6F, 0xE1462A2E,
+		0x66DE36E1, 0x7FC507A0, 0x54E85463, 0x4DF36522, 0x02B2F3E5, 0x1BA9C2A4, 0x30849167, 0x299FA026,
+		0xE4C5AEB8, 0xFDDE9FF9, 0xD6F3CC3A, 0xCFE8FD7B, 0x80A96BBC, 0x99B25AFD, 0xB29F093E, 0xAB84387F,
+		0x2C1C24B0, 0x350715F1, 0x1E2A4632, 0x07317773, 0x4870E1B4, 0x516BD0F5, 0x7A468336, 0x635DB277,
+		0xCBFAD74E, 0xD2E1E60F, 0xF9CCB5CC, 0xE0D7848D, 0xAF96124A, 0xB68D230B, 0x9DA070C8, 0x84BB4189,
+		0x03235D46, 0x1A386C07, 0x31153FC4, 0x280E0E85, 0x674F9842, 0x7E54A903, 0x5579FAC0, 0x4C62CB81,
+		0x8138C51F, 0x9823F45E, 0xB30EA79D, 0xAA1596DC, 0xE554001B, 0xFC4F315A, 0xD7626299, 0xCE7953D8,
+		0x49E14F17, 0x50FA7E56, 0x7BD72D95, 0x62CC1CD4, 0x2D8D8A13, 0x3496BB52, 0x1FBBE891, 0x06A0D9D0,
+		0x5E7EF3EC, 0x4765C2AD, 0x6C48916E, 0x7553A02F, 0x3A1236E8, 0x230907A9, 0x0824546A, 0x113F652B,
+		0x96A779E4, 0x8FBC48A5, 0xA4911B66, 0xBD8A2A27, 0xF2CBBCE0, 0xEBD08DA1, 0xC0FDDE62, 0xD9E6EF23,
+		0x14BCE1BD, 0x0DA7D0FC, 0x268A833F, 0x3F91B27E, 0x70D024B9, 0x69CB15F8, 0x42E6463B, 0x5BFD777A,
+		0xDC656BB5, 0xC57E5AF4, 0xEE530937, 0xF7483876, 0xB809AEB1, 0xA1129FF0, 0x8A3FCC33, 0x9324FD72,
+	},
+	{
+		0x00000000, 0x01C26A37, 0x0384D46E, 0x0246BE59, 0x0709A8DC, 0x06CBC2EB, 0x048D7CB2, 0x054F1685,
+		0x0E1351B8, 0x0FD13B8F, 0x0D9785D6, 0x0C55EFE1, 0x091AF964, 0x08D89353, 0x0A9E2D0A, 0x0B5C473D,
+		0x1C26A370, 0x1DE4C947, 0x1FA2771E, 0x1E601D29, 0x1B2F0BAC, 0x1AED619B, 0x18ABDFC2, 0x1969B5F5,
+		0x1235F2C8, 0x13F798FF, 0x11B126A6, 0x10734C91, 0x153C5A14, 0x14FE3023, 0x16B88E7A, 0x177AE44D,
+		0x384D46E0, 0x398F2CD7, 0x3BC9928E, 0x3A0BF8B9, 0x3F44EE3C, 0x3E86840B, 0x3CC03A52, 0x3D025065,
+		0x365E1758, 0x379C7D6F, 0x35DAC336, 0x3418A901, 0x3157BF84, 0x3095D5B3, 0x32D36BEA, 0x331101DD,
+		0x246BE590, 0x25A98FA7, 0x27EF31FE, 0x262D5BC9, 0x23624D4C, 0x22A0277B, 0x20E69922, 0x2124F315,
+		0x2A78B428, 0x2BBADE1F, 0x29FC6046, 0x283E0A71, 0x2D711CF4, 0x2CB376C3, 0x2EF5C89A, 0x2F37A2AD,
+		0x709A8DC0, 0x7158E7F7, 0x731E59AE, 0x72DC3399, 0x7793251C, 0x76514F2B, 0x7417F172, 0x75D59B45,
+		0x7E89DC78, 0x7F4BB64F, 0x7D0D0816, 0x7CCF6221, 0x798074A4, 0x78421E93, 0x7A04A0CA, 0x7BC6CAFD,
+		0x6CBC2EB0, 0x6D7E4487, 0x6F38FADE, 0x6EFA90E9, 0x6BB5866C, 0x6A77EC5B, 0x68315202, 0x69F33835,
+		0x62AF7F08, 0x636D153F, 0x612BAB66, 0x60E9C151, 0x65A6D7D4, 0x6464BDE3, 0x662203BA, 0x67E0698D,
+		0x48D7CB20, 0x4915A117, 0x4B531F4E, 0x4A917579, 0x4FDE63FC, 0x4E1C09CB, 0x4C5AB792, 0x4D98DDA5,
+		0x46C49A98, 0x4706F0AF, 0x45404EF6, 0x448224C1, 0x41CD3244, 0x400F5873, 0x4249E62A, 0x438B8C1D,
+		0x54F16850, 0x55330267, 0x5775BC3E, 0x56B7D609, 0x53F8C08C, 0x523AAABB, 0x507C14E2, 0x51BE7ED5,
+		0x5AE239E8, 0x5B2053DF, 0x5966ED86, 0x58A487B1, 0x5DEB9134, 0x5C29FB03, 0x5E6F455A, 0x5FAD2F6D,
+		0xE1351B80, 0xE0F771B7, 0xE2B1CFEE, 0xE373A5D9, 0xE63CB35C, 0xE7FED96B, 0xE5B86732, 0xE47A0D05,
+		0xEF264A38, 0xEEE4200F, 0xECA29E56, 0xED60F461, 0xE82FE2E4, 0xE9ED88D3, 0xEBAB368A, 0xEA695CBD,
+		0xFD13B8F0, 0xFCD1D2C7, 0xFE976C9E, 0xFF5506A9, 0xFA1A102C, 0xFBD87A1B, 0xF99EC442, 0xF85CAE75,
+		0xF300E948, 0xF2C2837F, 0xF0843D26, 0xF1465711, 0xF4094194, 0xF5CB2BA3, 0xF78D95FA, 0xF64FFFCD,
+		0xD9785D60, 0xD8BA3757, 0xDAFC890E, 0xDB3EE339, 0xDE71F5BC, 0xDFB39F8B, 0xDDF521D2, 0xDC374BE5,
+		0xD76B0CD8, 0xD6A966EF, 0xD4EFD8B6, 0xD52DB281, 0xD062A404, 0xD1A0CE33, 0xD3E6706A, 0xD2241A5D,
+		0xC55EFE10, 0xC49C9427, 0xC6DA2A7E, 0xC7184049, 0xC25756CC, 0xC3953CFB, 0xC1D382A2, 0xC011E895,
+		0xCB4DAFA8, 0xCA8FC59F, 0xC8C97BC6, 0xC90B11F1, 0xCC440774, 0xCD866D43, 0xCFC0D31A, 0xCE02B92D,
+		0x91AF9640, 0x906DFC77, 0x922B422E, 0x93E92819, 0x96A63E9C, 0x976454AB, 0x9522EAF2, 0x94E080C5,
+		0x9FBCC7F8, 0x9E7EADCF, 0x9C381396, 0x9DFA79A1, 0x98B56F24, 0x99770513, 0x9B31BB4A, 0x9AF3D17D,
+		0x8D893530, 0x8C4B5F07, 0x8E0DE15E, 0x8FCF8B69, 0x8A809DEC, 0x8B42F7DB, 0x89044982, 0x88C623B5,
+		0x839A6488, 0x82580EBF, 0x801EB0E6, 0x81DCDAD1, 0x8493CC54, 0x8551A663, 0x8717183A, 0x86D5720D,
+		0xA9E2D0A0, 0xA820BA97, 0xAA6604CE, 0xABA46EF9, 0xAEEB787C, 0xAF29124B, 0xAD6FAC12, 0xACADC625,
+		0xA7F18118, 0xA633EB2F, 0xA4755576, 0xA5B73F41, 0xA0F829C4, 0xA13A43F3, 0xA37CFDAA, 0xA2BE979D,
+		0xB5C473D0, 0xB40619E7, 0xB640A7BE, 0xB782CD89, 0xB2CDDB0C, 0xB30FB13B, 0xB1490F62, 0xB08B6555,
+		0xBBD72268, 0xBA15485F, 0xB853F606, 0xB9919C31, 0xBCDE8AB4, 0xBD1CE083, 0xBF5A5EDA, 0xBE9834ED,
+	},
+	{
+		0x00000000, 0xB8BC6765, 0xAA09C88B, 0x12B5AFEE, 0x8F629757, 0x37DEF032, 0x256B5FDC, 0x9DD738B9,
+		0xC5B428EF, 0x7D084F8A, 0x6FBDE064, 0xD7018701, 0x4AD6BFB8, 0xF26AD8DD, 0xE0DF7733, 0x58631056,
+		0x5019579F, 0xE8A530FA, 0xFA109F14, 0x42ACF871, 0xDF7BC0C8, 0x67C7A7AD, 0x75720843, 0xCDCE6F26,
+		0x95AD7F70, 0x2D111815, 0x3FA4B7FB, 0x8718D09E, 0x1ACFE827, 0xA2738F42, 0xB0C620AC, 0x087A47C9,
+		0xA032AF3E, 0x188EC85B, 0x0A3B67B5, 0xB28700D0, 0x2F503869, 0x97EC5F0C, 0x8559F0E2, 0x3DE59787,
+		0x658687D1, 0xDD3AE0B4, 0xCF8F4F5A, 0x7733283F, 0xEAE41086, 0x525877E3, 0x40EDD80D, 0xF851BF68,
+		0xF02BF8A1, 0x48979FC4, 0x5A22302A, 0xE29E574F, 0x7F496FF6, 0xC7F50893, 0xD540A77D, 0x6DFCC018,
+		0x359FD04E, 0x8D23B72B, 0x9F9618C5, 0x272A7FA0, 0xBAFD4719, 0x0241207C, 0x10F48F92, 0xA848E8F7,
+		0x9B14583D, 0x23A83F58, 0x311D90B6, 0x89A1F7D3, 0x1476CF6A, 0xACCAA80F, 0xBE7F07E1, 0x06C36084,
+		0x5EA070D2, 0xE61C17B7, 0xF4A9B859, 0x4C15DF3C, 0xD1C2E785, 0x697E80E0, 0x7BCB2F0E, 0xC377486B,
+		0xCB0D0FA2, 0x73B168C7, 0x6104C729, 0xD9B8A04C, 0x446F98F5, 0xFCD3FF90, 0xEE66507E, 0x56DA371B,
+		0x0EB9274D, 0xB6054028, 0xA4B0EFC6, 0x1C0C88A3, 0x81DBB01A, 0x3967D77F, 0x2BD27891, 0x936E1FF4,
+		0x3B26F703, 0x839A9066, 0x912F3F88, 0x299358ED, 0xB4446054, 0x0CF80731, 0x1E4DA8DF, 0xA6F1CFBA,
+		0xFE92DFEC, 0x462EB889, 0x549B1767, 0xEC277002, 0x71F048BB, 0xC94C2FDE, 0xDBF98030, 0x6345E755,
+		0x6B3FA09C, 0xD383C7F9, 0xC1366817, 0x798A0F72, 0xE45D37CB, 0x5CE150AE, 0x4E54FF40, 0xF6E89825,
+		0xAE8B8873, 0x1637EF16, 0x048240F8, 0xBC3E279D, 0x21E91F24, 0x99557841, 0x8BE0D7AF, 0x335CB0CA,
+		0xED59B63B, 0x55E5D15E, 0x47507EB0, 0xFFEC19D5, 0x623B216C, 0xDA874609, 0xC832E9E7, 0x708E8E82,
+		0x28ED9ED4, 0x9051F9B1, 0x82E4565F, 0x3A58313A, 0xA78F0983, 0x1F336EE6, 0x0D86C108, 0xB53AA66D,
+		0xBD40E1A4, 0x05FC86C1, 0x1749292F, 0xAFF54E4A, 0x322276F3, 0x8A9E1196, 0x982BBE78, 0x2097D91D,
+		0x78F4C94B, 0xC048AE2E, 0xD2FD01C0, 0x6A4166A5, 0xF7965E1C, 0x4F2A3979, 0x5D9F9697, 0xE523F1F2,
+		0x4D6B1905, 0xF5D77E60, 0xE762D18E, 0x5FDEB6EB, 0xC2098E52, 0x7AB5E937, 0x680046D9, 0xD0BC21BC,
+		0x88DF31EA, 0x3063568F, 0x22D6F961, 0x9A6A9E04, 0x07BDA6BD, 0xBF01C1D8, 0xADB46E36, 0x15080953,
+		0x1D724E9A, 0xA5CE29FF, 0xB77B8611, 0x0FC7E174, 0x9210D9CD, 0x2AACBEA8, 0x38191146, 0x80A57623,
+		0xD8C66675, 0x607A0110, 0x72CFAEFE, 0xCA73C99B, 0x57A4F122, 0xEF189647, 0xFDAD39A9, 0x45115ECC,
+		0x764DEE06, 0xCEF18963, 0xDC44268D, 0x64F841E8, 0xF92F7951, 0x41931E34, 0x5326B1DA, 0xEB9AD6BF,
+		0xB3F9C6E9, 0x0B45A18C, 0x19F00E62, 0xA14C6907, 0x3C9B51BE, 0x842736DB, 0x96929935, 0x2E2EFE50,
+		0x2654B999, 0x9EE8DEFC, 0x8C5D7112, 0x34E11677, 0xA9362ECE, 0x118A49AB, 0x033FE645, 0xBB838120,
+		0xE3E09176, 0x5B5CF613, 0x49E959FD, 0xF1553E98, 0x6C820621, 0xD43E6144, 0xC68BCEAA, 0x7E37A9CF,
+		0xD67F4138, 0x6EC3265D, 0x7C7689B3, 0xC4CAEED6, 0x591DD66F, 0xE1A1B10A, 0xF3141EE4, 0x4BA87981,
+		0x13CB69D7, 0xAB770EB2, 0xB9C2A15C, 0x017EC639, 0x9CA9FE80, 0x241599E5, 0x36A0360B, 0x8E1C516E,
+		0x866616A7, 0x3EDA71C2, 0x2C6FDE2C, 0x94D3B949, 0x090481F0, 0xB1B8E695, 0xA30D497B, 0x1BB12E1E,
+		0x43D23E48, 0xFB6E592D, 0xE9DBF6C3, 0x516791A6, 0xCCB0A91F, 0x740CCE7A, 0x66B96194, 0xDE0506F1,
+	}
+};
+
+void crc_slice_init(crc_state *const s)
+{
+	s->crc0[0] = ~0;
+}
+
+// compute CRC32 (Slicing-by-4 algorithm)
+void crc_slice(crc_state *const s, const unsigned char *src, long len)
+{
+	uint32_t crc = s->crc0[0];
+
+#if __BYTE_ORDER == __LITTLE_ENDIAN
+	// process first 1-3 bytes until input is aligned to 4 bytes
+	if ((uintptr_t)src & 3 && len >= 4)
+	{
+		int unaligned = 4 - (uintptr_t)src % 4;
+		while (unaligned-- != 0)
+		{
+			crc = ((crc >> 8) & 0x00FFFFFF) ^ crc32_tab[0][(crc ^ *src++) & 0xFF];
+			len--;
+		}
+	}
+
+	const uint32_t* current = (const uint32_t*)src;
+
+	// process four bytes at once (Slicing-by-4)
+	while (len >= 4)
+	{
+		uint32_t one = *current++ ^ crc;
+		crc = crc32_tab[0][(one >> 24) & 0xFF] ^
+			crc32_tab[1][(one >> 16) & 0xFF] ^
+			crc32_tab[2][(one >> 8) & 0xFF] ^
+			crc32_tab[3][one & 0xFF];
+
+		len -= 4;
+	}
+
+	// remaining 1 to 3 bytes (standard algorithm)
+	src = (uchar*)current;
+#endif
+
+	while (len-- != 0)
+	{
+		crc = ((crc >> 8) & 0x00FFFFFF) ^ crc32_tab[0][(crc ^ *src++) & 0xFF];
+	}
+
+	s->crc0[0] = crc;
+}
+
+uint32_t crc_slice_finish(crc_state *const s)
+{
+	return ~s->crc0[0];
+}
+
+void init_crc_slice()
+{
+	crc_init = &crc_slice_init;
+	crc_incr = &crc_slice;
+	crc_finish = &crc_slice_finish;
+}
+
+}

lib/yencode/Sse2Decoder.cpp

@@ -0,0 +1,50 @@
+/*
+ *  Based on node-yencode library by Anime Tosho:
+ *  https://github.com/animetosho/node-yencode
+ *
+ *  Copyright (C) 2017 Anime Tosho (animetosho)
+ *  Copyright (C) 2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+
+#include "nzbget.h"
+
+#include "YEncode.h"
+
+#ifdef __SSE2__
+#include <immintrin.h>
+#endif
+
+namespace YEncode
+{
+
+namespace Sse2
+{
+#ifdef __SSE2__
+#define SIMD_DECODER
+#include "SimdDecoder.cpp"
+#endif
+}
+
+void init_decode_sse2() {
+#ifdef __SSE2__
+	decode = &YEncode::Sse2::do_decode_simd<sizeof(__m128i), YEncode::Sse2::do_decode_sse<false>>;
+	YEncode::Sse2::decoder_init();
+	decode_simd = true;
+#endif
+}
+
+}

lib/yencode/Ssse3Decoder.cpp

@@ -0,0 +1,50 @@
+/*
+ *  Based on node-yencode library by Anime Tosho:
+ *  https://github.com/animetosho/node-yencode
+ *
+ *  Copyright (C) 2017 Anime Tosho (animetosho)
+ *  Copyright (C) 2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+
+#include "nzbget.h"
+
+#include "YEncode.h"
+
+#ifdef __SSSE3__
+#include <immintrin.h>
+#endif
+
+namespace YEncode
+{
+
+namespace Ssse3
+{
+#ifdef __SSSE3__
+#define SIMD_DECODER
+#include "SimdDecoder.cpp"
+#endif
+}
+
+void init_decode_ssse3() {
+#ifdef __SSSE3__
+	decode = &YEncode::Ssse3::do_decode_simd<sizeof(__m128i), YEncode::Ssse3::do_decode_sse<true>>;
+	YEncode::Ssse3::decoder_init();
+	decode_simd = true;
+#endif
+}
+
+}

lib/yencode/YEncode.h

@@ -0,0 +1,61 @@
+/*
+ *  Based on node-yencode library by Anime Tosho:
+ *  https://github.com/animetosho/node-yencode
+ *
+ *  Copyright (C) 2017 Anime Tosho (animetosho)
+ *  Copyright (C) 2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
+ *
+ *  This program is free software; you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation; either version 2 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+
+#ifndef YENCODE_H
+#define YENCODE_H
+
+namespace YEncode
+{
+
+void init();
+
+typedef enum : char {
+	YDEC_STATE_CRLF, // default
+	YDEC_STATE_EQ,
+	YDEC_STATE_CR,
+	YDEC_STATE_NONE,
+	YDEC_STATE_CRLFDT,
+	YDEC_STATE_CRLFDTCR,
+	YDEC_STATE_CRLFEQ // may actually be "\r\n.=" in raw decoder
+} YencDecoderState;
+
+extern int (*decode)(const unsigned char** src, unsigned char** dest, size_t len, YencDecoderState* state);
+extern int decode_scalar(const unsigned char** src, unsigned char** dest, size_t len, YencDecoderState* state);
+extern bool decode_simd;
+
+struct crc_state
+{
+#if defined(__i686__) || defined(__amd64__)
+	alignas(16) uint32_t crc0[4 * 5];
+#else
+	uint32_t crc0[1];
+#endif
+};
+
+extern void (*crc_init)(crc_state *const s);
+extern void (*crc_incr)(crc_state *const s, const unsigned char *src, long len);
+extern uint32_t (*crc_finish)(crc_state *const s);
+extern bool crc_simd;
+
+}
+
+#endif

linux/build-info.txt

@@ -30,15 +30,17 @@ Building toolchains
     output    - to store the results of build script;
     setup     - for extra files required for installer;
 
-4. Build buildroot-toolchain for one CPU-architecture (see below);
-	4.1. Download Buildroot distribution archive from http://buildroot.uclibc.org/download.html
-	     (currently using version "buildroot-2015.08.01");
-	4.2. Unpack the tarball into 'toolchain'-directory;
-	4.3. Rename the buildroot-directory according to the target CPU-architecture name;
-	     Be careful here, after the toolchain is built the directory cannot be renamed
-	     or moved, you will have to rebuild the toolchain if you want another name;
-	4.4. Run 'make nconfig';
-	4.5. Configure toolchain:
+4. Build buildroot-toolchain for one CPU-architecture:
+    4.1. Download Buildroot distribution archive 'buildroot-2015.08.01' from
+         http://buildroot.uclibc.org/download.html.
+         If you choose other version the instructions may not apply exactly;
+    4.2. Unpack the tarball into 'toolchain'-directory;
+    4.3. Rename the buildroot-directory according to the target CPU-architecture name,
+         for example 'x86_64' or 'armhf'; Be careful here, after the toolchain is
+         built the directory cannot be renamed or moved, you will have to rebuild
+         the toolchain if you want another name;
+    4.4. Run 'make menuconfig';
+    4.5. Configure toolchain:
          - Target architecture:
              - choose your target architecture;
          - Build options:
@@ -47,7 +49,8 @@ Building toolchains
              - Kernel Headers (Manually specified Linux version);
              - (2.6.32) linux version;
              - Custom kernel headers series (2.6.x);
-             - uClibc C library Version (uClibc-ng);
+             - C library: glibc (for x86_64 and armhf) or uClibc (for all other architectures);
+             - uClibc library version (uClibc-ng);
              - Enable large file (files > 2 GB) support;
              - Enable IPv6 support;
              - Enable toolchain locale/i18n support;
@@ -61,15 +64,44 @@ Building toolchains
                   - JSON/XML: libxml2;
                   - Text and terminal handling: ncurses;
          - Save configuration and exit;
-	4.6. Do few extra manual adjustments:
-         - in the file '.config' in the buildroot directory activate define to
-           build 'ubacktrace';
+    4.6. Do few extra manual adjustments:
          - in 'packages/ncurses/ncurses.mk' add extra configure parameters to
-           option 'NCURSES_CONF_OPTS‘ (without quotation marks):
+           option 'NCURSES_CONF_OPTS‘ (with quotation marks):
            "--with-fallbacks=xterm xterm-color xterm-256color xterm-16color linux vt100 vt200";
          - in 'packages/openssl/openssl.mk' replace 'zlib-dynamic' with 'zlib';
-	4.7. Run 'make' to build the toolchain. It may take an hour or so depending
-	     on your hardware;
+         - in 'packages/glibc/glibc.mk' add extra configure parameter
+           '--enable-static-nss';
+         - in directory 'packages/glibc/2.20' create new file with name 'allocatestack-pax.patch'
+           with the following content; the patch fixes incompatibility with hardened kernels:
+>>> START OF 'allocatestack-pax.patch' - this line marker is not part of the file >>>
+--- a/nptl/allocatestack.c	2014-09-07 10:09:09.000000000 +0200
++++ b/nptl/allocatestack.c	2017-11-02 18:02:47.165016000 +0100
+@@ -78,6 +78,7 @@
+ #ifndef STACK_ALIGN
+ # define STACK_ALIGN __alignof__ (long double)
+ #endif
++#define PF_X 0
+ 
+ /* Default value for minimal stack size after allocating thread
+    descriptor and guard.  */
+<<< END OF 'allocatestack-pax.patch' - this line marker is not part of the file <<<
+    4.7. Run 'make' to build the toolchain. It may take an hour or less depending
+         on your hardware;
+    4.8. If C library 'glibc' was chosen extra steps are needed to build proper
+         static libraries of OpenSSL in addition to already built dynamic
+         libraries (which are also required):
+         - delete directory 'output/build/openssl-x.x.x';
+         - in 'packages/openssl/openssl.mk' replace 
+           '$(if $(BR2_STATIC_LIBS),no-shared,shared)' with
+           '$(if $(BR2_STATIC_LIBS),no-shared,no-shared)'
+           and replace '$(if $(BR2_STATIC_LIBS),no-dso,dso)' with
+           '$(if $(BR2_STATIC_LIBS),no-dso,no-dso)';
+         - from root directory of toolchain run 'make' again;
+         - openssl will be rebuilt without shared libraries support;
+         - installation step of openssl fails with message:
+           "chmod: cannot access ‘../lib/engines/lib*.so’: No such file or directory";
+         - ignore the error, static libraries of openssl are already installed
+           and will be used by NZBGet;
 
 5. Now you should have a working toolchain for one CPU-architecture, let's
    test it.

linux/build-nzbget

@@ -29,6 +29,7 @@ ALLTARGETS="dist i686 x86_64 armel armhf mipsel mipseb ppc6xx ppc500 x86_64-bsd"
 ROOT=`pwd`
 ROOTPATH=$PATH
 OUTPUTDIR=$ROOT/output
+CROSSCLANG="clang-4.0"
 
 # Configuration
 BUILD=no
@@ -246,15 +247,9 @@ ConstructArchParams()
         mipseb)
             ARCH=mips
             ;;
-        armhf)
-            ARCH=arm
-            ;;
         arm*)
             ARCH=arm
             ;;
-        ppc500)
-            ARCH=powerpc
-            ;;
         ppc*)
             ARCH=powerpc
             ;;
@@ -263,15 +258,16 @@ ConstructArchParams()
     TOOLCHAIN_ROOT=$ROOT/toolchain/$TARGET$PLATSUFF
     TOOLPATH=$TOOLCHAIN_ROOT/output/host/usr/bin
     PATH=$TOOLPATH:$ROOTPATH
-    STAGING="$TOOLCHAIN_ROOT/output/staging/usr"
+    STAGING="$TOOLCHAIN_ROOT/output/staging"
     case $PLATFORM in
         linux)
             TOOLKIND=buildroot
             TOOLNAME=linux
         ;;
         freebsd)
-            TOOLKIND=crossgcc
-            TOOLNAME=pc-freebsd9
+            TOOLKIND=crossclang
+            TOOLNAME=pc-freebsd
+           	SYSROOT="$TOOLCHAIN_ROOT/sysroot"
         ;;
     esac
 }
@@ -288,11 +284,16 @@ ConfigureTarget()
         DEBUG="-g"
     fi
 
+    LUBACKTRACE=""
+    if [ -f "$STAGING/lib/libubacktrace.so.1" ] ; then
+	    LUBACKTRACE="-lubacktrace"
+    fi
+
     case "$TOOLKIND-$CONFIG" in
         buildroot-debug|buildroot-debug-strip)
             LDFLAGS="-static $STRIP" \
                 CXXFLAGS="-std=c++14 -g -fasynchronous-unwind-tables" \
-                LIBS="-lcrypto -ldl -lz -lubacktrace" \
+                LIBS="-lcrypto -ldl -lz $LUBACKTRACE" \
                 ./configure --disable-dependency-tracking --host=$ARCH-$TOOLNAME --enable-debug
             ;;
         buildroot-release|buildroot-release-nostrip)
@@ -301,17 +302,19 @@ ConfigureTarget()
                 LIBS="-lcrypto -ldl -lz" \
                 ./configure --disable-dependency-tracking --host=$ARCH-$TOOLNAME
             ;;
-        crossgcc-debug|crossgcc-debug-strip)
-            LDFLAGS="-static $STRIP" \
-                CXXFLAGS="-std=c++14 -g -fasynchronous-unwind-tables -I$STAGING/include" \
-                PKG_CONFIG_LIBDIR="$STAGING/lib/pkgconfig" \
+        crossclang-debug|crossclang-debug-strip)
+            CXX="$CROSSCLANG" \
+                LDFLAGS="-static $STRIP -fuse-ld=lld --target=$ARCH-$TOOLNAME -lc++ -lm --sysroot=$SYSROOT" \
+                CXXFLAGS="-g --target=$ARCH-$TOOLNAME --sysroot=$SYSROOT -I$SYSROOT/usr/include/c++/v1" \
+                PKG_CONFIG_LIBDIR="$STAGING/usr/lib/pkgconfig" \
                 ./configure --disable-dependency-tracking --host=$ARCH-$TOOLNAME --enable-debug
                 ;;
-        crossgcc-release|crossgcc-release-nostrip)
-            LDFLAGS="-static $STRIP" \
-                CXXFLAGS="-std=c++14 -O2 $DEBUG -I$STAGING/include" \
-                PKG_CONFIG_LIBDIR="$STAGING/lib/pkgconfig" \
-                ./configure --disable-dependency-tracking --host=$ARCH-$TOOLNAME
+        crossclang-release|crossclang-release-nostrip)
+            CXX="$CROSSCLANG" \
+                LDFLAGS="-static $STRIP -fuse-ld=lld --target=$ARCH-$TOOLNAME -lc++ -lm --sysroot=$SYSROOT" \
+                CXXFLAGS="-O2 $DEBUG --target=$ARCH-$TOOLNAME --sysroot=$SYSROOT -I$SYSROOT/usr/include/c++/v1" \
+                PKG_CONFIG_LIBDIR="$STAGING/usr/lib/pkgconfig" \
+                ./configure --disable-dependency-tracking --host=$ARCH-$TOOLNAME --enable-debug
                 ;;
     esac
 }
@@ -319,14 +322,14 @@ ConfigureTarget()
 PrecompileHeaders()
 {
     rm -f nzbget.h.gch
-    if [ $PCH == "yes" ]; then
+    if [ $PCH == "yes" -a $TOOLKIND != "crossclang" ]; then
         OPTIM=""
         if [ $CONFIG == "release" -o $CONFIG == "release-nostrip" ]; then
             OPTIM="-O2"
         fi
 
         $ARCH-$TOOLNAME-g++ -std=c++14 -DHAVE_CONFIG_H \
-            -I. -I$STAGING/include -I$STAGING/include/libxml2 \
+            -I. -I$STAGING/usr/include -I$STAGING/usr/include/libxml2 \
             -g $OPTIM daemon/main/nzbget.h -o nzbget.h.gch
     fi
 }

linux/build-toolchain-freebsd

@@ -2,7 +2,7 @@
 #
 #  This file is part of nzbget. See <http://nzbget.net>.
 #
-#  Copyright (C) 2015-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+#  Copyright (C) 2015-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
 #
 #  This program is free software; you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -29,20 +29,13 @@ set -o errexit
 #set -x
 
 
-GNU_TARGET="x86_64-pc-freebsd9"
+TARGET="x86_64-pc-freebsd"
 
 # FreeBSD
-FREEBSDIMAGE_URL="ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/amd64/9.1-RELEASE/base.txz"
+FREEBSDIMAGE_URL="https://download.freebsd.org/ftp/releases/amd64/10.1-RELEASE/base.txz"
 
-# GCC
-GCC_VERSION="5.3.0"
-GCC_ARCHIVE="gcc-$GCC_VERSION.tar.bz2"
-GCC_URL="ftp://ftp.gnu.org/gnu/gcc/gcc-$GCC_VERSION/$GCC_ARCHIVE"
-
-# Binutils
-BINUTILS_VERSION="2.25"
-BINUTILS_ARCHIVE="binutils-$BINUTILS_VERSION.tar.bz2"
-BINUTILS_URL="ftp://ftp.gnu.org/gnu/binutils/$BINUTILS_ARCHIVE"
+# CLANG (must be already installed)
+CLANG="clang-4.0"
 
 # Libxml
 LIBXML2_VERSION="2.9.2"
@@ -60,6 +53,7 @@ OPENSSL_URL="ftp://ftp.openssl.org/source/old/1.0.2/$OPENSSL_ARCHIVE"
 ROOTDIR=`pwd`
 
 rm -rf output
+rm -rf sysroot
 
 # Download all required tools
 mkdir -p downloads
@@ -67,12 +61,6 @@ cd downloads
 if [ ! -f base.txz ]; then
     wget $FREEBSDIMAGE_URL
 fi
-if [ ! -f $GCC_ARCHIVE ]; then
-    wget $GCC_URL
-fi
-if [ ! -f $BINUTILS_ARCHIVE ]; then
-    wget $BINUTILS_URL
-fi
 if [ ! -f $LIBXML2_ARCHIVE ]; then
     wget $LIBXML2_URL
 fi
@@ -83,54 +71,50 @@ cd ..
 
 # Creating sysroot for FreeBSD from official FreeBSD installation image.
 # Our sysroot contains only a small set of files needed to compile NZBGet and dependent libraries
-mkdir -p output/host/usr/$GNU_TARGET/sysroot
-cd output/host/usr/$GNU_TARGET/sysroot
-tar xvJf ../../../../../downloads/base.txz \
+mkdir -p sysroot
+cd sysroot
+tar xf ../downloads/base.txz \
+	./lib \
     ./usr/include \
-    ./usr/lib/crt1.o ./usr/lib/crti.o ./usr/lib/crtn.o ./usr/lib/crtbeginT.o \
-    ./usr/lib/libc.a ./usr/lib/libm.a ./usr/lib/libz.a ./usr/lib/libncurses.a \
-    ./usr/lib/libthr.a
-ln -s libthr.a usr/lib/libpthread.a
-cd ../../../../..
+    ./usr/lib/crt1.o \
+    ./usr/lib/crtbegin.o \
+    ./usr/lib/crtbeginS.o \
+    ./usr/lib/crtbeginT.o \
+    ./usr/lib/crtend.o \
+    ./usr/lib/crtendS.o \
+    ./usr/lib/crti.o \
+    ./usr/lib/crtn.o \
+    ./usr/lib/libc++.a \
+    ./usr/lib/libc.a \
+    ./usr/lib/libcompiler_rt.a \
+    ./usr/lib/libgcc.a \
+    ./usr/lib/libgcc_eh.a \
+    ./usr/lib/libgcc_s.so \
+    ./usr/lib/libm.a \
+    ./usr/lib/libncurses.a \
+    ./usr/lib/libpthread.a \
+    ./usr/lib/libthr.a \
+    ./usr/lib/libz.a
+# fix bad symlink
+ln -s -f ../../lib/libgcc_s.so.1 usr/lib/libgcc_s.so
+cd ..
  
 mkdir -p output/build && cd output/build
 
 # Unpack everything
-tar xvjf ../../downloads/$BINUTILS_ARCHIVE
-tar xvjf ../../downloads/$GCC_ARCHIVE
-tar xvzf ../../downloads/$LIBXML2_ARCHIVE
-tar xvzf ../../downloads/$OPENSSL_ARCHIVE
-
-## Build binutils (5 minutes)
-mkdir binutils-build && cd binutils-build
-../binutils-$BINUTILS_VERSION/configure --with-gnu-as --with-gnu-ld --disable-libssp --disable-nls --disable-multilib --target=$GNU_TARGET --prefix=$ROOTDIR/output/host/usr --with-sysroot=$ROOTDIR/output/host/usr/$GNU_TARGET/sysroot
-make -j 2
-make install
-cd ..
-
-## Build GCC (30 minutes)
-cd gcc-$GCC_VERSION
-./contrib/download_prerequisites
-rm isl
-cd ..
-mkdir gcc-build && cd gcc-build
-../gcc-$GCC_VERSION/configure --with-gnu-as --with-gnu-ld --enable-languages=c,c++ --disable-bootstrap --without-headers --disable-nls --disable-libssp --disable-libgomp --disable-libmudflap --disable-multilib --disable-decimal-float --disable-libffi --disable-libmudflap --disable-libquadmath --disable-shared --disable-host-shared --disable-multilib --disable-libsanitizer --target=$GNU_TARGET --prefix=$ROOTDIR/output/host/usr --with-sysroot=$ROOTDIR/output/host/usr/$GNU_TARGET/sysroot
-make -j 2
-make install
-cd ..
-
-PATH=$ROOTDIR/output/host/usr/bin:$PATH
+tar xf ../../downloads/$LIBXML2_ARCHIVE
+tar xf ../../downloads/$OPENSSL_ARCHIVE
 
 # Build libxml2 (2 minutes)
 cd libxml2-$LIBXML2_VERSION
-./configure --host=$GNU_TARGET --prefix=$ROOTDIR/output/staging/usr --disable-shared --disable-dependency-tracking --without-zlib --without-lzma --without-python
+CC="$CLANG" CFLAGS="--target=$TARGET --sysroot=$ROOTDIR/sysroot" LDFLAGS="-fuse-ld=lld -static --target=$TARGET --sysroot=$ROOTDIR/sysroot" ./configure --host=$TARGET --prefix=$ROOTDIR/output/staging/usr --disable-dependency-tracking --without-zlib --without-lzma --without-python --disable-shared
 make -j2
 make install
 cd ..
 
 # Build OpenSSL (5 minutes)
 cd openssl-$OPENSSL_VERSION
-CC=$GNU_TARGET-gcc LD=$GNU_TARGET-ld AS=$GNU_TARGET-ad AR=$GNU_TARGET-ar ./Configure --prefix=$ROOTDIR/output/staging/usr no-shared no-dso no-hw no-zlib no-unit-test BSD-x86_64
+CC="$CLANG --target=$TARGET --sysroot=$ROOTDIR/sysroot" LD="$CLANG -fuse-ld=lld --target=$TARGET --sysroot=$ROOTDIR/sysroot" ./Configure --prefix=$ROOTDIR/output/staging/usr no-shared no-dso no-hw no-zlib no-unit-test BSD-x86_64
 make -j2
 make install_sw
 cd ..

linux/build-unpack

@@ -2,7 +2,7 @@
 #
 #  This file is part of nzbget. See <http://nzbget.net>.
 #
-#  Copyright (C) 2015-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+#  Copyright (C) 2015-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
 #
 #  This program is free software; you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -29,6 +29,7 @@ ALLTARGETS="i686 x86_64 armel armhf mipsel mipseb ppc6xx ppc500 x86_64-bsd"
 ROOT=`pwd`
 OUTPUTDIR=$ROOT/setup
 BUILDDIR=temp
+CROSSCLANG="clang-4.0"
 
 echo "Usage:"
 echo "  $(basename $0) [targets] [clean] [unpacker]"
@@ -109,15 +110,18 @@ for TARGET in $TARGETS; do
 
     if [[ $TARGET == *-bsd ]]; then
         TARGET="${TARGET%-bsd}"
-        TOOLNAME=pc-freebsd9
+        TOOLKIND=crossclang
+        TOOLNAME=pc-freebsd
+       	SYSROOT="$TOOLCHAIN_ROOT/sysroot"
         PLATSUFF="-bsd"
     else
+        TOOLKIND=buildroot
         TOOLNAME=linux
         PLATSUFF=""
     fi
 
     case $TARGET in
-        mipsel|i?86|x86_64)
+        mipsel|i?86|x86_64|aarch64)
             ARCH=$TARGET
             ENDIAN=little
             ;;
@@ -144,7 +148,7 @@ for TARGET in $TARGETS; do
             cd unrar
             sed 's:^CXX=:#CXX=:' -i makefile
             sed 's:^STRIP=strip:#STRIP=strip:' -i makefile
-            sed 's:^LDFLAGS=:LDFLAGS=-static :' -i makefile
+            sed 's:^LDFLAGS=:#LDFLAGS=-static :' -i makefile
             sed 's:^CXXFLAGS=-O2:#CXXFLAGS=-O2:' -i makefile
             if [ "$ENDIAN" = "big" ]; then
                 sed 's:^DEFINES=:DEFINES=-DBIG_ENDIAN :' -i makefile
@@ -152,6 +156,7 @@ for TARGET in $TARGETS; do
             if [ "$PLATSUFF" != "-bsd" ]; then
                 sed 's:^DEFINES=:DEFINES=-DUSE_FALLOCATE :' -i makefile
             fi
+            sed 's:setlocale://setlocale:' -i rar.cpp
 
             EXEDIR=
             LICENSE=license.txt
@@ -168,6 +173,11 @@ for TARGET in $TARGETS; do
             cp makefile.linux_any_cpu_gcc_4.X makefile.machine
             sed 's:^CXX=g++:#CXX=g++:' -i makefile.machine
             sed 's:^CC=gcc:#CC=gcc:' -i makefile.machine
+            if [ "$TOOLKIND" == "crossclang" ] ; then
+                sed 's:^PRE_COMPILED_HEADER=:#PRE_COMPILED_HEADER=:' -i makefile.machine
+                sed 's:^ALLFLAGS=${OPTFLAGS} -pipe -s:ALLFLAGS=${OPTFLAGS} -pipe ${CPPFLAGS}:' -i makefile.machine
+            fi
+            sed 's:setlocale(LC_ALL,"")://setlocale(LC_ALL,""):' -i CPP/myWindows/mySplitCommandLine.cpp
 
             EXEDIR=bin/
             LICENSE=DOC/License.txt
@@ -179,12 +189,23 @@ for TARGET in $TARGETS; do
 
     make clean
 
-    CXX=$TOOLPATH/$ARCH-$TOOLNAME-g++ \
-        CC=$TOOLPATH/$ARCH-$TOOLNAME-gcc \
-        STRIP=$TOOLPATH/$ARCH-$TOOLNAME-strip \
-        CXXFLAGS=-O2 \
-        LDFLAGS=-static \
-        make -j $COREX $BUILDTARGET
+    if [ "$TOOLKIND" == "crossclang" ] ; then
+        CXX="$CROSSCLANG" \
+            CC="$CROSSCLANG" \
+            LDFLAGS="-static -fuse-ld=lld -lc++ -lm -lpthread --sysroot=$SYSROOT" \
+            CXXFLAGS="-O2 --target=$ARCH-$TOOLNAME --sysroot=$SYSROOT -I$SYSROOT/usr/include/c++/v1" \
+            CPPFLAGS="-O2 --target=$ARCH-$TOOLNAME --sysroot=$SYSROOT -I$SYSROOT/usr/include/c++/v1" \
+            STRIP=strip \
+            make -j $COREX $BUILDTARGET
+    else
+        CXX=$TOOLPATH/$ARCH-$TOOLNAME-g++ \
+            CC=$TOOLPATH/$ARCH-$TOOLNAME-gcc \
+            STRIP=$TOOLPATH/$ARCH-$TOOLNAME-strip \
+            CXXFLAGS=-O2 \
+            LDFLAGS=-static \
+            LIBS=-lpthread \
+            make -j $COREX $BUILDTARGET
+    fi
 
     cp $EXEDIR$EXENAME ../../setup/$EXENAME-$TARGET$PLATSUFF
     cp $LICENSE ../../setup/license-$UNPACKER.txt

linux/installer.sh

@@ -2,7 +2,7 @@
 #
 #  This file is part of nzbget. See <http://nzbget.net>.
 #
-#  Copyright (C) 2015-2016 Andrey Prygunkov <hugbug@users.sourceforge.net>
+#  Copyright (C) 2015-2017 Andrey Prygunkov <hugbug@users.sourceforge.net>
 #
 #  This program is free software; you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -44,6 +44,7 @@ PRINTEDTITLE=no
 JUSTUNPACK=no
 UPDATE=no
 VERIFY=yes
+OS=""
 
 Info()
 {
@@ -60,11 +61,12 @@ Error()
 
 ValidArch()
 {
-    LIMARCH=`echo " $ALLARCHS " | sed "s/ $1 //"`
-    if test " $ALLARCHS " = "$LIMARCH"; then
-        return 1
-    fi
-    return 0
+    for TARG in $ALLARCHS; do
+        if test "$TARG" = "$1"; then
+            return 0
+        fi 
+    done
+    return 1
 }
 
 PrintArch()
@@ -135,7 +137,7 @@ Verify()
     if test "$REQSIZE" != "$ACTSIZE"; then
         Error "Corrupted installer package detected: file size mismatch."
     fi
-    
+
     # Checking checksum (MD5) of package payload, only if command 'md5sum' is available
     ACTMD5=`dd "if=$0" bs=$HEADER skip=1 2>/dev/null | /bin/sh -c md5sum 2>/dev/null | cut -b-32 2>/dev/null | cat`
     LEN=${#ACTMD5}
@@ -160,7 +162,7 @@ DetectEndianness()
 DetectArch()
 {
     OS=`uname -s`
-    if test "(" "$PLATFORM" = "linux" -a "$OS" != "Linux" ")" -o \
+    if test "(" "$PLATFORM" = "linux" -a "$OS" != "Linux" -a "$OS" != "FreeBSD" ")" -o \
             "(" "$PLATFORM" = "freebsd" -a "$OS" != "FreeBSD" ")" ; then
         PrintHelp
         Error "Operating system ($OS) isn't supported by this installer."
@@ -195,7 +197,7 @@ DetectArch()
         esac
     fi
 
-    if test "$PLATFORM" = "linux"; then
+    if test "$OS" = "Linux"; then
         if test "$ARCH" = ""; then
             MIPS=`cat /proc/cpuinfo | sed -n 's/.*:.*\(mips\).*/&/p'`
             if test "$MIPS" != ""; then
@@ -258,7 +260,7 @@ Unpack()
         rm -f "$OUTDIR/installer.tmp"
     fi
 
-    # Rename unpacked binaries files and store arch selection
+    # Rename unpacked binary files and store arch selection
     if test "$JUSTUNPACK" = "no" -a "$ARCH" != "all"; then
         OLDDIR=`pwd`
         cd "$OUTDIR"
@@ -339,6 +341,8 @@ ConfigureLinux()
             sed 's:^WriteBuffer=.*:WriteBuffer=1024:' -i nzbget.conf
             Info "  Increasing par repair buffer (ParBuffer=100)"
             sed 's:^ParBuffer=.*:ParBuffer=100:' -i nzbget.conf
+            Info "  Activating direct rename (DirectRename=yes)"
+            sed 's:^DirectRename=.*:DirectRename=yes:' -i nzbget.conf
         elif test $TOTALFREE -gt 25000 -o $MEMFREE -gt 25000 -o $MEMCACHED -gt 25000; then
             Info "  Increasing write buffer (WriteBuffer=256)"
             sed 's:^WriteBuffer=.*:WriteBuffer=256:' -i nzbget.conf
@@ -366,6 +370,8 @@ ConfigureLinux()
             sed 's:^ScriptPauseQueue=.*:ScriptPauseQueue=yes:' -i nzbget.conf
         else
             Info "  Simultaneous download and post-processing is on"
+            Info "  Activating direct unpack (DirectUnpack=yes)"
+            sed 's:^DirectUnpack=.*:DirectUnpack=yes:' -i nzbget.conf
         fi
     fi
 }
@@ -373,7 +379,7 @@ ConfigureLinux()
 ConfigureFreeBSD()
 {
     # Adjusting config file to current system
-    
+
     # No memory check, assuming all supported FreeBSD machines have enough memory
 
     Info "  Activating article cache (ArticleCache=100)"
@@ -382,6 +388,10 @@ ConfigureFreeBSD()
     sed -i '' 's:^WriteBuffer=.*:WriteBuffer=1024:' nzbget.conf
     Info "  Increasing par repair buffer (ParBuffer=100)"
     sed -i '' 's:^ParBuffer=.*:ParBuffer=100:' nzbget.conf
+    Info "  Activating direct rename (DirectRename=yes)"
+    sed -i '' 's:^DirectRename=.*:DirectRename=yes:' nzbget.conf
+    Info "  Activating direct unpack (DirectUnpack=yes)"
+    sed -i '' 's:^DirectUnpack=.*:DirectUnpack=yes:' nzbget.conf
 }
 
 Configure()
@@ -392,18 +402,43 @@ Configure()
     if test ! -f nzbget.conf; then
         cp ./webui/nzbget.conf.template nzbget.conf
 
-        if test "$PLATFORM" = "linux"; then
+        if test "$OS" = "Linux"; then
             ConfigureLinux
         fi
 
-        if test "$PLATFORM" = "freebsd"; then
+        if test "$OS" = "FreeBSD"; then
             ConfigureFreeBSD
         fi
-        
+
         QUICKHELP=yes
     fi
 }
 
+Linux2FreeBSD()
+{
+    if test "$PLATFORM" = "linux" -a "$OS" = "FreeBSD" ; then
+        # Using Linux installer on FreeBSD machine
+
+        if test "$1" = "brand" ; then
+            Info "  Branding Linux binaries for use on FreeBSD"
+            brandelf -t Linux nzbget
+            brandelf -t Linux unrar
+            brandelf -t Linux 7za
+        fi
+
+        if test "$1" = "kernel-check" ; then
+            MODLINUX=`kldstat | sed -n 's/\(linux.ko\)/\1/p'`
+            if test "$ARCH" = "x86_64"; then
+                MODLINUX=`kldstat | sed -n 's/\(linux64.ko\)/\1/p'`
+            fi
+            if test "$MODLINUX" = ""; then
+                Info ""
+                Info "WARNING: Linux kernel module isn't loaded. See http://nzbget.net/installation-on-freebsd"
+            fi
+        fi
+    fi
+}
+
 # ParseCommandLine
 while true
 do
@@ -501,6 +536,7 @@ ABSOUTDIR=`cd "$OUTDIR"; pwd`
 if test "$JUSTUNPACK" = "no"; then
     Info "Configuring..."
     Configure
+    Linux2FreeBSD "brand"
 
     Info "Installation completed"
 
@@ -538,6 +574,7 @@ if test "$JUSTUNPACK" = "no"; then
         Info "Successfully installed into $ABSOUTDIR"
     fi
     Info "For support please visit http://nzbget.net/forum"
+    Linux2FreeBSD "kernel-check"
 else
     Info "Unpacked into $ABSOUTDIR"
 fi

nzbget.conf

@@ -85,7 +85,7 @@ LockFile=${MainDir}/nzbget.lock
 # Where to store log file, if it needs to be created.
 #
 # NOTE: See also option <WriteLog>.
-LogFile=${DestDir}/nzbget.log
+LogFile=${MainDir}/nzbget.log
 
 # Configuration file template.
 #
@@ -286,7 +286,7 @@ Server1.Notes=
 # IP on which NZBGet server listen and which clients use to contact NZBGet.
 #
 # It could be a dns-hostname (e. g. "mypc") or an IP address (e. g. "192.168.1.2" or
-# "127.0.0.1"). An IP-address is more effective because does not require dns-lookup.
+# "127.0.0.1").
 #
 # Your computer may have multiple network interfaces and therefore multiple IP
 # addresses. If you want NZBGet to listen to all interfaces and be available from
@@ -297,6 +297,10 @@ Server1.Notes=
 #
 # NOTE: If you set the option to "127.0.0.1" you will be able to connect to NZBGet
 # only from the computer running NZBGet. This restriction applies to web-interface too.
+#
+# NOTE: NZBGet also supports listening on Unix domain sockets instead of TCP/IP
+# sockets. To activate this mode set option <ControlIP> to a local path
+# (e. g. "ControlIP=/var/sock").
 ControlIP=0.0.0.0
 
 # Port which NZBGet server and remote client use (1-65535).
@@ -389,9 +393,13 @@ SecureControl=no
 SecurePort=6791
 
 # Full path to certificate file for encrypted communication.
+#
+# In case of Let's Encrypt: full path to fullchain.pem.
 SecureCert=
 
 # Full path to key file for encrypted communication.
+#
+# In case of Let's Encrypt: full path to privkey.pem.
 SecureKey=
 
 # IP-addresses allowed to connect without authorization.
@@ -427,6 +435,13 @@ AuthorizedIP=
 # NOTE: For more details visit http://nzbget.net/certificate-verification.
 CertCheck=no
 
+# Automatically check for new releases (none, stable, testing).
+#
+#  None    - do not show notifcations;
+#  Stable  - show notifications about new stable releases;
+#  Testing - show notifications about new stable and testing releases.
+UpdateCheck=stable
+
 # User name for daemon-mode, POSIX only.
 #
 # Set the user that the daemon normally runs at (POSIX in daemon-mode only).
@@ -739,11 +754,6 @@ DupeCheck=yes
 ##############################################################################
 ### DOWNLOAD QUEUE                                                         ###
 
-# Save download queue to disk (yes, no).
-#
-# This allows to reload it on next start.
-SaveQueue=yes
-
 # Flush download queue to disk (yes, no).
 #
 # Immediately flush file buffers for queue state file. This improves
@@ -755,9 +765,6 @@ SaveQueue=yes
 # in that case. Keep the option enabled if your system often crashes.
 FlushQueue=yes
 
-# Reload download queue on start, if it exists (yes, no).
-ReloadQueue=yes
-
 # Continue download of partially downloaded files (yes, no).
 #
 # If active the current state (the info about what articles were already
@@ -778,15 +785,6 @@ ContinuePartial=yes
 # after that they are downloaded.
 PropagationDelay=0
 
-# Decode articles (yes, no).
-#
-# yes - decode articles using internal decoder (supports yEnc and UU formats);
-# no - articles will not be decoded/joined. Useful to look at article's source text.
-#
-# NOTE: This option is primary for debugging purposes. You should not
-# disable it.
-Decode=yes
-
 # Memory limit for article cache (megabytes).
 #
 # Article cache helps to improve performance. First the amount of disk
@@ -837,7 +835,7 @@ ArticleCache=0
 # without article cache.
 DirectWrite=yes
 
-# Memory limit for per article write buffer (kilobytes).
+# Memory limit for per connection write buffer (kilobytes).
 #
 # When downloaded articles are written into disk the OS collects
 # data in the internal buffer before flushing it into disk. This option
@@ -864,13 +862,6 @@ DirectWrite=yes
 # NOTE: Also see option <ArticleCache>.
 WriteBuffer=0
 
-# Check CRC of downloaded and decoded articles (yes, no).
-#
-# Normally this option should be enabled for better detecting of download
-# errors. However checking of CRC needs CPU time. On a fast connection and
-# slow CPU disabling of CRC-Check may improve performance.
-CrcCheck=yes
-
 # How to name downloaded files (auto, article, nzb).
 #
 #  Article - use file names stored in article metadata;
@@ -878,7 +869,7 @@ CrcCheck=yes
 #  Auto    - prefer names from article metadata; for obfuscated files use
 #            names from nzb-file.
 #
-# NOTE: This option sets the naming convention for files listed in nzb. It has not
+# NOTE: This option sets the naming convention for files listed in nzb. It has no
 # effect on files extracted from archives.
 FileNaming=auto
 
@@ -969,6 +960,27 @@ KeepHistory=30
 # the feed history.
 FeedHistory=7
 
+# Discard downloaded data (do not write into disk) (yes, no).
+#
+# This option is for speed test purposes (benchmarking). When enabled the
+# downloaded data is not written into disk. The destination files are still
+# created but are either empty or contain zeros (depending on other
+# options). The post-processing (unpack, repair, etc.) is also completely
+# disabled.
+#
+# NOTE: This option is meant for development purposes. You should not
+# activate it except maybe for speed tests.
+SkipWrite=no
+
+# Write article raw data (yes, no).
+#
+# When enabled the article content is written into disk in raw form without
+# processing.
+#
+# NOTE: This option is meant for development purposes. You should not
+# activate it.
+RawArticle=no
+
 ##############################################################################
 ### CONNECTION                                                             ###
 
@@ -1009,11 +1021,10 @@ UrlInterval=10
 # Connection timeout when fetching nzb-files via URLs and fetching RSS feeds.
 UrlTimeout=60
 
-# Timeout until a download-thread should be killed (seconds).
+# Timeout for incoming connections (seconds).
 #
-# This can help on hanging downloads, but is dangerous.
-# Do not use small values!
-TerminateTimeout=600
+# Set timeout for connections from clients (web-browsers and API clients).
+RemoteTimeout=90
 
 # Set the maximum download rate on program start (kilobytes/sec).
 #
@@ -1022,24 +1033,6 @@ TerminateTimeout=600
 # Value "0" means no speed control.
 DownloadRate=0
 
-# Accurate speed rate calculation (yes, no).
-#
-# During downloading using several connections the download threads may
-# interfere with each other when updating statistical data for speed
-# meter. This may cause small errors in current download speed reported
-# by the program. The speed meter recovers automatically from such errors
-# after max. 30 seconds (time window used for speed calculation).
-#
-# Enable the option to use thread synchronisation mechanisms in order to
-# provide absolutely accurate speed calculations.
-#
-# NOTE: Thread synchronisation increases CPU load and therefore can
-# decrease download speed. Do not activate this option on computers with
-# limited CPU power. Before activating the option it is recommended to
-# run tests to determine how the option affects the CPU usage and the
-# download speed on a particular system.
-AccurateRate=no
-
 # Maximum number of simultaneous connections for nzb URL downloads (0-999).
 #
 # When NZB-files are added to queue via URL, the program downloads them
@@ -1113,9 +1106,8 @@ DetailTarget=log
 # debug-mode: "./configure --enable-debug".
 DebugTarget=log
 
-# Number of messages stored in buffer and available for remote
-# clients (messages).
-LogBufferSize=1000
+# Number of messages stored in screen buffer (messages).
+LogBuffer=1000
 
 # Create log for each downloaded nzb-file (yes, no).
 #
@@ -1123,12 +1115,6 @@ LogBufferSize=1000
 # at any time in download details dialog or history details dialog.
 NzbLog=yes
 
-# Create a log of all broken files (yes, no).
-#
-# It is a text file placed near downloaded files, which contains
-# the names of broken files.
-BrokenLog=yes
-
 # Print call stack trace into log on program crash (Linux and Windows) (yes, no).
 #
 # Call stack traces are very helpful for debugging. Call stack traces can be
@@ -1289,6 +1275,12 @@ UpdateInterval=200
 ##############################################################################
 ### CHECK AND REPAIR                                                       ###
 
+# Check CRC of downloaded and decoded articles (yes, no).
+#
+# Normally this option should be enabled for better detecting of download
+# errors and for quick par-verification (option <ParQuick>).
+CrcCheck=yes
+
 # Whether and how par-verification must be performed (auto, always, force, manual).
 #
 #  Auto   - par-check is performed when needed. One par2-file is always
@@ -1557,7 +1549,7 @@ SevenZipCmd=7z
 # characters * and ?.
 #
 # Example: .par2, .sfv
-ExtCleanupDisk=.par2, .sfv, _brokenlog.txt
+ExtCleanupDisk=.par2, .sfv
 
 # Files to ignore during unpack.
 #

nzbget.vcxproj

@@ -1,14 +1,22 @@
-<?xml version="1.0" encoding="utf-8"?>
+<?xml version="1.0" encoding="utf-8"?>
 <Project DefaultTargets="Build" ToolsVersion="14.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
   <ItemGroup Label="ProjectConfigurations">
     <ProjectConfiguration Include="Debug|Win32">
       <Configuration>Debug</Configuration>
       <Platform>Win32</Platform>
     </ProjectConfiguration>
+    <ProjectConfiguration Include="Debug|x64">
+      <Configuration>Debug</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
     <ProjectConfiguration Include="Release|Win32">
       <Configuration>Release</Configuration>
       <Platform>Win32</Platform>
     </ProjectConfiguration>
+    <ProjectConfiguration Include="Release|x64">
+      <Configuration>Release</Configuration>
+      <Platform>x64</Platform>
+    </ProjectConfiguration>
   </ItemGroup>
   <PropertyGroup Label="Globals">
     <ProjectGuid>{41BFB691-0127-4391-9629-F1BA6740DDFE}</ProjectGuid>
@@ -21,38 +29,58 @@
     <ConfigurationType>Application</ConfigurationType>
     <PlatformToolset>v140_xp</PlatformToolset>
   </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="Configuration">
+    <ConfigurationType>Application</ConfigurationType>
+    <PlatformToolset>v140_xp</PlatformToolset>
+  </PropertyGroup>
   <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="Configuration">
     <ConfigurationType>Application</ConfigurationType>
     <PlatformToolset>v140_xp</PlatformToolset>
   </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="Configuration">
+    <ConfigurationType>Application</ConfigurationType>
+    <PlatformToolset>v140_xp</PlatformToolset>
+  </PropertyGroup>
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.props" />
   <ImportGroup Label="ExtensionSettings">
   </ImportGroup>
   <ImportGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'" Label="PropertySheets">
     <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
   </ImportGroup>
+  <ImportGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'" Label="PropertySheets">
+    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+  </ImportGroup>
   <ImportGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'" Label="PropertySheets">
     <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
   </ImportGroup>
+  <ImportGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'" Label="PropertySheets">
+    <Import Project="$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props" Condition="exists('$(UserRootDir)\Microsoft.Cpp.$(Platform).user.props')" Label="LocalAppDataPlatform" />
+  </ImportGroup>
   <PropertyGroup Label="UserMacros" />
   <PropertyGroup>
     <_ProjectFileVersion>14.0.23107.0</_ProjectFileVersion>
   </PropertyGroup>
   <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
     <OutDir>..\bin\</OutDir>
-    <IntDir>..\bin\Debug\</IntDir>
-    <LinkIncremental>true</LinkIncremental>
+    <IntDir>..\bin\Debug32\</IntDir>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
+    <OutDir>..\bin\</OutDir>
+    <IntDir>..\bin\Debug64\</IntDir>
   </PropertyGroup>
   <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
     <OutDir>..\bin\</OutDir>
-    <IntDir>..\bin\Release\</IntDir>
-    <LinkIncremental />
+    <IntDir>..\bin\Release32\</IntDir>
+  </PropertyGroup>
+  <PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
+    <OutDir>..\bin\</OutDir>
+    <IntDir>..\bin\Release64\</IntDir>
   </PropertyGroup>
   <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">
     <ClCompile>
       <Optimization>Disabled</Optimization>
-      <AdditionalIncludeDirectories>.\daemon\connect;.\daemon\extension;.\daemon\feed;.\daemon\frontend;.\daemon\main;.\daemon\nserv;.\daemon\nntp;.\daemon\postprocess;.\daemon\queue;.\daemon\remote;.\daemon\util;.\daemon\windows;.\lib\par2;.\windows\resources;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
-      <PreprocessorDefinitions>WIN32;PACKAGE="nzbget";VERSION="19.1";_DEBUG;_CONSOLE;DEBUG;_WIN32_WINNT=0x0403;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <AdditionalIncludeDirectories>.\daemon\connect;.\daemon\extension;.\daemon\feed;.\daemon\frontend;.\daemon\main;.\daemon\nserv;.\daemon\nntp;.\daemon\postprocess;.\daemon\queue;.\daemon\remote;.\daemon\util;.\daemon\windows;.\lib\par2;.\lib\yencode;.\windows\resources;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+      <PreprocessorDefinitions>WIN32;PACKAGE="nzbget";VERSION="20.0";_DEBUG;_CONSOLE;DEBUG;_WIN32_WINNT=0x0403;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <MinimalRebuild>false</MinimalRebuild>
       <BasicRuntimeChecks>EnableFastChecks</BasicRuntimeChecks>
       <RuntimeLibrary>MultiThreadedDebug</RuntimeLibrary>
@@ -63,16 +91,36 @@
       <MultiProcessorCompilation>true</MultiProcessorCompilation>
     </ClCompile>
     <Link>
-      <AdditionalDependencies>WS2_32.lib;ole32.lib;OleAut32.Lib;comsuppwd.lib;Advapi32.lib;Winmm.lib;gdi32.lib;shell32.lib;dbghelp.lib;ssleay32MTd.lib;libeay32MTd.lib;regex.lib;zlib.lib</AdditionalDependencies>
+      <AdditionalDependencies>WS2_32.lib;ole32.lib;OleAut32.Lib;comsuppwd.lib;Advapi32.lib;Winmm.lib;gdi32.lib;shell32.lib;dbghelp.lib;ssleay32MTd.lib;libeay32MTd.lib;regex32MTd.lib;zlib32MTd.lib</AdditionalDependencies>
       <GenerateDebugInformation>true</GenerateDebugInformation>
       <SubSystem>Console</SubSystem>
       <TargetMachine>MachineX86</TargetMachine>
     </Link>
   </ItemDefinitionGroup>
+  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Debug|x64'">
+    <ClCompile>
+      <Optimization>Disabled</Optimization>
+      <AdditionalIncludeDirectories>.\daemon\connect;.\daemon\extension;.\daemon\feed;.\daemon\frontend;.\daemon\main;.\daemon\nserv;.\daemon\nntp;.\daemon\postprocess;.\daemon\queue;.\daemon\remote;.\daemon\util;.\daemon\windows;.\lib\par2;.\lib\yencode;.\windows\resources;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+      <PreprocessorDefinitions>WIN32;PACKAGE="nzbget";VERSION="20.0";_DEBUG;_CONSOLE;DEBUG;_WIN32_WINNT=0x0403;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <MinimalRebuild>false</MinimalRebuild>
+      <BasicRuntimeChecks>EnableFastChecks</BasicRuntimeChecks>
+      <RuntimeLibrary>MultiThreadedDebug</RuntimeLibrary>
+      <WarningLevel>Level3</WarningLevel>
+      <DisableSpecificWarnings>4996;%(DisableSpecificWarnings)</DisableSpecificWarnings>
+      <PrecompiledHeader>Use</PrecompiledHeader>
+      <PrecompiledHeaderFile>nzbget.h</PrecompiledHeaderFile>
+      <MultiProcessorCompilation>true</MultiProcessorCompilation>
+    </ClCompile>
+    <Link>
+      <AdditionalDependencies>WS2_32.lib;ole32.lib;OleAut32.Lib;comsuppwd.lib;Advapi32.lib;Winmm.lib;gdi32.lib;shell32.lib;dbghelp.lib;ssleay32MTd.lib;libeay32MTd.lib;regex32MTd.lib;zlib32MTd.lib</AdditionalDependencies>
+      <GenerateDebugInformation>true</GenerateDebugInformation>
+      <SubSystem>Console</SubSystem>
+    </Link>
+  </ItemDefinitionGroup>
   <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">
     <ClCompile>
-      <AdditionalIncludeDirectories>.\daemon\connect;.\daemon\extension;.\daemon\feed;.\daemon\frontend;.\daemon\main;.\daemon\nserv;.\daemon\nntp;.\daemon\postprocess;.\daemon\queue;.\daemon\remote;.\daemon\util;.\daemon\windows;.\lib\par2;.\windows\resources;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
-      <PreprocessorDefinitions>WIN32;PACKAGE="nzbget";VERSION="19.1";NDEBUG;_CONSOLE;_WIN32_WINNT=0x0403;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <AdditionalIncludeDirectories>.\daemon\connect;.\daemon\extension;.\daemon\feed;.\daemon\frontend;.\daemon\main;.\daemon\nserv;.\daemon\nntp;.\daemon\postprocess;.\daemon\queue;.\daemon\remote;.\daemon\util;.\daemon\windows;.\lib\par2;.\lib\yencode;.\windows\resources;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+      <PreprocessorDefinitions>WIN32;PACKAGE="nzbget";VERSION="20.0";NDEBUG;_CONSOLE;_WIN32_WINNT=0x0403;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <ExceptionHandling>Sync</ExceptionHandling>
       <RuntimeLibrary>MultiThreaded</RuntimeLibrary>
       <PrecompiledHeader>Use</PrecompiledHeader>
@@ -81,14 +129,53 @@
       <PrecompiledHeaderFile>nzbget.h</PrecompiledHeaderFile>
       <AdditionalOptions>/MP %(AdditionalOptions)</AdditionalOptions>
       <MultiProcessorCompilation>true</MultiProcessorCompilation>
+      <Optimization>Full</Optimization>
+      <InlineFunctionExpansion>AnySuitable</InlineFunctionExpansion>
+      <IntrinsicFunctions>true</IntrinsicFunctions>
+      <FavorSizeOrSpeed>Speed</FavorSizeOrSpeed>
+      <OmitFramePointers>true</OmitFramePointers>
+      <EnableFiberSafeOptimizations>true</EnableFiberSafeOptimizations>
+      <WholeProgramOptimization>true</WholeProgramOptimization>
+      <FloatingPointModel>Fast</FloatingPointModel>
     </ClCompile>
     <Link>
-      <AdditionalDependencies>WS2_32.lib;ole32.lib;OleAut32.Lib;comsuppwd.lib;Advapi32.lib;gdi32.lib;shell32.lib;Winmm.lib;ssleay32MT.lib;libeay32MT.lib;regex.lib;zlib.lib</AdditionalDependencies>
+      <AdditionalDependencies>WS2_32.lib;ole32.lib;OleAut32.Lib;comsuppwd.lib;Advapi32.lib;gdi32.lib;shell32.lib;Winmm.lib;ssleay32MT.lib;libeay32MT.lib;regex32MT.lib;zlib32MT.lib</AdditionalDependencies>
       <GenerateDebugInformation>true</GenerateDebugInformation>
       <SubSystem>Console</SubSystem>
       <OptimizeReferences>true</OptimizeReferences>
       <EnableCOMDATFolding>true</EnableCOMDATFolding>
       <TargetMachine>MachineX86</TargetMachine>
+      <LinkTimeCodeGeneration>UseLinkTimeCodeGeneration</LinkTimeCodeGeneration>
+    </Link>
+  </ItemDefinitionGroup>
+  <ItemDefinitionGroup Condition="'$(Configuration)|$(Platform)'=='Release|x64'">
+    <ClCompile>
+      <AdditionalIncludeDirectories>.\daemon\connect;.\daemon\extension;.\daemon\feed;.\daemon\frontend;.\daemon\main;.\daemon\nserv;.\daemon\nntp;.\daemon\postprocess;.\daemon\queue;.\daemon\remote;.\daemon\util;.\daemon\windows;.\lib\par2;.\lib\yencode;.\windows\resources;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+      <PreprocessorDefinitions>WIN32;PACKAGE="nzbget";VERSION="20.0";NDEBUG;_CONSOLE;_WIN32_WINNT=0x0403;%(PreprocessorDefinitions)</PreprocessorDefinitions>
+      <ExceptionHandling>Sync</ExceptionHandling>
+      <RuntimeLibrary>MultiThreaded</RuntimeLibrary>
+      <PrecompiledHeader>Use</PrecompiledHeader>
+      <WarningLevel>Level3</WarningLevel>
+      <DisableSpecificWarnings>4996;%(DisableSpecificWarnings)</DisableSpecificWarnings>
+      <PrecompiledHeaderFile>nzbget.h</PrecompiledHeaderFile>
+      <AdditionalOptions>/MP %(AdditionalOptions)</AdditionalOptions>
+      <MultiProcessorCompilation>true</MultiProcessorCompilation>
+      <Optimization>Full</Optimization>
+      <InlineFunctionExpansion>AnySuitable</InlineFunctionExpansion>
+      <IntrinsicFunctions>true</IntrinsicFunctions>
+      <FavorSizeOrSpeed>Speed</FavorSizeOrSpeed>
+      <OmitFramePointers>true</OmitFramePointers>
+      <EnableFiberSafeOptimizations>true</EnableFiberSafeOptimizations>
+      <WholeProgramOptimization>true</WholeProgramOptimization>
+      <FloatingPointModel>Fast</FloatingPointModel>
+    </ClCompile>
+    <Link>
+      <AdditionalDependencies>WS2_32.lib;ole32.lib;OleAut32.Lib;comsuppwd.lib;Advapi32.lib;gdi32.lib;shell32.lib;Winmm.lib;ssleay32MTd.lib;libeay32MTd.lib;regex32MT.lib;zlib32MT.lib</AdditionalDependencies>
+      <GenerateDebugInformation>true</GenerateDebugInformation>
+      <SubSystem>Console</SubSystem>
+      <OptimizeReferences>true</OptimizeReferences>
+      <EnableCOMDATFolding>true</EnableCOMDATFolding>
+      <LinkTimeCodeGeneration>UseLinkTimeCodeGeneration</LinkTimeCodeGeneration>
     </Link>
   </ItemDefinitionGroup>
   <ItemGroup>
@@ -166,8 +253,7 @@
     <ClCompile Include="daemon\util\Util.cpp" />
     <ClCompile Include="daemon\util\FileSystem.cpp" />
     <ClCompile Include="daemon\windows\StdAfx.cpp">
-      <PrecompiledHeader Condition="'$(Configuration)|$(Platform)'=='Debug|Win32'">Create</PrecompiledHeader>
-      <PrecompiledHeader Condition="'$(Configuration)|$(Platform)'=='Release|Win32'">Create</PrecompiledHeader>
+      <PrecompiledHeader >Create</PrecompiledHeader>
     </ClCompile>
     <ClCompile Include="daemon\windows\WinService.cpp" />
     <ClCompile Include="daemon\windows\WinConsole.cpp" />
@@ -190,6 +276,12 @@
     <ClCompile Include="lib\par2\reedsolomon.cpp" />
     <ClCompile Include="lib\par2\verificationhashtable.cpp" />
     <ClCompile Include="lib\par2\verificationpacket.cpp" />
+    <ClCompile Include="lib\yencode\SimdInit.cpp" />
+    <ClCompile Include="lib\yencode\ScalarDecoder.cpp" />
+    <ClCompile Include="lib\yencode\Sse2Decoder.cpp" />
+    <ClCompile Include="lib\yencode\Ssse3Decoder.cpp" />
+    <ClCompile Include="lib\yencode\SliceCrc.cpp" />
+    <ClCompile Include="lib\yencode\PclmulCrc.cpp" />
   </ItemGroup>
   <ItemGroup>
     <ClInclude Include="daemon\connect\Connection.h" />
@@ -290,6 +382,8 @@
     <ClInclude Include="lib\par2\reedsolomon.h" />
     <ClInclude Include="lib\par2\verificationhashtable.h" />
     <ClInclude Include="lib\par2\verificationpacket.h" />
+    <ClInclude Include="lib\yencode\YEncode.h" />
+    <ClInclude Include="lib\yencode\SimdDecoder.cpp" />
     <ClInclude Include="windows\resources\resource.h" />
   </ItemGroup>
   <ItemGroup>
@@ -304,4 +398,4 @@
   <Import Project="$(VCTargetsPath)\Microsoft.Cpp.targets" />
   <ImportGroup Label="ExtensionTargets">
   </ImportGroup>
-</Project>
+</Project>

osx/NZBGet-Info.plist

@@ -25,7 +25,7 @@
 	<key>LSUIElement</key>
 	<true/>
 	<key>NSHumanReadableCopyright</key>
-	<string>Copyright © 2007-2017 Andrey Prygunkov</string>
+	<string>Copyright © 2007-2018 Andrey Prygunkov</string>
 	<key>NSMainNibFile</key>
 	<string>MainApp</string>
 	<key>NSAppTransportSecurity</key>

posix/config.sub

@@ -1,44 +1,40 @@
 #! /bin/sh
 # Configuration validation subroutine script.
-#   Copyright (C) 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999,
-#   2000, 2001, 2002, 2003, 2004, 2005, 2006 Free Software Foundation,
-#   Inc.
+#   Copyright 1992-2014 Free Software Foundation, Inc.
 
-timestamp='2006-09-20'
+timestamp='2014-12-03'
 
-# This file is (in principle) common to ALL GNU software.
-# The presence of a machine in this file suggests that SOME GNU software
-# can handle that machine.  It does not imply ALL GNU software can.
-#
-# This file is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
+# This file is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3 of the License, or
 # (at your option) any later version.
 #
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street - Fifth Floor, Boston, MA
-# 02110-1301, USA.
+# along with this program; if not, see <http://www.gnu.org/licenses/>.
 #
 # As a special exception to the GNU General Public License, if you
 # distribute this file as part of a program that contains a
 # configuration script generated by Autoconf, you may include it under
-# the same distribution terms that you use for the rest of that program.
+# the same distribution terms that you use for the rest of that
+# program.  This Exception is an additional permission under section 7
+# of the GNU General Public License, version 3 ("GPLv3").
 
 
-# Please send patches to <config-patches@gnu.org>.  Submit a context
-# diff and a properly formatted ChangeLog entry.
+# Please send patches to <config-patches@gnu.org>.
 #
 # Configuration subroutine to validate and canonicalize a configuration type.
 # Supply the specified configuration type as an argument.
 # If it is invalid, we print an error message on stderr and exit with code 1.
 # Otherwise, we print the canonical config type on stdout and succeed.
 
+# You can get the latest version of this script from:
+# http://git.savannah.gnu.org/gitweb/?p=config.git;a=blob_plain;f=config.sub;hb=HEAD
+
 # This file is supposed to be the same for all GNU packages
 # and recognize all the CPU types, system types and aliases
 # that are meaningful with *any* GNU software.
@@ -72,8 +68,7 @@ Report bugs and patches to <config-patches@gnu.org>."
 version="\
 GNU config.sub ($timestamp)
 
-Copyright (C) 1992, 1993, 1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005
-Free Software Foundation, Inc.
+Copyright 1992-2014 Free Software Foundation, Inc.
 
 This is free software; see the source for copying conditions.  There is NO
 warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE."
@@ -120,12 +115,18 @@ esac
 # Here we must recognize all the valid KERNEL-OS combinations.
 maybe_os=`echo $1 | sed 's/^\(.*\)-\([^-]*-[^-]*\)$/\2/'`
 case $maybe_os in
-  nto-qnx* | linux-gnu* | linux-dietlibc | linux-newlib* | linux-uclibc* | \
-  uclinux-uclibc* | uclinux-gnu* | kfreebsd*-gnu* | knetbsd*-gnu* | netbsd*-gnu* | \
+  nto-qnx* | linux-gnu* | linux-android* | linux-dietlibc | linux-newlib* | \
+  linux-musl* | linux-uclibc* | uclinux-uclibc* | uclinux-gnu* | kfreebsd*-gnu* | \
+  knetbsd*-gnu* | netbsd*-gnu* | \
+  kopensolaris*-gnu* | \
   storm-chaos* | os2-emx* | rtmk-nova*)
     os=-$maybe_os
     basic_machine=`echo $1 | sed 's/^\(.*\)-\([^-]*-[^-]*\)$/\1/'`
     ;;
+  android-linux)
+    os=-linux-android
+    basic_machine=`echo $1 | sed 's/^\(.*\)-\([^-]*-[^-]*\)$/\1/'`-unknown
+    ;;
   *)
     basic_machine=`echo $1 | sed 's/-[^-]*$//'`
     if [ $basic_machine != $1 ]
@@ -148,10 +149,13 @@ case $os in
 	-convergent* | -ncr* | -news | -32* | -3600* | -3100* | -hitachi* |\
 	-c[123]* | -convex* | -sun | -crds | -omron* | -dg | -ultra | -tti* | \
 	-harris | -dolphin | -highlevel | -gould | -cbm | -ns | -masscomp | \
-	-apple | -axis | -knuth | -cray)
+	-apple | -axis | -knuth | -cray | -microblaze*)
 		os=
 		basic_machine=$1
 		;;
+	-bluegene*)
+		os=-cnk
+		;;
 	-sim | -cisco | -oki | -wec | -winbond)
 		os=
 		basic_machine=$1
@@ -166,10 +170,10 @@ case $os in
 		os=-chorusos
 		basic_machine=$1
 		;;
- 	-chorusrdb)
- 		os=-chorusrdb
+	-chorusrdb)
+		os=-chorusrdb
 		basic_machine=$1
- 		;;
+		;;
 	-hiux*)
 		os=-hiuxwe2
 		;;
@@ -214,6 +218,12 @@ case $os in
 	-isc*)
 		basic_machine=`echo $1 | sed -e 's/86-.*/86-pc/'`
 		;;
+	-lynx*178)
+		os=-lynxos178
+		;;
+	-lynx*5)
+		os=-lynxos5
+		;;
 	-lynx*)
 		os=-lynxos
 		;;
@@ -238,59 +248,90 @@ case $basic_machine in
 	# Some are omitted here because they have special meanings below.
 	1750a | 580 \
 	| a29k \
+	| aarch64 | aarch64_be \
 	| alpha | alphaev[4-8] | alphaev56 | alphaev6[78] | alphapca5[67] \
 	| alpha64 | alpha64ev[4-8] | alpha64ev56 | alpha64ev6[78] | alpha64pca5[67] \
 	| am33_2.0 \
-	| arc | arm | arm[bl]e | arme[lb] | armv[2345] | armv[345][lb] | avr | avr32 \
+	| arc | arceb \
+	| arm | arm[bl]e | arme[lb] | armv[2-8] | armv[3-8][lb] | armv7[arm] \
+	| avr | avr32 \
+	| be32 | be64 \
 	| bfin \
-	| c4x | clipper \
+	| c4x | c8051 | clipper \
 	| d10v | d30v | dlx | dsp16xx \
-	| fr30 | frv \
+	| epiphany \
+	| fido | fr30 | frv \
 	| h8300 | h8500 | hppa | hppa1.[01] | hppa2.0 | hppa2.0[nw] | hppa64 \
+	| hexagon \
 	| i370 | i860 | i960 | ia64 \
 	| ip2k | iq2000 \
+	| k1om \
+	| le32 | le64 \
+	| lm32 \
 	| m32c | m32r | m32rle | m68000 | m68k | m88k \
-	| maxq | mb | microblaze | mcore \
+	| maxq | mb | microblaze | microblazeel | mcore | mep | metag \
 	| mips | mipsbe | mipseb | mipsel | mipsle \
 	| mips16 \
 	| mips64 | mips64el \
-	| mips64vr | mips64vrel \
+	| mips64octeon | mips64octeonel \
 	| mips64orion | mips64orionel \
+	| mips64r5900 | mips64r5900el \
+	| mips64vr | mips64vrel \
 	| mips64vr4100 | mips64vr4100el \
 	| mips64vr4300 | mips64vr4300el \
 	| mips64vr5000 | mips64vr5000el \
 	| mips64vr5900 | mips64vr5900el \
 	| mipsisa32 | mipsisa32el \
 	| mipsisa32r2 | mipsisa32r2el \
+	| mipsisa32r6 | mipsisa32r6el \
 	| mipsisa64 | mipsisa64el \
 	| mipsisa64r2 | mipsisa64r2el \
+	| mipsisa64r6 | mipsisa64r6el \
 	| mipsisa64sb1 | mipsisa64sb1el \
 	| mipsisa64sr71k | mipsisa64sr71kel \
+	| mipsr5900 | mipsr5900el \
 	| mipstx39 | mipstx39el \
 	| mn10200 | mn10300 \
+	| moxie \
 	| mt \
 	| msp430 \
-	| nios | nios2 \
+	| nds32 | nds32le | nds32be \
+	| nios | nios2 | nios2eb | nios2el \
 	| ns16k | ns32k \
-	| or32 \
+	| open8 | or1k | or1knd | or32 \
 	| pdp10 | pdp11 | pj | pjl \
-	| powerpc | powerpc64 | powerpc64le | powerpcle | ppcbe \
+	| powerpc | powerpc64 | powerpc64le | powerpcle \
 	| pyramid \
+	| riscv32 | riscv64 \
+	| rl78 | rx \
 	| score \
-	| sh | sh[1234] | sh[24]a | sh[23]e | sh[34]eb | sheb | shbe | shle | sh[1234]le | sh3ele \
+	| sh | sh[1234] | sh[24]a | sh[24]aeb | sh[23]e | sh[34]eb | sheb | shbe | shle | sh[1234]le | sh3ele \
 	| sh64 | sh64le \
 	| sparc | sparc64 | sparc64b | sparc64v | sparc86x | sparclet | sparclite \
 	| sparcv8 | sparcv9 | sparcv9b | sparcv9v \
-	| spu | strongarm \
-	| tahoe | thumb | tic4x | tic80 | tron \
-	| v850 | v850e \
+	| spu \
+	| tahoe | tic4x | tic54x | tic55x | tic6x | tic80 | tron \
+	| ubicom32 \
+	| v850 | v850e | v850e1 | v850e2 | v850es | v850e2v3 \
+	| visium \
 	| we32k \
-	| x86 | xc16x | xscale | xscalee[bl] | xstormy16 | xtensa \
-	| z8k)
+	| x86 | xc16x | xstormy16 | xtensa \
+	| z8k | z80)
 		basic_machine=$basic_machine-unknown
 		;;
-	m6811 | m68hc11 | m6812 | m68hc12)
-		# Motorola 68HC11/12.
+	c54x)
+		basic_machine=tic54x-unknown
+		;;
+	c55x)
+		basic_machine=tic55x-unknown
+		;;
+	c6x)
+		basic_machine=tic6x-unknown
+		;;
+	leon|leon[3-9])
+		basic_machine=sparc-$basic_machine
+		;;
+	m6811 | m68hc11 | m6812 | m68hc12 | m68hcs12x | nvptx | picochip)
 		basic_machine=$basic_machine-unknown
 		os=-none
 		;;
@@ -300,6 +341,21 @@ case $basic_machine in
 		basic_machine=mt-unknown
 		;;
 
+	strongarm | thumb | xscale)
+		basic_machine=arm-unknown
+		;;
+	xgate)
+		basic_machine=$basic_machine-unknown
+		os=-none
+		;;
+	xscaleeb)
+		basic_machine=armeb-unknown
+		;;
+
+	xscaleel)
+		basic_machine=armel-unknown
+		;;
+
 	# We use `pc' rather than `unknown'
 	# because (1) that's what they normally are, and
 	# (2) the word "unknown" tends to confuse beginning users.
@@ -314,64 +370,87 @@ case $basic_machine in
 	# Recognize the basic CPU types with company name.
 	580-* \
 	| a29k-* \
+	| aarch64-* | aarch64_be-* \
 	| alpha-* | alphaev[4-8]-* | alphaev56-* | alphaev6[78]-* \
 	| alpha64-* | alpha64ev[4-8]-* | alpha64ev56-* | alpha64ev6[78]-* \
-	| alphapca5[67]-* | alpha64pca5[67]-* | arc-* \
+	| alphapca5[67]-* | alpha64pca5[67]-* | arc-* | arceb-* \
 	| arm-*  | armbe-* | armle-* | armeb-* | armv*-* \
 	| avr-* | avr32-* \
+	| be32-* | be64-* \
 	| bfin-* | bs2000-* \
-	| c[123]* | c30-* | [cjt]90-* | c4x-* | c54x-* | c55x-* | c6x-* \
-	| clipper-* | craynv-* | cydra-* \
+	| c[123]* | c30-* | [cjt]90-* | c4x-* \
+	| c8051-* | clipper-* | craynv-* | cydra-* \
 	| d10v-* | d30v-* | dlx-* \
 	| elxsi-* \
-	| f30[01]-* | f700-* | fr30-* | frv-* | fx80-* \
+	| f30[01]-* | f700-* | fido-* | fr30-* | frv-* | fx80-* \
 	| h8300-* | h8500-* \
 	| hppa-* | hppa1.[01]-* | hppa2.0-* | hppa2.0[nw]-* | hppa64-* \
+	| hexagon-* \
 	| i*86-* | i860-* | i960-* | ia64-* \
 	| ip2k-* | iq2000-* \
+	| k1om-* \
+	| le32-* | le64-* \
+	| lm32-* \
 	| m32c-* | m32r-* | m32rle-* \
 	| m68000-* | m680[012346]0-* | m68360-* | m683?2-* | m68k-* \
-	| m88110-* | m88k-* | maxq-* | mcore-* \
+	| m88110-* | m88k-* | maxq-* | mcore-* | metag-* \
+	| microblaze-* | microblazeel-* \
 	| mips-* | mipsbe-* | mipseb-* | mipsel-* | mipsle-* \
 	| mips16-* \
 	| mips64-* | mips64el-* \
-	| mips64vr-* | mips64vrel-* \
+	| mips64octeon-* | mips64octeonel-* \
 	| mips64orion-* | mips64orionel-* \
+	| mips64r5900-* | mips64r5900el-* \
+	| mips64vr-* | mips64vrel-* \
 	| mips64vr4100-* | mips64vr4100el-* \
 	| mips64vr4300-* | mips64vr4300el-* \
 	| mips64vr5000-* | mips64vr5000el-* \
 	| mips64vr5900-* | mips64vr5900el-* \
 	| mipsisa32-* | mipsisa32el-* \
 	| mipsisa32r2-* | mipsisa32r2el-* \
+	| mipsisa32r6-* | mipsisa32r6el-* \
 	| mipsisa64-* | mipsisa64el-* \
 	| mipsisa64r2-* | mipsisa64r2el-* \
+	| mipsisa64r6-* | mipsisa64r6el-* \
 	| mipsisa64sb1-* | mipsisa64sb1el-* \
 	| mipsisa64sr71k-* | mipsisa64sr71kel-* \
+	| mipsr5900-* | mipsr5900el-* \
 	| mipstx39-* | mipstx39el-* \
 	| mmix-* \
 	| mt-* \
 	| msp430-* \
-	| nios-* | nios2-* \
+	| nds32-* | nds32le-* | nds32be-* \
+	| nios-* | nios2-* | nios2eb-* | nios2el-* \
 	| none-* | np1-* | ns16k-* | ns32k-* \
+	| open8-* \
+	| or1k*-* \
 	| orion-* \
 	| pdp10-* | pdp11-* | pj-* | pjl-* | pn-* | power-* \
-	| powerpc-* | powerpc64-* | powerpc64le-* | powerpcle-* | ppcbe-* \
+	| powerpc-* | powerpc64-* | powerpc64le-* | powerpcle-* \
 	| pyramid-* \
-	| romp-* | rs6000-* \
-	| sh-* | sh[1234]-* | sh[24]a-* | sh[23]e-* | sh[34]eb-* | sheb-* | shbe-* \
+	| rl78-* | romp-* | rs6000-* | rx-* \
+	| sh-* | sh[1234]-* | sh[24]a-* | sh[24]aeb-* | sh[23]e-* | sh[34]eb-* | sheb-* | shbe-* \
 	| shle-* | sh[1234]le-* | sh3ele-* | sh64-* | sh64le-* \
 	| sparc-* | sparc64-* | sparc64b-* | sparc64v-* | sparc86x-* | sparclet-* \
 	| sparclite-* \
-	| sparcv8-* | sparcv9-* | sparcv9b-* | sparcv9v-* | strongarm-* | sv1-* | sx?-* \
-	| tahoe-* | thumb-* \
+	| sparcv8-* | sparcv9-* | sparcv9b-* | sparcv9v-* | sv1-* | sx?-* \
+	| tahoe-* \
 	| tic30-* | tic4x-* | tic54x-* | tic55x-* | tic6x-* | tic80-* \
+	| tile*-* \
 	| tron-* \
-	| v850-* | v850e-* | vax-* \
+	| ubicom32-* \
+	| v850-* | v850e-* | v850e1-* | v850es-* | v850e2-* | v850e2v3-* \
+	| vax-* \
+	| visium-* \
 	| we32k-* \
-	| x86-* | x86_64-* | xc16x-* | xps100-* | xscale-* | xscalee[bl]-* \
-	| xstormy16-* | xtensa-* \
+	| x86-* | x86_64-* | xc16x-* | xps100-* \
+	| xstormy16-* | xtensa*-* \
 	| ymp-* \
-	| z8k-*)
+	| z8k-* | z80-*)
+		;;
+	# Recognize the basic CPU types without company name, with glob match.
+	xtensa*)
+		basic_machine=$basic_machine-unknown
 		;;
 	# Recognize the various machine names and aliases which stand
 	# for a CPU type and a company and sometimes even an OS.
@@ -389,7 +468,7 @@ case $basic_machine in
 		basic_machine=a29k-amd
 		os=-udi
 		;;
-    	abacus)
+	abacus)
 		basic_machine=abacus-unknown
 		;;
 	adobe68k)
@@ -435,6 +514,10 @@ case $basic_machine in
 		basic_machine=m68k-apollo
 		os=-bsd
 		;;
+	aros)
+		basic_machine=i386-pc
+		os=-aros
+		;;
 	aux)
 		basic_machine=m68k-apple
 		os=-aux
@@ -443,10 +526,35 @@ case $basic_machine in
 		basic_machine=ns32k-sequent
 		os=-dynix
 		;;
+	blackfin)
+		basic_machine=bfin-unknown
+		os=-linux
+		;;
+	blackfin-*)
+		basic_machine=bfin-`echo $basic_machine | sed 's/^[^-]*-//'`
+		os=-linux
+		;;
+	bluegene*)
+		basic_machine=powerpc-ibm
+		os=-cnk
+		;;
+	c54x-*)
+		basic_machine=tic54x-`echo $basic_machine | sed 's/^[^-]*-//'`
+		;;
+	c55x-*)
+		basic_machine=tic55x-`echo $basic_machine | sed 's/^[^-]*-//'`
+		;;
+	c6x-*)
+		basic_machine=tic6x-`echo $basic_machine | sed 's/^[^-]*-//'`
+		;;
 	c90)
 		basic_machine=c90-cray
 		os=-unicos
 		;;
+	cegcc)
+		basic_machine=arm-unknown
+		os=-cegcc
+		;;
 	convex-c1)
 		basic_machine=c1-convex
 		os=-bsd
@@ -475,8 +583,8 @@ case $basic_machine in
 		basic_machine=craynv-cray
 		os=-unicosmp
 		;;
-	cr16c)
-		basic_machine=cr16c-unknown
+	cr16 | cr16-*)
+		basic_machine=cr16-unknown
 		os=-elf
 		;;
 	crds | unos)
@@ -514,6 +622,10 @@ case $basic_machine in
 		basic_machine=m88k-motorola
 		os=-sysv3
 		;;
+	dicos)
+		basic_machine=i686-pc
+		os=-dicos
+		;;
 	djgpp)
 		basic_machine=i586-pc
 		os=-msdosdjgpp
@@ -629,7 +741,6 @@ case $basic_machine in
 	i370-ibm* | ibm*)
 		basic_machine=i370-ibm
 		;;
-# I'm not sure what "Sysv32" means.  Should this be sysv3.2?
 	i*86v32)
 		basic_machine=`echo $1 | sed -e 's/86.*/86-pc/'`
 		os=-sysv32
@@ -668,6 +779,17 @@ case $basic_machine in
 		basic_machine=m68k-isi
 		os=-sysv
 		;;
+	leon-*|leon[3-9]-*)
+		basic_machine=sparc-`echo $basic_machine | sed 's/-.*//'`
+		;;
+	m68knommu)
+		basic_machine=m68k-unknown
+		os=-linux
+		;;
+	m68knommu-*)
+		basic_machine=m68k-`echo $basic_machine | sed 's/^[^-]*-//'`
+		os=-linux
+		;;
 	m88k-omron*)
 		basic_machine=m88k-omron
 		;;
@@ -679,10 +801,21 @@ case $basic_machine in
 		basic_machine=ns32k-utek
 		os=-sysv
 		;;
+	microblaze*)
+		basic_machine=microblaze-xilinx
+		;;
+	mingw64)
+		basic_machine=x86_64-pc
+		os=-mingw64
+		;;
 	mingw32)
-		basic_machine=i386-pc
+		basic_machine=i686-pc
 		os=-mingw32
 		;;
+	mingw32ce)
+		basic_machine=arm-unknown
+		os=-mingw32ce
+		;;
 	miniframe)
 		basic_machine=m68000-convergent
 		;;
@@ -704,6 +837,10 @@ case $basic_machine in
 		basic_machine=powerpc-unknown
 		os=-morphos
 		;;
+	moxiebox)
+		basic_machine=moxie-unknown
+		os=-moxiebox
+		;;
 	msdos)
 		basic_machine=i386-pc
 		os=-msdos
@@ -711,10 +848,18 @@ case $basic_machine in
 	ms1-*)
 		basic_machine=`echo $basic_machine | sed -e 's/ms1-/mt-/'`
 		;;
+	msys)
+		basic_machine=i686-pc
+		os=-msys
+		;;
 	mvs)
 		basic_machine=i370-ibm
 		os=-mvs
 		;;
+	nacl)
+		basic_machine=le32-unknown
+		os=-nacl
+		;;
 	ncr3000)
 		basic_machine=i486-ncr
 		os=-sysv4
@@ -779,6 +924,12 @@ case $basic_machine in
 	np1)
 		basic_machine=np1-gould
 		;;
+	neo-tandem)
+		basic_machine=neo-tandem
+		;;
+	nse-tandem)
+		basic_machine=nse-tandem
+		;;
 	nsr-tandem)
 		basic_machine=nsr-tandem
 		;;
@@ -809,6 +960,14 @@ case $basic_machine in
 		basic_machine=i860-intel
 		os=-osf
 		;;
+	parisc)
+		basic_machine=hppa-unknown
+		os=-linux
+		;;
+	parisc-*)
+		basic_machine=hppa-`echo $basic_machine | sed 's/^[^-]*-//'`
+		os=-linux
+		;;
 	pbd)
 		basic_machine=sparc-tti
 		;;
@@ -853,9 +1012,10 @@ case $basic_machine in
 		;;
 	power)	basic_machine=power-ibm
 		;;
-	ppc)	basic_machine=powerpc-unknown
+	ppc | ppcbe)	basic_machine=powerpc-unknown
 		;;
-	ppc-*)	basic_machine=powerpc-`echo $basic_machine | sed 's/^[^-]*-//'`
+	ppc-* | ppcbe-*)
+		basic_machine=powerpc-`echo $basic_machine | sed 's/^[^-]*-//'`
 		;;
 	ppcle | powerpclittle | ppc-le | powerpc-little)
 		basic_machine=powerpcle-unknown
@@ -880,7 +1040,11 @@ case $basic_machine in
 		basic_machine=i586-unknown
 		os=-pw32
 		;;
-	rdos)
+	rdos | rdos64)
+		basic_machine=x86_64-pc
+		os=-rdos
+		;;
+	rdos32)
 		basic_machine=i386-pc
 		os=-rdos
 		;;
@@ -925,6 +1089,9 @@ case $basic_machine in
 		basic_machine=sh-hitachi
 		os=-hms
 		;;
+	sh5el)
+		basic_machine=sh5le-unknown
+		;;
 	sh64)
 		basic_machine=sh64-unknown
 		;;
@@ -946,6 +1113,9 @@ case $basic_machine in
 		basic_machine=i860-stratus
 		os=-sysv4
 		;;
+	strongarm-* | thumb-*)
+		basic_machine=arm-`echo $basic_machine | sed 's/^[^-]*-//'`
+		;;
 	sun2)
 		basic_machine=m68000-sun
 		;;
@@ -1002,17 +1172,9 @@ case $basic_machine in
 		basic_machine=t90-cray
 		os=-unicos
 		;;
-	tic54x | c54x*)
-		basic_machine=tic54x-unknown
-		os=-coff
-		;;
-	tic55x | c55x*)
-		basic_machine=tic55x-unknown
-		os=-coff
-		;;
-	tic6x | c6x*)
-		basic_machine=tic6x-unknown
-		os=-coff
+	tile*)
+		basic_machine=$basic_machine-unknown
+		os=-linux-gnu
 		;;
 	tx39)
 		basic_machine=mipstx39-unknown
@@ -1081,6 +1243,9 @@ case $basic_machine in
 	xps | xps100)
 		basic_machine=xps100-honeywell
 		;;
+	xscale-* | xscalee[bl]-*)
+		basic_machine=`echo $basic_machine | sed 's/^xscale/arm/'`
+		;;
 	ymp)
 		basic_machine=ymp-cray
 		os=-unicos
@@ -1089,6 +1254,10 @@ case $basic_machine in
 		basic_machine=z8k-unknown
 		os=-sim
 		;;
+	z80-*-coff)
+		basic_machine=z80-unknown
+		os=-sim
+		;;
 	none)
 		basic_machine=none-none
 		os=-none
@@ -1127,7 +1296,7 @@ case $basic_machine in
 	we32k)
 		basic_machine=we32k-att
 		;;
-	sh[1234] | sh[24]a | sh[34]eb | sh[1234]le | sh[23]ele)
+	sh[1234] | sh[24]a | sh[24]aeb | sh[34]eb | sh[1234]le | sh[23]ele)
 		basic_machine=sh-unknown
 		;;
 	sparc | sparcv8 | sparcv9 | sparcv9b | sparcv9v)
@@ -1174,9 +1343,12 @@ esac
 if [ x"$os" != x"" ]
 then
 case $os in
-        # First match some system type aliases
-        # that might get confused with valid system types.
+	# First match some system type aliases
+	# that might get confused with valid system types.
 	# -solaris* is a basic system type, with this one exception.
+	-auroraux)
+		os=-auroraux
+		;;
 	-solaris1 | -solaris1.*)
 		os=`echo $os | sed -e 's|solaris1|sunos4|'`
 		;;
@@ -1197,29 +1369,31 @@ case $os in
 	# Each alternative MUST END IN A *, to match a version number.
 	# -sysv* is not here because it comes later, after sysvr4.
 	-gnu* | -bsd* | -mach* | -minix* | -genix* | -ultrix* | -irix* \
-	      | -*vms* | -sco* | -esix* | -isc* | -aix* | -sunos | -sunos[34]*\
-	      | -hpux* | -unos* | -osf* | -luna* | -dgux* | -solaris* | -sym* \
+	      | -*vms* | -sco* | -esix* | -isc* | -aix* | -cnk* | -sunos | -sunos[34]*\
+	      | -hpux* | -unos* | -osf* | -luna* | -dgux* | -auroraux* | -solaris* \
+	      | -sym* | -kopensolaris* | -plan9* \
 	      | -amigaos* | -amigados* | -msdos* | -newsos* | -unicos* | -aof* \
-	      | -aos* \
+	      | -aos* | -aros* \
 	      | -nindy* | -vxsim* | -vxworks* | -ebmon* | -hms* | -mvs* \
 	      | -clix* | -riscos* | -uniplus* | -iris* | -rtu* | -xenix* \
 	      | -hiux* | -386bsd* | -knetbsd* | -mirbsd* | -netbsd* \
-	      | -openbsd* | -solidbsd* \
+	      | -bitrig* | -openbsd* | -solidbsd* \
 	      | -ekkobsd* | -kfreebsd* | -freebsd* | -riscix* | -lynxos* \
 	      | -bosx* | -nextstep* | -cxux* | -aout* | -elf* | -oabi* \
 	      | -ptx* | -coff* | -ecoff* | -winnt* | -domain* | -vsta* \
 	      | -udi* | -eabi* | -lites* | -ieee* | -go32* | -aux* \
-	      | -chorusos* | -chorusrdb* \
-	      | -cygwin* | -pe* | -psos* | -moss* | -proelf* | -rtems* \
-	      | -mingw32* | -linux-gnu* | -linux-newlib* | -linux-uclibc* \
-	      | -uxpv* | -beos* | -mpeix* | -udk* \
+	      | -chorusos* | -chorusrdb* | -cegcc* \
+	      | -cygwin* | -msys* | -pe* | -psos* | -moss* | -proelf* | -rtems* \
+	      | -mingw32* | -mingw64* | -linux-gnu* | -linux-android* \
+	      | -linux-newlib* | -linux-musl* | -linux-uclibc* \
+	      | -uxpv* | -beos* | -mpeix* | -udk* | -moxiebox* \
 	      | -interix* | -uwin* | -mks* | -rhapsody* | -darwin* | -opened* \
 	      | -openstep* | -oskit* | -conix* | -pw32* | -nonstopux* \
 	      | -storm-chaos* | -tops10* | -tenex* | -tops20* | -its* \
 	      | -os2* | -vos* | -palmos* | -uclinux* | -nucleus* \
 	      | -morphos* | -superux* | -rtmk* | -rtmk-nova* | -windiss* \
 	      | -powermax* | -dnix* | -nx6 | -nx7 | -sei* | -dragonfly* \
-	      | -skyos* | -haiku* | -rdos* | -toppers*)
+	      | -skyos* | -haiku* | -rdos* | -toppers* | -drops* | -es* | -tirtos*)
 	# Remember, each alternative MUST END IN *, to match a version number.
 		;;
 	-qnx*)
@@ -1258,7 +1432,7 @@ case $os in
 	-opened*)
 		os=-openedition
 		;;
-        -os400*)
+	-os400*)
 		os=-os400
 		;;
 	-wince*)
@@ -1307,7 +1481,7 @@ case $os in
 	-sinix*)
 		os=-sysv4
 		;;
-        -tpf*)
+	-tpf*)
 		os=-tpf
 		;;
 	-triton*)
@@ -1343,12 +1517,14 @@ case $os in
 	-aros*)
 		os=-aros
 		;;
-	-kaos*)
-		os=-kaos
-		;;
 	-zvmoe)
 		os=-zvmoe
 		;;
+	-dicos*)
+		os=-dicos
+		;;
+	-nacl*)
+		;;
 	-none)
 		;;
 	*)
@@ -1371,10 +1547,10 @@ else
 # system, and we'll never get to this point.
 
 case $basic_machine in
-        score-*)
+	score-*)
 		os=-elf
 		;;
-        spu-*)
+	spu-*)
 		os=-elf
 		;;
 	*-acorn)
@@ -1386,8 +1562,23 @@ case $basic_machine in
 	arm*-semi)
 		os=-aout
 		;;
-        c4x-* | tic4x-*)
-        	os=-coff
+	c4x-* | tic4x-*)
+		os=-coff
+		;;
+	c8051-*)
+		os=-elf
+		;;
+	hexagon-*)
+		os=-elf
+		;;
+	tic54x-*)
+		os=-coff
+		;;
+	tic55x-*)
+		os=-coff
+		;;
+	tic6x-*)
+		os=-coff
 		;;
 	# This must come before the *-dec entry.
 	pdp10-*)
@@ -1407,13 +1598,13 @@ case $basic_machine in
 		;;
 	m68000-sun)
 		os=-sunos3
-		# This also exists in the configure program, but was not the
-		# default.
-		# os=-sunos4
 		;;
 	m68*-cisco)
 		os=-aout
 		;;
+	mep-*)
+		os=-elf
+		;;
 	mips*-cisco)
 		os=-elf
 		;;
@@ -1438,7 +1629,7 @@ case $basic_machine in
 	*-ibm)
 		os=-aix
 		;;
-    	*-knuth)
+	*-knuth)
 		os=-mmixware
 		;;
 	*-wec)
@@ -1543,7 +1734,7 @@ case $basic_machine in
 			-sunos*)
 				vendor=sun
 				;;
-			-aix*)
+			-cnk*|-aix*)
 				vendor=ibm
 				;;
 			-beos*)

posix/depcomp

@@ -1,7 +1,9 @@
 #! /bin/sh
-
 # depcomp - compile a program generating dependencies as side-effects
-# Copyright 1999, 2000 Free Software Foundation, Inc.
+
+scriptversion=2013-05-30.07; # UTC
+
+# Copyright (C) 1999-2014 Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -14,9 +16,7 @@
 # GNU General Public License for more details.
 
 # You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
-# 02111-1307, USA.
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 # As a special exception to the GNU General Public License, if you
 # distribute this file as part of a program that contains a
@@ -25,17 +25,112 @@
 
 # Originally written by Alexandre Oliva <oliva@dcc.unicamp.br>.
 
+case $1 in
+  '')
+    echo "$0: No command.  Try '$0 --help' for more information." 1>&2
+    exit 1;
+    ;;
+  -h | --h*)
+    cat <<\EOF
+Usage: depcomp [--help] [--version] PROGRAM [ARGS]
+
+Run PROGRAMS ARGS to compile a file, generating dependencies
+as side-effects.
+
+Environment variables:
+  depmode     Dependency tracking mode.
+  source      Source file read by 'PROGRAMS ARGS'.
+  object      Object file output by 'PROGRAMS ARGS'.
+  DEPDIR      directory where to store dependencies.
+  depfile     Dependency file to output.
+  tmpdepfile  Temporary file to use when outputting dependencies.
+  libtool     Whether libtool is used (yes/no).
+
+Report bugs to <bug-automake@gnu.org>.
+EOF
+    exit $?
+    ;;
+  -v | --v*)
+    echo "depcomp $scriptversion"
+    exit $?
+    ;;
+esac
+
+# Get the directory component of the given path, and save it in the
+# global variables '$dir'.  Note that this directory component will
+# be either empty or ending with a '/' character.  This is deliberate.
+set_dir_from ()
+{
+  case $1 in
+    */*) dir=`echo "$1" | sed -e 's|/[^/]*$|/|'`;;
+      *) dir=;;
+  esac
+}
+
+# Get the suffix-stripped basename of the given path, and save it the
+# global variable '$base'.
+set_base_from ()
+{
+  base=`echo "$1" | sed -e 's|^.*/||' -e 's/\.[^.]*$//'`
+}
+
+# If no dependency file was actually created by the compiler invocation,
+# we still have to create a dummy depfile, to avoid errors with the
+# Makefile "include basename.Plo" scheme.
+make_dummy_depfile ()
+{
+  echo "#dummy" > "$depfile"
+}
+
+# Factor out some common post-processing of the generated depfile.
+# Requires the auxiliary global variable '$tmpdepfile' to be set.
+aix_post_process_depfile ()
+{
+  # If the compiler actually managed to produce a dependency file,
+  # post-process it.
+  if test -f "$tmpdepfile"; then
+    # Each line is of the form 'foo.o: dependency.h'.
+    # Do two passes, one to just change these to
+    #   $object: dependency.h
+    # and one to simply output
+    #   dependency.h:
+    # which is needed to avoid the deleted-header problem.
+    { sed -e "s,^.*\.[$lower]*:,$object:," < "$tmpdepfile"
+      sed -e "s,^.*\.[$lower]*:[$tab ]*,," -e 's,$,:,' < "$tmpdepfile"
+    } > "$depfile"
+    rm -f "$tmpdepfile"
+  else
+    make_dummy_depfile
+  fi
+}
+
+# A tabulation character.
+tab='	'
+# A newline character.
+nl='
+'
+# Character ranges might be problematic outside the C locale.
+# These definitions help.
+upper=ABCDEFGHIJKLMNOPQRSTUVWXYZ
+lower=abcdefghijklmnopqrstuvwxyz
+digits=0123456789
+alpha=${upper}${lower}
+
 if test -z "$depmode" || test -z "$source" || test -z "$object"; then
   echo "depcomp: Variables source, object and depmode must be set" 1>&2
   exit 1
 fi
-# `libtool' can also be set to `yes' or `no'.
 
-depfile=${depfile-`echo "$object" | sed 's,\([^/]*\)$,.deps/\1,;s/\.\([^.]*\)$/.P\1/'`}
+# Dependencies for sub/bar.o or sub/bar.obj go into sub/.deps/bar.Po.
+depfile=${depfile-`echo "$object" |
+  sed 's|[^\\/]*$|'${DEPDIR-.deps}'/&|;s|\.\([^.]*\)$|.P\1|;s|Pobj$|Po|'`}
 tmpdepfile=${tmpdepfile-`echo "$depfile" | sed 's/\.\([^.]*\)$/.T\1/'`}
 
 rm -f "$tmpdepfile"
 
+# Avoid interferences from the environment.
+gccflag= dashmflag=
+
 # Some modes work just like other modes, but use different flags.  We
 # parameterize here, but still list the modes in the big case below,
 # to make depend.m4 easier to write.  Note that we *cannot* use a case
@@ -47,9 +142,32 @@ if test "$depmode" = hp; then
 fi
 
 if test "$depmode" = dashXmstdout; then
-   # This is just like dashmstdout with a different argument.
-   dashmflag=-xM
-   depmode=dashmstdout
+  # This is just like dashmstdout with a different argument.
+  dashmflag=-xM
+  depmode=dashmstdout
+fi
+
+cygpath_u="cygpath -u -f -"
+if test "$depmode" = msvcmsys; then
+  # This is just like msvisualcpp but w/o cygpath translation.
+  # Just convert the backslash-escaped backslashes to single forward
+  # slashes to satisfy depend.m4
+  cygpath_u='sed s,\\\\,/,g'
+  depmode=msvisualcpp
+fi
+
+if test "$depmode" = msvc7msys; then
+  # This is just like msvc7 but w/o cygpath translation.
+  # Just convert the backslash-escaped backslashes to single forward
+  # slashes to satisfy depend.m4
+  cygpath_u='sed s,\\\\,/,g'
+  depmode=msvc7
+fi
+
+if test "$depmode" = xlc; then
+  # IBM C/C++ Compilers xlc/xlC can output gcc-like dependency information.
+  gccflag=-qmakedep=gcc,-MF
+  depmode=gcc
 fi
 
 case "$depmode" in
@@ -57,10 +175,22 @@ gcc3)
 ## gcc 3 implements dependency tracking that does exactly what
 ## we want.  Yay!  Note: for some reason libtool 1.4 doesn't like
 ## it if -MD -MP comes after the -MF stuff.  Hmm.
-  "$@" -MT "$object" -MD -MP -MF "$tmpdepfile"
+## Unfortunately, FreeBSD c89 acceptance of flags depends upon
+## the command line argument order; so add the flags where they
+## appear in depend2.am.  Note that the slowdown incurred here
+## affects only configure: in makefiles, %FASTDEP% shortcuts this.
+  for arg
+  do
+    case $arg in
+    -c) set fnord "$@" -MT "$object" -MD -MP -MF "$tmpdepfile" "$arg" ;;
+    *)  set fnord "$@" "$arg" ;;
+    esac
+    shift # fnord
+    shift # $arg
+  done
+  "$@"
   stat=$?
-  if test $stat -eq 0; then :
-  else
+  if test $stat -ne 0; then
     rm -f "$tmpdepfile"
     exit $stat
   fi
@@ -68,13 +198,17 @@ gcc3)
   ;;
 
 gcc)
+## Note that this doesn't just cater to obsosete pre-3.x GCC compilers.
+## but also to in-use compilers like IMB xlc/xlC and the HP C compiler.
+## (see the conditional assignment to $gccflag above).
 ## There are various ways to get dependency output from gcc.  Here's
 ## why we pick this rather obscure method:
 ## - Don't want to use -MD because we'd like the dependencies to end
 ##   up in a subdir.  Having to rename by hand is ugly.
 ##   (We might end up doing this anyway to support other compilers.)
 ## - The DEPENDENCIES_OUTPUT environment variable makes gcc act like
-##   -MM, not -M (despite what the docs say).
+##   -MM, not -M (despite what the docs say).  Also, it might not be
+##   supported by the other compilers which use the 'gcc' depmode.
 ## - Using -M directly means running the compiler twice (even worse
 ##   than renaming).
   if test -z "$gccflag"; then
@@ -82,31 +216,31 @@ gcc)
   fi
   "$@" -Wp,"$gccflag$tmpdepfile"
   stat=$?
-  if test $stat -eq 0; then :
-  else
+  if test $stat -ne 0; then
     rm -f "$tmpdepfile"
     exit $stat
   fi
   rm -f "$depfile"
   echo "$object : \\" > "$depfile"
-  alpha=ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
-## The second -e expression handles DOS-style file names with drive letters.
+  # The second -e expression handles DOS-style file names with drive
+  # letters.
   sed -e 's/^[^:]*: / /' \
       -e 's/^['$alpha']:\/[^:]*: / /' < "$tmpdepfile" >> "$depfile"
-## This next piece of magic avoids the `deleted header file' problem.
+## This next piece of magic avoids the "deleted header file" problem.
 ## The problem is that when a header file which appears in a .P file
 ## is deleted, the dependency causes make to die (because there is
 ## typically no way to rebuild the header).  We avoid this by adding
 ## dummy dependencies for each header file.  Too bad gcc doesn't do
 ## this for us directly.
-  tr ' ' '
-' < "$tmpdepfile" |
-## Some versions of gcc put a space before the `:'.  On the theory
+## Some versions of gcc put a space before the ':'.  On the theory
 ## that the space means something, we add a space to the output as
-## well.
+## well.  hp depmode also adds that space, but also prefixes the VPATH
+## to the object.  Take care to not repeat it in the output.
 ## Some versions of the HPUX 10.20 sed can't process this invocation
 ## correctly.  Breaking it into two sed invocations is a workaround.
-    sed -e 's/^\\$//' -e '/^$/d' -e '/:$/d' | sed -e 's/$/ :/' >> "$depfile"
+  tr ' ' "$nl" < "$tmpdepfile" \
+    | sed -e 's/^\\$//' -e '/^$/d' -e "s|.*$object$||" -e '/:$/d' \
+    | sed -e 's/$/ :/' >> "$depfile"
   rm -f "$tmpdepfile"
   ;;
 
@@ -124,8 +258,7 @@ sgi)
     "$@" -MDupdate "$tmpdepfile"
   fi
   stat=$?
-  if test $stat -eq 0; then :
-  else
+  if test $stat -ne 0; then
     rm -f "$tmpdepfile"
     exit $stat
   fi
@@ -133,136 +266,302 @@ sgi)
 
   if test -f "$tmpdepfile"; then  # yes, the sourcefile depend on other files
     echo "$object : \\" > "$depfile"
-
     # Clip off the initial element (the dependent).  Don't try to be
     # clever and replace this with sed code, as IRIX sed won't handle
     # lines with more than a fixed number of characters (4096 in
     # IRIX 6.2 sed, 8192 in IRIX 6.5).  We also remove comment lines;
-    # the IRIX cc adds comments like `#:fec' to the end of the
+    # the IRIX cc adds comments like '#:fec' to the end of the
     # dependency line.
-    tr ' ' '
-' < "$tmpdepfile" \
-    | sed -e 's/^.*\.o://' -e 's/#.*$//' -e '/^$/ d' | \
-    tr '
-' ' ' >> $depfile
-    echo >> $depfile
-
+    tr ' ' "$nl" < "$tmpdepfile" \
+      | sed -e 's/^.*\.o://' -e 's/#.*$//' -e '/^$/ d' \
+      | tr "$nl" ' ' >> "$depfile"
+    echo >> "$depfile"
     # The second pass generates a dummy entry for each header file.
-    tr ' ' '
-' < "$tmpdepfile" \
-   | sed -e 's/^.*\.o://' -e 's/#.*$//' -e '/^$/ d' -e 's/$/:/' \
-   >> $depfile
+    tr ' ' "$nl" < "$tmpdepfile" \
+      | sed -e 's/^.*\.o://' -e 's/#.*$//' -e '/^$/ d' -e 's/$/:/' \
+      >> "$depfile"
   else
-    # The sourcefile does not contain any dependencies, so just
-    # store a dummy comment line, to avoid errors with the Makefile
-    # "include basename.Plo" scheme.
-    echo "#dummy" > "$depfile"
+    make_dummy_depfile
   fi
   rm -f "$tmpdepfile"
   ;;
 
+xlc)
+  # This case exists only to let depend.m4 do its work.  It works by
+  # looking at the text of this script.  This case will never be run,
+  # since it is checked for above.
+  exit 1
+  ;;
+
 aix)
   # The C for AIX Compiler uses -M and outputs the dependencies
-  # in a .u file.  This file always lives in the current directory.
-  # Also, the AIX compiler puts `$object:' at the start of each line;
-  # $object doesn't have directory information.
-  stripped=`echo "$object" | sed -e 's,^.*/,,' -e 's/\(.*\)\..*$/\1/'`
-  tmpdepfile="$stripped.u"
-  outname="$stripped.o"
+  # in a .u file.  In older versions, this file always lives in the
+  # current directory.  Also, the AIX compiler puts '$object:' at the
+  # start of each line; $object doesn't have directory information.
+  # Version 6 uses the directory in both cases.
+  set_dir_from "$object"
+  set_base_from "$object"
   if test "$libtool" = yes; then
+    tmpdepfile1=$dir$base.u
+    tmpdepfile2=$base.u
+    tmpdepfile3=$dir.libs/$base.u
     "$@" -Wc,-M
   else
+    tmpdepfile1=$dir$base.u
+    tmpdepfile2=$dir$base.u
+    tmpdepfile3=$dir$base.u
     "$@" -M
   fi
-
   stat=$?
-  if test $stat -eq 0; then :
-  else
-    rm -f "$tmpdepfile"
+  if test $stat -ne 0; then
+    rm -f "$tmpdepfile1" "$tmpdepfile2" "$tmpdepfile3"
     exit $stat
   fi
 
-  if test -f "$tmpdepfile"; then
-    # Each line is of the form `foo.o: dependent.h'.
-    # Do two passes, one to just change these to
-    # `$object: dependent.h' and one to simply `dependent.h:'.
-    sed -e "s,^$outname:,$object :," < "$tmpdepfile" > "$depfile"
-    sed -e "s,^$outname: \(.*\)$,\1:," < "$tmpdepfile" >> "$depfile"
-  else
-    # The sourcefile does not contain any dependencies, so just
-    # store a dummy comment line, to avoid errors with the Makefile
-    # "include basename.Plo" scheme.
-    echo "#dummy" > "$depfile"
-  fi
-  rm -f "$tmpdepfile"
+  for tmpdepfile in "$tmpdepfile1" "$tmpdepfile2" "$tmpdepfile3"
+  do
+    test -f "$tmpdepfile" && break
+  done
+  aix_post_process_depfile
   ;;
 
-icc)
-  # Must come before tru64.
-
-  # Intel's C compiler understands `-MD -MF file'.  However
-  #    icc -MD -MF foo.d -c -o sub/foo.o sub/foo.c
-  # will fill foo.d with something like
-  #    foo.o: sub/foo.c
-  #    foo.o: sub/foo.h
-  # which is wrong.  We want:
-  #    sub/foo.o: sub/foo.c
-  #    sub/foo.o: sub/foo.h
-  #    sub/foo.c:
-  #    sub/foo.h:
-
+tcc)
+  # tcc (Tiny C Compiler) understand '-MD -MF file' since version 0.9.26
+  # FIXME: That version still under development at the moment of writing.
+  #        Make that this statement remains true also for stable, released
+  #        versions.
+  # It will wrap lines (doesn't matter whether long or short) with a
+  # trailing '\', as in:
+  #
+  #   foo.o : \
+  #    foo.c \
+  #    foo.h \
+  #
+  # It will put a trailing '\' even on the last line, and will use leading
+  # spaces rather than leading tabs (at least since its commit 0394caf7
+  # "Emit spaces for -MD").
   "$@" -MD -MF "$tmpdepfile"
   stat=$?
-  if test $stat -eq 0; then :
-  else
+  if test $stat -ne 0; then
     rm -f "$tmpdepfile"
     exit $stat
   fi
   rm -f "$depfile"
-  # Each line is of the form `foo.o: dependent.h'.
-  # Do two passes, one to just change these to
-  # `$object: dependent.h' and one to simply `dependent.h:'.
-  sed -e "s,^[^:]*:,$object :," < "$tmpdepfile" > "$depfile"
-  sed -e "s,^[^:]*: \(.*\)$,\1:," < "$tmpdepfile" >> "$depfile"
+  # Each non-empty line is of the form 'foo.o : \' or ' dep.h \'.
+  # We have to change lines of the first kind to '$object: \'.
+  sed -e "s|.*:|$object :|" < "$tmpdepfile" > "$depfile"
+  # And for each line of the second kind, we have to emit a 'dep.h:'
+  # dummy dependency, to avoid the deleted-header problem.
+  sed -n -e 's|^  *\(.*\) *\\$|\1:|p' < "$tmpdepfile" >> "$depfile"
   rm -f "$tmpdepfile"
   ;;
 
-tru64)
-   # The Tru64 AIX compiler uses -MD to generate dependencies as a side
-   # effect.  `cc -MD -o foo.o ...' puts the dependencies into `foo.o.d'.
-   # At least on Alpha/Redhat 6.1, Compaq CCC V6.2-504 seems to put 
-   # dependencies in `foo.d' instead, so we check for that too.
-   # Subdirectories are respected.
+## The order of this option in the case statement is important, since the
+## shell code in configure will try each of these formats in the order
+## listed in this file.  A plain '-MD' option would be understood by many
+## compilers, so we must ensure this comes after the gcc and icc options.
+pgcc)
+  # Portland's C compiler understands '-MD'.
+  # Will always output deps to 'file.d' where file is the root name of the
+  # source file under compilation, even if file resides in a subdirectory.
+  # The object file name does not affect the name of the '.d' file.
+  # pgcc 10.2 will output
+  #    foo.o: sub/foo.c sub/foo.h
+  # and will wrap long lines using '\' :
+  #    foo.o: sub/foo.c ... \
+  #     sub/foo.h ... \
+  #     ...
+  set_dir_from "$object"
+  # Use the source, not the object, to determine the base name, since
+  # that's sadly what pgcc will do too.
+  set_base_from "$source"
+  tmpdepfile=$base.d
 
-   tmpdepfile1="$object.d"
-   tmpdepfile2=`echo "$object" | sed -e 's/.o$/.d/'` 
-   if test "$libtool" = yes; then
-      "$@" -Wc,-MD
-   else
+  # For projects that build the same source file twice into different object
+  # files, the pgcc approach of using the *source* file root name can cause
+  # problems in parallel builds.  Use a locking strategy to avoid stomping on
+  # the same $tmpdepfile.
+  lockdir=$base.d-lock
+  trap "
+    echo '$0: caught signal, cleaning up...' >&2
+    rmdir '$lockdir'
+    exit 1
+  " 1 2 13 15
+  numtries=100
+  i=$numtries
+  while test $i -gt 0; do
+    # mkdir is a portable test-and-set.
+    if mkdir "$lockdir" 2>/dev/null; then
+      # This process acquired the lock.
       "$@" -MD
-   fi
+      stat=$?
+      # Release the lock.
+      rmdir "$lockdir"
+      break
+    else
+      # If the lock is being held by a different process, wait
+      # until the winning process is done or we timeout.
+      while test -d "$lockdir" && test $i -gt 0; do
+        sleep 1
+        i=`expr $i - 1`
+      done
+    fi
+    i=`expr $i - 1`
+  done
+  trap - 1 2 13 15
+  if test $i -le 0; then
+    echo "$0: failed to acquire lock after $numtries attempts" >&2
+    echo "$0: check lockdir '$lockdir'" >&2
+    exit 1
+  fi
 
-   stat=$?
-   if test $stat -eq 0; then :
-   else
-      rm -f "$tmpdepfile1" "$tmpdepfile2"
-      exit $stat
-   fi
+  if test $stat -ne 0; then
+    rm -f "$tmpdepfile"
+    exit $stat
+  fi
+  rm -f "$depfile"
+  # Each line is of the form `foo.o: dependent.h',
+  # or `foo.o: dep1.h dep2.h \', or ` dep3.h dep4.h \'.
+  # Do two passes, one to just change these to
+  # `$object: dependent.h' and one to simply `dependent.h:'.
+  sed "s,^[^:]*:,$object :," < "$tmpdepfile" > "$depfile"
+  # Some versions of the HPUX 10.20 sed can't process this invocation
+  # correctly.  Breaking it into two sed invocations is a workaround.
+  sed 's,^[^:]*: \(.*\)$,\1,;s/^\\$//;/^$/d;/:$/d' < "$tmpdepfile" \
+    | sed -e 's/$/ :/' >> "$depfile"
+  rm -f "$tmpdepfile"
+  ;;
 
-   if test -f "$tmpdepfile1"; then
-      tmpdepfile="$tmpdepfile1"
-   else
-      tmpdepfile="$tmpdepfile2"
-   fi
-   if test -f "$tmpdepfile"; then
-      sed -e "s,^.*\.[a-z]*:,$object:," < "$tmpdepfile" > "$depfile"
-      # That's a space and a tab in the [].
-      sed -e 's,^.*\.[a-z]*:[ 	]*,,' -e 's,$,:,' < "$tmpdepfile" >> "$depfile"
-   else
-      echo "#dummy" > "$depfile"
-   fi
-   rm -f "$tmpdepfile"
-   ;;
+hp2)
+  # The "hp" stanza above does not work with aCC (C++) and HP's ia64
+  # compilers, which have integrated preprocessors.  The correct option
+  # to use with these is +Maked; it writes dependencies to a file named
+  # 'foo.d', which lands next to the object file, wherever that
+  # happens to be.
+  # Much of this is similar to the tru64 case; see comments there.
+  set_dir_from  "$object"
+  set_base_from "$object"
+  if test "$libtool" = yes; then
+    tmpdepfile1=$dir$base.d
+    tmpdepfile2=$dir.libs/$base.d
+    "$@" -Wc,+Maked
+  else
+    tmpdepfile1=$dir$base.d
+    tmpdepfile2=$dir$base.d
+    "$@" +Maked
+  fi
+  stat=$?
+  if test $stat -ne 0; then
+     rm -f "$tmpdepfile1" "$tmpdepfile2"
+     exit $stat
+  fi
+
+  for tmpdepfile in "$tmpdepfile1" "$tmpdepfile2"
+  do
+    test -f "$tmpdepfile" && break
+  done
+  if test -f "$tmpdepfile"; then
+    sed -e "s,^.*\.[$lower]*:,$object:," "$tmpdepfile" > "$depfile"
+    # Add 'dependent.h:' lines.
+    sed -ne '2,${
+               s/^ *//
+               s/ \\*$//
+               s/$/:/
+               p
+             }' "$tmpdepfile" >> "$depfile"
+  else
+    make_dummy_depfile
+  fi
+  rm -f "$tmpdepfile" "$tmpdepfile2"
+  ;;
+
+tru64)
+  # The Tru64 compiler uses -MD to generate dependencies as a side
+  # effect.  'cc -MD -o foo.o ...' puts the dependencies into 'foo.o.d'.
+  # At least on Alpha/Redhat 6.1, Compaq CCC V6.2-504 seems to put
+  # dependencies in 'foo.d' instead, so we check for that too.
+  # Subdirectories are respected.
+  set_dir_from  "$object"
+  set_base_from "$object"
+
+  if test "$libtool" = yes; then
+    # Libtool generates 2 separate objects for the 2 libraries.  These
+    # two compilations output dependencies in $dir.libs/$base.o.d and
+    # in $dir$base.o.d.  We have to check for both files, because
+    # one of the two compilations can be disabled.  We should prefer
+    # $dir$base.o.d over $dir.libs/$base.o.d because the latter is
+    # automatically cleaned when .libs/ is deleted, while ignoring
+    # the former would cause a distcleancheck panic.
+    tmpdepfile1=$dir$base.o.d          # libtool 1.5
+    tmpdepfile2=$dir.libs/$base.o.d    # Likewise.
+    tmpdepfile3=$dir.libs/$base.d      # Compaq CCC V6.2-504
+    "$@" -Wc,-MD
+  else
+    tmpdepfile1=$dir$base.d
+    tmpdepfile2=$dir$base.d
+    tmpdepfile3=$dir$base.d
+    "$@" -MD
+  fi
+
+  stat=$?
+  if test $stat -ne 0; then
+    rm -f "$tmpdepfile1" "$tmpdepfile2" "$tmpdepfile3"
+    exit $stat
+  fi
+
+  for tmpdepfile in "$tmpdepfile1" "$tmpdepfile2" "$tmpdepfile3"
+  do
+    test -f "$tmpdepfile" && break
+  done
+  # Same post-processing that is required for AIX mode.
+  aix_post_process_depfile
+  ;;
+
+msvc7)
+  if test "$libtool" = yes; then
+    showIncludes=-Wc,-showIncludes
+  else
+    showIncludes=-showIncludes
+  fi
+  "$@" $showIncludes > "$tmpdepfile"
+  stat=$?
+  grep -v '^Note: including file: ' "$tmpdepfile"
+  if test $stat -ne 0; then
+    rm -f "$tmpdepfile"
+    exit $stat
+  fi
+  rm -f "$depfile"
+  echo "$object : \\" > "$depfile"
+  # The first sed program below extracts the file names and escapes
+  # backslashes for cygpath.  The second sed program outputs the file
+  # name when reading, but also accumulates all include files in the
+  # hold buffer in order to output them again at the end.  This only
+  # works with sed implementations that can handle large buffers.
+  sed < "$tmpdepfile" -n '
+/^Note: including file:  *\(.*\)/ {
+  s//\1/
+  s/\\/\\\\/g
+  p
+}' | $cygpath_u | sort -u | sed -n '
+s/ /\\ /g
+s/\(.*\)/'"$tab"'\1 \\/p
+s/.\(.*\) \\/\1:/
+H
+$ {
+  s/.*/'"$tab"'/
+  G
+  p
+}' >> "$depfile"
+  echo >> "$depfile" # make sure the fragment doesn't end with a backslash
+  rm -f "$tmpdepfile"
+  ;;
+
+msvc7msys)
+  # This case exists only to let depend.m4 do its work.  It works by
+  # looking at the text of this script.  This case will never be run,
+  # since it is checked for above.
+  exit 1
+  ;;
 
 #nosideeffect)
   # This comment above is used by automake to tell side-effect
@@ -270,41 +569,49 @@ tru64)
 
 dashmstdout)
   # Important note: in order to support this mode, a compiler *must*
-  # always write the proprocessed file to stdout, regardless of -o,
-  # because we must use -o when running libtool.
-  test -z "$dashmflag" && dashmflag=-M
-  ( IFS=" "
-    case " $* " in
-    *" --mode=compile "*) # this is libtool, let us make it quiet
-      for arg
-      do # cycle over the arguments
-        case "$arg" in
-	"--mode=compile")
-	  # insert --quiet before "--mode=compile"
-	  set fnord "$@" --quiet
-	  shift # fnord
-	  ;;
-	esac
-	set fnord "$@" "$arg"
-	shift # fnord
-	shift # "$arg"
-      done
+  # always write the preprocessed file to stdout, regardless of -o.
+  "$@" || exit $?
+
+  # Remove the call to Libtool.
+  if test "$libtool" = yes; then
+    while test "X$1" != 'X--mode=compile'; do
+      shift
+    done
+    shift
+  fi
+
+  # Remove '-o $object'.
+  IFS=" "
+  for arg
+  do
+    case $arg in
+    -o)
+      shift
+      ;;
+    $object)
+      shift
+      ;;
+    *)
+      set fnord "$@" "$arg"
+      shift # fnord
+      shift # $arg
       ;;
     esac
-    "$@" $dashmflag | sed 's:^[^:]*\:[ 	]*:'"$object"'\: :' > "$tmpdepfile"
-  ) &
-  proc=$!
-  "$@"
-  stat=$?
-  wait "$proc"
-  if test "$stat" != 0; then exit $stat; fi
+  done
+
+  test -z "$dashmflag" && dashmflag=-M
+  # Require at least two characters before searching for ':'
+  # in the target name.  This is to cope with DOS-style filenames:
+  # a dependency such as 'c:/foo/bar' could be seen as target 'c' otherwise.
+  "$@" $dashmflag |
+    sed "s|^[$tab ]*[^:$tab ][^:][^:]*:[$tab ]*|$object: |" > "$tmpdepfile"
   rm -f "$depfile"
   cat < "$tmpdepfile" > "$depfile"
-  tr ' ' '
-' < "$tmpdepfile" | \
-## Some versions of the HPUX 10.20 sed can't process this invocation
-## correctly.  Breaking it into two sed invocations is a workaround.
-    sed -e 's/^\\$//' -e '/^$/d' -e '/:$/d' | sed -e 's/$/ :/' >> "$depfile"
+  # Some versions of the HPUX 10.20 sed can't process this sed invocation
+  # correctly.  Breaking it into two sed invocations is a workaround.
+  tr ' ' "$nl" < "$tmpdepfile" \
+    | sed -e 's/^\\$//' -e '/^$/d' -e '/:$/d' \
+    | sed -e 's/$/ :/' >> "$depfile"
   rm -f "$tmpdepfile"
   ;;
 
@@ -315,74 +622,93 @@ dashXmstdout)
   ;;
 
 makedepend)
-  # X makedepend
-  (
-    shift
-    cleared=no
-    for arg in "$@"; do
-      case $cleared in no)
-        set ""; shift
-	cleared=yes
-      esac
-      case "$arg" in
-        -D*|-I*)
-	  set fnord "$@" "$arg"; shift;;
-	-*)
-	  ;;
-	*)
-	  set fnord "$@" "$arg"; shift;;
-      esac
+  "$@" || exit $?
+  # Remove any Libtool call
+  if test "$libtool" = yes; then
+    while test "X$1" != 'X--mode=compile'; do
+      shift
     done
-    obj_suffix="`echo $object | sed 's/^.*\././'`"
-    touch "$tmpdepfile"
-    ${MAKEDEPEND-makedepend} 2>/dev/null -o"$obj_suffix" -f"$tmpdepfile" "$@"
-  ) &
-  proc=$!
-  "$@"
-  stat=$?
-  wait "$proc"
-  if test "$stat" != 0; then exit $stat; fi
+    shift
+  fi
+  # X makedepend
+  shift
+  cleared=no eat=no
+  for arg
+  do
+    case $cleared in
+    no)
+      set ""; shift
+      cleared=yes ;;
+    esac
+    if test $eat = yes; then
+      eat=no
+      continue
+    fi
+    case "$arg" in
+    -D*|-I*)
+      set fnord "$@" "$arg"; shift ;;
+    # Strip any option that makedepend may not understand.  Remove
+    # the object too, otherwise makedepend will parse it as a source file.
+    -arch)
+      eat=yes ;;
+    -*|$object)
+      ;;
+    *)
+      set fnord "$@" "$arg"; shift ;;
+    esac
+  done
+  obj_suffix=`echo "$object" | sed 's/^.*\././'`
+  touch "$tmpdepfile"
+  ${MAKEDEPEND-makedepend} -o"$obj_suffix" -f"$tmpdepfile" "$@"
   rm -f "$depfile"
-  cat < "$tmpdepfile" > "$depfile"
-  tail +3 "$tmpdepfile" | tr ' ' '
-' | \
-## Some versions of the HPUX 10.20 sed can't process this invocation
-## correctly.  Breaking it into two sed invocations is a workaround.
-    sed -e 's/^\\$//' -e '/^$/d' -e '/:$/d' | sed -e 's/$/ :/' >> "$depfile"
+  # makedepend may prepend the VPATH from the source file name to the object.
+  # No need to regex-escape $object, excess matching of '.' is harmless.
+  sed "s|^.*\($object *:\)|\1|" "$tmpdepfile" > "$depfile"
+  # Some versions of the HPUX 10.20 sed can't process the last invocation
+  # correctly.  Breaking it into two sed invocations is a workaround.
+  sed '1,2d' "$tmpdepfile" \
+    | tr ' ' "$nl" \
+    | sed -e 's/^\\$//' -e '/^$/d' -e '/:$/d' \
+    | sed -e 's/$/ :/' >> "$depfile"
   rm -f "$tmpdepfile" "$tmpdepfile".bak
   ;;
 
 cpp)
   # Important note: in order to support this mode, a compiler *must*
-  # always write the proprocessed file to stdout, regardless of -o,
-  # because we must use -o when running libtool.
-  ( IFS=" "
-    case " $* " in
-    *" --mode=compile "*)
-      for arg
-      do # cycle over the arguments
-        case $arg in
-	"--mode=compile")
-	  # insert --quiet before "--mode=compile"
-	  set fnord "$@" --quiet
-	  shift # fnord
-	  ;;
-	esac
-	set fnord "$@" "$arg"
-	shift # fnord
-	shift # "$arg"
-      done
+  # always write the preprocessed file to stdout.
+  "$@" || exit $?
+
+  # Remove the call to Libtool.
+  if test "$libtool" = yes; then
+    while test "X$1" != 'X--mode=compile'; do
+      shift
+    done
+    shift
+  fi
+
+  # Remove '-o $object'.
+  IFS=" "
+  for arg
+  do
+    case $arg in
+    -o)
+      shift
+      ;;
+    $object)
+      shift
+      ;;
+    *)
+      set fnord "$@" "$arg"
+      shift # fnord
+      shift # $arg
       ;;
     esac
-    "$@" -E |
-    sed -n '/^# [0-9][0-9]* "\([^"]*\)".*/ s:: \1 \\:p' |
-    sed '$ s: \\$::' > "$tmpdepfile"
-  ) &
-  proc=$!
-  "$@"
-  stat=$?
-  wait "$proc"
-  if test "$stat" != 0; then exit $stat; fi
+  done
+
+  "$@" -E \
+    | sed -n -e '/^# [0-9][0-9]* "\([^"]*\)".*/ s:: \1 \\:p' \
+             -e '/^#line [0-9][0-9]* "\([^"]*\)".*/ s:: \1 \\:p' \
+    | sed '$ s: \\$::' > "$tmpdepfile"
   rm -f "$depfile"
   echo "$object : \\" > "$depfile"
   cat < "$tmpdepfile" >> "$depfile"
@@ -392,42 +718,56 @@ cpp)
 
 msvisualcpp)
   # Important note: in order to support this mode, a compiler *must*
-  # always write the proprocessed file to stdout, regardless of -o,
-  # because we must use -o when running libtool.
-  ( IFS=" "
-    case " $* " in
-    *" --mode=compile "*)
-      for arg
-      do # cycle over the arguments
-        case $arg in
-	"--mode=compile")
-	  # insert --quiet before "--mode=compile"
-	  set fnord "$@" --quiet
-	  shift # fnord
-	  ;;
-	esac
-	set fnord "$@" "$arg"
-	shift # fnord
-	shift # "$arg"
-      done
+  # always write the preprocessed file to stdout.
+  "$@" || exit $?
+
+  # Remove the call to Libtool.
+  if test "$libtool" = yes; then
+    while test "X$1" != 'X--mode=compile'; do
+      shift
+    done
+    shift
+  fi
+
+  IFS=" "
+  for arg
+  do
+    case "$arg" in
+    -o)
+      shift
       ;;
+    $object)
+      shift
+      ;;
+    "-Gm"|"/Gm"|"-Gi"|"/Gi"|"-ZI"|"/ZI")
+        set fnord "$@"
+        shift
+        shift
+        ;;
+    *)
+        set fnord "$@" "$arg"
+        shift
+        shift
+        ;;
     esac
-    "$@" -E |
-    sed -n '/^#line [0-9][0-9]* "\([^"]*\)"/ s::echo "`cygpath -u \\"\1\\"`":p' | sort | uniq > "$tmpdepfile"
-  ) &
-  proc=$!
-  "$@"
-  stat=$?
-  wait "$proc"
-  if test "$stat" != 0; then exit $stat; fi
+  done
+  "$@" -E 2>/dev/null |
+  sed -n '/^#line [0-9][0-9]* "\([^"]*\)"/ s::\1:p' | $cygpath_u | sort -u > "$tmpdepfile"
   rm -f "$depfile"
   echo "$object : \\" > "$depfile"
-  . "$tmpdepfile" | sed 's% %\\ %g' | sed -n '/^\(.*\)$/ s::	\1 \\:p' >> "$depfile"
-  echo "	" >> "$depfile"
-  . "$tmpdepfile" | sed 's% %\\ %g' | sed -n '/^\(.*\)$/ s::\1\::p' >> "$depfile"
+  sed < "$tmpdepfile" -n -e 's% %\\ %g' -e '/^\(.*\)$/ s::'"$tab"'\1 \\:p' >> "$depfile"
+  echo "$tab" >> "$depfile"
+  sed < "$tmpdepfile" -n -e 's% %\\ %g' -e '/^\(.*\)$/ s::\1\::p' >> "$depfile"
   rm -f "$tmpdepfile"
   ;;
 
+msvcmsys)
+  # This case exists only to let depend.m4 do its work.  It works by
+  # looking at the text of this script.  This case will never be run,
+  # since it is checked for above.
+  exit 1
+  ;;
+
 none)
   exec "$@"
   ;;
@@ -439,3 +779,13 @@ none)
 esac
 
 exit 0
+
+# Local Variables:
+# mode: shell-script
+# sh-indentation: 2
+# eval: (add-hook 'write-file-hooks 'time-stamp)
+# time-stamp-start: "scriptversion="
+# time-stamp-format: "%:y-%02m-%02d.%02H"
+# time-stamp-time-zone: "UTC"
+# time-stamp-end: "; # UTC"
+# End:

posix/install-sh

@@ -1,276 +1,501 @@
 #!/bin/sh
-#
 # install - install a program, script, or datafile
-# This comes from X11R5 (mit/util/scripts/install.sh).
+
+scriptversion=2013-12-25.23; # UTC
+
+# This originates from X11R5 (mit/util/scripts/install.sh), which was
+# later released in X11R6 (xc/config/util/install.sh) with the
+# following copyright and license.
 #
-# Copyright 1991 by the Massachusetts Institute of Technology
+# Copyright (C) 1994 X Consortium
 #
-# Permission to use, copy, modify, distribute, and sell this software and its
-# documentation for any purpose is hereby granted without fee, provided that
-# the above copyright notice appear in all copies and that both that
-# copyright notice and this permission notice appear in supporting
-# documentation, and that the name of M.I.T. not be used in advertising or
-# publicity pertaining to distribution of the software without specific,
-# written prior permission.  M.I.T. makes no representations about the
-# suitability of this software for any purpose.  It is provided "as is"
-# without express or implied warranty.
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to
+# deal in the Software without restriction, including without limitation the
+# rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+# sell copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
+# X CONSORTIUM BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN
+# AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNEC-
+# TION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+#
+# Except as contained in this notice, the name of the X Consortium shall not
+# be used in advertising or otherwise to promote the sale, use or other deal-
+# ings in this Software without prior written authorization from the X Consor-
+# tium.
+#
+#
+# FSF changes to this file are in the public domain.
 #
 # Calling this script install-sh is preferred over install.sh, to prevent
-# `make' implicit rules from creating a file called install from it
+# 'make' implicit rules from creating a file called install from it
 # when there is no Makefile.
 #
 # This script is compatible with the BSD install script, but was written
-# from scratch.  It can only install one file at a time, a restriction
-# shared with many OS's install programs.
+# from scratch.
 
+tab='	'
+nl='
+'
+IFS=" $tab$nl"
 
-# set DOITPROG to echo to test this script
+# Set DOITPROG to "echo" to test this script.
 
-# Don't use :- since 4.3BSD and earlier shells don't like it.
-doit="${DOITPROG-}"
+doit=${DOITPROG-}
+doit_exec=${doit:-exec}
 
+# Put in absolute file names if you don't have them in your path;
+# or use environment vars.
 
-# put in absolute paths if you don't have them in your path; or use env. vars.
+chgrpprog=${CHGRPPROG-chgrp}
+chmodprog=${CHMODPROG-chmod}
+chownprog=${CHOWNPROG-chown}
+cmpprog=${CMPPROG-cmp}
+cpprog=${CPPROG-cp}
+mkdirprog=${MKDIRPROG-mkdir}
+mvprog=${MVPROG-mv}
+rmprog=${RMPROG-rm}
+stripprog=${STRIPPROG-strip}
 
-mvprog="${MVPROG-mv}"
-cpprog="${CPPROG-cp}"
-chmodprog="${CHMODPROG-chmod}"
-chownprog="${CHOWNPROG-chown}"
-chgrpprog="${CHGRPPROG-chgrp}"
-stripprog="${STRIPPROG-strip}"
-rmprog="${RMPROG-rm}"
-mkdirprog="${MKDIRPROG-mkdir}"
+posix_mkdir=
 
-transformbasename=""
-transform_arg=""
-instcmd="$mvprog"
-chmodcmd="$chmodprog 0755"
-chowncmd=""
-chgrpcmd=""
-stripcmd=""
+# Desired mode of installed file.
+mode=0755
+
+chgrpcmd=
+chmodcmd=$chmodprog
+chowncmd=
+mvcmd=$mvprog
 rmcmd="$rmprog -f"
-mvcmd="$mvprog"
-src=""
-dst=""
-dir_arg=""
+stripcmd=
 
-while [ x"$1" != x ]; do
-    case $1 in
-	-c) instcmd=$cpprog
-	    shift
-	    continue;;
+src=
+dst=
+dir_arg=
+dst_arg=
 
-	-d) dir_arg=true
-	    shift
-	    continue;;
+copy_on_change=false
+is_target_a_directory=possibly
 
-	-m) chmodcmd="$chmodprog $2"
-	    shift
-	    shift
-	    continue;;
+usage="\
+Usage: $0 [OPTION]... [-T] SRCFILE DSTFILE
+   or: $0 [OPTION]... SRCFILES... DIRECTORY
+   or: $0 [OPTION]... -t DIRECTORY SRCFILES...
+   or: $0 [OPTION]... -d DIRECTORIES...
 
-	-o) chowncmd="$chownprog $2"
-	    shift
-	    shift
-	    continue;;
+In the 1st form, copy SRCFILE to DSTFILE.
+In the 2nd and 3rd, copy all SRCFILES to DIRECTORY.
+In the 4th, create DIRECTORIES.
 
-	-g) chgrpcmd="$chgrpprog $2"
-	    shift
-	    shift
-	    continue;;
+Options:
+     --help     display this help and exit.
+     --version  display version info and exit.
 
-	-s) stripcmd=$stripprog
-	    shift
-	    continue;;
+  -c            (ignored)
+  -C            install only if different (preserve the last data modification time)
+  -d            create directories instead of installing files.
+  -g GROUP      $chgrpprog installed files to GROUP.
+  -m MODE       $chmodprog installed files to MODE.
+  -o USER       $chownprog installed files to USER.
+  -s            $stripprog installed files.
+  -t DIRECTORY  install into DIRECTORY.
+  -T            report an error if DSTFILE is a directory.
 
-	-t=*) transformarg=`echo $1 | sed 's/-t=//'`
-	    shift
-	    continue;;
+Environment variables override the default commands:
+  CHGRPPROG CHMODPROG CHOWNPROG CMPPROG CPPROG MKDIRPROG MVPROG
+  RMPROG STRIPPROG
+"
 
-	-b=*) transformbasename=`echo $1 | sed 's/-b=//'`
-	    shift
-	    continue;;
+while test $# -ne 0; do
+  case $1 in
+    -c) ;;
 
-	*)  if [ x"$src" = x ]
-	    then
-		src=$1
-	    else
-		# this colon is to work around a 386BSD /bin/sh bug
-		:
-		dst=$1
-	    fi
-	    shift
-	    continue;;
+    -C) copy_on_change=true;;
+
+    -d) dir_arg=true;;
+
+    -g) chgrpcmd="$chgrpprog $2"
+        shift;;
+
+    --help) echo "$usage"; exit $?;;
+
+    -m) mode=$2
+        case $mode in
+          *' '* | *"$tab"* | *"$nl"* | *'*'* | *'?'* | *'['*)
+            echo "$0: invalid mode: $mode" >&2
+            exit 1;;
+        esac
+        shift;;
+
+    -o) chowncmd="$chownprog $2"
+        shift;;
+
+    -s) stripcmd=$stripprog;;
+
+    -t)
+        is_target_a_directory=always
+        dst_arg=$2
+        # Protect names problematic for 'test' and other utilities.
+        case $dst_arg in
+          -* | [=\(\)!]) dst_arg=./$dst_arg;;
+        esac
+        shift;;
+
+    -T) is_target_a_directory=never;;
+
+    --version) echo "$0 $scriptversion"; exit $?;;
+
+    --) shift
+        break;;
+
+    -*) echo "$0: invalid option: $1" >&2
+        exit 1;;
+
+    *)  break;;
+  esac
+  shift
+done
+
+# We allow the use of options -d and -T together, by making -d
+# take the precedence; this is for compatibility with GNU install.
+
+if test -n "$dir_arg"; then
+  if test -n "$dst_arg"; then
+    echo "$0: target directory not allowed when installing a directory." >&2
+    exit 1
+  fi
+fi
+
+if test $# -ne 0 && test -z "$dir_arg$dst_arg"; then
+  # When -d is used, all remaining arguments are directories to create.
+  # When -t is used, the destination is already specified.
+  # Otherwise, the last argument is the destination.  Remove it from $@.
+  for arg
+  do
+    if test -n "$dst_arg"; then
+      # $@ is not empty: it contains at least $arg.
+      set fnord "$@" "$dst_arg"
+      shift # fnord
+    fi
+    shift # arg
+    dst_arg=$arg
+    # Protect names problematic for 'test' and other utilities.
+    case $dst_arg in
+      -* | [=\(\)!]) dst_arg=./$dst_arg;;
     esac
+  done
+fi
+
+if test $# -eq 0; then
+  if test -z "$dir_arg"; then
+    echo "$0: no input file specified." >&2
+    exit 1
+  fi
+  # It's OK to call 'install-sh -d' without argument.
+  # This can happen when creating conditional directories.
+  exit 0
+fi
+
+if test -z "$dir_arg"; then
+  if test $# -gt 1 || test "$is_target_a_directory" = always; then
+    if test ! -d "$dst_arg"; then
+      echo "$0: $dst_arg: Is not a directory." >&2
+      exit 1
+    fi
+  fi
+fi
+
+if test -z "$dir_arg"; then
+  do_exit='(exit $ret); exit $ret'
+  trap "ret=129; $do_exit" 1
+  trap "ret=130; $do_exit" 2
+  trap "ret=141; $do_exit" 13
+  trap "ret=143; $do_exit" 15
+
+  # Set umask so as not to create temps with too-generous modes.
+  # However, 'strip' requires both read and write access to temps.
+  case $mode in
+    # Optimize common cases.
+    *644) cp_umask=133;;
+    *755) cp_umask=22;;
+
+    *[0-7])
+      if test -z "$stripcmd"; then
+        u_plus_rw=
+      else
+        u_plus_rw='% 200'
+      fi
+      cp_umask=`expr '(' 777 - $mode % 1000 ')' $u_plus_rw`;;
+    *)
+      if test -z "$stripcmd"; then
+        u_plus_rw=
+      else
+        u_plus_rw=,u+rw
+      fi
+      cp_umask=$mode$u_plus_rw;;
+  esac
+fi
+
+for src
+do
+  # Protect names problematic for 'test' and other utilities.
+  case $src in
+    -* | [=\(\)!]) src=./$src;;
+  esac
+
+  if test -n "$dir_arg"; then
+    dst=$src
+    dstdir=$dst
+    test -d "$dstdir"
+    dstdir_status=$?
+  else
+
+    # Waiting for this to be detected by the "$cpprog $src $dsttmp" command
+    # might cause directories to be created, which would be especially bad
+    # if $src (and thus $dsttmp) contains '*'.
+    if test ! -f "$src" && test ! -d "$src"; then
+      echo "$0: $src does not exist." >&2
+      exit 1
+    fi
+
+    if test -z "$dst_arg"; then
+      echo "$0: no destination specified." >&2
+      exit 1
+    fi
+    dst=$dst_arg
+
+    # If destination is a directory, append the input filename; won't work
+    # if double slashes aren't ignored.
+    if test -d "$dst"; then
+      if test "$is_target_a_directory" = never; then
+        echo "$0: $dst_arg: Is a directory" >&2
+        exit 1
+      fi
+      dstdir=$dst
+      dst=$dstdir/`basename "$src"`
+      dstdir_status=0
+    else
+      dstdir=`dirname "$dst"`
+      test -d "$dstdir"
+      dstdir_status=$?
+    fi
+  fi
+
+  obsolete_mkdir_used=false
+
+  if test $dstdir_status != 0; then
+    case $posix_mkdir in
+      '')
+        # Create intermediate dirs using mode 755 as modified by the umask.
+        # This is like FreeBSD 'install' as of 1997-10-28.
+        umask=`umask`
+        case $stripcmd.$umask in
+          # Optimize common cases.
+          *[2367][2367]) mkdir_umask=$umask;;
+          .*0[02][02] | .[02][02] | .[02]) mkdir_umask=22;;
+
+          *[0-7])
+            mkdir_umask=`expr $umask + 22 \
+              - $umask % 100 % 40 + $umask % 20 \
+              - $umask % 10 % 4 + $umask % 2
+            `;;
+          *) mkdir_umask=$umask,go-w;;
+        esac
+
+        # With -d, create the new directory with the user-specified mode.
+        # Otherwise, rely on $mkdir_umask.
+        if test -n "$dir_arg"; then
+          mkdir_mode=-m$mode
+        else
+          mkdir_mode=
+        fi
+
+        posix_mkdir=false
+        case $umask in
+          *[123567][0-7][0-7])
+            # POSIX mkdir -p sets u+wx bits regardless of umask, which
+            # is incompatible with FreeBSD 'install' when (umask & 300) != 0.
+            ;;
+          *)
+            tmpdir=${TMPDIR-/tmp}/ins$RANDOM-$$
+            trap 'ret=$?; rmdir "$tmpdir/d" "$tmpdir" 2>/dev/null; exit $ret' 0
+
+            if (umask $mkdir_umask &&
+                exec $mkdirprog $mkdir_mode -p -- "$tmpdir/d") >/dev/null 2>&1
+            then
+              if test -z "$dir_arg" || {
+                   # Check for POSIX incompatibilities with -m.
+                   # HP-UX 11.23 and IRIX 6.5 mkdir -m -p sets group- or
+                   # other-writable bit of parent directory when it shouldn't.
+                   # FreeBSD 6.1 mkdir -m -p sets mode of existing directory.
+                   ls_ld_tmpdir=`ls -ld "$tmpdir"`
+                   case $ls_ld_tmpdir in
+                     d????-?r-*) different_mode=700;;
+                     d????-?--*) different_mode=755;;
+                     *) false;;
+                   esac &&
+                   $mkdirprog -m$different_mode -p -- "$tmpdir" && {
+                     ls_ld_tmpdir_1=`ls -ld "$tmpdir"`
+                     test "$ls_ld_tmpdir" = "$ls_ld_tmpdir_1"
+                   }
+                 }
+              then posix_mkdir=:
+              fi
+              rmdir "$tmpdir/d" "$tmpdir"
+            else
+              # Remove any dirs left behind by ancient mkdir implementations.
+              rmdir ./$mkdir_mode ./-p ./-- 2>/dev/null
+            fi
+            trap '' 0;;
+        esac;;
+    esac
+
+    if
+      $posix_mkdir && (
+        umask $mkdir_umask &&
+        $doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir"
+      )
+    then :
+    else
+
+      # The umask is ridiculous, or mkdir does not conform to POSIX,
+      # or it failed possibly due to a race condition.  Create the
+      # directory the slow way, step by step, checking for races as we go.
+
+      case $dstdir in
+        /*) prefix='/';;
+        [-=\(\)!]*) prefix='./';;
+        *)  prefix='';;
+      esac
+
+      oIFS=$IFS
+      IFS=/
+      set -f
+      set fnord $dstdir
+      shift
+      set +f
+      IFS=$oIFS
+
+      prefixes=
+
+      for d
+      do
+        test X"$d" = X && continue
+
+        prefix=$prefix$d
+        if test -d "$prefix"; then
+          prefixes=
+        else
+          if $posix_mkdir; then
+            (umask=$mkdir_umask &&
+             $doit_exec $mkdirprog $mkdir_mode -p -- "$dstdir") && break
+            # Don't fail if two instances are running concurrently.
+            test -d "$prefix" || exit 1
+          else
+            case $prefix in
+              *\'*) qprefix=`echo "$prefix" | sed "s/'/'\\\\\\\\''/g"`;;
+              *) qprefix=$prefix;;
+            esac
+            prefixes="$prefixes '$qprefix'"
+          fi
+        fi
+        prefix=$prefix/
+      done
+
+      if test -n "$prefixes"; then
+        # Don't fail if two instances are running concurrently.
+        (umask $mkdir_umask &&
+         eval "\$doit_exec \$mkdirprog $prefixes") ||
+          test -d "$dstdir" || exit 1
+        obsolete_mkdir_used=true
+      fi
+    fi
+  fi
+
+  if test -n "$dir_arg"; then
+    { test -z "$chowncmd" || $doit $chowncmd "$dst"; } &&
+    { test -z "$chgrpcmd" || $doit $chgrpcmd "$dst"; } &&
+    { test "$obsolete_mkdir_used$chowncmd$chgrpcmd" = false ||
+      test -z "$chmodcmd" || $doit $chmodcmd $mode "$dst"; } || exit 1
+  else
+
+    # Make a couple of temp file names in the proper directory.
+    dsttmp=$dstdir/_inst.$$_
+    rmtmp=$dstdir/_rm.$$_
+
+    # Trap to clean up those temp files at exit.
+    trap 'ret=$?; rm -f "$dsttmp" "$rmtmp" && exit $ret' 0
+
+    # Copy the file name to the temp name.
+    (umask $cp_umask && $doit_exec $cpprog "$src" "$dsttmp") &&
+
+    # and set any options; do chmod last to preserve setuid bits.
+    #
+    # If any of these fail, we abort the whole thing.  If we want to
+    # ignore errors from any of these, just make sure not to ignore
+    # errors from the above "$doit $cpprog $src $dsttmp" command.
+    #
+    { test -z "$chowncmd" || $doit $chowncmd "$dsttmp"; } &&
+    { test -z "$chgrpcmd" || $doit $chgrpcmd "$dsttmp"; } &&
+    { test -z "$stripcmd" || $doit $stripcmd "$dsttmp"; } &&
+    { test -z "$chmodcmd" || $doit $chmodcmd $mode "$dsttmp"; } &&
+
+    # If -C, don't bother to copy if it wouldn't change the file.
+    if $copy_on_change &&
+       old=`LC_ALL=C ls -dlL "$dst"     2>/dev/null` &&
+       new=`LC_ALL=C ls -dlL "$dsttmp"  2>/dev/null` &&
+       set -f &&
+       set X $old && old=:$2:$4:$5:$6 &&
+       set X $new && new=:$2:$4:$5:$6 &&
+       set +f &&
+       test "$old" = "$new" &&
+       $cmpprog "$dst" "$dsttmp" >/dev/null 2>&1
+    then
+      rm -f "$dsttmp"
+    else
+      # Rename the file to the real destination.
+      $doit $mvcmd -f "$dsttmp" "$dst" 2>/dev/null ||
+
+      # The rename failed, perhaps because mv can't rename something else
+      # to itself, or perhaps because mv is so ancient that it does not
+      # support -f.
+      {
+        # Now remove or move aside any old file at destination location.
+        # We try this two ways since rm can't unlink itself on some
+        # systems and the destination file might be busy for other
+        # reasons.  In this case, the final cleanup might fail but the new
+        # file should still install successfully.
+        {
+          test ! -f "$dst" ||
+          $doit $rmcmd -f "$dst" 2>/dev/null ||
+          { $doit $mvcmd -f "$dst" "$rmtmp" 2>/dev/null &&
+            { $doit $rmcmd -f "$rmtmp" 2>/dev/null; :; }
+          } ||
+          { echo "$0: cannot unlink or rename $dst" >&2
+            (exit 1); exit 1
+          }
+        } &&
+
+        # Now rename the file to the real destination.
+        $doit $mvcmd "$dsttmp" "$dst"
+      }
+    fi || exit 1
+
+    trap '' 0
+  fi
 done
 
-if [ x"$src" = x ]
-then
-	echo "$0: no input file specified" >&2
-	exit 1
-else
-	:
-fi
-
-if [ x"$dir_arg" != x ]; then
-	dst=$src
-	src=""
-
-	if [ -d "$dst" ]; then
-		instcmd=:
-		chmodcmd=""
-	else
-		instcmd=$mkdirprog
-	fi
-else
-
-# Waiting for this to be detected by the "$instcmd $src $dsttmp" command
-# might cause directories to be created, which would be especially bad
-# if $src (and thus $dsttmp) contains '*'.
-
-	if [ -f "$src" ] || [ -d "$src" ]
-	then
-		:
-	else
-		echo "$0: $src does not exist" >&2
-		exit 1
-	fi
-
-	if [ x"$dst" = x ]
-	then
-		echo "$0: no destination specified" >&2
-		exit 1
-	else
-		:
-	fi
-
-# If destination is a directory, append the input filename; if your system
-# does not like double slashes in filenames, you may need to add some logic
-
-	if [ -d "$dst" ]
-	then
-		dst=$dst/`basename "$src"`
-	else
-		:
-	fi
-fi
-
-## this sed command emulates the dirname command
-dstdir=`echo "$dst" | sed -e 's,[^/]*$,,;s,/$,,;s,^$,.,'`
-
-# Make sure that the destination directory exists.
-#  this part is taken from Noah Friedman's mkinstalldirs script
-
-# Skip lots of stat calls in the usual case.
-if [ ! -d "$dstdir" ]; then
-defaultIFS='
-	'
-IFS="${IFS-$defaultIFS}"
-
-oIFS=$IFS
-# Some sh's can't handle IFS=/ for some reason.
-IFS='%'
-set - `echo "$dstdir" | sed -e 's@/@%@g' -e 's@^%@/@'`
-IFS=$oIFS
-
-pathcomp=''
-
-while [ $# -ne 0 ] ; do
-	pathcomp=$pathcomp$1
-	shift
-
-	if [ ! -d "$pathcomp" ] ;
-        then
-		$mkdirprog "$pathcomp"
-	else
-		:
-	fi
-
-	pathcomp=$pathcomp/
-done
-fi
-
-if [ x"$dir_arg" != x ]
-then
-	$doit $instcmd "$dst" &&
-
-	if [ x"$chowncmd" != x ]; then $doit $chowncmd "$dst"; else : ; fi &&
-	if [ x"$chgrpcmd" != x ]; then $doit $chgrpcmd "$dst"; else : ; fi &&
-	if [ x"$stripcmd" != x ]; then $doit $stripcmd "$dst"; else : ; fi &&
-	if [ x"$chmodcmd" != x ]; then $doit $chmodcmd "$dst"; else : ; fi
-else
-
-# If we're going to rename the final executable, determine the name now.
-
-	if [ x"$transformarg" = x ]
-	then
-		dstfile=`basename "$dst"`
-	else
-		dstfile=`basename "$dst" $transformbasename |
-			sed $transformarg`$transformbasename
-	fi
-
-# don't allow the sed command to completely eliminate the filename
-
-	if [ x"$dstfile" = x ]
-	then
-		dstfile=`basename "$dst"`
-	else
-		:
-	fi
-
-# Make a couple of temp file names in the proper directory.
-
-	dsttmp=$dstdir/#inst.$$#
-	rmtmp=$dstdir/#rm.$$#
-
-# Trap to clean up temp files at exit.
-
-	trap 'status=$?; rm -f "$dsttmp" "$rmtmp" && exit $status' 0
-	trap '(exit $?); exit' 1 2 13 15
-
-# Move or copy the file name to the temp name
-
-	$doit $instcmd "$src" "$dsttmp" &&
-
-# and set any options; do chmod last to preserve setuid bits
-
-# If any of these fail, we abort the whole thing.  If we want to
-# ignore errors from any of these, just make sure not to ignore
-# errors from the above "$doit $instcmd $src $dsttmp" command.
-
-	if [ x"$chowncmd" != x ]; then $doit $chowncmd "$dsttmp"; else :;fi &&
-	if [ x"$chgrpcmd" != x ]; then $doit $chgrpcmd "$dsttmp"; else :;fi &&
-	if [ x"$stripcmd" != x ]; then $doit $stripcmd "$dsttmp"; else :;fi &&
-	if [ x"$chmodcmd" != x ]; then $doit $chmodcmd "$dsttmp"; else :;fi &&
-
-# Now remove or move aside any old file at destination location.  We try this
-# two ways since rm can't unlink itself on some systems and the destination
-# file might be busy for other reasons.  In this case, the final cleanup
-# might fail but the new file should still install successfully.
-
-{
-	if [ -f "$dstdir/$dstfile" ]
-	then
-		$doit $rmcmd -f "$dstdir/$dstfile" 2>/dev/null ||
-		$doit $mvcmd -f "$dstdir/$dstfile" "$rmtmp" 2>/dev/null ||
-		{
-		  echo "$0: cannot unlink or rename $dstdir/$dstfile" >&2
-		  (exit 1); exit
-		}
-	else
-		:
-	fi
-} &&
-
-# Now rename the file to the real destination.
-
-	$doit $mvcmd "$dsttmp" "$dstdir/$dstfile"
-
-fi &&
-
-# The final little trick to "correctly" pass the exit status to the exit trap.
-
-{
-	(exit 0); exit
-}
+# Local variables:
+# eval: (add-hook 'write-file-hooks 'time-stamp)
+# time-stamp-start: "scriptversion="
+# time-stamp-format: "%:y-%02m-%02d.%02H"
+# time-stamp-time-zone: "UTC"
+# time-stamp-end: "; # UTC"
+# End: