There is a bug where freshclam fails to detect if a downloaded CDIFF is
empty. In 0.103 this, combined with a CDN caching issue could result in
freshclam downloading a daily.cvd but failing to update, putting it in a
sort of infinite loop. In 0.104 this issue manifests slightly
differently, requiring freshclam to run up to 3x before you get over the
empty-CVD hump and are back to normal updates.
This commit updates an existing cdiff test with the zero-byte cdiff + an
out-of-date CVD to confirm the bug. The following commit will fix it.
If running multiple parallel processes of "xor_testfile.py" there was a
race condition between checking for the existence of the directory and
creating it. Now this is handled as a dependency in CMake.
Xcode (and perhaps some other generators?) do not like targets that have
only object files. See:
https://cmake.org/cmake/help/latest/command/add_library.html#object-libraries
And: https://cmake.org/pipermail/cmake/2016-May/063479.html
This issue manifests when using `-G Xcode` on macOS as the library
dylibs being missing when linking with other binaries.
This commit removes the object libraries for libclamav, libfreshclam,
libclamunrar_iface, libclamunrar, libclammspack, and (lib)common
because they were used by static or shared libs that didn't
themselves have any added sources.
Add getter & setter for the debug flag, so it isn't referenced by unit
tests or other code that links with libclamav. This is needed because
global variables are exported symbols on Windows.
The access-denied test and excludepath tests both relied on the full
path of the test file to be in the expected results. This fails if
you're working within a path that has a symlink because clamd and
clamdscan determine real-paths before scanning and end up sending
back the real-path in the results, not the original path.
This fixes the tests by removing the full paths from the expected
results.
I also cleaned up some type safety warnings.
Python 3.5 compatibility fixes for Debian 9, etc that lack 3.6+.
Change a python f-string to an old-style `"".format()`.
Convert Path objects to strings for older `shutil` APIs that don't
accept Paths.
Added a test to verify that clamscan can extract images from an XLS
document. The document has 2 images: a PNG and JPEG version of the
clamav demon/logo. The test requires the json metadata feature to verify
that the MD5 of the images are correct.
No other image formats were tested because despite the format allegedly
supporting other imate formats, Excel converts TIFF, BMP, and GIF images
to PNG files when you insert them.
The split test files are flagged by some AV's because they look like
broken executables. Instead of splitting the test files to prevent
detections, we should encrypt them. This commit replaces the "reassemble
testfiles" script with a basic "XOR testfiles" script that can be used
to encrypt or decrypt test files. This commit also of course then
replaces all the split files with xor'ed files.
The test and unit_tests directories were a bit of a mess, so I
reorganized them all into unit_tests with all of the test files placed
under "unit_tests/input" using subdirectories for different types of files.
Adds a basic test to validate that ExcludePath correctly excludes a
subdirectory but does not exclude subsequent files. As with the other
ClamD/Scan tests, it will test in each mode: regular, stream, and
fdpass (if available).
Unlike the other tests, this one tests ClamDScan with Valgrind instead
of ClamD.
Refactored the clamd_test.py file to reduce duplicate code, and support
enabling and disabling valgrind when running ClamDScan and ClamD.
Add pytest to the github actions environments because the results when
using pytest are far easier to read.
There appear to be minors leak in clamd that can occur when shutting-
down immediately after a command (e.g. RELOAD).
These are causing intermittent clamd test failures.
It seems like they're caused by a thread leaking occasionally,
due to not exiting before the program terminates.
I don't believe these to be a serious issue. Tracking down the exact
cause and crafting a fix for the leaks isn't worth the effort.
This commit adds valgrind suppression rules to stabilize the tests.
The non-existent file test has a hack to "expect" a wierd error message
caused by the '\v' character rather than the file not actually existing.
Recently something(?) changed and the test started reporting yet a
different message or no message.
Removing the '\v' special character fixes the test so it actually tests
a non-existent file and returns the same message as on other operating
systems.
Add a test where freshclam received a zero-byte cdiff to trigger a whole
CVD database download, and the CVD served is older than advertised.
This is a regression test for a bug found & fixed by Andrew Williams.
Adds 3 tests to validate that:
1. a CDIFF update works
2. a CDIFF partial update (with 1 missing CDIFF) works
and that a subsequent update is ok with being 1 behind
3. a CDIFF partial update (with 2 missing CDIFFs) works
and that a subsequent update will try to get the WHOLE CVD -
because being 2+ CDIFFs behind without any update isn't good enough.
Also fixed a minor bug so that the database name is properly displayed
when a partial update occurs instead of displaying "(null)".
Also changed the freshclam test port to 8001 to deconflict with
CVD-Update, in case that's running in the background.
TODO: Make the tests smarter so they find an open port instead of
hoping that 8001 is available.
Some users have scripts set up from long ago to delete mirrors.dat if
FreshClam failed. We used to recommend this if people had technical
issues because mirrors.dat would store a bunch of entries indicating
that all of their regional mirrors were failing and then FreshClam would
give up.
The new freshclam DAT file no longer stores that kind of information.
Deleting the DAT file is no longer sound advice.
We very much want the UUID, which is generated when creating the DAT
file, to persist between runs. So unless people go and change the
scripts to delete freshclam.dat instead, this commit should resolve the
concern.
Improvements to use modern block list and allow list verbiage.
blacklist -> block list
whitelist -> allow listed
blacklisted -> blocked
whitelisted -> allowed
In the case of certificate verification, use "trust" or "verify" when
something is allowed.
Also changed domainlist -> domain list (or DomainList) to match.
Have to manually link libtinfo (`-ltinfo`) because our FindLLVM
didn't add it to the LLVM_LIBRARIES variable for us. See:
- https://stackoverflow.com/questions/21477407/llvm-3-5-fails-to-link
Have to remove the CXX_STANDARD setting at the top of CMakeLists.txt
because of c++90 / c++11 ABI compatibility issues w/ LLVM. See:
- https://maleadt.github.io/LLVM.jl/dev/man/troubleshooting/
Rename "llvm/Config/config.h" "llvm/Config/llvm-config.h" because
LLVM renamed it in 2.8.
Have to link LLVM manually with the test binaries that use the
clamav object library instead of libclamav shared library.
CMake does not propagate library dependencies from object files.
I tested on ubuntu:16.04 with LLVM 3.6 built from source using:
```
/usr/local/bin/cmake .. -D CMAKE_INSTALL_PREFIX=/opt/llvm/3.6 \
-D LLVM_ENABLE_RTTI=ON
```
Then built clamav w/:
```
/usr/local/bin/cmake .. -D CMAKE_INSTALL_PREFIX=`pwd`/install \
-D BYTECODE_RUNTIME="llvm" \
-D LLVM_ROOT_DIR="/opt/llvm/3.6" \
-D LLVM_FIND_VERSION="3.6.0" && make && make install
```
CMake is now required to build.
The built-in LLVM is no longer available.
Also removed support for libltdl calls, which is not used in the CMake
builds, was only used when building with Autotools.
TODO: Fix CMake LLVM support & update to work with modern versions.
If you test in a docker container and don't create a user/switch to run
as something other than root, than the tests will fail.
Default freshclam behavior is to drop to run as the "clamav" user when
run as root. This change sets the DatabaseOwner to the current user, so
it the tests will pass when run as root.
The WebServerHandler classes used for the tests have a "Test_" prefix
which causes pytest to print some warnings thinking they're special test
classes. This commit gets rid of the warnings.
If pytest is available in the path, but wasn't found in the Python 3
installation discovered by CMake, then the test suite will fail.
This can happen when two different Python 3 installations exist, eg 3.6
and 3.7 and when the newest one (prefered by CMake) doesn't have pytest.
This commit will use the correct command depending on where pytest was
(or wasn't) found. It will also fail more gracefully if neither pytest
nor unittest was found (this happens with some wierd python installs).
The testcase.py script is using Python 3.6+ syntax for the NamedTuple.
This commit reverts to the old 3.5 syntax until we can drop support for
Python 3.5.
The named "shared" is confusing, especially now that these features are
built as a static library instead of being directly compiled into the
various applications.
Also add support for using pytest instead of python's unittest to make
it easier to find & read failed test results.
Clean up the log output in check_clamd when printing "wrong reply"
results.
Switch from using collections.namedtuple() to defining our own
NamedTuple subclass, which adds member variable typing.
If you set an ExcludePath regex in clamd.conf and then perform a
ClamDScan scan with --fdpass --multiscan, it will segfault.
The same issue also affects --fdpass --multiscan scans when using
ExcludePath when scanning a patch that doesn't exist.
The issue is that the filepath isn't being passed along for the path
exclusion regex match, resulting in a NULL deref.
This commit also fixes a possible memory leak if by duplicating the path
for the handle_entry() call _after_ the callback() runs, in case ret
isn't CL_SUCCESS and the function exits without every using the entry
structure or free'ing the copied filename.
The above work temporarily caused a test failure in check_clamd and a
valgrind failure in clamd for the nonexistent file test due to a minor
memory leak. This made it apparent that there were a few other nearby
possible memory leaks.
This commit fixes the above plus cleans up the error handling in clamd's
the file tree walk functions.
Valgrind occasionally reports a thread join leak from the multi-
threaded reload. This commit inserts a sleep to give ClamD more
time to clean up before terminating ClamD after the test.
Adds support to the pcre2 and pthreadw32 Find<Package>.cmake modules for
correctly discovering the debug versions. This change modeled after the
upstream FindBZip2.cmake module.
Also eliminated HAVE_STRUCT_TIMESPEC redefinition warnings in Windows
builds.
In testing on Alpine, I found that most libs were installing to
<prefix>/lib while libclamav installed to <prefix>/lib64. Those who like
multiarch will advocate for lib64, though I only actually noticed it
because clamscan failed to find libclamav.so! Anyways, they should all
install to lib64 by default if that's what how the system is set up.
Using ${CMAKE_INSTALL_FULL_LIBDIR} instead of <prefix>/lib will do that.
The clamd socket path was changed be an absolute path when
adding CTest support. This quietly broke the check_clamd libcheck
program when building with autotools because a relative path was
expected. I failed to notice because the autotools `make check`
doesn't actually care if check_clamd works!
It turns out that a relative path is required because the max length for
a socket path is *very* short.
This commit changes check_clamd and the associated CMake test to also
use a relative path for the clamd socket. Notably it also modifies the
testcase.py framework switch to the cls.path_tmp (generated) directory
before the tests and restore the CWD after the tests so as to ensure
that the socket file is dropped in somewhere in that tmp directory.
The test previously tried to limit the # of connections to
`ulimit -n` - 5. On most linux docker containers this failed
with a test timeout at about connection 285 or so.
The output in test-clamd.log would look something like this:
check_clamd.c:74:E:clamd stress test:test_connections:0: (after this
point) Test timeout expired
The same issue was observed with FreeBSD (12.2) when limiting to around
280 (noting that the FD # in the debug log actually hit around 288).
Limiting the # of connections to 250 resolves the issue in our test
pipeline.
Python 3.6 is not available on Debian 9 and other older LTS releases.
This patch removes use of Python f-strings which were introduced in
Python 3.6 so as to support Python 3.5.
TODO: Revert this commit when Debian 9 dies or gets f-string support
(whichver comes first).
On Windows, files open()'ed without the O_BINARY flag will have new-line
LF (aka \n) converted to CRLF (aka \r\n) automatically when read from or
written to. This is undesirable for all scan targets AND temp files
because it affects pattern matching and with hashing.
This commit converts a handful of instances throughout the codebase
where it appears that O_BINARY was mistakenly omitted and could result
in unexpected behavior on Windows.
Git on Windows also converts LF -> CRLF for "text" files, for editing
purposes.
This is problematic for scan files and test files that should match
verbatim.
We can prevent this issue by marking .ref test files as "binary" in the
.gitattributes file and by always opening scan files and temp files as
binary.
In this commit I've also removed the `ChangeLog merge=cl-merge` line
that was once used to reduce ChangeLog merge conflicts by using the
gnulib git-merge-changlog tool. This project now categorizes changes in
the NEWS.md.
For finer detail, git commit history is fully accessible on github.com.
Add missing CTest files to tarball.
Remove the generated version.h from libclamav sources so it isn't added
to the dist. version.h should be generated at build time by both
autotools builds and cmake builds. When included with the dist, it may
cause clamd VERSION command checks to fail because clamd is compiled
with the wrong version.h header.
Also bumped the minimum CMake version for Windows to accomodate the
file(GET_RUNTIME_DEPENDENCIES).
Updates to fix issues in the CMake install instructions.
Updates the README.md to indicate that CMake is now preferred
Adds a GitHub Actions badge, Discord badge, and logo to the README.md.
CMake:
- Renamed ENABLE_DOCS to ENABLE_MAN_PAGES.
- Fixed build issue when milter isn't enabled on Linux. Changed the
default to build milter on non-macOS, non-Windows operating systems.
- Fix LD_LIBRARY_PATH for tests including on macOS where LD_LIBRARY_PATH
and DYLD_LIBRARY_PATH must be manually propagated to subprocesses.
- Use UNKNOWN IMPORTED library instead of INTERFACE IMPORTED library for
pdcurses, but still use INTERFACE IMPORTED for ncurses.
UNKNOWN IMPORTED appears to be required so that we can use
$<TARGET_FILE_DIR:Curses::curses> to collected the pdcurses library at
install time on Windows.
- When building with vcpkg on Windows, CMake will automatically install
your app local dependencies (aka the DLL runtime dependencies).
Meanwhile, file(GET_RUNTIME_DEPENDENCIES ...) doesn't appear to work
correctly with vcpkg packages. The solution is to use a custom target
that has CMake perform a local install to the unit_tests directory
when using vcpkg.
This is in fact far easier than using GET_RUNTIME_DEPENDENCIES in the
unit_tests for assembling the test environment but we can't use this
method for the non-vcpkg install because it won't collect
checkDynamic.dll for us because we don't install our tests.
We also can't link with the static check.lib because the static
check.lib has pthreads symbols linked in and will conflict with our
pthread.dll.
TL;DR: We'll continue to use file(GET_RUNTIME_DEPENDENCIES ...) for
assembling the test enviornment on non-vcpkg builds, and use the local
install method for vcpkg builds.
testcase.py: Wrapped a Pathlib.unlink() call in exception handling as
the missing_ok optional parameter requires a Python version too new for
common use.
Remove localtime_r from win32 compat lib.
localtime_r may be present in libcheck when building with vcpkg and
while making it a static function would also solve the issue, using
localtime_s instead like we do everywhere else should work just fine.
check_clamd: Limited the max # of connections for the stress test on Mac
to 850, to address issues found testing on macos-latest on GitHub Actions.
Visual Studio projects removed in favor of CMake because it's far easier
to build and maintain. Also removed the old InnoSetup installer now that
CMake's CPack provides installer creation.
While working on this I found that the THIS_IS_CLAMAV macro was missing,
resulting in warnings for the `have_rar` and `have_clamjit` exported
global variables.
I also stumbled across some code duplication and more cl_error_t / int
type issues in the pcre code, so this commit includes a little cleanup.
Enabled the metadata collection feature, scan heuristics, and all-match
mode when fuzzing in the interest of better code coverage.
Also remove deprecated STREAM command.
An ENABLE_TESTS CMake option is provided so that users can disable
testing if they don't want it. Instructions for how to use this
included in the INSTALL.cmake.md file.
If you run `ctest`, each testcase will write out a log file to the
<build>/unit_tests directory.
As with Autotools' make check, the test files are from test/.split
and unit_tests/.split files, but for CMake these are generated at
build time instead of at test time.
On Posix systems, sets the LD_LIBRARY_PATH so that ClamAV-compiled
libraries can be loaded when running tests.
On Windows systems, CTest will identify and collect all library
dependencies and assemble a temporarily install under the
build/unit_tests directory so that the libraries can be loaded when
running tests.
The same feature is used on Windows when using CMake to install to
collect all DLL dependencies so that users don't have to install them
manually afterwards.
Each of the CTest tests are run using a custom wrapper around Python's
unittest framework, which is also responsible for finding and inserting
valgrind into the valgrind tests on Posix systems.
Unlike with Autotools, the CMake CTest Valgrind-tests are enabled by
default, if Valgrind can be found. There's no need to set VG=1.
CTest's memcheck module is NOT supported, because we use Python to
orchestrate our tests.
Added a bunch of Windows compatibility changes to the unit tests.
These were primarily changing / to PATHSEP and making adjustments
to use Win32 C headers and ifdef out the POSIX ones which aren't
available on Windows. Also disabled a bunch of tests on Win32
that don't work on Windows, notably the mmap ones and FD-passing
(i.e. FILEDES) ones.
Add JSON_C_HAVE_INTTYPES_H definition to clamav-config.h to eliminate
warnings on Windows where json.h is included after inttypes.h because
json-c's inttypes replacement relies on it.
This is a it of a hack and may be removed if json-c fixes their
inttypes header stuff in the future.
Add preprocessor definitions on Windows to disable MSVC warnings about
CRT secure and nonstandard functions. While there may be a better
solution, this is needed to be able to see other more serious warnings.
Add missing file comment block and copyright statement for clamsubmit.c.
Also change json-c/json.h include filename to json.h in clamsubmit.c.
The directory name is not required.
Changed the hash table data integer type from long, which is poorly
defined, to size_t -- which is capable of storing a pointer. Fixed a
bunch of casts regarding this variable to eliminate warnings.
Fixed two bugs causing utf8 encoding unit tests to fail on Windows:
- The in_size variable should be the number of bytes, not the character
count. This was was causing the SHIFT_JIS (japanese codepage) to UTF8
transcoding test to only transcode half the bytes.
- It turns out that the MultiByteToWideChar() API can't transcode
UTF16-BE to UTF16-LE. The solution is to just iterate over the buffer
and flip the bytes on each uint16_t. This but was causing the UTF16-BE
to UTF8 tests to fail.
I also split up the utf8 transcoding tests into separate tests so I
could see all of the failures instead of just the first one.
Added a flags parameter to the unit test function to open testfiles
because it turns out that on Windows if a file contains the \r\n it will
replace it with just \n if you opened the file as a text file instead of
as binary. However, if we open the CBC files as binary, then a bunch of
bytecode tests fail. So I've changed the tests to open the CBC files in
the bytecode tests as text files and open all other files as binary.
Ported the feature tests from shell scripts to Python using a modified
version of our QA test-framework, which is largely compatible and will
allow us to migrate some QA tests into this repo. I'd like to add GitHub
Actions pipelines in the future so that all public PR's get some testing
before anyone has to manually review them.
The clamd --log option was missing from the help string, though it
definitely works. I've added it in this commit.
It appears that clamd.c was never clang-format'd, so this commit also
reformats clamd.c.
Some of the check_clamd tests expected the path returned by clamd to
match character for character with original path sent to clamd. However,
as we now evaluate real paths before a scan, the path returned by clamd
isn't going to match the relative (and possibly symlink-ridden) path
passed to clamdscan. I fixed this test by changing the test to search
for the basename: <signature> FOUND within the response instead of
matching the exact path.
Autotools: Link check_clamd with libclamav so we can use our utility
functions in check_clamd.c.
VirusEvent commands may use %v to get the signature name (virus name)
for the alert but do not have a format option to get the file name.
This commit adds %f to get the file name.
The VirusEvent feature does provide two environment variables,
$CLAM_VIRUSEVENT_FILENAME and $CLAM_VIRUSEVENT_VIRUSNAME which provide
file and virus names, but they weren't documented in the sample configs.
This commit also adds these environment variables to the sample configs.
The fmap_duplicate function is used create a new fmap with a view into
an existing fmap. When the new view is a different size than the old
fmap, a new hash must be calculated for the duplicate fmap. However,
when the duplicated fmap is the same size as the original fmap, the hash
will be the same and there's no point recalculating.
The issue is apparent when scanning large EXE files because the hash was
being calculated at the beginning and end of the scan.
Digging into this issue revealed that hash calculations for fmaps were
also being performed at the wrong place. For scans of maps we use
fmap_duplicate() early in the process to apply the name API argument to
the duplicate fmap. Fixing the logic so we doing recalculate the hash
revealed that we never calculated hashes for fmap's created from buffers
in the first place, so that also had to be fixed be relocating where the
hash is calculated.
I also found that fmap_duplicate()'s offset argument used an off_t,
though it and all caller offsets are not allowed to be negative. This
was a bit of tangent to fix a bunch of off_t variables and paramters
that should've been size_t.
Added a couple unit tests to verify that making duplicate fmaps, and
duplicate-duplicate fmaps works as expected after the change.
Changed CLI_ISCONTAINED() and CLI_ISCONTAINED2() macros to cast to
size_t, because pointers and buffer sizes may not be negative, and these
two macros do not rely on substraction.
