hash_search() walks the entire hash-table chain for the current rolling
checksum at every byte offset of the source file. Disk and VM images
contain large runs of identical blocks, so a single weak checksum
(get_checksum1) can collide thousands of times and pile every one of
those blocks onto one chain. When the sender then rolls across a region
whose weak checksum keeps landing on that chain without ever producing a
strong-checksum match, it re-walks the whole chain for every byte, giving
O(file_size * chain_length) behaviour. The result is rsync sitting at
100% CPU for hours with no apparent progress -- the long-standing "rsync
hangs on large files" reports.
Cap the number of same-weak-checksum candidates examined per offset at
MAX_CHAIN_LEN. Once the cap is hit we treat the offset as a non-match and
roll forward a byte; any block skipped this way is simply sent as literal
data, so the transferred result is always correct -- only the transfer
size is marginally affected. This is purely a sender-side search limit:
it changes no checksum, emitted byte, or protocol field, so a capped
sender interoperates with an unmodified receiver and vice versa.
On a synthetic 40000-block basis sharing one weak checksum, syncing a
60KB source dropped from ~18.4s to ~0.7s; the unbounded cost grows with
the square of the file size.
testsuite/hashsearch-chain_test.py reproduces the pathology with a tiny
basis of weak-checksum-colliding decoy blocks and asserts, via the
existing false_alarms counter (--debug=deltasum1), that the per-hash-hit
chain walk stays bounded. The assertion is exact and machine-independent
rather than timing-based.
In daemon-as-sender mode, send_files() leaves mbuf NULL for an empty local file
(st.st_size == 0). receive_sums() then derives s->flength from the peer's
sum_head (count * blength), so a hostile client that sends a non-empty sum_head
against that empty file drives the append_mode==2 verify loop into
map_ptr(NULL, ...), dereferencing the NULL map. The diminished-file guard
compares against F_LENGTH(file) (the daemon's own flist size, also 0), not the
peer-supplied flength, so it does not catch this.
Clamp s->flength to len before the verify loop -- we cannot checksum bytes we do
not have, and last_match must not run past len for the trailing matched() flush.
(cherry picked from commit 43f43c92d3414f340029ddf7ddc316dc4d4e3e7b)
The check_want_i adjacent-match optimisation substitutes i = want_i after
verifying sum1 and sum2, but -- unlike the main chain loop and the aligned_i
path -- did not re-check s->sums[want_i].len == l. A hostile peer acting as the
generator against a daemon-as-sender can set s2length=0 (read_sum_head only
enforces >= 0, so the sum2 memcmp is a no-op) and craft count=2, blength=len+1,
remainder=len with both sum1 set to the file's rolling checksum. The chain head
(.len=remainder=len) passes the l==.len gate; want_i=0 (.len=blength=len+1) is
then substituted unchecked. After matched(), offset advances to len, k =
MIN(blength, len-offset) = 0, map_ptr(...,0) returns NULL, and the rolling-
checksum trim dereferences map[0] -- a remote unauthenticated SEGV of the
per-connection daemon child against any module serving a non-empty regular file.
Add the missing length re-check, matching the chain loop and aligned_i path.
(cherry picked from commit aad82d77aae098c8d8529f69614a0ffcbfbbb5d3)
- Put sum2_array into sum_struct to hold an array of sum2 checksums
that are each xfer_sum_len bytes.
- Remove sum2 buf from sum_buf.
- Add macro sum2_at() to access each sum2 array element.
- Throw an error if a sums header has an s2length larger than
xfer_sum_len.
- Size flist checksum data to hold the active size, not the max.
- Add a negotiated hash method to the daemon auth code.
- Use EVP for all openssl digests. This makes it easy to add more
openssl digest methods and avoids deprecation warnings.
- Support a way to re-enable deprecated digests via openssl conf
file and allow a default file to be configured.
- Supply a simple openssl-rsync.cnf file to enable legacy digests.
- All the memory-allocation macros now auto-check for failure and exit
with a failure message that incudes the caller's file and lineno
info. This includes strdup().
- Added the `--max-alloc=SIZE` option to be able to override the memory
allocator's sanity-check limit. It defaults to 1G (as before).
Fixes bugzilla bug 12769.
When checking a checksum that refers to a part of an --inplace file that
has been overwritten w/o getting SUMFLG_SAME_OFFSET set, we remove the
checksum from the list. This will speed up files that have a lot of
identical checksum blocks (e.g. sequences of zeros) that we can't use
due to them not getting marked as being the same. Patch provided by
Michael Chapman.
- Standardized the format of the opening comment, including adding a
brief description of what's in the file for those that lacked it.
- Added some missing copyright lines.
- Some minor whitespace tweaks (in a few of the files).
due to timing tests showing that the per-byte modulus calculation
slowed down regular sized files. Kept the other improvements
because they lessened our memory use and actually sped up the code.
Stuart Inglis