* Add 4 KiB logical block footprint tracking to stats
This adds a 'Number of 4 KiB logical blocks touched' metric to the --stats output to decouple network delta payload from actual local file modifications. Previously, a small amount of literal data scattered across a file (especially with --inplace) could result in a massive number of local file write operations with no visibility, and large sparse files masked their true write opreations (ignoring punch holes).
Technical details:
- Implemented a stateful block tracker in the receiver that calculates touched 4K boundaries using file offsets and lengths, including strict lseek awareness to accurately skip sparse file holes.
- Enforced strict per-file lifetime state with a reset hook inside receive_data(), successfully mitigating POSIX file descriptor (FD) recycling state leaks.
- Created MSG_BLOCK_STATS multiplex message to tunnel the block footprint safely out of the isolated receiver process and relay it over the network.
- Bumped PROTOCOL_VERSION to 33 and SUBPROTOCOL_VERSION to 8392 for safe PR testing.
- Added test suite covering contiguous, scattered, zero-byte, sparse file (hole-skipping), multi-file (FD reuse), batch mode, and maximum-I/O boundary conditions.
* io: fix block stats integration
---------
Co-authored-by: Zen Dodd <mail@steadytao.com>
A filter rule that fails to parse was printed back verbatim. When the
rule came from a file rather than an argument, that text is file
CONTENT, and the peer picks which file gets merged: a per-directory
merge rule travels over the protocol, so no argument of ours ever names
it and nothing a wrapper can see mentions it either. Any line that is
not valid filter syntax therefore came straight back to the peer -- a
read-any-line oracle over an rrsync restricted account or a daemon
module, neither of which confines the merge open.
The syntax errors turned out to be the smaller half. The MATCH trace
names the pattern that acted, and report_filter_result() logs at level 1
for a sender or generator, so plain -vv -- no --debug, nothing a stock
client cannot send -- returns a server-side merge file's rules:
[generator] protecting file X because of pattern <the file's text>
So provenance is carried on the rule itself (FILTRULE_FROM_FILE), not
just in the parser: a deferred ":" merge is processed long after the
file that named it was read, and its own name is file content too.
TEXT_FROM_FILE() consults the parse-time context and the rule, so both
the immediate and the deferred paths redact.
Rather than test the provenance at each message -- which is how the last
few of these were found, one at a time, after the ones before them were
fixed -- every string that is or is built from a rule's own text goes
through rule_text(). It returns the text for an argument-supplied rule
and a description of where it came from otherwise, so a message added
later cannot reintroduce the leak by forgetting to check, and there is
one place to audit. rule_detail() does the same for the extra detail a
message adds ABOUT the text: a character of it, an offset into it, the
[not found] bit.
Thirteen sites now route through them: the syntax errors; the modifier
character (one byte of the file, a slower oracle but still one); the
failed-open and merge-depth messages, whose pathname is file content
whenever a rule named it -- and errno with them, since it answers "does
this path exist"; both over-long messages, the deferred one of which
needed no verbosity at all; both merge-name overflows; the long-named
directory error; the [not found] openability bit; the match trace; the
add_rule, parse_filter_file and daemon-hidden traces; and the per-dir
mergelist label, which had the name baked in.
rule_detail() covers more than it first looks: the trailing-whitespace
CAUTION is computed from the rule's last byte, and "hidden by daemon
filter" distinguishes a daemon-filter rejection from an ordinary open,
so both would answer questions about text the peer cannot see.
The regression proves the chokepoint rather than the sites: making
rule_text() return its input unconditionally fails the test. It also
pins what must NOT change for the user's own rules -- the whitespace
warning still fires, and an over-long argument rule is still reported at
full length (the helper buffers at BIGPATHBUFLEN, as rprintf does, so
redaction does not quietly truncate what the user typed).
Bounded and left alone: the numeric rflags in the FILTER2 trace and the
in/exclude wording still describe a file-derived rule without quoting
it, and the daemon's own FLOG line records the name it filtered -- that
one goes to the operator's log, not the peer.
Rules given AS arguments are still echoed in full -- that text is the
user's own, and hiding it would only make ordinary typos harder to fix.
Where a rule did come from a file, the diagnostic names the file and
line instead, which is more useful anyway.
Two things the location itself needed: fname can point into
parse_merge_name()'s static buffer, which a merge rule inside the same
file overwrites while we are still reading it, so a rule after a nested
merge was blamed on the nested file -- keep our own copy. And a CRLF
pair was counted as two line endings while word-split mode counted
tokens rather than lines, so the number pointed at nothing; consume the
LF of a CRLF (preserving the byte for the next rule if pushback ever
fails), and report word-split sources without a line number.
Not covered, deliberately: a rule's provenance is not serialized by
send_filter_list(), so it does not survive to the far side. That is
right -- only the client sends that list, and the server already knows
the patterns the peer gave it.
The header claimed protocol-level crafting of MSG_IO_ERROR is covered by
msg-io-* crafted-server tests; no such test exists (msg-io-timeout-zero
crafts MSG_IO_TIMEOUT). Say what the test actually locks down.
A peer-supplied MSG_IO_ERROR value was OR'd into the local io_error
without masking, allowing a malicious peer to set arbitrary bits that
propagate to exit codes and get re-forwarded to other peers.
Fix: mask the incoming value with IOERR_VALID_MASK (IOERR_GENERAL |
IOERR_VANISHED | IOERR_DEL_LIMIT) before OR'ing.
Test: testsuite/ki62-io-error-mask_test.py
hashtable_create() and the grow path computed the slot-array byte count as
new_array0(char, size * node_size) in 32-bit int arithmetic; for a large
peer/data-driven size the product wrapped to a tiny value, bypassing my_alloc's
--max-alloc guard (which only saw the already-wrapped count), so the table was
under-allocated while tbl->size kept the huge size -- a later node access then
ran out of bounds (heap overflow; ASan-confirmed). Pass size and node_size as
SEPARATE factors so my_alloc checks both before multiplying; guard each *2
doubling against int overflow BEFORE it happens; make HASH_LOAD_LIMIT divide
before multiplying; promote the HT_NODE index multiply to size_t; and test
size < 16 first so a negative req short-circuits the size-1 (INT_MIN UB).
flist_expand()'s int growth math (used+extra and *=4 / *=2 / += FLIST_LINEAR)
could overflow past INT_MAX on a very large file list; guard each operation
before it overflows and refuse rather than under-size the realloc.
(Leonid Bugaev May-2026 re-audit, KI-11/12/13.)
Adds the coverage/regression tests from the code-scanner run and the
gcov plumbing they rely on:
- scanner-argv-bounds, scanner-batch-flag-mismatch,
scanner-delete-delay-overread, scanner-daemon-log-checksum:
regression tests for the argv/-v/--info/--skip-compress bounds, the
batch metadata-ndx corruption, the read_delay_line off-by-one, and
the daemon -c/%C checksum-slot leak.
- daemon-proxy-protocol, daemon-early-exec-nameconv, daemon-auth-group,
daemon-standalone-detach, misc-coverage, nonroot-restrictive-perms,
backup-acl-xattr-cache: daemon and path coverage tests.
- rsyncfns.py: CAP_MKNOD probe in devices_supported().
- gcov_flush() macro (rsync.h) + calls in the daemon fork/_exit paths
(clientserver.c, socket.c); no-op without --enable-coverage. Makefile.in
COVERAGE_EXCLUDE / gcovr / setuid .gcda refinements.
- CI: list the new TCP/root/ACL tests in the per-platform
RSYNC_EXPECT_SKIPPED sets.
Co-authored-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Refuse malformed/hostile wire input that could crash or corrupt the receiver:
io.c (out-of-range file index, count*blength OFF_T overflow, read_args NUL room,
deferred in_multiplexed), flist.c (sub-flist after the final flist is freed,
FLAG_HLINKED on dirs / gated on preserve_hard_links, parent_ndx bound, cleared-
slot ndx in the transfer phase), hlink.c (undeclared cross-flist gnum -> error not
assert), match.c (clamp peer flength, re-check len before want_i), log.c (drop
peer-reachable asserts and F_SUM deref), exclude.c (merge-file recursion cap,
trailing-backslash heap fix), lib/pool_alloc.c (ASan redzone for pool underflow).
The core symlink-race (TOCTOU) defense for the CVE-2026-29518 class: a portable
secure resolver that walks a path one component at a time holding an O_NOFOLLOW
dir fd per level (the dirfd-stack), plus the do_*_at() filesystem wrappers, the
held-directory fd cache, secure_relative_open[_at](), the operator-path ownership
walk (owner_walk_parent / open_no_attacker_symlinks, follow a uid0/euid symlink,
refuse a foreign one) and its module-root confinement (abspath_excluded_by_module).
util1.c routes change_dir / robust_rename / make_path / handle_partial_dir through
it; the resolver bounds its deep-path fd use against RLIMIT_NOFILE. Also drops the
obsolete android.c openat2 path and confines delete.c via the held dirfd.
read_del_stats() in main.c accumulates 5 wire-supplied counts into
the int32 stats.deleted_files field:
stats.deleted_files = read_varint_bounded(..., MAX_WIRE_DEL_STAT, ...);
stats.deleted_files += stats.deleted_dirs = ...;
stats.deleted_files += stats.deleted_symlinks = ...;
stats.deleted_files += stats.deleted_devices = ...;
stats.deleted_files += stats.deleted_specials = ...;
With the previous MAX_WIRE_DEL_STAT = 2^30 (1.07 GB) the worst-case
sum is 5 * 2^30 = 5.37 GB; three maximal values already exceed
INT32_MAX = 2.15 GB on the third "+=", triggering signed integer
overflow (C99 6.5/5 -- undefined behaviour, the compiler may assume
it cannot happen and elide subsequent checks).
The bound was introduced in f0155902 ("defence-in-depth: bound
wire-supplied counts and lengths") with a commit message claiming
"per-summand cap so the total can't overflow", but 2^30 * 5 does
overflow. Lower the per-summand cap to 2^28 (= 268M) so the worst
case is 5 * 2^28 = 1.34 GB < INT32_MAX with margin. 2^28 deletions
per category is still vastly above any plausible real transfer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Multiple receiver-side fields read from the wire were trusted
without upper-bound checks. A hostile peer could either request
extreme allocations (DoS via --max-alloc) or, on platforms where
read_varint returned a negative value, push ~SIZE_MAX through the
size_t conversion to wrap downstream length checks.
Introduce read_int_bounded(), read_varint_bounded() and
read_varint_size() in io.c so wire-derived integer ranges are
checked at the read site rather than scattered across each
caller, with RERR_PROTOCOL on out-of-range input.
Apply the bounded primitives to:
- sum->count (checksum count -- previously could overflow
(size_t)count * xfer_sum_len on 32-bit with raised max-alloc)
- xattrs: count, name_len, datum_len, plus rel_pos overflow
detect to stop chain wrapping the num accumulator
- acls: ida-entry count
- flist: file mode S_IFMT validation, modtime_nsec range check
- delete-stat counters in main: per-summand cap so the total
can't overflow a signed 32-bit accumulator
Reporters include Joshua Rogers (checksum-count overflow finding).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Put sum2_array into sum_struct to hold an array of sum2 checksums
that are each xfer_sum_len bytes.
- Remove sum2 buf from sum_buf.
- Add macro sum2_at() to access each sum2 array element.
- Throw an error if a sums header has an s2length larger than
xfer_sum_len.
- Size flist checksum data to hold the active size, not the max.
- Add a negotiated hash method to the daemon auth code.
- Use EVP for all openssl digests. This makes it easy to add more
openssl digest methods and avoids deprecation warnings.
- Support a way to re-enable deprecated digests via openssl conf
file and allow a default file to be configured.
- Supply a simple openssl-rsync.cnf file to enable legacy digests.
Some systems apparently put strlcpy() into a separate bsd/strings.h file
without putting the function into a separate library. Thus, configure
finds that the function exists for linking but the build does not have
the declaration (which rsync only supplies if it is also supplying its
own version of the function).
- Make the SIMD ASM code off by default. Use configure --enable-simd-asm
to enable.
- Allow MD5 ASM code to be requested even when OpenSSL is handling MD4
checksums. Use configure --enable-md5-asm to enable.
In 2004, an allocation optimization has been added to the file
list handling code, that preallocates 32k of file_struct pointers
in a file_list. This optimization predates the incremental
recursion feature, for which it is not appropriate anymore. When
copying a tree containing a large number of small directories,
using the incremental recursion, rsync allocates many short
file_lists. Suddenly, the unused file_struct pointers can easily
take 90-95% of the memory allocated by rsync.
- use `grep -E` and `grep -F` (`egrep` and `fgrep` are non-standard)
- use same hashbang style for all test scripts
- use explicit comparisons in test scripts
- remove redundant ; from test scripts
- make test script not executable, just like all the other scripts
- unify codestyle across all test scripts
- make openssl license exception clearer by having it at the top
- use modern links in COPYING. The text now matches:
https://www.gnu.org/licenses/gpl-3.0.txt
- fix typo
- Rename unchanged_file() to quick_check_ok().
- Enhance quick_check_ok() to work with non-regular files.
- Add a get_file_type() function to the generator.
- Use the new functions in the generator code to make the logic simpler.
- Fix a bug where the `--alt-dest` functions were not checking if a
special file fully matched the non-permission mode bits before
deciding if we have found an alt-dest match.
- Enhance the `--info=skip --ignore-existing` output to include extra
info on if the existing file differs in type or passes the standard
quick-check logic.
- Add `--info=skip2` that authorizes rsync to perform a slow checksum
"quick check" when ignoring existing files. This provides the uptodate
and differs info even if we need to checksum a file to get it.
- Use C99 flexible arrays when possible, and fall back on the existing
fname[1] kluges on older compilers.
- Avoid static initialization of a flexible array, which is not really
in the C standard.
Zen Dodd