* human-readable: group file-list and directory counts
Adds the thousand separators as file-list is computed. Useful on large
dirtrees:
Behavior before: 2523297101 files to consider
After: 2,523,297,101 files to consider (2.523.297.101 with a comma decimal point)
Also covers the file-list counter, "created N directories" and --max-delete.
* NEWS: record human-readable output changes
* Add 4 KiB logical block footprint tracking to stats
This adds a 'Number of 4 KiB logical blocks touched' metric to the --stats output to decouple network delta payload from actual local file modifications. Previously, a small amount of literal data scattered across a file (especially with --inplace) could result in a massive number of local file write operations with no visibility, and large sparse files masked their true write opreations (ignoring punch holes).
Technical details:
- Implemented a stateful block tracker in the receiver that calculates touched 4K boundaries using file offsets and lengths, including strict lseek awareness to accurately skip sparse file holes.
- Enforced strict per-file lifetime state with a reset hook inside receive_data(), successfully mitigating POSIX file descriptor (FD) recycling state leaks.
- Created MSG_BLOCK_STATS multiplex message to tunnel the block footprint safely out of the isolated receiver process and relay it over the network.
- Bumped PROTOCOL_VERSION to 33 and SUBPROTOCOL_VERSION to 8392 for safe PR testing.
- Added test suite covering contiguous, scattered, zero-byte, sparse file (hole-skipping), multi-file (FD reuse), batch mode, and maximum-I/O boundary conditions.
* io: fix block stats integration
---------
Co-authored-by: Zen Dodd <mail@steadytao.com>
* main: allow --contimeout for daemon connections over --rsh
The --contimeout guard rejected the option whenever connect_timeout was
set and the connection was not being made by a socket, so a daemon
reached through a remote shell (daemon_connection == 1) was refused the
same way as a plain non-daemon remote-shell transfer. That is how
rsync-ssl talks to a daemon: it runs rsync with --rsh pointing at its
helper, so "rsync-ssl --contimeout=60 host::mod" died with "The
--contimeout option may only be used when connecting to an rsync
daemon" before it could even connect.
Only reject --contimeout when there is no daemon connection at all, so
the option is accepted for a daemon-via-rsh connection (and still works
unchanged for the socket daemon path, which returns earlier through
start_socket_client).
Add a regression test that drives a daemon-via-rsh invocation and
asserts the option is not rejected, while a plain remote-shell
destination is still refused.
* main, io: honor --contimeout for daemon-via-rsh connections
The previous commit made --contimeout legal for a daemon reached
through a remote shell, but rsync still did not act on it: the socket
path times its connect() via an alarm, while the daemon-via-rsh path
spawned the helper and then blocked in start_inband_exchange() waiting
for the daemon greeting with no connect timeout at all. A hung helper
-- e.g. rsync-ssl's openssl connect to an unreachable host -- would hang
the transfer forever.
Give the client a second wall-clock deadline, client_connect_deadline,
set from --contimeout around do_cmd() + start_inband_exchange() and
cleared once the greeting exchange finishes. handshake_poll_timeout_ms()
now also checks it and exits with RERR_CONTIMEOUT ("timeout waiting for
daemon connection"), matching the socket path's exit code, instead of the
daemon-handshake deadline's RERR_TIMEOUT.
This covers the helper spawn, its connect/TLS handshake, and the greeting
exchange as one connection-establishment phase, so it works uniformly for
openssl, stunnel, and gnutls rsync-ssl backends.
Extend the contimeout-rsh test to drive a helper that never connects and
assert rsync aborts with exit 35 within the timeout, and that a working
daemon-via-rsh connection still completes.
* Add IDN support
rsync can now connect to IDN (internationalized domain name) hosts, and
IDN names are recognized in a daemon's hosts allow/deny.
* idn: convert host names label by label
The IDNA mapping folds some non-ASCII characters onto ASCII ones, so
running a whole hosts allow/deny token through idn2_to_ascii_8z() could
hand back a pattern the admin never wrote: a "＊" (U+FF0A FULLWIDTH
ASTERISK) entry came back as "*" and let every host in.
Convert label by label instead, keeping an ASCII label byte for byte and
using a converted label only when it comes back as a bare A-label. An
ASCII-only config now behaves as it did before there was IDN support, and
a token that cannot be converted is left alone and so matches nothing. The
client side shares the same helper, and neither side truncates a name at
its 1024-byte buffer any more. strlower() folds only ASCII now, since its
one caller is the hosts allow/deny list, which can hold UTF-8.
Adds testsuite/daemon-access-idn and extends testsuite/idn to cover
Unicode, punycode, mixed-case and invalid input on both sides.
* idn: hand libidn2 the same flags on both sides
The client path called idn2_lookup_ul() without IDN2_NFC_INPUT while the
daemon path passed it to idn2_to_ascii_8z(). Both normalize either way --
idn2_lookup_ul() ors the flag in itself, and TR46 normalizes as it maps --
but there is no reason for the two calls to read differently, so pass one set
of flags from one place. The flag asks libidn2 to normalize the label rather
than promising that it already is: it gates the u32_normalize() call, and
without it a decomposed label comes back IDN2_NOT_NFC.
Adds composed/decomposed cases to testsuite/idn, which sees the exact host
name rsync hands out, and a decomposed hosts allow token to
testsuite/daemon-access-idn.
sigusr2_handler called output_summary() (rprintf/vsnprintf/fwrite/iconv/malloc)
and close_all() (fstat/shutdown/close) directly from signal context before
_exit(). SIGUSR2 is sent by the generator/parent to tell the receiver child to
print its summary and exit; if it interrupted the receiver while it was inside
malloc/stdio, the handler re-entered those and could deadlock or corrupt.
The handler now only sets a flag (got_sigusr2, a volatile sig_atomic_t); the
actual summary + shutdown moves to receive_sigusr2(), run from a safe point in
the receiver's post-transfer wait paths:
- the perform_io() flag checks (next to got_kill_signal);
- the safe_read()/safe_write() loops (which a --read-batch / --write-batch fd
uses without going through perform_io);
- the whine_about_eof() kluge loop, where the receiver waits out the race of
the sender dying before the kill-signal arrives -- this loop polls for the
signal, so without the flag check it slept the full 10s and then errored with
RERR_STREAMIO instead of exiting cleanly;
- the trailing `while (!got_sigusr2) msleep()` loop in do_recv().
(Leonid Bugaev May-2026 re-audit, KI-14.)
hlink.c must confirm S_ISREG before quick_check_ok(FT_REG,...) reads
F_SUM, and start_server() must set sender_keeps_checksum when a daemon
sender runs -c with a %C log format so make_file() allocates
SUM_EXTRA_CNT. Without these, F_SUM() reads past the pool slot and (for
%C) hex-encodes adjacent heap into the transfer log.
Co-authored-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
The daemon exclude/filter chain is a name-based visibility/tamper filter, as in
stock rsync (verified against 3.2.7): a symlink whose own name is not excluded is
followed to an excluded target, and the documented symlink defense is `munge
symlinks`, not the filter. Collapse ".." (via sanitize_path) before the daemon
dest / temp-dir / backup-dir / partial-dir / basis filter checks so a "../excluded"
path is matched by name like 3.2.7 (clean_fname's CFN_COLLAPSE_DOT_DOT_DIRS does
not collapse "a/b/c/../../../secret"), and keep a leading "/" for a "path = /"
module so an absolute filter rule still matches. The module-ROOT confinement of
operator paths is unchanged (previous commit); only the in-module name match is
restored to its 3.2.7 behaviour.
Route every operator-supplied directory path (--temp-dir / --partial-dir /
--backup-dir / alt-dest basis) and the transfer engine's own dest/source opens
through the secure resolver + ownership walk, so a symlink owned by another uid
can no longer redirect a read, a backup, a staging open, a rename, an unlink or a
new-destination create outside the module. Covers backup.c, generator.c (alt/
link/in-place basis), receiver.c (basis open), sender.c (remove-source / source
open / copy-links leaf), rsync.c (held-fd attr stat), clientserver.c (pid-file
parent pin), and main.c (relative-basis make-absolute, mkpath dest-arg guard).
--insecure-links / "insecure links = yes" is the local opt-out (a daemon never
honors a peer-forwarded one).
A single-file --mkpath copy whose destination parent does not exist
failed under --dry-run: make_path() only *reports* the directories it
would create in a dry run, so change_dir#3 then tried to chdir into a
parent that isn't there and aborted with "change_dir#3 ... failed".
When the parent is genuinely missing in a dry run, skip the chdir and
mark the destination as not-yet-present (dry_run++), exactly as the
multi-file/dir-creation path already does, so the generator doesn't
probe the missing tree. Gating it on the missing-parent case keeps an
ordinary file-to-file dry run chdir'ing into and itemizing against an
existing destination.
Fixes: #880
Co-authored-by: Stiliyan Tonev (Bark) <stiliyan21@gmail.com>
Without --secluded-args, the client's safe_arg() backslash-escapes shell
and wildcard chars in option values before sending them to the server, so
--chown's --usermap=*:user is transmitted as --usermap=\*:user. Over ssh a
remote shell removes the backslashes before rsync parses the args, but a
daemon has no shell and read_args() stored option args verbatim -- so the
receiver saw the literal "\*", the usermap/groupmap wildcard never matched,
and the module's configured uid/gid won instead. A regression from the
secluded-args hardening; rsync 3.2.3 (protocol 31) worked.
Un-backslash option args in read_args() on the daemon's first
(non-protected) read, mirroring what the ssh-side shell does. File args
after the dot are already handled by glob_expand(); the protected (NUL,
already-unescaped) re-read and the server's stdin read pass unescape=0 so
their raw args are left untouched.
Fixes: #829
Two test-coverage build knobs (both behaviour-neutral by default):
--enable-coverage appends '--coverage -fprofile-update=atomic -O0' and adds
a 'make coverage' target (whole suite, run serially, then
gcovr HTML with branch + decision coverage). rsync forks
and its children exit without running the gcov atexit
flush -- the generator via its SIGUSR1 handler
(_exit_cleanup) and the receiver via the SIGUSR2 handler
-- so under GCOV_COVERAGE we call __gcov_dump() at both, or
receiver.c/generator.c record no coverage at all.
--disable-openat2 gates the Linux openat2(RESOLVE_BENEATH) sites in syscall.c
on HAVE_OPENAT2 (defined by default), so disabling it forces
the portable per-component O_NOFOLLOW resolver to run as the
primary on ordinary Linux -- exercising and
coverage-counting that fallback tier without a pre-5.6
kernel. NOTE: coordinate with the parallel syscall.c
path-resolution restructure.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two cumulative-snprintf patterns in log.c (rsyserr) and main.c
(output_itemized_counts) had the shape
len = snprintf(buf, sizeof buf, ...);
len += snprintf(buf+len, sizeof buf - len, ...);
with no guard between calls. snprintf returns the would-have-been
length on truncation, so a truncated first call leaves
"sizeof buf - len" as a negative-then-promoted-to-size_t value,
underflowing into a huge size_t and writing past buf.
Realistic exposure is small in both cases (log header well under
buffer, only ~5 itemized iterations writing ~25 chars each into a
1024-byte buffer) but the defect class matches bb0a8118 and the
fix is cheap. Guard before each subsequent call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Multiple receiver-side fields read from the wire were trusted
without upper-bound checks. A hostile peer could either request
extreme allocations (DoS via --max-alloc) or, on platforms where
read_varint returned a negative value, push ~SIZE_MAX through the
size_t conversion to wrap downstream length checks.
Introduce read_int_bounded(), read_varint_bounded() and
read_varint_size() in io.c so wire-derived integer ranges are
checked at the read site rather than scattered across each
caller, with RERR_PROTOCOL on out-of-range input.
Apply the bounded primitives to:
- sum->count (checksum count -- previously could overflow
(size_t)count * xfer_sum_len on 32-bit with raised max-alloc)
- xattrs: count, name_len, datum_len, plus rel_pos overflow
detect to stop chain wrapping the num accumulator
- acls: ida-entry count
- flist: file mode S_IFMT validation, modtime_nsec range check
- delete-stat counters in main: per-summand cap so the total
can't overflow a signed 32-bit accumulator
Reporters include Joshua Rogers (checksum-count overflow finding).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Size flist checksum data to hold the active size, not the max.
- Add a negotiated hash method to the daemon auth code.
- Use EVP for all openssl digests. This makes it easy to add more
openssl digest methods and avoids deprecation warnings.
- Support a way to re-enable deprecated digests via openssl conf
file and allow a default file to be configured.
- Supply a simple openssl-rsync.cnf file to enable legacy digests.
- Avoid implied rules on generator and (with extra certainty) on server
- Add -R implied-directory path elements as directory includes
- Log about extra file-list checking using a new --debug=FILTER3 level
The new default is to protect args and options from unintended shell
interpretation using backslash escapes. See the new `--old-args` option
for a way to get the old-style splitting. This idiom was chosen over
making `--protect-args` enabled by default because it is more backward
compatible (e.g. it works with rrsync). Fixes#272.
Without a DISPLAY var, ssh won't try to forward X11 when making an
ssh connection. This patch also makes use of setenv() and unsetenv()
if they are available.
- Rename daemon_over_rsh -> daemon_connection since it is also used to
indicate if a non-rsh daemon connection is active.
- Move the daemon-over-rsh exception out of server_options() to the one
caller that needs that behavior.
- Don't allow noop_io_until_death() to be short-circuited when talking
to a daemon over a socket, because it can't send errors via stderr.
- All the memory-allocation macros now auto-check for failure and exit
with a failure message that incudes the caller's file and lineno
info. This includes strdup().
- Added the `--max-alloc=SIZE` option to be able to override the memory
allocator's sanity-check limit. It defaults to 1G (as before).
Fixes bugzilla bug 12769.
graysky