* Add --delay-symlinks to create symlinks after all files are in place
The generator creates a symlink as soon as it reaches it in the file
list, while regular files arrive later through the sender and receiver
(and, with --delay-updates, are only renamed into place at the end).
For the rest of the transfer a new symlink can point at a file that
isn't there yet, so anything reading the destination during the sync --
e.g. a loader resolving libfoo.so -> libfoo.so.2 -- can fail.
The new --delay-symlinks option makes the generator defer new and
changed symlinks and create them once the receiver reports that every
file, including any --delay-updates renames, is in place. Each link is
itemized and logged after it has been created, before the generator
ends the delay-updates phase, and the deletion stats are sent after any
directory that a deferred link replaces.
The option implies --no-inc-recursive on the receiving side and is
passed to a remote receiver, which must also support it. It is an
error to combine it with --remove-source-files, --hard-links,
--compare-dest, --copy-dest, or --link-dest, or to use it with a
protocol older than 29. support/rrsync accepts the option and refuses
it under -no-overwrite, as it does --delay-updates.
Add delay-symlinks_test.py, which holds a push partway through the
link's target to check that new and changed links aren't created early,
and rrsync-no-overwrite-delay-symlinks_test.py.
* Document --delay-symlinks limits and test how it is passed to the server
The man page now says the option narrows the inconsistency window rather
than making the update atomic, that an interrupted transfer leaves new
symlinks missing and changed ones at their old targets, and that
--delete-before can also remove an old target early.
Add delay-symlinks-remote_test.py. Through a remote shell that records
the server command, it checks that the option is passed to a remote
receiver on a push but not to a remote sender on a pull, and that
incremental recursion is off in both directions. It also checks that
--protocol=28 fails with the protocol error, and that a push to an older
rsync (old_versions/rsync_3.4.1) is refused before anything is
transferred while a pull from it works.
* testsuite: seed delay-symlinks fixture with rsync
* NEWS: document --delay-symlinks
---------
Co-authored-by: Zen Dodd <mail@steadytao.com>
* human-readable: group file-list and directory counts
Adds the thousand separators as file-list is computed. Useful on large
dirtrees:
Behavior before: 2523297101 files to consider
After: 2,523,297,101 files to consider (2.523.297.101 with a comma decimal point)
Also covers the file-list counter, "created N directories" and --max-delete.
* NEWS: record human-readable output changes
* human-readable: do not add trailing space in output
Co-authored-by: Zen Dodd <mail@steadytao.com>
* human-readable: use dynamic precision length
Let's lower precision for huge numbers.
The output used to be:
3.45M -> 46.73M -> 523.11M -> 1.24G -> ...
With this change the code always gives the three most significant digits:
3.45M -> 46.7M -> 523M -> 1.24G -> ...
Co-authored-by: Zen Dodd <mail@steadytao.com>
* human-readable: also handle num < mult with the same code
... just make sure no precision is added.
Co-authored-by: Zen Dodd <mail@steadytao.com>
* human-readable: add an "i" to unit to indicate binary (1024) base
Co-authored-by: Zen Dodd <mail@steadytao.com>
* human-readable: also use it to format rate in progress
Let's also simplify the code for rate in progress, and benefit from
same functionality.
Co-authored-by: Zen Dodd <mail@steadytao.com>
* human-readable: append suffix "B" for byte to ofs...
... and also fix the formatting in man page.
Co-authored-by: Zen Dodd <mail@steadytao.com>
* human-readable: unbreak for do_big_dnum() with fract
Co-authored-by: Christian Hesse <mail@eworm.de>
* testsuite: cover human-readable output
---------
Co-authored-by: Zen Dodd <mail@steadytao.com>
This drops a lot of `else if` blocks and extends units by "E" (Exa),
which is 2^60 and thus the last to fit into int64.
Co-authored-by: Zen Dodd <mail@steadytao.com>
* Add 4 KiB logical block footprint tracking to stats
This adds a 'Number of 4 KiB logical blocks touched' metric to the --stats output to decouple network delta payload from actual local file modifications. Previously, a small amount of literal data scattered across a file (especially with --inplace) could result in a massive number of local file write operations with no visibility, and large sparse files masked their true write opreations (ignoring punch holes).
Technical details:
- Implemented a stateful block tracker in the receiver that calculates touched 4K boundaries using file offsets and lengths, including strict lseek awareness to accurately skip sparse file holes.
- Enforced strict per-file lifetime state with a reset hook inside receive_data(), successfully mitigating POSIX file descriptor (FD) recycling state leaks.
- Created MSG_BLOCK_STATS multiplex message to tunnel the block footprint safely out of the isolated receiver process and relay it over the network.
- Bumped PROTOCOL_VERSION to 33 and SUBPROTOCOL_VERSION to 8392 for safe PR testing.
- Added test suite covering contiguous, scattered, zero-byte, sparse file (hole-skipping), multi-file (FD reuse), batch mode, and maximum-I/O boundary conditions.
* io: fix block stats integration
---------
Co-authored-by: Zen Dodd <mail@steadytao.com>
* options: accept --max-alloc=0 again, resolved to the parser's own ceiling
3.5.0 rejected --max-alloc=0 (CVE-2026-53794). The stated rationale was that a
zero cap "disabled the per-allocation size cap (the defense behind
CVE-2024-12084)". That was true up to 3.2.7, where the check short-circuited:
if (max_alloc && num >= max_alloc/size)
but 2f9b963a ("Make `--max-alloc=0` safer", 3.3.0) removed the short-circuit and
mapped 0 to SIZE_MAX at parse time, leaving the guard unconditional. Since the
guard admits an allocation only when num < max_alloc/size, num*size stays below
max_alloc at every setting, so the num*size overflow check was armed for 0 just
as for any other value. From 3.3.0 onward, 0 raised the magnitude ceiling and
nothing else -- and an explicit 8191P raises it exactly as far, is accepted, and
is forwarded to the peer, so rejecting 0 removed no capability.
What it did remove is the only portable spelling. The parser's ceiling is
SIZE_MAX/2, so it tracks the build's word size: on ILP32 the suffix multiplier
alone exceeds the bound, making every P and T value an error whatever the digits
and capping the option at 2047M. Because max_alloc_arg goes on the wire
un-normalized, a 0 was re-resolved by each side against its own SIZE_MAX; any
literal is resolved once on the client and shipped verbatim, so nothing above
2047M survives a 64-bit client talking to a 32-bit daemon. There is no number a
user can compute that does what 0 did, which is what #1056 ran into.
So accept 0 again, but resolve it to SIZE_ARG_MAX (SIZE_MAX/2) rather than
SIZE_MAX, so it lands exactly on the largest value that could also be typed and
is no longer a limit only the 0 spelling can reach. Keep forwarding it
verbatim: that per-side resolution is the property worth having. This leaves
the substantive half of the 3.5.0 hardening -- bounding parse_size_arg() against
the unbounded `size *= atof(size_arg)` -- untouched.
The residual concern, a 0 forwarded to a <= 3.2.7 daemon that honours it, is not
something a client-side check can address: the client is the attacker's own
code, as daemon-max-alloc-zero_test.py noted in its own docstring. Operators
who want peers kept off their cap have `refuse options = max-alloc`.
Also drops the now-unreachable rejection message, which left the min-value error
recommending a value the parser refused ("min: 1.00M or 0 for unlimited").
Tests: max-alloc-zero replaces max-alloc-zero-rejected and
daemon-max-alloc-zero, whose assertions are the behaviour being reverted. It
checks that 0 is accepted, that it reaches the peer as the literal "0" rather
than a resolved number (verified against a negative control that normalizes it),
and that the parser's upper bound still rejects an out-of-range value.
Fixes#1056
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* testsuite: refresh Cygwin expectations
* docs: note max-alloc safety
* Add IDN support
rsync can now connect to IDN (internationalized domain name) hosts, and
IDN names are recognized in a daemon's hosts allow/deny.
* idn: convert host names label by label
The IDNA mapping folds some non-ASCII characters onto ASCII ones, so
running a whole hosts allow/deny token through idn2_to_ascii_8z() could
hand back a pattern the admin never wrote: a "＊" (U+FF0A FULLWIDTH
ASTERISK) entry came back as "*" and let every host in.
Convert label by label instead, keeping an ASCII label byte for byte and
using a converted label only when it comes back as a bare A-label. An
ASCII-only config now behaves as it did before there was IDN support, and
a token that cannot be converted is left alone and so matches nothing. The
client side shares the same helper, and neither side truncates a name at
its 1024-byte buffer any more. strlower() folds only ASCII now, since its
one caller is the hosts allow/deny list, which can hold UTF-8.
Adds testsuite/daemon-access-idn and extends testsuite/idn to cover
Unicode, punycode, mixed-case and invalid input on both sides.
* idn: hand libidn2 the same flags on both sides
The client path called idn2_lookup_ul() without IDN2_NFC_INPUT while the
daemon path passed it to idn2_to_ascii_8z(). Both normalize either way --
idn2_lookup_ul() ors the flag in itself, and TR46 normalizes as it maps --
but there is no reason for the two calls to read differently, so pass one set
of flags from one place. The flag asks libidn2 to normalize the label rather
than promising that it already is: it gates the u32_normalize() call, and
without it a decomposed label comes back IDN2_NOT_NFC.
Adds composed/decomposed cases to testsuite/idn, which sees the exact host
name rsync hands out, and a decomposed hosts allow token to
testsuite/daemon-access-idn.
CAN_HARDLINK_SYMLINK and CAN_HARDLINK_SPECIAL are decided by configure running
linkat() on whatever filesystem the build tree happened to sit on. The
destination is free to disagree, and one host can hold both answers: macOS
builds on APFS, which can hard-link a symlink, and backs up to HFS+, which
returns ENOTSUP.
A build that said yes had no fallback left. try_dests_non() reported the
refusal as a transfer error and returned a matched basis, so the caller created
the entry anyway -- correctly -- and the run still exited 23. Every
neighbouring case copes: a regular file whose link() fails goes to try_a_copy,
and a build compiled without either macro resorts to --copy-dest behaviour.
This was the same situation, discovered a little later, and the only one
treated as fatal.
Take the existing fallback on any refusal. Singling out the "cannot" errnos is
not possible: link(2) documents EPERM both for a filesystem with no hard-link
support and for an ordinary permission refusal, and FUSE reports ENOSYS for the
same thing. It is also what the regular-file path next door has always done
(try_dests_reg -> hard_link_one -> try_a_copy), and consistency between the two
was the point. Where the errno does matter the surrounding transfer says so
anyway: ENOSPC, EDQUOT and EROFS fail the creation independently, EMLINK and
EXDEV mean the link was never possible. EIO alone goes unremarked; reporting
it would put a line into --link-dest's itemised output, so it is left out on
purpose.
Returning -3 rather than -2 keeps the caller out of the "already up to date,
skip it" arm, which under --link-dest would drop the entry entirely. Both
callers give -3 the treatment the compile-time fallback already gets -- clearing
itemizing and code -- because try_dests_non() has itemised the match itself and
would otherwise report the entry twice.
The fallback is silent, matching a build that cannot link these at compile
time; documented under --link-dest instead.
The ownership walk that resolves operator-supplied paths asks who planted a
symlink, not where the path came out, so a symlink owned by uid 0 or the euid
is followed wherever it points. A daemon already narrows that with the served
module root; nothing else has a root to narrow it with.
That leaves a wrapper serving a restricted directory over a remote shell with
no way to bound the resolution. rrsync can vet the argv it is handed, but
filter rules travel over the protocol instead: a dir-merge rule can name a
merge file outside the restricted dir and the server reads it in as rules.
Redacting the resulting diagnostics does not close it, because an exclude-only
merge produces none -- every line becomes a pattern, so nothing fails to parse,
and the client reads the file's contents off which of its own names went
missing from the file list.
--confine-root gives that wrapper the root the daemon has. The existing module
check becomes a root check that takes its root from module_dir when we are a
daemon and from the option otherwise, so daemon behaviour is unchanged; a
daemon ignores the option outright, since it arrives in a peer-supplied argv
and could only widen the module.
The tracker is seeded from getcwd() rather than curr_dir, which is only the
lexical name change_dir() was given: descend into a source argument through a
trusted symlink and the two sit at different depths, so a ".." that really
escapes looks like it landed inside. When the cwd cannot be read there is
nothing to measure against and the open is refused -- an empty tracker does not
deny by itself, because a leading ".." pops nothing from it and an empty path
reads as an ancestor of the root.
An fd pin (/proc/self/fd/N, which rrsync uses so no later symlink can redirect
a validated option path) is spelled outside the root by construction, so the
walk transits the pin namespace and the pin is judged by what it points at.
Only a bare ".../fd/<digits>" is resolved that way, and one that will not
readlink to an absolute path is refused; rrsync's ".../fd/N/<leaf>" spelling
resolves through the magic link and has its remaining components checked
normally.
--insecure-links is refused alongside it: that opt-out returns the legacy open
before the walk that enforces the root runs, so the pair would have quietly
meant no confinement at all.
-D and --no-D do two jobs at once: they decide whether devices and
special files are created, and they decide whether those entries carry
their rdev fields on the wire. send_file_entry() and recv_file_entry()
frame those fields with the same condition, but each end evaluates its
own preserve_devices/preserve_specials, so the two only agree because
both normally parse the same command line.
That makes --no-D unusable for a wrapper that controls one end of a
connection and wants to deny creation. Give it to the receiver alone
and the client's -D sender writes rdev the receiver never reads: the
file list desynchronises from that entry on. A FIFO or socket breaks
below protocol 31 -- a hang at 29, "File-list index 0 not in 0 - -1" at
30 -- and a device node breaks at EVERY protocol, current ones included,
because its arm of the condition has no protocol clause at all.
--drop-D separates the two jobs: it refuses the creation and leaves the
encoding alone. The entry is skipped through the existing non-regular
fall-through, so the visible result matches --no-D, and because it
touches no wire state it can be applied to one end by itself.
It has no effect on a sending rsync, which creates nothing.
Corrections to the preceding man page rewrite, found in review:
- --max-alloc: 0 is rejected as invalid since 3.5.0 (CVE-2026-53794),
it does not mean "no limit"
- -t: restore "next transfer" -- it is the subsequent run that behaves
like --ignore-times when times aren't preserved; also note that the
timestamp-range limit is protocol < 30, not just an old remote
- --max-delete: re-add the caution that a pre-3.0.0 CLIENT treats
--max-delete=0 as unlimited (a new client forwards it as -1, so the
client version is the boundary that matters)
- --stats: the deleted-files line needs negotiated protocol >= 31, not
just a new remote rsync
- --version: the server does not ignore it; only the repeated-option
JSON output is client-side only
- SECURITY: restore the sentence distinguishing the pre-transfer -c
checksum from the whole-file transfer-verification checksum (which
runs unless --checksum-choice=none); fix a singular/plural clash in
the escape-path paragraph
- remove a leftover XXX comment marker (the paragraph it questioned is
correct: user@ sets the rsync module user, ssh -l sets the login)
- typo/mechanical: "If it is sufficient", "is output in a JSON",
missing space before --read-batch, unbalanced paren after
rsyncd.conf(5), missing "the" before --whole-file, trailing
whitespace
Updates to make statements more definite and explicit and improve
English expression, including:
- Globally change dir to directory, arg to argument, parens to
parentheses.
- Remove some references to behaviour of past versions, on the grounds
that the man page should describe the current version, and only
describe past behaviour when talking about how to get a current rsync
client to deal with an older remote rsync version.
- In most cases where a version number is mentioned, add the month
and year of release of that version.
- Be more explicit about when "file" means "regular file" vs. "any
filesystem object".
- Remove some parentheses, where the parenthesized comment is important
enough to warrant making it a first-class part of the sentence.
Tighten man-page prose for options whose documented behaviour was
imprecise or outright wrong, matching the behaviour now pinned by the
new oracle tests and verified against the C source:
- strict modes: the real secrets-file rule is "st_mode & 06" (other
read/write) plus root-owner-when-root, so 640 is accepted and 644
rejected -- not the old "any user ID other than the daemon's".
- munge symlinks: rewrite the default in terms of chroot and the "/./"
path split (disabled only for a plain chrooted module serving the
chroot root); drop the bogus "daemon chroot" clause. Fix the helper
reference: support/munge-symlinks is a python script, not perl.
- --no-implied-dirs: spell out that an existing in-tree dest symlink is
followed.
- --files-from: ".." handling is collapse-then-reject-survivors.
- --copy-unsafe-links: describe the lexical unsafe-symlink rule instead
of the old "verbose output" phrasing.
Move the uid/gid/acls/xattrs *_ndx assignments past check_batch_flags():
a mismatched-flag batch otherwise wrote F_XATTR(file) at offset 0 of
every file_struct, clobbering file->dirname. parse_negotiate_str() no
longer short-circuits on am_server, so each side picks its own #1 mutual
digest/checksum/compress choice rather than deferring to the peer's
order; man pages updated to match.
Co-authored-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
--insecure-links and the SECURITY sections in rsync(1)/rsyncd.conf(5), the
name-based exclude/filter + munge-symlinks clarification, the rsync-ssl hostname
note, and the README (incl. the thank-you to Wayne Davison, 2004-2024).
The description of user@host::module transfers over a remote shell only
documented the "ssh -l ssh-user" form, which led readers to conclude that
user@ never reaches the remote shell. In fact, for the simple
`--rsh=ssh user@host::module` form the user@ prefix is used both as the
ssh login user (ssh -l user) and as the rsync-user offered to the module;
the two are the same name. rsync only omits its own -l when the remote
shell command already specifies one, in which case user@ becomes the
rsync-user alone.
Spell out the default behaviour and why the explicit -l is needed to use
a different ssh login than the rsync-user.
The main purpose of the SHA checksums are to allow the daemon auth code
to pick a stonger digest method when negotiating the auth digest to use.
However, the SHA digests are also available for use in file checksums,
should someon really want to use one of them.
The new digests are listed from strongest to weakest at the start of the
daemon auth list, giving them the highest priority.
The new digests are listed from weakest to strongest near the end of the
checksum list, giving them the lowest priority of use for file
checksums.
Zen Dodd