The regression test honestly skips when it cannot force the receiver's
output mkstemp() to fail -- as root (root bypasses DAC) and on Cygwin
(chmod 0555 does not deny the owner a write). The ubuntu, ubuntu-22.04,
almalinux and macOS jobs run `make check` as root, and Cygwin can't
enforce the unwritable directory, so the test skips on all of them.
runtests.py fails a run on any skip-set mismatch, so add the test to
those jobs' RSYNC_EXPECT_SKIPPED lists; the BSD/Solaris jobs run as root
too but enforce no expected-skip set, so they need no change.
Also tighten the pass condition. The post-chmod writability probe already
guarantees the receiver discards (mkstemp must fail), so an exit 0 would
mean the file actually transferred and the discard path was never
exercised -- a silent false-pass. Require exactly exit 23 (the forced
discard leaves the file untransferred); 12 remains the pre-fix crash.
Drives a real sender<->receiver pair (client sender -> daemon receiver,
both the binary under test in the default pipe transport) so the receiver
actually takes the recv_files discard path -- a local `rsync a b` does
not. The basis and source share a leading block so the generator emits
real sums and the receiver gets a block MATCH; the destination directory
is made unwritable so the receiver's output mkstemp() fails and it
discards the delta. Pre-fix the receiver SIGSEGVs in full_fname(NULL),
which the client sees as a protocol-data-stream error (code 12); post-fix
it drains the delta and reports a benign code 23 (or 0).
Skips (exit 77) when run as root, since root bypasses DAC and the
unwritable destination would not make mkstemp() fail -- so the discard
path, and the bug, would never be reached.
Verified red-on-buggy / green-on-fixed against the 0d0399bb receiver.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
receive_data() crashed a receiver that was merely DISCARDING a file's
delta stream. discard_receive_data() calls receive_data() with
fname == NULL and fd == -1, so size_r == 0 and mapbuf == NULL. A normal
block-MATCH token (against a block the basis and source share) then
reaches the !mapbuf branch added in 31fbb17d ("receiver: fix absolute
--partial-dir delta resume"), which calls full_fname(fname). full_fname()
dereferences its argument unconditionally (util1.c: `if (*fn == '/')`),
so fname == NULL faults there -> receiver SIGSEGV.
This is a normal-operation crash with a stock cooperating sender, not an
adversarial one. The generator hands the sender real block sums whenever
the basis is readable and we're in delta mode; the receiver only decides
to discard afterwards, when its output cannot be produced -- e.g. the
destination directory is not writable (mkstemp fails), the basis turns
out to be a directory, or a --partial-dir resume is skipped. A MATCH
token arriving during that discard hit the NULL deref.
The 31fbb17d branch is correct only for a REAL output transfer (fd != -1,
fname valid): there, a block match with no mapped basis is a genuine
protocol inconsistency (the generator promised a basis the receiver could
not open), and honoring it would silently omit those bytes from the
verification checksum or leave a hole, so hard-erroring -- and
full_fname(fname) -- is right. It conflated that with the discard path.
The discriminator is fd, not mapbuf: on the discard path fd == -1 always;
on the real-output inconsistency fd != -1. Scope the "no basis file"
protocol error to fd != -1 (where fname is non-NULL and full_fname is
safe) and, on the discard path (fd == -1), absorb the matched bytes
benignly (offset += len; continue) -- symmetric with the literal-token
handling just above, and restoring the pre-31fbb17d behavior. The
real-transfer inconsistency check is preserved unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A slow or heavily-loaded fleet box can occasionally flake a concurrency-
sensitive test (e.g. a daemon/lsh test under -j8 on a nested-VM Solaris box).
Rather than dropping the whole target to a lower -j, add a per-target
"max_retry" property: after a run, each failed test is re-run on its own up to
max_retry more times, and any that then pass are dropped from the failure list.
Recovered tests are listed in a new "RECOVERED" report section, so a flake is
surfaced, never silently hidden.
Applies to every pass for the target (pipe, tcp, protoNN, nonroot). Default 0
keeps the current no-retry behaviour.
A target can list older "protocols" (e.g. [30, 29]) in the fleet config;
each runs as an extra stdio-pipe pass with runtests --protocol=N, the fleet
analogue of a workflow's check30/check29 steps. The passes reuse the same
parsed RSYNC_EXPECT_SKIPPED list as the default pipe run and appear as protoNN
columns in the report and --timing breakdown. Targets without the key run only
the default protocol and show "-" there.
The example config's ubuntu-2604 target (mirroring ubuntu-build.yml, which has
check30/check29 steps) now sets protocols: [30, 29].
Covers both halves: a --mkpath file-to-file --dry-run must succeed and
match the real run (the #880 abort), and a plain file-to-file --dry-run
onto an existing differing destination must still itemize the real change
rather than report it as brand new. Both compare "--dry-run -i" output
against the real run.
Co-authored-by: Stiliyan Tonev (Bark) <stiliyan21@gmail.com>
A single-file --mkpath copy whose destination parent does not exist
failed under --dry-run: make_path() only *reports* the directories it
would create in a dry run, so change_dir#3 then tried to chdir into a
parent that isn't there and aborted with "change_dir#3 ... failed".
When the parent is genuinely missing in a dry run, skip the chdir and
mark the destination as not-yet-present (dry_run++), exactly as the
multi-file/dir-creation path already does, so the generator doesn't
probe the missing tree. Gating it on the missing-parent case keeps an
ordinary file-to-file dry run chdir'ing into and itemizing against an
existing destination.
Fixes: #880
Co-authored-by: Stiliyan Tonev (Bark) <stiliyan21@gmail.com>
doc/rsync.sgml is a 1996-2002 DocBook user manual (with README-SGML
describing the docbook-utils build) that was long ago superseded by the
markdown man pages. It is unmaintained and referenced by nothing in the
build. This empties doc/.
rsync3.txt and rsyncsh.txt are Martin Pool's 2001 design proposals
("notes towards a new version of rsync", an interactive rsync shell),
neither of which reflects the current implementation. doc/profile.txt is
stale profiling notes. None are referenced by the build, tests, or docs.
This Python 2 test-tree generator (print statements, string.letters,
.next()) has been broken on modern Python for years and is referenced
nowhere in the build, tests, or any script. Drop it.
send_deflated_token() adds a matched block to the compressor history with
deflate(Z_INSERT_ONLY). Our bundled zlib implements Z_INSERT_ONLY (it
produces no output and consumes the input in one call), but a build
against a system zlib lacks it and falls back to Z_SYNC_FLUSH (see the top
of the file), which emits a flush block into obuf. For a large
incompressible matched token that block exceeds AVAIL_OUT_SIZE(CHUNK_SIZE),
so deflate returned with avail_in != 0 and the transfer aborted:
"deflate on token returned 0 (N bytes left)" at token.c
The insert output is never sent -- the receiver rebuilds the matching
history itself in see_deflate_token() -- so loop, resetting the output
buffer, and discard it. Drain with the same condition as the data loop
above: until the input is consumed AND avail_out != 0. Stopping at
avail_in == 0 alone can leave pending output in the deflate stream (a
full output buffer with bytes still buffered), which would then be emitted
by the next real deflate send and corrupt the stream. A bundled-zlib
build still finishes in one iteration.
Fixes: #951
fleettest is a developer tool meant to run on a modern Ubuntu box, so a
bitrot check belongs in its own ubuntu-latest job rather than in the
testsuite (which runs on the BSD/Solaris/macOS/Cygwin matrix, whose
older Pythons may not even parse it).
The job sets up passwordless ssh to localhost, writes a two-target
fleet config that both ssh to localhost (distinct build dirs), and runs
a real fleettest pass. Two targets exercise the parallel multi-target
path and the per-run dir / port isolation; the run exits 0 only if
every cell is OK. Triggered on changes to fleettest.py or this
workflow, manually, and weekly.
Records wall-clock per phase (push, build, each test transport, nonroot)
plus a total in TargetResult, and with --timing prints a breakdown after
the report, sorted slowest-target-first. Targets run in parallel, so the
run is gated by the slowest one; the phase columns show whether that
hold-up is the push, the build, or a test pass. A target that failed
early (no total) falls back to the sum of the phases it reached.
Address review findings on the cleanup paths:
- --cleanup no longer removes a bare <builddir>, only the suffixed
<builddir>-* run dirs it created. This keeps the sweep within its
documented scope and avoids clobbering an unrelated tree.
- Add _unsafe_builddir(): reject empty/root/$HOME and any absolute path
directly under / (e.g. a misconfigured builddir of "/tmp") before
building a destructive command, in both cleanup paths.
- Use `rm -rf --` so a path with a leading dash can't be read as options.
- Soften the docs: run-dir removal on Ctrl-C/kill is best-effort (a
signal arriving mid-push can still leave a remnant for --cleanup).
Each run now builds in its own randomly-named dir on every target
(<builddir>-<run_id>), so two or three fleettest runs can share the same
fleet without colliding on the pushed tree, the build, or the testtmp
scratch. Port collisions were already handled by claim_ports() locks.
The run dir is removed when the run ends -- on success, failure, or
Ctrl-C/kill (atexit + SIGINT/SIGTERM handlers); --keep retains it. A new
--cleanup mode sweeps stray <builddir>-* dirs left by a SIGKILL.
Incremental builds are dropped (every run is a fresh dir + full build):
--no-push removed, --clean removed.
Also look for the fleet config at ~/.fleettest.json first, then
testsuite/fleettest.json (still overridable with --fleet PATH).
Maps every source group to a second group the test user belongs to via a
daemon upload (--groupmap='*:GID') and checks the wildcard took effect.
Runs both arg modes: the default path (the '*' is safe_arg-escaped and the
daemon must un-backslash it -- the regression) and --secluded-args (the '*'
is sent raw over the protected channel, a guard that the fix left that path
alone). Needs no root -- a non-root receiver can chgrp to a member group --
and was verified RED on a pre-fix binary (the escaped '\*' is ignored, gid
unchanged) and GREEN after the fix.
Without --secluded-args, the client's safe_arg() backslash-escapes shell
and wildcard chars in option values before sending them to the server, so
--chown's --usermap=*:user is transmitted as --usermap=\*:user. Over ssh a
remote shell removes the backslashes before rsync parses the args, but a
daemon has no shell and read_args() stored option args verbatim -- so the
receiver saw the literal "\*", the usermap/groupmap wildcard never matched,
and the module's configured uid/gid won instead. A regression from the
secluded-args hardening; rsync 3.2.3 (protocol 31) worked.
Un-backslash option args in read_args() on the daemon's first
(non-protected) read, mirroring what the ssh-side shell does. File args
after the dot are already handled by glob_expand(); the protected (NUL,
already-unescaped) re-read and the server's stdin read pass unescape=0 so
their raw args are left untouched.
Fixes: #829
do_mknod_at() (the symlink-race-safe variant used by a non-chrooted
daemon receiver) calls mknodat()/mkfifoat(), but the at-variant was
gated only on AT_FDCWD. Older Darwin declares AT_FDCWD without
mknodat(), so the build failed with "mknodat undeclared".
Probe mknodat()/mkfifoat() in configure and require HAVE_MKNODAT for the
at-variant; without it do_mknod_at() falls back to do_mknod(), exactly
as it already does where AT_FDCWD is missing. Linux keeps the mknodat
path since HAVE_MKNODAT is defined there.
Fixes: #896
Commit d046525d made my_alloc() calloc every fresh allocation and made
expand_item_list() memset the freshly grown tail, to hand out predictably
zeroed memory. But that forces the kernel to back pages callers never
touch: each per-directory file_list pre-allocates a FLIST_START-entry
(32768) pointer array -- 256KB -- and calloc now zeroes the whole array
even for an empty directory. With incremental recursion over many
directories the resident set explodes; 80000 empty dirs went from ~336MB
to ~10.8GB.
Restore the pre-d046525d malloc/calloc split: fresh allocations use
malloc (so untouched tails stay lazy) and only explicit do_calloc
requests (new_array0) are zeroed. Callers that need zeroed memory
already ask for it, and the full test suite passes.
Fixes: #959
Forces --checksum-choice=xxh64 (an 8-byte transfer checksum) with a
corrupted-prefix --append-verify so the full-checksum redo path runs.
Before the generator capped s2length at MIN(SUM_LENGTH, xfer_sum_len)
this died with "Invalid checksum length 16 [sender]"; the test is RED on
the prior generator and GREEN with the cap. Reproduces on any build that
has xxhash, so it guards the fix without an old-libxxhash host; skips when
xxh64 is absent (a build without xxhash).
sum_sizes_sqroot() capped the strong-sum length at SUM_LENGTH (16), the
legacy MD4/MD5 digest size. Since 0902b52f the sum2 array elements are
xfer_sum_len bytes and the sender rejects a sums header whose s2length
exceeds xfer_sum_len. When the negotiated transfer checksum is shorter
than 16 bytes -- xxh64 (8), used when the build's libxxhash lacks
xxh128/xxh3 (e.g. Ubuntu 20.04) -- the generator still emitted s2length
up to 16, so --append-verify and other full-checksum (redo) transfers
died with "Invalid checksum length 16 [sender]" (protocol incompatibility).
Cap s2length at MIN(SUM_LENGTH, xfer_sum_len): unchanged for any checksum
>= 16 bytes (md5/xxh128/sha1), corrected for short ones. Also closes a
latent over-read of the xfer_sum_len-sized digest buffer.
Android's seccomp sandbox traps openat2() with SECCOMP_RET_TRAP, which
raises SIGSYS and kills the process instead of returning ENOSYS, so the
secure resolver cannot simply try openat2() and inspect errno. Add
openat2_usable() in a new android.c: it probes openat2() once behind a
temporary SIGSYS handler and caches the result.
Gate every SYS_openat2 call on openat2_usable(): in the resolver via an
openat2_beneath() wrapper, and in t_chmod_secure's kernel probe directly,
so a blocked openat2 reports ENOSYS and the caller falls back to the
portable O_NOFOLLOW resolver. Only openat2 is gated -- a plain openat()
(e.g. opening an operator-trusted absolute basedir) is left free.
The probe body compiles only on Android -- __ANDROID__ is a Bionic target
macro, so it is set for NDK cross-builds and native Termux alike and unset
everywhere else, where openat2_usable() collapses to a constant 1. Link
android.o into the secure-resolver test helpers too so their self-tests
survive on Termux.
Adapted from PR #909.
The openat2 secure resolver in syscall.c needs struct open_how and
RESOLVE_BENEATH from <linux/openat2.h>, not only the SYS_openat2 syscall
number. Some setups expose the syscall number via glibc without the
kernel header present, so probing SYS_openat2 alone still left the build
broken (#905). Exercise the header and struct in the configure check so
HAVE_OPENAT2 is defined only when both are actually usable.
To prevent using openat2() in situations where it is not supported, use
#if defined(__linux__) && defined(HAVE_OPENAT2)
in t_chmod_secure.c, just like it was already being done in syscall.c.
Signed-off-by: Markus Mayer <mmayer@broadcom.com>
Let configure detect if the openat2() syscall is supported by the kernel
headers we are building against. Do not attempt to use openat2() if
support is not present.
Users can still disable using the openat2() syscall manually if so
desired.
Signed-off-by: Markus Mayer <mmayer@broadcom.com>
fleettest.py builds the committed HEAD of a checkout on a fleet of remote machines over ssh and runs the test suite under both the stdio-pipe and --use-tcp transports in parallel, reporting only the unexpected results. Each target mirrors a .github/workflows/*.yml job: its configure flags, and the RSYNC_EXPECT_SKIPPED list parsed from the workflow.
The fleet is described by a JSON file (testsuite/fleettest.json, git-ignored); fleettest.json.example is a worked template. Use --fleet to point at another config and --repo to build a tree other than the current directory.
A target with nonroot:true reruns, as the unprivileged ssh user, the tests that declare a module-level fleet_nonroot=True (here ownership-depth and daemon). The set lives in the test files, so new privilege-sensitive tests join the non-root pass with no fleet-config change.
Also rename testsuite/README.testsuite to README.md and rewrite it as markdown documenting the current testsuite: runtests.py, the make check/check29/check30/installcheck/coverage targets, the result/exit-code conventions, and fleettest.py.
The symlink-race hardening routed the receiver's basis open through
secure_relative_open(), which rejects any '..' -- so a sibling
--link-dest=../01 on a use-chroot=no daemon was silently ignored and every file
re-transferred (#915/#928, a regression from 3.4.1).
Narrow the confinement to the sanitizing daemon (am_daemon && !am_chrooted) and
re-anchor it at the module root, the real trust boundary: secure_relative_open()
prefixes the cwd's module-relative path (from rsync's logical curr_dir[], a
guaranteed lexical prefix of module_dir) and resolves beneath module_dir, so
RESOLVE_BENEATH permits an in-module '..' climb while still rejecting one that
escapes the module. secure_basis_open() opens with a bare do_open() in the
non-sanitizing cases. t_stub.c gains weak curr_dir[]/curr_dir_len for the
helpers (via #pragma weak on non-GNU compilers, where rsync.h erases
__attribute__).
Two tests: link-dest-relative-basis asserts the in-module '..' is honoured;
link-dest-module-escape asserts a --link-dest=../../OUTSIDE climb that leaves
the module is refused (not hard-linked to an outside file). See upstream
PR #930.
A daemon module with path=/ makes F_PATHNAME absolute, so the secure_path built
for the content open starts with '/'. secure_relative_open() rejects an
absolute relpath with EINVAL, so a use-chroot=no daemon with path=/ could not
send any file ('failed to open ...: Invalid argument (22)') -- a regression
from 3.4.2. Strip leading slashes to a module-relative path; resolution stays
confined beneath module_dir.
