Andrew Tridgell
d12bdb1579
An LD_PRELOAD hook refuses linkat() for a symlink source only, so the arm is reachable on a filesystem that hard-links symlinks perfectly well. Three controls keep it from proving less than it looks: - the regular file in the same transfer must still be hard-linked, or "it fell back" would also be satisfied by --link-dest having been abandoned; - the itemised run must emit exactly one "cL... sym -> some-target" line. A plain -a run cannot see a duplicated itemisation, which is how that defect reached an HFS+ target before this was added; - EPERM and ENOSYS must fall back too, since errno does not separate "cannot" from "may not". itemize picked its expected change-type letter from the build capability, which is the wrong question -- the link happens on whichever filesystem holds the test data. Ask that one too, and drop the XFAIL the old mismatch needed. The hook is Linux-only, so the test joins the macOS and Cygwin skip lists, which are required to be sorted.
rsync testsuite
This directory holds rsync's automated regression tests. Ideally every code change or bug fix comes with a test that would have caught the problem.
The tests are Python scripts named testsuite/*_test.py, driven by the
runtests.py harness at the top of the tree (the old shell-based runtests.sh
is gone). Shared helpers live in testsuite/rsyncfns.py. A handful of C helper
programs (tls, getgroups, trimslash, …) are built alongside rsync and
used by some tests. Coverage notes are in COVERAGE.md.
Writing tests
Favour readability — a test is also documentation of the behaviour it pins, so prefer clarity over cleverness:
-
When a test writes an
rsyncd.conf, write it as a triple-quoted f-string so the actual config is readable top-to-bottom, with module parameters indented with plain spaces. Don't build it from adjacent string literals full of\n(and\t) escapes. The daemon's parser accepts space-indented parameters. -
Better still, use the structured helpers in
rsyncfns.pywhen a stock config will do:write_daemon_conf(modules, globals)(per-test modules/params) orbuild_rsyncd_conf()(the four standard modules). They also handle the root-onlyuid/gidlines for you (needed so ause chroot = nodaemon run as root can read a root-owned module). -
For config that varies (e.g. those root-only
uid/gidlines), interpolate a single optional block that expands when needed and is an empty string otherwise, rather than splicing pieces together:root = get_testuid() == get_rootuid() ids = f"uid = {get_rootuid()}\ngid = {get_rootgid()}" if root else "" conf.write_text(f"""\ pid file = {base}/rsyncd.pid use chroot = no {ids} log file = {base}/rsyncd.log [m] path = {mod} read only = yes """)
Running the tests
Via make
Run from the build directory:
make check— build the helper programs and run the whole suite in parallel (CHECK_J, default 8) against the just-built./rsync. You do not needmake installfirst; indeed you generally should not install before testing. Usemake check CHECK_J=1to run serially.make check29/make check30— the same, forcing protocol version 29 or 30.make installcheck— run the suite against the installed binary (e.g./usr/local/bin/rsync). Per the GNU standards this does not search$PATH. Handy for testing a distribution build.make check-progs— (re)build just the C helper programs the tests need, without running anything.make coverage/coverage-tcp/coverage-all— generate an HTML coverage report (needs./configure --enable-coverageandgcovr);coverage-allmerges runs across protocol versions and the tcp transport.
Via runtests.py directly
make check just drives runtests.py; run it directly for finer control. It
defaults --rsync-bin to ./rsync, so run it from the build directory (or pass
--rsync-bin / --tooldir):
./runtests.py # all tests
./runtests.py chmod-temp-dir # a single test by name
./runtests.py 'xattr*' # a glob of test names
Useful options:
-j N,--parallel N— run up to N tests at once--use-tcp— run daemon tests against a realrsyncdon127.0.0.1(the default runs them over a stdio pipe). Read the security warning below before using this on a shared machine.--protocol VER— force a protocol version--preserve-scratch— keep each test's scratch dir afterwards--log-level N,--always-log— more verbose output / show logs for passing tests too--stop-on-fail— stop after the first failure--timeout SECS— per-test timeout (default 300)--timing— after the run, list the tests by wall-clock, slowest first, with the serial sum and the floor set by the single longest test--race-timeout SECS— budget a TOCTOU race test may spend trying to win its race. These are the suite's slowest tests: a race test is a negative oracle, so it passes by spending its whole budget (5–15s each by default). Lowering this speeds the suite up and weakens the oracle in equal measure.--valgrind,--valgrind-opts OPTS— run rsync under valgrind--rsync-bin PATH,--tooldir DIR,--srcdir DIR— locate the binary / build / source dirs--expect-skipped LIST— see skip enforcement below
Security warning: --use-tcp
⚠️ Do not use
--use-tcpon a machine with untrusted local users.
--use-tcpstarts a realrsyncdaemon listening on a loopback TCP port (127.0.0.1/::1) and deliberately configures insecure test scenarios (daemon modules without authentication, unsafe options enabled, etc.). Loopback addresses are reachable by every local user, so for as long as the tests run, any other user on the machine can connect to that daemon and exploit those deliberately-insecure modules — potentially reading or writing files with the privileges of the user running the tests (which is root if you run the suite as root).Only run
--use-tcpwhere there are no possible local users who might try to exploit it — a single-user workstation or a dedicated, isolated CI machine. The default stdio-pipe transport carries no such risk: it talks to the daemon over a private pipe with nothing listening on the network, so prefer it on any shared or multi-user host.
Results and exit codes
Each test prints one result line — PASS, FAIL, ERROR, SKIP (with a
reason), or XFAIL (an expected failure) — and the run ends with a
passed / failed / skipped summary. Per-test exit-code convention:
| code | meaning |
|---|---|
| 0 | pass |
| 1 | fail |
| 2 | error |
| 77 | skip |
| 78 | xfail |
runtests.py exits non-zero if any test fails. Some tests need root or another
precondition and otherwise SKIP — read the individual test scripts for details.
Skip enforcement: on a full run, set RSYNC_EXPECT_SKIPPED=a,b,c (or
--expect-skipped a,b,c) and the run fails if the set of skipped tests does not
match. This is how the CI workflows pin each platform's expected skip set. An
@FILE entry reads a skip list (one test per line) instead, and several may be
composed: the workflows use
@testsuite/skiplist/common.txt,@testsuite/skiplist/linux.txt. Keeping the
lists one-name-per-line is what stops two branches that each add a skipping test
from conflicting -- see testsuite/skiplist/README.md.
Scratch dirs and debugging
Each test runs in testtmp/<name>/. On failure the scratch directory is left in
place (also --preserve-scratch); including its logs in a bug report is helpful.
Preconditions
You need python3, /bin/sh, and the normal build toolchain. The ACL/xattr
tests need the acl and attr tools (getfacl/setfacl,
getfattr/setfattr) and skip if they are absent. Some tests need root.
These tests also run in CI via GitHub Actions (see .github/workflows/).
Fleet testing (fleettest.py)
testsuite/fleettest.py builds the committed HEAD of an rsync checkout on a
fleet of remote machines over ssh and runs the suite under both transports
(stdio-pipe and --use-tcp) in parallel, reporting only the unexpected
results. It is a fast local pre-flight for the GitHub CI matrix: each target
mirrors a .github/workflows/*.yml job — its configure flags, and the
RSYNC_EXPECT_SKIPPED list parsed straight from the workflow.
Because every run includes a --use-tcp pass, the fleet stands up the insecure
loopback test daemon on each target — so only point it at machines with no
untrusted local users (see the security warning
above).
The fleet — which machines, and how to reach and build on each — is described in a JSON file. Copy the bundled example (it is git-ignored) and edit it for your hosts:
cp testsuite/fleettest.json.example testsuite/fleettest.json # then edit
# (or symlink it, or point elsewhere with --fleet PATH)
The config is looked up in order: ~/.fleettest.json first, then
testsuite/fleettest.json, unless overridden with --fleet PATH.
Each entry names an ssh host (null to run locally), the workflow it mirrors,
and its configure flags, plus optional per-target settings (make, privilege,
env_prefix, …). See the comments in fleettest.json.example.
A target with "nonroot": true does an extra pass, after the main (root) run,
that reruns the privilege-sensitive tests as the unprivileged ssh user. Which
tests those are is not listed in the fleet config — a test opts in by
setting a module-level fleet_nonroot = True, so the set is maintained in the
test files and new privilege-sensitive tests join automatically with no
fleet-config change.
A target with "protocols": [30, 29] runs one extra stdio-pipe pass per listed
version, each forcing that older wire version with runtests --protocol=N — the
fleet analogue of a workflow's check30/check29 steps. Each pass takes the
RSYNC_EXPECT_SKIPPED spec from the workflow's own check30/check29 step, so
a lane with extra protocol-gated skips (check29 adds
@testsuite/skiplist/proto29.txt) is enforced correctly. They show up as
protoNN columns in the report (and --timing breakdown); targets that don't
set protocols show - there.
Run it from inside a checkout (it builds the current directory's HEAD; use
--repo PATH for another tree):
python3 testsuite/fleettest.py # whole fleet, both transports
python3 testsuite/fleettest.py --list # list configured targets
python3 testsuite/fleettest.py --targets NAME[,NAME]
python3 testsuite/fleettest.py --fleet other.json --transport pipe
python3 testsuite/fleettest.py --timing # per-target wall-clock breakdown
python3 testsuite/fleettest.py --keep-on-fail # keep logs + tree where it broke
python3 testsuite/fleettest.py --full-tcp # whole suite in the tcp pass too
--timing adds a per-target breakdown after the report — total wall-clock plus
the push / build / pipe / tcp / protoNN / nonroot phases, sorted slowest-first. Targets
run in parallel, so the whole run is gated by the slowest one; the phase columns
show whether that target's hold-up is the push, the build, or a test pass. It
also passes --timing down to each target's runtests.py, so the captured
output attributes a slow pass to individual tests.
The tcp pass runs only the tests that can reach the daemon transport, since
it follows a full pipe pass over the very same build. --use-tcp is observable
through exactly one path — RSYNC_TEST_USE_TCP is read once in rsyncfns
(USE_TCP) and acted on once, in start_test_daemon() — so a test that never
gets there produces an identical result twice. That drops 186 of the 340 tests
and roughly a third of the pass's work; the count skipped is always printed.
Pass --full-tcp to sweep the whole suite there anyway. The narrowing applies
only when both transports run: under --transport tcp that pass is the only
one, so it runs the whole suite regardless.
--keep-on-fail [DIR] makes a failure inspectable without repeating the run.
For every target that came back with anything unexpected it writes the full
build and per-transport output to DIR/<run_id>/<target>/ (default
./fleettest-logs) and keeps that target's remote run dir, with the scratch
trees its failing tests left behind. Targets that came back clean are swept as
usual. This matters most for the race tests, which may not fail the same way
twice — and because a re-run costs a full configure + build on every machine.
Each run gets its own randomly-named build dir on every target
(<builddir>-<run_id>), so two or three runs can share the same fleet without
interfering. The dir is removed when the run ends — on success or failure, and
best-effort on Ctrl-C/kill; pass --keep to retain it for inspection. A hard
kill (SIGKILL), or a signal arriving mid-push, can leave a stray
<builddir>-<id> behind; sweep leftovers with
python3 testsuite/fleettest.py --cleanup (scope it with --targets, and only
run it when no other fleet runs are active, since it removes all matching run
dirs on the selected targets).
Each target must be provisioned with the build toolchain its workflow installs
(autoconf, automake, a C compiler, perl, a python3 markdown module such as
cmarkgfm or commonmark unless the flags pass --disable-md2man, and the dev
libraries its configure flags enable). A missing piece shows up as BUILD-FAIL.
Differential regression hunting (abdiff.py)
testsuite/abdiff.py is a developer tool — not a *_test.py, so runtests.py
ignores it. It hunts regressions by running the same benign transfer with
two rsync binaries (A = the build under test, B = a baseline) and comparing
the OUTCOME. The oracle is: for a benign input, a correctness/behaviour change
between the builds must be invisible, so A and B must produce an identical
result. Any divergence is a regression candidate to investigate and, if real,
minimize into a *_test.py.
It compares exit code, stderr (error markers + normalised text), --stats
"Literal data", the destination tree (content + full metadata: mode/uid/gid/
mtime/size/symlink target/xattrs/ACLs/hardlink grouping), the --itemize list,
and — with --cost — peak process-group RSS (a resource-regression oracle that
functional comparison misses). A stability gate runs each binary several
times and escalates on a candidate diff; nondeterministic scenarios are
quarantined FLAKY, never reported as regressions.
Run it from the build directory (so ./rsync and old_versions/ resolve):
testsuite/abdiff.py # default: ./rsync vs old_versions/rsync_3.4.1
testsuite/abdiff.py --sweep all -j5 # broad single pass, 5-way parallel
testsuite/abdiff.py --loop --timelimit 3600 --cost # hunt for an hour, resource oracle on
testsuite/abdiff.py --list --sweep all # list scenarios without running
Each finding is classed DIFF (regression candidate), ALLOW (an intentional,
documented behaviour change listed in the tool's allowlist), BETTER (A succeeds
where B fails), FLAKY, or TIMEOUT. Findings are printed and appended to a
per-run abdiff-log_<TIME>.txt (and the curated --findings log).
Key options: -j N parallelism; --sweep NAME|all; --loop (endless
random + systematic-combo stream) bounded by --timelimit SECS; --cost
(+--scale N for the large-tree fixtures); --repeat N (stability samples);
--rsync-a/--rsync-b the two binaries. Run as root to fold in the
owner/device/specials/fake-super and chroot-daemon sweeps automatically.
Transport lanes (a feature broken only over the wire is invisible to a local
copy): local, an ssh split (support/lsh.sh), a stdio-pipe daemon, a real TCP
daemon (bound port + greeting/handshake, and an auth challenge-response
variant), and the restricted rrsync wrapper (support/rrsh.sh). rrsync's
behaviour ships in the script, so pair each binary with its own version's
rrsync via --rrsync-a/--rrsync-b (give B's rrsync, e.g. one extracted from
that release's support/rrsync).
Cross-version baselines are the static binaries already in old_versions/;
old_versions/build_static.sh builds more from a git tag (and you can grab a
matching support/rrsync from the same tag for the rrsync lane).