Compare commits

..

21 Commits

Author SHA1 Message Date
Andrew Tridgell
806dff20d9 tests: add clang scan-build static-analysis CI (informational)
Run the clang static analyzer over a check-progs build, publish the HTML report
as an artifact, and print the bug count to the run summary. INFORMATIONAL only:
it does not pass --status-bugs, so it surfaces new analyzer findings without
going red on the existing (overwhelmingly false-positive) reports.

Runs on push/PR to master and via workflow_dispatch. No cron: it is
informational and its output only changes with the code (push/PR) or the clang
version, so a daily run on an unchanged tree would add noise without value.
2026-06-08 20:54:57 +10:00
Andrew Tridgell
8f63c498e9 tests: add ASan+UBSan CI gate
Add a clang AddressSanitizer + UndefinedBehaviorSanitizer workflow that builds
rsync with -fsanitize=address,undefined -fno-sanitize-recover=undefined -DNDEBUG
and runs the full test suite over both the stdio-pipe and TCP daemon transports.

UBSAN_OPTIONS=halt_on_error=1 together with -fno-sanitize-recover=undefined makes
any undefined behaviour fatal, so this job gates: the tree must stay UBSan-clean.
The remaining findings are fixed in code (hashtable/mdfour shifts, xattrs, and
log.c's file_struct, kept aligned via rounding.h); only byteorder.h's intentional
unaligned accessors are suppressed, with no_sanitize. -DNDEBUG builds as a release
does (assert() compiled out) so ASan covers the production code paths.

Runs on push/PR to master and via workflow_dispatch, plus a weekly cron to
catch breakage from a moving ubuntu-latest/clang toolchain (push/PR already
cover every code change, so daily would just re-run an unchanged tree).
2026-06-08 20:54:57 +10:00
Andrew Tridgell
df2833b318 io: drop the dead/unnecessary read_varint UBSan guard
The cherry-picked #428 wrapped no_sanitize attributes on read_varint() and
read_varlong() in `#ifndef CAREFUL_ALIGNMENT`, but byteorder.h always
#defines CAREFUL_ALIGNMENT (to 0 or 1), so that guard is never true and the
attributes were dead code.

They are also unnecessary: both functions read the assembled value through
an aligned union member (union { char b[5]; int32 x; }), not an unaligned
cast, so UBSan's alignment check never fires there (verified: the ASan+UBSan
suite is clean without them).  Remove the whole block rather than fix the
guard.  (The byteorder.h annotations from #428, which are real and correctly
placed inside the !CAREFUL_ALIGNMENT branch, are kept.)
2026-06-08 20:54:57 +10:00
Sam James
7214372a8a Disable UBSAN for alignment-sensitive functions when !CAREFUL_ALIGNMENT
rsync sets CAREFUL_ALIGNMENT for architectures which do not support
unaligned access. Disable UBSAN for functions which may use unaligned
accesses when CAREFUL_ALIGNMENT is set.

Bug: https://github.com/WayneD/rsync/issues/427
Signed-off-by: Sam James <sam@gentoo.org>
(cherry picked from commit 11c1e934e8)
2026-06-08 20:54:57 +10:00
Andrew Tridgell
497357800a log: align the file_struct built in log_delete()
log_delete() builds a struct file_struct inside a char buffer offset by the
(EXTRA_LEN-granular) extra data.  The EXTRA_ROUNDING block that rounds that
offset up to the struct's alignment (exactly as flist.c does for its pool
allocations) was dead code here: log.c never included rounding.h, so
EXTRA_ROUNDING was undefined and the rounding never ran, leaving the
file_struct pointer potentially under-aligned.  That trips UBSan's alignment
check and would fault on strict-alignment arches.

Include rounding.h (and add the Makefile dependency) so the existing rounding
actually applies -- fixing the alignment at the source rather than suppressing
the sanitizer.
2026-06-08 20:54:57 +10:00
Andrew Tridgell
fa084c4ae3 xattrs: fix UBSan-detected undefined behavior
Three pre-existing issues UBSan flags during the xattr tests:

  * xattr_lookup_hash(): the summed hashlittle2() values overflow the
    signed int64 accumulator (UB).  Accumulate in uint64_t and convert back
    at return -- the key is only used for hash-table equality, so the value
    is unchanged.
  * rsync_xal_get(): for an empty list (count == 0) the loop init
    `rxa += count-1` forms `items - 1` on a NULL `items` (UB).  Guard with
    `if (count)`.
  * rsync_xal_store(): `memcpy(dst, xalp->items, 0)` passes a NULL source for
    an empty list (UB).  Guard with `if (xalp->count)`.
2026-06-08 20:54:57 +10:00
Andrew Tridgell
4148419736 hashtable, mdfour: avoid signed left-shift overflow
UBSan flags two spots that shift a value into the top bits of a word via a
signed operand:

  * lib/mdfour.c copy64(): `in[i] << 24` promotes the uchar to int, so a
    byte >= 128 overflows int (UB).  Cast each byte to uint32.
  * hashtable.c NON_ZERO_64(): `(int64)(x) << 32` overflows int64 whenever
    x's high bit is set.  Shift as uint64_t (covers all four call sites).

Behavior-preserving -- only the intermediate type changes; the resulting
bit pattern is identical.
2026-06-08 20:54:57 +10:00
Andrew Tridgell
66712a90b3 rsync-web: updates for the 3.4.4 release 2026-06-08 14:45:03 +10:00
Andrew Tridgell
b780749ffb release.py: accept a git worktree in require_top_of_checkout()
In a git worktree .git is a file (a gitdir pointer), not a directory,
so os.path.isdir('.git') wrongly aborted with "no .git dir" when the
release was run from a worktree. Use os.path.exists() so it works from
both a normal checkout and a linked worktree.
2026-06-08 14:44:37 +10:00
Andrew Tridgell
d25c5e4b11 ci: move the daily scheduled jobs to weekly
Every platform build (the BSD/Solaris/macOS/cygwin/almalinux/ubuntu jobs),
coverage, the version-mix job and the android static build ran on a daily cron
*in addition to* push and pull_request to master. Since push/PR already cover
every code change, the cron only adds drift coverage -- catching breakage from a
moving runner image or toolchain that no commit triggers. Those images do not
change daily, so a daily run mostly re-tests an unchanged tree.

Move them all to a weekly cron (Mondays, keeping each job's existing time) to
keep that drift coverage at roughly a seventh of the Actions spend and log
noise. fleettest was already weekly. Per-change CI on push/PR is unchanged, and
workflow_dispatch still allows an on-demand run.
2026-06-08 10:25:38 +10:00
Andrew Tridgell
1ddfe17d65 fleettest: --cleanup also kills stray flippers/daemons and root-owned dirs
A run killed without a parent-death backstop can strand a TOCTOU path-flipper
(a busy `python -c` rename loop that pins a CPU) and an orphaned test rsyncd
(--no-detach --address=127.0.0.1) that squats its fixed port -- the wedge the
claim_ports() bind-probe now reports and points at --cleanup. Sweep both, best
effort, before removing the run dirs.

Each sweep counts the pattern, kills it (with a `sudo -n` retry for a process a
root-running test left), then re-counts after a settle: KILLED reports what
actually died, and a process that survives (pkill blocked, no passwordless sudo,
missing/limited pkill) is reported as SURVIVED and fails the run instead of
falsely claiming success.

Run-dir removal falls back to `sudo -n rm` so a dir whose contents a root test
owns is removed instead of failing with "Permission denied" (the failure mode
seen on the ubuntu/mac targets); only a dir that survives even sudo is failed.

The kill patterns use the pgrep self-exclusion trick ('r[e]name', 'det[a]ch')
so they match a real process's "rename"/"detach" but not the literal pattern in
the cleanup shell's own argv -- run_on() passes the whole script as the remote
argv, so without it --cleanup would signal itself. The patterns are host-global
(not scoped to one run), so --cleanup is documented to run between runs, not
during one.
2026-06-08 09:41:59 +10:00
Andrew Tridgell
6e6b4135ab testsuite: verify a claimed test port is actually bindable
claim_ports() takes a POSIX byte-range lock per port, which serializes
concurrent live test runs. But the kernel drops that lock the instant the
holding process dies, even if the run left an orphaned rsync --daemon still
bound to the port -- which happens when a run is SIGKILLed on a platform with
no parent-death backstop (rsyncfns only arms PR_SET_PDEATHSIG, Linux-only, so
the BSDs/Solaris/macOS can strand a daemon). A later run then wins the freed
lock while the socket is still squatted and dies with a cryptic "bind() failed:
Address already in use" / "did not see server greeting".

After taking each lock, actually bind the port (SO_REUSEADDR, so a port merely
in TIME_WAIT is not a false positive; only a live squatter fails) and close it
immediately. On failure stop with an actionable message naming the port and the
likely orphaned daemon. Closes the gap that masked the OpenBSD daemon-auth wedge.
2026-06-08 09:41:59 +10:00
Andrew Tridgell
c2b8e4532b fleettest: require runtests.py in --testsuite-repo, not the build tree
When --testsuite-repo provides the suite, the build tree (--repo) need not
carry runtests.py -- it may be an older release whose shell testsuite predates
the Python runtests.py (e.g. a 3.4.1 backport branch built and tested with the
current suite).  Check runtests.py in TESTSUITE_REPO and only require the build
tree to be rsync source (rsync.h).
2026-06-08 06:29:49 +10:00
Andrew Tridgell
7b66c0665f fleettest: add --testsuite-repo to run another tree's suite against this build
--repo couples the built source and the test suite that exercises it.
--testsuite-repo PATH overlays runtests.py + testsuite/ from a second tree onto
the staged build tree, and sources the expected-skip workflows from it, so one
can build an older release (e.g. a 3.4.x stable branch) and run the current
comprehensive suite against that binary. Defaults to --repo, so the existing
single-tree behaviour is unchanged.
2026-06-08 06:29:49 +10:00
Andrew Tridgell
49f8dd1ca4 runtests: stop discovering obsolete *.test shell tests
The shell testsuite was removed in 1f689ec0 (rewritten in Python); only
*_test.py remain, yet collect_tests still globbed *.test and _testbase mapped
foo.test and foo_test.py to the same canonical name. Harmless on a master tree
(no .test files), but when an older tree's *.test files are present -- e.g.
fleettest --testsuite-repo building a 3.4.x release whose shell suite still
exists -- both glob to the same test name and scratch dir and race under -j,
producing spurious failures. Drop .test discovery entirely.
2026-06-08 06:29:49 +10:00
Andrew Tridgell
6fad1d7d74 testsuite,ci: mark recv-discard-nullderef CI skip and tighten its check
The regression test honestly skips when it cannot force the receiver's
output mkstemp() to fail -- as root (root bypasses DAC) and on Cygwin
(chmod 0555 does not deny the owner a write). The ubuntu, ubuntu-22.04,
almalinux and macOS jobs run `make check` as root, and Cygwin can't
enforce the unwritable directory, so the test skips on all of them.
runtests.py fails a run on any skip-set mismatch, so add the test to
those jobs' RSYNC_EXPECT_SKIPPED lists; the BSD/Solaris jobs run as root
too but enforce no expected-skip set, so they need no change.

Also tighten the pass condition. The post-chmod writability probe already
guarantees the receiver discards (mkstemp must fail), so an exit 0 would
mean the file actually transferred and the discard path was never
exercised -- a silent false-pass. Require exactly exit 23 (the forced
discard leaves the file untransferred); 12 remains the pre-fix crash.
2026-06-06 18:56:51 +10:00
pterror
b8562dbf4a testsuite: regression for the receiver discard-path NULL deref
Drives a real sender<->receiver pair (client sender -> daemon receiver,
both the binary under test in the default pipe transport) so the receiver
actually takes the recv_files discard path -- a local `rsync a b` does
not. The basis and source share a leading block so the generator emits
real sums and the receiver gets a block MATCH; the destination directory
is made unwritable so the receiver's output mkstemp() fails and it
discards the delta. Pre-fix the receiver SIGSEGVs in full_fname(NULL),
which the client sees as a protocol-data-stream error (code 12); post-fix
it drains the delta and reports a benign code 23 (or 0).

Skips (exit 77) when run as root, since root bypasses DAC and the
unwritable destination would not make mkstemp() fail -- so the discard
path, and the bug, would never be reached.

Verified red-on-buggy / green-on-fixed against the 0d0399bb receiver.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-06 18:56:51 +10:00
pterror
d66846351d receiver: fix NULL deref on the delta discard path
receive_data() crashed a receiver that was merely DISCARDING a file's
delta stream. discard_receive_data() calls receive_data() with
fname == NULL and fd == -1, so size_r == 0 and mapbuf == NULL. A normal
block-MATCH token (against a block the basis and source share) then
reaches the !mapbuf branch added in 31fbb17d ("receiver: fix absolute
--partial-dir delta resume"), which calls full_fname(fname). full_fname()
dereferences its argument unconditionally (util1.c: `if (*fn == '/')`),
so fname == NULL faults there -> receiver SIGSEGV.

This is a normal-operation crash with a stock cooperating sender, not an
adversarial one. The generator hands the sender real block sums whenever
the basis is readable and we're in delta mode; the receiver only decides
to discard afterwards, when its output cannot be produced -- e.g. the
destination directory is not writable (mkstemp fails), the basis turns
out to be a directory, or a --partial-dir resume is skipped. A MATCH
token arriving during that discard hit the NULL deref.

The 31fbb17d branch is correct only for a REAL output transfer (fd != -1,
fname valid): there, a block match with no mapped basis is a genuine
protocol inconsistency (the generator promised a basis the receiver could
not open), and honoring it would silently omit those bytes from the
verification checksum or leave a hole, so hard-erroring -- and
full_fname(fname) -- is right. It conflated that with the discard path.

The discriminator is fd, not mapbuf: on the discard path fd == -1 always;
on the real-output inconsistency fd != -1. Scope the "no basis file"
protocol error to fd != -1 (where fname is non-NULL and full_fname is
safe) and, on the discard path (fd == -1), absorb the matched bytes
benignly (offset += len; continue) -- symmetric with the literal-token
handling just above, and restoring the pre-31fbb17d behavior. The
real-transfer inconsistency check is preserved unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-06 18:56:51 +10:00
Andrew Tridgell
bad790dd2e fleettest: add a per-target max_retry budget for flaky tests
A slow or heavily-loaded fleet box can occasionally flake a concurrency-
sensitive test (e.g. a daemon/lsh test under -j8 on a nested-VM Solaris box).
Rather than dropping the whole target to a lower -j, add a per-target
"max_retry" property: after a run, each failed test is re-run on its own up to
max_retry more times, and any that then pass are dropped from the failure list.
Recovered tests are listed in a new "RECOVERED" report section, so a flake is
surfaced, never silently hidden.

Applies to every pass for the target (pipe, tcp, protoNN, nonroot).  Default 0
keeps the current no-retry behaviour.
2026-06-06 16:41:51 +10:00
Zen Dodd
c67d1935d8 docs: fix option summary inconsistencies 2026-06-06 16:11:30 +10:00
Zen Dodd
1d6770edbc ci: test uninstall targets 2026-06-06 16:07:20 +10:00
32 changed files with 731 additions and 231 deletions

View File

@@ -18,7 +18,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/almalinux-8-build.yml'
schedule:
- cron: '42 8 * * *'
- cron: '42 8 * * 1'
jobs:
test:
@@ -62,7 +62,7 @@ jobs:
# crtimes-not-supported skip matches the other Linux jobs;
# daemon-chroot-acl and proxy-response-line-too-long skip because
# the default (secure) transport opens no listening socket.
run: RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long make check
run: RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long,recv-discard-nullderef make check
- name: check (TCP daemon transport)
# Second run exercising the real loopback-TCP daemon path.
run: ./runtests.py --rsync-bin="$PWD/rsync" --use-tcp -j 8

View File

@@ -21,7 +21,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/android-static-build.yml'
schedule:
- cron: '42 8 * * *'
- cron: '42 8 * * 1'
workflow_dispatch:
env:

72
.github/workflows/asan-build.yml vendored Normal file
View File

@@ -0,0 +1,72 @@
name: rsync ASan+UBSan (clang)
on:
push:
branches: [ master ]
paths-ignore:
- '.github/workflows/*.yml'
- '!.github/workflows/asan-build.yml'
pull_request:
branches: [ master ]
paths-ignore:
- '.github/workflows/*.yml'
- '!.github/workflows/asan-build.yml'
schedule:
# Weekly (Mon 09:42 UTC): catch breakage from a moving ubuntu-latest/clang
# toolchain (a new clang can add a UBSan check, or change ASan behaviour)
# that no code push would otherwise trigger. Push/PR already gate every
# code change, so daily would just re-run an unchanged tree.
- cron: '42 9 * * 1'
workflow_dispatch:
jobs:
asan:
runs-on: ubuntu-latest
name: rsync ASan+UBSan (clang)
env:
# rsync intentionally leaks small allocations at process exit, so leak
# detection would be all noise; chase only memory-safety errors.
ASAN_OPTIONS: detect_leaks=0:abort_on_error=1
# UBSan is a gate: -fno-sanitize-recover=undefined (below) aborts on the
# first finding and halt_on_error=1 makes that fatal, so any undefined
# behaviour fails the run. This needs the tree to be UBSan-clean: the
# remaining findings are fixed in code (hashtable/mdfour shifts, xattrs,
# and log.c's file_struct, kept aligned via rounding.h); only byteorder.h's
# intentional unaligned accessors are suppressed, with no_sanitize.
UBSAN_OPTIONS: print_stacktrace=1:halt_on_error=1
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: prep
run: |
sudo apt-get update
sudo apt-get install -y clang acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev openssl
echo "/usr/local/bin" >>"$GITHUB_PATH"
- name: configure
# -DNDEBUG builds as a shipped release does (assert() compiled out), so
# AddressSanitizer catches the over-reads/over-writes that an "assert()
# instead of a real bounds check" bug would cause in a production build.
# UBSan rides along on the same build; -fno-sanitize-recover=undefined
# makes any undefined behaviour abort (and thus fail the run) instead of
# merely printing it.
run: |
CC=clang \
CFLAGS="-fsanitize=address,undefined -fno-sanitize-recover=undefined -fno-omit-frame-pointer -g -O1 -DNDEBUG" \
LDFLAGS="-fsanitize=address,undefined" \
./configure --with-rrsync --disable-md2man
- name: make
# check-progs builds rsync plus the test helper programs (tls, trimslash,
# t_unsafe, ...) that runtests.py requires; plain "make" builds only rsync
# and runtests aborts on the missing helpers.
run: make check-progs
- name: info
run: ./rsync --version
- name: check (stdio-pipe transport)
# ASan+UBSan-instrumented coverage of the transfer, daemon, sender,
# receiver and metadata paths over the default stdio-pipe transport.
run: ./runtests.py --rsync-bin="$PWD/rsync" -j8
- name: check (TCP daemon transport)
# --use-tcp also exercises the loopback rsyncd listener and the client's
# TCP connection path.
run: ./runtests.py --rsync-bin="$PWD/rsync" --use-tcp -j8

View File

@@ -12,7 +12,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/coverage.yml'
schedule:
- cron: '42 9 * * *'
- cron: '42 9 * * 1'
workflow_dispatch:
jobs:

View File

@@ -12,7 +12,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/cygwin-build.yml'
schedule:
- cron: '42 8 * * *'
- cron: '42 8 * * 1'
jobs:
test:
@@ -46,7 +46,7 @@ jobs:
# RESOLVE_BENEATH symlink-race tests. symlink-dirlink-basis also now
# RUNS (the #915 non-daemon basis open uses a plain do_open, restoring
# following an in-tree dir-symlink basis without RESOLVE_BENEATH).
run: bash -c 'RSYNC_EXPECT_SKIPPED=acls-default,acls-depth,acls,bare-do-open-symlink-race,chdir-symlink-race,chown,daemon-access-ip,daemon-chroot-acl,devices,dir-sgid,open-noatime,protected-regular,proxy-response-line-too-long,sender-flist-symlink-leak,simd-checksum make check'
run: bash -c 'RSYNC_EXPECT_SKIPPED=acls-default,acls-depth,acls,bare-do-open-symlink-race,chdir-symlink-race,chown,daemon-access-ip,daemon-chroot-acl,devices,dir-sgid,open-noatime,protected-regular,proxy-response-line-too-long,recv-discard-nullderef,sender-flist-symlink-leak,simd-checksum make check'
- name: check (TCP daemon transport)
# Second run with daemon tests over a real loopback rsyncd; the default
# 'make check' above uses the secure stdio-pipe transport.

View File

@@ -12,7 +12,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/freebsd-build.yml'
schedule:
- cron: '42 8 * * *'
- cron: '42 8 * * 1'
jobs:
test:

View File

@@ -12,7 +12,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/macos-build.yml'
schedule:
- cron: '42 8 * * *'
- cron: '42 8 * * 1'
jobs:
test:
@@ -44,7 +44,7 @@ jobs:
# chown-fake / devices-fake / xattrs / xattrs-hlink now RUN on macOS
# (rsyncfns.py drives xattrs via the `xattr` command), verified on a
# real macOS host, so they're no longer in the skip set.
run: sudo RSYNC_EXPECT_SKIPPED=acls-default,acls-depth,chmod-temp-dir,daemon-access-ip,daemon-chroot-acl,dir-sgid,open-noatime,preallocate,protected-regular,proxy-response-line-too-long,simd-checksum,sparse make check
run: sudo RSYNC_EXPECT_SKIPPED=acls-default,acls-depth,chmod-temp-dir,daemon-access-ip,daemon-chroot-acl,dir-sgid,open-noatime,preallocate,protected-regular,proxy-response-line-too-long,recv-discard-nullderef,simd-checksum,sparse make check
- name: check (TCP daemon transport)
# Second run with daemon tests over a real loopback rsyncd; the default
# 'make check' above uses the secure stdio-pipe transport.

View File

@@ -12,7 +12,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/netbsd-build.yml'
schedule:
- cron: '42 8 * * *'
- cron: '42 8 * * 1'
jobs:
test:

View File

@@ -12,7 +12,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/openbsd-build.yml'
schedule:
- cron: '42 8 * * *'
- cron: '42 8 * * 1'
jobs:
test:

51
.github/workflows/scan-build.yml vendored Normal file
View File

@@ -0,0 +1,51 @@
name: rsync scan-build (clang analyzer)
on:
push:
branches: [ master ]
paths-ignore:
- '.github/workflows/*.yml'
- '!.github/workflows/scan-build.yml'
pull_request:
branches: [ master ]
paths-ignore:
- '.github/workflows/*.yml'
- '!.github/workflows/scan-build.yml'
workflow_dispatch:
jobs:
scan-build:
runs-on: ubuntu-latest
name: rsync scan-build (clang analyzer)
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: prep
run: |
sudo apt-get update
sudo apt-get install -y clang clang-tools acl libacl1-dev attr libattr1-dev liblz4-dev libzstd-dev libxxhash-dev openssl
- name: configure (under scan-build)
# Run configure under scan-build so its analyzer compiler-wrapper is baked
# into the Makefile's $(CC); --disable-md2man avoids the doc toolchain.
run: scan-build ./configure --with-rrsync --disable-md2man
- name: scan-build (informational)
# Static analysis only -- INFORMATIONAL, not a gate. rsync currently has
# a fair number of reports that are overwhelmingly known false positives
# (e.g. unix.Chroot "no chdir after chroot", core.NonNullParamChecker
# against functions that can't actually receive NULL). We publish the
# HTML report as an artifact and print the bug count to the run summary,
# but do NOT pass --status-bugs, so this surfaces new analyzer findings
# without going red on arrival. check-progs builds rsync + the test
# helpers without needing the man-page toolchain.
run: |
scan-build -o "$PWD/scan-report" make check-progs -j"$(nproc)" 2>&1 | tee scan-build.out
echo '## scan-build summary' >>"$GITHUB_STEP_SUMMARY"
grep -E 'scan-build: .* bugs? found|scan-build: No bugs found' scan-build.out >>"$GITHUB_STEP_SUMMARY" || true
- name: upload report
if: always()
uses: actions/upload-artifact@v4
with:
name: scan-build-report
path: scan-report
if-no-files-found: ignore

View File

@@ -12,7 +12,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/solaris-build.yml'
schedule:
- cron: '42 8 * * *'
- cron: '42 8 * * 1'
jobs:
test:

View File

@@ -16,7 +16,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/ubuntu-22.04-build.yml'
schedule:
- cron: '42 8 * * *'
- cron: '42 8 * * 1'
jobs:
test:
@@ -39,11 +39,11 @@ jobs:
- name: info
run: rsync --version
- name: check
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long make check
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long,recv-discard-nullderef make check
- name: check30
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long make check30
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long,recv-discard-nullderef make check30
- name: check29
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long make check29
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long,recv-discard-nullderef make check29
- name: check (TCP daemon transport)
# Second run with daemon tests over a real loopback rsyncd; the default
# 'make check' above uses the secure stdio-pipe transport.

View File

@@ -12,7 +12,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/ubuntu-build.yml'
schedule:
- cron: '42 8 * * *'
- cron: '42 8 * * 1'
jobs:
test:
@@ -30,16 +30,44 @@ jobs:
run: ./configure --with-rrsync
- name: make
run: make
- name: install/uninstall DESTDIR smoke test
run: |
set -e
tmp="$(mktemp -d)"
trap 'rm -rf "$tmp"' EXIT
make install-all DESTDIR="$tmp"
for path in \
/usr/local/bin/rsync \
/usr/local/bin/rsync-ssl \
/usr/local/bin/rrsync \
/usr/local/share/man/man1/rsync.1 \
/usr/local/share/man/man1/rsync-ssl.1 \
/usr/local/share/man/man1/rrsync.1 \
/usr/local/share/man/man5/rsyncd.conf.5 \
/etc/stunnel/rsyncd.conf
do
test -e "$tmp$path"
done
make uninstall-all DESTDIR="$tmp"
leftover="$(find "$tmp" -type f -print)"
if [ -n "$leftover" ]; then
printf '%s\n' "$leftover"
exit 1
fi
- name: install
run: sudo make install
- name: info
run: rsync --version
- name: check
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long make check
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long,recv-discard-nullderef make check
- name: check30
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long make check30
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long,recv-discard-nullderef make check30
- name: check29
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long make check29
run: sudo RSYNC_EXPECT_SKIPPED=crtimes,daemon-access-ip,daemon-chroot-acl,proxy-response-line-too-long,recv-discard-nullderef make check29
- name: check (TCP daemon transport)
# Second run with daemon tests over a real loopback rsyncd. The default
# 'make check' above uses the secure stdio-pipe transport (no listening

View File

@@ -33,7 +33,7 @@ on:
- '.github/workflows/*.yml'
- '!.github/workflows/ubuntu-version-mix.yml'
schedule:
- cron: '52 8 * * *'
- cron: '52 8 * * 1'
jobs:
version-mix:

View File

@@ -111,6 +111,21 @@ install-all: install install-ssl-daemon
install-strip:
$(MAKE) INSTALL_STRIP='-s' install
.PHONY: uninstall
uninstall:
rm -f $(DESTDIR)$(bindir)/rsync$(EXEEXT) $(DESTDIR)$(bindir)/rsync-ssl
rm -f $(DESTDIR)$(bindir)/rrsync
rm -f $(DESTDIR)$(mandir)/man1/rsync.1 $(DESTDIR)$(mandir)/man1/rsync-ssl.1
rm -f $(DESTDIR)$(mandir)/man1/rrsync.1
rm -f $(DESTDIR)$(mandir)/man5/rsyncd.conf.5
.PHONY: uninstall-ssl-daemon
uninstall-ssl-daemon:
rm -f $(DESTDIR)/etc/stunnel/rsyncd.conf
.PHONY: uninstall-all
uninstall-all: uninstall uninstall-ssl-daemon
rsync$(EXEEXT): $(OBJS)
$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(OBJS) $(LIBS)
@@ -124,6 +139,7 @@ usage.o: version.h latest-year.h help-rsync.h help-rsyncd.h git-version.h defaul
loadparm.o: default-dont-compress.h daemon-parm.h
flist.o: rounding.h
log.o: rounding.h
default-cvsignore.h default-dont-compress.h: rsync.1.md define-from-md.awk
$(AWK) -f $(srcdir)/define-from-md.awk -v hfile=$@ $(srcdir)/rsync.1.md

View File

@@ -68,10 +68,26 @@ SIVAL64(char *buf, int pos, int64 val)
#else /* !CAREFUL_ALIGNMENT */
/* We don't want false positives about alignment from UBSAN, see:
https://github.com/WayneD/rsync/issues/427#issuecomment-1375132291
*/
/* From https://gcc.gnu.org/onlinedocs/cpp/Common-Predefined-Macros.html */
#ifndef GCC_VERSION
#define GCC_VERSION (__GNUC__ * 10000 \
+ __GNUC_MINOR__ * 100 \
+ __GNUC_PATCHLEVEL__)
#endif
/* This handles things for architectures like the 386 that can handle alignment errors.
* WARNING: This section is dependent on the length of an int32 (and thus a uint32)
* being correct (4 bytes)! Set CAREFUL_ALIGNMENT if it is not. */
#ifdef __clang__
__attribute__((no_sanitize("undefined")))
#elif GCC_VERSION >= 409
__attribute__((no_sanitize_undefined))
#endif
static inline uint32
IVALu(const uchar *buf, int pos)
{
@@ -83,6 +99,11 @@ IVALu(const uchar *buf, int pos)
return *u.num;
}
#ifdef __clang__
__attribute__((no_sanitize("undefined")))
#elif GCC_VERSION >= 409
__attribute__((no_sanitize_undefined))
#endif
static inline void
SIVALu(uchar *buf, int pos, uint32 val)
{
@@ -94,6 +115,11 @@ SIVALu(uchar *buf, int pos, uint32 val)
*u.num = val;
}
#ifdef __clang__
__attribute__((no_sanitize("undefined")))
#elif GCC_VERSION >= 409
__attribute__((no_sanitize_undefined))
#endif
static inline int64
IVAL64(const char *buf, int pos)
{
@@ -105,6 +131,11 @@ IVAL64(const char *buf, int pos)
return *u.num;
}
#ifdef __clang__
__attribute__((no_sanitize("undefined")))
#elif GCC_VERSION >= 409
__attribute__((no_sanitize_undefined))
#endif
static inline void
SIVAL64(char *buf, int pos, int64 val)
{

95
chmod.c
View File

@@ -29,7 +29,7 @@ extern mode_t orig_umask;
struct chmod_mode_struct {
struct chmod_mode_struct *next;
int ModeAND, ModeOR, ModeCOPY_SRC, ModeCOPY_DST, ModeCOPY_AND, ModeOP;
int ModeAND, ModeOR;
char flags;
};
@@ -43,20 +43,6 @@ struct chmod_mode_struct {
#define STATE_2ND_HALF 2
#define STATE_OCTAL_NUM 3
static int mode_dest_special_bits(int where)
{
int bits = 0;
if (where & 0100)
bits |= S_ISUID;
if (where & 0010)
bits |= S_ISGID;
if (where & 0001)
bits |= S_ISVTX;
return bits;
}
/* Parse a chmod-style argument, and break it down into one or more AND/OR
* pairs in a linked list. We return a pointer to new items on success
* (appending the items to the specified list), or NULL on error. */
@@ -64,13 +50,13 @@ struct chmod_mode_struct *parse_chmod(const char *modestr,
struct chmod_mode_struct **root_mode_ptr)
{
int state = STATE_1ST_HALF;
int where = 0, what = 0, op = 0, topbits = 0, topoct = 0, flags = 0, copybits = 0;
int where = 0, what = 0, op = 0, topbits = 0, topoct = 0, flags = 0;
struct chmod_mode_struct *first_mode = NULL, *curr_mode = NULL,
*prev_mode = NULL;
while (state != STATE_ERROR) {
if (!*modestr || *modestr == ',') {
int bits, where_specified;
int bits;
if (!op) {
state = STATE_ERROR;
@@ -84,10 +70,9 @@ struct chmod_mode_struct *parse_chmod(const char *modestr,
first_mode = curr_mode;
curr_mode->next = NULL;
where_specified = where;
if (where) {
if (where)
bits = where * what;
} else {
else {
where = 0111;
bits = (where * what) & ~orig_umask;
}
@@ -96,35 +81,18 @@ struct chmod_mode_struct *parse_chmod(const char *modestr,
case CHMOD_ADD:
curr_mode->ModeAND = CHMOD_BITS;
curr_mode->ModeOR = bits + topoct;
curr_mode->ModeCOPY_SRC = copybits;
curr_mode->ModeCOPY_DST = where;
curr_mode->ModeCOPY_AND = where_specified ? CHMOD_BITS : ~orig_umask;
curr_mode->ModeOP = op;
break;
case CHMOD_SUB:
curr_mode->ModeAND = CHMOD_BITS - bits - topoct;
curr_mode->ModeOR = 0;
curr_mode->ModeCOPY_SRC = copybits;
curr_mode->ModeCOPY_DST = where;
curr_mode->ModeCOPY_AND = where_specified ? CHMOD_BITS : ~orig_umask;
curr_mode->ModeOP = op;
break;
case CHMOD_EQ:
curr_mode->ModeAND = CHMOD_BITS - (where * 7) - (topoct ? topbits : 0)
- (copybits ? mode_dest_special_bits(where) : 0);
curr_mode->ModeAND = CHMOD_BITS - (where * 7) - (topoct ? topbits : 0);
curr_mode->ModeOR = bits + topoct;
curr_mode->ModeCOPY_SRC = copybits;
curr_mode->ModeCOPY_DST = where;
curr_mode->ModeCOPY_AND = where_specified ? CHMOD_BITS : ~orig_umask;
curr_mode->ModeOP = op;
break;
case CHMOD_SET:
curr_mode->ModeAND = 0;
curr_mode->ModeOR = bits;
curr_mode->ModeCOPY_SRC = 0;
curr_mode->ModeCOPY_DST = 0;
curr_mode->ModeCOPY_AND = CHMOD_BITS;
curr_mode->ModeOP = op;
break;
}
@@ -135,7 +103,7 @@ struct chmod_mode_struct *parse_chmod(const char *modestr,
modestr++;
state = STATE_1ST_HALF;
where = what = op = topoct = topbits = flags = copybits = 0;
where = what = op = topoct = topbits = flags = 0;
}
switch (state) {
@@ -191,53 +159,26 @@ struct chmod_mode_struct *parse_chmod(const char *modestr,
case STATE_2ND_HALF:
switch (*modestr) {
case 'r':
if (copybits)
state = STATE_ERROR;
what |= 4;
break;
case 'w':
if (copybits)
state = STATE_ERROR;
what |= 2;
break;
case 'X':
if (copybits)
state = STATE_ERROR;
flags |= FLAG_X_KEEP;
/* FALL THROUGH */
case 'x':
if (copybits)
state = STATE_ERROR;
what |= 1;
break;
case 's':
if (copybits)
state = STATE_ERROR;
if (topbits)
topoct |= topbits;
else
topoct = 04000;
break;
case 't':
if (copybits)
state = STATE_ERROR;
topoct |= 01000;
break;
case 'u':
if (what || topoct || copybits)
state = STATE_ERROR;
copybits = 0100;
break;
case 'g':
if (what || topoct || copybits)
state = STATE_ERROR;
copybits = 0010;
break;
case 'o':
if (what || topoct || copybits)
state = STATE_ERROR;
copybits = 0001;
break;
default:
state = STATE_ERROR;
break;
@@ -271,20 +212,6 @@ struct chmod_mode_struct *parse_chmod(const char *modestr,
return first_mode;
}
static int mode_copy_bits(int mode, int copy_src, int copy_dst, int copy_and)
{
int copy_bits = 0;
if (copy_src & 0100)
copy_bits |= (mode >> 6) & 7;
if (copy_src & 0010)
copy_bits |= (mode >> 3) & 7;
if (copy_src & 0001)
copy_bits |= mode & 7;
return (copy_dst * copy_bits) & copy_and;
}
/* Takes an existing file permission and a list of AND/OR changes, and
* create a new permissions. */
@@ -292,25 +219,17 @@ int tweak_mode(int mode, struct chmod_mode_struct *chmod_modes)
{
int IsX = mode & 0111;
int NonPerm = mode & ~CHMOD_BITS;
int copy_bits;
for ( ; chmod_modes; chmod_modes = chmod_modes->next) {
if ((chmod_modes->flags & FLAG_DIRS_ONLY) && !S_ISDIR(NonPerm))
continue;
if ((chmod_modes->flags & FLAG_FILES_ONLY) && S_ISDIR(NonPerm))
continue;
copy_bits = mode_copy_bits(mode, chmod_modes->ModeCOPY_SRC,
chmod_modes->ModeCOPY_DST,
chmod_modes->ModeCOPY_AND);
mode &= chmod_modes->ModeAND;
if ((chmod_modes->flags & FLAG_X_KEEP) && !IsX && !S_ISDIR(NonPerm))
mode |= chmod_modes->ModeOR & ~0111;
else
mode |= chmod_modes->ModeOR;
if (chmod_modes->ModeOP == CHMOD_SUB)
mode &= CHMOD_BITS - copy_bits;
else
mode |= copy_bits;
}
return mode | NonPerm;

View File

@@ -351,7 +351,7 @@ void *hashtable_find(struct hashtable *tbl, int64 key, void *data_when_new)
*/
#define NON_ZERO_32(x) ((x) ? (x) : (uint32_t)1)
#define NON_ZERO_64(x, y) ((x) || (y) ? (y) | (int64)(x) << 32 | (y) : (int64)1)
#define NON_ZERO_64(x, y) ((x) || (y) ? (y) | (uint64_t)(x) << 32 | (y) : (int64)1)
uint32_t hashlittle(const void *key, size_t length)
{

View File

@@ -89,8 +89,8 @@ static void copy64(uint32 *M, const uchar *in)
int i;
for (i = 0; i < MD4_DIGEST_LEN; i++) {
M[i] = (in[i*4+3] << 24) | (in[i*4+2] << 16)
| (in[i*4+1] << 8) | (in[i*4+0] << 0);
M[i] = ((uint32)in[i*4+3] << 24) | ((uint32)in[i*4+2] << 16)
| ((uint32)in[i*4+1] << 8) | ((uint32)in[i*4+0] << 0);
}
}

1
log.c
View File

@@ -22,6 +22,7 @@
#include "rsync.h"
#include "itypes.h"
#include "inums.h"
#include "rounding.h" /* EXTRA_ROUNDING, so log_delete() aligns its file_struct */
extern int dry_run;
extern int am_daemon;

View File

@@ -1,4 +1,4 @@
TARGETS := all install install-ssl-daemon install-all install-strip conf gen reconfigure restatus \
TARGETS := all install install-ssl-daemon install-all install-strip uninstall uninstall-ssl-daemon uninstall-all conf gen reconfigure restatus \
proto man clean cleantests distclean test check check29 check30 installcheck splint \
doxygen doxygen-upload finddead rrsync

View File

@@ -104,8 +104,8 @@ def require_samba_host():
def require_top_of_checkout():
if not os.path.isfile('packaging/release.py'):
die("Run this script from the top of your rsync checkout.")
if not os.path.isdir('.git'):
die("There is no .git dir in the current directory.")
if not os.path.exists('.git'):
die("There is no .git in the current directory (run from the top of a git checkout or worktree).")
def replace_or_die(regex, repl, txt, die_msg):
@@ -636,6 +636,8 @@ If you have a 'samba' remote configured (git.samba.org:/data/git/rsync.git):
Then upload the tarball + .asc to the GitHub release for {v_ver},
and announce on rsync-announce@, rsync@, and Discord.
NOTE! Also update the PPAs if needed
""")

View File

@@ -423,16 +423,32 @@ static int receive_data(int f_in, char *fname_r, int fd_r, OFF_T size_r,
stats.matched_data += len;
/* A block match can only be honored if we actually mapped the
* basis. If we didn't (basis open failed), the sender should
* never have been told a basis existed -- treat it as a protocol
* inconsistency rather than silently omitting these bytes from
* the verification checksum (which yields a spurious failure) or
* leaving a hole in the output. */
/* A block match with no mapped basis is a protocol inconsistency
* ONLY when we are actually producing output (fd != -1): the
* generator told the sender a basis existed but the receiver could
* not open it, so honoring the match would silently omit these
* bytes from the verification checksum (a spurious failure) or
* leave a hole in the output. Fail cleanly in that case.
*
* On the DISCARD path (fd == -1, fname == NULL) there is no output
* and no verification: discard_receive_data() deliberately drains a
* delta the receiver never intends to write (basis fstat failed,
* basis is a directory, output open failed, batch skip, ...). The
* sender does not know the data is being discarded and streams an
* ordinary delta, so a match token here is NORMAL protocol, not
* malformed. Absorb it benignly (advance the offset and continue),
* as the pre-existing "if (mapbuf)" guards did before this check was
* added in 31fbb17d -- erroring would wrongly break legitimate
* transfers, and full_fname(fname) with fname==NULL would
* dereference NULL (a receiver crash on a normal transfer). */
if (!mapbuf) {
rprintf(FERROR, "got a block match with no basis file for %s [%s]\n",
full_fname(fname), who_am_i());
exit_cleanup(RERR_PROTOCOL);
if (fd != -1) {
rprintf(FERROR, "got a block match with no basis file for %s [%s]\n",
full_fname(fname), who_am_i());
exit_cleanup(RERR_PROTOCOL);
}
offset += len;
continue;
}
if (DEBUG_GTE(DELTASUM, 3)) {

View File

@@ -26,6 +26,58 @@ License</A> and is currently being maintained by
<img src="badge.svg">
</a></div>
<h3>Rsync version 3.4.4 released</h3>
<i class=date>June 8th, 2026</i>
<p>Rsync version 3.4.4 has been released. This is a regression fix
release for the issues that have been reported with the 3.4.3
security release. Many thanks to everyone who reported the issues
(see <a href="https://download.samba.org/pub/rsync/NEWS#3.4.4">NEWS.md</a>
for credits).<p>
The 3.4.3 release had so many issues for two main reasons:
<ul>
<li>the 3.4 testsuite did not have broad enough coverage to
catch the regressions notices by users
<li>the nature of a security release prevents wide beta testing,
resulting in not enough manual testing in disparate environments
</ul>
To fix this for future releases we have greatly expanded the test
suite for 3.5 (currently in master) and grown the development
team, especially with more people with security expertise.
Thanks for your patience!
<p>See the <a href="https://download.samba.org/pub/rsync/NEWS#3.4.4">3.4.4 NEWS</a> for a detailed changelog.
The latest manpages are also available for:<ul>
<li><a href="https://download.samba.org/pub/rsync/rsync.1"><b>rsync</b>(1)</a>
<li><a href="https://download.samba.org/pub/rsync/rsync-ssl.1"><b>rsync-ssl</b>(1)</a>
<li><a href="https://download.samba.org/pub/rsync/rsyncd.conf.5"><b>rsyncd.conf</b>(5)</a>
<li><a href="https://download.samba.org/pub/rsync/rrsync.1"><b>rrsync</b>(1)</a>
</ul>
<p>The source tar is available here:
<b><a href="https://download.samba.org/pub/rsync/src/rsync-3.4.4.tar.gz">rsync-3.4.4.tar.gz</a>
(<a href="https://download.samba.org/pub/rsync/src/rsync-3.4.4.tar.gz.asc">signature</a>)</b>,
and the diffs from version 3.4.3 are available here:
<b><a href="https://download.samba.org/pub/rsync/src-diffs/rsync-3.4.3-3.4.4.diffs.gz">rsync-3.4.3-3.4.4.diffs.gz</a>
(<a href="https://download.samba.org/pub/rsync/src-diffs/rsync-3.4.3-3.4.4.diffs.gz.asc">signature</a>)</b>.
<p>Patch sets are also available for the older stable series, for
distributors not yet able to move to 3.4.4. Each is GPG signed. The
<i>full</i> set applies to a pristine release tarball; the <i>update</i>
set has only the patches added since the previous security release:<ul>
<li>rsync 3.2.7:
<b><a href="https://download.samba.org/pub/rsync/src/rsync-3.2.7-full.tar.gz">rsync-3.2.7-full.tar.gz</a>
(<a href="https://download.samba.org/pub/rsync/src/rsync-3.2.7-full.tar.gz.asc">signature</a>)</b>,
<b><a href="https://download.samba.org/pub/rsync/src/rsync-3.2.7-update.tar.gz">rsync-3.2.7-update.tar.gz</a>
(<a href="https://download.samba.org/pub/rsync/src/rsync-3.2.7-update.tar.gz.asc">signature</a>)</b>
<li>rsync 3.4.1:
<b><a href="https://download.samba.org/pub/rsync/src/rsync-3.4.1-full.tar.gz">rsync-3.4.1-full.tar.gz</a>
(<a href="https://download.samba.org/pub/rsync/src/rsync-3.4.1-full.tar.gz.asc">signature</a>)</b>,
<b><a href="https://download.samba.org/pub/rsync/src/rsync-3.4.1-update.tar.gz">rsync-3.4.1-update.tar.gz</a>
(<a href="https://download.samba.org/pub/rsync/src/rsync-3.4.1-update.tar.gz.asc">signature</a>)</b>
</ul>
<h3>Rsync version 3.4.3 released</h3>
<i class=date>May 20th, 2026</i>

View File

@@ -425,6 +425,9 @@ has its own detailed description later in this manpage.
--archive, -a archive mode is -rlptgoD (no -A,-X,-U,-N,-H)
--no-OPTION turn off an implied OPTION (e.g. --no-D)
--recursive, -r recurse into directories
--inc-recursive, --i-r enable incremental recursion
--no-inc-recursive disable incremental recursion
--no-i-r same as --no-inc-recursive
--relative, -R use relative path names
--no-implied-dirs don't send implied dirs with --relative
--backup, -b make backups (see --suffix & --backup-dir)
@@ -435,7 +438,8 @@ has its own detailed description later in this manpage.
--append append data onto shorter files
--append-verify --append w/old data in file checksum
--dirs, -d transfer directories without recursing
--old-dirs, --old-d works like --dirs when talking to old rsync
--old-dirs works like --dirs when talking to old rsync
--old-d same as --old-dirs
--mkpath create destination's missing path components
--links, -l copy symlinks as symlinks
--copy-links, -L transform symlink into referent file/dir
@@ -469,12 +473,14 @@ has its own detailed description later in this manpage.
--preallocate allocate dest files before writing them
--dry-run, -n perform a trial run with no changes made
--whole-file, -W copy files whole (w/o delta-xfer algorithm)
--no-whole-file, --no-W use the delta-xfer algorithm
--checksum-choice=STR choose the checksum algorithm (aka --cc)
--one-file-system, -x don't cross filesystem boundaries
--block-size=SIZE, -B force a fixed checksum block-size
--rsh=COMMAND, -e specify the remote shell to use
--rsync-path=PROGRAM specify the rsync to run on remote machine
--existing skip creating new files on receiver
--ignore-non-existing skip creating new files on receiver
--ignore-existing skip updating files that exist on receiver
--remove-source-files sender removes synchronized files (non-dir)
--del an alias for --delete-during
@@ -868,7 +874,7 @@ expand it.
0. `--inc-recursive`, `--i-r`
This option explicitly enables on incremental recursion when scanning for
This option explicitly enables incremental recursion when scanning for
files, which is enabled by default when using the [`--recursive`](#opt)
option and both sides of the transfer are running rsync 3.0.0 or newer.
@@ -1144,9 +1150,13 @@ expand it.
seen in the listing). Specify `--no-dirs` (or `--no-d`) if you want to
turn this off.
There is also a backward-compatibility helper option, `--old-dirs`
(`--old-d`) that tells rsync to use a hack of `-r --exclude='/*/*'` to get
an older rsync to list a single directory without recursing.
See also the backward-compatibility helper option [`--old-dirs`](#opt).
0. `--old-dirs`, `--old-d`
This backward-compatibility helper tells rsync to use a hack of
`-r --exclude='/*/*'` to get an older rsync to list a single directory
without recursing.
0. `--mkpath`
@@ -1513,16 +1523,6 @@ expand it.
> --chmod=D2775,F664
Symbolic permission-copy modes are also allowed, such as `g=u`, `o=g` or
`g-o`. A permission-copy item may copy from one class only (`u`, `g` or
`o`) and cannot be combined with `rwxXst` permission letters in the same
item. Use comma-separated items when you need both behaviours, such as
`g=o,o=`.
A permission-copy `=` item also clears the special bit for each destination
class it updates (`u` clears setuid, `g` clears setgid, and `o` clears
sticky), matching GNU **chmod** behaviour.
It is also legal to specify multiple `--chmod` options, as each additional
option is just appended to the list of changes to make.

View File

@@ -191,35 +191,31 @@ _PY_TEST_SUFFIX = '_test.py'
def _is_test_path(path):
base = os.path.basename(path)
return base.endswith('.test') or base.endswith(_PY_TEST_SUFFIX)
return os.path.basename(path).endswith(_PY_TEST_SUFFIX)
def _testbase(path):
"""Strip the test extension to get the canonical test name."""
base = os.path.basename(path)
if base.endswith('.test'):
return base[:-len('.test')]
if base.endswith(_PY_TEST_SUFFIX):
return base[:-len(_PY_TEST_SUFFIX)]
return base
def collect_tests(suitedir, patterns):
"""Collect test scripts (.test or _test.py) matching the given patterns."""
"""Collect test scripts (_test.py) matching the given patterns."""
if not patterns:
candidates = (glob.glob(os.path.join(suitedir, '*.test'))
+ glob.glob(os.path.join(suitedir, '*' + _PY_TEST_SUFFIX)))
candidates = glob.glob(os.path.join(suitedir, '*' + _PY_TEST_SUFFIX))
tests = sorted(p for p in candidates if _is_test_path(p))
else:
seen = set()
tests = []
for pat in patterns:
# Accept either bare name ("mkpath"), explicit extension, or glob.
if pat.endswith('.test') or pat.endswith('.py'):
if pat.endswith('.py'):
pats = [pat]
else:
pats = [pat + '.test', pat + _PY_TEST_SUFFIX]
pats = [pat + _PY_TEST_SUFFIX]
for p in pats:
for m in sorted(glob.glob(os.path.join(suitedir, p))):
if _is_test_path(m) and m not in seen:

View File

@@ -11,8 +11,8 @@ import shutil
from rsyncfns import (
FROMDIR, SCRATCHDIR, TODIR,
build_rsyncd_conf, check_perms, checkit, makepath, rmtree,
run_rsync, start_test_daemon, test_fail,
build_rsyncd_conf, checkit, makepath, rmtree,
run_rsync, start_test_daemon,
)
@@ -62,37 +62,6 @@ for d in (checkdir, checkdir / 'dir1', checkdir / 'dir2'):
checkit(['-avv', '--chmod', 'ug-s,a+rX,D+w', f'{FROMDIR}/', f'{TODIR}/'],
checkdir, TODIR)
def check_permcopy(chmod_arg, start_mode, expected, is_dir=False):
rmtree(FROMDIR)
rmtree(TODIR)
makepath(FROMDIR)
permcopy = FROMDIR / 'permcopy'
if is_dir:
permcopy.mkdir()
else:
permcopy.write_text('permcopy\n')
os.chmod(permcopy, start_mode)
run_rsync('-avv', f'--chmod={chmod_arg}', f'{FROMDIR}/', f'{TODIR}/')
check_perms(TODIR / 'permcopy', expected)
# Exercise chmod(1)-style permission copies.
check_permcopy('g=o,o=', 0o647, 'rw-rwx---')
check_permcopy('g=u', 0o741, 'rwxrwx--x')
check_permcopy('g-o', 0o775, 'rwx-w-r-x')
check_permcopy('u=g', 0o4755, 'r-xr-xr-x')
check_permcopy('g=u', 0o2755, 'rwxrwxr-x')
check_permcopy('o=u', 0o1750, 'rwxr-xrwx', is_dir=True)
rmtree(FROMDIR)
rmtree(TODIR)
makepath(FROMDIR)
(FROMDIR / 'permcopy').write_text('permcopy\n')
proc = run_rsync('-avv', '--chmod=g=ur', f'{FROMDIR}/', f'{TODIR}/',
check=False, capture_output=True)
if proc.returncode == 0:
test_fail('--chmod=g=ur was not rejected')
# Now exercise the F-only chmod path.
rmtree(FROMDIR)
rmtree(checkdir)

View File

@@ -11,7 +11,8 @@
"this target mirrors), configure_flags. Optional (with defaults): make (\"make\"),",
"python (\"python3\"), rsync_bin (\"rsync\"; \"rsync.exe\" on Cygwin), privilege",
"(\"root\" | \"sudo\" | \"user\"), pipe_jobs/tcp_jobs (8), builddir (\"rsync-citest\",",
"relative to the remote $HOME), env_prefix, configure_pre, nonroot, protocols.",
"relative to the remote $HOME), env_prefix, configure_pre, nonroot, protocols,",
"max_retry.",
"",
"nonroot: true reruns -- as the non-root ssh user, after the sudo runs -- the",
"tests that declare `fleet_nonroot = True` at module level (so the set is",
@@ -19,8 +20,14 @@
"",
"protocols: [30, 29] adds one extra stdio-pipe test pass per listed version,",
"each run with runtests --protocol=N (the fleet analogue of a workflow's",
"check30/check29 steps) and shown as a protoNN column. Keys starting with",
"\"_\" are comments. See testsuite/README.md."
"check30/check29 steps) and shown as a protoNN column.",
"",
"max_retry: N (default 0) re-runs each failed test on its own up to N more",
"times and drops any that then pass (listed under RECOVERED, not hidden). Use",
"on a slow/loaded box where concurrency-sensitive tests occasionally flake,",
"instead of dropping the whole target to a lower pipe_jobs/tcp_jobs.",
"",
"Keys starting with \"_\" are comments. See testsuite/README.md."
],
"targets": [
{
@@ -40,12 +47,13 @@
"--disable-xxhash", "--disable-lz4"]
},
{
"_comment": "Nested-VM OpenBSD occasionally flakes a daemon/tcp test under load; max_retry re-runs just the failed test rather than throttling the whole box (tcp_jobs/pipe_jobs are still available if you prefer that).",
"name": "openbsd",
"ssh_host": "root@openbsd",
"workflow": "openbsd-build.yml",
"make": "gmake",
"configure_pre": "export AUTOCONF_VERSION=2.71 AUTOMAKE_VERSION=1.16;",
"tcp_jobs": 2,
"max_retry": 2,
"configure_flags": ["--with-rrsync", "--disable-zstd", "--disable-md2man",
"--disable-xxhash", "--disable-lz4"]
},

View File

@@ -31,8 +31,11 @@ without interfering: each pushes, builds and tests in isolation. The run dir is
removed when the run ends -- on success or failure, and best-effort on
Ctrl-C/kill (pass --keep to retain it for inspection). A run that is hard-killed
(SIGKILL), or signalled mid-push, or whose ssh dies during cleanup can leave a
stray <builddir>-<id> behind; sweep those with `fleettest.py --cleanup`
(optionally scoped with --targets). Because each
stray <builddir>-<id> behind -- plus an orphaned path-flipper or test rsyncd on
platforms without a parent-death backstop; sweep all of those (root-owned files
included, via sudo -n) with `fleettest.py --cleanup` (optionally scoped with
--targets). Run --cleanup between runs, not during one: its process kills are
host-global and would also catch a concurrent run's flipper/daemon. Because each
run starts from a fresh dir, every build is a full configure + build.
PROVISIONING: each target must have the build toolchain its workflow's prepare
@@ -83,7 +86,11 @@ from pathlib import Path
# source tree these point at, so it must be run from inside an rsync checkout
# or given --repo PATH.
REPO = Path.cwd()
WORKFLOWS = REPO / ".github" / "workflows"
# Source tree providing the test suite (runtests.py + testsuite/). Defaults to
# REPO; --testsuite-repo decouples it so one tree is built and another's suite is
# run against the result.
TESTSUITE_REPO = REPO
WORKFLOWS = TESTSUITE_REPO / ".github" / "workflows"
# Fleet config (overridable with --fleet): ~/.fleettest.json is tried first, then
# fleettest.json next to this script. The example template sits next to the
@@ -137,6 +144,12 @@ class Target:
# stdio-pipe pass with runtests --protocol=N (the fleet analogue of a
# workflow's check30/check29 steps). e.g. [30, 29]. Empty => proto pass off.
protocols: list[int] = dataclasses.field(default_factory=list)
# Per-target retry budget for FLAKY tests: after a run, each failed test is
# re-run on its own up to max_retry more times, and any that then pass are
# dropped from the failure list (and reported as "recovered", never hidden).
# Use on a slow/loaded box where concurrency-sensitive tests occasionally
# flake, instead of dropping the whole target to a lower -j. 0 => no retry.
max_retry: int = 0
def load_fleet(path: Path) -> list[Target]:
@@ -283,7 +296,7 @@ def build_script(t: Target) -> str:
def test_script(t: Target, transport: str, skip_csv: str | None, jobs: int,
protocol: int | None = None) -> str:
protocol: int | None = None, only: list[str] | None = None) -> str:
rb = f'--rsync-bin="$PWD/{t.rsync_bin}"'
tcp = " --use-tcp" if transport == "tcp" else ""
# protocol forces an older wire version (mirrors `make check30`/`check29`).
@@ -291,9 +304,14 @@ def test_script(t: Target, transport: str, skip_csv: str | None, jobs: int,
# PYTHONDONTWRITEBYTECODE: don't drop root-owned __pycache__/*.pyc into the
# tree (a sudo run would, breaking the next non-root push --delete).
env = "PYTHONDONTWRITEBYTECODE=1 "
if skip_csv:
# Named tests (a max_retry re-run) make runtests full_run False, so the
# expected-skip list does not apply -- only the named tests' pass/fail matter.
names = ""
if only:
names = " " + " ".join(only)
elif skip_csv:
env += f"RSYNC_EXPECT_SKIPPED={skip_csv} "
runtests = f'{t.python} runtests.py {rb}{tcp}{proto} -j {jobs}'
runtests = f'{t.python} runtests.py {rb}{tcp}{proto} -j {jobs}{names}'
# env_prefix (e.g. a brew PATH) must reach the test too: some tests build a
# helper binary on the fly (a test may invoke `make`, which needs gawk etc.),
# so the build tools must be on PATH at test time.
@@ -349,6 +367,10 @@ class TransportResult:
skip_expected: set[str]
skip_got: set[str]
raw: str
# Tests that failed the initial run but passed on a max_retry re-run, so they
# were dropped from `failed`. Surfaced in the report (a recovered flake is
# noted, never silently hidden).
recovered: list[str] = dataclasses.field(default_factory=list)
@property
def skip_mismatch(self) -> bool:
@@ -376,6 +398,35 @@ def parse_transport(transport: str, r: CmdResult, skip_checked: bool) -> Transpo
skip_checked, exp, got, r.out)
def retry_failed(t: Target, label: str, tr: TransportResult, rerun) -> None:
"""Honour the target's max_retry budget: re-run each failed test on its own
(serially) up to max_retry more times; drop any that pass and record them in
tr.recovered. `rerun(names)` runs the given tests and returns a CmdResult.
A no-op when max_retry is 0 or there were no failures."""
if not t.max_retry or not tr.failed:
return
remaining = list(tr.failed)
for attempt in range(1, t.max_retry + 1):
r = rerun(remaining)
still = [m.group(2) for m in RE_RESULT.finditer(r.out)
if m.group(1) in ("FAIL", "ERROR")]
recovered = [n for n in remaining if n not in still]
if recovered:
tr.recovered.extend(recovered)
log(f"[{t.name}] {label} retry {attempt}/{t.max_retry}: "
f"recovered {','.join(recovered)}"
+ (f"; still failing {','.join(still)}" if still else ""))
remaining = [n for n in remaining if n in still]
if not remaining:
break
tr.failed = remaining
# The initial run's non-zero exit was the now-recovered failures; once they
# all pass on retry the cell is OK, so clear the stale exit code (only the
# failed tests can make runtests exit non-zero on a no-skip-list re-run).
if not remaining and tr.recovered and tr.exit_code != 0:
tr.exit_code = 0
@dataclasses.dataclass
class TargetResult:
target: str
@@ -444,9 +495,12 @@ def run_target(t: Target, args, staging: str) -> TargetResult:
t0 = time.monotonic()
r = run_on(t, cmd, timeout=2400)
res.timings[transport] = time.monotonic() - t0
res.transports[transport] = parse_transport(transport, r, skip_csv is not None)
tr = parse_transport(transport, r, skip_csv is not None)
retry_failed(t, transport, tr, lambda names, tp=transport: run_on(
t, test_script(t, tp, None, 1, only=names), timeout=1200))
res.transports[transport] = tr
log(f"[{t.name}] {transport} done "
f"({'ok' if res.transports[transport].ok else 'ISSUE'})")
f"({'ok' if tr.ok else 'ISSUE'})")
# Extra older-protocol passes (mirroring the workflow's check30/check29
# steps): same stdio-pipe transport and skip list as `make check`, but with
@@ -461,9 +515,13 @@ def run_target(t: Target, args, staging: str) -> TargetResult:
t0 = time.monotonic()
r = run_on(t, cmd, timeout=2400)
res.timings[label] = time.monotonic() - t0
res.transports[label] = parse_transport(label, r, skip_csv is not None)
tr = parse_transport(label, r, skip_csv is not None)
retry_failed(t, label, tr, lambda names, pr=proto: run_on(
t, test_script(t, "pipe", None, 1, protocol=pr, only=names),
timeout=1200))
res.transports[label] = tr
log(f"[{t.name}] {label} done "
f"({'ok' if res.transports[label].ok else 'ISSUE'})")
f"({'ok' if tr.ok else 'ISSUE'})")
# Extra non-root pass (after the sudo runs) for targets that opt in, running
# the tests that declare `fleet_nonroot = True` (discovered in main()).
@@ -471,9 +529,12 @@ def run_target(t: Target, args, staging: str) -> TargetResult:
t0 = time.monotonic()
r = run_on(t, nonroot_test_script(t, args.nonroot_tests), timeout=2400)
res.timings["nonroot"] = time.monotonic() - t0
res.transports["nonroot"] = parse_transport("nonroot", r, skip_checked=False)
tr = parse_transport("nonroot", r, skip_checked=False)
retry_failed(t, "nonroot", tr, lambda names: run_on(
t, nonroot_test_script(t, names), timeout=1200))
res.transports["nonroot"] = tr
log(f"[{t.name}] nonroot done "
f"({'ok' if res.transports['nonroot'].ok else 'ISSUE'})")
f"({'ok' if tr.ok else 'ISSUE'})")
res.timings["total"] = time.monotonic() - started
return res
@@ -598,6 +659,17 @@ def print_report(results: list[TargetResult], args, fleet: list[Target]) -> bool
for d in details:
print(d)
print("=" * 64)
# Recovered flakes: tests that failed but passed within the target's
# max_retry budget. The cell counts as OK, but list them so a flaky test is
# never silently swallowed.
recovered = [f"{res.target} / {transport}: {','.join(tr.recovered)}"
for res in results for transport in transports
if (tr := res.transports.get(transport)) and tr.recovered]
if recovered:
print("==== RECOVERED (flaky -- failed, then passed on retry) ====")
for r in recovered:
print(f" {r}")
print("=" * 64)
print(f"{len(results)} targets x {len(transports)} transports = {cells} cells: "
f"{ok_cells} OK, {cells - ok_cells} not OK")
return all_ok
@@ -702,33 +774,94 @@ def _on_signal(signum, frame):
os._exit(130 if signum == signal.SIGINT else 143)
# sweep() counts a pattern, kills it (best effort; sudo -n retry for processes a
# root-running test left), then RE-counts after a settle so we report what
# actually died (KILLED = before-after) and flag any survivor (SURVIVED, sets
# fail) instead of claiming success when pkill couldn't reach it. The patterns
# use the pgrep self-exclusion trick -- 'r[e]name'/'det[a]ch' match a real
# process's "rename"/"detach" but not the bracketed literal in this script's own
# argv (run_on passes the whole script as the remote argv), so we never signal
# ourselves. @BASE@ is substituted with the target's run-dir prefix.
_CLEANUP_SCRIPT = r'''fail=0
sweep() {
command -v pgrep >/dev/null 2>&1 || return 0
before=$(pgrep -f "$2" 2>/dev/null | wc -l | tr -d ' ')
[ "$before" -gt 0 ] 2>/dev/null || return 0
pkill -f "$2" 2>/dev/null
sudo -n pkill -f "$2" 2>/dev/null
sleep 1
after=$(pgrep -f "$2" 2>/dev/null | wc -l | tr -d ' ')
killed=$((before - after))
[ "$killed" -gt 0 ] 2>/dev/null && echo "KILLED $killed stray $1(s)"
if [ "$after" -gt 0 ] 2>/dev/null; then
echo "SURVIVED $after stray $1(s)"
fail=1
fi
}
sweep flipper 'r[e]name.*r[e]name.*r[e]name'
sweep daemon 'det[a]ch --address=127.0.0.1'
for d in @BASE@-*; do
[ -e "$d" ] || continue
if rm -rf -- "$d" 2>/dev/null || sudo -n rm -rf -- "$d" 2>/dev/null; then
echo "REMOVED $d"
else
echo "FAILED $d"
fail=1
fi
done
exit $fail
'''
def cleanup_remnants(targets: list[Target]) -> int:
"""--cleanup mode: remove every <base>-* run dir on each target, reporting
what each removed. Returns a process exit code. Only suffixed run dirs are
swept -- a bare <base> is left alone."""
"""--cleanup mode: on each target, kill the stray processes a killed run can
leave behind, then remove every <base>-* run dir, reporting what went.
Returns a process exit code. Only suffixed run dirs are swept -- a bare
<base> is left alone.
A run that is SIGKILLed (or whose ssh drops) can strand two kinds of process
on platforms without a parent-death backstop: the TOCTOU path-flipper (a
busy `python -c` rename loop that pins a CPU) and an orphaned test rsyncd
(`--no-detach --address=127.0.0.1`, which then squats its fixed port -- the
very wedge claim_ports()' bind-probe now reports). Both are killed best
effort (sudo -n retry for root-owned ones); a kill is verified by re-counting
afterwards, and a process that survives is reported and fails the run.
CAVEAT: the kill patterns are host-global, not scoped to a particular run, so
--cleanup assumes no *other* fleettest run is active on the target -- it
would also kill a concurrent run's flipper/daemon (and any manual `rsync
--daemon --no-detach --address=127.0.0.1`). Run it between runs, not during
one. Run dirs whose contents a root test owns are removed via a `sudo -n rm`
fallback; only a dir that survives even that is a failure."""
rc = 0
for t in targets:
base = t.builddir
if _unsafe_builddir(base):
log(f"[{t.name}] skipped (unsafe builddir {base!r})")
continue
# Echo each match before removing it so the harness can report what
# went; an unmatched glob stays literal and is skipped by the -e test.
script = (f'set -e\n'
f'for d in {base}-*; do\n'
f' [ -e "$d" ] || continue\n'
f' echo "$d"\n'
f' rm -rf -- "$d"\n'
f'done\n')
r = run_on(t, script, timeout=120)
removed = [ln for ln in r.out.splitlines() if ln.strip()]
if r.rc != 0:
# Structured markers (KILLED/SURVIVED/REMOVED/FAILED) keep the report
# clean even though run_on() folds stderr into stdout.
r = run_on(t, _CLEANUP_SCRIPT.replace("@BASE@", base), timeout=120)
lines = r.out.splitlines()
removed = [ln.split(" ", 1)[1] for ln in lines if ln.startswith("REMOVED ")]
failed = [ln.split(" ", 1)[1] for ln in lines if ln.startswith("FAILED ")]
killed = [ln.replace("KILLED ", "killed ", 1)
for ln in lines if ln.startswith("KILLED ")]
survived = [ln.replace("SURVIVED ", "still alive: ", 1)
for ln in lines if ln.startswith("SURVIVED ")]
msgs = killed[:]
if removed:
msgs.append("removed: " + " ".join(removed))
if survived:
rc = 1
log(f"[{t.name}] cleanup error (rc={r.rc}): {r.out.strip()[:200]}")
elif removed:
log(f"[{t.name}] removed: {' '.join(removed)}")
else:
log(f"[{t.name}] nothing to remove")
msgs += survived
if failed:
rc = 1
msgs.append("could not remove (even with sudo): " + " ".join(failed))
if r.rc not in (0, 1):
rc = 1
msgs.append(f"cleanup error rc={r.rc}: {r.out.strip()[:160]}")
log(f"[{t.name}] " + ("; ".join(msgs) if msgs else "nothing to remove"))
return rc
@@ -744,24 +877,44 @@ def main() -> int:
ap.add_argument("--keep", action="store_true",
help="keep each run's build dir (default: remove it at exit)")
ap.add_argument("--cleanup", action="store_true",
help="remove stray <builddir>-* run dirs on the targets, then exit")
help="kill stray flippers/test daemons and remove stray "
"<builddir>-* run dirs (root-owned via sudo -n) on the "
"targets, then exit; run between runs, not during one "
"(kills are host-global)")
ap.add_argument("--jobs", type=int, help="override -j for both transports")
ap.add_argument("--timing", action="store_true",
help="report per-target wall-clock (push/build/test) to find "
"the slowest target")
ap.add_argument("--repo", help="rsync source tree to build (default: cwd)")
ap.add_argument("--testsuite-repo",
help="rsync tree to take runtests.py + testsuite/ from "
"(default: --repo). Build one tree and run another's test "
"suite against it, e.g. --repo ../rsync-v3.4 --testsuite-repo .")
ap.add_argument("--fleet", help="fleet config JSON (default: ~/.fleettest.json, "
"else fleettest.json next to this script)")
ap.add_argument("--list", action="store_true", help="list targets and exit")
args = ap.parse_args()
global REPO, WORKFLOWS
global REPO, WORKFLOWS, TESTSUITE_REPO
REPO = Path(args.repo).resolve() if args.repo else Path.cwd()
WORKFLOWS = REPO / ".github" / "workflows"
if not args.cleanup and not (REPO / "runtests.py").is_file():
print(f"{REPO} is not an rsync source tree (no runtests.py); "
f"run from inside a checkout or pass --repo", file=sys.stderr)
return 2
TESTSUITE_REPO = Path(args.testsuite_repo).resolve() if args.testsuite_repo else REPO
# The expected-skip lists travel with the suite, so read workflows from the
# tree that provides the tests.
WORKFLOWS = TESTSUITE_REPO / ".github" / "workflows"
if not args.cleanup:
# The Python test suite (runtests.py + testsuite/) comes from
# TESTSUITE_REPO, so that is where runtests.py must live. The build tree
# (REPO) only has to be a buildable rsync source -- it may be an older
# release whose runtests.py predates the Python suite, or lacks it.
if not (TESTSUITE_REPO / "runtests.py").is_file():
print(f"{TESTSUITE_REPO} has no runtests.py; run from inside a "
f"checkout or pass --testsuite-repo a tree with the Python "
f"test suite", file=sys.stderr)
return 2
if not (REPO / "rsync.h").is_file():
print(f"{REPO} is not an rsync source tree (no rsync.h); "
f"run from inside a checkout or pass --repo", file=sys.stderr)
return 2
if args.fleet:
config_path = Path(args.fleet).resolve()
@@ -840,6 +993,19 @@ def main() -> int:
print(f"git archive failed: {ar.stderr}", file=sys.stderr)
return 2
# --testsuite-repo: overlay another tree's runtests.py + testsuite/ onto
# the built source (merge, no delete). Build REPO's rsync, but run
# TESTSUITE_REPO's suite against it. The leftover .test files from REPO
# are ignored by a Python runtests.py (it globs *_test.py).
if TESTSUITE_REPO != REPO:
ov = subprocess.run(
f"git -C {TESTSUITE_REPO} archive HEAD -- runtests.py testsuite "
f"| tar -x -C {staging}",
shell=True, capture_output=True, text=True)
if ov.returncode != 0:
print(f"testsuite overlay archive failed: {ov.stderr}", file=sys.stderr)
return 2
# Tests that opt into the non-root pass (same for every target).
args.nonroot_tests = discover_nonroot_tests(Path(staging) / "testsuite")

View File

@@ -0,0 +1,126 @@
#!/usr/bin/env python3
# Regression test for a receiver NULL-deref on the delta DISCARD path.
#
# In receiver.c receive_data(), a block-MATCH token that arrives while the
# receiver is DISCARDING a file (discard_receive_data() -> receive_data() with
# fname==NULL, fd==-1, hence mapbuf==NULL) reached
# rprintf(FERROR, "...%s...", full_fname(fname), ...)
# with fname==NULL. full_fname() dereferences its argument unconditionally
# (util1.c: `if (*fn == '/')`), so the receiver SIGSEGVs. The faulty error
# branch was added in 31fbb17d ("receiver: fix absolute --partial-dir delta
# resume"); the fix discriminates on fd (not mapbuf) and, on the discard path
# (fd==-1), absorbs the matched bytes benignly instead of erroring.
#
# This is a NORMAL-operation crash, not adversarial: a stock cooperating sender
# triggers it. The generator sends real block sums (basis readable, delta mode);
# the receiver then has to discard because its output mkstemp() fails -- here
# because the destination directory is not writable. A block MATCH against the
# shared leading block reaches the discard path and crashes the pre-fix binary.
#
# We drive a real sender<->receiver pair (client sender -> daemon receiver) so
# the receiver actually takes the recv_files discard path; a local `rsync a b`
# does not. In the default (pipe) daemon transport both ends are the binary
# under test.
#
# Skipped (exit 77) when running as root (root bypasses DAC), or when the
# directory mode is not enforced (e.g. a non-root process holding
# CAP_DAC_OVERRIDE in an unprivileged container): in both cases the receiver's
# mkstemp() would succeed despite chmod 0555, the discard path would not be
# taken, and the test would silently pass against a buggy binary. The
# post-chmod writability probe converts that silent false-pass into an honest
# skip and subsumes the root check.
import os
import shlex
import subprocess
import tempfile
from rsyncfns import (
SCRATCHDIR, RSYNC, TMPDIR,
get_testuid, get_rootuid, makepath, start_test_daemon, write_daemon_conf,
test_fail, test_skipped,
)
DAEMON_PORT = 12895
if get_testuid() == get_rootuid():
test_skipped("root bypasses DAC: the unwritable dest dir wouldn't make "
"the receiver's mkstemp fail, so the discard path (and the "
"bug) is never reached")
os.chdir(TMPDIR)
MODDIR = SCRATCHDIR / 'recvdiscard-mod' # daemon module root (writable)
BASISDIR = MODDIR / 'd' # made read-only -> mkstemp fails
SRCDIR_ = SCRATCHDIR / 'recvdiscard-src' # client source tree
makepath(MODDIR, BASISDIR, SRCDIR_)
# Basis and source share a leading block (2000 'A's) so the generator emits
# real sums and the receiver gets a block MATCH; the tails differ and the
# source is larger so a delta (not a no-op) is sent.
basis = BASISDIR / 'f'
basis.write_bytes(b'A' * 2000 + b'C' * 1000)
src = SRCDIR_ / 'f'
src.write_bytes(b'A' * 2000 + b'B' * 3000)
# A read/write daemon module rooted at MODDIR.
conf = write_daemon_conf([('recvdiscard', {'path': str(MODDIR),
'read only': 'no'})])
url = start_test_daemon(conf, DAEMON_PORT, rsync_cmd=RSYNC)
# Make the destination directory unwritable so the receiver's output mkstemp()
# fails and it falls back to discarding the delta stream. Restore in finally so
# the per-test scratch tree can be cleaned up.
os.chmod(BASISDIR, 0o555)
# Probe that the chmod actually denies writes for *this* process. A non-root
# user holding CAP_DAC_OVERRIDE bypasses the directory write bit, so mkstemp
# would succeed in the daemon receiver too, the discard path would never be
# taken, and the test would silently pass on a buggy binary. Better to skip
# explicitly. (Root takes this path too: its probe succeeds → skip, which
# subsumes the uid==0 check.)
try:
_fd, _probe = tempfile.mkstemp(dir=BASISDIR)
os.close(_fd)
os.unlink(_probe)
os.chmod(BASISDIR, 0o755)
test_skipped("destination dir is writable despite chmod 0555 "
"(CAP_DAC_OVERRIDE?); cannot force the receiver discard path")
except OSError:
pass # EACCES -- good, the precondition is enforced
try:
argv = shlex.split(RSYNC) + [
'--no-whole-file', '-a',
str(src), f'{url}recvdiscard/d/f',
]
print('Running:', ' '.join(argv))
proc = subprocess.run(argv, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, text=True)
print(proc.stdout, end='')
finally:
os.chmod(BASISDIR, 0o755)
rc = proc.returncode
# A receiver SIGSEGV manifests to the client as a protocol error (the daemon's
# receiver child crashes mid-stream and the connection drops): exit code 12.
# With the fix the receiver drains the delta and, because the forced-unwritable
# destination leaves the file untransferred, the run reports the benign "some
# files were not transferred" -- exit code 23.
#
# 23 is the ONLY non-crash outcome here: the writability probe above guarantees
# the receiver's mkstemp() fails, so the file is always discarded. An exit 0
# would mean the file actually transferred -- the discard path was NOT exercised
# and the run proves nothing -- so require exactly 23 (and call out 12 as the
# pre-fix crash).
if rc == 12:
test_fail(f"receiver crashed on the discard path (rsync exited {rc}: "
"error in rsync protocol data stream -- the receiver child "
"SIGSEGV'd in full_fname(NULL))")
if rc != 23:
test_fail(f"expected rsync exit 23 (the forced discard leaves the file "
f"untransferred); got {rc} -- the discard path was not exercised, "
"so this run validates nothing (12 would be the pre-fix crash)")
print(f"OK: receiver discarded the delta without crashing (rsync exit {rc})")

View File

@@ -175,6 +175,42 @@ def _open_lock_file() -> int:
return fd
def _probe_bindable(port: int) -> 'None':
"""Confirm `port` is actually free once we hold its claim_ports() lock.
The byte-range lock only coordinates *live* test drivers, and the kernel
releases it the instant the holding process dies -- even if that driver left
an orphaned daemon still bound to the port. That happens when a run is
SIGKILLed (or its ssh drops) on a platform with no parent-death backstop:
rsyncfns only arms PR_SET_PDEATHSIG, which is Linux-only, so on the
BSDs/Solaris/macOS a killed fleettest run can strand its rsyncd, which then
squats the fixed test port forever. A later run wins the (now-free) lock but
the socket is still taken, and the daemon dies with a cryptic "bind() failed:
Address already in use" / the client "did not see server greeting".
So actually try to bind it. SO_REUSEADDR is used so a port merely in
TIME_WAIT (recently and cleanly closed) is NOT a false positive; only a
live bound/listening socket -- a real squatter -- makes the bind fail, and
then we stop here with an actionable message instead of failing obscurely
later. The probe socket is closed immediately, freeing the port for the
daemon that is about to bind it.
"""
s = _socket.socket(_socket.AF_INET, _socket.SOCK_STREAM)
s.setsockopt(_socket.SOL_SOCKET, _socket.SO_REUSEADDR, 1)
try:
s.bind(('127.0.0.1', port))
except OSError as e:
test_fail(
f"port {port} was claimed for this run but something is still bound "
f"to 127.0.0.1:{port} ({e.strerror}). The claim_ports() lock only "
"serializes live test runs, so a still-bound port almost always "
"means an orphaned 'rsync --daemon' from a previously killed run "
f"(find it with `fstat | grep {port}` / `netstat -an | grep {port}` "
"and kill it, or run `fleettest.py --cleanup`), then retry.")
finally:
s.close()
def claim_ports(*ports: int) -> 'None':
"""Reserve the given TCP port numbers for the rest of this process.
@@ -210,6 +246,9 @@ def claim_ports(*ports: int) -> 'None':
# F_SETLKW via fcntl.lockf(LOCK_EX, length, start): exclusive
# byte-range lock on byte `port`, blocking until acquired.
fcntl.lockf(_port_lock_fd, fcntl.LOCK_EX, 1, port)
# The lock only proves no other live test run owns the port; an orphaned
# daemon from a killed run can still squat it (see _probe_bindable).
_probe_bindable(port)
# --- standalone rsyncd helpers ---------------------------------------------

View File

@@ -295,8 +295,12 @@ static int rsync_xal_get(const char *fname, item_list *xalp)
rxa = xalp->items;
if (count > 1)
qsort(rxa, count, sizeof (rsync_xa), rsync_xal_compare_names);
for (rxa += count-1; count; count--, rxa--)
rxa->num = count;
/* Guard count==0: rxa is then xalp->items (possibly NULL) and the
* "rxa += count-1" init would form NULL-1 (undefined). */
if (count) {
for (rxa += count-1; count; count--, rxa--)
rxa->num = count;
}
return 0;
}
@@ -381,17 +385,19 @@ static int64 xattr_lookup_hash(const item_list *xalp)
{
const rsync_xa *rxas = xalp->items;
size_t i;
int64 key = hashlittle2(&xalp->count, sizeof xalp->count);
/* Accumulate unsigned: the summed hash values routinely overflow a
* signed int64 (UB), and we only care about the resulting bit pattern. */
uint64_t key = (uint64_t)hashlittle2(&xalp->count, sizeof xalp->count);
for (i = 0; i < xalp->count; i++) {
key += hashlittle2(rxas[i].name, rxas[i].name_len);
key += (uint64_t)hashlittle2(rxas[i].name, rxas[i].name_len);
if (rxas[i].datum_len > MAX_FULL_DATUM)
key += hashlittle2(rxas[i].datum, xattr_sum_len);
key += (uint64_t)hashlittle2(rxas[i].datum, xattr_sum_len);
else
key += hashlittle2(rxas[i].datum, rxas[i].datum_len);
key += (uint64_t)hashlittle2(rxas[i].datum, rxas[i].datum_len);
}
return key;
return (int64)key;
}
static int find_matching_xattr(const item_list *xalp)
@@ -460,7 +466,9 @@ static int rsync_xal_store(item_list *xalp)
* entire initial-count, not just enough space for one new item. */
*new_list = empty_xa_list;
(void)EXPAND_ITEM_LIST(&new_list->xa_items, rsync_xa, xalp->count);
memcpy(new_list->xa_items.items, xalp->items, xalp->count * sizeof (rsync_xa));
/* xalp->items is NULL for an empty list; memcpy(dst, NULL, 0) is UB. */
if (xalp->count)
memcpy(new_list->xa_items.items, xalp->items, xalp->count * sizeof (rsync_xa));
new_list->xa_items.count = xalp->count;
xalp->count = 0;