Some S3 providers, such as Wasabi and Alibaba, report success when
creating a bucket which already exists, so rclone logs that the bucket
was created. The during-sync hooks in the equal_skipped test create a
new Fs, which logged this in the middle of the bisync run and made the
log differ from the golden one. Ignore these messages.
If a file is changed on both sides and the conflict names it gets renamed
to (e.g. X.conflict1 and X.conflict2) were deleted on both sides since the
last run, the listing update removed the just-copied conflict files again
when processing the deleted-on-both queue. The rename bookkeeping then
could not find them and bisync aborted with "bisync.renames: internal
error: missing info".
Skip names that this run copied when removing files deleted on both sides,
and add a test case for the scenario.
Before this change, `cmd/bisync/testdata/bisync_vscode_debuggers_launch.json`
was in the repository, and every `-golden` run regenerated it from
`fstest/test_all/config.yaml`, adding unnecessary noise to commit diffs.
After this change, the file is in `.gitignore` and no longer tracked.
`-golden` still writes it locally, for copying debug configurations into
VS Code's `launch.json`.
Before this change, the `{eol}`, `{tab}` and `{chr:XX}` placeholders in a
scenario became raw bytes in the name the harness handed to `Put`. A raw
control character isn't a valid rclone name: rclone names are in the
`Standard` encoding, where a newline is `␊`, and `FsEncoding` in
`fstests` encodes every raw name before uploading it for the same
reason.
The `extended_filenames` scenario dates from 2021, when bisync was tested
on local, Drive, Dropbox, OneDrive, S3 and SFTP, but not on crypt or
memory. Its header said that all rclone backends, including local,
return `\n` as a glyph. That was an assumption, and it doesn't hold. A
backend with an encoder lists a raw control character as its glyph
because `ToStandardName` runs `Standard.Encode` on every name it
returns. A backend without one (crypt, memory, linkbox, netstorage,
pixeldrain, storj) lists exactly the name it was given. The same thing
happens with no bisync involved: `rclone touch` with a raw newline in
the name stores a raw newline on crypt, and `sync` from there to local
then copies and deletes the file on alternate runs.
The scenario has never passed on crypt or memory for the right reason.
Its golden listings were recorded on local with `␊`, so until January
2024 they would not have matched the raw names crypt lists, but the
scenario wasn't run there. `57624629d` ran it on crypt, treated the raw
names as an encoding difference, and normalized them away in both the
log and the listings, so from then on the goldens matched. `81d7b6f29`
added the check that each listing matches its path, which that
normalization doesn't cover, and the step has failed on those backends
since: local lists the copy as `␊` while the listing, built from the
source names, holds the raw newline. The pre-flight probe fixed in the
previous commit skipped the scenario on many nightly backends in the
meantime, which kept the failure out of the nightly results.
After this change, a placeholder for a control character (0x01-0x1f and
0x7f) becomes its `Standard` glyph, so every backend receives the same
name. On local the file on disk is unchanged, since the local backend
decodes the glyph to the raw character, and the golden logs are
unchanged. Placeholders for other bytes, such as the invalid UTF-8 in
`{chr:81}` and `{chr:fe}`, still become raw bytes, which `Standard`
leaves as they are.
Before this change, the pre-flight check for the `normalization` and
`extended_filenames` cases uploaded its probe file to the path root, which
already held the scenario's files, and compared the first entry each path
listed. That entry was usually an unrelated file, so the result depended on
listing order rather than on how each backend stored the probe name. On 42
of 71 backends in the 2026-10-15 nightly, the paths appeared to differ and
the cases were skipped; elsewhere the check passed without examining the
probe at all. The check also named `extended_char_paths`, a case renamed to
`ext_paths` in 2024. The probe's uploads counted in the step's transfer
stats, which removed `There was nothing to transfer` lines from the logs of
the checked cases.
After this change, the probe goes in a fresh subdirectory with its own
stats group, and the check compares the name listed back from there. It
names only `normalization` and `extended_filenames`: `ext_paths` uses none
of the probe's invalid UTF-8 or emoji, and doesn't use `--fix-case`. The
`normalization` and `extended_filenames` golden logs gain the lines the
probe's stats had hidden.
Before this change, the integration tests dropped Dropbox's "Too many
requests or write operations" notice from the logs only in the format it
had when the tests were written, which began with the error code
(`too_many_requests/...`). Since 110bf468a (v1.65), the notice begins with
`Error `, so it was left in the log and caused a miscompare whenever
Dropbox rate limited a run.
After this change, the notice is dropped in either format.
Before this change, the integration tests checked whether each remote could
set modtimes on every `bisync` step (`checkPreReqs`, called from
`runBisync`), not once per remote. On a remote that isn't `IsLocal`, each
check sleeps for 1s twice (the Google Cloud Storage per-object rate limit
workaround from be73a10a9), for each side. That added 2-4s to each of the
~110 `bisync` steps in each test, which on a fast remote was most of the
run: on the integration test server, the `:memory:` tests took about 14
minutes.
After this change, the result of the check is remembered for each remote,
so it runs, and sleeps, once per remote per test run. The `:memory:` tests
now take about 7s locally.
Before this change, nothing checked that the rc `sync/bisync` call accepted
every bisync flag, so the rc could fall behind the command line as flags
were added, as in #7799.
After this change, `TestRcFlagParity` passes each non-hidden bisync flag to
the rc, under the parameter name `rc.md` uses, with a value of the wrong
type. Every getter `rcBisync` uses rejects it, so the call fails only if
the parameter is read, and the test names any flag the rc doesn't handle.
Before this change, since v1.74.0 (bb78eb8ab), the rc `sync/bisync` call
read the `backupdir1` and `backupdir2` parameters (the names used before
they were renamed to `backupDir1` and `backupDir2`) only when the new name
was also given with an invalid value, because the fallback sat inside an
`rc.NotErrParamNotFound(err)` check, which is true only for an invalid
value. Otherwise, `backupdir1` and `backupdir2` had no effect.
The same change made an invalid value for any optional parameter (for
example `"resync": "yes"`, or a `filtersFile` that isn't a string) be
treated as if the parameter were missing, instead of returning an error.
After this change, the old backup dir names are read when the new ones are
missing, and an invalid value returns an error again, as it did before
v1.74.0.
Before this change, bisync treated a `--max-delete` above 100 as 100%, so
the `--max-delete` check never triggered. A run that left no file on a side
unchanged was still stopped, by the "all files were changed" or empty
listing checks, but one that deleted most files while leaving even one
unchanged was not.
For bisync, `--max-delete` has always been a
percentage, "in range 0..100" as Ivan put it when he designed it
(https://github.com/rclone/rclone/pull/5164#discussion_r633470815), but the
range was never enforced. So a value meant as a file count, such as
`--max-delete 500`, turned the check off. Now that the rc takes
`--max-delete` from the global config too, that includes a `--max-delete`
the rcd itself was started with.
After this change, a `--max-delete` above 100 is an error, as the rc's
`maxDelete` parameter already was.
Before this change, `--max-delete` and `--dry-run` were read from the
global config by `applyContext`, which only the command line called. The
rc `sync/bisync` call started from empty `Options` and never ran it, so it
ignored both settings:
- A dry run requested through the global option (`_config`
`{"DryRun": true}` or the flat `dry_run` parameter) was ignored.
`setDryRun` resets `ci.DryRun` from `opt.DryRun` before copying, and
`opt.DryRun` was only set by bisync's own `dryRun` parameter.
- When the `maxDelete` parameter was omitted, the `--max-delete` limit was
0%. A `max_delete` set through `_config` or the flat parameter was not
used for that limit, and applied to the inner syncs as a file count
instead.
After this change, `Bisync` runs `applyContext` itself, on its own copy of
the config, so the command line, the rc, and any other caller follow the
same rules, and `Options.MaxDelete` and `Options.DryRun` are set by
`Bisync`. With no `--max-delete` anywhere, the limit is 50%, as #5680 says
the rc should.
The rc's `maxDelete` and `dryRun` parameters now set the call's config.
`maxDelete` overrides the global `--max-delete`. `dryRun: true` asks for a
dry run, but `dryRun: false` doesn't cancel one from the global config, so
the run is a dry run if either asks for one. The `maxDelete` parameter is
also documented again; it was left out when the rc help became generated
from the command's flags.
Before this change, when bisync updated a listing entry, it kept the old
modtime if the new one was less than a second earlier. This dated back to
when bisync truncated modtimes to the destination's precision before
recording them (`ConvertPrecision`), so that a truncated time would not
overwrite the more precise original.
`ConvertPrecision` was removed in 956c2963f, and since then, bisync has kept
the original precision and relied on the modify window when comparing. But
the rule in `ls.put` was left behind, and it then fired only when a file
really had an earlier modtime, recording a time the file did not have. This
could cause:
- a `--resync` to abort with "path1 and path2 are out of sync" when Path1's
version was slightly older than Path2's
- later runs to report changes that didn't happen, for a file changed to a
version less than a second older, or copied from a remote with lower
modtime precision within the same second as the old version (for example,
from SFTP to local)
After this change, bisync records the modtime it is given, like any other
update. Comparisons already account for differences in precision through
the modify window.
- TestCheckConflictsSkipsCheck: the check for identical files is skipped
with --size-only, and with --ignore-checksum only when the paths have no
hash in common.
- test_equal_skipped: an identical file that is new on both paths with
--ignore-checksum is recorded, so deleting it on Path1 deletes it on
Path2. Adds the "ignore-checksum" scenario argument.
- test_conflict_window: Path1 changes between the listing snapshot and
the check for identical files.
Co-Authored-By: ASRodrigo1 <45823308+asrodrigo1@users.noreply.github.com>
Before renaming a sync conflict, bisync checks whether the two files are
identical. It compares hashes using check or cryptcheck, and as a last
resort, when neither is available, it compares the files by downloading
them (--download), so that the result is certain regardless of checksum
support.
Before this change, the fallback to --download happened only when bisync
knew up front that the remotes had no usable hash. It did not happen
when a hash that was expected to be present turned out to be missing
(for example, a backend that normally supports hashes returned a blank
one, or cryptcheck could not get the underlying hash). Those files were
counted as identical based on size alone.
After this change, a missing hash falls back to --download, the same as
when the remotes have no hash in common. --size-only and
--ignore-checksum still disable the fallback and keep the size-only
comparison.
Co-Authored-By: ASRodrigo1 <45823308+asrodrigo1@users.noreply.github.com>
Before this change, when a file was new or changed on both paths and the pre-sync
conflict check deemed the two copies identical, bisync skipped the copy and then
rechecked the file after the sync to get its listing info. If the recheck could
not confirm equality (for example, hashes unavailable due to --size-only or
(--ignore-checksum + no common hash)), the file was rolled back to be retried on
the next run, and every later run repeated the same check with the same result.
A later change on one side was then handled against a missing or stale listing
entry.
After this change, files we've declared identical pre-sync are no longer
rechecked post-sync. The pre-sync conflict check now also returns the values it
compared, and a skipped file is recorded from the listing snapshot. If either
path changed between the snapshot and the check, the prior listing is kept and
the next run rechecks.
When --size-only or (--ignore-checksum + no common hash) are in use, the "new or
changed on both paths and equal" case skips the conflict check entirely, because
it is unnecessary. The check could only compare sizes, which we already know are
equal from the snapshot.
The new `test_equal_skipped`, `test_conflict_window`, and
`test_conflict_window_size` scenarios cover identical files that change on
one path during the sync, files that change between the listing snapshot and
the check for identical files, and `--size-only` files with the same size but
different content.
Co-Authored-By: ASRodrigo1 <45823308+asrodrigo1@users.noreply.github.com>
Adds a test-only TestFnConflictCheck option, called before checking
potential conflicts, and a "hook <when> <step>" scenario command that runs
an ordinary scenario step once when the next bisync reaches that point:
before-check (TestFnConflictCheck) or during-sync (TestFn). A hook that
never runs is noted in the log so a scenario can't silently skip its setup.
On shutdown the batcher logged "Committing uploads - please wait..."
but nothing once the commit had completed, so the log looked as if
rclone had stopped part way through. Log "Committing uploads - done"
when the pending batches have been committed.
This affects the backends which use the batcher, dropbox and
googlephotos.
This change adds a step to the bisync tests that checks that after a successful
run, the files named in each listing are exactly the files on that path. This
ensures that nothing is silently missing from the listings.
Before this change, when an existing file had changed on both paths, the
conflict winner was chosen using only the attributes that had changed on each
side. An attribute that hadn't changed was treated as zero, so --conflict-resolve
larger could pick the smaller file (for example, when one side only had its
modtime updated), and --conflict-resolve newer could fail to pick a winner when one
side's modtime hadn't changed.
After this change, the current size, modtime, and checksum (to the extent
enabled with compare settings) are now recorded for every changed file.
Before this change, when --conflict-resolve picked Path2 as the winner, the winning
file was copied to Path1 correctly but then removed from both listings while
updating them for the renamed (or deleted) Path1 copy. The next run saw the file
as new on both paths and had to check it again, and a change to it on one path
in that run was handled as if the file had never been synced. When Path1 won,
the listings happened to be updated in an order that avoided this.
This change fixes the issue by adding a guard to remove the file from the
listing only when dstNewName != dstOldName.
See #9939
Add an opt-in preflight that excludes only guaranteed tracked renames
from the max-delete count. Reuse normal bisync listing metadata,
validate strategy and flag conflicts early, and skip preflight work
when --force bypasses the safety check.
The max_delete_track_renames scenario uses the default hash
track-renames strategy, which requires a hash common to both paths.
Skip it on remote combinations without one, such as crypt, instead of
failing the integration tests.
Fixes#8685
When bisync is interrupted with a graceful shutdown it keeps the files
which transferred successfully in its listings and rolls the rest back.
An operator precedence mistake in that check meant a transfer of an
empty file (or one of unknown size) was kept even when it had failed,
so bisync recorded it as synced when it had not been.
Before this change, `fastCopy` created a cancellable context for the sync and
stored its cancel func on the `bisyncRun`, but only ever called it when
gracefully shutting down. On a normal run, it was never called, and until it is
cancelled, a context from `context.WithCancel` stays registered with its nearest
cancellable ancestor.
For an rc job, that ancestor is the job's own context, which the job registry
retains for `--rc-job-expire-duration`. The sync context carries bisync's
`LoggerOpt`, whose `LoggerFn` is a method value on `*bisyncRun`, so a finished
run was kept alive -- including the Path1 and Path2 listings -- for as long as the
job was.
This change fixes the issue by cancelling the sync context when `fastCopy`
returns. The cancel func is still stored on the `bisyncRun`, so a graceful
shutdown can still interrupt a sync that is in progress.
Before this change, --conflict-loser pathname assumed --conflict-resolve none,
following the legacy behavior prior to v1.66. This produced unexpected behavior
when used with a different --conflict-resolve option.
This change fixes the issue by ensuring that --conflict-loser pathname looks for
the correct name on the side not being renamed, when only one side should be
renamed.
https://forum.rclone.org/t/bisync-does-not-copy-the-winner-file-to-the-loser-site/53768
Replace AuthRequired bool with NoAuth bool on the rc.Call struct and
flip the auth check logic. Previously endpoints were unauthenticated
by default and had to opt in with AuthRequired: true, which led to
security vulnerabilities when developers forgot to set the flag.
Now all endpoints require authentication by default. Only explicitly
safe read-only endpoints are marked with NoAuth: true:
- rc/noop
- rc/error
- rc/list
- core/version
- core/stats
- core/group-list
- core/transferred
- core/du
- cache/stats
- vfs/list
- vfs/stats
- vfs/queue
- job/status
- job/list
See GHSA-25qr-6mpr-f7qx, GHSA-jfwf-28xr-xw6q
The temp directory name used random.String(2) giving only 676 possible
values. When multiple concurrent tests started in the same second, they
shared the same timestamp prefix, causing name collisions and shared
temp directories. This led to lock file conflicts, listing file races,
and file deletion errors.
Increase to random.String(8) to make collisions effectively impossible.
Lockfiles with invalid JSON content caused bisync to fail permanently
because lockFileIsExpired() logged the decode error but still fell
through to the "valid lock file" path with zero-value TimeExpires.
Now when a JSON decode error is detected:
- If --max-lock is set (< basicallyforever): treat garbled lockfile as
expired, mark listings failed, and proceed (safe assumption: the
previous bisync run crashed and left garbage).
- If --max-lock is not set (default): log a clear error telling the
user the lockfile needs manual inspection, and return false.
This adds 11 previously-missing rc params for newer bisync features.
It also makes optional parameters truly optional. (Previously, callers were
required to supply every single one, even if using the default value.)
In this commit
d240d044c3 check: improved reporting of differences in sizes and contents
We adjusted the sense of operations.CheckIdenticalDownload to return
true if files are identical as is implied by the name, but we forgot
to invert the logic in the bisync DownloadCheckFn which caused lots of
tests to fail.
Before this change bisync adjusted the global MaxCompletedTransfers
variable which caused races.
This adds a SetMaxCompletedTransfers method and uses it in bisync.
The MaxCompletedTransfers global becomes the default. This can be
changed externally if rclone is in use as a library, and the commit
history indicates that MaxCompletedTransfers was added for exactly
this purpose so we try not to break it here.
Before this change bisync was adjusting MaxCompletedTransfers in order
to clear the done transfers from the stats.
This wasn't working (because it was only clearing one transfer) and
was part of a race adjusting MaxCompletedTransfers.
This fixes the problem by introducing a new method RemoveDoneTransfers
to clear the done transfers explicitly and calling it in bisync.
Before this change CaptureOutput could trip the race detector when
used concurrently. In particular if go routines using the logging are
outlasting the return from `fun()`.
This fixes the problem with a mutex.
Before this change, rclone could crash during modifyListing if a rename's
srcNewName is known but not found in the srcList
(srcNewName != "" && new == nil).
This scenario should not happen, but if it does, we should print an error
instead of crashing.
On #8458 there is a report of this possibly happening on v1.68.2. It is unknown
what the underlying issue was, and whether it still exists in the latest
version, but if it does, the user will now see an error and debug info instead
of a crash.
Before this change, koofr failed certain bisync tests because it can't set mod
time without deleting and re-uploading. This caused the "nothing to transfer" log
to not get printed where expected (as it is only printed when there are 0
transfers, but koofr requires extra transfers to set modtime.)
This change fixes the issue by ignoring the absence of the "nothing to transfer"
log line on backends that return `fs.ErrorCantSetModTimeWithoutDelete` for
`obj.SetModTime`.
The "There was nothing to transfer" log is only printed when the number of
transfers is exactly 0. However, there are a variety of reasons why the transfer
count would be expected to differ between backends. For example, if either side
lacks hashes, the sync may in fact need to transfer, where it would otherwise
skip based on hash or just update modtime. Transfer stats will also differ in
the "src and dst identical but can't set mod time without deleting and re-
uploading" scenario (because the re-upload is a transfer), and where --download-hash
is needed (because calculating the hash requires downloading the file, which is
a transfer).
Before this change, these expected differences would result in erroneous test
failures. This change fixes the issue by ignoring the absence of the "nothing to
transfer" log where it is expected.
Note that this issue did not occur before
https://github.com/rclone/rclone/commit/9e200531b1490000656031c42fff4d95477e1b46
because the number of transfers was not getting reset between test steps,
sometimes resulting in an artificially inflated transfers count.
Before this change, TestBisyncConcurrent would still run the "basic" test case
if a non-blank -case arg was used to specify a case other than "basic". This
change fixes it by skipping in this scenario.
Nick Craig-Wood