Document forward slashes and escaped backslashes in JSON, including
object parameters sent through a query string or form data. Distinguish
these from ordinary string parameters that do not use JSON encoding.
Fixes#6673
Co-authored-by: Codex <noreply@openai.com>
Before this change, `cmd/bisync/testdata/bisync_vscode_debuggers_launch.json`
was in the repository, and every `-golden` run regenerated it from
`fstest/test_all/config.yaml`, adding unnecessary noise to commit diffs.
After this change, the file is in `.gitignore` and no longer tracked.
`-golden` still writes it locally, for copying debug configurations into
VS Code's `launch.json`.
Before this change, `docs/content/bisync.md` kept its own changelog, and
each bisync pull request added a line under the same heading. The rclone
changelog is already made from commit messages (`bin/make_changelog.py`),
so the entries duplicated it, and caused unnecessary merge conflicts: open
pull requests conflicted with each other whenever one of them merged.
After this change, the section links to the rclone changelog and to the
last version of the bisync changelog on GitHub. The `## Changelog` heading
stays, so links to `/bisync/#changelog` still work.
Before this change, GitHub showed the bisync golden test files
(`cmd/bisync/testdata/*/golden/`) in full in pull request diffs, where a
new test scenario can add hundreds of lines of expected output.
After this change, they are marked `linguist-generated`, like `MANUAL.*`
and `rclone.1`, so GitHub collapses them in diffs and leaves them out of
the language statistics. They are still one click away in a diff.
Before this change, `cmd/bisync` was excluded on `TestS3Rclone`
(`rclone serve s3`), one of the backends `108504963` excluded because the
tests always failed there at the time. After this change, they run there
again: with `test_all` on Linux, every case passes or skips, with or
without the preceding changes to the test harness.
Before this change, the `{eol}`, `{tab}` and `{chr:XX}` placeholders in a
scenario became raw bytes in the name the harness handed to `Put`. A raw
control character isn't a valid rclone name: rclone names are in the
`Standard` encoding, where a newline is `␊`, and `FsEncoding` in
`fstests` encodes every raw name before uploading it for the same
reason.
The `extended_filenames` scenario dates from 2021, when bisync was tested
on local, Drive, Dropbox, OneDrive, S3 and SFTP, but not on crypt or
memory. Its header said that all rclone backends, including local,
return `\n` as a glyph. That was an assumption, and it doesn't hold. A
backend with an encoder lists a raw control character as its glyph
because `ToStandardName` runs `Standard.Encode` on every name it
returns. A backend without one (crypt, memory, linkbox, netstorage,
pixeldrain, storj) lists exactly the name it was given. The same thing
happens with no bisync involved: `rclone touch` with a raw newline in
the name stores a raw newline on crypt, and `sync` from there to local
then copies and deletes the file on alternate runs.
The scenario has never passed on crypt or memory for the right reason.
Its golden listings were recorded on local with `␊`, so until January
2024 they would not have matched the raw names crypt lists, but the
scenario wasn't run there. `57624629d` ran it on crypt, treated the raw
names as an encoding difference, and normalized them away in both the
log and the listings, so from then on the goldens matched. `81d7b6f29`
added the check that each listing matches its path, which that
normalization doesn't cover, and the step has failed on those backends
since: local lists the copy as `␊` while the listing, built from the
source names, holds the raw newline. The pre-flight probe fixed in the
previous commit skipped the scenario on many nightly backends in the
meantime, which kept the failure out of the nightly results.
After this change, a placeholder for a control character (0x01-0x1f and
0x7f) becomes its `Standard` glyph, so every backend receives the same
name. On local the file on disk is unchanged, since the local backend
decodes the glyph to the raw character, and the golden logs are
unchanged. Placeholders for other bytes, such as the invalid UTF-8 in
`{chr:81}` and `{chr:fe}`, still become raw bytes, which `Standard`
leaves as they are.
Before this change, the pre-flight check for the `normalization` and
`extended_filenames` cases uploaded its probe file to the path root, which
already held the scenario's files, and compared the first entry each path
listed. That entry was usually an unrelated file, so the result depended on
listing order rather than on how each backend stored the probe name. On 42
of 71 backends in the 2026-10-15 nightly, the paths appeared to differ and
the cases were skipped; elsewhere the check passed without examining the
probe at all. The check also named `extended_char_paths`, a case renamed to
`ext_paths` in 2024. The probe's uploads counted in the step's transfer
stats, which removed `There was nothing to transfer` lines from the logs of
the checked cases.
After this change, the probe goes in a fresh subdirectory with its own
stats group, and the check compares the name listed back from there. It
names only `normalization` and `extended_filenames`: `ext_paths` uses none
of the probe's invalid UTF-8 or emoji, and doesn't use `--fix-case`. The
`normalization` and `extended_filenames` golden logs gain the lines the
probe's stats had hidden.
Before this change, the default SFTP encoding (`Slash,Del,Ctl,Dot`, the same
as rclone's standard form) passed names through unchanged, so a name on the
server containing a control character, such as a raw newline, reached rclone
in a non-standard form. Other backends, such as local on Linux, list the same
name in the standard form (`␊`), so the two never matched: syncing such a file
from SFTP to local deleted it on every second run, and bisync recorded
listings that didn't match the files.
After this change, the default encoding is `Slash,Dot`, the same as the local
backend's default on Linux, so control characters in names on the server are
converted to and from the standard form. Servers running on Windows can add the
Windows restricted characters with `--sftp-encoding`, as now documented.
The issue was discovered on the bisync integration tests.
This also fixes `mkParentDir`, which encoded the parent `.` of a top-level
file as a name. With the new default encoding, that created a stray `．`
directory on the server.
A connection returned after an error without an FTP reply code can no
longer be trusted, so close it instead of returning it to the pool.
Errors carrying an FTP reply code still get a NOOP liveness probe.
Add a unit test driving putFtpConnection against a fake FTP server
covering network-error discard, live-connection pooling after an FTP
reply-code error, and discard when the NOOP probe fails.
deleteObject() replied 404 for any error from newObject(), while
serveObject() in the same file replies 404 only for
fs.ErrorObjectNotFound and 500 otherwise.
restic treats a 404 on DELETE as final and never retries, so a
transient lookup failure while removing a lock left the lock behind
and broke the next exclusive operation (eg forget --prune failing with
"repository is already locked").
This change mirrors serveObject() handling of 404/500 errors in
deleteObject().
Fixes#10100
Co-authored-by: bunnysayzz <bunnysayzz@users.noreply.github.com>
Before this change, renaming a file or directory through a name that
matched it only by ignoring case (`--vfs-case-insensitive`, the default on
macOS and Windows) or Unicode normalization left its old entry in the
parent directory. The item was listed twice, once under its new name and
once under the old key, and the old name still resolved to it until the
directory cache expired. `Dir.Rename` removed the parent's entry under the
name it was given rather than the item's real name. For a directory
renamed within its own parent, this was masked until `36ce1a28f`, which
removed a second deletion in `Dir.rename`.
After this change, `Dir.Rename` removes the entry under the item's real
name. The issue was discovered on the bisync integration tests.
Before this change, on backends without server-side `Move`, renaming a
file to an equivalent name deleted it: a name differing only in case on
a case-insensitive backend, or only in Unicode normalization on a
backend such as Cloudflare R2, which treats NFC and NFD names as
equivalent. `operations.move` fell back to copying the file to the new
name and then deleting the source, but both resolved to the same object,
so the delete removed the file it just copied. This affected
`--fix-case` renames in `sync` and `bisync`, `--track-renames` with
`--no-unicode-normalization`, and renames through the VFS
(`rclone mount`, `rclone serve`). `MoveFile` and backends with
server-side `Move` already handled these renames by moving through a
temporary name.
After this change, the copy-and-delete fallback also moves through a
temporary name when the two names are equivalent. The issue was discovered on
the bisync integration tests.
Cellar is the S3-compatible object storage service of Clever Cloud,
hosted in France.
rclone's defaults work for it: virtual-host style addressing, v2
listings, URL-encoded listings and BucketAlreadyExists behave like AWS.
Cellar does not apply Object Lock headers sent with
CreateMultipartUpload, so object_lock_supported is set to false and the
Object Lock integration tests are skipped. The docs tell users to set
--s3-object-lock-set-after-upload, which applies the lock on multipart
uploads with separate API calls.
The SSE-C options are exposed since Cellar supports customer-provided
keys. The docs note that Cellar rejects CopyObject with an SSE-C source,
so --no-update-modtime is needed and encrypted objects smaller than
--s3-copy-cutoff can't be moved server-side.
The integration tests pass against cellar-c2.services.clever-cloud.com.
jwt v5 removes StandardClaims so use RegisteredClaims instead. v5
sends a single audience as an array of strings, so the audience is
overridden in boxCustomClaims to keep sending it as a string as
before.
This adds a test which checks the JWT assertion sent to the token
endpoint is identical to the one sent with jwt v4.
The tcell v2 pin at v2.9.0 is removed as tcell v3 contains the fix for
duplicated key events on Windows (tcell issue 1124).
Note that tcell v3 uses the Windows VT console modes so ncdu requires
Windows 10 build 1703 or later.
See: #9539
Sorter.Send selects on both the output and error channels of the ext
sort. If the ext sort failed it closes both, so the select could pick
the closed output channel and return without the error. This was seen
as a flaky TestSorterExtTempFileError under the race detector.
extsort now only creates temp files when the data doesn't fit in
one chunk, so the test needs to add more than a chunk of entries to
provoke the temp file error.
This fixes the code to compile with the updated dependencies:
- dlna: pass a *slog.Logger to the ssdp server as required by
anacrolix/dms v1.8.0
github.com/Files-com/files-sdk-go/v3 was not updated as the latest
version requires go 1.26.9.
The tcell and go-systemd pins are kept and github.com/hanwen/go-fuse/v2
is pinned to v2.11.0 as v2.12.0 fails to compile on 32 bit linux - see
https://github.com/hanwen/go-fuse/issues/628
With a relative path override such as `--sftp-path-override @.` (useful
on chrooted hosting where the SSH shell starts in the directory that
SFTP shows as "/"), a file in the root gets the shell path "-name".
`quoteOrEscapeShellPath` does not escape "-", so the hash command
became `md5sum -name`, which md5sum parses as an option. The hash then
failed and the transfer was reported as "corrupted on transfer".
Absolute shell paths always start with "/", so only relative overrides
were affected. Prefix such paths with "./".
When the sftp backend uses an external ssh program (--sftp-ssh), one
goroutine waits for the ssh program to finish by calling cmd.Wait().
cmd.Wait() sets cmd.ProcessState when the program ends.
At the same time, CanReuse() calls exited(), which reads
cmd.ProcessState without any lock. This is a data race and the Go
race detector reports it. It happens in normal use, for example when
a connection is put back in the pool after a stat.
The race only decides whether a pooled connection can be reused, so
it is unlikely to cause problems in practice, but it makes programs
that use the sftp backend fail when they are tested with -race.
This fix sets an atomic.Bool after cmd.Wait() returns and makes
exited() read that instead of cmd.ProcessState.
It also adds a test which calls exited() while Wait() is running. It
fails with -race before this fix and passes after it.
Auto Drive is permanent storage on the Autonomys Network, with an S3
compatible API at s3.auto-drive.autonomys.xyz.
Storage is permanent: a delete hides an object, and every version is
kept. The provider sets:
- force_path_style, as the API uses path-style requests.
- copy_cutoff to the maximum. UploadPartCopy is not supported, so
server-side copies always use a single CopyObject.
- object_lock_supported false. Object Lock is always on in COMPLIANCE
mode and can't be set by clients, so rclone's Object Lock tests
can't pass.
- No Purge. Deleting a version is always refused, so the S3 Purge,
which deletes every version, fails. rclone's generic purge hides
the current version of each object instead.
- no_check_bucket. There are no bucket endpoints: buckets are created
by the first upload into them, and a CreateBucket request would be
stored as an object.
Before this change, the integration tests dropped Dropbox's "Too many
requests or write operations" notice from the logs only in the format it
had when the tests were written, which began with the error code
(`too_many_requests/...`). Since 110bf468a (v1.65), the notice begins with
`Error `, so it was left in the log and caused a miscompare whenever
Dropbox rate limited a run.
After this change, the notice is dropped in either format.
Before this change, the integration tests checked whether each remote could
set modtimes on every `bisync` step (`checkPreReqs`, called from
`runBisync`), not once per remote. On a remote that isn't `IsLocal`, each
check sleeps for 1s twice (the Google Cloud Storage per-object rate limit
workaround from be73a10a9), for each side. That added 2-4s to each of the
~110 `bisync` steps in each test, which on a fast remote was most of the
run: on the integration test server, the `:memory:` tests took about 14
minutes.
After this change, the result of the check is remembered for each remote,
so it runs, and sleeps, once per remote per test run. The `:memory:` tests
now take about 7s locally.
When a remote is configured with RCLONE_CONFIG_<NAME>_<OPTION> the remote
name is only converted to uppercase, so characters such as "-" and "." are
not changed to "_" and may not be settable in some shells.
Explain this in the environment variables section and link to it from
"Valid remote names".
Deleting the contents of a directory and then the directory itself,
before a listing has confirmed the deletes, left the directory's
virtual entries behind. They are counted in every parent directory, and
a directory which counts virtual entries refuses to forget its cache, so
vfs/forget could not make changes made outside rclone visible.
This lasted until the removed directory's own cleanup timer purged its
entries, which happens 2 x --dir-cache-time later.
This fix drops the removed directory's virtual entries once it has been
removed.
Fixes#10083
When a Drive listing error closed the ListR input while directories
were still in the overflow queue, the overflow goroutine sent on that
nil channel while holding the mutex. The main loop then blocked on the
same mutex, so the sync never returned the error.
refillListRInput now stops when the input is already closed, and it
does not add those jobs to the wait group.
Fixes#10069
If a file that hadn't been read was shrunk with Truncate, the cache
file was cut to the new size but the kept part of the file still had
to be fetched on close. The downloader wrote the whole remote object
into the cache file, which grew it back to its old size, so the upload
replaced the object with the original contents.
Stop the downloader writing past the end of the item, and add tests at
the vfscache and vfs levels.
Fixes#10047
gofakes3 prior to v0.0.11 defined the IsTruncated XML field in
ListMultipartUploadsResult and ListMultipartUploadPartsResult with
`xml:"IsTruncated,omitempty"`. When an upload result was not truncated
(IsTruncated == false), the <IsTruncated> tag was omitted from the XML
response.
According to the AWS S3 REST API specifications for ListParts and
ListMultipartUploads, IsTruncated is a required boolean element. Strict
S3 clients such as the official AWS Go SDK (used by Docker Distribution
and OCI registries) unmarshal this field as a *bool pointer. Because the
tag was omitted, the pointer remained nil, causing Docker Distribution
to panic with a nil-pointer dereference at s3.go:593 (for *resp.IsTruncated)
during multi-chunk blob uploads.
Updating github.com/rclone/gofakes3 to v0.0.11 brings in rclone/gofakes3#22,
which removes omitempty and ensures <IsTruncated>false</IsTruncated> is
always present in the response XML.
Fixes#10078
Before this change, nothing checked that the rc `sync/bisync` call accepted
every bisync flag, so the rc could fall behind the command line as flags
were added, as in #7799.
After this change, `TestRcFlagParity` passes each non-hidden bisync flag to
the rc, under the parameter name `rc.md` uses, with a value of the wrong
type. Every getter `rcBisync` uses rejects it, so the call fails only if
the parameter is read, and the test names any flag the rc doesn't handle.
Before this change, since v1.74.0 (bb78eb8ab), the rc `sync/bisync` call
read the `backupdir1` and `backupdir2` parameters (the names used before
they were renamed to `backupDir1` and `backupDir2`) only when the new name
was also given with an invalid value, because the fallback sat inside an
`rc.NotErrParamNotFound(err)` check, which is true only for an invalid
value. Otherwise, `backupdir1` and `backupdir2` had no effect.
The same change made an invalid value for any optional parameter (for
example `"resync": "yes"`, or a `filtersFile` that isn't a string) be
treated as if the parameter were missing, instead of returning an error.
After this change, the old backup dir names are read when the new ones are
missing, and an invalid value returns an error again, as it did before
v1.74.0.
Before this change, bisync treated a `--max-delete` above 100 as 100%, so
the `--max-delete` check never triggered. A run that left no file on a side
unchanged was still stopped, by the "all files were changed" or empty
listing checks, but one that deleted most files while leaving even one
unchanged was not.
For bisync, `--max-delete` has always been a
percentage, "in range 0..100" as Ivan put it when he designed it
(https://github.com/rclone/rclone/pull/5164#discussion_r633470815), but the
range was never enforced. So a value meant as a file count, such as
`--max-delete 500`, turned the check off. Now that the rc takes
`--max-delete` from the global config too, that includes a `--max-delete`
the rcd itself was started with.
After this change, a `--max-delete` above 100 is an error, as the rc's
`maxDelete` parameter already was.
Before this change, `--max-delete` and `--dry-run` were read from the
global config by `applyContext`, which only the command line called. The
rc `sync/bisync` call started from empty `Options` and never ran it, so it
ignored both settings:
- A dry run requested through the global option (`_config`
`{"DryRun": true}` or the flat `dry_run` parameter) was ignored.
`setDryRun` resets `ci.DryRun` from `opt.DryRun` before copying, and
`opt.DryRun` was only set by bisync's own `dryRun` parameter.
- When the `maxDelete` parameter was omitted, the `--max-delete` limit was
0%. A `max_delete` set through `_config` or the flat parameter was not
used for that limit, and applied to the inner syncs as a file count
instead.
After this change, `Bisync` runs `applyContext` itself, on its own copy of
the config, so the command line, the rc, and any other caller follow the
same rules, and `Options.MaxDelete` and `Options.DryRun` are set by
`Bisync`. With no `--max-delete` anywhere, the limit is 50%, as #5680 says
the rc should.
The rc's `maxDelete` and `dryRun` parameters now set the call's config.
`maxDelete` overrides the global `--max-delete`. `dryRun: true` asks for a
dry run, but `dryRun: false` doesn't cancel one from the global config, so
the run is a dry run if either asks for one. The `maxDelete` parameter is
also documented again; it was left out when the rc help became generated
from the command's flags.
Before this change, when bisync updated a listing entry, it kept the old
modtime if the new one was less than a second earlier. This dated back to
when bisync truncated modtimes to the destination's precision before
recording them (`ConvertPrecision`), so that a truncated time would not
overwrite the more precise original.
`ConvertPrecision` was removed in 956c2963f, and since then, bisync has kept
the original precision and relied on the modify window when comparing. But
the rule in `ls.put` was left behind, and it then fired only when a file
really had an earlier modtime, recording a time the file did not have. This
could cause:
- a `--resync` to abort with "path1 and path2 are out of sync" when Path1's
version was slightly older than Path2's
- later runs to report changes that didn't happen, for a file changed to a
version less than a second older, or copied from a remote with lower
modtime precision within the same second as the old version (for example,
from SFTP to local)
After this change, bisync records the modtime it is given, like any other
update. Comparisons already account for differences in precision through
the modify window.
deleteEmptyDirectories gated on the cumulative accounting stats
(accounting.Stats(ctx).Errored()) instead of the current operation's error.
On a long-running rcd that reuses one stats group (rc "_group") across many
sync/move calls, a single earlier error left Errored() true forever, which
permanently disabled empty-source-directory deletion for every later,
error-free operation sharing the group - each run just logged "not deleting
directories as there were IO errors" and never cleaned up.
Gate on s.currentError() instead, so empty-directory cleanup depends only on
the current operation's outcome. This matches the per-operation guards already
used elsewhere in run(), and is a no-op for normal single runs where
Errored() (s.errors != 0) tracks the same errors that reach s.currentError().
Opening an archive in the root of a remote, such as remote:file.zip,
failed with "file does not exist" because the archive's leaf name was
found with path.Base, which doesn't understand remote names. It is now
found with fspath.Split.
Drime won't list more than 20,000 entries of a directory. Asking for a
later page gets the last page it allows again, so rclone asked for the
same page forever and used more and more memory until it was killed.
Return an error instead. An error stops sync deleting files based on
the listing and uploading duplicates of files it couldn't see.
Also return listing errors from getItem rather than "object not found".
- TestCheckConflictsSkipsCheck: the check for identical files is skipped
with --size-only, and with --ignore-checksum only when the paths have no
hash in common.
- test_equal_skipped: an identical file that is new on both paths with
--ignore-checksum is recorded, so deleting it on Path1 deletes it on
Path2. Adds the "ignore-checksum" scenario argument.
- test_conflict_window: Path1 changes between the listing snapshot and
the check for identical files.
Co-Authored-By: ASRodrigo1 <45823308+asrodrigo1@users.noreply.github.com>
Before renaming a sync conflict, bisync checks whether the two files are
identical. It compares hashes using check or cryptcheck, and as a last
resort, when neither is available, it compares the files by downloading
them (--download), so that the result is certain regardless of checksum
support.
Before this change, the fallback to --download happened only when bisync
knew up front that the remotes had no usable hash. It did not happen
when a hash that was expected to be present turned out to be missing
(for example, a backend that normally supports hashes returned a blank
one, or cryptcheck could not get the underlying hash). Those files were
counted as identical based on size alone.
After this change, a missing hash falls back to --download, the same as
when the remotes have no hash in common. --size-only and
--ignore-checksum still disable the fallback and keep the size-only
comparison.
Co-Authored-By: ASRodrigo1 <45823308+asrodrigo1@users.noreply.github.com>
Before this change, when a file was new or changed on both paths and the pre-sync
conflict check deemed the two copies identical, bisync skipped the copy and then
rechecked the file after the sync to get its listing info. If the recheck could
not confirm equality (for example, hashes unavailable due to --size-only or
(--ignore-checksum + no common hash)), the file was rolled back to be retried on
the next run, and every later run repeated the same check with the same result.
A later change on one side was then handled against a missing or stale listing
entry.
After this change, files we've declared identical pre-sync are no longer
rechecked post-sync. The pre-sync conflict check now also returns the values it
compared, and a skipped file is recorded from the listing snapshot. If either
path changed between the snapshot and the check, the prior listing is kept and
the next run rechecks.
When --size-only or (--ignore-checksum + no common hash) are in use, the "new or
changed on both paths and equal" case skips the conflict check entirely, because
it is unnecessary. The check could only compare sizes, which we already know are
equal from the snapshot.
The new `test_equal_skipped`, `test_conflict_window`, and
`test_conflict_window_size` scenarios cover identical files that change on
one path during the sync, files that change between the listing snapshot and
the check for identical files, and `--size-only` files with the same size but
different content.
Co-Authored-By: ASRodrigo1 <45823308+asrodrigo1@users.noreply.github.com>
Adds a test-only TestFnConflictCheck option, called before checking
potential conflicts, and a "hook <when> <step>" scenario command that runs
an ordinary scenario step once when the next bisync reaches that point:
before-check (TestFnConflictCheck) or during-sync (TestFn). A hook that
never runs is noted in the log so a scenario can't silently skip its setup.
