Before this change, with --files-from and --no-traverse, a single file
that could not be read (for example permission denied) stopped all the
other files in the list being copied.
This happened because the error was returned from the listing, which
caused the whole source listing to be discarded.
This change counts and logs such per-file errors and carries on, so the
readable files are still copied and rclone exits with a non-zero error
code.
Fixes#9115
Option help strings are also used to generate the website documentation,
so some contain markdown links with root-relative targets such as
[encoding section in the overview](/overview/#encoding). These render
correctly on rclone.org but are confusing in the interactive config
prompt, where the user sees the raw markdown and the link has no
reachable root.
Rewrite such links to text (https://rclone.org/path) when showing an
option's help in the interactive config. The raw help is left unchanged
so documentation generation is unaffected.
The new "trace" dump flag attaches a net/http/httptrace ClientTrace to
each HTTP transaction and logs the connection level events - DNS
resolution, TCP connect, TLS handshake (including the negotiated TLS
version, cipher, ALPN protocol and server certificate), connection
reuse, request write and time to first response byte. Each line is
tagged with the time elapsed since the start of the transaction and the
request pointer so it can be correlated with the other dumps.
This is complementary to the existing dump flags: it shows how the
connection behaved rather than what was sent, which is useful for
debugging connectivity, DNS, TLS, proxy and keep-alive problems.
The new "errors" dump flag makes the HTTP dump conditional on the
transaction failing with a retryable error (a transport error, HTTP 429
or HTTP 5xx), so first-failure diagnostics can be captured without the
noise of dumping every transaction. The existing dump flags continue to
control what is dumped, for example --dump errors,bodies, and on its own
--dump errors dumps the headers.
ResetCounters unconditionally restarted the average loop, spawning a
ticker goroutine that pinned the StatsInfo even when no loop had been
running before. statsGroups.delete calls ResetCounters on every removed
group, so deleting N stats groups leaked N goroutines and prevented GC
of the underlying StatsInfo objects.
Only restart the loop if it was active before the reset.
A connection string can carry global.* options which change rclone's
process-wide configuration (e.g. global.http_proxy). This is
undesirable for the rc interface which was designed to have multiple
users or connections at once. The rc interface has the `_config`
mechanism for setting request scoped global config.
This blocks global.* options on all rc paths by marking the context as
a remote control request at the rc boundaries. fs.NewFs then skips
applying global.* to the process-wide config for a marked context.
The marker is reapplied in fs.CopyConfig, which is the call rclone
uses to detach context but keep config.
global.* options still apply to the individual backend they are set
on, exactly like override.* options; they just no longer leak into the
rest of the process. Remotes created directly on the command line are
unaffected as are remotes defined in the config file.
See: GHSA-qw24-gh76-8rvv
The --rc-serve GET/HEAD file serving path accepted bracketed inline
remotes from the URL and instantiated them, so a single
unauthenticated request could run a command as the rclone user via
backend options such as webdav bearer_token_command or sftp ssh, read
arbitrary local files, or change process-wide config via global.*
options.
This was the GET/HEAD equivalent of the POST hole fixed for
CVE-2026-41179, which only guarded the rc call dispatch path.
Now, unless the rc server has authentication configured or
--rc-no-auth is set, the serve path only allows remotes already
present in the config file: inline remotes, connection string
parameters and bare local paths are rejected. Connection string
global.* options are never honoured on the serve path, even when
authenticated.
See: GHSA-qw24-gh76-8rvv
JSON logs reported "source":"slog/logger.go:256" instead of the real
caller. getCaller skips logging-machinery frames by file path, but
release builds use -trimpath which rewrites the standard library slog
frame's path to "log/slog/logger.go" - matching neither the "/log/" nor
the "log.go" check, so it was reported as the source. Also skip frames
whose function belongs to the log/slog package, which is immune to
-trimpath.
Make rc respond with a 202 status code (instead of 200) if `Prefer: respond-
async` was passed. Keeps backwards compatibility for current clients while also
allowing the OpenAPI schema & generators to differentiate the responses
properly.
The metrics_addr option was registered twice: once explicitly and once
implicitly via AddPrefix(libhttp.ConfigInfo, "metrics", ...). Both
pointed at the same MetricsHTTP.ListenAddr field, so options/info
returned a duplicate entry.
Drop the explicit entry and use SetDefault to keep the empty default
(so the metrics server stays off unless configured), matching the
pattern already used for rc_addr.
Fixes#9419
operations.NeedTransfer's equality check may have deleted pair.Dst as
a precursor to re-uploading it if SetModTime returns
ErrorCantSetModTimeWithoutDelete (e.g. Dropbox). If so skip the eager
delete of the destination if --fix-case will rename it to a different
name. The rename itself replaces the destination, and any subsequent
re-upload happens at the correctly-cased path.
See: #8881
This reverts commit de67f29b3f.
This solved the original Dropbox "from_lookup/not_found" failure, but
broke --fix-case on case-sensitive backends that update modtime via a
server-side copy (such as S3 on Cloudflare R2).
The TransformFile tests in fs/sync call operations.TransformFile
immediately after MoveDir. On eventually-consistent backends the
internal NewObject lookup can momentarily fail with "object not
found", making the tests flaky.
This wraps the two operations.TransformFile calls in TestTransformFile
and TestManualTransformFile with fstest.Retry
When --fix-case was used (e.g. by bisync) on backends that can't set
modification times in place - such as Dropbox - files whose content
matched but whose modtimes differed would fail to rename with a
"from_lookup/not_found" error and abort the operation.
This happened because operations.NeedTransfer was called before the
fix-case rename. NeedTransfer's equality check would delete the
destination as a precursor to re-uploading it (the standard way to
update a modtime on these backends), so by the time the rename ran the
file no longer existed on the remote.
Fix by running the fix-case rename first, so that any subsequent
delete/re-upload happens at the correctly-cased destination path.
See: #8881
Avoid side effects by using own logger instance
- Importing fs/log only sets rclone's private logger via fs.SetLogger,
so internal rclone logging works from the moment the package is
imported but the process-wide slog default is left untouched.
- slog.SetDefault and slog.SetLogLoggerLevel move into InitLogging,
which is called explicitly from the CLI (cmd/cmd.go), the librclone
wrapper and the integration test framework. So rclone-as-a-program
keeps capturing log.Print/log.Fatal and slog.Default() output as
before.
Library consumers that import fs/log without calling InitLogging now
keep their own slog default and can safely route rclone output back
into it via log.Handler.SetOutput without recursing.
Fixes#8907
Co-authored-by: Nick Craig-Wood <nick@craig-wood.com>
Implement multipart upload support with configurable chunk size and concurrency options
Enable OpenChunkWriter with per-chunk encryption
Enhance multipart upload handling with new upload cutoff and error management for small files
Replace AuthRequired bool with NoAuth bool on the rc.Call struct and
flip the auth check logic. Previously endpoints were unauthenticated
by default and had to opt in with AuthRequired: true, which led to
security vulnerabilities when developers forgot to set the flag.
Now all endpoints require authentication by default. Only explicitly
safe read-only endpoints are marked with NoAuth: true:
- rc/noop
- rc/error
- rc/list
- core/version
- core/stats
- core/group-list
- core/transferred
- core/du
- cache/stats
- vfs/list
- vfs/stats
- vfs/queue
- job/status
- job/list
See GHSA-25qr-6mpr-f7qx, GHSA-jfwf-28xr-xw6q
The operations/fsinfo RC endpoint was registered without AuthRequired,
allowing unauthenticated callers to instantiate arbitrary backends via
inline backend definitions.
See GHSA-jfwf-28xr-xw6q
Snapshot the NoAuth setting when the RC server is created rather than
reading it from the mutable options struct on each request. This
prevents any runtime mutation of rc.NoAuth (e.g. via options/set)
from disabling the auth gate for protected RC methods.
See GHSA-25qr-6mpr-f7qx
The options/set RC endpoint was registered without AuthRequired,
allowing unauthenticated callers to mutate global runtime options
including rc.NoAuth, which disables the auth gate for all protected
RC methods. Require authentication for options/set.
See GHSA-25qr-6mpr-f7qx
The files.com integration tests for rcat/copyurl were failing because
fs/account.Account was declaring a ReadAt method when the underlying
handle did not support it. The files.com SDK decided to use the ReadAt
method to speed transfers up which failed.
ReadAt and Seek methods were added in this commit to support the
archive command:
409dc75328 accounting: add io.Seeker/io.ReaderAt support to accounting.Account
This fixes the problem by adding new methods to the Account object
WithSeeker/WithReaderAt/WithReadAtSeeker which produce an object with
the desired methods or errors if it isn't possible.
This stops Account advertising things it can't do which is bad Go
practice.
Bind the RC server to localhost:0 and read the bound URL back via a
new rcserver.Server.URLs() accessor instead of pre-allocating a port
in cmd/gui. This removes the small TOCTOU race window between
freePort() closing its listener and rcserver claiming the same port.
These Debugf calls in NewFilter() ran during GlobalOptionsInit(), before
InitLogging() configured the JSON log format. This caused plain-text
debug lines to leak to stderr when --use-json-log was set, breaking
tooling that expected only JSON output.
The resolved time values are already available via --dump filters so
this commit removes the debug messages.
When extsort.Strings() cannot create temporary files (e.g. due to
apparmor restrictions or permission denied), it returns a nil sorter
with the error on errChan. The code then called Sort() on the nil
sorter, causing a panic.
Check for nil sorter and return the error instead of panicking.
Fixes#9244
The format field was read in Handle() without synchronization while
setFormat() could write it concurrently from InitLogging(). This
caused a data race detected by the race detector, failing
TestListBucketsAuthProxy in cmd/serve/s3.
Fix by protecting all access to format with the existing mutex.
This reduces the number of go routines which can get out of hand when
using large --transfers and --multi-thread-streams from potentially
--multi-thread-streams * --transfers Go routines to --max-memory /
--multi-thread-chunk-size
It serializes the memory allocator in each transfer which should be
good for performance and reduce lock contention.
Before this change server side copies would show at 0% until they were
done then show at 100%.
With support from the backend, server side copies can now be accounted
in real time. This will only work for backends which have been
modified and themselves get feedback about how copies are going.
If the transfer fails, the bytes accounted will be reversed.
Before this change when doing a sync with `--no-traverse` and
`--files-from` we could call `NewObject` a total of `--checkers` *
`--checkers` times simultaneously.
With `--checkers 128` this can exceed the 10,000 thread limit and
fails when run on a local to local transfer because `NewObject` calls
`lstat` which is a syscall which needs an OS thread of its own.
This patch uses a weighted semaphore to limit the number of
simultaneous calls to `NewObject` to `--checkers` instead which won't
blow the 10,000 thread limit and is far more sensible use of OS
resources.
Fixes#9073
These stats weren't being updated in the global stats read by rc
core/stats:
- transferQueue
- deletesSize
- serverSideCopies
- serverSideCopyBytes
- serverSideMoves
- serverSideMoveBytes
Before the log re-organisation in:
8d353039a6 log: add log rotation to --log-file
rclone would write any backtraces to the --log-file which was very
convenient for users.
This got accidentally disabled due to a typo which meant backtraces
started going to stderr even if --log-file was supplied.
This fixes the problem.
