Before this change the parent process kept the remote control and
metrics servers it started in initConfig listening, so the daemon child
could not bind the same addresses and exited.
After this change the parent shuts those servers down just before
daemonizing, and the child starts them as usual.
With --auth-proxy, when several logins with the same credentials
arrived together and none was in the cache yet, as when an FTP client
opens several connections at once, each one ran the auth proxy program
and took its own reference to the backend. Only one of them was
remembered, so the others were never given back and the backend was
never shut down.
Logins which arrive together now share one run of the auth proxy
program and one reference to the backend.
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. With --vfs-cache-mode writes or full, files
are uploaded to the backend after the transfer which wrote them has
finished, which didn't count as a use. So an upload which hadn't
finished 5 minutes after the user's last command (or disconnection for
serve sftp) was stopped, leaving the file in the VFS cache but not on
the backend.
The backend is now kept until it has no files open for write and
nothing in the VFS cache waiting to be uploaded.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each request counted as a
use, so an upload or download lasting longer than that had its backend
shut down under it and failed.
Each request now holds the backend it uses until it has been served.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each request counted as a
use, so a download lasting longer than that had its backend shut down
under it and failed.
Each request now holds the backend it uses until it has been served.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each FTP command counted
as a use, so an upload or download lasting longer than that had its
backend shut down under it and failed with "context canceled".
Each FTP command now holds the backend it uses until it has finished,
which for a download is when the file is closed.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
Deleting a selection finished with "Successfully deleted all items!",
which claimed more than had happened - only the selected entries were
removed. It now names the number of items deleted.
The screen was not redrawn until the whole selection had been deleted,
which made the UI look hung, so a progress box is drawn before each
deletion.
Fixes#9516
Add an opt-in preflight that excludes only guaranteed tracked renames
from the max-delete count. Reuse normal bisync listing metadata,
validate strategy and flag conflicts early, and skip preflight work
when --force bypasses the safety check.
The max_delete_track_renames scenario uses the default hash
track-renames strategy, which requires a hash common to both paths.
Skip it on remote combinations without one, such as crypt, instead of
failing the integration tests.
Fixes#8685
To find the latest patch release of a minor version, for example
`rclone selfupdate --version 1.75`, selfupdate searched the listing of
downloads.rclone.org for href="./vX.Y.Z/". The leading ./ is a detail
of how Caddy's file server writes its links. A listing which linked to
the same directories as "vX.Y.Z/", which is an equally valid relative
URL, made selfupdate fail with "could not find the minor release".
This makes the ./ optional in the pattern, so selfupdate no longer
depends on which program wrote the listing, and takes the version from
a capture group rather than from fixed offsets into the match.
The listings written by rclone index now include the ./ for the
benefit of rclone versions without this fix, so this is to remove the
dependency for the future.
downloads.rclone.org is about to be served from static listings
written by rclone index instead of by Caddy's file server. Released
versions of rclone selfupdate find the latest patch release of a minor
version by searching that listing for href="./vX.Y.Z/", as written by
Caddy. rclone's listings linked to "vX.Y.Z/" without the ./ so `rclone
selfupdate --version X.Y` would have failed with "could not find the
minor release" for every rclone already installed.
Caddy prefixes every link with ./ so that a name with a colon in its
first path segment is not read as an absolute URL with a scheme (RFC
3986 section 4.2). rclone was already safe from that as url.URL.String
adds the ./ but only to names which need it, so links to plain names
had no prefix.
This prefixes all the links with ./ as Caddy does. It changes the
output of serve http, serve webdav and the rc server as well as rclone
index, since they share the code. The links resolve identically, and
it keeps the listings consistent with each other.
Now every URL has the prefix, the caddy.json template no longer needs
to add it.
After a sync which changed a known set of files there is no need to
walk the whole remote. --changed PATH, --changed-from FILE and
--changed-combined FILE tell rclone index what changed, and it
re-indexes only the directories containing those paths and their
ancestors, each with one non-recursive listing.
This makes directory listings for buckets and other remotes served as
static websites, for example S3 website endpoints or R2 behind
Cloudflare, so they can be browsed without directory listing support
on the host.
Use mime.FormatMediaType instead of unescaped string concatenation so a
directory name containing a double quote cannot inject extra disposition
parameters.
Fixes#9962
Use mime.FormatMediaType instead of unescaped string concatenation so a
directory name containing a double quote cannot inject extra disposition
parameters.
Fixes#9962
Copying an object onto itself, as clients do to replace its metadata,
reported success when the object didn't exist, and copying a missing
object to a different key failed with an internal error. Both now fail
with NoSuchKey as S3 does.
With --etag-hash auto the hash for the ETags was chosen once from the
remote serve s3 was started with. With --auth-proxy there is no such
remote so serve s3 crashed on startup, and when started by the rc it
was the wrong remote, so users whose backends lacked that hash got no
ETags.
The hash is now chosen from the backend of the user making each
request.
The metadata of every object uploaded was kept in memory for as long as
the server ran, even after the object was deleted, so it grew without
limit and the metadata of a deleted object reappeared on any object
later created at the same key other than through serve s3.
Deleting an object now forgets its metadata.
Listing a prefix with no delimiter walked the entire subtree below it into
memory and only then sliced out the requested page, so every page of a
listing cost a full traversal of the tree.
Walk the tree lazily instead, stopping as soon as the page is full and
skipping the subtrees an earlier page already returned. A page now costs a
number of directory reads proportional to the keys it returns rather than to
the size of the subtree, and ETags are computed only for the objects that
are actually returned.
Entries are emitted in the order their keys have in a flat keyspace, with a
directory sorting as if it carried its trailing slash, so that "a.txt" comes
before "a/b" as it does in a real S3 bucket. Without this a resumed listing
would silently skip keys across a page boundary.
The request log printed the escaped URL, so non-ASCII file names showed
up as long runs of %XX escapes. Log the unescaped URL path instead, as
the other serve commands do.
The box backend logged an int64 with %q which printed
%!q(int64=123) instead of the sequence ID.
serve s3 passed the message from gofakes3 as the format string, so
any % in it was interpreted as a formatting directive and the message
was mangled.
- Reject uploads when object lookup fails for reasons other than "not found".
- Serialize uploads to the same object to prevent races between the
existence check and the write.
When rclone archive create is run without a destination the archive
should be written to stdout, but ArchiveCreate called
CheckValidDestination on the nil dst and panicked with a nil pointer
dereference. Skip the destination check when there is no destination.
Also remove a leftover debug Printf that wrote to stdout before the
archive data.
Signed-off-by: Vladimir Babin <vovababin@gmail.com>
Listing a bucket without -R shows each prefix as one directory entry, so the
objects under it are not in the output. This is easy to read as a truncated
listing rather than as one level of a hierarchy.
Fixes#9797
The test asks InstallUpdate to install the latest beta into an
unwritable file and expects an error. When the binary under test
reports exactly the latest beta version (as make quicktest does right
after a beta is published from the same commit), InstallUpdate
correctly decides there is nothing to do and returns nil, and the test
then dereferences the nil error and panics.
Pin fs.Version to a fixed old value for the duration of the test so an
update is always attempted, and use require.Error so a missing error
fails the test instead of crashing it.
applyOptions consumes the "path" option into vol.Path rather than leaving
it in vol.Options, but restoreState rebuilt the options with only fs and
type. The explicit path was therefore dropped when the plugin restarted,
and since fsString is rebuilt from those options the volume was remounted
at the root of the remote instead of at its subpath.
Before this change a volume created with type + path lost its path
completely, and one created with remote + path silently fell back to the
path of the connection string. With a backend whose credentials are
scoped to the subpath the restored mount then failed every operation
rather than serving the wrong directory.
Feed the persisted path back like fs and type, so applyOptions applies
the same precedence on restore that it applies when the volume is
first created.
Fixes#9853
The code which opens the --combined, --differ etc report files (or
stdout for "-") and closes them afterwards was duplicated between the
check command and the sync logger flags. This moves it into
operations.OpenReportFiles which both now use. It also closes any
files already opened if a later one fails to open.
The deferred cleanup in the constructor checked a local error variable
rather than the error being returned, so failures after the VFS was
created (such as an invalid --passive-port) never shut it down. Name
the error return so the cleanup sees the returned error.
When the HTTP server failed to initialise, for example because the
listen address was already in use, rclone panicked with a nil pointer
dereference instead of reporting the error.
The deferred cleanup in the constructor read the provider from the
named return value, but `return nil, err` sets that to nil before the
deferred function runs. Use a local variable for the server instead.
Writing to the mount with os.WriteFile made the Go runtime register
the file with its poller so the kernel then polled the file from
epoll_ctl and epoll_wait, sending POLL requests to the FUSE server
running in this same process. A thread waiting inside epoll cannot be
preempted by the runtime, so a garbage collection starting while such
a POLL was outstanding stopped the world for good - the test binary
could not be killed even with SIGKILL and the mount was left behind,
wedging anything that touched it.
Now we write through the mount with a descriptor straight from
open(2), which os.NewFile keeps out of the poller, check that it
really is out of the poller with SetDeadline, and check at the end of
the test that the mountpoint is unmounted.
In this commit we fixed the same problem for mount by running in a
subprocess however changing one write file routine here was much
easier than re-arranging the tests.
4a382c09ec mount: run tests in a subprocess to fix deadlock - #3259
Note that go-fuse (and hence mount2) works around this problem it by
forcing an early POLL it can answer with ENOSYS.
See: https://github.com/golang/go/issues/21014
Move the archive entry name validation added for CVE-2026-59732 from
cmd/archive/extract into a new lib/sanitize package as sanitize.Path,
so the same check can be shared with the archive backend which mounts
archives as a filesystem.
sanitize.Path keeps the extract semantics - reject any name with a
".." path component, treating both "/" and "\" as separators - and
additionally cleans the name with path.Clean. This corrects two edge
cases in extract: a repeated "./" prefix ("././file.txt") is now fully
stripped rather than only the first, and a bare "." entry is now
treated as the archive root and skipped.
Add sanitize.Leaf, which rejects a name that is empty, ".", ".." or
contains a "/", for checking a single directory entry name read from
an archive.
The names handled are rclone remote paths, in which "/" is the only
separator and "\" an ordinary character, so Leaf does not reject a
backslash: making a name safe for its storage is the destination
backend's job (the local backend encodes "\" on Windows and refuses
paths which escape its root). Path's rejection of ".." between
backslashes is kept as defence in depth for extract.
An empty or "." volume name joined onto the base directory resolves to the
base directory itself. newVolume does not call validate, so such a name
would mount a remote over the base directory and shadow every other
volume's mountpoint.
Require the resolved mountpoint to be a strict descendant of the base
directory so these degenerate names are refused.
When the plugin restarts it reads its persisted state file and used the
stored mountpoint verbatim. A state file written by an older rclone that
allowed escaping volume names, or one that was tampered with, could point
the mountpoint outside the base directory, so upgrading did not remediate
an already-escaped volume.
Re-derive the mountpoint from the base directory and the volume name on
restore, confined to the base directory, rather than trusting the stored
path.
A Docker VolumeDriver.Create request carries a raw volume name that was
joined onto the base directory with filepath.Join and used verbatim as the
mountpoint. filepath.Join collapses ".." components, so a crafted name such
as "../../../etc/foo" resolved to a host path outside the base directory,
where the plugin then created a directory and mounted the remote.
Confine the mountpoint to the base directory and refuse any name that
resolves outside it.
When serving FTP with --auth-proxy, the obscured password was cached in a
driver-global map keyed only by the username. Two sessions that logged in
with the same username but different credentials shared one map entry, so a
later login overwrote it and every subsequent operation on the earlier,
still-authenticated session was re-authorized with the later session's
credential and executed against the later session's backend.
Bind the credential to the FTP session by storing the obscured password in
the per-session goftp Session.Data map instead, so each session always
resolves the backend it authenticated for.
With --auth-proxy set and --auth-key unset, serve s3 registered every client
supplied access key ID with an empty secret and verified the SigV4 signature
against that, so anyone could sign a request for an arbitrary access key ID with
an empty secret and be let in. The proxy program was only ever given the access
key ID (as both user and pass) so it had nothing with which to authenticate the
client either.
An S3 client never sends its secret, only a signature made with it, so the
server has to know the secret to check the request. The auth proxy protocol as
been changed to handle this. For serve s3 the proxy program is given just the
access key ID as the user (no pass or public_key) and must return the matching
secret as _secret_access_key in its output. rclone verifies the request's
signature against that secret, refusing the request if the proxy rejects the
access key ID, doesn't return a secret or returns an empty one, or the signature
doesn't match. The secret is only used for this server's own verification and is
never registered with gofakes3, so other serve s3 instances in the same process
don't honour it.
The proxy's answers are cached. If a signature fails against a cached secret the
proxy is consulted again so a rotated secret takes effect immediately - but only
for a signature mismatch, and at most once every 10 seconds per access key ID
and client IP, so a stream of bad signatures can't make the proxy program run
for every request. A rotation never shuts down the cached backend under requests
still using it. A cached answer is checked with the proxy again once it is 5
minutes old even if in constant use, so revoking an access key ID takes effect
within 5 minutes.
This means --auth-key is no longer needed with --auth-proxy: it is ignored and a
warning is given at startup if both are set. The proxy is the source of truth
for both the credentials and the backend they map to. Presigned URLs (credential
in the query string) are now recognised by the proxy middleware too. The auth
proxy docs are added to serve s3.
Note that the serve s3 auth proxy protocol has changed. The proxy program is now
given the access key ID as "user" (it was previously given an MD5 hash of it,
with the access key ID as "pass") and must return the matching secret as
"_secret_access_key".
This needs gofakes3 v0.0.9 for signature.V4SignVerifyWithSecret.
gofakes3 kept the keys given with --auth-key in a store global to the process,
so when more than one serve s3 was running in one rclone (eg started via the rc)
each accepted the others' credentials and a client with the key for one server
could read and write the backend of another.
This updates gofakes3 to v0.0.9 which keeps auth keys per instance and adds a
test that two servers only accept their own keys.
From v1.70.0, an SFTP server started through the rc serve/start API with
a per-server proxyOpt.AuthProxy decided whether to enable proxy
authentication by checking the process-global proxy.Opt.AuthProxy
instead of the supplied proxyOpt.AuthProxy. In the normal rc case the
global is empty, so the auth proxy was silently ignored: the server
either failed to start with "no authorization found" or authenticated
against the local authorized_keys file instead of routing each login
through the proxy the operator configured.
The serve Provider refactor (f425f8d46) fixed the constructor by building the
provider from the supplied proxyOpt, but the authorized-keys handling in
configure() still consulted the global option. Make it depend on whether
proxy mode is actually active, and add a regression test for the
per-server option.
From v1.70.0 until the serve Provider refactor (f425f8d46), an S3 server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server served the fixed
filesystem supplied to serve/start rather than routing each access key
to the backend chosen by the proxy, bypassing the operator's intended
per-key authorization.
The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again, and only
logs "allowing anonymous access" when neither an auth key nor an auth
proxy is configured so the log reflects the effective mode.
From v1.70.0 until the serve Provider refactor (f425f8d46), an FTP server started
through the rc serve/start API with a per-server proxyOpt.AuthProxy
decided whether to enable proxy authentication by checking the
process-global proxy.Opt.AuthProxy instead of the supplied
proxyOpt.AuthProxy. In the normal rc case the global is empty, so the
auth proxy was silently ignored and the server fell back to its
fixed-backend mode, whose default account accepts user "anonymous" with
any password - a complete authentication bypass.
The Provider refactor fixed this incidentally by building the provider
from the proxyOpt passed to the constructor. This adds a regression test
so the per-server option cannot silently stop working again.
The multipart reorder-buffer admission trusted the client-declared part length.
A negative length was accepted, and `buffered + size` could overflow int64 for
a huge declared length, wrapping the running total negative and admitting
further parts past --multipart-streaming-buffer-limit.
Reject a negative length and use the overflow-safe comparison `size <=
bufferLimit - buffered` so an untrusted Content-Length can neither poison nor
overflow the budget.
Streamed multipart UploadPart called Reserve(contentLength) before reading any
body bytes, so the pool immediately allocated one 1 MiB page per MiB of the
client-declared Content-Length (or X-Amz-Decoded-Content-Length). An client
could declare a huge part size, send no body, and force an arbitrarily large
allocation without paying the bandwidth cost of the declared body.
Drop the Reserve so the pool-backed buffer grows a page at a time as the body
is actually read: memory now tracks the bytes received, not the unverified
header.
When bisync is interrupted with a graceful shutdown it keeps the files
which transferred successfully in its listings and rolls the rest back.
An operator precedence mistake in that check meant a transfer of an
empty file (or one of unknown size) was kept even when it had failed,
so bisync recorded it as synced when it had not been.
kaii9