The Docker image had no /etc/mime.types, so MIME types came only from
Go's built-in table plus rclone's small extra list, and many
extensions uploaded as application/octet-stream.
This installs Alpine's mailcap package to fix the problem.
(cherry picked from commit 3cdf855547)
To find the latest patch release of a minor version, for example
`rclone selfupdate --version 1.75`, selfupdate searched the listing of
downloads.rclone.org for href="./vX.Y.Z/". The leading ./ is a detail
of how Caddy's file server writes its links. A listing which linked to
the same directories as "vX.Y.Z/", which is an equally valid relative
URL, made selfupdate fail with "could not find the minor release".
This makes the ./ optional in the pattern, so selfupdate no longer
depends on which program wrote the listing, and takes the version from
a capture group rather than from fixed offsets into the match.
The listings written by rclone index now include the ./ for the
benefit of rclone versions without this fix, so this is to remove the
dependency for the future.
(cherry picked from commit 8cd43746a3)
The temporary directory contains a drive letter colon so it needs
quoting in the crypt connection string, otherwise the password
parameter is dropped.
(cherry picked from commit 9dc8b71ae9)
The newer revive flags an exported function returning an unexported
type and a redundant type in a var declaration.
(cherry picked from commit 881cedd348)
MinIO no longer publishes images on quay.io (nor Docker Hub or ghcr.io)
so pulling quay.io/minio/minio returns "unauthorized". pgsty/minio is a
maintained community fork with the same entrypoint layout.
(cherry picked from commit 220c65f81d)
ENETUNREACH and ENETDOWN were not in the list of retriable errors on
non-Windows platforms, so a single failed connection aborted the
transfer instead of being retried as a low level retry. The Windows
list already includes the equivalent WSAENETUNREACH and WSAENETDOWN.
This is easy to hit on a host with IPv6 enabled but no IPv6 route: if
the A lookup fails transiently while the AAAA lookup succeeds, the only
address to dial is IPv6 and the connect fails with ENETUNREACH. A retry
resolves again and normally succeeds.
(cherry picked from commit 94e319d2fa)
BwPair.String printed a sub-KiB bandwidth as a bare number, and Set
reads a bare number as KiB, so a rate under 1 KiB grew by a factor of
1024 whenever it went through a string. rc core/bwlimit reports the
current rate in that format and accepts it back, so reading the limit
and setting it again raised it.
63c4fef27 fixed the same corruption for config values by suffixing bare
numbers with B inside Option.String. Move that into a SizeSuffix method
and use it from BwPair.String as well.
(cherry picked from commit 60bdc6d454)
Use mime.FormatMediaType instead of unescaped string concatenation so a
directory name containing a double quote cannot inject extra disposition
parameters.
Fixes#9962
(cherry picked from commit 14a1359c96)
Use mime.FormatMediaType instead of unescaped string concatenation so a
directory name containing a double quote cannot inject extra disposition
parameters.
Fixes#9962
(cherry picked from commit 661484f36c)
Copying an object onto itself, as clients do to replace its metadata,
reported success when the object didn't exist, and copying a missing
object to a different key failed with an internal error. Both now fail
with NoSuchKey as S3 does.
(cherry picked from commit ce5351c52e)
With --etag-hash auto the hash for the ETags was chosen once from the
remote serve s3 was started with. With --auth-proxy there is no such
remote so serve s3 crashed on startup, and when started by the rc it
was the wrong remote, so users whose backends lacked that hash got no
ETags.
The hash is now chosen from the backend of the user making each
request.
(cherry picked from commit 746eac73ef)
The metadata of every object uploaded was kept in memory for as long as
the server ran, even after the object was deleted, so it grew without
limit and the metadata of a deleted object reappeared on any object
later created at the same key other than through serve s3.
Deleting an object now forgets its metadata.
(cherry picked from commit c350f8ebbf)
Listing a prefix with no delimiter walked the entire subtree below it into
memory and only then sliced out the requested page, so every page of a
listing cost a full traversal of the tree.
Walk the tree lazily instead, stopping as soon as the page is full and
skipping the subtrees an earlier page already returned. A page now costs a
number of directory reads proportional to the keys it returns rather than to
the size of the subtree, and ETags are computed only for the objects that
are actually returned.
Entries are emitted in the order their keys have in a flat keyspace, with a
directory sorting as if it carried its trailing slash, so that "a.txt" comes
before "a/b" as it does in a real S3 bucket. Without this a resumed listing
would silently skip keys across a page boundary.
(cherry picked from commit 04697cc02a)
When the last user of a VFS shut it down at the same time as a new
user of the same remote asked for one, the new user could be given the
VFS being shut down, with its cache and background tasks stopped.
Only a VFS which is still in use is now reused, otherwise a new one is
made.
(cherry picked from commit c62aa2adc9)
A VFS is shut down when the last of the references to it from New is
given back with Shutdown. Hold takes another reference, for code using
a VFS it didn't create itself, which may otherwise be shut down under
it.
(cherry picked from commit 1dab0f3abd)
When a backend rejected the multipart upload as too small, the subtest
skipped without removing its local source file, so the following
upload subtests failed their local listing checks.
(cherry picked from commit ee9c228012)
Moving a directory can take longer than the API gateway allows, so the
request fails with a 520 or 502 error even though the move completes.
The retry then failed with "Folder ... was already moved to that
location (status 409)".
Treat that error as success.
(cherry picked from commit 0c52b183d0)
The Internxt API serves listings from read replicas which lag behind
writes, so for a short while after files or directories are moved or
deleted they can still be listed in their old location.
This caused removing a directory which had just been emptied to fail
with "directory not empty" (eg when moving a directory without server
side directory moves or purging a directory) and a directory which had
just been moved to be found in its old location.
Remember the directories this process has moved or deleted and ignore
directory and file entries which contradict that when listing, finding
directories and checking a directory is empty before removing it.
(cherry picked from commit 7ff5da8517)
Moving a file into a directory which had just been created failed with
"Not Found (status 404)", and moving a file over one which had just been
deleted (as sync does with --backup-dir and --suffix) failed with "A file
with the same name already exists in destination folder (status 409)".
The API checks moves against read replicas which lag behind writes, so
retry these errors until the replicas catch up.
(cherry picked from commit ee26daecd8)
The Internxt API serves lookups and listings from read replicas which
lag behind writes, so for a short while after a file is moved it can
still be returned from its old location.
When sync moved a file into the backup location and then uploaded its
replacement, the upload could find the moved file under its old name
and overwrite it. Overwriting renames the existing file by UUID and
deletes it once the upload succeeds, so the file that had just been
moved into the backup location was deleted.
Remember the files this process has moved or deleted for a minute and
ignore lookups and listing entries which contradict that.
(cherry picked from commit f068abd607)
With hard_delete set, deleted files and directories carried on being
listed for about a second afterwards because the Drime server doesn't
invalidate its cache of the parent folder listing when entries are
deleted forever. This made removing a directory straight after
emptying it fail with "directory not empty".
Moving an entry to the trash does invalidate the cache, so with
hard_delete set rclone now moves the entry to the trash first and then
deletes it forever.
(cherry picked from commit a2e2b73725)
When server-side copying to a destination which already existed, the
Drime server gave the copy the name "name (1)" and rclone only renamed
it if the source and destination leaf names differed. The existing file
was then deleted leaving the copy under the wrong name.
The server refuses to rename an entry to a name which is already in
use, so this removes the existing file straight after the copy, then
renames the copy whenever its name differs from the destination name.
(cherry picked from commit eb10c48a17)
The premiumize.me upload server has started truncating the multipart
file name at the first ";", so uploading "a;b.txt" created a file
called "a" and the upload then failed with "object not found".
Directory creation and renames still accept ";", so files whose names
contain ";" are now uploaded under a temporary name and renamed into
place. The encoding is left unchanged so existing files and
directories containing ";" remain accessible.
(cherry picked from commit 7c18e1eb86)
Shortcuts to shared items (and Personal Vault) are stored as remote
items pointing at another drive and listing them can fail with
"The provided drive id appears to be malformed". This aborts a
--fast-list listing of the whole drive.
(cherry picked from commit 8c38d3068a)
rest.ReadBody read the whole response body into memory with no limit.
It is used by the default error handler and by many backends' error
handlers and small API calls, so a server which answered with an error
status and then streamed an endless body could make rclone allocate
memory until it was killed.
ReadBody now reads at most 10 MiB (the same limit drainAndClose
already uses to discard unread bodies) and returns an error if the
body is bigger than that. Every caller reads small API responses -
error bodies, status documents and upload tokens - so no legitimate
response is affected.
Reported by @manus-pi
(cherry picked from commit 90e67915c8)
When S3 returns URL-encoded keys from ListObjectVersions, URL encoding
can change their lexical order. This could make mergeDeleteMarkers
place a delete marker after older versions of the same key, so
--s3-version-at reported deleted objects as live.
Compare decoded keys while merging, while preserving the encoded keys
for the existing listing decode path.
Fixes#9948
(cherry picked from commit cfb90e3ebe)
Testing against OneDrive personal (free) and OneDrive for Business shows
- personal accounts now create versions on setting the modification
time and can delete them, so --onedrive-no-versions and rclone
cleanup work there
- --onedrive-link-password works on OneDrive for Business
- personal free accounts can't set a link password or --expire
- --onedrive-link-type embed only works on OneDrive personal
- personal accounts store times with 1s precision, not mS
See #9917
(cherry picked from commit ff02636fe4)
With --vfs-metadata-extension set, looking up the metadata file of a
file which was open for write and not yet uploaded caused a nil pointer
panic. Such a file has no object to read the modification time from.
Use the modification time of the VFS node instead, which is valid
whether or not the file has been uploaded.
(cherry picked from commit 7a2d7c766d)
TestDeleteFatalError set --max-delete before writing its files, so on
chunker, which deletes while uploading, the setup failed.
TestDirMoveMoveError and TestDirMoveContext test the core DirMove logic
with a wrapping Fs. This fails on remotes without Move (eg s3, memory)
and on those whose objects don't belong to the wrapped Fs (eg archive),
so they now only run on local.
(cherry picked from commit ff958c999f)
findItem read the response status code when the lookup failed, so a
failure with no HTTP response panicked.
Thanks to @manus-pi for finding this problem.
(cherry picked from commit 1e92520076)
DirMove discarded the error from the destination check and read the
response status code, so a failure with no HTTP response panicked.
Other errors were reported as the destination existing; they are now
returned.
Thanks to @manus-pi for finding this problem.
(cherry picked from commit 35abcadfc7)
Rmdir and Purge read the response status code before checking for an
error, so a failed DeleteFolder call with no HTTP response (a network
error, or purging the root which fails validation) panicked.
Thanks to @manus-pi for finding this problem.
(cherry picked from commit e2cd9a5dfb)
The request log printed the escaped URL, so non-ASCII file names showed
up as long runs of %XX escapes. Log the unescaped URL path instead, as
the other serve commands do.
(cherry picked from commit 556940ea44)
VFS.AddVirtual always added a file entry to the directory cache, even
when called with isDir set, so a virtual directory showed up as a file
and nothing could be added inside it.
See #9310
(cherry picked from commit ebb1cc1221)
The default for --smb-user was the name of the user running rclone
config, and rclone does not write defaults to the config file, so
entering your own user name left it out. A remote made that way then
logged in as whoever ran it later, e.g. root under systemd.
Make the default blank and look up the current user when the remote is
used, as the ftp backend does. Existing configs work as before.
(cherry picked from commit 3c51b96774)
When the source matched a file in --copy-dest, copyDest server-side
copied it over an existing destination which differed, so sync, copy
and copyto with --copy-dest could modify a file --immutable should have
protected.
Leave such a destination for the caller, which rejects it with
ErrorImmutableModified as for any other modified file.
(cherry picked from commit dc98c216f2)
This was spotted by CodeQL after this change was merged:
976d05e1d rc: fix rc API accepting an out of range number and overflowing 64 bits
(cherry picked from commit 7723091be9)
The box backend logged an int64 with %q which printed
%!q(int64=123) instead of the sequence ID.
serve s3 passed the message from gofakes3 as the format string, so
any % in it was interpreted as a formatting directive and the message
was mangled.
(cherry picked from commit 1c3e432c3f)
job/status returns the complete output of the job so it is as
sensitive as the call which started the job, e.g. config/dump started
with _async returns the config file including any secrets. It was
possible to read this without authentication where an authenticated
and an unauthenticated rc server share the same process, e.g. rclone
gui --rc.
An unauthenticated client can't start jobs on a server which needs
authentication, other than with the calls which don't need it, so
this shouldn't affect existing users.
(cherry picked from commit 939f82d908)
Handle read the list of extra outputs without holding the mutex, so
calling AddOutput while logging caused a data race.
This reads the extra outputs once under the mutex.
(cherry picked from commit 4cd6359ae9)
TestRcJobList listed the jobs left running in the global job list by
the previous run of the tests, so reset the global job list first.
(cherry picked from commit dd07bb33ba)
The wait to obey a Retry-After error from the server used
time.Sleep() so it carried on sleeping when the context was
cancelled, e.g. by job/stop or --max-duration, and it also slept
after the last try when there was nothing left to retry.
Copying a file also logged the low level retry number counting from
0 rather than 1 as everything else does.
(cherry picked from commit 88bd49fab8)
When the backend has no DirMove method, operations.DirMove moves the
objects one by one. If one of the moves failed the movers stopped
without receiving the rest of the objects, so DirMove blocked forever
once the channel filled up. This could be seen renaming a directory on
a mount of a backend without DirMove, such as s3.
The moves were also done with context.Background() instead of the
caller's context, so they didn't have the caller's config, couldn't
be cancelled and weren't counted in the caller's stats.
(cherry picked from commit b82a024de0)
When a fatal error such as --max-delete being reached stopped the
deletions, the deleters returned without receiving the rest of the
objects, so whatever was sending them blocked forever once the
channel filled up.
For example "rclone delete --max-delete 1" on a directory with more
files than --checkers hung instead of returning the error.
The deleters now keep receiving objects without deleting them after a
fatal error.
(cherry picked from commit caa341878f)
Params.GetInt64 parsed string parameters with strconv.ParseInt(x, 10, 0).
A bitSize of 0 means int, which is 32 bits on 386, arm, mips and mipsle,
so any value outside the int32 range was rejected as out of range even
though it fits in the int64 the method returns.
Parameters reach the rc API as strings on the paths that matter here: the
rc server puts every URL query and form value into Params as a string, and
rclone rc turns each key=value argument into a string. So on a 32 bit build
operations/getfile could not be given an offset, count, head or tail beyond
2 GiB, and debug/set-soft-memory-limit could not be given a limit beyond
2 GiB, while the same commands worked on a 64 bit build.
Parse with a bitSize of 64 to match the declared return type. This is the
string half of the range checking that was added to the float64 branch of
the same function in 976d05e1.
(cherry picked from commit b5f83bdbf2)
Return the first non-EOF input error encountered while Rcat probes whether an
upload is small. Previously that error was ignored and the full probe buffer,
including bytes that were never read, could be passed to Put or PutStream.
(cherry picked from commit b93b73bcb0)
- Reject uploads when object lookup fails for reasons other than "not found".
- Serialize uploads to the same object to prevent races between the
existence check and the write.
(cherry picked from commit d632f8bb51)
Gzip metadata is read from the wrapped remote and could contain an
invalid block size or incomplete block index. A range read could then
panic in the seekable gzip reader.
Validate the gzip sidecar invariants before constructing a reader so
malformed remote metadata returns an error instead of crashing rclone.
(cherry picked from commit c8d60a67fc)
hsdfat