The newer revive flags an exported function returning an unexported
type and a redundant type in a var declaration.
(cherry picked from commit 881cedd348)
Moving a directory can take longer than the API gateway allows, so the
request fails with a 520 or 502 error even though the move completes.
The retry then failed with "Folder ... was already moved to that
location (status 409)".
Treat that error as success.
(cherry picked from commit 0c52b183d0)
The Internxt API serves listings from read replicas which lag behind
writes, so for a short while after files or directories are moved or
deleted they can still be listed in their old location.
This caused removing a directory which had just been emptied to fail
with "directory not empty" (eg when moving a directory without server
side directory moves or purging a directory) and a directory which had
just been moved to be found in its old location.
Remember the directories this process has moved or deleted and ignore
directory and file entries which contradict that when listing, finding
directories and checking a directory is empty before removing it.
(cherry picked from commit 7ff5da8517)
Moving a file into a directory which had just been created failed with
"Not Found (status 404)", and moving a file over one which had just been
deleted (as sync does with --backup-dir and --suffix) failed with "A file
with the same name already exists in destination folder (status 409)".
The API checks moves against read replicas which lag behind writes, so
retry these errors until the replicas catch up.
(cherry picked from commit ee26daecd8)
The Internxt API serves lookups and listings from read replicas which
lag behind writes, so for a short while after a file is moved it can
still be returned from its old location.
When sync moved a file into the backup location and then uploaded its
replacement, the upload could find the moved file under its old name
and overwrite it. Overwriting renames the existing file by UUID and
deletes it once the upload succeeds, so the file that had just been
moved into the backup location was deleted.
Remember the files this process has moved or deleted for a minute and
ignore lookups and listing entries which contradict that.
(cherry picked from commit f068abd607)
With hard_delete set, deleted files and directories carried on being
listed for about a second afterwards because the Drime server doesn't
invalidate its cache of the parent folder listing when entries are
deleted forever. This made removing a directory straight after
emptying it fail with "directory not empty".
Moving an entry to the trash does invalidate the cache, so with
hard_delete set rclone now moves the entry to the trash first and then
deletes it forever.
(cherry picked from commit a2e2b73725)
When server-side copying to a destination which already existed, the
Drime server gave the copy the name "name (1)" and rclone only renamed
it if the source and destination leaf names differed. The existing file
was then deleted leaving the copy under the wrong name.
The server refuses to rename an entry to a name which is already in
use, so this removes the existing file straight after the copy, then
renames the copy whenever its name differs from the destination name.
(cherry picked from commit eb10c48a17)
The premiumize.me upload server has started truncating the multipart
file name at the first ";", so uploading "a;b.txt" created a file
called "a" and the upload then failed with "object not found".
Directory creation and renames still accept ";", so files whose names
contain ";" are now uploaded under a temporary name and renamed into
place. The encoding is left unchanged so existing files and
directories containing ";" remain accessible.
(cherry picked from commit 7c18e1eb86)
When S3 returns URL-encoded keys from ListObjectVersions, URL encoding
can change their lexical order. This could make mergeDeleteMarkers
place a delete marker after older versions of the same key, so
--s3-version-at reported deleted objects as live.
Compare decoded keys while merging, while preserving the encoded keys
for the existing listing decode path.
Fixes#9948
(cherry picked from commit cfb90e3ebe)
Testing against OneDrive personal (free) and OneDrive for Business shows
- personal accounts now create versions on setting the modification
time and can delete them, so --onedrive-no-versions and rclone
cleanup work there
- --onedrive-link-password works on OneDrive for Business
- personal free accounts can't set a link password or --expire
- --onedrive-link-type embed only works on OneDrive personal
- personal accounts store times with 1s precision, not mS
See #9917
(cherry picked from commit ff02636fe4)
findItem read the response status code when the lookup failed, so a
failure with no HTTP response panicked.
Thanks to @manus-pi for finding this problem.
(cherry picked from commit 1e92520076)
DirMove discarded the error from the destination check and read the
response status code, so a failure with no HTTP response panicked.
Other errors were reported as the destination existing; they are now
returned.
Thanks to @manus-pi for finding this problem.
(cherry picked from commit 35abcadfc7)
Rmdir and Purge read the response status code before checking for an
error, so a failed DeleteFolder call with no HTTP response (a network
error, or purging the root which fails validation) panicked.
Thanks to @manus-pi for finding this problem.
(cherry picked from commit e2cd9a5dfb)
The default for --smb-user was the name of the user running rclone
config, and rclone does not write defaults to the config file, so
entering your own user name left it out. A remote made that way then
logged in as whoever ran it later, e.g. root under systemd.
Make the default blank and look up the current user when the remote is
used, as the ftp backend does. Existing configs work as before.
(cherry picked from commit 3c51b96774)
The box backend logged an int64 with %q which printed
%!q(int64=123) instead of the sequence ID.
serve s3 passed the message from gofakes3 as the format string, so
any % in it was interpreted as a formatting directive and the message
was mangled.
(cherry picked from commit 1c3e432c3f)
Gzip metadata is read from the wrapped remote and could contain an
invalid block size or incomplete block index. A range read could then
panic in the seekable gzip reader.
Validate the gzip sidecar invariants before constructing a reader so
malformed remote metadata returns an error instead of crashing rclone.
(cherry picked from commit c8d60a67fc)
OCI rejects a multipart completion request with no parts. Probe unknown-size
streams through a buffered reader and use a regular upload when the stream is
empty. Peek preserves a non-empty stream for the selected upload path.
(cherry picked from commit e85836d4c7)
Internxt stores a file as a (plainName, type) pair and never derives the
split itself, so it is a convention shared between clients. This backend
split at the final dot, storing and looking up ".bashrc" as an empty name
of type "bashrc", where the web, desktop, Linux and macOS clients all keep
the leading dot in plainName. List rebuilt the full name so such files
appeared, but NewObject looked them up by the split and missed them.
(cherry picked from commit 77ec281072)
- Replaced the preUploadCheck function with findFile for better clarity and efficiency in checking file existence.
- Introduced splitNameExt to encapsulate name and extension parsing logic.
- Updated NewObject and Update methods to utilize findFile for improved file metadata handling.
- Enhanced error handling and reduced redundant code in file checks.
(cherry picked from commit e441773d24)
Ceph RGW (and Linode Object Storage, which is Ceph-backed) can break
SigV4 when Accept-Encoding is included in the signature, especially
when a reverse proxy rewrites that header. GCS already sets this quirk;
apply the same default for Ceph and Linode as suggested in #8206.
(cherry picked from commit ea589de941)
Set the OCI source SSE-C request headers when using a customer key.
Server-side copies need these headers to decrypt the source object, in
addition to the existing headers that encrypt the destination.
(cherry picked from commit 3eee2c0dd2)
The Dropbox backend advertises CaseInsensitive: true, but the two
shared-mode lookup helpers compared names with an exact, case-sensitive
==, so a shared folder or received file named "Project" could not be
found when requested as "project". Use strings.EqualFold in both
findSharedFolder and findSharedFile to honour the advertised
case-insensitivity.
Fixes#9706
(cherry picked from commit b549554c31)
In shared_folders mode NewFs derived the shared folder name with
path.Dir(f.root), which returns the parent path rather than the first
path component. For a root like "SharedFolder/subdir/deeper" this yielded
"SharedFolder/subdir", which findSharedFolder cannot match, so NewFs
failed with ErrorDirNotFound. Use the first path component of the root,
as the shared_folders option documents, so deeply nested roots mount.
Fixes#9705
(cherry picked from commit ac7cfcc848)
IsErrNoSpace compared against syscall.ENOSPC. Go defines that constant on
Windows as a value in its application reserved range which no Windows API
returns, so the comparison could never be true there. A full disk on Windows
reports ERROR_DISK_FULL or ERROR_HANDLE_DISK_FULL instead.
Preallocation failures were still caught, because those return a separate
sentinel, but a disk that is already full fails at the directory creation or
at the open long before preallocation is reached. That is the case reported.
The errors are now held in a list which platform specific files add to in
their init, which is the shape retriable_errors already uses in this package,
and the comparison itself is unchanged. Windows appends the two codes that
lib/file already recognises when preallocation fails. Every other platform
keeps exactly the behaviour it had.
This also reaches the VFS cache, which uses the same helper and has no
preallocation path of its own, so its out of space handling has been inert
on Windows.
(cherry picked from commit ca41db095b)
The check that an entry returned by an archiver is a direct child of
the directory being listed normalised a parent of "/" to the root, so
an entry named "/x" passed as a child of the root while "x/" and
"dir//x" were rejected.
Decide by stripping the directory prefix and checking what is left
with sanitize.Leaf, which rejects an empty name, ".", ".." and any
name containing a "/". This also covers the leading slash case.
(cherry picked from commit b88e237e8c)
A file entry in a zip whose name refers to a directory, such as ".",
"/", "" or "sub/.", was only skipped when it named the root of an
archive which was itself the root of the remote. When the archive was
found by listing its parent directory the entry appeared as a file
with the same name as the archive alongside the directory for it, and
copying the archive tried to write both. When the entry named a
subdirectory it appeared as a file alongside that directory, and with
that subdirectory mounted as the archive root the entry was taken to
be the single file the root points at, hiding every real entry.
Skip any file entry whose last path component is "", "." or "..",
checked on the raw name before it is cleaned or joined on the prefix.
(cherry picked from commit da352a2a1b)
The zip archiver handed out its cached directory tree directly. Any
caller which filters a listing in place (as the core listing code
does) altered the cache, so later listings of the same directory could
be corrupted.
Return a copy of the cached listing instead.
(cherry picked from commit 68eab60564)
When an upload failed with a 500 error the upload was retried with a
new upload link but the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by returning a RetryError instead so the caller retries
the upload with a fresh stream, which will fetch a new upload link.
(cherry picked from commit 6cdd0ea761)
When an upload failed with a retryable error the pacer retried the
whole upload with the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
It also makes 5xx errors from the upload storage servers retryable.
These come back from the SDK as a different error type to API errors
so were not being retried at all.
(cherry picked from commit fa43f10af2)
When an upload failed with a retryable error the pacer retried the
whole PUT with the same input stream. The stream had already been
consumed by the first attempt so the retry uploaded an empty file.
This fixes it by using CallNoRetry for the upload, as the other
backends do, so retryable errors are returned wrapped in a RetryError
for the caller to retry the upload with a fresh stream.
(cherry picked from commit f4cd80a535)
The headers set with --http-headers are documented for passing
credentials such as Authorization or Cookie. The backend used the
default net/http redirect policy which copies all but a handful of
well known headers to any redirect target, so a redirect from the
configured server to another host would send those credentials to
that host, and a redirect from https to http would send them in
plaintext.
When headers are configured this installs a CheckRedirect function
which:
- removes the configured headers from every hop once the redirect
chain has left the originally requested host
- refuses a redirect from https to http with an error
(cherry picked from commit 0adc0082dff7ef6ed1597418f85ac691478a6fa5)
Whether an archive entry name can escape the archive's namespace was
left entirely to each archiver. Enforce it in the archive backend too.
List only passes on direct children of the directory listed and
NewObject only returns the object asked for, so a future archiver
which forgets to validate names cannot expose a traversal to fs/sync
and fs/operations.
(cherry picked from commit a6a7d95e081b91231b49c4004e8a2bff16da4cd8)
The path inside the archive was compared against the cleaned entry
names without being cleaned itself, so `archive.zip/sub/./dir` or
`archive.zip/sub//dir` failed to list even though `archive.zip/sub/dir`
worked.
(cherry picked from commit 34636f34079585f3a8f883038483ece58d39b084)
A zip containing a file entry whose name refers to the archive's own
root (".", "/" or "") was presented as a single file called "." and
all its other entries disappeared. A file at the root can only be the
archive member the backend was pointed at, so with no root such an
entry is skipped like any other unsafe name.
(cherry picked from commit c9fac578aae7707faf340b58cf45465a5f698cbd)
Entry names read from a squashfs directory are not sanitized by
go-diskfs. The squashfs backend joined each leaf name onto its
directory to form the object's remote, so a crafted image could escape
its directory.
Use sanitize.Leaf to skip unsafe entries in List. A "\" is an
ordinary character in a file name on the systems squashfs images are
made on and in an rclone remote path, so it is deliberately not
rejected; making it safe for the destination is the destination
backend's job.
Skipped entries are logged at DEBUG with a single NOTICE count per
listing so a crafted image under a mount cannot flood the log.
(cherry picked from commit 63385c66175141b63bb53b3e42b50908649f8437)
When a zip archive was mounted at a subdirectory root, readZip used a bare
strings.HasPrefix to decide which entries fell inside the root. This
matched on a raw string prefix rather than a path boundary, so mounting
root "foo" also exposed sibling entries such as "foobar/..." with their
names left uncorrected.
Require a path boundary when filtering by root.
(cherry picked from commit b444227264cee2a9307f03ba90c2a411e7eeb693)
The zip backend mounts a zip file as a browsable Fs. Go's archive/zip
does not sanitize entry names, and readZip applied path.Clean but did
not reject a cleaned name that still pointed outside the archive. A
crafted zip could make rclone copy/sync attempt writes outside the
intended destination.
Sanitize entry names with sanitize.Path - the same check used by
rclone archive extract - skipping any entry with a ".." path
component, whether separated by "/" or "\". A backslash is otherwise
kept as an ordinary character in the name, as archive extract does. It
is up to the destination backend to make names safe for its storage.
Skipped entries are logged as a single count per archive so a crafted
archive with many escaping entries cannot flood the log.
(cherry picked from commit 224fe97ac13105094651264a9e16ee3cf41ccf77)
With --links/-l, a symlink is served as a .rclonelink object whose
content is the target path. A Range request with a start offset beyond
the target length (e.g. "Range: bytes=99999999999-") reached
openTranslatedLink and sliced the target string at that offset, panicking
with "slice bounds out of range".
Clamp the offset to the target length so an out-of-range start reads
empty, matching how a real file read past EOF behaves.
(cherry picked from commit 3fa32192c20f0b4cfd4f4b07636dc3445026041a)
The birth-time (btime) write in writeMetadataToFile followed symlinks for
any object that was not a translated link, so under -l/--links a symlink
planted by an untrusted source at the destination path could redirect the
btime write to a target outside the backup destination on OSes where
birth time is settable (Windows).
Use the NOFOLLOW birth-time write whenever translating symlinks, not only
for translated links. It is a no-op on a real file or directory and stops
a planted symlink from being followed out of the destination.
(cherry picked from commit bd86336faac7cfc4e9057add38ec5fd12d762d02)
With -l/--links the local backend faithfully recreates a source ".rclonelink" as
a real symlink at the destination. Directory metadata (chmod/chown/chtimes),
however, was applied with the raw following syscalls
os.Chmod/os.Chown/os.Chtimes rather than through the os.Root sandbox used for
content writes. A Directory is never a translatedLink, so when the destination
path already existed as a symlink planted by an untrusted source, the metadata
was applied through it to a target outside the backup destination.
Route directory metadata through os.Root when translating symlinks, so a planted
symlink can no longer redirect chmod/chown/chtimes out of the destination, while
legitimate in-tree directories are unaffected.
(cherry picked from commit b764ccbd23582f29c611862f11f86dde3544035a)
Each upload chunk is buffered in a pool.RW from the global memory pool
but was never closed, so its pages were never returned to the pool.
Close the buffer after each chunk is uploaded and on the read error
path.
A chunk that failed with a retryable error was also retried without
rewinding the buffer, so the retry sent an empty body with the original
Content-Length and Content-Range and failed.
Seek the chunk back to the start inside the pacer closure so each
attempt re-sends it in full.
The FsPutRetry integration test covers the retry of a failed upload
request and checks the buffers are returned to the pool.
(cherry picked from commit 2f0228029e)
When both /me/drives and /me/drive fail during config (for example an
account-level 403 serviceReadOnly "Database Is Read Only"), send the
config state machine to the existing manual drive ID entry state
instead of dead-ending at choose_type with the raw error. The drive
itself remains usable when only the enumeration API is blocked.
Fixes#9794
(cherry picked from commit 03fe2ef794)
Directory names which look like they have a --b2-versions version
string are now encrypted in full, so directories created by older
rclone (which left the version string in plain text) no longer
decrypt and vanished silently from listings.
DecryptDirName now falls back to the old form for such names so the
directory is listed, and logs the name it needs to be renamed to on
the underlying remote to make it accessible again. Document this in
the crypt docs.
(cherry picked from commit 1583cce1e2)
The refresh endpoint returns a rotated token with a fresh expiry on
every successful call, but getUserInfo discarded it, so routine use
never extended the stored token's life. Once the stored token aged
out, accounts with 2FA enabled could not recover non-interactively
and required a manual reconnect.
Carry the rotated token out of getUserInfo and persist it in NewFs
via the same jwtToOAuth2Token + oauthutil.PutToken path that
refreshJWTToken uses, keeping f.cfg.Token in sync (same pattern as
refreshOrReLogin).
Fixes#9584
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 66761670da)
The --b2-versions support added in 3fe2aaf96 strips a version string
from the last segment of a path before encrypting it, so that the
plain text version suffixes which the underlying backend appends to
encrypted file leaf names can be handled. EncryptDirName and
DecryptDirName share that code, so the last segment of a *directory*
name was version stripped too. Only file leaf names are ever given a
version string by the backend - a directory gets a
version-string-like name from the user, and such a name is encrypted
verbatim when it appears as the parent of a file name, so the same
directory ended up with two different encryptions.
Before this change, with a directory whose name matches rclone's
version format, eg dir-v2001-02-03-040506-123:
rclone copy file.txt crypt:dir-v2001-02-03-040506-123/
rclone ls crypt:dir-v2001-02-03-040506-123
# => "directory not found" - the file is invisible to listings
rclone mkdir crypt:dir-v2001-02-03-040506-123
# => creates a second directory with the same decrypted name
After this change EncryptDirName and DecryptDirName encrypt directory
names verbatim, so a directory encrypts the same way whether it is
named on its own or as the parent of a file. Version strings are only
added to file names by the underlying backend, so --b2-versions is
unaffected and the existing version tests are untouched.
A directory which was created by the old EncryptDirName will no longer
decrypt and will be reported as undecryptable in listings. Such
directories were already unusable - anything copied into one was
written to a different encrypted directory - so nothing which worked
before is broken by this.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 67b184d6e7)
With no_head_object set, NewObject does not read any metadata, so the
destination object returned from a server side copy had a size of 0.
The size check in operations.Copy then failed with "corrupted on
transfer: sizes differ N vs 0" and deleted the newly copied object.
This also broke Move and hence renames through rclone mount.
Populate the destination object's size and MD5 from the source object
when no_head_object is set, as a server side copy produces an object
with identical content.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
(cherry picked from commit 6df7b8aba1)
Dropbox is case insensitive and the path_display it returns in
change notifications may not match the case of the configured root.
Before this change the root was trimmed with a case sensitive prefix
match, so when the cases differed the full path was passed to the
ChangeNotify callback and the notification was ignored.
This trims the root case insensitively while preserving the display
case of the remaining path.
(cherry picked from commit 4af64270cc)
Nick Craig-Wood