Directory listings showed every directory with a size of -1, so
`rclone lsd` and `operations/list` couldn't say how big a directory
was, and finding out meant walking it with `rclone size`.
Microsoft Graph already returns each folder's `size` as the total size
of its contents, and the backend read it but dropped it. Directories
now carry that size, as they already carry the item count. Listing a
directory's parent is enough to get the size of all its subdirectories,
without listing the subdirectories themselves.
OneDrive updates a folder's size in the background, so for a few
minutes after a change inside it the size may be out of date.
Fixes#7502
When container soft delete is enabled, listing the containers at the
root of the account returned the soft deleted ones too, because the
listing asked for them with Include Deleted. rclone treated them as
normal containers and then failed to list each of them with
"directory not found". With at least one error, sync refused to delete
anything on the destination ("not deleting files as there were IO
errors") and ended with exit code 3.
The listing now asks for live containers only, as it did before the
port to the new SDK in a131da2c3. Nothing else in the backend used the
deleted flag of the containers.
Nextcloud public shares served from public.php/dav/files reject requests
that lack an X-Requested-With header. Send "XMLHttpRequest" for these
URLs when vendor is nextcloud, unless the user already set the header
with --webdav-headers.
The headers set with the `headers` option were sent to any host the
server redirected to, as net/http only strips Authorization and Cookie
on a cross domain redirect and copies every other header onto each
hop. A server which could be made to redirect to another host would
therefore receive any credential carried in a custom header such as
X-Auth-Token.
Strip the configured headers from a redirect hop once the chain has
left the configured host (a different hostname, subdomain or port)
using rest.StripHeadersOnCrossHostRedirectFn. This logs at debug level
when it happens so a 401 from the redirect target can be diagnosed.
The `auth_redirect` option is unchanged as it is an explicit opt-in to
keep credentials across redirects.
The Authorization header set from `user`, `pass` and `bearer_token` is
deliberately left to net/http's policy, which strips it on a change of
domain but keeps it for a subdomain or a different port, as servers
which redirect to a subdomain rely on that today.
The resume start value returned by the NetStorage list API is joined
onto the endpoint URL as a URL reference to build the next listing
request. A value beginning with `//` is a network-path reference, so a
server (or an on-path attacker when `protocol = http`) which returns
`//other.example/path` could make rclone send the next request, signed
with the account secret, to a host other than the configured one.
Use rest.URLJoinRoot so the resume token can only change the path.
The netstorage backend joins the path after `remote:` onto the
configured host URL according to the URL standard. A path beginning
with `//` is a network-path reference under that standard, so a path
such as `//other.example/dir/` replaced the configured host (and port
and userinfo) rather than being treated as a path. Every request,
including the initial stat, was then signed with the account secret and
sent to the substituted host. The signature covers the request path and
action but not the host, so the substituted host received signatures
valid for replay against the real NetStorage endpoint for the duration
of their validity.
Use rest.URLJoinRoot, which refuses any root that changes the scheme,
host or userinfo of the configured URL.
The http backend resolves the path after `remote:` against the
configured url according to the URL standard. A path beginning with
`//` is a network-path reference under that standard, so a path such as
`//other.example/dir/` replaced the configured host (and port and
userinfo) rather than being treated as a path. The configured headers
were then sent to the substituted host on the initial HEAD request and
every request after it.
Use rest.URLJoinRoot, which refuses any root carrying a scheme, host
or userinfo of its own, to join the path onto the configured url.
When pCloud redirects the browser back to rclone's local OAuth
callback server it adds a `hostname` parameter saying which regional
API host (api.pcloud.com or eapi.pcloud.com) the account lives on.
rclone used this for the token exchange and saved it in the config as
the host for all of the remote's later traffic, without checking it.
This only accepts a well formed hostname within pCloud's own domain,
lower cased, failing the config with an error otherwise, and adds a
unit test for the check. This is defence in depth for GHSA-hvpr-p4pv-4f46
The local web server which receives the OAuth callback during
`rclone config` and `rclone authorize` checks that the `state`
parameter matches the random value rclone generated when starting the
flow. The pcloud backend opted out of this check with the
`StateBlankOK` option in lib/oauthutil, which made the local server
accept a callback with no state at all.
The callback URL (http://127.0.0.1:53682/) is fixed and well known,
so while a pcloud OAuth flow was in progress any web page open in the
user's browser could complete it with a single cross-site request
carrying an attacker's authorization code. This is a CSRF
(CWE-352) on the auth callback.
This commit removes the `StateBlankOK` option so that every callback
must carry the correct state and adds a test that the local auth
server rejects callbacks with a blank, missing or wrong state.
The option was added in May 2020 (#4210) because pCloud briefly
omitted the state parameter from its redirect. The callbacks recorded
five weeks later when EU region support was added show both
api.pcloud.com and eapi.pcloud.com returning `state=` again, so the
exemption has been unnecessary since then.
See: GHSA-hvpr-p4pv-4f46
In --links mode the local backend checked that a source name stayed inside
the destination root and only then removed the ".rclonelink" suffix from the
resulting OS path. A source entry called "...rclonelink" passed the check and
was then trimmed to "..", so the object addressed the parent of the
destination the user chose.
Writing the content of such an entry was already refused, as it is done
through an os.Root, but metadata was not: with --metadata rclone rewrote the
timestamps, ownership and extended attributes of the directory above the
destination from values supplied by the source.
The suffix is now removed from the remote name before it is joined to the
root, so the containment check applies to the name that is actually used. A
translated link must also leave a real native file name once the suffix is
removed.
A regular file whose name is exactly ".rclonelink" (or "..rclonelink" or
"...rclonelink") is not a valid translated link and is now skipped in
directory listings.
In --links mode the local backend removed the ".rclonelink" suffix from
directory names as well as file names when building the OS path. Only files
are ever translated into symlinks, so a directory called "sub.rclonelink"
addressed the sibling "sub" instead: listing it logged "Failed to read link
size", its metadata was read from the wrong place, and setting metadata on
it with --metadata failed or landed on the sibling.
Directories now use their name as is, whatever suffix it has, in the same
way DirSetModTime already does.
Every file download sends the live account authorization token and,
when configured, the SSE-C customer key headers to the download host.
With --b2-download-url that host is a CDN or proxy the user does not
operate, and the http client followed its redirects with Go's default
policy:
- Go only strips Authorization when the redirect target is not the same
host or a subdomain, and never looks at the scheme, so a same-host
HTTPS to HTTP redirect replayed the full-capability B2 token in
cleartext.
- Go has no idea that the X-Bz-Server-Side-Encryption-Customer-* headers
carry the customer encryption key, so it forwarded them to any
redirect target including a different host.
The B2 client now uses rest.StripHeadersOnCrossHostRedirectFn, so it
refuses to follow a redirect which downgrades HTTPS to HTTP and strips
the Authorization and SSE-C headers from any redirect which leaves the
host and port of the original request, matching the protection already
given to the S3, WebDAV and HTTP backends.
Fixes GHSA-gpw7-77cq-9f72
The s3 and http backends each carried their own copy of the same
CheckRedirect policy. This moves it into lib/rest as
StripHeadersOnCrossHostRedirectFn, so that any backend which sends
credentials in headers to a host it does not control can use the same
policy. It logs at debug level when it strips headers so that a
resulting 401 from the redirect target can be diagnosed. A header
starting with "*" is matched without canonicalising, as that is how
SetHeader and Opts.ExtraHeaders send it.
This changes the s3 policy in two edge cases:
- It compared the raw URL host string, so a redirect to the same host
in different case or with an explicit default port stripped the
credentials. It now uses SameHost like the http backend, which treats
those as the same host.
- The downgrade check compared each hop with the previous one. It now
compares with the original request like the other backends, so a
chain which starts on plaintext HTTP may return to it.
The behaviour tests move to lib/rest. The backend tests which check the
policy is wired into each client are kept.
Nextcloud chunk uploads currently return a retryable error when a
chunk PUT fails. The outer low-level retry then restarts the whole
upload, removing the temporary upload directory and sending previously
accepted chunks again.
Retry each chunk in place using the configured low-level retry budget,
rewinding its repeatable body before each attempt. Once that budget is
exhausted, prevent the outer retry from restarting the entire object.
WriteChunk checksums the chunk with io.Copy before sending it. For a
reader that implements neither WriterTo nor a memory backed Read - a
local file opened by a multi-thread copy - io.Copy falls back to its own
32 KiB buffer, so the source is read in 32 KiB pieces and the disk sees
thousands of requests per second where it should see hundreds.
Hand io.CopyBuffer a 1 MiB buffer instead. It is allocated for the
checksum pass rather than taken from the pool, where it could wait for
ever on a --max-buffer-memory below the page size. Readers that do
implement WriterTo, such as pool.RW, keep their fast path and don't get
a buffer.
A failed delta poll used to wipe the stored resume token: changeNotifyRunner
returned a named nextDeltaToken which stayed empty on error, and the caller
stored it unconditionally. Later polls then asked for /root/delta?token=
instead of resuming, so notifications stopped until the mount was restarted.
changeNotifyNextChange also called the API without the pacer, so a single
transient 503 was enough to trigger this.
While fixing that, walk @odata.nextLink as well. A change set spanning more
than one page returns nextLink without a deltaLink, and the token parsed from
the missing deltaLink was empty - changes were dropped even when no error
occurred.
Fixes#10009
A symlink in an archive was listed as an ordinary file holding the link
target by the zip backend, and dropped entirely by the squashfs one.
Symlinks are now skipped unless -l/--links is in use, as they are on the
other backends. With that flag they are listed with the .rclonelink
suffix and read back as the path they point at, so they can be copied to
a destination which supports symlinks.
This changes what the zip backend lists by default: a symlink used to
appear as a small file whose contents were the link target, and is now
skipped with a NOTICE naming the flag.
Fixes#9569
Listing "dir/" on a storage account with a hierarchical namespace
returns the file "dir" if it exists, so a file was listed as a
directory containing itself.
On storage accounts with a hierarchical namespace, directories can be
read as blobs so NewObject on a directory returned fs.ErrorNotAFile
rather than fs.ErrorObjectNotFound.
Storage accounts with a hierarchical namespace (ADLS Gen2) have real
directories. Rclone could not remove these - with directory_markers it
failed with "InvalidUri" and without it rmdir silently did nothing.
Directories are also left behind when the last file in them is
deleted.
This adds the hns option which is read from the storage account if
not set and saved in the config file. If set, rclone:
- supports empty directories whatever the directory_markers setting
- creates directories with Mkdir
- removes directories with Rmdir
- doesn't create directory markers for parent directories on upload
Add a new backend for dosya.dev, a cloud file storage, sharing, and
synchronization platform with workspace-based multi-tenancy.
Features:
- Server-side copy, move, rename (files and directories)
- Purge (recursive directory delete)
- Recursive listing (ListR) for --fast-list
- Multipart upload for large files (>10MB)
- Public link sharing via rclone link
- About (workspace storage quota)
- MIME type detection
- Empty directory support
- Workspace-based multi-tenancy (one remote per workspace)
- Download via presigned R2 URLs with Range header support
Co-authored-by: firatkaya <firat@netiket.com.tr>
When an upload failed during the transfer, the partial file was
removed but its handle was never closed. The server kept the deleted
file open, so its space stayed allocated until the pooled connection
was closed, which in rcd, mount or serve could be until rclone exited.
This closes the handle before removing the partial file.
When an upload was stopped part way through by --max-transfer, HiDrive
kept the part of the file it had received, leaving a truncated file at
the destination, or replacing an existing file with a truncated one.
This was introduced in e8f421d28 which accounted the buffered start of
each upload as it was sent rather than as it was read, so the transfer
limit could abort the request which creates the file half way through.
Account the requests which create or replace a file in one go as they
are buffered once more, so the limit is hit before anything is sent.
The chunks of larger uploads are still accounted as they are sent.
ownCloud 10.16 answers a PROPFIND for a path which doesn't exist with
a 207 Multi-Status response, rather than a 404, with a body which
starts a multistatus document then appends a Sabre NotFound error to
it. This isn't valid XML so rclone failed with
read metadata failed: XML syntax error on line 2: expected attribute name in element
whenever it was pointed at a directory which didn't exist yet.
Detect the NotFound error in the response and treat it as a 404.
With meta_format = none, listing a directory which contained the
temporary chunks of an interrupted upload, but no completed chunks for
that file, panicked with "invalid chunked object".
The listing made a placeholder object for the file on seeing a
temporary chunk, which then had no chunks in it when it was validated.
Only make the placeholder on seeing a data chunk, so files which have
nothing but temporary chunks are ignored as they are when metadata is
in use.
The Blob Listing with Apache Arrow feature is now public and shipped
in azblob v1.8.1, so the temporary backend/azureblob/arrowlist package
which implemented it on top of the previous SDK has been removed and
the backend now uses the SDK's own listing pager with ResponseFormat
set to Arrow.
As the SDK client handles every credential type this also makes Arrow
and parallel listing work with connection_string auth, and the
use_arrow_list and list_parallelism options are no longer hidden.
Pointing the archive backend at a file inside a squashfs image, for
example `rclone cat :archive:image.sqfs/dir/file.txt`, listed every
file in the directory containing it instead of just that file. The zip
archiver already behaved correctly.
The squashfs archiver now remembers the file the root points at and
only exposes that one, as zip does. Its Root() includes the file so
that the fs cache does not hand back the Fs for the whole directory in
its place.
With -l/--links a .rclonelink object is buffered in memory to become
the target of a symlink. The read was unbounded, so a hostile or
corrupt source serving a large .rclonelink object made rclone use that
much memory and then log the whole body in the resulting error.
A symlink target can never be longer than a path, so the read now stops
at 128 KiB and anything longer is refused without retrying.
Reading an object in an archive storage class failed with "Object in
GLACIER, restore first" even when the object was in DEEP_ARCHIVE or in
an Intelligent-Tiering archive access tier.
Use the storage class and access tier from the InvalidObjectState
error. The storage class is optional in that error, so fall back to
the storage class from the listing or HEAD, and then to GLACIER as
before.
When listing with --s3-versions or --s3-version-at the storage class
of each object was dropped, so it read as STANDARD unless the object's
metadata was fetched separately. This meant backend restore skipped
objects in GLACIER or DEEP_ARCHIVE with "Not GLACIER or DEEP_ARCHIVE
or INTELLIGENT_TIERING storage class", and lsf --format T showed the
wrong tier.
This happened because ObjectVersion.StorageClass has a different type
from Object.StorageClass so the generated setFrom helper does not copy
it. Convert it explicitly after the setFrom call.
Rclone always asked for the delta listing of the whole drive and threw
away the items which weren't under the directory being listed.
The delta API now works on any folder, so ask for the delta listing of
the directory being listed instead, falling back to listing from the
root of the drive for drives which only support delta there.
Moving a directory can take longer than the API gateway allows, so the
request fails with a 520 or 502 error even though the move completes.
The retry then failed with "Folder ... was already moved to that
location (status 409)".
Treat that error as success.
The Internxt API serves listings from read replicas which lag behind
writes, so for a short while after files or directories are moved or
deleted they can still be listed in their old location.
This caused removing a directory which had just been emptied to fail
with "directory not empty" (eg when moving a directory without server
side directory moves or purging a directory) and a directory which had
just been moved to be found in its old location.
Remember the directories this process has moved or deleted and ignore
directory and file entries which contradict that when listing, finding
directories and checking a directory is empty before removing it.
Moving a file into a directory which had just been created failed with
"Not Found (status 404)", and moving a file over one which had just been
deleted (as sync does with --backup-dir and --suffix) failed with "A file
with the same name already exists in destination folder (status 409)".
The API checks moves against read replicas which lag behind writes, so
retry these errors until the replicas catch up.
The Internxt API serves lookups and listings from read replicas which
lag behind writes, so for a short while after a file is moved it can
still be returned from its old location.
When sync moved a file into the backup location and then uploaded its
replacement, the upload could find the moved file under its old name
and overwrite it. Overwriting renames the existing file by UUID and
deletes it once the upload succeeds, so the file that had just been
moved into the backup location was deleted.
Remember the files this process has moved or deleted for a minute and
ignore lookups and listing entries which contradict that.
With hard_delete set, deleted files and directories carried on being
listed for about a second afterwards because the Drime server doesn't
invalidate its cache of the parent folder listing when entries are
deleted forever. This made removing a directory straight after
emptying it fail with "directory not empty".
Moving an entry to the trash does invalidate the cache, so with
hard_delete set rclone now moves the entry to the trash first and then
deletes it forever.
When server-side copying to a destination which already existed, the
Drime server gave the copy the name "name (1)" and rclone only renamed
it if the source and destination leaf names differed. The existing file
was then deleted leaving the copy under the wrong name.
The server refuses to rename an entry to a name which is already in
use, so this removes the existing file straight after the copy, then
renames the copy whenever its name differs from the destination name.
The premiumize.me upload server has started truncating the multipart
file name at the first ";", so uploading "a;b.txt" created a file
called "a" and the upload then failed with "object not found".
Directory creation and renames still accept ";", so files whose names
contain ";" are now uploaded under a temporary name and renamed into
place. The encoding is left unchanged so existing files and
directories containing ";" remain accessible.
When S3 returns URL-encoded keys from ListObjectVersions, URL encoding
can change their lexical order. This could make mergeDeleteMarkers
place a delete marker after older versions of the same key, so
--s3-version-at reported deleted objects as live.
Compare decoded keys while merging, while preserving the encoded keys
for the existing listing decode path.
Fixes#9948
Testing against OneDrive personal (free) and OneDrive for Business shows
- personal accounts now create versions on setting the modification
time and can delete them, so --onedrive-no-versions and rclone
cleanup work there
- --onedrive-link-password works on OneDrive for Business
- personal free accounts can't set a link password or --expire
- --onedrive-link-type embed only works on OneDrive personal
- personal accounts store times with 1s precision, not mS
See #9917
findItem read the response status code when the lookup failed, so a
failure with no HTTP response panicked.
Thanks to @manus-pi for finding this problem.
DirMove discarded the error from the destination check and read the
response status code, so a failure with no HTTP response panicked.
Other errors were reported as the destination existing; they are now
returned.
Thanks to @manus-pi for finding this problem.
Rmdir and Purge read the response status code before checking for an
error, so a failed DeleteFolder call with no HTTP response (a network
error, or purging the root which fails validation) panicked.
Thanks to @manus-pi for finding this problem.
