Since control characters in names are no longer encoded by default, a
file uploaded with a newline in its name keeps it on the server. Some
restricted shells, such as the one on rsync.net, end the command at a
newline even when it is quoted, so the hash command ran on a different
path and failed, or could have hashed a different file.
rclone now doesn't use a remote command to hash a file with a newline in
its name on a unix shell, and reports the hash as unavailable instead.
Jottacloud is case insensitive, so a server-side copy to "File1" while
"file1" was in the trash restored the trashed file under its old name
"file1". The restored file is now renamed to the requested name.
When a server-side copy landed on a file that was in the trash, rclone
restored it with a zero creation and modification time unless
--metadata was in use. The server then returned a time which couldn't be
parsed, the copy was reported as having size 0 and failed with
"corrupted on transfer".
The restored file now gets the source's times, as it did before
metadata support was added.
Server-side copies with the local backend on macOS failed with `errno -1` when the destination file already existed, for example with `--inplace` or when used under chunker. This was because copyfile(3) with COPYFILE_CLONE implies COPYFILE_EXCL.
This clones to a temporary name in the same directory and renames it over the destination, which keeps APFS cloning for overwrites and leaves the destination untouched if the clone fails.
PikPak's download node sometimes answers a ranged GET with 502 Bad
Gateway and a short HTML body. client.Do treats any completed response
as success, and shouldRetry returns immediately when err is nil, so
that HTML was saved as the file and rclone exited 0. With the VFS
cache those bytes were then served again.
Non-2xx download responses are now converted with the backend error
handler, which closes the body. Statuses in retryErrorCodes (429 and
5xx, including 502) are retried by the pacer. Any other status is
returned as an error. Successful 200 and 206 responses are unchanged.
Before this change, the default SFTP encoding (`Slash,Del,Ctl,Dot`, the same
as rclone's standard form) passed names through unchanged, so a name on the
server containing a control character, such as a raw newline, reached rclone
in a non-standard form. Other backends, such as local on Linux, list the same
name in the standard form (`␊`), so the two never matched: syncing such a file
from SFTP to local deleted it on every second run, and bisync recorded
listings that didn't match the files.
After this change, the default encoding is `Slash,Dot`, the same as the local
backend's default on Linux, so control characters in names on the server are
converted to and from the standard form. Servers running on Windows can add the
Windows restricted characters with `--sftp-encoding`, as now documented.
The issue was discovered on the bisync integration tests.
This also fixes `mkParentDir`, which encoded the parent `.` of a top-level
file as a name. With the new default encoding, that created a stray `．`
directory on the server.
A connection returned after an error without an FTP reply code can no
longer be trusted, so close it instead of returning it to the pool.
Errors carrying an FTP reply code still get a NOOP liveness probe.
Add a unit test driving putFtpConnection against a fake FTP server
covering network-error discard, live-connection pooling after an FTP
reply-code error, and discard when the NOOP probe fails.
Cellar is the S3-compatible object storage service of Clever Cloud,
hosted in France.
rclone's defaults work for it: virtual-host style addressing, v2
listings, URL-encoded listings and BucketAlreadyExists behave like AWS.
Cellar does not apply Object Lock headers sent with
CreateMultipartUpload, so object_lock_supported is set to false and the
Object Lock integration tests are skipped. The docs tell users to set
--s3-object-lock-set-after-upload, which applies the lock on multipart
uploads with separate API calls.
The SSE-C options are exposed since Cellar supports customer-provided
keys. The docs note that Cellar rejects CopyObject with an SSE-C source,
so --no-update-modtime is needed and encrypted objects smaller than
--s3-copy-cutoff can't be moved server-side.
The integration tests pass against cellar-c2.services.clever-cloud.com.
jwt v5 removes StandardClaims so use RegisteredClaims instead. v5
sends a single audience as an array of strings, so the audience is
overridden in boxCustomClaims to keep sending it as a string as
before.
This adds a test which checks the JWT assertion sent to the token
endpoint is identical to the one sent with jwt v4.
With a relative path override such as `--sftp-path-override @.` (useful
on chrooted hosting where the SSH shell starts in the directory that
SFTP shows as "/"), a file in the root gets the shell path "-name".
`quoteOrEscapeShellPath` does not escape "-", so the hash command
became `md5sum -name`, which md5sum parses as an option. The hash then
failed and the transfer was reported as "corrupted on transfer".
Absolute shell paths always start with "/", so only relative overrides
were affected. Prefix such paths with "./".
When the sftp backend uses an external ssh program (--sftp-ssh), one
goroutine waits for the ssh program to finish by calling cmd.Wait().
cmd.Wait() sets cmd.ProcessState when the program ends.
At the same time, CanReuse() calls exited(), which reads
cmd.ProcessState without any lock. This is a data race and the Go
race detector reports it. It happens in normal use, for example when
a connection is put back in the pool after a stat.
The race only decides whether a pooled connection can be reused, so
it is unlikely to cause problems in practice, but it makes programs
that use the sftp backend fail when they are tested with -race.
This fix sets an atomic.Bool after cmd.Wait() returns and makes
exited() read that instead of cmd.ProcessState.
It also adds a test which calls exited() while Wait() is running. It
fails with -race before this fix and passes after it.
Auto Drive is permanent storage on the Autonomys Network, with an S3
compatible API at s3.auto-drive.autonomys.xyz.
Storage is permanent: a delete hides an object, and every version is
kept. The provider sets:
- force_path_style, as the API uses path-style requests.
- copy_cutoff to the maximum. UploadPartCopy is not supported, so
server-side copies always use a single CopyObject.
- object_lock_supported false. Object Lock is always on in COMPLIANCE
mode and can't be set by clients, so rclone's Object Lock tests
can't pass.
- No Purge. Deleting a version is always refused, so the S3 Purge,
which deletes every version, fails. rclone's generic purge hides
the current version of each object instead.
- no_check_bucket. There are no bucket endpoints: buckets are created
by the first upload into them, and a CreateBucket request would be
stored as an object.
When a Drive listing error closed the ListR input while directories
were still in the overflow queue, the overflow goroutine sent on that
nil channel while holding the mutex. The main loop then blocked on the
same mutex, so the sync never returned the error.
refillListRInput now stops when the input is already closed, and it
does not add those jobs to the wait group.
Fixes#10069
Opening an archive in the root of a remote, such as remote:file.zip,
failed with "file does not exist" because the archive's leaf name was
found with path.Base, which doesn't understand remote names. It is now
found with fspath.Split.
Drime won't list more than 20,000 entries of a directory. Asking for a
later page gets the last page it allows again, so rclone asked for the
same page forever and used more and more memory until it was killed.
Return an error instead. An error stops sync deleting files based on
the listing and uploading duplicates of files it couldn't see.
Also return listing errors from getItem rather than "object not found".
Directory listings showed every directory with a size of -1, so
`rclone lsd` and `operations/list` couldn't say how big a directory
was, and finding out meant walking it with `rclone size`.
Microsoft Graph already returns each folder's `size` as the total size
of its contents, and the backend read it but dropped it. Directories
now carry that size, as they already carry the item count. Listing a
directory's parent is enough to get the size of all its subdirectories,
without listing the subdirectories themselves.
OneDrive updates a folder's size in the background, so for a few
minutes after a change inside it the size may be out of date.
Fixes#7502
When container soft delete is enabled, listing the containers at the
root of the account returned the soft deleted ones too, because the
listing asked for them with Include Deleted. rclone treated them as
normal containers and then failed to list each of them with
"directory not found". With at least one error, sync refused to delete
anything on the destination ("not deleting files as there were IO
errors") and ended with exit code 3.
The listing now asks for live containers only, as it did before the
port to the new SDK in a131da2c3. Nothing else in the backend used the
deleted flag of the containers.
Nextcloud public shares served from public.php/dav/files reject requests
that lack an X-Requested-With header. Send "XMLHttpRequest" for these
URLs when vendor is nextcloud, unless the user already set the header
with --webdav-headers.
The headers set with the `headers` option were sent to any host the
server redirected to, as net/http only strips Authorization and Cookie
on a cross domain redirect and copies every other header onto each
hop. A server which could be made to redirect to another host would
therefore receive any credential carried in a custom header such as
X-Auth-Token.
Strip the configured headers from a redirect hop once the chain has
left the configured host (a different hostname, subdomain or port)
using rest.StripHeadersOnCrossHostRedirectFn. This logs at debug level
when it happens so a 401 from the redirect target can be diagnosed.
The `auth_redirect` option is unchanged as it is an explicit opt-in to
keep credentials across redirects.
The Authorization header set from `user`, `pass` and `bearer_token` is
deliberately left to net/http's policy, which strips it on a change of
domain but keeps it for a subdomain or a different port, as servers
which redirect to a subdomain rely on that today.
The resume start value returned by the NetStorage list API is joined
onto the endpoint URL as a URL reference to build the next listing
request. A value beginning with `//` is a network-path reference, so a
server (or an on-path attacker when `protocol = http`) which returns
`//other.example/path` could make rclone send the next request, signed
with the account secret, to a host other than the configured one.
Use rest.URLJoinRoot so the resume token can only change the path.
The netstorage backend joins the path after `remote:` onto the
configured host URL according to the URL standard. A path beginning
with `//` is a network-path reference under that standard, so a path
such as `//other.example/dir/` replaced the configured host (and port
and userinfo) rather than being treated as a path. Every request,
including the initial stat, was then signed with the account secret and
sent to the substituted host. The signature covers the request path and
action but not the host, so the substituted host received signatures
valid for replay against the real NetStorage endpoint for the duration
of their validity.
Use rest.URLJoinRoot, which refuses any root that changes the scheme,
host or userinfo of the configured URL.
The http backend resolves the path after `remote:` against the
configured url according to the URL standard. A path beginning with
`//` is a network-path reference under that standard, so a path such as
`//other.example/dir/` replaced the configured host (and port and
userinfo) rather than being treated as a path. The configured headers
were then sent to the substituted host on the initial HEAD request and
every request after it.
Use rest.URLJoinRoot, which refuses any root carrying a scheme, host
or userinfo of its own, to join the path onto the configured url.
When pCloud redirects the browser back to rclone's local OAuth
callback server it adds a `hostname` parameter saying which regional
API host (api.pcloud.com or eapi.pcloud.com) the account lives on.
rclone used this for the token exchange and saved it in the config as
the host for all of the remote's later traffic, without checking it.
This only accepts a well formed hostname within pCloud's own domain,
lower cased, failing the config with an error otherwise, and adds a
unit test for the check. This is defence in depth for GHSA-hvpr-p4pv-4f46
The local web server which receives the OAuth callback during
`rclone config` and `rclone authorize` checks that the `state`
parameter matches the random value rclone generated when starting the
flow. The pcloud backend opted out of this check with the
`StateBlankOK` option in lib/oauthutil, which made the local server
accept a callback with no state at all.
The callback URL (http://127.0.0.1:53682/) is fixed and well known,
so while a pcloud OAuth flow was in progress any web page open in the
user's browser could complete it with a single cross-site request
carrying an attacker's authorization code. This is a CSRF
(CWE-352) on the auth callback.
This commit removes the `StateBlankOK` option so that every callback
must carry the correct state and adds a test that the local auth
server rejects callbacks with a blank, missing or wrong state.
The option was added in May 2020 (#4210) because pCloud briefly
omitted the state parameter from its redirect. The callbacks recorded
five weeks later when EU region support was added show both
api.pcloud.com and eapi.pcloud.com returning `state=` again, so the
exemption has been unnecessary since then.
See: GHSA-hvpr-p4pv-4f46
In --links mode the local backend checked that a source name stayed inside
the destination root and only then removed the ".rclonelink" suffix from the
resulting OS path. A source entry called "...rclonelink" passed the check and
was then trimmed to "..", so the object addressed the parent of the
destination the user chose.
Writing the content of such an entry was already refused, as it is done
through an os.Root, but metadata was not: with --metadata rclone rewrote the
timestamps, ownership and extended attributes of the directory above the
destination from values supplied by the source.
The suffix is now removed from the remote name before it is joined to the
root, so the containment check applies to the name that is actually used. A
translated link must also leave a real native file name once the suffix is
removed.
A regular file whose name is exactly ".rclonelink" (or "..rclonelink" or
"...rclonelink") is not a valid translated link and is now skipped in
directory listings.
In --links mode the local backend removed the ".rclonelink" suffix from
directory names as well as file names when building the OS path. Only files
are ever translated into symlinks, so a directory called "sub.rclonelink"
addressed the sibling "sub" instead: listing it logged "Failed to read link
size", its metadata was read from the wrong place, and setting metadata on
it with --metadata failed or landed on the sibling.
Directories now use their name as is, whatever suffix it has, in the same
way DirSetModTime already does.
Every file download sends the live account authorization token and,
when configured, the SSE-C customer key headers to the download host.
With --b2-download-url that host is a CDN or proxy the user does not
operate, and the http client followed its redirects with Go's default
policy:
- Go only strips Authorization when the redirect target is not the same
host or a subdomain, and never looks at the scheme, so a same-host
HTTPS to HTTP redirect replayed the full-capability B2 token in
cleartext.
- Go has no idea that the X-Bz-Server-Side-Encryption-Customer-* headers
carry the customer encryption key, so it forwarded them to any
redirect target including a different host.
The B2 client now uses rest.StripHeadersOnCrossHostRedirectFn, so it
refuses to follow a redirect which downgrades HTTPS to HTTP and strips
the Authorization and SSE-C headers from any redirect which leaves the
host and port of the original request, matching the protection already
given to the S3, WebDAV and HTTP backends.
Fixes GHSA-gpw7-77cq-9f72
The s3 and http backends each carried their own copy of the same
CheckRedirect policy. This moves it into lib/rest as
StripHeadersOnCrossHostRedirectFn, so that any backend which sends
credentials in headers to a host it does not control can use the same
policy. It logs at debug level when it strips headers so that a
resulting 401 from the redirect target can be diagnosed. A header
starting with "*" is matched without canonicalising, as that is how
SetHeader and Opts.ExtraHeaders send it.
This changes the s3 policy in two edge cases:
- It compared the raw URL host string, so a redirect to the same host
in different case or with an explicit default port stripped the
credentials. It now uses SameHost like the http backend, which treats
those as the same host.
- The downgrade check compared each hop with the previous one. It now
compares with the original request like the other backends, so a
chain which starts on plaintext HTTP may return to it.
The behaviour tests move to lib/rest. The backend tests which check the
policy is wired into each client are kept.
Nextcloud chunk uploads currently return a retryable error when a
chunk PUT fails. The outer low-level retry then restarts the whole
upload, removing the temporary upload directory and sending previously
accepted chunks again.
Retry each chunk in place using the configured low-level retry budget,
rewinding its repeatable body before each attempt. Once that budget is
exhausted, prevent the outer retry from restarting the entire object.
WriteChunk checksums the chunk with io.Copy before sending it. For a
reader that implements neither WriterTo nor a memory backed Read - a
local file opened by a multi-thread copy - io.Copy falls back to its own
32 KiB buffer, so the source is read in 32 KiB pieces and the disk sees
thousands of requests per second where it should see hundreds.
Hand io.CopyBuffer a 1 MiB buffer instead. It is allocated for the
checksum pass rather than taken from the pool, where it could wait for
ever on a --max-buffer-memory below the page size. Readers that do
implement WriterTo, such as pool.RW, keep their fast path and don't get
a buffer.
A failed delta poll used to wipe the stored resume token: changeNotifyRunner
returned a named nextDeltaToken which stayed empty on error, and the caller
stored it unconditionally. Later polls then asked for /root/delta?token=
instead of resuming, so notifications stopped until the mount was restarted.
changeNotifyNextChange also called the API without the pacer, so a single
transient 503 was enough to trigger this.
While fixing that, walk @odata.nextLink as well. A change set spanning more
than one page returns nextLink without a deltaLink, and the token parsed from
the missing deltaLink was empty - changes were dropped even when no error
occurred.
Fixes#10009
A symlink in an archive was listed as an ordinary file holding the link
target by the zip backend, and dropped entirely by the squashfs one.
Symlinks are now skipped unless -l/--links is in use, as they are on the
other backends. With that flag they are listed with the .rclonelink
suffix and read back as the path they point at, so they can be copied to
a destination which supports symlinks.
This changes what the zip backend lists by default: a symlink used to
appear as a small file whose contents were the link target, and is now
skipped with a NOTICE naming the flag.
Fixes#9569
Listing "dir/" on a storage account with a hierarchical namespace
returns the file "dir" if it exists, so a file was listed as a
directory containing itself.
On storage accounts with a hierarchical namespace, directories can be
read as blobs so NewObject on a directory returned fs.ErrorNotAFile
rather than fs.ErrorObjectNotFound.
Storage accounts with a hierarchical namespace (ADLS Gen2) have real
directories. Rclone could not remove these - with directory_markers it
failed with "InvalidUri" and without it rmdir silently did nothing.
Directories are also left behind when the last file in them is
deleted.
This adds the hns option which is read from the storage account if
not set and saved in the config file. If set, rclone:
- supports empty directories whatever the directory_markers setting
- creates directories with Mkdir
- removes directories with Rmdir
- doesn't create directory markers for parent directories on upload
Add a new backend for dosya.dev, a cloud file storage, sharing, and
synchronization platform with workspace-based multi-tenancy.
Features:
- Server-side copy, move, rename (files and directories)
- Purge (recursive directory delete)
- Recursive listing (ListR) for --fast-list
- Multipart upload for large files (>10MB)
- Public link sharing via rclone link
- About (workspace storage quota)
- MIME type detection
- Empty directory support
- Workspace-based multi-tenancy (one remote per workspace)
- Download via presigned R2 URLs with Range header support
Co-authored-by: firatkaya <firat@netiket.com.tr>
When an upload failed during the transfer, the partial file was
removed but its handle was never closed. The server kept the deleted
file open, so its space stayed allocated until the pooled connection
was closed, which in rcd, mount or serve could be until rclone exited.
This closes the handle before removing the partial file.
When an upload was stopped part way through by --max-transfer, HiDrive
kept the part of the file it had received, leaving a truncated file at
the destination, or replacing an existing file with a truncated one.
This was introduced in e8f421d28 which accounted the buffered start of
each upload as it was sent rather than as it was read, so the transfer
limit could abort the request which creates the file half way through.
Account the requests which create or replace a file in one go as they
are buffered once more, so the limit is hit before anything is sent.
The chunks of larger uploads are still accounted as they are sent.
ownCloud 10.16 answers a PROPFIND for a path which doesn't exist with
a 207 Multi-Status response, rather than a 404, with a body which
starts a multistatus document then appends a Sabre NotFound error to
it. This isn't valid XML so rclone failed with
read metadata failed: XML syntax error on line 2: expected attribute name in element
whenever it was pointed at a directory which didn't exist yet.
Detect the NotFound error in the response and treat it as a 404.
With meta_format = none, listing a directory which contained the
temporary chunks of an interrupted upload, but no completed chunks for
that file, panicked with "invalid chunked object".
The listing made a placeholder object for the file on seeing a
temporary chunk, which then had no chunks in it when it was validated.
Only make the placeholder on seeing a data chunk, so files which have
nothing but temporary chunks are ignored as they are when metadata is
in use.
The Blob Listing with Apache Arrow feature is now public and shipped
in azblob v1.8.1, so the temporary backend/azureblob/arrowlist package
which implemented it on top of the previous SDK has been removed and
the backend now uses the SDK's own listing pager with ResponseFormat
set to Arrow.
As the SDK client handles every credential type this also makes Arrow
and parallel listing work with connection_string auth, and the
use_arrow_list and list_parallelism options are no longer hidden.
Pointing the archive backend at a file inside a squashfs image, for
example `rclone cat :archive:image.sqfs/dir/file.txt`, listed every
file in the directory containing it instead of just that file. The zip
archiver already behaved correctly.
The squashfs archiver now remembers the file the root points at and
only exposes that one, as zip does. Its Root() includes the file so
that the fs cache does not hand back the Fs for the whole directory in
its place.
Nick Craig-Wood