Nextcloud chunk uploads currently return a retryable error when a
chunk PUT fails. The outer low-level retry then restarts the whole
upload, removing the temporary upload directory and sending previously
accepted chunks again.
Retry each chunk in place using the configured low-level retry budget,
rewinding its repeatable body before each attempt. Once that budget is
exhausted, prevent the outer retry from restarting the entire object.
WriteChunk checksums the chunk with io.Copy before sending it. For a
reader that implements neither WriterTo nor a memory backed Read - a
local file opened by a multi-thread copy - io.Copy falls back to its own
32 KiB buffer, so the source is read in 32 KiB pieces and the disk sees
thousands of requests per second where it should see hundreds.
Hand io.CopyBuffer a 1 MiB buffer instead. It is allocated for the
checksum pass rather than taken from the pool, where it could wait for
ever on a --max-buffer-memory below the page size. Readers that do
implement WriterTo, such as pool.RW, keep their fast path and don't get
a buffer.
FsOpenChunkWriter wrote three chunks of its own fixed sizes whatever
ChunkWriterInfo.ChunkSize the backend returned. That happened to be
the size the s3 backend uses by default, so the test passed there, but
a chunk writer which works out where a chunk goes from its number needs
every chunk but the last to be exactly the size it asked for - as a
multi-thread copy writes them - and the test could neither exercise nor
catch that.
Cut the file into chunks of info.ChunkSize, the last one short, still
written out of order. Where the backend lets the test set its chunk
size, ask for 5 MiB first so the file takes several chunks. Where it
doesn't and the chunks leave the file in fewer than three, start again
with a file of two full chunks and a short one, for chunks up to
128 MiB.
A failed delta poll used to wipe the stored resume token: changeNotifyRunner
returned a named nextDeltaToken which stayed empty on error, and the caller
stored it unconditionally. Later polls then asked for /root/delta?token=
instead of resuming, so notifications stopped until the mount was restarted.
changeNotifyNextChange also called the API without the pacer, so a single
transient 503 was enough to trigger this.
While fixing that, walk @odata.nextLink as well. A change set spanning more
than one page returns nextLink without a deltaLink, and the token parsed from
the missing deltaLink was empty - changes were dropped even when no error
occurred.
Fixes#10009
Before this change the parent process kept the remote control and
metrics servers it started in initConfig listening, so the daemon child
could not bind the same addresses and exited.
After this change the parent shuts those servers down just before
daemonizing, and the child starts them as usual.
A symlink in an archive was listed as an ordinary file holding the link
target by the zip backend, and dropped entirely by the squashfs one.
Symlinks are now skipped unless -l/--links is in use, as they are on the
other backends. With that flag they are listed with the .rclonelink
suffix and read back as the path they point at, so they can be copied to
a destination which supports symlinks.
This changes what the zip backend lists by default: a symlink used to
appear as a small file whose contents were the link target, and is now
skipped with a NOTICE naming the flag.
Fixes#9569
If a backend's ListR returned a directory after objects inside it,
the directory was listed twice when using --max-depth or filters. This
happens with azureblob on storage accounts with a hierarchical
namespace.
When a backend saved config, for example a refreshed OAuth token, for
a remote which wasn't in the config file, a section with just that
value was created in the config file for environment variable remotes
and the config file was rewritten for on the fly remotes.
Config is now only saved if the remote is in the config file.
Listing "dir/" on a storage account with a hierarchical namespace
returns the file "dir" if it exists, so a file was listed as a
directory containing itself.
On storage accounts with a hierarchical namespace, directories can be
read as blobs so NewObject on a directory returned fs.ErrorNotAFile
rather than fs.ErrorObjectNotFound.
Storage accounts with a hierarchical namespace (ADLS Gen2) have real
directories. Rclone could not remove these - with directory_markers it
failed with "InvalidUri" and without it rmdir silently did nothing.
Directories are also left behind when the last file in them is
deleted.
This adds the hns option which is read from the storage account if
not set and saved in the config file. If set, rclone:
- supports empty directories whatever the directory_markers setting
- creates directories with Mkdir
- removes directories with Rmdir
- doesn't create directory markers for parent directories on upload
With --auth-proxy, when several logins with the same credentials
arrived together and none was in the cache yet, as when an FTP client
opens several connections at once, each one ran the auth proxy program
and took its own reference to the backend. Only one of them was
remembered, so the others were never given back and the backend was
never shut down.
Logins which arrive together now share one run of the auth proxy
program and one reference to the backend.
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. With --vfs-cache-mode writes or full, files
are uploaded to the backend after the transfer which wrote them has
finished, which didn't count as a use. So an upload which hadn't
finished 5 minutes after the user's last command (or disconnection for
serve sftp) was stopped, leaving the file in the VFS cache but not on
the backend.
The backend is now kept until it has no files open for write and
nothing in the VFS cache waiting to be uploaded.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
Busy returns true if the VFS has files open for write or files in the
VFS cache which are open or waiting to be uploaded. It is for code
which wants to shut a VFS down only once that wouldn't lose any work.
An entry is expired when it hasn't been used for a while, but the user
of the cache may know it is still doing something in the background.
SetCanExpire sets a function which is asked before expiring an entry
and can keep it in the cache until it is ready to go.
The function is called without the cache locked as it may take a
while, so an entry which is used in the meantime isn't expired.
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each request counted as a
use, so an upload or download lasting longer than that had its backend
shut down under it and failed.
Each request now holds the backend it uses until it has been served.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each request counted as a
use, so a download lasting longer than that had its backend shut down
under it and failed.
Each request now holds the backend it uses until it has been served.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
With --auth-proxy the backend of each user is shut down once it has
been unused for 5 minutes. Only the start of each FTP command counted
as a use, so an upload or download lasting longer than that had its
backend shut down under it and failed with "context canceled".
Each FTP command now holds the backend it uses until it has finished,
which for a download is when the file is closed.
This was introduced in v1.75.1 by
f425f8d46 serve: refactor VFS and proxy handling into Provider
Add a new backend for dosya.dev, a cloud file storage, sharing, and
synchronization platform with workspace-based multi-tenancy.
Features:
- Server-side copy, move, rename (files and directories)
- Purge (recursive directory delete)
- Recursive listing (ListR) for --fast-list
- Multipart upload for large files (>10MB)
- Public link sharing via rclone link
- About (workspace storage quota)
- MIME type detection
- Empty directory support
- Workspace-based multi-tenancy (one remote per workspace)
- Download via presigned R2 URLs with Range header support
Co-authored-by: firatkaya <firat@netiket.com.tr>
When an upload failed during the transfer, the partial file was
removed but its handle was never closed. The server kept the deleted
file open, so its space stayed allocated until the pooled connection
was closed, which in rcd, mount or serve could be until rclone exited.
This closes the handle before removing the partial file.
Tests in fs/operations, fs/sync, cmd/bisync, cmd/gitannex and vfs are
run by the integration tests against every backend, so they need to
cope with backends with limited functionality, ideally by skipping on
the Fs.Features() flags.
Files.com now treats the replacement characters rclone uses for / and
\ in file names (／ and ＼) as if they were the originals, and the
names ． and ．． as if they were . and .., and rejects them all with
"Invalid path". There is no other way of encoding these names, so
document the restriction and ignore the integration tests for them.
When an upload was stopped part way through by --max-transfer, HiDrive
kept the part of the file it had received, leaving a truncated file at
the destination, or replacing an existing file with a truncated one.
This was introduced in e8f421d28 which accounted the buffered start of
each upload as it was sent rather than as it was read, so the transfer
limit could abort the request which creates the file half way through.
Account the requests which create or replace a file in one go as they
are buffered once more, so the limit is hit before anything is sent.
The chunks of larger uploads are still accounted as they are sent.
ownCloud 10.16 answers a PROPFIND for a path which doesn't exist with
a 207 Multi-Status response, rather than a 404, with a body which
starts a multistatus document then appends a Sabre NotFound error to
it. This isn't valid XML so rclone failed with
read metadata failed: XML syntax error on line 2: expected attribute name in element
whenever it was pointed at a directory which didn't exist yet.
Detect the NotFound error in the response and treat it as a 404.
Koofr refuses to download the HTML listings written by the index tests
with "FileBlocked: File download restricted due to possible dangerous
content" and OpenDrive refuses with 403 Forbidden, so the tests can't
read back what they wrote. The index tests which write other formats
still run.
With meta_format = none, listing a directory which contained the
temporary chunks of an interrupted upload, but no completed chunks for
that file, panicked with "invalid chunked object".
The listing made a placeholder object for the file on seeing a
temporary chunk, which then had no chunks in it when it was validated.
Only make the placeholder on seeing a data chunk, so files which have
nothing but temporary chunks are ignored as they are when metadata is
in use.
- Only check the MIME type of the listings on backends which declare
both ReadMimeType and WriteMimeType, and ignore its parameters, as
many backends report a MIME type they chose themselves or drop the
charset.
- Write the test file before turning --dry-run on. Chunker finishes
an upload with operations.Move, which honours --dry-run, so it left
temporary chunks behind which stopped the test directory being
removed and made the tests following fail.
- Don't check the transfer and delete counts on chunker, as moving
its chunks into place is counted in the same stats.
- Don't expect a new file to change the time of the directories above
it on backends which can't set modification times, as it may get the
same time as the files already there.
The Blob Listing with Apache Arrow feature is now public and shipped
in azblob v1.8.1, so the temporary backend/azureblob/arrowlist package
which implemented it on top of the previous SDK has been removed and
the backend now uses the SDK's own listing pager with ResponseFormat
set to Arrow.
As the SDK client handles every credential type this also makes Arrow
and parallel listing work with connection_string auth, and the
use_arrow_list and list_parallelism options are no longer hidden.
Pointing the archive backend at a file inside a squashfs image, for
example `rclone cat :archive:image.sqfs/dir/file.txt`, listed every
file in the directory containing it instead of just that file. The zip
archiver already behaved correctly.
The squashfs archiver now remembers the file the root points at and
only exposes that one, as zip does. Its Root() includes the file so
that the fs cache does not hand back the Fs for the whole directory in
its place.
With -l/--links a .rclonelink object is buffered in memory to become
the target of a symlink. The read was unbounded, so a hostile or
corrupt source serving a large .rclonelink object made rclone use that
much memory and then log the whole body in the resulting error.
A symlink target can never be longer than a path, so the read now stops
at 128 KiB and anything longer is refused without retrying.
Deleting a selection finished with "Successfully deleted all items!",
which claimed more than had happened - only the selected entries were
removed. It now names the number of items deleted.
The screen was not redrawn until the whole selection had been deleted,
which made the UI look hung, so a progress box is drawn before each
deletion.
Fixes#9516
The docs describe a duration as a sequence of numbers each with a unit
suffix, including d, w, M and y, but a sequence that used one of those
units, such as --max-age 1d12h, failed with a confusing error about
parsing it as a date. Only a single number with one of them, like 1.5d,
or a sequence of the time.ParseDuration units was accepted.
Parse such sequences before falling back to dates, rejecting ones too
long to represent.
Add an opt-in preflight that excludes only guaranteed tracked renames
from the max-delete count. Reuse normal bisync listing metadata,
validate strategy and flag conflicts early, and skip preflight work
when --force bypasses the safety check.
The max_delete_track_renames scenario uses the default hash
track-renames strategy, which requires a hash common to both paths.
Skip it on remote combinations without one, such as crypt, instead of
failing the integration tests.
Fixes#8685
Move strategy parsing, capability checks, and matcher behavior into a
reusable package without changing sync behavior. Add a lightweight
candidate API so callers can count guaranteed matches without
retaining filesystem objects.
Setting the modtime of a closed file refreshed the cache item's
fingerprint before the new modtime had been applied to the remote
object. The next open then saw a fingerprint mismatch, logged "removed
cache file as stale (remote is different)" and downloaded the whole
file again, even though only the modtime had changed.
Refresh the cache fingerprint again once the remote modtime has been
set.
When a file was reopened within the --vfs-handle-caching grace period
(default 5s) after its remote fingerprint changed - for example after a
touch or any other modtime change - _checkObject removed the stale
cache file and open recreated an empty one, but nothing sized it. The
next GetSize stat'd the empty file and set the item size to 0, so
ReadAt's _ensure clamped the request to nothing and skipped the
download, then the size check zero-extended the file and returned the
zeros. Reads through a mount returned the correct length but zeros for
the first read (128 KiB through FUSE).
This is easy to hit with git on a --vfs-cache-mode full mount: git
freshens pack files with utime, and a concurrent reader then sees
"not a GIT packfile".
Size the recreated cache file from the object before opening it, as a
normal open does.
James Truitt