Koofr refuses to download the HTML listings written by the index tests
with "FileBlocked: File download restricted due to possible dangerous
content" and OpenDrive refuses with 403 Forbidden, so the tests can't
read back what they wrote. The index tests which write other formats
still run.
With meta_format = none, listing a directory which contained the
temporary chunks of an interrupted upload, but no completed chunks for
that file, panicked with "invalid chunked object".
The listing made a placeholder object for the file on seeing a
temporary chunk, which then had no chunks in it when it was validated.
Only make the placeholder on seeing a data chunk, so files which have
nothing but temporary chunks are ignored as they are when metadata is
in use.
- Only check the MIME type of the listings on backends which declare
both ReadMimeType and WriteMimeType, and ignore its parameters, as
many backends report a MIME type they chose themselves or drop the
charset.
- Write the test file before turning --dry-run on. Chunker finishes
an upload with operations.Move, which honours --dry-run, so it left
temporary chunks behind which stopped the test directory being
removed and made the tests following fail.
- Don't check the transfer and delete counts on chunker, as moving
its chunks into place is counted in the same stats.
- Don't expect a new file to change the time of the directories above
it on backends which can't set modification times, as it may get the
same time as the files already there.
The Blob Listing with Apache Arrow feature is now public and shipped
in azblob v1.8.1, so the temporary backend/azureblob/arrowlist package
which implemented it on top of the previous SDK has been removed and
the backend now uses the SDK's own listing pager with ResponseFormat
set to Arrow.
As the SDK client handles every credential type this also makes Arrow
and parallel listing work with connection_string auth, and the
use_arrow_list and list_parallelism options are no longer hidden.
Pointing the archive backend at a file inside a squashfs image, for
example `rclone cat :archive:image.sqfs/dir/file.txt`, listed every
file in the directory containing it instead of just that file. The zip
archiver already behaved correctly.
The squashfs archiver now remembers the file the root points at and
only exposes that one, as zip does. Its Root() includes the file so
that the fs cache does not hand back the Fs for the whole directory in
its place.
With -l/--links a .rclonelink object is buffered in memory to become
the target of a symlink. The read was unbounded, so a hostile or
corrupt source serving a large .rclonelink object made rclone use that
much memory and then log the whole body in the resulting error.
A symlink target can never be longer than a path, so the read now stops
at 128 KiB and anything longer is refused without retrying.
Deleting a selection finished with "Successfully deleted all items!",
which claimed more than had happened - only the selected entries were
removed. It now names the number of items deleted.
The screen was not redrawn until the whole selection had been deleted,
which made the UI look hung, so a progress box is drawn before each
deletion.
Fixes#9516
The docs describe a duration as a sequence of numbers each with a unit
suffix, including d, w, M and y, but a sequence that used one of those
units, such as --max-age 1d12h, failed with a confusing error about
parsing it as a date. Only a single number with one of them, like 1.5d,
or a sequence of the time.ParseDuration units was accepted.
Parse such sequences before falling back to dates, rejecting ones too
long to represent.
Add an opt-in preflight that excludes only guaranteed tracked renames
from the max-delete count. Reuse normal bisync listing metadata,
validate strategy and flag conflicts early, and skip preflight work
when --force bypasses the safety check.
The max_delete_track_renames scenario uses the default hash
track-renames strategy, which requires a hash common to both paths.
Skip it on remote combinations without one, such as crypt, instead of
failing the integration tests.
Fixes#8685
Move strategy parsing, capability checks, and matcher behavior into a
reusable package without changing sync behavior. Add a lightweight
candidate API so callers can count guaranteed matches without
retaining filesystem objects.
Setting the modtime of a closed file refreshed the cache item's
fingerprint before the new modtime had been applied to the remote
object. The next open then saw a fingerprint mismatch, logged "removed
cache file as stale (remote is different)" and downloaded the whole
file again, even though only the modtime had changed.
Refresh the cache fingerprint again once the remote modtime has been
set.
When a file was reopened within the --vfs-handle-caching grace period
(default 5s) after its remote fingerprint changed - for example after a
touch or any other modtime change - _checkObject removed the stale
cache file and open recreated an empty one, but nothing sized it. The
next GetSize stat'd the empty file and set the item size to 0, so
ReadAt's _ensure clamped the request to nothing and skipped the
download, then the size check zero-extended the file and returned the
zeros. Reads through a mount returned the correct length but zeros for
the first read (128 KiB through FUSE).
This is easy to hit with git on a --vfs-cache-mode full mount: git
freshens pack files with utime, and a concurrent reader then sees
"not a GIT packfile".
Size the recreated cache file from the object before opening it, as a
normal open does.
Shutdown was calling expiryTimer.Stop without holding ts.mu, so it could
panic if OnExpiry had not created the timer yet, and raced the write in
OnExpiry. Hold the lock and skip Stop when the timer is still nil.
Fixes#9980
Reading an object in an archive storage class failed with "Object in
GLACIER, restore first" even when the object was in DEEP_ARCHIVE or in
an Intelligent-Tiering archive access tier.
Use the storage class and access tier from the InvalidObjectState
error. The storage class is optional in that error, so fall back to
the storage class from the listing or HEAD, and then to GLACIER as
before.
Until the first time slot of the week, the timetable used its last entry
as the limit carried over from the week before. That is only the latest
slot of the week when the entries are in order, so a timetable written
weekend first, like "Sat-00:00,off Mon-00:00,1M", limited Sunday to 1M
instead of leaving it unlimited.
Carry over the latest time slot of the week instead.
When listing with --s3-versions or --s3-version-at the storage class
of each object was dropped, so it read as STANDARD unless the object's
metadata was fetched separately. This meant backend restore skipped
objects in GLACIER or DEEP_ARCHIVE with "Not GLACIER or DEEP_ARCHIVE
or INTELLIGENT_TIERING storage class", and lsf --format T showed the
wrong tier.
This happened because ObjectVersion.StorageClass has a different type
from Object.StorageClass so the generated setFrom helper does not copy
it. Convert it explicitly after the setFrom call.
The Docker image had no /etc/mime.types, so MIME types came only from
Go's built-in table plus rclone's small extra list, and many
extensions uploaded as application/octet-stream.
This installs Alpine's mailcap package to fix the problem.
To find the latest patch release of a minor version, for example
`rclone selfupdate --version 1.75`, selfupdate searched the listing of
downloads.rclone.org for href="./vX.Y.Z/". The leading ./ is a detail
of how Caddy's file server writes its links. A listing which linked to
the same directories as "vX.Y.Z/", which is an equally valid relative
URL, made selfupdate fail with "could not find the minor release".
This makes the ./ optional in the pattern, so selfupdate no longer
depends on which program wrote the listing, and takes the version from
a capture group rather than from fixed offsets into the match.
The listings written by rclone index now include the ./ for the
benefit of rclone versions without this fix, so this is to remove the
dependency for the future.
downloads.rclone.org is about to be served from static listings
written by rclone index instead of by Caddy's file server. Released
versions of rclone selfupdate find the latest patch release of a minor
version by searching that listing for href="./vX.Y.Z/", as written by
Caddy. rclone's listings linked to "vX.Y.Z/" without the ./ so `rclone
selfupdate --version X.Y` would have failed with "could not find the
minor release" for every rclone already installed.
Caddy prefixes every link with ./ so that a name with a colon in its
first path segment is not read as an absolute URL with a scheme (RFC
3986 section 4.2). rclone was already safe from that as url.URL.String
adds the ./ but only to names which need it, so links to plain names
had no prefix.
This prefixes all the links with ./ as Caddy does. It changes the
output of serve http, serve webdav and the rc server as well as rclone
index, since they share the code. The links resolve identically, and
it keeps the listings consistent with each other.
Now every URL has the prefix, the caddy.json template no longer needs
to add it.
After a sync which changed a known set of files there is no need to
walk the whole remote. --changed PATH, --changed-from FILE and
--changed-combined FILE tell rclone index what changed, and it
re-indexes only the directories containing those paths and their
ancestors, each with one non-recursive listing.
The listing now shows the folder path with clickable breadcrumbs
above a card containing the entries:
- A summary of the directories, files and total size
- The directory and file counts are toggles
- The search box sits beside the Name heading (focussable with /)
- Sizes right aligned, empty columns are gone and the icons are simpler.
- The colours are CSS variables
The template data is unchanged so custom templates still work, and
the zip download links and ?sort= parameters work as before.
This makes directory listings for buckets and other remotes served as
static websites, for example S3 website endpoints or R2 behind
Cloudflare, so they can be browsed without directory listing support
on the host.
Index writes a directory listing (e.g. index.html) into every
directory of a remote so it can be browsed when served as a static
website. It is also available over the rc as operations/index.
It works like sync. It walks the remote once, renders the listings in
memory and compares them with the existing ones by size and hash from
the listing, or by reading them where the backend has no hash.
Listings in directories which contain nothing else are deleted on
backends which can't have empty directories.
Listings can be written in the serve http HTML format, the lsjson
format, Caddy's browse JSON format, or from a user template. A second
rule set (--index-include and friends) controls which directories get
listings, --dir-time controls the time shown for directories and
--no-modtime avoids reading modification times altogether.
- .Static hides the "up" link at the root
- .SetLinkIndex makes directory links point at an index doc
- .Render writes the listing to an io.Writer
- .Path, .IsRoot, .UpLink, .NumDirs, .NumFiles, .TotalSize and .MimeType.
- Sorting is now stable so the rendered output is deterministic
Clicking the Name, Size or Modified heading now sorts the listing in
the browser and clicking again reverses it, rather than reloading the
page with ?sort= and ?order= parameters. Names sort naturally so
v1.9.0 comes before v1.10.0, and directories stay first when sorting
by name.
The ?sort= and ?order= parameters still work for the initial order and
are shown in the heading, so existing links and custom templates are
unaffected.
MinIO no longer publishes images on quay.io (nor Docker Hub or ghcr.io)
so pulling quay.io/minio/minio returns "unauthorized". pgsty/minio is a
maintained community fork with the same entrypoint layout.
ENETUNREACH and ENETDOWN were not in the list of retriable errors on
non-Windows platforms, so a single failed connection aborted the
transfer instead of being retried as a low level retry. The Windows
list already includes the equivalent WSAENETUNREACH and WSAENETDOWN.
This is easy to hit on a host with IPv6 enabled but no IPv6 route: if
the A lookup fails transiently while the AAAA lookup succeeds, the only
address to dial is IPv6 and the connect fails with ENETUNREACH. A retry
resolves again and normally succeeds.
BwPair.String printed a sub-KiB bandwidth as a bare number, and Set
reads a bare number as KiB, so a rate under 1 KiB grew by a factor of
1024 whenever it went through a string. rc core/bwlimit reports the
current rate in that format and accepts it back, so reading the limit
and setting it again raised it.
63c4fef27 fixed the same corruption for config values by suffixing bare
numbers with B inside Option.String. Move that into a SizeSuffix method
and use it from BwPair.String as well.
Use mime.FormatMediaType instead of unescaped string concatenation so a
directory name containing a double quote cannot inject extra disposition
parameters.
Fixes#9962
Use mime.FormatMediaType instead of unescaped string concatenation so a
directory name containing a double quote cannot inject extra disposition
parameters.
Fixes#9962
Copying an object onto itself, as clients do to replace its metadata,
reported success when the object didn't exist, and copying a missing
object to a different key failed with an internal error. Both now fail
with NoSuchKey as S3 does.
With --etag-hash auto the hash for the ETags was chosen once from the
remote serve s3 was started with. With --auth-proxy there is no such
remote so serve s3 crashed on startup, and when started by the rc it
was the wrong remote, so users whose backends lacked that hash got no
ETags.
The hash is now chosen from the backend of the user making each
request.
The metadata of every object uploaded was kept in memory for as long as
the server ran, even after the object was deleted, so it grew without
limit and the metadata of a deleted object reappeared on any object
later created at the same key other than through serve s3.
Deleting an object now forgets its metadata.
Listing a prefix with no delimiter walked the entire subtree below it into
memory and only then sliced out the requested page, so every page of a
listing cost a full traversal of the tree.
Walk the tree lazily instead, stopping as soon as the page is full and
skipping the subtrees an earlier page already returned. A page now costs a
number of directory reads proportional to the keys it returns rather than to
the size of the subtree, and ETags are computed only for the objects that
are actually returned.
Entries are emitted in the order their keys have in a flat keyspace, with a
directory sorting as if it carried its trailing slash, so that "a.txt" comes
before "a/b" as it does in a real S3 bucket. Without this a resumed listing
would silently skip keys across a page boundary.
When the last user of a VFS shut it down at the same time as a new
user of the same remote asked for one, the new user could be given the
VFS being shut down, with its cache and background tasks stopped.
Only a VFS which is still in use is now reused, otherwise a new one is
made.
A VFS is shut down when the last of the references to it from New is
given back with Shutdown. Hold takes another reference, for code using
a VFS it didn't create itself, which may otherwise be shut down under
it.
Rclone always asked for the delta listing of the whole drive and threw
away the items which weren't under the directory being listed.
The delta API now works on any folder, so ask for the delta listing of
the directory being listed instead, falling back to listing from the
root of the drive for drives which only support delta there.
When a backend rejected the multipart upload as too small, the subtest
skipped without removing its local source file, so the following
upload subtests failed their local listing checks.
Moving a directory can take longer than the API gateway allows, so the
request fails with a 520 or 502 error even though the move completes.
The retry then failed with "Folder ... was already moved to that
location (status 409)".
Treat that error as success.
The Internxt API serves listings from read replicas which lag behind
writes, so for a short while after files or directories are moved or
deleted they can still be listed in their old location.
This caused removing a directory which had just been emptied to fail
with "directory not empty" (eg when moving a directory without server
side directory moves or purging a directory) and a directory which had
just been moved to be found in its old location.
Remember the directories this process has moved or deleted and ignore
directory and file entries which contradict that when listing, finding
directories and checking a directory is empty before removing it.
Moving a file into a directory which had just been created failed with
"Not Found (status 404)", and moving a file over one which had just been
deleted (as sync does with --backup-dir and --suffix) failed with "A file
with the same name already exists in destination folder (status 409)".
The API checks moves against read replicas which lag behind writes, so
retry these errors until the replicas catch up.
Nick Craig-Wood