Rmdir only removed the directory marker of a directory below the root
of the remote, so `rclone rmdir remote:bucket/dir` left the marker of
dir behind, and with it the directory.
gofakes3 v0.0.12 checks the bodies of requests which aren't object
data, such as DeleteObjects, against their Content-MD5 and
x-amz-checksum-* headers.
See #10082
Clients which rely on directory markers, such as rclone's s3 backend
with --s3-directory-markers, need the markers of directories holding
other keys listed, which serve s3 only does while it remembers the
marker was put.
With --directory-markers every directory is listed as a directory
marker, and isn't removed when the last key below it is deleted.
See #10082
A CompleteMultipartUpload which failed because the parts it listed
didn't match those uploaded, or had a gap in the part numbers, aborted
the whole upload. Leave the upload open instead, as S3 does, so the
client can upload any missing parts and complete it again.
See #10082
Check the Last-Modified of HEAD and GET is the modification time set
from the X-Amz-Meta-Mtime or mtime metadata, as in listings, rather
than the time of the PUT.
See #10082
A directory marker such as dir/ was only listed while its directory
was empty, as nothing recorded whether it had been put or the
directory made by putting a key below it. On S3 a marker which was put
is listed until it is deleted, which some clients rely on.
The markers put are now remembered with the object metadata, in memory,
and listed while the directory holds other keys, and their directories
are no longer removed when the last key below them is deleted.
See #10082
Replace the list of supported operations with a table of what is and
isn't compatible with AWS S3, and remove the bug about multipart server
side copies, which work now that UploadPartCopy is implemented.
See #7454
The MD5 of the whole object, needed only for MD5 ETags, was computed
for every multipart upload, costing about a CPU-second per GiB even
with --etag-hash "" or a different hash.
With a delimiter other than "/" found in directory names, each
directory rolling up into a common prefix was read to see if it had
any keys, even if that prefix had already been listed, on this page or
an earlier one.
With --etag-hash MD5, the default, on a remote which can't give the
MD5 of objects, such as crypt or sftp, PutObject returned the MD5 of
the data as the ETag but HeadObject and listings returned "".
Listings also ignored the MD5 kept for multipart uploads, and copying
an object onto itself to replace its metadata lost it, along with the
sizes of its parts.
The MD5 of a PutObject is now kept with the object's metadata, as for
multipart uploads, listings return it, and replacing the metadata of
an object keeps it.
See #10082
A PutObject or CreateMultipartUpload which failed left behind the
directories made for its key, which are listed as directory markers,
so the client saw keys such as "newdir/sub/" it never made.
They now remove the directories they made if they are still empty, as
aborting a multipart upload already did.
CopyObject with a directory marker such as "dir/" as its source failed
with InvalidArgument, although HeadObject and GetObject found it.
It now copies the marker, making the directory of another marker or an
empty object.
See #10082
Putting a directory marker such as "dir/" made the directory, but a
listing without a delimiter didn't return the "dir/" key, so clients
which make empty directories with markers, as the AWS console and many
sync tools do, lost them.
An empty directory is now listed as its directory marker. Other
directories aren't, as they may not have been made by putting one.
See #10082
With an --etag-hash other than MD5, including "" to turn hashing off,
PutObject returned the MD5 of the data and CompleteMultipartUpload an
S3 style multipart ETag, neither of which matched the ETag HeadObject
and listings returned for the object afterwards.
These now return the ETag HeadObject does. Listings now quote an empty
ETag as HeadObject does.
See #10082
This updates expected-pass.txt for the fixes to gofakes3 and serve
s3, which make 125 more of the s3-tests pass, 297 in all.
36 tests no longer pass, as they only passed because requests were
silently ignored or mishandled:
- 19 used SSE-C or SSE-KMS, now refused as they aren't supported
- 12 browser POST uploads expected 400, but unauthenticated requests
now get 403 as on AWS
- 5 set an ACL other than private, or object lock, now refused
The s3-tests config now gives the owner gofakes3 uses for the main
user, so the ACLs returned match it, and no api_name, as bucket
locations aren't stored.
gofakes3 now checks the If-Match and If-None-Match conditions of
PutObject and CompleteMultipartUpload before the object is written,
but another write can replace the object before this one finishes.
For example, of many concurrent uploads of a new object with
If-None-Match: *, most succeeded.
Now when the object is written to a temporary object and moved into
place, the conditions are checked again as it is moved, with the moves
of objects to the same path through the server serialised, so only one
of the uploads in the example succeeds.
CompleteMultipartUpload returned the AWS S3 style multipart ETag (the
MD5 of the parts' MD5s with a "-N" suffix), but HeadObject and GET
returned the hash of the object, so clients saw the ETag of an object
change as soon as it was uploaded.
Now with --etag-hash MD5, the default, the ETag returned by
CompleteMultipartUpload is the MD5 of the whole object, worked out as
it is streamed, so is the same as HeadObject returns. With other
hashes it is stored with the object's metadata and returned by
HeadObject while the metadata is kept.
The sizes of the parts are kept with the metadata too, so GET and
HEAD with partNumber and GetObjectAttributes can return the parts of
the object.
Keys ending in "/", such as "dir/", are used by S3 clients as
directory markers. gofakes3 used to strip the "/", so putting "dir/"
made a file "dir", which then stopped "dir/file" being written. Now
that gofakes3 keeps the "/" these keys were refused.
Now a key ending in "/" stands for the directory: putting an empty one
makes the directory, putting one with data is refused, HEAD and GET
return an empty object if the directory exists, and deleting one
removes the directory if it is empty.
Writing an object which conflicts with a directory, or whose directory
conflicts with an object, now fails with InvalidRequest rather than an
internal error, which clients retried.
Listings treated any delimiter as "/", so a listing with a delimiter
such as "a" or "." returned the keys and directories rather than the
keys rolled up into common prefixes at the delimiter. A delimiter of
whitespace was ignored altogether.
Now any delimiter is supported. Delimiters other than "/" are found by
walking the directory tree below the prefix, skipping directories
whose names contain the delimiter as all their keys roll up into the
same common prefix.
A prefix starting with "/" now matches nothing, as no key can start
with "/", rather than listing the whole bucket.
The ETags of objects in listings were returned without the double
quotes S3 uses, as in HEAD and GET responses, so clients which compare
them saw a difference.
Object tags given when an object is created are kept with its metadata
by gofakes3. This implements gofakes3's ObjectTaggingBackend so the
tags of an existing object can be changed with PutObjectTagging and
DeleteObjectTagging.
Like the rest of the metadata, tags are only kept in memory.
CopyObject merged the source object's metadata into the metadata it
was given, so a copy with x-amz-metadata-directive: REPLACE kept the
source's metadata. It also kept the source's modification time by
adding it to the metadata, so every copy gained "mtime" user metadata
the client hadn't set.
gofakes3 now works out the complete metadata of the copy from the
metadata directive, so it is used as given. The source's modification
time is still kept, unless the metadata sets one, but without adding
it to the metadata.
Starting a multipart upload makes the directory its object will be in,
but aborting the upload didn't remove it. The empty directory isn't
visible to S3 clients, so the bucket looked empty, but deleting it
failed with BucketNotEmpty.
Now aborting an upload removes the directories made for it if they are
still empty, and deleting a bucket holding nothing but empty
directories removes them and succeeds. Deleting a bucket which fails
for some other reason than holding objects now returns that error
rather than BucketNotEmpty.
Removing the empty directories left after deleting an object now
checks they are empty first, rather than logging an error for each
directory which isn't.
This adds cmd/serve/s3/s3tests/run.sh which runs the ceph/s3-tests S3
compatibility suite against rclone serve s3, built from the tree.
It fetches s3-tests at a pinned commit, makes a venv with pinned
dependencies, starts rclone serve s3 with the users s3-tests needs and
summarises the results by feature area.
As serve s3 doesn't implement many S3 features (ACLs, versioning,
lifecycle, policies, etc) the results are checked against
expected-pass.txt, a list of the tests which should pass, rather than
a list of failures. The run fails if any of these don't pass, and
run.sh --update records newly passing tests.
A small pytest plugin adapts the s3-tests cleanup to serve s3 so a
bucket which can't be deleted fails only the test which left it
rather than every test after it.
Linkbox renames an upload to name(1) when its name is the start of an
existing file's name, which breaks TestCaseInsensitiveMoveWithoutServerSideMove.
Internxt listings lag behind directory renames and deletes for longer than
the list retries allow, which breaks TestDirRename and
TestDirRenameParentEntries.
Remotes made by RandomRemote aren't in the fs cache so they were never
shut down, leaving any pooled connections open until they timed out.
On TestFTPProftpd this used up the server's 30 connection limit part way
through the fs/operations tests, making all the following tests fail
with EOF when connecting.
- CVE-2026-111943: crypto/x509: mitigation for security concerns with a specific Certificate Authority
This also updates the go1.26 test job to go1.26.10 which contains the
same fix.
Upgrade to v0.62.0 of golang.org/x/net in order to address:
- CVE-2026-78661: html: avoid quadratic complexity when coalescing text nodes
- CVE-2026-97045: html: check all ancestor namespace transitions when rendering raw text
Linkbox converts file extensions to lower case on upload, and renames an
uploaded file to name(1) if its name is the start of the name of a file
already in the directory, e.g. "apple" next to "applebanana". Document
both and ignore the TestTransform cases which they break.
Since control characters in names are no longer encoded by default, a
file uploaded with a newline in its name keeps it on the server. Some
restricted shells, such as the one on rsync.net, end the command at a
newline even when it is quoted, so the hash command ran on a different
path and failed, or could have hashed a different file.
rclone now doesn't use a remote command to hash a file with a newline in
its name on a unix shell, and reports the hash as unavailable instead.
GCS can't store CR or LF in object names and the S3 provider doesn't
encode them. These cases ran once the bisync pre-flight check was fixed
to stop skipping them.
Some S3 providers, such as Wasabi and Alibaba, report success when
creating a bucket which already exists, so rclone logs that the bucket
was created. The during-sync hooks in the equal_skipped test create a
new Fs, which logged this in the middle of the bisync run and made the
log differ from the golden one. Ignore these messages.
The test looked up the object it had just uploaded without retrying, so
it failed with "object not found" on backends such as Cloudinary and
Internxt where new objects take a moment to appear.
Jottacloud is case insensitive, so a server-side copy to "File1" while
"file1" was in the trash restored the trashed file under its old name
"file1". The restored file is now renamed to the requested name.
When a server-side copy landed on a file that was in the trash, rclone
restored it with a zero creation and modification time unless
--metadata was in use. The server then returned a time which couldn't be
parsed, the copy was reported as having size 0 and failed with
"corrupted on transfer".
The restored file now gets the source's times, as it did before
metadata support was added.
Add the hash type crc32c. It is the checksum that AWS S3 stores for
objects uploaded with x-amz-checksum-crc32c (FULL_OBJECT checksum),
and the checksum that iSCSI, ext4 and Btrfs use. Go calculates it with
the SSE 4.2 and ARMv8 CRC32 instructions where they are available.
With it, rclone can calculate the value that S3 returns in
ChecksumCRC32C for a local file:
rclone hashsum crc32c file
prints it as hex; `--base64` prints it in the URL-safe base64 alphabet
that rclone uses for every hash (`-` and `_` where S3 prints `+` and
`/`).
The new type is registered after the existing core types, so their
Type values do not change. The types that backends register (dropbox,
hidrive, mailru, quickxor) move up one bit, as they did when blake3,
xxh3 and xxh128 were added; no Type value is persisted (the hasher
cache and bisync listings store hash names).
If a file is changed on both sides and the conflict names it gets renamed
to (e.g. X.conflict1 and X.conflict2) were deleted on both sides since the
last run, the listing update removed the just-copied conflict files again
when processing the deleted-on-both queue. The rename bookkeeping then
could not find them and bisync aborted with "bisync.renames: internal
error: missing info".
Skip names that this run copied when removing files deleted on both sides,
and add a test case for the scenario.
Before this change, `TestMoveUnicodeEquivalentWithoutServerSideMove`
failed on backends that can't store a file under the NFD name it was given:
`mailru` rejects the name ("invalid characters in object name"), and
`cloudinary`, `filelu` and `seafile` don't find the file again under it.
Tests in `fs/operations` run against every backend, so this one needs to
skip where the precondition doesn't hold.
After this change, the test skips when the NFD name can't be uploaded or
the file isn't found again under it.
Before this change, `operations.Move` and `operations.MoveTransfer`
deleted a file when asked to move it onto its own path without a `dst`
object. Without server-side `Move`, `operations.move` copied the file onto
itself and then deleted the source, which was the same file, and the
`memory` backend's server-side `Move` did the same. `storj` stores every
name in NFC, so there a move from a file's NFD name to its NFC name is a
move onto its own path, and `TestMoveUnicodeEquivalentWithoutServerSideMove`
lost the file on `TestStorj`.
After this change, a move whose destination is the file's own path on the
same remote does nothing, including when the destination `Fs` has a
different root.
Server-side copies with the local backend on macOS failed with `errno -1` when the destination file already existed, for example with `--inplace` or when used under chunker. This was because copyfile(3) with COPYFILE_CLONE implies COPYFILE_EXCL.
This clones to a temporary name in the same directory and renames it over the destination, which keeps APFS cloning for overwrites and leaves the destination untouched if the clone fails.
PikPak's download node sometimes answers a ranged GET with 502 Bad
Gateway and a short HTML body. client.Do treats any completed response
as success, and shouldRetry returns immediately when err is nil, so
that HTML was saved as the file and rclone exited 0. With the VFS
cache those bytes were then served again.
Non-2xx download responses are now converted with the backend error
handler, which closes the body. Statuses in retryErrorCodes (429 and
5xx, including 502) are retried by the pacer. Any other status is
returned as an error. Successful 200 and 206 responses are unchanged.
gofakes3 adds the time of the request as "Last-Modified" to the
metadata it passes to PutObject, CopyObject and multipart uploads.
storeMeta cached it with the rest, so HeadObject and GetObject
replaced the object's modification time with the upload time until the
cache entry was evicted, while ListObjects reported the right time.
storeMeta now drops "Last-Modified" before caching the metadata.
Nick Craig-Wood