Cellar is the S3-compatible object storage service of Clever Cloud,
hosted in France.
rclone's defaults work for it: virtual-host style addressing, v2
listings, URL-encoded listings and BucketAlreadyExists behave like AWS.
Cellar does not apply Object Lock headers sent with
CreateMultipartUpload, so object_lock_supported is set to false and the
Object Lock integration tests are skipped. The docs tell users to set
--s3-object-lock-set-after-upload, which applies the lock on multipart
uploads with separate API calls.
The SSE-C options are exposed since Cellar supports customer-provided
keys. The docs note that Cellar rejects CopyObject with an SSE-C source,
so --no-update-modtime is needed and encrypted objects smaller than
--s3-copy-cutoff can't be moved server-side.
The integration tests pass against cellar-c2.services.clever-cloud.com.
Auto Drive is permanent storage on the Autonomys Network, with an S3
compatible API at s3.auto-drive.autonomys.xyz.
Storage is permanent: a delete hides an object, and every version is
kept. The provider sets:
- force_path_style, as the API uses path-style requests.
- copy_cutoff to the maximum. UploadPartCopy is not supported, so
server-side copies always use a single CopyObject.
- object_lock_supported false. Object Lock is always on in COMPLIANCE
mode and can't be set by clients, so rclone's Object Lock tests
can't pass.
- No Purge. Deleting a version is always refused, so the S3 Purge,
which deletes every version, fails. rclone's generic purge hides
the current version of each object instead.
- no_check_bucket. There are no bucket endpoints: buckets are created
by the first upload into them, and a CreateBucket request would be
stored as an object.
When a remote is configured with RCLONE_CONFIG_<NAME>_<OPTION> the remote
name is only converted to uppercase, so characters such as "-" and "." are
not changed to "_" and may not be settable in some shells.
Explain this in the environment variables section and link to it from
"Valid remote names".
Before this change, since v1.74.0 (bb78eb8ab), the rc `sync/bisync` call
read the `backupdir1` and `backupdir2` parameters (the names used before
they were renamed to `backupDir1` and `backupDir2`) only when the new name
was also given with an invalid value, because the fallback sat inside an
`rc.NotErrParamNotFound(err)` check, which is true only for an invalid
value. Otherwise, `backupdir1` and `backupdir2` had no effect.
The same change made an invalid value for any optional parameter (for
example `"resync": "yes"`, or a `filtersFile` that isn't a string) be
treated as if the parameter were missing, instead of returning an error.
After this change, the old backup dir names are read when the new ones are
missing, and an invalid value returns an error again, as it did before
v1.74.0.
Before this change, bisync treated a `--max-delete` above 100 as 100%, so
the `--max-delete` check never triggered. A run that left no file on a side
unchanged was still stopped, by the "all files were changed" or empty
listing checks, but one that deleted most files while leaving even one
unchanged was not.
For bisync, `--max-delete` has always been a
percentage, "in range 0..100" as Ivan put it when he designed it
(https://github.com/rclone/rclone/pull/5164#discussion_r633470815), but the
range was never enforced. So a value meant as a file count, such as
`--max-delete 500`, turned the check off. Now that the rc takes
`--max-delete` from the global config too, that includes a `--max-delete`
the rcd itself was started with.
After this change, a `--max-delete` above 100 is an error, as the rc's
`maxDelete` parameter already was.
Before this change, `--max-delete` and `--dry-run` were read from the
global config by `applyContext`, which only the command line called. The
rc `sync/bisync` call started from empty `Options` and never ran it, so it
ignored both settings:
- A dry run requested through the global option (`_config`
`{"DryRun": true}` or the flat `dry_run` parameter) was ignored.
`setDryRun` resets `ci.DryRun` from `opt.DryRun` before copying, and
`opt.DryRun` was only set by bisync's own `dryRun` parameter.
- When the `maxDelete` parameter was omitted, the `--max-delete` limit was
0%. A `max_delete` set through `_config` or the flat parameter was not
used for that limit, and applied to the inner syncs as a file count
instead.
After this change, `Bisync` runs `applyContext` itself, on its own copy of
the config, so the command line, the rc, and any other caller follow the
same rules, and `Options.MaxDelete` and `Options.DryRun` are set by
`Bisync`. With no `--max-delete` anywhere, the limit is 50%, as #5680 says
the rc should.
The rc's `maxDelete` and `dryRun` parameters now set the call's config.
`maxDelete` overrides the global `--max-delete`. `dryRun: true` asks for a
dry run, but `dryRun: false` doesn't cancel one from the global config, so
the run is a dry run if either asks for one. The `maxDelete` parameter is
also documented again; it was left out when the rc help became generated
from the command's flags.
Before this change, when bisync updated a listing entry, it kept the old
modtime if the new one was less than a second earlier. This dated back to
when bisync truncated modtimes to the destination's precision before
recording them (`ConvertPrecision`), so that a truncated time would not
overwrite the more precise original.
`ConvertPrecision` was removed in 956c2963f, and since then, bisync has kept
the original precision and relied on the modify window when comparing. But
the rule in `ls.put` was left behind, and it then fired only when a file
really had an earlier modtime, recording a time the file did not have. This
could cause:
- a `--resync` to abort with "path1 and path2 are out of sync" when Path1's
version was slightly older than Path2's
- later runs to report changes that didn't happen, for a file changed to a
version less than a second older, or copied from a remote with lower
modtime precision within the same second as the old version (for example,
from SFTP to local)
After this change, bisync records the modtime it is given, like any other
update. Comparisons already account for differences in precision through
the modify window.
Before renaming a sync conflict, bisync checks whether the two files are
identical. It compares hashes using check or cryptcheck, and as a last
resort, when neither is available, it compares the files by downloading
them (--download), so that the result is certain regardless of checksum
support.
Before this change, the fallback to --download happened only when bisync
knew up front that the remotes had no usable hash. It did not happen
when a hash that was expected to be present turned out to be missing
(for example, a backend that normally supports hashes returned a blank
one, or cryptcheck could not get the underlying hash). Those files were
counted as identical based on size alone.
After this change, a missing hash falls back to --download, the same as
when the remotes have no hash in common. --size-only and
--ignore-checksum still disable the fallback and keep the size-only
comparison.
Co-Authored-By: ASRodrigo1 <45823308+asrodrigo1@users.noreply.github.com>
Before this change, when a file was new or changed on both paths and the pre-sync
conflict check deemed the two copies identical, bisync skipped the copy and then
rechecked the file after the sync to get its listing info. If the recheck could
not confirm equality (for example, hashes unavailable due to --size-only or
(--ignore-checksum + no common hash)), the file was rolled back to be retried on
the next run, and every later run repeated the same check with the same result.
A later change on one side was then handled against a missing or stale listing
entry.
After this change, files we've declared identical pre-sync are no longer
rechecked post-sync. The pre-sync conflict check now also returns the values it
compared, and a skipped file is recorded from the listing snapshot. If either
path changed between the snapshot and the check, the prior listing is kept and
the next run rechecks.
When --size-only or (--ignore-checksum + no common hash) are in use, the "new or
changed on both paths and equal" case skips the conflict check entirely, because
it is unnecessary. The check could only compare sizes, which we already know are
equal from the snapshot.
The new `test_equal_skipped`, `test_conflict_window`, and
`test_conflict_window_size` scenarios cover identical files that change on
one path during the sync, files that change between the listing snapshot and
the check for identical files, and `--size-only` files with the same size but
different content.
Co-Authored-By: ASRodrigo1 <45823308+asrodrigo1@users.noreply.github.com>
Before this change setting LogLevel in the _config parameter of an rc
call had no effect as both the log functions and the log handler only
looked at the global log level.
The context aware log functions now use the log level from the config
in the context, and the log handler lets a log level in the context
which differs from the global one override its own level. This means
an rc call can be made more or less verbose than the rest of rclone
with, for example
"_config": {"LogLevel": "DEBUG"}
This applies to the logs made with the context aware log functions, so
not to those made by the backends which still use the global log level.
Note that a config copied into a context no longer follows later
changes to the global log level.
Several links on rclone.org pointed at anchors or pages that don't exist:
- bisync: --color and --dry-run anchors, relative commands/rclone_check/ links
- ftp: on-the-fly link pointed at a local anchor instead of /docs/
- faq: --sftp-http-proxy linked to the ftp page instead of sftp
- index: Arvan Cloud and Liara provider config anchors
- rc, sftp: Hugo drops leading dashes from heading ids (#--rc-no-auth -> #rc-no-auth)
- onedrive: webdav #sharepoint -> #sharepoint-online
- webdav: relative commands/rclone_serve_webdav/ link
- s3: https:/// typo in the China Mobile Ecloud link
Before this change, when an existing file had changed on both paths, the
conflict winner was chosen using only the attributes that had changed on each
side. An attribute that hadn't changed was treated as zero, so --conflict-resolve
larger could pick the smaller file (for example, when one side only had its
modtime updated), and --conflict-resolve newer could fail to pick a winner when one
side's modtime hadn't changed.
After this change, the current size, modtime, and checksum (to the extent
enabled with compare settings) are now recorded for every changed file.
Before this change, when --conflict-resolve picked Path2 as the winner, the winning
file was copied to Path1 correctly but then removed from both listings while
updating them for the renamed (or deleted) Path1 copy. The next run saw the file
as new on both paths and had to check it again, and a change to it on one path
in that run was handled as if the file had never been synced. When Path1 won,
the listings happened to be updated in an order that avoided this.
This change fixes the issue by adding a guard to remove the file from the
listing only when dstNewName != dstOldName.
See #9939
The manual release verification steps checked the signature with gpg
--verify and then checked the binaries against the whole SHA256SUMS
file. gpg only verifies the clearsigned part, so an unsigned hash line
added outside it would be used by sha256sum -c or by rclone hashsum -C.
The instructions now extract the signed hashes with gpg --decrypt and
check the binaries against those only.
See: GHSA-w33h-3qgq-8mv8
The transport strips the headers set with --header once a redirect
chain leaves the originally requested host, but the host comparison
ignores the URL scheme. A server which redirects https://host:8443/
to http://host:8443/ keeps the same host and port, so the headers
were added to the plaintext request and could be read by anyone on
the network path.
The redirect chain walk now also treats a hop which downgrades the
original HTTPS request to HTTP as leaving the host, so the headers are
removed from that hop and every one after it.
The transport also logs at debug level when it removes the headers,
for either reason, so that a resulting 401 from the redirect target
can be diagnosed.
The http backend resolves the path after `remote:` against the
configured url according to the URL standard. A path beginning with
`//` is a network-path reference under that standard, so a path such as
`//other.example/dir/` replaced the configured host (and port and
userinfo) rather than being treated as a path. The configured headers
were then sent to the substituted host on the initial HEAD request and
every request after it.
Use rest.URLJoinRoot, which refuses any root carrying a scheme, host
or userinfo of its own, to join the path onto the configured url.
Before this change the basic, htpasswd and auth proxy authentication
middlewares let every OPTIONS request through without checking
credentials so that browser CORS preflights, which can't carry
credentials, would succeed.
However in serve webdav the OPTIONS request is answered by the WebDAV
handler whose Allow header depends on whether the path is an existing
file, an existing directory or doesn't exist, so an unauthenticated
client could discover the existence and type of any path on the served
remote even with --user/--pass or --htpasswd set.
After this change he authentication middlewares no longer special-case
OPTIONS. Instead the CORS middleware, which runs before authentication
and owns the --allow-origin setting, answers genuine CORS preflights
itself with an empty 200 carrying the CORS headers only when
--allow-origin is set.
All other OPTIONS requests need credentials like any other method, so
clients which probe with OPTIONS before authenticating will get a 401
challenge and must retry with credentials.
A symlink in an archive was listed as an ordinary file holding the link
target by the zip backend, and dropped entirely by the squashfs one.
Symlinks are now skipped unless -l/--links is in use, as they are on the
other backends. With that flag they are listed with the .rclonelink
suffix and read back as the path they point at, so they can be copied to
a destination which supports symlinks.
This changes what the zip backend lists by default: a symlink used to
appear as a small file whose contents were the link target, and is now
skipped with a NOTICE naming the flag.
Fixes#9569
Add a new backend for dosya.dev, a cloud file storage, sharing, and
synchronization platform with workspace-based multi-tenancy.
Features:
- Server-side copy, move, rename (files and directories)
- Purge (recursive directory delete)
- Recursive listing (ListR) for --fast-list
- Multipart upload for large files (>10MB)
- Public link sharing via rclone link
- About (workspace storage quota)
- MIME type detection
- Empty directory support
- Workspace-based multi-tenancy (one remote per workspace)
- Download via presigned R2 URLs with Range header support
Co-authored-by: firatkaya <firat@netiket.com.tr>
Files.com now treats the replacement characters rclone uses for / and
\ in file names (／ and ＼) as if they were the originals, and the
names ． and ．． as if they were . and .., and rejects them all with
"Invalid path". There is no other way of encoding these names, so
document the restriction and ignore the integration tests for them.
With -l/--links a .rclonelink object is buffered in memory to become
the target of a symlink. The read was unbounded, so a hostile or
corrupt source serving a large .rclonelink object made rclone use that
much memory and then log the whole body in the resulting error.
A symlink target can never be longer than a path, so the read now stops
at 128 KiB and anything longer is refused without retrying.
The docs describe a duration as a sequence of numbers each with a unit
suffix, including d, w, M and y, but a sequence that used one of those
units, such as --max-age 1d12h, failed with a confusing error about
parsing it as a date. Only a single number with one of them, like 1.5d,
or a sequence of the time.ParseDuration units was accepted.
Parse such sequences before falling back to dates, rejecting ones too
long to represent.
Add an opt-in preflight that excludes only guaranteed tracked renames
from the max-delete count. Reuse normal bisync listing metadata,
validate strategy and flag conflicts early, and skip preflight work
when --force bypasses the safety check.
The max_delete_track_renames scenario uses the default hash
track-renames strategy, which requires a hash common to both paths.
Skip it on remote combinations without one, such as crypt, instead of
failing the integration tests.
Fixes#8685
Reading an object in an archive storage class failed with "Object in
GLACIER, restore first" even when the object was in DEEP_ARCHIVE or in
an Intelligent-Tiering archive access tier.
Use the storage class and access tier from the InvalidObjectState
error. The storage class is optional in that error, so fall back to
the storage class from the listing or HEAD, and then to GLACIER as
before.
Rclone always asked for the delta listing of the whole drive and threw
away the items which weren't under the directory being listed.
The delta API now works on any folder, so ask for the delta listing of
the directory being listed instead, falling back to listing from the
root of the drive for drives which only support delta there.
Shortcuts to shared items (and Personal Vault) are stored as remote
items pointing at another drive and listing them can fail with
"The provided drive id appears to be malformed". This aborts a
--fast-list listing of the whole drive.
Testing against OneDrive personal (free) and OneDrive for Business shows
- personal accounts now create versions on setting the modification
time and can delete them, so --onedrive-no-versions and rclone
cleanup work there
- --onedrive-link-password works on OneDrive for Business
- personal free accounts can't set a link password or --expire
- --onedrive-link-type embed only works on OneDrive personal
- personal accounts store times with 1s precision, not mS
See #9917
With -L/--copy-links a symlink pointing to one of its parent
directories was followed until the OS gave up with "too many levels
of symbolic links" 40 levels deep. As every looping symlink multiplies
the listing, a small tree with three such symlinks listed millions of
entries.
When a followed symlink points to a directory, compare it with the
directory being listed and each of its parents using os.SameFile and
if it matches report an error and skip it, the same way as a circular
symlink. A symlink to a sibling directory is still followed. Loops
excluded by the directory filters are skipped quietly as before.
Use OCI BatchDeleteObjects to remove up to 1,000 objects per request when
purging. Include directory markers so purging a bucket leaves it empty for
deletion.
Use OCI RenameObject for moves within a bucket, avoiding a server-side copy
followed by source deletion. Cross-bucket moves continue to fall back to copy
and delete.
The link pointed at a bare relative path (--check-filename) instead of
the in-page anchor for the ### --check-filename heading further down the
page, so it 404s on the rendered docs site. Point it at #check-filename
(Hugo strips leading dashes when generating header anchors).
---------
Co-authored-by: no-hup <19599684+no-hup@users.noreply.github.com>
David Legrand